DataDecide: How to Predict Best Pretraining Data with Small Experiments
Authors: Ian Magnusson
Nguyen Tai
Ben Bogin
David Heineman
Jena D. Hwang
Luca Soldaini
Akshita Bhagia
Jiacheng Liu
Dirk Groeneveld
Oyvind Tafjord
Noah A. Smith
Pang Wei Koh
Jesse Dodge
Venue: ICML 2025 poster (2025)
Keywords: Pretraining Data
Language Models
Scaling Laws
Evaluation
Benchmarks
Data Recipes
Data Ablations
Primary area: deep_learning->large_language_models
TL;DR: When do pretraining experiments at small scales correctly predict the ranking of different data recipes at larger scales as measured by downstream performance?

Abstract: Because large language models are expensive to pretrain on different datasets, using smaller-scale experiments to decide on data is crucial for reducing costs. Which benchmarks and methods of making decisions from observed performance at small scale most accurately predict the datasets that yield the best large models? To empower open exploration of this question, we release models, data, and evaluations in DataDecide—the most extensive open suite of models over differences in data and scale. We conduct controlled pretraining experiments across 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random seeds. We find that the ranking of models at a single, small size (e.g., 150M parameters) is a strong baseline for predicting best models at our larger target scale (1B) (∼ 80% of comparisons correct). No scaling law methods among 8 baselines exceed the compute-decision frontier of single-scale predictions, but DataDecide can measure improvement in future scaling laws. We also identify that using continuous likelihood metrics as proxies in small experiments makes benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval > 80% predictable at the target 1B scale with just 0.01% of the compute.

DataDecide
How to Predict Best Pretraining Data with Small Experiments
Jan Magnusson Nguyen Tai Ben Begin" David Heineman" Jena Hwang" Luca Soldaini
Akshita Bhagia
Jiacheng Liu" 12 Dirk Groeneveld Oyvind Tafjord'
Noah A. Smith
Pang Wei Koh
Jesse Dodge
Abstract
1. Introduction
The cost of training large language models (LMs) necessi- tates methods of trying out options at small scale, but it also makes it expensive to validate the accuracy of development decisions made with such methods. We focus on the ques- tion of choosing between pretraining datasets to use-one of the most impactful development decisions. Common practice (e.g., Li et al., 2024) uses single, small scale of experiments to cheaply test pretraining data intended for larger-scale models, where scale is determined by number of model parameters and training tokens. The other predom- inant approach to fit scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022; Choshen et al., 2024) to the trend in performance observed over multiple small scales, with recent work extending this to the prediction of downstream performance instead of language modeling loss (Gadre et al., 2024; Dubey et al., 2024; Bhagia et al., 2024).
Because large language models are expensive to pretrain on different datasets, using smaller-scale experiments to decide on data is crucial for re- ducing costs. Which benchmarks and methods of making decisions from observed performance small scale most accurately predict the datasets that yield the best large models? To empower open exploration of this question, we release mod- cls, data, and evaluations DATADECIDE- the most extensive open suite of models over differ- ences in data and scale. We conduct controlled pretraining experiments across 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random seeds. We find that the ranking of models at a single, small size (e.g.. 150M param- eters) is a strong baseline for predicting best mod- els our larger target scale (IB) (~ 80% of com- parisons correct). No scaling law methods among 8 baselines exceed the compute-decision fron- tier of single-scale predictions, but DATADECIDE can measure improvement in future scaling laws. We also identify that using continuous likelihood metries as proxies small experiments makes benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval > 80% predictable at the target 1B scale with just 0.01% of the compute.
So far decision-making approaches have only been validated without observing the counterfactual outcome, either by pro- ducing single large model on the chosen decision with impressive performance or by low error in predicting the magnitude of observed performance of small number of large models. Knowing what amount of error in predicting performance over scale is low enough to actually make correct decision among datasets, requires a suite of com- parable models trained on many datasets. Although a wide variety of open-source pretraining corpora are available, the scaling behavior of data difficult to assess from off-the- shelf models that vary simultaneously in data, optimizer, and modeling decisions.
To make it possible to empirically study what methods make the best decisions over data, we build DATA DECIDE suite of models we pretrain on 25 corpora up to 100B tokens, over 14 different model sizes ranging from 4M parameters up to 1B parameters (more than 30K model checkpoints in total). We evaluate all models across suite of 10 down- stream tasks and calculate how accurately small models pre- dict which pretraining corpora lead to better performance
'Equal contribution Allen Institute for AI *Paul G. Allen School of Computer Science & Engineering, University of Wash- ington *University of Pennsylvania. Correspondence Ian Mag- nusson Zianmag@ess fraammn@@s..wsshingnoo..co washington.edu>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
'DataDecide collection on HuggingFace

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Compute (FLOPs)
7e+15
7e+16
7e417 7e+18 7e419
7e+20
Targets
Predictions
Pretrain 25 datasets @ 150M to predict
Prediction Method
0.9
4M
Best
Best
pairs of 25 datasets
6M
Data:
Data:
0.8
@ 1B -80% correct
8M
1. Dolma
1. DCLM
10M
2. DCLM
?
2. Dolma
Decision Accuracy
14M
0.7
3....
3.
16M
20M
0.6
90M
Evaluation
(Proxy)
150M
0.5
Evaluation
300M
530M
Seeds
0.4
750M
1B
Smaller Scale(s)
Multi-Scale Fit
0.3
Large Scale
10-5
10
10
10
10
10
Proportion of Target Compute (%C)
Figure 1. Which pretraining data to use? Ideally, compare performance of large models with fixed configurations averaged over random seeds (left). In practice, cheaper, smaller-scale experiments used (center). Here DATADECIDE measures accuracy of pairwise decisions between 25 pretraining corpora to find efficient prediction methods (right).
at our largest scale. Our conclusions provide practical rec- ommendations for the best benchmarks, prediction methods, and metrics to use to make decisions.
deviation.
Measuring the tradeoff' of compute cost to better decisions lets us make the following recommendations about small experiments for making data decisions:
call the corpora we train on data recipes as they range across popular corpora including Dolma (Soldaini al., 2024), DCLM (Li al., 2024), RefinedWeb (Penedo al., 2023), C4 (Raffel al., 2019), and Web (Penedo et al., 2024) as well combinations of interventions on these datasets such as source mixing, deduplication, and filtering. Previous work has considered only (Biderman et al., 2023) or 6 recipes (Magnusson et al., 2024; Brand- fonbrener et al., 2024). We also offer novel affordance by including random seed reruns for even our largest runs, to help quantify whether variation occurs due to random initialization and data order or differences in the distribution of data.
$3.1 - The amount of compute you need to allocate for given decision accuracy depends heavily on task. MMLU and ARC are much cheaper to predict than Hel- laSwag and some tasks such as SocialIQA are difficult predict all scales.
$3.2 - 8 baseline scaling law methods do not exceed the compute to decision accuracy frontier set by ranking single scale experiments.
$3.3 - At small scales, continuous metrics using an- swer likelihood are better or equivalent predictors of decisions than using the same discrete accuracy target metric.
Concretely, DATA DECIDE allows analyses such as Figure (right), which shows the relationship between compute used to predict a ranking of datasets and how accurately that rank- ing reflects mean performance over seed runs (quantified here by OLMES; Gu et al., 2024) for models fully trained on those datasets at the target (1B) scale. We measure the accuracy of decisions as the percent of compared pairs of datasets where the prediction identifies the correct winner. Each point represents the average decision accuracy of given method over prediction attempts using small models with different random seeds, and shading shows standard
$3.4 - Better decisions can be explained in part by low run-to-run variance and a wide spread of benchmark performance values for different data, traits which can be improved by proxy metrics.
Future research can extend DETADECIDE with little extra compute by running new evaluations on our checkpoints, pretraining additional small models to compare against the large target models we provide, or trying new prediction
2

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Source/ / Recipe
Description
Dolmal.7 Original, No code, No math/code, No Reddit, No Flan
A 2.3T-token corpus (Dolma 1.7 Soldaini al., 2024) sampling common LM sources for open research. We ablate code, math/code, Reddit, or Flan subsets.
Dolmal.6++ Original
Dolma 1.6 plus additional sources from Dolma 1.7: RedPajama's arxiv subset, openwebmath, algebraic stack, flan, stareoder, falcon.
C4 Original
The C4 dataset (Raffel al., 2019) as prepared in Dolma 1.7, heuristically filtered from the April 2019 Common Crawl.
FineWer-Pro Original
The WineWeb Pro corpus (Zhou et al., 2024), featuring model-driven data cleaning on FineWeb.
FineWeb-Ede Original
The deduplicated FineWeb-Edu subset of SmolL (Ben Allal et al., 2024), focused on educational web pages.
Falcon Original
The Falcon WefinedWeb corpus (Penedo et al., 2023) in Dolma 1.7, derived from Common Crawl through June 2023 and more aggressively fillered/eduullictteed than C4.
CC Original, QC 10%, 20% QC Orig 10% QC Tulu 10%
Falcon and Dolma 1.7's Common Crawl. We quality filter top 10% or 20% documents with reproduced or original (Li et al., 2024) filter or retrain filter on pre-release version of Tuluu-3 (Lambert et al., 2024).
DCLM-Baseline Original, QC
SOTA Common Crawl corpus using best ablated deduplication, cleaning heuristics, and quality filter. We quality filter to top 7% of DCLM classified documents and further take 2+ or 3+ scores with FineWeb-ee classifier; or filter to top 3% or 10% with FineWeb-ede classifier; or take top 10% or 20% with reproduced DCLM classifier.
7% FW2. QC 7% FW3, QC FW
3%. QC FW 10%, QC 10%, QC 20%
X% WCLM-Baseling + 1 1 A% Dolma1.7
Fractional combinations of Dolmal.7 and SCLM-Baseling mixing different proportions of the two datasets for E (25%, 50%, 75%).
Table DATA enables the study of data differences over scales through controlled pretraining experiments on 25 data recipes. These take different source datasets and apply interventions from ablating domains, deduplication, mixing, to quality filtering with different classifiers and thresholds. We release all pretraining corpora, as well as models trained on cach recipe and each of the 14 model configurations in Table 2 with 3 random seeds.
methods with lightweight manipulations such as smoothing and curve fitting on top of our released evaluation results.
"Chinchilla" (5 C) optimal ratio (Hoffmaan al., 2022) captures the typical overtraining favored for inference sav- ings.
2. Methods
All IB (target size) models have 3 full reruns with differ- ent seeds, while other model sizes have second and third seed runs that are terminated early after 25% of the target compute budget. We train the 1B reruns all the way to com- pletion to allow our target "gold" predictions to account for run-to-run variance in evaluations due to weight initializa- tion and data order. For instance, we find that the standard deviation between runs at the IB 5xC' C' scale can be as high as 2% points of accuracy for some recipes on most tasks. Mcanwhile, at the non-target scales we wish to make pre- dictions with a small fraction of the target compute, so we avoid reruns that would use an impractically large prediction budget.
Our aim is to empirically test the predictability of down- stream performance at larger, target scale using small experiments. We describe DATA 12.1, the predic- tion methods we examine $2.2, the metrics we use assess predictions $2.3, how we measure downstream performance $2.4, and proxy metrics for our performance evaluations $2.5. We will release all models, checkpoints, pretraining corpora, and evaluations.
2.1. The DATADECIDE Suite
We pretrain a suite of 1,050 models using 25 data recipes x 14 model scales x 3 random seeds for initialization and data order. Table describes the 25 data recipes included in DATADECIDE that aim to provide coverage of common data preparation choices such as deduplication, ablating domains, mixes of existing datasets, as well quality filters with different implementations, training data, and thresholds for quality classifiers.
Whether for extrapolating scaling laws or ranking single scale experiments, it is important to select reasonable hy- perparameters for each scale to avoid confounding in per- formance differences that are simply due to suboptimal hy- perparameters. We use OLMo's model ladder (Groeneveld et al., 2024; OLMo al., 2025; Bhagia al., 2024) pro- grammatically create LM pretraining configurations for specified parameter size and token- aaken-paramete ratio to enable
We select a token to parameter ratio of 100, which at 5x
3

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
running grid of model scaling experiments. The model lad- der uses heuristics from the literature (Porian et al., 2024) to set global batch size and learning rate based on scaling fac- tors. The hyperparameters that determine parameter count (layers, hidden dimension, number of heads, MLP dimen- sion) were handpicked by OLMo developers for each scale to achieve the desired number of parameters. Appendix Table 2 details the configurations of all our models.
cost. To account for step-to-step noise in evaluation we average the last 10% of checkpoints as the final observed loss. Equation 2, however, is fit on all observations includ- ing intermediate checkpoints. We explore variations for a total of 8 multi scale approaches defined in Appendix C; none of these make for substantially better decisions than the method defined this section.
2.3. Prediction Metrics
2.2. Prediction Methods
Our predictive task is to forecast which of a pair of data recipes will perform better at some target scale based on small-scale experiments. We use the following metrics to measure the quality of these predictions.
Broadly, there are two approaches in the literature to pre- dicting large-scale performance based on small-scale exper- iments. We use straightforward implementations of each to assess where they succeed and fail at making decisions about which data recipes to use.
Prediction Error Scaling laws literature (Bhagia et al., 2024; Gadre et al., 2024) typically evaluates success from predicted and actual downstream performance, us- ing relative error [predicted acctuul
Ranking Single Scale Experiments (Single Scale) This simple approach is employed by work such as Li al. (2024) and consists of running a set of ablations or experiments over data recipe options while holding constant all other model- ing variables including scale. The winning data recipe by downstream accuracy (or proxies) at the small experimental scale is assumed to extrapolate to the target scale.
x 100%) or absolute error (ppredicted - actuall x 100%). We call these absolute or relative "prediction error" to distinguish from the following metric.
actual
Decision Accuracy Unlike previous work, we also mea- sure the impact of predictions decisions about which data recipe is better than another. The metric we use to capture this is decision accuracy, accuracy over all pairs of data recipes A and B where either A or B is defined as the cor- rect winner based on which achieves higher performance at the target scale. This is nearly equivalent to Kendall's т, but ranges from 0 to 1. We define the target-scale winner based on mean downstream performance over 3 random seeds. Thus decision accuracy can be formalized as follows. Let P be the set of all data recipe pairs (A, B) with observed mean performance yA. Vn and predicted performance 9 DR, re- spectively, then decision accuracy is:
Extrapolating Scaling Laws (Multi Scale) Another ap- proach to making decisions with predictions across scales used in works such as Dubey et al. (2024) is to fit scaling laws multiple small experiments across range of scales for each of the data recipes. The winning recipe is decided as the one whose scaling law shows the highest extrapolated performance at the target scale. Although scaling laws were first observed for language modeling loss (Kaplan et al., 2020; Hoffmann et al., 2022), they have been extended to predict downstream performance through a two-step ap- proach that also fits a function from loss to downstream performance (Gadre et al., 2024; Bhagia et al., 2024). We follow method from Bhagia et al. (2024). Their proposed approach incorporates separate parameters for number of model parameters and number of tokens trained to account for over or undertrained models. But as our suite only in- cludes one token- perem-parrmeter ratio, we use the simplified 3 parameter baseline, L(C'), as a first step which we chain with second step, Acc(L), defined as follows where A, a, E, o. b. k. Lo are optimized parameters:
P L(sigie (B) Sign(ya -yB) (3)
Percent of Target Compute Budget (%C) We measure compute in terms of theoretical FLOPs following the sim- plifying assumption made in most scaling literature that the costs associated with training a model are captured well enough by FLOPs 6NV D. based solely on the number of parameters (N) and tokens trained (D) (Kaplan et al., 2020). We consider the efficiency of a prediction based on the ratio of the experimental budget and the target budget in FLOPs, %CC = ? x 100%.
A
I(C)) = C
+ E
a
Acc(L)- 1 k(L-Loo)
+6
(2)
2.4. Performance Evaluation with OLMES
Following Bhagia et al. (2024) we fit Equation I only on observations of final, fully trained checkpoints as account- ing for the learning rate schedule's impact on intermediate checkpoints would require further parameters in the equa- tion increasing the required number of observations and
We use the OLMES suite of 10 multiple choice question an- swering benchmarks (Gu et al., 2024): MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019), ARC Chal- lenge (Clark et al., 2018), ARC Easy (Clark et al., 2018),
4

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
PIQA (Bisk et al., 2020), CommonsenseQA (Talmor al., 2019),SocialIQA (Sap et al., 2019), OpenBookQA (Mi- haylov et al., 2018), BoolQ (Clark et al., 2019), and Wino- Grande (Sakaguchi et al., 2020). These tasks are well suited for the model scales we examine with all but BoolQ receiv- ing non-trivial performance. Unless otherwise noted, we consider the macro average of these ten tasks. The under- lying metric for each task is accuracy, for which OLMES specifies different length normalization scheme per task. Our target "gold" rankings which we aim to predict are al- ways based on the "cloze" formulation (CF) accuracy with curated normalization per task, which we refer to as Accu- RACY. We diverge from OLMES only in that we make use of all available items in the specified split of each bench- mark rather than subsampling them, to reduce variance over the task distribution.
3. Results
3.1. What is the best way to spend compute for data decisions?
More compute makes better decisions. Decisions from intermediate checkpoints are as good as com- pute equivalent final checkpoints. The amount of compute needed to make good predictions varies between tasks. ARC and MMLU are predictable with much less compute than HellaSwag. The rest of OLMES tasks give markedly less reliable predictions across the scales we examine.
First looking at the aggregation of all 10 OLMES tasks (Figure I right), we see that there is a positive and roughly log-linear relationship between experimental compute and decision accuracy. Specifically, this figure illustrates the relationship between the compute used for predicting best data recipes and the decision accuracy those predictions achieve against targets ranked by OLMES performance at the IB scale. Each point represents the average decision accuracy over three runs with different random seeds, with shading indicating standard deviation. Points with the same color show all intermediate checkpoints from a given param- eter size. The color shows each model size for predicting using ranking single scale experiments. The stars show pre- dictions from extrapolating scaling laws using our default 3- p-rammeter approach, the details of which are discussed further in $3.2.
Note that while we focus just on OLMES multiple choice evaluations in this work, our method of validating decisions made through predictions can be applied to other bench- marks. We chose these tasks based on their appropriateness to our range of model scales, and one would have to se- lect different tasks when targeting larger scale. Moreover, DATADECIDE could be used to identify new evaluations that are sensitive within our range of scales.
2.5. Proxy Metries for Performance Evaluation
Previous work has noted how discrete metrics such as curacy cause jumps in performance across scale that otherwise see more predictable improvements with scale for continuous metrics (Schaeffer et al., 2023). We experiment with using continuous metrics at small scale as proxies of the accuracies selected by OLMES for each task (ACCURACY) at the target scale to improve decision accuracy. We use the following metrics: CORRECT PROB is the average probabil- ities of the correct continuations. MARGIN is the average difference between the probability of the correct continu- ation and the most likely incorrect continuation. NORM CORRECT PROB is the average probability of the correct continuation conditioned on the response being in the set of correct or incorrect continuations. TOTAL PROB is the average of the sum of probabilities of all correct and incor- rect continuations. ACCURACY is the fraction of instances where the correct continuation has the highest probability. Each of these can be computed with likelihoods normal- ized by number of tokens or characters; unless otherwise specified we use character length normalization. Appendix Table 3 shows formal definitions.
The ease of prediction is greatly influenced by which evalu- ation benchmark we use. In Figure 2, we show the relation- ship of compute and decision accuracy for each of the tasks in OLMES individually. The predictive sensitivity of tasks at given compute varies significantly, with ARC Easy be- ing consistently predictable with 5 orders of magnitude less compute and BoolQ only reaching beyond trivial decision accuracy for intermediate checkpoints of the target runs. HellaSwag, SocialTQA, WinoGrande show distinct periods of insensitivity followed by roughly log-linear increase after hitting some compute threshold.
3.2. How does extrapolating scaling laws compare to ranking single scale experiments?
A selection of 8 baseline scaling law methods are no more efficient than ranking single scale experi- ments. Future scaling law methods can be assessed on DATADECIDE.
Figure contrasts different approaches to fitting scaling laws over multiple scales of small experiments. Each of the approaches is shown in a different color. Multi-scale predictions have a compute budget equal to the training
5

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
ARC Challenge
ARC Easy
BoolQ
CommonsenseQA
HellaSwag
1.0
Decaion Acorncc
0.8
0.4
OpenBookQA
PIQA
SocialIQA
WinoGrande
MMLU
1.0
Disisen Recunee
0.8
0.4
10-1 10-4 10-1 10-1 10-1 10t
10-1 10-1 10-1 10-1 10-1 Proportion Target Compute 196C)
10"
10-h 10-4 10-1
10-1 10-1 10
10- 10-4 10-
10-3 10-1 10
10-1
10-4 10-1
10-7 10-1 10"
Proportionn of Target Compute (SC)
Proportion of Target Compute (SS)
Proportion barget Compute (SC)
Proportion of Tarpet Compute (SCC)
Figure 2. Accuracy in pairwise decisions on best data when evaluating on the 10 OLMES tasks with ACCURACY (shown aggregated in Figure Specific tasks have very distinct ranges of sensitivity, with like ARC Easy being predictable small scales and others like HellaSwag requiring substantially more compute to predict.
0.9
Prediction Methed
Single scale
Decision Accuracy
0.7
parameter
3-parameter with helper points
3- 3-parameter step 2 rit with >-50 % checkpoints
0.6
parameter helpers
checkpoints
S-parameter
S-parammttrr, single step
3-parameter, single step
2-parameter
0.4
0.3
10a
Proportion of Target Compute (9C)
Figure 3. Decision accuracy baseline scaling law variants. At best, these approaches reach only the compute decision accuracy frontier as ranking single scale experiments. DATADECIDE can be used to iterate on future scaling law prediction methods.
6

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
cost of the model sizes used make the prediction. We try the following combinations of models sizes: We use [sI....., } 3 I k S 14), where s is the ordered set of sizes, to explore the improvements of progressively adding larger model sizes beyond the minimum 3 required for fit- ting. We also use ([se)...,114) 2 < & I 11} to try re- moving potentially noisy information from small models. Unlike single scale results, we make only one prediction attempt with the default fully trained random seed, as final checkpoints are required for fitting the first step of these scaling law variants but are not available for all seeds.
incorrect but presumably relevant additional answers.
We notice two very distinct types of trends over the different tasks. Either the different proxy metrics are nearly indistin- guishable and increase in decision accuracy with compute or CORRECT PROB and TOTAL PROB are flat with respect to scale and the other metrics only rise up to that level of decision accuracy towards the full target compute budget. In the last order of magnitude below the target compute Ac- CURACY and the other metries tend to overtake CORRECT PROB and TOTAL PROB, while these two metrics some- times even decrease in decision accuracy. Notably these other metrics that trend with ACCURACY include continu- ous metrics that penalize probability assigned to incorrect answers, NORM CORRECT PROB and MARGIN.
Our scaling law approaches vary in the number of parame- ters fit, using hard coded points to define the minimum and maximum performance, using only the second half of in- termediate checkpoints for fitting the second step, or fitting function directly from compute to accuracy in single step. Each of the scaling law variants are defined formally in Appendix C. The 2 and 3 parameter variants all achieve among the top decision accuracy.
3.4. How can we make evaluation benchmarks more predictable?
The decision accuracy on a task is driven in part by low run-to-run variance and a wide spread of performance values for different data recipes. Us- ing CORRECT PROB sees wider spreads reduced noise for many tasks. Using this metric enables pre- dicting rankings for code tasks that are too hard for accuracy metrics at small scales.
A priori we know that ranking single scale experiments cannot correctly predict when the scaling trend of one data recipe overtakes another at scales between our small ex- periments and target scale. Such crossovers bound the decision accuracy of this constant approximation of per- formance. Nevertheless ranking single scale experiments sets a high baseline decision accuracy, implying relatively little crossover occurs. It is difficult to distinguish evalua- tion variance from true crossovers, but the scaling trends we empirically observe cross over frequently. Improved future scaling laws may be able to advance the Pareto frontier DATADEciDE as they are not bound by crossovers.
What underlies differences in decision accuracy when bench- marks and metrics change? The evaluation must separate pairs of data recipes by an amount greater than combined noise from run-to-run variance of each of the pair's ' runs. In Figure 5, we plot tasks with a given metric using fully trained 150M models over these two characteristics: 1) the standard deviation over 3 random seed runs aver- aged over all recipes, and 2) spread- the standard deviation among the mean performance of the different data recipes. Each point also shows the decision accuracy. We see that some highly predictable tasks (e.g., MMLU) are character- ized by having low run-to-run noise, while others (c.g.. ARC Easy) widely spread the different data recipes. We also see that improvements from using CORRECT PROB often align with improvements in one of these two characteristics.
3.3. What proxy metries give better signal for predictions at small scale?
At small scales, continuous metrics using the char- acter normalized likelihood of correct or all answer options serve as better or equivalent predictors of decisions than using the same ACCURACY as used at the target scale.
As a practical application of these insights, we demonstrate that a change of proxy metric makes predictable two code tasks (Austin et al., 2021; Chen et al., 2021) that are other- wise too challenging for our small models. Figure 6 shows how decision accuracy goes from trivial to 80% when us- ing CORRECT PROB. The switch of metric allows small models get above the noise floor for these tasks, while still predicting large-scale accuracy metrics. Notably, two math benchmarks (Lewkowyez et al., 2022; Cobbe et al., 2021) do not see such a benefit. They do however give decision accuracy above 80% if we switch the target metric to CORRECT PROB, raising a question for future work to
Figure 4 shows the decision accuracy over different proxy metrics. Here we chose a single length normalization, "_PER.CHAR. Metrics follow similar trends regardless of length normalization and this one is empirically optimal for most of the tasks that we observe.
Using CORRECT PROB TOTAL PROB leads to decision accuracy at least as good as any other metric for most small scales. These continuous metrics are simple likelihoods over answer strings. In particular, TOTAL PROB may be interpretable as signal of a model having exposure to the domain of a given task in the form of higher likelihoods on

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Accuracy
Correct Prob
Margin
Total Prob
Norm Correct Prob
ARC Challenge
ARC Easy
BoolQ
CommonsenseQA
HellaSwag
1.0
Dedition Accuraty
0.8
0.4
OpenBookQA
PIQA
SocialIQA
WinoGrande
MMLU
De-cision Accuracy
6.6
15F
Proportion Target Compute (NC) Proportion Target Compute (%C) Proportion Target Compute Proportion Target Compute (%C) Proportion Target Compute (NCC)
Figure 4. Per-task decision accuracy using character normalized proxy metrics for ACCURACY targets. 5 tasks benefit smaller scales from using raw likelihood of answers (CorRect PROB and TOTAL PROB), as opposed to discrete ACCURACY or continuous metrics penalize probability incorrect answers (NORM CORRECT PROB, MARGIN).
10
1.0
Accuracy
Spread (Performance STD over Data Recipes)
Correct Prob
ARC Easy
ARC Challenge
Decision Accuracy
Opak
0.6
Commonsense
MMLU
0.4
SociallOA
10
0.2
HellaSwag
PIQA
WinoGrande
0.0
10
10 -
10
Noise (Performance STD over Random Seed Runs)
Figure 5. Why do some tasks metrics get better worse decision accuracy? At 150M with CORRECT PROB tasks like HellaSwag succeed with low run-to-run variance tasks like SocialIQA widely spread performance assigned different pretraining data.
8

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
80%
random
4M Correct Prob
Decision Accuracy
4M Accuracy
60M Correct Prob
60%-
60M Accuracy
40%
20%
0%
Minerva
GSMBK
MBPP
HumanEval
Figure 6. Code tasks such as humaneval and MBPP go from trivial decision accuracy to largely predictable when using using continuous CORRECT PROB instead of discrete ACCURACY. Meanwhile common math tasks remain near trivial decision accuracy regardless of metric.
explore whether changing the target metric can be justified.
range of scales up to 10 (Brandfonbrener et al., 2024). Our DATADECIDE offers a range 14 scales up to x 1020 FLOPs, while including an order magnitude more fine- grained data differences. Meanwhile, DCLM also makes extensive use of ranking single scale experiments drive improvement in data recipes (Li et al., 2024). They release their best data and a model trained on it, but do not release models from their decision making experiments and do not search over multiple recipes at their largest scale. Where their goal is creating a proposed best recipe, our DATADE- CIDE enables the assessment of whether method for de- cision making really does find the best among proposed recipes.
4. Related Work
Prediction Much work studies scaling behavior in lan- guage models. Initially this focused on predicting LM loss from scale determined by parameter count and tokens trained (Kaplan et al., 2020; Hoffmaan et al., 2022). Special consideration is also given to the case of data constrained scaling (Muennighoff et al., 2023; Goyal et al., 2024). Un- like predicting loss, predicting downstream performance from scale generally harder (Schaeffer et al., 2024). How- ever, recent work has demonstrated it can be done based two step prediction that chains together predictions from scale to loss and loss to downstream performance (Gadre et al., 2024; Bhagia et al., 2024; Dubey et al., 2024), some- times using training loss (Du et al., 2024) or transferring losses from different data recipes (Brandfonbrener et al., 2024; Ruan et al., 2024). The one line of work targeting pretraining data considers the special ing proportions of several data sources optimized through scaling laws (Kang et al., 2024; Ye et al., 2024). Most relevant to our work, Choshen et al. (2024) consider practi- cal methods for better scaling prediction error such as how much compute to use or whether to include intermediate checkpoints. Orthogonally to these findings, we propose way assess the accuracy of decisions made with such predictions.
5. Limitations
The scope of our work is limited to just one ratio of tokens to parameters, 100 or 5x "Chinchilla" optimal ratio (Hoff- mann al., 2022). We believe this captures the typical case, most models now favor overtraining for inference savings. Due compute limitations and the need for stan- dardized set of model configurations over long period of time in which compute became available for pretraining, we opt for 14 specific configurations from 4M-1B parameter scale. While observations across more configurations would always be better, this must be traded off with exploring the other dimensions of data recipes and random seed reruns. Likewise, while our 25 data recipes is order of magnitude more than previous suites, there is always the possibility that findings across these will not be representative of future data recipes. In our evaluations we focus on multiple choice tasks with a "cloze" formulation as we find these to a good for our range of scales. Using DATADECIDE, new evaluations can be assessed easily by others without any additional pretraining.
of deciding mix-
Suites over Data Differences DATADECIDE follows in
the footsteps of the Pythia Suite (Biderman et al., 2023)
which was the first to offer a controlled comparison of 2
data recipes, using compute scales up to 2 x 20 FLOPs.
Subsequent suites have offered 6 data recipes at 9 x 1020 scale (Magnusson al., 2024) and 6 data recipes over

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Acknowledgments
Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https: /ttp:///axivvvv/.. orq/ abs/22/2/ / 2044.....4 01373.
We would like to thank Dave Wadden, Kyle Lo, Valentin Hofmann, and Hannaneh Hajishirzi for fruitful conversa- tions. This material based upon work supported by the U.S. National Science Foundation under Grant No. 2313998. Any opinions, findings, and conclusions or recommenda- tions expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Sci- ence Foundation. IM is supported by the NSF CSGrad4US Fellowship. PWK is supported by the Singapore National Research Foundation and the National AI Group in the Sin- gapore Ministry of Digital Development and Information under the AI Visiting Professorship Programme (award num- ber AIVP-20244000 and by the AI2050 program at Schmidt Sciences.
Bisk, Y., Zellers, R., Le bras, R., Gao, J., and Choi, Y. PIQA: Reasoning about physical commonsense in natural lan- guage. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-7439 Apr. 2020. doi: 10.1609/ aaai.v34i05.6233 UtL.:ttt:: index. AAAI httppp//....atttteeeee/eee//////
orgg/
Brandfonbrener, D., Anand, N., Vyas, N., Malach, E., and Kakade, S. Loss-to-loss prediction: Scaling laws for all datasets, 2024. URL https: /ttp/aaxivv.oro/.e/ea aba/ 2411. 12925.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S.,, Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021. URL https: http//axxvvvoro.//.o 2107.03374
Impact Statement
Training large language models is computationally expen- sive, especially when investigating thoroughly over dimen- sions of pretraining data composition, model scale, random initialization, and data order. The pretraining experiments in our DATA DECIDE required approximately 820K H100 GPU hours. We share the benefit of this cost through releasing all of our models, data, and evaluations so that others will not have repeat this expenditure. Moreover, our findings can guide efficient and cost-effective model development through the application of decision making with small-scale experiments. While DATADECIDE does not present direct ethical concerns beyond opportunity cost, we acknowledge ledge that decisions about pretraining data heavily impact down- stream model behavior. We encourage future research to explore potential biases in data selection methods and their implications for models deployed in the real world.
Choshen, L., Zhang, Y., and Andreas, J. A hitchhiker's guide to scaling law estimation, 2024. URL https: waaxxiv.org/bba. 241.
References
Clark, C., Lee, K., Chang. M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Doran, C., and Solorio, T. (eds.), NAACL, pp. 2924-2936. Minncapolis, Minnesota, June 2019. doi: 18.18653/v1/ N19-1300.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q. al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. 2021.
Ben Allal, L., Lozhkov, A., Penedo. G., Wolf, T., and von Werra, L. Smollm-corpus, July 2024. URL htttps://huggirge HuggingFaceTB//
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv, 2018. URL http: /ttp/a/avvvv/..////////ee///.aaa org/ htrg/ab///i aaaaabb///8 abs/.803. 05457.
soodataseess/
Bhagia, A., Liu, J., Wettig, A., Heineman, D., Tafjord, O., Jha, A. H., Soldaini, I.., Smith, N. A., Groeneveld, D.,
Koh, W., Dodge, J., and Hajishirzi, H. Establishing task
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv: 2021.
scaling laws via compute-efficient model ladders, 2024.
URL httpthtp//// /atssv///axvvvv// org/ abs 2412. 04403.
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H.,
O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S.,
10

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Common Crawl. Common crawl. URL https://
Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z.., Chen, Z.., Papakipos, Z., Singh, A. K., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., GGngidii A., Victoria, A., Goldstand, A., Memon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A.. Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, P.-Y. B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B.. Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Wang. C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, S.-W., Wyatt, D., Adkins. D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E.,, Arcaute, E.,, Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Ozgenel, F., Caggioni, F., Guzm' an, F., Kanayet, F. J., Seide, F., Flo- rez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpem, G., Thattai, G., Herman, G., Sizov, G. G., Zhang, G., Lak- shminarayanan, G., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Molybog, I., Tufanov, I., Veliche I.-E., Gat, I., Weissman, J., Geboski, J., Kohli. J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin. J., Yang. J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang. J., Wu, K., KamHou, U., Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakho- tia, K., Huang, K., Chen, L.,, Garg, L., Lavender, A., Silva, L., Bell, L., Zhang, L.,, Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Tsim- poukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Seltzer, M. L.,, Valko, M., Restrepo, M., Patel, M., Vy- atskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Sing- hal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chermoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollár, P., Zvyagina, P.. Ratanchandani, P., Yuvraj, P., Liang, Q. Alao, R., Ro- driguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang. R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Ya- mamoto, S., Ramaswamy, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang, S., Wang. S., Agarwal,
commoncrawl .rr.. Accessed: 2025-05-21.
Du, 7., Zeng. A., Dong, Y., and Tang. J. Understanding emergent abilities of language models from the loss per- spective. In The Thirty-eighth Anmual Conference on Neu- ral Information Processing Systems, 2024. URL https: /oppnneeview..o..// waet/fooruee id 30ee-33DD:00 35DAvigMFo.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., Goyal, A., Hartshorn, A. S., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C.. Touret, C., Wu, C., Wong, C., Ferrer, C. C., Niko- laidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia- Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E. A., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F. Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L.., Nail, G., Mialon, G., Pang. G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, M., Misra, L., Evtimov, I., Copet, J.. Lee, J., Geffert, I.., Vranes, J., Park, J., Mahadeokar, J., Shah, J. van der Linde, J., Billock, J.. Hong. J., Lee, J., Fu, J.. Chi, J., Huang, J., Liu, J., Wang. J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J.- Q. Alwala, K. V. Upasani, K., Plawiak, K., Li, K., neth Heafield, K.-. Stone, K., El-Arini, K., Iyer. K., Malik, K., Chiu, K. Bhalla, K., Rantala- Yeary, L., van der Maaten, L., Chen, L.. Tan, L.,, Jenkins, L., Martin, L.,, Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M. B., Singh, M., Paluri, M., Kar- das, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M. H. M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N. S., Duchenne, O., cCelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng. P., Bhargava, P., Dubal, P.. Krishnan, P., Koura, S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, S., Stojnic, R. Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S. C., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S. Batra, S., Whitman. S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speck- bacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., ney Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y.,

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Cho, S.- B., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., lonescu, V., Poenaru, V. A., Mihailescu, V. T., Ivanov, V., Li. W., Wang. W., Jiang, W. Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang, X., Xia, X., Wu, X., Gao, X., Chen, Y. Hu, Y., Jia, Y., Qi, Y. Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Wang. Y., Hao, Y., Qian, Y., He, Y., Rait, Z., De- Vito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. The llama 3 herd of models. ArXiv, abs/2407.2178.5 2024. URL Jttps://pp.semaaticcc. www.stiisshoooit...... org/ arg/Cosp Carp/oorpussSCC 2711511444
L. A., Welbi, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W. Vinyals, O., and Sifre, L. Training compute-optinal large language models, 2022. URL https: /aaps:///rxxvvi waaaaaiivvvooo..
Kang. F. Sun, Y., Wen, B., Chen, S., Song, D., Mahmood, R., and Jia, R. Autoscale: Auto- matic prediction of compute-optimal data compo- sition for training llms. ArXiv, abs/2407.20177, 2024. URL pistps://apts http:///aapps..aniieeessem www.stessccttteeeeeeeeeeeee axg//oorpuaD::271533855
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els. 2020. URL https: /aaxiv.vvvv.////ai org/ httg//b////tt abs///bs/0000 08361.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mereat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I,, Jitsev, J., Sol- daini. L., Dimakis, A. G., Ilharco, G., Koh, P. W. Song, S., Kollar, T., Carmon, Y., Dave, A., Heckel, R., Muen- nighoff, N., and Schmidt, L. Language models scale reli- ably with over-training and on downstream tasks, 2024. URL https: //t:://rraa /aaaiviiiiv..///.////a.a//aaaavv org. wwwgg//b////// ab/////3///33/../////////// 40.
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, J. D.. Yang, J., Bras, R. I.., Tafjord, O., Wilhelm, C., Soldaini, I.., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, H. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Goyal, S., Maini. P., Lipton, Z. C., Raghunathan, A., and Kolter, J. 7. Scaling laws for data filtering data curation cannot be compute agnostic. CoRR, abs/2404.07177,
Lewkowyzz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I. Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving quantitative rea- soning problems with language models, 2022. URL https:
2024. doi: 10.4855/AARX1.2244400777 URL https: //doi. //dbll.rg//..44555//
48:44/550/// wttp:///aiviiii. 2404. 07177.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I,, Wang, Y., Arora, S., Atkinson, D., Authur, R. Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V. Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M.. Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L.,, Dodge, J., Lo, K., Sol- daini, L., Smith, N. A., and Hajishirzi, H. Olmo: Ac- celerating the science of language models, 2024. URL ht attpps:/aare
14858.
Li, J., Fang, A., Smymis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E.. Keh, S., Arora, K., Garg. S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y., Ghosh, D., Gardner, J., Kilian, M., Zhang, H., Shao, R., Pratt, S., Sanyal, S., Ilharco, G., Daras, G., Marathe, K., Gokaslan, A., Zhang, J., Chandu, K., Nguyen, T., Vasiljevic, I., Kakade, S. Song. S., Sanghavi, S., Faghri, F., Oh, S., Zettlemoyer, L., Lo. K., El-Nouby, A., Pouransari, H., Toshev, A., Wang, S., Groeneveld, D., Soldaini, L., Koh, P. W., Jitsev, J., Kollar, T., Dimakis, A. G., Carmon, Y., Dave, A., Schmidt, L., and Shankar, V. Datacomp- Im: In search of the next generation of training sets for language models, 2024. URL https: / /araii..vvoo////....... abs / /24/2446..20 11794.
org/ abs/ 22022222222 00838.
Gu, Y., Tafjord, O., Kuehl, B., Haddad, D., Dodge, J., and Hajishirzi, H. Olmes: A standard for language model eval- uations, 2024. URL https: //t///aaxaa /aaxiv.vvvv.////i//aaa///aaaavla org/ abs/ 2406. 08446.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Magnusson, I., Bhagia, A., Hofmann, V., Soldaini, L., Jha, A. H., Tafjord, O., Schwenk, D., Walsh, E. P., Elazar, Y., Lo, K., Groeneveld, D.. Beltagy, I., Hajishirzi, H., Smith, N. A., Richardson, K., and Dodge, Paloma:
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, F., de Las Casas, D., Hendricks,
12

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
benchmark for evaluating language model fit, 2024. URL https: /ttp///avvvv//e////e//eee/e..evv org. abs 2212 10523.
the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.
Mihaylov, T., Clark, P.. Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujüi, J. (eds.), EMNLP, pp- 2381-2391, Brussels, Belgium, Detober-Novemlar 2018. doi: 10. 11151//11/DD1-122220
Ruan, Y., Maddison, C. J., and Hashimoto, T. Ob- servational scaling laws and the predictability of lan- gauge model performance. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: htttttp://ppeeeevieeeeee httpp/pppwweee..eeeeeeeee net/ forumm? id=On5WIN7xyD.
Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. WinoGrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):87322 (110))::332870000 Apr. 2020. doi: 10.1609/ aaai. w33303.6699999 URL hRLttts: i index. AAAI AwA./aaticee/eeeeeee....aaaaaaee 6399.
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 50358-50376. Curran Associates, Inc., 2023. URL httpt::pp/ooo...... https::/prrcceedinggg neurips. oc cc/ httppppe__fllleesppprr//////////
org/
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Inui, K., Jiang, J., Ng, V., and Wan, Coa.eennnnee...seaMMiM.... pp. 4463-4473, Hong Kong, China, November 2019. doi: 10. 11653//1//119. 1454.
pdf.
OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F., Clark, C.,, Dasigi, P., Dziri, N., Guerquin, M., Ivison, H., Koh, P. W., Liu, J., Malik, S., Merrill, W., Miranda, L. J. V., Morrison, J., Murray, T., Nam, C., Pyatkin, V., Rangapur, A., Schmitz, M., Skjonsberg, S., Wadden, D., Wilhelm, C., Wilson, M., Zettlemoyer L., Farhadi, A., Smith, N. A., and Hajishirzi, H. olmo 2 furious, 2025. URL https: org/ abs/ / 2501 00656.
Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models mirage?, 2023. URL https: / aaaii.iiiii...i. org/ abs/ /2004/30444 15004.
Schaeffer, R., Schoelkopf, H., Miranda. B.. Mukobi, G.,
Madan, V., Ibrahim, A., Bradley, H., Biderman, S., and Koyejo, S. Why has predicting downstream capabili- ties of frontier AI models with scale remained elusive? In Trustworthy Multi-modal Foundation Models and Al Agents (TiFA), 2024. URL net/fol fort/form?? ? (atprrmmmmdddAHieeeee dd
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson,
Penedo, G., Malartic, Q. Hesslow, D., Cojocaru, R.-A., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for fal- con I1m: Outperforming curated corpora with web data, and web data only. ArXiv, abs/2306.01116, 2023. URL https: httpppp...aaiissiiiaiaeae rrg/CooppuTD::
D., Authur, R., Bogin, B., Chandu, K., Dumas, J.. Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, I.,, Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, 7., Strubell, E., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L.., Smith, N. A., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo. K. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, arXiv preprint, 2024.
Penedo, G., Kydlfček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL attps://arxx org/ /bs/ 2406. 17557.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseQA: A question answering challenge targeting commonsense knowledge. In Burstein, J., Doran, C., and Solorio, T. (eds.), NAACL, pp. (149-4155 Minncapolis, Minnesota, June 2019. doi: 10.18663//. 1/N19-1411111
Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y. Resolving discrepancies in compute-optimal scaling of language models. ArXiv, abs/2406.191 19146, 2024. URL 270764838.
Ye, J., Liu, P., Sun, T., Zhou, Y., Zhan, J., and Qiu, X. Data mixing laws: Optimizing data mixtures by predicting lan- guage modeling performance. ArXiv, 0.x/2403.16955. 2024. URL /ttpttppp///ppp...enetttttttttot org/CorpusTD:2653 263
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring
13

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In Korhonen, A., Traum, D., and Marquez, L. (eds.), ACL. pp. 4791-4800, Florence, Italy, July 2019. doi: 10.18653/v1/1199-477
A. Hyperparameters
Table provides OLMo model ladder configurations for all models in DATADECIDE.
B. Proxy Metric Definitions
Zhou, F., Wang, Z., Liu, Q. Li, J., and Liu, P. Program- ming every example: Lifting pre- pre-training data quality like experts at scale. arXiv preprint M4Xx:2409..7115,, 2024.
Table 3 provides formal definitions for our proxy metrics ($2.5).
C. C.Calling Law Variants
Baseline 3-parameter fit. Our default setup (described in $2.2) follows the two-step fit from (Bhagia al., 2024) and uses Equation 1 to map compute C to task loss L, and Equation 2 to map task loss to metric score. This variant fits three parameters (A, a E) in the first step.
2-parameter fit. This is a restricted version of the base- line where the irreducible loss term E is removed from Equation leaving only parameters:
A C
L(C)
(4)
5-parameter (N, D) fit. Instead of modeling loss as function of compute C, this variant uses both number of tokens N and number of parameters D directly in the loss function:
A
B
L(N,D) =
+
(5)
No
D3
This introduces five parameters: A, B, B, and E.
Single-step prediction. In this variant, the two-stage fitting procedure is replaced with a single step that directly maps compute C' to accuracy:
Aco(C)= = 1 exp ( + Lo))
(6)
This combines the loss and accuracy mapping into func- tion.
5- 5-parameter, single step. We also test single-step variant that directly maps from (N, D) to accuracy using logistic function over the predicted loss. This merges Equations and 2 into:
Acc(N, D) 1 exp (-
+b
(7)
+ E)) DF
B
This formulation retains the same five parameters from the two-step (N, D) loss function. Following Bhagia et al. (2024), we merge the parameters k and Lo from the second- stage sigmoid into the loss-side parameters (A, B, E), yield- ing simplified single-stage fit with 7 total free parameters: (A, a. B, B3 F, b..
14

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Model
Batch
Hidden dim.
LR
Model size
Heads
Layers
Training steps
Tokens trained
name
size
32
64
1.4e-22
3.7M
5,725
0.4B
6M
32
96
1.2c-02
6.0M
9,182
0.6B
8M
32
128
1.1e-02
8.5M
13,039
0.9B
10M
32
144
1.0e-02
9.9M
8
8
15,117
1.0B
14M
32
192
9.2e-03
14.4M
8
8
21,953
1.4B
16M
32
208
8.9c-03
16.0M
8
24,432
1.6B
20M
64
192
8.4e-03
19.1M
16
14,584
1.9B
60M
96
384
5.8e-03
57.1M
29,042
5.7B
90M
160
528
4.9c-03
97.9M
12
29.901
9.8B
150M
192
768
4.2c-03
151.9M
12
12
38.157
15.0B
300M
320
1,024
3.3e-03
320.0M
16
16
45,787
30.0B
530M
448
1,344
2.8e-03
530. IM
16
16
57,786
53.0B
750M
576
1,536
2.5c-03
681.3M
16
16
63,589
75.0B
IB
704
2,048
2.1e-03
1176.8M
69,369
100.0B
Table DATADECIDE OLMo's model ladder (Groeneveld al., 2024; OL et 2025; Bhagia et al., 2024) programmatically create configurations for 14 model sizes with hyperparamcters determined by heuristics in Porian et al. (2024). All models have sequence length 2024 and MLP ratio Each configuration is pretrained over data recipes (Table 1). Each recipe and configuration also trained for 3 random seeds where model sizes < 1B are stopped early at 25% of the compute used to train the 1B model for all but the default seed. Model
number of non-embedding paramelers. Batch size the number sequences per batch.
Metric Name
Equation
CORRECT PROB
N
P(
( Conrect context;)
MARGIN
N
(P(coeree) Coorrett context;) max
Foner ECO) P(c' context;))
C of
((a (context,)
NORM CORRECT PROB
N Exec(s) P(ofoontexts)
TOTAL PROB
N E Eccor P(c| context;)
ACCURACY
1 N E T( (arg max P(c context;) = Coorrcct)
-per_token
log( (ns(P(llonntxx))/eal (ookonste)
+_per ..perchhar
Dog(((ccloontet))/cenns
Table Proxy metrics used alternative inputs prediction methods, (+) is the set of possible continuations for item i and N the number of items in benchmark. Fach each of the first 5 metrics have e per token and per char variants which likelihoods are normalized defined in the bottom rows.
Relative Error
Absolute Error
Scaling Law Variant
3-parameter with helpers and >> 50% checkpoints
5.6
2.6
3- 3-parameter with helper points
6.0
2.8
3-parameter step 2 fit with 550% checkpoints
5,9
2.9
3- --rrammtter
6.5
3.1
2 parameter
6.5
3.2
5 parameter, single step
42,8
17.4
3 parameter, single step 5-parameter
42,9
42.3
230.8
65.4
Table Average prediction error for targets for the different scaling law setups across tasks and recipes on ACCURACY all models but IB. We that other than the single step and 5-parameter variants errors comparable, and these variants also roughly follow the compute-decision decisson frontier in Figure 3.
15

---

DataDecide: How to Predict Best Pretraining Data with Small Experiments
Use of helper points. Following Bhagia al. (2024), optionally include extra point (I. 0.0, Acc 1.0) in the second-stage fit. This "helper" point anchors the upper asymptote of the accuracy prediction.
Filtering early checkpoints. We experiment with exclud- ing the first 50% intermediate checkpoints when fitting the sstoe sigmoid. This reduces noise from high-loss early training points and often improves the fit for extrapo- lation.
Helpers and >> 50% checkpoints. Lastly we experiment with combining the previous two techniques on the baseline 3-parameter fit.
We report prediction errors Table 4 for each setup, As the best scaling laws variants all roughly comparable to the simple 3-parameter set up, use this one as our baseline.
Prediction Error.
16


Reviews:
[{"review_id": "TpLmZqfgHm", "reviewer": "Reviewer_fgMX", "summary": "The paper introduces the DATADOS Suite, an extensive experimental framework designed to guide pretraining data decisions for large language models using small-scale experiments. By systematically exploring 25 data recipes (varying in sources, deduplication, filtering, and mixing) across 5 model scales with a fixed token-to-parameter ratio, and repeating experiments with 3 different random seeds (yielding 375 runs in total), the authors aim to predict which data recipes will yield the best downstream performance when scaled up to a 1B parameter, 100B token regime. It further introduces a set of continuous proxy metrics (e.g., normalized correct probability, margin) that improve decision accuracy over standard discrete metrics, reporting that even small models (150M parameters) trained with less than 2% of the target compute can correctly predict 80% of pairwise winners among data recipes. I believe the paper is well-motivated because the full scaling law for data quality decision can be very computational expensive and the benchmark score doesn't make sense if the model is only trained on small amount of data and doesn't emerge.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful questions and are glad they found our “paper is well motivated and the results are very helpful.”\n\nThe reviewer writes that “the main contribution of this paper is that small-scale pretraining experiments (e.g., using 150M–parameter models) can reliably predict which data recipes will perform best.” We highlight that our paper has two main contributions: 1) We release an extensible framework, easily updated with new prediction methods, benchmarks, and small models. 2) We provide practical recommendations based on empirical study of decision accuracy that we observe to be reliable over wide ranges of settings, such as model sizes (now from 4M to 1B parameters) and benchmarks (including knowledge QA, commonsense, and reasoning, as well as math and code now).\n\n**Expanded results**\n\nWe have extended our suite with more small runs and evaluations. Updated figures can be viewed [here](https://anonymous.4open.science/api/repo/tmp-pdf-4938/file/ICML_responce-2.pdf).\n|  | Initial Submission | Current Revision |\n|---|---|---|\n| Total Models | 375 models | 1,050 models |\n| Model Scales | 5 (from 150M to 1B) | 14 (from 4M to 1B) |\n| Random Seeds | 3 (<1B: up to 10% of target compute; 1B: 100%) | 3 (<1B: up to 25% of target compute; 1B: 100%) |\n| Evaluation Benchmarks | OLMES | OLMES + Code (MBPP, HumanEval) + Math (GSM8K, Minerva) |\n| Scaling Law Methods | 1 variant with 3 windows of model sizes used for compute | 8 variants, with 23 windows of model sizes down to 4 orders of magnitude less than target compute |\n\nWe find that **the conclusions to our research questions hold over these new results**: 1) Compute trades off with decision accuracy and different tasks get better signal for decisions. 2) 8 scaling law methods fail to outperform single-scale ranking; future work can use DataDos to iterate on these. 3) Continuous metrics over raw likelihood are always better at small compute; differences due to which length normalization is used are smaller. 4) Low run-to-run variance and large spread between recipes explain success; we can also use proxy metrics to get beyond noise floor for tasks like code benchmarks.\n\n**Ease of use and tooling**\n\nThe reviewer encourages us to make DataDos easy to use, for instance adding a “new benchmark (e.g. shopping_mmlu).” In our public release of 1K models, 25 corpora, and code for pretraining, evaluation, prediction, and decision accuracy (line 104) we will include step by step documentation for adding new tasks, prediction methods, and small pretraining experiments. We will host all checkpoints as native models on Hugging Face. Running new tasks can be as simple as adding a new argument to the open source evaluation framework that we leverage. Our aim is that researchers with a range of compute can build on our suite: 1) by trying new prediction methods with lightweight manipulations of our results CSV such as smoothing or curve fitting, 2) adding new benchmarks over our released checkpoints for just the cost of inference, 3) or even pretraining additional small models for a fraction of the target model cost (e.g., the 9 additional model sizes we trained since submission cost only ~2% of our total compute).\n\n**Reliable and generalizable claims**\nThe reviewer wrote that our experiments covered “a narrow range of models (150M to 1B)”, so we’ve significantly expanded our results to include 9 additional even smaller model sizes as small as  4M, or ~0.3% of the 1B compute. We continue to find consistent trends of compute to decision accuracy that support our claims for our 4 research questions (line 50). We agree that evaluation noise is a concern at small model scale, so our work is the first in this area to train multiple random seeds, even for the largest model sizes (line 152) We use the mean of these to make our decision targets more reliable. The reviewer asked about how are claims generalize, so we added four new heldout benchmarks and found that our preferred continuous proxy metric also leads to better decision accuracy when target performance is above the noise floor (new Figure #6).\n\n**Focus on exploring data difference**\n\nOur study investigates dimensions of differences not yet studied extensively; we explore 25 different data recipes, an order of magnitude more than previous suites.   We chose to spend our compute budget by overtraining (5x the Chinchilla multiplier) many models (14 different sizes) and evaluating many checkpoints per model to estimate what performance would be like with different token-parameter ratios. Our conclusions are very consistent across intermediate and final checkpoints, suggesting that different ratios might give similar results, but we leave that to future work to confirm.", "claims_and_evidence": "I think the main claim and contribution of this paper is that small‐scale pretraining experiments (e.g., using 150M–parameter models) can reliably predict which data recipes will perform best when scaled up to target levels (1B parameters, 100B tokens). They report an 80% decision accuracy in predicting the winner between data recipe pairs.\n\nThe paper conducted extensive experiments covering 25 recipes and multiple scales, with detailed reporting of decision accuracy and proxy metric performance. However, while the reported 80% decision accuracy is promising, the reliance on a fixed token-to-parameter ratio and a narrow range of model sizes (150M to 1B) may limit the robustness of these conclusions when applied to other settings.", "methods_and_evaluation_criteria": "The DATADOS is proposed to estimate the dataset performance/quality using relatively small model and data while keeping high accuracy. They also provide the proxy metrics for different primary benchmarks. The paper is well motivated and the results are very helpful as a guidance or technique report. However, the methodology is still holistic - one key issue is that if we have new benchmark (e.g. shopping_mmlu) and a new dataset/recipe (e.g. nemotron-cc), we have to re-run all the experiments to get the estimation but the estimation is not well-explained and may be difficult to extrapolate (compared to scaling law). For the proxy metrics, I agree the continuous metrics can be much better indicator for small scale experiments before the model emerges. However, there is a lack of analysis on how these proxy metrics are robust across different data recipes and models - if we use another models/data recipe, will the proxy keep accurate. Otherwise, some theoretical bound/guarantee can also be helpful.", "theoretical_claims": "The authors note that only 4.7% of scaling trends “crossover” at the current scales, but warn that as compute increases, more crossovers may occur.  It would be great if the author can provide more theoretical analysis/insights for the observations. It remains vulnerable to noise in small-scale experiments. A deeper discussion of how sensitive the extrapolations are to such noise and how uncertainty in the scaling law fits is handled would be valuable.", "experimental_designs_or_analyses": "The paper covers 25 data recipes across a fixed range of model sizes and including multiple seeds offers an extensive dataset for analysis. The clear comparison between ranking and scaling law approaches provides actionable insights. However, the experimental design is constrained to a single token-to-parameter ratio and a limited range of scales, which might not capture the full spectrum of scaling behavior (for example, we compare two dataset, one dataset is of high quality but has limited tokens, another is of low quality but has enough tokens, in this case we have to predict the performance in a larger data scale). Moreover, the evaluation is restricted to OLMES multiple choice tasks, raising questions about generalizability to other evaluation paradigms or domains. One key concern is that we believe the results are valuable as a summary but for new tasks/data, people can be unsure about how reliable the analysis results are.", "relation_to_broader_scientific_literature": "The work is well-situated within the literature on scaling laws, data curation, and pretraining data selection, building on foundational studies  as well as recent empirical efforts.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The experiments are extensive and the analysis is detailed. However, besides the empirical study, it would be great if the paper can further elaborate on 1) the reliance/theoretical support of the methods; 2) how well the methods/findings can be transferred to different model families and tasks; 3) if it is difficult to guarantee the former properties, it is great to make the proposed methods a easy to use tool.", "other_comments_or_suggestions": "Please kindly refer to the previous sections.", "questions_for_authors": "Please kindly refer to the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "to3eIeTIH9", "reviewer": "Reviewer_M26A", "summary": "The paper \"Data Differences over Scale (DataDos) Suite: How to Predict Best Pretraining Data with Small Experiments\" presents an empirical study on the predictability of pretraining data decisions at large scales using small-scale experiments. The authors conduct controlled pretraining experiments across 25 corpora, training models up to 1B parameters and 100B tokens, and introduce the DataDos Suite, which provides open-source models, data, and evaluations. The study finds that single-scale experiments, rather than multi-scale extrapolations, are the most efficient for predicting the best pretraining data, achieving 80% accuracy with only 2% of the compute budget of the target scale. The research also highlights the predictability of certain benchmarks, such as MMLU and ARC, using small-scale experiments while others, like CSQA, require more compute. The paper positions DataDos as a resource for researchers to explore data selection and scaling trends.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their clear and helpful feedback and supporting that “using 1B as the target model is sufficient for generalization purpose given the compute budget.”\n\n**Expanded results**\n\nSince our submission we have enriched our analysis and extended our suite with more small runs and examined additional benchmarks. Updated figures can be viewed [here](https://anonymous.4open.science/api/repo/tmp-pdf-4938/file/ICML_responce-2.pdf).\n|  | Initial Submission | Current Revision |\n|---|---|---|\n| Total Models | 375 models | 1,050 models |\n| Model Scales | 5 (from 150M to 1B) | 14 (from 4M to 1B) |\n| Random Seeds | 3 (<1B: up to 10% of target compute; 1B: 100%) | 3 (<1B: up to 25% of target compute; 1B: 100%) |\n| Evaluation Benchmarks | OLMES | OLMES + Code (MBPP, HumanEval) + Math (GSM8K, Minerva) |\n| Scaling Law Methods | 1 variant with 3 windows of model sizes used for compute | 8 variants, with 23 windows of model sizes down to 4 orders of magnitude less than target compute |\n\nWe find that **the conclusions to our research questions hold over these new results**: 1) Compute trades off with decision accuracy and different tasks get better signal for decisions. 2) 8 scaling law methods fail to outperform single-scale ranking; future work can use DataDos to iterate on these. 3) Continuous metrics over raw likelihood are always better at small compute; differences due to which length normalization is used are smaller. 4) Low run-to-run variance and large spread between recipes explain success; we can also use proxy metrics to get beyond noise floor for tasks like code benchmarks.\n\n**Explanations of better decisions**\n\nWe appreciate the reviewer’s encouragement to provide further discussion of the “roots” of differing decision accuracy. In our revised figure (#5) we provide an analysis of how decision accuracy of different tasks and metrics can be explained in part by low run-to-run variance and a wide spread of performance values for different data. Using the correct probability proxy metric sees wider spreads for many tasks, though some instead see reduced noise. The reviewer’s questions about saturation in particular are interesting, as this could either lead to easy decisions (if some recipes saturate at lower performance than others) or to noisy decisions (if all recipes saturate around a similar value by the target scale). However, in our suite we do not observe saturation on any of the benchmarks we examine. Instead we see near trivial performance on Boolq, which leads to it having low decision accuracy as its target rankings are mostly determined by noise. Thus we do not recommend it for use on predictions over the scales we examine. \n\nLikewise we appreciate the reviewer’s encouragement to provide further analysis of scaling laws vs. single ranking. Our smaller results let us consider a wider range of compute budgets for scaling law fits, and we elevate our 8 additional scaling law variants (previous appendix line 770) to the body of our paper (see new figure #3). Our finding that these baseline scaling laws do not outperform single scale ranking holds over the breadth of sizes and variants. The latest results show that the single ranking approach is strong at even smaller compute budgets than we previously found, and we believe future work is required to ascertain the specific ways in which scaling laws fail despite getting low prediction error. \n\n**Aggregate benchmark analysis**\nRegarding the reviewer’s question: when figures do not name a specific task for downstream performance, the quantity is the macro average of OLMES (line 202). We find that our claims hold when aggregating: intermediate checkpoints decide as well as compute equivalent final models, single scale experiments outperform scaling laws, continuous metrics outperform discrete ones at small scales. As we examine in new (and former) figure (#2) slope and range of the positive relationship between compute and decision accuracy depends a lot on which benchmark is used, but the other claims we make hold.", "claims_and_evidence": "Main claims:\n\n- C1: Single scale experiments are more efficient for performance prediction than multiple scales. Figure 2 clearly shows that single ranking provides better decision accuracy than multiple scale extrapolation at various compute budget and tasks. I consider the evidence for the claim is sufficient. \n\n- C2: No single metric gives better signal consistently. I consider the evidence (Sec 3.2) is sufficient.", "methods_and_evaluation_criteria": "**Benchmarks**\n\nOLMES suite of 10 multi-choice QA benchmarks is used. It contains common and standard benchmarks for LLM evaluation. \n\n**Metrics**\n\nThe study uses both performance prediction errors and decision accuracy to reflect the practical consequence of pretraining data prediction. \n\n**Methods**\nThe study effectively isolates the impact of scaling decisions by holding model architecture constant while varying training data and compute. It presents large scale experiments consists of a suite of 375 models using 25 data recipes, 5 model scales, and 3 random seeds for initialization and data order. I also find using 1B as the target model is sufficient for generalization purpose given the compute budget.", "theoretical_claims": "None", "experimental_designs_or_analyses": "1. The experimental design appears to be robust to me, covering 25 data recipes, 5 model scales, and 3 random seed variations to control for initialization effects (target model only).\n\n2. Models are trained using fixed architectures while varying datasets and compute budgets to isolate scaling relation and ensure a fair comparison.\n\n3. Insightful findings and analysis includes\n- Mixed results for proxy metrics choice; \n- limitation of single ranking method in the case of predicting performance orders of magnitude. \n- Mixed results for benchmark signals. \n\nI think the paper will benefits from more discussion on the roots of mixed results. For examples, what drives the discrepancy of benchmarks as a signal for large model performance? Why some benchmarks are saturated while others are not? Does it mean single rank method predict well simply because of the saturation of the benchmark (too easy?)?", "relation_to_broader_scientific_literature": "The study builds on prior work in language model scaling laws and data selection, referencing key papers such as Kaplan et al. (2020) and Hoffmann et al. (2022). It differentiates itself by focusing specifically on data selection rather than general model scaling, bridging a gap in the literature.", "essential_references_not_discussed": "Unknown.", "other_strengths_and_weaknesses": "**Strengths**: \n- Open-source contribution of models, datasets, and evaluations.\n\n**Weakness**\n- Limited depth analysis on the cause of scaling law failure and single rank success.\n- nitpick: lack of sensitivity analysis for hyperparameter choices.", "other_comments_or_suggestions": "None", "questions_for_authors": "- When performance is not reported per task, do you report the average of OLMES? \n- Does the conclusion hold when using the benchmark as a whole, by group? This tests the sensitivity of the conclusion to benchmark choices.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xsRrLQfL6Q", "reviewer": "Reviewer_B1fY", "summary": "This paper presents DataDos, a suite of experiments to examine the comparison of 25 variously processed corpora across scales by pretraining them with up to 1B models and 100B tokens. It finds that 150M models trained with < 2% compute of 1B targets correctly predict 80% of comparisons; spending compute towards smaller, more completely trained experiments for ranking is cost-effective for overall decision accuracy, but fitting scaling law predictions will provide additional information for extrapolating scales or extreme quality thresholds. Detailed per-task/metric analyses reveal different levels of predicting difficulty across tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed clarifying questions and are happy they found our work has “comprehensive experiments and detailed analyses.”\n\n**Expanded results**\n\nWe appreciate the reviewer’s encouragement to polish our paper. Since submission we have fully revised our paper and extended our suite with more small runs and examined additional benchmarks. Updated figures, which we hope are more readable, can be viewed [here](https://anonymous.4open.science/api/repo/tmp-pdf-4938/file/ICML_responce-2.pdf).\n|  | Initial Submission | Current Revision |\n|---|---|---|\n| Total Models | 375 models | 1,050 models |\n| Model Scales | 5 (from 150M to 1B) | 14 (from 4M to 1B) |\n| Random Seeds | 3 (<1B: up to 10% of target compute; 1B: 100%) | 3 (<1B: up to 25% of target compute; 1B: 100%) |\n| Evaluation Benchmarks | OLMES | OLMES + Code (MBPP, HumanEval) + Math (GSM8K, Minerva) |\n| Scaling Law Methods | 1 variant with 3 windows of model sizes used for compute | 8 variants, with 23 windows of model sizes down to 4 orders of magnitude less than target compute |\n\nWe find that **the conclusions to our research questions hold over these new results**: 1) Compute trades off with decision accuracy and different tasks get better signal for decisions. 2) 8 scaling law methods fail to outperform single-scale ranking; future work can use DataDos to iterate on these. 3) Continuous metrics over raw likelihood are always better at small compute; differences due to which length normalization is used are smaller. 4) Low run-to-run variance and large spread between recipes explain success; we can also use proxy metrics to get beyond noise floor for tasks like code benchmarks.\n\n**Clarity and Organization**\n\n\nFollowing the reviewer’s suggestion, we will move the discussion of scaling laws ahead of proxy metrics in section 3 (results) to better align with the order of topics in section 2 (methods). Also regarding the reviewer’s question 1: In the third figure in our submission we identified best metrics based on which achieves the highest decision accuracy over all of the small compute sizes used for prediction. Indeed, the *target* budget remains the same (1B 5xC) but we consider which metric gets best decision accuracy for the most small *experimental* budgets. Regarding the “best outcomes,” this quantity would be better defined as measuring the average decision accuracy over tasks and scales when using the best metric for each task. In our revised figure (#4), we instead use a new, more direct figure to visualize the relationship of compute to decision accuracy for each type of metric with a specific length normalization. Regarding question 2: 150 is the number of combinations visualized in the former Figure 5, since the number of rows is halved to save space. Thanks to the reviewer’s feedback we’ll be sure to make this more clear.", "claims_and_evidence": "Claims about how to predict the best pretraining data across scales are supported by the results of comprehensive experiments.", "methods_and_evaluation_criteria": "The methods and evaluation metrics for pretraining models on different corpora are reasonable and comprehensive.", "theoretical_claims": "This paper does not make theoretical claims and focuses mostly on the empirical part.", "experimental_designs_or_analyses": "The experiment and analysis design is overall sound.", "relation_to_broader_scientific_literature": "Scaling law has shown effective promises in guiding the optimal usage of computation. This paper examines the data comparison law across scales, which can further provide guidance and insights into the future development of pretraining data curation, especially extrapolating beyond small-scale experiments.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The strengths of this paper lie in its comprehensive experiments and detailed analyses about predicting pretraining data differences over scale.\n\nThe main weakness, from my perspective, is the clarity and the organization of paper writing.", "other_comments_or_suggestions": "I would suggest the authors run experiments on a target scale of 7B because this is likely to be the smallest mainstream LLM scale, but the feasibility definitely depends on the available computational resources the authors have because pretraining is always costly.\n\nThe writing also can be polished to make this work more impactful. For example, \n\n- Better arrangement of the order of subsections in Section 3, preferably aligned with Section 2.\n\n- Polishing all Figures to make them more readable.", "questions_for_authors": "Mainly about clarity issues.\n\n1. In the discussion of Figure 3, \"To identify a generally best metric per task we find the metric which achieves the highest decision accuracy in the most compute budgets.\" What do you mean by that? I suppose the decision accuracy is only measured in the target budget. Also, \"average decision accuracy for a metric over its best outcomes\", what do you mean by \"best outcomes\"?\n\n2. You mention \"150 extrapolated comparisons\" in 3.3. Why 150? I cannot interpret this number from Figure 5.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a well-executed empirical study showing that small-scale pretraining experiments can reliably predict the best data recipes for large-scale LLM training. All reviewers acknowledge the value of the extensive experiments and open-source contributions. While one reviewer raises concerns about generalization, the authors addressed these with expanded experiments and clearer analyses. I recommend acceptance of the paper.
Decision: Accept (poster)