CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
Authors: David Brandfonbrener
Hanlin Zhang
Andreas Kirsch
Jonathan Richard Schwarz
Sham M. Kakade
Venue: NeurIPS 2024 poster (2024)
Keywords: data selection
language models
LLMs
Primary area: active_learning

Abstract: Selecting high-quality data for pre-training is crucial in shaping the downstream task performance of language models. A major challenge lies in identifying this optimal subset, a problem generally considered intractable, thus necessitating scalable and effective heuristics. In this work, we propose a data selection method, CoLoR-Filter (Conditional Loss Reduction Filtering), which leverages an empirical Bayes-inspired approach to derive a simple and computationally efficient selection criterion based on the relative loss values of two auxiliary models.

In addition to the modeling rationale, we evaluate CoLoR-Filter empirically on two language modeling tasks: (1) selecting data from C4 for domain adaptation to evaluation on Books and (2) selecting data from C4 for a suite of downstream multiple-choice question answering tasks. We demonstrate favorable scaling both as we subselect more aggressively and using small auxiliary models to select data for large target models. As one headline result, CoLoR-Filter data selected using a pair of 150m parameter auxiliary models can train a 1.2b parameter target model to match a 1.2b parameter model trained on 25b randomly selected tokens with 25x less data for Books and 11x less data for the downstream tasks. 

Code: https://github.com/davidbrandfonbrener/color-filter-olmo

Filtered data: https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4

CoLoR-Filte:: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
David Brandfonbrener Kempner Institute at Harvard University
Hanlin Zhang
Andreas Kirsch University of Oxford
Harvard University
Jonathan Richard Schwarz
Sham Kakade Kempner Institute at Harvard University
Harvard University
Abstract
Selecting high-quality data for pre-training is crucial in shaping the downstream task performance of language models. A major challenge lies in identifying this optimal subset, a problem generally considered intractable, thus necessitating scalable and effective heuristics. In this work, we propose data selection method, CoLLoRFilter (Conditional Loss Reduction Filtering), which leverages an empirical Bayes-inspired approach derive simple and computationally efficient selection criterion based on the relative loss values of two auxiliary models.
In addition to the modeling rationale, we evaluate CoLoR-Fiin -Fllee empirically on two language modeling tasks: (1) selecting data from C4 for domain adaptation to evaluation on Books and (2) selecting data from C4 for a suite of downstream multiple-choice question answering tasks. We demonstrate favorable scaling both subselect more aggressively and using small auxiliary models to select data for large target models. As one headline result, CoLoR-Filter data selected using a pair of 150m parameter auxiliary models can train 1.2b parameter target model to match 1.2b parameter model trained on 25b randomly selected tokens with 25x less data for Books and 11x less data for the downstream tasks.
Code: httpps:/gittbb...o wwtp////w...rnnndedeeeeooooe.eee www.aarreeeeeeeeeeeeeeeeeeeeeeee filber olno
Filtered data: hespa://huggingfaee ca /at.aaaaataaaaaaaaaaaaaaaaaaaaaa www.://brraaafoorreeereeooo filterdd-cd
Introduction
The content of the data that language model is trained
have profound effects on its per- formance and the efficiency of the training process [Rae et al. [021] Longpre et al. 2023 Penedo
al. 2023 Soboleva et al. 2023 Li al. 2024). But it remains open research question how to decide which data to include in the training set. In this paper, we analyze family of loss- based approaches for targeted selection of pre-training data, propose simple approach that outperforms existing methods, and provide some preliminary evidence of favorable scaling properties.
To formulate the data selection problem, we first need to specify objective that quantifies whether the selected data is good. Defining this objective requires evaluating a pre-trained language model, which is Chang et al.) 2024]. For this paper, we will take the goal to be to maximize performance on a set of downstream tasks. Since the preferred metrics on given set of tasks not necessarily the same nor amenable to direct optimization, we consider the likelihood of sequences sampled from the downstream tasks proxy objective. With this objective, we have straightforward goal: given a very large corpus of sequences and a small amount of high-quality data from a set of downstream tasks, want select subset from the corpus that training on the selected data
of active research [aoo et al.) 2023 Magnusson al., 2023 Engstrom et al.) 2024
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Books Cross Entropy (4)
Average Downstream Accuracy
11x less data
50
25x less data
CoLoR Filter
64)
Random
3.5
40
0
3
15 Tokens (billions)
9
12
18
21
24
0
3
12 Tokens (billions)
9
15
18
21
24
Figure 1: Learning curves for 1.2 billion parameter language models trained on data selected by CoLoR-Filter using smaller 150 million parameter auxiliary models for two different target distributions. (Left) We target and evaluate loss on Books, lower is better. (Right) We target and evaluate accuracy on a suite of 8 downstream tasks from [Groeneveld al., 2024). higher is better. In both cases, test data is held out from the data used by CoLoR-Filter to guide selection. T is the subset size multiplier denoting the number of examples considered for each selected data point. The CoLoR-Filee Filter line terminates when we run out of data in C4 (s=17bb possible tokens).
maximizes likelihood on the downstream tasks. Then we can also test performance on the tasks under their preferred metrics.
From this objective, we derive an algorithm dubbed CoLoR-Filtes (Conditional Loss Reduction Filtering). In Section derive this method by applying Bayes' rule and approximate empirical Bayes to the downstream likelihood objective. The resulting method simple and intuitive: each sequence is scored by the difference in likelihood between "prior" model and "conditional" model that results from fine-tuning the prior model on the downstream data. Sequences that are more likely under the fine-tuned model are good. We also compare this algorithm prior work (e.g..
Mindermann et al. 2022 and discuss computational costs.
To evaluate method, consider two tasks. First, in Section we consider semi-synthetic task where the downstream task is language modeling on Books. Given access to C4 [Raffel et al.) 2020 potential pre- pre-training data and a small (25 million tokens) sample of data from Books, use CoLoR-Filter and a variety of baselines to select 3 billion tokens. We find that data selected by CoLoR-Filter substantially outperform models trained 8x much randomly chosen data. Second, in Section (6, we consider a suite of 8 downstream multiple-choice tasks from Grrenneell of the tasks, but evaluate accuracy
al. [2024). As downstream data take the training on the held-out test sets. We again find that selecting with CoLoR- outperforms training on 8x much randomly selected data. Moreover, in both tasks, performance scales smoothly with the hyperparameter that governs how aggressively we select the data, suggesting that further scaling would yield further improvements.
In addition to finding that CoLoR-Filter can select good subsets of data, we also consider the computational cost of the selection procedure itself. CoLoR-Filter only requires running inference of the two auxiliary models to select data. This is computationally beneficial compared to online methods like RHOLoss (Mindermann al. 2022) since inference is cheaper than training and is entirely parallelizable. To maximize the computational benefits we also show that data selected with small (150 million parameter) model can be transferred to larger (1.2 billion parameter) model. Results are shown in Figure [
showing substantial efficiency improvements.
2 Setting and Derivations
Assume that we given large pre-training dataset Dirain small downstream dataset Dawn from the downstream task(s) interest, and "prior" dataset Dpnia we can use as prior knowledge (in practice often just sample from Duain). We will assume for all practical purposes that D

---

is infinite and training proceeds in the "online" or "single pass" setting where we do not repeat data points. Our goal to choose a subset S Diain of fixed size [S] n that minimizes the downstream loss (maximizes the downstream likelihood).
This section introduces our Co-teR-Fiter algorithm, inspired by and building upon the RHOLoss approach from prior work [Mindermann et al. 2022 Evans et al. 2023]. We also discuss related
algorithms applicable to this setting such as DSIR [Xie al. 2023 and DSDM [Engstrom al.
2024].
Additional related work is discussed further in Section
2.1 Bayesian Data Selection
Our objective can be formulated Bayesian optimization problem, where the goal is to select set so as to maximize the posterior probability of Doown i.c.
min log Pr dowwn 15),
SC SC =
where Pr(Ddown (S) is the posterior probability. Applying Bayes rule we get:
min Sc ScDui.|S]
log (PS))Damm) log Pr(S) log Pr Pr(Ddow))
Note that the last term does not depend on S, so it can be ignored when optimizing over S. Introducing a prior over model parameters 0, we get:
min SC S|nn
log
Pr(S]0)Pr(( Doowa) + log Pr((10)
"cunditional"
"marginal"
We will refer to the terms as the conditional and marginal terms, respectively Note that the conditional and marginal terms together make up the negative pointwise mutual information between the selected and downstream data, which has deep connections prior work on active learning and
active sampling [Lindley] 1956 Moore and Lewis 2010 Houlsby et al. 2011 Bickford Smith et al. 2023
Kirsch 2023 Rainforth et al.] 2024
2.2
CoLoR-Filter
Given that we have access to prior knowledge from the dataset D priorr we can replace the uninformed prior over 8 with empirical Bayes prior that conditions D prior obtain:
log / Pr(S10) Doowa Pprior) + log Pr())
min SC Sl=n
(4)
As this integration still intractable, we now make our main simplifying assumption which is to replace this integration over parameters by point estimate:
min SCDi S|
log Pr(S 0 prior--down) + log Pr Pr(S)00rioc)
(5)
where ppnor is model trained on Dprior and 6 down is model trained on both Dprior and Doown (in practice, we use a model that is pre-trained on Dprior fine-tuned on Daown).
Moreover, this approximation leads to computational benefits by avoiding the full combinatorial optimization of subset selection. In particular, once we condition on a single model 0, and assuming the distribution over points S is independent, i.c. Pr(S]0) IIzes Pr(x((), we have:
n
min (R Fn Dean
log 1=1
Pr(x, doww log II
Pr(xa (prior)
(6)
which simplifies to:
min En Dusis log Pr
Pr (Iz dowor) (-log Pr(zz
Prior work Mindermann et al. 2022 Evans et al. terms as the "reference" and "learner" or "actor", respectively. We opt for the names conditional and marginal for clarity in connections the Bayesian viewpoint.
2023 has referred to the models that estimate these two
3

---

This gives our CoLoR-Filte --iteer criteria that we use to select data. This optimization selects the points with the largest conditional loss reduction (CoLoR), i.c. the points where the negative log- likelihood loss of the conditional model doown is lower than the marginal model price Intuitivelyy, this selects data points that are more likely under the conditional model than the marginal model.
A note on data diversity. While the factorization that results from our point estimate of the parameters is computationally convemient, it makes an important miportantimmplifiiin the CoLoR-Filte --Fter objective no longer encourages the selection of diverse dataset, as scores are applied independently to each point. In practice, this is remedied by a few considerations: (1) we can run CoL ooR-Fitter on corpus that has already been deduplicated to prevent degenerate duplications, (2) for large n, we must select many different data points, and (3) each datapoint itself a sequence that may contain diverse signal across tokens. We should also note this is not unique property of CoLoR- and also happens in other methods that do offline scoring like DSDM and DSIR. We defer detailed discussion of the nuances of this issue to Appendix C
assumption. In particular,
2.3 Related Algorithms
Connection to importance sampling. Since the CoLoR-Filter objective is written as a difference of logs, it can also be written as log of the ratio between probabilities under pprior-down and prior- If data were actually sampled from prrorr then this ratio would be the importance weight needed to reweight samples so that they are from the model defined by A drion Note that DSIR (Xie et al. 2023] directly attempts to perform importance sampling from Drain to Daown instead of optimizing performance on the downstream data. Thus, DSIR ends up with somewhat related algorithm except in DSIR: (I) there is no language model, just features of a full data point (hashed n-grams), and (2) the algorithm samples rather than optimizes.
Connections DSDM. Another closely related approach DSDM Engstrom et al. 2024] which uses TRAK Datamodel estimator [Ilyas et al., 2022 Park et al. 2023 to score datapoints and then selects the top-n points. The motivation and setting of DSDM are similar to CoLoR- Filter, but DSDM relies on TRAK which constructs a linear approximation of the influence that data points have on each other. Instead, CoLoR-Filee operates directly function space by comparing the loss between models directly rather than relying on linear approximations or Datamodels [Ilyas et al. 2022].
Connections to RHO-down. CoLoR-Filter is inspired by and builds on the RHOLoss approach introduced in prior work [Mindermann et al.] 2022] with subtle but significant differences in the setting: the original RHO paper focuses on cases where the hold-out data sampled from the same distribution as Drain over multiple epochs of training. In contrast, we focus on selecting data to target downstream distributions that are different from Dirain and where we only take a single pass over the data. Here, we derive straightforward adaptation of RHOLoss to our setting, which we call RHO-down.
We now derive RHO-down in our setting, aiming to illustrate the connections between RHO-down and FoLtR-Filte. First, RHO-down approximates the full subset selection problem from Equation (3) by a greedy (sequential) approximation where samples are added to S one (batch) at time. Using batch size 1, the ith-sample would be ideally added according to the following criterion:
min Drais
-
log Pr(x. Pr(O Dawa. F<()
+ log Pr(x:10) Pr(f)
(8)
where ranges from 1 to sequentially. RHO-down then uses point estimate of the parameters (as we do in CoLoR-Filter):
min - log Pr(x, (down+ + log Pr(a,10zec
(9)
2 LEDD
Finally, the RHO-down authors found that updating the conditional term to depend on Jeÿ was unstable, so they instead approximate this by fixed model 0sown'
min - log ((oon)) + log (r(21(0ue))
(10)
R D
Note that while both Co- and RHO-down approximate the posterior over parameters with point estimate, RHO-down makes a few additional approximations. This is largely result of

---

RHO-down attempting to increase data diversity by using sequential approach to selection that conditions on the previously selected data z<i- This is an understandable goal. but it introduces more approximations, can cause instability by creating non-stationary data distribution, and is computationally expensive since the data selection is no longer parallelizable. A continued discussion of the pros and cons of online selection is in Appendix
RHO-down + prior. We also consider a version of the algorithm that we call "RHO-down + prior" that replaces Dowwn: 8down in the RHO-down algorithm with prior U Dowwn +oon to incorporate the prior information. This corresponds to conditioning on both Dpeior and Doown instead of only Daown- Intuitively, this method can better leverage stronger features learned on the larger D Dprior to integrate the information from the small D Down-
3 Further Related Work
We now discuss some related work, more broadly, with regards to active learning and data curation.
Active & Curriculum learning, Our formulation of data selection has connections to classic and deep active learning [Houlsby et al., 201 Bickford Smith et al. [0233 Kirsch 2023]. which are deeply rooted in optimal Bayesian experimental design [Lindley] 1956 Rainorrh et al. 2024] whose
goal is select a set of experiments to optimize certain information criteria [Pukelsheim] 2006]
such as maximally reducing the uncertainty about model parameters, Various acquisition functions are proposed in deep learning regimes [Sener and Savarese 2018 Ash al., 2019 202T] and most of them focus on label-efficient image classification.. Another line of recent techniques share deep methodological connections but emphasize the sub-selection of available data during training (rather than the collection of additional examples typically considered in active learning) and could thus be classified as curriculum learning [e.g. Graves et al., 2017]. Among them, RHOLoss [Mindermann] al., 2022] seeks to select data based on the hold-out reference dataset from the same distribution as
the training data. It has been later implemented continual pre-training [Lin et al. 2024 and vision
domains [Evans et al. 2023 Tack et al. 2024).
Data curation practices in pre-training. Though large-scale public web-crawled data are common data sources for pre-training models, low-quality, toxic, and uninformative content that can prevent successful pre-training is prevalent [Wenzek al., 2020 [Flazar et al.) 2023 [Sorscher al. 2022]
Allen-Zhu and Li 2024). Therefore, practitioners design sophisticated data pre-processing pipe lines
such as filtering [Brown et al., 2020]. deduplication [Lee et al. 2022]. ) and mixing [Touvron et al. b to improve the data quality. Due to the immense scale, state-of-the-arr pre-training datasets
usually depend on simple heuristic filters [Raffel et al. 2020 Rae et al. 2021 Computer 2023)
(e.g.. URL, length, n-gram perplexity, fastest classifiers) that can be parallelized across CPU nodes. Besides the above rule-based filtering, model-based filtering concerns using machine learning models to score and filter data, which has been proven to be effective in vision and vision-text domains SShuhmaan et al. 2022, Abbas et al.) [2023] Fang et al., [2023]. J. Such approaches usually leverage a given trustworthy data source like Wikipedia or Books the reference and contrast the data with it. Due to computational cost, models are often designed to be small such as n-gram [Xie et al., 2202).
single-layer neural networks (Joulin et al., 2017 Brown al. 2020). k-means clustering [Tirumala ct al. 2024). There is also a growing line of work illustrating that data quality is important in shaping model training from a variety of perspectives, such increasing data scale (Hoffmann et al. 2022
Meta 2024 and using synthetic data [Gunasekar et al. 2023
4 Algorithms
4.1
From Derivations to Practical Algorithms
In our experiments, we will consider four algorithms based on the above derivations. In this section we go through each of these turn.
CoLoR-Filter. Our proposed algorithm is presented formally in Algorithm I Compared to the derivation, the main difference is the introduction of т, a hyperparameter that acts as a compute- performance trade-off controlling how expensive and aggressive the data selection is. Rather than
5

---

Algorithm CoLoR-FFiter
Require: Prior data Dpriors downstream data Doown training data Dunin budget n, subset size
multiplier
1: Pre-train (mae on Priio
2: fine-tune to get (rond on Down initialized from Omang
3: Select random subset D, of size tn from D
4: Select data:
S = bottom-n.ee Ex log Pr(x (gond) + log Pr(x)6uare
5: return Selected dataset S to train on.
selecting data from all of Diaain we take a random subset D, of size Tn. Thus, larger subselect more aggressively, but the cost of more computation. A full discussion of this cost is in Section 4.2
Conditional only. As an ablation of CoLoR- Filter, we follow prior work [Evans et al. 2023 and include baseline that only uses the conditional model to select data. Essentially, this is CoLoR Filter Litterrlter we always assume that log Pr( (=(mm)))) = 0 in Line of Algorithm !
Algorithm 2 RHO-down
Require: Downstream data Daown, train data Dinin budget n., subset size multiplier т. batch size b
1: Train @cond on Deown
2: Initialize a random g and S
3: for t E [l..... n1/bl do
4: Randomly select batch B, C Duaan of size 7b
5: Select data:
B, = bottom-beeete log (r(x)(sood)) + log Pr(x (
6: SU
7: Update 0.marg
0 by training on
8: end for
9: return Selected dataset S to train on.
RHO-down. We present practical variant of RHO-down in Algorithm 2 based on the derivation presented in Section [ The main changes to make practical algorithm are (1) the introduction of as in CoLoR- and (2) performing the algorithm batch-wise instead of using single data points.
We can also incorporate the prior data Dpricr into Algorithm by simply replacing Line 1 where goond is trained on Doown with procedure where we first pre-train geond on D price and then fine-tune on Dawwn-
RHO-down Prior.
4.2 Computational Cost
To evaluate the computational cost of the various algorithms, we use units of "model forwards" per token where we assume that a backward pass is twice as expensive forward pass [Fleuret 2023] Note that our 150m models take about 5e8 FLOPs per model forward of a single token [oofmmnn
al. 2022 Casson 2023] The cost of running the selection algorithms depends on m, n. 7 and L defined as follows: m is the size of the prior data Dprior- is the size of the selected dataset S,
is the hyperparameter controlling how aggressively we subselect data. Note that we assume that [aownl is so small that the cost of training a model on D is negligible towards the total cost (and all the methods we consider just fine-tune model once on Dewwn). We will also be careful to note when computation can be done in parallel before training versus computation that must happen serially during training run. Offline algorithms like CoLoR-Filter can take advantage of parallelism to improve efficiency. In this section, go through each method in turn and aggregate the computational costs in table
Scale transfer. We also include another parameter I. to cover the case where we select data using small models and use to train larger model [Evans al., 2023]. Specifically, T. the ratio of cost of one model forward of the large target model compared to the small auxiliary models used for data

---

Table 1: Compute cost of the various algorithms measured in "model forwards". The total cost of selection and training on the selected data is the sum of all costs across a row. The variables are m = [ peoors n = |S|, T is a hyperparameter that controls how aggressively we subselect, and is multiplier of the cost of model forwards between the selection model(s) and the target model (approximately the ratio of parameter counts between the models).
Method
Prior cost
Serial cost
Parallel cost
Training cost
CoLoR--Fiter
3m
2tn
3nL.
Conditional Only
3m
0
7h TT
3nL
RHO-down
2n
7h:
3nL
RHO-down+ Prior
3m
Tyl + 2n
TR
3nL
Random
0
0
3nL
selection. For example, in our experiments, when use 150 million parameter models to select data and then train 1.2 billion parameter model on the resulting data, then L R 5.53 Training thus costs 3nL across all methods since run forward and backward for the large model on all sequences.
The cost of selection is 2rn forward passes. But, this selection process is entirely parallelizable. Training the prior model costs 3m forwards since prier on the selected data costs 3nL forward passes. So the total cost is 3m + 2rn + 3nL, but the 2rn scoring computation can be done in parallel.
CoLoR-Filter
And training model
Conditional Only. The conditional-only method is almost the same as CoLoR-Filter, except we only need rn forward passes for selection since we only run one model over the data. The cost is thus 3m + 7n + 3nL, with 7n being parallelizable.
The cost of selection still 2rn forward passes. Then we need an additional 2n to backward the output model (since the forward is already handled during scoring). Note that we need to evaluate the marginal model online, so it is not parallelizable, but the conditional model is fixed and can be computed offline. So, the cost is 2rn + 2n + InL, and the rn conditional model computation can be done in parallel.
RHO-down.
RHO-down Prior. For the version with an added prior, we just add 3m cost for training the prior. Thus, the cost is 2rn + 2n + 3nL with rn parallelizable.
Overall, the methods all have comparable costs, with Conditional Only being the cheapest and RHO-down Prior the most expensive. The main difference is that CoLoR-Filter and Conditional Only are easily parallelized while RHO-down and RHO-down Prior are not. It should also be noted that when doing experimentation, offline methods like CoLoR-Filte also benefit from being able to re-use likelihoods multiple times, while RHO-bases methods need to recompute the serial cost any time that some hyperparameter of the algorithm.
Domain Transfer: a Simple Testbed
5.1 Setup
Training. We train language models with 150 million non-embedding parameters using the OLMo codebase [Groeneveld al., 2024) and following hyper-parameter choices from (Wortsman al.) 2024]. Unless otherwise noted, we use 150mm models as the auxiliary models (foond (marg) as well as the target model 0. Full hyperparameters are described in detail in Appendix H
We take Dawwn to be a small dataset of 25 million tokens sampled from the Project Gutenberg Books data subset of Dolma (Soldaini et al. 2024]. D prior to be dataset of 3.1 billion tokens from C4 RRffel et al. 2020). and Dusin to be all of C4. We select a dataset S of 3.1 billion tokens (which is
2 EEven though there are 8x as many parameters in the large model, the FLOP multiplier is less since the attention computations take the same number of FLOPs regardless of parameters.
7

---

approximately the "chinchilla optimal" amount for models of this size). To get 8 driwr or A down. we fine-tune train for one epoch on down-
Evaluation. To evaluate the efficacy of our data selection, we report cross-entropy loss of next token prediction on a held-out dataset D Daown from the same distribution Dawwn (Books).
Baselines. The simplest baseline we consider is Random sampling, which has been shown to be strong baseline for C4 pre-training Engstrom et al. [2024]. We consider all four algorithms described in Section 4 CoLoR-Filter, Conditional Only. RHO-down, and RHO-down prior. And one
extra baseline, we also include DSIR [Xie et al. [023] which estimates n- rrgram importance weights between Dirain and Down. and similarly has a parameter like r that controls how aggressively to subselect.
Note that while it is in similar setting ours, do not include DSDM [Engstrom et al. baseline since there is no publicly available code and based on the appendix of that paper, it much more computationally expensive than the methods we consider.
2024
5.2 Results
ColoRRFilte
RHO-down prior
Random 1x Random
Conditional only
DSIR
We first run the domain transfer experiments on 150m models, sweeping across T that controls the selected subset size. In Figure 2 we plot how the final performance scales with across methods. We see that CoL-oR-Filter has the best scaling performance with increased т, with no sign of saturation for = 16. We hypothesize that by using strong models to select the data, CoLoR-Filter able to more effectively scale to larger than the other methods. In Figure 7 in Appendix A we plot the learning curves (eval- uated on the held-out validation set) for the four methods introduced in Section 4 There, we see
Final Books Crass Entropy (J)
2
2F
2
Figure Scaling of final performance with r when targeting Books with 150m parameter models. especially clean scaling for CoLoR- Filter across the entire learning curve, substantially outperforming random selection with much less data, similar to Figure]
Books Cross Entropy
Scale generalization. Finally, we also con- duct an experiment in scale generalization (par- tially shown in Figure ! using the data selected by our 150m auxiliary models to train a 1.2b tar- get model. In Figure 3 we show learning curves for a sweep over т. We still see consistent gains we scale for fixed number of training to- kens. Interestingly, if we fix the total number of tokens we are selecting from (i.e. where the lines end when we run out of C4), then the final performance with
FileeFFilter (7 64)
4.7
ColoR-Filter (r 32)
CoLoR-Filter (r 16)
FiltRrFilter (r 7)
Random
3.8
3.5
15
18
24
32 is better than all other values of т. This shows how a strict subset of tokens can outperform a superset (e.g. T 16). We should also point out here the computational savings when using CoLoR-Filter, As an exam- ple, consider
Tokens (billions)
Figure 3: Scaling CoLoR- CoLoR-Filter with when train- ing 1.2b models with data selected by 150m mod- els. Curves end when exhaust the data in C4.
16 where we match the performance of 25 billion randomly selected tokens with about 1.5 billion filtered tokens. Considering the computational costs discussed above with L = 5.5 and measuring n in billions of tokens, the total cost for training the CoLoR-Fille model 3m + 27n + 3nL = 3 * 3.1 + * * 1.5 + * 1.5 x 5.5 = 82 while the cost for training on 25 billion random tokens is 3.VL = 3 25 5.5 = 412.5, illustrating more than 5x total compute savings to achieve the same performance on Books. A full plot visualizing the cost in FLOPSs for all is in Appendix
8

---

Improvement over Random 1x (r - 16) (1)
CoLoR
Conditional Only
-0.1
1.1
0.6
-2.2
0.4
-2.3
2.5
-2.1
-0.3
22
RHO-down
2.4
2.9
2.5
2.4
0.6
-10.2
-0.
prior
-2
DSIR
4.4
4.3
2.0
2.2
1.0
1.5
1.1
-0.7
2.0
Random
5.1
3.3
1.2
-0.2
2.0
-1.6
-0.7
-2.8
0.8
bellnnoss
pige
anc challenge
arc.nay
openkcok..g
son
booia
winogrande
Average
Test task
Figure 5: Performance improvement training equivalent amount of random data broken down by task (except for Random 8x, which uses 8x more data). table of results is in Appendix
6 Downstream Tasks
6.1 Setup
Training. We target the 8 tasks from the OLMo paper [Groeneveld et al., 2024]] Hellaswag [Zeleers
et al. 2019). PIQA [Bisk et al. 2020], ARC-challenge and ARC-easy [Clark et al. 2018 Openbook QA Mihaylov al. [0188]. SciQ [Welbl al., 2017]. BoolQ (Clark et 2019), and Winogrande Sakaguchi et 2021]. Each of these datasets has separate train split. We use these train splits to
construct Down as follows: for each question we concatenate the question and the correct answer formatted as grammatical continuation. Overall, this results in small Daown dataset of 7.4 million tokens. D Dprior and Dirain are the same as before. And we again get down by fine-tuning prioe for
one epoch on Doown
CoLoR Filter
RHO down prior
Random Ix
Conditional only
Evaluation. We evaluate on held-out data from each downstream task test or validation sets (using val if test not publicly available).
RHO-down
50
Final Aunrage Accuracy (1)
We use the evaluation procedure from OLMo [Groeneveld et al. 2024] which follows [Gao
49
al. 2023 for evaluating these multiple-choice tasks using the rank classification approach of Brown et al. [2020). We report aggregat perfro-
47
mance across tasks as well as the task-specific performance.
46
Baselines.
Same as in Section 5
Figure 4: Final performance versus on the suite of downstream tasks for 150m models. CoLoR- Filter scales the best with т.
6.2
Results
While the curves themselves are noisier now due to the noisier nature of accuracy evaluation on small datasets compared to cross entropy on large one, the same trends hold as we saw for domain transfer to Books. CoLoR- do not illustrate the same clean scaling as we increase т, which is nearly linear on log scale for CoLoR-Filtes seen in Figure 4
in particular is scaling the best
increase Other methods
Full learning curves are in Appendix
We can also look the performance broken down by task and illustrated relative to training on an equivalent amount (3.1 billion tokens) of randomly selected data for = 16 illustrated in Figure 5] We see especially large gains on Hellaswag, ARC easy, Openbook QA and SeiQ and actually see performance decreases on BoolQ and Winogrande. However, we should note that at this scale and with all data selected from C4, we actually found BoolQ and Winogrande to quite noisy and not correlated with training on 8x as much random data, it is not clear how much weight to place

---

on those results. Across the other tasks, the gains of CoLoR- --Fterr over the baselines are clear. It is an interesting direction for future work to probe more deeply into how task-dependent the gains from targeted data selection can be.
Scale generalization. We also consider scale generalization to a 1.2b target model and illus- trate the full results of a sweep over r in Figure Again we find significant benefits Filter across scales. A full table of per-task re- sults is in Appendix [ Again we notice that training strict subset of data can outperform a larger dataset.
Average Downstream Accuracy (1)
60
CoLoR- 55
50
FilterFilter (T 64)
CoLtR-Filter (r = 32)
CoLoR- Filter (r 16)
45
We can again do out the calculation of com- putational savings for about 3 billion tokens for CoLoR- Rillet to match the performance of training on 25 billion ran- dom tokens. This amounts to a total cost of 3mm+2rn 4. = 3+3.1 1-12 154.8, which is still an upwards of 2.5x reduc- tion in compute to achieve the same average performance across the suite of tasks. A full plot visualizing the cost in FLOPs for all is in Appendix D
CoLoR- Filter (T
16. now takes
Random
40
Takens (billions)
Figure 6: Scaling Co--oR-ilter with when train-
ing 1.2b models with data selected using smaller 150m models. Curves end when exhaust the data in C4.
Task generalization. We can also test task generalization beyond the tasks that were used to select the data on a few more tasks that test common sense reasoning [Wang et al. 2019 Socher jt al., [013]] Talmor et al. 2018 Sap et al., 2019]. Results are presented in Table 2 compared to a random model trained
10x as much data. The performance indicates that the data selected by CoLoR-Filter are not overfit to the particular evaluation tasks, but captures some general notion of good data for range of tasks.
Table 2: Task generalization for the 1.2b models with r = 64.
Method
copa
rtc
sst2
commonsense qa
social iqa
Random (25b tokens)
69.2
48.9
42.8
46.8
33.7
42.9
CoLoR-Filter (r = 64, 2.5b tokens)
65.8
52.6
46.0
55.8
32.6
42.7
Note, we also conduct a few more experiments and ablations in the appendix: Appendix F considers using CoLoR- Filter in-distribution target C4 loss, Appendix (ppendix(Fconsiders:: applying CoLoR-Filter batchwise rather than globally, Appendix Srrendix/Gcconsiee considerss finetuning on Doow after targeted pre-training, Appendix [ inspects some of the selected and excluded examples, and Appendix [ compared to Fine Web-edu [Penedo el al. 2024].
7 Discussion
While fairly simple to derive and implement, we show that Col LR--Fiiitter an effective method for data selection C4, with promising scaling behavior up to 1.2 billion models. In our experiments, CoLoR-Filte continues to improve when only using 1 out of 64 data points considered for selection and generalizes from small auxiliary models to larger target models. This opens many potential lines of research. First, while we have considered targeted pre- treitraining, it is possible that CoLoR- -ilter could be extended fine-tuning fine-tuning, continual pre-training, and more general open-domain pre-training. In particular, it is an interesting open question whether the lack of an explicit consideration of data diversity hinders CoLoR-Filted in any of these settings. Second, CoLoR-Fille could be applied to more challenging domains in language like code generation even applied beyond the language domain to other modalities. Finally, there is plenty of work to be done make the algorithm more efficient and to test the limits of scale generalization.
10

---

Acknowledgments
HZ. supported by an Eric and Susan Dunn Graduate Fellowship. SK acknowledges support from the Office of Naval Research under award N00014-22-1-2377 and the National Science Foundation Grant under award #IIS 2229881. This work has been made possible part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprini urXiv:2303.095400 2023.
Zeyuan Allen-ZMe -hu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024.
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34:8927- 8939, 2021.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03677 2019.
Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. Prediction-oriented Bayesian active learning. International Conference on Artificial Intelligence and Statistics, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 13011))
Adam Casson. Transformer flops, 2023.
URL https://sddmcaasoo..co//oo
transformer- flops
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-45, 2024.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1r0 2018.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL -Data
Abhimanyu Das and David Kempe. Approximate submodularity and its applications: Subset selection,
sparse approximation and dictionary selection. Journal Machine Learning Research, 19(3):
1-34, 2018. URL /tttt://jtmmmml wwwpallr.rgg/ppppr.//eooe http:/ppppsssvvee 534. html
Yanai Elazar, Akshita Bhagia, Ian Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? In The Twelfth International Conference on Learning Representations, 2023.
11

---

Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels, 2024.
Talfan Evans, Shreya Pathak, Hamza Merzie, Jonathan Schwarz, Ryutaro Tanno, and Olivier J Henaff. Bad students make great teachers: Active learning accelerates large-scale visual understanding. arXiv preprint arXiv:2312.053266 2023.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks, 2023.
François Fleuret. The little book of deep learning. A lovely concise introduction, page 297, 2023.
Lco Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot org arg/rrccords/102533300
language model evaluation, 12 2023. URL Bttp:tts:://eeno
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pages 1311-1320. Pmlr, 2017.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam. Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradcep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo. Luca Sol- daini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah. Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rac, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. stat, 1050:24, 2011.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Data- models: Predicting predictions from training data, arXiv preprint N2ti::2202.00622.. 2022.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427 -31. Association for Computational Linguistics, April 2017.
A Kirsch. Advanced deep active learning and data subset selection: unifying principles with information-theon theory intuitions. PhD thesis, University of Oxford, 2023.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424 8445, 2022.
12

---

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha. Sedrick Keh, Kushal Arora, et Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406. 11794, 2024.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujuu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need, 2024.
Dennis V Lindley. On measure of the information provided by an experiment. The Annals of Mathematical Statistics, 27(4))98-10055 1956.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Mcasuring the effects of data age, domain coverage, quality, toxicity. arXiv preprint arXiv:2305.13169, 2023.
lan Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo. Dirk Groeneveld, Iz Beltagy, Hannanch Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit, 2023.
Meta. Llama 3, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: arXiv: 1809.02789, 2018.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. International Conference on Machine Learning, pages 15630-15649, PMLR, 2022.
Robert Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACL 2010 conference short papers, pages 220-224, 2010.
George L. Nemhauser, Laurence A. Wolsey, and Marshall L. Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical Programming, 14:265-294, 1978. URL https::///// /taps:://ai.. api. www.ettiiaccool...oooo... org/Corpus 206800425
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303. 14186, 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 111001116,111 116, 2023.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406. 17557, 2024.
Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2 1446, 2021.
Colin Raffel Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-6) 2020.
Tom Rainforth, Adam Foster, Desi R Ivanova, and Freddie Bickford Smith, Modern bayesian experimental design. Statistical Science, 39(1):100-144 2024.
13

---

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.097 2019.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 1155278-55244 2022.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Daria Soboleva, Faisal Al-Khateeb,
Robert Myers,
Jacob R Steeves, A 627B token cleaned and
Joel
Hestness, and Nolan deduplicated version -awppaajaaaaa-22------2- 2023.
Dey.
SlimPajama:
RedPajama.
attps://wwen wwtp///wwwwweeersaae/e//eaeaaaaa neth//agsnet//bogg////o
-redaaajaja
627B
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational
Linguistics. URL https::t::// h/ww:///wwwww//w www.ww..wlwb....... org wttgg/nnhhlogyy/Dae (otpoolyyyyyy11111 DD13 1170
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint 24Xiv:240.00/559 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523- 2022.
Jihoon Tack, Subin Kim, Sihyun Yu, Jaeho Lee, Jinwoo Shin, and Jonathan Richard Schwarz. Learn ing large-scale neural fields via context pruned meta-learming Advances in Neural Information Processing Systems, 36, 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: question answering challenge targeting commonsense knowledge. arXiv preprint arXiv: 1.088777.07777... 2018.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving Ilm pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuner chat models, 2023b.
14

---

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-purpon language understanding systems. Advances in neural information processing systems, 32, 2019.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: 100.00209, 2017.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003-4012, 2020.
Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learning Representations, 2024.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang, Data selection for language models via importance resampling. Advances Neural Information Processing Systems, 36: 34201-34227, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:195.007830, 2019.
15

---

A Learning curves for 150m models
CoLoR Filtor
Conditional Only
Enempy
30
RHO-down
RHO-Jown Prior
5.0
Entropy (1)
Baselines
4.5
Final: Random
Final:: Random &
4.0
Tokens (billions)
Tokens (billions)
Figure 7: Sweeping over when targeting Books from C4 for 150m models.
Colto-FFilter
Conditional Only
Average Accuracy (1)
46
RHO-down
RHO-down Prior
Average Accuracy
Random Ix
Final: Random
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
(billions)
(billions)
Figure 8: Sweeping over 7 and measuring average performance on all downstream tasks for 150m models.
Tables of downstream results
16

---

Table 3: Performance for all tasks for 150m models for data selection with 16.
hella- swag
open- book qa
wino- grande
Method
piqa
arc-c
arc-e
sciq
boolg
Avg
Random 1x
33.2
64.5
22.4
44.4
26.8
66.9
58.8
53.3
46.3
CoLoR-Filter
38.6
68.7
25.3
51.8
32.0
72.8
54.3
49.4
49.1
Conditional Only
33.0
65.6
23.0
42.2
27.2
64.6
61.4
51.1
46.0
RHO-down
35.5
67.3
25.3
46.9
29.2
67.5
48.6
48.7
46.1
RHO-down + prior
35.6
66.6
25.3
49.3
29.4
69.0
61.6
50.9
48.5
DSIR Random 8x
37.6
68.8
24.4
46.6
27.8
68.4
59.9
52.6
48.3
38.2
67.8
23.5
44.2
28.8
65.3
58.1
50.5
47.1
Table 4: Final performance for all tasks for 1.2b models. Note that the CoLoR-Filce models do not train on as many tokens since we exhaust all of the tokens in C4 with these settings T.
Method
hella- swag
piqa
arc-c
arc-e
open- bnok qa
sciq
boolg
wino- grande
Avg
Random (25b tokens)
52.9
73.0
26.1
53.7
32.8
75.5
56.7
54.3
53.1
CoLoR-Filter (r 7, 25b tokens)
62.3
75.6
29.7
60.3
38.0
79.7
48.3
58.0
56.5
CoLoR-Filter (r 16, 10h tokens)
59.3
75.4
31.7
62.7
36.2
81.0
57.7
56.4
57.6
Col-oRFite (r -22 tokens)
54.8
74.3
29.4
60.9
35.4
78.4
59.1
54.1
55.8
CoLLoR-Filee (r 2.5h tokens)
49.3
73.2
28.9
59.7
35.6
77.1
59.8
53.0
54.6
C Data diversity and online vs. offline selection
Much work active learning focuses ensuring that we select a diverse set of data points that cover the test distribution interest. explained in the main text, by making a point estimate parameters, CoLoR-Filee is simplifying the problem and sacrificing an explicit term for diversity in the objective. practice, this seems to be saved by the facts that (1) C4 has already been deduplicated, (2) still select a fairly large subset without replacement, and (3) an individual sequence contains diversity across tokens.
However, the fact that CoLoR-Filted sacrifices notion of diversity in the objective is important to consider more deeply. Here, we derive what loss based algorithm for data selection that prioritizes diversity would look like and why it is computationally infeasible. Then we derive an approximation (that looks somewhat like RHOLoss [Mindermann et al., [002]) and show how empirically unstable, as was also observed previously by [Mindermann et al. 2022
To derive CoL.oR-Filter-l algorithm that values diversity, greedy approximation where samples are added to S' one (batch) at a time, like in RHO:
start from Equation by
log /. Pr(x.(0) Pr(0) Doown: x<3) + log Pr(z, Pr(f
min
(11)
Dinis
i=1
Note that this sort greedy algorithm for subset selection has a long history in active learning Das and Kempel 2018), is actually theoretically sound in some cases [Nemhauser et al. 1978), and is used in prior work [Ash et al., [021] Mindermann et al.) 2022). Importantly, this algorithm still prioritizes selecting diverse dataset. By conditioning on past data at step the objective encourages the algorithm to select data that is different from data that has already been selected.
We also make empirical bayes version by adding Dprior
min
log
Pr(x,10) Pr(0) Priior Drown: IZ3)
(12)
Ee Diran
+ log Pr(x:(0) Pr(0) Ies)
(13)
This is, of course, still intractable since it requires integrating the parameters. But, since we have already introduced the greedy algorithm that encourages diversity, if we now make the point estimate
17

---

Transfer from C4 to Books (T ()
Training loss
Books Entrapy (1)
Online Selection
Random
Training Cross Entropy (1)
3.6
Conditional Los
Marginal
5.0
Final: Random 1x
0.5
Final: Random 8x
4.4
3.3
0.6
5000 10000
15000
20000
5000
10000
25000
Stop
Step
Figure 9: (Left) Performance of online selection with fine-tuning outlined in Equation (14 selection is worse than random. (Right) Training curves for the conditional and marginal models on the selected data S. The conditional model faces training instability early (associated with forgetting), and then eventually becomes better than the marginal on the selected data.
Online
approximation, the incentive for data diversity remains. This results in:
min JP Pro Dean
log Pr(t::/ppporrtuwwaazeee + log Pr(x; (pi
(14)
The thorny issue here is how to define oprior-dowa--sss and 0 peior+=z2;; in practice. In theory, these parameters should be trained on an iid sample from the union of the datasets. If we add the datapoints one time, the dynamics of the distribution shift over time change how well the model corresponds to conditioning on the union of the dataset. But, this would require re-training the models every time add
2; which is clearly impractical.
In practice, this encourages using fine- tine-tuning approach (as RHO) where we continually fine-tune on the x; they are added. But when Daom is small and the data distribution changes over time, we can get catastrophic forgetting and unstable training dynamics. For these reasons, RHO avoids training the conditional model entirely (Appendix D of Mindermann et al. [2022). We also conduct an experiment on the Books task where we use this online fine-tuning algorithm that updates both the marginal and conditional models unstable and in fact performs worse than random.
add data to S. Results in Figure Ssoo show how the training is
Moreover, Note that the computational cost of even the cheapest fine-tuning algorithm is substantial compared to the algorithms the paper. In particular, the serial cost now 2rn + 4n (as compared to rn + 2n for RHO) since we need to pass the full 7n samples through both the conditional and marginal models. So this variant is clearly inferior in practice the other approaches we consider.
D Compute cost for scale generalization
FLOPs neach ranelom perfrmmrc
2.5
Rooks
FLGPs performance
2.5
<100
Downstream
2.0
2.0
1.5
tokens)
1.5
Training Cost (Serial)
1.0
Scoring Cost (Parallel)
1.0
0.5
0.5
Tau
Tau
Figure 10: Costs in FLOPs to reach equivalent performance to the final random model trained on 25b tokens (i.e. cost until we reach the dotted line in Figure [ We split cost into the scoring cost for filtering the data using the small auxiliary models and then training cost for the large model.
18

---

In the main text we computed the cost for T 16 in terms of model forwards of billion tokens. Here we can convert this to FLOPs and compute the cost for all values of т. Results are in Figure To showing the breakdown of costs into scoring FLOOPs for running the small auxiliary models over the data and training FLOPss for training the large model. We measure the cost it takes to reach the final performance of the random model, l.e. until the CoLoRRfilter learning curve crosses the dotted line in Figure | The main tradeoff is that lower T values require more scoring cost and less training cost because they are able to select better data.
We should also note that if multiple models are being trained with the same dataset, then this scoring cost can be amortized over those runs and the larger T values will look even better.
E
Can we do data selection in distribution?
2
ColidRrFileer
C1 Crass Entropy (
Conditional Only
RHD
3.6
Final: Random
Final: Random 8x
2.0 Tokens (billions)
Takens (billions)
Using sample of C4 as Ddown. RHO provides marginal gains here, while CoLoR-Fille does not provide gains at all. Conditional Only is worse than random. Scaling 7 does not change results as much when target downstream tasks.
Figure 11:
One obvious question raised by these data selection techniques is whether they can work in distribu- tion, i.e. can we select data make the iid loss on C4 go down faster? In Figure T1 we present results for running this experiment with CoLoR-Fille no difference between RHO and RHO prior now (and we drop the "down" from the name) since the prior distribution and the downstream distribution are the same. To implement CoLoR-Filter in this setting, we just take two checkpoints from pre-training the prior model and call the earlier one (at 2.5b tokens) the marginal model and the later one (at 3.lb tokens) the conditional model.
well as RHO and Conditional Only. Note that there
We find that in distribution selection does not work effectively with these methods. There are small gains to RHO loss, but here they are massively outweighed by the computational cost of the selection. CoLoR-Filter sees no gain all over random and Conditional Only worse than random. These preliminary results suggest why it is important to recognize that data selection (especially with these methods) will be most effective when we genuinely want to target a different distribution from D
Clobal FaltR-Filtr
Batchwise CaLaR-Filter
5.0
Books Cross Entropy (1)
4.5
Baselines
Randum
Final: Rondom
Final: Random
Tokens (billions)
Tokens (billions)
Figure 12: Comparison between global and batchwise variants of CoLoR-Filter on Books. The two
perform nearly identically here.
19

---

Global vs. batchwise selection
One more minor implementation aspect about CoLoR- is that as presented in Algorithm 1. we do global selection where we take the best data points across the entire train set, while in RHO-down in Algorithm[ selection is done batchwise. Here we ablate whether the ability to do global selection actually helpful for CoLtR-Filter, Results in Figure 12 suggest that there not much difference between the two and at small т, batchwise selection maybe even beat global selection. We provide this result to illustrate that CoLoR- is fairly robust to how the selection is performed.
G Finetuning after targeted pre-training
One possible question about the targeted pre-training setting consider is: what happens if finetune on Doown after the targeted pre- treitrainng??
This is interesting since while the pre-trained models presented in the main text never have direct access to Dawn: the selection algorithm does. In this section, we also allow access Daown after pre-training and then compare the final performance of the finetuned models that are pre-trained on random data vs. selected data.
First, in Table 55 and Table O we present finetuning results for the 150m models. We find that CoLoR- Filter data outperforms 8x much random data after finetuning. Note that the conditional model that we use to guide the selection of CoLoR- -ilter is equivalent to model that has been pre-trained on 3B random tokens and then finetuned on the task. Thus, these results show that we are substantially outperforming the conditional model when both models are finetuned on the downstream data.
Table 5: Performance after finetuning on Books for different pre-trained 150m models. Note that the Random (3.1b tokens) model equivalent the conditional model used to select data with CoLoR-Filten (r 16).
Pre-training data
Finetuned Books Val Cross Entropy
Random (3. tokens)
3.441
Random (25b tokens)
3.357
CoLoR-Filter (b.1b tokens)
3.258
Table Held out performance after finetuning on downstream data for different pre-trained 150m models. Note that the Random (3.1b tokens) model is equivalent the conditional model used to select data with MoLoR-Filted (T 16).
Pre-training data
hella- swag
piqa
are-c
arc-e
open- book qa
sciq
boolq
wino- grande
Avg
Random (3.lb tokens)
34.4
66.6
24.8
51.7
28.0
89.9
65.6
53.1
51.8
Random (25b tokens)
39.5
29.2
53.9
30.2
91.4
64.2
52.9
53.9
CoLoR-Filter (b.lb tokens)
39.2
71.1
29.1
55.3
33.2
90.0
65.1
51.6
54.3
Table Performance after finetuning on Books for different pre-trained 1.2b models. Note that the conditional model that selects data is only 150m parameters.
Pre-training data
Finetuned Books Val Cross Entropy
Random (25b tokens)
3.074
CoLoR-Filter (2.6b tokens)
2.964
Next, we present results the 1. 1.2b models in Table 7 and Table
find that the CoLoR-Filte
model outperforms or is competitive with training on about 10x as much data randomly selected data. We should also note that the Col Fo---err models are now dramatically outperforming the 150m conditional models that were used to filter the data, showing positive scale transfer of data selection.
20

---

Table 8: Held out performance after finetuning on downstream data for different pre-trained 1.2b models.
Pre-training data
hella- swag
piqa
arc-C
open- book qa
sciq
boolq
wino- grande
Avg
Random (25b tokens)
55.3
74.6
35.2
63.0
35.8
94.6
72.0
62.5
61.6
CoLoR-Filter (2.6b tokens)
53.4
76.1
35.8
65.6
36.8
93.2
66.6
58.9
60.8
H
Hyperparameters
Table 9: 150m model parameters. based on Wortsman et al. [2024), Groeneveld et al. 2024
Parameter
Value
Residual dimension
1024
Depth
12
MLP hidden dimension
4096
Activation
GeLU
Head dimension
64
Context length
512
Positional encoding
RoPE
Bíases
False
Normalization
PyTorch Layernorm
QK normalization
True
Precision
Mixed, bfloat16
Tokenizer
GPTNeox
Table 10: 1.2b model, based on Wortsman et al. 2024 differences from 150m.
Groeneveld et al. 2004 Only reporting
Parameter
Value
Residual dimension
2048
Depth
24
MLP hidden dimension
8192
21

---

Table 11: Training parameters, based on Wortsman al. [024). Groeneveld al. 2024
Parameter
Value
Optimizer
Adam
Batch size
256
Learning rate
le-3
Schedule
Linear warmup, cosine decay
Warmup steps
5% of total steps
z-loss coefficient
le-4
Weight decay
0.0
B1
0.9
B2
0.95
le-15
I Inspecting the selected data
In this section, we conduct some basic analysis of the data that selected by CoLoR-Filter We leave full analysis to future work, but here we provide some high level statistics about the distributions of the scores of the conditional vs. marginal models and some representative examples from datasets.
L.1 Distribution of scores
First, simply plot the CDFs of the conditional loss reduction (CoLoR) score function used select the data. We find that there are relatively few outliers and the CoLoR scores are fairly concentrated and normally distributed. Moreover, meaning that the conditional model actually has higher losses on the datapoints in C4 than the marginal model. This makes sense because the conditional model has been finetuned on Doown which is out of distribution relative to C4.
note that the mean CoLoR in both experiments is positive,
Turgeting Books
Targeting Downstream
1.0
0.8
0.8
CDF
0.4
0.4
6.0
10
Conditional Loss Reduction
Conditional Loss Rodiactio
Figure 13: CDFs for the conditional loss reduction (CoLoR), i.c. loog Pr(x 6 0 (prior-downl (- log pr(r))ppor)) The dashed line highlights the cutoff point for 64. We select the points with the lowest CoLoR.
1.2 Representative examples
Now just list few representative examples to give flavor for the types of outliers that exist under our ranking of sequences and the sorts of typical sequences that are selected versus excluded. The sequences are sampled randomly from different quantiles of the distribution and we shorten all the sequences so that they fit more easily on the page.
Figure shows outliers when targeting Books and Figure 15 shows more typical examples when targeting Books. Generally, we found that the documents with very high scores contain things like old English, poetry, and tables of contents that are particularly unusual in books compared to the rest of the internet. Other things like fiction and dialogue are also highly scored. Negative outliers typically have things like poorly encoded text advertisements.
22

---

Figure 1o]shows outliers when targeting downstream tasks and Figure shows more typical examples when targeting downstream tasks. Here the patterns are less clear since the target tasks are more diverse, but we did observe many scientific and wiki-style documents with high scores as well as some descriptions of physical interactions that may be useful for common sense tasks. Again, the negative outliers tend to have things like poorly encoded text or advertisements.
shall ye wyt, what tyme the day shall angle. From the begynning of Maye vntill it September: the byting tyme is early the morow from four the clocke vnto eyght of the clocke, at after none from foure to eyght also, not good as in the mornyng, and if colde wynde and a lowryng day, it is muche better than cleere daye.
??????????????????????????????
P???????????????777
??????????????????
????????????????????
????????????????????????????????
???????????????????????
???????"
??????????
Also many poole fyashes will byte best in the morne tyde. And if ye any tyme the day the Troute or greylyng lepe angle to him with dub according to the same moneth. And where the water ebbeth and floweth: the fish wyll byte in some place at place the flud after they haue restyng
?????????????????
???????
????????????
m88
???????????????? ????
m88
????????????????????????????????
???еии????????си? ????
?????????????????????????????"?"
the ebbe and
???????????????????? ? ????????????? (b) Bad outlier, CoLoR 5.45
(a) Good outlier, CoLoR=
Figure 14: Examples of outliers when targeting Books. Examples sampled randomly from the top or bottom 1000 sequences. The positive outlier is written an older dialect of English which may be related some documents in the Project Gutenberg corpus, while the negative outlier appears to be poorly encoded.
23

---

Chamber of Commerce and other business venues, such as the Gwinnett Civic & Convention Centers and is ideal working environment for commercial businesses and corporations in Northeast Atlanta. The prominent location is heavily wooded, landscaped 6.5 acre site fronting I-85. The exterior features green-tinted thermal glass and the entrance features curtain wall glass leading into a granite-floored lobby with vaulted ceilings. Gvinnett County is home to leading Fortune 500 companies, drawn by its reputation as commerce and technology hub, providing businesses with a regional market of five million people. SERVPRO Gurnee can simplify the restoration process by handling both the initial water damage mitigation and rebuilding the affected areas. Having one qualified company for the entire process can save time and keep costs low.
Mrs Mackenzie, was there ever a time when you felt like you could just hop on a plane and make that flight down to the next State to be with your boys? B: Oh my dear, yes. I feel sometimes
if I'm twenty and so fit and active and can whatever want do and then remember, good grief, I'm 86, you old fool, you can't do that. wish could just fly down there and live with them all together just how was when they were little and I was their Mum and they followed because was so bright and cheery and smart and active and all the things that I'm not now. Oh, I'm so sorry, listen to me. Maybe I'm just losing marbles, what do you think, dear? C: Smiling Imagine if waved magic wand and miraculously you were twenty again. What would you see yourself doing Beryl. Is it ok if call you Beryl?
(a) Sequence from best 3%, CoLoR 0.40
Sequence from median 3%, CoLoR 0.73
Figure 15: Examples of more typical documents when targeting Books. First document from the top 3% that would be selected with 32, and then a document that scores near the median of all documents. The selected document is fictional dialogue while the median document is an advertisement.
24

---

among the pinacoderm are the ostia that allow entry of water into the body of the sponge. These pores have given the sponges their phylum name In some sponges, ostia are formed by porocytes, single tube-shaped cells that act as valves to regulate the flow of water into the spongocoel. other eponges, ostia are formed by folds in the body wall of the sponge. Between the outer layer and the feeding chambers of the sponge is jelly-like substance called the mesohyl, which contains collagenous fibers. Various cell types reside within the mesohyl, including amoebocytes, the "stem cells" of sponges, and sclerocytes, which produce skeletal materials. The gel-like consistency mesohyl acts like an endoskeleton and maintains the tubular morphology sponges. The feeding chambers inside the
******* **** EEE8 #E SEEEE ***.
Porifera-pore-bearers
plates ***E OR
(***
******* #* tested),
***..*.
********* 8. SSSSEEEER
*****
capture *HHH 88888334
****** ** *4*4 X BO.. ***.**, >10 ..*.** *.*.. OF ****, ..* *****
field 1541, 2EESS 8*) *EEE S4STEEE
S0EETE SEEEES to BESEER 225*** ** night,
SHEEES S5S3SES *RR888S 44 8888888S.
8/888,
**.***** .OU.ON
front
*8*K B00SS. HoNever, *4**
***** ** night, **E.E* * *** scene.
sponge are lined by choanocytes ("colllrr cells").
*** ****888
(a) Good outlier, CoLoR -0.46
(b) Bad outlier, CoLoR 5.36
Figure 16: Examples of outliers when targeting downstream tasks. Examples are sampled randomly from the top or bottom 1000 sequences. The positive outlier is a scientific document that could be relevant for tasks like SciQ, while the negative outlier appears to be poorly encoded.
summer plans. After thinking for while I decided to spend summer in Squamish, where I would work for the Admissions Team. Hovever, due to a very large number of students interested to work on campus and limited number of work positions, ended up not getting job on campus. was very upset indeed and I began to think that there were not any job openings elsewhere, which would then result in travelling back home. Surprisingly, there were many job opportunities in the Squamish community. Since Quest University Canada hosted job fair on campus I, along with all the students, had the chance to meet local businesses that were looking for summer employees. It was great opportunity and give my resume to the ones that interested me. (a) Sequence from best 3%, CoLoR 0.33
Can I install PDF Stacks on more than computer? The license key is valid for only one device and is non-tramsferable. You obtain additional license key by placing order. How do Click "File" and then "Import Folder" Once you import the PDF files, your files will be copied into PDP Stacks for easier ability to read, search, organize, take notes, print and share. Any questions, ask us! How do create collections (virtual binders) and match/tag my documents for better organization? It's easy. Watch the video for creating collections and tagging documents. Can multiple users access the same documents or can I access and sync my documents through multiple devices?
PDF Stacks?
network
(b) Sequence from median 3%, CoLoR 0.55
Figure 17: Examples of more typical documents when targeting downstream tasks. First document from the top 3% that would be selected with = 32, and then a document that scores near the median of all documents. The selected document appears to be journal entry while the median document is software documentation
25

---

Comparison to Fineweb-Edu
Concurrent our initial work, Penedo et al. 2024 released FineWeb-edu, a classifier for educational content that can filter the Fine Web dataset. Here we provide comparison between CoLoR-Filter and this classifier- based approach.
Specifically, we re-implement the CoLoR-Filter pipeline on top of the Fineweb dataset and with slightly smaller auxiliary models (125m) make more fair comparison FineWeb-edi Then compare on the same suite of 8 downstream tasks over various settings of r using the two scores: CoLoR-Filter the WeneWeb-edu classifier. We then train larger models (680M parameters) for consistently
10B tokens of selected data. Results are shown in fig. [8] We find that CoLoR- outperforms Web-euu, which is so surprising since are doing more targeted data selection by specifically targeting the downstream NLP tasks rather than a general notion of "educational content".
CoLoR-Fite
Average Downtrom Acurscy (17
Finettib
0.54
0.52
Figure 18: A comparison the performance of 680m models trained on 10B tokens selected with various between CoLoR-Filter and FibeWed-
K
Broader Impact
The development of the CoLoR-Filter for data selection has notable broader impacts both ma- chine learning and society. It enhances efficiency in language model training, leading to reduced computational resources and environmental footprint, while its scalability democratizes access to high-performing models. The method's 's success in diverse downstream tasks promises advancements in fields like medical text processing and legal analysis. However, it also raises concerns about dataset bias, necessitating continuous evaluation and updates. Future research should focus on ensuring models do not inherit biases from the selected training data, extending applications, improving efficiency, and implementing safeguards to maximize societal benefits while minimizing risks.
L Compute resources
All training is conducted on an internal cluster using H100 GPUs. On one GPU, each 150m training run for 3. lb tokens takes about 4 hours, running the auxiliary models offline and in parallel can be faster. Training the 1.2b model completion takes about days 4 GPUs.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and intro clearly state the key results directly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss limitations with respect to approximations, data diversity, and computational cost.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

We do not have formal theorems, but each step of our derivations is fully stated and assumptions are clear.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All hyperparameters listed and algorithms clearly explained.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer:
[Yes]
Justification:
Code is included in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniioiscyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See Appendix Section
Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since do computationally intensive language modeling experiments and results are clear cut, we do not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwuuopcccpppu..cccccccaa
Answer: [Yes]
Justification:
We conform to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See K
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: We only train small models on well established benchmark data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the code and data we use for the project (all very standard tools in open source ML).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: No new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: No human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: No human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "g3oADUcyTo", "reviewer": "Reviewer_yqth", "summary": "This paper proposes a data selection method that can improve the performance of language models on downstream tasks. The method uses two auxiliary models: one pretrained on the pretraining dataset, and the other is finetuned on the downstream task using that pretrained model. Then the method selects the data that has the largest difference in loss between the two models. Authors formulate the method using Bayesian optimization and show that the method is trying to maximize the likelihood of sequences that will appear in the downstream task. The authors conduct experiments using a 150M parameter model and show that it can be transferred to a 1.2B model. Using the selcted data, the model can achieve better performance on tasks, even when it's pretrained on much fewer tokens.", "questions": "1. What will happen to the downstream tasks that are not included in the data selection process? Will the model perform worse on those tasks?\n\n2. How do you choose the downstream tasks? Any criteria?\n\n3. Do you only consider maximal likelihood for the downstream tasks? What about other downstream finetuining methods, such as RLHF, DPO, etc?", "limitations": "Limitations with respect to approximations, data diversity, and computational cost are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is novel and intuitive. The idea of selecting data based on the difference in loss between two models is interesting and can be easily understood. The method is also well-motivated and well-explained in the paper.\n\n2. The method is well-formulated. It can be viewed as maximizing the posterior likelihood of the downstream sequences.\n\n3. The authors have thoroughly discussed the relevance of the method to other data selction methods. \n\n4. The experiments can very well support the claims. The method is shown to be effective in improving the performance of the model on downstream tasks. Moreover, the method seem to reduce the computational cost of pretraining because the model can be pretrained on much fewer tokens when using selected data.", "weaknesses": "1. The paper has an aim of \"Pre-training\", but the models being tested are relatively small (150M and 1.2B). It's unclear whether the method can scale to larger models (eg, 10B, 100B). The authors should test the method on larger models to show the scalability of the method.\n\n2. In order to select the pretraining data that can improve the downstream performance, the method requires knowing the downstream tasks ahead of time. This is a limitation of the method because in practice, we might not always know the downstream tasks when we are pretraining the model. And people using want the pretrained model to be versatile and not just good at a few tasks.\n\n3. It's not very clear what will happen to the downstream tasks that are not included in the data selection process. It's unclear whether there are side effects of the data selection process on unknown tasks.\n\n4. As the authors mentioned, \"In particular, the FRHOzen objective no longer encourages the selection of a diverse dataset\". So a lack of diversity and overfitting might be a concern when using the selected data.\n\n5. Using a probability-based view, the method is effective in improving the likelihood of the downstream sequences. However, it does not explain or guarantee anything about RLHF/preference-based fine-tuning. And it's unclear how it will impact the model's safety and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their thorough review and largely positive comments. In particular, they highlight that the method is novel, intuitive, well-formulated, situated wrt related work, and has strong experimental results. \n\nIn the rest of this response we will address the weaknesses and questions raised in the review.\n\n## Weaknesses\n\n1. We totally agree that the scale is relatively small, but unfortunately training a 10B model is beyond our computational constraints right now. We hope that the publication of the work will encourage those with more compute to attempt such scaling.\n2. This is a reasonable concern and indeed we may not always know all of the downstream tasks. We would like to point out that this is a larger issue that faces the entire field of language model training. It is not clear how we should define the desired behaviors of a language model or how we should evaluate the performance in general, and it is an active area of research. One primary approach these days is to use a suite of downstream tasks. Our goal with this work is to show how this notion of quality can be leveraged to perform data selection. Moreover, there is no limit on the downstream tasks that can be used as targets in D_down. In our experiments we use a suite of 8 tasks, but we hypothesize that this method could scale to larger suites of tasks (of course this would have to be proven in future work). \nWe also add a new experiment (see the general response and attached PDF) showing that the selected data using our suite of 8 tasks also improves performance by a similar amount on 6 novel tasks.\n\n3. This is highly related to point 2 above, but also a reasonable concern. Note that we do not use the evaluation data directly, but we do use training sets derived from the same tasks. We also add a new experiment to test whether the 1.2B models that target our suite of 8 downstream tasks generalize to other downstream tasks that are unrelated to the selection process. We select a suite of 6 tasks (mostly for glue/super glue) related to natural language understanding and we find that even the model trained on 8x less data selected by FRHOzen outperforms the model trained on randomly selected data by 2 percent. The results are attached as a table in the PDF and in markdown in the global response, we will update the paper accordingly. Thanks for suggesting this issue so that we could add this experiment!\n4. Yes, we agree that a lack of diversity is definitely a worry with this method (as we raise in the paper). However, as we also point out, related existing ideas for how to maintain diversity have serious computational problems and just don’t work as well. We hope that future work can uncover whether this is really a problem at scale or whether starting from enormous web scraped data (which is inherently diverse) and not selecting too aggressively means that this is not a practical issue. \n5. Again we agree that the likelihood is merely a proxy for things we may care about downstream like reward functions or safety, but we think it is a reasonable proxy. For some further evidence that likelihood is often a good proxy see e.g. [1][2] and references therein. Moreover, using likelihood importantly facilitates the bayesian analysis that yields a highly efficient method for selection. It’s not exactly clear how to target other metrics with a similar method, but it is definitely an interesting direction for future work. \n\n[1] Huang et al., 2024, https://arxiv.org/abs/2404.09937\n\n[2] Ruan et al., 2024, https://arxiv.org/abs/2405.10938\n\n## Questions\n\n1. See point 3 above and the added experiment.\n2. We chose the suite of downstream tasks following prior work (OLMo). But in general, this is up to the user. Our paper is focused on presenting the methodology, but this methodology is very flexible (which is why we tried to show it on two very distinct targets: multiple choice evals and books).\n3. Yes, we only consider likelihood at the metric for the tasks when performing selection. This follows directly from our bayesian derivations. It is possible that a similar method could be derived with a different objective, but it is not clear how the bayesian machinery would work when there is an unknown reward function we are targeting. An interesting direction for future work!\n\nWe hope that this clarifies any misunderstandings and we encourage you to increase your score if we have resolved your concerns or to let us know otherwise so we may try to clear up any remaining confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lKeBhfT4uC", "reviewer": "Reviewer_q13t", "summary": "The paper presents FRHOzen, a new data selection method for targeted pre-training of language models, which uses an empirical Bayes-inspired approach to derive a simple, efficient selection criterion based on the relative loss values of two auxiliary models. Evaluated on tasks such as domain adaptation from C4 to Books and multiple-choice question answering, FRHOzen consistently outperforms training on eight times more randomly selected data. It also scales effectively across model sizes, with data selected by 150 million parameter models yielding improvements when used to train a 1.2 billion parameter model.", "questions": "I am curious, when the model is larger (for example, at 7B), will the acceleration still be as significant （8x) ?", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written. The comparison and discussion is very sound, e.g., section 4.2 on computational cost.\n2. The proposed method is quite effective -- outperforms training on 8x as much randomly selected data.\n3. The method's effectiveness in data selection is transferable across models of different sizes, making it scalable.", "weaknesses": "1. Section 2.1, Bayesian Data Selection, lacks rigorous derivation. This part can be considered an intuitive understanding of Bayesian optimization, but it does not constitute a strict derivation.\n\n2. The acceleration effect brought by this paper is quite significant, but the experimental setting uses Books as D_{down} and tests it on the Books held-out set. However, in real-world scenarios, the downstream dataset used and tested should be more general, such as a combination of multiple corpora or multiple end tasks. I hope the authors can confirm that the acceleration effect brought by this paper is applicable to more general datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their positive assessment of our paper. In particular, they highlight the clarity, the discussion of computational costs, the effectiveness of the method, and the scalability of the approach. \n\nIn the rest of this response we will address the weaknesses and questions raised in the review.\n\n## Weaknesses\n\n1. We are not sure we understand the point being raised here. Section 2.1 presents the main objective that we attempt to optimize and then two equalities (Bayes rule and introducing a prior). If any step here in particular is troubling, please let us know and we are happy to explain it further or improve the exposition.\n2. Indeed we present the Books task as more of a didactic example than a real-world application to show how the method can be applied. That is why we also consider a suite of 8 downstream multiple choice tasks (results in figure 1a, figure 3, and figure 4). And in the new experiment presented in the global response also show that this generalizes to 6 more tasks. We agree that future work could apply the method even more broadly than this, but we think this is a reasonable proof of concept for a conference paper. \n\n## Questions\n\n1. We totally agree that this is interesting! But unfortunately training a 7B model is beyond our computational constraints right now. We hope that publication of the work can encourage those with more compute to attempt such scaling.\n\nWe hope that this clarifies any misunderstandings and we encourage the reviewer to increase their score or confidence if we have resolved their concerns or to let us know otherwise so we may try to clear up any remaining confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "35gRDacbEV", "reviewer": "Reviewer_ZLEF", "summary": "This work proposes a simple, intuitive approach for data selection based on empirical bayes formulation minimizing the difference in the likelihood assigned to a candidate training sample by a model trained on a base distribution and a model trained on the base distribution plus a smaller sample of high quality target (test) data of interest. The method is scalable in that it leverages only standard training computation plus additional forward passes and empirical results suggest that it enjoys scale transfer properties where small scale experiments can be used to select data for larger training runs.", "questions": "- A diversity term seems reasonably easy to incorporate. Did the authors experiment with a diversity regularizer term or step of any sort?\n\n- Can we explain why conditional-only worsens with relaxation of the pre-sampling efficiency constraint tau? (Figure 2)", "limitations": "- See weakness.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The setup is easy to follow, intuitive. Theoretically principled from a bayesian perspective while still feasible in practice is rare :]\n\n- Connection to related algorithms is well detailed including complementarity and different tradeoffs against contemporary RHOLoss. \n\n- Use of actual conditional-marginal loss gap is a strength of this method over DSDM and other influence function techniques which have been demonstrated to generalize poorly to realistic scenarios for both computational and broken assumption reasons.\n\n- The scaling results, headline figure, but last section... are impressive. Suggests that this simple method is worth further empirical exploration and expenditure of compute in the future.", "weaknesses": "- The domain transfer experiment should be broadened. Does the result of figure 2 only hold this favorably for the Project Gutenberg Books downstream target? it is possible that improvements are only strong when there exist very distinct sub-distibutions in the prior that match the target. It would improve this empirical section if the authors considered more than one downstream target distribution.\n\n- (More minor, academic constraints assumed) Only one model architecture and scale extrapolation test setting are considered. It would be more convincing if another model family were considered and a few more scales, especially beyond 1B, as some trends in O(100M) models change dramatically beyond a few billion parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their thorough review and very positive assessment of our paper. In particular, they highlight the intuitive and principled algorithm, the connections to related work, the improvement over influence functions, and the impressive scaling results.\n\nIn the rest of this response we will address the weaknesses and questions raised in the review.\n\n## Weaknesses\n\n1a. Yes, we agree that the Books task is a somewhat toyish task. We present it since it is very low-noise and clearly conveys the potential of the algorithm (we should also note that likelihood likelihood is often a good proxy for downstream tasks, see e.g. [1][2] and references therein). For a more real-world analysis we present the same experiment for a suite of 8 downstream multiple choice tasks in figure 4, the curves are slightly less clean due to the noisy nature of multiple choice accuracy evaluations, but the general trend is the same. This mixture of 8 target distributions shows that the method can be useful across many tasks at once by leveraging the downstream data as more of a generic definition of “high quality” data rather than targeting the specific tasks. Of course it would be great to expand this to even more tasks, and we look forward to future work attempting to do this.\n\n\n1b. As a step in this direction, we *add a new experiment* to test whether the 1.2B models that target our suite of 8 downstream tasks generalize to other downstream tasks that are unrelated to the selection process. We select a suite of 6 tasks (mostly for glue/super glue) related to natural language understanding and we find that even the model trained on 8x less data selected by FRHOzen outperforms the model trained on randomly selected data by 2 percent. The results are attached as a table in the PDF and in markdown in the global response, we will update the paper accordingly.\n\n2. We totally agree that the scale is relatively small, but unfortunately pre-training beyond 1B to a ~7B model is beyond our computational constraints right now. We hope that publication of the work can encourage those with more compute to attempt such scaling.\n\n[1] Huang et al., 2024, https://arxiv.org/abs/2404.09937\n\n[2] Ruan et al., 2024, https://arxiv.org/abs/2405.10938\n\n## Questions\n\n1. In keeping with Bayesian formulation, it is actually computationally difficult to solve the full diversity issue posed by general subset selection. We did experiment with different approximations (e.g. RHO) that attempt to somewhat incorporate diversity, and find that they perform worse. We agree that this is a great direction for future work to figure out how to best incorporate a notion of diversity, but doing so with extra terms in the objective function itself is beyond the scope of this paper.\n2. This is a good question. One hypothesis is that when we use conditional-only we are essentially selecting for datapoints that have low loss under the conditional model. There are (roughly) two types of datapoints where this is true (1) data that is relevant for the downstream task, and (2) data that is just easy for a language model (e.g. data that is highly repetitive). By only selecting with the conditional model we mix up both kinds of data, so selecting too aggressively can start to hurt by selecting too much of type (2). On the contrary, FRHOzen explicitly focuses the model on only selecting type (1) and not type (2).\n\nWe hope that this clarifies any misunderstandings and we encourage the reviewer to increase their score or confidence if we have resolved their concerns or to let us know otherwise so we may try to clear up any remaining confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fi3HSvz2Ri", "reviewer": "Reviewer_UH6A", "summary": "This work propose a novel data selection method, FRHOzen (Frozen Reducible Hold Out Loss), which leverages an empirical Bayes-inspired approach to derive a simple and computationally efficient selection criterion based on the relative loss values of two auxiliary models. They provide an empirical evaluation of  FRHOzen on two language modeling tasks: (1) selecting data from C4 for domain adaptation to evaluation on Books and (2) selecting data from C4 for a suite of downstream multiple-choice question answering tasks.", "questions": "1.Could you provide more details for how you fine-tune the olmo using downstream data? What’s the loss function?\n2. What is the unit of data point x in Equation 6?\n\n3.Is the selected optimal data used to optimize the fine-tune phase or the pre-train phase of training?\n4.What’s the difference of data points and sequences (Section 4.2 Line 215)?", "limitations": "As the authors have stated comprehensive in the limitation Section, I have no further comments for this part.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.They analyze a family of loss-based approaches for targeted selection of pre-training data, propose a simple approach that outperforms existing methods, and provide some preliminary evidence of favorable scaling properties.\n2. They analyzed and compared the computation cost of FRHOzen with other similar methods and proved its computational efficiency.", "weaknesses": "1.I think this work could not match its objective of selecting optimal subsets of data for language model pre-training. The evaluation is done by fine-tuning an already-trained LM using downstream data, which could not prove the selection could work for LM pre-training. In fact, a pre-trained LM usually should not be specifically optimized for a particular downstream task (with regard to its domain-specific data).\n2.In addition, the evaluation are based on olmo, which is an decoder-only LM. Exisiting pipeline don’t fine-tune for an decoder-only LMs, usually sft is needed for prompt alignment. I doubt the evaluation results may not be applicable for the real scenario.\n3.The assumption of a pre-set budget n is less valid (in Algorithm 1), and the total number of optimal n is hard to predict during pre-training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their positive comments about how our simple method outperforms alternatives, has evidence of favorable scaling, and computational efficiency. We think these are all important strengths of the paper, and the other reviewers largely agreed. \n\nNow we will address each of the weaknesses and questions raised in the review.\n\n## Weaknesses\n\n1. We think there is a *major* misunderstanding here of our methodology. We are not evaluating a finetuned model, and we are not using any pretrained models from prior work. FRHOzen is a method for filtering a pre-training corpus to select high quality data for training from scratch. We do use a small amount of downstream data as a *guide* to help us define “high quality data” for our selection procedure. In particular, if we look at algorithm 1, we first pretrain and then finetune some small *auxiliary* models in lines 1 and 2. But, these models are then used in lines 3 and 4 to create a new dataset S of high quality data from the pre-training corpus and we then pre-train models from scratch. Thus, all results in the paper are the results for the models that are pre-trained on S in various settings. For example, Figure 1a shows how this yields a pre-training corpus that is much better on a *suite* of 8 downstream tasks for a 1.2B model.\n2. Again we think there is a major misunderstanding here. We use the olmo *code base*, but we do not use the olmo pre-trained model. We train models from scratch. We should note that indeed the models that we pre-train from scratch could be subsequently finetuned with SFT, but that is beyond the scope of the paper for now.\n3. We do not agree here. N can be seen as the compute budget we have for pre-training (it is directly proportional to FLOPs when using e.g. chinchilla scaling to set model size). A user who is attempting to train a model can set this according to their particular constraints for their budget.\n\n### Questions\n\n1. Again we want to reiterate that we are not proposing a finetuning method. However, we understand there may be some confusion since we do finetune the small auxiliary conditional model. For that finetuning, we simply use the standard next-token prediction loss.\n2. We are not sure we understand the question. In equation 6, x_i are sequences sampled from the unfiltered pre-training corpus and we evaluate their log likelihood under various language models (i.e. the sum of the per-token losses across the sequence).\n3. The selected data (S in algorithm 1) is used for pre-training, i.e. training models from scratch. All reported results are for these models that are trained from scratch.\n4. Indeed, datapoints and sequences are the same thing. Since we consider language modeling, each datapoint is a sequence of tokens. For example, in our case this will be a sequence of 512 tokens of tokenized data from a chunk of a paragraph from C4. \n\nWe hope that this clarifies any misunderstandings and that you will reconsider your review and raise your score. If there are any lingering questions, do not hesitate to post them and we will try to clear them up.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to all the reviewers for their constructive comments. We hope we have resolved any misunderstandings.

One experiment suggested by the reviews (particularly reviewers yqth and ZLEF) was to test if the data selected by FRHOzen for downstream tasks generalizes to new downstream tasks that are unrelated to the conditioning set. To this end, we evaluated the final checkpoints of the 1.2B models from figure 1a on a new suite of 6 downstream natural language understanding tasks and we report the results in the table below (and in the attached PDF). We find that the FRHOzen data does generalize, indicating that it is not overfit to the tasks, but is picking up on generic notions of data quality for natural language understanding. Even with 8x less data, the FRHOzen models outperform randomly selected training data by nearly 2 points.


| Method                         | copa  | rte   | commitment bank | sst2  | commonsense qa | social iqa | Average |
|--------------------------------|-------|-------|-----------------|-------|----------------|------------|---------|
| Random (24b tokens)            | 69.2  | 49.1  | 43.2            | 46.9  | 33.9           | 42.6       | 47.5    |
| FRHOzen ($\tau=16$, 6b tokens) | **70.2** | **51.4** | 41.6            | 55.8  | **35.6**       | **44.3**   | **49.8**    |
| FRHOzen ($\tau=32$, 3b tokens) | 67.8  | 50.1  | **46.0**         | **55.8** | 32.5           | 43.5       | 49.3    |

Meta-review: This paper presents FRHOzen (Frozen Reducible Hold Out Loss), a novel data selection method for targeted pre-training of language models. The approach uses an empirical Bayesian approach to derive a computationally efficient selection criterion based on the relative loss values of two auxiliary models which is reminiscent of the Moore and Lewis (2010) cross-entropy difference method. The method is evaluated on two language modeling tasks: (1) selecting data from C4 for domain adaptation to Books, and (2) selecting data from C4 for a suite of downstream multiple-choice question answering tasks.


**Strengths** The paper is novel, intuitive, and  well-motivated, leading to 
solid empirical results, consistently outperforming training randomly selected data. 

**Weaknesses** The techniques are limited to small models (up to 1.2B parameters) and a limited set of downstream tasks.
Decision: Accept (poster)