DataRater: Meta-Learned Dataset Curation
Authors: Dan A. Calian
Gregory Farquhar
Iurii Kemaev
Luisa Zintgraf
Matteo Hessel
Jeremy Shar
Junhyuk Oh
András György
Tom Schaul
Jeff Dean
Hado van Hasselt
David Silver
Venue: NeurIPS 2025 poster ()
Keywords: meta-learning
dataset curation
data rating
bilevel optimization
Primary area: deep_learning
TL;DR: We introduce the "DataRater", a meta-learning approach to automatically learn the value of data and use it to improve the compute efficiency of training foundation models.

Abstract: The quality of foundation models depends heavily on their training data.
Consequently, great efforts have been put into dataset curation.
Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics.
An approach that is ultimately more scalable (let alone more satisfying) is to \emph{learn} which data is actually valuable for training.
This type of meta-learning could allow more sophisticated, fine-grained, and effective curation.
Our proposed \emph{DataRater} is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data.
In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.

DataRater: Meta-Learned Dataset Curation
Dan A. Calian"
Gregory Farquhar"
Iurii Kemaev"
Luisa M. Zintgraf
Matteo Hessel
Jeremy Shar
Junhyuk Oh
András György
Tom Schaul
Jeffrey Dean
Hado van Hasselt
David Silver
Google DeepMind
Abstract
The quality of foundation models depends heavily their training data. Conse- quently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is learn which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine- -re-grained, and effective curation. Our proposed DataRater is an instance of this idea. estimates the value of training any particular data point. This is done by meta- meea-lcarning using ineta-gradients' with the objective of improving training efficiency on held out data. In extensive experiments across range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.
Introduction
is widely acknowledged in machine learning that the quality of model fundamentally depends on the data used to train it. In the era of large foundation models, an enormous quantity of data, encompassing a wide spectrum of sources and reliability, is ingested for pre-training. Careful filtering and curation of this data has repeatedly proven critical for efficiency and capability [Hoffmann et al., 2022 Llama 3 Authors 2024 Parmar et al., 2024). Despite the vast seale of this challenge, laborious human curation approaches are the status quo: in general, data is filtered by hand-crafted heuristics, and final coarse-grained mixtures across sources are manually tuned.
In the near term, these data challenges will only become more acute: the next frontier is synthetic data, which can be generated in unbounded quantity, but can be biased, redundant, or otherwise of low quality. Hence it will be critical to employ systems able to ingest arbitrary amounts and types of unfiltered data, while processing this merged stream of data so as maximize learning efficiency.
Addressing this necessitates automated methods for filtering and optimising the data stream. A highly scalable filtering method would let humans specify what they want at the end of training (e.g.. in terms of a validation loss), rather than specifying how to achieve it (i.e., in terms of manually curating the data stream). This is precisely what meta-learning, as solution method, offers and is the approach we take in our work. Instead of manually specifying the filtering process, meta-learning enables us automatically learn the criteria by which to filter or mix the data stream in data-driven way. If we let it, the data can reveal its own value.
To this end, we introduce the DataRater, method that estimates the value of any particular piece of data for training: the DataRater assigns a (meta-learned) preference weight to any data item, which can be used for filtering re- re-weighting. It is grounded in very simple meta objective: improving
Equal contribution. Correspondence (dancalian, gregfar, iukemaev, 2intgraf)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fraction of compute needed to achieve baseline performance
100%
14.9%
11.95
27.8%
33.2%
22.4%
39.7%
75%
46.6%
DataRater ovemead Net compute gain
Validation set
Validation set
Validation set
wikipedi
BellaSwag
Wikipedis
HellsSwaa
Wikipedia
HellaSwag
Trained The Pile
Trained on C4lnoclean (discard frection 50%)
Trained C4 (discard fraction 10%)
(discard fraction 75%)
Figure Compute needed achieve baseline performance using the DataRater, baseline 1B model is trained on the unfiltered dataset, while a second 1B model is trained analogously on the same dataset, but filtered by a DataRater. The x-axis states the underlying dataset, with evaluation metrics each, while the y-axis shows the fraction of compute needed to match the baseline in grey. The overhead using the DataRater for filtering online shown in pink. calculate this metric convert both the LLM training step cost and the DR inference step costs into FLOPs (while also accounting for batch-size oversampling). "Validation set' refers to the respective validation set the underlying dataset. Net compute gain is shown in green. The figure shows that filtering data using the DataRater results significant overall compute gains for lower quality datasets like the Pile 04/noclean.
training efficiency held out data (Section 2). The DataRater is trained using meta-gradients, which makes it highly sample-efficient compared to black box methods that see the consequence of the data, but do not the function connecting data to performance.It It is effective empirically at reducing training compute for matching performance, especially for filtering low quality training datasets (Section 33. Finally, the principles underlying the DataRater open path toward further gains and use-cases, such as online adaptation during training, robustness to shifts in the data distribution, or tailoring data streams narrowly targeted preferences, discussed in Appendix A
The main contributions this work are:
DataRater framework. We introduce the DataRater, a novel meta-learning framework designed to estimate the value of individual data points for training foundation models, relative to specified meta-objective. Our approach aims to automate dataset curation by meta-learning the value of data.
Scalable meta objective. The DataRater model is optimised using meta-gradients specify objective of improving training efficiency on held-out data (Section 2).
simple-to-
Significant compute efficiency and performance gains. Extensive experiments demonstrate that DataRater-fitered data substantially reduces training FLOPP (achieving up to 46.6% net compute gain, Figure M and frequently improves final performance for language models across diverse pre-training corpora (e.g., the Pile, C4/noclean). as illustrated in Figure
Cross-scale generalisation of data valuation. Critically, DataRater model meta meta-trained with fixed-scale inner models (400M parameters) effectively generalizes its learned data valuation policy to enhance training across spectrum of target model scales (50M to 1B parameters, also Figure 6). Moreover, optimal data discard proportions are shown to be consistent across these model scales (Fi
Insight into data quality, Our analysis (Section 3 reveals that the DataRater learns to identify and down-weite data that aligns with human intuitions of low quality, such incorrect text encodings, OCR errors, and irrelevant content.
2 Meta-Learned Dataset Curation
In this section we formally introduce our framework and derive our method.
The filtering problem. Suppose our aim develop a predictor over an input set D, for example,
D can be the set of all finite sequences of tokens (Io, .......... -F(xi) x(pp) over token set. To do this, we are given a training set Dirain C DD on which we train foundation model with a loss function f, and we aim to use model on possibly different dataset Drest D and loss function L. For

---

DataRater
nnur data batch
Inner duta batch
Inner data batch
Outer data batch
Ratings
Weighted (xx
Weighted
Weighood ennrr lon
Inner model
Oc+1
OT
Ouser loss Oyx;)y(n))
DataRater parameters update trajectory shown green. Inner model parameters trajectory shown blue.
Figure 2: DataRater meta-learning schematic. This diagram shows how the DataRater is updated using meta- seee-grrdientt to minimise an outer loss, by pack-propagating through multiple inner model updates (we experimented with up inner updates) over weighted inner data batches. For more details on the implementation see Algorithm
Algorithm Meta-learning DataRater (Φr))
1: Inputs:
Training dataset Dusin' testing dataset Dies's DataRater o,, with meta- peeta-paramteers 7 and meta-optimiser H; Inner model få parameterised by O' with optimiser θ; K outer update steps; T inner model update steps; inner batch size Nineer and outer batch size Nouters inner loss function E(0; x) and outer loss function ((0; x); number of inner models Meedlss-
2: 7Jo INITMETAPARAMS]
Initialise DataRater. b Initialise inner models.
3: for = O...N....ueee Nmndeels - do
4: 8 + INITPARAMS))
5:
for k
0. K 1 do
P> Outer loop (1)).
6:
for
= 0 O... ...dels - do
7:
fj UPDATE/NNERMODDELCC 7k) Be PNam (Dreu)
> Update inner model for T steps.
8:
Sample outer batch.
9:
gf = Naw ExEB, v,L(xx (xl(xx;ii.. 0):1
>> Form meta- meta-gradient.
T ( sk)
Compute per-mor per-model meta-paramen update.
11:
71+1 Nuraie E'
Update DataRater.
12:
Return 7]K
b Optimised DataRater meta-parameters
13: function UPDATELWERMMODELCC, 17) for / O...T do
Inner loop (0). Sample inner batch. > Form weighted gradient using data ratings. Update parameters. T-steps updated inner model parameters.
15:
B, ~ P Nunar (Duain)
9: Σ ExeB, Vol(x; 0o) 0B, (d,(xx()))
0,+1 (ft.gtt)
18: Return 0r
example, Diest can be clean dataset for problem while D can be its noisy version (e.g.. see Figure 3 for a motivating toy example). Given a parameter vector of the predictor, the loss of sample D is f(x; ()) for the training and L(x; 0) for the test problem.
In this setting a typical gradient-basee based learning algorithm, A, performs steps on random minibatches B, (x) (x...... x²") selected from the training data Dirin for each step O,..,,
N
1
0,11, 8141 ((,, St, 9t.9t)
with
Σ N
Vol(x;
(1)

---

Meta iter. (R² 0.163)
Meta iter. 10 (R? 0.5611
Meta iter.. 30 (R² 006933)
Meta iter 70 (R² 0.813)
0.019
0.020
0.013
0.016
DataRaler ght
0.011
0.008
0.007
0.005
0.005
0.004
0.003
0.000
0.003
00,000
0.50
0.25
0,50
Noise fraction
fraction
fraction
Figure 3: Motivating toy example: DataRater learns weight examples in proportion to their corruption level. Given datasets, D and D. drawn from the same distribution, but where sequences in 'D have been corrupted with token-level noise of various levels, from O (clean data) to 1 (fully random tokens). The DataRater weights correlate increasingly well with the noise fractionn showing that the DataRater recognises that corrupt sequences are worse for training than clean ones.
where θ is the update function of the learning algorithm A, and denotes its state at step t. After T steps, this process leads to (random) parameter vector 0y; we will denote this by (Puuii)))) ~ (Diiin)) where in the notation we omit the dependence on the initial parameter vector 0g and optimiser state 8o for simplicity. The performance of (one run of) the algorithm measured on the test data by
1(0rrrruaii (Duriin; Drea) = Ex [L(x; (Duran))
where x Diest denotes uniform distribution over the dataset Diest (and could be replaced by an arbitrary test distribution over D).
The goal of filtering process is to select subset of training data that leads to the smallest test error,
Dirain = argmin [D)
(3)
Dmin
where the expectation is taken over the randomness of the learning algorithm A.
Note that even when the training and test distributions are the same, Dimin Drean it might be more efficient to train on a subset of Duam only, as the learning algorithm typically does not find the optimal parameter 6* and/or may train faster on well-selected subset of the data (e.g.. omitting data points which are already learnt).
The DataRater algorithm. Diserete subsampling of which data keep known be non- deterministic polmnomial-time hard problem, even for linear models [Davis et al.) 1997 Natarajan 1995]. Like many others [e.g. Liu et al. 2018). solve continuous relaxation instead.
We proceed in three steps. First, instead optimising over the best subset of Drrain in (3). we optimise over which data points (from a larger minibatch) to include in the gradient calculation yr. step this is equivalent to introducing binary weight for each data point in the minibatch.
Secondly, we perform a continuous relaxation, replacing the binary weights with continuous weights in [0, 1]. Thus, instead of (I), we calculate weighted gradient for the minibatch in the form St E Vol;)Vol(x fe), where w(x) E [o, 1] are the continuous weights. To keep the gradient of similar order, enforce that the weights sum up in each batch.
Finally, to represent the weights we use function approximation, rather than tabulating a weight per data point: we introduce a DataRater model (i.e., score function) d,, D - R, parameterised by which measures the value of each data point; we obtain the normalised weights for every point in
batch through softmax preference function: OB, (x), Exee, (
(9960)
Thus, the filtering weights specified by 7) induce training algorithm A(Duaini that for O,...,T T - does: sample B, = (XI...,,Xy) uniformly at random from Dirain and compute
N
8:41 st,,,,,,9))
with
ge =
V Vol(x;; 01) σB, (0,,())))),
(4)

---

We denote the resulting (random) parameter vector by (1) ~ A(Duain
To optimise the filtering algorithm, want to find the DataRater parameters v minimizing the expected test error:
17' = argmin E[L (n): Dieu)
where again the expectation is taken over the randomness of the algorithm.
Meta-optimisation. To optimise the parameters, 17. of DataRater model approximately, we use a meta-gradient method (Xu et al. 2018]: v is optimised via a stochastic gradient method, where the gradient of the loss L (which called outer loss in this context) with respect to rj is computed by pack-propagating through multiple optimiser updates for 0 with respect to the so-called inner loss E, as illustrated in Figure 2 the method is given formally in Algorithm
Meta- Meta-Learning objective. We described the general DataRater framework above. For the remainder of the paper, including all experimental evaluation, we choose to focus on the most widely applicable objective: where the train and test datasets share the same distribution. This choice does not enforce that one must be interested in optimising for some specific downstream task, but rather, it assumes that one simply wants to maximise training efficiency with respect to given dataset.
So, the held out data, which the DataRater outer loss is defined, is disjoint subset of the input training data. Put differently, our objective is to produce a curated variant 'D' of given original dataset, that training on it results in faster learning on the original training dataset. In this case, the inner (é) and outer (L) loss share the same functional form, i.c., cross entropy loss w.r.t the next token prediction, {(x: 9) = L(x: 0) log (x Σ log (2) .o))
=0
Implementation. We instantiate the DataRater as a non-causal transformer and optimise its parame- ters y) using meta-gradient descent; i.e. pack-propagating through the unrolled optimisation of the inner model. We back-propagate through a truncated window of 2 inner model updates to compute the meta- (eeta-gradien[] with second-order derivatives, such as Hessian matrices, which in practice are computationally expen- sive. To make this computation feasible, we use the scalable bilevel optimisation reparameterisation Kemavv et al. [225 called MixFlow-MG. This method exploits implicit symmetries present in bilevel gradient optimisation problems and uses mixed-mode differentiation to significantly reduce RAM (HBM) usage. In particular, we use all the core techniques from MixFlow-Me -GG. including block-level rematerialisation with saving inner-level gradients, and mixed- mode differenniation itself. This enables us to optimise DataRater meta-paramesn parameters efficiently - even for 50M DataRater models and 400M inner models, while mack-propagating through multiple inner model update steps.
3 This process, handled implicitly by automatic differentiation, involves operations
To stabilise meta-gradient computation we: (1) use a population of inner models; (2) pass the meta-gradient for each inner model through an individual Adam optimiser (Kingma and Ba 2015), averaging the resulting meta-gradient updates; (3) periodically reinitialise inner models to stratify learning progress. In the appendix in Figure 12 we plot the norm of the meta-gradient updates over the course of meta- meta-training, for each inner model, showing that they remain stable and bounded. Furthermore, in Figure 13 we show that the temporal auto-correlation of DataRater scores approaches > 0.95 after a few thousand meta-update steps, indicating that meta- meta-trainte training is converging.
Data curation. For data curation, with trained DataRater, we use top- K filtering at the batch-level: instead of weighting data, we remove data with low predicted value. To implement top-K filtering at the batch level, for given target discard fraction E [0, 1). and a nominal batch size N, we upsample larger batch of data points sized _N.. bottom (100 p)-percent of samples.
I-p' score them using a DataRater, and filter out the
DataRater scores can also be equivalently used select data points individually without requiring access to batch of data. This requires access to the CDF of the ratings distribution, F4 R - [0, 1]. To emulate the batch-level decision for top-K filtering given a batch size of B, for a data point x with (o((x)) we can calculate the probability of keeping it as P (x)cep(X) = K-1 (B=1)(1
p)* "''p" x-1 (since we keep a data point if select at most K - 1 "better" points in a batch). This is
PPseudo-code omits optimiser states and periodic resetting of inner model parameters. i We experimented with 1, 2, 4 and 8 inner model updates, and found that using 2 inner updates results in very similar DataRater performance as using more.
5

---

The Pile
G4/hoclean
3.2
3.0
3.0
2.8
Loss on validation
2.8
2.6
2.4
3.4
Model scale
2.6
400M
2.4
2.0
3.0
150M
50M
2.2
1.8
2.8
2.0
1.6
2.6
fraction
fraction
ction
Figure 4: Loss on validation subsets, over multiple model scales, as function of the percentage of data filtered out according to DataRater models (one DataRater model is trained for each dataset). For each underlying dataset (the Pile, C4/noclean and C4), the data filtering proportion resulting in the best validation set performance remains constant across evaluated model scales (highlighted in green).
useful for building massively parallel data filtering pipelines using, e.g., Apache Beam Dean and Ghemawat 2004). as filtering decisions can be made locally.
3 Experimental Results
Evaluation protocol. To evaluate the effectiveness of our method in curating a dataset we perform the following three main steps: (1) meta-learn a DataRater for the input dataset DD; (2) use the trained DataRater to curate it, forming D*; (3) train language models (of 50M, 150M, 400M and 1B sizes) from random initialisation the curated dataset and subsequently evaluate their performance.
Datasets. Our experiments utilise three datasets selected for their varying degrees of pre-filtering: C4 Raffel et al. 2020], the most stringently filtered; C4/noclean, a less-filtered version of C4; and the Pile [Gao et al. 22200. representing the least filtered data. We we use the English subsets of C4, i.c. 04 and C4/noclean refer to c4/en en and c4/ en noclean from huggingface co respectively.
Models and training, All inner language models and DataRater models based the Chin- chilla [Hoffmann et al., 2022 transformer architectures [Vaswani et al. [017]. Models are trained from random initialisation, adhering the Chinchilla training protocols and token budgets; i.e. the learning algorithm (and optimiser update function θ) is defined by the Chinchilla scaling laws.
DataRater configuration. For each of the three datasets, we train distinct 50M parameter non- causal transformer as the DataRater model. Each DataRater is meta-learned using a population of eight 400M parameter inner language models (see [A] meta- meta-learning checkpoint selection details). Importantly, for each input dataset, the inner models and their corresponding DataRater model are optimised on disjoint subsets of that dataset's training data.
Evaluation. We measure negative log likelihood (NLL) on the validation set of each of the three input datasets, as well as on (English-language Wikipedia. We measure accuracy on the following downstream tasks: HellaSwag [Zellers et al., 2019), SIQA [Sap et al. 2019 PIQA [Bisk et al., 2020). ARC Easy [Clark al. 2018) and Commonsense QA [Talmor et 2019), which we selected as they provide useful signal at the model scales we use (i.e. benchmark performance increases during the course of training and across model scales). In Figure we plot the fraction of compute needed to match baseline performance for IB models. To calculate this metric we convert both the LLM training step cost into FLOPs well as the DR inference step cost into FLOPs, while also accounting for batch-size oversampling prior to DR online filtering. We leave this metric undefined for cases where training on the filtered datasets does not reach baseline performance (c.g. for some cases on the C4 dataset).
Can the DataRater accelerate learning? The DataRater indeed accelerates learning, resulting in considerable FLOPS savings for lower quality datasets like the Pile and C4/noclean. For 1B models, Figure [ [shows compared to baseline training, while also showing the compute overhead due to DataRater filtering. Figure 5 shows performance metrics during training of 1B models. This shows that, particularly for
shows precisely the amount of compute saved by using the DataRater, when

---

Velidetion sel, (4)
Wiripedia, NLL14)
Helaaimng, Accuracy (1)
225
43.7%
2.14
2.42
14.20
2.25
41.45
2.24
35.0%
1.40
24.05
1.24
11.00
2.50
Thaining Slapo
Figure 5: Learning curves of 1B models trained three separate underlying datasets, with DataRater filtering and without. On both C4/noclean and the Pile, models trained on DataRater filtered data match the baseline's final performance while using considerably fewer training steps, while also resulting in improved final performance. Using a DataRater to filter C4, which is a consid- erably higher quality dataset, results in minor performance improvements on average (i.e. see Table in appendix for full results) while exhibiting trade-offs in downstream evaluations as exemplified here.
Wikperta (4)
Validviin ML 16)
Heels Swany Acc.
5KyA Apc.
PIDA Arc.
ARC Eary Arc 1t1
Commersmoeeakkrr Art
285
23.15
0.41
6678
1377%
30.45
280
16.8
35.45
5525
19.25
13.25
2500
0..5
1.75
19.15
21.75
224
11.39
01.7
13244
31.65
36.0
31.91
5489
1911
33.55
385
36.8
4324
1808
0.25
1335
35.55
0075
3425
0.25
15:M0 J0M
155 6:24
1510 4716
10tx
1204
1950 4714
Figure 6: Rate filtering results in performance improvements across the majority of model scales and downstream tasks. Fach individual plot shows downstream evaluation performance (measured either as accuracy or NI.L) as a function of model scale. Each row corresponds to one of three separate training datasets, while each column corresponds to specific downstream evaluation.
lower-quality datasets, using DataRater filtered datasets often results in improved final performance, and not just faster learning.
Of course, training the DataRater model itself must also be accounted for. Meta-training a DataRater requires approximately 58.4% of the FLOPS needed to train single IB LLM. Given that the filtered dataset produced by the DataRater will be used to train numerous other LLMs, of possibly much larger scale, we argue that the cost of training the DataRater can be amortised away.
In Section A in the appendix we compare the DataRater approach with a perplexity- -bsed filtering method adapted from prior work [[nknnr al. [025] showing that the DataRater results in better performance in a large majority of evaluations, l.e. in 16 out of 21 cases.
How much data should be removed? To select the best discard proportion for each dataset, we run a sweep over 5 candidate discard proportions (10%, 25%, 50%, 75% and 90%), using the smallest model (50M) and select the discard proportion resulting in the best validation set NLL. In Figure
7

---

1.0
Publled Central
Gthub
AXXiv
M4xtur Woight
Other
0.6-
LSPTD Bar bgrounds
Github
PubMed Central
Open
ArXiv
Other
StackExchange
USPTO Backgroondd ds
Operittaa Twet2
Rice cC
0.2
Plc-CC
Pub Med Abstracts
Putiffed Absitacts
Witipedia (ur)
Witipedia
5.0
6.5
0.0
0.2
0.2
0.3
0.5
6.6
0.8
DataRater Score
Discard Proportion
Figure 7: (a) Distribution of DataRater scores on data subsets from the Pile. (b) DataRater induced mixture weights for the Pile data subsets as a function of discard proportion (from 0% to 80%).
(P') Prodayd
Late Py Katlie
coorr 3.254 (Py) (Sidie
acor L437 (P)) Frodbookrr
6:50 Tiel poossbriannp
p004 tux
0
DE OP JPME spirs
eve: IPU X2RO0V2
DM sow
Ty The Syattbe
5-215 (Pa) Kie
Figure 8: Qualitative samples from the Pile annotated with their DataRater score, score percentile and provenance.
we show that using the smallest model size is sufficient, as the best discard proportion is shared across model sizes. The optimal value depends on the underlying data quality: for C4 we drop 10%, for C4/noclean 50% and for the Pile 75%.
Is the DataRater robust to model scale? We trained single DataRater per dataset, with fixed inner model size of 400M. The DataRater generalises across model scales: in experiments over 3 input datasets over model scales and 7 metrics, 73/84 showed improvements (see Figure For the lower quality datasets, C4/noclean and the Pile, the performance improvements are consistent over model scale for NI.. metrics and HellaSwag, whereas downstream evaluations have more variance.
What does the DataRater learn to do? The DataRater learns to assign fine-grained ratings to different data subsets, as shown in Figure 7 (a). This plot shows the distribution of raw DataRater scores (i.e., o, (x)) for multiple subsets of the the Pile; most subsets are assigned a heavy left tail- identifying data points which should be discarded even at very low discard proportions. In effect, the DataRater also learns to re- weight data at the mixture level, as shown in Figure (b) for increasing discard proportions.
The DataRater also learns to identify low quality data; i.e. samples which are assigned low ratings by the DataRater on the Pile dataset tend to be low quality. As exemplified in Figure these include incorrect text encodings, OCR errors, large amounts of whitespace, non non-standard non- non-printable characters (e.g., in the FreeLaw subset of the Pile), high entropy text such as lists and tables of numerical or string data as well as private SSH keys, English language in all capital letters, multilingual data (the Pile contains over 97% English data [Gao et al. 2020)), etc.
In Section A in the appendix we perform correlation analysis between the DataRater score and common language filtering heuristics, with results shown in Figure [4] The DataRater score is most positively correlated with the number of packed sub-sequences as well as with various metrics of text length; it is most negatively correlated with the proportion of con-alphanumeric characters, of punctuation and of unique characters. An OLS model results in an R² of 0.766. As many of the heuristics are collinear, we fit cross- validatee Lasso (L1) model on Z-scored features to identify
8

---

a small but predictive subset of heuristics. The Lasso model explains 75.3% of the variance in the DataRater scores while using only 11 non-zero coefficients. We find that the DataRater score can mostly be explained by the number of packed nub-sequences, the proportion of non-alphanumeric characters, the word count, the sequence length, the packing density as well as a few other heuristics. Since neither regression model can perfectly predict the DataRater's score, it is likely the DataRater is identifying valuable patterns in the data that are not captured by these simple heuristics.
Further analysis about what the DataRater learns is presented in Appendix B
4 Related Work
The curation of training data remains essential in ML and LLMs [Touvron et al.) 2023 Zhou it al. [2023] Albalak et al., [024]. Automated data selection strategies include heuristic filtering. learning- basen strategies (Albalak et al. 2024 Jiachen Wang
2024), and an emerging trend in bilevel
optimisation methods [Pan et al. 2024 Wang et al. [020 Shen et al. 2025 Dagréou et al. 2022).
Heuristic Data Filtering. Predefined heuristics clean massive raw corpora remain prevalent. The C4 dataset [Raffel et al. 2020), for example, applied rules like language identification using simple classifiers (e.g., using fastText (Joulin al., (0166), removal of lines without terminal punctuation, and n-gram-based deduplication. More recent large-scale efforts such as Web (Penedo et al., 2024] and Dolma (Soldaini et al., 2024] detail extensive, multi-star stage pipelines encompassing URL filtering, quality heuristics (c.g., adapted from Gopher [Rae et al. [022]] or C4), content filtering for boilerplate text, navigation menus, error messages, personally identifiable information, toxic language, and various forms of deduplication using techniques like hashing (e.g.. SHA1, MinHash with LSH), suffix arrays for exact substring matching, or Bloom filters. General categories of these heuristics, including document length constraints, symbol-to-word ratios, and boilerplate removal, are well-established [Albalak et al., 2024 Diachen Wang 2024). While effective, these methods require manual tuning and may not consider less human-interpretable or intuitive features.
Learned Data Selection Methods. To move beyond fixed heuristics, various machine learning techniques have been proposed. Data valuation and influence- based techniques attempt to quantify the contribution of data points to model performance or capabilities. Foundational ideas include influence
functions [Hampel. 1974 Koh and Liang, 2017). Cook's s distance [Cook] 1977]. and the Shapley
value (Shapley et al. 1953 Ghorbani and Zou| 2019 Wang et al. 2025 More recently, JST (Zhang et al. 2025 uses a two-stage valuation to refine the selection of high-quality data by initially
leveraging identified low-quality data. PreSelect (Shum al. 2025) quantifies "predictive strength"
how well data predicts downstream abilities - to guide selection. Relatedly, (Gu et al. (202)) formulates data selection as generalized optimal control problem, and solves it for LLMs approximately, Some approaches train classifiers to distinguish high-quality from low-quality text (Shum et al. 2025 Penedo et al.) 2024]. J Others leverage signals (such as perplexity scores) from pre-trained models Wenzek et al. 22200] or use comparisons to baseline predictors [Lin et al. [2024] Mindermann cct al. 2022 Sow et al. [025]. Dynamic methods aim to sclect data during the training process itself; for example, GREATS Wang et al. 2024] proposes an online batch selection algorithm using Taylor approximations of data quality. Our method shares the goal of learning the data value but uses meta-learning to directly optimise for the outcome of the learning process.
Bilevel Optimisation and Meta-Learning for Data Selection. The most analogous approaches to the DataRater frame data selection as a bilevel optimisation problem to learn a selection or weighting policy. Maclaurin et al. (2015) demonstrate computing exact gradients of validation performance by reversing learning dynamics, while many subsequent works (e.g., [Pedregosa] (016]] Lorraine ctt al. 20200 use the implicit function theorem for efficient approximation. Differentiable Data Selection (Wang et al., 2020 uses bilevel setup where scorer network learns to weight data instances to optimise a model's performance, using gradient alignment as a reward. A similar bilevel weighting approach was used by [Guo et al. 2020 for safe semi-supervised learning. Others use similar bilevel frameworks to learn training data distributions under domain shift [Grangier et al. 2024] to adaptively re-weight per-task gradients to improve multilingual training [Li and Gong 2021
Contemporaneously, SEAL Shen et al. 2025 learns a 'data ranker' via bilevel optimisation to select fine-tuning data to enhance LLM safety while preserving utility; our formulation mathematically allows for arbitrary inner and outer losses, but, in this work, we focus on improving training efficiency

---

specifically. SEAL optimises the bilevel problem using penalty method [Clarke] 1990], while we use exact meta-gradients Concurrent work [Engstrom et al. [025] similarly applies meta- meta-gradients to optimise training configurations, including to select multi-modal data for CLIP [Radford et al. 202T] and instruction tuning data for Gemma-2B [Gemma Team, 2024) with a similar meta-objective as our method. Their method tracks one meta-parameter per data point, while we use function approximation to learn to assign value to arbitrary individual data points. We use all techniques from MixFlow-Me [Kemaev et al., 2025] (gradient checkpointing, mixed-differentiation etc.) to efficiently compute meta-gradients, while their method introduces the 'REPLAY" algorithm for the same purpose.
Conclusion
This paper introduces the DataRater, novel meta-learning approach for dataset curation that estimates the value data to enhance the compute efficiency of training foundation models. By employing meta-gradients, our method is trained with the objective of improving training efficiency on held out data. Our experiments across various model scales and datasets demonstrate that our method is highly effective in filtering data, leading to significant compute efficiency improvements.
Our method can capture fine- fine-grained distinctions in data quality, identifying and discarding less valuable data points which correlate with human intuitions of low quality text data. We also demon- strated robustness compute efficiency improvements across different model sizes. The DataRater is particularly effective when the underlying dataset is low quality, may be well suited to contexts where the quality of data is highly variable and human intuition struggles to easily define heuristics to estimate that value. Future work applying the DataRater to synthetic data or sensor data may therefore particularly promising.
This approach offers meaningful step towards automating and improving current dataset curation practices, which currently rely heavily on manual tuning and handcrafted heuristies. Our method provides a scalable and principled way to determine the value of individual data relative to the specified meta-objective, and shows considerable compute efficiency gains for filtering multiple real-world datasets when training on models of significant scale thus providing a first proof of concept that meta-learning the data curation pipeline for modern LLMs is possible.
Acknowledgements
The authors would like to express their gratitude to John E. Reid, Paul Michel, Katie Millican, Andrew Dai, Andrei A. Rusu, Jared Lichtarge, Chih-Kuan Yeh, Oriol Vinyals, the Gemini Data Team and the RL team for fruitful discussions, valuable feedback, and support throughout the project. The authors are also grateful the anonymous reviewers for their insightful comments and constructive suggestions.
References
Alon Albalak, Yanai Elazar, Sang Michael Xie. Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language mod- els. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https Survey Certification.
//openceview.net/eeeee net.
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity- based data pruning with small reference models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://
openreview htt.entttttooommdaamaaaaaaa
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium Educational Adyunces in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439. AAAI Press, 2020. doi: A0.1609/AAAL V34105.62 URL attpp:///oo org/ 1609/ hww/aaai..... v34105. 6239
10

---

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milnee, al. Jax: compos- able transformations of python+ numpy programs. 2018.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning chal- lenge, ArXiv, abs/ 11011103350555 2018. URL https: somanticscholar wwwpeoorggoriioeee Org/CorpusID 3922816]
Frank H Clarke. Optimization and nonsmooth analysis. SIAM, 1990.
R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1):15-18. 1977.
Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
G Davis, S Mallat, and M Avellaneda. Adaptive greedy approximations. Constructive approximation, 13(1)557-98, 1997. ISSN 0176-4276.
Jeffrey Dean and Sanjay Ghemawat. Mapreduce: Simplified data processing on large clusters. In OSD004: Sixth Symposium Operating System Design and Implementation, pages 137-150, San Francisco, CA, 2004.
DeepMind, Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju. Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Clau- dio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Turii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojević, Wojeiech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL
http://github. com wwtp://o///oooeeedeeeimiiiooooie
Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, and Aleksander Madry. Optimizing ml training with metagradient descent, 2025. URL https::/asx:::ii org/
abs/ 13751
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101,00 10,00007 2020.
Gemma Team. Gemma: Open models based on gemini research and technology, 2024. URL
https://arxiv. orgz//iivorr//ooo abs/2403 28295
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning.
In International conference on machine learning, pages 2242-2251. PMLR, 2019.
David Grangier, Pierre Ablin, and Awni Hannun. Bilevel optimization learn training distributions for language modeling under domain shift. In NeurlPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2024. URL D67r01BYYP
wttp://iiiiiieeeetteotreeeeiaei id=
Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttp: wwweeeeeeeeeeeeeeeeeeeeeeee net forummorrmm/mm www.ormmmmddddaleettaaaaaa
Lan-Zhe Guo, Zhen-Yo Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zui-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In Hal Daumé III and Aarti Singh, edi- tors, Proceedings of the 37th International Conference Proceedings of Machine Learning Research, pages 3897-3906. PMLR, 13-18 Jul 2020. URL htttps:/proceedding mlr. w.r.ppressvviv wwtps//vi////// wttps///011///0//i.ooo html guo20i
Machine Learning, volume 119 of

---

Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-393, 1974.
Matteo Hessel, Manuel Kroiss, Aidan Clark, Iurii Kemaev, John Quan, Thomas Keck, Fabio Viola, and Hado van Hasselt. Podracer architectures for scalable reinforcement learning. arXiv preprint arXiv:2/04.06277 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals. and Laurent Sifre. Training compute-optimal large language models, 2022. URL htttttts////aiii/. http:/aaxxvv..oraabse/e. org/abs/2203 15556
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International
Conference on Learning Representations, 2022. URL https::/ wet.:eiwwwwwweieeee.......eeee.e mettfforumiit nZeVKeeFYf9
Nathan Zixia Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. Meta-learning online adaptation of language models. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL
Ruoxi Jia Jiachen Wang, Ludwig Schmidt. Advancing data selection for foundation models: From heuristics to principled methods. NeurIPS 2024 Tutorial, 2024. URL https://neurips. cc/ virtual/2024/uuue http:///222//tt//////////0 tutorral/999900 Accessed on: 2025-04-30.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint
2016.
Jurüi Kemaev, Dan A. Calian, Luisa M Zintgraf, Gregory Farquhar, and Hado Van Hasselt, Scalable meta-learning via mixed-mode differentiation. In Proceedings of the 42nd International Conference on Machine Learning. volume 267 of Proceedings of Muchine Learning Research, pages 29687- v267 /kemaev2a
29705. PMLR, 13-19 Jul 2025. URL htpp::/p///prreeiiings mlr pww.pp.ss///.e.. press/v267
html
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International conference on learning representations. 2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In PMLR, 2017.
International conference on machine learning, pages
Xian Li and Hongyu Gong. Robust optimization for multilingual translation with imbalanced data. Advances in Neural Information Processing Systems, 34:25086-25099, 2021.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, yelong shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Not all tokens are what you need for pretraining. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL wwt mett//Forummii--oiaBBaaaa.
btttps://pperrett
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 10X.v:1806.09055, 2018.
Llama 3 Authors. The llama 3 herd of models, 2024. URI httpps://aaxiv.or//aaaaaa
21783
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference artificial intelligence and statistics, pages 1540-1552. PMLR, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL U iid Bkg6
12

---

Dougal Maclaurin, David Duvenaud, and Ryan Adams. tradieet-bbssed hyperparameter optimization through reversible learning. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference Machine Learning. volume 37 of Proceedings of Machine Learning Research, pages 2113-2122, Lille, France, 07-09 Jul 2015. PMLR. URL URR.https:/ htapttps:/pproceednggg..
mlr. w...press/vee v37 v37 maclaurin15 html
S" oren Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H" oltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15630-15649. PMLR, 17-23 Jul 2022. URL mlr
httpss://proceeeeeee
hhtmmmm222mmmm1mm11mmmmmm html
B. K. Natarajan. Sparse approximate solutions linear systems. SIAM Journal on Computing, 24(2):227-234, 1995, doi: 10.1113/5009975539922244444 URL
50097639792240405
Rui Pan, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Xiaoyu Wang, and Tong Zhang. Scalebio: Scalable bilevel optimization for Ilm data reweighting, 2024. URL https://:: /tctpp::/arxiv.org/// abs/ 19976
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Data, data everywhere: A guide for pretraining dataset construction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages (0671-10695, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 100533/11/222.mmmlppp 596. URL httpps://aclanthooooo...ooo/ org. matnnlpmmain5550
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning- Volume 28, ICML' 13, page I11-130--11-1318 JMLR.org, 2013.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pages 737-746, New York. New York, USA, 20-22 Jun 2016. PMLR. URL httpps:/pproceddinng mlr. mww.mrrss............. v48/ pedregosa16 html
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Uirich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024. Vancouver, BC, Canada, December 10 15, 2024, 2024. URL /ttpattp:ppppppss..iiiiiie cctps///ppr...ppp.ppppp.a.ppeppe files/ Files/paper/ and Benchmarks Track. html
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https:
Jack W. Rac, Sebastian Borgeaud, Trevor Cai, Katic Millican, Jordan Hoffmaan, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang. Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadch, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch,
13

---

Tean-Bapting Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Masson d' d'Autume. Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/21 12. 1446, 2021. https::/aaxxv..oortt hbsgg//bb/// 2112. 11446
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), January 2020. ISSN 1532-4435.
Ingo Rechenberg. Evolutionsstrategie Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Stuttgart: Frommann-Holzbogg 1973. ISBN 3772803733. URL be/ catalag//ugggttooo :011802000022204
Tim Salimans, Jonathan Ho. Xi Chen, Szymon Sidor, and Ilaa Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017. URL httts::/arxiv.orrtt abs/ 1703
03864
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 4463-4473, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 10.18653/v/DD9-.. 1454. /attpss:s//aclaan
org/ 1454/
Lloyd S Shapley et al. A value for n-person games. 1953.
Han Shen, Pin- Yu Chen, Payel Das, and Tianyi Chen. SEAL: Safety-enhanced aligned L.M fine-tuning via bilevel data selection. In The Thirteenth International Conference on Learning Representations, 2025. /ttttttt/ppeeerreeeeee wwtp:/iwww..neeeoooooeee wwtp:e/www...eeooooo////d/iiiiia Vi-Vhuuhvcc
Kashun Shum, Yuzhen Huang, Hongjian Zou, Ding Qi, Yixuan Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts is the data that teaches. arXiv preprint arXiv:2503.00808, 2025.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy. Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15725-15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v 1/222/2224...211on 840. URL BRLLhtpps:// htttps::/aclanthollloo wwrlllooo..ogg//oooooooo 2024/2024.00 acl acl-long. 840/
Daouda Sow, Herbert Woisetschläger, Saikiran Bulusu, Shiqiang Wang. Hans Arno Jacobsen, and Yingbin Liang. Dynamic loss-based sample reweighting for improved large language model pretraining. In The Thirteenth International Conference on Learning Representations, 2025.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), pages 4149-4158 4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18555v11/N19/1// 1421. RRR.itttps::/caaatttoll
119-1421/
14

---

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, llyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models, 2023. htRlshttts:/// /ttp:://axxvvvvoo org/ abs 22// 09288
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 6000-6010 Red Hook, NY. Curran Associates Inc. ISBN 9781510860964.
USA, 2017.
Jiachen T. Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online se- lection of high-quality data for Ilm training in every iteration. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems, volume 37, pages 131197-131223, Curran Associates, wewpiinnngnneeuuiii cc/ haperrppr___a f files/ pttp///pppp////////eeeee Paper- Conference, pdf
Inc., 2024. URL
Jiachen T. Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. Data shapley in one training run. In The Thirteenth International Conference on Learning Representations, 2025. URL https
// //opereevee wwt.:eeetfooueeeeee ft////orumm/// bobormmiid-iiie H8D6
Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. In Proceedings of the 37th International Conference on Machine Learning, ICML'20, JMLR.org. 2020.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave CCNet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors. Proceedings of the Twelfth Language Resources and Evaluation Conference, pages
Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-444 URL https 7/aclanthology. .rg/ 2020. 1rec- 1. 494/
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36. pages 19798-69811 Cur- ran Associates, Inc., 2023. URL htt.:tttt///rrcceediigg htwp://ooooeeeiioooo.ierreeeeeee wttplppss.cpppppppeeeepep
caatarrfffllspppppeaee /
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning.
Advances in neural information processing systems, 31, 2018.
Rowan Zellers, Ari Holtzman, Yonatan Bisk. Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Märquez. editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-111 1472. URL http:t/tt//:clceeea /aapl:/:///cllaholoogyy http://caaccaalllooor//////-//-/ org/
Yifei Zhang, Yusen Jiao, Jiayi Chen, Jieyu Zhang, and Frederic Sala. Just select twice: Leveraging low quality data to improve data selection. In Second NeurlPS Workshop on Auributing Model Behavior at Scale, 2025.
15

---

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: less is more for alignment. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
16

---

A
Appendix
Evaluation 150M LLM
Evaluation on 400M LLM
80%
140%
Relative perf, improvement (t)
60%
120%
40%
100%
DR population cfg. 400M
20%
80%
0%
60%
-20%
40%
20%
2000
4000
6000
8000
2000
4000
6000
8000
Meta checkpoint
Meta checkpaint
Figure 9: Relative performance improvement induced by DataRater models trained on popula- tions consisting of different sizes of inner models, as a function of the meta-training checkpoint. The underlying training dataset is C4/noclean and all evaluations are done with 50% discard fraction. The relative performance improvement is calculated with respect to a 1B baseline trained on the unfiltered C4/noclean dataset. The plots show the average (and 95% CI) of the aggregate performance metric on HellaSwag, CommonsenseeA. PIQA, SIQA and Wikipedia. The DataRater trained with eight 400M inner models obtains the highest and most consistent performance across the two evaluated model scales and four meta-training checkpoints.
Selecting the inner model scale and checkpoint. As explained in the main paper, to meta-train a DataRater model, we approximately solve bilevel optimization problem using stochastic gradient descent. We optimise a DataRater model's meta- peeta-prammters while computing meta-gradients using a population of individually trained inner language models.
We experimented with various size of the population, etc.) and found that meta-evaluation performance depends mostly on the scale of the inner models in the population. In Figure we show the relative performance improvement due to DataRater models trained on populations of inner models with different scales (150M and 400M). We perform this experiment only on the C4/noclean underlying dataset and use a fixed discard fraction of 50%.
choices (including varying the meta-unroll length, the
In this plot (Fig. ( we show relative performance improvement as a function of the meta-training checkpoint index (i.c., number of meta-update steps) and inner model scale. We calculate the relative performance improvement as the average percentage improvement in the raw metric (NLL for the C4/noclent can validation set; accuracy for all the downstream tasks stated in the figure caption) of an LLM trained on the DataRater filtered data vs. a 1B baseline LLM (trained like-for-like on the unfiltered C4/noclean dataset).
We observe that the DataRater trained a population of eight 400M inner models results in the best performance across the board. So, we choose this confi guration (8 x 400M) and the meta-training checkpoint of 8000 steps to produce all results in the main paper body including for experiments which use other underlying training datasets (C4 and the Pile).
Perplexity- based filtering, We compare the DataRater to haplexity-baseed filtering, perplexity is a common heuristic for assessing data quality that does not require choosing a specific validation set. For a fair and balanced comparison, all experiments in this section conducted at the 150M model scale. We use 50M DataRater model that was meta-trained on inner models of 150M seale (as opposed the 400M inner models used for the results presented in the main paper).
We implement a perplexity-basee filtering method similar to the one by Ankner et al. [2025] To mirror the DataRater's online, batch-level filtering mechanism, we first train three separate 150M
17

---

TheFlie
C4moceean
3.2
2.9
3,40
1050 on valibation subsvet
2.8
3.35
2.7
processing
2.9
Baseline
3.30
DstaReter (150M)
PFLX (Mid)
3.25
PPLX (Tbp)
2.0
3.20
10%
10% Filter traction
traction
Filber traction
Figure 10: Comparison of data filtering methods by sweeping over the filtering fraction. The plots show validation loss as function of the proportion of data filtered out for the DataRater (150M), there perplexity-based filtering baselines (PPLX- (Top, Mid, Bot)), and a no-fittering baselines. The PPLX (Mid) variant is the most effective of the perplexity-basee methods, but the DataRater (150M) achieves the best performance across the majority of filtering fractions.
VOK (T)
POCAA (T)
Da
17:6
132
204
306
117
61
1.76
60%
804
30h
Figure 11: Detailed comparison of DataRater (150M) to the best perplexity- -based filter PPLX (Mid). Baseline (no- (ooffilteiing) performance is shown using dotted horizontal grey lines. This
comparison uses the best filtering fractions for each method as identified from the sweep in Figure T0
Performance is evaluated across seven metrics for each of the three datasets. The DataRater (150M)
results in the best performance on 16/21 evaluations, while PPLX (Mid) shows better performance on four.
bascline models on held-out subsets of each of the three training sets used throughout this paper. Each of these bascline models then acts as a scorer, assigning a perplexity score each data point (i.e. packed tokenized text sequence). Filtering is then performed within each oversampled batch by selecting data based on these scores. We use the same oversampling mechanism for filtering as for the DataRater method. We test the three variants of perplexity- -asee filtering introduced in [Ankner] al. 2025]: keeping only the data with the lowest perplexity (PPLX (Bot)), the highest perplexity (PPLX (Top)), and the data from the middle of the perplexity distribution within each batch (PPLX (Mid)).
Figure presents a sweep over different filtering fractions for the perplexity-based methods alongside the DataRater (150M) and the no-filtering baseline on the three datasets. From this experiment we observe that PPLX (Mid) is the most effective of the perplexity-basee filtering variants. The best validation set performance is achieved by filtering data using the DataRater (150M), across the majority of the filtering fractions evaluated. Interestingly, filtering by perplexity can result in improvements when filtering with very high filtering proportions. Based on this experiment, we select the best variant of perplexity-based filter as the PPLX (Mid) for the next comparison together with the best filtering fractions: 90% for the PPLX (Mid) method on all three datasets, and 75%, 50% and 25% for the DataRater (150M) for the Pile, C4/noclean and C4 datasets respectively. In Figure TT we present an extensive comparison of the selected methods on the seven evaluation metrics used in the main paper. This comparison shows that filtering using the DataRater still results
18

---

Model
Nodel
0.05
Modill
Model
0.05
0.04
0.03
Meta gradient update
0.02
0.01
Model
Model
0.05
0.04
0.02
0.02
0.01
0.90
Modell
Model
0.05
0.34
0.02
0.42
0.91
0.90
1000
2000
3000
4000
5000
0000
7000
1000
2000
3000
4000
5000
6000
7000
8000
Meta update index
Figure 12: Meta-gradient update norms through meta-training. The plots show the L2 norm of meta updates originating from each individual inner model over the meta-training step counts of the C4/noclean 150M DataRater. The individual plots highlight when inner model parameters are reset by vertical dotted lines and by plotting individual lifetimes in different colours.
in the best performance across most evaluations (i.c. 16 out of 21 evaluations). Perplexity- filtering (specifically, keeping the "middle" perplexity data from each oversampled batch) shows better performance on four evaluations (SIQA on the Pile, ARC Easy and Commonsense(AA on C4/noclean, and PIQA C4), while best performance SIQA results from not filtering C4.
Meta-training stability. As mentioned in the main paper body, we stabilise meta-gradient com- putation by using population of inner models, passing their meta-gradients through individual Adam optimisers (averaging the resulting meta-gradient updates) and periodically reinitialising inner models. Since our goal is to trained and frozen DataRater to filter datasets for future downstream runs we require the DataRater to generalise across all stages of training rather than to overfit to a particular inner model to particular stage of training. Having population of inner models, whose parameters are reset occasionally, helps maintain diversity over the stage of training and promotes this type of generalisation. Our stabilisation strategies result in stable and bounded meta-updates as shown in Figure [2] the figure also highlights how inner models' lifetimes are stratified. Furthermore, Figure 13 shows the correlation between DataRater scores across meta-training, demonstrating that meta-training starts converging after few thousand steps when scores start tending towards perfect correlation (> 0.95).
Relation to heuristic filters. We analyse the relationship between the DataRater score and common heuristics used for natural language data filtering, We adopted the heuristics used for filtering the C4 dataset, sourced from the Drtaerove library along with other common metries (c.g., non- alphanumeric character fraction, packed sub-sequence count, and word-level type-token ratio). A complete list of these heuristics is provided in T
To perform this analysis, we sampled 25600 packed sequences from the Pile, calculated the DataRater score (using the the Pile 150M DR model) for each, and extracted all correspond-
filters/c4 filters/c4fiiterrs .py
19

---

100
0:73
0.53
9.65
0.51
-1.0
300
0.73
0.90
1.00
0.94
0.91
0.84
0.50
0.7.
0.80
0.71
0.76
0.71
0.65
0.70
0.57
0.69
0.66
400
0.64
0.80
0.94
1.00
0.93
0.88
0.85
0.85
0.05
0.71
0.80
0.7.
0.73
0.71
0.
72
0.72
7.75
71
0.71
500
0.91
0.91
0.57
0.85
0.51
700
0.53
0.70
0.850
0.855
0.91
0.91
1.00
0.97
0.96
0.95
0.93
0.90
0.86
0.56
85
0.83
0.87
8.54
0.85
0.8
800
0.55
3.71
0.778
0.85
0.87
0.93
0.97
1.00
0.98
0.95
0.95
0.93
0.89
0.85
0.88
0.88
0.89
0.87
0.88
900
0.80
0.85
0.96
0.98
1.00
0.97
0.94
0.91
0.89
update
0.96
2K
0.76
0.85
0.93
0.91
0.95
1.00
0.95
0.92
0.90
0.93
0.93
0.91
3k
0.54
0.85
0.71
0.26
0.81
0.89
0.90
0.93
0.94
0.91
0.95
1.00
0.98
0.97
0. 97
0.97
0.98
0.96
0.97
0.6
3.63
0.60
0.73
0.77
0.85
0.80
0.89
0.90
0.87
0.92
0.98
1.90
98
60
0.88
0.91
0.97
0.98
0.99
7k
0.54
3.83
0.82
0.22
0.75
0.85
0.83
0.88
0.89
0.84
0.91
0.97
0.98
0.97
0. 96
1.00
0.98
0.98
0.98
0.54
0.85
0.23
0.78
0.86
0.87
0.89
0.91
0.87
0.93
0.98
0.98
0.99
0. 99
0.98
1.00
0.99
0.99
10k
0.66
0.88
0.91
0.97
0.98
1.00
100
200
300
400
500
600
700
800
900
2k
3k
4k
5k
6k 7k
8k
9k
10k
update index
Correlation of DataRater scores between different stages of meta-training. This heatmap displays the Spearman correlation between scores (of the Pile 150M DR) obtained at various meta update checkpoints throughout training. All correlations are calculated using static set of 25600 examples from the Pile. The plot demonstrates that the DR scores stabilize after a few thousand meta-training steps. While scores from early checkpoints show weaker correlation with later ones (as well as each other), scores from checkpoints after approximately 3000 steps become almost perfectly correlated (correlation 0.95), indicating convergence.
Figure 13:
ing heuristic features. We first computed the Pearson correlation between the DataRater score and all the heuristics which is shown in Figure 14] This analysis shows that the DataRater score is most strongly correlated with features related document size, such as the number of packed sub- sequences, characters, words, and sentences. Conversely, the score is anti-correlated with measures of non-standard content, such as the fractions hon-alphanumeric characters and punctuation. Notably, the DataRater score has only a weak correlation (0.22) with the composite C4 filter, c4 _4__passe_____ all, which aggregates all individual C4 checks.
A simple linear regression model predicting the DataRater score from these features achieves high R² of 0.766. However, as many heuristics are highly collinear (i.e., strongly correlated or anti- correlated), this model is unsuitable for identifying the most predictive features. To isolate minimal set of predictors, we instead fit Lasso (L,) regression model. We applied strong regularisation (a 0.03) and normalised all features (Z-scoring). We trained the Lasso model with 10-fold cross-validation (repeated 5 times), achieving an R² of 0.753. The resulting coefficients, shown in Figure 15 reveal that the Lasso model set most feature weights to zero, identifying only few as significant predictors. The score is most strongly and positively predicted by the number of packed sub-sequences (num_artionn articles, +0.42). The presence of curly braces (c4_ curly. ccurlybbrace,, brace, +0.27)
20

---

0.45
0.2
0.30
.002
3:28
01
3.03
40:00 40.5 -0.03 -130 +032 4.03 4.22
-151
4.35
-3.33
num, artices
0.42
3.00
0.22
3.03
0:11
3,02 4.30 0.
-3.00 1.02 0.1 4.23 6.22
-1.0
-228
AnmL count
2.43
2:34
3.00
6:37
0013
3.22
02
3.03 44.40 40.52 -0.05 -121 -030 4.12 4.22 0030
-3.0
emply Ire Inctoon
0.26
8:47
6.12
3.02
0.62
3.03 6.30 66.01 +0.00 9.30 602 4.05 6.23
6.22
-.11
0.75
poCk denciy
6.16
627
0144
5.44
0:55
0L55
3.51
0.5 5.01 6.32 18.55 10.6 0:01 601 4.01 0.55 01
4.0
6:36
012
-313
022
8:30
0.0
0.32
223
0.04
2.00
0:44
+0.2
2:35
0
5,02
4:33 1.11
-3.03 -133 0.33 4.07 0.21 10.8 1350 4027
0.3 -3.3
Of DESOCL al-0222
5:47
6.0
0.57
3.0
0.05
2.44
3.00
0.15
3.3
0.60
3.03
4.37
6.002
-3.03
-134 6.13 4.05 4.22
0.33
-.50 6.11
0445
-337
Of tha 102 Aoe words 0.21
615
6.18
0.2
0.66
4.20
6:15
1.64
0.00
0.6 0.03 6.75
0.61
3.00 0:01 011 0.36 0.64 0.16 6.21 6:37 000
-.10
SentEncc.cnnt 9.09 9:18
601 0.19
0.01
1.39
0003
1.00
601
5.03
4.16
0.52
3.00 1562 0.09 0.30 0.09 0032 -113 6.55 0.04 -387
0.25
cl.jewsotttt 1.06 :101
611 6.02 0:51 0.65
6:95
0.64
0:1
150
201
6:04
1006 0:00 505 6.02 6.61 0.6 5/12 6:33 0.60
-.51
po Koyward 9.06
1:05
007 0.75
3.55
3.0
605
2,07
4:72
00.00
.0.00 8:00 001 4.00 0.01
0L
-0.35
-003
-384
curly brace 0.04
(130
4.43
02
6:37
-324
800
-3.2
2.00
0.
0.03 8:00 002 4.02 0.06
0.05
0.30 -6.21
0.37
comm Iosum 9.00
66
6:22
1.02
-.00 G6L
4.00 0.01 0.60 9.01 66-11
0.25
uspersaze,
-0.50
svreage jenghh
repettion
-0.75
fun.innemminoloon 0.28 -353
4007
4.35
-323
004 -3.00
0.23
0.0
0.00
0.30
0.38
0.30
0.23
rumeri Inclor 0.0
-336
4.15
6.11
000 3.55
-3000 4.31
4.60 0.33 8.06 601
0.12 0.00
0.32
0.12
-1.00
Suncualion mnctor
262
0.32
3.31
6.51 0002 3.44 001
6.37 (6.0 3.0 6:06
0.34
6616
5.91
6.30
rot mwt.lporumekkkkko fnnckrn 207
1.29
4:11
0.41
0.31
13:00
0.37
1013
-337 1.
0.00
0.27
013
0.75
0:31
0.32
1.41
6.31
2.00
Danstatte Sove
bec Jrrgh cutt
mooy Ins. frad
part deeetyy
semerce cour.
r4.p0rr 11
cLido Jawwdd.
44 Ja 102 word
WIn DVP. J0000 28b
Span 11200300 fraction
ungie conmer
mC 28, berrnn punt.
numer Stntin
pur www.iaaaa Soudien
Figure 14: Pearson correlation between DataRater score and various text filtering heuristics. The score is most strongly correlated with document size metries and anti-correlatdd with non-standard content, such as non-alphanumeric text and punctuation. Notably, the correlation with the composite C4 filter, c4_passes_ all, is low (0.20).
and greater text length (text_ hea__lnnnnhh_htaeeeeeeeee cutoff, +0.19; word_count, +0.13) also yield moderate positive coefficients. The only strong negative predictor is a high fraction of non-alphanumeric characters (-0.33). In summary, the Lasso model indicates the DataRater score is primarily function of document count (i.e., packed articles) and length, penalised by non-alphanumeric content. The Lasso model fit indicates that DataRater score is notably insensitive most other common linguistic heuristics or C4-style filters.
Experimental details. For evaluation, we train causal auto-regressive LLMs from random initialisa- tion, following the Chincilla model architecture, on data filtered by a DataRater model. We use the Adam optimiser [Kingma and Ba, 2015) with a learning rate of 0.001 for LLMSs of size 50M, 150M and 400M and one of 0.0002 for 1B models, with a standard cosine learning rate decay schedule. We train 50M, 150M, 400M and 1B models for 5k, 12k, 30k and 48k update steps respectively, with 128 batch size (except for 1B for which we use a doubled batch size of 256), with a sequence length of 2048. Token budgets per model size can be inferred from the number of update steps, batch size and sequence length. For meta-training, for each inner LLM, we use a batch size of 128 to compute the inner loss, and an outer batch size of 128 to compute the outer loss. We used the Adam optimiser, with a decoupled weight decay of 0.1 [Loshchilov and Hutter 2019], with 100 steps of linear warmup and global norm clipping (Pascanu al.) 2013] of 0.01.
Infrastructure details. We implemented our infrastructure using the jax framework (Bradbury et al. 2018 and its ecosystem of associated libraries [DeepMind et al. 2020]. Our meta- meta-training framework
21

---

Lasso model (a=0.03) fit to DR score: 0.753 R
num, articiis
non_ jon_aaphanumericc.om
20:33
c4_ L_curlybbbrce
text wwnggehnnttoeftt
.0.1M
word_coume
-0.13
c4 M_passes_pa all
jeck_density
0.09
c4 [4_frac_too, wtwooo/eowweeeeeeeeeeeeeoeeeeeee _eod__ewddddsssss words
fraction
0.00
5gram, Deaam_rpeettion..mm
repetition_taattion fraction
0.00
Feature (Z-score)
average average_word, word_length
0.001
sentence_coum count
0.00
popplicy_kawwordd keyword
00.00
c4, ww__sntence_com count
00.00
c4 frarrranooo_ ret_noot_tmnna__unnccc
c4 jajavascript
.0.00
word word_type wttp_tyeeettteeeaaeee token_ratio
c4_ M__as__ong, long_word
00.00
punctuation fraction
uppercase_
.0.00
_in_ciieeeaeoonnn emply_ fraction
0.00
unique_cherr characters
.0.00
64_ Jorem jpsum
.0.00
0.4
0.2
0.0
0.2
0.4
Coefficient
from Z-scored heuristic features. The model uses strong regularisation (a = 0.03) and achieves an R² of 0.753 (vs. 0.766 of OLS) assigning most features zero coefficient. The DR score is most strongly predicted by the number of packed sub-sequences, the presence of curly braces and features related to the document length. The only strong negative predictor the proportion of non-alphanumeric content.
Figure 15: Coefficients of Lasso (L₃) regression model fit to predict the DataRater
is inspired by the Podracer architecture for distributed reinforcement learning Hessel et al. 2021 and it enables parallel computation of meta- gratdgggasents from the inner LLM population. We have run our experiments on Google TPUs: we used 4 x 4 x topology v5 TPUs for meta-training, and up topology v6e TPUs for evaluation. We carefully optimised our infrastructure, using data parallelism, model sharding as well as by using most techniques from MixFlow-Me [Kemaev et al. 2025 to keep RAM (i.e. HBM) consumption within hardware constraints.
framework extensions. We dedicated the core paper to present evaluations the most widely applicable instantiation of the DataRater framework, focusing on accelerating learning progress on given and fixed pre-training dataset. This has clear practical benefits and, as we have shown, can result in significant computational efficiency gains.
Discussion on
In other experiments, we investigated online adaptation during training using dynamic data selection: i.c., we provided additional features (similar and derived from RHO-LOSS [Mindermann et al. 2022 which depend on the inner LLM state, as inputs to the DataRater model here, results showed that the DataRater weights (aggregated at the coarse mixture level) become function of the inner LLM training progress.
We also experimented with different loss weighting granularities: at the level of individual tokens (as the weighting used by (Hu al.] (2023) and at the coarse-grained mixture level (as the weighting used by Xic et al. 20033) For training efficiency meta-objective, still obtained the best performance
22

---

by doing sequence-level filtering (vs. token-level weighting or coarse mixte-m weighting). We also experimented with different downstream metrics (e.g. using NLL on English Wikipedia to meta-leam to identify high quality English data for pre-training) well non-differentiable meta-objectives (i.e., accuracy metrics, where we used Evolutionary Strategies [Rechenberg] 1973 Salimans et al. 2017] to estimate eeta-gradients).
Finally, we also experimented with tailoring the datastream to narrowly targeted preferences by meta-lcarning a DataRater for curating instruction tuning data for the supervised finetuning regime; in this case we used set of human curated conversational data for the meta-objective, and this resulted in conversational data (e.g. forums, Quora, podcast data) being heavily upweighted by the DataRater.
B What can we expect from the DataRater?
To analyse the DataRater simple setting, consider the case when the test dataset Prest consists of clean samples only, while the training set Drain Duss U comrupped contains a mix of clean data and fully corrupted data. Assume that the inner and outer loss functions are the same, that is, l L and are both next token loss.
In this case, if we assume that the corrupted data is completely useless, then want the ratings assigned by the DataRater to clean data to be much larger than the ones assigned to corrupted data (i.e. (x) > (x,(x') for every x E Pess and x' Deomupted) so that the effective final weight for each corrupted sample will be close to zero.
However, if the noise in the corrupted samples is much more benign, that is, the clean data has lower noise level, then we would expect that would still be valuable to use this corrupted data for training. Of course, with more noise, the smaller the DataRater weight should be such scenario is illustrated in Figure 3f of the main paper.
To understand this scenario better, consider the case of an overparametrised model in the interpolation regime, that is, where the loss for all clean data points O, i.e. {(x; 0r) for all Diest
Then the outer loss over the whole dataset (2) (defined in the main paper) can be replaced with an arbitrary weighted combination of losses, with positive weights w(x), where the goal is minimise L. Dres) = Exx-Dex [xx fxx; 0r (Perain)))
Assuming each data point x' E Dirain is possibly noisy version of a test point x E Diea such that the gradient has the form ox((x'; 8) = (x; 0) + c(x) where <(x) is some zero-mean independent noise, then the training data results in unbiased gradient estimate: Vol(x; 0) Ex(x) [Vo((VV)'' ()] with variance Ex(x) [|] Thus, have each sample contribute gradient with uniform variance, one could choose (xx') = 1/ (x)) [[]].
C Limitations
While the DataRater framework demonstrates promising results in enhancing training efficiency through meta-learned dataset curation, several limitations should be acknowledged:
Sensitivity to Meta-Objective. The effectiveness of the DataRater is inherently tied to the choice of the meta-objective (i.e., the outer loss and the held-out test data distribution) which defines the data valuation policy that will be encoded by the DataRater upon meta-training.
The DataRater framework itself explicitly allows for arbitrary inner and outer losses, and, in our empirical evaluation we chose the most widely applicable setting, of curating the initial dataset - i.c., where the meta-objective encodes improvement in training efficiency on the given training dataset.
However, the selection of appropriate meta- veja-obbeetties which align with diverse end-goals remains critical consideration; i.e., the learned data preferences might not generalise fully significantly different downstream tasks or data modalities if the chosen held-out data for the meta-objective does not adequately represent these.
Potential for Bias Amplification. As discussed above, the DataRater learns to value data based on its provided meta-objective. If this objective, or the held-out data used to define it, contains biases,
23

---

the DataRater could inadvertently learn to amplify these biases by upweighting data that aligns with them and bown-weighting data that does not.
For example, if the held-out data underrepresents certain demographics or perspectives, a DataRater model might learn to devalue data pertinent to those groups, leading to models trained on the curated dataset that are less fair or equitable.
While our experiments focus training efficiency with respect to a given data distribution, the framework's s flexibility to use arbitrary outer losses also means it could be misused to optimise for undesirable model capabilities (i.e., if the meta-objective is chosen maliciously). Prior to releasing curated dataset, careful consideration and auditing of the meta-objective (and the resulting data) are crucial to mitigate such negative societal impacts.
Scalability of Meta-Gradient Computation. by techniques like MixFlow-MG [Kemaev et al. [025], still presents computational challenges, especially as model scales continue to grow. Baokapgoo through multiple inner model updates involves second-order derivatives and can, depending on the scale and number of updates involved, be resource-intensive in terms of both computation and memory (HBM).
The meta-gradient computation, while made feasible
We demonstrated upward scale generalisation: from 400M parameter inner models to 1B models for evaluation (which are trained on DataRater-filtere data, not used as inner models themselves) i.c. showing generalisation to LLMs which require an order of magnitude more FLOPS to train. As discussed above, we have also experimented with scaling up meta-gradient computation for larger models (i.e., using inner models with 27B parameters) for the supervised fine-tuning regime, where inner model updates are made in low-rank projection of parameter space using LORA (Hu et al. [022]. This setting was non-trivial to tune for staying within hardware limitations on HBM consumption, but feasible. However, the scalability of meta-training DataRater models for extremely large foundation models with fully dense inner updates remains an open question, and may require further algorithmic advancements in scalable bilevel optimisation.
24

---

Table Description of filtering heuristics use for correlation analysis.
Heuristic
Description
packing.density
Ratio of valid tokens to total tokens.
cum_articles
Number of packed sub-sequences.
text_length_cutoff
Number of characters.
word_count
Total word count chiitespace-delimiiee
sentence_count
Count terminal punctuation (regex [.17]).
empty_line_fraction
Ratio of empty lines (regex \n (n) to total newline
characters.
unique_characten
Ratio of unique characters to total text length. Word-level type-token ratio, i.c. number of unique words to number of words.
word_typeetokee_ratia
oo..allpanummriic_rrcttioo
Fraction of non-alphanumeric characters (regex AW). Fraction of uppercase letters (A-Z).
uppercase_letters fractiion
punctuation_fraction average_ woodage_word__ennth
Fraction of punctuation characters (regex [*\w\s]).
Mean word length (total non-w won-whiesppace chars
word_coure couunt.
cumerrc_fractiine
Fraction of numeric digits (0-9).
geraa_repetition..o fraction
Fraction of duplicate (non-unique) word-level 5-grams.
c4_lorem_ipeum
True text contains "lorem ipsum".
cd_curly_brace
True if the text contains a curly brace (().
c4_ ce_cootaains_javascrppp c4_has_long.a
True if any line contains the keyword "javascript". True if any space-delimited word exceeds length of 1000.
c4_ 1e4.raacttot.Ilnsss..aaloooooaan
Fraction of lines not ending with standard terminal punctuation (??!"?).
c4_ co_ftrattioo__Lnes_eail__.. worddooooaa worda
Fraction of lines containing fewer than 3 words. True if the text contains policy keywords (e.g., "privacy policy", "terms of use").
c4_ col_oonaain_pollccy.eeee
cd_sentence_count
Sentence count derived from NLTK's Punkt sentence tokenizer, assuming English text.
passes all_c4_filterrr
Composite filter: applies document-level (lorem ipsum, () and line-level (length, punctuation, kcywords) filterin, then checks for a minimum of five sentences.
NLL
Ascuracy % (7)
Wikipedia Baseline
HellaSwag Baseline DR
Easy
Commonsense@U
DR
Boseline
DR
Roseline
DR
Baseline
DR
Raseline
DR
Boseline DR
Train Dataset
Model
The Palw
50M
3.66
3.30
3.17
2.94
26.01
26.47
39.41
40,07
55.22
59.36
28.25
$1.58
22.19
23.51
150M
61.15
26.04
200M
2.69
2.52
2.36
2.27
31.59
34.71
42.43
43.91
62.62
65.02
40.35
40.53
29.65
28.17
44.08
09.70
30.71
04/socleam
50M
4.17
3.94
2.93
2.69
26.05
25.31
39.57
34,46
$4,90
$1.34
29.07
31.23
22:85
22.85
2.44
40.02
61.37
23.10
400M
3.35
3.24
2.04
1.91
29.52
31.28
40.74
42.12
61.32
63.44
32.28
34.74
24.37
26.62
1.64
43.65
C4
S0M
3.91
3.91
3.65
3.62
26.76
26.93
41.50
40.69
59.19
60.77
32.11
32.28
25.96
25.06
150M
3:45
3.20
43.91
43.24
63.28
64.36
37.02
27:35
27.52
600M
3.22
2.92
2.91
37.95
44.93
43.91
68.66
68. 59
40.18
41.58
29.16
29.98
2.95
2.97
2.64
44.17
45.65
31.20
32.02
Table 2: Performance metrics (as NLL. or accuracy) at convergence across three train datasets, four model sizes and seven evaluation tasks (two negative log likelihood tasks, and five downstream accuracy tasks). "DR" stands for DataRater.
25

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract claims DataRater estimates data point value, is trained with meta-gradients for improving performance on held out data, and shows effectiveness in experiments. The introduction reiterates these points, emphasizing automated, fine-grained curation and improved learning efficiency. These are all consistent with the contributions listed (e.g., DataRater framework, scalable meta objective, compute efficiency gains) and supported by the extensive empirical evaluations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses the conditions where the DataRater is particularly effective (low-quality datasets) and mentions the compute cost of online filtering using the DataRater and the cost of meta-training.
Justification:
The appendix includes dedicated "Limitations" section, where explicit discussion of more limitations is given (e.g., on sensitivity to the meta-objective choice, scalability of meta-gradient computation in general, and potential negative societal impacts if data is filtered in a biased way).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
26

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper primarily presents an empirical method and experimental results. While does provide formal problem definition and derives the DataRater approach, the paper does not include new theoretical results with formal proofs in the traditional sense (e.g., convergence guarantees).
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides details on datasets used (with specific references), model architectures (Chinchilla-based transformers), training protocols (Chinchilla training proto- cols and token budgets), DataRater configuration (50M parameter transformer, population of eight 400M inner models), and evaluation metrics. The appendix includes further de- tails about the selection of inner model scale, meta- veta-training checkpoint and details on the syper-parameten used.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
27

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: No code or models are publicly released. The datasets used are existing publicly available datasets, and the models are trained following publicly available research papers (as referenced and detailed in the paper and appendix).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips cripss.cc/ubbic//uuii wttp://giiiieeseeeeeeeeee http:///alsttttddeeeoo////oaaaaa wwtp://ooeeeiiiiiiiiei.oo
ssoiiiliccyiiiiiiiccocoooooiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper describes all training and test details in the main paper body, in subsections the evaluation protocol, experimental setup and evaluation metrics. The ap- pendix includes further details on additional syyee--praamteen (such as optimisers, learning rates, token budgets, data splits, etc.).
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Figure 9 in the Appendix A.1 shows average performance with 95% confidence intervals for the meta-training checkpoint selection experiments. The main experimental results in the main paper body mostly present point estimates with each plotted point being the result of a full L.LM training run, or its learning trajectory. Note that the consistency of results across multiple datasets, model scales, and metrics provides degree of robustness. It is also well known that the pre- pre-training performance of LLMs can be captured by simple scaling laws, which relate the training flops to final performance so, in most cases, there is little need to run multiple seeds of the same (expensive) experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The appendix includes specific details on the type of compute workers (TPUs, topologies, training setup), memory requirements (i.c., HBM) and execution time (in TPU hours) that have been used for the experiments in the paper. The paper body also includes details the model scales used for the DataRater, inner models and evaluation models, and includes reference to the model architectures and training protocols adopted from the Chinchilla paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
29

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuripps wwtpppblic/tchicsuuiieeeeeeeeeee
Answer: [Yes]
Justification:
The research focuses improving the efficiency of training foundation models by curating datasets. The datasets used are standard, non-deprecated, publicly available corpora; datasets are introduced. The research did not involve human subjects or participants.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yss]
The paper focuses on specific technical contributions in automating the data curation process. The experiments presented in the main paper body focus on improving the training efficiency of LLMs. However, the DataRater framework allows specifying an arbitrary outer loss, which sets the goal of the resulting data curation process. Of course, this empowers practitioners to improve objectively desirable capabilities LLMs, but intentional misuse can result in improving harmful capabilities of LLMs. This case is discussed more detail in the appendix.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
30

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not introduce new datasets nor does it release models which could inherently pose a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper properly cites the sources for the datasets used: C4/en, C4/en noclenn and the Pile, and various downstream task datasets like HellaSwag, SIQA, PIQA, ARC Easy, and Commonsense QA. All models used in the paper are trained from random initialisation and the specific training details used cited (i.e., Chinchilla paper).
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpprswipteatteee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
31

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
The research described in the paper does not involve crowdsourcing or research with human subjects. The term "human intuition" mentioned in the main body of the paper refers to qualitative analysis of data quality.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects, so IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The research this paper is about training foundation models (which are LLMs in this context) more efficiently by curating their training data. The LLMs the
32

---

subject of the study but are not a tool used for developing the methodology itself in non-standard way (e.g., an LLM was not used to design the DataRater framework nor algorithm).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, ntn-standard components.
Please refer to our LLM policy for what should or should not be described.
33


Reviews:
[{"review_id": "eYHIZJCt10", "reviewer": "Reviewer_i2Zt", "summary": "The paper deals with finding most influencial training data for foundational models in order to improve training efficiency, but also possibly predictive performance. The goal of a filtering process is to select a subset of training data that leads to the smallest test error. The DataRater approach is to solve a continuous linear relaxation for discrete subsampling of data, where for each gradient update and minibatch a continuous weight in [0,1] is chosen. The weights are represented by a learned function, the DataRater model, where the outputs are normalized by a softmax preference function Parameter opt. is done via meta-Learning of DataRater model parameteres by backpropagrating through multiple inner loops. The authors build their implementation on prior work MixFlow-MG and a population of inner models for meta-gradient calculation, in order make optimization feasible and stable. The data curation is then performed using top-K filtering, i.e., removing data with low predicted weight.\n\nThe system is evaluated by learning DataRater, then training a LM of sizes up to 1B from random init. on the resulting data. Datasets used are C4 variants and Pile. The Inner LLMs and DataRater model are based on Chinchilla architectures. The final accuracy is measured on HellaSwag, SIQA, PIQA, ARC Easy, Commonsense QA downstream tasks. The results show that DataRater accelerates learning (i.e., FLOP savings) esp. for lower quality datasets. The authors also evaluate the best data drop rates, which correlates with the amount of available noise.", "questions": "- The pipeline first trained a DataRater model using multiple inner LLMs, then trains the final LLM using the curated data. Could one also train such a final LLM directly, similar to the inner LLMs? I understand that the DataRater model should be reused multiple times to actually save compute, but could end-to-end optimization also fine the optimal amount of needed data?\n- Would Multi-criteria optimization be possible to also reduce dataset size directly?\n- What are the advantages over competing approach mentioned in related works - e.g., influence functions, SHAP values, JST, ... ? More efficient? Better quality assessments?\n- T=2 inner models updates have been found to be best. Did the performance remain stable with more updates and steadily increase? Can you please give possible explanations for the good fit of T=2?\n- How complex are the chosen datasets? Do the findings generalize to any complexity of text-based downstream task?\n- Is the optimization process sufficiently stable for updating a trained DataRater wrt new data?", "limitations": "Limitations become clearer when answering the questions in the above section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Pros\n- Clear presentation of methods and results\n- Sensible and original meta-learning approach to learn a data value function\n- Good empirical results for the method, showing efficiency and predictive performance improvements compared to training with all data\n\n## Cons\n- No empirical comparison to competitors and not sufficient conceptual argumentation why new approach is adding value wrt existing work.\n- Missing argumentation for the choice of datasets wrt (task-) complexity", "was_revised": true, "final_justification": "I thank the authors for their answers to my questions. After reading the answers and the comments of the other reviewers, my assessment to accept the paper clearly remains. The proposed method to estimate the value of training data points is clearly described, sensible and provides reliable results. The proposition to add further baselines is of course welcomed.", "rebuttal": "We thank the reviewer for their comprehensive summary of our work, and are grateful they found our paper's methods and results clear, our approach sensible, and our empirical results good. \n\nWe provide individual replies to the reviewer's questions below:\n\n> The pipeline first trained a DataRater model using multiple inner LLMs, then trains the final LLM using the curated data. Could one also train such a final LLM directly, similar to the inner LLMs? I understand that the DataRater model should be reused multiple times to actually save compute, but could end-to-end optimization also fine the optimal amount of needed data?\n\nYes, one could train a single DataRater model side-by-side with a single LLM whose data could be filtered online by the DR. This type of DR model could provide targeted data relevant for the current state of the LLM -- e.g., if performance on maths appears to lag, this could be rectified immediately with relevant maths data. We conducted a few experiments in this direction (see Appendix L951), but we leave a full treatment of this line of research to future work. \n\nIn our case, indeed, as the reviewer points out, we save compute cost by using a trained DR many times. We can amortize compute costs due to the observed scale generalization: as our results show, a DR trained on 400M inner models generalizes to LLMs with parameters of 50M to 1B parameters.\n\n\n> Would Multi-criteria optimization be possible to also reduce dataset size directly?\n\nTo target reducing the dataset size directly would indeed be possible; this would require introducing a different continuous relaxation of the main problem formulation (Eq 3), one which would operate at the dataset level instead of at the mini-batch level.\n\nAnother way to use multi-criteria optimization is in the setting where one has a specific set of capabilities to target and corresponding dataset proxies. Then, the validation dataset on which the outer loss is calculated could include multiple such proxy datasets, for example, the validation dataset could include Wikipedia (as a proxy for high quality text), GitHub data (as a proxy for coding capability) or podcasts (as a proxy for conversational capability), etc. This would optimize a DR model for multiple criteria at the same time. The outer loss could be modified to explicitly make use of this (e.g. robustness against worst-case performance across the objectives).\n\n\n> What are the advantages over competing approaches mentioned in related works - e.g., influence functions, SHAP values, JST, ... ? More efficient? Better quality assessments?\n\nInfluence functions (e.g. the TracIn algorithm from [1]) and SHAP values (e.g. as applied to deep learning in [2]) both use tabular representations (i.e. one score per datapoint). They also have myopic objectives, more closely related to a variant of our method with a single inner-loop unroll. In contrast, the DR is a scoring model that can generalise to new datapoints, and across model scales. Our meta-optimisation also accounts for multiple inner model steps, which may better characterise the value of the data.\n\nJST [3] performs a two stage filtering process, which relies on a held out set of low-quality data for its predictions. This is in contrast to our work, where the validation set can be sampled from the same distribution as the training set, as evidenced by all our results.\n\nOur paper contains a short comparison to these and related approaches (in Sec 4.1), which is due in part to the page limit constraints on the initial submission. In the final paper version we will expand this section to better contextualize our method.\n\n[1] Pruthi et al., Estimating Training Data Influence by Tracing Gradient Descent\n\n[2] Wang et al., Data Shapley in One Training Run\n\n[3] Zhang et al., Just Select Twice: Leveraging Low Quality Data to Improve Data Selection\n\n\n> T=2 inner models updates have been found to be best. Did the performance remain stable with more updates and steadily increase? Can you please give possible explanations for the good fit of T=2?\n\nPerformance remains relatively stable when performing more inner updates, but static device memory (static HBM) consumption increases significantly, since memory linearly depends on the number of inner updates, even with MixFlow-MG, as it mainly addresses dynamic HBM. One could use more advanced techniques from MixFlow-MG (such as host offloading) to handle static memory however our experiments showed that T=2 or 4 achieved the best trade-off between the performance wins and cost of experiments. The good performance of shorter unrolls provides some evidence that the influence of a given training example on the overall optimisation trajectory is relatively localised.\n\n\n> How complex are the chosen datasets? Do the findings generalize to any complexity of text-based downstream task?\n\n\nThe datasets we used (C4 and The Pile) are standard in the field and were chosen specifically for their varying degrees and types of complexity, which provides a robust testbed for DataRater.\nThe Pile has a lot of compositional complexity, as it is composed of many distinct smaller corpora. These range from highly structured/technical sources (from arXiv) to more informal and conversational text (from YouTube).This diversity presents the kind of complex, mixed-quality scenario where an effective data curation is most valuable.\nC4 is derived from a broad crawl of the internet and contains the full spectrum of complexity of human knowledge as reflected in the internet. A key feature of C4 is the presence of significant noise and quality variation. Our results show that the DataRater is particularly effective on datasets with this type of complexity and variation.\nThe downstream tasks (HellaSwag, SIQA, PIQA, ARC Easy, Commonsense QA) were chosen because they measure more than simple pattern recognition. These are challenging benchmarks that require sophisticated reasoning, and indicate whether a model has developed a nuanced understanding from its training data. We believe our findings generalise well to other text-based downstream tasks.\n\n\n> Is the optimization process sufficiently stable for updating a trained DataRater wrt new data?\n\n\nYes. The DataRater utilizes a standard non-causal Transformer architecture. While optimization via meta-gradients involves a complex bilevel optimization landscape, we address the stability of this process explicitly. As described in Section 2, we employ several key techniques to stabilize meta-gradient computation, including: using a population of inner models (line 129); passing the meta-gradient for each inner model through an individual Adam optimizer (line 130) and averaging the resulting updates; and periodically reinitializing the inner models to stratify learning progress (lines 131-132).\n\nThese stabilization mechanisms ensure robust optimization during initial training and are equally applicable when updating a trained DataRater on new data. Furthermore, updating the model (fine-tuning) begins from an already converged initialization, a process that is typically more stable than training from scratch. To further highlight this stability, we will include a plot of the meta-gradient update norms throughout training in the final submission.\n\n\nWe hope we have addressed your concerns. Please let us know if you have any other questions."}, {"review_id": "txJdEU5PZc", "reviewer": "Reviewer_JMg5", "summary": "This paper introduces DataRater, a meta-learning framework for automated dataset curation that learns to estimate the training value of individual data points for foundation models. The approach addresses the current reliance on manual heuristics and coarse-grained data mixing in large-scale model training.\n\nDataRater uses a transformer-based scorer that assigns scores to data points, trained via bilevel optimization with meta-gradients. The meta-learning objective is to improve training efficiency on held-out validation data. The method employs a continuous relaxation of discrete data selection, using softmax-weighted gradients during training and top-K filtering during actual curation.\n\nThe authors evaluate DataRater across three datasets of varying quality (C4, C4/noclean, The Pile) and four model scales (50M to 1B parameters). A key finding is cross-scale generalization: a DataRater trained with 400M parameter inner models effectively transfers to target models ranging from 50M to 1B parameters, with consistent optimal filtering proportions across scales.\n\nThe method achieves substantial compute savings on lower-quality datasets, with up to 46.6% net compute reduction on The Pile while matching baseline performance. Results show DataRater learns to down-weight data that aligns with human intuitions of poor quality, such as OCR errors, encoding issues, and non-English content. The approach demonstrates improvements across 73 out of 84 experimental configurations spanning multiple datasets, model scales, and evaluation metrics.\n\nThe technical implementation leverages MixFlow-MG for scalable bilevel optimization, making the approach computationally feasible for realistic model and dataset sizes. The framework supports both batch-level filtering for efficiency and individual data point scoring for parallel processing pipelines.", "questions": "**1. Baseline Comparisons** How does DataRater compare against established curation methods beyond unfiltered data? Please provide comparisons against: (a) heuristic filtering similar to C4's pipeline applied to The Pile, (b) perplexity-based filtering, and (c) random subsampling at equivalent discard rates. **Criterion**: Adding proper baselines would increase Quality if DataRater shows meaningful advantages over ablated heuristics.\n\n**2. Cost-Benefit Analysis and Decision Framework** When is DataRater worth the ~58.4% training cost versus cheaper alternatives? Please provide: (a) clear decision criteria for when to use DataRater vs. heuristics, (b) break-even analysis for different dataset quality levels, and (c) guidelines for practitioners. **Criterion**: Clear practical guidance would increase Significance.\n\n**3. Systematic Analysis of Learned Patterns** Can you extract interpretable rules from DataRater beyond qualitative examples? Please provide: (a) quantitative correlation analysis between scores and measurable text features, (b) comparison of learned patterns vs. existing heuristics, and (c) feature importance analysis that could inform simpler filtering methods.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- **Quality**: The technical approach is sound with proper bilevel optimization formulation and scalable implementation using MixFlow-MG. The cross-scale generalization finding (DataRater trained on 400M models works for 50M-1B targets) is empirically robust and theoretically interesting. Experiments span multiple datasets, model scales, and metrics, providing comprehensive evaluation.\n- **Clarity**: The paper is well-written with clear mathematical formulation and algorithmic specification. The meta-learning framework is explained intuitively, and experimental setup is thoroughly described. Figures effectively communicate key results, particularly the compute efficiency gains.\n- **Significance**: Addresses an important scalability challenge in foundation model training. Achieves substantial compute savings (up to 46.6%) on noisy datasets while maintaining performance. The cross-scale generalization property has important practical implications for amortizing curation costs across model sizes.\n\n### Weaknesses\n- **Quality**: gap in baseline comparisons—only compares against unfiltered data, not against established heuristic filtering, perplexity-based methods, or random subsampling. This makes it impossible to assess whether neural approaches justify their computational overhead versus simpler alternatives.\n- **Clarity**: Limited analysis of what DataRater actually learned beyond qualitative examples. No systematic feature analysis or rule extraction that could provide transferable insights to the community.\n**Significance**: Minimal gains on well-curated datasets (C4) where most practitioners operate, limiting immediate practical value. Training cost (~58.4% of target model) isn't justified by small improvements on quality data.\n- **Originality**: While the application is novel, the core bilevel optimization techniques are established. The continuous relaxation approach follows standard practices in differentiable discrete optimization.", "was_revised": true, "final_justification": "I keep a borderline-accept because the paper’s main strengths—a meta-learned data filter that delivers compute savings and robust cross-scale transfer—are technically solid and practically promising. The rebuttal usefully adds quantitative correlation evidence and pledges a fuller analysis of what the DataRater learns, partly addressing my interpretability concern, and provides qualitative guidance on amortising its 58 % meta-training cost. However, it still lacks empirical comparisons with heuristic, perplexity-based, or random filtering, so the key baseline gap and cost-benefit uncertainty remain. On balance, the contributions merit publication, but I urge the authors to include the missing baselines and clearer decision guidelines in the final version.", "rebuttal": "We thank the reviewer for their extensive summary of our work; we are grateful they found our technical approach sound and scalable, our writing clear, and our problem statement practical.\n\nPlease find our replies below.\n\n***Baseline Comparisons***\n\nOur experiments, in particular on C4 and C4/en.noclean, give some context for (a) as to the comparison with strong heuristic filtering methods. Using the DR on the no-clean variant does not recover the performance of C4 using the fully tuned heuristic filtering pipeline. However, we can still improve over C4 on most metrics when using that as the starting point, albeit with diminishing returns.\n\nWe think the DR approach is exciting because it requires so little prior knowledge, deriving a filtering strategy merely from the pre-training data itself. We consider this finding to be of significant scientific interest. It may be of significant value to practitioners as well. Clean data sources, especially for niche domains, are quickly becoming exhausted. Teams working on foundational models are constantly searching for new sources of data, which have increasingly poorer base quality and poorly understood patterns that may be used for heuristic filtering.\n\nRegarding random subsampling (c), we highlight that our method often improves the final performance (Figures 5 and 6) compared to training on the full, unfiltered dataset. Random subsampling reduces compute but typically does not improve final performance over the full dataset. The substantial gains we observe strongly indicate that the DR is performing non-trivial, quality-based selection superior to random chance, selectively removing data found to hinder training (e.g., OCR errors, encoding issues).\n\nConcerning perplexity-based filtering (b), these methods typically require a high-quality reference corpus or a well-trained reference model to establish scores. A key advantage of the DR is its self-supervised nature, learning value directly from the pre-training data without external references/validation sets. Furthermore, perplexity can be a coarse proxy for quality—potentially discarding complex yet valuable data—whereas DR optimizes directly for learning efficiency via the meta-objective.\n\n***Decision Framework***\n\nIt is difficult to provide a comprehensive decision framework as the break-even point depends significantly on the intended downstream use of the DR. If one does multiple training runs using the filtered data (e.g. for modeling ablations), or scales up a downstream training run as we show is possible, the cost of training the DR is rapidly amortized.\nFor a dataset where existing heuristics have not been extensively tuned, and a practitioner expects to use an order of magnitude more compute in downstream use (for repeated future training runs or scaling up), we can recommend trying the DR strategy for filtering.\n\n***Learned Patterns***\n\nWe conducted a preliminary analysis of the DR, and found low (Pearson) correlation (< 0.15) with standard heuristics from the C4 pipeline like maximum word length, terminal punctuation checks, word repetition counts, curly brace checks. We found the DR tended to pick up on unusual text (poorly encoded, OCR errors, non-standard characters) and highly-patterned uninformative text (long lists, tables, indices, log outputs). Overall our analysis indicates that the DataRater is not a simple heuristic (since there’s not a single strong correlation to any of the heuristics we tested) and that most correlations are negative, i.e., the DataRater learns to penalize markers of low-quality text. We will include the full analysis in the final paper version.\n\nIt would certainly be possible to construct heuristic filters to capture some of these features. It is also possible to distill the DR into a smaller and simpler model.\n\nWe hope our answers have suitably addressed your concerns. Please let us know if you have any other questions."}, {"review_id": "K3EmMSBPvg", "reviewer": "Reviewer_pnjF", "summary": "The paper \"DataRater: Meta-Learned Dataset Curation\" proposes a method for estimating the value of individual data points for training foundation models, using meta-learning with meta-gradients. The authors frame their contribution as making LLM training more data-efficient, especially since we are running out of human-curated text to train on.", "questions": "* How does DataRater compare against recently learned data selection methods?\n\n* Since the DataRater is trained to optimize performance on a held-out subset of the same dataset, how do you ensure that it isn’t overfitting to the meta-validation metric or gaming spurious correlations in that specific split? Could you explore robustness to alternative outer losses/distribution shift between D_train and D_test/generalization to truly different downstream tasks?\n\n* On a similar note, how sensitive is DataRater to the choice of outer loss, or shifts in domain/task?\n\nNitpicking: Please make sure that your figures are in the right place and that you refer to them in the text itself (e.g., the first time you refer to figure 1, which is on page 2, is on page 6; figures 2-4 were never referred to).", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n* Topic: The problem of curating training data for foundation models, especially with increasing reliance on synthetic and noisy data, is critical and underexplored at scale.\n\n* Empirical results: Consistent performance and compute savings across model scales (50M to 1B) and diverse datasets (The Pile, C4, C4/noclean). Also, the authors show that a DataRater trained on 400M inner models generalizes to smaller and larger models. This is practically useful and rarely demonstrated in other works. Cute qualitative analysis. \n\n* Theoretical grounding: Meta-learning data value, bilevel optimization, and filtering via data scoring are all well-explored. The novelty here is in their combination. This is a rather incremental novelty, but still a novelty.\n\n\n**Weaknesses:**\n\n* Baseline: There’s no head-to-head comparison with existing learned filtering approaches like SEAL, JST, or PreSelect (only to the unfiltered baseline). There's also no vanilla baseline comparison, as I'd expect in figure 4. \n\n* Downstream utility: The evaluation focuses on proxy tasks (HellaSwag, PIQA), not on a large downstream deployment (e.g., instruction tuning, chat models). Claims about general usability or production utility would benefit from such an example.\n\n* Compute cost: Training the DataRater takes ~58% the compute of training a 1B LLM. The argument about amortizing this cost is valid, but practically speculative without end-to-end downstream deployment scenarios.\n\n* High-quality data: The method is much more effective on noisy data (Pile, C4/noclean); on already filtered datasets like C4, the benefits are small and sometimes ambiguous.\n\n* Technical details: While referencing MixFlow-MG and other tricks for scalability, the paper provides limited detail on hyperparameters, stability, or how these techniques differ from baseline approaches in practical terms. For reproducibility or for researchers hoping to build on this, more clarity would be welcome.", "was_revised": true, "final_justification": "I am keeping my score as it captures the scope (that excludes downstream tasks) and contributions since the authors addressed the baseline issue, I am not reducing my score.", "rebuttal": "We thank the reviewer for their comments; we are grateful they found our choice of problem domain important, results strong, and approach novel.\n\nPlease find our replies below.\n\n***Learned data selection methods***\n\nSEAL [1] targets the fine-tuning regime (i.e. adapting pre-trained LLMs to excel at specific downstream tasks), while our method targets the pre-training regime (i.e. training foundational models from scratch). SEAL builds a data selector for enhancing safety / lowering toxicity; our method learns to generally curate data. SEAL requires as input a safety-aligned LLM as well as a golden validation set of safe data points; our method does not require either. SEAL uses a bilevel optimization formulation as well, and approaches it using a penalty-based reformulation, instead of computing exact meta-gradients.\n\nJST [2] performs a two stage filtering process, which relies on a held out set of low-quality data for its predictions. This is in contrast to our work, where the validation data is sampled from the same distribution as the training set, as evidenced by all our results.\n\nPreSelect [3] identifies data points which correlate well with improvements in downstream metrics and shows that filtering those that do not result in performance improvements. Their method makes considerably stronger assumptions: it requires a set of downstream evaluation metrics (12 used, §2.3), as well as a held out validation set of high quality data (they select random samples from the top 3000 most frequent domains, like wikipedia.org, §2.3). Our method requires no curated validation set nor downstream metrics.\n\nTo summarise, the recent learned data selection methods differ considerably from the DR in multiple significant dimensions, precluding meaningful direct comparisons. I.e., SEAL and the DR differ in the problem setting, goal and assumptions. The DR does not require validation data, while both JST and PreSelect do, while PreSelect also depends on having a set of downstream metrics as input.\n\nThe points in Figure 4 with an x-axis value of 0% filter fraction correspond to no filtering, i.e. the raw dataset, which could be considered a default baseline.\n\n[1] Shen et al., SEAL: Safety-Enhanced Aligned LLM Fine-Tuning via Bilevel Data Selection\n\n[2] Zhang et al., Just Select Twice: Leveraging Low Quality Data to Improve Data Selection\n\n[3] Shum, Huang et al., Predictive Data Selection: The Data that predicts is the Data that teaches\n\n***Overfitting to meta-validation metrics***\n\nWe use strictly held-out validation sets as well as downstream evaluations in order to guard against meta-overfitting. In fact, we find in our setting this is not an issue. Since we use a split of the original (large) dataset, it is quite easy to have a diverse enough meta-validation set to prevent overfitting.\n\n***Different downstream tasks***\n\nWe have also tried using targeted meta-validation sets with a significant distribution shift from the original dataset. This also works quite effectively, although meta-overfitting is a greater risk. For this paper, we decided to focus on the matching-distribution setting, as we were excited by the generality of the approach which does not require any curation of a golden targeted/high-quality meta-validation dataset.\n\n***Sensitivity to outer loss / domain / task***\n\nOverall we found the approach quite robust to the outer loss and choice of dataset. For example, preliminary experiments with code-focused datasets in the inner and/or outer loss showed similar qualitative results to those presented.\n\n***Figures*** \n\nThank you for pointing this out. All figures are referenced but not in order. For the final paper version we will correct the Latex referencing so all figures are numbered in the order they appear in the PDF.\n\n***Downstream utility***\n\nWe use standard proxy tasks (HellaSwag, PIQA, etc.) as they are established benchmarks for assessing the quality of pre-trained foundational models at these scales. While evaluation on end-to-end instruction tuning or chat models is an interesting avenue for future work, the consistent improvements we observe across these diverse benchmarks (73 out of 84 configurations) strongly suggest the general utility of the DR.\n\n***Compute cost***\n\nWe acknowledge the upfront cost of training the DataRater which is ~58% of the FLOPS for a 1B LLM. This cost is rapidly amortized because the filtered LM datasets are typically reused. Given the demonstrated cross-scale generalization, the DR can be used for subsequent training of much larger models or multiple ablation studies, leading to significant net compute savings.\n\n***Technical details***\n\nWe provide more experimental details in the appendix, including on hyper-parameters (learning rate, model sizes,  optimizer, batch sizes, etc.), and infrastructure (Jax codebase, types of accelerators used for training, libraries, etc.). However, to improve clarity, for the final version we will expand this section to better detail how we make use of techniques from MixFlow-MG in our work.\nOn the topic of meta-training stability, we employ several key techniques to stabilize meta-gradient computation, including: using a population of inner models (line 129); passing the meta-gradient for each inner model through an individual Adam optimizer (line 130) and averaging the resulting updates; and periodically reinitializing the inner models to stratify learning progress (lines 131-132).\n\nWe hope our answers have suitably addressed your concerns. Please let us know if you have any other questions."}, {"review_id": "LyQe5EOvgx", "reviewer": "Reviewer_VwDw", "summary": "The paper introduces a novel meta-learning approach called DataRater for curating data for training LLMs. The DataRater framework trains a data-selector model in a meta-learning fashion by optimizing through unrolled inner optimizations of an actual LM being trained. The paper conducts experiments across different model scales on three publicly available datasets. Empirical evidence shows that the trained DataRater models can select high-quality training data for LMs, thereby improving compute efficiency of training. Further analysis on the type of training data samples selected by the DataRater are also provided to guide intuitions on how DataRater works.", "questions": "- Doesn’t DataRater require manual tuning of the threshold as well? It seems a bit disingenuous to claim that your method does not need manual tuning (in line )? It requires the same amount of tuning as a fast-text classifier pretty much?\n- Does DataRater also work for >1-epoch training or is it limited to the single-repeat setting? From the description, it is unclear if the experiments are conducted for a single or multi-epoch setting (especially given the different discard fractions used).\n- Why do the inner model parameters need to be reset? Does this not making the DataRater somewhat off-policy, compared to the learning signal gotten from the updated inner models? A more detailed explanation of this would be good to add in the paper.\n- One thing that is not clear to me from this statement in lines 168-172: Is this using offline DataRater or an online version that filters out every batch? Do you assume amortization of running the DataRater model on the full dataset once, or do you still count that towards the total FLOPs needed for training? The claims in the paper around compute efficiency sometimes use number of training steps, and sometimes use FLOPs. If indeed the cost of running inference of DataRater is not accounted for, making claims about compute efficiency purely using number of training steps is not fair in my opinion.", "limitations": "Yes, the paper provides a good list of limitations, both technical and societal, in the appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written and provides a great overview of the method, both from an intuition perspective and a technical implementation perspective.\n- The problem outlined by the paper---improving training efficiency via data curation---is an extremely important problem and the paper does a good job of characterizing it.\n- The intuitions provided by the analysis in Section 3 (Figures 7 and 8) are very helpful in understanding what data samples are selected by DataRater. More papers in the data curation literature should adopt this.\n\nWeaknesses:\n- From figure 1 and as discussed in Section 3, it seems like DataRater brings large boosts for relatively under-filtered datasets like Pile and C4/no-clean, whereas for a reasonably well filtered dataset like C4, the gains seem to be a little bit more modest. This raises the question if DataRater is complementary to other LM data curation / filtering methods or not? The benefits of the SoTA filtering pipelines used in current LMs is that they are, for the most part, complementary i.e. you can use multiple fast-text classifiers together, and performance improves (see [Nemotron-CC](https://arxiv.org/abs/2412.02595) paper). Hence, it is an important question to answer if the DataRater method is complementary or orthogonal to more such filters i.e. can DataRater be used as one filter in a stack of many more filters?\n- DataRater assumes access to a test distribution. In the set of experiments conducted, that is just from the same distribution as the training distribution. What happens when your test distribution is quite different from your training distribution? For example, could the authors discuss / ground their findings with respect to the experiments conducted in the [Color-filter](https://proceedings.neurips.cc/paper_files/paper/2024/file/b0f25f0a63cc544d506e4c1374a3c807-Paper-Conference.pdf) did, where they explicitly have a “validation set” that is the target distribution of interest.\n- The paper misses out on a lot of relevant literature that also tackles the same problem of data curation for LMs. Some notable omissions are: [DataComp-LM](https://arxiv.org/abs/2502.10341), [AskLLM](https://arxiv.org/pdf/2402.09668), [Qurating](https://arxiv.org/abs/2402.09739), [Nemotron-CC](https://arxiv.org/abs/2412.02595), [DSIR](https://arxiv.org/abs/2302.03169), [WebOrganizer](https://arxiv.org/abs/2502.10341), [SemDeDup](https://arxiv.org/abs/2303.09540) etc.\n- There is a lack of good baselines. All the experiments conducted only show comparisons to a standard baseline that is trained on uniformly sampled data. However, simple baselines like training a fast-text classifier on the same training data distribution as DataRater as the positives are missing. It is unclear if DataRater would substantially outperform such simple data selection methods, justifying the additional cost of the meta-learning training (given that training a fast-text classifier is extremely cheap in comparison).", "was_revised": true, "final_justification": "After the rebuttal, most of my concerns have been addressed, and so I am slightly leaning towards acceptance.", "rebuttal": "We are grateful the reviewer found our paper very well written, the problem statement important, and our qualitative results insightful.\n\nPlease find our replies below.\n\n***Tuning of DR***\n\nThe DataRater does not need a labeled golden high-quality dataset, so in this sense requires significantly less curation effort than building a fast-text classifier. The DR filtering threshold (i.e. discard proportion) can be tuned at the smallest model scale, as shown in Figure 4, as this is shown to generalize upwards to larger model scales.\n\n***Multi-epoch setting***\n\nWe can train the DR in a single or multi-epoch setting, and found it works well in both. However, in our presented version, the DR does not explicitly account for repeats since this information is not provided as an input to the DR model. A more contextually-aware DR, which would include model-state dependent info as DR inputs, could be an interesting avenue for future work.\n\n***Why reset inner model parameters***\n\nWe wish to use a frozen trained DR to filter datasets for future downstream training runs. For this reason, we want it to generalise across all stages of training, rather than fit to a particular model at a particular stage of training. We found that having a population of inner models with resets to maintain diversity over the stage of training helps to promote this generalisation. As suggested, we will expand on this point in the final paper version.\n\n***Offline vs Online, Amortization***\n\n> One thing that is not clear to me from this statement in lines 168-172: Is this using offline DataRater or an online version that filters out every batch?\n\nThis is using an online DataRater version to filter data batches online. The overhead incurred by the DR inference cost is shown in pink in Figure 1.\n\n> Do you assume amortization of running the DataRater model on the full dataset once, or do you still count that towards the total FLOPs needed for training?\n\nWe do not assume DataRater inference amortization. We count its inference costs in the total FLOPs needed for training.\n\n> The claims in the paper around compute efficiency sometimes use number of training steps, and sometimes use FLOPs. If indeed the cost of running inference of DataRater is not accounted for, making claims about compute efficiency purely using number of training steps is not fair in my opinion.\n\nIn Figure 1, we explicitly account for the cost of running inference to filter data online using the DataRater model: the compute overhead (as a proportion of FLOPS) due to the inference cost of the DataRater is shown in pink. To calculate this we converted both the LLM training step cost into FLOPs as well as the DR inference step cost into FLOPs (while accounting for batch-size oversampling prior to DR online filtering). As the reviewer points out this is a critical point for the fairness of our claims, and we will update lines 169-170 as well as the caption of Figure 1 to make this accounting unambiguous.\n\n***Complementary of filtering pipelines***\n\nAll datasets we use have various degrees of heuristic filtering applied to them (e.g. for the Pile see Appendix C in [1], for C4 see page 6 of [2]).\n\nOur quantitative results show that DR filtering is complementary to the filtering done in the Pile, as well to the one in C4/en.noclean, with less complementarity over the filtering used in C4/en. \n\nOur qualitative findings indicate that the DR tends to pick up on unusual text (poorly encoded, OCR errors, non-standard characters) and highly-patterned uninformative text (long lists, tables, indices, log outputs), which can be complementary to various heuristic filters. The fact that the DataRater still finds gains on top of these strong pre-filtered datasets is, in our view, a key strength of the method.\n\n[1] Gao et al., The Pile: An 800gb 314 dataset of diverse text for language modeling.\n\n[2] Raffel, Shazeer, Roberts, Lee et al., Exploring the limits of transfer learning with a unified text-to-text transformer.\n\n***Test distribution choices***\n\nWe have experimented with applying the DR framework to test distributions different from the training distribution (Appendix A). E.g., when using English Wikipedia as the test distribution, the resulting DR downweighted user forum data and foreign language text while upweighting data from the most common websites. Similarly, when using a dataset of code examples as the test distribution, the resulting DR upweighted scientific and mathematical text. \n\nThis use case requires selecting downstream metrics a priori which is often hard to reconcile with the breadth of capabilities required by foundational model training. So we did not regard this use case as the most practical one, and instead we focused our paper on the general case of self-supervised data curation.\n\n***Additional data curation literature***\n\nWe thank the reviewer for their comprehensive suggestions and will include the additional references provided in the final version of the paper.\n\n***Fast-text classifier***\n\nA fast-text classifier needs labeled data in the form of, e.g., positive and negative examples. In the case where such labelled data is given, of course practitioners should experiment with a fast-text classifier. However, our method tackles the more general case of self-supervision, where no golden set is given.\n\nWe hope our replies have suitably addressed your concerns. Please let us know if you have any other questions."}]

Author rebuttal:

Meta-review: This work proposed the "DataRater", a meta-learning approach to automatically learn the value of data and use it to improve the compute efficiency of training foundation models. The DataRater framework trains a data-selector model in a meta-learning fashion by optimizing through unrolled inner optimizations of an actual LM being trained. The paper conducts experiments across different model scales on three publicly available datasets. Empirical evidence shows that the trained DataRater models can select high-quality training data for LMs, thereby improving compute efficiency of training.

This work has multiple significant strengths. 

- The problem outlined by the paper---improving training efficiency via data curation---is an extremely important problem and and underexplored at scale.
- Meta-learning data value, bilevel optimization, and filtering via data scoring are all well-explored comprehensively.
- The paper is well-written with clear mathematical formulation and algorithmic specification.


While a reviewer still suggests that this work lacks empirical comparisons with heuristic, perplexity-based, or random filtering, so the key baseline gap and cost-benefit uncertainty remain. This does not change the consensus to accept this work.

The work touched an important problem and made significant progress with solid empirical results and clear theoretical motivation.

Given the consensus among the reviewers and the foregoing strengths, I recommend Accept (Poster).
Decision: Accept (poster)