MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
Authors: Zichun Yu
Spandan Das
Chenyan Xiong
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Pretraining
Data Curation
Primary area: natural_language_processing
TL;DR: We introduce model-aware data selection with data influence models (MATES), where a data influence model continuously adapts to the evolving data preferences of the pretraining model and selects the data to optimize the efficacy of the pretraining.

Abstract: Pretraining data selection has the potential to improve language model pretraining efficiency by utilizing higher-quality data from massive web data corpora. Current data selection methods, which rely on either hand-crafted rules or larger reference models, are conducted statically and do not capture the evolving data preferences during pretraining. In this paper, we introduce *model-aware data selection with data influence models (MATES)*, where a data influence model continuously adapts to the evolving data preferences of the pretraining model and then selects the data most effective for the current pretraining progress. Specifically, we collect oracle data influence by locally probing the pretraining model and fine-tune a small data influence model to approximate it accurately. The data influence model then predicts data influence over the whole pretraining corpus and selects the most influential data for the next pretraining stage. Experiments of pretraining 410M and 1B models on the C4 dataset demonstrate that MATES significantly outperforms random data selection on extensive downstream tasks. It doubles the gains achieved by the state-of-the-art data selection approach that leverages larger reference models and reduces the total FLOPs required to reach certain performances by half. Further analyses validate the effectiveness of the locally probed oracle data influence and the approximation with data influence models. Our code is open-sourced at https://github.com/cxcscmu/MATES.

MATES Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
Zichun Yu Spandan Das
Chenyan Xiong
School of Computer Science
Carnegie Mellon University
{zichunyu, spandand, cx)@andrew.
cmu, cmu. edu
Abstract
Pretraining data selection has the potential improve language model pretraining efficiency by utilizing higher-quality data from massive web data corpora. Current data selection methods, which rely on either hand-crafted rules larger reference models, are conducted statically and do not capture the evolving data preferences during pretraining. In this paper, we introduce model-aware data selection with data influence models (MATES), where a data influence model continuously adapts the evolving data preferences of the pretraining model and then selects the data most effective for the current pretraining progress. Specifically, we collect oracle data influence by locally probing the pretraining model and fine-tune a small data influence model to approximate it accurately. The data influence model then predicts data influence over the whole pretraining corpus and selects the most influential data for the next pretraining stage. Experiments of pretraining 10M and models on the C4 dataset demonstrate that MATES significantly outperforms random data selection extensive downstream tasks. It doubles the gains achieved by the state-of-the-art data selection approach that leverages larger reference models and reduces the total FLOPs required to reach certain performances by half. Further analyses validate the effectiveness of the locally probed oracle data influence and the approximation with data influence models. Our code is open-sourced at httts://gthbbbcoo///cccc//MMM
Introduction
The power of large language models (LLMs) rises with scaling up 7 with more parameters on more data using more compute resources 23
23 58
pretraining models
Among these three aspects of scaling, compute is often the most restrictive factor, as current large-scale pretraining frequently demands millions of GPU hours 2 8 58], while the model parameters and the pretraining data amounts are determined based on the pre-allocated compute budget [23 (27)
27
This provides a unique opportunity to elevate the scaling law of pretraining through data selection since the available data sources, such as the web 422 (55), are orders of magnitude bigger than available compute resources and contain data of varying quality. Recent research has shown that effective data selection can improve the generalization ability pretrained models [15 62). enhance
scaling efficiency [5], and introduce specialized capabilities [34]. These explorations mainly focus on
heuristic-based methods, such as rule-based filtering 49) 50 (55), deduplication 45 57/ proximity
to high-quality corpora [17] 33 61 64],, and prompting LLMs
[52] 62 Despite their success on
certain datasets and models, these techniques rely heavily on the static heuristics and often overlook
the evolving nature of pretraining [39). As result, their performance tends to be limited when
applied to the real-worke pretraining scenarios. [
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The data preferences of pretraining models dynamically shift as they progress through different stages of pretraining
39 40). For instance, Figure Ia Illlsrrttts how the data influence measured by the pretraining model evolves at different pretraining steps. As result, the data quality measurement should also keep pace with the model's 's evolving data preferences during the pretraining. This leads us to our core research question: How can we precisely track the data influence with the pretraining model and efficiently select pretraining data based on the acquired inftuence?
In this paper, we introduce Model-A ware data selec- tion with daTa influencE modelS (MATES), new pretraining paradigm where pretraining data is selected on-the-fly by a data influ-
1.0
Data Influcncc Correlation
Avg. Downstream Ace
48
10k-
1.000
0.413
0.505
0.317
MATES
Trininn Steps
47
Random
0.8
20k-0.413 0.413
1.000
0.522
0.389
46
2.3x Faster
30k-0..0 0,505
0.522
1.000
0.623
0.6
4.5
ence model capturing the
40k 0.317
0.389
0.623
1,000
00.
44
4
8
12
16
10k
20k 30k
40k
ever-changing data prefer- ences of the pretraining model. To track the model's data preferences, we locally probe the oracle data in- fluence by evaluating the pretraining model's perfor- mance on a reference task after training on individual data points. Then, we train a small data influence model using the locally probed oracle data influence, which then predicts data influence over the whole pretraining corpus and selects the most influential data for the next pretraining stage. This side-by-side learning framework ensures that the data influence model continuously adapts to the evolving data preferences of the pretraining model, providing the most valuable data accordingly,
Training Steps
Training FLOPs (1e19)
(a) Preference correlation.
(b) Accuracy w.r.t FLOPs.
Figure 1: Correlation of locally probed data influences at different pretraining steps (a) and the zero-shot performance with model-aware data selection (b). The experiments are based on 1B models.
Our pretraining experiments with 410M and IB models on the C4 dataset (50] demonstrate that MATES can significantly outperform random selection by an average zero-shot accuracy of 1.3% (410M) and 1.1% (IB) across various downstream tasks, ranging from reading comprehension, commonsense reasoning, and question answering. MATES doubles the gains obtained by statc-of-the- art data selection approaches that rely on signals from reference models larger than the pretraining model. Furthermore, our model-aware data selection significantly elevates the scaling curve of pretraining models, shown in Figure 1b reducing the total FLOPs required to achieve certain downstream performances by more than half. Further analyses confirm the advantages of our locally probed oracle data influence and the effective approximation of this oracle with data influence models. Ablation studies demonstrate the robustness of MATES across various hyperparameter settings and different design choices of the data influence models.
We summarize our main contributions as follows:
1. We propose a model- -aware data selection framework, MATES, where a small data influence model continuously adapts to the constantly changing data preferences of the pretraining model and selects the training data to optimize the efficacy of the pretraining process.
2. We effectively collect oracle data influence through local probing with the pretraining model and use small BERT-base model to approximate it accurately.
3. We empirically verify the superiority of MATES over rule-based, wiweencceffnction--e and LLM-based selection methods and the effectiveness the probed oracle and its approximation.
2 Related work
Early approaches on data selection relied heavily on manual intuitions. For example, T5 [50] first proposed the C4 pipeline, followed by Gopher rules [49]. which utilized criteria like document length, mean word length, and the presence of harmful or stop words to curate data. Recent Fine Web
dataset (46 further applied quality and repetition filters on top of these basic rules. These rule-based data selection methods have been shown effective as an initial data curation step [46] though manual intuitions may not capture the nuances of models' data preferences (36).

---

Language Model
Langunge Model
language Model
Training w/ random data
Training w/ selected data
Training w/ selected data
Collect data influence
Collect data influence
Objective
Maximize the target performance of the language model within the given FLOPs
Predictt influence
Predict data influence
Training
Training
Data Influence Model
Data influence Model
Initialization
Initializaaion
Figure 2: Overview of MATES. The language model is first pretrained with a random data. Then, data influence model is trained to approximate data influences on the target performance of the pretraining model and select the most effective data for the next pretraining stage.
Deduplication is another standard approach in pretraining data selection. Specifically, Lee et al. 32 and Penedo et al. [45] explored exact string match and fuzzy MinHash to filter duplicate sequences. SemDeDup [ further leveraged pretrained embeddings to identify semantic duplicates, and D4 [57] introduced diversification factors into the deduplication process. These methods effectively narrow down the number of similar documents in corpus and are often used together with quality-oriented selection techniques improve performance.
Selecting data that is proximate to high-quality corpora can enhance the data quality as well 17 Techniques leveraging n-gram similarity [4] and language modeling perplexity 8 61
61
have been adopted to evaluate how closely sequences in large datasets approximate high-quality data. As the size of pretraining data grows, the effectiveness of proximity- bassed methods becomes unclear, as they potentially reduce the diversity of the pretraining data and, consequently, the general capability of the pretrained models (36]
Recent advancements explore the use LLMSs to improve the pretraining data quality. For instance, QuRating (62 and Ask-LLM [52] employed LLMS like GPT-3.5 to annotate high-quality documents. Maini et al. (38 rephrased web corpora by providing L.MMs with detailed prompts to balance quality and diversity. These methods leverage the capabilities of strong reference LLMs, which are often several orders of magnitude larger, to guide the pretraining of smaller models.
Influence functions 229 59] provide theoretical tool to assess the impact of individual data points on a model's performance. However, they face scalability challenges the context of LLMs due to the expensive gradient calculations [21] [54]. To efficiently and robustly approximate influence functions, TRAK 44 performed Taylor approximation and gradient dimension reduction, making influence computation feasible for pretraining experiments [15). Nevertheless, the computational cost remains prohibitive for model-aware data selection, which requires tracking the evolving data preferences of pretraining models on the fly.
On the other hand, many researchers have proposed curriculum learning strategies that dynamically adjust the data distribution in the pretraining [ 24 47 [1 (56 62]. ELECTRA-style models (I] incorporated curriculum learning their pretraining process by synchronously training the model with an auxiliary generator, which provided more and more difficult training signals for the discriminator. This implicit curriculum significantly improved the pretraining efficiency on denoising language models [4] [1 19
39) 66]. Other methods have explicitly designed the curriculum for pretraining data selection, such as decreasing gradient norms (56), least certainty [24] 51. and increasing expertise (62), demonstrating the benefits of adapting the pretraining data signal according to the model's s ever-changing preferences.
Methods
This section first introduces the model-aware data selection framework MATES 3.1 and then proposes local probing technique to collect oracle data influence during pretraining ($ 3.2

---

Algorithm Model-Aware Data Selection
Require: Training data Dr, hold-out data Dr. reference data D,, pretraining model M, optimizer A, total training step T, selected size k, data influence model O sampling temperature т, update step U
Initialize pretraining model parameters MM Initialize ST. as randomly sampled size-k subset from Dr for t I,..., do if mod U 0 then
Collect oracle data influence {IM D,) 24 E Dx}
Fine-tune data influence model e on [T; [M; D,)) I E Dn)
Select data S (Gumbel-Top-Ae (zx,) x4 E D
end if
Sample batch of data 8 from St
MM < A(M, B*)
end for
3.1 Model-aware data selection framework
MATES selects the most effective data for pretraining a language model, aiming to maximize its final target performance, as illustrated in Figure The target performance here can evaluated using any downstream task or their combinations. Specifically, we leverage non-evaluation data as a reference for the model's target performance and select the pretraining data according to the reference loss.
Formally, given a size-n pretraining dataset Dj and the current model state M,, in each iteration, the objective of data selection to find an optimal batch 8 from D, minimize the loss L over the reference data D, after training M on B":
B arg ming L(D, B)M, where L(D, I A(M, B)) = ((ay)D, fly A(M, B)), M + A*M,
(1)
(2)
where A(M,, B) denotes the optimization of model M on a batch B, e.g., one-step training with Adam (28] and denotes the function to compute the model loss on an input-output pair (z, y).
There are two challenges to implement this framework. First, enumerating all possible batches will exponentially increase computational complexity. Second, obtaining oracle data influence for all pretraining data points is challenging. To address these issues, we introduce two techniques: pointwise data influence and data influence parameterization.
Pointwise Data Influence. To avoid the computationally intensive task of enumerating all possible batches, more practical workaround is to decompose the group influence into the pointwise influence [5] 4 Following previous research [44), we aggregate all the data influences by the summation, assuming that each data point 2; has an independent influence irrespective of the others:
C(D, A(M,) = Σ Im(a;:D,)
(4)
Z.EB
where IM is the oracle pointwise data influence function based on model state M. IM continuously changes along with the model pretraining.
Data Influence Parameterization. Estimating oracle pointwise data influence normally involves 29 44), which impractical to perform over millions of pretraining examples for every pretraining model state. To make the data influence collection feasible, we propose to collect the oracle data influence on a small hold-out dataset D,, (sampled from the same distribution as D,) and fine-tune a small data influence model θ on {(1,, [m( D,)) x; E Dr) to approximate the oracle. This data influence parameterization process transfers the costly influence computation to the small data influence model's inference.
gradient-based calculation 29

---

Then, we obtain the influence prediction (z;;) with the fine-tuned data influence model over all the training examples 2; E D, For better efficiency, we only asynchronously update the data influence model every U steps with the new oracle TM. Therefore, one data influence model checkpoint can select the entire data subset St for the next U steps of pretraining. The selection process uses the Gumbel-Top-k: algorithm [30] 62) to sample data from D;, with influence scores as weights:
S -GGumbel-Top-kl (z+))
|,, Di).
(5)
where т is the sampling temperature. The update step U is chosen to balance the data selection efficiency with the evolving data influence. Empirically, we warm up the model with a randomly sampled size-k subset from D, in the initial U steps. The overall pretraining and data selection pipeline of MATES is illustrated in Algorithm
3.2 Locally probed oracle data influence
The rest of this section presents our method to estimate the oracle data influence IM- We start the derivation from the standard influence functions [29] that quantify the reference loss change if one data point Ij in the training data is upweighted by a small c. We denote the optimal model state after the upweighting Maz, arg min M 2 fary .M) effor M) and simplify the optimal model under upweighting zù is given by:
case (i.e., no upweighting) M Then, the oracle data influence of
IM (2;; de dL(D, 1 M
(6)
de
l==0
dMi.z.
VMC(D, | M.) de
(7)
-VMR(DD M') H MMLa M*)
(8)
where HM. = " Σ
VM M*) is the Hessian and positive definite. The derivation
from Eq. 7 to Eq. 8 given by building a quadratic approximation to the empirical risk around M and taking a Newton step (29). Now consider the case that we incorporate z; into the training data, which means e A. then the parameter change due to the inclusion æ; is M w
M R
Hzl MML(,, MM) and the influence in Eq. 8 can be further represented as:
= nV MC(D, M 6::: -M")
(9)
~ n(((D, M - L(D, MM)) CCD, M*) L(D, M M
(10)
(11)
In practice, we manage to obtain the data influence based on the current model state M, while the above influence calculation still remains meaningful in the non-converged state by adding a damping term À that ensures HM + Af is positive definite [29]. Under this assumption, the first term in Eq. [T] can be regarded as fixed value whatever z; is, since 2; sampled from the hold-out data Dj. The second term in Eq. 11 can be locally probed with the one-step training of the current model with the new 24, i.e., A(M, 2.)). This one-step training incorporates x; into the optimization of the current model. Finally, the influence of z; on the reference loss is:
[m (z) CLDD, M) L(D, A(M,
(12)
This formula means, for each I; in the hold-out data D,,, we run one-step training with the current model M and evaluate the difference reference loss before and after one-step training. ensure that a positive influence score reflects a beneficial impact on model performance, we empirically define the negative influence, L(D, M) - L(D, A(M, 2x)), as our locally probed oracle data influence of z;. A full derivation of locally probed oracle data influence can be found in Appendix A
5

---

Table 1: Zero-shot evaluation of pretraining 410M/1B models with different data selection methods. We report the accuracy (standard error) and the total GPU FLOPs for each method. Dependencies on stronger reference models (e.g., GPT-3.5) are denoted by Best performances marked bold.
Methods anoon -1119
SeQ
ARC-E ARC-C
LogiQA
OBQA
BooiQ
HellaSwag
PIQA
WinoGrande
Average
410M Setting: 410M model, 25B tokens
Random 6635
(4.1
25.6
24.7
29.4
58.9
39.7
(7.1
50.6
44.5
DSIR
(3.1
19.9
23.81.2)
27.0,u.)
284(23:
58.3w0
39.6
66.8La))
51.5ua
44.3 44.3
LESS 23625
64.6
23.1
25.2,1m)
30/42a:
55.6
4199a
67.2qu.)
51.0
44.6
SemDeDuup (2.3)
63.5..5
42.4
24.40.3)
27.6
30.0
58.2
40.8
67
$2.5
45.2
66 65.4...5
41.7
24.7
27.5
29.0g2a
57.500)
40.3
68. 68.
50. 50.10a
44.9
QuRaaiin" (26.3)
(4.8
25.3
20.2
58.9
40.7as
67.5
52.1
45.2
MATES (Ours) (1331)
66.0
41.8
25.0
25.7
30.8233
60.6
41.0gas)
68.7
$2.7(14)
45.8
IB Setting: IB model, 25B tokens
Random
65.8:149
(3.7
25.6
27. 27.5
31.8
600.2a.
43.8,a))
68.M(aa)
50.7
46404
DSIR (17e))
65.8:115)
12.5
24.7
28.7(18)
29.2
59.7
44.2mm
68.3
53.2
16.3(14)
SemDDDDp
66.5.1.9
(1:5
25.3
27.6
30.6p
60.2g.z)
45.3
69.7
52.5
47.1
DsDm pron
68.2
26.5
26.6
29.4
59.0mm
44.8.a
$1.9
46.7
QuRaring" prosi
67.1:15)
45.5
25.6
26.20
29.8
60.3,00)
45.2aa
70.2
51.604
46.9
MATES (Ours)
67.3
(4.9
25.9
28.7(18)
32.2
660.9au))
45.3
69.5(La)
$2.4114
47.5
4 Experimental methodologies
Implementation Details. We pretrain 410M/IB models with Pythia [5] architecture from scratch on the C4 dataset (50) as pretraining model M, and continuously fine-tune BERT-base [13] as data influence model O The data influence model is smaller than the pretraining model, ensuring the efficiency of data selection. More details of data influence models be found in Appendix B.3 For model pretraining, we utilize which offers flexibility in the varying training length 22 More details of WSD scheduler can be found in Appendix [.2
Saanuu--table-Decay (WSD) scheduler proposed in MiniCPM 255,
For MATES selection, sample 20% data with their influence scores as weights at each pretraining stage (10k steps). The sampling temperature T is set to 1.0 to balance the data quality and the diversity. The update step U is also set to 10k that the selected data is trained by epoch. Following 43) as our reference data D,.. LAMBADA is a widely-used language modeling task and often serves validation task for language model pretraining 77 23 A summary of these details can be found in Appendix B.I
DsDm [15]. we leverage LAMBADA
18 codebase to perform holistic evalu- ation of the pretraining models across downstream tasks, including SciQ (60], ARC-E [[2), ARC- C [2]. LogiQA [35], OBQA [4]], BoolQ [0). HellaSwag [666. PIQA [6], and WinoGrande (53). These tasks cover the core abilitics of the pretrained language model, ranging from reading compre- hension, commonsense reasoning, and question answering. We report zero-shot accuracy for BoolQ and WinoGrande; otherwise, normalized zero-shot accuracy. We also report the total GPU FLOPs, including model pretraining and data selection, as both are parts of the scaling formulae (20]
Evaluation Methods. We use
The learning outcome of our data influence model is evaluated by the validation Spearman correlation between its predictions and the oracle data influence on a 10% hold-out validation set. This set is sampled from the collected data-oracle mapping ((x;, (r,;; D,)) x E D,))
Baselines. We compare MATES with random selection as well as the state-of-the-art pretraining data selection baselines, which include (1) DSIR (64]: proximity with Wikipedia by n-gram fea- tures. (2) SemDeDup [I removal of semantically duplicate sentences. (3) LESS 63] computing
cosine similarity between training and validation (LAMBADA) gradients as influential scores. [
(4) DsDm [5] static approximation of influence scores on LAMBADA by well- well-trained proxy model 48 (5) QuRating [62]: ranking with Llarn-learned (58] quality scores identified by GPT-3.5
in terms of educational values. These baselines cover all the mainstream data selection schemes, ranging from simple heuristics, statie influence functions, and I.I.M rating. Some recent methods, such as Ask-L.MM (2 are not open-sourced yet, prohibiting direct comparisons.
Due to the high computational cost of LESS to compute the gradients for all the pretraining examples, we only report its results in the 410M setting.

---

46
Avg. Downstream Ace
46
Avg Downstream Ace
48
Avg. Downstrcan Ape
48
Downstream
-8 Random
45
Randon
47
Randum
47
-B- Kandem
46
46
44
44
45
45
43
43
10k 20k 30k 40k 50k
10
15
20
10k 20k 30k 40k 50k Training Steps
Training FLOPs (Ic19)
Training Steps
Training FLOPs (Ic19)
(a) 410M FL.PPs.
(b) 410M w/ steps.
(c) IB w/ FL.PPs
(d) 1B w/ steps.
Figure 3: Downstream performance of 410M and IB models w.r.t. pretraining FLOPs and steps. The data selection procedure of MATES only accounts for 21.7% and 11.5% of the total FLOPs for 410M and IB models, respectively.
Evaluation results
This section evaluates the effectiveness MATES ($ 3.I locally probed oracle data influence ($ and data influence model ($ 5.3). It further presents study to illustrate the model's changing data preferences ($ 5.4.. More ablation studies can be found in Appendix
5.1 Overall performance
MATES outperforms the state-of-the-art data selection approach. Table presents the zero- shot evaluation of pretraining 410M/IB models with different data selection methods. MATES significantly outperforms random selection by average downstream accuracy gain of 1.3% and 1.1% in the 410M and IB settings, respectively. These gains are nearly double those achieved by the state-of-tte the-art data selection method, QuRating, which depends larger reference models. Notably, we observe an absolute improvement of nearly 2.0% across most tasks in the 410M setting, except for ARC-C and LogiQA, where the predictions of 410M models close random guessing. In the 1B setting, MATES outperforms random selection across all 9 tasks, demonstrating the strong scalability and generalization capabilities of our method.
MATES selects the data with low costs. We also show detailed breakdown of the pre- training cost of MATES in Table [ The rel- ative selection cost of larger models is generally smaller since their pretraining cost dominates the total FL.OPs while the training and inference cost of our data influence model remains stable. Even with the IB pretraining model, the wall clock time to collect one oracle data influence only around 2.5 seconds one GPU, which means we can get all the required 160k oracle scores during 50k-step pretraining on node (8 GPUs) around 14 hours, which is significantly lower than the actual pretraining time (4 days). The inference speed of our data influence model also be improved with data parallelism a fast-inference framework like vLLMM (31), re-
Table 2: FL.OPs breakdown of MATES steps.
Process
#FLOPs +ce19
Ratio
410M Setting:
410M model, 25B tokens
Model pretraining
6.35
78.3%
Oracle data influence collection
0.29
3.6%
Data influence model training Data influence model inference Total
1.46
18.0%
8.11
100%
IB Setting: 1B model. 25B tokens
Model pretraining Oracle data influence collection
17.67
88.5%
0.83
4.1%
Data influence model training
0.1%
Data influence model inference Total
1.46
7.3%
19.97
100%
ducing the selection cost further. This breakdown analysis underscores the low expense of MATES in achieving model-aware data selection.
MATES significantly elevates the scaling curves. Figure 3 plots the performance of the pretraining models w.r.t. different FLOPs and steps. Measuring by FLOPs counts both the model pretraining and data selection costs, demonstrating the total compute expense during pretraining. Measuring by steps reflects the compute cost of the model pretraining alone, as the data selection can be trivially parallelized when more computational resources available. At both 410M and IB scales, MATES
7

---

Stundard Deviation=O.. 114 Pos (46%), Neg (54%)
Standard Deviation Pos (15%). Neg (85%)
Stundard Deviation-O.05. Pos (28%), Neg (72%) Val. Spearmin-0.
Stundard Deviation 110 Pos (18%). Neg (82%)
Probability Density -
Spearman-0..
Probubiltty Density
Speurman-0.33
Probability Density
Probability Density S
Spearman-0422
CJ 2
1
0.0 0.3
0.6
0.3 Oracle Data Influence
0.0
0.3
Oracle Data Influence
Oracle Data Influence
Oracle Data Influence
(a) LAMBADA.
ARC-E (MC).
(c) ARC-E (LM).
(d) FLAN.
Figure 4:
Oracle data influence distribution in the 410M setting with different reference tasks at 50k steps. MC: multiple choice. LM: language modeling. We also present the standard deviation of the distribution and the proportions of the data with positive/negative oracle data influence.
Table 3: Performances of oracle selected data with different reference tasks in the 410M setting. We run the decay stage starting from the MATES model at 50k steps.
D.
SeiQ
ARC-E
ARC-C
LogiQA
OBQA
BoolQ
HellaSwag
PIQA
WinoGrande
Average
LAMBADA
66.0
4202
24.((13)
27.2
30.8
(9.
41.9
68.5
52.3
45 45.9
ARC-E (MC)
64.9(a))
42.4(m.)
24.9(z.)
27,8(18)
30.4g2:;
58.0pa)
41.s
68.1(1a)
51.7n4
45.5;14
ARC-E (LM)
65.3(1)))
43.0
24.8
28.0
31.8pt
58.5
40.7
67.2
$2.5
45.8:14)
FLAN
66.4
(5.
25.1
28.7
32.0
40.5
67.9
52.3
46.0
significantly elevates the scaling curves compared to random selection, reducing the FLOPs and pretraining steps required reach a certain downstream performance by more than half. Scaling efficiency is more evident at the IB scale, where model pretraining dominates 88.5% of FLOPs versus 11.5% for data selection. As result, MATES reaches the same performance random selection with only 43.3% % of the total FL.OPs. A similar observation can be found in Figure 9 where the performance of MATES comparable to or higher than the full pretraining using data. These results reveal the promising potential of MATES in elevating the scaling law of foundation models.
5.2 Effectiveness of locally probed oracle data influence
This set of experiments analyzes the effectiveness of oracle data influence with different reference tasks. Besides LAMBADA used in our main experiment, we also consider taking the training sets of ARC-E [2] and FLAN [ as the reference tasks show the generalization ability of our method. tasks, while FLAN represents data. For ARC-E, we construct each example either multiple-choice selection (i.e., outputting the correct option [A-D]) or as a language modeling task (i.e., outputting the verbalized answer) to investigate whether the task format will affect the data influence collection and parameterization.
ARC-E represents one of the knowledge-baad beseed question-answertt large set of varied
In Figure 4 we demonstrate the oracle data influence distribution with different reference tasks, the standard deviation of the distribution, and the proportions of the data with positive/negative oracle data influence. The oracle distribution remains spread-out across all reference tasks, indicating that our oracle effectively differentiates data influences. Notably, the positive influence proportion for LAMBADA in Figure 4al is higher than others by more than 20%, suggesting that more data is deemed beneficial when LAMBADA serves as the reference. We hypothesize that this is because LAMBADA is essentially word prediction task, which aligns more closely with the pretraining objective compared to knowledge- based or instruction-followiii supported by the results for ARC-E (Figure/4b[4c) 4c). where the language modeling format identifies more positively influential data points compared to the multiple-choice format.
tasks. This hypothesis further
We further validate the effectiveness of the oracle across different reference tasks by sampling 20% data with oracle scores as weights. Due to the infeasibility obtaining oracle scores for all the pretraining data, we only one short decay stage with the data selected by different oracle scores.
8

---

As shown in Table 3 taking ARC-E as the reference task can benefit the model's in in-domain accuracy. but its generalization performance worse than using LAMBADA. In contrast, FLAN benefits wider range of downstream tasks due to its diverse instructions. However, there remains trade-off between the performance of different tasks, so the average accuracy of choosing FLAN is similar to that of LAMBADA. This experiment highlights the robustness of our oracle, as it is not limited to specific reference data but generalizes effectively across multiple reference tasks.
5.3 Effectiveness of data influence model
This experiment studies the effective- ness of data influence model. As shown in Figure our data influence model effectively approximates the or- acle data influence across various ref- erence tasks. Nevertheless, higher standard deviation in the oracle dis- tribution tends enhance the valida- tion Spearman correlation. This sug- gests that greater variability influ- ence scores can provide more diverse signals for the data influence model to capture, making it better approximate the oracle distribution.
0.7
Avg. Downstreunt Ace
47
MATES
Val. Spearman
0.6
46
1OR Selection
MATES
50k Sclection
0.5
Training
45
Training
0.4
44-
43
10k
20k
30k
40k
20k
30k
40k
50k
Training Steps
Training Steps
(a) Influence modeling.
(b) Downstream accuracy.
Figure 5: Static (based on a 10k or 50k random random-pretraaneed model checkpoint) data selection versus model-awan data selection in influence modeling and downstream accuracy.
Sogupare MATES with static
data influence models trained influence from 10k a 50k aandom-pretraing model checkpoint. In Figure55a we measure the validation Spearman correlation between the predictions of data influence models and the oracle data influence probed with each pretraining checkpoint. The correlations of static data influence models are always below 0.5, while data influence models in MATES, with dynamic updates, can capture the ever-changing data preferences more and more precisely along with the model pretraining (e.g., the correlation is around 0.7 in 40k steps). The effects of model-aware data selection are directly reflected in downstream accuracy. In Figure [5] the data selected by static data influence models will cause a notable performance drop, especially at the early pretraining stage. These observations confirm the ever-changing nature of data preferences in pretraining and the advantages of model-aware data selection to elevate the scaling curves of pretraining.
5.4
Case study
This case study demonstrates representative examples to illustrate the evolving preferences of the pretraining model in detail. As shown in Table the model the early pretraining stage (the 10k checkpoint) tends to favor learning natural narrative examples without delving too deeply into specific knowledge. At the 20k checkpoint, appears to shift focus toward factual knowledge (e.g., pages from Wikipedia) while gradually reducing reliance on natural narratives. At the 30k checkpoint, the model shows a preference for more detailed academic text, such as official teaching slides. By the 40k checkpoint, the model may begin learning more long-tail knowledge, like Telescopic Forklift Training. These examples provide insights into the evolving nature of the model's 's data preferences throughout pretraining. Although they may not cover every aspect of the selected data, observation highlights the necessity to adapt data selection strategies to different model learning stages.
Discussion and limitations
Combinational Measurement of Data Influence. One primary assumption in our work is that each data point contributes independently to the pretraining outcome, without accounting for its interactions with other data points. Despite common hypothesis [15] 44] the pretraining essentially applies the long-term combinational effect batched data on language models, and the learning of many advanced capabilities is accumulative. Efficiently and effectively measuring and learning the combinational and accumulative nature of the pretraining process can make us better understand and leverage the value of data [37].

---

Table 4: Case study in the 1B setting. We show the 300 character excerpts of data points with the ranks of their influence scores among 80k randomly sampled data at different pretraining checkpoints. A lower rank denotes higher influence.
Influence Rank |
Source
Text
O at 10k checkpoint 6376 at 20k checkpoint
Blog
She went to the place where Jonathan lay and gave to his servant's 's David's 's richest garment to he placed next him he lay crying out in his sickness. She went in and out of house. She went in and out the city gates. She waited for David in the place...
1 at 20k checkpoint 22998 at 30k checkpoint
Wikipodia
Two weeks later, Friedman threw three touchdown passes in 27-0 victory over Northwestern, One of Michi- gan's 's touchdowns was up when Friedman intercepted a Northwestern pass and returned it 13 yards. On the next play, Fricdman threw a touchdown pass to Dutch Marion...
3 30k checkpoint
CRITHINKEDU
Critical Thinking Across the European Higher Educa- tion Curricula), Education and Regional Development in Southern Europe: Should we invest in Critical Thinking across the Higher Education Curricula? First... What is Critical Thinking (CT)? CT is not only a high quality way of thinking (skill), but also a way of being (disposition)...
452 at 40k checkpoint
I 40k checkpoint 5954 at 10k checkpoint
forkLiftcertificatinn
It's group training course resources to help you master telescopic forklifts in record time. Or else you' re taking the course and throwing bunch of forklift telescopic training against wall and hoping something sticks. And forklift is only getting more popular. This chapter about handler course and certification...
Exploratory Scale. As an exploratory research work, our experiments are conducted at a moderate scale, with pretraining model of 410M or 1B parameters. Although the trend from 410M 1B indicates the robustness of our observations, remains unclear how well our methods scale up to production-level models with billions of parameters and trillions of pretraining tokens. On the one hand, moving to that scale provides more headroom for data selection with more urgent needs for efficiency, more leniency of candidate data. On the other hand, large-scale pretraining may yield various stability issues that require dedicated work to introduce new techniques [58] 67]. models to future work.
the relatively small compute spent data selection, and larger pool
We leave the exploration of larger
7 Conclusion
In this paper, introduce MATES, novel framework enhance the efficiency and effectiveness of language model pretraining through model-aware data selection. MATES leverages a data influence model to continuously capture the evolving data preferences of the pretraining model throughout the pretraining process, thereby selecting the training data most effective for the current pretraining stage. To achieve that, we locally probe the oracle data influence on a reference task using the pretraining model and fit the data influence model on the probed oracle. Our empirical results demonstrate that MATES surpasses random, rule-based, influence-funed function-basee and LLMM-baed data selection methods on pretraining, significantly elevating the scaling curves of pretraining LLMs. Further analyses confirm the effectiveness of our locally probed oracle data influence and the accurate approximation of this oracle with data influence models. Our work successfully demonstrates the potential of model-aware data curation pretraining, and we hope it will motivate further explorations on improving the scaling law of foundation models through better data curation techniques.
Acknowledgements
We sincerely thank Jie Lei, Fei Peng, Scott Yih, and Arnold Overwijk for discussing ideas and providing helpful feedback on this work.
10

---

References
[1] Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S. Morcos. SemDeDup: Data-efficient learning web-scale through semantic deduplication. In Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, flge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. ArXiv preprint, 2023.
[3] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, al. A survey on data selection for language models. ArXiv preprint, 2024.
[4] Payal Bajaj. Chenyan Xiong, Guolin Ke, Xiaodong Liu, Di He, Saurabh Tiwary, Tie- Yan Liu, Paul Bennett, Xia Song, and Jianfeng Gao. METRO: Efficient denoising pretraining of large scale autocncoding language models with model generated signals. ArXiv preprint, 2022.
[5] Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Proc. of ICML, 2023.
[6] Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense natural language. In Pruc. AAAI, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Kruegor, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proc. of NeurlPS, 2020.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, and et al. PaLM: Scaling language modeling with pathways. ArXiv preprint, 2022.
[9] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu. Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quae V. Le, and Jason Wei. Scaling cnrrucctoonfinctuned language models. JMLR, 2024.
[10] Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proc. of NAACL-HLT, 2019.
[11] Kevin Clark, Minh- Thang Luong, Quoe V. Lc, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In Proc. of ICLR, 2020.
[12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the ai2 reasoning challenge. ArXiv preprint, 2018.
[13] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidircctional transformers for language understanding. In Proc. of NAACL-HLT, 2019.
[14] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellic Webster, Maric Pellat, Kevin Robinson, Kathlcen S. Meier-Hellster, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le. Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with
In Proc, of ICML, 2022.
11

---

[15] Logan Engstrom, Axel Feldmann, and Aleksander Madry, DsDm: Model-aware dataset selection with datamodels. In Proc. of ICML, 2024.
[16] Ruyi Gan, Ziwei Wu, Renliang Sun, Junyu Lu, Xiaojun Wu, Dixiang Zhang, Kunhao Pan, Ping Yang. Qi Yang, Jiaxing Zhang, et al. Ziya2: Data-centric learning is all LLMs need. ArXiv preprint, 2023.
[17] Lco Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800gb dataset diverse text for language modeling. ArXiv preprint, 2021.
[18] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain L. Noac'h, h, Haonan I.i, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. framework for few-shot language model evaluation, 2023.
[19] Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, and Xia Song. Model-generated pretraining signals improves zero-shot generalization of text-to-text transformers, Proc. of ACL, 2023.
[20] Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data filtering-data curation cannot be compute agnostic. In Proc. of CVPR, 2024.
[21] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Studying large language model generalization with influence functions. ArXiv preprins, 2023.
[22] Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Workshop on Efficient Systems for Foundation Models I1, 2024.
[23] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute- optimal large language model training. In Proc. of NeurlPS, 2022.
[24] Xudong Hong, Sharid Lošiciga, and Asad Sayeed. A surprisal oracle for active curriculum language modeling. In Proc. of CoNLL, 2023.
[25] Shengding Hu, Yuge Tu, Xu Han, Chaoqua He. Ganqu Cui, Xiang Long. Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. ArXiv preprint, 2024.
[26] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. In Proc. of ICML, 2022.
[27] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv preprint, 2020.
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
[29] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proc. of ICML, 2017.
[30] Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel- top-k trick for sampling sequences without replacement. Proc. of ICMI, 2019.
[31] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proc. of SOSP, 2023.
12

---

[32] Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch and Nicholas Carlini. Deduplicating training data makes language models better. In Proc. of ACL, 2022.
[33] Jeffiey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. DataComp-I In search of the next generation of training sets for language models. Proc, of NeurIPS, 2024.
[34]
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Runchen Xu, Chen Lin, Yujuu Yang, Jian Jiao, Nan Duan, et al. Rho- I: Not all tokens are what you need. ArXiv preprint, 2024.
[35] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang. and Yue Zhang. LogiQA: A challenge dataset for machine reading comprehension with logical reasoning. In Proc. of IJCAI, 2020.
[36] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph. Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proc. of NAACL-HLLT, 2024.
[37] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proc. of NeurIPS, 2017.
[38] Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly, Rephrasing the web: A recipe for compute and data-efficient language modeling. In Proc. of ACL, 2024.
[39] Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. COCO- LM: correcting and contrasting text sequences for language model pretraining. In Proc. of NeurlPS, 2021.
[40] Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Pretraining text encoders with adversarial mixture of training signal generators. In Proc. of ICLR, 2022.
[41] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Proc. of EMNLP, 2018.
[42] Amold Overwijk, Chenyan Xiong, and Jamie Callan. ClueWeh22: 10 billion web documents with rich information. Proc. of SIGIR, 2022.
[43] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proc. of ACL, 2016.
[44] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. TRAK: Attributing model behavior at scale. Proc. of ICML, 2023.
[45] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RetinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only. Proc. of NeurIPS, 2023.
[46] Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. ArXiv preprint, 2024.
[47] Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, Yang You, et al. InfoBatch: Lossless training speed up by unbiased dynamic data pruning. In Proc. of ICLR, 2024.
[48] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, Jeffrey Dean, and Sanjay Ghemawat. Language models are unsupervised multitask learners. In Proc. of OSDI, 2019.
[49] Jack W Rac, Schastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmaann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods. analysis & insights from training Gopher. ArXiv preprins, 2021.
13

---

[50]
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JML.R 2020.
[51] Ravi S Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning. ArXiv preprint, 2021.
[52] Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient LLMs. ArXiv preprint, 2024.
[53] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proc. of AAAI, 2020.
[54] Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proc. of AAAI, 2022.
[55] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighofi, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannanch Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Proc. of ACL, 2024.
[56] Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. In Proc. of EMNLP, 2023.
[57] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving LLM pretraining via document de-duplication and diversification. In Proc. of NewrIPS, 2023.
[58] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, 2023.
[59] Sanford Weisherg and R Dennis Cook. Residuals and influence in regression. 1982.
[60] Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Workshop on Noisy Useerrtnde Text, 2017.
[61] Guillaume Wenzek, Marie-Anne Lachaux. Alexis Conneau, Vishrav Chaudhary. Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data, Proc. of LREC, 2020.
[62] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting high-quality data for training language models. In Proc. of ICMI. 2024.
[63] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. Proc. of ICML., 2024.
[64] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Proc. of NewrlPS, 2023.
[65] Zhenhui Xu, Linyuan Gong, Guolin Ke, Di He, Shuxin Zheng, Liwei Wang, Jiang Bian, and Tic- Yan Liu. MC-BERT: Efficient language pre- pre-training via a meta controller. ArXiv preprint, 2020.
[66] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proc. of ACL, 2019.
[67] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shlcifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar. Tianlu Wang. and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. ArXiv preprint, 2022.
14

---

A Full derivation of locally probed oracle data influence
This section presents full derivation of locally probed oracle data influence. First, we define the optimal model state after upweighting one training data point x; as:
M
arg min M
Σ clz (M) el(o, (M),
(13)
simplify the optimal model under e = 0 case (i.e., no upweighting) M Following Koh Liang 29)] data influence quantifies the reference loss change before and after one data point 2; in the training data is upweighted by a small e, i.e.:
IM- (I:; del dL(D, [
(14)
de
l==0
d.M..
| (a)
(15)
de
LmD
d.M..
To compute
we consider the optimality condition at Me,ag
cm0
V
Σ
Clay
M + eC(os Menss = 0.
(16)
71
Differentiating both sides with respect
dMM..
HM-
+ VML(ai MM) = O,
(17)
de
c=
dMG.z,
VMLLi, IM),
(18)
de
where HM- i E== VMrlar
(M") the Hessian and is positive definite. Therefore.
Im (oi; D,) VMC(D, M') H VMC(a, M').
(19)
With the first-order Taylor expansion, approximate the change in model parameters:
dM e,
Mr,. - M =
= -cH VmL(E IM).
(20)
de
===0
For 1. i.e., incorporating into the training data, this becomes:
M M - H VML((x M*)
(21)
Substituting H Mhl, M') in Eq.
with - -n(Mi :.:; - MM), we get:
(xt D.) R V IM)
-M")
(22)
= (L((D, M - L(D, IM'))
(23)
& L(D, M') + L(D, IMI
(24)
In practice, manage to obtain the data influence based the current model state M, while the above influence calculation still remains meaningful in the non-com non-converged state by adding a damping term À that ensures H. + X/ is positive definite (29). Finally, the locally probed oracle data influence of is:
[m( Dz) x C(D, M) + L(D, A(M,
(25)
ensure that positive influence score reflects beneficial impact on model performance, we empirically define the negative influence, L(Dr .M) L(D, A(M, xx)), our locally probed oracle data influence of
15

---

Table 5: Experimental configurations.
Configuration Name
Value
Pretraining
Dataset
C4
Tokens
25B
Model
Pythia-410M/IB (randomly initialized)
Steps
50k
Sequence length
1024
Batch size
512
Max learning rate
0.001
Data influence parameterization
80k at 10k steps 20k the following steps Pretrained BERT at 10k steps Last fine-tuned checkpoint the following steps
Amount collected oracle data
Initialization data influence model
Selection ratio
20%
Update step of data influence model Amount of data in reference tasks
10k
1024
Epochs
5
Batch size Max learning rate
256
5e-5
Validation set of data influence model
10% sampled from the collected oracle data
B Experimental details
This section provides our experimental configurations ($ [.J). details of WSD scheduler ($ B.2). and design of data influence model B.3).
B.1 Experimental configurations
We provide all experimental configurations in Table
B.2 WSD scheduler
WSD lcarning rate scheduler is initially proposed in MiniCPM (25) and is found to scale predictably and reliably similar to widely-used cosine learning rate scheduler 22). Moreover, WSD offers hetter flexibility than cosine for training across different lengths since its learning rate is constant during the stable stage. The learning rate of WSD scheduler is configured as follows:
W
A
T
if W
Ir(t)=
n
if <t<s
(26)
0.5 .,, i
where I, W. S. and D represent the number of steps now, at the end of the warmup, stable, and decay stages, respectively. 7) the max learning rate. our main experiments,
choose 2000. 50000. and 200. Inspired by MiniCPM (25), each checkpoint is evaluated after the short decay stage for better stability. We run all experiments on 8 A6000 GPUs, which will take days for 410M models and days for IB models.
B.3 Design of data influence model
Our BERT-based data influence model averages all the hidden representations last model layer to obtain the sequence representation h E R where H is the hidden size of the model. Note that BERT only support a maximum input sequence length of 512. To deal with our pretraining sequence length of 1024, we divide one sequence into two chunks and forward them separately. Then, we average the hidden representations from both chunks obtain the final h. This vector will be multiplied by regression output weight w. _ get the
16

---

model prediction w. h. The training objective is the mean squared error between the model prediction and the normalized ground truth IM.
Note that BERT has different tokenizer from our pretraining models, but the average sequence length of all the examples after the tokenization is almost the same. Our chunk-based design can be easily extended to longer pretraining sequenees the future.
C Additional experiments
This section presents the comparison of different data influence attribution methods C.I
the ablation study
on data influence models ($ C.2 and the analysis of oracle data influence ($
C.1 Comparison of different data influence attribution methods
Table 6: Performances locally probed oracle data influence, MATES, and DsDm in 410M setting at 40k steps. We show trro--soo/two-shot results.
Methods
SciQ
ARC-E
ARC-C
LogiQA
OBQA
Oracle MATES
(5.401))70.44)
(1) /43.6
(..2 (25.0
26.1am (25.7
(1 /3.,4 (2.1)
67.3 77..7 (1.3) 66.0
41.7 (1.) /44.4 (L.)
24.7 (1.7)/3/24..00 (2..0
26.9 /26.3
28.8(2.0) / 228.0
DsDm
(1.7 (43.2
23.7 (25.2
24.4 (3..
29.2 (29.4
Methods
BoolQ
HellaSwag
PIQA
WinoGrande
Average
Oracle
58.9 /59.1
(1.) (3..
(1.2
45.6 /4. (1.3)
MATES 59.6 (5.
(0.l /39.//99.6.. (3..6(0.5)
67.6
52.1 (5..33
45.4(L3) /6. 14.. (1.3)
DsDm
60.3 /58.1 (o.)
66.55.)
40.4(4))52.22/ (4.8 /4..6
We compare the performance of data selection directly using our oracle with DsDm the short decay stage in T Our locally probed oracle, utilizing one-step training, outperforms DsDm, which relies on Taylor approximation and gradient dimension reduction. The performance gap (0.8 zero-shot accuracy gain) significant, considering the decay stage only consists of 200 steps. This improvement can be attributed to factors: (I) we leverage the current model state calculate the data influence rather than relying on an existing checkpoint like DsDm, and (2) we perform one-step training to obtain oracle score, considering the training dynamics of the pretraining model and eliminating the precision loss from multiple approximation manipulations in DsDm. It is always costly to acquire the optimal data influence oracle for the entire pretraining dataset, but our local probing method with data influence parameterization beats Taylor approximation with fewer FLOPs, offering direction for future exploration.
C.2 Allaion study data influence models
Table 7: Ablation study of pretraining 410M models with different update steps U and sampling temperatures between 40k to 50k steps. We show sero-shot/too-shol results.
Hyperparameters
ScQQ
ARC-E
ARC-C
LogiQA
OBQA
0=10k, 7=1.0
25.0
25.7(L.)/24.901)
30.8|2.);/0066211
U=5k, 7=1.0
(5.8;.s;/744.ppp
/44.1
34:55.3//25511
22),
31.0,21;33..
U=2.5k, 1=100
65.0
41.8
(43.. (1.)
24.9/13//24./133
37.6(u9)/2644u)
30.6 /2..://22..0 22,88
U=10k, 7=2.0
(4.00..5/72.221.444
/4/8 (1.)
24.7
27.8
29.6 (3.6:,0/30..22u (30.6
0=10k, 7:0.0
65.9/u.5/7455((.4
41.8(uo)) (40.//43.5m00
24.6(u3//23.5(u2)
28.0(.3//24.6..))
30.0 /20.88//8888p
Hyperparameters
BoolQ
HellaSwag
PIQA
WinoGrande
Average
U=10k, 7=1.0
60.6,o.:572 /57
(8.7,1.;667Ann.
52.7(14)53344p
45,8
U=5k, 7=1.0
/58.8 (19)
41.1
(0.8
68.2 18.1
/2.5
/46.6113)
U=2.5k, 1=1.0
60.2 /57.5 ((99)
41.0p.5)/44 /40.6
67.5
(qm)
$1.5
(566:14/446001s
U=10k, 7=2.0
56.2 (
40.8
/40.55/440050 ((ss))
67.7
52.2
45.0::4::45.444
U=10k, 7=0.0
57.4 /55.8
/40.0
66.18,1666.8,11)
50.6(14)/511....
45:5554
This group of experiments conducts ablation studies on the key hyperparameters.
17

---

Influence Score (BS 16)
Spearman 0.898
Avg. Downstream Ace
Spearman 00.87
Spearman 0.399
Spearman 0.284
0.45
0.45
0.45
2.9
0.44
0.44
0.44
2.6
2.8
3.0
3.2
0.35 0.40 0.45 LAMBADA Ace
0.40 0.41 0.42 ARC-E Ace
0.66 0.67
0.68
Influence Score (BS
PIQA
(a) Batch size (BS).
(b) LAMBADA as D,.
(c) ARC-E as D,-
(d) PIQA as D,-
Figure 8: Correlation of oracle data influence probed with different batch sizes (a) and the correlatio between different reference task accuracy with average downstream accuracy (b-d).
Update Step and Sampling Temperature. Table shows the impact of hyperparameters downstream performance. We initialize the model with the 40k-step MATES checkpoint and adopt different hyperparameters the 40k-50k training. Decreasing the update step U from 10k to and 2.5k leads to little fluctuations since the model preferences may not dramatically change within 5k steps. Varying the sampling temperature to 2.0 and 0.0 causes decreased performances. The zero temperature extremely up- weiwesshts high qual-qualit data, reducing data diversity, while higher temperatures like 2.0 do not sufficiently emphasize data quality. Our observations highlight the necessity of balancing quality and diversity in the data selection.
Data Selection Ratio, As shown in Figure 6 MATES shows the consistent gains compared to random selection using either low/tigh-selecting ratio, ranging from 1/200 to 1/2. We also find the optimal sampling rate of a larger model (1B) is smaller
Ave Dewnstrem.. Age
0.47
that smaller model (410M). We hypothesize aggressive selection ratio, such
0.46
01-41084
that larger model may require 10%, since it is more robust to the subtle distribution change of the training data. However, too (1/200) or high selection ratio (1/2) does not perform good, low ratio may harm the diversity and high ratio does not leverage the strength of the data influence enough.
0.45
17200
1:10 115
Selection Ratio
Figure Zero-shot perfor- mances of MATES with dif- ferent data selection ratios.
Oracle Label Amount and Data Influence Model Scale. Figure demon- strates the impact of oracle label amounts by comparing the validation Spearman correlation different data influence models at 40k steps. The data influence model is initialized from either the pretrained BERT its
last checkpoint at 30k steps. We observe that continuous fine-tuning from 30k steps requires less than half of the oracle data compared training from the pretrained BERT, which can significantly reduce cost collecting new oracle data. Furthermore, Figure 7b studies the impact of parameter counts in the data influence model. Generally, the approximation becomes more accurate as the numher of parameters increases until BERT-large, which may have become saturated with current learning algorithm using available oracle label. have proved the effectiveness of our data influence models through extensive experiments and thus, leave the exploration of better data influence parameterization algorithms and stronger data influence models to future work.
C.3 Analysis of oracle data influence
To further demonstrate stability of our oracle data influence, enlarge the one-step training batch size (BS) from to 16. Following llyas ct al. (26). we utilize LASSO regression to separate each data's influence in the BS 16 setup and calculate the Spearman correlation between BS = 1 and B$ 16 scores. More details of LASSO regression can be found in Appendix C Figure &a illustrates that the oracle data influence is not sensitive to the batch size same model state. We calculate batch-level data influences, where
0.70
0.70
46.0
Val. Speurmmn
0.65
0.65
45.5
0.60
0.60
45.0
Cantinouu Trinits
0.55
1020
0.55
5:
50100
300
44.5
Avg.
Number Training Data (k)
#Parameters (M)
(a) Oracle label amount.
(b) Model scale.
Figure 7: Data influence parameterization with dif- ferent amounts of oracle labels and model scales.
long as they correspond
18

---

4n8 Dww.mmonnnnn....... ALC
Ang 8x
Are
000
Truvag (3406
Toanin Stege
Traning FL.P% 13419)
Trasiea Stege
(a) 410M w/ FLOPs.
(b) 410M w/ steps.
(c) IB w/ FLOPs.
IB w/ steps.
Figure 9: Downstream performance of 410M and IB models w.r.t. pretraining FLOPs and steps.
data point can appear on multiple batches so that separating individual data influence from batch influences regression problem. Formally, the input of the regressor is m-dimensiooal binary vector [M1, 12; -.. Unn] vector, where means the size of probing data pool (80k in our experiment) and = 1/e; = denotes the inclusion/cxclusion of the i-th data in the current batch; the output is the oracle data influence score probed by one-batch one-step training. In experiment, to 16 to train a LASSO regressor and then acquire mn-dimensional individual data influence vector from the regressor's prediction. This vector represents influence score shown the y-axis in Figure Sa The idea of adopting LASSO regression is inspired by the linear datamodels in DsDm 15 that separate individual data influence from the collection of batch data influences.
first collect 80k input-output pairs with batch size equal
We also investigate the traits of our reference task, LAMBADA, when collecting oracle data influences. Specifi- cally, we collect the task accuracy at each model checkpoint and measure the Spearman correlation between single task and average downstream accuracy. As shown in Figure 8b LAMBADA has a significantly positive
Spearman correlation (0.877) with average downstream accuracy. In contrast, the correlation between ARC- E/PIQA and average downstream accuracy is not high in Figure Sc
[d although they parts of the evaluation tasks. These results imply that ARC-E/PIQA may not reflect the target performance as accurately as LAMBADA when acting the reference.
19

---

NeurIPS Paper Checklist
Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions, such the introduction of aware data selection framework, the use of small data influence model, and the empirical validation of the proposed method. These claims are consistent with experimental results and discussions provided in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses several limitations, such the computational constraints that limit the model size to 1B parameters and the need for more advanced data sampling strategies.
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results violations of these assumptions (c.g., independence assumptions, noiseless settings. model well sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or specch-to-tove system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
appplicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't ack acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
20

---

Question: For each theoretical result. does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yss]
Justification: The paper provides detailed and complete thcoretical proof of oracle data influence proposed in the method.
Guidelines:
The NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should he numbered and cross-referenced.
All assumptions should clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the and data are provided or not)?
Answer: [Yes]
The paper includes detailed descriptions of the experimental setup, including the datasets used, the models employed. and the training procedures. also releases code, data, and model checkpoints facilitate reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contrihution is novel architecture, describing the architecture fully might suffice, the contribution is specific model empirical evaluation, may necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also provided via detailed instructions for how to replicate results. access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g.. to registered users), hut should be possible for other rescarchers to have some path to reproducing verifying the results.
21

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce main experimental results, as described supplemental material?
Answer: [Yes]
Justification: The paper releases code, data, and model checkpoints, providing sufficient instructions to reproduce the main experimental results.
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. http:/ippp.cppppbiil.l./
for more details.
While we encourage the release of code and data, we understand that this might not be possihle,
so "No" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g.. for a open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines lattps://nipp ccppiipiccc/ppbccc for more details.
reproduce
guides/ ChwoCoooeeieeiioonnonooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify training and test details (e-g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specifies details such as the datasets, model architectures, hyperparameters, and the training procedures used the experiments.
Guidelines:
The answer means that paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary appreciate the results and make of them.
The full details can provided either with the code, appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly detined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper includes standard emorr in the experimental results, ensuring that the reported performance gains are robust and reliable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing some parameter, or overall run with given experimental conditions).
22

---

The method for calculating the enror bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard of the mean.
is OK to report I-sigma enror bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be carcful not to show in tables or figures
symmetric error hars would yield results that are out of range (c.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding ligures or tables in the text.
Experiments Compute Resources
Question: each experiment, does paper provide sufficient information on computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
The paper provides details on the type of compute resources used, including the specific hardware, memory, and the approximate time required for execution of the experiments.
Justification:
Guidelines:
The answer means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
The research conducted the paper conforms, in every respect, with the NeurIPS Code of Ethies.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special cireumstances that require deviation from the Code of Ethies.
authors should make sure preserve anonymity (e.g., if there special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss hoth potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both positive and negative socictal impacts, including the importance of high-quality data in preventing bias and ensuring the responsible of language models.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper credits the original creators of datascts and models used in the experiments and provides references to the licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL..
The name of the license (c.g., .. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, http:/pppppaaaae thcode. wwt. datassta has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
existing datasets that are re-packaged, both original license and the license of derived asset (if it has changed) should be provided.
24

---

If this information is not available online, the authors encouraged to reach out to the asset's
creators.
13. New Assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets, such the data influence model, and provides documentation alongside them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as of their sub- missions via structured templates. This includes details about training. license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjeets
Does paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/revicw based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
25


Reviews:
[{"review_id": "3Uy1hGGFkO", "reviewer": "Reviewer_2Zbt", "summary": "The paper proposes a new method “MATES,” which aims to select pretraining data using a reference high-quality data source. MATES uses an estimated influence function to iteratively select the most influential datapoints. Experiments on the Pythia model + dataset show good promise over existing data curation methodologies.", "questions": "While I completely understand the cost of experiments included in the paper, I believe a few critical questions/extra experiments might greatly improve the paper. Please feel free to ignore any of the following questions in the interest of compute.\n\n\n- I couldn’t find details on which dataset was used. While the abstract mentions C4, the main text doesn’t specify this, making me question if the Pythia dataset was used instead?\n- The above question also makes way for a deeper question: since the selection ratio is fixed to 20% (of the entire dataset, I assume), this translates to how many *tokens sampled*? Are the training runs multi-epoch or is the sampled data greater than the training budget?\n- If the runs are multi-epoch, what are the results in the full-data (no sampling) case?\n- How does MATES compare to other approaches when the sampling rate is not equal to 20%? Is MATES also useful in the super low/high-sampling rate regime?\n- What if a different target dataset is used vs. only using lambada? Can we use a mixture of target datasets? How does the design of the target dataset/mixture affect different evals?\n- Finally, the evaluation benchmarks used in the paper seem to be cherry-picked based on the “high-level” proximity to the kind of data present in lambada. The paper would greatly benefit from using more diverse evaluation scenarios like coding, math, knowledge, etc.", "limitations": "The authors address the major technical limitations of their work in Section 6. No obvious negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Better trained models on MATES’ curated data vs. other data curation approaches\n- The distillation of influence scores into a smaller proxy model (BERT-base in the paper) allows MATES to perform iterative, model-state dependent data selection vs. other baselines which perform one-step, model-state-independent data selection.\n- Thorough experiments: various experiments and ablations are included, e.g., various relevant baseline comparisons, ablating the effect of various hyper-parameters in MATES, etc.", "weaknesses": "- I found the paper writing to be quite loose and missing context in various places. I had to read a lot of the text multiple times as well as filling in the missing context myself.\n- While I appreciate the extensive experiments covered in the paper, I feel the paper misses a few important results and/or ablations. See the questions section for more on this.\n- The choice of evaluation benchmarks doesn’t seem obvious and feels quite debatable to me. See the questions section for more on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper! We will address your questions/comments below:\n\n**Weakness 1:** Loose writing and missing context.\n\n**Response:** Thank you for pointing it out! We will carefully revise the paper from the following perspectives:\n\n1. Clarify the experimental setup, as mentioned in general response 2\n2. Add more theoretical derivation from the classic influence function formulae [1] to the representation of our oracle data influence\n3. Avoid grammar errors and missing context and add more detailed explanations for both our method and results\n\n**Question 1:** Pretraining dataset used.\n\n**Response**: We use the C4 dataset (the same as DsDm) only to pretrain all our main models from scratch. We will clarify it in the next version of the paper.\n\n**Question 2:** Can the selection ratio translate to the number of tokens? Are the training runs multi-epoch, or is the sampled data greater than the training budget?\n\n**Response:** The ratio can be translated into the number of tokens. In every 10k steps, we update the data influence model and select the data from separate pools randomly sampled from the entire C4. Each data selection pool is 5x larger than the training data of 10k steps, so we only select 20% from it. The whole pretraining data pool (C4) is huge, so we run one epoch training as most previous work (DsDm, QuRating) did.\n\n**Question 3:** Full-data results?\n\n**Response:** For the full training, we follow the original Pythia paper and pretrain the models with 150k steps. As shown in Figure 1 in our supplementary pdf, MATES can improve the pretraining efficiency by more than 3x, given that 50k MATES performance is comparable to or higher than the 150k full data training. Please see the general response 3.1 for more details.\n\n**Question 4:** How does MATES compare to other approaches when the sampling rate is not equal to 20%? Is MATES also useful in the super low/high-sampling rate regime?\n\n**Response:** Interesting question! We only have time to provide ablation results of different selection ratios in Figure 3a in our supplementary pdf, in which MATES shows consistent gains compared to random selection using either low/high-selection ratio, ranging from 1/200 to 1/2. We also find the optimal sampling rate of a larger model (1B) is smaller than that of a smaller model (410M). However, too low (1/200) or high selection ratio (1/2) will decrease the performance, as a low ratio may harm the diversity, and a high ratio does not leverage the strength of the data influence enough.\n\nFor your reference, this is the adopted selection ratio from our baseline papers. Our optimal selection ratio is close to QuRating/DsDm, which pretrains a model of a similar size as ours.\n\n| Methods   | Max Model Size    | Selection Ratio |\n| ----------- | ---------------- | --------------- |\n| DSIR    | 110M | 3.2%      |\n| SemDeDup  | 6.7B | 63%       |\n| QuRating  | 1.3B | 11.5%      |\n| DsDm    | 1.3B | 20%       |\n| DataComp-LM | 6.9B  | 10%       |\n\n**Question 5:** A different or a mixture of target datasets? How does the design of the target dataset/mixture affect different evals?\n\n**Response:** Thanks for your suggestion. We use FLAN (the mixture of multiple NLP tasks) as the target dataset to run our MATES on the recently released FineWeb datasets [2]. As shown in Figure 2a in our supplementary pdf, MATES still demonstrates superior average downstream performance on FineWeb compared to the state-of-the-art data selection methods, such as the FineWeb-Edu classifier and the fasttext-oh-eli5 classifier. Compared with taking LAMBADA as the target task, using FLAN can improve the model’s performance on both knowledge QA and commonsense tasks (Figure 2c, Figure 2d). We hypothesize that multi-task data (e.g., FLAN) has better coverage of the language task types and thus is more suitable for serving as the target task. Please see the general response 3.3 for more details.\n\n**Question 6:** Choice of the evaluation tasks\n\n**Response:** We followed the original Pythia paper and our main baseline QuRating for the evaluation benchmark selection and didn’t cherry-pick any tasks that resemble LAMBADA. LAMBADA is essentially a word prediction task and greatly differs from our QA-like evaluation tasks. It is found to be a great reference task due to its high correlation (0.877) with the average downstream accuracy (Figure 5b) and the high data influence modeling (>0.7) performance (Figure 6a).\n\nWe agree that coding, math, and knowledge are essential to evaluate a language model’s capabilities. For the knowledge part, actually most of our adopted QA tasks (SciQ, ARC-E, ARC-C, OBQA) require factual and commonsense knowledge to solve. In our supplementary pdf, we also evaluate MMLU [3] in Figure 2b. On MMLU, MATES can outperform random selection but not the FineWeb-Edu classifier. As mentioned in general response 3.3, the FineWeb-Edu classifier is optimized towards educational values derived from LLama3-70B-Instruct, which can benefit knowledge-intensive tasks like MMLU with the price of losing generalization abilities to other types of evaluation tasks. For the coding and math tasks, we didn’t include them since they are not commonly used to evaluate pretrained checkpoints w/o additional instruction tuning. A supportive evidence is that even the Llama 3 series [4] **didn’t choose to evaluate pretrained checkpoints** on HumanEval (coding) and MATH (math) due to the non-indicative performance.\n\n[1]: Koh, Pang Wei, and Percy Liang. \"Understanding black-box predictions via influence functions.\" Proceedings of the 34th ICML-Volume 70. 2017.\n\n[2]: Penedo, Guilherme, et al. \"The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.\" arXiv preprint arXiv:2406.17557 (2024).\n\n[3]: Hendrycks, Dan, et al. \"Measuring massive multitask language understanding.\" arXiv preprint arXiv:2009.03300 (2020).\n\n[4]: Dubey, Abhimanyu, et al. \"The Llama 3 Herd of Models\" arXiv preprint arXiv:2407.21783 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YI5Zi61Dt1", "reviewer": "Reviewer_xrwH", "summary": "It is important to carefully select data during the pretraining stage, as the pretraining data are often obtained from web crawling and can be extensive and noisy. Existing methods for data selection include heuristic-based approaches, clustering-based techniques, and the use of influence models. However, these methods result in static selected datasets and do not adapt to the evolving training process. This paper introduces a novel approach by developing an influence model that co-evolves with the main pretraining model. The selected data is no longer static; now, it is dynamic and sensitive to the model. The study demonstrates that continuously adapting data preferences enables efficient training, and the influence model is both small and accurate. Empirical evidence presented in the framework, MATES, proves its better performance compared to other methods, showcasing advantages in both performance and efficiency.", "questions": "1. It's an interesting paper, and I am curious about the changes in data preference. Can the authors provide some examples of what the model likes to learn at the beginning but discards later, or vice versa?\n2. This is not critical: but looking forward to seeing this approach being employed in other data curation benchmarks, such as DataComp.", "limitations": "I didn't see any potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Originality: This paper introduces a new data selection framework in the pretraining stage by learning a smaller data influence model and co-evolving with the main model's data preference.\n* Quality: Empirically, their results show promise to enable efficient and improved pretraining.\n* Clarity: Their writing and their result are clear to me.\n* Significance: Data curation is crucial to denoise massive pretraining datasets. This paper presents a new approach to selecting data points by learning model preferences. Their empirical results support their claim, and I believe their notion is significant to the field.", "weaknesses": "1. In Figure 2, it appears that the data influence model needs to be reinitialized at each iteration. It's not clear why we need to reset it instead of continuing to learn with the main model.\n2. It's unclear how the influence model includes and excludes data points and converts them into labels for learning the influence score. Could the authors clarify this? Additionally, what is the performance of the influence model across training epochs and how do we evaluate it? I believe there is an imbalance in the labels.\n3. Although the learning curve indicates efficient and improved performance, it's uncertain how much computational burden is involved in maintaining the trained influence model. It would be helpful to include these details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper! We will address your questions/comments below:\n\n**Weakness 1**: The initialization of the data influence model.\n\n**Response**: We only initialize our data influence model with pretrained BERT at 10k steps and continuously fine-tune it at the following steps. Figure 6a shows that initializing the data influence model with pretrained weights still works well with enough training data (80k), while continuous fine-tuning demands less training data to achieve the same performance, saving the oracle probing cost by 75% (80k -> 20k).\n\n**Weakness 2**: Unclear how the influence model includes and excludes data points and converts them into labels for learning the influence score.\n\n**Response**: The oracle data influence score is defined as the difference in the target loss before and after one-step training on a hold-out example $x_i$ (Equation 4). Please see the general response 2.2 for more details.\n\n**Weakness 3**: What is the performance of the influence model across training epochs, and how do we evaluate it? Is there an imbalance in the labels?\n\n**Response**: We already reported the performance of data influence models during pretraining in Figure 4. Please see the general response 2.3 for more details.\n\nThe oracle data influence distribution conforms nearly to the normal distribution, so there is no imbalance in the labels (Figure 3b in our supplementary pdf). As training steps increase, our data influence model can better capture the data influence of the main model with increasing validation performance.\n\n**Weakness 4**: The computational burden in maintaining the trained influence model.\n\n**Response**: As stated in lines 198-199, we reported the total pretraining FLOPs, including the cost of maintaining the data influence model, in Table 1. The actual additional data selection cost incurred by the MATES is (8.11 - 6.35) * 10^19 = **1.76 * 10^19** FLOPs for the 410M model and (19.97 - 17.67) * 10^19 = **2.3 * 10^19** FLOPs for the 1B model, which is near 9x lower than the selection cost of the state-of-the-art data selection method QuRating (**20 * 10^19** FLOPs). Figure 3 also reports the FLOPs-ACC curves and MATES greatly elevates the scaling curves (even with additional data selection cost) compared to random selection.\n\n**Question 1**: Some examples of what the model likes to learn at the beginning but discards later, or vice versa?\n\n**Response**:  We would like to present a table of the high-influence cases on a hold-out 500k subset whose rank decreases a lot across the training.\n\n| Selected Data Rank (Lower means higher Influence) | Source                | Text                                                         |\n| -------------------------------------------------- | --------------------- | ------------------------------------------------------------ |\n| 0 in 10k checkpoint; 6376 in 20k checkpoint                             | Blog                  | She went to the place where Jonathan lay and gave to his servant's David's richest garment to be placed next to him as he lay crying out in his sickness. She went in and out of the house. She went in and out of the city gates. She waited for David in the place |\n| 1 in 20k checkpoint; 22998 in 30k checkpoint                             | Wikipedia             | Two weeks later, Friedman threw three touchdown passes in a 27–0 victory over Northwestern. One of Michigan's touchdowns was set up when Friedman intercepted a Northwestern pass and returned it 13 yards. |\n| 3 in 30k checkpoint; 452 in 40k checkpoint                              | Slides in CRITHINKEDU | Critical Thinking Across the European Higher Education Curricula), Education and Regional Development in Southern Europe: Should we invest in Critical Thinking across the Higher Education Curricula? First… What is Critical Thinking (CT)? CT is not only a high quality way of thinking (skill), but also a way of being (disposition). |\n| 1 in 40k checkpoint; 5954 in 10k checkpoint                            | forkliftcertification | It’s a group of training course resources to help you master telescopic forklifts in record time. Or else you’re taking the course and throwing a bunch of forklift telescopic training against a wall and hoping something sticks. And forklift is only getting more popular. This chapter is about handler course and certification. |\n\nWe find that the model at the early pretraining stage (10k steps) prefers to learn the natural narrative w/o too much specific knowledge. In the 20k steps, the model focuses more on factual knowledge (e.g., the page from Wikipedia). In the 30k checkpoint, the model prefers more academic text, such as the official teaching slides. In 40k steps, the model turns to more long-tail knowledge like Telescopic forklift Training. This study shows that the model’s preferences for the data continuously evolve during the pretraining. We will add this analysis in the next version of the paper.\n\n**Question 2**: Employ MATES on other data curation benchmarks, such as DataComp.\n\n**Response**: We agree that DataComp [1]  is a good future experiment. In another recently released state-of-the-art pretraining data FineWeb [2] (Figure 2 in our supplementary pdf), MATES still demonstrates better average performance compared with the strong FineWeb-Edu classifier (similar to QuRating but using annotations generated by LLama3-70B-Instruct) and fasttext-oh-eli5 classifier (the best-performing classifier to mine instruction-like data in the DataComp paper). This result gives us confidence that MATES can contribute to the top-performing runs on the DataComp leaderboard under its standardized data selection setup.\n\n[1]: Li, Jeffrey, et al. \"DataComp-LM: In search of the next generation of training sets for language models.\" arXiv preprint arXiv:2406.11794 (2024).\n\n[2]: Penedo, Guilherme, et al. \"The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.\" arXiv preprint arXiv:2406.17557 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tJLzBCFk2Q", "reviewer": "Reviewer_T2sB", "summary": "This paper introduces model-aware data selection with data influence models (MATES) that selects high-quality data for pre-training large language models (LLMs). MATES addresses the issue that existing data selection methods don’t adapt to evolving data preferences during pre-training. MATES continuously adapts to the main model’s evolving data preferences by uses a small data influence model to select the most effective pre-training data for the next stage. The data influence model also reduces the cost of computing the influence scores on-the-fly. Experiments on Pythia and the C4 dataset show that MATES outperforms random data selection and existing static data selection approaches. Further analysis validates the ever-changing data preferences of the pre-training models and the effectiveness of our data influence models to capture it.", "questions": "1. The authors only compare against a full set of baseline methods in the Pythia-410M experiments. In the Pythia-1B experiments, the authors only compare against the random selection baseline. Why are the results of other baselines omitted in the Pythia-1B experiment? How do other baseline methods such as SemDedup perform in the Pythia 1B experiments? \n\n2. How does the proposed method MATES perform compared to using the full dataset for training?", "limitations": "The authors adequately discuss the limitations of the work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method outperforms existing static data selection baselines on the Pythia-410 model. Compared to random selection, it can achieve a certain accuracy with less GPU FLOPS.\n2. The authors demonstrate the importance of adapting to fluctuating data influence in data selection methods.", "weaknesses": "1. The authors only compare MATES to other static baseline methods on the Pythia-410M model. For most of the rest experiments, the authors only compare to random selection. \n2. Missing references. There are existing dynamic data pruning methods [1][2] that the authors may want to include in the discussion of related works. These works also noticed the influence of data points changes as the training proceeds.\n\n[1] Raju, Ravi S., Kyle Daruwalla, and Mikko Lipasti. \"Accelerating deep learning with dynamic data pruning.\" arXiv preprint arXiv:2111.12621 (2021).\n[2] Qin, Ziheng, et al. \"InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning.\" The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper! We will address your questions/comments below:\n\n**Weakness & Question 1**: Only comparing MATES to other static baseline methods on the Pythia-410M model. For most of the experiments, the authors only compare random selection.\n\n**Response**: We acknowledge the reviewer’s concern regarding the limited comparison of MATES to other baselines, specifically for larger-scale models. It is important to note that large-scale experiments are costly and often infeasible for many university research groups. In research or even the productive large language models like T5 [1] and Llama 3.1 [2], it’s a common practice to conduct comprehensive ablations at a smaller scale and then test the final performance at large scales. \nWith more resources, we are able to show all the 1B baselines in Table 1 in our supplementary pdf, where MATES still performs the best among all the baselines. This demonstrates our commitment to validating our approach at larger scales within our resource constraints, and we believe our results can provide valuable insights into MATES’s effectiveness across different model scales. Please see the general response 3.2 for more details.\n\n**Weakness 2**: Missing references.\n\n**Response**: Thanks for the references. Our work builds upon the growing emphasis on dynamic data selection for efficient model training. As with Raju et al. [3] and Qin. et al. [4], we share the common motivation of adapting the data selection as training progresses, as the value of training samples may change throughout the learning process.\nUnlike Raju et al.’s ε-greedy and upper confidence bound (UCB) approach, MATES uses a dedicated data influence model to capture the complex and changing relationships between training data and model performance. As a result, MATES is able to model more nuanced data preferences than methods based on simpler heuristics like the sum of mean and variance. \nQin et al.’s InfoBatch maintains unbiased gradient expectations through gradient rescaling on the instruction fine-tuning setup. In MATES, we employ local probing to collect oracle data influence, providing a more direct measure of sample importance compared to InfoBatch’s loss-based soft pruning policy, allowing for a more accurate selection in the complex pretraining settings.\nWe will cover these related works carefully in the next version.\n\n**Question 2**: How does the proposed method MATES perform compared to using the full dataset for training?\n\n**Response**: For the full training, we follow the original Pythia paper and pretrain the models with 150k steps. As shown in Figure 1 in our supplementary pdf, MATES can improve the pretraining efficiency by more than 3x, given that 50k MATES performance is comparable to or higher than the 150k full data training. Please see the general response 3.1 for more details.\n\n[1]: Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" Journal of machine learning research 21.140 (2020): 1-67.\n\n[2]: Dubey, Abhimanyu, et al. \"The Llama 3 Herd of Models\" arXiv preprint arXiv:2407.21783 (2024).\n\n[3] Raju, Ravi S., Kyle Daruwalla, and Mikko Lipasti. \"Accelerating deep learning with dynamic data pruning.\" arXiv preprint arXiv:2111.12621 (2021).\n\n[4] Qin, Ziheng, et al. \"InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning.\" The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kyjdlwVXTp", "reviewer": "Reviewer_R3ZA", "summary": "This paper introduces MATES, a method termed \"model-aware data selection with data influence models\". MATES is designed to optimize data selection for large language model (LLM) pre-training efficiently. The method dynamically considers various influence models during different stages of pre-training, tailored to the current state of the training model. Experiments validating this approach have been conducted using the C4 dataset with Pythia models.", "questions": "- For results presented in Table 1 and Figure 3, what is the amount of sampled data used for computing $I_m$? How to make the computation of $I_m$ efficient, considering that for each sampled data, one pass of the Adam and recomputation of the reference task loss are required? \n- In relation to Figure 5(a), how is LASSO regression employed to \"separate\" influence scores? Does this imply that data points within the same batch can have varying influences?\n- For Figures 4 and 6, what is implied by \"Val. Spearman\"? Was the influence computed for the entire dataset?", "limitations": "Besides the limitations discussed by the authors themselves, an additional limitation of MATES is that it uses a single pass of the Adam to approximate the change in loss. This approach assumes that each data point will be trained exactly once. While this assumption might be valid for certain pretraining scenarios, it does not hold for fine-tuning, where all data are used multiple times.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper reveals an important observation that the influence of data changes throughout the training process. It underscores the necessity for model-aware data selection methods to adapt accordingly.\n- The utilization of the Gumbel-Top-k algorithm is particularly interesting and well-suited for balancing data quality and diversity. This diversity is crucial as it helps mitigate the independent influence assumption of the training data.\n- Extensive experiments have been conducted to validate the method's effectiveness.", "weaknesses": "- **Gap between Motivation and Experiment Design**: While the authors have compared MATES with multiple baselines concerning data selection, there remains a gap regarding the necessity of data selection for pretraining as opposed to full training or no continuous pretraining.\n  - **Full Training**: The results and computational costs of pretraining the entire C4 dataset have not been demonstrated. If the performance significantly drops or the computational cost reduction is minimal, then data selection may not be necessary initially.\n  - **No Continuous Pretraining**: While MATES focuses on pretraining, the chosen model, Pythia, is already pretrained on the Pile dataset (i.e., pretraining a pretrained model). If Pythia already performs adequately on those tasks, then data selection might not be necessary initially.\n\n- **Significance of the Results**: The results presented in Table 1 are not significantly robust. For instance, for Pythia-1B, only 5 out of 18 tasks show improvements greater than one **standard error**, suggesting that the p-values are likely higher than 0.05. In some recent similar works [1], the improvement over random selection can be up to seven times the **standard deviation**. Given that MATES also adds computational costs for influence computation, the significance of these results is arguable.\n\n- **Missing Implementation Details**: Some critical details about the implementation are not sufficiently detailed, making the presentation somehow confusing. See the Question section.\n\n[1] LESS: Selecting Influential Data for Targeted Instruction Tuning, ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper! We will address your questions/comments below:\n\n**Weakness 1:** Comparison with no continuous pretraining and full training\n\n**Response:** We first want to clarify that we only followed the Pythia architecture but **didn’t use any of its pretrained weights.** All the pretraining models in the paper are trained from scratch using C4. We chose C4 instead of the Pile since pretraining models on C4 can outperform the Pile, which has also been verified in the recent FineWeb paper. We will make this setup clearer in the next version.\n\nFor the full training, we followed the original Pythia paper and pretrain the models using 150k steps. As shown in Figure 1 in our supplementary pdf, MATES can improve the pretraining efficiency by more than 3x, given that 50k MATES performance is comparable to or higher than the 150k full data training. Please see the general response 3.1 for more details.\n\n**Weakness 2:** Significance of the results\n\n**Response:** We hope to clarify that the **standard error** in Table 1 is computed across the accuracy of all the evaluation examples, **not the standard deviation** across different random seeds. Actually, this value is the “stderr” returned by the `lm-evaluation-harness`. It only shows the **variability** of the model’s accuracy across all the examples, **not the stability of each method**. We are aware that using $\\pm$ to show this metric is inappropriate and will replace it with parentheses in the next version.\n\nLESS computes the standard deviation using different training seeds, while pretraining with different seeds is not feasible with our available computing resources. Our main baselines, DsDm and QuRating, also didn’t report the standard deviation across different seeds.\n\nWe also notice that LESS and QuRating are papers from the same research group, while QuRating is mainly designed for pretraining data selection and LESS is for instruction tuning data selection. Generally, the average performance gain in pretraining is smaller than that in instruction tuning since pretrained models are typically evaluated on a broader set of benchmarks. Table 1 (both in our original paper and supplementary pdf) demonstrates that MATES **doubles the gain** achieved by the state-of-the-art method, QuRating, with fewer total FLOPs, so our improvement is really significant.\n\nSince LESS requires the computation of all training data’s gradients to perform the selection, our compute resources only allow us to run it at the short 40k decay stage (the same setup as Table 2). The results are shown below:\n\n|    | SciQ   | ARC-E  | ARC-C  | LogiQA  | OBQA   | BoolQ  | HellaSwag | PIQA   | WinoGrande | Average |\n| ----- | -------- | -------- | -------- | -------- | -------- | -------- | --------- | -------- | ---------- | -------- |\n| LESS | 63.0   | 41.3   | 23.6   | 24.0   | **28.8** | 54.6   | **42.1** | 64.4   | 51.3    | 43.7   |\n| MATES | **67.3** | **41.7** | **24.7** | **26.9** | **28.8** | **59.6** | 40.1   | **67.6** | **52.1**  | **45.4** |\n\nThis shows that directly applying instruction data selection methods may not work well in the pretraining setup, suggesting that pretraining and instruction selections are two different research scenarios.\n\n**Question 1:** The amount of sampled data used for computing oracle scores and the way to make the computation efficient\n\n**Response:** The sampled data for computing oracle scores is 80k in the first round and 20k in the following rounds since we continuously fine-tune the data influence model. Even with the 1B model, the wall clock time to compute one oracle score is only 2.5s on one GPU, which means we can get all the required oracle scores during 50k-step pretraining on one node (8 GPUs) **around 12 hours**, which is significantly lower than the actual pretraining time (**4 days+**).\n\nTo further make the influence computation efficient, we fine-tune a small data influence model to learn the oracle scores and predict the data influence. Even with the additional data selection costs, MATES greatly elevates the scaling curves (Figure 3) w.r.t. the total FLOPs, making this extra computation justifiable.\n\n**Question 2:** How is LASSO regression employed to separate influence scores? Does this imply that data points within the same batch can have varying influences?\n\n**Response:** Yes, data points within the same batch can have varying influences. We sample different batches to calculate the influence scores, and one data point can appear on multiple batches. Formally, the input of the regressor is a $m$-dimensional binary vector **$v: [v_1, v_2, ..., v_m]$** vector ($m$ means the total probing data pool, as mentioned in Algorithm 1), where $v_i=1$ denotes the inclusion of the $i$-th data in the current batch and vice versa; the output is the **oracle data influence score probed by one-batch one-step training**. We probed 80k input-output pairs that can be directly utilized to train the LASSO regressor, and we can get a $m$-dimensional individual data influence vector from the regressor’s weights. This method is inspired by the linear datamodels in DsDm [3] to separate the individual data influence from the collection of batch data influences. We will add more details about LASSO in the next version.\n\n**Question 3:** What is implied by \"Val. Spearman\"? Was the influence computed for the entire dataset?\n\n**Response:** The oracle influence is only computed on a tiny subset of 80k/20k examples (general response 2.2), not the entire. For the “Val. Spearman”, please see the general response 2.3 for more details.\n\n**Limitation 1:** Assumption that each data point will be trained exactly once does not apply for the fine-tuning.\n\n**Response:** Thank you for pointing out this. We assumed that the pretraining data is infinite w.r.t. current computing resources (line 26-27) and thus unique from other data selection scenarios, while most of the fine-tuning scenarios only have limited supervised examples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## 0 Overview

First, we thank all the reviewers for their great efforts and insightful feedback.

In this post, we summarize positive points from the reviews, clarify the experimental setup, and address the shared questions proposed by the reviewers with additional experiments to support and strengthen our findings.

## 1 Positive points

We sincerely thank all the reviewers for recognizing our key contributions. We appreciate the acknowledgment of our novel data selection framework MATES, which learns a smaller data influence model that co-evolves with the main model’s data preference to perform iterative, model-state dependent data selection ( `2Zbt`, `xrwH`). Furthermore, we are grateful for their recognition of our method’s superior performance: MATES underscores the necessity for model-aware data selection in pretraining, outperforming static baselines with fewer GPU FLOPs (`2Zbt`, `xrwH`, `T2sB`, `R3ZA`). We are also pleased that our extensive and varied experiments and ablations were noted for validating the method’s effectiveness (`2Zbt`, `R3ZA`).

## 2 Clarification of the setup

We summarize the most important clarifications and discuss each in detail in the following points:

1. **The pretraining dataset used in this work (`2Zbt` and `R3ZA`):** We use the **C4** dataset only to pretrain all our main models from **scratch.**

1. **The training data of the data influence model (`xrwH` and `R3ZA`):** The oracle data influence score is defined as **the difference in the target loss** **before and after one-step training** (i.e., $\mathcal{L}(\mathcal{D_t} \mid \mathcal{A}(\mathcal{M}, x\mid_{i \to -1}))$ and $\mathcal{L}(\mathcal{D_t} \mid \mathcal{A}(\mathcal{M}, x\mid_{i \to +1}))$ in Equation 4) using the Adam optimizer on a hold-out example $x_i$. Then, we construct the mapping of the data point to its influence score $(x_i, \mathcal{I}_\mathcal{M}(x_i;\mathcal{D}_t))$ (line 148) as the training data for our BERT-based data influence model. The size of sampled data for computing data influence scores (i.e., $m$ in Algorithm 1) is **80k** at 10k steps and **20k** at 20k, 30k, 40k, 50k steps.

1. **The evaluation of the data influence model (`xrwH` and `R3ZA`):** We evaluate our data influence model’s performance by its prediction’s Spearman correlation with the oracle data influence on the **10% hold-out validation set** sampled from the collected data influence training set {$(x_i, \mathcal{I}_\mathcal{M}(x_i;\mathcal{D}_t)) \mid x_i \in S_m$} (Algorithm 1). Spearman correlation is an intermediate metric and a higher correlation denotes the better learning of our data influence models that approximate the oracle scores (Figure 4a). Our ultimate evaluation of the data influence model is still the downstream performance of the main model pretrained with the selected data (Figure 4b).

We will clarify all of those in the next version of the paper.

## 3 Additional experiments

We summarize observations of the additional experiments we have provided in the supplementary pdf below:

1. **Pythia-410M/1B full data training run (`2Zbt`, `T2sB`, and `R3ZA`), in Figure 1:** We provide the full data training experiments (following the original Pythia pretraining steps of 150k) of Pythia-410M/1B in Figure 1. For the full data training curve, we notice that the average downstream performance increases very slowly, while MATES significantly elevates the model performance in the early pretraining steps. MATES can improve the pretraining efficiency by more than 3x, given that 50k MATES performance is comparable to or higher than the 150k full data training. 
2. **All baselines in the Pythia-1B (`T2sB`), in Table 1:** We are able to secure more compute resources to run all the 1B baseline results and present those in Table 1. Among all the baselines, MATES still performs the best in both zero-shot and two-shot performances. We would like to highlight that zero-shot evaluation is the more important and robust one to reflect a model's pretraining abilities, while two-shot evaluation depends on the in-context examples used, which can be less stable.
4. **MATES on FineWeb [1] with LAMBADA or FLAN as target tasks (`2Zbt`), in Figure 2:** To verify MATES’s effectiveness on a more advanced dataset, we use LAMBADA or FLAN [2] (the mixture of multiple NLP tasks) as the target dataset to run MATES on FineWeb. FineWeb is the state-of-the-art open-source pretraining dataset released by HuggingFace recently. It is processed with complicated and delicate data curation steps. We also include two new baselines in the FineWeb comparison: one is the FineWeb-Edu classifier that is similar to QuRating and the LLM quality filter in LLama3.1, using educational scores generated by LLama3-70B-Instruct; the other is fasttext-oh-eli5 classifier that mines instruction-like data from DataComp corpus and achieves the best selection results in the DataComp paper [3]. As shown in Figure 2a, MATES demonstrates superior average downstream performance on FineWeb compared to these state-of-the-art data selection methods. Using multi-task FLAN as the target can make MATES generalize better to downstream tasks than using LAMBADA only. In contrast, the FineWeb-Edu classifier overfits the knowledge QA tasks (Figure 2c) but underperforms in commonsense tasks (Figure 2d), as the scores it assigns to data are influenced by human-written criteria that may disproportionately favor certain downstream capabilities.

We will add and analyze all of those in the next version of the paper.

[1]: Penedo, Guilherme, et al. "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale." arXiv preprint arXiv:2406.17557 (2024).

[2]: Chung, Hyung Won, et al. "Scaling instruction-finetuned language models." Journal of Machine Learning Research 25.70 (2024): 1-53.

[3]: Li, Jeffrey, et al. "DataComp-LM: In search of the next generation of training sets for language models." arXiv preprint arXiv:2406.11794 (2024).

Meta-review: Reviews are somewhat borderline, though certainly leaning on the side of accepting since all lean positive. Reviewers praise the originality of the work, and the findings themselves, as well as aspects of the data/experiments, etc. Arguably somewhat "generic" praise though still uniformly positive. Some issues raised about the writing, evaluation, and various other small things, though these seemed fairly minor.
Decision: Accept (poster)