Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Authors: KaShun SHUM
Yuzhen Huang
Hongjian Zou
dingqi
YiXuan Liao
Xiaoxin Chen
Qian Liu
Junxian He
Venue: ICML 2025 poster (2025)
Keywords: Pretraining Data Selection
Efficient Pretraining
Primary area: deep_learning->large_language_models

Abstract: Language model pretraining involves training on extensive corpora, where data quality plays a pivotal role. In this work, we aim to directly estimate the contribution of data during pretraining and select pretraining data in an efficient manner. Specifically, we draw inspiration from recent findings showing that compression efficiency (i.e., normalized loss) of diverse models on certain text correlates strongly with their downstream performance, when the text domain aligns with the downstream benchmarks (Huang et al., 2024). Building on this observation, we hypothesize that data on which model losses are predictive of downstream abilities also contribute effectively to learning, which shares similar intuition with Thrush et al. (2024). To leverage this insight, we introduce predictive data selection (PreSelect), a lightweight and efficient data selection method that requires training and deploying only a fastText-based scorer. Through comprehensive experiments with 1B and 3B parameter models, we demonstrate that models trained on 30B tokens selected with PreSelect surpass the performance of the vanilla baseline trained on 300B tokens, achieving a 10x reduction in compute requirements. Furthermore, PreSelect significantly outperforms other competitive data selection baselines, such as DCLM and FineWeb-Edu on a scale of 3B models trained on 100B tokens. We open-source our trained data selection scorer along with the curated datasets at https://github.com/hkust-nlp/PreSelect.

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Kashun Shum
Yuzhen Huang"
Hongjian Zou 2 Qi Ding? Yixuan Liao?
Xiaoxin Chen
Qian Liu
Junxian He
Abstract
42
Language model pretraining involves training on extensive corpoaa, where data quality plays piv-
10x faster
40
otal role. In this work, aim to directly estimate the contribution of data during pretraining and se- lect pretraining data in an efficient manner. Specif- ically, we draw inspiration from recent findings showing that compression efficiency (i.c., normal- ized loss) of diverse models on certain text corre- lates strongly with their downstream performance, when the text domain aligns with the downstream benchmarks (Huang et al., 2024). Building on this observation, we hypothesize that data on which model losses are predictive of downstream abili- ties also contribute effectively to learning, which shares similar intuition with Thrush et al. (2024). To leverage this insight, we introduce predictive data selection (PRESELECT), lightweight and efficient data selection method that requires train- ing and deploying only fastText- -asee scorer. Through comprehensive experiments with IB and 3B parameter models, we demonstrate that mod- els trained on 30B tokens selected with PRESE- LECT surpass the performance the vanilla base- line trained 300B tokens, achieving 10x re- duction in compute requirements. Furthermore, PRESELECT significantly outperforms other com- petitive data selection baselines, such as DCLM and Fine Web-Edu on a scale of 3B models trained on 100B tokens. We open-source our trained data selection scorer along with the curated datasets https///gttuh..oo//kkussst
Average Performance
3.1%
38
36
Random Selection
34
FineWeb- Edu
DCLM
PreSelect
32
3
6
15
30
90
150
300
Training Tokens (B)
Figure 1: PRESELECT outperforms random selection by an average of 3.19 % accuracy across 15 downstream bench- marks and achieves 10x reduction in compute requirements on the scale of 1B models using RefinedWeb corpus.
where the existence of low-quality data leads to slow scaling law (Kaplan et al., 2020; Chowdhery et al., 2023). Given the growing available token budget and limited training budget nowadays, data selection has become a standard step in the pre-training stage. This step has played a critical role in enhancing data quality and optimizing training efficiency during the development of various LLMs (Touvron et al., 2023b; DeepSeek-Al AI, 2024). While previous practices of- ten rely on human heuristics such as pre-defined rules and domain classification to filter data (Penedo et al., 2024b; Li et al., 2024a), in this work we aim to directly identify data that could contribute effectively to learning various abilities.
We are inspired by the belief that "compression represents intelligence" and the corresponding empirical finding re- cently (Huang et al., 2024), that the compression efficiency (i.e., the normalized loss)' of series of models on cer- tain raw text, strongly correlates with their downstream benchmark performance, when the text data aligns with the benchmark domains. For example, Huang et al. (2024) observed that model losses on raw GitHub code exhibit an
1. Introduction
Large language model (LLM) pre-training typically requires training on huge data source such web crawl data
'Equal contribution 'HKUST VVivo AI Lab. Correspondence to: Kashun Shum <ksshumab @ cos.e Junxian He <junx- janh @ ukk.
For given model, the lossless compression efficiency, such as bits per character certain data, is equal to the normalized loss that data (Deletang al., 2024; Huang et al., 2024). See $2.1 further details.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Downstream Performance
different models
rank not align
COMMON CRAWL
sample
rank align
Pos Features
Neg Features
Normalized Loss
Normalized Loss
<!M', <33,,33777> 33777>>
<' Register', -3375>
<'MIT', 248..6>
Comments', , >3
positive data
negative data
< <'Math', "taath,,990 ,193.7>
COVID--19 19', 308.99>
"'Cite',84. /144.7>
20200 223.77>
<'1.', (138111 (138.1>
"hippingg' , -219.5>
fi
FastText Bi-gram Classifier
The overall framework of PRESELECT. It first samples subset of pre-training data for computing the predictive strength score of each document ($2.2). Then fastText- basteee--bsed scorer is trained based on the identified positive data and negative data. Finally the fest scorer is trained to enable scalable data selection.
Figure 2:
almost linear correlation with performance on various cod- ing tasks, whereas model losses on Common Crawl data are more indicative of performance on knowledge-intensive tasks. These insights suggest that compression efficiency on certain data is more reflective of model's intelligence than that on other data. Motivated by this, we hypothesize that data on which compression more effectively represents intelligence also facilitates the learning of that respective intelligence more effectively. similar hypothesis "I.IM losses on many pretraining texts are correlated with down- stream benchmark performance" was proposed by Thrush et al. (2024) previously, but the empirical validation was limited and the effectiveness remained inconclusive.
(2024) in several key aspects: (1) While they compute per- plexity correlations for data domains (e.g.. wikipedia.com), our method operates much finer granularity, directly operating and selecting individual documents.? As will demonstrate in $3.4 and $4.3, this finer granularity leads to sigmificant differences in the selected data and, con- sequently, in the final results; (2) Perplexity correlation suggested using a large number of (e.g., around 90) open- sourced models to approximate the correlation. However, we argue that using many open-sourced LLMs brings poten- tial evaluation noises when dealing with different families of the models and causes computation inefficiency. Thus in our design, we employ 6 I.lama models only during data se- lection. (3) They performed very limited experiments with 160M-size model and compared with several baselines on limited benchmarks, while we train models with up to 3B parameters on up 100B tokens and evaluate on 17 diverse benchmarks, a more realistic scale in typical pre- training setting. The updated, concurrent version of Thrush et al. (2024) further added 1B-scale experiments assesed on comprehensive benchmarks, where the results indicate good performance on raw data pools but no improvement on high-quality, well- nell-processes data pools an observation distinct from our empirical results. PRESELECT targets a well de-duplicated, high-quality pre-training corpus which we believe represents realistic setting and shows the effectiveness of selecting data from high-quality data pools
Building on this hypothesis, we first define the predictive strength of a given text to quantify how well the compres- sion efficiency on correlates with the models' ability, then, introduce predicative data selection (PRESELECT), data selection method based on the data's 's predictive strength. Different from previous works that try to quantify the pre- training data influence requiring controlled training (Yu et al., 2024; Engstrom et al., 2024), our approach is ex- tremely lightweight, and only utilizes open pretrained mod- els in the wild without training any deep models in the pro- cess. After collecting a small seed dataset, a fastText (Joulin al., 2017) is trained enable scalable data selection.
While we are motivated independently by Huang et al. (2024). our approach shares similar intuition with Perplexity Correlation (Thrush et al., 2024) and can be viewed under the framework of Perplexity Correlation, which is the first to explore domain-level correlation-base methods data selection. However, this work differs from Thrush et al.
"We note that Thrush al. (2024) preregistered document-leve experiments future tasks, which were not present as we released the first version of this paper. The updated version of Perplexity Correlation added document-leved results concurrent to this work, but the final results very different we will dicuss in $2.3.
2

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
in $3.4.
While Huang al. (2024) use the Pearson correlation co- efficient to measure the relationship between losses and the model's downstream performance, we take a simpli- fied approach by examining how well the rankings of the models based on losses align with their rankings on downstream benchmark scores. Specifically, assume we have series of N open-weight pre-trained language mod- els {M1 [M, ..., ...] alongside their averaged benchmark score {S1 S2 < ... < Sy). Given any document in the pre-training corpus d E D, we can compute each model's normalized losses per character on d, and obtain { C1, C2. CN}. We adopt a simple matching score as the correlation between the ranks of normalized losses and ranks of benchmark scores:
Experimental results show that PRESELECT naturally se- lects high-quality data, achieving an average of 5.3% ab- solute improvement over random selection at IB scale and outperforms current state-of-the-art data selection methods by 2.2% absolutely on tasks across understanding, knowl- edge, math, and code. PRESELECT also show its adaptabil- ity to different model architectures (Llama (Touvron al., 2023a) and Pythia (Biderman et al., 2023)), as well as differ- ent pre-training corpora across different model sizes from 400M to 3B. Compared to training without data selection, PRESELECT reduces training steps by up to 10x as shown in Figure 1. At the meanwhile, by eliminating data with negative effect, PRESELECT achieves superior performance to training on the entire corpora.
S == s= E 1G > C3H/2, ISKNNKJSN
(I)
2. Data Selection with Predictive Strength
where Z N²-N is the normalization factor ensure S [0. Intuitively, Eq. evaluates whether the rank- ing of losses for each pair of models is inversely aligned with their task score rankings (since lower values are better for losses). The score S increases when the rankings are inversely aligned, indicating that the losses on this dataset ef- fectively reflect the models' rankings on downstream scores. A score of implies that the losses on the document can perfectly predict the models' downstream performance rank- ings. Accordingly, refer to the score in Eq. as the document's 's predictive strength.
2
2.1. Preliminaries SeaiiiinrrissRRelationiisi Between Compression and Intelligence
Language modeling can be considered a form of lossless compression (Deletang et al., 2024). From this regard, given a sequence of text tokens {a:] that contain T characters, the average language model loss per character E - -log Pmodet (z; 11: represents the minimal number of bits required to encode one character on aver- age. Therefore, the normalized language loss on given text x represents how efficiently a model can compress æ in a lossless manner. Recently, Huang et al. (2024) empirically verified that the average downstream task scores across vari- ous diverse models strongly correlate with their normalized losses on certain raw text data, even though these models pretrained on distinct data with different tokenizers. For example, they found that normalized losses on Common Crawl correlated strongly with the models' performance on wwwlledee-nttnsive tasks. A similar level of correla- tion was also observed between losses on GitHub code files and performance on code generation tasks. However, such strong correlations are not universally present for instance, the losses on Common Crawl show a weak correlation with tasks that do not rely on factual knowledge.
Beyond considerations of simplicity, we argue that this ranking-buss based correlation is more robust to noise compared to the Pearson correlation used by Huang et al. (2024) in document-level calculation. As in prior, Perplexity Correla- tion (Thrush al., 2024) has explored ranking-based metric to estimate correlation coefficient in domain-level. Loss computation on single document can be sensitive to noisy or anomalous text, especially when the document is short. In such cases, numerical correlation estimation such as Pear- son correlation, which reacts sharply to minor variations in loss values, may not be suitable metric. This instability can, some extent, be mitigated by grouping documents first and selecting based these groups However, such grouping step introduces additional human heuristics into the process that we think may be suboptimal, as we will verify empirically in $3.4.
This insight underscores that not all data are equally reflec- tive of models' abilities through the losses on them. Inspired by this observation, we hypothesize that data on which com- pression or losses more effectively represent models' ability is better suited for learning that respective ability. This per- spective forms the foundation of our proposed approach, which we introduce next.
We propose to select documents with high predictive strength scores, hypothesizing that the data that can be used to predict the model's ability is also the data that con- tributes to learning it. Different from selection strategies based on pre-defined rules such as Weh-FEddEdu (Penedo et al., 2024b) which prioritizes education-related documents, or DCLM (Li et al., 2024a) which selects referring to supervised -iintiuinnnnn data, our method is more principled and bypasses strong human heuristics.
2.2. Data's Predictive Strength
Here, we aim to quantify how effectively the losses on given dataset represent the model's capabilities.
3

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
2.3. The Overall Framework
scorer. Notably, our approach requires only the fastText scorer at deployment, making it highly accessible and scal- able to select data from a large corpus. We refer to our data selection method as PRESELECT, short for predictive data selection.
Our core hypothesis is that the data on which the losses can help predict the performance well is the data that can contribute to training effectively, thus the documents' pre- dictive strength scores as defined in Eq. 1 are our metric to select data. However, computing Eq. for every document is expensive since it requires computing losses of several models on all the data, thus we follow Li et al. (2024a) and only select a small seed set of documents that are later used to train fastText classifier (Joulin et al., 2017). As shown in Figure 2, we first randomly sample a small subset of data from the pre-training corpus, which is kept separate from the data used for later training, to compute the normal- ized losses. Specifically, we pick the 3,000 most frequent domains (e.g., wikipedia.org org) in the corpus and randomly sample 300 examples for each domain for a wide coverage over the pre-training corpora, thus obtaining 900K samples in total. Then we choose the models from the Llama and 2 series (Touvron et al., 2023abb) ranging from 7B to 65B parameters- - in total 6 models are used to compute the predictive strength for the data. We do not choose various models from different families to compute the predictive strength on purpose - in our preliminary experiments, we found that models from various families introduce signif- icant evaluation noises, for example, they are sensitive to prompts and the prompt that is good for one model may not be suitable for another. This issue is particularly salient given that we are evaluating base models that are generally more sensitive to prompts than instruction tuned models. Such noises would dramatically affect the computation of predictive strength, and thus we found that data selected this way could not outperform random data selection baseline in our initial trials. This design differs from Thrush et al. (2024) which employs diverse models to estimate perplexity correlation. Such distinction probably explains why per- plexity correlation fails to improve random data selection baselines significantly on a high-quality, pre-filtered data pool, as shown in the concurrent version of their paper. We run inference of these models the 900K documents to obtain the loss ranks of these models on each document respectively. Next, we obtain the downstream score ranks by directly using the averaged scores on diverse set of 12 benchmarks from Huang et al. (2024). We also explored how performance change when choosing one specific task (e.g. HellaSwag) as the target that elicits a different ranking to averaged score in Appendix A.7.2.
3. Experiments
In this section, first introduce our pre-training corpus ($3.1). Then various pre-training data selection baselines we compared ($3.2) and evaluation settings ($3.3) are presented. In the last part, we will demonstrate our experimental re- sults. Full implementation details and evaluation details are illustrated in Appendix and Appendix C.
3.1. Pre-training Corpus and Models
For a fair comparison and ease of processing, we follow and directly use the large data pool created in Li et al. (2024a). Concretely, this data pool utilized version of RefinedWeb that undergoes processing through resiliparse text extrac- tion (Bevendorff et al., 2018), RefinedWeb's ' heuristic filter- ing (Penedo et al., 2024d) and deduplication using bloom filters (Soldaini et al., 2024), solely filtered from Common Crawl. This large data pool has over 20 trillion tokens in total, which will serve as the source for us to sample smaller data pools to conduct data selection experiments in various scales. We refer to the pool as RefinedWeb in the following sections.
Following the suggested setting in Li et al. (2024a), we ran- domly sample 80 billion, 300 billion, and I trillion tokens as the data selection pool for 400M, IB, and 3B model training respectively and select 10% of the data for training unless otherwise specified, which corresponds to the Chinchilla optimal training data size (Hoffmann et al., 2024). For the models, use Llama architecture (Touvron et al., 2023a) unless otherwise specified. We also apply PRESELECT on the C4 dataset to show the effectiveness of our method on different corpora and to compare with more data selection baselines.
3.2. Baselines
To make a fair comparison, we keep the data pool and train- ing setting of baselines the same as PRESELECT, such as model architecture and training hyper-parameters, while only the data selection part is different. We consider the fol- lowing baselines: (1) Random, which randomly selects the documents; (2) filterinn, which keeps low perpeeelyy documents following CCNet (Wenzek et al., 2020); (3) Fine Web (Penedo et al., 2024b), which scores the educa- tion level of documents with an LLM and then train a small scorer to identify and select educational contents (Penedo et al., 2024b); (4) PPL Correlation (DD) (Thrush al.,
Once we compute the predictive strength scores for all the documents, we identify the positive and negative samples from them to train fastText classifier. Concretely, we select the documents with the highest predictive strength scores as the positive examples, and those with the lowest predictive strength scores as the negative ones. We select around 200K positive and 200K negative examples to train fastText
4

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
2024), which utilizes domain-level correlation estimation and performs domain-level selection. (5) PPL Correlation (DP) (Thrush et al., 2024), which utilizes domain-lever cor- relation estimation and trains a fasttext based on domain- level positive/negative samples to perform page-level selec- tion. As discussed before in $1, these two baselines are the most relevant to PRESELECT while our approach does not rely on human heuristics to group documents and op- erates at the document level directly; and (6) DCLM (Li et al., 2024a), which is the state-of-the-art pretraining data selection method that trains the fastText scorer using su- pervised fine-tuning (SFT) data as the positive data. For FineWeb-Edu and DCLM, we directly use their released BERT and TastText scorer to select documents in our pool. Due to the high cost of running all the basclines across all settings, we mainly include all of them in our setting of IB model training for 30B tokens. (7) ScalingFilter (Li et al., 2024b), which uses the perplexity difference between a large model and small model as the metric to select data, which we compared and discussed how intermediate level models actually help in Appendix A.7.1. In other experi- ment scales, we only compare to DCLM which is superior to other baselines. We provide more details about the model architecture and training aeperpparaeetes in Appendix B.3 and Appendix B.4 respectively.
benchmarks such as GSM8K (Cobbe al., 2021) and HumanEval (Chen al., 2021). Therefore, we follow the advice of Huang et al. (2024) and report the bits per char- acter (BPC) on Math-related and GitHub raw texts, where we directly use their released Math and Code evaluation corpora. For experiments under C4 corpus, we follow MATES's setting and evaluate the zero-shot performance on Arc-Easy, Chehlhhleeeee, SciQ. LogiQA (Liu et al.. 2020), OpenBookQA, HellaSwag, PIQA and WinoGrande The full evaluation details are illustrated in Appendix C.
3.4. Main Results
Comparison with Baselines As shown in Table 1 top part, we verify the effectiveness of our method by con- ducting initial experiments on 400M models. PRESELECT demonstrates remarkable performance, with an average ab- solute improvement of 2.8% over the random selection and 20% gains in Math and Code raw text BPC, which shows a promising trend. Scaling to the IB model, as shown in the middle part of Table 1, the results show that PRESELECT has the best performance on most tasks, outperforming ran- dom selection with an averaged absolute improvement of 5.3% on 7 representative tasks, including significant boosts such as 8.8% on Are-Easy, 8.4% on BBH and 6.7% on SciQ. In greater detail, as shown in the full evaluation table C.2, on an average of 15 benchmarks, we outperform random selec- tion by 3.1 % absolute improvements. FineWeb-Edu shows notable improvements on exam-related tasks such ARC, however, the improvement on other abilities such as un- derstanding is relatively marginal. More importantly, PRE- SELECT outperforms the strongest data selection method DCLM by over 2%, achieving better performance on most tasks. Similar to the 400M scale, PRESELECT consistently shows significant improvements in Math and Code domain with averaged improvement of 19% and 18% on raw text BPC.
In an ablation experiment to validate our approach on the C4 data pool, we also compare with more baselines such as DSIR (Xie et al., 2023) which selects data by importance resampling, DsDm (Engstrom et al., 2024) which uses data- models approximate the relationship between data subset and benchmark, QuRating (Wettig al., 2024) which aims to capture human intuitions and MATES (Yu et al., 2024) which selects data dynamically based on data influence model. These baseline numbers are directly taken from Yu et al. (2024). There are other widely used methods aimed at improving data quality such as data refinement (Zhou et al., 2024). We do not include comparison with such methods as data refinement is orthogonal to data selection.
Compared with Perplexity Correlation (DD)which selects domains based on the correlation of each domain, it only achieves relatively marginal improvements over random se- lection and is significantly underperformed PRESELECT by 4.8% As we will further discuss in 4.3, the removal of many domains leads to significantly decreased diversity, thus, some tasks such as HellaSwag would even have a significant drop. In addition, we compare with Perplexity Correlation (DP) which perform page-level selection using fastText trained with domain-level positive/negative sam- ples. We see that though it has 1.2% improvements over random bascline, still underperformed PRESELECT by 4.1% showing the effectiveness of using example-level cor- relation estimation. This difference is because by operating at the domain level, it may inadvertently include low-quality data from the high-correlation domains while overlooking high-quality data from unselected domains when building
3.3. Evaluation Settings
We evaluate the pre-trained base models on 17 tasks across different general domains which include: MMLU (Hendrycks et al., 2020). Arc-Easy, Are- Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019),
SciQ (Welbl et al., 2017), RTE (Dagan et al., 2005), BBH (Suzgun et al., 2022), LAMBADA (Paperno et al., 2016), OpenBookQA (Mihaylov et al., 2018), RACE- Middle, RACE-High (Lai et al., 2017), MultiRC (Khashabi et al., 2018), WinoGrande (Sakaguchi et al., 2021). We also aim to evaluate the Math and Code domains, however, we found that our models of these scales can only yield negligible performance on widely used problem-solving
5

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Table 1: The experimental results of different data selection baselines ranges from 400M to 3B on the Refined data pool. For case of space, we only show a subset of representative tasks here from each domain, while we include additional results in Appendix C.2. Bold denotes the best. Gray provided for reference only, as is trained on more tokens. Math and Code are measured by bits per character(BPPC) while others use accuracy.
Method
Tokens
ARC-E
ARC-C
MMLU
LAMBADA
RACE
SeiQ
BBH
Avg.
Math (L)
Code
400
Model with 10% selection threshold
Random
33.5
19.3
25.9
13.2
21.5
56.2
24.2
1.224
1.111
DCLM
38.6
21.7
60.2
PRESELECT(OUrS)
41.6
26.0
14,8
23.1
61.2
3.7
27.0
0.995
0.885
IB Model with 10% selection threshold
Random
300B
42.2
27.8
24.5
27.6
22.3
70,9
12.8
31.3
0.892
0.804
Random
39.2
24.4
26.0
19.0
21.9
64.8
7.8
28.1
1.023
0.901
PPL Filtering
42.5
24.6
25.8
18.8
22.6
67.5
8.5
29.1
0.957
0.853
Web-EEu
48.3
26.1
26.0
18.2
24.4
69.0
12.8
31.1
0.906
0,816
PPL. Correlation (DD)
30B
39.7
26.1
20.7
22.8
63.7
28.6
0.919
PPL Correlation (DP)
44.2
24.6
25.2
22.9
65.7
29.3
DCLM
45.2
24.8
26.3
24.3
70.0
12.6
31.2
0.857
0.773
PRESELECT(OUS))
48.0
26.8
26.0
23,5
27.7
71.5
16.2
33.4
0.830
0.744
Model with 30% selection threshold
DCLM
90B
47.6
27.5
26.3
26.2
22.7
74.7
13.3
32.6
0.847
0.771
PRESELECT(OUrS)
90B
49.2
27.5
26.0
27.0
25.0
75.4
17.2
34.0
0.831
0.757
3B Model with 10% selection threshold
Random
51.2
29.2
24.8
33.2
22.5
79.5
15.3
34.7
0.818
0.726
DCLM
100B
55.7
25.3
35.1
26.0
82.5
37.8
0.664
PRESELECT(OUr))
61.2
26.2
36.1
25.8
85.6
23.3
0.694
0.648
the fastText classifier.
3.5. Using C4 as the Data Pool
In the scale the 3B model trained on 100B tokens, as shown the bottom part of Table 1, the results indicate that PRESELECT consistently achieves the best performance across almost all representative tasks compared to other baselines, demonstrating the scalability and effectiveness of PRESELECT. Specifically, similar to the 1B setting, PRESE- LECT outperforms random selection by significant margin in tasks such as Arc-Easy and BBH, with 10% and 8% ab- solute increase respectively. Moreover, PRESELECT still achieves the best performance on Math and Code.
Next, we perform experiments on a different setting to further validate PRESELECT with different data pools and model architectures. Specifically, following the MATES's (Yu et al., 2024) setting, we train 410M and 1B aithia-architecture (Biderman al., 2023) model with 25B tokens as shown in Table 2. We directly include the baseline results from Yu al. (2024). PRESELECT demonstrates an absolute averaged improvement of 2.2% and 2.1% over ran- dom selection at 410M and 1B model respectively. as well as notable improvements over all other data selection base- lines, showing its applicability across different pretraining corpora and model architectures.
Efficient Training with Data Selection Data selection as standard approach to computation, it is also common to see how many training steps can save compared to train- ing without data selection. We directly train the 1B model from scratch with full data size 300B (10x selected data). surprising that PRESELECT with 30B tokens shows superior results to the model trained with 300B tokens, indi- cating 10x reduction in computation requirement. Also, by comparing DCLM-selected data and PRESELECTT-Selccted data at 30% selection ratio (90B tokens), PRESELECT con- sistently show better performance than DCLM. In addition, by comparing DCLM trained with 90B and tokens and PRE- SELECT trained with 30B tokens, DCLM needs 3x more training tokens to show comparable results to PRESELECT, indicating greater efficiency of PRESELECT.
4. Analysis
In this section, we analyze the data selected by various data selection methods to gain insights into the characteristics of the selected data.
4.1. Positive/Negative Data Analysis
As discussed in $2.3, PRESELECT defines positive data as those with high predictive strength scores. What data are selected as positive data and negative data respectively? We observed that literature-relaedd domains contain higher percentage of positive data such literotica and ukessays,
6

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Table 2: The zero-shot performance comparison between data selection baselines and PRESELECT on C4 with Pythia model architecture. All reference results of baselines are obtained directly from (Yu et al., 2024). Bold denotes the best.
Method
SciQ
ARC-E
ARC-C LogiQA
OBQA
HellaSwag
PIQA
WinoGrande
Average
410M Model trained with 25B tokens
Random
64.1
40.2
25.6
24.7
29.4
39.7
67.1
50.6
42.7
DSIR
63.1
39.9
23.8
27.0
28.4
39.6
66.8
51.5
42.5
DsDm
65.4
41.7
24.7
27.5
29.0
40.3
68.1
50.1
43.4
QuRating
64,8
42.0
25.4
25.3
30.2
40.7
67.5
52.1
43.5
MATES
66.0
41.8
25.0
25.7
30.8
41.0
68.7
52.7
44.0
PRESELECT(OUS)
67.9
47.0
26.2
26.4
31.0
40.5
67.3
52.9
44.9
IB Model trained with 25B tokens
Random
65.8
43.7
25.6
27.5
31.8
43.8
68.9
50.7
44.7
DSIR
65.8
42.6
24.7
28.7
29.2
44.2
68.3
53.2
44.6
DsDm
68.2
45.0
26.5
26.6
29.4
44.8
68.9
51.9
45.2
QuRating
67.1
45.5
25.6
26.9
29.8
45.2
70.2
51.6
45.2
MATES
67.3
44,9
25.9
28.7
32.2
45.3
69.5
52.4
45.8
PRESELECT(OUrS)
69.5
50.4
27.4
28.9
32.4
44.8
68.7
52.6
46.8
Table 3: The 1op-15 fastText learned fearures with corresponding influence scores of different methods. Positive and Negative are defined according the magnitude of the influence score.
Method
Top-15 fest Features with Influence Scores
DCLM (pos)
("1, 168.194), (Why, 671.558), (Answer: 620.404), (Additionally, 599.6), (assistant. 567.250 (67.250), (Edit:, 565. 565.787). (Generate, 537.11), (answer., 502.883), (Given, 474.243), (A:, 457.440), (Question;, 447.825), (Write, 440.69), (task., 423.878), (ELIS, 419.001), (AI, 403.29)
DCLM (neg)
(-, 1.7.88888). (Comment, -466.0545). (Posted, -450.19833), (Post. -442.57623), ((...). -386,1609), (Comments, 377.41232), (- -344. 9523), (About, 340.93152), (.., -221,9244, ("The, -321.8486), (Re:, -310.0053), (More, -297.77707), (x. 491434, (See, -289.7501), (2012, 227.6626)
PPL Correlation (DP)(pos)
(Archives. 347.5), (Navigation, 299.7), (Commenting, 288.5), (Section. 280.3), (E-mail, 276.1). (Played:, 263.9), (HB, 249.7), (Lyries, 240.4), (Received, 231.4). (Gamercard, 226.7), (40000, 226.6), (Discussion: 224.3), (Sleeps. 219.0). (Playing: 216.9). (Forgot, 216.3)
PPL Correlation (DP)(pos)
(Login. -459.9), ((permalink). -367.6). ((ANS), -358.0), (Images), -344.0), (macrumors, -343.3), ((eredit:, -320.8). (Shipping, -314.9), (Subscription, -293.5), (Defines, -291.0), (Number: -286.5), (Lainey, -279.2), (Maine, -278.0). (Tutors, -277.6), (Nearby, -269.7), (Bys, 2269.5)
PRESELECT
(3337.7). (Likes, 241.387), (sharclimprove, 241.28), (MIT, 240.614), (-xa0,, 240.006), (Retrieved, 232.997), (Received. 214.872). (Photo:, 808), (Azure, 200.453), (Tolkien, 195.01), (Math, 193.741), (answer, 187.706), (Contributed. 185.518), (Cite, 184.729), (MATLAB, 176.991)
(pos)
PRESELECT
(Register, -375.421), (Details, -385.247), (Comments, -320.821), (Updated: -310.755), (COVID- -303.878,, (tho, -281.664). (Print, -271.024), (Profile, -262.2222 262.205), (Leave, -258.85), (Published:, -235.235), (Product. -228. -228.646). (coron- avirus, -227.227), (2020, -223.704), (By:, -222.939), (Shipping, -219.518)
(neg)
followed by knowledge related domains such as wikipedia, stackexchange and wikihow. While for negative data, the top domains vary widely, encompassing reference materials, commerce sites, genealogy platforms, and others such as abbreviations.com, onegreafaamll.com. citysearch.com
tains two linear transformations input matrix and output matrix, where the input matrix maps all uni-gram features to the hidden dimension and the output matrix maps hidden di- mension to final classification dimension, and our case, 2, we are able to track the influence of individual uni-gram final prediction score (the detailed calculation can be found in $A.3). This helps understand what uni-gram pat- terns the fastText classifier looks for and potentially explains the word-level characteristic of good/bad data.
4.2. Fast Feature Contribution
After selecting the small set of representative positive and negative data, PRESELECT trains a fastText-bant based scorer so that the method can be adapted the large corpus. Because of the simple architecture of fastText classifier which con-
As shown in Table 3, for DCLM, the baseline that also uses fastText classifier, the uni-gram features with high influ-

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Table 4: The selected data distribution (RefinedWeb) over source domains of different data selection baselines. The top-15 domains are listed for comparison and the density are counted based on character percentage.
Method
Top-15 Domains (Percentage Density)
Random
nnwiiipeddaaorg (0.36), issuu.com (0.15), ufdc.ufl.edu (0.13), www. www.thegurrdian.com (0.12), www.fanfiction.net (0. (0.)
www.nifty.org (0.11), www.beeradvocate.com (0.10), openjurist.org (0.09). www.nytimes.com (0.08). bleacherre-
port.com (0.08). www.brmesandnoble.com (0.06), www.washingtoppsst.omm (0.06). www.huffigtooppsst.com (0.06).
www.literotica.com (0.06), www.nbenews.com (0.06)
FineWeb-Edu
en. wi.iipppdia.org (0.85), en.wikisource..rg (0.12), www.reference.com (0.10). www.mdpi.com (0.09).
www.britamnicaccom (0.09), en.m. wikipedia org (0.08), phys.org (0.08), www.enotes.com (0.08), www.ncbi.nlm.nhh gov
(0.07), www.sacred-texk. temts.com (0.07), www.slideshare.net (0.07), bbooks.dellidd..du.. (0.07), sacred-texts.com (0.07).
www.ukessays.com (0.06), www. www.wisegeek.com (0.06)
PPL Correlation
www. www.theguardian.com (1.12), issuu.com (1.11), www.nifty.org (0.90), www.fanfiction.ne (0.83), openjurist.org
(DD)
(0.68), www.beeradvocate.com com (0.68), www.nytimes.om com (0.65), www.literotica.com (0.58), www. www.washing www.waahinggoopss.coom
(0.54). www.slideshare.nee net (0,54), www.mdpi.com (0.48), archive.org (0.45). www.businessinsider.com (0.43).
www.harnesandnohle.omm (0.43), www.forhes.co com (0.43)
PPL. Correlation
(ww..aafictiion.ee 0.36), (www.opentable.com, 0.32), www.wnebi.noo www.nebbiil..ii...... gov, 0.30), (www.alldiscountbook... 0.29),
(DP)
(www.hindawi.com 0.26), (pubmeddentrallannadacco ca, 0.26), wwww.ockpapershooguu.co com, 0.23), (www.iteerotia.ccom 0.22), (link.springerr.com 0.20), (politicaltickerrbbll con.com, 0.20), (sixstat.com, 0.19), (openjurist.org, 0.19),
(www.homaaway.com, 17), (www.nifty.org. 0.17), (elinicaltrials.gov, 0.16)
DCLM
www.physiesffrums.com (0.59), nath.stackexchange.coo (0.56), www.reddit.com (0.53), slashdot.org (0.47), stackover- flow.com (0.46), whysses.stackexchaaeee
(0.46), wn.iiiipedi..org (0.44), ovtropes.org (0.35), www.yycombinator.co (0.31). quizlet.com (0.27), everything2.com (0.27), programmers. stackexchunge.comm (0.24), www.scribd.com (0.23).
www.reference.ccom (0.23), www.freszingblue.com (0.23)
PRESELECT
on.wikipedia.org (3.1 www.fanfiction.ne (0.74), www.nifty.org (0.69), www.litcrotica.com (0.45), stackovertlow.com
(0.38), en.m. wt..w.ikipedia.org (0.35), Ivtropes.org (0.27), www.sexstories.com com (0.25), slashdot.org (0.21), wn.wikisource.org
(0.21). archiveofourown.org (0.20), sacred-texts.com (0.20), www.sacred-texts.om (0.19). whwwhound.chow.ccom (0.17).
lists.w3.org (0.16)
ence scores include (Why, 672), (Answer:, 620), (assistant., 567), (Question:, 448), (Describe, 368)... It is clearly ob- served that most positive uni-gram features have supervised fine-tuning data patterns and if the data in the pretraining corpus has such uni-gram, they have a high probability to be selected. While for PRESELECT, the uni- ani-gram features with high influence scores include (", 338), (MIT, 241), (Math, 194), (answer; 188), which widely covers many features ap- peared in code, education, and question answering domains. For example, ..... exists in many code snippets, "MIT" and "Math" exist in many education-related examples. These rep- resentative uni-gram features does not show in domain-level correlation estimation method, Perplexity Correlation (DP), where various diverse meaningless features are learned. We attribute this to the noise in the demain-level correlation cal- culation, where inadvertently include low-quality data from the nigh-correlation domains while overlooking high-quality data from unselected domains. The full learned positive and negative uni-grams alongside their influence score can be found in appendix A.3.2 which provides more insights into what word patterns are preferred by different methods.
lines? We begin by analyzing the data distribution according to the source domain of selected data. As shown in Table 4, we list top-15 source domains with their percentage density in terms of character number in descending order. We list more domains in Appendix A.S.
For domain distribution of the original corpus where we just randomly sampled a subset, wikipedia.org org takes the highest percentage and followed by general utility domains such as news (theguardian, nytimes, washingtonpostt..) and litera- ture/digital libraries (fanfiction, niify, literotica...), which spreads quite evenly.
Fine Web-Edu aims to select education-relate contents which prioritize many dduuctionnrelatee domains such as wikisource.org britannica.com phys.org, ncbi.nlm.nih.ge while down-sampled those general utility domains. This makes FineWeb-Eda achieve great performance on examination--elatte benchmarks such as ARC, hurting the performance on understanding such as LAMBADA. DCLM aims to select SIT-like data and is clearly reflected in the selected data, where large number of question-ansse answer re- lated domains are up-sampled such as physicsforums.com stackexchange.com reddit.com, stackoverflow.com. These SFT-like data brings DCLM significant advantages over random-sampled baseline on many downstream tasks. Per- plexity correlation (DD) selects domains based on domain- level correlation, from the highest correlation domains until filling up token budgets. This means will discard all the
4.3. Distribution of Selected Data
After showing what kind of documents PRESELECT tends to look for in the corpora, we would like to know (1) what data PRESELECT actually select? and (2) How PRESELECC selected data different from the data selected by other base-
8

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
DCLM: 2462.8
Edu show a significant skew toward short data and long data, with average length of 2500 and 5100 characters respectively, PRESELECT selects broader range of data with an average length of 4000, mitigating the risk of bias towards specific document lengths.
0.025
Random: 3489.3
PreSelect: 4039.9
0.020
FineWeb-Ede Edu: 5088.3
Density
0.015
5. Related Work
0.010
Rule-based Data Selection Rule-based methods typically employ human-crafeen heuristic filters to select data. For example, the C4 pipeline (Raffel et al., 2020) and Gopher rules (Rae et al., 2021) filtered documents by their document length, mean word length, and URL domains. More recently, RefinedWeb (Penedo et al., 2024d) and Web (Penedo et al., 2024a) brought more heuristic quality filters such as character repetition, the fraction of lines ending with punc- tuation, etc. While straightforward and computationally efficient, these approaches often fail to generalize across diverse domains and contain strong human bias.
0.005
0.000
10
10'
# characters
Random
FnneWebeedu
DCLM
PreSelect
Figure Length distribution of different data selection base- lines on RefinedWeb measured by the number of characters. The length annotations are averaged characters.
remaining domains with relatively medium/low correlations once sufficient training tokens are obtained (e.g. only 1500 domains are selected to fill up the required training tokens). As shown in Table 4 row 4, many high-quality domains such as wikipedia.org org may be dropped according to medium cor- relation and automatically upsampled each single selected domain, such as theguardian.com weights from 0. 0.12% in original data distribution to 1.12% in perplexity correla- tion (DD). This significantly hurt the data diversity and quality, even leading to a performance drop on HellaSwag, Are-Challenge compared to random selection, as shown in Appendix Table 12.
Model-based Data Selection In contrast to ale-based ap- proaches, model- data selection leverages trained mod- els to score the data. For instance, CC Net retained the text that is assigned low perplexity by a language model trained on Wikipedia (Wenzek et al., 2020), Marion et al. (2023) and Ankner et al. (2024) also use perplexity and effective metric for data pruning, Win--WebEdu trained BERT classifier to prioritize educational contents (Penedo et al., 2024b), DsDm and MATES employed data influ- ence models to assess the impact of specific data points on pretraining (Engstrom et al., 2024; Yu et al., 2024). And recently, DCLM (Li et al., 2024a) also uses a festText-based scorer which selects referring supervised- supervised-fmettuinng data, These methods offer stronger performance than rule-based methods, however, they either require heavy computation or still contain strong human heuristics. Different from them, PRESELECT is more principled and lightweight, thus by- passing strong human heuristics and offering adaptability.
practical
Perplexity correlation (DP) utilizes domain-level correla- tion estimation and train a fasttext based on domain-level positive/negative samples to perform page-level selection. This mitigate the problems in Perplexity correlation (DD) to some extent, but suffers from the noise inside each domains we discussed in $4.2. This also reflected in final selected data where the learned features fail to upsample high-quality domains as shown in Table 4.
Different from them, our method tends to select data that cor- relates with the downstream abilities while broadly contain- ing previously mentioned domains such wikipedia. com, fanfiction. net, stackoverflow.com and wikisowrce.org. In addition, our method operates at the document level, which allows for fine-gre fine-grained filtering of content within each do- main and may achieve higher diversity and better quality.
6. Conclusion
In this work, we introduce PRESELECT, novel predic- tive data selection tailored for language model pretrain- ing that leverages the predictive strength of the data (i.e. how effectively the normalized losses on the data repre- sent the model's capabilities) identify high-quality and low-quality data. By operating at the document level, PRES- ELECT significantly improves granularity and adaptability in data selection. Experimental results across multiple model scales validate the effectiveness of PRESELECT, demon- strating substantial improvements over random selection and recent pretraining data selection basclines in both down- stream task performance and computational efficiency.
4.4. Length Distribution of Selected Data
The data preference difference is also reflected in the length distribution of different methods as shown in Figure 3. The candom-samples data, marked in red, shows a reference distribution of the whole corpus with an average character number of around 3500. While both DCLM and Fine Web-

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Acknowledgments
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko,
This project is partially supported by NSFC Grant 62306177.
S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer,
N. Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L.., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph. B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. Pel- lat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang. X., Saeta, B., Diaz, M., Firar, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. URL http: hhmll.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
AI,
https:///gttiee com/ Lightning-AT/litepa 2023. 2023.
Litgpt.
Ankner, Z.,, Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-basee data pruning with small reference mod- els. 2024. URL https:///// aaaiiiiiiivv...e http://awwwi.../aa////////s/o 20541.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.00 803.00457. 2018.
Bevendorff, J., Stein, B., Hagen, M., and Potthast, M. Elas- tic ChatNoir: Search Engine for the ClueWeb and the Common Crawl. In Azzopardi, L., Hanbury, A., Pasi, G., and Piwowarski, B. (eds.), Advances Information Retrieval. 40th European Conference on IR Research (ECIR 2018), Lecture Notes in Computer Science, Berlin Heidelberg New York, March 2018. Springer.
Cobbe, K., Hilton, J., and Schulman, J. Training verifiers solve math word problems. ArXiv, abs/2110.14168. 2021.
Contributors, O. Opencompass: universal evaluation platform for foundation models. https: petps::/githiin /tgtttb.ttt..ttg com/ open- htt///pennccssssee htcp-/comaaas//aae http:/asss//ppccccpppasss 2023.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp- 2397-2430. PMLR, 2023.
Dagan, I., Glickman, O., and Magnini, B. The pascal recog- nising textual entailment challenge. In Machine Learning Challenges Workshop. 2005.
DeepSeek-AL. Deepseek 1lm: Scaling open-source
language models with longtermism. arXiv preprint ftti:240010295444 2024. URL https: httt://gttttttt /githbb. com/ con/deepseek-aae ai DDeepSeeek LIM.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. AAAI, 2020.
Deletang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Ge- newein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression. In The Twelfth International Conference on Learning Representations, 2024. URL https: https::/ /oppreevieeeeeeeeeeeeeeeeee net/ /orum? Ad=jznbgii inw.iiggiiiuu.... ynus.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M.. Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N.. Tang, J. Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J.. Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M.. Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code.
Engstrom, L.,, Feldmann, A., and Madry, A. Dsdm: Model- aware dataset selection with datamodels, 2024. URL https: hwwp:axxvvvvvoo/.. abs 2401. 12926.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L.,, Hsu, J., Le Noac baach,, A., Li,
H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
ArXiv, 2021.
J., Reynolds, L.,, Schoelkopf, H., Skowron, A., Sutawika,
10

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: /ttp::://eeooooo/ wttp://orrrr/rreeeeeoooooooo 60866...
Toshev, A., Wang, S., Groeneveld, D., Soldaini, L., Koh, P. W. Jitsev, J., Kollar, T., Dimakis, A. G., Carmon, Y., Dave, A., Schmidt, L., and Shankar, V. Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406. / 1794, 2024a.
Hendrycks, D., Basart, S., Mazeika, M., Zou, A., and Stein- hardt, J. Measuring massive multitask language under- standing. ArXiv, abs/2009.0330000 03300, 2020.
Li, R., Wei, Y., Zhang, M., Yu, N.. Hu, H., and Peng. H. ScalingFilter: Assessing data quality through in- verse utilization of scaling laws. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 3209-3222 Miami, Florida, USA, November 2024b. Association for Computational Linguistics. URL https: / /ac//acaaaccoa aaaaaaaanoollooooooooooooooo org/ 2024 4. -mn.ppmmainneee main.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L. Training compute-optimal large lan- guage models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781 0781713877 1781713871088.
Liu, J., Cui, I.., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. 2020.
Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadace, M., and Hooker, S. When less is more: Investigating data pruning for pretraining ilms at scale, 2023. URL https: / /tt///awvvvvvv////////ee//e/eavv harg/abb// 2309 04564.
Huang, Y., Zhang, J., Shan, Z., and He, Compression rep- resents intelligence linearly. In First Conference on Lan- guage Modeling, 2024. URL https: /tpps//////eeeewww///// net/ fott/foru??a -id--iddd--aae (HM---HM1884S0
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? new dataset for open book question answering. EMNLP, 2018.
Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the As- sociation for Computational Linguistics: Volume 2, Short Papers, pp. 427-431, Association for Computational Lin- guisties, April 2017.
Paperno, D.,, Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring broad discourse context. ACL, 2016.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- cls, 2020. URL https:://///// arxiv. org/ abs/ 08361.
Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2024a. URL https::////// wpwprevien.eeeeeepeepeeee.ror.we net/ forum? i id=n6sCkn2@ac.
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., and Roth, D. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACI, 2018.
Penedo, G., Kydlíčck, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024b. URL https://araa httss///xxivvorrr 2406 2406.17557.
Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. H. Race: Large-scale reading comprehension dataset from exami- nations. In EMNLP. 2017.
Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., Garg, S., Xin. R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhuriaa, M., Abbas, A., Hsieh, C.-Y., Ghosh, D., Gardner, J., Kilian, M., Zhang, H., Shao, R., Pratt, S., Sanyal, S., Ilaacoo G., Daras, G., Marathe, K., Gokaslan, A., Zhang, J., Chandu, K., Nguyen, T., Vasiljevic, I., Kakade, S., Song. S., Sanghavi, S., Faghri, F., Oh, S., Zettlemoyer, I.,, Lo, K., El-Nouby, A., Pouransari, H.,
Penedo, G., Kydlíček, H., Cappelli, A., Sasko, M., and Wolf, T. Datatrove: large scale data pro- cessing, 2024c. URL https://gitthb. com/ bogginggface/dttatte
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobei- dli, H., Cappelli, A., Pannier, B., Almazrouci, E., and Launay, J. The refinedweb dataset for falcon 1lm: out- performing curated corpora with web data only. In Pro- ceedings of the 37th International Conference on Neural

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024d. Curran Associates Inc.
D., Dodge, J., and Lo, K. Dolma: an open corpus of three trillion tokens for language model pretraining re- search. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15725-15788, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10.186553 840. URL https: / //aclanthologg..rrgg
Rac, J. W., Borgeaud, S., Cai, T., Millican, K., Hoff- mann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbi, J., Dathathri, S., Huang, S., Uesato, J., Mel- lor, J. F. J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S. M., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Pagannii, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Ne- matzadeh, A., Gribovskaya, E.,, Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.- Tsimpoukelli, M., Grig- orev, N. K., Fritz, D., Sottiaux, T.., Pajarskas, M., Pohlen, T., Gong. Z. Toyama, D., de Masson d' Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., John- son, M. G., Hechtman, B. A., Weidinger, L., Gabriel, I. Isaac, W. S., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K. W., Stanway, J., Ben- nett, I.. I.,, Hassabis, D., Kavukcuoglu, K., and Irving, G. Scaling language models: Methods, analysis & in- sights from training gopher. ArXiv, abs/2112.11446, 2021. URL https:/ api. www.nt #tg/CorpuSID:2244 245353475.
acl- 840/.
Suzgun, M., Scales, N., Schärli, N., Tay, Y., Gehrmann, S., Tran, V. Q., Chen, X., Freitag, M., Chung, H. W., Fedus, W., Wang, X., Wei, J., Bommasani, R., Zhou, D., Liang, P., Le, Q. V., and Chi, E. H. Challenging big-bench tasks and whether Mhain-of-though can solve them. ArXiv, abs/2210.09261, 2022.
Thrush, T., Potts. C., and Hashimoto, T. Improving pre- training data using perplexity correlations, 2024. URL https: / /awxxvivvv//////////////////a/
http://wwxxx....a///////// wttp:bs//////..//eaae 05816.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a. URL https: aaaaxiivvorr abs/ 2302. 13971.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaei, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y.. Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E.,, Tang, B., Taylor, R., Williams, A., Kuan, J. X.,
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), January 2020. ISSN 1532-4435.
Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, 2021.
Sap, M., Rashkin, H., Chen, D.. Bras, R. L., and Choi, Y. Socialiqa: Commonsense reasoning about social interac- tions. In EMNLP. 2019.
Xu, P., Yan, Z., Zarov, L., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https: http///axivvv.o// abs /2307 09288.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J. and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model par- allelism, 2020. URL 1909.08055 08053.
abs/
Welbl, J., Liu, N. F,, and Gardner, M. Crowdsourcing multiple choice science questions. In Workshop on Noisy User- geerrgenerated Text, 2017.
Soldaini, L.. Kinney, R., Bhagia, A., Schwenk, D., Atkin- son, D., Authur, R., Bogin, B., Chandu, K., Dumas, J. Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M., Ravichander, A., Richardson, K., Shen, 2., Strubell, E., Subramani, N., Tafjord, O., Walsh, E., Zettlemoyer,
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. CCNet: Ex- tracting high quality monolingual datasets from web crawl data. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A.,
L., Smith, N., Hajishirzi, H., Beltagy, I., Groeneveld,
12

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Odijk, J., and Piperidis. S. (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4012, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546- 34-.. URL https::/p//.. : / /acl anthhlohhllly org/2020. 1rec-1.494/
Wettig, A., Gupta, A., Malik, S., and Chen, D. QuRating: Selecting high-quality data for training language mod- els. In International Conference on Machine Learning (ICML), 2024.
Xie, S. M., Santurkar, S., Ma, T., and Liang, P. Data lection for language models via importance resampling. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. httpp::::/ httpt:ttttppspseeeeeeeee ppppreveen.e.eee net bet/fozze utt.:errmmidiusSS00Soeee uPVoeleeu...
Yu, 7 Das, S., and Xiong, C. MATES: Model-aware data selection for efficient pretraining with data influence mod- els. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https: //openreview net/ forum? 1d=6g2
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In ACL, 2019.
Zhang, G., Qu, S., Liu, J., Zhang, C., Lin, C., Yu, C. L., Pan, D., Cheng, E., Liu. J.. Lin, Q., Yuan, R., Zheng, T., Pang. W., Du, X., Liang. Y., Ma, Y., Li, Y., Ma, Z., Lin, B., Benetos, E., Yang. H., Zhou, J., Ma, K., Liu, M., Niu, M., Wang, N., Que, Q.. Liu, R., Liu, S., Guo, S., Gao, S., Zhou, W., Zhang, X., Zhou, Y., Wang, Y., Bai, Y., Zhang, Y., Zhang, Y., Wang. Z., Yang, Z., Zhao, Z., Zhang, J., Ouyang, W., Huang, W., and Chen, W. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv: 2405. 19327, 2024a.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model, 2024b.
Zhou, F., Wang, Z., Liu, Q. Li, J., and Liu, P. Program- ming every example: Lifting pre-training data quality like experts at scale. arXiv preprint arXiv:2409. 15, 2024.
13

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
A. Filtering Details
This section serves as an extension of $ 2, which introduce the detail implementation our compression- -based data selection and fastText classifier training, and pretrianing data filtering.
A.1. Identified Positive/Negative Data
We first sampled a small subset data which also from pretraining corpus data distribution but not from the data pool for later model training. We counted the 3,000 most frequent domains in the pretraining corpus (e.g. wikipedia. org) and randomly sampled 300 documents for each domain which result 900,000 documents in total. We then choose series open-sourced models to compute compression efficiency. Here choose Llama-1-7B Llama-1-13hh Llama- 1-30B, Llama-1-65B, Llama-2-7B, Llama-2-13B which cover wide range in terms of model size. From Huang al. (2024), we already know that the following order holds in terms of averaged downstream performance:
Ilama- 1-65B > Llama-1-300 > Llama-2-13B Llama-1-13 1-13B > Llama-2-7B > Llama-1-77
So next step, for each data, calculate the compression efficiency for each model. This means each data document has corresponding 6 compression efficiency. We than score each document according to the matching between compression efficiency and downstream task performance by Eq. 1. The score distribution shown in Figure below.
Density
0.15
0.30
0.05
0.00
0.00
0.07 0.13 0.20 0.27 0.33 0.40 0.47 0.53 0.60 0.67 0.73 0.80 0.87 0.93 1.00
Matcing Score
Figure 4: The matching score distribution on RefinedWeb subset where reflect the alignment between compression efficiency and averaged downstream performance. The higher the better.
We then select examples with score as the positive(good) data and select negative(bad) data gradually from score as dicussed in $ 2.3. We have already give discussion about what kind of data are these positive and negative data in $ 4.1, here we list more domains (e.g. top-30 domains) these positive/negative documents from Figure and Figure respectively.
14

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
100
Percentage (%)
60
40
www.secstoviessc con
worre www.dita.agg dirtuarg
www. www.iieerttic..comm cor
wrine uf eesaysssssss.... cam
wwwtt thetiggtttt org
www.sdesrtars not
slddep da cor
wrchvenfouurwnn org
www. www.utiikkioooooomm org
programmersscom www.ss.sscceecc.... www.sttcceccchaagcooo cor
ent www.ppepiiinnnoommmm org
www.thettgeee. com
www. frezzingggiggn cor
www.wikkioo de corn
secunity secunty Souuttystackkccharr waackkchaanggg.... corn
www www. troughtso comn
bloge discov discovemaapa wwwooveeaagaainnn com
www. www.rgereberf.com cm
wwwe sacredteeettt com
www.sspperchere com com
bletko.com com
en-wikipedia org
mikivisualle com
sacred comn
were cram. com
ufac uff eduu
wwwe pojibajjjbb..oo con
wera cotor com
us us.stackectaangeoo com
www.zerohesig.cc
Source Demains
Figure 5: The Top-30 domains where the identified positive documents from, measured by percentage of positive documents inside that domain.
100
Percentage (%)
www. sw cstonies.coo ccm
www Hetorttic.coom com
www.kkssan www.ukssays.comm com
www. www.tthhhhhhaa...oo org
www.slleeshar...o Furret
ulldeplayer.com com
archieer www.oooooooroooooo org
arch www. www.thefulliii..oomm org
www.zerohesge www.eerrreeege.coom com
programmets sta.keddceeceeeccaccc eecherrccaeeccccc camaage.com com
en www.wittpediab.oo og
www.thetsptterec can
wwwe freazingblus blueeee...o con
wore www.wwkkhho..ccom nowcccm
secuuiity sta.ktystackeechheec trangeg.... comn
www. www.shhuutht... www.hhuuhhhooooomm com
boogs www.overaaaaggooo petecvvemmaaaiinne wwm.amamnn..c.m..
www. www.trogerebee www.r/ggebbbrrr.oo
www.sacceee-eee www.sssseeeeeeeeet..oooo
www www.rheeecceyyccc
bledro comm
en wikipedia org
wikivisuully com
sncred text..co comm
www. www.cram..com com
urdc-ull edu
www www.pajta.com com
www. dicsttccccom com
us uu.stacreeccaang comm
Figure 6: The Top-30 domains where the identified negative documents from, measured by percentage of negative documents inside that domain.
15

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
A.2. Fast Training Details
We the official fastText python library' for training the classifier, some important hyper-parameters are listed in Table below. Basically follow the default setting of fastText training, however, since both positive examples and negative examples from pretraining data distribution which tend to be diverse, increase the epoch number to 5 for better trianing convergence.
Table 5: The training hyper-parameters of our fastText classifier.
Ir
0.1
dim
100
epoch
minn
0
maxn
0
WordNgrams 2
Specifically we found that fastText will internally add 
 as end of sentence token the end of each documents, which will be learned uni-gram feature well. However, this token/uni-ght gramm feature generally does not exist in the data want to score (e.g. the pretraining corpus). Given every documents will be affected by this uni-gram feature with weight 1, we found this would potentially bring length bias towards data selection. If <ls> was learned as a negative feature, than short data will be influenced more according the mechanism fastText scoring calculation, which lead to a general low score of short data. Similarly, if <ls> was learned as positive feature, short data will also be influenced more, which lead a high score of short data. To bypass such bias towards length, we manually set weight to this uni-gres uni-gram feature.
A.3. Learned FastText Feature
1. PRELIMINARY FASTTEXT FEATURE VISUALIZATION
According to the fastText classifier model architecture (Joulin et al., 2017), there are basically two components - Input Matrix and Output Matrix, where input matrix maps all uni-gram feature to hidden dim (e.g. 100) and output matrix maps hidden dim to final classification dimension, in our case, Finally, a softmax layer is applied obtain the classification score for positive label and negative label.
Thus we can calculate the importance of each feature (uni-gram). Specifically, we define the input with N n-grams features 1 NN which represent the occurrence of each individual n-gram feature. We then define input matrix to be A E R 1000NN for simplicity where 100 is the hidden dimension size and output matrix to be B E 22x100 Thus the final layer is
Fi
T2
8/1
BA
(2)
3/2
LTN_
where y1 is the value before softmax for positive label and y2 is the value before softmax for negative label. Thus, for each individual feature we can calculate its contribution to and y given our trained A and B. Specifically, for each feature
...
Y/1 112
Ir we can calculate BA
I;
where I, 1. And (ya - y:2) is used as the feature importance where positive
value represents a positive feature.
Another problem for such importance visualization that we train the fastText classifier with Ngram equals where bi- bi-gram features are hashed into buckets which prevent from tracking the importance of specific uni-gram bi-gram features.
16

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
However, we found for fastText classifier trained with uni-gram + bi-gram features, if we only give its uni-gram and make the final score prediction, that does not have huge difference using all features (with 0.02 score difference out of 1.0). Then we explore the feature importance for uni-gram only.
A.3.2. FASTTEXT FEATURE INFLUENCE SCORE
We briefly discussed the learned features with their influence scores in $ 4.2, and we list the top-50 positive/negative features alongside the influence scores in Table 6 below.
Table 6: The Top-50 fastText learned fearures with corresponding influence scores of different methods. Positive and Negative are defined according to the magnitude of the influence score.
Method
Top-50 fast Features with Influence Scores
DCLM (pos)
(*' 968.194), (Why, 671.558). (Answer. 620.404). (Additionally.. 599.6). (assistunt. 567. (20), (Edit:, 565. 565.787). (Generate, 537.11 537.11), (answer., 502.883). (Given, 474.243), (At, 457.440). (Question:, 447.825). (Write, 440.69), (task., 423.878). (ELI5, 419.001). (AI, 20). (question: 994.,96). (WW, 378.738). (why, 378.032), (answering, 370.485), (Describe, 368.243), (explanation.. 349.408) (basically. 347.061), (sentence, 338.57). (&amp: 331,964). (answer. 328.221). (ussistant., 324.112). (How, 322.649), ([deleted]. 320.465), ((Your, 317.477), (Overall,, 316.189), (Reddit, 315.919), (EDIT; 314,785), (explain, 307.202 207),02), (Provide, 292.085 992.085, (step-by-step. 290.89), (old., 283.723), (Rewrite, 280.710), (user), 280.342), ([deleted, 279, 535), (Title: 279, 145), (is:, 378,694) (can.. 278.337), (Identify, 270.603), (Explain, 268.238), (Imagine, 259.265). (Summarize, 254. 946). (Create, 254.661), ([removed], 254,198), (Roleplay. 253.866), (Whats, 250.611)
DCLM (neg)
(-. (779.8888) (Comment, -466.0545). (Posted, -450.19833), (Post, 442.576233, ((...) -386.1609), (Comments,
377.41232), (-, -344,9523), (About, -340.93152), (..,, -321 (2254,, ("The, 325886) (Re:, -310,0053), (More, -297.77707), (v, -294.49173), (See, -289.7501), (2012, -278.6206) (View, -278.62042), (PM, -274.32016, (0, 272.98975), (2017, -269.229998 48788), (Read, -267.5124), (Leave, -65 90674), (Related, 220057003, (2013, -200 260.09442). (Love, 2254.712), (Free, -247.34369), (Home, -247.21219), (2014, -144.1049 (Q: -40.6447,,,,,,, (Click, 2239.25022, (Questions, -238,72769), (blog, -232 80008), (&, -23104167), (2010, -225.71875), (Be, -225.16672), (Ruy, 2222,3644),
(Originally, 2219775381,, (..,, -218.0946) (Last, 2217.36478) (Search, 221.. 1242), (2011, 2144555688,,, (, -214,43338), (Get.-200.06116, (2019. -00799997) (Continue, -203.56561),
(by:, 211391111), (to:, -213.87567), (Don't, -222004221,,,22644, (Although, -203.0502), (Date:, -201.52185), -19099,199.80663
PRESELECT
(,337.7), (Likes, 241.387), (sharelimprove, 241.28), (MIT, 240.614), (-xx0,, 240.006), (Retrieved, 232.997), (Received, 214.872), (Photo:, 205.808), (Azure, 200.453), (Tolkien, 195.01), (Math, 193.741), (answer, 187.706), (Contributed, 185.518), (Cite. 184.729). (MATLAB. 176.991). (Document, 172.13). (Youre, 168.549), (APL, 168.372). (Rhapsody. 166.203). (Name. 165,407) (Novell, 165.201), ((for, 162.172). (Dismiss, 159.38), (Geek, 156.977), (Examples. 153.852). (Expand, 153.352), (add-ons, 153.074). ((and, 151.961), (Techdirt, 151.241), (Peri, 148.295), (12, 147.958), (Youll, 146.352). (Random, 144.253), (macrumors, 142.24). (p.. 141.384), (Fool, 140.61), (secs, 140.129), (1., 138.052), (bass, 136.025). (Users. 132.508). (Boards, 132.233), (IBM. 131.088), (Journey. 130.867), (Step. 130.648). (BrainyQuod) 130.109). (ofxx0tthe 127. 127.833), (Preview, 127.324). (Comentários), 126.514). (contributors, 126.057). (Early, 124.897)
(pos)
PRESELECT
(Register, -375.421). (Details, -335.247). (Comments. -320.821). (Updated:, -310.755). (COVID-19, 19.-I---9--33.87777 -303.878), (tho. -281.664), (Print, -271 -271.024). (Profile, -262.205). (Leave, -258.85). (Published:, -235.235), (Product, -228.646). (coro- navirus, -227.227), (2020. -223,704), (By:, -222.939). (Shipping. -219.518), (Share, -219,401), (Login, -219.266). (Deals, -211.159), (Add, -207.725). (item, -201.697). (Reviews, -201.59,-201..9), (Sponsored, -200.319). (Current, -196.23), (s. -195,792), (Follow.-194.77) -994.77, (2021, -19..621.. (C. 192.3044. (Join.-189.933 -189.937) ("".. -189.647) (Post, -16.328), (View. -183,529), (Item, -1822289), (Privacy.- -176.569), ("W,, -176.348) (Search, -175.912), (Publication, - -175.804) (Author, -173,699), -abbsrribe.-70.55111 (Contact, 166.304), (ISBN, -163.925), (About, -163,91,,, (account?, -163.671), (re- views, -162.936) (Posted, -162.205). (Customer, -160,427) (Level, -155.532), (Paperback, 115. (Author, 115,,952)
(neg)
(Submit, 1152.988), (Ratings, 155.9985
A.4. Extend to Pretraining Corpora
We use DataTrove (Penedo et al., 2024c) which is a library for processing data at very large scale. The final filtering process on pretraining corpora does not need any GPU resrouces and the filtering process is done on the machine with 4 AMD EPYC 9654 96-Core Processor.
A.5. Selected Pretraining Data Details
As an extension of Table 4, we plot the selected data distribution in Figure 7 where we use the same y-axis scale for all four sub-plot which means the magnitudes of the bars are directly comparable. And we further include the selected pretrianing data distribution of perplexity correlation (Thrush et al., 2024) which uses domain-level correlation and then selects domains in Figure& 8 below.
17

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
0.830
0.230
0.225
0.025
0.520
9.020
0.015
0.015
0.505
0,505
0.000
0.000
up "rppan wowaaaiiieeee up cosuunsio un sesuuae aopee sosooo u. cettunee Soruun Be mnteee
un contiaatie uo Worfome Meae
con waweeaaaeeee Seree wogeeeeeeo
ard
coouune un Soooen pertoo
de vou Comteeeee cor
uno Storrra SeA we meeeee woy the Thooee pne repoon
Torne Bhat un Sarappae
con con careeraee onn dooeeeeeee ao serilleeeee un
on Respeeee
Density
wore
0,900
-
0.020
0.520
0,525
0.015
0.015
0.560
0.010
0
0.805
Ln pulooe Sodddie slere wat.eoa
ooy Sonoonnn cooloooooo Beeobooo booeeee do arpi P0pp Unt 3000 audr en aooratine
poettnt in
Contiaaoo Dro coiing Dorogoo
Dio comuunuee
C. com ane uon pittnn tou aodiiii
subeend di com seree soree
com
pompertt an Careeeaenn aannaatee (inoon oo com
corn
miwiiipekk org
sedee pettine nel te
wnt difio org dereeee snd
Li Aopreeee an contte tn Co sereoreee sotturee Co Co aiceeeee don aiseeeeee uun comeoaa
uon pol 230 pittee Bro un Ro cod sooe
womoo ooloor on Splas Tia
udt pepigntaui Sinbay//aia Bi86
Lddy "1100) haie Boxt 1013) Ci
unco 2260p44 slcxit SxtttRuu
Source Domains
Random Selection
FineWeb-Edu
DCLM
Ours
Figure 7: The selected data distribution (RefinedWeb) over source domains of different data selection baselines in descending order. The top-20 domains are listed for comparison and the density are counted based on character percentage. The y-axis scale are same across four subplots.
A.6. Filtering Computation Overhead
Similar to many model-based methods such as MATES, DsDm (Yu et al., 2024; Engstrom et al., 2024), PRESELECT involves pre-computing the compression efficiency on a small set of data. As we show in $ 2.3, we compute compression on around 0.6B tokens by Llama models ranges from 7B to 65B only once for all experiments. Accoding to Kaplan et al. (2020), the inference FLOPs is Cinfer A 2 N (D) = 1.8 x 1020 FLOPs, equivalent to 25 H100 hours, which is small, acceptable even negligible when training large model.
18

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
0.325
0.200
Density
0.025
0,500
und yeast Mdr cartaaa
1000 pass Ban AAA Rur
Arr Wil anten secedovtttie No. di un you pada Go
to s/o Hall arou
P'03 secoooooa 27
Heee Lu Mautyyaan so Aun
enovertt dertuure aut ann 14 projil uor pertooron anr
saall ai ao tee un co an piteen vor
(o a cor un ond do oo
un ie mer eolorrt
une corman dis on une iorreiiii ceecctoo san en wo ool be Saadhan tee corcesti aare
Muoy Coctte Rio corrodaii Rotie "y-o
uoo Too comoooolggaa Bio ueruueooo aaigua
Do EN 2000 aa Mig centrrriieee
Jar COo und 23448 JRMMMM and
Source Domains
Perplexity Correlation
Figure 8: The selected data (istributton(RefnneWWeb)) over source domains of Perplexity Correlation in descending order. The Top-50 domains are listed for comparison and the density are counted based on character percentage. The y-asis scale are same as Figure 7.
19

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
A.7. Additional Analysis Filtering
A.7.1. COMPARISON WITH SCALINGFILTER
ScalingFilter (Li al., 2024b) uses the perplexity difference between large model and small model metric to select data, which can be viewed as method only contrasts losses between two model scales. To show these intermediate level models actually help, we compare with ScalingFilter under our setting, using llama-65B and Ilama-7B to select data under ScalingFilter while removing the mid-size models to compute the Quality factor. Results of training 1B model on 30B tokens with our data pool (Table 1 setting) are shown in Table 7 below.
Table 7: The averaged performance comparison between ScalingFilter and PRESELECT.
Method Random
Average Benchmark Performance
37.2
ScalingFilter
37.6
PRESELECT
40.3
From Table 7, we see that after removing the 4 mid-size models, the results drop by 2.7 points on average. Take further step, we observe that data with a significant perplexity difference is skewed toward shorter texts, with an average character length of 2300, compared to the original corpus distribution's 's average of 3500 characters. We further see that the top domains where these data with significant perplexity difference from contains (regatorcom, msbusiness.com com, www. www.yourobservercoo www.qucollections.c comm www. www.sselleistro.com com, libn. com, www. www.buinnesmmnnggennntaii quality domains exist. These evidence suggests that using the perplexity difference between large model and small model tends to select easy and short data.
where none of typical high-
In addition, large loss difference between the largest model and the smallest model does not necessarily imply a rank preservation among a series of models. On the other hand, a rank matching between losses and downstream performance of a series of models does not necessarily imply a large losses difference between the largest model and the smallest model. For example, we calculate the spearman correlation between the ScalingFilter perplexity ratios and our PRESELECT predictive strength, which is 0.0533. And they also have Pearson correlation of -0.079 which indicate a low correlation between these two metrics (measured based on the actual predictive strength score where we calculate based the sampled subset), indicating the efffcctiveness of intermediate level models.
A.7.2. TARGETING SPECIFIC DOWNSTREAM TASK
Since our setting is pre-training, we didn't 't intend to choose a specific task as the target. However, in our initial experiments, we did explore such differences. For example, when choosing HellaSwag to represent intelligence that led to a different model ranking from using the average. This results by 5% lower losses on wiki-related raw texts while having much higher (worse) losses on other domains such as 8% on math and 16% on code, at a 400M model and 8B token training scale. Afterwards, we consistently choose the average performance represent intelligence under pre-training setting, but We think this is evidence that method actually predicts "downstream abilities" beyond only pre-training scale.
performance improvement on knowledge-intensive tasks, indicated
A.7.3. MITIGATING SENSITIVE CONTENTS
In our observation, literature and knowledge related contents are more reflective of downstream tasks (generally have a better rank alignment) which includes essays and some adult contents. We believe filtering adult content is an orthogonal direction that could be done separately, which is not the focus of this work. For example, we think rule-based pre pre-processing step to filter adult contents out could greatly mitigate this issue.
20

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
B. Pretraining Details
For pretrianing, mainly we have two settings, one for our main experiments (Refinedweb (Penedo et al., 2024d)) and another one follows MATES (Yu et al., 2024) which use Pythia (Biderman al., 2023) architecture with C4 (Raffel et al., 2020) as the training corpora. And we will separately discuss their details in the following subsections.
B.1. Pretraining Infra
Framework For our main experiments on RefinedWeb, we follow MAP-NEO (Zhang et al., 2024a) and adapted Megatron- based (Shoeybi al., 2020) training framework which allows us to efficient training models with different sizes under single-node multi-node. Since the largest model size is 3B, do not have any tensor parallelism pipeline parallelism. While for the experiments under C4, following MATES, use litgpt (AI, 2023), which is the training framework used by Tiaya (Zhang et al., 2024b).
Resource For our pre-training, we mainly use H800 (1 node) for training 1B models. While for some relatively large experiments such as 3B models, 4 nodes x8 H800 distributed training.
B.2. Pretraining Corpus
Given the different size of corpus and different source of corpus across our whole experiments, we list our pretrianing corpus setting for each part of experiments alongside their selection ratio, number of tokens etc. in Table 8.
Table 8: Pretraining corpus details for different experiments. *; which subset of the whole corpus is used in code.
Table
Corpus
Subset From
Method
Model Size
Pool Size
Selection Ratio
Trained Tokens
Random Selection
N/A
k E [0,
DCLM
400M
80B
10%
8B
PRESELECT
10%
Random Selection
N/A
30B
Random Selection
N/A
300B
Perplexity Filter
10%
30B
Web-Web
10%
30B
DCLM
10%
30B
RefinedWeb
A E [0,40]
Perplexity Correlation(DD)
10% (domaiiiwwwiis))
30B
Table
IB
300B
Perplexity Correlation(PP)
10%
30B
PRESELECT
10%
30B
DCLM
30%
90B
PRESELECT
DCLM
50%
150B
PRESELECT
Random Selection
N/A
1: [0, 130
DCLM
3B
1T
10%
100B
PRESELECT
10%
410M
Table
C4
N/A
PRESELECT
200B
top-25B
25B
IB
For RefinedWeb. we use a version' that after heuristic filtering and deduplication following DCLM and randomly sampled subset of needed number of tokens. We evenly sampled from each global index and local index. For C4, we use the whole dataset, which is about 198B tokens.
21

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
B.3. Model Architecture
Since train the models from scratch, list our model architectures in Table 9, For fair comparison, different data selection baselines compared the same scale (e.g. IB) the same model architecture.
Table 9: The pretraining model architecture details. For more configuration, please refer our code repo
Model Size
# Heads
# Layers
Context Length
Vocabulary Size
Hidden Size
FFN Hidden Size
NEO Architecture
400M
8
12
4,096
64,005
1,024
8,192
IB
12
4,096
64,005
1,536
12,288
3B
8
24
4,096
64,005
2,048
16,384
Pythia Architecture
410M
16
24
1,024
50,304
1024
4096
IB
16
1024
50,304
2048
8192
B.4. Pretraining Hyperparameters
For pretraining hyperparameters, consistent with many open-sourced models, we consistently use batch size 1,048,576 tokens, which is 4096 context length x 256 global batch size. Also widely used AdamW optimizer and cosine decay learning rate schedular are used. For Pythia models, we try the best to keep the same training setting with MATES (Yu et al., 2024). The detailed training hyperprameters are listed in Table below.
Table 10: The detailed pretraining hyperparameters for all experiments. For more configuration, please refer to our code repo
Model Size
Batch Size(Token)
Global Batch Size
Learning Rate
Schedular
LR Warmup
Optimizer
TP/PP
NEO Architecture
400M
IM
256
2c-4 + 2c-5
cosine
0.5%
Adam W
1/1
IB
IM
256
2e-4 + 2e-5
cosine cosine
0.5%
AdamW
1/1
3B
2e-4 - 2e-5
0.5%
Adam W
1/1
Pythia Architecture
410M
500K
1,024
1e-3 - 6.25e-5
WSD
2,000 2,000
Adam
IB
500K
1,024
le-3 + 6.25e-5
WSD
Adam W
1/1
22

---

Predictive Data Selection:
The Data That Predicts Is the Data That Teaches
C. Evaluation Details
C.1. Evaluation Framework
For our evaluation framework, we mainly adapt two widely used framework Opencompass (Contributors, 2023) and LM--valuation- Sitraation-aanessss (Gao et al., 2024) cover broader range of tasks as either one may lack some tasks in specific field. For Math and Code domain, adapted the code from Huang et al. (2024) and calculating the bpc based on provided math and code raw texts (e.g. Math Arxiv paper and GitHub code). Considering specific task, we list our used evaluation details, including evaluation method, metric, framework etc. for each task in Table below. We also categorize each task into an ability domain according to opencompass which provides better visualization.
bugs of these framework to
Table 11: The full details of evaluation framework, mode and metrics for each task. *; adapt the code from Huang et al. (2024) for BPC computation.
Domain
Task
Evaluation Mode
Metric
Framework
Arc-Easy
Perplexity
Accuracy
Opencompass
Examination
Arc-Challenge
Perplexity
Accuracy
Opencompass
MMLU
Perplexity
Accuracy
Opencompass
LAMBADA
Generation
Accuracy
Opencompass
OpenBookQA
Perplexity
Accuracy
Opencompass
Understanding
RACE-Middle
Perplexity
Accuracy
Opencompass
RACE-High
Perplexity
Accuracy
Opencompass
MultiRC
Perplexity
Accuracy
Opencompass
HellaSwag
Perplexity
Accuracy
Opencompass
PIQA
Perplexity
Aceuracy
Opencompass
SIQA
Perplexity
Accuracy
Opencompass
Reasoning
SciQ
Perplexity
Accuracy
RTE
Perplexity
Aceuracy
Opencompass
BBH
Generation
Exact-Match
AM-Evaluation-Hane Harness
Language
WinoGrande
II
Accuracy
Opencompass
Math
Math (L)
loss
BPC
Code
Code (1)
loss
BPC
C.2. Full Evaluation Results
Due the page limitation of main body, unable include all 17 task performance in Table 1 in $ 3.4. Here we list the full evaluation results of our method and various data selection baselines on 17 tasks, grouped by model size. The full evaluation results of IB models and 3B models are presented in Table 12 and Table 13 respectively.
23

---

Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Table 12: Full evaluation results of various data selection baselines and our method on 17 tasks at scale 30B token.
Task
Random Selection
Perplexity Filter
FineWeb-Edu
Perplexity Correlation(DD)
DCLM
PRESELECT
Arc-Easy
39.2
42.5
48.3
39.7
45.2
48.0
Are-Challenge
24.4
24.6
26.1
23.7
24.8
26.8
MMLU
26.0
25.8
26.0
26.1
26.3
26.0
LAMBADA
19.0
18.8
18.2
20.7
22.2
23,5
OpenBookQA
30.0
30.2
30.6
29.0
31.2
31.0
RACE-Middie
21.5
22.5
24.8
21.9
24.6
27.9
RiCE-High
22.4
22.7
23.9
23.7
27.5
MultiRC
57.2
57.2
57.2
57.2
57.2
HellaSwag
40.0
38.5
40.0
36.3
38.9
38.9
PIQA
69.2
67.6
67.3
67.6
67.6
67.7
SIQA
32.1
32.5
33.6
33.5
33,9
34,9
SciQ
64.8
67.5
69
63.7
70.0
71.5
RTE
52.7
52.7
52.0
52.7
52.7
53,8
BBH
7.8
8.5
12.8
9.5
12,6
16.2
WinoGrande
51.6
51.5
51.2
50.4
53.0
Average
37.2
37.5
38.7
37.1
38.8
40.3
Math (L)
1.023
0.957
0.906
0.980
0.857
0.830
Code (L)
0.901
0.853
0.816
0.919
0.773
0.744
Table 13:
Full evaluation results of various data selection baselines and our method on 17 tasks at 3B scale 100B token. Bold denotes the best.
Task
Random Selection
DCLM
PRESELECT
Are-Easy
51.2
55.7
61.2
Arc-Challenge
29.2
31.2
31.9
MMLU
24.8
25.3
26.2
LAMBADA
33.2
35.1
36.1
OpenBookQA
34.0
38.2
36.0
RACE-Middle
22.1
25.1
26.0
RACE-High
22.8
27.0
25.5
MultiRC
57.2
57.2
57.2
HellaSwag
57.5
58.4
59.2
PIQA
75.2
74.1
74.2
SIQA
35.8
33.4
33.8
SciQ
79.5
82.5
85.6
RTE
52.4
52.7
52.4
BBH
15.3
20.5
23.3
WinoGrande
54.3
55.8
55.0
Average
43.0
44.8
45.6
Math (4)
0.818
0.712
0.694
Code ({)
0.726
0.664
0.648
24


Reviews:
[{"review_id": "rmym5FmDm6", "reviewer": "Reviewer_VRK4", "summary": "This paper leverages the intuition that data on which losses of N models correlate with the benchmark performance (ranking) of the N models is the most useful. The paper proposes a simple numerical score for measuring this consistency in ranking (different and more numerically stable than Pearson correlation). To scale this effectively, after applying this method on a small subset of data, a fastText classifier is trained to classify text into that with high and low score using this methodology.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VRK4,\n\nThank you for your insightful comments! And we are grateful that you found our work very useful and effective.\n\n---\n\n---\n\nAs Reviewer 95of raises serious concerns about our paper and our response to that is probably related to other reviewers as well, due to the imposed character limitation, we have to list some responses to Reviewer 95of here, we are sorry for any inconvenience and thank you so much for the understanding.\n\n---\n\nQ6. [The paper should compare document-level vs. domain-level ranking on the same benchmarks to fairly assess their differences]\n\nOur experiments in Table 1 already compared document-level vs. domain-level ranking on the same benchmarks in a fair setting. Following your suggestions, we train the domain-level fasttext which takes high correlation domains as positive examples while low remaining domains as negative examples, to fairly assess the difference. \n\n|**Method**|**Average Performance**|\n|:-:|:-:|\n|Random |37.2|\n|domain-level filtering|37.1|\n|domain-level fasttext| 37.5|\n|PreSelect|**40.3**|\n\n\nBy using fasttext, it performs slightly better than random baseline and domain-level filtering, with 37.5% accuracy across 15 accuracy-based benchmarks, which is lower than document-level PreSelect. As we will also discuss some details in Q8 below, incorporating many domains (especially there is noise inside each domain) in fasttext training cannot lead to robust and dominant features. \n\n---\n\nQ7. [Is there concrete evidence that document-level selection is superior to domain-level selection?]\n\nAnd from the top selected data ((‘fanfiction.net', 0.36%), ('opentable.com', 0.32%),('ncbi.nlm.nih.gov', 0.30%), ('alldiscountbooks.net', 0.29%), ('hindawi.com', 0.26%), there is no significant good domains upsampled. Because of the word limit in rebuttal, we will add more comparisons (e.g. Learned fasttext features, selected data distribution…) to support the advantages of document-level over domain-level.\n\n---\n\nQ8. [How does PRESELECT fundamentally differ from PPL correlation, apart from working at a document level instead of a domain level?]\n\nAs we already acknowledged in the submission, PRESELECT and PPL correlation share similar ideas, thus the high-level idea is not that “fundamentally” different, which we never claimed to be. Even so, there are significant differences:\n\n1. PPLv1 uses domain level to classify documents while we work on finer granularity, this causes large empirical difference as we show in our original experiments and the added results in the response to Q6,7.\n\n2. PPLv1 suggested using a large number of open-source LLMs to approximate the correlation, while we only used 6 Llama models – while we did not mention in the submission, we tried using many open LLMs in the beginning and it did not significantly outperform the random baseline, and we finally opted to models from the same family to reduce potential noises from evaluating different-family models. \n\n3. Empirically, compared to PPLv1 which only reported 160M-scale results, our experiments cover larger scales and comprehensive benchmarks with strong results, representing significant empirical contributions as well.\n\n---\n\nQ9. [Why does the paper suggest that PPL correlation only uses Pearson correlation when the PPL paper explicitly discusses rank-based methods?]\n\nPlease refer to Q3.\n\n---\n\nQ10. [Given that the PPL paper now evaluates on 22 benchmarks (per the OpenReview update), does PRESELECT still maintain a clear experimental advantage?]\n\nFirst, the new PPL paper is updated after the ICML submission deadline,, which should not be considered when reviewing our paper. Second, even compared to PPLv2, our experimental advantage is very clear:\n\nWith a pre-filtered data pool, PPLv2 indicates quite negative results and it does not outperform the random baseline or DCLM significantly, while we achieve pretty strong results beating all the baselines under the DCLM pre-filtered data pool. We think that working with a pre-filtered data pool represents a more realistic setting, and empirically outperforming the baselines or not on a pre-filtered data pool makes a huge empirical difference and it directly impacts whether this method will really be used in practice or not.", "claims_and_evidence": "Yes, the main claim that the data that \"predicts\" is the most useful for training is empirically validated through experiments. \nAfter the main empirical results showing how on the DCLM benchmark, this method achieves better accuracy than baselines, the paper provides a qualitative exploration of what data is actually selected by this method. We see a preference for selecting more from reliable sources such as wikipedia etc. Also, it seems that this method removes the bias that other data selection methods make towards shorter phrases (for example, perplexity based selection).", "methods_and_evaluation_criteria": "The DCLM benchmark is the most appropriate benchmark for evaluating data selection for pre-training.", "theoretical_claims": "No theoretical claims are made.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This is a very useful contribution in the literature of pre-training data selection. The idea of selection based on a validation set has been around for some time now, but the approach of rank correlation using a variety of open source models & distilling this ability into a fastText classifier is highly intuitive and effective as demonstrated by the results.", "essential_references_not_discussed": "Essential references are all discussed.", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vOGWD8VSVH", "reviewer": "Reviewer_95of", "summary": "The paper Predictive Data Selection (PRESELECT) introduces a method for selecting pretraining data based on its predictive strength, defined as the correlation between normalized loss values and downstream task performance rankings. It proposes using a fastText classifier trained on documents with high predictive strength scores to efficiently select high-quality pretraining data. The paper claims that PRESELECT outperforms other data selection methods, including PPL correlation, by offering finer granularity and improved efficiency in language model pretraining.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 95of, \n\nThank you for your review. Our submission was flagged with “Research Integrity Issues (e.g., plagiarism)” due to concerns related to the PPL correlation paper.  We believe this is an unreasonable accusation and misleading to others. As we respond to the specific points below, we hope you can properly revise the statement. We are open to resolve concerns in our paper to discuss the relations between the two works more clearly (as we will mention below), but we also refute unreasonable accusations.\n\nFirst of all, we would like to sort out the timeline of different versions of the PPL correlation paper. The first version on arxiv was released on Sep 9, 2024 (PPLv1). The second version on arxiv was updated on March 10, 2025 (PPLv2) after the ICML submission. In our submission, we mainly referred to PPLv1. **While we will also discuss PPLv2 below, it should not be considered when judging this paper.**\n\nSecond, As PPLv1 was released 5 months prior to this submission, we did not really mention our works are concurrent following the common definition of “concurrency” – **but in reality, these two works are indeed concurrent and we independently developed this idea inspired by Huang et al. 2024. PPLv1 was released early partially because it only conducted 160M-parameter experiments on small scales, and we aimed for 1B and 3B scales, more serious experimental settings (pre-filtered data pool), and more benchmarks so it took longer.** This may be the main reason why we write this paper in the current style and the reviewer feels PPLv1 was not acknowledged enough. \n\nEven so, we note that our submission and PPL correlation still admit significant differences, and lead to very different empirical results in the end. We continuously acknowledged the prior contribution of PPL correlation throughout our paper – where we included a dedicated paragraph in the Introduction section to discuss it, and made comparison with it empirically in the experiments. We address the concerns one by one below:\n\n---\n\nQ1. [ This claim is similar to that of the PPL correlation paper but is framed as an original insight, despite the PPL paper pre-registering experiments on a similar premise.]\n\nIn the introduction section, we explicitly pointed out that our submission shares similar intuition with PPLv1 (Line 95) – we were quite open acknowledging this. However, as we mentioned above, our idea was independently inspired by Huang et al. 2024 and this project started prior to release of PPLv1, thus we wrote the paper following our actual logic flow of performing this project. \n\nTo mitigate the concerns on this point, we will revise our description of PPL correlation as “the first to explore domain-level correlation-based methods in data selection”. \n\n---\n\nQ2.[The authors state they will open-source their trained data selection scorer and datasets. However, they have not done it yet]\n\nWe were planning to release them after the review period.\n\n---\n\nQ3. [Misrepresent PPL correlation by suggesting that it only uses Pearson correlation]\n\nWe didn’t suggest  PPL correlation only uses Pearson correlation. The reviewer got this impression maybe due to Line 116? We never intended to imply that, Line 116 only meant that grouping documents together as in PPLv1 could enable more stable computation of Pearson correlation. We will revise that sentence to avoid misunderstanding, and in the meanwhile, we will acknowledge PPLv1’s discussion on ranking-based methods in Section 2.2. \n\n---\n\nQ4. [They do not cite the updated version of the PPL correlation paper.]\n\nIf the reviewer refers to PPLv2, we could not cite it in the submission because it was released afterwards.\n\n---\n\nQ5. [The novelty claim is questionable, given that PPL correlation pre-registered similar ideas months prior]\n\nWe do not think the community has reached agreement on how to understand or treat the  “preregistration experiments”, and it is not common practice to write them. All the conferences have no guidelines about it. While we fully understand the motivation of doing this from PPLv1, we do not consider a long list of \"we will do xxxx in the future\" statements to constitute prior contributions. We respectfully disagree with the notion that it is good practice for the community to simply post preliminary results and preregister experiments to claim priority. \n\n---\n\nBecause of the imposed character limitation, we keep addressing the remaining concerns in the response to reviewer VRK4. Sorry for any inconvenience and thank you for your understanding.", "claims_and_evidence": "Normalized loss (compression efficiency) is predictive of downstream task performance. The authors cite prior work (Huang et al., 2024) showing that losses on certain text domains correlate with task performance. They extend this idea to a finer document level.\nPRESELECT is more effective than existing data selection methods, including PPL correlation. Experiments demonstrate performance improvements on 17 benchmarks, outperforming PPL correlation and other baselines. PRESELECT offers a more scalable and efficient approach to data selection. The method eliminates human heuristics, operates at the document level (not just domain level), and requires only a lightweight fastText classifier.", "methods_and_evaluation_criteria": "The paper evaluates PRESELECT against several baselines, including: PPL correlation, DCLM and FineWeb-Edu, and random selection and low-perplexity filtering.\n\nPerformance is assessed on 17 NLP benchmarks spanning general understanding, knowledge-intensive tasks, mathematics, and coding. The evaluation criteria include: Accuracy on NLP tasks, bpc for math, and code tasks, compute efficiency.", "theoretical_claims": "There's no theoretical claims in the paper, but emperic investigations of a hypothesis that ranking-based correlation is more robust than Pearson correlation for identifying predictive pretraining data. It suggests that documents where normalized losses strongly correlate with model rankings are more useful for training. This claim is similar to that of the PPL correlation paper but is framed as an original insight, despite the PPL paper pre-registering experiments on a similar premise.", "experimental_designs_or_analyses": "The experiments compare PRESELECT with multiple baselines using standardized datasets. The study measures predictive strength using rank-based correlation rather than Pearson correlation, arguing for better robustness. The claim that PRESELECT outperforms PPL correlation is supported by experimental results but does not fully acknowledge that the core methodology (perplexity-based ranking) was introduced in the PPL paper.", "relation_to_broader_scientific_literature": "The paper references prior work on compression as intelligence (Huang et al., 2024) and perplexity-based filtering (Thrush et al., 2024).\nIt places PRESELECT within the context of data selection research, contrasting it with heuristic-based and supervised selection methods.\nHowever, it does not adequately credit perplexity correlation methods in prior literature as a direct predecessor.", "essential_references_not_discussed": "The authors misrepresent PPL correlation by suggesting that it only uses Pearson correlation, despite the PPL paper pre-registering rank correlation-based experiments. They do not cite the updated version of the PPL correlation paper that includes additional benchmarks and domain-level experiments.", "other_strengths_and_weaknesses": "Strengths: efficient, scalable data selection method, evaluated on many bench marks, demonstrates compute savings.\n\nWeaknesses: The novelty claim is questionable, given that PPL correlation pre-registered similar ideas months prior. The comparison with PPL correlation is misleading: The authors claim PPL correlation only uses Pearson correlation, which is false. They do not properly acknowledge the prior work’s pre-registration of page-level results. Overlap in experimental setup suggests that PRESELECT may be an incremental extension rather than a fundamentally new contribution.", "other_comments_or_suggestions": "The paper should explicitly acknowledge the PPL correlation work as a foundation and clarify what aspects are genuinely novel.\nInstead of dismissing domain-level PPL correlation, the paper should compare document-level vs. domain-level ranking on the same benchmarks to fairly assess their differences.\nThe authors should engage with the PPL paper authors to resolve these concerns before publication.", "questions_for_authors": "Why does the paper claim to be the first to propose rank-based correlation when the PPL correlation paper pre-registered similar results months earlier?\n\nHow does PRESELECT fundamentally differ from PPL correlation, apart from working at a document level instead of a domain level?\n\nWhy does the paper suggest that PPL correlation only uses Pearson correlation when the PPL paper explicitly discusses rank-based methods?\n\nGiven that the PPL paper now evaluates on 22 benchmarks (per the OpenReview update), does PRESELECT still maintain a clear experimental advantage?\n\nHow do the selected datasets from PRESELECT and PPL correlation compare qualitatively? Is there concrete evidence that document-level selection is superior to domain-level selection?\n\nWould the authors be open to revising their claims to reflect PPL correlation’s prior contributions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Research Integrity Issues (e.g., plagiarism)"]}}, {"review_id": "opUIu8wr4T", "reviewer": "Reviewer_k3Rj", "summary": "This paper explores the problem of data selection for pretraining language models. The authors propose a lightweight method that leverages predictive strength as an indicator to determine whether a document should be included in the pretraining data. To evaluate their approach, they train a group of language models of varying sizes on datasets of different scales, selected using various methods. Their findings suggest that the proposed method outperforms other data selection techniques, leading to more diverse data domains and a more balanced distribution of data length.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer k3Rj,\n\nThank you for your valuable suggestions and insightful comments! We are grateful that you found our work well-structured and logically organized, and we deeply appreciate your recognition of the novelty of our method and the comprehensive analysis in our experiments.\n\nWe address your comments one by one as follows:\n\n---\n\nQ1. [Experimental design does not fully verify the core assumption]\n\nWe think the most direct way to verify our core assumption is to compare the results between our selected data and randomly selected data, with the same amount of data trained. And we show this in Table 1 400M/1B/3B row Random & PreSelect, Table 2 410M/1B row Random & PreSelect, where training using data selected by PreSelect consistently and significantly outperforms random selection across model sizes and tasks.\n\n---\n\nQ2. [Missing baseline that trains on the entire dataset without selection could be added]\n\nWe thank you for highlighting this important point. In fact, on top of the random selection (i.e. without selection strategy), we did include a baseline that trains on the entire dataset (i.e. without selection). In Table 1 Row 4, we refer to it as “random with 300B”, the model is trained on the entire 300B dataset without selection, and this 300B dataset was our data pool for 1B experiments. The following rows are different selection strategies performed on this entire dataset/pool. We can see that by filtering the high-quality examples, PreSelect not only achieves a 10x reduction in compute requirements but also yields additional performance gains.\n\n---\n\nThank you again for your suggestions and we will describe this baseline more clearly in the revision.", "claims_and_evidence": "One key question regarding the experimental setup is why the baseline does not include a method that uses the full dataset without applying any selection. Including such a baseline could serve as a valuable reference to measure how much performance improves or declines due to data selection. This would help in assessing the effectiveness of the proposed method more comprehensively.", "methods_and_evaluation_criteria": "Both methods and evaluation criteria make sense.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "The experimental designs are valid.", "relation_to_broader_scientific_literature": "Several prior works have explored different data selection methods for pretraining. Additionally, some studies suggest that the loss on specific data can reflect model performance on downstream benchmarks. This paper builds on that idea by assuming that data which better reflects model capability is more beneficial for pretraining. Based on this assumption, the authors introduce the predictive strength score as a criterion for selecting pretraining data.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "### Strengths:\n1. The paper is well-structured and logically organized.\n2. It introduces a new method for data selection in language model pretraining.\n3. The authors conduct a comprehensive analysis comparing their method to baseline approaches.\n4. In addition to evaluating performance, the paper provides a detailed analysis of the characteristics of selected data and how they differ from those chosen by previous methods.\n\n### Weaknesses\n1. The experimental design does not fully verify the core assumption of the paper. The assumption is that data on which loss can better predict performance is more beneficial for pretraining. To validate this, a baseline that trains on the entire dataset without selection could be added. This would help determine the actual contribution of the unselected data and provide a clearer picture of the impact of data selection.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5PBk1veWJx", "reviewer": "Reviewer_xzzW", "summary": "The authors propose a new data selection based on the rankings of perplexity which a range of llama models assign to documents. The method is scaled to large datasets via means of a fasttext filter. The authors claim that this captures which training examples are predictive of broad downstream abilities. The data selected by this method achieves better downstream results than a number of baselines when evaluated in a comparable setting. The authors perform some high-level inspection of domains upweighted and downweighted by the different methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer xzzW,\n\nThank you for your valuable suggestions and insightful comments! \n\nWe address your comments one by one as follows:\n\n---\n\nQ1.[ScalingFilter [1], which I believe is an important point of comparison for the proposed method]\n\nThanks for pointing out a related work ScalingFilter[1] which uses the perplexity difference between a large model and a small model as a metric to select data. To show that the intermediate levels of models actually help, we compare with ScalingFilter under our setting, using llama-65B and llama-7B to select data under ScalingFilter while removing the mid-size models. Results of training 1B model on 30B tokens with our data pool (Table 1 setting) are below.\n\n|**Method**|**Average Performance**|\n|:-:|:-:|\n|Random |37.2|\n|ScalingFilter| 37.6|\n|PreSelect|**40.3**|\n\n\nWe see that after removing the 4 mid-size models, the results drop by 2.7 points on average. We will add these results to the next revision of the paper.\n\n---\n\nQ2. [choose a setting where different downstream tasks elicit different model rankings]\n\nSince our setting is pre-training, we didn’t intend to choose a specific task as the target which was analyzed by Perplexity Correlation[2].\n\nHowever, in our initial experiments, we did explore such differences. For example,  we chose a HellaSwag to represent intelligence that led to a different model ranking from using the average. This results in a performance improvement on knowledge-intensive tasks, indicated by 5% lower losses on wiki-related raw texts while having much higher (worse) losses on other domains such as  8% on math and 16% on code, at a 400M model and 8B token training scale. We think this is evidence that our method actually predicts “downstream abilities” beyond only pre-training scale. \n\n---\n\nQ3. [Importance of group-level selection]\n\nThank you for your suggestions, we further compare with computing perplexity rankings on a group level to select data. The results and analysis can be seen in Q6,7 of the response to reviewer 95of.\n\n---\n\nQ4. [Add few-shot performance]\n\nThank you for your suggestions! We apologize for this confusion. For experiments in Table 1, some of these are zero-shot while some are few-shot. We follow the common practice to use zero-shot for classification tasks (perplexity-based metric), and few-shot for generation tasks. We will indicate this more clearly in the revised version.\n\n---\n\nQ5. [Low performance in Table 1]\n\nWe believe this performance difference stems from differences in the experimental configurations between Tables 1 and 2, including variations in pre-training corpus (DCLM pool from CC vs. C4, quality differs in each subset), training framework, different evaluation framework (prompt), slightly different model size etc. Thus these two tables are not that comparable. Regarding the “strange” MMLU results from 1B models in Table 1, those numbers are all around random guess (25%) and do not reflect much, because MMLU is too difficult for small models to learn quickly. For example, in DCLM’s paper, even a 7B model trained at chinchilla optimal scale still shows a nearly random accuracy on MMLU. \n\n---\n\nQ6. [Additional ablations]\n\nThank you for this suggestion. We did try using many additional models (especially mixing different model families) to compute the ranking coefficient, but that brought significant noise due to variance of evaluation results from different models. Thus the trained models underperformed the current version significantly. \n\nAnd following your advice, we compute the ranking coefficient based on the pythia series (14M, 31M, 70M, 160M, 410M,1B),  where we find the top domains of the llama produced data also appears in the top domains of the pythia produced data. And same for negative domains, they also share a large portion of overlap. However, due to the limited time period, we didn’t train them. We will show analyses of detailed overlap, differences and more insights about using different families of models in the revised version following your suggestions.\n\n---\n\nQ7. [Missing comparison to [1][3][4]]\n\nWe have added comparison to ScalingFilter[1] in the rebuttal (Q1). And for [3,4], we think they share the same insights with one of our baseline perplexity filtering[5]. And we will add them into related work for discussion. \n\n---\n\nQ8. [Questions about adult content]\n\nIn our observation, literature and knowledge related contents are more reflective of downstream tasks (generally have a better rank alignment) which includes essays and some adult contents. We believe filtering adult content is an orthogonal direction that could be done separately, which is not the focus of this work.  For example, we think a rule-based pre-processing step to filter adult contents out could greatly mitigate this issue.\n\n---\n\n[5] Wenzek et.al., LREC 2020. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data", "claims_and_evidence": "While the experimental results look impressive, I have major concerns about the justification and motivation for the method.\n\n**Confounding factors**\n\nThe central thesis is that \"data on which model losses are predictive of downstream abilities also contribute effectively to learning\". However, their method only vaguely involves “prediction of downstream abilities” as they employ a coarse-grained rank-correlational method, where the average benchmark ranking of a small set of models is compared to the models’ perplexity ranking on a specific document.\n\nTheir “downstream abilities” model ranking is fixed across all experiments:\n\nLlama-1-65B > Llama-1-30B > Llama-2-13B > Llama-1-13B > Llama-2-7B > Llama-1-7B\n\nwhich suggests the equally valid hypotheses that either “*documents on which additional model parameters benefits loss are useful for pre-training*” or  “*documents on which additional pre-training compute benefits loss are useful for pre-training.*. The first hypothesis has previously been explored by ScalingFilter [1], which I believe is an important point of comparison for the proposed method. Their method only contrasts losses between two model scales, and you could show that intermediate levels actually help.\n\nTo show that the method actually predicts “downstream abilities” beyond only pre-training scale, the authors should choose a setting where different downstream tasks elicit different model rankings. Note that this fine-grained downstream analyses is performed in Perplexity Correlations [2], which the authors discuss as the most similar prior work (although I think that ScalingFilter [1] is actually the most directly related paper).\n\n\n**Importance of group-level selection**\n\nThe authors claim that sample-level selection is an important factor to their success of their method compared to Perplexity Correlations [2], which only performs group-level selection. However, I believe these methods are very different in nature. Therefore, the empirical justification for this claim should be to apply their method and compute perplexity rankings on a group level to select data, rather than comparing to the results of Perplexity Correlations.\n\n\n[1] Li et al., EMNLP 2024. ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws\n\n[2] Thrush et al., ICLR 2025. Improving Pretraining Data Using Perplexity Correlations", "methods_and_evaluation_criteria": "The method makes sense and the evaluation/experimental setting is reasonable for evaluating training data selection. Scaling the document selection to large corpora with fasttext is also understandable. \n\n**Reliance of large pre-trained models**\n\nOne potential issue with the experiments is that they rely on much larger models (from 7B-65B) for selecting training data of a small model (1B).\nThis may limit the utiliy of this method when wanting to scale to even larger models. \nNote for example, that recent work in perplexity filtering [3] (another very relevant paper) highlights the use of a smaller model for evaluating perplexity than used for training. \nI would suggest repeating the method with smaller models, for example the pythia models of sizes 14M, 31M, 70M, 160M and 410M.\n\n\n[3] Ankner et al., ICLR 2025. Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The experimental design follows standard practices from prior works. The choice of downstream tasks is also good. However, besides the missing comparisons discussed above, \n\n**Add few-shot performance**\n\nIt is not clear whether the main results in Table 1 are from zero-shot or few-shot prompting (whereas Table 2 is explicitly zero-shot). I would suggest adding few-shot results in the appendix at least.\n\n**Low performance**\n\nI'm confused why the general results in Table 1 are so low. For example, a 1B Random selection baseline on 25B tokens on C4 performs in Table 2 outperforms the 30B token DCLM baseline in Table 1 on ARC-easy, ARC-challenge and HellaSwag, despite the known strong performance of DCLM-baseline over C4. The low performance of the 1B model at 300B token is also a bit strange, including the decrease in MMLU score compared to 30B tokens.\n\n**Additional ablations**\n\nThe proposed data selection objective is only instantiated with a single fairly arbitrary series of existing models. I would guess that the proposed rank coefficient is quite sensitive to the number and kind of models used. I think it is essential to introduce ablations to understand some of the implicit assumptions on reference models and demonstrate the robustness of the model, i.e. how would the results change when including or excluding large models and when using a different reference model family (e.g. Pythia).\nI realize that every type of ablation would be computationally too expensive, but even showing analyses like overlap and differences in selected data would be valuable to the reader.", "relation_to_broader_scientific_literature": "In my view, the key contribution is that the paper provides further empirical evidence that \nthe loss from pre-trained models can be useful signal for selecting pre-training data,\nfollowing from work on using loss from one, two or many models to select data ([1], [2], and [3, 4] respectively). The authors achieve strong results with their method, and beat heuristic quality-based data selection baseline, highlighting the promise of automatic loss-based methods requiring less human intervention. However, there is no discussion or comparison to [1, 3, 4] in the paper.\n\n[1] Li et al., EMNLP 2024. ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws\n\n[2] Thrush et al., ICLR 2025. Improving Pretraining Data Using Perplexity Correlations\n\n[3] Ankner et al., ICLR 2025. Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models\n\n[4] Marion et al., 2023. When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale", "essential_references_not_discussed": "As mentioned above, the paper misses a comparison and discussion of ScalingFilter [1] and Perplexity Pruning [3, 4].", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "* Equation (1) would be much clearer if you indicated that the score S and the ranks C's are per document!\n* Btw Figures 6 and 7 have the same data, despite meaning to show the most negative and positive domains in terms of selected data", "questions_for_authors": "* I'm curious why you don't address the fact that the method seems to have a strong bias towards selecting ''adult content`` (in Figure 3)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a data selection method PRESELECT that operates on a similar intuition to previous successful work on data selection (Perplexity Correlations): that data where model loss correlates with performance are also the data most useful for training. Reviewers all agreed that the method and the experiments were sound and impressive. All reviewers were positive (xzzW did not update their score but acknowledged their concerns were addressed in the rebuttal discussion) except for one, which I discuss below.

The main negative point raised by one reviewer was severe concern about misrepresentation of the prior Perplexity Correlations work in this work. On one hand, I sympathize with the reviewers point: the authors should be careful about making statements like "we hypothesize that data on which model losses are predictive of downstream abilities also contribute effectively to learning" - even if the authors were independently inspired, it is not appropriate to frame this as a hypothesis of this work given that the hypothesis has already been posed (and verified!) by the Perplexity Correlations work.

On the other hand, I don't see good evidence that there is genuine academic misconduct here: there is no explicit plagiarism of text (as far as I can tell) nor is there evidence of the "parallel structure" between the two papers that would indicate some kind of purposeful copying. Instead, it seems as though the authors were independently inspired, found out their idea was very similar to Perplexity Correlations, and tried to differentiate their work - falling short in some respects.

As for the "truly novel" contributions of the paper, I think the combination of document-level selection, larger-scale experiments, and model curation are of general interest to the community - there is still a lot of disagreement in the broader community about the utility and scalability of data selection for pre-training, so even incremental progress here is quite interesting.

Given that the authors seem open to editing their paper, I strongly encourage them to:

1. Mention perplexity correlations in the abstract, and make the contributions of this paper more upfront. Something like "this hypothesis was first explored by [PPL]---in this work, we further corroborate and refine this hypothesis via A, B, and C." 
2. Remove statements like the one quoted above (there is a similar statement in the introduction, and scattered throughout the paper).
3. More explicitly compare the design choices made to perplexity correlations in each subsection - even if the authors were independently inspired, it is important (and does not detract from the paper) to put their method into context with the current state-of-the-art.
Decision: Accept (poster)