Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
Authors: Zachary Ankner
Cody Blakeney
Kartik Sreenivasan
Max Marion
Matthew L Leavitt
Mansheej Paul
Venue: ICLR 2025 Poster (2025)
Keywords: Data
Data Filtering
Data Pruning
Pretraining
Perplexity
Large Language Model
LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: We demonstrate that the perplexity of a small language model can be used to prune the dataset that a significantly larger language model is trained on.

Abstract: In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that pruning based on the perplexity of a larger model can yield high-quality data, we investigate whether smaller models can be used for perplexity-based pruning and how pruning is affected by the domain composition of the data being pruned. We demonstrate that for multiple dataset compositions, perplexity-based pruning of pretraining data can significantly improve downstream task performance: pruning based on perplexities computed with a 125 million parameter model improves the average performance on downstream tasks of a 3 billion parameter model by up to 2.04 and achieves up to a 1.45× reduction in pretraining steps to reach commensurate baseline performance. Furthermore, we demonstrate that such perplexity-based data pruning also yields downstream performance gains in the over-trained and data-constrained regimes.

Published as a conference paper at ICLR 2025
PERPLEXED BY PERPLEXITY: PERPLEXITY-BASED DATA PRUNING WITH SMALL REFERENCE MODELS
Zachary Ankner 1.2 Cody Blakeney'
Kartik Sreenivasan' Mansheej Paul
Max Marion Matthew I.. Leavitt
1 Databricks MIT Aatology
ABSTRACT
In this work, we investigate whether small language models can determine high- quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that pruning based on the perplexity of larger model can yield high-quality data, investigate whether smaller models can be used for perplexity-based pruning and how pruning is affected by the domain composition of the data being pruned. We demonstrate that for multiple dataset compositions, perplexity-baat based pruning of pretraining data can significantll improve downstream task performance: pruning based on perplexities computed with a 125 million parameter model improves the average performance on downstream tasks of 3 billion parameter model by up to 2.04 and achieves up to 1.45x reduction in pretraining steps to reach commensurate baseline performance. Furthermore, we demonstrate that such perplexity- -based data pruning also yields downstream performance gains in the over-trained and data-constrained regimes.
INTRODUCTION
large focus of the machine learning community has been improving the performance of large language models (LLMs) while reducing their training costs. In this work, we consider how to improve the quality of an LLM by improving the quality of its pretraining data. Although there are many techniques to improve data quality, such as augmenting training samples with additional information (Li et al. 2024 Korbak et al.) 2023), in this work we focus on the predominant method of data pruning: inte ligently selecting a high-quality subset of a larger dataset to train on.
Data pruning commonly used for quality filtering of noisy text data. Simple approaches include using symbolic rules (Bane et al. 2022 Raffel et al. 2020) or using simple classifiers to determine high-quality samples (Wenzek et 2020). However, in addition basic quality filtering, more complex data pruning techniques
also applied to datasets to farther improve their quality. Xie et al.) (2023b) perform importance resampling where importance scores are calculated based on feature similarity target text. Tirumala al. (2023) prune datasets by deduplicating and diversifying data based on a pretrained language model's embeddings of the text samples. Xie et al. (2023a) re-wey ee-weight domain proportions based on learnability as determined by smaller proxy model. Marion al. (2023) investigate data pruning based on multiple neural heuristics of sample difficulty, ultimately concluding that the perplexity of sample under reference language model is the best pruning metric.
In this work, we thoroughly investigate the impact that data pruming based on sample perplexity (Mar ion et al.] (202)) has on LLM pretraining. In particular, we focus on the interplay between pretraming dataset composition and pruning methodology. We further evaluate perplexity pruning in the over- trained and lata-constrained regimes. We also investigate whether evaluating the quality of data interventions based on upstream test set perplexity is sound methodology for gauging downstream performance. To perform perplexity-based data pruning, random subset of the given pretraining dataset and then evaluate its perplexity on each sample in the dataset. We then prune the dataset to only include samples within some range of perplexities (i.e.. sub-sample to the highest or lowest perplexity samples). We demonstrate that for two vastly different pretraining data compositions, small language model can be used to effectively prune the
train small language model on

---

Published as a conference paper at ICLR 2025
pretraining dataset of a significantly larger model, leading to significant gains in the final model's downstream performance.
Our work differs from previous work perplexity-basee data pruning for LLM pretraining in three key ways: (i) our emphasis on downstream model quality evaluation, (ii) our exploration different pretraining dataset domain compositions, and (iii) our analysis of pruning in non-standard training regimes. While previous works evaluate the resulting LLM's quality based on upstream metrics such as perplexity the test split of the pretraining dataset, we evaluate data pruning's 's impact based on downstream evaluation benchmarks (e.g. mmlu (Hendrycks et al. (021), hellaswag Zellers et al. 2019), etc.). Evaluating on more meaningful benchmarks enables us to make stronger, more ngorous conclusions about the impact of perplexity- -bsee data pruning, as we find that some techniques that significantly improve downstream performance have no, or even adverse, effects on upstream performance. This difference in metrics enables us to conclude that smaller models can prune the data for larger models, which was not observed in previous perplexity-ban -hased pruning works. Secondly, while previous work only investigates pruning datasets composed of just one domain (CommonCraw D. we consider two datasets with different domain compositions: the Pile (Gao et al. 2020) and Dolma (Soldaini al. 2024). The Pile is composed of many diverse curated domains, with only 15.61% of the data being derived from general web-scrapes, while Dolma is a web-scrape skewed dataset, with 81.31% % of its data being derived from the CommonCrawl. We find that successful pruning techniques vary greatly for different dataset compositions to the point that the best technique for one dataset composition may degrade performance for different composition. Finally, we also evaluate perplexity-basee data pruning in the less standard regimes of over-training and data-constrained training. This investigation provides a broader understanding for when practitioners should use perplexity pruning for their data.
Contributions Our work makes the following contributions:
We demonstrate that, across three datasets of varying domain compositions, small reference model can effectively prune the pretraining dataset of a significantly larger language model (30x greater parameters), providing both a significant increase in downstream performance and decrease in pretraining steps (Table and Figure D
We show that data pruming techniques can be highly sensitive to the domain composition of the dataset, suggesting the need to evaluate multiple distinct dataset compositions when conducting data pruning research (Table aad Table
We investigate perplexity-based data pruning in multiple non-standard settings demonstrating that can still lead to gains when over-training and when data-constrained (Section 3.4 and Section 3.5).
We find that test set perplexity can be a misleading metric for evaluating the efficacy of data pruning techniques, as interventions that result in significantly higher test set perplexity can still achieve better performance on downstream tasks (Table
2 PERPLEXITY-BASE DATA PRUNING
We start by training reference model that will be used to calculate the perplexity of all samples in our dataset. First, we partition the original dataset into two splits: one for training the reference model and one for training the final model. After training the reference model on the standard next-token prediction objective, we compute the reference model's 's perplexity on each of the samples in the final model's 's training split. We then prune the final model's dataset split to a fraction of its original size, referred to as the selection rate (ra), by selecting samples according to selection criteria which can be one of low, medium, or high. In low selection, samples with the lowest perplexity are selected. In medium selection, we select samples whose perplexity is close to the median perplexity, that is, samples with perplexity in the [50- 2 50 + "= percentiles of all perplexities. In high selection, samples with the highest perplexity are selected. After pruning our dataset, we train a final model using the standard next token prediction objective on the pruned version of the final model training split. We present pseudocode for pruning based on perplexity in Algorithm
https://data.commmcrraaaa
2

---

Published as a conference paper at ICLR 2025
Algorithm 1: Psuedocode for performing perplexity-based data pruning.
Input: Raw dataset D
(x(1) ===1, where each 2²1)) is tokenized text sample;
selection_cn Aaleecion_criteriin
etw.eciiiiaaaeee E {low, medium, high}; selection rate ra (0, 1); reference
training split size R.
Output: Parameters of final model trained on the perplexity pruned dataset fial-
Dref, Dirain random_sppl spltt(Dppp. Rt)
Ore + random parameter initialization
0e + train(Ppdt Diet)
P +-1 []
for (0() E Dirain do
NLLg(( [zO Et,Ez(0) - log P(tyltzzi00d
(P 2
P[ [[ =PPLX
end
selectionncriteriieei = "low" then
mir_percentile + 0.0
max__perceta max_percentile++ r
end seeeectionncriteeeeeieee = "medium" then
else
mir_percentile+ 0.5
max_percentile+ 0.5
end
else phelecttoo_criteriis == "high" then mir_percentilee+
max max_percentile+ 1.0 end
Fp + empirical CDF of P. values
D prined
for
(0) (PLX,6) E P do
if inin_percentile < Fp(PPLX,() < max_percent le then
D Dpnned append (2()))
end
end
Oinal f- random parameter initialization
0 Final train D promed)
return /imal
We consider the setting in which the reference model is significantly smaller than the final model. While this assumption is not strictly necessary, believe that is the most practically relevant setup, as it best reflects a data pruning paradigm that would be used for the next generation of LLMs where the models being trained are larger than any existing models.
EXPERIMENTS
3.1 SETUP
Models. All models are based on the MPT family of transformer models (Vaswani et al. 2017 MosaicML 2023c). All reference models have 125 million parameters, and we consider final models with billion and billion parameters.
Data. We consider two datasets this work. The Pile (Gao et al. 2020 is composed of 22 different domains that range from general web scrapes to legal text. Dolma (Soldaini et al. 2024) is composed of 7 different domains and is derived mainly from general web scrapes. We tokenize all datasets using the GPT-4 tokenizer (OpenAl 2022

---

Published as a conference paper at ICLR 2025
Table 1: Average normalized accuracy grouped by task category for both datasets and both final model sizes. For all datasets and model sizes we find that training on perplexity pruned data outperforms the baseline. Bold results are within one standard error of the highest score.
World Common
Language
Symbolic
Reading
Sense Under- stand- ing
Prob- lem Solving
Com- prehen- sion
Pruning Method
Knowl- edge
Average
Reason- ing
IB Parameters Trained on Pile
No Pruning (Baseline)
15.51
10.31
28.11 11
3.53
11.16
13.73
High Perplexity Selected
18.18
12.75
33.2
3.36
10.63
15.62
3B Parameters Trained on Pile
No Pruning (Baseline)
21.82
13.09
39.08
4.88
14.28
18.63
High Perplexity Selected
25.8
16.24
43.32
2.91
15.07
20.67
IB Parameters Trained on Dolma
No Pruning (Baseline)
16.48
12.32
28.86
3.58
7.95
13.84
Medium Perplexity Selected
17.98
13.03
31.87
3.44
10.41
15.35
3B Parameters Trained on Dolma
No Pruming (Baseline)
23.56
14.29
39.57
4.4
14.2
19.2
Medium Perplexity Selected
24.19
16.48
41.8
3.3
13.19
19.79
Training and hyperparameters. All reference models trained for fixed duration of 26 billion tokens. Unless otherwise specified, all final models are trained to Chinchilla optimal (Hoffmann) al. 2022), meaning that each final model's training duration in tokens is 20 times its parameter count. All models are trained using the decoupled Lion optimizer (Chen et al.) 2024) with a cosine learning rate schedule, All reference models and 1B parameter models are trained with maximum learning rate and weight decay of ee-4 and all 3B models are trained with maximum learning rate and weight decay of 1 6e-4. Training is conducted using lIm-foundry (MosaicML 2023b) and using both Nvidia A1 s100s and H100s. We perform two trials for each experiment.
Evaluation. We evaluate models on 33 different downstream question-answering tasks using the MosaicML evaluation gauntlet (MosaicML 2023a). Before averaging the accuracy across tasks, normalize the accuracy on each task by the baseline random guessing Specifically, we normalize the accuracy of each individual task an Sm Tm where am is the accuracy of the model and a, is the expected accuracy of random guessing, We report the average normalized accuracy for each task category as well as the average normalized accuracy across task categories. More details on tasks and task categories are listed in Section 9
3.2 PERPLEXITY-BASER DATA PRUNING IMPROVES DOWNSTREAM PERFORMANCE
If a certain range of perplexities good heuristic for data quality, training on that perplexity-prune -prunned-- subset should improve downstream performance. We sweep across pruning selection criteria and selection rates (Section 7 and find that the best settings are select high-perplexity samples at 50% rate for the Pile and to select medium-perplesn most performant pruning settings to baseline models trained on the original datasets without pruning in Table [ Across all datasets and model sizes, models pretrained on the perplexity pruned version of the dataset significantly outperform the baseline model on average. Specifically, perplexity- data pruning outperforms the average downstream performance of no pruning for IB models by 1.89 and 1.51 for the Pile and Dolma respectively, and improves the performance of 3B models by 2.04 and 0.59 for the Pile and Dolma respectively. These results suggest that the perplexity of small model provides strong signal of data quality for a much larger model, as training on the data selected by the small model leads to significant downstream performance improvements.
samples at 50% rate for Dolma. We compare the
Not to he confused, the random accuracy normalization is different from the normalized accuracy reported by the EleutherAl LMM Evaluation Harness) which normalizes hased the Byte-length of the response.
4

---

Published as a conference paper at ICLR 2025
(4)
Pile 1B Parameters
(6)
Pile 38 Parameters
Avg. Normalzed Accuncyy
14
Norma
12
Baseline High Perplexity
Baseline
High Perplexity Selected
Taken Duration (Billion)
Taken Duration (Billion)
10
Dolma 1B Parameters
(4)
Dolma 3B Parameters
14
Avg. Normalized Accuracy
Avg. Normalized Accuracy'
36
30
Baseline
Baseline
Medium Perplexity Selected
Medium Perplexity Selected
Token Duration (Billion)
Token Duration (Dillion)
Figure Average normalized task accuracy evaluated intermittently throughout pretraining for each dataset and model size investigated. Perplexity-based data pruning leads to an improvement in performance for all intermediate training steps evaluated.
3.3 PERPLEXITY-BASED DATA PRUNING IMPROVES TRAINING EFFICIENCY
Since perplexity- based data pruning improves the final performance of models, also investigate how pruned data affects the training dynamics models. Specifically, we investigate whether training perplexity pruned data enables models achieve the as models trained on the unpruned data in training fewer steps. We plot the average downstream performance of partially trained checkpoints from the IB baseline and perplexity pruned models in Figure [ Perplexity pruning outperforms the baseline model for all intermediate pretraining durations evaluated. Furthermore, perplexity pruned models reach the same average normalized accuracy as the baseline models in 1.31 x and 1.45x fewer steps for Pile 1B and 3B respectively and in 1.29x and 1.14x fewer steps for Dolma IB and Dolma 3B respectively. These results demonstrate that the resulting high-quality data from perplexity-basee data pruning enables faster learning which can leveraged achieve the same downstream performance as training on unpruned data with fewer pretraining steps.
downstream performance
3.4 PERPLEXITY BASED DATA PRUNING FOR OVER- EVER-TRAINED MODELS
recent trend with LLMs has been to over-train models by training them on more tokens than the Chinchilla optimal number of tokens (Touvron et al.) 2023 Gadre et al., 2024). As our work targets the data component of LLM pretraining, we investigate the hypothesis that over- over-training would be more beneficial for models trained on perplexity pruned datasets as the data is of higher quality. We test this hypothesis by training a 1B parameter model for 130B tokens, which is 5x the Chinchilla optimal number of tokens. We evaluate the downstream performance of each over-trained model in Table The main observation is that while the absolute gain average downstream normalized accuracy from perplexity-basee data pruning the Pile is similar for both compute optimal and
5

---

Published as a conference paper at ICLR 2025
Downstream task performance for Chinchilla Optimal and 5x over-trained data budgets. The "Improvement Over Baseline" column refers to the gain observed from perplexity pruning as compared to the baseline trained in the same setting.
Table 2:
Pruning Method
Average
Improvement Over Baseline
IB Parameters Trained on High Perplexity Pile Chinchilla Optimal
15.62
1.89
5x TrainTr
18.83
1.74
1B Parameters Trained Medium Perplexity Dolma
Chinchilla Optimal
15.35
1.51 0.84
5x Over-Trained
18.67
over-trained models, the gain decreases for Dolma when over-training. On the Pile, we find that the gain from perplexity pruned data is similar in the compute optimal regime and the over-trained regime: we see a gain in average performance of 1.89 when training compute optimal and gain of 1.74 when over- training. On Dolma, the gain from perplexity pruned data decreases in the over-trained regime: we see gain of 1.51 when training for a compute optimal duration but this decreases to gain of 0.84 when over-training. These results show that while the higher quality data resulting from perplexity-basee data pruning does still lead to improvement in downstream performance in the over-trained regime, there is not a relative increase in downstream improvement over the baseline when over-training.
3.5 PERPLEXITY- BASED DATA PRUNING FOR THE DATA CONSTRAINED REGIME
Our experiments far conducted in the setting where there exists sufficient abundance of data such that even after pruming with the desired selection rate there are enough data points to fill the desired token budget without requiring any data be repeated. However, there are many training settings that do not fall under this data-abundant regime. Consequently, we evaluate how perplexity-
data pruning performs when the number of tokens is constrained, and pruning induces greater number of repetitions of the data. For each dataset, we vary the available data such that training for a Chinchilla optimal number of tokens requires a different number of repetitions. Specifically, we investigate data budgets that require (0.5, 5, 1, 2, 4, 8 repetitions to reach the Chinchilla optimal number tokens As each number of repeats refers to the total number of tokens available, for all pruming experiments the number of repetitions after pruming is actually greater by factor of 1 since we prune the available tokens according to rs, the selection rate. Since all models use a selection rate 0.5, the models trained on the pruned data see the data for 2 more repetitions.
We plot the average downstream performance On both the Pile and Dolma, we find that training on perplexity pruned data yields an improvement for up two repetitions. These results suggest that perplexity-based data pruning still provide performance gains for some degree of data constraint. Furthermore, our results replicate the findings of Muennigofff al. (2023) that more than four repetitions yields negligible gains. Specifically, the bascline model without pruning maintains commensurate performance for up to four repetitions. Similarly, models trained on perplexity-prung data maintain commensurate performance for up to two repetitions through the base data, which corresponds to four repetitions after pruning. That training on repeated perplexity-prut -rruned--pned data leads diminishing gains after four repetitions post- pruning suggests that the higher quality data resulting from pruning does not change the point for which repeating data yields diminishing improvements in performance.
function of the number of repetitions in Figure 2
3.6 UPSTREAM PERPLEXITY IS NOT A RELIABLE EVALUATION METRIC FOR DATA PRUNING
As previous works have used the perplexity of the model on a test split of the pretraining dataset as an approximation downstream performance, wanted to explore how well such perplexity-basee
Repeat=0.55 means that the available number of tokens is twice the training budget, i.c. the data-ahundant setting

---

Published as a conference paper at ICLR 2025
(+)
Pile 1B Parameters
18)
Dolma 18 Parameters
Baseline
Baseline
Nemmalzed Accurscy
Perplexity Selected
Perplexity Selected
32
32
11
11
20
Repeats
Repeats
Downstream task performance as a function of available dataset size. The number of repeats denotes the number of repeats over the dataset necessary to achieve the Chinchilla optimal number of tokens. Training on perplexity pruned data leads to an improvement for up to two repeats on both the Pile Dolma.
Figure 2:
Table 3:
Performance as evaluated by perplexity on test split of the original dataset as well as average normalized task accuracy for billion parameter final models trained on the Pile. The model trained on pruned data worse pretraining test split perplexity even though it significantly improves average downstream normalized accuracy.
Pruning Method
Test Set Pplx. (L) Downstream Task Avg. (t)
IB Parameters Trained on Pile
No Pruning (Baseline) High Perplexity Selected
7.83
13.73
8.51
15.62
1B Parameters Trained on Dolma
No Pruning (Baseline)
13.53
13.84
Medium Perplexity Selected
14.33
15.35
evaluations agree with downstream performance for data intervention techniques. Pruning performs an intervention on the dataset, making models trained on the pruned dataset biased estimators of the original data distribution. Therefore, it is unlikely that the performance on the original data distribution a fair evaluation of model quality. We compare the test set perplexity and average downstream performance for billion parameter models trained the original and pruned version of the Pile and Dolma in Table 3 For both the Pile and Dolma, training on perplexity pruned data significantly worsens perplexity performance is significantly improved. This result suggests that test set perplexity may not always be sound metric for data pruning work and that researchers should instead directly evaluate on downstream benchmarks.
test split of the pretraining data, while the average downstream
4
UNDERSTANDING THE EFFECTS OF PERPLEXITY-BASER PRUNING
In this section, we investigate how data pruning works by exploring some of the properties of perplexity-based pruning.
4.1 How ARE REFERENCE PERPLEXITIES DISTRIBUTED
In order to better understand how perplexity-bante -based data pruning works, we investigate the distribution of the computed reference model perplexities for each dataset. For each dataset, we randomly sample 10% of the calculated perplexities and perform kernel density estimation to estimate the distribution of log perplexities for given dataset. We repeat this procedure for the optimal pruned version the dataset. We plot the resulting estimates of the log perplexity distribution in Figure 3] We find that
the log perplexity distribution for the Pile is multimodal and asymmetric, while for Dolma and is unimoda and symmetric.
7

---

Published as a conference paper at ICLR 2025
64)
Pile Log Perplexity Distribution
(6)
Dolma Log Perplexity Distribution
2.00
No Pruning
No Pruning
High Perplexity
Perplexity
1.50
Denety
Deneh
1.22
1.00
0.73
0.50
0.25
0.00
Log Perplexity
Log Perplexity
Figure 3: Distribution of sample perplexities as evaluated by the reference model for the Pile and Dolma. We show both the original distribution the full dataset without pruning well the distribution after applying the optimal perplexity-based data pruning technique for a given dataset.
(n)
Pile Domain Composition
(b)
Dolma Domain Composition
No Pruning
No Pruning
Prepartion of Dataset (N6)
High Perplexity
Prsportion Dataret (86)
Perplexity
0.20
0.5
0.15
0.10
0.2
0.1
0.00
plac
Aomon cent sane
(ua) weteees aag
8
bode
wid
Domains
Domains
Proportion of the total dataset each domain makes up before and after pruning. For all datasets, pruning tends to select more samples from general web domains while leaving out samples from highly specific domains.
Figure 4:
4.2 How PRUNING AFFECTS DOMAIN COMPOSITION
We can also interpret the effect that perplexity-basee data pruning has on dataset by examining how pruning affects each domain's 's proportion of the total dataset. We plot the pre and post-pruning domain compositions for the Pile and Dolma in Figure 4 Interestingly, for all datasets pruning increases the proportion of data coming from web-scraped domains while decreasing the proportion of data coming from highly specific technical domains such as code or scientific papers. This trend more pronounced in the Pile, where the proportions of Pile-CC and Open WebTee2 nearly double, while the proportions of domains such as Pubmed Central, ArXiv, and Github are reduced by at least factor of three. Future work should investigate how perplexity based pruning affects model's performance on downstream tasks that are in the same category as the highly pruned domains.
5 RELATED WORK
Classical methods for pruning text data. In order to improve the quality of raw web scrapes, which often contain very noisy samples, pruning via quality filtering has become a common practice. Simple rules-based methods have been employed to prune datasets by filtering out low-quality samples according to some hand-crafted heuristic such as whether the text contains prohibited words, is predominantly English, etc. (Bane al.) 2022 Raffel al. 2020 Rae al. 2022 Penedo et al.) 2023). N-gram perplexity-bai based methods, in which an n-gram model is first trained on a high quality, curated corpus and then used score another corpus, have also been applied filter text
8

---

Muennighoff et al., Published as a conference paper at ICLR 2025
data (Moore & Lewis|
2010 Axelrod 2017
Gao 2021
Laurençon et al.) 2022
2023). Although our method also uses perplexity to prune data, it does so in a very different manner. In n-gram perplexity pruning, perplexity is used to estimate whether new text is in distribution as compared to the currated text the n-gram was trained on, while in our model-based perplexity pruning, the reference model is trained
the same distribution of text and the perplexity is more akin to an estimate of the difficulty of an example. In this work, the datasets we leverage already have some basic rules-based pruning applied, and as such, the method we investigate is largely complementary to these existing techniques.
Neural network based methods for pruning text data. Recently, there has been much interest in using neural networks to compute metrics that can be used to intelligently prune datasets. A common technique in this family of methods is using a model to sample high-quality data from large datasets based on the sample's similarity to a curated high-quality corpus that serves as target distribution (Feng et al.) 2022 Xie et al.] 2023b). Xie et al.] (2023a) also consider how to use a small reference model to prune pretraining data for much larger model, by using a small reference model to learn the optimal weighting of domain proportions to maximize the "learnability" of the resulting dataset. Pruning based on the difficulty or loss of sample has previously been explored for text data, but the majority of such work focuses on curating data for finetuming (Swayamdiptai
et al. 2020 Attendu & Corbeil 2023; Coleman al., 2020 Mindermann et al. 2022 Mekala et al. 2024). Marion et al. (2023); : Yang et al. (2025) ; (Sachdeva et al. (2024), however, investigate multiple model-based sample difficulty heuristics for pruning pretraining text datasets. Although we use the same method for pruning text pretraining datasets as Marion et al. (2023), our analysis differs
substantially as we evaluate model quality based on downstream metries and extend our analysis to multiple different dataset compositions which enables us to conclude that the reference model can be smaller than the final model.
Data pruning on vision tasks. While data pruning becoming more and more relevant with large amounts of text data, it has also been extensively applied in the vision domain (Paul et al. 2 Toneva et al. 2018 Park et al., 2023). These works often prune data points based on their loss or gradients during training (Killamsetty et al. 2021] Mirzasoleiman et al. 2020). Model-basee methods
have also been leveraged for image data pruning (Fang et al.) 2024] Schuhmann et al. 2222 }. Note that in the literature, data pruning is also sometimes referred to as coreset selection (Guo et al. 2022).
More recently, Park et al. (2022) show that, somewhat surprisingly, active learning (Castro & Nowak 2008) based algorithms tend to outperform most data subset selection algorithms. In the context of contrastive learning, hard-negative mining has been effective as a data pruning method (Kalantidis) let al. 2020 Robinson et al., 20200] /Zhang& & Stratos 2021). Recently, Goyal al. (2024) investigated scaling laws for training on pruned data in the context of vision models.
6 CONCLUSION
In this work, we conduct an empirical investigation of the impact that perplexity- data pruning has on model performance. We demonstrate that small reference models can be used to prune the data of models with up to 30x more parameters, leading to both significant downstream performance improvements and increased training efficiency. We then investigate perplexity-based data pruning in two non-standard settings: the over-trained and data-constrained regimes. We find that for both settings, training on perplexity pruned data can outperform training on unpruned data, demonstrating that perplexity-baa -ased data pruning is widely applicable and extensible technique. We also investigate upstream metrics for evaluating data pruning techniques and provide an example where evaluating models based on their perplexity on the test split of the pretraining dataset does not align with evaluating based on downstream model performance. One limitation of our work is that while domains such as code are largely pruned, we do not have the computational budget required for training models of sufficient size to test the coding ability of models on benchmarks such as HumanEval (Chen et al., 2021). It remains important future work to scale perplexity pruning to larger models such that the impact of pruning specific domains can be better understood. Our work takes a key step towards establishing perplexity- data pruning as a technique in the modern data researcher's 's toolkit.
9

---

Published as a conference paper at ICLR 2025
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10.18653/v1/N19. 1245.
htthhtps:// http///aaaaathooooooo wtg/N19-1 1245
Quentin Anthony, Stella Biderman, and Hailey Schoelkopf. Transformer math 101.
Jean-michel Attendu and Jean-philippe Corbeil. NLU on data diets: Dynamic data subset selection for NLP classification tasks. In Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schuster, and Ameeta Agrawal (eds.), Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pp. 129-146, Toronto, Canada (Hybrid), July 2023. Association for Computational Linguistics. doi: 11811/81/22223//////s 9. URL wttp:httt://tt //ac:://cclaaa http///caanttooooogooo htsp:oooy.ooro2222.ssssaaaa..a.. sustainlp-1.
Amittai Axelrod. Cynical selection of language model training data. arXiv preprint arXiv:1709.02279 2017.
Fred Bane, Celia Soler Uguet, Wiktor Stribizew, and Anna Zaretskaya. A comparison of data filtering methods for neural machine translation. In Janice Campbell, Stephen Larocca, Jay Marciano, Konstantin Savenkov, and Alex Yanishevsky (eds.), Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pp. 313-325, Orlando, USA. September 2022. Association for Machine Translation in the Americas. URL htt.thts:///
amta- 22
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Rui Castro and Robert Nowak. Active learning and sampling. In Foundations and Applications of Sensor Management, pp. 177-200. Springer, 2008
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. 2021.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems, 36, 2024.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computationnal Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pp. 2936, Minncapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: (0.18653/v1/119.1 1300. URL httts::/aaac..c antho wwt.ooog...o...111111
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: (00:: 05557v1. 2018.
10

---

Published as a conference paper at ICLR 2025
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Perey Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representattinns 2020. URL https:
//openreview. net. (o
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference Learning Represen- tations, 2024, htthttt::://c/t
ptttt:::///penrevieee wtw.rreeewweeeeeoouuuuuu. 6ng
Yukun Feng. Patrick Xia, Benjamin Van Durme, and João Sedoc. Automatic document selection for efficient encoder pretraining. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 9522-9530, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 111.//633/1/2222222 122.6666////222.mmmmllp main.647. URL attps::/aclanthhooorr org/ 2022. www.p-mannnn......emeemmmmmmmawm 647
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang. Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and downstream tasks. arXiv preprint arXiv:2403.085400 2024.
Leo Gao. An empirical exploration in quality filtering of text data, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Sachin Goyal, Pratyush Maini, Zachary Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data filtering-date curation cannot be compute agnostic. arXiv preprint 27X11:2404.07177, 2024.
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL
net/ / forum? ?id=
d7KB jmI 3GmQ,
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoe, Au- relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. httpaaaeerreelleeeepeepppppp ciates, Inc., 2022. UtL/atttsss/pprrreeeinn....eooep 220.://..0//2aaff338000011111110
Curran Asso-
Aditi Jha, Sam Havens, Jeremy Dohmann, Alexander Trott, and Jacob Portes. LIMIT: Less is more for instruction tuning across evaluation paradigms. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https: ftttp:://openrevte
nwt. feet/forum?i0
[xtee01enz
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. IJCNLP), pp. 2567-2577, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DD 1259. URL htl.https:/aaa. http:p/://...aaaaooooooooooo////

---

Published as a conference paper at ICLR 2025
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33: 21798-2188 21809, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.08360 2020.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5445 5474, PMLR, 2021.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 17506-1733 17533. PMLR, 2023. URL btttps://proceee ngs. ml press/ #202/korbal html
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez- Dios, Aitor Soroa, Kyle Lo. Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Luccioni, and Yacine Jernite. The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset. In sixhh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL htttt::: / /wpp//ppeeeeie.eeeoo/eeee. www.:eneeeffouree
Id=üoEw6Ki
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirleenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-aligmment with instruction backtranslation. In The Twelfth International Conference Learning Representations, 2024. httttt::///ppmet/m
ew wwt. forum?
id= (HJRRST]
Jian Liu, Leyang Cui, Hammeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere (cd.), Proceedings of the Twenty-Nint International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3622 3628. ijeai.org, 2020. doi: 1022966/ICAA222225550 URL. https //doi. 0. 24963/ feaii 2020/502
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less more: Investigating data pruning for pretraining LLMs scale. In NeurlPS Workshop on Attributing Model Behavior at Scale, 2023. URL
net wwt.:nntttnooeeeeee www.oormmui..
XUIYn3Jo5T
Dheeraj Mekala, Alex Nguyen, and Jingbo Shang. Smaller language models are capable of selecting instruction-tunne training data for larger language models. arXiv preprini arXiv:2402. 10430, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? dataset for open book question answering. In Conference Empirical Methods in Natural Language Processing, 2018. URL https: ://p:::://p....// semanticscholar. ozrg
CorpusTD: :218::5517777
12

---

Published as a conference paper at ICLR 2025
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15630-15649, PMLR, 17-23 Jul 2022. URL html
https: http:::/pprccedinnne
mlr press wtw.l.pp.ee...../...... https///wwmmmmmmmmmmmmmmnn
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovee. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Jan Hajič, Sandra Carberry, Stephen Clark, and Joakim Nivre (eds.). Proceedings of the ACL 2010 Con- ference Short Papers, pp. 220-224, Uppsala, Sweden, July 2010. Association for Computational
Linguisties. URL https: http://aaaaahhhhoooo logy. org/ 101//110-20000 2041
MosaicML. Llm evaluation scores, 2023a.
URL
https: www mosai was.slmmmmmm..... com/
11m-eve eva 1 iuation
MosaicML. Lim foundry. htt.h//wwww..gtt////g.a..aooo//n
http:///tthhb..commmmsaaaam/////
foundryoudrrrrrrr 2023b.
MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023c.
URL httph: http://ww.waaati.....o.oooooaaaa www.c/////o...o.
aatatabbicks
7b
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirty- Conference on Neural Information Processing Systems, 2023. URL
https://cpenreviie wwt.eewwweeeeeeeeee wwtp:/rummtooeeeeeeeoo
J3BuTr
OpenAl. Tiktoken: A fast bpe tokeniser for use with openai's 's models. https://github. openas/Eiktoken 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1111888111/11111111 11PP1111114 144. UtLtttps::///a http://://.....a.aaooooo....////
Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee, Active learning is strong baseline for data subset selection. In Has it Trained Yer? NeurlPS 2022 Workshop, 2022.
Dongmin Park, Seola Choi, Doyoung Kim, Hwanjun Song, and Jae-Gil Lee. Robust data pruning under label noise via maximizing re-labeling accuracy. arXiv preprint arXiv:231 1.222311000 (000:231,.0002,, 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Thirth-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. net forum? id=kM5eGedCzq
URL
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young. Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen. Siddhant Jayakumar, Elena Buchatskaya. David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro,
13

---

Published as a conference paper at ICLR 2025
Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baph cun--Bptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autmme, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis and insights from training gopher. 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1677, 2020. URL
http: //mtp: wwt.ooog/gppp....... wwtp/v/1/////// 00021/200074.ttmm htm
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/ D16- 1264.
Joshua Robinson, Ching-Yau Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives:
An evaluation of commonsense causal reasoning. In Logical Formalizations of Commonsense Reasoning, Papers from the 201 AAAI Spring Symposium, Technical Report 1-06, Stanford.
California, USA, March 21-23, 2011. AAAI, 2011. URL http:tp///// //wp:///wwwwww www. aaai orgg /
index.ppp.. pttp:://w....sll1ppppp///reee//e
Noveen Sachdeva, Benjamin Coleman, BhangChhng Kang, Jianmo Ni, James Caverlee, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. Improving data efficiency for recommenders and 1lms. In Proceedings of the 18th ACM Conference on Recommender Systems, pp. 790-792, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An
adversarial winograd schema challenge at scale. AAAI, pp. 8732-8740, 2020. htts//
//aaai. org/ o htssiindess php AAAT/ //rttittii.. è e/ ew 6399
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarzzyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXiv preprint 2tt.::::2111.22//11 14, 2021.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartiomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk
14

---

Published as a conference paper at ICLR 2025
Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt. Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez. Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang. Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolå, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang. Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed Francesca Happé. Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Loppe Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang. Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishhaan Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo. Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Ieo Gao. Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochandoo Louis-Philippe Morency, Luca Moschella, Lucas Lam, I Lucy Noble, Ludwig Schmidt, Luheng He, Luis Ciiveros-Coóón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana. Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng. Nathan Andrew Chi, Naycon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormllabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade. Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Techan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Aaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo- Hwan Lee, Spencer Torene. Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu,
Evans,
15

---

Published as a conference paper at ICLR 2025
Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg. Trenton Chang. Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang. Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions 2835-8856. URL hnttps://opereviiee www.e/wwweeeeeeeeeeeeeeeeeeeeeee net wwt//feoooonee forum? i dduuTII5Bvos
Muchine Learning Research, 2023. ISSN
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannanch Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275- 9293, Online, November 2020. Association for Computational I Linguistics. doi: 10. (0.18633/1/20020 emnlp-maing 746.. URL httpttt://aaaaaatl.ooo//
org/2020
746
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos. D4: Improving LLM a
pretraining via document de-duplication and diversification. In shirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2023. URI. https
7/openreview.o net
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: /8/2.05 2018
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:::://pprooediggg.s..m wwweeurpp..cpppppeeeeeeee waapeer_fieespppppee pespppppp//221/// 3de
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave CCNet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4014 Marscille, France, May 2020. European Language Resources Association. ISBN 979- 1795444-5546-3444 V/aclanthology.co
URL https
Thom Wolfe, Lewis Tunstall, and Patrick von Platen. Jeopardy dataset on hugging face hub. https 2022.
Sang Michael Xie, Hieu Pham, Xuanyi Dong. Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In seeeth- Conference on Neural Information Processing Systems, 2023a. URL.
nnt/ www.:/ettttte//.ete itt.//ormmmadaayyuuelliiill LXuByUeHhd
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Sevenhh Conference on Neural Information Processing Systems, 2023b. URL RRl.tttp:: fttt:p:://oppnreeeee hwtperiwwwweeeeeeeeeeeeeeeeeeeee wttp:/ooooonniddooorrooo eAu
16

---

Published as a conference paper at ICLR 2025
Table 4: Results from sweeping different selection criteria. We report the average normalized accuracy for each task grouping as well as across all tasks. While high perplexity selection is optimal for the Pile, medium perplexity selection optimal for Dolma. Bold results are within one standard error of the highest normalized accuracy.
Common
Language Under- stand- ing
Symholic Prob- lem Solving
Reading Com- prehen- sion
World
Pruning Method
Knowl- edge
Sense Reason- ing
Average
IB Parameters Trained on Pile
No Pruming (Baseline)
15.51
10.31
28.11
3.53
11.16
13.73
Low Perplexity Selected
11.14
5.76
18.66
3.54
8.72
9.56
Medium Perplexity Selected
16.12
9.01
28.1
3.41
10.86
13.5
High Perplexity Selected
18.18
12.75
33.2
3.36
10.63
15.62
IB Parameters Trained on Dolma
No Pruning (Baseline)
16.48
12.32
28.86
3.58
7.95
13.84
Low Perplexity Selected
16.13
10.1
27.28
3.45
7.85
12.96
Medium Perplexity Selected
17.98
13.03
31.87
3,44
10.41
15.35
High Perplexity Selected
16.65
13.12
31.14
3.15
8.55
14.52
Yu Yang, Siddhartha Mishra, Jeffrey Chiang, and Baharan Mirzasoleiman. Smalttoaagge (s2l): Scalable data selection for fine-tnning large language models by summarizing training trajectories of small models. Advances in Neural Information Processing Systems, 37:83465-83496, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Wenzheng Zhang and Karl Stratos. Understanding hard negatives in noise contrastive estimation. arXiv preprint arXiv:2104.0624 2021.
7 FULL DATA PRUNING SETTINGS SWEEP
In this section, we report the results of sweeping over different perplexity-bant pruning setting configurations. In particular, for each dataset, we first sweep over the selection criteria to determine where from the distribution of perplexities samples should selected. Then, using the best selection criteria, we sweep the selection rate to determine how much we should prune.
Setup. We use the same training and evaluation setup as detailed in Section 3.1 We only perform the sweep pruning settings for billion parameter final models for computational budget reasons; however, we find that the best selection criteria at the billion parameter scale also confers a performance improvement the 3 billion parameter scale, as detailed in 3.2
7.1 FINDING THE BEST SELECTION CRITERIA
For each dataset, we first sweep the selection criteria while keeping the selection rate fixed at 50%. We report the performance of each selection criteria in Table 4 We find that on the Pile high perplexity selection works the best and on Dolma medium perplexity selection works the best, improving the average downstream performance by 1.89 and 1.51 respectively. An important observation from the sweep is that the best selection criteria from one dataset does not transfer to another dataset and may actually degrade performance compared to the baseline. Although medium -ertiuextttp selection is the best method Dolma, selecting medium-perplexit! samples the Pile leads decrease in the average downstream performance of 0.23 as compared to not performing pruning. These results inform that high and medium perplexity selection the optimal selection criteria for the Pile and Dolma respectively, and that the optimal selection criteria does not necessarily transfer between datasets with different domain compositions.
17

---

Published as a conference paper at ICLR 2025
Table 5: Results from sweeping different selection rates. We report the average normalized accuracy for each task grouping as well as across all tasks. Bold results are within one standard error of the highest normalized accuracy.
Common
Language Under- stand- ing
Symbolic
Reading
World Knowl- edge
Sense Reason- ing
Prob- lem Solving
Com- prehen- sion
Pruning Method
Average
1B Parameters Trained on Pile
25% Selection Rate
18.21
12.88
34.44
3.73
9.44
15.74
50% Selection Rate
18.18
12.75
33.2
3,36
10.63
15.62
75% Selection Rate
17.08
10.11
31.37
3.81
9.02
14.28
1B Parameters Trained on Dolma
25% % Selection Rate
17.94
12.16
31.63
3.58
8.91
14.85
50% Selection Rate
17.98
13.03
31.87
3.44
10.41
15.35
75% Selection Rate
18.2
11.78
29.96
3.32
10.82
14.82
7.2 FINDING THE BEST SELECTION RATE
Using the optimal selection criteria that we found for each dataset, we next investigate the best selection rate for each dataset. We investigate three different selection rates: 25% 50%, and 75%. We present the results for each selection rate in Table On the Pile, we find that there is significant difference in downstream performance for selection rates of 25% and 50%; on Dolma we find that selection rate of 50% achieves the best average downstream performance. For simplicity, to conduct the rest of the experiments in the paper using a selection rate of 50% on both datasets. Furthermore, we find that all the selection rates tested outperform the baseline of no data pruning as measured by average downstream performance. This suggests that the selection criteria has a greater impact the performance of pruning configuration than the selection rate.
chose
EFFICIENCY ANALYSIS
We now turn towards the total computational complexity of perplexity pruning. As our method does change the computation performed on the forward or backward pass, and only affects which tokens are trained on, we can analyze the compute requirements in terms of the total number of operations. We will approximate the cost of training as 6.V where N the number of parameters and D is the number of training tokens, and we will approximate the cost of computing a sequence's perplexity in inference mode as 2ND (Kaplan et al., 2020 Anthony al.). Assuming reference model of size Nier number of reference training tokens Dier, final model size Ninat- number of final tokens Danal, selection rate T. and fraction tokens for the pruned data to achieve the same performance the baseline F, the relative cost of perplexity pruning compared to the baseline is:
2.N Dink + 6N Dret + 6. Drinall
Nief
Dief
+
6.Vina Dinal
Ninal
3r,
Dinal
All our reference models are 125M parameters, trained for 26B tokens, and we use Ts Nu.. + Dat D
0.5 throughout. The IB models are trained on 26B tokens giving us Nuu 3r,
10
26x 10 0.16. For the Pile, F 0.76 giving relative cost of 0.16+0.76 0.92 26x10
1.3x
and for Dolma F = 0.78 giving us relative cost of 0.16 + 0.78 0.94.
The 3B models are trained on 54B tokens giving us New 3r. + Dmi 125x 10
Nut.
2
10 51x10
Dime
2.7 10
0.05. For the Pile, F 0.69 giving relative cost of 0.69 0.05 0.74 and for Dolma F 0.88 giving us a relative cost of 0.88 + 0.05 = 0.93.
As can be seen, perplexity pruning leads to total reduction in cost across all experiments. Further- more, the cost reference model training is typically amortized as it's only performed once per dataset. T.e., used the same reference models prune the data for both the 1B and 3B models.
18

---

Published as a conference paper at ICLR 2025
DETAILED EVALUATION SETUP
Jha et al. 2023) also use the MosaicML evaluation gauntlet to perform evaluations in their work. As such, with explicit permission from the authors, we exactly reproduce their text describing the tasks and tasks categories in the evaluation gauntlet. The following is from Section D of their paper:
The World Knowledge category includes the following datasets:
Jeopardy (2,117 questions that custom subset of the dataset originally obtained from Wolfe et al. (0022)
MMLU (14,042 four-choice multiple choice questions distributed across 57 categories Hendrycks et al. (2021)
BIG-bench wikidata (20,321 questions regarding factual information pulled from wikipedia) Srivastavaa et al. (0233
ARC easy (2,376 easy multiple choice middle school science questions) Clark et al. 2018
ARC challenge (1,172 hard multiple choice science questions) Clark al. 20188)
BIG-bench: misconceptions (219 true or false questions regarding common misconceptions) Srivastava et al. (023)
The Commonsense Reasoning category loosely assesses a model's ability to basic reasoning tasks that require commonsense knowledge of objects, their properties, and their behavior. It includes the following datasets:
BIG-bench Strategy QA (2,289 very eclectic yes/no questions on wide range common- sense subjects e.g "Can fish get "onsilitis?" [Srivastava et al. (2023)
BIG-bench Strange Stories (174 short stories followed by questions about the charac- ters Srivastava et al. (2023)
BIG-bench Novel Concepts (32 find-the common-concppll problems Srivastava et al. (2023)
COPA (100 cause/effect multiple choice questions) Roemmele et al. 2011
PIQA (1,838 commonsense physical intuition 2-choice questions) Bisk et al. 2020
OpenBook QA (500 questions that rely on basic physical and scientific intuition about common objects and entities) Mihaylov et (2018).
Language Understanding tasks evaluate the model's 's ability to understand the structure and properties of languages, and include the following datasets:
LAMBADA (6, 1.3 passages take from books we use the formatting adopted by OpenAl's version Paperoo et al. (2016)
HellaSwag (10,042 multiple choice scenarios in which the model is prompted with scenario and choose the most likely conclusion to the scenario from four possible options o
et al. (019)
Winograd Schema Challenge (273 scenarios in which the model must semantics
correctly resolve the anaphora in a sentence. The Eval Gauntlet uses the partial evaluation technique technique introduced Trinh Le 2019 Levesque al. (2012)
Winogrande (1,267 scenarios in which two possible beginnings of a sentence are presented along with single ending) (Saaauuch et al. (2020)
BIG-bench language identification (10,000 questions on multiple choice language identifica-
tion) Srivastava et al. (2023)
BIG-bench conceptual combinations (103 questions using made up words) Srivastava et al. (003)
BIG-bench conlang translation (164 example problems in which the model is given transla-
tions simple sentences between English and some fake constructed language) Srivastaval
et al. (2023)
19

---

Published as a conference paper at ICLR 2025
Symbolic problem solving tasks test the model's abilitt to solve diverse range of symbolic tasks including arithmetic, logical reasoning, algorithms, and algebra. These datasets include:
BIG-bench elementary math QA (38,160 four-choice multiple choice arithmetic word problems) Srivastava et al. (2023)
BIG-bench dyck languages (1000 complete-thee-sequence questions) Srivastava et al. (2023
BIG- BiG-bench algorithms (1,320 questions) Srivastava et al. (2023)
BIG-bench logical deduction (1500 four-choice multiple choice questions relating to relative ordering of objects) (Srivastava et al. (2023)
BIG-bench operators (210 questions involving mathematical operators) Srivastava et al.
(2023)
BIG-bench repeat copy logic (32 samples in which the model is required to follow some instructions for copying words/symbols)
Simple arithmetic with spaces (1000 arithmetic problems consisting of up to 3 operations and using numbers of up to 3 digits, developed by MosaicML)
Simple arithmetic without spaces (1000 arithmetic problems consisting of up to 3 operations and using numbers of up to 3 digits, developed by MosaicML)
Math QA (2,983 four-choice multiple choice math word problems) Amini al. (0199)
LogiQA (651 four-logical word problems) Liu et al. (2200)
The Reading comprehension benchmarks test model's ability to answer questions based on the information passage of text. The datasets include:
BIG-bench Understanding fables (189 short stories) Srivastava al. (2023)
Pubmed QA Labeled (1000 hand-labeled medical documents followed by related question for which the model must respond yes/no/maybo be) Jin et al. (2019)
SQuAD (10,570 short documents followed by a related question. The model is expected to output the exact correct answer) Rajpurkar et al. (2016)
BoooQ (3,270 short passages on diverse range of subjects followed by yes/no questions)
Clark et al. (2019)
9.1 EVALUATION PROCEDURE
To compute model performance the above datasets, the Eval Gauntler one of the following three ICL metrics for each dataset (from MosaicML's composer library).
1. LaConiexLeamiinnAAAAAooo
This metric uses the query, the corresponding correct answer and list of alternative answers measure model's prediction. If the model's response conditioned on the query starts with either the correct answer or with one of the alternative answers, it is considered correct. This used for question-ane -anstionnannswrring tasks such TriviaQA.
2. InContextLearringg MAcurra This metric tests model's ability to output precise set of tokens. model's 's output conditioned on a given query is judged to be correct only if the model's highest probability tokens match the correct sequence of tokens. This is used for language modeling tasks such as LAMBADA.
3. aaanreeeLeuminggMtippCCooooeeeee
This metric is used for testing model's ability to answer multiple choice questions accurately. It compares the respective perplexity of the query prepended to each of the possible choices, according to the model. the query-choice pair with the lowest per token perplexity is indeed the correct choice, then the model's output judged be correct. This used for multiple choice tasks such as HellaSwag, Winograd etc.
20


Reviews:
[{"review_id": "ZywBbWh82H", "reviewer": "Reviewer_qYWL", "summary": "The paper proposes that smaller language models effectively prune large datasets in a way that benefits the training of much larger model. Applying perplexity-based pruning techniques, they explore using a small model to filter high-quality subsets of data for training larger models. This approach is interesting because it’s a cost-effective alternative to using large models for pruning, and is applicable in real settings. The findings indicate benefits for downstream accuracy and training efficiency.\n\nThe paper demonstrates that a 125m parameter model can successfully prune data for large models and improve downstream task performance. The paper shows empirical results testing on The Pile and Dolma, two datasets with very different domain structures.\nThey also study the two settings of over-training and data-constrained setups and provide additional insights.", "questions": "How do you expect the results to scale on models larger than 3B parameters? \n\nHow does models' performance change on domains which are pruned the most?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The goal, and the process, and algorithm are defined and presented very clearly. Experiments cover multiple settings, with different model sizes and training algorithms. \nThe proposed method is super useful for researchers who investigate practical techniques for data curation, with insightful empirical results. \nExperiments include two very different dataset distributions, the Pile dataset and Dolma. The work shows thorough experiments for various selection rates and perplexity criteria, presenting strong evidence about settings in which perplexity pruning does and does not work.", "weaknesses": "Authors claim that datasets pruning increases the proportion of general domain data from web-scraped domains, and decreases the proportion of specific and technical domains. But it is unclear and counter intuitive why training on general domain data improves performance of models on benchmarks. I think the paper lacks analysis to explain this observation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HIsB4Qqu4J", "reviewer": "Reviewer_wjeV", "summary": "This paper presents a perplexity-based pruning method for reducing the size of pre-training datasets. The effect of pruning is evaluated through the performance on downstream tasks as well. Two datasets are used for evaluation: Pile and Dogma. The pruning efficacy is determined for over-trained and data-constrained regimes as well.", "questions": "- A quantized model may lead to better inference efficiency while calculating the perplexity. Was this considered while running the experiments?\n- High perplexity selection will also inevitably lead to the inclusion of a significant portion of the noisier examples in the overall dataset. How can we determine the proportion of such examples in the final dataset and exclude them reliably?\n- Minor typo (line 66): perplexity-basd -> perplexity-based\n- It would be useful to include the following closely related data pruning works in the related work section:\n    - https://arxiv.org/abs/2403.07384\n    - https://arxiv.org/abs/2402.09668", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper addresses an important problem of pruning the pre-training datasets to enable efficient training of LLMs.\n- The experiments are thorough and cover different dimensions of perplexity-based pruning. \n- The paper is well-written and the results are presented clearly. \n-  The findings are significant, as they show that perplexity-based data filtering can not only reduce the size of the pre-training datasets, it also leads to better performance on certain downstream tasks.", "weaknesses": "- The paper does not currently cover the computational complexity of the proposed pruning procedure. A few important questions that need to be considered in this regard:\n    - How do the computational requirements for perplexity-based pruning increase with the size of the dataset to be pruned?\n    - How does the cost of computing perplexity (before pruning) amortize over the efficiency improvements achieved while pretraining the model on the pruned datasets? \n- A discussion for choosing the right perplexity pruning method (low, medium, high) for the dataset should be included for the practitioners. From the experimental results, we can see that high perplexity selection performs better on Pile while medium perplexity selection is better for dolma. Can we extract any patterns from these results and other experiments that can be generalized to other datasets? \n    - For example, prior theory on data pruning for vision tasks shows that the optimal pruning strategy changes depending on the amount of initial data. When data is abundant, the better pruning strategy is to keep harder examples. In contrast, for smaller datasets, keeping the easier examples leads to better performance. [1] \n- The results show that test set perplexity may not always be a sound metric for evaluating a pruning strategy and that downstream evaluation is necessary. What should be the cheapest way of conducting the downstream evaluation of the correct perplexity pruning method, i.e., the one that can yield reliable results at a minimal cost? For example, could there be a small set of representative downstream tasks or metrics that could serve as efficient proxies for full downstream evaluation?\n\nReferences:\n\n[1] https://arxiv.org/abs/2206.14486", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vPqafKGHIJ", "reviewer": "Reviewer_wYdV", "summary": "The authors filter LLM pre-training data by using the perplexity of a smaller language model. They demonstrate that dataset filtering improves the [initial] learning curve of LLM pre-training.", "questions": "- Fig.4 is interesting, but I'm not sure how Fig. 3 is relevant in practice - could you clarify?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is well motivated. Except for some uncommon terminology that is explained in later sections like \"non-standard training regime\", \"over-training\" (which is not over-fitting) the paper is clearly written.", "weaknesses": "L186 suggests that the final models are (pre-)trained for a fixed number of steps, no matter the dataset size. This sets the stage for dataset filtering, since training on the full dataset may go through fewer epochs. It would be interesting to train for long enough to show convergence in the plots in Fig. 1.  The story would be more convincing if there is an offset between the blue and red curves even after convergence. In fact, the \"over-training\" experiment in Sec. 3.4 shows diminishing gains, so I can imagine that they disappear fully at some point. The method would still have merits (steeper pre-training curve), just not the ones claimed in the paper.\n\nNovelty. Perplexity-based pruning and countless variations of it are well-studied. The authors set their work apart from prior work in L058, but neither of the arguments (i)-(iii) (evaluation on downstream task, exploration of domain compositions, \"non-standard\" evaluation regimes) strike me as particularly strong.\n\nI don't think that Algorithm 1 is really helping clarity. 1-2 normal equations would be just as expressive and more concise.\n\nEdit: my point about novelty was unjustified - I have increased my scores after the rebuttal", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8BIMgXGP7t", "reviewer": "Reviewer_13P9", "summary": "This paper investigates whether a small model can be used to perform perplexity based data selection for a larger model. The key findings are that 1) a reference model with 30x fewer parameters compared to the larger model can be used to identify a subset of the training data which can improve the performance of the larger model relative to no pruning. 2) the filtered data subset can speed up training of the larger model, 2) the improvements carry over to some extent to over training and data constrained regimes, 3) ideal pruning criteria can vary by dataset e.g. for Pile, a high perplexity subset performs better while for Dolma, a medium perplexity subset works the best. The paper shows that test data perplexity is not a good indicator of the downstream task performance when using perplexity based pruning.", "questions": "* L290: \"These results show that while the higher quality data resulting from perplexity-based data pruning does still lead to an improvement in downstream performance in the over-trained regime, there is not a relative increase in downstream improvement over the baseline when over-training.\" It would be good to understand why this is the case since there are no repeats. \n* L314: \"That training on repeated perplexity-pruned data leads to diminishing gains after four repetitions post- pruning suggests that the higher quality data resulting from pruning does not change the point for which repeating data yields diminishing improvements in performance.\" This sentence is confusing and should be reworded.\n* In section 4.2, the paper presents results showing that the pruning affects data composition such that some domains (e.g. web) are oversampled compared to others (e.g. pubmed). It would be useful to perform additional analysis to understand why this is the case e.g. is it possible that the training split (L113) resulted in a smaller proportion of these domains for the reference dataset?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Describes a simple approach to improve the performance of large language models using perplexity based data filtration using a smaller reference model.\n* Presents useful results e.g. 1) filtration criteria varies by dataset type and 2) test set perplexity is not a good indicator of the downstream task performance.", "weaknesses": "* The main results (Table 1) do not include a random baseline i.e. what is the performance of a model trained on a subset of the data which has a similar size as the perplexity filtered buckets but is selected randomly?\n* The paper does not contain ablations on the size of the reference model and sensitivity of the results to the random split (L113) used for training the reference model. Though exploring this space is computationally expensive, it may be useful to present 1-2 additional data points.\n* It would be good to see some additional analysis to understand why a high perplexity set works better for one domain while a medium perplexity set works better for others.\n\nNote: The authors have addressed some of these concerns (random baseline/sensitivity to random split) in  the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)