SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training (2024)

Abstract: The effectiveness of large language models 001 (LLMs) is often hindered by duplicated data 002 in their extensive pre-training datasets. Current 003 approaches primarily focus on detecting and 004 removing duplicates, which risks the loss of 005 valuable information and neglects the varying 006 degrees of duplication. To address this, we pro007 pose a soft deduplication method that maintains 008 dataset integrity while selectively reducing the 009 sampling weight of data with high common010 ness. Central to our approach is the concept of 011 "data commonness", a metric we introduce to 012 quantify the degree of duplication by measuring 013 the occurrence probabilities of samples using 014 an n-gram model. Empirical analysis shows 015 that this method significantly improves train016 ing efficiency, achieving comparable perplexity 017 scores with at least a 26% reduction in required 018 training steps. Additionally, it enhances aver019 age few-shot downstream accuracy by 1.77% 020 when trained for an equivalent duration. Im021 portantly, this approach consistently improves 022 performance, even on rigorously deduplicated 023 datasets, indicating its potential to complement 024 existing methods and become a standard pre025 training process for LLMs. 026

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4011–4022 August 11-16, 2024 ©2024 Association for Computational Linguistics
The effectiveness of large language models001 (LLMs) is often hindered by duplicated data002 in their extensive pre-training datasets. Current003 approaches primarily focus on detecting and004 removing duplicates, which risks the loss of005 valuable information and neglects the varying006 degrees of duplication. To address this, we pro-007 pose a soft deduplication method that maintains008 dataset integrity while selectively reducing the009 sampling weight of data with high common-010 ness. Central to our approach is the concept of011 "data commonness", a metric we introduce to012 quantify the degree of duplication by measuring013 the occurrence probabilities of samples using014 an n-gram model. Empirical analysis shows015 that this method significantly improves train-016 ing efficiency, achieving comparable perplexity017 scores with at least a 26% reduction in required018 training steps. Additionally, it enhances aver-019 age few-shot downstream accuracy by 1.77%020 when trained for an equivalent duration. Im-021 portantly, this approach consistently improves022 performance, even on rigorously deduplicated023 datasets, indicating its potential to complement024 existing methods and become a standard pre-025 training process for LLMs.026
1 Introduction027
In recent years, the expansion of pre-training028 datasets has played a pivotal role in advancing029 LLMs (Raffel et al., 2023; Gao et al., 2020; Penedo030 et al., 2023). However, a large fraction of these031 datasets is derived from uncurated snapshots of the032 internet, resulting in a significant amount of du-033 plication. Such redundancy, particularly beyond034 certain levels, can severely impair the performance035 of LLMs (Hernandez et al., 2022). While repeti-036 tion under specific conditions may be beneficial,037
∗Work done during internships at Tencent AI Lab. † Corresponding author. ‡ Kai Zhang proposed the initial prototype of the method.
the marginal gains from additional computation di- 038 minish to zero over time (Muennighoff et al., 2023). 039 Thus, it is imperative to ensure that data repetition 040 is a deliberate choice rather than an unintentional 041 consequence. In light of this, data deduplication 042 has emerged as a critical procedure in the manage- 043 ment of pre-training datasets. 044
Most current data deduplication strategies can be 045 classified as hard deduplication methods, focusing 046 on identifying and removing redundant data. For 047 example, MinHashLSH (Leskovec et al., 2020), 048 a widely utilized method (Soboleva et al., 2023; 049 Penedo et al., 2023), approximates Jaccard similar- 050
1 4011
Large Row Dataset
n-gram Calculate commonness for each document ··· low high high low Commonness Sampling weight Train n-gram model
LLM
ity among samples by generating MinHash (Broder,051 1997) signatures and using locality sensitive hash-052 ing to map these signatures into multiple buckets.053 Samples are considered duplicates if their MinHash054 values exactly match in at least one bucket, indicat-055 ing they exceed a predefined similarity threshold.056 In the subsequent removal stage, a common prac-057 tice involves clustering samples across all buckets058 (for instance, if samples A and B match in one059 bucket, and B and C in another, then A, B, and060 C are considered a cluster) (Penedo et al., 2023).061 Within each cluster, only one sample is preserved.062
These methods face several principal limitations.063 First, the concept of duplicates within a set of sam-064 ples is symmetric. Randomly retaining one sample065 while discarding the others may introduce bias by066 ignoring the differences among them. Second, set-067 ting a specific threshold for duplication presents068 a challenge since the degree of duplication is con-069 tinuous. A high threshold might overlook near-070 duplicates that bear significant similarities, whereas071 a low threshold could result in the exclusion of072 valuable data. Moreover, data categorized as non-073 duplicates according to these thresholds are uni-074 formly treated, despite the variations in the degree075 of duplication among them.076
To address these limitations, we introduce a soft077
deduplication method (Figure 1). This method di- 078 verges from traditional practices by preserving the 079 entirety of the dataset and avoids the need for set- 080 ting thresholds to determine duplicates. We intro- 081 duce the concept of "data commonness", a metric 082 that quantifies the degree of duplication by measur- 083 ing the occurrence probabilities of samples using an 084 n-gram model. Samples with high commonness are 085 assigned a lower sampling weight, while those with 086 low commonness receive a higher weight. This 087 method reduces the risk of inadvertently discarding 088 valuable data and leverages the spectrum of data 089 duplication, offering a refined and comprehensive 090 perspective on data deduplication. 091
Our empirical analysis reveals that the proposed 092 method enables language models to achieve base- 093 line performance with at least 26% fewer train- 094 ing steps, ultimately leading to improved perfor- 095 mance on downstream tasks. It exhibits superior 096 temporal efficiency and outperforms existing meth- 097 ods in terms of effectiveness. Significantly, even 098 when applied to rigorously deduplicated datasets, 099 our method still delivers substantial improvements. 100 These results suggest that our approach can com- 101 plement existing methods and can be adopted as a 102 standard procedure in the pre-training of LLMs. 103
2
2 Related Work104
2.1 Data deduplication105
Research has revealed that many existing pre-106 training datasets contain a substantial number of107 duplicate samples (Lopes et al., 2017; Bandy and108 Vincent, 2021; Penedo et al., 2023). To explore109 the impact of duplicate data on model performance,110 numerous studies have been conducted on both111 general and domain-specific datasets (Allamanis,112 2019; Lee et al., 2022; Biderman et al., 2023; Xue113 et al., 2023). The results indicate that repetition at114 certain frequencies can significantly harm model115 performance (Hernandez et al., 2022). Although116 appropriate repetition under specific circumstances117 can be beneficial (Muennighoff et al., 2023), this118 should result from careful selection rather than be-119 ing an unintended consequence of data duplication.120
Therefore, data deduplication is crucial for pre-121 training large language models. Exact dedupli-122 cation is typically achieved through suffix arrays123 (Manber and Myers, 1993). MinHash (Broder,124 1997) and SimHash (Charikar, 2002) are widely125 used fuzzy deduplication methods. In recent re-126 search, some studies have shifted towards semantic-127 based deduplication. Abbas et al. (2023) and128 Sorscher et al. (2023) utilize pre-trained embed-129 dings for clustering to remove semantically redun-130 dant data. Tirumala et al. (2023) combines both131 methods.132
2.2 Data reweighting133
Adjusting the significance of training samples134 through data reweighting has proven to be an ef-135 fective strategy for enhancing model performance,136 either through modifying loss function weights or137 changing the sampling probabilities. Focal Loss,138 as introduced by Lin et al. (2018), employs a soft139 weighting scheme to allocate higher weights to140 more challenging samples. Ren et al. (2019) assign141 weights to training samples based on the direction142 of their gradients. In DSIR (Xie et al., 2023b),143 sampling based on importance weights is utilized,144 allowing the training data to align with the distri-145 bution of high-quality corpora such as Wikipedia.146 DoReMi (Xie et al., 2023a) explores an automated147 scheme for determining the sampling weights of148 different data sources.149
3 Method 150
3.1 Hard deduplication 151 Hard deduplication methods identify and remove 152 duplicate samples. This process can be seen as 153 partitioning the dataset D into numerous distinct 154 subsets Di, such that D = ⋃k i=1Di. Each of these 155 subsets contains samples deemed to be duplicates 156 based on a specific similarity threshold. Within 157 each subset Di, only one sample, denoted as xi, is 158 retained, while the rest are discarded. If a subset 159 consists of only one sample, it indicates that this 160 sample has no duplicates within the dataset. 161
In the context of pre-training LLMs, the funda- 162 mental training goal is to maximize the log likeli- 163 hood of the training data. Incorporating hard dedu- 164 plication into this process can be formulated as: 165
L = ∑
x∈D I(x) logP (x|Θ),
I(x) = { 1, x ∈ {x1, x2, · · · , xk} 0, otherwise
(1) 166
where L denotes the log likelihood function, Θ 167 represents the model parameters. Despite its utility, 168 hard deduplication may inadvertently omit valuable 169 data and fail to adequately consider the degree of 170 redundancy. 171
3.2 Soft deduplication 172 To address the limitations of hard deduplication, we 173 propose a soft deduplication method. This method 174 employs sampling weights W (x), allowing for a 175 nuanced handling of data redundancy by adjusting 176 the influence of each sample on the model based 177 on its commonness: 178
L = ∑
x∈D W (x)·logP (x|Θ), W (x) ∈ (0, 1). (2) 179
We assume that the sampling weight of sample x 180 can be represented as follows: 181
W (x) ∝ 1 p(x) . (3) 182
Here, p(x) denotes the occurrence probability of 183 sample x, serving as a direct measure of its com- 184 monness. This probability-based measure effec- 185 tively captures the degree of duplication of each 186 sample. This approach ensures that samples with 187 higher commonness are assigned lower weights, 188 thus mitigating the impact of duplicates without 189 discarding potentially valuable information. 190
3
3.3 Implementation of commonness191 calculation192
In practical implementation, we leverage an n-gram193 model to process data, achieving high temporal194 efficiency in calculating the commonness of each195 data sample (Figure 2). This process consists of196 three steps.197
1. Tokenization. The n-gram model assumes198 that the appearance of a word is determined199 by the previous n − 1 words. The first step200 is to tokenize the original corpus. We use the201 same tokenizer as the pre-training models for202 consistency.203
2. Train n-gram model. In the training process204 of an n-gram model (where n = 4), maximum205 likelihood estimation is used to calculate the206 probability of each n-gram. We empirically207 choose n = 4 after conducting early exper-208 iments. To alleviate the issue of data spar-209 sity, we employ the Kneser-Ney smoothing210 technique (Ney et al., 1994). We utilize the211 KenLM toolkit* to accomplish this step.212
3. Calculate commonness. We utilize the ob-213 tained n-gram model to compute the common-214 ness (measured by the occurrence probability)215 for each data sample. For a given x containing216 N tokens,217
p(x) =
( N∏
i=1
P (wi|wi−1, . . . , wi−n+1) ) 1 N .
(4)218 By employing the geometric mean, the influ-219 ence of sample length can be eliminated.220
3.4 Approximate sampling for large-scale221 data222
Due to the vast volume of data, directly assigning223 individual sampling weights to each data point is224 impractical. To overcome this, we introduce an225 approximate method for data sampling that seg-226 ments M samples into K categories. This process227 initiates by sorting the M samples in ascending228 order of data commonness, followed by dividing229 the dataset into K distinct segments according to230 K quantiles. For the k-th segment, the sampling231 weight Wk is determined by the k-th quantile, pk,232 as follows:233
Wk = C · ( 1
pk
)T (5)234
*https://kheafield.com/code/kenlm
where T is a hyperparameter that adjusts the sam- 235 pling weight and C is a normalization constant, 236 which ensures that the sum of the weights across 237 all segments equals one. 238
4 Experimental Setup 239
4.1 Datasets 240
We conduct experiments on different versions of 241 the Common Crawl dataset, which is a compre- 242 hensive and publicly accessible collection of data 243 obtained through web crawling. 244 RedPajama CommonCrawl is a subset of the 245 RedPajama dataset (Computer, 2023). It involves 246 the original Common Crawl data undergoing pro- 247 cessing through the CCNet pipeline (Wenzek 248 et al., 2019). This dataset has been subjected to 249 paragraph-level deduplication; however, it has not 250 undergone rigorous deduplication procedures. 251 SlimPajama CommonCrawl is a subset of the 252 SlimPajama dataset (Soboleva et al., 2023). The 253 SlimPajama dataset represents a further refined iter- 254 ation of the RedPajama corpus, boasting enhanced 255 data cleansing procedures and the implementation 256 of MinHashLSH (Leskovec et al., 2020) for more 257 effective deduplication. 258 Falcon RefinedWeb is introduced as a pre-training 259 dataset for the Falcon series (Penedo et al., 2023; 260 Almazrouei et al., 2023). It undergoes rigorous 261 deduplication processes using exact matching and 262 MinHashLSH. 263
4.2 Model training 264
In the experiments, we employ the same model 265 architecture as the LLaMA (Touvron et al., 2023) 266 series. Our models are configured with 1.3B pa- 267 rameters, incorporating 16 attention heads and 24 268 layers. The hidden size is set to 2048, and the 269 dimension of feed-forward network is 5504. Pre- 270 vious research has demonstrated the feasibility of 271 pre-training validation on models of this scale (Tiru- 272 mala et al., 2023; Xie et al., 2023a). All models are 273 trained from scratch to 40B tokens. The batch size 274 is 512, and the training sequence length is 1024. 275 The learning rate is decayed from 2e-4 to 2e-5. 276
4.3 Baselines 277
Our primary baseline is defined by directly training 278 on a dataset that has been randomly sampled to en- 279 compass 40B tokens. In our study, we implement 280 the SoftDedup method across all three datasets, 281
4
facilitating a comparative analysis between our pro-282 posed technique and the established baseline for283 each dataset. Furthermore, for experiments con-284 ducted on the RedPajama CommonCrawl dataset,285 the SlimPajama CommonCrawl, which employs286 MinHashLSH for deduplication directly on it, is287 considered a hard deduplication baseline.288
4.4 Evaluation metrics289 We evaluate the models by measuring their perplex-290 ity on the test sets and their few-shot performance291 on downstream tasks.292 Test set perplexity. Our test sets come from the293 Pile (Gao et al., 2020) and SlimPajama (Soboleva294 et al., 2023). The Pile test set consists of 22 sub-295 sets, including BookCorpus, DM Mathematics, and296 others. SlimPajama also includes 7 subsets, such297 as Common Crawl, C4, and GitHub. We measure298 the perplexity of the models on each sample and299 report the average for each subset. We investigate300 data leakage and remove the contaminated samples.301 Specifically, if a sample in the training set has more302 than 50 tokens of overlap with a sample in the test303 set, the former will be removed from the training304 set.305 Downstream task accuracy. In order to further306 evaluate the performance of the models, we mea-307 sure their accuracy on 12 downstream tasks. The308 tasks cover the models’ abilities in reading compre-309 hension (SQuADv2 (Rajpurkar et al., 2018), Trivia310 QA (Joshi et al., 2017)), commonsense reason-311 ing (ARC easy and challenge (Clark et al., 2018),312 WinoGrande (Sakaguchi et al., 2019), HellaSwag313 (Zellers et al., 2019), PIQA (Bisk et al., 2020),314 Social IQa (Sap et al., 2019)), world knowledge315 (WebQuestions (Berant et al., 2013), NQ Open316 (Lee et al., 2019)), and contextual understanding317 (LAMBADA standard and openai (Paperno et al.,318 2016)). The evaluation of downstream tasks is pri-319 marily accomplished through the utilization of the320 lm-evaluation-harness (Gao et al., 2023).321
4.5 Hyperparameter impact analysis322 In exploring the impact of hyperparameters on our323 method, we focus on two key hyperparameters:324 the number of data partitions (K) and the weight325 parameter (T ).326 Our experiments involves varying levels of data327 partition granularity by dividing the dataset into 10,328 20, 50, and 100 segments. Regarding weight as-329 signment, we modify the hyperparameter T within330 Equation 5 to alter weight disparities. We investi-331
gate three configurations that result in maximum- 332 minimum weight differences of approximately 2- 333 fold, 5-fold, and 10-fold, respectively. A larger 334 disparity exerts a greater suppression on data with 335 higher commonness. 336
5 Results 337
In this section, we provide a detailed report of the 338 results under various experimental settings. 339
5.1 Enhanced performance and efficiency in 340 language model pre-training 341
To verify the effectiveness of our soft deduplication 342 method, we conduct experiments on the RedPajama 343 CommonCrawl dataset, which has not subjected 344 to meticulous deduplication. Our findings indicate 345 a significant improvement with our method com- 346 pared to the direct training baseline, as illustrated 347 in Figure 3. 348
Our approach consistently outperforms the base- 349 line in terms of average perplexity across all evalu- 350 ated datasets. Specifically, on the Pile test set, our 351 method enables models to achieve baseline perplex- 352 ity within 50,000 iterations, saving nearly 30,000 353 training steps. Furthermore, models continue to 354 improve, ultimately reaching a lower perplexity, as 355 shown in Figure 3a. Similar advancements are ob- 356 served in the SlimPajama test set, confirming our 357 method’s effectiveness (Figure 3b). Additionally, 358 we report the average perplexity for each subset 359 upon completion of training (Appendices A.1 and 360 A.2). Our method enables the models to yield per- 361 formance improvements across the majority of the 362 test subsets. 363
In our evaluation of downstream tasks, our 364 method outperforms the baseline in accuracy. It 365 accelerates learning on the RedPajama dataset, 366 achieving baseline performance nearly 30,000 steps 367 sooner and improving average accuracy by 1.77% 368 at the end of training, as shown in Figure 3c. De- 369 tailed scores for each individual task at the fi- 370 nal training checkpoint are delineated in Table 1. 371 Our approach yields improvements in all evaluated 372 tasks. 373
In summary, our experiments on the RedPajama 374 CommonCrawl dataset substantiate that the soft 375 deduplication method is capable of reducing per- 376 plexity and enhancing the accuracy of downstream 377 tasks more efficiently compared to the baseline 378 model. Such accelerated convergence is crucial for 379 pre-training large language models, considering the 380
5
significant costs associated with training duration381 and resource utilization.382
5.2 Surpassing hard deduplication in383 effectiveness384
In the experiments carried out using the RedPajama385 CommonCrawl dataset, we also contrast the SoftD-386 edup method against traditional hard deduplication387 techniques (refer to Table 1). Considering that the388 SlimPajama dataset originates from the RedPajama389 dataset, refined through MinHashLSH deduplica-390 tion, we employ models trained on the SlimPajama391 CommonCrawl dataset as the hard deduplication392 baseline.393
The evaluation results of models on various394 downstream tasks reveal our method’s superior per-395 formance over both the hard deduplication tech-396 nique and the direct training baseline. In detail,397 while the hard deduplication method surpasses the398 direct training baseline in nine out of twelve tasks,399
showing an average increase in accuracy of 0.6%, 400 our SoftDedup method demonstrates more consis- 401 tent and significant improvements. It outperforms 402 the direct training baseline across all evaluated 403 tasks, achieving an average accuracy enhancement 404 of 1.77%. These findings underscore the advan- 405 tages over conventional deduplication methods in 406 enhancing downstream task performance. 407
5.3 A powerful complement to existing 408 techniques 409
To further assess the effectiveness of our method 410 when applied in sequence with extant hard dedu- 411 plication processes, we conduct experiments on 412 the SlimPajama CommonCrawl and Falcon Re- 413 finedWeb datasets, which have undergone stringent 414 deduplication processes (as shown in Figures 4 and 415 5). 416
In the evaluations conducted on the Pile and 417 SlimPajama test sets, our method exhibits consis- 418
6
tent superiority over the baseline models. Notably,419 our models achieve equivalent baselines in perplex-420 ity with a reduction of 26% to 39% in the number of421 required training steps. Additionally, the ultimate422 performance of the models demonstrates a tangible423 enhancement, as evidenced by the results displayed424 in Figures 4a, 4b, 5a, and 5b. In terms of accuracy425 on downstream tasks, Figures 4c and 5c highlight426 the training efficiency achieved by our models. It427 is particularly noteworthy that our method reaches428 baseline performance with around 20,000 fewer429 training steps. We report the detailed scores in430 Appendices A.1, A.2, and A.3.431
In summary, even when applied to already dedu-432 plicated datasets, our method significantly en-433 hances training efficiency and effectiveness. This434 underscores its capability to address the shortcom-435 ings of current deduplication techniques. Specif-436 ically, our approach reweights the data to reflect437 varying levels of duplication, thus avoiding one-438 size-fits-all solutions. This integration has the po-439 tential to become a standard practice in the pre-440 training of large language models.441
5.4 Finer data partitioning for improved 442 downstream task performance 443
In Figure 6, we illustrate the impact of different 444 numbers of data partitions on model performance. 445 We argue that investigating methods to further en- 446 hance the training effectiveness of higher-quality 447 data is a more critical concern. Therefore, our ex- 448 periments are conducted on the Falcon RefinedWeb 449 dataset. 450
In evaluations conducted on both the Pile and 451 SlimPajama test sets, models exhibit negligible 452 variations in average perplexity across a range of 453 data partition counts, specifically 10, 20, 50, and 454 100. This observation indicates that perplexity, as 455 a metric, demonstrates relatively low sensitivity to 456 changes in the granularity of data partitioning. 457
In contrast, we observe that as the granularity 458 of data partitioning increases, the accuracy of the 459 language model in downstream tasks also improves. 460 As demonstrated in Figure 6c, there is a clear corre- 461 lation between the number of data partitions and the 462 model’s accuracy. This indicates that finer-grained 463 data partitioning can make the training data more 464 balanced, thereby enhancing performance in down- 465
7
stream tasks.466
5.5 Effects of sampling weight disparities on467 model performance468
Figure 7 presents the outcomes of experimental in-469 vestigations into the effects of varying disparities470 between maximum and minimum weights assigned471 to different data partitions. The methodology em-472 ployed ensures a consistent ascending order in the473 allocation of weights, with greater disparities indi-474 cating a more pronounced suppression of data with475 high commonness.476
Experiments conducted utilizing disparities in477 the maximum to minimum weight ratios of 2-fold,478 5-fold, and 10-fold reveal a consistent trend: in-479 creased disparities between the maximum and min-480 imum weights lead to a reduction in average model481 perplexity. Although slight variations are observed482 in the performance outcomes for downstream tasks,483 the experiments demonstrate that the largest weight484 disparity consistently facilitates the most optimal485 model performance.486
5.6 Cost of data reweighting 487
The computational processes of n-gram training 488 and commonness calculation are executed solely 489 on CPU resources. For a 40B token corpus, the 490 n-gram training procedure (with n = 4) requires 491 4 CPU cores for 5 hours, followed by computing 492 data commonness using 4 CPU cores in 2 hours. 493 Compared to the substantial costs of GPU con- 494 servation (at least 930 V100 GPU hours in our 495 experiments), these expenses can be considered 496 negligible. This underscores the efficiency of Soft- 497 Dedup and the feasibility of its implementation in 498 resource-constrained environments. 499
6 Conclusion 500
In this study, we introduce a soft deduplication 501 method that effectively addresses the primary lim- 502 itations associated with traditional hard dedupli- 503 cation methods. Unlike its predecessors, this ap- 504 proach retains all samples of data while reallocat- 505 ing sampling weights according to data common- 506 ness. Experimental analyses demonstrate that this 507 technique can significantly expedite the training 508
8
process for large language models, evidenced by509 a reduction of over 26% in the number of training510 steps required. The proposed method surpasses511 existing deduplication techniques in effectiveness512 and can serve as a valuable complement to these513 methods. Due to its low operational cost and su-514 perior efficiency, we advocate for the integration515 of this soft deduplication approach with traditional516 hard deduplication methods as a standard practice517 in the pre-training phase of large language models.518
Limitations519
Due to current limitations in computational re-520 sources, the extension of SoftDedup to larger-scale521 models will be deferred to future research endeav-522 ors. Moreover, future studies will seek to conduct523 a more comprehensive evaluation of the method’s524 effectiveness across various mixed data sources.525
References526 Amro Abbas, Kushal Tirumala, Dániel Simig, Surya527 Ganguli, and Ari S. Morcos. 2023. Semdedup: Data-528 efficient learning at web-scale through semantic dedu-529 plication.530
Miltiadis Allamanis. 2019. The adverse effects of code531 duplication in machine learning models of code.532
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al-533 shamsi, Alessandro Cappelli, Ruxandra Cojocaru,534 Mérouane Debbah, Étienne Goffinet, Daniel Hesslow,535 Julien Launay, Quentin Malartic, Daniele Mazzotta,536 Badreddine Noune, Baptiste Pannier, and Guilherme537 Penedo. 2023. The falcon series of open language538 models.539
Jack Bandy and Nicholas Vincent. 2021. Addressing540 "documentation debt" in machine learning research:541 A retrospective datasheet for bookcorpus.542
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy543 Liang. 2013. Semantic parsing on Freebase from544 question-answer pairs. In Proceedings of the 2013545 Conference on Empirical Methods in Natural Lan-546 guage Processing, pages 1533–1544, Seattle, Wash-547 ington, USA. Association for Computational Linguis-548 tics.549
Stella Biderman, Hailey Schoelkopf, Quentin Anthony,550 Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mo-551 hammad Aflah Khan, Shivanshu Purohit, USVSN Sai552 Prashanth, Edward Raff, Aviya Skowron, Lintang553 Sutawika, and Oskar van der Wal. 2023. Pythia:554 A suite for analyzing large language models across555 training and scaling.556
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng557 Gao, and Yejin Choi. 2020. Piqa: Reasoning about558
physical commonsense in natural language. In Thirty- 559 Fourth AAAI Conference on Artificial Intelligence. 560
A.Z. Broder. 1997. On the resemblance and con- 561 tainment of documents. In Proceedings. Compres- 562 sion and Complexity of SEQUENCES 1997 (Cat. 563 No.97TB100171), pages 21–29. 564
Moses S Charikar. 2002. Similarity estimation tech- 565 niques from rounding algorithms. In Proceedings of 566 the thiry-fourth annual ACM symposium on Theory 567 of computing, pages 380–388. 568
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, 569 Ashish Sabharwal, Carissa Schoenick, and Oyvind 570 Tafjord. 2018. Think you have solved question an- 571 swering? try arc, the ai2 reasoning challenge. ArXiv, 572 abs/1803.05457. 573
Together Computer. 2023. Redpajama: an open dataset 574 for training large language models. 575
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- 576 ing, Travis Hoppe, Charles Foster, Jason Phang, 577 Horace He, Anish Thite, Noa Nabeshima, Shawn 578 Presser, and Connor Leahy. 2020. The pile: An 579 800gb dataset of diverse text for language modeling. 580
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, 581 Sid Black, Anthony DiPofi, Charles Foster, Laurence 582 Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, 583 Kyle McDonell, Niklas Muennighoff, Chris Ociepa, 584 Jason Phang, Laria Reynolds, Hailey Schoelkopf, 585 Aviya Skowron, Lintang Sutawika, Eric Tang, An- 586 ish Thite, Ben Wang, Kevin Wang, and Andy Zou. 587 2023. A framework for few-shot language model 588 evaluation. 589
Danny Hernandez, Tom Brown, Tom Conerly, Nova 590 DasSarma, Dawn Drain, Sheer El-Showk, Nelson 591 Elhage, Zac Hatfield-Dodds, Tom Henighan, Tris- 592 tan Hume, Scott Johnston, Ben Mann, Chris Olah, 593 Catherine Olsson, Dario Amodei, Nicholas Joseph, 594 Jared Kaplan, and Sam McCandlish. 2022. Scaling 595 laws and interpretability of learning from repeated 596 data. 597
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke 598 Zettlemoyer. 2017. Triviaqa: A large scale distantly 599 supervised challenge dataset for reading comprehen- 600 sion. In Proceedings of the 55th Annual Meeting of 601 the Association for Computational Linguistics, Van- 602 couver, Canada. Association for Computational Lin- 603 guistics. 604
Katherine Lee, Daphne Ippolito, Andrew Nystrom, 605 Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, 606 and Nicholas Carlini. 2022. Deduplicating training 607 data makes language models better. 608
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 609 2019. Latent retrieval for weakly supervised open 610 domain question answering. In Proceedings of the 611 57th Annual Meeting of the Association for Computa- 612 tional Linguistics, pages 6086–6096, Florence, Italy. 613 Association for Computational Linguistics. 614
9
Jure Leskovec, Anand Rajaraman, and Jeffrey David615 Ullman. 2020. Mining of massive data sets. Cam-616 bridge university press.617
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He,618 and Piotr Dollár. 2018. Focal loss for dense object619 detection.620
Cristina V. Lopes, Petr Maj, Pedro Martins, Vaibhav621 Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan622 Vitek. 2017. Déjàvu: A map of code duplicates on623 github. Proc. ACM Program. Lang., 1(OOPSLA).624
Udi Manber and Gene Myers. 1993. Suffix arrays: A625 new method for on-line string searches. SIAM Jour-626 nal on Computing, 22(5):935–948.627
Niklas Muennighoff, Alexander M. Rush, Boaz Barak,628 Teven Le Scao, Aleksandra Piktus, Nouamane Tazi,629 Sampo Pyysalo, Thomas Wolf, and Colin Raffel.630 2023. Scaling data-constrained language models.631
Hermann Ney, Ute Essen, and Reinhard Kneser. 1994.632 On structuring probabilistic dependences in stochas-633 tic language modelling. Computer Speech Language,634 8(1):1–38.635
Denis Paperno, Germán Kruszewski, Angeliki Lazari-636 dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro637 Pezzelle, Marco Baroni, Gemma Boleda, and Raquel638 Fernández. 2016. The lambada dataset: Word predic-639 tion requiring a broad discourse context.640
Guilherme Penedo, Quentin Malartic, Daniel Hesslow,641 Ruxandra Cojocaru, Alessandro Cappelli, Hamza642 Alobeidli, Baptiste Pannier, Ebtesam Almazrouei,643 and Julien Launay. 2023. The refinedweb dataset for644 falcon llm: Outperforming curated corpora with web645 data, and web data only.646
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine647 Lee, Sharan Narang, Michael Matena, Yanqi Zhou,648 Wei Li, and Peter J. Liu. 2023. Exploring the limits649 of transfer learning with a unified text-to-text trans-650 former.651
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.652 Know what you don’t know: Unanswerable questions653 for squad.654
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel655 Urtasun. 2019. Learning to reweight examples for656 robust deep learning.657
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhaga-658 vatula, and Yejin Choi. 2019. Winogrande: An ad-659 versarial winograd schema challenge at scale. arXiv660 preprint arXiv:1907.10641.661
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan662 Le Bras, and Yejin Choi. 2019. Social iqa: Com-663 monsense reasoning about social interactions. In664 Proceedings of the 2019 Conference on Empirical665 Methods in Natural Language Processing and the 9th666 International Joint Conference on Natural Language667 Processing (EMNLP-IJCNLP), pages 4463–4473.668
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Ja- 669 cob R Steeves, Joel Hestness, and Nolan Dey. 2023. 670 SlimPajama: A 627B token cleaned and deduplicated 671 version of RedPajama. 672
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya 673 Ganguli, and Ari S. Morcos. 2023. Beyond neural 674 scaling laws: beating power law scaling via data 675 pruning. 676
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, 677 and Ari S. Morcos. 2023. D4: Improving llm pre- 678 training via document de-duplication and diversifica- 679 tion. 680
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier 681 Martinet, Marie-Anne Lachaux, Timothée Lacroix, 682 Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal 683 Azhar, Aurelien Rodriguez, Armand Joulin, Edouard 684 Grave, and Guillaume Lample. 2023. Llama: Open 685 and efficient foundation language models. 686
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- 687 neau, Vishrav Chaudhary, Francisco Guzmán, Ar- 688 mand Joulin, and Edouard Grave. 2019. Ccnet: Ex- 689 tracting high quality monolingual datasets from web 690 crawl data. 691
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, 692 Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, 693 Tengyu Ma, and Adams Wei Yu. 2023a. Doremi: 694 Optimizing data mixtures speeds up language model 695 pretraining. 696
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and 697 Percy Liang. 2023b. Data selection for language 698 models via importance resampling. 699
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei 700 Zheng, and Yang You. 2023. To repeat or not to 701 repeat: Insights from scaling llm under token-crisis. 702
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali 703 Farhadi, and Yejin Choi. 2019. Hellaswag: Can a 704 machine really finish your sentence? In Proceedings 705 of the 57th Annual Meeting of the Association for 706 Computational Linguistics. 707
A Appendix 708
A.1 Average perplexity for each subset in the 709 Pile test set 710
In Table 2, we provide a detailed report on the 711 average perplexity for each subset within the Pile 712 test set. For models trained on the RedPajama 713 CommonCrawl dataset, our method results in im- 714 provements across 18 out of 22 subsets. For models 715 trained on the SlimPajama CommonCrawl dataset, 716 our method leads to improvements in 17 subsets. 717 For models trained on the Falcon RefinedWeb, im- 718 provements are observed in 19 subsets. Due to 719 the exceedingly small number of documents in the 720
10
Ubuntu IRC subset, we exclude it from the cal-721 culation of the average perplexity on the Pile test722 set.723
A.2 Average perplexity for each subset in the724 SlimPajama test set725
In Table 3, we provide a detailed report on the av-726 erage perplexity for each subset within the SlimPa-727 jama test set. Our method has led to improvements728 across nearly all subsets.729
A.3 Accuracy for each downstream task730 In Table 4, we provide a detailed report on the731 accuracy for each downstream task. For models732 trained on the RedPajama CommonCrawl dataset,733 our method has led to improvements across all734 tasks. For models trained on the SlimPajama Com-735 monCrawl and Falcon RefinedWeb datasets, our736 approach has also resulted in accuracy improve-737 ments on the majority of tasks.738
11
12

heading: Future Work
text: 518
Limitations519
Due to current limitations in computational re-520 sources, the extension of SoftDedup to larger-scale521 models will be deferred to future research endeav-522 ors. Moreover, future studies will seek to conduct523 a more comprehensive evaluation of the method’s524 effectiveness across various mixed data sources. 525
References526 Amro Abbas, Kushal Tirumala, Dániel Simig, Surya527 Ganguli, and Ari S. Morcos. 2023. Semdedup: Data-528 efficient learning at web-scale through semantic dedu-529 plication. 530
Miltiadis Allamanis. 2019. The adverse effects of code531 duplication in machine learning models of code. 532
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al-533 shamsi, Alessandro Cappelli, Ruxandra Cojocaru,534 Mérouane Debbah, Étienne Goffinet, Daniel Hesslow,535 Julien Launay, Quentin Malartic, Daniele Mazzotta,536 Badreddine Noune, Baptiste Pannier, and Guilherme537 Penedo. 2023. The falcon series of open language538 models. 539
Jack Bandy and Nicholas Vincent. 2021. Addressing540 "documentation debt" in machine learning research:541 A retrospective datasheet for bookcorpus. 542
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy543 Liang. 2013. Semantic parsing on Freebase from544 question-answer pairs. In Proceedings of the 2013545 Conference on Empirical Methods in Natural Lan-546 guage Processing, pages 1533–1544, Seattle, Wash-547 ington, USA. Association for Computational Linguis-548 tics. 549
Stella Biderman, Hailey Schoelkopf, Quentin Anthony,550 Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mo-551 hammad Aflah Khan, Shivanshu Purohit, USVSN Sai552 Prashanth, Edward Raff, Aviya Skowron, Lintang553 Sutawika, and Oskar van der Wal. 2023. Pythia:554 A suite for analyzing large language models across555 training and scaling. 556
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng557 Gao, and Yejin Choi. 2020. Piqa: Reasoning about558
physical commonsense in natural language. In Thirty- 559 Fourth AAAI Conference on Artificial Intelligence. 560
A. Z. Broder. 1997. On the resemblance and con- 561 tainment of documents. In Proceedings. Compres- 562 sion and Complexity of SEQUENCES 1997 (Cat. 563 No. 97TB100171), pages 21–29. 564
Moses S Charikar. 2002. Similarity estimation tech- 565 niques from rounding algorithms. In Proceedings of 566 the thiry-fourth annual ACM symposium on Theory 567 of computing, pages 380–388. 568
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, 569 Ashish Sabharwal, Carissa Schoenick, and Oyvind 570 Tafjord. 2018. Think you have solved question an- 571 swering? try arc, the ai2 reasoning challenge. ArXiv, 572 abs/1803. 05457. 573
Together Computer. 2023. Redpajama: an open dataset 574 for training large language models. 575
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- 576 ing, Travis Hoppe, Charles Foster, Jason Phang, 577 Horace He, Anish Thite, Noa Nabeshima, Shawn 578 Presser, and Connor Leahy. 2020. The pile: An 579 800gb dataset of diverse text for language modeling. 580
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, 581 Sid Black, Anthony DiPofi, Charles Foster, Laurence 582 Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, 583 Kyle McDonell, Niklas Muennighoff, Chris Ociepa, 584 Jason Phang, Laria Reynolds, Hailey Schoelkopf, 585 Aviya Skowron, Lintang Sutawika, Eric Tang, An- 586 ish Thite, Ben Wang, Kevin Wang, and Andy Zou. 587 2023. A framework for few-shot language model 588 evaluation. 589
Danny Hernandez, Tom Brown, Tom Conerly, Nova 590 DasSarma, Dawn Drain, Sheer El-Showk, Nelson 591 Elhage, Zac Hatfield-Dodds, Tom Henighan, Tris- 592 tan Hume, Scott Johnston, Ben Mann, Chris Olah, 593 Catherine Olsson, Dario Amodei, Nicholas Joseph, 594 Jared Kaplan, and Sam McCandlish. 2022. Scaling 595 laws and interpretability of learning from repeated 596 data. 597
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke 598 Zettlemoyer. 2017. Triviaqa: A large scale distantly 599 supervised challenge dataset for reading comprehen- 600 sion. In Proceedings of the 55th Annual Meeting of 601 the Association for Computational Linguistics, Van- 602 couver, Canada. Association for Computational Lin- 603 guistics. 604
Katherine Lee, Daphne Ippolito, Andrew Nystrom, 605 Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, 606 and Nicholas Carlini. 2022. Deduplicating training 607 data makes language models better. 608
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 609 2019. Latent retrieval for weakly supervised open 610 domain question answering. In Proceedings of the 611 57th Annual Meeting of the Association for Computa- 612 tional Linguistics, pages 6086–6096, Florence, Italy. 613 Association for Computational Linguistics. 614
9
Jure Leskovec, Anand Rajaraman, and Jeffrey David615 Ullman. 2020. Mining of massive data sets. Cam-616 bridge university press. 617
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He,618 and Piotr Dollár. 2018. Focal loss for dense object619 detection. 620
Cristina V. Lopes, Petr Maj, Pedro Martins, Vaibhav621 Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan622 Vitek. 2017. Déjàvu: A map of code duplicates on623 github. Proc. ACM Program. Lang. , 1(OOPSLA). 624
Udi Manber and Gene Myers. 1993. Suffix arrays: A625 new method for on-line string searches. SIAM Jour-626 nal on Computing, 22(5):935–948. 627
Niklas Muennighoff, Alexander M. Rush, Boaz Barak,628 Teven Le Scao, Aleksandra Piktus, Nouamane Tazi,629 Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 630 2023. Scaling data-constrained language models. 631
Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. 632 On structuring probabilistic dependences in stochas-633 tic language modelling. Computer Speech Language,634 8(1):1–38. 635
Denis Paperno, Germán Kruszewski, Angeliki Lazari-636 dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro637 Pezzelle, Marco Baroni, Gemma Boleda, and Raquel638 Fernández. 2016. The lambada dataset: Word predic-639 tion requiring a broad discourse context. 640
Guilherme Penedo, Quentin Malartic, Daniel Hesslow,641 Ruxandra Cojocaru, Alessandro Cappelli, Hamza642 Alobeidli, Baptiste Pannier, Ebtesam Almazrouei,643 and Julien Launay. 2023. The refinedweb dataset for644 falcon llm: Outperforming curated corpora with web645 data, and web data only. 646
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine647 Lee, Sharan Narang, Michael Matena, Yanqi Zhou,648 Wei Li, and Peter J. Liu. 2023. Exploring the limits649 of transfer learning with a unified text-to-text trans-650 former. 651
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. 652 Know what you don’t know: Unanswerable questions653 for squad. 654
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel655 Urtasun. 2019. Learning to reweight examples for656 robust deep learning. 657
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhaga-658 vatula, and Yejin Choi. 2019. Winogrande: An ad-659 versarial winograd schema challenge at scale. arXiv660 preprint arXiv:1907. 10641. 661
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan662 Le Bras, and Yejin Choi. 2019. Social iqa: Com-663 monsense reasoning about social interactions. In664 Proceedings of the 2019 Conference on Empirical665 Methods in Natural Language Processing and the 9th666 International Joint Conference on Natural Language667 Processing (EMNLP-IJCNLP), pages 4463–4473. 668
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Ja- 669 cob R Steeves, Joel Hestness, and Nolan Dey. 2023. 670 SlimPajama: A 627B token cleaned and deduplicated 671 version of RedPajama. 672
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya 673 Ganguli, and Ari S. Morcos. 2023. Beyond neural 674 scaling laws: beating power law scaling via data 675 pruning. 676
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, 677 and Ari S. Morcos. 2023. D4: Improving llm pre- 678 training via document de-duplication and diversifica- 679 tion. 680
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier 681 Martinet, Marie-Anne Lachaux, Timothée Lacroix, 682 Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal 683 Azhar, Aurelien Rodriguez, Armand Joulin, Edouard 684 Grave, and Guillaume Lample. 2023. Llama: Open 685 and efficient foundation language models. 686
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- 687 neau, Vishrav Chaudhary, Francisco Guzmán, Ar- 688 mand Joulin, and Edouard Grave. 2019. Ccnet: Ex- 689 tracting high quality monolingual datasets from web 690 crawl data. 691
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, 692 Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, 693 Tengyu Ma, and Adams Wei Yu. 2023a. Doremi: 694 Optimizing data mixtures speeds up language model 695 pretraining. 696
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and 697 Percy Liang. 2023b. Data selection for language 698 models via importance resampling. 699
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei 700 Zheng, and Yang You. 2023. To repeat or not to 701 repeat: Insights from scaling llm under token-crisis. 702
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali 703 Farhadi, and Yejin Choi. 2019. Hellaswag: Can a 704 machine really finish your sentence? In Proceedings 705 of the 57th Annual Meeting of the Association for 706 Computational Linguistics. 707
A Appendix 708
A. 1 Average perplexity for each subset in the 709 Pile test set 710
In Table 2, we provide a detailed report on the 711 average perplexity for each subset within the Pile 712 test set. For models trained on the RedPajama 713 CommonCrawl dataset, our method results in im- 714 provements across 18 out of 22 subsets. For models 715 trained on the SlimPajama CommonCrawl dataset, 716 our method leads to improvements in 17 subsets. 717 For models trained on the Falcon RefinedWeb, im- 718 provements are observed in 19 subsets. Due to 719 the exceedingly small number of documents in the 720
10
Ubuntu IRC subset, we exclude it from the cal-721 culation of the average perplexity on the Pile test722 set. 723
A. 2 Average perplexity for each subset in the724 SlimPajama test set725
In Table 3, we provide a detailed report on the av-726 erage perplexity for each subset within the SlimPa-727 jama test set. Our method has led to improvements728 across nearly all subsets. 729
A. 3 Accuracy for each downstream task730 In Table 4, we provide a detailed report on the731 accuracy for each downstream task. For models732 trained on the RedPajama CommonCrawl dataset,733 our method has led to improvements across all734 tasks. For models trained on the SlimPajama Com-735 monCrawl and Falcon RefinedWeb datasets, our736 approach has also resulted in accuracy improve-737 ments on the majority of tasks. 738
11
12