SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Curation is not a silver bullet for zero-shot generalization: small-scale models trained on  $\bullet$ REFINEDWEB outperform models trained on web data (C4, OSCAR), and on curated corpora (▼ The Pile). Average accuracy in zero-shot on the small-agg aggregate. All models trained with identical architectures and pretraining hyperparameters, for the same amount of tokens. We find that OSCAR-22.01 underperforms other datasets significantly, perhaps because dedduplication is only optional. C4 is a strong baseline, with OSCAR-21.09 lagging slightly behind, but we find that RefinedWeb outperforms both web datasets and the most popular curated dataset, The Pile. Both filtering and dedduplication contribute significantly to improving zero-shot performance.
MASSIVE WEB DATSETS	CURATED	OURS
OSCAR-21.09	OSCAR-22.01	C4	▼ The Pile	RW-Raw	RW-Filtered	●REFINEDWEB
1B@27GT	55.0%	52.7%	55.7%	53.4%	52.7%	54.3%	56.2%
3B@60GT	59.1%	55.9%	59.6%	57.9%	57.4%	58.2%	59.8%
We now validate that models trained on RefinedWeb can match the zero-shot performance obtained with curated corpora and by state-of-the-art models. We first discuss our evaluation and pretraining setup, and models with which we compare. We perform experiments at small scale to internally compare with other datasets, and ablate the stages of RefinedWeb (raw, filtered, final). Then, we scale to 1B and 7B models trained on 350GT to compare with the state-of-the-art. Finally, we apply the MDR pipeline to existing datasets, and show that it can potentially deliver further improvements.
Evaluation. At variance with previous works studying pretraining datasets [12, 30], we focus our evaluation on zero-shot generalization across many tasks rather than measuring validation loss. Perplexity alone can be at odds with end-task performance [68], and modern works on LLMs predominantly report zero-shot performance [2, 12, 22]. Furthermore, zero-shot generalization is the "natural" setting for autoregressive decoder-only models, in which they perform best [69]. Our evaluation setup is inspired by the one used by the architecture and scaling group of Big Science [11]. We base our evaluation on the Eleuther AI evaluation harness [49], allowing us to evaluate across a wide range of tasks. We identified aggregates allowing us to: (1) obtain signal (i.e., non zero zero-shot performance) at small scale for ablations; (2) compare with results reported by other models. We outline these aggregates small (for ablations), and core, main, ext (for comparisons) in Table 3.
Comparisons across models trained and evaluated in different settings are difficult to untangle, as many externalities may influence the results (e.g., numerical precision of training vs inference, prompts used). We distinguish three levels of comparisons: (1) internal comparisons, with models trained and evaluated within our codebase, for which only the pretraining datasets differ; (2) benchmark-level comparisons, with models trained with a different codebase but evaluated with the Eleuther AI harness, taking results from [11, 70, 71, 48], thereafter flagged with a  $*$ ; (3) external comparisons with [2, 22], thereafter flagged with a  $\dagger$ . For further details on evaluation, see Appendix H.1
Models. We train 1B, 3B, and 7B parameters autoregressive decoder-only models, based on configurations and hyperparameters similar to GPT-3 [2], diverging mostly on our use of ALiBi [72]. We use FlashAttention [73] in a custom codebase. We train internal models on both The Pile and RefinedWeb to control for deviations caused by our pretraining setup—we found The Pile models to perform in-line with others. For small-scale and ablation studies (first half of Section 4.2; Section 4.3), we train models to optimality according to the scaling laws of Hoffmann et al. [4]: on 27B and 60B tokens respectively for our 1B and 3B parameters models. For the main experiments demonstrating our approach (Falcon-RW models in Section 4.2), we train the models to 350GT, in line with popular public models [2, 74, 23]. Note that we do not compare against the recently introduced LLaMA models [7], as the smallest of them is trained on x2.5 more compute than our largest model, preventing a meaningful comparison from being made dataset-wise. For a more in-depth overview of the models and pretraining datasets with which we compare, see Appendix H.
196   
197   
198   
199   
200   
201   
202   
203   
204   
205   
206   
207   
208   
209   
210   
211   
212   
213   
214   
215   
216   
217   
218   
219   
220   
221   
222   
223   
224   
225   
226   
227   
228   
229   
230   
231   
232
4.2 Can web data alone outperform curated corpora?
We endeavour to demonstrate that web data alone can result in models outperforming models trained on curated corpora. To do so, we first perform a small-scale study with 1B and 3B parameters models trained to optimality (27GT and 60GT) on popular web and curated datasets. Then, we scale up to 1B and 7B models trained on 350GT, and compare zero-shot generalization to state-of-the-art models.
Small-scale study. We first consider public web datasets (OSCAR-2019 [8], OSCAR-2022 [75], C4 [9]), The Pile [10] as the most popular publicly available curated dataset, and variations of RefinedWeb (RW-Raw, RW-Filtered, and RW as described in Section 3). All models are trained with the same architecture, for the same amount of tokens, and using the same internal codebase; they are also all evaluated within the same framework-only pretraining datasets differ.
Results averaged on the small aggregate of 6 tasks are presented in Table 4. We observe relatively strong performance of all web datasets compared to The Pile, showcasing that curation is not a silver bullet for performant language models. We find C4 to be a strong pretraining dataset, in line with the findings of Scao et al. [11]–however, The Pile underperforms more in our benchmarks. The disappointing results on OSCAR-22.01 may be due to the dataset being distributed without dedduplication by default. Regarding RefinedWeb, both filtering and deduplication significantly improve performance. We also note that a 3B@60GT model trained on OSCAR-22.1 performs worse than a 1B@27GT model trained on RefinedWeb: data alone accounts for a 4x difference in pretraining compute, highlighting that compute budgets alone cannot compensate efficiently for inadequate data.
Full-scale models. We now validate these results with comparisons with state-of-the-art models. We scale our previous experiments by training 1B and 7B models on 350GT; we also train a 1B model on 350GT on The Pile, as a control for the influence of our pretraining setup. We compare with the following models: the GPT-3 series [2], the FairSeq series [76], the GPT-Neo(X)/J models [77,74,70], the OPT series [43], the BigScience Architecture and Scaling Pile model [11], PaLM-8B [22], Aleph Alpha Luminous 13B [71], the Pythia series [42], and the Cerebras-GPT series [48]. For GPT-3, we distinguish between results obtained through the API (babbage and curie) with the EleutherAI LM evaluation harness [49] (*), and results reported in their paper, with a different evaluation setup ( $\dagger$ ). For PaLM and OPT, results were obtained also with a different evaluation suite ( $\dagger$ ); for most other models they were obtained with the evaluation harness ( $*$ ), allowing for more direct comparisons.
Results on main-agg are presented in Figure 1, and in Figure 3 for core-agg and ext-agg. We find that open models consistently underperform models trained on private curated corpora, such
(b)
Figure 3: Models trained on  $\bullet$ REFINEDWEB alone outperform models trained on curated corpora. Zero-shot performance averaged on our core-agg (left) and ext-agg (right) task aggregates (see Section 4.1 for details, and Figure 1 for results on main-agg). Existing open models fail to match the performance of the original GPT-3 series (left); however, models trained on RefinedWeb significantly outperform models trained on  $\nabla$  The Pile: including our direct comparison model (right), ruling out our pretraining setup as the main source of increased performance. In fact, our RefinedWeb models even match the performance of the  $\blacksquare$ GPT-3 models.
234   
235   
236   
237   
238   
239   
240   
241   
242   
243   
244   
245   
246   
247   
248   
249   
250   
251   
252   
253   
254   
255   
256   
257   
258   
259   
260   
261   
262   
263
Table 5: Although improvements from filtering are not systematic across datasets, dedduplication brings a steady performance boost across the board. Zero-shot accuracy averaged on small1-agg aggregate;  $[+\mathrm{x.x}]$  reports absolute gains compared to base, removal rates reported against base. Due to limitations in our pipeline, we cannot apply the dedduplication stage independently for RefinedWeb.
MASSIVE WEB DATSETS	CURATED	OURS
OSCAR-21.09	OSCAR-22.01	C4	▼Pile	RefinedWeb
Base	55.0%	52.7%	55.7%	53.4%	52.7%
Filtered	55.4% [+.4]	52.3% [-.4]	56.2% [+.5]	54.2% [+.8]	54.3% [+1.6]
removal rate	-25.0%	-39.8%	-16.4%	-27.1%	-50.8%
Deduplicated	55.6% [+.6]	55.6% [+2.9]	55.9% [+.2]	54.5% [+1.1]
removal rate	-10.8%	-60.8%	-7.59%	-45.3%
Filt.+Dedup.	55.5% [+.5]	55.4% [+2.7]	56.4% [+.7]	55.2% [+1.8]	56.2% [+3.5]
removal rate	-28.2%	-62.2%	-17.9%	-66.0%	-75.4%
as GPT-3—even when using a similar evaluation setup. Conversely, models trained on RefinedWeb are able to match the performance of the GPT-3 series using web data alone, even though common high-quality sources used in The Pile are excluded from RefinedWeb (see Table 14 in Appendix). Finally, we note that our internal model trained on The Pile performs in line with the BigScience Architecture and Scaling model; this highlights that our pretraining setup is unlikely to be the main source of increased performance for models trained on RefinedWeb.
Finding. Challenging beliefs on data quality, filtered and deduplicated web data alone allows models to match the natural language tasks performance of models trained on curated data.
4.3 Do other corpora benefit from MDR?
Ablating the contributions and evaluating the performance of individual components in the MDR pipeline is difficult: for most heuristics, there is no agreed-upon ground truth, and changes may be too insignificant to result in sufficient zero-shot signal after pretraining. In the first half of Section 4.2 we identified that subsequent stages of RefinedWeb (raw, filtered, final) led to improvements in performance. In this section, we propose to apply independently the filtering and dedduplication stages of MDR to popular pretraining datasets, studying whether they generalize widely.
We report results on the small-agg in Table 5. First, we find that improvements from filtering are not systematic. On The Pile, we had to adjust our line length and characters ratio heuristics to avoid expunging books and code. Despite improvements on OSCAR-21.09, C4, and The Pile, our filters worsen performance on OSCAR-22.01; generally, removal rates from filtering are not strongly correlated with downstream accuracy. Conversely, dedduplication delivers a steady boost across all datasets, and removal rates are better correlated with zero-shot improvements. OSCAR-21.09 and C4 are already well deduplicated, while The Pile and OSCAR-22.01 exhibit  $40 - 60\%$  duplicates. OSCAR-22.01 is distributed without dedduplication by default; for The Pile, this is consistent with the findings of Zhang et al. [43]. Finally, combining filtering and dedduplication results in further improvements; although performance is now more uniform across datasets, differences remain, suggesting that flaws in the original text extraction and processing are not fully compensated for.
By processing C4 with MDR, we are able to obtain subsets of data which might slightly outperform RefinedWeb; this combines both the stringent filtering of C4 (e.g., strict NSFW word blacklist, 3-sentence span dedduplication) with our own filters and dedduplication. While this results in rejection rates that are unacceptable for our target of 3-6 trillions tokens, this is an interesting perspective for shorter runs, which may be able to extract extremely high-quality subsets from large datasets.
Finding. While filtering heuristics may require source-dependent tuning, stringent dedduplication improves zero-shot performance across datasets consistently.
264   
265   
266   
267   
268   
269   
270   
271   
272   
273   
274   
275   
276   
277   
278   
279   
280   
281   
282   
283   
284   
285   
286   
287   
288   
289   
290   
291   
292
Biases and harmfulness. We conduct an analysis of the toxicity of RefinedWeb in Figure 5 of the Appendix. We find RefinedWeb to be about as toxic as The Pile, based on the definition of toxicity of the Perspective API: "content that is rude or disrespectful". Notably, this definition does not cover social biases or harmfulness. Although it is unlikely that our pipeline introduces further issues than is already documented for popular datasets, we encourage quantitative work on our public extract.
Performance beyond natural language. Our evaluation aggregates are overwhelmingly targeting natural language tasks, and do not include code or mathematics evaluation—which are popular use cases for fully-fledged models. A natural question may be whether web data alone is sufficient to achieve strong code/mathematics performance; we do not think this is the case, and encourage tractioners to combine RefinedWeb with code datasets such as The Stack [78] when training modules. However many of our findings apply equally: notably, Li et al. [79] found that dedduplication helped with code data collected from GitHub as well. Broadly speaking, like web data is massively collected from CommonCrawl, code data is usually collected from GitHub, before undergoing extensive filtering and dedduplication. This is similar to the spirit of RefinedWeb, and does not rely on a collection of curated sources. Finally, we note that specific domains (e.g., code, technical papers) exist on a spectrum, and that general natural language improvements may benefit technical tasks too: for instance, we find that models trained on RefinedWeb outperform on PubMedQA models trained on The Pile, despite not including any explicit medical data (The Pile includes PubMed).
And beyond pretraining... Our study is strictly limited to language model pretraining, and does not address finetuning existing models. We note the value of high-quality samples for downstream specialization, for instance for improving chattiness or instruction-following capabilities [80].
Multiple epochs. Instead of looking for "unique" tokens for a trillion-scale pretraining dataset, one could simply repeat data over multiple epochs. Popular models like OPT and NeoX-20B train on up to 2 epochs [43, 70], and most curated datasets upsample corpora 2-5 times [2, 10]. However, Hernandez et al. [41] has recently shown that models with  $100\mathrm{B}+$  parameters may be sensitive to even just a few epochs. Orthogonal to our work one could explore tradeoffs in the data-constrained regime: can dedduplication help sustain more epochs? Are multiple epochs on higher quality data better than one epoch on lower quality data? See Appendix G.3 for a more in-depth discussion.
Other results on dedduplication. Biderman et al. [42] found a limited impact on zero-shot performance from deduplicating The Pile; we discuss in Appendix H.2 and suspect dedduplication may be unreasonably effective on web datasets because it predominantly removes low quality content (see Appendix J for top samples). Muennighoff et al. [81] studied scaling laws for multiple epochs, and found that up to four epochs carried limited degradation—however, we note that many of the duplicates we find are present hundred to thousands of time in the raw data, far from this safe regime.
As LLMs are widely adopted, models trained past the recommendations of scaling laws are bound to become increasingly common to amortize inference costs [7]. This will further drive the need for pretraining datasets with trillions of tokens, an order of magnitude beyond publicly available corpora. We have demonstrated that stringent filtering and dedduplication could result in a five trillion tokens web only dataset suitable to produce competitive models, even outperforming LLMs trained on curated corpora. We publicly release a 600GT extract of RefinedWeb, and note that RefinedWeb has already been used to train state-of-the-art language models, such as Falcon-40B [82].
We publicly release the following artefacts:
- A 600B tokens extract of RefinedWeb: https://huggingface.co/datasets/tiiuae/falcon-refinedweb;
- The 1B and 7B models trained on RefinedWeb in this paper: https://huggingface.co/tiiuae/falcon-rw-1b and https://huggingface.co/tiiuae/falcon-rw-7b.
293   
294   
295   
296   
297   
298   
299   
300   
301   
302   
303   
304   
305   
306   
307   
308   
309   
310   
311   
312   
313   
314   
315   
316   
317   
318   
319   
320   
321   
322   
323   
324   
325   
326   
327   
328   
329   
330   
331   
332   
333   
334   
335   
336   
337   
338   
339
341   
342   
343   
344   
345   
346   
347   
348   
349   
350   
351   
352   
353   
354   
355   
356   
357   
358   
359   
360   
361   
362   
363   
364   
365   
366   
367   
368   
369   
370   
371   
372   
373   
374   
375   
376   
377   
378   
379   
380   
381   
382   
383   
384   
385   
386
387   
388   
389   
390   
391   
392   
393   
394   
395   
396   
397   
398   
399   
400   
401   
402   
403   
404   
405   
406   
407   
408   
409   
410   
411   
412   
413   
414   
415   
416   
417   
418   
419   
420   
421   
422   
423   
424   
425   
426   
427   
428   
429   
430   
431
432   
433   
434   
435   
436   
437   
438   
439   
440   
441   
442   
443   
444   
445   
446   
447   
448   
449   
450   
451   
452   
453   
454   
455   
456   
457   
458   
459   
460   
461   
462   
463   
464   
465   
466   
467   
468   
469   
470   
471   
472   
473   
474   
475   
476   
477
487   
479   
480   
481   
482   
483   
484   
485   
486   
487   
488   
489   
490   
491   
492   
493   
494   
495   
496   
497   
498   
499   
500   
501   
502   
503   
504   
505   
506   
507   
508   
509   
510   
511   
512   
513   
514   
515   
516   
517   
518   
519   
520   
521   
522   
523
524   
525   
526   
527   
528   
529   
530   
531   
532   
533   
534   
535   
536   
537   
538   
539   
540   
541   
542   
543   
544   
545   
546   
547   
548   
549   
550   
551   
552   
553   
554   
555   
556   
557   
558   
559   
560   
561   
562   
563   
564   
565   
566   
567
568   
569   
570   
571   
572   
573   
574   
575   
576   
577   
578   
579   
580   
581   
582   
583   
584   
585   
586   
587   
588   
589   
590   
591   
592   
593   
594   
595   
596   
597   
598   
599   
600   
601   
602   
603   
604   
605   
606   
607   
608   
609   
610   
611   
612
613   
614   
615   
616   
617   
618   
619   
620   
621   
622   
623   
624   
625   
626   
627   
628   
629   
630   
631   
632   
633   
634   
635   
636   
637   
638   
639   
640   
641   
642   
643   
644   
645   
646   
647   
648   
649   
650   
651   
652   
653   
654   
655   
656   
657   
658
[92] Dick Sites. Compact language detector 2. Software available at https://github.com/CLD2Owners/cld2 (last updated on August 2015), 2013.
[93] Laura Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020.
[94] P. Jaccard. The distribution of the flora in the alpine zone.1. New Phytologist, 11:37-50, 1912.
659   
660   
661   
662