SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Falcon LLM Team
Guilherme Penedo2 Quentin Malartic1 Daniel Hesslow2 Ruxandra Cojocaru1
Hamza Alobeidli $^{1}$  Alessandro Cappelli $^{2}$  Baptiste Pannier $^{2}$
Ebtesam Almazrouei<sup>1</sup>  
Julien Launay<sup>2,3</sup>
Technology Innovation Institute, Abu Dhabi
https://huggingface.co/datasets/tiiuae/falcon-refinedweb
Large language models are commonly trained on a mixture of filtered web data and curated "high-quality" corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation, and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our REFINEDWEB dataset, and 1.3/7.5B parameters language models trained on it.
Figure 1: Models trained on  $\bullet$ REFINEDWEB alone outperform models trained on curated corpora. Zero-shot performance on our main-agg task aggregate (see Section 4.1 for details). At equivalent compute budgets (in PetaFLOPS-days), our models significantly outperform publicly available models trained on  $\nabla$  The Pile, and match the performance of the GPT-3 models.
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13
Progress in natural language processing is increasingly driven by sheer compute scale alone [1]: as more compute is expended to train large language models (LLM), they gain and exhibit powerful emergent capabilities [2, 3]. To best benefit from scaling, recent scaling laws dictate that both model size and dataset size should jointly be increased [4]. This is at variance with earlier findings, which had argued that scaling should focus on model size first and foremost, with minimal data scaling [5].
This joint scaling paradigm raises significant challenges: although plentiful, text data is not infinite, especially so when accounting for data quality and licensing—leading some researchers to argue scaling may soon be bottlenecked by data availability [6]. Concretely, optimally training a GPT-3 sized model (175B parameters) would require no less than 3,500 billion tokens according to [4]. This is twice as much as the largest pretraining datasets publicly demonstrated [4, 7], and ten times more than the largest publicly available English datasets such as OSCAR [8], C4 [9], or The Pile [10].
Massively scaling-up pretraining data is made even more challenging by the fact LLMs are commonly trained using a mixture of web crawls and so-called "high-quality" data [2, 10]. Typical high-quality corpora include curated sources of books, technical documents (e.g., research papers), human-selected web pages, code or social media conversations. The increased diversity and quality brought forth by these curated corpora is believed to be a key component of performant models [11]. Unfortunately, curation is labour intensive: typically, each source requires specialized processing, while yielding a limited amount of data. Furthermore, licensed sources can raise legal challenges.
Nevertheless, most pretraining data is still sourced out of necessity from massive web crawls—as they can be scaled up to trillions of tokens with limited human intervention. However, the quality of this data has traditionally been seen as (much) inferior to that of the manually curated data sources. Even finely processed sources of web data, such as C4 [9] or OSCAR [8], are regarded as inferior to curated corpora for LLMs [12, 11], producing less performant models.
To sustain the ever-increasing needs of larger and larger LLMs, and to streamline data pipelines and reduce the need for human-intensive curation, we explore how web data can be better processed to significantly improve its quality, resulting in models as capable as models trained on curated corpora.
Contributions. We make the following contributions:
- We introduce REFINEDWEB, a five trillion tokens web-only English pretraining dataset;
- We demonstrate that web data alone can result in models outperforming both public and private curated corpora, challenging current views about data quality;
- We publicly release a 600B tokens extract of RefinedWeb, and 1/7B parameters LLMs trained on it, to serve as a new baseline high-quality web dataset for the community.
Table 1: REFINEDWEB improves on existing English pretraining datasets for large language models by combining extensive filtering with stringent dedduplication at unprecedented scale. For additional details, see the full version in Table 12 of Appendix H.3
Dataset	Size	Availability	Web	CC Processing	Deduplication
MASSIVE WEB DATASETS
C4	~ 360GT	Public	100%	Rules + NSFW words blocklist	Exact: spans of 3 sentences
OSCAR-21.09	~ 370GT	Public	100%	Built at the line-level	Exact: per line (~ 55% removed)
OSCAR-22.01	~ 283GT	Public	100%	Line-level rules + optional rules & NSFW URL blocklist	Exact: per line (optional, not used for results in this paper)
CURATED DATASETS
■ GPT-3	300GT	Private	60%	Content filter trained on known high-quality sources	Fuzzy: MinHash (~ 10% removed)
▼ The Pile	~ 340GT	Public	18%	jusText for extraction, filter trained on curated data	Fuzzy: MinHash (~ 26% removed)
★ PaLM	780GT	Private	27%	Filter trained on HQ data	Unknown
OURS
●REFINEDWEB	~ 5,000GT	Public (500GT)	100%	trafilatura for text extraction, document and line-level rules, NSFW URL blocklist	Exact & fuzzy: exact sub-string+MinHash (~ 50% removed)
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25   
26   
27   
28   
29   
30   
31   
32   
33   
34   
35   
36   
37   
38   
39   
40   
41   
42   
43   
44   
45   
46
Pretraining data for large language models. Both GPT and BERT identified the importance of datasets with long, coherent documents [13, 14]. Moving from sentence-wise datasets [15], they instead leveraged document-focused, single-domain corpora like Wikipedia or BookCorpus [16]. As models increased in scale, datasets based on massive web-scrape gained prevalence [8, 9]. However, further work argued that these untargeted web scrape fell short of human-curated data [17], leading to the wide adoption of curated datasets such as The Pile [10], combining web data with books, research articles, conversations, and more. At scale, it has been proposed to emulate the human curation process by leveraging weak signals: for instance, by crawling the top links of a forum [18]. Targeted corpora can also produce domain-specific models [19], or broaden the expressiveness of models (e.g., for conversational modalities [20, 21]). Latest large language models [2, 12, 22, 23] are trained on giant aggregated corpora, combining both massive web-scrape and so-called "high-quality" curated single-domain sources. These targeted sources are often upsampled—from one to five times is most common—to increase their representation in the final dataset. The assumed diversity and higher-quality brought fourth by these aggregated datasets is thought to be central to model quality; web data alone is considered insufficient to train powerful large language models [24, 11].
Pipelines for web data. Massive web datasets are typically built upon CommonCrawl, a publicly available scrape of the internet. Working with data scraped from all over the internet presents unique challenges: notably, a significant portion is machine-generated spam or pornographic content [25, 26]. Accordingly, training on unfiltered web data is undesirable, resulting in poorly performing models [9]. Modern pipelines focus on filtering out undesirable content [27]. Broadly speaking, these pipelines usually combine a variety of stages: (1) language identification, leveraging inexpensive n-gram models (e.g., fastText [28]); (2) filtering rules and heuristics, such as only keeping lines with valid punctuation, discarding lines with too many symbols, or removing documents containing banned words [29, 9]; (3) ML-based quality filtering, using lightweight models trained on known gold data to identify similar high-quality web documents [27, 2]; (4) dedduplication, removing either exact duplicate spans or similar documents [30]. While some filtering is necessary, excessive filtering can introduce undesirable biases: this can overly impact minorities [31], motivating the adoption of practices such as pseudo-crawling, wherein allowed URLs are manually curated [32].
Deduplication. Deduplication removes repeated extracts and documents from a dataset: these could either be exact matches, identical in every character, or approximate matches, based on some similarity metric. For exact duplicates, it is common to match exact substrings of a minimum length using suffix arrays [33]. For fuzzy duplicates, methods based on locally-sensitive hashes such as MinHash [34] or SimHash [35] have seen wide adoption [2, 36, 12]. Recently, [37] has proposed to leverage embeddings to imbue semantic understanding in approximate matching algorithms. Deduplication has been identified as playing a significant role in improving language models [38, 30]. Notably, it reduces memorization [39], which is especially problematic in large models [40]. Furthermore, repeated data has been shown to be increasingly harmful to model quality as parameter count increases [41]: for a 1B parameters model, a hundred duplicates are harmful; at 175B, even a few duplicates could have a disproportionate effect. Concurrently to this work, the Pythia suite of models found that deduplicating The Pile had a limited impact on zero-shot performance [42], questioning whether dedduplication is as relevant for curated corpora as it for predominantly web-based datasets as studied in Lee et al. [30]. We provide an overview of some widely adopted pretraining English datasets for LLMs in Table 1, with additional information in Table 12 of Appendix H.3. We also note that recent popular open models [43, 7] often indirectly leverage The Pile [10] by doing a mix-and-match of its components.
With REFINEDWEB, we extend upon the state-of-the-art in three ways: (1) we aggregate and combine best-practices for document preparation and filtering across multiple pipelines, and introduce linewise corrections to fix lingering issues with text extraction; (2) we combine both exact and fuzzy dedduplication at very large-scale; (3) the scale of our final dataset is unique, with a total 5,000 billion tokens, and a 600 billion tokens extract available for public use with permissive licensing. Training large models on RefinedWeb also lead us to challenge the commonly held belief that web data is worse than curated corpora, as our models outperform others trained on so-called "high-quality" data.
47   
48   
49   
50   
51   
52   
53   
54   
55   
56   
57   
58   
59   
60   
61   
62   
63   
64   
65   
66   
67   
68   
69   
70   
71   
72   
73   
74   
75   
76   
77   
78   
79   
80   
81   
82   
83   
84   
85   
86   
87   
88   
89   
90   
91   
92   
93   
94   
95   
96   
97   
98
3 Macrodata Refinement and RefinedWeb
We introduce MDR (MacroData Refinement), a pipeline for filtering and deduplicating web data from CommonCrawl at very large scale. Using MDR, we produce REFINEDWEB, an English pretraining dataset of five trillion tokens based on web data only. We leverage strict filtering and stringent dedduplication to uplift the quality of web data, distilling it down to a corpus matching the quality of aggregated corpora used to train state-of-the-art models.
Design principles. We abide by the following guidelines:
- Scale first. We intend MDR to produce datasets to be used to train 40-200B parameters models, thus requiring trillions of tokens [4]. For English-only RefinedWeb, we target a size of 3-6 trillion tokens. Specifically, we eschew any labour intensive human curation process, and focus on CommonCrawl instead of disparate single-domain sources.
- Strict dedduplication. Inspired by Lee et al. [30], which demonstrated the value of dedduplication for LLMs, we implement a rigorous dedduplication pipeline. We combine both exact and fuzzy dedduplication, and use strict settings leading to high removal rates.
- Neutral filtering. To avoid introducing further undesirable biases into the model [31, 44], we avoid using ML-based filtering outside of language identification. We stick to simple rules and heuristics, and use only URL filtering for adult content.
3.1 Document preparation: reading data, filtering URLs, extracting text, and language identification
Reading the data. CommonCrawl is available in either WARC (raw HTML response), or WET files (preprocessed to only include plain text). Individual files correspond to a page/document/sample at a given URL. WET files would spare us from running our own HTML extraction; however, in line with previous works [10, 12], we found WET files to include undesirable navigation menus, ads, and other irrelevant texts. Accordingly, we start from raw WARC files, read with the warcio library.
URL filtering. Before undertaking any compute-heavy processing, we perform a first filtering based on the URL alone. This targets fraudulent and/or adult websites (e.g., predominantly pornographic, violent, related to gambling, etc.). We base our filtering on two rules: (1) an aggregated blocklist of 4.6M domains; (2) a URL score, based on the presence of words from a list we curated and weighed by severity. We found that commonly used blocklists include many false positives, such as popular blogging platforms or even pop culture websites. Furthermore, word-based rules (like the one used in
Figure 2: Subsequent stages of Macrodata Refinement remove nearly  $90\%$  of the documents originally in CommonCrawl. Notably, filtering and deduplication each result in a halving of the data available: around  $50\%$  of documents are discarded for not being English,  $24\%$  of remaining for being of insufficient quality, and  $12\%$  for being duplicates. We report removal rate (grey) with respect to each previous stage, and kept rate (shade) overall.
99   
100   
101   
102   
103   
104   
105   
106   
107   
108   
109   
110   
111   
112   
113   
114   
115   
116   
117   
118   
119   
120   
121   
122   
123   
124   
125   
126   
127   
128
Table 2: Macrodata Refinement aggregates best practices from the state-of-the-art and novel approaches (URL scoring, line-wise filtering, etc.) to produce high-quality web data. On dedduplication, we note that MDR is unique in both the scale at which it is performed, and in applying subsequently fuzzy and exact substring methods to improve coverage and scalability.
DOCUMENT PREPARATION	FILTERING	DEDUPLICATION
URL filtering	Text extraction	Language identification	Document-wise filtering	Line-wise filtering	Dedduplication	URL dedupli-cation
Aggregated blocklist, URL scor-ing, common HQ sources blocked	From WARC using sarcio, trafilatura for extraction	fastText classi-fier from CCNet, thresholding on top language score	In-document repetition removal and quality heuristics from MassiveWeb	Remove unde-sirable lines (call to actions, navigation buttons, social counters, etc.)	Fuzzy dedu-plication w/ MinHash + exact substring dedduplication w/ suffix arrays	Remove URLs revisited across Common- Crawl dumps
Appendix [1]	Barbaresi [46]	Wenzek et al. [27]	Rae et al. [12]	Appendix [12]	Lee et al. [30]	Section [3.3]
C4, [9]) can easily result in medical and legal pages being blocked. Our final detailed rules based on this investigation are shared in Appendix [1]. Since we intend RefinedWeb to be used as part of an aggregate dataset along with curated corpora, we also filtered common sources of high-quality data: Wikipedia, arXiv, etc. The detailed list is available in Appendix [1.1.3]
Text extraction. We want to extract only the main content of the page, ignoring menus, headers, footers, and ads among others: Lopukhin [45] found that trafilatura [46] was the best non-commercial library for retrieving content from blog posts and news articles. Although this is only a narrow subset of the kind of pages making up CommonCrawl, we found this finding to hold more broadly. We use trafilatura for text extraction, and apply extra formatting via regular expressions: we limit new lines to two consecutive ones, and remove all URLs.
Language identification. We use the fastText language classifier of CCNet [27] at the document-level: it uses characters n-gram and was trained on Wikipedia, supporting 176 languages. We remove documents for which the top language scores below 0.65: this usually corresponds to pages without any natural text. For this paper, we focus on English; RefinedWeb can also be derived for other languages, see Appendix F for details.
The data we retrieve at this stage, called RW-RAW, corresponds to what we can extract with the minimal amount of filtering. At this stage, only  $48\%$  of the original documents are left, mostly filtered out by language identification (and a small fraction by failures of the text extraction).
3.2 Filtering: document-wise and line-wise
Repetition removal. Due to crawling errors and low-quality sources, many documents contain repeated sequences: this may cause pathological behavior downstream [47]. The later dedduplication stage could catch this, but it is cheaper to catch it earlier document-wise. We implement the heuristics of Rae et al. [12], removing any document with excessive line, paragraph, or n-gram repetitions.
Document-wise filtering. A significant fraction of pages are machine-generated spam, made predominantly of lists of keywords, boilerplate, or sequences of special characters. Such documents are not suitable for language modeling; to filter them out, we adopt the quality filtering heuristics of Rae et al. [12]. These remove outliers in terms of overall length, symbol-to-word ratio, and other criteria ensuring the document is natural language. We note we adapted these filters on a per language basis, as they may result in overfiltering if naively transferred from English to other languages.
Line-wise corrections. Despite the improvements brought forth by using trafilatura instead of relying on preprocessed files, many documents remain interlaced with undesirable lines (e.g., social media counters [3 comments], navigation buttons [Home]). Accordingly, we devised a line-correction filter, targeting these undesirable items leftover from text extraction imperfections. If these corrections remove more than  $5\%$  of a document, we remove it entirely. See Appendix I.2 for details.
The data we retrieve at this stage has gone through all of the filtering heuristics in the MDR pipeline. We refer to this dataset as RW-FILTERED. Only  $23\%$  of the documents of CommonCrawl are left, with around  $50\%$  of the documents of RW-Raw removed by the filtering.
129   
130   
131   
132   
133   
134   
135   
136   
137   
138   
139   
140   
141   
142   
143   
144   
145   
146   
147   
148   
149   
150   
151   
152   
153   
154   
155   
156   
157   
158   
159   
160   
161   
162   
163   
164   
165
3.3 Dedduplication: fuzzy, exact, and across dumps
After filtering, although data quality has improved, a large fraction of the content is repeated across documents. This may be due to the crawler indirectly hitting the same page multiple times, to boilerplate content being repeated (e.g., licences), or even to plagiarism. These duplicates can strongly impact models, favoring memorization instead of generalization [30, 41]. Since dedduplication is expensive, it has seen limited adoption in public datasets [8, 9]. We adopt an aggressive dedduplication strategy, combining both fuzzy document matches and exact sequences removal.
Fuzzy dedduplication. We remove similar documents by applying MinHash [34]: for each document, we compute a sketch and measure its approximate similarity with other documents, eventually removing pairs with high overlap. MinHash excels at finding templated documents: licenses with only specific entities differing, placeholder SEO text repeated across websites—see examples of the biggest clusters in Appendix J.1. We perform MinHash dedduplication using 9,000 hashes per document, calculated over 5-grams and divided into 20 buckets of 450 hashes. We found that using less aggressive settings, such as the 10 hashes of The Pile [10], resulted in lower dedduplication rates and worsened model performance. See Appendix I.3.1 for more details about our MinHash setup.
Exact dedduplication. Exact substring operates at the sequence-level instead of the document-level, finding matches between strings that are exact token-by-token matches by using a suffix array [33] (e.g., specific disclaimers or notices, which may not compromise the entire document as showcased in Appendix J.2). We remove any match of more than 50 consecutive tokens, using the implementation of Lee et al. [30]. We note that exact substring alters documents, by removing specific spans: we also experimented with dropping entire documents or loss-masking the duplicated strings instead of cutting them, but this didn't result in significant changes in zero-shot performance—see Appendix I.3.2
URL dedduplication. Because of computational constraints, it is impossible for us to perform dedduplication directly on RW-Filtered. Instead, we split CommonCrawl into 100 parts, where each part contains a hundredth of each dump, and perform dedduplication on individual parts. Most of the larger duplicate clusters (e.g., licences, common spams) will be shared across parts, and effectively removed. However, we found that CommonCrawl dumps had significant overlap, with URLs being revisited across dumps despite no change in content. Accordingly, we keep a list of the URLs of all samples we have kept from each part, and remove them from subsequent parts being processed.
Table 3: To evaluate models trained on RefinedWeb and compare to the state-of-the-art, we build four aggregates across 18 tasks on which to measure zero-shot performance. small was built for internal ablations, based on tasks with consistent performance at small scale, core is based on tasks commonly reported for public suites of models [48, 42], main is based on tasks from the GPT-3 and PaLM paper [2, 22], and ext is based on tasks used by the BigScience Architecture and Scaling group [11]. We flag with  $\dagger$  results obtained in an arbitrary evaluation setup, and with  $*$  results obtained with the EAI Harness [49], which we also employ for all our models.
Tasks	Type	Random	small	core	main	ext
HellaSwag [50]	Sentence completion	25.0	✓	✓	✓	✓
LAMBADA [51]	Sentence completion	0.0	✓	✓	✓
Winogrande [52]	Coreference resolution	50.0	✓	✓	✓	✓
PIQA [53]	Multiple-choice question answering	50.0	✓	✓	✓	✓
ARC [54]	Natural language inference	25.0	✓	✓	✓	✓
OpenBookQA [55]	Multiple-choice question answering	25.0	✓	✓	✓
BoolQ [56]	Multiple-choice question answering	50.0	✓	✓	✓
COPA [57]	Sentence completion	50.0	✓	✓
CB [58]	Natural language inference	33.3	✓	✓
RTE [59]	Natural language inference	50.0	✓	✓
ReCoRD [60]	Question answering	0.0	✓
ANLI [61]	Natural language inference	33.3	✓
LogiQA [62]	Multiple-choice question answering	25.0	✓
HeadQA [63]	Multiple-choice question answering	20.0	✓
MathQA [64]	Multiple-choice question answering	20.0	✓
PROST [65]	Paraphrase identification	50.0	✓
PubMedQA [66]	Multiple-choice question answering	50.0	✓
SciQ [67]	Multiple-choice question answering	25.0	✓	✓
166   
167   
168   
169   
170   
171   
172   
173   
174   
175   
176   
177   
178   
179   
180   
181   
182   
183   
184   
185   
186   
187   
188   
189   
190   
191   
192   
193   
194   
195