The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only
Authors: Guilherme Penedo
Quentin Malartic
Daniel Hesslow
Ruxandra Cojocaru
Hamza Alobeidli
Alessandro Cappelli
Baptiste Pannier
Ebtesam Almazrouei
Julien Launay
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: web data
crawl
curated
deduplication
NLP
LLM
TL;DR: Adequately filtered and deduplicated web data alone can train models outperforming others trained on curated corpora such as The Pile

Abstract: Large language models are commonly trained on a mixture of filtered web data and curated ``high-quality'' corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation, and whether we will run out of unique high-quality data soon.  At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 500 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only
The Falcon LLM Team
Guilherme Penedo2
Quentin Malartic'
Daniel Hesslow
Ruxandra Cojocaru'
Hamza A Alessandro Cappelli"
Baptiste Pannier
Ebtesam Almazrouei"
Julien Launay2
Technology Innovation Institute, Dhabi
Abstract
Large language models are commonly trained on mixture of filtered web data and curated "high-qualit"" corpora, such social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it unclear how scalable is curation, and whether we will run out of unique high-quality data soon. variance with previous beliefs, we show that properly filtered and deduplicated web data alone lead powerful models; even significantly outperforming models trained The Pile. Despite extensive filtering, the high-quality data extract from the web is still plentiful, and we are able obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our REFINED WBB dataset, and 1. 1.3/7.5B parameters language models trained on it.
7
12
13
Aggregated zero-shot performance [main-ase aggagg %6
BS-ARS
GPT-Neo"
PaLM-68t
GPT-3 API"
GPT-3!
GPT-3 (paper))
PaLM
Ours (Pile)
Ours (RW)
56-
RefinedWeb (aurs)
GPT-3 (API)*
The Pile (ours)
45
Compute [PF-days]
Figure 1: Models trained REFINEDWEB alone outperform models trained curated corpora. Zero-shot performance on our main-agg task aggregate (see Section 4. details). At equivalent compute budgets (in PotaFLOPS-day our models significantly outperform publicly available models trained on The Pile, and match the performance the GPTT- models.
Submitted to 37th Conference on Neural Information Processing Systems (NcurIPS 2023) Track on Datasets and Benchmarks. Do not distribute.

---

Introduction
14
15
Progress in natural language processing is increasingly driven by sheer compute scale alone M: as more compute is expended to train large language models (LLM), they gain and exhibit powerful emergent capabilities 22] [. To best benefit from scaling, recent scaling laws dictate that both model size and dataset size should jointly be increased []). This is at variance with earlier findings, which had argued that scaling should focus on model size first and foremost, with minimal data scaling [S].
18
19
This joint scaling paradigm raises significant challenges: although plentiful, text data is not infinite, 21 especially so when accounting for data quality and licensing-leadin some researchers argue scaling may soon be bottlenecked by data availability [ Concretely, optimally training GPT-3 23 sized model (175B parameters) would require no less than 3,500 billion tokens according to [. This
20
24 is twice much as the largest pretraining datasets publicly demonstrated 44] 7 and ten times more C4 or The Pile ITO-
25 than the largest publicly available English datasets such as OSCAR g
26 Massively scaling-up up pretraining data made even more challenging by the fact LLMs are commonly Typical high-quality corpora include curated sources of books, technical documents research papers), duman-selecter
27 trained using mixture of web crawls and so-called "high-quality" data [
TC
29 web pages, code or social media conversations. The increased diversity and quality brought forth by
30 these curated corpora is believed to be a key component of performant models []]. Unfortunately,
31 curation is labour intensive: typically, each source requires specialized processing, while yielding a
32 limited amount of data. Furthermore, licensed sources can raise legal challenges.
Nevertheless, most pretraining data is still sourced out of necessity from massive web crawls-ar 34 they can be scaled up to trillions of tokens with limited human intervention. However, the quality of
35 this data has traditionally been seen as (much) inferior to that of the manually curated data sources.
36 Even finely processed sources of web data, such as C4 9 or OSCAR g are regarded as inferior to producing less performant models.
37
curated corpora for LLMs 12
38 To sustain the ever-increasing needs of larger and larger LLMs, and to streamline data pipelines and reduce the need for significantly improve its quality, resulting in models as capable as models trained on curated corpora.
curation, explore how web data can be better processed to
41
Contributions.
We make the following contributions:
We introduce REFINEDWEB, five trillion tokens web-only English pretraining dataset;
43
We demonstrate that web data alone can result in models outperforming both public and private curated corpora, challenging current views about data quality; We publicly release a 600B tokens extract of RefinedWeb, and 1/7B parameters LLMs trained on it, to serve as a new baseline high-quality web dataset for the community.
44
45
46
Table 1: REFINED W ER improves existing English pretraining datasets for large language models by combining extensive filtering with stringent deduplication at unprecedented scale. For additional details, the full version in Table 12 of Appendix
Dataset
Size
Availability
Web
CC Processing
Deduplication
MASSIVE WEB DATASETS
C4
360GT ~ 370GT
Public Public
100%
Rules NSFW words blocklist Built at the line-level
Exact: spans of sentences
OSCAR-21.09
100%
Exact: per line (~ 55% re- moved)
OSCAR-2201
~ 283GT
Public
100%
Line level rules + optional URT. blocklist
Exuct: line (optional, not for results in this paper)
CURATED DATASETS
GPT-3
300GT
Private
60%
Content filter trained on known high quality sources JusTert for extraction, filter trained on curated data
Fuzzy: MinHash (~ 10% re- moved)
The Pile
~ -400GT
Public
18%
MinHash (~> 26% re- moved)
Fuzzy:
Pal.M
780GT
Private
27%
Filter trained on HQ data
Unknown
OURS
REFINEDWEB
5, 000GT
Public (500GT)
100%
slaurr for text extrac- tion, document and line-level nules, NSFW URL blocklist
Exact fuzzy: exact sub-
string+MinHash (N 50% re- moved)

---

47 2
Related works
48
Pretraining data for large language models. Both GPT and BERT identified the importance of
49
datasets with long, coherent documents [33 14.
Moving from sentence-wise datasets [5). they
50 instead leveraged document-focused, single-domain corpora like Wikipedia or BookCorpus [6 As
51 models increased in scale, datasets based on massive web-scrape gained prevalence g 9 However, further work argued that these untargeted web scrape fell short of human-curated data [ leading to the wide adoption of curated datasets such as The Pile [0], combining web data with books, research articles, conversations, and more. At scale, it has been proposed to emulate the human curation process by leveraging weak signals: for instance, by crawling the top links of a forum [[]. Targeted corpora can also produce domain-spectie models (e.g., for conversational modalities 200 (1I). Latest large language models
53
54
55
56
models [19] or broaden the expressiveness of
57
22 [ are
trained giant aggregated corpora, combining both massive web-scrape and so-called "high-quality" curated single-domain sources. These targeted sources are often upsampled-from one to five times is most common-fo increase their representation in the final dataset. The assumed diversity and higher-quality brought fourth by these aggregated datasets is thought to be central to model quality; web data alone is considered insufficient to train powerful large language models 24
60
61
62
63
Pipelines for web data. Massive web datasets are typically built upon CommonCrawl, publicly available scrape of the internet. Working with data scraped from all over the internet presents unique challenges: notably, a significant portion is machine-generated spam pornographic content 225 26
64
65
66
Accordingly, training on unfiltered web data is undesirable, resulting in poorly performing models 9
87
Modern pipelines focus on filtering out undesirable content (27]. Broadly speaking, these pipelines usually combine a variety of stages: (1) language identification, leveraging inexpensive n-gram models (e.g., fastText (2)): (2) filtering rules and heuristics, such as only keeping lines with valid punctuation, discarding lines with too many symbols, or removing documents containing banned
68
69
70
71 words 22 1 (3) ML-based quality filtering, using lightweight models trained on known gold data to identify similar high-quality web documents
[27 2 (4) deduplication, removing either exact 73 duplicate spans or similar documents 30 While some filtering is necessary, excessive filtering
74 can introduce undesirable biases: this can overly impact minorities [31, motivating the adoption of
75
practices such as pseudo-crawling, wherein allowed URLs are manually curated 32)
76
Deduplication. Deduplication removes repeated extracts and documents from a dataset: these could either be exact matches, identical in every character, or approximate matches, based on some similarity metric. For exact duplicates, is common to match exact substrings of minimum length using [3 For fuzzy duplicates, methods based on locally-sensitive hashes such MinHash 35 have seen wide adoption 2
77
78
79
suffix arrays
80
56 12 Recently, 67 has proposed to leverage embeddings to imbue semantic understanding in approximate matching algorithms. Deduplication has 82 been identified as playing a significant role in improving language models
k
or SimHash
81
Notably, it reduces Furthermore, repeated data 84 has been shown to be increasingly harmful to model quality as parameter count increases 41:: for a
388
30
83
memorization 39]. which is especially problematic in large models 40
85 IB parameters model, hundred duplicates are harmful; at 175B, even few duplicates could have disproportionate effect. Concurrently to this work, the Pythia suite of models found that deduplicating The Pile had limited impact on zero-shot performance 42 questioning whether deduplication is as relevant for curated corpora as it for predominantly web-based datasets as studied in Lee et al. [30]
87
88
89 We provide an overview of some widely adopted pretraining English datasets for LLMS in Table T with additional information in Table 12 of Appendix H.3 We also note that recent popular open often indirectly leverage The Pile 110 by doing a mix-and-match of its components.
90
91
models 4 7
92
With REFINEDWEB, we extend upon the state-of-the-art in three ways: (1) we aggregate and combine best- pesttpractices for document preparation and filtering across multiple pipelines, and introduce line- wise corrections to fix lingering issues with text extraction; (2) we combine both exact and fuzzy deduplication at very large-scale; (3) the scale of our final dataset is unique, with a total 5,000 billion
93
94
95
96 tokens, and 600 billion tokens extract available for public use with permissive licensing. Training large models on RefinedWeb also lead to challenge the commonly held belief that web data is worse than curated corpora, our models outperform others trained on so-called "high-quality" data.
3

---

Macrodata Refinement and RefinedWeb
99
We introduce MDR (MacroData Refinement), pipeline for filtering and deduplicating web data from CommonCrawl very large scale. Using MDR, we produce REFINEDWEB, English pretraining dataset of five trillion tokens based web data only. We leverage strict filtering and stringent deduplication to uplift the quality of web data, distilling it down to corpus matching the quality of aggregated corpora used to train state-of-the-art models.
103
104
Design principles. We abide by the following guidelines:
106
Scale first. We intend MDR to produce datasets to be used to train 40-200B parameters models, thus requiring trillions of tokens 4 For English-only RefinedWeb, we target a size of 3-6 trillion tokens. Specifically, we eschew any labour intensive human curation process, and focus on CommonCrawl instead of disparate single-domain sources.
107
108
109
Strict deduplication. Inspired by Lee et al. [30] which demonstrated the value of dedupli- cation for LLMs, we implement rigorous deduplication pipeline. We combine both exact and fuzzy deduplication, and use strict settings leading to high removal rates.
112
113
Neutral filtering. To avoid introducing further undesirable biases into the model we avoid using ML-based filtering outside of language identification. We stick to simple rules and heuristics, and use only URL filtering for adult content.
114
115
116
3.1
Document preparation: reading data, filtering URLs, ss, extracting text, and language identificatinn
117
118
Reading the data. CommonCrawl available in either WARC (raw HTML response), or WET files (preprocessed to only include plain text). Individual files correspond to page/document/sample given URL. WET files would spare us from running our own HTML extraction; however, in line with previous works [TO 12,, other irrelevant texts. Accordingly, we start from raw WARC files, read with the warcio library.
121
we found WET files to include undesirable navigation menus, ads, and
122
URL filtering. Before undertaking any compute-hear processing, perform first filtering based on the URL alone. This targets fraudulent and/or adult websites (e.g., predominantly pornographic, violent, related to gambling, etc.). We base our filtering on two rules: (1) an aggregated blocklist of 4.6M domains; (2) a URL score, based on the presence of words from a list we curated and weighed by severity. We found that commonly used blocklists include many false positives, such as popular blogging platforms or even pop culture websites. Furthermore, word-based rules (like the one used in
125
126
127
128
URL
Text extraction
Langunge stenciftration
Repetition reenoval
Document-wie mitriig
Eineswiee commetions
Fassy dedupiiattion
Exact dediepitratinn
Ailering
23.34%
18.1%%
35,97%
47,5995
50%
16.9%
24.2%%
toons
01,200
96.31%
preparation
Flitcting
Deduplication
Figure 2: Subsequent stages of Macrodata Refinement remove nearly 90% of the documents originally in CommonCrawl. Notably, filtering and deduplication each result in a halving of the data available: around 50% of documents are discarded for being English, 24% of remaining for being of insufficient quality, and 12% for being duplicates. We report removal rate (grey) with respect each previous stage, and kept rate (shade) overall.

---

Table 2: Macrodata Refinement aggregates best practices from the state-of-the-art and novel approaches (URL scoring, line-wise filtering, etc.) to produce high-quality web data. On deduplication, We note that MDR is unique in both the scale at which it is performed, and in applying subsequently fuzzy and exact substring methods to improve coverage and scalability.
DOCUMENT PREPARATION
FILTERING
DEDUPLICATION
URL filternng
Text extraction
Langmage identification
Document-wise filtering
Line-wise filtering
Deduplication
URL dedupli- cation
Aggregated blocklist, URT. scor-
WARC using varcio, trafilatura for extraction
From
fastText classi- fier from CCNet, thresholding on top language score
In-document repetition removal and quality heuris- from Massive Web Rae et al. [
Remove unde- sirable lines (call actions,
Fuzzy dedu-
Remove URLs
plication w/
revisited across
MinHash +
Common- Crawl dumps
ing, common HQ sources
navigation buttons, social counters, Appendix 0.2
exact substring
lies
deduptication w/ sullix arayy
blocked
Appendix
1.
Barbaresi
16
Wenzek et al.
Lee et al. 20
Section 3.3
129
C4, can easily result in medical and legal pages being blocked. Our final detailed rules based on this investigation are shared in Appendix [.] Since we intend RefinedWeb be used as part of an aggregate dataset along with curated corpora, also filtered common sources of high-quality data: Wikipedia, arXiv, etc. The detailed list is available in Appendix [13
130
131
132
135
Text extraction. We want to extract only the main content of the page, ignoring menus, headers, footers, and ads among others: Lopukhin
134
45 found that traf ilatura (46) was the best non- commercial library for retrieving content from blog posts and news articles. Although this only narrow subset of the kind of pages making up CommonCrawl, we found this finding to hold more broadly. We use traf i latura for text extraction, and apply extra formatting via regular expressions: we limit new lines to two consecutive ones, and remove all URLs.
135
136
137
138
139
Language identification. We use the fastText language classifier of CCNet 2 at the document- level: it uses characters n-gram and trained on Wikipedia, supporting 176 languages. We remove documents for which the top language scores below 0.65: this usually corresponds to pages without any natural text. For this paper, we focus on English; RefinedWeb can also be derived for other languages, see Appendix F for details.
140
143
The data retrieve this stage, called RW-RAW, corresponds to what minimal amount of filtering. At this stage, only 48% of the original documents are left, mostly filtered out by language identification (and a small fraction by failures of the text extraction).
extract with the
145
146
147
3.2
Filtering: document-wise and line-wise
148
Repetition removal. Due to crawling errors and low-quality sources, many documents contain
149
repeated sequences: this may cause pathological behavior dowstream 47 The later deduplication stage could catch this, but it is cheaper to catch it earlier document-wise. We implement the heuristics of Rae et al. 12. removing any document with excessive line, paragraph, or n-gram repetitions.
151
152
significant fraction of pages are srcchine-generaee spam, made predominantly of lists of keywords, boilerplate, or sequences of special characters. Such documents not suitable for language modeling; to filter them out, we adopt the quality filtering heuristics of Rae et al. [12] These remove outliers in terms of overall length, symbol-to-wol wood ratio, and other criteria ensuring the document is natural language. We note we adapted these filters on a per language basis, as they may result in overfiltering if naively transferred from English to other languages.
Document-wist filtering.
153
155
156
157
158 I Line-wise corrections. Despite the improvements brought forth by using traf i latura instead of relying on preprocessed files, many documents remain interlaced with undesirable lines (c.g., social media counters [3 comments], navigation buttons [Home]). Accordingly, we devised line-correction filter, targeting these undesirable items leftover from text extraction imperfections. If these corrections
160
161
162
remove more than 5% of a document, we remove it entirely. See Appendix [.2 for details.
163
The data we retrieve at this stage has gone through all of the filtering heuristics in the MDR pipeline. We refer to this dataset as RW-FILTERED. Only 23% of the documents of CommonCrawl are left, with around 50% of the documents of RW-Raw removed by the filtering.
5

---

186
3.3
Deduplication: fuzzy, exact, and across dumps
187
After filtering, although data quality has improved, large fraction of the content is repeated across documents. This may be due to the crawler indirectly hitting the same page multiple times, to boilerplate content being repeated (e.g., licences), or even to plagiarism. These duplicates can strongly impact models, favoring memorization instead of generalization 60 !I
188
169
170
Since deduplication is
expensive, it has seen limited adoption in public datasets S We adopt an aggressive deduplication
strategy, combining both fuzzy document matches and exact sequences removal.
173
F uuzzy deduplication. We remove similar documents by applying MinHash [4] for each document, we compute a sketch and measure its approximate similarity with other documents, eventually removing pairs with high overlap. MinHash excels at finding templated documents: licenses with only specific entities differing, placeholder SEO text repeated across websites-see examples of the biggest clusters in Appendix J.I We perform MinHash deduplication using 9,000 hashes per document, calculated over 5-grams and divided into 20 buckets of 450 hashes. We found that using less aggressive settings, such as the 10 hashes of The Pile 10,, resulted in lower deduplication rates and worsened model performance. See Appendix
174
175
176
177
133.1 for more details about our MinHash setup.
181
Exact deduplication. Exact substring operates at the sequence-leve instead of the document-level. finding matches between strings that are exact token- ooken-by-token matches by using a suffix array [K] (e.g., specific disclaimers or notices, which may not compromise the entire document as showcased in Appendix
182
183
184
We remove any match of more than 50 consecutive tokens, using the implementation of Lee et al. 30 We note that exact substring alters documents, by removing specific spans: also experimented with dropping entire documents or loss-masking the duplicated strings instead of cutting them, but this didn't 't result in significant changes in zero-shot performance-see Appendix I.3.2
187
188
URL deduplication. Because of computational constraints, it is impossible for perform deduplication directly on RW-Filtered. Instead, we split CommonCrawi into 100 parts, where each part contains hundredth of each dump, and perform deduplication on individual parts. Most of the larger duplicate clusters (e.g.. licences, common spams) will be shared across parts, and effectively removed. However, we found that CommonCrawl dumps had significant overlap, with URLs being revisited across dumps despite no change in content. Accordingly, we keep list of the URLs of all samples have kept from each part, and remove them from subsequent parts being processed.
192
193
194
195
Table 3: To evaluate models trained on RefinedWeb and compare to the state-of-the-art, build four aggregates across 18 tasks on which to measure zero-shot performance. small was built for internal ablations, based on tasks with consistent performance at small scale, core is based on tasks commonly reported for public suites of models 48 42 main is based on tasks from the GPT-3 and PaLM paper [2] 22] and ext based on tasks used by the BigSeience Architecture and Scaling group IT. We flag with results obtained in an arbitrary evaluation setup, and with results obtained with the EAI Harness 4 which also employ for all our models.
Tasks
Type
Random
smlll
main
HellaSwap
M
Sentence completion
25.0
LAMBADA
51
Sentence completion
0.0
Winogrande
Corefereace resolution
50.0
PIQA
[3]
Maliiple-chiiee question unswering
50.0
ARC
Natural language inference
25.0
OpenBookOA
35
Multiple-choice question answering
25.0
BcolQ 300
Multiple-choice question answering
50.0
Sentence completion
CB 28
Natural language inference
33.3
RIE kX
Natural language inference
50.0
ReCaRD
Question answering
0.0
ANLI 61
Natural language inference
33.3
LogiQA
Multiplo-choice question answering
25.0
HcadQA
Multiple-choice question answering
20.0
MainQA
Muliiple choice question answering
PROST X
Paraphrase identification
50.0
PubMelOA AR
Mulliple-hhoie question unswering
50.0
SciQ
Multiple choice question answering
25.0

---

Table 4: Curation is not a silver bullet for zero-shot generalization: small-scale models trained on REFINEDWEB outperform models trained on web data (C4, OSCAR), and on curated corpora (V The Pile). Average accuracy in zero-shot on the small-agg aggregate. All models trained with identical architectures and pretraining hyperparameters, for the same amount of tokens. We find that OSCAR-22.01 underperforms other datasets signficantly, perhaps because deduplication is only optional. C4 is a strong baseline, with OSCAR-21 .09 lagging slightly behind, but we find that RefinedWeb outperforms both web datasets and the most popular curated dataset, The Pile. Both filtering and deduplication contribute significantly to improving zero-shot performance.
MASSIVE DATASETS
CURATED
OURS
OSCAR-21.09
OSCAR-22.01
Pile
RW-Raw
RW-Filterd
REFINEDWES
IB@27GT
55.0%
52.7%
55.7%
53.4%
52.7%
54.3%
56.2%
@
59.1%
55.9%
59.6%
57.9%
57.4%
58.2%
59.8%
Experiments
196
197
We now validate that models trained on Wef can match the zero-shot performance obtained with curated corpora and by state-of-the-art models. We first discuss our evaluation and pretraining setup, and models with which we compare. We perform experiments at small scale to internally compare with other datasets, and ablate the stages of RefinedWeb (raw, filtered, final). Then, we scale
198
199
200
201
to IB and 7B models trained on 350GT to compare with the state-of-the-as Finally, we apply the
202
MDR pipeline to existing datasets, and show that it can potentially deliver further improvements.
203
4.1 Setting
204
we focus our evaluation on zero-shot generalization across many tasks rather than measuring validation loss. Perplexity alone can be at odds with end-task performance 68) and modern works on LLMs
Evaluation. At variance with previous works studying pretraining datasets 12 30
205
206
predominantly report zero-shot performance 12 22 Furthermore, zero-shot generalization is the "natural" setting for autoregressive decoder-only models, in which they perform best [69]. Our evaluation setup is inspired by the one used by the architecture and scaling group of Big Science []]
209
210
We base our evaluation on the Fleuther Al evaluation harness 449 allowing us to evaluate across a wide range of tasks. We identified aggregates allowing us to: (1) obtain signal (i.e., non zero zero-shot performance) at small scale for ablations; (2) compare with results reported by other models. We outline these aggregates small (for ablations), and core, main, ext (for comparisons) in Table B
211
212
214
Comparisons across models trained and evaluated in different settings are difficult to untangle, as many externalities may influence the results (e.g., numerical precision of training vs inference, prompts used). We distinguish three levels of comparisons: (1) internal comparisons, with models trained and evaluated within our codebase, for which only the pretraining datasets differ; (2) benchmark-level comparisons, with models trained with different codebase but evaluated with the Eleuther AI
215
216
217
harness, taking results from [II 70 71 48), thereafter flagged with *; (3) external comparisons with thereafter flagged with f. For further details on evaluation, see Appendix H.1
220
221
Models. We train 1B, 3B, and 7B parameters autoregressive decoder-only models, based on configu- rations and hyperparameters similar to GPT-3 22] diverging mostly on our use of ALiBi FlashAttention
222
We use
223
in a custom codebase. We train internal models on both The Pile and RefinedWeb to control for deviations caused by our pretraining setup- we found The Pile models to perform in-line with others. For small-scale and ablation studies (first half of Section 4.2 Section
4.3) we train
226
models to optimality according to the scaling laws of Hoffmann et al. 4 on 27B and 60B tokens respectively for our 1B and 3B parameters models. For the main experiments demonstrating our we train the models to 350GT, in line with popular Note that we do not compare against the recently introduced L.aMA models as the smallest of them trained x2.5 more compute than our largest model, preventing meaningful comparison from being made dataset-wise For more in-depth overview of the models and pretraining datasets with which we compare, Appendix H
227
228
approach (Falcon-FF) RR models in Section
229
public models 2
74 2
230
7

---

233
4.2
Can web data alone outperform curated corpora?
234
We endeavour to demonstrate that web data alone can result in models outperforming models trained on curated corpora. To do so, we first perform a small-scale study with IB and 3B parameters models trained to optimality (27GT and 60GT) on popular web and curated datasets. Then, we scale up to 1B and 7B models trained on 350GT, and compare zero-shot generalization to state-of-the-an art models.
235
236
237
238
Small-scale study. We first consider public web datasets (OSCAR-201) [ OSCAR-2022 75),
239
C4 9), The Pile [TO] as the most popular publicly available curated dataset, and variations of Web (RW-Raw, RW-Filtered, and RW described in Section B All models are trained with
240
the same architecture, for the same amount of tokens, and using the same internal codebase; they also all evaluated within the same framework-onl pretraining datasets differ.
243
Results averaged on the small aggregate of 6 tasks are presented in Table We observe relatively strong performance of all web datasets compared to The Pile, showcasing that curation not a silver bullet for performant language models. We find C4 to be a strong pretraining dataset, in line with the findings of Scao al. hhwwever The Pile underperforms more in our benchmarks. The disappointing results OSCAR-22.01 may be due the dataset being distributed without deduplication by default. Regarding RefinedWeb, both filtering and deduplication significantly improve performance. We also note that 3B @ 60GT model trained on OSCAR-22.1 performs worse than a IB 22GG model trained on RefinedWeb: data alone accounts for a 4x difference in pretraining compute, highlighting that compute budgets alone cannot compensate efficiently for inadequate data.
244
245
248
249
250
251
Full-scale models. We now validate these results with comparisons with state-of-the-ar models. We scale our previous experiments by training IB and 7B models on 350GT; we also train a IB model on 350GT on The Pile, as a control for the influence of our pretraining setup. We compare with the
253
254
255
following models: the GPT-3 series (2) the FairSeq series (76), the GPT-Neo(X)// yJ models 77 74 70)
256
the OPT series [3]. the BigScience Architecture and Scaling Pile model [T PaLM-8B 22 Aleph
257
Alpha Luminous 13B 771 the Pythia series [2) and the Cerebras-GPT series 48
For GPT-3, we
distinguish between results obtained through the API (babbage and curie) with the the EleutherAI
259
49 (*), and results reported in their paper, with a different evaluation setup (t). For PaLM and OPT, results were obtained also with a different evaluation suite (t); for most other models they were obtained with the evaluation harness (*), allowing for more direct comparisons.
LM evaluation harness
280
281
262
Results on main-agg are presented in Figure D and in Figure for core-agg and ext-agg. We find that open models consistently underperform models trained private curated corpora, such
60
Aggregitad zero sbot performnca [corerage N.
BS
RefasedWeb (ours)
GPT:3 APT
Nacers porrspoedd ze1o shat perfo perfrrance fernance (art
(Pii)
RefiowdWWll (nurs)
OPT!
Oum (RW)
Alegh Alpha"
Cordras GPT
FaiSSeo
50
GPT-3"
Prehia
GPT
Pychia
OFT!
GPT-NeoX 1"
The Pau (ours)
13 models trained The Pie
Ours (RW)
10
1/2
10
107
Compute [PF-days]
Compute [PF-dao]
(a)
(b)
Figure 3: Models trained on REFINEDWER alone outperform models trained on curated cor- pora. Zero-shot performance averaged on core-agg (left) and ext-agg (right) task aggregates (see Section 4.T for details, and Figure for results on main-agg). Existing open models fail to match the performance of the original GPT-3 series (left); however, models trained on RefinedWeb significantly outperform models trained on v The Pile: including our direct comparison model (right), ruling out pretraining setup the main source of increased performance. In fact, our RefinedWeb models even match the performance of the GPT-3 models.
8

---

Table 5: Although improvements from filtering not systematic across datasets, deduplication brings a steady performance boost across the board. Zero-shot accuracy averaged on small-agg aggregate: [+x.x] reports absolute gains compared to base, removal rates reported against base. Due to limitations in our pipeline, we cannot apply the deduplication stage independently for RefinedWeb.
MASSIVE WEB DATASETS
CURATED
OurS
OSCAR-21.09
OSCAR-22.01
C4
Pile
RelinedWeb
Base
55.0%
52.7%
55.7%
53.4%
52.7% 54.3% [+1.6]
Filtered
55.4% [+.4]
52.3% [-.4]
56.2% [+.5]
54.2% [+.8]
removal rate
-25.0%
-39.8%
-16.4%
-27,
-50.8%
Deduplicated
55.6% [+,6]
55.6% [+2.9]
55.9% [+.2]
54.5% [+1.1]
removal rate
-10.8%
-608%%
-7.59%
-45,3%
Dedu.
55.5% [+.5]
55.4% [+2.7]
56.4% [+.7]
55.2% [+1.8]
56.2% [43.5]
removal rate
-28.2%
-62.2%
-17.9%
-66.0%
-75.4%
284
as GPT-3-even when using a similar evaluation setup. Conversely, models trained on Refined are able to match the performance of the GPT-3 series using web data alone, even though common high-quality sources used in The Pile are excluded from RefinedWeb (see Table [4 in Appendix). Finally, we note that our internal model trained on The Pile performs in line with the BigScience Architecture and Scaling model; this highlights that our pretraining setup is unlikely to be the main source of increased performance for models trained on RefinedWeb.
265
266
267
Finding, Challenging beliefs on data quality, filtered and deduplicated web data alone allows models to match the natural language tasks performance of models trained on curated data.
4.3
Do other corpora benefit from MDR?
271
Ablating the contributions and evaluating the performance of individual components in the MDR pipeline is difficult: for most heuristics, there is no agreed-upon ground truth, and changes may be too insignificant to result in sufficient zero-shot signal after pretraining. In the first half of Section 4.2 we identified that subsequent stages of RefinedWeb (raw, filtered, final) led to improvements in performance. In this section, we propose apply independently the filtering and deduplication stages of MDR to popular pretraining datasets, studying whether they generalize widely.
272
273
274
275
277
We report results on the small-agg in Table 5 First, we find that improvements from filtering are not systematic. On The Pile. we had to adjust our line length and characters ratio heuristics to avoid expunging books and code. Despite improvements on OSCAR-21.00 2.009, C4, and The Pile, our filters worsen performance on OSCAR-22.01, generally, removal rates from filtering are not strongly correlated with downstream accuracy. Conversely, deduplication delivers steady boost across all datasets, and removal rates are better correlated with zero-shot improvements. OSCAR-21 -1.09 and
278
279
280
283
C4 are already well deduplicated, while The Pile and OSCAR-22 -20001 exhibit 40-60% duplicates. OSCAR-22.01 is distributed without deduplication by default; for The Pile, this is consistent with the findings of Zhang et al. 4 Finally, combining filtering and deduplication results in further improvements; although performance is now more uniform across datasets, differences remain, suggesting that flaws in the original text extraction and processing are not fully compensated for.
284
285
286
287
288
By processing C4 with MDR, we are able to obtain subsets of data which might slightly outperform RefinedWeb; this combines both the stringent filtering of C4 (e.g.. strict NSFW word blocklist, 3-sentence span deduplication) with our own filters and deduplication. While this results in rejection rates that are unacceptable for our target of 3-6 trillions tokens, this is an interesting perspective for shorter runs, which may be able extract extremely high-quality subsets from large datasets.
289
290
291
292
Finding. While filtering heuristics may require source-dependent tuning, stringent deduplication improves zero-shot performance across datasets consistently.

---

5
Limitations
293
294
Biases and harmfulness. We conduct an analysis of the toxicity of RefinedWeb in Figure 5 of the Appendix. We find RefinedWeb to be about as toxic as The Pile, based on the definition of toxicity of the Perspective API: "content that is rude or disrespectful". Notably, this definition does not cover social biases or harmfulness. Although it is unlikely that our pipeline introduces further issues than is already documented for popular datasets, we encourage quantitative work on our public extract.
285
296
297
298
299
Performance beyond natural language. Our evaluation aggregates are overwelmingly targeting natural language tasks, and do not include code or mathematics evaluation- are popular use cases for fully-fledged models. A natural question may be whether web data alone is sufficient to achieve strong code/mathematies performance; we do not think this is the case, and encourage practionners to combine RefinedWeb with code datasets such The Stack 78 when training modles. However many of our findings apply equally: notably, Li et al. 779 found that deduplication helped with code data collected from GitHub as well. Broadly speaking, like web data is massively collected from CommonCrawl, code data is usually collected from GitHub, before undergoing extensive
300
301
302
303
305
306
307
filtering and deduplication. This is similar to the spirit of RefinedWeb, and does not rely on a collection of curated sources. Finally, we note that specific domains (e-g., code, technical papers) exist on a spectrum, and that general natural language improvements may benefit technical tasks too: for instance, we find that models trained on RefinedWeb outperform PubMedQA models trained on The Pile, despite not including any explicit medical data (The Pile includes PubMed).
308
309
312
And beyond pretraining... Our study is strictly limited language model pretraining, and does not address finetuning existing models. We note the value of high-quality samples for downstream specialization, for instance for improving chattiness or instruction capabilities
313
80
315
Multiple epochs. Instead of looking for "unique" tokens for trillion-scale pretraining dataset, one could simply repeat data over multiple epochs. Popular models like OPT and NeB train on up to 2 epochs [3 77,, and most curated datasets upsample corpora 2-5 times 2 J0 However,
316
318
41 has recently shown that models with 100B+ parameters may be sensitive to even just few epochs. Orthogonal to our work one could explore tradeoffs in the data-constrained regime: can deduplication help sustain more epochs? Are multiple epochs on higher quality data
Hernandez et al.
319
320
321
better than one epoch on lower quality data? See Appendix G.3 for a more in-depth discussion.
322
Other results on deduplication. Biderman et al. found limited impact on zero-shot per- formance from deduplicating The Pile; we discuss in Appendix H.2 and suspect deduplication may be unreasonably effective on web datasets because it predominantly removes low quality content (see Appendix
323
324
325
! for top samples). Muennighoff et al. [8] studied scaling laws for multiple epochs, and found that up to four epochs carried limited -ewaddatioo-oowveeee we note that many of the duplicates we find are present hundred to thousands of time in the raw data, far from this safe regime.
328
Conclusion
328
329 As LLMs are widely adopted, models trained past the recommendations of scaling laws are bound to become increasingly common to amortize inference costs 7 This will further drive the need for pretraining datasets with trillions of tokens, an order of magnitude beyond publicly available corpora. We have demonstrated that stringent filtering and deduplication could result in a five trillion tokens web only dataset suitable to produce competitive models, even outperforming LLMs trained on curated corpora. We publicly release a 600GT extract of RefinedWeb, and note that Reb has already been used to train state-of-the-as language models, such as Falcon-40B 82
331
332
333
334
335
We publicly release the following artefacts:
337 A 600B tokens extract of Refined
338
inedweeb
339
The 1B and 7B models trained on RefinedWeb in this paper:
340
tiiuae/falcon rw-1b and http::://hhhgggggcccee////eee//e
10

---

References
341
[I] Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalo- bos. Compute trends across three eras of machine learning, arXiv preprint 4r21022202.05924, 2022.
343
344
345
[2] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
348
[3] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
349
350
[4] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training optpual large language models. arXiv preprint 22..::22203.55566 2022.
352
353
354
[5] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
357
[6] Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets machine learning. arxiv preprint arXiv:2211.04322 2022.
358
359
380
[7] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
381
362
383
[8] Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd
365
386
July 2019, pages 9 - 16, Mannheim, 2019. Leibniz-Institut für Deutsche Sprache. doi: 10. 44618/isspubb- 902- URL hhtt:://nie resoovo-rrooliiiine.o de/ur /urunnn... nbn : de bsz : mh39-90215
387
368
[9]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020. URL
371
http://jmlr. www.rrrg/gpeee pttp:/paprrs///1//20022 0mmmmmmlm
372
[10] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00022, 2020.
373
375
[11] Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train you have one million gpu hours? arXiv preprint (tt.:22200114222 2022.
376
377
[12] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Ucsato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lor- raine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Ban-ttttist Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug
380
381
382
383
384
11

---

387
Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. 2021. doi: 10.4850/ARXI22212211444 URL
388
389
390
391
httpaaaiww.orrgbb///aa// wwt..orggabbb///..aa////a 11446
395
[13] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
396
397
[14] Jacob Devlin, Ming-Woit Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 4171-4186, 2019.
398
399
[15] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arviv: 13 /2.3055, 2013.
402
403
404
[16] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
405
408
[17] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. 2019.
409
410
[18] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus.
411
http://Skylion007. .gg www.hhu...//pp//eeetttttto www.pppen/eeTextcorrppppopo 2019.
412
[19]
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, 2019.
413
415
[20] Daniel Adiwardana, Minh- Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppi- lan, Zi Yang, Apoory Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 2020.
417
418
419
[21]
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint (iX.:2220.00823333 2022.
420
421
422
[22] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
424
425
[23]
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 776b-parameter open-access multilingual language model. arXiv preprint 10X0002211.05100, 2022.
426
427
428
429
[24] Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 19X.v:197.11192. 2019.
12

---

432
[25]
Trieu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint /rXivv/806.. 02847 2018.
438
[26]
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan Esch, Nasanbayar Ulzii- Orshikh, Allahsera Auguste Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguisties, 10:50-72, 2022.
435
436
437
[27]
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003 4012, 2020.
439
440
441
442
[28] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv: 2016.
443
444
[29] Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomáš Mikolov. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
447
[30] Katherine Lec, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language mod- els better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8445, 2022.
450
451
[31] Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1305, 2021.
453
454
455
[32]
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 th composite multilingual dataset. In siitt Conference on Neural Information Processing Systems Dutasets and Benchmarks Truck, 2022.
457
458
459
460
461
[33] Udi Manber and Gene Myers. Suffix arrays: new method for on-line string searches. Journal on Computing, 22(5):935-944) 1993.
462
[34] Andrei Z. Broder. On the resemblance and containment of documents. In Proceedings. Com- pression and Complexity of Sequences 1997. pages 21-29. IEEE, 1997.
484
[35] Moses Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium Theory of computing, pages 380-388. 2002.
467
(36] Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, al. Pangu-alpha: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXh::2104.12369, 2021.
468
471
[37] Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data-efficient learning web-scale through semantic deduplication. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
474
475
[38] Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143-153, 2019.
13

---

478
[39] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski. Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07666, 2022.
479
480
[40]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium ( (USENIX Security 21),
483
484
pages 2633-2650, 2021.
485
[41] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv: 10487, 2022.
488
[42] Stella Biderman, Hailey Schoelkopf, Quentin Authony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023.
489
490
492
[43] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint NXNiv:2205000066 2022.
493
494
[44] Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguisties:
496
487
498
EMNLP 2021, pages 2447-2469, 2021.
[45]
Konstantin Lopukhin. Evaluating quality of article body extraction for mercial services and open-source libraries.
501
2019.
502
[46] Adrien Barbaresi. Trafilatura: A Web Scraping Library and Command Line Tool for Text Discovery and Extraction. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 122-131. Association for Computational Linguistics, 2021. URL httpp://aclaattooll
503
506
org/2021. aal-demoo 15
507
[47] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
508
509
[48] Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint arXiv:2304.03200 2023.
510
511
[49] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https: //doi. org/10. 5281/zenodo. 5371628
514
515
516
[50] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
519
[51] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-15555 1534,
520
521
2016.
14

---

524
[52]
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99-106.
525
626
2021.
527
[53]
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
528
529
530
[54] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint 1803v.8803.0555 2018.
531
532
633
[55] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages 2381-2391 2018.
534
535
536
[56] Christopher Clark, Kenton Lee, Ming-Wai Wei Chang, Tom Kwiatkowski, Michael Collins, and
537
Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, pages 2924-2936, 2019.
538
539
[57] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. Semeval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoming. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume I: Proceedings the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394-398, 2012.
540
(58) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124, 2019.
546
[59] Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. Recognizing textual entailment: Rational, evaluation and approaches- -eraactmm Natural Language Engineering, 16(1):105-105. 2010.
548
549
[60] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: 12885, 2018
551
552
553
[61] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela, Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv: 14599, 2019.
554
555
[62] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: a 556
557
challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3622-3628, 2021.
558
559
560
[63] David Vilares and Carlos Gómez Rodriguuz Head-qa: A healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 966.966, 2019.
561
562
563
[64] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Concel-Kedzioree Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 2357-2361 2019.
564
585
15

---

588
[65]
Stéphane Prea-OOuellette Cory Paik, Alessandro Roncone, and Katharina Kann. Prost: Phys- ical reasoning about objects through space and time. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4597-4608, 2021.
569
570
571
[66] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567-2577, 2019.
572
575
[67] Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94-106, 2017.
578
577
578
[68] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representations, 2021.
579
580
[69]
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? In International Conference on Machine Learning, 2022.
583
584
585
586
[70]
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, al. Gpt-neox-22 20b: An open-source autoregressive language model. Challenges & Perspectives in Creating Large Language Models, page 95, 2022.
587
588
580
[71] Aleph Alpha. Luminous: performance benchmarks. arXiv preprint 18100 2023. URL httthttt///www/ aleph
591
592
[72] Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2021.
593
594
585
[73] Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, 2022.
598
[74] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. http://thhwbb.ccoooooooooo////aa
599
jax May 2021.
600
[75] Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. Ungoliant: An
601
optimized pipeline for the generation of a very large-scale multilingual web corpus. Proceedings of the Workshop on Challenges the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event) pages 1 9, Mannheim, 2021. Leibniz-Institut für Deutsche Sprache. doi: 14.1 8/ids huads-pub-lin 10468. URL
602
603
org/
605
bsz : mh39- 104688
608
[76] Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Vic- toria Lin, Jingfei Du, Srinivasan lyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684 2021.
607
608
609
[77] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL If you use this software, please cite it using these
610
//doi. org/ f0rg//1..e0 1...0.5281/zeoddd 5297715
metadata.
16

---

613
[78]
Denis Kocetkov, Raymond Li, LI Jia, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, et al. The stack: 3 th of permissively licensed source code. Transactions on Machine Learning Research, 2022.
614
615
616
[79] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.066. 166,,, 2023.
617
618
[80] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 1164,, 2023.
620
621
[81] Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2023.
623
624
[82] Ebtesam Almazrouei, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40b: an open large language model with state-of-the-art performance. 2023.
627
628
[83] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, 2021.
632
633
634
[84] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 38-45, 2020.
635
636
637
638
[85] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-99) 2021.
639
640
[86] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220-229, 2019.
642
643
644
645
[87] David M. Eberhard, Gary F. Simons, and Charles D. Fennig. Ethnologue: Languages of the World. SIL International, Dallas, TX, USA, twenty-sixth edition, 2023.
646
[88] Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:1 17044-177777 2021.
648
649
650
189] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text trans- former. In Proceedings of the 2021 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498. 2021.
652
653
654
655
[90] Julien Abadji. Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. Towards Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv e-prints, art. arXiv:2201 1.6642,, January 2022.
656
657
[91] Jan Pomikálek. Justext. 2011.
17

---

659
[92] Dick Sites. Compact language detector 2. Software available at https://githu.
com/CI.D20wnersccd22 (last updated on August 2015), 2013.
680
[93] Laura Hanu and Unitary team. Detoxify. Github. http://sittib.commmiiaaaaaaaaaia 2020.
682
[94] Jaccard. The distribution of the flora the alpine zone.1. New Phytologist, 11:37-50, 1912.
18


Reviews:
[{"review_id": "I3zD2GlFXW", "reviewer": "Reviewer_GCg1", "summary": "", "questions": "", "limitations": "The authors have fully addressed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The related works section is thorough. It covers many aspects of this work and I feel confident in the authors’ research in this area.\n2. MacroData Refinement. This pipeline for filtering and deduplicating datasets has multiple steps in which to catch low-quality or duplicated data, and used on a large scale dataset not previously been done before. \n3. There are several places where the authors mention how their design and pipelining decisions took bias amplification or bias mitigation into consideration which I appreciate.\n4. It is valuable insight to see how model performance changes across different steps in the MacroData Refinement pipeline. \n5. Evaluating models based on zero-shot performance instead of perplexity makes sense and is an interesting approach. I would have liked to see both evaluations done as there are benefits to both metrics, but I understand the authors’ reasoning.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Vz7gXar3fj", "reviewer": "Reviewer_2Ywv", "summary": "", "questions": "", "limitations": "The paper includes a study of toxicity (measured by Perspective API), and defers to future work for studying potential social biases as part of the dataset is publicly available. It would be good if the paper could use a subset of the data to do a study on potential NSFW content remaining after filtering based on URLs only.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The findings are interesting and the carefully filtered and deduplicated dataset that was released can greatly benefit future work.\n- The data preprocessing steps are clearly explained and well-motivated. The paper also reports how much data was left after each step, as well as showing the added benefits obtained from their preprocessing steps.\n- The paper compares baselines using both the same (internal) codebase as well as codebases used by previous work.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0KNnGp7oPg", "reviewer": "Reviewer_om9q", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper is well written and thoughtfully organized, including throughout the appendix.\n- It is full of interesting details about the dataset and filtering process\n- The evaluation is thoughtful and well-executed, and includes training several models from scratch, evaluating other models with the same codebase, and reporting results directly from other works. This gets around many of the core challenges with LM evaluations and comparisons. Additionally, it’s nice to have different evaluation splits, including the small split to get signal for smaller models.\n- Even though the story with respect to the Pythia models isn’t cleanly in line with this work’s, I appreciate that it is discussed openly (and provides future questions to investigate)\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "8O0178xTCG", "reviewer": "Reviewer_TYb3", "summary": "", "questions": "", "limitations": "N/A\n", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "\n1. Better scalability: Using filtered and deduplicated web data for model training can significantly reduce the reliance on curated corpora, which have scalability limitations.\n\n2. Good performance on LLM pre-training: The study shows that models trained using these extensive web-only datasets can outperform models trained on curated corpora.\n\n3. Resource contribution: The authors have created and released the RefinedWeb dataset (500B subset). This is a valuable contribution to the machine learning community, which would benefit resource-constrained researchers and projects. \n\n4. Toxicity: According to the perspective API's toxicity definition, the toxicity level of RefinedWeb is comparable to that of The Pile, which suggests a reasonable content quality.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper produces a large dataset from common crawl, and via careful cleaning, demonstrates that trained models are very capable.  The results in this paper will be interesting to the open-source community and could also provide insights on data curation and cleaning more broadly.
Decision: Accept (Poster)