What's In My Big Data?
Authors: Yanai Elazar
Akshita Bhagia
Ian Helgi Magnusson
Abhilasha Ravichander
Dustin Schwenk
Alane Suhr
Evan Pete Walsh
Dirk Groeneveld
Luca Soldaini
Sameer Singh
Hannaneh Hajishirzi
Noah A. Smith
Jesse Dodge
Venue: ICLR 2024 spotlight (2024)
Keywords: nlp
dataset
analaysis
data-statistics
data-quality
PII
Primary area: datasets and benchmarks
TL;DR: We propose "What's In My Big Data", a framework and a set of analyses to understand what's in large text-based corpora used to train language models.

Abstract: Large text corpora are the backbone of language models.
However, we have a limited understanding of the content of these corpora, including general statistics, quality, social factors, and inclusion of evaluation data (contamination).
In this work, we propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities---count and search---*at scale*, which allows us to analyze more than 35 terabytes on a standard compute node. 
We apply WIMBD to ten different corpora used to train popular language models, including *C4*, *The Pile*, and *RedPajama*.
Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content, personally identifiable information, toxic language, and benchmark contamination. 
For instance, we find that about 50% of the documents in *RedPajama* and *LAION-2B-en* are duplicates. In addition, several datasets used for benchmarking models trained on such corpora are contaminated with respect to important benchmarks, including the Winograd Schema Challenge and parts of GLUE and SuperGLUE.
We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.

Published as a conference paper at ICLR 2024
WHAT'S IN MY BIG DATA?
WIMBO
Yanai Elazar12
Akshita Bhagia'
lan Magnusson' Pete Walsh
Abhilasha Ravichander'
Dustin Schwenk'
Alane Suhr*
Dirk Groeneveld'
Luca Soldaini!
Sameer Singh*
Hannaneh Hajishirzi"?
Noah A. Smith 1.2 Jesse Dodge'
Allen Institute for AI
Paul G. Allen School of Computer Science & Engineering, University of Washington
3 UUniversity of California, Berkeley
*University of California, Irvine
https://ggttubb.mmmaalllaaaa/i/i
Q wimbd.apps.allena...
ABSTRACT
Large text corpora the backbone of language models. However, have limited understanding of the content of these corpora, including general statistics, quality, social factors, and inclusion of evaluation data (contamination). In this work, we propose WILAT'S S IN MY BIG DATA? (WIMBD), a platform and set of sixteen analyses that allow to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities count and search-as at scale, which allows us to analyze more than 35 terabytes on standard compute node, We apply WIMBD to ten different corpora used to train popular language models, including C4, The Pile, and RedPajama. Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content, personally identifiable information, toxie language, and benchmark contamination. For instance, find that about 50% of the documents in RedPajama and LAION-2B are duplicates. In addition, several datasets used for benchmarking models trained such corpora are contaminated with respect to important benchmarks, including the Winograd Schema Challenge and parts of GLUE and SuperGLUE, We open-source WIMBD'S code and artifacts to provide a standard set of evaluations for new text-based corpora and encourage more analyses and transparency around them.
INTRODUCTION
Data is the foundation upon which machine learning (ML) is built. The introduction of new datasets drives progress, playing a crucial role in facilitating research and the creation models with novel capabilities. Over time, the computational cost of Al experiments has dramatically increased, partly due to training increasingly large models on increasingly large datasets (Schwartz et al.) 20200 Sevilla al. 2022); today, some of the most impactful datasets are being created by scraping text from the entire yublicl--available internet (Raffel et al. 2020 Together Computer 2023 Penedo et al. 2023 Soldaini et al. 2024). These are some the largest text datasets that have ever been built, and they are typically introduced with only description of how they were made but no documentation of their contents. This is an important distinction, we are now training models on massive text corpora without knowing what ideas, topics, toxicity, or personal information they contain.
Meanwhile, language models (LMs) have become ubiquitous and are used by people worldwide daily. These Al systems directly impact people's lives, and thus, it has become vitally important to understand their capabilities and drawbacks. Models are only capable of learning from the data they were trained on, but analysis of pretraining corpora is hindered by lack of public release and by their massive size. Work analyzing the contents web-scale corpora typically focuses on a subset of important dimensions, and there has been almost work analyzing multiple datasets across the same dimensions. This means that ML practitioners have no practical tools to describe differences between datasets before choosing which one(s) to use.

---

Published as a conference paper at ICLR 2024
WIMBD
Building Blocks
Analyses
Distribution
Personslly
Contamination
Information (PII)
Ngrams
Counts
Bool
WIMBD
NNLU
Krel Xrel-typeerlg "5gg
Doman
Court
X0000
X
Figure I: An overview of WIMBD. We implement two fundamental capabilities: Count and Search, allowing quick processing and access to large text corpora, which enables wide range of analyses.
In this work, we propose investigate the content of large text corpora using WHAT'S IN MY BIG DATA (WTMBD), set of tools that enables practitioners to easily explore and quickly analyze large language datasets. We also use this tool to provide some of the first measurements across different web-scale datasets that are directly comparable. WIMBD has two components: (1) search tool that enables programmatic access to search for documents containing a query using an Elasticsearcl (ES) index. ES is search engine that allows retrieving strings from a corpus, the documents where they appeared, and the number of times they appeared. (2) a count functionality, built using map- map-reduce (Dean & Ghemawat 2008). , allowing quick iteration over entire dataset and extraction of relevant information, e.g., the character length distribution of documents, duplicates, domain counts, finding personally identifiable information (PII), and more. WIMBD is extendable and can be used to index, count, and analyze other corpora at scale (we benchmark the runtimes in Appendix[D.
Using these tools, we perform set of sixteen analyses on ten different English corpora used to train LMs, including C4 (used to train T5: Raffel et al. 2020). The Pile (used to train Pythia: Gao ct al.
2020 Biderman et al., 2022; 2023). J and RedPajama (used to reproduce Llama, Touvron et al.
2023 and to train RedPajjam--NN INCITE; Together Computer 2023). We divide our analyses into four
categories: (1) data statistics (e.g., number of tokens and domain distribution; $4.2); (2) data quality (e.g., most frequent re-grams and measuring duplicate documents; $4.3); (3) community- and society- relevant measurements (e-g., benchmark contamination and personally identifiable information detection; $4.4); and (4) cross-corpora analysis (c.g., comparing the most common r--ram and document overlap; $B.41 An illustration of WIMBD is presented in Figure
Our work presents many insights data distribution and anomalies. For example, inspecting the distribution over document lengths exposes anomalies where specific lengths are overrepresented relative to neighboring lengths; these anomalies often correspond to near-duplicate template genarrted text or documents arbitrarily truncated to a specific character length. As another example, punctuation sequences frequently the most common n-grams, such as dash ('-") repeated ten times as the most common 10-gram in The Pile. WIMBD offers both retrospective documentation and grounding of model behavior to their training data and actionable insights for higher-quality corpora curation.
BACKGROUND: ON THE IMPORTANCE OF DATA UNDERSTANDING
There have been repeated calls for ML practitioners to provide better data documentation (e.g.. McMillan--Majo et al. 2023] Bender& & Friedman 2018 Mitchell et al.) 2023 Pistilli et al.. 2023 Paullada et al.. [021 Gebru et al. (021)) On the other hand, some of the most impactful ML models are increasingly opaque, specifically with respect to the most important component of recent advancements: data. With the increasingly competitive nature of the field, developers of systems like GPT-4 (OpenAI 2023) and PaLM-2 (Google) (023) have been offering little transparency into the most important development decisions, including the sources, size, and contents of their training data.
As web-scale datasets drive this rapid progress in modern ML systems, the gap between data transparency and documentation is more striking than ever (Kaddour et al. 2023). From a technical standpoint, the massive size of these datasets makes analysis of their contents challenging; even if OpenAl or Google shared their training data, it's unclear where to start understanding it in its entirety. Tools like the Data Measurements Tool (Luccioni et al. 2021) and Know Your Data (Google) 2021) work towards improving data documentation, but focus on smaller datasets since the scale of web data leads to significant technical challenges. Our work aims to address this critical missing component.
httpss//wweelatticcoolllltttccct
2

---

Published as a conference paper at ICLR 2024
While other works support indexing and analyses of large corpora (Piktus al. 2023a Marone & Van Durme 2023] Simig et al. [002 Piktus et al.] 2023b; Razeghi et al. 2022b). these efforts support single corpus and often do not support programmatic access to the data or the analysis. Instead, we offer a holistic approach that combines search and counting with package that allows programmatic access through wrappers top of the ES API and extendable efficient counting capabilities.
Additional efforts are concerned with the effect of data on model behavior, Longpre et al. (2023)
investigate how the composition of LMs' pretraining data influences their downstream performance. Razeghi et al. (2022a) measure high correlation between term frequency and LMs' few-shot reasoning capabilities with those terms. Shin et al.) (2022) study the effect of pretraining corpora on in-context abilities. [Seshadri et al. (2023) demonstrate that text-to-image models mimic biases from their training data. Akkyrekk et (2022) study fact tracing for identifying pretraining examples that enable factual assertion, while Guu et al. (2023) offer training run simulator, which allows making counterfactual queries on what a model would have learned under a different training procedure. These efforts separately built dedicated infrastructure to perform the studies. Our work provides a dedicated interface and tooling that allows performing a wide range of analyses on large-scale corpora, categorizing and offering novel analyses that highlight new insights into these large corpora.
WIMBD: THE PLATFORM
core desideratum of WIMBD to enable quick processing of terabytes of data. As such, we fo- cus on uncomplicated, standard methods from the information retrieval and data management communities. WIMBD is comprised of two ba- sic components: counting and search (retrieval). Fast counting and retrieving enable us to an- swer fundamental questions about data, as demonstrate in Section 4 We summarize the framework abilities and types of analyses in Table We run our experiments using a compute node machine with 224 CPUs and 882GB RAM, and an Elasticsearch cluster for the indexed corpora.
Table I: Summary of the capabilities WIMBD provides and the analyses enabled by them.
Busic Ability
Analyses
Document Counts, mishuss dac langth, Rokero, divaibation, utterance statistiss
Foort Coante (63.1)
gcolocation, language distibutioon leagth distribation, loxic langunge. personallyy identifihhic information, damngraphic sentiment coweuurresss
Compressed Coont
3.1
Doplicates, most & least commos n-grams
Scorch o
1.2
Benchmak contamination, n-gram counts
3.1 COUNTING
Due to the sparsity of language data and the scale of the data of interest, accurate counting can be challenging. We leverage the map-reduce framework (Dean Ghemawat 2008). We provide two approaches for counting, described below.
The exact counts approach is designed for cases where the number of possible values is tractable and can fit in memory. This fits cases where we are interested in calculating a bound number of variables of interest (e.g., number of documents. 44.2 document length, $4.3.)..
Exact Counts
Compressed Counts The compressed counts approach is designed for cases where the number of
possible values is intractable. For instance, the total 10-grams in a large corpus can be very high, and the memory usage compute all of them would be overwhelming. Similarly, finding duplicates requires keeping and comparing the strings of all documents in memory. In the case of C4, that would require over 800 GB of RAM. Instead, we apply a compression function (e.g.. hashing, (Boom 1970) to those values, reducing memory footprint while sacrificing some accuracy (due to hash collisions). For example, when finding the most common 10- (0-grams, we store a table of counts where the keys in the table correspond to hashes of 10-grams. The hash table size is configurable according to the amount of memory available. The larger the hash table, the smaller the probability of hash collisions and, therefore, the higher the accuracy of the counts. E.g., unigram estimates are more accurate than 10-gram estimates since the number of possible values is much smaller.
3.2
SEARCHING
The second part of WIMBD allows fast text retrieval. For instance, can get the number of documents mentioning a word or sequence (document frequency). It also allows more complex Boolean queries. While search and retrieval have numerous implementations, such reverse indices, suffix arrays,
3

---

Published as a conference paper at ICLR 2024
Table 2: Summary statistics of the corpora, along with the models trained them. signifies that the model not trained on exact version we consider, either due some data mismatch, or the original data being private.
Corpos
Origin
Medel
Siae (GBl
+ Decuments
Tookens
maxcc Tokeess
minid Tokens!
OpenWeolest
Selladon & Cater
2019
GPT
Badtunl acal. 2019
41.2
5,000 539
7,767,7155, 349
95,129
125
838:
354. 200
153,807
966
3.938.73..774
2.703.877.566.901
RedPuuna
2025
5.002.0
930.453.48
28,121.329
S20RC
ScIBERT
11.241,44 .499
59.363.121.
370,081
p:S2a
2003
504.3
8,342.162
44,0045900,20 590,299
92,043
154
LAIUN 28-088880 en
Stakie Difflssoon Rombsch or 2002
57012
2.319.00 X27
29,641,500155
131,077
U
The Stack
2004
StarCoder" (L
20233
73808
546.741672
1,228018,225000
36,296.154
0
suffix trees for exact match search, and dense retrieval for fuzzy search, in this work, use ES, an inverted index. We build a wrapper on top of the ES API, allowing tailored and customized searches fit our analysis requirements. We leave future work explore other search alternatives.
WIMBD: THE ANALYSES
This section presents analyses conducted in WIMBD, grouped by category. First, we describe the corpora considered in this study ($4.1) We then consider four high-level categories, each split into several analyses: data statistics ($4.2). data quality ($4.3) and community- and society-relevant
measurements ($4.4). Cross-corpus analyses, well elaborations and more analyses presented in the appendix ($B) Our analyses are inspired by previous works (Dodge et al., [202]] Gao et al. 2020). but we expand them to multiple corpora, extend the types of analyses, and open-source our modular toolkit to encourage researchers to scrutinize their corpora. We offer the first extensive analyses on ten, combining extension of previous analyses and several novel ones.
4.1 CORPORA
We cover ten different large corpora, spanning across text-only (e.g., C4) to image captions (LAION- 2B-en) and code (The Stack). These corpora have been used in training language models (or similar large-scale models, such as Stable Diffusion; Rombach er al. 2022). A high-level description of these datasets using WIMBD is presented in Table 2 and further details about the construction and origin of these corpora are detailed Appendix A
4.2
DATA STATISTICS
Main Findings
Four out of the ten corpora we consider have 'empty' documents (meaning they contain only space-like characters), while The Pile and RedPajama contain the same longest document (with over 28 million tokens) of an encyclopedia.
While the most common source of webpages in C4 originates from www.nyt www.nytimmmm..mmm tintiimmsscm com consists of less than 0.05% of the total web pages, mC4-en most common domain is google.com (over 5% of the documents), and cdn.shopify.com contributes almost 6% the total documents in LAION-2B-en.
4.2. SUMMARY STATISTICS
We begin by computing some summary statistics and present the results in Table 2 Using the Exact Counts compute the following high-level statistics of corpus: (1) size, (2) number of documents, (3) number of tokens shortest document. Out of all corpora, mC4-en is the largest, which takes 14.7TB of disk, and 2.7 trillion tokens. After that comes The Stack with size of 7.8TB, and more than 1.5 trillion tokens. Interestingly, four corpora contain documents with empty strings: LAION-2B-en (81 total), which typically contain a sequence of white spaces. The Stack (1,350 total), RedPajama (3,877), and The
(4) the size of the longest document, and (5) the size of the
We use Unicode text segmentation (Unicode 2023 as tokenizer, but support any tokenizer supported
by HuggingFace's tokenizers library (Moi & Patry 2023
4

---

Published as a conference paper at ICLR 2024
RedPajema
greett.ggoggl..oo
Latring.com .co cdr.stoplyycom images slddep
anoiverg
stwippidaooo
caskswertliw.com
www.nylimss.cm
tragetens..slmmaaae
www.gulenberg.oom
joumals plo
Lakayling com sola photoshclerr con asdt alieón con gettylmages
de wkipesib.ang
www. www.latimes.co .om
wrkppeci..ogg
www.theuarddam..
walpecis.cog
whlipesia
www.hulboot.com
www.acrbbd.com
122r
0.0
0.1
0.2
0.9
04
10
Documents
% Doo, ments
Documents
Figure 2: Domain distribution of the most common domains per token for C4, LAION MAION-2B-cr 2B-en, and RedPajama.
Pile (7,533), documents typically contain mix of special characters that denote spacing (e.g., 'An', or "\t"). ). In RedPajama, all of the empty strings are from the arXiv subset. The longest document in The Stack is json file, with 26,298, 134 tokens from htttp://jqqee
com/ The longest document in The Pile and RedPajama is the same encyclopedia book called "NTEERATIINAL ENCYCLOPEDIA OF THE SOCIAL & BEHAVIORAL SCIENCES" from the Books3 subset with 28,121,329 tokens.
4.2.2 INTERNET DOMAIN DISTRIBUTION
Some corpora contain metadata information about the URL where the documents came from. As such, we employ the Exact Counts functionality, to parse the entire corpus, and extract informa- tion from the URLs about the (1) schemas (e.g., hilp, hilps). (2) domains (e.g.. www.google.com en.wikipedia.org. etc.), and (3) suffixes (e.g., com, org, de, etc.).
We apply these counts the corpora that contain this information, namely C4. mC4-en, OSCAR, RedPajama, and LAION-2B-er Starting with the domain analysis, we perform these counts twice: when each domain is counted per document (yielding documents per domain) and another where each domain is counted per token (yielding tokens per domain). We present the results of three corpora per token in Figure 2 (and the full results in Appendix [B.I]. First, that C4 contains documents from a diverse set of domains, and even the percentage of the most common one, patents gwwgle.oooooooooooooooooooo com is less than 0.05%. On the other hand, in the case of LAION- 2B-en, shopi wtw.comcl..... is responsible for more than 6% of the documents. Similarly, arxiv.org responsible for more than 12% of the documents in RedPajama. We showcase the results of the domains for the other corpora, as well as the schemas and suffixes in Appendix B.1
note
4.3
DATA QUALITY
Main Findings
The most common n- n-grams often correspond to repeated punctuation marks and duplicates. While more than 60% of documents in The Pile are duplicates (unsurprisingly due to oversampling), RedPajama and LAION-2B-en also contain about 50% duplicate documents. Document length distribution reveals interesting (and unexpected) outliers of documents, often resulting from duplicate documents and idiosyncratic data decisions.
4.3.1 MOST & LEAST COMMON n-GRAMS
Measuring outliers can reveal interesting insights about a corpus (Mitchell et al. 2023), We explore the most and least common token n-grams of cach corpus using the Compressed Counts We compute the 10K most common n-grams for all corpora, with n E 21, 10). We report the results of the ten most common 10-grams in Table and of the ten most common uni-, bi-, and tri-grams in Table in the Appendix. Identical n-grams across corpora are highlighted in the same colors.
The different соrрoга contain lot of uncleaned html markdown format (c.g., ten times "?' or 'amp"), ), or boilerplate texts such as: ". You can follow any responses to this entry through" in C4, or " Log Out / Change ) You are commenting using" in OSCAR, and formatting (" [1] [2] [3] (") in S20RC and peS2o, which signifies references.
A striking finding from this analysis is the vast repetition of such 10-grams. For instance, '.'.
and '_' repeated ten times appear 9. 7.2, and 4.4 million times, respectively, in C4. We perform manual analysis on the repeating question marks in C4 to better understand the scenarios where they
5

---

Published as a conference paper at ICLR 2024
Table Most common 10- grams in five of the corpora consider. y-grams from the top- 10 that occur in than one corpus are highlighted in same color.
MM
Coot
0000000000
K70
LIM
2112
2018
SeWil LEMILLEA (100331
10009)
Corphry speciigg
1-2
appear on the ten consecutive question marks symbols and categorize each appearance into writing, noise, and format occurrence. Analyzing 100 random documents, found that 68% of documents use such n-grams as part of their wriling style (e.g.. possible? .r.or... So what do thin??????
How is that due to noise as we could not understand the context or content of the writing (e.g,, kap chit-koa ??), and finally, 14% of the documents were due to styles or issues (e.g., a sequence of question marks following by a 'normal' text, or different format formaa
$6????????????
18%
???????????????
sequence of question marks between keywords).
4.3.2 DUPLICATES
Previous work has found that duplication can affect the quality of pretraining data, impacting sample efficiency (Lee et al. [022 Tirumala al. (023) and memorization (Carlini et al. (202)). While more recent work finds contradictory evidence on data with less web-scraped text (Biderman et al. 2023).
measuring duplication in pretraining data is necessary for future research on its effects. We calculate duplicates by matching documents with an MD5 hash of their texts (using Compressed Counts )
If more than a single document has the same hash, we consider them duplicates We examine the duplication of document text and URLs within each dataset. While some datasets explicitly deduplicate their content, others do not, and some even oversample some sources.
60-
tomal
Table 4: Most frequent text duplicates from four datasets with text duplicates, along with their counts. Truncation for visualization is marked by [...].
52
Ouptcale
42
Corpus
22
OSCAR
login you must registered, only moments gives you incroas)...
19
18M
Pile
"info" ('n "version" "authoe" "scoude" [u]
OSCAR
The Pile
RedPajama
S20RC
LA-ON-28-en
Count: 3.8K
RodPPajama
ACCEPTEDionAUtS According Ivlennteraatinnll
Figure 3: Percentages of document and document cluster duplicates in corpora with > 1% documents duplicated (corresponding blue and orange bars). Duplicate counts are above hars.
at Namesindestnineww Published inmnullalcstte Original n[...]
Coun:: 213.9K
LAION-2B-cn
Fromt Cover
IM
In Figure 3 we show counts and ratios of duplication across datasets with greater than 1% documents duplicated, and all datasets are shown Table i3 in the appendix. These are based on two kinds of counts: (1) the count of documents in all clusters of duplicate text (in blue) and (2) the count of duplicate clusters (in orange). As expected, deduplicated corpora such as C4 have no exact duplicates (as those were filtered out of the corpus). In contrast, The Pile, which intentionally oversampled some data sources, has many duplicates (139M documents belonging to 64.6M duplicate text clusters). LAION-2B-er has the second highest ratio of duplicate documents (1.25B documents belonging to 342M duplicate text clusters), perhaps due to the smaller space of short sentences common in
'To test for hash collisions, we rerun the analysis with a different random seed. None of the > 7 billion hashes across the ten corpora had a different count. This could only occur if identical number of collisions conflated identical set of counts or, more likely, there were no collisions.

---

Published as a conference paper at ICLR 2024
its image "alt text" source. Figure 15 in the appendix showcase the images of the most common duplicates LAION-2B with the most common images describe mainly receipts.
Table showcases duplicates with the most occurrences in four corpora. These duplicates vary dramatically in length and domain. LAION-2B-or en, OSCAR, and RedPajama have clusters with the most occurrences, in the hundreds of thousands and above. Top duplicates in LAION-2B-en shorter and describe products and website features. OSCAR's top duplicates are all instances of website boilerplate RedPajama's top duplicates come from similar templated citation information.
4.3.3 DOCUMENT LENGTH DISTRIBUTION
We compute document length distributions with Exact Counts We expect a smooth distribution over document lengths, and deviation from such distribution may indicate the presence of artificial documents or near duplicates acter length distribution and present results for three corpora in Figure (additional results in Appendix B.2.3).
OBCAR The Fie C4
DeepMind Mathematios
3 We compute the char-
Decumetts
08
04
"In order login you musi be regitered..
While C4 free of duplicate documents, it include clusters of cemplate-generated near- wear-duplicate documents exposed by out- liers of identical document lengths. Beyond cemplate-generater user-facing copy (e.g., cemplate-generatee documents from reverse phone lookup site, each associated with a unique phone number), we find clusters of cemplate-generater JavaScript snippets, and large collections of unique documents, including numerous permutations of the same keywords, likely crafted for SEO purposes.
FAQ for forum software php68
0.0
Charaders
Figure 4: Distribution over character document lengths (in log-scale) for C4, OSCAR and The Pile.
The Pile, featuring the longest documents, has a notable outlier with nearly 1% of its documents pre- cisely 8,194 characters long. These outliers are derived from the DeepMind Mathematics dataset (Sax ton et al., 2019), truncated to this length. The Pile also contains a significant number of short template-generateed code snippets, e.g., a number of documents (of lengths 9, 18, and 36 tokens) each corresponding to a unique publication in various medical journals, and to auto-generated metadata files (of length 20 tokens) used the Unity game engine. While OSCAR has no documents shorter than 100 characters, as those were filtered, it contains many near-duplicate documents that correspond website boilerplate, e.g., template-generated FAQS about how to use the forum software phpBB.
4.4 COMMUNITY- AND RELEVATT MEASUREMENTS
Main Findings
Instances of popular benchmarks like GLUE and SuperGLUE, were found in various corpora (e.g., C4 and RedPajama), render them unusable for fair model evaluation.
Automatic toxicity detection reveals that 1-16.5% of the documents in the corpora contain toxic language using an automatic classifier and between 0.01-16.6% using a taxonomy.
An estimated 200M, 4B, and 97M of email addresses, phone numbers, and IP addresses were found in the most PIl-contaminated corpora per token (mC4-en).
4.4.1 BENCHMARK CONTAMINATION
As corpora grow and new evaluation datasets are created, the risk of contaminationwhhee evaluation data are included in a (pre)traini)) training corpus increases. As such, it is important to track contamination
Sainz et al. 2023 Jacovi et al. 2023 b Using Search, we provide a contamination analysis of 82 datasets for four popular corpora: The Pile, C4, RedPajama, and OSCAR. We consider all datasets
Many of these duplicate documents indicate that the user agent used to collect the cataset received automatic responses blocking it from crawling the website's 's contents.
'Outlier lengths are those whose prevalence across the corpus significantly higher than neighboring lengths. 'When evaluating a model trained on an existing corpus. one should exempt contaminated evaluation sets. However, in the case of new corpus construction, practitioners may use WIMBD for decontaminating the corpus itself maintain the evaluation data integrity.
7

---

Published as a conference paper at ICLR 2024
100
Mories
supolgtin gio
glee 20ch
det, mdd
hudto 1eit
e000 oidt
argongion M
supregtion cupt
weseoed sen
subar gtre, un
Dutssct
Figure S: Most contaminated evaluations test sets out of 82 PromptSource Bach al. 2022 datasets.
from PromptSource (Bach et al. 2022), D. repository containing prompts for 279 different datasets
(as of May 2023). We filter datasets we cannot automatically download, from Huggingface datasets (Lhoest et al., 202T), and datasets that do not have a test split. In addition, only consider datasets that contain at least two inputs (e.g., natural language inference), leaving us with 82 datasets.
We measure contamination by testing whether all input fields are present in a single document and report the percentage of contaminated examples from the test set. Our contamination evaluation serves as an upper bound of exact-match dataset contamination. We provide more details of our analysis and design choices in Appendix B.3.
Contaminated datasets We present the results in Figure We showcase all benchmarks whose contamination percentages are at least 5% in one of the four corpora. We find that RedPajama is the most contaminated dataset out of the four, where in eight out of the 15 corpora, its contamination rate is above 50%, and fully contaminated in the case of COPA (Roemmele et al.) 2011). The Pile's contamination rates are lower, but it is also contaminated with few datasets, such as aesic (Zhang & Tetreault 2019), WSC (Levesque et al. 2012) and WIC (Pilehvar & Camacho- Collados 2019). which were included in the SuperGL.UE evaluation benchmark (Wang et al. 2019)..
Most examined datasets were not found in the corpora. It is important to note that while we find some contamination, most of the considered benchmarks do not appear in the corpora we investigated (67 out of the 82 datasets). For instance, Winogrande (Sakaguchi et al.) (021), a large corpus in the style of the Winograd schema, does not appear in any of the examined corpora.
4.4.2 PERSONALLY IDENTIFIABLE INFORMATION
PII is "information which can be used distinguish trace an individual's identity, such as their name, social se-
Table 5: Extrapolated PII frequencies. Count is extrapolated frequency and Prec. is our identification precision accuracy, estimated by manual analysis of 100 random examples.
curity number, biometric records, etc." (Johnson III (007). Recent research has sought to extract PII from LMs (Car- lini et al. 202T). These attacks highlight that LMs can ingest and reproduce PII contained in their training data, and show the risks of training on data that contains such information, even if the data remains private.
Curpus
Email Addresss
Phune Numbers
Addresus
OpentidrTox
364K
19
$13K
70%
OSCAR
(2
100)
10TM
51
3.2M
C4
7.436
00
19.7M
02
7063
mC3ken
201M
57.8M
38M
We document three kinds of personally identifiable in- formation in pretraining corpora: phone numbers, email addresses, and IP addresses. We employ regular expres-
S2ORC
ABIK
1.4M
1000
[K
0
pe52:
418K
57
227X
22
OK
0
LAION-9B-m
6T6l
04
IM
7
OK
0
The Stack
4.304
53
45.4M
4.4M
sions corresponding to each PII type using the Exaci Counts We provide more details about our
methodology, the regexes, additional results, and error analyses in Appendix BB..2 We conduct manual analysis to estimate the precision of these methods on all corpora. The results of this analysis, as well as the extrapolated frequency of these matches, are presented in Table Our identification method is highly precise (>80% precision) for email addresses on eight out of 10 corpora, and for phone numbers on five of the 10 corpora. Overall, most corpora contain a high volume of PII information, varying in type based on the corpus. For instance, RedPajama contain mainly phone numbers (70.2M) and a smaller amount of IP Addresses (M)) but S20RC and pe52o contain mainly email addresses (630K and 418K, respectively) and no IP addresses were identified. The most common PII across corpora is phone numbers, followed by email addresses and IP addresses (except for The Stack, which has more IP addresses than email addresses: 4.4M vs. 4.3M, and peS2o. which has more email addresses than phone numbers). Finally, we observe that mC4-en contains the largest amount of PII, also when controlling for the number of tokens (Table 19 in the Appendix).
8

---

Published as a conference paper at ICLR 2024
5 DISCUSSION
Data is one of the most poorly understood and studied components in ML research since "everyone wants to do the model work, not the data work" (Sambasivan et al., 2021). Yet, it is one of the most critical factors for successfully training a state-of-the language model. While the benefit of increasing model size is evident from the trend of recent years, it is not enough by itself, as the amount and quality of data are crucial (Kaplan et al. 2020).
Data Curation With the increasing data needed to train LMs (and other models for other modalities), it remains challenging to curate high-quality datasets. Besides the technical challenges of composing a large-scale dataset and the decisions that go into making it, these decisions and their influence on the final models are costly to assess due to the high computational resources required to train such models. With WIMBD, we hope to case the decisions that go into crafting large-scale datasets by surfacing patterns and trends about what goes into them and what is left out from different aspects, such as data quality, community and society measurements, etc. Once decisions upon what data is important, and which should be left out of dataset, practitioners that adhere to such decisions. The curation of the Dolma dataset (Soldaini et al. (0224) that happened while developing this work benefited from iterations over the insights from this work, such as the finding of "noisy" most-common vi-grams, and bugs in the initial 'de-duplication' implementation.
filter documents or passages
Data Documentation Adding to previous works that call for more data documentation, such as Datasheets (Gebru et al., (021) and Data Statements (McMillan-Major et al. (023). we argue for the importance of documenting such information. While previous works often focused and tailored the documentation for supervised-style datasets (c.g.. "Is there a label or target associated with each instance?", "How was the data associated with each instance acquired?" from Datasheets, and "What are the demographic characteristics of the annotators and annotation guideline developers?" from Data Statements) we call for more tailored documentation of large-scale pretraining corpora This work offers a superset of the automatic full-corpus analyses proposed by Dodge et al. (2021): Gao eet al. (2020), with several additions, categorization, and programmatic interface, allowing better understanding of the content of current and future large text corpora.
Grounding Models to their Training Data Unlike other factors of language model training, such as model architecture or optimizer choice, training data comes in the same natural language format as language model's outputs and thus can be measured and described in all the same ways. As such, the data offers unique opportunity for grounding models. For instance, model's 's ability to recall factual knowledge is derived from its training data (Jiang et al.) |2200] Elazar et al., (02 On the other hand, models often perform better frequent occurrences (Razeghi al.
2022a McCoy 2023). The path
eet al. 2023), and on documents similar to models' training data (Longpre et al.
to holistic comprehension of model behavior is through the data, which requires infrastructure investment to access big datasets and the right abstraction of data attributes.
6 CONCLUSION
In this work, propose WIMBD, framework for processing and analyzing large text corpora. Using WIMBD, we study ten different corpora that were used to train language models (or vision and language models, such as Stable Diffusion). We uncover interesting insights about these corpora using sixteen different analyses across four aspects: high-level statistics, data quality, community- and society- relevant measurements, and cross-data analysis. For instance, the most common source of texts for the LAION-2B-en dataset are the commercial websites Pinterest, Shopify, SlidePlayer, Amazon, and eBay. Regarding data quality, we find that about 50% of RedPajama and LAION-2B- en's documents are duplicates. In addition, we find that many evaluation benchmarks, including several from GLUE and SuperGLUE, such WSC, WIC, and RTE, are contaminated due their appearance in corpora such as RedPajama. Besides the analyses, WIMBD offers an extendable platform for reproducing our analyses on other corpora, developing new ones, and answering research questions about data. We release all the code and artifacts for WIMBD to encourage researchers to adopt and extend our framework and analyze existing and new corpora.
Many questions are still relevant for large pretraining corpora (c.g., "What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?").
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We want to thank Ludwig Schmidt, Maarten Sap, and Emma Strubell, and the anonymous reviewers for discussions and feedback on this paper, Elizabeth Salesky for the help with Unicode rendering and getting excited about obscure Unicode characters with me, and Carissa Schoenick, Jon Borchardt, and Johann Dahm for assisting with visuals.
REFERENCES
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. Towards a cleaner document- oriented multilingual crawled corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 133444335, Marseille, France, June 2022. European Language 1.463
Resources Association. URL
Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing knowledge in language models back to the training data, In Findings of the Association for Computational Linguisties: EMNLP 2022, pp. 2429-2446, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational I Linguistics. doi: 10. 18653vv/1
ho02.inndings-cmnl..115
BtRLhhtps:/// htttts:://aclanthooogg.oooo 202..
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't 't reach for the stars! arXiv preprint arXiv:2301.039881 2023. URL
2331803988
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang. Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 93-1944--0 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ 20//653//1/2022.2 acl-2emo..cl 9. URL www.oooo...rgg22...aadeeee
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615-3620 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/w1/DD19-1337
URL htt.thss//acccaatooooo.ooooo wtwpaaathhooooooo//1111133333
Emily M. Bender and Batya Friedman. Data statements for natural language processing: To- ward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604, 2018 doi: 10.1162/aalaa URL https
//acllantholooyyyor///111-111
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile, 2022. URL https:// arxiv.org/abs/2201..0
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430 PMLR, 2023. /
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leaby, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-203B An open-source autoregressive language model. In Proceedings of BigScience Episode #5- Workshop Challenges & Perspectives in Creating Large Language Models, pp. 95-136, virtual+ virtual+Dublin.
10

---

Published as a conference paper at ICLR 2024
May 2022. Association for Computational Linguistics. doi: 10. 10.18633///2022bbgcciiecc 1.9. URL hrrp:/aannnnoooooogg//.........
Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Commun. ACM, 13(7): 422 222-426,J jul 1970. ISSN 0001-0782. URL httphhttt///oooooo 10. 1453362686.366665
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Ex- tracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). pp. 2633-2650 2650. USENIX Association, August 2021. ISBN 978-1-939133-24-
3.
extracting
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. The Eleventh International Conference on Learning Representations, 2023. URL
TatRHT_ 1ck
Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, and Noah Constant. Unimax:
Fairer and more effective language sampling for large-scale multilingual pretraining. In The Eleventh International Conference on Learning Representations, 2023. URL
Jeffrey Dean and Sanjay Ghemawat. Mapreduce: Simplified data processing on large clusters. Commun. ACM, 51(():107-113, jan 2008. URL http:::///ioororr wttp:/i...r//0011/11111 1ttg//0..115//332772222323
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 10. 10.18653/v1/2211 .emmlp- main.98. URL RRRLLttts///acaaantoooocc org ww2.oo/gg/22.2.... wtt.://.ww..222......mmm.......
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:3012-1031, 2021a. URL
https://aclanthoo.gg. org/ 2021. t.61
Yanai Elazar, Hongming Zhang, Yoav Goldberg. and Dan Roth. Back to square Arti- fact detection, training and commonsense disentanglement in the Winograd schema. In Pro- ceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 10486-10500, Online and Punta Cana, Dominican Republic, November 2021b. Associ- ation for Computational Linguistics. doi: 10.18653/w1/0021 1201.:/11/22...mmmpmmm.. emnlp-main.88 URL https //aclanthology.oou 202 w.emnp-mmain.B1
Ali Emami, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. An analysis of dataset overlap on Winograd-style tasks. Proceedings of the 28th International Conference on Com- patational Linguistics, pp. 5855-5865, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational I Linguistics. doi: 10. 1663//1122020.ccoin-mmmii https://aclaanhoooog.or/////// wwwp::000....iioiaam.......
URL
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy, The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020. URL
https://arxiv.or//ab////
.00277
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Commun. ACM, 64(12):86-99 92. nov 2021. ISSN 0001-0782, doi: 10. 1144/345558723. URI. httphhttt////ioorrgg..... 0tt.:/..000011111///355577777771 1145/3458723.
Aaron Gokaslan and Vanya Cohen.
Openwebtext corpus, 2019.
URL https:/ ///

---

Published as a conference paper at ICLR 2024
Google. Know your data, 2021. URL httpt:/s/ggttttitoco//////rrrrr/
Google, Palm technical report. arXiv preprint arXiv:2305 2023. URL /Rthtttps://arx//wor
abs/2305. 10403
Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs. arXiv preprint 0rXii:2303308114, 2023. URL htt.t:t////axi//rrrrg///////////
2303.08114
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5075-5084, Singapore. December 2023, Association for Computational Linguistics. Lttguisiiii..////tp//////aaaattn http://:///.claataoooo.....oo/// org.g//ooooooggoooooooo 2023. wttp://rg./2.....mmamaaaa.a...
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020. doi: 10. 162/tacl a_ 0000244 URL htt.httss/// httppss//acclaantoooooooo//22222 tac1-1128
Clay Johnson III. Us office of management and budget memorandum m-07-16, 2007. URL https //georgewbush-e tehouse
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert
McHardy. Challenges and applicationns of large language models. arXiv preprint arXiv2330.10016, 2023. https://arri..org/ab////////////
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint XXXv:2001.08361, 2020. UtL/https:/ /ttp:i/.r//xxxxoo///oo//// www.rr///////////e .83661
Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia L.I, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. The stack: 3 TB of permissively licensed source code. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL https://openrevieewnet/
fForu?id==pppTduEppl
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 8424-8445, Dublin, Ireland, May 2022, Association for Computational Linguistics. doi: 10. (0.118663/1122222acccoo 1ong.577
httpttpps//cccaaatoooooooorr/r// http://olooooorgg/22......./ 2022.ac11
Hector J. Levesque, Ernest Davis, and Leora Morgenstemn. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR' 12. pp. 552-561. AAAI Press, 2012. ISBN 9781577355601. URL https
3031 0303333183333333 909
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Majje Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 185, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. URL //aclanthology.oorg .rgg 2021 -ew.-ee-neemeeee
Raymond Li, Loubna Ben Allal, Yangtian Zi. Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki. Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 2023. URL httptts://aaxxx..////aa//// /
12

---

Published as a conference paper at ICLR 2024
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 4969-4983, Online, July 2020. Association for Computational Linguistics.. doi: 10. wwwp://ww/222..lmmaaaaaa... 2020.acl-main.44.
URL htttp::/www.aclwe..or///aaaa
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint URL http://////rrxx..rg///////////// ab0..rr.ggaab/22221333135
1.Xiv:2300513169 2023.
Sasha Luccioni, Yacine Jernite, and Margaret Mitchell. Data measurements tool, 2021. URL http:://hhugggggga...coo//////// http:////ggg.baeeeeeereeeeeentse
Mare Marone and Benjamin Van Durme, Data portraits: Recording foundation model training data. arXiv preprint arXhv:2303.0391,, 2023. httpl//t////xxiior//a/aa////////
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309. 16338, 2023. URL httpp:///axxiv.orrr httpg//g//////////////
Angelina McMillan Major, Emily M. Bender, and Batya Friedman. Data statements: From technical concept to community practice. ACM J. Responsib. Comput., may 2023. doi: 10.1145/3594777.
URL /TKK//ttpp.//0011o//////11117733
Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Nazneen Ozoani, Ezinwanne Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela. Measuring data. In arXiv, 2023. htt.hhttsss/axxi.orr///aaa 221.b//12222.099
Anthony Moi and Nicolas Patry. HuggingFace's Tokenizers, April 2023. URL husgingfacce/tokeiizers
com/
OpenAL. Gpt-4 technical report. arXiv preprint arXiv: 2023. URL https://aa//////rrr///// abs/2303.08774
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. Data and its (dis)contents: A survey of dataset development and use in machine learning re- search. In Patterns, 2021. URL
52666389921001188
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116 2023. /BLL/tttps:///xxxxor/////////
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, and Anna Rogers. The ROOTS search tool: Data transparency for LLMs. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 3: System Demonstrations), pp. 304-314, Toronto, Canada, July 2023a. Association for Computational Linguisties. doi: 16533/v/2022.ac//
URL
Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hailey Schoelkopf. Stella Biderman, Martin Potthast, and Jimmy Lin. GAIA search: Hugging face and pyserini interoperability for NLP training data exploration. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 388-598, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023 acl-dcmo.5511 URL
Mohammad Taher Pilehvar and Jose Camacho-Colladee WiC: the word-in-contedd dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1267-1273, Minneapolis, Minnesota, June
13

---

Published as a conference paper at ICLR 2024
2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N199 1128. URL https Taaclanthology.orr/N553 1128
Giada Pistilli, Carlos Muñoz Ferrandis, Yacine Jemitte, and Margaret Mitchell. Stronger together: On the articulation of ethical charters, legal tools, and technical documentation in ml. Proceedings of the 2023 ACM Conference on Fairness, Accouniability, and Transparency, FAccT '23, pp. 343-354, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701924, doi: (0.1145//593301333590000 /tLL1ttto/////oooo/////////////3
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog post, 2019. URL /ttl/ttts////peaiicco
researct/bbetter--aangg------eee
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http://jhml..or/ppppers//1121111 229.4h2222200
Yasaman Razeghi, Robert Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings the Association for Computational Linguistics: EMNLP 2022, pp. 840-854, Abu Dhabi, United Arab Emirates, December 2022a.
Association for Computational Linguistics. htkLinttpss//alllattooooo.... wttp://oww..rrr//.........
em
Yasaman Razeghi, Raja Sekhar Reddy Mekala, Robert I. Logan Iv, Matt Gardner, and Sameer Singh. Snoopy: An online interface for exploring the effect of pretraining term frequencies on few-shot LM performance. In Wanxiang Che and Ekaterina Shutova (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 389-395, Abu Dhabi, UAE December 2022b. Association for Computational Linguistics. URL https: http:://llaaaaaooo...o.oooo wwwpthhoooggo.rrr22//eeeeeem 10m.222emmmdddmmm...333- demos.39
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pp. 90-95, 2011. http//////aaaacrr/e/e//////eeeea
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Oscar Sainz, Jon Ander Campos, Iker Garcfa-Ferrero, Julen Etxaniz, and Eneko Agirre Did chatgpt cheat on your test?, Jun 2023. URL httthss//htttt-zettrroaet wtt.aittimmmmmmmommttiiinanaaaaa
blog/
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, aug 2021. URL
https://doi.org// htt.://...rr//11.11/333333 3474388
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "everyone wants to do the model work, not the data work": Data cascades in high-stakes ai.
In Proceedings of the 202/ CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 97819993155 450380966. doi: 10.1144/341 1764.344518.
URL//ttss: /ttp:p/////.o...rr//.1114//////1 01011..1114//3117444445555
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019. URL
Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana Ili's lli'c, Daniel Hesslow, Roman Castagn's e, Alexandra Sasha Luccioni, Franccois Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Rose Biderman, Albert Webson, Pawan Sasanka Ammanamanchi,
14

---

Published as a conference paper at ICLR 2024
Thomas Wang. Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurenccon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa Etxabe, Al- ham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris C. Emezue, Christopher Klamm, Colin Leong, Daniel Alexander van Strien, David Ifce oluwa Adelani, Dragomir R. Radev, Eduardo Gonz' alez Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady ElSahar, Hamza Benyamina, Hieu Trung Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Josephine L. Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro von Werra, Leon Weber, I Long Phan, Loubna Ben Allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz. Maraim Masoud, Mar ia Grandury, Mario vSavsko, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang. Minh Chien Vu, Mohammad Ali Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priseilla A. Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto L' L'opez, Rui Ribeiro, Salomey Osei, Sampo Pyysalo. Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, S. Longpre, Somaich Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala. Violette Lepercg, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea San- tilli, Antoine Chaffin, Arnad Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani. Matteo Manica, Nihal V. Nayak, Ryan Teehan, Samuel Albanie. Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Févry, Trishala Neeraj. Urmish Thakker, Vikas Raunak, Xiang Tang, Zheng Xin Yong. Zhiqing Sun, Shaked Brody, Y Uri, Hadar Tojarich, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang. Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette. Pierre Franccois Lavall' ce, Rémi Lacroix. Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aur' elie N' ev' eol, Charles Lovering, Daniel H Garrette, Deepak R. Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Xiangru Tang, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuar, Miruna Clinciu, Najoung Kim, Newton Cheng. Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, S. Osher Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdenvek Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ananda Santa Rosa Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Olusola Ajibade, Bharat Kumar Saxena, Carlos Muñoz Ferrandis, Danish Contractor, David M. Lansky, Davis David, Douwe Kiela, Duong Anh Nguyen, Edward Tan, Emily Baylor, Ezinwanne Ozoani, Fatim T Mirza, Frankline Ononiwu, Habib Rezanejad. H.A. Jones, In- drani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jan Passmore, Joshua Seltzer, Julio Bonis Sanz, Karen Fort, Lívia Macedo Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, M. K. K. Ghauri, Mykola Burynok, Nañi Abrar, Nazneen Rajani, Nour Elkott, Nourhan Fahmy, Olanrewaju Samuel, Ran An, R. P. Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas L. Wang, Sourav Roy, Sylvain Viguier, Thanh-Cong Le, Tobi Oyebade, Trieu Nguyen Hai Le. Yoyo Yang, Zachary Kyle Nguyen, Abhinav Ramesh Kashyap. Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Fsscalada, Ayush Kumar Singh, Benjamin Beilharz, Bo Wang, Caio Matheus Fonseca de Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel Le' on Perin' an, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully A. Burns, Helena U. Vrabec, Iman LBB Bello, Isha Dash, Ji Soo Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthi Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pamies, María Andrea
15

---

Published as a conference paper at ICLR 2024
Castillo, Marianna Nezhurina, Mario Sanger, Matthias Samwald, Michael Cullan, Michael Wein- berg, M Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patricia Haller, R. Chandrasekhar, R. Fisenberg. Robert Martin, Rodrigo I.. Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Pratap Bharati, T. A. Laud, Th co Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yashasv Bajaj. Y. Venkatraman, Yifan Xu, Ying Xu, Yun chao Xu, Zhee Xao Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. ArXiv, abs/2211 1,0100, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk. and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL http:////ppeerrreeewnnettttfffrr
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. Commun. ACM, 63(12): 64-63, nov 2020. ISSN 0001-0782. URL. htttpp:://doi..rg/11... 1145/331311
Preethi Seshadri, Sameer Singh, and Yanai Elazar. The bias amplification paradox in text-to-image generation. arXiv preprint wwXiv:2308.00755, 2023. /tL/
Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN) pp. 1-8, 2022. URL WRLLhhtps:://// htttps://ieeexplrre wee.lorrreieeeooooo weee. /
ddcument/9891914
Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Woo Ha, and Nako Sung. On the effect of pretraining corpora on in-context learning by large-scale language model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5168 5186, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 www.aaaaaacnnm.mmmm URL https 2ww.. -ma..aacllmmmim8800
Daniel Simig, Tianlu Wang, Verna Dankers, Peter Henderson, Khuyagbaatar Batsuren, Dieuwke Hupkes, and Mona Diab. Text characterization toolkit (TCT). In Proceedings of the 2nd Con- ference of the Asiu-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joinl Conference on Natural Language Processing: System Demonstrations, pp. 72-87, Taipei, Taiwan, November 2022. Association for Computational Linguisties. URL httpp::/aaclanthologgy.. wttp://.or//22....adeee...
Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently on S2ORC) Dataset. Technical report, Allen Institute for AI, 2023. ODC-By, https::/ggithub.com/////
Luca Soldaini, Rodney Kinney. Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Flazar, Valentin Hofmann, A. Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacoh Daniel Morrison, Niklas Muennighoff. Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hanna Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.001 02244000.99,,,. 2024. URL http::::///axx..og///aaa faabs/2402.00 2402.00152
Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. Detecting personal information in training corpora: an analysis. In Proceedings of the 3rd Workshop on Trust- worthy Natural Language Processing (TrustNLP 2023), pp. 208 220, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 1/2033 http//200.....ulp--111o trustnlp-1.11 1.18. URL
ottpss//accaattoooooooor/////sss
16

---

Published as a conference paper at ICLR 2024
MosaicML NLP Team. Introducing mpt-7b: A standard for open- commercially usable Ilms, 2023. URL www.mosaicml.commm https/sa..mm..cc/oooooo///////// Accessed: 2023-05-05.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S Morcos. D4: Improving Ilm
pretraining via document de-duplication and diversification. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Together Computer. RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset, April 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint 2337 2023. URL httpss//arrkivoor/////2222111//
Unicode. Unicode Text Segmentation, Aug 2023. URL http:://uuntccoe.oor/rrprtt/////
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' J'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curra Associates, Inc., 2019. URL
wwtp::eeerrrisccccpp.eee feteerfffleeepppeeee //// Paper .pdf
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model. attps://ght http://ooo///iigiooooooo///i/rre
jax May 2021.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.188533v1/2055 //aclanthology.org /tter///////22/22/ naacl-main
https/
Rui Zhang and Joel Tetreault. This email could save your life: Introducing the task of email subject line generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 446-456, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18663/11/119-004 URL URLLittps///ecc.etttooooo.yo.o otrpaaaaaantttoooorr/11/11----
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3153-3155, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v 1/252//////221.11 leacl-
m7...274. URL / htt.:t/ps//accattooooooo//////// 2aa../2222....--m.......... main.274
17

---

Published as a conference paper at ICLR 2024
A CORPORA: ELABORATION
We cover ten different corpora, including text-only corpora (e.g., C4), captions from image-captioning (LAION-2B-en), and code (The Stack). high level description of these corpora using WIMBD presented in Table 2 and details about the information contained in those corpora are detailed in B
We analyze all corpora fully, including the different subsets (e.g., The Pile is constructed of multiple sources, such as Wikipedia, arXiv, etc.). The only exceptions are mC4, and LAION, which the original released data consist of non-English texts well, and we focus while we focus on English text corpora, most of our analyses are not language dependent, and can be easily applied to other languages well. The only exception is the toxic language analysis ($B.3.3) that relies on an English lexicon and classifier. However, we note that given non-English lexicon and classifier, the analysis can be easily repeated for other languages using our framework.
the English subset. Note that
OPEN WEXTEXT is open-source reproduction Gokaslan & Cohen 2019) of the data used to train GPT-2 (Radford et al. 2019). Due to the limited information provided by Radford et al. (2019), and never releasing the data, is unelear how similar WeeTet is to the original data (WebText), but similar steps to the paper's reports were conducted (such as deduplication, non-English filtering, min-length filtering, etc.).
C4 is the dataset used by Raffel al. 2020) for training T5. The dataset: The Colossal Clean Crawled Corpus (C4 in short) is based on Common Crawl as a source of text that was scraped from the web. As such, lot of the data is noisy, and set of heuristics were employed clean it up, such as filtering documents by length, obscene/bad words, duplicate texts, non-english, etc. C4 was not released by Raffel al. (2020), and instead, was scraped, cleaned, filtered, and released by Dodge itt al. (2021).
MC4-EN is multilingual version of C4 that was used to train mT5 (uue et al. (021), and later umT5 (Chung al. 2023). We use the latest version (v.3.1.0) which was used train umT5, containing documents collected from Common Crawl through August 2022, and in practice the portion of the data that is classified as English. The main difference of mC4-en over C4 is higher confidence by language classifier (from 0.7 to 0.96), while also allowing a 0. random set of documents that contain "bad words" to pass through, and adaptation of the "bad words" list that resulted in filtering more than 10% of the documents in language.
OSCAR is a multilingual corpus based on Common Crawl (Abadji et al. 2022). It contains length filter for improving data quality that filters out documents with short sentences. They also annotate the data with different labels, such as the language of the document, adult content, and language identification, which they use for different analyses. is an ongoing effort, and the corpus is maintained and updated regularly.
THE PILE is a corpus consisting of 22 different domains (Gao et al. 2020). Unlike C4, the data was not scrapped from the web and then filtered, but pre-selected, with the motivation that this way the data will be of higher quality. The included domains in The Pile are diverse: they include data such as Wikipedia, Github, Arxiv, EuroParl, and more. By design, most datasets are upsampled in the hope to increase data quality, from 1.5x with domains such OpenSubtitles, up to 3x with
Wikipedia. Models such as GPT-J (Wang & Komatsuzaki 2021), GPT--eo (Black et al. 2022) and Pythia (Biderman et al. 2023) were trained this dataset.
REDPAJAMA is open-source version reproduction of the data used to train LLaMA TTuvron et al. 2023). and was used to train RedPajama-INNrrre (Together Computer) 2023).
S20RC is a large corpus of English academic papers, which consists the abstracts, full text, including figures, tables, and references (Lo et al. 2020). The texts are automatically extracted from pdfs and LATEX sources.
18

---

Published as a conference paper at ICLR 2024
PES20 is a derivative of S2ORC, cleaned and filtered to obtain more usable version of the data intended to train language models. We use peS2o V2 (Soldaini & Lo (023).
LAION is large dataset of images and captions scraped from Common Crawl (Schuhmann et al. 2022). The main dataset (LAION-5B) contains 5.8 billion examples, of which 2.32 billion of the captions are English (LAION-2B-en)) which we use in this work. We focus on the text captions but demonstrate qualitative examples using the associated URLs and images when appropriate.
THE STACK (Koetkov et al. 2023) is source-code dataset that was collected for training language models, and parts of it were used to train SantaCoder (Allal al. 2023) and MPT (Team 2023). was compiled from GHArchiva with some filters: files that cannot contribute to training code such binary files, files larger than IMB, and some extensions. In addition, only repositories with permissive licenses were included (18 license types in the version vl.0, and 193 in version vl.1). and the v1.2. While the main purpose of code is provide machine instructions to perform different functionalities, it also contain natural language in the form of comments: "Roughly 40 natural languages are present in docstrings and comments with English being the most prevalent. In python files, it makes up 96% of the dataset."
Table 6: Metadata information contained in the ten corpora we consider. Text refers to the main information contained in those datasets, while the type of text is different, e.g. The Stack contains source code, and LAION2B-en descibes images. URL indicates the URL that the document LAION2B-en. link
image that the text refers to. Scrape Date collected from, in the case that the document was scraped from the web, Date Added is the date the data was incorporated into the corpora. Domain/Lang indicates a subcategory of the text (c.g. field of study, the source from The Pile, code language in The Stack). ID is the document ID. Has Split signifies whether not the released data contains train-test split.
Corpus
Text
Url
Scrape Date
Date Added
Domain/Lang
ID
Has Split
OpenWebText
x
C4
mC4-en
OSCAR
V
X
/
The Pile
X
X
X
X
RedPajama
J
V
S2ORC
X
PeS2o
LAION-2B-en
The Stack
X
httpps://gharchive.orr//
19

---

Published as a conference paper at ICLR 2024
Corpus
1
50
75
99
N.
C4
26
264
964
3,886
137,117
15,668,300
OSCAR
21
303
1,351
6,108
440,577
15,424,393
LAION-2B-en
1
11
25
892
1,470,243
mC4-en
48
580
1,448
5,984
477,951
62,209,45 ,454
RedPajama
26
264
963
3,882
136,937
15,658,463
Table Internet domain quantiles each corpora URL information. values correspond to the number tokens from each internet domain quantile. N. corresponds to the number of unique internet domains.
B ADDITIONAL RESULTS
We provide additional details and extended results on all the corpora considered in this work. This appendix is structured in a similar way to the structure the main paper, categorized by the four different high-level analyses: (1) Data Statisties (Appendix [.II. (2) Data Quality (Appendix (.2)). (3) Community- and Society- -eleevtt Measurements (Appendix B.3 and (4) Cross-Data Analysis (Appendix B.4
B.1 DATA STATISTICS
The summary statistics are composed of different analyses that mainly involve the additional metadata associated with the textual documents, such as the URL from which the document was extracted, the date it was collected, etc. We also consider some statistics about the corpora, described in the main paper (4.2). The analyses we propose for data statistics are the following:
1. Summary statistics ($4 ($42
2. Internet domain distribution 4.2.2
3. Internet domain schemes ($B B.1.2
1.2
4. Internet domain suffixes ( 1.3
5. Utterance date statistics B.I.4p
6. Geolocation (BI.5
7. Language distribution (BB.1.6)
B.1.1 INTERNET DOMAIN DISTRIBUTION
Here, we provide complete analyses on the five corpora that contain URL information in the corpus metadata. Using the Exact Counts we conduct two analyses: (1) each domain counted per document (yielding documents per domain), and another where each domain is counted per token in the document (yielding tokens per domain). The results are presented Figure [] where the (1) document per domain figures presented the left, and the (2) document per token figures are presented on the right.
In Table 7 analyze the number of tokens in each domain, and calculate the 1, 25, 50, 75, and 99 quantiles of these distributions. Interestingly, the 1% quantile in LAION-2B-en include domains which have 1-or-less tokens.
B.1.2 INTERNET DOMAIN SCHEMES
This analysis computes the domain schemes of the associated URLs using the Exact Counts The results are presented in Figure 7 HTTP and HTTPS are two internet protocols, with the latter being an extension of the first that provides more secure communication. While the exact portion of websites across the web that uses each protocol is hard to assess, traffic that goes through Google primarily uses HTTPS 95%
September 16th, 2023.
20

---

Published as a conference paper at ICLR 2024
The trend of recent years shows increase in the portion of HTTPS-supported websites, and as such, we can use this portion as a proxy for the internet age of a website: HTTP websites are more likely to be older. In addition, the portion of corpus is an interesting comparison with the reported portion from Google's traffic.
All corpora containing URL information show significant proportions from Google's reports of 95% for the HTTPS protocol. OSCAR contains the highest proportion with 87.6% HTTPS URLs, while C4 is the lowest with only 62.5%.
B.1.3 INTERNET DOMAIN SUFFIXES
Next, we compute the suffix distribution of the different corpora using the Exact Counts and present the results of the ten most common ones in Figure Compared to the internet domain distribution, the suffixes provide us with higher-level description of the sources of the documents.
Perhaps surprisingly, the most common suffix is com, which is between 60. 1% of the documents in OSCAR and 77.5% LAION-2B-en. The distribution of suffixes for each dataset exhibits a long tail with a total of over 3,000 different suffixes the different corpora. While the top 10 typically represent suffixes from English-speaking countries (c.g., co.uk, and ca), LAION-2B-en 's top- 10 contains a lot of non-English speaking countries as well, such as Germany (de, 0.7%), Russia (ru, 0.5%), France (fr, 0.4%) and Italy (ir, 0.4%).
B.1.4 UTTERANCE DATE STATISTICS
In this section, we examine the temporal diversity of documents from corpora with either reliable creation timestamps in their metadata or URL source information from which creation time can be estimated. Language usage drifts, new concepts are introduced over time, and the truth of much commonsense knowledge depends on the date an utterance was made. While some datasets we consider (S2ORC and peS2o) have reliable, A APl-generated creation timestamps, most have creation dates that reflect the time of document ingestion into the source dataset and not its origin date (C4, mC4-en, RedPajama, and (AION-2B-en) To characterize their temporal distribution, we directly count and bin documents by year for those with reliable creation time metadata. For datasets without this information, we fall back on using either the earliest date the URL associated with a document was indexed by the Internet Archive the date of ingestion into the dataset (whichever is earlier) IT] Note that such a procedure does not provide us with the timestamp of the document that was scraped, and as such, it serves as lower bound on the document's time creation. Given the limitations of the Internet Archive's API, we do this for a 10,000 document random sample of each dataset, which allows rough estimate of the collection time for documents these corpora. Results are shown in Figure 9 We can see that RedPajama and OSCAR are dominated by documents created in the previous five years (as of September 2023), while other datasets have a more substantial proportion of documents from the first half of the 2010s and earlier. Notably, S2ORC and pes2o contain non-negligible fraction of documents from the pre-internet era.
B.5 GEOLOCATION
In this section, we gauge the geographic diversity of corpora with URL source information in their metadata. We use a commercially developed database [2] to estimate the country of origin for 100,000 randomly sampled URLs from each of the five corpora with this information included. While there are limitations to using the location of a hosting server as a stand-in for the content creator's s location (i.e., websites are not always hosted locally nor in one unique location), it does provide rough geographic origin for source material. As seen in Figure 10 most web pages across corpora hosted in the United States, with the bulk of the remainder distributed amongst the anglosphere. This is unsurprising given the focus on under consideration.
sources in the construction of the corpora
21

---

Published as a conference paper at ICLR 2024
Table 8: Percentage of documents in English per dataset.
Corpus
Percentage
OpenWebText
99.68
C4
99.67
mC4-en
99.56
OSCAR
99.92
The Pile
96.12
RedPajama
96.93
S2ORC
96.44
peS2o
100.00
LAION-2B-en
95.90
B.1.6 LANGUAGE DISTRIBUTION
Here, aim assess the proportion of languages in all corpora. We the CLD classifier to make a prediction about what language is being used in each document, and use this prediction as label that we analyze in aggregate. Note that we use the classifier label also in mixed- lannuu-a documents (if CLD2's is reliable flag is False, we apply the label UN). Table repporss the percentages of English-language documents across corpora. As expected, the English fraction is quite high, given the targeted construction of most datasets we consider. The remaining percentages of non-English documents are broken down for the ten remaining most common languages in Figure [] Note that the classifier we use, as with other classifiers, is imperfect, and as such the identified languages may be wrong.
1heee Internet Archive is massive library that has been preserving the web since 1996. https
//archive.org
This work includes Io2Lici LITE data available from https:/litteippllcctii..cmm
22

---

Published as a conference paper at ICLR 2024
Demam Dennent
GeDenemapet
netees Deniin Den ror
irilis DeraampeeTooon
www.oe cleay.cor
945
1.00
Documera
E.02 Sc Dacumaria
0.5
0.21
6.0
RedPajuno Dovaine per Docuree:
RedPajana DenclasperTien
selooeefewccc
waggaseegagg
LACO-D-- DenriaaeeeeeaeeeeeDerreeee
LADN22 Danaina Tala
lamy/nacee
Iregis www.sddeeeeowoooommmmmmm
Image eaape-carriiggeeeaaar.o
Sunba
Raacce
media pay.eaaooooo
Figure 6: Internet domain distributions of the ten most common domains for each corpus.
23

---

Published as a conference paper at ICLR 2024
Rohene
Behoro
Rehuma
Rediaara Cromes
LAKONGG
Schese
Schere
Figure 7: Schema distributions the common domains for each corpus. We show the results for the five corpora that contain URL information.
58
Figure 8: Suffix distributions of the domains for each corpus. We show the results for the five corpora that contain URL information.
24

---

Published as a conference paper at ICLR 2024
80
2023
% of Documens
2022
2021
60
2020
40
pro
20
mC4-en-
OSCARt
RedPalama
S20Rc
ne52o
LAIoN-2B-en-
Figure 9: Fraction documents in each corpus produced per year. Corpora marked with are estimates based on the Internet Archive index dates for 10,000 document sample.
100
SolDaammnts
FR
mDdeen
ascaet
ReePaiana
LeKa 2 kn
mcieen
oscup
RedPgema
LACHM-B4n em
Percentage URLs by country
(b) Percentage of URLs (excluding unresolved URLS)
Figure 10: Percentage of documents for each dataset originating in given country. Only the nine most common countries across corpora are shown with the remainder combined in 'other.' We label URLs we were unable to geolocate UN (Unknown), and provide results with and without these documents included.
3.5
2.5
Docunerts
Documents
2.0
2.0
1.5
0.6
0.0
OpenWebTedd Tead
C4
mc444n
OBCAR
The Pile
RedPaanit
S20RC
PeSto
LAION-2B-en
0.
OpanViebTon Fast
cd
mn
OSCAR
Pile
RedPajama
SooRc
PeSto
LAIoN-2--an un
(a) Non-English language content
(b) Non-English language content excluding unknown languages
Figure 11: Percentage of non-English language documents detected in each corpus.
25

---

Published as a conference paper at ICLR 2024
Table 9: Most common unigrams, bigrams and trigrams and their estimated counts.
LAMIN
1:3
5289
5289
1.50
1230
1008
120
4240
0818
4.39
1748
4503
4330
305
2604
4240
2181
20388
4.03
01104
15
-200
15:24
4.00
7104
1000
20468
+1.0
13354
19004
12044
0.242
100
01750
0.550
4330
1.800
17100
$1A1
B.2 DATA QUALITY
While we reported all the different analyses under data quality in the main paper, here we elaborate and provide the full results on all corpora and the different variations (e.g., most common unigrams, bigrams, and length distribution on token level). The analyses we propose for data quality are the following:
1. Most and least common n n-grams 4.3.1
B.2.1
2. Duplicate 4.3.2 $B.2.2)
3. Document length distribution 4.3.
B.2.3
B.2.1 MOST & LEAST COMMON R-GRAMS
Most common n-grams In addition to the most common 10-grams reported in Section 4.3.1
report the results for the most common unigrams, bigrams, and trigrams. Stop words and punctuation the most common unigrams across the different datasets, with some differences in their ranking. Moving to bigrams, we observe more differences between the corpora. For instance, in LAION-2B-en, we observe some marketing mentions, such as "for sale" and r- Shirt", "of the" and "in t the" are repeating bigrams in all corpora. In the trigram results, we notice a larger diversion between the corpora. C4 contains common English expressions, such as "one of the", "a lot of". and "as well as". However, LAION-2B-en contains much more marketing material, such as "T Shirt" "for sale in". OSCAR and The Pile have many n-grams that look like uncleaned html (": /", 'https : "","type "") or markdown ("--"," "===", "##*").
Least common n-grams Similarly to the most common 7-grams, we look at the other side of n-grams distribution on the least common in corpus. We showcase a random set of 25 unique unigrams from the different corpora in Figures 12 [3] We observe two noticeable trends from such unigrams: (1) non-standard Unicode fonts like "negative squared latin" (for instance COTD in mC4-en), and (2) non- -on-lnglish strings. Non-Englieg strings are quite diverse. The sample from Open Webxet contains unigrams from 12 languages other than English: Urdu, Arabic, Korean, Sanskrit, Hebrew, Armenian, Bengali, Persian, Japanese, Latvian, Sindhi, and Russian.
In addition to the unique unigrams inspection, we estimate the number of unique unigrams in each corpus and present the results in Table [0] The unique unigrams results reveal that a non con-trivial amount of unique unigrams appear in these corpora. Even the smallest corpus, Web contains more than 88 million unique unigrams, about 1.1% of the total unigrams in this corpus. The ratio of unique unigrams is about an order of magnitude smaller in the other corpora, except for LAION-2B-en, with over 554 million unique unigrams, which constitute 1.8% of the total unigrams.
26

---

Published as a conference paper at ICLR 2024
Table 10: Estimated unique unigrams, and their percentage of the total unigrams.
Corpus
Count
Percentage
WebTeetText
88.551,499
1.1
C4
759.392,762
0,5
mC4-en
4,290.392,741
0.2
OSCAR
1,280,686,44
0.3
The Pile
1,809,24 2496
0.6
RedPajama
2,530,085,09
0,2
S2ORC
287, 196,445
0,5
PeS2o
201.729,350
0.5
LAION-2B-en
554,850,812
1.9
The Stack
4,294,966,820
0.3
27

---

Published as a conference paper at ICLR 2024
HYO
7
@0|0|77
Gäzān
2
0
وسااة
m
omptue
+7 r /
(a) OpenWebText
@aaaes
favoured
Accel erated RECNA
nights
BUSINESS
BopRk
added
ICS
s| Particlaryy
G john
IIGIO
_
arrived,tt
to
#|
(b) C4
nornanconics
#d.
BREED
Tomie
Forbearance
pepper
2
3980
C0TD
CAi
N
dt's
CHIGANA
y'all's
HIPSTERS
Hostens
coke
B RRDIS
S
Veggle
(c) mC4-en
3pm
Sunohara
Candy
3|
E| 2)
3/22
corniculotus
sccuagl!
μ0H
Leo
1975
deff'
#EOMG
Franklin
#
$ Atol
SPEA
ränsi
(d) OSCAR
011
Bima
NonoUe
0
REM
P
67
?
Traurig
R'
0/19
SPV235
2
I
22/
:20
(e) The Pile
Figure 12: Unique unigrams in Open WebTen, C4, mC4-en, OSCAR, and The Pile.
28

---

Published as a conference paper at ICLR 2024
ho
6
#
4
Wherever
Ciphergen
-
4
NOTTY
2
fageffazione
$21B
(a) RedPajama
4.22
* A. [
ávcrooui
Ju
=
0
سارن
4plale
Yj
microelectrc
2
P
2
(b) S2ORC
flloviridae
érglis
Знаните//ННb/
негативни
OHcomponent
cIpina
bagt
Infipuplalllin
21X
парезиттbБ
орyтуриaм
Hussein
бауыиддйй
бчсттаиичскаа
MаHтиA
ممررة
Byra
(c) peS2o
ドッテェジリーースス HHammock
4
アインアイリシア
サイイグラス
クイ カインドカーストト
TEAR
リトルクラコ
Kennudy
$=//$Llo_DDancing
Pークインタビスー
BUNDLES
821
トップスナニニック
アレンズセツ A
パイ カーー
クレー7チ
gaaz
クーリンミマイタレイス ソ/バン
(d) LAION-2B-en
ANKNIU
Bernoulli
util
7ROVORK
rhetoric
Palona
#B
SACHIN
DOZEN
a/á
ótanc
TMAIKKNNRN
yy1970
F'
V197 Px' *
=
poppies
(e) The Stack
Figure 13: Unique unigrams in RedPojama, S2ORC, peS2o, LAION-2B and The Stack.
29

---

Published as a conference paper at ICLR 2024
Table 11: Top 5 most occurring text duplicates from datasets with duplicates (OpenWebText and C4 don't ton't have any duplicate documents). Truncation for visualization is marked by [..].
Corpus
Property
Duplicate
Deplicate
Dupliccte
Duplicate
Duplicate
Test
tigiodigg cnforne bricc bikcchrkkgparneeee goarddeaaooraal
Tads has the woeld's les ding snset porking toches
4K Uneecker pietur with eespiions pictut quality, lag and play, H H.basai 2n
"ercksllgnndeett coferne
markersne 'click' me CkeeChickk ditppeo & indeemmot murker crii. retiee. turgee marker Ltd 73
mC+eet
56411;7 11 (ff of allowdd
stugy and las many of the
5skal:;) ) 19 of by dhanwtt vari t5pe a SchMia -.]
vori a SC J
world's experts. bug [..)
54, Max 5120 SDD cantt -
Count
154
114
so
76
USCAR
In lagin you neghttned Registoring tskes only
Bevakeriggt diselted, cuuble before pent.. J
experiance, plase
Privsuy Cuskioe This coskice
Inooskeripe your
pu aininistaae
may
your
this you agree their seet...
experienas
boced whither you regiatr (t..]
pives you increen)..) J
be sort..l
The Pile
Test
Ii "saibee": "soskk"n | lo]
('n "anti" (te "weden".
IC-TTaak R-Filinghhaiiiee
1e Locallood weooons of In! ogliis lays "Maln
<heel vansinee 1 enco dinge LTTE- "TTe--"ssteett "StwetDO CTYPE plua PUBLIC -lAggpeltDT PLIST L.V IEN" "ungeff....
cewilzeslleeieeieeaaale tettoutttiaiieee Inavaald ewiezenteent --] 2,941
Count
8.755
2,013
2,744
2,214
RedFajaste aajaa
Test
ThlaThe Catalo Lif,, 3ed 201 Duluttte Published ieleniiiiattterr Oct...] 146,434
SYNONYMWiisiaa
ooding
ACCHrTEDilasiaa tet
ADCEMTHDeeluecd Aus
ulaletziin Regiass Marine
Plan Sumre Indestatuas
3ed 201 tuctesh Published inlanal Turutett Ocf... 92.922
fautonya Teltaaii Publi shed invadiiuluttet lgitel sunéaaalll..) 15.008
Publisted lnmmniiu
Generiulatiii Publisaed inlanulfal... 10,089
Originl 21322
Count
S2ORC
Text
Aburact not sahmineed I sr unlize publicsstetalllei
Alsiracs P1 - Pin see e dacational and nue incha del for publication unli
Almtnas wifhdroeetehlle wiu2022 Coneenicat anlii se vhhmiiin 1u2022 The mouhh peer review vil20 No space constrinns I..
Educaional abdraclviett Validaim of now sutson and valamesic breas d easht nessurmeut ayae in I-.]
Modkling and analyds of monkeypus discasc wingg fracional derivwiveeeiii he frequescy of mookeypo x1-.1
delub022 Research which is fieely available for rod issib....
neautiiute A PR
ESEN TI...
Count
35
30
26
14
14
Text
Educational alsimsfulaal Validation of volumetik syste
Raphy RCZaiaTTis escripts invesigates screpancy evilmated catf.
CGP27 climatc change edded and worddaWThe 2022 report 11...
Reply RCDutaaullinii sugaestion, luve revised the careuull.. The liss
Reply RClluaa pap estuary
astion
vadiblitt ovenide undes varyial...]
I..]
Count
14
LAION 28 20
Text
Frozt Covcer
Wall View 002
Market position of the clocsed tochnologies 414,996
Poiuswise: Reliable CFD
Go to Europzan Conni ssion ochsite 314.423
maaking
Count
1003.865
681,753
319.324
Suckk
#uluuRaaCruppllrr IoNa. lid movemmenta
Mhal Wechta, Aur.s............. all WechasAuhh SDKuo Creazed lebllllutll le 13-1 1:00 Cogyrght 2013k55774
OUTFUT FORMAT e*
Inteam"" "elf32 "etr52 unteorm""
Maat WBitnpRoqustt ihoToken WeboSDK Whhhl' Crested Donaton 14/11//m0 Cop
thail Wox 1624th Www.AAl
Walhll Created Wechot
inNTTYrieee hondle SYSEEACCH_DDE
2: 24 Copyrigte
yrihhl..)
2012u5e74 Tenocul.
Count
45
24
20
Table 12: Top 5 most occurring URL duplicates from datasets with URLs for each document and non-zero URL duplication.
LAION-2B-ca
OSCAR
Text
Count
Test
Count
UNLIKELY
2,154
/twpw//tttcccccceoeoeo gg// grddriver downlead Leoges/driver downlood wwwttrricattinss.or..
27.162
235
10,700
sper
100
aceho 1PR
10,144
L5p27//ta1...oo rogporocc tonteennn appr ouroel 301000000
9,935
100
B.2.2 DUPLICATES
URL Duplicates We also examine duplication between document URLs for the datasets that have that metadata, which we show the top-5 URL duplicates from datasets with URL duplicates in [2] LAION's most frequent URL (with 33,142 occurrences) is an invalid URL- - "UNLIKELY", likely resulting from a parsing error. The second most frequent URL (with 27.162 occurrences) from LAION-2B-en leads to an all-white image from computer driver website, and in Figure we see that among the top 25 duplicated URLs in LAION-2B-en, there are instances of image duplicates hosted at different URLs. Meanwhile, OSCAR has notable artifact wherein, after the top two duplicate URLs, the next 234 URLs are duplicated exactly 100 times. Table [4 in the Appendix shows counts and ratios for these URL duplicates as previously specified for text hashes. These find that URL duplicate ratios are roughly an order of magnitude smaller than their text hash counterparts, and that the count of documents duplicated by URL is not dominated by only few clusters.
30

---

Published as a conference paper at ICLR 2024
total
60
uniq of total
50
Duplicate %
40
205M
20
1.EM
10
13:3M
5176
mC4-en
OSCAR
The Pite
RedPajama
S20RC
peS2o
LAION-2B-en
The Stack
Figure 14: Pereentages of duplicates to totals for datasets with any. The percentages documents and percentages of unique document clusters are each shown as bars. Duplicate counts are presented above the bars.
Table 13:
Statistics about text duplicates per dataset. Counts of duplicate documents and ratio of duplicate total documents as well as equivalent counts for unique text clusters.
Corpus
Duplicates
Ratio of total
Unique duplicates
Uniq ratio of total
OpenWebTee C4
0.00
0.00
0
0.00
0
0.00
mC4-en
48,255
0.00
21,991
0.00
OSCAR
164.740,386
0.38
19,934,531
0.07
The Pile
138,716,558
0.66
64,623,82.
0,47
RedPajama
459,530,754
0.49
218,875,070
0.32
S2ORC
3,703,001
0.33
1,767,564
0.19
peS2o
33,903
0.00
16,924
0.00
LAION-2B-en 1,254,910,523
0.54
342,174,466
0,24
The Stack
517,396
0.00
232,151
0.00
B.2.3 DOCUMENT LENGTH DISTRIBUTION
We elaborate on the results from the main paper and report the length distribution for all corpora. both for the character and token distribution. Figure T6 showcases these distributions, and Table T5 depicts the median token and character length distributions.
LAION-2B-ert containing image alt text, has the smallest average document lengths. Beyond the exact duplicates described above, which commonly describe products (especially home appliances), LAION-2B- also contains significant number of aeneerttee alt texts paired with maps describing the location rental boats. The only outlier Open WebText in terms of document length
Table 14: Statistics about URL duplicates for datasets URLs for all documents. Counts duplicate documents and ratio of duplicate to total documents as well as equivalent counts for unique URL clusters.
Corpus
Duplicates
Ratio of total
Unique duplicates
Unique ratio of total
C4
0.00
0
0.00
mC4-en OSCAR
0.00
0
0.00
3,958,969 158.824,858
0.01
2,542,577
0.01
LAION-2B-en
0.07
61.674,276
0.03
31

---

Published as a conference paper at ICLR 2024
NO NETOTT
hp
hp
AllPosters
House for sale
GROUPON
ZELTAURTION
SOLARIS
Jun Nabio
Figure 15: Images from the top 25 most duplicated URLs in LATON-2B-en.
is at exactly 100,000 characters; all documents over this length were chunked into multiple documents of length 100,000 by the dataset builders.
RedPajama also contains -egeraat-g user-facing copy. including, e.g., placeholder pages for alumni of various secondary schools (each associated with unique individual's name). This analysis also reveals collection of documents comprising nearly 0.01% % of the dataset, containing what appear to be usernames or titles associated with pornographic content.
Finally, The Stack contains many template-generated new-duplicate documents; for example, large number of auto-generated metadata files for Unity assets, each of length 20 tokens. It also contains a significant number of documents of length 20,000 characters that contain float and bit matrices.
The Pile also includes a significant number of auto-generated metadata files corresponding to Unity assets, e.g.:
fileFormatVersionn 2
guid: e32ro07feearaac4488bee3eeee2 timeCreated: 1435687483 license Pro SriieeForaaIImooree. userData:
assetBundleNamm:
assetBundleVariant:
as well as auto autor-eeeerted files corresponding to publications in medical journals, e.g.:
!C]
(#sp1 .384)
32

---

Published as a conference paper at ICLR 2024
Characters Distribution
8.07
804
0.7
3.58
8:44
8.22
015
0.00
600
3.00
10'
15
Chocclesspp par Docanerd
Gursaain per Docamnor
Claraden Docanen
Ouracton per Docorart
Clarsdees Docanert
G0ORD
LACN Biier
The Staak
0,000
4150
0.000
00000
ant
0000
0.000
0008
1.00
<7'
of
12
12'
12
15"
Tokens Distribution
0.12
0.09
0.30
030
0.10
81100
0.5
025
0.4
0.5
020
005
0.10
0.00
5.0%
5.500
002
002
0.0
0.000
5.00
Talem Document
Toler per Oeourent
TeleEs Dotumen
Talans per Coounett
Tekoro Donnne
S20RO
LAKON-D
The Stack
0018
00150
0012
0702
0.12
03100
630%
0.00
0229
03000
0.00
"6"
18
10"
Figure 16: Distribution of document lengths for each of the datasets.
Table 15: Median document lengths for tokens and characters.
Corpus
Median Token per Document
Median Character per Document
OpenWebText
634
3,185
C4
227
1,153
mC4-en
397
1,988
OSCAR
423
2,163
The Pile
361
1,835
RedPajama
514
2,604
S2ore
4,538
23,418
PeS2o
4,582
23,852
LAION-2B-en
10
54
The Stack
430
1,953
33

---

Published as a conference paper at ICLR 2024
B.3 COMMUNITY- AND RELEVANT MEASUREMENTS
In this section, we provide additional results on the contamination and PII analyses from the main pa- per, as well conduct two more analyses: toxic language and demographic sentiment co-occurrences. Overall the community- and society-relevant measurements contain the following analyses:
1. Benchmark contamination (B.31T
2. Personally identifiable information (B3.22
3. Toxic language ($3.3.3)
4. Demographic sentiment co-occurrences ($B.3.4)
B.3.1 BENCHIMARK CONTAMINATION
We measure contamination by testing whether all of the input fields are present in single document, and report the percentage of examples from the test set that are contaminated and present the results Table [6 We do not test for the presence of the labels those are not always available, and they can come in different forms (e.g., in RTE they may appear either as 'entailment', 'not-entailment or as '0', '1'). Moreover, we not test for consecutive appearance of these inputs, as they might appear in different orders and with different separators. As such, our contamination evaluation serves as an upper bound of exact-match dataset contamination. By employing exact match comparison with the pretraining data, we ignore minor changes in words phrases that models trained on such similar texts may exploit. An example of such influence is introduced by |maami al. (2020), who showed how high overlap between sentences in the Winograd Schema Challenge (Levesque et al., 2012) and pretraining corpora inflates the results on the test set, while Elazar et al. (2021b) argue that knowledge and reasoning capabilities from large pretraining corpora leak and inflate evaluation benchmarks.
Rationales of the Design Choices Here, we provide the rationals behind our design choices for the contamination experiment. Overall, our desiderata required large benchmark that can be processed automatically, and that matched in an inspected corpora would be of high precision. We details these rationals in the following points:
Choice of task type. We chose to use tasks that include two more inputs (e.g., natural language inference) as the co-occurrence of both inputs in the same document increase the likelihood of these inputs to originate from existing evaluation dataset. In contrary, texts from tasks containing single input (c.g., sentiment analysis) may naturally occur some text corpus, which decreases the likelihood of contamination.
Ignoring the output. We decided to ignore the output of the inspected datasets since these can appear in different formats (c.g., numeric values, text labels, etc.).
Choice of PromptSource. Finally, use PromptSource (Bach al. 2022) as is the only large scale benchmark which we could automatically process and discern the different input parts (e-g., this is important since many datasets contain additional fields like metadata which are not directly part of the task).
Note that different design choices be made for inspecting additional contamination of benchmarks.
34

---

Published as a conference paper at ICLR 2024
Table Contamination percentages of the datasets filtered from PromptSource (Bach et al. 2022 C4, OSCAR, The Pile, and RedPajama.
Dataset/Corpus
OSCAR
The Pile
RedPajama
peersarial-g--adeeeeee
0.03
0.03
0.03
0.03
adversarial- qa-dbert
adversurial qu qu-dbiduf
0.00
0.00
0.00
0.00
adversarial-gee Dherssriallgidddrbeeer -gaddrbbera
0.10
0.10
0.10
0.10
anazon-reviews-nute
2.28
2,10
148
2.06
billsum
0.06
0.05
0.03
0.06
cosmos-qa qa
crows-puie pairs
0.00
0.20
0.00
0.60
Paooe-ParaphasccCCC
0.00
0.00
0.00
0.00
Selte-
canli
0.04
0.08
1.13
1.24
gigaward
0.15
0.36
1.18
2.82
gine-ax
5.07
6.16
glue-muli-mmatched
1.65
1,77
2.17
2.26
glue miliimiismtthhed
1.73
1,91
2.11
2.17
glue corpe
glue quli
0.13
0.04
1.48
1.21
ginc-qnii
0.09
0.04
1.48
1.21
gluc-rte
0.20
0.17
0.13
67,47
glue-stab
3.48
3.12
11.09
9.86
giue-wall
0.00
0.00
0.00
2.05
hend-qa-cn
5.22
5.29
5.11
3.9
health-fact
7,53
3.40
1.94
18.70
liar
29.23
13.95
10.91
45.05
math saahh-dtaset-lll algehet lineur- Id
0.00
0.00
0.00
0.00
math-dataset alac--llggg--iieeeee linea---nnna-2-
0.00
math-dataset- sal---atttt--geebrrre algebra lincar- 2d-composed math-qu
0.00
0.00
0.00
0.00
0.34
0.03
0.00
0.07
me-taco
mocha
0.00
0.00
0.00
0.03
opensa-tumineval
0.00
1.22
0.00
0.00
paws-x-en
paws labeled -inall
0.05
0.04
0.25
0.35
0.06
0.03
0.06
0.13
race-high
0.11
0.00
0.00
0.26
race-middle
0.21
0:21
0.00
0.35
ropes
samsum
0.00
0.00
0.00
0.12
-ann-addrrm--jamm
0.00
0.00
0.05
0.16
sca-aaddprim-tur
0.08
0.00
can--iller-numl)
0.00
0.00
0.00
0.09
Lenn-length
0.00
0.00
0.03
0.00
simple
0.26
scan-Icmplate-arroue
0.00
0.00
0.00
0.18
template-ejuueee jump
0.00
0.00
0.00
0.09
0.00
0.00
0.04
0.16
scun template right
0.00
0.00
0.11
0.16
seicite
1.78
0.86
1.72
scital-sall-foomae
0.09
0.38
0.28
0.71
scitail-tor format
0.09
0.38
0.28
0.71
sem-eval-
0.35
4.89
$2.81
skck
0.18
4.79
52.61
snli
0.04
0.08
1.11
1.22
squadstifts-anszzon
squndshifis new-wiki
0.01
0.01
0.01
0.03
squndshifts-nnt
0.01
0.03
0.02
0.04
stab-molti-nst
subjqu-bood books
0.00
0.00
0.00
0.00
subjge-grocery
0.00
0:00
0.00
0.00
subjeu
su beigg-restturanttt
0.00
0.00
0.00
0.00
super-gh pepr--lle--xb
1.99
1.45
5.07
6.16
super glue-axg
super glue nee-hook
0.00
3.05
0.00
0.03
super glue-boolg
0.00
3.05
0.00
0.03
super- glue-ch
supor-pl peor-glle-coee gluc-copa
0.60
1.00
1.20
100.00
super glue glue-multire
0.00
0.00
0.00
0.00
super glue
esper-gluertte
0.20
0.17
0.13
67.47
super-pe glue-wic
64,43
49.43
60.21
swag-regular
2.48
1.65
2.21
2.79
tab fac-fact-tab
0.00
0.00
0.00
0.00
wiki-uu
0.24
0.18
0.19
0.91
winegrad-wso-weee
winngrande-winggren
0.00
0.00
0.00
0.00
xnli-en
0.12
0.24
0.36
0.44
xsum
2.13
0.13
3.30
4.28
zest
0.00
0.00
0.00
0.00
35

---

Published as a conference paper at ICLR 2024
B.3.2 PII
We use three regular expressions inspired by |Subramani al. (2023) to identify email addresses, phone numbers, and IP addresses across pretraining corpora. In addition, we improved the phone numbers regex for better precision. These regexes provide us with high precision performance (which we manually evaluate) and allows fast PII identification. We apply postprocessing rules to the resulting matches, to improve the precision of detecting personal information by seeking to eliminate common classes of false positíves (such as ISBN numbers that may be flagged as phone numbers). These rules are enumerated in Table
Applying these regular expressions to the ten corpora we study in the paper, Table contains the number of matches of each PII type in each corpus. For faster processing, we filter documents containing large amount of special characters (such documents with >50 consecutive ":)" emoticons). We further normalize this statistic, by the number of tokens in each pretraining dataset, in order to estimate the relative proportion of PII in each corpus. These results are in Table [9]] We observe that even when controlling for the number of tokens the different corpora, mC4-en has a large amount of personal information compared to the other pretraining corpora.
We manually evaluate the precision of the heuristics. In order to compute this statistic, we sample 100 examples of strings detected PII (when available), for the three PII types, over the pretraining corpora in this study. These results are in Table [8 The nature of this retrieval task makes challenging to estimate the recall of our method, and more work is needed on the topic. We show the types of examples that may be incorrectly identified as PII by our method in each corpus in Table
2
Table 17: Regular expressions and postprocessing rules used to identify three PII types (email/ phone numbers/IP addresses).
PII Type
Regular Expression
Postpracessing Filter
(1) The usemname cannot be only "("
Email Addresses
(2) There
*." in domain
()) 'ISNN' 'DOI", or "#"
appear in context window of 50 characters from the match
Phone Numbers
(s+()(d(3))))1L |, (P(((4))
(2) Cannot contain URL
(::0:2510-5)2(004110 9]][ 00100/000--09111111 9178)13 91000000000000000001000000000000
(I) 'ISBN", 'DOI", "#" cannot appear in context window of 50 characters from the match
Addresses
Assumptions and Limitations: We make a number of assumptions in doing this analysis, and we describe them below:
We choose three types of PII: phone numbers, email addresses and IP addresses. These three types of PII have relatively standardized formats (for example, IP addresses are always 32-bit numbers expressed in dotted decimal format), which allows construct regular expressions to search for these information types in text. However, the retrieved information types may not correspond any one individual- for example, government organizations have email addresses and phone numbers.
Conversely, many types of personally identifiable information are not easily specifiable in the structured format not identify them in pretraining corpora.
for the information types in this study, and as result do
While many types of information individually may not appear to identify a specific individual, they be combined with information elsewhere the internet to form PII. In this work, we only identify a small proportion of potential personal information that is present in pretraining datasets, but further work is needed analyze the extent to which pretraining corpora include personal information as well as how this information can be sanitized.
Finally, we do not claim to estimate the risk level or sensitivity of the information types we extract from the pretraining corpus, acknowledging that this is highly context-dependent and personalized.
36

---

Published as a conference paper at ICLR 2024
Table 18: Extrapolated frequency of matches for regex searches different kinds PII (email/ phone numbers/IP addresses) in pretraining corpora. This is computed by multiplying the precision of our PII identification module for each pretraining corpus with the number of detections, in order to estimate the number of true matches. Prec, contain the precision of our identification method, estimated by manual verification, Precision indicates the proportion samples detected that we can reasonably infer accurately matching the PII type. We sample 100,000 documents from cach corpora, and analyze 100 samples of cach detected PII type when available. indicates that less than 100 samples for PII type were found in corpus, and we report the precision amongst the available PII detections. The number of samples for these corpora/PII type combinations follows: LAION-2B-en /Email Addresses (17), LAION-2B-en /IP Addresses (16), PeS2o/Phone Numbers (13), PeS2o /IP Addresses (12), RedPajama/IP Addresses (95), S2ORC Email Addresses (10), S2ORC/ Phone Numbers (1), S2ORC / IP Addresses (0)
each corpora.
Corpus
Email Addresses
Phone Numbers
IP Addresses
Count
Prec.
Count
Prec.
Count
Prec.
OpenWebText Web'Tett
363,789.4
99
532,929.8
87
70,430.0
54
OSCAR
62,802.224.0
100
107.163.132.4
91
3,237,420.8
43
C4
7,614,759.2
99
19,702, 198.4
92
796,494.7
56
mC4-en
201,368,9455 945.0
92
1967,997,4207
97,887,5100
44
The Pile
19,882.348.2
43
38.019,831.8
65
4,078,794.7
48
RedPajama
35,217,396.0
100
70,264,985 985.9
1,126, 129.5
#30
S2ORC
630,130.0
#100
1,465,9477
#100
0.0
*0
PeS2o
418,136.9
97
226,937.5
#30.8
0.0
*0
LAION-2B-en
636,252.1
#94
1,029,066.6
7
0.0
*0
The Stack
4,329,620.3
53
45,473,381.9
4,481,490.
55
Table 19: Extrapolated ratios of PII frequency (the number PIl PIl matches multiplied by the estimated precision).
PII Precision
normalized by number of tokens in corpus
#Tokens
PII Type
Email Addresses
Phone Numbers
IP Addresses
OpenWebText
0.000047
0.000069
0.000009
OSCAR
0.000409
0.000698
0.00002 1
C4
0.000003
0,000007
0.000000
mC4-en
0.000423
0.008546
0.000206
The Pile
0.000070
0.000133
0.000014
RedPajama
0.000034
0.000069
0.000001
S2ORC
0.000011
0.000024
0.000000
PeS2o
0.000009
0.000005
0.000000
LAION-2B-en
0.000021
0.000035
0,000000
The Stack
0.000003
0.000030
0.000003
Corpus
Email Addresses
Phone Numbers
IP Addresses
OpenWeeTexxt
367,464
612,563
130,426
OSCAR
62,802,224
117,761,684
7,528,885
C4
7,691,676
21,415,433
1,422,312
mC4-en
218,879,28
6.163.632,464
222,471,614
The Pile
46,238.01
58,492,049
8,497.488
RedPajama
35,217,396
74,749. 985
3,753,765
S2ORC
630,130
1,465,947
373,095
peS2o
431,069
736,810
239,912
LAION-2B-en
676,001 8,969,0
14,700,951 505,259,799
522,005
The Stack
8,148,165
Table 20: Frequency of matches for regex searches of different kinds of PIl in pretraining corpora.
37

---

Published as a conference paper at ICLR 2024
Table Abbreviated examples incorrect detections by our method, for each PII type, each pretraining dataset. The exact span that was matched red. Offensive content and personal information have been redacted from the presented examples.
Corpus
Email Addresses
Phone Numbers
IP Addresses
skremoved) has joined trayvonmartin sets "!" "!"@n="s,ee see trayvonmartin has kicked whitepower from #*****
...2017 limitation pes. article 472172730 ean
[stdout] awy was overriden from notenoughitems 1.6.1.9.jar 2014-03-24 20:25:06 [info] [minecraftcclent)...
OpenWebText
4012138149625 the model
was produced in the usual minichamps...
"you ever googled our email dress? try googling "@ "fm..com" and "charity" together, and you will an idea"
your morigage. claimer property reference 100103003249. the information displayed about this property
not load file assembly smmswaappers version 3.0.0.0
mC4-en
sinwwappe wrote in mes- sagenews:a.3cc1pp6.
"stat-major-fauu" faults": 1213, "stal- total-memory", 3975217152, "stat-swap-in".
s not constitute consent re- quired by n.j.a.c. 11.5.6.J for
vffggr..4ffg7vve8 44axx @44ax .om... bought iii garage sale and did not
the advertisement of listings clusively
OSCAR
getty images) michael jones9 october 2021 21:53 1633812509 andorra england player ratings: phil foden shi...
...latest update software with version number 10.0.0.16 currently the update available in the...
The Pile
ciguren3]] label="table4"
undefinee behavior. for ple, get that 2083899728 and d = -557766888. the persis- tent thing are
such damage. according cema-262, sections 8.6.2.2 and 8.6.2.3 you'r re not // allowed override rea
RedPajama
watercolor baby bring book card printable png 1525458984 watercolor baby bring book card printable png
wikipedia) 18:54, 15 july 2013 (utc) if can. 86. 146. 46.46.88 john of reading (talk) 06:38, 25 july 2013 (utc)
$20rc
PeS2o
65%60..0022
izona institutional review board (approval number 2003521636a002 bascline, the participants reported thei
LAION-2B-cn
Democrat-
queen creek 85142 cher- rywood property id: throne apk 3.8.10.1
gods and glory: war the
Gazette/Michael Woods
-03/15/2015-
131
w w&NWAMICHAELW.
The Stack
reminrorfu@@0...3
crmine the vision-agent service is running 15010872669 add missing heartheatresponsee
atoaune - have you upgraded oracle soa suite 12.2.1.1 and find partitions configu- ration any
timersecs the
38

---

Published as a conference paper at ICLR 2024
Table 22: Toxic language percentages based taxonomy and classifier over entire documents in the corpora we consider. Toxic language statistics the corpora we consider. The document toxicity (the first two columns) reports the percentage of documents that contain at least one mention of toxic language detected by each of the approaches. The classifier is applicd separately on cach sentence. The fine-graince taxonomy mention (the last three columns) reports the number of toxic mentions overall, and their relative appearance normalized by the number of tokens each corpus.
Documents with Detected Toxicity
Fine-grained Taxonomy Statistics
Corpus
Classifier
Taxonomy
Offensive- minnorty
Offensive minnnoirt
Harmless minority
OpenWebText
16.47
13.8
149K (1.92 (1.92c-05)
3.55M 04)5
13.5M 13.5M (1.74c-03)
C4
5.75
0.01
158K (1.03e-06)
47 (3.06e-10)
146M (9.51e-04)
mC44en
6.09
0.15
31.4M (1. 16c-05)
6.55M (2.42c-06)
2,85B (1.05c-03)
OSCAR
9.58
8.97
8.91M (77-75)
236M (4.95e-04)
549M (1.15e-03)
The Pile
8.27
7.67
4.55M (1 (1559e-05)
84.7M (2.96e-04)
238M (8.32e-04)
RedPajama
10.3
7.88
15.2M (1 (1.49c-05)
283M (2. (2.76c-04)
1.43 (1.40c-03)
S2ORC
10.52
16.55
95.9K (1.60e-66)
8.02M (1.34c-04)
33M (5.52e-04)
peS2o
9.56
17.0
47.8K (1.09c-06)
5.96M (1.35e-04)
26.7M (6.07e-04)
LAION2B-en
1.09
0.89
2.69M (9.09e-05)
25.4M (8.55e-04)
182M (6. 14c-03)
The Stack
1.16
1.85
4.63M (3.04e-06)
84.8M (5.56e-05)
228M (1.50e-04)
B.3.3 TOXIC LANGUAGE
How common is toxic language used in corpora? We employ two complementary methods for computing toxicity. The first is based on the work of (Zhou et al. 2021). who compiled a lexicon of terms (TOXTRIG) into three categories: possibly offensive minority identity mentions, possibly offensive non-identity mentions, and non-offensive minority identity mentions. It is then used by matching these "toxic triggers" over texts. The model-based method uses an SVM classifier trained on a dataset consisting of 200K examples based on Wikipedia and Twitter to identify toxic language [4] We apply such a classifier on each sentence separately and consider the document toxic in case any sentence is found to be toxic. We present the results in Table 22 C4 is the least toxic based on the taxonomy: only 0.01% % were found to be toxic, which expected due to the filters used in the curation process of the dataset. On the other hand, the classifier finds more documents to be toxic: 5.75%, which may indicate subtlcties that the lexicon used for filtering documents from C4 did not catch. Open WebText is the most toxic corpus based the classifier, while PeS2o is the most toxic one based on the taxonomy, perhaps surprisingly, as it is not a web-based corpus.
Explicit Content Filtering The only dataset we analyze that explicitly filtered for toxic content (in the form of keyword matching) is C4. Indeed, the matching category from our analysis are the "Offensive ** categories. Our analysis, that uses a fine- finengrained lexicon (Zhou et al., [002)). splits this category into "offensive-minn "infensiveeminority"" and "offensivee-ot-minorig 47 mentions of the "offensive-not-nee not-minority" category, likely due to a difference in filter used to create C4 and our lexicon. In comparison, other datasets that did not employ such filters contain several million references of such phrases. Interestingly, C4 also contains 158K occurrences of the ooffensive-minoriyy category, which were not filtered from the dataset.
In C4 we only found
B.3.4 DEMOGRAPHIC SENTIMENT CO-OCCURRENCES
In this section, we turn to detecting biases in the corpora based on demographic factors. We constructed a set of unigrams and bigrams associated with gender (male and female pronouns), religion (the proper names of several major religions), and race (combinations of racial identifiers and words like man, woman, people, etc.). The sentiment of sentences containing these terms was computed using SpacyTextBBlc
and averaged over a given corpus. The results for all corpora shown in Figure 17 The Stack is excluded from this analysis since the contexts in which these terms appeared were not typically natural language. Overall, we observe a neutral weakly positive sentiment for sentences in which most of our demographic terms appear, with the exception of those including 'black' being uniformly more negative across all corpora. With minor exceptions we don't ' observe substantial variation in the sentiment for individual terms among datasets. The weak positivity seen for all sources is in opposition to a related analysis performed Gao et al. (2020). which measured weak negativity for most terms. It's likely this is due to differences in the way
httpps: wttp:iitth....ciiiiiiiiieem http:///ttt.tttoooooooototaaottt
39

---

Published as a conference paper at ICLR 2024
OpenWebText
C4
mC4-en*
-0.05
OSCAR
Sentiment
The Pile
-0.00
RedPajama
S2ORC
-0.05
peS2o
LAION-2B-en
male
female
asian
black
hispanic
white
athiest
buddhist
christian
hindu
jew
muslim
0.15
Figure 17: The average sentiment associated with several gender, racial, and religious demographic terms for each dataset. Note: averages for datasets marked with * were computed for 10% samples.
average sentiment is computed (we compute sentiment at the sentence level while Gao et al. (0000) computes sentiment only for the most frequent co-occurring terms).
40

---

Published as a conference paper at ICLR 2024
10000
10000
LNDD-22--
4000
4000
0
Figure 1,000 most common unigrams LAION-2B-en (rank on æ-axis), their corresponding rank in C4 (y-axis), and visa-versa. The dashed red line corresponds to y = æ. Points below and above that line indicates differences between the corpora. For instance, common unigrams in LAION-2B-oo are of different adjectives and words often used describe objects (e.g., Black, Light, Happy, Woman's s), but those are much less common C4.
B.4 CROSS-DATA ANALYSIS
Main Findings
Comparing unigrams of different corpora reveals distributional and topical differences.
OSCAR unigram distribution is the most similar to all other corpora on average.
50% of RedPajama unique documents originate from C4 and 50% of OpenWebText unique documents originate from The Pile.
While mC4-en was supposedly superset of C4, documents from C4 constitute only 0.04% of mC4-en, while the later being only 10x larger size.
Using the analyses from the previous sections we can now perform targeted comparisons between different corpora. Such analysis is the first step of better understand the similarities and differences between corpora. We perform the following analyses:
Distributional similarities ($ (B.41.
2. Corpus overlap B.4.2
B.4.1 DISTRIBUTIONAL SIMILARITY
Unigram Ranking Using the most common n-gram statistics 44..1p. can compare the ranking of these n-grams, to gain insights into their different usage between corpora. For the following analysis we consider the top 10,000 most common unigrams of two corpora, and display the 1,000 most common unigrams in one corpus as a function of the same unigram rank in the other corpus. In Figure [8 we display the rank of unigrams C4 as a function of their ranks in LAION-2B-en Some very common unigrams in LAION-2B-en describing objects such as "Two", "Black", "blue", and "Light" are very common LAION-2B-en top 500 unigrams, but much more rare in C4's top 1,000. Another category is models such BNW and Toyota whose ranking is about 900 in LAION-2B-en, but above 6,000 C4. Figures 28 show the paired ranks for all corpora pairs.
41

---

Published as a conference paper at ICLR 2024
402
1000
Opentes
322
250
120
100
100
60
470
030
200
Figure 19: WpenWebText top 1,00 unigrams, and their corresponding indices in the other corpora.
42

---

Published as a conference paper at ICLR 2024
100
130
100
74
103
300
300
30
100
X33
430
200
Figure 20: C4 top 1,00 unigrams, and their corresponding indices in the other corpora.
43

---

Published as a conference paper at ICLR 2024
200
200
130
432
100
430
200
Figure 21: mC4-en top 1,00 unigrams, and their corresponding indices in the other corpora.
44

---

Published as a conference paper at ICLR 2024
A00
120
100
200
1.23
420
100
X30
400
422
200
Figure 22: OSCAR top 1,00 unigrams, and their corresponding indices in the other corpora.
45

---

Published as a conference paper at ICLR 2024
100
4.00
200
4000
423
120
120
000
200
152
100
130
470
200
222
Figure 23: The Pile top 1,00 unigrams, and their corresponding indices in the other corpora.
46

---

Published as a conference paper at ICLR 2024
100
2.0
102
300
100
1003
470
638
300
Figure 24: RedPajama top 1,00 unigrams, and their corresponding indices in the other corpora.
47

---

Published as a conference paper at ICLR 2024
1000
1100
120
480
0.0
122
250
200
Figure 25: S2ORC top 1,00 unigrams, and their corresponding indices in the other corpora.
48

---

Published as a conference paper at ICLR 2024
1000
4.0
1.00
1.00
1.00
100
422
4.00
200
200
Figure 26: peS2o top 1,00 unigrams, and their corresponding indices in the other corpora.
49

---

Published as a conference paper at ICLR 2024
1082
6.00
4.0
0.00
0000
X33
4.0
200
202
Figure 27: LAION-2B-en top 1,00 unigrams, and their corresponding indices the other corpora.
50

---

Published as a conference paper at ICLR 2024
1000
1023
0.00
4.00
1.00
A30
4.0
1.03
200
202
Figure 28: The Stack top 1,00 unigrams, and their corresponding indices in the other corpora.
51

---

Published as a conference paper at ICLR 2024
PM
1.3
X33
SaPore
314
021
338
HBC
X011
50
1000
23
24
23
(a) Intersection IS distance
(b) Union JS distance
Figure 29: The Jensen Shannon distance between top 1,000 most common unigrams in each corpus. The lower the numbers the more similar the corpora are. OpenWebText, C4, mC4-en, OSCAR, The Pile and RedPajama are quite similar to one another (in terms of the common unigrams distribution), and S2ORC, peS2o, LAION-2B-en, and The Stack are quite different from all other corpora.
Table 23: Top 10 exact text overlaps between than 2 datasets. C4, OSCAR, and RedPajama share the most amount of documents, with over million shared documents. Interestingly, even LAION-2B an image-caption corpus overlaps with other corpora, such as C4 and RedPajama (which all share more than 30 thousand documents).
Corpus Intersection
Count
C4 n OSCAR n RedPajama
1,680,953
C4 n mC4-en O RedPajama
1,375,088
The Pile O RedPajama n The Stack
592,364
C4 n The Pile RedPajama
118,432
C4 n RedPajama n LAION-2B-en
30,602
mC4-en OSCAR n RedPajama
14,319
C4 n mC4-en n OSCAR
12,854
C4 n mC4-en n OSCAR O RedPajama
12,854
OSCAR n The Pile RedPajama
6,112
C4 n OSCAR n The Pile
6,096
Unigram Overlap Next, by comparing the 10.000 most common unigrams, we compare the similarity between each corpora pair using the Jensen Shannon distance using (1) the intersection and (2) the union of the two vocabularies. We present the results in Figure 29 On average, we find that OSCAR's unigram distribution is the most similar to all other corpora (0.19 on average). The Stack, as expected, is the most distance corpus from all other corpora.
B.4.2 CORPUS OVERLAP
In this analysis, we compute the overlap between the different corpora, by comparing (1) the texts, and (2) the URLs, when available. The pairwise results are presented in Figure 30 for the texts overlap, and Figure 3T for the URL overlap. We see that text overlap diminishes quickly to zero more datasets are considered. Table 23 shows the largest text overlaps between more than two datasets. While the largest two are over million document clusters, this is less than 1% clusters in any of the involved datasets, and overlap size drops rapidly from there. This trend is similar for URL overlaps. The largest 3-corpora overlap is between C4, mCC-en, and OSCAR, with 6,77 shared URLS, while the rest of the overlaps share at most a single URL.
We find that documents from S2ORC and peS2o do not appear in other corpora. While it is likely that some of the academic papers shared with other corpora, e.g., The Pile and RedPajama
52

---

Published as a conference paper at ICLR 2024
Count Overlipe UPA 12
Pullo Overtars Inineeyed Doaumees
02 071 E2) 172)
LBKC
223
2.04
LOK
1E+20
S5-02
15:05
16-36
Daceet 3032
91 D.Daaen 1912
2578
283M
OCCAR
Undgae in
1196
4.0K
43.2K
132M
JA.2K
6:24
Phe
32-04
16-36
21:04
1C:60
16:23
02
15
26-22
19K
340M
1,75
1.M
504K
600M
11.2K
13:20
Andegerra
26:04
10+-20
-6-24
12:03
12-23
22+00
06
4E-00
20
26-32
52302
1M
223nc
01++31
68--6
01--3
54+66
51--31
1E-60
51+00
(e--0n
51+00
02:20
02--2
0C.06
02-20
DE.
20
22102
02112
12.08
Reddor
Le00 Whan
Phe
Openies
LesoR
Vtggu Doummons Colact
Uhigu m Catest
Figure 30: Overlaps of hashed full text between all pairs of datasets as counts and as ratio to dataset size.
Dount of Over lap I12 2222
Fadts of Quadapp Liiiuu UBLs in CD iBt 622 (D7))
LISOUI UILA Dabone COU
Lniour LRLL Cetaret (D3)
12400
42-42
32:02
mCAwet
243M
3.00
120M
112
4E:01
16+00
35:43
66
6SCAR
OSCAR
22:00
32:58
92:09
18+:0
Urigue URL Derasen (D2)
Unigue IRL 2 (02)
Figure 31: Overlaps of URL string between all pairs of datasets as counts and as ratio to dataset size.
that included arXiv as a data source, there are likely formatting differences that cause the exact string matching to be different. Interestingly, even S2ORC and peS2o do not contain any exact-text overlapping documents, despite pe52o being a cleaned version of S2ORC, due to a difference in formatting for parsed paper sections.
While RedPajama is 2.5 times larger than C4 in number of documents and 6.6 larger in number of tokens, find that 50% of RedPajama unique documents originate from C4. This can be explained by larger documents (as evident from the largest average document length in The Stack of 2,800 tokens per document on average, compared 420 tokens per document in C4, or by duplicate contents of C4 documents in RedPajama. Similarly, 50% of Open Web Text unique documents overlap with The Pile, which includes Open WebText as source. Another expected overlap is between datasets with Github as source (RedPajama and The Pile), and The Stack (which purely consist of Github code).
Finally, we also notice that while mC4-en was created from a superset the Common Crawl data used to make C4, documents from C4 only constitute 0.04% of mC4-en, while the later is only 10 times larger in size. We speculate that this due to formatting differences, between the C4 and mC4-en collection.
C LIMITATIONS
WIMBD has few limitations, described below:
The search tool we use is Elasticsearch. While it is scalable, was not designed for scaling with large text corpora. In addition, indexing these massive text corpora take few days,
53

---

Published as a conference paper at ICLR 2024
and keeping it running is costly. In the future, we hope to explore cost effective and faster indexing tools.
Search is currently enabled using Elasticsearch, which only enables exact-match search.
Fuzzy, and semantic search are important abilities that we currently do not support.
54

---

Published as a conference paper at ICLR 2024
Table 24: Time benchmark of the different analyses C4. We all of these analyses 224-CPUs machine, with 881 Gb memory. * contamination time was calculated on the test of COPA, which contains test examples. We also report the estimated cost in dollars based on Google's pricing of the machine we used, that $9.46 per hour.
Category
Analysis
Time
Estimated Cost ($)
Summary Statistics
6:32
1
Data Statistics
Internet Schemas
2:25
0.4
Internet Domains
5:38
0.9
Internet Domains per Token
3:32:07
33.4
Internet Suffixes
1:56
0.3
Utterance Date Statistics
2:12
0.3
Geolocation
1:17
0.2
Language ID
5:52
0.9
Top-1
9:08
1.4
Top-2
2:14:26
21.2
Data Quality
Top-3
5:45:10
54,4
Top-5
3:43:58
35.3
Top-10
8:43:40
82.6
Top-100
3:00:14
28.4
Bot-1
18:17
2.9
Duplicates
8:36
1.4
Length Distribution
8:56
1.4
Contamination
*:48
0.1
Comm. Measures
Toxic Classifier
3:19:12
31,4
Toxic Taxonomy
3:15:27
30.8
PII
24:44
3.9
Demographic Sentiment
11:11:17
110.5
Total
46:51:51
443.1
D BENCHMARKING RUNTIMES
This section describes the benchmark times each analysis took to run on the C4 corpus. While C4 is not the largest corpora we analyze, it popular one, and representative in size. All out analyses on Google cloud compute node with 882GB RAM and 224 CPUs. While the machine is rich RAM, our analyses typically did not use more than 250GB, and the reason for choosing such machine the availability of machine with enough CPU cores, that came along with this amount of memory.
We report the benchmark runs in Table 24 All of the analyses conducted took less than 12 hours to run, with 13 (out of 22) that took only several minutes, and all of the analyses on C4 took an estimated of 46 hours and 51 seconds (excluding repeated runs, and the contamination analyses other evaluation datasets). Note that while the measured time for each run were calculated using the TIME command in linux, there some variance, and those should be taken
rough estimate.
We also calculate the estimated costs for each analysis and report it in the same table (Table 24). We use the estimated $9.46 per hour based on httpt::: //ts:://lood..ooo soud.googlee coe/cco/////mmmm cotp://oommppeeallpriiiiiiccciia for our calculations, making the total cost on C4 $443.
This estimation does include the Flasticscarch hosting costs.
55

---

Published as a conference paper at ICLR 2024
E TECHNICAL DETAILS
This section describes the algorithms for computing the most common, least common, and total number of unique n-grams in large corpus. Each of these algorithms uses the same trick that was inspired by Bloom filters (Bloom) 1970) as described in section 3.1] As a result these algorithms do not provide exact results, and the accuracy determined by the amount of memory available for the hash table.
E.I MOST COMMON 7-GRAMS
To collect the (approximate) top-k n-grams we start by initializing a hash table of zeros (either u32 u64) which represent occurrence counts for each n-gram, and empty collection of the top-k n-grams. Then we iterate over the n-grams in the corpus and for each n-gram encountered we take its hash, increment the corresponding count in the hash table, and if that count is least large as the current minimum count in the top-l we add that n-gram to the top-k, potentially evicting another n-gram from the top-k.
After completing the iteration over the corpus the top-k will be complete and, in the absence of hash collisions, correct. However, the larger the corpus is relative the hash table, the higher the probability of hash collisions. A large enough corpus will have more unique n-grams than there are entries in the hash table, which guarantees hash collisions in the table, leading to inflated counts for some n-grams and the potential for false positives in the top-k. That's where the eeuracy-memor tradeoff comes in. The final counts reported for the top-k n-grams will always be upper bound of the true counts.
E.2 LEAST COMMON R-GRAMS
To collect the (approximate) bottom-k n-grams we also start by initializing a hash table of u32 zeros represent occurrence counts for each n-gram, and empty collection of the bottom-k n-grams. But this time we have to iterate over the corpus' 7t- n-grams twice.
During the first iteration we tally up the counts just like we do the top-k algorithm, except that don't add any 7-grams to the bottom-k collection. During the second iteration we now already have the final counts of all n-grams, so we simply look up the count of each n-gram encountered and then add it to the bottom-k collection if its count is low enough, potentially evicting another n-gram.
Hash collisions might cause false negatives with the bottom-k. i.c. some rare n-grams may be missing from bottom-k if they had hash collisions with more frequent n-grams. The final counts reported will for the bottom-k n-grams always be lower bound of the true counts.
E.3 UNIQUE R-GRAMS
To estimate the number of unique n-grams initialize hash table of booleans set 'false'. Then we iterate over all n-grams in the corpus and for each n-gram encountered we take its hash and update the corresponding boolean in the table to 'true'. After iterating over the whole corpus simply have to tally up the number of 'true' entries. This number is the estimate for the number of unique n-grams, which will always be lower bound of the actual number of unique n-grams.
1ttts not necessary use u64 integers when collecting the bottom- & even if there's possibility of overflow counts, provided overflows are caught and kept at 242 since only care about the exact count of rare n-grams which are unlikely ever reach overflow.
56


Reviews:
[{"review_id": "pwYJ8b8jY8", "reviewer": "Reviewer_c8KC", "summary": "The premise of the paper is, that we have a poor understanding of the datasets that have been used during the past years to pretrain LLMs. While the curation and preprocessing steps for many popular pretraining datasets have been described, we still lack an understanding on a common set of dimensions with comparable measurements. \n\nThis paper proposes a platform that produces a set of dimensions, such as duplication, top-repeated n-grams, domain statistics ect. to characterize a dataset. Throughout the paper, it demonstrates this approach on a set of popular pretraining datasets, such as C4, mC4-EN, Redpajama, OSCAR and others and summarizes findings and provides anecdotes that have been discovered with this approach.", "questions": "- Datasets: Some datasets (e.g. PILE, Redpajama) consist of several datasets, yet it seems the paper only considers the web-crawls in those datasets?\n- Token counts: What is a token here? Are you counting whitespace tokens or tokens from a subword tokenizer (which one, the same for all datasets)?\n- Domain counts: As the text tells us \"C4 contains documents from a diverse set of domains, and even the percentage of the most common one, patents.google.com, is less than 0.05%. [...]. Similarly, arxiv.org is responsible for more than 12% of the documents in RedPajama.\" While this is interesting, it feels like this topic is just scratching the surface and its still unclear how the rest of the data looks like.\n- N-Grams: Over tokens or characters?\n- Document duplication: The text says MD5, but then refers to \"Compressed counts\", which seems to describe MinHash?\n- Benchmark contamination: Frankly, the Llama2 technical report set a high bar for investigating contamination of relevant benchmarks. The selection here is not helpful for SOTA research, and the selection criteria is questionable: Why are benchmarks with less than one input field discarded? Why Promptsource? This part should either left out or fleshed out considerably with more details.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is a timely contribution to increase our understanding of data properties that are feeding the LLMs during pretraining. I am not aware of any comparable publication. The techniques used in the approach do not contain any novelty by themselves, but this would not be necessary for the scope of this publication. The paper is easy to follow and well organized.", "weaknesses": "While I really like the premise and the execution of the paper, I feel it often falls short of delivering enough insights into the data and could increase its impact. I like the various anecdotes and statistics of those popular datasets, yet many of the dimensions are slightly underdeveloped or lack clarity (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BVmP9VYyeq", "reviewer": "Reviewer_XCiq", "summary": "This paper proposes a tool for analyzing the contents of big textual datasets efficiently. It describes the implementation of the platform and gives examples for data statistics and data quality findings from 10 different large corpora for English as a result.", "questions": "- It would be helpful for each inspected corpus to describe briefly what kind of quality guards/filters were employed at its creation (if known), so that e.g. the presence of PII can be interpreted in context. If a corpus is supposed to be PII-filtered already, it would be very surprising to find more PII with this tool - or the regexes used here should be used to make the next generation of PII filter.\n- What are the limitations of the platform?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The implemented solution seems efficient and uses scalable methods, making use of simple tools and scaling techniques. \n- The proposed tool could be useful to basically any NLP practitioner working with data to be analyzed. It fills a gap of existing tools.\n- The shown examples illustrate well how the tool can be used to unearth surprising and previously unknown quality issues and outliers. The appendix has a generous number of analyses and results that might be helpful for anyone working with these datasets.\n- The report of benchmark overlap is a nice side finding that will hopefully affect how future models trained on this data are evaluated. It is hard to publish such findings individually, so I am grateful for the authors to include these here and prevent future misinterpretations of results.", "weaknesses": "- What the paper doesn’t address is how these instances that are problematic can be efficiently filtered from the data. Would the same tool be able to serve that purpose? Then one could nicely iterate with inspection and cleaning. This is not strictly a weakness as it goes beyond the scope of the proposed tool, but this should at least be discussed to make the observations more actionable.\n- The tool is only evaluated on English, and this is not explicitly stated. It is unclear if there are any English-specific biases in the way that e.g. PII regexes are designed, or if the tool can be expected to seamlessly generalize to other languages.\n- Given that it is a tool that is supposed to be user-friendly, it would be good to evaluate the usability, as the main contribution of the paper is not a method but a tool. How hard was it to dig out these quality findings, would an average user spot these quickly? Some kind of user study would be good to confirm that the tool is indeed usable. Are the statistics and summaries presented with visualizations to browse? The competitor, Know Your Data, might be good to compare against in that aspect, even if it doesn’t come with the same efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d05zOZycHh", "reviewer": "Reviewer_5Nw1", "summary": "The authors present a platform and set of analyses to explore the large-scale datasets used for training LLMs.", "questions": "See my comments in Weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The presentation and writing of the paper is good\n2. The insights are presented from multiple large-scale datasets and the proposed methodology could be extended to other large-scale datasets\n3. The authors document the set of observations from the dataset that several researchers and practitioners of LLMs in part experience.", "weaknesses": "I feel the analyses presented in the paper could be extended or strengthened along multiple lines such as:\n1. Can we look beyond the n-gram repetition to measure the duplicity? Is every n-gram duplication bad for training LLMs?\n2. What might help to better understand the relevance of the proposed platform is to (re)train the LLMs after removing the unwanted artifacts from the dataset. \n3. The impact on the output generated by the LLMs due to such unwanted artifacts in the dataset is still not clear. Some analysis/exploration along this line may help to better motivate the link of work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iquuuxWCR6", "reviewer": "Reviewer_jsfy", "summary": "The paper introduces an automated system called WIMBD. WIMBD is a framework for processing and analyzing large text corpora, with a variety of evaluation tools. In the work, using WIMBD the quality of ten different corpora widely used train language models (or vision and language models, etc.) has been evaluated. Insightful findings were found and provided, along with a data quality & impact assessment.", "questions": "N/A", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I think this work is a substantial contribution to the ML community in general, and even broader (like the NLP community). It covers how important good and qualitative data is, above the quantity of the data. As a consequence, I believe it is crucial to adopt tools like WIMDB in order to make sure ML models are trained on good, non-toxic, data in order to avoid reinforcing existing societal biases, and exhibition of toxic language that is negatively impacting as ML models used in day-to-day lives. Systems like WIMDB could also play an important role (as data is) in the interpretability and expandability of ML models' performance + reproducible models.", "weaknesses": "Not a weakness per se, but it would have been good to see the authors discuss how scalable the tool is (they tested 10 massively huge datasets) to 100, 1000, etc datasets (with different lengths of magnitudes).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)