LLM-Datasets: An Open Framework for Pretraining Datasets of Large Language Models
Authors: Malte Ostendorff
Pedro Ortiz Suarez
Lucas Fonseca Lage
Georg Rehm
Venue: COLM (2024)
Keywords: Demo paper, Dataset collection, Data preprocessing, European languages, Multilingual
TL;DR: We present an open framework for pretraining datasets of LLMs and showcase it with a multilingual dataset of 2.3T tokens.

Abstract: Large language models have become the cornerstone of today's natural language processing research. To facilitate the training, evaluation, and deployment of language models, the community has developed a series of tools and frameworks and made them openly available. This joint community effort has led to more collaboration, standardization, and overall more progress in language model research. However, one crucial aspect of large language models has been neglected so far: the pretraining datasets. To address this gap, we present an open framework for the collection and systematic compilation of pretraining datasets, called LLM-Datasets. With LLM-Datasets, we make a community-effort and collaborate with experts from the individual languages to collect and systematically compile datasets suitable in terms of data quantity and quality for pretraining language models in a multilingual setting.  The framework provides a unified interface to pretraining datasets enabling the download, text extraction, filtering, and sampling of the pretraining data. It is modular and extensible with new datasets and designed with high-performance-computing requirements in mind that are needed to achieve the scale of today's language models. Users of the framework can focus on the actual data composition and reuse existing datasets from the community while ensuring reproducibility. To showcase LLM-Datasets, we compiled a pretraining dataset with 2.3 trillion tokens for a large language model covering 32 European languages.

Published as a conference paper at COLM 2024
LLM-Datasets: An Open Framework for Pretraining Datasets of Large Language Models
Malte Ostendorff Pedro Ortiz Suarez?, Lucas Fonseca Lage*, Georg Rehm 4.1
!Occiglot, Deutsche Telekom, *Common Crawl Foundation, *Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKT) [firstname] /occiglot.or
Abstract
Large language models have become the cornerstone of today's natural language processing research. To facilitate the training, evaluation, and deployment of language models, the community has developed a series of tools and frameworks and made them openly available. This joint com- munity effort has led to more collaboration, standardization, and overall more progress in language model research. However, one crucial aspect of large. language models has been neglected so far: the pretraining datasets. To address this gap, we present an open framework for the collection and systematic compilation of pretraining datasets, called LLM-Datasets, With LLM-Datasets, we make community-effort and collaborate with experts from the individual languages to collect and systematically compile datasets suitable in terms of data quantity and quality for pretraining language mod- els in multilingual setting. The framework provides unified interface to pretraining datasets enabling the download, text extraction, filtering, and sampling of the pretraining data. It is modular and extensible with new requirements
datasets and designed with weerperformancccoopuue in mind that are needed to achieve the scale of today's 's language models. Users of the framework can focus on the actual data composition and reuse existing datasets from the community while ensuring reproduci To showcase LLM-Datasets, we compiled a pretraining dataset with 2.3 trillion tokens for a large language model covering 32 European languages.
1 Introduction
Language models are the cornerstone of today's natural language processing research but their scale has made the training, evaluation, and deployment an increasingly complex project. The community reacted to the increasing complexity by developing dedicated tooling and framework for parts of the language model stack. For instance, DeepSpeed (Rasley et al., 2020), Megatron-LMM (Shoeybi et al., 2019) or its forks (Black et al., 2022) are common codebases for large-scale model training. The lm-evaluationn-aarnss (Gao al., 2023) has become the defacto standard for benchmarking and evaluations. For inference and deployment, open source projects such as vLLM (Kwon et al., 2023) have been adopted by the community. Moreover, fine-tuning datasets are openly shared on the Hugging Face Hub (Lhoest al., 2021). However, the pretraining datasets of language models and their processing pipelines are rarely shared with the community. Notable exceptions are Bloom (BigScience-Workshop Workshop
al., 2023), Pythia (Biderman et al., 2023), and OLMo (Groeneveld et al., 2024). Even language models that explicitly claim to be open, such as OPT (Zhang et al., 2022), do not provide sufficient details to reproduce the pretraining data. This gap is even more severe given that there is more and more evidence that the pretraining data is one of the primary factors that determine model quality (Gunasekar et al., 2023; Gemini- et al., 2023).
In this paper, we address the aforementioned gap and present LLM-Datasets as an open framework for pretraining datasets of language models. The remainder of this paper
1

---

Published as a conference paper at COLM 2024
describes the principles that guided the development of the framework, explains its com- ponents such as the processing pipeline and the dataset viewer, and finally showcases pretraining dataset that created with the framework. LLM-Datasets is publicly available as a Python package and its source code is on GitHub with an Apache-2.0 license." !
Related Work
Since the introduction of language models in the form of pre-trained fixed-word embeddings like word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014) and fastText (Mikolov et al., 2018). Preprocessing the pre-trained data has become a crucial step in developing state-of-the-art language models, and while data demands have dramatically increased in recent years (Hoffmann et al., 2022), some of these foxed-word embeddings already used pre-training datasets in the Billions of tokens with Pennington et al. (2014) using 840 billion token dataset based on Common Crawl to train GloVe. And while data demands have increased to point where we are beginning to question we will run out of data (Villalobos et al., 2022) our data preprocessing pipelines have become more and more sophisticated throughout the years, and have started to target more diverse data sources in recent years.
One of the first data pipelines for processing pre-training data to be publicly available was that of fastText (Grave et al., 2018), this was a rather simple pipeline with very little documentation and based on bash scripts?. It was later with the release of OSCAR with its first pipeline goclassy (Ortiz Suárez et al., 2019) and CCNet (Wenzek et al., 2020) that pre-processing data pipelines became more central in scientific NLP publications.
From these first pipelines, up to more recent ones like C4's Raffel et al. (2020), Ungoliant (Abadji et al., 2021), Redpajama (Computer, 2023), Ref (Penedo et al., 2023), MAD- al., 2023) and Datatrove (Penedo et al., 2024); the main focus have been on quickly processing, cleaning and filtering Web data, more precisely Common Crawl data, although some very recent projects have also used the Internet Archive as a source (Aulamo et al., 2023). Most of the differences between these pipelines and libraries now revolve around the infrastructure optimization (Abadji et al., 2021; Penedo et al., 2024), the choice of boilerplate removal tool (Pomikálek, 2011; Endrédy & Novák, 2013; Barbaresi, 2021), the language identification method (Grave et al., 2018; Caswell et al., 2020; Kreutzer et al., 2022, NLLB Team et al., 2022; Burchell et al., 2023), and the quality metrics or filters used to select which Web documents to retain (Heafield et al., 2013; Rae et al., 2021; Jansen et al., 2022; Caswell et al., 2023).
LAD (Kudugunta
Since most modern models are now being trained with more than just Web data, recent initiatives have published data pipelines and management tools that can handle wide variety of data sources. some of these include widely used data libraries like HF Datasets (Lhoest et al., 2021) but also data preparing and pre-processsing pipelines specifically targeted to the training of LLMs such as The Pile (Gao et al., 2020), ROOTS (Laurençon et al., 2022), DOLMA (Soldaini et al., 2024), the NeMo Data C Some data exploration tools have also been released at times along these pipelines (Piktus et al., 2023), and some newer initiatives like the Common Corpus have moved away from using Web data altogether.
However, the aforementioned related work focuses either on techniques for data preprocess- ing or the collection of datasets with a narrow predefined scope, e.g,, only English datasets. combine both aspects and widen the scope of the data collection
With LLM-Datasets, such that framework users can concentrate on the data composition and easily pick the most suitable datasets for their own LLM use case.
Phttps: h/tmm///c//ommmmoow/cccccmmmmm .rg
Phttps://gi thub. wowaf//ccoooeeeeee//sss/etrrrree
Chttps://github com//WWWDLLLNNeeo---Craaaooo /tttrpnr//huggggggrarrroooooooop
2

---

Published as a conference paper at COLM 2024
3 The Framework
3.1
Principles
The training of large language models is a complex project. To account to this complexity, the LLM-Datasets framework follows set of key principles.
HPC-ready. Language models require large computing resources that are typically only available in dedicated high performance computing (HPC) facilities. We developed LLM- Datasets with the requirements of an HPC system mind. For instance, data will most likely be stored in network file system affecting the reading and writing of files. To account for this, all data produced by LLM-Datasets (10GB of uncompressed data). The file chunks are small enough to fit into memory (e. g., for shuffling) but at the same time they are large enough to reduce inode count.
stored in medium-sized chunks by default
Modular and extensible. The framework comes with a preconfigured collection of 2241 datasets. In addition these datasets, users can extend the framework with their own custom datasets without modifying the main code base. We also provide abstractions for common datasets, e. g., Hugging Face datasets, that allow adding new datasets with few lines of code. For all other data formats, only the text extraction logic needs to be implemented.
Private data support. While we are strong supporters of open data, acknowledge that not all data can be shared openly due to license restrictions or other reasons. For this reason, the framework supports the use of private datasets. All user datasets can be assigned to specific registries and tagged according to their availability (public, on-request, or private).
Reproducible dataset composition. A key requirement for rigorous science is the repro- duciblity of experiments. Thus, the framework is designed to produce the identical output dataset as long as the input configuration is identical. At the same time the underlying data may change during the development process, e. g., because novel data has become available. In LLM-Datasets, the final dataset composition is based on configuration files that enable the simple selection and sampling of underlying sources while being reproducible.
Bring your own model. Model architectures and implementations are evolving rapidly. To reflect this, I.M.-Dataetss is model agonistic, i. e., the composed and tokenized training data can be consumed via Python API or common modeling frameworks like Hugging Face Transformers (Wolf et al., 2020) or NVIDIA's Megatron-LM (Shoeybi et al., 2019).
3.2 Schema
Fig. 1 illusstrates the schema in which information is organized in the framework. The central element is the Datasel class. It represents specific implementation of one dataset. For example, all Wikipedia articles in a certain language from a specific date would correspond to one dataset. A Dataset associated with attributes of metadata such title, homepage, citation, or genre. However, more important are the attributes that are relevant for language modeling, e., the languages of the contained texts, license information, and the size of the dataset in terms of tokens and bytes. We also tag a Dataset for potential short texts or encoding errors) and regarding their availability (public domain, on request etc.). All the metadata allows users selecting the datasets that are most suitable for their use case. Groups of datasets that share many attributes are combined into a Data Source, g., all Wikipedia versions are represented as single data source. Besides the metadata, each Dataset implements a method that extracts the text from the raw dataset files. Taking again the example of Wikipedia: Extracting the plain text of Wikipedia's 'as XML dumps would corresponds to this method. For common data formats or sources, text extraction methods are implemented and can be reused.
issues (e. g,,
To make a Dataset accessible within the framework, it needs to registered in the Dataset Registry. The registry holds essential the information of all available datasets. If needed,
3

---

Published as a conference paper at COLM 2024
Dataset
Config
specific implemeniation
reprudecibie configuration the findl dulasel
(eg. Wikipedia 2023-41 de)
Selected datasets
Seed
Dataset Source
anglaages
Availability lity
License
Sampling
Split sioce
Data pullis
A common souree for differont
issues
Genre
Filter thresholdss
(eg. Wikipedia)
Token count
Byte count
JSONI datuset
HP dataset
Dataset Registry List of all default datasets
Panquct dataset
Extract text from dataset
Custom Dataset Registry
List of custom dutaseet
Figure 1: Each dataset is represented with dataset class, which implements its own text extraction logic or inherits from common data sources or other abstractions, e. 8. Hugging Face datasets. The final dataset composition is defined through reproducible configuration files with the information about the selection and sampling individual datasets.
custom registeries can be defined, e. e.g,, to have clear distinction between public and private datasets. In addition to the data-centric elements, the Config pivotal to the framework. A Config holds all information to make the creation of the final dataset reproducible and shareable such as random seeds, selected datasets, and sampling setting.
3.3 Processing Pipeline
Extract text
Compose
Tokenize
Raw Data
Data Split
Tokenized Data
JSONL
Parquet
Training dataset
Megatron-LM LM
Web
Shuftie
Validation dataset
Transfermers
HF Dutasets
Split by size
Deduplicate
Count tokens
The data originate from different sources and be in diverse formats.
The intermediate data rupresentation contains primarily the "plaintext".
Based the config, training and validation data selected and sampled.
The tokenized data be direcily used model training frameworks.
Figure 2: The framework's processing pipeline. The three main processing steps include the text extraction, data composition, and tokenization.
To obtain language data, which can be used for model training, the framework employs pipeline with three main processing steps: text extraction, data composition, and tokeniza- tion. Fig. 2 summarizes the processing pipeline. Initially all data is considered as raw data, meaning the data resides in its original format as provided by the data source. The format in that raw data is stored can range from simple text files on the local file system or on the Web to PDFs or other formats that require sophisticated processing for the text extraction. As described in $3.2, each dataset class implement its own independent text extraction step or reuse existing abstractions for common formats.
The extracted text is then stored in format unified for all datasets, e.g,, JSONL or Parquet. To reduce the file size, compression enabled by default. At this stage, additional processing steps such as shuffling, counting tokens splitting large files into small chunks be performed. These processing steps are implemented but optional as they are not mandatory for all L.M use cases.

---

Published as a conference paper at COLM 2024
Dataset: curlicat_bg bg
Todt provicm:
(a) Details, e.g., extracted texts.
(b) Statistics, e-g, token count.
Figure 3: The framework comes with a Web-based dataset viewer that allows inspecting the text content of all datasets and provides overview about statistics such as token count.
The next pipeline step is the data composition. The composition involves the mixing and sampling of individual datasets into one final dataset that can be split training and validation subsets. Configuration files ($3.2) define how the final dataset is composed. The composition step can be performed on-the-fly, e., the resulting samples are diretly consumed by a tokenizer, or stored the file system. On top of the composed dataset, the framework can perform exact deduplication using locality sensitive hashing. As the final pipeline step, the composed data is fed into a tokenizer to produce the input data for the language model training framework. As tokenization frameworks, Google SentencePiece and Hugging Face tokenizers" are implemented.
The framework intentionally does not implement common preprocessing techniques like near-deduplication or toxicity filters since other dedicated tools exist for these preprocessing steps, e.g., text-dedup or Perspectioe". However, these dedicated tools can be applied on the data files, either on the extracted texts or on the final composed dataset. For the Data Trove processing framework, we provide even a dedicated interface that allows ingesting all datasets implemented in LLM-Datasets with DataTrove.
A short documentation about installation, usage, and configuration of the framework can be found in Appendix A.1. For more comprehensive overview and detailed examples, we refer to our Web-based documentation.
3.4 Dataset Viewer
Training datasets of large language models are becoming so large that it is no longer feasible to inspect all the data manually. However, when deciding for data composition it is still crucial to manually look at least at sample of the data. For this purpose, the framework comes with Web-based data viewer that allows users to view the extracted text and statistics over the available datasets. With the dataset viewer, one spot systematic errors that might occur during the processing of the datasets, e. g., bad encodings or false language identification. The dataset viewer is implemented as Streamlit application." 11
/https://gi com/ /https://gtt
https/////ggttttboooooo///////// com///oooglee/eetteenepieeee
hhttts::/cchenghaommuuu titb io/text-dedup/
ptttps:://errspeciiveaps com/
/ttttps:/malteess. gi github.... wwt.iio/doaaaeeeeeeeeeeeeeeeaaae datasets/
Phttps://streamlit...
5

---

Published as a conference paper at COLM 2024
4 A European Pretraining Dataset
The framework is language-agnostic and can be used for any kind of monolingual or multilingual pretraining datasets. As a first example, we compiled a pretraining dataset with 2.3 trillion tokens for multilingual language model covering 32 European languages. The number of tokens is based on the Bloom tokenizer (BigScience-Workshie dataset-level separately for each data subset by
et al., 2023).
We estimate the number of tokens counting the white spaces in all documents, tokenizing 1,000 randomly selected documents, and then multiplying the white space count of all documents with the tokens per white space in the 1,000 documents. An overview of the European dataset is presented in the following.
4.1 Languages
17
Tokens (log scale)
100 B
8
100
80
Language
Figure 4: Tokens per language. English is by far the most dominant language.
The dataset contains text from the 24 official EU languages and eight additional regional and non-EU languages. Specifically, it covers English, German, Spanish, French, Greek, Italian, Ukrainian, Hungarian, Portuguese, Polish, Dutch, Czech, Slovak, Bulgarian, Swedish, Finnish, Romanian, Danish, Slovene, Norwegian, Croatian, Serbian, Estonian, Lithuanian, Catalan, Latvian, Maltese, Basque, Trish, Galician, Serbo-Croation Norwegian Nynorsk (sorted by number of tokens). Moreover, the dataset has diverse set of programming languages, primarily from the StarCoder dataset (Li et al., 2023). Fig. 4 shows the distribution of tokens per language. English is by far the most dominant language, reflecting studies on the technology support of Europe's languages. There are also translation datasets that mix two languages (see "mixed" in Fig. 4).
table with the token distribution by language can be found in the Appendix A.2. Please note that for a final data composition specific languages can be up- or down-sampled to achieve a more balanced language distribution tailored to the specific LLL use case.
4.2
Data Sources
The European pretraining dataset comprises 62 different data sources. A detailed list of all data sources can found in Appendix A.3. The large number of different data sources is due to the diverse set of languages covered by the dataset.
Most of the data sources are community contributes, i.e. they were suggested by researchers from the individual language communities. When compiling the dataset, the initial goal was to have at least one dedicated monolingual data source per language and to not only rely multilingual data sources such as Web-crawled data. The intuition behind this approach was that curated language-specific data will presumably have higher quality compared to their multilingual counterparts. However, such language-specific data sources could be integrated for low-resource languages like Irish. For these languages, the data originates only from multilingual sources.
The largest data source is Web-crawled data from Colossal OSCAR, separately described in $4.3. Other notable sources the multilingual corpora from MaCoCu (Bañón et al.,

---

Published as a conference paper at COLM 2024
17
Tokers (log scale))
100
calosnal mecar
startader
maxecu
redpejama
auviex
ligal not
court decisiond rd
weel
syn ug
pronf pile
great wob corpus
nor purweegian Cz
math ampo
pile
Figure 5: Tokens per data source for the top-5 sources. The largest data source is Web-
crawled data from Colossal OSCAR.
2022), EURLex12 and Wikimedia projects like Wikipedia, Wikinews, and Wikibooks. We also make use of high quality English sources. For instance, scientific literature from peS2o (Soldaini & Lo, 2023) or mathematical corpora like Proof Pile and AMPS Hendrycks et al. (2021). To avoid duplicated content, we discard specific subsets from data sources, e. g,, only the StackExchange subset from RedPajama is used and other RedPajama subsets such as Wikipedia or CommonCrawl are discarded. Other subsets like Reddajamm--owt BBokk are removed due to legal reasons. In general, the European pretraining dataset has a higher share of legal literature (laws and court decisions) compared to related work since in many European countries this data is public domain and available in large quantities. For the curated data sources mon--We-ccrwwles content), no additional filtering was performed apart from removing texts with less than 450 characters.
4.3
Web-crawled Data from Colossal OSCAR
Original docs.
Removed docs. by filter [%]
Filtered docs.
Dump
# [M]
Quality W.
Harmfulpp Categories
# [M]
Tokens [B]
2015-14
388.9
89.91%
0.67%
1.94%
38.2
102.9
2016-40
553.9
88.52%
1.08%
7.12%
58.4
158.3
2017-43
1101.9
87.03%
1.62%
1.54%
138.4
282.3
2018-47
768.8
84.99%
1.43%
0.50%
113.1
242.6
2019-22
815.3
86.46%
1.52%
0.32%
108.3
226.7
2020-24
814.0
87.94%
2.18%
0.32%
95.7
209.6
2020-45
801.8
88.00%
2.36%
0.36%
93.6
196.4
2021-49
760.1
87.40%
3.35%
0.23%
92.3
197.9
2022-27
997.7
87.70%
2.80%
0.25%
119.0
245.8
2022-49
1096.8
87.61%
2.89%
0.23%
131.7
279.2
2023-14
1002.2
87.35%
2.75%
0.22%
123.0
245.5
2023-23
1029.5
87.16%
2.85%
0.19%
128.2
257.9
Table Number original documents, percentage of documents removed by quality warmings, by harmful KenLM model perplexity and by Blocklist Category. The filters are applied in sequential order. The table also shows final number of documents and and estimated tokens after filtering. All of the numbers are given by Common Crawl dump.
/ http:///hggggoeoooooooc////////e

---

Published as a conference paper at COLM 2024
By far the largest data source is Colossal OSCAR (Abadji et 2022; Jansen et al., 2022) which an annotated version of twelve Common Crawl snapshots (also called dumps), i. i.e, Web- crawled data using the Ungoliant pipeline Abadji et al. (2021). The OSCAR project (Open Super-large Crawled Aggregated coRpus) is an open source project that aims to provide Web-based multilingual textual resources. The project provides language identification and annotations for some Common Crawl snapshots releasing them for more than 150 languages while trying to optimize the quality of these annotations (Kreutzer et al., 2022).
For our framework, we decided to include the latest release of OSCAR, namely Colossal OSCAR 1 The release comprises ten Common Crawl snapshots and uses the latest quality metrics introduced into the Ungoliant pipeline by Abadji et al. (2022) and Jansen et al. (2022). We also used OSCAR 2023.01* 15 that uses the exact same version of Ungoliant as Colossal OSCAR 1, and we also run Ungoliant ourselves on the CC-MAIN-2023-1 to create one more OSCAR like dataset. However, OSCAR provides only annotations to the Common Crawl data. The decision of how to filter the data is left to the user. Therefore, describe in the following what filter settings we used to derive our final dataset. All settings are found by manually inspecting samples of the data and deciding for the settings that produce to highest quality of content.
First of all, we extract only the European languages mentioned in $4.1 from OSCAR. We rely on the annotations made by the Ungoliant pipeline in separate steps. First, we use the quality warnings introduced by Abadji et al. (2021) including:
1. tiny: The document has low (< 5) number of lines.
2. short sentences: The document has high number (> 50%) of short lines (< 400 bytes)
3. header: The document has high number of short lines at its head, suggesting the presence of low quality content.
4. footer: The document has a high number of short lines at its tail, suggesting the presence of low quality content.
5. noisy: The document has high percentage of punctuation (> 50%)
We remove every document containing any of these warnings. To the remaining documents, we then apply the KenLM-based (Heafield et al., 2013) harmful perplexity filter introduced by Jansen et al. (2022). We remove all documents with perplexity below 25 and above 100,000. Finally, we used the URL-based category blocklist annotations based on the Toulouse UT1 Blocklist removing all of the remaining documents that are annotated with the following categories:
aggressive
ddos
adult
hacking
cryptojacking
malware
dangerous material
phishing
mixed adult
warez
sect
Tab. and Tab. 2 show the number documents by both date of the CommonCrawl dump and language, the percentage of documents removed at each filtering step, and the final document and token estimate count after filtering.
/hhttps:/huugiingaacee wttttess//iggggggee..oooottattee http://uuuu..ctiioooo/./f/aaaaaa en.
8

---

Published as a conference paper at COLM 2024
Original docs. [k]
Removed docs. by filter [%]
Filtered docs.
Lang.
Quality W.
Harmfulpp Categories
[k]
Tokens [M]
Top-5
6,137,246.1
86.84%
2.42%
0.76%
782015.3
950,517.47
de
841,750,9
88.19%
5.56%
0.19%
93685.4
134,711.25
es
654,498.5
87. 87.43%
1.13%
0.17%
81177.7
89,646.34
629,855.4
88.62%
1.45%
0.35%
70387.0 .0
78,929.40
it
340,549.9
89.96%
0.33%
0.19%
34024.0
50,417.61
Bottom-5
ga 140.2
87.80%
0.00%
0.10%
17.1
24.44
nn
73.9
95.27%
0.00%
0.00%
3.5
2.00
hr 138,9
98,72%
0.00%
0.00%
1.8
0.99
mt
21.7
98.02%
0.00%
0.00%
0.4
0.41
sh
20.3
99.81%
0.00%
0.00%
0.0
0.02
Table Number of original documents, percentage of documents removed by quality warnings, by harmful KenLM model perplexity and by Blocklist Calegory. The filters are applied in sequential order. The table also shows final number of documents and estimated tokens. The statistics are computed for the top and bottom 5 languages.
Conclusions
We have presented LLM-Datasets, open framework for pretraining datasets of large language models. LLM-Datasets provides a unified interface for downloading, text ex- traction, filtering, and sampling of large text datasets, including additional tools such as a dataset viewer. At the time of writing, 2241 datasets including subsets from 62 sources in 164 languages are implemented in the framework and readily available for language model training. In this paper, we further showcase the framework by presenting a European pretraining dataset that represents subset of all implemented datasets. The European dataset contains 2.3 trillion tokens of multilingual text, covers 32 European languages, and fully reproducible with the LLM-Datasets framework. Parts of this dataset are already being used to train language models.!?
Acknowledgments
The work presented in this paper has received funding from the German Federal Ministry for Economic Affairs and Climate Action (BMWK) through the project OpenGPT-X (project no.68GX21007D). The majority of the work presented in this paper was carried out while all four authors were with DFKI, working on the OpenGPT--X project. Some of the work carried out by Pedro Ortiz Suarez was also done within Occiglot where he remains member and active contributor.
References
Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus. In Harald Lüngen, Marc Kupietz, Piotr Bański, Adrien Barbaresi, Simon Clematide, and Ines Pisetta (eds.), Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event)) Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 COnline-Events pp. 1 Mannheim, 2021. Leibniz-Institut für Deutsche Sprache. doi: 10.14618 / 114466///s-puuie ids-pub-1la 10468. URL https: / /nnn resslvvinnn or g/g /urn: nbn: de: bsz mh399-04688.
17

---

Published as a conference paper at COLM 2024
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. Towards cleaner document-oriented multilingual crawled corpus. In Nicoletta Calzolari, Frédéric Béchet, Phillippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirleenth Language Resources and Evaluation Conference, pp. 4344- 4355, Marseille, France, June 2022. European Language Resources Association. URL https://aclaattoloogy. www.://...r../....../// 1rec-1. .43..
Mikel Artetxe, Itziar Aldabe, Rodrigo Agerri, Olatz Perez de Viñaspre, and Aitor Soroa. Does corpus quality really matter for low-resource languages?, 2022.
Mikko Aulamo, Nikolay Bogoychev, Shaoxiong Ji, Graeme Nail, Gema Ramírez-Sánchez, Jörg Tiedemann, Jelmer van der Linde, and Jaume Zaragoza. HPLT: High performance language technologies. In Proceedings of the 24th Annual Conference of the European As-
sociation for Machine Translation, pp. 517-518, Tampere, Finland, June 2023. European Association for Machine Translation. URL https: http:ss///clanntooooo.rorgr wrg./223.. hta.t/tt2..m..eeeme.m..mmmm eamt-1 61.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023.
Marta Bañón, Miquel Esplà-Gomis, Mikel L. Forcada, Cristian García-Romero, Taja Kuz- man, Nikola Ljubešić, Rik van Noord, Leopoldo Pla Sempere, Gema Ramirez-Sánchez, Peter Rupnik, Vit Suchomel, Antonio Toral, Tobias van der Werff, and Jaume Zaragoza. MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pp. 303-304, Ghent, Belgium, June 2022. European Association for Machine Translation. URL htttpss:/aalaaahlllle..o
org/2022. 010222.eamt-11 eamt-1.44 .41.
Adrien Barbaresi. Trafilatura: A web scraping library and command-lin tool for text discovery and extraction. In Heng Ji, Jong C. Park, and Rui Xia (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lingwistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 122-131, Online, August 2021. Association for Computational L doi: 15. URL https: http://aaaaaaaholocoooo tholooog org/2021. atrg/2221.acldemm... acl- 15.
Loic Barrault, Ondřej Bojar, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, August 2019. Association for Computational Linguistics. doi: (0.18653/v1/W119-550 URL https://aclannthologg/ or9/W19-
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430.
PMLRR 2023.
BigScience-Workee ;, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Belt- agy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponfer- rada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont,
10

---

Published as a conference paper at COLM 2024
Germán Kruszewski, Giada Pistilli,, et al. Bloom: A 176b-parameter open-access multilin- gual language model, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. G0T- An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. An open dataset and model for language identification. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volumee 2: Short Papers), pp. 865-879, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v /ttp://///22233lsrrr....o. URL https://acllanthologgy. org/ acl -oort..hr..5 75.
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Confer- ence on Computationall Linguistics, pp. 6588-6608, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/200 coling-main.577 URL https: //tps:::/// wwwaanhhoogy..rg////0
coling-main.55 579.
Isaac Caswell, Lisa Wang, and Isabel Papadimitriou. Separating the Wheat from the Chaff with BREAD: An open-source benchmark and metrics to detect redundancy text. arXiv e-prints, art arXiv:2311.06440 November 2023. doi: 10.48550/ 10.48500/arXiv.33 /at//a..221..0444.
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023. URL https: //ts:s://tttttttt// com/ www.m//ttteeerooooooeeeeoeeeeeee
István Endrédy and Attila Novák. More Effective Boilerplate Removal the GoldMiner Algorithm. Polibits Research journal on Computer science and computer engineering with applications, (48):79-83, 2013. ISSN 1870-9044. URL http://polibits. gettuuhh com/ com/2013 2013. 48.
Tomaž Erjavec, Maciej Ogrodniczuk, Petya Osenova, Nikola Ljubešić, Kiril Simov, Andrej Pančur, Michal Rudolf, Matyáš Kopp, Starkaur Barkarson, Steinór Steingrimsson, et al. The parlamint corpora of parliamentary proceedings. Language resources and evaluation, 57 (1):415-448, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. ar Xiv e-prints, art. arXiv:2101.00027, December 2020. doi: 10.48550/ att.aa21111121002222222
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, December 2023.
Gemini- Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Battiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Toannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Érica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, et al. Gemini: A family of highly capable multimodal models, 2023.
11

---

Published as a conference paper at COLM 2024
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Nicoletta Calzolari, Khalid Choukri, Christo- pher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Mae- gaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Fleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https://aclanthologyy org/L1 ArggL18-1558.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Ab- hilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023.
Asier Gutiérrez-Fandiioo, Jordi Armengol-Estapé, Aitor Gonzalez-Agirre, and Marta Villegas. Spanish legalese language model and corpora, 2021.
Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. Scalable modified Kneser-Ney language model estimation. In Hinrich Schuetze, Pasccaee Fung, and Massimo Poesio (eds.), Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 690-696, 696, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL
org/P13-2122
Peter Henderson*, Mark S. Krass*, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and 256gb open-source legal dataset, 2022. URL https://arxxv org/ abs/2207.e .02220.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models. arXiv e-prints, art arXiv:2203.15556, March 2022. doi: 10.48550/ arXiv.2203.15555.
Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. Perplexed by Quality: A Perplexity- -ased Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data. arXio e-prints, art. arXiv:2212.10444 December 2022. doi: 4448550/arXiv.2111
Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. Perplexed by
D quality: A perplexity-based method for adult and harmful content detection in multilingual heterogeneous web data, 2022.
Michal Kfen, Václav Cvrček, Jan Henyš, Milena Hnátková, Tomáš Jelínek, Jan Kocek, Dominika Kováříková, Jan Křivan, Jifí Milička, Vladimír Petkevič, Pavel Procházka, Hana Skoumalová, Jana Sindlerová, and Michal Škrabal. SYN v9: large corpus of written handle. saande..ee//il..
LINDAT COADIACCAARAA-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
czech, 2021. URL http: /
12

---

Published as a conference paper at COLM 2024
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzi-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50-72, 2022. //aclanthology.org httphhollgg.org/22222222 tac.21..cl11.....
doi: 10.1162 004477
URL https:
Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. Madlad-400: A mul- tilingual and document-level large audited dataset. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 67284-67296. Curran Associates, Inc., 2023. URL httpps:/pproceedingg neerrieeeeddnnssnnnnnisssss cttpaper.ccppppppepeepee wttppppe...llle/ppppeeeeeeeeeeee Bewchmaaarss.ssss..e............ pdf.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention.. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada,
Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Or- tiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The bigscience roots corpus: A 1.6tb composite multilingual dataset. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 31809-31826 Curran Associates, Inc., 2022. URL http///t//// neurseipiiinnnnnnnnpppee.....ns cc/paper files/ paper/2022/ Tileee992ek2222224933337777e-2e2
and and_Benchmarrk..... pdf.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tun- stall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre De- but, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagu- nas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Heike Adel and Shuming Shi (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Associ- ation for Computational Linguistics. doi: 10.18653 v www...emmlpddeoo...... URL https: http:s//aclanthologg.ooo org/2021 ewnee-dneeeeeeeeeeee memlp-ddmo... 21.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj,
13

---

Published as a conference paper at COLM 2024
Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muh- tasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Ku- nakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! 2023.
Nikola Ljubešić and Filip Klubička. bs,hr,srWaC- web corpora of Bosnian, Croatian and Serbian. In Felix Bildhauer and Roland Schäfer (eds.), Proceedings of the 9th Web as Corpus Workshop (WaC-9), pp. 29-35, Gothenburg, Sweden, April 2014. Association for Computational Linguistics. doi: 10.3115v11/1144005 URL https://aclanthollee org/w14-0405.
Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, and Claudia Borg. Pre- training data quality and quantity for a low-resource language: New corpus and BERT models for Maltese. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pp. 90-101, Hybrid, July 2022. Association for Computational Linguistics. doi: deeplo-1.10.
URL https::/aclanttollooe
oogoorgg/// 2022.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL htttp::/prroceedings neurips. wwtpcc/ppppppe///e//eep http:///a44431882229999999554333
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https:///claatthologs 111g:///oooooooo.//1111-1-111111 org/L18- 1008.
Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. Swiss-judgment-reediice A mul- tilingual legal judgment prediction benchmark. In Proceedings of the 2021 Natural Legal Languge Processing Workshop, 2021.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No Language Left Behind: Scaling Hluman-Centera Machine Translation. arXiv e-prints, art trXiv:2207.046777 July 2022. doi: 10.48550/ arXiv. 2207.04672.
Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora medium to low resource infrastructures. In Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald Lüngen, and Caroline Iliadi (eds.), Proceedings of the Workshop Challenges in the Management of Large Corpora (CMLCC7) 2019. Cardiff. 22nd July 2019, pp. 9- 16, Mannheim, 2019. Leibniz-Instituu für Deutsche Sprache. doi: 10.14618 ids-pub-9022 URL http: //nbn-resolvingg de/urn: de bsz 10th3-90225.
Malte Ostendorff, Till Blume, and Saskia Ostendorff. Towards an open platform for legal information. In Proceedings of the ACM/IEEE Joint Conference Digital Libraries in 2020,
14

---

Published as a conference paper at COLM 2024
JCDL '20, pp. 385-388, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450375856. doi: 3383583. 3398616.
URL https://doi.org//e/
Stamatis Outsios, Konstantinos Skianis, Polykarpos Meladianos, Christos Xypolopoulos, and Michalis Vazirgiannis. Word embeddings from large-scale greek web content. ar Xiv preprint 0riit:180000669 2018.
Christos Papaloukas, llias Chalkidis, Konstantinos Athinaios, Despina-Athant AAthansiaa Pantazi, and Manolis Koubarakis. Multi-granular legal topic classification on greek legislation. In Proceedings of the Natural Legal Language Processing Workshop 2021, PP. 63-75, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: /0.48550/ arXiv:2109.15298. URL https: /ttp:i:///xxvv.o/o www..orgg/bbb/geeeeeee 15298.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobei- dli, Alessandro Cappelli, Baptiste Pannierr, Ebtesam Almazrouei, and Julien Launay, The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data only. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 79155-79172. Curran Associates, Inc., 2023. URL httphttt////prooeeddnngg neurips cc/paper files/ /lles/paper/2023/ and httpiaastttssaad.Beehaaarareee Benchmarks.h.....m pdf.
Guilherme Penedo, Alessandro Cappelli, Thomas Wolf, and Mario Sasko. Datatrove: large scale data processing, 2024. URL htttht/tp///tthieie com/ wwwgiinface//dttatee
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. D14-
Linguisiss doi:
URL https: hattpp://aclantholoog) org/
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Alexandra Sasha Luccioni, Yacine Jernite, and Anna Rogers. The roots search tool: Data transparency for Ilms, 2023.
Jan Pomikálek. Removing Boilerplate and Duplicate Content from Web Corpora. Doctoral theses, dissertations, Masaryk University, Faculty of InformaticsBrno, 2011. URL https: //theses. cz/ c2/id/ngo9nn/. SUPERVISOR: prof. PhDr. Karel Pala, CSc.
Murielle Popa-Fabre, Pedro Javier Ortiz Suárez, Benoit Sagot, and Éric de la Clergerie. French contextualized word-embeddings with sip of CaBeRnet: a new French balanced reference corpus. In Proceedings of the 8th Workshop on Challenges in the Management of Large Corpora, pp. 15-23, Marseille, France, May 2020. European Language Ressources Association. ISBN 979-10-95546-61-0 URL https: // /allanthllogyy....oo org/ cm1c-1.3.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprinf, 2019. URL 155007.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganíni, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhigunaa Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean lean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autmme, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger,
15

---

Published as a conference paper at COLM 2024
lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. Xiv e-prints, art. arXiv:2112.11446. December 2021. doi: 10.48550/ 2tt00//axxivv2121142222
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: http:///iiin..ogapppps//////////
074.html
Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E. Ho, and Joel Niklaus. Scale: Scaling up the complexity for advanced language model evaluation, 2023.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20 pp. 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978-1-4503-7998-4
doi: /0.1145/3394486.3444 340644003
Florian Richter, Philipp Koch, Oliver Franke, Jakob Kraus, Fabrizio Kuruc, Anja Thiem, Judith Högerl, Stella Heine, and Konstantin Schöps. Open Discourse, 2020. URL https:
//doi.org/10 http///...910D////// 79KIODW/FIKKIIO
Eva Sharma, Chen Li, and Lu Wang. Bigpatent: A large-scale dataset for abstractive and coherent summarization, 2019.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. 2019.
Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently S2ORC) Dataset. Technical report, Allen Institute for AI, 2023.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnus- son, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv e-prints, art. arXiv:2402.00159, January 2024. doi: 10.48550/ 10445500/arxiv2402.000 arXiv.2402.00159,
Leon Stromberg--Derzzynski. Manuel Ciosici, Rebekah Baglini, Morten H. Christiansen, Jacob Aarup Dalsgaard, Riccardo Fusaroli, Peter Juel Henrichsen, Rasmus Hvingelby, Andreas Kirkedal, Alex Speed Kjeldsen, Claus Ladefoged, Finn Arup Nielsen, Jens Madsen, Malte Lau Petersen, Jonathan Hvithamar Rystrøm, and Daniel Varab. The Danish Gigaword corpus. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 413-421 Reykjavik, Iceland (Online), May 31-2 June 2021. orrg/0211
Linköping University Electronic Press, Sweden. URL nodal iala-mmai.466.
Jörg Tiedemann. Parallel data, tools and interfaces in OPUS. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Dogan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Eighth In- ternational Conference on Language Resources and Evaluation (LREC'12), pp. 2214-2218, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL http:// http://www. wwwc-conf..coo .rg wttpp//rcceeiing///eee/2222222
/663 Paper pdf..
16

---

Published as a conference paper at COLM 2024
Jörg Tiedemann. Finding alternative translations in a large corpus of movie subtitle. In Pro- ceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pp. 351--3522, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https: /ottsho//aclanthoologyy org/L16-1559.
Daniel Varab and Natalie Schluter. DaNewsroom: A large-scale Danish summarisation dataset. In Proceedings of The 12th Language Resources and Evaluation Conference, PP. 6731- 6739, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 URL https: www.wwwwwww.....t.tttwwwwwwwww.w org/ wergganthologyy//// 2020. (rrec-1.83)
Tamás Váradi, Bence Nyēki, Svetla Koeva, Marko Tadić, Vanja Stefanec, Maciej Ogrodniczuk, Bartlomieij Niton, Piotr Pezik, Verginica Barbu Mititelu, Elena Irimia, Maria Mitrofan, Dan Tufis, Radovan Garabík, Simon Krek, and Andraž Repar. Introducing the CURLICAT Seven-language domain specific annotated corpora from curated sources. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 100-108, Marseille, France, June 2022. European Language Resources Association. URL ht htss:///aaathhoogg..ooom org/ 1rec-1.111
corpora:
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning. Xiv e-prints, art 1rXiv:2211.04325, October 2022. doi: 10.48550/ arxiv. 2211.04325.
Jorge A Wagner Filho, Rodrigo Wilkens, Marco Idiart, and Aline Villavicencio. The brwac corpus: A new open resource for brazilian portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality mono-
lingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4012, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-93546-34-4 URL https: httpss///ccaaahhooooooo wrw.oogg///// 22200000.2. 1rec-1.494.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- Natural Language Processing. pp. 38-45. Association for Computational Linguistics, October 2020. URL https://wwi http://wwww..wwwl.l........ org/ wwwgg/nthhoog//220/00 emnlp-demos demos. 6.
Ales Zagar, Matic Kavaš, Marko Robnik-Šikonjp Tomaž Erjavec, Darja Fišer, Nikola Ljubešić, Marko Ferme, Mladen Borović, Borko Boškovič, Milan Ojsteršek, and Goran Hrovat. Corpus of academic slovene KAS 2.0, 2022. ISSN 2820-4042. URL http://hdl, handle net/11356/ 1448. Slovenian language resource repository CLARIN.SI.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilin- gual neural machine translation and zero-shot translation, 2020.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
17

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Usage example
Getting started.
The framework can be installed via the Python Package Index (PyPI):
pip install 1lm-datasets
After installing the Python package, the individual processing steps can be run via the command line. For example, the download and extraction of plain-text of one more datasets can be run with the following command:
1lm-datasets extract_text $DATASET_ID $OUTPUT_DIR [--output_format parquet]
Configuration. LLM-Datasets allows you to specific general settings through config files so you do not need to specific always the same command line arguments. Several commands support passing the -configs argument which should point to one or more YAML-files on your file system. For example, the text extraction command:
1lm-datasets extract_text
--configs $PATH_TO_ COL_LAAALLOOOIGGGFFCCOCC
In the config files, you can store for example system specific settings like the local paths, where the raw dataset files are located:
//examples/lin__aatstttt cttsssettscooffoeeeeeeeeeeoeeees htwp://asssttt...css.......sss. local-dirs byrlldddrssbyyyoooo soy-source... id:
yaml
redpajama: / http:y/ttw.peeeiiiicaaaaaeeeeeee
Dataset selection and sampling. The configuration files are also needed for specifying the final dataset composition, including the selection of the datasets and their sampling. The following examples shows a config for an Italian dataset:
#/ /ltss/lmmmmpa dat wtt.aaastttscooso/iiaaaaaoos htt.aanll///.aaaaaaaaaoooooaaeoa
yaml
fixed random seed for shuffling etc.
seed: 0
selected www.ddaaaaetticeeeee
# italian subsets
itwac
eurlex_it
wikibooks.it
wikinews it
colossal oscar 2023 -23 it parlamint_it
fown-sample webcrawled sp-sampled high quality faatiingffctorbbb.s... bosucccce.idd colossal.oscar: 0.1
sampling factor by dataset id:
itwac: 0.5
eurlex_it: 2
wikipediaa.2231101.:.
18

---

Published as a conference paper at COLM 2024
Language
Estimated Tokens (M)
Percentage (%)
English (en)
1,079,192
46,90
Code (code)
223,834
9.73
German (de)
161,521
7.02
Spanish (es)
102,586
4.46
French (fr)
85,394
3.71
Greek (el)
66,371
2.88
Italian (it)
62,408
2.71
Ukrainian (uk)
55,650
2.42
Hungarian (hu)
51,686
2.25
Polish (pl)
48,640
2.11
Portuguese (pt)
46,726
2.03
Dutch (nl)
45,090
1,96
Czech (cs)
43,286
1.88
Bulgarian (bg)
32,168
1.40
Slovak (sk)
32,145
1.40
Romanian (ro)
20,267
0.88
Swedish (sv)
19, 458
0.85
Finnish (fi)
19,393
0.84
Slovene (sl)
13,726
0.60
Danish (da)
13,604
0.59
Norwegian (no)
13,387
0.58
Serbian (sr)
12,858
0.56
Catalan (ca)
12,582
0.55
Croatian (hr)
10,559
0.46
Estonian (et)
8,759
0.38
Lithuanian (It)
7,239
0.31
Latvian (Iv)
4,719
0.21
Maltese (mt)
3,668
0.16
Mixed (mixed)
2,665
0.12
Basque (eu)
792
0.03
Irish (ga)
274
0.01
Galician (gl)
260
0.01
Serbo-Croation (sh)
184
0.01
Norwegian Nynorsk (nn)
76
0.00
Total tokens:
2,301,165
100.0
Table 3: Language distribution millions of tokens (based on the Bloom tokenizer).
A.2 Language distribution in the European dataset
The European dataset contains 1,138 datasets (including subsets) from 67 sources 32 follows:
languages. The languages English, Spanish, Estonian, Basque, Finnish, French, Irish, Galician, Croatian, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Norwegian Nynorsk, Norwegian, Polish, Portuguese, Romanian, Serbo-Croatian, Slovak, Slovene, Serbian, Swedish, Ukrainian. Tab. 3 lists all languages and estimated number of tokens.
Bulgarian, Catalan, Czech, Danish, German, Greek,
A.3 List data sources in the European dataset
LLM-Datasets implements different data sources ranging from Web-crawled data such Colossal OSCAR to small curated datasets with high quality content. The data sources used in the European dataset are listed in Tab. 4 including estimated number of tokens.
19

---

Published as a conference paper at COLM 2024
Table 4: Available European data sources and their estimated token count.
Title
Description
Tokens
CURLICAT Corpus (Váradi et al., 2022)
The CURLICAT corpus includes monolingual corpora (Bulgarian, Croatian, Hungarian, Polish, Romanian, Slo- vak and Slovenian) containing selected samples from re- spective national corpora.
963 M
MaCoCu web corpus (Bañón et al., 2022)
MaCoCu focuses on collecting monolingual and parallel data from the Internet, specially for under-resourced lan- guages and DSI-specific data. See https:/ https://macocu.eu/ /macoc..u/////////
RedPajama-Data T1 (selected subsets) (Computer, 2023)
An Open Source Recipe to Reproduce LLaMA training dataset
B
pe52o (Soldaini & Lo, 2023)
The peS2o dataset is collection of "40M creative open- access academic papers, cleaned, filtered, and formatted for pre-training language models. is derived from the Semantic Scholar Open Research Corpus(Lo et al, 2020), or S2ORC.
57 B
Auxiliary Mathematics Prob- lems and Solutions (AMPS) dataset (Hendrycks et al., 2021)
pretraining dataset, the Auxiliary Mathematics Prob- lems and Solutions (AMPS) dataset, has problems and step-by-step solutions typeset in LATEX. AMPS contains over 100,000 problems pulled from Khan Academy and ap- proximately 5 million problems generated from manually designed Mathematica scripts. Problems include various aspects of algebra, calculus, counting and statistics, geom- etry, linear algebra, and number theory.
Open Legal Data German court decisions and laws (Os- tendorff et al., 2020)
OPENLEGALDATA.LO a free and open platform that makes legal documents and information accessible to the public.
7
DeWaC
DeWaC is 1.7 billion word corpus constructed from the Web limiting the crawl to the ,de domain and using medium--reequece words from the SudDeutsche Zeitung corpus and basic German vocabulary lists as seeds.
3 B
The Gaois bilingual corpus of English-Irish legislation (Irish legislation)
Bilingual corpus of English-Irish legislation provided by the Department of Justice.
4
Irish Universal Dependen- cies
Universal Dependencies (UD) is framework for consis- annotation of grammar (parts speech, morphologi- cal features, and syntactic dependencies) across different human languages.
40
Croatian web corpus hrWaC 2.1 (Ljubesić Klubička, 2014)
hrWaC is web corpus collected from the .hr top-level domain. The current version the corpus (v2.0) contains 1.9 billion tokens and annotated with the lemma, mor- phosyntax and dependency syntax layers.
B
ITWaC
itWaC: a billion word corpus constructed from the Web limiting the crawl to the domain and using medium- frequency words from the Repubblica corpus and basic Italian vocabulary lists as seeds.
3
Continued on next page
20

---

Published as a conference paper at COLM 2024
Table continued from previous page
Title
Description
Tokens
Korpus Malti (Micallef et al., 2022)
General Corpora the Maltese Language. This dataset is composed of texts from various genres domaans written in Maltese.
816
SoNaR Corpus NC 1.2
The SoNaR Corpus contains more than 500 million words from texts in standard Dutch later than 1954. All texts were tokenized, tagged for part of speech and lemmatized. The named entities were also labelled. All annotations were produced automatically, no manual verification took place.
746 M
Corpus of academic Slovene KAS 2.0 (Zagar et al., 2022)
The KAS corpus of Slovene academic writing consists of almost 65,000 BSc/BA, 16,000 MSe/MA and 1,600 PhD theses (82 thousand texts, 5 million pages 1,5 billion tokens) written 2000 2018 and gathered from the digital libraries of Slovene higher education institutions via the Slovene Open Science portal (http:/ /openscience.s///
3
sIWaC web corpus
sIWaC is web corpus collected from the si top-level do- main 2011 and 2014. The corpus tokenized and anno- tated with the lemma and the morphosyntax layer.
B
od-justice 2.0
Slovak court decisions. The corpus is based data made available by the Ministry of Justice the Slovak Republic.
24 B
Korpus slovenských právnych predpisov v1.9
Slovak body of laws (1955-2022)
105 M
SYN v9: large corpus of writ- ten Czech (Křen et al., 2021)
Corpus of contemporary written (printed) Czech sized GW (ie. 5.7 billion tokens). It covers mostly the 1990-2019 period and features rich metadata including detailed bibli- ographical information, text-type dassification etc. SYN v9 contains wide variety of text types (fiction, non-fiction, newspapers), but the newspapers prevail noticeably.
Danish
GigaWord Sirombber--erczyynnn al., 2021)
A billion-word corpus of Danish text. Split into many sections, and covering many dimensions of variation (spo- hen//witten, formal/infome //noomal,, modern/oll /old, rigsdansk/- dialect, and so The license is CC-BY 4.0, Creative Com- mons with Attribution. Owners: ITU; Leon Derczynski, Manuel R. Ciosici
2
DaNewsroom Schluter, 2020)
(Varab &
A Large-scale Danish Summarisation Dataset
835 M
DK-CLARIN Reference Cor- pus of General Danish
Reference Corpus of General Danish
80 M
CaBeRnet: New French Balanced Reference Corpus (Popa-Fabre et al.2200))
A new balanced French corpus, CaBeRnet, that features representative range of language usage, including a bal- anced variety of genres (oral transcriptions, newspapers, popular magazines, technical reports, fiction, academic texts), in oral and written styles.
599 M
Continued on next page
21

---

Published as a conference paper at COLM 2024
Table continued from previous page
Title
Description
Tokens
Norwegian Colossal Corpus
The Norwegian Colossal Corpus a collection multi- ple smaller Norwegian corpuses suitable for training large language models. We have done extensive cleaning the datasets, and have made them available in common format. The total of the NCC currently 45GB. Docu- ments: 20,830,348; Words/ Words/document. document: 331
B1B
NKJP- TodkorpusMillonowy-- (National Corpus Polish)
A reference corpus of Polish language containing over fif- hundred millions of words. The list of sources for the corpora contains classic literature, daily newspapers, specialist periodicals and journals, transcripts conversa- tions, and a variety of short-lived and internet texts.
3 M
Polish Parliamentary Corpus / Korpus Dyskursu Parla- mentarnego
The Polish Parliamentary Corpus (PPC) is Polish corpus made up of documents from the proceedings of the Polish Parliament, Sejm, and Senate. The corpus includes data of the Polish Sejm corpus and consists of stenographic records of plenary sittings and committee sittings, seg- ments of interpellations and questions. Texts the PPC corpus cover the period of hundred years from 1919 to 2019.
1B
ParlamentoPT
The ParlamentoPT is Portuguese language data set ob- tained by collecting publicly available documents contain- ing transcriptions of debates the Portuguese Parliament. The data was collected from the Portuguese Parliament portal in accordance with its open data policy.
732 M
Brazilian Portuguese Web as Corpus (Wagner Filho et al., 2018)
The BrWaC (Brazilian Portuguese Web as Corpus) a large corpus constructed followingthe Wacky framework, which was made public for research purposes. The current corpus version, released in January 2017, is composed by 3.53 million documents,2.66 billion tokens and 5.79 million types.
B
Bilingual English-Liin LnglishnLihhuanian parallel corpus from Seimas of the Republic Lithuania website
Contents of http:/ /www.lr..t It crawled, aligned document and sentence level and converted into parallel corpus.
12 k
Corpus of State-related con- tent from the Latvian Web (Processed)
Latvian Web, home pages of ministries and state public services, army, etc. were crawled, and parallel Latvian- English content was collected.
52
Greek Legal Code (Pa-
Greek.Legal.Code (GLC) dataset consisting of approx. 47k legal resources from Greek legislation. The origin of GLC "Permanent Greek Legislation Code Raptarchis", collection of Greek legislative documents classified into multi-level (from broader to more specialized) categories.
80
M
paloukas et al., 2021)
Greek Web Corpus (Outsios
A corpus Greek Web used training 'GreekBART: 11
et al., 2018)
The First Pretrained Greek Sequence- Sequence-to-Sequeee to-Sequence Model'
Continued on next page
22

---

Published as a conference paper at COLM 2024
Table continued from previous page
Title
Description
Tokens
Estonian Reference Corpus
This corpus includes Estonian texts (fiction, PhD the- ses, newspapers, magazines, parliamentary transcriptions, monputer-mediated communication) published between 1990 and 2007. The corpus is encoded in TEI. The corpus available for online browsing through a dedicated con- cordancer and is available for download from CELR.
M
Estonian National Corpus
Corpus is based on Estonian National Corpus 2013, which was renewed by Lexical Computing Ltd. 2017 and 2019 at the request of Estonian Language Destitute.Suborrpon are: Estonian Reference Corpus 1990-2008, Estonian Web 2013, Estonian Web 2017, Estonian Web 2019, Estonian Wikipedia 2017, Estonian Wikipedia 2019, Estonian Open Access Journals (DOAJ), blogs, discussion, education, fic- tion, food, health, journals, news, religion, science, sex, society, sports.
B
2021
EusCrawl (filtered: Wikipedia, no NC-licenses) (Artetxe et al., 2022)
EusCrawl high-quality corpus for Basque comprising 12.5 million documents and 423 million tokens, totalling 2.1 GiB of uncompressed text. EusCrawl was built using ad- hoc scrapers to extract text from 33 Basque websites with high-quality content, resulting in cleaner text compared to general purpose approaches.
335 M
Spanish Legal Corpora (Guniérrez-Fandiño al., 2021)
Domain
A collection of corpora
of Spanish legal domain.
1 1B
Yle Finnish News Archive
The corpus, containing the articles from YLE https:// yle.fi from 2019 and 2020, is available at www.kielipankkiiff/ hotoooooakiiiffdddoooddd download
286 M
The Swedish Culturomics Gi- gaword Corpus
One billion Swedish words from 1950 and on- wards. Code to extract data from the corpus, well as usage instructions, can be down- loaded from arkiv akkiv/tools tools/ gigaword/
528 M
https://sn.spraakaatguuee
SrpKorSubset (news, legal, academic, conversation, liter- ary)
The Corpus of contemporary Serbian, SrpKor, consists of 4,925 texts.
866 M
MARCELL Romanian legisla- tive subcorpus v2
The Romanian corpus contains 163,274 files, which rep- resent the body of national legislation ranging from 1881 to 2021. This corpus includes mainly: governmental deci- sions, ministerial orders, decisions, decrees and laws. All the texts were obtained via crawling from the public Ro- manian legislative portal.
1B
Corpus of laws and legal acts Ukraine
A large (more than Gb) corpus of laws and legal acts of Ukraine.
B
BurlexResources
A Corpus Covering the Largest EURLEX Resources.
41 B
LegalMC4
MC4.Legal: Corpus Covering the Legal Part of MC4 for Furopean Languages
Continued on next page
23

---

Published as a conference paper at COLM 2024
Table continued from previous page
Title
Description
Tokens
Wikipedia
The free encyclopedia that anyone can edit.
Wikibooks
The open-content textbooks collection that anyone can edit.
313 M
Wikiquote
The free quote compendium that anyone can edit.
247 M
Wikinews
News written by volunteers.
90 M
Wikisource
The free library that anyone can improve.
Wikivoyage
The free worldwide travel guide that you can edit.
119 M
Colossal OSCAR (Jansen et al., 2022)
The OSCAR project (Open Super-large Crawled Aggre- gated coRpus) is an Open Source project aiming to pro- vide web-based multilingual resources and datasets for Machine Learning (ML) and Artificial Intelligence (AT) ap- plications. The project focuses specifically in providing large quantities of unannotated raw data that is commonly used in the pre-training of large deep learning models.
Starcoder (Li et al., 2023)
The dataset used for training StarCoder and StarCoder- Base. It contains 783GB of code 86 programming lan- guages, and includes 54GB GitHub Issues 13GB Jupyter notebooks in scripts and text-code pairs, and 32GB of GitHub commits, which is approximately 250 Billion to- kens.
212 B
Pile of Law (Henderson" et al., 2022)
We curate a large corpus of legal and administrative data. The utility of this data is twofold: (1) to aggregate legal and administrative data sources that demonstrate different norms and legal standards for data filtering: (2) to collect dataset that can be used in the future for pretraining legal-domain language models, a key direction in access- to-justice initiatives.
34
CATalog 1.0 (without OS-
CATalog is diverse, open-source Catalan corpus for lan- guage modelling. It consists of text documents from 26 different sources, including web crawling, news, forums, digital libraries and public institutions, totaling in 17.45 billion words.
BB
CAR subsets)
Open Discourse Bundestag (Richter et al., 2020)
Open Discourse is the first fully comprehensive corpus of the plenary proceedings of the federal German Parliament (Bundestag).
442 M
Tagesschau Archive Article Dataset
A scrape of Tagesschau.de articles from 01.01.2018 to 26.04.2023. Find all source code in github.com/ waaalb.comboojojee /bjormpo////////////// /tggssscca.......
M
PhilPaperr (2023 version)
A filtered version open access collection philoso- phy publications PhilPapers, data-ready for The-Pile.
M
Project Gutenberg books pub- lished before 1919 (Rae al., 2019)
3 B
Continued on next page
24

---

Published as a conference paper at COLM 2024
Table continued from previous page
Title
Description
Tokens
BigPatent (Sharma et al., 2019)
BigPatent consisting 1.3 million records of U.S. patent documents along with human written abstractive sum- maries.
Proof-Pile-2 (algebraic-stack) (Azerbayev et al., 2023)
AlgebraicStack, an 11B-token dataset of source code from 17 languages, spanning numerical, symbolic, and formal math. The dataset consists of filtered code from the Stack (Kocetkov et al., 2022), public GitHub repositories, and formal proofstep data
Wwvis-Judgmenn-Peediiiee (Niklaus et al., 2021)
twiss-Judgment-e judgment-Prediction Prediction a multilingual, diachronic dataset of 85K Swiss Federal Supreme Court (FSCS) cases
55 M
Swiss Legislation (Rasiah et al., 2023)
Swiss Legislation multilingual, diachronic dataset of 36K Swiss laws. This dataset is part of challenging Infor- mation Retreival task.
227 M
Tatoeba (Tiedemann, 2012)
Tatoeba collection of sentences and translations.
7 M
OPUS-100 (Zhang et al., 2020)
OPUS-100 English-centric, meaning that all training pairs include English on either the source or target side.
901 M
WMT19 (Workshop Sta- tistical Machine Translation) (Barrault et al., 2019)
Shared Task: Machine Translation of News.
1 1B
Opensubtitles (Tiedemann, 2016)
OpenSubtitles.org provides large collection of con- tributed subtitles various languages for movies and TV programs.
30 B
ParlaMint (Erjavec et al., 2023)
ParlaMint 4.0 is set of comparable corpora containing transcriptions of parliamentary debates of 29 European countries and autonomous regions, mostly starting 2015 and extending to mid-2022.
2 B
25


Reviews:
[{"review_id": "NQAtK7F3uQ", "reviewer": "Reviewer_pGJL", "summary": "This work develops a framework to compile and collect pretraining datasets for large language models. They call their framework LLM-Datasets. LLM-Datasets contains an interface that makes downloading, extracting text, filtering, and sampling of pretraining data possible. They also talk about how modular it is: aiding in quick adaptability and extensibility to new data sources. Finally they use the framework to curate a 2.3T token dataset covering 32 european languages to demonstrate efficacy of LLM-Datasets. The framework claims to be HPC-ready, modular, supportive of private data, reproducible, and model agnostic.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the thoughtful comments about our paper.\n\n@Reasons To Reject:\n\nR1:  Our framework is designed with modularity in mind, as such, additional preprocessing steps from other open source projects can be easily integrated. For instance, the next release will include an `LLMDatasetsReader` for DataTrove (a data processing framework recently released by Huggingface) that would allow users to apply all filters from DataTrove on the datasets integrated into our framework. \n\nR2: The value lies in the integration of many different data sources that makes all the datasets available via a unified interface. Users do not need to implement the text extraction logic and cleaning themselves, which is a time consuming but necessary work. Users can instead focus on finding the best dataset composition for their own LLM use case. For our framework, we focused on providing text extraction and filtering for some datasets in underrepresented languages, as such, our framework provides tools to clean and extract texts from some sources that were not previously available on any other framework.\n\n@Questions:\n\nQ1: The value is the reduced effort in data sourcing (select the datasets that are relevant your use LLM use case), integrated datasets come with relevant information (license, language, size, what other models used this data already …). Also, as mentioned above, we focused on multilinguality and we cover dataset and sources that were not previously easily usable by researchers wanting to train LLMs in underrepresented languages.\n\nQ2: The modularity of our framework allows the datasets being processed by other tools, for example, with DataTrove (see R1). Moreover, the intermediate datasets (after text extraction) are stored in common file formats (Parquet or JSONL) that many other tools support, making cross-compatibility easy.\n\nQ3: A hub would be out of scope for our project however on the long run every useful (apart from existing hubs like the one from Huggingface). We primarily want to avoid repeated work and make it easier for researcher to compile better datasets for LLM pretrainingsince we see this as the major weakness of current academic LLM projects. Also as mentioned previously, we also focused on multilinguality and worked on some underrepresented European languages, as we want to provide easy to use datasets for practitioners wanting to develop these technologies in their own languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The problem that the authors are trying to solve is motivated adequately. It's clear to me that as we train more and more LLMs, having a unified, easily accessible and extensible framework for pretraining data is crucial.\n\n2) The framework addresses some of the key aspects that such a framework needs to have, such as modularity and extensibility and private data support.\n\n3) The adoption or integration of already created datasets into the framework is sizable. 2241 datasets including subsets from 62 sources in 164 languages is quite a lot of data!", "reasons_to_reject": "1) The curation and filtering of the dataset ignores some filters that some recent datasets contain such as deduplication and toxicity filters. The authors say that tools already exist for these, but it seems like those were not utilized in the creation of this dataset. Personal information filters and other data contamination filters and things would be good to integrate such as those from Dolma (Soldaini et al. 2024).\n\n2) Its not clear to me how much additional value this framework creates on top of existing toolkits if most of the filters are not native to the framework.", "questions_to_authors": "1) If you were trying to create some ideal pretraining corpus for an LLM using your framework and other tools that exist out there, what additional value add does the current instantiation of your framework have? This is the question that I'm confused about that would really help me evaluate the work more clearly.\n\n2) How easy would it be for the framework to integrate existing toolkits such that you can use the framework to easily create your dataset that uses a specific config that you like on an already existing dataset? For example, if I wanted to use corpus X and use filters A, B, C from some toolkit on corpus X. It'd be awesome if I could press a couple of buttons and the framework would allow me to create that.\n\n3) Is my question 2 beyond the scope of your project, where you simply are looking at creating a reproducible hub basically of datasets that can easily just be used to train LMs?", "extra_scores": {}}, {"review_id": "kLRwH3UDds", "reviewer": "Reviewer_vKjo", "summary": "This work seeks to address an important problem: the lack of public disclosure and documentation surrounding pretraining datasets. It is largely well-written, and the authors have considered several desiderata for such a framework. However, I found that the central motivation of the paper needs to be clarified more carefully (see my comments in questions to authors), and some key claims need to be justified better (such as those surrounding reproducibility of pretraining datasets, see my questions to authors).\n\nEDIT: I am updating my score to 6 on the basis of the author response", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "@Reasons To Reject:\n\nR1: Our audience are researchers who want to conduct open LLM research - not the ones who want to remain closed - and help them make their data work more reproducible by having standardized preprocessing for a large number of datasets. Moreover, we have focused on multilingual content, specially on low-resource European languages. As such we have curated and developed text extraction pipelines for datasets that were previously published only through country-specific platforms, we thus believe that we have made this resource far more discoverable and accessible.\n\nR2: We provide cleaning and text extraction scripts that are reproducible. The norm right now is that most LLM projects tend to use the same data sources but use different processing pipelines, this makes it hard to directly compare two modes that have been trained on the same data source as the actual processed training data might be different. Our goal with reproducibility is that researchers can report what dataset they processed with LLM-Datasets (our framework) and other researchers will be able to get the exact same data.\n\nR3: All datasets were manually inspected for their quality with the provided “dataset viewer”, which enables this. However, we acknowledge that the manual inspections do not necessarily correlate with the “data quality in terms of model performance”. Model training and evaluation are required to truly judge the quality of the data. We consider this out of scope for this paper due to the computational cost associated with training a multilingual LLM, but are working on this already.\n\n@Questions:\n\nQ1: See R1.\n\nQ2: https://pastebin.com/4q5L4nSX (will be added to Appendix)\n\nQ3:  Yes, it is possible to reproduce the dataset with exactly the same documents. See config config above.\n\nQ4: Yes, it is an ongoing effort, we also review and accept pull requests.\n\nQ5: Monolingual data sources are stratified by language, but datasource like Wikipedia or OSCAR are viewed as multilingual. However, our framework also allows to extract specific languages from those multilingual sources, that is, effectively restricting them to one language if the user so desires it.\n\nQ6:  Thank you, we will make the changes for the camera-ready version of the paper, if it is accepted.\n\nQ7: We are currently working on this and we will include some of these more sophisticated quality filters in future versions of the framework. See also R1 for reviewer 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) A centralized framework that makes it easy to construct pretraining corpora is likely to be useful to the research community and I am supportive of the goals of this paper.\n\n(2) The paper itself is fairly clear and well-written.\n\n(3) I appreciate that the authors included a link to an anonymous github repository, which was very helpful in understanding the framework.\n\n(4) The authors use their framework to construct a sizable multilingual pretraining corpus.", "reasons_to_reject": "(1) Some of the main claims of this work are not well-motivated in the manuscript. For example “However, the pretraining datasets of language models and their processing pipelines are rarely shared with the community…In this paper, we address the aforementioned gap”. It is unclear to me how the contributed framework could help facilitate the release of pretraining datasets by entities who do not release them currently.\n\n(2) I am not entirely sure that the pretraining datasets will be truly reproducible (see my question to authors), but will be willing to update this assessment based on the author response,\n\n(3) It would have been nice to get an idea of the quality of the pretraining datasets it is possible to construct using this framework. I appreciate the authors were able to construct a large multilingual pretraining dataset, but it would have also been nice to get an idea of the quality of possible pretraining datasets.", "questions_to_authors": "(1) Could the authors clarify how their framework can improve the disclosure of pretraining data, by entities who do not currently disclose the data used to train their LLMs?\n\n(2) Could the authors please provide more details on what an input configuration might look like?\n\n(3) The authors state that “the framework is designed to produce the identical output dataset as long as the input configuration is identical”. Does this mean that it is possible to reproduce the dataset with exactly the same documents (and not just the correct mixture of tokens from similar sources)? If so, how detailed does the input configuration have to be to support this? If not, could the authors comment on the exact reproducibility of the dataset?\n\n(4) Are there plans to update the datasets in LLM-datasets? How frequently will these updates happen?\n\n(5) Are the different data sources stratified by language or type? For example, are wikipedias of all different languages considered a single data source within the framework?\n\n(6) Minor writing suggestion: in section 4.1, how is ‘code’ in “from the 24 official EU languages and eight additional regional and non-EU languages. Specifically, it covers English, Code” different from “Moreover, the dataset has a diverse set of programming languages, primarily from the StarCoder dataset”. It was also odd to see code in the first sentence in the context of human languages.\n\n(7) Minor suggestion: It would be quite useful to include recipes for standard quality filters within the framework as well, such as for near-deduplication, toxicity and PII (though the authors have justified their decision to not do this in the paper).", "extra_scores": {}}, {"review_id": "EAqfFOSa1h", "reviewer": "Reviewer_WmR4", "summary": "This paper makes the following contributions: \n- LLM-Datasets, a open-source python package to facilitate the creation large-scale pre-training data for LLMs, with unified interfaces to handle the loading and processing of datasets of diverse sources \n- The authors apply LLM-Datasets to create a large corpus of 2.3T trillion tokens covering 32 European Languages.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the thoughtful comments about our paper.\n\n@Reasons To Reject:\n\nR1: The current release of the framework already supports exact deduplication and the next release will integrate all processing steps from Huggingface's recently published DataTrove framework, which also includes near deduplication.\n\nR2: As mentioned in R3@Reviewer vKjo, we consider this out of scope for this paper due to the computational cost associated with training a multilingual LLM, even smaller ones, but are working on this already for a future release of our framework, and we expect support from the open source community in order to perform said ablations in the future. Probably the interesting thing that our framework can offer right now in this regard is the ease of reproducibility and of easily making changes to the data mixture, making ablations far more easier to perform for researchers and users of our framework.\n\nR3: Our framework relies on big-data technologies like parquet, which allows us to make everything as efficient in terms of I/O as possible. We will add the details of the infrastructure we used and the processing times for our 2.3T tokens dataset if our paper is accepted.\n\nR4: Please take a look at the example config (https://pastebin.com/4q5L4nSX) or the anonymized repo linked in the paper, more details will be also added to the appendix if the paper is accepted.\n\nR5: As mentioned above we consider this out of scope for this paper due to the computational cost associated with training a multilingual LLM, even smaller ones, but are working on this already for a future release of our framework, and we expect support from the open source community in order to perform said ablations in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is nicely written and the proposed library is simple and clear.\n\n1. The unified framework in LLM-Datasets (illustrated in figure 1 and 2) can help unify datasets of many different sources to create a larger corpus for training LLMs. The clear characterization of the needed steps can streamline the data preprocessing and makes it easier to adopt. \n2. The proposed dataset artifact can also be a helpful resource to the community, albeit missing some ablation studies/analysis that can help us better understand the dataset as well as how it helps the actual training of an LLM.", "reasons_to_reject": "Albeit the simple and clear design of LLM-Datasets, it seems the library is missing some of the key functionalities for data engineering for LLM pretraining; also the paper misses details about the proposed python library. \n1. **Understanding the overlapping and enabling deduplication among data sources**: Deduplicating training entries in datasets is crucial to the training data quality. In the case of LLM-Datasets, since its primary goal is to mix multiple sources of datasets, it would be helpful if the tool can help identify content overlapping among the sources, and support deduplication.  While in figure 2 and Sec 3  the author mentioned that LLM-Datasets supports the use of external tools to deduplicate data, it does not specifically address the need for cross-dataset duplication analysis and removal, which can be a great/desired component for such a library. \n2. **Supporting ablation of different data mixtures**: another major recent finding is that proper mixture of datasets of different sources can have a major impact in LLM training. While currently LLM-Datasets can help mix datasets of different sources, there is no direct support to help understand which subsets of mixture can lead to desired model performance, which should be an important function for such a library. \n3. **Lack of details about the implementation and computational efficiency**: processing trillion token data involves a lot of compute, memory, and storage. The author barely mentions the underlying implementation of the proposed library (only vague descriptions like “LLM-Datasets is publicly available as a Python package” (page 2) or “We developed LLM- Datasets with the requirements of an HPC system in mind” (page 3)); neither does the author specify the required computation cost (# of CPUs, peak memory, etc) for the created dataset of 2.3T tokens. Having a fast and efficient implementation should be a desired (and perhaps necessary) property of such a library; adding additional discussion (and, ideally, comparisons to libraries like [datatrove](https://github.com/huggingface/datatrove)) can strengthen the paper significantly. \n4. **Examples of the APIs (or actual code) for LLM-Datasets usage**: while the architecture design of the proposed library is clearly documented (in fig 1 and 2), this work does not mention specific use / APIs of LLM-Datasets in detail. Having the python library as the primary contribution of this paper, it’s important to detail not only design but also concrete implementations of the tool in the paper (at least in the appendix).  \n5. **Missing in-depth analysis / ablation studies for the proposed dataset**: this paper details the processing pipeline for the processed dataset, however it is unclear whether the mixture is meaningfully helpful for LLM training – as mentioned above, fine-grained analysis in terms of the data mixtures ratios/duplication among datasets is essential, and ideally one can show the benefits of the dataset via training smaller/proxy LLMs that can help extrapolate the performance at a larger scale.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper the authors propose an open source software framework, LLM-Datasets for collecting pretraining data for LLMs. Their pipeline consists of downloading raw web documents, extracting text, filtering, and subsampling examples, and the authors also include a Streamlit frontend interface for data visualization. They demonstrate the framework by using it to collect a 2.3T token dataset that includes data from 32 European languages. 

Reviewers agreed that the paper was well written, the new dataset represents a useful contribution (though perhaps could use more analysis/study), and that having a unified framework for pertaining data curation could help alleviate some of the challenges in pretraining data curation, notably duplicated engineering effort and reproducibility. Of course, in order for the tool to serve these purposes, it must be highly flexible and easy to use. Reviewers expressed concerns about the maturity of the tool in terms of key functionality such as easily incorporating existing and new filters and data sources, and efficiency optimizations. The authors noted that they are working on adding functionality that would allow for running any filters available through DataTrove, but it remains unclear how easily the framework can be updated to include similar functionality on an ongoing basis.
Decision: Accept