Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
Authors: Pierre-Carl Langlais
Pavel Chizhov
Catherine Arnett
Carlos Rosas Hinostroza
Mattia Nee
Eliot Krzysztof Jones
Irène Girard
David Mach
Anastasia Stasenko
Ivan P. Yamshchikov
Venue: ICLR 2026 Oral (2026)
Keywords: dataset
pre-training
large language models
open data
open science
multilingual
Primary area: datasets and benchmarks
TL;DR: We assemble and release the largest truly open multilingual dataset for LLM pre-training consisting of 2 trillion tokens

Abstract: Large Language Models (LLMs) are pre-trained on large data from different sources and domains. These datasets often contain trillions of tokens, including large portions of copyrighted or proprietary content, which raises questions about the legal use of such models. This underscores the need for truly open pre-training data that complies with data security regulations. In this paper, we introduce Common Corpus, the largest open dataset for LLM pre-training. The data assembled in Common Corpus are either uncopyrighted or under permissive licenses and amount to about two trillion tokens. The dataset contains a wide variety of languages, ranging from the high-resource European languages to some low-resource languages rarely represented in pre-training datasets. In addition, it includes a large amount of code data. The diversity of data sources in terms of covered domains and time periods opens up the paths for both research and entrepreneurial needs in diverse areas of knowledge. In this paper, we present the detailed provenance of data assembling and the details of dataset filtering and curation. We train two small language models on Common Corpus and find that they perform comparably to other models of their size, indicating that our dataset is suitable for multilingual pretraining. Common Corpus represents a key contribution to the ecosystem for open science research on Large Language Models.

Published as a conference paper at ICLR 2026
COMMON CORPUS: THE LARGEST COLLECTION OF ETHICAL DATA FOR LLM PRE- TRAINIG
Pierre-Carl Langlais"
Pavel Chizhov*
Catherine Arnett 1
Carlos Rosas Hinostroza"
Mattia Nee
Eliot Krzystof Jones'
Irêne Girard
David Mach"
Anastasia Stasenko"
Ivan Yamshchikor*
Correspondence to: pierre-carl fr
ABSTRACT
Large Language Models (LLMs) are pre-trained on large data from different sources and domains. These datasets often contain trillions of tokens, including large por- tions of copyrighted or proprictary content, which raises questions about the legal use of such models. This underscores the need for truly open pre-training data that complies with data security regulations. In this paper, we introduce Common Corpus, the largest open dataset for LLM pre-training. The data assembled Common Corpus are either uncopyrighted or under permissive licenses and amount to about two trillion tokens. The dataset contains a wide variety languages, rang- ing from the high-resource European languages to some low-resource languages rarely represented in pre-training datasets. In addition, it includes a large amount of code data. The diversity of data sources in terms of covered domains and time periods opens up the paths for both research and entrepreneurial needs diverse areas of knowledge. In this paper, we present the detailed provenance of data assembling and the details of dataset filtering and curation. We train two small language models on Common Corpus and find that they perform comparably other models of their size. indicating that our dataset is suitable for multilingual pretraining. Common Corpus represents key contribution the ecosystem for open science research on Large Language Models.
FleIAs/comnon_comt corpus
INTRODUCTION
Large Language Models demand large amounts of training data. GPT-3 (Brown et al., 2020) is considered the first "large" language models. Trained on 300 billion tokens, GPT-3 introduced standard training data pipeline shared by nearly all language models to date: large-scale processing of web datasets (45 TB of compressed source data from Common Crawl) and additional digitized sources (Books3). Until 2025, LLM training data sizes have grown logarithmically. The latest generation of publicly documented language models including DeepSeek
(Liu al.) (025), Gemma (Kamath) et al. 2025), Llama 4 (Meta (025) or Qwen (Yang et al. (025) have been trained on 14-36 trillion tokens. Even "small language models" (Wang et al.) (025) rely on large amounts of training data to fit scaling laws: Qwen 3 0.6B was trained on 36 trillion tokens, which is 3,000 times more than Chinchilla-optimal estimates (Hoffmann al. 2022).
As data quality has increasingly been area of concern, the collection, maintenance, processing, and filtering of data has become one of the main costs language model training. But data curation
PleeAs.. Paris, France https://pleias.tt
1 'CAIRO, Technical University of Applied Sciences Work done PleTAs. Now FleutherAT.
Germany

---

Published as a conference paper at ICLR 2026
Other
Englich
Mulli- linguall
LA
English
Eiglish
Jarabittll
Other
Franch
[ana
English
Open Science
OpenSemanis
Open Source
Open Wib
Open Culture
Open Goverament
Figure 1:
Proportional treemap of Common Corpus collections and their most popular languages.
scale may also have hidden costs: negative externalities affecting competing markets, the digital commons, and society at large. While data scraped from the web is publicly available, it is not always in the public domain. Most web data does not have sufficient metadata determine whether it is permissively licensed. NLP practitioners have relied on the protection of fair use, claiming that the transformative nature the use of the data allows them to use this data to train language models. There are increasingly more legal challenges to the use of this data. The New York Times sued OpenAl for copyright infringement, alleging that OpenAl trained their models on NYT articles (Roth 2023 Pope 2024). Due to concerns about indirect commercial exploitation, many rightholders have implemented either hard technical measures or legal provisions against model training. In 2024, it estimated that for Terms of Service crawling restrictions, full 45% of C4 is restricted (Longpre et al. (0024)) and 5% is fully blocked for scraping with a disproportionate impact over quality sources (Longpre et al. 2024b). Restrictions not only affect LLM pre-training but also the quality of search engine indexation and variety of research projects analyzing and collecting content at scale. Even projects dedicated knowledge access have faced significant pressure from Al crawlers and implemented protections that negatively impact access and user experience.
Legal uncertainties have significantly impeded the development of open science research on LLMs. Previously reproducible research artifacts have been removed or taken down, impacting pre-training data, continuous pre-trained models, and evaluation datasets. Books3, which has been used to train numerous models, faced legal challenges (Brittain 2023). The original dataset was ultimately removed due to DMCA take-down (Van der Sar 2023). The LAION dataset was demonstrated to contain CSAM (Birhane al. 2021; Thiel 2023), and taken down (LAION 2023), and then re-released once suspected CSAM was removed (LAION) 2024). The Dutch model GEITje was taken down (Rijgersberg) (025), due to complaints about training the Dutch Gigacorpus, in order to avoid legal disputes. Finally, the widely used benchmark, the Mathematics Aptitude Test of Heuristics removed from Hugging Face via DMCA takedown. All of these artifacts, which were released to further open development and evaluation of language models, were removed suddenly, making previous work unreplicable. These takedowns and legal challenges also represent a sizeable loss of investment for developers, who are often independent or small research organizations.
(MATH) dataset (Hendrycks et al. [202T),
In part as reaction to the use of publicly available but not permissively licensed data, web text is also becoming harder to acquire and use. In an analysis of popular datasets such as C4 (Raffel et 2020), RefinedWeb (Penedo et al. 2023), and Dolma (Soldaini et al. 2024), I Longpre al. (2024b) found that just the last year, 5% all tokens in C4 now have restricted use, with disproportionate number of those tokens coming from the best-maintained, most critical sources. This is largely due to changes in content owners' and hosts' preferences, which are changing to no longer allow scraping, especially for the purposes of training AI models.
Since 2024, several initiatives have emerged to collect open data in English with clear licensing.
This includes: C4C (Habernal et al. 2016); Open License Corpus (Min et al. 2024), 228 billion token corpus from a mix of public domain texts and open source code under free licenses; KL3M (Bommarito al. 2025) 1.2 trillion tokens corpus of administrative texts and structured data
2

---

Published as a conference paper at ICLR 2026
Total number of documents (log-aoali, 10*N)
Figure 2: A schematic world map of languages in Common Corpus with a log-scaled distribution of document counts. For each language, we chose city that is located in the region where this language is most specific to. To avoid outliers, we show only languages with 10,000+ documents.
mostly from the US federal public domain; Common Pile (Kandpal et al. 2025 a data collection of 1 trillion tokens from a variety of recent sources, including a filtered common crawl (Creative Commons Common Crawl). All these projects are monolingual, restricting in effect the reach of language models to the English-speaking audience. In contrast, the most ambitious multilingual collection of permissive content predates Large Language Models: C4C contains 12 million web pages in more than 50 languages filtered by Creative Commons Licenses (Habernal et al.) 2016).
Common Corpus has grown to become the largest fully open pre-training dataset at about 2 trillion tokens and the only one in its size range having high multilingual diversity. Through this release, we show that open LLM research and development is possible while meeting legal and regulatory requirements in compliance with even the strictest Al regulations, such as the European Union. In this paper, we detail the composition of Common Corpus and the process of data collection and curation, and license clearing. Despite its size, Common Corpus is still far from covering the entire range of available resources. We attribute this discrepancy an open data paradox major sources of open content are paradoxically little visible online and even more so the leading pre-training sources. By deseribing the unique challenges coming with the aggregation of large open source, aim to inspire further initiatives. We also train two small language models on our dataset and find that it offers comparable performance to existing multilingual models.
2 ABOUT COMMON CORPUS
When talking about Common Corpus data, we use the word "open" the strongest possible sense. Not only is the data available, but we also provide essential details about the data provenance, data processing, and important information about the contents of each dataset. The Open Source Initiative has also defined open-source AI in terms of openness of use, where open means that is permitted for "any purpose and without having to ask for permission" (Open Source Initiative, 2024). To achieve this, models must be trained datasets that are free from copyright or other legal limitations. This is currently a limitation of existing open datasets for training L.LMs.
Common Corpus, therefore, provides valuable training tokens that will not be subject to the same restrictions. Additionally, the data in Common Corpus are different from other corpora, primarily composed web text. It contains multilingual data in a variety of high- and low-resource languages (see Figure 2 for language distribution), covering diverse genres, time periods, and domains (in Section we detail each part of the dataset). Therefore, it contributes to data diversity in the open pre- rre-training data ecosystem. This is important for developing powerful and generalizable model performance. Common Corpus can be used on its own or conjunction with existing open datasets, according to one's needs and the desired case of a language model.
3

---

Published as a conference paper at ICLR 2026
Common Corpus developed with consideration for developing best practices for open-soure
LLM development (The AI Alliance, 2024 Longpre et al., 2024a Duprieu & Berkouk 2024 Baack
et al. 2025).
We highlight our adherence to those suggested by Baack al. (2025)
Provide useful documentation. We provide information about dataset provenance and processing (Sections and 5) and share key statistics to help potential users understand the applications of the dataset. Dataset documentation improves reproducibility, helps prevent misuse, and aids downstream users to best utilize the dataset (Longpre et al., 2024a).
Follow and record preference signals. the metadata, we include the source URL and license information for the vast majority of the corpus.
Increase diversity and involve local communities to identify relevant data sources. This dataset includes data from a variety of languages, coming from high-quality sources, and the multilingual part was never machine-translated.
Share advancements to foster reciprocity and give back. In addition to the dataset, we release many of the tools we developed in order create the final dataset (Section 5
Do not use openly licensed data without regard for its quality fitness for purpose. In particular, for the dataset in the public domain, we engage in extensive OCR correction and toxicity filtering in order to bring datasets up standard (Section 5
Do not capture highly sensitive data. We remove personally identifiable information from our datasets (Section
Common Corpus aims to support the pre-training of fully open and auditable LLMs by making it legal to release the source even without the provision of fair use. It has been used to create a wider range of language model artifacts, including multimodal datasets, classifiers, synthetic datasets, and benchmarks. Beyond the main dataset, Common Corpus works as open science infrastructure dedicated to the entire lifecycle of language models. As defined by UNESCO, it is a shared research infrastructure that is needed support open science and serve the needs of different communities (Unesco) 2021). We argue this is the first point in time where there has been sufficient knowledge and infrastructure collect and clean a dataset on this scale, which meets the legal and ethical criteria we have outlined.
Common Corpus is composed of six collections: Open Government, Open Culture, Open Science, Open Web, Open Code, and Open Semantic (Figure and in total amounts to 1,998,647, 1198,647,18,, 168,282
tokens The token counts in each collection are listed in Appendix B To highlight the diversity, we visualize the timeline of the collected documents and embeddings of subsample in Figure [3] Each collection comprises multiple datasets, for which we provide details on provenance and other key information in the corresponding subsections. Each data object contains license, language(s), collection or domain of specialization, and other metadata, allowing one to filter out a desired subset. We note that the majority of the data in Common Corpus is in the public domain (see Appendix (B).
Common Corpus is multilingual (Figure 2 and language distribution in Appendix B). with at least 10B tokens for nine languages(] The issues discussed above for making datasets are compounded in languages other than English. Even in relatively high-resource languages like French, these problems are compounded by the fact that there is much less data available, and most tools generalize poorly to languages other than English. Additionally, Kreutzer et al. (2022) showed that many multilingual datasets contain a lot of low-quality or entirely unusable data. Many of the existing datasets they analyzed contained less than 50% of usable text, with 15 sources containing no usable data at all.
3 RELATED WORK
Table illlutrates the similarities and differences between prominent open pre-training datasets and Common Corpus. Multiple other efforts, including C4 (Raffel al. 2220), ROOTS (Laurençon et al. 2022). DCAD-2000 (Shen et al., 2025). and Fine Web 2 (Penedo et al. 2025), contain multilingual data from several distinct domains, but consist mostly of web crawl data with not only permissive
'The token counts are calculated using PlelAs tokenizer trained on a representative subsample of Common Corpus, available at: Ple As/e1e aiassss 35 350mmPPevii Preview
Language distrihution was computed using the Tast language identification model.
4

---

Published as a conference paper at ICLR 2026
Open Qulture
Open Suinne
Open
Searce
Open Source
Syye
Open Wedb
Document Count
p6Ni Comporent
304
//
-40
-40
1800
-10
-40
-31
20
40
Year
5NN Component
(a) A timeline of the main collections with their num- bers of documents in the Common Corpus.
(b) A two-component I-SNE visualization of subsets of Common Corpus collections, C4, and Web.
Figure 3: Temporal and semantic overview of the Common Corpus collections.
licenses. Dolma (Soldaini et al. 2024) ex expands beyond web crawl, but is primarily comprised of English data. KL3M (Bommarito et al. 2025) is offering strictly permissive licensing beyond crawled data, but it is limited to administrative and legal documents in English. Common Pile (Kandpal et al.) 2025) shares Common Corpus's 's commitment to varied data with only permissive licensing, but is English-only. Common Corpus is unique satisfying all four criteria simultaneously: multilingual and multi-domain coverage, data sources beyond web crawls, and fully permissive licensing.
DCAD 2000
Common
Common
KL3M
Dolma
ROOTS
FineWeb
Pile
Corpus
Multidomain
X
V
Beyond Web Crawl
x
Multilinuual
X
X
/
/
Permissive data
X
X
Table 1: Comparison of the contemporary datasets for LLM training.
We also stress the difference in data sources. An investigation the top 1000 domains of Web shows very low overlap with the Common Corpus: less than 2% of pages and 1% of domain names. Within overlapping collections, content can differ substantially: we focused on large PDF sets, whereas agnostic crawl pipelines mostly retrieve HTML pages (e.g., abstracts from scientific papers). This means that Common Corpus mostly adds new, diverse content to the open pretraining ecosystem and significantly differs from crawl corpora, also illustrate in Figure 3b
PROVENANCE
In this section, we present the details about collections that comprise the Common Corpus, accompa- nied by the information about the data sources and the main included languages in Appendix D
4.1 OPEN GOVERNMENT
Open Government is a set of financial, legal, and administrative data in the public domain. In total, the dataset contains more than 407B tokens and comprises two main datasets: Finance Commons and Legal Commons. See Appendix D.I for detailed data composition.
Finance Commons. This is the largest collection of financial documents in the public domain, comprising more than 14 billion words (more than 23 billion tokens). The documents come from wide time range, all the way to 2024. Like many of our other datasets, Finance Commons is also multilingual. Most of the documents languages such as Romanian, Bulgarian, and Latvian. Additionally, this a multimodal dataset. includes more than 1.36 million original PDF documents from AMF and the WTO.. The documents
in English, French, and German, but there also texts in
5

---

Published as a conference paper at ICLR 2026
constitute a wide coverage of in-house layouts and formats produced by industrial and economic sectors. This makes this dataset ideal for developing the next generation of open-data multimodal models. One application for this dataset is to develop vision-language models (VLMs) for advanced document segmentation and processing. These documents also contain vast amounts of structured data, which is also promising area of research that Finance Commons can help drive forward.
Legal Commons. This is a collection of legal and administrative datasets. The datasets come mostly from the EU and the US and cover wide range of languages. These datasets are useful for developing language models with legal knowledge, as well as models that are ideal for document processing in official administrative applications.
4.2
OPEN CULTURE
Open Culture is an aggregation of vast cultural heritage datasets containing both monographs and periodicals for over 13 languages: French, English, German, Spanish, Portuguese, Italian, Dutch, Luxembourgish, Danish, Swedish, Serbian, Czech, and Greek. There are also small portions of data in other languages, such as Arabic, Bengali, Latin, Persian, Russian, Sanskrit, and Urdu.
Composition, large part of Open Culture compiled from Collections As Data (CAD) large dumps of texts, datasets, PDFs, and even raw XML output (METS/ALTO). CAD initiatives considerably simplify dataset aggregation and are a major contribution to the digital commons ecosystem. The remaining sources have been collected on resource-by-resoune APIS and other standard retrieval methods whenever available. The largest extractions of this kind include Internet Archive (about 2 million monographs) and Delpher (50,000 Dutch monographs and periodicals filtered to match the Dutch copyright law for public domain). We managed to compile a large multilingual collection despite such challenges, like poor OCR quality, through the development of OCR correction tools (see Section 5). text segmentation issues, and sometimes irrecoverable deterioration of the original support. For the detailed dataset composition, refer Appendix D.2]
basis using
Licenses. All Open Culture documents are in the public domain, which means their copyright has expired after a given term, and there are limitations on their reuse. For certain content, or cases where we could not rely on the guarantee of established cultural heritage institutions, we implemented our own internal rights verification process. This process follows specific criteria, including author life and data object creation time, and takes into account that we only collected cultural heritage content from institutions based in the US or the EU (see the complete criteria list Appendix E).
Value. Open Culture data is also rich from a cultural and stylistic standpoint and can be used to train multilingual language models with more diverse and creative writing styles. As LLMS are trained on extremely large corpora to maximize next-word prediction accuracy, LLM-generated text can often lack in personality and be boring or generic (Jones & Bergen 2024). This feature of language models stands in contrast with one of their most common uses. In analysis of WildChat (Zhao et al. 2024 a dataset of million user interactions with ChatGPT, Longpre et al. (2024b) found that over 30% of user requests involved creative compositions such as fictional stories, role-play, or poetry generation. At the same time, creative writing is poorly represented among datasets used to train L.Ms,, which mainly comprise web text (Longpre et al. 2024b). Therefore, Open Culture contributes data that can be used to train models for creative writing without violating copyright law. In addition, as many of the Open Culture datasets are historical (coming from the 18th-19th centuries, or even earlier; see Figure Ba), this collection also enables the development of historical language models. The metadata includes the document creation year, which enables researchers to develop language models with a cutoff of the training data creation date.
4.3
OPEN SCIENCE
The Open Science collection includes scientific papers and other documents (theses, book reviews, clinical trials, etc). Following the development of global open access movement, these documents have been made increasingly available in open archives (preprints) or directly through open science publishers and infrastructure. Scientific content has become a primary focus of training data, due to its impact reasoning capacities. Yet, the lack of licensing information has until now partly hindered reuse. The Semantic Scholar Open Research Corpus from Allen AI includes 81.1 million articles in English under an Open Data Commons Attribution License, allowing for the free reuse of

---

Published as a conference paper at ICLR 2026
the aggregated metadata while still acknowledging the remaining copyright of individual authors (Lo
et al. 2020). The Pile incorporated data from arXiv and PubMed Central, also exclusively in En glish Biderman et al.; 2022). Finally, the BigScience project assembled several curated multilingual scientific datasets like the French HAL as part of the training data for Bloom (Scao et al. 2023
The Open Science collection was made possible largely due to the recent development of Ope- nAlex (Priem et al. 2022), the largest open catalogue of scientific documents. OpenAlex maintains an expansive API search engine tracking detailed metadata for each indexed item, including the licensing, as well as link to the original resource, which is generally in PDF format. We filtered OpenAlex on the following three licenses: CC-By. Public Domain/CCO, and CC-By-SA. The largest share of resources is available under CC-By. which is currently the recommended license by the Open Access definition. Open Science also includes smaller subsets, such as a direct extraction of arXiv articles available in CC-By and some Suropean-specie resources not currently well indexed on OpenAlex (the exact distribution of token counts can be found in Appendix DD3)
Due to the specificity of open scientific publishing, the Open Science collection has less linguistic diversity, with nearly 85% of documents currently available in English.
4.4
OPEN CODE
The Open Code collection comprises code data under vast variety of free licenses, which allows NL. practitioners to train models on public domain code for either coding applications or in order to improve certain model performance on natural language reasoning, world knowledge tasks,
mathematics, and structured output tasks (Aryabumi et al. 2025 Petty et 20044 Ma et al. 20024).
The code data we use comes from the Stack v1 and v2 (Kocetkov et al., 2023; Lozhkov et al. 2024).
The Stack yl contains 6.4TB of data and covers 30 programming languages, while the Stack v2 is approximately ten times bigger at 67.5TB and covers over 600 programming languages. All the code data is made available with a direct link to the original resource on GitHub. In total, Open Code contains 283,227, 22,,402,898 tokens (see most common languages in Appendix (D.4).
To prepare the collection, we ran pipeline of varied filters. We first removed files that were not in our desired set of languages and formats according to their file extensions, including SVG files containing mostly encoded shapes, data storage formats: csv, json, jscn5, json isonld, d, and other file types with non-informative content, typically in small amounts: python-t tthhonntaceebck, unity3d-asset, numpy, and http. We then filtered out the licenses to keep only permissible ones. To discard the low-quality data, we ran a series of manual filters described by Lozhkov et al. (2024). In addition those, we removed files consisting of 75% or more of digits, which are mostly files containing raw numeric data. Before the filters, we also replaced sequences of [\r]+)) with An and recalculated line lengths avoid false positives by maximum line length.
4.5 OPEN WEB
In accordance with the general focus of Common Corpus curated content, the Open Web collection currently includes four major web sources:
Wikipedia and Wikisource. Wikimedia projects are popular sources for language model training data due to their reliability, extensive coverage, and textbook- style. Despite this centrality, there is still range of unresolved challenges with the most common versions available for training. The raw source of Wikimedia projects is made available in a specific mediawiki syntax, including a lot of project-specific models, tags, and conventions. The parsing of models is especially not straightforward, as they can either format existing text or remove or include external content (transclusion). As part of Wikimedia Enterprise, the Wikimedia Foundation created entirely new dumps from the rendered HTML sources, which in effect ensure that they include all the text made available to readers.
Youtube Commons*] For YouTube Commons, collected audio transcripts of 2,063,066 videos uploaded on YouTube under standardized CC-By license.
StackExchange. This a collection of user-generated forums and Q&A made available under the CC-By-SA license. We reused the version from The Pile (Biderman et al. 2022).
P1e As As/Youuuue- Commons

---

Published as a conference paper at ICLR 2026
A major aim for the future work will be the integration of web archives filtered by permissive licenses. Since 2016, several projects have attempted to reidentify Creative Commons licenses from web archives at scale including C4C (multilinguull; Habernal et al. 2016), CCCC (English: Dodge et al. 2021), and most recently multilingual Common Crawl Creative Commons Corpus (C5; Vanroy 2025). All these projects struggled with license identification. While license mentions are frequently normalized with a direct link or logo to Creative Commons, there is no guarantee they really concern the entire content: "a blog page contains many photos, and each photo licensed under a different CC-license type, or blog home page with many articles, and each article is licensed under a different CC-license type." (Habernal et al., (016)). We hope this limitation could be overcome by a combination of web domain curation and fine-grained curation and annotation by language model.
4.6 OPEN SEMANTIC
Semantic data is the latest set added to Common Corpus and currently includes only one collection: Wikidata. First created in 2011, Wikidata hosts 100 million documented items and several billion factual statements encoded as RDF triples. It has grown become critical web infrastructure, used ambitious vision
by Google for search disambiguation and currently embodying Tim Berners- for "a web of data". Despite the rising interest in mixed LLM/Knowledge graph methods, Wikidata has hardly been used in language models. The largest initiative to date is Kelm, a collection of 15 million synthetic sentences generated by Google from English-speaking statements (Agarwal et al., 2021). A persistent challenge has been the exclusive availability of Wikidata dumps under formats optimized for data exchange rather than language model training.
Thanks to collaboration with Wikimedia Deutschland, the entire set of Wikidata has been adapted in natural language and added to Common Corpus. This is to date the only available textual collection of Wikidata covering the entire range of 300 languages. Data processing involved the translation of items and properties into formal language sequences as simple natural language sequences, without textual synthesis: "Q41309 P:27 Q171 150" becoming "Franz Liszt country of citizenship Kingdom of Hungary." " Within each entry, we provide all the available translations as consecutive blocks separated by newline, anticipating that this may contribute to language alignment.
5 CLEANING AND CURATION
In order to curate our dataset, we developed a number of custom tools to handle the issues unique to multilingual, historical, and OCRed data. We document the tools in detail in Appendix [F]
Text Segmentation. We developed Segmentext, a specialized language model for text segmentation (see example in Appendix F.I). Segmentext has been trained to be resilient to broken and unstructured texts with digitization artifacts and ill-recognized layout formats. Given the diversity of the training data, Segmentext should work correctly on diverse document formats in the main European languages.
OCR Error Detection We developed two pipelines to determine the digitization quality of different datasets in order determine the appropriate treatment before datasets can be used for pre-training. The first is OCRoscope. This is a tool based on the language identification tool cld2 (Al-Rfou 2013), which identifies documents with non-recognized 7 7-grams provide measure of OCR quality for at least 80 languages (for further details, see Appendix F.2). The other is OCRerrer. This is a DeBERTa-v2-style language model with 400M parameters, which is trained to label OCR errors. This enables error rate estimation and supports OCR error correction. OCRoscope tends to have lower accuracy, especially for documents with fewer OCR errors. However, it is less computationally expensive and is faster, which allows it to be better used at larger scales. OCRerrer, on the other hand, achieves higher accuracy, but is more computationally intensive.
OCR Correction. We developed OCRonos model based Llama 3 8B (Grattafiori et al. 2024). supports the correction of OCR errors, cutting or merging of the wrong word, and overall broken text structures. The training data includes highly diverse set of -CR-ed texts in multiple languages, mostly from uncorrected versions of Open Culture and Open Government. On highly deteriorated content, OCRonos act as synthetic rewriting tool rather than a strict correction tool (e.g.,
'All described tools are publicly released:
Ple.AAs/ SellAs//eementexx
P P As OCRosccp
P1 As/ /OCkerror
Ple As /OCRoos
Ple AAA/cs/ ladon
8

---

Published as a conference paper at ICLR 2026
Character Repetition
Type- Ratio
Top Wood Propoction
Dresity
Desiit
010
1.0
0.0
0.4
00
0.4
Alphanumenic Characters
Whitespace Characters
Uppercase Characters
Demiiy
Denny
000
022
0.4
06
0.5
1.0
0.0
0.2
0-4
066
0.8
1.0
0.0
012
01.
0.6
04
1.0
Open
Semandii
Open Source
Open
Open Culture
Open Covemment
Figure 4: Stacked histograms of probability density for qualitative evaluations of Common Corpus on sample of 300,000 documents. Metric descriptions can be found in Appendix G
Appendix F:4
OCRonos is generally faithful to the original material, provides sensible restitution of deteriorated text, and will rarely rewrite correct words. However, when original PDF sources are too damaged for accurate OCR, unavailable, or otherwise difficult to retrieve, OCRonos provides a practical alternative for recovering usable text.
PIT Removal. Personally Identifiable Information (PII), i.e., any information that can be used to distinguish or trace an individual's identity, is protected under legislation such as GDPR. Consequently, the new regulations put restrictions on LLM training data. In large open datasets, there is staggering amount of personal data widely used datasets, e.g., large quantities of phone numbers in RedPajama, email addresses in S2ORC and peS2o, and IP addresses in the Stack (Elazar et al. 2024). To identify and replace PII, we use Microsoft's 'ss Presidio (Mendels et al. 2018). an open- state-of- the-art art tool. With Presidio, we filtered out phone numbers, email addresses, IBANs, IP addresses, and URLs. With the base settings, Presidio identified on average 55-60% of texts that included phone numbers due to different possible number formats. By applying custom regular expression patterns that include most phone numbers, we increased this accuracy to 85%. Typical methods of handling PII include removing it, replacing it with tags, and partial anonymization. These transformations substantially alter the format of PII, which could undermine the model's 's understanding of the text or interfere with its ability process text with real PII. Instead, we replace PII with fictitious but realistic values.
Deduplication. Our early experiments showed a negligible rate of duplication, which we attribute to the initial data curation: large institutions are incentivized to avoid re-digitizing the same texts. also filtered out duplicates based on PDF metadata and used deduplicated sources wherever possible.
Toxicity Detection. addition to posing legal and regulatory issues, web data is major source of harmful and biased content (Common Crawl was shown to contain sexual content, hate speech, and racial and gender biases (Luccioni Viviano, (02I]) and often suffers from low-quality and machine-generater text (Dodge et al. 2021). Public Domain data, such as that in Open Culture, comprise historical periodicals and monographs from at least 80 years ago. As cultural norms have changed dramatically, many of these texts do not meet modern ethical standards. Training language models on these texts would lead to the reproduction and circulation of harmful language. To address this, we developed pipeline filter the public domain training data. We created multilingual toxicity classifier, Celadon, DeBERTa-v3-smal model 1~~000M parameters), trained from scratch on 2M synthetically annotated samples. Celadon and the training dataset were released as parts of separate work (Amett et al. 2024).
6 EVALUATION
We evaluate sample from Common Corpus using various qualitative metries and present the distributions in Figure 2 and provide details on metrics in Appendix Most data objects present high peaks at similar proportions in the expected range. Specific collections exhibit expected deviations:

---

Published as a conference paper at ICLR 2026
code data has generally higher proportion of repetitions and a lower proportion of whitespace characters due to strict syntax and the presence of punctuation, while Open Government data presents lower linguistic diversity due to fixed terminology and expressions.
We train two models on Common Corpus: PleIAs 350M and PicIAs I.2E The architecture is based Llama. We train a custom Llama-style tokenizer with vocabulary size of 65536 representative subset of Common Corpus. PleIAs 350M is trained on a filtered subset of Common Corpus, amounting approximately IT tokens. PlelAs 1 1.2B is trained on Common Corpus and three epochs of the filtered subset. The models were trained for 2944 and 23040 H100 hours, respectively.
We evaluate our models MultiBL Jumelet et al.
XStoryCloze (Lin et al. 2022), and XCOPA (Ponti et al. 2020] see Table 2 for aggregated scores and Appendix G for per- language details). All evaluations were run using the LM Evaluation Harness (Biderman al. 2024). Our models perform comparably to models trained on closed or non-permissively licensed data, and show outstanding performance on MultiBLIMP, which has more languages compared to other benchmarks. This is especially notable for our 350M model, which we compare to bigger models; it also outperforms models from the IB range, except for Gemma 3 IB. Our models stably outperform OLMo IB, which was also pre-trained on publicly released dataset.
2025
Ours
Gemma 3
XGLM
BLOOM
Ours
Gemma 3
XGLM
OLMo
Model
350M
270M
564M
560M
1.2B
1B
1.7B
1B
MultiBLiMP
0.774
0.762
0.711
0.683
0.797
0.799
0.710
0.699
XStoryCloze
0.509
0.533
0.537
0.532
0.526
0.594
0.569
0.517
XCOPA
0.533
0.544
0.550
0.541
0.541
0.593
0.574
0.518
Table 2: Benchmarking results. "Ours" refers to PleIAs models pre- pre-trained on Common Corpus.
7 CONCLUSION
Through the release of Common Corpus and this paper with thorough documentation of data collection and curation, we show that LLM development is possible while strictly adhering to the regulatory norms. While Common Corpus is only large enough to train small models currently, the tools and methods we used to identify and curate the data may be used to expand the amount of permissively licensed open data. We hope that Common Corpus will grow as a critical infrastructure for open science LLM research and development and inspire future initiatives in the open.
LIMITATIONS
Common Corpus far from collecting the whole range of available open data, which we described as the open data paradox. Therefore, the future collection of permissible data is highly encouraged by this work. Furthermore, the collected amount of data (2 trillion tokens), when used alone, as our own small language model family (see Section (6) is suitable for pre-training of models of limited size, while larger ones require significantly larger amounts of data. In addition, Common Corpus naturally does not contain data for instruction-tuning and any forms of specialized tasks. Therefore, it is not directly suitable for task-specific fine-tuning However, due to the multilingual, temporal, and semantic diversity of data, Common Corpus opens the opportunities for the creation of ethical fine-tuning datasets.
In Section 5] we described the tools we used for the data curation, filtering, and editing. Although we used these methods responsibly and mitigated many issues overlooked by the counterparts (e.g., with toxicity detection), none of the curation methods could naturally facilitate hundred percent accuracy. However, some issues, like OCR errors, present considerable challenges to the models and might even account for better handling of typos in the future. We would also like mention that each data object is accompanied by sufficient metadata, and, if desired, LLM practitioners are free to filter out collections that might contain potential issues (as described in Section 5
Repositooiis:
Ple ISeAA//AA As/Pleiassss.. 3500m-350mmmmmmmm Preview
#TelAs/Peeias 1 2b- Preview
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
Common Corpus is part of the Current AI initiative to accelcrate the development of the building blocks of open AI - notably data - that serves and is shaped by the public interest. It was built up with the support and concerted efforts of AI Alliance, the state start-up LANGU:IA (start-up E'Eaa),, supported by the French Ministry of Culture and DINUM, as part of the prefiguration of the service offering of the Alliance for Language Technologies EDIC (ALT-EDIC). The creation of the OpenCulture datasets was made possible by the support of LANGU:IA The Wikidata and Wikipedia datasets were made in partnership with Wikimedia Enterprise and Germany. The corpus was stored and processed with the generous support of Genci (Jean-Zay, Idris, Eviden), Scaleway, and Tracto AI. The collection of the corpus has been largely facilitated thanks to major organizations committed to an open science approach for AI, namely AI Alliance, Mozilla, HuggingFace, Occiglot, and Eleuther AL.
REFERENCES
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Fnhanced I Language Model Pre-training. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cot- terell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3554-3565, Online, June 2021. Association for Computational Linguisties. doi: 10. 10.18653/v1/2021 mamacclmmin.2777 278. URL http::httsss/cccce httpt:::s//allattoloogg.oooo air. 278/
2021 naacl-m
Rami Al-Rfou. pycld2: Python bindings for the Compact Language Detector 2, 2013. URL https::/gitte com/ coom/aboSamoor/yyllddd Python bindings for the CLD2 C++ library by Dick Sites.
Catherine Arnett, Eliot Jones, Ivan P Yamshchikov, and Pierre-Carl Langlais. Toxicity of the Commons: Curating Open-Source Pre-Training Data. arXiv preprint urXiv:2410.22587, 2024.
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadace, Ahmet Üstün, and Sara Hooker. To Code or Not To Code? Exploring Impact of Code in Pre-training. In The Thirteenth International Conference on Learning Representations, 2025. URL
Stefan Baack, Stella Biderman, Kasia Odrozek, Aviya Skowron, Ayah Bdeir, Jillian Bommarito, Jennifer Ding, Maximilian Gahntz, Paul Keller, Pierre-Car Langlais, et al. Towards Best Practices for Open Datasets for LLM Training. arXiv preprint arXiv:2501, (55.:2501000355,, 2025.
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the Pile, January 2022. URL httt:://axxlv.orr//bb///20202 07311 arXiv:2201.0331 [es].
Stella Biderman. Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the Trenches Reproducible Evaluation of Language Models. arXiv preprint arXiv:2405. 14782, 2024.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes. arXiv preprint / 2021.
Michael J. Bommarito, Jillian Bommarito, and Daniel Martin Katz. The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models, April 2025. URL http: // a faaxiiv wwt.:o/g//abaaaaa
07854 arXiv:2504.07854 [cs].
Blake Brittain. Authors sue Meta, Microsoft, Bloomffr latest Al copyright olopp Reuters, October 18 |tt.a///4ea///////araaaaaaaaaaaa sta-mi ott.

---

Published as a conference paper at ICLR 2026
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL http:: URL/Etppss//pprcceeiinns urips. cttpsscccpppper 2 2F64a-Paper.pet
Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. Large- Scale Multi-Label Text Classification on EU Legislation. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6314 1-22222 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1636. URL / / anthol www....g........ 636
Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad. Kenneth Heafelld Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffmaa,, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shan- non L. Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm' an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Saftyaah Saleem, Holger Schwenk, and Jeff Wang. No Language Left Behind: Scaling Human-Centered Machine Translation. ArXiv, 0id/2207.046722 2022. URL https://aph. semant www.mmnnniichh..oooo icscho ar.
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v http:///2222.mmmlmm main.98. URL https://aci antho ogy. oro. wwwnmpmmmee ma i n. 98
Henri Duprieu and Nicolas Berkouk. Techniques d' d'audit des grands modèles de langage. Technical report, Commission Nationale Informatique et Libertés (CNIL), November 2024. https/
// /ha www.sciinnneaaa-00082222211
Yanai Elazar, Akshita Bhagia, Ian Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. What's In My Big Data? In The Twelfth International Conference on Learning Representations, 2024. URL https: www./epeevvvieeee/ee ew net. fitorrummii-vve
nOkPV4
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chem, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruy Mahajan, Diego Garcia-Oland Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, et al. The Llama 3 Herd of Models, 2024. URL https: wwtp:iiwwwrgga/////..oo http:////222220221////
Ivan Habernal, Omnia Zayed, and Iryna Gurevych. C4Corpus: Multilingual Web-size Corpus with Free License. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko
12

---

Published as a conference paper at ICLR 2026
Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pp. 914 922, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https::tp///.. / / www.hhhloggyorr/1116111111 6146//
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Geooding-Enhancee BERT with Disentangled Attention. In International Conference on Learning Representations, 2021.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradiemt-Disentangled Embedding Sharing, 2023, URL
https://axxi> org/ http:/bs//2// /alss//111.. 09543
Peter Henderson, Mark Simon Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel E. Ho. Pile of Law: Learning Responsible Data Filtering from the Law and 256GB Open-Source Legal Dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL potttps://ppereeees
nww.:wn..eeeeeeeeeeeeeeeee www./fourffrrto ?
CT N
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fisth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httl:tttts///eeeeetii//
2mQsCe
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models, March 2022. URL wwt.hhtp:/// pestp://arxisn aarxiv..g.gg../........./....... arXiv:2203.1 15556 [cs].
abs/2203 15556
Cameron Jones and Ben Bergen. Does GPT-4 pass the Turing test? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume Long Papers), pp.
Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1865386//vll 1/2024. 290. htt.tt /acc/acaaaac http://aaaaaaaaooo..ooooooo org/2 024. nwac long. 290
Jaap Jumelet, Leonie Weissweiler, and Arianna Bisazza. MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs, 2025. URL
2768
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Fyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov. Surya Bhupatiraju, Rishabh Agarwal, Mchran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky. Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György. André Su- sano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, al. Gemma Technical Report, 2025. URL wwwp:/wwwww..//////......./
Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Ab- basi, Luca Soldaini, Enrico Shippole. A. Feder Cooper, Aviya Skowron, Shayne Longpre, Lin- tang Sutawika, Alon Albalak, Zhenlin Xu, Guilherme Penedo. Loubna Ben allal, Elie Bakouch, John David Pressman, Honglu Fan, Dashiell Stander, Guangyu Song, Aaron Gokaslan, John Kirchenbauer, Tom Goldstein, Brian R. Bartoldson, Bhavya Kailkhura, and Tyler Murray. The
13

---

Published as a conference paper at ICLR 2026
Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text. In The Thirty- ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. B httttp:://ppeneeeee htt.://wwwwweeeeeeeeeee.ee.eee Eor.mmid-
Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jemite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. The Stack: TB of Permissively Licensed Source Code. Transactions net
on Machine Learning Research, 2023. ISSN 2835-8856. URL
7 /forum?io
[oorum?id-pppSTaEEpp pxpbTdUEpD)
Philipp Koehn. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of Machine Translation Summit X: Papers, pp. 79-86, Phuket, Thailand, September 13-15 2005. URL https: / /ac///cccc//// antho ogy. org/ 2005. mt t-paperp...........
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orsikkh Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Pa- padimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the As- sociation for Computational Linguistics, 10:50-72, 2022.
doi: 10.1162/tacl URL
tacl-1.....11.4..
LAION. Safety Review for LAION 5B, December 19 2023, URL Flaion-maintenanne
LAION. Releasing Re-LAION 5B: Transparent Iteration on LAION-5B with Additional Safety Fixes, August 30 2024. URL https::t/t/i http://////l..o..a//lloogo////// 10n-://
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMiilan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaich Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Vil- legas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak. Minh Chien Vu, Itziar Gonzalez- Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepereq. Suzana Ilie, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The Big- Science ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 31809-31826, Curran Associates, Inc., 2022. URL ottps::/prroceccngg.. neur paperr http:///aeeeelleeeeppee///////// 2
wttpppee//22//1ii1e wtt.2////////////// 9e9 and
hannDDafaaatttttannnnnnenneaaaaa www.e/eennhhhheeeeeeeaaeee pdf
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishray Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot Learning with Multilingual Generative Language Models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9019-9052, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 1/2022.e www.lllotttoototoooooooooooooo org 2022. emn p- 6167
mnlp-main.616 URL httpt: /ttttpp://cclartthooooy..oo.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen,
14

---

Published as a conference paper at ICLR 2026
Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, J. Chen, R. L. Jin, Ruiqi Ge, et al. DeepSeek- V3 Technical Report, 2025. URL https::/tppp.ttieeeee
org/ abs/ /ttp/2122.110 http/////a/////1593133... 19437
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC:
The Semantic Scholar Open Research Corpus. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969-4983, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl- maieel4-aii.447...
URL httpsss::
2020
Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Ilharco, Nay San, Maribeth Rauh, Aviya Skowron, Bertie Vidgen, Laura Weidinger, Arvind Narayanan, Victor Sanh, David Ifeoluwa Adelani, Percy Liang, Rishi Bommasani, Peter Henderson, Sasha Luccioni, Yacine Jernite, and Luca Soldaini. The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856. URL lattss://openeeeveew.cco Survey Certification.
or sxum?id=thidod
Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu. Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Mustafa Anis, An Dinh, Caroline Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnic Liang. Mohammed Hamdy, Niklas Muennighoff, Sconghyeon Yc, Seungone Kim, Shrestha Mohanty. Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Sandy Pentland. Consent in Crisis: The Rapid Decline of the Al Data Commons. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information P Processing Systems, volume 37, pp. 108042-108087 Curran Associates, Inc., 2024b. doi: 10.5220020079017-33333 [gtp:///aaaaaiiaaaaae///////1112
URL
Track pdf
Lefteris Loukas, Manos Fergadiotis, Ion Androutsopoulos, and Prodromos Malakasiotis. EDGAR- CORPUS: Billions of Tokens Make The World Go Round. In Udo Hahn, Veronique Hoste, and Amanda Stent (eds.), Proceedings of the Third Workshop on Economics and Natural Lan- guage Processing, pp. 13-18, Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. doi: (..18653/v1/2221 weconlpp.e..ccplpe....cc 1.2. URL https wot.p//o....rgg/.......oooooo/..
Eneldo Loza Mencía and Johannes Fürnkranz. Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain. In Semantic Processing of Legal Texts. Springer, 2010.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu. Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry. Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder 2 and The Stack v2: The Next Generation, 2024. URL WtRl.tttss: / .rg./////.oo abs 2402. 11.442..1911 19173
15

---

Published as a conference paper at ICLR 2026
Alexandra Luccioni and Joseph Viviano. What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus. Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 182-189, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.ac1-short.24 /tt.st/sss///eaaattooooooo/////p achtsssst......ssttt..oo.o......
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At Which Training Stage Does Code Data Help LLMs Reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL htt.ttp:///penzeieew..... http://wwww.e.......eeeeee
?id- KIPJKST4gW
Omri Mendels, Coby Peled, Nava Vaisman Levy, Sharon Hart, Tomer Rosenthal, Limor Lahiani, et al. Microsoft Presidio: Context aware, pluggable and customizable PII anonymization service for text and images, 2018. htttttts:://mnt crosoft..ooooooooooooooooocooo.. thub. io www.s/siiporre.i.oo.o.o..
Meta, The Llama 4 herd: The beginning of new era of natively multimodal Al innovation, 2025. httph:://// //ai://aii.aa mett.......m com www/bloo//////oocccoooooooooooot hatgg//llaaaaaaa hto-mmmmmmmmmmimmmmeeee 1iigeegeg////
Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah A. Smith, and Luke Zettlemoyer. SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore. In The Twelfth International Conference on Learning Representations, 2024. URL https
//openreview net feretfforoee totee//oormmiid-oooonyye
Clemens Neudecker. An Open Corpus for Named Entity Recognition in Historic Newspapers. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.). Proceedings of the Tenth International Conference on Language Resources and Evalua- tion (LREC' pp. 4348 4352, Portorož, Slovenia, May 2016. European Language Resources org/ 116-1689
Association (ELRA). URL https:
Open Source Initiative. The Open Source Al Definition 1.0, 2024. URL https: https://opensqu /ppensour Accessed: 2024-1 11-20.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only. Advances in Neural Information Processing Systems, 36:79155-79172, 2023.
Guilherme Penedo, Hynek Kydlūček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hos- sein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. Feb2: One Pipeline to Scale Them All Adapting Pre-Training Data Processing to Every Language. In Second Conference on Language Modeling, 2025. URL jnn2iid=jRBBe2aa.
Jackson Petty, Sjoerd van Steenkiste, and Tal Linzen. How Does Code Pretraining Affect Language Model Task Performance? In The 7th BlackboxNIPP Workshop, 2024. URL hww.::tppppp..... :/tt::s:///perre
view. net/ (tt.rmmmiddsgghhllla..
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulié, and Anna Korhonen. XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2362-2376, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.11863/v11/ 1/2020 wtt.:20.emllpmmaaii 185. URL https //acllantholooy.orrcc//o
wmw..mmmmlppmmmm.mm 1857
Audrey Pope. NYT OpenAl: The Times's About-Face, Harvard Law Review Blog, April 2024. imess- imess-about-face.
Jason Priem, Heather Piwowar, and Richard Onr OpenAlex: A Fully-Open Index of Scholarly Works, Authors, Venues, Institutions, and Concepts. arXiv preprint arXiv:2205.01833, 2022. URL
https://arxiv..or http://wwwwwworroro////ooooss///
01333
16

---

Published as a conference paper at ICLR 2026
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
Edwin Rijgersberg. The end of GEITje. GoingDutch.ai, 2025. URL htLLttss///ggigdur.... Accessed: 2025-02-20.
h/en/posts/geitjee jeakedew takedown/ /ackk
Emma Roth. New York Times sues OpenAl and Microsoft over copyright infringement. The Verge,
December 2023. httpp///tt////////ooooo//oo//aaa
tyok-times- openaa wateenaammmmcoosooo wtw.:/oototwwwtttttttttooeeeoee- -ow-attooo.oo..ggtt--i----
Lwwuit
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexan- der M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan- MMjorr, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Mar- garet Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisherg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang. Jörg Frohberg, Joseph Tobing, Joydeep Bhat- tacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, et al. BLOOM: A Parameter Open-Access Multilingual Language Model, 2023. URL https://arxxt /ttp:///axxivio wttp:/iiiorrro.ooooo httpabbb///11..111
Yingli Shen, Wen Lai, Shuo Wang, Xueren Zhang, Kangyang Luo, Alexander Fraser, and Maosong Sun. DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detection. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL BRl.https:// http:s:///p/eeevveeeeeeee ew, net/forun
?id Hld-Hooywh222V
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannanch Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 15725-15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/w112022..ooo acl-ooo 180. URL 84C/
The AI Alliance. Dataset Specification, 2024. Utt.nttts://hhe--iaalll hithub ot /eenrr/ee/eetttttteeeeeeeeeee//r Accessed: 2025-02-20.
David Thiel. Identifying and Eliminating CSAM in Generative ML Training Data and Models. Technical report, Stanford Digital Repository, December 20 2023. /tt/httss:/pprrl..e
nford. effordeedd/kh/522mm122
Unesco. Recommendation on Open Science, 2021. URL http::///www.unseoo.o/g///////// gal wtt.://rrroooemmmtaaiiiiiiooo.. wewnmn-ccennneee ence
17

---

Published as a conference paper at ICLR 2026
Ernesto Van der Sar. Anti- Anti-Piracy Group Takes Prominent AI Training Dataset "Books3" Offine
TorrentFreak, August 16 2023. [tt.////wr/.oorrraaaaaaaa.aaaaaa
Bram Vanroy, CommonnCree-CrrrrriiiCoooooooss February 2025. URL URI.https://ithhbb.co
Chengyu Wang. Taolin Zhang, Richang Hong, and Jun Huang. A Short Survey on Small Reasoning
Models: Training, Inference, Applications and Research Directions, April 2025. URL https
//axxiv.org/abs/25500 /ttp://aab/200000100001 09100v1
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 Technical Report, 2025. URL https: /ttp:///aaavv..o www.aawwwww...rgaaaaaaaaaoee www.oorgggbbb..................e 2505 09388
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: IM ChatGPT Interaction Logs in the Wild. In The Twelfth International Conference Learning Representations, 2024. http://pppp/opeeeeeeeeeeeeeeeeee
18

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
the process of developing this work, we utilized LLMs for grammar correction and occasionally as rewriting tool. In addition, involved LLMs in the process of data visualization.
B CORPUS COMPOSITION
The documents in Common Corpus have the following structure:
identffeed unique text identifier. many cases, this also links to the original resource.
open_type: one of the six leading collection groupings.
collection: name the sub-collection (Appendix D).
license: details on the content licensing.
date: date of creation of the resource, where known.
title: title of the resource, when known, alternatively, the filename.
creator: institution that published, collected, curated the resource.
lanquage: automatically identified language.
word.count: number space-delimited words.
token_count: number of tokens calculated by PleIAs tokenizer.
text: full text, without formatting.
In Table
we present the token, word, and document counts for the Common Corpus collections.
Dataset
Documents
Words
Tokens
Open Government
75,652,998 998
257,561 1816,,830682
407,067,5544 1554,1899
Open Culture
93,156,602
549,608,766 763,887776666
885,982,490.090
Open Science
19,220,942
147.305,783,44
281,193,563,77
Open Code
202,765,051
77.669,169
(16669,169,999 69,0
283,227,403 802,898
Open Web
96,165,348
33,208,509 ,065
73,217,485, 489,489
Open Semantic
30,072,707
23,284,201,78%, 2722
67.958,671,88
Total
517.033,648
1.088.638,258,044
1,998,647,166 1185644,118,,88
Table Dataset composition of Common Corpus. For each collection, we report the total number of documents, words cuhitespace-separateeee and tokens.
In Table we present the top ten licences of Common Corpus documents.
License type
Tokens
Public Domain
1,138,508, 5338,508,775 3,,75,955
CC-By
287,749, 2,444,246,4577
MIT
442,694,227, 22776777
CC-By-SA
74,768, (66668866088866
Apache-2.0
68,750,977, 7,037
OSD-3-Clause
18,483, 944,333
Open license
10,432.513,767
BSD-2-Clause
5,497,145,480
CC-BY-4.0
2,110,966,243
CC0-1.0
1,877,206,195
Table 4: Token counts for the ten most common licenses in Common Corpus.
In Table 5 we present the top-50 languages in Common Corpus by token count. The token counts presented in tenms of our BPE tokenizer used to train the models described in Section which was trained on a representative subsample of Common Corpus. To verify that our tokenizer serves as strong baseline for token counts, show its fertility in Appendix C
19

---

Published as a conference paper at ICLR 2026
Language
Documents
Words
Tokens
English
154,175,907
634,794,970,595
968,757,721,747
French
35,245,624
162,061,620,874
275.358,437,630
German
11,385,377
56,674,819,173
112.127,458.251
Spanish
6,530,094
26,215,767,271
46,514,142,421
Latin
2,367,110
16,189,444.325
36,031,551,540
Italian
3,804,052
13,207,129.356
24.681637,,55
Polish
2,640,613
5,086.555,167
12,146,688, .69
Greek
844,122
3,796.018,483
11,376,498.055
Portuguese
1,756,922
5,234,373,473
10,262,747,944
Russian
2,762,818
3,222.919.854
9,439,453,633
Dutch
3,206,382
3,791,928.728
8,058,934,00 080
Danish
2,270,459
2.840,121,206
6,941,822 9827,931
Slovak
683,174
2,320,831,403
5,148,967,88
Czech
946,534
1,966,829,784
4,798.558.09
Indonesian
1,023,361
1,660,129,567
4,381,878,82
Estonian
685,180
1,613,093,565
4,379,534.617
Hungarian
888,780
1,527,981,88
4,110,87,,72
Swedish
3,250,289
1,782,642,556
4,014,927,88
Finnish
931,201
1,356,653,003
3,943,036,413
Maltese
480,491
1,421,372,608
3,646,102,921
Bulgarian
576,997
1,444,748,66 621
3,422,182.324
Lithuanian
539,906
1,192,586,834
3,097,400,90
Romanian
725,766
1,398.178002.
2,909,452,579
Japanese
1,409,956
204,698,43.
2.738,872,745
Arabic
1,291,439
827,392,227
2,682,255,180
Slovenian
504,492
1,107, 467, 467,643
2,602,943,380
Latvian
364,503
966,005,785
2.579,350,119
Ukrainian
1,378,390
786,829,868
2,561,253,212
Croatian
443,669
984,577 685
2,400,762,140
Chinese
1,426,017
329.649,628
2,238,230,225
Haitian Creole
292,718
762,225,686
1,420,886.044
Turkish
572,633
380,551,359
1,206,732,266
Cebuano
6,123,694
598,016,557
1,080,404 400,400,899
Norwegian Nynorsk
1,026,745
405,911,671
1,036,914,977
Irish
139,815
371,175,86 864
958, 688.647
Castilian
50,785
457,490,798
893,080,621
Serbian
698,037
292.399.22 267
822,519,322
Hebrew
343,406
227,869.199
763,029 4888
Catalan
803,532
14,,044
736,220,789
Korean
653,460
143,729.419
677.716.397
Persian
983,632
209,078,44 446
668,287,97 28.975
Vietnamese
.229,789
267,190,147
593.203,801
Norwegian
628,798
152,133,513
661,694.223
Armenian
306,959
109,852,686
539,900.475
Hindi
210,984
127,830,222
463,832,707
Yiddish
59,158
122,100,684
463,526.449
Welsh
309,573
191,827,552
138,133,529
Occitan
288,195 195
195.380,666
357,267 ,635
Georgian
172,532
71,678,084
349,884,144
Basque
439,582
339,077.600
348,891,265
Table 5: T0p-50 languages in Common Corpus by token count. Each language is presented with its number documents, words, and tokens the corpus. The rows are ordered by the token count.
20

---

Published as a conference paper at ICLR 2026
C TOKENIZER DETAILS
Engkin
Frech
German
5.20
3.9
40
1.99
Port By Rolters
0.75
0.00
Fon
0.00
1.9
Gamera 1B
Obor 18
Geostaaz 1.1
Liane 2.2 18
Dane 1.1
Awetaa 1.
Oene 2.78
Sarr
Llan
Inlas
Palee
2.24
Fwelity Hossre Ste Cunn
3/2
Pob Hekees Ser (000)
2.05
23-
1.49
2.00
2:08
2
7.44
141
10
Sevmaar 1.70
Liene R2
Oven 1.8
Gemera 18
Genmoe 2.8
Oom
nater
dest 178
Usen 22
1:7
Genmo P
Quea
Figure 5: Comparing the fertility of PlelAs tokenizer (marked "Ours IB") and other language mod- els for six languages. The data source for all languages the devtest set of FLORES200 (Costa jussá et al. 2022).
we show how tokenizer with vocabulary of size 65,536, trained on representative subsample of Common Corpus, compares to other language model tokenizers. Our tokenizer is outperformed only by Gemma which has tokenizer four times larger.
In Figure
D PROVENANCE
D.1 OPEN GOVENMENT
In this section, we describe the provenance and present token counts and main languages for the two sub-collections of Open Govermment: Finance Commons and Legal Commons.
D.1.1
FINANCE COMMONS
Dataset
Main Languages
Documents
Tokens
SEC
English
1,085,113
9,653,919,837
WTO
English, Spanish, French, and small parti-
772,508
2,835,007,015
tions of others
AMF
French, English
595.397
9,823,755.281
TED EU
German, French, Polish, Spanish, Dutch, Czech, Romanian, English, Swedish, Ital- ian, Bulgarian, Finnish. Latvian, Danish, Lithuanian, Croatian, Estonian, Hungarian, Portuguese, Slovenian, Slovak, Greek, Irish
137,837
650,396,761
Tenders
GATT Li-
English, French, Spanish, Catalan, Por-
67,596
224,526,628
brary
tuguese, German
Table 6: Finance Commons sources distribution with languages.
The datasets that make up Finance Commons are presented in Table Here, we also present the provenance details for each of the parts of Finance Commons:
21

---

Published as a conference paper at ICLR 2026
Securities and Exchange Commission (SEC). This dataset comprises the SEC annual reports (Form 10-K) for the years 1993 to 2024. Entries up to 2020 were compiled by Loukas al. 2021). We added the reports from 2021-2024, which come from the EDGAR datahase compiled using the EDGAR-Crawler toolkit
World Trade Organization (WTO). This dataset comprises documents from WTO's official Documents Online platform. The documents cover the years 1995 to 2024. Documents are available in three official languages: English, French, and Spanish. Some documents are available other languages, e.g., Chinese, Korean, Arabic, German, and Portuguese. Also released separately WTO-PDF.
French Authority for Financial Market (AMF). This is a dataset documents from the French Authority for Financial Market, or the Autorité des marchés financiers R (AMF), which an independent public authority that regulates the French market. The documents are primarily in French. Also released separately as AMF-PDF.
Tenders Electronic Daily (TED) EU Tenders. This dataset is collection of procurement notices published by the EU. The documents are published in the online version of the "Supplement the Official Journal" of the EL dedicated European public procurement. The documents are mostly in German, with French, Polish, and Spanish making up relatively large portions of the remaining documents. There are also small portions of other languages (see details in T
General A Agreement Tariffs and Trade (GATT) Library. This dataset comprises documents from GATT, which was an organization that promoted international commerce and the reduction of trade barriers among member states. Public documents available by the General Council of the WTO in 2006 The documents span from January 1, 1946, to September 6, 1996. Most of the documents documents in French, Spanish, and other languages.
made
English, but there are also
D.1.2 LEGAL COMMONS
Here, we present the provenance details for each of the parts of Legal Commons:
Europarl. This dataset a multilingual parallel corpus, drawn from the proceedings the
European Parliamen [T] It includes texts from 21 EU languages. It was originally compiled by Koehn (2005).
Caselaw Access Project. This dataset consists of 6,773,632 legal cases, digitized from Harvard Law School Library's physical collection of American case law The dataset spans the years 1658 to 2020.
CourtL.istened This is datase of opinions, oral arguments. judges, judicial financial records, and federal filings put together by the Free Law Project
EUR-lex. This is dataset of 57, 57,000 legislative documents from the EU 15
It is based on the
dataset by Loza Mencía & Fürnkranz (2010) and developed by Chalkidis al. (2019). The documents have also been annotated by the Publications Office of EU with concepts from
Eoro
The dataset covers all 24 EU languages.
dgar-datal
htpps///ed.europs.e///aa
https://ww..stet
https://case.laa/
https//eurr-ex.eezoa..
http://eurovvo.eurgpp...l
22

---

Published as a conference paper at ICLR 2026
Dataset
Languages
Tokens
Caselaw
Access
English
13,821,842,995
Project
Court Listener
English
22,625,121,735
EUR-lex
Bulgarian, Croatian, Czech, Danish, Dutch,
65,044,763,781
English, Estonian, Finnish, French, German,
Greek, Hungarian, Irish, Italian, Latvian,
Lithuanian, Maltese, Polish, Portuguese, Ro- manian, Slovak, Slovenian, Spanish, Swedish
Eurovoc
English, German, French. Croatian, Italian, Lithuanian, Portuguese, Finnish, Danish, Bul- garian, Dutch, Polish, Greek, Swedish, Hun- garian, Czech, Spanish, Maltese, Latvian, Slo- vak, Slovenian, Romanian, Estonian, Arabic, Tigrinya, Farsi, Russian, Urdu, Serbian, Alba- nian, Kurdish, Pushto, Irish, Norwegian, Ice- landic, Dari, Armenian, Japanese.
11,648,136,899
French open data
French
24,597,392,089
USPTO
English
200,509,900,,18
UN Digital Library
Arabic, Chinese, English, French, Russian, Spanish
1,781,037,875
European
Open
EU languages
7,098,502,579
Data
OECD
English, French
584,969,458
Table 7: Legal Commons sources distribution with languages.
Eurovoc. Eurovoe is a dataset containing 1,528,402 documents in 39 languages with associated EuroVoe labels. The documents come from Cellau which is data repository for the Publications Office of the European Union. This dataset was originally compiled by Sébastien Campior 199
French Open Data. This dataset comes from French administrative bodies' websites, for example, the French Directorate of Legal and Administrative Information (Direction de I'information légale administrative DILA), which is French public administrative entity that disseminates information about laws and their applications to the public.
USPTO.
This dataset comprises documents from the United States Patent and Trademark Office (USPTO), the federal agency that grants patents and registers trademarks. This dataset consists of actions from this agency from 2019 to 2022. It was originally published as part of the Pile of Law (Henderson et al. 2022
UN Digital Library. This dataset comes from the UN Digital Library
22
European Legal Dataset. We also collect datasets from various EU websites, e.g., Archives of the EU Institute and the Council of the EU
httees//iigtallbbbe.......////.
https://archiess.eu.cc
les/traangareenoy//aa---ta
23

---

Published as a conference paper at ICLR 2026
Corpus
Language
Domain
Tokens
English PD
English
Books and Newspapers
174.2B
US PD Books
English
Books
82.2B
French PD Books
French
Books
24.0B
French PD Newspapers
French
Newspapers
110.8B
French PD Diverse
French
Books and Newspapers
69.6B
LoC Books
English
Books
10.6B
US PD Newspapers
English
Newspapers
199.3B
New Zealand PD Newspa-
English, Mãori
Newspapers
12.6B
pers
Europeana Newspapers
Multilingual
Newspapers
21.0B
German PD Newspapers
German
Newspapers
18.4B
German PD
German
Books
58.0B
Portuguese PD
Portuguese
Books and Newspapers
2.6B
Spanish PD Newspapers
Spanish
Newspapers
8.0B
Spanish PD Books
Spanish
Books
15,4B
Italian PD
Italian
Books
18.2B
Dutch PD
Dutch
Books and Newspapers
2.7B
BnL Newspapers
German, French, Lux-
Newspapers
0.3B
embourgish
Danish PD
Danish
Books and Newspapers
0.5B
Serbian PD
Serbian
Books and Newspapers
0.3B
Czech PD
Czech
Books and Newspapers
0.7B
Greek PD
Greek
Books and Newspapers
4.2B
Multilingual PD
Multilingual
Books and Newspapers
8.4B
Polish PD
Polish
Books and Newspapers
5.9B
Latin PD
Latin
Books
27.2B
Russian PD
Russian
Books
1.9B
Arabic PD
Arabic
Books
0.3B
Table 8: Subsets of Open Culture with language coverage, type of document, and token count.
OECD. These data come from the Organisation for Economic Co-operation and Develop- ment (OECD 25
D.2 OPEN CULTURE
Large portion of data in Open Culture part of the Common Corpus was built on top of the following collection-as-dat initiatives:
Chronicle America: about 100B words (150B tokens) of digitized US newspapers by the Library of Congress, made available as a raw text file.
Europeana: about 21B tokens digitized European newspapers through large-scale cross-
national contributions and new digitizations.
Gallica: about 85B words of digitized French newspapers and monographs made available on the open data portal of the French digitized library through entire dumps or API access 26
Biblioteca: about 15B words of digitized Spanish newspapers and monographs.
Combined with the other retrieved data, the collections were dispatched into smaller individual subsets, which were also separately released as parts of the Open Culture collection (Table 8 The Open Culture data in Common Corpus have been post-processed and filtered, as described below, which results in slightly different final word and token count:
http:://www..edd.rrr////a/tattta html 2order
https://api.N5 .nf.
24

---

Published as a conference paper at ICLR 2026
French PD. This corpus is based on the training corpus for gallicagram It comprises 289,000 books from the French National Library (Gallica). This initial aggregation was made possible thanks to the open data program of the French National Library and the consolidation of public domain status for cultural heritage works in the EU following the 2019 Copyright Directive (Art. 14).
French PD Newspapers. This dataset was also based on the Gallicagram corpus. It comprises nearly three million unique newspaper and periodical editions from the French National Library (Gallica).
LoC Books. This dataset comprises 140,000 English books, digitized by the Library of Congress. The books come from the Selected Digitized Books Collection 28 The dataset was curated by using the Library of Congress JSON API. This dataset contains only the books in the English collection. The dataset was compiled by Sebastian Majstorovic.
US PD Newspapers. This dataset comprises 21 million digitized newspapers from Chroni- cling America The newspapers were digitized by the Library of Congress. The dataset can be fully explored through an original corpus map created by Nomic A The dataset is mostly in English, but it also contains articles other languages, mostly German and Spanish. The articles were published between the years 1690 and 1963.
New Zealand PD Newspapers. This dataset comprises historic newspapers from New Zealand and the Pacific from the 19th and 20th centuries. The data were made available by the National Library of New Zealand as part of Papers Past 31 The articles are primarily in English, but include some articles te reo Mãori.
Europeana Newspapers. This dataset contains over 1,000 digitized newspapers from 23 libraries around Europe. It contains articles in at least 40 languages, and its articles were published between 1618 and 1990 (Neudecker, 2016 The original sources available via Europeana, and were made available by Big Science
German PD Newspapers. This dataset contains articles from 4,299,653 9,99,,3333,, issues from over 1900 different newspapers, The articles come from the German Digital Library, hosted by Deutsches Zeitungsportal" The articles were originally published between 1794 and 1957.
This dataset was curated and first made available by Sebastian Majstorovid
German PD. This dataset contains texts from various sources, including the Mannheim Cor- pus of Historical Newspapers and Magazines (Mannheimer Korpus Historischer Zeitungen und Zeitschriften). This dataset is made up of 21 German newspapers and magazines. The texts were originally published between 1737 and 1905. The corpus was originally digitized between 2009 and 2011 and made available by the Institut für Deutsche Sprache 2013.
Spanish PD Books. This dataset contains 302.640 individual texts from various sources, including the leading cultural heritage institution Biblioteca Digital Hispánica (60 (BDH). To ensure that these texts are in the public domain, we have retained exclusively titles published prior to 1884.
Dutch PD. This dataset contains approximately 176,000 books and 540,000 periodicals, which come from various sources including Delphe Delpher is repository of digitized printed material from the Netherlands, which is maintained by the Koninklijke Bibliotheck, the national library of the Netherlands. To ensure that these texts in the public domain, we have retained exclusively titles published prior to 1884.
ollection/
/
25

---

Published as a conference paper at ICLR 2026
BnL Newspapers. This dataset contains 630,709 articles from 21 different newspaper titles and 24,415 unique issues. The articles were digitized by the National Library of Luxembourg (BnL) part of their Open Data Initiative OCR done using Nautilus- OCR39 39 The articles are in German, French, and Luxembourgish. The newspapers were originally published between 1841 and 1879. The dataset was published and made accessible by BigScience.
The rest of the datasets, including French PD Diverse, Portuguese PD, Italian PD, Polish PD, Danish PD, Swedish PD, Serbian PD, Czech PD, and Multilingual PD, come from various sources, including several European national libraries and cultural heritage institutions. To ensure that these texts in the public domain, have retained exclusively titles published prior to 1884.
D.3 OPEN SCIENCE
In Table we present the token counts inside of the Open Science collections.
Dataset
Tokens
OpenAlex
191,616,437,384
Open Science Pile
11,096,766.322
Open Science French
46,961,690,792
Open Science Spanish
16,523,491,767
Open Science German
7,806,446,050
ArXiv
7,188,731,472
Total
18,,193,663,789
Table 9; Token count by dataset Open Science
D.4
OPEN CODE
Table 10 shows the number of tokens for the top ten coding languages and frameworks in Open Code.
Language
Tokens
Java
35,697,451,454
JavaSeript
28,894.772,110
Python
26,681,331,771
C++
25,481,950,314
C
23,277,000,
РHP
23,077,121,733
C#
16,806,995,110
Go
11,200,587,099
Rust Ruby
1.888,428,11
3,718,918,988
Table 10: Token counts by programming language framework.
E
OPEN CULTURE VERIFICATION
Here, we describe the rights verification process that we applied for cultural data objects:
Author life + 70 years for all non-US authors. Among most signatories of the Berne
Convention for the Protection Literary and Artistic Works [0 this is the most common
https::/Gata.bnl..1
26

---

Published as a conference paper at ICLR 2026
approach determining documents in the public domain. This approach requires not only identifying the author but also their date death. On top of the information already made available by cultural heritage institutions, reconciliation pipeline based on the complete dump of Wikidata.
also implemented an internal data
All publications after 1884. In cases where the author could not be identified or for collective works like newspapers, we applied "universal" public domain rule based on 70 years prior to the current term of the author's life + 70 years. Simplified rules like these are commonly applied in cultural heritage projects, especially for the release of newspaper collections.
Publication 95 years for US authors. This is the copyright based approach currently in place in the US. For an international project, this will only affect US-born authors. Due to lack of further legal expertise, we did not attempt to include works whose copyright might not have been renewed.
No digitization rights. Following on the 2019 Copyright Directive (Art. 14) and common practice among GLAM reusers like Wikimedia Commons, consider that the simple act digitization does not provide any additional rights.
F CLEANING AND CURATION
We release the cleaning and curation tools as part of our Bad Data Toolbox, a suite designed to make challenging digitized resources usable for downstream NLP tasks and large-scale corpus construction.
F.1 TEXT SEGMENTATION: SEGMENTEXT
Repository: PleIAAs/Segmentex"
Segmentext is token classification model trained for the structural segmentation of documents, with specific focus on robustness to digitisation artefacts and broken or unstructured input text. In contrast to layout-based segmentation approaches that operate visual signals such as bounding boxes or font information, Segmentext works exclusively on raw character sequences, reconstructing editorial structure from text alone. This makes it applicable in post-OCR pipelines where layout information has been lost.
The model is DeBERTa-v2 (He et al. 2021) variant for token classification, fine-tuned 3,500 manually annotated documents. The training data was drawn from three datasets spanning comple- mentary domains: Open Government, Open Culture, and Open Science. This domain diversity is intended to give the model broad applicability across document types and languages.
Segmentext supports the following text segmentation:
Text: the text itself.
Title: any type of heading.
Table: tabular content.
Separator: a segmentation separator, mostly based on newline, with some variations due to text segmentation understanding.
Dialog: any kind of speaker-atrribtted intervention.
Bibliography: statement of specific bibliographic reference, either in a bibliography section or a footnote.
Contact: personal information, which can be especially useful in the context of PII removal.
Paratext: any non-me meaningful text included in standard documents like headers, page numbering, section recall, etc.
Author: author names and signatures.
Date: statement of date and time, common in letters and newspaper articles.
Keyword: list of keywords, especially common in scientific publications.
27

---

Published as a conference paper at ICLR 2026
Here is an example input text for the Segmentext model:
In this respect, the insurance business investment portfolio can be considered conservatively managed as it is largely composed of corporate, sovereign, and supranational bonds, term loans as well as demand deposits. Following the previous year, the group continued to diversify its holdings into investment-grade corporate bonds. It should be noted that bonds and term loans are held to maturity accordance with the group's ' business model policy of "inflows".
Technical liabilities on insurance contracts.
The guarantees offered cover death, disability, redundancy, and unemployment part of loan protection insurance policy. These types of risk controlled through the use of appropriate mortality tables, statistical checks on loss ratios for the population groups insured, and through the insurance program.
Liability adequacy test.
A goodness-of- test aimed at ensuring that insurance liabilities are adequate with respect current statements future cash flows generated by the insurance contracts is performed at each statement of account. Future cash flows resulting from the contracts take into account the guarantees and options inherent therein. In the event of inadequacy. the potential losses are fully recognized in the income statement. The modeling of future cash flows the insurance liability adequacy test are based on the following assumptions: At the end of 2022, this liability adequacy test did not reveal any anomalics.
Income statement.
The income and expenses recognized for the insurance contracts issued by the group appear in the income statement in "Net income of other activities" and "Net expense of other activities".
Risk management.
The group adopts a "prudent approach" to its management of the risks to which it could be exposed through its insurance activities. Risk of counterparty. As stated above, insurance companies only invest in assets (bank deposits, sovereign bonds, supranational agencies, or corporate bonds).
Example output:
Editorial Segmentation
[Text] In this respect, the insurance business investment portfolic can be considered conservetively managed as it largely composed of the
dimand Following the contirued to diversify its hold ngs into investeent-rraee corporate bonds, It should noted that bonds and term loans are held maturity in accordance with the group's business model policy of "infoows..
and supranational bonds,
meee
Technical liabilities on insurance
[ent]
The guarantees offered cover desth, dissiiity, redundancy, and unemolayment as part of loan proteczion insurance policy. These types risk controlled through the appropriate mortality tables, statistical checks ratios for the populat groups insured, and through the insurance program.
[Ttia]
Liability adequacy test.
[Toxt]
A goodness test
ensuring that Insurance flows generated by the insurance contracts is performed at each statement nent of account Future cash flows resulting from the contracts potential
adequase with respect to current statemonts
inherent therein. the income statement The modeling future cash flows in the insurance libiliiy adequacy test are based on the following assumptions At the end 2022, this liailiity adequacy test did not reveal any anomalles.
take
the
Inadequacy,
in
[iitle]
Income statement.
[Text]
The income and expenses recogn zed for the insurance contracts issued by the group appear in the income statement in "Nec activ
activities..
[Tibee]
Risk management.
[Text]]
The edopts prrudent approach' to management of the risles which could exposed through its insurence activities
[T36a
Risk of counterparty.
[Text]
As stated above, Insurance companies only mvest assets jbank deposits, sovereign conds, supranational agenc es, corporate
28

---

Published as a conference paper at ICLR 2026
F.2
OCR ERROR DETECTION: OCROSCOPE
Repository: PleIAs/OCRoscom
OCRoscope lightweight Python library for estimating the quality of OCR-processed text without access to a ground-truth reference. Rather than relying on character- or word-level edit distance against a gold standard, OCRoscope exploits the differential behaviour of language detection models across text granularities. On long texts, language detection is robust to noise; short character sequences, degrades rapidly in the presence of OCR errors. The library leverages this property by applying language detection (via pycld2) to rolling 7-grams extracted from the input text and comparing the per per-gram language assignments against the document-leve language detected on the full text. The proportion of n-grams assigned to mismatching unknown language yields the OCR quality score: a value close to 1 indicates high-quality text, while lower values signal increasing levels of digitisation noise.
OCRoscope returns two primary metrics: a standardised OCR quality rate and a standardised rate of non-character content (punctuation, digits, whitespace anomalies). The tool supports over 80 languages via cld2's i language coverage. The choice of cld2 over more accurate successors such as cld3 is intentional: cld2's greater sensitivity to short noisy sequences makes a more discriminating signal for OCR quality estimation, yielding a score distribution that is well spread across the quality spectrum and therefore easier to threshold for downstream filtering decisions.
The tool is designed for pipeline use cases in which large corpora must be rapidly triaged: it provides fast, reference-free quality signal that can guide decisions about whether texts warrant OCR correction, manual review, or outright exclusion.
Below we illustrate OCRoscope performance on this long text, which is correctly identified French with >99% confidence by cld2. Despite the many mistakes, there are enough non-ambiguous French words:
NOUVELLES POLI TI Q. Suede. Stockholm le 2 5 décembre 1792. Le général Toll ira Varsovie en quarté e'voyé de Suede auprès du roi et de la république A I même rey.u I' l'ordre de s'y rendra incessamment. 11 paraît que k Uc-régeik a des craintes ; il a fait venir chez lji les membres Iij" tribunal 4e cour, et leur rtmis son lesfca n at. La fermentation qu' a causée 1 ''ari r?tavh n k M P v riote Thorild tf' est pas appaisée y le luigage qv'il yailé an duc-régent a été bien entendu par le peu) y M (U n' eeetendrrii pas I' l'apostrdphe suivante uRRx3xa7nd >l libuk à r otre raison, et ne et nous force pas de i'ache' ef r te n te sang..
Le duc a fait x.épa4idre sur-le-champ une fjtbprijuun à te us les habitans diS Toyaume pour les detourntr de mr laisser sé luire par de fa,ux bruits et des jugemens pe rveră. e en même temps l'ordre a. été donné la garnison de charger et de se tenir prête à marcher.
(Mercure Français, 1793, January 25th)
Yet one short n-gram ('n k M riote Thorild") is classified unknown by cld2. Complete processing by OCRoscope yielded a 41% rate of non-recognized 7-grams, resulting in an OCR quality rate of 59%. In comparison, the self-estimated rate of OCR-validate words by the French National Library is significantly higher (85% ) for the whole document.
F.3 OCR ERROR DETECTION: OCRERRCR
Repository: PleIAs
OCRerrer is small token classification model trained to identify individual OCR errors at the token level within text. Unlike OCRoscope, which produces document-level quality score, OCRerrer generates a fine-grained annotation of which tokens are likely to be erroneous, making it suitable for targeted correction workflows and for producing spatially explicit error rate estimates.
The model is a DeBERTA-v3 (He et al.) 2023) with 400M parameters fine-tuned on a corpus of approximately 1,000 documents with manually labelled OCR errors, drawn from Open Govemment
29

---

Published as a conference paper at ICLR 2026
and Open Culture. This dual-domain training is intended to give the model coverage across both modern administrative language and historical or literary language, which present different error profiles. OCRerrer provides an accurate reference-fiee OCR enror rate estimate. outperforming OCRoscope in precision, particularly on documents with a low or moderate error rate. However, scales less efficiently than OCRoscope very large corpora due its per-token inference cost.
The model's name is itself playful illustration of the problem it addresses: OCRerrcr is a plausible OCR misreading of OCRerror, with the vowels dropped as they might be in degraded scan.
The following is a low-error example sentence taken from Open Culture:
They did not approach cer, but turned away and passed irom her presence, filled with sorrow and moved with sympathy, which her intense emotions seemed to communicate these thoughtless young men of the tho plains.
The version with the errors highlighted by OCRerrer is presented below:
They did not approach <er>cer,<<er: but turned away and passed <er>irom</er> her presence, filled with sorrow and moved with sympathy, which her intense emotions seemed to communicate to even these thoughtless young men of the <er>tho</er>> plains.
F.4 OCR ERROR CORRECTION: OCRONOS
Repository:
PleIAs/OCRonos
OCRonos is generative language model trained for the correction of badly digitised text, including OCR errors, incorrect word splits and merges, and broader structural artefacts introduced during digitisation. It belongs to the corrective tier of the Bad Data Toolbox, operating downstream of quality detection tools such as OCRoscope and OCRerrer.
The training data for OCRonos consists of diverse multilingual set of OCR-processed texts drawn from Open Culture and Open Government. The model is tinettueed from Llama-3-8B (Grattafiori al. 2024).
OCRonos is designed to be conservative: it generally faithful to the original source material and rarely rewrites words that not erroneous. On highly degraded inputs, however, it function as synthetic rewriting tool rather than a strict corrector. A known issue at scale is the occasional inclusion of spuriously repeated words, which can be filtered in post-processing. The model also mitigates a common failure mode observed in smaller generalist LLMs, namely language switching: faced with heavily noisy input, models such GPT-3.5 Claude Haiku have been observed to drift into a different language or script during correction. OCRonos is specifically trained to resist this behaviour.
Below we present an exemplar text containing various OCR errors:
Theguaran offered cover disallyis dundancy andunem ployment aspartof aloanprotect ion insurance policy. These types f risk are controlled throu ghthe app ropriate morta litytables,tattstice Ichecksonloss rat ios for thepopulation groups insure dandthrough ar e insurance program.
The following the version corrected by OCRonos:
The guarantees offered cover death, disability, redundancy, and unemployment part of a loan protection insurance policy. These types of risk are controlled through the use of appropriate mortality tables, statistical checks on loss ratios for the population groups insured, and through the insurance program.
F.5 TOXICITY DETECTION: CELADON
Repository:
PleTAs/celados
30

---

Published as a conference paper at ICLR 2026
Data: Ple PLeIAsTToxicoomme
Celadon is a multilingual toxicity classifier designed specifically for the challenges posed by public- domain pre-training data: historical language. OCR noise, and multilingual content spanning nine languages. It was developed as part of the Toxic Commons project, described in Arnett et al. (2024). and was applied to filter harmful content in Common Corpus prior to its release.
The model is a fine-tune tune of DeBERTa-v3 -aaaaal (He et al. 2023) with five independent classification heads, each corresponding to dimension of toxicity: xenophobia, and bias against immigration status), gender and sexuality-bane -based bias (including sexism, homophobia, and transphobia), religious bias, ability bias (bias relating physical, mental, or intellectual disability), and violence and abuse (graphic violence, threats, and incitement). Each head separate linear layer with custom weighted cross-entropy loss function handle class imbalance.
and origin-based bias (including racism,
The model was trained on 600,000 samples drawn from ToxicCommons, dataset of approximately 2 million annotated mubiic-domain text segments from 13 multilingual cultural heritage collections. Annotations were generated in two stages: a small set of human annotations was used to calibrate prompt for the instruct version of Llama 3. 1-B (Grattafirii et al. 2024), which then produced annotations at scale across five toxicity dimensions using a 0-3 severity seale. The training set was balanced to equalize non-toxic and toxic samples.
Celadon's key design rationale is efficiency: it achieves toxicity detection performance comparable to Llama 3.1-8B 1-8B-Instruct while being over 40 times faster (5 minutes versus 3.4 hours per 100,000 samples), making it practical for large-scale corpus filtering. The model is optimised for the specific challenges of its target domain: historical text, OCR noise, and non English content, and may not generalise without adaptation to web-text corpora, for which it was not designed.
G EVALUATIONS
Figure 4 presents the distribution of six document-level quality and character composition metrics the five open types in 300,000--dcumeet sample of Common Corpus:
Character Repetition measures the proportion of character pairs belonging to repeated character bigram sequences within a document. This value is expected to be higher for code data, due to specific syntax and varied punctuation.
Type-Token Ratio is the ratio of unique word forms to the total number of tokens. It captures lexical diversity; documents with values close to 1 are typically very short, while low values can indicate highly repetitive text. For Open Government, this value is expected be lower as regulatory texts usually contain fixed, distinct terms and repetitive formulations.
Top Word Proportion is the share of all tokens accounted for by the single most frequent word in document. This is another measure of linguistic difersity.
A Alphanumeric Characters denotes the fraction of characters that are letters digits. This proportion might be lower in code due to reliance on punctuation and whitespace symbols.
Whitespace Characters gives the proportion of whitespace characters. which tends to be higher in source code and tabular data than in running prose.
Uppercase Characters measures the share of uppercase letters and separates Open Source content, which is densely populated with identifiers, constants, and type annotations. from Open Culture and Open Government documents that follow standard typographic conven- tions.
In Tables II and 14 we present per-language scores for the studied benchmarks. On MultiBLIMP, most of the scores are significantly above random (0.5); therefore, we also highlight the best and second-best scores.
31

---

Published as a conference paper at ICLR 2026
Ours
Gemma 3
XGLM
BLOOM
Ours
Gemma
XGLM
OLMo
Model
350M
270M
564M
560M
1.2B
1B
1.7B
1B
abk
0.550
0.750
0.475
0.325
0.675
0.675
0.325
0.800
aln
0.733
0.755
0.709
0.700
0.728
0.750
0.675
0,690
amh
0.946
0.929
0.911
0.973
1.000
0.955
0,848
0.964
apu
0.964
0.964
0.893
0.786
0.964
0.929
0,964
0.893
aqz
0.214
0.357
0.429
0.500
0.429
0.429
0.714
0.214
arb
0.877
0.913
0.895
0.923
0.900
0.951
0.887
0.782
azz
0.734
0.744
0.729
0.720
0.729
0.758
0.773
0,686
bel
0.799
0.795
0.574
0.608
0.853
0.896
0.577
0.611
0.571
0.762
1.000
0.810
0.762
0.905
0,857
0.524
bho
0.676
0.647
0.588
0.588
0.706
0.794
0.618
0.588
bor
0.722
0,631
0.697
0.610
0.697
0.627
0.680
0,668
bre
0.942
0.815
0.554
0.604
0.938
0.946
0.615
0.685
bua
0.680
0.718
0.670
0.718
0.670
0.641
0.699
0.660
0.872
0.880
0.969
0.623
0.897
0.945
0.976
0.735
cat 0.885
0.852
0.961
0.950
0,919
0.931
0.953
0.735
ces
0.824
0.808
0,579
0.597
0,858
0.891
0.603
0.668
chu 0.670
0.648
0.582
0,635
0.659
0.663
0.593
0.632
cym 0.771
0.730
0.633
0.611
0.828
0.796
0.610
0.796
dan 0.980
1.000
0.840
0.800
0.980
1.000
0.740
0,940
deu 0.967
0.949
0.961
0.754
0.977
0.981
0.969
0.886
egy
0.409
0.409
0.409
0.455
0,500
0.455
0.409
0.455
ell 0.931
0.937
0.985
0.676
0.948
0.975
0.984
0.842
eng 0.981
0.979
0.973
0,960
0.983
0.987
0.974
0.984
est 0.729
0.699
0.885
0.561
0,800
0.800
0.915
0.587
eus 0.916
0.927
0.963
0.952
0.916
0.938
0.982
0.905
fao 0.707
0.647
0,509
0,556
0.772
0.806
0.552
0.681
0.756
0.810
0.567
0.577
0.837
0.919
0,565
0.655
fin 0.736
0.744
0.947
0,562
0,809
0.893
0,935
0.645
fra 0.994
0.963
0.963
0.984
0.993
0.989
0.976
0.928
frm 0.997
0.741
0.745
0.905
0.997
0.847
0.820
0.765
fro 0.782
0.701
0.686
0.709
0.822
0.725
0.694
0.679
gla 0.955
0.924
0,924
0.939
0.909
0.924
0,939
0.970
gle 0.750
0,821
0.750
0.786
0.679
0.750
0.750
0.714
glg
0.849
0.807
0.798
0.788
0.879
0.895
0.789
0.754
got 0.630
0.642
0.588
0.599
0.645
0.631
0.580
0.598
gre
0.824
0.719
0.683
0.623
0.887
0.758
0.711
0.707
guj
1.000
1.000
1.000
1.000
1.000
0.857
1.000
1.000
hbo 0.711
0.712
0.683
0.706
0.737
0.764
0.629
0.679
hbs 0.892
0.848
0.619
0.603
0,922
0.929
0.616
0.723
heb 0.853
0.829
0.609
0.642
0.876
0.868
0.585
0.667
0.854
0.934
0.975
0.971
0,916
0.966
0.977
0.748
hit 0.620
0.640
0.520
0.600
0,560
0.480
0,540
0,540
hsb 0.683
0.677
0,624
0.629
0.667
0.694
0.624
0.591
hun 0.928
0.867
0.728
0.692
0.938
0.925
0.686
0.740
hye 0.883
0.898
0.631
0.623
0,929
0.946
0.662
0.671
hyw 0.813
0.781
0.583
0.608
0.894
0.859
0.551
0.629
isl 0.710
0.751
0.653
0.660
0.767
0.863
0.636
0.667
ita 0.925
0.910
0.915
0.670
0.952
0.965
0.915
0.791
Table 11: Multilingual benchmarking results on MultBLIMP (ISO 639 language codes a*-i* in alphabetical order). "Ours" refers to PleIAs models pre-trained on Common Corpus. Within each model group, the best score is in bold, and the second-best is underlined.
32

---

Published as a conference paper at ICLR 2026
Ours
Gemma 3
XGLM
BLOOM
Ours
Gemma
XGLM
OLMo
Model
350M
270M
564M
560M
1.2B
1B
1.7B
1B
kat
0.931
0.951
0.917
0.760
0.951
0.951
0.809
0.907
kaz
0.705
0.792
0.647
0.682
0.780
0.844
0.682
0,688
kir
0.930
0.843
0.914
0.919
0.935
0.924
0.843
0.941
kmr
0.710
0.662
0.588
0.579
0.761
0.748
0.577
0.588
0.628
0.488
0.605
0.558
0.558
0.651
0.628
0.605
kpv
0.641
0.591
0.553
0.547
0.700
0.628
0.581
0.547
krl
0.650
0.612
0.688
0,573
0.612
0.642
0.704
0.573
kxh 0.483
0.433
0,475
0.333
0,458
0.450
0,442
0.483
0.874
0.651
0.578
0,575
0,925
0.730
0,568
0.625
lav 0.791
0.747
0.616
0.604
0.844
0.862
0.611
0.623
lij 0.783
0.744
0.669
0.638
0.780
0.807
0.701
0.665
lit 0.928
0.848
0.745
0.740
0.947
0.932
0.736
0.779
0.737
0.717
0.735
0.667
0.713
0.776
0.726
0.776
mdf 0.537
0.622
0.524
0.537
0.622
0.585
0,561
0,500
mkd 0.923
0.974
0.769
0.769
0.821
1.000
0.590
0.718
myv 0.608
0.614
0,565
0,560
0,636
0.619
0.532
0.547
nds 0.736
0.729
0.674
0.663
0.749
0.732
0.674
0.700
0.526
0.579
0,474
0.632
0.553
0.579
0,447
0.500
0.924
0.912
0.620
0.627
0.954
0.963
0.663
0.829
olo 0.679
0.668
0.795
0.611
0.753
0.711
0.842
0.595
0.733
0.721
0.690
0.636
0.757
0.744
0.707
0.667
0,879
0.929
0.899
0.848
0,939
0.949
0.889
0.828
pem 1.000
1,000
L.000
0.923
1.000
0.962
1.000
0.885
pol 0.892
0.849
0.624
0.634
0,930
0.931
0.628
0.725
por
0.948
0,933
0.939
0.955
0,965
0.972
0.920
0.872
que 0.779
0.672
0.649
0.740
0.740
0.664
0.679
0.656
ron
0.868
0.874
0.638
0.608
0.903
0.928
0.640
0.793
rus
0.921
0.916
0.937
0.727
0.952
0.963
0.954
0.819
sah 0.688
0.771
0.736
0.792
0.708
0.681
0.701
0.764
0.657
0.666
0.612
0.609
0.670
0.678
0.618
0.620
slk
0.797
0.739
0.528
0.570
0.824
0.861
0.533
0.588
0.882
0.796
0.618
0.636
0.903
0.854
0,622
0.711
sme
0.689
0.705
0.653
0.660
0.681
0.700
0.668
0.659
sms
0.833
0.802
0.779
0.757
0.821
0.779
0.764
0.768
spa
0.959
0.945
0.950
0.966
0.970
0.973
0.956
0.896
sqi
0.786
0.823
0.494
0.588
0.823
0.881
0.539
0.765
swe
0.995
0.995
0,970
0.950
0.995
1.000
0,985
0,990
tam
0.942
0,942
0.969
0.966
0,932
0.953
0.976
0.746
0.111
0.000
0.111
0.111
0.222
0.111
0.000
0,000
0.478
0.478
0.493
0.449
0.449
0.464
0.435
0.478
0.766
0.804
0.829
0.700
0.814
0.908
0.857
0.716
uig 0.764
0.760
0.722
0.732
0.755
0.757
0.686
0.740
ukr 0.874
0.892
0.640
0.606
0.911
0,946
0.648
0.704
urb 0.538
0.462
0.462
0.231
0.538
0.462
0.462
0.462
urd 0.858
0.925
0.956
0.935
0.896
0.964
0.960
0.736
0.900
0.880
0.780
0.720
0.900
0.940
0.760
0.880
vep 0.572
0,588
0,588
0.481
0.631
0.626
0,567
0.524
wbp 0.250
0,000
0.167
0.250
0.250
0.083
0.250
0.250
wol 0.892
0.881
0,851
0,891
0.868
0.879
0.854
0.823
xcl 0.679
0.674
0.585
0.636
0.711
0.702
0.613
0.616
xnr 0.791
0.756
0.779
0.616
0.744
0.814
0.721
0.733
0.820
0.900
0.820
0.860
0.880
0.900
0.900
0.900
yrl 0.689
0.722
0.594
0.604
0.683
0.669
0.626
0.601
Table 12: Multilingual benchmarking results on MultiBLIMP (ISO 639 language codes k*-y* in alphabetical order). "Ours" refers to PleIAs models pre-trained on Common Corpus. Within each model group, the best score is in bold, and the second-best is underlined.
33

---

Published as a conference paper at ICLR 2026
Ours
Gemma
XGLM
BLOOM
Ours
Gemma
XGLM
OLMo
Model
350M
270M
564M
560M
1.2B
1B
1.7B
1B
ar
0.475
0.492
0.500
0.521
0.477
0.572
0.525
0.473
0.514
0.513
0.567
0.561
0.535
0.600
0.602
0.509
0.569
0.614
0.606
0.612
0.617
0.698
0.645
0.704
0.520
0.558
0.549
0.555
0.543
0.628
0,593
0.556
eu 0.516
0.524
0.531
0.538
0.514
0.531
0,561
0.503
0.490
0,486
0.461
0.467
0.532
0.576
0.484
0,459
hi 0.509
0.541
0.520
0.549
0.515
0.598
0.557
0.493
id 0.500
0.544
0.542
0.555
0.518
0.631
0.583
0.498
my 0.492
0.507
0.515
0.475
0.498
0.518
0.537
0.475
ru 0.503
0.547
0.562
0.488
0,531
0.634
0.600
0.512
0.500
0.507
0,531
0,500
0.510
0.551
0.563
0,494
0.542
0.562
0.559
0.557
0.553
0.592
0.581
0.532
0.491
0.536
0.533
0,545
0.494
0.594
0.561
0.511
Table 13: Multilingual benchmarking results on XStoryCloze. "Ours" refers to PleIAs models pre-trained on Common Corpus. Languages are represented two-letter codes ISO 639.
Ours
Gemma
XGLM
BLOOM
Ours
Gemma
XGLM
OLMo
Model
350M
270M
564M
560M
1.2B
1B
1.7B
1B
es
0.566
0.590
0.604
0.620
0.614
0.678
0.664
0.524
0.518
0.498
0.554
0.488
0.500
0.536
0.568
0,480
0.504
0.514
0.512
0.502
0.518
0.502
0.534
0.516
0.522
0.504
0.548
0.500
0.524
0.518
0.556
0,534
0.534
0.578
0.574
0.596
0.558
0.690
0.646
0.544
it
0.542
0.524
0.536
0.502
0.562
0.648
0,536
0,488
qu
0.522
0.492
0.492
0.500
0.500
0.502
0.522
0.506
0.528
0.546
0.530
0.516
0.538
0.544
0.562
0.510
0.536
0.560
0.562
0.558
0.556
0.566
0,550
0.550
0.546
0.538
0.550
0.538
0.550
0,584
0,580
0.532
0.530
0.566
0.544
0.528
0.542
0.606
0,536
0.530
0.550
0.598
0.584
0,602
0.526
0.694
0,630
0.494
0.536
0.564
0.554
0.588
0.544
0.640
0.584
0.522
Table 14: Multilingual benchmarking results on XCopa. "Ours" refers to PleIAs models pre-trained on Common Corpus. Languages are represented two-letter codes ISO 639.
34


Reviews:
[{"review_id": "bkY1AsL5B0", "reviewer": "Reviewer_aTGr", "summary": "The paper introduces Common Corpus, a ~2T-token multilingual dataset  for LLM pre-training that contains only data that are in the public domain or released under permissive open licenses.  The authors document data sources, licensing, cleaning tools (Segmentext, OCRonos, Celadon, PII filtering), and present benchmark results for small Llama-style models trained solely on this corpus.\n\nOverall, this is a valuable contribution to open LLM research. The motivation is strong, and the authors make extensive efforts to demonstrate the usefulness of the dataset for pre-training purposes. \n\nHowever, several aspects including the very Euro-centric nature of the multilingual coverage and lack of quantitative information about ambiguous licenses need further clarification.", "questions": "1. Can the authors provide a full language inventory and token count distribution, including low-resource languages?\n\n2. How are ambiguous or missing licenses handled, and what proportion of the dataset do they represent?\n\n3. What steps were taken to ensure deduplication across overlapping sources?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Timely contribution to open-data infrastructure for LLMs, especially given that many existing datasets had to be taken down due to license issues.\n\n2. Transparent and detailed documentation of dataset provenance, content, and filtering pipeline\n\n3. Releases and discusses useful tools for OCR correction and sentence segmentation\n\n4. Demonstrates that open data sourced from only permissive licenses can still yield competitive model performance.", "weaknesses": "1. The claim of “multilingual diversity” is overstated as the dataset is heavily Euro-centric. The top ten languages are all European, and there is little inclusion of African or Asian languages.\n\n2. It is not clear how trustworthy the licensing information is. The authors do not report how many documents had ambiguous, conflicting, or missing license information.\n\n3. For certain types of data, licenses can apply inconsistently across sub-components. It is not clear how these were disambiguated.\n\n4. The data pre-processing pipeline, including Segmentext and OCRonos are underdescribed. It is not clear if any de-duplication was done or required.\n\n5. The evaluation lacks detail on per-language performance and broader benchmark coverage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C4Fuk8oFGb", "reviewer": "Reviewer_BY7t", "summary": "This paper accompanies the release of the largest open dataset for LLM pretraining: the Common Corpus. The authors fully emphasize the “open” aspect of their data, providing full transparency around the provenenance, source, and licensing of their data. They also engage in extensive data cleaning and quality improvement steps for different segments of their data. Unlike its recent open data precusors, Common Corpus also includes substantial multilingual data.", "questions": "Data may be copied and then mislicensed. Have you checked for overlap between your dataset and data that has less permissive licenses? This might be really tricky to do, so I am just curious. \n\nIs there a way for someone to remove data about themselves or produced by themselves from this dataset? Right to be forgotten, etc.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The writing of this paper was very clear, and this work was done with much care, detail, and rigor. For example, the authors “involve local communities” in gathering data from diverse sources and did not machine-translate the multilingual component of their dataset. This resource is an invaluable contribution and far exceeds its predecessors in size and composition. Aside from data, this work also contributes several data cleaning tools.", "weaknesses": "I understand that the main focus of this paper is on the data and not on model training, so it is okay that the model training results aren’t groundbreaking. One minor detail is that your choice of benchmarking results to show in Section 5 is a little strange. You focus on a few multilingual benchmarks, but compare against OLMo 1B, which may have been intended to be monolingually English. \n\nIt would have been nice to see a clear tabulation (like, a table) of how this dataset overlaps, extends, or differs from existing open datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6RgsXhH2eE", "reviewer": "Reviewer_LPfF", "summary": "This paper introduces the Common Corpus, an open dataset for pre-training LLMs. It is constituted of texts that are either uncopyrighted or under permissible licenses, for a total of two trillion tokens across a variety of languages and tasks such as coding, as well as a diversity in terms of regions and time. \n\nThe authors go into the data collection and curation process in great detail, and also train two LLMs of small size on the dataset that show that they perform on a similar level to other models of their size across a variety of datasets.", "questions": "- How are the six collections defined, what are the criteria? \n\n- How are you sure (are you sure) that none of the data is LLM-generated?\n\n- Are the audio transcripts AI- or human- generated?\n\n- How has the set of Wikidata been adapted in natural language? The example provided isn't very clear\n\n- \"Segmentext should work correctly on diverse document formats\" - did you do testing? In general, providing more information about the tools that you developed and how they work would help understand their limitations and applicabliity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Developing an explicitly curated dataset based on data licensing is important and a crucial contribution to the AI community.\n\nThere is an emphasis on diversity in terms of languages and regions.\n\nThe code used for creating the dataset is available, allowing others to reproduce it.\n\nThe resulting dataset can be filtered based on different criteria, including license and language, which makes it useful for developers and researchers working on specific languages or historical periods.\n\nPersonally Identifiable Information is removed with the Presidio tool, which means that there is almost no risk of data leakage from the trained models.", "weaknesses": "- \"Ethical data\" is a very relative/hard-to-define concept -- maybe \"consensual data\" or \"legal data\" would be better alternatives?\n\n- Figure 1 is hard to read because languages such as English, Spanish or French are spoken in multiple places, so simply putting a dot on Madrid or Paris isn't representative of where the language is from \n\n- \"synthetically rewrite the document without the harmful language\" - how is this done and verified? Doesn't introducing synthetically-generated text dilute the corpus?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1sxAsc5X2", "reviewer": "Reviewer_GQ3w", "summary": "The paper introduces Common Corpus, a 2T-token multilingual dataset built entirely from open or permissively licensed sources (public domain, CC, open code). It aims to offer a legally clean, transparent alternative to web-scraped corpora. The authors describe the collection process, cleaning (PII removal, OCR correction, toxicity filtering), and train two small models showing comparable multilingual performance to baselines.", "questions": "- How scalable is the current pipeline to truly support trillion-token multilingual expansion?\n- How do the authors ensure consistent quality across OCRed historical data?\n- Could releasing the filtering tools lead to reproducibility or bias-transfer risks?\n- Do they plan to release validation splits or subsets for standardized benchmarking?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Timely and relevant: strong contribution to open and compliant LLM research.\n- Impressive scale (2T tokens) and careful documentation of provenance and licenses.\n- Multilingual coverage beyond English, rare for open corpora.\n- Clear adherence to emerging best-practice frameworks (e.g., dataset documentation, PII filtering).\n- Demonstrates feasibility through working models and released tools.", "weaknesses": "- Empirical section is limited, only small models and a few benchmarks.\n- No clear comparison to similarly “open” corpora (e.g., Dolma, KL3M) in terms of quality or coverage.\n- Curation process, though detailed, lacks quantitative measures of data quality after filtering.\n- Language balance is heavily skewed to English (~50%).\n- Evaluation of ethical filtering (toxicity, PII accuracy) could be better substantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)