An Expanded Massive Multilingual Dataset for High-Performance Language Technologies
Authors: Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Malik, Farrokh Mehryary, Vladislav Mikhailov, Nikita Moghe, Amanda Myntti, Dayyán O'Brien, Stephan Oepen, Proyag Pal, Jousia Piha, Sampo Pyysalo, Gema Ramírez-Sánchez, David Samuel, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, Tereza Vojtěchová, Jaume Zaragoza-Bernabeu
Accepted at: acl25

An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)

Laurie Burchell1, Ona de Gibert2, Nikolay Arefyev3, Mikko Aulamo2, Marta Bañón4, Pinzhen Chen1, Mariia Fedorova3, Liane Guillou1, Barry Haddow1, Jan Hajic5, Jindřich Helcl6, Erik Henriksson6, Mateusz Klimaszewski1, Ville Komulainen6, Andrey Kutuzov3, Joona Kytoniemi6, Veronika Laippala6, Petter Mæhlum3, Bhavitvya Malik1, Farrokh Mehryary6, Vladislav Mikhailov3, Nikita Moghe7, Amanda Mynttii6, Dayyán O’Brien1, Stephan Oepen3, Proyag Pal1, Jousia Piha6, Sampo Pyysalo6, Gema Ramírez-Sánchez7, David Samuel1, Pavel Stepachev1, Jörg Tiedemann2, Duan Varsí5, Tereza Vojtěchová5, Jaume Zaragoza-Bernabéu4

1University of Edinburgh, 2University of Helsinki, 3University of Oslo, 4Prompsit Language Engineering, 5Charles University, 6University of Turku, 7Amazon *

Contact: https://hplt-project.org

Abstract

Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

1 Introduction

In order to train the state-of-the-art large language models (LLMs) required for modern NLP, large amounts of high-quality textual training data are essential. However, obtaining a sufficient quantity of such data is far from easy. In addition, effective NLP research requires open training data so that results can be replicated and verified.

In this paper, we introduce a new set of text corpora dubbed HPLT v2.1 It is extracted from 4.5 petabytes (PB) of the Internet Archive (IA)2 and Common Crawl (CC)3 data. We build on the work of de Gibert et al. (2024) (hereafter referred to as HPLT v1.2) with an improved extraction pipeline and a much larger set of input crawls to produce the HPLT v2 collection of monolingual and parallel corpora. To our knowledge, our new corpus is the only large-scale text collection extracted from the IA, apart from HPLT v1.2. We release HPLT v2 under the permissive Creative Commons Zero (CC0) license4 and provide the code to replicate our pipeline. Our main contributions can be summarised as:
• We release monolingual corpora covering 193 languages and containing approximately 52 trillion characters and 8 trillion tokens.
• We derive parallel corpora from our monolingual data for 50 languages paired with English, containing over 380 million sentence pairs.
• We make the tools and pipelines used to create the collection openly available.5
• We conduct an in-depth analysis of our data including descriptive statistics, manual inspection, and automatic register labelling.
• We demonstrate the quality of HPLT v2 by using it to train a range of high-performing language and machine translation models.

2 Related work

The increasing data demands of state-of-the-art LLMs have driven a rapid growth in both the number and the size of text corpora. We provide a summary of some well-known collections in Appendix A. Whilst LLMs trained on ostensibly English data have shown impressive multilingual capabilities (Armengol-Estapé et al., 2022), of particular relevance to this work is the growing

1Work was done prior to joining Amazon.
2https://hplt-project.org/datasets/v2.0
3https://archive.org
4https://creativecommons.org/share-your-work/public-domain/cc0/. We do not claim ownership of any of the text from which this data has been extracted.
5https://github.com/hplt-project/HPLT-textpipes
Figure 1: The distribution of documents in the HPLT v2 cleaned dataset by language family and variety. Shortened ISO 639-3 language codes are used here; see the full plots at https://hplt-project.org/datasets/v2.0

shift towards explicitly multilingual corpora. Compared with earlier efforts (e.g. OSCAR (Suárez et al., 2019), CC-100 (Conneau et al., 2020) and mC4 (Xue et al., 2021)), more recent multilingual datasets cover increasing numbers of languages, e.g. CulturaX (Nguyen et al., 2024a) and MADLAD-400 (Kudugunta et al., 2024). HPLT v2 continues this trend by aiming for significant coverage of a wide range of languages. We note that the majority of previous multilingual datasets are sourced from CC, whereas much of HPLT v2 is composed of IA crawls. This means that HPLT v2 can be used in conjunction with these existing datasets as a complementary source.

Producing large-scale datasets by crawling the Web is helpful for scale, but raises questions around dataset quality such as the prevalence of boilerplate, explicit material or non-linguistic content (Kreutzer et al., 2022). One way to tackle low-quality data is through human audit and curation (e.g. ROOTS (Laurençon et al., 2022), Glot500-c (Imani et al., 2023), Serengeti (Adebora et al., 2023) and the MaLA Corpus (Ji et al., 2024)). However, such an approach is difficult to scale. Instead, we ensure the quality of HPLT v2 through a robust dataset construction pipeline (Section 4) and by verifying our data through extensive analysis and downstream evaluation (Sections 5 and 6).

In addition to large-scale monolingual data in multiple languages, HPLT v2 contains high-quality parallel data. Whilst causal language models (CLMs) with decoder-only architectures rely primarily on monolingual data, recent studies have shown that incorporating parallel data during the pretraining stage significantly boosts multilingual, cross-lingual and machine translation (MT) performance for such models (Kale et al., 2021; Briakou et al., 2023; Alves et al., 2024). Because of this, we expect that there is still significant demand for parallel data.

This work is a direct successor of our HPLT v1.2 dataset introduced in (de Gibert et al., 2024). Compared to HPLT v1.2, we feature more data (21 billion vs. 5 billion documents) using an improved pipeline (Section 4), resulting in a significantly larger dataset (52 trillion characters compared to 42 trillion). The HPLT v2 collections are of higher quality than those in HPLT v1.2, as shown through comparative analysis and evaluation (Sections 5 and 6). For clarity, we list the main differences between HPLT v1.2 and HPLT v2 here:
• The size of the source web collections is 2.5x larger, totalling 4.5 petabytes of compressed web data.
• The text extraction pipeline uses Trafilatura rather than warc2text, which results in more efficient boilerplate removal.
• Language identification uses a modified version of OpenLID rather than CLD2, increasing coverage from 75 to 193 language varieties.
• Documents are annotated with their compliance to the robots.txt files of the corresponding websites, which can be used to filter out documents explicitly forbidden for crawling by website owners. The cleaned
variant of the HPLT v2 dataset contains only robots.txt compliant documents.
• Deduplication is done at collection level rather than globally.
• Documents are annotated for PII information.
• Document quality scores computed with web-docs-scorer rather than segment-level language model based scores
• Simpler and more interpretable filtering and cleaning criteria.

3 Dataset description

In this section, we describe the HPLT v2 collection of monolingual and parallel corpora, before explaining how it was constructed in Section 4.

3.1 Monolingual datasets

The monolingual portion of HPLT v2 covers 193 language varieties6 and is published in two variants: ‘deduplicated’ (21 terabytes) and ‘cleaned’ (15 terabytes). In the latter variant, the documents filtered by our cleaning heuristics (see Section 4.2) are excluded. For training LLMs, we recommend using the cleaned variant, but we also publish the datasets before cleaning (“deduplicated”) so that it is possible to apply custom cleaning pipelines to the HPLT v2 data. In total, the deduplicated monolingual HPLT v2 datasets contain approximately 7.6 trillion white-space separated tokens and 52 trillion characters, extracted from 21 billion documents. HPLT v2 is published in the JSONL format, with one document per line.

Figure 1 shows the distribution of documents in the cleaned monolingual data by language families and language variety. Indo-European languages, and especially English, make up the majority of the data. Unfortunately, this is the reality of current web crawls; increasing the amount of data available for other languages is not an easy task and is important future work. Appendix B gives a full breakdown of the statistics of the monolingual data.

3.2 Parallel datasets

We extract parallel data from the monolingual HPLT v2 to cover 50 languages paired with English. We aimed for a diverse range of language varieties and scripts in the low to medium resource range (listed in Table 5). We align these to English since this configuration has the highest potential for finding high-quality parallel data. We release our data in both XML and bitext format.7

Table 1 gives the number of sentence pairs and English words per language prior to filtering (Raw) and after processing (Filtered). We provide both the total over the entire dataset and the median count by language variety. Our results show that the deduplicated HPLT v2 parallel corpora have a 70% reduction in sentence pairs compared to the raw data. The final dataset contains over 380 million sentence pairs, with the English side of the dataset containing over 6 billion words. The median number of sentence pairs by language variety is 4 million, but individual sizes vary greatly by language: the smallest, Sinhala, contains around 273 thousand pairs, whereas the largest, Finnish, contains over 29 million pairs. We give full statistics for each included language variety in Table 5 in the appendix.

We assume the large number of Finnish sentence pairs is due to the pipeline’s bias toward European languages. In contrast, languages such as Japanese and Korean, which we would expect to have larger corpora, may have lower counts because of lower-quality monolingual data and limited support in key pipeline components such as sentence splitting and tokenization. This results in reduced yields during data cleaning and filtering for non-European languages written in non-Latin scripts.

DocHPLT v2  We also introduce context, by providing the documents that our bitext data comes from. These documents are annotated with both sentence and paragraph alignment. Overall, we provide 74,078,581 aligned documents covering all 51 languages. This ranges from 38,199,407 documents in English to 20,448 documents in Xhosa.8

MultiHPLT v2  We leverage the English-centric HPLT v2 parallel resources to further create a multi-

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">Raw</th>
    <th colspan="2">Filtered</th>
  </tr>
  <tr>
    <th>Pairs</th>
    <th>Eng. words</th>
    <th>Pairs</th>
    <th>Eng. words</th>
  </tr>
  <tr>
    <td>Total</td>
    <td>1277M</td>
    <td>16849M</td>
    <td>380M</td>
    <td>6780M</td>
  </tr>
  <tr>
    <td>Median</td>
    <td>11M</td>
    <td>170M</td>
    <td>4M</td>
    <td>80M</td>
  </tr>
</table>

Table 1: Counts in millions (M) of sentence pairs and English words in the parallel HPLT v2 data before filtering (Raw) and after filtering and deduplication (Filtered), both in total and the median across all languages.

6Language varieties are labelled with an ISO 639-3 code denoting the variety plus an ISO 15924 four-letter code denoting the script, separated by an underscore: e.g., gla_Latn.

7https://opus.nlpl.eu/HPLT/corpus/version/HPLT

8https://opus.nlpl.eu/legacy/DocHPLT.php
way parallel corpus, obtained by pivoting via English sentences. This corpus includes 1275 language pairs and contains over 16.7 billion parallel sentences.9

A similar pivoting is also done for DocHPLT and we create the additional bitexts through sentences aligned to the same English document. However, anchoring sentences in aligned documents creates a different subset across the pivoted language pairs.

4 Dataset construction

In the following section, we explain the dataset construction pipeline for HPLT v2. We first extract text from web crawls via HTML (Section 4.1), deduplicate and clean this monolingual text (Section 4.2), and finally extract and process the parallel data (Section 4.3). Figure 2 provides a high-level overview of the pipeline.

4.1 Text extraction from web crawls

Sources In total, we ingest 4.5 PB of web crawl data to build HPLT v2. 3.7 PB is sourced from IA from crawls conducted mostly between 2012 and 2020, with the remaining 0.8 PB coming from CC. We use CC crawls conducted mostly between 2014 and 2022. A detailed description of the crawls we use is in Appendix C.

Extracting HTML Both IA and CC crawls are provided in the Web ARChive (WARC) format10 which stores HTTP requests and responses between a web crawler and web servers. We use the warc2text tool11 to extract HTML and related metadata from these WARC files. It selects relevant WARC records containing HTML pages, removes documents from a list of known trash websites,12 and finally saves the results in the ZSTD-compressed JSONL format. The extracted metadata includes document URLs, paths to the original WARC files and record positions inside, content types and timestamps. Additionally, WARC records with URLs ending with "robots.txt" are stored for later use in filtering.

Extracting text This stage of the pipeline extracts the main textual content from HTML pages and groups it into language-specific subsets. It first parses the HTML pages into a tree representation. Next, it removes likely machine-translated texts by searching for indicative HTML tags and attributes. It then removes boilerplate (i.e. parts of a web page that do not contribute to its main content) using Trafilatura 1.8.0 (Barbaresi, 2021). Following hyperparameter experimentation, we set include_comments=False, include_tables=False, no_fallback=False and MIN_EXTRACTED_SIZE=0, with all other hyperparameters set to their defaults. We chose not to use fallback to Trafilatura’s simple extraction baseline since it leaves most boilerplate intact, and we preferred sacrificing some documents but avoiding extra boilerplate in HPLT v2. Finally, we predict the language of the text using a modified version of the OpenLID model13 (Burchell et al., 2023; Burchell, 2024), where the Arabic dialects are combined under one macrolanguage label and the model training data has undergone improved pre-processing. These changes are intended to improve classification reliability. After text extraction, the dataset size reduces to 62 TB, 15 times smaller than the HTML data and 75 times smaller than the original web crawls.

Figure 2: Overview of the data acquisition and processing pipeline for HPLT v2.

4.2 Monolingual text processing

Following text extraction, we proceed to monolingual text cleaning in which we apply various criteria to select the cleanest documents. With the exception of fixing encoding, we do not alter the text in this process.

We first discard all documents for which the predicted probability of the language label is < 0.5.

9https://opus.nlpl.eu/MultiHPLT/corpus/version/MultiHPLT
10https://www.iso.org/standard/68004.html
11https://github.com/bitextor/warc2text
12Mostly containing auto-generated lists of phone numbers, addresses, etc.: https://github.com/paracrawl/cirrus-scripts/blob/master/url-filter-list.annotated
13https://data.statmt.org/lid/lid193_merged_arabics.bin
We then perform crawl-level deduplication with a MinHash index (Broder et al., 1998), using 240 hashes and a Jaccard similarity threshold of 0.8. We keep one document from each computed disjoint-set (Galler and Fischer, 1964), thus removing near-duplicates within each crawl.

To respect robots.txt14 rules specified by each domain, we use the extracted robots.txt records to identify patterns disallowing the crawlers we use.15 We use the fst16 tool to create a compressed index of URLs to exclude and use it to remove documents originating from these URLs.

We then use a range of heuristics to discard low-quality documents. We calculate a document quality score using Web Docs Scorer (WDS),17 discarding documents with a score < 5. We remove any documents where the length of the document is < 500 characters, or where the average number of words per segment is < 5 (< 10 characters for Japanese, Chinese or Korean). We also filter documents where the URL is in the UTI adult list.18

Finally, we enrich documents with additional metadata. We add a unique identifier for the document hash derived from the WARC file name, the URL and the timestamp. We also carry out segment-level language identification (LID) using the Rust port19 of HeLI-OTS (Jauhiainen et al., 2022), trained on the OpenLID dataset. Finally, we add the Unicode character offsets of any personally identifiable information found by the PII tool.20

Although the CC crawls are less than 20% of the input data, they are the source of about 60% of the final text. This is likely because CC focuses on textual content whereas IA includes much multimedia content, resulting in 4-8x lower yields in general. However, for some languages (e.g. Chinese, Persian, and a few smaller languages), IA provides more texts than CC. Appendix D presents a detailed study of the contributions of different source crawls to the final dataset.

4.3 Parallel data extraction

Our parallel data extraction pipeline is adapted from Bitextor.21 We make the following changes to increase the quality of the final dataset:
• Input data comes from cleaned monolingual HPLT v2 rather than WARCs.
• We use Loomchild, a SRX-based sentence splitter (Milkowski and Lipski, 2011), to cover more languages.
• During sentence splitting, paragraph and sentence identifiers are added as persistent metadata through the pipeline.
• Minimal length rule and fluency filtering in bicleaner-hardrules are disabled as this duplicates other processing steps.
• Bicleaner AI (Zaragoza-Bernabeu et al., 2022) uses a multilingual model able to handle unseen language pairs during training.
• Document-level output from the document-matching step is collected to allow the creation of document-level parallel data.
To avoid the possible introduction of new bugs in the pipeline, given that many of the steps in it are made for 2-letter language codes, we convert 3-letter language codes to 2-letter before processing.

5 Data analysis

In this section, we present our analysis of the HPLT v2 data based on indirect quality indicators, manual inspection, and register labels.

5.1 Indirect quality indicators

We consider two types of indirect quality indicators: descriptive statistics and website domains.

Descriptive statistics  We calculate descriptive statistics for HPLT v2 using the HPLT Analytics tool.22 We compare to the cleaned HPLT v1.2 dataset (our previous release).

For the monolingual data, there are far more unique segments (22.2% of HPLT v1.2 vs. 40.9% of HPLT v2) but far fewer documents longer than 25 segments (90.8% vs. 23.2%). Similarly, the proportion of short segments is reduced (39.6% vs. 13.3%). These changes can be attributed to the use of Trafilatura and WDS. More segments match the document language (58.6% vs. 81.5%), driven by improvements in LID accuracy and a more aggressive document filtering strategy. Finally, we iden-

14https://www.robotstxt.org/
15a, CCBot, ia-archiver
16https://burntsushi.net/transducers/
17https://github.com/pablopl6n/web-docs-scorer/
18https://dsi.ut-capitole.fr/blacklists
19https://github.com/ZJaume/heliport
20https://github.com/mmanteli/multilingual-PII-tool
21https://github.com/bitextor/bitextor
22https://github.com/hplt-project/data-analytics-tool
tify frequent \( n \)-grams and find that a substantial amount of textual boilerplate remains, particularly from Wikipedia and blogging platforms.

Regarding parallel data, we find that the number of source and target tokens per language pair is much higher in HPLT v2 (by 47% and 49% on average) than in HPLT v1.2. Furthermore, 80% of the sentence pairs have a translation likelihood score from 0.8 to 1 (as computed by Bicleaner AI) which attests to their high quality. The frequent \( n \)-grams in the parallel datasets are similar among all languages: larger datasets tend to focus on hotels and legal notices, whereas the smaller datasets exhibit more variety and the frequent \( n \)-grams in these datasets reflect local content likely from news websites, e.g. political figures and place names. Appendix D contains further examples.

Domains We explore the website domain names and geographic top-level domains (TLDs) present in the data in order to understand its origins better.

We find different patterns of website domain names in the corpora depending on the size of the language dataset. Languages with more data available contain a diverse range of website domain names in the monolingual data but more travel-related webpages in the parallel data. However, smaller language datasets tend to contain more Wikipedia and religious content in both the monolingual and parallel data. Appendix F contains further information about common domains.

Whilst most of the TLDs in our dataset are general purpose (e.g., .com, .org), we found that the most common geographic TLDs in the monolingual language corpora were usually from the country with the most speakers. This gave us confidence in the reliability of the text. We found the proportion of geographic TLDs from an indicative country was highest for those in Europe, whereas the datasets for many languages primarily spoken in Africa mostly consisted of general-purpose TLDs. The parallel data exhibits more diversity in TLDs than the monolingual data. For example, .eu is much more frequent, appearing in the top-10 TLDs of all mid-size and large parallel datasets of nearly all European languages. A more detailed discussion of our observations is in Appendix G.

5.2 Manual data inspection

To assess human-perceived quality, we manually inspected a random sample of documents from the cleaned monolingual datasets in 22 languages. Specifically, for each language spoken by the authors, we sampled 50 random documents extracted from each of the four groups of crawls: the older CC/IA crawls from 2012–2014, and the newer CC/IA crawls from 2017–2020. The main goal of this stratification was to compare the quality of texts we get depending on the crawl source and age, and select the most promising crawls for the next release of our datasets.

We asked participants to annotate any documents which look like pornographic content, look unnatural, and/or are not in the target language. Appendix H describes the inspection procedure and the results. Overall, for most languages, both the proportions of pornographic content and texts not in the target language are around 0–3%, with no significant difference between groups of crawls. Asturian, Scottish Gaelic and Norwegian Nynorsk are notable exceptions, with 31, 11 and 7 percent of texts not in the target language respectively. The proportion of unnatural texts is around 10% on average and up to 30% for some languages, leaving space for improvements. We also observe that the probability of getting an unnatural text from the newer CC crawls is roughly half of that of the other three inspected groups of crawls. This is probably related to the introduction of harmonic centrality ranking for domain prioritization in the CC crawler queue since 2017 (Nagel, 2023), which is stated to be more efficient in avoiding spam compared to the previously used techniques.

5.3 Register labels

As noted in Section 5.1, web crawls cover a vast range of different kinds of documents from various sources. We use automatic register (or genre) classification to create metadata about this variation, allowing users to make informed decisions when sampling from the data.

We use the multilingual register classifier described in Henriksson et al. (2024) to label the entire monolingual HPLT v2. This classifier covers 16 languages and is based on an XLM-R Large model (Conneau et al., 2020), fine-tuned on a multilingual web corpus manually annotated with register information. The classifier employs a hierarchical taxonomy with 25 register classes organized into 9 main categories (listed in Table 2). This label scheme is specifically designed for the linguistic characteristics of web texts and is therefore well-suited for our dataset.

The system achieves a mean micro F1 score of
<table>
  <tr>
    <th>Register</th>
    <th>Percentage</th>
  </tr>
  <tr>
    <td>How-to Interactive (HI)</td>
    <td>1.8 %</td>
  </tr>
  <tr>
    <td>Interactive Discussion (ID)</td>
    <td>6.5 %</td>
  </tr>
  <tr>
    <td>Informative Description (IN)</td>
    <td>27.1 %</td>
  </tr>
  <tr>
    <td>Informative Persuasion (IP)</td>
    <td>10.8 %</td>
  </tr>
  <tr>
    <td>Lyrical (LY)</td>
    <td>0.5 %</td>
  </tr>
  <tr>
    <td>Machine Translated (MT)</td>
    <td>3.3 %</td>
  </tr>
  <tr>
    <td>Narrative (NA)</td>
    <td>18.1 %</td>
  </tr>
  <tr>
    <td>Opinion (OP)</td>
    <td>5.4 %</td>
  </tr>
  <tr>
    <td>Spoken (SP)</td>
    <td>0.2 %</td>
  </tr>
  <tr>
    <td>Multiple labels</td>
    <td>23.6 %</td>
  </tr>
  <tr>
    <td>No label</td>
    <td>2.5 %</td>
  </tr>
</table>

Table 2: Register label distribution in HPLT v2 English dataset for classification threshold 0.4. See Henriksson et al. (2024) for the full scheme and explanation of the contents of the classes.

77% on the 5 languages used during fine-tuning. It also demonstrates good performance for 11 unseen languages, with a mean micro F1 score of 66%. These results allow us to extend register labelling to a broad range of languages, though we limit predictions to languages within the 100 languages covered by XLM-RoBERTa.

We provide the classification certainty as well as the label, so that the threshold can be optimized by use case. Table 2 presents the distribution for register labels in our English data for a classification threshold of 0.4. Further work could use our derived labels to improve dataset quality, by e.g. filtering out MT content.

6 Empirical evaluation

In this section, we describe our empirical evaluation of the quality of the HPLT v2 monolingual corpora. We conduct this evaluation by employing the datasets as training material for several natural language processing (NLP) models.

6.1 Basic linguistic tasks and MLMs

We train masked language models (MLMs) on 52 different languages from the HPLT v2 datasets, choosing those with available benchmarks.23 We use LTG-BERT (Samuel et al., 2023) to allow comparison with HPLT v1.2. We give full details about LTG-BERT in Appendix J.

We evaluate the trained MLMs on part-of-speech tagging, lemmatisation and dependency parsing using the Universal Dependencies (UD) treebanks (de Marneffe et al., 2021), as well as named entity recognition (NER) using WikiAnn datasets (Pan et al., 2017). We compare to mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) models as multilingual baselines, and to HPLT v1.2 BERT models24 as monolingual baselines. The performance is measured using the official CoNLL 2018 evaluation code (Zeman et al., 2018) for the UD tasks, and seqeval (Nakayama, 2018) balanced F1 score for the NER task.

Figure 3 shows the win rates achieved by the models for the four tasks ('win rate' here is the number of languages on which a given model outperforms other models). Models trained on the HPLT v2 datasets show a considerably higher win rate compared to the baselines in all the tasks except lemmatisation, where XLM-R and HPLT v1.2 yield competitive results. However, we note that the difference between XLM-R, HPLT v1.2 and HPLT v2 on the lemmatisation task is less than 1% of accuracy, meaning that no model significantly outperforms any other. Detailed scores by language and task are to be found in Table 15. We make the HPLT v2 BERT models with intermediate checkpoints publicly available.25

![Win rates for MLMs at part-of-speech tagging, lemmatisation, dependency parsing, and named entity recognition.](page_1012_186_388_282.png)

Figure 3: Win rates for MLMs at part-of-speech tagging, lemmatisation, dependency parsing, and named entity recognition.

23https://github.com/hplt-project/HPLT-WP4
24https://hf.co/collections/HPLT/hplt-bert-models-6625a8f3e0f8ed1c9a4fa96d
25https://hf.co/collections/HPLT/hplt-20-bert-models-67ba52ae96b1fb8aae673493
for one high-resource and one low-resource language: English and Norwegian. We train 1.7B decoder-only LMs using 100B/30B tokens sampled from the English/Norwegian parts of our HPLT v2 dataset respectively. We compare our English and Norwegian models with models trained on same-sized samples of HPLT v1.2 (de Gibert et al., 2024) and FineWeb (Penedo et al., 2024a), and additionally compare our Norwegian models with FineWeb-2 (Penedo et al., 2024b), CulturaX (Nguyen et al., 2024b), and mC4 (Xue et al., 2021). We replicate the design by Penedo et al. (2024a) and train the models with a fixed pretraining setup except for the pretraining corpus (English: four corpora; Norwegian: five corpora). We provide full details on pretraining and evaluation in Appendix I and describe our key results below.

Figure 4: Performance comparison of the trained generative LMs on English.

English Average results over the English benchmarks are presented in Figure 4. Our models trained on the cleaned HPLT v2 datasets reach similar performance to the models trained on FineWeb data in downstream tasks, and considerably outperform the models trained on HPLT v1.2 and on the deduplicated subset of HPLT v2. This implies that our cleaning approach has successfully improved the data quality with respect to these benchmarks.

Norwegian Average normalized scores over the Norwegian tasks are shown in Figure 5. We observe that the Norwegian models trained on FineWeb, CulturaX, and mC4 perform on par with HPLT v2 and outperform those trained on HPLT v1.2. Performance gains start to level off after 16B tokens, with the FineWeb and HPLT v2 scores being more stable during pretraining. This suggests that CulturaX, FineWeb, and HPLT v2 are more effective corpora for Norwegian, and their mixtures potentially provide further benefits.

![Performance comparison of the trained generative LMs on Norwegian.](page_820_120_601_377.png)

Figure 5: Performance comparison of the trained generative LMs on Norwegian.

mixtures potentially provide further benefits.

6.3 Machine translation tasks

We extrinsically evaluate the quality of the HPLT v2 parallel data by measuring the performance of MT models trained on it. This section considers two settings:
• We compare HPLT v2 to its predecessor, HPLT v1.2, where we measure their performance in 10 overlapping languages.
• We assess HPLT v2 as a complementary dataset to existing MT resources. We compare models in three data conditions: 1) solely on HPLT v2 parallel data; 2) on the data from Tatoeba Challenge (which includes most of the OPUS collection for each direction Tiedemann, 2012, 2020); and 3) on a combination of the two.

We carry out training for each individual language pair (to and from English) using the Transformer-base architecture (Vaswani et al., 2017) and the Marian NMT toolkit (Junczys-Dowmunt et al., 2018). Data processing and training are streamlined with OpusPocus26 following the configuration of Arefyev et al. (2024). We evaluate all models on the FLORES-200 benchmark (NLLB Team et al., 2024) using BLEU (Papineni et al., 2002), chrF++ (Popović, 2017), and COMET-22-DA (Rei et al., 2022). We use sacrebleu’s implementation of BLEU27 and chrF++28 with signatures footnoted (Post, 2018). Full results for each metric and each direction are given in Table 16 and Table 17 in Appendix K.

We report average BLEU and COMET scores in Table 3. It is worth noting that we only average

26 https://github.com/hplt-project/OpusPocus
27 nrefs:1|case:mixed|eff:off|smooth:exp|version:2.5.1, and where applicable: tok: ja-mecab, tok: ko-mecab, or tok:13a
28 nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">xx-en</th>
    <th colspan="2">en-xx</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>COMET</th>
  </tr>
  <tr>
    <td>HPLT v1.2</td>
    <td>28.5</td>
    <td>0.7943</td>
    <td>24.4</td>
    <td>0.7623</td>
  </tr>
  <tr>
    <td>HPLT v2</td>
    <td><b>32.7</b></td>
    <td><b>0.8343</b></td>
    <td><b>27.9</b></td>
    <td><b>0.8137</b></td>
  </tr>
  <tr>
    <td>HPLT v2</td>
    <td>28.7</td>
    <td>0.8144</td>
    <td>23.4</td>
    <td>0.7941</td>
  </tr>
  <tr>
    <td>OPUS</td>
    <td>29.6</td>
    <td>0.8142</td>
    <td>23.4</td>
    <td>0.8074</td>
  </tr>
  <tr>
    <td>HPLT v2+OPUS</td>
    <td><b>30.5</b></td>
    <td><b>0.8237</b></td>
    <td><b>24.2</b></td>
    <td><b>0.8083</b></td>
  </tr>
</table>

Table 3: Average scores for the HPLT v1.2 and v2 comparison (top) and HPLT v2 as a complimentary resource to OPUS (bottom). Only numbers that are available for all models in a comparison are averaged.

numbers for translation directions that are covered by all models in a comparison. The top half features a clear advantage of HPLT v2 over HPLT v1.2, reflecting the effect of our improved data extraction pipeline discussed in Section 4. Results in the bottom of Table 3 shows that HPLT v2 and OPUS perform comparably. However, combining the two datasets leads to improvements in both BLEU and COMET. This indicates that HPLT v2 contains non-overlapping content relative to existing OPUS corpora, making it a valuable complementary resource for MT.

7 Conclusions and future work

This paper introduced the HPLT v2 dataset, a large-scale multilingual collection of openly-available monolingual and parallel web-crawled data. Building on our previous HPLT v1 effort, we focused on improving the quality of data available for a wider range of languages, and we make our data processing pipelines publicly available for easy reuse. We presented extensive data analysis as well as intrinsic and extrinsic evaluation, demonstrating the value of HPLT v2 for various NLP tasks.

Further work will focus on expanding language coverage and data quality, particularly for under-served languages. We also plan to conduct more experiments on the document-level aligned parallel corpus.

Limitations

Like many large-scale corpora, the majority of the data in HPLT v2 is in Indo-European languages, especially English, and the parallel data is English-centric. To an extent, this is a result of the dominance of these languages in the source web-crawl data. In addition, the evaluation in the paper only covers a subset of the languages in HPLT v2 due to a lack of resources for all languages present. We hope that the data we release in multiple under-served languages will be used to improve language technologies for more communities.

Whilst we focus on improving the HPLT v2 data processing pipeline, there are still residual errors in the final dataset in LID, boilerplate removal (particularly Wiki* boilerplate) and other cleaning steps. We make the code for our pipeline available to facilitate its evaluation and improvement. We note that there is only limited removal of machine-generated content in HPLT v2 (i.e., content generated by technologies like MT and LLMs), as detecting such content remains a difficult task (Yang et al., 2024).

It is possible that some of the test data we use for evaluation is contained within HPLT v2 (for example, the Wikipedia-based test set for named entity recognition). Nevertheless, we believe that the results reported in Section 6 are still indicative of the quality of HPLT v2, since the large-scale datasets we compare against are likely to have similar contamination issues.

During the evaluation, we discovered that the punctuation for Chinese languages (and probably Korean and Japanese) in HPLT v2 had been normalised incorrectly to its Latin equivalent, causing a drop in measured performance for languages in this script. We will fix this in the next iteration of the HPLT pipeline.

Ethical considerations

We source our data from web crawls and since Internet text is largely unregulated, our final dataset may contain harmful content or amplify existing biases, despite the extensive filtering applied to mitigate these issues. One notable bias is the over-representation of religious content in smaller language corpora, which could lead to models trained on this data being biased towards this particular domain.

Another pressing ethical consideration is the significant environmental impact of producing large-scale datasets. We mitigate this impact by making the data openly available in multiple formats, limiting the need to reproduce the processing pipeline.

We report the estimated CPU and GPU cost in hours for our work to allow for more informed decision-making in future research efforts:
• WARC to HTML extraction: 250K CPU
• HTML to text extraction: 1.7M CPU
• Monolingual data processing: 600K CPU
• Parallel data cleaning and deduplication:
1.8M CPU and 23K GPU
• Register labels classification: 36.7K GPU
• MLM experiments: 1.8K CPU and 4K GPUs
• Generative LMs training and evaluation: 21.5K CPU and 43K GPU
• MT models training and evaluation: 20K GPU
The total amount of hours spent would be roughly 4.4M CPU hours and 106K GPU hours. The most expensive task is the evaluation of our data through generative model training. We mitigate the environmental impact of our work by using one of the most eco-efficient HPC clusters in the world (LUMI) to carry out much of our computation. The LUMI supercomputer uses renewable, carbon-neutral energy.29

Acknowledgements

We acknowledge Tsz Kin Lam, Otto Tarkka, and Maryam Teimouribadelehdaeh for their annotation work; we thank Jelmer van der Linde for his help with Go.

This project has received funding from the European Union’s Horizon Europe research and innovation programme under Grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]. The contents of this publication are the sole responsibility of its authors and do not necessarily reflect the opinion of the European Union.

The authors wish to thank CSC - IT Center for Science, Finland for computational resources and support. The project has also been supported by the Czech MEYS project No. CZ.02.01.01/00/23_025/0008691 and Research Infrastructure project LM2023062.

References

Ife Adebara, AbdelRahim Elmaldany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. 2023. SERENGETI: Massively multilingual language models for Africa. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1498–1537, Toronto, Canada. Association for Computational Linguistics.

Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An Open Multilingual Large Language Model for Translation-Related Tasks. arXiv preprint.

Nikolay Arefyev, Mikko Aulamo, Pinzhen Chen, Ona De Gibert Bonet, Barry Haddow, Jindřich Helcl, Bhavitvya Malik, Gema Ramírez-Sánchez, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, and Jaume Zaragoza-Bernabeu. 2024. HPLT’s first release of data and models. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2), pages 53–54, Sheffield, UK. European Association for Machine Translation (EAMT).

Jordi Armengol-Estapé, Ona de Gibert Bonet, and Maite Melero. 2022. On the multilingual capabilities of very large-scale English language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3056–3068, Marseille, France. European Language Resources Association.

Adrien Barbaresi. 2021. Trafilatura: A web scraping library and command-line tool for text discovery and extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 122–131, Online. Association for Computational Linguistics.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqua: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.

Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in PaLM’s translation capability. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9432–9452, Toronto, Canada. Association for Computational Linguistics.

Andrei Z. Broder, Moses Charikar, Alan M. Frieze, and Michael Mitzenmacher. 1998. Min-wise independent permutations (extended abstract). In Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing, STOC ’98, page 327336, New York, NY, USA. Association for Computing Machinery.

Laurie Burchell. 2024. Improving natural language processing for under-served languages through increased training data diversity. Ph.D. thesis, University of Edinburgh.

Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. 2023. An open dataset and model for language identification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 865–879, Toronto, Canada. Association for Computational Linguistics.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question

29https://lumi-supercomputer.eu/sustainable-future/
answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. 2024. A new massive multilingual dataset for high-performance language technologies. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1116–1128, Torino, Italia. ELRA and ICCL.

Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305.

Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation.

Bernard A. Galler and Michael J. Fischer. 1964. An improved equivalence algorithm. Commun. ACM, 7:301–303.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noa’ch, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Erik Henriksson, Amanda Myntti, Saara Hellström, Anni Eskelinen, Selcen Erten-Johansson, and Veronika Laippala. 2024. Automatic register identification for the open web using multilingual deep learning. Preprint, arXiv:2406.19892.

Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023. Glot500: Scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1082–1117, Toronto, Canada. Association for Computational Linguistics.

Sardana Ivanova, Fredrik Andreassen, Matias Jentoft, Sondre Wold, and Lilja Øvrelied. 2023. NorQuAD: Norwegian question answering dataset. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 159–168, Tórshavn, Faroe Islands. University of Tartu Library.

Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2022. HeLI-OTS, off-the-shelf language identifier for text. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3912–3922, Marseille, France. European Language Resources Association.

Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyan O’Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, and Barry Haddow. 2024. EMMA-500: Enhancing massively multilingual adaptation of large language models. arXiv preprint arXiv:2409.17892.

Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121, Melbourne, Australia. Association for Computational Linguistics.

Mihir Kale, Aditya Siddhant, Rami Al-Rfou, Linting Xue, Noah Constant, and Melvin Johnson. 2021. nmT5 - is parallel data still relevant for pre-training massively multilingual language models? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 683–691, Online. Association for Computational Linguistics.

Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nesanbayar Ulzii-Orshikh, Allahsena
Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50–72.

Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2024. Madlad-400: A multilingual and document-level large audited dataset. Advances in Neural Information Processing Systems, 36.

Hynek Kydlíček, Guilherme Penedo, Clémentine Fourier, Nathan Habib, and Thomas Wolf. 2024. FineTasks: Finding signal in a haystack of 200+ multilingual tasks.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Fonferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809–31826.

Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2024. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3245–3276, Mexico City, Mexico. Association for Computational Linguistics.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.

Vladislav Mikhailov, Tita Enstad, David Samuel, Hans Christian Farsethås, Andrey Kutuzov, Erik Vellido, and Lilja Øvrelid. 2025a. NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark. arXiv preprint arXiv:2504.07749.

Vladislav Mikhailov, Petter Mehlem, Victoria Ovedie Chruckshank Langø, Erik Vellido, and Lilja Øvrelid. 2025b. A collection of question answering datasets for Norwegian. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 397–407, Tallinn, Estonia. University of Tartu Library.

Marcin Miłkowski and Jarosław Lipski. 2011. Using srx standard for sentence segmentation. In Human Language Technology. Challenges for Computer Science and Linguistics, pages 172–182, Berlin, Heidelberg. Springer Berlin Heidelberg.

Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena. arXiv preprint arXiv:2406.07545.

Sebastian Nagel. 2023. Common Crawl: Data collection and use cases for NLP. HPLT & NLPL Winter School on Large-Scale Language Modeling and Neural Machine Translation with Web Data, February, 6.

Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from https://github.com/chakki-works/seqeval.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024a. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4226–4237, Torino, Italia. ELRA and ICCL.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024b. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4226–4237, Torino, Italia. ELRA and ICCL.

NLLB Team et al. 2024. No language left behind: Scaling human-centered machine translation. Nature, 630(8018):841.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Guilherme Penedo, Hynek Kydlíček, Loubna Ben alal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024a. The FineWeb datasets: Decanting the web for the finest text data at scale. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Guilherme Penedo, Hynek Kydlicek, Vinko Sabolec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. 2024b. FineWeb2: A sparkling update with 1000s of languages.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only. In Advances in Neural Information Processing Systems, volume 36, pages 79155–79172. Curran Associates, Inc.

Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

David Samuel. 2023. Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 221–237, Singapore. Association for Computational Linguistics.

David Samuel, Andrey Kutuzov, Lilja Øvrelid, and Erik Velldal. 2023. Trained on 100 million words and still in shape: BERT meets British National Corpus. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1954–1974, Dubrovnik, Croatia. Association for Computational Linguistics.

Shaden Smith, Mostafa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model. Preprint, arXiv:2201.11990.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand. Association for Computational Linguistics.

Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Leibniz-Institut für Deutsche Sprache.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).

Jörg Tiedemann. 2020. The tatoeba translation challenge – realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174–1182, Online. Association for Computational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.

Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Ruth Petzold, William Yang Wang, and Wei Cheng. 2024. A survey on detection of LLMs-generated content. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9786–9805, Miami, Florida, USA. Association for Computational Linguistics.
Jaume Zaragoza-Bernabeu, Gema Ramírez-Sánchez, Marta Bañón, and Sergio Ortiz Rojas. 2022. Bicleaner AI: Bicleaner goes neural. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 824–831, Marseille, France. European Language Resources Association.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21, Brussels, Belgium. Association for Computational Linguistics.
A  Comparison of multilingual collections

<table>
  <tr>
    <th>Dataset</th>
    <th>Size (TB)</th>
    <th>Tokens (T)</th>
    <th>Langs</th>
    <th>% English</th>
    <th>Source</th>
  </tr>
  <tr>
    <th colspan="6">English only</th>
  </tr>
  <tr>
    <td>The Pile (<a href="https://arxiv.org/abs/2012.04877">Gao et al., 2020</a>)</td>
    <td>0.8</td>
    <td>0.39</td>
    <td>1</td>
    <td>100</td>
    <td>various (c.f Section 2)</td>
  </tr>
  <tr>
    <td>C4.en (<a href="https://arxiv.org/abs/2010.15679">Raffel et al., 2020; Dodge et al., 2021</a>)</td>
    <td>0.3</td>
    <td>0.16</td>
    <td>1</td>
    <td>100</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <td>RefinedWeb (<a href="https://arxiv.org/abs/2302.07694">Penedo et al., 2023</a>)</td>
    <td>2.8</td>
    <td>0.6</td>
    <td>1</td>
    <td>100</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <td>Dolma-Web (<a href="https://arxiv.org/abs/2401.07785">Soldaini et al., 2024</a>)</td>
    <td>-</td>
    <td>2.28</td>
    <td>1</td>
    <td>100</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <td>FineWeb (<a href="https://arxiv.org/abs/2401.07785">Penedo et al., 2024a</a>)</td>
    <td>-</td>
    <td>15</td>
    <td>1</td>
    <td>100</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <th colspan="6">Multilingual</th>
  </tr>
  <tr>
    <td>OSCAR-23.01 (<a href="https://arxiv.org/abs/1910.01108">Suárez et al., 2019</a>)</td>
    <td>-</td>
    <td>1.1</td>
    <td>153</td>
    <td>48.43</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <td>CC-100 (<a href="https://arxiv.org/abs/2009.04895">Conneau et al., 2020</a>)</td>
    <td>2.39</td>
    <td>0.3</td>
    <td>100</td>
    <td>18.84</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <td>mC4 (<a href="https://arxiv.org/abs/2106.04227">Xue et al., 2021</a>)</td>
    <td>-</td>
    <td>6.3</td>
    <td>101</td>
    <td>5.67</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <td>ROOTS (<a href="https://arxiv.org/abs/2204.07647">Laurençon et al., 2022</a>)</td>
    <td>1.6</td>
    <td>0.4</td>
    <td>46</td>
    <td>30.03</td>
    <td>BigScience Catalogue Data, Common Crawl, OSCAR</td>
  </tr>
  <tr>
    <td>Glot500-c (<a href="https://arxiv.org/abs/2301.05177">Imani et al., 2023</a>)</td>
    <td>0.6</td>
    <td>-</td>
    <td>511</td>
    <td>*2.16</td>
    <td>various (c.f. Appendix C)</td>
  </tr>
  <tr>
    <td>Serengeti (<a href="https://arxiv.org/abs/2301.05177">Adebare et al., 2023</a>)</td>
    <td>0.042</td>
    <td>-</td>
    <td>517</td>
    <td>-</td>
    <td>various (c.f. Appendix C)</td>
  </tr>
  <tr>
    <td>CulturaX (<a href="https://arxiv.org/abs/2401.07785">Nguyen et al., 2024a</a>)</td>
    <td>27</td>
    <td>6.3</td>
    <td>167</td>
    <td>45.13</td>
    <td>OSCAR, mC4</td>
  </tr>
  <tr>
    <td>MADLAD-400-clean (<a href="https://arxiv.org/abs/2401.07785">Kudugunta et al., 2024</a>)</td>
    <td>-</td>
    <td>2.6</td>
    <td>419</td>
    <td>50</td>
    <td>Common Crawl</td>
  </tr>
  <tr>
    <td>MaLA (<a href="https://arxiv.org/abs/2401.07785">Ji et al., 2024</a>)</td>
    <td>-</td>
    <td>0.074</td>
    <td>939</td>
    <td>4</td>
    <td>various (c.f. Section 2.1.4)</td>
  </tr>
  <tr>
    <td>monoHPLT v1.2-dedup (<a href="https://arxiv.org/abs/2401.07785">de Gibert et al., 2024</a>)</td>
    <td>11</td>
    <td>5.6</td>
    <td>75</td>
    <td>41</td>
    <td>Common Crawl, Internet Archive</td>
  </tr>
  <tr>
    <td>HPLT v2 (monolingual, deduplicated)</td>
    <td>21</td>
    <td>7.6</td>
    <td>193</td>
    <td>44</td>
    <td>Common Crawl, Internet Archive</td>
  </tr>
</table>

Table 4: Comparison of selected massively multilingual collections of monolingual data listed in chronological order. We report size, token counts, language coverage, and the proportion of English content. - indicates that data is not available. * indicates that the English percentage was computed over sentence counts, instead of token counts.
B  Parallel and monolingual data statistics

<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="2">Raw</th>
    <th colspan="2">Filtered</th>
    <th colspan="2">TMX</th>
  </tr>
  <tr>
    <th>Sentence Pairs</th>
    <th>English Words</th>
    <th>Sentence Pairs</th>
    <th>English Words</th>
    <th>Sentence Pairs</th>
    <th>English Words</th>
  </tr>
  <tr><td>sin_Sinh</td><td>929,844</td><td>15,647,062</td><td>450,122</td><td>8,248,007</td><td>273,430</td><td>5,932,234</td></tr>
  <tr><td>npi_Deva</td><td>1,058,740</td><td>18,514,145</td><td>523,022</td><td>10,176,931</td><td>317,120</td><td>7,145,363</td></tr>
  <tr><td>xho_Latn</td><td>1,223,514</td><td>16,524,728</td><td>655,790</td><td>9,339,359</td><td>405,605</td><td>5,998,358</td></tr>
  <tr><td>mal_Mlym</td><td>1,686,113</td><td>25,600,791</td><td>795,653</td><td>12,475,940</td><td>547,168</td><td>9,656,086</td></tr>
  <tr><td>nno_Latn</td><td>2,358,129</td><td>34,771,540</td><td>1,175,108</td><td>21,352,540</td><td>563,791</td><td>10,548,302</td></tr>
  <tr><td>mar_Deva</td><td>2,067,311</td><td>34,952,324</td><td>952,116</td><td>19,606,305</td><td>656,962</td><td>15,113,175</td></tr>
  <tr><td>guj_Gujr</td><td>2,134,977</td><td>38,906,708</td><td>1,165,483</td><td>23,631,881</td><td>716,777</td><td>16,564,683</td></tr>
  <tr><td>kan_Knda</td><td>2,354,299</td><td>37,451,816</td><td>1,238,033</td><td>21,344,021</td><td>720,157</td><td>13,965,655</td></tr>
  <tr><td>tel_Telu</td><td>2,924,532</td><td>46,227,504</td><td>1,513,237</td><td>25,963,464</td><td>902,962</td><td>17,487,796</td></tr>
  <tr><td>tam_Taml</td><td>3,859,610</td><td>55,779,718</td><td>1,759,372</td><td>28,369,233</td><td>1,111,471</td><td>20,718,487</td></tr>
  <tr><td>uzn_Latn</td><td>2,791,412</td><td>37,715,209</td><td>1,571,871</td><td>25,823,124</td><td>1,159,869</td><td>19,667,785</td></tr>
  <tr><td>urd_Arab</td><td>3,866,815</td><td>101,346,427</td><td>2,200,602</td><td>65,830,839</td><td>1,399,893</td><td>47,591,409</td></tr>
  <tr><td>eus_Latn</td><td>5,907,808</td><td>79,485,282</td><td>2,526,198</td><td>38,107,950</td><td>1,491,873</td><td>24,303,464</td></tr>
  <tr><td>epo_Latn</td><td>5,664,237</td><td>91,081,114</td><td>3,190,135</td><td>60,141,119</td><td>1,521,821</td><td>30,986,721</td></tr>
  <tr><td>mlt_Latn</td><td>7,434,717</td><td>114,046,030</td><td>2,651,758</td><td>50,044,197</td><td>1,529,471</td><td>32,243,598</td></tr>
  <tr><td>kaz_Cyril</td><td>3,827,170</td><td>55,027,673</td><td>2,628,328</td><td>39,138,283</td><td>1,943,935</td><td>30,216,073</td></tr>
  <tr><td>swh_Latn</td><td>10,125,330</td><td>145,685,653</td><td>3,680,151</td><td>68,766,541</td><td>1,985,899</td><td>39,952,916</td></tr>
  <tr><td>ben_Beng</td><td>6,376,109</td><td>106,303,435</td><td>3,920,955</td><td>70,350,081</td><td>2,328,136</td><td>49,851,040</td></tr>
  <tr><td>isl_Latn</td><td>11,929,153</td><td>146,981,787</td><td>6,624,589</td><td>91,089,371</td><td>2,694,541</td><td>47,440,271</td></tr>
  <tr><td>gle_Latn</td><td>7,685,880</td><td>133,441,028</td><td>4,421,130</td><td>89,932,030</td><td>2,697,582</td><td>59,065,530</td></tr>
  <tr><td>glg_Latn</td><td>8,680,808</td><td>145,602,784</td><td>5,166,276</td><td>99,562,132</td><td>2,783,727</td><td>58,437,672</td></tr>
  <tr><td>bel_Cyril</td><td>11,493,046</td><td>154,657,914</td><td>6,092,481</td><td>90,760,902</td><td>3,140,958</td><td>50,113,002</td></tr>
  <tr><td>azj_Latn</td><td>8,506,772</td><td>118,079,751</td><td>4,765,278</td><td>72,026,597</td><td>3,188,231</td><td>51,425,346</td></tr>
  <tr><td>pes_Arab</td><td>9,434,306</td><td>192,718,387</td><td>5,391,049</td><td>130,840,005</td><td>3,448,296</td><td>95,822,037</td></tr>
  <tr><td>cym_Latn</td><td>9,390,284</td><td>156,087,956</td><td>6,348,606</td><td>125,442,540</td><td>3,867,402</td><td>82,244,645</td></tr>
  <tr><td>afr_Latn</td><td>15,901,372</td><td>246,703,185</td><td>7,452,216</td><td>139,249,006</td><td>3,987,340</td><td>80,857,410</td></tr>
  <tr><td>mkd_Cyril</td><td>10,815,504</td><td>185,651,668</td><td>7,175,217</td><td>131,839,062</td><td>3,991,617</td><td>78,629,993</td></tr>
  <tr><td>tha_Thai</td><td>13,818,095</td><td>102,830,296</td><td>7,551,187</td><td>52,171,172</td><td>4,088,354</td><td>34,155,503</td></tr>
  <tr><td>als_Latn</td><td>11,171,352</td><td>208,460,688</td><td>6,943,910</td><td>145,918,660</td><td>4,166,536</td><td>94,263,904</td></tr>
  <tr><td>bos_Latn</td><td>12,480,871</td><td>193,998,734</td><td>7,527,232</td><td>139,457,685</td><td>4,559,328</td><td>92,723,229</td></tr>
  <tr><td>srp_Cyril</td><td>17,605,882</td><td>244,921,478</td><td>9,618,806</td><td>153,171,621</td><td>5,291,686</td><td>90,518,351</td></tr>
  <tr><td>zsm_Latn</td><td>47,173,963</td><td>558,911,698</td><td>22,298,471</td><td>301,446,773</td><td>8,432,285</td><td>147,009,838</td></tr>
  <tr><td>heb_Hebr</td><td>34,004,891</td><td>431,453,938</td><td>21,600,460</td><td>279,563,405</td><td>8,686,089</td><td>162,768,846</td></tr>
  <tr><td>est_Latn</td><td>29,934,421</td><td>362,843,780</td><td>16,629,846</td><td>223,025,816</td><td>8,797,574</td><td>133,824,400</td></tr>
  <tr><td>hin_Deva</td><td>26,345,062</td><td>500,967,390</td><td>16,337,234</td><td>372,581,817</td><td>9,926,620</td><td>263,709,932</td></tr>
  <tr><td>slv_Latn</td><td>30,956,083</td><td>449,919,060</td><td>18,290,300</td><td>301,699,622</td><td>10,336,528</td><td>188,709,019</td></tr>
  <tr><td>lvs_Latn</td><td>39,599,210</td><td>476,030,125</td><td>24,504,355</td><td>316,955,851</td><td>11,294,618</td><td>183,588,490</td></tr>
  <tr><td>lit_Latn</td><td>47,035,968</td><td>553,786,167</td><td>27,879,310</td><td>351,354,617</td><td>12,881,354</td><td>205,285,778</td></tr>
  <tr><td>cat_Latn</td><td>40,922,098</td><td>671,563,410</td><td>26,451,844</td><td>521,397,424</td><td>13,080,859</td><td>292,854,267</td></tr>
  <tr><td>hrv_Latn</td><td>45,617,022</td><td>627,929,707</td><td>27,783,979</td><td>420,953,899</td><td>14,263,908</td><td>250,103,294</td></tr>
  <tr><td>ara_Arab</td><td>41,671,896</td><td>759,192,353</td><td>31,389,602</td><td>618,611,620</td><td>17,505,366</td><td>424,460,900</td></tr>
  <tr><td>kor_Hang</td><td>76,980,595</td><td>865,790,290</td><td>46,010,282</td><td>530,544,997</td><td>18,393,859</td><td>294,720,264</td></tr>
  <tr><td>jpn_Jpan</td><td>105,291,263</td><td>575,207,953</td><td>48,568,992</td><td>178,462,644</td><td>18,894,019</td><td>80,778,230</td></tr>
  <tr><td>vie_Latn</td><td>47,831,389</td><td>1,077,677,445</td><td>35,072,681</td><td>831,280,934</td><td>19,231,770</td><td>502,683,444</td></tr>
  <tr><td>slk_Latn</td><td>62,840,882</td><td>798,323,342</td><td>40,704,524</td><td>566,060,898</td><td>20,056,339</td><td>332,818,300</td></tr>
  <tr><td>tur_Latn</td><td>84,823,944</td><td>1,054,338,198</td><td>46,493,019</td><td>656,237,839</td><td>21,616,652</td><td>402,325,110</td></tr>
  <tr><td>bul_Cyril</td><td>63,059,982</td><td>939,666,726</td><td>40,936,972</td><td>670,141,259</td><td>22,725,326</td><td>420,768,808</td></tr>
  <tr><td>nob_Latn</td><td>71,277,875</td><td>969,526,777</td><td>45,204,041</td><td>700,446,147</td><td>22,912,722</td><td>395,447,564</td></tr>
  <tr><td>ukr_Cyril</td><td>68,111,464</td><td>862,726,167</td><td>46,141,511</td><td>637,243,802</td><td>25,125,019</td><td>400,949,773</td></tr>
  <tr><td>fin_Latn</td><td>98,138,078</td><td>1,028,494,366</td><td>59,836,942</td><td>667,840,953</td><td>29,067,875</td><td>383,463,787</td></tr>
</table>

Table 5: Statistics for the parallel portion of HPLT v2 before filtering (Raw), after Bicleaner AI (Filtered) and after deduplication (TMX). Languages are in increasing order of deduplicated sentence pairs.
<table>
  <tr>
    <th>Language</th>
    <th>Segments</th>
    <th>Tokens</th>
    <th>Characters</th>
    <th>Documents</th>
  </tr>
  <tr><td>ace_Arab</td><td>1.170e+02</td><td>8.363e+03</td><td>4.973e+04</td><td>1.600e+01</td></tr>
  <tr><td>ace_Latn</td><td>2.062e+05</td><td>8.196e+06</td><td>5.083e+07</td><td>1.293e+04</td></tr>
  <tr><td>afr_Latn</td><td>3.774e+07</td><td>1.000e+09</td><td>5.947e+09</td><td>1.457e+06</td></tr>
  <tr><td>als_Latn</td><td>9.510e+07</td><td>2.713e+09</td><td>1.610e+10</td><td>5.385e+06</td></tr>
  <tr><td>amh_Ethi</td><td>7.006e+06</td><td>1.959e+08</td><td>1.031e+09</td><td>2.955e+05</td></tr>
  <tr><td>ara_Arab</td><td>2.200e+09</td><td>4.814e+10</td><td>2.795e+11</td><td>8.267e+07</td></tr>
  <tr><td>asm_Beng</td><td>2.677e+06</td><td>7.344e+07</td><td>4.757e+08</td><td>1.757e+05</td></tr>
  <tr><td>ast_Latn</td><td>7.426e+06</td><td>1.950e+08</td><td>1.244e+09</td><td>2.732e+05</td></tr>
  <tr><td>awa_Deva</td><td>1.315e+05</td><td>6.049e+06</td><td>2.877e+07</td><td>7.281e+03</td></tr>
  <tr><td>ayr_Latn</td><td>1.885e+05</td><td>3.068e+06</td><td>2.508e+07</td><td>9.223e+03</td></tr>
  <tr><td>azb_Arab</td><td>2.389e+06</td><td>3.958e+07</td><td>2.602e+08</td><td>6.611e+04</td></tr>
  <tr><td>azj_Latn</td><td>1.266e+08</td><td>2.569e+09</td><td>1.962e+10</td><td>6.485e+06</td></tr>
  <tr><td>bak_Cyril</td><td>3.139e+06</td><td>7.533e+07</td><td>5.585e+08</td><td>1.708e+05</td></tr>
  <tr><td>bam_Latn</td><td>9.172e+04</td><td>3.982e+06</td><td>2.074e+07</td><td>5.721e+03</td></tr>
  <tr><td>ban_Latn</td><td>6.011e+05</td><td>1.134e+07</td><td>7.724e+07</td><td>1.070e+04</td></tr>
  <tr><td>bel_Cyril</td><td>4.884e+07</td><td>1.212e+09</td><td>8.540e+09</td><td>2.320e+06</td></tr>
  <tr><td>bem_Latn</td><td>1.335e+05</td><td>4.532e+06</td><td>3.232e+07</td><td>6.136e+03</td></tr>
  <tr><td>ben_Beng</td><td>1.760e+08</td><td>4.639e+09</td><td>3.016e+10</td><td>1.104e+07</td></tr>
  <tr><td>bho_Deva</td><td>4.583e+05</td><td>1.347e+07</td><td>6.865e+07</td><td>2.864e+04</td></tr>
  <tr><td>bjn_Arab</td><td>1.953e+04</td><td>5.483e+05</td><td>3.317e+06</td><td>1.112e+03</td></tr>
  <tr><td>bjn_Latn</td><td>3.663e+05</td><td>8.048e+06</td><td>5.597e+07</td><td>1.876e+04</td></tr>
  <tr><td>bod_Tibt</td><td>4.650e+05</td><td>5.781e+06</td><td>2.685e+08</td><td>2.744e+04</td></tr>
  <tr><td>bos_Latn</td><td>2.682e+08</td><td>7.255e+09</td><td>4.607e+10</td><td>1.461e+07</td></tr>
  <tr><td>bug_Latn</td><td>3.855e+04</td><td>2.705e+06</td><td>1.931e+07</td><td>2.023e+03</td></tr>
  <tr><td>bui_Cyril</td><td>6.814e+08</td><td>1.530e+10</td><td>9.693e+10</td><td>2.809e+07</td></tr>
  <tr><td>cat_Latn</td><td>3.833e+08</td><td>1.002e+10</td><td>6.019e+10</td><td>1.855e+07</td></tr>
  <tr><td>ceb_Latn</td><td>2.865e+06</td><td>8.589e+07</td><td>5.157e+08</td><td>1.388e+05</td></tr>
  <tr><td>ces_Latn</td><td>1.927e+09</td><td>4.208e+10</td><td>2.739e+11</td><td>7.529e+07</td></tr>
  <tr><td>cjk_Latn</td><td>3.670e+04</td><td>9.647e+05</td><td>7.432e+06</td><td>1.196e+03</td></tr>
  <tr><td>ckb_Arab</td><td>5.226e+06</td><td>1.426e+08</td><td>9.128e+08</td><td>2.737e+05</td></tr>
  <tr><td>crh_Latn</td><td>1.381e+06</td><td>3.676e+07</td><td>2.811e+08</td><td>1.227e+05</td></tr>
  <tr><td>cym_Latn</td><td>1.557e+07</td><td>4.090e+08</td><td>2.402e+09</td><td>7.581e+05</td></tr>
  <tr><td>dan_Latn</td><td>8.730e+08</td><td>2.120e+10</td><td>1.334e+11</td><td>3.384e+07</td></tr>
  <tr><td>deu_Latn</td><td>1.113e+10</td><td>2.515e+11</td><td>1.782e+12</td><td>4.821e+08</td></tr>
  <tr><td>dik_Latn</td><td>3.465e+04</td><td>2.295e+06</td><td>1.154e+07</td><td>2.325e+03</td></tr>
  <tr><td>dyu_Latn</td><td>2.456e+04</td><td>1.194e+06</td><td>5.552e+06</td><td>1.390e+03</td></tr>
  <tr><td>dzo_Tibt</td><td>3.997e+04</td><td>4.222e+05</td><td>3.735e+06</td><td>1.626e+03</td></tr>
  <tr><td>ell_Grek</td><td>1.849e+09</td><td>4.270e+10</td><td>2.835e+11</td><td>7.033e+07</td></tr>
  <tr><td>eng_Latn</td><td>1.165e+11</td><td>2.862e+12</td><td>1.708e+13</td><td>4.389e+09</td></tr>
  <tr><td>epo_Latn</td><td>2.035e+07</td><td>4.716e+08</td><td>2.976e+09</td><td>8.189e+05</td></tr>
  <tr><td>est_Latn</td><td>2.644e+08</td><td>4.742e+09</td><td>3.602e+10</td><td>8.449e+06</td></tr>
  <tr><td>eus_Latn</td><td>3.762e+07</td><td>7.767e+08</td><td>6.052e+09</td><td>1.974e+06</td></tr>
  <tr><td>ewe_Latn</td><td>1.434e+05</td><td>4.308e+06</td><td>2.132e+07</td><td>3.772e+03</td></tr>
  <tr><td>fao_Latn</td><td>4.526e+06</td><td>9.345e+07</td><td>5.818e+08</td><td>2.399e+05</td></tr>
  <tr><td>fi_j_Latn</td><td>1.789e+05</td><td>7.263e+06</td><td>3.769e+07</td><td>8.914e+03</td></tr>
  <tr><td>fin_Latn</td><td>9.766e+08</td><td>1.845e+10</td><td>1.557e+11</td><td>3.482e+07</td></tr>
  <tr><td>fon_Latn</td><td>1.476e+04</td><td>1.233e+06</td><td>5.335e+06</td><td>1.226e+03</td></tr>
  <tr><td>fra_Latn</td><td>1.056e+10</td><td>2.370e+11</td><td>1.457e+12</td><td>4.018e+08</td></tr>
  <tr><td>fur_Latn</td><td>7.300e+05</td><td>2.082e+07</td><td>1.147e+08</td><td>3.667e+04</td></tr>
  <tr><td>fu_v_Latn</td><td>1.340e+05</td><td>5.143e+06</td><td>2.990e+07</td><td>7.760e+03</td></tr>
  <tr><td>gaz_Latn</td><td>9.736e+05</td><td>2.888e+07</td><td>2.192e+08</td><td>4.914e+04</td></tr>
  <tr><td>gla_Latn</td><td>3.307e+06</td><td>8.066e+07</td><td>4.836e+08</td><td>1.374e+05</td></tr>
  <tr><td>gle_Latn</td><td>1.099e+07</td><td>2.957e+08</td><td>1.749e+09</td><td>4.908e+05</td></tr>
  <tr><td>glg_Latn</td><td>6.118e+07</td><td>1.639e+09</td><td>1.011e+10</td><td>3.020e+06</td></tr>
  <tr><td>grn_Latn</td><td>1.713e+06</td><td>3.072e+07</td><td>2.186e+08</td><td>7.342e+04</td></tr>
  <tr><td>gu_Gujr</td><td>2.064e+07</td><td>5.768e+08</td><td>3.386e+09</td><td>1.134e+06</td></tr>
</table>

Table 6: Counts of segments, tokens, characters and documents for each language in the monolingual HPLT v2 datasets. Tokens are words as defined by Unix wc.
<table>
  <tr>
    <th>Language</th>
    <th>Segments</th>
    <th>Tokens</th>
    <th>Characters</th>
    <th>Documents</th>
  </tr>
  <tr><td>hat_Latn</td><td>4.635e+06</td><td>1.223e+08</td><td>6.389e+08</td><td>2.127e+05</td></tr>
  <tr><td>hau_Latn</td><td>5.688e+06</td><td>1.526e+08</td><td>8.535e+08</td><td>3.159e+05</td></tr>
  <tr><td>heb_Hebr</td><td>4.666e+08</td><td>9.966e+09</td><td>5.682e+10</td><td>1.712e+07</td></tr>
  <tr><td>hin_Deva</td><td>2.674e+08</td><td>8.637e+09</td><td>4.396e+10</td><td>1.365e+07</td></tr>
  <tr><td>hne_Deva</td><td>5.500e+04</td><td>2.199e+06</td><td>1.059e+07</td><td>2.806e+03</td></tr>
  <tr><td>hrv_Latn</td><td>2.971e+08</td><td>7.307e+09</td><td>4.800e+10</td><td>1.230e+07</td></tr>
  <tr><td>hun_Latn</td><td>1.419e+09</td><td>3.052e+10</td><td>2.252e+11</td><td>5.187e+07</td></tr>
  <tr><td>hye_Arnn</td><td>6.524e+07</td><td>1.405e+09</td><td>1.072e+10</td><td>3.599e+06</td></tr>
  <tr><td>ibo_Latn</td><td>1.411e+06</td><td>3.829e+07</td><td>2.052e+08</td><td>5.629e+04</td></tr>
  <tr><td>ilo_Latn</td><td>1.120e+06</td><td>2.478e+07</td><td>1.568e+08</td><td>4.875e+04</td></tr>
  <tr><td>ind_Latn</td><td>2.389e+09</td><td>5.462e+10</td><td>3.842e+11</td><td>9.814e+07</td></tr>
  <tr><td>isl_Latn</td><td>6.964e+07</td><td>1.536e+09</td><td>9.593e+09</td><td>2.841e+06</td></tr>
  <tr><td>ita_Latn</td><td>5.127e+09</td><td>1.274e+11</td><td>8.206e+11</td><td>2.218e+08</td></tr>
  <tr><td>jav_Latn</td><td>6.431e+06</td><td>1.378e+08</td><td>9.375e+08</td><td>1.960e+05</td></tr>
  <tr><td>jpn_Jpan</td><td>2.327e+10</td><td>4.236e+10</td><td>9.011e+11</td><td>4.177e+08</td></tr>
  <tr><td>kab_Latn</td><td>3.452e+05</td><td>9.222e+06</td><td>5.419e+07</td><td>1.510e+04</td></tr>
  <tr><td>kac_Latn</td><td>1.594e+05</td><td>5.955e+06</td><td>2.840e+07</td><td>7.587e+03</td></tr>
  <tr><td>kam_Latn</td><td>1.426e+04</td><td>6.740e+05</td><td>4.645e+06</td><td>1.183e+03</td></tr>
  <tr><td>kan_Knda</td><td>2.493e+07</td><td>5.329e+08</td><td>4.298e+09</td><td>1.336e+06</td></tr>
  <tr><td>kas_Arab</td><td>2.711e+04</td><td>6.780e+05</td><td>3.468e+06</td><td>9.490e+02</td></tr>
  <tr><td>kas_Deva</td><td>1.357e+03</td><td>3.194e+04</td><td>1.854e+05</td><td>1.060e+02</td></tr>
  <tr><td>kat_Geor</td><td>6.372e+07</td><td>1.244e+09</td><td>1.016e+10</td><td>3.335e+06</td></tr>
  <tr><td>kaz_Cyril</td><td>8.101e+07</td><td>1.409e+09</td><td>1.113e+10</td><td>2.637e+06</td></tr>
  <tr><td>kbp_Latn</td><td>4.679e+04</td><td>4.258e+06</td><td>2.090e+07</td><td>7.075e+03</td></tr>
  <tr><td>kea_Latn</td><td>4.391e+04</td><td>1.143e+06</td><td>6.144e+06</td><td>1.962e+03</td></tr>
  <tr><td>khk_Cyril</td><td>5.347e+07</td><td>1.342e+09</td><td>9.327e+09</td><td>2.121e+06</td></tr>
  <tr><td>khm_Khmr</td><td>9.864e+06</td><td>1.138e+08</td><td>2.122e+09</td><td>7.010e+05</td></tr>
  <tr><td>kik_Latn</td><td>5.193e+04</td><td>1.428e+06</td><td>9.292e+06</td><td>3.995e+03</td></tr>
  <tr><td>kin_Latn</td><td>1.917e+06</td><td>5.074e+07</td><td>3.671e+08</td><td>9.270e+04</td></tr>
  <tr><td>kir_Cyril</td><td>1.004e+07</td><td>2.467e+08</td><td>1.925e+09</td><td>6.761e+05</td></tr>
  <tr><td>kmb_Latn</td><td>1.180e+04</td><td>3.831e+05</td><td>2.068e+06</td><td>5.310e+02</td></tr>
  <tr><td>kmr_Latn</td><td>7.147e+06</td><td>1.959e+08</td><td>1.123e+09</td><td>3.643e+05</td></tr>
  <tr><td>knc_Arab</td><td>1.085e+04</td><td>2.620e+05</td><td>1.302e+06</td><td>2.450e+02</td></tr>
  <tr><td>knc_Latn</td><td>1.052e+04</td><td>2.409e+06</td><td>1.195e+07</td><td>2.472e+03</td></tr>
  <tr><td>kon_Latn</td><td>4.748e+04</td><td>1.944e+06</td><td>1.127e+07</td><td>2.542e+03</td></tr>
  <tr><td>kor_Hang</td><td>1.358e+09</td><td>1.970e+10</td><td>8.923e+10</td><td>3.887e+07</td></tr>
  <tr><td>lao_Laoo</td><td>3.200e+05</td><td>5.178e+06</td><td>8.468e+07</td><td>2.950e+04</td></tr>
  <tr><td>lij_Latn</td><td>1.577e+05</td><td>5.593e+06</td><td>3.146e+07</td><td>8.371e+03</td></tr>
  <tr><td>lim_Latn</td><td>7.140e+06</td><td>1.806e+08</td><td>1.125e+09</td><td>3.679e+05</td></tr>
  <tr><td>lin_Latn</td><td>2.003e+05</td><td>5.555e+06</td><td>3.292e+07</td><td>7.588e+03</td></tr>
  <tr><td>lit_Latn</td><td>3.222e+08</td><td>6.676e+09</td><td>5.039e+10</td><td>1.334e+07</td></tr>
  <tr><td>lmo_Latn</td><td>2.125e+06</td><td>5.964e+07</td><td>3.454e+08</td><td>1.462e+05</td></tr>
  <tr><td>ltg_Latn</td><td>1.514e+05</td><td>3.790e+06</td><td>2.688e+07</td><td>9.209e+03</td></tr>
  <tr><td>ltz_Latn</td><td>5.059e+06</td><td>1.072e+08</td><td>7.104e+08</td><td>2.469e+05</td></tr>
  <tr><td>lua_Latn</td><td>3.869e+04</td><td>1.368e+06</td><td>9.005e+06</td><td>1.083e+03</td></tr>
  <tr><td>lug_Latn</td><td>4.075e+05</td><td>9.176e+06</td><td>6.796e+07</td><td>2.128e+04</td></tr>
  <tr><td>luo_Latn</td><td>8.412e+04</td><td>3.727e+06</td><td>2.033e+07</td><td>4.153e+03</td></tr>
  <tr><td>lus_Latn</td><td>3.433e+06</td><td>1.252e+08</td><td>6.520e+08</td><td>1.604e+05</td></tr>
  <tr><td>lvx_Latn</td><td>1.738e+08</td><td>3.461e+09</td><td>2.518e+10</td><td>6.772e+06</td></tr>
  <tr><td>mag_Deva</td><td>1.929e+04</td><td>8.906e+05</td><td>4.283e+06</td><td>3.280e+02</td></tr>
  <tr><td>mai_Deva</td><td>6.455e+05</td><td>1.779e+07</td><td>9.674e+07</td><td>2.498e+04</td></tr>
  <tr><td>mal_Mlym</td><td>4.800e+07</td><td>9.737e+08</td><td>9.489e+09</td><td>3.105e+06</td></tr>
  <tr><td>mar_Deva</td><td>3.632e+07</td><td>9.807e+08</td><td>6.622e+09</td><td>2.080e+06</td></tr>
  <tr><td>min_Latn</td><td>6.008e+05</td><td>1.098e+07</td><td>7.477e+07</td><td>2.504e+04</td></tr>
  <tr><td>mkd_Cyril</td><td>5.701e+07</td><td>1.485e+09</td><td>9.440e+09</td><td>3.566e+06</td></tr>
  <tr><td>mlt_Latn</td><td>8.675e+06</td><td>1.958e+08</td><td>1.442e+09</td><td>3.673e+05</td></tr>
</table>

Table 6: Counts of segments, tokens, characters and documents for each language in the monolingual HPLT v2 datasets. Tokens are words as defined by Unix wc.
<table>
  <tr>
    <th>Language</th>
    <th>Segments</th>
    <th>Tokens</th>
    <th>Characters</th>
    <th>Documents</th>
  </tr>
  <tr><td>mni_Beng</td><td>6.576e+04</td><td>1.627e+06</td><td>1.179e+07</td><td>2.934e+03</td></tr>
  <tr><td>mos_Latn</td><td>1.910e+04</td><td>8.075e+05</td><td>3.864e+06</td><td>9.310e+02</td></tr>
  <tr><td>mri_Latn</td><td>2.795e+06</td><td>8.676e+07</td><td>4.243e+08</td><td>1.083e+05</td></tr>
  <tr><td>mya_Mymr</td><td>3.050e+07</td><td>4.532e+08</td><td>5.819e+09</td><td>1.368e+06</td></tr>
  <tr><td>nld_Latn</td><td>3.075e+09</td><td>7.141e+10</td><td>4.511e+11</td><td>1.387e+08</td></tr>
  <tr><td>nno_Latn</td><td>3.460e+07</td><td>8.603e+08</td><td>5.404e+09</td><td>1.423e+06</td></tr>
  <tr><td>nob_Latn</td><td>6.760e+08</td><td>2.154e+10</td><td>1.332e+11</td><td>2.705e+07</td></tr>
  <tr><td>npi_Deva</td><td>3.714e+07</td><td>1.128e+09</td><td>7.256e+09</td><td>2.778e+06</td></tr>
  <tr><td>nso_Latin</td><td>1.433e+05</td><td>5.322e+06</td><td>2.749e+07</td><td>6.066e+03</td></tr>
  <tr><td>nus_Latin</td><td>8.514e+03</td><td>3.932e+05</td><td>1.882e+06</td><td>2.720e+02</td></tr>
  <tr><td>nya_Latin</td><td>1.344e+06</td><td>2.706e+07</td><td>2.029e+08</td><td>5.312e+04</td></tr>
  <tr><td>oci_Latin</td><td>4.195e+06</td><td>1.027e+08</td><td>6.354e+08</td><td>1.899e+05</td></tr>
  <tr><td>ory_Orya</td><td>3.596e+06</td><td>1.201e+08</td><td>7.815e+08</td><td>4.129e+05</td></tr>
  <tr><td>pag_Latin</td><td>8.583e+04</td><td>5.657e+06</td><td>3.352e+07</td><td>6.900e+03</td></tr>
  <tr><td>pan_Guru</td><td>1.174e+07</td><td>3.722e+08</td><td>1.902e+09</td><td>5.846e+05</td></tr>
  <tr><td>pap_Latin</td><td>1.387e+06</td><td>4.671e+07</td><td>2.541e+08</td><td>8.981e+04</td></tr>
  <tr><td>pbt_Arab</td><td>8.455e+06</td><td>2.794e+08</td><td>1.304e+09</td><td>4.665e+05</td></tr>
  <tr><td>pes_Arab</td><td>3.963e+09</td><td>8.855e+10</td><td>4.551e+11</td><td>9.050e+07</td></tr>
  <tr><td>plt_Latin</td><td>4.736e+06</td><td>1.171e+08</td><td>8.103e+08</td><td>2.078e+05</td></tr>
  <tr><td>pol_Latin</td><td>4.461e+09</td><td>8.953e+10</td><td>6.316e+11</td><td>1.754e+08</td></tr>
  <tr><td>por_Latin</td><td>6.125e+09</td><td>1.463e+11</td><td>8.965e+11</td><td>2.378e+08</td></tr>
  <tr><td>prs_Arab</td><td>6.900e+07</td><td>1.844e+09</td><td>9.567e+09</td><td>2.839e+06</td></tr>
  <tr><td>quy_Latin</td><td>4.943e+05</td><td>1.731e+07</td><td>1.434e+08</td><td>3.694e+04</td></tr>
  <tr><td>ron_Latin</td><td>1.697e+09</td><td>4.005e+10</td><td>2.507e+11</td><td>6.588e+07</td></tr>
  <tr><td>run_Latin</td><td>1.752e+06</td><td>4.444e+07</td><td>3.165e+08</td><td>1.373e+05</td></tr>
  <tr><td>rus_Cyril</td><td>2.629e+10</td><td>5.409e+11</td><td>3.908e+12</td><td>8.847e+08</td></tr>
  <tr><td>sag_Latin</td><td>5.190e+04</td><td>3.612e+06</td><td>1.674e+07</td><td>3.161e+03</td></tr>
  <tr><td>san_Deva</td><td>3.281e+06</td><td>4.380e+07</td><td>3.592e+08</td><td>5.491e+04</td></tr>
  <tr><td>sat_Olck</td><td>4.580e+04</td><td>1.085e+06</td><td>6.266e+06</td><td>2.566e+03</td></tr>
  <tr><td>scn_Latin</td><td>1.650e+06</td><td>4.239e+07</td><td>2.523e+08</td><td>8.197e+04</td></tr>
  <tr><td>shn_Mymr</td><td>9.214e+04</td><td>1.648e+06</td><td>2.121e+07</td><td>6.003e+03</td></tr>
  <tr><td>sin_Sinh</td><td>3.371e+07</td><td>7.956e+08</td><td>4.981e+09</td><td>1.153e+06</td></tr>
  <tr><td>slk_Latin</td><td>4.945e+08</td><td>1.063e+10</td><td>7.037e+10</td><td>2.183e+07</td></tr>
  <tr><td>slv_Latin</td><td>2.386e+08</td><td>5.435e+09</td><td>3.526e+10</td><td>1.028e+07</td></tr>
  <tr><td>smo_Latin</td><td>1.012e+06</td><td>3.709e+07</td><td>1.861e+08</td><td>4.586e+04</td></tr>
  <tr><td>sna_Latin</td><td>1.202e+06</td><td>2.392e+07</td><td>1.926e+08</td><td>6.108e+04</td></tr>
  <tr><td>snd_Arab</td><td>2.826e+06</td><td>8.953e+07</td><td>4.286e+08</td><td>1.003e+05</td></tr>
  <tr><td>som_Latin</td><td>1.638e+07</td><td>3.888e+08</td><td>2.565e+09</td><td>9.665e+05</td></tr>
  <tr><td>sot_Latin</td><td>1.085e+06</td><td>3.100e+07</td><td>1.715e+08</td><td>4.392e+04</td></tr>
  <tr><td>spa_Latin</td><td>1.212e+10</td><td>3.220e+11</td><td>1.954e+12</td><td>5.031e+08</td></tr>
  <tr><td>srd_Latin</td><td>9.171e+05</td><td>2.389e+07</td><td>1.487e+08</td><td>5.382e+04</td></tr>
  <tr><td>srp_Cyril</td><td>9.381e+07</td><td>2.519e+09</td><td>1.616e+10</td><td>4.123e+06</td></tr>
  <tr><td>ssw_Latin</td><td>6.213e+04</td><td>9.943e+05</td><td>8.821e+06</td><td>2.036e+03</td></tr>
  <tr><td>sun_Latin</td><td>3.238e+06</td><td>6.963e+07</td><td>4.753e+08</td><td>1.148e+05</td></tr>
  <tr><td>swe_Latin</td><td>1.755e+09</td><td>4.011e+10</td><td>2.511e+11</td><td>6.681e+07</td></tr>
  <tr><td>swl_Latin</td><td>3.431e+07</td><td>7.177e+08</td><td>4.664e+09</td><td>1.374e+06</td></tr>
  <tr><td>szl_Latin</td><td>6.366e+05</td><td>1.468e+07</td><td>1.038e+08</td><td>4.093e+04</td></tr>
  <tr><td>tam_Taml</td><td>1.686e+08</td><td>2.981e+09</td><td>2.624e+10</td><td>6.106e+06</td></tr>
  <tr><td>tag_Latin</td><td>1.388e+04</td><td>1.544e+06</td><td>8.845e+06</td><td>1.747e+03</td></tr>
  <tr><td>tat_Cyril</td><td>1.345e+07</td><td>2.967e+08</td><td>2.157e+09</td><td>6.307e+05</td></tr>
  <tr><td>tel_Telu</td><td>3.919e+07</td><td>8.354e+08</td><td>6.505e+09</td><td>2.058e+06</td></tr>
  <tr><td>tgk_Cyril</td><td>2.485e+07</td><td>6.248e+08</td><td>4.590e+09</td><td>1.261e+06</td></tr>
  <tr><td>tgl_Latin</td><td>5.288e+07</td><td>1.346e+09</td><td>8.131e+09</td><td>1.869e+06</td></tr>
  <tr><td>tha_Thai</td><td>3.391e+08</td><td>3.506e+09</td><td>5.998e+10</td><td>1.770e+07</td></tr>
  <tr><td>tir_Ethi</td><td>1.128e+06</td><td>3.672e+07</td><td>1.816e+08</td><td>6.469e+04</td></tr>
  <tr><td>tpi_Latin</td><td>2.824e+05</td><td>1.251e+07</td><td>6.453e+07</td><td>1.398e+04</td></tr>
</table>

Table 6: Counts of segments, tokens, characters and documents for each language in the monolingual HPLT v2 datasets. Tokens are words as defined by Unix wc.
<table>
  <tr>
    <th>Language</th>
    <th>Segments</th>
    <th>Tokens</th>
    <th>Characters</th>
    <th>Documents</th>
  </tr>
  <tr><td>tsn_Latn</td><td>1.322e+05</td><td>5.273e+06</td><td>2.767e+07</td><td>6.050e+03</td></tr>
  <tr><td>tso_Latn</td><td>2.212e+05</td><td>8.668e+06</td><td>4.929e+07</td><td>1.101e+04</td></tr>
  <tr><td>tuk_Latn</td><td>3.355e+06</td><td>7.068e+07</td><td>5.700e+08</td><td>1.710e+05</td></tr>
  <tr><td>tum_Latn</td><td>9.901e+04</td><td>2.876e+06</td><td>2.110e+07</td><td>4.384e+03</td></tr>
  <tr><td>tur_Latn</td><td>2.575e+09</td><td>5.167e+10</td><td>3.896e+11</td><td>1.166e+08</td></tr>
  <tr><td>twi_Latn</td><td>1.256e+05</td><td>4.696e+06</td><td>2.418e+07</td><td>5.860e+03</td></tr>
  <tr><td>uig_Arab</td><td>8.982e+06</td><td>2.239e+08</td><td>1.747e+09</td><td>4.424e+05</td></tr>
  <tr><td>ukr_Cyril</td><td>1.169e+09</td><td>2.523e+10</td><td>1.829e+11</td><td>4.740e+07</td></tr>
  <tr><td>umb_Latn</td><td>5.991e+04</td><td>2.431e+06</td><td>1.541e+07</td><td>2.471e+03</td></tr>
  <tr><td>urd_Arab</td><td>5.063e+07</td><td>2.126e+09</td><td>1.001e+10</td><td>3.194e+06</td></tr>
  <tr><td>uzn_Latn</td><td>1.480e+07</td><td>3.513e+08</td><td>2.846e+09</td><td>7.069e+05</td></tr>
  <tr><td>vec_Latn</td><td>1.579e+06</td><td>3.526e+07</td><td>2.180e+08</td><td>8.480e+04</td></tr>
  <tr><td>vie_Latn</td><td>3.020e+09</td><td>8.320e+10</td><td>3.795e+11</td><td>1.007e+08</td></tr>
  <tr><td>war_Latn</td><td>2.009e+05</td><td>5.889e+06</td><td>3.557e+07</td><td>1.387e+04</td></tr>
  <tr><td>wol_Latn</td><td>1.615e+05</td><td>5.463e+06</td><td>2.754e+07</td><td>5.679e+03</td></tr>
  <tr><td>xho_Latn</td><td>1.821e+06</td><td>3.034e+07</td><td>2.587e+08</td><td>6.309e+04</td></tr>
  <tr><td>ydd_Hebr</td><td>2.940e+06</td><td>7.753e+07</td><td>4.585e+08</td><td>1.283e+05</td></tr>
  <tr><td>yor_Latn</td><td>1.469e+06</td><td>4.281e+07</td><td>2.178e+08</td><td>6.613e+04</td></tr>
  <tr><td>yue_Hant</td><td>1.235e+06</td><td>3.268e+06</td><td>7.430e+07</td><td>6.129e+04</td></tr>
  <tr><td>zho_Hans</td><td>4.245e+10</td><td>7.403e+10</td><td>2.352e+12</td><td>1.247e+09</td></tr>
  <tr><td>zho_Hant</td><td>4.480e+09</td><td>9.510e+09</td><td>2.868e+11</td><td>1.571e+08</td></tr>
  <tr><td>zsm_Latn</td><td>5.798e+08</td><td>1.148e+10</td><td>7.843e+10</td><td>1.842e+07</td></tr>
  <tr><td>zul_Latn</td><td>2.710e+06</td><td>4.436e+07</td><td>3.808e+08</td><td>1.136e+05</td></tr>
</table>

Table 6: Counts of segments, tokens, characters and documents for each language in the monolingual HPLT v2 datasets. Tokens are words as defined by Unix wc.
C Sources of web crawls

<table>
  <tr>
    <th>Name</th>
    <th>Years</th>
    <th>Size (TB)</th>
  </tr>
  <tr>
    <th colspan="3">IA full crawls</th>
    <th>2012–2020</th>
    <th>3390</th>
  </tr>
  <tr><td>wide5</td><td>2012–2012</td><td>365</td></tr>
  <tr><td>wide6</td><td>2012–2013</td><td>204</td></tr>
  <tr><td>wide10</td><td>2014–2014</td><td>91</td></tr>
  <tr><td>wide11</td><td>2014–2014</td><td>420</td></tr>
  <tr><td>wide12</td><td>2015–2015</td><td>449</td></tr>
  <tr><td>wide15</td><td>2016–2017</td><td>358</td></tr>
  <tr><td>wide16</td><td>2017–2018</td><td>768</td></tr>
  <tr><td>wide17</td><td>2018–2020</td><td>641</td></tr>
  <tr><td>survey3</td><td>2015–2016</td><td>94</td></tr>
  <tr>
    <th colspan="3">CC full crawls</th>
    <th>2014–2022</th>
    <th>743</th>
  </tr>
  <tr><td>CC-MAIN-2014-35</td><td>2014</td><td>43</td></tr>
  <tr><td>CC-MAIN-2014-42</td><td>2014</td><td>54</td></tr>
  <tr><td>CC-MAIN-2015-11</td><td>2015</td><td>29</td></tr>
  <tr><td>CC-MAIN-2015-48</td><td>2015</td><td>30</td></tr>
  <tr><td>CC-MAIN-2017-04</td><td>2017</td><td>54</td></tr>
  <tr><td>CC-MAIN-2018-05</td><td>2018</td><td>75</td></tr>
  <tr><td>CC-MAIN-2018-22</td><td>2018</td><td>52</td></tr>
  <tr><td>CC-MAIN-2018-43</td><td>2018</td><td>59</td></tr>
  <tr><td>CC-MAIN-2021-43</td><td>2021</td><td>86</td></tr>
  <tr><td>CC-MAIN-2022-27</td><td>2022</td><td>85</td></tr>
  <tr><td>CC-MAIN-2022-40</td><td>2022</td><td>83</td></tr>
  <tr><td>CC-MAIN-2022-49</td><td>2022</td><td>93</td></tr>
  <tr>
    <th colspan="3">Partial crawls</th>
    <th>2013-2023</th>
    <th>317</th>
  </tr>
  <tr><td>1% of WARCs from 81 CC crawls</td><td>2013-2023</td><td>46</td></tr>
  <tr><td>7% of IA ArchiveBot</td><td>2013-2023</td><td>271</td></tr>
</table>

Table 7: List of web crawls used to construct HPLT v2. From IA, we use 8 Wide crawls, 1 Survey crawl containing main pages of websites and a random sample of 7% of items from IA ArchiveBot. From CC, we use 12 randomly-selected full crawls, plus a 1% sample of WARCs from each of the other 81 available crawls.

D Yields of different crawls

To figure out how different web crawls contribute to our datasets and which crawls are the most promising sources of monolingual corpora in general, we compared crawls from two points of view: the amount of texts extracted from each crawl and the quality of these texts. In this section, we study crawls from the first point of view, while in H the results of manual quality inspection are presented.

To make a comparison, we group all crawls into groups according to their age and source. The oldest IA wide crawls from 2012-2014 (from wide5 up to wide11) are assigned to the group ia_o, the newest wide16, wide17 crawls from 2017-2020 to the group ia_n, and the wide12, wide15 crawls in the middle to the group ia_m. CC crawls are split by age following the same time periods, but additionally a group cc_r is introduced for the recent CC crawls from 2021-2023 (we don’t have IA wide crawls from this time period). Finally, the IA survey3 and ArchiveBot crawls form their own groups ia_survey and ia_archivebot. In total, we have 9 groups of crawls.

For different processing stages, Figure 6 visualizes how much data comes from different groups of crawls. While originally less than 20% of our crawls are CC crawls, they contribute about half of the raw text before duplication and more than 60% of the text after deduplication and cleaning. Especially high-yielding are the new and recent CC crawls, they are only 6% and 8% of all crawls in size but contribute 28% and 30% of text (both when counting in characters and in documents) to the cleaned version. On the other hand, the newest IA wide crawls are 32% of all crawls in size but contribute only about 11% of text.

Figure 7 suggests another point of view showing yields for different crawls, or more specifically, how much text (measured in the number of characters) is extracted from 1 GB of compressed WARC files for each crawl. Evidently, CC crawls have the highest yields, especially the newer ones. Compared to the newer CC crawls, for the older CC crawls more data is filtered during deduplication and cleaning, giving finally lower yields despite a bit higher yields of raw texts. IA wide crawls have 4-8x smaller yields than CC crawls. The survey IA crawl has a comparable yield to the wide crawls in the final dataset. Since they are publicly available, it probably makes sense to employ more of these crawls in the future. Finally, the ArchiveBot IA crawl has remarkably low yields.

Despite having a lower contribution in general, for some languages, IA crawls supply the majority of texts. Figure 8 shows 15 languages with the highest proportion of texts from IA crawls. They include both high-resourced (Chinese, Western Persian) and low-resourced languages. Deduplication and cleaning significantly reduce the number of languages with high contribution of IA. For instance, before deduplication and cleaning there are 49 languages having more than 70% of texts (characters) coming from IA and only 6 such languages after.

E Frequent n-grams

We obtain frequent n-grams (up to order 5) in each dataset after tokenizing text and applying some restrictions:
• n-grams must start and end in the same seg-
Figure 6: Proportions of data from different groups of crawls at various processing stages. Crawls were quantified in TB of compressed WARC files, while raw texts and deduplicated cleaned texts in characters.

Figure 7: Yields (in characters per 1 GB of raw compressed crawls) of different crawls at different stages.

ment (i.e. no line breaks are allowed in the middle of a \( n \)-gram)
• \( n \)-grams containing any punctuation are discarded
• \( n \)-grams that start or end in stopwords are discarded
• \( n \)-grams are calculated case-insensitive
• all tokens in the \( n \)-gram must have at least one alphabetic character

Tables 8 and 9 show the 5 most frequent \( n \)-grams (orders 1 to 5) in HPLT v2. In the case of parallel datasets, \( n \)-grams are selected from the target (non-English) side of the segments. Translation to English is obtained with Google Translate.\(^{30}\)

We find that most datasets (both monolingual and parallel) contain frequent \( n \)-grams that seem to be boilerplate, such as "edit source", "read more", "click button" or "view map". This kind of content usually comes from Wikipedia and Blogspot. In the monolingual datasets, there is a large amount of text that seems to come from headers or footers in news webpages, e.g. "latest news". Biblical \( n \)-grams (such as "god" or "jehovah") are also very frequent in some datasets, notably African languages, matching our observations about frequent domains (Appendix F). Some frequent \( n \)-grams suggest poor-quality content in some datasets, since they seem to be related to downloads webpages, online game platforms or betting sites.

For the parallel datasets, we observe that on the English side the frequent \( n \)-grams are very similar across all languages. For the languages with the most data, hotels and legal notices are the most

Figure 8: Proportions of texts from different groups of crawls for the 15 languages with the largest contribution of IA crawls.

\(^{30}\)https://translate.google.com/
common kind of \( n \)-grams. The smaller parallel datasets tend to exhibit more variety of \( n \)-grams and include \( n \)-grams alluding to political leaders or city names, which suggest more locally-generated content (probably from news sites). Finally, frequent \( n \)-grams in parallel datasets from Eastern European countries usually contain mentions to European institutions (such as the European Parliament or the European Commission). This matches our observations on TLDs in Appendix G.

F Frequent domains

Inspecting the most common domain names in the datasets is one way to understand the type of content we can find in it. Table 10 gives the datasets with the highest proportion of frequent domain classes, and Table 11 gives the datasets with the highest proportion of frequent domain classes for the parallel data. We make the following general observations:
• Mid-to-large-sized datasets show a wider variety of domains with no clear majority source. However, in monolingual datasets, blogging platforms usually get a significant portion of the total (Table 10).
• Wikipedia tends to be among the most frequent domains for both monolingual and parallel datasets. It is usually the most frequent domain for smaller language datasets (Table 10).
• Hotel and travel webpages are much more frequent in the larger parallel datasets and very infrequent in the monolingual data (Table 11).
• News and media outlets are also a frequent content source in monolingual datasets, with some news websites getting a significant percentage in different datasets: for example, regional websites from the Free Radio Europe\(^{31}\) or Voice of America\(^{32}\) networks (Table 10).
• Religious and biblical content is also very frequent in the smaller monolingual and parallel datasets. This is specially notable in the case of African languages, which often get more than three quarters of their content from such sites (Table 10).
• Software and online gaming websites are usually among the top-10 most frequent domains in almost all parallel datasets.
• Chinese shopping websites are common in the larger parallel datasets of non-European languages (Vietnamese, Japanese, Korean, Arabic and Turkish).
• No pornographic webpages appear in the top domains, implying our filter for such content worked as expected.

G Geographic TLDs

Tables 12 and 13 list the most frequent examples of geographic TLDs in the monolingual and parallel HPLT v2 corpora respectively. We make the following observations in addition to those made in the main text:
• In general, the most frequent TLDs in many of the datasets are generic (such as .com, .org or .info).
• Some TLDs are frequent because they "sound good" rather than indicating the kind of content or language: .icu (because it reads like "I see you"), .is (official TLD for Iceland, but used as a verb and very noticeably in bible.is, a religious webpage whose domain is usually in the top-10 most frequent TLDs), .tv (for the country of Tuvalu, but widely used for TV-related web pages), .co (for Colombia, but mostly used for companies), .no (for Norway, but used as a negative particle), .nu (for the island of Niue, but used because it sounds like "new"), etc.
• There are common TLDs for super-national territories: .eu (European Union), .africa, .asia, etc.
• In our monolingual datasets, there is frequently one geographic TLD among the 10 most frequent ones that clearly surpasses the others. The "winning" TLD is usually from the country where the dataset language is spoken most, indicating that the text content is probably in the correct language. The percentage of this "winning country" varies depending on the amount of general purpose TLD in the dataset, but it is in general higher for European countries. This "winning" geographic TLD, in the case of parallel datasets, is less frequent and, when present, its portion of the total is noticeably lower than for the monolingual datasets.
• Many African languages do not have a significant portion of geographic TLDs (beyond the aforementioned .is, .no, etc).
• For some languages, there are a few coun-

\(^{31}\)https://www.rferl.org/navigation/allsites
\(^{32}\)https://www.voanews.com/navigation/allsites
<table>
  <tr>
    <th>n-gram (original)</th>
    <th>n-gram (translated)</th>
    <th>Dataset</th>
    <th>Occurrences</th>
  </tr>
  <tr>
    <th colspan="4">Blogging & social networks boilerplate</th>
  </tr>
  <tr>
    <td>read more</td>
    <td>-</td>
    <td>Tosk Albanian</td>
    <td>318,636</td>
  </tr>
  <tr>
    <td>read more</td>
    <td>-</td>
    <td>Malayalam</td>
    <td>244,935</td>
  </tr>
  <tr>
    <td>read more</td>
    <td>-</td>
    <td>Telugu</td>
    <td>225,178</td>
  </tr>
  <tr>
    <td>posted by</td>
    <td>-</td>
    <td>Malayalam</td>
    <td>177,507</td>
  </tr>
  <tr>
    <td>read more</td>
    <td>-</td>
    <td>Nepali</td>
    <td>176,500</td>
  </tr>
  <tr>
    <td>வேலைவாய்க்குறியீடு மூன்றுக்கும் சிலரை பேசுதல்</td>
    <td>readers’ comments</td>
    <td>Malayalam</td>
    <td>414,905</td>
  </tr>
  <tr>
    <td>you must be logged in</td>
    <td>-</td>
    <td>Tagalog</td>
    <td>138,209</td>
  </tr>
  <tr>
    <td>пістапад каstryчнік верасень</td>
    <td>november october september</td>
    <td>Belarusian</td>
    <td>126,015</td>
  </tr>
  <tr>
    <th colspan="4">Wikipedia boilerplate</th>
  </tr>
  <tr>
    <td>editar</td>
    <td>edit</td>
    <td>Galician</td>
    <td>3,177,972</td>
  </tr>
  <tr>
    <td>redakti</td>
    <td>edit</td>
    <td>Esperanto</td>
    <td>3,013,130</td>
  </tr>
  <tr>
    <td>editar a fonte</td>
    <td>edit source</td>
    <td>Galician</td>
    <td>1,491,606</td>
  </tr>
  <tr>
    <td>redakti fonton</td>
    <td>edit source</td>
    <td>Esperanto</td>
    <td>1,405,770</td>
  </tr>
  <tr>
    <td>aldatu iturburu kodea</td>
    <td>edit source code</td>
    <td>Basque</td>
    <td>921,847</td>
  </tr>
  <tr>
    <td>endre wikiteksten</td>
    <td>edit wiki text</td>
    <td>Norwegian Nynorsk</td>
    <td>777,797</td>
  </tr>
  <tr>
    <td>правіць зиходнік</td>
    <td>edit source</td>
    <td>Belarusian</td>
    <td>710,669</td>
  </tr>
  <tr>
    <td>modificar la font</td>
    <td>edit source</td>
    <td>Occitan</td>
    <td>568,442</td>
  </tr>
  <tr>
    <td>editar la fonte</td>
    <td>edit source</td>
    <td>Asturian</td>
    <td>567,301</td>
  </tr>
  <tr>
    <td>հայտարարի լուծում</td>
    <td>edit source text</td>
    <td>Armenian</td>
    <td>517,252</td>
  </tr>
  <tr>
    <td>ბერძნული ენციკლოპედია</td>
    <td>edit source</td>
    <td>Georgian</td>
    <td>355,269</td>
  </tr>
  <tr>
    <td>уреди извор</td>
    <td>edit source</td>
    <td>Macedonian</td>
    <td>506,624</td>
  </tr>
  <tr>
    <td>wysig bron</td>
    <td>edit source</td>
    <td>Afrikaans</td>
    <td>345,356</td>
  </tr>
  <tr>
    <td>quelltext ändern</td>
    <td>edit source</td>
    <td>Luxembourgish</td>
    <td>278,814</td>
  </tr>
  <tr>
    <th colspan="4">Religious & biblical content</th>
  </tr>
  <tr>
    <td>uyehova</td>
    <td>jehovah</td>
    <td>Xhosa</td>
    <td>47,227</td>
  </tr>
  <tr>
    <td>yehova</td>
    <td>jehovah</td>
    <td>Tumbuka</td>
    <td>27,729</td>
  </tr>
  <tr>
    <td>chiuta</td>
    <td>god</td>
    <td>Tumbuka</td>
    <td>23,684</td>
  </tr>
  <tr>
    <td>yehowa</td>
    <td>jehovah</td>
    <td>Ewe</td>
    <td>16,739</td>
  </tr>
  <tr>
    <td>biblia</td>
    <td>bible</td>
    <td>Ewe</td>
    <td>10,777</td>
  </tr>
  <tr>
    <td>yehova</td>
    <td>jehovah</td>
    <td>Chokwe</td>
    <td>9,174</td>
  </tr>
  <tr>
    <td>yesu</td>
    <td>jesus</td>
    <td>Chokwe</td>
    <td>6,984</td>
  </tr>
  <tr>
    <td>nin dijos</td>
    <td>by god</td>
    <td>Pangasinan</td>
    <td>6,636</td>
  </tr>
  <tr>
    <td>yeova</td>
    <td>jehovah</td>
    <td>Kamba</td>
    <td>3,785</td>
  </tr>
  <tr>
    <th colspan="4">Low-quality content indicators</th>
  </tr>
  <tr>
    <td>ਪੰਜਾਬੀ ਵੀਡੀਓ</td>
    <td>porn video</td>
    <td>Punjabi</td>
    <td>143,816</td>
  </tr>
  <tr>
    <td>piala donya</td>
    <td>world cup</td>
    <td>Javanese</td>
    <td>121,060</td>
  </tr>
  <tr>
    <td>piala dunya</td>
    <td>world cup</td>
    <td>Sundanese</td>
    <td>93,086</td>
  </tr>
  <tr>
    <td>compartir descargar reproducir</td>
    <td>share download play</td>
    <td>Asturian</td>
    <td>71,042</td>
  </tr>
  <tr>
    <td>ndajnë këtë lojë me miqtë</td>
    <td>share this game with friends</td>
    <td>Tosk Albanian</td>
    <td>67,888</td>
  </tr>
  <tr>
    <td>tohan maén bal</td>
    <td>soccer betting</td>
    <td>Javanese</td>
    <td>52,873</td>
  </tr>
  <tr>
    <td>luaj online flash lojë</td>
    <td>play online flash game</td>
    <td>Tosk Albanian</td>
    <td>47,112</td>
  </tr>
  <tr>
    <td>fifa world cup</td>
    <td>-</td>
    <td>Khmer</td>
    <td>45,367</td>
  </tr>
  <tr>
    <td>qatar world cup</td>
    <td>-</td>
    <td>Khmer</td>
    <td>35,895</td>
  </tr>
  <tr>
    <td>gêm hon gyda ’ch ffrindiau</td>
    <td>(share) this game with your friends</td>
    <td>Welsh</td>
    <td>12,984</td>
  </tr>
  <tr>
    <td>jwèt sou entènèt</td>
    <td>online game</td>
    <td>Haitian Creole</td>
    <td>12,686</td>
  </tr>
  <tr>
    <td>play ar líne a flash</td>
    <td>play flash online</td>
    <td>Irish</td>
    <td>10,954</td>
  </tr>
  <tr>
    <td>सेक्सी मूवी</td>
    <td>sexy movie</td>
    <td>Bhojpuri</td>
    <td>8,812</td>
  </tr>
  <tr>
    <td>tohan piala dunya</td>
    <td>world cup betting</td>
    <td>Sundanese</td>
    <td>6,691</td>
  </tr>
</table>

Table 8: Frequent n-grams in monolingual datasets.
<table>
  <tr>
    <th>n-gram (original)</th>
    <th>n-gram (translated)</th>
    <th>Dataset</th>
    <th>Occurrences</th>
  </tr>
  <tr>
    <th colspan="4">Legal boilerplate</th>
  </tr>
  <tr>
    <td>osobných údajov</td>
    <td>personal data</td>
    <td>Slovak</td>
    <td>776.876</td>
  </tr>
  <tr>
    <td>лични данни</td>
    <td>personal data</td>
    <td>Bulgarian</td>
    <td>487.125</td>
  </tr>
  <tr>
    <td>osobnih podataka</td>
    <td>personal data</td>
    <td>Croatian</td>
    <td>290.381</td>
  </tr>
  <tr>
    <td>osebnih podatkov</td>
    <td>personal data</td>
    <td>Slovene</td>
    <td>244.755</td>
  </tr>
  <tr>
    <td>asmens duomen</td>
    <td>personal data</td>
    <td>Lithuanian</td>
    <td>239.844</td>
  </tr>
  <tr>
    <td>personas datu</td>
    <td>personal data</td>
    <td>Latvian</td>
    <td>202.714</td>
  </tr>
  <tr>
    <td>персональних даних</td>
    <td>personal data</td>
    <td>Ukrainian</td>
    <td>153.272</td>
  </tr>
  <tr>
    <td>ђађа ёзунна</td>
    <td>personal information</td>
    <td>Thai</td>
    <td>134.876</td>
  </tr>
  <tr>
    <td>personas datus</td>
    <td>personal data</td>
    <td>Latvian</td>
    <td>128.843</td>
  </tr>
  <tr>
    <th colspan="4">Hotels & travels</th>
  </tr>
  <tr>
    <td>hotel</td>
    <td>hotel</td>
    <td>Malay</td>
    <td>681.806</td>
  </tr>
  <tr>
    <td>תלון</td>
    <td>hotel</td>
    <td>Hebrew</td>
    <td>633.416</td>
  </tr>
  <tr>
    <td>bilik</td>
    <td>rooms</td>
    <td>Malay</td>
    <td>295.982</td>
  </tr>
  <tr>
    <td>ห้อง โรงแรม</td>
    <td>find hotels</td>
    <td>Thai</td>
    <td>152.206</td>
  </tr>
  <tr>
    <td>tripadvisor er stolt</td>
    <td>tripadvisor is proud</td>
    <td>Norwegian Bokmål</td>
    <td>145.267</td>
  </tr>
  <tr>
    <td>ห้อง หัวหน้า</td>
    <td>book rooms</td>
    <td>Thai</td>
    <td>136.329</td>
  </tr>
  <tr>
    <td>tatil hem de i seyahatleri</td>
    <td>holidays and business travel</td>
    <td>Turkish</td>
    <td>122.057</td>
  </tr>
  <tr>
    <td>hôtel</td>
    <td>hotels</td>
    <td>Icelandic</td>
    <td>113.560</td>
  </tr>
  <tr>
    <td>kävelymatkan päässä</td>
    <td>walking distance</td>
    <td>Finnish</td>
    <td>103.208</td>
  </tr>
  <tr>
    <th colspan="4">European Union</th>
  </tr>
  <tr>
    <td>euroopa liidu</td>
    <td>european union</td>
    <td>Estonian</td>
    <td>45.900</td>
  </tr>
  <tr>
    <td>eiropas parlamenta un padomes</td>
    <td>european parliament and council</td>
    <td>Latvian</td>
    <td>33.524</td>
  </tr>
  <tr>
    <td>европейский парламент и на съвета</td>
    <td>european parliament and council</td>
    <td>Bulgarian</td>
    <td>31.860</td>
  </tr>
  <tr>
    <td>euros parlamento ir tarybos</td>
    <td>european parliament and council</td>
    <td>Lithuanian</td>
    <td>30.219</td>
  </tr>
  <tr>
    <td>evropskega parlamenta in sveta</td>
    <td>of the european parliament and council</td>
    <td>Slovene</td>
    <td>27.164</td>
  </tr>
  <tr>
    <td>euroopan parlamentin ja neuvoston</td>
    <td>european parliament and council</td>
    <td>Finnish</td>
    <td>24.667</td>
  </tr>
  <tr>
    <td>tal-parlament ewropew</td>
    <td>of the european parliament</td>
    <td>Maltese</td>
    <td>21.479</td>
  </tr>
  <tr>
    <th colspan="4">Boilerplate</th>
  </tr>
  <tr>
    <td>הצג מפה</td>
    <td>show map</td>
    <td>Hebrew</td>
    <td>344.614</td>
  </tr>
  <tr>
    <td>지도 보기</td>
    <td>show map</td>
    <td>Korean</td>
    <td>304.644</td>
  </tr>
  <tr>
    <td>haritay göster</td>
    <td>show map</td>
    <td>Turkish</td>
    <td>191.672</td>
  </tr>
  <tr>
    <td>عرض الخريطة</td>
    <td>show map</td>
    <td>Arabic</td>
    <td>187.615</td>
  </tr>
  <tr>
    <td>kullancsndan yenido bloglad</td>
    <td>reblogged from</td>
    <td>Turkish</td>
    <td>180.189</td>
  </tr>
  <tr>
    <td>עיטור דף מוקד</td>
    <td>edit code</td>
    <td>Hebrew</td>
    <td>83.767</td>
  </tr>
  <tr>
    <td>modifica el codi</td>
    <td>edit code</td>
    <td>Catalan</td>
    <td>64.734</td>
  </tr>
  <tr>
    <td>editar a fonte</td>
    <td>edit source</td>
    <td>Galician</td>
    <td>54.873</td>
  </tr>
  <tr>
    <td>редактиране на кода</td>
    <td>edit code</td>
    <td>Bulgarian</td>
    <td>46.778</td>
  </tr>
  <tr>
    <td>लिंक पर क्लिक</td>
    <td>click link</td>
    <td>Hindi</td>
    <td>30.258</td>
  </tr>
  <tr>
    <th colspan="4">Religious & biblical</th>
  </tr>
  <tr>
    <td>mungu</td>
    <td>god</td>
    <td>Swahili</td>
    <td>166.221</td>
  </tr>
  <tr>
    <td>uyehova</td>
    <td>jehovah</td>
    <td>Xhosa</td>
    <td>24.281</td>
  </tr>
  <tr>
    <td>heilige gees</td>
    <td>holy spirit</td>
    <td>Afrikaans</td>
    <td>14.284</td>
  </tr>
  <tr>
    <td>thotë zoti</td>
    <td>says god</td>
    <td>Albanian</td>
    <td>13.480</td>
  </tr>
  <tr>
    <td>uyesu</td>
    <td>jesus</td>
    <td>Xhosa</td>
    <td>9.135</td>
  </tr>
  <tr>
    <td>diras la eternulo</td>
    <td>says the lord</td>
    <td>Esperanto</td>
    <td>7.425</td>
  </tr>
  <tr>
    <th colspan="4">Software & games</th>
  </tr>
  <tr>
    <td>permainan dalam talian</td>
    <td>online game</td>
    <td>Malay</td>
    <td>19.342</td>
  </tr>
  <tr>
    <td>این نرم افزار</td>
    <td>this software</td>
    <td>Farsi</td>
    <td>18.992</td>
  </tr>
  <tr>
    <td>call of duty</td>
    <td>call of duty</td>
    <td>Farsi</td>
    <td>18.765</td>
  </tr>
  <tr>
    <td>mchezo huu</td>
    <td>this game</td>
    <td>Swahili</td>
    <td>16.014</td>
  </tr>
  <tr>
    <td>gêm ar-lein</td>
    <td>online game</td>
    <td>Welsh</td>
    <td>11.774</td>
  </tr>
  <tr>
    <td>luaj online flash</td>
    <td>play online flash</td>
    <td>Albanian</td>
    <td>9.298</td>
  </tr>
  <tr>
    <td>споделување на играта</td>
    <td>sharing the game</td>
    <td>Macedonian</td>
    <td>8.170</td>
  </tr>
</table>

Table 9: Frequent n-grams in parallel datasets (non-English side).
<table>
  <tr>
    <th>Dataset</th>
    <th>% of documents</th>
  </tr>
  <tr>
    <th colspan="2">Blogging platforms</th>
  </tr>
  <tr>
    <td>Standard Malay</td>
    <td>50%</td>
  </tr>
  <tr>
    <td>Magahi</td>
    <td>48%</td>
  </tr>
  <tr>
    <td>Greek</td>
    <td>24%</td>
  </tr>
  <tr>
    <td>Cantonese</td>
    <td>22%</td>
  </tr>
  <tr>
    <td>Portuguese</td>
    <td>16%</td>
  </tr>
  <tr>
    <td>Finnish</td>
    <td>16%</td>
  </tr>
  <tr>
    <td>Swedish</td>
    <td>13%</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>10%</td>
  </tr>
  <tr>
    <th colspan="2">Wikipedia</th>
  </tr>
  <tr>
    <td>Santali</td>
    <td>90%</td>
  </tr>
  <tr>
    <td>Ligurian</td>
    <td>80%</td>
  </tr>
  <tr>
    <td>Waray</td>
    <td>74%</td>
  </tr>
  <tr>
    <td>Iloko</td>
    <td>66%</td>
  </tr>
  <tr>
    <td>Esperanto</td>
    <td>66%</td>
  </tr>
  <tr>
    <td>Occitan</td>
    <td>62%</td>
  </tr>
  <tr>
    <td>Sicilian</td>
    <td>55%</td>
  </tr>
  <tr>
    <th colspan="2">News & media</th>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>61%</td>
  </tr>
  <tr>
    <td>Tigrinya</td>
    <td>48%</td>
  </tr>
  <tr>
    <td>Banjari (Arabic)</td>
    <td>46%</td>
  </tr>
  <tr>
    <td>Nigerian Fulfulde</td>
    <td>38%</td>
  </tr>
  <tr>
    <td>Turkmen</td>
    <td>32%</td>
  </tr>
  <tr>
    <td>Kyrgyz</td>
    <td>30%</td>
  </tr>
  <tr>
    <td>Rundi</td>
    <td>29%</td>
  </tr>
  <tr>
    <th colspan="2">Religious & biblical</th>
  </tr>
  <tr>
    <td>Dyula</td>
    <td>99%</td>
  </tr>
  <tr>
    <td>Fon</td>
    <td>96%</td>
  </tr>
  <tr>
    <td>Bemba</td>
    <td>95%</td>
  </tr>
  <tr>
    <td>Tumbuka</td>
    <td>94%</td>
  </tr>
  <tr>
    <td>Kamba</td>
    <td>93%</td>
  </tr>
  <tr>
    <td>Chokwe</td>
    <td>92%</td>
  </tr>
  <tr>
    <td>Central Kanuri (Latin)</td>
    <td>91%</td>
  </tr>
  <tr>
    <td>Luba-Lulua</td>
    <td>88%</td>
  </tr>
  <tr>
    <td>Sango</td>
    <td>88%</td>
  </tr>
  <tr>
    <td>Umbundu</td>
    <td>84%</td>
  </tr>
</table>

Table 10: Languages with the biggest proportion of frequent domain classes in the monolingual HPLT v2 corpora.

<table>
  <tr>
    <th>Language</th>
    <th>% of segments</th>
  </tr>
  <tr>
    <th colspan="2">Hotels & travels</th>
  </tr>
  <tr>
    <td>Icelandic</td>
    <td>42%</td>
  </tr>
  <tr>
    <td>Malay</td>
    <td>34%</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>26%</td>
  </tr>
  <tr>
    <td>Lithuanian</td>
    <td>21%</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>18%</td>
  </tr>
  <tr>
    <td>Thai</td>
    <td>16%</td>
  </tr>
  <tr>
    <td>Norwegian Bokmål</td>
    <td>16%</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>15%</td>
  </tr>
  <tr>
    <th colspan="2">Wikipedia</th>
  </tr>
  <tr>
    <td>Norwegian Nynorsk</td>
    <td>72%</td>
  </tr>
  <tr>
    <td>Galician</td>
    <td>37%</td>
  </tr>
  <tr>
    <td>Esperanto</td>
    <td>36%</td>
  </tr>
  <tr>
    <td>Kannada</td>
    <td>22%</td>
  </tr>
  <tr>
    <td>Macedonian</td>
    <td>21%</td>
  </tr>
  <tr>
    <td>Telugu</td>
    <td>19%</td>
  </tr>
  <tr>
    <td>Catalan</td>
    <td>19%</td>
  </tr>
  <tr>
    <th colspan="2">Religious & biblical</th>
  </tr>
  <tr>
    <td>Xhosa</td>
    <td>70%</td>
  </tr>
  <tr>
    <td>Esperanto</td>
    <td>28%</td>
  </tr>
  <tr>
    <td>Swahili</td>
    <td>20%</td>
  </tr>
  <tr>
    <td>Nepali</td>
    <td>16%</td>
  </tr>
  <tr>
    <td>Icelandic</td>
    <td>14%</td>
  </tr>
  <tr>
    <td>Albanian</td>
    <td>14%</td>
  </tr>
</table>

Table 11: Frequent domain classes in parallel HPLT v2 datasets for different languages (non-English side).

tries TLDs in the top-10 from closely related countries or territories (for example, from former colonial rulers (i.e. African languages datasets) or with geostrategic interests (i.e. .ru (Russia) appearing in all former Soviet states). This may indicate "language contamination" in the data.

H Manual quality inspection

In this section, we study how the quality of the extracted texts varies between older and newer crawls, and also between IA and CC crawls. More specifically, for a particular language we wanted to understand if there are any substantial differences in the proportions of texts classified as this language by mistake or just undesirable texts.

For this study, we carried out manual annotation of documents from the cleaned version of our dataset asking our annotators to provide three binary annotations for each document.

• LID ok: 0 if most of the text is not in the target language, otherwise 1;
• Unnatural: 1 if most of the text looks unnatural (e.g. word lists for SEO, mostly boilerplate, etc.), otherwise leave empty;
• Porn: 1 if the text looks like pornographic content, otherwise leave empty.

We compared four groups of crawls: among wide IA crawls and CC crawls separately we selected old crawls from 2012-2014 and new crawls from 2017-2020. Among languages spoken by the paper authors, 22 languages were selected for annotation.

For each language and each group of crawls, 50 random documents from the cleaned version of our datasets were annotated by a native or a fluent speaker of this language. In total, 200 documents for each language were annotated, except for Russian where three native speakers annotated 600 documents. Only texts extracted from the documents were shown to the annotators, they did not know which crawl each text came from or any other
<table>
  <tr>
    <th>Language</th>
    <th>% of documents</th>
    <th>TLD</th>
    <th>Country or Territory</th>
  </tr>
  <tr>
    <th colspan="4">One geographic TLD</th>
  </tr>
  <tr>
    <td>Manipuri</td>
    <td>80%</td>
    <td>.in</td>
    <td>India</td>
  </tr>
  <tr>
    <td>Lithuanian</td>
    <td>79%</td>
    <td>.lt</td>
    <td>Lithuania</td>
  </tr>
  <tr>
    <td>Polish</td>
    <td>77%</td>
    <td>.pl</td>
    <td>Poland</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>76%</td>
    <td>.hu</td>
    <td>Hungary</td>
  </tr>
  <tr>
    <td>Danish</td>
    <td>76%</td>
    <td>.dk</td>
    <td>Denmark</td>
  </tr>
  <tr>
    <td>Icelandic</td>
    <td>73%</td>
    <td>.is</td>
    <td>Iceland</td>
  </tr>
  <tr>
    <td>Faroese</td>
    <td>73%</td>
    <td>.fo</td>
    <td>Faroe Islands</td>
  </tr>
  <tr>
    <td>Macedonian</td>
    <td>73%</td>
    <td>.mk</td>
    <td>N. Macedonia</td>
  </tr>
  <tr>
    <td>Latgalian</td>
    <td>73%</td>
    <td>.lv</td>
    <td>Latvia</td>
  </tr>
  <tr>
    <td>Latvian</td>
    <td>72%</td>
    <td>.lv</td>
    <td>Latvia</td>
  </tr>
  <tr>
    <th colspan="4">Related territories</th>
  </tr>
  <tr>
    <td>Slovak</td>
    <td>77%</td>
    <td>.sk</td>
    <td>Slovakia</td>
  </tr>
  <tr>
    <td></td>
    <td>3%</td>
    <td>.cz</td>
    <td>Czechia</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>71%</td>
    <td>.kz</td>
    <td>Kazakhstan</td>
  </tr>
  <tr>
    <td></td>
    <td>3%</td>
    <td>.ru</td>
    <td>Russia</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>65%</td>
    <td>.ru</td>
    <td>Russia</td>
  </tr>
  <tr>
    <td></td>
    <td>5%</td>
    <td>.ua</td>
    <td>Ukraine</td>
  </tr>
  <tr>
    <td></td>
    <td>2%</td>
    <td>.by</td>
    <td>Belarus</td>
  </tr>
  <tr>
    <td>Croatian</td>
    <td>47%</td>
    <td>.hr</td>
    <td>Croatia</td>
  </tr>
  <tr>
    <td></td>
    <td>3%</td>
    <td>.ba</td>
    <td>Bosnia</td>
  </tr>
  <tr>
    <td></td>
    <td>3%</td>
    <td>.rs</td>
    <td>Serbia</td>
  </tr>
  <tr>
    <td>Kyrgyz</td>
    <td>33%</td>
    <td>.kg</td>
    <td>Kyrgyzstan</td>
  </tr>
  <tr>
    <td></td>
    <td>7%</td>
    <td>.ru</td>
    <td>Russia</td>
  </tr>
  <tr>
    <td>Bosnian</td>
    <td>29%</td>
    <td>.rs</td>
    <td>Serbia</td>
  </tr>
  <tr>
    <td></td>
    <td>13%</td>
    <td>.ba</td>
    <td>Bosnia</td>
  </tr>
  <tr>
    <th colspan="4">Language variants</th>
  </tr>
  <tr>
    <td>Romanian</td>
    <td>72%</td>
    <td>.ro</td>
    <td>Romania</td>
  </tr>
  <tr>
    <td></td>
    <td>3%</td>
    <td>.md</td>
    <td>Moldova</td>
  </tr>
  <tr>
    <td>Dutch</td>
    <td>66%</td>
    <td>.nl</td>
    <td>Netherlands</td>
  </tr>
  <tr>
    <td></td>
    <td>12%</td>
    <td>.be</td>
    <td>Belgium</td>
  </tr>
  <tr>
    <td>German</td>
    <td>60%</td>
    <td>.de</td>
    <td>Germany</td>
  </tr>
  <tr>
    <td></td>
    <td>6%</td>
    <td>at</td>
    <td>Austria</td>
  </tr>
  <tr>
    <td></td>
    <td>5%</td>
    <td>.ch</td>
    <td>Switzerland</td>
  </tr>
  <tr>
    <td>Portuguese</td>
    <td>45%</td>
    <td>.br</td>
    <td>Brazil</td>
  </tr>
  <tr>
    <td></td>
    <td>9%</td>
    <td>.pt</td>
    <td>Portugal</td>
  </tr>
  <tr>
    <td>Lombard</td>
    <td>47%</td>
    <td>.ch</td>
    <td>Switzerland</td>
  </tr>
  <tr>
    <td></td>
    <td>5%</td>
    <td>.it</td>
    <td>Italy</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>36%</td>
    <td>.cn</td>
    <td>China</td>
  </tr>
  <tr>
    <td></td>
    <td>3%</td>
    <td>.kz</td>
    <td>Kazakhstan</td>
  </tr>
  <tr>
    <td>French</td>
    <td>30%</td>
    <td>.fr</td>
    <td>France</td>
  </tr>
  <tr>
    <td></td>
    <td>3%</td>
    <td>.be</td>
    <td>Belgium</td>
  </tr>
  <tr>
    <td></td>
    <td>2%</td>
    <td>.ca</td>
    <td>Canada</td>
  </tr>
  <tr>
    <td></td>
    <td>2%</td>
    <td>.ch</td>
    <td>Switzerland</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>15%</td>
    <td>.es</td>
    <td>Spain</td>
  </tr>
  <tr>
    <td></td>
    <td>4%</td>
    <td>.ar</td>
    <td>Argentina</td>
  </tr>
  <tr>
    <td></td>
    <td>4%</td>
    <td>.mx</td>
    <td>Mexico</td>
  </tr>
  <tr>
    <td></td>
    <td>2%</td>
    <td>.cl</td>
    <td>Chile</td>
  </tr>
  <tr>
    <td></td>
    <td>1%</td>
    <td>.pe</td>
    <td>Peru</td>
  </tr>
</table>

Table 12: Frequent geographic TLDs in monolingual HPLT v2 datasets for different languages.

<table>
  <tr>
    <th>Language</th>
    <th>% of segments</th>
    <th>TLD</th>
    <th>Country or Territory</th>
  </tr>
  <tr>
    <th colspan="4">One geographic TLD</th>
  </tr>
  <tr>
    <td>Norwegian</td>
    <td>36%</td>
    <td>.no</td>
    <td>Norway</td>
  </tr>
  <tr>
    <td>Nynorsk</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Norwegian</td>
    <td>34%</td>
    <td>.no</td>
    <td>Norway</td>
  </tr>
  <tr>
    <td>Bokmål</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>33%</td>
    <td>.az</td>
    <td>Azerbaijan</td>
  </tr>
  <tr>
    <td>Macedonian</td>
    <td>25%</td>
    <td>.mk</td>
    <td>North Macedonia</td>
  </tr>
  <tr>
    <td>Vietnamese</td>
    <td>23%</td>
    <td>.vn</td>
    <td>Vietnam</td>
  </tr>
  <tr>
    <td>Farsi</td>
    <td>22%</td>
    <td>.ir</td>
    <td>Iran</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>20%</td>
    <td>.il</td>
    <td>Israel</td>
  </tr>
  <tr>
    <td>Sinhala</td>
    <td>19%</td>
    <td>.lk</td>
    <td>Sri Lanka</td>
  </tr>
  <tr>
    <td>Serbian</td>
    <td>16%</td>
    <td>.rs</td>
    <td>Serbia</td>
  </tr>
  <tr>
    <td>Malay</td>
    <td>15%</td>
    <td>.my</td>
    <td>Malaysia</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>15%</td>
    <td>.in</td>
    <td>India</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>15%</td>
    <td>.jp</td>
    <td>Japan</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>15%</td>
    <td>.kr</td>
    <td>South Korea</td>
  </tr>
  <tr>
    <th colspan="4">Related territories (European Union)</th>
  </tr>
  <tr>
    <td>Maltese</td>
    <td>68%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td></td>
    <td>4%</td>
    <td>.mt</td>
    <td>Malta</td>
  </tr>
  <tr>
    <td>Slovene</td>
    <td>31%</td>
    <td>.si</td>
    <td>Slovenia</td>
  </tr>
  <tr>
    <td></td>
    <td>17%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td>Estonian</td>
    <td>35%</td>
    <td>.ee</td>
    <td>Estonia</td>
  </tr>
  <tr>
    <td></td>
    <td>16%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td>Latvian</td>
    <td>30%</td>
    <td>.lv</td>
    <td>Latvia</td>
  </tr>
  <tr>
    <td></td>
    <td>16%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td>Lithuanian</td>
    <td>35%</td>
    <td>.lt</td>
    <td>Lithuania</td>
  </tr>
  <tr>
    <td></td>
    <td>14%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td>Slovak</td>
    <td>44%</td>
    <td>.sk</td>
    <td>Slovakia</td>
  </tr>
  <tr>
    <td></td>
    <td>11%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td></td>
    <td>4%</td>
    <td>.cz</td>
    <td>Czechia</td>
  </tr>
  <tr>
    <td>Croatian</td>
    <td>26%</td>
    <td>.hr</td>
    <td>Croatia</td>
  </tr>
  <tr>
    <td></td>
    <td>10%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td></td>
    <td>2%</td>
    <td>.ba</td>
    <td>Bosnia</td>
  </tr>
  <tr>
    <td>Bulgarian</td>
    <td>26%</td>
    <td>.bg</td>
    <td>Bulgaria</td>
  </tr>
  <tr>
    <td></td>
    <td>10%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td>Irish</td>
    <td>20%</td>
    <td>.ie</td>
    <td>Ireland</td>
  </tr>
  <tr>
    <td></td>
    <td>10%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
  <tr>
    <td>Finnish</td>
    <td>44%</td>
    <td>.fi</td>
    <td>Finland</td>
  </tr>
  <tr>
    <td></td>
    <td>7%</td>
    <td>.eu</td>
    <td>European Union</td>
  </tr>
</table>

Table 13: Frequent geographic TLDs in the parallel HPLT v2 datasets for different languages (non-English side).
<table>
  <tr>
    <th>Language</th>
    <th>% Porn ↓</th>
    <th>% Unnat. ↓</th>
    <th>% LID ↑</th>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>0 (-)</td>
    <td>9 (5-13)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Asturian</td>
    <td>0 (-)</td>
    <td>28 (22-35)</td>
    <td>69 (62-75)</td>
  </tr>
  <tr>
    <td>Bengali</td>
    <td>1 (-)</td>
    <td>0 (-)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Catalan</td>
    <td>0 (-)</td>
    <td>14 (9-19)</td>
    <td>99 (-)</td>
  </tr>
  <tr>
    <td>Czech</td>
    <td>0 (-)</td>
    <td>9 (4-13)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>0 (-)</td>
    <td>25 (18-31)</td>
    <td>99 (-)</td>
  </tr>
  <tr>
    <td>Dutch</td>
    <td>1 (-)</td>
    <td>5 (-)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>English</td>
    <td>1 (-)</td>
    <td>13 (8-18)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Finnish</td>
    <td>1 (-)</td>
    <td>4 (-)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>German</td>
    <td>1 (-)</td>
    <td>2 (-)</td>
    <td>98 (-)</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>2 (-)</td>
    <td>2 (-)</td>
    <td>98 (-)</td>
  </tr>
  <tr>
    <td>Iran. Persian</td>
    <td>0 (-)</td>
    <td>25 (18-31)</td>
    <td>99 (-)</td>
  </tr>
  <tr>
    <td>Marathi</td>
    <td>0 (-)</td>
    <td>6 (-)</td>
    <td>97 (-)</td>
  </tr>
  <tr>
    <td>Modern Greek</td>
    <td>0 (-)</td>
    <td>3 (-)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Nor. Bokmål</td>
    <td>2 (-)</td>
    <td>8 (4-11)</td>
    <td>99 (-)</td>
  </tr>
  <tr>
    <td>Nor. Nynorsk</td>
    <td>0 (-)</td>
    <td>3 (-)</td>
    <td>93 (-)</td>
  </tr>
  <tr>
    <td>Polish</td>
    <td>1 (-)</td>
    <td>7 (3-11)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>2 (1-3)</td>
    <td>18 (15-21)</td>
    <td>98 (-)</td>
  </tr>
  <tr>
    <td>Scot. Gaelic</td>
    <td>0 (-)</td>
    <td>3 (-)</td>
    <td>89 (85-93)</td>
  </tr>
  <tr>
    <td>Slovak</td>
    <td>0 (-)</td>
    <td>10 (6-14)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>1 (-)</td>
    <td>9 (5-13)</td>
    <td>100 (-)</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>6 (-)</td>
    <td>10 (5-14)</td>
    <td>99 (-)</td>
  </tr>
</table>

Table 14: Manual quality inspection of a random sample of documents from the cleaned version, stratified by crawls groups. Percentages of extracted texts considered as pornography (% Porn), unnatural texts (% Unnat.), and texts correctly classified by language identification (% LID) (the 95% confidence intervals for the percentage estimates are given in brackets when applicable).

meta-information. For documents longer than 1000 characters, the first 500 characters and 500 characters from the beginning of the second half were shown.

Table 14 shows the results for the four groups combined together.33 We see that the proportion of pornographic content is low, usually between 0-2% with a maximum of 6% for Turkish. The precision of our LID model for the inspected languages is above 97%, with a few notable exceptions. The worst precision is for Asturian where we observed about 30% of texts being in Spanish or other Spanish minority languages (e.g. Extremeno, Aragonese), or just SEO lists consisting of e.g. song names not in Asturian. The proportions of unnatural texts vary a lot from language to language. Annotators report the following major types of unnaturalness: lists of services and goods, commercial ads with varying degrees of grammaticality, traces of Wikipedia markup, documents consisting mostly of menus, and boilerplate missed by boilerplate removal.

33 Since the sample is stratified by group and the crawls from these groups give about 52% of all texts in our dataset, one should carefully interpret these statistics in the context of the full dataset.

Figure 9 shows proportions of unnatural texts for each language and group of crawls. Looking at individual languages, for most of them the group of new CC crawls give a much lower proportion of unnatural texts than other groups. However, since only 50 documents were labelled from each group and language, the confidence intervals are large and statistically significant conclusions cannot be made for each individual language. However, when annotations for all languages are combined (denoted as TOTAL on the figure) it becomes clear that for a random language (among those annotated) a random document has about 2x lower probability to be unnatural if it comes from the group of newer CC crawls compared to older CC crawls or any of two groups of IA crawls. For the proportions of pornographic content and documents misclassified by LID we did not observe any consistent differences for different groups of crawls.
1 Model training and evaluation

I.1 Corpora comparison: English

Pretraining We fully replicated the original FineWeb training and evaluation setup by Penedo et al. (2024a), with the same architecture and pre-training settings (1.71B parameters, Llama architecture with a sequence length of 2048 tokens, GPT 2 tokenizer, and a global batch size of ~2 million tokens). We train 4 models that are differentiated only by training data, and evaluate their performance at different stages of model training. Each model is trained on 100 billion tokens, randomly sampled from the following datasets:
• English HPLT v2 data, cleaned
• English HPLT v2 data, deduplicated
• English HPLT v1.2 (de Gibert et al., 2024)
• FineWeb dataset (Penedo et al., 2024a)

We use NVIDIA’s Megatron-LM (https://github.com/NVIDIA/Megatron-LM) training framework instead of HuggingFace’s nanotron (https://github.com/huggingface/nanotron) framework used by Penedo et al. (2024a). Each model is trained on the LUMI supercomputer with 16 nodes, each with 4 AMD MI250x GPUs with dual-GCD (graphics compute die) design, amounting to 8 logical devices. In total, we used 128 devices and a single 64-core CPU for approximately 84 hours, totalling 11 008 GPU hours per model.

Evaluation Evaluation is performed using HuggingFace’s LightEval tool (Fourrier et al., 2023) on the tasks listed below. Results per task are presented in Figure 10.
• HellaSwag: a dataset to evaluate commonsense reasoning. Its questions are designed to be trivial for humans but challenging for LLMs (Zellers et al., 2019).
• PIQA: a dataset focusing on reasoning with multiple-choice questions about physical interactions, evaluating the LLM’s understanding of how different objects are used in various situations (Bisk et al., 2020).
• OpenBookQA: a dataset consisting of multiple-choice questions which require understanding concepts and their relations, benchmarking the complex reasoning and inference performance of the LLM (Mihaylov et al., 2018).
• ARC Easy and ARC Challenge: both parts of the AI2 Reasoning Challenge dataset, containing easier and more complex questions to test the LLM’s reasoning skills (Clark et al., 2018).

I.2 Corpora comparison: Norwegian

Pretraining We mirrored the pretraining setup used for the English ablation studies in Appendix I.1, except for two details: 1) we trained a new tokenizer specifically for Norwegian, using a single tokenizer for all experiments trained on equal number of samples from all ablated corpora using the tokenizers library; 2) we pretrained the models for 30B tokens (roughly corresponding to 1 epoch on most of the ablated corpora) instead of 100B, mirroring the multilingual experiments for FineTasks (Kydliček et al., 2024).

We compared five different filtered corpora that support Norwegian. Most of these discriminate between two written variants of Norwegian – Bokmål and Nynorsk – in those cases, we simply concatenate these subcorpora. The ablated corpora are:
• Norwegian HPLT v2 data, cleaned;
• Norwegian CulturaX (Nguyen et al., 2024a);
• Norwegian HPLT v1.2 (de Gibert et al., 2024);
• Norwegian FineWeb-2 (Penedo et al., 2024b);
• Norwegian mC4 (Xue et al., 2021).

The pretraining code is built on the Megatron-DeepSpeed framework (Smith et al., 2022). All models were trained on the LUMI supercomputer using 32 compute nodes, each with 4 AMD MI250x GPUs. The full pretraining run of each model took approximately 15 hours (wall-clock time), or 1 920 GPU-hours (15 × 32 × 4 hours), respectively.

Evaluation Evaluation is performed using NorEval (Mikhailov et al., 2025a), an open-source benchmark for Norwegian built upon LM Evaluation Harness (Gao et al., 2024). We consider the following ten multiple-choice QA, generative QA, sentence completion, and sentence pair ranking tasks that target different aspects of the model understanding and generation abilities in Norwegian Bokmål and Nynorsk:
• Commonsense reasoning: performing logical and commonsense reasoning (NorCommonsenseQA, Mikhailov et al., 2025b).
• Norwegian-specific & world knowledge: answering questions about facts and Norwegian culture (NorOpenBookQA and NRK-Quiz-QA, Mikhailov et al., 2025b).
• Norwegian language knowledge: understanding Norwegian punctuation rules (NCB,
Figure 10: Final checkpoint scores of the English models trained on the datasets shown, grouped based on the benchmarks conducted. The models perform quite similarly with the exception of the model trained on the HPLT v1.2 dataset, the scores of which are noticeably lower.

Mikhailov et al., 2025a)34 and idioms (NorIdiom, Mikhailov et al., 2025a).35

• Machine reading comprehension: understanding a given text and extracting an answer from it (NorQuAD, Ivanova et al., 2023).

We aim to find tasks that provide a reliable signal during pretraining. We evaluate the models in a zero-shot regime at regular checkpoint intervals (approx. 1B tokens) on all tasks. Next, we discard tasks that provide a low signal based on two criteria (Penedo et al., 2024b):

• Monotonicity: the Spearman rank correlation between the number of steps and the target performance score is at least 0.5 across all model checkpoints.

• Non-random performance: the difference between the random baseline (zero for generative tasks, one divided by the number of answer choices for multiple-choice tasks, and a coin flip probability for sentence pair ranking tasks) and the maximum score across all models is positive and satisfactory.

The filtering results in four datasets: NCB (accuracy), NRK-Quiz-QA Bokmål (accuracy), NorCommonsenseQA Bokmål (accuracy), and NorQuAD (F1-score). We aggregate the performance across the datasets using the average normalized score (Myrzakhan et al., 2024). We report the performance results for our 150 checkpoints in Figure 5 (see §6.2) and final checkpoint performance in Figure 11.

34https://huggingface.co/datasets/hcfa/ncb
35https://huggingface.co/datasets/Sprakbanken/Norwegian_idioms

J LTG-BERTs training and evaluation details

Following the HPLT v1.2,36 we use UD treebanks of version 2.1337 for most languages, except for Albanian and Georgian. These languages were not used in the HPLT v1.2 report due to missing training and development splits in UD 2.13. However, UD 2.15 does contain the required splits, and we use them. We do not evaluate NER on Maltese, since its WikiAnn training split contains only 100 samples. Table 15 shows detailed MLM evaluation results by language and task.

LTG-BERT architecture (Samuel et al., 2023) is a version of the original masked BERT model (Devlin et al., 2019). The differences include removing the next sentence prediction objective, swapping subword masking to span masking, and other minor architectural improvements. LTG-BERT was shown to perform well for small-sized training datasets (Samuel, 2023), which fits our evaluation setup. The models were trained with the same hyperparameters as in the aforementioned HPLT report.

We trained separate models for Bosnian and Croatian, in addition to the joint Bosnian-Croatian model. Since the UD does not provide Bosnian treebanks, we evaluated all three models on the Croatian datasets. We did not include Serbian, because it uses the Cyrillic writing system in HPLT v2, while UD features Serbian data only in Latin. Exploring whether mixing the scripts still improves the

36https://hplt-project.org/HPLT_D4_1___First_language_models_trained.pdf
37https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-5287
Figure 11: Final checkpoint scores of the Norwegian models trained on the datasets shown, grouped based on the evaluation datasets in NorEva1. The models perform quite similarly with the exception of the model trained on the HPLT v1.2 dataset, the NorQuAD and NCB scores of which are generally lower.

results is left for future work. It is difficult to give any clear recommendations on which of the three models to use for practical tasks, since all of them yield satisfactory evaluation results (ranking varies from task to task).

LTG-BERT models were trained for 31 250 steps on 16 compute nodes with 4 physical AMD INSTINCT MI250x GPUs each for approximately 9.8 hours. Sharding, training a tokenizer and tokenizing for larger languages required up to 3.5, 0.5 and 1 hours correspondingly on 7 AMD EPYC 7763 CPUs (these numbers are estimated from the processing of English, the largest data subset in HPLT v2. The processing time of different languages may vary, for instance, languages without whitespace separation between words require an additional pretokenizing step). UD fine-tuning and NER fine-tuning required 1.1 hours and 8 minutes correspondingly on 1 GPU (estimated for English).

MT models for translation into English and from English respectively. For reference, we also include the performance of models trained on the HPLT v1.2 dataset, which shares the same underlying extraction pipeline. Note that we did not perform any language-specific hyper-parameter tuning which possibly led to low scores for a few model instances.

K Full Results for Translation Models Built on Parallel Data

We compare models trained on HPLT v2, Tatoeba (Tiedemann, 2012, 2020), and the combination of the two datasets. The language selection is the intersection of the languages covered by both datasets. We evaluate the models on the FLORES-200 evaluation benchmark (NLLB Team et al., 2024) using SacreBLEU implementation of BLEU\(^{38}\) and chrF++\(^{39}\) metrics (Post, 2018) and COMET-22-DA (Rei et al., 2022).

Tables 16 and 17 present the full results of the

\footnotetext{38 nrefs:1|case:mixed|eff:no|smooth:exp|version:2.5.1, and where applicable: tok:ja-mecab, tok:ko-mecab, or tok:13a}
\footnotetext{39 nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1}
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="4">POS tags</th>
    <th colspan="4">Lemmas</th>
    <th colspan="4">Dependency parsing</th>
    <th colspan="4">NER</th>
  </tr>
  <tr>
    <th>mBERT</th>
    <th>XLm-R v1.2</th>
    <th>HPLT v1.2</th>
    <th>HPLT v2</th>
    <th>mBERT</th>
    <th>XLm-R v1.2</th>
    <th>HPLT v1.2</th>
    <th>HPLT v2</th>
    <th>mBERT</th>
    <th>XLm-R v1.2</th>
    <th>HPLT v1.2</th>
    <th>HPLT v2</th>
    <th>mBERT</th>
    <th>XLm-R v1.2</th>
    <th>HPLT v1.2</th>
    <th>HPLT v2</th>
  </tr>
  <tr><td>als_Latn</td><td>99.1</td><td>61.6</td><td>64.5</td><td>78.2</td><td>75.0</td><td>76.3</td><td>77.2</td><td>33.1</td><td>29.5</td><td>25.3</td><td>24.7</td><td>92.3</td><td>92.9</td><td>92.4</td><td>93.9</td></tr>
  <tr><td>bel_Cyr</td><td>94.1</td><td>94.6</td><td>95.5</td><td>95.7</td><td>93.2</td><td>93.8</td><td>97.1</td><td>88.1</td><td>89.9</td><td>91.1</td><td>91.7</td><td>91.1</td><td>90.3</td><td>92.8</td><td>92.8</td><td>92.8</td></tr>
  <tr><td>bos_Latn</td><td>95.5</td><td>96.2</td><td>96.4</td><td>96.6</td><td>97.2</td><td>97.4</td><td>97.2</td><td>97.1</td><td>90.2</td><td>91.3</td><td>91.3</td><td>91.7</td><td>91.5</td><td>91.6</td><td>89.3</td><td>92.8</td></tr>
  <tr><td>hrv_Latn</td><td>95.5</td><td>96.2</td><td>96.4</td><td>96.8</td><td>97.2</td><td>97.4</td><td>97.2</td><td>97.1</td><td>90.2</td><td>91.3</td><td>91.3</td><td>91.7</td><td>91.5</td><td>91.6</td><td>89.3</td><td>92.5</td></tr>
  <tr><td>bul_Cyr</td><td>97.9</td><td>97.9</td><td>97.8</td><td>97.5</td><td>97.3</td><td>97.3</td><td>97.3</td><td>97.2</td><td>94.4</td><td>94.4</td><td>94.5</td><td>92.2</td><td>92.2</td><td>91.5</td><td>91.5</td><td>93.0</td></tr>
  <tr><td>ces_Latn</td><td>97.1</td><td>97.2</td><td>97.4</td><td>97.5</td><td>99.4</td><td>99.4</td><td>99.4</td><td>97.5</td><td>91.6</td><td>94.4</td><td>94.4</td><td>94.4</td><td>91.0</td><td>91.0</td><td>90.1</td><td>94.5</td></tr>
  <tr><td>cze_Latn</td><td>97.8</td><td>98.0</td><td>98.3</td><td>98.4</td><td>99.3</td><td>99.3</td><td>99.4</td><td>99.4</td><td>93.5</td><td>94.2</td><td>94.4</td><td>94.6</td><td>91.2</td><td>91.2</td><td>89.0</td><td>91.8</td></tr>
  <tr><td>cym_Latn</td><td>87.2</td><td>88.3</td><td>89.2</td><td>90.0</td><td>94.6</td><td>94.4</td><td>93.7</td><td>92.3</td><td>80.8</td><td>82.8</td><td>82.8</td><td>82.5</td><td>90.0</td><td>89.4</td><td>89.4</td><td>93.4</td></tr>
  <tr><td>dan_Latn</td><td>97.9</td><td>97.9</td><td>97.9</td><td>97.9</td><td>97.6</td><td>97.6</td><td>97.6</td><td>97.6</td><td>88.2</td><td>88.2</td><td>88.2</td><td>88.2</td><td>91.1</td><td>91.1</td><td>90.8</td><td>92.0</td></tr>
  <tr><td>deu_Latn</td><td>88.8</td><td>89.4</td><td>89.7</td><td>89.9</td><td>97.6</td><td>97.7</td><td>95.5</td><td>97.5</td><td>84.6</td><td>87.1</td><td>76.4</td><td>87.6</td><td>89.4</td><td>87.7</td><td>64.1</td><td>89.2</td></tr>
  <tr><td>ell_Grek</td><td>94.6</td><td>95.7</td><td>96.1</td><td>96.2</td><td>94.6</td><td>94.7</td><td>94.1</td><td>94.1</td><td>91.7</td><td>93.5</td><td>92.2</td><td>93.2</td><td>90.2</td><td>90.7</td><td>90.2</td><td>92.6</td></tr>
  <tr><td>eng_Lam</td><td>98.1</td><td>98.6</td><td>98.7</td><td>98.7</td><td>98.1</td><td>98.1</td><td>98.1</td><td>98.1</td><td>91.2</td><td>92.6</td><td>92.6</td><td>92.6</td><td>91.1</td><td>91.1</td><td>91.1</td><td>92.7</td></tr>
  <tr><td>est_Latn</td><td>96.0</td><td>96.6</td><td>97.1</td><td>97.1</td><td>94.8</td><td>95.0</td><td>95.2</td><td>95.2</td><td>88.1</td><td>89.7</td><td>90.8</td><td>91.0</td><td>91.8</td><td>90.4</td><td>89.6</td><td>93.0</td></tr>
  <tr><td>eus_Latn</td><td>91.0</td><td>91.4</td><td>92.3</td><td>92.3</td><td>95.7</td><td>95.9</td><td>96.0</td><td>95.9</td><td>85.3</td><td>87.3</td><td>88.1</td><td>88.2</td><td>91.3</td><td>90.7</td><td>89.8</td><td>92.9</td></tr>
  <tr><td>faa_Latn</td><td>98.0</td><td>98.4</td><td>98.6</td><td>98.3</td><td>99.4</td><td>99.4</td><td>99.4</td><td>99.4</td><td>95.2</td><td>97.0</td><td>97.0</td><td>97.0</td><td>92.9</td><td>92.9</td><td>92.9</td><td>95.8</td></tr>
  <tr><td>fin_Latn</td><td>95.1</td><td>96.4</td><td>96.8</td><td>97.0</td><td>90.6</td><td>91.5</td><td>91.6</td><td>91.4</td><td>90.2</td><td>93.0</td><td>93.3</td><td>94.0</td><td>90.2</td><td>90.0</td><td>90.2</td><td>91.6</td></tr>
  <tr><td>fra_Latn</td><td>97.8</td><td>98.1</td><td>98.0</td><td>98.0</td><td>98.8</td><td>98.8</td><td>98.8</td><td>98.6</td><td>93.8</td><td>94.4</td><td>94.5</td><td>94.8</td><td>90.5</td><td>88.7</td><td>87.2</td><td>90.0</td></tr>
  <tr><td>gle_Latn</td><td>86.4</td><td>87.3</td><td>88.2</td><td>89.2</td><td>95.1</td><td>95.1</td><td>95.1</td><td>95.1</td><td>83.6</td><td>84.4</td><td>84.4</td><td>84.0</td><td>88.0</td><td>88.0</td><td>88.0</td><td>89.2</td></tr>
  <tr><td>heb_Hebr</td><td>96.9</td><td>97.1</td><td>97.1</td><td>97.0</td><td>98.3</td><td>98.3</td><td>98.2</td><td>98.0</td><td>82.3</td><td>82.6</td><td>82.3</td><td>82.2</td><td>92.5</td><td>93.3</td><td>91.1</td><td>94.1</td></tr>
  <tr><td>hin_Deva</td><td>92.4</td><td>93.3</td><td>93.6</td><td>93.7</td><td>98.7</td><td>99.0</td><td>99.0</td><td>99.0</td><td>92.6</td><td>93.5</td><td>93.5</td><td>93.6</td><td>88.6</td><td>88.0</td><td>84.3</td><td>89.3</td></tr>
  <tr><td>hun_Latn</td><td>93.0</td><td>94.3</td><td>93.0</td><td>94.1</td><td>93.0</td><td>94.3</td><td>93.0</td><td>92.3</td><td>84.3</td><td>86.7</td><td>82.4</td><td>86.1</td><td>92.2</td><td>91.9</td><td>92.8</td><td>93.1</td></tr>
  <tr><td>hye_Armn</td><td>88.7</td><td>91.2</td><td>92.7</td><td>92.7</td><td>94.4</td><td>94.5</td><td>93.9</td><td>94.7</td><td>80.4</td><td>85.3</td><td>84.1</td><td>86.8</td><td>95.7</td><td>95.3</td><td>94.8</td><td>95.9</td></tr>
  <tr><td>ind_Latn</td><td>97.8</td><td>98.0</td><td>98.1</td><td>98.3</td><td>98.6</td><td>98.6</td><td>98.6</td><td>98.6</td><td>86.2</td><td>86.6</td><td>86.9</td><td>87.4</td><td>81.7</td><td>63.9</td><td>55.9</td><td>78.3</td></tr>
  <tr><td>isl_Latn</td><td>98.0</td><td>98.0</td><td>98.1</td><td>98.3</td><td>98.6</td><td>98.7</td><td>98.7</td><td>98.4</td><td>94.1</td><td>94.4</td><td>94.6</td><td>95.1</td><td>90.5</td><td>89.7</td><td>87.8</td><td>91.2</td></tr>
  <tr><td>jpn_Jpan</td><td>97.5</td><td>97.8</td><td>97.8</td><td>97.8</td><td>98.1</td><td>98.1</td><td>98.1</td><td>98.1</td><td>94.6</td><td>94.6</td><td>94.6</td><td>94.8</td><td>65.5</td><td>67.4</td><td>67.2</td><td>71.2</td></tr>
  <tr><td>kaz_Latn</td><td>91.3</td><td>92.6</td><td>92.6</td><td>92.8</td><td>93.7</td><td>93.7</td><td>92.5</td><td>92.5</td><td>79.5</td><td>80.1</td><td>80.1</td><td>81.7</td><td>87.4</td><td>87.0</td><td>86.9</td><td>90.7</td></tr>
  <tr><td>kor_Hang</td><td>88.6</td><td>89.7</td><td>89.9</td><td>90.1</td><td>94.0</td><td>94.3</td><td>94.4</td><td>94.8</td><td>88.0</td><td>89.0</td><td>89.4</td><td>89.7</td><td>87.8</td><td>87.0</td><td>88.3</td><td>89.3</td></tr>
  <tr><td>lvv_Latn</td><td>91.6</td><td>92.8</td><td>92.4</td><td>93.6</td><td>96.9</td><td>96.1</td><td>96.8</td><td>97.7</td><td>88.9</td><td>90.9</td><td>90.9</td><td>92.1</td><td>93.2</td><td>92.6</td><td>90.7</td><td>93.9</td></tr>
  <tr><td>ltz_Latn</td><td>87.7</td><td>91.9</td><td>92.0</td><td>92.5</td><td>90.2</td><td>91.6</td><td>91.5</td><td>91.2</td><td>79.3</td><td>85.7</td><td>84.9</td><td>86.8</td><td>89.1</td><td>89.3</td><td>87.0</td><td>91.0</td></tr>
  <tr><td>mkd_Cyr</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>mit_Latn</td><td>94.7</td><td>94.5</td><td>97.0</td><td>97.5</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>78.2</td><td>78.5</td><td>83.2</td><td>87.2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>mnk_Latn</td><td>97.0</td><td>97.4</td><td>97.6</td><td>97.5</td><td>98.5</td><td>98.8</td><td>98.5</td><td>98.7</td><td>93.2</td><td>94.3</td><td>94.5</td><td>94.7</td><td>91.9</td><td>92.6</td><td>91.1</td><td>93.2</td></tr>
  <tr><td>nld_Latn</td><td>96.2</td><td>96.9</td><td>97.1</td><td>97.2</td><td>94.1</td><td>94.7</td><td>94.4</td><td>94.1</td><td>91.6</td><td>92.9</td><td>93.8</td><td>94.1</td><td>91.7</td><td>90.4</td><td>88.6</td><td>91.0</td></tr>
  <tr><td>nno_Latn</td><td>96.6</td><td>97.0</td><td>97.7</td><td>97.8</td><td>98.2</td><td>98.2</td><td>98.5</td><td>98.5</td><td>92.9</td><td>93.9</td><td>94.6</td><td>95.0</td><td>95.8</td><td>93.6</td><td>93.5</td><td>95.5</td></tr>
  <tr><td>nor_Latn</td><td>96.2</td><td>96.9</td><td>97.2</td><td>97.2</td><td>98.2</td><td>98.2</td><td>98.2</td><td>98.2</td><td>95.7</td><td>95.7</td><td>95.7</td><td>95.7</td><td>88.5</td><td>88.2</td><td>88.2</td><td>90.6</td></tr>
  <tr><td>por_Latn</td><td>93.6</td><td>94.0</td><td>94.1</td><td>94.1</td><td>98.1</td><td>98.3</td><td>98.3</td><td>98.2</td><td>83.4</td><td>84.5</td><td>84.9</td><td>85.3</td><td>91.2</td><td>90.3</td><td>88.0</td><td>91.5</td></tr>
  <tr><td>ron_Latn</td><td>97.3</td><td>97.6</td><td>97.7</td><td>97.7</td><td>97.9</td><td>97.9</td><td>97.8</td><td>97.8</td><td>89.5</td><td>91.0</td><td>90.6</td><td>91.6</td><td>94.5</td><td>93.6</td><td>91.2</td><td>93.6</td></tr>
  <tr><td>russ_Latn</td><td>91.4</td><td>91.4</td><td>91.4</td><td>91.4</td><td>94.7</td><td>94.7</td><td>94.7</td><td>94.7</td><td>89.5</td><td>91.0</td><td>91.0</td><td>91.0</td><td>88.6</td><td>88.6</td><td>88.6</td><td>90.9</td></tr>
  <tr><td>sik_Latn</td><td>89.1</td><td>97.6</td><td>98.1</td><td>91.9</td><td>95.7</td><td>96.1</td><td>95.6</td><td>95.5</td><td>92.9</td><td>94.4</td><td>93.8</td><td>95.0</td><td>93.2</td><td>92.9</td><td>91.2</td><td>93.3</td></tr>
  <tr><td>slv_Latn</td><td>96.7</td><td>97.6</td><td>98.1</td><td>98.2</td><td>98.5</td><td>98.7</td><td>98.6</td><td>98.7</td><td>93.4</td><td>94.7</td><td>94.8</td><td>95.3</td><td>93.4</td><td>93.1</td><td>93.6</td><td>94.2</td></tr>
  <tr><td>ser_Cyr</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>shq_Latn</td><td>96.5</td><td>97.4</td><td>97.4</td><td>97.3</td><td>97.3</td><td>97.6</td><td>97.1</td><td>97.0</td><td>89.4</td><td>92.1</td><td>90.8</td><td>91.7</td><td>94.5</td><td>93.5</td><td>94.4</td><td>96.4</td></tr>
  <tr><td>tat_Cyr</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>tur_Latn</td><td>94.1</td><td>91.0</td><td>91.5</td><td>91.4</td><td>91.1</td><td>91.3</td><td>91.0</td><td>91.4</td><td>70.9</td><td>73.0</td><td>73.6</td><td>74.6</td><td>92.2</td><td>92.0</td><td>90.8</td><td>92.5</td></tr>
  <tr><td>ukr_Cyr</td><td>93.1</td><td>94.2</td><td>95.3</td><td>95.0</td><td>97.2</td><td>97.9</td><td>97.9</td><td>97.9</td><td>89.4</td><td>91.8</td><td>91.3</td><td>92.1</td><td>91.2</td><td>91.2</td><td>87.5</td><td>92.8</td></tr>
  <tr><td>ukr_Hans</td><td>89.8</td><td>92.1</td><td>91.8</td><td>92.1</td><td>99.9</td><td>99.9</td><td>99.9</td><td>99.9</td><td>66.5</td><td>70.3</td><td>68.0</td><td>70.3</td><td>91.9</td><td>90.6</td><td>89.2</td><td>90.3</td></tr>
  <tr><td>zho_Hans</td><td>96.2</td><td>96.0</td><td>96.0</td><td>99.9</td><td>99.9</td><td>99.9</td><td>99.9</td><td>86.1</td><td>86.9</td><td>84.6</td><td>85.6</td><td>0.1</td><td>76.5</td><td>75.5</td><td>74.5</td><td>-</td></tr>
</table>

Table 15: Results of monolingual masked language models trained on the HPLT v2 datasets compared to the baselines on part-of-speech (POS) tagging, lemmatization, dependency parsing and named entity recognition. For POS tagging, we evaluate the AllTags performance, which is the exact match accuracy of the UPOS, XPOS and UFeats UDtags. For dependency parsing, we report LAS, and for lemmatization accuracy.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">HPLT v1.2</th>
    <th colspan="3">HPLT v2</th>
    <th colspan="3">Tatoeba (OPUS)</th>
    <th colspan="3">HPLT v2+OPUS</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
  </tr>
  <tr><td>en-af</td><td>39.2</td><td>64.5</td><td>0.8398</td><td>38.3</td><td>63.6</td><td>0.8397</td><td>38.8</td><td>63.8</td><td>0.8409</td></tr>
  <tr><td>en-ar</td><td>26.6</td><td>55.0</td><td>0.8442</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr>
  <tr><td>en-az</td><td>12.2</td><td>41.0</td><td>0.8128</td><td>11.3</td><td>38.7</td><td>0.8074</td><td>11.5</td><td>38.7</td><td>0.8011</td></tr>
  <tr><td>en-be</td><td>11.6</td><td>39.0</td><td>0.7756</td><td>11.2</td><td>37.4</td><td>0.7767</td><td>11.8</td><td>38.2</td><td>0.7800</td></tr>
  <tr><td>en-bg</td><td>38.0</td><td>62.4</td><td>0.8680</td><td>0.9</td><td>14.5</td><td>0.6774</td><td>30.0</td><td>51.9</td><td>0.8122</td></tr>
  <tr><td>en-bn</td><td>16.0</td><td>45.2</td><td>0.8109</td><td>16.6</td><td>45.9</td><td>0.8282</td><td>16.8</td><td>46.1</td><td>0.8275</td></tr>
  <tr><td>en-bs</td><td>4.7</td><td>26.0</td><td>0.4314</td><td>26.8</td><td>53.9</td><td>0.8672</td><td></td><td></td><td></td><td></td><td></td><td></td></tr>
  <tr><td>en-ca</td><td>38.4</td><td>61.7</td><td>0.8461</td><td>37.8</td><td>61.0</td><td>0.8334</td><td>39.8</td><td>62.2</td><td>0.8440</td><td>39.5</td><td>62.1</td><td>0.8425</td></tr>
  <tr><td>en-cy</td><td></td><td></td><td></td><td>50.4</td><td>69.9</td><td>0.8611</td><td>47.7</td><td>67.6</td><td>0.8536</td><td>48.4</td><td>67.8</td><td>0.8506</td></tr>
  <tr><td>en-eo</td><td></td><td></td><td></td><td>27.2</td><td>54.7</td><td>0.8264</td><td>25.5</td><td>54.1</td><td>0.8500</td><td>25.9</td><td>54.4</td><td>0.8523</td></tr>
  <tr><td>en-et</td><td>23.7</td><td>53.4</td><td>0.8664</td><td>24.5</td><td>53.8</td><td>0.8684</td><td>24.5</td><td>53.3</td><td>0.8600</td><td>24.4</td><td>53.3</td><td>0.8578</td></tr>
  <tr><td>en-eu</td><td>12.1</td><td>43.4</td><td>0.7674</td><td>16.5</td><td>49.5</td><td>0.8215</td><td>14.9</td><td>47.2</td><td>0.8098</td><td>14.8</td><td>47.1</td><td>0.8122</td></tr>
  <tr><td>en-fa</td><td></td><td></td><td></td><td>21.5</td><td>47.5</td><td>0.7947</td><td>23.4</td><td>50.0</td><td>0.8336</td><td>23.6</td><td>50.0</td><td>0.8349</td></tr>
  <tr><td>en-fi</td><td></td><td></td><td></td><td>21.3</td><td>51.1</td><td>0.8709</td><td>21.3</td><td>51.2</td><td>0.8725</td><td>22.1</td><td>51.6</td><td>0.8752</td></tr>
  <tr><td>en-ga</td><td>27.3</td><td>52.6</td><td>0.7561</td><td>29.0</td><td>53.9</td><td>0.7543</td><td>30.2</td><td>53.9</td><td>0.7715</td><td>30.8</td><td>54.6</td><td>0.7717</td></tr>
  <tr><td>en-gl</td><td>27.9</td><td>54.0</td><td>0.8033</td><td>30.0</td><td>55.7</td><td>0.8179</td><td>31.4</td><td>56.1</td><td>0.8302</td><td>31.4</td><td>56.1</td><td>0.8264</td></tr>
  <tr><td>en-gu</td><td></td><td></td><td></td><td>19.3</td><td>46.5</td><td>0.8066</td><td>22.5</td><td>49.9</td><td>0.8518</td><td>22.6</td><td>49.9</td><td>0.8479</td></tr>
  <tr><td>en-he</td><td></td><td></td><td></td><td>28.1</td><td>54.0</td><td>0.8320</td><td>29.7</td><td>55.9</td><td>0.8532</td><td>29.6</td><td>55.4</td><td>0.8503</td></tr>
  <tr><td>en-hi</td><td>32.8</td><td>55.5</td><td>0.7621</td><td>32.0</td><td>54.6</td><td>0.7612</td><td>33.1</td><td>55.5</td><td>0.7728</td><td>32.5</td><td>54.9</td><td>0.7658</td></tr>
  <tr><td>en-hr</td><td></td><td></td><td></td><td>27.5</td><td>53.7</td><td>0.8504</td><td></td><td></td><td></td><td></td><td></td><td></td></tr>
  <tr><td>en-is</td><td>20.6</td><td>45.1</td><td>0.7651</td><td>22.2</td><td>47.1</td><td>0.7766</td><td>22.8</td><td>47.1</td><td>0.7800</td><td>23.1</td><td>47.5</td><td>0.7859</td></tr>
  <tr><td>en-ja</td><td></td><td></td><td></td><td>27.0</td><td>26.6</td><td>0.8244</td><td>29.9</td><td>30.2</td><td>0.8640</td><td>29.6</td><td>29.9</td><td>0.8633</td></tr>
  <tr><td>en-kk</td><td></td><td></td><td></td><td>21.0</td><td>51.4</td><td>0.8651</td><td>16.5</td><td>45.1</td><td>0.8315</td><td>16.9</td><td>45.3</td><td>0.8347</td></tr>
  <tr><td>en-kn</td><td>13.8</td><td>43.5</td><td>0.7746</td><td>19.5</td><td>50.8</td><td>0.8348</td><td>19.2</td><td>51.1</td><td>0.8369</td><td></td><td></td><td></td></tr>
  <tr><td>en-ko</td><td>25.0</td><td>31.2</td><td>0.8268</td><td>26.6</td><td>42.2</td><td>0.8424</td><td>26.5</td><td>32.0</td><td>0.8402</td><td></td><td></td><td></td></tr>
  <tr><td>en-lt</td><td>22.9</td><td>50.9</td><td>0.8214</td><td>25.0</td><td>52.7</td><td>0.8486</td><td>24.4</td><td>52.5</td><td>0.8417</td><td></td><td></td><td></td></tr>
  <tr><td>en-lv</td><td>26.8</td><td>53.1</td><td>0.8214</td><td>23.9</td><td>50.0</td><td>0.7898</td><td>24.3</td><td>50.6</td><td>0.7891</td><td></td><td></td><td></td></tr>
  <tr><td>en-mk</td><td>32.2</td><td>58.6</td><td>0.8478</td><td>33.9</td><td>60.1</td><td>0.8718</td><td>34.1</td><td>60.1</td><td>0.8660</td><td></td><td></td><td></td></tr>
  <tr><td>en-nl</td><td>0.6</td><td>20.2</td><td>0.5753</td><td>14.4</td><td>47.9</td><td>0.8458</td><td>14.6</td><td>48.0</td><td>0.8427</td><td></td><td></td><td></td></tr>
  <tr><td>en-nr</td><td>11.0</td><td>37.9</td><td>0.6086</td><td>13.9</td><td>42.3</td><td>0.6808</td><td>14.2</td><td>42.4</td><td>0.6792</td><td></td><td></td><td></td></tr>
  <tr><td>en-ms</td><td>38.3</td><td>63.9</td><td>0.8490</td><td>24.4</td><td>32.6</td><td>0.8534</td><td>25.1</td><td>33.2</td><td>0.8540</td><td></td><td></td><td></td></tr>
  <tr><td>en-mt</td><td>34.7</td><td>63.2</td><td>0.7072</td><td>35.8</td><td>64.1</td><td>0.7102</td><td>36.0</td><td>64.4</td><td>0.7111</td><td></td><td></td><td></td></tr>
  <tr><td>en-nb</td><td>33.0</td><td>58.2</td><td>0.8661</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr>
  <tr><td>en-ne</td><td>0.9</td><td>20.9</td><td>0.4880</td><td>12.9</td><td>42.8</td><td>0.7404</td><td>12.8</td><td>43.1</td><td>0.7386</td><td></td><td></td><td></td></tr>
  <tr><td>en-nn</td><td>22.8</td><td>47.1</td><td>0.7766</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr>
  <tr><td>en-si</td><td>1.2</td><td>18.6</td><td>0.6289</td><td>13.1</td><td>41.0</td><td>0.8542</td><td>13.2</td><td>41.2</td><td>0.8548</td><td></td><td></td><td></td></tr>
  <tr><td>en-sk</td><td>29.3</td><td>54.0</td><td>0.8279</td><td>29.3</td><td>53.9</td><td>0.8353</td><td>29.9</td><td>54.5</td><td>0.8423</td><td></td><td></td><td></td></tr>
  <tr><td>en-sl</td><td>26.8</td><td>52.2</td><td>0.8295</td><td>26.5</td><td>52.0</td><td>0.8339</td><td>27.5</td><td>52.6</td><td>0.8414</td><td></td><td></td><td></td></tr>
  <tr><td>en-sq</td><td>27.8</td><td>54.6</td><td>0.8509</td><td>27.7</td><td>54.2</td><td>0.8398</td><td>29.9</td><td>55.8</td><td>0.8659</td><td>29.3</td><td>55.3</td><td>0.8600</td></tr>
  <tr><td>en-sw</td><td>28.4</td><td>54.6</td><td>0.7743</td><td>32.5</td><td>58.2</td><td>0.7964</td><td>31.2</td><td>57.0</td><td>0.8058</td><td>31.3</td><td>57.0</td><td>0.8031</td></tr>
  <tr><td>en-ta</td><td>14.1</td><td>46.7</td><td>0.8139</td><td>16.7</td><td>50.3</td><td>0.8565</td><td>16.4</td><td>50.1</td><td>0.8553</td><td></td><td></td><td></td></tr>
  <tr><td>en-te</td><td>20.2</td><td>51.3</td><td>0.8104</td><td>22.1</td><td>53.7</td><td>0.8378</td><td>22.7</td><td>53.9</td><td>0.8383</td><td></td><td></td><td></td></tr>
  <tr><td>en-th</td><td>9.9</td><td>40.9</td><td>0.7977</td><td>8.1</td><td>40.6</td><td>0.8053</td><td>8.7</td><td>40.9</td><td>0.8053</td><td></td><td></td><td></td></tr>
  <tr><td>en-tr</td><td>25.3</td><td>53.7</td><td>0.8368</td><td>27.8</td><td>56.4</td><td>0.8685</td><td>27.5</td><td>55.8</td><td>0.8638</td><td></td><td></td><td></td></tr>
  <tr><td>en-uk</td><td>26.7</td><td>52.6</td><td>0.8457</td><td>27.2</td><td>53.4</td><td>0.8532</td><td>26.8</td><td>52.8</td><td>0.8471</td><td></td><td></td><td></td></tr>
  <tr><td>en-ur</td><td>18.9</td><td>43.2</td><td>0.7548</td><td>19.3</td><td>44.0</td><td>0.7537</td><td>19.5</td><td>44.6</td><td>0.7584</td><td></td><td></td><td></td></tr>
  <tr><td>en-uz</td><td>16.3</td><td>49.1</td><td>0.8397</td><td>15.9</td><td>47.3</td><td>0.8497</td><td>17.1</td><td>48.8</td><td>0.8532</td><td></td><td></td><td></td></tr>
  <tr><td>en-vi</td><td>37.8</td><td>55.8</td><td>0.8358</td><td>39.3</td><td>57.1</td><td>0.8489</td><td>38.8</td><td>56.6</td><td>0.8451</td><td></td><td></td><td></td></tr>
  <tr><td>en-xh</td><td>12.0</td><td>44.2</td><td>0.7323</td><td>0.0</td><td>3.5</td><td>0.2317</td><td>0.0</td><td>3.5</td><td>0.1647</td><td></td><td></td><td></td></tr>
</table>

Table 16: MT results (BLEU, chrF++, COMET22-DA) for models translating from English, trained on our HPLT v2, Tatoeba (OPUS), a combination of both, and the existing HPLT v1.2 (numbers reported where available).
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">HPLT v1.2</th>
    <th colspan="3">HPLT v2</th>
    <th colspan="3">Tatoeba (OPUS)</th>
    <th colspan="3">HPLT v2+OPUS</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
  </tr>
  <tr><td>az-en</td><td>18.5</td><td>47.1</td><td>0.8290</td><td>17.4</td><td>44.7</td><td>0.8039</td><td>18.6</td><td>46.2</td><td>0.8175</td></tr>
  <tr><td>be-en</td><td>16.1</td><td>46.7</td><td>0.7886</td><td>14.8</td><td>44.7</td><td>0.7621</td><td>15.5</td><td>45.8</td><td>0.7743</td></tr>
  <tr><td>bg-en</td><td>35.5</td><td>61.1</td><td>0.8556</td><td>7.4</td><td>32.5</td><td>0.5104</td><td>34.8</td><td>60.6</td><td>0.8524</td></tr>
  <tr><td>bn-en</td><td>27.9</td><td>53.7</td><td>0.8468</td><td>28.4</td><td>53.7</td><td>0.8498</td><td>29.0</td><td>54.2</td><td>0.8523</td></tr>
  <tr><td>bs-en</td><td>12.8</td><td>38.0</td><td>0.5882</td><td>35.2</td><td>59.9</td><td>0.8508</td></tr>
  <tr><td>ca-en</td><td>41.0</td><td>64.4</td><td>0.8676</td><td>39.2</td><td>63.1</td><td>0.8478</td><td>41.1</td><td>64.4</td><td>0.8580</td><td>40.3</td><td>63.9</td><td>0.8541</td></tr>
  <tr><td>cy-en</td><td>51.5</td><td>70.7</td><td>0.8615</td><td>50.0</td><td>68.8</td><td>0.8456</td><td>50.6</td><td>69.1</td><td>0.8455</td></tr>
  <tr><td>eo-en</td><td>35.5</td><td>59.2</td><td>0.8362</td><td>35.6</td><td>59.2</td><td>0.8437</td><td>35.6</td><td>59.3</td><td>0.8429</td></tr>
  <tr><td>et-en</td><td>30.6</td><td>56.6</td><td>0.8611</td><td>30.3</td><td>55.8</td><td>0.8517</td><td>30.7</td><td>56.1</td><td>0.8510</td><td>30.7</td><td>55.6</td><td>0.8510</td></tr>
  <tr><td>eu-en</td><td>19.4</td><td>45.7</td><td>0.7810</td><td>23.3</td><td>49.2</td><td>0.8121</td><td>22.2</td><td>47.5</td><td>0.8064</td><td>22.0</td><td>47.4</td><td>0.8042</td></tr>
  <tr><td>fa-en</td><td>31.1</td><td>56.4</td><td>0.8447</td><td>33.7</td><td>58.2</td><td>0.8585</td><td>32.7</td><td>57.6</td><td>0.8546</td></tr>
  <tr><td>fi-en</td><td>26.6</td><td>52.4</td><td>0.8442</td><td>26.6</td><td>51.9</td><td>0.8451</td><td>26.2</td><td>51.6</td><td>0.8421</td></tr>
  <tr><td>ga-en</td><td>29.9</td><td>54.9</td><td>0.7653</td><td>34.1</td><td>58.8</td><td>0.8006</td><td>32.3</td><td>56.3</td><td>0.7754</td><td>33.1</td><td>57.7</td><td>0.7918</td></tr>
  <tr><td>gl-en</td><td>31.4</td><td>57.2</td><td>0.8236</td><td>33.7</td><td>59.2</td><td>0.8374</td><td>34.5</td><td>59.1</td><td>0.8395</td><td>35.0</td><td>59.9</td><td>0.8441</td></tr>
  <tr><td>gu-en</td><td>28.5</td><td>54.6</td><td>0.8475</td><td>32.0</td><td>57.0</td><td>0.8646</td><td>33.0</td><td>57.6</td><td>0.8667</td></tr>
  <tr><td>he-en</td><td>38.2</td><td>62.2</td><td>0.8534</td><td>39.7</td><td>62.9</td><td>0.8622</td><td>40.4</td><td>63.6</td><td>0.8665</td></tr>
  <tr><td>hi-en</td><td>35.2</td><td>59.9</td><td>0.8741</td><td>34.7</td><td>59.5</td><td>0.8701</td><td>35.8</td><td>60.1</td><td>0.8738</td><td>36.9</td><td>61.0</td><td>0.8773</td></tr>
  <tr><td>hr-en</td><td>31.7</td><td>56.4</td><td>0.8389</td></tr>
  <tr><td>is-en</td><td>25.3</td><td>50.0</td><td>0.7815</td><td>29.0</td><td>53.4</td><td>0.8189</td><td>29.0</td><td>52.8</td><td>0.8163</td><td>28.7</td><td>52.8</td><td>0.8136</td></tr>
  <tr><td>ja-en</td><td>19.9</td><td>46.8</td><td>0.8255</td><td>24.6</td><td>52.5</td><td>0.8628</td><td>23.6</td><td>50.6</td><td>0.8533</td></tr>
  <tr><td>kk-en</td><td>27.0</td><td>53.4</td><td>0.8403</td><td>22.6</td><td>47.8</td><td>0.8003</td><td>22.6</td><td>47.7</td><td>0.7998</td></tr>
  <tr><td>kn-en</td><td>3.8</td><td>24.5</td><td>0.6246</td><td>27.9</td><td>53.4</td><td>0.8391</td><td>27.4</td><td>53.2</td><td>0.8396</td></tr>
  <tr><td>ko-en</td><td>24.1</td><td>51.3</td><td>0.8458</td><td>25.7</td><td>52.7</td><td>0.8586</td><td>25.8</td><td>52.7</td><td>0.8578</td></tr>
  <tr><td>lt-en</td><td>26.5</td><td>51.9</td><td>0.8138</td><td>27.2</td><td>52.6</td><td>0.8224</td><td>27.0</td><td>52.2</td><td>0.8201</td></tr>
  <tr><td>lv-en</td><td>29.3</td><td>56.0</td><td>0.8368</td><td>25.0</td><td>50.9</td><td>0.7862</td><td>26.6</td><td>53.3</td><td>0.8113</td></tr>
  <tr><td>mk-en</td><td>37.0</td><td>61.4</td><td>0.8522</td><td>38.2</td><td>62.0</td><td>0.8558</td><td>38.8</td><td>62.6</td><td>0.8596</td></tr>
  <tr><td>ml-en</td><td>2.9</td><td>23.5</td><td>0.5978</td><td>26.4</td><td>51.7</td><td>0.8342</td><td>26.4</td><td>51.9</td><td>0.8363</td></tr>
  <tr><td>mr-en</td><td>23.8</td><td>49.8</td><td>0.8063</td><td>26.1</td><td>51.9</td><td>0.8299</td><td>26.9</td><td>52.2</td><td>0.8320</td></tr>
  <tr><td>ms-en</td><td>37.2</td><td>61.3</td><td>0.8561</td><td>38.5</td><td>61.8</td><td>0.8579</td><td>38.0</td><td>61.7</td><td>0.8583</td></tr>
  <tr><td>mt-en</td><td>45.0</td><td>68.4</td><td>0.7777</td><td>47.1</td><td>68.5</td><td>0.7892</td><td>47.5</td><td>68.9</td><td>0.7895</td></tr>
  <tr><td>nb-en</td><td>37.3</td><td>61.0</td><td>0.8480</td></tr>
  <tr><td>ne-en</td><td>22.1</td><td>47.9</td><td>0.7929</td><td>26.6</td><td>52.0</td><td>0.8408</td><td>27.7</td><td>52.8</td><td>0.8436</td></tr>
  <tr><td>nn-en</td><td>32.9</td><td>55.8</td><td>0.7945</td></tr>
  <tr><td>si-en</td><td>3.0</td><td>24.2</td><td>0.5979</td><td>26.0</td><td>51.2</td><td>0.8381</td><td>26.9</td><td>51.9</td><td>0.8418</td></tr>
  <tr><td>sk-en</td><td>31.6</td><td>58.1</td><td>0.8456</td><td>32.7</td><td>58.6</td><td>0.8486</td><td>33.2</td><td>59.0</td><td>0.8535</td></tr>
  <tr><td>sl-en</td><td>29.2</td><td>55.0</td><td>0.8371</td><td>28.7</td><td>54.4</td><td>0.8345</td><td>29.7</td><td>55.6</td><td>0.8402</td></tr>
  <tr><td>sq-en</td><td>31.7</td><td>58.3</td><td>0.8468</td><td>32.1</td><td>58.6</td><td>0.8453</td><td>33.7</td><td>58.8</td><td>0.8448</td><td>34.8</td><td>59.8</td><td>0.8534</td></tr>
  <tr><td>sr-en</td><td>37.4</td><td>62.7</td><td>0.8544</td></tr>
  <tr><td>sw-en</td><td>27.2</td><td>51.0</td><td>0.7542</td><td>35.3</td><td>57.8</td><td>0.8086</td><td>34.3</td><td>56.3</td><td>0.7979</td><td>33.5</td><td>55.6</td><td>0.7932</td></tr>
  <tr><td>ta-en</td><td>24.0</td><td>49.6</td><td>0.8068</td><td>23.4</td><td>49.0</td><td>0.8139</td><td>24.3</td><td>49.5</td><td>0.8154</td></tr>
  <tr><td>te-en</td><td>30.2</td><td>55.3</td><td>0.8328</td><td>31.5</td><td>55.9</td><td>0.8438</td><td>31.9</td><td>56.4</td><td>0.8446</td></tr>
  <tr><td>th-en</td><td>24.9</td><td>52.3</td><td>0.8452</td><td>22.9</td><td>51.0</td><td>0.8381</td><td>23.7</td><td>51.7</td><td>0.8410</td></tr>
  <tr><td>tr-en</td><td>29.5</td><td>54.9</td><td>0.8392</td><td>32.2</td><td>57.3</td><td>0.8622</td><td>32.7</td><td>57.4</td><td>0.8602</td></tr>
  <tr><td>uk-en</td><td>33.1</td><td>58.7</td><td>0.8444</td><td>33.4</td><td>59.2</td><td>0.8470</td><td>33.9</td><td>59.6</td><td>0.8478</td></tr>
  <tr><td>ur-en</td><td>26.3</td><td>52.1</td><td>0.8138</td><td>26.2</td><td>50.9</td><td>0.8097</td><td>27.4</td><td>52.0</td><td>0.8144</td></tr>
  <tr><td>uz-en</td><td>24.8</td><td>51.5</td><td>0.8110</td><td>23.6</td><td>48.6</td><td>0.7990</td><td>24.8</td><td>50.0</td><td>0.8064</td></tr>
  <tr><td>vi-en</td><td>32.0</td><td>56.4</td><td>0.8514</td><td>33.5</td><td>57.9</td><td>0.8602</td><td>32.9</td><td>57.2</td><td>0.8543</td></tr>
  <tr><td>xh-en</td><td>22.1</td><td>45.5</td><td>0.6703</td><td>0.1</td><td>11.8</td><td>0.2871</td><td>0.2</td><td>13.1</td><td>0.2628</td></tr>
</table>

Table 17: MT results (BLEU, chrF++, COMET22-DA) for models translating into English, trained on our HPLT v2, Tatoeba (OPUS), a combination of both, and the existing HPLT v1.2 (numbers reported where available).