# Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures

Pedro Javier Ortiz Suárez, Benoît Sagot, Laurent Romary

► To cite this version:

Pedro Javier Ortiz Suárez, Benoît Sagot, Laurent Romary. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Jul 2019, Cardiff, United Kingdom. 10.14618/IDS-PUB-9021. hal-02148693

HAL Id: hal-02148693

https://inria.hal.science/hal-02148693

Submitted on 13 Jul 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Asynchronous Pipeline for Processing Huge Corpora
on Medium to Low Resource Infrastructures

Pedro Javier Ortiz Suárez¹,²

Benoît Sagot¹

Laurent Romary¹

¹Inria, Paris, France

²Sorbonne Université, Paris, France

{pedro.ortiz, benoit.sagot, laurent.romary}@inria.fr

**Abstract**

Common Crawl is a considerably large, hetergeneous multilingual corpus comprised of crawled documents from the internet, surpassing 20TB of data and distributed as a set of more than 50 thousand plain text files where each contains many documents written in a wide variety of languages. Even though each document has a metadata block associated to it, this data lacks any information about the language in which each document is written, making it extremely difficult to use Common Crawl for monolingual applications. We propose a general, highly parallel, multithreaded pipeline to clean and classify Common Crawl by language; we specifically design it so that it runs efficiently on medium to low resource infrastructures where I/O speeds are the main constraint. We develop the pipeline so that it can be easily reapplied to any kind of heterogeneous corpus and so that it can be parameterised to a wide range of infrastructures. We also distribute a 6.3TB version of Common Crawl, filtered, classified by language, shuffled at line level in order to avoid copyright issues, and ready to be used for NLP applications.

# 1 Introduction

In recent years neural methods for Natural Language Processing (NLP) have consistently and repeatedly improved the state-of-the-art in a wide variety of NLP tasks such as parsing, PoS-tagging, named entity recognition, machine translation, text classification and reading comprehension among others. Probably the main contributing factor in this steady improvement for NLP models is the raise in usage of *transfer learning* techniques in the field. These methods normally consist of taking a pre-trained model and reusing it, with little to no retraining, to solve a different task from the original one it was intended to solve;

in other words, one *transfers* the *knowledge* from one task to another.

Most of the transfer learning done in NLP nowadays is done in an unsupervised manner, that is, it normally consist of a *language model* that is fed unannotated plain text in a particular language; so that it *extracts* or *learns* the basic *features* and patterns of the given language, the model is subsequently used on top of an specialised architecture designed to tackle a particular NLP task. Probably the best known example of this type of model are *word embeddings* which consist of real-valued vector representations that are trained for each word on a given corpus. Some notorious examples of word embeddings are word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014) and fastText (Mikolov et al., 2018). All these models are *context-free*, meaning that a given word has one single vector representation that is independent of context, thus for a polysemous word like Washington, one would have one single representation that is reused for the city, the state and the US president.

In order to overcome the problem of polysemy, *contextual* models have recently appeared. Most notably ELMo (Peters et al., 2018) which produces deep contextualised word representations out of the internal states of a deep bidirectional language model in order to model word use and how the usage varies across linguistic contexts. ELMo still needs to be used alongside a specialised architecture for each given downstream task, but newer architectures that can be fine-tuned have also appear. For these, the model is first fed unannotated data, and is then fine-tuned with annotated data to a particular downstream task without relying on any other architecture. The most remarkable examples of this type of model are GPT-1, GPT-2 (Radford et al., 2018, 2019), BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019);
the latter being the current state-of-the-art for mul-
tiple downstream tasks. All of these models are
different arrangements of the Transformer archi-
tecture (Vaswani et al., 2017) trained with differ-
ent datasets, except for XLNet which is an in-
stance of the Transformer-XL (Dai et al., 2019).

Even though these models have clear advan-
tages, their main drawback is the amount of data
that is needed to train them in order to obtain a
functional and efficient model. For the first En-
glish version of word2vec, Mikolov et al. (2013)
used a one billion word dataset consisting of vari-
ous news articles. Later Al-Rfou et al. (2013) and
then Bojanowski et al. (2017) used the plain text
from Wikipedia to train distributions of word2vec
and fastText respectively, for languages other than
English. Now, the problem of obtaining large
quantities of data aggravates even more for con-
textual models, as they normally need multiple
instances of a given word in order to capture all
its different uses and in order to avoid overfitting
due to the large quantity of hyperparameters that
these models have. Peters et al. (2018) for exam-
ple use a 5.5 billion token<fnref n="1" /> dataset comprised of
crawled news articles plus the English Wikipedia
in order to train ELMo, Devlin et al. (2018) use a
3.3 billion word<fnref n="2" /> corpus made by merging the En-
glish Wikipedia with the BooksCorpus (Zhu et al.,
2015), and Radford et al. (2019) use a 40GB En-
glish corpus created by scraping outbound links
from Reddit.<fnref n="3" />

While Wikipedia is freely available, and mul-
tiple pipelines exist<fnref n="4" />,<fnref n="5" /> to extract plain text from
it, some of the bigger corpora mentioned above
are not made available by the authors either due
to copyright issues or probably because of the in-
frastructure needed to serve and distribute such big
corpora. Moreover the vast majority of both these
models and the corpora they are trained with are
in English, meaning that the availability of high
quality NLP for other languages, specially for low-
resource languages, is rather limited.

To address this problem, we choose Common
Crawl,<fnref n="6" /> which is a 20TB multilingual free to use
corpus composed of crawled websites from the

internet, and we propose a highly parallel multi-
threaded asynchronous pipeline that applies well-
known concurrency patterns, to clean and classify
by language the whole Common Crawl corpus to a
point where it is usable for Machine Learning and
in particular for neural NLP applications. We op-
timise the pipeline so that the process can be com-
pleted in a sensible amount of time even in infras-
tructures where Input/Output (I/O) speeds become
the main bottleneck.

Knowing that even running our pipeline will not
always be feasible, we also commit to publish-
ing our own version of a classified by language,
filtered and ready to use Common Crawl corpus
upon publication of this article. We will set up an
easy to use interface so that people can download
a manageable amount of data on a desired target
language.

## 2 Related Work

Common Crawl has already been successfully
used to train language models, even multilingual
ones. The most notable example in probably fast-
Text which was first trained for English using
Common Crawl (Mikolov et al., 2018) and then
for other 157 different languages (Grave et al.,
2018). In fact Grave et al. (2018) proposed a
pipeline to filter, clean and classify their fastText
multilingual word embeddings, which we shall
call the “fastText pre-processing pipeline.” They
used the fastText linear classifier (Joulin et al.,
2016, 2017) to classify each line of Common
Crawl by language, and downloaded the initial
corpus and schedule the I/O using some simple
Bash scripts. Their solution, however, proved to
be a synchronous blocking pipeline that works
well on infrastructures having the necessary hard-
ware to assure high I/O speeds even when storing
tens of terabytes of data at a time. But that down-
scales poorly to medium-low resource infrastruc-
tures that rely on more traditional cost-effective
electromechanical mediums in order to store this
amount of data.

Concerning contextual models, Baevski et al.
(2019) trained a BERT-like bi-directional Trans-
former for English using Common Crawl. They
followed the “fastText pre-processing pipeline”
but they removed all copies of Wikipedia inside
Common Crawl. They also trained their model
using News Crawl (Bojar et al., 2018) and using
Wikipedia + BooksCorpus, they compared three

<footnote n="1">¹Punctuation marks are counted as tokens.</footnote>

<footnote n="2">²Space separated tokens.</footnote>

<footnote n="3">³https://www.reddit.com/</footnote>

<footnote n="4">⁴https://github.com/attardi/wikiextractor</footnote>

<footnote n="5">⁵https://github.com/hghodrati/wikifil</footnote>

<footnote n="6">⁶http://commoncrawl.org/</footnote>
models and showed that Common Crawl gives the best performance out of the three corpora.

The XLNet model was trained for English by joining the BookCorpus, English Wikipedia, Giga5 (Parker et al., 2011), ClueWeb 2012-B (Callan et al., 2009) and Common Crawl. Particularly for Common Crawl, Yang et al. (2019) say they use “heuristics to aggressively filter out short or low-quality articles” from Common Crawl, however they don’t give any detail about these “heuristics” nor about the pipeline they use to classify and extract the English part of Common Crawl.

It is important to note that none of these projects distributed their classified, filtered and cleaned versions of Common Crawl, making it difficult in general to faithfully reproduce their results.

## 3 Common Crawl

Common Crawl is a non-profit foundation which produces and maintains an open repository of web crawled data that is both accessible and analysable.<fnref n="7" /> Common Crawl's complete web archive consists of petabytes of data collected over 8 years of web crawling. The repository contains raw web page HTML data (WARC files), metadata extracts (WAT files) and plain text extracts (WET files). The organisation's crawlers has always respected *nofollow*<fnref n="8" /> and *robots.txt*<fnref n="9" /> policies.

Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web pages written in a large variety of languages and covering all possible types of topics. Thus, in order to effectively use this corpus for the previously mentioned Natural Language Processing and Machine Learning applications, one has first to extract, filter, clean and classify the data in the snapshot by language.

For our purposes we use the WET files which contain the extracted plain texts from the websites mostly converted to UTF-8, as well as headers containing the metatada of each crawled document. Each WET file comes compressed in gzip format<fnref n="10" /> and is stored on Amazon Web Services.

We use the November 2018 snapshot which surpasses 20TB of uncompressed data and contains more than 50 thousand plain text files where each file consists of the plain text from multiple websites along its metadata header. From now on, when we mention the “Common Crawl” corpus, we refer to this particular November 2018 snapshot.

## 4 fastText's Pipeline

In order to download, extract, filter, clean and classify Common Crawl we base ourselves on the “fastText pre-processing pipeline” used by Grave et al. (2018). Their pipeline first launches multiple process, preferably as many as available cores. Each of these processes first downloads one Common Crawl WET file which then proceeds to decompress after the download is over. After decompressing, an instance of the fastText linear classifier (Joulin et al., 2016, 2017) is launched, the classifier processes each WET file line by line, generating a language tag for each line. The tags are then stored in a tag file which holds a one-to-one correspondence between lines of the WET file and its corresponding language tag. The WET file and the tag files are read sequentially and each on the WET file line holding the condition of being longer than 100 bytes is appended to a language file containing only plain text (tags are discarded). Finally the tag file and the WET files are deleted.

Only when one of these processes finishes another can be launched. This means that one can at most process and download as many files as cores the machine has. That is, if for example a machine has 24 cores, only 24 WET files can be downloaded and processed simultaneously, moreover, the 25th file won't be downloaded until one of the previous 24 files is completely processed.

When all the WET files are classified, one would normally get around 160 language files, each file holding just plain text written in its corresponding language. These files still need to be filtered in order to get rid of all files containing invalid UTF-8 characters, so again a number of processes are launched, this time depending on the amount of memory of the machine. Each process reads a language file, first filters for invalid UTF-8 characters and then performs deduplication. A simple non-collision resistant hashing algorithm is used to deduplicate the files.

The fastText linear classifier works by repre-

<footnote n="7">⁷http://commoncrawl.org/about/</footnote>

<footnote n="8">⁸http://microformats.org/wiki/rel-nofollow</footnote>

<footnote n="9">⁹https://www.robotstxt.org/</footnote>

<footnote n="10">¹⁰https://www.gnu.org/software/gzip/</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: A scheme of the goclassy pipeline. The red square represents the Compressed WET files stored on Amazon Web Services. The 📁 icons represent the gzip files stored locally, the 📁 represent one of the 50K WET files. The 📁 represents the filtered file and the 🔖 represents a file of language tags, one tag per line in 📁 . The 🔖 represents one of the 166 classified files. Each arrow represents an asynchronous non blocking worker and dotted arrows represent a line filtering process.</figcaption></figure>

sending sentences for classification as Bags of Words (BoW) and training a linear classifier. A weight matrix A is used as a look-up table over the words and the word representations are then averaged into a text representation which is fed to the linear classifier. The architecture is in general similar to the CBoW model of Mikolov et al. (2013) but the middle word is replaced by a label. They uses a softmax function f to compute the probability distribution over the classes. For a set of N documents, the model is trained to minimise the negative log-likelihood over the classes:

$$-\frac{1}{N} \sum_{n=1}^{N} y_n \log (f(BAx_n)),$$

where $x_n$ is the normalised bag of features of the n-th document, $y_n$ is the n-th label, and A, B are the weight matrices. The pre-trained fast-Text model for language recognition (Grave et al., 2018) is capable of recognising around 176 different languages and was trained using 400 million tokens from Wikipedia as well as sentences from the Tatoeba website<fnref n="11" />.

# 5 Asynchronous pipeline

We propose a new pipeline derived from the fast-Text one which we call goclassy, we reuse the fastText linear classifier (Joulin et al., 2016, 2017) and the pre-trained fastText model for language recognition (Grave et al., 2018), but we

completely rewrite and parallelise their pipeline in an asynchronous manner.

The order of operations is more or less the same as in the fastText pre-processing pipeline but instead of clustering multiple operations into a single blocking process, we launch a worker for each operation and we bound the number of possible parallel operations at a given time by the number of available threads instead of the number of CPUs. We implement goclassy using the Go programming language<fnref n="12" /> so we let the Go runtime<fnref n="13" /> handle the scheduling of the processes. Thus in our pipeline we don't have to wait for a whole WET file to download, decompress and classify in order to start downloading and processing the next one, a new file will start downloading and processing as soon as the scheduler is able to allocate a new process.

When using electromechanical mediums of storage, I/O blocking is one of the main problems one encounters. To overcome this, we introduced buffers in all our I/O operations, a feature that is not present in the fastText pre-processing pipeline. We also create, from the start, a file for each of the 176 languages that the pre-trained fastText language classifier is capable of recognising, and we always leave them open, as we find that getting a file descriptor to each time we want to write, if we wanted leave them open just when needed, introduces a big overhead.

<footnote n="11">¹¹https://tatoeba.org/</footnote>

<footnote n="12">¹²https://golang.org/</footnote>

<footnote n="13">¹³https://golang.org/src/runtime/mprof.go</footnote>
<table><caption>Table 1: Benchmarks are done using the UNIX time tool, are repeated 10 times each and are done for random samples of 10, 100 and 200 WET files. Only the classifying and filtering part are benchmarked. The table shows the minimum, maximum and mean time for the user, real and sys time over the 10 runs. Here “fastText” is used as short for the pipeline.</caption><thead><tr><th rowspan="2"></th><th colspan="3">10 files</th><th colspan="3">100 files</th><th colspan="3">200 files</th></tr><tr><th>Min</th><th>Max</th><th>Mean</th><th>Min</th><th>Max</th><th>Mean</th><th>Min</th><th>Max</th><th>Mean</th></tr></thead><tbody><tr><td>real</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>fastText</td><td>2m50s</td><td>6m45s</td><td>3m31s</td><td>13m46s</td><td>38m38s</td><td>17m39s</td><td>26m20s</td><td>47m48s</td><td>31m4s</td></tr><tr><td>goclassy</td><td>1m23s</td><td>3m12s</td><td>1m42s</td><td>7m42s</td><td>12m43s</td><td>9m8s</td><td>15m3s</td><td>15m47s</td><td>15m16s</td></tr><tr><td>user</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>fastText</td><td>26m45s</td><td>27m2s</td><td>26m53s</td><td>4h21m</td><td>4h24m</td><td>4h23m</td><td>8h42m</td><td>8h48m</td><td>8h45m</td></tr><tr><td>goclassy</td><td>10m26s</td><td>12m53s</td><td>11m0s</td><td>1h46m</td><td>1h54m</td><td>1h49m</td><td>3h37m</td><td>3h40m</td><td>3h38m</td></tr><tr><td>sys</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>fastText</td><td>40.14s</td><td>40.85s</td><td>40.56s</td><td>6m14s</td><td>6m17s</td><td>6m15s</td><td>12m26s</td><td>12m45s</td><td>12m31s</td></tr><tr><td>goclassy</td><td>37.34s</td><td>45.98s</td><td>39.67s</td><td>5m7s</td><td>5m34s</td><td>5m16s</td><td>9m57s</td><td>10m14s</td><td>10m5s</td></tr></tbody></table>

We also do the filtering and cleaning processes at line level before feeding each line to the classifier, which makes us create a new filtered file so that we can have a correspondence with the tag file, which in turn will consume more space, but that will also reduce the amount of unnecessary classifications performed by fastText. The filtered and file tags are then read and lines are appended to its corresponding language file. The writing in the classification step is asynchronous, meaning that process writing a line to the filtered files does not wait for the classifier to write a tag on the tag file. Figure 1 shows the pipeline up to this point.

After all WET files are processed, we then use Isaac Whitfield's deduplication tool runiq<fnref n="14" /> which is based on Yann Collet's xxhash64¹⁵, an extremely fast non-cryptographic hash algorithm that is resistant to collisions. We finally use the Mark Adler's pigz<fnref n="16" /> for data compression, as opposed to the canonical UNIX tools proposed in the original fastText pipeline. We add both tools to our concurrent pipeline, executing multiple instances of them in parallel, in order to ensure we use the most of our available resources at a given time.

Beyond improving the computational time required to classify this corpus, we propose a simple improvement on the cleaning scheme in the fast-Text pre-processing pipeline. This improvement allows our pipeline to better take into account the multilingual nature of Common Crawl; that is, we count UTF-8 characters instead of bytes for setting the lower admissible bound for the length of a line to be fed into the classifier. This straightforward

modification on the fastText pre-processing pipeline assures we take into account the multiple languages present in Common Crawl that use non-ASCII encoded characters.

Given that our implementation is written in Go, we release binary distributions<fnref n="17" /> of goclassy for all major operating systems. Both pigz and runiq are also available for all major operating systems.

## 6 Benchmarks

We test both pipelines against one another in an infrastructure using traditional electromechanical storage mediums that are connected to the main processing machine via an Ethernet interface, that is, a low I/O speed environment as compared to an infrastructure where one would have an array of SSDs connected directly to the main processing machine via a high speed interface. We use a machine with an Intel® Xeon® Processor E5-2650 2.00 GHz, 20M Cache, and 203.1 GiB of RAM. We make sure that no other processes apart from the benchmark and the Linux system processes are run. We do not include downloading, decompression or deduplication in our benchmarks as downloading takes far too much time, and deduplication and compression were performed with third party tools that don't make part of our main contribution. We are mainly interested in seeing how the way the data is fed to the classifier impacts the overall processing time.

Benchmarks in table 1 of our goclassy pipeline show a drastic reduction in processing time compared to the original fastText prepossessing pipeline. We show that in our particular infrastructure, we are capable of reducing the real time

<footnote n="14">¹⁴https://github.com/whitfin/runiq</footnote>

<footnote n="15">¹⁵https://github.com/Cyan4973/xxHash</footnote>

<footnote n="16">¹⁶https://zlib.net/pigz/</footnote>

<footnote n="17">¹⁷https://github.com/pjox/goclassy</footnote>
as measured by the *time* UNIX tool almost always by half. The *user* time which represents the amount of CPU time spent in user-mode code (outside the kernel) within the process is almost three times lower for our goclassy pipeline, this particular benchmark strongly suggest a substantial reduction in energy consumption of goclassy with respect to the fastText pipeline.

As we understand that even an infrastructure with more than 20TB of free space in traditional electromechanical storage is not available to everyone and we propose a simple parametrization in our pipeline that actively deletes already processed data and that only downloads and decompresses files when needed, thus ensuring that no more than 10TB of storage are used at a given time. We nevertheless note that delaying decompression increases the amount of computation time, which is a trade-off that some users might make as it might be more suitable for their available infrastructure.

## 7 OSCAR

Finally, we are aware that some users might not even have access to a big enough infrastructure to run our pipelines or just to store all the Common Crawl data. Moreover, even if previously used and cited in NLP and Machine Learning research, we note that there is currently no public distribution of Common Crawl that is filtered, classified by language and ready to use for Machine Learning or NLP applications. Thus we decide to publish a pre-processed version of the November 2018 copy of Common Crawl which is comprised of usable data in 166 different languages, we publish<fnref n="18" /> our version under the name OSCAR which is short for *Open Super-large Crawled ALMAnaCH*<fnref n="19" /> *coRpus*.

After processing all the data with goclassy, the size of the whole Common Crawl corpus is reduced to 6.3TB, but in spite of this considerable reduction, OSCAR still dwarfs all previous mentioned corpora having more 800 billion “words” or spaced separated tokens and noting that this in fact in an understatement of how big OSCAR is, as some of the largest languages within OSCAR such as Chinese and Japanese do not use spaces. The sizes in bytes for both the original and the deduplicated versions of OSCAR can be found in table 2. OSCAR is published under the *Creative Commons*

CC0 license (“no rights reserved”)<fnref n="20" />, so it is free to use for all applications.

## 8 Conclusions

We are sure that our work will greatly benefit researchers working on an either constrain infrastructure or a low budget setting. We are also confident, that by publishing a classified version of Common Crawl, we will substantially increase the amount of available public data for medium to low resource languages, thus improving and facilitating NLP research for them. Furthermore, as our pipeline speeds-up and simplifies the treatment of Common Crawl, we believe that our contribution can be further parallelised and adapted to treat multiple snapshots of Common Crawl opening the door to what would be otherwise costly diachronic studies of the use of a given language throughout the internet.

Finally, we note that both our proposed pipeline is data independent, which means that they can be reused to process, clean and classify any sort of big multilingual corpus that is available in plain text form and that is UTF-8 encoded; meaning that the impact of our work goes way beyond a single corpus.

## Acknowledgements

The authors are grateful to Inria Paris “rioc” computation cluster for providing resources and support, and for allowing us to store the complete copies of both the raw and the filtered Common Crawl versions on their infrastructure.

<footnote n="18">¹⁸https://team.inria.fr/almanach/oscar/</footnote>

<footnote n="19">¹⁹https://team.inria.fr/almanach/</footnote>

<footnote n="20">²⁰http://creativecommons.org/publicdomain/zero/1.0/</footnote>
<table><caption>Table 2: Size of the OSCAR corpus by language measured in bytes and number of words. Standard UNIX human-readable notation is used for the size in byte. We define “words” as spaced separated tokens, which gives a good estimate of the size of each corpus for languages using Latin or Cyrillic alphabets, but might give a misleading size for other languages such as Chinese or Japanese.</caption><thead><tr><th rowspan="2">Language</th><th colspan="2">Size</th><th colspan="2">Words</th><th rowspan="2">Language</th><th colspan="2">Size</th><th colspan="2">Words</th></tr><tr><th>Orig</th><th>Dedup</th><th>Orig</th><th>Dedup</th><th>Orig</th><th>Dedup</th><th>Orig</th><th>Dedup</th></tr></thead><tbody><tr><td>Afrikaans</td><td>241M</td><td>163M</td><td>43,482,801</td><td>29,533,437</td><td>Lower Sorbian</td><td>13K</td><td>7.1K</td><td>1,787</td><td>966</td></tr><tr><td>Albanian</td><td>2.3G</td><td>1.2G</td><td>374,196,110</td><td>186,856,699</td><td>Luxembourgish</td><td>29M</td><td>21M</td><td>4,403,577</td><td>3,087,650</td></tr><tr><td>Amharic</td><td>360M</td><td>206M</td><td>28,301,601</td><td>16,086,628</td><td>Macedonian</td><td>2.1G</td><td>1.2G</td><td>189,289,873</td><td>102,849,595</td></tr><tr><td>Arabic</td><td>82G</td><td>32G</td><td>8,117,162,828</td><td>3,171,221,354</td><td>Maitili</td><td>317K</td><td>11K</td><td>69,161</td><td>874</td></tr><tr><td>Aragones</td><td>1.3M</td><td>801K</td><td>52,896</td><td>45,669</td><td>Malagasy</td><td>21M</td><td>13M</td><td>3,068,360</td><td>1,872,044</td></tr><tr><td>Armenian</td><td>3.7G</td><td>1.5G</td><td>273,919,388</td><td>110,196,043</td><td>Malay</td><td>111M</td><td>42M</td><td>16,696,882</td><td>6,045,753</td></tr><tr><td>Assamese</td><td>113M</td><td>71M</td><td>6,956,663</td><td>4,366,570</td><td>Malayalam</td><td>4.9G</td><td>2.5G</td><td>189,534,472</td><td>95,892,551</td></tr><tr><td>Asturian</td><td>2.4M</td><td>2.0M</td><td>381,005</td><td>325,237</td><td>Maltese</td><td>24M</td><td>17M</td><td>2,995,654</td><td>2,163,358</td></tr><tr><td>Avaric</td><td>409K</td><td>324K</td><td>24,720</td><td>19,478</td><td>Marathi</td><td>2.7G</td><td>1.4G</td><td>162,609,404</td><td>82,130,803</td></tr><tr><td>Azerbaijani</td><td>2.8G</td><td>1.5G</td><td>322,641,710</td><td>167,742,296</td><td>Mazanderani</td><td>691K</td><td>602K</td><td>73,870</td><td>64,481</td></tr><tr><td>Bashkir</td><td>128M</td><td>90M</td><td>9,796,764</td><td>6,922,589</td><td>Minangkabau</td><td>608K</td><td>310K</td><td>5,682</td><td>4,825</td></tr><tr><td>Basque</td><td>848M</td><td>342M</td><td>120,456,652</td><td>45,359,710</td><td>Mingrelian</td><td>5.8M</td><td>4.4M</td><td>299,098</td><td>228,629</td></tr><tr><td>Bavarian</td><td>503</td><td>503</td><td>399</td><td>399</td><td>Mirandese</td><td>1.2K</td><td>1.1K</td><td>171</td><td>152</td></tr><tr><td>Belarusian</td><td>1.8G</td><td>1.1G</td><td>144,579,630</td><td>83,499,037</td><td>Modern Greek</td><td>62G</td><td>27G</td><td>5,479,180,137</td><td>2,412,419,435</td></tr><tr><td>Bengali</td><td>11G</td><td>5.8G</td><td>623,575,733</td><td>363,766,143</td><td>Mongolian</td><td>2.2G</td><td>838M</td><td>181,307,167</td><td>68,362,013</td></tr><tr><td>Bihari</td><td>110K</td><td>34K</td><td>8,848</td><td>2,875</td><td>Nahuatl languages</td><td>12K</td><td>11K</td><td>1,234</td><td>1,193</td></tr><tr><td>Bishnupriya</td><td>4.1M</td><td>1.7M</td><td>198,286</td><td>96,940</td><td>Neapolitan</td><td>17K</td><td>13K</td><td>5,282</td><td>4,147</td></tr><tr><td>Bosnian</td><td>447K</td><td>116K</td><td>106,448</td><td>20,485</td><td>Nepali</td><td>1.8G</td><td>1.2G</td><td>107,448,208</td><td>71,628,317</td></tr><tr><td>Breton</td><td>29M</td><td>16M</td><td>5,013,241</td><td>2,890,384</td><td>Newari</td><td>5.5M</td><td>4.1M</td><td>564,697</td><td>288,995</td></tr><tr><td>Bulgarian</td><td>32G</td><td>14G</td><td>2,947,648,106</td><td>1,268,114,977</td><td>Northern Frisian</td><td>4.4K</td><td>4.4K</td><td>1,516</td><td>1,516</td></tr><tr><td>Burmese</td><td>1.9G</td><td>1.1G</td><td>56,111,184</td><td>30,102,173</td><td>Northern Luri</td><td>76K</td><td>63K</td><td>8,022</td><td>6,740</td></tr><tr><td>Catalan</td><td>8.0G</td><td>4.3G</td><td>1,360,212,450</td><td>729,333,440</td><td>Norwegian</td><td>8.0G</td><td>4.7G</td><td>1,344,326,388</td><td>804,894,377</td></tr><tr><td>Cebuano</td><td>39M</td><td>24M</td><td>6,603,567</td><td>3,675,024</td><td>Norwegian Nynorsk</td><td>85M</td><td>54M</td><td>14,764,980</td><td>9,435,139</td></tr><tr><td>Central Bikol</td><td>885</td><td>885</td><td>312</td><td>312</td><td>Occitan</td><td>5.8M</td><td>3.7M</td><td>750,301</td><td>512,678</td></tr><tr><td>Central Khmer</td><td>1.1G</td><td>581M</td><td>20,690,610</td><td>10,082,245</td><td>Oriya</td><td>248M</td><td>188M</td><td>14,938,567</td><td>11,321,740</td></tr><tr><td>Central Kurdish</td><td>487M</td><td>226M</td><td>48,478,334</td><td>18,726,721</td><td>Ossetian</td><td>13M</td><td>11M</td><td>1,031,268</td><td>878,765</td></tr><tr><td>Chavacano</td><td>520</td><td>520</td><td>130</td><td>130</td><td>Pampanga</td><td>760</td><td>304</td><td>130</td><td>52</td></tr><tr><td>Chechen</td><td>8.3M</td><td>6.7M</td><td>711,051</td><td>568,146</td><td>Panjabi</td><td>763M</td><td>460M</td><td>61,847,806</td><td>37,555,835</td></tr><tr><td>Chinese</td><td>508G</td><td>249G</td><td>14,986,424,850</td><td>6,350,215,113</td><td>Persian</td><td>79G</td><td>38G</td><td>9,096,554,121</td><td>4,363,505,319</td></tr><tr><td>Chuvash</td><td>39M</td><td>26M</td><td>3,041,614</td><td>2,054,810</td><td>Piemontese</td><td>2.1M</td><td>1.9M</td><td>362,013</td><td>337,246</td></tr><tr><td>Cornish</td><td>44K</td><td>14K</td><td>8,329</td><td>2,704</td><td>Polish</td><td>109G</td><td>47G</td><td>15,277,255,137</td><td>6,708,709,674</td></tr><tr><td>Croatian</td><td>226M</td><td>110M</td><td>34,232,765</td><td>16,727,640</td><td>Portuguese</td><td>124G</td><td>64G</td><td>20,641,903,898</td><td>10,751,156,918</td></tr><tr><td>Czech</td><td>53G</td><td>24G</td><td>7,715,977,441</td><td>3,540,997,509</td><td>Pushto</td><td>361M</td><td>242M</td><td>46,559,441</td><td>31,347,348</td></tr><tr><td>Danish</td><td>16G</td><td>9.5G</td><td>2,637,463,889</td><td>1,620,091,317</td><td>Quechua</td><td>78K</td><td>67K</td><td>10,186</td><td>8,691</td></tr><tr><td>Dhiveshi</td><td>126M</td><td>79M</td><td>7,559,472</td><td>4,726,660</td><td>Romanian</td><td>25G</td><td>11G</td><td>3,984,317,058</td><td>1,741,794,069</td></tr><tr><td>Dimli</td><td>146</td><td>146</td><td>19</td><td>19</td><td>Romanish</td><td>7.4K</td><td>6.5K</td><td>1,093</td><td>960</td></tr><tr><td>Dutch</td><td>78G</td><td>39G</td><td>13,020,136,373</td><td>6,598,786,137</td><td>Russia Buriat</td><td>13K</td><td>11K</td><td>963</td><td>809</td></tr><tr><td>Eastern Mari</td><td>7.2M</td><td>6.0M</td><td>565,992</td><td>469,297</td><td>Russian</td><td>1.2T</td><td>568G</td><td>92,522,407,837</td><td>46,692,691,520</td></tr><tr><td>Egyptian Arabic</td><td>66M</td><td>33M</td><td>7,305,151</td><td>3,659,419</td><td>Sanskrit</td><td>93M</td><td>37M</td><td>4,331,569</td><td>1,713,930</td></tr><tr><td>Emilian-Romagnol</td><td>25K</td><td>24K</td><td>6,376</td><td>6,121</td><td>Scottish Gaelic</td><td>1.9M</td><td>1.3M</td><td>310,689</td><td>207,110</td></tr><tr><td>English</td><td>2.3T</td><td>1.2T</td><td>418,187,793,408</td><td>215,841,256,971</td><td>Serbian</td><td>3.9G</td><td>2.2G</td><td>364,395,411</td><td>207,561,168</td></tr><tr><td>Erzya</td><td>1.4K</td><td>1.2K</td><td>90</td><td>78</td><td>Serbo-Croatian</td><td>25M</td><td>5.8M</td><td>5,292,184</td><td>1,040,573</td></tr><tr><td>Esperanto</td><td>299M</td><td>228M</td><td>48,486,161</td><td>37,324,446</td><td>Sicilian</td><td>3.3K</td><td>2.8K</td><td>554</td><td>468</td></tr><tr><td>Estonian</td><td>4.8G</td><td>2.3G</td><td>643,163,730</td><td>309,931,463</td><td>Sindhi</td><td>347M</td><td>263G</td><td>43,530,158</td><td>33,028,015</td></tr><tr><td>Finnish</td><td>27G</td><td>13G</td><td>3,196,666,419</td><td>1,597,855,468</td><td>Sinhala</td><td>1.4G</td><td>802M</td><td>93,053,465</td><td>50,864,857</td></tr><tr><td>French</td><td>282G</td><td>138G</td><td>46,896,036,417</td><td>23,206,776,649</td><td>Slovak</td><td>9.1G</td><td>4.5G</td><td>1,322,247,763</td><td>656,346,179</td></tr><tr><td>Galician</td><td>620M</td><td>384M</td><td>102,011,291</td><td>63,600,602</td><td>Slovenian</td><td>2.5G</td><td>1.3G</td><td>387,399,700</td><td>193,926,684</td></tr><tr><td>Georgian</td><td>3.6G</td><td>1.9G</td><td>171,950,621</td><td>91,569,733</td><td>Somali</td><td>61K</td><td>16K</td><td>1,202</td><td>472</td></tr><tr><td>German</td><td>308G</td><td>145G</td><td>44,878,908,446</td><td>21,529,164,172</td><td>South Azerbaijani</td><td>27M</td><td>19M</td><td>2,175,054</td><td>1,528,709</td></tr><tr><td>Goan Konkani</td><td>2.2M</td><td>1.8M</td><td>124,277</td><td>102,306</td><td>Spanish</td><td>278G</td><td>149G</td><td>47,545,122,279</td><td>25,928,290,729</td></tr><tr><td>Guarani</td><td>36K</td><td>24K</td><td>7,382</td><td>4,680</td><td>Sundanese</td><td>211K</td><td>141K</td><td>30,321</td><td>20,278</td></tr><tr><td>Gujarati</td><td>1.1G</td><td>722M</td><td>72,045,701</td><td>50,023,432</td><td>Swahili</td><td>13M</td><td>8.1M</td><td>2,211,927</td><td>1,376,963</td></tr><tr><td>Haitian</td><td>3.9K</td><td>3.3K</td><td>1,014</td><td>832</td><td>Swedish</td><td>44G</td><td>25G</td><td>7,155,994,312</td><td>4,106,120,608</td></tr><tr><td>Hebrew</td><td>20G</td><td>9.8G</td><td>2,067,753,528</td><td>1,032,018,056</td><td>Tagalog</td><td>573M</td><td>407M</td><td>98,949,299</td><td>70,121,601</td></tr><tr><td>Hindi</td><td>17G</td><td>8.9G</td><td>1,372,234,782</td><td>745,774,934</td><td>Tajik</td><td>379M</td><td>249M</td><td>31,758,142</td><td>21,029,893</td></tr><tr><td>Hungarian</td><td>40G</td><td>18G</td><td>5,163,936,345</td><td>2,339,127,555</td><td>Tamil</td><td>9.3G</td><td>5.1G</td><td>420,537,132</td><td>226,013,330</td></tr><tr><td>Icelandic</td><td>1.5G</td><td>846M</td><td>219,900,094</td><td>129,818,331</td><td>Tatar</td><td>670M</td><td>305M</td><td>51,034,893</td><td>23,825,695</td></tr><tr><td>Ido</td><td>147K</td><td>130K</td><td>25,702</td><td>22,773</td><td>Telugu</td><td>2.5G</td><td>1.6G</td><td>123,711,517</td><td>79,094,167</td></tr><tr><td>Iloko</td><td>874K</td><td>636K</td><td>142,942</td><td>105,564</td><td>Thai</td><td>36G</td><td>16G</td><td>951,743,087</td><td>368,965,202</td></tr><tr><td>Indonesian</td><td>30G</td><td>16G</td><td>4,574,692,265</td><td>2,394,957,629</td><td>Tibetan</td><td>187M</td><td>138M</td><td>1,483,589</td><td>936,556</td></tr><tr><td>Interlingua</td><td>662K</td><td>360K</td><td>180,231</td><td>100,019</td><td>Tosk Albanian</td><td>5.0M</td><td>2.8M</td><td>841,750</td><td>459,001</td></tr><tr><td>Interlingue</td><td>24K</td><td>1.6K</td><td>5,352</td><td>602</td><td>Turkish</td><td>60G</td><td>27G</td><td>7,577,388,700</td><td>3,365,734,289</td></tr><tr><td>Irish</td><td>88M</td><td>60M</td><td>14,483,593</td><td>10,017,303</td><td>Turkmen</td><td>11M</td><td>6.8M</td><td>1,113,869</td><td>752,326</td></tr><tr><td>Italian</td><td>137G</td><td>69G</td><td>22,248,707,341</td><td>11,250,012,896</td><td>Tuvinian</td><td>12K</td><td>7.9K</td><td>759</td><td>540</td></tr><tr><td>Japanese</td><td>216G</td><td>106G</td><td>4,962,979,182</td><td>1,123,067,063</td><td>Uighur</td><td>122M</td><td>83M</td><td>8,657,141</td><td>5,852,225</td></tr><tr><td>Javanese</td><td>659K</td><td>583K</td><td>104,896</td><td>86,654</td><td>Ukrainian</td><td>53G</td><td>28G</td><td>4,204,381,276</td><td>2,252,380,351</td></tr><tr><td>Kalmyk</td><td>113K</td><td>112K</td><td>10,277</td><td>10,155</td><td>Upper Sorbian</td><td>4.2M</td><td>1.8M</td><td>545,351</td><td>236,867</td></tr><tr><td>Kannada</td><td>1.7G</td><td>1.1G</td><td>81,186,863</td><td>49,343,462</td><td>Urdu</td><td>2.7G</td><td>1.7G</td><td>331,817,982</td><td>218,030,228</td></tr><tr><td>Karachay-Balkar</td><td>2.6M</td><td>2.3M</td><td>185,436</td><td>166,496</td><td>Uzbek</td><td>21M</td><td>12M</td><td>2,450,256</td><td>1,381,644</td></tr><tr><td>Kazakh</td><td>2.7G</td><td>1.5G</td><td>191,126,469</td><td>108,388,743</td><td>Venetian</td><td>18K</td><td>17K</td><td>3,492</td><td>3,199</td></tr><tr><td>Kirghiz</td><td>600M</td><td>388M</td><td>44,194,823</td><td>28,982,620</td><td>Vietnamese</td><td>68G</td><td>32G</td><td>12,036,845,359</td><td>5,577,159,843</td></tr><tr><td>Komi</td><td>2.3M</td><td>1.2M</td><td>201,404</td><td>95,243</td><td>Volapk</td><td>2.0M</td><td>2.0M</td><td>321,121</td><td>318,568</td></tr><tr><td>Korean</td><td>24G</td><td>12G</td><td>2,368,765,142</td><td>1,120,375,149</td><td>Walloon</td><td>273K</td><td>203K</td><td>50,720</td><td>37,543</td></tr><tr><td>Kurdish</td><td>94M</td><td>60M</td><td>15,561,003</td><td>9,946,440</td><td>Waray</td><td>2.5M</td><td>2.2M</td><td>397,315</td><td>336,311</td></tr><tr><td>Lao</td><td>174M</td><td>114M</td><td>4,133,311</td><td>2,583,342</td><td>Welsh</td><td>213M</td><td>133M</td><td>37,422,441</td><td>23,574,673</td></tr><tr><td>Latin</td><td>26M</td><td>8.3M</td><td>4,122,201</td><td>1,328,038</td><td>Western Frisian</td><td>35M</td><td>26M</td><td>5,691,077</td><td>4,223,816</td></tr><tr><td>Latvian</td><td>4.0G</td><td>1.8G</td><td>520,761,977</td><td>236,428,905</td><td>Western Mari</td><td>1.2M</td><td>1.1M</td><td>93,338</td><td>87,780</td></tr><tr><td>Lezghian</td><td>3.3M</td><td>3.0M</td><td>247,646</td><td>224,871</td><td>Western Panjabi</td><td>12M</td><td>9.0M</td><td>1,426,986</td><td>1,111,112</td></tr><tr><td>Limburgan</td><td>29K</td><td>27K</td><td>4,730</td><td>4,283</td><td>Wu Chinese</td><td>109K</td><td>32K</td><td>11,189</td><td>4,333</td></tr><tr><td>Lithuanian</td><td>8.8G</td><td>3.9G</td><td>1,159,661,742</td><td>516,183,525</td><td>Yakut</td><td>42M</td><td>26M</td><td>2,547,623</td><td>1,789,174</td></tr><tr><td>Lojban</td><td>736K</td><td>678K</td><td>154,330</td><td>141,973</td><td>Yiddish</td><td>141M</td><td>84M</td><td>13,834,320</td><td>8,212,970</td></tr><tr><td>Lombard</td><td>443K</td><td>433K</td><td>75,229</td><td>73,665</td><td>Yoruba</td><td>55K</td><td>27K</td><td>8,906</td><td>3,518</td></tr><tr><td>Low German</td><td>18M</td><td>13M</td><td>2,906,347</td><td>2,146,417</td><td>Yue Chinese</td><td>3.7K</td><td>2.2K</td><td>186</td><td>128</td></tr><tr><td>Total</td><td>6.3T</td><td>3.2T</td><td>844,315,434,723</td><td>425,651,344,234</td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
## References

Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. *Polyglot: Distributed word representations for multilingual NLP*. In *Proceedings of the Seventeenth Conference on Computational Natural Language Learning*, pages 183–192, Sofia, Bulgaria. Association for Computational Linguistics.

Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. *Cloze-driven pretraining of self-attention networks*. *CoRR*, abs/1903.07785.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. *Enriching word vectors with subword information*. *Transactions of the Association for Computational Linguistics*, 5:135–146.

Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In *Proceedings of the Third Conference on Machine Translation: Shared Task Papers*, pages 272–303, Belgium, Brussels. Association for Computational Linguistics.

Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. 2009. *Clueweb09 data set*.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. *Transformer-xl: Attentive language models beyond a fixed-length context*. *CoRR*, abs/1901.02860.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. *arXiv e-prints*, page arXiv:1810.04805.

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. *Learning word vectors for 157 languages*. In *Proceedings of the 11th Language Resources and Evaluation Conference*, Miyazaki, Japan. European Language Resource Association.

Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomas Mikolov. 2016. *Fasttext.zip: Compressing text classification models*. *CoRR*, abs/1612.03651.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. *Bag of tricks for efficient text classification*. In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, pages 427–431, Valencia, Spain. Association for Computational Linguistics.

Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In *Proceedings of the Eleventh International*

Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. *Distributed representations of words and phrases and their compositionality*. In *Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS'13*, pages 3111–3119, USA, Curran Associates Inc.

Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition, linguistic data consortium. *Technical report, Technical Report. Linguistic Data Consortium*.

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. *Glove: Global vectors for word representation*. In *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. *Deep contextualized word representations*. In *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*, pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. *Improving language understanding by generative pre-training*. *OpenAI Blog*.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. *OpenAI Blog*, 1:8.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In *Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA*, pages 6000–6010.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. *Xlnet: Generalized autoregressive pretraining for language understanding*. *CoRR*, abs/1906.08237.

Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In *2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015*, pages 19–27. IEEE Computer Society.