SlowGuess/ACL_2020_OCR huggingface dataset:
Marta Bañón†, Pinzhen Chen‡, Barry Haddow‡, Kenneth Heafield‡, Hieu Hoang‡  
Miquel Esplà-Gomis*, Mikel Forcada*, Amir Kamran*, Faheem Kirefu‡  
Philipp Koehn§, Sergio Ortiz-Rojas†, Leopoldo Pla*, Gema Ramírez-Sánchez†  
Elsa Sarrias*, Marek Strelec‡, Brian Thompson§, William Waites‡, Dion Wiggins†  
Jaume Zaragoza†
†Promptsit, ‡University of Edinburgh, †University of Alicante
$^{\S}$ Johns Hopkins University,  $\spadesuit$ TAUS,  $\spadesuit$ Omniscien Technologies
We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.
Parallel corpora are essential for building high-quality machine translation systems and have found uses in many other natural language applications, such as learning paraphrases (Bannard and Callison-Burch, 2005; Hu et al., 2019) or cross-lingual projection of language tools (Yarowsky et al., 2001).
We report on work to create the largest publicly available parallel corpora by crawling hundreds of thousands of web sites, using open source tools. The processing pipeline consists of the steps: crawling, text extraction, document alignment, sentence alignment, and sentence pair filtering. We describe these steps in detail in Sections 4-8. For some of these steps we evaluate several methods empirically in terms of their impact on machine translation quality. We provide the data resources used in these evaluations as benchmarks for future research.
As part of these effort, several open source components have been developed. These are integrated into the open-source tool Bitextor, $^{1}$  a highly modular pipeline that allows harvesting parallel corpora from multilingual websites or from preexisting or historical web crawls such as the one available as part of the Internet Archive. $^{2}$
The execution of the pipeline has focused on official European Union languages, but also targeted Russian, Sinhala, Nepali, Tagalog, Swahili, and Somali. We show that the obtained parallel corpora improve state-of-the-art results on common benchmarks, such as the WMT Shared Task on News Translation.
While the idea of mining the web for parallel data has been already pursued in the 20th century (Resnik, 1999), the most serious efforts have been limited to large companies such as Google (Uszkoreit et al., 2010) and Microsoft (Rerrick et al., 2011), or targeted efforts on specific domains such as the Canadian Hansards and Europarl (Koehn, 2005). The book Bitext Alignment (Tiedemann, 2011) describes some of the challenges in greater detail.
2.1 Acquisition Efforts
Most publicly available parallel corpora are the result of targeted efforts to extract the translations from a specific source. The French-English Canadian Hansards<sup>3</sup> were used in the earliest work on statistical machine translation. A similar popular corpus is Europarl (Koehn, 2005), used throughout the WMT evaluation campaign.
Multi-lingual web sites are attractive targets. Rafalovitch and Dale (2009); Ziemski et al. (2015) extract data from the United Nations, Tager (2011) from European Patents, Lison and Tiedemann (2016) from a collection of TV and movie subtitles. Cettolo et al. (2012) explain the creation of a multilingual parallel corpus of subtitles from the TED Talks website which is popular due to its use in the IWSLT evaluation campaign.
<https://github.com/bitextor/bitextor>
$^{2}$ https://archive.org/
3https://www.isi.edu/natural-language/download/hansard/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555-4567 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
There are also various efforts targeted at a single language pair. Martin et al. (2003) build a parallel corpus for Inuktitut-English. Utiyama and Isahara (2003); Fukushima et al. (2006) worked on creating Japanese-English corpora. Uchiyama and Isahara (2007) report on the efforts to build a Japanese-English patent corpus and Macken et al. (2007) on efforts on a broad-based Dutch-English corpus. Li and Liu (2008) mine the web for a Chinese-English corpus. A large Czech-English corpus from various sources was collected (Bojar et al., 2010), linguistically annotated (Bojar et al., 2012), and has been continuously extended to over 300 million words (Bojar et al., 2016).
All these efforts rely on methods and implementations that are quite specific for each use case, not documented in great detail, and not publicly available. A discussion of the pitfalls during the construction of parallel corpora is given by Kaalep and Veskis (2007). A large collection of corpora is maintained at the OPUS web site $^{4}$  (Tiedemann, 2012).
2.2 Document Alignment
Document alignment can be defined as a matching task that takes a pair of documents and computes a score that reflects the likelihood that they are translations of each others. The task is typically limited to a single web domain (all web pages from www.aaa.com and aaa.com, possibly aaa.de but notbbb.com) for efficiency.
Matching may take the HTML structure into account, or purely rely on the textual content. Examples of structural matching is the use of edit-distance between linearized documents (Resnik and Smith, 2003) and probability of a probabilistic DOM-tree alignment model (Shi et al., 2006). Using the URL for matching is a very powerful indicator for some domains, typically by using a predefined set of patterns for language marking or simple Levenshtein distance (Le et al., 2016).
Content matching requires crossing the language barrier at some point, typically by using bilingual dictionaries or translating one of the documents into the other document's language (Uszkoreit et al., 2010).
Documents may be represented by vectors over word frequencies, typically tdidf-weighted. Vectors may also be constructed over bigrams (Dara and Lin, 2016) or even higher order n-grams
(Uszkoreit et al., 2010). The vectors are then typically matched with cosine similarity (Buck and Koehn, 2016a). The raw vectors may be recentered around the mean vector for a web domain (Germann, 2016)
Document alignment quality can be improved with additional features such ratio of shared links, similarity of link URLs, ratio of shared images, binary feature indicating if the documents are linked, DOM structure similarity (Esplà-Gomis et al., 2016), same numbers (Papavassiliou et al., 2016), or same named entities (Lohar et al., 2016).
Guo et al. (2019) introduce the use of document embeddings, constructed from sentence embeddings, to the document alignment task.
2.3 Sentence Alignment
Early sentence aligners (Brown et al., 1991; Gale and Church, 1993) use scoring functions based only on the number of words or characters in each sentence and alignment algorithms based on dynamic programming. Europarl, for example, used metadata to align paragraphs, typically consisting of 2-5 sentences, and using Gale and Church (1993)'s method to align sentences within corresponding paragraphs. Later work added lexical features and heuristics to speed up search, such as limiting the search space to be near the diagonal (Moore, 2002; Varga et al., 2005).
More recent work introduced scoring methods that use MT to get both documents into the same language (Sennrich and Volk, 2010) or use pruned phrase tables from a statistical MT system (Gomes and Lopes, 2016). Both methods "anchor" high-probability 1-1 alignments in the search space and then fill in and refine alignments. They later propose an extension (Sennrich and Volk, 2011) in which an SMT system is bootstrapped from an initial alignment and then used in Bleualign.
Vecalign (Thompson and Koehn, 2019) is a sentence alignment method that relies on bilingual sentence embeddings and achieves linear run time with a coarse-to-fine dynamic programming algorithm.
2.4 Sentence Pair Filtering
Parallel corpora that have been crawled from unverified web sites and processed by error-prone extraction and alignment methods are likely to contain noise, such as random text fragments, text in the wrong language, translations produced by machine translation tools or bad translators, and
<sup>4</sup>http://opus.lingfil.uu.se/
misaligned sentence pairs. Such noise is specially harmful for neural machine translation (Khayrallah and Koehn, 2018), so filtering it out is an essential processing step.
There is a robust body of work on filtering out noise in parallel data but most recently this topic has gained a lot of momentum, partly due to the lack of robustness of neural models and fostered by recent shared tasks on parallel corpus filtering under high-resource (Koehn et al., 2018) and low-resource data conditions (Koehn et al., 2019).
Most participants in these shared tasks used three components: pre-filtering rules, scoring functions for sentence pairs, and a classifier that learned weights for feature functions.
Pre-filtering rules. Some of the training data can be discarded based on simple deterministic filtering rules. This may remove over  $80\%$  of the data (Kurfalı and Östling, 2019; Soares and Costa-jussà, 2019). Such rules remove too short or too long sentences, sentences that have too few words (tokens with letters instead of just special characters), either absolute or relative to the total number of tokens, sentences whose average token length is too short or too long, sentence pairs with mismatched lengths in terms of number of tokens, sentence pairs where names, numbers, dates, email addresses, URLs do not match between both sides, sentence pairs that are too similar, indicating simple copying instead of translating, and sentences where language identifier do not detect the required language.
Scoring functions. Sentence pairs that pass the pre-filtering stage are assessed with scoring functions which provide scores that hopefully correlate with quality of sentence pairs. Participants used a variety of such scoring functions, including  $n$ -gram or neural language models on clean data (Rossenbach et al., 2018), language models trained on the provided raw data as contrast, neural translation models (Junczys-Dowmunt, 2018), bag-of-words lexical translation probabilities (González-Rubio, 2019), or even existing off-the-shelf tools like Zipporah and Bicleaner (Chaudhary et al., 2019).
Learning weights for scoring functions. Given a large number of scoring functions, simply averaging their resulting scores may be inadequate. Learning weights to optimize machine translation system quality is computationally intractable
due to the high cost of training these systems to evaluate different weight settings. A few participants used instead a classifier that learns how to distinguish between good and bad sentence pairs (where bad sentence pairs are either synthesized by scrambling good sentence pairs or selected from the raw crawled data).
A novel method that was central to the best-performing submission in WMT 2019 was the use of cross-lingual sentence embeddings that were directly trained from parallel sentence pairs (Chaudhary et al., 2019). Other submissions used monolingual word embeddings (Soares and Costajussa, 2019; Kurfalu and Ostling, 2019; Bernier-Colborne and Lo, 2019).
Another approach is to first train a translation system on the clean data, then use it to translate the non-English side into English and use monolingual matching methods to compare it against the English side of the parallel corpus. Different matching metrics were used: METEOR (Erdmann and G Winnup, 2019), Levenshtein distance (Sen et al., 2019), or BLEU (Parcheta et al., 2019),
As Rarrick et al. (2011) point out, one type of noise in parallel corpora extracted from the web are translations that have been created by machine translation. Venugopal et al. (2011) propose a method to watermark the output of machine translation systems to aid this distinction, with a negligible loss of quality. Antonova and Misyurev (2011) report that rule-based machine translation output can be detected due to certain word choices, and statistical machine translation output can be detected due to lack of reordering. Rarrick et al. (2011) train a classifier to learn the distinction and show that removing such data leads to better translation quality.
2.5 Comparable Corpus Mining
Our work exploits web sites that provide roughly the same content in multiple languages, leading us to the assumption to find pairs of web pages which are translations of each other, with translated sentences following the same order. This assumption does not hold in less consistently translated web content such as Wikipedia, or accidental parallel sentence found in news stories about the same subject matter written in multiple languages.
There have been increasing efforts to mine sentence pairs from large pools of multi-lingual text, which are treated as unstructured bags of sen
tences. Munteanu and Marcu (2005) use document retrieval and a maximum entropy classifier to identify parallel sentence pairs in a multi-lingual collection of news stories.
Bilingual sentence embeddings (Guo et al., 2018) and multilingual sentence embeddings (Artetxe and Schwenk, 2018) were tested on their ability to reconstruct parallel corpora. This lead to work to construct WikiMatrix, a large corpus of parallel sentences from Wikipedia (Schwenk et al., 2019) based on cosine distance of their cross-lingual sentence embeddings.
3 Identifying Multi-Lingual Web Sites
Since the start of the collection effort in 2015, we identified potential web sites to crawl in various ways, but mainly by exploiting statistics from CommonCrawl. By splitting this large collection of crawled web pages by web domain and running text extraction and language identification (Buck et al., 2014), we can extract statistics on what language content exists on each of them. Web domains with sufficient content in a targeted language and English are selected for crawling.
The thresholds of what constitutes sufficient content varied depending on language. Typically, we require minimum amounts of content in the targeted language and English (measured in bytes of text), and consider the ratio between the two. For instance, we identified 19,616 web domains with at least 100KB of content in German and English (max ratio 10), but only 438 web domains with at least 20KB of content in Maltese and English (max ratio 10).
It is worth noting that by targeted crawling of web sites we are able to collect many more web pages than present in CommonCrawl. In an exploratory study, only  $5\%$  of a collection of web pages with useful content were found in CommonCrawl. This may have improved with recent more extensive crawls by CommonCrawl but there is still a strong argument for targeted crawling.
4 Crawling
Crawling is the initial step of the pipeline. It entails downloading documents from a number of websites and looking for any documents that contain text. These documents are stored as single or multi-domain Web ARCHive (WARC) files. WARC is an archiving format for crawled data originally proposed by the Internet Archive
Figure 1: Workflow diagram of Bitextor
and developed by a consortium of libraries and archives into the ISO 28500:2009 standard (ISO, 2009). It consists of a list of gzip-compressed records, each comprising a header with metadata and a crawled document.
Four different crawling tools are currently supported in Bitextor:
HTTrack<sup>5</sup> Well-known multi-platform tool for crawling. It has been for long time in Bitextor, even though it is now deprecated as the support for the tool is discontinued.
Heritrix<sup>6</sup> Internet Archive's web crawler; it is fully compatible with WARC format and supports
5https://www.httrack.com/
$^{6}$ https://github.com/internetarchive/ heritrix3
a variety of options that make it one of the most suitable options for large scale data crawling.
Creepy<sup>7</sup> Python library with basic resources for crawling. A crawler has been implemented on top of it, and is currently experimental.
Wget One of the most popular tools for retrieving files through HTTP and HTTPS in Unix systems. It is fully compatible with WARC format.
Most of our crawling in ParaCrawl has been done using HTTrack. To deal with the I/O-intensive process of writing small files with high frequency, data is first stored on local SSD drives and then transferred to a network file system for subsequent processing.
5 Text Extraction
After crawling, all documents are pre-processed to extract and normalize the text and identify their language. The resulting cleaned and sorted text is the input for the subsequent steps of document and segment alignment (see Sections 6 and 7).
Conversion to HTML WARC files contain one web-crawled document per record. The documents can be in a variety of formats that contain text: plain text, HTML, Open Document Format $^8$  (".odt"), Office Open XML $^9$  (".docx") or PDF files containing text. With the exception of the small number of documents that are already in plain text format, the bitextor-warc2htmlwarc.py module converts any of these formats to HTML (see fig. 1) and produces WARC files containing only HTML or plain text documents.
Text extraction from HTML Given WARC files containing HTML, we extract the text content. We preserve sentence breaks indicated by HTML tags such as  $\langle \mathsf{p} \rangle$  or  $\langle \mathsf{br} \rangle$  (paragraph and line break), but remove formatting tags such as  $\langle \mathsf{b} \rangle$  (for bold text) without a trace.
Language identification with c1d2 and text extraction are currently performed by Python module bitextor-warc2preprocess.py; as text extraction is a rather intensive operation, an alternative workflow uses an experimental module written in the Go language, giawarc.
6 Document Alignment
There are two main workflows for document alignment.
Using bilingual lexica The traditional workflow in Bitextor until version 5 used bilingual lexica. Module bitextor-buildidx.py builds indexes of documents containing, for each word in the lexicon for each language, the documents containing it. Then bitextor-idx2ridx uses the bilingual lexica to translate these words and build reverse indexes where each document is paired to a list of documents and bag-of-words-based overlap scores in the other language. A series of modules (bitextor-urlscomparison.py, bitextor-)urlsetoverlap.py, bitextor-imagestooverlap.py, etc.), compute a series of features for each language direction based on mutual linking and the comparison of document URLs, the set of outgoing URLs, HTML structure and image content; these features are integrated by bitextor-rank.py into two new reverse-index file with new scores, which are used to obtain the final document alignment.
Using machine translation This workflow uses machine translation to decide whether two documents have to be aligned, and is the one that has been used for the parallel data releases of the project (Buck and Koehn, 2016b). After extract-lett.py extracts plain-text documents in each language, a machine translation system translates each document from language  $A$  to  $B$ . We then generate a (sparse) matrix of tfidf scores between machine translated versions of documents in language  $A$  and documents in language  $B$ . These scores are used by compute_MATCHes.py to compute a list of document pairs (score, source URL, target URL).
Document pairs are stored in a file in which each line contains the URLs of both documents and their plain-text content encoded in base64.
7 Sentence Alignment
During the ParaCrawl project, we made use of a few sentence alignment tools. In this paper, we compare their performance on five language pairs. The sentence aligners are:
Hunalign (Varga et al., 2005) is a widely used tool that relies on a bilingual dictionary that we
$^{7}$ https://github.com/aitjcize/creepy  
 $^{8}$ https://www.oasis-open.org/standards#opendocumentv1.2  
 $^{9}$ http://www.ecma-international.org/publications/standards/Ecma-376.htm
4559
Language	Web Domains	Document Pairs	English Tokens
German	21,806	17,109,018	10,788,923,009
Czech	12,179	6,661,650	4,089,806,440
Hungarian	5,560	2,770,432	1,504,698,348
Estonian	5,129	2,301,309	1,427,328,440
Maltese	933	303,198	134,232,546
generated from the Europarl corpus or other available parallel corpora.
Bleualign (Sennrich and Volk, 2010) aligns an English translation of the foreign sentences and the English sentences based on their similarity, as measured by a variant of the BLEU score. We implemented a faster version of Bleualign in  $\mathrm{C + + }$ .
Vecalign (Thompson and Koehn, 2019) is a new sentence aligner based on sentence embeddings, using an efficient coarse-to-fine algorithm with linear run time. We used pre-trained LASER embeddings<sup>10</sup> which cover all the languages of ParaCrawl, except for Irish.
We compared the quality of the sentence pairs extracted from document pairs for these tools. To our knowledge, this is the first evaluation of sentence aligners on large-scale real-world web-crawled data. We selected five languages, ranging from low resource (Maltese) over mid-resource (Estonian, Hungarian) to high-resource (Czech, German). We selected a subset of web domains, for details see Table 1.
The data is provided as document pairs from the usual upstream ParaCrawl processing. The text of web pages needs to be further split into sentences, and then aligned using the different sentence aligners. The resulting sentence pairs are deduplicated are assessed for quality using Bicleaner (more on sentence pair filtering in the next section).
Since different sentence aligners generate different amounts of data (for instance, Blealign filters quite aggressively for noise), we selected differently sized subsets of the data for evaluation by selecting the best sentence pairs according to Bicleaner quality scores. We built neural machine translation models on these subsets using
Table 1: Corpus statistics for data used in the sentence alignment evaluation. Number of English tokens is computed with the Unix command wc.
Language	Hunalign	Vecalign	Bleualign
German	35.1 (100m)	35.8 (150m)	35.0 (100m)
Czech	21.0 (50m)	21.2 (50m)	21.0 (50m)
Hungarian	16.5 (30m)	16.8 (30m)	16.6 (15m)
Estonian	21.8 (20m)	21.6 (20m)	21.4 (20m)
Maltese	33.5 (5m)	34.1 (7m)	30.3 (2m)
Table 2: BLEU scores for systems trained on corpora generated by different sentence aligners. Different subsets are selected based on Bicleaner scores, size of the subsets is given in number of million English tokens.
Fairseq and evaluated them on test sets drawn from the WMT news translation task (newstest2018 for German, Czech, Estonian; newstest2009 for Hungarian) and the EU Bookshop<sup>11</sup> corpus (Maltese).
See Table 2 for the BLEU scores and corpus sizes for the best-performing subsets for each sentence aligner and language. Vecalign gives the best results for 4 of the languages, and is slightly behind Hunalign for Estonian.
We published the document pairs to be aligned, as well as the testing environment<sup>12</sup> to promote the evaluation of novel sentence alignment methods.
8 Sentence Pair Filtering
Our processing pipeline is aimed at high recall at the cost of precision, thus creating large but very noisy corpora. So, as a last processing step, we aim to filter out sentence pairs that are not useful as training data for machine translation or any other purpose.
This is especially important since training on noisy corpora is a challenge for neural machine translation which motivated the organization of two shared tasks in 2018 and 2019, on the high resource language German-English and the low resource languages Sinhala and Nepali, respectively. Here, we extend this evaluation to European languages with medium-sized resources.
Building on the data sets generated by the sentence alignment evaluation of the previous section, we compared three sentence pair filtering methods used in the ParaCrawl effort: Zipporah (Xu and Koehn, 2017), Bicleaner (Sánchez-Cartagena et al., 2018), and LASER (Chaudhary et al., 2019).