SlowGuess/ACL_2020_OCR huggingface dataset:
We carried out the evaluation (see Table 3) in the same fashion, as in the previous section. Filtering by LASER scores gives the best results except for Maltese (for which the publicly available
<sup>11</sup>http://opus.nlpl.eu/EUbbookshop.php
12http://www.statmt.org/
paracrawl-benchmarks/
$^{10}$ https://engineering.fb.com/ai-research/laser-multilingual-sentence-embeddings/
Setup	Zipporah	Bicleaner	LASER
de, Hunalign	34.4 (100m)	35.1 (100m)	36.0 (100m)
de, Vecalign	34.6 (100m)	35.8 (100m)	36.3 (50m)
cs, Hunalign	19.1 (15m)	21.0 (50m)	22.2 (30m)
cs, Vecalign	21.4 (30m)	21.2 (50m)	22.2 (30m)
hu, Hunalign	16.2 (10m)	16.5 (30m)	17.2 (10m)
hu, Vecalign	16.9 (15m)	16.8 (30m)	17.2 (15m)
et, Hunalign	21.2 (15m)	21.8 (20m)	22.1 (15m)
et, Vecalign	21.3 (20m)	21.6 (20m)	22.9 (20m)
mt, Hunalign	32.8 (5m)	33.5 (7m)	32.6 (7m)
mt, Vecalign	33.8 (5m)	34.1 (5m)	30.2 (7m)
LASER model has not been trained). Moreover, in almost all settings, we achieve better results with Bicleaner than Zipporah.
9 Released Corpora
Overall, the ParaCrawl corpus release v5.0 contains a total of 223 million filtered $^{13}$ , unique sentence pairs from around 150k website domains and across 23 EU languages with English (see Table 5). However, the data release is highly imbalanced with  $73\%$  of sentence pairs comprising of just five languages: French, German, Spanish, Italian and Portuguese. The average (untokenised) English sentence length (over all languages) is 22.9 words, with some notable anomalies. For example, the low-resourced Irish-English pair (27.6 words) has over  $50\%$  of sentence pairs originating from the legal domain, where sentences are longer than usual. Furthermore, we noticed that filtered sentences which had been aligned using Hunalign were significantly shorter than those aligned by Bleualign (26.1 and 20.1 words respectively), although we are unsure of the exact reason for this discrepancy.
Our main motivation for creating the ParaCrawl corpus is to improve the quality of machine translation systems. To test this, we trained neural machine translation models where we added the corpus to existing data sets for language pairs that were tackled in the shared task on news translation at the Conference on Machine Translation (WMT) — which we consider a strong baseline.
Table 3: BLEU scores for systems trained on subsets of the data selected by different sentence pair filtering methods. The size of the subsets in millions of English words is also reported.
Pair	BLEU14WMT	BLEUWMT+ParaCrawl-5
en-cs	19.0	(52m)	19.8	(52m+5.3m)
cs-en	25.0	(52m)	25.7	(52m+5.3m)
en-de	26.2	(5.8m)	27.7	(5.8m+37m)
de-en	31.2	(5.8m)	34.0	(5.8m+37m)
en-fi	19.9	(2.6m)	23.3	(2.6m+3.0m)
fi-en	24.2	(2.6m)	29.9	(2.6m+3.0m)
en-lv	12.8	(4.5m)	16.2	(4.5m+1.0m)
lv-en	16.2	(4.5m)	20.2	(4.5m+1.0m)
en-ro	26.5	(0.6m)	28.6	(0.6m+2.8m)
ro-en	30.2	(0.6m)	35.7	(0.6m+2.8m)
Table 4: BLEU scores for machine translation systems trained with WMT data adding ParaCrawl release v5.0 data. All the training and test sets are from WMT17 except for Romanian, taken from WMT16. The systems are transformer base trained with Marian using SentencePiece. Sentences are reported in millions.
We trained Transformer-Base models with Martian using SentencePiece. See Table 4 for results. For most language pairs, we see gains of several BLEU points (up to 6 BLEU points for English-Romanian). We even see gains for English-Czech, were ParaCrawl is quite a bit smaller than existing data sets (+0.7 BLEU when adding 5.3m sentence pairs to the existing set of 52m sentence pairs).
10 Computational Costs Concerns
Several of the steps involved in producing and evaluating the ParaCrawl corpora are computationally expensive. Even as some of the steps are embarrassingly parallel and amenable processing in a high-performance computing setting, even pre-processing of 100TB of source data to produce candidate documents consumes on the order of 50,000 CPU-hours equivalent to an estimated $^{15}$  720kWh of power. Training of a neural network model for translating one of the more resource-rich languages such as German may take a week on a dozen GPUs again consuming about 750kWh. Translating 500 million German sentences to English for evaluation consumed roughly 7MWh. In practice, these computations are not simply performed once, they are performed many times as parameters are changed and different strategies tried.
This energy cost is significant. The Typical Domestic Consumption Values published by
<sup>13</sup>Sentence pairs with a Bicleaner score of less than 0.7 were discarded, but remain in the RAW release.
14 sacreBLEU signatures:
BLEU+case.mixed+lang.\*-+numrefs.1+smooth.exp+ tok.13a+version.1.4.2
[15] The datasheet of an Intel E5-2695 processor says that it uses 115W of power or about 9.5W/core. This estimate includes a  $50\%$  margin for main board power and other overhead.
Language Pair	Web domains	Raw Corpus	Clean Corpus
Sentence Pairs	English Words	Sentence Pairs	English Words
Bulgarian-English	4,762	248,555,951	1,564,051,100	2,586,277	55,725,444
Croatian-English	8,889	273,330,006	1,738,164,401	1,861,590	43,464,197
Czech-English	14,335	665,535,115	4,025,512,842	5,280,149	117,385,158
Danish-English	19,776	447,743,455	3,347,135,236	4,606,183	106,565,546
Dutch-English	17,887	1,101,087,006	6,792,400,704	10,596,717	233,087,345
Estonian-English	9,522	168,091,382	915,074,587	1,387,869	30,858,140
Finnish-English	11,028	460,181,215	2,731,068,033	3,097,223	66,385,933
French-English	48,498	4,273,819,421	24,983,683,983	51,316,168	1,178,317,233
German-English	67,977	5,038,103,659	27,994,213,177	36,936,714	929,818,868
Greek-English	11,343	640,502,801	3,768,712,672	3,830,643	88,669,279
Hungarian-English	9,522	461,181,772	3,208,285,083	4,187,051	104,292,635
Irish-English	1,283	64,628,733	667,211,260	782,769	21,909,039
Italian-English	31,518	2,251,771,798	13,150,606,108	22,100,078	533,512,632
Latvian-English	3,557	176,113,669	1,069,218,155	1,019,003	23,656,140
Lithuanian-English	4,678	198,101,611	963,384,230	1,270,933	27,214,054
Maltese-English	672	3,693,930	38,492,028	177,244	4,252,814
Polish-English	13,357	723,052,912	4,123,972,411	6,382,371	145,802,939
Portuguese-English	18,887	1,068,161,866	6,537,298,891	13,860,663	299,634,135
Romanian-English	9,335	510,209,923	3,034,045,929	2,870,687	62,189,306
Slovak-English	7,980	269,067,288	1,416,750,646	2,365,339	45,636,383
Slovenian-English	5,016	175,682,959	1,003,867,134	1,406,645	31,855,427
Spanish-English	36,211	2,674,900,280	16,598,620,402	38,971,348	897,891,704
Swedish-English	13,616	620,338,561	3,496,650,816	6,079,175	138,264,978
Russian-English	14,035	1,078,819,759	-	12,061,155	157,061,045
Dutch-French	7,700	38,164,560	Dutch: 770,141,393 French: 817,973,481	2,687,331	Dutch: 60,504,313 French: 64,650,034
Polish-German	5,549	11,060,105	Polish: 202,765,359 German: 198,442,547	916,522	Polish: 18,883,576 German: 20,271,637
Table 5: Size of corpus release 5. The corpus is released in two versions: Raw is very noisy data before the sentence pair filtering step. Clean has been proven to be useful for training machine translation systems. We release the raw corpus to allow use of other filtering methods, or different thresholds for quality cutoffs.
$\mathrm{Ofgem}^{16}$ , the UK energy regulator, say that a high-consuming household with electric heating is expected to consume 7.1MWh/year. Does an increase of one or two BLEU points justify this cost? For ParaCrawl, we argue that yes, it does, because we are producing an enabling data set whose cost will, we hope, be amortised across many future experiments.
But there is a more general point to be made here: it is not currently the practice in the machine translation community to publish figures about the cost involved in achieving an increase in performance as measured with the standard metrics. It is not straightforward to evaluate when or if we, as a community, have reached a point of diminishing returns where small changes to a family of methods consume an ever-increasing amount of resources yielding only marginal improvements. We therefore suggest adopting a practice of disclosing energy use for experiments in machine translation alongside BLEU scores to make the
cost-benefit trade-off explicit.
11 Conclusions
We released the largest publicly available parallel corpora for many language pairs and demonstrated their benefit to train machine translation systems. Going beyond providing data, the goals of this project include the creation of publicly available infrastructure to explore new research directions on parallel corpus mining by releasing open source code for the entire pipeline and public benchmarks for individual processing steps.
Each of the processing steps we describe here still have great potential for improvement, and we hope that our work contributes to the development of novel methods both in terms of better processing of raw parallel data sources, but also increasing the robustness of neural machine translation training when faced with noisy data.
We are especially interested in further extending this work into low resource languages where resources tend to be noisier and underlying models to support data mining less reliable.
<sup>16</sup>https://www.ofgem.gov.uk/electricity/ retail-market/monitoring-data-and-statistics/ typical-domestic-consumption-values
This work has been supported in part by three projects funded by the Con
necting Europe Facility of the European Union (paracrawl.eu), two Google Faculty Research Awards to Philipp Koehn, a Mozilla research grant to Kenneth Heafield, and a donation from eBay to Kenneth Heafield. Hosting is provided by the AWS Public Dataset Program. This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (http://www.csd3.cam.ac.uk/), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/P020259/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk). This paper is the authors' opinion and not necessarily that of the funders.
Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 136-144, Portland, Oregon. Association for Computational Linguistics.
Mikel Artetxe and Holger Schwenk. 2018. Margin-based parallel corpus mining with multilingual sentence embeddings. CoRR, abs/1811.01136.
Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 597-604, Ann Arbor, Michigan. Association for Computational Linguistics.
Gabriel Bernier-Colborne and Chi-kiu Lo. 2019. NRC parallel corpus filtering system for WMT 2019. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Ondrej Bojar, Ondrej Dušek, Tom Kocmi, Jindrich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. 2016. CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered. In Text, Speech, and Dialogue: 19th International Conference, TSD 2016, number 9924 in Lecture Notes in Computer Science, pages 231–238, Cham / Heidelberg / New York / Dordrecht / London. Masaryk University, Springer International Publishing.
Ondrej Bojar, Adam Liska, and Zdenek Žabokrtský. 2010. Evaluating utility of data sources in a large parallel Czech-English corpus CzEng 0.9. In Proceedings of LREC2010.
Ondrej Bojar, Zdenek Žabokrtsky, Ondrej Dusek, Petra Galuscakova, Martin Majlis, David Marecek, Jiri Maršík, Michal Novák, Martin Popel, and Ales Tamchyna. 2012. The Joy of Parallelism with CzEng 1.0. In Proceedings of LREC2012, Istanbul, Turkey. ELRA, European Language Resources Association.
Peter F. Brown, Jennifer C. Lai, and Robert L. Mercer. 1991. Aligning sentences in parallel corpora. In Proceedings of the 29th Annual Meeting on Association for Computational Linguistics, ACL '91, pages 169-176, Stroudsburg, PA, USA. Association for Computational Linguistics.
Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from the common crawl. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavík, Iceland.
Christian Buck and Philipp Koehn. 2016a. Findings of the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pages 554-563, Berlin, Germany. Association for Computational Linguistics.
Christian Buck and Philipp Koehn. 2016b. Quick and reliable document alignment via tfidf-weighted cosine distance. In Proceedings of the First Conference on Machine Translation, pages 672-678, Berlin, Germany. Association for Computational Linguistics.
M. Cettolo, C. Girardi, and M. Federico. 2012. WIT3: Web inventory of transcribed and translated talks. In Proceedings of th 16th International Conference of the European Association for Machine Translation (EAMT), pages 261-268.
Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Aswarth Abhilash Dara and Yiu-Chang Lin. 2016. Yoda system for wmt16 shared task: Bilingual document alignment. In Proceedings of the First Conference on Machine Translation.
Grant Erdmann and Jeremy Gwinnup. 2019. Quality and coverage: The afl submission to the wmt19 parallel corpus filtering for low-resource conditions task. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Miquel Esplà-Gomis, Mikel Forcada, Sergio Ortiz Rojas, and Jorge Ferrández-Tordera. 2016. Bitextor's participation in wmt'16: shared task on document alignment. In Proceedings of the First Conference on Machine Translation, pages 685-691, Berlin,
Germany. Association for Computational Linguistics.
Ken'ichi Fukushima, Kenjiro Taura, and Takashi Chikayama. 2006. A fast and accurate method for detecting English-Japanese parallel texts. In Proceedings of the Workshop on Multilingual Language Resources and Interoperability, pages 60-67, Sydney, Australia. Association for Computational Linguistics.
William A Gale and Kenneth W Church. 1993. A program for aligning sentences in bilingual corpora. Computational linguistics, 19(1):75-102.
Ulrich Germann. 2016. Bilingual document alignment with latent semantic indexing. In Proceedings of the First Conference on Machine Translation, pages 692-696, Berlin, Germany. Association for Computational Linguistics.
Luís Gomes and Gabriel Pereira Lopes. 2016. First steps towards coverage-based sentence alignment. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 2228-2231, Portož, Slovenia. European Language Resources Association (ELRA).
Jesús González-Rubio. 2019. Webinterpret submission to the wmt2019 shared task on parallel corpus filtering. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernand ez Abrego, Keith Stevens, Noah Constant, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Effective parallel corpus mining using bilingual sentence embeddings. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 165-176, Belgium, Brussels. Association for Computational Linguistics.
Mandy Guo, Yinfei Yang, Keith Stevens, Daniel Cer, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Hierarchical document encoder for parallel corpus mining. In Proceedings of the Fourth Conference on Machine Translation, pages 64-72, Florence, Italy. Association for Computational Linguistics.
J. Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 2019. Parabank: Monolingual bitext generation and sentential paraphrasing via lexically-constrained neural machine translation. In Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19).
International Organization for Standardization ISO. 2009. ISO 28500:2009 information and documentation-WARC file format.
Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics.
Heiki-Jaan Kaalep and Kaarel Veskis. 2007. Comparing parallel corpora and evaluating their quality. In Proceedings of the MT Summit XI.
Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74-83. Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X), Phuket, Thailand.
Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019. Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 54-72, Florence, Italy. Association for Computational Linguistics.
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726-739, Belgium, Brussels. Association for Computational Linguistics.
Murathan Kurfali and Robert Östling. 2019. Noisy parallel corpus filtering through projected word embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Thanh Le, Hoa Trong Vu, Jonathan Oberlander, and Ondrej Bojar. 2016. Using term position similarity and language modeling for bilingual document alignment. In Proceedings of the First Conference on Machine Translation, pages 710-716, Berlin, Germany. Association for Computational Linguistics.
Bo Li and Juan Liu. 2008. Mining Chinese-English parallel corpora from the web. In Proceedings of the 3rd International Joint Conference on Natural Language Processing (IJCNLP).
Pierre Lison and Jörg Tiedemann. 2016. Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Language Resources and Evaluation Conference (LREC).
Pintu Lohar, Haithem Afli, Chao-Hong Liu, and Andy Way. 2016. The adapt bilingual document alignment system at wmt16. In Proceedings of the First Conference on Machine Translation, pages 717-723, Berlin, Germany. Association for Computational Linguistics.
Lieve Macken, Julia Trushkina, and Lidia Rura. 2007. Dutch parallel corpus: MT corpus and translator's aid. In Proceedings of the MT Summit XI.
Joel Martin, Howard Johnson, Benoit Farley, and Anna Maclachlan. 2003. Aligning and using an English-Inuktitut parallel corpus. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, Edmonton, Alberta, Canada. Association for Computational Linguistics.
Robert C Moore. 2002. Fast and accurate sentence alignment of bilingual corpora. In Conference of the Association for Machine Translation in the Americas, pages 135-144. Springer.
Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4).
Vassilis Papavassiliou, Prokopis Prokopidis, and Stelios Piperidis. 2016. The ilsp/arc submission to the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pages 733-739, Berlin, Germany. Association for Computational Linguistics.
Zuzanna Parcheta, Germán Sanchis-Trilles, and Francisco Casacuberta. 2019. Filtering of noisy parallel corpora based on hypothesis generation. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Alexandre Rafalovitch and Robert Dale. 2009. United Nations General Assembly resolutions: A six-language parallel corpus. In Proceedings of the Twelfth Machine Translation Summit (MT Summit XII). International Association for Machine Translation.
Spencer Rarrick, Chris Quirk, and Will Lewis. 2011. MT detection in web-scraped parallel corpora. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII), pages 422-430. International Association for Machine Translation.
Philip Resnik. 1999. Mining the web for bilingual text. In Proceedings of the 37th Annual Meeting of the Association of Computational Linguistics (ACL).
Philip Resnik and Noah A Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349-380.
Nick Rossenbach, Jan Rosendahl, Yunsu Kim, Miguel Graça, Aman Gokrani, and Hermann Ney. 2018. The RWTH Aachen University filtering system for the WMT 2018 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics.
VICTOR M. Sánchez-Cartagena, Marta Banón, Sergio Ortiz Rojas, and Gema Ramírez. 2018. Promptsit's submission to wmt 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 955-962, Belgium, Brussels. Association for Computational Linguistics.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. CoRR, abs/1907.05791.
Sukanta Sen, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Parallel corpus filtering based on fuzzy string matching. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Rico Sennrich and Martin Volk. 2010. MT-based sentence alignment for OCR-generated parallel texts. In *The Ninth Conference of the Association for Machine Translation in the Americas* (AMTA 2010).
Rico Sennrich and Martin Volk. 2011. Iterative, MT-based sentence alignment of parallel texts. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011), pages 175-182.
Lei Shi, Cheng Niu, Ming Zhou, and Jianfeng Gao. 2006. A dom tree alignment model for mining parallel data from the web. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 489-496. Association for Computational Linguistics.
Felipe Soares and Marta R. Costa-jussà. 2019. Unsupervised corpus filtering and mining. In Proceedings of the Fourth Conference on Machine Translation (WMT).
Wolfgang Täger. 2011. The sentence-aligned european patent corpus. In Proceedings of the 15th International Conference of the European Association for Machine Translation (EAMT), pages 177-184.
Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Hong Kong. Association for Computational Linguistics.
Jörg Tiedemann. 2011. Bitext Alignment. Synthesis Lectures on Human Language Technologies. Morgan & Claypool, San Rafael, CA.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA). ACL Anthology Identifier: L12-1246.
Masao Uchiyama and Hitoshi Isahara. 2007. A Japanese-English patent parallel corpus. In Proceedings of the MT Summit XI.
Jakob Uszkoreit, Jay Ponte, Ashok Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proceedings of the