WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
Authors: Holger Schwenk
Vishrav Chaudhary
Shuo Sun
Hongyu Gong
Francisco Guzmán
Venue:  (2020)
Keywords: multilinguality
bitext mining
neural MT
Wikipedia
low-resource languages
joint sentence representation

Abstract: We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 85 languages, including several dialects or low-resource languages.  We do not limit the extraction process to alignments with English, but systematically consider all possible language pairs.  In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English.  This corpus of parallel sentences is freely available (URL anonymized)
  
To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs.  The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.

Under review as a conference paper at ICLR 2020
WIKIMATRIX: MINING 135M PARALLEL SENTENCES IN 1620 LANGUAGE PAIRS FROM WIKIPEDIA
A Anonymous authors Paper under double-blind review
ABSTRACT
We present an approach based on multilingual sentence embeddings to automat- ically extract parallel sentences from the content of Wikipedia articles 85 lan- guages, including several dialects or low-resource languages. We do not limit the extraction process to alignments with English, but systematically consider all pos- sible language pairs. In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English. This corpus of parallel sentences is freely available.'
To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.
INTRODUCTION
Most of the current approaches in Natural Language Processing (NLP) are data-driven. The size of the resources used for training is often the primary concern, but the quality and large variety of topies may be equally important. Monolingual texts are usually available in huge amounts for many topics and languages. However, multilingual resources, typically sentences in two languages which are mutual translations, are more limited, in particular when the two languages do not involve English. An important source of parallel texts are international organizations like the European Parliament (Koehn, 2005) or the United Nations (Ziemski et al., 2016). These are professional human translations, but they are in a more formal language and tend be limited political topics. There are several projects relying on volunteers to provide translations for public texts, e.g. news commentary (Tiedemann, 2012). OpensubTitles (Lison & Tiedemann, 2016) or the TED corpus (Qi et al., 2018)
Wikipedia is probably the largest free multilingual resource on the Internet. The content of Wikipedia is very diverse and covers many topics. Articles exist in more than 300 languages. Some content on Wikipedia was human translated from an existing article into another language, not nec- essarily from or into English. Eventually, the translated articles have been later independently edited and are not parallel any more. Wikipedia strongly discourages the use of unedited machine transla- tion,? but the existence of such articles can not be totally excluded. Many articles have been written independently. but may nevertheless contain sentences which are mutual translations. This makes Wikipedia a very appropriate resource to mine for parallel texts for large number of language pairs. To the best of our knowledge, this is the first work to process the entire Wikipedia and systematically mine for parallel sentences in all language pairs. We hope that this resource will be useful for several research areas and enable the development of NLP applications for more languages.
In this work, build recent approach to mine parallel texts based a distance measure in joint multilingual sentence embedding space (Schwenk, 2018; Artetxe & Schwenk, 2018b). For this, the freely available LASER toolkit² which provides a language agnostic sentence encoder which was trained on 93 languages (Artetxe & Schwenk, 2018a). We approach the computational
Anooymiizdd for review
//enps://en..e soon.wwkiig wikipedia..ore. http://wkk//iiiiiiaaaaaaaaaaaaaa rransaaaaatiooooiiii 'https: //tp::////ii //gi thub. com/ faceboo LAS/ LASKR

---

Under review as a conference paper at ICLR 2020
challenge to mine in almost six hundred million sentences by using fast indexing and similarity search algorithms.
The paper is organized as follows. In the next section, first discuss related work. We then sum- marize the underlying mining approach. Section 4 describes in detail how we applied this approach to extract parallel sentences from Wikipedia in 1620 language pairs. To asses the quality of the extracted bitexts, we train NMT systems for a subset of language pairs and evaluate them on the TED corpus (Qi al., 2018) for 45 languages. These results are presented in section 5. The paper concludes with a discussion of future research directions.
2 RELATED WORK
There is a large body of research on mining parallel sentences in collections of monolingual texts, usually named "comparable coprora". ". Initial approaches to bitext mining have relied on heavily en- gineered systems often based on metadata information, e.g. (Resnik, 1999; Resnik & Smith, 2003). More recent methods explore the textual content of the comparable documents. For instance, it was proposed to rely on cross-lingual document retrieval, e.g. (Utiyama & Isahara, 2003; Munteanu & Marcu, 2005) or machine translation, e.g. (Abdul-Rauf & Schwenk, 2009; Bouamor & Sajjad, 2018), typically to obtain an initial alignment that is then further filtered. In the shared task for bilingual document alignment (Buck& & Koehn, 2016), many participants used techniques based on n-gram or neural language models, neural translation models and bag-of-words lexical translation probabilities for scoring candidate document pairs. The STACC method uses seed lexical transla- tions induced from IBM alignments, which are combined with set expansion operations to score translation candidates through the Jaccard similarity coefficient (Etchegoyhen & Azpeitia, 2016; Azpeitia al., 2017; 2018). Using multilingual noisy web-crawls such as ParaCrawl" for filtering good quality sentence pairs has been explored in the shared tasks for high resource (Koehn et al., 2018) and low resource (Koehn et al., 2019) languages.
In this work, we rely on massively multilingual sentence embeddings and margin- margin-based mining in the joint embedding space, as described in (Schwenk, 2018; Artetxe & Schwenk, 201 2018b;a). This approach has also proven to perform best in a low resource scenario (Chaudhary et al., 2019; Koehn ct al., 2019). Closest to this approach is the research described in España- Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019). However, in all these works, only bilingual sentence representations have been trained. Such an approach does not scale to many languages, in particular when considering all possible language pairs in Wikipedia. Finally, related ideas have been also proposed in Bouamor& & Sajjad (2018) or Grégoire & Langlais (2017). However, in those works, mining is not solely based on multilingual sentence embeddings. but they are part of larger system. To the best of our knowledge, this work is the first one that applies the same mining approach to all combinations of many different languages. written in more than twenty different scripts.
Wikipedia is arguably the largest comparable corpus. One of the first attempts to exploit this re- source was performed by Adafre & de Rijke (2006). An MT system was used translate Dutch sentences into English and to compare them with the English texts. This method yielded several hundreds of Dutch/English parallel sentences. Later, a similar technique was applied to the Per- sian/English pair (Mohammadi & GhasemAghaee, 2010). Structural information in Wikipedia such as the topic categories of documents was used in the alignment of multilingual corpora (Otero & López, 2010). In another work, the mining approach of Munteanu & Marcu (2005) was applied to extract large corpora from Wikipedia in sixteen languages (Smith et al., 2010). Otero et al. (2011) measured the comparability of Wikipedia corpora by the translation equivalents on three languages Portuguese, Spanish, and English. Patry & Langlais (201 1) came up with a set of features such as Wikipedia entities to recognize parallel documents, and their approach was limited to a bilingual setting. Tufis et al. (2013) proposed an approach to mine parallel sentences from Wikipedia textual content, but they only considered high-resource languages, namely German, Spanish and Romanian paired with English. Tsai & Roth (2016) grounded multilingual mentions to English wikipedia by training cross-lingual embeddings on twelve languages. Gottschalk & Demidova (2017) searched for parallel text passages in Wikipedia by comparing their named entities and time expressions. Fi- nally, Aghaebrahimian (2018) propose an approach based on bilingual BiI.TTM sentence encoders to mine German, French and Persian parallel texts with English. Parallel data consisting of aligned
2

---

Under review as a conference paper at ICLR 2020
Wikipedia titles have been extracted for twenty-thres languages'. Since Wikipedia titles are rarely entire sentences with a subject, verb and object, seems that only modest improvements were ob- served when adding this resource to the training material of NMT systems.
We are not aware of other attempts to systematically mine for parallel sentences in the textual content of Wikipedia for large number of languages.
DISTANCE-BAL -BASED MINING APPROACH
The underling idea of the mining approach used in this work is first learn multilingual sentence embedding, i.e. an embedding space in which semantically similar sentences are close independently of the language they written in. This means that the distance in that space can be used as an indicator whether two sentences are mutual translations or not. Using simple absolute threshold on the cosine distance was shown to achieve competitive results (Schwenk. 2018). However, has been observed that an absolute threshold on the cosine distance is globally not consistent, e.g. (Guo al., 2018). The difficulty to select one global threshold is emphasized in our setting since we are mining parallel sentences for many different language pairs.
3.1 MARGIN CRITERION
The alignment quality can be substantially improved by using a margin criterion instead of an ab- solute threshold (Artetxe & Schwenk, 2018b). In that work, the margin between two candidate sentences I and is defined as the ratio between the cosine distance between the two sentence embeddings, and the average cosine similarity of its nearest neighbors both directions:
cos(æ, y)
margin(x,
cos(x, 2)
cos(y, 2)
2k
2k
xENNe(w)
2ENN&(M)
where & (z) denotes the i unique nearest neighbors of in the other language, and analogously for NN, (v). We used k: = 4 in all experiments.
We follow the "max" strategy as described (Artetxe & Schwenk, 2018h): the margin is first cal- culated in both directions for all sentences in language L, and L2. We then create the union of these forward and backward candidates. Candidates are sorted and pairs with source or target sentences which were already used are omitted. We then apply threshold on the margin score decide whether two sentences are mutual translations or not. Note that with this technique, we always get the same aligned sentences, independently of the mining direction, e.g. searching translations of French sentences in a German corpus, or in the opposite direction. The reader is referred to Artetxe & Schwenk (201 8b) for detailed discussion with related work.
The complexity of a distance-base based mining approach is O(N x M), where N and M are the num- ber of sentences in each monolingual corpus. This makes brute-force approach with exhaustive distance calculations intractable for large corpora. Margin-based mining was shown to significantly outperform the state-of-the-art on the shared-task of the workshop on Building and Using Compa- rable Corpora (BUCC) (Artetxe & Schwenk, 2018b). The corpora in the BUCC corpus are rather small: most 567k sentences.
The languages with the largest Wikipedia are English and German with 134M and 51M sentences, respectively, after pre-prooe processscsn (see Section 4.1 for details). This would require 6.8 x 10 distance calculations. We show in Section 3.3 how to tackle this computational challenge.
3.2 MULTILINGUAL SENTENCE EMBEDDINGS
Distance-based based bitext mining requires a joint sentence embedding for all the considered languages. One may be tempted train bi-lingual embedding for each language pair, e.g. (España-Bon
'ttpps: / 1 inguatools www.g.....g..ooo.oooo. orgg tool s //ccrpora//wkkkt wetporr/wwwiippdiiii e--araall papralllll -escooooaaoo 'Strictly speaking, Cebuano and Swedish are larger than German, yet mostly consist of template/machine translated text https: /en. wi pedi org/ wikk / st__s____f _f___o____e_____ w pedias

---

Under review as a conference paper at ICLR 2020
ENCODER
sent cmb
DECODER
4/5P
BILSTM
BISTM
BR:STM
sottmax
softmax
softmax
ISTM
ISTM
ISTM
BPE cmb
BPE
BPE emb
sent
BPE
sent
BPE
sent
RPE
</>>
Table 1: Architecture of the system used to train massively multilingual sentence embeddings. See Artetxe & Schwenk (2018a) for details.
et al., 2017; Hassan et al., 2018; Guo et al., 2018; Yang et al., 2019), but this is difficult to scale to thousands of language pairs present in Wikipedia. Instead, we chose multilingual sentence embedding for all languages, namely the one proposed by the open-source LASER toolkit (Artetxe & Schwenk, 2018a). Training one joint multilingual embedding on many languages at once also has the advantage that low-resource languages can benefit from the similarity to other language in the same language family. For example, we were able mine parallel data for several Romance (minority) languages like Aragonese, Lombard, Mirandese or Sicilian although data in those languages was not used to train the multilingual LASER embeddings.
one single massively
The underlying idea of LASER is to train a once using a shared BPE vocabulary and a shared encoder for all languages. The sentence represen- tation is obtained by max-pooling over all encoder output states. Figure I illustrates this approach. The reader is referred to Artetxe & Schwenk (2018a) for a detailed description.
system many language pairs at
3.3 FAST SIMILARITY SEARCH
Fast large-scale similarity search is an area with a large body of research. Traditionally, the ap- plication domain is image search, but the algorithms are generic and can be applied to any type of vectors. In this work, we use the open-source FAISS library" which implements highly ef- ficient algorithms to perform similarity search billions of vectors (Johnson et al., 2017). An additional advantage is that FAISS has support to run on multiple GPUs Our sentence represen- tations are 1024-dimensional. This means that the embeddings of all English sentences require 153. 10 x 1024 x 4 = 513 GB of memory. Therefore, dimensionality reduction and data compres- sion are needed for efficient search. In this work, chose rather aggressive compression based on a 64t product-quantize (Jégou et al., 2011), and portioning the search space in 32k cells. This corresponds to the index type "OPQ64, IV732768, PQ64" in FAISS terms. Another interesting compression method is scalar quantization. A detailed comparison is left for future research. We build and train one FAISS index for each language.
The compressed FAISS index for English requires only 9.2GB, i.c. more than fifty times smaller than the original sentences embeddings. This makes possible load the whole index on a standard GPU and to run the search in very efficient way multiple GPUs in paraliel, without the need to shard the index. The overall mining process for German/English requires less than 3.5 hours on GPUs, including the nearest neighbor search in both direction and scoring all candidates
4 BITEXT MINING IN WIKIPEDIA
For each Wikipedia article, it is possible get the link to the corresponding article in other lan- guages. This could be used to mine sentences limited to the respective articles. One one hand, this local mining has several advantages: 1) mining is very fast since each article usually has few hundreds of sentences only; 2) it seems reasonable to assume that translation of sentence is more likely to be found in the same article than anywhere in the whole Wikipedia. On the other hand, we
htttps///gi thub. com http://tt....cooccoooooeeeeeeeaa
/aai i sa- indexs
4

---

Under review as a conference paper at ICLR 2020
T (French)
Ceci est une très grande maison
L2 (German)
Das ist ein sehr großes Haus This is very big house Ez egy nagyon nagy ház Ini rumah yang sangat besar
Table 2: Illustration how sentences in the wrong language can hurt the alignment process with margin criterion. See text for detailed discussion.
hypothesize that the margin criterion will be less efficient since one article has usually few sentences which are similar. This may lead to many sentences in the overall mined corpus of the type "NAME was born DATE in CITY", "BUILDING is monument in CITY built DATE", etc. Although those alignments may be correct, we hypothesize that they are of limited use to train an NMT sys- tem, in particular when they are too frequent. In general, there risk that will get sentences which are close structure and content.
The other option is to consider the whole Wikipedia for each language: for each sentence in the source language, we mine in all target sentences. This global mining has several potential advan- tages: we can try align
languages even though there are only few articles in common; 2) many short sentences which only differ by the name entities are likely to be excluded by the margin criterion. A drawback of this global mining potentially increased risk of misalignment and lower recall.
In this work, we chose the global mining option. This will allow us to scale the same approach to other, potentially huge, corpora for which document-level alignments are not easily available, e.g. Common Crawl. An depth comparison local and global mining (on Wikipedia) left for future research.
4.1 CORPUS PREPARATION
Extracting the textual content of Wikipedia articles in all languages is rather challenging task, i.e. removing all tables, pictures, citations, footnotes or formatting markup. There are several ways to download Wikipedia content. In this study, we the so-called CirrusSearch dumps since they directly provide the textual content without any meta information." 9 We downloaded this dump in March 2019. A total of about 300 languages are available, but the size obviously varies lot between languages. We applied the following processing:
extract the textual content;
split the paragraphs into sentences;
remove duplicate sentences;
perform language identification and remove sentences which not in the expected lan- guage (usually, citations or references to texts in another language).
should be pointed out that sentence segmentation is not a trivial task, with many exceptions and specific rules for the various languages. For instance, is rather difficult to make an exhaustive list of common abbreviations for all languages. In German, points are used after numbers in enumerations, but numbers may also appear the end of sentences. Other languages do not use specific symbols to mark the end of a sentence, namely Thai. We are not aware of a reliable and freely available sentence segmenter for Thai and we had exclude that language. We used the freely available Python to which is based on Moses scripts. Regular expressions were used for most of the Asian languages, falling back to English for the remaining languages. This gives us 879 million sentences in 300 languages. The margin criterion to mine for parallel data requires that the texts do not contain duplicates. This removes about 25% the sentences."
pietps://dunee wikimedia wwk.ma/daa.cr//cteeee//rrssssss "https: http//ypp..ot weroojec/seeeteeee /ww./eneeeeeeeeeeeeeeeeeeeee i itter/ "The Cehuano and Waray Wikipedia were largely created by hot and contain than 65% of duplicates.
5

---

Under review as a conference paper at ICLR 2020
3020
630
213
1020
16
csde
550
24
2230
25
450
2010
450
1850
24
23
1820
1400
300
22.
1200
1030
200
22.
803
21.3
50
1.56
1,005
1.05
3.015
1.01
1.035
1.03
1.06
1.255
1.05
1.045
3.00
1.035
1.03
Margin treeshold
Hargin threshole
Figure 1: BLEU scores (continuous lines) for several NMT systems trained on bitexts extracted from Wikipedia for different margin thresholds. The size of the mined bitexts are depicted as dashed lines.
LASER's s sentence embeddings are totally language agnostic. This has the side effect that the sen- tences in other languages (c.g. citations or quotes) may be considered closer in the embedding space than a potential translation in the target language. Table 2 illustrates this problem. The algorithm would not select the German sentence although it a perfect translation. The sentences in the other languages are also valid translations which would yield a very small margin. To avoid this problem, we perform language identification (LID) on all sentences and remove those which are not in the expected language. LID is performed with fa (Joulin et al., 2016). Fasttext does not support all the 300 languages present in Wikipedia and we disregarded the missing ones (which typically have only few sentences anyway). After deduplication and LID, we dispose of 595M sentences in 182 languages. English accounts for 134M sentences, and German with 51M sentences is the second largest language. The sizes for all languages given in Tables 4 and 6.
4.2 THRESHOLD OPTIMIZATION
Artetxe & Schwenk (2018h) optimized their mining approach for each language pair on provided corpus of gold alignments. This is not possible when mining Wikipedia, in particular when con- sidering many language pairs. In this work, shared task on parallel corpus filtering for low-resource conditions (Koehn et al., 2019): an NMT system is trained the extracted bitexts for different thresholds and the resulting BLEU scores are compared. We choose newstest204 parallel test sets for English, French, German and Czech. We favoured the translation between two morphologically rich languages from different families and considered the following language pairs: Cerman/Englisse German/French, Czech/German and Czech/French. The size of mined bitexts is in the range of 100k to more than 2M (see Table 3 and Figure 1). We did not try to optimize the archi- tecture of the NMT system the size of the bitexts and used the same architecture for all systems: the encoder and decoder are 5-layer transformer models as implemented in fa fairsed (Ott et al., 2019). The goal of this study is not to develop the best performing NMT system for the considered languages pairs, but to compare different mining parameters.
an evaluation protocol inspired by the WMT
of the WMT evaluations since it provides an N-way
The evolution of the BLEU score function of the margin threshold given in Figure 1. De- creasing the threshold naturally leads to more mined data we observe an exponential increase of the data size. The performance of the NMT systems trained on the mined data seems to change as expected, in surprisingly smooth way. The BLEU score first improves with increasing amounts of available training data, reaches a maximum and than decreases since the additional data gets more and more noisy, Lee. contains wrong translations. It is also not surprising that careful choice of the margin threshold is more important in a low-resource setting. Every additional parallel sentence important. According Figure 1, the optimal value of the margin threshold seems be 1.05 when many sentences can be extracted, in our case German/English and German/French. When less parallel data available, i.c. Czech/German and Czech/French, value the range of 1.03-1.04 seems to be a better choice. Aiming at one threshold for all language pairs, we chose value of 1.04. seems be good compromise for most language pairs. However, for the open release of this corpus, we provide all mined sentence with a margin of 1.02 or better. This would enable end users
"Phttps://faattee wwtp/cs/e/////e////ec/...cc lanuuage-uugggeeeee dent ficati on. html

---

Under review as a conference paper at ICLR 2020
Bitexts
de-en
de-fr
cs-de
cs-fr
1.9M
1.9M
568k
627k
21.5
23.6
14.9
21.5
Europarl
1.0M
370k
200k
220k
21.2
21.1
12.6
19.2
Mined 1.0M
372k
201k
219k
Wikipedia 24.4
22.7
13.1
16.3
Europarl 3.0M
2.3M
768k
846k
Wikipedia 25.5
25.6
17.7
24.0
Comparison of NMT systems trained on the Europarl corpus and on bitexts automatically mined in Wikipedia by our approach at a threshold of 1.04. We give the number of sentences (first line) and the BLEU score (second line of each bloc) on newstest
Table 3:
choose an optimal threshold for their particular applications. However, it should be emphasized that we do not expect that many sentence pairs with margin as low as 1.02 are good translations.
For comparison, also trained NMT systems the Europarl corpus V7 (Koehn, 2005), i.e. pro- fessional human translations, first on all available data, and then on the same number of sentences than the mined ones (see Table 3). With the exception of Czech/French, we were able achieve better BLEU scores with the automatically mined bitexts in Wikipedia than with Europarl of the same size. Adding the mined text to the full Europarl corpus, also leads to further improvements of 1.1 to 3.1 BLEU. We argue that this is a good indicator of the quality of the automatically extracted parallel sentences.
RESULT ANALYSIS
We run the alignment process for all possible combinations languages in Wikipedia. This yielded 1620 language pairs for which were able mine least thousand sentences. Remember that mining L, + L2 is identical to L2 + Lt, and is counted only once. We propose to analyze and evaluate the extracted bitexts two ways. First, we discuss the amount of extracted sentences (Section 5.1). We then turn to a qualitative assessment by training NMT systems for all language pairs with more than twenty-five thousand mined sentences (Section 5.2).
5.1 QUANTITATIVE ANALYSIS
Due to space limits, Table 4 summarizes the number of extracted parallel sentences only for lan- guages which have a total of at least five hundred thousand parallel sentences (with all other lan- guages a margin threshold of 1.04). Additional results are given in Table 6 in the Appendix.
There are many reasons which can influence the number of mined sentences. Obviously, the larger the monolingual texts, the more likely is mine many parallel sentences. Not surprisingly, we observe that more sentences could be mined when English is one of the two languages. Let us point out some languages for which it is usually not obvious to find parallel data with English, namely Indonesian (1M), Hebrew (545k), Farsi (303k) or Marathi (124k sentences). The largest mined texts not involving English are Mussian/UUkaanne (2.5M), Catalan/Spanish (1.6M), between the Romance languages French, Spanish, Italian and Portuguese (480k-923k), and German/French (626k).
striking to see that we were able to mine more sentences when Galician and Catalan are paired with Spanish than with English. On one hand, this could be explained by the fact that LASER's multilingual sentence embeddings may be better since the involved languages are linguistically very similar. On the other, it could be that the Wikipedia articles in both languages share a lot of content, or are obtained by mutual translation.
Services from the European Commission provide human translations of (legal) texts in all the 24 official languages of the European Union. This N-way parallel corpus enables training of MT system to directly translate between these languages, without the need to pivot through English. This is

---

tobil
86 4135
885
697
8933
3770
312488
86- 85-1115
780 1011111
39 22111
17422800
40 4551
25 2699
88 4::11
04 64.99
19441
68 2:300
22 75
459
9/511
371329
$1391
1648
584
(364
4771
706
6835
4515
14311311
3337
1464
2726
3510
5107
662
1315
1315
3408
603
7043
4415
80
26
14
15
57
6
W 02
213
60
10
11
90
80
31
38
22
68
22
1072
42
60
105
38
16
143
25
29
33
57
14
14
79
96
122
42
21 14
38
48
20
56
24
10
19
16
17
77
126
104
08
30
120
51
144
48
ST
104
9
172
156
14
12
14
11
32
26
12
11
25
28
77 17
10
89
8
75
55
29
180
54
112
21
7
7
02
140
119
ST
Table 4: WikiMatii: number of extracted sentences for each language pair (in thousands), e.g. Es-Ja=219 corresponds to 219.260 sentences (rounded), The column "size" gives the number of lines in the monolingual texts after deduplication and LID.
te
18
21
23
21
8
13
31
24
24
21
21
32
21
72
91
17
15
17
22
13
17
30
20
21
12
31
12
11
16
34
26
3
26
54
45
29
17
4
4 16
15
17
14
30
35
SYOW ta
13
5
22
14
30
15
11
35
8
16
6
13
821
19.36
13
10
13
44
19
12
7
13
3
16
17:44
8
W
14
9
10
16
58
10
16
63
97
16
216
62
55
58
58
42
54
56
88
K5
31
46
48
56
17
12
121
46
48
51
50
51
89
65
25
63
43
81
305
21
20
39
45
21
55
26
KS
51
29
29
100
M
7
60
114
373
15
34
25
kg
30
26
17
30
100
19
18
12
17
48
23
23
27
48
28
17
16
35
19
10
17
19
39
45
26
57
64
90
318
32
35
20
24
86
35
38
53
58
W
30
10
62
6
OK
85
7K
22
32
10
43
25
50
57
39
35
178
20
88
29
36
18
28
56
37
15
72
25
38
29
10
6
81
27
22
32
31
52
32
50
31
115
28
9
74
30
21
26
ON
15
19
5
40
42
35
9
41
20
50
32
38
224
26
19
72
30
33
680
46
67
28
23
18
53
6
56
71
Nu
125
47
43
160
270
62
189
185
100
114
180
81
96
47
72
410
84
149
30
19
M9
K0
88
19
199
14
136
70
14
15
69
46
82
70
78
631
43
48
33
56
87
20
161
79
39
14
157
23
114
76
358
155
123
144
2001
76
58
77
558
227
135
148
304
480
175
95
62
86
21
24
2001
pl
74
22
96
52
11 176
89
185
77
688
43
30
235
65
126
20210
128
63
70
56
4 17
177
10S
13
de
8
8
37
9
124
10
6
58
13
10
38
35
86
300
201
60
196688
636
29
32
52
28
28
130
39
38
13
Ill
73
22
82
50
176
139
21
81
42
59
60
86
05
121
27
240
57
55
46
mnr le
12
13
15
1212
15
28 9
39 11
16
10
24
4
32
35
41
35
22
124
21
10
10
25
28
78
25
10
16
22
M mnk
53.32
3629
7
23
61
36
51
3930
52
098
71
3028
20
11
19
11
27
83
35
48
52
48
47
12
55
73
20
26
28
16
31
42
55
157
28
19
20
27
48
62
ko
18
38
20
22
52
55
37
82
35
306
26
29
18
26
89
28
45
27
49
45
85
jA
6
4
9
6
20
7
5
8
9
7
M
1
24
68
25
12
71
38
108
105
76
69
351
48
59
35
46
30
82
48
99
77
18
19
123
102
318
101
115
119
2128
101
54
244
121
121
146
146
18
17
29
20
15
85
14
10
9
58
14
14
20
id
9
61
107
78
65
107
1010
27
161
20
ha
60
15
11
68
41
39
92
105
69
62
688
52
56
35
29
16
30
65
58
36
9
47
63
259
21
33
Md
38
30
21
37
38
25
26
201
22
20
13
20
60
21
28
68
58
72
55
545
25
136
50
10
45
21
32
268
54
24
48
450
46
32
43
25
154
ft
163
29
117
490
185
142
157
2157
134
65
53
10
61
37
35
95
75
55
375
375
N0
33
36
fa
36
31
45
303
23
ea
24
25
14
77
36
26
23
119
25
22
43
62
<5
41
263
31
68
L74
31
28
28
122
81
70
5M0
151
120
145
60
31
9
40
25
34
29
en
950
31
35
3ST
280
2205
519
436
630
el
8
de
99
34
20
13
20
233
CR da
11
9
53
67
14
79
siie Wu tai be bu be Ce
S4
76
34
34
10
11:51
14
16
3
15
6510
1873
1090
3327
1412
8332
7634
2881
50944
3211
114431
2371
23003
2289
3954
34404
2221
6962
2220
77012
3899
J8NN
21025
31614
54M
2031
1487
272
5US
235
34091
472
16220
13354
3014
32537
2060
.30
1991
1836
740
4236
179966
235
1629
1500
312
4067
440
12591
6727
12306
Langunge Family
Arabic
Turkic
Slavie
Slovic
Indo
Slavic
Romunce
Slavic
Cemanic
Helenic
Gemaaic
Komunce
Urali
boliee
Romancc
Romince
Scmisc
Stavie
Urali
Malayo-
Polynesian
Germunc
Romance
Japnnic
Turkic
Koraani
Bilfe
mi Macolionia Stevic
Dranidin
Anda-Arpan
Indi-Aryan
Gemanic
Germnic
Romance
Stavie
Romonee
Stavie
South
Mlavic
Indo-Aspan
Slavie
Stevie
Albusian
Stevie
Gemenic
Conga
Drvvidon
Drscidin
Malago-
Polynesine
Tirkic
Turkie
N Wolmmmow: Vasie
Choose
ISO Name
Arabic
Azeiborjan
bn Belamviaz
Bulgaman
bn Mengali
C1 Cantin
Czech
Damish
Cross
English
Spansn
Ehoonian
01 Basque
Pansi
Pinnsh
French
Hebrew
Hindi
ur Crossan
Hanngmin
Indouciiat
Icelandic
Iallin
Kazakh
kn Korean
Eithuenian
ml Malayaam
mr Mandh
ne Neyali
Dukh
10 Novegian
Oceiten
Polkh
Ponigese
10 Romaina
II Russan
Salbo-
Crosiian
langsages
Siniala
& Skoak
M Albasian
Scrtam
Swshil
Tomil
Telngn
Taplleg
Turkish
Tontr
Ukninin
dh Chinese
be
bg
8

---

Under review as a conference paper at ICLR 2020
Sre/Trgar
bg
bs
da
de
el
89
e5
Idrcca
g
be
hr
hu
id
it
ja
ko
mk
mb
ml
p1
pipt-br
ru
ak
5
SV
tr
uk
vich-cnch-tw
ar
4.9
1.8
3.0
4.5
3.8 6.
6.70
4.
1.22222
9.0
5.6
35
22
2.7
9.2
9.9
4.2
5.3
5.5
4.9
4.4
3.012.0
12.2
5.6 5.6
1.5
2.7
1.2
4.0
2.4
4,
4,512
822
4.9
3.0
4.2
6.7
8.7
8.510.225
7.7
16.16.1
11.
8.4
3.3
64
4.7
4..02.
4.0
5
K1
1.7
6.514.7
154
1.
40
64
2.7
7.1
2,1044117
6.4
4.7
4.1
4.52
129
11101106
78
83
2-000.0
104
5.0
65
7.8 96
41
5.5
63
76
8.
10.
12.1
6.3
16
2.6
9.0
6.6
4.7
da
89
46 52
14.0
9.0
6.167716719
7.3
3.5
44
4.188114
4.7
6.0
6.2
12.
4.814.0
16.2
8.5
7.8
5.2
14258
2.7
6.111
73
4.9
2.4
9.7
8.1169
7.62445
1551741831144
688
55
6613.
6.4
6.6
5611
5174
6.811.2
15.2
8.7
54
1.512.
3.6
7.11.
4.3
5.5
11.3
56
111
12.001
10.117.6361
18:0
10.7
L8 6.4
7.3 7.13.5
8. 8.123
17.11.11..7
10.5
1.9
3.0
4.5
9.2
2.5
4.8 6.2
72212.1
6.7 16.0
6.9 8.6
7.4
S6
0.6
8.6
188 5.4 7.7
5.2
3.6
6.114.3 68
5015712.916
33/2 13.5
25.6
19.11999300
8.7
1.716
123.8
7.9
919.119553144
K.127.6 27.8
14.711
6.0
8.1
26
1.200.778
12.3
6.6
6.022.
833156145155
1862
8710715..155
Trea
49125
1.814.12..155
27.8
7
68
13.2233
7.5
8810.3
[5.]
7.218.6
23.2
15.31133
6.1
3.5
5.0
97154
6.2
26 7.3
4.
29 74
5.3
8.5234
9 9 160152
2.5 35
25
9.8
36
4.2
5.6
69
6.5
4,502.4
23.7
7.7
5.9
17
5.1
0.2
4.3
2.3
3. 94
58
4.2
5.7
37
5.4
6.325.2
4,2153
5.1
25
89112
4.1 4.5
46
54
6.5
3.7
15.0
6.3
19
1.0
52
5.1
4.6
6.6
57
62104
12.6
56
7.0
5.516.7
6.5
3
36
6.6
6.
4.3
7.1
34
1.2
5.
3.0
4.4
8.5
4.2
id
4.
9.1
422
1210
1.7
4.2
11.1
9.9
5.1
56
52
12.7
3.8
9.1
7.010.0
9
5.514.
16.7
9.8 8.1
3.6
56
18
9.3
4.6
7.318.
11.0
6.2
5.1L 50
6913.111.5
500.0
62
70
7.3
9.
9.133312.
7.522.8
24.9
1.
54
2
4.6
8.15
10.7
5.8
ko
099
L.
13
2.0
1.7
1.7 8.7
1.3
4.7 4 3.4
1.5
09
07
15
3.1
3.2
922
1.2
13
1.
1.4
2.0
2.1
1.0
1.5
04
1.5
L4
1.5
4.3
mk
2418
120
54
7.2
4.500
3234
85152115
11.00
5.4
40
S7
1
3.5
47
7.5
6.7
4.5133.
15.6
8.3
7.5
1.5
7
37
56
2.0
64108
63 4.3
27
8
9.8
8 1
9.5176
14.6
8.0
3.1
36
9.915
5.5
57 67
4.7144.
9.5
7.5
32
35
0
1.9
6.2
78
13.116.161
13.7
7.3
5.313.6
7.8
4.1
16
D
L8 7
82
6.5
5 15.1
411
8.6
5.2
488
7.5
4.2
5
5.2
4.1
6.2
9.9
6.2
62
1.5
56
2.4
8.9
7.7
63
3.6
2 T4 8.015.55 13:218.
12
26
19.023.0
103
8.117.5240
7.8
10.6
14.
8.9
15.
6.6
8.1
2.0
6.2
9.318U
12.9
588
pi-br
6.5 2
16168112.11
A 16.0
26.6
20.32300
8.7
98
8. 18.624
7.8
10.712.5
16.
K.5
15.1
6.1
28
5.3108188
13.2
6.7
11.15
17/7
36
Tu
3.312.6
4.2
7.
8.5
8.8
8.5
9514411
14.5
11.0
6.0
4.8
68
9.511.7
6.1
7.2
74
80
8.1
19124
13.5
8.2
58
54
2.7
8.2
2
2.422
11.5
9.1
5.2
sk
0.7
5
2
270
4.3
5.7
3.2
9.3
9.1
8.5
6.7
2.7
LC
5.1
3.7
49
6
22
399
35
499
SC
7.1
7.8
8.5
3.5
66
5C
1.5
4.3
16
5.1
5.4
2.3
2.5
62
55
766
56114
8.5
3.2
2
6.5
78
2.7
63
4.
5.5
4.8
9.9
488
5.
36
19
2.2
4.1
74
388
266
3.5
6.1
2913
11.
5.6
28
SY
1.2
7.4
5.926
12.6
8. B
11C151157100
10.7
7.
55
5111
48
6.2
5
11
5.I50
17.4
7.9
8
38
1.0
3.2
68128
7.8
4.8
tr
2.2
3.5
26
26
3.9
4,1
4.715.9
25 9.
13
6.7
3.6
1.
21
34
6.7
6.4
4,3
7
35
3.1
422
2.5
9.0
8A4
4.6
4.
18
22
08
3.5
3.3
82
6.7
4.4
2.11..3
74
7.5
8 20.7
6.514.14.
11.2
5.5
66
9.5112
4.9
7.2
63
9.6
12.8
49
26
2.6
11.
4,9
7.5
6.0
620
6.5
10.
14.1
zh-en
2.1
3.2
16
22
32
4.5118
38
82
1.6
322
1.7
19
3L
6.6
6.0
3.4
38
2.2
7.1
7.9
4.1
4.1
16
24
89
3.1
2.3
3.0J0U
2h-tw
2.2
3.1
111
2.1
3.7
28
3.9
3.4
7.5
72
6.1
288
18
1.6
3X
6.2
5.4
28
3.5
2.3
6.3
6.9
3.5
39
14
2.
0.9
3.0
2.4
2.900.0
Table 5: BLEU scores on the TED test set as proposed in (Qi et al., 2018). NMT systems were trained on bitexts mined in Wikipedia only (with least twenty-five thousand parallel sentences). No other resources were used.
usually not the case when translating between other major languages, for example in Asia. Let us list some interesting language pairs for which we were able to mine more than hundred thousand sentences: Korcan/Japanese (222k), Russian/Japanese (196k), Viteesiian//ietameeee (146k), or Hebrew/Romance languages (1 2050k50k sentences).
Overall, were able extract at least ten thousand parallel sentences for 85 different languages. I For several low-resource languages, we were able to extract more parallel sentences with other languages than English. These include, among others, Aragonse with Spanish, Lombard with Italian, Breton with several Romance languages, Western Frisian with Dutch, Luxembourgish with German or Egyptian Arabic and Wu Chinese with the respective major language.
Finally, Cebuano (ceb) falls clearly apart: it has a rather huge Wikipedia (17.9M filtered sentence), but most of it was generated by a bot, as for the Waray language This certainly explains that only a very small number of parallel sentences could be extracted. Although the same bot was also used to generate articles in the Swedish Wikipedia, our aligmments seem to be better for that language.
5.2 QUALITATIVE EVALUATION
Aiming to perform a large-scale assessment of the quality of the extracted parallel sentences, we trained NMT systems on the extracted parallel sentences. We identified a publicly available data set which provide test sets for many language pairs: translations of TED talks as proposed in the context of study on pretrained word embeddings for NMT (Qi et al., 2018). We would like to emphasize that we did not use the training data provided by TED- we only trained on the mined sentences from Wikipedia. The goal of this study is not to build state-of-the-art NMT system for for the TED task, but to get an estimate of the quality of our extracted data, for many language pairs. In
1399 languages have more than 5,000 parallel sentences. "https: / http///ewwwwiiiee wiki wikipedia a htrg/wikk /isjjor "htpps: / / thub. com/ neul ab/worddwwaaeeoooeeeewww embedd' ngs- for
9

---

Under review as a conference paper at ICLR 2020
particular, there may be mismatch in the topic and language style between Wikipedia texts and the transcribed and translated TED talks.
For training NMT systems, we used transformer model from fairseg (Ott et al., 2019) with the parameter settings shown in Figure in the appendix. For preprocessing, the text was tokenized using the Moses tokenizer (without true casing) and 5000 subword vocabulary SentencePiece (Kudo & Richardson, 2018). Decoding was done with beam size 5 and length nor- malization 1.2.
learnt using
We evaluate the trained translation systems on the TED dataset (Qi et al., 2018). The TED data con- sists of parallel TED talk transcripts multiple languages, and provides development and test sets for 50 languages. Since the development and test sets were already tokenized, we first detokenize them using Moses. We trained NMT systems for all possible language pairs with more than twenty- five thousand mined sentences. This gives us in total 1886 language pairs in 45 languages. We train Ln + L2 and L2 + L1 with the same mined bitexts L,/L.2. Scores on the test sets were computed with SacreBLEU (Post, 2018). Table 5 summarizes all the results. Due to space constraints, we are unable to report BLEU score for all language combinations in that table. Some additional results are reported in Table 7 in the annex. 23 NMT systems achieve BLEU scores over 30, the best one being 37.3 for Brazilian Portuguese to English. Several results are worth mentioning, like Farsi/Englishe 16.7, Hebrew/English. 25.7, Indonesian/Engllih:: 24.9 or English/Hindi: 25.7 We also achieve inter- esting results for translation between various non English language pairs for which is usually not easy to find parallel data, e.g. Norwegian ++ Danish <333, Norwegian Swedish 222,, Indonesian + Vietnamese & 16 or Japanese/ / Korean 177
Our results on the TED give an indication on the quality of the mined parallel sentences. These BLEU scores should be of course appreciated in context of the sizes of the mined corpora as given in Table 4. Obviously, we can not exclude that the provided data contains some wrong alignments even though the margin is large. Finally, we would like to point out that we run our approach on all available languages in Wikipedia, independently of the quality of LASER's sentence embeddings for each one.
6 CONCLUSION
We have presented approach systematically mine for parallel sentences in the textual content of Wikipedia, for all possible language pairs. We use a recently proposed mining approach based on massively multilingual sentence embeddings (Artetxe & Schwenk, 2018a) and margin criterion (Artetxe & Schwenk, 2018b). The same approach is used for all language pairs without the need of a language specific optimization. In total, we make available 135M parallel sentences in 85 languages, out of which only 34M sentences are aligned with English. We were able to mine more than ten thousands sentences for 1620 different language pairs. This corpus of parallel sentences freely available. 16 We also performed large scale evaluation of the quality of the mined sentences by training 1886 NMT systems and evaluating them on the 45 languages of the TED corpus (Qi et al., 2018).
This work opens several directions for future research. The mined texts could be used to first re- train LASER's multilingual sentence embeddings with the hope to improve the performance low-resource languages, and then to rerun mining in Wikipedia. This process could be iteratively repeated. We also plan apply the same methodology to other large multilingual collections. The monolingual texts made available by ParaCrawl or CommonCraw!!" are good candidates.
We expect that the WikiMatrix corpus has mostly well-formed sentences and it should not contain social media language. The mined parallel sentences are not limited to specific topics like many of the currently available resources (parliament proceedings, subtitles, software documentation, ...,, but are expected to cover many topics of Wikipedia. The fraction of unedited machine translated text also expected to be low. We hope that this resource will be useful to support research in multilingualit,, in particular machine translation.
16 Anonymized for review "ttp::/ http:///cccm///////////t/////att orrg/
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sadaf Abdul-Rauf and Holger Schwenk. On the Use of Comparable Corpora to Improve SMT performance. EACL, pp. 16-23, 2009, URL http://www.wtt. web. wwt.:/ww............eeeeeeeeee aegggnnthllogy/// E09-10033
Sisay Fissaha Adafre and Maarten de Rijke. Finding similar sentences across multiple languages in Wikipedia. In Proceedings of the Workshop NEW TEXT Wikis and blogs and other dynamic text sources, 2006.
Ahmad Aghacbrahimian. Deep neural networks the service of multilingual parallel sentence extraction. In Coling, 2018.
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. In https://arxix: http://wwww..rrr/////////// 10464, 2018a.
Mikel Artetxe and Holger Schwenk. Margin-based Parallel Corpus Mining with Multilingual Sen- tence Embeddings. https: /ttp::///rxiiv org/ att//b//////// /1111. 01136, 2018b.
Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martínez Garcia. Weighted Set-Theoretic Align- ment of Comparable Sentences. In BUCC, pp. 41-45, 2017. URL http: http::////.wwbbr/////// org/ anthology w17-2506
Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martínez Garcia. Extracting Parallel Sentences from Comparable Corpora with STACC Variants. In BUCC, may 2018.
Houda Bouamor and Hassan Sajjad. H2@ BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings. In BUCC, may 2018.
Christian Buck and Philipp Koehn. Findings of the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pp. 554-563, Berlin, Ger- many, August 2016. Association for Computational Linguistics. URL http: www. aclweb. org/ anthology /w////////// 616-2347.
Vishrav Chaudhary. Yuging Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. Low- resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (WM7), 2019.
Cristina -opaña-Booet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Josef van Genabith. An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification. IEEE Journal of Selected Topics in Signal Processing, pp. 1340-1348, 2017.
Thierry Etchegoyhen and Andoni Azpeitia. Set-Theoretic Alignment for Comparable Corpora. In ACL, pp. 2009-2018, 2016. doi: 10. 11181///1/111--.. 1189. URL http: //t:://wwwww htwp//www/ooooo///oo wwwp//wwwwww.....////.///////..h aclweb. org/ anthology/P166-118
Simon Gottschalk and Elena Demidova. Multiwiki: Interlingual text passage alignment in Wikipedia. ACM Transactions the Web (TWEB), 11(1):6, 2017.
Francis Grégoire and Philippe Langlais. BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora. BUCC, pp- 46-50, 2017. URL http: / aclweb. org/ anthology/ /W1 -52509.
Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Effective Parallel Corpus Mining using Bilingual Sentence Embeddings. arXiv: 10.11 1996, 2018
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang. Marcin Junczys- -owomutt William Lewis, Mu Li, Shujie Liu, Tie- Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingee Xia, Dongdong Zhang. Zhirui Zhang, and Ming Zhou. Achieving Human Parity on Automatic Chinese English News Translation. arXiv: 803.0556... 2018.

---

Under review as a conference paper at ICLR 2020
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scall similarity search with GPUs. arXiv preprint arXiv:1702.087355 2017.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient wwtpa/////gg/////////////aa/// 607. 01 759, 2016.
text classification. ht
H. Jégou, M. Douze, and Schmid. Product quantization for nearest neighbor search. IEEE Trans. PAMI, 33 2011.
Philipp Kochn. Europarl: A parallel corpus for statistical machine translation. In MT summit, 2005.
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L.. Forcada. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 726-739, Belgium, Brussels, October 2018. Association for Computational Lingusstiss. URL https: /www. aclweb org/ wttg//nnhologyy/// /W11 6453.
Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan M. Pino. Findings of the wmt 2019 shared task parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation, Volume Shared Task Papers, Florence, Italy, August 2019. Association for Computational Linguisties.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference Em- pirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, 2018. Association for Computational Linguistics. URL https: www. aclweb. httg//nnhhloogyDDBBgg
Lison and J. Tiedemann. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In LREC, 2016.
Mehdi Zadeh Mohammadi and Nasser Ghasem Building bilingual parallel corpora based on Wikipedia. In 2010 Second International Conference Computer Engineering and Applications, pp. 264-268, 2010.
Dragos Stefan Munteanu and Daniel Marcu. Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. Computational Linguistics, 31(4):477-504, 2005. URL
http:/ /ww. wowwacck wwwwwb.w........ccc...........e. http://anhloooyooo--o 4003.
P Otero, I López, Cilenis, and Santiago de Compostela. Measuring comparability of multilingual corpora extracted from Wikipedia. Iberian Cross-Language Natural Language Processings Tasks (ICL), pp. 8, 2011.
Pablo Gamallo Otero and Isaac González López. Wikipedia as multilingual source of comparable corpora. In Proceedings of the 3rd Workshop on Building and Using Comparable Corpora, LREC, pp. 21-25, 2010.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48-53, 53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. URL ht www. ac web. www..orggg..oooooo///oooooo 9-4009.
Alexandre Patry and Philippe Langlais. Identifying parallel documents from a large bilingual col- lection of texts: Application to parallel article extraction in Wikipedia. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pp-
87-95. Association for Computational Linguistics, 2011.
Matt Post. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https: / /www. aclweb. org/antholoogy/
w1 2118-6319 9.
12

---

Under review as a conference paper at ICLR 2020
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. When and why are pre-trained word embeddings useful for neural machine translation? In Pro- ceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Techhologies, Volume 2 (Short Papers), pp. 529- 535, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL http: http///www// hwwp://wwwwww.....o//o////.....p org/ aaaaantthooooyy. /N18- 20044.
Philip Resnik. Mining the Web for Bilingual Text. In ACL, URL org/ warg/antholoog ogy 9t- 1068
Philip Resnik and Noah A. Smith. The Web as Parallel Corpus. Computational Linguistics, 29(3):
349-380, 2003. URL http: /www. aclweb org/ antaanthoooho JJ0-//00323222
Holger Schwenk. Filtering and mining parallel data in joint multilingual space. In ACL, pp- 228-234, 2018.
Jason R. Smith, Chris Quirk, and Kristina Toutanova. Extracting parallel sentences from comparable corpora using document level alignment. In NAACL, pp. 403-414311 411,,11,110 2010.
Tiedemann. Parallel data, tools and interfaces in OPUS. In IREC, 2012.
Chen-Tse Tsai and Dan Roth. Cross- looss-lingual wikification using multilingual embeddings. In Pro- ceedings of the 2016 Conference of the North American Chapter of the Association for Computa- fional Linguistics: Human Language Technologies, pp. 589-598, 2016.
Dan Tufis, Radu Ion, Stefan Daniel, Dumitrescu, and Dan Stefānescu. Wikipedia as an smt training corpus. RANLP, pp. 702-709, 2013.
Masao Utiyama and Hitoshi Isahara. Reliable Measures for Aligning Japanese-English News Articles and Sentences. In ACL, 2003. URL http::/wwwwwwwlww.oogoo///oooo/. PO3 1010.
Yinfei Yang, Gustavo Hernández Ábrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun- Hsuan Sung, Brian Strope, and Ray Kurzweil. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In 1902. 08564, 2019.
Michat Ziemski, Marein Junezys-Dowmund and Bruno Pouliquen. The United Nations Parallel Corpus v1.0. In LREC, may 2016.
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
Table 6 provides the amounts of mined parallel sentences for languages which have a rather small Wikipedia. Aligning those languages obviously yields a very small amount of parallel sentences. Therefore, we only provide these results for alignment with high resource languages. It is also likely that several these alignments are of low quality since the LASER embeddings were directly trained on most these languages, but we still hope to achieve reasonable results since other languages of the same family may be covered.
ISO
Name
Language Family
sine cm da de
en
fr
It
pl
pt sv rm zotatal
an
Angonese Rominee
222
24
12122
23 33
16
13
9
10
14
II
324
Egyztia
Arabic
120
12
10
278
14 Assmese
Indo-Aryan
124
10
azb
South Azer Turkic
395
9
8
9
10
7
172
buijani
bpy
Dishnupriya Indo Aryan
128
71
br
Braton
Celtic
413
20
16
22
23
22
19
16
200
Chechen
Northeast
315
Cehuano
Malayo-
17919
22
24
20
Polynesian
ckb
Central Kur
Iramian
127
Chuvash
Tuskic
198
129
dy
Maldiviun
Inda-
52
2
6
3
3
3
5
96
fo
Germanic
Germanic
Frisian
ed
Gaelic
Celtic
66
41
Insh
Irish
216
gom
Indo-Aryau
Konkami
ht
Haitian Cre Creole
60
2
4
2
72
Philippine
to
Ido
constructed
153
143
ge
Javanese
Malayo-
220
10
18
11
8
219
Polynesian
Georgian
ku
Kurdish
Iranian
165
5
6
222
la
Lasin
Romance
558
12
17
20
18
17
13
18
478
Garmanic
Lombard
Romance
mg
Malagasy
Malayo-
263
12
199
Polynesian
mhr
Unalic
min
MintaggaabanMallaa
255
121
Polynesian
Mongalian
Mongolic
255
Romance
nds Low
Ger Germanic
10
151
man/Saton
ps
Pashto
sah
Yokut
Taakic/Sih
134
6
134
Sizliaa
Romance
81
143
Sinchi
Tomian
Sandanese
Malayo-
Polynesian
1k.
Tuekmen
Tuskic
56
2
3
4
3
76
1g
Ionian
ug
Lighar
ur
Unhu
Andan
150
2
123
Walloon
Romanes
56
5
4
3
93
Chines
Chinese
yi
Germanic
Table 6: WikiMatrix (part 2): number of extracted sentences (in thousands) languages with a rather small Wikipedia. Alignments with other languages yield less than 5k sentences and are omitted for clairty.
14

---

Under review as a conference paper at ICLR 2020
Table 2 gives the detailed configuration which used to train NMT models the mined data in Section 5.
arch transformer
share-alllembeddinns
caccder-layers
decoder-layers 5
encoder-embbdddlln 512
decoder--embed-ddi 512
encoder-ffn-emmod--in 2048
deccdder-fff--emiddd--- 2048
encoder-sttenttoor-eeadd
teccoder-stteentioor-hdddd 2
eaccoeer-norma]liee-oooooce
deccoer-nnormaa122ooofoooo
dropout 0.4
attention-dropout 0.2
relu-dropout 0.2
weight-decay 0.0001
-abeel-smotthing 0.2
-criterion
-optimizer adam
-adam-betss (0.9, 0.98)'
-clip-norm -- -1r-scheduler inverse_sqrt
-warmup-update 4000
-warmp-initt1r 1e-7
1e-3 --min-lr 1e-9
max-tckens 4000
urdate-freq
max-epoch 100
savc-interval 10
Figure 2: Model settings for NMT training with fairseq
Finally, Table gives the BLEU scores on the TED corpus when translating into and from English for some additional languages.
Lang
->
et
15.9
14.3
10.1
7.6
16.7
8.8
fi
10.9
10.9
It
13.7
10.0
hi
17.8
21.9
mr
2.6
3.5
Table 7: BLEU scores the TED test proposed in (Qi al., 2018). NMT systems
trained on bitexts mined in Wikipedia only. No other resources were used.
15


Reviews:
[{"review_id": "ryecO-JoYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents WikiMatrix, an approach to automatically extract parallel sentences from the free text content of Wikipedia. The paper considers 1620 languages and the final dataset contains 135M parallel sentences. The language pairs are general and therefore the data does not require the use of English as a common language between two other languages.\n\nTo evaluate the quality of the extracted pairs, a neural machine translation system has been trained on them and tested on the TED dataset, obtaining good results in terms of BLEU score.\n\nThe article provides information on the system used to extract parallel sentences and opens up different directions for future investigations.\n\nThe dataset seems, from the given details, useful. However, without access to the data and, more importantly, extensive testing of it, it is difficult to say how and how much it would help the advancement of the field. For the moment it seems to be good. However, I am not really sure that this paper could be of interest to a wide audience, except for those involved in machine translation.\n\nIn general, the article describes everything at a high level, without going into the real details.\nAn example of this is on page 6, section 4.2, where the article says that its purpose is to compare different mining parameters, but I do not see any real comparison. Some words are spent for the mining threshold, but there is no real comparison, while other possible parameters are not considered at all.\n\nFor this reason, I would tend to give a low score, which does not mean that the dataset is not good. It means that the real content of the paper seems to me to be too little to be published at ICLR, since the paper only informs about the presence of this new dataset, saying that it contains a large number of sentences and seems to allow good translations based on the results of a preliminary test.\n\nTypos:\n- on page 9 \"Aragonse\"\n- on page 9, end penultimate line, the word \"for\" is repeated.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SklmxWok9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This ICLR submission deals with an strategy for the automatic extraction of parallel sentences from Wikipedia articles in 85 languages, based on multilingual sentence embeddings.\nThe review is delivered with the caveat that I am not an expert in this particulat field.\nThe paper is well written and structured, being within the scope of the conference. \nThe literature review is very up to date and overall relevant to provide an appropriate context to the investigation.\nI reckon this is a very interesting piece of work, but also that it draws too heavily on previous work from which the study is just an incremntal extension.\nMinor issues:\nAll acronyms in the text should be defined the first time they appear in the text.\n1st sentence of section 2: typo on “comparable coprora”.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HygOmHZm2H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a multi-lingual multi-way pseudo-parallel text corpus automatically extracted from Wikipedia.\n\nThe authors use a variety of pre-existing techniques applied at large scale with substantial engineering effort to extract a large number of sentence pairs in 1620 language pairs from 85 languages.\n\nIn the proposed method 1) raw sentences are extracted from a Wikipedia dump, 2) LASER sentence embeddings and language IDs are computed for each sentence, 3) for each language pair candidate sentence pairs are extracted using a FAISS approximate K-nearest neighbor index on the cosine distance between sentence embeddings, 4) sentence similarity scores are computed between the candidate pairs using the \"max margin\" criterion of Artetxe & Schwenk, 2018 and finally 5) sentence pairs are selected according to a language-pair-agnostic threshold on the similarity scores. \n\nThe extraction method is symmetric w.r.t. language directions for each language pair.\n\nStructural metadata of Wikipedia, such as cross-lingual document alignments, is deliberately not exploited (some discussion is provided but I would have preferred an empirical comparison of local vs global extraction). \n\nThe similarity threshold is determined by evaluating training corpora extracted at different thresholds on a machine translation task on De->En, De->Fr, Cs->De and Cs->Fr translation directions, evaluated on WMT newstest2014, and manually selecting the threshold based on BLEU scores. The paper also reports that combining the automatically extracted corpora with Europarl results in strong BLEU improvements over training only on Europarl. BLEU scores on TED test sets obtained using only the automatically extracted corpus are also reported. The corpus has been released.\n\nOverall the methodology presented in the paper is strong and the corpus is likely going to become a valuable tool to build machine translation systems and other multi-lingual applications. However, I am concerned that ICLR 2020 may not be the appropriate venue for this paper, as in my understanding dataset release papers are not explicitly solicited in the Call for Papers https://iclr.cc/Conferences/2020/CallForPapers . The corpus generation method is based on existing techniques, and to the extent that the engineering effort is innovative, it might not necessarily transfer well to data sources other than Wikipedia, thus limiting its broad scientific value. Therefore I suggest to submit the paper to a different venue.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1eiO93RYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper creates a large dataset for machine translation, called WikiMatrix, that contains 135M parallel sentences in 1620 language pairs from Wikipedia. The paired sentences from different languages are mined based on the sentence embeddings. Training NMT systems based on the mined dataset, and comparing with those trained based on existing dataset, the authors claim that the quality of the dataset is good. The effect of thresholding values of similarity scores for selecting parallel sentences is studied. Since the data is huge, dimension reduction and data compression techniques are used for efficient mining. The study is the first one that systematically mine for parallel sentences of Wikipedia for a large number of languages. The results are solid and the dataset is valuable for research in multilinguality.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present an approach to large scale bitext extraction from Wikipedia. This builds heavily on previous work, with the novelty being somewhat minor efficient approximate K-nearest neighbor search and language agnostic parameters such as cutoffs. These techniques have not been validated on other data sets and it is unclear how well they generalise. The major contribution of the paper is the corpus created, consisting of 85 languages, 1620 language pairs and 135M parallel sentences, of which most do not include English. This corpus is very valuable and already in use in the field, but IMO ICLR is not the right venue for this kind of publication. There were four reviews, all broadly in agreement, and some discussion with the authors. 

Decision: Reject