WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

To get an indication on the quality of the extracted bitexts , we train neural MT baseline systems on the mined data only for 1886 languages pairs , and evaluate them on the TED corpus , achieving strong BLEU scores for many language pairs . The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English . 1 INTRODUCTION . Most of the current approaches in Natural Language Processing ( NLP ) are data-driven . The size of the resources used for training is often the primary concern , but the quality and a large variety of topics may be equally important . Monolingual texts are usually available in huge amounts for many topics and languages . However , multilingual resources , typically sentences in two languages which are mutual translations , are more limited , in particular when the two languages do not involve English . An important source of parallel texts are international organizations like the European Parliament ( Koehn , 2005 ) or the United Nations ( Ziemski et al. , 2016 ) . These are professional human translations , but they are in a more formal language and tend to be limited to political topics . There are several projects relying on volunteers to provide translations for public texts , e.g . news commentary ( Tiedemann , 2012 ) , OpensubTitles ( Lison & Tiedemann , 2016 ) or the TED corpus ( Qi et al. , 2018 ) Wikipedia is probably the largest free multilingual resource on the Internet . The content of Wikipedia is very diverse and covers many topics . Articles exist in more than 300 languages . Some content on Wikipedia was human translated from an existing article into another language , not necessarily from or into English . Eventually , the translated articles have been later independently edited and are not parallel any more . Wikipedia strongly discourages the use of unedited machine translation,2 but the existence of such articles can not be totally excluded . Many articles have been written independently , but may nevertheless contain sentences which are mutual translations . This makes Wikipedia a very appropriate resource to mine for parallel texts for a large number of language pairs . To the best of our knowledge , this is the first work to process the entire Wikipedia and systematically mine for parallel sentences in all language pairs . We hope that this resource will be useful for several research areas and enable the development of NLP applications for more languages . In this work , we build on a recent approach to mine parallel texts based on a distance measure in a joint multilingual sentence embedding space ( Schwenk , 2018 ; Artetxe & Schwenk , 2018b ) . For this , we use the freely available LASER toolkit3 which provides a language agnostic sentence encoder which was trained on 93 languages ( Artetxe & Schwenk , 2018a ) . We approach the computational 1Anonymized for review 2https : //en.wikipedia.org/wiki/Wikipedia : Translation 3https : //github.com/facebookresearch/LASER challenge to mine in almost six hundred million sentences by using fast indexing and similarity search algorithms . The paper is organized as follows . In the next section , we first discuss related work . We then summarize the underlying mining approach . Section 4 describes in detail how we applied this approach to extract parallel sentences from Wikipedia in 1620 language pairs . To asses the quality of the extracted bitexts , we train NMT systems for a subset of language pairs and evaluate them on the TED corpus ( Qi et al. , 2018 ) for 45 languages . These results are presented in section 5 . The paper concludes with a discussion of future research directions . 2 RELATED WORK . There is a large body of research on mining parallel sentences in collections of monolingual texts , usually named “ comparable coprora ” . Initial approaches to bitext mining have relied on heavily engineered systems often based on metadata information , e.g . ( Resnik , 1999 ; Resnik & Smith , 2003 ) . More recent methods explore the textual content of the comparable documents . For instance , it was proposed to rely on cross-lingual document retrieval , e.g . ( Utiyama & Isahara , 2003 ; Munteanu & Marcu , 2005 ) or machine translation , e.g . ( Abdul-Rauf & Schwenk , 2009 ; Bouamor & Sajjad , 2018 ) , typically to obtain an initial alignment that is then further filtered . In the shared task for bilingual document alignment ( Buck & Koehn , 2016 ) , many participants used techniques based on n-gram or neural language models , neural translation models and bag-of-words lexical translation probabilities for scoring candidate document pairs . The STACC method uses seed lexical translations induced from IBM alignments , which are combined with set expansion operations to score translation candidates through the Jaccard similarity coefficient ( Etchegoyhen & Azpeitia , 2016 ; Azpeitia et al. , 2017 ; 2018 ) . Using multilingual noisy web-crawls such as ParaCrawl4 for filtering good quality sentence pairs has been explored in the shared tasks for high resource ( Koehn et al. , 2018 ) and low resource ( Koehn et al. , 2019 ) languages . In this work , we rely on massively multilingual sentence embeddings and margin-based mining in the joint embedding space , as described in ( Schwenk , 2018 ; Artetxe & Schwenk , 2018b ; a ) . This approach has also proven to perform best in a low resource scenario ( Chaudhary et al. , 2019 ; Koehn et al. , 2019 ) . Closest to this approach is the research described in España-Bonet et al . ( 2017 ) ; Hassan et al . ( 2018 ) ; Guo et al . ( 2018 ) ; Yang et al . ( 2019 ) . However , in all these works , only bilingual sentence representations have been trained . Such an approach does not scale to many languages , in particular when considering all possible language pairs in Wikipedia . Finally , related ideas have been also proposed in Bouamor & Sajjad ( 2018 ) or Grégoire & Langlais ( 2017 ) . However , in those works , mining is not solely based on multilingual sentence embeddings , but they are part of a larger system . To the best of our knowledge , this work is the first one that applies the same mining approach to all combinations of many different languages , written in more than twenty different scripts . Wikipedia is arguably the largest comparable corpus . One of the first attempts to exploit this resource was performed by Adafre & de Rijke ( 2006 ) . An MT system was used to translate Dutch sentences into English and to compare them with the English texts . This method yielded several hundreds of Dutch/English parallel sentences . Later , a similar technique was applied to the Persian/English pair ( Mohammadi & GhasemAghaee , 2010 ) . Structural information in Wikipedia such as the topic categories of documents was used in the alignment of multilingual corpora ( Otero & López , 2010 ) . In another work , the mining approach of Munteanu & Marcu ( 2005 ) was applied to extract large corpora from Wikipedia in sixteen languages ( Smith et al. , 2010 ) . Otero et al . ( 2011 ) measured the comparability of Wikipedia corpora by the translation equivalents on three languages Portuguese , Spanish , and English . Patry & Langlais ( 2011 ) came up with a set of features such as Wikipedia entities to recognize parallel documents , and their approach was limited to a bilingual setting . Tufis et al . ( 2013 ) proposed an approach to mine parallel sentences from Wikipedia textual content , but they only considered high-resource languages , namely German , Spanish and Romanian paired with English . Tsai & Roth ( 2016 ) grounded multilingual mentions to English wikipedia by training cross-lingual embeddings on twelve languages . Gottschalk & Demidova ( 2017 ) searched for parallel text passages in Wikipedia by comparing their named entities and time expressions . Finally , Aghaebrahimian ( 2018 ) propose an approach based on bilingual BiLSTM sentence encoders to mine German , French and Persian parallel texts with English . Parallel data consisting of aligned 4http : //www.paracrawl.eu/ Wikipedia titles have been extracted for twenty-three languages5 . Since Wikipedia titles are rarely entire sentences with a subject , verb and object , it seems that only modest improvements were observed when adding this resource to the training material of NMT systems . We are not aware of other attempts to systematically mine for parallel sentences in the textual content of Wikipedia for a large number of languages . 3 DISTANCE-BASED MINING APPROACH . The underling idea of the mining approach used in this work is to first learn a multilingual sentence embedding , i.e . an embedding space in which semantically similar sentences are close independently of the language they are written in . This means that the distance in that space can be used as an indicator whether two sentences are mutual translations or not . Using a simple absolute threshold on the cosine distance was shown to achieve competitive results ( Schwenk , 2018 ) . However , it has been observed that an absolute threshold on the cosine distance is globally not consistent , e.g . ( Guo et al. , 2018 ) . The difficulty to select one global threshold is emphasized in our setting since we are mining parallel sentences for many different language pairs . 3.1 MARGIN CRITERION . The alignment quality can be substantially improved by using a margin criterion instead of an absolute threshold ( Artetxe & Schwenk , 2018b ) . In that work , the margin between two candidate sentences x and y is defined as the ratio between the cosine distance between the two sentence embeddings , and the average cosine similarity of its nearest neighbors in both directions : margin ( x , y ) = cos ( x , y ) ∑ z∈NNk ( x ) cos ( x , z ) 2k + ∑ z∈NNk ( y ) cos ( y , z ) 2k ( 1 ) where NNk ( x ) denotes the k unique nearest neighbors of x in the other language , and analogously for NNk ( y ) . We used k = 4 in all experiments . We follow the “ max ” strategy as described in ( Artetxe & Schwenk , 2018b ) : the margin is first calculated in both directions for all sentences in language L1 and L2 . We then create the union of these forward and backward candidates . Candidates are sorted and pairs with source or target sentences which were already used are omitted . We then apply a threshold on the margin score to decide whether two sentences are mutual translations or not . Note that with this technique , we always get the same aligned sentences , independently of the mining direction , e.g . searching translations of French sentences in a German corpus , or in the opposite direction . The reader is referred to Artetxe & Schwenk ( 2018b ) for a detailed discussion with related work . The complexity of a distance-based mining approach is O ( N ×M ) , where N and M are the number of sentences in each monolingual corpus . This makes a brute-force approach with exhaustive distance calculations intractable for large corpora . Margin-based mining was shown to significantly outperform the state-of-the-art on the shared-task of the workshop on Building and Using Comparable Corpora ( BUCC ) ( Artetxe & Schwenk , 2018b ) . The corpora in the BUCC corpus are rather small : at most 567k sentences . The languages with the largest Wikipedia are English and German with 134M and 51M sentences , respectively , after pre-processing ( see Section 4.1 for details ) . This would require 6.8×1015 distance calculations.6 We show in Section 3.3 how to tackle this computational challenge .

Summary: The paper creates a large dataset for machine translation, called WikiMatrix, that contains 135M parallel sentences in 1620 language pairs from Wikipedia. The paired sentences from different languages are mined based on the sentence embeddings. Training NMT systems based on the mined dataset, and comparing with those trained based on existing dataset, the authors claim that the quality of the dataset is good. The effect of thresholding values of similarity scores for selecting parallel sentences is studied. Since the data is huge, dimension reduction and data compression techniques are used for efficient mining. The study is the first one that systematically mine for parallel sentences of Wikipedia for a large number of languages. The results are solid and the dataset is valuable for research in multilinguality.