Machine Translation With Weakly Paired Bilingual Documents

1 INTRODUCTION . Neural Machine Translation ( NMT ) is a great success of deep learning for natural language processing . Thanks to recently developed advanced neural network architectures ( Cho et al. , 2014 ; Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ; Jean et al. , 2015 ; Vaswani et al. , 2017 ; Gehring et al. , 2017 ) , NMT has significantly outperformed statistical machine translation and reached near humanlevel performance for several language pairs ( Wu et al. , 2016 ; Hassan et al. , 2018 ) . Such breakthroughs heavily depend on the availability of large scale of bilingual sentence pairs . Taking the WMT14 English→French task as an example , NMT uses 38 million parallel sentence pairs for training ( Vaswani et al. , 2017 ) . As bilingual sentence pairs are costly to collect , the success of NMT is not fully realized on the vast majority of language pairs , especially for low-resource languages . Recently , Artetxe et al . ( 2017 ) ; Lample et al . ( 2017 ) tackle this challenge by training NMT models using only monolingual data , which achieves considerably good accuracy but still far away from that of the state-of-the-art supervised models . While it is costly to collect bilingual sentence pairs by human translation , we notice that there exist many weakly paired bilingual documents on the Web . For example , for the entity “ machine learning ” , Wikipedia has multiple articles in different languages , e.g. , the English article and German article . The two articles have very similar content , but they are not sentence-by-sentence translations , since they may be independently created by different people . Similarly , an English news in BBC and a Chinese news in China Daily talk about the same event but maybe with differences in details . Furthermore , a popular novel in different languages is usually liberal translation instead of literal translation . We call such weakly aligned documents weakly paired bilingual documents . In this paper , we explore a new direction of learning NMT models from weakly paired documents , which has several advantages . First , weakly paired documents are much easier to obtain than bilingual sentence pairs . We can obtain bilingual document pairs from Wikipedia pages , from aligned news articles on international news websites , even from books . Second , such weakly paired documents have great coverage of different languages . For example , Wikipedia covers 178 languages and most of them have paired pages to English . This means that learning translation models from paired bilingual documents are possible for many language pairs . Weakly aligned document pairs can be utilized for NMT training from two aspects . First , although such two documents are not exactly sentence-by-sentence translations , it is possible that one specific sentence in one document is the translation of one sentence in the other document . Such a sentence pair can be used as bilingual signals for model training . For example in Wikipedia , although the web structure and paragraphs are generally different for entity “ Beijing ” in English and “ Pekin ” in French , we found the first sentences of the pages are quite semantically similar , in which both of them define Beijing as the capital of China ( English : Beijing , formerly romanized as Peking , is the capital of the People ’ s Republic of China . French : Pekin , galement appele Beijing , est la capitale de la Rpublique populaire de Chine . ) The challenge is how to mine such sentence pairs from those document pairs . Second , although the sentences in two weakly paired documents are not aligned , the topics of the documents are well aligned . Such topic alignment is a strong signal that can be used to train NMT models . In this paper , we focus on Wikipedia data and propose a method to train the machine translation models by leveraging weakly paired bilingual documents from Wikipedia . The key idea of our method is to mine implicitly aligned sentence pairs and leverage topic alignment as regularization . First , we provide a simple and efficient method to mine bilingual sentence pairs from each weakly aligned document pair . We first train cross-lingual word embeddings from two monolingual corpora ( one for each language ) using MUSE ( Conneau et al. , 2017 ) , and then use weighted average of embeddings of words in a sentence as the sentence embedding . With the sentence embedding , we select the sentence pairs with large cosine similarity as bilingual sentence pairs and use them as supervised signals to train NMT models . Second , many previous works suggest that the word distribution can be used to well characterize the topic of the document ( Petterson et al. , 2010 ; Du et al. , 2015 ; Funatsu et al. , 2014 ; Pedrosa et al. , 2016 ; Chemudugunta et al. , 2007 ) . To leverage the topic consistency between two weakly paired documents , we minimize the KL-divergence of the word distributions between the ground-truth document and the model-generated document . Taking Wikipedia corpus as the training data , we test our method on the widely used WMT16 German↔English and WMT13 Spanish↔English translation tasks . Our method achieves 24.1/30.3 BLEU points for WMT16 German↔English translations and 28.1/27.6 BLEU points for WMT13 Spanish↔English translations , outperforming the state-of-the-art unsupervised method by more than 5 BLEU points and reduce the gap between unsupervised translation and supervised translation up to 50 % . 2 RELATED WORK . Using monolingual data to boost the machine translation performance has attracted a lot of attention in the literature ( Gulcehre et al. , 2015 ; Sennrich et al. , 2016a ; Zhang & Zong , 2016 ; Wu et al. , 2018 ; He et al. , 2016 ) , especially when the bilingual supervision is limited . Sennrich et al . ( 2016a ) propose the back-translation approach which is a popular and effective way to augment the training bilingual sentence pairs with the target-side monolingual data . He et al . ( 2016 ) leverage both the source-side and target-side monolingual data in a dual learning framework . However , these methods still require a relatively large amount of labeled bilingual data . Recently , Lample et al . ( 2017 ) and Artetxe et al . ( 2017 ) make an initial study of unsupervised machine translation , in which the model is trained from the monolingual data only . Lample et al . ( 2017 ) leverage two key components to learn translation models from monolingual data : 1 ) suitable initialization of the translation model by cross-lingual word embeddings , 2 ) denoising auto-encoder as language model and reconstruction loss based on translation-back-translation . Both works leverage monolingual sentences only but do not leverage rich weakly paired documents from Web . We study how to leverage document pairs for learning translation without sentence pairs and implement the proposed approach using Wikipedia data . Leveraging the free online Wikipedia database as an additional source to improve the natural language processing tasks has also attracted interest in recent years . For example , Conneau et al . ( 2017 ) show that word translation can be effectively learned based on the embedding trained from Wikipedia . This embedding further becomes one of the key components for unsupervised machine translation . Different from using Wikipedia to train warm-start word embeddings , we aim to leverage more and stronger signals from such weakly paired documents to train translation model without parallel sentence pairs . Besides , Hálek et al . ( 2011 ) use the category information in Wikipedia corpus to improve the translation of named entities . Drexler et al . ( 2014 ) incorporate language models from target language documents that are comparable to the source documents in Wikipedia pages to improve the document translation . There are several works aim at extracting potential sentence pairs from comparable corpus , but most of them rely on a set of bilingual sentence pairs to train a model and use this model to select sentence pairs . For example , Adafre & De Rijke ( 2006 ) and Yasuda & Sumita ( 2008 ) use a strong machine translation system to obtain a rough translation of a given page in one language into another , and then calculate word overlap or BLEU score between sentences as measure . Smith et al . ( 2010 ) and Munteanu & Marcu ( 2005 ) develop a ranking model/binary classifier to learn how likely a sentence in target language is a translation of the source language using parallel corpora . However , in our scenario , we have no bilingual sentence pairs available . That is , we have no bilingual sentence pairs to train such a model to further select new sentence pairs . Our work is also related to document-to-document translation ( Tu et al. , 2018 ) , but with different goals and settings . The goal of document-to-document translation is to enhance sentence-tosentence translation with stronger signals beyond sentence pairs by using richer inputs , e.g. , the topic information from the document that contains this sentence . During training , it takes one sentence as well as the cross-sentence ( document-level ) information as input and predicts the ground-truth translation sentence in other languages . Therefore , training a document-to-document translation model requires bilingual sentence pairs and their surrounding contexts . In our scenario , our goal is to learn a sentence-to-sentence translation model with weaker signals than sentence pairs . We target to extract useful information from weakly paired documents to train a translation model without human-labeled bilingual data . 3 OUR METHOD . We develop two ways to leverage weakly paired documents : mining implicitly aligned sentence pairs from the document pairs and aligning the topic distributions of two documents in a weakly aligned pair . Before diving into details , we first introduce some notations . Denote D = { ( dXi , dYi ) } , i ∈ { 1 , 2 , ... , M } as the set of weakly paired documents , in which document dXi is aligned to d Y i . For example , d X i and d Y i are two cross-lingual linked Wikipedia pages . Denote nXi and n Y i as the number of sentences in document d X i and d Y i respectively . Note that usually nXi 6= nYi . Without any confusions , we denote x as a sentence in languageX and y as a sentence in language Y . We denote enc as the encoder for language X and Y , which maps a sentence x or y into a sequence of real vectors using parameter θenc . We use dec with parameter θdec as the decoder , which takes the encoded vectors and target language tag ( X or Y ) as inputs and outputs a probability distribution over sentences in the target language . Let θ denote all the parameters of the translation model . Similar to Artetxe et al . ( 2017 ) ; Lample et al . ( 2017 ) , such a model can handle both X → Y translation and Y → X translation . 3.1 MINING IMPLICITLY ALIGNED SENTENCE PAIRS . Different language versions of Wikipedia pages about the same entity/event are usually created by different people speaking different native languages , and therefore most sentences in two weakly aligned documents are not aligned . Even though , there is still a small chance that some bilingual sentences are aligned , and we try to mine such implicitly aligned sentence pairs and use them as supervision for NMT model training . Imankulova et al . ( 2017 ) extract bilingual sentence pairs from Wikipedia using a well-trained translation model learned from supervised sentence pairs . This method does not work for us since we do not have aligned bilingual sentence pairs . Instead , our idea is to compute the similarity of two bilingual sentences using their cross-lingual sentence embeddings and choose the pairs with large similarity as aligned bilingual sentence pairs . Sentence embedding is widely used to measure textual similarity in text classification tasks ( Arora et al. , 2017 ; Le & Mikolov , 2014 ; Wieting et al. , 2015 ) . Arora et al . ( 2017 ) compute the weighted average of the word embedding in one sentence where the weight depends on word frequency and then project away the weighted average sentence embeddings from their first principal component . This method achieves good performance on a range of monolingual textual similarity task . We extend their method from monolingual sentence embedding to cross-lingual sentence embedding , given the cross-lingual word emebddings are pre-trained using MUSE ( Conneau et al. , 2017 ) . The detailed method is described as follows . For each word w , we denote ew as the word embedding trained from MUSE ( Conneau et al. , 2017 ) , p ( w ) as the estimated frequency from another document and a as a predefined parameter to calculate the weight of word embedding . We denote ês as the weighted average sentence embedding of sentence s and E as the embedding matrix of all the sentences over the monolingual corpus . Then we remove the first principal components u1 of E for every weighted average sentence embedding ês and use the resulting embedding es as the final sentence embedding , i.e. , ês = ∑ w∈s a a+ p ( w ) ew , ( 1 ) u1 → PCA ( E ) , ( 2 ) es = ês − u1uT1 ês . ( 3 ) Based on the sentence embedding , we estimate the similarity between two sentences in different languages by their cosine similarity sim ( sX , sY ) = < esX , esY > ‖esX ‖‖esY ‖ . For each weakly aligned document pair ( dXi , d Y i ) , we have n X i × nYi pairs of sentences and form a bipartite graph between sentences in two documents where the weight of an edge between two cross-lingual sentences is their cosine similarity score . The goal is to find the most confident edges ( sentence pairs ) from this weighted bipartite graph . We adopt a greedy selection approach with two constraints : The first constraint is that the weight of a selected edge must be larger than threshold c1 , which is to ensure that the two sentences are similar enough . The second constraint is that the weight of a selected edge must be larger than the weights of all other edges connected to these two nodes ( sentences ) by threshold c2 . This ensures that the pair we selected is unique enough . Denote S = { ( sXj , sYj ) } as the set of selected sentence pairs . We use those pairs as supervision for model training , i.e. , minimizing the negative log-likelihood as below . Lp ( S ; θ ) = 1 |S| ∑ ( sX , sY ) ∈S logPX→Y ( s Y |sX ; θ ) + 1 |S| ∑ ( sX , sY ) ∈S logPY→X ( s X |sY ; θ ) ( 4 )

Summary: This paper proposes a method to train a machine translation system using weakly paired bilingual documents from Wikipedia. A pair of sentences from a weak document pair are used as training data if their cosine similarity exceeds c1, and the similarity between this sentence pair is c2 greater than any other pair in the documents, under sentence representations formed from word embeddings trained with MUSE. The neural translation model learns to translate from language X to Y, and from Y to X using the same encoder and decoder parameters, but the decoder is aware of the intended target language given an embedding of the intended language. The model is also trained to minimise the KL divergence between the distribution of terms in the target language document and the distribution of terms in the current model output. The model also uses the denoising autoencoding and reconstruction objectives of Lample et al. (2017). The results show improvements over the Lample et al. (2017) and that performance is heavily dependent on the number of sentences extracted from the weakly aligned documents.