Cross-model Back-translated Distillation for Unsupervised Machine Translation

1 INTRODUCTION . Machine translation ( MT ) is a core task in natural language processing that involves both language understanding and generation . Recent neural approaches ( Vaswani et al. , 2017 ; Wu et al. , 2019 ) have advanced the state of the art with near human-level performance ( Hassan et al. , 2018 ) . However , they continue to rely heavily on large parallel data . As a result , the search for unsupervised alternatives using only monolingual data has been active . While Ravi & Knight ( 2011 ) and Klementiev et al . ( 2012 ) proposed various unsupervised techniques for statistical MT ( SMT ) , Lample et al . ( 2018a ; c ) established a general framework for modern unsupervised MT ( UMT ) that works for both SMT and neural MT ( NMT ) models . The framework has three main principles : model initialization , language modeling and iterative back-translation . Model initialization bootstraps the model with a knowledge prior like word-level transfer ( Lample et al. , 2018b ) . Language modeling , which takes the form of denoising auto-encoding ( DAE ) in NMT ( Lample et al. , 2018c ) , trains the model to generate plausible sentences in a language . Meanwhile , iterative back-translation ( IBT ) facilitates cross-lingual translation training by generating noisy source sentences for original target sentences . The recent approaches differ on how they apply each of these three principles . For instance , Lample et al . ( 2018a ) use an unsupervised word-translation model ( Lample et al. , 2018b ) for model initialization , while Conneau & Lample ( 2019 ) use a pretrained cross-lingual masked language model ( XLM ) . In this paper , we focus on a different aspect of the UMT framework , namely , its data diversification2 process . If we look from this view , the DAE and IBT steps of the UMT framework also perform some form of data diversification to train the model . Specifically , the noise model in the DAE process generates new , but noised , versions of the input data , which are used to train the model with a reconstruction objective . Likewise , the IBT step involves the same UMT model to create synthetic parallel pairs ( with the source being synthetic ) , which are then used to train the model . Since the NMT model is updated with DAE and IBT simultaneously , the model generates fresh translations in each back-translation step . Overall , thanks to DAE and IBT , the model gets better at translating 1Anonymized code : https : //tinyurl.com/y2ru8res . 2By diversification , we mean sentence level variations ( not expanding to other topics or genres ) . by iteratively training on the newly created and diversified data whose quality also improves over time . This argument also applies to statistical UMT , except for the lack of the DAE ( Lample et al. , 2018c ) . However , we conjecture that these diversification methods may have reached their limit as the performance does not improve further the longer we train the UMT models . In this work , we introduce a fourth principle to the standard UMT framework : Cross-model Backtranslated Distillation or CBD ( §3 ) , with the aim to induce another level of diversification that the existing UMT principles lack . CBD initially trains two UMT agents ( models ) using existing approaches . Then , one of the two agents translates the monolingual data from one language s to another t in the first level . In the second level , the generated data are back-translated from t to s by the other agent . In the final step , the synthetic parallel data created by the first and second levels are used to distill a supervised MT model . CBD is applicable to any existing UMT method and is more efficient than ensembling approaches ( Freitag et al. , 2017 ) ( §5.3 ) . In the experiments ( §4 ) , we have evaluated CBD on the WMT ’ 14 English-French , WMT ’ 16 English-German and WMT ’ 16 English-Romanian unsupervised translation tasks . CBD shows consistent improvements of 1.0-2.0 BLEU compared to the baselines in these tasks . It also boosts the performance on IWSLT ’ 14 English-German and IWSLT ’ 13 English-French tasks significantly . In our analysis , we explain with experiments why other similar variants ( §5.1 ) and other alternatives from the literature ( §5.4 ) do not work well and cross-model back-translation is crucial for our method . We further demonstrate that CBD enhances the baselines by achieving greater diversity as measured by back-translation BLEU ( §5.2 ) . 2 BACKGROUND . Ravi & Knight ( 2011 ) were among the first to propose a UMT system by framing the problem as a decipherment task that considers non-English text as a cipher for English . Nonetheless , the method is limited and may not be applicable to the current well-established NMT systems ( Luong et al. , 2015 ; Vaswani et al. , 2017 ; Wu et al. , 2019 ) . Lample et al . ( 2018a ) set the foundation for modern UMT . They propose to maintain two encoder-decoder networks simultaneously for both source and target languages , and train them via denoising auto-encoding , iterative back-translation and adversarial training . In their follow-up work , Lample et al . ( 2018c ) formulate a common UMT framework for both PBSMT and NMT with three basic principles that can be customized . The three principles are : • Initialization : A non-randomized cross- or multi-lingual initialization that represents a knowledge prior to bootstrap the UMT model . For instance , Lample et al . ( 2018a ) and Artetxe et al . ( 2019 ) use an unsupervised word-translation model MUSE ( Lample et al. , 2018b ) as initialization to promote word-to-word cross-lingual transfer . Lample et al . ( 2018c ) use a shared jointly trained sub-word ( Sennrich et al. , 2016b ) dictionary . On the other hand , Conneau & Lample ( 2019 ) use a pretrained cross-lingual masked language model ( XLM ) to initialize the unsupervised NMT model . • Language modeling : Training a language model on monolingual data helps the UMT model to generate fluent texts . The neural UMT approaches ( Lample et al. , 2018a ; c ; Conneau & Lample , 2019 ) use denoising auto-encoder training to achieve language modeling effects in the neural model . Meanwhile , the PBSMT variant proposed by Lample et al . ( 2018c ) uses the KenLM smoothed n-gram language models ( Heafield , 2011 ) . • Iterative back-translation : Back-translation ( Sennrich et al. , 2016a ) brings about the bridge between source and target languages by using a backward model that translates data from target to source . The ( source and target ) monolingual data is translated back and forth iteratively to progress the UMT model in both directions . During training , the initialization step is conducted once , while the denoising and back-translation steps are often executed in an alternating manner.3 It is worth noting that depending on different implementations , the parameters for backward and forward components may be separate ( Lample et al. , 2018a ) or shared ( Lample et al. , 2018c ; Conneau & Lample , 2019 ) . A parameter-shared cross-lingual NMT model has the capability to translate from either source or target , while a UMT system with parameter-separate models has to maintain two models . Either way , we deem a standard UMT system to be bidirectional , i.e . it is capable of translating from either source or target language . 3The KenLM language model in PBSMT ( Lample et al. , 2018c ) was kept fixed during the training process . Our proposed cross-model back-translated distillation ( CBD ) works outside this well-established framework . It employs two UMT agents to create extra diversified data apart from what existing methods already offer , rendering it a useful add-on to the general UMT framework . Furthermore , different implementations of UMT as discussed above can be plugged into the CBD system to achieve a performance boost , even for future methods that may potentially not employ the three principles . 3 CROSS-MODEL BACK-TRANSLATED DISTILLATION . Let Xs and Xt be the sets of monolingual data for languages s and t , respectively . We first train two UMT agents independently with two different parameter sets θ1 and θ2 using existing methods ( Lample et al. , 2018a ; c ; Conneau & Lample , 2019 ) .4 Since a UMT agent with parameter set θ is deemed bidirectional in our setup , we denote yt ∼ P ( ·|xs , θ ) to be the translation sample from language s to t of input sentence xs using parameters θ . In addition , for each language direction s→ t ( and vice versa for t→ s ) , we introduce a supervised MT model with parameters θ̂ that will be trained through our CBD . With these notations , we denote the following training loss L with respect to input variables θα , θβ ∈ { θ1 , θ2 } , the supervised parameters θ̂ and the learning rate η . Figure 1 also illustrates the translation flows of function L. L ( θα , θβ , θ̂ ) = −Ezs∼P ( ·|yt , θβ ) , yt∼P ( ·|xs , θα ) , xs∼Xs [ logP ( yt|xs , θ̂ ) + logP ( yt|zs , θ̂ ) ] − Ezt∼P ( ·|ys , θβ ) , ys∼P ( ·|xt , θα ) , xt∼Xt [ logP ( xt|ys , θ̂ ) + logP ( zt|ys , θ̂ ) ] ( 1 ) In the CBD strategy ( Algorithm 1 , Eq . 1 ) , at each step , each agent θ ∈ { θ1 , θ2 } samples translations from the monolingual data of both languages s and t to acquire the first level of synthetic parallel data ( xs , yt ) and ( xt , ys ) . In the second level , the other agent { θ1 , θ2 } \ θ is used to sample the translation zs of the translation yt of xs ( and similarly for zt from the translation ys of xt ) . This process is basically back-translation , but with the backward model coming from a different regime than that of the forward model . The fact that the first level agent must be different from the second level agent is crucial to achieve the desirable level of diversity in data generation . After this , we update the final model θ̂ using all the synthetic pairs ( x , y ) and ( y , z ) with maximum likelihood estimation ( MLE ) . Algorithm 1 Cross-model Back-translated Distillation : Given monolingual data Xs and Xt of languages s and t , return a s→ t UMT model with parameters θ̂ . 1 : procedure CBD ( s , t ) 2 : Train the 1st UMT agent with parameters θ1 3 : Train the 2nd UMT agent with parameters θ2 4 : Randomly initialize the parameters for MT model , θ̂ 5 : while until convergence do 6 : θ̂ ← θ̂ − η∇θ̂L ( θ1 , θ2 , θ̂ ) 7 : θ̂ ← θ̂ − η∇θ̂L ( θ2 , θ1 , θ̂ ) 8 : return θ̂ In this way , the model θ̂ first gets trained on the translated products ( x − y ) of the UMT teachers , making it as capable as the teachers . Secondly , the model θ̂ is also trained on the second-level data ( y − z ) , which is slightly different from the first-level data . Thus , this mechanism provides extra data diversification to the system θ̂ in addition to what the UMT teachers already offer , resulting in our final model outperforming the UMT baselines ( §4 ) . However , one may argue that since θ1 and θ2 are trained in a similar fashion , z will be the same as x , resulting in a duplicate pair . In our experiments , on the contrary , the back-translated dataset contains only around 14 % duplicates across different language pairs , as shown in our analysis on data diversity in §5.2.In Appendix §8.1 , we provide a more generalized version of CBD with n ( ≥ 2 ) UMT agents , where we also analyze its effectiveness on the IWSLT translation tasks . 4For neural approaches , changing the random seeds would do the trick , while PBSMT methods would need to randomize the initial embeddings and/or subsample the training data .

Summary: This paper describes a method to enhance unsupervised machine translation through data augmentation. The idea is pretty straight-forward, if not altogether intuitive, you begin by training two bidirectional (i.e.: they can translate source to target and target to source) unsupervised MT systems A and B. The tested scenarios always have A and B be identical architectures trained with different initializations. They then produce synthetic source-target pairs by first having A (source->target) translate the provided source sentence x to y, and then having B (target->source) translate y back to z. They then train supervised MT on both x,y and z,y. The same procedure can be repeated with source and target reversed. The authors show substantial (1-2) BLEU improvements with 3 different UMT systems in 5 low-data scenarios (En-Fr, Fr-En, En-De, En-Ro, Ro-En), all subsampled to 5M monolingual sentences for each language. In En-Fr and Fr-En and En-De, they are able to match reported XLM results from Conneau and Lample 2019, despite using much less data.