Incorporating BERT into Neural Machine Translation

1 INTRODUCTION . Recently , pre-training techniques , like ELMo ( Peters et al. , 2018 ) , GPT/GPT-2 ( Radford et al. , 2018 ; 2019 ) , BERT ( Devlin et al. , 2019 ) , cross-lingual language model ( briefly , XLM ) ( Lample & Conneau , 2019 ) , XLNet ( Yang et al. , 2019b ) and RoBERTa ( Liu et al. , 2019 ) have attracted more and more attention in machine learning and natural language processing communities . The models are first pre-trained on large amount of unlabeled data to capture rich representations of the input , and then applied to the downstream tasks by either providing context-aware embeddings of an input sequence ( Peters et al. , 2018 ) , or initializing the parameters of the downstream model ( Devlin et al. , 2019 ) for fine-tuning . Such pre-training approaches lead to significant improvements on natural language understanding tasks . Among them , BERT is one of the most powerful techniques that inspires lots of variants like XLNet , XLM , RoBERTa and achieves state-of-the-art results for many language understanding tasks including reading comprehension , text classification , etc ( Devlin et al. , 2019 ) . Neural Machine Translation ( NMT ) aims to translate an input sequence from a source language to a target language . An NMT model usually consists of an encoder to map an input sequence to hidden representations , and a decoder to decode hidden representations to generate a sentence in the target language . Given that BERT has achieved great success in language understanding tasks , a question worthy studying is how to incorporate BERT to improve NMT . Due to the computation resource limitation , training a BERT model from scratch is unaffordable for many researchers . Thus , we focus on the setting of leveraging a pre-trained BERT model ( instead of training a BERT model from scratch ) for NMT . ∗This work is conducted at Microsoft Research Asia . The first two authors contributed equally to this work . Given that there is limited work leveraging BERT for NMT , our first attempt is to try two previous strategies : ( 1 ) using BERT to initialize downstream models and then fine-tuning the models , and ( 2 ) using BERT as context-aware embeddings for downstream models . For the first strategy , following Devlin et al . ( 2019 ) , we initialize the encoder of an NMT model with a pre-trained BERT model , and then finetune the NMT model on the downstream datasets . Unfortunately , we did not observe significant improvement . Using a pre-trained XLM ( Lample & Conneau , 2019 ) model , a variant of BERT for machine translation , to warm up an NMT model is another choice . XLM has been verified to be helpful for WMT ’ 16 Romanian-to-English translation . But when applied to a language domain beyond the corpus for training XLM ( such as IWSLT dataset ( Cettolo et al. , 2014 ) , which is about spoken languages ) or when large bilingual data is available for downstream tasks , no significant improvement is observed neither . For the second strategy , following the practice of ( Peters et al. , 2018 ) , we use BERT to provide context-aware embeddings for the NMT model . We find that this strategy outperforms the first one ( please refer to Section 3 for more details ) . This motivates us to go along this direction and design more effective algorithms . We propose a new algorithm , BERT-fused model , in which we exploit the representation from BERT by feeding it into all layers rather than served as input embeddings only . We use the attention mechanism to adaptively control how each layer interacts with the representations , and deal with the case that BERT module and NMT module might use different word segmentation rules , resulting in different sequence ( i.e. , representation ) lengths . Compared to standard NMT , in addition to BERT , there are two extra attention modules , the BERT-encoder attention and BERT-decoder attention . An input sequence is first transformed into representations processed by BERT . Then , by the BERTencoder attention module , each NMT encoder layer interacts with the representations obtained from BERT and eventually outputs fused representations leveraging both BERT and the NMT encoder . The decoder works similarly and fuses BERT representations and NMT encoder representations . We conduct 14 experiments on various NMT tasks to verify our approach , including supervised , semi-supervised and unsupervised settings . For supervised NMT , we work on five tasks of IWSLT datasets and two WMT datasets . Specifically , we achieve 36.11 BLEU score on IWSLT ’ 14 Germanto-English translation , setting a new record on this task . We also work on two document-level translations of IWSLT , and further boost the BLEU score of German-to-English translation to 36.69 . On WMT ’ 14 datasets , we achieve 30.75 BLEU score on English-to-German translation and 43.78 on English-to-French translation , significantly better over the baselines . For semi-supervised NMT , we boost BLEU scores of WMT ’ 16 Romanian-to-English translation with back translation ( Sennrich et al. , 2016b ) , a classic semi-supervised algorithm , from 37.73 to 39.10 , achieving the best result on this task . Finally , we verify our algorithm on unsupervised English↔French and unsupervised English↔Romanian translations and also achieve state-of-the-art results . 2 BACKGROUND AND RELATED WORK . We briefly introduce the background of NMT and review current pre-training techniques . NMT aims to translate an input sentence from the source language to the target one . An NMT model usually consists of an encoder , a decoder and an attention module . The encoder maps the input sequence to hidden representations and the decoder maps the hidden representations to the target sequence . The attention module is first introduced by Bahdanau et al . ( 2015 ) , which is used to better align source words and target words . The encoder and decoder can be specialized as LSTM ( Hochreiter & Schmidhuber , 1997 ; Sutskever et al. , 2014 ; Wu et al. , 2016 ) , CNN ( Gehring et al. , 2017 ) and Transformer ( Vaswani et al. , 2017 ) . A Transformer layer consists of three sublayers , a self-attention layer that processes sequential data taking the context of each timestep into consideration , an optional encoder-decoder attention layer that bridges the input sequence and target sequence which exists in decoder only , and a feed-forward layer for non-linear transformation . Transformer achieves the state-of-the-art results for NMT ( Barrault et al. , 2019 ) . In this work , we will use Transformer as the basic architecture of our model . Pre-training has a long history in machine learning and natural language processing ( Erhan et al. , 2009 ; 2010 ) . Mikolov et al . ( 2013 ) and Pennington et al . ( 2014 ) proposed to use distributional representations ( i.e. , word embeddings ) for individual words . Dai & Le ( 2015 ) proposed to train a language model or an auto-encoder with unlabeled data and then leveraged the obtained model to finetune downstream tasks . Pre-training has attracted more and more attention in recent years and achieved great improvements when the data scale becomes large and deep neural networks are employed . ELMo was proposed in Peters et al . ( 2018 ) based on bidirectional LSTMs and its pre-trained models are fed into downstream tasks as context-aware inputs . In GPT ( Radford et al. , 2018 ) , a Transformer based language model is pre-trained on unlabeled dataset and then finetuned on downstream tasks . BERT ( Devlin et al. , 2019 ) is one of the widely adopted pre-training approach for model initialization . The architecture of BERT is the encoder of Transformer ( Vaswani et al. , 2017 ) . Two kinds of objective functions are used in BERT training : ( 1 ) Masked language modeling ( MLM ) , where 15 % words in a sentence are masked and BERT is trained to predict them with their surrounding words . ( 2 ) Next sentence prediction ( NSP ) : Another task of pre-training BERT is to predict whether two input sequences are adjacent . For this purpose , the training corpus consists of tuples ( [ cls ] , input 1 , [ sep ] , input 2 , [ sep ] ) , with learnable special tokens [ cls ] to classify whether input 1 and input 2 are adjacent and [ sep ] to segment two sentences , and with probability 50 % , the second input is replaced with a random input . Variants of BERT have been proposed : In XLM ( Lample & Conneau , 2019 ) , the model is pre-trained based on multiple languages and NSP task is removed ; in RoBERTa ( Liu et al. , 2019 ) , more unlabeled data is leveraged without NSP task neither ; in XLNet ( Yang et al. , 2019b ) , a permutation based modeling is introduced . 3 A PRELIMINARY EXPLORATION . While a few pieces of work ( Lample & Conneau , 2019 ; Song et al. , 2019 ) design specific pretraining methods for NMT , they are time and resource consuming given that they need to pre-train large models from scratch using large-scale data , and even one model for each language pair . In this work , we focus on the setting of using a pre-trained BERT model . Detailed model download links can be found in Appendix D. Considering that pre-trained models have been utilized in two different ways for other natural language tasks , it is straightforward to try them for NMT . Following previous practice , we make the following attempts . ( I ) Use pre-trained models to initialize the NMT model . There are different implementations for this approach . ( 1 ) Following ( Devlin et al. , 2019 ) , we initialize the encoder of an NMT model with a pretrained BERT . ( 2 ) Following ( Lample & Conneau , 2019 ) , we initialize the encoder and/or decoder of an NMT model with XLM . ( II ) Use pre-trained models as inputs to the NMT model . Inspired from ( Peters et al. , 2018 ) , we feed the outputs of the last layer of BERT to an NMT model as its inputs . We conduct experiments on the IWSLT ’ 14 English→German translation , a widely adopted dataset for machine translation consisting of 160k labeled sentence pairs . We choose Transformer ( Vaswani et al. , 2017 ) as the basic model architecture with transformer iwslt de en configuration ( a six-layer model with 36.7M parameters ) . The translation quality is evaluated by BLEU ( Papineni et al. , 2002 ) score ; the larger , the better . Both BERTbase and XLM models are pre-trained and we get them from the Web . More details about the experimental settings are included in Appendix A.2 . The results are shown in Table 1 . We have several observations : ( 1 ) Using BERT to initialize the encoder of NMT can only achieve 27.14 BLEU score , which is even worse than standard Transformer without using BERT . That is , simply using BERT to warm up an NMT model is not a good choice . ( 2 ) Using XLM to initialize the encoder or decoder respectively , we get 28.22 or 26.13 BLEU score , which does not outperform the baseline . If both modules are initialized with XLM , the BLEU score is boosted to 28.99 , slightly outperforming the baseline . Although XLM achieved great success on WMT ’ 16 Romanian-to-English , we get limited improvement here . Our conjecture is that the XLM model is pre-trained on news data , which is out-of-domain for IWSLT dataset mainly about spoken languages and thus , leading to limited improvement . ( 3 ) When using the output of BERT as context-aware embeddings of the encoder , we achieve 29.67 BLEU , much better than using pretrained models for initialization . This shows that leveraging BERT as a feature provider is more effective in NMT . This motivates us to take one step further and study how to fully exploit such features provided by pre-trained BERT models .

Summary: This paper explores the use of BERT to improve Neural Machine Translation (NMT) both in supervised, semi-supervised and unsupervised settings. The authors first show that using BERT to initialize the encoder and/or the decoder does not bring any clear improvement, while using it as a feature extractor performs better. Based on this finding, the authors propose a new approach to integrate BERT in NMT, named BERT-fused NMT, which incorporates BERT representations from the input sequence into the encoder and decoder attention mechanisms.