Incorporating BERT into Neural Machine Translation
Authors: Jinhua Zhu
Yingce Xia
Lijun Wu
Di He
Tao Qin
Wengang Zhou
Houqiang Li
Tieyan Liu
Venue:  (2020)
Keywords: BERT
Neural Machine Translation

Abstract: The recently proposed BERT (Devlin et al., 2019) has shown great power on a variety of natural language understanding tasks, such as text classification, reading comprehension, etc. However, how to effectively apply BERT to neural machine translation (NMT) lacks enough exploration. While BERT is more commonly used as fine-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for fine-tuning. This motivates us to think how to better leverage BERT for NMT along this direction. We propose a new algorithm named BERT-fused model, in which we first use BERT to extract representations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mechanisms. We conduct experiments on supervised (including sentence-level and document-level translations), semi-supervised and unsupervised machine translation, and achieve state-of-the-art results on seven benchmark datasets. Our code is available at https://github.com/bert-nmt/bert-nmt

Published as a conference paper at ICLR 2020
INCORPORATING BERT INTO NEURAL MACHINE TRANSLATION
Lijun Wa*, Di He', Jinhua Zhu Yingee Xia Lijun
Tao Qin', Wengang Zhou', Houqiang Li Tie Van Liu?
ICAS Key Laboratory of GIPAS, EEIS Department, University of Science and Technology of China;
Microsoft Research;
8 Sun Yat-sen University;
4 Key Laboratory of Machine Perception (MOE), School of EECS, Peking University
"teslazhu@mail ust.c edu. en, {zhwg, Buste.
Pyingce www.gmaii.eemi..........aa.iaiea com, ftaoqin, wwiiuu/mmicrooof.. com
Swulijun3emaille sysu. edu. *di.he@pku.een eduu. cn
ABSTRACT
The recently proposed BERT (Devlin et al., 2019) has shown great power on va- riety of natural language understanding tasks, such as text classification, reading comprehension, etc. However, how to effectively apply BERT to neural machine translation (NMT) lacks enough exploration. While BERT is more commonly used as fine-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as con- textual embedding is better than using for fine-tuning. This motivates us to think how to better leverage BERT for NMT along this direction. We propose new algorithm named BERT-fused model, in which we first use BERT to extract rep- resentations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mech- anisms. We conduct experiments on supervised (including sentence-level and document-leva translations), semi-supervised and unsupervised machine trans- lation, and achieve state-of-the- results on seven benchmark datasets. Our code is available atthtps: /ttp:::////ttttbbt http:////gthaabb.o/bbbbrtrt nmt bbett- -mmt
INTRODUCTION
Recently, pre-training techniques, like ELMo (Peters et al. 2018), GPT/GPT-2 (Radford et al. 2018 2019). BERT (Devlin et al., 2019). longulinggul language model (briefly, XLM) (Lample & Con- neau 2019). XLNet (Yang al. (019bb and RoBERTa (Liu al. 2019) have attracted more and more attention in machine learning and natural language processing communities. The models are first pre-trained on large amount of unlabeled data to capture rich representations of the input, and then applied to the downstream tasks by either providing context-aware embeddings of an input se- quence (Peters al., 2018). or initializing the parameters of the downstream model (Devlin al. 2019) for fine-tuning. Such pre-training approaches lead to significant improvements on natural lan- guage understanding tasks. Among them, BERT is one of the most powerful techniques that inspires lots of variants like XLNet, XLM, RoBERTa and achieves state-of-the-ar results for many language understanding tasks including reading comprehension, text classification, etc (Devlin et al. 2019).
Neural Machine Translation (NMT) aims to translate an input sequence from source language to target language. An NMT model usually consists of an encoder to map an input sequence to hidden representations, and decoder to decode hidden representations to generate sentence in the target language. Given that BERT has achieved great success in language understanding tasks, a question worthy studying is how to incorporate BERT to improve NMT. Due to the computation resource limitation, training a BERT model from scratch is unaffordable for many researchers. Thus, we focus on the setting of leveraging a pre-trained BERT model (instead of training BERT model from scratch) for NMT.
TThis work is conducted Microsoft Research Asia. The first two authors contributed equally to this work.

---

Published as a conference paper at ICLR 2020
Given that there is limited work leveraging BERT for NMT, our first attempt is to try two previous strategies: (1) using BERT to initialize downstream models and then fine-tuning the models, and (2) using BERT as context-aware embeddings for downstream models. For the first strategy, follow- ing Devlin et al. 2019), we initialize the encoder of an NMT model with a pre-trained BERT model, and then finetune the NMT model the downstream datasets. Unfortunately, significant improvement. Using a pre-trained XLM (Lample & Conneau (019) model, variant of BERT for machine translation, to warm up an NMT model another choice. XLM has been ver- ified to be helpful for WMT 16 Romanian-to Centiiiinntt--nliis to-Ellilsh translation. But when applied to a language domain beyond the corpus for training XLM (such as IWSLT dataset (Cettolo et al., 2014), which is about spoken languages) or when large bilingual data is available for downstream tasks, no sig- nificant improvement is observed neither. For the second strategy, following the practice of (Peters) er al. 2018). we use BERT to provide context-aware embeddings for the NMT model. We find that this strategy outperforms the first one (please refer to Section 3 for more details). This motivates us to go along this direction and design more effective algorithms.
did not observe
We propose a new algorithm, BERT-fused model, in which we exploit the representation from BERT by feeding it into all layers rather than served as input embeddings only. We use the attention mechanism to adaptively control how each layer interacts with the representations, and deal with the case that BERT module and NMT module might use different word segmentation rules, resulting in different sequence (i.e., representation) lengths. Compared to standard NMT, in addition to BERT, there are two extra attention modules, the BERT-encoder attention and BERT-decoder attention. An input sequence is first transformed into representations processed by BERT. Then, by the BERT- encoder attention module, each NMT encoder layer interacts with the representations obtained from BERT and eventually outputs fused representations leveraging both BERT and the NMT encoder. The decoder works similarly and fuses BERT representations and NMT encoder representations.
We conduct 14 experiments on various NMT tasks to verify our approach, including supervised, semi-supervised and unsupervised settings. For supervised NMT, we work on five tasks of IWSLT datasets and two WMT datasets. Specifically, we achieve 36. 11 BLEU score on IWSLT 14 German- to-English translation, setting a new record on this task. We also work on two document-leww translations of IWSLT, and further boost the BLEU score of Germanntoo-ngllis translation to 36.69. On WMT' 14 datasets, we achieve 30.75 BLEU score on English-to-Germall translation and 43. 43.78 on English-to - translation. significantly better over the baselines. For semi-supervised NMT, boost BLEU scores of WMT' 16 Romanian-to- to--nglish translation with back translation (Sennrich) et al. 2016b), classic semi-supervised algorithm, from 37.73 to 39.10, achieving the best result on this task. Finally, we verify our algorithm on unsupervised Engeish and unsupervised Englisht+RRoaniee translations and also achieve sfate-of-the-arr results.
2 BACKGROUND AND RELATED WORK
We briefly introduce the background of NMT and review current pre- pre-training techniques.
NMT aims to translate an input sentence from the source language to the target one. An NMT model usually consists of encoder, decoder and an attention module. The encoder maps the input sequence to hidden representations and the decoder maps the hidden representations to the target sequence. The attention module is first introduced by [Bahdanau et al. (2015), which used to better align source words and target words. The encoder and decoder can be specialized as LSTM (Hochreiter & Schmidhuber] 1997, (Sutskever et al. 2014 Wu et al. 2016), CNN (Gehring et al. 2017) and Transformer (Vaswani et al., 2017)) A Transformer layer consists of three sub- layers, self-attention layer that processes sequential data taking the context of each timestep into consideration, an optional encoder-decoder attention layer that bridges the input sequence and tar- get sequence which exists in decoder only, and a feed-forward layer for non-linear transformation. Transformer achieves the state-of-the-an results for NMT (Barrault et al. 2019). In this work, we will use Transformer as the basic architecture of our model.
Pre-trainine has long history in machine learning and natural language processing (Erhan et al. 2009 2010). Mikolov et al. (2013) and [Pennington et al. (2014) proposed to use distributional representations (i.e., word embeddings) for individual words. Dai & Le (2015) proposed to train a language model or an auto-encoder with unlabeled data and then leveraged the obtained model
to finetune downstream tasks. Pre-training has attracted more and more attention in recent years
2

---

Published as a conference paper at ICLR 2020
and achieved great improvements when the data scale becomes large and deep neural networks are employed. ELMo was proposed in Peters et al. (2018) based on bidirectional LSTMs and its pre-trained models are fed into downstream tasks context-aware inputs. In GPT (Radford et al. 2018), Transformer based language model is pre-trained on unlabeled dataset and then finetuned on downstream tasks. BERT (Devlin al.) 2019) is one of the widely adopted pre-training approach for model initialization. The architecture of BERT is the encoder of Transformer (Vaswani et al.) 2017). Two kinds of objective functions are used in BERT training: (1) Masked language modeling (MLM), where 15% words in sentence are masked and BERT is trained to predict them with their surrounding words. (2) Next sentence prediction (NSP): Another task of pre-training BERT is to predict whether two input sequences are adjacent. For this purpose, the training corpus consists of tuples ( ([cls]. input 1, [sep], ; input 2, [sep]), with learnable special tokens [cls] to classify whether i nput. I and i nppt 2 are adjacent and [sep] to segment two sentences, and with probability 50%. the second input is replaced with a random input. Variants of BERT have been proposed: In XLM (Lample & Conneau 2019), the model is pre- pre-trained based on multiple languages and NSP task is removed; in RoBERTa Liu et al. 2019). more unlabeled data is leveraged without NSP task neither; in XLNet (Yang al. 2019b). permutation based modeling introduced.
A PRELIMINARY EXPLORATION
While few pieces of work Lample & Conneau 2019 Song et al. 2019) design specific pre- training methods for NMT, they are time and resource consuming given that they need to pre-train large models from scratch using large-scale data, and even one model for each language pair. In this work, we focus on the setting of using a pre-trained BERT model. Detailed model download links can be found in Appendix D
Considering that pre-trained models have been utilized in two different ways for other natural lan- guage tasks, is straightforward try them for NMT. Following previous practice, we make the following attempts.
(I) Use pre-trained models initialize the NMT model. There are different implementations for this approach. (1) Following (Devlin et al. 22019), we initialize the encoder of an NMT model with pre- trained BERT. (2) Following (Lample & Conneau 2019). we initialize the encoder and/or decoder of an NMT model with XL.M.
(II) Use pre-trained models as inputs to the NMT model. Inspired from (Peters et al. 2018). we feed the outputs of the last layer of BERT to an NMT model as its inputs.
We conduct experiments on the IWSLT' English->German translation, a widely adopted dataset for machine translation consisting of 160k labeled sentence pairs. We choose Transformer (Vaswani al., 2017) as the basic model architecture with tran former _ ws t_de_en configuration (a layel model with 36.7M parameters). The translation quality is evaluated by BLEU (Papineni cct al. 2002) score; the larger, the better. Both BERT and XLM models are pre-trained and we get them from the Web. More details about the experimental settings included in Appendix A.2
Table I: Preliminary explorations IWSLT' 14 English -nermnh translation.
Algorithm
BLEU score
Standard Transformer
28.57
Use BERT to initialize the encoder of NMT
27.
Use XLM to initialize the encoder NMT Use XLM to initialize the decoder of NMT Use XLM to initialize both the encoder and decoder of NMT
28.22
26.13
28.99
Leveraging the output of BERT as embeddings
29.67
The results are shown in Table We have several observations: (1) Using BERT to initialize the en- coder of NMT can only achieve 27.14 BLEU score, which is even worse than standard Transformer without using BERT. That is, simply using BERT warm up NMT model is not good choice. (2) Using XLM to initialize the encoder or decoder respectively, we get 28.22 or 26.13 BLEU score, which does not outperform the baseline. If both modules are initialized with XLM, the BLEU score
3

---

achieve 29.67 BLEU, much better than using pre- Published as a conference paper at ICLR 2020
is boosted to 28.99, slightly outperforming the baseline. Although XLM achieved great success on WMT Romamian-to to-Engiihh, we get limited improvement here. Our conjecture is that the XLM model is pre-trained news data, which is out-of-domain for IWSLT dataset mainly about spoken languages and thus, leading to limited improvement. (3) When using the output of BERT context-aware embeddings of the encoder, trained models for initialization. This shows that leveraging BERT as feature provider is more effective in NMT. This motivates features provided by pre- pre-trained BERT models.
take one step further and study how to fully exploit such
ALGORITHM
In this section, we first define the necessary notations, then introduce our proposed BERT-fused model and finally provide discussions with existing works.
Notations Let X' and y denote the source language domain and target language domain respectively, which are the collections of sentences with the corresponding languages. For any sentence and y E V, let ly and ly denote the number of units (e.g., words or sub-words) in æ and y. The i-th unit in w/y is denoted x/////- Denote the encoder, decoder and BERT as Enc, Dec and BERT respectively. For case of reference, we call the encoder and decoder in our work as the NMT module. W.l.o.g., we assume both the encoder and decoder consists of L layers. Let att K, V) denote the attention layer, where q. K and V indicate query, key and value respectively (Vaswani et al.) 2017]. We use the same feed-forward layer as that used in (Vaswani et al. 2017) and denote it as FFN. Mathematical formulations of the above layers are left at Appendix
X
4.1 BERT-FUSED MODEL
illustration of our algorithm is shown in Figure [I Any input x E X' progressively processed by the BERT, encoder and decoder.
decoder layer
Add & Norm
Feed Forwanl
4 x
encoder layer
Add & Norm
A & Nomn
Feed Forwani
BERT-Dec
Enc- Dec Atlottion 1a Tug
Ha
A ttention
Ad Norm
Add & Norm
BERT- Enc
Self A ttention
A Atention
Self A ttention
BERT
ST-1
The architecture of BERT-fused model. The left and right figures represent the BERT, encoder and decoder respectively. Dash lines denote residual connections. HB (red part) and H (green part) denote the output of the last layer from BERT and encoder.
Figure 1:
Step-1: : Given any input æ E X, BERT first encodes it into representation H = BERT(T). Hp is the output of the last layer in BERT. The bB E H the representation of the i-th wordpiece x.
Step-2: Let H. denote the hidden representation of 1-th layer in the encoder, and let HP denote word embedding of sequence æ. Denote the i-th element in H 1 hĄ for any [Zz] In the 1-th
4

---

Published as a conference paper at ICLR 2020
layer. 1 [L],
1 = = (attnss(h , ") attnn(h! HB, HB)), Vi E [x].
(I)
where attns and attng are attention models (see (qn. with different parameters. Then each hf is further processed by FFN(.) defined in Eqn. and get the output of the 1-th layer: H (FFN(M)), ... ,FFN), )). The encoder will eventually output H from the last layer.
Step-3: Let Sk, denote the hidden state of I-th layer in the decoder preceding time step f, i.c., S = (s)..... ... s1-1). , Note of is a special token indicating the start sequence, and sy is the embedding of the predicted word at time-step 1. At the i-th layer, have
attns(s, 22t1 S
(2)
8 (attnn((), Hn, Hn) + attnn(8), H Hb)), & FFN(())
The attns, a and a represent self-attention model, BERT-decoder attention model and encoder-decoder attention model respectively. (q iterates over layers and we can eventually obtain sf Finally sf is mapped via a linear transformation and softmax to get the f-th predicted word it. The decoding process continues until meeting the end-of-sentence token.
In our framework, the output of BERT serves as an external sequence representation, and we use an attention model incorporate it into the NMT model. This is general way to leverage the pre-trained model regardless of the tokenization way.
4.2 DROP-NET TRICK
Inspired by dropout (Srivastava et al. 2014) and drop-path (Larsson et al. (017). which can regular- ize the network training, propose drop-net trick ensure that the features output by BERT and the conventional encoder are fully utilized. The drop-net will effect Eqn. and Eqn. Denote the drop-net rate as Pnet E [0, 1]. At each training iteration, for any layer we uniformly sample a random variable U from [0, 1]. then all the hi in Eqn. are calculated in the following way:
R1 dropp- =1(U" Puet
attns(( .HE "H'H''')) ) I(U' Puet attno(h, ,He, He)
2
(3)
11 (
Puu < S - P(n) (attns(h)- 1-1 H . attno(h HH, Ha)),
2 2
where I(-) is the indicator function. For any layer, with probability Pret/2, either the BERT-encoder attention or self-attention is used only; w.p. (1-Pnet). both the two attention models used. For example, at a specific iteration, the first layer might uses attnş only while the second layer uses attng only. During inference time, the expected output of each attention model is used, which is Bu [0, (A drop-net)) The expectation is exactly Eqn.
Similarly, for training of the decoder, with the drop-net trick, have
St delppennt I(U'*
Poel attng(s Ha, H8) + I(U 1 Pnet
2 attne(s), H H
(4)
+ [ Pun
1
U <
Poet (át Ha, Hy) + attne(s), H H 2
For inference, it is calculated in the same way as Eqn. (). Using this technique can prevent network from overfitting (see the second part of Section 6 for more details).
4.3 DISCUSSION
Comparison with ELMo As introduced in Section 2 ELMo (Peters et al. 2018) provides context- aware embeddings for the encoder in order to capture richer information the input sequence. Our approach is more effective way of leveraging the features from the pre-trained model: (1) The output features of the pre- trained model are fused in all layers of the NMT module, ensuring the well- pre- trained features are fully exploited; (2) We use the attention model to bridge the NMT module and the pre-trained features of BERT, in which the NMT module can adaptively determine how to leverage the features from BERT.
5

---

Published as a conference paper at ICLR 2020
Limitations We are aware that our approach has several limitations. (1) Additional storage cost: approach leverages a BERT model. which results in additional storage cost. However, considering the BLEU improvement and the fact that we do not need additional training of BERT, we believe that the additional storage is acceptable. (2) Additional inference time: We use BERT to encode the input sequence, which takes about 45% additional time (see Appendix Oopendix[ffor details). We will leave the improvement of the above two limitations as future work.
5 APPLICATION TO SUPERVISED NMT AND SEMI-SUPERVISED NMT
We first verify our BERT-fused model on the supervised setting, including low-resource and rich- resource scenarios. Then conduct experiments on document-level translation to verify our ap- proach. Finally, we combine BERT-fused model with back translation (Sennrich et al. 2016b) to verify the effectiveness of our method semi-supervised NMT.
5.1 SETTINGS
Dataset For the low-resource scenario, we choose IWSLT' 14 English++German (En++De), English-4Spanisk (En-=Es), IWSLT' English-FFrench (En-FFr) and English-sChinese (En+>Zh) translation. There are 160k, 1834, 236k, 235k bilingual sentence pairs for De En-+Es. En-Fr and En-ZZ tasks. Following the common practice (Edunov al., 2018 for En++De, we lowercase all words. All sentences are preprocessed by BPE (Sennrich et al. (01 The model configuration is trans iwslt stl.t.dde.. .ee en, representing six-layer model with embedding size 512 and FFN layer dimension 1024. For the rich-resource scenario, we work on WMT' 14 En->De and En-Frr, whose corpus sizes are 4.5.M and 36M respectively. We concate- nate newstest2012 and newstesr2013 as the validation set and use newstest2014 model configuration rans fo wor.errriiig another six-layer network with embedding size 1024 and FFN layer dimension 4096. More details about data and model are left in Appendix A.I]
the test set. The
We choose BERT base for IWSLT tasks and BERT large for WMT tasks, which can ensure that the dimension of the BERT and NMT model almost match. The BERT models are fixed during training. Detailed BERT information for each task is Appendix D The drop-net rate Poet is set as 1.0.
Training Strategy We first train an NMT model until convergence. then initialize the encoder and decoder of the BERT-fused model with the obtained model. The BERT-encoder attention and BERT- IWSLT and WMT tasks conducted
decoder attention randomly initialized. Experiments on 1 and 8 M40 GPUs respectively. The batchsize is 4k tokens per GPU. Following (Ott et al. 2018 for WMT tasks, accumulate the gradient for 16 iterations and then update simulate 128-GPU environment. It takes 1, 8 and 14 days to obtain the pre-trained NMT models, and additional 1, 7 and 10 days finish the whole training process. The optimization algorithm is Adam (Kingma & Ba 2014) with initial learning rate 0.0005 and inverse .sgrt learning rate scheduler /Vaswani al. 2017). For WMT 14 En De, we use beam search with width and length penalty 0.6 for inference following (Vaswani et al. 2017). For other tasks, we use width 5 and length penalty 1.0.
Evaluation We use multi-bles bw..bbeu.pp.... pprrl to evaluate IWSLT'14 En and WMT translation tasks for fair comparison with previous work. For the remaining tasks, we use more advance implementation of BLEU score, sacreBLEU for evaluation. Script urls are in Appendix A.I
5.2
RESULTS
The results of IWSLT translation tasks are reported in Ta- 2le We implemented standard Transformer as baseline. Our proposed BERT-fused model can improve the BLEU scores of the five tasks by 1.88, 1.47, 2.4, 1.9 and 2.8 points respectively, demonstrating the effectiveness of our method. The consistent improvements on various tasks shows that our method works well for low-resource trans- lations. We achieved sfate-of-hh--ee art results on IWSLT" 14 De->En translation, widely investigated baseline in ma-
Table 2: BLEU of all IWSLT tasks.
Transformer
BERT-fused
En De
28.57
30.45
De
34.64
36.11
En-
39.0
41.4
En-
26.3
28.2
En-+Fr
35.9
38.7

---

Published as a conference paper at ICLR 2020
chine translation. The comparison with previous methods are shown in Appendix B.4 due to space limitation.
The results of WMT 14 En and En FFr are shown in Table Our reproduced Transformer matches the results reported Ott et al. (2018), and we can see that our BERT-fused model can improve these two numbers 30.75 and 43.78, achieving 1.63 and 0.82 points improvement. Our approach also outperforms the well-designed model DynamicCony (Wu et al. 2019) and a model obtained through neural architecture search (So al. 2019).
Table 3: BLEU scores of WMT" 14 translation.
Algorithm
En
En-
DynamicConv (wu al.
2019
29.7
43.2
Evolved Transformer (So et al.
2019
29.8
41.3
Transformer + Large Batch (Ott al. 2018
29.3
43.0
Our Reproduced Transformer Our BERT-fused model
29.12
42.96
30.75
43.78
5.3 TRANSLATION WITH DOCUMENT-LEVED CONTEXTUAL. INFORMATION
BERT is able to capture the relation between two sentences, since the next sentence prediction (NSP) task is to predict whether two sentences are adjacent. We can leverage this property to improve translation with document-level contextual information (Miculicich et al.] 2018), which is briefly denoted as document-level translation. The inputs are a couple of sentences extracted from a para- graph/document, -1, -2.-.. 27., where the T x's are contextually correlated. We want to translate them into target language by considering the contextual information.
Algorithm In implementation, translate sentence to target domain, leverage the textual information by taking both æ and its preceding sentence &prev as inputs. æ is fed into Enc, which is the same sentence-level translation. For the input of BERT, it the concatenation of two sequences: ([cls], Tprev, [sep], x, [sep]). ), where both [cls] and [sep] are special tokens of BERT.
Setting We use IWSLT' 14 Ent+De dataset as introduced in Section 5.1 The data is a collection of TED talks, where each talk consists of several sequences. We can extract the adjacent sentences for training, validation and test sets. The training strategy, hyperparameter selection and evaluation metric are the same for sentence-level translation.
Baselines We use two baselines here. (1) To demonstrate how BERT works in our model, we replace BERT by Transformer with configu- ration azansformer.ivee _ee.seed_deeen.. which is randomly initialized and jointly trained. (2) Another baseline is proposed by MMculiiiich et al.] (2018), where multiple preceding sen-
Table 4: BLEU of document-level translation.
En
De-+En
Sentence-level
28.57
34.64
Our Document-level
28.90
34.95
Miculicich (2018)
27.94
33.97
Sentence-level BERT
30.45
36.11
tences in a document are leveraged using a hi- erarchical attention network.
Document-level + BERT
31.02
36.69
Results The results are shown Table We can see that introducing contextual information from an additional encoder boost the sentence-level baselines, but the improvement limited (0.33 for En->e and 0.31 for De->En). For Micullicc et al. (2018), the best results we obtain are 27.94 and 33.97 respectively, which are model and document-level information, we can eventually achieve 31.02 for En-+De and 36.69 for De->En. We perform significant tes(] best[beteeen between sentence-level and document-level translation. Our document-level BERT-fused model significantly outperforms sentence-leve baseline with p-value less than 0.01. This shows that our approach only works for sentence-level translation, but can also be generalized to document-level translation.
than the sentence-level baselines. Combining BERT-fused
pts/
ana hnt.y///bborrrrr-yyhhaaaeeeeeeee

---

Published as a conference paper at ICLR 2020
5.4 APPLICATION TO SEMI-SUPERVISED NMT
We work on WMT 16 Romanian ~English (Ro~>En) translation to verify whether our approach can still make improvement over back translation (Sennrich al. 2016b), the standard and powerful semi-supervised way to leverage monolingual data in NMT.
The number of bilingual sentence pairs for Ro-=n is 0.6M. Sennrich et al. (20 provided 2M back translated date We use newsdev2016 validation set and newstest2016 as test set. Sentences were encoded using BPE with a shared source-target vocabulary about 32k tokens. We use trans wwransformee former big configuration. Considering there is Romanian BERT, we use the cased multilingual BERT (please refer to Appendix D) to encode inputs. The drop-net rate Pnet is set as 1.0. The translation quality is evaluated by multi settibbleu.pe..
The results are shown in Table Transformer baseline achieves 33.12 BLEU
The
Table 5: BLEU scores WMT" Ro >En.
Methods
BLEU
score. With back-translation, the performance is boosted 37.73. We use the model obtained with back-translation to initialize BERT-fused model, and eventually reach 39. BLEU. Such a score surpasses the previous best result 38.5
Sennrich et al. (2116a)
33.9
XLM (Lample & Conneau 2019
38.5
Standard Transformer
33.12
+ back translation
37.73
achieved by XLM (Lample & Conneau 2019)
+ BERT--used model
39.10
and sets new record. This demonstrates that
our proposed approach is effective and can still achieve improvement over strong baselines.
6 ABLATION STUDY
We conduct two groups of ablation studies on IWSLT' En->De translation to better understand our model.
Table 6: Ablation study IWSLT" 14 En
Standard Transformer
28.57
BERT-fused model
30.45
Randomly initialize encoder/decoder of BERT-fused model
27.03
Jointly tune BERT and encoder/decoder of BERT-fused model
28.87
Feed BERT feature into all layers without attention Replace BERT output with random vectors
29.61
28.91
Replace BERT with the encoder of another Transformer model
28.99
Remove BERT-encoder attention Remove BERT-decoder attention
29.87
29.90
Study for training strategy and network architecture
We conduct ablation study to investigate the performance of each component of our model and training strategy. Results are reported in Table 6
(1) We randomly initialize the NMT module (i.e., encoder and decoder) of BERT-fused model in- stead of using a warm-stan start one as introduced in the training strategy of Section 5.1 In this way, we can only achieve 27.03 BLEU score, which cannot catch up with the baseline. We also jointly train BERT model with the NMT module. Although can also boost the baseline from 28.57 28.87, is not good as fixing the BERT part, whose BLEU is 30.45.
(2) We feed the output of BERT into all layers of the encoder without attention models. That is, Whhh-t)). where W is learnable.
the Eq is revised to R (at ( H. In this case, the encoder and BERT have to share the same vocabulary. The BLEU score is 29.61, which is better than the standard Transformer but slightly worse than leveraging the output of BERT
Data http: http:///aaa.aaaaaaa.ce.eeeeeeee/ http://cccccccctooooeeeerrrrreee www.srannaaaiiooooooeeeeee

---

Published as a conference paper at ICLR 2020
embedding. This shows that the output of BERT should not be fused into each layer directly, and using the attention model to bridge the relation is better than using simple transformation, More results different languages are included in Appendix [B.]] To illustrate the effectiveness of our method, we choose another two kinds of ways to encode the input sequence rather than using BERT: (1) Using fixed and randomly initialized embedding: (2) Using the encoder from another NMT model. Their BLEU scores are 28.91 91 and 28.99 respectively, indicating that the BERT pre-trained on large amount of unlabeled data can provide more helpful features NMT.
(3) To verify where the output of BERT should be connected to, we remove the BERT-encoder atten- tion (i.e., attng in Eqn. T and the BERT-decoder attention (i.e., a in Eqn. respectively. Correspondingly, the BLEU score drops from 30.45 to 29.87 and 29.90. This indicates that the out- put of BERT should be leveraged by both encoder and decoder to achieve better performances. At last, considering that there are two stacked encoders in our model, we also choose ensemble models and deeper NMT models as baselines. Our approach outperforms the above baselines. The results are left in Appendix B.2 due to space limitation.
Study on drop-net
To investigate the effect of drop-net, we conduct experiments IWSLT" 14 En->De dataset with different drop-net probability, Phct E (0, 0.2, 0.4, 0.6, 0.8, 1.0). The results are shown Figure ? As can been scen, although larger Post leads larger training loss, it leads smaller validation loss and so better BLUE scores. This shows that the drop-net trick can indeed improve the generalization
ability of our model. We fix Puet 1.0 in other experiments unless specially specified.
4.5
32
1O55
SaLEd
Rie 0.
Por 0.6
Rue 0.2
Pu. 0.8
Roe
Pe. 1.0
00
20
40
60 EPOCH
80
100
120
3.50
20
40
60
80
100
120
280
20
40
60
80
100
120
EPOCH
EPOCH
(a) Training loss.
(b) Validation loss.
(c) Validation BLEU.
Figure 2: Training/validation curves with different Poet ss
APPLICATION TO UNSUPERVISED NMT
We work unsupervised En++Fr and En translation. The data processing, architecture selec- tion and training strategy is the same as Lample & Conneau (2019).
Settings For Ent+Fr, we use 190M monolingual English sentences and 62.M monolingual French sentences from WMT News Crawl datasets, which is the same as that used in (Song et al. 2019
For unsupervised En translation, we use 50M English sentences from News Crawl (sampled from the data for En->Fr) and collect 2.9M sentences for Romanian by concatenating News Crawl data sets and 16 Romanian monolingual data following [Lample et al. (2018). The data is preprocessed in the same way as Lample & Conneau (2019).
We use the same model configuration as Lample Conneau (209)), with details in Appendix A.3 The BERT is the pre-trained XL.M model (see Appendix D). We first train an unsupervised NMT model following Lample & Conneau (2019) until convergence. Then we initialize our BERT-fused model with the obtained model and continue training. We train models on 8 M40 GPUs, and the batchsize is 2000 tokens per GPU. We use the same optimization hyper-parameter as that described
in Lample & Conneau 2019).
Data source: ht htttp::///odeee

---

Published as a conference paper at ICLR 2020
Table 7: BLEU scores of unsupervised NMT.
En-
Fr-+En
En En-RRo
Ro->n
Lample et al. 2018
27.6
27.7
25.1
23.9
XLM (Lample & Conneau 2019)
33.4
33.3
33.3
31.8
MASS (Song al. 2019)
37.50
34.90
35.20
33.10
Our BERT-fused model
38.27
35.62
36.02
33.20
Results The results of unsupervised NMT are shown Table 7 With our proposed BERT-fused model, we can achieve 38. 27, 35.62, 36.02 and 33.20 BLEU scores on the four tasks, setting state- of-the-art results on these tasks. Therefore, our BERT-fused model also benefits unsupervised NMT.
8 CONCLUSION AND FUTURE WORK
In this work, we propose an effective approach, BERT-fused model, to combine BERT and NMT, where the BERT leveraged by the encoder and decoder through attention models. Experiments on supervised NMT (including sentence-leve and document-level translations), semi-supervised NMT and unsupervised NMT demonstrate the effectiveness of our method.
For future work, there are many interesting directions. First, we will study how to speed up in- ference time. Second, we can apply such an algorithm more applications, like questioning and answering. Third, how to compress BERT-fused model into a light version is another topic. There are some contemporary works leveraging knowledge distillation to combine pre-trained models with NMT (Ynng et al. 2019a Chen et al. 2019), which is direction to explore.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 6th International Conference on Learning Representations, 2015. URL
2tt1//11419.00333.... pddf
Loic Barrault, Ondfej Bojar, Marta R. Costa-jussá, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference ma- chine translation (wmt19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, August 2019. Association for Computational Linguistics. URL http: / www awwewb. www.lleww.rgg.g........o.....l.. http://w.....bb.oorrraaaaoaaoaa/
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop on Spoken Language Translation, Hanoi, Vietnam, pp. 57, 2014.
Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. Distilling the knowledge of bert for text generation. arXiv preprint 1911:v1111.3300 2019.
Andrew M Dai and Quoc Le. Sumi-supe sequence learning. In Advances in neural infor- mation processing systems, pp. 3079-30000 3087, 2015.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. Latent alignment and variational attention. In Advances in Neural Information Processing Systems, pp. 9712-9724, 2018.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACI, 2019. htl./ttts://arxiv arg/pdf/18 04805.000.. f
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marcaurelio Ranzato. Classical struc- tured prediction losses for sequence sequence learning. NAACI, 2018.
10

---

Published as a conference paper at ICLR 2020
Dumitru Erhan, Pietre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. The difficulty of training deep architectures and the effect of unsupervised pre- trainingg...... In Artificial Intelligence and Statistics, pp. 153-160, 2009.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11(Feb):625-6 660, 2010.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Voiie Volume 70, pp. 1243-1252. JMLR. org, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98): URL http://dx.
1780, November 1997. ISSN 0899-7667. doi: 10.1 wtt.////c..//........ doi.org/10. 11t.//....11111ooooo 9tt.
Marcin Junezys-Dowmunt and Roman Grundkiewicz. Ms-uedin submission to the wmt2018 ape shared task: Dual-source transformer for automatic post-editing. EMNLP 2018 THIRD CON- FERENCE ON MACHINE TRANSLATION (WMT18), 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: T0t::///2.6000 2,680, 2014.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. NeurIPS, 2019.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. arXiv preprint arXtv:/804.07755, 2018.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ulua-deep neural net- works without residuals. ICLR, 2017. URL pdf
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 11692, 2019.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document-level neural machine translation with hicrarchical attention networks. arXiv preprint 1809:0110000017700 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. EMNLP 2018 third conference on machine translation (WMT18), 2018 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311 311-318. Association for Computational Linguistics, 2002.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 20/4 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew Peters, Mark Neumann, Mohit Iyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint arXiv: 1rXi::8022.05365, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training- URL https://s3-us-wess--. amazonaws. com/openai- aaeer//earrrhoerrrlgggggggnngere
understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
11

---

Published as a conference paper at ICLR 2020
Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for wmn 16. In Proceedings of the First Conference on Machine Translation, volume 2, pp. 371- 376, 2016a. URL/ttt::/// www. statmt
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. ACL, 2016b. URL httpps///aclvebb.og////ttttttot
P16-1009
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. ACI., 2016c.
David So, Quoe Le, and Chen Liang. The evolved transformer. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5877-5886, Long Beach, Cali-
fornia, USA, 09-15 Jun 2019. PMLR. URL http: http://pooceeeenneeeeeee ml p.ppsss//97 50 9a htm
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Leurning Research, pp. 5936-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR. Utthttp::/ http::///poceedeiingneoo m r press / v wtt///ng/l///// song 19.. html
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, (((1):1929-199))) 2014.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 1104-3112, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tic- Yan Liu. Multi- agent dual learning. ICLR, 2019.
Dirk Weissenborn, Douwe Kiela, Jason Weston, and Kyunghyun Cho. Contextualized role interac- tion for neural machine translation, 2019.
hwtp://wwwwwww.....eeeeeeeee ?id=
ryx3 1AcY7
Felix Wu, Angela Fan, Alexci Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019. URL http:tttss//peeeeeeeee
n for.mrid=ussseerr ftte///ooooummm11-//L1/1100
Lijun Wu, Fei Tian, Yingee Xia, Yang Fan, Tao Qin, Lai Jian Jian-Huang, and Tie- Yan Liu. Learning to teach with dynamic loss functions. In Advances in Neural Information Processing Systems, pp- 1466-6477 2018
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint UrXiv:1609.000 0444, 2016.
Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied transformers: Neural machine translation with shared encoder and decoder. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5466-5473, 2019.
Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Yong Yu, Weinan Zhang, and Lei Li. Towards making the most of bert neural machine translation. arXiv preprint arXiv: 2019a.
arxii v. wwwpoo/gg//o.oooo//o 1988. 05672.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 17X1071906.08237,, 2019b.
12

---

Published as a conference paper at ICLR 2020
A EXPERIMENT SETUP
A.1 IWSLT' 14 & WMT 14 SETTINGS
We mainly follow the scripts below to preprocess the data:
Dataset For the low-resource scenario, we choose IWSLT' 14 English++German (En++De), English-+Spanish (En->Es), IWSLT' 17 English->French (En->Fr) and English-&Chinese (En->Zh) translation. There are 160k, 1834, 236k, 235k bilingual sentence pairs for En En-+Es, En-Fr and En-+Zh tasks. Following the common practice (Edunov al. 2018), for En++De, we lowercase all words, split 7k sentence pairs from the training dataset for validation and concatenate dev2010, dev2012, tst2010, tst2011, tst2012 as the set. For other tasks, we do not lowercase the words and use the official validation/test sets of the corresponding years.
For rich-resource scenario, work WMT En and En-+Fr, whose corpus sizes are 4.5. and 36M respectively. We concatenate newstest2012 and newstest2013 as the validation set and use newstest2014 as the test set.
We apply BPE (Sennrich et al. 2016c to split words into sub-units. The numbers of BPE merge operation for IWSLT tasks, WMT 14 En->De and En->Fr are 10k, 32k and 40k respectively. We merge the source and target language sentences for all tasks to build the vocabulary except En->Zh.
Model Configuration For IWSLT tasks, we use the transformes iwslt _eeeddnennnn en setting with dropout ratio 0.3. In this setting, the embedding dimension, FFN layer dimension and number of layers are 512, 1024 and For WMT En -Dn and En-+Fr, we use rans transformer.2 formerbiig setting (short for rrans Former-vaswar www.waaa.n...................... wartaaiiimttttt www.:eeeteeeeiiiiemmmmm with dropout 0.3 and 0.1 respectively. In this setting, the aforementioned three parameters are 1024, 4096 and respectively.
Evaluation We use multi soltt-bllee bleu. per...eer.rree. to evaluate IWSLT' 14 Ene+De and WMT translation tasks for fair comparison with previous work. For the remaining tasks, we use a more advance implementation of BLEU score, detokenized sacreBLEU for evaluation
A.2 DETAILED EXPERIMENT SETTING SECTION 3
The IWSLT" 14 Aaglish-to-German data and model configuration is introduced in Section A.I
For the training stategy, we use Adam Kingma & Ba 2014) to optimize the network with 31 = 0.9, 82 0.98 and weight-decay 0.0001. The learning rate scheduler is i invers aeerse sqrt, where wa harmup-init-i = 10-², wa caarmuppupdates = 4000 and max-1r = 0.0005.
A.3 DETAILED MODEL CONFIGURATION IN UNSUPERVISED NMT
We leverage one Transformer model with GELU activation function to work on translations two directions, where each language associated with language tag. The embedding dimension, FFN layer dimension and number of layer are 1024, 4096 and 6. The BERT is initialized by the pre- trained XLM model provided by (Lample & Conneau 2019).
B MORE EXPERIMENT RESULTS
B.1 MORE RESULTS ON PRELIMINARY EXPLORATION OF LEVERAGING BERT
We use XL.M initialize the model for WMT' 14 English~+German translation task, whose training corpus is relative large. We eventually obtain 28.09 after 90 epochs, which is still underperform the baseline, 29.12 as we got. Similar problem is also reported in https: htt///tttut thub. com/ We leave the improvement of supervised NMT with
XLM as future work.
mu t 1iibbeu. ti-bleu.per
13

---

Published as a conference paper at ICLR 2020
B.2 MORE ABLATION STUDY
Part I: A different way to deal with multiple attention models
Munczys-Dowmine Grundkiewicz (2018) proposed new way to handle multiple attention models. Instead of using Eqn.2). the input is processed by self-attention, encoder-decoder attention and BERT-decoder attention sequentially. Formally.
& = a S
1 Skct+)):
5 attng (8), 1 H1)
(5)
S5 = attng(s, s HB. HB):
s = FFN($)).
The BLEU score is 29.35 for this setting, not as good as our proposed method.
Part II: More results IWSLT'14 En-*De translation
Since our BERT-fused model contains two stacked encoders, we carry out two groups of additional baselines:
(I) Considering that stacking the BERT and encoder can be seen as deeper model, we also train another NMT models with deeper encoders, with 18 layers (since BERT huse consists of 12 layers) and the other with 12 layers (which achieved best validation performance ranging from 6 to 18 layers).
(2) We also compare the results of our approach with ensemble methods. To get an M-model ensemble, we independently train M models with different random seeds (M E 2 We ensemble both standard Transformers and our BERT-fused models, which are denoted as M-model ensemble (standard) and M-model ensemble (BERT-fused) respectively. Please note that when we aggregate multiple BERT-fused models, we only need to store one replica of the BERT model because the BERT part is not optimized.
Table More ablation study on IWSLT' 14 De.
Algorithm
BLEU
Standard Transformer
28.57
BERT-fused model
30.45
12-layer encoder 18-layer encoder
29.27
28.92
2 moddl ensemble (standard)
29.71
3 3-moeel ensemble (standard)
30.08
4- model ensemble (standard)
30.18
2-model ensemble (BERT-fused)
31.09
3-model ensemble (BERT-fused)
31.45
model ensemble (BERT-fused)
31.85
The results shown in Table g We have the following observations:
1. Adding more layers indeed boost the bascline, but still not good as BERT-fused model. According to our experiments, when increasing the number layers to 12, we achieve the best BLEU score, 29.27.
2. We also compare the baseline by more than one point. However, even if using ensemble of four models, the BLEU score is still lower than our BERT-fused model (30.18 18 v.s. 30.45), which shows the effectiveness of our method.
results ensemble methods. Indeed, ensemble significantly boost
We want to point out that our method is intrinsically different from ensemble. Ensemble approaches usually refer "independently" train several different models for the same task, and then aggregate
14

---

Published as a conference paper at ICLR 2020
the output of each model to get the eventually task. In BERT-fused model, although include pre-trained BERT into our model, there is still only one model serving for the translation task.
In this sense, from ensemble too. When ensembling two models, we can achieve 31.09 BLEU score. When adding the number of models to four, eventually achieve 31.85 BLEU score, which is 1.67 point improvement over the ensemble of standard Transformer.
also combine BERT-fused model with ensemble. Our approach benefits
Part III: More results IWSLT'14 14 De-En translation
We report the ensemble results on IWSLT" 14 De->E translation in Table 9 conclusion compared to that of IWSLT" 14 En->De.
We can get similar
Table 9: More ablation study on IWSLT' 14 De-
Algorithm
BLEU
Standard Transformer
34.67
BERT-fused model
36.11
2-model ensemble (standard)
35.92
3-model ensemble (standard)
36.40
4-model ensemble (standard)
36.54
2-model ensemble (BERT-fused)
37.42
3-model ensemble (BERT-fusee
37.70
4-model ensemble (BERT-fused) 37.71
B.3
MORE RESULTS ON FEEDING BERT OUTPUT TO NMT MODULE
The ablation study more languages is shown in Table TO Our method achieves the best results compared to all baselines.
Table 10: BLEU scores IWSLT translation tasks.
Algorithm
En DD
De->En
En-
En-*Zh
En-
Standard Transformer
28.57
34.64
39.0
26.3
35.9
Feed BERT feature into embedding
29.67
34.90
39.5
28.1
37.3
Feed BERT feature into all layers of encoder
29.61
34.84
39.9
28.1
37.4
Our BERT-fused model
30.45
36.11
41.4
28.2
38.7
B.4 MORE BASELINES OF IWSLT'14 14 GERMAN-TO-ENGUAE TRANSLATION
We summarize the BLEU scores on IWSLT' 14 De existed works and our BERT-fused model approach in Table II
Table 11: Previous results of IWSLT' 14 De->En.
Approach
BLEU
Multi-agent dual learning (Wang al. 2019
35.56
Tred-Transformer (Xia et al. (2019)
35.52
Loss teach (Wu et al. 2018)
34.80
Role- cole-intcracttve layer (Weissenborn et al
2019
34.74
Variational attention (Deng al. 2018)
33.68
Our BERT-fused model
36.11
15

---

Published as a conference paper at ICLR 2020
B.5 COMPARISON WITH BACK TRANSLATION
When using unlabeled data to boost machine learning systems, one of the most notable approaches back translation (briefly. BT) (Sennrich et al. 2016b): We first train a reversed translation model, use the obtained model to translate the unlabeled data in the target domain back to source domain, obtain a synthetic dataset where the source data back-translated and finally train the forward model on the augmented dataset.
Our method has two main differences with BT method.
1. In BT, the monolingual data from the target side leveraged. In our proposed approach, we use a BERT of the source language, which indirectly leverages the monolingual data from the source side. In this way, our approach and BT are complementary to each other. In Section 5.4 we have already verified that our method can further improve the results of standard BT on
tn-Elglish translation.
2. To use BT, we have to train a reversed translation model and then back translate the mono- lingual data, which is time-cost due the decoding process. In BERT-fused model, only need to download a pre-trained BERT model, incorporate it into our model and con- tinue training. Besides, the BERT module is fixed during training.
On IWSLT' 14, also implement BT on wikipedia data, which is subset of the corpus of training BERT. The model used for back translation are standard Transformer baselines introduced Sec- tion 5 whose BLEU scores are 28.57 and 34.64 respectively. We back translate 1M, 2M, 5M, 15M and 25M randomly selected German sentences.
The results reported in Table 12 The rows started with BT(-) represent the results of BT, and the numbers the brackets are the number of sentences for back translation.
Table 12: BLEU scores IWSLT' 14 En De by BT.
En BERT-fused model
Algorithm
Standard Transformer
28.57
30.45
BT (IM)
29.42
BT (2M)
29.76
BT (5M)
29.10
BT (15M)
28.26
BT (25M)
27.34
IWSLT dataset is collection of spoken language, and the bilingual training corpus is small (160k). In Wikipedia, the sentences are relatively formal compared to the spoken language, which is out- of-domain of spoken languages. We can see that when using 1M or 2M monolingual data for BT, the BLEU scores can indeed improve from 28.57 to 29.42/29.76. However, simply adding more wikipedia data for BT does not result in more improvement. There is even a slight drop when adding more than 15M monolingual sentences. However, our BERT-fused model can achieve better performances than BT with wikipedia data.
C COMPARISON OF INFERENCE TIME
Table 13: Comparisons on inference time (seconds), '+' is the increased ratio of inference time.
Dataset
Transformer
Ours
(+)
IWSLT' 14 En En
70
97
38.6%
IWSLT' 14 De-
69
103
49.3%
WMT 14 En-+De
67
99
47.8%
WMT' 14 En-+Fr
89
128
43.8%
16

---

Published as a conference paper at ICLR 2020
We compare the inference time of our approach to the baselines. The results shown in Table B where from the second column to the last column, the numbers are the inference time of standard Transformer, BERT-fused model, and the increase of inference time.
Indeed, introducing BERT to encode the input brings additional inference time, resulting in about 40% 49% increase. But considering the significant improvement of BLEU score, it is acceptable of such extra cost. We will study how to reduce inference time in the future.
D DOWNLOAD LINK OF PRE-TRAINED BERT MODELS
We leverage the pre-trained models provided by Chorch-Transformery
For IWSLT' 14 tasks, we choose BERT model with 12 layers and hidden dimension 768.
1. IWSLT14 En (De, Es, Fr, Zh}. we choose bert-base-uncased.
2. IWSLT14 De- De-En, choose bert-baase-grrmn--caee
For WMT14 (nr, De), we choose weztt-large-uncasee.e which BERT model with 24 layers and hidden dimension 1024.
For WMT16 Ro-En, we choose bert-base-muiiiiingnaa----as because there is no BERT specially trained for the Romanian.
For unsupervised En and unsupervised En we choose xim-mlm-enfr100m and xlm-mlm-enee 024 respectively.
The download links are summarized as follows:
bert://ert-baae-nnccaa..........
bert-large-uncased:
berrbass-mulliingualcaadddd
bert-base-german-ccscee
tar.gz
xln-mlm-enffr0024
xlm-mlm-cnnoo004:
E DETAILS OF THE NOTATIONS
Let attn(g, K, V) denote the attention layer, where q. K and V indicate query, key and value respectively. Here q is a dy-dimensional vector (d E Z), K and are two sets with K = [V]. Each k; K and U; V are also dg/dd-ddimensionl (dg, dy and d, be different) vectors, i e [[]]]. The attention model works as follows:
attn(q, K, = W.W, a;= cxp (W (WW,k) Z = exp (V (WWk))
M
K
Z
(6) where W. Wi and W., are the parameters to be learned. In Vaswani et al. (201)), attn is im- plemented as a multi-head attention model and we omit the details here to increase readability. Following Vaswani et al. (2017), we define the non-linear transformation layer
FFN(#) W2 max(W,, b,, by.
17

---

Published as a conference paper at ICLR 2020
where is the imput; W1. W2. d1, b2 the parameters to be learned; max is element- operator. Layer normalization is also applied following Transformer (Vaswani et al. (017).
18


Reviews:
[{"review_id": "rJxBSkziYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the use of BERT to improve Neural Machine Translation (NMT) both in supervised, semi-supervised and unsupervised settings. The authors first show that using BERT to initialize the encoder and/or the decoder does not bring any clear improvement, while using it as a feature extractor performs better. Based on this finding, the authors propose a new approach to integrate BERT in NMT, named BERT-fused NMT, which incorporates BERT representations from the input sequence into the encoder and decoder attention mechanisms.\n\nI am ambivalent about this paper. On the one hand, the paper presents a thorough experimental evaluation, with strong baselines (often outperforming their original implementation) and results that can be interesting from different angles, and the reported improvements are consistent. However, the paper is rather poorly written and some important details are not adequately described, which left me with some concerns and an overall negative impression as I read through the paper. More concretely:\n\n- The paper is rather poorly written. There are many expressions that sound ungrammatical or otherwise unnatural to me (although I am not a native speaker myself) and, more importantly, the overall exposition of ideas is not sufficiently clear. I found the paper difficult to follow, and I was left with many doubts as I read through it. In addition, the style in which some results are presented is inappropriate for an academic paper (e.g. \"Obviously, our proposed BERT-fused NMT can improve the BLEU scores\"), although I understand that this was probably not intentional.\n\n- To make things worse, the paper is 10 pages long, and according to the CFP reviewers are \"instructed to apply a higher standard to papers in excess of 8 pages\". I think that the paper could be fit in the regular 8 page limit.\n\n- The pre-trained BERT models that the authors use were trained on different (and generally larger) training data than what they use for the NMT training (e.g. they all use Wikipedia). As such, the models that build on BERT are indirectly using this additional training data. How can we make sure that the reported improvements are not due to this additional data? What would happen if the same data was used for the baseline systems (e.g. through back-translation)? Also, please clearly state which pre-trained model you use for each specific experiment.\n\n- The treatment of subword tokenization is not given sufficient attention and raises some concerns to me. It seems clear that the authors combine different subword tokenizations for their proposed system (i.e. BERT and the NMT encoder/decoders use a different subword vocabulary). However, it is not clear to me how this is handled in the baseline systems that use BERT for initialization only, for which a mismatch in tokenization would be problematic.\n\n- I often find it difficult to understand what the authors did exactly for each of the reported systems. For instance, what is the difference between \"Standard transformer\" and \"Training NMT module from scratch\" in Table 6? I cannot see any yet the difference in BLEU is 1.5.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJgB3-oRtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nThis paper discusses a method that effectively incorporates a (large) pre-trained LM, such as BERT and XMN, for improving the performance of NMT.\n \nThe motivation of this paper is rather straightforward and not novel; many researchers can quickly think of such an idea of incorporating the power of the recent (rapid) development of pre-training LMs into NMT.\nFrom this perspective, this paper is not very exciting. \nHowever, as described in the paper, we often fail to improve (or even degrade) the performance of NMT when we straightforwardly incorporate a pre-trained LM.\nThus, many researchers/developers might want to know a practical approach to integrate a pre-trained LM into NMT.\nThis paper provides a straightforward but smart way to incorporate pre-trained LMs, which is not trivial in the community.\nIn this sense, this paper might have a considerable influence on the community.\nI was a bit surprised by the apparent effectiveness of the proposed method since I also have attempted to apply pre-trained LMs to NMT and have not obtained a good result.\n \n \nExperimental results are mostly convincing; the authors conducted comprehensive and extensive experiments on many settings, such as supervised NMT with low- and hi-resource settings, a semi-supervised NMT setting by back-translation, document-level MT, and unsupervised NMT.\nThe results were also promising; the proposed method consistently outperformed conventional methods. \nI think these results are useful for many readers.\nMoreover, such findings also offer further insights for many researchers who aim to apply BERT to many other tasks, especially for text generation tasks.\n \n\nHere are my concerns about this paper.\n\n1, unclear explanations\nThe writing can be much improved. Readers might be able to guess, but several descriptions are hard to follow, or detailed explanations are missing.\nFor example, what is the exact operation of \"function cascade\"?  \nWhat is the difference between the \"Training NMT module from scratch\" and \"Standard Transformer\" in Table 6?  What is the main reason for the lower performance of (Miculicich et al. (2018)) than that of sentence-level NMT in Table 4?\n\n2, better comparisons\nI think the authors need to confirm another model setting for a fairer comparison, something like \"The proposed architecture with (fixed) random vectors instead of the BERT's contextualized embeddings.\nIt is because we sometimes observe the improved performance for the above model comparing with the original one.\nWe can interpret this improvement by the effect of increasing the weight parameters for injecting the additional random vectors to the original architecture.\nTherefore, I think the above model settings can improve the performance of standard Transformers, which can be a preferable counterpart of the proposed method.\nMoreover, the proposed method is closely related to the model ensembling since the method utilizes two separate models.\nTherefore, the authors should also report the results of model ensembling for better comparisons.\n \n3, less discussion for the experimental results\nI found minimal discussions about the results.\nFor example, in the ablation study, the authors only show (list) the observations of their results and no discussions.\nThe authors should provide discussions about how and why their method (architecture) can improve the performance compared with a similar (and current de facto standard) approach, like the fine-tuning setting that can often improve most of the other NLP tasks.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HyeCZACtYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to incorporate BERT pretrained sentence representations within a NMT architecture.\nIt shows that simply pretraining the encoder of a NMT model with BERT does not necessarily provide gains (and can even be detrimental) and proposes instead to add a new attention mechanism, both in the encoder and in the decoder. The modification is relatively simple, but provides significant improvements in supervised and unsupervised MT, although it makes the model slower and computationally more expensive. The paper contains a lot of experiments, and a detailed ablation study.\n\n===\n\nI'm very surprised by the results in Table 1, i.e. the fact that pretraining can decrease the performance significantly. The provided explanation \"Our conjecture is that the XLM model is pre-trained on news data, which is out-of-domain for IWSLT dataset mainly about spoken languages\" is not satisfactory to me. The domain mismatch is also there in the majority of GLUE tasks, SQUAD, etc. and yet pretraining with BERT significantly improves the performance on these tasks. When the encoder is pretrained with a BERT/XLM model, I assume the encoder is not frozen, but finetuned?\n\nThe description of the algorithm in Section 4 could be simplified a lot I feel. Overall, the attention in the encoder is simply replaced by two attention layers: one over the previous layer like in a standard setting, and one on top of the BERT representation. Also I don't understand why the attention over the BERT sequence is also necessary in the decoder. Shouldn't this information already be captured by the encoder output?\n\nThe Drop-Net Trick is interesting. But the fact that 1.0 gives the best performance (Section 6.2) is very unintuitive to me. This means that the model will never consider the setting with two attentions at training time, although this is what it does at test time.\n\nIn Table 6, you propose experiments with 12 and 18 layers for fair comparison, because as you mention, your model with BERT-fused has more parameters. But IWSLT is a very small dataset and it would have been surprising that using 18 layers actually helps (overfitting is much more likely in that setting). Instead, I think something like an ensemble model would be a more fair comparison. In fact, the BERT-fused is essentially an ensemble model of the encoder.\nCould you try the following experiment on IWSLT, where you do not pretrain the BERT model with the BERT objective, but with a NMT encoder trained in a regular supervised setting (i.e. do not reload a BERT model, but a NMT encoder that you previously trained without the fused architecture)?\n\nOverall, I think the gains are nice, but I would really like to see the comparison I mentioned just above, and comparisons with ensemble models. The proposed model is significantly larger / slower than the baseline models considered, and I wonder if you could not achieve the same gains with ensemble models.\n\nSomething I like about the approach is that is it quite generic in the sense that you can provide any external sequence of vectors as input to your encoder. As a result, it is possible to leverage a model pretrained with a different tokenization. Tokenization is often an issue with pretraining in NLP (how do you leverage a model trained without BPE if you actually want to use BPE in your new model). The proposed approach does not has this constraint and I think this is something you should highlight more in the paper.\n\n===\n\nSmall details in the related work section:\n- I would cite \"Sutskever et al, 2014\" for the LSTM encoder, along with \"Hochreiter & Schmidhuber\", and not only \"Wu et al, 2016\"\n- Removing the NSP task was proposed in \"Lample & Conneau, 2019\", not in \"Liu et al, 2019\"", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel way of incorporating a large pretrained language model (BERT) into neural machine translation using an extra attention model for both the NMT encoder and decoder.   The paper presents thorough experimental design, with strong baselines and consistent positive results for supervised, semi-supervised and unsupervised experiments. The reviewers all mentioned lack of clarity in the writing and there was significant discussion with the authors. After improvements and clarifications, all reviewers agree that this paper would make a good contribution to ICLR and be of general use to the field. 
Decision: Accept (Poster)