Introduction
Neural Machine Translation (NMT) has made considerable progress in recent years BIBREF0 , BIBREF1 , BIBREF2 . Traditional NMT has relied solely on parallel sentence pairs for training data, which can be an expensive and scarce resource. This motivates the use of monolingual data, usually more abundant BIBREF3 . Approaches using monolingual data for machine translation include language model fusion for both phrase-based BIBREF4 , BIBREF5 and neural MT BIBREF6 , BIBREF7 , back-translation BIBREF8 , BIBREF9 , unsupervised machine translation BIBREF10 , BIBREF11 , dual learning BIBREF12 , BIBREF13 , BIBREF14 , and multi-task learning BIBREF15 .
We focus on back-translation (BT), which, despite its simplicity, has thus far been the most effective technique BIBREF16 , BIBREF17 , BIBREF18 . Back-translation entails training an intermediate target-to-source model on genuine bitext, and using this model to translate a large monolingual corpus from the target into the source language. This allows training a source-to-target model on a mixture of genuine parallel data and synthetic pairs from back-translation.
We build upon edunov2018understanding and imamura2018enhancement, who investigate BT at the scale of hundreds of millions of sentences. Their work studies different decoding/generation methods for back-translation: in addition to regular beam search, they consider sampling and adding noise to the one-best hypothesis produced by beam search. They show that sampled BT and noised-beam BT significantly outperform standard BT, and attribute this success to increased source-side diversity (sections 5.2 and 4.4).
Our work investigates noised-beam BT (NoisedBT) and questions the role noise is playing. Rather than increasing source diversity, our work instead suggests that the performance gains come simply from signaling to the model that the source side is back-translated, allowing it to treat the synthetic parallel data differently than the natural bitext. We hypothesize that BT introduces both helpful signal (strong target-language signal and weak cross-lingual signal) and harmful signal (amplifying the biases of machine translation). Indicating to the model whether a given training sentence is back-translated should allow the model to separate the helpful and harmful signal.
To support this hypothesis, we first demonstrate that the permutation and word-dropping noise used by BIBREF19 do not improve or significantly degrade NMT accuracy, corroborating that noise might act as an indicator that the source is back-translated, without much loss in mutual information between the source and target. We then train models on WMT English-German (EnDe) without BT noise, and instead explicitly tag the synthetic data with a reserved token. We call this technique “Tagged Back-Translation" (TaggedBT). These models achieve equal to slightly higher performance than the noised variants. We repeat these experiments with WMT English-Romanian (EnRo), where NoisedBT underperforms standard BT and TaggedBT improves over both techniques. We demonstrate that TaggedBT also allows for effective iterative back-translation with EnRo, a technique which saw quality losses when applied with standard back-translation.
To further our understanding of TaggedBT, we investigate the biases encoded in models by comparing the entropy of their attention matrices, and look at the attention weight on the tag. We conclude by investigating the effects of the back-translation tag at decoding time.
Related Work
This section describes prior work exploiting target-side monolingual data and discusses related work tagging NMT training data.
Leveraging Monolingual Data for NMT
Monolingual data can provide valuable information to improve translation quality. Various methods for using target-side LMs have proven effective for NMT BIBREF20 , BIBREF7 , but have tended to be less successful than back-translation – for example, BIBREF7 report under +0.5 Bleu over their baseline on EnDe newstest14, whereas BIBREF19 report over +4.0 Bleu on the same test set. Furthermore, there is no straighforward way to incorporate source-side monolingual data into a neural system with a LM.
Back-translation was originally introduced for phrase-based systems BIBREF21 , BIBREF22 , but flourished in NMT after work by sennrich2016improving. Several approaches have looked into iterative forward- and BT experiments (using source-side monolingual data), including cotterell2018explaining, hoang2018iterative, and niu2018bidirectional. Recently, iterative back-translation in both directions has been devised has a way to address unsupervised machine translation BIBREF23 , BIBREF11 .
Recent work has focused on the importance of diversity and complexity in synthetic training data. BIBREF24 find that BT benefits difficult-to-translate words the most, and select from the back-translated corpus by oversampling words with high prediction loss. BIBREF25 argue that in order for BT to enhance the encoder, it must have a more diverse source side, and sample several back-translated source sentences for each monolingual target sentence. Our work follows most closely BIBREF19 , who investigate alternative decoding schemes for BT. Like BIBREF25 , they argue that BT through beam or greedy decoding leads to an overly regular domain on the source side, which poorly represents the diverse distribution of natural text.
Beyond the scope of this work, we briefly mention alternative techniques leveraging monolingual data, like forward translation BIBREF26 , BIBREF27 , or source copying BIBREF28 .
Training Data Tagging for NMT
Tags have been used for various purposes in NMT. Tags on the source sentence can indicate the target language in multi-lingual models BIBREF29 . BIBREF30 use tags in a similar fashion to control the formality of a translation from English to Japanese. BIBREF31 use tags to control gender in translation. Most relevant to our work, BIBREF32 use tags to mark source sentence domain in a multi-domain setting.
Experimental Setup
This section presents our datasets, evaluation protocols and model architectures. It also describes our back-translation procedure, as well as noising and tagging strategies.
Data
We perform our experiments on WMT18 EnDe bitext, WMT16 EnRo bitext, and WMT15 EnFr bitext respectively. We use WMT Newscrawl for monolingual data (2007-2017 for De, 2016 for Ro, 2007-2013 for En, and 2007-2014 for Fr). For bitext, we filter out empty sentences and sentences longer than 250 subwords. We remove pairs whose whitespace-tokenized length ratio is greater than 2. This results in about 5.0M pairs for EnDe, and 0.6M pairs for EnRo. We do not filter the EnFr bitext, resulting in 41M sentence pairs.
For monolingual data, we deduplicate and filter sentences with more than 70 tokens or 500 characters. Furthermore, after back-translation, we remove any sentence pairs where the back-translated source is longer than 75 tokens or 550 characters. This results in 216.5M sentences for EnDe, 2.2M for EnRo, 149.9M for RoEn, and 39M for EnFr. For monolingual data, all tokens are defined by whitespace tokenization, not wordpieces.
The DeEn model used to generate BT data has 28.6 SacreBleu on newstest12, the RoEn model used for BT has a test SacreBleu of 31.9 (see Table 4 .b), and the FrEn model used to generate the BT data has 39.2 SacreBleu on newstest14.
Evaluation
We rely on Bleu score BIBREF33 as our evaluation metric.
While well established, any slight difference in post-processing and Bleu computation can have a dramatic impact on output values BIBREF34 . For example, BIBREF35 report 33.3 Bleu on EnRo using unsupervised NMT, which at first seems comparable to our reported 33.4 SacreBleu from iterative TaggedBT. However, when we use their preprocessing scripts and evaluation protocol, our system achieves 39.2 Bleu on the same data, which is close to 6 points higher than the same model evaluated by SacreBleu.
We therefore report strictly SacreBleu , using the reference implementation from post2018call, which aims to standardize Bleu evaluation.
Architecture
We use the transformer-base and transformer-big architectures BIBREF2 implemented in lingvo BIBREF36 . Transformer-base is used for the bitext noising experiments and the EnRo experiments, whereas the transformer-big is used for the EnDe tasks with BT. Both use a vocabulary of 32k subword units. As an alternative to the checkpoint averaging used in edunov2018understanding, we train with exponentially weighted moving average (EMA) decay with weight decay parameter $\alpha =0.999$ BIBREF37 .
Transformer-base models are trained on 16 GPUs with synchronous gradient updates and per-gpu-batch-size of 4,096 tokens, for an effective batch size of 64k tokens/step. Training lasts 400k steps, passing over 24B tokens. For the final EnDe TaggedBT model, we train transformer-big similarly but on 128 GPUs, for an effective batch size of 512k tokens/step. A training run of 300M steps therefore sees about 150B tokens. We pick checkpoints with newstest2012 for EnDe and newsdev2016 for EnRo.
Noising
We focused on noised beam BT, the most effective noising approach according to BIBREF19 . Before training, we noised the decoded data BIBREF10 by applying 10% word-dropout, 10% word blanking, and a 3-constrained permutation (a permutation such that no token moves further than 3 tokens from its original position). We refer to data generated this way as NoisedBT. Additionally, we experiment using only the 3-constrained permutation and no word dropout/blanking, which we abbreviate as P3BT.
Tagging
We tag our BT training data by prepending a reserved token to the input sequence, which is then treated in the same way as any other token. We also experiment with both noising and tagging together, which we call Tagged Noised Back-Translation, or TaggedNoisedBT. This consists simply of prepending the $<$ BT $>$ tag to each noised training example.
An example training sentence for each of these set-ups can be seen in Table 1 . We do not tag the bitext, and always train on a mix of back-translated data and (untagged) bitext unless explicitly stated otherwise.
Results
This section studies the impact of training data noise on translation quality, and then presents our results with TaggedBT on EnDe and EnRo.
Noising Parallel Bitext
We first show that noising EnDe bitext sources does not seriously impact the translation quality of the transformer-base baseline. For each sentence pair in the corpus, we flip a coin and noise the source sentence with probability $p$ . We then train a model from scratch on this partially noised dataset. Table 2 shows results for various values of $p$ . Specifically, it presents the somewhat unexpected finding that even when noising 100% of the source bitext (so the model has never seen well-formed English), Bleu on well-formed test data only drops by 2.5.
This result prompts the following line of reasoning about the role of noise in BT: (i) By itself, noising does not add meaningful signal (or else it would improve performance); (ii) It also does not damage the signal much; (iii) In the context of back-translation, the noise could therefore signal whether a sentence were back-translated, without significantly degrading performance.
Tagged Back-Translation for EnDe
We compare the results of training on a mixture of bitext and a random sample of 24M back-translated sentences in Table 3 .a, for the various set-ups of BT described in sections "Conclusion" and "Future Work" . Like edunov2018understanding, we confirm that BT improves over bitext alone, and noised BT improves over standard BT by about the same margin. All methods of marking the source text as back-translated (NoisedBT, P3BT, TaggedBT, and TaggedNoisedBT) perform about equally, with TaggedBT having the highest average Bleu by a small margin. Tagging and noising together (TaggedNoisedBT) does not improve over either tagging or noising alone, supporting the conclusion that tagging and noising are not orthogonal signals but rather different means to the same end.
Table 3 .b verifies our result at scale applying TaggedBT on the full BT dataset (216.5M sentences), upsampling the bitext so that each batch contains an expected 20% of bitext. As in the smaller scenario, TaggedBT matches or slightly out-performs NoisedBT, with an advantage on seven test-sets and a disadvantage on one. We also compare our results to the best-performing model from edunov2018understanding. Our model is on par with or slightly superior to their result, out-performing it on four test sets and under-performing it on two, with the largest advantage on Newstest2018 (+1.4 Bleu).
As a supplementary experiment, we consider training only on BT data, with no bitext. We compare this to training only on NoisedBT data. If noising in fact increases the quality or diversity of the data, one would expect the NoisedBT data to yield higher performance than training on unaltered BT data, when in fact it has about 1 Bleu lower performance (Table 3 .a, “BT alone" and “NoisedBT alone").
We also compare NoisedBT versus TaggedNoisedBT in a set-up where the bitext itself is noised. In this scenario, the noise can no longer be used by the model as an implicit tag to differentiate between bitext and synthetic BT data, so we expect the TaggedNoisedBT variant to perform better than NoisedBT by a similar margin to NoisedBT's improvement over BT in the unnoised-bitext setting. The last sub-section of Table 3 .a confirms this.
Tagged Back-Translation for EnRo
We repeat these experiments for WMT EnRo (Table 4 ). This is a much lower-resource task than EnDe, and thus can benefit more from monolingual data. In this case, NoisedBT is actually harmful, lagging standard BT by -0.6 Bleu. TaggedBT closes this gap and passes standard BT by +0.4 Bleu, for a total gain of +1.0 Bleu over NoisedBT.
Tagged Back-Translation for EnFr
We performed a minimal set of experiments on WMT EnFr, which are summarized in Table 5 . This is a much higher-resource language pair than either EnRo or EnDe, but BIBREF19 demonstrate that noised BT (using sampling) can still help in this set-up. In this case, we see that BT alone hurts performance compared to the strong bitext baseline, but NoisedBT indeed surpasses the bitext model. TaggedBT out-performs all other methods, beating NoisedBT by an average of +0.3 Bleu over all test sets.
It is worth noting that our numbers are lower than those reported by BIBREF19 on the years they report (36.1, 43.8, and 40.9 on 2013, 2014, and 2015 respectively). We did not investigate this result. We suspect that this is an error/inoptimlaity in our set-up, as we did not optimize these models, and ran only one experiment for each of the four set-ups. Alternately, sampling could outperform noising in the large-data regime.
Iterative Tagged Back-Translation
We further investigate the effects of TaggedBT by performing one round of iterative back-translation BIBREF38 , BIBREF39 , BIBREF40 , and find another difference between the different varieties of BT: NoisedBT and TaggedBT allow the model to bootstrap improvements from an improved reverse model, whereas standard BT does not. This is consistent with our argument that data tagging allows the model to extract information out of each data set more effectively.
For the purposes of this paper we call a model trained with standard back-translation an Iteration-1 BT model, where the back-translations were generated by a model trained only on bitext. We inductively define the Iteration-k BT model as that model which is trained on BT data generated by an Iteration-(k-1) BT model, for $k>1$ . Unless otherwise specified, any BT model mentioned in this paper is an Iteration-1 BT model.
We perform these experiments on the English-Romanian dataset, which is smaller and thus better suited for this computationally expensive process. We used the (Iteration-1) TaggedBT model to generate the RoEn back-translated training data. Using this we trained a superior RoEn model, mixing 80% BT data with 20% bitext. Using this Iteration-2 RoEn model, we generated new EnRo BT data, which we used to train the Iteration-3 EnRo models. SacreBleu scores for all these models are displayed in Table 4 .
We find that the iteration-3 BT models improve over their Iteration-1 counterparts only for NoisedBT (+1.0 Bleu, dev+test avg) and TaggedBT (+0.7 Bleu, dev+test avg), whereas the Iteration-3 BT model shows no improvement over its Iteration-1 counterpart (-0.1 Bleu, dev+test avg). In other words, both techniques that (explicitly or implicitly) tag synthetic data benefit from iterative BT. We speculate that this separation of the synthetic and natural domains allows the model to bootstrap more effectively from the increasing quality of the back-translated data while not being damaged by its quality issues, whereas the simple BT model cannot make this distinction, and is equally “confused" by the biases in higher or lower-quality BT data.
An identical experiment with EnDe did not see either gains or losses in Bleu from iteration-3 TaggedBT. This is likely because there is less room to bootstrap with the larger-capacity model. This said, we do not wish to read too deeply into these results, as the effect size is not large, and neither is the number of experiments. A more thorough suite of experiments is warranted before any strong conclusions can be made on the implications of tagging on iterative BT.
Analysis
In an attempt to gain further insight into TaggedBT as it compares with standard BT or NoisedBT, we examine attention matrices in the presence of the back translation tag and measure the impact of the tag at decoding time.
Attention Entropy and Sink-Ratio
To understand how the model treats the tag and what biases it learns from the data, we investigate the entropy of the attention probability distribution, as well as the attention captured by the tag.
We examine decoder attention (at the top layer) on the first source token. We define Attention Sink Ratio for index $j$ ( $\textrm {ASR}_j$ ) as the averaged attention over the $j$ th token, normalized by uniform attention, i.e. $ \textrm {ASR}_j(x, \hat{y}) = \frac{1}{|\hat{y}|} \sum _{i = 1}^{ | \hat{y} | } \frac{\alpha _{ij}}{\tilde{\alpha }} $
where $\alpha _{ij}$ is the attention value for target token $i$ in hypothesis $\hat{y}$ over source token $j$ and $\tilde{\alpha } = \frac{1}{|x|}$ corresponds to uniform attention. We examine ASR on text that has been noised and/or tagged (depending on the model), to understand how BT sentences are treated during training. For the tagged variants, there is heavy attention on the tag when it is present (Table ), indicating that the model relies on the information signalled by the tag.
Our second analysis probes word-for-word translation bias through the average source-token entropy of the attention probability model when decoding natural text. Table reports the average length-normalized Shannon entropy: $ \tilde{\textrm {}{H}(x, \hat{y}) = - \frac{1}{|\hat{y}|} \sum _{i=1}^{| \hat{y}|} \frac{1}{\log |x|} \sum _{j=1}^{|x|}\alpha _{ij} \log (\alpha _{ij})  The entropy of the attention probabilities from the model trained on BT data is the clear outlier. This low entropy corresponds to a concentrated attention matrix, which we observed to be concentrated on the diagonal (See Figure~\ref {pretty-attention:ende-bt} and \ref {pretty-attention:enro-bt}). This could indicate the presence of word-by-word translation, a consequence of the harmful part of the signal from back-translated data. The entropy on parallel data from the NoisedBT model is much higher, corresponding to more diffuse attention, which we see in Figure~\ref {pretty-attention:ende-nbt} and ~\ref {pretty-attention:enro-nbt}. In other words, the word-for-word translation biases in BT data, that were incorporated into the BT model, have been manually undone by the noise, so the model^{\prime }s understanding of how to decode parallel text is not corrupted. We see that TaggedBT leads to a similarly high entropy, indicating the model has learnt this without needing to manually ``break" the literal-translation bias. As a sanity check, we see that the entropy of the P3BT model^{\prime }s attention is also high, but is lower than that of the NoisedBT model, because P3 noise is less destructive. The one surprising entry on this table is probably the low entropy of the TaggedNoisedBT. Our best explanation is that TaggedNoisedBT puts disproportionately high attention on the sentence-end token, with 1.4x the \textrm {ASR}_{|x|} that TaggedBT has, naturally leading to lower entropy. }$

Question: What datasets was the method evaluated on?
Answer: WMT18 EnDe bitext, WMT16 EnRo bitext, WMT15 EnFr bitext, We perform our experiments on WMT18 EnDe bitext, WMT16 EnRo bitext, and WMT15 EnFr bitext respectively. We use WMT Newscrawl for monolingual data (2007-2017 for De, 2016 for Ro, 2007-2013 for En, and 2007-2014 for Fr). For bitext, we filter out empty sentences and sentences longer than 250 subwords. We remove pairs whose whitespace-tokenized length ratio is greater than 2. This results in about 5.0M pairs for EnDe, and 0.6M pairs for EnRo. We do not filter the EnFr bitext, resulting in 41M sentence pairs.