Tagged Back-translation Revisited: Why Does It Really Work?
Benjamin Marie
Raphael Rubino
Atsushi Fujita
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{bmarie, raphael.rubino, atsushi.fujita}@nict.go.jp

Abstract
In this paper, we show that neural machine translation (NMT) systems trained onlarge back-translated data overfit some of
the characteristics of machine-translated texts.
Such NMT systems better translate humanproduced translations, i.e., translationese, but
may largely worsen the translation quality oforiginal texts. Our analysis reveals that adding
a simple tag to back-translations prevents thisquality degradation and improves on average
the overall translation quality by helping the
NMT system to distinguish back-translated
data from original parallel data during training.
We also show that, in contrast to high-resource
configurations, NMT systems trained in lowresource settings are much less vulnerable tooverfit back-translations. We conclude that the
back-translations in the training data should always be tagged especially when the origin of
the text to be translated is unknown.

1

Introduction

During training, neural machine translation (NMT)
can leverage a large amount of monolingual datain the target language. Among existing ways of
exploiting monolingual data in NMT, the so-called
back-translation of monolingual data (Sennrichet al., 2016a) is undoubtedly the most prevalentone, as it remains widely used in state-of-the-art
NMT systems (Barrault et al., 2019). NMT systems
trained on back-translated data can generate more
fluent translations (Sennrich et al., 2016a) thanks to
the use of much larger data in the target language tobetter train the decoder, especially for low-resource
conditions where only a small quantity of parallel
training data is available. However, the impact of
the noisiness of the synthetic source sentences generated by NMT largely remains unclear and understudied. Edunov et al. (2018) even showed that
introducing synthetic noise in back-translations actually improves translation quality and enables theuse of a much larger quantity of back-translated
data for further improvements in translation quality.
More recently, Caswell et al. (2019) empiricallydemonstrated that adding a unique token at the
beginning of each back-translation acts as a tag
that helps the system during training to differentiate back-translated data from the original parallel
training data and is as effective as introducing synthetic noise for improving translation quality. It isalso much simpler since it requires only one editing operation, adding the tag, and non-parametric.
However, it is not fully understood why adding a
tag has such a significant impact and to what extent
it helps to distinguish back-translated data from theoriginal parallel data.
In this paper, we report on the impact of tagging back-translations in NMT, focusing on thefollowing research questions (see Section 2 for our
motivation).
Q1. Do NMT systems trained on large backtranslated data capture some of the characteristics of human-produced translations, i.e.,translationese?
Q2. Does a tag for back-translations really help differentiate translationese from original texts?
Q3. Are NMT systems trained on back-translation
for low-resource conditions as sensitive totranslationese as in high-resource conditions?

2

Motivation

During the training with back-translated data (Sennrich et al., 2016a), we can expect the NMT system
to learn the characteristics of back-translations, i.e.,translations generated by NMT, and such characteristics will be consequently exhibited at test time.
However, translating translations is a rather artificial task, whereas users usually want to perform
translation of original texts. Nonetheless, manyof the test sets used by the research community
for evaluating MT systems actually contain a large
portion of texts that are translations produced by humans, i.e., translationese. Translationese texts are
known to be much simpler, with a lower mean sentence length and more standardized than original
texts (Laviosa-Braithwaite, 1998). These characteristics overlap with those of translations generated
by NMT systems that have been shown simpler,shorter, and to exhibit a less diverse vocabulary
than original texts (Burlot and Yvon, 2018). Thesesimilarities raise Q1.
Caswell et al. (2019) hypothesized that tagging
back-translations helps the NMT system during
training to make some distinction between the backtranslated data and the original parallel data. Even
though the effectiveness of a tag has been empirically demonstrated, the nature of this distinction
remains unclear. Thus, we pose Q2.
The initial motivation for back-translation is toimprove NMT for low-resource language pairs byaugmenting the training data. Therefore, we verify whether our answers to Q1 and Q2 for highresource conditions are also valid in low-resource
conditions, answering Q3.

3

Experiments

3.1

Data

As parallel data for training our NMT systems,
we used all the parallel data provided for theshared translation tasks of WMT191 for English–
German (en-de), excluding the Paracrawl corpus,
and WMT152 for English–French (en-fr).3 As
monolingual data for each of English, German, andFrench to be used for back-translation, we concatenated all the News Crawl corpora provided by
WMT, and randomly extracted 25M sentences. For
our simulation of low-resource conditions, we randomly sub-sampled 200k sentence pairs from theparallel data to train NMT systems and used thesesystems to back-translate 1M sentences randomly
sub-sampled from the monolingual data. For validation, i.e., selecting the best model after training, wechose newstest2016 for en-de and newstest2013 foren-fr, since they are rather balanced on their sourceside between translationese and original texts. For1

http://www.statmt.org/wmt19/
translation-task.html
2
http://www.statmt.org/wmt15/
translation-task.html
3
After pre-processing and cleaning, we obtained 5.2M and
32.8M sentence pairs for en-de and en-fr, respectively.

evaluation, since most of the WMT test sets are
made of both original and translationese texts, weused all the newstest sets, from WMT10 to WMT19
for en-de, and from WMT08 to WMT15 for en-fr.4
All our data were pre-processed in the same
way: we performed tokenization and truecasing
with Moses (Koehn et al., 2007).
3.2

NMT Systems

For NMT, we used the Transformer (Vaswani et al.,2017) implemented in Marian (Junczys-Dowmunt
et al., 2018) with standard hyper-parameters fortraining a Transformer base model.5 To compress
the vocabulary, we learned 32k byte-pair encoding(BPE) operations (Sennrich et al., 2016b) for eachside of the parallel training data.
The back-translations were generated throughdecoding with Marian the sampled monolingual
sentences using beam search with a beam sizeof 12 and a length normalization of 1.0. The
back-translated data were then concatenated tothe original parallel data and a new NMT model
was trained from scratch using the same hyperparameters used to train the model that generated
the back-translations.
We evaluated all systems with BLEU (Papineni
et al., 2002) computed by sacreBLEU (Post, 2018).
To evaluate only on the part of the test set that haveoriginal text or translationese on the source side, weused the --origlang option of sacreBLEU withthe value “non-L1” for translationese texts and “L1”for original texts, where L1 is the source language,
and report on their respective BLEU scores.6
3.3

Results in Resource-Rich Conditions

Our results with back-translations (BT) and tagged
back-translations (T-BT) are presented in Table 1.
When using BT, we consistently observed a dropof BLEU scores for original texts for all the translations tasks, with the largest drop of 12.1 BLEU
points (en→fr, 2014). Conversely, BLEU scoresfor translationese texts were improved for mosttasks, with the largest gain of 10.4 BLEU points
4
For WMT14, we used the “full” version instead of thedefault filtered version in sacreBLEU that does not contain
information on the origin of the source sentences.
5
The full list of hyper-parameters is provided in the supplementary material (Appendix A).
6sacreBLEU signatures where “L1” and “L2” respectively
indicates a two-letter identifier for the source and target
languages of either de-en, en-de, fr-en, or en-fr, and “XXX”
the name of the test set: BLEU+case.mixed+lang.L1L2+numrefs.1+{origlang.L1,origlang.nonL2}+smooth.exp+test.XXX+tok.13a+version.1.4.2

all

de→en
o

n-o

all

en→de
o

n-o

BT

2010
2011
2012
2013
2014
2015
2017
2018
2019

28.9 (+0.5)
25.3 (-0.3)
27.1 (+0.3)
30.3 (+0.3)
32.8 (+2.2)
33.8 (+2.4)
35.5 (+3.0)
43.9 (+4.6)-

33.2 (-0.9)
29.9 (-1.0)
27.9 (-1.6)
34.7 (-1.6)
27.4 (-2.5)
22.5 (-1.9)
27.2 (-1.1)
32.0 (-1.0)
33.1 (-1.5)

27.9 (+0.7)
24.2 (-0.2)
27.0 (+0.7)
29.2 (+0.6)
36.8 (+7.0)
39.5 (+5.5)
42.8 (+7.4)
53.8 (+10.4)-

21.8 (-2.3)
19.9 (-1.4)
20.4 (-1.2)
23.8 (-1.9)
25.4 (-0.5)
27.2 (-1.1)
26.4 (-0.1)
38.0 (-1.4)-

24.6 (-5.7)
23.8 (-1.9)
24.5 (-4.6)
25.1 (-2.8)
23.2 (-3.3)
28.1 (-2.9)
26.3 (-3.6)
38.9 (-5.9)
31.4 (-4.8)

21.0 (-1.2)
19.0 (-1.1)
19.3 (-0.2)
23.6 (-1.7)
27.9 (+2.7)
24.7 (+1.9)
25.5 (+3.3)
35.0 (+3.8)
-

T-BT

2010
2011
2012
2013
2014
2015
2017
2018
2019

29.5 (+1.1)
26.4 (+0.8)
28.1 (+1.3)
30.8 (+0.8)
32.4 (+1.8)
33.9 (+2.5)
35.5 (+3.0)
43.2 (+3.9)-

34.4 (+0.3)
31.7 (+0.8)
30.2 (+0.7)
36.0 (-0.3)
29.6 (-0.3)
24.9 (+0.5)
28.1 (-0.2)
33.0 (+0.0)
35.0 (+0.4)

28.4 (+1.2)
25.2 (+0.8)
27.7 (+1.4)
29.6 (+1.0)
33.8 (+4.0)
37.7 (+3.7)
41.2 (+5.8)
50.4 (+7.0)-

25.0 (+0.9)
22.1 (+0.8)
22.8 (+1.2)
26.4 (+0.7)
27.9 (+2.0)
29.9 (+1.6)
28.7 (+2.2)
41.8 (+2.4)-

30.5 (+0.2)
25.8 (+0.1)
30.0 (+0.9)
28.1 (+0.2)
26.7 (+0.2)
32.1 (+1.1)
30.7 (+0.8)
45.6 (+0.8)
37.6 (+1.4)

23.4 (+1.2)
21.0 (+0.9)
20.9 (+1.4)
26.1 (+0.8)
29.4 (+4.2)
25.6 (+2.8)
26.0 (+3.8)
35.5 (+4.3)-

Systemtest set

all

fr→en
o

n-o

all

en→fr
o

n-o

BT

2008
2009
2010
2011
2012
2014
2015

22.9 (-1.7)
26.5 (-2.3)
29.3 (-1.4)
29.4 (-1.9)
29.7 (-1.4)
36.6 (+0.6)
36.2 (+0.0)

27.9 (-2.6)
41.1 (-5.3)
27.4 (-7.8)
29.3 (-4.7)
34.3 (-4.3)
31.4 (-4.7)
40.9 (-3.1)

22.2 (-1.5)
23.9 (-1.6)
29.5 (+0.5)
29.4 (-1.1)
28.6 (-0.6)
40.3 (+5.6)
29.8 (+3.5)

23.2 (-0.2)
27.7 (+1.1)
28.2 (-0.5)
30.9 (+0.0)
28.4 (+1.1)
32.9 (-3.1)
35.7 (+1.7)

21.2 (-3.3)
22.7 (-2.0)
22.5 (-11.1)
36.7 (-8.2)
26.3 (-4.1)
26.1 (-12.1)
25.1 (-4.4)

23.6 (+0.5)
28.4 (+1.4)
29.8 (+2.5)
29.3 (+2.1)
29.0 (+2.5)
39.6 (+6.1)
44.9 (+6.5)

T-BT

2008
2009
2010
2011
2012
2014
2015

24.5 (-0.1)
28.9 (+0.1)
31.2 (+0.5)
31.8 (+0.5)
31.8 (+0.7)
37.3 (+1.3)
36.6 (+0.4)

29.5 (-1.0)
46.4 (+0.0)
35.1 (-0.1)
33.3 (-0.7)
38.3 (-0.3)
36.1 (+0.0)
43.2 (-0.8)

23.7 (+0.0)
25.7 (+0.2)
29.6 (+0.6)
31.4 (+0.9)
30.1 (+0.9)
37.2 (+2.5)
27.9 (+1.6)

23.8 (+0.4)
27.3 (+0.7)
30.0 (+1.3)
31.6 (+0.7)
28.9 (+1.6)
38.2 (+2.2)
36.0 (+2.0)

25.1 (+0.6)
25.1 (+0.4)
34.1 (+0.5)
45.3 (+0.4)
31.9 (+1.5)
39.7 (+1.5)
30.7 (+1.2)

23.5 (+0.4)
27.7 (+0.7)
28.9 (+1.6)
28.0 (+0.8)
28.1 (+1.6)
36.5 (+3.0)
41.2 (+2.8)

Systemtest set

Table 1: BLEU scores for NMT systems trained with back-translations (BT) and tagged back-translations (T-BT)
for each origin of the source text: original (o) or translationese (n-o). The values in parentheses are the differences
between the BLEU scores of the evaluated system and the vanilla system trained without any back-translated data.

(de→en, 2018). These results give an answer to Q1:
NMT overfits back-translations, potentially due totheir much larger size than the original parallel
data used for training. Interestingly, using backtranslations does not consistently improve translation quality. We assume that newstest sets may
manifest some different characteristics of translationese from one year to another.
Prepending a tag (T-BT) had a strong impact on
the translation quality for original texts, recovering
or even surpassing the quality obtained by the NMT
system without back-translated data, always beating BT. The large improvements of BLEU scores
over BT show that a tag helps in identifying translationese (answer for Q2). In the supplementarymaterial (Appendix B), we present additional results obtained using more back-translations (up to
150M sentences) showing a similar impact of tags.

However, while a tag in such a configuration prevents an even larger drop of the BLEU scores, it is
not sufficient to attain a BLEU score similar to theconfigurations that use less back-translations.
Interestingly, the best NMT system was not always the same depending on the translation direction and the origin of the test sets. It is thus possible to select either of the models to obtain the besttranslation quality given the origin of the source
sentences, according to the results on the validation
set for instance.7

7
Since this observation is rather secondary, we present
results for best model selection in the supplementary material(Appendix C). Note also that these BLEU scores can potentially be further increased by using a validation set whose
source side is either original texts or translationese respectively to translate original texts or translationese at test time.

SystemBT

T-BT

System

BT

T-BT

all

de→en
o

n-o

all

en→de
o

n-o

2010
2011
2012
2013
2014
2015
2017
2018
2019

24.1 (+9.5)
21.0 (+8.1)
22.2 (+8.6)
25.0 (+9.0)
25.1 (+11.3)
27.1 (+11.8)
27.6 (+12.5)
34.3 (+16.4)-

27.1 (+12.4)
23.9 (+10.3)
21.6 (+8.7)
28.1 (+9.6)
20.9 (+8.4)
18.4 (+6.9)
21.5 (+8.2)
25.2 (+10.7)
26.1 (+11.9)

23.3 (+8.8)
20.3 (+7.6)
22.3 (+8.5)
24.1 (+8.7)
27.7 (+13.3)
31.0 (+14.3)
32.4 (+16.2)
41.0 (+21.1)-

18.0 (+2.9)
16.3 (+2.3)
16.4 (+2.5)
19.6 (+2.9)
19.7 (+4.5)
21.5 (+4.0)
20.7 (+4.0)
29.3 (+6.7)-

21.6 (+2.7)
19.1 (+2.9)
19.8 (+2.6)
20.0 (+3.2)
18.7 (+3.3)
22.5 (+3.6)
20.8 (+2.7)
30.4 (+5.4)
24.8 (+4.8)

17.0 (+3.0)
15.6 (+2.1)
15.5 (+2.5)
19.5 (+2.8)
20.3 (+6.1)
18.3 (+5.0)
19.3 (+5.5)
26.3 (+8.3)
-

2010
2011
2012
2013
2014
2015
2017
2018
2019

24.4 (+9.8)
21.8 (+8.9)
22.8 (+9.2)
25.9 (+9.9)
25.1 (+11.3)
27.0 (+11.7)
27.8 (+12.7)
34.2 (+16.3)-

27.4 (+12.7)
25.3 (+11.7)
22.9 (+10.0)
29.4 (+10.9)
22.1 (+9.6)
19.4 (+7.9)
22.5 (+9.2)
26.4 (+11.9)
26.8 (+12.6)

23.6 (+9.1)
20.9 (+8.2)
22.8 (+9.0)
24.9 (+9.5)
26.8 (+12.4)
30.5 (+13.8)
32.0 (+15.8)
39.8 (+19.9)-

18.8 (+3.7)
16.8 (+2.8)
17.2 (+3.3)
20.2 (+3.5)
20.1 (+4.9)
22.0 (+4.5)
21.1 (+4.4)
30.5 (+7.9)-

22.6 (+3.7)
20.2 (+4.0)
21.3 (+4.1)
20.5 (+3.7)
19.5 (+4.1)
23.5 (+4.6)
22.2 (+4.1)
32.9 (+7.9)
26.9 (+6.9)

17.7 (+3.7)
16.0 (+2.5)
16.1 (+3.1)
20.2 (+3.5)
20.6 (+6.4)
18.2 (+4.9)
19.2 (+5.4)
25.5 (+7.5)
-

all

fr→en
o

n-o

all

en→fr
o

n-o

2008
2009
2010
2011
2012
2014
2015

20.5 (+2.8)
24.0 (+3.3)
26.4 (+4.7)
26.2 (+3.4)
26.4 (+4.0)
32.2 (+7.8)
30.0 (+5.9)

26.3 (+1.6)
39.7 (+5.4)
28.3 (+4.0)
26.9 (+0.7)
31.4 (+1.3)
28.9 (+4.5)
34.0 (+5.0)

19.7 (+3.0)
21.2 (+3.1)
25.4 (+5.0)
26.0 (+4.1)
25.2 (+4.6)
33.6 (+10.5)
24.8 (+7.1)

21.3 (+4.1)
24.8 (+6.1)
26.1 (+6.0)
28.1 (+6.3)
26.4 (+6.3)
31.4 (+7.6)
29.9 (+8.0)

21.2 (+3.2)
21.6 (+5.1)
29.9 (+6.3)
38.1 (+8.6)
27.2 (+5.9)
28.9 (+4.5)
23.7 (+5.4)

21.3 (+4.3)
25.2 (+6.2)
24.9 (+5.8)
25.5 (+5.8)
26.1 (+6.3)
32.9 (+10.3)
35.5 (+10.1)

2008
2009
2010
2011
2012
2014
2015

21.3 (+3.6)
24.6 (+3.9)
27.0 (+5.3)
27.4 (+4.6)
27.3 (+4.9)
31.8 (+7.4)
30.6 (+6.5)

27.5 (+2.8)
41.6 (+7.3)
29.6 (+5.3)
29.7 (+3.5)
33.3 (+3.2)
29.9 (+5.5)
35.6 (+6.6)

20.4 (+3.7)
21.5 (+3.4)
25.7 (+5.3)
26.7 (+4.8)
25.7 (+5.1)
32.1 (+9.0)
23.7 (+6.0)

20.8 (+3.6)
23.7 (+5.0)
25.6 (+5.5)
27.3 (+5.5)
25.6 (+5.5)
31.0 (+7.2)
29.2 (+7.3)

21.7 (+3.7)
20.8 (+4.3)
29.8 (+6.2)
36.9 (+7.4)
26.8 (+5.5)
30.4 (+6.0)
24.0 (+5.7)

20.6 (+3.6)
24.1 (+5.1)
24.3 (+5.2)
24.8 (+5.1)
25.2 (+5.4)
30.9 (+8.3)
34.2 (+8.8)

test set

test set

Table 2: BLEU scores for low-resource configurations.

3.4

Results in Low-Resource Conditions

In low-resource conditions, as reported in Table 2,
the translation quality can be notably improvedby adding back-translations. Using BT, we observed improvements of BLEU scores ranging from
0.7 (fr→en, 2011) to 12.4 (de→en, 2010) BLEU
points for original texts and from 2.1 (en→de,
2011) to 21.1 (de→en, 2018) BLEU points fortranslationese texts. These results remain in linewith one of the initial motivations for using backtranslation: improving translation quality in lowresource conditions. In this setting without backtranslated data, the data in the target language is too
small for the NMT system to learn reasonably good
representations for the target language. Adding 5
times more data in the target language, throughback-translation, clearly helps the systems without
any negative impact of the noisiness of the backtranslations that were generated by the initial sys-tem. We assume here that since the quality of the
back-translations is very low, their characteristics
are quite different from the ones of translationese
texts. This is confirmed by our observation that
adding the tag has only a negligible impact on the
BLEU scores for all the tasks (answer to Q3).
3.5

Tagged Test Sets

A tag on back-translations helps identifying translationese during NMT training. Thus, adding thesame tag on the test sets should have a very different impact depending on the origin of the source
sentences. If we tag original sentences and decodethem with a T-BT model, then we enforce the decoding of translationese. Since we mislead the decoder, translation quality should drop. On the other
hand, by tagging translationese sentences, we help
the decoder that can now rely on the tag to be very
confident that the text to decode is translationese.
Our results presented in Table 3 confirm theseSystemtagged original
tagged non-originalde→en2017 2018

en→de2017 2018

fr→en
2012 2015

en→fr
2012 2015

-2.0
+1.6

-5.9
+0.8

-7.5
-3.1

-10.1
-0.3

-2.6
+3.4

-9.6
+1.6

-4.9
+1.4

-11.1
+3.6

Table 3: Results with tagged test sets, either original or non-original, decoded with the T-BT model in the highresource condition. Delta BLEU scores are computed relatively to the configurations with untagged test sets.

assumptions. We observed a drop of BLEU scores
when decoding tagged original texts with the T-BT
model, while we saw an improvement of translation
quality for 6 out of 8 test sets when decoding tagged
translationese texts. The remaining 2 test sets forwhich we did not observed any improvements are
newstest2012 for both translation directions of enfr. It potentially indicates a mismatch between thecharacteristics of translationese in newstest2012
and those exhibited by back-translations used fortraining the T-BT model.

4Discussions

We empirically demonstrated that training NMT
on back-translated data overfits some of its characteristics that are partly similar to those of translationese. Using back-translation improves translation quality for translationese texts but worsens
it for original texts. Previous work (Graham et al.,2019; Zhang and Toral, 2019) showed that stateof-the-art NMT systems are better in translatingtranslationese than original texts. Our results showthat this is partly due to the use of back-translations
which is also confirmed by concurrent and independent work (Bogoychev and Sennrich, 2019; Edunov
et al., 2019). Adding a tag to back-translations prevents a large drop of translation quality on original texts while improvements of translation quality for translationese texts remain and may be further boosted by tagging test sentences at decoding time. Moreover, in low-resource conditions,
we show that the overall tendency is significantly
different from the high-resource conditions: backtranslation improves translation quality for bothtranslationese and original texts while adding a tag
to back-translations has only a little impact.
We conclude from this study that training NMT
on back-translated data, in high-resource conditions, remains reasonable when the user knows inadvance that the system will be used to translatetranslationese texts. If the user does not know it apriori, a tag should be added to back-translations
during training to prevent a possible large drop oftranslation quality.

For future work, following the work on automatic identification of translationese (Rabinovich
and Wintner, 2015; Rubino et al., 2016), we plan
to investigate the impact of tagging translationese
texts inside parallel training data, such as parallel
sentences collected from the Web.

Acknowledgments
We would like to thank the reviewers for theiruseful comments and suggestions. A part ofthis work was conducted under the program “Research and Development of Enhanced Multilingualand Multipurpose Speech Translation System” of
the Ministry of Internal Affairs and Communications (MIC), Japan. Benjamin Marie was partly
supported by JSPS KAKENHI Grant Number
20K19879 and the tenure-track researcher start-up
fund in NICT. Atsushi Fujita was partly supported
by JSPS KAKENHI Grant Number 19H05660.

ReferencesLoı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà,Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn,
Shervin Malmasi, Christof Monz, Mathias Müller,
Santanu Pal, Matt Post, and Marcos Zampieri. 2019.
Findings of the 2019 Conference on Machine Translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared
Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics.
Nikolay Bogoychev and Rico Sennrich. 2019. Domain, translationese and noise in synthetic data
for neural machine translation. arXiv preprint
arXiv:1911.03362.
Franck Burlot and François Yvon. 2018. Using monolingual data in neural machine translation: a systematic study. In Proceedings of the Third Conference
on Machine Translation: Research Papers, pages
144–155, Brussels, Belgium. Association for Computational Linguistics.
Isaac Caswell, Ciprian Chelba, and David Grangier.
2019. Tagged back-translation. In Proceedings ofthe Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 53–63, Florence,
Italy. Association for Computational Linguistics.

Sergey Edunov, Myle Ott, Michael Auli, and David
Grangier. 2018. Understanding back-translation atscale. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing,pages 489–500, Brussels, Belgium. Association for
Computational Linguistics.
Sergey Edunov, Myle Ott, Marc’Aurelio Ranzato, andMichael Auli. 2019. On the evaluation of machinetranslation systems trained with back-translation.
arXiv preprint arXiv:1908.05204.
Yvette Graham, Barry Haddow, and Philipp Koehn.
2019. Translationese in machine translation evaluation. arXiv preprint arXiv:1906.09833.
Marcin Junczys-Dowmunt, Roman Grundkiewicz,
Tomasz Dwojak, Hieu Hoang, Kenneth Heafield,
Tom Neckermann, Frank Seide, Ulrich Germann,
Alham Fikri Aji, Nikolay Bogoychev, André F. T.
Martins, and Alexandra Birch. 2018. Marian: Fast
neural machine translation in C++. In Proceedingsof ACL 2018, System Demonstrations, pages 116–
121, Melbourne, Australia. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra
Constantin, and Evan Herbst. 2007. Moses: Open
source toolkit for statistical machine translation. In
Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion
Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
Sara Laviosa-Braithwaite. 1998. Universals of translation. Routledge encyclopedia of translation studies.
London: Routledge, pages 288–291.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings ofthe 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia,
USA. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU
scores. In Proceedings of the Third Conference on
Machine Translation: Research Papers, pages 186–
191, Brussels, Belgium. Association for Computational Linguistics.
Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. Transactionsof the Association for Computational Linguistics,
3:419–432.
Raphael Rubino, Ekaterina Lapshinova-Koltunski, andJosef van Genabith. 2016. Information density andquality estimation features as translationese indicators for human translation classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational

Linguistics: Human Language Technologies, pages
960–970, San Diego, USA. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016a. Improving neural machine translation models with monolingual data. In Proceedings of the54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
86–96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016b. Neural machine translation of rare words
with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715–
1725, Berlin, Germany. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In I. Guyon, U. V. Luxburg, S. Bengio,
H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
Mike Zhang and Antonio Toral. 2019. The effect oftranslationese in machine translation test sets. In
Proceedings of the Fourth Conference on MachineTranslation (Volume 1: Research Papers), pages 73–
81, Florence, Italy. Association for Computational
Linguistics.

A

NMT system hyper-parameters

For training NMT systems with Marian 1.7.6(1d4ba73), we used the hyper-parameters, on 8
GPUs, presented by Table 4 and kept the remaining
ones with their default values.

--type transformer--train-sets para.L1 para.L2--model model.npz --max-length
150 --mini-batch-fit--valid-freq 5000 --save-freq
5000 --workspace 4000--disp-freq 500 --valid-sets
dev.bpe32k.L1 dev.bpe32k.L2--beam-size 12 --normalize=1--valid-mini-batch 16--overwrite --early-stopping
5 --cost-type=ce-mean-words--valid-metrics ce-mean-words
bleu --keep-best--enc-depth 6 --dec-depth
6 --transformer-dropout
0.1 --learn-rate 0.0003--lr-warmup 16000--lr-decay-inv-sqrt 16000--label-smoothing 0.1--dim-vocabs 32000 32000--optimizer-params 0.9 0.98
1e-09 --clip-norm 5 --sync-sgd--exponential-smoothing
Table 4: Parameters of Marian used for training our
NMT systems.

B

Experiments with Larger Quantity of
Back-translaitons

Table 5 presents the results using much larger backtranslations in the high-resource conditions.

C

Best Model Selection

As discussed in Section 3.3, among the originalmodel, the one trained with back-translation (BT),and the one trained with tagged back-translation (TBT), the best-performing model is not always thesame depending on the translation direction. For
de→en and en→de, the best model is always T-BT.
However, for fr→en, the system that does not use
any back-translation is the best to translate original texts while T-BT is the best for translationese
texts. For en→fr, the best system for translatingtranslationese texts is BT while the best system fortranslating original texts is T-BT. This selection
is performed by evaluating the translation quality
for each model on the validation sets original andtranslationese texts.
By applying this selection strategy, we can significantly improve the overall translation quality

for given test sets, as reported in Table 6.

all

de→en
o

n-o

all

en→de
o

n-o

BT

2010
2011
2012
2013
2014
2015
2017
2018
2019

28.7 (+0.3)
24.6 (-1.0)
26.4 (-0.4)
29.6 (-0.4)
32.4 (+1.8)
33.4 (+2.0)
34.6 (+2.1)
43.2 (+3.9)-

32.0 (-2.1)
29.2 (-1.7)
27.1 (-2.4)
33.1 (-3.2)
25.7 (-4.2)
21.2 (-3.2)
25.7 (-2.6)
30.1 (-2.9)
31.4 (-3.2)

27.9 (+0.7)
23.5 (-0.9)
26.2 (-0.1)
28.6 (+0.0)
37.3 (+7.5)
39.4 (+5.4)
42.2 (+6.8)
53.9 (+10.5)-

22.3 (-1.8)
19.9 (-1.4)
20.7 (-0.9)
23.8 (-1.9)
26.0 (+0.1)
27.4 (-0.9)
26.6 (+0.1)
38.1 (-1.3)-

25.8 (-4.5)
23.1 (-2.6)
25.2 (-3.9)
24.4 (-3.5)
23.4 (-3.1)
27.7 (-3.3)
25.9 (-4.0)
38.8 (-6.0)
32.1 (-4.1)

21.3 (-0.9)
19.1 (-1.0)
19.5 (+0.0)
23.7 (-1.6)
28.9 (+3.7)
25.7 (+2.9)
26.4 (+4.2)
35.4 (+4.2)
-

T-BT

2010
2011
2012
2013
2014
2015
2017
2018
2019

29.5 (+1.1)
25.9 (+0.3)
27.5 (+0.7)
30.7 (+0.7)
32.5 (+1.9)
33.7 (+2.3)
35.2 (+2.7)
43.4 (+4.1)-

34.1 (+0.0)
30.4 (-0.5)
28.8 (-0.7)
35.2 (-1.1)
28.2 (-1.7)
23.7 (-0.7)
27.3 (-1.0)
32.4 (-0.6)
34.3 (-0.3)

28.3 (+1.1)
24.8 (+0.4)
27.3 (+1.0)
29.6 (+1.0)
35.4 (+5.6)
38.3 (+4.3)
41.5 (+6.1)
51.5 (+8.1)-

24.9 (+0.8)
21.9 (+0.6)
22.7 (+1.1)
26.1 (+0.4)
28.2 (+2.3)
29.6 (+1.3)
28.3 (+1.8)
41.7 (+2.3)-

29.3 (-1.0)
26.0 (+0.3)
28.8 (-0.3)
27.4 (-0.5)
26.8 (+0.3)
31.1 (+0.1)
29.8 (-0.1)
45.0 (+0.2)
36.5 (+0.3)

23.7 (+1.5)
20.7 (+0.6)
21.1 (+1.6)
25.9 (+0.6)
30.0 (+4.8)
26.7 (+3.9)
26.3 (+4.1)
36.1 (+4.9)-

Systemtest set

all

fr→en
o

n-o

all

en→fr
o

n-o

BT

2008
2009
2010
2011
2012
2014
2015

20.8 (-3.8)
23.9 (-4.9)
27.2 (-3.5)
27.3 (-4.0)
26.8 (-4.3)
33.5 (-2.5)
31.7 (-4.5)

27.4 (-3.1)
38.3 (-8.1)
27.7 (-7.5)
27.3 (-6.7)
31.4 (-7.2)
28.8 (-7.3)
35.5 (-8.5)

19.8 (-3.9)
21.2 (-4.3)
27.0 (-2.0)
27.3 (-3.2)
25.7 (-3.5)
36.9 (+2.2)
27.1 (+0.8)

21.6 (-1.8)
26.4 (-0.2)
26.7 (-2.0)
28.9 (-2.0)
26.5 (-0.8)
29.9 (-6.1)
32.4 (-1.6)

17.5 (-7.0)
20.4 (-4.3)
19.2 (-14.4)
31.4 (-13.5)
22.2 (-8.2)
20.6 (-17.6)
18.4 (-11.1)

22.5 (-0.6)
27.3 (+0.3)
28.8 (+1.5)
28.2 (+1.0)
27.7 (+1.2)
39.4 (+5.9)
44.9 (+6.5)

T-BT

2008
2009
2010
2011
2012
2014
2015

24.7 (+0.1)
28.4 (-0.4)
31.2 (+0.5)
31.8 (+0.5)
31.6 (+0.5)
37.9 (+1.9)
36.2 (+0.0)

30.6 (+0.1)
45.3 (-1.1)
34.2 (-1.0)
32.7 (-1.3)
37.5 (-1.1)
35.6 (-0.5)
42.2 (-1.8)

23.8 (+0.1)
25.2 (-0.3)
29.8 (+0.8)
31.5 (+1.0)
30.2 (+1.0)
38.7 (+4.0)
28.3 (+2.0)

24.1 (+0.7)
27.7 (+1.1)
30.6 (+1.9)
31.6 (+0.7)
29.2 (+1.9)
38.5 (+2.5)
36.3 (+2.3)

25.6 (+1.1)
25.7 (+1.0)
34.5 (+0.9)
45.5 (+0.6)
31.9 (+1.5)
39.7 (+1.5)
30.2 (+0.7)

23.7 (+0.6)
28.0 (+1.0)
29.5 (+2.2)
28.0 (+0.8)
28.4 (+1.9)
37.0 (+3.5)
42.1 (+3.7)

Systemtest set

Table 5: BLEU scores for all the systems in the high-resource conditions using 150M back-translations or theentire news crawl corpus for en→fr (76.6M sentences).

Sys.

2008
fr→en
en→fr

2009
fr→en
en→fr

2010
fr→en
en→fr

2011
fr→en
en→fr

2012
fr→en
en→fr

2014
fr→en
en→fr

2015
fr→en
en→fr

vanillaBT
T-BT
selection

24.6
22.9
24.5
24.7

28.8
26.5
28.9
29.0

30.7
29.3
31.2
31.5

31.3
29.4
31.8
33.0

31.1
29.7
31.8
32.5

36.0
36.6
37.3
37.5

36.2
36.2
36.6
36.3

23.4
23.2
23.8
23.9

26.6
27.7
27.3
28.2

28.7
28.2
30.0
30.9

30.9
30.9
31.6
32.7

29.5
28.4
28.9
29.9

36.0
32.9
38.2
38.9

34.0
35.7
36.0
37.8

Table 6: BLEU scores for all the systems for en-fr on the overall test sets. “selection” denotes that decoding is
performed by using the best model given the origin of the source sentence.