SlowGuess/ACL_2020_OCR huggingface dataset:
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2019. On the evaluation of machine translation systems trained with back-translation. arXiv preprint arXiv:1908.05204.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation. arXiv preprint arXiv:1906.09833.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in  $\mathbf{C} + +$ . In Proceedings of ACL 2018, System Demonstrations, pages 116-121, Melbourne, Australia. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Sara Laviosa-Braithwaite. 1998. Universals of translation. Routledge encyclopedia of translation studies. London: Routledge, pages 288-291.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, USA. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419-432.
Raphael Rubino, Ekaterina Lapshinova-Koltunski, and Josef van Genabith. 2016. Information density and quality estimation features as translationese indicators for human translation classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages 960-970, San Diego, USA. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 73-81, Florence, Italy. Association for Computational Linguistics.
A NMT system hyper-parameters
For training NMT systems with Marian 1.7.6 (1d4ba73), we used the hyper-parameters, on 8 GPUs, presented by Table 4 and kept the remaining ones with their default values.
5995
Table 4: Parameters of Marian used for training our NMT systems.
B Experiments with Larger Quantity of Back-translaitons
Table 5 presents the results using much larger backtranslations in the high-resource conditions.
C Best Model Selection
As discussed in Section 3.3, among the original model, the one trained with back-translation (BT), and the one trained with tagged back-translation (T-BT), the best-performing model is not always the same depending on the translation direction. For de  $\rightarrow$  en and en  $\rightarrow$  de, the best model is always T-BT. However, for fr  $\rightarrow$  en, the system that does not use any back-translation is the best to translate original texts while T-BT is the best for translationese texts. For en  $\rightarrow$  fr, the best system for translating translationese texts is BT while the best system for translating original texts is T-BT. This selection is performed by evaluating the translation quality for each model on the validation sets original and translationese texts.
By applying this selection strategy, we can significantly improve the overall translation quality
for given test sets, as reported in Table 6.
5996
System	test set	de→en	en→de
all	o	n-o	all	o	n-o
BT	2010	28.7 (+0.3)	32.0 (-2.1)	27.9 (+0.7)	22.3 (-1.8)	25.8 (-4.5)	21.3 (-0.9)
2011	24.6 (-1.0)	29.2 (-1.7)	23.5 (-0.9)	19.9 (-1.4)	23.1 (-2.6)	19.1 (-1.0)
2012	26.4 (-0.4)	27.1 (-2.4)	26.2 (-0.1)	20.7 (-0.9)	25.2 (-3.9)	19.5 (+0.0)
2013	29.6 (-0.4)	33.1 (-3.2)	28.6 (+0.0)	23.8 (-1.9)	24.4 (-3.5)	23.7 (-1.6)
2014	32.4 (+1.8)	25.7 (-4.2)	37.3 (+7.5)	26.0 (+0.1)	23.4 (-3.1)	28.9 (+3.7)
2015	33.4 (+2.0)	21.2 (-3.2)	39.4 (+5.4)	27.4 (-0.9)	27.7 (-3.3)	25.7 (+2.9)
2017	34.6 (+2.1)	25.7 (-2.6)	42.2 (+6.8)	26.6 (+0.1)	25.9 (-4.0)	26.4 (+4.2)
2018	43.2 (+3.9)	30.1 (-2.9)	53.9 (+10.5)	38.1 (-1.3)	38.8 (-6.0)	35.4 (+4.2)
2019	-	31.4 (-3.2)	-	-	32.1 (-4.1)	-
T-BT	2010	29.5 (+1.1)	34.1 (+0.0)	28.3 (+1.1)	24.9 (+0.8)	29.3 (-1.0)	23.7 (+1.5)
2011	25.9 (+0.3)	30.4 (-0.5)	24.8 (+0.4)	21.9 (+0.6)	26.0 (+0.3)	20.7 (+0.6)
2012	27.5 (+0.7)	28.8 (-0.7)	27.3 (+1.0)	22.7 (+1.1)	28.8 (-0.3)	21.1 (+1.6)
2013	30.7 (+0.7)	35.2 (-1.1)	29.6 (+1.0)	26.1 (+0.4)	27.4 (-0.5)	25.9 (+0.6)
2014	32.5 (+1.9)	28.2 (-1.7)	35.4 (+5.6)	28.2 (+2.3)	26.8 (+0.3)	30.0 (+4.8)
2015	33.7 (+2.3)	23.7 (-0.7)	38.3 (+4.3)	29.6 (+1.3)	31.1 (+0.1)	26.7 (+3.9)
2017	35.2 (+2.7)	27.3 (-1.0)	41.5 (+6.1)	28.3 (+1.8)	29.8 (-0.1)	26.3 (+4.1)
2018	43.4 (+4.1)	32.4 (-0.6)	51.5 (+8.1)	41.7 (+2.3)	45.0 (+0.2)	36.1 (+4.9)
2019	-	34.3 (-0.3)	-	-	36.5 (+0.3)	-
System	test set	fr→en	en→fr
all	o	n-o	all	o	n-o
BT	2008	20.8 (-3.8)	27.4 (-3.1)	19.8 (-3.9)	21.6 (-1.8)	17.5 (-7.0)	22.5 (-0.6)
2009	23.9 (-4.9)	38.3 (-8.1)	21.2 (-4.3)	26.4 (-0.2)	20.4 (-4.3)	27.3 (+0.3)
2010	27.2 (-3.5)	27.7 (-7.5)	27.0 (-2.0)	26.7 (-2.0)	19.2 (-14.4)	28.8 (+1.5)
2011	27.3 (-4.0)	27.3 (-6.7)	27.3 (-3.2)	28.9 (-2.0)	31.4 (-13.5)	28.2 (+1.0)
2012	26.8 (-4.3)	31.4 (-7.2)	25.7 (-3.5)	26.5 (-0.8)	22.2 (-8.2)	27.7 (+1.2)
2014	33.5 (-2.5)	28.8 (-7.3)	36.9 (+2.2)	29.9 (-6.1)	20.6 (-17.6)	39.4 (+5.9)
2015	31.7 (-4.5)	35.5 (-8.5)	27.1 (+0.8)	32.4 (-1.6)	18.4 (-11.1)	44.9 (+6.5)
T-BT	2008	24.7 (+0.1)	30.6 (+0.1)	23.8 (+0.1)	24.1 (+0.7)	25.6 (+1.1)	23.7 (+0.6)
2009	28.4 (-0.4)	45.3 (-1.1)	25.2 (-0.3)	27.7 (+1.1)	25.7 (+1.0)	28.0 (+1.0)
2010	31.2 (+0.5)	34.2 (-1.0)	29.8 (+0.8)	30.6 (+1.9)	34.5 (+0.9)	29.5 (+2.2)
2011	31.8 (+0.5)	32.7 (-1.3)	31.5 (+1.0)	31.6 (+0.7)	45.5 (+0.6)	28.0 (+0.8)
2012	31.6 (+0.5)	37.5 (-1.1)	30.2 (+1.0)	29.2 (+1.9)	31.9 (+1.5)	28.4 (+1.9)
2014	37.9 (+1.9)	35.6 (-0.5)	38.7 (+4.0)	38.5 (+2.5)	39.7 (+1.5)	37.0 (+3.5)
2015	36.2 (+0.0)	42.2 (-1.8)	28.3 (+2.0)	36.3 (+2.3)	30.2 (+0.7)	42.1 (+3.7)
Table 5: BLEU scores for all the systems in the high-resource conditions using 150M back-translations or the entire news crawl corpus for en→fr (76.6M sentences).
Sys.	2008	2009	2010	2011	2012	2014	2015
fr→en	en→fr	fr→en	en→fr	fr→en	en→fr	fr→en	en→fr	fr→en	en→fr	fr→en	en→fr	fr→en	en→fr
vanilla	24.6	23.4	28.8	26.6	30.7	28.7	31.3	30.9	31.1	29.5	36.0	36.0	36.2	34.0
BT	22.9	23.2	26.5	27.7	29.3	28.2	29.4	30.9	29.7	28.4	36.6	32.9	36.2	35.7
T-BT	24.5	23.8	28.9	27.3	31.2	30.0	31.8	31.6	31.8	28.9	37.3	38.2	36.6	36.0
selection	24.7	23.9	29.0	28.2	31.5	30.9	33.0	32.7	32.5	29.9	37.5	38.9	36.3	37.8
Table 6: BLEU scores for all the systems for en-fr on the overall test sets. "selection" denotes that decoding is performed by using the best model given the origin of the source sentence.
5997