SlowGuess/ACL_2020_OCR huggingface dataset:
The first row of Table 4 shows that both language models achieve similar perplexity on  $Y^{*}$  (37.2 VS 36.8), suggesting that the translations of BT and OP are equally close to translationese. Interestingly, both system outputs are closer to translationese than natural text since PPL on  $Y^{*}$  is significantly lower than the PPL on  $Y$  (second row of Table 4). This is also supported by BLEU being higher when using  $Y^{**}$  as a reference compared to  $Y$  for the same input  $X^{*}$  (second and last row of Table 3).
Our results support the hypothesis that the outputs of  $BT$  and  $OP$  are equally close to translationese. This in turn may explain why BLEU cannot distinguish between  $OP$  and  $BT$  in direct mode where the reference is translationese.
4.6 BT Generates More Natural Text
Back-translation augments the training corpus with automatic translations from target original data. Training models on large amounts of target original data may bias BT systems to produce outputs that are closer to naturally occurring text. In contrast, OP systems have been trained on the original parallel data, a mix of direct and reverse data which contains a much smaller amount of target original sentences. This may explain why BLEU evaluation with translationese references (direct portion) does not capture the human preference for BT.
To understand this better, we conduct two experiments. The first experiment is based on the language models we trained previously (§4.5) to assess how close our systems are to translationese and naturally occurring text. The second experiment is based on a human study where native speakers assess the fluency of each system output.
For the first experiment we reuse the two language models from §4.5 to measure how close the system outputs are to natural text  $(Y)$ . The second
BT	OP	draw
De-En	28	16	63
En-De	50	33	18
En-Ru	37	21	42
Table 5: Human preference in terms of fluency for system outputs of BT and OP. Judgements are based on a pair-wise comparison between the two systems without the source sentence and conducted by native speakers. All results are based on 100 judgements and the preference of BT over OP is statistically significant at  $p = 0.05$ .
row of Table 4 shows that the BT language model assigns much higher probability to naturally occurring text,  $Y$ , compared to the OP language model (82.2 VS 57.4 perplexity), suggesting that BT does indeed produce outputs that are much closer to natural text than OP. We surmise that this difference, which is captured by a language model trained on system outputs and evaluated on  $Y$ , could be at least partially responsible for the marked human preference towards BT translations.
In the second experiment, native speakers of English, German and Russian rate whether the output of OP is more fluent than the output of BT for 100 translations of the De-En, En-De and En-Ru systems. Human raters perform a pair-wise ranking and raters can only see two translations but not the source; the system identity is unknown to raters.
Table 5 shows that BT is judged to be significantly more fluent by native speakers than OP in three languages.
5 Improving BT Evaluation
In the previous sections, we gathered mounting evidence that BLEU fails at capturing the improved fluency of BT in direct mode. Next, we propose to use a language model to assess fluency as an additional measure to complement BLEU. Different to the setup above (§4.5, 4.6), where we used a separate LM for each system, we propose to use a single LM for all systems in order to simplify the evaluation.
The language model is trained on a large monolingual dataset disjoint from the monolingual dataset used for generating back-translated data for BT training. This restriction is critical, otherwise the language model is likely to assign higher probably to BT generations simply because training and evaluation sets overlap. To train these language models we sample 315M, 284M and 120M com
BT PPL	OP PPL
De-En	74.8	78.7
En-De	48.6	52.6
Ru-En	57.6	68.6
En-Ru	61.7	72.4
Table 6: Automatic fluency analysis with language models trained on the Common Crawl corpus in the respetive target language. BT receives lower perplexity (PPL) throughout, despite attaining the same BLEU score of OP, see Table 1.
moncrawl sentences for each of the three target languages, namely English, German and Russian, respectively.
The language model is used to score the outputs of BT and OP on the direct portion of the test set. If two systems have similar BLEU scores, then a lower perplexity with the LM indicates higher fluency in the target natural language. This fluency assessment is complementary to BLEU which in turn is more sensitive to adequacy.
Table 6 shows that the language model assigns lower perplexity to BT in all four setups. This shows that a language model can help to assess the fluency of system output when a human evaluation is not possible.
In future work, we intend to further investigate how to best combine BLEU and language model scoring in order to maximize correlation with human judgements, particularly when evaluating BT in direct mode. Meantime, practitioners can use this additional metric in their evaluation to break ties in BLEU scoring.
According to our findings, back-translation improves translation accuracy, for both source and target original sentences. However, automatic metrics like BLEU fail to capture human preference for source original sentences (direct mode).
We find that BT produces outputs that are closer to natural text than the output of OP, which may explain human preference for BT. We recommend distinguishing between direct and reverse translations for automatic evaluation, and to make final judgements based on human evaluation. If human evaluation is not feasible, complementing standard metrics like BLEU with a language model (§5) may help assessing the overall translation quality.
In the future, we plan to investigate more thor-
oughly the use of language models for evaluating fluency, the effect of domain mismatch in the choice of monolingual data, and ways to generalize this study to other applications beyond MT.
We thank Barry Haddow for initially pointing out the BLEU discrepancy between the forward and reverse portions of the WMT 2018 test set.
Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv, abs/1809.10853.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.
Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and technology: In honour of John Sinclair, 233:250.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measure for Machine Translation.
Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Proc. of WMT.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proc. of WMT.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2019. Findings of the 2019 conference on machine translation (wmt19). In Proc. of WMT.
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proc. of WMT.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proc. of EMNLP.
Markus Freitag, Isaac Caswell, and Scott Roy. 2019. Text repair model for neural machine translation. arXiv, abs/1904.04790.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. *Natural Language Engineering*, 23(1):330.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation. arXiv, abs/1906.09833.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Proc. of 2nd Workshop on Neural Machine Translation and Generation.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP.
Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proc. of ACL.
David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proc. of MT Summit.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based and neural unsupervised machine translation. In EMNLP.
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011. Language models for machine translation: Original vs. translated texts. In Proc. of EMNLP.
Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proc. of ACL: Demonstrations.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proc. of WMT.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair's wmt19 news translation task submission. In Proc. of WMT.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL: Demonstrations.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. of WMT.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL.
Alberto Poncelas, Dimitar Sht. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018a. Investigating backtranslation in neural machine translation. arXiv, 1804.06189.
2843
Alberto Poncelas, Dimitar Sht. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018b. Investigating backtranslation in neural machine translation. arXiv, 1804.06189.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proc. of WMT.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proc. of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proc. of ACL.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of AMTA.
Milos Stanojevic and Khalil Sima'an. 2014. Beer: Better evaluation as ranking. In Proc. of WMT.
Sara Stymne. 2017. The effect of translationese on tuning for statistical machine translation. In Proc. of NoDaLiDa.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proc. of WMT.
Gideon Toury. 2012. Descriptive translation studies and beyond: Revised edition, volume 100. John Benjamins Publishing.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. arXiv, abs/1906.08069.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv, abs/1904.09675.
A Forward/reverse BLEU for WMT'18 English-German systems
system	fwd	rev	delta
online-Y	47.1	30.3	-16.8
MMT-production-system	51.8	36.7	-15.1
online-B.0	52.9	39.1	-13.8
NTT	50.7	39.7	-11.0
Microsoft-Marian	52.5	41.6	-10.9
KIT	50.3	39.5	-10.8
LMU-nmt	43.5	33.4	-10.1
uedin	47.8	37.8	-10.0
online-A	37.8	28.6	-9.2
JHU	46.0	38.2	-7.8
online-F	23.5	16.4	-7.1
UCAM	48.9	42.1	-6.8
RWTH-UNSUPER	16.7	12.0	-4.7
online-G	25.9	22.5	-3.4
LMU-unsup	15.2	14.3	-0.9
Facebook-FAIR	45.8	46.1	0.4
Table 7 shows that a large-scale back-translation system, Facebook-FAIR, mostly improves BLEU on the reverse portion whereas it is outperformed by many other entrants in the forward portion.
B Results with WMT references
Table 7: Forward/reverse BLEU for WMT'18 English-German systems.
src	ref	sys	en-de	de-en	en-ru	ru-en
BLEU	human	BLEU	human	BLEU	human	BLEU	human
X	Y*	OP	33.7	-0.18	40.3	-0.07	31.3	-0.66	43.8	-0.37
BT	32.3	-0.05	38.6	0.03	31.9	-0.35	41.2	-0.12
X	YWMT	OP	28.7	-0.18	35.4	-0.07	31.8	-0.66	39.7	-0.37
BT	29.9	-0.05	34.2	0.03	31.9	-0.35	38.5	-0.12
Table 8: BLEU results with respect to the original WMT references (document-level) and the sentence-level references used throughout this study. Sentence-level references result in higher BLEU but OP and BT still achieve very similar BLEU.
Table 8 shows that BLEU does not strongly distinguish between BT and OP, regardless of whether the reference was obtained at the document-level  $(Y_{WMT}^{*})$  or at the sentence-level  $(Y^{*})$ .
C Other metrics than BLEU
src	ref	sys	human	BLEU	TER	en-de BEER	METEOR	BERTScore
X	Y*	OP	-0.18	33.7	0.466	0.635	0.531	0.849
BT	-0.05	32.3	0.473	0.619	0.512	0.843
X*	Y	OP	-0.01	31.3	0.504	0.609	0.530	0.841
BT	0.10	38.9	0.431	0.652	0.580	0.866
X**	Y*	OP	-0.05	39.7	0.403	0.677	0.590	0.878
BT	0.03	39.2	0.409	0.669	0.578	0.876
X*	Y**	OP	-0.01	39.5	0.410	0.670	0.599	0.876
BT	0.10	41.8	0.383	0.683	0.610	0.884
Table 9: BLEU and other metrics as well as human preference judgements for English-German translations.
Table 9 shows results for automatic metrics other than BLEU (Papineni et al., 2002). The metrics TER (Snover et al., 2006), BEER (Stanojevic and Sima'an, 2014), METEOR (Banerjee and Lavie, 2005) and BERTScore (Zhang et al., 2019) show similar trends as BLEU, i.e., they do not indicate human preference of BT over bitext for the direct portion of the test set  $(X\to Y^{*})$