SlowGuess/ACL_2020_OCR huggingface dataset:
Test set → Tagging ↓	Src-Origin
Decode	Lex. Var.	Lex. Density	Len. Var.
Untagged	-	0.258	0.393	0.246
FT clf.	Transl.	0.255	0.396	0.264
FT clf.	Natural	0.260	0.397	0.245
Table 5: Measuring the degree of translationese for WMT English  $\rightarrow$  French newtest2014/full on the source-original half. Higher lexical variety, lexical density, and length variety indicate less translationese output.
Lexical Variety : Using the tag to decode as natural text (i.e. more like original target text) increases lexical variety. This is expected as original sentences tend to use a larger vocabulary.
Lexical Density : We also increase lexical density when decoding as natural text. In other words, the model has a higher percentage of content words in its output, which is an indication that it is more like original target-language text.
Length Variety: Unlike the previous two metrics, decoding as natural text does not lead to a more "natural" (i.e. larger) average length variety. One reason may be related to the fact that this is the only metric that also depends on the source sentence: since all of our training pairs feature translationese on either the source or target side, both the tagged and untagged training pairs will
feature similar sentence structures, so the model never fully learns to produce different structures. This further illustrates the problem of the lack of original  $\rightarrow$  original training data noted in the introduction.
5.2 Tagging using Translationese Heuristics
Rather than tagging training data with a trained classifier, as explored in the previous sections, it might be possible to tag using much simpler heuristics, and achieve a similar effect. We explore two options here.
5.2.1 Length Ratio Tagging
Here, we partition the training pairs  $(x,y)$  according to a simple length ratio  $\frac{|x|}{|y|}$ . We use a threshold  $\hat{\rho}_{\text{length}}$  empirically calculated from two large monolingual corpora,  $M_x$  and  $M_y$ :
$$
\hat {\rho} _ {l e n g t h} = \frac {\frac {1}{| M _ {x} |} \sum_ {x _ {i} \in M _ {x}} | x _ {i} |}{\frac {1}{| M _ {y} |} \sum_ {y _ {i} \in M _ {y}} | y _ {i} |} \tag {4}
$$
For English  $\rightarrow$  French, we found  $\hat{\rho}_{\text{length}} = 0.8643$ , meaning that original French sentences tend to have more tokens than English. We tag all pairs with length ratio greater than  $\hat{\rho}_{\text{length}}$  (49.8% of the training bitext). Based on the discussion in Section 5.1.3, we expect that  $\frac{|x|}{|y|} \approx 1.0$  indicates translationese, so in this case the tag should mean "produce translationese" instead of "produce original text."
5.2.2 Lexical Density Tagging
We tag examples with a target-side lexical density of greater than 0.5, which means that the target is more likely to be original than translationese. Please refer to Section 5.1.2 for an explanation of this metric.
5.2.3 Results
Table 6 shows the results for this experiment, compared to the untagged baseline and the classifier-tagged model from Table 2a. This table specifically looks at the effect of controlling whether the output should feature more or less translationese on each subset of the test set. We see that the lexical density tagging approach yields expected results, in that the tag can be used to effectively increase BLEU on the target-original portion of the test set. The length-ratio tagging, however, has the opposite effect: producing shorter outputs ("decode as if translationese") produces higher target-original
BLEU and lower source-original BLEU. We speculate that this data partition has accidentally picked up on some artifact of the data.
Two interesting observations from Table 6 are that 1) both heuristic tagging methods perform much more poorly than the classifier tagging method on both test set halves, and 2) all varieties of tagging produce large performance changes (up to -7.2 BLEU). This second observation highlights that tagging can be powerful – and dangerous when it does not correspond well with the desired feature.
5.3 Back-Translation Experiments
We also investigated whether using a classifier to tag training data improved model performance in the presence of back-translated (BT) data. Caswell et al. (2019) introduced tagged back-translation (TBT), where all back-translated pairs are tagged and no bitext pairs are. They experimented with decoding the model with a tag ("as-if-back-translated") but found it harmed BLEU score. However, in our early experiments we discovered that doing this actually improved the model's performance on the target-original portion of the test set, while harming it on the source-original half. Thus, we frame TBT as a heuristic method for identifying target-original pairs: the monolingual data used for the back-translations is assumed to be original, and the target side of the bitext is assumed to be translated. We wish to know whether we can find a better tagging scheme for the combined BT+bitext data, based on a classifier or some other heuristic.
Results for English  $\rightarrow$  French models trained with BT data are presented in Table 7a. While combining the bitext classified by the FT classifier with all-tagged BT data yields a minor gain of 0.2 BLEU over the TBT baseline of Caswell et al. (2019), the other methods do not beat the baseline. This indicates that assuming all of the target monolingual data to be original is not as harmful as the error introduced by the classifiers.
English  $\rightarrow$  German results are presented in Table 7b. Combining the bitext classified by the RTT classifier with all-tagged BT data matched the performance of the TBT baseline, but none of the models outperformed it. This is expected, given the poor performance of the bitext-only models for this language pair.
Test set →
Decode as if →
: Domain match? → Train data tagging ↓	Src-Orig
Natural
X	Src-Orig
Transl.	Trg-Orig
Transl.	Trg-Orig
Natural
Untagged	39.5	39.5	44.5	44.5
FT clf.	37.7	40.0	42.5	45.0
Length Variety	38.2	36.1	43.6	36.2
Lex. Density	36.9	36.7	41.2	43.4
Table 6: Comparing heuristic- and classifier-based tagging. BLEU scores are averaged for newstest2014/full and newstest2015 English→French. The trained classifier outperforms both heuristics, and length-ratio tagging has the reverse effect from what we expect.
Test set →
Decode as if →
: Domain match? →
Bitext tagging ↓ | BT tagging ↓	Src-Orig	Trg-Orig	Combined
Both
Natural	Transl.	Transl.	Natural
X	✓	X	✓	✓
a. English  $\rightarrow$  French: Avg. newstest20{14/full, 15}
Untagged	All Tagged	38.4	40.8	47.5	49.8	45.5
FT clf.	All Tagged	38.8	40.8	47.3	50.3	45.7
FT clf.	FT clf.	38.2	40.9	45.5	49.0	45.2
RTT clf.	RTT clf.	38.3	40.1	49.4	49.5	45.1
b. English  $\rightarrow$  German: Avg. newstest20{14/full,16,17,18}
Untagged	All Tagged	33.5	37.3	36.7	37.1	37.6
FT clf.	All Tagged	33.4	37.2	36.2	37.2	37.5
RTT clf.	All Tagged	33.6	37.4	36.6	37.1	37.6
RTT clf.	RTT clf.	31.6	35.7	36.8	36.7	36.4
FT clf.	FT clf.	30.5	35.5	36.5	37.0	36.5
Table 7: Average BLEU scores for models trained on (a) WMT 2018 English→French bitext plus 39M back-translated monolingual sentences, and (b) WMT 2018 English→German bitext plus 24M back-translated monolingual sentences. As before, we tag by heuristics and/or classifier predictions on the target (German) side.
6 Example Output
In Table 8, we show example outputs for WMT English  $\rightarrow$  French comparing the Untagged baseline with the  $FT$  clf. natural decodes. In the first example, avec suffisamment d'art is an incorrect word-for-word translation, as the French word art cannot be used in that context. Here the word habilitation, which is close to "skilfully" in English, sounds more natural. In the second example, libre d'impôt is the literal translation of "tax-free", but French documents rarely use it, they prefer pas imposable, meaning "not taxable".
7.1 Translationese
The effects of translationese on MT training and evaluation have been investigated by many prior authors (Kurokawa et al., 2009; Lembersky et al., 2012; Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2019; Freitag et al., 2019; Edunov et al., 2019; Freitag et al., 2020). Training classifiers to detect translationese has also been done (Kurokawa et al., 2009; Koppel and Ordan, 2011).
Similarly to this work, Kurokawa et al. (2009) used their classifier to preprocess MT training data; however, they completely removed target-original pairs. In contrast, Lembersky et al. (2012) used both types of data (without explicitly distinguishing them with a classifier), and used entropy-based measures to cause their phrase-based system to favor phrase table entries with target phrases that are more similar to a corpus of translationese than original text. In this work, we combine aspects from each of these: we train a classifier to partition the training data, and use both subsets to train a single model with a mechanism allowing control over the degree of translationese to produce in the output. We also show with human evaluations that source-original test sentence pairs result in BLEU scores that do not correlate well with translation quality when evaluating models trained to produce more original output.
7.2 Training Data Tagging for NMT
In addition to the methods in Caswell et al. (2019), tagging training data and using the tags to control output is a technique that has been growing in popularity. Tags on the source sentence have
Source	Sorry she didn’t phrase it artfully enough for you.
Untagged	Désolée, elle ne l’a pas formulé avec suffisamment d’art pour vous.
FT clf.	Désolé elle ne l’a pas formulé assez habilement pour vous.
Source	Your first 10,000 is tax free.
Untagged	Votre première tranche de 10 000 est libre d’impôt.
FT clf.	La première tranche de 10 000 n’est pas impossibly.
Table 8: Example English→French output comparing the untagged baseline with the  $FT$  clf. natural decode.
been used to indicate target language in multilingual models (Johnson et al., 2016), formality level in English  $\rightarrow$  Japanese (Yamagishi et al., 2016), politeness in English  $\rightarrow$  German (Sennrich et al., 2016a), gender from a gender-neutral language (Kuczmarski and Johnson, 2018), as well as to produce domain-targeted translation (Kobus et al., 2016). Shu et al. (2019) use tags at training and inference time to increase the syntactic diversity of their output while maintaining translation quality; similarly, Agarwal and Carpuat (2019) and Marchisio et al. (2019) use tags to control the reading level (e.g. simplicity/complexity) of the output. Overall, tagging can be seen as domain adaptation (Freitag and Al-Onaizan, 2016; Luong and Manning, 2015).
We have demonstrated that translationese and original text can be treated as separate target languages in a "multilingual" model, distinguished by a classifier trained using only monolingual and synthetic data. The resulting model has improved performance in the ideal, zero-shot scenario of original  $\rightarrow$  original translation, as measured by human evaluation of adequacy and fluency. However, this is associated with a drop in BLEU score, indicating that better automatic evaluation is needed.
Acknowledgments We are grateful to the anonymous reviewers for suggesting useful additions.
Swetha Agarwal and Marine Carpuat. 2019. Controlling Text Complexity in Neural Machine Translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Mona Baker, Gill Francis, and Elena Tognini-Bonelli. 1993. Corpus Linguistics and Translation Studies: Implications and Applications, chapter 2. John Benjamins Publishing Company, Netherlands.
Nikhil Buduma and Nicholas Locascio. 2017. Fundamentals of deep learning: Designing next-generation machine intelligence algorithms. O'Reilly Media, Inc.
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 53-63, Florence, Italy. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2019. On the evaluation of machine translation systems trained with back-translation. arXiv preprint arXiv:1908.05204.
J.L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.
Markus Freitag and Yaser Al-Onaizan. 2016. Fast Domain Adaptation for Neural Machine Translation. CoRR, abs/1612.06897.
Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at Scale and Its Implications on MT Evaluation Biases. In Proceedings of the Fourth Conference on Machine Translation, pages 34-44, Florence, Italy. Association for Computational Linguistics.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be Guilty but References are not Innocent.
Martin Gellerstam. 1986. Translationese in swedish novels translated from english. In Lars Wollin and Hans Lindquist, editors, Translation Studies in Scandinavia, page 8895. CWK Gleerup.
Martin Gellerstam. 1996. Translations as a source for cross-linguistic studies. Lund Studies in English, 88:53-62.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation. CoRR, abs/1906.09833.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Vi'egas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. CoRR, abs/1611.04558.
Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74-83, Melbourne, Australia. Association for Computational Linguistics.
Catherine Kobus, Josep Maria Crego, and Jean Senellart. 2016. Domain Control for Neural Machine Translation. CoRR, abs/1612.06140.
Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit, pages 79-86, Phuket, Thailand. AAMT, AAMT.
Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 1318-1326, Stroudsburg, PA, USA. Association for Computational Linguistics.
James Kuczmarski and Melvin Johnson. 2018. Gender-aware natural language translation. Technical Disclosure Commons.
David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proceedings of MT-Summit XII, pages 81-88.
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationese improves SMT. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL '12, pages 255-265, Stroudsburg, PA, USA. Association for Computational Linguistics.
Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation, pages 76-79.
Kelly Marchisio, Jialiang Guo, Cheng-I Lai, and Philipp Koehn. 2019. Controlling the reading level of machine translation output. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 193-203, Dublin, Ireland. European Association for Machine Translation.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311-318. Association for Computational Linguistics.
Matt Post. 2018. A Call for Clarity in Reporting Bleu Scores. arXiv preprint arXiv:1804.08771.
Federica Scarpa. 2006. Corpus-based quality-assessment of specialist translation: A study using parallel and comparable corpora in English and Italian. Insights into specialized translation-linguistics insights. Bern: Peter Lang, pages 155-172.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In Proceedings of
the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 35-40, San Diego, California. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X. Chen, Ye Jia, Anjuli Kannan, Tara N. Sainath, and Yuan Cao et al. 2019. Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling. CoRR, abs/1902.08295.
Raphael Shu, Hideki Nakayama, and Kyunghyun Cho. 2019. Generating Diverse Translations with Sentence Codes. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics.
Antonio Toral. 2019. Post-editese: an exacerbated translationese. CoRR, abs/1907.00900.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? Reassessing claims of human parity in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113-123, Belgium, Brussels. Association for Computational Linguistics.
Gideon Toury. 2012. Descriptive translation studies and beyond: Revised edition, volume 100. John Benjamins Publishing.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Hayahide Yamagishi, Shin Kanouchi, Takayuki Sato, and Mamoru Komachi. 2016. Controlling the voice of a sentence in Japanese-to-English neural machine translation. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 203–210.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. CoRR, abs/1906.08069.
7746