SlowGuess/NAACL_2022_OCR huggingface dataset:
Ablation: Architecture vs. Data One reason Unsup might produce more natural-sounding output could be simply that it develops language-modeling capabilities from natural German alone, whereas Sup must see some translated data (being trained on bitext of human translations). Next, we ask whether the improved naturalness and structural diversity is due to the unsupervised NMT architecture, or simply the natural training data.
We build a supervised end-de MT system with 329,000 paired lines of translated English source and natural German, where the source is back-translated German News Crawl from a supervised system. In other words, we train on backtranslated data only on the source side and natural German as the target. The model thus develops its language-modeling capabilities on natural sentences alone. If more natural output is simply a response to training on natural data, then this supervised system should perform as well in naturalness as Unsup, or better.
We train another unsupervised system on translated text only. Source-side training data is synthetic English from translating German News Crawl with a supervised system. Target-side is synthetic German which was machine-translated from English News Crawl. If naturalness solely results
from data, this system should perform worst, being trained only on translated (unnatural) text.
Table 8 shows the results. The original unsupervised system (Unsup) performs best according to both LMs, having output that is more natural and less like translated text. When given only natural German to build a language model, the supervised system (Sup En-Trns/De-Orig) still produces more unnatural output than Unsup. Even when the unsupervised system uses translated data only (Unsup-Trns), its output is still more natural than the original supervised system (Sup) according to both LMs. This is a surprising result, and is interesting for future study. Together, these findings suggest that both German-original data and the unsupervised architecture encourage output to sound more natural.
5 Application: Leveraging Unsupervised Back-translation
Our results indicate that high-adequacy unsupervised MT output is more natural and more structurally diverse in comparison to human translation, than is supervised MT output. We are thus motivated to use these advantages to improve translation. We explore how to incorporate unsupervised MT into a supervised system via back-translation. We train for  $\sim 500,000$  updates for each experiment, and select models based on validation performance on newstest2018. We test on newstest2019(p).
The first row of Table 9 is the supervised baseline trained on the WMT18 bitext. The second row is Unsup, used throughout this work.
We back-translate 24 million randomly-selected sentences of German News CWEl twice: once using a supervised German-English system trained on WMT18 bitext with a transformer-big architecture, and once using Unsup. Both use greedy decoding for efficiency. We augment the WMT18 bitext with either the supervised or unsupervised BT.
Seen in Table 9, adding supervised BT (+SupBT) performs as expected; minorly declining on the source-original test set (orig-en), improving on the target- original set (orig-de), and improving on
LM Perplexity (PPL)	BLEU
Natural Text LM	Translated Text LM	Overall	orig-en	orig-de
Supervised (Sup)	72.69	41.06	29.2	34.0	21.1
Sup En-Trns/De-Orig	69.75	50.65	35.4	35.5	34.1
Unsup	67.01	58.17	30.1	30.9	27.1
Unsup-Trns	69.88	48.90	33.4	35.4	28.4
Table 8: Comparison of 4 English  $\rightarrow$  German MT systems: ppl from LMs trained on natural or translated text, BLEU on newestest2018.  $\downarrow$  ppl = model prefers the output. Sup En-Trns/De-Orig is supervised, trained on translated English and German-original News Crawl. Unsup is trained on natural English and German News Crawl. Unsup-Trns uses translated News Crawl only. Unsup performs best == more like natural text and less like translated text.
Figure 1: Back-translation selection method. Both systems translate the same source sentences. If an unsupervised output sentence is more than  $\mathrm{T}\%$  as likely as the supervised one, select the unsupervised. Here,  $\mathrm{T} = 65\%$ .
the paraphrase set (nt19p). Conversely, adding unsupervised BT (+UnsupBT) severely lowers BLEU on source-original and paraphrase test sets. Randomly-partitioning the BT sentences such that  $50\%$  are supervised BT and  $50\%$  are unsupervised also lowers performance on orig-en  $(+50 - 50\mathrm{BT})$
5.2 Tagged BT
Following Caswell et al. (2019), we tag BT on the source-side. Tagging aids supervised BT (+SupBT_Tag) and greatly improves unsupervised BT (+UnsupBT_Tag), which outperforms the baseline and is nearly on-par with +SupBT_Tag. Combining supervised and unsupervised BT using the same tag for both (+50-50BT_Tag) shows no benefit over +SupBT_Tag. +50-50BT_TagDiff uses different tags for supervised vs. unsupervised BT.
5.3 Probability-Based BT Selection
We design a BT selection method based on translation probability to exclude unsupervised BT of low quality. We assume that supervised BT is "good enough." Given translations of the same source sentence (one supervised, one unsupervised) we assert that an unsupervised translation is "good enough" if its translation probability is similar or better than that of the supervised translation. If much lower, the unsupervised output may be low-quality.
- Score each supervised and unsupervised BT with a supervised de-en system.
- Normalize the translation probabilities to con
trol for translation difficulty and output length.
- Compare probability of the supervised and unsupervised BT of the same source sentence:
$$
\Delta P = \frac {P \mathrm {n o r m} (\mathrm {u n s u p})}{P \mathrm {n o r m} (\mathrm {s u p})}
$$
- Sort translation pairs by  $\Delta P$ .
- Select the unsupervised BT for pairs scoring highest  $\Delta P$  and the supervised BT for the rest.
This filters out unsupervised outputs less than a hyperparameter  $\mathrm{T\%}$  as likely as the corresponding supervised sentence and swaps them with the corresponding supervised sentence. Importantly, the same 24M source sentences are used in all experiments. The procedure is shown in Figure 1.
Full results varying T are in the Appendix for brevity, but we show two example systems in Table 9. The model we call “+MediumMixTAG” uses the top  $\sim 40\%$  of ranked unsupervised BT with the rest supervised (9.4M unsupervised, 14.6M supervised). “+SmallMixTAG” uses the top  $\sim 13\%$  of unsupervised BT (3.1M unsupervised, 20.9M supervised). We use the same tag for all BTs. Improvements are modest, but our goal was to demonstrate how one might use unsupervised MT output rather than build a state-of-the-art system.
+SmallMixTAG performs better than the previous best on newstest2018p and +MediumMixTAG performs highest overall on nt19p. We recall
The numbers are not round because data was selected using round numbers for the hyperparameter T.
newstest2018	newstest2019
Overall	orig-en	orig-de	nt18p	orig-en	orig-de	nt19p
Supervised Baseline (5.2M)	41.8	46.1	34.3	12.6	38.8	30.4	11.7
Unsup (used throughout this work)	30.1	30.9	27.1	9.6	24.6	28.5	8.8
Supervised Baseline + SupBT	43.4	43.7	41.8	12.5	37.0	39.9	12.0
+ UnsupBT	33.3	33.8	31.1	9.9	27.2	30.8	9.5
+ 50-50BT	38.0	36.4	39.0	12.9	29.4	38.3	10.0
+ SupBTTAG	44.8	47.0	40.7	13.0	40.3	38.2	12.4
+ UnsupBTTAG	43.3	46.9	36.9	12.9	39.1	35.0	12.2
+ 50-50BTTAG	44.4	47.1	39.6	12.9	39.4	38.0	12.2
+ 50-50BT_TaqDiff	44.4	46.8	40.1	13.0	39.9	37.9	12.4
+ SmallMixTAG	44.8	46.8	40.8	13.2	39.8	38.8	12.5
+ MediumMixTAG	44.7	46.8	40.8	13.0	40.1	38.2	12.6
that small differences on paraphrase test sets can signal tangible quality differences (Freitag et al., 2020). Trusting BLEU on nt19p, we use  $+\text{MediumMix\_Tag}$  as our model for human evaluation.
One might inquire whether improved performance is due to the simple addition of noise in light of Edunov et al. (2018), who conclude that noising BT improves MT quality. Subsequent work, however, found that benefit is not from the noise itself but rather that noise helps the system distinguish between parallel and synthetic data (Caswell et al., 2019; Marie et al., 2020). Yang et al. (2019) also propose tagging to distinguish synthetic data. With tagging instead of noising, Caswell et al. (2019) outperform Edunov et al. (2018) in 4 of 6 test sets for En-De, furthermore find that noising on top of tagging does not help. They conclude that "tagging and noising are not orthogonal signals but rather different means to the same end." In light of this, our improved results are likely not due to increased noise but rather to systematic differences between supervised and unsupervised BT.
We run human evaluation with professional translators for  $+MediumMix\_ Tag$ , comparing its output translation of the newstest2019 test set with two baseline models. Table 10 shows that humans prefer the combined system over the baseline outputs.7 Table 11 shows the percentage of sentences judged as "worse than," "about the same as," or "better than" the corresponding  $+SupBT\_ Tag$  output, based on fluency. Raters again prefer the combined system. The improvements are modest, but
Table 9: SacREBLEU of supervised baseline plus 24M supervised or unsupervised BTs. +MediumMixTAG and +SmallMixTAG use the BT selection method of §5.3. +MediumMixTAG has 9.4M unsupervised BT and 14.6M supervised BT. +SmallMixTAG has 3.1M and 20.9M, respectively. nt18p and nt19p are paraphrase references from Freitag et al. (2020), where small BLEU score changes can indicate tangible quality difference.
Adequacy
+ UnsupBT_Tag	54.82
+ SupBT_Tag	56.13
+ MediumMix_Tag	58.62
Table 10: Human-eval direct assessment (adequacy) of supervised MT with supplemental back-translation.
Better	Same	Worse
51.1%	3.7%	45.2%
Table 11: Human side-by-side fluency eval. Shown: % of +MediumMix_Tag sentences judged "worse than," "about the same," or "better than" +SupBT_Tag output.
encouragingly indicate that unsupervised MT may have something to contribute to machine translation, even in high-resource settings.
Recent unsupervised MT systems can reach reasonable translation quality under clean and controlled data conditions, and could bring alternative translations to language pairs with ample parallel data. We perform the first systematic comparison of supervised and unsupervised MT output from systems of similar quality. We find that systematic differences do exist, and that high-quality unsupervised MT output appears more natural and more structurally diverse when compared to human translation, than does supervised MT output. Our findings indicate that there may be useful differences between supervised and unsupervised MT systems that could contribute to a system better than either achieves alone. As a first step, we demonstrate an unsupervised back-translation augmented model that takes advantage of the differences between the translation methodologies to outperform a traditional supervised system on human-evaluated measures of adequacy and fluency.
<sup>7</sup>Scores are low because we use only WMT18 bitext + BT, and translators score more harshly than crowd workers.
Stefanie Albert, Jan Anderssen, Regine Bader, Stephanie Becker, Tobias Bracht, Sabine Brants, Thorsten Brants, Vera Demberg, Stefanie Dipper, Peter Eisenberg, et al. 2003. tiger annotationsschema. Universitat des Saarlandes and Universitat Stuttgart and Universitat Potsdam.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 194-203, Florence, Italy. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of the Sixth International Conference on Learning Representations.
Mona Baker. 1993. Corpus Linguistics and Translation Studies: Implications and Applications. Text and technology: in honour of John Sinclair, pages 233-252.
Alexandra Birch. 2011. Reordering metrics for statistical machine translation.
Alexandra Birch, Phil Blunsom, and Miles Osborne. 2009. A quantitative analysis of reordering phenomena. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 197-205.
Alexandra Birch and Miles Osborne. 2011. Reordering metrics for mt. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1027-1035.
Alexandra Birch, Miles Osborne, and Phil Blunsom. 2010. Metrics for mt evaluation: evaluating reordering. Machine Translation, 24(1):15-26.
Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745-754, Honolulu, Hawaii. Association for Computational Linguistics.
Arianna Bisazza and Marcello Federico. 2016. A survey of word reordering in statistical machine translation: Computational models and language phenomena. Computational linguistics, 42(2):163-205.
Nikolay Bogoychev and Rico Sennrich. 2019. Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation. arXiv preprint arXiv:1911.03362.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (wmt18). In Proceedings of the Third Conference on Machine
Translation, Volume 2: Shared Task Papers, pages 272-307, Belgium, Brussels. Association for Computational Linguistics.
Sabine Brants, Stefanie Dipper, Peter Eisenberg, Silvia Hansen-Schirra, Esther König, Wolfgang Lezius, Christian Rohrer, George Smith, and Hans Uszkoreit. 2004. Tiger: Linguistic interpretation of a german corpus. Research on language and computation, 2(4):597-620.
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 53–63, Florence, Italy. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7057-7067.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2836-2846.
Marina Fomicheva and Lucia Specia. 2016. Reference bias in monolingual machine translation evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 77-82, Berlin, Germany. Association for Computational Linguistics.
Heidi Fox. 2002. Phrasal cohesion and statistical machine translation. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 304-3111. Association for Computational Linguistics.
Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 34-44, Florence, Italy. Association for Computational Linguistics.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61-71, Online. Association for Computational Linguistics.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation.
Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63-S63.
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81-93.
Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 1318-1326, Stroudsburg, PA, USA. Association for Computational Linguistics.
David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proceedings of MT-Summit XII, pages 81-88.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. CoRR, abs/1804.07755.
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationese improves SMT. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL '12, pages 255-265, Stroudsburg, PA, USA. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210.
Qingsong Ma, Yvette Graham, Timothy Baldwin, and Qun Liu. 2017. Further investigation into reference bias in monolingual evaluation of machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2476-2485, Copenhagen, Denmark. Association for Computational Linguistics.
Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020. Tagged back-translation revisited: Why does it really work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5990–5997, Online. Association for Computational Linguistics.
Anna Mauranen and Pekka Kujamäki. 2004. *Translation universals: Do they exist?*, volume 48. John Benjamins Publishing.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020. *Translationese as a language in “multilingual” NMT*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7737–7746, Online. Association for Computational Linguistics.
Larry Selinker. 1972. Interlanguage. International Review of Applied Linguistics, pages 209-241.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X. Chen, Ye Jia, Anjuli Kannan, Tara N. Sainath, and Yuan Cao et al. 2019. Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling. CoRR, abs/1902.08295.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020. Leveraging monolingual data with self-supervision for multilingual neural machine translation. arXiv preprint arXiv:2005.04816.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, 6. Cambridge, MA.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In ICML.
Antonio Toral. 2019. Post-edites: an exacerbated translationese. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 273-281, Dublin, Ireland. European Association for Machine Translation.
Antonio Toral. 2020. Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019. arXiv preprint arXiv:2005.05738.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113-123, Belgium, Brussels. Association for Computational Linguistics.
Gideon Toury. 1995. Descriptive Translation Studies and Beyond. Benjamins translation library. John Benjamins Publishing Company.
Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. 2021. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2203-2213.
Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019. Lost in translation: Loss and decay of linguistic richness in machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 222-232, Dublin, Ireland. European Association for Machine Translation.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Benjamin Wellington, Sonjia Waxmonsky, and I. Dan Melamed. 2006. Empirical lower bounds on the complexity of translational equivalence. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 977-984, Sydney, Australia. Association for Computational Linguistics.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2019. Effectively training neural machine translation models with monolingual data. Neurocomputing, 333:240-247.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. CoRR, abs/1906.08069.