Venue: NAACL 2022

# On Systematic Style Differences between Unsupervised and Supervised MT

and an Application for High-Resource Machine Translation

Kelly Marchisio\({}^{*}\)

Johns Hopkins University

kmarc@jhu.edu &Markus Freitag   David Grangier

Google Research

{freitag, grangier}@google.com

Work completed at Google Translate Research.

###### Abstract

Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask whether the different training methods result in systematically different output beyond what is visible via quality metrics like adequacy or BLEU. We compare translations from supervised and unsupervised MT systems of similar quality, finding that unsupervised output is more fluent and more structurally different in comparison to human translation than is supervised MT. We then demonstrate a way to combine the benefits of both methods into a single system which results in improved adequacy and fluency as rated by human evaluators. Our results open the door to interesting discussions about how supervised and unsupervised MT might be different yet mutually-beneficial.

## 1 Introduction

Supervised machine translation (MT) utilizes parallel bitext to learn to translate. Ideally, this data consists of natural texts and their human translations. In a way, the goal of supervised MT training is to produce a machine that mimicks human translators in their craft. Unsupervised MT, on the other hand, uses monolingual data alone to learn to translate. Critically, unsupervised MT _never sees an example of human translation_, and therefore _must create its own style of translation_. Unlike supervised MT where one side of each training sentence pair must be a translation, unsupervised MT can be trained with natural text alone.

In this work, we investigate the output of supervised and unsupervised MT systems of similar quality to assess whether systematic differences in translation exist. Our exploration of this research area focuses on English\(\rightarrow\)German for which abundant bilingual training examples exist, allowing us to train high-quality systems with both supervised and unsupervised training.

Our main contributions are:

* We observe systematic differences between the output of supervised and unsupervised MT systems of similar quality. High-quality unsupervised output appears **more natural**, and more **structurally diverse** when compared to human translation.
* We show a way to incorporate unsupervised back-translation into a standard supervised MT system, improving adequacy, naturalness, and fluency as measured by human evaluation.

Our results provoke interesting questions about what unsupervised methods might contribute beyond the traditional context of low-resource languages which lack bilingual training data, and suggest that unsupervised MT might have contributions to make for high-resource scenarios as well. It is worth exploring how combining supervised and unsupervised setups might contribute to a system better than either creates alone.

We discuss related work in SS2. In SS3, we introduce the dataset, model details, and evaluation setups. In SS4, we characterize the differences between the output of unsupervised and supervised neural MT systems of similar quality. In SS5, we demonstrate a combined system which benefits from the complementary strengths of the two methods. We summarize the paper in SS6.

## 2 Related Work

Unsupervised MTTwo paradigms for unsupervised MT are finding a linear transformation to align two monolingual embedding spaces [9, 8, 7], and pretraining a bi-/multilingual language model then finetuning on a translation task [10, 11, 12]. We study the Masked Sequence

to-Sequence Pretraining (MASS) language model pretraining paradigm of Song et al. (2019). MASS is an encoder-decoder trained jointly with a masked language modeling objective on monolingual data. Iterative back-translation (BT) follows pretraining.

Monolingual Data in MTBT is widely-used to exploit monolingual data (Sennrich et al., 2016). "Semi-supervised" systems use monolingual and parallel data to improve performance (e.g. Artetxe et al. (2018)). Siddhant et al. (2020) combine multilingual supervised training with MASS for many languages and zero-shot translation.

Source Artifacts in Translated TextBecause supervised MT is trained ideally on human-generated translation, characteristics of human translation affects the style of machine translation from such systems. Dubbed "translationese," human translation includes source language artifacts (Koppel and Ordan, 2011) and source-independent artifacts--_Translation Universals_ (Mauranen and Kujamaki, 2004). There are systematic biases inherent to translated texts (Baker, 1993; Selinker, 1972), and biases coming from interference from source text (Toury, 1995). In MT, Freitag et al. (2019, 2020) attribute these patterns as a source of mismatch between BLEU (Papineni et al., 2002) and human evaluation measures of quality, raising concerns that overlap-based metrics reward hypotheses with the characteristics of translated text more than those with natural language. Vanmassenhove et al. (2019, 2021) note loss of linguistic diversity and richness from MT, and Toral (2019) see related effects even after human post-editing. The impact of translated text on human evaluation has also been studied (Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2019; Fomicheva and Specia, 2016; Ma et al., 2017), as has the impact in training data (Kurokawa et al., 2009; Lembersky et al., 2012; Bogoychev and Sennrich, 2019; Riley et al., 2020).

Measuring Word ReorderingWord reordering models are well-studied because they formed a critical part of statistical MT (see Bisazza and Federico (2016) for a review). Others examined metrics for measuring reordering in translation (e.g. Birch et al., 2008, 2009, 2010). Wellington et al. (2006) and Fox (2002) use part-of-speech (POS) tags in the context of parse trees, and Fox (2002) measure the similarity of French and English with respect to phrasal cohesion by calculating alignment crossings using parse trees. Most similar to us, Birch (2011) view simplified word alignments as permutations and compare distance metrics over these to quantify the amount of reordering done. They use TER computed over the alignments as a baseline. Birch and Osborne (2011)'s LRScore interpolates a reordering metric with a lexical translation metric.

## 3 Experimental Setup

### Data

TrainingExperiments are in English\(\rightarrow\)German. For the main study comparing supervised and unsupervised MT, we use News Commentary v14 (329,000 sentences) as parallel bitext for the supervised system, and News Crawl 2007-17 as monolingual data for the unsupervised system. Deduplicated News Crawl 2007-17 has 165 million English sentences and 226 million German sentences.

The combined system demonstration at the end of our work utilizes a BT selection method. We use the bilingual training data from WMT2018 (Bojar et al., 2018) (News Commentary v13, Europarl v7, Common Crawl, EU Press Release) so that our model can be compared with well-known work using BT (e.g. Edunov et al., 2018; Caswell et al., 2019). We deduplicate and filter out pairs with \(>250\) tokens in either language or length ratio over 1.5, resulting in 5.2 million paired sentences.

Development and Test SetsFor the main experiments, we use newstest2017 as the dev set with newstest2018 and newstest2019 for test. newstest2018 was originally created by translating one half of the test data from English\(\rightarrow\)German (origin) and the other half from German\(\rightarrow\)English (origde). Since 2019, WMT produces newstset sets with only source-original text and human translations on the target side to mitigate known issues when translating and evaluating on target-original data (e.g. Koppel and Ordan, 2011; Freitag et al., 2019).

For most experiments, we evaluate on orig-en sentences only to reflect the real use-case for translation and modern evaluation practice. We examine orig-de only for BLEU score as an additional data point of difference between supervised and un-supervised MT. Zhang and Toral (2019) show that target-language-original text should not be used for human evaluation (orig-de, in our case).

We use the newstest2018 "paraphrased" test references from Freitag et al. (2020),1 which are made 

for orig-en sentences only. These additional references have different structure than the source sentence but maintain semantics, and provide a way to measure system quality without favoring translations with the same structure as the source. Observing work that uses these references, BLEU is typically much lower than on original test sets, and score differences tend to be small but reflect tangible quality difference Freitag et al. (2020).

For the system combination demonstration, we use newstest2018 for development and newstest2019 for test. We also use newstest2019 German\(\rightarrow\)English and swap source and target to make an orig-de English\(\rightarrow\)German test set, and use paraphrase references for newtest2019 (orig-en).

Testing on the official newstest2018 in the main experiments allows us to see interesting differences between unsupervised and supervised MT that are hidden with newstest2019 because it is orig-en only. Using newstest2018 for development in the system combination demonstration aligns with similar literature e.g. Edunov et al. (2018); Caswell et al. (2019). We use SacreBLEU throughout Post (2018).2

Footnote 2: BLEU+case.mixed+lang.ende+numrefs.l+smooth.exp+{TESTSET}+tok.13a+version.1.4.12

### Part-of-Speech Tagging

We use part-of-speech taggers for some experiments: universal dependencies (UD) implemented in spaCy3 and spaCy's language-specific fine-grained POS tags for German from the TIGER Corpus Albert et al. (2003); Brants et al. (2004).

Footnote 3: [https://spacy.io/](https://spacy.io/), [https://universaldependencies.org/](https://universaldependencies.org/)

### Models

Our **unsupervised MT** model is a MASS transformer with the hyperparameters of Song et al. (2019). We train MASS on the News Crawl corpora, hereafter called "Unsup." Our **supervised MT** systems use the transformer-big Vaswani et al. (2017) as implemented in _Lingvo_Shen et al. (2019) with a vocabulary of 32k subword units.

To investigate differences between approaches, we train two **language models** (LMs) on different types of data and calculate the perplexity of translations generated by the supervised and unsupervised MT systems. We train one LM on the monolingual German News Crawl dataset with a decoder-only transformer, hereafter called the "natural text LM" (nLM). We train another on machine translated sentences which we call the "translated text LM" (tLM). We generate the training corpus by translating the English News Crawl dataset into German with a English\(\rightarrow\)German transformer-big model trained on the WMT18 bitext.

### Human Evaluations

Human evaluation complements automatic evaluation and abstracts away from comparison to a human reference which favors the characteristics of translated text Freitag et al. (2020). We score adequacy using direct assessment and run side-by-side evaluations measuring fluency and adequacy preference between systems. Each campaign has 1,000 test items. For side-by-side eval, a test item includes a pair of translations of the same source sentence: one from the supervised system and one from the unsupervised. We hire 12 professional translators, who are more reliable than crowd workers Toral (2020); Freitag et al. (2021).

Direct Assessment AdequacyWe use the template from the WMT 2019 evaluation campaign. Human translators assess a translation by how adequately it expresses the meaning of the source sentence on a 0-100 scale. Unlike WMT, we report the average rating and do not normalize the scores.

Side-by-side AdequacyRaters see a source sentence with two translations (one supervised, one unsupervised) and rate each on a 6-point scale.

Side-by-side FluencyRaters assess the alternative translations (one supervised, one unsupervised) without the source, and rate each on a 6-point scale.

## 4 Unsupervised vs. Supervised MT

The goal of this section is to analyse supervised and unsupervised systems of similar overall translation quality so that differences in quality do not confound analyses. As unsupervised systems underperform supervised systems, we use a smaller parallel corpus (news commentary) to train systems of similar quality. Table 1 summarizes the BLEU scores and human side-by-side adequacy results for both systems. Although the supervised system is below state-of-the-art, these experiments help elucidate how unsupervised and supervised output is different. Overall BLEU and human ratings suggest similar translation quality. Nevertheless, we observe notable differences between orig-de and orig-en sides of the test set when comparing both systems. Recall that orig-de has natural German text on the target side. Unsup scores higher than Sup on orig-de, suggesting that its output is more natural-sounding 

as it better matches text originally written in German. Performance discrepancies on orig-en and orig-de indicate that differences in system output may exist and prompt further investigation.

### Selecting Translations of Same Adequacy

To assess the translation style and compare linguistic aspects of supervised and unsupervised MT, we further must compare translations that have the same accuracy level on the segment level, so that neither confounds analysis. We use the adequacy evaluation from Table 1 and retain sentences for which both approaches yield similar adequacy scores. We divide the rating scale into bins of low (0-2), medium (3-4), and high (5-6) adequacy. Table 2 shows the percentage of sentences in each bin. For each source sentence, there is one translation by Unsup and one by Sup. If human judges assert that both translations belong in the same adequacy bin, that sentence also appears in "Both." There are 86, 255, and 218 sentences in "Both" for low, medium, and high bins, respectively. For subsequent analyses, we examine sentences falling into "Both."

### Comparing Translation Style

Measuring Structural SimilarityWe develop a metric to ascertain the degree of structural similarity between two sentences, regardless of language. When evaluated on a source-translation pair, it measures the influence of the source structure on the structure of the output without penalizing for differing word choice; thus it is a measure of "monotonicity" - the degree to which words are translated in-order. Given alternative translations in the same language, it assesses the degree of structural similarity between the two. Thus given a machine translation and a human translation of the same source sentence, it can measure the structural similarity between the machine and human translations.

Word alignment seems well-suited here. Like Birch (2011), we calculate Kendall's tau (Kendall, 1938) over alignments of source-translation pairs, but do not simplify alignments to permutations. We use fast_align (Dyer et al., 2013) but observe that it struggles to align words not on the diagonal, so sometimes skipped alignments.4 Because of this issue, we instead estimate monotonicity/structural similarity using the new metric, introduced next.

Footnote 4: We ran fast_align with and without diagonal-favoring and all 5 symmetrization heuristics, and see similar trends.

We propose measuring translation edit rate (TER, Snover et al. (2006)) over _POS tag sequences_. TER is a well-known word-level translation quality metric which measures the number of edits required to transform a "hypothesis" sentence into the reference, outputting a "rate" by normalizing by sentence length. Between languages, we compute TER between POS tag sequences of the source text (considered the reference) and the translation (considered the hypothesis), so that TER now measures changes in _structure_ independent of word choice. Source-target POS sequences which can be mapped onto each other with few edits are considered similar--a sign of a monotonic translation. Given a machine translation (hypothesis) and a human reference in the same language, TER over POS tags measures _structural similarity between the machine and human translations_. Outputs with identical POS patterns score 0, increasing to 1+ as sequences diverge. Lower TER for (source, translation) pairs indicates monotonic translation; Lower TER for (machine translation, human translation) pairs indicates structural similarity to human translation. We call the metric "posTER".

MonotonicityPOS sequences are comparable across languages thanks to universal POS tags. Table 3 has a toy example with two possible German translations of an English source. Next to each sentence is its universal dependencies POS sequence. In the third column, TER is calculated with the POS sequence of the English source as reference and the sequence of the translation as hypothesis.

Table 4 shows posTER over universal dependencies of German translations versus the newstest2018 (orig-en) source sentences. While the standard newstest2018 references (Ref) score 0.410, newstest2018p's (RefP) higher score of 0.546 reflects the fact that the paraphrase references are designed to have different structure than the source. Difference in overall monotonicity between Sup and Unsup is unapparent at this granularity.

Because universal dependencies are designed to

\begin{table}
\begin{tabular}{c|c c c|c}  & **Overall** & **orig-en** & **orig-de** & **nt18p** \\ \hline
**Sup** & 29.2 & 34.0 & 21.1 & 9.3 & 3.89 \\
**Unsup** & 30.1 & 30.9 & 27.1 & 9.6 & 3.82 \\ \end{tabular}
\end{table}
Table 1: SacreBLEU & human adequacy (orig-en) on newstest2018 and newstest2018p (nt18p = paraphrase reference). nt18p is available for orig-en only.

\begin{table}
\begin{tabular}{r|c|c|c}  & **Low** & **Medium** & **High** \\ \hline
**Sup** & 18.7\% & 42.1\% & 39.2\% \\
**Unsup** & 19.3\% & 44.6\% & 36.1\% \\
**Both** & 8.6\% & 25.5\% & 21.8\% \\ \end{tabular}
\end{table}
Table 2: Percentage of sentences with low, medium and high human-evaluated adequacy ratings. “Both” are sentences which have same rating from both systems.

 

suit many languages, the 17 UD categories may be too broad to adequately distinguish moderate structural difference. Whereas UD has a single class for "VERB," the finer-grained German TIGER tags distinguish between 8 sub-verb types including infinitive, modal, and imperative. We use these language-specific categories next to uncover differences between systems that broad categories conceal.

Similarity to Human TranslationRecall that supervised MT essentially mimics human translators, while unsupervised MT learns to translate without examples. Intuitively, supervised MT output might be stylistically more like human translation, even when controlling for quality. The first indication is Sup's lower BLEU score on nt18p--the paraphrase test set designed to have structure different than the original human translation.

We compare the structure of MT output with the human reference using German TIGER tags. Lower posTER indicates more structural similarity, while higher posTER indicates stylistic deviation from human translation. Comparison with the newstest2018 orig-en human reference is in Table 5. Sup and Unsup show negligible difference overall, but binning by adequacy shows Unsup output as _less structurally similar to the human reference on the high-end of adequacy_, and more similar on the low-end. This suggests systematic difference between systems, and that unsupervised MT might have more structural diversity as quality improves.

NaturalnessThe first hint that Unsup might produce more natural output than Sup is its markedly higher BLEU on the orig-de test set: 27.1, versus 21.1 from Sup. Recall that orig-de has natural German on the target side, so higher BLEU here means higher n-gram overlap with natural German.

Edunov et al. (2020) recommend augmenting BLEU-based evaluation with perplexity from a language model (LM) to assess fluency or naturalness of MT output. Perplexity Jelinek et al. (1977) measures similarity of a text sample to a model's training data. We contrast the likelihood of output according to two LMs: one trained on machine-translated text (tLM) and another trained on non-translated natural text (nLM). While machine-translated and human-translated text differ, the LMs are nonetheless a valuable heuristic and contribute insights on whether systematic differences between MT system outputs exist. Low perplexity from the nLM indicates natural language. Low perplexity from the tLM (trained on English News Crawl that has been machine-translated into German) shows proximity to training data composed of translated text, indicating simplified language.

Sup perplexity is lower than Unsup across adequacy bins for the tLM, seen in Table 6. Conversely, Sup generally has higher perplexity from the nLM. All adequacy levels for Unsup have similar nLM perplexity, suggesting it is particularly skilled at generating fluent output. Together, these findings suggest that _unsupervised MT output is more natural_ than supervised MT output.

Stronger Supervised MTThough analyzing systems of similar quality is important for head-to-head comparison, we evaluate a stronger supervised system for context.5 We do not have human evaluation scores, but automatic results give insight: see Table 7. The model has overall BLEU = 40.9 and a similarly large discrepancy on orig-en vs. orig-de as did the Sup system used throughout this

\begin{table}
\begin{tabular}{c||c|c|c|c} \hline  & **nt18** & **nt18p** & **Sup** & **Unsup** \\ \hline
**Src** & 0.410 & 0.546 & 0.409 & 0.399 \\ \hline \end{tabular}
\end{table}
Table 4: posTER (0-1+) over universal dependencies of translations of newstest2018 (orig-en) vs. the source. \(\downarrow\) = more monotonic translation. nt18p=paraphrase ref.

\begin{table}
\begin{tabular}{c||c c} \multicolumn{3}{l|}{} \\
* \\  & _PNOUN DET NOUN PUNCT_ & \\ \hline Ich habe mir heute Morgen einee Tasse & PRON AUX PRON ADV NOUN DET & 0.5 \\ Kaffee gemacht. & NOUN NOUN VERR PUNCT & \\ \hline Heute morgen habe ich mir einee Tasse & ADV ADV AVUX PRON PRON DET & 0.7 \\ Kaffee gemacht. & NOUN NOUN VERG PUNCT & \\ \hline \end{tabular}
\end{table}
Table 3: posTER over universal dependencies POS sequences for example toy German translations of an English source. Row 1 is the source with its POS tag sequence. Rows 2/3 are example translations with the POS tags of each. posTER is calculated via the POS sequences of the translation (hypothesis) and the source (considered the reference).
 

work: 44.6 for orig-en and 34.9 for orig-de. As for structural similarity, this stronger system has lower overall posTER vs. the human reference--0.238 vs. 0.280/0.287 from Sup/Unsup--indicating even more structural similarity with the reference. For naturalness, the stronger system has lower perplexity from the nLM. As a higher-quality system, this is expected. At the same time, it scores much lower than Sup and Unsup by the tLM, where higher indicates more natural-sounding output: 29.23 vs. 41.06/58.17 for Sup/Unsup.

Ablation: Architecture vs. DataOne reason Unsup might produce more natural-sounding output could be simply that it develops language-modeling capabilities from natural German alone, whereas Sup must see some translated data (being trained on bitext of human translations). Next, we ask whether the improved naturalness and structural diversity is due to the unsupervised NMT architecture, or simply the natural training data.

We build a supervised en-de MT system with 329,000 paired lines of translated English source and natural German, where the source is backtranslated German News Crawl from a supervised system. In other words, we train on backtranslated data only on the source side and natural German as the target. The model thus develops its language-modeling capabilities on natural sentences alone. If more natural output is simply a response to training on natural data, then this supervised system should perform as well in naturalness as Unsup, or better.

We train another unsupervised system on translated text only. Source-side training data is synthetic English from translating German News Crawl with a supervised system. Target-side is synthetic German which was machine-translated from English News Crawl. If naturalness solely results from data, this system should perform worst, being trained _only_ on translated (unnatural) text.

Table 8 shows the results. The original unsupervised system (Unsup) performs best according to both LMs, having output that is more natural and less like translated text. When given only natural German to build a language model, the supervised system (Sup En-Trns/De-Orig) _still_ produces more unnatural output than Unsup. Even when the unsupervised system uses translated data only (Unsup-Trns), its output is _still_ more natural than the original supervised system (Sup) according to both LMs. This is a surprising result, and is interesting for future study. Together, these findings suggest that both German-original data _and_ the unsupervised architecture encourage output to sound more natural.

## 5 Application: Leveraging Unsupervised Back-translation

Our results indicate that high-adequacy unsupervised MT output is more natural and more structurally diverse in comparison to human translation, than is supervised MT output. We are thus motivated to use these advantages to improve translation. We explore how to incorporate unsupervised MT into a supervised system via back-translation. We train for \({\sim}\)500,000 updates for each experiment, and select models based on validation performance on newstest2018. We test on newstest2019(p).

### Baselines

The first row of Table 9 is the supervised baseline trained on the WMT18 bitext. The second row is Unsup, used throughout this work.

We back-translate 24 million randomly-selected sentences of German News Crawl twice: once using a supervised German-English system trained on WMT18 bitext with a transformer-big architecture, and once using Unsup. Both use greedy decoding for efficiency. We augment the WMT18 bitext with either the supervised or unsupervised BT.

Seen in Table 9, adding supervised BT (+SupBT) performs as expected; minorly declining on the source-original test set (orig-en), improving on the target-original set (orig-de), and improving on

\begin{table}
\begin{tabular}{c||c c c c||c|c c c}  & \multicolumn{4}{c||}{**Natural Text LM**} & \multirow{2}{*}{\(\downarrow\)} & \multicolumn{4}{c}{**Translated Text LM**} \\ \hline  & _Overall_ & _Low_ & _Medium_ & _High_ & _Overall_ & _Low_ & _Medium_ & _High_ \\ \hline
**Supp** & 72.69 & 90.61 & 76.36 & **68.37** & 41.06 & 51.91 & 40.32 & 36.70 \\
**Unsp** & **67.01** & **68.32** & **60.56** & 69.88 & **58.17** & **61.50** & **53.71** & **57.95** \\ \end{tabular}
\end{table}
Table 6: Perplexity of MT output on newstest2018 based on LMs trained on natural text vs. translated text, binned by adequacy. Sup and Unsup are comparable supervised and unsupervised MT systems, respectively. \(\downarrow\) from the Natural Text LM and \(\uparrow\) from the Translated Text LM indicate more natural-sounding output.

 

the paraphrase set (nt19p). Conversely, adding unsupervised BT (+UnsupBT) severely lowers BLEU on source-original and paraphrase test sets. Randomly-partitioning the BT sentences such that 50% are supervised BT and 50% are unsupervised also lowers performance on orig-en (+50-50BT).

### Tagged BT

Following Caswell et al. (2019), we tag BT on the source-side. Tagging aids supervised BT (+SupBT_Tag) and greatly improves unsupervised BT (+UnsupBT_Tag), which outperforms the baseline and is nearly on-par with +SupBT_Tag. Combining supervised and unsupervised BT using the same tag for both (+50-50BT_Tag) shows no benefit over +SupBT_Tag. +50-50BT_TagDiff uses different tags for supervised vs. unsupervised BT.

### Probability-Based BT Selection

We design a BT selection method based on translation probability to exclude unsupervised BT of low quality. We assume that supervised BT is "good enough." Given translations of the same source sentence (one supervised, one unsupervised) we assert that an unsupervised translation is "good enough" if its translation probability is similar or better than that of the supervised translation. If much lower, the unsupervised output may be low-quality.

* Score each supervised and unsupervised BT with a supervised de-en system.
* Normalize the translation probabilities to control for translation difficulty and output length.
* Compare probability of the supervised and unsupervised BT of the same source sentence: \[\Delta P=\frac{P\text{norm}(\text{unsup})}{P\text{norm}(\text{sup})}\]
* Sort translation pairs by \(\Delta\)P.
* Select the unsupervised BT for pairs scoring highest \(\Delta\)P and the supervised BT for the rest.

This filters out unsupervised outputs less than a hyperparameter T% as likely as the corresponding supervised sentence and swaps them with the corresponding supervised sentence. Importantly, the same 24M source sentences are used in all experiments. The procedure is shown in Figure 1.

Full results varying T are in the Appendix for brevity, but we show two example systems in Table 9. The model we call "+MediumMix_Tag" uses the top \(\sim\)40% of ranked unsupervised BT with the rest supervised (9.4M unsupervised, 14.6M supervised). "+SmallMix_Tag" uses the top \(\sim\)13% of unsupervised BT (3.1M unsupervised, 20.9M supervised).6 We use the same tag for all BTs. Improvements are modest, but our goal was to demonstrate how one might use unsupervised MT output rather than build a state-of-the-art system.

Footnote 6: The numbers are not round because data was selected using round numbers for the hyperparameter T.

+SmallMix_Tag performs better than the previous best on newstest2018p and +MediumMix_Tag performs highest overall on nt19p. We recall

Figure 1: Back-translation selection method. Both systems translate the same source sentences. If an unsupervised output sentence is more than T% as likely as the supervised one, select the unsupervised. Here, T=65%.

 

that small differences on paraphrase test sets can signal tangible quality differences Freitag et al. (2020). Trusting BLEU on nt19p, we use _+MediumMix\(\_\)Tag_ as our model for human evaluation.

One might inquire whether improved performance is due to the simple addition of noise in light of Edunov et al. (2018), who conclude that noising BT improves MT quality. Subsequent work, however, found that benefit is not from the noise itself but rather that noise helps the system distinguish between parallel and synthetic data Caswell et al. (2019); Marie et al. (2020). Yang et al. (2019) also propose tagging to distinguish synthetic data. With tagging instead of noising, Caswell et al. (2019) outperform Edunov et al. (2018) in 4 of 6 test sets for En-De, furthermore find that noising on top of tagging does not help. They conclude that "tagging and noising are not orthogonal signals but rather different means to the same end." In light of this, our improved results are likely not due to increased noise but rather to systematic differences between supervised and unsupervised BT.

### Human Evaluation

We run human evaluation with professional translators for _+MediumMix\(\_\)Tag_, comparing its output translation of the newstest2019 test set with two baseline models. Table 10 shows that humans prefer the combined system over the baseline outputs.7 Table 11 shows the percentage of sentences judged as "worse than," "about the same as," or "better than" the corresponding +SupBT\(\_\)Tag output, based on fluency. Raters again prefer the combined system. The improvements are modest, but encouragingly indicate that unsupervised MT may have something to contribute to machine translation, even in high-resource settings.

Footnote 7: Scores are low because we use only WMT18 bitext + BT, and translators score more harshly than crowd workers.

## 6 Conclusion

Recent unsupervised MT systems can reach reasonable translation quality under clean and controlled data conditions, and could bring alternative translations to language pairs with ample parallel data. We perform the first systematic comparison of supervised and unsupervised MT output from systems of similar quality. We find that systematic differences do exist, and that high-quality unsupervised MT output appears **more natural** and **more structurally diverse** when compared to human translation, than does supervised MT output. Our findings indicate that there may be useful differences between supervised and unsupervised MT systems that could contribute to a system better than either achieves alone. As a first step, we demonstrate an unsupervised back-translation augmented model that takes advantage of the differences between the translation methodologies to outperform a traditional supervised system on human-evaluated measures of adequacy and fluency.

\begin{table}
\begin{tabular}{c|c|c} \hline Better & Same & Worse \\ \hline
51.1\% & 3.7\% & 45.2\% \\ \end{tabular}
\end{table}
Table 11: Human side-by-side fluency eval. Shown: % of +MediumMix\(\_\)Tag sentences judged “worse than,” “about the same,” or “better than” +SupBT\(\_\)Tag output.

\begin{table}
\begin{tabular}{l||c|c|c|c||c|c|c} \hline  & \multicolumn{4}{c||}{**newstest2018**} & \multicolumn{4}{c}{**newstest2019**} \\ \hline  & **Overall** & **orig-en** & **orig-de** & **nt18p** & **orig-en** & **orig-de** & **nt19p** \\ \hline Supervised Baseline (5.2M) & 41.8 & 46.1 & 34.3 & 12.6 & 38.8 & 30.4 & 11.7 \\ Unsup _(used throughout this work)_ & 30.1 & 30.9 & 27.1 & 9.6 & 24.6 & 28.5 & 8.8 \\ \hline _Supervised Baseline_ & & & & & & & \\ + SupBT & 43.4 & 43.7 & 41.8 & 12.5 & 37.0 & **39.9** & 12.0 \\ + UnsupBT & 33.3 & 33.8 & 31.1 & 9.9 & 27.2 & 30.8 & 9.5 \\ + 50-50BT & 38.0 & 36.4 & 39.0 & 12.9 & 29.4 & 38.3 & 10.0 \\ \hline + SuppT\(\_\)Tag & **44.8** & 47.0 & 40.7 & 13.0 & **40.3** & 38.2 & 12.4 \\ + UnsupBT\(\_\)Tag & 43.3 & 46.9 & 36.9 & 12.9 & 39.1 & 35.0 & 12.2 \\ + 50-50BT\(\_\)Tag & 44.4 & **47.1** & 39.6 & 12.9 & 39.4 & 38.0 & 12.2 \\ + 50-50BT\(\_\)TagDiff & 44.4 & 46.8 & 40.1 & 13.0 & 39.9 & 37.9 & 12.4 \\ \hline + SmallMix\(\_\)Tag & **44.8** & 46.8 & 40.8 & **13.2** & 39.8 & 38.8 & 12.5 \\ + MediumMix\(\_\)Tag & 44.7 & 46.8 & 40.8 & 13.0 & 40.1 & 38.2 & **12.6** \\ \hline \end{tabular}
\end{table}
Table 9: SacreBLEU of supervised baseline plus 24M supervised or unsupervised BTs. +MediumMix\(\_\)Tag and +SmallMix\(\_\)Tag use the BT selection method of §5.3. +MediumMix\(\_\)Tag has 9.4M unsupervised BT and 14.6M supervised BT. +SmallMix\(\_\)Tag has 3.1M and 20.9M, respectively. nt18p and nt19p are paraphrase references from Freitag et al. (2020), where small BLEU scores changes can indicate tangible quality difference.

 



## References

* Albert et al. (2003) Stefanie Albert, Jan Anderssen, Regine Bader, Stephanie Becker, Tobias Bracht, Sabine Brants, Thorsten Brands, Vera Demberg, Stefanie Dipper, Peter Eisenberg, et al. 2003. tiger annotationsschema. _Universitat des Saarlandes and Universitati Stuttgart and Universitat Potsdam_.
* Artetxe et al. (2019) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 194-203, Florence, Italy. Association for Computational Linguistics.
* Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In _Proceedings of the Sixth International Conference on Learning Representations_.
* Baker (1993) Mona Baker. 1993. Corpus Linquistics and Translation Studies: Implications and Applications. _Text and technology: in honour of John Sinclair_, pages 233-252.
* Birch (2011) Alexandra Birch. 2011. Reordering metrics for statistical machine translation.
* Birch et al. (2009) Alexandra Birch, Phil Blunsom, and Miles Osborne. 2009. A quantitative analysis of reordering phenomena. In _Proceedings of the Fourth Workshop on Statistical Machine Translation_, pages 197-205.
* Birch and Osborne (2011) Alexandra Birch and Miles Osborne. 2011. Reordering metrics for mt. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 1027-1035.
* Birch et al. (2010) Alexandra Birch, Miles Osborne, and Phil Blunsom. 2010. Metrics for mt evaluation: evaluating reordering. _Machine Translation_, 24(1):15-26.
* Birch et al. (2008) Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_, pages 745-754, Honolulu, Hawaii. Association for Computational Linguistics.
* Bisazza and Federico (2016) Arianna Bisazza and Marcello Federico. 2016. A survey of word reordering in statistical machine translation: Computational models and language phenomena. _Computational linguistics_, 42(2):163-205.
* Bogoychev and Sennrich (2019) Nikolay Bogoychev and Rico Sennrich. 2019. Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation. _arXiv preprint arXiv:1911.03362_.
* Bojar et al. (2018) Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (wmt18). In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 272-307, Belgium, Brussels. Association for Computational Linguistics.
* Brants et al. (2004) Sabine Brants, Stefanie Dipper, Peter Eisenberg, Silvia Hansen-Schirra, Esther Konig, Wolfgang Lezius, Christian Rohrer, George Smith, and Hans Uszkoreit. 2004. Tiger: Linguistic interpretation of a german corpus. _Research on language and computation_, 2(4):597-620.
* Caswell et al. (2019) Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 53-63, Florence, Italy. Association for Computational Linguistics.
* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In _Advances in Neural Information Processing Systems_, pages 7057-7067.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 489-500.
* Edunov et al. (2020) Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2836-2846.
* Fomicheva and Specia (2016) Marina Fomicheva and Lucia Specia. 2016. Reference bias in monolingual machine translation evaluation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 77-82, Berlin, Germany. Association for Computational Linguistics.
* Fox (2002) Heidi Fox. 2002. Phrasal cohesion and statistical machine translation. In _Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)_, pages 304-311. Association for Computational Linguistics

Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 34-44, Florence, Italy. Association for Computational Linguistics.
* Freitag et al. [2021] Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation.
* Freitag et al. [2020] Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 61-71, Online. Association for Computational Linguistics.
* Graham et al. [2019] Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation.
* Jelinek et al. [1977] Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity--a measure of the difficulty of speech recognition tasks. _The Journal of the Acoustical Society of America_, 62(S1):S63-S63.
* Kendall [1938] Maurice G Kendall. 1938. A new measure of rank correlation. _Biometrika_, 30(1/2):81-93.
* Volume 1_, HLT '11, pages 1318-1326, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Kurokawa et al. [2009] David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In _Proceedings of MT-Summit XII_, pages 81-88.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Lample et al. [2018b] Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. _CoRR_, abs/1804.07755.
* Lembersky et al. [2012] Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationes improves SMT. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_, EACL '12, pages 255-265, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Liu et al. [2020] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. _arXiv preprint arXiv:2001.08210_.
* Ma et al. [2017] Qingsong Ma, Yvette Graham, Timothy Baldwin, and Quin Liu. 2017. Further investigation into reference bias in monolingual evaluation of machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2476-2485, Copenhagen, Denmark. Association for Computational Linguistics.
* Marie et al. [2020] Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020. Tagged back-translation revisited: Why does it really work? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5990-5997, Online. Association for Computational Linguistics.
* Mauranen and Kujamaki [2004] Anna Mauranen and Pekka Kujamaki. 2004. _Translation universals: Do they exist?_, volume 48. John Benjamins Publishing.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_, pages 311-318. Association for Computational Linguistics.
* Post [2018] Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Riley et al. [2020] Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020. Translationale as a language in "multilingual" NMTI. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7737-7746, Online. Association for Computational Linguistics.
* Selinker [1972] Larry Selinker. 1972. Interlanguage. _International Review of Applied Linguistics_, pages 209-241.
* Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Shen et al. [2019] Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X. Chen, Ye Jia, Anjuili Kannan, Tara N. Sainath, and Yuan Cao et al. 2019. Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling. _CoRR_, abs/1902.08295.
* Siddhant et al. [2020] Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020. Leveraging monolingual data with self-supervision for multilingual neural machine translation. _arXiv preprint arXiv:2005.04816_ 

Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In _Proceedings of association for machine translation in the Americas_, 6. Cambridge, MA.
* Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In _ICML_.
* Toral (2019) Antonio Toral. 2019. Post-editese: an exacerbated translationese. In _Proceedings of Machine Translation Summit XVII Volume 1: Research Track_, pages 273-281, Dublin, Ireland. European Association for Machine Translation.
* Toral (2020) Antonio Toral. 2020. Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019. _arXiv preprint arXiv:2005.05738_.
* toral et al. (2018) Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 113-123, Belgium, Brussels. Association for Computational Linguistics.
* Toury (1995) Gideon Toury. 1995. _Descriptive Translation Studies and Beyond_. Benjamins translation library. John Benjamins Publishing Company.
* Vanmassenhove et al. (2021) Eva Vanmassenhove, Dimitar Shterionov, and Matthew GWilliam. 2021. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2203-2213.
* Vannassenhove et al. (2019) Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019. Lost in translation: Loss and decay of linguistic richness in machine translation. In _Proceedings of Machine Translation Summit XVII Volume 1: Research Track_, pages 222-232, Dublin, Ireland. European Association for Machine Translation.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Wellington et al. (2006) Benjamin Wellington, Sonja Waxmonsky, and I. Dan Melamed. 2006. Empirical lower bounds on the complexity of translational equivalence. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_, pages 977-984, Sydney, Australia. Association for Computational Linguistics.
* Yang et al. (2019) Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2019. Effectively training neural machine translation models with monolingual data. _Neurocomputing_, 333:240-247.
* Zhang and Toral (2019) Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. _CoRR_, abs/1906.08069.

 



\begin{table}
\begin{tabular}{l||c|c|c|c|c|c|c|c}  & \multicolumn{4}{c||}{**newstest2018**} & \multicolumn{3}{c}{**newstest2019**} \\ \hline  & **Joint** & **Orig-En** & **Orig-De** & **nt18p** & **Orig-En** & **Orig-De** & **nt19p** \\ \hline Supervised Baseline (5.2M) & 41.8 & 46.1 & 34.3 & 12.6 & 38.8 & 30.4 & 11.7 \\ Unsup _(same used throughout this work)_ & 30.1 & 30.9 & 27.1 & 9.6 & 24.6 & 28.5 & 8.8 \\ \hline _Supervised Baseline_ & & & & & & & \\ + SupBT & 43.4 & 43.7 & 41.8 & 12.5 & 37.0 & **39.9** & 12.0 \\ + UnsupBT & 33.3 & 33.8 & 31.1 & 9.9 & 27.2 & 30.8 & 9.5 \\ + 50-50BT & 38.0 & 36.4 & 39.0 & 12.9 & 29.4 & 38.3 & 10.0 \\ \hline + SupBT\_Tag & **44.8** & 47.0 & 40.7 & 13.0 & **40.3** & 38.2 & 12.4 \\ + UnsupBT\_Tag & 43.3 & 46.9 & 36.9 & 12.9 & 39.1 & 35.0 & 12.2 \\ + 50-50BT\_Tag & 44.4 & **47.1** & 39.6 & 12.9 & 39.4 & 38.0 & 12.2 \\ + 50-50BT\_TagDiff & 44.4 & 46.8 & 40.1 & 13.0 & 39.9 & 37.9 & 12.4 \\ \hline + 21.7M Tagged Unsup \& 2.3M Sup BT & 44.0 & 46.6 & 39.3 & 13.0 & 39.6 & 36.9 & 12.3 \\ + 17.4M Tagged Unsup \& 6.6M Sup BT & 44.0 & 46.2 & 40.0 & 13.0 & 40.0 & 37.7 & 12.3 \\ + 9.4M Tagged Unsup \& 14.6M Sup BT (+MediumMix\_Tag) & 44.7 & 46.8 & 40.8 & 13.0 & 40.1 & 38.2 & **12.6** \\ + 3.1M Tagged Unsup \& 20.9M Sup BT (+SmallMix\_Tag) & **44.8** & 46.8 & 40.8 & **13.2** & 39.8 & 38.8 & 12.5 \\ + 1.5M Tagged Unsup \& 22.5M Sup BT & **44.8** & **47.1** & 40.7 & **13.2** & 40.0 & 38.4 & 12.5 \\ + 680K Tagged Unsup \& 23.3M Sup BT & 44.4 & 46.4 & 40.7 & 12.9 & 40.0 & 38.1 & 12.4 \\ \end{tabular}
\end{table}
Table 12: SacreBLEU of supervised baseline plus 24M supervised or unsupervised BTs. Systems using both use the BT selection method of §5.3 with increasing values for hyperparameter \(T\). nt18p and nt19p are paraphrase references from Freitag et al. (2020), where small BLEU score changes can indicate tangible quality difference.