SlowGuess/NAACL_2022_OCR huggingface dataset:
Kelly Marchisio* Johns Hopkins University kmarc@jhu.edu
Markus Freitag David Grangier Google Research {freiag, grangier}  $@$  google.com
Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask whether the different training methods result in systematically different output beyond what is visible via quality metrics like adequacy or BLEU. We compare translations from supervised and unsupervised MT systems of similar quality, finding that unsupervised output is more fluent and more structurally different in comparison to human translation than is supervised MT. We then demonstrate a way to combine the benefits of both methods into a single system which results in improved adequacy and fluency as rated by human evaluators. Our results open the door to interesting discussions about how supervised and unsupervised MT might be different yet mutually-beneficial.
Supervised machine translation (MT) utilizes parallel bitext to learn to translate. Ideally, this data consists of natural texts and their human translations. In a way, the goal of supervised MT training is to produce a machine that mimics human translators in their craft. Unsupervised MT, on the other hand, uses monolingual data alone to learn to translate. Critically, unsupervised MT never sees an example of human translation, and therefore must create its own style of translation. Unlike supervised MT where one side of each training sentence pair must be a translation, unsupervised MT can be trained with natural text alone.
In this work, we investigate the output of supervised and unsupervised MT systems of similar quality to assess whether systematic differences in translation exist. Our exploration of this research area focuses on English  $\rightarrow$  German for which abundant bilingual training examples exist, allowing us
to train high-quality systems with both supervised and unsupervised training.
- We observe systematic differences between the output of supervised and unsupervised MT systems of similar quality. High-quality unsupervised output appears more natural, and more structurally diverse when compared to human translation.
- We show a way to incorporate unsupervised back-translation into a standard supervised MT system, improving adequacy, naturalness, and fluency as measured by human evaluation.
Our results provoke interesting questions about what unsupervised methods might contribute beyond the traditional context of low-resource languages which lack bilingual training data, and suggest that unsupervised MT might have contributions to make for high-resource scenarios as well. It is worth exploring how combining supervised and unsupervised setups might contribute to a system better than either creates alone.
We discuss related work in §2. In §3, we introduce the dataset, model details, and evaluation setups. In §4, we characterize the differences between the output of unsupervised and supervised neural MT systems of similar quality. In §5, we demonstrate a combined system which benefits from the complementary strengths of the two methods. We summarize the paper in §6.
Unsupervised MT Two paradigms for unsupervised MT are finding a linear transformation to align two monolingual embedding spaces (Lample et al., 2018a,b; Conneau et al., 2018; Artetxe et al., 2018, 2019), and pretraining a bi-/multilingual language model then finetuning on a translation task (Conneau and Lample, 2019; Song et al., 2019; Liu et al., 2020). We study the Masked Sequence
* Work completed at Google Translate Research.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2214 - 2225 July 10-15, 2022 ©2022 Association for Computational Linguistics
to-Sequence Pretraining (MASS) language model pretraining paradigm of Song et al. (2019). MASS is an encoder-decoder trained jointly with a masked language modeling objective on monolingual data. Iterative back-translation (BT) follows pretraining.
Monolingual Data in MT BT is widely-used to exploit monolingual data (Sennrich et al., 2016). "Semi-supervised" systems use monolingual and parallel data to improve performance (e.g. Artetxe et al. (2018)). Siddhant et al. (2020) combine multilingual supervised training with MASS for many languages and zero-shot translation.
Source Artifacts in Translated Text Because supervised MT is trained ideally on human-generated translation, characteristics of human translation affects the style of machine translation from such systems. Dubbed "translationese," human translation includes source language artifacts (Koppel and Ordan, 2011) and source-independent artifacts—Translation Universals (Mauranen and Kujamäki, 2004). There are systematic biases inherent to translated texts (Baker, 1993; Selinker, 1972), and biases coming from interference from source text (Toury, 1995). In MT, Freitag et al. (2019, 2020) attribute these patterns as a source of mismatch between BLEU (Papineni et al., 2002) and human evaluation measures of quality, raising concerns that overlap-based metrics reward hypotheses with the characteristics of translated text more than those with natural language. Vanmassenhove et al. (2019, 2021) note loss of linguistic diversity and richness from MT, and Toral (2019) see related effects even after human post-editing. The impact of translated text on human evaluation has also been studied (Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2019; Fomicheva and Specia, 2016; Ma et al., 2017), as has the impact in training data (Kurokawa et al., 2009; Lembersky et al., 2012; Bogoychev and Sennrich, 2019; Riley et al., 2020).
Measuring Word Reordering Word reordering models are well-studied because they formed a critical part of statistical MT (see Bisazza and Federico (2016) for a review). Others examined metrics for measuring reordering in translation (e.g. Birch et al., 2008, 2009, 2010). Wellington et al. (2006) and Fox (2002) use part-of-speech (POS) tags in the context of parse trees, and Fox (2002) measure the similarity of French and English with respect to phrasal cohesion by calculating alignment crossings using parse trees. Most similar to us, Birch
(2011) view simplified word alignments as permutations and compare distance metrics over these to quantify the amount of reordering done. They use TER computed over the alignments as a baseline. Birch and Osborne (2011)'s LRScore interpolates a reordering metric with a lexical translation metric.
Training Experiments are in English  $\rightarrow$  German. For the main study comparing supervised and unsupervised MT, we use News Commentary v14 (329,000 sentences) as parallel bitext for the supervised system, and News Crawl 2007-17 as monolingual data for the unsupervised system. Deduplicated News Crawl 2007-17 has 165 million English sentences and 226 million German sentences.
The combined system demonstration at the end of our work utilizes a BT selection method. We use the bilingual training data from WMT2018 (Bojar et al., 2018) (News Commentary v13, Europarl v7, Common Crawl, EU Press Release) so that our model can be compared with well-known work using BT (e.g. Edunov et al., 2018; Caswell et al., 2019). We deduplicate and filter out pairs with  $>250$  tokens in either language or length ratio over 1.5, resulting in 5.2 million paired sentences.
Development and Test Sets For the main experiments, we use newstest2017 as the dev set with newstest2018 and newstest2019 for test. newstest2018 was originally created by translating one half of the test data from English  $\rightarrow$  German (origen) and the other half from German  $\rightarrow$  English (origde). Since 2019, WMT produces newstest sets with only source-original text and human translations on the target side to mitigate known issues when translating and evaluating on target- original data (e.g. Koppel and Ordan, 2011; Freitag et al., 2019).
For most experiments, we evaluate on orig-en sentences only to reflect the real use-case for translation and modern evaluation practice. We examine orig-de only for BLEU score as an additional data point of difference between supervised and unsupervised MT. Zhang and Toral (2019) show that target-language-original text should not be used for human evaluation (orig-de, in our case).
We use the newstest2018 "paraphrased" test references from Freitag et al. (2020),<sup>1</sup> which are made
$^{1}$ github.com/google/wmt19-paraphrased-references
for orig-en sentences only. These additional references have different structure than the source sentence but maintain semantics, and provide a way to measure system quality without favoring translations with the same structure as the source. Observing work that uses these references, BLEU is typically much lower than on original test sets, and score differences tend to be small but reflect tangible quality difference (Freitag et al., 2020).
For the system combination demonstration, we use newstest2018 for development and newstest2019 for test. We also use newstest2019 German  $\rightarrow$  English and swap source and target to make an orig-de English  $\rightarrow$  German test set, and use paraphrase references for newtest2019 (orig-en).
Testing on the official newstest2018 in the main experiments allows us to see interesting differences between unsupervised and supervised MT that are hidden with newstest2019 because it is orig-en only. Using newstest2018 for development in the system combination demonstration aligns with similar literature (e.g. Edunov et al., 2018; Caswell et al., 2019). We use SacreBLEU throughout (Post, 2018).<sup>2</sup>
3.2 Part-of-Speech Tagging
We use part-of-speech taggers for some experiments: universal dependencies (UD) implemented in spaCy<sup>3</sup> and spaCy's language-specific fine-grained POS tags for German from the TIGER Corpus (Albert et al., 2003; Brants et al., 2004).
Our unsupervised MT model is a MASS transformer with the hyperparameters of Song et al. (2019). We train MASS on the News Crawl corpora, hereafter called "Unsup." Our supervised MT systems use the transformer-big (Vaswani et al., 2017) as implemented in Lingvo (Shen et al., 2019) with a vocabulary of  $32k$  subword units.
To investigate differences between approaches, we train two language models (LMs) on different types of data and calculate the perplexity of translations generated by the supervised and unsupervised MT systems. We train one LM on the monolingual German News Crawl dataset with a decoder-only transformer, hereafter called the "natural text LM" (nLM). We train another on machine translated sentences which we call the "translated text LM" (tLM). We generate the training corpus
by translating the English News Crawl dataset into German with a English  $\rightarrow$  German transformer-big model trained on the WMT18 bitext.
3.4 Human Evaluations
Human evaluation complements automatic evaluation and abstracts away from comparison to a human reference which favors the characteristics of translated text (Freitag et al., 2020). We score adequacy using direct assessment and run side-by-side evaluations measuring fluency and adequacy preference between systems. Each campaign has 1,000 test items. For side-by-side eval, a test item includes a pair of translations of the same source sentence: one from the supervised system and one from the unsupervised. We hire 12 professional translators, who are more reliable than crowd workers (Toral, 2020; Freitag et al., 2021).
Direct Assessment Adequacy We use the template from the WMT 2019 evaluation campaign. Human translators assess a translation by how adequately it expresses the meaning of the source sentence on a 0-100 scale. Unlike WMT, we report the average rating and do not normalize the scores.
Side-by-side Adequacy Raters see a source sentence with two translations (one supervised, one unsupervised) and rate each on a 6-point scale.
Side-by-side Fluency Raters assess the alternative translations (one supervised, one unsupervised) without the source, and rate each on a 6-point scale.
4 Unsupervised vs. Supervised MT
The goal of this section is to analyse supervised and unsupervised systems of similar overall translation quality so that differences in quality do not confound analyses. As unsupervised systems underperform supervised systems, we use a smaller parallel corpus (news commentary) to train systems of similar quality. Table 1 summarizes the BLEU scores and human side-by-side adequacy results for both systems. Although the supervised system is below state-of-the-art, these experiments help elucidate how unsupervised and supervised output is different. Overall BLEU and human ratings suggest similar translation quality. Nevertheless, we observe notable differences between orig-de and orig-en sides of the test set when comparing both systems. Recall that orig-de has natural German text on the target side. Unsup scores higher than Sup on orig-de, suggesting that its output is more natural-sounding
$^{2}$ BLEU+case.mixed+langende+numrefs.1+smooth.exp+ {TESTSET}+tok.13a+version.1.4.12
<sup>3</sup>https://spacy.io/, https://universaldependencies.org/
as it better matches text originally written in German. Performance discrepancies on orig-en and orig-de indicate that differences in system output may exist and prompt further investigation.
Overall	orig-en	orig-de	nt18p	Human Adq.
Sup	29.2	34.0	21.1	9.3	3.89
Unsup	30.1	30.9	27.1	9.6	3.82
Table 1: SacreBLEU & human adequacy (orig-en) on newstest2018 and newstest2018p (nt18p = paraphrase reference). nt18p is available for orig-en only.
4.1 Selecting Translations of Same Adequacy
To assess the translation style and compare linguistic aspects of supervised and unsupervised MT, we further must compare translations that have the same accuracy level on the segment level, so that neither confounds analysis. We use the adequacy evaluation from Table 1 and retain sentences for which both approaches yield similar adequacy scores. We divide the rating scale into bins of low (0-2), medium (3-4), and high (5-6) adequacy. Table 2 shows the percentage of sentences in each bin. For each source sentence, there is one translation by Unsup and one by Sup. If human judges assert that both translations belong in the same adequacy bin, that sentence also appears in "Both." There are 86, 255, and 218 sentences in "Both" for low, medium, and high bins, respectively. For subsequent analyses, we examine sentences falling into "Both."
Low	Medium	High
Sup	18.7%	42.1%	39.2%
Unsup	19.3%	44.6%	36.1%
Both	8.6%	25.5%	21.8%
Table 2: Percentage of sentences with low, medium and high human-evaluated adequacy ratings. "Both" are sentences which have same rating from both systems.
4.2 Comparing Translation Style
Measuring Structural Similarity We develop a metric to ascertain the degree of structural similarity between two sentences, regardless of language. When evaluated on a source-translation pair, it measures the influence of the source structure on the structure of the output without penalizing for differing word choice; thus it is a measure of "monotonicity" – the degree to which words are translated in-order. Given alternative translations in the same language, it assesses the degree of structural similarity between the two. Thus given a machine translation and a human translation of the same source sentence, it can measure the structural similarity between the machine and human translations.
Word alignment seems well-suited here. Like Birch (2011), we calculate Kendall's tau (Kendall,
1938) over alignments of source-translation pairs, but do not simplify alignments to permutations. We use fast_align (Dyer et al., 2013) but observe that it struggles to align words not on the diagonal, so sometimes skipped alignments. $^4$  Because of this issue, we instead estimate monotonicity/structural similarity using the new metric, introduced next.
We propose measuring translation edit rate (TER, Snover et al. (2006)) over POS tag sequences. TER is a well-known word-level translation quality metric which measures the number of edits required to transform a "hypothesis" sentence into the reference, outputting a "rate" by normalizing by sentence length. Between languages, we compute TER between POS tag sequences of the source text (considered the reference) and the translation (considered the hypothesis), so that TER now measures changes in structure independent of word choice. Source-target POS sequences which can be mapped onto each other with few edits are considered similar—a sign of a monotonic translation. Given a machine translation (hypothesis) and a human reference in the same language, TER over POS tags measures structural similarity between the machine and human translations. Outputs with identical POS patterns score 0, increasing to  $1+$  as sequences diverge. Lower TER for (source, translation) pairs indicates monotonic translation; Lower TER for (machine translation, human translation) pairs indicates structural similarity to human translation. We call the metric "posTER".
Monotonicity POS sequences are comparable across languages thanks to universal POS tags. Table 3 has a toy example with two possible German translations of an English source. Next to each sentence is its universal dependencies POS sequence. In the third column, TER is calculated with the POS sequence of the English source as reference and the sequence of the translation as hypothesis.
Table 4 shows postTER over universal dependencies of German translations versus the newstest2018 (orig-en) source sentences. While the standard newstest2018 references (Ref) score 0.410, newstest2018p's (RefP) higher score of 0.546 reflects the fact that the paraphrase references are designed to have different structure than the source. Difference in overall monotonicity between Sup and Unsup is unapparent at this granularity.
Because universal dependencies are designed to
<sup>4</sup>We ran fast_align with and without diagonal-favoring and all 5 symmetrization heuristics, and see similar trends.
Sentence	POS Sequence	TER
I made myself a cup of coffee this morning.	PRON VERB PRON DET NOUN ADP PNOUN DET NOUN PUNCT	-
Ich habe mir heute Morgen eine Tasse Kaffee gemacht.	PRON AUX PRON ADV NOUN DET NOUN NOUN VERB PUNCT	0.5
Heute morgen haben ich mir eine Tasse Kaffee gemacht.	ADV ADV AUX PRON PRON DET NOUN NOUN VERB PUNCT	0.7
Table 3: posTER over universal dependencies POS sequences for example toy German translations of an English source. Row 1 is the source with its POS tag sequence. Rows 2/3 are example translations with the POS tags of each. posTER is calculated via the POS sequences of the translation (hypothesis) and the source (considered the reference).
nt18	nt18p	Sup	Unsup
Src	0.410	0.546	0.409	0.399
suit many languages, the 17 UD categories may be too broad to adequately distinguish moderate structural difference. Whereas UD has a single class for "VERB," the finer-grained German TIGER tags distinguish between 8 sub-verb types including infinitive, modal, and imperative. We use these language-specific categories next to uncover differences between systems that broad categories conceal.
Similarity to Human Translation Recall that supervised MT essentially mimics human translators, while unsupervised MT learns to translate without examples. Intuitively, supervised MT output might be stylistically more like human translation, even when controlling for quality. The first indication is Sup's lower BLEU score on nt18p—the paraphrase test set designed to have structure different than the original human translation.
We compare the structure of MT output with the human reference using German TIGER tags. Lower posTER indicates more structural similarity, while higher posTER indicates stylistic deviation from human translation. Comparison with the newstest2018 origin human reference is in Table 5. Sup and Unsup show negligible difference overall, but binning by adequacy shows Unsup output as less structurally similar to the human reference on the high-end of adequacy, and more similar on the low-end. This suggests systematic difference between systems, and that unsupervised MT might have more structural diversity as quality improves.
Table 4: posTER (0-1+) over universal dependencies of translations of newest2018 (orig-en) vs. the source.  $\downarrow =$  more monotonic translation. nt18p=paraphrase ref.
Overall	Low	Med	High
Sup	0.280	0.348	0.282	0.255
Unsup	0.287	0.313	0.298	0.296
Table 5: postTER (0-1+ scale) over TIGER POS tags of system output vs. the human reference, grouped by adequacy (newest2018, orig-en).  $\downarrow =$  greater structural similarity to the human reference.
Naturalness The first hint that Unsup might produce more natural output than Sup is its markedly higher BLEU on the orig-de test set: 27.1, versus 21.1 from Sup. Recall that orig-de has natural German on the target side, so higher BLEU here means higher n-gram overlap with natural German.
Edunov et al. (2020) recommend augmenting BLEU-based evaluation with perplexity from a language model (LM) to assess fluency or naturalness of MT output. Perplexity (Jelinek et al., 1977) measures similarity of a text sample to a model's training data. We contrast the likelihood of output according to two LMs: one trained on machine-translated text (tLM) and another trained on non-translated natural text (nLM). While machine-translated and human-translated text differ, the LMs are nonetheless a valuable heuristic and contribute insights on whether systematic differences between MT system outputs exist. Low perplexity from the nLM indicates natural language. Low perplexity from the tLM (trained on English Newscrawl that has been machine-translated into German) shows proximity to training data composed of translated text, indicating simplified language.
Sup perplexity is lower than Unsup across adequacy bins for the tLM, seen in Table 6. Conversely, Sup generally has higher perplexity from the nLM. All adequacy levels for Unsup have similar nLM perplexity, suggesting it is particularly skilled at generating fluent output. Together, these findings suggest that unsupervised MT output is more natural than supervised MT output.
Stronger Supervised MT Though analyzing systems of similar quality is important for head-to-head comparison, we evaluate a stronger supervised system for context. We do not have human evaluation scores, but automatic results give insight: see Table 7. The model has overall BLEU = 40.9 and a similarly large discrepancy on orig-en vs. orig-de as did the Sup system used throughout this
$^{5}$ Trained on 4.5 million lines of WMT14 bitext.
Natural Text LM ↓	Translated Text LM ↑
Overall	Low	Medium	High	Overall	Low	Medium	High
Sup	72.69	90.61	76.36	68.37	41.06	51.91	40.32	36.70
Unsup	67.01	68.32	60.56	69.88	58.17	61.50	53.71	57.95
Table 6: Perplexity of MT output on newstest2018 based on LMs trained on natural text vs. translated text, binned by adequacy. Sup and Unsup are comparable supervised and unsupervised MT systems, respectively.  $\downarrow$  from the Natural Text LM and  $\uparrow$  from the Translated Text LM indicate more natural-sounding output.
work: 44.6 for orig-en and 34.9 for orig-de. As for structural similarity, this stronger system has lower overall posTER vs. the human reference—0.238 vs. 0.280/0.287 from Sup/Unsup—indicating even more structural similarity with the reference. For naturalness, the stronger system has lower perplexity from the nLM. As a higher-quality system, this is expected. At the same time, it scores much lower than Sup and Unsup by the tLM, where higher indicates more natural-sounding output: 29.23 vs. 41.06/58.17 for Sup/Unsup.
Quality	Structural Sim.	Naturalness
BLEU	nt18p	v. Src	v. Ref	nLM	tLM
40.9	12.1	0.401	0.238	54.35	29.23
Table 7: Strong supervised model trained on WMT14. Structural sim. is posTER: v. Src is comparable to Table 4, v. Ref to Overall in Table 5.  $\downarrow =$  more monotonic. nLM/tLM are Natural/Translated Text LMs of Table 6.