SlowGuess/ACL_2020_OCR huggingface dataset:
Translationese as a Language in “Multilingual” NMT
Parker Riley $^{\triangle*}$ , Isaac Caswell $^{\nabla}$ , Markus Freitag $^{\nabla}$ , David Grangier $^{\nabla}$
$\triangle$  University of Rochester
Machine translation has an undesirable propensity to produce "translationese" artifacts, which can lead to higher BLEU scores while being liked less by human raters. Motivated by this, we model translationese and original (i.e. natural) text as separate languages in a multilingual model, and pose the question: can we perform zero-shot translation between original source text and original target text? There is no data with original source and original target, so we train a sentence-level classifier to distinguish translationese from original target text, and use this classifier to tag the training data for an NMT model. Using this technique we bias the model to produce more natural outputs at test time, yielding gains in human evaluation scores on both adequacy and fluency. Additionally, we demonstrate that it is possible to bias the model to produce translationese and game the BLEU score, increasing it while decreasing human-rated quality. We analyze these outputs using metrics measuring the degree of translationese, and present an analysis of the volatility of heuristic-based train-data tagging.
"Translationese" is a term that refers to artifacts present in text that was translated into a given language that distinguish it from text originally written in that language (Gellerstam, 1986). These artifacts include lexical and word order choices that are influenced by the source language (Gellerstam, 1996) as well as the use of more explicit and simpler constructions (Baker et al., 1993).
These differences between translated and original text mean that the direction in which parallel data (bitext) was translated is potentially important for machine translation (MT) systems. Most
Figure 1: Illustration of MT train+test parallel data, organized into quadrants based on whether the source or target is translated or original.
parallel data is either source-original (the source was translated into the target) or target-original (the target was translated into the source), though sometimes neither side is original because both were translated from a third language.
Figure 1 illustrates the four possible combinations of translated and original source and target data. Recent work has examined the impact of translationese in MT evaluation, using the WMT evaluation campaign as the most prominent example. From 2014 through 2018, WMT test sets were constructed such that  $50\%$  of the sentence pairs are source-original (upper right quadrant of Figure 1) and the rest are target-original (lower left quadrant). Toral et al. (2018), Zhang and Toral (2019), and Graham et al. (2019) have examined the effect of this testing setup on MT evaluation, and have all argued that target-original test data should not be included in future evaluation campaigns because the translationese source is too easy to translate. While target-original test data does have the downside of a translationese source side, recent work has also shown that human raters prefer MT output that is closer in distribution to original target text than
*Work done while at Google Research.
7737
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7737-7746 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
translationese (Freitag et al., 2019). This indicates that the target side of test data should also be original (upper left quadrant of Figure 1); however, it is unclear how to produce high-quality test data (let alone training data) that is simultaneously source- and target-original.
Because of this lack of original-to-original sentence pairs, we frame this as a zero-shot translation task, where translationese and original text are distinct languages or domains. We adapt techniques from zero-shot translation with multilingual models (Johnson et al., 2016), where the training pairs are tagged with a reserved token corresponding to the domain of the target side: translationese or original text. Tagging is helpful when the training set mixes data of different types by allowing the model to 1) see each pair's type in training to preserve distinct behaviors and avoid regressing to a mean/dominant prediction across data types, and 2) elicit different behavior in inference, i.e. providing a tag at test time yields predictions resembling a specific data type. We then investigate what happens when the input is an original sentence in the source language and the model's output is also biased to be original, a scenario never observed in training.
Tagging in this fashion is not trivial, as most MT training sets do not annotate which pairs are source- original and which are target-original  ${}^{1}$  ,so in order to distinguish them we train binary classifiers to distinguish original and translated target text.
Finally, we perform several analyses of tagging these "languages" and demonstrate that tagged back-translation (Caswell et al., 2019) can be framed as a simplified version of our method, and thereby improved by targeted decoding.
1. We propose two methods to train translationese classifiers using only monolingual text, coupled with synthetic text produced by machine translation.
2. Using only original  $\rightarrow$  translationese and translationese  $\rightarrow$  original training pairs, we apply techniques from zero-shot multilingual MT to enable original  $\rightarrow$  original translation.
3. We demonstrate with human evaluations that this technique improves translation quality, both in terms of fluency and adequacy.
4. We show that biasing the model to instead produce translationese outputs inflates BLEU scores while harming quality as measured by human evaluations.
2 Classifier Training + Tagging
Motivated by prior work detailing the importance of distinguishing translationese from original text (Kurokawa et al., 2009; Lembersky et al., 2012; Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2019; Freitag et al., 2019; Edunov et al., 2019) as well as work in zero-shot translation (Johnson et al., 2016), we hypothesize that performance on the source-original translation task can be improved by distinguishing target-original and target-translationese examples in the training data and constructing an NMT model to perform zero-shot original→original translation.
Because most MT training sets do not annotate each sentence pair's original language, we train a binary classifier to predict whether the target side of a pair is original text in that language or translated from the source language. This follows several prior works attempting to identify translations (Kurokawa et al., 2009; Koppel and Ordan, 2011; Lembersky et al., 2012).
To train the classifier, we need target-language text annotated by whether it is original or translated. We use News Crawl data from WMT $^2$  as target- original data. It consists of news articles crawled from the internet, so we assume that most of them are not translations. Getting translated data is trickier; most human-translated pairs where the original language is annotated are only present in test sets, which are generally small. To sidestep this, we choose to use machine translation as a proxy for human translationese, based on the assumption that they are similar. This allows us to create classifier training data using only unannotated monolingual data. We propose two ways of doing this: using forward translation (FT) or round-trip translation (RTT). Both are illustrated in Figure 2.
To generate FT data, we take source-language News Crawl data and translate it into the target language using a machine translation model trained on WMT training bitext. We can then train a classifier to distinguish the generated text from monolingual target-language text.
One potential problem with the FT data set is that the original and translated pairs may differ not only
<sup>1</sup>Europarl (Koehn, 2005) is a notable exception, but it is somewhat small and not in the news domain.
$^{2}$ http://www.statmt.org/wmt18/translation-task.html
7738
Figure 2: Illustration of data set creation for the FT and RTT translationese classifiers. The Source  $\rightarrow$  Target and Target  $\rightarrow$  Source nodes represent NMT systems.
in the respects we care about (i.e. translationese), but also in content. Taking English  $\rightarrow$  French as an example language pair, one could imagine that certain topics are more commonly reported on in original English language news than in French, and vice versa, e.g. news about American or French politics, respectively. The words and phrases representing those topics could then act as signals to the classifier to distinguish the original language.
To address this, we also experiment with RTT data. For this approach we take target-language monolingual data and round-trip translate it with two machine translation models (target→source and then source→target), resulting in another target-language sentence that should contain the same content as the original sentence, alleviating the concern with FT data. Here we hope that the noise introduced by round-trip translation will be similar enough to human translationese to be useful for our downstream task.
In both settings, we use the trained binary classifier to detect and tag training bitext pairs where the classifier predicted that the target side is original.
3 Experimental Set-up
We perform our experiments on WMT18 English  $\rightarrow$  German bitext and WMT15 English  $\rightarrow$  French bitext. We use WMT News Crawl for monolingual data (2007-2017 for German and 2007-2014 for French). We filter out sentences longer than 250 subwords (see Section 3.2 for the vocabulary used) and remove pairs whose length ratio is greater than 2. This results in about 5M pairs for English  $\rightarrow$  German. We do not filter the English  $\rightarrow$  French bitext, resulting in 41M sentence pairs.
For monolingual data, we deduplicate and filter sentences with more than 70 tokens or 500 characters. For the experiments described later in Section 5.3, this monolingual data is back-translated with a target-to-source translation model; after doing so, we remove any sentence pairs where the back-translated source is longer than 75 tokens or 550 characters. This results in 216.5M sentences for English  $\rightarrow$  German (of which we only use 24M at a time) and 39M for English  $\rightarrow$  French. As a final step, we use an in-house language identification tool based on the publicly-available Compact Language Detector  $2^{3}$  to remove all pairs with the incorrect source or target language. This was motivated by observing that some training pairs had the incorrect language on one side, including cases where both sides were the same; Khayrallah and Koehn (2018) found that this type of noise is especially harmful to neural models.
The classifiers were trained on the target language monolingual data in addition to either an equal amount of source language monolingual data machine-translated into the target language (for the FT classifiers) or the same target sentences round-trip translated through the source language with MT (for the RTT classifiers). In both cases, the MT models were trained only with WMT bitext.
The models used to generate the synthetic data have BLEU (Papineni et al., 2002) performance as follows on newestest2014/full: German  $\rightarrow$  English 31.8; English  $\rightarrow$  German 28.5; French  $\rightarrow$  English 39.2; English  $\rightarrow$  French 40.6. Here and elsewhere, we report BLEU scores with SacreBLEU (Post, 2018); see Section 3.3.
Both language pairs considered in this work are high-resource. While translationese is a potential concern for all language pairs, in low-resource settings it is overshadowed by general quality concerns stemming from the lack of training data. We leave for future work the application of these techniques to low-resource language pairs.
3.2 Architecture and Training
Our NMT models use the transformer-big architecture (Vaswani et al., 2017) implemented in lingvo (Shen et al., 2019) with a shared source-target byte-pair-encoding (BPE) vocabulary (Sennrich et al., 2016b) of  $32\mathrm{k}$  types. To stabilize training, we use exponentially weighted moving average (EMA) decay (Buduma and Locascio, 2017).
<sup>3</sup>https://github.com/CLD2Owners/cld2
7739
Language	Classifier Type	Bitext % Orig.	BT % Orig.
French	FT	47%	84%
RTT	30%	68%
German	FT	22%*	82%
RTT	29%*	70%
Checkpoints were picked by best dev BLEU on a set consisting of a tagged and untagged version of every input.
For the translationese classifier, we trained a three-layer CNN-based classifier optimized with Adagrad. We picked checkpoints by F1 on the development set, which was newstest2015 for English  $\rightarrow$  German and a subset of newstest2013 containing 500 English-original and 500 French- original sentence pairs for English  $\rightarrow$  French. We found that the choice of architecture (RNN/CNN) and hyperparameters did not make a substantial difference in classifier accuracy.
We report BLEU (Papineni et al., 2002) scores with SacreBLEU (Post, 2018) and include the identification string<sup>4</sup> to facilitate comparison with future work. We also run human evaluations for the best performing systems (Section 4.3).
4.1 Classifier Accuracy
Before evaluating the usefulness of our translationese classifiers for the downstream task of machine translation, we can first evaluate how accurate they are at distinguishing original text from human translations. We use WMT test sets for this evaluation, because they consist of source-original and target-original sentence pairs in equal number.
For French, the FT classifier scored 0.81 F1 and the RTT classifier scored 0.68 on newstest2014/full. For German, the FT classifier achieved 0.85 F1 and the RTT classifier scored 0.65 on newstest2015. We note that while the FT classifiers perform reasonably well, the RTT classifiers are less effective. This result is in line with prior work by
Table 1: Percentage of training data where the target side was classified as original. English  $\rightarrow$  German pairs with predicted original German (marked with a *) were upsampled to balance both bitext subsets' sizes.
Test set →	Src-Orig	Trg-Orig	Both
Decode →	Nt.	Tr.	Tr.	Nt.	Match
Match? →	X	✓	X	✓	✓
a. En→Fr: Avg. newstest20{14/full,15}
Untagged	39.5	39.5	44.5	44.5	42.0
FT clf.	37.7	40.0	42.5	45.0	42.5
RTT clf.	38.0	39.4	43.2	44.1	41.8
b. En→De: Avg. newstest20{14/full,16,17,18}
Untagged	36.3	36.3	30.0	30.0	34.0
FT clf.	28.3	36.0	29.4	29.8	33.6
RTT clf.	32.3	36.2	30.0	30.2	33.9
Table 2: Average BLEU for models trained on (a) WMT 2014 English  $\rightarrow$  French bitext and (b) WMT 2018 English  $\rightarrow$  German bitext, tagged according to target side classifier predictions. The tag controls the output domain: translationese ("Tr") or original/natural text ("Nt"). Matching output and test domains ("Match?") row) for both halves ("Both" column) achieves the highest combined BLEU.
Kurokawa et al. (2009), who trained an SVM classifier on French sentences to detect translations from English. They used word n-gram features for their classifier and achieved 0.77 F1, but were worried about a potential content effect and so also trained a classifier where nouns and verbs were replaced with corresponding part-of-speech (POS) tags, achieving 0.69 F1. Note that they tested on the Canadian Hansard corpus (containing Canadian parliamentary transcripts in English and French) while we tested on WMT test sets, so the numbers are not directly comparable, but it is interesting to see the similar trends in comparing content-aware and content-unaware versions of the same method. We also point out that Kurokawa et al. (2009) both trained and tested with human-translated sentences, while we trained our classifiers with machine-translated sentences while still testing on human-translated data.
The portion of our data classified as target- original by each classifier is reported in Table 1.
4.2 NMT with Translationese-Classified Bitext
Table 2a shows the BLEU scores of three models all trained on WMT 2014 English→French bitext. They differ in how the data was partitioned: either it wasn't, or tags were applied to those sentence pairs with a target side that a classifier predicted to be original French. We first note that the model trained on data tagged by the round-trip translation
4BLEU + case.mixed + lang.LANGUAGEPAIR + numrefs.1 + smooth.exp + test.SET + tok.intl + version.1.2.15
Test set → Tagging ↓	Src-Orig	Test set → Tagging ↓	Src-Orig
Decode	BLEU	% Preferred	Decode	BLEU	% Preferred
Untagged	-	43.9	26.6%	FT clf.	Transl.	44.6	24.2%
FT clf.	Natural	41.5	31.9%	FT clf.	Natural	41.5	30.7%
Table 3: Fluency side-by-side human evaluation for WMT English  $\rightarrow$  French newstest2014/full (Table 2a). We evaluate only the source- original half of the test set because it corresponds to our goal of original  $\rightarrow$  original translation. Despite a BLEU drop, humans rate the natural decode on average as more fluent than both the bitext model output and the same model with the translationese decode.
(RTT) classifier performs slightly worse than the baseline. However, the model trained with data tagged by the forward translation (FT) classifier is able to achieve an improvement of 0.5 BLEU on both halves of the test set when biased toward translationese on the source-original half and original text on the target-original half. This, coupled with the observation that the BLEU score on the source-original half sharply drops when adding the tag, indicates that the two halves of the test set represent quite different tasks, and that the model has learned to associate the tag with some aspects specific to generating original text as opposed to translationese.
However, we were not able to replicate this positive result on the English  $\rightarrow$  German language pair (Table 2b). Interestingly, in this scenario the relative ordering of the FT and RTT models is reversed, with the German RTT-trained model outperforming the FT-trained one. This is also interesting because the German FT classifier achieved a higher F1 score than the French one, indicating that a classifier's performance alone is not a sufficient indicator of its effect on translation performance. One possible explanation for the negative result is that the English  $\rightarrow$  German bitext only contains 5M pairs, as opposed to the 41M for English  $\rightarrow$  French, so splitting the data into two portions could make it difficult to learn both portions' output distributions properly.
4.3 Human Evaluation Experiments
In the previous subsection, we saw that BLEU for the source-original half of the test set went down when the model trained with FT classifications (FT clf.) was decoded it as if it were target-original (Table 2a). Prior work has shown that BLEU has a low correlation with human judgments when the reference contains translationese but the system output is biased toward original/natural text (Freitag et al., 2019). This is the very situation we find ourselves in now. Consequently, we run a human evaluation to see if the output truly is more natu
ral and thereby preferred by human raters, despite the loss in BLEU. We run both a fluency and an adequacy evaluation for English  $\rightarrow$  French to compare the quality of this system when decoding as if source-original vs. target-original. We also compare the system with the Untagged baseline. All evaluations are conducted with bilingual speakers whose native language is French, and each is rated by 3 different raters, with the average taken as the final score. Our two evaluations are as follows:
- Adequacy: Raters were shown only the source sentence and the model output. Each output was scored on a 6-point scale.
- Fluency: Raters saw two target sentences (two models' outputs) without the source sentence, and were asked to select which was more fluent, or whether they were equally good.
Fluency human evaluation results are shown in Table 3. We measured inter-rater agreement using Fleiss' Kappa (Fleiss, 1971), which attains a maximum value of 1 when raters always agree. This value was 0.24 for the comparison with the untagged baseline, and 0.16 for the comparison with the translationese decodes. The agreement levels are fairly low, indicating a large amount of subjectivity for this task. However, raters on average still indicated a preference for the  $FT$  clf. model's natural decodes. This provides evidence that they are more fluent than both the translationese decodes from the same model and the baseline untagged model, despite the drop in BLEU compared to each.
Adequacy human ratings are summarised in Table 4. Both decodes from the  $FT$  clf. model scored significantly better than the baseline. This is especially true of the natural decodes, demonstrating that the model does not suffer a loss in adequacy by generating more fluent output, and actually sees a significant gain. We hypothesize that splitting the data as we did here allowed the model to learn a sharper distribution for both portions, thereby increasing the quality of both decode types. Some
Test set → Tagging ↓	Src-Orig
Decode	BLEU	Adequacy
Untagged	-	43.9	4.51
FT clf.	Transl.	44.6	4.67*
FT clf.	Natural	41.5	4.72**
Table 4: Human evaluation of adequacy for WMT English  $\rightarrow$  French on the source- original half of newstest2014/full. Humans rated each output separately on a 6-point scale. As with fluency (Table 3), the natural decode scores the best, despite a BLEU loss. The single and double asterisks indicate that the adequacy value is significantly greater than the first row's value at significance level  $\alpha = 0.05$  and  $\alpha = 0.01$ , respectively, according to a one-tailed paired  $t$ -test. The difference between the second and third rows was not significant at  $\alpha = 0.1$ .
additional evidence for this is the fact that the  $FT$  clf. model's training loss was consistently lower than that of the baseline.
5 Supplemental Experiments
5.1 Measuring Translationese
Translationese tends to be simpler, more standardised and more explicit (Baker et al., 1993) compared to original text and can retain typical characteristics of the source language (Toury, 2012). Toral (2019) proposed metrics attempting to quantify the degree of translationese present in a translation. Following their work, we quantify lexical simplicity with two metrics: lexical variety and lexical density. We also calculate the length variety between the source sentence and the generated translations to measure interference from the source.
5.1.1 Lexical Variety
An output is simpler when it uses a lower number of unique tokens/words. By generating output closer to original target text, our hope is to increase lexical variety. Lexical variety is calculated as the type-token ratio (TTR):
$$
T T R = \frac {\text {n u m b e r o f t y p e s}}{\text {n u m b e r o f t o k e n s}} \tag {1}
$$
5.1.2 Lexical Density
Scarpa (2006) found that translationese tends to be lexically simpler and have a lower percentage of content words (adverbs, adjectives, nouns and verbs) than original written text. Lexical density is
$$
l e x \_ d e n s i t y = \frac {\text {n u m b e r o f c o n t e n t w o r d s}}{\text {n u m b e r o f t o t a l w o r d s}} \tag {2}
$$
5.1.3 Length Variety
Both MT and humans tend to avoid restructuring the source sentence and stick to sentence structures popular in the source language. This results in a translation with similar length to that of the source sentence. By measuring the length variety, we measure interference in the translation because its length is guided by the source sentence's structure. We compute the normalized absolute length difference at the sentence level and average the scores over the test set of source-target pairs  $(x,y)$ :
$$
l e n g t h v a r i e t y = \frac {\left| \left| x \right| - \left| y \right| \right|}{\left| x \right|} \tag {3}
$$
5.1.4 Results
Results for all three different translationese measurements are shown in Table 5.