SlowGuess/NAACL_2022_OCR huggingface dataset:
Jingwei Ni*
ucabjni@ucl.ac.uk
Zhijing Jin*
MPI & ETH Zürich
zjin@tue.mpg.de
Markus Freitag
Google Research freitag@google.com
Mrinmaya Sachan
ETH Zürich  
msachan@ethz.ch
Human-translated text displays distinct features from naturally written text in the same language. This phenomena, known as translationese, has been argued to confound the machine translation (MT) evaluation. Yet, we find that existing work on translationese neglects some important factors and the conclusions are mostly correlational but not causal. In this work, we collect CAUSALMT, a dataset where the MT training data are also labeled with the human translation directions. We inspect two additional critical factors, the train-test direction match (whether the human translation directions in the training and test sets are aligned), and data-model direction match (whether the model learns in the same direction as the human translation direction in the dataset). We show that these two factors have a large causal effect on the MT performance, in addition to the test-model direction mismatch highlighted by existing work on translationese. In light of our findings, we provide a set of suggestions for MT training and evaluation.
MT has long been concerned with the artifacts introduced by translationese, the human-translated text that is systematically different from naturally written text in the same language (Toury, 1980; Gellerstam, 1986; Toury, 1995; Baker, 1993; Baroni and Bernardini, 2006). For a translation system translating from language  $X$  to language  $Y$ , there can be two types of test data: sentences that originated in language  $X$  and are human-translated into language  $Y$  (denoted as  $X \xrightarrow{\mathrm{H}} Y$ ), and sentences that originated in language  $Y$  and human-translated into language  $X$  (denoted as  $X \xleftarrow{\mathrm{H}} Y$ ).<sup>2</sup>
Figure 1: Three different factors illustrate the impact of translationese on MT performance. Previous work mainly focuses on how translationese in the test set (Factor 1) inflates BLEU score and makes it favor some translation systems over others. Our work investigates the causal effects of the other two key factors, the train-test direction match (Factor 2; i.e., whether the training set and the test set share the same human translation direction), and data-model direction match (Factor 3; i.e., whether the dataset collection direction and model translation direction are the same).
The main concern raised by this distinction of the two sets is whether the reported performance on a mixed test set truly reflects the actual translation quality. Previous work in MT has shown that translationese is a confounder in evaluating translation quality (Lembersky et al., 2012; Toral et al., 2018; Läubli et al., 2018; Freitag et al., 2020).
Recent studies on causality have also brought to attention the importance of distinguishing the data-model direction match, namely whether the data collection direction is the same as or opposite to the model direction, also known as causal or anti-causal learning (Jin et al., 2021b; Veitch et al., 2021; Scholkopf et al., 2012). If the dataset is collected by human annotators who see the input  $X$  and produce an output  $Y$ , then learning an  $X$ -to-
<sup>1</sup>Our code and data are at https://github.com/EdisonNi-hku/CausalMT.
2Note that the scope of this work does not cover pivot translation through a third language, but we encourage exploration in future work.
Human Language Technologies, pages 5303 - 5320
$Y$  model is causal learning, and learning a  $Y$ -to- $X$  model is anticausal learning.
In this work, we study the artifacts in MT brought by translationese from the viewpoint of causality, specifically, the match between the data and model directions. We consider two factors of variation in MT: human translation direction (in both the training and the test set) and model translation direction. Thus, we formulate the effect of translationese in the test set as the test-model direction match problem, and causal/anticausal learning as the data-model direction match problem. Further, we identify the third important factor, the train-test direction match problem, namely, whether the training set and the test set are collected with the same human translation direction. We study the causal influences of these three factors on MT performance in Figure 1. Previous work has mainly studied the artifacts introduced by the test-model direction match on MT performance (Toral et al., 2018; Graham et al., 2020; Edunov et al., 2020), but little considers the other two factors, the train-test and data-model direction match (Kurokawa et al., 2009; Larkin et al., 2021). Moreover, most analyses are based on correlation but not causation (Graham et al., 2020).
In this paper, we use causal inference (Pearl, 2009; Peters et al., 2017) to re-investigate previous conclusions about the most studied relationship between the test-model direction match and MT performance. We calibrate the effect of test-model direction match by analyzing the causal effects of the two other factors, the train-test and data-model direction match. In our causal analyses, we use interventions to study the effect of train-test direction match, and covariate matching to study the effect of data-model direction match, by controlling for covariates such as sentence lengths and content.
We build CAUSALMT, a new dataset on five language pairs labeled with the human translation directions, and statistically verify that translationese tends to be simpler and more verbose, corroborating previous observations on translationese (Toury, 1980; Gellerstam, 1986; Toury, 1995; Baker, 1993). Then, we rigorously analyze CAUSALMT, leading to the following new insights and contributions:
C1. Previous work claims that translationese in the test set inflates MT model performance and thus suggests removing the translationese-to- original half of the test set (Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2020;
Barrault et al., 2019). Our work shows that the translationese-to-original half of the test set does not necessarily inflate MT performance in all cases. In some cases, it can even be more challenging than the other half, depending on the human translation direction in the training corpus. Hence, we suggest still reporting performance on both test sets, and also reporting the training data direction if available.
C2. Previous work (Burlot and Yvon, 2018) claims that back-translation (BT) (Sennrich et al., 2016) is usually more effective than self-training (ST) (He et al., 2019). Our work shows that BT is not necessarily better than ST in all cases. This result also depends on how the pseudo-parallel corpus aligns with the human translation direction in the test set. We suggest choosing BT or ST depending on this direction match.
C3. Previous work claims that BT's performance improvement is largely reflected on the translationese-to-original half of the test set, but the improvement is very small on the other half (Toral et al., 2018; Freitag et al., 2019). Our work shows that the improvement of BT can be larger on the other half of the test set as well, as long as the pseudo-parallel corpus aligns with the human translation direction in the test set.
C4. Our work shows that data-model direction match also has a large causal effect on the MT performance of up to 12.25 BLEU points after adjusting for other covariates using the backdoor adjustment (Pearl, 1995).
2 CAUSALMT Dataset
To investigate the effect of train-test direction match and data-model direction match, we need to collect translation data in different human translation directions.
To construct our CAUSALMT dataset consisting of a large number of translation pairs labeled with the human translation direction, we use the EuroparlExtract toolkit (Ustaszewski, 2019) to filter
3Most existing datasets do not distinguish the human translation direction for the training set (Kolias et al., 2014; Barrault et al., 2019). Some works train a classifier to identify the human translation direction (Kurokawa et al., 2009; Riley et al., 2020), but they are not our ideal choice since this classification may rely on the domain difference of the two directions (Rabi-
DeH→En	DeH→En	DeH→Fr	DeH→Fr	EnH→Fr	EnH→Fr	EnH→Es	EnH→Es	EsH→Fr	EsH→Fr
# Training Samples	248K	248K	220K	220K	203K	203K	93K	93K	92K	92K
# Words/Sample	22.4/25.5	22.9/23.9	22.6/28.7	25.4/30.4	24.5/28.9	27.5/30.5	24.0/25.7	31.6/31.9	32.4/36.5	27.9/30.5
# Sents/Sample	1.05/1.04	1.04/1.02	1.05/1.86	1.07/1.94	1.03/1.89	1.05/1.95	1.03/1.05	1.08/1.08	1.09/2.18	1.07/1.95
Passive Voice (%)	-12.90	-11.48	-/-	-/-	11.70/-	13.45/-	11.49/-	14.94/-	-/-	-/-
Vocab Size	119K/37K	113K/40K	108K/49K	106K/55K	40K/53K	38K/56K	29K/46K	26K/47K	47K/38K	48K/42K
Expansion Factor	en:de=1.13	en:de=1.04	fr:de=1.26	fr:de=1.19	fr:en=1.18	fr:en=1.10	es:en=1.06	es:en=1.01	fr:es=1.12	fr:es=1.09
Table 1: Detailed characteristics of the CAUSALMT dataset. We first report the number of translation pairs in the training set (# Training Samples), and for each parallel corpus  $(X\stackrel {\mathrm{H}}{\longrightarrow} / \stackrel {\mathrm{H}}{\leftarrow}Y)$ , we report the following statistics in both language  $X$  and  $Y$  (denoted as "stats in  $X/$  stats in  $Y$ ): the number of words per sample (# Words/Sample), number of sentences per sample (# Sents/Sample), percentage of samples with passive voice, vocabulary size, and the expansion factor. The expansion factor from language  $X$  to language  $Y$  ( $X:Y$ ) is calculated by the average word count per sample in language  $X$  divided by the average word count per sample in language  $Y$ .
translation pairs by meta-information (e.g., the tag specifying the original language of the speaker). Specifically, in the Europarl corpus (Koehn, 2005), we iterate over each transcript that has an origina-tion label and mark a sentence as original text if the original language of the speaker is the same as the language this sentence is in, or otherwise mark it as the translated text. After extracting the direction-labeled language pairs, we remove all duplicates in the entire dataset. Since our study needs to compare training on parallel corpora of the same language pair but with two different human translation directions, e.g.,  $\mathrm{De}\xrightarrow{\mathrm{H}}\mathrm{En}$  and  $\mathrm{De}\xleftarrow{\mathrm{H}}\mathrm{En}$ , we control the size of the two corpora to be the same by downsampling the larger set.
Among all language pairs we can obtain, we keep five language pairs with the largest number of data samples. As in Table 1, the CAUSALMT dataset contains over 200K translation pairs in each training set of three language pairs and over 90K translation pairs in each training set of the other two language pairs. The development set and test set contain 1K and 2K translation pairs for all language pairs in each direction, respectively.
2.2 Dataset Characteristics
We analyze the characteristics of the CAUSALMT dataset in light of how translated text differs from naturally written text in the same language.
Our findings echo the observations by previous work on the distinct features of translationese (Toury, 1980; Gellerstam, 1986; Toury, 1995; Baker, 1993; Baroni and Bernardini, 2006; Volansky et al., 2015). For example, translationese tends to be simpler and more standardized (Baker, 1993; Toury, 1995; Laviosa-Braithwaite, 1998), such as
novich and Wintner, 2015). Our dataset is an extended version of our prior study (Jin et al., 2021b), but ours is significantly larger to enable the various analyses in our study.
having a smaller vocabulary and using certain discourse markers more often (Baker, 1993, 1995, 1996). Translationese also tends to be influenced by the source language in terms of its lexical and word order choice (Gellerstam, 1986).
In the CAUSALMT data, we observe three properties. (1) Within each language pair (e.g., German and English), the same language's translationese always has a smaller vocabulary than its naturally written text corpus. For example, the translationese German in  $\mathrm{De}\overset{\mathrm{H}}{\leftarrow}\mathrm{En}$  has only 113K vocabulary, which is 5K smaller than the vocabulary of the German corpus in  $\mathrm{De}\overset{\mathrm{H}}{\rightarrow}\mathrm{En}$ . (2) Translationese tends to be more verbose. For each language pair, we calculate the expansion factor from language  $X$  to language  $Y$  ( $X:Y$ ) as the average word count per sample in language  $X$  divided by the average word count per sample in language  $Y$ . For example, for each (English, German) translation pair, the number of English words is 1.13 times that of German words when English is the translationese (i.e., en:de expansion factor=1.13). On the other hand, the en:de expansion factor is only 1.04 when English is the naturally written text. (3) We use a syntax-based parser to detect the percentage of samples with passive voice in English, details of which are in Appendix B. There is a clear distinction that translationese English tends to use more passive voice than original English, e.g., 14.94% translationese samples in the passive voice in the  $\mathrm{En}\overset{\mathrm{H}}{\leftarrow}\mathrm{Es}$  corpus in contrast with 11.49% original English samples in the reverse direction.
3 The Overshadowing Effect of Train-Test Direction Match
The first analysis of this paper aims to calibrate the most studied relationship of the test-model direction match and MT performance by considering the additional effect of the train-test direction match.
Previous work observes that the translationese-to- original test set inflates the score. To evaluate a model with the  $X$  -to-  $Y$  translation direction, traditionally, the test set is a mixture of two halves, one with the human translation direction  $X\stackrel {\mathrm{H}}{\rightarrow}Y$  (aligned) and the other  $X\stackrel {\mathrm{H}}{\leftarrow}Y$  (unaligned, or translationese-to- original) (Bojar et al., 2018).
Previous studies propose that the unaligned, translationese-to- original test set is easier to translate than the other aligned test set because translationese inputs are easy for the MT model to handle (Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2020). The inflated test performance caused by translationese has long been speculated (Lembersky et al., 2012; Toral et al., 2018; Läubli et al., 2018), and recent work has statistically verified the correlation (Graham et al., 2020).
With the previous understanding, some works suggest removing the unaligned half of the test set (Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2020), which was adopted by the 2019 WMT shared task (Barrault et al., 2019), whereas others suggest keeping both but reporting the performance separately (Freitag et al., 2019; Edunov et al., 2020). The motivations from the two sides are that in the unaligned half, although the source text being translationese is an easy input to the model, its target text being naturally written text makes the evaluation more natural.
This "inflation" depends on train-test direction match. We take a step back from the argument on whether the unaligned test set positively or negatively affects the MT performance evaluation. Instead, we call attention to the fact that, beyond the test-model direction match, there can be other factors also playing a critical in the MT performance evaluation, i.e., the train-test direction match.
For a given machine translation task to learn the  $X$ -to- $Y$  translation, there can be two questions: the question from previous work is whether we should use the test set aligned with the model translation direction (T1) or the test set unaligned with the model translation direction (T2) to evaluate the model fairly, whereas the question answered by our work is which training data should be used to achieve the best performance.
Our analysis aims to obtain causal conclusions on how intervening on the train-test direction match affects the MT performance. Therefore, we control all other possible confounders. For each language pair, we control the total training data size
De-to-En Translation	En-to-De Translation
α% T1 (de, en*)	T2 (de*, en)	α% T1 (en, de*)	T2 (en*, de)
0%	24.68	35.86	0%	21.24	26.27
25%	28.98	35.40	25%	25.60	25.44
50%	30.86	34.53	50%	27.29	24.70
75%	31.52	31.92	75%	27.82	23.23
100%	31.33	27.07	100%	28.94	20.32
De-to-Fr Translation	Fr-to-De Translation
α% T1 (de, fr*)	T2 (de*, fr)	α% T1 (fr, de*)	T2 (fr*, de)
0%	24.37	36.44	0%	18.85	22.62
25%	28.60	36.21	25%	24.30	22.88
50%	28.87	34.06	50%	25.91	22.10
75%	30.11	32.42	75%	27.41	20.94
100%	30.45	27.65	100%	27.79	18.68
En-to-Fr Translation	Fr-to-En Translation
α% T1 (en, fr*)	T2 (en*, fr)	α% T1 (fr, en*)	T2 (fr*, en)
0%	31.74	38.09	0%	31.91	40.74
25%	36.64	37.84	25%	35.94	38.69
50%	38.00	36.83	50%	37.36	37.51
75%	39.00	36.10	75%	39.11	36.61
100%	39.74	33.88	100%	40.27	33.01
En-to-Es Translation	Es-to-En Translation
α% T1 (en, es*)	T2 (en*, es)	α% T1 (es, en*)	T2 (es*, en)
0%	31.74	38.09	0%	31.91	40.74
25%	36.64	37.84	25%	35.94	38.69
50%	38.00	36.83	50%	37.36
75%	39.00	36.10	75%	39.11	36.61
100%	39.74	33.88	100%	40.27	33.01
Es-to-Fr Translation	Fr-to-Es Translation
α% T1 (es, fr*)	T2 (es*, fr)	α% T1 (fr, es*)	T2 (fr*, es)
0%	37.32	46.25	0%	39.16	41.60
25%	40.60	46.43	25%	41.81	40.64
50%	41.94	45.57	50%	43.48	39.66
75%	42.39	43.88	75%	45.13	39.03
100%	42.46	40.00	100%	45.42	37.56
Table 2: BLEU points of all five language pairs on training sets mixed by  $\alpha \% X\stackrel {\mathrm{H}}{\longrightarrow}Y$  and  $(1 - \alpha \%)$ $X\stackrel {\mathrm{H}}{\leftarrow}Y$  data, where the mixture rate  $\alpha = 0,25,50,75,100$  We always use T1 to denote the test set aligned with the model direction, and T2 to denote the unaligned one. For readability, we use \* to denote the translationese language. For example,  $(\mathrm{de},\mathrm{en}^{*})$  means original German and translated English pairs.
to be the same<sup>4</sup> when varying the portion of data in two directions. We also enumerate all other possible interventions, such as varying the two model translation directions and reporting performance on two different halves of the test set with two human translation directions. We also control that all translation models use the same Transformer architecture (Vaswani et al., 2017) by faireq (Ott et al., 2019), with experimental details in Appendix C.
A side benefit of controlling the training data size is that our experiments can help answer what the best nature (i.e., human translation direction) of the training data given a fixed annotation or computation budget is. We leave the space for future work to increase the total training set size with all available training data in both directions.
We report the experiment results of how intervening the train-test direction match affects the MT performance in BLEU score (Papineni et al., 2002) in Table 2. The main takeaways are as follows:
(1) It is not always the case that, for the same model, the unaligned test set T2 yields higher/more inflated results than the aligned test set T1. When the training data has  $75 - 100\%$  aligned training samples, performance reported on T2 is, in most cases, no longer larger than that on the other half. With such training data, usually, T1 inflates the BLEU score more.
(2) The train-test direction match can have an overshadowing effect over the artifacts introduced by the translationese-to-original test set, since no matter which test set we use, the more matched the train and test directions are, the higher the performance reported on T1 than T2 is. Specifically, as we vary the portion of the aligned training data from 0 to  $100\%$ , the performance on T1 keeps increasing, and the performance on T2 keeps decreasing. Additionally, if the training data has about  $0 - 50\%$  samples aligned with the model translation direction, then, in many cases, T2 is higher than T1, which might explain the previous observations that T2 inflates the BLEU score (Toral et al., 2018; Graham et al., 2020). To account for another possible interpretation, such as the domain shift between the training and test sets, we also conduct an additional evaluation using the newstest2014 test sets, which do not share any domain similarity with our training sets, but still support our observation (in Appendix Table 6).
Hence, the two constructive suggestions for future work are that (1) it is important to still report on both test sets, and also the training data direction if available, and (2) parallel training data in the same direction with the test set is more helpful.
Monolingual data in the original language of the test set is more helpful. With the intuition that the train-test direction match is a crucial factor for MT performance, we also look into its implications on semi-supervised learning.
Given additional monolingual data, a common question in MT is what type of monolingual data to use, and the accompanying question, whether to use self-training (ST) for the source language monolingual corpus (He et al., 2019; Yarowsky, 1995) or back-translation (BT) for the target language monolingual corpus (Bojar and Tamchyna, 2011; Sennrich et al., 2016; Poncelas et al., 2018).
English-to-French (en-to-fr) Translation
Test 1 (en, fr*)	Test 2 (en*, fr)
Sup. on Equal Mix	16.16	16.65
+ ST (en, fr**)	+2.04 (Aligned)	+1.74
+ BT (en**, fr)	+1.91	+2.45 (Aligned)
French-to-English (fr-to-en) Translation
Test 1 (fr, en*)	Test 2 (fr*, en)
Sup. on Equal Mix	18.39	15.09
+ ST (fr, en**)	+2.64 (Aligned)	+2.24
+ BT (fr**, en)	+2.17	+3.26 (Aligned)
English-to-German (en-to-de) Translation
Test 1 (en, de*)	Test 2 (en*, de)
Sup. on Equal Mix	10.59	8.80
+ ST (en, de**)	+1.92 (Aligned)	+1.60
+ BT (en**, de)	+1.86	+2.25 (Aligned)
German-to-English (de-to-en) Translation
Test 1 (de, en*)	Test 2 (de*, en)
Sup. on Equal Mix	11.99	13.46
+ ST (de, en**)	+2.28 (Aligned)	+1.25
+ BT (de**, en)	+1.99	+3.72 (Aligned)
Table 3: Performance on the en-fr and en-de test sets of newtest2014. There are two test sets for each task, where * marks the translated language. We use an equal mixture of supervised data in two human translation directions ("Sup. on Equal Mix"). Both ST and BT generate pseudo-parallel data (marked by **), with which we find that aligned directions between the test set and the pseudo-parallel data lead to larger performance gain.
We reframe the question as "with abundant monolingual data from both languages, but limited computation resources, which data (together with the corresponding semi-supervised learning method) should we choose?"
In previous work, BT is the most widely used technique (Bojar et al., 2018; Edunov et al., 2018; Ng et al., 2019; Barrault et al., 2019, p. 15), and is reported to outperform ST (Burlot and Yvon, 2018). Another line of previous work inspects the performance gain by BT. Some argue that BT is helpful mostly on the test set aligned with the model (Toral et al., 2018; Freitag et al., 2019; Edunov et al., 2020, Appendix A Table 7) but not the unaligned test set, while others show that BT improves performance on both test sets (Edunov et al., 2020).
We re-inspect the two previous lines of work, and find (1) BT does not always outperform ST, especially when ST can make use of the monolingual data in the original language of the test set (to produce pseudo-aligned training data), and (2) the performance gain by BT is not always larger on the unaligned test set, but depends on the model direction, especially when BT generates pseudo-aligned training data with the test set.
We implement BT by Edunov et al. (2020), and ST by He et al. (2019). To fairly compare the per
formance of ST vs. BT, for each language pair  $X$  and  $Y$ , we split half both training corpora into  $X \xrightarrow{\mathrm{H}} Y$ -Half1,  $X \xrightarrow{\mathrm{H}} Y$ -Half2,  $Y \xrightarrow{\mathrm{H}} X$ -Half1, and  $Y \xrightarrow{\mathrm{H}} X$ -Half2. We construct the supervised training data as an equal mix (i.e.,  $\alpha = 50$ ) combining  $X \xrightarrow{\mathrm{H}} Y$ -Half1 and  $Y \xrightarrow{\mathrm{H}} X$ -Half1. The development data is the combination of both development sets, which is also an equal mix.