Venue: NAACL 2022

_Original_ or _Translated_? A Causal Analysis of the Impact of Translationese on Machine Translation Performance

Jingwei Ni1University College Londonucabjni@ucl.ac.uk

Zhijing Jin2MPI & ETH Zurichzjin@tue.mpg.deMarkus FreitagGoogle Researchfreitag@google.com

Footnote 1: Equal contributions.

###### Abstract

Human-translated text displays distinct features from naturally written text in the same language. This phenomena, known as _translationes_, has been argued to confound the machine translation (MT) evaluation. Yet, we find that existing work on transitiones neglects some important factors and the conclusions are mostly correlational but not causal. In this work, we collect CausalMT, a dataset where the MT training data are also labeled with the human translation directions. We inspect two additional critical factors, the _traintest direction match_ (whether the human translation directions in the training and test sets are aligned), and _data-model direction match_ (whether the model learns in the same direction as the human translation direction in the dataset). We show that these two factors have a large causal effect on the MT performance, in addition to the _test-model direction mismatch_ highlighted by existing work on translationese. In light of our findings, we provide a set of suggestions for MT training and evaluation.3

Footnote 3: Our code and data are at [https://github.com/EdisONi-hku/CausalMT](https://github.com/EdisONi-hku/CausalMT).

+
Footnote †: Equal contributions.

+
Footnote †: Note that the scope of this work does not cover pivot translation through a third language, but we encourage exploration in future work.

+
Footnote †: Note that the scope of this work does not cover pivot translation through a third language, but we encourage exploration in future work.

## 1 Introduction

MT has long been concerned with the artifacts introduced by _translationese_, the human-translated text that is systematically different from naturally written text in the same language [54, 68, 70, 9]. For a translation system translating from language \(X\) to language \(Y\), there can be two types of test data: sentences that originated in language \(X\) and are human-translated into language \(Y\) (denoted as \(X\smash{\stackrel{{\text{H}}}{{\rightarrow}}}Y\)), and sentences that originated in language \(Y\) and human-translated into language \(X\) (denoted as \(X\smash{\stackrel{{\text{H}}}{{\leftarrow}}}Y\)).4 The main concern raised by this distinction of the two sets is whether the reported performance on a mixed test set truly reflects the actual translation quality. Previous work in MT has shown that translationese is a confounder in evaluating translation quality [11, 12, 13, 14, 15].

Footnote 4: Note that the scope of this work does not cover pivot translation through a third language, but we encourage exploration in future work.

Recent studies on causality have also brought to attention the importance of distinguishing the _data-model direction match_, namely whether the data collection direction is the same as or opposite to the model direction, also known as _causal_ or _anti-causal_ learning [16, 17, 18]. If the dataset is collected by human annotators who see the input \(X\) and produce an output \(Y\), then learning an \(X\)-to

Figure 1: Three different factors illustrate the impact of translationese on MT performance. Previous work mainly focuses on how translatiocese in the test set (Factor 1) inflates BLEU score and makes it favor some translation systems over others. Our work investigates the causal effects of the other two key factors, the traintest direction match (Factor 2; i.e., whether the training set and the test set share the same human translation direction), and data-model direction match (Factor 3; i.e., whether the dataset collection direction and model translation direction are the same).

 

\(Y\) model is causal learning, and learning a \(Y\)-to-\(X\) model is anticausal learning.

In this work, we study the artifacts in MT brought by translationese from the viewpoint of causality, specifically, the match between the data and model directions. We consider two factors of variation in MT: _human translation direction_ (in both the training and the test set) and _model translation direction_. Thus, we formulate the effect of translationese in the test set as the _test-model direction match_ problem, and causal/anticausal learning as the _data-model direction match_ problem. Further, we identify the third important factor, the _train-test direction match_ problem, namely, whether the training set and the test set are collected with the same human translation direction. We study the causal influences of these three factors on MT performance in Figure 1. Previous work has mainly studied the artifacts introduced by the test-model direction match on MT performance [23, 45], but little considers the other two factors, the train-test and data-model direction [76, 80]. Moreover, most analyses are based on correlation but not causation [45].

In this paper, we use causal inference [9, 10] to re-investigate previous conclusions about the most studied relationship between the test-model direction match and MT performance. We calibrate the effect of test-model direction match by analyzing the causal effects of the two other factors, the train-test and data-model direction match. In our causal analyses, we use interventions to study the effect of train-test direction match, and covariate matching to study the effect of data-model direction match, by controlling for covariates such as sentence lengths and content.

We build CausalMT, a new dataset on five language pairs labeled with the human translation directions, and statistically verify that translationese tends to be simpler and more verbose, corroborating previous observations on translationese [23, 10, 109, 11]. Then, we rigorously analyze CausalMT, leading to the following new insights and contributions:

* Previous work claims that translationese in the test set inflates MT model performance and thus suggests removing the translationalese-to-original half of the test set [23, 45, 45, 46]. Our work shows that the translationese-to-original half of the test set does not necessarily inflate MT performance in all cases. In some cases, it can even be more challenging than the other half, depending on the human translation direction in the training corpus. Hence, we suggest still reporting performance on both test sets, and also reporting the training data direction if available.
* Previous work [23] claims that back-translation (BT) [23] is usually more effective than self-training (ST) [22]. Our work shows that BT is not necessarily better than ST in all cases. This result also depends on how the pseudo-parallel corpus aligns with the human translation direction in the test set. We suggest choosing BT or ST depending on this direction match.
* Previous work claims that BT's performance improvement is largely reflected on the translationese-to-original half of the test set, but the improvement is very small on the other half [23, 10]. Our work shows that the improvement of BT can be larger on the other half of the test set as well, as long as the pseudo-parallel corpus aligns with the human translation direction in the test set.
* Our work shows that data-model direction match also has a large causal effect on the MT performance of up to 12.25 BLEU points after adjusting for other covariates using the backdoor adjustment [9].

## 2 CausalMT Dataset

To investigate the effect of train-test direction match and data-model direction match, we need to collect translation data in different human translation directions.

### Data Collection

To construct our CausalMT dataset consisting of a large number of translation pairs labeled with the human translation direction,3 we use the EuroparlExtract toolkit [10] to filter 

translation pairs by meta-information (e.g., the tag specifying the original language of the speaker). Specifically, in the Europarl corpus [14], we iterate over each transcript that has an origination label and mark a sentence as original text if the original language of the speaker is the same as the language this sentence is in, or otherwise mark it as the translated text. After extracting the direction-labeled language pairs, we remove all duplicates in the entire dataset. Since our study needs to compare training on parallel corpora of the same language pair but with two different human translation directions, e.g., De\({}^{\rm H}\)En and De\({}^{\rm E}\)En, we control the size of the two corpora to be the same by downsampling the larger set.

Footnote 2: [https://github.com/hugging-in-the-language-learning-of-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-the-language-with-the-the-language-with-the-the-language-with-the-the-language-with-the-the-language-with-the-the-the-language-with-the](https://github.com/hugging-in-the-language-learning-of-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-language-with-the-the-language-with-the-the-language-with-the-the-language-with-the-the-language-with-the-the-language-with-the-the-the-language-with-the) 

Previous work observes that the translationese-to-original test set inflates the score.To evaluate a model with the \(X\)-to-\(Y\) translation direction, traditionally, the test set is a mixture of two halves, one with the human translation direction \(X{\stackrel{{\mathrm{H}}}{{\rightarrow}}}Y\) (aligned) and the other \(X{\stackrel{{\mathrm{H}}}{{\leftarrow}}}Y\) (unaligned, or translationese-to-original) Bojar et al. (2018).

Previous studies propose that the unaligned, translationese-to-original test set is easier to translate than the other aligned test set because translationese inputs are easy for the MT model to handle Toral et al. (2018); Zhang and Toral (2019); Graham et al. (2020). The inflated test performance caused by translationese has long been speculated Lembersky et al. (2012); Toral et al. (2018); Laubli et al. (2018), and recent work has statistically verified the correlation Graham et al. (2020).

With the previous understanding, some works suggest removing the unaligned half of the test Toral et al. (2018); Zhang and Toral (2019); Graham et al. (2020), which was adopted by the 2019 WMT shared task Barrault et al. (2019), whereas others suggest keeping both but reporting the performance separately Freitag et al. (2019); Edunov et al. (2020). The motivations from the two sides are that in the unaligned half, although the source text being translationese is an easy input to the model, its target text being naturally written text makes the evaluation more natural.

**This "inflation" depends on train-test direction match.** We take a step back from the argument on whether the unaligned test set positively or negatively affects the MT performance evaluation. Instead, we call attention to the fact that, beyond the test-model direction match, there can be other factors also playing a critical in the MT performance evaluation, i.e., the train-test direction match.

For a given machine translation task to learn the \(X\)-to-\(Y\) translation, there can be two questions: the question from previous work is whether we should use the test set aligned with the model translation direction (T1) or the test set unaligned with the model translation direction (T2) to evaluate the model fairly, whereas the question answered by our work is _which training data should be used to achieve the best performance_.

Our analysis aims to obtain causal conclusions on how intervening on the train-test direction match affects the MT performance. Therefore, we control all other possible confounders. For each language pair, we control the total training data size to be the same4 when varying the portion of data in two directions. We also enumerate all other possible interventions, such as varying the two model translation directions and reporting performance on two different halves of the test set with two human translation directions. We also control that all translation models use the same Transformer architecture Vaswani et al. (2017) by fairseq Ott et al. (2019), with experimental details in Appendix C.

Footnote 4: A side benefit of controlling the training data size is that our experiments can help answer what the best nature (i.e., human translation direction) of the training data given a fixed annotation or computation budget is. We leave the space for future work to increase the total training set size with all available training data in both directions.

\begin{table}
\begin{tabular}{c c|c|c|c|c} \hline \multicolumn{3}{c|}{**De-to-En Translation**} & \multicolumn{3}{c}{**En-to-De Translation**} \\ \(\alpha\%\) & T1 (de, en*)  & \(\alpha\%\) & T1 (f, de*)  & t2 (en*, ee)  & t2 (en*, ee) [FOOTNOTE:]Footnote *: footnotemark: [ENDFOOTNOTE] \\ \hline
0\% & 24.68 & 35.86 & 0\% & 21.24 & 26.27 \\
25\% & 28.98 & 35.40 & 25.56 & 25.44 \\

[MISSING_PAGE_POST]

 \hline \end{tabular}
\end{table}
Table 2: BLEU points of all five language pairs on training sets mixed by \(\alpha\%\)\(X{\stackrel{{\mathrm{H}}}{{\rightarrow}}}Y\) and \((1-\alpha\%)\)\(X{\stackrel{{\mathrm{H}}}{{\leftarrow}}}Y\) data, where the mixture rate \(\alpha=0,25,50,75,100\). We always use T1 to denote the test set aligned with the model direction, and T2 to denote the unaligned one. For readability, we use \({}^{*}\) to denote the translationese language. For example, “(de, en*)” means original German and translated English pairs.

 

We report the experiment results of how intervening the train-test direction match effects the MT performance in BLEU score Papineni et al. (2002) in Table 2. The main takeaways are as follows:

(1) It is not always the case that, for the same model, the unaligned test set T2 yields higher/more inflated results than the aligned test set T1. When the training data has 75-100% aligned training samples, performance reported on T2 is, in most cases, no longer larger than that on the other half. With such training data, usually, T1 inflates the BLEU score more.

(2) The train-test direction match can have an overshadowing effect over the artifacts introduced by the translationese-to-original test set, since no matter which test set we use, the more matched the train and test directions are, the higher the performance reported on T1 than T2 is. Specifically, as we vary the portion of the aligned training data from 0 to 100%, the performance on T1 keeps increasing, and the performance on T2 keeps decreasing. Additionally, if the training data has about 0-50% samples aligned with the model translation direction, then, in many cases, T2 is higher than T1, which might explain the previous observations that T2 inflates the BLEU score Toral et al. (2018); Graham et al. (2020). To account for another possible interpretation, such as the domain shift between the training and test sets, we also conduct an additional evaluation using the newstest2014 test sets, which do not share any domain similarity with our training sets, but still support our observation (in Appendix Table 6).

Hence, the two constructive suggestions for future work are that (1) it is important to still report on both test sets, and also the training data direction if available, and (2) parallel training data in the same direction with the test set is more helpful.

**Monolingual data in the original language of the test set is more helpful.** With the intuition that the train-test direction match is a crucial factor for MT performance, we also look into its implications on semi-supervised learning.

Given additional monolingual data, a common question in MT is what type of monolingual data to use, and the accompanying question, whether to use self-training (ST) for the source language monolingual corpus He et al. (2019); Yarowsky (1995) or back-translation (BT) for the target language monolingual corpus Bojar and Tamchyna (2011); Sennrich et al. (2016); Poncelas et al. (2018). We reframe the question as _"with abundant monolingual data from both languages, but limited computation resources, which data (together with the corresponding semi-supervised learning method) should we choose?"_

In previous work, BT is the most widely used technique Bojar et al. (2018); Edunov et al. (2018); Ng et al. (2019); Barrault et al. (2019), p. 15), and is reported to outperform ST Burlot and Yvon (2018). Another line of previous work inspects the performance gain by BT. Some argue that BT is helpful mostly on the test set aligned with the model Toral et al. (2018); Freitag et al. (2019); Edunov et al. (2020), Appendix A Table 7) but not the unaligned test set, while others show that BT improves performance on both test sets Edunov et al. (2020).

We re-inspect the two previous lines of work, and find (1) BT does _not always_ outperform ST, especially when ST can make use of the monolingual data in the original language of the test set (to produce pseudo-aligned training data), and (2) the performance gain by BT is _not always_ larger on the unaligned test set, but depends on the model direction, especially when BT generates pseudo-aligned training data with the test set.

We implement BT by Edunov et al. (2020), and ST by He et al. (2019). To fairly compare the per

\begin{table}
\begin{tabular}{l l|l} \hline \multicolumn{3}{c}{**English-to-French (en-to-fr) Translation**} \\  & Test (1 en, fr\({}^{*}\)) & Test 2 (en’, fr) \\ \hline Sup. on Equal Mix & 16.16 & 16.65 \\ + ST (en, fr\({}^{**}\)) & **+2.04 (Aligned)** & +1.74 \\ + BT (en\({}^{**}\), fr) & +1.91 & **+2.45 (Aligned)** \\ \hline \multicolumn{3}{c}{**French-to-English (fr-to-en) Translation**} \\  & Test (1 f, en\({}^{*}\)) & Test 2 (f\({}^{*}\), en) \\ \hline Sup. on Equal Mix & 18.39 & 15.09 \\ + ST (fr, en\({}^{**}\)) & **+2.64 (Aligned)** & +2.24 \\ + BT (fr\({}^{**}\), en) & +2.17 & **+3.26 (Aligned)** \\ \hline \multicolumn{3}{c}{**English-to-German (en-to-de) Translation**} \\  & Test 1 (en, de\({}^{*}\)) & Test 2 (en’, de) \\ \hline Sup. on Equal Mix & 10.59 & 8.80 \\ + ST (en, de\({}^{**}\)) & **+1.92 (Aligned)** & +1.60 \\ + BT (en\({}^{**}\), de) & +1.86 & **+2.25 (Aligned)** \\ \hline \multicolumn{3}{c}{**German-to-English (de-to-en) Translation**} \\  & Test 1 (de, en\({}^{*}\)) & Test 2 (de ’, en) \\ \hline Sup. on Equal Mix & 11.99 & 13.46 \\ + ST (de, en\({}^{**}\)) & **+2.28 (Aligned)** & +1.25 \\ + BT (de\({}^{**}\), en) & +1.99 & **+3.72 (Aligned)** \\ \hline \end{tabular}
\end{table}
Table 3: Performance on the en-fr and en-de test sets of _newstest2014_. There are two test sets for each task, where \({}^{*}\) marks the translated language. We use an equal mixture of supervised data in two human translation directions (“_Sup. on Equal Mix_”). Both ST and BT generate pseudo-parallel data (marked by \({}^{**}\)), with which we find that **aligned** directions between the test set and the pseudo-parallel data lead to larger performance gain.

 

formance of ST vs. BT, for each language pair \(X\) and \(Y\), we split half both training corpora into \(X{\stackrel{{\rm H}}{{\rightarrow}}}Y\)-Half1, \(X{\stackrel{{\rm H}}{{\rightarrow}}}Y\)-Half2, \(Y{\stackrel{{\rm H}}{{\rightarrow}}}X\)-Half1, and \(Y{\stackrel{{\rm H}}{{\rightarrow}}}X\)-Half2. We construct the supervised training data as an equal mix (i.e., \(\alpha\)=50) combining \(X{\stackrel{{\rm H}}{{\rightarrow}}}Y\)-Half1 and \(Y{\stackrel{{\rm H}}{{\rightarrow}}}X\)-Half1. The development data is the combination of both development sets, which is also an equal mix.

To train ST or BT, we use the second halves of the training data only as the monolingual corpora. For example, if the translation task is English-to-German translation, ST generates a pseudo-parallel corpus with original English paired with machine-translated pseudo-German, which we denote as (en, de**). For readability, we mark the machine-translation direction with ST and BT by ** and the human translation direction by *.

Footnote 4: [https://github.com/google-research/gid/](https://github.com/google-research/gid/)

Our hypothesis is that the machine-translated text pairs (en, de**) will also show similar properties as the human-translated training data (en, de*). Specifically, the more the pseudo-training data is aligned with the test set, the higher performance the semi-supervised learning method will achieve. This is confirmed by the experiment results in Table 3, where, across all settings, no matter which semi-supervised learning method is used, when the pseudo-training data has the same translation direction as the test set, the resulting performance is generally higher. The experiments conducted on CausalMT test sets also generally show the same trend, and, due to the space limit, we include the results in the Appendix Table 7.

## 4 Causal Effect of Data-Model Direction Match

The second contribution of this work is to inspect how much another factor, the data-model direction match, causally affects the MT performance. Formally, our research question is that, for a given translation task \(X\)-to-\(Y\), considering an equal mix of the test set, does the human translation direction of the training data still matter? If so, how large is the effect, and is it language-/task-dependent?

This section will use causal inference to isolate the effect of data-model direction match from other possible confounders and discuss its effect in different languages and translation tasks.

### Correlation in Previous Experiments

Our previous experiments show that data-model direction match _correlates_ with MT performance. Specifically, for each translation task in Table 2, there is a clear difference between the causal learning and anticausal learning model. We copy this naive difference to the "Diff" column of Table 5.

This naive difference represents \(\mathbb{E}_{\mathrm{aligned\_corpus}}[S]-\mathbb{E}_{\mathrm{ unaligned\_corpus}}[S]\), which compares the performance score \(S\) on the aligned corpus and \(S\) on the unaligned corpus, without controlling for potential confounders. The famous "correlation does not imply causation" implies that this formulation cannot answer the causal question, as the two expectation terms are taken over two different distributions that are not necessarily comparable. However, by Reichenbach's Common Cause Principle (Reichenbach, 1956), correlation implies the existence of some common cause behind the two correlated variables, which motivates us to investigate the causal relationship in the next section.

Setup of the Causal Effect Estimation Formulating the causal effect.Instead of just correlational analyses, we aim to estimate the average causal effect (ACE) Holland (1988); Pearl (2009) of the data-model direction match (i.e., causal vs. anticausal learning) \(M\) on the translation performance \(S\):5

Footnote 5: Note that there are two notions of causality here, one is the intervention we are interested in, namely the data-model direction match, known as causal vs. anticausal learning, and the other is the meta-level causality we are interested in, namely how much the data-model direction match (as a binary variable) causally affect the translation performance.

\[\begin{split}\mathrm{ACE}&=P(S=s|\mathrm{do}(M=1))\\ &-P(S=s|\mathrm{do}(M=0))\,,\end{split} \tag{1}\]

where, according to do-calculus Pearl (1995) in causal inference, the operator \(\mathrm{do}(M=0~{}\text{or}~{}1)\) means to intervene on the data-model direction match to be 0 (i.e., anticausal learning) or 1 (i.e., causal learning). The ACE formulation is about how much the model performance \(S\) will differ if intervening the data-model direction match \(M\) to be 0 or 1.

To estimate the ACE, we first draw the causal graph considering all variables that can interfere with the relationship between data-model direction match and MT performance. The main additional factors we need to control for are in the causal graph in Figure 2. We make the assumption that it is very likely that the two corpora of different human translation directions also vary by sentence 

lengths and the distribution of content (Bogoychev and Sennrich, 2019) due to a hidden confounder (i.e., a common cause) such as the nature of Europarl. Note that since our research question is about which training data matters for given a translation task, the data-model direction match is equivalent to the human translation direction of the training data, as the model translation direction is fixed.

Given the causal graph in Figure 2, the ACE in Eq. (1) can be calculated by conditioning on the set of variables \(Z\) which blocks the backdoor paths Pearl (1995) between \(M\) and \(S\). (\(Z\) fits the backdoor criterion Pearl (1993) in that the sentence lengths and content block all non-directed paths from \(M\) to \(S\), and neither is a descendant of any node on the directed path from \(M\) to \(S\).) An intuitive interpretation can be that when we directly look at the correlation between the data-model direction match and MT performance, it might also be due to that different corpora have different distributions of sentence lengths and content. Therefore, we need to control the sentence lengths and content so that the performance difference will be solely due to the data-model direction match.

Formally, the ACE using the do-notation can be calculated by conditioning on \(Z\). Specifically, we integrate over the distribution of \(P(Z)\), and calculate the difference in the conditional probability distribution \(P(S=s|M=1,Z={\mathbf{z}})-P(S=s|M=0,Z={\mathbf{z}})\) of \(S\) given the data-model direction match value \(M\) conditioned on the other key variables \(Z\) for each of its possible value \({\mathbf{z}}\), as shown in Eq. (2):

\[\begin{split}\mathrm{ACE}&=\int_{{\mathbf{z}}}[(P(S= s|M=1,Z={\mathbf{z}})\\ &-P(S=s|M=0,Z={\mathbf{z}}))P({\mathbf{z}})]\\ &=\mathbb{E}_{{\mathbf{z}}}[S|M=1,Z={\mathbf{z}}]\\ &-\mathbb{E}_{{\mathbf{z}}}[S|M=0,Z={\mathbf{z}}]~{}.\end{split} \tag{2}\]

Finally, we estimate it by comparing the expected values of the model performance score \(S\) given \(M=0\) or \(1\) over all possible values of the covariate \(Z\) as in Eq. (3).

**Causal effect estimation by matching.** To estimate the ACE in Eq. (3), the intuition is that we need to take care of the covariates in \(Z\) so that the aligned setting and the unaligned setting are comparable. We follow the covariate matching method in causal inference Rosenbaum and Rubin (1983); Iacus et al. (2012) and adjustment in the high-dimensional setting of text Roberts et al. (2020); Veitch et al. (2020). Specifically, matching is a method in causal inference to subsample the treated (i.e., the aligned corpus with the model direction) and control samples (i.e., the unaligned corpus with the model direction) so that the covariates of interest are matched.

We aim to match pairs of samples, one from the causal corpus and the other from the anticausal corpus, where we constrain them to share similar contents and similar sentence lengths. In our

Figure 3: Distribution of topics and sentence lengths after matching, using the German-English language pair (between De\(\stackrel{{\text{H}}}{{\longrightarrow}}\)En and De\(\stackrel{{\text{H}}}{{\longleftarrow}}\)En) as an example.

 

implementation, for each sentence in the causal corpus, we select its most similar match in the anticausal corpus using Dinic's maximal matching algorithm (Dinic, 1970).

Empirically, we limit the sentence length ratio of each matched pair to be no larger than 1.1 and the content to have a cosine similarity larger than 0.7, following the threshold to match a content-similar pseudo-parallel corpus in Jin et al. (2019). To calculate the content-wise similarity of a pair of samples, we represent each sentence by the sentence BERT embedding (Reimers and Gurevych, 2019). In cases of multiple languages as candidates to match the sentence embeddings in, we set a prioritization order of "En\(>\)De\(>\)Fr\(>\)Es" for sentence embedding matching. The prioritization order roughly follows the training data sizes of the multilingual version (Reimers and Gurevych, 2020) of the sentence transformer in the four languages. Note that since the set of factors to control is in a high-dimensional vector space, it is less realistic to use other common matching methods such as propensity score stratification and matching, as pointed out by Roberts et al. (2020).

We check the quality of our matching heuristics. Taking the German-English language pair as an example, we plot the distributions of topics by Latent Dirichlet Allocation (LDA) topic modeling (Blei et al., 2001) and distributions of sentence lengths across the De\(\stackrel{{\text{H}}}{{\rightarrow}}\)En and De\(\stackrel{{\text{H}}}{{\leftarrow}}\)En corpora in Figure 3. We also list some example matched pairs in English in Table 4. Further statistics of the matched corpora are in Appendix E.1.

Finally, based on the matched datasets that control for the sentence lengths and contents, ACE can be calculated by the average difference in MT performance of models trained on the two directions of the new datasets.

### Causal Effect Results

We have three observations from the results in Table 5: (1) The data-model direction match is a clear cause for MT performance. The ACE of data-model direction match on MT performance can be up to 12.25 BLEU points, for example, in the Spanish-to-English translation task. (2) The ACE varies by language and translation tasks. For the English-Spanish language pair, both translation directions get higher BLEU points if the models are trained in the causal learning direction. For other language pairs, the causal effects of data-model direction match are clear, although varying from positive to negative values, the reasons for which are worth future studies. (3) The results of naive differences (Diff) are, in most cases, smaller than that of the causal analysis by ACE. This indicates that the correlational analysis neglects other important factors such as the sentence length and content,

\begin{table}
\begin{tabular}{l r c l|r c c c c} \hline \multicolumn{6}{c|}{**English-to-German (en-to-de) Translation**} & \multicolumn{4}{c}{**German-to-English (de-to-en) Translation**} \\  & T1 (en, de\({}^{*}\)) & T2 (en\({}^{*}\), de) & **ACE** & Diff & & T1 (de, en\({}^{*}\)) & T2 (de\({}^{*}\), en) & **ACE** & Diff \\ \hline Cau. (en, de\({}^{*}\)) & 21.88 & 28.77 & \multirow{3}{*}{+3.13} & \multirow{3}{*}{+1.75} & Cau. (de, en\({}^{*}\)) & 31.70 & 28.68 & \multirow{3}{*}{**-1.89**} & \multirow{3}{*}{-2.14} \\ Ant. (en\({}^{*}\), de) & 25.33 & 22.19 & & Ant. (de\({}^{*}\), en) & 26.35 & 35.92 & & \\ \cline{1-1} \cline{5-10} \multicolumn{6}{c|}{**French-to-German (fr-to-de) Translation**} & \multicolumn{4}{c}{**German-to-French (de-to-fr) Translation**} \\  & T1 (fr, de\({}^{*}\)) & T2 (fr\({}^{*}\), de) & **ACE** & Diff & & T1 (de, fr\({}^{*}\)) & T2 (de\({}^{*}\), fr) & **ACE** & Diff \\ \hline Cau. (fr, de\({}^{*}\)) & 18.36 & 25.45 & \multirow{3}{*}{+5.57} & \multirow{3}{*}{+5.0} & Cau. (de, fr\({}^{*}\)) & 32.25 & 29.98 & \multirow{3}{*}{**-3.58**} & \multirow{3}{*}{-2.71} \\ Ant. (fr\({}^{*}\), de) & 20.46 & 11.78 & & Ant. (de\({}^{*}\), fr) & 28.07 & 37.74 & & \\ \cline{1-1} \cline{5-10} \multicolumn{6}{c|}{**French-to-English (fr-to-en) Translation**} & \multicolumn{4}{c}{**English-to-French (en-to-fr) Translation**} \\  & T1 (fr, en\({}^{*}\)) & T2 (fr\({}^{*}\), en) & **ACE** & Diff & & T1 (em, fr\({}^{*}\)) & T2 (en\({}^{*}\), fr) & **ACE** & Diff \\ \hline Cau. (fr, en\({}^{*}\)) & 33.77 & 37.42 & \multirow{3}{*}{+1.43} & \multirow{3}{*}{+2.53} & Cau. (en, fr\({}^{*}\)) & 42.60 & 37.34 & \multirow{3}{*}{-0.14} & \multirow{3}{*}{-0.65} \\ Ant. (fr\({}^{*}\), en) & 37.67 & 32.09 & & Ant. (en\({}^{*}\), fr) & 38.05 & 42.03 & & \\ \cline{1-1} \cline{5-10} \multicolumn{6}{c|}{**Spanish-to-English (es-to-en) Translation**} & \multicolumn{4}{c}{**English-to-Spanish (en-to-es) Translation**} \\  & T1 (es, en\({}^{*}\)) & T2 (es\({}^{*}\), en) & **ACE** & Diff & & T1 (en, es\({}^{*}\)) & T2 (en\({}^{*}\), es) & **ACE** & Diff \\ \hline Cau. (es, en\({}^{*}\)) & 37.79 & 33.64 & \multirow{3}{*}{+12.25} & \multirow{3}{*}{+0.63} & Cau. (en, es\({}^{*}\)) & 39.04 & 33.68 & \multirow{3}{*}{**+3.50**} & \multirow{3}{*}{+3.79} \\ Ant. (es\({}^{*}\), en) & 21.69 & 25.24 & & Ant. (en\({}^{*}\), es) & 30.76 & 34.96 & & \\ \cline{1-1} \cline{5-10} \multicolumn{6}{c|}{**French-to-Spanish (fr-to-es) Translation**} & \multicolumn{4}{c}{**Spanish-to-French (es-to-fr) Translation**} \\  & T1 (ff, es\({}^{*}\)) & T2 (fra\({}^{*}\), es) & **ACE** & Diff & & T1 (es, fr\({}^{*}\)) & T2 (es\({}^{*}\), fr) & **ACE** & Diff \\ \hline Cau. (fr, es\({}^{*}\)) & 37.09 & 43.40 & \multirow{3}{*}{+5.84} & \multirow{3}{*}{+2.22} & Cau. (es, fr\({}^{*}\)) & 41.67 & 41.57 & \multirow{3}{*}{41.62} & \multirow{3}{*}{**-2.74**} & \multirow{3}{*}{-1.11} \\ Ant. (fr\({}^{*}\), es) & 38.45 & 36.20 & & Ant. (es\({}^{*}\), fr) & 39.36 & 46.62 & & \\ \cline{1-1} \cline{5-10} \end{tabular}
\end{table}
Table 5: BLEU points of causal learning (Cau.) vs. anticausal (Ant.) directions after controlling for topics and lengths. We calculate the ACE by taking each model’s average performance on T1 and T2, and comparing how much causal models outperform anticausal models (**ACE**). In comparison, we show the naïve difference (Diff) by directly comparing the results of causal and anticausal models in Table 2 without topic and length control.

 

which might also be reflected in the overall difference. The causal analysis is a more appropriate method to isolate the influence of the data-model direction match.

## 5 Related Work

Linguistic studies have long observed the distinct properties of translationese from text originally authored in the same language Toury (1980); Gellerstam (1986); Baker (1993); Toury (1995). Recent work in MT identifies that the source side of the translationese-to-original portion of the test sets (i.e., test sets unaligned with the model direction) is easier Graham et al. (2020), echoing with many previous observations Toral et al. (2018); Lembersky et al. (2012); Laubli et al. (2018) and thus some suggest to exclude this portion from future test sets, in particular human evaluations Toral et al. (2018); Zhang and Toral (2019); Graham et al. (2020); Barrault et al. (2019). Nevertheless, Freitag et al. (2019) demonstrate that it is worth reporting automatic metric scores on both directions separately as both types of test sets evaluate different properties of translation quality. In follow-up work, Freitag et al. (2020) introduce paraphrased test sets that overcome the problems of translationese for test sets, not solving the problem for the training data though.

Based on this speculated inflation of MT performance due to translationese in the test set, further work inspects what previous conclusions about the effectiveness of MT models should be recalibrated. Some discover that models with BT mostly improve on the inflated test set but not the other more challenging portion Toral et al. (2018); Freitag et al. (2019); Edunov et al. (2020), Appendix A Table 7 and raise concerns that BT is not as effective as expected. Others argue that BT can still improve on both test sets Edunov et al. (2020).

While for almost all test sets the original language of each example is known, the majority of training data does not contain this meta information. Studies of the impact of translationese on training examples are thus mostly based on Europarl where meta information is given. For instance, Ozdowska and Way (2009) argue that the original side of each training sample is important when building a statistical machine translation (SMT) system. Kurokawa et al. (2009); Koppel and Ordan (2011); Sominsky and Wintner (2019) build classifiers to automatically detect the direction of each training sample. Riley et al. (2020) use a CNN classifier to separate the training data at scale and bias the NMT model via tagging to generate translations that look like the original text. Human evaluation demonstrates that this produces more accurate and natural translations. Further, Amponsah-Kaakyire et al. (2021) investigate the impact of training samples generated with pivot ("relay") languages.

Our work differs from all previous work in that we conduct causal inference Pearl (2009); Peters et al. (2017) to contribute causal insights on how translationese affects MT.

## 6 Future Work

We list several directions for potential future work: (1) It will be meaningful to explore whether the conclusions of this paper can generalize to higher-resource data and a wider variety of languages. (2) In real-world MT systems, it is important not only to answer whether aligned training data or unsigned training data is better (when deciding how to distribute the budget to collect data for a usage scenario with a fixed direction), but also to investigate how to utilize the mixed training data (when trying to make the best use of the existing data) and contribute to the best possible translation systems. Riley et al. (2020); Larkin et al. (2021); Freitag et al. (2022) suggest adding a tag per sample specifying the direction to make use of the unaligned data. Future work can also explore if there can be an end-to-end model jointly inferring the human translation direction and signaling the model to deal with the aligned and unaligned directions differently. (3) In our study, Table 5 mainly confirms the causal effects of the data-model direction match on the MT performance, but the actual reasons for positive and negative ACEs are still worth further investigation.

## 7 Conclusion

In conclusion, this work study the causal effects of three important factors on MT performance: the test-model, train-test, and data-model direction match. We provide suggestions for future study in MT, such as using more training data in the aligned direction and paying attention to whether the nature of the translation task is causal or anticausal.

### Ethical Considerations

**Data Privacy and Bias:** This research mainly focuses on translation using the Europarl Koehn ( 

2005) corpus, which is widely adopted in the community. There are no data privacy issues or bias against certain demographics with regard to this dataset.

**Potential Use:** The potential use of this study is to improve future MT practice in terms of both evaluation and training.

**Generalizability:** Most conclusions in this study are language-agnostic and potentially help MT in all language pairs, although due to the limitations of available data, the study mainly uses the common languages, English, German, French, and Spanish, in a relatively low-resource setting of around 100K to 200K data. It will be meaningful to explore whether the conclusions of this paper can generalize to higher-resource data and a wider variety of languages. There is a possibility that the findings of the study will need to be further adjusted in different settings, which we strongly encourage future work to explore.

**Limitations:** First, the current study mainly looks into clear cases of causal or anticausal learning, but there can potentially be a third case where both languages are translated from a third language, as pointed out in Riley et al. (2020, Figure 1), which is worth exploring for future work.

Second, this work extracts human translation directions from the Europarl corpus, with the assumption that the speakers at the European Parliament tend to be native speakers. It might also be possible that the Europarl corpus contains text annotated as originals but from non-native speakers, but since the European Parliament is a highly formal and important venue, the speakers tend to be at least proficient users of that language, if not native. In addition, for non-English languages in our corpus, it is highly likely that the speech comes from a native speaker.

Third, in addition to the length and content factors considered in this work, it could be interesting to look at other factors that can constitute the \(Z\) variable in Eq. (3). Some motivations include that the data-model direction match seems to be a clear cause for the MT performance, and the fact that it does not always show a very large causal effect might mean that there are additional hidden variables to take into consideration.

Lastly, due to financial budgets, we did not use human evaluation in addition to the BLEU score reported in this work. We released all outputs of our model so future work can feel free to evaluate our results by human evaluation or various other automatic evaluation metrics. See more discussions on recommended evaluation metrics in Appendix D.3.

## Acknowledgments

We sincerely thank Prof Rico Sennrich for his constructive suggestions on improving the message we present from our experiments to the MT community, and Isaac Caswell at Google Translate for proofreading our paper. We thank Claudia Shi for the insightful discussions on the matching method and suggestions on papers to cite. We thank Shaoshu Yang and Di Jin for supporting us with computational resources. We thank Zhiheng Lyu for helping to code a fast version of the sentence similarity matching.

This material is based in part upon works supported by the German Federal Ministry of Education and Research (BMBF): Tubingen AI Center, FKZ: 01IS18039B; the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645; the John Templeton Foundation (grant #61156); a Responsible AI grant by the Haslerstiftung; and an ETH Grant (ETH-19 21-1).

Zhijing Jin is supported by the Open Phil AI Fellowship and the Vitalik Buterin PhD Fellowship from the Future of Life Institute.

## Author Contributions

**Jingwei Ni** conducted most of the MT experiments, collected the most updated version of CausalMT dataset, and performed various analyses.

**Zhijing Jin** designed the project, conducted the causal inference experiments, structured the first version of the codes to run the MT experiments, and collected the first version of CausalMT dataset which is also used in our previous study (Jin et al., 2021a).

**Markus Freitag** helped design the storyline of the translationalese part, gave insights on what analyses are important, and provided suggestions on the evaluation.

**Mrinmaya Sachan** and **Bernhard Scholkopf** guided the project and provided substantial contributions to the storyline of the paper.

**Everyone** contributed to writing the paper 



## References

* A. Amponsah-Kaakyire, D. Pylypenko, C. Espana-Bonet, and J. van Genabith (2021)Do not rely on relay translations: multilingual parallel direct Europarl. In Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age, Online, pp. 1-7. External Links: Link Cited by: SS3.4.
* M. Baker (1993)Corpus linguistics and translation studies: implications and applications. In Text and Technology, External Links: Link Cited by: SS5.
* M. Baker (1995)Corpora in translation studies: an overview and some suggestions for future research. Target. International Journal of Translation Studies7 (2), pp. 223-243. External Links: Document, ISSN 0862-455X Cited by: SS1.
* M. Baker (1996)Corpus-based translation studies: The challenges that lie ahead. Benjamins Translation Library18, pp. 175-186. External Links: Link Cited by: SS1.
* M. Baroni and S. Bernardini (2006)A new approach to the study of translationese: machine-learning the difference between original and translated text. Literary and Linguistic Computing21 (3), pp. 259-274. External Links: Link Cited by: SS1.
* L. Barrault, O. Bojar, M. R. Costa-jussa, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, W. Koehn, S. Malmasi, C. Monz, M. Muller, S. Pal, M. Post, and M. Zampieri (2019)Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 1-61. External Links: Link Cited by: SS1.
* S. Bird, E. Klein, and E. Loper (2009)Natural language processing with Python: analyzing text with the natural language toolkit.. O'Reilly Media. Inc.. Cited by: SS1.
* D. M. Blei, A. Y. Ng, and M. I. Jordan (2001)Latent dirichlet allocation. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], pp. 601-608. External Links: Link Cited by: SS1.
* N. Bogoychev and R. Sennrich (2019)Domain, translationese and noise in synthetic data for neural machine translation. CoRRabs/1911.03362. External Links: Link Cited by: SS1.
* O. Bojar, C. Federmann, M. Fishel, Y. Graham, B. Haddow, P. Koehn, and C. Monz (2018)Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Belgium, Brussels, pp. 272-303. External Links: Link Cited by: SS1.
* O. Bojar and A. Tamchyna (2011)Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland, pp. 330-336. External Links: Link Cited by: SS1.
* F. Burlot and F. Yvon (2018)Using monolingual data in neural machine translation: a systematic study. In Proceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium, pp. 144-155. External Links: Link Cited by: SS1.
* E. A. Dinic (1970)Algorithm for solution of a problem of maximum flow in networks with power estimation. In Soviet Math. Doklady, Vol. 11, pp. 1277-1280. External Links: Link Cited by: SS1.
* S. Edunov, M. Ott, M. Auli, and D. Grangier (2018)Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 489-500. External Links: Link Cited by: SS1.
* S. Edunov, M. Ott, M. Kanzato, and M. Auli (2020)On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2836-2846. External Links: Link Cited by: SS1.
* M. Freitag, I. Caswell, and S. Roy (2019)APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Florence, Italy, pp. 34-44. External Links: Link Cited by: SS1.
* M. Freitag, D. Grangier, and I. Caswell (2020)BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 61-71. External Links: Link Cited by: SS1.
* M. Freitag, R. Rei, N. Mathur, C. Lo, C. Stewart, G. Foster, A. Lavie, and O. Bojar (2021)Results of the wmt21 metrics shared task: evaluating metrics with expert-based human evaluations on tcd and news domain. In Proceedings of the Sixth Conference on Machine Translation, Online, pp. 733-774. External Links: Link Cited by: SS1.
* M. Freitag, D. Vilar, D. Grangier, C. Cherry, and G. Foster (2022)A natural diet: towards improving naturalness of machine translation output. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, pp. 3340-3353. External Links: Link Cited by: SS1.



Martin Gellerstam. 1986. Translationese in swedish novels translated from english. _Translation studies in Scandinavia_, 1:88-95.
* Glymour et al. (2019) Clark Glymour, Kun Zhang, and Peter Spirtes. 2019. Review of causal discovery methods based on graphical models. _Frontiers in Genetics_, 10:524.
* Graham et al. (2020) Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and transitiones in machine translation evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 72-81, Online. Association for Computational Linguistics.
* He et al. (2019) Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. _CoRR_, abs/1909.13788.
* Heckerman et al. (1999) David Heckerman, Christopher Meek, and Gregory Cooper. 1999. A bayesian approach to causal discovery. _Computation, causation, and discovery_, 19:141-166.
* Holland (1988) Paul W Holland. 1988. Causal inference, path analysis and recursive structural equations models. _ETS Research Report Series_, 1988(1):i-50.
* Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
* Huang et al. (2018) Biwei Huang, Kun Zhang, Yizhu Lin, Bernhard Scholkopf, and Clark Glymour. 2018. Generalized score functions for causal discovery. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018_, pages 1551-1560. ACM.
* Macus et al. (2012) Stefano M Macus, Gary King, and Giuseppe Porro. 2012. Causal inference without balance checking: Coarsened exact matching. _Political analysis_, 20(1):1-24.
* Jin et al. (2019) Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. IMaT: Unsupervised text attribute transfer via iterative matching and translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3097-3109, Hong Kong, China. Association for Computational Linguistics.
* Jin et al. (2021a) Zhijing Jin, Zeyu Peng, Tejas Vaidhya, Bernhard Schoelkopf, and Rada Mihalcea. 2021a. Mining the cause of political decision-making from social media: A case study of COVID-19 policies across the US states. In _Findings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021_. Association for Computational Linguistics.
* Jin et al. (2021b) Zhijing Jin, Julius von Kugelgen, Jingwei Ni, Tejas Vaidhya, Ayush Kaushal, Marinnas Sachan, and Berhard Schoelkopf. 2021b. Causal direction of data collection matters: Implications of causal and anticausal learning for NLP. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9499-9513, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In _Proceedings of the Sixth Conference on Machine Translation_, pages 478-494, Online. Association for Computational Linguistics.
* Koehn (2005) Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _Proceedings of Machine Translation Summit X: Papers_, pages 79-86, Phuket, Thailand.
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zeng, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
* Kolias et al. (2014) Vassilis Kolias, Ioannis Anagnostopoulos, and Elefterrios Kayafas. 2014. Exploratory analysis of a terabyte scale web corpus. _CoRR_, abs/1409.5443.
* Koppel and Ordan (2011) Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 1318-1326, Portland, Oregon, USA. Association for Computational Linguistics.
* Kurokawa et al. (2009) David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In _Proceedings of Machine Translation Summit XII: Papers_, Ottawa, Canada.
* Larkin et al. (2021) Samuel Larkin, Michel Simard, and Rebecca Knowles. 2021. Like chalk and cheese? On the effects of translations in MT training. In _Proceedings of Machine Translation Summit XVIII: Research Track_, pages 103-113, Virtual. Association for Machine Translation in the Americas.
* Laubli et al. (2018) Samuel Laubli, Rico Senrich, and Martin Volk. 2018. Has machine translation achieved human parity? A case for document-level evaluation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics 

Sara Lavisoas-Braithwaite. 1998. Universals of translation. _Routledge encyclopedia of translation studies__London: Routledge_, pages 288-291.
* Lembersky et al. (2012) Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Language models for machine translation: Original vs. translated texts. _Computational Linguistics_, 38(4):799-825.
* Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 314-319, Florence, Italy. Association for Computational Linguistics.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ozdowska and Way (2009) Sylvia Ozdowska and Andy Way. 2009. Optimal bilingual data for French-English PB-SMT. In _Proceedings of the 13th Annual conference of the European Association for Machine Translation_, Barcelona, Spain. European Association for Machine Translation.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Pearl (1993) Judea Pearl. 1993. [Bayesian analysis in expert systems]: Comment: Graphical models, causality and intervention. _Statistical Science_, 8(3):266-269.
* Pearl (1995) Judea Pearl. 1995. Causal diagrams for empirical research. _Biometrika_, 82(4):669-688.
* Pearl (2009) Judea Pearl. 2009. _Causality: Models, reasoning, and inference_. Cambridge University Press.
* Peters et al. (2017) Jonas Peters, Dominik Janzing, and Bernhard Scholkopf. 2017. _Elements of causal inference: Foundations and learning algorithms_. The MIT Press.
* Poncelas et al. (2018) Alberto Poncelas, Dimitar Sht. Shterionov, Andy way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating backtranslation in neural machine translation. _CoRR_, abs/1804.06189.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 101-108, Online. Association for Computational Linguistics.
* Rabinovich and Winther (2015) Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. _Transactions of the Association for Computational Linguistics_, 3:419-432.
* Reichenbach (1956) Hans Reichenbach. 1956. _The direction of time_, volume 65. University of California Press.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4512-4525, Online. Association for Computational Linguistics.
* Riley et al. (2020) Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020. Translationese as a language in "multilingual" NMT. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7737-7746, Online. Association for Computational Linguistics.
* Roberts et al. (2020) Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen. 2020. Adjusting for confounding with text matching. _American Journal of Political Science_, 64(4):887-903.
* Rosenbaum and Rubin (1983) Paul R Rosenbaum and Donald B Rubin. 1983. The central role of the propensity score in observational studies for causal effects. _Biometrika_, 70(1):41-55.
* July 1, 2012_. icml.cc / Omnipress.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Sominsky and Wintner (2019) Ilia Sominsky and Shuly Wintner. 2019. Automatic detection of translation direction. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_, pages 1131-1140, Varna, Bulgaria. INCOMA Ltd.
* Sutskever et al. (2015) 

Pater Spirtes, Clark Glymour, and Richard Scheines. 2000a. Constructing bayesian network models of gene expression networks from microarray data. In _Proceedings of the Atlantic Symposium on Computational Biology_.
* Spirtes et al. (2000b) Peter Spirtes, Clark Glymour, and Richard Scheines. 2000b. _Causation, Prediction, and Search, Second Edition_. Adaptive computation and machine learning. MIT Press.
* Toral et al. (2018) Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? Reassessing claims of human parity in neural machine translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 113-123, Brussels, Belgium. Association for Computational Linguistics.
* Toury (1980) Gideon Toury. 1980. _In search of a theory of translation_. Porter Institute for Poetics and Semiotics, Tel Aviv University.
* Toury (1995) Gideon Toury. 1995. _Descriptive translation studies and beyond_, volume 4. John Benjamins.
* Ustaszewski (2019) Michael Ustaszewski. 2019. Optimising the Europarl corpus for translation studies with the EuroparlExtract toolkit. _Perspectives_, 27(1):107-123.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Veitch et al. (2021) Victor Veitch, Alexander D'Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021. Counterfactual invariance to spurious correlations in text classification. _Advances in Neural Information Processing Systems_, 34.
* Veitch et al. (2020) Victor Veitch, Dhanya Sridhar, and David M. Blei. 2020. Adapting text embeddings for causal inference. In _Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020_, volume 124 of _Proceedings of Machine Learning Research_, pages 919-928. AUAI Press.
* Volansky et al. (2015) Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. _Digit. Scholarsh. Humanist._, 30(1):98-118.
* Yarowsky (1995) David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In _33rd Annual Meeting of the Association for Computational Linguistics_, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
* Zhang and Toral (2019) Mike Zhang and Antonio Toral. 2019. The effect of translationes in machine translation test sets. In _Translation (Volume 1: Research Papers)_, pages 73-81, Florence, Italy. Association for Computational Linguistics.

 

Reproducibility, License, and Copyright

We open-source our codes and datasets, which are both uploaded to the submission system. In our data, we include all three variations: the full CausalMT dataset, the split used for the semi-supervised learning experiments, and the subset after matching the contents and sentence lengths. In our codes, we include all commands with hyperparameters to help future work to reproduce our results.

The codes and data are under MIT license. Note that the Europarl dataset has no copyright restriction, according to its official website.6

Footnote 6: [https://www.statmt.org/euorap1/](https://www.statmt.org/euorap1/)

## Appendix B Linguistic Property Analysis

We also open-source the codes to calculate the linguistic properties of our dataset in Table 1. We use the Python library Stanza7(Qi et al., 2020) to tokenize the sentences when calculating the number of sentences per sample. For computational efficiency, we use NLTK8(Bird et al., 2009) to tokenize the words and count the vocabulary.

Footnote 7: [https://stanfordnlp.github.io/stanza/fairseq_cli/generate.py](https://stanfordnlp.github.io/stanza/fairseq_cli/generate.py)

Footnote 8: [https://github.com/pytorch/fairseq/tree/main/examples/backtranslation](https://github.com/pytorch/fairseq/tree/main/examples/backtranslation)

We use the Python library spaCy9(Honnibal and Montani, 2017) to calculate the punctuation per sample. We use a passive voice checker10(only available in English). For the expansion factor, we formatted Table 1 using the ratio of the two languages in a descending alphabetical order of each language pair. In our table, it happens to be the ratio of the more verbose language to the less verbose language in each language pair.

Footnote 10: [https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl)

## Appendix C Implementation Details

### Preprocessing

To prepare the text for the models, we follow the preprocessing scripts of fairseq Ott et al. (2019).11 Specifically, we use the Moses tokenizer Koehn et al. (2007),12 the default byte pair encoding (BPE) size of 40K subwords, and remove sentence pairs that of larger than 1.5 length ratio from the training set.

Footnote 12: [https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl)

### Evaluation Script

We use the fairseq script13 to calculate the BLEU score Papineni et al. (2002) of each translation model, with a beam width of 5, BPE removed, and detokenized by moses.

Footnote 13: [https://github.com/pytorch/fairseq/blob/main/fairseq_cli/generate.py](https://github.com/pytorch/fairseq/blob/main/fairseq_cli/generate.py)

### Model Details

We use the sequence-to-sequence Transformer model Vaswani et al. (2017) implemented by the fairseq library Ott et al. (2019). Specifically, we use a six-layer Transformer, a label smoothing of 0.1, a weight decay of 0.0001, a dropout of 0.3, 4000 warming updates, and a learning rate of 0.0005. All results are reported by a single run but a fixed random seed.

For the semi-supervised learning, we implement the BT model following Edunov et al. (2020) to use the Facebook-FAIR system of the WMT'19 news shared translation task.14 All the hyperparameters are the same as the supervised system, with a learning rate of 0.0007 on both the supervised training data and the generated pseudo-parallel corpus. We implement the ST model by He et al. (2019) following their script,15 and also keep the hyperparameters the same as the supervised model.

Footnote 14: [https://github.com/pytorch/fairseq/tree/main/examples/backtranslation](https://github.com/pytorch/fairseq/tree/main/examples/backtranslation)

Footnote 15: [https://github.com/jxhe/self-training-text-generation/blob/master/self_train.sh](https://github.com/jxhe/self-training-text-generation/blob/master/self_train.sh)

### Training Details

We train the supervised learning model and each step in the semi-supervised learning scripts for 1000 epochs. We select the model with the best performance on the development set and report the final evaluation results on the test set.

All experiments are run on NVIDIA RTX2080 GPUs. Each supervised learning experiment takes around 32 GPU hours, and each semi-supervised learning experiment takes about 128 GPU hours.

## Appendix D Additional Experimental Results

### Effect of Train-Test Direction Match on Supervised Learning

To inspect the influence of train-test direction match on the MT performance, we conduct all experiments on our CausalMT test sets and also the standard newestest2014 test sets. For the supervised learning performance, we list the performance on 

the CausalMT test sets in the main paper in Table 2, and list the additional performance on the newstest2014 test sets in Table 6.

For better visualization of the trends, we also provide line plots of the same experimental results in Table 2. Specifically, we plot the results of German-English translation in Figure 3(a) using our previous experiment results in Table 2. We also include the diagram of all five language pairs in Figure 3(b).

In Figure 3(a), we use lines with the same darkness of color for the same model trained on different data directions. Results show that the data-model direction match matters significantly. Taking the German-to-English translation models (- - - and --), the two data directions can cause up to 4.53 difference in BLEU points. In the current figures, we also see that the data direction with a smaller expansion factor is a better training corpus than the other one.

We use the same line type (dashed or solid) for models trained on the same data. Using the same data, the performance of the two different directions of models cannot be compared directly because the target language is different, causing the BLEU calculation to be different.

### Effect of Train-Test Direction Match on Semi-Supervised Learning

For the semi-supervised learning performance, we show the performance on the newstest2014 test sets in Table 3 in the main paper, and performance on the test sets of CausalMT in Table 7. Note that the decrease of ST performance on En-Es and Es-Fr pairs is possible because ST is more sensitive to the quality of the model learned on the supervised data, and these language pairs have a smaller training data size of 90K compared with 200K+ data for all the other language pairs.

### Non-BLEU Evaluation Metrics

Due to financial constraints, we did not use human evaluation in addition to the BLEU score in our study. However, we encourage future studies in this line of work to include human evaluation results, as evaluation is important given the nature of such work, and human evaluation is reported to be more reflective of the real translation quality Edunov et al. (2020).

To make it convenient for follow-up work, we open-source all the outputs generated by our model

\begin{table}
\begin{tabular}{c c|c|c|c|c} \hline \multicolumn{3}{c|}{**de-to-en Translation**} & \multicolumn{3}{c}{**en-to-de Translation**} \\ \(\alpha\%\) T1 (de, en°)T2 (de, en) & \(\alpha\%\) T1 (fn, ee)T2 (en°, de) & \(\alpha\%\) T1 (fn, ee)T2 (en°, de) \\ \hline
0v & 14.21 & 19.10 & 0\% & 11.18 & 15.49 \\
25\% & 15.01 & 18.09 & 25\% & 12.69 & 12.49 \\
50\% & 16.71 & 18.17 & 50\% & 13.30 & 14.33 \\
75\% & 16.91 & 16.27 & 75\% & 13.38 & 13.16 \\
100\% & 16.02 & 12.91 & 100\% & 13.28 & 10.68 \\ \hline \multicolumn{3}{c|}{**en-to-fr Translation**} & \multicolumn{3}{c}{**fr-to-en Translation**} \\ \(\alpha\%\) T1 (en, fr°)T2 (en°, fr) & \(\alpha\%\) T1 (f, en°)T2 (fi°, en) & \(\alpha\%\) T1 (f, en°)T2 (fi°, en) \\ \hline
0v & 16.61 & 21.33 & 0\% & 16.34 & 23.26 \\
25\% & 18.56 & 20.95 & 25\% & 18.81 & 23.31 \\
50\% & 20.45 & 21.66 & 50\% & 19.75 & 23.20 \\
75\% & 21.19 & 21.05 & 75\% & 21.09 & 22.01 \\
100\% & 21.43 & 19.30 & 100\% & 20.02 & 19.78 \\ \hline \end{tabular}
\end{table}
Table 6: Effect of train-test direction match on the en-fr and en-de test sets of _newstest2014_.

 

to our GitHub. We also encourage future work to adopt more evaluation metrics such as COMET and BLUERT, which are among the metrics that correlates the best with human judgements according to the metric task at WMT Freitag et al. (2021). COMET, for instance, correlates better with human judgements than BLEU Kocmi et al. (2021) and goes further string matching.

## Appendix E Implementation Details for Causal Inference

### Statistics of the Matched Corpora

We list the statistics of the matched corpora in Table 8, and analyze its linguistic properties in Table 9.

### Confirming the Causal Graph by Causal Discovery

To check our causal graph assumption, we first verify whether data-model direction match is a cause for MT performance using causal discovery.

We use the causal discovery algorithm, fast causal inference (FCI) Spirtes et al. (2000a), to verify that the data-model direction match causally affects the translation performance, conditioned on other factors such as the sentence length and topics.

FCI is the most appropriate causal inference method for this analysis since there might exist hidden confounders that affect the MT performance, which normal causal discovery methods such as score-based methods Heckerman et al. (1999); Huang et al. (2018) and other constraint-based algorithms like Peter-Clark (PC) algorithm Spirtes et al. (2000b, SS5.4.2, pp. 84-88) cannot handle Glymour et al. (2019). FCI gives asymptotically correct results in the presence of confounders, and outputs Markov equivalence classes, i.e., a set of causal structures satisfying the same conditional independences.

Given a language pair \(X\) and \(Y\), we generate eight sets of experiment results, by varying the two training directions, two test directions, and two model directions. We extract the test samples of all eight experiments, and since each test set is 2K, there are 16K samples in total. On the 16K samples, besides keeping the label of their data-model direction match, translation performance in BLEU, we also calculate the other factors such as the test-model direction match, train-test direction

\begin{table}
\begin{tabular}{c c|l} \hline \multicolumn{3}{c}{**German-to-English (de-to-en) Translation**} \\  & Test (1, de-en) & Test (2 e/o*, en) \\ \hline Sup. on Equal Mix & 25.52 & 29.02 \\ + ST (de, en*) & **+3.25 (Aligned)** & +2.59 \\ + BT (de*, en) & -0.97 & **+2.67 (Aligned)** \\ \hline \multicolumn{3}{c}{**English-to-German (en-to-de) Translation**} \\  & Test 1 (en, de*) & Test 2 (en*, de) \\ \hline Sup. on Equal Mix & 23.76 & 21.48 \\ + ST (ca, de*) & **+1.39 (Aligned)** & +1.44 \\ + BT (em*, de) & -0.63 & -0.51 (Aligned) \\ \hline \multicolumn{3}{c}{**German-to-French (de-to-fr) Translation**} \\  & Test 1 (dre, fr*) & Test 2 (de*, fr) \\ \hline Sup. on Equal Mix & 25.42 & 30.10 \\ + ST (de, fr*) & **+1.79 (Aligned)** & +1.23 \\ + BT (de*, fr) & -0.35 & **+1.69 (Aligned)** \\ \hline \multicolumn{3}{c}{**French-to-Gearman (fct-de) Translation**} \\  & Test 1 (ft, dc*) & Test 2 (fi*, der) \\ \hline Sup. on Equal Mix & 21.89 \\ + ST (fc, d*) & **+2.46 (Aligned)** & +2.09 \\ + BT (ff*, df) & +1.07 & +0.87 (Aligned) \\ \hline \multicolumn{3}{c}{**English-to-French (en-to-fr) Translation**} \\  & Test 1 (cn, fr*) & Test 2 (en*, f) \\ \hline Sup. on Equal Mix & 35.64 & 36.19 \\ + ST (en, fr*) & **+2.04 (Aligned)** & +1.89 \\ + BT (cm*, rf) & +0.13 & +1.33 (Aligned) \\ \hline \multicolumn{3}{c}{**French-to-GeDefinition (fo-ten) Translation**} \\  & Test 1 (ft, cn*) & Test 2 (fi*, en) \\ \hline Sup. on Equal Mix & 34.35 & 33.75 \\ + ST (cf, es*) & **+1.76 (Aligned)** & +2.28 \\ + BT (ff*, en) & +0.43 & +1.89 (Aligned) \\ \hline \multicolumn{3}{c}{**English-to-Spanish (en-to-es) Translation**} \\  & Test 1 (en, es*) & Test 2 (en*, es) \\ \hline Sup. on Equal Mix & 33.65 & 34.01 \\ + ST (en, es*) & -0.10 (Aligned) & -0.75 \\ + BT (em*, es) & -0.36 & **+1.40 (Aligned)** \\ \hline \multicolumn{3}{c}{**Spanish-to-English (st-to-ea) Translation**} \\  & Test 1 (es, en*) & Test 2 (es*, en) \\ \hline Sup. on Equal Mix & 35.00 \\ + ST (cs*, em) & -0.46 (Aligned) & -0.41 \\ + BT (ss*, en) & +0.63 & **+1.77 (Aligned)** \\ \hline \multicolumn{3}{c}{**Spanish-to-French (est-to-fr) Translation**} \\  & Test 1 (es, fr*) & Test 2 (es*, fi) \\ \hline Sup. on Equal Mix & 38.30 \\ + ST (cs*, fs*) & +0.58 (Aligned) & +0.83 \\ + BT (ss*, ff) & +1.00 & **+2.01 (Aligned)** \\ \hline \multicolumn{3}{c}{**French-to-Spanish (fo-te-es) Translation**} \\  & Test 1 (fn, cs*) & Test 2 (fi*, es) \\ \hline Sup. on Equal Mix & 40.61 \\ + ST (fs*, es*) & -0.84 (Aligned) & -1.09 \\ + BT (fm*, es) & -0.14 & **+0.12 (Aligned)** \\ \hline \end{tabular}
\end{table}
Table 7: Performance analogous to Table 3 but on our CausalMT test sets.

\begin{table}
\begin{tabular}{c c c c} \hline \multicolumn{3}{c}{**Human Trans. Dir. Train**} & **Dev** & **Test** \\ \hline De \({}^{\frac{H}{2}}\)En & 107K & 1K & 2K \\ En \({}^{\frac{H}{2}}\)De & 107K & 1K & 2K \\ De \({}^{\frac{H}{2}}\)Fr & 133K & 1K & 2K \\ Fr \({}^{\frac{H}{2}}\)De & 133K & 1K & 2K \\ En \({}^{\frac{H}{2}}\)Fr & 87K & 1K & 2K \\ Fr \({}^{\frac{H}{2}}\)En & 87K & 1K & 2K \\ En \({}^{\frac{H}{2}}\)Es & 47K & 1K & 2K \\ Es \({}^{\frac{H}{2}}\)Fn & 47K & 1K & 2K \\ Es \({}^{\frac{H}{2}}\)Fr & 50K & 1K & 2K \\ Fr \({}^{\frac{H}{2}}\)Es & 50K & 1K & 2K \\ \hline \end{tabular}
\end{table}
Table 8: Dataset statistics for five language pairs after matching. Each language pair has data from two human translation directions (Human Trans. Dir.), e.g., De\({}^{\frac{H}{2}}\)En and De\({}^{\frac{H}{2}}\)En.

 

match, source sentence length, and the topic vector by topic modeling on all the training data of the language pair \(X\) and \(Y\). We run the FCI algorithm using the causal-learn Python package16 over all the variables of interest. The implementation details are in the Appendix.

Footnote 16: [https://github.com/cmu-phil/causal-learn](https://github.com/cmu-phil/causal-learn)

The resulting causal graph on the German-English language pair is in Figure 2. The results confirm our hypothesis that the data-model direction match (causal vs. anticausal direction) does have a causal effect on the BLEU score, together with other factors such as the sentence length and topics.