SlowGuess/NAACL_2022_OCR huggingface dataset:
To train ST or BT, we use the second halves of the training data only as the monolingual corpora. For example, if the translation task is English-to-German translation, ST generates a pseudo-parallel corpus with original English paired with machine-translated pseudo-German, which we denote as (en, de**) . For readability, we mark the machine-translation direction with ST and BT by  $^{**}$  and the human translation direction by \*
Our hypothesis is that the machine-translated text pairs (en,  $\mathrm{de}^{**}$ ) will also show similar properties as the human-translated training data (en,  $\mathrm{de}^*$ ). Specifically, the more the pseudo-training data is aligned with the test set, the higher performance the semi-supervised learning method will achieve. This is confirmed by the experiment results in Table 3, where, across all settings, no matter which semi-supervised learning method is used, when the pseudo-training data has the same translation direction as the test set, the resulting performance is generally higher. The experiments conducted on CAUSALMT test sets also generally show the same trend, and, due to the space limit, we include the results in the Appendix Table 7.
4 Causal Effect of Data-Model Direction Match
The second contribution of this work is to inspect how much another factor, the data-model direction match, causally affects the MT performance. Formally, our research question is that, for a given translation task  $X$ -to-  $Y$ , considering an equal mix of the test set, does the human translation direction of the training data still matter? If so, how large is the effect, and is it language-/task-dependent?
This section will use causal inference to isolate the effect of data-model direction match from other possible confounders and discuss its effect in different languages and translation tasks.
4.1 Correlation in Previous Experiments
Our previous experiments show that data-model direction match correlates with MT performance.
Specifically, for each translation task in Table 2, there is a clear difference between the causal learning and anticausal learning model. We copy this naive difference to the "Diff" column of Table 5.
This naive difference represents  $\mathbb{E}_{\mathrm{aligned\_corpus}}[S] - \mathbb{E}_{\mathrm{unaligned\_corpus}}[S]$ , which compares the performance score  $S$  on the aligned corpus and  $S$  on the unaligned corpus, without controlling for potential confounders. The famous "correlation does not imply causation" implies that this formulation cannot answer the causal question, as the two expectation terms are taken over two different distributions that are not necessarily comparable. However, by Reichenbach's Common Cause Principle (Reichenbach, 1956), correlation implies the existence of some common cause behind the two correlated variables, which motivates us to investigate the causal relationship in the next section.
4.2 Setup of the Causal Effect Estimation
Formulating the causal effect. Instead of just correlational analyses, we aim to estimate the average causal effect (ACE) (Holland, 1988; Pearl, 2009) of the data-model direction match (i.e., causal vs. anticausal learning)  $M$  on the translation performance  $S$ :
$$
\begin{array}{l} \mathrm {A C E} = P (S = s \mid \mathrm {d o} (M = 1)) \tag {1} \\ - P (S = s \mid \operatorname {d o} (M = 0)), \\ \end{array}
$$
where, according to do-calculus (Pearl, 1995) in causal inference, the operator  $\mathrm{do}(M = 0$  or 1) means to intervene on the data-model direction match to be 0 (i.e., anticausal learning) or 1 (i.e., causal learning). The ACE formulation is about how much the model performance  $S$  will differ if intervening the data-model direction match  $M$  to be 0 or 1.
To estimate the ACE, we first draw the causal graph considering all variables that can interfere with the relationship between data-model direction match and MT performance. The main additional factors we need to control for are in the causal graph in Figure 2. We make the assumption that it is very likely that the two corpora of different human translation directions also vary by sentence
Note that there are two notions of causality here, one is the intervention we are interested in, namely the data-model direction match, known as causal vs. anticausal learning, and the other is the meta-level causality we are interested in, namely how much the data-model direction match (as a binary variable) causally affect the translation performance.
Figure 2: Causal graph about how the data-model direction match  $M$  affects MT performance score  $S$ , considering the other covariates  $Z$  including the distribution of sentence lengths and content.
Figure 3: Distribution of topics and sentence lengths after matching, using the German-English language pair (between  $\mathrm{De}\xrightarrow{\mathrm{H}}\mathrm{En}$  and  $\mathrm{De}\xleftarrow{\mathrm{H}}\mathrm{En}$ ) as an example.
lengths and the distribution of content (Bogoychev and Sennrich, 2019) due to a hidden confounder (i.e., a common cause) such as the nature of Europarl. Note that since our research question is about which training data matters for given a translation task, the data-model direction match is equivalent to the human translation direction of the training data, as the model translation direction is fixed.
Given the causal graph in Figure 2, the ACE in Eq. (1) can be calculated by conditioning on the set of variables  $Z$  which blocks the backdoor paths (Pearl, 1995) between  $M$  and  $S$ . ( $Z$  fits the backdoor criterion (Pearl, 1993) in that the sentence lengths and content block all non-directed paths from  $M$  to  $S$ , and neither is a descendant of any node on the directed path from  $M$  to  $S$ .) An intuitive interpretation can be that when we directly look at the correlation between the data-model direction match and MT performance, it might also be due to that different corpora have different distributions of sentence lengths and content. Therefore, we need to control the sentence lengths and content so that the performance difference will be solely due to the data-model direction match.
Formally, the ACE using the do-notation can be calculated by conditioning on  $Z$ . Specifically, we integrate over the distribution of  $P(Z)$ , and calculate the difference in the conditional probability distribution  $P(S = s|M = 1,Z = z) - P(S = s|M = 0,Z = z)$  of  $S$  given the data-model di
Corpus	Matched Pairs
DeH→En	Let me make some comments on specific points.
DeH→En	However, I have one or two points.
DeH→En	From our perspective, it is now urgently necessary that the Council also accepts this proposal, so that the negotiations can commence as soon as possible.
DeH→En	That greater urgency has been recognised in the Council suggestion that we should have an intergovernmental conference beginning next year, something which we subscribe to.
DeH→En	We want a European Union that is strong, effective and democratic, and all those who want to make it no more than a free trade zone within Europe will have a fight on their hands.
DeH→En	I agree that the European Union needs an integrated, coherent and consistent European energy policy that maintains Europe's competitiveness, safeguards our environmental objectives and ensures our security of supply.
Table 4: Examples of matched pairs between the  $\mathrm{De}\xrightarrow{\mathrm{H}}\mathrm{En}$  and  $\mathrm{De}\xleftarrow{\mathrm{H}}\mathrm{En}$  corpora.
rejection match value  $M$  conditioned on the other key variables  $Z$  for each of its possible value  $\mathbf{z}$ , as shown in Eq. (2):
$$
\begin{array}{l} \operatorname {A C E} = \int_ {z} [ (P (S = s | M = 1, Z = z) (2) \\ - P (S = s | M = 0, Z = \boldsymbol {z})) P (\boldsymbol {z}) ] \\ = \mathbb {E} _ {\boldsymbol {z}} [ S | M = 1, Z = \boldsymbol {z} ] (3) \\ - \mathbb {E} _ {z} [ S | M = 0, Z = z ]. \\ \end{array}
$$
Finally, we estimate it by comparing the expected values of the model performance score  $S$  given  $M = 0$  or 1 over all possible values of the covariate  $Z$  as in Eq. (3).
Causal effect estimation by matching. To estimate the ACE in Eq. (3), the intuition is that we need to take care of the covariates in  $Z$  so that the aligned setting and the unaligned setting are comparable. We follow the covariate matching method in causal inference (Rosenbaum and Rubin, 1983; Iacus et al., 2012) and adjustment in the high-dimensional setting of text (Roberts et al., 2020; Veitch et al., 2020). Specifically, matching is a method in causal inference to subsample the treated (i.e., the aligned corpus with the model direction) and control samples (i.e., the unaligned corpus with the model direction) so that the covariates of interest are matched.
We aim to match pairs of samples, one from the causal corpus and the other from the anticausal corpus, where we constrain them to share similar contents and similar sentence lengths. In our
English-to-German (en-to-de) Translation	German-to-English (de-to-en) Translation
T1 (en, de*)	T2 (en*, de)	ACE	Diff	T1 (de, en*)	T2 (de*, en)	ACE	Diff
Cau. (en, de*)	21.88	28.77	+3.13	+1.75	Cau. (de, en*)	31.70	28.68	-1.89	-2.14
Ant. (en*, de)	25.33	22.19	Ant. (de*, en)	26.35	35.92
French-to-German (fr-to-de) Translation	German-to-French (de-to-fr) Translation
T1 (fr, de*)	T2 (fr*, de)	ACE	Diff	T1 (de, fr*)	T2 (de*, fr)	ACE	Diff
Cau. (fr, de*)	18.36	25.45	+5.57	+5.0	Cau. (de, fr*)	32.25	29.98	-3.58	-2.71
Ant. (fr*, de)	20.46	17.78	Ant. (de*, fr)	28.07	37.74
French-to-English (fr-to-en) Translation	English-to-French (en-to-fr) Translation
T1 (fr, en*)	T2 (fr*, en)	ACE	Diff	T1 (en, fr*)	T2 (en*, fr)	ACE	Diff
Cau. (fr, en*)	33.77	37.42	+1.43	+2.53	Cau. (en, fr*)	42.60	37.34	-0.14	-0.65
Ant. (fr*, en)	37.67	32.09	Ant. (en*, fr)	38.05	42.03
Spanish-to-English (es-to-en) Translation	English-to-Spanish (en-to-es) Translation
T1 (es, en*)	T2 (es*, en)	ACE	Diff	T1 (en, es*)	T2 (en*, es)	ACE	Diff
Cau. (es, en*)	37.79	33.64	+12.25	+0.63	Cau. (en, es*)	39.04	33.68	+3.50	+3.79
Ant. (es*, en)	21.69	25.24	Ant. (en*, es)	30.76	34.96
French-to-Spanish (fr-to-es) Translation	Spanish-to-French (es-to-fr) Translation
T1 (fr, es*)	T2 (fr*, es)	ACE	Diff	T1 (es, fr*)	T2 (es*, fr)	ACE	Diff
Cau. (fr, es*)	37.09	43.40	+5.84	+2.22	Cau. (es, fr*)	41.67	41.57	-2.74	-1.11
Ant. (fr*, es)	38.45	36.20	Ant. (es*, fr)	39.36	46.62
Table 5: BLEU points of causal learning (Cau.) vs. anticausal (Ant.) directions after controlling for topics and lengths. We calculate the ACE by taking each model's average performance on T1 and T2, and comparing how much causal models outperform anticausal models (ACE). In comparison, we show the naive difference (Diff) by directly comparing the results of causal and anticausal models in Table 2 without topic and length control.
implementation, for each sentence in the causal corpus, we select its most similar match in the anticausal corpus using Dinic's maximal matching algorithm (Dinic, 1970).
Empirically, we limit the sentence length ratio of each matched pair to be no larger than 1.1 and the content to have a cosine similarity larger than 0.7, following the threshold to match a content-similar pseudo-parallel corpus in Jin et al. (2019). To calculate the content-wise similarity of a pair of samples, we represent each sentence by the sentence BERT embedding (Reimers and Gurevych, 2019). In cases of multiple languages as candidates to match the sentence embeddings in, we set a prioritization order of "En>De>Fr>Es" for sentence embedding matching. The prioritization order roughly follows the training data sizes of the multilingual version (Reimers and Gurevych, 2020) of the sentence transformer in the four languages. Note that since the set of factors to control is in a high-dimensional vector space, it is less realistic to use other common matching methods such as propensity score stratification and matching, as pointed out by Roberts et al. (2020).
We check the quality of our matching heuristics. Taking the German-English language pair as an example, we plot the distributions of topics by Latent Dirichlet Allocation (LDA) topic modeling (Blei et al., 2001) and distributions of sentence lengths
across the  $\mathrm{De}\xrightarrow{\mathrm{H}}\mathrm{En}$  and  $\mathrm{De}\xleftarrow{\mathrm{H}}\mathrm{En}$  corpora in Figure 3. We also list some example matched pairs in English in Table 4. Further statistics of the matched corpora are in Appendix E.1.
Finally, based on the matched datasets that control for the sentence lengths and contents, ACE can be calculated by the average difference in MT performance of models trained on the two directions of the new datasets.
4.3 Causal Effect Results
We have three observations from the results in Table 5: (1) The data-model direction match is a clear cause for MT performance. The ACE of data-model direction match on MT performance can be up to 12.25 BLEU points, for example, in the Spanish-to-English translation task. (2) The ACE varies by language and translation tasks. For the English-Spanish language pair, both translation directions get higher BLEU points if the models are trained in the causal learning direction. For other language pairs, the causal effects of data-model direction match are clear, although varying from positive to negative values, the reasons for which are worth future studies. (3) The results of naive differences (Diff) are, in most cases, smaller than that of the causal analysis by ACE. This indicates that the correlational analysis neglects other important factors such as the sentence length and content,
which might also be reflected in the overall difference. The causal analysis is a more appropriate method to isolate the influence of the data-model direction match.
Linguistic studies have long observed the distinct properties of translationese from text originally authored in the same language (Toury, 1980; Gellerstam, 1986; Baker, 1993; Toury, 1995). Recent work in MT identifies that the source side of the translationese-to-original portion of the test sets (i.e., test sets unaligned with the model direction) is easier (Graham et al., 2020), echoing with many previous observations (Toral et al., 2018; Lembersky et al., 2012; Läubli et al., 2018) and thus some suggest to exclude this portion from future test sets, in particular human evaluations (Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2020; Barrault et al., 2019). Nevertheless, Freitag et al. (2019) demonstrate that it is worth reporting automatic metric scores on both directions separately as both types of test sets evaluate different properties of translation quality. In follow-up work, Freitag et al. (2020) introduce paraphrased test sets that overcome the problems of translationese for test sets, not solving the problem for the training data though.
Based on this speculated inflation of MT performance due to translationese in the test set, further work inspects what previous conclusions about the effectiveness of MT models should be recalibrated. Some discover that models with BT mostly improve on the inflated test set but not the other more challenging portion (Toral et al., 2018; Freitag et al., 2019; Edunov et al., 2020, Appendix A Table 7) and raise concerns that BT is not as effective as expected. Others argue that BT can still improve on both test sets (Edunov et al., 2020).
While for almost all test sets the original language of each example is known, the majority of training data does not contain this meta information. Studies of the impact of translationese on training examples are thus mostly based on Europarl where meta information is given. For instance, Ozdowska and Way (2009) argue that the original side of each training sample is important when building a statistical machine translation (SMT) system. Kurokawa et al. (2009); Koppel and Ordan (2011); Sominsky and Wintner (2019) build classifiers to automatically detect the direction of each training sample.
Riley et al. (2020) use a CNN classifier to separate the training data at scale and bias the NMT model via tagging to generate translations that look like the original text. Human evaluation demonstrates that this produces more accurate and natural translations. Further, Amponsah-Kaakyire et al. (2021) investigate the impact of training samples generated with pivot ("relay") languages.
Our work differs from all previous work in that we conduct causal inference (Pearl, 2009; Peters et al., 2017) to contribute causal insights on how translationese affects MT.
We list several directions for potential future work: (1) It will be meaningful to explore whether the conclusions of this paper can generalize to higher-resource data and a wider variety of languages. (2) In real-world MT systems, it is important not only to answer whether aligned training data or unaligned training data is better (when deciding how to distribute the budget to collect data for a usage scenario with a fixed direction), but also to investigate how to utilize the mixed training data (when trying to make the best use of the existing data) and contribute to the best possible translation systems. Riley et al. (2020); Larkin et al. (2021); Freitag et al. (2022) suggest adding a tag per sample specifying the direction to make use of the unaligned data. Future work can also explore if there can be an end-to-end model jointly inferring the human translation direction and signaling the model to deal with the aligned and unaligned directions differently. (3) In our study, Table 5 mainly confirms the causal effects of the data-model direction match on the MT performance, but the actual reasons for positive and negative ACEs are still worth further investigation.
In conclusion, this work study the causal effects of three important factors on MT performance: the test-model, train-test, and data-model direction match. We provide suggestions for future study in MT, such as using more training data in the aligned direction and paying attention to whether the nature of the translation task is causal or anticausal.
Data Privacy and Bias: This research mainly focuses on translation using the Europarl (Koehn,
2005) corpus, which is widely adopted in the community. There are no data privacy issues or bias against certain demographics with regard to this dataset.
Potential Use: The potential use of this study is to improve future MT practice in terms of both evaluation and training.
Generalizability: Most conclusions in this study are language-agnostic and potentially help MT in all language pairs, although due to the limitations of available data, the study mainly uses the common languages, English, German, French, and Spanish, in a relatively low-resource setting of around 100K to 200K data. It will be meaningful to explore whether the conclusions of this paper can generalize to higher-resource data and a wider variety of languages. There is a possibility that the findings of the study will need to be further adjusted in different settings, which we strongly encourage future work to explore.
Limitations: First, the current study mainly looks into clear cases of causal or anticausal learning, but there can potentially be a third case where both languages are translated from a third language, as pointed out in Riley et al. (2020, Figure 1), which is worth exploring for future work.
Second, this work extracts human translation directions from the Europarl corpus, with the assumption that the speakers at the European Parliament tend to be native speakers. It might also be possible that the Europarl corpus contains text annotated as originals but from non-native speakers, but since the European Parliament is a highly formal and important venue, the speakers tend to be at least proficient users of that language, if not native. In addition, for non-English languages in our corpus, it is highly likely that the speech comes from a native speaker.
Third, in addition to the length and content factors considered in this work, it could be interesting to look at other factors that can constitute the  $Z$  variable in Eq. (3). Some motivations include that the data-model direction match seems to be a clear cause for the MT performance, and the fact that it does not always show a very large causal effect might mean that there are additional hidden variables to take into consideration.
Lastly, due to financial budgets, we did not use human evaluation in addition to the BLEU score reported in this work. We released all outputs of our model so future work can feel free to evaluate our
results by human evaluation or various other automatic evaluation metrics. See more discussions on recommended evaluation metrics in Appendix D.3.
We sincerely thank Prof Rico Senrich for his constructive suggestions on improving the message we present from our experiments to the MT community, and Isaac Caswell at Google Translate for proofreading our paper. We thank Claudia Shi for the insightful discussions on the matching method and suggestions on papers to cite. We thank Shaoshu Yang and Di Jin for supporting us with computational resources. We thank Zhiheng Lyu for helping to code a fast version of the sentence similarity matching.
This material is based in part upon works supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B; the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645; the John Templeton Foundation (grant #61156); a Responsible AI grant by the Haslerstiftung; and an ETH Grant (ETH-19 21-1).
Zhijing Jin is supported by the Open Phil AI Fellowship and the Vitalik Buterin PhD Fellowship from the Future of Life Institute.
Jingwei Ni conducted most of the MT experiments, collected the most updated version of CAUSALMT dataset, and performed various analyses.
Zhijing Jin designed the project, conducted the causal inference experiments, structured the first version of the codes to run the MT experiments, and collected the first version of CAUSALMT dataset which is also used in our previous study (Jin et al., 2021a).
Markus Freitag helped design the storyline of the translationese part, gave insights on what analyses are important, and provided suggestions on the evaluation.
Mrinmaya Sachan and Bernhard Scholkopf guided the project and provided substantial contributions to the storyline of the paper.
Everyone contributed to writing the paper.