SlowGuess/NAACL_2022_OCR huggingface dataset:
(a) Translation performance between German and English on different mixtures of training sets combining  $\alpha \% X\stackrel {\mathrm{H}}{\longrightarrow}Y$  data and  $(1 - \alpha \%)$ $X\stackrel {\mathrm{H}}{\leftarrow}Y$  data, where  $\alpha = 0,25,50,75,100$  Note that there are four settings between German and English, by varying two different data origins.  $(X\stackrel {\mathrm{H}}{\rightarrow}Y$  data  $=$  De  $\stackrel {\mathrm{H}}{\rightarrow}\mathrm{En}$  or De  $\stackrel {\mathrm{H}}{\leftarrow}\mathrm{En})$  and two different translation task directions (German-to-English (De-En) translation or English-toGerman (En-De) translation).
(b) Translation performance between all five language pairs on different mixtures of training sets combining  $\alpha \% X\stackrel {\mathrm{H}}{\longrightarrow}Y$  data and  $(1 - \alpha \%)$ $X\stackrel {\mathrm{H}}{\leftarrow}Y$  data, where the mixture rate  $\alpha = 0,25,50,75,100$
data size of  $90\mathrm{K}$  compared with  $200\mathrm{K}+$  data for all the other language pairs.
D.3 Non-BLEU Evaluation Metrics
Due to financial constraints, we did not use human evaluation in addition to the BLEU score in our study. However, we encourage future studies in this line of work to include human evaluation results, as evaluation is important given the nature of such work, and human evaluation is reported to be more reflective of the real translation quality (Edunov et al., 2020).
To make it convenient for follow-up work, we open-source all the outputs generated by our model
German-to-English (de-to-en) Translation
Test 1 (de, en*)	Test 2 (de*, en)
Sup. on Equal Mix	25.52	29.02
+ ST (de, en**)	+3.25 (Aligned)	+2.59
+ BT (de**, en)	+0.97	+2.67 (Aligned)
English-to-German (en-to-de) Translation
Test 1 (en, de*)	Test 2 (en*, de)
Sup. on Equal Mix	23.76	21.48
+ ST (en, de**)	+1.39 (Aligned)	+1.44
+ BT (en**, de)	-0.63	+0.51 (Aligned)
German-to-French (de-to-fr) Translation
Test 1 (de, fr*)	Test 2 (de*, fr)
Sup. on Equal Mix	25.42	30.10
+ ST (de, fr**)	+1.79 (Aligned)	+1.23
+ BT (de**, fr)	+0.35	+1.69 (Aligned)
French-to-German (fr-to-de) Translation
Test 1 (fr, de*)	Test 2 (fr*, de)
Sup. on Equal Mix	21.89	18.60
+ ST (fr, de**)	+2.46 (Aligned)	+2.09
+ BT (fr**, de)	+1.07	+0.87 (Aligned)
English-to-French (en-to-fr) Translation
Test 1 (en, fr*)	Test 2 (en*, fr)
Sup. on Equal Mix	35.64	36.19
+ ST (en, fr**)	+2.04 (Aligned)	+1.89
+ BT (en**, fr)	+0.13	+1.33 (Aligned)
French-to-Engligh (fr-to-en) Translation
Test 1 (fr, en*)	Test 2 (fr*, en)
Sup. on Equal Mix	34.35	33.75
+ ST (fr, en**)	+1.76 (Aligned)	+2.28
+ BT (fr**, en)	+0.43	+1.89 (Aligned)
English-to-Spanish (en-to-es) Translation
Test 1 (en, es*)	Test 2 (en*, es)
Sup. on Equal Mix	33.65	34.01
+ ST (en, es**)	-0.10 (Aligned)	-0.75
+ BT (en**, es)	+0.36	+1.04 (Aligned)
Spanish-to-English (es-to-en) Translation
Test 1 (es, en*)	Test 2 (es*, en)
Sup. on Equal Mix	35.00	33.82
+ ST (es, en**)	-0.46 (Aligned)	-0.41
+ BT (es**, en)	+0.63	+1.77 (Aligned)
Spanish-to-French (es-to-fr) Translation
Test 1 (es, fr*)	Test 2 (es*, fr)
Sup. on Equal Mix	38.30	40.40
+ ST (es, fr**)	+0.58 (Aligned)	+0.83
+ BT (es**, fr)	+1.00	+2.01 (Aligned)
French-to-Spanish (fr-to-es) Translation
Test 1 (fr, es*)	Test 2 (fr*, es)
Sup. on Equal Mix	40.61	38.55
+ ST (fr, es**)	-0.84 (Aligned)	-1.09
+ BT (fr**, es)	-0.14	+0.12 (Aligned)
to our GitHub. We also encourage future work to adopt more evaluation metrics such as COMET and BLUERT, which are among the metrics that correlates the best with human judgements according to the metric task at WMT Freitag et al. (2021). COMET, for instance, correlates better with human judgements than BLEU (Kocmi et al., 2021) and goes further string matching.
E Implementation Details for Causal Inference
E.1 Statistics of the Matched Corpora
We list the statistics of the matched corpora in Table 8, and analyze its linguistic properties in Ta
Table 7: Performance analogous to Table 3 but on our CAUSALMT test sets.
Human Trans. Dir.	Train	Dev	Test
De H→En	107K	1K	2K
En H→De	107K	1K	2K
De H→Fr	133K	1K	2K
Fr H→De	133K	1K	2K
En H→Fr	87K	1K	2K
Fr H→En	87K	1K	2K
En H→Es	47K	1K	2K
Es H→En	47K	1K	2K
Es H→Fr	50K	1K	2K
Fr H→Es	50K	1K	2K
Table 8: Dataset statistics for five language pairs after matching. Each language pair has data from two human translation directions (Human Trans. Dir.), e.g.,  $\mathrm{De}\xrightarrow{\mathrm{H}}\mathrm{En}$  and  $\mathrm{De}\xleftarrow{\mathrm{H}}\mathrm{En}$ .
ble 9.
E.2 Confirming the Causal Graph by Causal Discovery
To check our causal graph assumption, we first verify whether data-model direction match is a cause for MT performance using causal discovery.
We use the causal discovery algorithm, fast causal inference (FCI) (Spirtes et al., 2000a), to verify that the data-model direction match causally affects the translation performance, conditioned on other factors such as the sentence length and topics.
FCI is the most appropriate causal inference method for this analysis since there might exist hidden confounders that affect the MT performance, which normal causal discovery methods such as score-based methods (Heckerman et al., 1999; Huang et al., 2018) and other constraint-based algorithms like Peter-Clark (PC) algorithm (Spirtes et al., 2000b, §5.4.2, pp. 84-88) cannot handle (Glymour et al., 2019). FCI gives asymptotically correct results in the presence of confounders, and outputs Markov equivalence classes, i.e., a set of causal structures satisfying the same conditional independences.
Given a language pair  $X$  and  $Y$ , we generate eight sets of experiment results, by varying the two training directions, two test directions, and two model directions. We extract the test samples of all eight experiments, and since each test set is 2K, there are 16K samples in total. On the 16K samples, besides keeping the label of their data-model direction match, translation performance in BLEU, we also calculate the other factors such as the test-model direction match, train-test direction
DeH→En	DeH→En	DeH→Fr	DeH→Fr	EnH→Fr	EnH→Fr	EnH→Es	EnH→Es	EsH→Fr	EsH→Fr
# Words/Sample	21.05/23.82	22.64/23.82	23.81/30.09	24.39/29.19	25.40/29.76	25.33/28.02	26.69/28.18	26.96/27.14	30.59/34.33	30.39/33.45
# Sents/Sample	1.032/1.031	1.041/1.020	1.041/1.925	1.056/1.902	1.033/1.949	1.040/1.872	1.028/1.053	1.057/1.070	1.076/2.100	1.068/2.088
Sent Expansion Factor	en:de=1.00	en:de=0.98	fr:de=1.85	fr:de=1.80	fr:en=1.88	fr:en=1.80	es:en=1.02	es:en=1.01	fr:es=1.95	fr:es=1.96
Passive Voice (%)	-0/1.128	-0/1.036	-/-	-/-	0.1073/-	0.1185/-	0.1155/-	0.1256/-	-/-	-/-
# Punctuation/Sample	3.04/2.83	3.04/2.63	3.45/6.04	3.35/6.43	2.82/5.89	3.11/6.16	2.93/2.71	3.12/3.07	3.42/7.02	3.50/7.44
# Syllables/Word	2.002/1.744	2.059/1.755	1.988/1.553	2.068/1.546	1.758/1.562	1.780/1.550	1.760/2.022	1.780/2.010	2.010/1.567	2.030/1.544
Flesch Reading Ease	31.90/35.22	29.30/33.78	35.25/46.30	28.0/46.1	31.93/45.80	31.09/49.91	30.55/50.22	30.05/51.94	48.82/43.26	46.87/42.04
MATTR	58.93/52.68	60.58/53.31	59.32/52.77	61.74/52.89	53.19/52.55	53.38/52.32	53.90/54.91	52.33/53.90	53.60/51.85	54.84/52.29
Lexical Density	49.15/49.24	50.75/49.91	48.86/55.30	50.82/55.21	49.99/55.18	50.28/55.16	50.24/49.76	49.56/48.88	48.72/55.02	50.01/55.14
Vocab Size	58K/22K	56K/23K	78K/37K	71K/39K	22K/31K	21K/31K	19K/31K	16K/29K	32K/26K	34K/29K
Table 9: Detailed characteristics of the matched dataset.
match, source sentence length, and the topic vector by topic modeling on all the training data of the language pair  $X$  and  $Y$ . We run the FCI algorithm using the causal-learn Python package $^{16}$  over all the variables of interest. The implementation details are in the Appendix.
The resulting causal graph on the German-English language pair is in Figure 2. The results confirm our hypothesis that the data-model direction match (causal vs. anticausal direction) does have a causal effect on the BLEU score, together with other factors such as the sentence length and topics.
$^{16}$ https://github.com/cmu-phil/causal-learn