SlowGuess/NAACL_2022_OCR huggingface dataset:
Kwabena Amponsah-Kaakyire, Daria Pylypenko, Cristina Espana-Bonet, and Josef van Genabith. 2021. Do not rely on relay translations: Multilingual parallel direct Europarl. In Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age, pages 1-7, online. Association for Computational Linguistics.
Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. In Text and Technology. John Benjamins.
Mona Baker. 1995. Corpora in translation studies: An overview and some suggestions for future research. *Target*. International Journal of Translation Studies, 7(2):223-243.
Mona Baker. 1996. Corpus-based translation studies: The challenges that lie ahead. *Benjamins Translation Library*, 18:175-186.
Marco Baroni and Silvia Bernardini. 2006. A new approach to the study of translationese: Machine-learning the difference between original and translated text. Literary and Linguistic Computing, 21(3):259-274.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc."
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2001. Latent dirichlet allocation. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], pages 601-608. MIT Press.
Nikolay Bogoychev and Rico Sennrich. 2019. Domain, translationese and noise in synthetic data for neural machine translation. CoRR, abs/1911.03362.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272-303, Belgium, Brussels. Association for Computational Linguistics.
Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 330-336, Edinburgh, Scotland. Association for Computational Linguistics.
Franck Burlot and François Yvon. 2018. Using monolingual data in neural machine translation: a systematic study. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 144-155, Brussels, Belgium. Association for Computational Linguistics.
Efim A Dinic. 1970. Algorithm for solution of a problem of maximum flow in networks with power estimation. In Soviet Math. Doklady, volume 11, pages 1277-1280.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2836-2846, Online. Association for Computational Linguistics.
Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 34-44, Florence, Italy. Association for Computational Linguistics.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61-71, Online. Association for Computational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondrej Bojar. 2021. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733-774, Online. Association for Computational Linguistics.
Markus Freitag, David Vilar, David Grangier, Colin Cherry, and George Foster. 2022. A natural diet: Towards improving naturalness of machine translation output. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3340-3353, Dublin, Ireland. Association for Computational Linguistics.
Martin Gellerstam. 1986. Translationese in swedish novels translated from english. Translation studies in Scandinavia, 1:88-95.
Clark Glymour, Kun Zhang, and Peter Spirtes. 2019. Review of causal discovery methods based on graphical models. Frontiers in Genetics, 10:524.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 72-81, Online. Association for Computational Linguistics.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. CoRR, abs/1909.13788.
David Heckerman, Christopher Meek, and Gregory Cooper. 1999. A bayesian approach to causal discovery. Computation, causation, and discovery, 19:141-166.
Paul W Holland. 1988. Causal inference, path analysis and recursive structural equations models. ETS Research Report Series, 1988(1):i-50.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Biwei Huang, Kun Zhang, Yizhu Lin, Bernhard Schölkopf, and Clark Glymour. 2018. Generalized score functions for causal discovery. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pages 1551-1560. ACM.
Stefano M Iacus, Gary King, and Giuseppe Porro. 2012. Causal inference without balance checking: Coarsened exact matching. Political analysis, 20(1):1-24.
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. IMaT: Unsupervised text attribute transfer via iterative matching and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3097-3109, Hong Kong, China. Association for Computational Linguistics.
Zhijing Jin, Zeyu Peng, Tejas Vaidhya, Bernhard Schoelkopf, and Rada Mihalcea. 2021a. Mining the cause of political decision-making from social media: A case study of COVID-19 policies across the US states. In *Findings of the 2021 Conference on Empirical Methods in Natural Language Processing*, EMNLP 2021. Association for Computational Linguistics.
Zhijing Jin, Julius von Kugelgen, Jingwei Ni, Tejas Vaidhya, Ayush Kaushal, Mrinmaya Sachan, and Bernhard Schoelkopf. 2021b. Causal direction of data collection matters: Implications of causal and anticausal learning for NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9499-9513, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478-494, Online. Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86, Phuket, Thailand.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Vassilis Kolias, Ioannis Anagnostopoulos, and Eleftherios Kayafas. 2014. Exploratory analysis of a terabyte scale web corpus. CoRR, abs/1409.5443.
Moshe Koppel and Noam Ordan. 2011. *Translationese and its dialects*. In *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*, pages 1318–1326, Portland, Oregon, USA. Association for Computational Linguistics.
David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proceedings of Machine Translation Summit XII: Papers, Ottawa, Canada.
Samuel Larkin, Michel Simard, and Rebecca Knowles. 2021. Like chalk and cheese? On the effects of translationese in MT training. In Proceedings of Machine Translation Summit XVIII: Research Track, pages 103-113, Virtual. Association for Machine Translation in the Americas.
Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? A case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics.
Sara Laviosa-Braithwaite. 1998. Universals of translation. Routledge encyclopedia of translation studies. London: Routledge, pages 288-291.
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Language models for machine translation: Original vs. translated texts. Computational Linguistics, 38(4):799-825.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314-319, Florence, Italy. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Sylvania Ozdowska and Andy Way. 2009. Optimal bilingual data for French-English PB-SMT. In Proceedings of the 13th Annual conference of the European Association for Machine Translation, Barcelona, Spain. European Association for Machine Translation.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Judea Pearl. 1993. [Bayesian analysis in expert systems]: Comment: Graphical models, causality and intervention. Statistical Science, 8(3):266-269.
Judea Pearl. 1995. Causal diagrams for empirical research. Biometrika, 82(4):669-688.
Judea Pearl. 2009. Causality: Models, reasoning, and inference. Cambridge University Press.
Jonas Peters, Dominik Janzing, and Bernhard Scholkopf. 2017. Elements of causal inference: Foundations and learning algorithms. The MIT Press.
Alberto Poncelas, Dimitar Sht. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating backtranslation in neural machine translation. CoRR, abs/1804.06189.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419-432.
Hans Reichenbach. 1956. The direction of time, volume 65. University of California Press.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, Online. Association for Computational Linguistics.
Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020. *Translationese as a language in "multilingual" NMT*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7737-7746, Online. Association for Computational Linguistics.
Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen. 2020. Adjusting for confounding with text matching. American Journal of Political Science, 64(4):887-903.
Paul R Rosenbaum and Donald B Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55.
Bernhard Scholkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris M. Mooij. 2012. On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012. icml.cc / Omni-press.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Ilia Sominsky and Shuly Wintner. 2019. Automatic detection of translation direction. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1131-1140, Varna, Bulgaria. INCOMA Ltd.
Pater Spirtes, Clark Glymour, and Richard Scheines. 2000a. Constructing bayesian network models of gene expression networks from microarray data. In Proceedings of the Atlantic Symposium on Computational Biology.
Peter Spirtes, Clark Glymour, and Richard Scheines. 2000b. Causation, Prediction, and Search, Second Edition. Adaptive computation and machine learning. MIT Press.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? Reassessing claims of human parity in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113-123, Brussels, Belgium. Association for Computational Linguistics.
Gideon Toury. 1980. In search of a theory of translation. Porter Institute for Poetics and Semiotics, Tel Aviv University.
Gideon Toury. 1995. Descriptive translation studies and beyond, volume 4. John Benjamins.
Michael Ustaszewski. 2019. Optimising the Europarl corpus for translation studies with the EuroparlExtract toolkit. Perspectives, 27(1):107-123.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Victor Veitch, Alexander D'Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021. Counterfactual invariance to spurious correlations in text classification. Advances in Neural Information Processing Systems, 34.
Victor Veitch, Dhanya Sridhar, and David M. Blei. 2020. Adapting text embeddings for causal inference. In Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020, volume 124 of Proceedings of Machine Learning Research, pages 919-928. AUAI Press.
Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digit. Scholarsh. Humanit., 30(1):98-118.
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine
Translation (Volume 1: Research Papers), pages 73-81, Florence, Italy. Association for Computational Linguistics.
A Reproducibility, License, and Copyright
We open-source our codes and datasets, which are both uploaded to the submission system. In our data, we include all three variations: the full CAUSALMT dataset, the split used for the semi-supervised learning experiments, and the subset after matching the contents and sentence lengths. In our codes, we include all commands with hyperparameters to help future work to reproduce our results.
The codes and data are under MIT license. Note that the Europarl dataset has no copyright restriction, according to its official website.[6]
B Linguistic Property Analysis
We also open-source the codes to calculate the linguistic properties of our dataset in Table 1. We use the Python library Stanza $^{7}$  (Qi et al., 2020) tockenize the sentences when calculating the number of sentences per sample. For computational efficiency, we use NLTK $^{8}$  (Bird et al., 2009) tockenize the words and count the vocabulary.
We use the Python library spaCy $^9$  (Honnibal and Montani, 2017) to calculate the punctuation per sample. We use a passive voice checker $^{10}$  (only available in English). For the expansion factor, we formatted Table 1 using the ratio of the two languages in a descending alphabetical order of each language pair. In our table, it happens to be the ratio of the more verbose language to the less verbose language in each language pair.
To prepare the text for the models, we follow the preprocessing scripts of fairseq (Ott et al., 2019).<sup>11</sup> Specifically, we use the Moses tokenizer (Koehn et al., 2007),<sup>12</sup> the default byte pair encoding (BPE) size of 40K subwords, and remove sentence pairs that of larger than 1.5 length ratio from the training set.
C.2 Evaluation Script
We use the fairseq script<sup>13</sup> to calculate the BLEU score (Papineni et al., 2002) of each translation model, with a beam width of 5, BPE removed, and detokenized by moses.
We use the sequence-to-sequence Transformer model (Vaswani et al., 2017) implemented by the faireq library (Ott et al., 2019). Specifically, we use a six-layer Transformer, a label smoothing of 0.1, a weight decay of 0.0001, a dropout of 0.3, 4000 warming updates, and a learning rate of 0.0005. All results are reported by a single run but a fixed random seed.
For the semi-supervised learning, we implement the BT model following Edunov et al. (2020) to use the Facebook-FAIR system of the WMT'19 news shared translation task. $^{14}$  All the hyperparameters are the same as the supervised system, with a learning rate of 0.0007 on both the supervised training data and the generated pseudo-parallel corpus. We implement the ST model by He et al. (2019) following their script, $^{15}$  and also keep the hyperparameters the same as the supervised model.
We train the supervised learning model and each step in the semi-supervised learning scripts for 1000 epochs. We select the model with the best performance on the development set and report the final evaluation results on the test set.
All experiments are run on NVIDIA RTX2080 GPUs. Each supervised learning experiment takes around 32 GPU hours, and each semi-supervised learning experiment takes about 128 GPU hours.
D.1 Effect of Train-Test Direction Match on Supervised Learning
To inspect the influence of train-test direction match on the MT performance, we conduct all experiments on our CAUSALMT test sets and also the standard newstest2014 test sets. For the supervised learning performance, we list the performance on
<sup>6</sup>https://www.statmt.org/europarl/
<sup>7</sup>https://stanfordnlp.github.io/stanza/
<sup>8</sup>https://www.nltk.org/
9https://spacy.io/
10https://github.com/armsp/active_or Passive
11https://github.com/pytorch/fairseq/
$^{12}$ https://github.com/moses-smt/mosesdecoder/ blob/master/scriptstokenizertokenizer.perl
$^{13}$ https://github.com/pytorch/fairseq/blob/main/fairseq-cli/generate.py
$^{14}$ https://github.com/pytorch/fairseq/tree/main/ examples/backtranslation
$^{15}$ https://github.com/jxhe/self-training-text-generation/blob/master/self_train.sh
de-to-en Translation	en-to-de Translation
α% T1 (de, en*)	T2 (de*, en)	α% T1 (en, de*)	T2 (en*, de)
0%	14.21	19.10	0%	11.18	15.49
25%	15.71	18.69	25%	12.69	14.29
50%	16.77	18.17	50%	13.30	14.33
75%	16.91	16.27	75%	13.38	13.16
100%	16.02	12.91	100%	13.28	10.68
en-to-fr Translation	fr-to-en Translation
α% T1 (en, fr*)	T2 (en*, fr)	α% T1 (fr, en*)	T2 (fr*, en)
0%	16.61	21.33	0%	16.34	23.26
25%	18.56	20.95	25%	18.81	23.31
50%	20.45	21.66	50%	19.75	23.20
75%	21.19	21.05	75%	21.09	22.01
100%	21.43	19.30	100%	20.02	19.78
Table 6: Effect of train-test direction match on the en-fr and en-de test sets of newtest2014.
the CAUSALMT test sets in the main paper in Table 2, and list the additional performance on the newestest2014 test sets in Table 6.
For better visualization of the trends, we also provide line plots of the same experimental results in Table 2. Specifically, we plot the results of German-English translation in Figure 4a using our previous experiment results in Table 2. We also include the diagram of all five language pairs in Figure 4b.
In Figure 4a, we use lines with the same darkness of color for the same model trained on different data directions. Results show that the data-model direction match matters significantly. Taking the German-to-English translation models (--- and —), the two data directions can cause up to 4.53 difference in BLEU points. In the current figures, we also see that the data direction with a smaller expansion factor is a better training corpus than the other one.
We use the same line type (dashed or solid) for models trained on the same data. Using the same data, the performance of the two different directions of models cannot be compared directly because the target language is different, causing the BLEU calculation to be different.
D.2 Effect of Train-Test Direction Match on Semi-Supervised Learning
For the semi-supervised learning performance, we show the performance on the newstest2014 test sets in Table 3 in the main paper, and performance on the test sets of CAUSALMT in Table 7. Note that the decrease of ST performance on En-Es and Es-Fr pairs is possible because ST is more sensitive to the quality of the model learned on the supervised data, and these language pairs have a smaller training
Mixture Rate  $\alpha (\%)$  .Train Set  $= \alpha \% X\stackrel {\mathrm{H}}{\rightarrow}Y + (1 - \alpha \%)X\stackrel {\mathrm{H}}{\leftarrow}Y.$