SlowGuess/NAACL_2022_OCR huggingface dataset:
Kwabena Amponsah-Kaakyire, Daria Pylypenko, Cristina Espana-Bonet, and Josef van Genabith. 2021. Do not rely on relay translations: Multilingual parallel direct Europarl. In Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age, pages 1-7, online. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674-7684, Online. Association for Computational Linguistics.
Mona Baker, Gill Francis, and Elena Tognini-Bonelli. 1993. Corpus linguistics and translation studies: Implications and applications. In Text and Technology: In Honour of John Sinclair, page 233-, Netherlands. John Benjamins Publishing Company.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Advances in Neural Information Processing Systems, 29:4349-4357.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. Transactions of the Association for Computational Linguistics, 8:454-470.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Sunipa Dev and Jeff Phillips. 2019. Attenuating bias in word vectors. In Proceedings of the Twenty-Second
International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 879-887. PMLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Koel Dutta Chowdhury, Cristina España-Bonet, and Josef van Genabith. 2020. Understanding translationese in multi-view embedding spaces. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6056-6062.
Koel Dutta Chowdhury, Cristina Espana-Bonet, and Josef van Genabith. 2021. Tracing source language interference in translation with graph-isomorphism measures. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 375-385.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2836-2846, Online. Association for Computational Linguistics.
Martin Gellerstam. 1986. Translationese in Swedish novels translated from English. Translation studies in Scandinavia, 1:88-95.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 609-614, Minneapolis, Minnesota. Association for Computational Linguistics.
Hila Gonen, Ganesh Jawahar, Djamé Seddah, and Yoav Goldberg. 2020. Simple, interpretable and stable method for detecting words with usage change across corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 538-555, Online. Association for Computational Linguistics.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 72-81, Online. Association for Computational Linguistics.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Moshe Koppel and Noam Ordan. 2011. *Translationese and its dialects*. In *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*, pages 1318–1326, Portland, Oregon, USA. Association for Computational Linguistics.
Ira Leviant and Roi Reichart. 2015. Separated by an un-common language: Towards judgment language informed vector space modeling. arXiv preprint arXiv:1508.00106.
Sheng Liang, Philipp Dufter, and Hinrich Schütze. 2020. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5082-5093, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314-319, Florence, Italy. Association for Computational Linguistics.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020. Translationese as a language in "multilingual" NMT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7737-7746, Online. Association for Computational Linguistics.
Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019. Xlda: Cross-lingual data augmentation for natural language inference and question answering. arXiv preprint arXiv:1905.11471.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113-123, Brussels, Belgium. Association for Computational Linguistics.
Ivan Vulic, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibean, Roi Reichart, and Anna Korhonen. 2020. Multi-SimLex: A large-scale evaluation of multilingual and crosslingual lexical semantic similarity. Computational Linguistics, 46(4):847-897.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 73-81, Florence, Italy. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3143-3155, Online. Association for Computational Linguistics.
A.1 Experimental Setup and Hyperparameters
Each run in Table 1 took around 1.5 hours on a GTX1080Ti GPU. Each classification and debi- asing step for SNLI sentence-level experiment in Table 3 took approximately 2 hours on V100-32GB GPU. Other hyperparameter settings are shown in the Table 4.
Model	Hyperparameters
fastText	minCount=5, dim=300
Logistic Regression	warm_start = True, penalty = '12', verbose=5, solver="saga", random_state=23, max_iter=7
BiLSTM	hid_dim=300, dropout=0.2, batch_size=32, Adam optimizer with lr=0.0001, epochs=15
SNLI-sentence	nclfs=45, max_iter=1500
SNLI-Aligned (word)	nclfs=34
INLP(single,word=human
SNLI-Joint (word)	nclfs=35
A.2 Debiased Word Representations
A.2.1 Word Analogy Tests
To verify that debiasing does not hurt the quality of the word representations, we estimate the performance of the original and debiased embeddings on the word analogy task using the MultiSIMLEX benchmark (Vulic et al., 2020). As MultiSIMLEX does not cover German, we use German-Simlex from Leviant and Reichart (2015). After debiasing, Spearman's  $\rho$  correlation coefficients show negligible decreases of 0.02 on En-De and En-Es, 0.01 on Es-En and Es-De, 0.3 on De-En and an increase of 0.01 for De-Es.
A.2.2 Visualisation
Figure 1 shows the t-SNE (Van der Maaten and Hinton, 2008) projection of the vectors in Direct Joint Space Projection before and after debiasing with INLP.
Table 4: Hyperparameter settings.
De-En	De-Es	En-De	En-Es	Es-De	Es-En
466	510	429	483	547	504
Table 5: Size of translationese word lists created with the usage change algorithm (Gonen et al., 2020).
Direct Joint Space Projection at  $t = 0$
Direct Joint Space Projection at  $t = 35$
Figure 1: Clustering on Direct Joint Space, before (upside) and after (down-side) debiasing.
A.3 Translationese Word Lists
The size of the translationese word lists created via the usage change algorithm of (Gonen et al., 2020) is shown in Table 5 and our top-50 elements per language are shown in Table 6.
rank	De-En	De-Es	En-De	En-Es	Es-De	Es-En
1	hin	weise	each	hand	ahí	-
2	heißt	art	while	down	”	”
3	zumindest	gilt	hand	mind	-	modo
4	hohe	hinaus	upon	labour	supuesto	duda
5	dessen	sowohl	direct	while	terreno	supuesto
6	dabei	interesse	mean	due	igual	algún
7	außerdem	bleibt	secondly	form	“	bajo
8	weise	tat	light	example	incluso	pues
9	bleibt	außerdem	mind	others	luego	general
10	völlig	gesamte	beyond	another	sí	igual
11	derzeit	beim	takes	high	largo	cual
12	sorge	ebenso	kind	each	allá	luego
13	sowohl	beispiel	comes	food	origen	sí
14	daran	kommt	practice	throughout	)	tipó
15	darin	letztendlich	capacity	whole	bajo	cabo
16	art	dabei	full	true	orden	alto
17	seite	davon	itself	call	precisamente	tercer
18	ihrem	sorge	open	single	corresponde	único
19	beispielsweise	heißt	sort	air	solamente	resulta
20	erster	linie	share	capacity	interés	fuera
21	macht	weder	labour	organisation	general	incluso
22	beispiel	völlig	large	power	idea	línea
23	beim	zwar	words	practice	tercer	p Pública
24	bzw.	erster	short	become	objeto	largo
25	ebenso	genau	individual	takes	p Pública	siempre
26	gilt	darin	nor	without	saben	alghuna
27	bedeutet	selbst	value	nature	donde	)
28	hinaus	dessen	sense	reality	alto	terminos
29	voll	diejenigen	allow	comes	da	donde
30	grund	jedem	least	doubt	algún	orden
31	gesamte	macht	side	euro	cara	asimismo
32	unser	ihrem	behind	close	línea	riesgo
33	jedem	zumindest	currently	words	todà	ningún
34	stehen	eindeutig	longer	far	asimismo	“
35	tatsache	sei	doubt	course	misma	mIENTras
36	kommen	seite	whole	circumstances	propio	vista
37	eindeutig	sorgen	down	currently	(	claro
38	interesse	innerhalb	effect	non	través	efecto
39	tatsächlich	form	become	yet	fuera	práctica
40	all	handelt	outside	long	duda	régimen
41	ganze	gesamten	indeed	itself	cual	ella
42	hohen	ebene	board	least	central	misma
43	weder	jeder	free	every	alghuna	pódico
44	aller	teil	needs	fisheries	resulta	carácer
45	gleichzeitig	beispielsweise	without	interest	claro	plan
46	gesamten	liegt	rural	general	junto	defensa
47	allein	weit	throughout	main	tipó	dentro
48	ort	schließlich	across	mean	demás	lado
49	innerhalb	übrigen	close	free	falta	número
50	jeder	reihe	another	term	siempre	personal
Table 6: Top-50 translationese words as obtained by the application of the use change concept for the three languages L1-L2 (L1 being En, Es and De) when they are translated from the other two languages L2.