SlowGuess/ACL_2020_OCR huggingface dataset:
for the 2005 iwslt speech translation evaluation. In International Workshop on Spoken Language Translation (IWSLT) 2005.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
Tomer Levinboim and David Chiang. 2015. Multi-task word alignment triangulation for low-resource languages. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1221-1226, Denver, Colorado. Association for Computational Linguistics.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135, Florence, Italy. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2015: Extracting large parallel corpora from movie and tv subtitles. In International Conference on Language Resources and Evaluation.
Patrick Littell, David R Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8-14.
Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark.
Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv:1309.4168.
Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 1659-1666.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In The 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy.
8667
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv:2003.07082.
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In *Meeting of the North American Chapter of the Association for Computational Linguistics* (NAACL), New Orleans, USA.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. *Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia*. arXiv:1907.05791.
Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of the Fifth International Conference on Learning Representations.
Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384-394, Uppsala, Sweden. Association for Computational Linguistics.
Haifeng Wang, Hua Wu, and Zhanyi Liu. 2006. Word alignment for languages with scarce resources using bilingual corpora of other language pairs. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 874-881, Sydney, Australia. Association for Computational Linguistics.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1959-1970.
Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. 2016. Ten pairs to tag-multilingual pos tagging via coarse mapping between embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1307-1317.
Chunting Zhou, Xuezhe Ma, Di Wang, and Graham Neubig. 2019. Density matching for bilingual word embedding. In *Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL)*, Minneapolis, USA.
Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1393-1398.
8668
A Does evaluation directionality matter?
We also explored whether there are significant differences between the evaluated quality of aligned spaces, when computed on both directions (src-trg and trg-src). We find that the evaluation direction indeed matters a lot, when the languages of the evaluation pair are very distant, in terms of morphological complexity and data availability (which affects the quality of the original embeddings). A prominent example, from our Europeanlanguages experiment, are evaluation pairs involving Az or BE. When evaluating on the Az-XX and BE-XX dictionaries, the word translation  $\mathrm{P@1}$  is more than 20 percentage points higher than when evaluating on the opposite direction (XX-Az or XX-BE). For example, Es-Az has a mere  $\mathrm{P@1}$  of 9.9, while Az-Es achieves a  $\mathrm{P@1}$  of 44.9. This observation holds even between very related languages (cf. RU-BE: 12.8, BE-RU: 41.1 and TR-Az: 8.4, Az-Tr: 32.0), which supports our hypothesis that this difference is also due to the quality of the pre-trained embeddings. It is important to note that such directionality differences are not observed when evaluating distant pairs with presumably high-quality pre-trained embeddings e.g. TR-SK or TR-Es; the P@1 for both directions is very close.
B Complete results for all experiments
Here we provide complete evaluation results for our multilingual experiments. Table 7 presents the P@1 of the bilingual experiments using MUSE, and Table 8 presents accuracy using VecMap. Tables 9-14 present P@1, P@5, and P@10 respectively, for the experiment on the 10 European languages. Similarly, results on the distant languages experiment are shown in Tables 15, 16, and 17.
8669
Table 7: BWE results (P@1) with MUSE
Source	Target
Az	BE	Cs	EN	Es	GL	Pt	RU	Sk	Tr
Az	-	4.8	21.4	23.6	32.6	13.6	26.7	10.4	15.0	31.8
BE	4.0	-	26.1	3.8	12.3	9.3	11.3	42.0	23.1	2.9
Cs	2.6	5.4	-	57.1	55.5	11.9	52.3	44.7	71.2	31.6
EN	12.2	2.5	47.3	-	79.3	32.0	72.9	39.7	34.3	40.6
Es	7.8	2.4	45.0	76.7	-	37.1	83.4	38.9	34.3	38.2
GL	2.7	1.8	14.0	38.5	61.2	-	53.3	11.4	12.9	8.5
Pt	2.9	2.3	44.9	72.2	88.7	36.3	-	33.7	33.7	34.6
RU	1.7	12.0	48.6	50.2	49.4	6.6	46.8	-	44.6	21.1
SK	0.3	5.2	71.8	48.0	46.4	9.3	44.4	43.2	-	21.2
TR	10.8	0.3	35.8	48.0	50.9	3.5	45.9	26.9	20.3	-
Source	Target
EN	FR	HI	Ko	RU	Sv	UK
EN	-	80.3	17.9	9.5	39.7	60.0	25.9
FR	76.6	-	11.9	5.1	38.0	52.4	26.8
HI	24.2	17.0	-	0.4	3.1	3.3	2.3
Ko	12.4	7.1	0.4	-	2.5	2.2	0.6
RU	50.2	47.3	3.2	1.6	-	35.8	58.8
Sv	53.3	47.8	5.2	2.3	27.8	-	19.9
UK	37.4	40.3	4.1	0.3	60.7	30.2	-
Table 8: BWE results (P@1) with VecMap
Source	Target
Az	BE	Cs	EN	Es	GL	PT	RU	Sk	Tr
Az	-	15.86	32.43	32.38	37.81	28.48	37.29	26.58	29.38	28.71
BE	14.41	-	35.31	32.74	43.67	30.56	36.58	43.49	30.0	20.77
Cs	6.78	8.65	-	57.45	56.75	35.66	54.09	44.59	73.49	34.75
EN	20.12	18.06	46.41	-	69.91	40.83	63.5	40.13	40.19	37.7
Es	11.66	8.9	45.09	69.49	-	39.37	81.19	40.52	40.7	39.89
GL	5.34	2.44	29.14	46.11	58.44	-	51.64	26.57	28.53	22.47
PT	6.72	6.97	43.48	66.21	85.68	41.17	-	38.29	39.81	36.61
RU	8.06	10.33	52.43	59.03	59.29	29.87	55.55	-	49.93	27.73
SK	2.92	9.26	70.16	56.73	52.35	36.62	50.96	45.47	-	31.23
TR	14.2	9.74	42.37	45.51	50.21	28.06	49.11	32.33	34.57	-
Source	Target
EN	FR	HI	Ko	RU	Sv	UK
EN	-	69.82	35.0	19.2	40.56	56.49	23.63
FR	68.44	-	28.27	15.53	38.18	49.71	26.95
HI	44.61	38.52	-	14.01	20.39	26.26	14.72
Ko	32.69	18.32	12.93	-	11.72	18.45	7.21
RU	59.24	55.18	21.65	10.65	-	47.58	55.12
SV	51.94	46.92	27.46	12.66	34.29	-	26.96
UK	42.61	47.82	17.92	5.21	57.64	43.23	-
Table 9: All results from the European-languages MWE experiment: P@1 (part 1).
Test	Hub language	μ
Az	BE	Cs	EN	Es	GL	Pt	RU	SK	TR
Az-BE	13.7	12.6	14.2	17.2	16.4	13.9	15.0	15.6	14.5	15.8	14.9
Az-Cs	33.7	34.0	32.3	34.5	35.1	34.0	34.8	34.5	32.9	33.7	33.9
Az-EN	31.1	34.7	32.8	32.6	35.7	34.2	33.6	33.6	34.0	33.2	33.5
Az-Es	42.7	46.6	45.2	46.1	44.9	44.4	44.9	43.3	46.1	48.0	45.2
Az-GL	25.9	27.2	29.0	26.5	29.0	24.7	27.2	32.7	31.5	25.9	28.0
Az-PT	37.5	41.5	39.3	41.5	39.8	39.0	39.8	41.5	38.5	40.0	39.8
Az-RU	27.9	27.1	27.1	27.4	27.7	29.0	29.8	26.3	26.3	28.5	27.7
Az-SK	28.8	30.1	31.7	29.1	30.4	30.4	28.8	28.5	29.5	30.4	29.8
Az-Tr	29.8	30.8	32.0	30.1	31.3	30.8	32.0	31.1	32.0	31.8	31.2
BE-Az	10.4	13.3	14.1	13.0	11.9	12.7	12.4	13.0	13.3	13.0	12.7
BE-Cs	30.5	31.6	33.3	33.0	30.8	31.6	32.5	32.2	33.0	35.9	32.5
BE-EN	24.8	26.5	27.8	27.8	28.2	24.8	29.9	28.2	26.5	25.6	27.0
BE-Es	36.4	38.1	36.4	39.5	35.5	38.1	39.0	37.0	36.1	34.4	37.0
BE-GL	24.4	24.4	22.9	24.9	25.8	22.6	24.9	23.5	22.6	24.4	24.0
BE-PT	33.2	33.2	32.7	33.7	34.4	31.7	33.9	31.7	31.9	31.4	32.8
BE-RU	40.9	40.9	40.6	40.3	40.0	41.1	39.1	38.9	39.7	40.0	40.1
BE-SK	30.1	27.7	30.7	27.4	28.6	29.2	28.9	30.7	27.7	27.4	28.8
BE-Tr	17.7	17.2	18.9	19.9	17.4	18.9	20.4	18.7	16.9	18.4	18.5
Cs-Az	3.5	4.6	4.9	6.0	6.9	4.9	3.7	4.9	4.0	6.0	4.9
Cs-BE	8.6	7.8	8.6	8.6	8.8	7.8	8.8	9.3	9.3	8.6	8.6
Cs-EN	59.7	60.5	59.4	59.2	61.0	60.4	60.1	59.7	60.2	58.8	59.9
Cs-Es	59.0	59.1	57.5	60.5	59.2	58.7	58.9	59.6	59.1	57.6	58.9
Cs-GL	27.1	26.9	27.1	27.6	27.0	21.4	27.9	27.1	26.5	26.1	26.5
Cs-PT	56.9	55.6	55.4	57.8	55.5	56.9	55.6	57.3	56.1	54.1	56.1
Cs-RU	44.2	45.5	45.5	45.0	45.5	45.3	45.9	45.0	45.2	45.9	45.3
Cs-SK	69.8	69.8	70.2	71.2	70.6	70.2	70.4	69.7	68.4	70.2	70.0
Cs-Tr	35.3	35.2	34.6	35.1	34.7	34.7	35.1	35.0	35.8	34.2	35.0
EN-AZ	15.8	17.7	16.6	17.5	17.9	16.9	17.5	16.1	16.6	17.2	17.0
EN-BE	16.4	15.1	17.6	14.9	18.4	17.4	15.6	17.1	15.9	16.4	16.5
EN-Cs	49.2	49.0	47.6	47.4	50.2	49.8	50.1	48.3	48.8	49.3	49.0
EN-Es	76.3	77.5	77.2	77.0	76.8	76.5	76.6	77.5	77.3	76.6	76.9
EN-GL	35.0	35.8	36.0	35.2	36.3	31.9	35.9	36.2	35.3	35.0	35.3
EN-PT	71.3	71.8	71.3	72.1	71.5	72.0	71.0	71.5	72.3	71.3	71.6
EN-RU	42.5	43.3	42.7	40.8	43.1	43.3	43.3	41.3	41.4	42.8	42.4
EN-SK	38.7	39.6	40.2	38.0	40.4	39.3	38.5	38.6	36.8	40.4	39.0
EN-TR	40.5	41.7	41.3	41.6	39.4	40.9	41.9	41.0	41.3	40.9	41.0
Es-Az	8.4	10.8	9.0	12.1	10.5	10.5	10.8	9.6	11.8	11.8	10.5
Es-BE	9.9	7.2	8.5	9.3	7.5	9.9	9.9	10.1	9.1	8.8	9.0
Es-Cs	45.3	46.0	44.2	43.4	45.8	45.5	47.4	46.3	45.4	44.7	45.4
Es-EN	73.0	74.5	73.8	73.2	74.0	74.1	73.1	73.5	74.6	73.6	73.7
Es-GL	37.1	37.0	37.1	36.9	37.5	33.7	36.8	37.0	36.8	36.7	36.7
Es-PT	82.1	82.9	82.7	83.0	83.1	83.1	82.5	83.0	82.9	83.0	82.8
Es-RU	41.4	41.5	41.2	39.4	41.3	41.9	40.9	40.3	40.2	41.9	41.0
Es-SK	37.0	39.2	38.8	37.4	40.0	39.2	39.5	39.5	35.2	38.8	38.5
Es-TR	37.5	38.0	37.7	38.2	37.6	37.8	38.4	37.8	38.6	37.9	38.0
8671
Table 10: All results from the European-languages MWE experiment: P@1 (part 2).
Test	Hub language	μ
Az	BE	Cs	EN	Es	GL	Pt	RU	SK	TR
GL-Az	4.0	4.6	4.3	5.5	5.0	4.1	5.2	4.7	4.8	5.0	4.7
GL-BE	3.6	3.0	2.4	3.0	3.0	2.4	3.0	2.4	1.2	3.0	2.7
GL-Cs	23.2	25.7	25.0	23.8	26.5	23.0	25.6	25.4	25.6	26.5	25.0
GL-EN	40.3	41.8	41.9	39.6	43.2	40.8	41.5	41.9	41.6	42.1	41.5
GL-Es	60.0	60.5	60.1	59.9	60.4	59.0	60.0	60.3	59.6	60.8	60.1
GL-PT	52.5	52.5	52.9	52.0	52.0	50.4	52.5	51.9	52.1	52.0	52.1
GL-RU	22.5	22.7	22.9	21.7	23.3	21.9	23.7	22.7	22.5	23.8	22.8
GL-SK	26.0	26.3	26.8	25.6	26.4	23.4	25.5	25.1	23.2	26.4	25.5
GL-Tr	18.5	19.3	19.7	18.6	17.8	18.3	18.9	19.2	19.4	17.6	18.7
PT-AZ	3.8	4.7	5.8	5.0	5.0	3.2	5.8	5.0	5.5	4.7	4.8
PT-BE	7.3	5.3	7.3	7.3	6.1	7.1	6.8	6.1	8.6	7.1	6.9
PT-Cs	45.5	47.0	46.3	45.0	45.5	47.2	45.5	46.7	46.5	45.6	46.1
PT-EN	69.9	70.9	70.2	71.3	71.1	70.5	70.6	71.3	70.6	70.8	70.7
PT-Es	87.4	88.1	87.7	87.6	88.0	87.4	88.1	87.8	87.6	88.1	87.8
PT-GL	35.7	36.9	36.3	36.3	37.1	32.7	36.0	35.9	35.2	36.4	35.8
PT-RU	37.4	37.7	36.4	36.5	38.0	38.0	36.2	37.0	37.1	37.4	37.2
PT-SK	37.6	37.0	37.3	36.7	38.7	37.7	38.3	37.9	33.6	38.0	37.3
PT-TR	36.5	37.4	37.2	38.1	35.9	36.4	35.5	37.2	36.2	36.3	36.7
RU-AZ	5.0	6.4	6.2	7.8	8.7	7.3	7.5	7.3	6.7	7.5	7.0
RU-BE	12.8	9.9	10.7	11.5	11.2	11.0	11.5	12.3	11.0	11.8	11.4
RU-Cs	49.2	50.0	49.2	50.1	49.7	50.3	50.3	49.8	50.1	50.1	49.9
RU-EN	53.6	53.8	54.4	52.7	54.7	55.5	54.8	52.0	54.5	55.5	54.1
RU-Es	53.7	53.4	54.8	54.5	52.3	53.5	54.0	53.2	53.9	51.2	53.4
RU-GL	20.9	21.3	22.1	22.3	22.9	17.2	23.0	21.8	21.7	21.9	21.5
RU-PT	50.4	50.3	50.4	52.4	51.1	51.1	49.6	49.8	51.0	47.6	50.4
RU-SK	45.0	44.7	44.7	45.2	45.2	44.7	44.3	43.7	43.7	45.5	44.7
RU-TR	25.9	27.0	26.2	26.9	26.0	25.9	26.1	25.6	26.8	24.7	26.1
SK-AZ	2.8	4.0	1.5	3.7	2.1	2.8	3.4	3.1	1.8	3.4	2.9
SK-BE	10.2	7.5	9.9	9.4	9.6	8.3	10.4	10.9	10.9	9.1	9.6
SK-Cs	71.4	72.5	70.9	70.8	70.5	71.1	71.3	70.6	71.0	71.4	71.1
SK-EN	54.8	55.0	54.0	52.9	55.4	54.7	54.8	54.6	53.0	55.6	54.5
SK-Es	52.5	51.6	52.2	53.9	52.3	52.0	50.4	50.5	51.5	51.1	51.8
SK-GL	27.0	27.3	27.2	28.4	27.8	20.6	26.2	26.0	27.0	27.0	26.4
SK-PT	49.3	50.3	48.2	50.4	52.0	49.2	49.1	48.7	48.5	47.7	49.3
SK-RU	43.8	43.4	43.5	43.2	43.7	44.0	42.8	42.9	41.2	43.4	43.2
SK-TR	28.2	27.5	27.2	28.5	27.1	26.1	26.2	27.6	27.4	26.0	27.2
TR-AZ	9.8	12.1	10.1	11.1	10.1	11.4	11.4	10.8	12.1	11.1	11.0
TR-BE	9.0	4.8	8.7	8.1	7.8	7.5	8.1	6.9	7.5	7.2	7.6
TR-Cs	40.3	41.6	40.3	41.6	41.6	40.8	41.6	41.8	40.9	39.2	41.0
TR-EN	51.1	49.3	51.1	50.2	50.4	48.5	50.5	50.2	50.7	50.1	50.2
TR-Es	53.8	53.6	55.0	55.0	52.5	53.0	54.6	52.9	54.1	53.3	53.8
TR-GL	17.0	17.3	17.3	15.9	16.8	11.6	17.5	17.1	17.1	18.4	16.6
TR-PT	50.1	50.1	51.4	51.6	49.3	48.9	48.7	49.9	50.5	49.5	50.0
TR-RU	34.0	34.3	32.3	34.6	34.3	33.6	33.2	32.0	33.0	32.9	33.4
TR-SK	27.5	29.2	27.9	28.5	29.4	27.7	27.9	27.5	25.2	27.9	27.9
8672
Table 11: All results from the European-languages MWE experiment: P@5 (part 1).
Test	Hub language	μ
Az	BE	Cs	EN	Es	GL	Pt	RU	SK	TR
Az-BE	26.0	22.5	26.5	26.0	26.5	25.2	25.7	26.0	25.7	25.7	25.6
Az-Cs	53.4	54.8	53.7	57.5	54.8	55.9	55.6	54.5	53.2	54.8	54.8
Az-EN	44.7	48.0	47.6	45.7	45.9	47.4	46.8	46.3	46.1	47.2	46.6
Az-Es	60.1	62.6	60.7	62.6	60.7	60.4	60.7	62.4	61.8	62.9	61.5
Az-GL	38.3	37.7	40.1	41.4	38.9	35.8	40.1	41.4	38.9	39.5	39.2
Az-PT	52.8	55.3	55.3	56.3	55.8	55.3	55.8	57.8	55.3	56.8	55.7
Az-RU	45.2	46.5	46.8	48.1	49.2	47.3	48.4	45.5	46.8	50.0	47.4
Az-SK	43.9	46.1	47.0	48.3	49.2	48.3	49.2	48.3	46.7	46.7	47.4
Az-Tr	45.2	49.1	51.3	49.1	46.7	48.7	49.1	49.4	49.6	49.4	48.8
BE-Az	20.6	20.6	23.4	23.2	24.6	22.0	22.9	24.9	22.3	24.6	22.9
BE-Cs	44.5	44.8	47.6	48.5	46.5	47.9	48.7	46.8	45.7	47.9	46.9
BE-EN	42.3	42.3	42.7	41.5	44.4	42.7	42.3	42.7	41.0	43.2	42.5
BE-Es	50.4	53.0	54.2	53.3	50.4	53.6	54.4	51.0	54.2	52.4	52.7
BE-GL	38.8	36.5	37.7	38.8	38.0	36.5	38.3	38.0	38.6	37.7	37.9
BE-PT	49.5	50.8	52.8	51.5	52.0	50.0	49.0	49.0	50.5	49.5	50.5
BE-RU	53.0	53.2	52.1	51.8	53.8	52.7	53.0	53.0	53.2	51.8	52.8
BE-SK	43.8	40.1	44.7	43.5	41.6	43.8	44.4	43.5	40.1	43.5	42.9
BE-Tr	33.4	33.2	34.6	37.8	32.2	34.4	36.9	33.4	33.2	32.2	34.1
Cs-Az	10.3	11.2	11.2	13.8	14.1	11.8	12.1	10.6	11.2	12.6	11.9
Cs-BE	14.8	15.5	15.5	16.3	16.3	16.6	16.1	16.1	14.8	15.8	15.8
Cs-EN	75.6	76.4	75.1	75.7	76.2	76.9	76.1	75.8	75.9	76.0	76.0
Cs-Es	75.5	75.3	74.1	76.5	75.9	74.9	74.3	75.5	75.9	74.1	75.2
Cs-GL	40.8	41.8	43.0	43.7	43.1	36.5	42.1	42.6	42.1	41.2	41.7
Cs-PT	72.9	74.1	72.2	74.3	73.1	73.7	72.7	73.8	72.7	71.6	73.1
Cs-RU	64.5	64.4	63.6	63.9	63.9	64.5	64.9	64.5	64.3	65.5	64.4
Cs-SK	81.7	82.9	83.2	82.8	82.5	83.0	83.2	82.7	81.6	82.7	82.6
Cs-Tr	56.2	56.0	55.1	57.1	56.4	54.2	54.9	55.5	54.9	53.8	55.4
EN-AZ	28.3	29.1	30.3	29.9	28.9	29.2	30.2	29.1	28.8	30.6	29.4
EN-BE	32.8	28.3	34.0	31.5	34.0	34.5	30.3	32.8	33.3	32.8	32.4
EN-Cs	74.7	74.9	73.4	74.5	76.1	76.5	74.8	75.1	73.8	75.5	74.9
EN-Es	88.9	89.5	88.8	89.3	89.1	89.3	89.1	89.3	89.0	89.1	89.1
EN-GL	49.0	50.4	50.5	50.4	51.3	47.8	50.9	51.4	49.1	50.7	50.1
EN-PT	86.0	86.6	86.2	86.6	86.2	86.4	86.3	86.3	86.4	85.8	86.3
EN-RU	68.0	68.1	68.2	66.0	68.6	69.6	68.7	67.7	67.4	68.2	68.1
EN-SK	62.3	62.7	62.5	60.8	62.5	62.1	63.5	62.7	59.9	63.2	62.2
EN-TR	63.6	62.6	64.3	62.4	62.4	63.8	63.8	63.0	63.2	63.2	63.2
Es-AZ	16.3	16.9	16.9	17.5	18.4	17.8	17.2	17.2	19.0	18.1	17.5
Es-BE	16.8	15.5	17.1	18.9	16.3	18.9	18.7	17.1	18.1	16.5	17.4
Es-Cs	64.4	65.7	63.5	65.2	66.1	65.5	65.9	66.0	65.8	65.9	65.4
Es-EN	85.2	86.3	86.0	85.5	85.8	85.5	85.8	86.1	86.0	86.0	85.8
Es-GL	45.6	46.0	45.7	46.1	46.4	43.2	45.9	45.7	45.8	46.2	45.7
Es-PT	90.8	91.1	90.7	91.3	91.4	91.1	91.3	90.7	90.9	90.9	91.0
Es-RU	61.5	62.5	61.4	62.5	62.1	61.7	62.2	60.8	61.6	62.9	61.9
Es-SK	57.9	59.1	58.7	58.5	59.1	57.8	58.1	57.6	57.0	58.5	58.2
Es-TR	57.0	57.4	57.2	56.7	55.0	56.3	56.3	55.5	56.6	56.5	56.5
8673
Table 12: All results from the European-languages MWE experiment: P@5 (part 2).
Test	Hub language	μ
Az	BE	Cs	EN	Es	GL	Pt	RU	SK	TR
GL-Az	8.4	9.0	8.8	9.8	9.6	10.0	9.7	9.4	9.2	9.7	9.4
GL-BE	7.3	6.1	6.1	6.7	6.7	6.7	7.9	6.1	6.1	7.3	6.7
GL-Cs	41.8	42.1	43.0	42.3	44.5	40.2	42.5	42.5	42.0	43.0	42.4
GL-EN	56.8	57.4	58.6	56.3	59.7	57.6	57.2	57.8	56.7	58.1	57.6
GL-Es	68.3	68.8	68.1	68.8	68.6	67.9	68.3	68.8	68.2	68.8	68.5
GL-PT	63.9	64.3	63.4	64.1	63.2	62.8	63.4	64.0	63.7	63.9	63.7
GL-RU	40.2	39.8	39.3	39.6	39.5	37.0	40.0	39.5	39.3	40.8	39.5
GL-SK	41.6	42.4	41.1	41.9	43.7	38.5	41.0	41.4	39.2	41.5	41.2
GL-Tr	33.5	33.4	34.9	33.9	33.3	29.4	32.4	32.6	34.0	31.5	32.9
Pt-AZ	8.7	11.1	10.2	12.5	11.1	10.2	10.5	9.9	12.0	11.1	10.7
Pt-BE	14.4	12.1	14.4	17.4	14.1	15.9	14.9	14.9	14.9	14.6	14.8
Pt-Cs	65.6	66.6	64.7	65.8	66.5	66.6	65.9	66.3	65.5	65.1	65.9
Pt-EN	81.3	82.1	82.0	82.1	81.9	82.0	81.5	81.7	81.5	82.0	81.8
Pt-Es	92.1	92.6	92.4	92.1	92.0	91.8	92.4	92.4	92.0	92.3	92.2
Pt-GL	45.4	46.4	46.2	46.9	46.8	43.5	45.8	45.4	45.2	46.7	45.8
Pt-RU	57.6	57.8	57.7	58.7	58.1	58.5	57.0	57.5	57.6	57.6	57.8
Pt-SK	57.2	56.9	57.0	57.8	56.6	55.4	56.6	56.8	53.1	56.4	56.4
Pt-TR	53.9	54.8	54.2	56.3	53.3	53.6	52.7	54.5	54.4	54.6	54.2
RU-AZ	12.0	15.6	15.9	15.6	15.9	14.8	15.4	14.2	14.2	15.9	15.0
RU-BE	20.1	18.3	20.6	20.1	20.9	20.6	20.6	20.9	21.1	20.4	20.4
RU-Cs	65.7	65.0	65.1	64.7	65.0	66.7	66.1	65.8	65.1	65.5	65.5
RU-EN	72.8	73.0	73.9	72.0	73.8	73.5	72.7	72.3	72.9	73.5	73.0
RU-Es	70.1	69.8	69.7	71.3	69.2	70.3	71.2	68.8	70.7	68.4	69.9
RU-GL	36.1	35.9	36.1	36.8	37.1	30.9	36.5	36.6	35.9	35.3	35.7
RU-PT	66.8	66.8	67.0	69.3	67.9	67.6	65.8	66.6	67.3	65.2	67.0
RU-SK	61.1	62.6	61.4	61.1	62.0	61.8	61.8	60.9	59.8	61.6	61.4
RU-TR	48.0	48.0	47.6	49.9	47.1	47.5	48.0	46.0	47.0	47.4	47.7
SK-AZ	7.7	9.2	7.1	9.5	7.4	8.3	8.9	8.9	8.3	8.6	8.4
SK-BE	17.4	16.7	18.5	18.2	17.7	18.5	18.2	19.3	19.3	18.5	18.2
SK-Cs	82.1	82.1	81.3	81.6	82.1	82.4	81.6	81.6	81.3	81.9	81.8
SK-EN	70.7	71.7	71.3	69.6	71.2	71.4	71.5	70.9	70.3	71.4	71.0
SK-Es	69.2	69.7	70.2	71.2	70.1	68.8	70.0	68.6	69.2	69.4	69.6
SK-GL	43.4	43.3	42.9	45.1	43.7	36.0	42.9	42.0	43.0	42.7	42.5
SK-PT	68.2	67.5	67.5	68.7	69.9	67.6	66.1	67.6	66.7	66.7	67.7
SK-RU	59.2	58.1	58.2	58.8	59.4	59.5	58.8	58.5	57.5	59.5	58.8
SK-TR	47.2	48.7	47.6	48.7	47.1	46.7	48.2	47.8	46.7	46.2	47.5
TR-AZ	19.5	22.2	19.9	21.2	20.9	20.9	20.5	19.5	21.9	20.2	20.7
TR-BE	17.1	12.3	16.2	17.1	16.8	15.6	16.5	16.5	16.2	16.2	16.1
TR-Cs	61.6	62.1	60.1	61.8	62.4	61.9	61.6	61.5	61.4	60.1	61.4
TR-EN	68.0	68.2	68.1	67.2	67.8	67.5	69.6	67.7	67.9	67.2	67.9
TR-Es	69.8	69.0	70.4	70.5	68.0	69.2	70.5	69.4	69.8	69.5	69.6
TR-GL	30.5	30.7	31.1	30.0	30.4	23.6	31.4	31.1	29.7	30.7	29.9
TR-PT	67.1	66.9	66.9	67.9	66.5	65.9	65.2	67.1	67.5	66.6	66.8
TR-RU	55.4	55.9	54.0	55.4	55.3	55.1	55.1	53.0	52.9	53.5	54.6
TR-SK	48.2	49.9	48.9	49.7	48.7	47.8	48.9	48.1	44.2	47.7	48.2
8674
Table 13: All results from the European-languages MWE experiment: P@10 (part 1).
Test	Hub language	μ
Az	BE	Cs	EN	Es	GL	Pt	RU	SK	TR
Az-BE	31.1	27.1	30.8	31.4	31.9	31.1	29.8	30.3	32.2	31.1	30.7
Az-Cs	60.3	62.5	60.8	62.7	63.6	61.4	62.7	61.1	60.3	63.6	61.9
Az-EN	49.3	51.1	52.6	50.5	49.5	50.7	51.4	50.3	50.1	50.7	50.6
Az-Es	63.8	65.7	65.4	67.1	65.2	66.3	68.0	64.6	66.6	67.4	66.0
Az-GL	42.6	42.6	45.1	45.1	43.8	39.5	45.1	43.8	42.6	43.8	43.4
Az-PT	58.5	61.2	62.7	62.5	61.5	61.7	61.0	61.2	61.7	62.5	61.5
Az-RU	50.8	52.7	52.9	50.8	54.0	53.2	54.3	51.6	51.9	54.5	52.7
Az-SK	48.9	52.0	53.0	52.0	53.9	54.2	53.0	52.4	51.7	51.7	52.3
Az-Tr	53.3	55.5	56.7	57.0	55.0	55.3	55.7	56.5	57.0	56.7	55.9
BE-Az	25.7	25.4	29.7	28.5	29.4	26.8	27.7	28.2	26.8	28.0	27.6
BE-Cs	50.7	51.0	52.1	51.3	51.8	53.8	52.7	51.8	50.7	51.8	51.8
BE-EN	46.6	48.7	50.0	46.2	48.3	50.9	46.2	48.3	46.2	47.9	47.9
BE-Es	54.7	57.3	58.7	58.7	56.2	57.9	57.9	55.9	58.5	57.9	57.4
BE-GL	47.0	45.2	44.6	46.1	43.8	41.4	43.5	43.8	44.3	42.9	44.3
BE-PT	55.3	55.8	57.0	57.8	57.0	56.5	55.8	54.5	55.5	56.0	56.1
BE-RU	56.3	56.3	56.1	56.1	56.9	56.1	56.3	56.3	56.9	55.5	56.3
BE-SK	48.0	45.6	48.3	47.7	48.0	48.6	49.8	48.6	46.2	48.0	47.9
BE-Tr	38.3	40.5	41.5	43.2	40.3	40.3	41.8	41.5	40.3	38.3	40.6
Cs-Az	13.8	14.9	15.5	16.1	17.5	14.9	15.8	14.1	14.9	15.5	15.3
Cs-BE	18.9	17.9	19.2	19.9	19.4	19.9	19.9	19.2	17.9	19.2	19.1
Cs-EN	80.2	80.5	79.8	80.0	80.1	81.0	80.2	80.5	80.5	81.1	80.4
Cs-Es	80.1	79.6	78.8	80.0	79.9	79.4	79.9	79.3	80.2	79.0	79.6
Cs-GL	47.2	48.0	47.9	49.9	49.3	42.4	48.2	48.3	49.1	47.1	47.7
Cs-PT	77.5	78.7	77.5	78.3	77.1	77.7	76.9	77.7	76.9	76.8	77.5
Cs-RU	70.1	70.3	69.1	69.6	69.4	70.7	69.6	69.5	69.5	70.5	69.8
Cs-SK	85.5	85.6	85.7	85.2	84.9	85.1	86.2	85.2	84.9	85.6	85.4
Cs-Tr	63.2	62.7	62.5	63.5	62.7	62.5	62.7	63.4	62.6	61.6	62.7
EN-AZ	32.2	33.3	34.3	34.3	33.8	32.5	34.4	33.0	34.3	33.8	33.6
EN-BE	38.5	34.0	40.4	39.0	40.0	41.2	38.7	38.2	38.7	38.5	38.7
EN-Cs	81.2	81.1	79.9	80.7	81.9	82.5	80.6	80.7	80.7	81.5	81.1
EN-Es	91.3	92.1	91.7	91.5	91.9	91.7	91.8	91.6	91.9	91.7	91.7
EN-GL	53.9	56.3	56.4	55.7	55.8	53.2	55.9	56.2	54.9	55.5	55.4
EN-PT	89.4	90.0	89.2	89.5	89.1	89.5	89.3	89.0	89.4	89.0	89.3
EN-RU	74.6	74.0	75.8	72.2	74.8	76.0	74.8	73.8	74.0	74.4	74.4
EN-SK	69.3	69.7	69.9	68.0	69.6	68.7	69.9	69.5	67.1	69.9	69.2
EN-Tr	69.9	70.1	71.0	69.3	69.5	69.8	70.3	71.1	70.0	69.2	70.0
Es-AZ	20.2	20.8	20.2	21.1	20.8	20.2	19.3	20.2	21.1	21.1	20.5
Es-BE	20.8	18.9	20.8	22.9	21.3	22.4	21.1	23.2	21.3	21.3	21.4
Es-Cs	70.5	70.7	70.8	70.9	71.0	71.1	71.3	71.8	72.2	70.9	71.1
Es-EN	88.5	88.4	88.5	88.3	88.5	88.5	88.5	88.5	88.5	88.4	88.5
Es-GL	49.5	49.4	49.4	49.8	50.0	46.0	49.6	49.6	49.4	50.2	49.3
Es-PT	92.7	92.5	92.5	92.5	93.0	92.9	92.8	92.4	92.1	92.7	92.6
Es-RU	67.5	67.1	67.4	68.9	67.4	67.6	67.8	66.8	68.7	68.5	67.8
Es-SK	64.5	64.3	63.9	65.4	65.4	63.5	64.3	64.8	63.0	63.8	64.3
Es-Tr	63.6	63.8	64.3	62.7	61.6	62.6	63.7	62.2	63.8	61.7	63.0
8675