SlowGuess/ACL_2020_OCR huggingface dataset:
Translation, pages 228-231. Association for Computational Linguistics.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150-157.
Chi-kiu Lo. 2019. YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507-513, Florence, Italy. Association for Computational Linguistics.
Chi-kiu Lo, Meriem Beloucif, Markus Saers, and Dekai Wu. 2014. XMEANT: Better semantic MT evaluation without reference translations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 765-771, Baltimore, Maryland. Association for Computational Linguistics.
Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018a. Results of the WMT18 metrics shared task. In Proceedings of the Third Conference on Machine Translation (WMT).
Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018b. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671-688, Belgium, Brussels. Association for Computational Linguistics.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Florence, Italy". Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799-2808, Florence, Italy. Association for Computational Linguistics.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013a. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on
Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 3111-3119.
Bari Saiful M Mohiuddin, Tasnim and Shafiq Joty. 2020. Lnmap: Departures from isomorphic assumption in bilingual lexicon induction through non-linear mapping in latent space. CoRR, abs/1309.4168.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 311-318, Stroudsburg, PA, USA. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL).
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Maja Popovic. 2012. Morpheme- and POS-based IBM1 and language model scores for translation quality estimation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 133-137, Montréal, Canada. Association for Computational Linguistics.
Maja Popović. 2017.  $\mathrm{chrF}++$ : Words Helping Character N-grams. In Proceedings of the Second Conference on Machine Translation, pages 612-618, Copenhagen, Denmark.
Maja Popovic, David Vilar, Eleftherios Avramidis, and Aljoscha Burchardt. 2011. Evaluation without references: IBM1 scores as evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 99-103, Edinburgh, Scotland. Association for Computational Linguistics.
Ella Rabinovich, Noam Ordan, and Shuly Wintner. 2017. Found in translation: Reconstructing phylogenetic language trees from translations. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 530-540, Vancouver, Canada. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper.pdf.
Andreas Rückle, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. 2018. Concatenated power mean word embeddings as universal cross-lingual sentence representations. arXiv.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389. Association for Computational Linguistics.
Peter H Schonemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1599–1613, Minneapolis, Minnesota. Association for Computational Linguistics.
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. RUSE: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference on Machine Translation (WMT).
Anders Søgaard, Sebastian Ruder, and Ivan Vulic. 2018. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778-788, Melbourne, Australia. Association for Computational Linguistics.
Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. *Machine translation*, 24(1):39-50.
Lucia Specia, Kashif Shah, Jose G.C. de Souza, and Trevor Cohn. 2013. QuEst - a translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 79-84, Sofia, Bulgaria. Association for Computational Linguistics.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1171-1181. Association for Computational Linguistics.
Ivan Vulic, Goran Glavaš, Roi Reichart, and Anna Korhonen. 2019. Do we really need fully unsupervised cross-lingual embeddings? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4398-4409.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307.
Elizaveta Yankovskaya, Andre Tattar, and Mark Fishel. 2019. Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 101-105, Florence, Italy. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China. Association for Computational Linguistics.
A.1 Zero-shot Transfer to Resource-lean Language
Our metric allows for estimating translation quality on new domains. However, the evaluation is limited to those languages covered by multilingual embeddings. This is a major drawback for low-resource languages—e.g., Gujarati is not included in LASER. To this end, we take multilingual USE (Yang et al., 2019) as an illustrating example which covers only 16 languages (in our sample Czech, Latvian and Finish are not included in USE). We re-align the corresponding embedding spaces with our re-mapping functions to induce evaluation metrics even for these languages, using only 2k translation pairs. Table 4 shows that our metric with a composition of re-mapping functions can raise correlation from zero to 0.10 for cs-en and to 0.18 for lv-en. However, for one language pair, fi-en, we see correlation goes from negative to zero, indicating that this approach does not always work. This observation warrants further investigation.
Metrics	cs-en	fi-en	lv-en
BLEU	0.849	0.834	0.946
COSINE + LAS	-0.001	-0.149	0.019
COSINE + CLP(USE)	0.072	-0.068	0.109
COSINE + UMD(USE)	0.056	-0.061	0.113
COSINE + CLP ∘ UMD(USE)	0.089	-0.030	0.162
COSINE + UMD ∘ CLP(USE)	0.102	-0.007	0.180
Table 4: The Pearson correlation of merics on segment-level WMT17. 'o' marks the composition of two remapping functions.
Setting	Metrics	cs-en	de-en	fi-en	lv-en	ru-en	tr-en	zh-en	Average
m(y*,y)	BLEU	0.971	0.923	0.903	0.979	0.912	0.976	0.864	0.933
CHRF++	0.940	0.965	0.927	0.973	0.945	0.960	0.880	0.941
Baseline with Original Embeddings
MOVER-1 + M-BERT	0.408	0.905	0.570	0.571	0.855	0.576	0.816	0.672
COSINE + LASER	0.821	0.821	0.744	0.754	0.895	0.890	0.676	0.800
Cross-lingual Alignment for Sentence Embedding
COSINE + CLP(LASER)	0.824	0.830	0.760	0.766	0.900	0.942	0.757	0.826
COSINE + UMD(LASER)	0.833	0.858	0.735	0.754	0.909	0.870	0.630	0.798
Cross-lingual Alignment for Word Embedding
MOVER-1 + RCSLS	-0.693	-0.053	0.738	0.251	0.538	0.380	0.439	0.229
MOVER-1 + CLP(M-BERT)	0.796	0.960	0.879	0.874	0.894	0.864	0.898	0.881
MOVER-2 + CLP(M-BERT)	0.818	0.971	0.885	0.887	0.878	0.893	0.896	0.890
MOVER-1 + UMD(M-BERT)	0.610	0.956	0.526	0.599	0.906	0.538	0.898	0.719
MOVER-2 + UMD(M-BERT)	0.650	0.973	0.574	0.649	0.888	0.634	0.901	0.753
Combining Language Model
COSINE + CLP(LASER) ⊕ LM	0.986	0.909	0.868	0.968	0.858	0.910	0.800	0.900
COSINE + UMD(LASER) ⊕ LM	0.984	0.904	0.861	0.968	0.850	0.922	0.817	0.901
MOVER-2 + CLP(M-BERT) ⊕ LM	0.977	0.923	0.873	0.944	0.863	0.880	0.803	0.895
MOVER-2 + UMD(M-BERT) ⊕ LM	0.968	0.934	0.832	0.951	0.871	0.862	0.821	0.891
Table 5: Pearson correlations with system-level human judgments on the WMT17 dataset.
Setting	Metrics	cs-en	de-en	et-en	fi-en	ru-en	tr-en	zh-en	Average
m(y*,y)	SENTBLEU	0.233	0.415	0.285	0.154	0.228	0.145	0.178	0.234
Y1S1-1	0.319	0.488	0.351	0.231	0.300	0.234	0.211	0.305
m(x,y)	Baseline with Original Embeddings
MOVER-1 + M-BERT	0.005	0.229	0.179	0.115	0.100	0.039	0.082	0.107
COSINE + LASER	0.072	0.317	0.254	0.155	0.102	0.086	0.064	0.150
Cross-lingual Alignment for Word Embedding
COSINE + CLP(LASER)	0.093	0.323	0.254	0.151	0.112	0.086	0.074	0.156
COSINE + UMD(LASER)	0.077	0.317	0.252	0.145	0.136	0.083	0.053	0.152
COSINE + UMD ○ CLP(LASER)	0.090	0.337	0.255	0.139	0.145	0.090	0.088	0.163
COSINE + CLP ○ UMD(LASER)	0.096	0.331	0.254	0.153	0.122	0.084	0.076	0.159
Cross-lingual Alignment for Sentence Embedding
MOVER-1 + CLP(M-BERT)	0.084	0.279	0.207	0.147	0.145	0.089	0.122	0.153
MOVER-2 + CLP(M-BERT)	0.063	0.283	0.193	0.149	0.136	0.069	0.115	0.144
MOVER-1 + UMD(M-BERT)	0.043	0.264	0.193	0.136	0.138	0.051	0.113	0.134
MOVER-2 + UMD(M-BERT)	0.040	0.268	0.188	0.143	0.141	0.055	0.111	0.135
MOVER-1 + UMD ○ CLP(M-BERT)	0.024	0.282	0.192	0.144	0.133	0.085	0.089	0.136
MOVER-1 + CLP ○ UMD(M-BERT)	0.073	0.277	0.208	0.148	0.142	0.086	0.121	0.151
MOVER-2 + CLP ○ UMD(M-BERT)	0.057	0.283	0.194	0.149	0.137	0.069	0.114	0.143
Combining Language Model
COSINE + UMD ○ CLP(LASER) ⊕ LM	0.288	0.455	0.226	0.321	0.263	0.159	0.192	0.272
COSINE + CLP ○ UMD(LASER) ⊕ LM	0.283	0.457	0.228	0.321	0.265	0.150	0.198	0.272
MOVER-1 + CLP ○ UMD(M-BERT) ⊕ LM	0.268	0.428	0.292	0.213	0.261	0.152	0.192	0.258
MOVER-2 + CLP ○ UMD(M-BERT) ⊕ LM	0.254	0.426	0.285	0.203	0.251	0.146	0.193	0.251
Table 6: Kendall correlations with segment-level human judgments on the WMT18 dataset.
Setting	Metrics	cs-en	de-en	et-en	fi-en	ru-en	tr-en	zh-en	Average
m(y*,y)	BLEU	0.970	0.971	0.986	0.973	0.979	0.657	0.978	0.931
METEOR++	0.945	0.991	0.978	0.971	0.995	0.864	0.962	0.958
m(x,y)	Baseline with Original Embeddings
MOVER-1 + M-BERT	-0.629	0.915	0.880	0.804	0.847	0.731	0.677	0.604
COSINE + LASER	-0.348	0.932	0.930	0.906	0.902	0.832	0.471	0.661
Cross-lingual Alignment for Sentence Embedding
COSINE + CLP(LASER)	-0.305	0.934	0.937	0.908	0.904	0.801	0.634	0.688
COSINE + UMD(LASER)	-0.241	0.944	0.933	0.906	0.902	0.842	0.359	0.664
COSINE + UMD o CLP(LASER)	0.195	0.955	0.958	0.913	0.896	0.899	0.784	0.800
COSINE + CLP o UMD(LASER)	-0.252	0.942	0.941	0.908	0.919	0.811	0.642	0.702
Cross-lingual Alignment for Word Embedding
MOVER-1 + CLP(M-BERT)	-0.163	0.943	0.918	0.941	0.915	0.628	0.875	0.722
MOVER-2 + CLP(M-BERT)	-0.517	0.944	0.909	0.938	0.913	0.526	0.868	0.654
MOVER-1 + UMD(M-BERT)	-0.380	0.927	0.897	0.886	0.919	0.679	0.855	0.683
MOVER-2 + UMD(M-BERT)	-0.679	0.929	0.891	0.896	0.920	0.616	0.858	0.633
MOVER-1 + UMD o CLP(M-BERT)	-0.348	0.949	0.905	0.890	0.905	0.636	0.776	0.673
MOVER-1 + CLP o UMD(M-BERT)	-0.205	0.943	0.916	0.938	0.913	0.641	0.871	0.717
MOVER-2 + CLP o UMD(M-BERT)	-0.555	0.944	0.908	0.935	0.911	0.551	0.863	0.651
Combining Language Model
COSINE + UMD o CLP(LASER) ⊕ LM	0.979	0.967	0.979	0.947	0.942	0.673	0.954	0.919
COSINE + CLP o UMD(LASER) ⊕ LM	0.974	0.966	0.983	0.951	0.951	0.255	0.961	0.863
MOVER-1 + CLP o UMD(M-BERT) ⊕ LM	0.956	0.960	0.949	0.973	0.951	0.097	0.954	0.834
MOVER-2 + CLP o UMD(M-BERT) ⊕ LM	0.959	0.961	0.947	0.979	0.951	-0.036	0.952	0.815
Table 7: Pearson correlations with system-level human judgments on the WMT18 dataset.
Setting	Metrics	Direct Assessment
de-en	fi-en	gu-en	kk-en	lt-en	ru-en	zh-en	Average
m(y*, y)	BLEU	0.849	0.982	0.834	0.946	0.961	0.879	0.899	0.907
m(x, y)	Existing Reference-free Metrics
IBM1-MORPHEME(Popović, 2012)	0.345	0.740	-	-	0.487	-	-	-
IBM1-POS4GRAM(Popović, 2012)	0.339	-	-	-	-	-	-	-
LASIM(Yankovskaya et al., 2019)	0.247	-	-	-	-	0.310	-	-
LP(Yankovskaya et al., 2019)	0.474	-	-	-	-	0.488	-	-
YiSI-2(Lo, 2019)	0.796	0.642	0.566	0.324	0.442	0.339	0.940	0.578
YiSI-2-SRL(Lo, 2019)	0.804	-	-	-	-	-	0.947	-
Baseline with Original Embeddings
MOVER-1 + M-BERT	0.358	0.611	-0.396	0.335	0.559	0.261	0.880	0.373
COSINE + LASER	0.217	0.891	-0.745	-0.611	0.683	-0.303	0.842	0.139
Our Cross-lingual based Metrics
MOVER-2 + CLP(M-BERT)	0.625	0.890	-0.060	0.993	0.851	0.928	0.968	0.742
COSINE + CLP(LASER)	0.225	0.894	0.041	0.150	0.696	-0.184	0.845	0.381
COSINE + UMD o CLP(LASER)	0.074	0.835	-0.633	0.498	0.759	-0.201	0.610	0.277
Our Cross-lingual based Metrics ⊕ LM
COSINE + CLP(LASER) ⊕ LM	0.813	0.910	-0.070	-0.735	0.931	0.630	0.711	0.456
COSINE + UMD(LASER) ⊕ LM	0.817	0.908	-0.383	-0.902	0.929	0.573	0.781	0.389
MOVER-2 + CLP(M-BERT) ⊕ LM	0.848	0.907	-0.068	0.775	0.963	0.866	0.827	0.731
MOVER-2 + UMD(M-BERT) ⊕ LM	0.859	0.914	-0.181	-0.391	0.970	0.702	0.874	0.535
Table 8: Pearson correlations with system-level human judgments on the WMT19 dataset. ’-’ marks the numbers not officially reported in (Ma et al., 2019).