SlowGuess/ACL_2020_OCR huggingface dataset:
Setting	Metrics	cs-en	de-en	fi-en	lv-en	ru-en	tr-en	zh-en	Average
m(y*,y)	SENTBLEU	43.5	43.2	57.1	39.3	48.4	53.8	51.2	48.1
CHRF++	52.3	53.4	67.8	52.0	58.8	61.4	59.3	57.9
Baseline with Original Embeddings
MOVER-1 + M-BERT	22.7	37.1	34.8	26.0	26.7	42.5	48.2	34.0
COSINE + LASER	32.6	40.2	41.4	48.3	36.3	42.3	46.7	41.1
Cross-lingual Alignment for Sentence Embedding
COSINE + CLP(LASER)	33.4	40.5	42.0	48.6	36.0	44.7	42.2	41.1
COSINE + UMD(LASER)	36.6	28.1	45.5	48.5	31.3	46.2	49.4	40.8
Cross-lingual Alignment for Word Embedding
MOVER-1 + RCSLS	18.9	26.4	31.9	33.1	25.7	31.1	34.3	28.8
MOVER-1 + CLP(M-BERT)	33.4	38.6	50.8	48.0	33.9	51.6	53.2	44.2
MOVER-2 + CLP(M-BERT)	33.7	38.8	52.2	50.3	35.4	51.0	53.3	45.0
MOVER-1 + UMD(M-BERT)	22.3	38.1	34.5	30.5	31.2	43.5	48.6	35.5
MOVER-2 + UMD(M-BERT)	23.1	38.9	37.1	34.7	33.0	44.8	48.9	37.2
Combining Language Model
COSINE + CLP(LASER) ⊕ LM	48.8	46.7	63.2	66.2	51.0	54.6	48.6	54.2
COSINE + UMD(LASER) ⊕ LM	49.4	46.2	64.7	66.4	51.1	56.0	52.8	55.2
MOVER-2 + CLP(M-BERT) ⊕ LM	46.5	46.4	63.3	63.8	47.6	55.5	53.5	53.8
MOVER-2 + UMD(M-BERT) ⊕ LM	41.8	46.8	60.4	59.8	46.1	53.8	52.4	51.6
Table 1: Pearson correlations with segment-level human judgments on the WMT17 dataset.
Figure 1: Average results of our best-performing metric, together with reference-based BLEU on WMT17.
Figure 1 shows that our metric MOVER-2 + CLP(M-BERT)  $\oplus$  LM, operating on modified M-BERT with the post-hoc re-mapping and combining a target-side LM, outperforms BLEU by 5.7 points in segment-level evaluation and achieves comparable performance in the system-level evaluation. Figure 2 shows that the same metric obtains 15.3 points gains (73.1 vs. 57.8), averaged over 7 languages, on WMT19 (system-level) compared to the state-of-the-art reference-free metric YiSi-2. Except for one language pair, gu-en, our metric performs on a par with the reference-based BLEU (see Table 8 in the Appendix) on system-level.
In Table 1, we exhaustively compare results for several of our metric variants, based either on M-BERT or LASER. We note that re-mapping has
Figure 2: Average results of our metric best-performing metric, together with the official results of reference-free metrics, and reference-based BLEU on system-level WMT19.
considerable effect for M-BERT (up to 10 points improvements), but much less so for LASER. We believe that this is because the underlying embedding space of LASER is less 'misaligned' since it has been (pre-)trained on parallel data.7 While the re-mapping is thus effective for metrics based on M-BERT, we still require the target-side LM to outperform BLEU. We assume the LM can address challenges that the re-mapping apparently is not able to handle properly; see our discussion in §5.1.
Overall, we remark that none of our metric com
However, in the appendix, we find that re-mapping LASER using 2k parallel sentences achieves considerable improvements on low-resource languages, e.g., kk-en (from -61.1 to 49.8) and lt-en (from 68.3 to 75.9); see Table 8.
binations performs consistently best. The reason may be that LASER and M-BERT are pretrained over hundreds of languages with substantial differences in corpora sizes in addition to the different effects of the re-mapping. However, we observe that MOVER-2 + CLP(M-BERT) performs best on average over all language pairs when the LM is not added. When the LM is added, MOVER-2 + CLP(M-BERT) ⊕ LM and COSINE + UMD (LASER) ⊕ LM perform comparably. This indicates that there may be a saturation effect when it comes to the LM or that the LM coefficients should be tuned individually for each semantic similarity metric based on cross-lingual representations.
We first analyze preferences of our metrics based on M-BERT and LASER (§5.1) and then examine how much parallel data we need for re-mapping our vector spaces (§5.2). Finally, we discuss whether it is legitimate to correlate our metric scores, which evaluate the similarity of system predictions and source texts, to human judgments based on system predictions and references (§5.3).
5.1 Metric preferences
To analyze why our metrics based on M-BERT and LASER perform so badly for the task of reference-free MT evaluation, we query them for their preferences. In particular, for a fixed source sentence  $\mathbf{x}$ , we consider two target sentences  $\hat{\mathbf{y}}$  and  $\hat{\mathbf{y}}$  and evaluate the following score difference:
$$
d (\tilde {\mathbf {y}}, \hat {\mathbf {y}}; \mathbf {x}) := m (\mathbf {x}, \tilde {\mathbf {y}}) - m (\mathbf {x}, \hat {\mathbf {y}}) \tag {1}
$$
When  $d > 0$ , then metric  $m$  prefers  $\tilde{\mathbf{y}}$  over  $\hat{\mathbf{y}}$ , given  $\mathbf{x}$ , and when  $d < 0$ , this relationship is reversed. In the following, we compare preferences of our metrics for specifically modified target sentences  $\tilde{\mathbf{y}}$  over the human references  $\mathbf{y}^{\star}$ . We choose  $\tilde{\mathbf{y}}$  to be (i) a random reordering of  $\mathbf{y}^{\star}$ , to ensure that our metrics do not have the BOW (bag-of-words) property, (ii) a word-order preserving translation of  $\mathbf{x}$ , i.e., (ii-a) an expert reordering of the human  $\mathbf{y}^{\star}$  to have the same word order as  $\mathbf{x}$  as well as (ii-b) a word-by-word translation, obtained either using experts or automatically. Especially condition (ii-b) tests for preferences for literal translations, a common MT-system property.
Expert word-by-word translations. We had an expert (one of the co-authors) translate 50 Ger-
man sentences word-by-word into English. Table 2 illustrates this scenario. We note how bad the word-by-word translations sometimes are even for closely related language pairs such as German-English. For example, the word-by-word translations in English retain the original German verb final positions, leading to quite ungrammatical English translations.
Figure 3 shows histograms for the  $d$  statistic for the 50 selected sentences. We first check condition (i) for the 50 sentences. We observe that both MOVER + M-BERT and COSINE+LASER prefer the original human references over random reorderings, indicating that they are not BOW models, a reassuring finding. Concerning (ii-a), they are largely indifferent between correct English word order and the situation where the word order of the human reference is the same as the German. Finally, they strongly prefer the expert word-by-word translations over the human references (ii-b).
Condition (ii-a) in part explains why our metrics prefer expert word-by-word translations the most: for a given source text, these have higher lexical overlap than human references and, by (ii-a), they have a favorable target language syntax, viz., where the source and target language word order are equal. Preference for translationese, (ii-b), in turn is apparently a main reason why our metrics do not perform well, by themselves and without a language model, as reference-free MT evaluation metrics. More worryingly, it indicates that cross-lingual M-BERT and LASER are not robust to the 'adversarial inputs' given by MT systems.
Automatic word-by-word translations. For a large-scale analysis of condition (ii-b) across different language pairs, we resort to automatic word-by-word translations obtained from Google Translate (GT). To do so, we go over each word in the source sentence  $\mathbf{x}$  from left to right, look up its translation in GT independently of context and replace the word by the obtained translation. When a word has several translations, we keep the first one offered by GT. Due to context-independence, the GT word-by-word translations are of much lower quality than the expert word-by-word translations since they often pick the wrong word senses—e.g., the German word sein may either be a personal pronoun (his) or the infinitive to be, which would be selected correctly only by chance; cf. Table 2.
Instead of reporting histograms of  $d$ , we define a "W2W" statistic that counts the relative number of
x	Dieser von Langsamkeit geprügte Lebensstil scheint aber ein Patentrezept für ein hohes Alter zu sein.
y*	However, this slow pace of life seems to be the key to a long life.
y*-random	To pace slow seems be the this life. life to a key however, of long
y*-reordered	This slow pace of life seems however the key to a long life to be.
x'-GT	This from slowness embossed lifestyle seems but on nostrum for on high older to his.
x'-expert	This of slow pace characterized life style seems however a patent recipe for a high age to be.
x	Putin teilte aus und beschuldigte Ankara, Russland in den Rücken gefallen zu sein.
y*	Mr Putin lashed out, accusing Ankara of stabbing Moscow in the back.
y*-random	Moscow accusing lashed Putin the in Ankara out, Mr of back. stabbing
y*-reordered	Mr Putin lashed out, accusing Ankara of Moscow in the back stabbing.
x'-GT	Putin divided out and accused Ankara Russia in the move like to his.
x'-expert	Putin lashed out and accused Ankara, Russia in the back fallen to be.
Figure 3: Histograms of  $d$  scores defined in Eq. (1). Left: Metrics based on LASER and M-BERT favor gold over randomly-shuffled human references. Middle: Metrics are roughly indifferent between gold and reordered human references. Right: Metrics favor expert word-by-word translations over gold human references.
times that  $d(\mathbf{x}',\mathbf{y}^{\star})$  is positive, where  $\mathbf{x}'$  denotes the described literal translation of  $\mathbf{x}$  into the target language:
$$
\mathrm {W} 2 \mathrm {W} := \frac {1}{N} \sum_ {\left(\mathbf {x} ^ {\prime}, \mathbf {y} ^ {\star}\right)} I \left(d \left(\mathbf {x} ^ {\prime}, \mathbf {y} ^ {\star}\right) > 0\right) \tag {2}
$$
Here  $N$  normalizes W2W to lie in  $[0,1]$  and a high W2W score indicates the metric prefers translationese over human-written references. Table 3 shows that reference-free metrics with original embeddings (LASER and M-BERT) either still prefer literal over human translations (e.g., W2W score of  $70.2\%$  for cs-en) or struggle in distinguishing them. Re-mapping helps to a small degree. Only when combined with the LM scores do we get adequate scores for the W2W statistic. Indeed, the LM is expected to capture unnatural word order in the target language and penalize word-by-word translations by recognizing them as much less likely to appear in a language.
Note that for expert word-by-word translations, we would expect the metrics to perform even worse.
Table 2: Original German input sentence  $\mathbf{x}$  ,together with the human reference  ${\mathbf{y}}^{ \star  }$  ,in English,and a randomly  $\left( {{\mathbf{y}}^{ \star  }\text{-random}}\right)$  and expertly reordered  $\left( {{\mathbf{y}}^{ \star  }\text{-reordered}}\right)$  English sentence as well as expert word-by-word translation  $\left( {\mathrm{x}}^{\prime }\right)$  of the German source sentence. The latter is either obtained by the human expert or by Google Translate (GT).
Metrics	cs-en	de-en	fi-en
COSINE + LASER	70.2	65.7	53.9
COSINE + CLP(LASER)	70.7	64.8	53.7
COSINE + UMD(LASER)	67.5	59.5	52.9
COSINE + UMD(LASER) ⊕ LM	7.0	7.1	6.4
MOVER-2 + M-BERT	61.8	50.2	45.9
MOVER-2 + CLP(M-BERT)	44.6	44.5	32.0
MOVER-2 + UMD(M-BERT)	54.5	44.3	39.6
MOVER-2 + CLP(M-BERT) ⊕ LM	7.3	10.2	6.4
Table 3: W2W statistics for selected language pairs. Numbers are in percent.
5.2 Size of Parallel Corpora
Figure 4 compares sentence- and word-level remapping trained with a varying number of parallel sentences. Metrics based on M-BERT result in the highest correlations after re-mapping, even with a small amount of training data (1k). We observe that COSINE + CLP(LASER) and MOVER-2 + CLP(M-BERT) show very similar trends with a sharp increase with increasing amounts of parallel data and then level off quickly. However, the M-BERT based Mover-2 reaches its peak and outperforms the original baseline with only 1k data, while LASER needs 2k before beating the corre
Figure 4: Average results of our metrics based on sentence- and word-based re-mappings of vector spaces as a function of different sizes of parallel corpus (x-axis).
sponding original baseline.
5.3 Human Judgments
The WMT datasets contain segment- and system-level human judgments that we use for evaluating the quality of our reference-free metrics. The segment-level judgments assign one direct assessment (DA) score to each pair of system and human translation, while system-level judgments associate each system with a single DA score averaged across all pairs in the dataset. We initially suspected the DA scores to be biased for our setup—which compares x with y—as they are based on comparing  $\mathbf{y}^{\star}$  and y. Indeed, it is known that (especially) human professional translators "improve"  $\mathbf{y}^{\star}$ , e.g., by making it more readable, relative to the original x (Rabinovich et al., 2017). We investigated the validity of DA scores by collecting human assessments in the cross-lingual settings (CLDA), where annotators directly compare source and translation pairs  $(\mathbf{x},\mathbf{y})$  from the WMT17 dataset. This small-scale manual analysis hints that DA scores are a valid proxy for CLDA. Therefore, we decided to treat them as reliable scores for our setup and evaluate our proposed metrics by comparing their correlation with DA scores.
Existing semantically-motivated metrics for reference-free evaluation of MT systems have so far displayed rather poor correlation with human estimates of translation quality. In this work, we investigate a range of reference-free metrics based on cutting-edge models for inducing cross-lingual semantic representations: cross-lingual (contextualized) word embeddings and cross-lingual
sentence embeddings. We have identified some scenarios in which these metrics fail, prominently their inability to punish literal word-by-word translations (the so-called "translationese"). We have investigated two different mechanisms for mitigating this undesired phenomenon: (1) an additional (weakly-supervised) cross-lingual alignment step, reducing the mismatch between representations of mutual translations, and (2) language modeling (LM) on the target side, which is inherently equipped to punish "unnatural" sentences in the target language. We show that the reference-free coupling of cross-lingual similarity scores with the target-side language model surpasses the reference-based BLEU in segment-level MT evaluation.
We believe our results have two relevant implications. First, they portray the viability of reference-free MT evaluation and warrant wider research efforts in this direction. Second, they indicate that reference-free MT evaluation may be the most challenging ("adversarial") evaluation task for multilingual text encoders as it uncovers some of their shortcomings—prominently, the inability to capture semantically non-sensical word-by-word translations or paraphrases—which remain hidden in their common evaluation scenarios.
We release our metrics under the name XMoverScore publicly: https://github.com/AIPHES/ACL20-Reference-Free-MT-Evaluation.
We thank the anonymous reviewers for their insightful comments and suggestions, which greatly improved the final version of the paper. This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) at the Technische Universität Darmstadt under grant No. GRK 1994/1. The contribution of Goran Glavaš is supported by the Eliteprogramm of the Baden-Württemberg-Stiftung, within the scope of the grant AGREE.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Linguial Evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497-511, San Diego,
California. Association for Computational Linguistics.
Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017a. Results of the WMT17 metrics shared task. In Proceedings of the Conference on Machine Translation (WMT).
Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017b. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pages 489-513, Copenhagen, Denmark. Association for Computational Linguistics.
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In International Conference on Learning Representations.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. 2019. Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748-2760, Florence, Italy. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL.
Sunipa Dev and Jeff M. Phillips. 2019. Attenuating bias in word vectors. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, pages 879-887.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Cooccurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT '02, pages 138-145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Markus Dreyer and Daniel Marcu. 2012. Hyter: Meaning-equivalent semantics for translation evaluation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 162-171. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2019. On the evaluation of machine translation systems trained with back-translation. CoRR, abs/1908.05204.
Marina Fomicheva and Lucia Specia. 2016. Reference bias in monolingual machine translation evaluation. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016-Short Papers, pages 77-82. ACL Home Association for Computational Linguistics.
Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate crosslingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710-721.
Goran Glavas and Ivan Vulic. 2020. Non-linear instance-based cross-lingual mapping for non-isomorphic embedding spaces. In Proceedings of ACL.
Yinuo Guo and Junfeng Hu. 2019. Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 501-506, Florence, Italy. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080.
Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernand a ViA@gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Martin Josifoski, Ivan S Paskov, Hristo S Paskov, Martin Jaggi, and Robert West. 2019. Crosslingual document embedding as reduced-rank ridge regression. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 744-752.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979-2984, Brussels, Belgium. Association for Computational Linguistics.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459-1474, Mumbai, India. The COLING 2012 Organizing Committee.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In  $MT$  summit, volume 5, pages 79-86. Citeseer.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957-966.
Anne Lauscher, Goran Glavas, Simone Paolo Ponzetto, and Ivan Vulic. 2019. A general framework for implicit and explicit debiasing of distributional word vector spaces. arXiv preprint arXiv:1909.06092.
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine