SlowGuess/NAACL_2021_OCR huggingface dataset:
From Table 2, we can observe the Pearson correlation results of our metrics by comparing the source sentences with machine translated sentences using both single metrics and their combinations in the Multi-30K dataset. The result reveals that SSS + WMD outperforms all individual metrics and the other combined metrics. It is clear that SSS is better than both WMD and BERTScore, with WMD outperforming BERTScore in this specific crosslingual task.
In Table 3, the benefit of SSS becomes even more evident. It again outperforms WMD and BERTScore, with BERTScore also significantly outperforming WMD in this case. Moreover, SSS + BERTScore showed the best and more stable performance for all language pairs in the WMT-17 dataset. This can be clearly visualised for en-lv as an example in Figure 1, where we plot metric scores in the Y axis against human scores in the X axis.
We believe the differences in the performance of the combined metric in the Multi-30K and WMT-17 datasets happens because the sentence length differs significantly in these datasets: sentences in Multi-30K have on average 12-14 words, much shorter than those in the WMT-17 dataset. Because WMD optimizes the word alignment globally for the whole sentence, instead of optimizing word alignment locally like BERTScore, the performance of WMD is better than BERTScore when sentence length is shorter, but it becomes a harder optimization problem when the sentence
$^{2}$ https://github.com/Rain9876/Unsupervised-crosslingual-Compound-Method-For-MT
MT-REF
Metrics	de-en	zh-en	fi-en	lv-en	ru-en	cs-en	tr-en	Avg
BLEU	0.366	0.440	0.444	0.321	0.413	0.344	0.441	0.396
METEOR	0.460	0.557	0.631	0.450	0.525	0.480	0.596	0.528
MEANT 2.0	0.565	0.639	0.687	0.586	0.607	0.578	0.596	0.608
WMDo (Word2Vec)	0.531	0.595	0.689	0.505	0.562	0.513	0.561	0.565
WMDo (BERT)	0.546	0.623	0.710	0.543	0.585	0.531	0.637	0.596
WMD	0.730	0.769	0.827	0.736	0.733	0.698	0.770	0.752
BERTScore	0.745	0.775	0.833	0.756	0.746	0.710	0.751	0.759
SSS	0.612	0.653	0.730	0.703	0.700	0.622	0.654	0.668
SSS + WMD	0.755	0.779	0.847	0.781	0.786	0.731	0.781	0.780
SSS + BERTScore	0.770	0.785	0.860	0.792	0.796	0.746	0.782	0.790
Table 4: Pearson Correlation with human scores for the WMT-17 dataset (to English) with Roberta-Large in the MT-REF setting. MEANT 2.0 (Lo, 2017) was the winning metric in that year,  $\mathrm{WMD}_o$  (Word2Vec) is from (Chow et al., 2019) using Word2Vec embeddings, and  $\mathrm{WMD}_o$  (BERT) our modification of it using BERT embeddings.
SRC-MT
Metrics	ne-en	en-de	et-en	en-zh	ro-en	si-en	ru-en	Avg
Leaderboard baseline	0.386	0.146	0.477	0.190	0.685	0.374	0.548	0.322
D-TP	0.558	0.259	0.642	0.321	0.693	0.460	—	0.489
D-Lex-Sim	0.600	0.172	0.612	0.313	0.663	0.513	—	0.479
WMD	0.361	0.456	0.463	0.251	0.647	0.308	0.315	0.400
BERTScore	0.357	0.459	0.460	0.260	0.673	0.309	0.320	0.405
SSS	0.313	0.330	0.481	0.401	0.694	0.404	0.441	0.438
SSS + WMD	0.390	0.472	0.553	0.427	0.724	0.426	0.476	0.495
SSS + BERTScore	0.392	0.484	0.553	0.427	0.727	0.426	0.475	0.498
Table 5: Pearson Correlation with human scores for the WMT-20 dataset with Roberta-Base in the SRC-MT setting. Metrics like D-TP and D-Lex-Sim (Fomicheva et al., 2020b) are unsupervised metrics which show good performance in the WMT-20 quality estimation shared task, while Leaderboard baseline is a supervised model provided by the organizers that uses training data to finetune pretrained representations.
Figure 1: Comparing BERTScore and SSS + BERTScore for lv-en in WMT-17 SRC-MT case.
length is long. This may explain why the performance of SSS + WMD is better than that of SSS + BERTScore in Multi-30K but lower than that of SSS + BERTScore in the WMT-17 dataset.
SSS also outperforms WMD and BERTScore in the WMT-20 dataset, as Table 5 shows. SSS + BERTScore reaches the best performance in three out of seven language pairs and is the best metric in comparison with BERTScore or WMD alone. The metrics that outperform SSS + BERTScore
for three language pairs require multiple passes of the neural machine translation decoder to score or generate multiple translations (D-TP and D-Lex-Sim, respectively), or require supervised machine learning (Leaderboard baseline).
5.2 MT-REF Setting
In the machine generated sentence to reference sentence case, as Table 2 shows,  $\mathbf{SSS} + \mathbf{WMD}$  achieves the best result in the monolingual Multi-30K tasks for both German to German and French to French using XLM-Roberta-Base embeddings. However, for other datasets in this standard setting where we compare sentences in a monolingual fashion, as we can observe from Table 4 for the WMT-17 dataset,  $\mathbf{SSS} + \mathbf{BERTScore}$  is the best metric. The reason for the differences is again likely to be the sentence lengths in the two datasets. If taken independently, the performance of SSS is not as good here as that of WMD or BERTScore. The two variants of the combined metrics still outperform any metric on their own, and reach the best performance results in this dataset. It can also be observed from Table 4 that  $\mathbf{WMD}_o$  with Word2Vec is far behind than that with BERT embedding or
BERTScore	SSS	SentSim
E1	REF	The food tastes good.
E1	MT1	The food tastes not good.	0.954	0.778	0.821
E1	MT2	The food tastes not bad.	0.948	0.948	0.950
E2	REF	President Barack Obama also backs the proposal.
E2	MT1	President Obama also supported this proposal.	0.8419	0.954	0.844
E2	MT2	Supported President Obama also this proposal.	0.4604	0.625	0.405
E3	REF	She is recovering, and police are still searching for a suspect.
E3	MT1	She is recovering, and police are searching for a suspect.	0.984	0.903	0.912
E3	MT2	Police searched for a suspect, and she recovered.	0.911	0.688	0.713
E4	SRC	The food tastes good.
E4	MT1	这食物味道好.	0.882	1.000	0.958
E4	MT2	好道味物食这.(word order shuffled)	0.207	0.682	0.309
Table 6: Examples from various datasets including the comparisons among BERTScore, SSS and SentSim (SSS + BERTScore).
our WMD with Roberta-Large. It indicates that the importance of using the pretrained contextual embedding as the representation of tokens. A visual example of correlation plots can be seen in Figure 2 for the en-lv language pair again.
Generally, the metrics' performances in the case of SRC-MT are much lower than in the MT-REF setting. This can be attributed to the embeddings used. First, the models' embeddings are not the same in these two cases. In the case of MT-REF, monolingual embeddings are used, which are known to be stronger; however these cannot be used in the case of SRC-MT evaluation, where crosslingual embeddings are used instead, which have been trained on more than 100 languages. Also, the way the crosslingual embeddings were generated does not rely on specific alignments or mappings between tokens or sentences in different languages, which can make them suboptimal. Second, the size of pretrained model for the case of MT-REF (Roberta-Large) is much larger than that of SRC-MT (XLM-Roberta-Base). As previously mentioned, pre-trained semantic sentence embeddings using XLM-Roberta-Large are not available, so we instead provide a comparison with Roberta-Base for the MT-REF case with WMT-17 in Section 5.5 to show the impact of model size.
5.3 Effect of Embedding Layers
Since both XLM-Roberta-Base and Roberta-Large have multiple layers, selecting a good layer or combination of layers is important for WMD and BERTScore. Here we use the WMT-17 dataset to study these representation choices. The Pearson Correlation of WMD with human judgement scores for the SRC-MT setting by specific XLM-Roberta-Base's layers is shown in Figure 3. Se
Figure 2: Comparing BERTScore and SSS + BERTscore for lv-en in WMT-17 MT-REF case.
lecting Layer 9 as the token embeddings for XLM-Roberta-Base leads to the best average Pearson Correlation among 9 language pairs in this SRC-MT setting.
Figure 3: Pearson Correlation of WMD with different layers of XLM-Roberta-Base embeddings in the WMT-17 dataset, SRC-MT setting.
For Roberta-Large, in Figure 4 we study the performance of different layers using the WMT-17 dataset in the MT-REF setting. Among the 24
Metrics	de-en	zh-en	fi-en	lv-en	ru-en	cs-en	tr-en	Avg
WMD	0.667	0.743	0.818	0.693	0.705	0.663	0.744	0.719
BERTScore	0.683	0.740	0.818	0.693	0.707	0.675	0.718	0.719
SSS	0.612	0.655	0.705	0.680	0.642	0.599	0.644	0.648
SSS + WMD	0.718	0.767	0.832	0.755	0.736	0.703	0.764	0.754
SSS + BERTScore	0.728	0.767	0.843	0.755	0.744	0.717	0.758	0.759
Table 7: Pearson Correlation with human scores for WMT-17 dataset with Roberta-Base in the MT-REF setting (to English).
output layers, the best layer seems to be 17. This is inline with the results described in (Zhang et al., 2019), where the best layer for Roberta-Large to use in BERTScore is also found to be layer 17.
Figure 4: Pearson Correlation of WMD with different layers of Roberta-Large embeddings in the WMT-17 dataset, MT-REF setting.
5.4 Analysis of SSS vs token-level metrics
For illustration purposes, Table 6 shows a few cases where SSS performs better than token-level metric because it adds the notion of sentence meaning and where, as a consequence, SentSim performs better (examples E1 and E2). It also show cases where SSS is too sensitive to semantic changes (example E3). SSS also performs well in the SRC-MT case (example E4). Here, the second machine translation has very different and incorrect word order, and the token-level metric (BERTScore) has very low performance compared to SSS, but both token-level and SSS metrics capture the incorrect word order. The combined metric (SentSim), therefore, is very robust.
5.5 Effect of Pretrained Embeddings
To analyse the impact of pre-trained embeddings, Table 7 shows the performance of Roberta-Base in the case of WMT-17 MT-REF. As with the general trend in NLP, this confirms that stronger embeddings (Roberta-Large, Table 4) lead to better performance. The same trend was observed for the
other test sets.
In this paper, we propose to combine sentence-level and token-level evaluation metrics in an unsupervised way. In our experiments on a number of standard datasets, we demonstrate that this combination is more effective for MT evaluation than the current state-of-the-art unsupervised token-level metrics, substantially outperforming these as well as sentence-level semantic metrics on their own. The sentence level metric seems to capture higher-level or compositional semantic similarity, which complements the token-level semantic similarity information.
We also show that this combination approach can be applied both in the standard monolingual evaluation setting, where machine translations are compared to reference translations, and in a crosslingual evaluation setting, where reference translations are not available and machine translations are directly compared with the source sentences.
In future work, we will aim to improve the crosslingual metric and explore other types of multilingual embeddings for better mapping across different languages.
Lucia Specia was supported by funding from the Bergamot project (EU H2020 grant no. 825303).
Milton Abramowitz and Irene A. Stegun, editors. 1965. Handbook of Mathematical Functions with Formulas, Graphs and Mathematical Tables. Dover Publications, Inc., New York.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Ar
bor, Michigan. Association for Computational Linguistics.
Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 489-513, Copenhagen, Denmark. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Julian Chow, Lucia Specia, and Pranava Madhyastha. 2019. WMDO: Fluency-based word mover's distance for machine translation evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 494-500, Florence, Italy. Association for Computational Linguistics.
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748-2760, Florence, Italy. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Hiroshi Echizen'ya, Kenji Araki, and Eduard Hovy. 2019. Word embedding-based automatic MT evaluation metric using word position information. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1874-1883,
Minneapolis, Minnesota. Association for Computational Linguistics.
Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70-74, Berlin, Germany. Association for Computational Linguistics.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Vishrav Chaudhary, Mark Fishel, Francisco Guzmán, and Lucia Specia. 2020a. Bergamot-latte submissions for the wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 1008-1015, Online. Association for Computational Linguistics.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020b. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555.
Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 199-209, Valencia, Spain. Association for Computational Linguistics.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. volume 37 of Proceedings of Machine Learning Research, pages 957-966, Lille, France. PMLR.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, pages 589-597, Copenhagen, Denmark. Association for Computational Linguistics.
Chi-kiu Lo. 2019. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507-513, Florence, Italy. Association for Computational Linguistics.
Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined MT metric based on direct assessment — CASICT-DCU submission to WMT17 metrics task. In Proceedings of the Second Conference on Machine Translation, pages 598–603, Copenhagen, Denmark. Association for Computational Linguistics.
Pranava Madhyastha, Josiah Wang, and Lucia Specia. 2019. VIFIDEL: Evaluating the visual fidelity of image descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6539-6550, Florence, Italy. Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799-2808, Florence, Italy. Association for Computational Linguistics.
Nitika Mathur, Johnny Wei, Qingsong Ma, and Ondrej Bojar. 2020. Results of the wmt20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 686-723, Online. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR, abs/1301.3:1-12.
João Moura, miguel vera, Daan van Stigt, Fabio Kepler, and Andre F. T. Martins. 2020. Ist-unbabel participation in the wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 1027-1034, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. Transquest at wmt2020: Sentence-level direct assessment. In Proceedings of the Fifth Conference on Machine Translation, pages 1047-1053, Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, Online. Association for Computational Linguistics.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. 2000. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99-121.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Kashif Shah, Trevor Cohn, and Lucia Specia. 2015. A bayesian non-linear method for feature selection in machine translation quality estimation. In Machine Translation 29, pages 79-84.
Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and Andre F. T. Martins. 2020. Findings of the wmt 2020 shared task on quality estimation. In Proceedings of the Fifth Conference on Machine Translation, pages 741-762, Online. Association for Computational Linguistics.
Lucia Specia, Kashif Shah, Jose G.C. de Souza, and Trevor Cohn. 2013. QuEst - a translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 79-84, Sofia, Bulgaria. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020. Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. CoRR, abs/2010.08240.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2014. Cider: Consensus-based image description evaluation. CoRR, abs/1411.5726.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. *Bertscore: Evaluating text generation with BERT*. CoRR, abs/1904.09675.
Zachariah Zhang, Jingshu Liu, and Narges Razavian. 2020. BERT-XML: Large scale automated ICD coding using BERT pretraining. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 24-34, Online. Association for Computational Linguistics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. CoRR, abs/1909.02622.
A.1 Comparison to Sentence Mover's Distance
Sentence Mover's Distance (SMD) (Zhao et al., 2019) is an alternative sentence level metric which for sentence semantic similarity. It compares two text documents using sentence embeddings which are not semantically fine-tuned but based on averaging or pooling the sentences' combined contextual word embeddings. The SMD is defined as follows:
$$
S M D \left(x ^ {n}, y ^ {n}\right) := \left\| E \left(x _ {1} ^ {l _ {x}}\right) - E \left(y _ {1} ^ {l _ {y}}\right) \right\| \tag {9}
$$
where  $E$  is the embedding function which maps an n-gram to its vector representation,  $l_{x}$  and  $l_{y}$  are the size of sentences. As a comparison, we experimented with the linear combination between SMD and each of our token-level metrics – WMD and BERTScore. The metrics performances for WMT-17 in both cases of SRC-MT and MT-REF, and WMT-20 SRC-MT are shown in Table 8, Table 9 and Table 10.
The overall performance of this metric is inferior to that of SSS, which is to be expected since this is simply averaging token-level embeddings. Similar to our SSS, the SMD metric performance improves when it is combined with token-level metrics. The combined metrics' performance drops when there is a big difference between the scores of the two combined metrics, e.g. more than  $10\%$ . To pick an example, in Table 8 the gap between BERTScore and SMD for zh-en is 0.115, and the combined SMD + BERTScore only reaches a score of 0.503, compared to 0.51 from BERTScore alone. For other languages with closer BERTScore and SMD scores, the performance of the combined metric remains the same or improves, for example, ru-en.
A.2 Plots with Metrics' Performance
To facilitate visualisation of our main tabular results presented in the paper, Figures 5, 6, 7 show them as bar plots.
Figure 5: Metrics' performance in WMT-20 SRC-MT case.
Figure 6: Metrics' performance in WMT-17 SRC-MT case.
Figure 7: Metrics' performance in WMT-17 MT-REF case.