Venue: NAACL 2021

# SentSim: Crosslingual Semantic Evaluation of Machine Translation

 Yurun Song\({}^{*}\)

Imperial College London

UK

yuru.song19@imperial.ac.uk &Junchen Zhao\({}^{*}\)

University of California, Irvine

CA, US

junchez3@uci.edu &Lucia Specia

Imperial College London

UK

l.specia@imperial.ac.uk

###### Abstract

Machine translation (MT) is currently evaluated in one of two ways: in a monolingual fashion, by comparison with the system output to one or more human reference translations, or in a trained crosslingual fashion, by building a supervised model to predict quality scores from human-labeled data. In this paper, we propose a more cost-effective, yet well performing unsupervised alternative **SentSim**: relying on strong pretrained multilingual word and sentence representations, we directly compare the source with the machine translated sentence, thus avoiding the need for both reference translations and labelled training data. The metric builds on state-of-the-art embedding-based approaches - namely BERTScore and Word Mover's Distance - by incorporating a notion of sentence semantic similarity. By doing so, it achieves better correlation with human scores on different datasets. We show that it outperforms these and other metrics in the standard monolingual setting (MT-reference translation), a well as in the source-MT bilingual setting, where it performs on par with glass-box approaches to quality estimation that rely on MT model information.

+
Footnote †: Contributed equally to this work.

_Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3143–3156

June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Automatically evaluating machine translation (MT) as well as other language generation tasks has been investigated for decades, with substantial progress in recent years due to the advances of pretrained contextual word embeddings. The general goal of such evaluation metrics is to estimate the semantic equivalence between the input text (e.g. a source sentence or a document) and an output text that has been modified in some way (e.g. a translation or summary), as well as the general quality of the output (e.g. fluency). As such, by definition metrics should perform some forms of input-output comparisons.

However, this direct comparison has been proven hard in the past because of the natural differences between the two versions (such as different languages). Instead, evaluation metrics have resorted to comparison against one or more correct outputs produced by humans, _a.k.a._ reference texts, where comparisons at the string level are possible and straightforward. A multitude of evaluation metrics have been proposed following this approach, especially for MT, the application we focus on in this paper. These include the famous BLEU Papineni et al. (2002) and METEOR Banerjee and Lavie (2005) for machine translation, ROUGE Lin (2004) for summarization, and CIDER Vedantam et al. (2014) for image captioning. These traditional metrics are based on simple-word, n-gram matching mechanisms or slight relaxations of these (e.g. synonyms) which are computationally efficient, but suffer from various limitations.

In order to overcome the drawbacks of the traditional string-based evaluation metrics, recent work Williams et al. (2018); Bowman et al. (2015); Echizen'ya et al. (2019); Cer et al. (2017); Echizen'ya et al. (2019) has investigated metrics that perform comparisons in the semantic space rather than at the surface level. Notably, applications of Word Mover's Distance (WMD; Kusner et al. (2015), such as WMDo Chow et al. (2019), VIFIDEL Madhyastha et al. (2019) and moverscore Zhao et al. (2019), which compute similarity based on continuous word embeddings using pre-trained representations. These have been shown to consistently outperform previous metrics on various language generation evaluation tasks.

However, these metrics have two limitations: (i) they still rely on reference outputs, which are expensive to collect, only cover one possible correct answer, and do not represent how humans do evaluation; (ii) they are bag-of-embeddings approaches which capture semantic similarity at the token level, but are unable to capture the meaning of the sen

tence or text as a whole, including correct word order.

In this paper, focusing on MT, to address these limitations we first posit that evaluation can be done by directly comparing the source to the machine translation using multilingual pretrained embeddings, such as multilingual BERT, avoiding the need of reference translations. We note that this is different from quality estimation (QE) metrics Speccia et al. (2013); Shah et al. (2015) , which also compare source and machine translated texts directly, but assume an additional step of supervised learning against human labels for quality. Second, we introduce Sentence Semantic Similarity (SSS) , an additional component to be combined with bag-of-embeddings distance metrics such as BERTScore. More specifically, we propose to explore semantic similarity at the sentence level - based on sentence embeddings Sellam et al. (2020); Reimers and Gurevych (2020); Thakur et al. (2020) - and linearly combine it with existing metrics that use word embeddings. By doing so, the resulting metrics have access to word and compositional semantics, leading with improved performance. The combination is a simple weighted sum, and does not require training data.

As a motivational example, consider the case in Table 1, from the WMT-17 Metrics task Zhang et al. (2019). When faced with MT sentences that contain a negated version of the reference (MT3 and MT4), token-level metrics such as BERTScore and WMD cannot correctly penalize these sentences since they match representations of words in both versions without a full understanding of the semantics of the sentences. As a consequence, they return a high score for these incorrect translations, higher than the score for correct paraphrases of the reference (MT1 and MT2). Sentence similarity, on the other hand, correctly captures this mismatch in meaning, returning relatively lower scores for Translations 3 and 4. However on their own they may be too harsh, since the remaining of the sentence has the same meaning. The combination of these two metrics (last column) balances between these two sources of information and, as we will later show in this paper, has higher correlation with human scores.

Our **main contributions** are:

1. We investigate and show the effectiveness of linearly combining sentence-level semantic similarity with different metrics using token-level semantic similarity. The resulting combined metric, SentSim, consistently achieves higher Pearson Correlation with human judgements of translation quality than both word and sentence similarity alone.
2. We show, for the first time, that these metrics can be effective when comparing system-generated sentences directly against source sentences, in a crosslingual fashion.
3. Our SentSim metric outperforms existing metrics on various MT datasets in monolingual and crosslingual settings.

## 2 Related Work

Various natural language generation tasks, including machine translation, image captioning, among others, produce sentences as output. These are evaluated either manually or automatically by comparison against one or multiple reference sentences. A multitude of metrics have been proposed for the latter, which perform comparisons at various granularity levels, from characters to words to embedding vectors. The goal of such metrics is to replace human judgements. In order to understand how well they fare at this task, metrics are evaluated by how similar their scores are to human assigned judgements on held-out datasets. For absolute quality judgements, Pearson Correlation is the most popularly used metric for such a comparison Mathur et al. (2020).

Recent studies have showed that the new generation of automatic evaluation metrics, which instead of lexical overlap are based on word semantics using continuous word embedding, such as BERT Devlin et al. (2019), ElMo Peters et al. (2019), XLNet Yang et al. (2019) or XLM-Roberta Conneau et al. (2019), have significantly higher Pearson Correlation with the human judgements when comparing reference sentences with system generated sentences. Zhang et al. (2019) introduce **BERTScore**, an automatic evaluation metric based on contextual word embeddings, and tests it for text generation tasks such as machine translation and imaging captioning, using embeddings including BERT, XLM-Roberta, and XLNet (more details in Section 3.2). Mathur et al. (2019) present supervised and unsupervised metrics which are based on BERT embeddings for improving machine translation evaluation. Zhao et al. (2019) introduce **moverscore**, a metric which generates high-quality evaluation 

results on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, using BERT embeddings. Clark et al. (2019) present semantic metrics for text summarization based on the **sentence mover's similarity** and ELMo embeddings. Chow et al. (2019) introduce a fluency-based word mover's distance (**WMDo**) metric for machine translation evaluation using Word2Vec embeddings Mikolov et al. (2013). Lo (2019) presents **Yisi**, a unified automatic semantic machine translation quality evaluation and estimation metric using BERT embeddings.

There is also a bulk of work on metrics that take a step further to optimize their scores using machine learning algorithms trained on human scores for quality Sellam et al. (2020); Ma et al. (2017). They often perform even better, but the reliance on human scores for training, in addition to reference translations at inference time, makes them less applicable in practice. A separate strand of work that relies on contextual embeddings is that of Quality Estimation Moura et al. (2020); Fomicheva et al. (2020); Ranasinghe et al. (2020); Specia et al. (2020). These are also trained on human judgements of quality, but machine translations are compared directly to the source sentences rather than against reference translations.

In addition to embeddings for words, embeddings for full sentences have been shown to work very well to measure semantic similarity. These are extracted using Transformer models that are specifically trained for capturing sentence semantic meanings using BERT, Roberta, and XLM-Roberta embeddings Reimers and Gurevych (2019); Reiners and Gurevych (2020); Thakur et al. (2020) and provide state-of-art performance pretrained models for many languages.1

Footnote 1: [https://github.com/UKPLab/sentence-transformers](https://github.com/UKPLab/sentence-transformers)

In this paper, we take inspiration from these lines of previous works to propose unsupervised metrics that combine word and sentence semantic similarity and show that this can be effective for both MT-reference and source-MT comparisons.

## 3 Method

In this section, we first describe in more detail the metrics that we have used in our experiments, namely semantic sentence cosine similarity, WMD and BERTScore. Then we present our simple approach to linearly combine these metrics.

### Word Mover's Distance (WMD)

Kusner et al. (2015) presents word mover's distance (WMD) metric, a special case of Earth mover's distance Rubner et al. (2000), computing the semantic distance between two text documents by aligning semantically similar words and capturing the word traveling flow between the similar words utilizing the vectorial relationship between their word embeddings Mikolov et al. (2013). WMD has been proven to generate consistently high-quality results for the tasks of measuring text similarity and text classification Kusner et al. (2015). A text document is represented as a vector \(D\), where each element is denoted as the normalized frequency of a word in the document such that:

\[D=[d_{1},d_{2},....,d_{n}]^{T} \tag{1}\]

where \(d_{i}\) = \(c_{i}/\sum_{j}^{n}c_{j}\) and \(c_{i}\) is the frequency that the \(i^{th}\) word which appears \(c_{i}\) times in a given text document. Assuming there are two given words from different text document denoted as \(i\) and \(j\), then the euclidean distance in the embedding \(x_{i}\) and \(x_{j}\) for the two words is defined as:

\[c(i,j)=\|x_{i}-x_{j}\|_{2} \tag{2}\]

where \(c(i,j)\) is defined as the "word traveling cost" from \(x_{i}\) in one document to \(x_{j}\) in the other document. Now, assuming there are two documents,

\begin{table}
\begin{tabular}{l l c c c}  & & BERTScore & SSS & SSS + BERTScore \\ \hline REF & We have made a complete turnaround. & & & & \\ MT1 & We did a complete turnaround. & 0.7975 & 0.9578 & 0.8111 \\ MT2 & We made a turnaround. & 0.7748 & 0.8898 & 0.7427 \\ MT3 & We have not made a complete turnaround. & 0.8296 & 0.3878 & 0.4832 \\ MT4 & We have made an incomplete turnaround. & 0.8318 & 0.4431 & 0.5107 \\ \end{tabular}
\end{table}
Table 1: An example from the WMT-17 dataset. Given the reference (REF) sentence, BERTScore assigns higher similarity to its negated versions (MT3 and MT4) than to semantically similar variants (MT1 and MT2). Contrarily, SSS gives a very low score to MT3 and MT4. Their combination provides a more balanced score.

 

one is the source document denoted as \(A\) where the word \(i\) belongs to, and another one is the target document denoted as \(B\) where the word \(j\) belongs to. A flow matrix \(T\) is defined in which every element is denoted as \(T_{ij}\), suggesting the number of times the word \(i\) in document \(A\) moves to the word \(j\) in document \(B\). Then, the value of the flow matrix is normalized based on the total count of words in the vocabulary such that:

\[\sum_{j}T_{ij}=d_{i},\sum_{i}T_{ij}=d_{j} \tag{3}\]

The semantic distance calculated by WMD can be then defined as follows:

\[\text{WMD}=\min_{T\geq 0}\sum_{i,j=1}^{n}T_{ij}c(i,j) \tag{4}\]

WMD, or the semantic distance between two text documents, can thus be computed by optimizing values in the flow matrix \(T\). In other words, WMD corresponds to the minimal semantic distance to move semantically similar words (via their embeddings) from one text document to another.

### BERTScore

BERTScore Zhang et al. (2020) is designed to evaluate semantic similarity between sentences in the same language, namely a reference sentence and a machine-generated sentence. Assume a reference sentence is denoted as \(x\) = \((x_{1},....,x_{k})\) and a candidate sentence is denoted as \(\hat{x}\) = \((\hat{x_{1}},....,\hat{x_{k}})\), BERTScore uses contextual embeddings such as BERT Devlin et al. (2019) or ELMo Peters et al. (2019) to represent word tokens in the sentences. It finds word matchings between the reference and candidate sentence using cosine similarity, which can be optionally reweighted by the inverse document frequency scores (IDF) of each word. BERTScore matches each word token \(x\) in reference sentence to the closest word token \(\hat{x}\) in candidate sentence for computing recall, and matches each word token \(\hat{x}\) in candidate sentence to the closest word token \(x\) in reference sentence for computing precision. It combines recall with precision to produce an F1 score. However, only recall is used for evaluation in most cases, which is defined as follows:

\[R_{BERT}=\frac{1}{|x|}\sum_{x_{i}\in x}\max_{\hat{x_{j}}\in\hat{x}}x_{i}^{T} \hat{x_{j}} \tag{5}\]

In essence, BERTScore can be viewed as a hard word alignment given a pair of sentences using contextual embeddings, in which each word is aligned to one other word, the closest in the embedding space according to the cosine distance between their vectors.

### Semantic Sentence Similarity (SSS)

A commonly used method to measure sentence similarity is using the cosine distance between the two vectors summarizing the sentences:

\[cos(\theta)=\frac{\alpha\cdot\beta}{\lVert\alpha\rVert\lVert\beta\rVert} \tag{6}\]

where \(\alpha\) and \(\beta\) are the vectors representing the two sentences. The higher the value obtained through cosine similarity between two sentences vectors based on the pretrained sentence representation Reimers and Gurevych (2019); Reimers and Gurevych (2020); Thakur et al. (2020), the stronger their similarity.

### SentSim

In order to bring the notion of semantic similarity to token similarity metrics, we combine the sentence cosine similarity using semantically fine-tuned sentence embedding with the metrics using contextual word embeddings. Assume that the generated score from sentence level metric is denoted as \(A\), the value generated from token-level metric is denoted as \(B\) and the gold truth from human judgement is denoted as \(S\). Our combination metric, namely SentSim, is as follows:

\[\text{SentSim}(A,B)= w_{1}*e^{A}+w_{2}*e^{B} \tag{7}\]

where \(A\) and \(B\) are normalized to the range between 0 and 1, \(w_{1}\) and \(w_{2}\) are the weights given to two metric scores. If metric \(B\) is negatively correlated with \(S\), i.e., if it is a distance metric like WMD, we give it \(e^{1-B}\). We use \(e^{B}\) for similarity metrics such as SSS and BERTScore.

In equation 7, we apply exponential for similarity scores as the linear addition of two similarity scores (\(A+B\)) in lower-order leads to a large variance and inconsistency in the correlation with human scores. Lower-order models are too simple to fit the relationship between similarities. Therefore, a non-linear model is required to project these similarities into higher-order (\(A^{n}+B^{n}\)). Given the Taylor Series Expansion Abramowitz and Stegun ( 

1965) of exponential function, we can get a factorial average of two similarities from lower-order to higher-order as follows:

\[{\rm SentSim}(A,B)= \sum_{n=1}^{\infty}\frac{w_{1}*A^{n}+w_{2}*B^{n}}{n!} \tag{8}\]

Our final metric is given in Equation 8, which follows from Equation 7 using Taylor Series Expansion. This was also shown in Kilickaya et al. (2017); Clark et al. (2019), which convert distance scores to similarities by using the exponential function.

In Section 5, we report experiments with two linear metric combinations: **SSS + WMD** and **SSS + BERTScore**, where we give equal weight to each metric (\(w_{1}\) = \(w_{2}\) = 0.5). We have also investigated the linear combination between Sentence Mover's Distance Zhao et al. (2019) and token-level metrics, but the performance is poorer than SSS, so we only show results in the Appendix A.1.

## 4 Experiment Setup

In this section, we describe two types of experimental scenarios, monolingual and crosslingual evaluation, as well as the three datasets and pre-trained embeddings we used.

### Task Scenarios

The first evaluation setting we experimented with is the standard monolingual evaluation task scenario (MT-REF), which takes reference sentences and machine generated sentences in the same language as input. The second one is the crosslingual evaluation task scenario (SRC-MT), which directly assesses the similarity between source sentences and machine generated sentences in different languages. We compute our combined metrics for each task scenario separately.

### Datasets

We use various datasets with absolute human judgements from recent evaluation campaigns.

Multi-30K Elliott et al. (2016) is a multilingual (English-German (en-de) and English-French (en-fr)) image description dataset. We use the 2018 test set, in which each language pair contains more than 2K sentence tuples, including source sentences, reference sentences, machine generated sentences, and the corresponding human judgement scores in an (0-100) continuous range. Therefore, this dataset can be used for both crosslingual and monolingual task scenarios.

Wmt-17Bojar et al. (2017) is a dataset containing multiple language pairs from the WMT News Translation task used for segment-level system evaluation in the Metrics task. We used all seven to-English datasets: German-English (den), Chinese-English (zh-en), Latvian-English (lv-en), Czech-English (cs-en), Finnish-English (fi-en), Russian-English (ru-en), Turkish-English (tr-en) and two from-English datasets: English-Russian (en-ru), English-Chinese (en-zh). Each language has 560 sentence tuples, where each tuple has a source sentence, a reference sentence and multiple system generated sentences, in addition to a human score varying from 0 to 100. WMT-17 can be used in both monolingual and crosslingual evaluation task scenarios, and is our main experimental data. More recent WMT Metrics task datasets do not report metrics results using absolute judgements, but rather convert these into pairwise judgements. While such relative judgements are useful to assess metrics ability to rank different MT systems, they are not applicable to assess metrics in their ability to estimate quality in absolute terms, which are what we are interested in.

Wmt-20Fomicheva et al. (2020b) is the dataset used in the WMT20 quality estimation task, where participants are expected to directly predict the translation quality between source sentences and machine generated sentences without using reference sentences. This dataset has seven language pairs: Sinhala-English (si-en), Nepalese-English (ne-en), Estonian-English (et-en), English-German (en-de), English-Chinese (enzh), Romanian-English (ro-en), Russian-English (ru-en). We use the test set, withdrew each language pair contains 1K tuples with source and machine generated sentences, as well as human judgements in the 0-100 range. Therefore, with this dataset we can only perform crosslingual evaluation.

### Embeddings

For each language model, we consider embeddings at the token level and sentence level individually and in combination. In our experiments, Roberta-Large and XLM-Roberta-Base for monolingual and crosslingual assessments respectively.

For crossligual embeddings we use XLM-Roberta instead of multilingual BERT (mBERT) 

because the former significantly outperforms the latter Conneau et al. (2019), as also shown by Reimers and Gurevych (2020) for crosslingual semantic textual similarity (STS) tasks Cer et al. (2017). For a fair comparison with previous metrics like WMD\({}_{0}\), we replaced their original embeddings with XLM-Roberta-Base embeddings.

For the semantic sentence embedding, we used XLM-Roberta-Base embeddings from Sentence Transformer, which were trained on SNLI Bowman et al. (2015) + MultiNLI Williams et al. (2018) and then fine-tuned on the STS benchmark training data. These sentence embeddings have been shown to provide good representations of the semantic relationship between two sentences, but they had not yet been tested for machine translation evaluation. Without using semantic embeddings, the performance of SSS is not consistent across different languages pairs given our experimental datasets (see Appendix A.1). XLM-Roberta-Large embeddings are not used in our experiments because they are not available in the pre-trained Sentence Transformer package yet.

For monolingual word and semantic sentence embeddings we use the Roberta-Large model, which has shown the best performance with BERTScore Zhang et al. (2019).

## 5 Results

The evaluation results are presented in this section. Our code and data can be found on github2.

Footnote 2: [https://github.com/Rain9876/Unsupervised-crosslingual-Compound-Method-For-MT](https://github.com/Rain9876/Unsupervised-crosslingual-Compound-Method-For-MT)

### SRC-MT Setting

From Table 2, we can observe the Pearson correlation results of our metrics by comparing the source sentences with machine translated sentences using both single metrics and their combinations in the Multi-30K dataset. The result reveals that **SSS + WMD** outperforms all individual metrics and the other combined metrics. It is clear that SSS is better than both WMD and BERTScore, with WMD outperforming BERTScore in this specific crosslingual task.

In Table 3, the benefit of SSS becomes even more evident. It again outperforms WMD and BERTScore, with BERTScore also significantly outperforming WMD in this case. Moreover, **SSS + BERTScore** showed the best and more stable performance for all language pairs in the WMT-17 dataset. This can be clearly visualised for en-lv as an example in Figure 1, where we plot metric scores in the Y axis against human scores in the X axis.

We believe the differences in the performance of the combined metric in the Multi-30K and WMT-17 datasets happens because the sentence length differs significantly in these datasets: sentences in Multi-30K have on average 12-14 words, much shorter than those in the WMT-17 dataset. Because WMD optimizes the word alignment globally for the whole sentence, instead of optimizing word alignment locally like BERTScore, the performance of WMD is better than BERTScore when sentence length is shorter, but it becomes a harder optimization problem when the sentence

\begin{table}
\begin{tabular}{|l|c c c|c c c|} \hline  & \multicolumn{4}{c|}{**SRC-MT**} & \multicolumn{4}{c|}{**REF-MT**} \\ \hline Metrics & en-de & en-fr & Avg & de-de & fr-fr & Avg \\ \hline BLEU & - & - & - & 0.262 & 0.387 & 0.325 \\ METEOR & - & - & - & 0.461 & 0.411 & 0.436 \\ WMD & 0.360 & 0.319 & 0.340 & 0.492 & 0.425 & 0.459 \\ BERTScore & 0.335 & 0.291 & 0.313 & 0.434 & 0.352 & 0.393 \\ SSS & 0.483 & 0.449 & 0.466 & 0.487 & 0.446 & 0.467 \\ \hline SSS + WMD & **0.508** & **0.477** & **0.492** & **0.546** & **0.501** & **0.524** \\ SSS + BERTScore & 0.486 & 0.434 & 0.460 & 0.527 & 0.462 & 0.494 \\ \hline \end{tabular}
\end{table}
Table 2: Pearson Correlation with human scores for Multi-30K with Roberta-Base in the SRC-MT and MT-REF settings. For the latter we evaluate German to German and French to French as monolingual tasks.

\begin{table}
\begin{tabular}{|l|c c c c c c c c c c|} \hline  & \multicolumn{8}{c|}{**SRC-MT**} \\ \hline Metrics & de-en & zh-en & fi-en & lv-en & ru-en & cs-en & en-ru & en-zh & tr-en & Avg \\ \hline WMD & 0.366 & 0.501 & 0.373 & 0.373 & 0.308 & 0.267 & 0.404 & 0.408 & 0.350 & 0.372 \\ BERTScore & 0.409 & 0.510 & 0.414 & 0.402 & 0.337 & 0.319 & 0.434 & 0.446 & 0.382 & 0.406 \\ SSS & 0.456 & 0.514 & 0.540 & 0.555 & 0.541 & 0.464 & 0.505 & 0.458 & 0.540 & 0.508 \\ \hline SSS + WMD & 0.504 & 0.594 & 0.566 & 0.569 & 0.534 & 0.476 & 0.538 & 0.513 & 0.562 & 0.540 \\ SSS + BERTScore & **0.523** & **0.600** & **0.578** & **0.574** & **0.551** & **0.499** & **0.553** & **0.531** & **0.569** & **0.553** \\ \hline \end{tabular}
\end{table}
Table 3: Pearson Correlation with human scores for the WMT-17 with Roberta-Base in the SRC-MT setting.

 

length is long. This may explain why the performance of **SSS + WMD** is better than that of **SSS + BERTScore** in Multi-30K but lower than that of **SSS + BERTScore** in the WMT-17 dataset.

SSS also outperforms WMD and BERTScore in the WMT-20 dataset, as Table 5 shows. **SSS + BERTScore** reaches the best performance in three out of seven language pairs and is the best metric in comparison with BERTScore or WMD alone. The metrics that outperform **SSS + BERTScore** for three language pairs require multiple passes of the neural machine translation decoder to score or generate multiple translations (D-TP and D-LexSim, respectively), or require supervised machine learning (Leaderboard baseline).

### MT-REF Setting

In the machine generated sentence to reference sentence case, as Table 2 shows, **SSS + WMD** achieves the best result in the monolingual Multi-30K tasks for both German to German and French to French using XLM-Roberta-Base embeddings. However, for other datasets in this standard setting where we compare sentences in a monolingual fashion, as we can observe from Table 4 for the WMT-17 dataset, **SSS + BERTScore** is the best metric. The reason for the differences is again likely to be the sentence lengths in the two datasets. If taken independently, the performance of SSS is not as good here as that of WMD or BERTScore. The two variants of the combined metrics still outperform any metric on their own, and reach the best performance results in this dataset. It can also be observed from Table 4 that WMD\({}_{o}\) with Word2Vec is far behind than that with BERT embedding or

Figure 1: Comparing BERTScore and SSS + BERTScore for lv-en in WMT-17 SRC-MT case.

\begin{table}
\begin{tabular}{|l|c c c c c c c c|} \hline  & \multicolumn{6}{c|}{**STR-MET**} \\ \hline Metrics & de-en & zh-en & fi-en & h-ven & ru-en & cs-en & tr-en & Avg \\ \hline BLEU & 0.366 & 0.440 & 0.444 & 0.321 & 0.413 & 0.344 & 0.441 & 0.396 \\ METEOR & 0.460 & 0.557 & 0.631 & 0.450 & 0.525 & 0.480 & 0.596 & 0.528 \\ MEANT 2.0 & 0.565 & 0.639 & 0.687 & 0.586 & 0.607 & 0.578 & 0.596 & 0.608 \\ WMD\({}_{o}\) (Word2Vec) & 0.531 & 0.595 & 0.689 & 0.505 & 0.562 & 0.513 & 0.561 & 0.565 \\ WMD\({}_{o}\) (BERT) & 0.546 & 0.623 & 0.710 & 0.543 & 0.585 & 0.531 & 0.637 & 0.596 \\ \hline WMD & 0.730 & 0.769 & 0.827 & 0.736 & 0.733 & 0.698 & 0.770 & 0.752 \\ BERTScore & 0.745 & 0.775 & 0.833 & 0.756 & 0.746 & 0.710 & 0.751 & 0.759 \\ SSS & 0.612 & 0.653 & 0.730 & 0.703 & 0.700 & 0.622 & 0.654 & 0.668 \\ \hline SSS + WMD & 0.755 & 0.779 & 0.847 & 0.781 & 0.786 & 0.731 & 0.781 & 0.780 \\ SSS + BERTScore & **0.770** & **0.785** & **0.860** & **0.792** & **0.796** & **0.746** & **0.782** & **0.790** \\ \hline \end{tabular}
\end{table}
Table 4: Pearson Correlation with human scores for the WMT-17 dataset (to English) with Roberta-Large in the MT-REF setting. MEANT 2.0 (Lo, 2017) was the winning metric in that year, WMD\({}_{o}\) (Word2Vec) is from (Chow et al., 2019) using Word2Vec embeddings, and WMD\({}_{o}\) (BERT) our modification of it using BERT embeddings.

 

our WMD with Roberta-Large. It indicates that the importance of using the pretrained contextual embedding as the representation of tokens. A visual example of correlation plots can be seen in Figure 2 for the en-lv language pair again.

Generally, the metrics' performances in the case of SRC-MT are much lower than in the MT-REF setting. This can be attributed to the embeddings used. First, the models' embeddings are not the same in these two cases. In the case of MT-REF, monolingual embeddings are used, which are known to be stronger; however these cannot be used in the case of SRC-MT evaluation, where crosslingual embeddings are used instead, which have been trained on more than 100 languages. Also, the way the crosslinkingal embeddings were generated does not rely on specific alignments or mappings between tokens or sentences in different languages, which can make them suboptimal. Second, the size of pretrained model for the case of MT-REF (Roberta-Large) is much larger than that of SRC-MT (XLM-Roberta-Base). As previously mentioned, pre-trained semantic sentence embeddings using XLM-Roberta-Large are not available, so we instead provide a comparison with Roberta-Base for the MT-REF case with WMT-17 in Section 5.5 to show the impact of model size.

### Effect of Embedding Layers

Since both XLM-Roberta-Base and Roberta-Large have multiple layers, selecting a good layer or combination of layers is important for WMD and BERTScore. Here we use the WMT-17 dataset to study these representation choices. The Pearson Correlation of WMD with human judgement scores for the SRC-MT setting by specific XLM-Roberta-Base's layers is shown in Figure 3. Selecting Layer 9 as the token embeddings for XLM-Roberta-Base leads to the best average Pearson Correlation among 9 language pairs in this SRC-MT setting.

For Roberta-Large, in Figure 4 we study the performance of different layers using the WMT-17 dataset in the MT-REF setting. Among the 24



output layers, the best layer seems to be 17. This is inline with the results described in Zhang et al. (2019), where the best layer for Roberta-Large to use in BERTScore is also found to be layer 17.

### Analysis of SSS vs token-level metrics

For illustration purposes, Table 6 shows a few cases where SSS performs better than token-level metric because it adds the notion of sentence meaning and where, as a consequence, SentSim performs better (examples E1 and E2). It also show cases where SSS is too sensitive to semantic changes (example E3). SSS also performs well in the SRC-MT case (example E4). Here, the second machine translation has very different and incorrect word order, and the token-level metric (BERTScore) has very low performance compared to SSS, but both token-level and SSS metrics capture the incorrect word order. The combined metric (SentSim), therefore, is very robust.

### Effect of Pretrained Embeddings

To analyse the impact of pre-trained embeddings, Table 7 shows the performance of Roberta-Base in the case of WMT-17 MT-REF. As with the general trend in NLP, this confirms that stronger embeddings (Roberta-Large, Table 4) lead to better performance. The same trend was observed for the other test sets.

## 6 Conclusions

In this paper, we propose to combine sentence-level and token-level evaluation metrics in an unsupervised way. In our experiments on a number of standard datasets, we demonstrate that this combination is more effective for MT evaluation than the current state-of-the-art unsupervised token-level metrics, substantially outperforming these as well as sentence-level semantic metrics on their own. The sentence level metric seems to capture higher-level or compositional semantic similarity, which complements the token-level semantic similarity information.

We also show that this combination approach can be applied both in the standard monolingual evaluation setting, where machine translations are compared to reference translations, and in a crosslingual evaluation setting, where reference translations are not available and machine translations are directly compared with the source sentences.

In future work, we will aim to improve the crosslingual metric and explore other types of multilingual embeddings for better mapping across different languages.

## Acknowledgements

Lucia Specia was supported by funding from the Bergamot project (EU H2020 grant no. 825303).

## References

* Abramowitz and Stegun (1965) Milton Abramowitz and Irene A. Stegun, editors. 1965. _Handbook of Mathematical Functions with Formulas, Graphs and Mathematical Tables_. Dover Publications, Inc., New York.
* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_, pages 65-72, Ann Ar

\begin{table}
\begin{tabular}{|l|c c c c c c c c|} \hline Metrics & de-en & zh-en & fi-en & iv-en & ru-en & cs-en & tr-en & Avg \\ \hline WMD & 0.667 & 0.743 & 0.818 & 0.693 & 0.705 & 0.663 & 0.744 & 0.719 \\ BERTScore & 0.683 & 0.740 & 0.818 & 0.693 & 0.707 & 0.675 & 0.718 & 0.719 \\ SSS & 0.612 & 0.655 & 0.705 & 0.680 & 0.642 & 0.599 & 0.644 & 0.648 \\ \hline SSS + WMD & 0.718 & 0.767 & 0.832 & 0.755 & 0.736 & 0.703 & **0.764** & 0.754 \\ SSS + BERTSCore & **0.728** & **0.767** & **0.843** & **0.755** & **0.744** & **0.717** & 0.758 & **0.759** \\ \hline \end{tabular}
\end{table}
Table 7: Pearson Correlation with human scores for WMT-17 dataset with Roberta-Base in the MT-REF setting (to English).

Figure 4: Pearson Correlation of WMD with different layers of Roberta-Large embeddings in the WMT-17 dataset, MT-REF setting.

 

bor, Michigan. Association for Computational Linguistics.
* Bojar et al. (2017) Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 489-513, Copenhagen, Denmark. Association for Computational Linguistics.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
* Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
* Chow et al. (2019) Julian Chow, Lucia Specia, and Pranava Madhyastha. 2019. WMDO: Fluency-based word mover's distance for machine translation evaluation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 494-500, Florence, Italy. Association for Computational Linguistics.
* Clark et al. (2019) Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2748-2760, Florence, Italy. Association for Computational Linguistics.
* Conneau et al. (2019) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. _CoRR_, abs/1911.02116.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Echizen'ya et al. (2019) Hiroshi Echizen'ya, Kenji Araki, and Eduard Hovy. 2019. Word embedding-based automatic MT evaluation metric using word position information. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1874-1883, Minneapolis, Minnesota. Association for Computational Linguistics.
* Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalili Sima'an, and Lucia Specia. 2016. Multi3OK: Multilingual English-German image descriptions. In _Proceedings of the 5th Workshop on Vision and Language_, pages 70-74, Berlin, Germany. Association for Computational Linguistics.
* Fomicheva et al. (2020a) Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frederic Blain, Vishrav Chaudhary, Mark Fishel, Francisco Guzman, and Lucia Specia. 2020a. Bergamot-lattle submissions for the wmrt20 quality estimation shared task. In _Proceedings of the Fifth Conference on Machine Translation_, pages 1008-1015, Online. Association for Computational Linguistics.
* Fomicheva et al. (2020b) Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frederic Blain, Francisco Guzman, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020b. Unsupervised quality estimation for neural machine translation. _Transactions of the Association for Computational Linguistics_, 8:539-555.
* Kilickaya et al. (2017) Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 199-209, Valencia, Spain. Association for Computational Linguistics.
* Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. volume 37 of _Proceedings of Machine Learning Research_, pages 957-966, Lille, France. PMLR.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lo (2017) Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In _Proceedings of the Second Conference on Machine Translation_, pages 589-597, Copenhagen, Denmark. Association for Computational Linguistics.
* a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 507-513, Florence, Italy. Association for Computational Linguistics.
* Ma et al. (2017) Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined MT metric based on direct assessment -- CASICT-DCU submission to WMT1? metrics task. In _Proceedings of the Second Conference on Machine Translation_, pages 598-603, Copenhagen, Denmark. Association for Computational Linguistics.

 

Pranava Madhyastha, Josiah Wang, and Lucia Specia. 2019. VIFIDEL: Evaluating the visual fidelity of image descriptions. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6539-6550, Florence, Italy. Association for Computational Linguistics.
* Mathur et al. (2019) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2799-2808, Florence, Italy. Association for Computational Linguistics.
* Mathur et al. (2020) Nitika Mathur, Johnny Wei, Qingsong Ma, and Ondrej Bojar. 2020. Results of the wmt20 metrics shared task. In _Proceedings of the Fifth Conference on Machine Translation_, pages 686-723, Online. Association for Computational Linguistics.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _CoRR_, abs/1301.3:1-12.
* Moura et al. (2020) Joao Moura, miguel vera, Daan van Stigt, Fabio Kepler, and Andre F. T. Martins. 2020. Ist-unbabel participation in the wmt20 quality estimation shared task. In _Proceedings of the Fifth Conference on Machine Translation_, pages 1027-1034, Online. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Peters et al. (2019) Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 43-54, Hong Kong, China. Association for Computational Linguistics.
* Ranasinghe et al. (2020) Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. Transquest at wmt2020: Sentence-level direct assessment. In _Proceedings of the Fifth Conference on Machine Translation_, pages 1047-1053, Online. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4512-4525, Online. Association for Computational Linguistics.
* Rubner et al. (2000) Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. 2000. The earth mover's distance as a metric for image retrieval. _International Journal of Computer Vision_, 40(2):99-121.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Shah et al. (2015) Kashif Shah, Trevor Cohn, and Lucia Specia. 2015. A bayesian non-linear method for feature selection in machine translation quality estimation. In _Machine Translation 29_, pages 79-84.
* Specia et al. (2020) Lucia Specia, Frederic Blain, Marina Romicheva, Eric Fonseca, Vishrav Chaudhary, Francisco Guzman, and Andre F. T. Martins. 2020. Findings of the wmt 2020 shared task on quality estimation. In _Proceedings of the Fifth Conference on Machine Translation_, pages 741-762, Online. Association for Computational Linguistics.
* a translation quality estimation framework. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 79-84, Sofia, Bulgaria. Association for Computational Linguistics.
* Thakur et al. (2020) Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020. Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. _CoRR_, abs/2010.08240.
* Vedantam et al. (2014) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2014. Cider: Consensus-based image description evaluation. _CoRR_, abs/1411.5726.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. _CoRR_, abs/1906.08237 

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. CoRRabs/1904.09675.
* Zhang et al. (2020) Zachariah Zhang, Jingshu Liu, and Narges Razavian. 2020. BERT-XML: Large scale automated ICD coding using BERT pretraining. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, Online. Association for Computational Linguistics.
* Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. CoRRabs/1909.02622.

 

Appendix

### Comparison to Sentence Mover's Distance

Sentence Mover's Distance (SSMD) Zhao et al. (2019) is an alternative sentence level metric which for sentence semantic similarity. It compares two text documents using sentence embeddings which are not semantically fine-tuned but based on averaging or pooling the sentences' combined contextual word embeddings. The SMD is defined as follows:

\[SMD(x^{n},y^{n}):=\|E(x_{1}^{l_{x}})-E(y_{1}^{l_{y}})\| \tag{9}\]

where \(E\) is the embedding function which maps an n-gram to its vector representation, \(l_{x}\) and \(l_{y}\) are the size of sentences. As a comparison, we experimented with the linear combination between SMD and each of our token-level metrics - WMD and BERTScore. The metrics performances for WMT-17 in both cases of SRC-MT and MT-REF, and WMT-20 SRC-MT are shown in Table 8, Table 9 and Table 10.

The overall performance of this metric is inferior to that of SSS, which is to be expected since this is simply averaging token-level embeddings. Similar to our SSS, the SMD metric performance improves when it is combined with token-level metrics. The combined metrics' performance drops when there is a big difference between the scores of the two combined metrics, e.g. more than 10%. To pick an example, in Table 8 the gap between BERTScore and SMD for **zh-en** is 0.115, and the combined SMD + BERTScore only reaches a score of 0.503, compared to 0.51 from BERTScore alone. For other languages with closer BERTScore and SMD scores, the performance of the combined metric remains the same or improves, for example, **ru-en**.

 



\begin{table}
\begin{tabular}{|l|c c c c c c c c c|} \hline Metrics & ne-en & en-de & et-en & en-zh & ro-en & si-en & ru-en & Avg \\ \hline WMD & 0.361 & 0.456 & 0.463 & 0.251 & 0.647 & 0.308 & 0.315 & 0.400 \\ BERTScore & 0.357 & **0.459** & **0.460** & 0.260 & **0.673** & 0.309 & 0.320 & 0.405 \\ SMD & 0.436 & 0.368 & 0.302 & 0.277 & 0.570 & 0.298 & 0.281 & 0.362 \\ \hline SMD + WMD & **0.452** & 0.423 & 0.401 & 0.279 & 0.618 & 0.355 & 0.326 & 0.408 \\ SMD + BERTScore & 0.449 & 0.439 & 0.413 & **0.289** & 0.638 & **0.363** & **0.327** & **0.417** \\ \hline \end{tabular}
\end{table}
Table 10: Pearson Correlation with human scores in WMT-20 SRC-MT case with Sentence Mover’s Distance.

\begin{table}
\begin{tabular}{|l|c c c c c c c c c|} \hline Metrics & de-en & zh-en & fi-en & lv-en & ru-en & cs-en & tr-en & Avg \\ \hline WMD & 0.730 & 0.769 & 0.827 & 0.736 & 0.733 & 0.698 & 0.770 & 0.752 \\ BERTScore & 0.745 & **0.775** & 0.833 & 0.756 & 0.746 & 0.710 & 0.751 & 0.759 \\ SMD & 0.703 & 0.686 & 0.763 & 0.693 & 0.698 & 0.648 & 0.644 & 0.691 \\ \hline SMD + WMD & 0.745 & 0.757 & 0.832 & 0.750 & 0.736 & 0.705 & **0.753** & 0.754 \\ SMD + BERTScore & **0.757** & 0.771 & **0.846** & **0.764** & **0.752** & **0.717** & **0.752** & **0.766** \\ \hline \end{tabular}
\end{table}
Table 9: Pearson Correlation with human scores in WMT-17 MT-REF case with Sentence Mover’s Distance.