SlowGuess/NAACL_2021_OCR huggingface dataset:
Yurun Song*
Imperial College London UK
yurun.song19@imperial.ac.uk
Junchen Zhao*
University of California, Irvine CA, US
junchez3@uci.edu
Machine translation (MT) is currently evaluated in one of two ways: in a monolingual fashion, by comparison with the system output to one or more human reference translations, or in a trained crosslingual fashion, by building a supervised model to predict quality scores from human-labeled data. In this paper, we propose a more cost-effective, yet well performing unsupervised alternative SentSim: relying on strong pretrained multilingual word and sentence representations, we directly compare the source with the machine translated sentence, thus avoiding the need for both reference translations and labelled training data. The metric builds on state-of-the-art embedding-based approaches – namely BERTScore and Word Mover's Distance – by incorporating a notion of sentence semantic similarity. By doing so, it achieves better correlation with human scores on different datasets. We show that it outperforms these and other metrics in the standard monolingual setting (MT-reference translation), a well as in the source-MT bilingual setting, where it performs on par with glass-box approaches to quality estimation that rely on MT model information.
Automatically evaluating machine translation (MT) as well as other language generation tasks has been investigated for decades, with substantial progress in recent years due to the advances of pretrained contextual word embeddings. The general goal of such evaluation metrics is to estimate the semantic equivalence between the input text (e.g. a source sentence or a document) and an output text that has been modified in some way (e.g. a translation or summary), as well as the general quality of the output (e.g. fluency). As such, by definition metrics should perform some forms of input-output comparisons.
However, this direct comparison has been proven hard in the past because of the natural differences between the two versions (such as different languages). Instead, evaluation metrics have resorted to comparison against one or more correct outputs produced by humans, a.k.a. reference texts, where comparisons at the string level are possible and straightforward. A multitude of evaluation metrics have been proposed following this approach, especially for MT, the application we focus on in this paper. These include the famous BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) for machine translation, ROUGE (Lin, 2004) for summarization, and CIDER (Vedantam et al., 2014) for image captioning. These traditional metrics are based on simple-word, n-gram matching mechanisms or slight relaxations of these (e.g. synonyms) which are computationally efficient, but suffer from various limitations.
In order to overcome the drawbacks of the traditional string-based evaluation metrics, recent work (Williams et al., 2018; Bowman et al., 2015; Echizen'ya et al., 2019; Cer et al., 2017; Echizen'ya et al., 2019) has investigated metrics that perform comparisons in the semantic space rather than at the surface level. Notably, applications of Word Mover's Distance (WMD; Kusner et al., 2015), such as WMDo (Chow et al., 2019), VIFIDEL (Madhyastha et al., 2019) and moverscore (Zhao et al., 2019), which compute similarity based on continuous word embeddings using pretrained representations. These have been shown to consistently outperform previous metrics on various language generation evaluation tasks.
However, these metrics have two limitations: (i) they still rely on reference outputs, which are expensive to collect, only cover one possible correct answer, and do not represent how humans do evaluation; (ii) they are bag-of-embeddings approaches which capture semantic similarity at the token level, but are unable to capture the meaning of the sen
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3143-3156 June 6-11, 2021. ©2021 Association for Computational Linguistics
tence or text as a whole, including correct word order.
In this paper, focusing on MT, to address these limitations we first posit that evaluation can be done by directly comparing the source to the machine translation using multilingual pretrained embeddings, such as multilingual BERT, avoiding the need of reference translations. We note that this is different from quality estimation (QE) metrics (Specia et al., 2013; Shah et al., 2015), which also compare source and machine translated texts directly, but assume an additional step of supervised learning against human labels for quality. Second, we introduce Sentence Semantic Similarity (SSS), an additional component to be combined with bag-of-embeddings distance metrics such as BERTScore. More specifically, we propose to explore semantic similarity at the sentence level - based on sentence embeddings (Sellam et al., 2020; Reimers and Gurevych, 2020; Thakur et al., 2020) - and linearly combine it with existing metrics that use word embeddings. By doing so, the resulting metrics have access to word and compositional semantics, leading with improved performance. The combination is a simple weighted sum, and does not require training data.
As a motivational example, consider the case in Table 1, from the WMT-17 Metrics task (Zhang et al., 2019). When faced with MT sentences that contain a negated version of the reference (MT3 and MT4), token-level metrics such as BERTScore and WMD cannot correctly penalize these sentences since they match representations of words in both versions without a full understanding of the semantics of the sentences. As a consequence, they return a high score for these incorrect translations, higher than the score for correct paraphrases of the reference (MT1 and MT2). Sentence similarity, on the other hand, correctly captures this mismatch in meaning, returning relatively lower scores for Translations 3 and 4. However on their own they may be too harsh, since the remaining of the sentence has the same meaning. The combination of these two metrics (last column) balances between these two sources of information and, as we will later show in this paper, has higher correlation with human scores.
1. We investigate and show the effectiveness of linearly combining sentence-level semantic similarity with different metrics using token
level semantic similarity. The resulting combined metric, SentSim, consistently achieves higher Pearson Correlation with human judgements of translation quality than both word and sentence similarity alone.
2. We show, for the first time, that these metrics can be effective when comparing system-generated sentences directly against source sentences, in a crosslingual fashion.
3. Our SentSim metric outperforms existing metrics on various MT datasets in monolingual and crosslingual settings.
Various natural language generation tasks, including machine translation, image captioning, among others, produce sentences as output. These are evaluated either manually or automatically by comparison against one or multiple reference sentences. A multitude of metrics have been proposed for the latter, which perform comparisons at various granularity levels, from characters to words to embedding vectors. The goal of such metrics is to replace human judgements. In order to understand how well they fare at this task, metrics are evaluated by how similar their scores are to human assigned judgements on held-out datasets. For absolute quality judgements, Pearson Correlation is the most popularly used metric for such a comparison (Mathur et al., 2020).
Recent studies have showed that the new generation of automatic evaluation metrics, which instead of lexical overlap are based on word semantics using continuous word embedding, such as BERT (Devlin et al., 2019), ElMo (Peters et al., 2019), XLNet (Yang et al., 2019) or XLM-Roberta (Conneau et al., 2019), have significantly higher Pearson Correlation with the human judgements when comparing reference sentences with system generated sentences. Zhang et al. (2019) introduce BERTscore, an automatic evaluation metric based on contextual word embeddings, and tests it for text generation tasks such as machine translation and imaging captioning, using embeddings including BERT, XLM-Roberta, and XLNet (more details in Section 3.2). Mathur et al. (2019) present supervised and unsupervised metrics which are based on BERT embeddings for improving machine translation evaluation. Zhao et al. (2019) introduce moverscore, a metric which generates high-quality evaluation
BERTScore	SSS	SSS + BERTScore
REF	We have made a complete turnaround.
MT1	We did a complete turnaround.	0.7975	0.9578	0.8111
MT2	We made a turnaround.	0.7748	0.8898	0.7427
MT3	We have not made a complete turnaround.	0.8296	0.3878	0.4832
MT4	We have made an incomplete turnaround.	0.8318	0.4431	0.5107
Table 1: An example from the WMT-17 dataset. Given the reference (REF) sentence, BERTScore assigns higher similarity to its negated versions (MT3 and MT4) than to semantically similar variants (MT1 and MT2). Contrarily, SSS gives a very low score to MT3 and MT4. Their combination provides a more balanced score.
results on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, using BERT embeddings. Clark et al. (2019) present semantic metrics for text summarization based on the sentence mover's similarity and ELMo embeddings. Chow et al. (2019) introduce a fluency-based word mover's distance (WMDo) metric for machine translation evaluation using Word2Vec embeddings (Mikolov et al., 2013). Lo (2019) presents Yisi, a unified automatic semantic machine translation quality evaluation and estimation metric using BERT embeddings.
There is also a bulk of work on metrics that take a step further to optimize their scores using machine learning algorithms trained on human scores for quality (Sellam et al., 2020; Ma et al., 2017). They often perform even better, but the reliance on human scores for training, in addition to reference translations at inference time, makes them less applicable in practice. A separate strand of work that relies on contextual embeddings is that of Quality Estimation (Moura et al., 2020; Fomicheva et al., 2020a; Ranasinghe et al., 2020; Specia et al., 2020). These are also trained on human judgements of quality, but machine translations are compared directly to the source sentences rather than against reference translations.
In addition to embeddings for words, embeddings for full sentences have been shown to work very well to measure semantic similarity. These are extracted using Transformer models that are specifically trained for capturing sentence semantic meanings using BERT, Roberta, and XLM-Roberta embeddings (Reimers and Gurevych, 2019; Reimers and Gurevych, 2020; Thakur et al., 2020) and provide state-of-art performance pretrained models for many languages. $^{1}$
In this paper, we take inspiration from these lines
of previous works to propose unsupervised metrics that combine word and sentence semantic similarity and show that this can be effective for both MT-reference and source-MT comparisons.
In this section, we first describe in more detail the metrics that we have used in our experiments, namely semantic sentence cosine similarity, WMD and BERTScore. Then we present our simple approach to linearly combine these metrics.
3.1 Word Mover's Distance (WMD)
Kusner et al. (2015) presents word mover's distance (WMD) metric, a special case of Earth mover's distance (Rubner et al., 2000), computing the semantic distance between two text documents by aligning semantically similar words and capturing the word traveling flow between the similar words utilizing the vectorial relationship between their word embeddings (Mikolov et al., 2013). WMD has been proven to generate consistently high-quality results for the tasks of measuring text similarity and text classification (Kusner et al., 2015). A text document is represented as a vector  $D$ , where each element is denoted as the normalized frequency of a word in the document such that:
$$
D = \left[ d _ {1}, d _ {2}, \dots ., d _ {n} \right] ^ {T} \tag {1}
$$
where  $d_{i} = c_{i} / \sum_{j}^{n}c_{j}$  and  $c_{i}$  is the frequency that the  $i^{th}$  word which appears  $c_{i}$  times in a given text document. Assuming there are two given words from different text document denoted as  $i$  and  $j$ , then the euclidean distance in the embedding  $x_{i}$  and  $x_{j}$  for the two words is defined as:
$$
c (i, j) = \left\| x _ {i} - x _ {j} \right\| _ {2} \tag {2}
$$
where  $c(i,j)$  is defined as the "word traveling cost" from  $x_{i}$  in one document to  $x_{j}$  in the other document. Now, assuming there are two documents,
<sup>1</sup>https://github.com/UKPLab/sentence-transformers
one is the source document denoted as  $A$  where the word  $i$  belongs to, and another one is the target document denoted as  $B$  where the word  $j$  belongs to. A flow matrix  $T$  is defined in which every element is denoted as  $T_{ij}$ , suggesting the number of times the word  $i$  in document  $A$  moves to the word  $j$  in document  $B$ . Then, the value of the flow matrix is normalized based on the total count of words in the vocabulary such that:
$$
\sum_ {j} T _ {i j} = d _ {i}, \sum_ {i} T _ {i j} = d _ {j} \tag {3}
$$
The semantic distance calculated by WMD can be then defined as follows:
$$
\mathrm {W M D} = \min  _ {T \geq 0} \sum_ {i, j = 1} ^ {n} T _ {i j} c (i, j) \tag {4}
$$
WMD, or the semantic distance between two text documents, can thus be computed by optimizing values in the flow matrix  $T$ . In other words, WMD corresponds to the minimal semantic distance to move semantically similar words (via their embeddings) from one text document to another.
3.2 BERTScore
BERTScore (Zhang et al., 2020) is designed to evaluate semantic similarity between sentences in the same language, namely a reference sentence and a machine-generated sentence. Assume a reference sentence is denoted as  $x = (x_{1},\dots,x_{k})$  and a candidate sentence is denoted as  $\hat{x} = (\hat{x}_1,\dots,\hat{x}_k)$ . BERTScore uses contextual embeddings such as BERT (Devlin et al., 2019) or ELMo (Peters et al., 2019) to represent word tokens in the sentences. It finds word matchings between the reference and candidate sentence using cosine similarity, which can be optionally reweighted by the inverse document frequency scores (IDF) of each word. BERTScore matches each word token  $x$  in reference sentence to the closest word token  $\hat{x}$  in candidate sentence for computing recall, and matches each word token  $\hat{x}$  in candidate sentence to the closest word token  $x$  in reference sentence for computing precision. It combines recall with precision to produce an F1 score. However, only recall is used for evaluation in most cases, which is defined as follows:
$$
R _ {B E R T} = \frac {1}{| x |} \sum_ {x _ {i} \in x} \max  _ {\hat {x} _ {j} \in \hat {x}} x _ {i} ^ {T} \hat {x} _ {j} \tag {5}
$$
In essence, BERTScore can be viewed as a hard word alignment given a pair of sentences using contextual embeddings, in which each word is aligned to one other word, the closest in the embedding space according to the cosine distance between their vectors.
3.3 Semantic Sentence Similarity (SSS)
A commonly used method to measure sentence similarity is using the cosine distance between the two vectors summarizing the sentences:
$$
\cos (\theta) = \frac {\alpha \cdot \beta}{\| \alpha \| \| \beta \|} \tag {6}
$$
where  $\alpha$  and  $\beta$  are the vectors representing the two sentences. The higher the value obtained through cosine similarity between two sentences vectors based on the pretrained sentence representation (Reimers and Gurevych, 2019; Reimers and Gurevych, 2020; Thakur et al., 2020), the stronger their similarity.
3.4 SentSim
In order to bring the notion of semantic similarity to token similarity metrics, we combine the sentence cosine similarity using semantically finetuned sentence embedding with the metrics using contextual word embeddings. Assume that the generated score from sentence level metric is denoted as  $A$ , the value generated from token-level metric is denoted as  $B$  and the gold truth from human judgement is denoted as  $S$ . Our combination metric, namely SentSim, is as follows:
$$
\operatorname {S e n t S i m} (A, B) = w _ {1} * e ^ {A} + w _ {2} * e ^ {B} \tag {7}
$$
where  $A$  and  $B$  are normalized to the range between 0 and 1,  $w_{1}$  and  $w_{2}$  are the weights given to two metric scores. If metric  $B$  is negatively correlated with  $S$ , i.e., if it is a distance metric like WMD, we give it  $e^{1 - B}$ . We use  $e^{B}$  for similarity metrics such as SSS and BERTScore.
In equation 7, we apply exponential for similarity scores as the linear addition of two similarity scores  $(A + B)$  in lower-order leads to a large variance and inconsistency in the correlation with human scores. Lower-order models are too simple to fit the relationship between similarities. Therefore, a non-linear model is required to project these similarities into higher-order  $(A^n + B^n)$ . Given the Taylor Series Expansion (Abramowitz and Stegun,
1965) of exponential function, we can get a factorial average of two similarities from lower-order to higher-order as follows:
$$
\operatorname {S e n t S i m} (A, B) = \sum_ {n = 1} ^ {\infty} \frac {w _ {1} * A ^ {n} + w _ {2} * B ^ {n}}{n !} \tag {8}
$$
Our final metric is given in Equation 8, which follows from Equation 7 using Taylor Series Expansion. This was also shown in (Kilickaya et al., 2017; Clark et al., 2019), which convert distance scores to similarities by using the exponential function.
In Section 5, we report experiments with two linear metric combinations: SSS + WMD and SSS + BERTScore, where we give equal weight to each metric ( $w_{1} = w_{2} = 0.5$ ). We have also investigated the linear combination between Sentence Mover's Distance (Zhao et al., 2019) and token-level metrics, but the performance is poorer than SSS, so we only show results in the Appendix A.1.
In this section, we describe two types of experimental scenarios, monolingual and crosslingual evaluation, as well as the three datasets and pretrained embeddings we used.
4.1 Task Scenarios
The first evaluation setting we experimented with is the standard monolingual evaluation task scenario (MT-REF), which takes reference sentences and machine generated sentences in the same language as input. The second one is the crosslingual evaluation task scenario (SRC-MT), which directly assesses the similarity between source sentences and machine generated sentences in different languages. We compute our combined metrics for each task scenario separately.
We use various datasets with absolute human judgements from recent evaluation campaigns.
Multi-30K (Elliott et al., 2016) is a multilingual (English-German (en-de) and English-French (en-fr)) image description dataset. We use the 2018 test set, in which each language pair contains more than 2K sentence tuples, including source sentences, reference sentences, machine generated sentences, and the corresponding human judgement scores in an (0-100) continuous range. Therefore, this
dataset can be used for both crosslingual and monolingual task scenarios.
WMT-17 (Bojar et al., 2017) is a dataset containing multiple language pairs from the WMT News Translation task used for segment-level system evaluation in the Metrics task. We used all seven to-English datasets: German-English (deen), Chinese-English (zh-en), Latvian-English (lv-en), Czech-English (cs-en), Finnish-English (fi-en), Russian-English (ru-en), Turkish-English (tr-en) and two from-English datasets: English-Russian (en-ru), English-Chinese (en-zh). Each language has 560 sentence tuples, where each tuple has a source sentence, a reference sentence and multiple system generated sentences, in addition to a human score varying from 0 to 100. WMT-17 can be used in both monolingual and crosslingual evaluation task scenarios, and is our main experimental data. More recent WMT Metrics task datasets do not report metrics results using absolute judgements, but rather convert these into pairwise judgements. While such relative judgements are useful to assess metrics ability to rank different MT systems, they are not applicable to assess metrics in their ability to estimate quality in absolute terms, which are what we are interested in.
WMT-20 (Fomicheva et al., 2020b) is the dataset used in the WMT20 quality estimation task, where participants are expected to directly predict the translation quality between source sentences and machine generated sentences without using reference sentences. This dataset has seven language pairs: Sinhala-English (si-en), Nepalese-English (ne-en), Estonian-English (eten), English-German (en-de), English-Chinese (enzh), Romanian-English (ro-en), Russian-English (ru-en). We use the test set, witwhere each language pair contains 1K tuples with source and machine generated sentences, as well as human judgements in the 0-100 range. Therefore, with this dataset we can only perform crosslingual evaluation.
4.3 Embeddings
For each language model, we consider embeddings at the token level and sentence level individually and in combination. In our experiments, Roberta-Large and XLM-Roberta-Base for monolingual and crosslingual assessments respectively.
For crossligual embeddings we use XLM-Roberta instead of multilingual BERT (mBERT)
SRC-MT	REF-MT
Metrics	en-de	en-fr	Avg	de-de	fr-fr	Avg
BLEU	-	-	-	0.262	0.387	0.325
METEOR	-	-	-	0.461	0.411	0.436
WMD	0.360	0.319	0.340	0.492	0.425	0.459
BERTScore	0.335	0.291	0.313	0.434	0.352	0.393
SSS	0.483	0.449	0.466	0.487	0.446	0.467
SSS + WMD	0.508	0.477	0.492	0.546	0.501	0.524
SSS + BERTScore	0.486	0.434	0.460	0.527	0.462	0.494
Table 2: Pearson Correlation with human scores for Multi-30K with Roberta-Base in the SRC-MT and MT-REF settings. For the latter we evaluate German to German and French to French as monolingual tasks.
SRC-MT
Metrics	de-en	zh-en	fi-en	lv-en	ru-en	cs-en	en-ru	en-zh	tr-en	Avg
WMD	0.366	0.501	0.373	0.373	0.308	0.267	0.404	0.408	0.350	0.372
BERTScore	0.409	0.510	0.414	0.402	0.337	0.319	0.434	0.446	0.382	0.406
SSS	0.456	0.514	0.540	0.555	0.541	0.464	0.505	0.458	0.540	0.508
SSS + WMD	0.504	0.594	0.566	0.569	0.534	0.476	0.538	0.513	0.562	0.540
SSS + BERTScore	0.523	0.600	0.578	0.574	0.551	0.499	0.553	0.531	0.569	0.553
Table 3: Pearson Correlation with human scores for the WMT-17 with Roberta-Base in the SRC-MT setting.
because the former significantly outperforms the latter (Conneau et al., 2019), as also shown by Reimers and Gurevych (2020) for crosslingual semantic textual similarity (STS) tasks (Cer et al., 2017). For a fair comparison with previous metrics like  $\mathrm{WMD}_0$ , we replaced their original embeddings with XLM-Roberta-Base embeddings.
For the semantic sentence embedding, we used XLM-Roberta-Base embeddings from Sentence Transformer, which were trained on SNLI (Bowman et al., 2015) + MultiNLI (Williams et al., 2018) and then fine-tuned on the STS benchmark training data. These sentence embeddings have been shown to provide good representations of the semantic relationship between two sentences, but they had not yet been tested for machine translation evaluation. Without using semantic embeddings, the performance of SSS is not consistent across different languages pairs given our experimental datasets (see Appendix A.1). XLM-Roberta-Large embeddings are not used in our experiments because they are not available in the pre-trained Sentence Transformer package yet.
For monolingual word and semantic sentence embeddings we use the Roberta-Large model, which has shown the best performance with BERTScore (Zhang et al., 2019).
The evaluation results are presented in this section. Our code and data can be found on github<sup>2</sup>.
5.1 SRC-MT Setting