SlowGuess/ACL_2020_OCR huggingface dataset:
Marina Fomincheva<sup>1</sup> Lucia Specia<sup>1,2</sup> Francisco Guzmán<sup>3</sup>
$^{1}$ Department of Computer Science, University of Sheffield, UK
$^{2}$ Department of Computing, Imperial College London, UK
$^{3}$ Facebook AI, Menlo Park, CA, USA
m.fomicheva@sheffield.ac.uk
l.specia@imperial.ac.uk
Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. One of the main challenges is the fact that multiple outputs can be equally valid. Attempts to minimise this issue include metrics that relax the matching of MT output and reference strings, and the use of multiple references. The latter has been shown to significantly improve the performance of evaluation metrics. However, collecting multiple references is expensive and in practice a single reference is generally used. In this paper, we propose an alternative approach: instead of modelling linguistic variation in human reference we exploit the MT model uncertainty to generate multiple diverse translations and use these: (i) as surrogates to reference translations; (ii) to obtain a quantification of translation variability to either complement existing metric scores or (iii) replace references altogether. We show that for a number of popular evaluation metrics our variability estimates lead to substantial improvements in correlation with human judgements of quality by up  $15\%$ .
Translation is an open-ended task with multiple valid solutions. There are often multiple equivalent translations for the same source sentence. This is due to inherent differences between languages and various sources of ambiguity, which is often impossible to solve without access to additional context. Furthermore, the source might suffer substantial changes in translation due to translator's need to adapt it to the target audience. With rare exceptions, translations are not literal, they can differ from the source text at any linguistic level – lexical, syntactic, semantic or even discourse – and still be considered correct. The ability to produce non-literal, more natural translations is one of the goals in the field of Machine Translation (MT).
Neural MT (NMT) approaches have certainly made significant progress in this direction.
However, the diversity of possible outcomes makes it harder to evaluate MT models. Evaluation metrics (or humans in the case of monolingual manual evaluation) are given a single reference translation against which to compare the MT output. Fomicheva and Specia (2016) found differences of up to 1 point on a 1-5 point quality scale (i.e.  $20\%$ ) between groups of annotators who use different references for manual evaluation. In automatic evaluation, which computes a similarity score between MT output and human reference, they found differences of up to 6 BLEU points depending on the reference used, showing that metrics strongly penalise perfectly correct translations that happen to be different from the reference provided.
Dreyer and Marcu (2012) showed that if multiple human translations are used, any automatic MT evaluation metric achieves a substantially higher correlation with human judgments. However, multiple translations are hardly ever available in practice due to the cost of collecting them.
Alternatives strategies for modelling linguistic variation in automatic MT evaluation include using paraphrasing, synonyms, or comparing linguistic structures of MT output and the reference translation (e.g. semantic role labels) instead of surface forms  $(\S 2)$ . It is worth noticing that this line of work focuses on varying the reference translation. No existing work accounts for the diversity of possible MT outputs.
Instead of using multiple references or relaxing the string matching process, we use the MT system to generate multiple additional hypotheses representing potentially valid translation variations. We do so by exploring model uncertainty in output probability distributions. To generate a diverse set
<sup>1</sup>We focus on sentence-level evaluation, as system-level
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1218-1232 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Hypothetical similarity space where a low-quality MT output (left) and a high-quality MT output (right) are equally distant from the reference but can be distinguished based on the similarity to additional MT hypotheses.
of hypotheses from neural MT (NMT) systems, we leverage recent work on uncertainty quantification for neural networks (§3).
The additional hypotheses produced for a given source sentence are then used for evaluation with or without human references. Intuitively, if some of the hypotheses match the reference, it is probable that the MT output under evaluation is also of high quality. Furthermore, we posit that the differences between system hypotheses produced for the same source capture uncertainty. The more similar they are among themselves, the higher the confidence of the model. As illustrated in Figure 1, this could provide additional information for discriminating translation quality when measuring the distance to the reference translation does not suffice. We devise various new metrics based on this intuition and obtain large improvements in correlation with human judgments over traditional reference-based metrics.
Our main contributions are as follows: (1) We study different ways to generate additional MT hypotheses by exploring uncertainty in NMT models. We show that a light-weight Bayesian approximation method – Monte Carlo Dropout, which allows for uncertainty quantification by using dropout at inference time (Gal and Ghahramani, 2016) – works the best for the purpose of automatic MT evaluation; (2) We devise methods to effectively explore multiple MT hypotheses to better evaluate MT output quality with existing evaluation metrics. On two different datasets, we achieve a large improvement in correlation with human judgments
over using both single reference and multiple references. To the best of our knowledge, this is the first work to leverage NMT model uncertainty for automatic MT evaluation.
Meteor (Banerjee and Lavie, 2005) was the first MT evaluation metric to relax the exact match constraint between MT system output and reference translation by allowing matching of lemmas, synonyms or paraphrases. However, this requires linguistic resources which do not exist for most languages. Character-based metrics (Popovic, 2015; Wang et al., 2016) also relax the exact word match constraint by allowing the matching of characters. However, ultimately they still assume a surface-level similarity between reference and MT.
A more recent direction compares MT and reference sentences in the embedding space. Chen and Guo (2015) extract word embedding representations for the two sentences and measure the (cosine) similarity between them. Similarly, in (Fomicheva et al., 2015; Servan et al., 2016; Tattar and Fishel, 2017) two words are considered to match if their cosine distance in the embedding space is above a certain threshold. The embeddings are thus used to provide a binary decision. MEANT 2.0 (Lo, 2017) and YISI (Lo, 2019) also relies on matching of words in the embedding space, but this is only used to score the similarity between pairs of words that have already been aligned based on their semantic roles, rather than to find the alignments between words. Finally, Chow et al. (2019) and Echizen'ya et al. (2019) perform the alignment in the embedding space using Earth Mover's Distance with some special treatment for word order. All of these metrics are however still limited to variance in the words used (even in the continuous space),
automatic evaluation can be by and large considered a solved problem (Ma et al., 2019a).
2The goal of this paper is not to evaluate the search space of the MT system, but to improve the evaluation of the given MT output by using additional hypotheses. Evaluating the NMT search space beyond the generated output could be an interesting direction to explore in future work.
rather than more general stylistic or structural variations which can only be captured with multiple references.
Another way of incorporating linguistic variation is pseudo-reference approach by Albrecht and Hwa (2007). They leverage various off-the-shelf MT systems to generate additional imperfect references and use them instead or alongside the original reference during evaluation. Evaluation scores obtained using each of the pseudo references and the available human references are combined as features by training a classifier to predict human judgments. Thus, this line of work implicitly learns the quality of the MT systems used to generate pseudo references. We revisit this idea in our paper by having pseudo-references as one type of diverse MT output.
3 Generating Diverse Hypotheses
We posit that using multiple MT hypotheses can help automatic MT evaluation in two ways. First, the difference between them may reflect model confidence and potential ambiguity or complexity of the source. Second, they provide an additional point of comparison with the reference, such that if the initial MT output is different from the provided reference due to acceptable linguistic variation, the risk of over-penalising this translation is lower.
3.1 Neural MT
Most recent work on NMT is based on the sequence-to-sequence approach with encoder and decoder networks (Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017b). In these models probability of generating the output sequence  $\vec{y}$  given the input sequence  $\vec{x}$  is decomposed as follows:
$$
p (\vec {y} | \vec {x}, \theta) = \prod_ {j = 1} ^ {J} p (y _ {j} | \vec {y} _ {<   j}, \vec {x}, \theta)
$$
where  $\theta$  represents model parameters. The decoder produces the probability distribution  $p(y_{j}|\vec{y}_{< j},\vec{x},\theta)$  over system vocabulary at each time step using softmax function.
In this work we use state-of-art Transformer architecture proposed by Vaswani et al. (2017b), an encoder-decoder model that uses stacked self-attention and fully connected layers for both encoder and decoder.
3.2 Search Algorithm
One way to obtain multiple MT hypotheses is by taking top MT hypotheses resulting from the search algorithms used in NMT for decoding.
Beam Search. Hypotheses spaces in NMT are very large and it is not feasible to explore them exhaustively. Beam search is traditionally used for decoding in NMT by exploring the search space in a greedy left-to-right manner retaining the top-N candidates with the highest probability. While effective to select a likely translation, beam search tends to result in a list of N-best translations which lack linguistic diversity (Vijayakumar et al., 2016).
Diverse Beam Search. Vijayakumar et al. (2016) proposed the Diverse Beam Search algorithm to improve the diversity of top hypotheses. The algorithm promotes diversity by optimising a diversity-augmented objective.
3.3 Uncertainty
We propose that a better method for obtaining diverse MT hypotheses for automatic MT evaluation is by exploiting uncertainty in NMT. For the intuition, consider three different cases. First, if there is only one correct translation at each time step, the output probabilities will have "peakier" distributions with low entropy and a single word receiving a large portion of the probability mass. In this case, there is very little variation in the hypotheses space. Second, if there are various correct translation options at a given generation step, the output probability distribution will have higher entropy, with multiple target words receiving similar probabilities. In this case, generating hypotheses from the model will result in similar sentences containing synonyms or paraphrases. Finally, if the NMT model has not seen enough data during training for a given combination of words, we would expect output probabilities to exhibit high entropy, approximating a uniform probability distribution. In this case, generating MT hypotheses from the model should result in a highly diverse set with lower quality translations.
Below we explore various approaches to uncertainty quantification in neural networks in order to generate a set of additional hypotheses for MT evaluation.
Monte Carlo Dropout. It has been shown that softmax function used in neural networks to generate output probability distribution does not properly
capture uncertainty as it produces overconfident predictions (Gal and Ghahramani, 2016). Most of the work on uncertainty quantification in deep learning relies on Bayesian formalism (MacKay, 1992; Graves, 2011; Welling and Teh, 2011; Gal and Ghahramani, 2016; Tran et al., 2019). Representing uncertainty through Bayesian neural networks usually comes with prohibitive computational costs and various approximations have been developed to alleviate this issue. One such approximation by Gal and Ghahramani (2016) is called Monte Carlo (MC) dropout. Dropout is a method developed to reduce overfitting when training neural models Srivastava et al. (2014). It consists in randomly masking neurons to zero based on Bernoulli distribution. Gal and Ghahramani (2016) use dropout at test time before every weight layer. They perform  $N$  forward passes through the network and collect posterior probabilities generated by the model with parameters perturbed by dropout:  $\{\hat{p} (\vec{y} |\vec{x},\hat{\theta})_{i = 1}^{N}\}$  where  $\hat{\theta}$  represents the perturbed parameters. They show that this is equivalent to an approximation to the probabilistic deep Gaussian process. Previous work has applied this method to quantify model uncertainty by taking the variance of the resulting probability distribution (Dong et al., 2018; Wang et al., 2019). We instead look at the linguistic differences between MT hypotheses generated as a result of  $N$  forward passes through the model with perturbed parameters. If the top MT output for a given source sentence is of high quality, it is probable that other hypotheses will be similar.
Ensembling. Ensemble model combination is another strategy commonly used for estimating predictive uncertainty (Lakshminarayanan et al., 2017; Pearce et al., 2018; Liu et al., 2019). We take an assembling strategy typically applied in NMT to improve translation quality: we train four NMT models initialised with different random seeds. At decoding time, prediction distributions from the four models are combined by averaging. To generate additional hypotheses, the four models in the ensemble are used separately, each generating an independent set of translations.
Mixture of Experts. Shen et al. (2019) applied mixture of experts (MoE) framework to capture the inherent uncertainty of the MT task and generate diverse hypotheses. A mixture model introduces a multinomial latent variable  $z \in 1, \dots, K$ . The
marginal likelihood is then decomposed as:
$$
\begin{array}{l} p (\vec {y} | \vec {x}; \theta) = \sum_ {z = 1} ^ {K} p (\vec {y}, z | \vec {x}; \theta) \\ = \sum_ {z = 1} ^ {K} p (z | \vec {x}; \theta) p (\vec {y} | z, \vec {x}; \theta) \\ \end{array}
$$
The model is trained with the EM algorithm where the E-step estimates the responsibilities of each mixture component ("expert") and M-step updates parameters  $\theta$  with gradients weighted by their responsibilities. For our experiments, one of the mixture components was randomly selected to produce the MT output for human evaluation and the rest of them were used for the generation of additional hypotheses (§5).
3.4 Pseudo-Reference Translations
Here we revisit the approach previously used for statistical MT (Albrecht and Hwa, 2007) where outputs of other off-the-shelf MT systems are used as additional reference translations, with some differences. First, NMT outputs on average have substantially higher quality. Second, to avoid the need for labelled data, we do not rely on supervised training and treat the outputs of other MT systems in the same way we treat additional hypotheses that were produced using the methods described in the previous sections. We use publicly available online NMT systems (§5).
4 Scoring with Multiple Hypotheses
Using the methods described above we are able to produce a set of MT hypotheses for each given source segment. The final dataset which we use for evaluation contains a human reference translation  $(r)$ , the top MT output  $(o)$  and this set of alternative  $N$  MT hypotheses  $(H = \{h_{1}..h_{N}\})$ . We devise the following ways of combining similarities between possible translations and between these and the reference to obtain more accurate evaluation. This accuracy will be measured by Pearson correlation with a direct assessment (DA) score collected for the  $o$  translation, as is common practice in the evaluation metrics field (Ma et al., 2019b).
4.1 Addressing Linguistic Variation
Here we compute the similarity against the reference translation for the set of all generated translation candidates, including the initial MT output and additional hypotheses, and take the average
similarity score (micro-average). If the MT output is of high quality but does not match the provided human reference due to acceptable linguistic variation, other hypotheses may serve as paraphrases to match the reference.
However, it is important to assign a higher weight to the MT output that was actually evaluated  $(o)$ , as compared to the alternative MT hypotheses. This is done using a simple variant of the above metric where we first take an average of the hypotheses-reference similarities, and then average this score with the MT output-reference similarity score (macro-average). This results in two metrics:
$$
\operatorname {h y p - r e f} _ {\text {m i c r o}} ^ {*} = N ^ {- 1} \sum_ {i = 1} ^ {N + 1} \operatorname {s i m} \left(h _ {i} ^ {\prime}, r\right), h _ {i} ^ {\prime} \in H ^ {\prime}
$$
$$
\operatorname {h y p - r e f} _ {\text {m a c r o}} ^ {*} = \frac {N ^ {- 1} \sum_ {i = 1} ^ {N} \operatorname {s i m} (h _ {i} , r) + \operatorname {s i m} (o , r)}{2}
$$
where  $H' = \{h_1', h_N', o\}$  is a set including additional hypotheses and the MT output, and sim corresponds to a similarity function of choice (§4.3). The * represents different ways of combining hypotheses-reference similarities: average (as shown in the equations above), minimum (i.e. choosing the score for the most distant hypothesis) and maximum (i.e. choosing the score of the closest hypotheses).
4.2 Incorporating Model Uncertainty
As discussed in Section §3.3, similarity between translation hypotheses capture model confidence and could thus be indicative of translation quality. We propose two metric variants to capture this idea. First, we compute the similarity between all translations candidates including the additional hypotheses and the MT output:
$$
\mathrm {h y p - s e l f} ^ {*} = \frac {1}{C} \sum_ {i = 1} ^ {| H ^ {\prime} |} \sum_ {j = 1} ^ {| H ^ {\prime} |} \operatorname {s i m} \left(h _ {i} ^ {\prime}, h _ {j} ^ {\prime}\right)
$$
where  $h_i' \in H'$ ,  $i \neq j$  and  $C = 2^{-1}|H'|(|H'| - 1)$  is the number of pairwise comparisons for  $H'$  hypotheses. As before,  $*$  corresponds to different ways of combining similarity scores: average, minimum and maximum.
Second, as before, we give a higher weight to the MT output whose quality we wish to evaluate  $(o)$ . To that end we compare the MT output against additional generated hypotheses. This comparison
indirectly captures the similarity between MT hypotheses themselves:
$$
\mathrm {h y p - m t} ^ {*} = N ^ {- 1} \sum_ {i = 1} ^ {N} s i m (h _ {i}, o)
$$
Both of these variants can be used with and without reference translation. Interestingly, as will be shown in §6.2, they perform comparably to other methods even without the reference, putting into question the need for human reference in MT evaluation. As in the previous section, to add human reference translations into the mix, we average the results as follows:
$$
\mathrm {h y p - m t} ^ {*} \text {- r e f} = \frac {N ^ {- 1} \sum_ {i = 1} ^ {N} s i m (h _ {i} , o) + s i m (o , r)}{2}
$$
Figure 2 summarises the methods discussed above.
Figure 2: Methods to explore similarities between MT output, system hypotheses and references.
4.3 Similarity Functions
To measure similarity amongst hypotheses and against the reference(s), we experiment with the following standard MT evaluation metrics:
sentBLEU (Papineni et al., 2002). BLEU measures the similarity between MT and the reference translation based on the number of matching n-grams. We use a smoothed version of BLEU as described by Lin and Och (2004) with  $N = 4$ .
3We use these metrics out of the box. Better results could possibly be achieved by adapting them to our settings, e.g. by changing the weight of precision and recall depending on the direction of the comparison between MT output, hypotheses and the reference. For instance, when using BLEU as similarity function for computing  $\mathrm{hyp - mt}^*$ , we are evaluating recall on the MT output, whereas BLEU is designed as a precision-oriented metric. But the choice of similarity function is orthogonal to the goal of this paper, and we leave further refinements in this direction to future work.
TER (Translation Edit Rate) (Snover et al., 2006). TER computes the edit distance defined as the minimum number of word substitutions, deletions, insertions and shifts that are needed to convert MT into the reference.
ChrF (Popovic, 2015). ChrF calculates the F-score of character n-grams of maximum length 6.
Meteor (Denkowski and Lavie, 2014). Meteor aligns MT output to the reference translation using synonyms and paraphrases besides exact word matching. The similarity is based on the proportion of aligned words in the candidate and in the reference and a fragmentation penalty.
BERTScore. (Zhang et al., 2019). We also looked at this very recent metric (published after the submission of this paper), which uses powerful pre-trained embeddings. BERTScore computes a cosine similarity score for each token in the MT output with each token in the reference sentence using contextual embeddings from BERT (Devlin et al., 2019), which can generate different vector representations for the same word depending on the context, thus better capturing meaning. Maximum similarity values for MT and reference words are then used to compute a soft F1-score. We use the implementation available at https://github.com/Tiiger/bert_score.
To test whether our methods improve correlation with human judgments, we need to have access to the NMT model and human judgments for the translations generated by this model. This data is not generally readily available in evaluation campaigns such as Metrics Task at WMT conferences. Below we describe two datasets that satisfy these conditions. They cover two different language pairs and two different domains.
News English-Czech dataset. We use available data from the WMT19 News Translation Task. We focus on the University of Edinburgh's submission (Bawden et al., 2019) to the English-Czech translation task, since its NMT model is available. The system was trained using the MarianMT toolkit with a standard Transformer architecture (Vaswani et al., 2017a). Details on model training and architecture are described in (Bawden et al., 2019). For producing pseudo-references, we use all five
"online" systems whose submissions were provided as part of the WMT19 Translation Task.
Human judgments were collected in the form of Direct Assessments (DA) following the methodology proposed by Graham et al. (2015), which suggests that 15 segment-level DA judgements are required for trustworthy correlation analysis. However the number of DA judgements in the WMT19 Metrics Task was much smaller. We select segments with at least two DA annotations (795 segments with an average DA score of 80.22) to minimise this issue, but the results reported here for English-Czech should be interpreted with caution.
Wikipedia Estonian-English dataset. This is a new dataset we collected which contains 1K sentences randomly selected from Wikipedia articles in Estonian and translated into English. Two human reference translations were generated independently by two professional translators.