SlowGuess/ACL_2020_OCR huggingface dataset:
All the NMT models were trained using the Fairesq toolkit based on the standard Transformer architecture (Vaswani et al., 2017a) and the training settings described in Ott et al. (2018). We used publicly available parallel datasets for training the models: the Rapid corpus of EU press releases (Rozis and Skadin, 2017) and Europarl (Koehn, 2005), which amount to around 4M parallel sentences in total.
A set of 400 segments were translated by the model variants described in §3 to assess the impact of uncertainty types. The following settings were used for model variants. For MC dropout we use dropout rate of 0.3, same as for training the basic Transformer model. Additional hypotheses were produced by performing  $N$  stochastic forward passes through the network with dropout, as described in §3. For this analysis we use  $N = 30$ , which was shown to perform well for uncertainty quantification (Dong et al., 2018). We also test how the number of hypotheses affects the results (see Appendix B). For MoE we use hard mixture model with uniform prior and  $K = 5$  mixture components. To produce the translations we generate from a randomly chosen component with greedy search following the settings in Shen et al. (2019). For generating additional hypotheses with beam search the top-K sentences  $\mathrm{K} \in [2..5]$  from the beam were used ( $K = 1$  corresponds to the initial MT output). For pseudo-reference approach we use three online systems: Systran, Google and Bing.
Human judgements were given by professional
translators following the FLORES setup (Guzmán et al., 2019) which presents a form of DA judgements (Graham et al., 2013). The annotators were asked to rate each sentence from 0–100 according to the perceived translation quality. Specifically, the 0–10 range represents an incorrect translation; 11–29, a translation with few correct keywords, but the overall meaning is different from the source; 30–50, a translation with major mistakes; 51–69, a translation which is understandable and conveys the overall meaning of the source but contains typos or grammatical errors; 70–90, a translation that closely preserves the semantics of the source sentence; and 90–100, a perfect translation. Each segment was annotated by up to 6 translators. Raw scores were converted into z-scores, i.e. standardised according to each individual annotator's overall mean and standard deviation. The scores collected for each segment were averaged to obtain the final score.
The judgments were collected for the 1K segments translated by the standard Transformer model and for the 400 segments produced by four MT model variants in §3, resulting in a total of  $1000 + 4 * 400 = 2600$  source-MT pairs annotated with DA judgments. The distribution of DA scores for English-Czech and 1K Estonian-English datasets is shown in the Appendix A. $^{4}$
In this section, we present the results of our experiments for generating additional MT hypotheses (§6.1) and the methods for exploiting similarities between them (§6.2).
6.1 Diverse MT Generation Approaches
We start by comparing the different strategies for generating multiple MT hypotheses described in §3 for the Estonian-English dataset. Note that some variants also produce different top MT outputs  $(o)$ , as they were trained using different architectures or decoding algorithms. As a result we have four sets of DA annotations collected for 400 segments for system variants with different MT outputs: standard Transformer, Transformer with diverse beam search, MoE and ensembling. MT outputs for beam search and MC dropout variants correspond to the same underlying NMT model.
Table 1 presents the results. First, beam search performs the poorest. This is in line with the well known fact that beam suffers from low diversity of produced hypotheses (Vijayakumar et al., 2016). As expected, diverse beam search results in a higher difference in correlation compared to mt-ref. However, it is still outperformed by all other methods that capture model uncertainty, with MC dropout achieving the highest difference in correlation against mt-ref. We note that this is not related to the number of generated hypotheses (see Appendix B for details). We suggest that this is due to the fact that linguistic differences between additional hypotheses for high vs. low-quality MT outputs is more discriminating when the hypotheses are generated using MC dropout for representing model uncertainty (see example in Table 3). The difference in correlation observed between different system variants is not related with the quality of MT outputs, as demonstrated by the average DA scores in Table 1. Pseudo-references also perform very well, potentially due to the high quality of the MT systems used to generate them. We select MC dropout and pseudo-references as the two best performing options to conduct a more detailed analysis below.
6.2 Scoring Approaches
Table 2 shows the results for the 1K Estonian-English dataset and for English-Czech dataset.  ${}^{5}$  mt-ref stands for the standard reference-based evaluation. The remaining methods correspond to those described in §4. The methods pseudo-mt-max and pseudo-mt-max-ref are equivalent to the hyp-mt- * and hyp-mt-*-ref but instead of dropout-based hypotheses, the outputs of other MT systems are used.
For Estonian-English, since we have two human references we compute the correlation for each of them separately (mt-ref-1 and mt-ref-2), as well as in a multi-reference scenario (mt-ref-multi). We use mt-ref-1 to calculate all the remaining methods that involve a reference translation.
Significance of the differences in correlation for the proposed methods with respect to mt-ref-1 and mt-ref-multi is assessed using Hotelling-Williams
For the full set of results see the Appendix C.
In the multi-reference scenario, BLEU score is computed by counting the n-gram matches between the MT output and all references as in (Papineni et al., 2002). For the rest of the metrics, the closest reference is used for each segment to compute the score, as in (Denkowski and Lavie, 2014).
4The dataset and the NMT models required to reproduce our results are available at https://github.com/facebookresearch/mlqe/tree/master/data-multi-hyp.
Top-5 Beam	Top-5 Diverse Beam	MC-dropout	MoE	Ensemble	Pseudo
mt-ref	0.316	0.340	0.316	0.286	0.312	0.316
hyp-ref-avgmacro	0.325	0.345	0.323	0.310	0.354	0.167
hyp-ref-avgmicro	0.327	0.345	0.319	0.316	0.372	0.223
hyp-mt-avg-ref	0.275	0.380	0.438	0.371	0.413	0.408
hyp-mt-avg	0.022	0.340	0.433	0.352	0.388	0.424
Average DA	58.88	55.12	58.88	51.20	61.19	58.88
Table 1: Pearson correlation with human judgments for single reference (mt-ref) and the metrics based on MT hypotheses in §4 for the different MT systems generating diverse MT in §3 for the Estonian-English dataset, using BLEU as similarity function. The last row shows the average absolute DA scores for each model variant.
Estonian-English	English-Czech
BLEU	TER	ChrF	Meteor	BERT	BLEU	TER	ChrF	Meteor	BERT
mt-ref-1	0.417	-0.413	0.508	0.550	0.653	0.312	-0.335	0.346	0.380	0.422
mt-ref-2	0.432	-0.436	0.521	0.521	0.662	-	-	-	-	-
mt-ref-multi	0.494	-0.497	0.554	0.547	0.688	-	-	-	-	-
pseudo-mt-max	0.526	-0.350	0.589	0.550	0.672	0.271	-0.219	0.285	0.279	0.350
pseudo-mt-max-ref	0.539	-0.453	0.600	0.607	0.699	0.352	-0.346	0.398	0.388	0.473
hyp-ref-avgmacro	0.448	-0.445	0.532	0.580	0.671	0.319	-0.334	0.348	0.382	0.423
hyp-ref-avgmicro	0.455	-0.454	0.532	0.583	0.658	0.320	-0.328	0.346	0.380	0.419
hyp-mt-avg-ref	0.553	-0.543	0.601	0.638	0.700	0.378	-0.382	0.389	0.431	0.464
hyp-mt-avg	0.562	-0.548	0.597	0.610	0.665	0.296	-0.281	0.288	0.334	0.326
hyp-self-avg	0.557	-0.567	0.589	0.614	0.660	0.313	-0.306	0.309	0.351	0.355
Table 2: Pearson correlation with human judgments for single reference (mt-ref-1/2), multiple references (mt-ref-multi) and the metrics based on MT hypotheses in §4 for the Estonian-English 1K dataset and English-Czech dataset. Results significantly improving on single-reference evaluation are underlined, and those significantly improving on multi-reference evaluation are marked in bold. We use Hotelling-Williams test (Williams, 1959) for significant differences in correlation.
type	text	info
high-quality	Source	Siis aga võib tekkida seesmise ja ∀iisse vaate vahele lõhe.	DA = 75
Reference	This could however lead to a split between the inner and outer view.	ME = 0.262
MT Output	Then there may be a split between internal and external viewpoints.
Dropout	Then, however, there may be a split between internal and external viewpoints.	HY = 0.532
Then, however, there may be a gap between internal and external viewpoints.
Then there may be a split between internal and external viewpoints.
Then there may be a split between internal and external viewpoints.
Beam	Then there may be a split between internal and external viewpoints.	HY = 0.182
Then there may be a gap between internal and external viewpoints.
Then there may be a split between internal and external viewpoints.
Then there may be a gap between internal and external viewpoints.
low-quality	Source	Kant on see, kellele kuulub see teene, et ta tāiustas mateeria kāsitust seeläbi, et ta vaatles seda tõukumise ja tõmbumise ühtusena.	DA = 3
Reference	It is Kant who has the merit of refining the concept of matter by seeing it as a unity of pushing and pulling.	ME = 0.304
MT Output	It is the person who owes it to the merit of pardoning it by looking at it as a unity of push and withdrawal.
Dropout	It is the person who owes it to the merit of pardoning this approach by looking at it as a unity of push and withdrawal.	HY = 0.182
It is Mrs Kant who owes to the fact that he has perfected his approach by looking at it as a unity of impetus and resignation.
It is the one who owes the service that he has perfected his approach by seeing it as a catalyst and a stand-alone.
It is the person who owes it to the merit of perfecting this approach by looking at it as a means of push and withdrawal.
Beam	It is the person who owes it to the merit of pardoning it by seeing it as a unity of push and withdrawal.	HY = 0.182
It is the person who owes it to the merit of pardoning the concept by looking at it as a unity of push and withdrawal.
It is the person who owes it to the merit of pardoning it by looking at it as a unity of impetus and withdrawal.
It is the person who owes it to the merit of pardoning it by seeing it as a unity of impetus and withdrawal.
Table 3: Example of MC dropout and beam search hypotheses for a high-quality and a low-quality MT output. The last column shows the DA score for these two translations, as well as the Meteor score (ME) and our hyp-mt-avg-ref (HY) score obtained for them.
test (Williams, 1959), as described in Graham et al. (2015).
First, we observe that the methods based on the similarity against the reference (hyp-ref-*) do not perform as well as those relying more on the relation between MT hypotheses (hyp-mt-'). As discussed in §4, the latter capture the uncertainty of NMT models when generating the output for a given source sentence. Overall, hyp-mt-avg-ref consistently outperforms all the other variants by a large margin, for all automatic evaluation metrics considered. Logically, the improvement is larger for exact-matching metrics, but also significant for Meteor, ChrF and BERTScore, which attempt to capture linguistic variation.
Surprisingly, hyp-mt-avg-ref performs better than the mt-ref-multi. Reasons may be that it can potentially cover a larger number of paraphrases than one additional reference translation, and that besides computing similarity to a reference translation, it incorporates information on model uncertainty.
Interestingly, our reference-free metric hyp-mtavg, which only compares the MT output against additional generated hypotheses and does not rely on human references, also performs competitively. This result confirms the important role played by the model confidence component in measuring MT quality. Note that for Estonian-English dataset it performs better than the evaluation with single reference, indicating that model confidence alone can be more reliable for assessing MT quality than using a single reference translation.
Finally, we observe that using translations from online MT systems also outperforms reference-based evaluation. The differences are larger for Estonian-English. This could be because for into-English translation the quality of pseudo-references is higher, making them as good as actual reference translations, while yet closer to the MT output under evaluation. For English-Czech, pseudo-references are closer to mt-ref and generally worse than hyp-mt-avg-ref.
Table 3 illustrates the advantage of our uncertainty-aware evaluation over standard reference-based scoring. We show MC dropout and top beam hypotheses for a high quality and for a low quality MT output. First, note that MC dropout hypotheses are very different for a low-quality MT output and fairly similar for good-quality translation. By contrast, beam
hypotheses are similar or the same in both cases. Second, the evaluation scores obtained using MC dropout hypotheses result in a large difference between low-quality and high-quality MT outputs, whereas Meteor assigns a higher score to the low-quality example due to surface word and synonym matches that are in this case not indicative of MT quality.
The proposed approach has some limitations. First, it requires access to the NMT system that was used to generate the translations. Second, we note that this idea works better if the NMT model is reasonably well trained, as additional hypotheses could be less informative otherwise. Finally, it is not clear how the methods presented here would work for comparing the output quality of different MT systems, but this is a different application of our proposed approach and we leave this question to future work.
We proposed to explore NMT model uncertainty to generate additional hypotheses for MT evaluation. We showed that by exploiting similarities in the space of translation hypotheses generated by the model, along with methods to effectively combine information from these multiple hypotheses, we can achieve more accurate estimation on the quality of MT output than standard reference-based comparison, including cases with multiple references. This suggests that model uncertainty alone can be more reliable for assessing MT quality than standard reference-based evaluation.
This work can be extended in numerous ways. First, we plan to test whether similar observations will hold for more language pairs and text domains. Second, the score combination strategies could be improved by learning weights for each component. Finally, we would like to test this approach for comparing different MT systems.
Marina Fomicheva and Lucia Specia were supported by funding from the Bergamot project (EU H2020 Grant No. 825303). We thank Mark Fishel (University of Tartu) for collecting one set of Estonian-English references.
Joshua Albrecht and Rebecca Hwa. 2007. Regression for sentence-level mt evaluation with pseudo references. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 296-303.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Rachel Bawden, Nikolay Bogoychev, Ulrich German, Roman Grundkiewicz, Faheem Kirefu, Antonio Valerio Miceli Barone, and Alexandra Birch. 2019. The university of edinburgh's submissions to the wmt19 news translation task. arXiv preprint arXiv:1907.05854.
Boxing Chen and Hongyu Guo. 2015. Representation based translation evaluation metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 150-155, Beijing, China. Association for Computational Linguistics.
Julian Chow, Lucia Specia, and Pranava Madhyastha. 2019. Wmdo: Fluency-based word movera™s distance for machine translation evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 494-500, Florence, Italy.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Li Dong, Chris Quirk, and Mirella Lapata. 2018. Confidence Modeling for Neural Semantic Parsing. arXiv preprint arXiv:1805.04604.
Markus Dreyer and Daniel Marcu. 2012. Hyter: Meaning-equivalent semantics for translation evaluation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 162-171. Association for Computational Linguistics.
Hiroshi Echizen'ya, Kenji Araki, and Eduard Hovy. 2019. Word embedding-based automatic MT evaluation metric using word position information. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1874-1883, Minneapolis, Minnesota. Association for Computational Linguistics.
Marina Fomicheva, Núria Bel Rafecas, Iria Da Cunha Fanego, and Anton Malinovskiy. 2015. Upf-cobalt submission to wmt15 metrics task. In Bojar O, Chatterjee R, Federmann C, Haddow B, Hokamp C, Huck M, Logacheva V, Pecina P, editors. Proceedings of the Tenth Workshop on Statistical Machine Translation; 2015 Sep 17-18; Lisboa, Portugal: Association for Computational Linguistics; 2015. p. 373-9. ACL (Association for Computational Linguistics).
Marina Fomicheva and Lucia Specia. 2016. Reference bias in monolingual machine translation evaluation. In 54th Annual Meeting of the Association for Computational Linguistics, pages 77-82. Association for Computational Linguistics.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In international conference on machine learning, pages 1050-1059.
Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183-1191.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33-41.
Alex Graves. 2011. Practical variational inference for neural networks. In Advances in neural information processing systems, pages 2348-2356.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6097-6110, Hong Kong, China. Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. CiteSeer.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems, pages 6402-6413.
Chin-Yew Lin and Franz Josef Och. 2004. Orange: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of the 20th international conference on Computational Linguistics, page 501. Association for Computational Linguistics.
Jeremiah Liu, John Paisley, Marianthi-Anna Kioumourtzoglou, and Brent Coull. 2019. Accurate uncertainty estimation and decomposition in ensemble learning. In Advances in Neural Information Processing Systems, pages 8950-8961.
Chi-kiu Lo. 2017. Meant 2.0: Accurate semantic mt evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 589-597, Copenhagen, Denmark. Association for Computational Linguistics.
Chi-kiu Lo. 2019. Yisi-a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507-513.
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019a. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Florence, Italy. Association for Computational Linguistics.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019b. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90.
David JC MacKay. 1992. Bayesian methods for adaptive models. Ph.D. thesis, California Institute of Technology.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of