SlowGuess/ACL_2020_OCR huggingface dataset:
the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Tim Pearce, Mohamed Zaki, Alexandra Brintrup, and Andy Neel. 2018. Uncertainty in neural networks: Bayesian ensembling. arXiv preprint arXiv:1810.05546.
Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 372-375, Lisboa, Portugal.
Roberts Rozis and Raivis Skadinš. 2017. Tilde model-multilingual open data for eu languages. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 263-265. Tilde Model RAPID Corpus and EU data therein licensed under CC-BY 4.0.
Christophe Servan, Alexandre Berard, Zied Elloumi, Hervé Blanchon, and Laurent Besacier. 2016. Word2Vec vs DBnary: Augmenting METEOR using Vector Representations or Lexical Resources? Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1159-1168.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc'Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. arXiv preprint arXiv:1902.07816.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Dustin Tran, Mike Dusenberry, Mark van der Wilk, and Danijar Hafner. 2019. Bayesian layers: A module for neural network uncertainty. In Advances in Neural Information Processing Systems, pages 14633-14645.
Andre Tattar and Mark Fishel. 2017. bleu2vec: the painfully familiar metric on continuous vector space steroids. Proceedings of the Second Conference on Machine Translation, page 619-622.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017b. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424.
Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019. Improving Back-Translation with Uncertainty-based Confidence Estimation. arXiv preprint arXiv:1909.00157.
Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. Character: Translation edit rate on character level. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 505-510.
Max Welling and Yee W Teh. 2011. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681-688.
Evan James Williams. 1959. Regression Analysis, volume 14. Wiley, New York, USA.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
A Distribution of DA Scores
Figure 3 shows the distribution of DA scores for Estonian-English 1K and English-Czech datasets.
B Number of MC Dropout Hypotheses
Figure 4 Pearson correlation with human judgments for hyp-mt-avg-ref with sentBLEU metric as a function of the number of stochastic forward passes with MC dropout. The improvements in correlation become small after 10 hypotheses for both Estonian-English and English-Czech.
C Combination Methods
Tables 4 and 5 show a full set of Pearson correlation results for the scoring approaches described in Section 3.2.
Figure 3: Distribution of DA scores for Estonian-English 1K and English-Czech datasets
Figure 4: Pearson correlation with human judgments for hyp-mt-avg-ref with sent-BLEU metric as a function of the number of stochastic forward passes with MC dropout
BLEU	TER	ChrF	Meteor	BERT
mt-ref1	0.417	-0.413	0.508	0.550	0.653
mt-ref2	0.432	-0.436	0.521	0.521	0.662
mt-ref-multi	0.494	-0.497	0.554	0.547	0.688
pseudo-mt-avg	0.494	-0.485	0.541	0.545	0.547
pseudo-mt-min	0.363	-0.531	0.396	0.400	0.289
pseudo-mt-max	0.526	-0.350	0.589	0.550	0.672
pseudo-mt-avg-ref	0.508	-0.506	0.570	0.604	0.657
pseudo-mt-min-ref	0.445	-0.531	0.505	0.547	0.493
pseudo-mt-max-ref	0.539	-0.453	0.600	0.607	0.699
hyp-ref-avgmacro	0.448	-0.445	0.532	0.580	0.671
hyp-ref-minmacro	0.438	-0.443	0.514	0.586	0.667
hyp-ref-maxmacro	0.446	-0.440	0.536	0.548	0.648
hyp-ref-avgmicro	0.455	-0.454	0.532	0.583	0.658
hyp-ref-minmicro	0.388	-0.429	0.460	0.555	0.622
hyp-ref-maxmicro	0.427	-0.415	0.525	0.497	0.591
hyp-mt-avg-ref	0.553	-0.543	0.601	0.638	0.700
hyp-mt-min-ref	0.479	-0.536	0.550	0.615	0.676
hyp-mt-max-ref	0.577	-0.505	0.617	0.614	0.686
hyp-mt-avg	0.562	-0.548	0.597	0.610	0.665
hyp-mt-min	0.421	-0.546	0.479	0.550	0.593
hyp-mt-max	0.549	-0.423	0.571	0.528	0.595
hyp-self-avg	0.557	-0.567	0.589	0.614	0.660
Table 4: Pearson correlation with human judgments for single reference (mt-ref), multiple references (mt-ref-multi) and the metrics based on MT hypotheses described in Section 3.2 for Estonian-English dataset
BLEU	TER	ChrF	Meteor	BERT
mt-ref	0.312	-0.335	0.346	0.380	0.422
pseudo-mt-avg	0.283	-0.293	0.304	0.304	0.375
pseudo-mt-min	0.214	-0.272	0.221	0.258	0.280
pseudo-mt-max	0.271	-0.219	0.285	0.279	0.350
pseudo-mt-avg-ref	0.356	-0.385	0.394	0.404	0.476
pseudo-mt-min-ref	0.325	-0.382	0.347	0.381	0.425
pseudo-mt-max-ref	0.352	-0.346	0.398	0.388	0.473
hyp-ref-avgmacro	0.319	-0.334	0.348	0.382	0.423
hyp-ref-minmacro	0.322	-0.334	0.344	0.391	0.429
hyp-ref-maxmacro	0.326	-0.339	0.346	0.379	0.412
hyp-ref-avgmicro	0.320	-0.328	0.346	0.380	0.419
hyp-ref-minmicro	0.310	-0.320	0.324	0.382	0.421
hyp-ref-maxmicro	0.321	-0.328	0.333	0.362	0.391
hyp-mt-avg-ref	0.378	-0.382	0.389	0.431	0.464
hyp-mt-min-ref	0.329	-0.357	0.339	0.371	0.448
hyp-mt-max-ref	0.359	-0.373	0.375	0.422	0.438
hyp-mt-avg	0.296	-0.281	0.288	0.334	0.326
hyp-mt-min	0.225	-0.218	0.199	0.246	0.295
hyp-mt-max	0.240	-0.248	0.228	0.260	0.204
hyp-self-avg	0.313	-0.306	0.309	0.351	0.355
Table 5: Pearson correlation with human judgments for single reference (mt-ref) and the metrics based on MT hypotheses described in Section 3.2 for English-Czech dataset