SlowGuess/NAACL_2022_OCR huggingface dataset:
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumatica: technologies de la traduccio, 0:455-463.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984-4997, Online. Association for Computational Linguistics.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. 2020b. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688-725, Online. Association for Computational Linguistics.
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173-2185, Online. Association for Computational Linguistics.
Mathias Müller and Rico Sennrich. 2021. Understanding the properties of minimum Bayes risk decoding in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 259-272, Online. Association for Computational Linguistics.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212-223, Brussels, Belgium. Association for Computational Linguistics.
Graham Neubig. 2013. Travatar: A forest-to-string machine translation engine based on tree transducers. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 91-96, Sofia, Bulgaria. Association for Computational Linguistics.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine
Translation (Volume 2: Shared Task Papers, Day 1), pages 314-319, Florence, Italy. Association for Computational Linguistics.
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160-167, Sapporo, Japan. Association for Computational Linguistics.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3956-3965. PMLR.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Maja Popovic. 2015. *chrF: character n-gram F-score* for automatic MT evaluation. In *Proceedings of the Tenth Workshop on Statistical Machine Translation*, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for MT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 751-762, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest: Translation quality estimation with cross-lingual transformers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5070-5081, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, Andre F. T. Martins, and Alon Lavie.
2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b. Unbabel's participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 911-920, Online. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Uri Shaham and Omer Levy. 2021. What do you get when you cross beam search with nucleus sampling?
Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 177-184, Boston, Massachusetts, USA. Association for Computational Linguistics.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
Raphael Shu and Hideki Nakayama. 2017. Later-stage minimum bayes-risk decoding for neural machine translation.
David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787-794, Sydney, Australia. Association for Computational Linguistics.
Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020. Findings of the WMT 2020 shared task on quality estimation. In Proceedings of the Fifth Conference on Machine Translation, pages 743-764, Online. Association for Computational Linguistics.
Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. 2021. Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 684-725, Online. Association for Computational Linguistics.
Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3356-3362, Hong Kong, China. Association for Computational Linguistics.
Felix Stahlberg, Adrià de Gispert, Eva Hasler, and Bill Byrne. 2017. Neural machine translation by minimising the Bayes-risk with respect to syntactic translation lattices. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 362-368, Valencia, Spain. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826.
Chaojun Wang and Rico Senrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544-3552, Online. Association for Computational Linguistics.
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU:training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344-4355, Florence, Italy. Association for Computational Linguistics.
Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054-3059, Brussels, Belgium. Association for Computational Linguistics.
Kyra Yee, Yann Dauphin, and Michael Auli. 2019. Simple and effective noisy channel modeling for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5696-5701, Hong Kong, China. Association for Computational Linguistics.
Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomás Kocisky. 2017. The neural noisy channel. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Chrysoula Zerva, Daan van Stigt, Ricardo Rei, Ana C Farinha, Pedro Ramos, José G. C. de Souza, Taisiya Glushkova, Miguel Vera, Fabio Kepler, and André F. T. Martins. 2021. IST-unbabel 2021 submission for the quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 961-972, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
For the experiments using IWSLT17, we train a small transformer model (6 layers, 4 attention heads, 512 embedding dimensions, and 1024 hidden dimensions) from scratch, using Fairseq (Ott et al., 2019). We tokenize the data using SentencePiece (Kudo and Richardson, 2018), with a joint vocabulary with 20000 units. We train using the Adam optimizer (Kingma and Ba, 2015) with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.98$  and use an inverse square root learning rate scheduler, with an initial learning rate of  $5 \times 10^{-4}$  and with a linear warm-up in the first 4000 steps. For models trained with label smoothing, we use the default value of 0.1.
For completeness, we include in Table 4 results to evaluate the impact of the metrics presented in §3 as features and objectives for ranking using the other language pairs: EN  $\rightarrow$  RU (large model) and EN  $\rightarrow$  FR (small model).
Large (WMT20)	Small (IWSLT)
BLEU	chrF	BLEURT	COMET	BLEU	chrF	BLEURT	COMET
Baseline	23.86	51.16	0.6953	0.5361	38.12	63.20	0.6532	0.4809
F-RR w/ COMET-QE	20.32	49.18	0.7130	0.6207	35.59	60.90	0.6628	0.5553
F-RR w/ MBART-QE	22.39	50.59	0.6993	0.5481	36.68	62.17	0.6593	0.5091
F-RR w/ OpenKiwi	20.88	48.72	0.7040	0.5688	32.03	55.68	0.5996	0.2581
F-RR w/ Transquest	21.60	50.14	0.7060	0.5836	36.02	62.26	0.6681	0.5397
T-RR w/ BLEU	23.87	51.51	0.7042	0.5669	39.10	64.22	0.6968	0.6189
T-RR w /BLEURT	22.84	51.25	0.7265	0.6470	38.60	63.76	0.7042	0.6405
F-RR w/ COMET	22.42	50.91	0.7243	0.6441	38.60	63.77	0.7020	0.6392
MBR w/ BLEU	24.03	51.12	0.6938	0.5393	37.97	63.13	0.6484	0.4764
MBR w/ BLEURT	23.01	50.87	0.7314	0.5984	37.29	62.82	0.6886	0.5361
MBR w/ COMET	23.67	51.18	0.7093	0.6242	37.77	63.24	0.6710	0.6127
T-RR+MBR w/ BLEU	24.11	51.44	0.6967	0.5482	38.96	64.04	0.6781	0.5636
T-RR+MBR w/ BLEURT	23.18	51.30	0.7344	0.6277	37.43	63.14	0.7092	0.5961
T-RR+MBR w/ COMET	23.21	51.26	0.7238	0.6736	38.33	63.53	0.6883	0.6610
Table 4: Evaluation metrics for EN  $\rightarrow$  RU for the large model setting and EN  $\rightarrow$  FR for small model settings, using a fixed  $N$ -best reranker (F-RR), a tuned  $N$ -best reranker (T-RR), MBR decoding, and a two-stage approach. Best overall values are bolded and best for each specific group are underlined.
C Human Study
In order to perform human evaluation, we recruited professional translators who were native speakers of the target language on the freelancing site Upwork. 300 sentences were evaluated for each language pair, sampled randomly from the test sets after a restriction that sentences were no longer than 30 words. All translation hypotheses for a single source sentence were first deduplicated, and then shown to the translator side-by-side in randomized order to avoid any ordering biases.
Sentences were evaluated according to a 1-5 rubric slightly adapted from that of Wieting et al. (2019):
1. There is no overlap in the meaning of the source sentence whatsoever.
2. Some content is similar but the most important information in the sentence is different.
3. The key information in the sentence is the same but the details differ.
4. Meaning is essentially equal but some expressions are unnatural.
5. Meaning is essentially equal and the sentence is natural.
<sup>5</sup>https://upwork.com. Freelancers were paid a market rate of 18-20 US dollars per hour, and finished approximately 50 sentences in one hour.
D MQM Framework
Human evaluations were performed by Unbabel's PRO Community, made of professional translators and linguists with relevant experience in linguistic annotations and translation errors annotations. In order to properly assess translations quality, annotators must be native speakers of the target language and with a proven high proficiency of the source language, so that they can properly capture errors and their nuances. The systems' outputs were evaluated by using the annotation framework adopted internally at Unbabel, which is an adaptation of the MQM Framework (Lommel et al., 2014).
We asked the annotators to identify all errors and independently label them with an error category and a severity level. We consider three categories (each of them containing a set of different subcategories) that may affect the quality of the translations:
- Accuracy, if the target text does not accurately reflect the source text (e.g., changes in the meaning, addition/omission of information, untranslated text, MT hallucinations);
- Fluency, if there are issues that affect the reading and the comprehension of the text (e.g., grammar and spelling errors);
- Style, if the text has stylistic problems (e.g., grammatical and lexical register).
Additionally, each error is labeled according to three severity levels (minor, major, and critical), depending on the way they affect the accuracy, the fluency, and the style of the translation. The final sentence-level score is obtained using a weighting scheme where minor, major, and critical errors are weighted as 1, 5, and 10, respectively.
Figures 4 and 5 show the counts of errors breakdown by typology and severity level for EN  $\rightarrow$  DE and EN  $\rightarrow$  RU, respectively.
Figure 4: Error typology and severity level breakdown for WMT20 (large models) EN→DE.
Figure 5: Error typology and severity level breakdown for WMT20 (large models) EN→RU.