SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 7: Analysis of the ten DE-EN test set segments with the most favoritism in SNMT (S) or UNMT (U), according to  $\mathrm{MACROF_1}$  (left) and BLEU(right). Fav is the favored system by metrics. The complete text of the sentences is in the Appendix, Tables 15 and 16.
4.1 Pairwise Maximum Difference Discriminator
We consider cases where a metric has a strong opinion of one translation system over another, and analyze whether the opinion is well justified. In order to obtain this analysis, we employ a pairwise segment-level discriminator from within a corpus-level metric, which we call favoritism.
We extend the definition of  $T$  from Section 2 to  $T = \{x, h_S, h_U, y\}$  where each of  $h_S$  and  $h_U$  is a separate system's hypothesis set for  $x$ .<sup>13</sup> Let  $M$  be a corpus-level measure such that  $M(h, y) \in \mathbb{R}$  and a higher value implies better translation quality.  $M(h^{(-i)}, y^{(-i)})$  is the corpus-level score obtained by excluding  $h^{(i)}$  and  $y^{(i)}$  from  $h$  and  $y$ , respectively. We define the benefit of segment  $i$ ,  $\delta_M(i; h)$ :
$$
\delta_ {M} (i; h) = M (h, y) - M \left(h ^ {(- i)}, y ^ {(- i)}\right)
$$
If  $\delta_M(i; h) > 0$ , then  $i$  is beneficial to  $h$  with respect to  $M$ , as the inclusion of  $h^{(i)}$  increases the corpus
level score. We define the favoritism of  $M$  toward  $i$  as  $\delta_{M}(i;h_{S},h_{U})$ :
$$
\delta_ {M} (i; h _ {S}, h _ {U}) = \delta_ {M} (i; h _ {S}) - \delta_ {M} (i; h _ {U}) \tag {1}
$$
If  $\delta_M(i; h_S, h_U) > 0$  then  $M$  favors the translation of  $x^{(i)}$  by system  $S$  over that in system  $U$ .
Table 7 reflects the results of a manual examination of the ten sentences in the DE-EN test set with greatest magnitude favoritism; complete results are in the Appendix, Tables 15 and 16. Meaning-altering changes such as 'untranslation', (wrong) 'time', and (wrong) 'translation' are marked in italics, while changes that do not fundamentally alter the meaning, such as 'synonym,' (different) 'inflection,' and (different) 'word order' are marked in plain text.[14]
The results indicate that  $\mathrm{MACROF_1}$  generally favors SNMT, and with good reasons, as the favored translation does not generally alter sentence meaning, while the disfavored translation does. On
13The subscripts represent SNMT and UNMT in this case, though the definition is general.
<sup>14</sup>Some changes, such as 'word order' may change meaning; these are italicized or not on a case-by-case basis.
6th	δMACROF1(i,hS,hU): .044, δBLEU(i,hS,hU): -.00087, δBLEURT(i,hS,hU): .97
Ref	Ever since I joined Labour 32 years ago as a school pupil, provoked by the Thatcher government's neglect that had left my comprehensive school classroom literally falling down, I've sought to champion better public services for those who need them most - whether as a local councillor or government minister.
SNMT	32 years ago, I joined Labour as a student because of the neglect of the Thatcher government, which had led to my classroom literally collapsed, and as a result I tried to promote better public services for those who need it most, whether as a local council or ministers.
UNMT	Last 32 years ago, as a student, because of the disdain for the Thatcher-era government, Labour joined Labour.
Problems	SNMT: synonym, word_order UNMT: subject, truncation, word_order
Table 8: An example of favoritism that illustrates the differences between  $\mathrm{MACROF_1}$  and BLEU. Translations of the DE-EN test sentence with sixth largest magnitude favoritism according to  $\mathrm{MACROF_1}$ , along with the favoritism according to BLEU (not in the top ten). UNMT's translation does not include the second half of the sentence.  $\mathrm{MACROF_1}$  favors SNMT, but BLEU favors UNMT.
the other hand, for the ten most favored sentences according to BLEU, four do not contain meaning-altering divergences in the disfavored translation. Importantly, none of the sentences with greatest favoritism according to  $\mathrm{MACROF_1}$ , all of which having meaning altering changes in the disfavored alternatives, appears in the list for BLEU. This indicates relatively bad judgment on the part of BLEU. One case of good judgment from  $\mathrm{MACROF_1}$  and bad judgment from BLEU regarding truncation is shown in Table 8.
From our qualitative examinations,  $\mathrm{MACROF}_1$  is better than BLEU at discriminating against untranslations and trucations in UNMT. The case is similar for FR-EN and RO-EN, except that RO-EN has more untranslations for both SNMT and UNMT, possibly due to the smaller training data. Complete tables and annotated sentences are in the Appendix, in Section C.
5.1 MT Metrics
Many metrics have been proposed for MT evaluation, which we broadly categorize into model-free or model-based. Model-free metrics compute scores based on translations but have no significant parameters or hyperparameters that must be tuned a priori; these include BLEU (Papineni et al., 2002), NIST (Doddington, 2002), TER (Snover et al., 2006), and  $\mathrm{CHRF}_1$  (Popovic, 2015). Model-based metrics have a significant number of parameters and, sometimes, external resources that must be set prior to use. These include METEOR (Banerjee and Lavie, 2005), BLEURT (Sellam et al., 2020), YiSi (Lo, 2019), ESIM (Mathur et al., 2019), and BEER (Stanojevic and Sima'an, 2014). Model-based metrics require significant effort and resources when adapting to a new language or domain, while model-free metrics require only a test
set with references.
Mathur et al. (2020) have recently evaluated the utility of popular metrics and recommend the use of either  $\mathrm{CHRF}_1$  or a model-based metric instead of BLEU. We compare our  $\mathrm{MACROF}_1$  and  $\mathrm{MICROF}_1$  metrics with BLEU,  $\mathrm{CHRF}_1$ , and BLEURT (Sellam et al., 2020). While Mathur et al. (2020) use Pearson's correlation coefficient  $(r)$  to quantify the correlation between automatic evaluation metrics and human judgements, we instead use Kendall's rank coefficient  $(\tau)$ , since  $\tau$  is more robust to outliers than  $r$  (Croux and Dehon, 2010).
5.2 Rare Words are Important
That natural language word types roughly follow a Zipfian distribution is a well known phenomenon (Zipf, 1949; Powers, 1998). The frequent types are mainly so-called "stop words," function words, and other low-information types, while most content words are infrequent types. To counter this natural frequency-based imbalance, statistics such as inverted document frequency (IDF) are commonly used to weigh the input words in applications such as information retrieval (Jones, 1972). In NLG tasks such as MT, where words are the output of a classifier, there has been scant effort to address the imbalance. Doddington (2002) is the only work we know of in which the 'information' of an n-gram is used as its weight, such that rare n-grams attain relatively more importance than in BLEU. We abandon this direction for two reasons: Firstly, as noted in that work, large amounts of data are required to estimate n-gram statistics. Secondly, unequal weighing is a bias that is best suited to datasets where the weights are derived from, and such biases often do not generalize to other datasets. Therefore, unlike Doddington (2002), we assign equal weights to all n-gram classes, and in this work we limit our scope to unigrams only.
While BLEU is a precision-oriented measure,
METEOR (Banerjee and Lavie, 2005) and CHRF (Popovic, 2015) include both precision and recall, similar to our methods. However, neither of these measures try to address the natural imbalance of class distribution. BEER (Stanojevic and Sima'an, 2014) and METEOR (Denkowski and Lavie, 2011) make an explicit distinction between function and content words; such a distinction inherently captures frequency differences since function words are often frequent and content words are often infrequent types. However, doing so requires the construction of potentially expensive linguistic resources. This work does not make any explicit distinction and uses naturally occurring type counts to effect a similar result.
5.3 F-measure as an Evaluation Metric
F-measure (Rijsbergen, 1979; Chinchor, 1992) is extensively used as an evaluation metric in classification tasks such as part-of-speech tagging, named entity recognition, and sentiment analysis (Derczynski, 2016). Viewing MT as a multi-class classifier is a relatively new paradigm (Gowda and May, 2020), and evaluating MT solely as a multiclass classifier as proposed in this work is not an established practice. However, we find that the  $F_{1}$  measure is sometimes used for various analyses when BLEU and others are inadequate: The compare-mt tool (Neubig et al., 2019) supports comparison of MT models based on  $F_{1}$  measure of individual types. Gowda and May (2020) use  $F_{1}$  of individual types to uncover frequency-based bias in MT models. Sennrich et al. (2016) use corpus-level unigram  $F_{1}$  in addition to BLEU and CHRF, however, corpus-level  $F_{1}$  is computed as MICROF<sub>1</sub>. To the best of our knowledge, there is no previous work that clearly formulates the differences between micro- and macro-averages, and justifies the use of MACROF<sub>1</sub> for MT evaluation.
We have evaluated NLG in general and MT specifically as a multi-class classifier, and illustrated the differences between micro- and macro- averages using  $\mathrm{MICROF_1}$  and  $\mathrm{MACROF_1}$  as examples (Section 2).  $\mathrm{MACROF_1}$  captures semantic adequacy better than  $\mathrm{MICROF_1}$  (Section 3.1). BLEU, being a micro-averaged measure, served well in an era when generating fluent text was at least as difficult as generating adequate text. Since we are now in an era in which fluency is taken for granted and seman
tic adequacy is a key discriminating factor, macro-averaged measures such as  $\mathrm{MACROF}_1$  are better at judging the generation quality of MT models (Section 3.2). We have found that another popular metric,  $\mathrm{CHRF}_1$ , also performs well on direct assessment, however, being an implicitly micro-averaged measure, it does not perform as well as  $\mathrm{MACROF}_1$  on downstream CLIR tasks (Section 3.3.1). Unlike BLEURT, which is also adequacy-oriented,  $\mathrm{MACROF}_1$  is directly interpretable, does not require retuning on expensive human evaluations when changing language or domain, and does not appear to have uncontrollable biases resulting from data effects. It is both easy to understand and to calculate, and is inspectable, enabling fine-grained analysis at the level of individual word types. These attributes make it a useful metric for understanding and addressing the flaws of current models. For instance, we have used  $\mathrm{MACROF}_1$  to compare supervised and unsupervised NMT models at the same operating point measured in BLEU, and determined that supervised models have better adequacy than the current unsupervised models (Section 4).
Macro-average is a useful technique for addressing the importance of the long tail of language, and  $\mathrm{MACROF_1}$  is our first step in that direction; we anticipate the development of more advanced macroaveraged metrics that take advantage of higher-order and character n-grams in the future.
Since many ML models including NMT are themselves opaque and known to possess data-induced biases (Prates et al., 2019), using opaque and biased evaluation metrics in concurrence makes it even harder to discover and address the flaws in modeling. Hence, we have raised concerns about the opaque nature of the current model-based evaluation metrics, and demonstrated examples displaying unwelcome biases in evaluation. We advocate the use of the  $\mathrm{MACROF_1}$  metric, as it is easily interpretable and offers the explanation of score as a composition of individual type performances. In addition,  $\mathrm{MACROF_1}$  treats all types equally, and has no parameters that are directly or indirectly estimated from data sets. Unlike  $\mathrm{MACROF_1}$ ,  $\mathrm{MICROF_1}$  and other implicitly or explicitly micro-averaged metrics assign lower importance to rare concepts and their associated rare types. The use of microaveraged metrics in real world evaluation could lead to marginalization of rare types.
Failure Modes: The proposed  $\mathrm{MACROF_1}$  metric is not the best measure of fluency of text. Hence we suggest caution while using  $\mathrm{MACROF_1}$  to draw fluency related decisions.  $\mathrm{MACROF_1}$  is inherently concerned with words, and assumes the output language is easily segmentable into word tokens. Using  $\mathrm{MACROF_1}$  to evaluate translation into alphabetical languages such as Thai, Lao, and Khmer, that do not use white space to segment words, requires an effective tokenizer. Absent this the method may be ineffective; we have not tested it on languages beyond those listed in Section B.
Reproducibility: Our implementation of  $\mathrm{MACROF_1}$  and  $\mathrm{MICROF_1}$  has the same user experience as BLEU as implemented in SACRE-BLEU; signatures are provided in Section A. In addition, our implementation is computationally efficient, and has the same (minimal) software and hardware requirements as BLEU. All data for MT and NLG human correlation studies is publicly available and documented. Data for reproducing the IR experiments in Section 3.3.2 is also publicly available and documented. The data for reproducing the IR experiments in Section 3.3.1 is only available to participants in the CLSSTS shared task.
Climate Impact: Our proposed metrics are on par with BLEU and such model-free methods, which consume significantly less energy than most model-based evaluation metrics.
The authors thank Shantanu Agarwal, Joel Barry, and Scott Miller for their help with CLSSTS CLIR experiments, and Daniel Cohen for discussions on IR evaluation methods. This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via AFRL Contract FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pages 489-513, Copenhagen, Denmark. Association for Computational Linguistics.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pages 131-198, Berlin, Germany. Association for Computational Linguistics.
Elizabeth Boschee, Joel Barry, Jayadev Billa, Marjorie Freedman, Thamme Gowda, Constantine Lignos, Chester Palen-Michel, Michael Pust, Banriskhem Kayang Khonglah, Srikanth Madikeri, Jonathan May, and Scott Miller. 2019. SARAL: A low-resource cross-lingual domain-focused information retrieval system for effective rapid document triage. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 19–24, Florence, Italy. Association for Computational Linguistics.
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136-158, Prague, Czech Republic. Association for Computational Linguistics.
Nancy Chinchor. 1992. MUC-4 Evaluation Metrics. In Proceedings of the 4th Conference on Message Understanding, MUC4 '92, page 22-29, USA. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7059-7069. Curran Associates, Inc.
Christophe Croux and Catherine Dehon. 2010. Influence functions of the spearman and kendall correlation measures. Statistical methods & applications, 19(4):497-515.
Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85-91, Edinburgh, Scotland. Association for Computational Linguistics.
Leon Derczynski. 2016. Complementarity, F-score, and NLP evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 261-266, PortoRoZ, Slovenia. European Language Resources Association (ELRA).
George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram cooccurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT '02, page 138-145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for NLG micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 179-188. Association for Computational Linguistics.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243-1252.
Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3955-3964, Online. Association for Computational Linguistics.
Gregory Grefenstette. 2012. *Cross-language information retrieval*, volume 2. Springer Science & Business Media.
Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11-21.
Karen Sparck Jones, Steve Walker, and Stephen E. Robertson. 2000. A probabilistic model of information retrieval: development and comparative experiments: Part 2. Information processing & management, 36(6):809-840.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. Proc. 10th Machine Translation Summit (MT Summit), 2005, pages 79-86.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049, Brussels, Belgium. Association for Computational Linguistics.
Constantine Lignos, Daniel Cohen, Yen-Chieh Lien, Pratik Mehta, W. Bruce Croft, and Scott Miller. 2019. The challenges of optimizing machine translation for low resource cross-language information retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3497-3502, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation.
Chi-kiu Lo. 2019. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507-513, Florence, Italy. Association for Computational Linguistics.
Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671-688, Belgium, Brussels. Association for Computational Linguistics.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume