SlowGuess/NAACL_2022_OCR huggingface dataset:
Jungo Kasai\* Keisuke Sakaguchi\* Ronan Le Bras\* Lavinia Dunagan\* Jacob Morrison\* Alexander R. Fabbri\* Yejin Choi\* Noah A. Smith\* Paul G. Allen School of Computer Science & Engineering, University of Washington \*Allen Institute for AI Department of Linguistics, University of Washington Salesforce Research billboard.nlp@gmail.com
Natural language processing researchers have identified limitations of evaluation methodology for generation tasks, with new questions raised about the validity of automatic metrics and of crowdworker judgments. Meanwhile, efforts to improve generation models tend to depend on simple n-gram overlap metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics should each more directly benefit and inform the other. We therefore propose a generalization of leaderboards, bidimensional leaderboards (BILLBOARDs), that simultaneously tracks progress in language generation models and metrics for their evaluation. Unlike conventional unidimensional leaderboards that sort submitted systems by predetermined metrics, a BILLBOARD accepts both generators and evaluation metrics as competing entries. A BILLBOARD automatically creates an ensemble metric that selects and linearly combines a few metrics based on a global analysis across generators. Further, metrics are ranked based on their correlation with human judgments. We release four BILLBOARDs for machine translation, summarization, and image captioning.<sup>1</sup> We demonstrate that a linear ensemble of a few diverse metrics sometimes substantially outperforms existing metrics in isolation. Our mixed-effects model analysis shows that most automatic metrics, especially the reference-based ones, overrate machine over human generation, demonstrating the importance of updating metrics as generation models become stronger (and perhaps more similar to humans) in the future.
Recent modeling advances have led to improved natural language generation in applications such as machine translation and summarization (Ng et al.,
Figure 1: Bidimensional leaderboard (BILLBOARD). When a generator developer submits output text (output.txt), BILLBOARD computes all metric scores. When a metric developer submits an executable program (e.g., metric.py), BILLBOARD computes correlation with the human judgments, updates the ensemble metric ( $\S 2.2$ ), and measures how much the metric overrates machines ( $\S 2.3$ ).
2019; Raffel et al., 2020; Brown et al., 2020, inter alia). This progress is typically measured with automatic scores, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), executed by modeling researchers themselves. These metrics allow for fast, inexpensive development cycles. They were adopted based on reported correlations with human judgments at the time the metrics were introduced, but it has since been established that the correspondence can collapse when models of different types are compared (Callison-Burch et al., 2006) or models become increasingly powerful (Ma et al., 2019; Edunov et al., 2020).
Meanwhile, many evaluation metrics that improve correlation with human judgments have been proposed (Clark et al., 2019; Zhang et al., 2020b; Sellam et al., 2020; Hessel et al., 2021, inter alia), but this progress has yet to be broadly adopted by the community of researchers focused on advancing models. Indeed, consistent with prior metaevaluations (Marie et al., 2021), we found that  $68\%$
*Work was done during an internship at AI2.  
'https://nlp.cs.washington.edu/billboard/.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3540 - 3557 July 10-15, 2022 ©2022 Association for Computational Linguistics
of the machine translation papers from NAACL and ACL 2021 evaluated their models solely by BLEU, and only  $5\%$  measured the performance using recent metrics with contextual representations such as COMET (Rei et al., 2020). Similarly, automatic evaluation in  $66\%$  of the summarization papers was done only in terms of ROUGE. We believe this separation between generation modeling and automatic evaluation represents a missed opportunity for each subcommunity to more rapidly benefit from the advances of the other.
We therefore propose an abstraction of conventional leaderboards, bidimensional leaderboards (BILLBOARDs), that simultaneously facilitates progress in natural language generation and its evaluation (Fig. 1). A BILLBOARD accepts two types of submissions related to a given task and dataset: generators and metrics. Unlike conventional leaderboards, model ranking is not tied to a predetermined set of metrics; the generators are ranked based on the metric that currently correlates best with human judgments. Metric submissions are ranked by their correlations to human judgments, and each is stored as an executable program, which will then be used to evaluate future generation submissions. Our BILLBOARD includes a sparse regression that selects and linearly combines three existing metrics, revealing complementary strengths. All leaderboard scores are readily reproducible, allowing research on generation models and automatic metrics to benefit from each other.
We release four BILLBOARD interfaces (https://nlp.cs.washington.edu/billboard/) spanning three generation tasks: the WMT20 EN-DE and WMT20 ZH-EN machine translation tasks (Barrault et al., 2020), the CNNDM summarization task (Hermann et al., 2015), and the MSCOCO image captioning task (Lin et al., 2014).
Key Findings Using the collective analyses of BILLBOARDs, our main findings are as follows.
- A simple linear combination of a few (diverse) metrics can sometimes improve correlation. This finding quantifies complementary effects of different metrics and encourages metric developers to seek out aspects of generated text quality not yet measured by existing metrics.
- Using linear mixed-effects models, we find that most automatic metrics, especially conventional, reference-based ones such as BLEU and ROUGE, overrate machines over humans in all tasks. This result provides further support for the claim that the metrics should be continually evaluated and updated as our generation models become stronger (and perhaps, closer to humans).
- When only one reference is available per instance, COMET-QE (a strong referenceless metric with crosslingual contextual representations; Rei et al., 2020) achieves higher correlation with human judgments than all reference-based metrics. This raises a concern about the current standard evaluation practice in machine translation and summarization that uses reference-based metrics with a single reference per instance.
- Our findings confirm many others who report that recent metrics achieve substantially higher correlation with human judgments than popular metrics like BLEU and ROUGE in BILLBOARDs. We believe these older metrics continue to be used mainly because modeling researchers value consistency and accessibility of evaluation practice over long periods of time. BILLBOARDs provide a way to maintain long-term comparability of system output while also drawing better conclusions about system quality, using advances in evaluation. All generators continue to be evaluated with new metrics on BILLBOARDs.
2 Bidimensional Leaderboards
We propose BILLLBOARDS to simultaneously drive progress in natural language generation and its evaluation, which are often disconnected in current research. We first describe the general framework (§2.1) and the automatic analyses they provide (§2.2-2.3). We then discuss our design choices (§2.4) and the rubric-based, human judgment data necessary to initialize BILLLBOARDS (§2.5).
2.1 BILLBOARD Framework
The leaderboard paradigm has driven research on state-of-the-art model performance on many tasks in various fields (e.g., ImageNet, Russakovsky et al., 2015; SQuAD, Rajpurkar et al., 2016). As applications and tasks become more diverse, however, the conventional leaderboard paradigm presents a serious challenge: the assumption becomes too strong that predetermined, automatic metrics can
2We examined all papers whose title contains "machine translation" and "summarization." See Appendix A for details.
reliably score the system performance over time. In particular, scores from automatic metrics often diverge from human judgments in language generation tasks, especially when models become increasingly powerful (Ma et al., 2019).
Much recent work proposed new evaluation metrics that improve correlations with human judgments in certain generation tasks (Clark et al., 2019; Zhang et al., 2020b; Sellam et al., 2020; Hesselet al., 2021, inter alia), but most developers of generation models are not benefiting from them (See Appendix A for our analysis of papers from NAACL/ACL 2021). From the perspective of generation model developers, it is not clear which of these many metrics in the literature is most reliable in which generation task or dataset, resulting in community-wide overuse of long-standing metrics like BLEU and ROUGE. Developers of evaluation metrics, on the other hand, are missing the opportunity to apply their metrics to new generation models and compare them with the existing ones. We propose BILLBOARDs that bridge this gap between generation modeling and evaluation development.
Generators, Metrics, and Scores A BILLBOARD for a language generation task consists of sets of generators and evaluation metrics:  $\mathcal{G} = \{G_i\}_{i=1}^I$ ,  $\mathcal{M} = \{M_j\}_{j=1}^J$ . Each generator  $G_i$  takes as input  $X_k$  (e.g., source text in machine translation) and generates text:  $Y_{i,k} = G_i(X_k)$ . A metric  $M_j$  assigns a score to each generated text given the generation input and the corresponding set of references  $\mathcal{R}_k$ :  $s_{i,j,k} = M_j(Y_{i,k}, \mathcal{R}_k, X_k)$ . The last two arguments to the function are optional; some metrics do not require references (i.e., referenceless or quality estimation metrics) or the generation input (e.g., BLEU). We then compute the aggregate score  $s_{i,j}$  by averaging  $s_{i,j,k}$  over  $K$  test examples.
Rankings In contrast to standard leaderboards, BILLBOARDS have a dynamic set of evaluation metrics, and generators are not ranked by a predefined metric. We first rank the metrics by measuring their correlations to human judgments as commonly done in the generation evaluation literature (Zhang et al., 2020b; Sellam et al., 2020). Let  $h_{i,k}$  be a human score for  $Y_{i,k}$  (i.e., output from generator  $G_i$  on input  $X_k$ ). We compute the instance-level Pearson correlation for every metric  $M_j$  between  $h_{i,k}$  and  $s_{i,j,k}$  ( $M_j$  score for  $Y_{i,k}$ ). All metrics are ranked by their correlations. We then use the top metric  $M_{j^*}$  to rank the generators in
the descending order of  $s_{i,j^*}$ . We defer our discussions on alternative design choices (§2.4) and human evaluations (§2.5). We note, however, that the overall framework of BILLBOARDs still holds regardless of these decisions.
2.2 Ensemble of Metrics
So far, we have assumed that metrics are used individually in isolation, but BILLBOARDs provide a unique opportunity to examine metrics collectively. Different metrics can capture different aspects of generation quality; even if a metric is not sufficiently informative in isolation, it might reflect an important aspect of text quality that the existing metrics overlook. Here we consider a straightforward and interpretable ensemble of metrics using a regression model with  $\ell_1$  regularization (Tibshirani, 1994). Let the ensemble's score be
$$
\hat {h} _ {i, k} = \sum_ {j = 1} ^ {J} w _ {j} \cdot s _ {i, j, k},
$$
where  $w_{j}$  is a scalar coefficient associated with the  $j$ th metric and the intercept term is suppressed. We optimize the vector of coefficients  $\mathbf{w}$  with the pairs of output text and a human score  $\{Y_{i,k}, h_{i,k}\}_{k=1}^{K}$  from the test data:
$$
\mathbf {w} ^ {*} = \arg \min  _ {\mathbf {w}} \sum_ {k = 1} ^ {K} \left(h _ {i, k} - \hat {h} _ {i, k}\right) ^ {2} + \lambda \| \mathbf {w} \| _ {1}
$$
The  $\ell_1$  regularization produces sparse coefficients and improves interpretability by removing highly correlated metrics. Moreover, it avoids the need for practitioners to run many metrics to obtain an ensemble score when used outside our BILLBOARDs. Our goal for the ensemble is to provide a useful signal to the research community, rather than to achieve the best possible correlation with human judges at a given time; we tune  $\lambda$  to get three nonzero coefficients. Every metric is standardized by its mean and standard deviation on the test data.
Similar to the individual metrics, we rank this ensemble metric by its correlation to the human judgments. To make fair comparisons, we simulate situations where the ensemble is applied to a newly submitted generator that has no human evaluations. Specifically, we perform cross validation that holds out the human judgments for each generator  $G_{i}$  and runs regression on the rest; we then apply these  $I$  regression models to the corresponding held-out data and calculate the overall correlation. We will see that the ensemble metric outperforms all individual metrics in some cases, suggesting that different
metrics can capture different aspects.
Reproducibility The ensemble metric is updated every time a new metric is submitted (Fig. 1). For reproducibility, we keep track of every past ensemble metric with a signature that indicates its coefficients,  $\lambda$ , and input metrics in the backend. Similar to SACREBLEU (Post, 2018), model developers can report the signature for easy replication of their scores from the ensemble metric. Further, all generation outputs are saved on the leaderboards, so model developers can download outputs from all past models and compare in any way.
2.3 Mixed-Effects Model Analysis
Recent work (Kasai et al., 2022) observed that automatic metrics tend to overrate machine-generated text over human one on the MSCOCO image captioning task (Chen et al., 2015). This problem is particularly severe in conventional metrics that are based on n-gram overlap such as BLEU and CIDEr (Vedantam et al., 2015). This raises a significant concern about the continuous use of these conventional metrics in generation tasks as models become increasingly powerful (and more similar to humans); those metrics unintentionally discourage researchers from developing human-like, strong generation models. To quantify this undesirable property, we propose a linear mixed-effects model that compares the two groups of machine-and human-generated text. The underlying model assumes that  $s_{i,j,k}$ , the score from metric  $M_j$  for generator  $G_i$  and test example  $k$ , can be expressed as (the intercept term is suppressed for brevity):
$$
s _ {i, j, k} = \beta_ {0} ^ {j} \mathbb {1} \left\{G _ {i} \text {i s m a c h i n e} \right\} + \beta_ {1} ^ {j} h _ {i, k} + \gamma_ {k} + \epsilon_ {i, j, k}
$$
where  $\gamma_{k}$  is the random effect for example  $k$ , and  $\epsilon_{i,j,k}$  is Gaussian noise. Intuitively,  $\beta_0^j$  measures how much metric  $M_{j}$  overrates machine generation over human one, compared against the human judgment  $h_{i,k}$ .  $\beta_0^j = 0$  means being neutral, and indeed we will find that  $\beta_0^j$  is significantly positive in most cases (§4). We standardize all metric scores over the test samples to compare the size of  $\beta_0^j$ . We apply the lme4 package (Bates et al., 2015).
2.4 Design Choices and Discussion
In our current setup, we make several design choices for metrics and their rankings:
- M.1 Metrics are expected to positively correlate with the generation output quality.
- M.2 By default, metrics are ranked based on their instance-level Pearson correlations with human judgments. We also compute and present their system-level Kendall rank correlations.
- M.3 When available, reference-based metrics use multiple references per instance.
M.1 implies that we need to take the negative of metric scores that are intended to negatively correlate (e.g., TER, Snover et al., 2006). This normalization is also done in WMT metric competitions (Callison-Burch et al., 2007, 2008, inter alia).
While instance-level correlations are commonly used to evaluate and compare automatic metrics for various language generation tasks (Sellam et al., 2020; Fabbri et al., 2021; Hessel et al., 2021, inter alia), there are several alternatives to M.2. For example, Pearson, Spearman's rank, or Kendall rank correlations can be used on a system (i.e., generator) level (Callison-Burch et al., 2007; Macháček and Bojar, 2014; Mathur et al., 2020b). However, such system-level correlations would substantially reduce data points to compare automatic scores, resulting in many ties in the ranking. Spearman's and Kendall rank correlations become brittle when multiple generators are similar in overall output quality; penalizing a metric for swapping two similar generators is misleading (Macháček and Bojar, 2014). Moreover, if a metric can perform well on an instance level, it can be used to augment human judgments by, for example, flagging likely wrong ratings (Mathur et al., 2020b). Thus, we encourage researchers to develop metrics that correlate well with human judgments on an instance level. Prior work also points out other problems in ranking metrics like outlier effects where outlier systems have a disproportionately large effect on the overall correlation (Mathur et al., 2020a,b). We therefore assume M.2 in the current version of BILLBOARDs, but this can be modified in a future version.
M.3 is supported by our experimental results in §4 that multiple references substantially improve reference-based metrics, and a single reference is often insufficient to outperform strong referenceless metrics. Some metrics have specifications for multiple references (e.g., BLEU, CIDEr). In the other cases, we evaluate outputs against every reference and take the maximum score, following prior work on image captioning evaluation (Zhang et al.,
$^{3}$ E.g., ensemble.wmt20-zh-en+refs.AB+version.1.
Japanese Prime Minister Shinzo Abe posted a video to celebrate the 70th anniversary of the founding of the People's Republic of China.
Generated Translation
Japanese Prime Minister Shinzo Abe congratulated the 70th anniversary of the founding of the People's Republic of China via video.
It may have been her impressive vocals... It's now been revealed that X Factor-finalist and pop star Ella Henderson has joined forces with dry shampoo brand Batiste. The 19-year-old has been announced as the official face of the Batiste's 2015 "Ready For It" campaign, and this will mark the star's first brand collaboration... The partnership between the platinum award-winning artist and the UK's number 1 dry shampoo brand...
Generated Summary
Ella, 19, announced as face of Batiste's 2015 "Ready for It" campaign. X Factor Finalist's first brand collaboration with the UK's No 1 dry shampoo.
Figure 2: Comparisons and meta-evaluations of crowdworker and rubric-based, expert evaluations for WMT20 ZH-EN and CNNDM summarization. Every dot represents one test instance that is evaluated by the same numbers of experts and crowdworkers (one for WMT20 ZH-EN and three for CNNDM) for fair comparisons. We randomly sampled instances with diverging evaluations in two areas  $\square$  and conducted binary meta-evaluations (good  $\bullet$  or bad quality  $\bullet$ ). Meta-evaluations agree more with the expert evaluations:  $\bullet > \bullet$  in the upper left squares and  $\bullet > \bullet$  in the lower right squares. We suspect that the highlighted text might have caused the disagreement.
Xining will implement the Xining Civilized Behavior Promotion Regulations from October 1st, which focus on 15 types of uncivilized behavior, such as pedestrians who do not follow the traffic lights and throw objects from buildings.
Xining City will implement the "Xining City Civilized Behavior Promotion Regulation" from October 1, focusing on 15 types of uncivilized behaviors such as pedestrians not passing traffic lights and throwing objects from buildings.
Arsenal manager Arsene Wenger does not know the exact reason Alexis Sanchez chose the Emirates Stadium over Anfield - but he is glad the Chile forward will be lining up for his side rather than against them for Liverpool on Saturday...
Alexis Sanchez was courted by a number of elite clubs last summer. Sanchez has scored 19 goals so far this season. Arsenal boss Arsene Wenger does not know if Sanchez decided to join the North London club.
2020b; Hessel et al., 2021).<sup>4</sup>
2.5 Human Evaluation
Human evaluations are required to initialize BILL-BOARDs; they are used to rank metrics, train the metric ensembling model, and assess how much each metric overrates machines. Recent work, however, points out problems when evaluations are done by crowdworkers even when extensive quality controls are performed (Gillick and Liu, 2010; Toral et al., 2018; Freitag et al., 2021; Clark et al., 2021). Freitag et al. (2021) show that rubric-based machine translation evaluations by professional translators led to substantially different generator rankings from the crowdsourced evaluations in WMT 2020 (Barrault et al., 2020), where WMT participants or Amazon Mechanical Turkers directly assess each translation's adequacy by a single score (direct assessment). These crowdworker evaluations depend highly on individual annotators' discretion and understanding of the annotation scheme (Freitag et al., 2021; Clark et al., 2021), making it difficult to decompose, interpret, and validate
(Kasai et al., 2022). Moreover, these direct assessment scores make it difficult to interpret evaluation results for downstream applications where some aspects are particularly important (e.g., accessibility for people with visual impairments in image captioning, Gleason et al., 2020; gender bias in machine translation, Stanovsky et al., 2019).
Motivated by this line of work, we perform metaevaluations to compare crowdsourced and rubric-based expert evaluations. Fig. 2 plots overall scores for test examples from WMT20 ZH-EN (Barrault et al., 2020; Freitag et al., 2021) and CNNDM summarization (Fabbri et al., 2021). Each instance is evaluated by averaging the same number of crowdworkers and expert scores for fair comparisons. We see that substantially many instances fall into disagreement: crowdworkers give much higher scores than experts (lower right square) or the reverse (upper left square). We sample and shuffle 20/25 examples from either type and ask a meta-evaluator to make a binary decision (good  $\oplus$  or bad quality  $\oplus$ ). Meta-evaluations agree more with the expert evaluations (e.g., 22 and 0  $\oplus$  in the upper left
4 Intuitively, the maximum score measures the distance to the closest out of equally valid generations.
5The meta-evaluations were done by a bilingual speaker (WMT20 ZH-EN) and the first author of this paper (CNNDM).
and lower right squares for CNNDM, respectively). In the examples on the left, crowdworkers fail to properly assess a valid translation with different structure than the reference (posted a video to celebrate vs. congratulated via video) or a summary that combines information from different parts of the article. The examples on the right illustrate that crowdworkers can be fooled by inaccurate yet fluent generations (does not know the reason vs. does not know if Sanchez decided). Given this result, we decide to initialize our BILLBOARDs with rubric-based expert evaluations for all generation tasks. We still encourage future work to explore ways to improve crowdsourced evaluations for scalability.
Having established the framework, we set up BILLBOARDs for three natural language generation tasks: machine translation, summarization, and image captioning. To maximize the performance of reference-based metrics, we use as many references as possible for each task. See §4 for an analysis on the effect of varying numbers of references.