SlowGuess/NAACL_2022_OCR huggingface dataset:
Daniel Deutsch, Rotem Dror, and Dan Roth
{ddeutsch,rtmdrr,danroth}@seas.upenn.edu
How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are commonly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.
Automatic evaluation metrics are the most common method that researchers use to quickly and cheaply approximate how humans would rate the quality of a summarization system (Lin, 2004; Louis and Nenkova, 2013; Zhao et al., 2019; Zhang et al., 2020; Deutsch et al., 2021a, among others). The quality of a metric — how similarly it replicates human judgments of systems — is quantified by calculating the correlation between the metric's scores and human judgments on a set of systems, known as the system-level correlation (Louis and Nenkova, 2013; Deutsch et al., 2021b).
Accurately estimating system-level correlations is critically important. Summarization researchers
use automatic metrics during system development to make decisions about which ideas work and which do not, and systems from different research groups are ranked by automatic metrics to define which system is the "state-of-the-art." If we do not have precise estimates of metric quality, it is not clear how much trust the community should put in such evaluation methodologies.
At present, there are disconnects between how automatic metrics are evaluated and how they are used to evaluate systems. First, the metrics' scores which are used in practice are not the ones which are evaluated in system-level correlations; researchers compare systems based on metric scores calculated on the entire test set but calculate scores for system-level correlations when evaluating metrics on a much smaller subset of judged summaries. Second, metrics are evaluated in a setting that is much easier than how they are actually used. Metric correlations are calculated using systems that vary greatly in quality, whereas researchers compare new systems to recent work, which are likely to be very close in quality. Discriminating between two systems of similar quality is much harder than doing so between low and high quality systems.
In this work, we re-examine how system-level correlations are calculated and propose two independent changes to make the evaluation of metrics better aligned to how they are actually used to evaluate systems.
First, we propose to modify the system-level correlation definition to use the entire test set to calculate the system scores for automatic metrics instead of only the subset of summaries judged by humans (§3). With this change, the scores which are used to compare systems are directly evaluated, and we further demonstrate how the precision of our estimate of system-level correlations improves as a result. Calculating system scores over a larger number of instances reduces the variance of the scores, which results in confidence intervals (CIs)
1Our code is available at http://cogcomp.org/page/publication_view/973.
Human Language Technologies, pages 6038 - 6052
for the correlations that are  $16 - 51\%$  more narrow on average (§3.2).
Second, we redefine a high quality metric to be one for which a small difference in score reliably indicates a difference in quality (§4). Then, instead of calculating the correlation with all available system pairs, we only evaluate with pairs of systems whose automatic metric scores differ by some threshold. This allows us to show, for example, that a ROUGE-1 score difference of less than 0.5 between systems has almost no correlation to how humans would rank the same two systems according to our best estimates (§4.2). For two other metrics, BERTScore (Zhang et al., 2020) and QAEval (Deutsch et al., 2021a), we show their correlations calculated on system pairs of similar quality are much worse than under the standard correlation definition. These results cast doubt on how reliable automatic evaluation metrics are for measuring summarization system quality in realistic scenarios.
Our analyses point to the need to collect more high-quality human judgments of summaries in order to have more accurate estimates of metric correlations as well as the need to improve the ability of automatic metrics to discriminate between similarly performing systems.
Automatic evaluation metrics are most commonly used to argue that one summarization system is better than another, typically by showing that the value of a metric improves with the "better" system. How similarly automatic metrics replicate human judgments of system quality is quantified by system-level correlations as follows.
The summaries from  $N$  systems on  $M_{\mathrm{jud}}$  input documents are judged by humans  $\mathcal{Z}$  and scored with an automatic metric  $\mathcal{X}$ . Then, the system-level correlation between  $\mathcal{X}$  and  $\mathcal{Z}$  is calculated as
$$
r _ {\mathrm {S Y S}} = \operatorname {C O R R} \left(\left\{\left(\frac {1}{M _ {\mathrm {j u d}}} \sum_ {j} ^ {M _ {\mathrm {j u d}}} x _ {i} ^ {j}, \frac {1}{M _ {\mathrm {j u d}}} \sum_ {j} ^ {M _ {\mathrm {j u d}}} z _ {i} ^ {j}\right) \right\} _ {i = 1} ^ {N}\right)
$$
where  $x_{i}^{j}$  and  $z_{i}^{j}$  are the scores of  $\mathcal{X}$  and  $\mathcal{Z}$  for the summary produced by the  $i$ -th system on the  $j$ -th input document and CORR is some correlation function. See Fig. 1 for an illustration of this calculation.
Figure 1: The system-level correlation is calculated between the average  $\mathcal{X}$  and  $\mathcal{Z}$  scores on a set of summarization systems.  $x_{i}^{j}$  and  $z_{i}^{j}$  are the scores for the summary produced by system  $i$  (represented by rows) on input document  $j$  (represented by columns).
In this work, we use Kendall's  $\tau$  (the "b" variant) as the correlation function because we are most concerned with a metric's ability to correctly determine whether one system is better than another since that is how metrics are used in practice. Kendall's  $\tau$  is computed based on the number of system pairs out of  $\binom{N}{2}$  which are ranked the same by  $\mathcal{X}$  and  $\mathcal{Z}$ . It is defined as
$$
\tau = \frac {P - Q}{\sqrt {(P + Q + T) \cdot (P + Q + U)}} \tag {1}
$$
where  $P$  and  $Q$  are the number of pairs ranked the same or different by  $\mathcal{X}$  and  $\mathcal{Z}$ , respectively, and  $T$  and  $U$  are the number of ties only in  $\mathcal{X}$  or  $\mathcal{Z}$ , respectively.
Because the computation of  $r_{\mathrm{SYS}}$  involves randomness — its value depends on which  $M_{\mathrm{jud}}$  input documents (and even which  $N$  systems) were used — it is only an approximation of the true correlation between  $\mathcal{X}$  and  $\mathcal{Z}$ . As such, Deutsch et al. (2021b) proposed various methods for calculating confidence intervals for  $r_{\mathrm{SYS}}$ . For instance, their BOOT-INPUTS method uses bootstrapping to repeatedly resample the  $M_{\mathrm{jud}}$  input documents used to calculate  $r_{\mathrm{SYS}}$ , thereby calculating a confidence interval for the true  $r_{\mathrm{SYS}}$  value for  $\mathcal{X}$  and  $\mathcal{Z}$ .
Datasets The datasets that are used in this paper's analyses are SummEval (Fabbri et al., 2021) and REALSumm (Bhandari et al., 2020), two recently collected datasets with human annotations for summary quality collected from the
$^{2}$ https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html
CNN/DailyMail dataset (Nallapati et al., 2016). SummEval has  $M_{\mathrm{jud}} = 100$  summaries annotated with a summary relevance score for  $N = 16$  systems. REALSumm has  $M_{\mathrm{jud}} = 100$  summaries annotated with a Lightweight Pyramid score (Shapira et al., 2019) for  $N = 25$  systems. We correlate the scores of the automatic metrics to these annotations. The CNN/DailyMail test split has 11,490 instances.
Automatic Metrics Our experiments will analyze three different reference-based automatic evaluation metrics which were chosen because they were demonstrated to have the best correlations with human judgments on the SummEval and REALSumm datasets (Deutsch et al., 2021b). ROUGE- $n$  (Lin, 2004) evaluates a generated summary by calculating an  $\mathrm{F}_1$  score on the number of  $n$ -grams it has in common with a human-written reference summary. BERTScore (Zhang et al., 2020) aligns the generated and reference summaries' tokens based on their BERT embeddings (Devlin et al., 2019) and calculates a score based on the similarity of the aligned tokens' embeddings. QA-Eval (Deutsch et al., 2021a) compares the two summaries by automatically generating questions from the reference and calculating what proportion of those questions are answered correctly by the generated summary.
3 Evaluating with All Available Instances
Although the above definition of the system-level correlation has been used by recent meta-evaluation studies of metrics (Bhandari et al., 2020; Fabbri et al., 2021; Deutsch et al., 2021b), there is a disconnect between how the automatic metrics are evaluated and how they are used in practice.
Researchers who develop summarization systems evaluate those systems with automatic metrics on all  $M_{\mathrm{test}}$  test instances, not just the subset of  $M_{\mathrm{jud}}$  instances which were judged by humans. Evaluating a system on a larger number of summaries may end up changing the system's score, which could potentially alter the overall ranking of a set of systems. Therefore, the rankings that are used by practitioners to determine system quality are not the ones which are being evaluated in the standard definition of system-level correlation. $^3$
Figure 2: The bootstrapped  $95\%$  confidence intervals for the BERTScore of each system in the REALSumm dataset using  $M_{\mathrm{jud}}$  judged instances in blue and  $M_{\mathrm{test}}$  instances in orange. Evaluating systems with  $M_{\mathrm{test}}$  instances leads to far better estimate of their true scores.
To that end, we propose to modify the correlation definition to use all  $M_{\mathrm{test}}$  instances to calculate the system scores for the automatic metrics. That is (differences in bold):
$$
r _ {\mathrm {S Y S}} = \operatorname {C O R R} \left(\left\{\left(\frac {1}{\mathbf {M} _ {\text {t e s t}}} \sum_ {j} ^ {\mathbf {M} _ {\text {t e s t}}} x _ {i} ^ {j}, \frac {1}{M _ {\text {j u d}}} \sum_ {j} ^ {M _ {\text {j u d}}} z _ {i} ^ {j}\right) \right\} _ {i = 1} ^ {N}\right)
$$
In practice with modern, large-scale datasets, this minor change could mean estimating system quality based on  $\approx 10\mathrm{k}$  inputs instead of around 100. This new definition now properly evaluates the way metrics are actually used by researchers.
We expect that scoring systems with  $M_{\mathrm{test}}$  inputs instead of  $M_{\mathrm{jud}}$  should lead to a better estimate of the true automatic metric score, which would in turn result in a lower-variance estimate of the correlation between  $\mathcal{X}$  and  $\mathcal{Z}$  in the form of smaller confidence intervals for  $r_{\mathrm{SYS}}$ . In the next sections, we carry out analyses to demonstrate that this is true.
3.1 Reducing Automatic Metric Variance
First, we empirically show that scoring systems with  $M_{\mathrm{test}}$  instances instead of  $M_{\mathrm{jud}}$  does indeed reduce the variance of the estimate of the automatic metric scores and subsequently increases the stabilities of the system rankings.
Ideally, the  $\mathcal{X}$  score for a system would be its "oracle"  $\mathcal{X}$  score, equal to the expected value of  $\mathcal{X}$  for a document sampled from the latent distribution over documents defined by the dataset (e.g., a system's ROUGE score on an infinite number of examples from a dataset). Since this cannot be calculated, it is approximated by averaging the  $\mathcal{X}$  score on a sample (i.e., either the  $M_{\mathrm{jud}}$  or  $M_{\mathrm{test}}$  input documents). Because  $M_{\mathrm{test}} \gg M_{\mathrm{jud}}$ , we expect that the variance of this estimate using  $M_{\mathrm{test}}$  inputs should be lower than when using  $M_{\mathrm{jud}}$ .
<sup>3</sup>We suspect this methodology is an artifact of how system-level correlations were first calculated for summarization in the DUC shared tasks when the dataset sizes were small enough that  $M_{\mathrm{jud}} = M_{\mathrm{test}}$  (e.g., Dang and Owczarzak, 2008).
Figure 3: Bootstrapped estimates of the stabilities of the system rankings for automatic metrics and human annotations on SummEval (left) and REALSumm (right). The  $\tau$  value quantifies how similar two system rankings would be if they were computed with two random sets of  $M$  input documents. When all  $M_{\mathrm{test}}$  test instances are used, the automatic metrics' rankings become near constant. The error regions represent  $\pm 1$  standard deviation.
To quantify this, we calculated the variance of estimating the oracle  $\mathcal{X}$  score using both  $M_{\mathrm{jud}}$  and  $M_{\mathrm{test}}$  input documents via bootstrapping. We randomly sampled  $M$  input documents with replacement, recomputed the system scores, and calculated the variance of those scores over 1k iterations. For all three metrics on both datasets, we found around a  $99\%$  reduction in the variance when  $M_{\mathrm{test}}$  inputs were used instead of  $M_{\mathrm{jud}}$ , clearly demonstrating that evaluating systems with  $M_{\mathrm{test}}$  inputs results in a better estimate of the system scores. In Fig. 2, this is visualized for BERTScore on the REALSumm dataset.
However, because we are interested in evaluating the metrics' rankings, we also quantify how much of an effect this reduction in variance has on the stability of the system rankings induced by  $\mathcal{X}$ . Similarly to the system scores, there is an oracle ranking of systems for  $\mathcal{X}$ , equal to the ordering of systems by their respective oracle  $\mathcal{X}$  scores (e.g., systems sorted by their ROUGE scores calculated on an infinite number of examples from a dataset). As the variance of the system score estimates decreases, the computed ranking of systems should begin to converge to the oracle  $\mathcal{X}$  ranking. We aim to understand to what extent this happens if  $M_{\mathrm{test}}$  instances are used for evaluation instead of  $M_{\mathrm{jud}}$ .
To quantify this notion, we calculate the Kendall's  $\tau$  between two system rankings for  $\mathcal{X}$  that were based on two sets of  $M$  input documents, each sampled with replacement from the set of available documents. This simulates how much the system rankings would change if the evaluation procedure was run twice, each time with  $M$  random input documents. This quantity is calculated
1k times for various values of  $M$  and plotted in Fig. 3.
As  $M$  approaches  $M_{\mathrm{test}}$ , the automatic metrics'  $\tau$  values approach 1, which is much higher than the respective values at  $M_{\mathrm{jud}}$ , typically around 0.6-0.8. A value near 1 means that the rankings calculated using  $M_{\mathrm{test}}$  inputs are almost constant, implying the rankings have converged to the oracle ranking. Therefore, the reduction in variance from evaluating on  $M_{\mathrm{test}}$  instances does indeed greatly stabilize the system rankings.
Fig. 3 also contains the same analysis performed for the human judgments  $\mathcal{Z}$  in both datasets, although it is limited to a maximum of  $M_{\mathrm{jud}}$  input documents. We see that on both datasets the judgments' rankings are still quite variable, reaching a maximum  $\tau$  of around 0.8-0.85.
3.2 Confidence Interval Analysis
Next, we show that the improved estimate of system scores leads to a more precise estimate of  $r_{\mathrm{SYS}}$  by demonstrating the widths of the confidence intervals for  $r_{\mathrm{SYS}}$  decrease.
The confidence intervals for  $r_{\mathrm{SYS}}$  calculated using bootstrapping methods proposed by Deutsch et al. (2021b) are rather wide. For instance, the 95% CI for ROUGE-2 on SummEval is [-.09, .84], demonstrating a rather high level of uncertainty in its value. This is problematic because it means we do not have a good picture of how reliable automatic evaluation metrics are. Reducing the width of the CIs will help us better understand the true metric quality.
We suspect that the large width of the confidence interval is due to the variance of the system rank
Figure 4:  $95\%$  confidence intervals for  $r_{\mathrm{SYS}}$  calculated with the BOOT-INPUTS resampling method when the system rankings for the automatic metrics are calculated using only the judged data (orange) versus the entire test set (blue). Scoring systems with more summaries leads to better (more narrow) estimates of  $r_{\mathrm{SYS}}$ .
ings of the automatic metrics and human judgments. The more unstable the rankings are with respect to the  $M$  inputs, the larger the variance of the estimate of  $r_{\mathrm{SYS}}$  should be since very different system rankings would be compared on each bootstrapping iteration. Deutsch et al. (2021b) used  $M_{\mathrm{jud}}$  input documents to calculate their CIs. Therefore, we expect the improved stability of the automatic metric system rankings from evaluating on  $M_{\mathrm{test}}$  instances should result in a more narrow confidence interval for  $r_{\mathrm{SYS}}$  since some noise has been removed from this computation.
To demonstrate this, we calculated  $95\%$  CIs for  $r_{\mathrm{SYS}}$  using the BOOT-INPUT method on SummEval and REALSumm using both  $M_{\mathrm{jud}}$  and  $M_{\mathrm{test}}$  input documents, shown in Fig. 4. We find that the widths of the CIs shrank on average by  $51\%$  on SummEval and  $16\%$  on REALSumm. The largest decrease in width is in the ROUGE family of metrics on SummEval, potentially because that metric and dataset combination saw the biggest improvement in ranking stability (see Fig. 3). Thus, the improved estimate of the system scores did result in more precise estimates of  $r_{\mathrm{SYS}}$ . We repeated this analysis using the other bootstrapping methods proposed by Deutsch et al. (2021b), and the results are discussed in Appendix A.
3.3 Conclusions & Recommendations
By estimating system quality using automatic metrics on all available instances instead of only those which were judged, we showed that the variances of the system scores and subsequent rankings reduce significantly, resulting in better estimates of  $r_{\mathrm{SYS}}$ . Because this methodology additionally directly evaluates the system scores used by researchers, we recommend future work do the same.
In order to continue to improve the estimate of  $r_{\mathrm{SYS}}$ , as much variance as possible needs to be removed from the system rankings. Evaluating systems using  $M_{\mathrm{test}}$  instances removed a large amount of variance from the automatic metric rankings, but as demonstrated in Fig. 3, the human judgments still have a large amount of variance.
The human rankings' variances can either be reduced by judging more summaries per system or making the judgments more consistent. Since the human rankings' stabilities in Fig. 3 are mostly beginning to plateau — especially for SummEval — it may be prohibitively expensive to collect a sufficient number of judgments to better stabilize the rankings (Wei and Jia, 2021). Therefore, we expect the more feasible solution is to improve the consistency of the human judgments, for example by better training the annotators or improving the annotation protocol.
4 Evaluating with Realistic System Pairs
Next, we argue that the set of systems used to evaluate metrics is not reflective of how metrics are used in practice and propose a new system-level correlation variant to address this problem.
4.1 Evaluating with All System Pairs
The  $N$  systems which are used for calculating system-level correlations are typically those which participated in a shared task, as in DUC/TAC (Dang and Owczarzak, 2008, among others), or those which have been published in the previous 3-4 years (Bhandari et al., 2020; Fabbri et al., 2021). As such, they are typically rather diverse in terms of their qualities, both as rated by human annotators and automatic metrics.
The system scores of all of the systems in the REALSumm dataset as evaluated by humans and automatic metrics are shown in Fig. 5. Clearly, the scores are rather diverse. For example, the systems cluster into low, medium, and high quality groups (with an additional outlier) as evaluated by ROUGE.
Figure 5: The systems (each represented by a point) on the two datasets (shown here for REALSumm) are rather diverse in quality as measured by both human judgments and automatic metrics.
A difference of around 5 ROUGE points between them is a rather large gap for ROUGE scores.
The standard definition for a high quality evaluation metric is one which correctly ranks a set of systems with respect to human judgments. As such, the implementation of the system-level correlation calculated with Kendall's  $\tau$  will rank all  $N$  systems according to the human judgments and an automatic metric, then count how many pairs were ranked the same out of all  $\binom{N}{2}$  pairs (see §2). As a consequence, even pairs of systems which are separated by a large margin according to the automatic metric — likely systems with a clear difference in quality — are included in the evaluation. Therefore, automatic metrics are rewarded for correctly ranking such "easy" system pairs.
4.2 Evaluating with Realistic Pairs
This standard evaluation setting does not reflect how summarization metrics are actually used by researchers. New systems are typically only slightly better than previous work. Based on a survey of recent summarization papers in *ACL conferences (see Appendix C), we found that the average improvement over baseline/state-of-the-art models that was reported on the CNN/Dailymail dataset was on average 0.5 ROUGE-1. It is rarely the case that the improvement in automatic metrics is very large. Therefore, evaluating metrics using pairs of systems which are separated by a large margin does not reflect the reality that metrics are very frequently used to compare those separated by a small margin. Including "easy" system pairs in the system-level correlation likely overestimates the
quality of the metrics in settings which occur in practice.
To that end, we redefine a high quality evaluation metric to be one for which a small difference in scores reliably indicates a difference in quality. We quantify this by proposing a variant of the system-level  $\tau$  which is calculated between system pairs which are separated by a pre-defined automatic metric score margin. Instead of using all  $\binom{N}{2}$  system pairs, only pairs whose difference in scores falls within the margin are used to calculate the system-level correlation. We denote this correlation variant as  $r_{\mathrm{SYS}}\Delta (\ell ,u)$  where  $\ell$  and  $u$  are the lower- and upper-bounds of the allowable differences in automatic metrics' scores. This would enable, for example, evaluating how well ROUGE correlates to human judgments on system pairs that are separated by 0.0-0.5 ROUGE points, thereby directly evaluating the scenario in which ROUGE is used to make decisions about system quality.