Venue: NAACL 2022

# Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

Daniel Deutsch, Rotem Dror, and Dan Roth

Department of Computer and Information Science

University of Pennsylvania

{ddeutsch,rtdmrr,danroth}@seas.upenn.edu

###### Abstract

How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are commonly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.1

Footnote 1: Our code is available at [http://cogcomp.org/page/publication_view/973](http://cogcomp.org/page/publication_view/973).

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6038 - 6052

## 1 Introduction

Automatic evaluation metrics are the most common method that researchers use to quickly and cheaply approximate how humans would rate the quality of a summarization system Lin (2004); Louis and Nenkova (2013); Zhao et al. (2019); Zhang et al. (2020); Deutsch et al. (2021a), among others). The quality of a metric -- how similarly it replicates human judgments of systems -- is quantified by calculating the correlation between the metric's scores and human judgments on a set of systems, known as the system-level correlation Louis and Nenkova (2013); Deutsch et al. (2021b).

Accurately estimating system-level correlations is critically important. Summarization researchers use automatic metrics during system development to make decisions about which ideas work and which do not, and systems from different research groups are ranked by automatic metrics to define which system is the "state-of-the-art." If we do not have precise estimates of metric quality, it is not clear how much trust the community should put in such evaluation methodologies.

At present, there are disconnects between how automatic metrics are evaluated and how they are used to evaluate systems. First, the metrics' scores which are used in practice are not the ones which are evaluated in system-level correlations; researchers compare systems based on metric scores calculated on the entire test set but calculate scores for system-level correlations when evaluating metrics on a much smaller subset of judged summaries. Second, metrics are evaluated in a setting that is much easier than how they are actually used. Metric correlations are calculated using systems that vary greatly in quality, whereas researchers compare new systems to recent work, which are likely to be very close in quality. Discriminating between two systems of similar quality is much harder than doing so between low and high quality systems.

In this work, we re-examine how system-level correlations are calculated and propose two independent changes to make the evaluation of metrics better aligned to how they are actually used to evaluate systems.

First, we propose to modify the system-level correlation definition to use the entire test set to calculate the system scores for automatic metrics instead of only the subset of summaries judged by humans (SS3). With this change, the scores which are used to compare systems are directly evaluated, and we further demonstrate how the precision of our estimate of system-level correlations improves as a result. Calculating system scores over a larger number of instances reduces the variance of the scores, which results in confidence intervals (CIs)

for the correlations that are 16-51% more narrow on average (SS3.2).

Second, we redefine a high quality metric to be one for which a small difference in score reliably indicates a difference in quality (SS4). Then, instead of calculating the correlation with all available system pairs, we only evaluate with pairs of systems whose automatic metric scores differ by some threshold. This allows us to show, for example, that a ROUGE-1 score difference of less than 0.5 between systems has almost no correlation to how humans would rank the same two systems according to our best estimates (SS4.2). For two other metrics, BERTScore Zhang et al. (2020) and QAEval Deutsch et al. (2021a), we show their correlations calculated on system pairs of similar quality are much worse than under the standard correlation definition. These results cast doubt on how reliable automatic evaluation metrics are for measuring summarization system quality in realistic scenarios.

Our analyses point to the need to collect more high-quality human judgments of summaries in order to have more accurate estimates of metric correlations as well as the need to improve the ability of automatic metrics to discriminate between similarly performing systems.

## 2 Background

Automatic evaluation metrics are most commonly used to argue that one summarization system is better than another, typically by showing that the value of a metric improves with the "better" system. How similarly automatic metrics replicate human judgments of system quality is quantified by system-level correlations as follows.

The summaries from \(N\) systems on \(M_{\text{jud}}\) input documents are judged by humans \(\mathcal{Z}\) and scored with an automatic metric \(\mathcal{X}\). Then, the system-level correlation between \(\mathcal{X}\) and \(\mathcal{Z}\) is calculated as

where \(x^{j}_{i}\) and \(z^{j}_{i}\) are the scores of \(\mathcal{X}\) and \(\mathcal{Z}\) for the summary produced by the \(i\)-th system on the \(j\)-th input document and Corr is some correlation function. See Fig. 1 for an illustration of this calculation.

In this work, we use Kendall's \(\tau\) (the "b" variant2) as the correlation function because we are most concerned with a metric's ability to correctly determine whether one system is better than another since that is how metrics are used in practice. Kendall's \(\tau\) is computed based on the number of system pairs out of \(\binom{N}{2}\) which are ranked the same by \(\mathcal{X}\) and \(\mathcal{Z}\). It is defined as

Footnote 2: [https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html)

\[\tau=\frac{P-Q}{\sqrt{(P+Q+T)\cdot(P+Q+U)}} \tag{1}\]

where \(P\) and \(Q\) are the number of pairs ranked the same or different by \(\mathcal{X}\) and \(\mathcal{Z}\), respectively, and \(T\) and \(U\) are the number of ties only in \(\mathcal{X}\) or \(\mathcal{Z}\), respectively.

Because the computation of \(r_{\textsc{sys}}\) involves randomness -- its value depends on which \(M_{\text{jud}}\) input documents (and even which \(N\) systems) were used -- it is only an approximation of the true correlation between \(\mathcal{X}\) and \(\mathcal{Z}\). As such, Deutsch et al. (2021b) proposed various methods for calculating confidence intervals for \(r_{\textsc{sys}}\). For instance, their Boot-Inputs method uses bootstrapping to repeatedly resample the \(M_{\text{jud}}\) input documents used to calculate \(r_{\textsc{SYS}}\), thereby calculating a confidence interval for the true \(r_{\textsc{SYS}}\) value for \(\mathcal{X}\) and \(\mathcal{Z}\).

DatasetsThe datasets that are used in this paper's analyses are SummEval Fabbri et al. (2021) and REALSummBhandari et al. (2020), two recently collected datasets with human annotations for summary quality collected from the

Figure 1: The system-level correlation is calculated between the average \(\mathcal{X}\) and \(\mathcal{Z}\) scores on a set of summarization systems. \(x^{j}_{i}\) and \(z^{j}_{i}\) are the scores for the summary produced by system \(i\) (represented by rows) on input document \(j\) (represented by columns).

 

CNN/DailyMail dataset (Nallapati et al., 2016). SummEval has \(M_{\text{jud}}=100\) summaries annotated with a summary relevance score for \(N=16\) systems. REALSumm has \(M_{\text{jud}}=100\) summaries annotated with a Lightweight Pyramid score (Shapira et al., 2019) for \(N=25\) systems. We correlate the scores of the automatic metrics to these annotations. The CNN/DailyMail test split has \(11,490\) instances.

Automatic MetricsOur experiments will analyze three different reference-based automatic evaluation metrics which were chosen because they were demonstrated to have the best correlations with human judgments on the SummEval and REALSumm datasets Deutsch et al. (2021). ROUGE-\(n\)Lin (2004) evaluates a generated summary by calculating an F\({}_{1}\) score on the number of \(n\)-grams it has in common with a human-written reference summary. BERTScore Zhang et al. (2020) aligns the generated and reference summaries' tokens based on their BERT embeddings Devlin et al. (2019) and calculates a score based on the similarity of the aligned tokens' embeddings. QAEval Deutsch et al. (2021) compares the two summaries by automatically generating questions from the reference and calculating what proportion of those questions are answered correctly by the generated summary.

## 3 Evaluating with All Available Instances

Although the above definition of the system-level correlation has been used by recent meta-evaluation studies of metrics (Bhandari et al., 2020; Fabbri et al., 2021; Deutsch et al., 2021), there is a disconnect between how the automatic metrics are evaluated and how they are used in practice.

Researchers who develop summarization systems evaluate those systems with automatic metrics on all \(M_{\text{test}}\) test instances, not just the subset of \(M_{\text{jud}}\) instances which were judged by humans. Evaluating a system on a larger number of summaries may end up changing the system's score, which could potentially alter the overall ranking of a set of systems. Therefore, the rankings that are used by practitioners to determine system quality are not the ones which are being evaluated in the standard definition of system-level correlation.3

Footnote 3: We suspect this methodology is an artifact of how system-level correlations were first calculated for summarization in the DUC shared tasks when the dataset sizes were small enough that \(M_{\text{jud}}=M_{\text{test}}\) (e.g., Dang and Owczarzak, 2008).

To that end, we propose to modify the correlation definition to use all \(M_{\text{test}}\) instances to calculate the system scores for the automatic metrics. That is (differences in bold):

In practice with modern, large-scale datasets, this minor change could mean estimating system quality based on \(\approx\)10k inputs instead of around 100. This new definition now properly evaluates the way metrics are actually used by researchers.

We expect that scoring systems with \(M_{\text{test}}\) inputs instead of \(M_{\text{jud}}\) should lead to a better estimate of the true automatic metric score, which would in turn result in a lower-variance estimate of the correlation between \(\mathcal{X}\) and \(\mathcal{Z}\) in the form of smaller confidence intervals for \(r_{\text{sys}}\). In the next sections, we carry out analyses to demonstrate that this is true.

### Reducing Automatic Metric Variance

First, we empirically show that scoring systems with \(M_{\text{test}}\) instances instead of \(M_{\text{jud}}\) does indeed reduce the variance of the estimate of the automatic metric scores and subsequently increases the stabilities of the system rankings.

Ideally, the \(\mathcal{X}\) score for a system would be its "oracle" \(\mathcal{X}\) score, equal to the expected value of \(\mathcal{X}\) for a document sampled from the latent distribution over documents defined by the dataset (e.g., a system's ROUGE score on an infinite number of examples from a dataset). Since this cannot be calculated, it is approximated by averaging the \(\mathcal{X}\) score on a sample (i.e., either the \(M_{\text{jud}}\) or \(M_{\text{test}}\) input documents). Because \(M_{\text{test}}\gg M_{\text{jud}}\), we expect that the variance of this estimate using \(M_{\text{test}}\) inputs should be lower than when using \(M_{\text{jud}}\).

Figure 2: The bootstrapped 95% confidence intervals for the BERTScore of each system in the REALSumm dataset using \(M_{\text{jud}}\) judged instances in blue and \(M_{\text{test}}\) instances in orange. Evaluating systems with \(M_{\text{test}}\) instances leads to far better estimate of their true scores.

 

To quantify this, we calculated the variance of estimating the oracle \({\cal X}\) score using both \(M_{\text{jud}}\) and \(M_{\text{test}}\) input documents via bootstrapping. We randomly sampled \(M\) input documents with replacement, recomputed the system scores, and calculated the variance of those scores over 1k iterations. For all three metrics on both datasets, we found around a 99% reduction in the variance when \(M_{\text{test}}\) inputs were used instead of \(M_{\text{jud}}\), clearly demonstrating that evaluating systems with \(M_{\text{test}}\) inputs results in a better estimate of the system scores. In Fig. 2, this is visualized for BERTScore on the REALSUmm dataset.

However, because we are interested in evaluating the metrics' rankings, we also quantify how much of an effect this reduction in variance has on the stability of the system rankings induced by \({\cal X}\). Similarly to the system scores, there is an oracle ranking of systems for \({\cal X}\), equal to the ordering of systems by their respective oracle \({\cal X}\) scores (e.g., systems sorted by their ROUGE scores calculated on an infinite number of examples from a dataset). As the variance of the system score estimates decreases, the computed ranking of systems should begin to converge to the oracle \({\cal X}\) ranking. We aim to understand to what extent this happens if \(M_{\text{test}}\) instances are used for evaluation instead of \(M_{\text{jud}}\).

To quantify this notion, we calculate the Kendall's \(\tau\) between two system rankings for \({\cal X}\) that were based on two sets of \(M\) input documents, each sampled with replacement from the set of available documents. This simulates how much the system rankings would change if the evaluation procedure was run twice, each time with \(M\) random input documents. This quantity is calculated 1k times for various values of \(M\) and plotted in Fig. 3.

As \(M\) approaches \(M_{\text{test}}\), the automatic metrics' \(\tau\) values approach 1, which is much higher than the respective values at \(M_{\text{jud}}\), typically around 0.6-0.8. A value near 1 means that the rankings calculated using \(M_{\text{test}}\) inputs are almost constant, implying the rankings have converged to the oracle ranking. Therefore, the reduction in variance from evaluating on \(M_{\text{test}}\) instances does indeed greatly stabilize the system rankings.

Fig. 3 also contains the same analysis performed for the human judgments \({\cal Z}\) in both datasets, although it is limited to a maximum of \(M_{\text{jud}}\) input documents. We see that on both datasets the judgments' rankings are still quite variable, reaching a maximum \(\tau\) of around 0.8-0.85.

### Confidence Interval Analysis

Next, we show that the improved estimate of system scores leads to a more precise estimate of \(r_{\textsc{sys}}\) by demonstrating the widths of the confidence intervals for \(r_{\textsc{sys}}\) decrease.

The confidence intervals for \(r_{\textsc{sys}}\) calculated using bootstrapping methods proposed by Deutsch et al. (2021b) are rather wide. For instance, the 95% CI for ROUGE-2 on SummEval is \([-.09,,.84]\), demonstrating a rather high level of uncertainty in its value. This is problematic because it means we do not have a good picture of how reliable automatic evaluation metrics are. Reducing the width of the CIs will help us better understand the true metric quality.

We suspect that the large width of the confidence interval is due to the variance of the system rank

Figure 3: Bootstrapped estimates of the stabilities of the system rankings for automatic metrics and human annotations on SummEval (left) and REALSumm (right). The \(\tau\) value quantifies how similar two system rankings would be if they were computed with two random sets of \(M\) input documents. When all \(M_{\text{test}}\) test instances are used, the automatic metrics’ rankings become near constant. The error regions represent \(\pm 1\) standard deviation.

 

ings of the automatic metrics and human judgments. The more unstable the rankings are with respect to the \(M\) inputs, the larger the variance of the estimate of \(r_{\text{SYS}}\) should be since very different system rankings would be compared on each bootstrapping iteration. Deutsch et al. (2021b) used \(M_{\text{jud}}\) input documents to calculate their CIs. Therefore, we expect the improved stability of the automatic metric system rankings from evaluating on \(M_{\text{test}}\) instances should result in a more narrow confidence interval for \(r_{\text{SYS}}\) since some noise has been removed from this computation.

To demonstrate this, we calculated 95% CIs for \(r_{\text{SYS}}\) using the Boot-Input method on SummEval and REALSumm using both \(M_{\text{jud}}\) and \(M_{\text{test}}\) input documents, shown in Fig. 4. We find that the widths of the CIs shrank on average by 51% on SummEval and 16% on REALSumm. The largest decrease in width is in the ROUGE family of metrics on SummEval, potentially because that metric and dataset combination saw the biggest improvement in ranking stability (see Fig. 3). Thus, the improved estimate of the system scores did result in more precise estimates of \(r_{\text{SYS}}\). We repeated this analysis using the other bootstrapping methods proposed by Deutsch et al. (2021b), and the results are discussed in Appendix A.

### Conclusions & Recommendations

By estimating system quality using automatic metrics on all available instances instead of only those which were judged, we showed that the variances of the system scores and subsequent rankings reduce significantly, resulting in better estimates of \(r_{\text{SYS}}\). Because this methodology additionally directly evaluates the system scores used by researchers, we recommend future work do the same.

In order to continue to improve the estimate of \(r_{\text{SYS}}\), as much variance as possible needs to be removed from the system rankings. Evaluating systems using \(M_{\text{test}}\) instances removed a large amount of variance from the automatic metric rankings, but as demonstrated in Fig. 3, the human judgments still have a large amount of variance.

The human rankings' variances can either be reduced by judging more summaries per system or making the judgments more consistent. Since the human rankings' stabilities in Fig. 3 are mostly beginning to plateau -- especially for SummEval -- it may be prohibitively expensive to collect a sufficient number of judgments to better stabilize the rankings (Wei and Jia, 2021). Therefore, we expect the more feasible solution is to improve the consistency of the human judgments, for example by better training the annotators or improving the annotation protocol.

## 4 Evaluating with Realistic System Pairs

Next, we argue that the set of systems used to evaluate metrics is not reflective of how metrics are used in practice and propose a new system-level correlation variant to address this problem.

### Evaluating with All System Pairs

The \(N\) systems which are used for calculating system-level correlations are typically those which participated in a shared task, as in DUC/TAC (Dang and Owczarzak, 2008, among others), or those which have been published in the previous 3-4 years (Bhandari et al., 2020; Fabbri et al., 2021). As such, they are typically rather diverse in terms of their qualities, both as rated by human annotators and automatic metrics.

The system scores of all of the systems in the REALSumm dataset as evaluated by humans and automatic metrics are shown in Fig. 5. Clearly, the scores are rather diverse. For example, the systems cluster into low, medium, and high quality groups (with an additional outlier) as evaluated by ROUGE.

Figure 4: 95% confidence intervals for \(r_{\text{SYS}}\) calculated with the Boot-Inputs resampling method when the system rankings for the automatic metrics are calculated using only the judged data (orange) versus the entire test set (blue). Scoring systems with more summaries leads to better (more narrow) estimates of \(r_{\text{SYS}}\).

 

A difference of around 5 ROUGE points between them is a rather large gap for ROUGE scores.

The standard definition for a high quality evaluation metric is one which correctly ranks a set of systems with respect to human judgments. As such, the implementation of the system-level correlation calculated with Kendall's \(\tau\) will rank all \(N\) systems according to the human judgments and an automatic metric, then count how many pairs were ranked the same out of all \({N\choose 2}\) pairs (see SS2). As a consequence, even pairs of systems which are separated by a large margin according to the automatic metric -- likely systems with a clear difference in quality -- are included in the evaluation. Therefore, automatic metrics are rewarded for correctly ranking such "easy" system pairs.

### Evaluating with Realistic Pairs

This standard evaluation setting does not reflect how summarization metrics are actually used by researchers. New systems are typically only slightly better than previous work. Based on a survey of recent summarization papers in *ACL conferences (see Appendix C), we found that the average improvement over baseline/state-of-the-art models that was reported on the CNN/Dailymail dataset was on average 0.5 ROUGE-1. It is rarely the case that the improvement in automatic metrics is very large. Therefore, evaluating metrics using pairs of systems which _are_ separated by a large margin does not reflect the reality that metrics are very frequently used to compare those separated by a small margin. Including "easy" system pairs in the system-level correlation likely overestimates the quality of the metrics in settings which occur in practice.

To that end, we redefine a high quality evaluation metric to be one for which a small difference in scores reliably indicates a difference in quality. We quantify this by proposing a variant of the system-level \(\tau\) which is calculated between system pairs which are separated by a pre-defined automatic metric score margin. Instead of using all \({N\choose 2}\) system pairs, only pairs whose difference in scores falls within the margin are used to calculate the system-level correlation. We denote this correlation variant as \(r_{\textsc{sys}}\Delta(\ell,u)\) where \(\ell\) and \(u\) are the lower- and upper-bounds of the allowable differences in automatic metrics' scores. This would enable, for example, evaluating how well ROUGE correlates to human judgments on system pairs that are separated by 0.0-0.5 ROUGE points, thereby directly evaluating the scenario in which ROUGE is used to make decisions about system quality.

In Fig. 6 we report the \(r_{\textsc{sys}}\Delta(\ell,u)\) correlations for \(\ell=0.0\) and various values of \(u\) on both the SummEval and REALSumm datasets (more combinations of \(\ell\) and \(u\) are included in Appendix B). That is, we evaluate \(r_{\textsc{sys}}\) only on system pairs which are separated by at most an automatic score of \(u\). The values of \(u\) were selected by picking the minimum \(u\) which would result in evaluating on \(10\%,20\%,\ldots,100\%\) of the \({N\choose 2}\) possible system pairs closest in score to be consistent across all three metrics.

The correlations for each metric on the system pairs closest in score are far lower than the correlations evaluated on all of the system pairs. For instance, the correlation of BERTScore on SummEval with the closest 20% of system pairs (\(u\approx 0.2\)) is only 0.42 compared to 0.77 under the standard definition of \(r_{\textsc{sys}}\). Thus, it is clear that the metrics are much less reliable approximations of human judgments when the system scores are close than was previously known. Evaluating on all possible system pairs leads to an overly optimistic view of automatic metric quality.

The \(r_{\textsc{sys}}\Delta(\ell,u)\) correlation of ROUGE for \(\ell=0.0\) and \(u=0.5\) -- a typical improvement reported by researchers -- is \(0.08\) and \(0.0\) on the SummEval and REALSumm datasets. Therefore, these results suggest the most popular summarization evaluation metric agrees with human judgments of system quality in realistic scenarios only slightly better than or equal to random chance.

Figure 5: The systems (each represented by a point) on the two datasets (shown here for REALSumm) are rather diverse in quality as measured by both human judgments and automatic metrics.

 

This result also offers an explanation for why a naive metric such as ROUGE achieves moderately strong correlations under the standard definition of the system-level correlation (0.45 and 0.73 on SummEval and REALSumm) despite well known flaws and criticisms Passonneau et al. (2005); Conroy and Dang (2008); Deutsch and Roth (2020), among others): it has benefited from an easy evaluation protocol. Despite its simplicity, it is not too surprising that a large gap of 5-10 ROUGE points actually does correctly rank system pairs. Most of its positive correlation comes from such easy examples.

### Conclusions & Recommendations

If it is assumed that we have enough high-quality judgments to accurately discriminate between two similarly performing systems, then the results in Fig. 6 show that the correlations in realistic settings are trending very low, meaning automatic metrics are not nearly sensitive enough to distinguish between systems with only minor differences in quality. This is problematic because this is the scenario in which metrics are most frequently used, and therefore they are not very reliable methods of evaluating summarization systems. However, it is not all bad news. Because the standard system-level \(\tau\) values are moderately positive, consistent improvements in automatic metrics over time will likely result in better quality systems. Similarly to stochastic gradient descent, not every reported improvement is real, but on average over time, the quality does improve. Nonetheless, future work should focus on improving the quality of evaluation metrics when the differences in system performance are small, and researchers who compare systems should invest more effort into their human evaluations since automatic evaluations are not very reliable.

However, because the available number of system pairs to calculate the correlations in Fig. 6 is rather small -- especially when evaluating on the closest system pairs -- and recent work suggests we may not have enough human judgments to accurately distinguish between similarly performing systems Wei and Jia (2021), it could be difficult to reach any definitive conclusions about the metrics' correlations. That being said, these are our best estimates of the correlations with the available data. Not knowing how much we can trust automatic metrics is not a good outcome. In this scenario, future work should focus on collecting more, high-quality human judgments so that we can better meta-evaluate automatic metrics. Since we argue that it is important to distinguish between similarly performing systems, new data collection efforts should consider using targeted pairwise judgments between those systems instead of direct assessments across a variety of systems of diverse quality.

We recommend that proposals of new evaluation metrics also report correlations on system pairs with various differences in scores in addition to the standard system-level correlation definition. Re 

porting this information would better inform users of metrics about how likely humans would agree their observed improvement is real based on its value.

## 5 Related Work

The methodology behind meta-evaluating summarization evaluation metrics was established during the DUC/TAC shared tasks (Dang and Owczarzak, 2008, among others). In addition to competitions for developing high-quality summarization systems, there were also shared tasks for creating automatic metrics that correlated well with human judgments. The benchmark datasets created during DUC/TAC were small in size by today's standards because they were manually collected multi-document summarization datasets, which are hard to create at scale. As such, all of the model-generated summaries on the full test set were judged (so \(M_{\text{jud}}=M_{\text{test}}\); SS3), unlike for current datasets which are too large to fully judge.

Recently, there has been growing interest in revisiting the meta-evaluation of automatic evaluation metrics for summarization, in part due to the large differences between currently popular summarization datasets and those used in DUC/TAC. We view our work as continuing this direction of research, described next.

Peyrard (2019) argues that current evaluation metrics do not work as well when they are used to evaluate high-performing systems compared to those which were evaluated in DUC/TAC.

Both Fabbri et al. (2021) and Bhandari et al. (2020) re-evaluated how well existing evaluation metrics work on the popular CNN/DailyMail dataset (Nallapati et al., 2016) by collecting judgments of summary quality using recent state-of-the-art systems. These datasets were used in our analyses. While the goal of these works was to identify which metrics correlated best with human judgments, our goal is to point out the ways in which the current methodology of meta-evaluating metrics is inconsistent with how they are used.

Then, the work of Deutsch et al. (2021) proposed statistical methods for estimating and comparing correlation values. In contrast to our work, they provide statistical tools for analyzing correlations, whereas we propose new definitions of correlations.

Finally, Wei and Jia (2021) provided a theoretical analysis of the bias and variance of automatic and human evaluations of machine translations and summaries. Among their conclusions, they argue for evaluating metrics with pairwise accuracy (Kendall's \(\tau\)) and that it may be prohibitively expensive to collect enough human judgments to distinguish between two systems with very similar quality. Our work further argues that metrics should be evaluated with a variant of Kendall's \(\tau\) calculated using realistic system pairs (SS4). Unfortunately, their results suggest that collecting enough human judgments to accurately measure how well automatic metrics perform in this setting may be very difficult.

Related studies to ours have examined how the choice of which systems to include in metric evaluations impacts the correlation values. Both Mathur et al. (2020) and Bhandari et al. (2020) identify that metrics perform worse when scoring only the top-\(k\) systems in machine translation and summarization, respectively, and examine the use of pairwise comparisons for metric evaluation. Further, Mathur et al. (2020) demonstrate that outlier systems have an out-sized influence on the correlation values and recommend removing them from the metric evaluations. In contrast, our work proposes to change the evaluation methodology for metrics so that it more closely resembles how they are used in practice. This results in evaluating only on system pairs which are realistically compared by researchers, that is, those separated by small margins in automatic metric scores. We believe that this is a more principled approach to how to select which system pairs to evaluate on compared to previous work.

## 6 Conclusion

In this work, we proposed two independent changes to how the system-level correlation of metrics is calculated to better align with how they are used to evaluate systems. Our analyses showed that these modifications led to lower-variance estimates of correlations and that commonly reported improvements in metric scores may not reliably predict how humans would judge system quality. The results from the analyses point to the need for future data collection efforts of high-quality human judgments and improving automatic evaluation metrics when differences in system performance are small.

## Acknowledgments

The authors would like to thank the anonymous reviewers for their insightful feedback on our work 

This work was supported by Contracts FA8750-19-2-1004 and FA8750-19-2-0201 with the US Defense Advanced Research Projects Agency (DARPA). Approved for Public Release, Distribution Unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.

This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

This research is supported by a Focused Award from Google.

The second author is supported by the Eric and Wendy Schmidt Postdoctoral Award for Women in Mathematical and Computing Sciences.

## References

* Balach et al. (2021) Balach, Vidhisha ran, Artidoto Pagnoni, Jay Yoon Lee, Dheeraj Rajagopal, Jaime Carbonell, and Yulia Tsvetkov. 2021. StructSum: Summarization via Structured Representations. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2575-2585, Online. Association for Computational Linguistics.
* Bhandari et al. (2020) Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating Evaluation in Text Summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9347-9359, Online. Association for Computational Linguistics.
* Chen and Yang (2021) Jiaoa Chen and Diyi Yang. 2021. Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1380-1391, Online. Association for Computational Linguistics.
* Conroy and Dang (2008) John M. Conroy and Hoa Trang Dang. 2008. Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality. In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_, pages 145-152, Manchester, UK. Coling 2008 Organizing Committee.
* Dang and Owczarzak (2008) Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the TAC 2008 Update Summarization Task. In _Proc. of the Text Analysis Conference (TAC)_.
* Desai et al. (2020) Shrey Desai, Jiacheng Xu, and Greg Durrett. 2020. Compressive Summarization with Plausibility and Salience Modeling. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6259-6274, Online. Association for Computational Linguistics.
* Deutsch et al. (2021a) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021a. Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. _Transactions of the Association for Computational Linguistics_, 9.
* Deutsch et al. (2021b) Daniel Deutsch, Rotem Dror, and Dan Roth. 2021b. A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods. _Transactions of the Association for Computational Linguistics_, 9:1132-1146.
* Deutsch and Roth (2020) Daniel Deutsch and Dan Roth. 2020. Understanding the Extent to which Summarization Evaluatrics Measure the Information Quality of Summaries. _ArXiv_, abs/2010.12495.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dou et al. (2021) Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A General Framework for Guided Neural Abstractive Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4830-4842, Online. Association for Computational Linguistics.
* Fabbri et al. (2021) Alexander Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev. 2021. Sum-mEval: Re-evaluating Summarization Evaluation. _Transactions of the Association for Computational Linguistics_, 9:391-409.
* Huang et al. (2020) Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5094-5107, Online. Association for Computational Linguistics 

Yin Jou Huang and Sadao Kurohashi. 2021. Extractive Summarization Considering Discourse and Coreference Relations based on Heterogeneous Graph. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 3046-3052, Online. Association for Computational Linguistics.
* Jia et al. (2021) Ruipeng Jia, Yanan Cao, Fang Fang, Yuchen Zhou, Zheng Fang, Yanbing Liu, and Shi Wang. 2021. Deep Differential Amplifier for Extractive Summarization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 366-376, Online. Association for Computational Linguistics.
* Jia et al. (2020) Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, and Shi Wang. 2020. Neural Extractive Summarization with Hierarchical Attentive Heterogeneous Graph Network. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3622-3631, Online. Association for Computational Linguistics.
* Jin and Wan (2020) Hanqi Jin and Xiaojun Wan. 2020. Abstractive Multi-Document Summarization via Joint Learning with Single-Document Summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2545-2554, Online. Association for Computational Linguistics.
* Li et al. (2020) Zhenwen Li, Wenhao Wu, and Sujian Li. 2020. Composing Elementary Discourse Units in Abstractive Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6191-6196, Online. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2021a) Yang Liu, Sheng Shen, and Mirella Lapata. 2021a. Noisy Self-Knowledge Distillation for Text Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 692-703, Online. Association for Computational Linguistics.
* Liu et al. (2021b) Yixin Liu, Zi-Yi Dou, and Pengfei Liu. 2021b. Refactoring Neural Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1437-1448, Online. Association for Computational Linguistics.
* Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021. SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 1065-1072, Online. Association for Computational Linguistics.
* Louis and Nenkova (2013) Annie Louis and Ani Nenkova. 2013. Automatically Assessing Machine Summary Content Without a Gold Standard. _Computational Linguistics_, 39:267-300.
* Mathur et al. (2020) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4984-4997, Online. Association for Computational Linguistics.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Caglar Gulcehe, and Bing Xiang. 2016. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016_, pages 280-290. ACL.
* Nan et al. (2021) Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathlen McKeown, Ramesh Nallapati, Dejjao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving Factual Consistency of Abstractive Summarization via Question Answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6881-6894, Online. Association for Computational Linguistics.
* Narayan et al. (2020) Shashi Narayan, Joshua Maynez, Jakub Adamek, Daniele Pighin, Blaz Bratanic, and Ryan McDonald. 2020. Stepwise Extractive Summarization and Planning with Structured Transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4143-4159, Online. Association for Computational Linguistics.
* Padmakumar and He (2021) Vishakh Padmakumar and He He. 2021. Unsupervised Extractive Summarization using Pointwise Mutual Information. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2505-2512, Online. Association for Computational Linguistics.
* Passonneau et al. (2005) Rebecca J Passonneau, Ani Nenkova, Kathleen McKeown, and Sergey Sigelman. 2005. Applying the Pyramid Method in DUC 2005. In _Proceedings of the document understanding conference (DUC 05), Vancouver, BC, Canada_.
* Peyrard (2019) Maxime Peyrard. 2019. Studying Summarization Evaluation Metrics in the Appropriate Scoring Range. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5093-5100, Florence, Italy. Association for Computational Linguistics 

Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdam, and Ido Dagan. 2019. Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 682-687. Association for Computational Linguistics.
* Wang et al. (2020) Zhengjue Wang, Zhibin Duan, Hao Zhang, Chaojie Wang, Long Tian, Bo Chen, and Mingyuan Zhou. 2020. Friendly Topic Assistant for Transformer Based Abstractive Summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 485-497, Online. Association for Computational Linguistics.
* Wei and Jia (2021) Johnny Wei and Robin Jia. 2021. The Statistical Advantage of Automatic NLG Metrics at the System Level. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6840-6854, Online. Association for Computational Linguistics.
* Xiao et al. (2020) Liqiang Xiao, Lu Wang, Hao He, and Yaohui Jin. 2020. Modeling Content Importance for Summarization with Pre-trained Language Models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3606-3611, Online. Association for Computational Linguistics.
* Xing et al. (2021) Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the Lead Bias in News Summarization via Alternating Adversarial Learning. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 948-954, Online. Association for Computational Linguistics.
* Xu et al. (2020a) Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020a. Discourse-Aware Neural Extractive Text Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5021-5031, Online. Association for Computational Linguistics.
* Xu et al. (2020b) Song Xu, Haoran Li, Peng Yuan, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020b. Self-Attention Guided Copy Mechanism for Abstractive Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1355-1362, Online. Association for Computational Linguistics.
* Yang et al. (2020) Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve. 2020. TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1865-1874, Online. Association for Computational Linguistics.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In _Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Zhong et al. (2020) Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive Summarization as Text Matching. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6197-6208, Online. Association for Computational Linguistics.
* Zou et al. (2020) Yanyan Zou, Xingxing Zhang, Wei Lu, Furu Wei, and Ming Zhou. 2020. Pre-training for Abstractive Document Summarization by Reinstating Source Text. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3646-3660, Online. Association for Computational Linguistics.

## Appendix A Additional Confidence Interval Results

In addition to the Boot-Inputs CI method proposed by Deutsch et al. (2021), the authors also proposed Boot-Systems and Boot-Both. Each of the three methods makes assumptions about whether the set of \(N\) systems and \(M\) input documents are fixed or variable during the bootstrapping calculation. For instance, Boot-Inputs assumes the \(N\) systems are always the same and the \(M\) input documents are random, then subsequently re-samples \(M\) input documents on each bootstrapping iteration to calculate the confidence interval. BootstrapSystems does the opposite by resampling which \(N\) systems are used while holding the original \(M\) input documents fixed. Boot-Both assumes both the systems and inputs are variable.

Figs. 7 and 8 contain the 95% CIs for ROUGE, BERTScore, and QAEval on the SummEval and REALSumm datasets using the Boot-Systems and Boot-Both methods calculated using all \(M_{\text{test}}\) test instances and only the \(M_{\text{jud}}\) annotated instances (Boot-Inputs included in the main body 

of the paper, Fig. 4). The widths of the Bootstrap CIs decreased by 14% and 12%, whereas the Boot-Systems CIs only decreased by 1% and 6%.

The Boot-Systems widths likely decreased less because its estimation of is not dependent on the variance of the system score estimates. Since the set of input documents is fixed, the system scores do not change at all during bootstrapping, so increasing the number of summaries used to estimate those scores should not have a major effect on the estimation of .

## Appendix B Additional Results

Fig. 9 contains the correlations for when for ROUGE-1, ROUGE-2, and ROUGE-L, equivalent to those shown in Fig. 6 in the main body of the paper (ROUGE-1 is shown in both). The ROUGE-2 and ROUGE-L results are largely consistent with those of ROUGE-1. The metrics' correlations to human annotations are low (or even negative) when the differences between system scores are small. As more pairs are added that differ by larger margins, the correlations increase.

Figs. 10 and 11 contain the correlations for ROUGE, BERTScore, and QAEval for various combinations of and on both the SummEval and REALSumm datasets. The first rows of each heatmap are plotted in Figs. 6 and 9.

We see that as the allowed score gap between system pairs is allowed to increase (i.e., adding "easier" pairs to rank), the correlation increases by a large margin over the correlation on pairs close in score. All of the metrics have nearly perfect correlation when the system pairs are separated by large margins.

## Appendix C Summarization Paper Survey

To estimate the difference in ROUGE-1 score that is commonly reported in papers, we performed a survey of recently published summarization papers. We selected papers from 2020 and 2021 that were published in a *ACL conference (including Findings), had "summary" or "summarization" in the title, proposed a new system, and compared systems on the CNN/DailyMail dataset with ROUGE. We selected the differences between the best two models that were compared with ROUGE on the test set. We did not include ablation experiments for which the differences are likely smaller than the differences between the top two performing systems. The results are shown in Table 1. The average reported difference was found to be 0.49 ROUGE-1.

 



\begin{table}
\begin{tabular}{l l c c r r} \hline
**Paper** & **Model 1** & **Score 1** & **Model 2** & **Score 2** & \(\Delta\) \\ \hline Xu et al. (2020b) & SAGCopy Indegree-2 & 42.56 & MASS+Copy & 41.71 & 0.85 \\ Xu et al. (2020a) & DiscoBert w. \(\mathcal{G}_{R}\) \& \(\mathcal{G}_{C}\) & 43.77 & BERTSum & 43.25 & 0.52 \\ Huang et al. (2020) & BART & 44.16 & ASGRAD-DOC + \(R_{\text{ROUGE}}+R_{\text{Clare}}\) & 43.93 & 0.23 \\ Li et al. (2020) & EDUSum & 41.40 & Fast-Abs & 40.88 & 0.52 \\ Zhong et al. (2020) & MatchSum (RoBERTa-Base) & 44.41 & BERTSum & 43.85 & 0.56 \\ Wang et al. (2020) & BERTSum+TA & 43.06 & BERTSum & 42.13 & 0.93 \\ Jia et al. (2020) & HAHSum-Large & 44.68 & MatchSum-Base & 44.41 & 0.27 \\ Zou et al. (2020) & STEP (SIGA-CM) & 44.07 & UniLM & 43.47 & 0.60 \\ Yang et al. (2020) & TED 10LSH & 38.73 & Pretrained 10L8H & 38.38 & 0.35 \\ Desai et al. (2020) & MatchSum + CUPS & 44.69 & MatchSum & 44.41 & 0.28 \\ Xiao et al. (2020) & IHP + BERT (MLM) & 37.53 & ImP + XLNet (PLM) & 37.04 & 0.49 \\ Jin and Wan (2020) & Ours & 41.70 & CopyTransformer & 41.39 & 0.31 \\ Jia et al. (2021) & DifferSum-Large & 44.70 & MatchSum-Base & 44.41 & 0.29 \\ Nan et al. (2021) & Q-C-O & 44.70 & MLE & 44.24 & 0.46 \\ Xing et al. (2021) & Shuffling & 41.00 & Our Method & 40.88 & 0.12 \\ Liu and Liu (2021) & SimCLS & 46.67 & GSum & 45.94 & 0.73 \\ Padmauxam and He (2021) & PacSum & 40.26 & Lead-k & 39.69 & 0.57 \\ Balach et al. (2021) & Explicit-Structure Attention & 39.63 & Pointer-Generator + Coverage & 39.07 & 0.56 \\ Huang and Kurohashi (2021) & DiscoSBERT & 43.77 & Proposed & 43.61 & 0.16 \\ Liu et al. (2021a) & UniLMv2 + SKD + Noisy T + Noisy S & 43.77 & UniLMv2 & 43.45 & 0.43 \\ Chen and Yang (2021) & S-BART w. Discourse \& Action & 46.07 & Multi-View Seq2Seq & 45.56 & 0.51 \\ Liu et al. (2021b) & GSUm-Fine-Tuned & 46.18 & Base & 45.93 & 0.25 \\ Dou et al. (2021) & BART + MatchSum & 45.94 & BART & 44.66 & 1.28 \\ \multicolumn{6}{c}{Average} & & & 0.49 \\ \hline \end{tabular}
\end{table}
Table 1: A survey of recent summarization papers published in *ACL conferences and the differences in ROUGE-1 score they reported on the CNN/DailyMail dataset. Narayan et al. (2020) is not included because it is not clear which results were considered comparable to their model.



Figure 10: \(r_{\text{sys}}\Delta(\ell,u)\) correlations for various combinations of \(\ell\) and \(u\) (see §4.2) for ROUGE (top), BERTScore (middle), and QAEval (bottom) on SummEval (left) and REALSumm (right). The values of \(\ell\) and \(u\) were chosen so that each value in the heatmaps evaluates on 10% more system pairs than the value to its left. For instance, the first row evaluates on \(10\%,20\%,\dots,100\%\) of the system pairs. The second row evaluates on \(10\%,20\%,\dots,90\%\) of the system pairs, never including the \(10\%\) of pairs which are closest in score. The first row of each of the heatmaps is plotted in Fig. 6. The correlations on realistic score differences between systems are in the upper left portion of the heatmaps and contain the lowest correlations overall. Evaluating on all pairs is the top-rightmost entry, and the “easiest” pairs (those separated by a large score margin) are in the bottom right.

 

Figure 11: See Fig. 10 for a description of the heatmaps, shown here for ROUGE-2 (top) and ROUGE-L (bottom).