SlowGuess/NAACL_2022_OCR huggingface dataset:
In Fig. 6 we report the  $r_{\mathrm{SYS}} \Delta(\ell, u)$  correlations for  $\ell = 0.0$  and various values of  $u$  on both the SummEval and REALSumm datasets (more combinations of  $\ell$  and  $u$  are included in Appendix B). That is, we evaluate  $r_{\mathrm{SYS}}$  only on system pairs which are separated by at most an automatic score of  $u$ . The values of  $u$  were selected by picking the minimum  $u$  which would result in evaluating on  $10\%, 20\%, \ldots, 100\%$  of the  $\binom{N}{2}$  possible system pairs closest in score to be consistent across all three metrics.
The correlations for each metric on the system pairs closest in score are far lower than the correlations evaluated on all of the system pairs. For instance, the correlation of BERTScore on SummEval with the closest  $20\%$  of system pairs  $(u \approx 0.2)$  is only 0.42 compared to 0.77 under the standard definition of  $r_{\mathrm{SYS}}$ . Thus, it is clear that the metrics are much less reliable approximations of human judgments when the system scores are close than was previously known. Evaluating on all possible system pairs leads to an overly optimistic view of automatic metric quality.
The  $r_{\mathrm{SYS}} \Delta(\ell, u)$  correlation of ROUGE for  $\ell = 0.0$  and  $u = 0.5$  — a typical improvement reported by researchers — is 0.08 and 0.0 on the Summ-Eval and REALSumm datasets. Therefore, these results suggest the most popular summarization evaluation metric agrees with human judgments of system quality in realistic scenarios only slightly better than or equal to random chance.
Figure 6: The  $r_{\mathrm{SYS}} \Delta(\ell, u)$  correlations on the SummEval (top) and REALSumm (bottom) datasets for  $\ell = 0$  and various values of  $u$  (additional combinations of  $\ell$  and  $u$  can be found in Appendix B). The  $u$  values were chosen to select the  $10\%$ ,  $20\%$ , ...,  $100\%$  of the pairs of systems closest in score. Each  $u$  is displayed on the top of each plot. For instance,  $20\%$  of the  $\binom{N}{2}$  system pairs on SummEval are separated by  $< 0.5$  ROUGE-1, and the system-level correlation on those pairs is around 0.08. As more systems are used in the correlation calculation, the allowable gap in scores between system pairs increases, and are therefore likely easier to rank, resulting in higher correlations.
This result also offers an explanation for why a naive metric such as ROUGE achieves moderately strong correlations under the standard definition of the system-level correlation (0.45 and 0.73 on SummEval and REALSumm) despite well known flaws and criticisms (Passonneau et al., 2005; Conroy and Dang, 2008; Deutsch and Roth, 2020, among others): it has benefited from an easy evaluation protocol. Despite its simplicity, it is not too surprising that a large gap of 5-10 ROUGE points actually does correctly rank system pairs. Most of its positive correlation comes from such easy examples.
4.3 Conclusions & Recommendations
If it is assumed that we have enough high-quality judgments to accurately discriminate between two similarly performing systems, then the results in Fig. 6 show that the correlations in realistic settings are trending very low, meaning automatic metrics are not nearly sensitive enough to distinguish between systems with only minor differences in quality. This is problematic because this is the scenario in which metrics are most frequently used, and therefore they are not very reliable methods of evaluating summarization systems. However, it is not all bad news. Because the standard system-level  $\tau$  values are moderately positive, consistent improvements in automatic metrics over time will likely result in better quality systems. Similarly to stochastic gradient descent, not every reported improvement is real, but on average over time, the
quality does improve. Nonetheless, future work should focus on improving the quality of evaluation metrics when the differences in system performance are small, and researchers who compare systems should invest more effort into their human evaluations since automatic evaluations are not very reliable.
However, because the available number of system pairs to calculate the correlations in Fig. 6 is rather small — especially when evaluating on the closest system pairs — and recent work suggests we may not have enough human judgments to accurately distinguish between similarly performing systems (Wei and Jia, 2021), it could be difficult to reach any definitive conclusions about the metrics' correlations. That being said, these are our best estimates of the correlations with the available data. Not knowing how much we can trust automatic metrics is not a good outcome. In this scenario, future work should focus on collecting more, high-quality human judgments so that we can better meta-evaluate automatic metrics. Since we argue that it is important to distinguish between similarly performing systems, new data collection efforts should consider using targeted pairwise judgments between those systems instead of direct assessments across a variety of systems of diverse quality.
We recommend that proposals of new evaluation metrics also report correlations on system pairs with various differences in scores in addition to the standard system-level correlation definition. Re
porting this information would better inform users of metrics about how likely humans would agree their observed improvement is real based on its value.
The methodology behind meta-evaluating summarization evaluation metrics was established during the DUC/TAC shared tasks (Dang and Owczarzak, 2008, among others). In addition to competitions for developing high-quality summarization systems, there were also shared tasks for creating automatic metrics that correlated well with human judgments. The benchmark datasets created during DUC/TAC were small in size by today's standards because they were manually collected multi-document summarization datasets, which are hard to create at scale. As such, all of the model-generated summaries on the full test set were judged (so  $M_{\mathrm{jud}} = M_{\mathrm{test}}$ ; §3), unlike for current datasets which are too large to fully judge.
Recently, there has been growing interest in revisiting the meta-evaluation of automatic evaluation metrics for summarization, in part due to the large differences between currently popular summarization datasets and those used in DUC/TAC. We view our work as continuing this direction of research, described next.
Peyrard (2019) argues that current evaluation metrics do not work as well when they are used to evaluate high-performing systems compared to those which were evaluated in DUC/TAC.
Both Fabbri et al. (2021) and Bhandari et al. (2020) re-evaluated how well existing evaluation metrics work on the popular CNN/DailyMail dataset (Nallapati et al., 2016) by collecting judgments of summary quality using recent state-of-the-art systems. These datasets were used in our analyses. While the goal of these works was to identify which metrics correlated best with human judgments, our goal is to point out the ways in which the current methodology of meta-evaluating metrics is inconsistent with how they are used.
Then, the work of Deutsch et al. (2021b) proposed statistical methods for estimating and comparing correlation values. In contrast to our work, they provide statistical tools for analyzing correlations, whereas we propose new definitions of correlations.
Finally, Wei and Jia (2021) provided a theoretical analysis of the bias and variance of automatic
and human evaluations of machine translations and summaries. Among their conclusions, they argue for evaluating metrics with pairwise accuracy (Kendall's  $\tau$ ) and that it may be prohibitively expensive to collect enough human judgments to distinguish between two systems with very similar quality. Our work further argues that metrics should be evaluated with a variant of Kendall's  $\tau$  calculated using realistic system pairs ( $\S 4$ ). Unfortunately, their results suggest that collecting enough human judgments to accurately measure how well automatic metrics perform in this setting may be very difficult.
Related studies to ours have examined how the choice of which systems to include in metric evaluations impacts the correlation values. Both Mathur et al. (2020) and Bhandari et al. (2020) identify that metrics perform worse when scoring only the top- $k$  systems in machine translation and summarization, respectively, and examine the use of pairwise comparisons for metric evaluation. Further, Mathur et al. (2020) demonstrate that outlier systems have an out-sized influence on the correlation values and recommend removing them from the metric evaluations. In contrast, our work proposes to change the evaluation methodology for metrics so that it more closely resembles how they are used in practice. This results in evaluating only on system pairs which are realistically compared by researchers, that is, those separated by small margins in automatic metric scores. We believe that this is a more principled approach to how to select which system pairs to evaluate on compared to previous work.
In this work, we proposed two independent changes to how the system-level correlation of metrics is calculated to better align with how they are used to evaluate systems. Our analyses showed that these modifications led to lower-variance estimates of correlations and that commonly reported improvements in metric scores may not reliably predict how humans would judge system quality. The results from the analyses point to the need for future data collection efforts of high-quality human judgments and improving automatic evaluation metrics when differences in system performance are small.
The authors would like to thank the anonymous reviewers for their insightful feedback on our work.
This work was supported by Contracts FA8750-19-2-1004 and FA8750-19-2-0201 with the US Defense Advanced Research Projects Agency (DARPA). Approved for Public Release, Distribution Unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNl, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
This research is supported by a Focused Award from Google.
The second author is supported by the Eric and Wendy Schmidt Postdoctoral Award for Women in Mathematical and Computing Sciences.
Balach, Vidhisha ran, Artidoro Pagnoni, Jay Yoon Lee, Dheeraj Rajagopal, Jaime Carbonell, and Yulia Tsvetkov. 2021. StructSum: Summarization via Structured Representations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2575-2585, Online. Association for Computational Linguistics.
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Reevaluating Evaluation in Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online. Association for Computational Linguistics.
Jiaao Chen and Diyi Yang. 2021. Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1380-1391, Online. Association for Computational Linguistics.
John M. Conroy and Hoa Trang Dang. 2008. Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality. In Proceedings
of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 145-152, Manchester, UK. Coling 2008 Organizing Committee.
Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the TAC 2008 Update Summarization Task. In Proc. of the Text Analysis Conference (TAC).
Shrey Desai, Jiacheng Xu, and Greg Durrett. 2020. Compressive Summarization with Plausibility and Salience Modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6259-6274, Online. Association for Computational Linguistics.
Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021a. Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. Transactions of the Association for Computational Linguistics, 9.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2021b. A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods. Transactions of the Association for Computational Linguistics, 9:1132-1146.
Daniel Deutsch and Dan Roth. 2020. Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries. ArXiv, abs/2010.12495.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A General Framework for Guided Neural Abstractive Summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4830-4842, Online. Association for Computational Linguistics.
Alexander Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5094-5107, Online. Association for Computational Linguistics.
Yin Jou Huang and Sadao Kurohashi. 2021. Extractive Summarization Considering Discourse and Coreference Relations based on Heterogeneous Graph. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3046-3052, Online. Association for Computational Linguistics.
Ruipeng Jia, Yanan Cao, Fang Fang, Yuchen Zhou, Zheng Fang, Yanbing Liu, and Shi Wang. 2021. Deep Differential Amplifier for Extractive Summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 366-376, Online. Association for Computational Linguistics.
Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, and Shi Wang. 2020. Neural Extractive Summarization with Hierarchical Attentive Heterogeneous Graph Network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3622-3631, Online. Association for Computational Linguistics.
Hanqi Jin and Xiaojun Wan. 2020. Abstractive Multi-Document Summarization via Joint Learning with Single-Document Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2545–2554, Online. Association for Computational Linguistics.
Zhenwen Li, Wenhao Wu, and Sujian Li. 2020. Composing Elementary Discourse Units in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6191-6196, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu, Sheng Shen, and Mirella Lapata. 2021a. Noisy Self-Knowledge Distillation for Text Summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 692-703, Online. Association for Computational Linguistics.
Yixin Liu, Zi-Yi Dou, and Pengfei Liu. 2021b. RefSum: Refactoring Neural Summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1437-1448, Online. Association for Computational Linguistics.
Yixin Liu and Pengfei Liu. 2021. SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
on Natural Language Processing (Volume 2: Short Papers), pages 1065-1072, Online. Association for Computational Linguistics.
Annie Louis and Ani Nenkova. 2013. Automatically Assessing Machine Summary Content Without a Gold Standard. Computational Linguistics, 39:267-300.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984-4997, Online. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Caglar Gülçehre, and Bing Xiang. 2016. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 280-290. ACL.
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving Factual Consistency of Abstractive Summarization via Question Answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6881-6894, Online. Association for Computational Linguistics.
Shashi Narayan, Joshua Maynez, Jakub Adamek, Daniele Pighin, Blaz Bratanic, and Ryan McDonald. 2020. Stepwise Extractive Summarization and Planning with Structured Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4143-4159, Online. Association for Computational Linguistics.
Vishakh Padmakumar and He He. 2021. Unsupervised Extractive Summarization using Pointwise Mutual Information. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2505-2512, Online. Association for Computational Linguistics.
Rebecca J Passonneau, Ani Nenkova, Kathleen McKeown, and Sergey Sigelman. 2005. Applying the Pyramid Method in DUC 2005. In Proceedings of the document understanding conference (DUC 05), Vancouver, BC, Canada.
Maxime Peyrard. 2019. Studying Summarization Evaluation Metrics in the Appropriate Scoring Range. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5093-5100, Florence, Italy. Association for Computational Linguistics.
Ori Shapira, David Gabay, Yang Gao, Hadar Rosen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 682-687. Association for Computational Linguistics.
Zhengjue Wang, Zhibin Duan, Hao Zhang, Chaojie Wang, Long Tian, Bo Chen, and Mingyuan Zhou. 2020. Friendly Topic Assistant for Transformer Based Abstractive Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 485-497, Online. Association for Computational Linguistics.
Johnny Wei and Robin Jia. 2021. The Statistical Advantage of Automatic NLG Metrics at the System Level. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6840-6854, Online. Association for Computational Linguistics.
Liqiang Xiao, Lu Wang, Hao He, and Yaohui Jin. 2020. Modeling Content Importance for Summarization with Pre-trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3606-3611, Online. Association for Computational Linguistics.
Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the Lead Bias in News Summarization via Alternating Adversarial Learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 948-954, Online. Association for Computational Linguistics.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020a. Discourse-Aware Neural Extractive Text Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computational Linguistics.
Song Xu, Haoran Li, Peng Yuan, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020b. Self-Attention Guided Copy Mechanism for Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1355–1362, Online. Association for Computational Linguistics.
Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve. 2020. TED: A Pretrained Unsupervised Summarization Model
with Theme Modeling and Denoising. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1865-1874, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive Summarization as Text Matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208, Online. Association for Computational Linguistics.
Yanyan Zou, Xingxing Zhang, Wei Lu, Furu Wei, and Ming Zhou. 2020. Pre-training for Abstractive Document Summarization by Reinstating Source Text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3646-3660, Online. Association for Computational Linguistics.
A Additional Confidence Interval Results