SlowGuess/NAACL_2021_OCR huggingface dataset:
Since our benchmark contains diverse summaries from different datasets and models, dataset biases can hamper accurate reporting. In Figure 3, we visually show correlations between two factuality metrics (FEQA and FactCC) and human judgement on the entire data and on partitions of the data. For both metrics, we notice that the slope (an unscaled measure of correlation) of the line fitted through the entire data (red line) is significantly larger. In FEQA, the dotted lines (fitted on subsets of the data of each model and dataset) are almost horizontal. This likely indicates the presence of a confounding variable associated with the properties of each system and dataset. This can lead to false measures of high correlation if not accounted for. To address this, we suggest to control for confounding variables using partial correlations. We include details on partial correlations in the Appendix. In this case, both the system and the dataset are taken to be confounding variables.
In Table 2, we report the partial Pearson correlation and Spearman rank correlation coefficients with human judgements for each metric, along with their
All data	CNN/DM	XSum
Metrics	Pearson	Spearman	Pearson	Spearman	Pearson	Spearman
ρ	p-val	r	p-val	ρ	p-val	r	p-val	ρ	p-val	r	p-val
BLEU	0.10	0.00	0.07	0.00	0.08	0.01	0.08	0.01	0.14	0.00	0.20	0.00
METEOR	0.14	0.00	0.11	0.00	0.12	0.00	0.10	0.00	0.15	0.00	0.10	0.00
Rouge-1	0.14	0.00	0.10	0.00	0.12	0.00	0.10	0.00	0.15	0.00	0.09	0.01
Rouge-2	0.12	0.00	0.08	0.00	0.08	0.00	0.07	0.01	0.17	0.00	0.14	0.00
Rouge-L	0.13	0.00	0.09	0.00	0.11	0.00	0.09	0.00	0.16	0.00	0.10	0.00
OpenIE	0.11	0.00	0.02	0.36	0.16	0.00	0.15	0.00	0.00	0.93	-0.45	0.00
BERTS P	-0.02	0.35	-0.01	0.69	0.00	0.95	-0.01	0.65	-0.04	0.25	0.02	0.57
BERTS R	-0.03	0.14	-0.05	0.03	-0.04	0.18	-0.06	0.04	-0.03	0.34	0.02	0.58
BERTS F1	-0.03	0.16	-0.03	0.13	-0.02	0.43	-0.05	0.10	-0.04	0.26	0.02	0.53
FEQA	0.00	0.83	0.01	0.60	-0.01	0.76	-0.01	0.72	0.02	0.45	0.07	0.04
QAGS	0.06	0.00	0.08	0.00	0.13	0.00	0.09	0.00	-0.02	0.48	0.01	0.65
DAE	0.16	0.00	0.14	0.00	0.25	0.00	0.24	0.00	0.04	0.16	0.28	0.00
FactCC	0.20	0.00	0.30	0.00	0.36	0.00	0.33	0.00	0.07	0.02	0.25	0.00
Table 2: Partial Pearson correlation and Spearman rank correlation coefficients and p-values between human judgements and metrics scores. Comparisons should be made along with the pairwise Williams test found in Table 4.
$p$ -values indicating statistical significance.
How do different metrics correlate with human judgements? From Table 2 we observe that all metrics exhibit low correlations with human judgements of factuality. The best metric overall is FactCC with 0.20 Pearson and 0.30 Spearman correlation. Interestingly, we observe that general summarization metrics BLEU, Rouge, and METEOR, and the OpenIE baseline have statistically significant correlations with factuality, close to FactCC ( $\rho = 0.14$  for Rouge-1 and METEOR versus  $\rho = 0.20$  for FactCC). The entailment metrics (FactCC and DAE) have the two highest correlations and are statistically significant. The two QGA metrics have lower overall correlation. FEQA's correlation is not statistically significant. QAGS has low, but significant correlation of  $\rho = 0.06$ .
How well do different metrics capture errors in different datasets? In Figure 4, we observe that entailment metrics have significantly higher partial Pearson correlation on the CNN/DM dataset than XSum where their correlation is reduced by a factor of four. QAGS and the OpenIE baseline have similar behavior. This suggests that these metrics capture the error types from CNN/DM better that those from XSum. Specifically, XSum has uniquely high Out of Article (OutE) errors which they might not capture well. This also highlights the importance of data diversity in building and benchmarking factuality metrics to avoid overfit
ting to certain types of errors.
How well do different metrics capture errors from pretrained and non-pretrained models? On the CNN/DM dataset we observe that entailment metrics and QAGS perform significantly better on non-pretrained models. This indicates that the artificial factual errors on which entailment metrics are trained on are closest to the mistakes that non-pretrained models make. This also suggests that the errors made by pretrained models might be more difficult to capture by these metrics. These trends are less clear on the XSum dataset which we again attribute to high Out of Article (OutE) errors in the pretrained and non-pretrained models (ref Figure 2)
Figure 4 shows partial Pearson correlation on six subsets of the data. To understand capabilities of metrics across the broad categories of errors (semantic frame errors, discourse errors, and content verifiability errors) we perform an ablation study. For each category, we compute the variation in partial correlation with errors from that category omitted. In Figure 5, we visualize the influence of a given type of error using the variation for each metric and category. A higher positive bar indicates that the error type was a significant contributor to the overall correlation (or metric highly correlates with error) causing the correlation without it to
Figure 4: Partial Pearson correlation on different partitions of the data. Entailment metrics have highest correlation on pretrained models in the CNN/DM dataset. Their performance degrades significantly on XSum.
Figure 5: Variation in partial Pearson correlation when omitting error types. Higher variation indicates greater influence of an error type in the overall correlation.
General Summarization metrics Unsurprisingly, we observe that Rouge L is best correlated with content verifiability errors (which contains Out of Article Errors) as n-gram matches detect them. Rouge L has negative correlation with semantic frame errors and low correlation with discourse level errors indicating that n-gram matching fails to capture them. We observe that OpenIE is more correlated with semantic frame errors. The metric matches entities and verifies the predicate that relates them and hence is able to capture semantic frame errors. BertScore has low correlation overall, being more correlated with content verifiability errors and negatively correlated with discourse errors.
QGA metrics Both QGA metrics have negative correlation with discourse errors suggesting that QGA metrics are not able to capture coreference errors or discourse link errors potentially due to the entity oriented questions in their training data. FEQA additionally is also negatively correlated with semantic frame errors and has low positive correlation with content verifiability errors. In contrast QAGS is best correlated with semantic frame errors.
Entailment metrics Both entailment metrics correlate well with semantic frame and content verifiability errors. DAE has the highest correla
tion of all metrics with discourse errors suggesting that entailment at the dependency level can help model discourse errors (CorefE and LinkE). FactCC is nearly uncorrelated in this category, indicating that artificially generated factual errors need to go beyond simple pronoun swaps to train models to capture discourse errors. FactCC had best overall partial correlation which can be attributed to FactCC being able to capture semantic frame and content verifiability errors well.
Kryscinski et al. (2019) and Fabbri et al. (2020) find that standard n-gram based metrics have low correlation with human judgements of factuality. Motivated by this, several automated metrics falling in two paradigms were proposed to improve the evaluation of factuality.
Entailment Classification Goodrich et al. (2019); Kryscinski et al. (2020); Maynez et al. (2020); Goyal and Durrett (2020) model factuality as entailment classification breaking down the summary into smaller units, such as sentences, which are verified against the original article. However, modeling factuality as a classification task requires supervision on factual and hallucinated data. FactCC (Kryscinski et al., 2020) is trained on the CNN/DM dataset augmented with four types of artificial mistakes as supervision.
Question Generation and Answering (QGA) FEQA (Durmus et al., 2020) and QAGS (Wang et al., 2020) are two metrics which reduce factuality evaluation to question generation and answering. These methods use a question generation model to obtain questions from the output summary and a question answering model to answer them, separately using the article and the output summary.
Prior Efforts on Factuality Annotations of Summaries Fabbri et al. (2020) and Maynez et al.
(2020) have collected annotations on the CNN/DM and XSum dataset respectively. In this work we cover both datasets to ensure greater data diversity. Other efforts (Kryscinski et al., 2020; Wang et al., 2020; Durmus et al., 2020) were smaller in scale Durmus et al. (2020) and Kryscinski et al. (2020) annotated 200 and 503 sentences while Wang et al. (2020) annotated 470 summaries (we collect judgements on 2250 summaries). Crucially, all previous efforts portray factuality as a binary label without variations in degree or type of factual errors.
In this work we provide a linguistically grounded typology of factual errors which we use to collect FRANK, a dataset of human annotations of 2250 summaries covering both CNN/DM and XSum datasets. We use FRANK to assess the factuality of summarization systems and benchmark recently proposed factuality metrics highlighting the types of errors they can capture. With the FRANK benchmark we have started moving away from a summary-level binary understanding of factuality.
We have collected crowd annotations using the Amazon Mechanical Turk platform. Workers were paid  $50\%$  more than the average American minimum wage and offered additional bonuses as an incentive to maintain high quality work. No information about the workers will be released and worker IDs will be anonymized.
The authors are grateful to the anonymous reviewers for their feedback, and to Anjalie Field, Rishabh Joshi, Alissa Ostapenko, Dheeraj Rajagopal, Evangelia Spiliopoulou, Shuly Wintner, and the members of the Tsvetshop group for their invaluable feedback and support in various stages of the project. This material is based upon work supported by the DARPA CMO under Contract No. HR001120C0124, and in part by the National Science Foundation under Grants No. IIS2040926 and No. IIS2007960. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily state or reflect those of the United States Government or any agency thereof.
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 86-90, Montreal, Quebec, Canada. Association for Computational Linguistics.
Vidhisha Balachandran, Artidoro Pagnoni, Jay Yoon Lee, Dheeraj Rajagopal, Jaime Carbonell, and Yulia Tsvetkov. 2021. StructSum: Summarization via structured representations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2575-2585, Online. Association for Computational Linguistics.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, IJCAI'07, pages 2670-2676, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Gillian Brown and G. Yule. 1983. Discourse Analysis. Cambridge Textbooks in Linguistics. Cambridge University Press.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4784-4791. AAAI Press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Alexander R Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Charles J Fillmore et al. 1976. Frame semantics and the nature of language. In Proceedings of the Annals of the New York Academy of Sciences: Conference on the origin and development of language and speech, volume 280, pages 20-32. New York.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 166-175. ACM.
Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3592-3603, Online. Association for Computational Linguistics.
Yvette Graham. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128-137, Lisbon, Portugal. Association for Computational Linguistics.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693-1701.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228-231, Prague, Czech Republic. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243-281.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of
the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264-280.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Roser Sauri. 2008. A Factuality Profiler for Eventualities in Text. Ph.D. thesis, Brandeis University, USA.
Roser Sauri and James Pustejovsky. 2012. Are you sure that this happened? assessing the factuality degree of events in text. Computational Linguistics, 38(2):261-299.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885-895, New Orleans, Louisiana. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with BERT.* In *Proceedings of the 8th International Conference on Learning Representations*, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A.1 Model details
We provide details of the models used in the human evaluation task to construct FRANK.
A.1.1 CNN/DM dataset
On the CNN/DM (Hermann et al., 2015) dataset we use five different models. We use the preprocessed model outputs provided by Fabbri et al. (2020). S2S an LSTM based Sequence-to-Sequence with attention model (Rush et al., 2015)
PGN an LSTM based Pointer-Generator Network with Copy Mechanism (See et al., 2017)
BUS Bottom-Up Summarization (Gehrmann et al., 2018) - a Pointer-Generator model with a data-efficient content selector to over-determine phrases in a source document that should be part of the summary.
BERTSum summarization with pretrained encoders (Liu and Lapata, 2019)
BART (Lewis et al., 2020)
A.1.2 XSum dataset
On the XSum dataset (Narayan et al., 2018) we use four different models. All model outputs for this dataset are taken from (Maynez et al., 2020)
PGN pointer-generator network from above (See et al., 2017)
TConvS2S Topic-Aware Convolution Sequence-to-Sequence (Narayan et al., 2018)
TranS2S A randomly initialized Transformer (Vaswani et al., 2017) encoder-decoder model fine-tuned on the XSum dataset
BERTS2S Transformer encoder-decoder model with parameter sharing (Rothe et al., 2020) where both encoder and decoder are initialized with the BERT-Base checkpoints (Devlin et al., 2019) and fine-tuned on XSum
In this work we compare the following five metrics.
BERTScore (Zhang et al., 2020): We report BERTScore Precision, Recall, and F1 between the model output and the reference summary. Our experiments show that recall and F1 do not correlate as well with the human judgement of factuality for BERTScore.
OpenIE : We use a simple baseline based on OpenIE (Banko et al., 2007) and Sentence-BERT (Reimers and Gurevych, 2019). We use OpenIE (Banko et al., 2007) to extract subject-relation-object triplets from the article, reference summary, and model generated summary. We consider binary relations only and thus use the first two arguments of the relation.<sup>5</sup> After replacing corefering entity mentions with the main mention of the cluster<sup>6</sup>, we use BERT base Sentence-BERT (Reimers and Gurevych, 2019) to obtain embeddings of each element of the subject-relation-object triplets extracted by OpenIE. Two relation triplets are considered to be equivalent if their embeddings have cosine similarity higher than a threshold for all three elements of the triplet (we use 0.6 as threshold after a grid search between 0.5 and 0.9 on data from our pilot study).
FEQA (Durmus et al., 2020): FEQA is a question generation and answering (QGA) factuality metric. We relied on the original implementation of the authors for this metric as well as their pretrained model weights. We used the full summary to generate questions and we answer them both using the summary and article text.
QAGS (Wang et al., 2020): QAGS is another QGA metric. The authors kindly provided outputs on the FRANK benchmark generating 10 questions for each summary.