SlowGuess/ACL_2020_OCR huggingface dataset:
We aim to understand to what extent the proposed QA-based metric and existing metrics capture faithfulness of a summary. Given pairs of documents and summary sentences without reference summaries, we measure correlations between human-annotated faithfulness scores (Section 2.3) and scores computed using each metric described below.
4.1 Automated Metrics for Faithfulness
Word overlap-based metrics. A straightforward metric for faithfulness is the word overlap between the summary sentence and the document. We compute ROUGE (R), BLEU (B), between the output sentence and each of the source sentences (i.e. taking the source sentence as the reference). We then take the average scores and maximum score across all the source sentences. Since according to our analysis taking the average score consistently has higher correlation, we report only the correlation for the average.
Embedding-based metrics. Word embeddings extend word overlap-based metrics beyond exact match. Recently, BERTScore (Zhang et al., 2019b) was proposed to compute the similarity between two sentences using contextual word embeddings from BERT. It has higher correlation
with human judgements on image captioning and machine translation than word overlap based metrics. We compute BERTScore (BERTSc) between each source sentence and the summary sentence.<sup>7</sup> To get the final score, we experiment with both the average and the maximum scores computed from each source sentence and the summary sentence. We report results using the maximum score since it has better performance.
Model-based metrics. In addition to QA, recent work has used relation extraction and textual entailment models for faithfulness evaluation (Falke et al., 2019a; Goodrich et al., 2019). For the relation extraction metric (RE), we compute the precision for the relation triplets extracted from the summary sentence and the source document using an off-the-shelf model (Angeli et al., 2015) from Stanford Open IE. For the textual entailment metric (ENT), we measure whether the summary sentence is entailed by the source using the pretrained ESIM model (Chen et al., 2017) from AllenNLP (Gardner et al., 2018).
Metric Comparison. We first compute scores for each metric on document and output sentence pairs on both CNN/DM and XSum datasets (748 and 286 pairs respectively). We then compute Pearson and Spearman correlation coefficients between scores given by each metric and human-annotated scores. Table 7 includes correlation coefficients for the examples from CNN/DM and XSum, respectively. We observe that for both CNN/DM and XSum, the score of QA-based evaluation has a higher correlation with faithfulness than other metrics. Although word-overlap based metrics are correlated with the faithfulness in more extractive settings (i.e. for CNN/DM), these metrics have no correlation with faithfulness in more abstractive settings (i.e. for XSum). We further notice that all the metrics have significantly lower correlation with human scores for XSum, suggesting that evaluating faithfulness is more difficult in highly abstractive settings; deeper understanding of the source and the summary sentence is necessary here.
Consistent with the findings of Falke et al. (2019b), the entailment metric does not have a significant correlation with faithfulness in most cases. These models fail to distinguish entailed (faithful)
<sup>6</sup>We report only BLUE-4 since it performed the best for CNN/DM and no variation of BLEU has significant correlation with faithfulness for XSum.
<sup>7</sup>https://github.com/Tiiger/bert_score.
Source Sentence	Output Sentence	Metric	Score
Health Inspectorate Wales said Wrex-ham Maelor Hospital staff were under “considerable pressure” for long periods as ambulances waited outside.	A hospital ward in Wrexham has been rated “inadequate” by inspectors after inspectors found patients at risk of harm.	Entailment	72.83%
The Black Poplar is one of the rarest native trees in the UK, with only 2,500 thought to be left.	Northern Ireland's first trees are among those recognised in the Welsh Architecture Trust's list of the year's best trees.	BertScore	83.06%
Table 6: Unfaithful examples missed by Entailment and BertScore. Score: Output score of the metrics; higher score indicates stronger entailment and similarity respectively.
Metric	CNN/DM	XSum
P	S	P	S
Word overlap-based
R-1	12.02**	15.86**	-2.57	0.07
R-2	13.25**	15.99**	-5.78	-8.47
R-L	12.58**	16.49**	-6.37	-9.68
B-4	12.09**	11.68**	-6.76	-10.02
Embedding-based
BERTSc	11.07*	10.70*	10.06	10.69
Model-based
RE	8.58*	5.52	1.62	2.32
ENT	2.80	3.65	-5.62	-3.85
FEQA	32.01**	28.23**	26.31**	21.34**
and non-entailed (unfaithful) summary sentences when both overlap largely with the source document, because models trained on current entailment datasets may rely on simple heuristics such as lexical overlap (McCoy et al., 2019). Similarly, BERTScore tends to give higher scores when there are overlapping concepts between the sentences even though the content is not the same. See Table 6 for examples.
Content selection and faithfulness. Current evaluation metrics for summarization produce a single measure of the overall quality of the summary. Typically, the output summary is compared against the reference summary in terms of n-gram overlap. These metrics mainly evaluate content selection, i.e. whether the content of the output is similar to the content of the reference. In contrast, to evaluate faithfulness, we compare the output summary against the source document. One natural question that follows is whether high content matching sufficient for faithfulness. We compute the correlation coefficients between human-annotated faithfulness scores and ROUGE scores computed from the reference and the output sentence. As shown in Table 8, while there is a weak
Table 7: Pearson (P) and Spearman (S) correlation between human-annotated faithfulness scores and the metric scores. *,** indicates  $p$ -values  $< 0.05, < 0.001$ , respectively. FEQA has the highest correlation with human scores for both CNN/DM and XSum.
Metric	CNN/DM	XSum
P	S	P	S
ROUGE-1	15.31**	14.92**	5.44	5.79
ROUGE-2	15.10**	16.39**	8.25	6.79
ROUGE-L	13.33**	13.35**	4.61	3.97
Table 8: Pearson (P) and Spearman (S) correlation between human-annotated faithfulness scores and ROUGE scores of content selection (computed between the reference and the output sentence). High content selection scores (typical ROUGE score for summarization) do not necessarily imply faithfulness of the summary.
correlation between ROUGE scores of content selection and faithfulness on CNN/DM, the correlation is significantly lower than ROUGE scores of faithfulness (i.e. computed between the source and the output sentence). For XSum, there is no significant correlation between the content selection metrics and faithfulness. We provide unfaithful examples with high content selection scores in Appendix D.3. This suggests that content selection and faithfulness should be measured separately as opposed to using a unified score.
Analysis and limitations of QA-based evaluation. Table 9 shows examples for a faithful and an unfaithful output sentence and the corresponding QA pairs. Note that the QA system is able to capture common errors such as conflicting information in the output sentence. To measure the reliability of FEQA, we further perform a manual error analysis using 100 randomly sampled QA pairs. We observe that around  $94\%$  of generated questions are mostly grammatical and correct given the mask. For  $78\%$  of the questions, the QA system has the correct behaviour: it answers the question correctly if the sentence is faithful to the article, otherwise it produces "unanswerable" or an incorrect answer. Majority of the errors of the QA system are because it either didn't detect unanswerable questions or produces "unanswerable" when there exists an answer  $(14\%)$ . More
Source	Output Sentence	Question	OA	SA
...However, Winger Ross Wallace (knee) and right-back Steven Reid (calf) could return for the Barclays pre-mier league contest...	Dean Marney and Steven Reid could return for the Barclays Premier League match.	Who and Steven Reid could return for the premier league match?	Dean Mar-ney	Ross Wal-lace
...Miss Bruck, 22, from maybe has not been seen since the early hours of October 26, 2014. She has not been seen for six months...	Miss Bruck, 22, from maybe has not been seen for six months.	How long has Miss Bruck, 22 from not been seen for?	six months	six months
Table 9: Examples detection results from FEQA. OA:Output Answer, SA:Source Answer. The output sentence in the first example is unfaithful, whereas the one for the second example is faithful. Bold text indicates the span that was masked to generate the question.
over, when the article is long, QA system tends to make more mistakes. Especially for more abstractive settings, F1-score penalizes the correct answers when the answer from the article does not exactly match with the gold answer (i.e. "Donald Trump" vs. "the President of the United States Donald Trump")  $(16\%)$ .
Problems in current neural generation models. Since the beginning of neural text generation, problems with repetition and generic responses have received lots of attention (Sordoni et al., 2015; Li et al., 2016; Holtzman et al., 2019). Recently, more work has focused on semantic errors in model outputs, such as adequacy in machine translation (Tu et al., 2017), faithfulness in summarization (Cao et al., 2018), and consistency in dialogue (Li et al., 2019). Our analysis on the abstractiveness-faithfulness tradeoff reveals additional limitation of current models, and suggests that we need new inductive bias on how to summarize beyond copying.
QA as a proxy. Question answering is a broad format that subsumes many tasks (Gardner et al., 2019). To the best of our knowledge, Mani et al. (1999) first use QA as an extrinsic evaluation for summarization: A good summary should answer key questions a reader might have about an article. Later, QA is incorporated in human evaluation where one person writes questions and another person answers them based on the summary (Clarke and Lapata, 2010; Liu and Lapata, 2019). The closest to our work are recent efforts in automating this protocol, including rule-based approaches (Chen et al., 2018) and cloze-test QA (Eyal et al., 2019; Scialom et al., 2019). Our work is the first to apply automated question generation. While we focus on faithfulness, our QA-based metric is applicable to semantic comparison
between any two pieces of text.
Automated evaluation for NLG. Automated NLG evaluation is challenging as it often requires deep understanding of the text. Although metrics based on word overlap with the reference text are commonly used, it is widely known that they do not correlate well with human judgments (Novikova et al., 2017; Liu et al., 2016). Recently, more work has focused on model-based evaluation using discriminators (Lowe et al., 2017; Hashimoto et al., 2019), entailment models (Falke et al., 2019a), information extraction (Wiseman et al., 2017; Goodrich et al., 2019), and question answering (Chen et al., 2018; Eyal et al., 2019).
We investigate the faithfulness problem in neural abstractive summarization and propose a QA-based metric for evaluating summary faithfulness. We show that current models suffer from an inherent trade-off between abstractiveness and faithfulness. They are good at copying important source content, but tend to concatenate unrelated spans and hallucinate details when generating more abstractive sentences. A new inductive bias or additional supervision is needed for learning reliable models. While our QA-based metric correlates better with human judgment and is useful for model development, it is limited by the quality of the QA model. The final evaluation should still rely on human annotation or human-in-the-loop methods (Chaganty et al., 2018).
We would like to thank Faisal Ladhak, the Lex and Comprehend groups at Amazon Web Services AI, and the anonymous reviewers for their feedback on this work.
R. Alec, W. Jeff, C. Rewon, L. David, A. Dario, and S. Ilya. 2019. Language models are unsupervised multitask learners.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 344-354, Beijing, China. Association for Computational Linguistics.
Z. Cao, F. Wei, W. Li, and S. Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Association for the Advancement of Artificial Intelligence (AAAI).
A. Chaganty, S. Mussmann, and P. Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In Association for Computational Linguistics (ACL).
Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University.
P. Chen, F. Wu, T. Wang, and W. Ding. 2018. A semantic QA-based approach for text summarization evaluation. In Association for the Advancement of Artificial Intelligence (AAAI).
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657-1668, Vancouver, Canada. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In ACL.
J. Clarke and M. Lapata. 2010. Discourse constraints for document compression. Computational Linguistics, 36.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. *ArXiv*, abs/1809.02922.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
M. Eyal, T. Baumel, and M. Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL).
T. Falke, L. F. R. Ribeiro, P. A. Utama, I. Dagan, and I. Gurevych. 2019a. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Association for Computational Linguistics (ACL).
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019b. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL '05, pages 363-370, Stroudsburg, PA, USA. Association for Computational Linguistics.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational AI. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 2-7, Melbourne, Australia. Association for Computational Linguistics.
M. Gardner, J. Berant, H. Hajishirzi, A. Talmor, and S. Min. 2019. Question answering is a format; when is it useful? arXiv preprint arXiv:1909.11291.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
B. Goodrich, V. Rao, P. J. Liu, and M. Saleh. 2019. Assessing the factual accuracy of generated text. In International Conference on Knowledge Discovery and Data Mining (KDD).
M. Grusky, M. Naaman, , and Y. Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In North American Association for Computational Linguistics (NAACL).
T. Hashimoto, H. Zhang, and P. Liang. 2019. Unifying human and statistical evaluation for natural language generation. In North American Association for Computational Linguistics (NAACL).
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA. MIT Press.
A. Holtzman, J. Buys, M. Forbes, and Y. Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676-2686, Melbourne, Australia. Association for Computational Linguistics.
K. Knight and D. Marcu. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence, 139:91-107.
W. Krysciński, N. S. Keskar, B. McCann, C. Xiong, and R. Socher. 2019. Neural text summarization: A critical evaluation. In Empirical Methods in Natural Language Processing (EMNLP).
L. Lebanoff, J. Muchovej, F. Dernoncourt, D. S. Kim, S. Kim, W. Chang, and F. Liu. 2019. Analyzing sentence fusion in abstractive summarization. arXiv preprint arXiv:1910.00203.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv, abs/1910.13461.
J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110-119.
M. Li, S. Roller, I. Kulikov, S. Welleck, Y. Boureau, K. Cho, and J. Weston. 2019. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv:1911.03860.
C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Charlin, and J. Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Empirical Methods in Natural Language Processing (EMNLP).
Y. Liu and M. Lapata. 2019. Text summarization with pretrained encoders. In Empirical Methods in Natural Language Processing (EMNLP).
R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard-Gontier, Y. Bengio, and J. Pineau. 2017. Towards an automatic tuning test: Learning to evaluate dialogue responses. In Association for Computational Linguistics (ACL).
I. Mani, G. Klein, L. Hirschman, T. Firmin, D. House, and B. Sundheim. 1999. The TIPSTER SUMMAC text summarization evaluation. In European Association for Computational Linguistics (EACL).
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
R. T. McCoy, E. Pavlick, and T. Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Guilçehre, and Bing Xiang. 2016b. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
J. Novikova, O. Dusek, A. C. Curry, and V. Rieser. 2017. Why we need new evaluation metrics for NLG. In Empirical Methods in Natural Language Processing (EMNLP).
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
T. Scialom, S. Lamprier, B. Piwowarski, and J. Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Empirical Methods in Natural Language Processing (EMNLP).
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL.
A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In North American Association for Computational Linguistics (NAACL).
Z. Tu, Y. Liu, L. Shang, X. Liu, and H. Li. 2017. Neural machine translation with reconstruction. In Association for the Advancement of Artificial Intelligence (AAAI).
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76-85, Berlin, Germany. Association for Computational Linguistics.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700. Curran Associates, Inc.
S. Wiseman, S. M. Shieber, and A. M. Rush. 2017. Challenges in data-to-document generation. In Empirical Methods in Natural Language Processing (EMNLP).
F. Zhang, J. Yao, and R. Yan1. 2018. On the abstractiveness of neural document summarization. In Empirical Methods in Natural Language Processing (EMNLP).
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. 2019a. BERTSCORE: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675.
A Summarization Datasets
All of our experiments are run on the CNN/DM and XSum datasets. We show basic statistics of the two datasets in Table 10.
CNN/DM	XSum
# Training Documents	287,227	204,045
# Validation Documents	13,368	11,332
# Test Documents	11,490	11,334
Document: avg # of tokens	781.00	431.07
Document: avg # of sents.	40.00	33.00
Summary: avg # tokens	56.00	23.26
Summary: avg # of sents.	3.75	1.00
Table 10: Statistics of CNN/DM and XSum datasets.
B Summarization Models
The characteristics of each model used in our experiments are detailed below.