SlowGuess/ACL_2020_OCR huggingface dataset:
Table 2: Intrinsic vs. Extrinsic Hallucinations. The numbers in "Hallucinated" columns show the percentage of summaries where at least one word was annotated by all three annotators as an intrinsic (I) or extrinsic (E) hallucination. When a summary is not marked with any hallucination, it is "faithful" (100 - I $\cup$ E), column "Faith." The final "+"Fact." column shows the total percentage of faithful and/or factual summaries, which includes all faithful summaries plus the percentage of non-faithful summaries annotated by all three annotators as factual. Higher numbers for faithful/factual and lower numbers for hallucinations are boldfaced.
firmed that the BBC gold summaries often have extrinsic hallucinations due to the dataset artifact that gold summaries are introductory sentences prefacing each article. It was not surprising that most models also had significant extrinsic hallucinations.
Intrinsic Hallucination is Also Common in Abstractive Summaries. Gold summaries can also display intrinsic hallucinations. For example, a news article could describe an event related to "Barack Obama" and "the office of the President of the United States" without inferring that "Obama is the President of the United States." A journalist with the knowledge of the event in the article could write a summary stating "President Obama."
However, the percentage of system summaries with intrinsic hallucination was much higher than in gold summaries (7.4% vs others). This phenomenon particularly revealed the models' tendency to misrepresent information in the document due to the lack of document-level understanding and inference. The copy mechanism in PTGEN is good at copying from the source (showing the least percentage of extrinsic hallucination of 63.3%), but the mechanism lacks the inference capability and is prone to generate a summary that is not supported by the document (19.9% intrinsic hallucination). TRANs2S showed similar performance to PTGEN and ranked second worst. The BERTS2S showed the least number of intrinsic hallucination (16.9%) among all four abstractive systems.
Pretraining Improves Faithfulness. Hallucinations do not result from the artifacts in the training data only, but also due to model shortcomings. The PTGEN model with the copy mechanism (Gu et al., 2016; See et al., 2017) had the lowest extrinsic
hallucination (63.3%), but BERTS2S reported the highest number of faithful summaries. It appears that BERTS2S is overall most conservative among all four abstractive systems while getting closer to reference summaries in terms of ROUGE. The pretraining prepares BertS2S to be more aware of the domain of the document and less prone to language model vulnerabilities. Consequently, BertS2S is more confident in predicting tokens from the document than TranS2S, hence, improving faithfulness.
5.3 Assessment of Factual Hallucinations.
Hallucinations are not necessarily erroneous. In our second human assessment, we measured to what extent this is the case. Our annotators were presented a single-sentence summary with hallucinations and were asked to assess whether it is true or false. To better explain the context of the summary, annotators were made available the source document as well as the external resources such as Wikipedia or Google Search. The source document can be particularly important for generic summaries to better understand context. External resources assisted the evaluators to validate grounded facts in public knowledge bases.
Annotators were expected to validate the summary by looking for supporting evidence for the information found on the summary. If information in the summary contradicts the document, then the summary is not factual. If supporting evidence is found for all the information, then the summary is factual. The document is not useful when the summary has information that is neither supported nor contradicted in the article. For example, the summary in Figure 2 mentions "Conservative MP Zac Goldwin" which can not be verified from the article in Figure 1. Here, annotators could use Wikipedia or Google Search to check that there had not been a Conservative MP named Zac Goldwin who tried to change their party and become a Labour's candidate in the 2016 London mayoral election.
We dropped the human authored gold summaries from this evaluation; they were presumably factual. We also dropped the abstracts that were faithful to their input documents from the previous study. Finally, there were 1869 document-summary pairs where the summaries were marked with at least one intrinsic or extrinsic hallucination. We elicited judgments from three different annotators for each of them. Results from this assessment are also presented in Table 2 (see the column labelled “+Fact.”) along with the hallucination assessment.
Pretraining Helps Generating Factual Summaries. In total,  $34.7\%$  of the BERTS2S abstracts were faithful  $(26.9\%)$  and/or factual  $(+7.8\%)$ . This is  $7.4\%$  absolute better than the next-best model (PTGEN). The number of unfaithful yet factual summaries for BERTS2S,  $7.8\%$ , was also the highest. In fact, for extrinsic hallucinations, even though PTGEN hallucinates less than BERTS2S  $(63.3\%$  vs.  $64.1\%)$ ,  $6.6\%$  of BERTS2S hallucinations were factual, compared to  $2.2\%$  of PTGEN. Thus, if we consider factual hallucinations to be valid, this means that even for extrinsic cases, BERTS2S hallucinates the least.
The superior performance of BERTS2S is most likely due to its exposure to vast amount of text through pretraining, allowing it to integrate background knowledge with generation. Even so, over  $90\%$  of BERTS2S hallucinations are erroneous.
Finally, we carried out pairwise comparisons between all models (using a one-way ANOVA with post-hoc Tukey HSD tests;  $p < 0.01$ ). For intrinsic hallucinations (the second column in Table 2), GOLD is significantly different from all other systems. For extrinsic hallucinations (the third column in Table 2), there were significant differences between PTGEN and TCONVS2S, PTGEN and GOLD, and BERTS2S and GOLD. For factuality, the differences between PTGEN, TCONVS2S, and TRANs2S were insignificant.
5.4 Automatic Measures for Hallucinations
Summaries are a proxy for their source documents under the assumption that they highlight the most important content. With this assumption, we further studied the extent to which the hallucinated content can be measured by semantic inference related measures, such as textual entailment and question answering.
Textual Entailment. We trained an entailment classifier by finetuning a BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018). We calculated the entailment probability score between the document and its abstractive summaries. Note that this entailment classifier is not optimal for the BBC article-summary pairs; the Multi-NLI dataset contains sentence-sentence pairs.
Ideally a summary should entail the document or perhaps be neutral to the document, but never contradict the document. As can be seen in Table 3, the BERTS2S abstracts showed the least number of
Models	Textual Entailment	QA
entail.	neut.	cont.
PTGEN	38.4	34.4	27.2	20.2
TCONVS2S	29.6	37.4	33.0	19.9
TRANS2S	34.6	39.8	25.6	22.4
BERTS2S	41.8	37.8	20.4	23.0
GOLD	32.8	47.2	20.0	19.3
Table 3: Textual entailment and question answering (QA) based measures for summary evaluation. For entailment, we show the percentage of times a summary entails (entail.) the document, is neutral (neut.) to the document and contradicts (cont.) the document. For QA, we report the percentage of questions that were correctly answered by a system. The highest numbers for entail., neut. and QA, and the lowest number for cont. are boldfaced.
contradictions compared to other system-generated abstracts and was at par with the GOLD summaries. Similar to the performance on extrinsic hallucination in Table 2, the TCONVS2S abstracts also displayed the highest number of contradictions. Interestingly, the GOLD summaries are more neutral to their documents, whereas the BERTS2S summaries are more entailed by their documents. This is probably due to the nature of the data and that journalists tend to add color and have a high number of extrinsic (but valid) hallucinations.
Question Answering. QA frameworks have been used to assess or promote summary informativeness (Narayan et al., 2018b; Arumae and Liu, 2019). We adapted the QA framework to assess hallucination in model generated summaries; a faithful model will generate a summary that only has information that is supported by its document. Under this assumption, any question answerable by the summary should also be answerable by the source.
Given an abstractive summary, we used the round-trip consistency method of Alberti et al. (2019), which combines question generation and answer extraction models to generate synthetic question-answer pairs. For the 500 document-summary pairs, we generated 731, 708, 720, 725 and 820 question-answer pairs for PTGEN, TCONVS2S, TRANs2S, BERTS2S and GOLD, respectively. Finally, we used a machine reading comprehension model to answer these questions using the document as context. As in Alberti et al. (2019), we trained all models: question generation, answer extraction and reading comprehension models; using a BERT-Base pretrained model (Devlin et al., 2019) finetuned on the Natural Questions dataset (Kwiatkowski et al., 2019).
Similar to textual entailment results, the
See Appendix for full results.
PTGEN	Leeds United fought back from 2-0 down to beat Huddersfield town in the first round of the EFL cup. (Q: What team did Leeds United beat in the first round of the EFL cup?, A: Huddersfield town)
TCONVS2S	A coal mine in South Yorkshire has collapsed as a result of the loss of a coal mine. (Q: What type of mine has collapsed?, A: Coal)
TRANS2S	Star Wars actor James Davis said he was “locked in a caravan” and had his caravan stolen during a break-in. (Q: Who said he was locked in a caravan?, A: Davis)
Figure 3: Sample of question-answer pairs generated from hallucinated summaries that are correctly answered by their source articles. Highlighted spans in the summaries are marked as extrinsic or intrinsic hallucinations by our annotators.
Metric	Faithful	Factual
ROUGE-1	0.197	0.125
ROUGE-2	0.162	0.095
ROUGE-L	0.162	0.113
BERTScore	0.190	0.116
QA	0.044	0.027
Entailment	0.431	0.264
BERTS2S abstracts were the most faithful to their source documents in terms of question answering. The GOLD abstracts were the least accurate due to a high number of extrinsic hallucination in them.
Spearman's Correlation. We estimate Spearman's correlation coefficients of different metrics with the faithful and factual human scores (see Table 4). We found that the textual entailment scores are best correlated with both faithful (moderate,  $0.40 \leq |r_s| \leq 0.59$ ) and factual (weak,  $0.20 \leq |r_s| \leq 0.39$ ) human scores. Comparatively, ROUGE-based metrics and BERTScore have very weak correlation, our findings are consistent with the recent studies (Goodrich et al., 2019; Kryscinski et al., 2019a; Wang et al., 2020). Surprisingly, the question answering scores showed a very weak correlation ( $0.0 \leq |r_s| \leq 0.19$ ) with faithful and factual human scores. We hypothesize that this is due to a compounding of errors where (i) the question generator is used to generate questions from the systems' generated abstracts, instead of human-written text on which they were trained, (ii) the question generator is susceptible to generate questions with hallucinated content when fed in with hallucinated summaries, and (iii) our assumption that a summary is faithful if the answers from the source and the summary match, is rather poor for extreme summarization. We demonstrate these issues in Figure 3. Irrespective of questions with hallucinated content, our reading comprehension
Table 4: Spearman's correlation coefficient  $\left( {\left| {r}_{s}\right| }\right)$  of different metrics with faithful and factual annotations.
Models	R1	R2	RL	Faith.	+Fact.
BERTS2S	38.42	16.96	31.27	26.9	34.7
ENTAIL	35.93	14.02	28.87	31.5	38.6
→FAITH	37.31	15.21	30.12	31.7	38.8
Table 5: ROUGE and faithfulness/factuality scores for BERTS2S plus systems that use textual entailment as a criteria or fine-tuned on faithful annotations.
model can fortuitously answer them correctly from their source articles. Better ways of generating questions (Narayan et al., 2020) and measuring factual consistency may alleviate some of these issues (Wang et al., 2020).
5.5 Model Selection with Entailment
Our study suggests that entailment could be used as an automatic measure for faithfulness. However, we should point out that this measure is referenceless. Thus, it can easily be gamed, i.e., the first sentence of any source document is always entailed by the whole document. Because of this, entailment-based measures for evaluation need to be coupled with reference-based measures like ROUGE.
However, one major advantage of the measure being reference-less is that we can use it as a model selection objective or during decoding. We tested the former. Specifically, we used the probability that a summary is entailed by a document as a selection criteria to select a summary between four candidates generated by systems evaluated: PTGEN, TCONVS2S, TRANS2S, and BERTS2S. Results are shown in the ENTAIL row of Table 5. We can see that indeed this is a strong metric to optimize towards if we want faithful summaries - almost  $5\%$  absolute better. There is a trade-off in terms of ROUGE, but this model must select amongst 4 systems, 3 of which have significantly lower ROUGE than the best model.
A further experiment is to train a model explicitly to predict faithfulness. In order to do this, we further fine-tuned the entailment model using the 'faithful' annotations generated during our evaluation. For all summary-document pairs marked as 'faithful', we set the associated class to 'entailment', otherwise we set it to 'neutral'. This allowed for us to also fine-tune the last classification layers taking advantage of the correlation between 'entailment' and 'faithfulness'. Results using 5-fold cross validation are shown in the ENTAIL  $\rightarrow$  FAITH row of Table 5. We see here that indeed this does improve the ability to select faithful summaries from a set of candidates, though slightly. We would expect to see larger gains with more training data. However, this model is significantly better than ENTAIL
on ROUGE-based metrics and seems like a good balance between ROUGE and better faithfulness.
Following the Document Understanding Conference (DUC; Dang, 2005), a majority of work has focused on evaluating the content and the linguistic quality of summaries (Nenkova, 2005). Most popular among them is the automatic metric ROUGE (Lin and Hovy, 2003) that measures the unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a proxy for assessing informativeness and the longest common subsequence (ROUGE-L), for fluency. ROUGE, however, can be misleading when used as the only means to assess the informativeness of summaries (Schluter, 2017). Hence, the ROUGE score is often complemented with subjective human assessment of summaries. More objective measures have been proposed to improve agreement among human annotators. Pyramid method (Nenkova and Passonneau, 2004) requires summaries to be annotated by experts for salient information. Narayan et al. (2018a,b) used a question-answering based approach where a summary is used as context to answer questions which were written based on its reference summary. Hardy et al. (2019) proposed a reference-less approach where a summary is assessed against the source document, highlighted with its pertinent content.
There has not been much work on evaluating faithfulness and truthfulness of abstractive summaries. The automatic evaluation such as ROUGE and the human evaluation of saliency and linguistic quality of summaries are not sufficient due to the complex nature of the task. Recently, Chen and Bansal (2018) asked human annotators to assess the summary relevance measuring both the saliency and the presence of contradictory/unrelated information. Dhingra et al. (2019) proposed a new automatic metric, PARENT, for data-to-text generation (Lebret et al., 2016; Wiseman et al., 2017) which aligns  $n$ -grams from the reference and generated texts to the source table to measure the accuracy of  $n$ -grams that are entailed from the source table. Goodrich et al. (2019) proposed a model-based automatic metric to assess the faithfulness of Wikipedia summaries; they trained an end-to-end model to extract a complete set of OpenIE-style (Banko et al., 2007) facts from both the source text and the generated summary. The summary is faithful if it is precise in generating facts from the source text. In our experiments with OpenIE-based measures, we found that they are not suited
for evaluating extreme summarization models; all models perform poorly on these metrics without any significant differences. Like ours, few recent works (some in parallel) have explored natural language inference and question answering models to detect factual consistency in generated text (Welleck et al., 2019; Falke et al., 2019; Kryscinski et al., 2019b; Wang et al., 2020). In line with our findings, Falke et al. (2019) observed that the BERT-based NLI models substantially improved summaries reranking in terms of their correctness. Kryscinski et al. (2019b) proposed an NLI-based fact checking model that is trained on a dataset tailored for detecting factual inconsistencies in generated text. Wang et al. (2020) proposed a question answering and generation based automatic evaluation protocol that is designed to identify factual inconsistencies in a generated summary. Future work will likely investigate better ways of generating questions and measuring factual consistency to address poor correlation with faithfulness and factuality annotations.
Finally, others have used reinforcement learning to improve informativeness and reduce contradictory information in abstractive summaries, e.g., Pasunuru and Bansal (2018) used a textual entailment-based reward and Arumae and Liu (2019), a question-answering based reward. However, these approaches don't evaluate if these rewards improve faithfulness of summaries.
We conducted a large-scale study of hallucinations in abstractive document summarization. We found that (i) tackling hallucination is a critical challenge for abstractive summarization, perhaps the most critical, (ii) NLU-driven pretraining in neural text generators is key to generate informative, coherent, faithful and factual abstracts, but it is still far from solving the problem; and (iii) measures such as ROUGE or BERTScore will not be sufficient when studying the problem; semantic inference-based automatic measures are better representations of true summarization quality.
We thank Ratish Puduppully, Yova Kementchedjheva, Ankur Parikh, Peter Liu, Slav Petrov, the reviewers and the action editor for invaluable feedback. The hard work of Muqthar Mohammad, Mohd Majeed and Ashwin Kakarla made our human annotation possible.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168-6173, Florence, Italy.
Kristjan Arumae and Fei Liu. 2019. Guiding extractive summarization with question-answering rewards. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2566-2577, Minneapolis, Minnesota.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, San Diego, CA, USA.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pages 2670-2676, Hyderabad, India.
Ann L. Brown and Jeanne D. Day. 1983. Macrorules for summarizing texts: The development of expertise. Journal of Verbal Learning and Verbal Behaviour, 22(1):1-14.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 675-686, Melbourne, Australia.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy.
Hoa Trang Dang. 2005. Overview of DUC 2005. In Proceedings of the Document Understanding Conference, pages 1-12.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186, Minneapolis, Minnesota.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884–4895, Florence, Italy.
Teun A. van Dijk and Walter Kintsch. 1978. Cognitive psychology and discourse: Recalling and summarizing stories. In Wolfgang U. Dressler, editor, Current Trends in Textlinguistics, pages 61-80.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32, pages 13063-13075. Curran Associates, Inc.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1243-1252, Sydney, NSW, Australia.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium.
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 166-175, New York, NY, USA.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1631-1640, Berlin, Germany.
Hardy, Shashi Narayan, and Andreas Vlachos. 2019. HighRES: Highlight-based reference-less evaluation of summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3381-3392, Florence, Italy.
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of the 8th International Conference on Learning Representations, Virtual Conference, Formerly Addis Ababa Ethiopia.