SlowGuess/ACL_2020_OCR huggingface dataset:
Joshua Maynez* Shashi Narayan* Bernd Bohnet Ryan McDonald Google Research
{joshuahm,shashinarayan,bohnetbd,ryanmcd}@google.com
It is well known that the standard likelihood training and approximate decoding objectives in neural text generation models lead to less human-like responses for open-ended tasks such as language modeling and story generation. In this paper we have analyzed limitations of these models for abstractive document summarization and found that these models are highly prone to hallucinate content that is unfaithful to the input document. We conducted a large scale human evaluation of several neural abstractive summarization systems to better understand the types of hallucinations they produce. Our human annotators found substantial amounts of hallucinated content in all model generated summaries. However, our analysis does show that pretrained models are better summarizers not only in terms of raw metrics, i.e., ROUGE, but also in generating faithful and factual summaries as evaluated by humans. Furthermore, we show that textual entailment measures better correlate with faithfulness than standard metrics, potentially leading the way to automatic evaluation metrics as well as training and decoding criteria.<sup>1</sup>
Current state of the art conditional text generation models accomplish a high level of fluency and coherence, mostly thanks to advances in sequence-to-sequence architectures with attention and copy (Sutskever et al., 2014; Bahdanau et al., 2015; Gu et al., 2016), fully attention-based Transformer architectures (Vaswani et al., 2017; Dai et al., 2019) and more recently pretrained language modeling for natural language understanding (Devlin et al., 2019; Radford et al., 2018; Yang et al., 2019; Liu et al., 2019). There has been a growing interest in
understanding how maximum likelihood training and approximate beam-search decoding in these models lead to less human-like text in open-ended text generation such as language modeling and story generation (Holtzman et al., 2020; Welleck et al., 2020; See et al., 2019). In this paper we investigate how these models are prone to generate hallucinated text in conditional text generation, specifically, extreme abstractive document summarization (Narayan et al., 2018a).
Document summarization — the task of producing a shorter version of a document while preserving its information content (Mani, 2001; Nenkova and McKeown, 2011) — requires models to generate text that is not only human-like but also faithful and/or factual given the document. The example in Figure 1 illustrates that the faithfulness and factuality are yet to be conquered by conditional text generators. The article describes an event of "Conservative MP Zac Smith winning the primary for 2016 London mayoral election", but summaries often forge entities (e.g., "Nigel Goldsmith" or "Zac Goldwin") or information (e.g., "UKIP leader Nigel Goldsmith", "Nigel Goldsmith winning the mayoral election", "Sadiq Khan being the former London mayor" or "Zac Goldwin being the Labour's candidate") that are not supported by the document or are factually wrong. Interestingly, all summaries are topical and fluent, and perform well in terms of ROUGE scores (Lin and Hovy, 2003).
We conducted a large-scale human evaluation of hallucinated content in systems that use Recurrent Neural Network (RNN) (See et al., 2017), Convolutional Neural Network (CNN) (Narayan et al., 2018a), and Transformers (Radford et al., 2019; Rothe et al., 2020), as well as human written summaries for the recently introduced eXtreme Summarization task (XSUM, Narayan et al., 2018a). We seek to answer the following questions: (i) How frequently do abstractive summarizers hallucinate content?; (ii) Do models hal
* The first two authors contributed equally.  
<sup>1</sup>Our human annotated summaries for faithfulness and factuality will be released at https://github.com/google-research-datasets/xsum_hallucination_annotations.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
GOLD	Zac Goldsmith will contest the 2016 London mayoral election for the Conservatives, it has been announced.
DOCUMENT: The Richmond Park and North Kingston MP said he was "honoured" after winning 70% of the 9,227 votes cast using an online primary system.
He beat London Assembly Member Andrew Boff, MEP Syed Kamall and London's deputy mayor for crime and policing Stephen Greenhalgh.
Mr Goldsmith's main rival is likely to be Labour's Sadiq Khan. (2 sentences with 59 words are abbreviated here.)
Mr Goldsmith, who was the favourite for the Tory nomination, ballotd his constituents earlier this year to seek permission to stand.
At the very point of his entry into the race for London mayor, Zac Goldsmith's decision revealed two big characteristics.
(5 sentences with 108 words are abbreviated here.)
Mr Goldsmith - who first entered Parliament in 2010 - told the BBC's Daily Politics that he hoped his environmental record would appeal to Green and Lib Dem voters and he also hoped to "reach out" to UKIP supporters frustrated with politics as usual and the UK's relationship with the EU.
Zac Goldsmith Born in 1975, educated at Eton and the Cambridge Centre for Sixth-form Studies (5 sentences with 76 words are abbreviated here.)
Mr Goldsmith, who has confirmed he would stand down from Parliament if he became mayor, triggering a by-election, said he wanted to build on current mayor Boris Johnson's achievements. (3 sentences with 117 words are abbreviated here.)
Both Mr Khan and Mr Goldsmith oppose a new runway at Heathrow airport, a fact described by the British Chambers of Commerce as "depressing". (1 sentences with 31 words is abbreviated here.)
Current mayor Boris Johnson will step down next year after two terms in office. He is also currently the MP for Uxbridge and South Ruislip, having been returned to Parliament in May.
Some Conservatives have called for an inquiry into the mayoral election process after only 9,227 people voted - compared with a 87,884 turnout for the Labour contest. (4 sentences with 121 words are abbreviated here.)
PTGEN	UKIP leader Nigel Goldsmith has been elected as the new mayor of London to elect a new Conservative MP.	[45.7, 6.1, 28.6]
TCONVS2S	Former London mayoral candidate Zac Goldsmith has been chosen to stand in the London mayoral election.	[50.0, 26.7, 37.5]
TRANS2S	Former London mayor Sadiq Khan has been chosen as the candidate to be the next mayor of London.	[35.3, 12.5, 23.5]
GPT-TUNED	Conservative MP Zac Goldwin's bid to become Labour's candidate in the 2016 London mayoral election.	[42.4, 25.8, 36.4]
BERTS2S	Zac Goldsmith has been chosen to contest the London mayoral election.	[66.7, 40.0, 51.9]
Figure 1: Hallucinations in extreme document summarization: the abbreviated article, its gold summary and the abstractive model generated summaries (PTGEN, See et al. 2017; TCONVS2S, Narayan et al. 2018a; and, GPT-TUNED, TRANs2S and BERTS2S, Rothe et al. 2020) for a news article from the extreme summarization dataset (Narayan et al., 2018a). The dataset and the abstractive models are described in Section 3 and 4. We also present the [ROUGE-1, ROUGE-2, ROUGE-L]  $\mathrm{F}_1$  scores relative to the reference gold summary. Words in red correspond to hallucinated information whilst words in blue correspond to faithful information.
lucinate by manipulating the information present in the input document (intrinsic hallucinations) or by adding information not directly inferable from the input document (extrinsic hallucinations)?; (iii) How much hallucinated content is factual, even when unfaithful?; and (iv) Are there automatic means of measuring these hallucinations?
Our main conclusions are as follows: First, intrinsic and extrinsic hallucinations happen frequently – in more than  $70\%$  of single-sentence summaries. Second, the majority of hallucinations are extrinsic, which potentially could be valid abstractions that use background knowledge. However, our study found that over  $90\%$  of extrinsic hallucinations were erroneous. Thus, hallucinations happen in most summaries and the majority of these are neither faithful nor factual. Third, models initialized with pretrained parameters perform best both on automatic metrics and human judgments of faithfulness/factuality. Furthermore, they have the highest percentage of extrinsic hallucinations that are factual. This suggests that while some studies
argue that large-scale pretrained models are merely better at learning data-specific regularities (Niven and Kao, 2019), at least on in-domain summarization the gains in automatic metrics are realized in observable differences by humans. Fourth, ROUGE (Lin and Hovy, 2003) and BERTScore (Zhang et al., 2020) correlates less with faithfulness/factuality than metrics derived from automatic semantic inference systems, specifically the degree to which a summary is entailed by the source document. This presents an opportunity for improved automatic evaluation measures as well as model training and decoding objectives. We show preliminary experiments in this direction.
2 Hallucinations in Summarization
Open-ended generation — the task of generating text that forms a natural continuation from the input text — requires the model to hallucinate text; hence the focus has been to ensure that the model learns to generate text that is more human-like (i.e., less repetitive or dull with more content-related words)
(Holtzman et al., 2020; Welleck et al., 2020; See et al., 2019). In contrast, tasks such as document summarization (Nenkova and McKeown, 2011; See et al., 2017; Paulus et al., 2018) and data-to-text generation (Lebret et al., 2016; Wiseman et al., 2017) which are not open-ended, require models to be factual and/or faithful to the source text.
Despite recent improvements in conditional text generation, most summarization systems are trained to maximize the log-likelihood of the reference summary at the word-level, which does not necessarily reward models for being faithful. Moreover, models are usually agnostic to the noises or artifacts of the training data, such as reference divergence, making them vulnerable to hallucinations (Kryscinski et al., 2019a; Wiseman et al., 2017; Dhingra et al., 2019). Thus, models can generate texts that are not consistent with the input, yet would likely have reasonable model log-likelihood.
2.1 Intrinsic and Extrinsic Hallucinations
Given a document  $D$  and its abstractive summary  $S$ , we try to identify all hallucinations in  $S$  with respect to the content of  $D$ , regardless of the quality of the summary. In this work, we define a summary as being hallucinated if it has a span(s)  $w_{i}\ldots w_{i + j}$ ,  $j \geq i$ , that is not supported by the input document. To distinguish hallucinations further in the context of a document and a summary, we categorize hallucinations by the information source as intrinsic and extrinsic hallucinations. Note, paraphrases or any information that can be inferred from the document are not categorized as hallucinations.
Intrinsic hallucinations are consequences of synthesizing content using the information present in the input document. For example, in Figure 1, "Former London mayoral candidate" in the TCONVS2S abstract and "Former London mayor" in the TRANS2S abstract are hallucinations of intrinsic nature; both use terms or concepts from the document but misrepresent information from the document, making them unfaithful to the document. The article does not confirm if "Zac Goldsmith" was a "Former London mayoral candidate" or if "Sadiq Khan" was a "Former London mayor". One may suspect that a model with poor input document representation will fail to do document level inference, often required for abstraction, and will be vulnerable to such errors.
Extrinsic hallucinations are model generations that ignore the source material altogether. For example, in Figure 1, "Nigel" in the PTGEN abstract and "2016" in both GOLD and GPT-TUNED are
extrinsic hallucinations; these terms are not introduced in the document. A model with a poorly-informed decoder and that is agnostic to the divergence issue between the source and target texts (Wiseman et al., 2017; Dhingra et al., 2019), will function more as an open-ended language model and will be prone to extrinsic hallucinations.
2.2 Factual Hallucinations in Summarization
A summary  $S$  of a document  $D$  contains a factual hallucination if it contains information not found in  $D$  that is factually correct. Factual hallucinations may be composed of intrinsic hallucinations or extrinsic hallucinations.
By definition, abstractive summaries are written to preserve the salient information in the input document, but they are expressed in the words of the summary author as opposed to the input document author (Nenkova and McKeown, 2011). As such, it is natural to construct summaries that integrate with the author's background knowledge (van Dijk and Kintsch, 1978; Brown and Day, 1983). Such knowledge integration can also be desirable in real world applications. For instance, an engaging sports report will reflect an understanding of the game to provide color and context. Another example is audience-targeted summarization where a good summary will reflect understanding of both the article domain and the desired audience. Nonetheless, there is no consensus in the research community if the summary should be faithful (without any hallucinations) to the input document or if there is tolerance for factual hallucinations.
Recent deep learning approaches to abstractive summarization naturally learn to integrate knowledge from the training data while generating an abstractive summary for a document (See et al., 2017; Gehrmann et al., 2018). More advanced pretrained text generators (Radford et al., 2018, 2019; Dong et al., 2019; Song et al., 2019; Khandelwal et al., 2019; Rothe et al., 2020) are even better at capturing world knowledge as they are informed by a vast amount of background text. This can be observed in the example shown in Figure 1 as the input document does not mention that the discussed "London mayoral election" is from "2016"; but the abstract generated by the pretrained text generator GPT-TUNED correctly predicts this information similar to the human-authored abstract.
2Despite the correct extrinsic hallucination ("2016"), the GPT-TUNED abstract overall is still not factual due to the incorrect extrinsic hallucination in "Conservative MP Zac Goldwin." There is no Conservative MP named Zac Goldwin.
In this paper we stand in favour of the assertion that abstractive systems may integrate with the background knowledge to generate rich and meaningful summaries. More concretely, "hallucinations in summarization are acceptable if they lead to better summaries that are factual with respect to the document and the associated background knowledge." This assumption also allows us to assess the capability of recent neural models to integrate with the background knowledge to generate factual abstracts (see Section 5.3).
3 Extreme Document Summarization
We focus on the recently introduced extreme summarization dataset (XSUM, Narayan et al., 2018a)<sup>3</sup> which comprises 226,711 British Broadcasting Corporation (BBC) articles paired with their single-sentence summaries, provided by the journalists writing the articles. The dataset is split into three subsets: training (90%, 204,045), validation (5%, 11,332), and test (5%, 11,334) sets. All models in §4 trained to generate abstractive summaries are trained and evaluated using this standard split, provided by the authors.
We choose to focus our study on extreme summarization for the following reasons: First, this task aims to create a single-sentence summary of a news article; these shorter summaries are relatively easier to annotate and analyze than longer summaries such as story highlights from the CNN/Dailymail dataset (Hermann et al., 2015) or abstracts from the NY Times (Sandhaus, 2008) or the WikiSum (Liu et al., 2018) dataset. Secondly, the gold summary in the extreme summarization dataset is an introductory sentenceprefacing each article. By virtue of this property, the extreme summarization task is not amenable to extractive strategies and requires an abstractive modeling approach. Hence, it provides us a better benchmark to assess abstractive models' abilities to produce abstractions which are faithful and factual. Finally, since we conclude that hallucination is a problem on this dataset, then we can safely conclude it is a problem for summarization datasets with longer summaries, as modeling longer-distance dependencies and discourse structures make the task harder.
4 Abstractive Summaries
We evaluate summaries from RNN, CNN and Transformer-based state-of-the-art abstractive summarization methods and the reference human writ
ten summaries. See the Appendix for hyperparameter and decoding details for all models.
Human Written Reference Summaries. The single-sentence summaries contained in the extreme summarization dataset (XSUM) are also evaluated as part of this study. These summaries were written by journalists as introductions to the news articles they precede. These summaries, therefore, often have true additional information not found in the document. Such divergence issue between source and target is not uncommon in conditional text generation (Kryscinski et al., 2019a; Wiseman et al., 2017; Dhingra et al., 2019).
RNN-based Seq2Seq. We use the Pointer-Generator model (PTGEN) introduced by See et al. (2017), an RNN-based attention-based sequence to sequence model which not only generates from the target vocabulary but can copy words from the source text.
Topic-aware Convolutional Seq2Seq. The Topic-aware Convolutional Sequence to Sequence model (TCONVS2S) introduced by Narayan et al. (2018a) is an abstractive system which is conditioned on the article's topics and based entirely on Convolutional Neural Networks (Gehring et al., 2017). TCONVS2S is better suited for extreme summarization, as convolution layers capture long-range dependencies between words in the document more effectively than RNNs. Simultaneously, the convolutional encoder associates each word with a topic vector, capturing whether it is representative of the document's content.
Transformer-based Abstractive Methods. We experiment with three Transformer-based model variants, all of which have 12 layers, a hidden size of 768, filter size of 3072, and 12 attention heads. GPT-TUNED: Radford et al. (2019) proposed Transformer-based Generative Pre-Trained (GPT) language models that can generate high quality text in open-ended generation setups. The proposed decoder-only architecture for language modeling can be easily adapted to abstractive summarization where the model first sees the document and, given a prompt, such as TL;DR;, generates its summary. Our GPT-TUNED is warm-started with a publicly available GPT checkpoint (Radford et al., 2019), but fine-tuned with supervised training on the extreme summarization dataset.
TRANS2S and BERTS2S: TRANS2S and BERTS2S are sequence to sequence models
<sup>3</sup>https://github.com/EdinburghNLP/XSum
Models	Human Eval Test Set
R1	R2	RL	BERTScore
PTGEN	30.01	9.38	23.76	74.30
TCONVS2S	30.89	11.47	25.80	75.23
TRANS2S	32.28	11.66	24.65	75.69
GPT-TUNED	21.82	4.72	16.28	70.35
BERTS2S	38.42	16.96	31.27	78.85
Table 1: ROUGE and BERTScore  $F_1$  scores for non-pretrained (the top block) and pretrained (the bottom block) models reported on the XSum dataset. These results are on the sampled human evaluation (500 items) dataset. The best results are boldfaced.
where both encoder and decoder are composed of Transformer layers (Vaswani et al., 2017; Rothe et al., 2020). All weights in TRANS2S are randomly initialized, but in BERTS2S, both encoder and decoder are initialized with the BERT-Base checkpoints (Devlin et al., 2019), with parameter sharing between the encoder and decoder, following Rothe et al. (2020). The only variable that is initialized randomly is the encoder-decoder attention in BERTS2S. Both models are then trained on the extreme summarization dataset.
The main focus of this work is not to propose a solution to hallucination related issues, but to achieve a better understanding of hallucinations in abstractive summarization through their human assessment. We randomly sampled 500 articles from the test set to facilitate our study. Using the full test set was unfeasible given its size and the cost of human judgments. We have trained annotators (fluent in English) specifically for our assessment. Our annotators went through two pilot studies to have a better understanding of intrinsic and extrinsic hallucinations, and factuality of summaries. Documents used in the pilot studies were not used in the final annotation. We also report on ROUGE (Lin and Hovy, 2003) scores, BERTScore (Zhang et al., 2020) and semantic inference metric such as textual entailment (Pasunuru and Bansal, 2018; Welleck et al., 2019; Falke et al., 2019; Kryscinski et al., 2019b) and question answering (Arumae and Liu, 2019; Wang et al., 2020).
5.1 Automatic Evaluation of Summaries
ROUGE (Lin and Hovy, 2003) provides a means to quickly assess a model's ability to generate summaries closer to human authored summaries. We report on ROUGE-1 and ROUGE-2 for informativeness and ROUGE-L, for fluency. Like ROUGE, BERTScore (Zhang et al., 2020) computes a similarity score for each token in the candidate sum
Figure 2: Human assessment of a system generated summary for the article in Figure 1. The annotation user interface is shown as it was shown to raters.
mary with each token in the reference summary. However, instead of exact matches, it computes token similarity using contextual embeddings. Results are presented in Table 1.
For both cases, the pretrained encoder-decoder architecture BERTS2S performed far superior to any other randomly initialized models, such as PTGEN, TCONVS2S and TRANs2S, and the decoder-only architecture GPT-TUNED. The differences between PTGEN, TCONVS2S and TRANs2S are not significant; all other differences are significant.
ROUGE and BERTScore are indicators of informativeness of summaries but they are not sufficient metrics to assess the overall quality of summaries. This becomes evident from our human assessments in the following sections where we employ human annotators to evaluate summaries generated with PTGEN, TCONVS2S, TRANS2S and BERTS2S, and the human authored summaries. We excluded GPT-TUNED abstracts from our study after their poor performance on the automatic measures.
5.2 Assessment of Hallucinations
In this assessment, human annotators were presented an article and a single-sentence summary for the article. They were stringently told to only assess the hallucinations in the summary and to not confuse their assessment with the quality of the summary. For summaries containing hallucinations, annotators were tasked with (i) identifying those text spans that were unfaithful to the article and (ii) for each text span, annotating whether the hallucination was intrinsic or extrinsic. We elicited judgments from three different annotators for each of 2500 (500x5) document-summary pairs. Figure 2 shows an example assessment of a summary of an article from Figure 1. Results from the full assessment are shown in Table 2, which shows the percentage of documents per system that were annotated as faithful or hallucinated (faithful  $= 100$  - hallucinated). The Appendix provides inter-annotator agreement of hallucinations as well as hallucinated span characteristics.
Extrinsic Hallucination due to Divergence between Source and Target. Our results con
Pairwise comparisons between all models using a one-way ANOVA with post-hoc Tukey HSD tests;  $p < 0.01$ .
Models	Hallucinated	Faith.	+Fact.
I	E	I ∪ E
PTGEN	19.9	63.3	75.3	24.7	27.3
TCONVS2S	17.7	71.5	78.5	21.5	26.9
TRANS2S	19.1	68.1	79.3	20.7	25.3
BERTS2S	16.9	64.1	73.1	26.9	34.7
GOLD	7.4	73.1	76.9	23.1	—