SlowGuess/ACL_2020_OCR huggingface dataset:
Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. 2019. Sample efficient text summarization using a single pre-trained transformer. CoRR, abs/1905.08836.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019a. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 540-551, Hong Kong, China.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2019b. Evaluating the factual consistency of abstractive text summarization. CoRR, abs/1910.12840.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. CoRR, abs/1808.06226.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas.
Chin Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150-157.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In Proceedings of the 6th International Conference on Learning Representations, Vancouver Canada.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Inderjeet Mani. 2001. Automatic summarization, volume 3. John Benjamins Publishing.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don't give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1747-1759, New Orleans, Louisiana.
Shashi Narayan, Goncalo Simoes, Ji Ma, Hannah Craighead, and Ryan T. McDonald. 2020. QURI-OUS: Question generation pretraining for text generation. CoRR, abs/2004.11026.
Ani Nenkova. 2005. Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference. In Proceedings of the 20th National Conference on Artificial Intelligence - Volume 3, pages 1436-1441.
Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2-3):103-233.
Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The Pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 145-152, Boston, Massachusetts, USA.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 646-653, New Orleans, Louisiana.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. To appear in Transactions of the Association for Computational Linguistics, abs/1907.12461.
Evan Sandhaus. 2008. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia, 6(12).
Natalie Schluter. 2017. The limits of automatic summarisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 41-45, Valencia, Spain.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1073-1083, Vancouver, Canada.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 843-861, Hong Kong, China.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, California.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Alex Wang, Kyunghyun Cho, and Michael Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual Conference, Formerly Seattle, USA.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In Proceedings of the 8th International Conference on Learning Representations, Virtual Conference, Formerly Addis Ababa Ethiopia.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112-1122, New Orleans, Louisiana.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proceedings of the 8th International Conference on Learning Representations, Virtual Conference, Formerly Addis Ababa Ethiopia.
A Model Hyperparameters and Predictions
PTGEN and TCONVS2S model predictions are provided by Narayan et al. (2018a) and Transformer model predictions from GPT-TUNED, TRANS2S and BERTS2S, by Rothe et al. (2020). Both PTGEN and TCONVS2S use a Stanford tokenized vocabulary size of 50k. TRANS2S and BERTS2S use a vocabulary size of around  $\sim 30$ k WordPieces (Wu et al., 2016) to match BERT pretrained vocabulary and, GPT-TUNED, a vocabulary size of around  $\sim 50$ k SentencePieces (Kudo and Richardson, 2018) to match the GPT-2 pretrained vocabulary. All models use the same uncased vocabulary on both source and target sides. Both PTGEN and TCONVS2S summaries were generated using beam search with beam size 10, the Transformer models use beam size of 4. See Narayan et al. (2018a) and Rothe et al. (2020) for more details on these models.
Models	Fleiss' Kappa
Hall.	Fact.	Rept.	Inco.
PTGEN	0.70	0.91	0.89	0.84
TCONVS2S	0.73	0.91	0.93	0.90
TRANS2S	0.67	0.91	0.92	0.90
BERTS2S	0.67	0.88	0.94	0.93
GOLD	0.71	—	1.00	0.98
B Inter annotator agreement
We estimated Fleiss's Kappa  $(k)$  to assess the agreement among our raters when categorizing a word in the summary as one of faithful, intrinsically hallucinated and extrinsically hallucinated. The results are shown in Table 6. All models showed substantial agreement  $(0.61 \leq k \leq 0.80$ ; Landis and Koch, 1977) among their annotations.
Table 6 also shows Fleiss's Kappa  $(k)$  to assess the agreement among our raters for factuality. All models showed almost perfect agreement (0.81  $\leq k\leq 1.0$ ; Landis and Koch, 1977) among their annotations.
C Highlighted Span Characteristics
Results in Table 7 shed some light on the characteristics of hallucinated spans observed in different abstracts. GOLD abstracts showed the least number of intrinsically hallucinated spans (0.55 per document), whereas, PTGEN abstracts showed the
Table 6: Fleiss's Kappa scores measuring word-level agreements among annotators for different annotation tasks: hallucination (Hall.), factuality (Fact.), repetition (Rept.) and incoherence (Inco.) assessments.
Models	Intrinsic total (avg.)	Extrinsic total (avg.)	avg. length
PTGEN	625 (1.35)	1424 (2.85)	8.48
TCONVS2S	518 (1.04)	1556 (3.11)	8.44
TRANS2S	589 (1.18)	1556 (3.11)	7.39
BERTS2S	530 (1.06)	1520 (3.04)	6.12
GOLD	276 (0.55)	1807 (3.61)	7.11
Table 7: Total number of spans and the average number of spans per document, annotated as intrinsic or extrinsic hallucinations for all 500 document-summary pairs by three annotators. We also show the average span length for each system.
Models	Repetition	Incoherence
PTGEN	17.5	20.3
TCONVS2S	16.7	17.7
TRANS2S	8.9	11.5
BERTS2S	8.7	9.5
GOLD	0.0	0.8
Table 8: Repetition and Incoherence Evaluation. The numbers show the percentage of 500 summaries where at least one word in a summary was annotated by all three annotators with the "Repetition" or "Incoherence" related issue. The lowest numbers are boldfaced.
Metric	Faithful	Factual
ROUGE-1	0.197	0.125
ROUGE-2	0.162	0.095
ROUGE-L	0.162	0.113
BERTScore	0.190	0.116
Repetition	0.064	0.075
Incoherence	0.067	0.082
QA	0.044	0.027
Entailment	0.431	0.264
Table 9: Spearman's correlation coefficient  $\left( {\left| {r}_{s}\right| }\right)$  of different metrics with faithful and factual annotations.
least number of extrinsically hallucinated spans (2.85 per document). Interestingly, the average span length for PTGEN summaries was 8.48 words, much higher than 6.12 words for BERTS2S summaries. Our result demonstrates that (i) the effect of hallucination in BERTS2S is more local than what we observe in PTGEN and (ii) despite a lower number of extrinsically hallucinated spans or documents in PTGEN compared to that in BERTS2S (2.85 vs 3.04 spans per document,  $63.3\%$  vs  $64.1\%$  documents), the total number of words that were annotated as extrinsic hallucination is much higher in PTGEN than in BERTS2S (12075 vs 9302 words).
D Assessment of Linguistic Irregularities.
Following standard practice in summarization, all 2500 document-summary pairs were annotated for repetition and incoherence related linguistic irregularities. Annotators were presented only a single-sentence summary and were asked to identify all
Models	Faithful	Hallucinated	Factual
I total	factual	E total	factual	I ∪ E total	factual
PTGEN	24.7	19.9	0.4	63.3	2.2	75.3	2.6	27.3
TCONVS2S	21.5	17.7	0.8	71.5	5.0	78.5	5.4	26.9
TRANS2S	20.7	19.1	1.4	68.1	3.4	79.3	4.6	25.3
BERTS2S	26.9	16.9	1.8	64.1	6.6	73.1	7.8	34.7
GOLD	23.1	7.4	—	73.1	—	76.9	—	—
Table 10: Intrinsic vs Extrinsic Hallucinations and their factuality. The numbers in "Hallucinated" columns show the percentage of summaries out of 500 where at least one word was annotated by all three annotators as an intrinsic (I) or extrinsic (E) hallucination. When a summary is not marked with any hallucination, it is "faithful" (1- IUE). The "factual" columns within the "Hallucinated" column show for each type (I, E and IUE), the percentage of summaries out of 500 annotated by all three annotators as factual. The final "Factual" column shows the total percentage of factual summaries (Faithful  $+\mathrm{I}\cup \mathrm{E}_{\mathrm{factual}}$  ). The highest numbers for faithful and factual, and the lowest numbers for hallucinations are boldfaced.
spans of text in the summary that were either repeated or made the summary incoherent. We again elicited judgments from three different annotators for each document-summary pair. Results are shown in Table 8.
Overall, all neural text generation systems are getting better in generating repetition-free and coherent single-sentence summaries of news articles. Transformer-based models, TRANs2S and BERTS2S in particular, perform superior to RNN-based PTGEN and CNN-based TCONVS2S models. Nonetheless, Table 9 shows that these metrics fail to correlate with faithful, hallucinated and factual assessments of summaries. Fleiss's Kappa  $(k)$  values for repetition and incoherence assessments showed almost a perfect agreement  $(0.81 \leq k \leq 1.0$ ; Landis and Koch, 1977) among our raters (see Table 6).
E Full Hallucination Results
Table 10 has the full results from our human study of hallucinations.