SlowGuess/NAACL_2022_OCR huggingface dataset:
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345.
Yizhu Liu, Zhiyi Luo, and Kenny Zhu. 2018. Controlling length in abstractive summarization using a convolutional neural network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4110-4119.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity-level factual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2727-2733, Online. Association for Computational Linguistics.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
Shashi Narayan, Yao Zhao, Joshua Maynez, Goncalo Simoes, and Ryan McDonald. 2021. Planning with entity chains for abstractive summarization. arXiv preprint arXiv:2104.07606.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the
58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. 2011. Ontonotes release 4.0. LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium.
Haopeng Zhang and Jiawei Zhang. 2020. Text graph transformer for document classification. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. 2020a. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv: Arxiv-2001.05140.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020b. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 718-733, Online. Association for Computational Linguistics.
We use Huggingface libraries (Wolf et al., 2020) for all our experiment implementations. Our backbone abstractive summarization model is BARTlarge (Lewis et al., 2020), a pre-trained denoising autoencoder language model with  $336M$  parameters based on the sequence-to-sequence transformer (Vaswani et al., 2017). For fair comparison, we fine-tune BART-large on each dataset for on 8 Tesla A100 GPU pods with same learning rate  $5e - 5$  with weight decay using Adam optimizer (Kingma and Ba, 2014). We set hyperparameter  $k = 3$  for all experiments. Larger number of  $k$  doesn't increase the performance significantly. The three discrete ECC bins are represented with control codes: <FF-low>, <FF-mid> and <FF-high> respectively. The entity coverage precision boundaries are 0.36 and 0.5 for Pubmed, 0.33 and 0.66 for SAMsum and Xsum.
For entity recognition, we use a neural Named Entity Recognition (NER) system from the Stanza NLP toolkit (Qi et al., 2020) trained on the OntoNotes corpus (Weischedel et al., 2011) except for Pubmed dataset. Since Pubmed is a medical scientific article collection, we use biomedical, scientific, and clinical text Named Entity Recognition toolkit scispaCy (Neumann et al., 2019) instead.
B Representative Examples Analysis
In Table 8, we provide several representative examples from XSum dataset. Example 1 (first row) shows how our entity control method gets rid of hallucination terms from BART output. The reference summary here is not faithful since 'Los Angeles' is not covered in the source document. The correction baseline changes 'Los Angeles' to 'Mexico', which is a factual error. In contrast, the ECCoutput is totally faithful to the source document and contains salient information.
Example 2 (second row) shows the outputs decoded with different control codes during inference. We can see the output decoded with low faithfulness control code is still fluent and reasonable, but contains less faithful entities compared to the output decoded with high faithfulness control code.
Example 3 (third row) shows an example of factual statement, which is verifiable in the real world independent of the source text. The reference summary uses 'most of Wales' to summarize the county names in the source document. This type of hallucination needs more external knowledge and com
BART: A video game based on one of the world's most popular wrestling traditions has been launched at the E3 gaming show in Los Angeles.'
Correction: A video game based on one of the world's most popular wrestling traditions has been launched at the E3 gaming show in Mexico.
ECC: A video game dedicated to Mexican wrestling has been released at E3.
Reference: One of the more unusual titles at E3, the worlds largest video games exhibition held each year in Los Angeles, is Konami's Lucha Libre AAA: Heroes del Ring.
BART: Tourists in Spain have been accused of harassing a dolphin after it became stranded on a beach.
Low Code: A dolphin that became stranded in the sea off the coast of Spain has been harassed by a group of tourists.
High Code: A dolphin that became stranded in the sea off the coast of Andalucia has been harassed by tourists.
Reference: A baby dolphin has died after it was surrounded by tourists looking to take photographs on a beach in southern Spain.
Document: The warning begins at 22:00 GMT on Saturday and ends at 10:00 on Sunday. The ice could lead to difficult driving conditions on untreated roads and slippery conditions on pavements, the weather service warned.
Only the southernmost counties and parts of the most westerly counties are expected to escape. Counties expected to be affected are Carmarthenshire, Powys, Ceredigion, Pembrokeshire, Denbighshire, Gwynedd, Wrexham, Conwy, Flintshire, Anglesey, ..., Rhondda Cynon Taff and Torfaen.
Reference:The Met Office has issued a yellow weather warning for ice across most of Wales.
Table 8: Representative examples from the XSum test set.
Model	Faith. %	Ex. %	In. %	Quality
BART	15.0 ± 7.4	54.0 ± 11.2	39.0 ± 5.8	2.31 ± 0.14
ECC	28.0 ± 6.2	41.0 ± 7.2	37.0 ± 8.3	2.43 ± 0.17
ECC-zero	31.0 ± 2.8	48.0 ± 9.3	38.0 ± 7.2	1.73 ± 0.07
Table 9: Human evaluation results of 50 test examples sampled from XSum dataset.
monsense reasoning to decide its factuality. Our method only focuses on entity level hallucination problems instead.
C Human Evaluation Confidence
Our human evaluation follows the setting of prior work (Chen et al., 2021). We calculate the inter-annotator agreement with additional annotations from two other experts. We estimate the adjusted mean and  $95\%$  confidence interval from the mean and standard deviation. The full results are shown in Table 9.