SlowGuess/NAACL_2021_OCR huggingface dataset:
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Rada Mihalcea and Hakan Ceylan. 2007. Explorations in automatic book summarization. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 380-389, Prague, Czech Republic. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9308-9319, Online. Association for Computational Linguistics.
Vahed Qazvinian and Dragomir R Radev. 2008. Scientific paper summarization using citation summary networks. arXiv preprint arXiv:0807.1560.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213, Florence, Italy. Association for Computational Linguistics.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596-4604, Stockholm, Sweden. PMLR.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In Proceedings of the
57th Annual Meeting of the Association for Computational Linguistics, pages 331-335, Florence, Italy. Association for Computational Linguistics.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020a. Sparse sinkhorn attention.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020b. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732.
Amy JC Trappey, Charles V Trappey, and Chun-Yi Wu. 2009. Automatic patent document summarization for collaborative knowledge systems and services. Journal of Systems Science and Systems Engineering, 18(1):71-94.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Christopher Walker, Stephanie Strassel, Stephanie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, et al. 2020b. Cord-19: The Covid-19 open research dataset. ArXiv.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020c. Linformer: Self-attention with linear complexity.
Chih-Hsuan Wei, Alexis Allot, Robert Leaman, and Zhiyong Lu. 2019. Pubtator central: automated concept annotation for biomedical full text articles. *Nucleic acids research*, 47(W1):W587-W593.
E.J. Williams. 1959. Regression Analysis. WILEY SERIES in PROBABILITY and STATISTICS: APPLIED PROBABILITY and STATIST ICS SECTION Series. Wiley.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus
Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3011-3021, Hong Kong, China. Association for Computational Linguistics.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontonon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. arxiv e-prints, art. arXiv preprint arXiv:2007.14062.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. CoRR, abs/1912.08777.
Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. Seal: Segment-wise extractive-abstractive long-form text summarization.
A GovReport Dataset Collection and Processing
For GAO reports, their summaries are organized as highlights. We collect GAO reports that include corresponding highlights and were published before Jul 7, 2020. The reports and highlights are published in PDF files. Most of the highlights are also reorganized and shown on the web page as HTML. Since PDF parsing is more prone to errors than web parsing, we only keep the reports whose highlights can be obtained on the corresponding web page to ensure the quality of extracted gold-standard summaries. For reports, we first convert the PDF files to HTML using PDFMiner<sup>6</sup>. We then parse the HTML into text into sections and paragraphs with handcrafted parsing rules. We remove the reports that do not have cover pages, as our rules are constructed for documents with then. We further remove parsed documents with empty sections, non-capitalized section titles, or a single section, since these are common patterns of incorrectly parsed documents. Failed parsing would also result in short documents. Therefore, we examine the reports with shorter length and then filter out  $10\%$  of the shortest reports.
$^{6}$ https://github.com/euske/pdfminer
We collect CRS reports that were published before May 20, 2020 from EveryCRSReport<sup>7</sup> where the original PDF files are already parsed into HTML. We only keep documents with expert-written summaries. We then gather texts from the html files.
FactCC Training Data Construction. Kryscinski et al. (2020) generate training data by applying rule-based transformations to sentences from source documents. We leverage reference summaries, where we train a FactCC model by reading a summary sentence (i.e., the claim) and a context to predict the corresponding label. A context is constructed by greedily selecting sentences that maximize the improvement of its ROUGE-2 when compared against the reference summary sentence. Following FactCC, we apply sentence negation, entity swap, and number swap to summary sentences to construct negative claims and use the original sentences as positive claims. During testing, we first find the context for each system summary sentence. The model then predicts a sentence-level faithfulness score by reading the system summary sentence and the context.
Evaluation Model Training. We fine-tune BERT (Devlin et al., 2019) for both FactCC and QA models. We include an additional classification head to predict entailment label or answer spans based on the [CLS] token. For GovReport dataset, we consider a base version of BERT with uncased tokens. For PubMed, we use a BERT model which is fine-tuned on PubMed abstracts to obtain better performance<sup>8</sup>.
Entity Extraction Model. We use OneIE to extract entities from the reference summary (Lin et al., 2020). OneIE is a unified framework that combines entities, relations, and events extraction in one model. The model leverages the BERT pretrained weights as the sentence embedding to produce entities, relations, and events from a sentence. Two OneIE models are built.
The first model for government reports is trained on the Automatic Content Extraction (ACE) 2005 dataset (Walker et al., 2006). This model can extract entities from general conversation contexts
Genia 2011	Genia 2013	PubMed
Entity Type
Anaphora	-	105	-
Entity	480	121	-
CellLine	-	-	614
Chemical	-	-	14,051
Disease	-	-	62,228
Mutation	-	-	164
Protein	11,539	3,562	15,577
Species	-	-	52,954
Event Type
Binding	880	167	-
Gene Expression	2,076	666	-
Localization	264	44	-
Negative Regulation	338	273	-
Phosphorylation	175	105	-
Positive Regulation	1,123	311	-
Protein Catabolism	100	23	-
Protein Modification	-	8	-
Regulation	292	72	-
Transcription	580	97	-
Ubiquitination	-	4	-
Table 9: Dataset description for training OneIE Biomedical extraction. While Genia 2011 and 2013 datasets focus more on event extraction, PubMed covers more entities.
such as People, Location, or Organization, and events such as Movement, Conflict, or Justice, etc.
The second model for scientific domain information extraction is trained on the Genia 2011 (BioNLP, 2011), Genia 2013 (BioNLP, 2013), and PubMed (Wei et al., 2019) datasets. It extracts entity such as Gene, Variant, Disease, Chemical, or Species, and events such as Gene Expression, Binding, Protein Modification, or Positive Regulation, etc. The full list of entity and event types can be found in Table 9. To train this model, we fine-tune the BioBERT pre-trained model (Lee et al., 2020) on the COVID-19 Open Research (CORD-19) dataset (Wang et al., 2020b). As we proposed, this model is applied to the PubMed data.
C Additional Sample Outputs
We include two samples from GovReport and PubMed to further illustrate that our model with HEPOS attention generates more faithful and informative summaries in Fig. 7 and Fig. 8.
D Human Evaluation Guideline
In human evaluation, annotators are asked to evaluate the system summaries generated for a report or a paper. In addition to the summaries, annotators are provided with the report or the paper to be summarized and a corresponding human-written
7https://www.everycrsreport.com 8https://huggingface.co/monologg/biobert_v1.0_pubmed_pmc
reference. Human judges evaluate each system summary sentence by sentence. The annotation consists of three tasks, which are described below.
Task 1: Aspect Labeling. First, annotators are asked to decide which aspect each sentence belongs to. For government reports, each sentence should be categorized into three aspects: (1) Why GAO did this study, (2) What GAO found, and (3) What GAO recommends. For scientific papers, summaries have four aspects: (1) Introduction and Literature, (2) Methods, (3) Results, and (4) Discussion and Conclusion. Table 10 and Table 11 contain example reference summaries with labeled aspects.
Task 2: Sentence-level Faithfulness Error Labeling. Next, annotators will judge whether each sentence contains any unfaithful content. Unfaithful content is categorized into three types. A "0" or "1" label will be given to each type, where "0" indicates the sentence is free of such type of error, and "1" otherwise.
Concretely, unfaithful content is the fabricated or contradictory content which is not present or contradicts the facts in the source article. It can also be ambiguous expression which distorts the meaning. Here are detailed descriptions for the three types of errors:
- Hallucination error refers to fabricated content that cannot be found or inferred from the source.
- Misconstruction error that is due to deletion of entities, events, or clauses, resulting in sentences that are incomplete, missing context, or ungrammatical.
- Misconstruction error that is caused by false concatenation of content from different places in the source.
Task 3: Aspect-level Summary Quality Rating. After reading the full summary, annotators will evaluate the informativeness of the summary for each aspect—whether the summary provides a necessary and enough coverage of information in the reference. For instance, whether the summary covers all the salient points in "Why GAO did this study".
Here are detailed descriptions of informativeness:
- 5: Summary covers enough key points in the
reference (only misses minor topics), and is free of unfaithful errors.
- 4: Summary covers major key points (e.g., 80 percent) and may miss one or two key points in the reference. Summary can contain one unfaithful error.
- 3: Summary covers roughly half of the key points in the reference or contains 2 or 3 unfaithful errors.
- 2: Summary only covers 1 or 2 key points and misses many important topics (e.g.  $>80$  percent) in the reference, or contains more than 3 major unfaithful errors, e.g. summary fabricates or distorts some facts.
- 1: Summary is irrelevant and does not cover any content in the reference.
What GAO found
The Food and Drug Administration (FDA) and U.S. Department of Agriculture (USDA) have begun collaborating on regulatory oversight of cell-cultured meat. For example, in 2019, the agencies signed an interagency agreement and created three working groups to carry out the terms of the agreement. However, the agreement and working groups could more fully incorporate practices to enhance and sustain collaboration, such as defining outcomes. For example, the agreement identifies the development of labeling principles as an outcome, but does not describe how the agencies will track and monitor progress toward this outcome, and the working groups identify a lead agency but not members' roles. (...)
Developing and updating written guidance and agreements is also a leading practice for interagency collaboration. By fully incorporating leading practices into their efforts to collaborate, the agencies could minimize potential overlap and fragmentation, use resources in a more efficient manner, and better ensure the public and other key stakeholders have clarity about the agencies' oversight responsibilities. (...)
The frequency of inspections the agencies conduct varies, as follows: FDA. FDA conducts risk-based inspections of high- risk and non-high-risk food facilities. FDA's authority requires a risk- based approach, in which inspection rates vary depending on the level of risk associated with a food product. (...)
In 2010, FDA began conducting inspections of meat slaughter facilities to determine whether they were complying with the agency's longstanding interpretation of its statutes requiring it to do so. In addition, USDA conducted inspections of slaughter facilities that process catfish to determine if they are complying with USDA's requirement to inspect catfish slaughter facilities. USDA also conducts inspections of processing facilities to verify that they are following USDA's requirements to inspect these facilities. In 2011, USDA began inspecting processing facilities for catfish for the first time. (...)
Model w/ HEPOS enc-dec attn.:
The Departments of Health and Human Services (HHS) and Agriculture have taken several steps to coordinate their oversight of the development, commercialization, and regulation of cell cultured meat and seafood, but they have not fully incorporated leading practices for interagency collaboration. For example, HHS and USDA have established interagency working groups that have met to discuss issues such as coordination and coordination. However, the agencies have not yet fully incorporated some leading practices, such as establishing clear roles and responsibilities, into their interagency agreements. (...)
Figure 7: Sample summaries for a government report. Model with truncated input generates unfaithful content. Our HEPOS encoder-decoder attention with Sinkhorn encoder attention covers more salient information in “What GAO found” aspect.
Original Paper:
(...) anatomical abnormalities included complex pelvicalyceal anatomy in 98, pelviureteric junction obstruction (pujo) in 35, ectopic, horseshoe, cross-fused ectopic kidneys in 40, and duplex system with nonfunctioning upper or lower moiety in 14 patients. (...)
of the 1404 patients from rural area, 547  $(39\%)$  were managed by open surgery as compared to 362  $(22\%)$  of the patients from urban areas. (...)
the mean size of the kidney stone was  $5.05\ 5.88\ \mathrm{cm}$ , ureter  $1.95\ 1.33\ \mathrm{cm}$ , and bladder  $7.6\ 3.2\ \mathrm{cm}$ . (…) overall success rate was  $90\%$  in pcnl,  $96\%$  in urs,  $100\%$  in cystolithotripsy, and  $81\%$  in swl.
Results of 3969 surgeries, 2794 (70%) were minimally invasive surgery (mis) techniques to include sw1  $19\%$ , pcnl  $16\%$ , urs  $18.9\%$ , and pucl + pccl  $16\%$  and 1175 (30%) were open surgeries.
the main factors necessitating open surgery were large stone burden  $37\%$ , anatomical abnormalities  $16\%$ , stones with renal failure  $34\%$ , gross hydronephrosis with thin cortex  $58\%$ , urinary tract infection (uti)  $25\%$ , and failed mis  $18\%$ . Nearly  $50\%$  of the surgeries were necessitated by economic constraints and long distance from center where one-time treatment was preferred by the patient. tone-free rates by open surgeries were pyelolithotomy  $91\%$ , ureterolithotomy  $100\%$ , and cystolithotomy  $100\%$  with complication rate of up to  $3\%$ .
results: the mean age of the patients was 6.5 years. the mean follow-up period was 12.5 months. mean stone size was  $1.5\mathrm{cm}$ . the majority of the stones were larger stones  $(n = 454)$ . the stone free rate was  $100\%$ . open surgery was performed in children with large bulk of stones, anatomical abnormalities, marked obstructive cortical atrophy and scarring, gross hydronephrosis, or urinary tract infection (uti).
the mean age of the patients was 6.35, 3.7 years with a male-to-female ratio of 2.8:1, and a range of 25 days (<1 month) to 15 years. Anatomical abnormalities included complex pelvicalyceal anatomy in 98, pelviureteric junction obstruction (pujo) in 35, ectopic, horseshoe, cross-fused ectopic kidneys in 40, and duplex system with nonfunctioning upper or lower moiety in 14 patients. The mean size of the kidney stone was  $5.055.88\mathrm{cm}^3$ . Of the 1404 patients from rural areas, 547  $(39\%)$  were managed by surgery as compared to 362  $(22\%)$  patients from urban areas. overall success rate was  $90\%$  in pcnl,  $96\%$  in urs,  $100\%$  in cystolithotripsy, and  $81\%$  in swl.
Figure 8: Sample summaries for a scientific paper. Model with truncated input generates fabricated facts. Our HEPOS encoder-decoder attention with LSH encoder attention are more faithful for the aspect of “results”.
Aspect	Example
Why GAO Did This Study	To protect data that are shared with state government agencies, federal agencies have established cybersecurity requirements and related compliance assessment programs. Specifically, they have numerous cybersecurity requirements for states to follow when accessing, storing, and transmitting federal data. GAO was asked to evaluate federal agencies' cybersecurity requirements and related assessment programs for state agencies. The objectives were to determine the extent to which (...)
What GAO Found	Although the Centers for Medicare and Medicaid Services (CMS), Federal Bureau of Investigation (FBI), Internal Revenue Service (IRS), and Social Security Administration (SSA) each established requirements to secure data that states receive, these requirements often had conflicting parameters. Such parameters involve agencies defining specific values like the number of consecutive unsuccessful logon attempts prior to locking out the user. Among the four federal agencies, the percentage of total requirements with conflicting parameters ranged from 49 percent to 79 percent. Regarding variance with National Institute of Standards and Technology guidance, GAO found that the extent to which the four agencies did not fully address guidance varied from 9 percent to 53 percent of total requirements. The variances were due in part to the federal agencies' insufficient coordination in establishing requirements. (...)
What GAO Recommends	GAO is making 12 recommendations to the four selected agencies and to OMB. Three agencies agreed with the recommendations and one agency (IRS) partially agreed or disagreed with them. OMB did not provide comments. GAO continues to believe all recommendations are warranted.
Table 10: Sample reference summary with aspects in a GAO report.
Aspect	Keywords	Example
Introduction and Literature	introduction, case, objectives, purposes, objective, purpose, background, literature, related work	background: the present study was carried out to assess the effects of community nutrition intervention based on advocacy approach on malnutrition status among school - aged children in shiraz , iran .
introduction: low serum vitamin d levels are associated with increased postural sway . vitamin d varies seasonally . this study investigates whether postural sway varies seasonally and is asso-ciated with serum vitamin d and falls .
Methods	materials and methods, techniques, methodology, materials, research de-sign, study design	materials and methods: this case - control nutritional intervention has been done between 2008 and 2009 on 2897 primary and secondary school boys and girls ( 7 - 13 years old ) based on advocacy approach in shiraz , iran . the project provided nutritious snacks in public schools over a 2 - year period along with advocacy oriented actions in order to implement and promote nutritional intervention . for evaluation of effectiveness of the intervention growth monitoring indices of pre- and post - intervention were statistically compared .
Results	results, experiments, observations	results: the frequency of subjects with body mass index lower than 5% decreased significantly after intervention among girls ( p = 0.02 ) . however, there were no significant changes among boys or total population . (...)
Discussion and Conclusion	discussion, limitation, conclusions, concluding	conclusion: this study demonstrates the potential success and scalability of school feeding programs in iran . community nutrition intervention based on the advocacy process model is effective on reducing the prevalence of underweight specifically among female school aged children .
Table 11: Sample reference summary with aspects labeled in a PubMed article. Keywords are used to match different parts of the summaries to the four aspects.