SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 3: Results on evaluating encoder and encoder-decoder attentions on input of the same length. Best ROUGE scores of fixed patterns, learnable patterns, and enc-dec attentions are in red, orange, and purple, respectively.  $\ast$ : significantly better than comparison(s) using the same encoder or enc-dec attention (approximation randomization test,  $p < 0.0005$ ).
System (MAXLEN)	GovReport	PubMed
R-1	R-2	R-L	R-1	R-2	R-L
Baselines
PEGASUS (1024)	-	-	-	45.97	20.15	41.34
TLM (full)	-	-	-	42.13	16.27	39.21
SEAL (full)	-	-	-	46.50	20.10	42.20
DANCER (full)	-	-	-	46.34	19.97	42.42
BIGBIRD (3072)	-	-	-	46.32	20.65	42.33
Encoder variants w/ full enc-dec attn.
FULL (1024)	52.83	20.50	50.14	45.36	18.74	40.26
STRIDE (4096)	54.29	20.80	51.35	46.95	19.98	41.67
LIN. (3072)	44.84	13.87	41.94	43.69	16.35	38.66
LSH (4096)	54.75	21.36	51.27	47.54	20.79	42.22
SINKHORN (5120)	55.45	21.45	52.48	47.96	20.78	42.53
Encoder variants w/ HEPOS enc-dec attn. (ours)
LSH (7168)	55.00	21.13	51.67	48.12	21.06	42.72
SINKHORN (10240)	56.86	22.62	53.82	47.93	20.74	42.58
Table 4: ROUGE scores for models trained on the same GPU. SINKHORN with HEPOS enc-dec attention and LSH with HEPOS both read more text and obtain significantly better scores than other models on GovReport and PubMed  $(p < 0.0005)$ .
System (MAXLEN)	R-1	R-2	R-L
Baselines
PEGASUS (1024)	44.21	16.95	38.83
TLM (full)	41.62	14.69	38.03
SEAL (full)	44.3	18.0	39.3
DANCER (full)	45.01	17.60	40.56
BIGBIRD (3072)	46.63	19.02	41.77
Encoder variants w/HEPOS enc-dec attn. (ours)
LSH (7168)	48.24	20.26	41.78
SINKHORN (10240)	47.87	20.00	41.50
Table 5: Automatic evaluation on arXiv. Our best model yields better ROUGE scores than previous state-of-the-art models.
full encoder attention, implying the effectiveness of HEPOS on both identifying the salient content and capturing the global context.
6.3 Reading More Input Boosts Informativeness
We investigate whether processing more words generates more informative summaries.
Comparisons include recent top-performing abstractive models: PEGASUS (Zhang et al., 2019), a large pre-trained summarization model with truncated inputs; TLM (Pilault et al., 2020), DANCER (Gidiotis and Tsoumakas, 2020), and SEAL (Zhao et al., 2020), all of them using hybrid extract-then-abstract methods; and BIGBIRD (Za-heer et al., 2020), which combines sliding window,
global and random token attentions in the encoder.
For encoder variants, we pick the best performing model from fixed patterns to be combined with full encoder-decoder attention, i.e., sliding window with stride (STRIDE), low-rank method (LIN.), and learnable patterns (LSH and SINKHORM). We then combine learnable patterns with HEPOS to support processing more text. All models consume as long an input as the memory allows.
Results. Overall, models that read more text obtain higher ROUGE scores, according to results on Gov-Report and PubMed in Table 4. First, different encoder variants with full encoder-decoder attentions attain better results than the full attentions baseline except Linformer. Second, adding HEPOS encoder-decoder attention almost doubles the words that can be processed and further improves the performance. This highlights the importance of handling both encoder attentions and encoder-decoder attentions efficiently. Notably, HEPOS with an LSH encoder achieves new state-of-the-art results on PubMed, outperforming BigBird which only uses sparse attentions on the encoder. We also report performances of our two best models with HEPOS on arXiv in Table 5, and they outperform all competitive abstractive models.
As can be seen from the sample summaries in Fig. 3, our model that reads in  $10\mathrm{k}$  tokens generates more informative summary than the full attention model that only processes 1k tokens. Fig. 4 further shows that ROUGE-2 scores can be consistently lifted when reading more input, with similar trends observed on ROUGE-1 and ROUGE-L. More sample outputs are presented in Appendix C.
6.4 Reading More Input Improves Faithfulness
Here we first show human evaluation results on informativeness and unfaithful errors in the generated summaries. We sample 100 documents from GovReport and PubMed (50 each) with structured references that are labeled with aspects as described in § 4 and Appendix D. Each sample is evaluated by two fluent English speakers, who have cumulatively annotated tens of thousands of sentences for the same tasks before this work. Annotators are asked to label each summary sentence with an aspect and then decide whether it contains any type of error. Three types of unfaithful errors are considered: (i) hallucination—fabricating content not present in the input, (ii) deletion—incorrectly
Human-written Summary:
In fiscal year 2018, Medicaid covered approximately 75 million individuals at an estimated cost of  $629 billion,$ 393 billion of which were federal funds. (...)
While CMS is generally required to disallow, or recoup, federal funds from states for eligibility-related improper payments if the state's eligibility error rate exceeds 3 percent, it has not done so for decades, because the method it used for calculating eligibility error rates was found to be insufficient for that purpose. To address this, in July 2017, CMS issued revised procedures through which it can recoup funds for eligibility errors, beginning in fiscal year 2022. (...)
Model w/ full attn.:
Medicaid is a federal-state program that provides health care coverage to low-income individuals and families. (...) CMS officials stated that they have provided states with guidance on how to use data from SSA's automated system for eligibility determinations, (...
CMS officials said that they did not have guidance on when states should use SSA data to evaluate eligibility based on nonfinancial or financial criteria. (...)
Model w/ HEPOS enc-dec attn. (ours):
The Patient Protection and Affordable Care Act (PPACA) expanded Medicaid coverage to millions of low-income adults and children with disabilities and their eligible dependents. (...)
The selected states also reported that they did not have adequate processes to address these issues. CMS has taken steps to improve its oversight of the Medicaid program, including issuing guidance to states on the use of MAGI-exempt bases for determining eligibility, but these efforts have not been fully implemented. (...)
Figure 3: Sample summaries for a government report. The model with truncated input generates unfaithful content. HEPOS attention with a Sinkhorn encoder covers more salient information.
Figure 4: Summarizing articles truncated at different lengths by the best models: LSH (7168)  $^+$  HEPOS on PubMed and SINKHORN (10240)  $^+$  HEPOS on GovReport. Reading more consistently improves ROUGE-2.
deleting crucial entities, events, or clauses, and (iii) false concatenation—inappropriately concatenating components from different sentences. 1 is given if any judge determines that a certain type of error exists in the sentence, 0 otherwise.
After reading the full summaries, each judge also scores aspect-level informativeness—whether the
System (MaxLen)	Inf.↑	Hal.↓	Del.↓	Concat.↓
GovReport
Encoder variants w/ full enc-dec attn.
FULL (1024)	3.29	15.2%	3.5%	9.5%
SINKHORN (5120)	3.32	11.0%	2.3%	9.4%
Encoder variant w/ HEPOS enc-dec attn. (ours)
SINKHORN (10240)	3.53	11.5%	3.4%	8.8%
PubMed
Encoder variants w/ full enc-dec attn.
FULL (1024)	3.27	20.1%	2.8%	14.3%
SINKHORN (5120)	3.94	4.8%	1.6%	9.6%
Encoder variant w/ HEPOS enc-dec attn. (ours)
SINKHORN (10240)	4.18	3.5%	2.2%	9.1%
Table 6: Human evaluation on informativeness (Inf.) (1-to-5), and percentages of unfaithful errors due to hallucination (Hal.), deletion (Del.), and false concatenation (Concat.). Inter-rater agreement with Krippendorf's  $\alpha$  for all columns: 0.59, 0.59, 0.53 and 0.60.
summary covers important information of an aspect when compared with the reference. All system summaries and references are presented in a random order. Human evaluation guidelines and sample summaries for different aspects are included in Appendix D.
Results. Overall, reading more text significantly improves informativeness as well as reduces fabricated content. From Table 6, we observe that HEPOS attention, combined with a SINKHORN encoder, obtains better informativeness scores than comparisons that read in less text on both datasets. This echos results from automatic evaluation in the previous section. Moreover, both models that use efficient attentions reduce unfaithfulness, especially hallucination errors, when compared with the full attention model, which only reads 1024 tokens. As the models read more content, they learn to surface more factual and richer content in the summaries, as seen in Fig. 3.
Next, we explore if reading more helps correctly reflect the content in documents' later sections. We plot aspect-level human ratings of informativeness and unfaithful errors on PubMed and GovReport in Fig. 5 and Fig. 6. We report percentages of sentences with unfaithful errors by majority voting (i.e., at least one error is found by both annotators in the sentence). As can be seen, our models consistently improve informativeness and reduce errors across sections, especially for "Results" and "Conclusions" on PubMed and "What GAO recommends" on GovReport—these sections often appear in the later part of the source documents.
Figure 5: Aspect-level informativeness and percentages of sentences containing unfaithful errors as labeled by both human judges on PubMed. Models with efficient attentions reduce errors for later sections in the sources, e.g., "Results" and "Conclusion".
Figure 6: Aspect-level informativeness and percentages of sentences with unfaithful errors on GovReport.
Especially, we find that the full attention model tends to produce fabricated numbers in resultant summaries, whereas our models are able to correct them.
Lastly, we report the entailment-based FactCC and QA scores APES and  $\mathrm{APES}_{src}$  for top performing models in Table 7. The results again show that consuming longer input leads to more faithful summaries, though the differences are less pronounced.
6.5 Correlations between Human and Automatic Metrics
Finally, we study whether the faithfulness evaluation metrics correlate with human judgment. As shown in Table 8, on both government reports and scientific papers,  $QA$  metrics are better correlated with human ratings, with our newly pro
System (MaxLen)	GovReport	PubMed
F. APESAPESrc	F. APESAPESrc
FULL (1024)	58.9	42.7	42.7	74.6	43.2	31.5
Encoder variants w/ full enc-dec attn.
STRIDE (4096)	55.3	43.1	42.5	72.7	43.8	31.9
LIN. (3072)	48.4	35.7	36.3	67.7	39.3	29.5
LSH (4096)	55.7	44.0	43.6	73.2	46.7	35.1
SINKHORN (5120)	57.0	43.6	42.1	72.9	46.8	35.4
Encoder variants w/ HEPos enc-dec attn. (ours)
LSH (7168)	59.6	44.0	44.2	73.3	47.5	35.6
SINKHORN (10240)	60.1	44.0	44.3	71.9	46.2	34.8
Table 7: Evaluation with FactCC (F.), APES, and the new  $\mathrm{APES}_{src}$  metric, with higher numbers indicating more faithful summaries.
Metric	GovReport	PubMed
Inf.↑	Err.↓	Inf.↑	Err.↓
FactCC	0.07	-0.08	0.10	-0.14
APES	0.16	-0.15	0.25	-0.31
\( APES_{src} \)	0.21	-0.23*	0.32*	-0.32
Table 8: Pearson correlation between human ratings and metrics. We use aggregated unfaithful errors (Err.). \*: significantly better than other metrics based on William's test (Williams, 1959)  $(p <   0.05)$
posed  $\mathrm{APES}_{src}$  being the stronger of the two. After inspection, we find that human-written summaries contain paraphrases or acronyms that APES cannot capture via strict lexical matching. For instance, for the question "Diabetes may worsen in patients", the reference answer is "death rate", whereas answers from the source and the system summary are both "mortality".  $\mathrm{APES}_{src}$  captures this, but not APES.
Summarizing long inputs has been investigated in many domains, including books (Mihalcea and Ceylan, 2007), patents (Trappey et al., 2009), movie scripts (Gorinski and Lapata, 2015), and scientific publications (Qazvinian and Radev, 2008). However, the datasets are often too small to train neural models. Cohan et al. (2018) publish two large-scale datasets by collecting articles from ARXIV and PUBMED. Popular methods rely on extractive summarizers that identify salient sentences based on positional information (Dong et al., 2020) or combined global and local contexts (Xiao and Carenini, 2019), where each sentence is represented as aggregated word embeddings. However, extractive summaries are often redundant and in
coherent, highlighting the need for handling long documents via abstractive summarization.
To that end, extract-then-abstract methods are proposed. For example, Pilault et al. (2020) first extract relevant sentences and then rewrite them into paper abstracts. Our work is in line with building end-to-end abstractive summarization models for long input. Cohan et al. (2018) design a hierarchical encoder to read different sections separately, and then use combined attentions over words and sections to generate the summary. Multiple agents are created to read segments separately, and then collaboratively write an abstract (Celikyilmaz et al., 2018). However, both work truncates articles to 2K words. Although efficient encoder attentions have been studied in Zaheer et al. (2020) for abstractive summarization, at most 3K tokens can be consumed by their models. Our HEPOS encoder-decoder attention are able to process more than 10K tokens, significantly improving summary informativeness and faithfulness.
We investigate efficient attentions for long document summarization. We propose a novel encoder-decoder attention, HEPOS, based on head-wise positional strides that can effectively identify salient content. Models based on HEPOS attention can process at least twice as many words and produce more informative summaries with less unfaithful errors, according to both automatic evaluation and human evaluation. We further show that our new cloze QA metric better correlates with human judgment than prior faithfulness evaluation metrics.
This research is supported in part by Oracle for Research Cloud Credits, National Science Foundation through Grant IIS-1813341, and by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract # FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. We thank three anonymous reviewers for their valuable suggestions and comments.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.
BioNLP. 2011. Genia event extraction (genia).
BioNLP. 2013. Genia event extraction for nfkb knowledge base.
Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1662-1675.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Rethinking attention with performers.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yue Dong, Andrei Romascanu, and Jackie CK Cheung. 2020. Hiporank: Incorporating hierarchical and positional information into graph-based unsupervised long document extractive summarization. arXiv preprint arXiv:2005.00513.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3938-3948, Minneapolis, Minnesota. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109.
Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of long documents. arXiv: Computation and Language.
Philip John Gorinski and Mirella Lapata. 2015. Movie script summarization as graph-based scene extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1066-1076, Denver, Colorado. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations.
Anastassia Kornilova and Vladimir Eidelman. 2019. *BillSum: A corpus for automatic summarization of US legislation*. In *Proceedings of the 2nd Workshop on New Frontiers in Summarization*, pages 48–56, Hong Kong, China. Association for Computational Linguistics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3744-3753, Long Beach, California, USA. PMLR.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics.
Peter J. Liu*, Mohammad Saleh*, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.