SlowGuess/NAACL_2021_OCR huggingface dataset:
Luyang Huang  $^{1}$  Shuyang Cao  $^{1}$  Nikolaus Parulian  $^{2}$  Heng Ji  $^{2}$  Lu Wang
$^{1}$ Computer Science and Engineering, University of Michigan, Ann Arbor, MI  $^{2}$ Department of Computer Science, University of Illinois at Urbana-Champaign, IL
1{lyhuang, caoshuy, wangluxy}@umich.edu
2{nnp2, hengji}@illinois.edu
The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose HEPOS, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attention. Combined with HEPOS, we are able to process ten times more tokens than existing models that use full attention. For evaluation, we present a new dataset, GOVREPORT, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.
Long documents, such as scientific papers and government reports, often discuss substantial issues at length, and thus are time-consuming to read, let alone to comprehend. Generating abstractive summaries can help readers quickly grasp the main topics, yet prior work has mostly focused on short texts (containing hundreds of words), e.g., news articles (Gehrmann et al., 2018; Liu and Lapata, 2019; Zhang et al., 2019).
Model training efficiency and summary quality present a pair of challenges for long document summarization. State-of-the-art systems (Lewis et al., 2020; Zhang et al., 2019) are built upon Transformer (Vaswani et al., 2017), which uses attentions to compute pairwise relations between tokens. Such framework has quadratic time and memory complexities, and is too costly for long documents<sup>1</sup>. Solutions have been proposed to reduce
the calculation of encoder self-attentions (Wang et al., 2020c; Zaheer et al., 2020) by selectively attending to neighboring tokens (Beltagy et al., 2020; Child et al., 2019) or relevant words (Kitaev et al., 2020; Tay et al., 2020a). Yet, these methods do not apply to encoder-decoder attentions in summarization models since they collaborate and dynamically pinpoint salient content in the source as the summary is decoded. Truncation is commonly used to circumvent the issue. However, training on curtailed content further aggravates "hallucination" in existing abstractive models (Maynez et al., 2020).
We argue that summarizing long documents (e.g., with thousands of words or more) requires efficient handling of both types of attentions. To this end, we propose an efficient encoder-decoder attention with head-wise positional strides (HEPOS), where the attention heads follow a strided pattern and have varying starting positions. HEPOS reduces computational and memory costs while (1) maintaining the power of emphasizing important tokens, and (2) preserving the global context per head. HEPOS successfully doubles the processed input sequence size, when combined with any encoder. To the best of our knowledge, we are the first to study efficient encoder-decoder attentions and provide a systematic comparison of diverse encoder attentions for the task of summarization.
For evaluation, we collect a new large-scale dataset, GOVREPORT, consisting of about 19.5k U.S. government reports with expert-written abstractive summaries.3 GOVREPORT has two important features: (1) It contains significantly longer documents (9.4k words) and summaries (553 words) than existing datasets, such as PubMed and arXiv (Cohan et al., 2018) (see Table 2); (2) Salient
tokens with a batch size of 1, 70GB of memory is needed for encoder attentions, and 8GB for encoder-decoder attentions.
2Our code is released at https://github.com/ luyang-huang96/LongDocSum.
${}^{3}$  GovREPORT can be downloaded from https:// gov-report-data.github.io.
For instance, to fine-tune BART on documents of 10K
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436 June 6-11, 2021. ©2021 Association for Computational Linguistics
content is spread throughout the documents, as opposed to cases where summary-worthy words are more heavily concentrated in specific parts of the document. These properties make GOVREPORT an important benchmark for producing long document summaries with multiple paragraphs.
We conduct experiments on GOVREPORT and scientific papers in PubMed and arXiv. First, when summarizing documents of the same length, HEPOS attention yields significantly better ROUGE scores than a non-trivial comparison that projects attentions into low-rank space (Wang et al., 2020c). Second, when trained on the same GPU, HEPOS attention, combined with sparse encoder attentions, is able to read more than 10K words and obtains significantly higher ROUGE scores on GOVREPORT and new state-of-the-art results on PubMed, compared with full encoder-decoder attention models which can process at most 5K input words. Human judges further rate the summaries generated by our models to be more informative and faithful.
We further propose a new evaluation metric for faithfulness, inspired by APES (Eyal et al., 2019), a fill-in-the-blank QA metric for summary evaluation. With questions generated from references, our metric,  $\mathrm{APES}_{src}$ , compares QA answers by reading the source and the system summary. It is shown to be better correlated with human judgment than the original metric and an entailment-based scorer (Kryscinski et al., 2020).
The rest of the paper is organized as follows. We describe efficient encoder attentions in prior work in § 2, and formulate our proposed encoder-decoder attention in § 3. The GOVREPORT data is presented in § 4. We then share details on evaluation metrics (§ 5) and experimental results (§ 6). Additional related work is listed in § 7, with conclusion in §8.
2 Prior Work on Efficient Encoder Attentions
Transformer models are built upon multi-head attentions in multiple layers. The attention is calculated as Attention  $(\mathbf{Q},\mathbf{K},\mathbf{V}) = \mathrm{softmax}(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}})\mathbf{V}$  where  $\mathbf{Q}$ ,  $\mathbf{K}$ , and  $\mathbf{V}$  are query, key, and value matrices, each consisting of  $n$  vectors for a document with  $n$  tokens, thus the quadratic memory footprint.
Here, we present an overview of representative methods for efficient encoder self-attention (henceforth "encoder attentions") that can be built upon large pre-trained seq2seq models, e.g., BART (Lewis et al., 2020). We follow the naming
Model	Complexity	# New Para.
Full	O(n2)	—
Encoder Self-attention
I. Fixed Patterns
Sliding Window (2020)	O(nw)	0
Adaptive Span (2019)	O(nw)	O(1)
Global Tokens (2020)	O(2ng)	0
Stride (2019)	O(n2/s)	0
Random (2020)	O(nr)	0
II. Low-rank
Linformer (2020c)	O(nk)	O(n)
III. Learnable Patterns
LSH (2020)	O(lnbl)	0
Sinkhorn (2020a)	O(2nbS)	0
Encoder-decoder Attentions
Heros (ours)	O(mn/sh)	0
Linformer	O(mk)	O(n)
Table 1: Summary of efficient Transformer attentions on memory complexity and newly learned parameters compared with full attentions at each layer.  $m$  and  $n$  are lengths of the input and the output. See §2 and §3 for model-specific hyperparameters.
convention of Tay et al. (2020b), and summarize their memory complexities and numbers of newly learned parameters in Table 1.
2.1 Fixed Patterns
Fixed patterns are used to limit the scope of attentions. In our experiments, in addition to window-based attentions, we also combine them with global tokens, stride patterns, or random attentions.
Sliding window attentions (Beltagy et al., 2020) aim to capture the local context, which is critical for language understanding (Liu* et al., 2018; Child et al., 2019). Concretely, each query token attends to  $w / 2$  neighboring tokens on both left and right, yielding a memory complexity of  $\mathcal{O}(nw)$ .
Adaptive span is proposed by Sukhbaatar et al. (2019) to learn attention windows at different layers. This is implemented by learning a masking function for each head independently. In practice, the adaptive span attention has a complexity of  $\mathcal{O}(nw)$ , where  $\hat{w}$  is the maximum values of predicted spans for all heads. Besides, it introduces  $\mathcal{O}(1)$  new parameters for learning spans.
Global tokens (Beltagy et al., 2020) are often added to sliding windows to let pre-selected tokens attend to the full sequence, to build global representations. Importantly, global attention operations are symmetric, i.e., a global token is also attendable to all tokens in the sequence. We select the first  $g$  tokens as global tokens, as leading sentences are
often important for summarization. Memory complexity is  $\mathcal{O}(2ng)$  due to the symmetric attentions.
Stride patterns are proposed by Child et al. (2019) to capture long term interactions, where each query attends to every  $s$ -th token, with  $s$  as the stride size. It thus has a complexity of  $\mathcal{O}(n^2 / s)$ .
Random attention is motivated by the fact that randomly constructed graphs with  $\tilde{\Theta}(n)$  edges can approximate the complete graphs spectrally (Zaheer et al., 2020). Zaheer et al. (2020) propose to allow each query to attend to  $r$  random keys, resulting in a complexity of  $\mathcal{O}(nr)$ . For efficient implementations, input tokens are first segmented into blocks. Tokens in the same block attend to tokens in another randomly selected block.
2.2 Low-rank Methods
Wang et al. (2020c) show that self-attention matrices are low-rank. They propose Linformer that linearly projects key and value matrices into a low-dimensional space, e.g., from  $n$  to  $k$ , to achieve a  $\mathcal{O}(nk)$  complexity. It also introduces  $\mathcal{O}(n)$  new parameters for projection matrix learning.
2.3 Learnable Patterns
Recently, learnable sparse attentions are proposed to better capture both local and global contexts than attentions based on fixed patterns.
Locality-sensitive hashing (LSH) attentions use a random-projection hashing function to hash similar queries and keys into the same buckets in  $l$  rounds (Kitaev et al., 2020). Attentions are then computed among tokens within each bucket. For bucket size  $b_{l}$ , the complexity of LSH attention is  $\mathcal{O}(lnb_l)$ .
Sinkhorn attentions first segment a sequence into blocks, which are then arranged by a learned Sinkhorn sorting network (Tay et al., 2020a). Given the new permutation, each query attends to  $b_{s}$  tokens within the same block to maintain the local context and another  $b_{s}$  tokens in a neighboring block to capture global interactions. Its complexity is  $\mathcal{O}(2nb_s)$ .
2.4 Other Attentions
We also describe several notable methods that are not suitable for our experiments and excluded from this study: Recurrence over input segments are tailored for an autoregressive decoder only (Dai et al., 2019); memory methods use a separate memory module to attend to full sequences (Lee et al.,
Figure 1: A toy example of our HEPOS attention, with a stride of 2 and four attention heads. Dark colors indicate that heads 1 and 3 attend to the first and third tokens ("Job" and "home") in the input, heads 2 and 4 look at the second and fourth words ("in" and "care").
2019), which share a similar theoretical foundation as global tokens; and kernel methods over attention require training models from scratch (Choromanski et al., 2020; Katharopoulos et al., 2020).
3 Encoder-decoder Attention with Head-wise Positional Strides (Hepos)
The efficient design of encoder-decoder attentions with head-wise positional strides (HEPOS) allows models to consume longer sequences. Concretely, our design is motivated by two observations: (1) Attention heads are redundant (Voita et al., 2019). (2) Any individual head rarely attends to several tokens in a row (Clark et al., 2019). Therefore, as illustrated in Fig. 1, HEPOS uses separate encoder-decoder heads on the same layer to cover different subsets of source tokens at fixed intervals. Each head starts at a different position, and all heads collectively attend to the full sequence.
Given a stride size of  $s_h$ , for the  $h$ -th head, its attention value between decoder query  $\mathbf{q}_j$  (at step  $j$ ) and encoder key vector  $\mathbf{k}_i$  (for the  $i$ -th input token) can be formulated as:
$$
a _ {j i} ^ {h} = \left\{ \begin{array}{l l} \operatorname {s o f t m a x} \left(\mathbf {q} _ {j} \mathbf {k} _ {i}\right), & \text {i f} (i - h) \bmod s _ {h} = 0 \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {1}
$$
In HEPOS attention, each query token attends to  $n / s_h$  tokens per head, yielding a memory complexity of  $\mathcal{O}(mn / s_h)$ , where  $m$  is the output length.
For comparison, Linformer (§ 2.2) can be straightforwardly adapted for encoder-decoder attentions by using decoder queries for attention calculation instead. We do not adapt pattern-based attentions (§ 2.1 and § 2.3), since they rely on local token grouping which makes it difficult to pinpoint salient content.
4 GOVREPORT Dataset
We introduce a new large-scale dataset, GOVREPORT, containing 19,466 long reports published by U.S. Government Accountability Office (GAO) $^{4}$  to fulfill requests by congressional members, and Congressional Research Service (CRS) $^{5}$ , covering researches on a broad range of national policy issues. A human-written summary is provided along with each report. During data collection, we remove boilerplates from crawled files, and keep the section and paragraph structure of the documents and summaries. Additional data cleaning and processing details are in Appendix A.
We obtain 12,228 GAO reports and 7,238 CRS reports of high quality evidenced by human inspection of 200 parsed reports. Collected GAO reports and CRS reports have on average 6.9 and 4.6 sections, respectively. We split train, validation and test set by publication date on each dataset, and end up with 17519 training samples, 974 validation documents, and 973 test samples.
Notably, summaries of GAO reports are written by experts, and are often structured into three aspects in order: "Why GAO did this study"—motivation and problem(s) under discussion, "What GAO found"—findings of the report, and "What GAO recommends"—suggestions and solutions to the problem(s). All but three GAO summaries include "What GAO Found". The percentages of GAO summaries that contain "Why GAO did this study" and "What GAO recommends" are  $94.8\%$  and  $29.0\%$ . For comparison, structured summaries are also observed on PUBMED (Cohan et al., 2018) samples. Though they do not contain explicit aspect labels, the summaries can often be broken down into "Introduction", "Methods", "Results", and "Conclusion" via keyword matching. Details about keyword choices for each aspect are provided in Table 11 in Appendix D.
Comparison with Existing Long Document Summarization Datasets. In Table 2, we compare GOVREPORT with several existing long document summarization datasets, including PUBMED and ARXIV (Cohan et al., 2018) that consist of scientific publications; BILLSUM (Kornilova and Eidelman, 2019), a collection of congressional bills; and BIGPATENT (Sharma et al., 2019), a corpus of
Dataset	# Doc	Summary # word # sent	Doc # word	Comp.	Den.
PUBMED	133,215	202.4	6.8	3049.0	16.2
ARXIV	215,913	272.7	9.6	6029.9	39.8
BILLSUM	23,455	207.7	7.2	1813.0	13.6
BIGPATENT	1,341,362	116.5	3.7	3573.2	36.3
GOVREPORT	19,466	553.4	17.8	9409.4	19.0
Table 2: Statistics of GOVREPORT and existing long document summarization datasets. Comp.: compression ratio, Den.: extractive fragment density (Grusky et al., 2018). All values are mean over the whole dataset except for the "#Doc" column. Documents and summaries in GOVREPORT are significantly longer.
Figure 2: Percentage of unique salient bigrams accumulated from the start to  $\mathrm{X\%}$  of the source. Key information is spread over the documents in GOVREPORT, highlighting the importance of understanding longer text.
U.S. patent documents.
First, documents and summaries in GovReport are significantly longer than prior datasets. Next, we inspect the distribution of summary-worthy bigrams in the source by dividing each document into ten equisized partitions. For each partition, we count the occurrence of unique bigrams that also appear in the reference, accumulated from the start of the document to the end of the partition. Fig. 2 shows that key information is spread throughout documents in GovREPORT, with new salient bigrams being steadily added as more content is consumed. For ARXIV and BIGPATENT, only about  $10\%$  of new salient bigrams are accumulated in the second half of the documents, reflecting the heavy positional bias in these two datasets. In contrast, in GovReport and BILLSUM, more than  $18\%$  of new summary-worthy bigrams appear in the later half of the articles, showing a more even distribution. A similar trend is observed on unigrams. However, BILLSUM has the shortest documents among the five datasets.
4. www.gao.gov
5crsreports.congress.gov
5 Summary Evaluation with Cloze QA
This work aims to evaluate whether processing more text improves both informativeness and faithfulness of abstractive summaries. In addition to ROUGE (Lin, 2004) and human evaluation, we extend existing QA-based metric (Eyal et al., 2019) and consider an entailment-based scorer.
QA-based Evaluation. We present a new faithfulness evaluation metric by extending the APES score (Eyal et al., 2019). We follow APES to construct a set of cloze questions,  $\{q\}$ , from each reference summary by masking entities. Events, dates, and numbers are also masked, as they are prevalent in our data. Each masked phrase becomes the gold-standard answer  $a_{ref}$  for a question  $q$ . We do not generate natural language questions (Durmus et al., 2020; Wang et al., 2020a), due to the lack of accurate question generation models for the domains of government reports and scientific papers.
QA models are trained by reading a question and a context to label the answer span in the context. We construct context by greedily selecting sentences that maximize the improvement of ROUGE-2 recall when compared with the reference summary. If the answer  $a_{ref}$  cannot be found in the context, the sample is excluded from training. We train all QA models by fine-tuning BERT (Devlin et al., 2019) to predict the answer span.
To evaluate the faithfulness of a system summary, APES uses the QA model to read the summary and a question  $q$  to label an answer  $a_{sys}$ . It calculates a unigram F1 score by comparing  $a_{sys}$  and  $a_{ref}$ . Different from APES, we further use the QA model to read the context (sentences selected from the source) and give an answer  $a_{cxt}$  to the question  $q$ . We compute a unigram F1 by comparing  $a_{sys}$  and  $a_{cxt}$ , denoted as  $\mathbf{APES}_{src}$ . Given that existing summarization models rarely rewrite names or numbers correctly, our metric can better capture faithfulness by using a gold-standard answer constructed from the source article than from the human-written abstract.
To extract entities and events, we deploy a state-of-the-art IE framework, OneIE (Lin et al., 2020) on GOVREPORT. On PubMed, we retrain OneIE on Genia 2011 (BioNLP, 2011) and 2013 (BioNLP, 2013), and PubMed (Wei et al., 2019) datasets to extract domain-specific entities and events, such as entities of Gene and Disease. We additionally include numbers and dates extracted by spaCy (Honnibal and Montani, 2017).
Entailment-based Evaluation. We further consider FactCC (Kryscinski et al., 2020), which evaluates factual consistency of a system summary by predicting an entailment score between the source and the summary. We reproduce their method on our datasets.
Additional details for implementing the evaluation models and the entity extraction models are given in Appendix B.
In this section, we start with describing training details in § 6.1. We then compare attention variants on documents of the same length (§ 6.2) and study whether reading more text can generate more informative summaries (§ 6.3). We further report human evaluation on summary informativeness and faithfulness as well as automatic faithfulness scores (§ 6.4). Finally, we investigate whether automatic metrics correlate with human judgment (§ 6.5).
We fine-tune BART (Lewis et al., 2020) for all experiments. We implement our models with PyTorch (Paszke et al., 2019) and Fairseq (Ott et al., 2019). Additional position embeddings are initialized randomly for models that handle longer inputs. The learning rate is set to  $1 \times 10^{-4}$  and learning rate warm-up is applied for the first 10,000 steps. Adafactor (Shazeer and Stern, 2018) optimizer with a gradient clipping of 0.1 is used. All models are trained on two Quadro RTX 6000 GPUs with 24GB memory or one Quadro RTX 8000 with 48GB memory. We set a batch size of 2 per step and accumulate gradient every 32 steps. During test, we adopt a beam size of 4 and a length penalty of 2 (Wu et al., 2016) on all datasets.
6.2 Comparing Attention Variants
Comparisons. We first experiment with articles that are all truncated at 1024 tokens. For encoder attentions, we consider the following variants: (1) sliding WINDOW; (2) adaptive span (ADASpan); (3) GLOBAL tokens; (4) STRIDE; (5) RANDOM tokens; (6) Linformer (LIN.); (7) locality sensitive hashing (LSH); and (8) SINKHORN. We ensure models are comparable by setting hyperparameters to satisfy  $w = \hat{w} = k = lb_{l} = 2b_{s} = 256$ , so that models have similar memory complexity. For LSH attentions, we select  $l = 4$  rounds of hashing. Following prior work (Zaheer et al.,
System	GovReport (new)	PubMed
R-1	R-2	R-L	R-1	R-2	R-L
FULL	52.83	20.50	50.14	45.36	18.74	40.26
Encoder variants w/ full enc-dec attn.
I. Fixed Patterns
WINDOW	50.78	18.59	48.10	42.74	16.83	37.96
+ GLOBAL	51.24	19.01	48.58	43.44	17.07	38.55
+ STRIDE	51.53	19.14	48.68	43.73	17.25	38.82
+ RANDOM	51.49	18.90	48.75	43.38	16.87	38.45
ADASpan	50.76	18.69	48.13	43.42	17.16	38.60
+ GLOBAL	50.33	18.56	47.80	43.24	17.01	38.42
+ STRIDE	51.56	19.19	48.57	43.71	17.25	38.76
+ RANDOM	51.39	18.89	48.74	43.28	16.87	38.45
II. Low-Rank Methods
LIN.	50.70	18.48	47.85	43.65	17.12	38.71
III. Learnable Patterns
LSH	51.95	19.36	48.85	44.74	18.07	39.76
SINKHORN	53.00*	20.05*	50.25*	45.10	18.40*	40.11*
Enc-dec variants w/ full encoder attn.
LIN.	47.79	14.93	45.15	45.16	17.66	40.25
HEPOS (ours)	51.05*	19.44*	48.51*	45.80*	18.61*	40.69*
Enc-dec variants w/ Sinkhorn encoder attn.
LIN.	42.90	12.86	40.32	44.84	17.65	39.98
HEPOS (ours)	51.34*	19.09*	48.73*	44.85	18.19*	39.91
2020), we combine GLOBAL, STRIDE, and RANDOM with WINDOW and ADASpan, where we set  $g = n^2 / s = r = 128$  for a fair comparison. We adapt Linformer to encoder-decoder attentions to compare with HEPOS, where we use  $s_h = n / k = 4$  for all experiments. Finally, we report results using FULL, i.e., the original, encoder and encoder-decoder attentions.
Results. Among all encoder variants, learnable patterns perform the best, approaching the performance of full attentions on both GovReport and PubMed, as shown in Table 3. Within learnable patterns, Sinkhorn attention consistently obtains better ROUGE scores. Moreover, combining techniques in fixed patterns is more effective than simply using window-based sparse attentions, though with an increased memory cost.
For encoder-decoder attentions, HEPOS consistently yields higher ROUGE scores than Linformer on both datasets, using either full or Sinkhorn encoder. Notably, coupled with a Sinkhorn attention, our model's performance matches the variant using