SlowGuess/NAACL_2021_OCR huggingface dataset:
Ramakanth Pasunuru<sup>1</sup> Mengwen Liu<sup>2</sup> Mohit Bansal<sup>1</sup> Sujith Ravi<sup>2</sup> Markus Dreyer<sup>2</sup>  
<sup>1</sup>UNC Chapel Hill  
<sup>2</sup>Amazon Alexa
{ram,mbansal}@cs.unc.edu,{mengwliu,sujithai,mddreyer}@amazon.com
This paper presents an efficient graph-enhanced approach to multi-document summarization (MDS) with an encoder-decoder Transformer model. This model is based on recent advances in pre-training both encoder and decoder on very large text data (Lewis et al., 2019), and it incorporates an efficient encoding mechanism (Beltagy et al., 2020) that avoids the quadratic memory growth typical for traditional Transformers. We show that this powerful combination not only scales to large input documents commonly found when summarizing news clusters; it also enables us to process additional input in the form of auxiliary graph representations, which we derive from the multi-document clusters. We present a mechanism to incorporate such graph information into the encoder-decoder model that was pre-trained on text only. Our approach leads to significant improvements on the Multi-News dataset, overall leading to an average 1.8 ROUGE score improvement over previous work (Li et al., 2020). We also show improvements in a transfer-only setup on the DUC-2004 dataset. The graph encodings lead to summaries that are more abstractive. Human evaluation shows that they are also more informative and factually more consistent with their input documents.<sup>1</sup>
Abstractive Multi-Document Summarization (MDS), the task of writing a consolidated summary of the main information from multiple documents, has seen advancements with the introduction of large-scale datasets and powerful Transformer-based models (Liu et al., 2018; Liu and Lapata, 2019; Fabbri et al., 2019). However, some of the key challenges of MDS include lack of proper inter-document context-aware information, improper logical flow of information, and need
Figure 1: Illustration of our dual-encoder approach to summarizing multi-document clusters with graph encodings. The truncated concatenated text contains the beginnings of each cluster document; the graphs contain information from the full documents.
for external deep context representations. Liu and Lapata (2019) and Li et al. (2020) have addressed the inter-document context modeling to some extent with local and global attention, and document-level similarity graphs. Further, Li et al. (2020) have addressed the later part of using external contextual information (large pre-trained language models, e.g., RoBERTa (Liu et al., 2019)) to improve the performance of MDS models. However, these pre-trained language models are (1) not scalable for long documents because of their encoding length limit and quadratic memory growth; and (2) they do not jointly explore alternate auxiliary information, e.g., semantic graphs derived from multi-document clusters.
Addressing these issues, we present an efficient graph-enhanced approach to multi-document summarization using a pre-trained encoder-decoder Transformer model (Lewis et al., 2019), depicted in Fig. 1, along with an efficient encoding mech
1All our code publicly available at: https://github. com/amazon-research/BartGraphSumm.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4768-4779 June 6-11, 2021. ©2021 Association for Computational Linguistics
anism to encode longer input texts. To this end, we first provide a strong baseline for MDS on the Multi-News dataset (Fabbri et al., 2019) using a pre-trained encoder-decoder model, called BART (Lewis et al., 2019). Next, we incorporate a Longformer-based approach (Beltagy et al., 2020) into the pre-trained BART model, replacing the quadratic memory growth of the full self-attention mechanism with an efficient context window-based attention mechanism that scales the memory linearly w.r.t. the input length. This enables us to encode longer documents than previous work. This efficient encoding mechanism comprises local and global attention mechanisms that address the challenge of modeling inter-document context.
Further, we build consolidated semantic graph representations of the multiple input documents and explore ways to incorporate them into the encoder-decoder model. The semantic graph for a given multi-document cluster is a compact representation of subject-predicate-object triplets (Stanovsky et al., 2018) extracted from the text of the documents; see Fig. 3 for an example. We propose a dual encoding mechanism that separately encodes the regular text of a multi-document cluster and a text representation of its graph. The regular text is encoded by the pre-trained BART encoder, while the graph text is encoded by a transformer encoder that is not pre-trained.
Empirically, we show that our approach (including the ability to use longer parts of the input documents and add auxiliary graph encodings) leads to significant improvements on the Multi-News dataset (achieving state-of-the-art), overall leading to an average 1.8 ROUGE score improvement over previous work (Li et al., 2020). Based on various automatic evaluation metrics, we show that adding graph encodings can help the model abstract away from the specific lexical content of the input and generate summaries that are more abstractive. Further human evaluation shows that they are also more informative and factually more consistent with their input documents. We also test our model with auxiliary graph encodings on the DUC-2004 dataset (Over and Yen, 2004) in a test-only transfer setup, and show that it improves the generalization performance better than a non-graph baseline model. Finally, we present ablations, such as analyzing the effect of input document length on the performance, qualitative analysis of the output summaries, and effect of various graph encoding
approaches on the performance of the MDS system.
Researchers have been interested in automatically summarizing multiple documents since the late 1990s. First works (Mani and Bloedorn, 1997; Radev and McKeown, 1998) cited the gaining popularity of the World Wide Web (WWW) as a motivation for the task. They modeled multi-document collections as graph structures – perhaps influenced by the link structure of the WWW itself. Mani and Bloedorn (1997) summarized pairs of documents by building a graph representation of each and performing graph matching to find salient regions across both documents. Radev and McKeown (1998) summarized multiple documents by mapping them to abstract template representations, then generating text from the templates.
In the early 2000s, datasets from the Document Understanding Conference (DUC), which included human-written summaries for multi-document clusters, sparked increased research interest. In LexRank, Erkan and Radev (2004) extracted the most salient sentences from a multi-document cluster by constructing a graph representing pairwise sentence similarities and running a PageRank algorithm on the graph. Subsequent approaches followed the same paradigm while improving diversity of the extracted sentences (Wan and Yang, 2006) or adding document-level information into the graph (Wan, 2008). Dasgupta et al. (2013) incorporated dependency graph features into their sentence relation graphs. Baralis et al. (2013) built graphs over sets of terms, rather than sentences. Li et al. (2016) built a graph over event mentions and their relationships, in order to summarize news events using sentence extraction techniques. Liu et al. (2015) and Liao et al. (2018) leveraged AMR formalism to convert source text into AMR graphs and then generate a summary using these graphs.
More recently, the introduction of larger datasets for MDS has enabled researchers to train neural models for multi-document summarization. Liu et al. (2018) introduced a large-scale dataset for MDS called WikiSum, based on Wikipedia articles. Liu and Lapata (2019) introduced a hierarchical Transformer model to better encode global and local aspects in multiple documents and showed improvements on WikiSum. Fabbri et al. (2019) introduced an MDS dataset of human-written abstracts from the newser.com website, along with
the source articles that are cited from these abstracts. Further, they also proposed a hierarchical neural model for MDS with an additional Maximal Marginal Relevance (MMR) module that calculates sentence ranking scores based on relevancy and redundancy. Li et al. (2020) further showed the usefulness of pre-trained language models to improve the performance on MDS. However, this approach lacks a pre-trained decoder, and it also limits the document length that can be encoded by the pre-trained language models. In contrast, our work utilizes the pre-trained seq2seq BART (Lewis et al., 2019) model to improve the performance on MDS. We have also incorporated the Longformer-based attention mechanism (Beltagy et al., 2020) into BART model to encode long documents.
To encode graphs into an MDS neural model, Fan et al. (2019) constructed a semantic graph representing key phrases and entities from the documents, as well as their expressed relationships; they used linearized forms of these graphs as inputs to their Transformer model. In contrast, we use dual encoders for encoding both documents text and linearized graph text information. Recently, Li et al. (2020) constructed a similarity graph, topic graph, and discourse graph between input documents and encoded this information directly, rather than in linearized form, into a Transformer. In our work, we build semantic graphs at the sentence level and create a consolidated graph representation by efficiently removing less useful information.
In this section, we first discuss our baseline MDS model utilizing the pre-trained BART sequence-to-sequence model (Lewis et al., 2019). Next, we integrate a Longformer approach (Beltagy et al., 2020) into the BART model for encoding long documents. Finally, we discuss our integration of graph encodings into the BART model.
3.1 BART Baseline
Bidirectional Auto-Regressive Transformer (BART) (Lewis et al., 2019) is a sequence-to-sequence Transformer-based model where the encoder is bi-directional and the decoder is uni-directional. The objective of this model is to reconstruct the actual input from given noisy text input. Input noising strategies include token masking, sentence permutation, document rotation, token deletion, and text infilling. The BART model
Figure 2: Pictorial overview of various attention mechanisms. Each block represents a token. Texture-filled blocks have global self-attention.
is pre-trained on large amounts of text.
To perform multi-document summarization (MDS), we use the pre-trained BART model (trained as described above) and fine-tune it on the MDS datasets. Following Fabbri et al. (2019), we feed cluster documents as a single string joined by a special marker to the BART encoder.
3.2 BART-Long
Recently, the Longformer model (Beltagy et al., 2020) was introduced to allow the pre-trained RoBERTa model (Liu et al., 2019) to encode longer documents than its pre-fixed 512 limit. This is achieved by replacing the traditional full self-attention mechanism (top diagram in Fig. 2) in the Transformers ( $n^2$  memory complexity) with a sparse context window-based attention mechanism which has linear memory in complexity w.r.t. the document length. Further, a small number of tokens are selected to attend over all other tokens, thus creating global attention along with the local context window-based attention (bottom diagram in Fig. 2).
Previously, Longformer has only been explored for pre-trained encoder-only based models, e.g., RoBERTa. In our work, we explore this approach to the pre-trained sequence-to-sequence BART model. We integrate the Longformer including both local and global attention mechanisms, into the BART model, named BART-Long, to encode documents much longer than its maximum token limit of 1024. In order to better encode the information from multiple documents, we incorporate global attention after every sentence and explore various context window sizes for local attention.
(a) Input documents
(b) Graph overview
(c) Linearized graph text
Figure 3: Our graph construction pipeline: (a) Text showing parts of input documents. (b) Overview of graph representation of the information using OIE triplets. (c) Conversion of graph information into text form.
3.3 BART with Graph Encodings
Recently, Fan et al. (2019) converted each multidocument input of the MDS into a graph and then pass the linearized form of this graph as input to a non-pre-trained sequence-to-sequence model, replacing the original text input. In contrast, our work explores the integration of graph encodings into a pre-trained BART model with a separate graph encoder. It is important and also challenging to encode graph representations into the pretrained model while leveraging the pre-existing knowledge from pre-trained models. Moreover, we utilize the BART-Long model described in Sec. 3.2 to avoid the limitation in the input length for encoding both graph and textual information. Next, we describe how we convert multiple input documents into a consolidated graph representation and later describe how we encode this information into an extended BART architecture.
Graph Construction. Following Fan et al. (2019), we perform three steps for constructing a consolidated graph from multiple input documents. First, we do co-reference resolution within each document and extract open information extraction triplets (OIE) at the sentence level from all input documents. Each OIE triplet consists of a subject, a predicate, and an object. Once we have all the triplets, in the second step, we build a graph with subjects and objects as nodes and the predicates as the edge relationship between the nodes. We also calculate the TF-IDF scores for each word in a document. This is useful in identify
ing similar phrases and merging their corresponding nodes in the graph.3 Once we build the graph, we remove the clusters (sub-graphs) with only two nodes, thereby creating a consolidated graph. In the third step, we convert the graph into a linearized form. For this, we traverse sub-graphs in the order of their size, and within each sub-graph we simply start from a node with the highest centrality and move down the sub-graph in a breadth-first search approach to generate linearized text. We concatenate these texts together to form the linearized graph text. Fig. 3 gives an overview of our graph construction approach with examples of linearized graph. Here, we use special tokens like <sub>for subject, <pred> for predicate, <obj> for object, and <cat> for concatenating multiple predicates between a pair of a subject and an object.
Linear Graph Model. Our initial experiments combining both the documents text and linearized graph text into one single input for the BART model gave a slight improvement. To further enable better encoding, we used two encoders: (1) encoding the documents' original text via the pre-trained BART encoder; and (2) encoding the linearized graph text via a new graph encoder, as shown in Fig. 4. Let  $x_{i}$  and  $g_{i}$  represent the tokens at position  $i$  corresponding to the documents text and linearized graph text, respectively. Also, let the corresponding token embeddings be  $e_{i}^{x}$  and  $e_{i}^{g}$ , and the positional embeddings be  $p_{i}^{x}$  and  $p_{i}^{g}$ . Then, the input to the BART encoder  $(x_{i}^{0})$  and graph encoder
2We use AllenNLP (https://allenlp.org/) library for co-reference resolution and extracting OIE triplets.
<sup>3</sup>We define one representative unique string (as a node) from the pool of all matched strings. We manually set the TF-IDF matching threshold to 0.5 based on graph size.
Figure 4: Overview of our approach with BART encoder and a graph encoder. All the Transformer layers use Longformer attention. We use pre-trained representations for BART encoder.
$(g_i^0)$  are:
$$
x _ {i} ^ {0} = e _ {i} ^ {x} + p _ {i} ^ {x}; \quad g _ {i} ^ {0} = e _ {i} ^ {g} + p _ {i} ^ {g}; \qquad (1)
$$
Let the final outputs of the graph encoder with  $M$  Transformer layers be  $g^{M}$ . Let the outputs of the BART encoder after  $K$  Transformer layers be  $x^{K}$ .4 Now, we combine these outputs and give it as a single input to the  $(K + 1)^{th}$  layer of the BART encoder (as shown in Fig. 4). The combined input to  $(K + 1)^{th}$  Transformer layer is defined as:
$$
\hat {x} ^ {K} = \left[ x ^ {K}; g ^ {M} \right] \tag {2}
$$
where  $[;]$  represents the concatenation and  $\hat{x}^K$  represents the input to  $(K + 1)^{th}$  layer (total number of inputs at this layer is equal to the sum of documents text and graph text tokens). Our approach of having separate encoders for graph information could bring the linearized graph text representations closer to that of the pre-trained BART representations.
Multi-News Dataset. The Multi-News dataset (Fabbri et al., 2019) consists of English news articles and the corresponding summaries written by professionals on the newser.com website. The articles in this dataset are curated from a diverse set of news sources (over 1,500 sites). In this work, we use the same splits provided by Fabbri et al. (2019), i.e., 44,972/5,622/5,622 examples for training/validation/test, respectively. Following Fabbri et al. (2019), we truncate  $N$  documents to a total length of  $L$  tokens such that
we choose  $L / N$  tokens from each document and concatenate the truncated documents as input.
DUC-2004 Dataset. The DUC-2004 dataset (Over and Yen, 2004) consists of 50 topics with 10 English documents per topic. Each topic has 4 human-written summaries. In our work, we use this dataset as test-only setup to analyze the transfer skills of our models.
Evaluation Metrics. We evaluate our models via automatic evaluation metrics using ROUGE $^6$  (Lin, 2004), as well as human evaluations of informativeness, coherence, and factual consistency. Following previous work (Fabbri et al., 2019), we report the F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L on Multi-News dataset, and report the F1 scores of ROUGE-1, ROUGE-2, and ROUGESU on DUC-2004 dataset with a 100 word limit. In order to have a fair comparison with previous work, we report summary-level ROUGE-L scores.
Training Details. We tune all our models based on the validation performance. We start with the pre-trained BART large model and fine-tune on the Multi-News dataset.<sup>7</sup> All our new methods are implemented on top of fairseq library.<sup>8</sup> We train each model on 4 Nvidia V100 GPUs. By default, we use Adam optimizer with a learning rate of  $2 \times 10^{5}$  and manually tune in the range:  $[1 \times 10^{5}, 4 \times 10^{5}]$  with 500 warm-up steps. We apply dropout of 0.1 and a label smoothing of 0.1. We perform standard tokenization following previous work (Fabbri et al., 2019) and lowercase both source and target. During inference, we use a minimum decoding length of 50 and a maximum decoding length of 500. For our BART model with Longformer attention, we use a default attention context window size of 128. We train our BART-Long model for 5 epochs which approximately takes 6 hours. For BART-Long-Graph model we train for 8 epochs which approximately takes 8 hours. In terms of total number of trainable parameters, BART-Long has 447 million parameters and BART-Long-Graph has 463 million parameters.<sup>9</sup>
<sup>5</sup>https://duc.nist.gov/duc2004/
<https://pypi.org/project/pyrouge/
7The BART-Large model has 12 Transformer encoders and decoders.
Note that models based on pre-trained RoBERTa also have a similar number of parameters given that RoBERTa has 24 Transformer layers with 1024 hidden size, whereas BART-based models have 12 Transformer layers each on the encoder and decoder sides with 1024 hidden size.
4We set  $K = 1$  and  $M = 1$  in all our experiments.
Model	ROUGE-1	ROUGE-2	ROUGE-L	Average
PREVIOUS WORK
PG-BRNN (Gehrmann et al., 2018)	43.77	15.38	20.84	26.66
HiMAP (Fabbri et al., 2019)	44.17	16.05	21.38	27.20
Flat Transformer	44.32	15.11	20.50	26.64
Hierarchical Transformer (Liu and Lapata, 2019)	42.36	15.27	22.08	26.57
RoBERTa + Transformer Decoder (Li et al., 2020)	44.26	16.22	22.37	27.62
GraphSum (Li et al., 2020)	45.02	16.69	22.50	28.07
GraphSum + RoBERTa (Li et al., 2020)	45.87	17.56	23.39	28.94
OUR MODELS
BART-Long	48.54	18.56	23.78	30.29
BART-Long-Graph (500 tokens graph text)	49.03	19.04	24.04	30.70
BART-Long-Graph (1000 tokens graph text)	49.24	18.99	23.97	30.73
Table 1: Performance of various models on the Multi-News test set. We report the reproduced results of previous works provided by Li et al. (2020). We report 'summary-level' ROUGE-L scores following Fabbri et al. (2019).
In this section, we discuss the performance of various previous works on Multi-News and DUC-2004 datasets, and compare it with our proposed models.
Baseline Results. Table 1 presents the performance of various previous works. First, we report the results of PG-BRNN, HiMAP, and Flat Transformer following Fabbri et al. (2019). Next, we report the results of Hierarchical Transformers, RoBERTa+Transformer Decoder, and the variants of GraphSum models following Li et al. (2020). Some of these previous works use RoBERTa based encoder representations while training MDS models, hence achieving strong results. Note that all these models use 500 tokens for the source input.
BART-Long Results. Table 1 also presents the results on our BART-Long model as described in Sec. 3.2. Our BART-Long model is better than all previous works by a large margin, achieving a new state-of-the-art. This is because of two reasons: (1) The BART model has pre-trained encoder and decoder representations, whereas the previous works have pre-trained encoder-only models such as RoBERTa+Transformer Decoder and GraphSum + RoBERTa; (2) BART model has more number of parameters. $^{10}$  Apart from the performance, our BART-Long model has the advantage to encode longer parts of the input documents more efficiently than the traditional Transformer models or RoBERTa style pre-trained models (more results
on this in Sec. 5.4; Table 4). This is because BART-Long model uses linear memory complexity via its local and global attention mechanism.
BART-Long-Graph Results. The results of our novel graph-based encodings into the BART model are shown in the last two rows of Table 1. Both these models perform statistically significantly better than our strong BART-Long baseline, where the main difference between these two models is the number of tokens used in the graph encoder.[11] Note that we construct our graph using 2,000 tokens of input documents and use 500 or 1,000 tokens of linearized graph text as input along with 500 tokens of input documents text.[12] We further calculated BERTScore (Zhang et al., 2020) for our models, and the F1 scores are 44.06, 44.52, and 44.64 for BART-Long, BART-Long-Graph with 500 tokens of graph, and BART-Long-Graph with 1,000 tokens of graph, respectively. We have also tried pre-training the BART-Long-Graph with the criteria of decoding the original documents' text using noisy input, i.e., by removing some sentences randomly from linearized graph text and documents text. However, we do not see any significant improvement with this approach.
11Our BART-Long-Graph with 500 tokens of graph text is significantly better than BART-Long baseline on all ROUGE metrics with  $p < 0.05$  based on ROUGE script's  $95\%$  confidence interval. Whereas, our BART-Long-Graph with 1,000 tokens of graph text is statistically significantly better on ROUGE-1/2 metrics with  $p < 0.05$ , and it achieves the best average ROUGE score.
$^{12}$ BART-Long with longer inputs perform on par w.r.t. BART-Long-Graph (see Table 4), however, they suffer from generating more extractive summaries, whereas our graph methods generate more abstractive summaries (see Table 8).
<sup>10</sup>Note that RoBERTa+Transformer decoder (Li et al., 2020) also has a similar number of parameters (see training details in Sec. 4).