SlowGuess/NAACL_2021_OCR huggingface dataset:
Model	R-1	R-2	R-SU
EXTRACTIVE METHODS
MMR (Carbonell and Goldstein, 1998)	30.14	4.55	8.16
LexRank (Erkan and Radev, 2004)	35.56	7.87	11.86
TextRank (Mihalcea and Tarau, 2004)	33.16	6.13	10.16
ABSTRACTIVE METHODS TRAINED ON CNN/DAILY MAIL
Copy-Transformer (Gehrmann et al., 2018)	28.54	6.38	7.22
PG-BRNN (Gehrmann et al., 2018)	29.47	6.77	7.56
Hi-MAP (Fabbri et al., 2019)	35.78	8.90	11.43
PG-MMR (Lebanoff et al., 2018)	36.42	9.36	13.23
OUR ABSTRACTIVE METHODS TRAINED ON MULTI-NEWS
BART-Long (500 text tokens)	33.82	8.09	10.53
BART-Long-Graph (500 text tok. + graph)	34.72	7.97	11.04
Table 2: ROUGE scores on DUC-2004 test-only setup.
Informative	Coherent
BART-Long is better	17.0%	24.0%
BART-Long-Graph is better	25.5%	20.0%
None	57.5%	56.0%
5.2 Transfer Results on DUC-2004
We also evaluate our proposed models in a test-only transfer setup using the DUC-2004 multi-document summarization dataset. Table 2 presents the results on this dataset comparing our models with previous works. Our models perform better than some of the extractive summarization methods (TextRank and MMR). However, some of the previous works perform better than our models, but we cannot strictly compare with them since they are trained on CNN/Daily Mail dataset.[13] Comparing our baseline model and our model with graph encodings, we observe that graph encodings help improve the performance by 0.9 on ROUGE-1 and 0.5 on ROUGE-SU. This suggests that graph information is useful in transfer setups as well.
We conduct a human evaluation on Amazon MTurk to analyze the effect of adding graph input to the BART-Long model (setup details in Appendix B).
Informativeness and coherence: To evaluate how graph encodings impact informativeness and coherence of the generated summaries, we show human annotators pairs of summaries from the BART-Long model and the BART-Long-Graph model and ask them to indicate which one is more informative
Table 3: Human evaluation of informativeness and coherence of generated summaries.
Model	Input Length	R-1	R-2	R-L	Avg.
BART	500	49.22	18.88	23.88	30.66
BART-Long	500	48.54	18.56	23.78	30.29
BART-Long	1000	49.15	19.50	24.47	31.04
BART-Long	1500	48.79	19.14	24.16	30.70
BART-Long	2000	48.96	19.34	24.37	30.89
Table 4: Performance of BART models at various input lengths on the Multi-News dataset.
and which one is more coherent; definitions are listed in the Appendix B. There is also an option for choosing None. The summaries are labeled as A and B using random permutation; we also show the target summary from the test set for reference. We obtain judgments from two annotators on 200 examples from the Multi-News test set. Table 3 shows the results; None represents all cases where either both annotators picked None or the two annotators did not give the same answer. We observe that BART-Long-Graph summaries were picked as more informative by both judges  $25.5\%$  of the time, compared to  $17\%$  for the BART-Long model. The results are closer for coherence, with a slight disadvantage for the BART-Long-Graph model. We hypothesize that using graph information, which has a different structure than natural text, makes the summary less coherent.
Factual consistency: We evaluate factual consistency by highlighting single summary sentences and asking the annotators if it is consistent with the input articles. We ask three annotators to judge the factual consistency of the highlighted summary w.r.t. the articles on 200 outputs per model. For the BART-Long-Graph model,  $72\%$  of the summaries are judged as factually consistent by two or more annotators, compared to  $68\%$  for the BART-Long model. Frequently, news sources are hallucinated, e.g., "as reported by TMZ". This error accounts for 18 of the 136 errors of the BART-Long-Graph model and 19 of the 144 errors of the BART-Long model. More details in Appendix B.
5.4 Ablations and Analyses
What is the effect of input documents length over the performance? Table 4 presents the performance comparison of BART with Longformer (BART-Long) over different input lengths. At the same input length, BART-Long performance is slightly lower than the BART model without Longformer attention, i.e., using full self-attention. This is expected as we replace the full self-attention with
<sup>13</sup>If we compare these models (e.g., Hi-MAP) on MultiNews dataset, our models perform much better (see Table 1).
Window Size	R-1	R-2	R-L	Avg.
32	48.39	18.75	23.68	30.27
64	48.93	19.28	24.25	30.82
128	48.96	19.34	24.37	30.89
256	49.47	19.94	24.82	31.41
512	49.43	19.86	24.77	31.35
Table 5: Performance of BART-Long model at various attention window sizes. We use Multi-News dataset with 1000 text tokens as input in this comparison to consider longer context window sizes.
Model	R-1	R-2	R-L	Avg.
BART-Long (BL)	48.54	18.56	23.78	30.29
BL-Graph-Only	44.91	14.19	20.46	26.52
BL-Graph-Concat	48.85	18.78	23.79	30.47
BL-Separate-Graph	49.03	19.04	24.04	30.70
local and global attention with lower memory footprint. More importantly, BART-Long can encode longer documents and can achieve better results which is evident from the results in Table 4. $^{14}$  Overall, we observe that the best results are achieved at a document length of 1,000 tokens, and no further improvement for any input length greater than that.
What is the effect of attention context window size over the performance? We also compare the effect of various attention context window sizes in the local attention mechanism of BART-Long model over the summarization performance. Table 5 presents such ablation with attention context window sizes of 32, 64, 128, 256, and 512, on the Multi-News dataset with 1,000 tokens input.15 Here, we observe that performance linearly improves till certain context window size and then stays more or less similar. Note that in Table 1 we use an attention context window size of 128 to trade-off between memory and performance.
Different approaches of graph encodings. Table 6 presents the results on various graph encoding methods. First, we replace the original input with linearized graph text and we observe a significant drop in the performance ('BL-Graph-Only'; second row in Table 6). This suggests that documents' text as input is very important to achieve good results.
Table 6: Performance of various graph encoding methods. We use 500 tokens of graph text.
Target Graph	R-1	R-2	R-L	Avg.
0 %	49.03	19.04	24.04	30.70
25 %	49.99	20.66	24.93	31.86
50 %	54.17	27.30	29.79	37.09
75 %	53.70	28.09	30.28	37.36
100 %	61.32	39.15	38.66	46.38
Table 7: Ablation of the performance of BART-Long-Graph model with 500 tokens input graph text over additionally using a varying percentage of target summary graph information.
Model	Density	LCS(%)	4-gr (%)
BL (2,000 text tokens)	15.5	68.8	55.0
BL (1,000 text tokens)	13.6	66.8	50.7
BL (500 text tokens)	11.6	62.5	44.4
BL (500 text tok. + graph)	10.0	59.3	41.3
Reference summaries	5.0	45.9	17.9
Table 8: Abstractiveness: Measuring lexical overlap between summaries and their inputs; lower numbers mean higher abstractiveness. Adding graphs increases abstractiveness.
Next, we concatenate the documents' text with linearized graph text and give it has input to the BART model ('BL-Graph-Concat') which achieves slightly better results over the baseline. However, when we add the linearized graph text as a separate graph encoder ('BL-Separate-Graph'; same as our 'BART-Long-Graph' model in Table 1), we achieve the best results.
How abstractive are the summaries? Abstractive summarizers generate surprisingly extractive summaries, copying large fragments unmodified from the input documents into the summaries (Webber et al., 2018; Pilault et al., 2020). We hypothesize that providing graph representations of the input can help the model abstract away from the specific lexical content of the input and generate summaries that are more abstractive. Table 8 shows the lexical overlap between the summaries and their inputs when truncating the input documents to different numbers of words, and when adding a graph representation of the input (truncated to 1k graph tokens). Density measures the expected length of the extractive fragment that any randomly chosen summary word belongs to (Grusky et al., 2018);  $\mathrm{LCS}(\%)$  is the length of the longest common subsequence divided by the length of the summary; and  $4\text{-gr} (\%)$  is the proportion of 4-grams in the summaries that are extracted from the input. We
$^{14}$ Encoding more than 500 tokens is not feasible with full self-attention BART model due to high memory requirements.  ${ }^{15}$  We use 1,000 tokens instead of 500 tokens in this setup to effectively compare the 512 window size, otherwise it would be same as full self-attention with 500 tokens input.
BART-Long	Microsoft's acquisition of Nokia's cell phone business is a big step in the company's evolution into a devices and services company, but it's not going to be the Apple of the mobile world. Nokia, which had a 35% market share of the cell phone market in 2003, made an operating profit of 5.48 billion Euros that year, according to the Wall Street Journal, but today's sale price for the company — which includes 1.65 billion Euros in patents — is just 5.44 billion Euros. The acquisition is part of a larger effort by Microsoft to "move further away from the moribund world of the beige desktop and towards the sunlit world of smartphones and tablets ," writes Chris Cillizza at Forbes. "Owning the desktop (via windows) and building additional services on top, like office or search, has been vital for Microsoft's strategy until now," He writes. " As our interest shifts from the desktop to the tablet or smartphone, it'll be essential to Microsoft that it has a presence in the smartphone and tablet market." Nokia will continue to operate under the Nokia brand, but the company will be renamed Microsoft mobile.
BART-Long-Graph	Microsoft is buying Nokia's cell phone business for $8.5 billion, a price tag that includes $1.65 billion in patents, reports the Wall Street Journal. The move is part of Microsoft's plan to shift away from the desktop and toward smartphones and tablets, and the journal sees the move as "the latest acceleration of that strategy — to move further away from a moribund world of the beige desktop and towards the sunlit world of smartphones and tablet." Nokia has been in trouble for a while now, notes TechCrunch, but the deal is a sign that the company is finally ready to move on from its mobile roots. "The acquisition of Nokia is the right move for Microsoft ," says one analyst. "It's a step in the right direction." But the journal notes that the move could complicate Apple's plans to buy Nokia, which it has been working on for some time.
Table 9: Examples of two summaries generated from the same input by the BART-Long and BART-Long-Graph models. Long extractive fragments are marked in red, shorter ones in orange and yellow. Summaries are recapitalized and de-tokenized for better readability.
observe that longer text inputs make summaries more extractive, while adding a graph makes summaries more abstractive.[16]
Would better graph leads to better improvements? In order to answer how good is our graph construction approach, we choose to convert the target summary into a graph and use its linearized text as input to the model along with the original input documents' text and its linearized graph text. Table 7 presents such ablation where we linearly increase the amount of target graph information given as input, and we observe that using more target graph information leads to better performance. This suggests that a better way of including more salient information in the graph construction process could lead to a better summarization model.[17]
Qualitative analysis of output summaries. Table 9 presents generated summaries from two models, BART-Long and BART-Long-Graph. Both examples have the misattribution of source error as mentioned in Sec.5.3, motivating the need to improve factual consistency of abstractive summaries. The overlapped  $n$ -grams between the summary and the original source articles are highlighted in colors. Yellow and red stand for shorter and longer  $n$ -gram overlap, respectively. The visualizations show that
BART-Long-Graph produces more abstractive summaries, as shown in Table 8, due to the fact that it incorporates triplet-based information that abstract away from the surface of the source articles.
Extra Ablations. We provide graph visualization of input documents in Appendix A.
We presented an efficient graph-enhanced approach to MDS that achieves state-of-the-art results on the Multi-News dataset using the pre-trained encoder-decoder Transformer model along with an efficient encoding mechanism. We also show improvements in a transfer-only setup on the DUC-2004 dataset. The graph encodings lead to summaries that are more abstractive. Human evaluation shows that they are also more informative and factually more consistent with their input documents. Finally, we present extensive ablations to better understand the usefulness of our method.
We thank the anonymous reviewers for their helpful comments. This work was partially supported by NSF-CAREER Award 1846185, an Amazon ML Research Award, and a Microsoft PhD Fellowship.
<sup>16</sup>We observe similar trends when we add graph to longer or shorter text inputs.
17We 'randomly' choose  $x\%$  of the target graph.
Elena Baralis, Luca Cagliero, Naeem Mahoto, and Alessandro Fiori. 2013. GraphSum: Discovering correlations among multiple terms for graph-based summarization. Information Sciences, 249:96-109.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335-336.
Anirban Dasgupta, Ravi Kumar, and Sujith Ravi. 2013. Summarization through submodularity and dispersion. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1014-1022.
Günes Erkan and Dragomir R Radev. 2004. LexRank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457-479.
Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019. Multi-News: a large-scale multi-document summarization dataset and abstractive hierarchical model. In ACL.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale seq2seq models to multi-document inputs. In EMNLP.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In EMNLP.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In *NAACL*.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131-4141.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Wei Li, Lei He, and Hai Zhuge. 2016. Abstractive News Summarization based on Event Semantic Link Network. In Proc. of COLING, pages 236-246.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summarization. In ACL.
Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract meaning representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1178-1190.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A Smith. 2015. Toward abstractive summarization using semantic representations. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1077-1086.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In ICLR.
Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. In Proc. of ACL.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Inderjeet Mani and Eric Bloedorn. 1997. Multi-document summarization by graph search and matching. In Proceedings of the National Conference on Artificial Intelligence, pages 622-628. AAAI.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Paul Over and James Yen. 2004. An introduction to duc-2004. In Proceedings of the 4th Document Understanding Conference (DUC 2004).
Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9308-9319, Online. Association for Computational Linguistics.
Dragomir R Radev and Kathleen R McKeown. 1998. Generating natural language summaries from multiple on-line sources. Computational Linguistics, 24(3):470-500.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised Open Information Extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 885-895, Stroudsburg, PA, USA. Association for Computational Linguistics.
Xiaojun Wan. 2008. An exploration of document impact on graph-based multi-document summarization. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 755-762.
Xiaojun Wan and Jianwu Yang. 2006. Improved Affinity Graph Based Multi-Document Summarization. In Proc. of NAACL (short papers), pages 181-184.
Noah Weber, Leena Shekhar, Niranjan Balasubramanian, and Kyunghyun Cho. 2018. Controlling decoding for more abstractive summaries with copy-based networks. arXiv preprint arXiv:1803.07038.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In ICLR.
A Extra Ablations
Qualitative Analysis of Graphs. Each graph in Fig. 5 corresponds to an example in the Multi-News dataset with multiple input documents, where we convert them into a graph using our graph construction process as described in Sec. 3.3. We observe that these graphs are highly connected forming only a few clusters. We further remove clusters with only two nodes to create a very consolidated graphs. It is also worth noting that input documents with a total of 2,000 tokens can be represented with less than 100 nodes (and their corresponding relations).
B Details on the Mechanical Turk Setup
For all our evaluations on Mechanical Turk (see Sec. 5.3 of the main paper), we first set up a short qualification test that can be taken by any worker from a country whose main language is English, who has completed 100 or more HITs so far with an acceptance rate of  $95\%$  or higher. The qualification test consists of just three questions from our factual consistency setup; two of which must be answered correctly, along with an explanation text (5 words or more) to explain when "not factually consistent" was chosen.  $53\%$  of workers who start the test provide answers to all three questions, and  $27.6\%$  of these answers at least two correctly
and provide a reasonable explanation text, i.e., only \(14.6\%\) of the test takers are granted the qualification. The qualification enables workers to work on our factual consistency HITs as well as our HITs judging informativeness and coherence. The rate per HIT differs widely between the two tasks, as the factual consistency task can be done quickly, given the fact that a single summary sentence is evaluated, which is often extractive, and the related sentences in the article are highlighted. The factual consistency task pays \\(0.07 per hour with a bonus of \\)0.03; the informativeness and coherence task pays \$0.25 per hour with a bonus of \$0.50. Overall, this amounts to an average pay of \\)12.50, incl. the bonus. The bonus is paid to workers who spend at least 10 seconds per HIT for the factual consistency task and 60 seconds per HIT for the informativeness and coherence task and who give short explanation texts for their decisions.
We give the following guidelines on deciding which summary is more informative or more coherent, respectively:
- Informativeness: The more informative summary is better at expressing the main points of the news story. It contains information that is more relevant and important. It has fewer unimportant details. Its content is more similar to the human-written summary.
- Coherence: The more coherent summary has better structure and flow, is easier to follow. The facts are presented in a more logical order.
(a) Example Graph-1
(b) Example Graph-2
Figure 5: Visualization of input documents in a graph form, where nodes represent subject or object phrases and edges represent the relationship between them.
(c) Example Graph-3