SlowGuess/ACL_2020_OCR huggingface dataset:
Model	R-1	R-2	R-L
LEAD-3 (See et al., 2017)	40.34	17.70	36.57
ORACLE (Liu and Lapata, 2019b)	52.59	31.24	48.87
REFRESH (Narayan et al., 2018)	40.00	18.20	36.60
LATENT (Zhang et al., 2018)	41.05	18.77	37.54
BanditSum (Dong et al., 2018)	41.50	18.70	37.60
NeuSUM (Zhou et al., 2018)	41.59	19.01	37.98
JECS (Xu and Durrett, 2019)	41.70	18.50	37.90
LSTM+PN (Zhong et al., 2019a)	41.85	18.93	38.13
HER w/o Policy (Luo et al., 2019)	41.70	18.30	37.10
HER w Policy (Luo et al., 2019)	42.30	18.90	37.60
Ext-BiLSTM	41.59	19.03	38.04
Ext-Transformer	41.33	18.83	37.65
HSG	42.31	19.51	38.74
HSG + Tri-Blocking	42.95	19.76	39.23
Table 1: Performance (Rouge) of our proposed models against recently released summarization systems on CNN/DailyMail.
5.1 Single-document Summarization
We evaluate our single-document model on CNN/DailyMail and NYT50 and report the unigram, bigram and longest common subsequence overlap with reference summaries by R-1, R-2 and R-L. Due to the limited computational resource, we don't apply pre-trained contextualized encoder (i.e. BERT (Devlin et al., 2018)) to our models, which we will regard as our future work. Therefore, here, we only compare with models without BERT for the sake of fairness.
Results on CNN/DailyMail Table 1 shows the results on CNN/DailyMail. The first part is the LEAD-3 baseline and ORACLE upper bound, while the second part includes other summarization models.
We present our models (described in Section 4.3) in the third part. Compared with ExtBiLSTM, our heterogeneous graphs achieve more than  $0.6 / 0.51 / 0.7$  improvements on R-1, R-2 and R-L, which indicates the cross-sentence relationships learned by our sentence-word-sentence structure is more powerful than the sequential structure. Besides, Our models also outperform ExtTransformer based on fully connected relationships. This demonstrates that our graph structures effectively prune unnecessary connections between sentences and thus improve the performance of sentence node classification.
Compared with the second block of Figure 1, we observe that HETERSUMGRAPH outperforms all previous non-BERT-based summarization systems
3The detailed experimental results are attached in the Appendix Section.
6214
and trigram blocking leads to a great improvement on all ROUGE metrics. Among them, HER (Luo et al., 2019) is a comparable competitor to our HETERSUMGRAPH, which formulated the extractive summarization task as a contextual-bandit problem and solved it with reinforcement learning. Since the reinforcement learning and our trigram blocking plays a similar role in reorganizing sentences into a summary (Zhong et al., 2019a), we additionally compare HER without policy gradient with HETERSUMGRAPH. Our HETERSUMGRAPH achieve 0.61 improvements on R-1 over HER without policy for sentence scoring, and HETERSUMGRAPH with trigram blocking outperforms by 0.65 over HER for the reorganized summaries.
Model	R-1	R-2	R-L
First sentence (Durrett et al., 2016)	28.60	17.30	-
First k words (Durrett et al., 2016)	35.70	21.60	-
LEAD-3	38.99	18.74	35.35
ORACLE	60.54	40.75	57.22
COMPRESS (Durrett et al., 2016)	42.20	24.90	-
SUMO (Liu et al., 2019)	42.30	22.70	38.60
PG* (See et al., 2017)	43.71	26.40	-
DRM (Paulus et al., 2017)	42.94	26.02	-
Ext-BiLSTM	46.32	25.84	42.16
Ext-Transformer	45.07	24.72	40.85
HSG	46.89	26.26	42.58
HSG + Tri-Blocking	46.57	25.94	42.25
Table 2: Limited-length ROUGE Recall on NYT50 test set. The results of models with * are copied from Liu and Lapata (2019b) and '-' means that the original paper did not report the result.
Results on NYT50 Results on NYT50 are summarized in Table 2. Note that we use limited-length ROUGE recall as Durrett et al. (2016), where the selected sentences are truncated to the length of the human-written summaries and the recall scores are used instead of F1. The first two lines are baselines given by Durrett et al. (2016) and the next two lines are our baselines for extractive summarization. The second and third part report the performance of other non-BERT-based works and our models respectively.
Again, we observe that our cross-sentence relationship modeling performs better than BiLSTM and Transformer. Our models also have strong advantages over other non-BERT-based approaches on NYT50. Meanwhile, we find trigram block doesn't work as well as shown on CNN/DailyMail, and we attribute the reason to the special formation
of summaries of CNN/DailyMail dataset. 4
Ablation on CNN/DailyMail In order to better understand the contribution of different modules to the performance, we conduct ablation study using our proposed HETERSUMGRAPH model on CNN/DailyMail dataset. First, we remove the filtering mechanism for low TF-IDF words and the edge weights respectively. We also remove residual connections between GAT layers. As a compensation, we concatenate the initial sentence feature after updating messages from nearby word nodes in Equal 8:
$$
\mathbf {H} _ {s} ^ {1} = \operatorname {F F N} \left(\left[ \mathbf {U} _ {s \leftarrow w} ^ {1}; \mathbf {H} _ {s} ^ {0} \right]\right). \tag {13}
$$
Furthermore, we make iteration number  $t = 0$  which deletes the word updating and use the sentence representation  $\mathbf{H}_s^1$  for classification. Finally, we remove the BiLSTM layer in the initialization of sentence nodes.
As Table 3 shows, the removal of low TF-IDF words leads to increases on R-1 and R-L but drops on R-2. We suspect that filtering noisy words enable the model to better focus on useful word nodes, at the cost of losing some bigram information. The residual connection plays an important role in the combination of the original representation and the updating message from another type of nodes, which cannot be replaced by the concatenation. Besides, the introduction of edge features, word update and BiLSTM initialization for sentences also show their effectiveness.
5.2 Multi-document Summarization
We first take the concatenation of the First-k sentences from each source document as the baseline and use the codes and model outputs<sup>5</sup> released by Fabbri et al. (2019) for other models.
To explore the adaptability of our model to multi-document summarization, we concatenate multisource documents to a single mega-document and apply HETERSUMGRAPH as the baseline. For comparison, we extend HETERSUMGRAPH to multi-document settings HETERSDOCSUMGRAPH
${}^{4}$  Nallapati et al. (2016) concatenate summary bullets, which are written for different parts of the article and have few overlaps with each other, as a multi-sentence summary. However, when human write summaries for the whole article (such as NYT50 and Multi-News), they will use key phrases repeatedly. This means roughly removing sentences by n-gram overlaps will lead to loss of important information.
$^{5}$ https://github.com/Alex-Fabbri/Multi-News
6215
Model	R-1	R-2	R-L
HSG	42.31	19.51	38.74
- filter words	42.24	19.56	38.68
- edge feature	42.14	19.41	38.60
- residual connection	41.59	19.08	38.05
- sentence update	41.59	19.03	38.04
- word update	41.70	19.16	38.15
- BiLSTM	41.70	19.09	38.13
as described in Section 3.5. Our results are presented in Table 4.
Specifically, we observe that both of our HETERSUMGRAPH and HETERDOCSUMGRAPH outperform previous methods while HETERDOCSUMGRAPH achieves better performance improvements. This demonstrates the introduction of document nodes can better model the document-document relationships and is beneficial for multi-document summarization. As mentioned above, trigram blocking does not work for the Multi-News dataset, since summaries are written as a whole instead of the concatenations of summary bullets for each source document.
Table 3: Ablation studies on CNN/DailyMail test set. We remove various modules and explore their influence on our model. '-' means we remove the module from the original HETERSUMGRAPH. Note that HETERSUMGRAPH without the updating of sentence nodes is actually the Ext-BiLSTM model described in Section 4.3.
Model	R-1	R-2	R-L
First-1	25.44	7.06	22.12
First-2	35.70	10.28	31.71
First-3	40.21	12.13	37.13
ORACLE	52.32	22.23	47.93
LexRank* (Erkan and Radev, 2004)	41.77	13.81	37.87
TextRank* (Mihalcea and Tarau, 2004)	41.95	13.86	38.07
MMR* (Carbonell and Goldstein, 1998)	44.72	14.92	40.77
PG† (Lebanoff et al., 2018)	44.55	15.54	40.75
BottomUp† (Gehrmann et al., 2018)	45.27	15.32	41.38
Hi-MAP† (Fabbri et al., 2019)	45.21	16.29	41.39
HSG	45.66	16.22	41.80
HSG + Tri-Blocking	44.92	15.59	40.89
HDSG	46.05	16.35	42.08
HDSG + Tri-Blocking	45.55	15.78	41.29
Table 4: Results on the test set of Multi-News. We reproduce models with  $*$  via the released code and directly use the outputs of  $\dagger$  provided by Fabbri et al. (2019) for evaluation.
5.3 Qualitative Analysis
We further design several experiments to probe into how our HETERSUMGRAPH and HETERDOC
Figure 4: Relationships between the average degree of word nodes of the document (x-axis) and  $\tilde{R}$ , which is the mean of R-1, R-2 and R-L (lines for left y-axis), and between  $\Delta \tilde{R}$ , which is the delta  $\tilde{R}$  of HETERSUMGRAPH and Ext-BiLSTM (histograms for right y-axis).
SUMGRAPH help the single- and multi-document summarization.
Degree of word nodes In HETERSUMGRAPH, the degree of a word node indicates its occurrence across sentences and thus can measure the redundancy of the document to some extent. Meanwhile, words with a high degree can aggregate information from multiple sentences, which means that they can benefit more from the iteration process. Therefore, it is important to explore the influence of the node degree of words on the summarization performance.
We first calculate the average degree of word nodes for each example based on the constructed graph. Then the test set of CNN/DailyMail is divided into 5 intervals based on it (x-axis in Figure 4). We evaluate the performance of HETERSUM-GRAPH and Ext-BiLSTM in various parts and the mean score of R-1, R-2, R-L is drawn as lines (left y-axis  $\tilde{R}$ ). The ROUGE increases with the increasing of the average degree of word nodes in the document, which means that articles with a high redundancy are easier for neural models to summarize.
To make  $\Delta \tilde{R}$  between models more obvious, we draw it with histograms (right y-axis). From Figure 4, we can observe that HETERSUMGRAPH performs much better for documents with a higher average word node degree. This proves that the benefit brought by word nodes lies in the aggregation of information from sentences and the propagation of their global representations.
Number of source documents We also investigate how the number of source documents influences the performance of our model. To this end,
6216
Figure 5: Relationship between number of source documents (x-axis) and  $\tilde{R}$  (y-axis).
we divide the test set of Multi-News into different parts by the number of source documents and discard parts with less than 100 examples. Then, we take First-3 as the baseline, which concatenates the top-3 sentences of each source document as the summary.
In Figure 5, we can observe that the lead baseline raises while both of our model performance degrade and finally they converge to the baseline. This is because it is more challenging for models to extract limited-number sentences that can cover the main idea of all source documents with the increasing number of documents. However, the First-3 baseline is forced to take sentences from each document which can ensure the coverage. Besides, the increase of document number enlarges the performance gap between HETERSUMGRAPH and HETERDOCSUMGRAPH. This indicates the benefit of document nodes will become more significant for more complex document-document relationships.
In this paper, we propose a heterogeneous graph-based neural network for extractive summarization. The introduction of more fine-grained semantic units in the summarization graph helps our model to build more complex relationships between sentences. It is also convenient to adapt our single-document graph to multi-document with document nodes. Furthermore, our models have achieved the best results on CNN/DailyMail compared with non-BERT-based models, and we will take the pretrained language models into account for better encoding representations of nodes in the future.
This work was supported by the National Natural Science Foundation of China (No. U1936214 and 61672162), Shanghai Municipal Science and Technology Major Project (No. 2018SHZDZX01) and ZJLab.
Kristjan Arumae and Fei Liu. 2018. Reinforced extractive summarization with question-focused rewards. In Proceedings of ACL 2018, Student Research Workshop, pages 105-111.
Jaime G Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In SIGIR, volume 98, pages 335-336.
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 484-494.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739-3748.
Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. arXiv preprint arXiv:1603.08887.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457-479.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109.
6217
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In ICML, pages 1263-1272.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684-1692.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131-4141.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics.
Hu Linmei, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. 2019. Heterogeneous graph attention networks for semi-supervised short text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4823-4832.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. Proceedings of the 6th International Conference on Learning Representations.
Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070-5081.
Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721-3731, Hong Kong, China. Association for Computational Linguistics.
Yang Liu, Ivan Titov, and Mirella Lapata. 2019. Single document summarization as tree induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1745-1755.
Ling Luo, Xiang Ao, Yan Song, Feiyang Pan, Min Yang, and Qing He. 2019. Reading like her: Human reading inspired extractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3024-3034.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ca glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. CoNLL 2016, page 280.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1747-1759.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1073-1083.
Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip. 2016. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering, 29(1):17-37.
Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, and Bowen Zhou. 2019. Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. arXiv preprint arXiv:1905.07374.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019a. Exploring domain shift in extractive text summarization. arXiv preprint arXiv:1908.11664.
Hsiu-Yi Wang, Jia-Wei Chang, and Jen-Wei Huang. 2019b. User intention-based document summarization on heterogeneous sentence networks. In International Conference on Database Systems for Advanced Applications, pages 572-587. Springer.
Yang Wei. 2012. Document summarization method based on heterogeneous graph. In 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery, pages 1285-1289. IEEE.
Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. arXiv preprint arXiv:1902.00863.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive model for text summarization. arXiv preprint arXiv:1910.14142.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. arXiv preprint arXiv:1706.06681.
Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. Neural latent extractive document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 779-784.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Conference of the Association for Computational Linguistics.
Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2019a. Searching for effective neural extractive summarization: What works and what's next. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1049-1058.
Ming Zhong, Danqing Wang, Pengfei Liu, Xipeng Qiu, and Xuan-Jing Huang. 2019b. A closer look at data bias in neural extractive summarization models. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 80-89.
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 654-663.
In order to select the best iteration number for HETERSUMGRAPH, we compare performances of different  $t$  on the validation set of CNN/DM. All models are trained on a single GeForce RTX 2080 Ti GPU for about 5 epochs. As Table 5 shows, our HETERSUMGRAPH has comparable results for  $t = 1$  and  $t = 3$ . However, when the iteration number goes from 1 to 3, the time for one epoch nearly doubles. Therefore, we take  $t = 1$  as a result of the balance of time cost and model performance.
Number	R-1	R-2	R-L	Time
t = 0	43.63	19.58	37.39	3.16h
t = 1	44.26	19.97	38.03	5.04h
t = 2	44.13	19.85	37.87	7.20h
t = 3	44.28	19.96	37.98	8.93h
Table 5: Different turns of iterative updating of sentence nodes. The experiments are performed on the validation set of CNN/DM. Time is the average time of one epoch.
6219