SlowGuess/ACL_2020_OCR huggingface dataset:
Table 2: Results on the test set of the CNNDM dataset. ROUGE-1, -2 and -L  $\mathrm{F}_{1}$  are reported. Models with the asterisk symbol (*) used extra data for pre-training. R-1 and R-2 are shorthands for unigram and bigram overlap; R-L is the longest common subsequence.
encoding is based on DGL (Wang et al., 2019). Experiments are conducted on a single NVIDIA P100 card, and the mini-batch size is set to 6 due to GPU memory capacity. The length of each document is truncated to 768 BPEs. We use the pre-trained 'bert-base-uncased' model and fine tune it for all experiments. We train all our models for up to 80,000 steps. ROUGE (Lin, 2004) is used as the evaluation metrics, and 'R-2' is used as the validation criteria.
The realization of discourse units and structure is a critical part of EDU pre-processing, which requires two steps: discourse segmentation and RST parsing. In the segmentation phase, we use a neural discourse segmenter based on the BiLSTM CRF framework (Wang et al., 2018). The segmenter achieved  $94.3\mathrm{F}_1$  score on the RST-DT test set, in which the human performance is 98.3. In the parsing phase, we use a shift-reduce discourse parser to extract relations and identify nuclearity (Ji and Eisenstein, 2014).
The dependencies among EDUs are crucial to the grammaticality of selected EDUs. Here are the two steps to learn the derivation of dependencies: head inheritance and tree conversion. Head inheritance defines the head node for each valid non-terminal tree node. For each leaf node, the
$^{7}$ https://github.com/PKU-TANGENT/NeuralEDUSeg
<sup>8</sup>https://github.com/jiyfeng/DPLP
$^{6}$ https://catalog.ldc.upenn.edu/LDC2008T19
head is itself. We determine the head node(s) of non-terminal nodes based on their nuclearity. For example, in Figure 2, the heads of text spans [1-5], [2-5], [3-5] and [4-5] need to be grounded to a single EDU. We propose a simple yet effective schema to convert RST discourse tree to a dependency-based discourse tree. We always consider the dependency restriction such as the reliance of Satellite on Nucleus, when we create oracle during preprocessing and when the model makes the prediction. For the example in Figure 2, if the model selects “[5] being carried ... Liberia.” as a candidate span, we will enforce the model to select “[3] and shows ... 8,” and “[2] This ... series,” as well.
The number of chosen EDUs depends on the average length of the reference summaries, dependencies across EDUs as mentioned above, and the length of the existing content. The optimal average number of EDUs selected is tuned on the development set.
Results on CNNDM Table 2 shows results on CNNDM. The first section includes Lead3 baseline, sentence-based oracle, and discourse-based oracle. The second section lists the performance of baseline models, including non-BERT-based and BERT-based variants. The performance of our proposed model is listed in the third section. BERT is our implementation of sentence-based BERT model. DISCOBERT is our discourse-based BERT model without Discourse Graph Encoder. DISCOBERT w.  $\mathcal{G}_C$  and DISCOBERT w.  $\mathcal{G}_R$  are the discourse-based BERT model with Coreference Graph and RST Graph, respectively. DISCOBERT w.  $\mathcal{G}_R$  &  $\mathcal{G}_C$  is the fusion model encoding both graphs.
The proposed DISCOBERT beats the sentence-based counterpart and all the competitor models. With the help of Discourse Graph Encoder, the graph-based DISCOBERT beats the state-of-the-art BERT model by a significant margin (0.52/0.61/1.04 on R-1/-2/-L on  $\mathbf{F}_1$ ). Ablation study with individual graphs shows that the RST Graph is slightly more helpful than the Coreference
Model	R-1	R-2	R-L
Lead3	41.80	22.60	35.00
Oracle (Sentence)	64.22	44.57	57.27
Oracle (Discourse)	67.76	48.05	62.40
JECS (Xu and Durrett, 2019)	45.50	25.30	38.20
BERT (Zhang et al., 2019)	48.38	29.04	40.53
HIBERTS	48.92	29.58	41.10
HIBERTM	49.06	29.70	41.23
HIBERTS*	49.25	29.92	41.43
HIBERTM	49.47	30.11	41.63
BERT	48.48	29.01	40.62
DISCOBERT	49.78	30.30	42.44
DISCOBERT w. GC	49.79	30.18	42.48
DISCOBERT w. GR	49.86	30.25	42.55
DISCOBERT w. GR & GC	50.00	30.38	42.70
Table 3: Results on the test set of the NYT dataset. Models with the asterisk symbol (*) used extra data for pre-training.
Graph, while the combination of both achieves better performance overall.
Results on NYT Results are summarized in Table 3. The proposed model surpasses previous state-of-the-art BERT-based model by a significant margin. HIBERT\* and HIBERT\* used extra data for pre-training the model. We notice that in the NYT dataset, most of the improvement comes from the use of EDUs as minimal selection units. DISCOBERT provides 1.30/1.29/1.82 gain on R-1/-2/-L over the BERT baseline. However, the use of discourse graphs does not help much in this case.
4.5 Grammaticality
Due to segmentation and partial selection of sentence, the output of our model might not be as grammatical as the original sentence. We manually examined and automatically evaluated model output, and observed that overall, the generated summaries are still grammatical, given the RST dependency tree constraining the rhetorical relations among EDUs. A set of simple yet effective post-processing rules helps to complete the EDUs in some cases.
Automatic Grammar Checking We followed Xu and Durrett (2019) to perform automatic grammar checking using Grammarly. Table 4 shows the grammar checking results, where the average number of errors in every 10,000 characters on CNNDM and NYT datasets is reported. We compare DISCOBERT with sentence-based BERT model. 'All' shows the summation of the number of errors in all categories. As shown in the table, the
If both children are N(ucleus), then the head of the current node inherits the head of the left child. Otherwise, when one child is N and the other is S, the head of the current node inherits the head of the N child.
<sup>10</sup>If one child node is N and the other is S, the head of the S node depends on the head of the N node. If both children are N and the right child does not contain a subject in the discourse, the head of the right N node depends on the head of the left N node.
Source	M	All	CR	PV	PT	O
CNNDM	Sent	33.0	18.7	9.0	2.3	3.0
Disco	34.0	18.3	8.4	2.6	4.7
NYT	Sent	23.3	13.5	5.9	0.8	3.1
Disco	23.8	13.9	5.7	0.8	3.4
Table 4: Number of errors per 10,000 characters based on automatic grammaticality checking with Grammarly on CNNDM and NYT. Lower values are better. Detailed error categories, including correctness (CR), passive voice (PV) misuse, punctuation (PT) in compound/complex sentences and others (O), are listed from left to right.
Model	All	Coherence	Grammaticality
Sent	3.45 ± 0.87	3.30 ± 0.90	3.45 ± 1.06
Disco	3.24 ± 0.84	3.15 ± 0.95	3.25 ± 1.02
Ref	3.28 ± 0.99	3.12 ± 0.94	3.29 ± 1.06
Table 5: Human evaluation results. We ask Turkers to grade the overall preference, coherence and grammaticality from 1 to 5. Mean values along with standard deviations are reported.
summaries generated by our model have retained the quality of the original text.
Human Evaluation We sampled 200 documents from the test set of CNNDM and for each sample, we asked two Turkers to grade three summaries from 1 to 5. Results are shown in Table 5. SentBERT model (the original BERTSum model) selects sentences from the document, hence providing the best overall readability, coherence, and grammaticality. In some cases, reference summaries are just long phrases, so the scores are slightly lower than those from the sentence model. DISCOBERT model is slightly worse than Sent-BERT model but is fully comparable to the other two variants.
Examples & Analysis We show some examples of model output in Table 6. We notice that a decent amount of irrelevant details are removed from the extracted summary.
Despite the success, we further conducted error analysis and found that the errors mostly originated from the RST dependency resolution and the upstream parsing error of the discourse parser. The misclassification of RST dependencies and the hand-crafted rules for dependency resolution hurted the grammaticality and coherence of the 'generated' outputs. Common punctuation issues include extra or missing commas, as well as missing quotation marks. Some of the coherence issue
Clare Hines, who lives in Brisbane, was diagnosed with a brain tumour after suffering epileptic seizures. After a number of tests doctors discovered she had a benign tumour that had wrapped itself around her acoustic, faecal and balance nerve—and told her she had have it surgically removed or she risked the tumour turning malignant. One week before brain surgery she found out she was pregnant.
Jordan Henderson, in action against Aston Villa at Wembley on Sunday, has agreed a new Liverpool deal. The club's vice captain puts pen to paper on a deal which will keep him at Liverpool until 2020. Rodgers will consider Henderson for the role of club captain after Steven Gerrard moves to LA Galaxy at the end of the campaign but, for now, the England international is delighted to have agreed terms on a contract that will take him through the peak years of his career.
Table 6: Example outputs from CNNDM by DISCOBERT. Strikethrough indicates discarded EDUs.
originates from missing or improper or missing anaphora resolution. In this example "[Johnny is believed to have drowned,  $]_{1}$  [but actually he is fine,'  $]_{2}$  [the police say.  $]_{3}$ ", only selecting the second EDU yields a sentence "actually he is fine", which is not clear who is 'he' mentioned here.
Neural Extractive Summarization Neural networks have been widely used in extractive summarization. Various decoding approaches, including ranking (Narayan et al., 2018), index prediction (Zhou et al., 2018) and sequential labelling (Nallapati et al., 2017; Zhang et al., 2018; Dong et al., 2018), have been applied to content selection. Our model uses a similar configuration to encode the document with BERT as Liu and Lapata (2019) did, but we use discourse graph structure and graph encoder to handle the long-range dependency issue.
Neural Compressive Summarization Text summarization with compression and deletion has been explored in some recent work. Xu and Durrett (2019) presented a two-stage neural model for selection and compression based on constituency tree pruning. Dong et al. (2019) presented a neural sentence compression model with discrete operations including deletion and addition. Different from these studies, as we use EDUs as minimal selection basis, sentence compression is achieved automatically in our model.
Discourse & Summarization The use of discourse theory for text summarization has been explored before. Louis et al. (2010) examined the
benefit of graph structure provided by discourse relations for text summarization. Hirao et al. (2013); Yoshida et al. (2014) formulated the summarization problem as the trimming of the document discourse tree. Durrett et al. (2016) presented a system of sentence extraction and compression with ILP methods using discourse structure. Li et al. (2016) demonstrated that using EDUs as units of content selection leads to stronger summarization performance. Compared with them, our proposed method is the first neural end-to-end summarization model using EDUs as the selection basis.
Graph-based Summarization Graph approach has been explored in text summarization over decades. LexRank introduced a stochastic graph-based method for computing relative importance of textual units (Erkan and Radev, 2004). Yasunaga et al. (2017) employed a GCN on the relation graphs with sentence embeddings obtained from RNN. Tan et al. (2017) also proposed graph-based attention in abstractive summarization model. Fernandes et al. (2018) developed a framework to reason long-distance relationships for text summarization.
In this paper, we present DISCOBERT, which uses discourse unit as the minimal selection basis to reduce summarization redundancy and leverages two types of discourse graphs as inductive bias to capture long-range dependencies among discourse units. We validate the proposed approach on two popular summarization datasets, and observe consistent improvement over baseline models. For future work, we will explore better graph encoding methods, and apply discourse graphs to other tasks that require long document encoding.
Thanks to Junyi Jessy Li, Greg Durrett, Yen-Chun Chen, and to the other members of the Microsoft Dynamics 365 AI Research team for the proof-reading, feedback and suggestions.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the Original: Fact Aware Neural Abstractive Summarization. In AAAI Conference on Artificial Intelligence.
Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue.
Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1662-1675, New Orleans, Louisiana. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686. Association for Computational Linguistics.
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484-494, Berlin, Germany. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393-3402, Florence, Italy. Association for Computational Linguistics.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. BanditSum: Extractive Summarization as a Contextual Bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739-3748. Association for Computational Linguistics.
Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1998–2008. Association for Computational Linguistics.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text
5029
summmarization. Journal of artificial intelligence research, 22:457-479.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018. Structured neural summarization. arXiv preprint arXiv:1811.01824.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1- 6, Melbourne, Australia. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-Up Abstractive Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Tsutomu Hirao, Yasuhisa Yoshida, Masaaki Nishino, Norihito Yasuda, and Masaaki Nagata. 2013. Single-document summarization as a tree knapsack problem. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA.
Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13-24, Baltimore, Maryland. Association for Computational Linguistics.
Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818-1828.
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of ICLR.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summarization. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and
Dialogue, pages 137-147. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3728-3738, Hong Kong, China. Association for Computational Linguistics.
Annie Louis, Aravind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In Proceedings of the SIGDIAL 2010 Conference, pages 147-156, Tokyo, Japan. Association for Computational Linguistics.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243-281.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60. Association for Computational Linguistics.
Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The penn discourse treebank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04).
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. SummaRuNNer: A Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents. In AAAI Conference on Artificial Intelligence.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747-1759. Association for Computational Linguistics.
Ani Nenkova, Kathleen McKeown, et al. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2-3):103-233.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389. Association for Computational Linguistics.
Evan Sandhaus. 2008. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083. Association for Computational Linguistics.
Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. 2019. An entity-driven framework for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3271-3282.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1171-1181, Vancouver, Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J Smola, and Zheng Zhang. 2019. Deep graph library: Towards efficient and scalable deep learning on graphs. ICLR Workshop on Representation Learning on Graphs and Manifolds.
Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. Toward fast and accurate neural discourse segmentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 962-967, Brussels, Belgium. Association for Computational Linguistics.
Florian Wolf and Edward Gibson. 2005. Representing discourse coherence: A corpus-based study. Computational linguistics, 31(2):249-287.
Jiacheng Xu, Danlu Chen, Xipeng Qiu, and Xuanjing Huang. 2016. Cached long short-term memory neural networks for document-level sentiment classification. In Proceedings of the 2016 Conference on