SlowGuess/ACL_2020_OCR huggingface dataset:
Jiacheng Xu $^{1}$ , Zhe Gan $^{2}$ , Yu Cheng $^{2}$ , Jingjing Liu $^{2}$
<sup>1</sup>The University of Texas at Austin <sup>2</sup>Microsoft Dynamics 365 AI Research jcxu@cs.utexas.edu; {zhe.gan,yu.cheng, jingjl}@microsoft.com
Recently BERT has been adopted for document encoding in state-of-the-art text summarization models. However, sentence-based extractive models often result in redundant or uninformative phrases in the extracted summaries. Also, long-range dependencies throughout a document are not well captured by BERT, which is pre-trained on sentence pairs instead of documents. To address these issues, we present a discourse-aware neural summarization model - DISCOBERT<sup>1</sup>. DISCOBERT extracts sub-sentential discourse units (instead of sentences) as candidates for extractive selection on a finer granularity. To capture the long-range dependencies among discourse units, structural discourse graphs are constructed based on RST trees and coreference mentions, encoded with Graph Convolutional Networks. Experiments show that the proposed model outperforms state-of-the-art methods by a significant margin on popular summarization benchmarks compared to other BERT-base models.
Neural networks have achieved great success in the task of text summarization (Nenkova et al., 2011; Yao et al., 2017). There are two main lines of research: abstractive and extractive. While the abstractive paradigm (Rush et al., 2015; See et al., 2017; Celikyilmaz et al., 2018; Sharma et al., 2019) focuses on generating a summary word-by-word after encoding the full document, the extractive approach (Cheng and Lapata, 2016; Zhou et al., 2018; Narayan et al., 2018) directly selects sentences from the document to assemble into a summary. The abstractive approach is more flexible
Figure 1: Illustration of DISCOBERT for text summarization. Sentence-based BERT model (baseline) selects whole sentences 1, 2 and 5. The proposed discourse-aware model DISCOBERT selects EDUs  $\{1-1, 2-1, 5-2, 20-1, 20-3, 22-1\}$ . The right side of the figure illustrates the two discourse graphs we use: (i) Coreference Graph (with the mentions of 'Pulitzer prizes' highlighted as examples); and (ii) RST Graph (induced by RST discourse trees).
and generally produces less redundant summaries, while the extractive approach enjoys better factuality and efficiency (Cao et al., 2018).
Recently, some hybrid methods have been proposed to take advantage of both, by designing a two-stage pipeline to first select and then rewrite (or compress) candidate sentences (Chen and Bansal, 2018; Gehrmann et al., 2018; Zhang et al., 2018; Xu and Durrett, 2019). Compression or rewriting aims to discard uninformative phrases in the selected sentences. However, most of these hybrid systems suffer from the inevitable disconnection between the two stages in the pipeline.
Meanwhile, modeling long-range context for document summarization remains a challenge (Xu
* Most of this work was done when the first author was an intern at Microsoft.
$^{1}$ Code, illustration and datasets are available at: https://github.com/jiacheng-xu/DiscoBERT.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021-5031 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
et al., 2016). Pre-trained language models (Devlin et al., 2019) are designed mostly for sentences or a short paragraph, thus poor at capturing long-range dependencies throughout a document. Empirical observations (Liu and Lapata, 2019) show that adding standard encoders such as LSTM or Transformer (Vaswani et al., 2017) on top of BERT to model inter-sentential relations does not bring in much performance gain.
In this paper, we present DISCOBERT, a discourse-aware neural extractive summarization model built upon BERT. To perform compression with extraction simultaneously and reduce redundancy across sentences, we take Elementary Discourse Unit (EDU), a sub-sentence phrase unit originating from RST (Mann and Thompson, 1988; Carlson et al., 2001) $^{2}$  as the minimal selection unit (instead of sentence) for extractive summarization. Figure 1 shows an example of discourse segmentation, with sentences broken down into EDUs (annotated with brackets). By operating on the discourse unit level, our model can discard redundant details in sub-sentences, therefore retaining additional capacity to include more concepts or events, leading to more concise and informative summaries.
Furthermore, we finetune the representations of discourse units with the injection of prior knowledge to leverage intra-sentence discourse relations. More specifically, two discourse-oriented graphs are proposed: RST Graph  $\mathcal{G}_R$  and Coreference Graph  $\mathcal{G}_C$ . Over these discourse graphs, Graph Convolutional Network (GCN) (Kipf and Welling, 2017) is imposed to capture long-range interactions among EDUs. RST Graph is constructed from RST parse trees over EDUs of the document. On the other hand, Coreference Graph connects entities and their coreference clusters/mentions across the document. The path of coreference navigates the model from the core event to other occurrences of that event, and in parallel explores its interactions with other concepts or events.
The main contribution is threefold: (i) We propose a discourse-aware extractive summarization model, DISCOBERT, which operates on a subsentential discourse unit level to generate concise and informative summary with low redundancy. (ii) We propose to structurally model
(a) [1] Winner: [2] This iconic photo by New York Times photographer Daniel Berehulak, was part of a winning series, [3] and shows James Dorbar, 8, [4] suspected of being infected with Ebola, [5] being carried by medical staff to an Ebola treatment center in Monrovia, Liberia.
Figure 2: Example of discourse segmentation and RST tree conversion. The original sentence is segmented into 5 EDUs in box (a), and then parsed into an RST discourse tree in box (b). The converted dependency-based RST discourse tree is shown in box (c). Nucleus nodes including [2], [3] and [5], and Satellite nodes including [2] and [4] are denoted by solid lines and dashed lines, respectively. Relations are in italic. The EDU [2] is the head of the whole tree (span [1-5]), while the EDU [3] is the head of the span [3-5].
inter-sentential context with two types of discourse graph. (iii) DISCOBERT achieves new state of the art on two popular newswire text summarization datasets, outperforming other BERT-base models.
2 Discourse Graph Construction
In this section, we first introduce the Rhetorical Structure Theory (RST) (Mann and Thompson, 1988), a linguistic theory for discourse analysis, and then explain how we construct discourse graphs used in DISCOBERT. Two types of discourse graph are considered: RST Graph and Coreference Graph. All edges are initialized as disconnected, and connections are later added for a subset of nodes based on RST discourse parse tree or coreference mentions.
2.1 Discourse Analysis
Discourse analysis focuses on inter-sentential relations in a document or conversation. In the RST framework, the discourse structure of text can be represented in a tree format. The whole document can be segmented into contiguous, adjacent and non-overlapping text spans called Elementary Discourse Units (EDUs). Each EDU is tagged as either Nucleus or Satellite, which characterizes its nuclearity or saliency. Nucleus nodes are generally more central, and Satellite nodes are more peripheral and less important in terms of content and grammatical reliance. There are dependencies among EDUs that represent their rhetorical relations.
In this work, we treat EDU as the minimal unit for content selection in text summarization. Fig-
2We adopt RST as the discourse framework due to the availability of existing tools, the nature of the RST tree structure for compression, and the observations from Louis et al. (2010). Other alternatives include Graph Bank (Wolf and Gibson, 2005) and PDTB (Miltsakaki et al., 2004).
ure 2 shows an example of discourse segmentation and the parse tree of a sentence. Among these EDUs, rhetorical relations represent the functions of different discourse units. As observed in Louis et al. (2010), the RST tree structure already serves as a strong indicator for content selection. On the other hand, the agreement between rhetorical relations tends to be lower and more ambiguous. Thus, we do not encode rhetorical relations explicitly in our model.
In content selection for text summarization, we expect the model to select the most concise and pivotal concept in the document, with low redundancy. However, in traditional extractive summarization methods, the model is required to select a whole sentence, even though some parts of the sentence are not necessary. Our proposed approach can select one or several fine-grained EDUs to render the generated summaries less redundant. This serves as the foundation of our DISCOBERT model.
2.2 RST Graph
When selecting sentences as candidates for extractive summarization, we assume each sentence is grammatically self-contained. But for EDUs, some restrictions need to be considered to ensure grammaticality. For example, Figure 2 illustrates an RST discourse parse tree of a sentence, where “[2] This iconic ... series” is a grammatical sentence but “[3] and shows ... 8” is not. We need to understand the dependencies between EDUs to ensure the grammaticality of the selected combinations. The detail of the derivation of the dependencies could be found in Sec 4.3.
The construction of the RST Graph aims to provide not only local paragraph-level but also long-range document-level connections among EDUs. We use the converted dependency version of the tree to build the RST Graph  $\mathcal{G}_R$ , by initializing an empty graph and treating every discourse dependency from the  $i$ -th EDU to the  $j$ -th EDU as a directed edge, i.e.,  $\mathcal{G}_R[i][j] = 1$ .
2.3 Coreference Graph
Text summarization, especially news summarization, usually suffers from the well-known 'position bias' issue (Kedzie et al., 2018), where most of the key information is described at the very beginning
of the document. However, there is still a decent amount of information spread in the middle or at the end of the document, which is often ignored by summarization models. We observe that around  $25\%$  of oracle sentences appear after the first 10 sentences in the CNNDM dataset. Besides, in long news articles, there are often multiple core characters and events throughout the whole document. However, existing neural models are poor at modeling such long-range context, especially when there are multiple ambiguous coreferences to resolve.
To encourage and guide the model to capture the long-range context in the document, we propose a Coreference Graph built upon discourse units. Algorithm 1 describes how to construct the Coreference Graph. We first use Stanford CoreNLP (Manning et al., 2014) to detect all the coreference clusters in an article. For each coreference cluster, all the discourse units containing the mention of the same cluster will be connected. This process is iterated over all the coreference mention clusters to create the final Coreference Graph.
Figure 1 provides an example, where 'Pulitzer prizes' is an important entity and has occurred multiple times in multiple discourse units. The constructed Coreference Graph is shown on the right side of the document<sup>4</sup>. When graph  $\mathcal{G}_C$  is constructed, edges among 1-1, 2-1, 20-1 and 22-1 are all connected due to the mentions of 'Pulitzer prizes'.
3 DISCOBERT Model
Figure 3 provides an overview of the proposed model, consisting of a Document Encoder and a Graph Encoder. For the Document Encoder, a pretrained BERT model is first used to encode the
4We intentionally ignore other entities and mentions in this example for simplicity.
For example, in Figure 2, details such as the name of the suspected child in [3], the exact location of the photo in [5], and who was carrying the child in [4], are unlikely to be reflected in the final summary.
Figure 3: (Left) Model architecture of DISCOBERT. The Stacked Discourse Graph Encoders contain  $k$  stacked DGE blocks. (Right) The architecture of each Discourse Graph Encoder (DGE) block.
whole document on the token level. Then, a self-attentive span extractor is used to obtain the EDU representations from the corresponding text spans. The Graph Encoder takes the output of the Document Encoder as input and updates the EDU representations with Graph Convolutional Network based on the constructed discourse graphs, which are then used to predict the oracle labels.
Assume that document  $D$  is segmented into  $n$  EDUs in total, i.e.,  $D = \{d_{1}, d_{2}, \dots, d_{n}\}$ , where  $d_{i}$  denotes the  $i$ -th EDU. Following Liu and Lapata (2019), we formulate extractive summarization as a sequential labeling task, where each EDU  $d_{i}$  is scored by neural networks, and decisions are made based on the scores of all EDUs. The oracle labels are a sequence of binary labels, where 1 stands for being selected and 0 for not. We denote the labels as  $Y = \{y_{1}^{*}, y_{2}^{*}, \dots, y_{n}^{*}\}$ . During training, we aim to predict the sequence of labels  $Y$  given the document  $D$ . During inference, we need to further consider discourse dependency to ensure the coherence and grammaticality of the output summary.
3.2 Document Encoder
BERT is a pre-trained deep bidirectional Transformer encoder (Vaswani et al., 2017; Devlin et al., 2019). Following Liu and Lapata (2019), we encode the whole document with BERT and finetune the BERT model for summarization.
BERT is originally trained to encode a single sentence or sentence pair. However, a news article
typically contains more than 500 words, hence we need to make some adaptation to apply BERT for document encoding. Specifically, we insert  $\langle \mathrm{CLS} \rangle$  and  $\langle \mathrm{SEP} \rangle$  tokens at the beginning and the end of each sentence, respectively. $^5$  In order to encode long documents such as news articles, we also extend the maximum sequence length that BERT can take from 512 to 768 in all our experiments.
The input document after tokenization is denoted as  $D = \{d_1, \dots, d_n\}$ , and  $d_i = \{w_{i1}, \dots, w_{il_i}\}$  where  $\ell_i$  is the number of BPE tokens in the  $i$ -th EDU. If  $d_i$  is the first EDU in a sentence, there is also a  $\langle \mathrm{CLS} \rangle$  token prepended to  $d_i$ ; if  $d_j$  is the last EDU in a sentence, there is a  $\langle \mathrm{SEP} \rangle$  token appended to  $d_j$  (see Figure 3). The schema of insertion of  $\langle \mathrm{CLS} \rangle$  and  $\langle \mathrm{SEP} \rangle$  is an approach used in Liu and Lapata (2019). For simplicity, these two tokens are not shown in the equations. BERT model is then used to encode the document:
$$
\{\mathbf {h} _ {1 1} ^ {B}, \dots , \mathbf {h} _ {n \ell_ {n}} ^ {B} \} = \mathrm {B E R T} (\{w _ {1 1}, \dots , w _ {n \ell_ {n}} \}),
$$
where  $\{\mathbf{h}_{11}^B,\dots ,\mathbf{h}_{n\ell_n}^B\}$  is the BERT output of the whole document in the same length as the input.
After the BERT encoder, the representation of the  $\langle \mathrm{CLS}\rangle$  token can be used as sentence representation. However, this approach does not work in our setting, since we need to extract the representation for EDUs instead. Therefore, we adopt a
We also tried inserting  $\langle \mathrm{CLS}\rangle$  and  $\langle \mathrm{SEP}\rangle$  at the beginning and the end of every EDU, and treating the corresponding  $\langle \mathrm{CLS}\rangle$  representation as the representation for each EDU, but the performance drops drastically.
Self-Attentive Span Extractor (SpanExt), proposed in Lee et al. (2017), to learn EDU representation.
For the  $i$ -th EDU with  $\ell_i$  words, with the output from the BERT encoder  $\{\mathbf{h}_{i1}^B,\mathbf{h}_{i2}^B,\dots ,\mathbf{h}_{i\ell_i}^B\}$ , we obtain EDU representation as follows:
$$
\begin{array}{l} \alpha_ {i j} = \mathbf {W} _ {2} \cdot \mathrm {R e L U} (\mathbf {W} _ {1} \mathbf {h} _ {i j} ^ {B} + \mathbf {b} _ {1}) + \mathbf {b} _ {2} \\ \mathbf {a} _ {i j} = \frac {\exp (\alpha_ {i j})}{\sum_ {k = 1} ^ {\ell_ {i}} \exp (\alpha_ {i k})}, \quad \mathbf {h} _ {i} ^ {S} = \sum_ {j = 1} ^ {\ell_ {i}} \mathbf {a} _ {i j} \cdot \mathbf {h} _ {i j} ^ {B}, \\ \end{array}
$$
where  $\alpha_{ij}$  is the score of the  $j$ -th word in the EDU,  $\mathbf{a}_{ij}$  is the normalized attention of the  $j$ -th word w.r.t. all the words in the span.  $\mathbf{h}_i^S$  is a weighted sum of the BERT output hidden states. Throughout the paper, all the  $\mathbf{W}$  matrices and  $\mathbf{b}$  vectors are parameters to learn. We abstract the above Self-Attentive Span Extractor as  $\mathbf{h}_i^S = \mathrm{SpanExt}(\mathbf{h}_{i1}^B,\dots ,\mathbf{h}_{il_\ell}^B)$ .
After the span extraction step, the whole document is represented as a sequence of EDU representations:  $\mathbf{h}^S = \{\mathbf{h}_1^S,\dots ,\mathbf{h}_n^S\} \in \mathbb{R}^{d_h\times n}$ , which will be sent to the graph encoder.
3.3 Graph Encoder
Given the constructed graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , nodes  $\mathcal{V}$  correspond to the EDUs in a document, and edges  $\mathcal{E}$  correspond to either RST discourse relations or coreference mentions. We then use Graph Convolutional Network to update the representations of all the EDUs, to capture long-range dependencies missed by BERT for better summarization. To modularize architecture design, we present a single Discourse Graph Encoder (DGE) layer. Multiple DGE layers are stacked in our experiments.
Assume that the input for the  $k$ -th DGE layer is denoted as  $\mathbf{h}^{(k)} = \{\mathbf{h}_1^{(k)}, \ldots, \mathbf{h}_n^{(k)}\} \in \mathbb{R}^{d_h \times n}$ , and the corresponding output is denoted as  $\mathbf{h}^{(k+1)} = \{\mathbf{h}_1^{(k+1)}, \ldots, \mathbf{h}_n^{(k+1)}\} \in \mathbb{R}^{d_h \times n}$ . The  $k$ -th DGE layer is designed as follows:
$$
\begin{array}{l} \mathbf {u} _ {i} ^ {(k)} = \mathbf {W} _ {4} ^ {(k)} \operatorname {R e L U} \left(\mathbf {W} _ {3} ^ {(k)} \mathbf {h} _ {i} ^ {(k)} + \mathbf {b} _ {3} ^ {(k)}\right) + \mathbf {b} _ {4} ^ {(k)} \\ \mathbf {v} _ {i} ^ {(k)} = \operatorname {L N} \left(\mathbf {h} _ {i} ^ {(k)} + \operatorname {D r o p o u t} \left(\mathbf {u} _ {i} ^ {(k)}\right)\right) \\ \mathbf {w} _ {i} ^ {(k)} = \operatorname {R e L U} \left(\sum_ {j \in \mathcal {N} _ {i}} \frac {1}{| \mathcal {N} _ {i} |} \mathbf {W} _ {5} ^ {(k)} \mathbf {v} _ {j} ^ {(k)} + \mathbf {b} _ {5} ^ {(k)}\right) \\ \mathbf {h} _ {i} ^ {(k + 1)} = \operatorname {L N} (\operatorname {D r o p o u t} (\mathbf {w} _ {i} ^ {(k)}) + \mathbf {v} _ {i} ^ {(k)}), \\ \end{array}
$$
where  $\mathrm{LN}(\cdot)$  represents Layer Normalization,  $\mathcal{N}_i$  denotes the neighborhood of the  $i$ -th EDU node.  $\mathbf{h}_i^{(k + 1)}$  is the output of the  $i$ -th EDU in the  $k$ -th DGE layer, and  $\mathbf{h}^{(1)} = \mathbf{h}^S$ , which is the output from the Document Encoder. After  $K$  layers of
Dataset	Document	Sum. # tok.	# E in Graph G_R G_C
# sent.	# EDU	# tok.
CNNDM	24	67	541	54	66	233
NYT	22	66	591	87	65	143
Table 1: Statistics of the datasets. The first block shows the average number of sentences, EDUs and tokens in the documents. The second block shows the average number of tokens in the reference summaries. The third block shows the average number of edges in the constructed RST Graphs  $(\mathcal{G}_R)$  and Coreference Graphs  $(\mathcal{G}_C)$ , respectively.
graph propagation, we obtain  $\mathbf{h}^G = \mathbf{h}^{(K + 1)}\in$ $\mathbb{R}^{d_h\times n}$  , which is the final representation of all the EDUs after the stacked DGE layers. For different graphs, the parameter of DGEs are not shared. If we use both graphs, their output are concatenated:  $\mathbf{h}^G = \mathrm{ReLU}(\mathbf{W}_6[\mathbf{h}_C^G;\mathbf{h}_R^G ] + \mathbf{b}_6)$
3.4 Training & Inference
During training,  $\mathbf{h}^G$  is used for predicting the oracle labels. Specifically,  $\hat{y}_i = \sigma (\mathbf{W}_7\mathbf{h}_i^G +\mathbf{b}_7)$  where  $\sigma (\cdot)$  represents the logistic function, and  $\hat{y}_i$  is the prediction probability ranging from 0 to 1. The training loss of the model is binary cross-entropy loss given the predictions and oracles:  $\mathcal{L} = -\sum_{i = 1}^{n}(y_i^*\log (\hat{y}_i) + (1 - y_i^*)\log (1 - \hat{y}_i))$  For DISCOBERT without graphs, the output from Document Encoder  $\mathbf{h}^S$  is used for prediction instead. The creation of oracle is operated on EDU level. We greedily pick up EDUs with their necessary dependencies until R-1 F<sub>1</sub> drops.
During inference, given an input document, after obtaining the prediction probabilities of all the EDUs, i.e.,  $\hat{\mathbf{y}} = \{\hat{y}_1,\dots ,\hat{y}_n\}$ , we sort  $\hat{\mathbf{y}}$  in descending order, and select EDUs accordingly. Note that the dependencies between EDUs are also enforced in prediction to ensure grammacality of generated summaries.
In this section, we present experimental results on two popular news summarization datasets. We compare our proposed model with state-of-the-art baselines and conduct detailed analysis to validate the effectiveness of DISCOBERT.
We evaluate the models on two datasets: New York Times (NYT) (Sandhaus, 2008), CNN and Daily-mail (CNNDM) (Hermann et al., 2015). We use the
5025
script from See et al. (2017) to extract summaries from raw data, and Stanford CoreNLP for sentence boundary detection, tokenization and parsing (Manning et al., 2014). Due to the limitation of BERT, we only encode up to 768 BERT BPEs.
Table 1 provides statistics of the datasets. The edges in  $\mathcal{G}_C$  are undirected, while those in  $\mathcal{G}_R$  are directional. For CNNDM, there are 287,226, 13,368 and 11,490 samples for training, validation and test, respectively. We use the un-anonymized version as in previous summarization work. NYT is licensed by LDC<sup>6</sup>. Following previous work (Zhang et al., 2019; Xu and Durrett, 2019), we use 137,778, 17,222 and 17,223 samples for training, validation, and test, respectively.
4.2 State-of-the-art Baselines
We compare our model with the following state-of-the-art neural text summarization models.
Extractive Models: BanditSum treats extractive summarization as a contextual bandit problem, trained with policy gradient methods (Dong et al., 2018). NeuSum is an extractive model with seq2seq architecture, where the attention mechanism scores the document and emits the index as the selection (Zhou et al., 2018).
Compressive Models: JECS is a neural text-compression-based summarization model using BLSTM as the encoder (Xu and Durrett, 2019). The first stage is selecting sentences, and the second stage is sentence compression by pruning constituency parsing tree.
BERT-based Models: BERT-based models have achieved significant improvement on CNNDM and NYT, when compared with LSTM counterparts. BertSum is the first BERT-based extractive summarization model (Liu and Lapata, 2019). Our baseline model BERT is the re-implementation of BertSum. PNBert proposed a BERT-based model with various training strategies, including reinforcement learning and Pointer Networks (Zhong et al., 2019). HiBert is a hierarchical BERT-based model for document encoding, which is further pretrained with unlabeled data (Zhang et al., 2019).
We use AllenNLP (Gardner et al., 2018) as the code framework. The implementation of graph
Model	R-1	R-2	R-L
Lead3	40.42	17.62	36.67
Oracle (Sentence)	55.61	32.84	51.88
Oracle (Discourse)	61.61	37.82	59.27
NeuSum (Zhou et al., 2018)	41.59	19.01	37.98
BanditSum (Dong et al., 2018)	41.50	18.70	37.60
JECS (Xu and Durrett, 2019)	41.70	18.50	37.90
PNBERT (Zhong et al., 2019)	42.39	19.51	38.69
PNBERT w. RL	42.69	19.60	38.85
BERT (Zhang et al., 2019)	41.82	19.48	38.30
HIBERTS	42.10	19.70	38.53
HIBERT*	42.31	19.87	38.78
HIBERTM	42.37	19.95	38.83
BERTSUM (Liu and Lapata, 2019)	43.25	20.24	39.63
T5-Base (Raffel et al., 2019)	42.05	20.34	39.40
BERT	43.07	19.94	39.44
DISCOBERT	43.38	20.44	40.21
DISCOBERT w. GC	43.58	20.64	40.42
DISCOBERT w. GR	43.68	20.71	40.54
DISCOBERT w. GR & GC	43.77	20.85	40.67