SlowGuess/NAACL_2021_OCR huggingface dataset:
Thanh-Tung Nguyen†, Xuan-Phi Nguyen†, Shafiq Joty§, Xiaoli Li†
Salesforce Research Asia
†Institute for Infocomm Research, A-STAR
{ng0155ng@e.;nguyenxu002@e.;srjoty@}ntu.edu.sg
x11i@i2r.a-star.edu.sg
We introduce a novel top-down end-to-end formulation of document level discourse parsing in the Rhetorical Structure Theory (RST) framework. In this formulation, we consider discourse parsing as a sequence of splitting decisions at token boundaries and use a seq2seq network to model the splitting decisions. Our framework facilitates discourse parsing from scratch without requiring discourse segmentation as a prerequisite; rather, it yields segmentation as part of the parsing process. Our unified parsing model adopts a beam search to decode the best tree structure by searching through a space of high scoring trees. With extensive experiments on the standard English RST discourse treebank, we demonstrate that our parser outperforms existing methods by a good margin in both end-to-end parsing and parsing with gold segmentation. More importantly, it does so without using any handcrafted features, making it faster and easily adaptable to new languages and domains.
In a document, the clauses, sentences and paragraphs are logically connected together to form a coherent discourse. The goal of discourse parsing is to uncover this underlying coherence structure, which has been shown to benefit numerous NLP applications including text classification (Ji and Smith, 2017), summarization (Gerani et al., 2014), sentiment analysis (Bhatia et al., 2015), machine translation evaluation (Joty et al., 2017) and conversational machine reading (Gao et al., 2020).
Rhetorical Structure Theory or RST (Mann and Thompson, 1988), one of the most influential theories of discourse, postulates a hierarchical discourse structure called discourse tree (DT). The leaves of a DT are clause-like units, known as elementary discourse units (EDUs). Adjacent EDUs and higher-order spans are connected hierarchically through coherence relations (e.g., Contrast, Expla
nation). Spans connected through a relation are categorized based on their relative importance — nucleus being the main part, with satellite being the subordinate one. Fig. 1 exemplifies a DT spanning over two sentences and six EDUs. Finding discourse structure generally requires breaking the text into EDUs (discourse segmentation) and linking the EDUs into a DT (discourse parsing).
Discourse parsers can be singled out by whether they apply a bottom-up or top-down procedure. Bottom-up parsers include transition-based models (Feng and Hirst, 2014; Ji and Eisenstein, 2014; Braud et al., 2017; Wang et al., 2017) or globally optimized chart parsing models (Soricut and Marcu, 2003; Joty et al., 2013, 2015). The former constructs a DT by a sequence of shift and reduce decisions, and can parse a text in asymptotic running time that is linear in number of EDUs. However, the transition-based parsers make greedy local decisions at each decoding step, which could propagate errors into future steps. In contrast, chart parsers learn scoring functions for sub-trees and adopt a CKY-like algorithm to search for the highest scoring tree. These methods normally have higher accuracy but suffer from a slow parsing speed with a complexity of  $\mathcal{O}(n^3)$  for  $n$  EDUs. The top-down parsers are relatively new in discourse (Lin et al., 2019; Zhang et al., 2020; Kobayashi et al., 2020). These methods focus on finding splitting points in each iteration to build a DT. However, the local decisions could still affect the performance as most of the methods are still greedy.
Like most other fields in NLP, language parsing has also undergone a major paradigm shift from traditional feature-based statistical parsing to end-to-end neural parsing. Being able to parse a document end-to-end from scratch is appealing for several key reasons. First, it makes the overall development procedure easily adaptable to new languages, domains and tasks by surpassing the expensive feature engineering step that often requires more time
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1613-1625 June 6-11, 2021. ©2021 Association for Computational Linguistics
and domain/language expertise. Second, the lack of an explicit feature extraction phase makes the training and testing (decoding) faster.
Because of the task complexity, it is only recently that neural approaches have started to outperform traditional feature-rich methods. However, successful document level neural parsers still rely heavily on handcrafted features (Ji and Eisenstein, 2014; Yu et al., 2018; Zhang et al., 2020; Kobayashi et al., 2020). Therefore, even though these methods adopt a neural framework, they are not "end-to-end" and do not enjoy the above mentioned benefits of an end-to-end neural parser. Moreover, in existing methods (both traditional and neural), discourse segmentation is detached from parsing and treated as a prerequisite step. Therefore, the errors in segmentation affect the overall parsing performance (Soricut and Marcu, 2003; Joty et al., 2012).
In view of the limitations of existing approaches, in this work we propose an end-to-end top-down document level parsing model that:
- Can generate a discourse tree from scratch without requiring discourse segmentation as a prerequisite step; rather, it generates the EDUs as a by-product of parsing. Crucially, this novel formulation facilitates solving the two tasks in a single neural model. Our formulation is generic and works in the same way when it is provided with the EDU segmentation.
- Treats discourse parsing as a sequence of splitting decisions at token boundaries and uses a seq2seq pointer network (Vinyals et al., 2015) to model the splitting decisions at each decoding step. Importantly, our seq2seq parsing model can adopt beam search to widen the search space for the highest scoring tree, which to our knowledge is also novel for the parsing problem.
- Does not rely on any handcrafted features, which makes it faster to train or test, and easily adaptable to other domains and languages.
- Achieves the state of the art (SoTA) with an  $F_{1}$  score of 46.6 in the Full (label+structure) metric for end-to-end parsing on the English RST Discourse Treebank, which outperforms many parsers that use gold EDU segmentation. With gold segmentation, our model achieves a SoTA  $F_{1}$  score of 50.2 (Full), outperforming the best existing system by 2.1 absolute points. More importantly, it does so without using any handcrafted features (not even part-of-speech tags).
We make our code available at https://ntunlpsg.github.io/project/rst-parser
Assuming that a document has already been segmented into EDUs, following the traditional approach, the corresponding discourse tree (DT) can be represented as a set of labeled constituents.
$$
\mathbb {C} := \left\{\left(\left(i _ {t}, k _ {t}, j _ {t}\right), r _ {t}\right) \mid i _ {t} \leq k _ {t} <   j _ {t} \right\} _ {t = 1} ^ {m} \tag {1}
$$
where  $m = |\mathbb{C}|$  is the number of internal nodes in the tree and  $r_t$  is the relation label between the discourse unit containing EDUs  $i_t$  through  $k_t$  and the one containing EDUs  $k_t + 1$  through  $j_t$ .
Traditionally, in RST parsing, discourse segmentation is performed first to obtain the sequence of EDUs, which is followed by the parsing process to assemble the EDUs into a labeled tree. In other words, traditionally discourse segmentation and parsing have been considered as two distinct tasks that are solved by two different models.
On the contrary, in this work we take a radically different approach that directly starts with parsing the (unsegmented) document in a top-down manner and treats discourse segmentation as a special case of parsing that we get as a by-product. Importantly, this novel formulation of the problem allows us to solve the two problems in a single neural model. Our parsing model is generic and also works in the same way when it is fed with an EDU-segmented text. Before presenting the model architecture, we first formulate the problem as a splitting decision problem at the token level.
2.1 Parsing as a Splitting Decision Problem
We reformulate the discourse parsing problem from Eq. (1) as a sequence of splitting decisions at token boundaries (instead of EDUs). Specifically, the input text is first prepended and appended with the special start  $(<\mathrm{sod}>)$  and end  $(<\mathrm{eod}>)$  tokens, respectively. We define the token-boundary as the indexed position between two consecutive tokens. For example, the constituent spanning "But he added :" in Fig. 2 is defined as  $(0,4)$ .
Following the standard practice, we convert the discourse tree by transforming each multi-nuclear constituent into a hierarchical right-branching binary sub-tree. Every internal node in the resulting binary tree will have a left and a right constituent, allowing us to represent it by its split into the left and right children. Based on this, we define the
Figure 1: A discourse tree for two sentences in the RST discourse treebank. The internal nodes (e.g., Attribution, Contrast) denote the coherence relations and the edge labels reflect the nuclearity of the child span. Below the tree, we show the sequence of splitting decisions  $\mathbb{S}_{\mathrm{edu}}$  when EDUs are provided and  $\mathbb{S}$  when EDUs are not provided (end-to-end parsing). The bold splitting decision represents the final split of the span, forming an EDU.
Figure 2: Relation between token-boundary (above) and token (below) representations. A token-boundary position  $k$  is located between the tokens at  $k$  and  $k + 1$ .
Boundary-based splitting representation when EDUs are provided
$$
\mathbb {S} _ {\mathrm {e d u}} = \{(0, 4 4) \rightarrow 4, (4, 4 4) \rightarrow 2 5, (4, 2 5) \rightarrow 1 7, (2 5, 4 4) \rightarrow 3 7, (2 5, 3 7) \rightarrow 3 3 \}
$$
Boundary-based splitting representation for end-to-end parsing
$$
\mathbb {S} = \{(0, 4 4) \rightarrow 4, (0, 4) \rightarrow 4, (4, 4 4) \rightarrow 2 5, (4, 2 5) \rightarrow 1 7, (4, 1 7) \rightarrow 1 7, (1 7, 2 5) \rightarrow 2 5,
$$
$$
(2 5, 4 4) \rightarrow 3 7, (2 5, 3 7) \rightarrow 3 3, (\mathbf {2 5}, \mathbf {3 3}) \rightarrow \mathbf {3 3}, (\mathbf {3 3}, \mathbf {3 7}) \rightarrow \mathbf {3 7}, (\mathbf {3 7}, \mathbf {4 4}) \rightarrow \mathbf {4 4} \}
$$
$$
\begin{array}{c c c c c c c c c c c c} 0 & 1 & & 2 & & 3 & 4 & 4 1 & & 4 2 & 4 3 & 4 4 \\ \hline \text {s o d} > & \text {B u t} & & \text {h e} & & \text {a d d e d} & : & \dots & \text {m o n t h s} & . & \text {＂} & \text {e e o d} > \\ 0 & 1 & & 2 & & 3 & 4 & & 4 2 & 4 3 & 4 4 & 4 5 \end{array}
$$
Parsing as a set of splitting decisions  $\mathbb{S}$  at tokenboundaries by the following proposition:
Proposition 1 Given a binarized discourse tree for a document containing  $n$  tokens, the tree can be converted into a set of token-boundary splitting decisions  $\mathbb{S} = \{(i,j)\rightarrow k|i < k\leq j\}$  such that the parent constituent  $(i,j)$  either gets split into two child constituents  $(i,k)$  and  $(k,j)$  for  $k < j$ , or forms a terminal EDU unit for  $k = j$ , i.e., the span will not be split further (i.e., marks segmentation).
Notice that  $\mathbb{S}$  is a generalized formulation of RST parsing, which also includes the decoding of EDUs as a special case ( $k = j$ ). It is quite straightforward to change this formulation to the parsing scenario, where discourse segmentation (sequence of EDUs) is provided. Formally, in that case, the tree can be converted into a set of splitting decisions  $\mathbb{S}_{\mathrm{edu}} = \{(i,j)\to k|i < k < j\}$  such that the constituent  $(i,j)$  gets split into two constituents  $(i,k)$  and  $(k,j)$  for  $k < j$ , i.e., we simply omit the special case of  $k = j$  as the EDUs are given. In other words, in our generalized formulation, discourse segmentation is just one extra step of parsing, and
can be done top-down end-to-end.
An example of our formalism of the parsing problem is shown in Fig. 1 for a discourse tree spanning over two sentences (44 tokens); for simplicity, we do not show the relation labels corresponding to the splitting decisions (marked by  $\rightarrow$ ). Since each splitting decision corresponds to one and only one internal node in the tree, it guarantees that the transformation from the tree to  $\mathbb{S}$  (and  $\mathbb{S}_{\mathrm{edu}}$ ) has a one-to-one mapping. Therefore, predicting the sequence of such splitting decisions is equivalent to predicting the discourse tree (DT).
Seq2Seq Parsing Model. In this work, we adopt a structure-then-label framework. Specifically, we factorize the probability of a DT into the probability of the tree structure and the probability of the relations (i.e., the node labels) as follows:
$$
P _ {\theta} (D T | \boldsymbol {x}) = P _ {\theta} (\mathbb {S}, \mathbb {L} | \boldsymbol {x}) = P _ {\theta} (\mathbb {L} | \mathbb {S}, \boldsymbol {x}) P _ {\theta} (\mathbb {S} | \boldsymbol {x}) \tag {2}
$$
where  $\pmb{x}$  is the input document, and  $\mathbb{S}$  and  $\mathbb{L}$  respectively denote the structure and labels of the DT. This formulation allows us to first infer the best tree structure (e.g., using beam search), and then find the corresponding labels.
As discussed, we consider the structure prediction problem as a sequence of splitting decisions to generate the tree in a top-down manner. We use a seq2seq pointer network (Vinyals et al., 2015) to model the sequence of splitting decisions (Fig. 3). We adopt a depth-first order of the decision sequence, which showed more consistent
Figure 3: Our discourse parser along with a few decoding steps for a given document. The input to the decoder at each step is the representation of the span to be split. We predict the splitting point using the biaffine function between the corresponding decoder state and the token-boundary encoder representations. The figure is for end-to-end parsing, where each EDU-corresponding span points to its right edge to mark the EDU. The coherence relations between the left and right spans are assigned using a label classifier after the (approximately) optimal tree structure is formed using beam search.
performance in our preliminary experiments than other alternatives, such as breath-first order.
First, we encode the tokens in a document  $\mathbf{x} = (x_0, \ldots, x_n)$  with a document encoder and get the token-boundary representations  $(h_0, \ldots, h_n)$ . Then, at each decoding step  $t$ , the model takes as input an internal node  $(i_t, j_t)$ , and produces an output  $y_t$  (by pointing to the token boundaries) that represents the splitting decision  $(i_t, j_t) \to k_t$  to split it into two child constituents  $(i_t, k_t)$  and  $(k_t, j_t)$ . For example, the initial span  $(0, 44)$  in Fig. 1 is split at boundary position 4, yielding two child spans  $(0, 4)$  and  $(4, 44)$ . If the span  $(0, 4)$  is given as an EDU (i.e., segmentation given), the splitting stops at  $(0, 4)$ , thus omitted in  $\mathbb{S}_{\mathrm{edu}}$  (Fig. 1). Otherwise, an extra decision  $(0, 4) \to 4 \in \mathbb{S}$  needs to be made to mark the EDUs for end-to-end parsing. With this, the probability of  $\mathbb{S}$  can be expressed as:
$$
\begin{array}{l} P _ {\theta} (\mathbb {S} | \boldsymbol {x}) = \prod_ {y _ {t} \in \mathbb {S}} P _ {\theta} (y _ {t} | y _ {<   t}, \boldsymbol {x}) \\ = \prod_ {t = 1} ^ {| \mathbb {S} |} P _ {\theta} \left(\left(i _ {t}, j _ {t}\right)\rightarrow k _ {t} \mid \left(\left(i, j\right)\rightarrow k\right) _ {<   t}, \boldsymbol {x}\right) \\ \end{array}
$$
This end-to-end conditional splitting formulation is the main novelty of our method and is in contrast to previous approaches which rely on offline-inferred EDUs from a separate discourse segmenter. Our formalism streamlines the overall parsing process, unifies the neural components seamlessly and smoothes the training process.
In the following, we describe the components of our parsing model: the document encoder, the
boundary and span representations, the decoding process through the decoder and the label classifier.
Document Encoder. Given an input document of  $n$  words  $\pmb{x} = (x_{1},\dots ,x_{n})$ , we first add  $< \mathrm{sod}>$  and  $< \mathrm{eod}>$  markers to the sequence. After that, each token  $x_{i}$  in the sequence is mapped into its dense vector representation  $e_i$  as:  $e_i = [e_i^{\mathrm{char}},e_i^{\mathrm{word}}]$ , where  $e_i^{\mathrm{char}}$ , and  $e_i^{\mathrm{word}}$  are respectively the character and word embeddings of token  $x_{i}$ . For word embedding, we experiment with (i) randomly initialized, (ii) pretrained static embeddings, e.g., GloVe (Pennington et al., 2014). To represent the character embedding of a token, we apply a character bidirectional LSTM i.e., BiLSTM (Hochreiter and Schmidhuber, 1997) or pretrained contextualized embeddings, e.g., XLNet (Yang et al., 2019). The token representations are then passed to a sequence encoder of a three-layer Bi-LSTM to obtain their forward  $f_{i}$  and backward  $b_{i}$  contextual representations.
Token-boundary Span Representations. To represent each token-boundary position  $k$  between token positions  $k$  and  $k + 1$ , we use the fencepost representation (Cross and Huang, 2016):
$$
\boldsymbol {h} _ {k} = \left[ \boldsymbol {f} _ {k}; \boldsymbol {b} _ {k + 1} \right] \tag {3}
$$
where  $\pmb{f}_k$  and  $\pmb{b}_{k + 1}$  are the forward and backward LSTM hidden vectors of positions  $k$  and  $k + 1$  respectively, and  $[\cdot ;\cdot ]$  is the concatenation operation.
Then, to represent the token-boundary span  $(i,j)$ , we use the linear combination of the two endpoints  $i$  and  $j$  as:
$$
\boldsymbol {h} _ {i, j} = \boldsymbol {W} _ {1} \boldsymbol {h} _ {i} + \boldsymbol {W} _ {2} \boldsymbol {h} _ {j} \tag {4}
$$
Figure 4: Illustration of token-boundary span encoder. The figure lays out an example representation for the boundary at 2 and the representation of the token-boundary span (0, 44), which corresponds to the whole document.
where  $W_{1}$  and  $W_{2}$  are trainable weights. These span representations will be used as input to the decoder or the label classifier. Fig. 4 illustrates an example boundary span representation.
The Decoder. Our model uses a unidirectional LSTM as the decoder. At each decoding step  $t$ , the decoder takes as input the corresponding span  $(i,j)$  (i.e.,  $h_{i,j}$ ) and its previous LSTM state  $d_{t-1}$  to generate the current state  $d_t$  and then the bioaffine function (Dozat and Manning, 2017) is applied between  $d_t$  and all the encoded token-boundary representations  $(h_0,h_1,\dots ,h_n)$  as follows:
$$
\boldsymbol {d} _ {t} ^ {\prime} = \operatorname {M L P} _ {d} \left(\boldsymbol {d} _ {t}\right) \quad \boldsymbol {h} _ {i} ^ {\prime} = \operatorname {M L P} _ {h} \left(\boldsymbol {h} _ {i}\right) \tag {5}
$$
$$
s _ {t} ^ {i} = \boldsymbol {d} _ {t} ^ {\prime T} \boldsymbol {W} _ {d h} \boldsymbol {h} _ {i} ^ {\prime} + \boldsymbol {h} _ {i} ^ {\prime T} \boldsymbol {w} _ {h} \tag {6}
$$
$$
a _ {t} ^ {i} = \frac {\exp \left(s _ {t} ^ {i}\right)}{\sum_ {i = 0} ^ {n} \exp \left(s _ {t} ^ {i}\right)} \quad \text {f o r} i = 0, \dots , n \tag {7}
$$
where each MLP operation comprises a linear transformation with LeakyReLU activation (Maas et al., 2013) to transform  $\pmb{d}_i$  and  $\pmb{h}_i$  into equal-sized vectors  $\pmb{d}_t^{\prime}, \pmb{h}_i^{\prime} \in \mathbb{R}^{d}$ , and  $\pmb{W}_{dh} \in \mathbb{R}^{d \times d}$  and  $\pmb{w}_h \in \mathbb{R}^d$  are respectively the weight matrix and weight vector for the biaffine function. The resulting biaffine scores  $s_t^i$  are then fed into a softmax layer to acquire the pointing distribution  $\pmb{a}_t^i \in [0,1]^{n+1}$  for the splitting decision. During inference, when decoding the tree at step  $t$ , we only examine the "valid" splitting points between  $i$  and  $j$ , and we look for  $k$  such that  $i < k \leq j$ .
Label Classifier. We perform label assignment after decoding the entire tree structure. Each assignment takes into account the splitting decision that generated it since the label represents the relation between the child spans. Specifically, for a constituent  $(i,j)$  that was split into two child constituents  $(i,k)$  and  $(k,j)$ , we determine the coherence relation between them as follows:
$$
\boldsymbol {h} _ {i k} ^ {l} = \operatorname {M L P} _ {l} ([ \boldsymbol {h} _ {i}; \boldsymbol {h} _ {k}]); \quad \boldsymbol {h} _ {k j} ^ {r} = \operatorname {M L P} _ {r} ([ \boldsymbol {h} _ {k}; \boldsymbol {h} _ {j} ]) \tag {8}
$$
$$
P _ {\theta} (l | (i, k), (k, j)) = \operatorname {s o f t m a x} \left(\left(\boldsymbol {h} _ {i k} ^ {l}\right) ^ {T} \boldsymbol {W} _ {l r} \boldsymbol {h} _ {k j} ^ {r}\right.
$$
$$
\left. + \left(\boldsymbol {h} _ {i k} ^ {l}\right) ^ {T} \boldsymbol {W} _ {l} + \left(\boldsymbol {h} _ {k j} ^ {r}\right) ^ {T} \boldsymbol {W} _ {r} + \boldsymbol {b}\right) \tag {9}
$$
$$
l _ {(i, k), (k, j)} ^ {*} = \underset {l \in L} {\arg \max } P _ {\theta} (l | (i, k), (k, j)) \tag {10}
$$
where  $L$  is the total number of labels (i.e., coherence relations with nuclearity attached); each of  $\mathrm{MLP}_l$  and  $\mathrm{MLP}_r$  includes a linear transformation with LeakyReLU activation to transform the left and right spans into equal-sized vectors  $\pmb{h}_{ik}^{l},\pmb{h}_{kj}^{r}\in \mathbb{R}^{d};\pmb{W}_{lr}\in \mathbb{R}^{d\times L\times d},\pmb{W}_{l}\in \mathbb{R}^{d\times L},\pmb{W}_{r}\in \mathbb{R}^{d\times L}$  are the weights and  $\pmb{b}$  is a bias vector.
Training Objective. Our parsing model is trained by minimizing the total loss defined as:
$$
\mathcal {L} \left(\theta_ {e}, \theta_ {d}, \theta_ {l}\right) = \mathcal {L} _ {s} \left(\theta_ {e}, \theta_ {d}\right) + \mathcal {L} _ {l} \left(\theta_ {e}, \theta_ {l}\right) \tag {11}
$$
where structure  $\mathcal{L}_s$  and label  $\mathcal{L}_l$  losses are cross-entropy losses computed for the splitting and labeling tasks respectively, and  $\theta_e$ ,  $\theta_d$  and  $\theta_l$  denote the encoder, decoder and labeling parameters.
2.3 Complete Discourse Parsing Models
Having presented the generic framework, we now describe how it can be easily adapted to the two parsing scenarios: (i) end-to-end parsing and (ii) parsing with EDUs. We also describe the incorporation of beam search for inference.
End-to-End Parsing. As mentioned, previous work for end-to-end parsing assumes a separate segmenter that provides EDU-segmented texts to the parser. Our method, however, is an end-to-end framework that produces both the EDUs as well as the parse tree in the same inference process. To guide the search better, we incorporate an inductive bias into our inference based on the finding that most sentences have a well-formed subtree in the document-level tree (Soricut and Marcu, 2003), i.e., discourse structure tends to align with the text structure (sentence boundary in this case); for example, Fisher and Roark (2007); Joty et al. (2013) found that more than  $95\%$  of the sentences have a well-formed subtree in the RST discourse treebank.
Our goal is to ensure that each sentence corresponds to an internal node in the tree. This can be achieved by a simple adjustment in our inference. When decoding at time step  $t$  with the span  $(i_t, j_t)$  as input, if the span contains  $M > 0$  sentence boundaries within it, we pick the one that
has the highest pointing score (Eq. 7) among the  $M$  alternatives as the split point  $k_{t}$ . If there is no sentence boundary within the input span  $(M = 0)$ , we find the next split point as usual. In other words, sentence boundaries in a document get the chance to be split before the token boundaries inside a sentence. This constraint is indeed similar to the 1S-1S (1 subtree for 1 sentence) constraint of Joty et al. (2013)'s bottom-up parsing, and is also consistent with the property that EDUs are always within the sentence boundary. Algorithm 1 illustrate the end-to-end inference algorithm.