Venue: EMNLP 2017

# Learning Contextually Informed Representations for Linear-Time

Discourse Parsing

Yang Liu &Mirella Lapata

Institute for Language, Cognition and Computation

School of Informatics, University of Edinburgh

10 Crichton Street, Edinburgh EH8 9AB

Yang.Liu2@ed.ac.uk,mlap@inf.edu.ac.uk

###### Abstract

Recent advances in RST discourse parsing have focused on two modeling paradigmisms: (a) high order parsers which jointly predict the tree structure of the discourse and the relations it encodes; or (b) linear-time parsers which are efficient but mostly based on local features. In this work, we propose a linear-time parser with a novel way of representing discourse constituents based on neural networks which takes into account global contextual information and is able to capture long-distance dependencies. Experimental results show that our parser obtains state-of-the art performance on benchmark datasets, while being efficient (with time complexity linear in the number of sentences in the document) and requiring minimal feature engineering.

## 1 Introduction

The computational treatment of discourse phenomena has recently attracted much attention, due to their increasing importance for potential applications. Knowing how text units can be composed into a coherent document and how they relate to each other e.g., whether they express contrast, cause, or elaboration, can usefully aid downstream tasks such summarization Yoshida et al. (2014), question answering Chai and Jin (2004), and sentiment analysis Somasundaran (2010).

Rhetorical Structure Theory (RST, Mann and Thompson 1988), one of the most influential frameworks in discourse processing, represents texts by trees whose leaves correspond to Elementary Discourse Units (EDUs) and whose nodes specify how these and larger units (e.g., multi-sentence segments) are linked to each other by rhetorical relations. Discourse units are further characterized in terms of their importance in textbook: nuclei denote central segments, whereas satellites denote peripheral ones. Figure 1 shows an example of a discourse tree representing two sentences with three EDUs (\(e_{1},e_{2},\text{ and }e_{3}\)). EDUs \(e_{1}\) and \(e_{2}\) are connected with a mononuclear relation (i.e., _Consequence_), where \(e_{1}\) is the nucleus and \(e_{2}\) the satellite (indicated by the left pointing arrow in the figure). Span \(e_{1:2}\) is related to \(e_{3}\) via _List_, a multi-nuclear relation, expressing the fact that both spans are equally important and therefore both nucleus.

Given such tree-based representations of discourse structure, it is not surprising that RST-style document analysis is often viewed as a parsing task. State-of-the-art performance on RST parsing is achieved by cubic-time parsers Li et al. (2016); Li et al. (2016), with \(O(n^{3})\) time complexity (where \(n\) denotes the number of sen

Figure 1: Example text (bottom) composed of two sentences (three EDUs) and its RST discourse tree representation (top).



tences in the document). These systems model the relations between all possible adjacent discourse segments and use a CKY-style algorithm to generate a global optimal tree. The high order complexity renders such parsers inefficient in practice, especially when processing large documents. As a result, more efficient linear-time discourse parsers have been proposed [14, 20] which make local decisions and model the structure of the discourse and its relations separately. In this case, features are extracted from a local context (i.e., a small window of discourse constituents) without considering document-level information, which has been previously found useful in discourse analysis [13].

In this paper, we propose a simple and efficient linear-time discourse parser with a novel way of learning contextual representations for discourse constituents. To guarantee linear-time complexity, we use a two-stage approach: we first parse each sentence in a document into a tree whose leaves correspond to EDUs, and then parse the document into a tree whose leaves correspond to already pre-processed sentences. The feature learning process for both stages is based on neural network models. At the sentence level, Long-Short Term Memory Networks (LSTMs; hochreiter2005learning lstm_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn__rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn__rnn_rnn_rnn_rnn_rnn_rnn_rnn_rnn__ 

transition stack and queue. Li et al. (2014) proposed a CKY-based parser which uses recursive neural networks to learn representations for EDUs and their composition during the tree-building process. More recently, Li et al. (2016) designed a CKY-based parser which uses LSTMs to model text spans and a tensor-based transformation to compose adjacent spans.

Our own work joins others Feng and Hirst (2014); Joty et al. (2013) in adopting a two-stage architecture for our discourse parser. However, rather than considering each discourse constituent independently, we learn contextually-aware representations capturing long-range dependencies across sentences and documents. Discourse constituents at all levels are modeled with recurrent neural networks adopting a relatively simple, yet efficient, architecture compared to previously proposed neural systems Li et al. (2016). Finally, we experimentally assess whether sequence-based representations are expressive enough by comparing them to those obtained (with recursive neural networks) from structured inputs.

## 3 Discourse Parser Overview

In RST discourse parsing, a document is first segmented into EDUs and a parser then builds a discourse tree with the EDUs as leaves. The first subtask is considered relatively easy with state-of-art accuracy at above 90% Hernault et al. (2010). As a result, recent research focuses on the second subtask and often uses manual EDU segmentation.

Joty et al. (2013) found that a two-stage parsing strategy, which separates intra-sentential from multi-sentential parsing, has some advantages for document-level discourse parsing, since the distribution of discourse relations and useful features are different in the two stages. Based on their findings, our model also follows a two-stage approach and is composed by two components: an _intra-sentential parser_ and a _multi-sentential parser_. Given a document pre-segmented into EDUs, our intra-sentential parser first builds a sentence-level discourse tree for individual sentences. Then, our multi-sentential parser creates a discourse tree for the entire document.

To guarantee linear-time complexity, both parsers adopt a greedy bottom-up tree-building process Hernault et al. (2010) and are based on two conditional random field (CRF) models, one for creating discourse structure and another one for assigning relations. The intra-sentential parser considers adjacent EDUs and decides whether they should be connected (based on the scores predicted by the first CRF) and their relation (based on predictions of the second CRF). The multi-sentential parser follows the same procedure while operating over sentences.

The CRFs employ feature representations which we obtain using neural networks. Specifically, discourse constituents at all levels are modeled with recurrent neural networks (see Section 4.2 for details). In addition, for inter-sentential constituents, we complement the flat text span representation with recursive neural networks (see Section 4.4). We argue that the combination is advantageous; a sequential text span representation is in principle unsuitable for capturing hierarchical discourse structure, whereas a tree-based representation can be more precise, albeit less robust (due to the accumulation of errors from the recursive tree-building process).

## 4 Parsing Model

### Intra-sentential Parser

To parse a sentence, we start with EDUs, which can be viewed as discourse constituents at the first level. As mentioned earlier, our intra-sentential parser is based on two linear-chain CRFs, the structure CRF decides which pair of constituents should be merged at the current level and the relation CRF assigns discourse relations to non-leaf constituents. For example, let \(C_{1}=\{e_{1},e_{2},\cdots,e_{m}\}\) denote a sentence with a sequence of EDUs, where \(e_{i}\) is the \(i^{th}\) EDU in the sentence. Suppose the structure CRF decides to merge together \(e_{2}\) and \(e_{3}\), then the next-level sequence is \(C_{2}=\{e_{1},e_{2:3},e_{4},\cdots,e_{m}\}\) and the relation CRF will assign a discourse relation to \(e_{2:3}\), the only non-leaf constituent so far. This process iterates until all EDUs are merged and a discourse subtree is generated for the entire sentence.

A linear-chain CRF for intra-sentential discourse parsing is shown in Figure 2. Here, \(C=\{c_{0},\cdots,c_{t},\cdots,c_{n}\}\) are observed discourse constituents and \(L=\{l_{1},\cdots,l_{t},\cdots,l_{n}\}\) are hidden structure nodes; label \(l_{l}\in\{1,0\}\) denotes whether constituents \(c_{t}\) and \(c_{t+1}\) should be connected. Our model differs from standard linear-chain CRFs in that the score between adjacent hidden nodes is not calculated based on a transition matrix, but is learned from observations instead. Specifically, 

given a constituent sequence \(C\), the probability of forming the structure sequence \(Y^{str}\) is written as:

\[p(Y^{str}|C)=\frac{1}{Z}\prod_{t=1}^{n}exp(u_{t}^{str}+b_{t,t+1}^{ str}) \tag{1}\] \[u_{t}^{str}=\theta(c_{t},c_{t+1},y_{t})\] (2) \[b_{t,t}^{sr}=\gamma(c_{t-1},c_{t},c_{t+1},y_{t},y_{t+1}) \tag{3}\]

where the \(u_{t}^{str}\) represents the unary potential s-core of structure node \(I_{t}=y_{t}\), depending on constituents \(c_{t}\) and \(c_{t+1}\). The binary potential score \(b_{t,t+1}^{str}\) for \(s_{t}=y_{t},s_{t+1}=y_{t+1}\) is calculated based on \(c_{t-1},c_{t}\) and \(c_{t+1}\). The binary potential provides information for discriminating between \(s_{t}=0,s_{t+1}=1\) and \(s_{t}=1,s_{t+1}=0\). Also, we impose the constraint that one constituent can be merged with at most one other adjacent constituent.

Figure 3 depicts the relation CRF for intra-sentential parsing. Similar to the structure CRF, \(C=\{c_{0},\cdots,c_{t},\cdots,c_{n}\}\) are observed constituents and \(R=\{r_{1},\cdots,r_{t},\cdots,r_{n}\}\) hidden nodes, corresponding to discourse relations. The CRF will assign a relation to a non-leaf constituent \(c_{t}\) based on its right and left children, \(c_{t,L}\) and \(c_{t,R}\), respectively. If a constituent is a single EDU, we force its hidden node to be the special label \(LEAF\), whereas hidden nodes for constituents which are the product of merging cannot be \(LEAF\). Given a sequence of constituents \(C\), the probability of the relation label sequence \(Y^{rel}\) can be written as:

\[p(Y^{rel}|C)=\frac{1}{Z}\prod_{t=1}^{n}exp(u_{t}^{rel}+b_{t,t+1}^{ rel}) \tag{4}\] \[u_{t}^{rel}=\theta(c_{t},y_{t})\] (5) \[b_{t,t+1}^{rel}=\gamma(c_{t},c_{t+1},y_{t},y_{t+1}) \tag{6}\]

### Intra-sentential Feature Learning

Instead of adopting a high order parsing model, we use neural networks to capture contextual information and recover the meaning of discourse constituents. Aside from modeling long distance dependencies, our representation learning process alleviates the need for elaborate feature engineering and selection. Our approach is based on L-STMs Hochreiter and Schmidhuber (1997) which have recently emerged as a popular architecture for modeling sequences and have been successfully applied to a variety of tasks ranging from machine translation Sutskever et al. (2014), to speech recognition Graves et al. (2013), and image description generation Vinyals et al. (2015). LSTMs have also been incorporated into syntactic parsing in a variety of ways Vinyals et al. (2015); Kiperwasser and Goldberg (2016); Dyer et al. (2015, _inter alia_). Of particular relevance to this work is LSTM-minus, a method for learning embeddings of text spans, which has achieved competitive performance in both dependency and constituency parsing Wang and Chang (2016); Cross and Huang (2016). We describe below how we extend this method which is based on subtraction between LSTM hidden vectors to discourse parsing.

We represent each sentence as a sequence of word embeddings \([\mathbf{w}_{sss},\mathbf{w}_{1},\cdots,\mathbf{w}_{i},\cdots,\mathbf{w}_{n},\mathbf{w}_{eos}]\) and insert a special embedding \(w_{E}\) to indicate the boundaries of EDUs. We run a bidirectional LSTM over the sentence and obtain the output vector sequence \([\mathbf{h}_{0},\cdots,\mathbf{h}_{i},\cdots,\mathbf{h}_{t}]\), where \(\mathbf{h}_{i}=[\vec{\mathbf{h}}_{i},\vec{\mathbf{h}}_{i}]\) is the output vector for the \(i^{th}\) word, and \(\vec{\mathbf{h}}_{i}\) and \(\vec{\mathbf{h}}_{i}\) are the output vectors from the forward and backward directions, respectively. We represent a constituent \(c\) from position \(a\) to \(b\) with a span vector \(\mathbf{sp}\) which is the concatenation of the vector d-ifferences \(\vec{\mathbf{h}}_{b+1}-\vec{\mathbf{h}}_{a}\) and \(\vec{\mathbf{h}}_{a-1}-\vec{\mathbf{h}}_{b}\):

\[\mathbf{sp}=[\vec{\mathbf{h}}_{b+1}-\vec{\mathbf{h}}_{a},\vec{\mathbf{h}}_{a-1}-\vec{\mathbf{h}}_{ b}] \tag{7}\]

As illustrated in Figure 4, spans are represented using output from both backward and forward L-STM components. Intuitively, this allows to obtain representations for EDUs and larger constituents in context, as embeddings are learned based on in

Figure 3: Intra-sentential relation CRF with pairwise modeling.

Figure 2: Intra-sentential structure CRF with pairwise modeling.

 

formation from the span itself and the words surrounding it.

The structure CRF model calculates the unary potential score \(u_{t}^{str}\) and the binary potential s-core \(b_{t,t+1}^{gr}\) based on the span vector as follows:

\[\mathbf{u}_{t}^{str}=\mathbf{W}_{u}^{str}[\mathbf{s}\mathbf{p}_{j},\mathbf{s}\mathbf{p}_{ j+1}] \tag{8}\] \[\mathbf{b}_{t,t+1}^{sr}=\mathbf{W}_{b}^{str}[\mathbf{s}\mathbf{p}_{j-1},\mathbf{s}\mathbf{ p}_{j},\mathbf{s}\mathbf{p}_{j+1}] \tag{9}\]

where \(\mathbf{s}\mathbf{p}_{j}\) is the span vector for the \(j^{th}\) constituent in the current sequence; and \(\mathbf{W}_{u}^{str}\in\mathbb{R}^{d\times 2}\), \(\mathbf{W}_{b}^{str}\in\mathbb{R}^{d\times 4}\) are weight matrices. \(\mathbf{b}_{t}^{tr}\) is reshaped into a \(2\times 2\) matrix, where the \((i,j)^{th}\) entry indicates the score of the transition from label \(i\) to label \(j\) between constituent \(c_{t}\) and \(c_{t+1}\).

Analogously, unary and binary potential scores for the relation CRF are calculated as:

\[\mathbf{u}_{t}^{rel}=\mathbf{W}_{u}^{rel}[\mathbf{s}\mathbf{p}_{j,L},\mathbf{s}\mathbf{p} _{j,R}] \tag{10}\] \[\mathbf{b}_{t,t+1}^{rel}=\mathbf{W}_{b}^{rel}[\mathbf{s}\mathbf{p}_{j,L},\mathbf{s} \mathbf{p}_{j,R},\mathbf{s}\mathbf{p}_{j+1,L},\mathbf{s}\mathbf{p}_{j+1,R}] \tag{11}\]

where \(\mathbf{s}\mathbf{p}_{j,L}\) and \(\mathbf{s}\mathbf{p}_{j,R}\) are span vector representations for the left and right child of the \(j^{th}\) constituent; \(\mathbf{W}_{u}^{rel}\in\mathbb{R}^{d\times n_{r}},\mathbf{W}_{b}^{rel}\in\mathbb{R}^{ d\times n_{r}^{2}}\), and \(n_{r}\) is the number of discourse relations; \(\mathbf{b}_{t}^{rel}\) is also reshaped into a \(n_{r}\times n_{r}\) matrix. For a constituent that is a s-single EDU, \(\mathbf{s}\mathbf{p}_{j,R}\) and \(\mathbf{s}\mathbf{p}_{j+1,R}\) are special vectors \(\mathbf{w}_{LEAF_{l}}\) and \(\mathbf{w}_{LEAF_{R}}\).

### Multi-sentential Parser

The multi-sentential discourse parser treats sentences as the smallest possible discourse units, following a process similar to the intra-sentential model. Unfortunately, it is practically unfeasible to model all constituents with one CRF when processing entire documents. The forward-backward algorithm for calculating the CRF normalization factor on a sequence with \(T\) units has time complexity \(O(TM^{2})\), where \(M\) is the number of labels, leading to \(O(T^{2}M^{2})\) time for parsing a document.

We therefore modify the two CRFs into sliding-window versions. Specifically, at each level, when decoding a constituent sequence, for each hidden structure node \(l_{i}\) or relation node \(r_{i}\), we find all windows of constituents that contain the hidden node, and set the hidden node's label according to the window with the maximum joint probability.

 

We present the modified sliding-window CRFs in Figure 5 (structure) and Figure 6 (relation).

### Multi-sentential Feature Learning

For multi-sentential parsing, we also use the LSTM-minus method to model constituents, however, the minimum units here are sentences rather than EDUs. We represent individual sentences with the same bidirectional LSTM used for intra-sentential parsing. For sentence \(s_{i}\), the LSTM output vector of the first token \(\mathbf{h}_{0}\) and the last token \(\mathbf{h}_{n}\) are taken to form the sentence representation \(\mathbf{v}_{i}\):

\[\mathbf{v}_{i}=[\mathbf{h}_{0},\mathbf{h}_{n}] \tag{12}\]

In order to capture additional contextual information, we then build another bidirectional LSTM over a sequence of sentence vectors. We learn representations for constituents (aka text spans within a document) with the LSTM-minus method and use \(\mathbf{fl}\) to denote the resulting vectors. Although this flat representation is relatively straightforward to obtain, it is perhaps overly simplistic for modeling documents where the number of sentences can be relative large. Moreover, it is not clear that it is discriminating enough for modeling relations between constituents. Such relations follow structural regularities (e.g., they can be asymmetric or symmetric, left- or right-branching) which cannot be captured when adopting a sequence-based document view.

To inject structural knowledge in our representation, we also model constituents as subtrees with recursive neural networks Socher et al. (2012). The latter operate over tree structures which we obtain during training from the RST Discourse Treebank Carlson et al. (2003). The representation for each parent is computed based on its children iteratively, in a bottom-up fashion. More formally, let vectors \(\mathbf{h}_{L}\) and \(\mathbf{h}_{R}\) denote the left and right children of constituent \(c\) and \(dis\) their rhetorical relation. The vector for parent \(c\) is:

\[\mathbf{h}_{c}=\text{tanh}(\mathbf{W}_{dis}[\mathbf{h}_{L},\mathbf{h}_{R}]+\mathbf{b}_{dis}) \tag{13}\]

where \([\mathbf{h}_{L},\mathbf{h}_{R}]\) is the concatenation of the children representations \(\mathbf{h}_{L}\in\mathbb{R}^{d}\) and \(\mathbf{h}_{R}\in\mathbb{R}^{d}\), \(\mathbf{W}_{dis}\in\mathbb{R}^{2d\times d}\), and \(\mathbf{b}_{dis}\in\mathbb{R}^{d}\) is the bias vector. We henceforth use the term tree vector \(\mathbf{tr}\) to refer to the representation in Equation (13) since constituents are now subtrees in a document-wide discourse tree.

In multi-sentential parsing, the span vector \(\mathbf{sp}\) now becomes the concatenation of the flat vector \(\mathbf{fI}\) and the tree vector \(\mathbf{tr}\):

\[\mathbf{sp}=[\mathbf{fl},\mathbf{tr}] \tag{14}\]

The representation above, attempts to capture richer semantic features during the tree building process while benefiting from the robustness afforded by the flat LSTM-based text spans. An example of this representation is given in Figure 7.

Unary and binary potential scores for the structure and relation CRFs are calculated as in intra-sentential parsing (see Section 4.2).

### Training

We first train the intra-sentential parser and use the learned LSTM as a component of the multi-sentential parser. During training, we maximize the log-likelihood of the correct label sequence \(Y\) given a constituent sequence \(C\), which is \(p(Y^{str}|C)\) for the structure CRFs and \(p(Y^{rel}|C)\) for the relation CRFs. Stochastic gradient descent with momentum is used to update the parameters of the network. In our experiments, the momentum is set to 0.9 and the learning rate is 0.001. The LSTMs in our paper have one hidden layer.

## 5 Experimental Setup

In this section we present our experimental setup for assessing the performance of the discourse parser described above. We give details on the

Figure 7: Document-level constituents with LSTM-minus features. Blue nodes are LSTM outputs for words, light green nodes represent vectors for sentences, dark green nodes are LSTM outputs for sentences, and black nodes are learned constituent representations.

 

datasets we used, evaluation protocol, and model training.

EvaluationWe evaluated our model on the RST Discourse Treebank (RST-DT; Carlson et al.2003), which is partitioned into 347 documents for training and 38 documents for testing. Following previous work Joty et al. (2013); Li et al. (2016), we converted non-binary relations into a cascade of right-branching binary relations.

Predicted RST-trees are typically evaluated by computing F1 against gold standard trees Marcu (2000). Evaluation metrics for RST-style discourse parsing include: (a) _span_ (S) which measures whether the predicted subtrees match the goldstandard; (b) _nucleus_ (N) which measures whether subtrees have the same nucleus as in the goldstandard and (c) _relation_ (R) which measures whether discourse relations have been identified correctly. The three metrics are interdependent, errors on the span metric propagate to the nuclearity metric, and in turn to the relation metric. Following other RST-style discourse parsing systems Hernault et al. (2010), we evaluate the relation metric using 18 coarse-grained relation classes, and with nuclearity attached, we have a total of 41 distinct relations.1 Since EDU segmentation falls outside the scope of this work, we evaluate our system on gold-standard EDUs. Comparison systems are also assessed in the same setting.

Footnote 1: For calculating the binary potential scores, 41 relations will lead to a large number of parameters; to avoid this, we only use 18 relations without nuclearity.

Training DetailsWord embeddings were pre-trained with the Gensim2 implementation of word2vec Mikolov et al. (2013) on the English GigaWord corpus (with case left intact). The dimensionality of the word embeddings was set to 50. Following Li et al. (2016), the embeddings were fine-tuned using a mapping matrix \(\boldsymbol{W}\in\mathbb{R}^{50\times 50}\) trained with the following criterion:

Footnote 2: [https://radimrehurek.com/gensim/](https://radimrehurek.com/gensim/)

\[\min_{\boldsymbol{W},\boldsymbol{b}}\|\boldsymbol{LT}_{tuned}-\boldsymbol{ LT}_{pre}\boldsymbol{W}+\boldsymbol{b}\| \tag{15}\]

where \(\boldsymbol{LT}_{tuned}\), and \(\boldsymbol{LT}_{pre}\) are lookup tables for fine-tuned and pre-trained word embeddings in the training set. Matrix \(W\) can be subsequently used to to estimate fine-tuned embeddings for words in the test set.

Tokenization, POS tagging and sentence s-splitting were performed using the Stanford CoreNLP toolkit Manning et al. (2014). All neural network parameters were initialized randomly with Xavier's initialization Glorot and Bengio (2010). The hyper-parameters are tuned by cross-validation on the training set.

Additional FeaturesMost existing state-of-the-art systems rely heavily on handcrafted features Hernault et al. (2010); Feng and Hirst (2014); Joty et al. (2013) some of which have been also proved helpful in neural network models Li et al. (2014, 2016). In our experiments, we use the following basic features which have been widely adopted in various discourse parsing models: (1) the first three words and the last two words of each constituent; (2) the POS tags of the first three words and the last two words of each constituent; (3) the number of EDUs; and (4) the number of tokens of each constituent. We concatenate these features with the constituent vectors learned by our neural networks, and train new CRF models.

## 6 Results

In this paper we have presented two views for modeling discourse constituents, namely as trees or sequences. We experimentally assessed whether these two views are overlapping or complementary. Table 1 reports the performance of our parser which we call CIDer (as a shorthand for **C**ontextually **I**nformed **D**iscourse **Parser**) without the additional features introduced in Section 5. The first row presents a version of CIDer based solely on tree span representations. In the second row, CIDer uses flat representations without LSTM minus features. Specifically, each constituent is represented as the average of the LSTM output-s within it. In the third row, CIDer's representations are computed with the LSTM minus method, while the fourth row shows results for the full system.

As can be seen, on the span (S) metric, CIDer with flat span representations is much better than CIDer with tree span representations; on the nuclearity (N) metric tree representations are still inferior to flat representations but the performance

\begin{table}
\begin{tabular}{|l|p{28.5pt}|p{28.5pt}|p{28.5pt}|} \hline CIDer & S & N & R \\ \hline \hline Tree Span & 79.5 & 68.1 & 56.6 \\ Flat Span (\(-\)minus) & 82.7 & 69.3 & 55.6 \\ Flat Span (\(+\)minus) & 83.6 & 70.1 & 55.4 \\ Tree \(+\) Flat Span (\(+\)minus) & 83.6 & 71.1 & 57.3 \\ \hline  & \multicolumn{1}{p{28.5pt}}{} & \multicolumn{1}{p{28.5pt}}{} \\ \end{tabular}
\end{table}
Table 1: CIDer performance using different constituent representations (RST-DT test set).

 

gap is narrower, whereas on the the relation (R) metric, tree span representations are superior. We believe this can be explained by the fact that relation identification relies on more semantic information while span identification relies on more shallow features, like the beginning and the end of the span Ji and Eisenstein (2014). Span features based on the LSTM minus method bring improvements over vanilla LSTM representations on the s-pan and nuclearity metrics. Perhaps unsurprisingly, the combination of span and tree representations achieves the best results overall.

In Table 2, we compare our system with several state-of-the-art discourse parsers, which can be classified in two groups depending on their time complexity. Linear-time systems (first block in the table) include two transition-based parsers Ji and Eisenstein (2014); Heilman and Sagae (2015) and one CRF-based parser Feng and Hirst (2014), whereas cubic-time parsers (second block) include two neural network models Li et al. (2016); Li et al. (2016). CIDer falls in the first group as it is a liner-time parser, while it shares with parsers in the second group the use of neural architectures for automated feature extraction. We report CIDer scores with and without the additional features (AF) discussed in the previous section. As an upper bound, we also report inter-annotator agreement on the discourse parsing task (last row in the table).

Amongst linear-time systems, our parser achieves comparable results on the span and relation metric, and best performance on the nuclearity metric. Note that the three metrics evaluate different aspects of a discourse parser, and CIDer achieves the most balanced results across all metrics. As far as other comparison systems are concerned, Ji and Eisenstein (2014) employ a shift-reduce discourse parser. They represent EDUs with word-count vectors and use a projection matrix to combine them into text spans. A support vector machine classifier is used to decide the actions of the parser. heilman2015deep also adopt a shift-reduce approach and use multi-class logistic regression to select the best parsing action. Their classifier considers a variety of lexical, syntactic, and positional features. feng2014deep system is closest to ours in their use of linear-chain CRFs, but their features are mainly extracted from local constituents. Furthermore, they adopt a post-editing method which modifies the discourse trees their parser creates with height features.

With regard to previously proposed cubic-time systems, CIDer outperforms li2014deep across all metrics. Their CYK-based parser adopts a recursive deep model for composing EDUs hierarchically together with several additional features to boost performance. CIDer performs s-lightly worse on span and nuclearity compared to li2016deep, but is better at identifying relationships. Their system uses an attention-based hierarchical neural network for modeling text spans and a tensor-based transformation for combining two s-pans. A CKY-like algorithm is used to generate the discourse tree structure. In comparison, CIDer is conceptually simpler, and more efficient.

We used paired bootstrap re-sampling Efron and Tibshirani (1993) to assess whether differences in performance are statistically significant. CIDer is significantly better than Feng and Hirst's 2014 system on the relation metric (\(p<0.05\)); it is also significantly better (\(p<0.05\)) than heilman2015deep on all three metrics and better than ji2014deep on the span metric. Compared to li2014deep, CIDer is significantly better on the span and relation metrics (\(p<0.05\)). Unfortunately, we cannot perform significance tests against li2016deep as we do not have access to the output of their system.

We also evaluated the speed of CIDer and comparison discourse parsers on a platform with Intel Core-i5-7200U CPU at 2.50GHz. We report the average number of seconds taken to parse a document in the RST-DT test set. The times shown in Table 2 do not include pre-processing, which for CIDer is only part-of-speech tagging, whereas all other linear-time systems rely on a syntactic parser. As can be seen CIDer is quite efficient compared to related systems Feng and Hirst (2014);

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline Discourse Parsers & S & N & R & Speed \\ \hline Ji and Eisenstein (2014) & 82.1 & 71.1 & 61.6 & 0.21 \\ Feng and Hirst (2014) & 85.7 & 71.0 & 58.2 & 9.88 \\ Heiliman and Sagae (2015) & 83.5 & 68.1 & 55.1 & 0.40 \\ CIDer (\(-\)AF) & 83.6 & 71.1 & 57.3 & 3.80 \\ CIDer (+ AF) & 85.0 & 71.1 & 59.0 & 3.80 \\ \hline \hline Li et al. (2014) & 84.0 & 70.8 & 58.6 & 26.00 \\ Li et al. (2016) & 85.8 & 71.7 & 58.9 & – \\ \hline \hline Human & 88.7 & 77.7 & 65.8 & – \\ \hline \end{tabular}
\end{table}
Table 2: Comparison with state-of-the-art systems (RST-DT test set). Speed indicates the average number seconds taken to parse a document.

 

Heilman and Sagae, 2015) whilst requiring less feature engineering.

## 7 Conclusions

In this paper we described CIDer, a simple and efficient discourse parser which adopts a two-stage parsing strategy, whilst exploiting a more global feature space. We proposed a novel way to learn contextually informed representations of constituents with the LSTM minus method, at the sentence and document level. We also demonstrated that flat representations of text spans can be usefully complemented with tree-based ones leading to a more accurate characterization of discourse relations. Experimental results showed that CIDer performs on par with the state of the art (Li et al., 2016), despite the greedy parsing algorithm and relatively simple neural architecture. In the future, we would like to improve parsing accuracy by leveraging unlabeled text rather than relying exclusively on human annotated training data.

AcknowledgmentsThe authors gratefully acknowledge the support of the European Research Council (award number 681760).

## References

* Baldridge and Lascarides (2005) Jason Baldridge and Alex Lascarides. 2005. Probabilistic head-driven parsing for discourse structure. In _Proceedings of the CoNLL conference_. pages 96-103.
* Carlson et al. (2003) Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In _Current and new directions in discourse and dialogue_, Springer, pages 85-112.
* Chai and Jin (2004) Joyce Y. Chai and Rong Jin. 2004. Discourse structure for context question answering. In _HLT-NAACL 2004: Workshop on Pragmatics of Question Answering_. pages 23-30.
* Cross and Huang (2016) James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In _Proceedings of the EMNLP conference_. pages 1-11.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the ACL conference_. pages 334-343.
* Efron and Tibshirani (1993) Bradley Efron and Robert J. Tibshirani. 1993. _An Introduction ot the Bootstrap_. Chapman & Hall, New York, NY.
* Feng and Hirst (2012) Vanessa Wei Feng and Graemeo Hirst. 2012. Text-level discourse parsing with rich linguistic features. In _Proceedings of the ACL conference_. pages 60-68.
* Feng and first (2014) Vanessa Wei Feng and Graemeo Hirst. 2014. A lineartime bottom-up discourse parser with constraints and post-editing. In _Proceedings of the ACL conference_. pages 511-521.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of the 13th International Conference on Artificial Intelligence and Statistics_. volume 9, pages 249-256.
* Graves et al. (2013) Alex Graves, Jaitly Navdeep, and A-R Mohamed. 2013. Hybrid speech recognition with deep bidirectional lstm. In _IEEE Workshop on Automatic Speech Recognition and Understanding_. pages 293-278.
* Heilman and Sagea (2015) Michael Heilman and Kenji Sagea. 2015. Fast rhetorical structure theory discourse parsing. _arXiv preprint arXiv:1505.02425_ .
* Hernault et al. (2010) Hugo Hernault, Helmut Prendinger, David A DuVerle, Mitsuru Ishizuka, and Tim Paek. 2010. Hilda: a discourse parser using support vector machine classification. _Dialogue and Discourse_ 1(3):1-33.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Ji and Eisenstein (2014) Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In _Proceedings of the ACL conference_. pages 13-24.
* Joty et al. (2013) Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In _Proceedings of the ACL conference_. pages 486-496.
* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional lstm feature representations. _Transactions of the Association for Computational Linguistics_ 4:313-327.
* LeThanh et al. (2004) Huong LeThanh, Geetha Abeysinghe, and Christian Huyck. 2004. Generating discourse structures for written texts. In _Proceedings of the 20th international conference on Computational Linguistics_. page 329.
* Li et al. (2014) Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In _Proceedings of the EMNLP conference_. pages 2061-2069.
* Li et al. (2016) Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In _Proceedings of the EMNLP conference_. pages 362-371 

William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. _Text-Interdisciplinary Journal for the Study of Discourse_ 8(3):243-281.
* Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Proceedings of the ACL conference: System Demonstrations_. pages 55-60.
* Marcu (2000) Daniel Marcu. 2000. The rhetorical parsing of unrestricted texts: A surface-based approach. _Computational linguistics_ 26(3):395-448.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_ .
* Prasad et al. (2008) Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn discourse treebank 2.0. In _Proceedings of LREC_.
* Sagae (2009) Kenji Sagae. 2009. Analysis of discourse structure with syntactic dependencies and data-driven shift-reduce parsing. In _Proceedings of the 11th International Conference on Parsing Technologies_. pages 81-84.
* Socher et al. (2012) Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In _Proceedings of the EMNLP conference_. pages 1201-1211.
* Somasundaran (2010) Swapna Somasundaran. 2010. _Discourse-level Relation for Opinion Analysis_. Ph.D. thesis, University of Pittsburgh.
* Soricut and Marcu (2003) Radu Soricut and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In _Proceedings of the NAACL conference_. pages 149-156.
* Subba and Di Eugenio (2009) Rajen Subba and Barbara Di Eugenio. 2009. An effective discourse parser that uses rich linguistic information. In _Proceedings of the ACL conference_. pages 566-574.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of the NIPS conference_. pages 3104-3112.
* Vinyals et al. (2015a) Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015a. Grammar as a foreign language. In _Proceedings of the NIPS conference_. pages 2773-2781.
* Vinyals et al. (2015b) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015b. Show and tell: A neural image caption generator. In _Proceedings of the CVPR conference_. pages 3156-3164.
* Wang and Chang (2016) Wenhui Wang and Baobao Chang. 2016. Graph-based dependency parsing with bidirectional lstm. In _Proceedings of the ACL conference_. pages 2306-2315.
* Yoshida et al. (2014) Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In _Proceedings of the EMNLP conference_. pages 1834-1839.