Venue: EMNLP 2017

# Joint Syntaco-Discourse Parsing and the Syntacto-Discourse Treebank +
Footnote †: The source code and the joint treebank are available at [https://github.com/kaayy/josydipa](https://github.com/kaayy/josydipa).

 Kai Zhao   Liang Huang

School of Electrical Engineering and Computer Science

Oregon State University

Corvallis, Oregon, USA

{kzhao.hf, liang.huang.sh}@gmail.com

###### Abstract

Discourse parsing has long been treated as a stand-alone problem independent from constituency or dependency parsing. Most attempts at this problem are pipelined rather than end-to-end, sophisticated, and not self-contained: they assume gold-standard text segmentations (Elementary Discourse Units), and use external parsers for syntactic features. In this paper we propose the first end-to-end discourse parser that jointly parses in both syntax and discourse levels, as well as the first syntacto-discourse treebank by integrating the Penn Treebank with the RST Treebank. Built upon our recent span-based constituency parser, this joint synatoc-discourse parser requires no preprocessing whatsoever (such as segmentation or feature extraction), achieves the state-of-the-art end-to-end discourse parsing accuracy.

## 1 Introduction

Distinguishing the semantic relations between segments in a document can be greatly beneficial to many high-level NLP tasks, such as summarization Louis et al. (2010); Yoshida et al. (2014), sentiment analysis Voll and Taboada (2007); Somasundaran et al. (2009); Bhatia et al. (2015), question answering Ferrucci et al. (2010); Jansen et al. (2014), and textual quality evaluation Tetreault et al. (2013); Li and Jurafsky (2016).

There has been a variety of research on discourse parsing Marcu (2000a); Soricut and Marcu (2003); Pardo and Nunes (2008); Hernault et al. (2010); da Cunha et al. (2012); Joty et al. (2013); Joty and Moschitti (2014); Feng and Hirst (2014); Ji and Eisenstein (2014); Li et al. (2014); Heilman and Sagae (2015); Wang et al. (2017). But most of them suffer from the following limitations:

1. _pipelined rather than end-to-end_: they assume pre-segmented discourse, and worse yet, use gold-standard segmentations, except Hernault et al. (2010);
2. _not self-contained_: they rely on external syntactic parsers and pretrained word vectors;
3. _complicated_: they design sophisticated features, including those from parse-trees.

We argue for the first time that discourse parsing should be viewed as an extension of, and be performed in conjunction with, constituency parsing. We propose the first _jointsyntacto-discourse treebank_, by unifying constituency and discourse tree representations. Based on this, we propose the first _end-to-end_ incremental parser that jointly parses at both constituency and discourse levels. Our algorithm builds up on the span-based parser Cross and Huang (2016); it employs the strong generalization power of bi-directional LSTMs, and parses efficiently and robustly with an extremely simple span-based feature set that does not use any tree structure information.

We make the following contributions:

1. We develop a combined representation of constituency and discourse trees to facilitate parsing at both levels without explicit conversion mechanism. Using this representation, we build and release a joint treebank based on the Penn Treebank Marcus et al. (1993) and RST Treebank Marcus (2000a, b) (Section 2).
2. We propose a novel joint parser that parses at both constituency and discourse levels. Our

parser performs discourse parsing in an end-to-end manner, which greatly reduces the efforts required in preprocessing the text for segmentation and feature extraction, and, to our best knowledge, is the first end-to-end discourse parser in literature (Section 3).
3. Even though it simultaneously performs constituency parsing, our parser does _not_ use any explicit syntactic feature, nor does it need any binarization of discourse trees, thanks to the powerful span-based framework of Cross and Huang (2016) (Section 3).
4. Empirically, our end-to-end parser outperforms the existing pipelined discourse parsing efforts. When the gold EDUs are provided, our parser is also competitive to other existing approaches with sophisticated features (Section 4).

## 2 Combined Representation & Treebank

We first briefly review the discourse structures in Rhetorical Structure Theory Mann and Thompson (1988), and then discuss how to unify discourse and constituency trees, which gives rise to our syntacto-discourse treebank PTB-RST.

### Review: RST Discourse Structures

In an RST discourse tree, there are two types of branchings. Most of the internal tree nodes are binary branching, with one _nucleus_ child containing the core semantic meaning of the current node, and one _satellite_ child semantically decorating the nucleus. Like dependency labels, there is a _relation_ annotated between each satellite-nucleus pair, such as "Background" or "Purpose". Figure 1(a) shows an example RST tree. There are also nonbinary-branching internal nodes whose children are conjunctions, e.g., a "List" of semantically similar EDUs (which are all nucleus nodes); see Figure 2(a) for an example.

### Syntacto-Discourse Representation

It is widely recognized that lower-level lexical and syntactic information can greatly help determining both the boundaries of the EDUs (i.e., discourse segmentation) Bach et al. (2012) as well as the semantic relations between EDUs Soricut and Marcu (2003); Hernault et al. (2010); Joty and Moschitti (2014); Feng and Hirst (2014); Ji and Eisenstein (2014); Li et al. (2014); Heilman and Sagae (2015). While these previous approaches rely on pre-trained tools to provide both EDU segmentation and intra-EDU syntactic parse trees, we instead propose to directly determine the low-level segmentations, the syntactic parses, and the high-level discourse parses using a single joint parser. This parser is trained on the combined trees of constituency and discourse structures.

We first convert an RST tree to a format similar

Figure 1: Examples of the RST discourse treebank and our syntacto-discourse treebank (PTB-RST).

 

to those constituency trees in the Penn Treebank Marcus et al. (1993). For each binary branching node with a nucleus child and a satellite child, we use the relation as the label of the converted parent node. The nucleus/satellite relation, along with the direction (either \(\leftarrow\) or \(\rightarrow\), pointing from satellite to nucleus) is then used as the label. For example, at the top level in Figure 2, we convert

For a conjunctive branch (e.g. "List"), we simply use the relation as the label of the converted node.

After converting an RST tree into the constituency tree format, we then replace each leaf node (i.e., EDU) with the corresponding syntactic (sub)tree from PTB. Given that the sentences in the RST Treebank Marcu (2000b) is a subset of that of PTB, we can always find the corresponding constituency subtrees for each EDU leaf node. In most cases, each EDU corresponds to one single (sub)tree in PTB, since the discourse boundaries generally do not conflict with constituencies. In other cases, one EDU node may correspond to multiple subtrees in PTB, and for these EDUs we use the lowest common ancestor of those subtrees in the PTB as the label of that EDU in the converted tree. E.g., if C-D is one EDU in the PTB tree it might be converted to \(\stackrel{{\text{Purpose}}}{{\text{B CA D}}}\) if the relation annonated in RST is B \(\stackrel{{\text{Purpose}}}{{\longrightarrow}}\)C-D.

Figures 1-2 are two examples of discourse trees and their combined syntacto-discourse trees.

### Joint PTB-RST Treebank

Using the conversion strategy described above we build the first joint syntacto-discourse treebank based on the Penn Treebank and RST Treebank. This PTB-RST treebank is released as a set of tools to generate the joint trees given Penn Treebank and RST Treebank data. During the alignment between the RST trees and the PTB trees, we only keep the common parts of the two trees.

We follow the standard training/testing split of the RST Treebank. In the training set, there are 347 joint trees with a total of 17,837 tokens, and the lengths of the discourses range from 30 to 2,199 tokens. In the test set, there are 38 joint trees with a total of 4,819 tokens, and the lengths vary from 45 to 2,607. Figure 3 shows the distribution of the discourse lengths over the whole dataset, which on average is about 2x of PTB sentence length, but longest ones are about 10x the longest lengths in the Treebank.

## 3 Joint Syntacto-Discourse Parsing

Given the combinedsyntacto-discourse treebank, we now propose a joint parser that can perform end-to-end discourse segmentation and parsing.

### Extending Span-based Parsing

As mentioned above, the input sequences are substantially longer than PTB parsing, so we choose linear-time parsing, by adapting a popular greedy constituency parser, the span-based constituency parser of Cross and Huang (2016).

As in span-based parsing, at each step, we maintain a a stack of spans. Notice that in conventional incremental parsing, the stack stores the subtrees

Figure 3: PTB-RST: length distribution (# tokens).

Figure 2: Another example of RST vs. PTB-RST, demonstrating a discourse tree over two sentences and a non-binary relation (List). The lower levels of the PTB-RST tree are collapsed due to space contraints.

 

constructed so far, but in span-based constituency parsing, the stack only stores the boundaries of subtrees, which are just a list of indices \(...i{\bigtriangleup}_{k}{\bigtriangleup}_{j}\). In other words, quite shockingly, no tree structure is represented anywhere in the parser. Please refer Cross and Huang (2016) for details.

Similar to span-based constituency parsing, we alternate between structural (either shift or combine) and label (label\({}_{X}\) or nolabel) actions in an odd-even fashion. But different from Cross and Huang (2016), after a structural action, we choose to keep the last branching point \(k\), i.e., \({}_{j}\) (mostly for combine, but also trivially for shift). This is because in our parsing mechanism, the discourse relation between two EDUs is actually determined after the previous combine action. We need to keep the splitting point to clearly find the spans of the two EDUs to determine their relations. This midpoint \(k\) disappears after a label action; therefore we can use the shape of the last span on the stack (whether it contains the split point, i.e., \({}_{i}\) / \({}_{j}\) or \({}_{i}\) / \({}_{j}\)) to determine the parity of the step and thus no longer need to carry the step \(z\) in the state as in Cross and Huang (2016).

The nolabel action makes the binarization of the discourse/constitucency tree unnecessary, because nolabel actually combines the top two spans on the stack \(\sigma\) into one span, but without annotating the new span a label. This greatly simplifies the preprocessing and post-processing efforts needed.

### Recurrent Neural Models and Training

The scoring functions in the deductive system (Figure 4) are calculated by an underlying neural model, which is similar to the bi-directional LSTM model in Cross and Huang (2016) that evaluates based on span boundary features. Again, it is important to note that no discourse or syntactic tree structures are represented in the features.

During the decoding time, a document is first passed into a two-layer bi-directional LSTM model, then the outputs at each text position of the two layers of the bi-directional LSTMs are concatenated as the positional features. The spans at each parsing step can be represented as the feature vectors at the boundaries. The span features are then passed into fully connected networks with softmax to calculate the likelihood of performing the corresponding action or marking the corresponding label.

We use the "training with exploration" strategy Goldberg and Nivre (2013) and the dynamic oracle mechanism described in Cross and Huang (2016) to make sure the model can handle unseen parsing configurations properly.

## 4 Empirical Results

We use the treebank described in Section 2 for empirical evaluation. We randomly choose 30 documents from the training set as the development set.

We tune the hyperparameters of the neural model on the development set. For most of the hyperparameters we settle with the same values suggested by Cross and Huang (2016). To alleviate the overfitting problem for training on the relative small RST Treebank, we use a dropout of 0.5.

One particular hyperparameter is that we use a value \(\beta\) to balance the chances between training following the exploration (i.e., the best action chosen by the neural model) and following the correct path provided by the dynamic oracle. We find that \(\beta=0.8\), i.e., following the dynamic oracle with a probability of 0.8, achieves the best performance. One explanation for this high chance to follow the oracle is that, since our combined trees are signif

\begin{table}
\begin{tabular}{c|c|c|c}  & Prec. & Recall & F1 \\ \hline \hline Constituency & 87.6 & 86.9 & 87.2 \\ Discourse & 46.5 & 40.2 & 43.0 \\ \hline Overall & 83.5 & 81.6 & 82.5 \\ \end{tabular}
\end{table}
Table 1: Accuracies on PTB-RST at constituency and discourse levels.

 

icantly larger than the constituency trees in Penn Treebank, lower \(\beta\) makes the parsing easily divert into wrong trails that are difficult to learn from.

Since our parser essentially performs both constituency parsing task and discourse parsing task. We also evaluate the performances on sentence constituency level and discourse level separately. The result is shown in Table 1. Note that in constituency level, the accuracy is not directly comparable with the accuracy reported in Cross and Huang (2016), since: a) our parser is trained on a much smaller dataset (RST Treebank is about 1/6 of Penn Treebank); b) the parser is trained to optimize the discourse-level accuracy.

Table 2 shows that, in the perspective of end-to-end discourse parsing, our parser first outperforms the state-of-the-art segmentator of Bach et al. (2012), and furthermore, in end-to-end parsing, the superiority of our parser is more pronounced comparing to the previously best parser of Hernault et al. (2010).

On the other hand, the majority of the conventional discourse parsers are not end-to-end: they rely on gold EDU segmentations and pre-trained tools like Stanford parsers to generate features. We perform an experiment to compare the performance of our parser with them given the gold EDU segments (Table 3). Note that most of these parsers do not handle multi-branching discourse nodes and are trained and evaluated on binarized discourse trees Feng and Hirst (2014); Li et al. (2014a,b); Ji and Eisenstein (2014); Heilman and Sagae (2015), so their performances are actually not directly comparable to the results we reported.

## 5 Conclusion

We have presented a neural-based incremental parser that can jointly parse at both constituency and discourse levels. To our best knowledge, this is the first end-to-end parser for discourse parsing task. Our parser achieves the state-of-the-art performance in end-to-end parsing, and unlike previous approaches, needs little pre-processing effort.

#### Acknowledgments

We thank the anonymous reviewers for helpful comments. We are also grateful to Mingbo Ma, James Cross, and Dezhong Deng for suggestions. This work is supported in part by NSF IIS-1656051, DARPA N66001-17-2-4030 (XAI), a Google Faculty Research Award, and HP.

\begin{table}
\begin{tabular}{c|c|c|c c c c}  & description & syntactic feats. & segmentation & structure & +nuclearity & +relation \\ \hline \hline Bach et al. (2012) & segmentation only & Stanford & 95.1 & - & - & - \\ \hline Hernault et al. (2010) & end-to-end pipeline & Penn Treebank & 94.0 & 72.3 & 59.1 & 47.3 \\ \hline joint syntactic \& discourse parsing & - & **95.4** & **78.8** & **65.0** & **52.2** \\ \end{tabular}
\end{table}
Table 2: F1 scores of end-to-end systems. “+nuclearity” indicates scoring of tree structures with nuclearity included. “+relation” has both nuclearity and relation included (e.g., \(\leftarrow\)Elaboration).

\begin{table}
\begin{tabular}{l|c|c|c c c}  & & syntactic feats & structure & +nuclearity & +relation \\ \hline human annotation Ji and Eisenstein (2014) & - & 88.7 & 77.7 & 65.8 \\ \hline \hline  & Hernault et al. (2010) & Penn Treebank & 83.0 & 68.4 & 54.8 \\  & Joty et al. (2013) & Charniak (retrained) & 82.7 & 68.4 & 55.7 \\ sparse & Joty and Moschitti (2014) & Charniak (retrained) & - & - & 57.3 \\  & Feng and Hirst (2014) & Stanford & 85.7 & 71.0 & 58.2 \\  & Heilman and Sagae (2015) & ZPar (retrieadied) & 83.5 & 68.1 & 55.1 \\  & Wang et al. (2017) & Stanford & **86.0** & **72.4** & 59.7 \\ \hline  & Li et al. (2014a) & Stanford & 82.4 & 69.2 & 56.8 \\ neural & + sparse features & & 84.0 & 70.8 & 58.6 \\  & Ji and Eisenstein (2014) & MALT & 80.5 & 68.6 & 58.3 \\  & + sparse features & & 81.6 & 71.1 & **61.8** \\ \hline \hline  & span-based discourse parsing & - & 84.2 & 67.7 & 56.0 \\ \end{tabular}
\end{table}
Table 3: Experiments using gold segmentations. The column of “syntactic feats” shows how the syntactic features are calculated in the corresponding systems. Note that our parser predicts solely based on the span features from bi-directional LSTM, instead of any explicitly designed syntactic features.

 



## References

* Bach et al. (2012) Ngo Xuan Bach, Nguyen Le Minh, and Akira Shimazu. 2012. A reranking model for discourse segmentation using subtree features. In _Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 160-168. Association for Computational Linguistics.
* Bhatia et al. (2015) Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from rst discourse parsing. _arXiv preprint arXiv:1509.01599_.
* Cross and Huang (2016) James Cross and Liang Huang. 2016. Incremental parsing with minimal features using bi-directional lstm. _arXiv preprint arXiv:1606.06406_.
* Cunha et al. (2012) Iria da Cunha, Eric SanJuan, Juan-Manuel Torres-Moreno, M Teresa Cabre, and Gerardo Sierra. 2012. A symbolic approach for automatic detection of nuclearity and rhetorical relations among intra-sentence discourse segments in spanish. In _International Conference on Intelligent Text Processing and Computational Linguistics_, pages 462-474. Springer.
* Feng and Hirst (2014) Vanessa Wei Feng and Graeme Hirst. 2014. A lineartime bottom-up discourse parser with constraints and post-editing. In _ACL (1)_, pages 511-521.
* Ferrucci et al. (2010) David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building watson: An overview of the deepqa project. _AI magazine_, 31(3):59-79.
* Goldberg and Nivre (2013) Yoav Goldberg and Joakim Nivre. 2013. Training deterministic parsers with non-deterministic oracles. _Transactions of the association for Computational Linguistics_, 1:403-414.
* Heilman and Sagae (2015) Michael Heilman and Kenji Sagae. 2015. Fast rhetorical structure theory discourse parsing. _arXiv preprint arXiv:1505.02425_.
* Hernault et al. (2010) Hugo Hernault, Helmut Prendinger, David A DuVerle, Mitsuru Ishizuka, and Tim Paek. 2010. Hilda: a discourse parser using support vector machine classification. _Dialogue and Discourse_, 1(3):1-33.
* Jansen et al. (2014) Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for nonfactoid answer reranking. In _ACL (1)_, pages 977-986.
* Ji and Eisenstein (2014) Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In _ACL (1)_, pages 13-24.
* Joty and Moschitti (2014) Shafiq Joty and Alessandro Moschitti. 2014. Discriminative reranking of discourse parses using tree kernels. _a) A_, 4(5):6.
* Joty et al. (2013) Shafiq R Joty, Giuseppe Carenini, Raymond T Ng, and Yashar Mehhad. 2013. Combining intra-and multi-sentential rhetorical parsing for document-level discourse analysis. In _ACL (1)_, pages 486-496.
* Li and Jurafsky (2016) Jiawei Li and Dan Jurafsky. 2016. Neural net models for open-domain discourse coherence. _arXiv preprint arXiv:1606.01545_.
* Li et al. (2014a) Jiwei Li, Rumeng Li, and Eduard Hovy. 2014a. Recursive deep models for discourse parsing. In _EMNLP_, pages 2061-2069.
* Li et al. (2014b) Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014b. Text-level discourse dependency parsing. In _ACL (1)_, pages 25-35.
* Louis et al. (2010) Annie Louis, Arxind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In _Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 147-156. Association for Computational Linguistics.
* Mann and Thompson (1988) William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. _Text-Interdisciplinary Journal for the Study of Discourse_, 8(3):243-281.
* Marcu (2000a) Daniel Marcu. 2000a. The rhetorical parsing of unrestricted texts: A surface-based approach. _Computational linguistics_, 26(3):395-448.
* Marcu (2000b) Daniel Marcu. 2000b. _The theory and practice of discourse parsing and summarization_. MIT press.
* Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. _Computational linguistics_, 19(2):313-330.
* Pardo and Nunes (2008) Thiago Alexandre Salgueiro Pardo and Maria das Gracas Volpe Nunes. 2008. On the development and evaluation of a brazilian portuguese discourse parser. _Revista de Informatica Teorica e Aplicada_, 15(2):43-64.
* Somasundaran et al. (2009) Swapna Somasundaran, Galileo Namata, Janyce Wiebe, and Lise Getoor. 2009. Supervised and unsupervised methods in employing discourse relations for improving opinion polarity classification. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1_, pages 170-179. Association for Computational Linguistics.
* Soricut and Marcu (2003) Radu Soricut and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_, pages 149-156. Association for Computational Linguistics.
* Tetreault et al. (2013) ETS Tetreault et al. 2013. Holistic discourse coherence annotation for noisy essay writing.
* Voll and Taboada (2007) Kimberly Voll and Maite Taboada. 2007. Not all words are created equal: Extracting semantic orientation as a function of adjective relevance. In _Australasian Joint Conference on Artificial Intelligence_, pages 337-346. Springer

Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In _Proceedings of ACL_.
* Yoshida et al. (2014) Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In _EMNLP_, pages 1834-1839. Citeseer.