SlowGuess/NAACL_2021_OCR huggingface dataset:
Parsing with EDUs. When segmentation information is provided, we can have a better encoding of the EDUs to construct the tree. Specifically, rather than simply taking the token-boundary representation corresponding to the EDU boundary as the EDU representation, we adopt a hierarchical approach, where we add another Bi-LSTM layer (called "Boundary LSTM") that connects EDU boundaries (a figure of this framework is in the Appendix). In other words, the input sequence to this LSTM layer is  $(\overline{h}_0,\dots ,\overline{h}_m)$ , where  $\overline{h}_0 = h_0$ ,  $\overline{h}_m = h_n$  and  $\overline{h}_j\in \{h_1,\ldots ,h_{n - 1}\}$  such that  $\overline{h}_j$  is an EDU boundary. For instance, for the example in Fig. 1, the input to the Boundary LSTM layer is  $(h_0,h_4,h_{17},h_{25},h_{33},h_{37},h_{44})$ .
This hierarchical representation facilitates better modeling of relations between EDUs and higher order spans, and can capture long-range dependencies better, especially for long documents.
Incorporating Beam Search. Previous work (Lin et al., 2019; Zhang et al., 2020) which also uses a seq2seq architecture, computes the pointing scores over the token or span representations only within the input span. For example, for an input span  $(i,j)$ , the pointing scores are computed considering only  $(h_i,\dots,h_j)$  as opposed to  $(h_1,\dots,h_n)$  in our Eq. 7. This makes the scales of the scores uneven across different input spans as the lengths of the spans vary. Thus, such scores cannot be objectively compared across sub-trees globally at the full-tree level. In addition, since efficient global search methods like beam search cannot be applied properly with non-uniform scores, these previous methods had to remain greedy at each decoding step. In contrast, our decoder points to all the encoded token-boundary representations in every step (Eq. 7). This ensures that the pointing scores are evenly scaled, allowing fair comparisons between the scores of all candidate sub-trees. Therefore, our method enables the effective use of beam search through highly probable candidate trees. Algorithm 2 illustrates the beam search inference when EDUs are given.
We conduct our experiments on discourse parsing with and without gold segmentation. We use the standard English RST Discourse Treebank or RST-DT (Lynn et al., 2002) for training and evaluation.
It consists of 385 annotated Wall Street Journal news articles: 347 for training and 38 for testing. We randomly select  $10\%$  of the training set as our development set for hyper-parameter tuning. Following prior work, we adopted the same 18 courser relations defined in (Carlson and Marcu, 2001). For evaluation, we report the standard metrics Span, Nuclearity, Relation and Full F1 scores, computed using the standard Parseval (Morey et al., 2017, 2018) and RST-Parseval (Marcu, 2000) metrics.
3.1 Parsing with Gold Segmentation
Settings. Discourse parsing with gold EDUs has been the standard practice in many previous studies. We compare our model with ten different baselines as shown in Table 1. We report most results from Morey et al. (2018); Zhang et al. (2020); Kobayashi et al. (2020), while we reproduce Yu et al. (2018) using their provided source code.
For our model setup, we use the encoder-decoder framework with a 3-layer Bi-LSTM encoder and 3-layer unidirectional LSTM decoder. The LSTM hidden size is 400, the word embedding size is 100 for random initialization, while the character embedding size is 50. The hidden dimension in MLP modules and biaffine function for structure prediction is 500. The beam width  $B$  is 20. Our model is trained by Adam optimizer (Kingma and Ba, 2015) with a batch size of 10000 tokens. Our learning rate is initialized at 0.002 and scheduled to decay at an exponential rate of 0.75 for every 5000 steps. Model selection for testing is performed based on the Full F1 score on the development set. When using pretrained word embeddings, we use the 100D vectors from GloVe (Pennington et al., 2014). For pretrained model, we use the XLNet-base-cased version (Yang et al., 2019). The pretrained models/embeddings are kept frozen during training.
Results. From the results in Table 1, we see that our model with GloVe (static) embeddings achieves a Full F1 score of 46.8, the highest among all the parsers that do not use pretrained models (or contextual embeddings). This suggests that a BiLSTM-based parser can be competitive with effective modeling. The model also outperforms the one proposed by Zhang et al. (2020), which is closest to ours in terms of modelling, by  $3.9\%$ ,  $4.1\%$ ,  $2.4\%$  and  $2.5\%$  absolute in Span, Nuclearity, Relation
Systems	Span	Nuc	Rel	Full
Parseval Metric (Morey et al., 2017)
Human Agreement	78.7	66.8	57.1	55.0
Ji and Eisenstein (2014)+	64.1	54.2	46.8	46.3
Feng and Hirst (2014)+	68.6	55.9	45.8	44.6
Joty et al. (2015)+	65.1	55.5	45.1	44.3
Li et al. (2016)+	64.5	54.0	38.1	36.6
Braud et al. (2016)	59.5	47.2	34.7	34.3
Braud et al. (2017)*	62.7	54.5	45.5	45.1
Yu et al. (2018)+§	71.4	60.3	49.2	48.1
Zhang et al. (2020)+	67.2	55.5	45.3	44.3
Our with GloVe	71.1	59.6	47.7	46.8
Our with XLNet§	74.3	64.3	51.6	50.2
RST-Parseval Metric (Marcu, 2000)
Human Agreement	88.7	77.3	65.4
Yu et al. (2018)+§	85.5	73.1	60.2
Wang et al. (2017)+§	86.0	72.4	59.7
Kobayashi et al. (2020)+§	87.0	74.6	60.0
Our with XLNet§	87.6	76.0	61.8
Table 1: Parsing results with gold segmentation. The sign  ${}^{ + }$  denotes that systems use handcrafted features such as lexical, syntactic, sentence/paragraph boundary features and so on, * denotes that systems use external cross-lingual features and  ${}^{8}$  means that systems use pretrained models.
and Full, respectively. More importantly, our system achieves such results without relying on external data or features, in contrast to previous approaches. In addition, by using XLNet-base pretrained model, our system surpasses all existing methods (with or without pretraining) in all four metrics, achieving the state of the art with  $2.9\%$ ,  $4.0\%$ ,  $2.4\%$  and  $2.1\%$  absolute improvements. It also reduces the gap between system performance and human agreement. When evaluated with the RST-Parseval (Marcu, 2000) metric, our model outperforms the baselines by  $0.6\%$ ,  $1.4\%$  and  $1.8\%$  in Span, Nuclearity and Relation, respectively.
3.2 End-to-end Parsing
For end-to-end parsing, we compare our method with the model proposed by Zhang et al. (2020). Their parsing model uses the EDU segmentation from Li et al. (2018). Our method, in contrast, predicts the EDUs along with the discourse tree in a unified process ( $\S 2.3$ ). In terms of model setup, we use a setup identical to the experiments with gold segmentation ( $\S 3.1$ ).
Table 2 reports the performance for document-level end-to-end parsing. Compared to Zhang et al. (2020), our model with GloVe embeddings yields  $1.5\%$ ,  $2.9\%$ ,  $2.4\%$  and  $2.5\%$  absolute gains in Span, Nuclearity, Relation and Full F1 scores, respectively. Furthermore, the model with XLNet
<sup>1</sup>Our initial attempt with BERT did not offer significant gain as BERT is not explicitly designed to process long documents and has a limit of maximum 512 tokens.
Model	Span	Nuc	Rel	Full
Zhang et al. (2020)	62.3	50.1	40.7	39.6
Our model
with GloVe	63.8	53.0	43.1	42.1
with XLNet	68.4	59.1	47.8	46.6
Table 2: End-to-end parsing performance.
Model	Span	Nuc	Rel	Full
Final model	71.1	59.6	47.7	46.8
Beam-seareh	70.1	58.1	46.8	45.8
Boundary-LSTM	68.5	55.5	46.1	44.7
achieves even better performance and outperforms many models that use gold segmentation (Table 1).
EDU Segmentation Results. Our end-to-end parsing method gets an F1 score of 96.30 for the resulting EDUs. Our result rivals existing SoTA segmentation methods – 92.20 F1 of Li et al. (2018) and 95.55 F1 of Lin et al. (2019). This shows the efficacy of our unified framework for not only discourse parsing but also segmentation. $^{2}$
To further understand the contributions from the different components of our unified parsing framework, we perform an ablation study by removing selected components from a network trained with the best set of parameters.
With Gold Segmentation. Table 3 shows two ablations for parsing with gold EDUs. We see that both beam search and boundary LSTM (hierarchical encoding as shown in Fig. 7) are important to the model. The former can find better tree structure by searching a larger searching space. The latter, meanwhile, connects the EDU-boundary representations, which enhances the model's ability to capture long-range dependencies between EDUs.
Table 3: Ablation test of our models with gold EDUs. Beam search indicates the full model with greedy decoding (beam width 1), while Boundary-LSTM is the full model with greedy decoding and no LSTM connection between EDU-boundary representations.
Model	Span	Nuc	Rel	Full
Final model (GloVe)	63.8	53.0	43.1	42.1
GloVe	63.3	52.3	42.4	41.4
Sentence-guidance	59.2	48.8	40.7	38.9
Table 4: Ablation test of our end-to-end model. GloVe is the full model with randomized word embeddings, while Sentenee guidancee is the full model with randomized word embeddings and without sentence guidance.
Figure 5: Confusion matrix for the 10 most frequent relations on the RST-DT test set. The vertical axis represents true and horizontal axis represents predicted relations. The relations are: Elaboration (EL), Attribution (AT), Joint (JO), Same-Unit (SA), Contrast (CONT), Background (BA), Explanation (EX), Cause (CA), Temporal (TEM), Condition (COND).
End-to-end Parsing. For end-to-end parsing, Table 4 shows that the sentence boundary constraint (\$2.3) is indeed quite important to guide the model as it decodes long texts. Since sentence segmentation models are quite accurate, they can be employed if ground truth sentence segmentation is not available. We also notice that pretraining (GloVe) leads to improved performance.
Error Analysis. We show our best parser's (with gold EDUs) confusion matrix for the 10 most frequent relation labels in Fig. 5. The complete matrix with the 18 relations is shown in Appendix (Fig. 8). The imbalanced relation distribution in RST-DT affects our model's performance to some extent. Also semantic similar relations tend to be confused with each other. Fig. 6 shows an example where our model mistakenly labels Summary as Elaboration. However, one could argue that the relation Elaboration is also valid here because the parenthesized text brings additional information (the equivalent amount of money). We show more error examples in the Appendix (Fig. 9 - 11), where our parser la
2We could not compare our segmentation results with the DISRPT 2019 Shared Task (Zeldes et al., 2019) participants. We found few inconsistencies in the settings. First, in their "gold sentence" dataset, instead of using the gold sentence, they pre-process the text with an automatic tokenizer and sentence segmenter. Second, in the evaluation, under the same settings, they do not exclude the trivial BeginSegment label at the beginning of each sentence which we exclude in evaluating our segmentation result (following the RST standard).
Figure 6: An error example where our system incorrectly labels a Summary as Elaboration.
System	Gold Seg.	End-to-End	Time (s)	Speedup
(Feng and Hirst, 2014)	✓	210	1.0x
(Yu et al., 2018)	✓	79	2.7x
Our parser (Glove)	✓	19	11.1x
Our parser (XLNet)	✓	33	6.4x
Our parser (GloVe)	✓	45	4.7x
Our parser (XLNet)	✓	60	3.5x
Table 5: The wall time for parsing the RST-DT test set.
bels a Condition as Background, Temporal as Joint and Explanation as Elaboration. As we can see, all these relations are semantically close and arguably interchangeable.
3.4 Parsing Speed
Table 5 compares the parsing speed of our models with a representative non-neural (Feng and Hirst, 2014) and neural model (Yu et al., 2018). We measure speed empirically using the wall time for parsing the test set. We ran the baselines and our models under the same settings (CPU: Intel Xeon W-2133 and GPU: Nvidia GTX 1080 Ti).
With gold-segmentation, our model with GloVe embeddings can parse the test set in 19 seconds, which is up to 11 times faster than (Feng and Hirst, 2014), and this is when their features are precomputed. The speed gain can be attributed to  $(i)$  to the efficient GPU implementation of neural modules to process the decoding steps, and  $(ii)$  the fact that our model does not need to compute any handcrafted features. With pretrained models, our parser with gold segmentation is about 2.4 times faster than (Yu et al., 2018). Our end-to-end parser that also performs segmentation is faster than the baselines that are provided with the EDUs. Nonetheless, we believe there is still room for speed improvement by choosing a better network, like the Longformer (Beltagy et al., 2020) which has an  $\mathcal{O}(1)$  parallel time complexity in encoding a text, compared to the  $\mathcal{O}(n)$  complexity of the recurrent encoder.
Discourse analysis has been a long-established problem in NLP. Prior to the neural tsunami in NLP, discourse parsing methods commonly em
ployed statistical models with handcrafted features (Soricut and Marcu, 2003; Hernault et al., 2010; Feng and Hirst, 2014; Joty et al., 2015). Even within the neural paradigm, most previous studies still rely on external features to achieve their best performances (Ji and Eisenstein, 2014; Wang et al., 2017; Braud et al., 2016, 2017; Yu et al., 2018). These parsers adopt a bottom-up approach, either transition-based or chart-based parsing.
Recently, top-down parsing has attracted more attention due to its ability to maintain an overall view of the input text. Inspired by the Stack-Pointer network (Ma et al., 2018) for dependency parsing, Lin et al. (2019) first propose a seq2seq model for sentence-level parsing. Zhang et al. (2020) extend this to the document level. Kobayashi et al. (2020) adopt a greedy splitting mechanism for discourse parsing inspired by Stern et al. (2017)'s work in constituency parsing. By using pretrained models/embeddings and extra features (e.g., syntactic, text organizational features), these models achieve competitive results. However, their decoder infers a tree greedily.
Our approach differs from previous work in that it can perform end-to-end discourse parsing in a single neural framework without needing segmentation as a prerequisite. Our model can parse a document from scratch without relying on any external features. Moreover, it can apply efficient beam search decoding to search for the best tree.
We have presented a novel top-down end-to-end method for discourse parsing based on a seq2seq model. Our model casts discourse parsing as a series of splitting decisions at token boundaries, which can solve discourse parsing and segmentation in a single model. In both end-to-end parsing and parsing with gold segmentation, our parser achieves state-of-the-art, surpassing existing methods by a good margin, without relying on handcrafted features. Our parser is not only more effective but also more efficient than the existing ones.
This work leads us to several future directions. Our short-term goal is to improve the model with better architecture and training mechanisms. For example, joint training on discourse and syntactic parsing tasks could be a good future direction since both tasks are related and can be modeled within our unified conditional splitting framework. We also plan to extend our parser to other languages.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from RST discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2212-2218, Lisbon, Portugal. Association for Computational Linguistics.
Chloe Braud, Maximin Coavoux, and Anders Søgaard. 2017. Cross-lingual RST discourse parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292-304, Valencia, Spain. Association for Computational Linguistics.
Chloe Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of RST discourse parsers. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1903-1913, Osaka, Japan. The COLING 2016 Organizing Committee.
Lynn Carlson and Daniel Marcu. 2001. Discourse tagging reference manual. Technical Report ISI-TR-545, University of Southern California Information Sciences Institute.
James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1-11, Austin, Texas. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511-521, Baltimore, Maryland. Association for Computational Linguistics.
Seeger Fisher and Brian Roark. 2007. The utility of parse-derived features for automatic discourse segmentation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 488-495, Prague, Czech Republic. Association for Computational Linguistics.
Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty Steven C.H. Hoi, Caiming Xiong, Irwin King, and
Michael Lyu. 2020. Discern: Discourse-aware entailment reasoning network for conversational machine reading. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2439-2449, Online. Association for Computational Linguistics.
Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T. Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1602-1613, Doha, Qatar. Association for Computational Linguistics.
Hugo Hernault, Helmut Prendinger, David A. duVerle, and Mitsuru Ishizuka. 2010. Hilda: A discourse parser using support vector machine classification. Dialogue and Discourse, 1(3):1-33.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13-24, Baltimore, Maryland. Association for Computational Linguistics.
Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 996-1005, Vancouver, Canada. Association for Computational Linguistics.
Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2012. A novel discriminative framework for sentence-level discourse analysis. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL'12, pages 904-915, Jeju Island, Korea. ACL.
Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385-435.
Shafiq Joty, Giuseppe Carenini, Raymond T. Ng, and Yashar Mehdad. 2013. Combining Intra- and Multisentential Rhetorical Parsing for Document-level Discourse Analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL'13, pages 486-496, Sofia, Bulgaria. ACL.
Shafiq Joty, Francisco Guzmán, Lluis Márquez, and Preslav Nakov. 2017. Discourse structure in machine translation evaluation. Computational Linguistics, 43(4):683-722.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamiigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down rst parsing utilizing granularity levels in documents. In Proceedings of the 2020 Conference on Artificial Intelligence for the American (AAAI), pages 8099-8106.
Jing Li, Aixin Sun, and Shafiq Joty. 2018. Segbot: A generic neural text segmentation model with pointer network. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4166-4172. International Joint Conferences on Artificial Intelligence Organization.
Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362-371, Austin, Texas. Association for Computational Linguistics.
Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and M Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4190-4200, Florence, Italy. Association for Computational Linguistics.
Carlson Lynn, Daniel Marcu, and Mary Ellen Okurowski. 2002. Rst discourse treebank (rst-dt) ldc2002t07. Linguistic Data Consortium.
Xuezhe Ma, Zeong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stackpointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403-1414, Melbourne, Australia. Association for Computational Linguistics.
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243-281.
Daniel Marcu. 2000. The rhetorical parsing of unrestricted texts: a surface-based approach. Computational Linguistics, 26(3):395-448.
Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017