SlowGuess/NAACL_2021_OCR huggingface dataset:
Kobayashi et al. (2020) achieved the best results in the D2P2S2E setting, training the models in three levels of granularity, i.e., paragraph trees for documents, sentence trees for paragraphs, and EDU trees for sentences. This setting requires us to train many models corresponding to multiple granularity levels. To simplify this, we trained only the model for building an RST tree whose leaves are EDUs for a document, which corresponds to their D2E setting. In decoding, we split spans at sentence and paragraph boundaries to make the setting closer to D2P2S2E.
We also used ensemble decoding by following
<sup>3</sup>We could not adopt multiple different parser architectures, for example, the Two-stage parser and Span-based parser, because the agreement was low.
4https://github.com/PKU-TANGENT/ NeuralEDUSeg 5https://github.com/nttcslab-nlp/ Top-Down-RST-Parser
Model	Average	Ensemble
S	N	R	F	S	N	R	F
SBP	86.3	73.1	57.6	57.3	87.1	74.6	60.0	59.6
SBP+DT	86.9	74.1	61.8	61.0	87.4	74.7	62.7	61.7
SBP+ADT	86.6	73.5	59.5	58.8	86.9	74.3	60.5	59.7
SBP+AST	86.8	74.7	62.5	61.8	87.1	75.0	63.2	62.6
Table 2: Micro-averaged  $\mathbf{F_1}$  scores of span-based neural top-down parser with or without silver data on the test dataset of the RST-DT. S, N, R, and F represent Span, Nuclearity, Relation, and Full scores, respectively. The best score for each metric in the average and ensemble settings is indicated with bold.
Kobayashi et al. (2020). Since it takes a large amount of time to train multiple models in pretraining, we trained only a single model in the pretraining stage, while multiple models were trained in the fine-tuning stage with the pre-trained model as the initial state.
Teacher parser: We used the official code of the two-stage parsing method and re-trained it four times with different random seeds. A smaller value of  $k$  made reliability of the agreement lower since we could not exclude coincidentally agreed trees. On the other hand, a larger value required us more time to create silver data, while the reliability of the agreement is high. Thus, we set  $k$  to 4, that is a moderate number in terms of both the reliability of the agreement and the data creation time.
By following previous studies (Sagae and Lavie, 2005), we transformed RST-trees into right-heavy binary trees and evaluated system results with micro-averaged  $\mathrm{F_1}$  scores of Span, Nuclearity, Relation, and Full, based on RST-Parseval (Marcu, 2000). Span, Nuclearity, Relation, and Full were used to evaluate unlabeled, nuclearity-labeled, relation-labeled, and fully-labeled tree structures, respectively. Since Morey et al. (2017) made a suggestion to use a standard parseEval toolkit for evaluation, we also report the results using this in Appendix C.
4.4 Compared Methods
To demonstrate the effectiveness of our proposed method, we pre-trained the span-based neural top-down parser, i.e., our student parser, in various settings for creating the silver data and compared
the performance after fine-tuning on the RST-DT. Table 1 summarizes the statistics of the different types of silver data. 'DT' denotes RST trees obtained by using a single two-stage parser. The number of RST trees is the same as that of documents in the CNN dataset. 'ADT' denotes agreement document-level RST trees, i.e., the cases in which the parsers built the same trees for the whole document. 'AST' denotes ASTs of RST trees obtained from the teacher parsers.
5.1 Different Methods for Constructing Silver Data
Table 2 shows the average and ensemble scores with five models for different types of silver data. In the table, SBP indicates the results obtained from the original span-based neural top-down parser, which means the parser was trained only with the RST-DT; this setting is without any silver data.
With AST as the silver data, performance in all metrics improved against the baseline. In most metrics, AST achieved the best scores. In particular, the gains in Relation and Full were impressive. DT and ADT, which consist of document-level RST trees, also outperformed the baseline. However, the gains against the baseline were smaller than those by AST. We believe this is related to the size and quality of the silver data. The number of trees and nodes in ADT is only 2,142 and 57,940, respectively, while AST has 175,709 trees and 2,279,275 nodes. Thus, a small number of silver data for pre-training is not effective. On the other hand, while DT has only 91,536 trees, the number of their nodes is huge, at about  $8,000\mathrm{K}$ . The lower score of DT would come from unreliable parse trees contained in the silver data built by a single teacher parser. As described above, to pre-train
<sup>6</sup>We show the results for a case of using SBP as a teacher parser in Appendix B.
<sup>7</sup>https://github.com/yizhongw/StageDP
the student parser, we do not need to use the entire RST trees for documents. Thus, AST, with a large collection of RST subtrees, is more effective than the other approaches. Since the training time depends on the number of nodes contained in the data, SBP+AST can be learned in a quarter of the time required by SBP+DT. Consequently, AST has another advantage against DT.
Furthermore, the performance of averaging five models was greatly improved by pre-training with the silver data. The gains against the baseline were larger than those for 'Ensemble,' and the differences between their performances became small. The neural model tends to converge to a different local optimum solution by mini-batch training, so the convergence is not stable when the data size is small. Pre-training can improve this. This is another advantage of pre-training with silver data.
We also compare the results of our parser pretrained with AST with and without fine-tuning in Appendix D.
5.2 Effect of Data Size
To investigate how the data size of AST for pretraining affects the performance, we show Span, Nuclearity, Relation, and Full scores while varying the size in Figure 3. Span scores showed only small gains even by increasing the amount of data because identifying splitting points for spans is a simple 2-class classification problem. On the other hand, identifying nuclearity and relation labels is a multi-class classification problem. Thus, we believe we need more training data than that for identifying splitting points. In particular, the Relation score could be improved with more silver data.
5.3 Detailed Analysis of Relation Labeling
To investigate the effectiveness of SBP+AST in more detail, we show Relation  $\mathbf{F_1}$  scores for relation labels with SBP, SBP+AST, and the two-stage parser in Figure 4. The results of SBP and SBP+AST were obtained from a five-model ensemble. In most relation labels, since the two-stage parser, the teacher parser, is comparable or superior to SBP, i.e., the student parser, the performance of SBP+AST can be improved. It finally outperformed the two-stage parser by introducing pretraining with silver data, even for less frequent relation labels. Furthermore, SBP+AST can correctly parse for some relation labels that the student parser
Figure 3: Results of changing the data size used for pretraining. The scores are from a five-model ensemble with SBP+AST. The solid lines represent the scores for SBP+AST by changing the data size. The dotted lines represent SBP, which does not depend on the size.
Model	S	N	R	F
Two-stage Parser	86.0	72.4	59.7	58.8
NNDisParser*	85.5	73.1	60.2	59.9
NNDis Parser	85.9	72.5	59.5	58.9
SpanBased Parser	87.1	74.6	60.0	59.6
SBP+AST	87.1	75.0	63.2	62.6
Table 3: Comparison of state-of-the-art parsers. * indicates reported scores. The best score in each metric is indicated in **bold**. Our model is statistically significantly better than **underlined** scores at p-level  $< 0.01$  in pairwise comparison. $^{8}$
cannot handle, by acquiring training instances with the help of the teacher parser.
5.4 Comparison with state-of-the-art parsers
Finally, we compare our SBP+AST with the ensemble to current state-of-the-art parsers. Table 3 shows the micro-averaged  $\mathbf{F_1}$  scores. We used Paired Bootstrap Resampling (Koehn, 2004) for the significance test. We can see that our method achieved the best scores except for Span. The gains against the previous best scores were 0.4, 3.0, and 2.7 points for Nuclearity, Relation, and Full, respectively. In particular, the gains for Relation and Full are remarkable.
<sup>8</sup>Since the previous best scores for Relation and Full are reported scores, and we could not obtain the authors' data, we could not perform a significance test against them.
Figure 4:  $\mathbf{F_1}$  scores of SBP, SBP+AST, and the two-stage parser (TSP) for each relation label: ELABORATION, JOINT, ATTRIBUTION, SAME-UNIT, CONTRAST, TEMPORAL, BACKGROUND, EXPLANATION, CAUSE, EVALUATION, CONDITION, ENABLEMENT, TOPIC-COMMENT, COMPARISON, SUMMARY, MANNER-MEANS, TOPIC-CHANGE, and TEXTUAL-ORGANIZATION. Relation labels are arranged in descending order of their frequency, shown in parenthesis under their label.
To solve the problem of the limited amount of training data available for neural RST parsing, we proposed a method of exploiting agreement subtrees as silver data: We pre-train a parser with the silver data and fine-tune it with the gold data. We also presented an algorithm that efficiently extracts overlapping subtrees as the agreement subtrees from multiple trees.
Experimental results on the RST-DT demonstrated that our method significantly improves the performance of relation-labeled and fully-labeled  $\mathrm{F_1}$  scores, which are strongly affected by data sparseness due to a small number of training data. Furthermore, the results showed that our method achieves the state-of-the-art nuclearity-labeled, relation-labeled, and fully-labeled  $\mathrm{F_1}$  scores.
Sonia Badene, Kate Thompson, Jean-Pierre Lorré, and Nicholas Asher. 2019. Data programming for learning discourse structure. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 640-645, Florence, Italy. Association for Computational Linguistics.
Chloe Braud, Maximin Coavoux, and Anders Søgaard. 2017. Cross-lingual RST discourse parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292-304,
Valencia, Spain. Association for Computational Linguistics.
Chloe Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of RST discourse parsers. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1903-1913, Osaka, Japan. The COLING 2016 Organizing Committee.
Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue.
Iria da Cunha, Juan-Manuel Torres-Moreno, and Gerardo Sierra. 2011. On the development of the RST Spanish treebank. In Proceedings of the 5th Linguistic Annotation Workshop, pages 1-10, Portland, Oregon, USA. Association for Computational Linguistics.
Akshay Deepak and David Fernandez-Baca. 2014. Enumerating all maximal frequent subtrees in collections of phylogenetic trees. Algorithms Mol. Biol., 9:16.
David duVerle and Helmut Prendinger. 2009. A novel discourse parser based on support vector machine classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 665-673, Suntec, Singapore. Association for Computational Linguistics.
Vanessa Wei Feng and Graeme Hirst. 2012. Text-level discourse parsing with rich linguistic features. In
Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 60-68, Jeju Island, Korea. Association for Computational Linguistics.
Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511-521, Baltimore, Maryland. Association for Computational Linguistics.
Michael Heilman and Kenji Sagae. 2015. Fast rhetorical structure theory discourse parsing.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Patrick Huber and Giuseppe Carenini. 2019. Predicting discourse structure using distant supervision from sentiment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2306-2316, Hong Kong, China. Association for Computational Linguistics.
Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13-24, Baltimore, Maryland. Association for Computational Linguistics.
Kailang Jiang, Giuseppe Carenini, and Raymond Ng. 2016. Training data enrichment for infrequent discourse relations. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2603-2614, Osaka, Japan. The COLING 2016 Organizing Committee.
Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multisentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 486-496, Sofia, Bulgaria. Association for Computational Linguistics.
Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385-435.
Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamiigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down rst parsing utilizing granularity levels in
documents. In Proceedings of the 2020 Conference on Artificial Intelligence for the American, pages 8099-8106, New York, America.
Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019. Split or merge: Which is better for unsupervised RST parsing? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5797-5802, Hong Kong, China. Association for Computational Linguistics.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Jiwei Li, Rumeng Li, and Eduard Hovy. 2014a. Recursive deep models for discourse parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061-2069, Doha, Qatar. Association for Computational Linguistics.
Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362-371, Austin, Texas. Association for Computational Linguistics.
Zhenghua Li, Min Zhang, and Wenliang Chen. 2014b. Ambiguity-aware ensemble training for semi-supervised dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 457-467, Baltimore, Maryland. Association for Computational Linguistics.
Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and M Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4190-4200, Florence, Italy. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2017. Learning contextually informed representations for linear-time discourse parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1289-1298, Copenhagen, Denmark. Association for Computational Linguistics.
W.C. Mann and S.A Thompson. 1987. Rhetorical structure theory: A theory of text organization. Technical Report ISI/RS-87-190, USC/ISI.
Daniel Marcu. 2000. The Theory and Practice of Discourse Parsing and Summarization. MIT Press.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152-159, New York City, USA. Association for Computational Linguistics.
Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319-1324, Copenhagen, Denmark. Association for Computational Linguistics.
Xuan-Phi Nguyen, Shafiq Joty, Kui Wu, and Ai Ti Aw. 2020. Data diversification: A simple strategy for neural machine translation. In Advances in Neural Information Processing Systems, volume 33, pages 10018-10029. Curran Associates, Inc.
Noriki Nishida and Hideki Nakayama. 2020. Unsupervised discourse constituency parsing using viterbi em. Transactions of the Association for Computational Linguistics, pages 215-230.
Viktor Pekar, Juntao Yu, Mohab El-karef, and Bernd Bohnet. 2014. Exploring options for fast domain adaptation of dependency parsers. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 54–65, Dublin, Ireland. Dublin City University.
Kenji Sagae and Alon Lavie. 2005. A classifier-based parser with linear run-time complexity. In Proceedings of the Ninth International Workshop on Parsing Technology, pages 125-132, Vancouver, British Columbia. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Manfred Stede and Arne Neumann. 2014. Potsdam commentary corpus 2.0: Annotation for discourse research. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 925-929, Reykjavik, Iceland. European Language Resources Association (ELRA).
Xinhao Wang, James Bruno, Hillary Molloy, Keelan Evanini, and Klaus Zechner. 2017a. Discourse annotation of non-native spontaneous spoken responses using the Rhetorical Structure Theory framework. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 263-268, Vancouver, Canada. Association for Computational Linguistics.
Yizhong Wang, Sujian Li, and Houfeng Wang. 2017b. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184-188, Vancouver, Canada. Association for Computational Linguistics.
Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. Toward fast and accurate neural discourse segmentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 962-967, Brussels, Belgium. Association for Computational Linguistics.
David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 323-333, Beijing, China. Association for Computational Linguistics.
Juntao Yu, Mohab Elkaref, and Bernd Bohnet. 2015. Domain adaptation for dependency parsing via self-training. In Proceedings of the 14th International Conference on Parsing Technologies, pages 1-10, Bilbao, Spain. Association for Computational Linguistics.
Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based neural RST parsing with implicit syntax features. In Proceedings of the 27th International Conference on Computational Linguistics, pages 559-570, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Longyin Zhang, Yuqing Xing, Fang Kong, Peifeng Li, and Guodong Zhou. 2020. A top-down neural architecture towards text-level parsing of discourse rhetorical structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6386-6395, Online. Association for Computational Linguistics.
Figure 5: Fully-labeled  $\mathbf{F_1}$  scores with different  $l_{\mathrm{min}}$  on the development dataset.
Model	S	N	R	F
SBP+DT (SBP)	87.1	74.5	60.3	59.7
SBP+DT (TSP)	87.4	74.7	62.7	61.7
SBP (Kobayashi et al., 2020)	87.1	74.6	60.0	59.6
A Effects of Parameter  $l_{\mathrm{min}}$
Figure 5 shows fully-labeled  $\mathrm{F_1}$  scores in changing  $l_{\mathrm{min}}$  on the development dataset. From the figure, it is clear that the  $\mathrm{F_1}$  score was changed with  $l_{\mathrm{min}}$ . The best  $\mathrm{F_1}$  score was achieved by  $l_{\mathrm{min}} = 9$ . The results indicate that a large number of smaller subtrees prevents better pre-training of SBP. A small number of larger subtrees is also not useful for the pre-training.
B Performance of a case when the Span-based parser was used both as teacher and student parsers
We used different parsers for teacher and student parsers. In this section, we examine the setting of using the Span-based parser both as teacher and student parsers. We compare DT (SBP) and DT (TSP) as the silver datasets used for pre-training and show the results in Table 4. The results show that SBP+DT (SBP) does not obtain any gain of performance compared to SBP, that does not use any silver dataset. The better results with SBP+DT (TSP) demonstrate the effectiveness of using different types of parsers for teacher and student parsers.
C Performance with Original Parseval
In this paper, we used the gold EDU segmentation following conventional studies and evaluated the model performance for binarized trees with RST-Parseval (Marcu, 2000). Morey et al. (2017) reported that when evaluating the performance of binarized trees over manual EDU segmentation, the level of agreement between RST trees is artificially raised. To avoid this, they recommended using the original Parseval for the trees of label-attachment decisions. Following them, we evaluated our models with the original Parseval<sup>9</sup> and show the results in Table 5.
The results show the same tendency as that when employing RST-Parseval as the evaluation metrics. That is, SBP+AST obtained the best results for Nuclearity, Relation, and Full.
Table 4: Comparison of teacher parsers.
Model	S	N	R	F
(Feng and Hirst, 2014)*	68.6	55.9	45.8	44.6
(Ji and Eisenstein, 2014)*	64.1	54.2	46.8	46.3
(Wang et al., 2017b)**	72.0	60.5	50.4	48.2
(Yu et al., 2018)**	71.8	60.3	49.4	48.4
(Zhang et al., 2020)	67.2	55.5	45.3	44.3
(Kobayashi et al., 2020)**	74.1	63.7	48.8	47.9
SBP+AST	74.1	64.7	54.1	52.7
Table 5: Micro-averaged  $\mathrm{F}_{1}$  scores with the original Parseval (Morey et al., 2017). * indicates the reported scores in (Morey et al., 2017). ** indicates the scores for the re-produced models.
D Performance of Parser with Pre-training Alone