SlowGuess/NAACL_2021_OCR huggingface dataset:
Naoki Kobayashi<sup>1</sup>, Tsutomu Hirao<sup>2</sup>, Hidetaka Kamigaito<sup>1</sup>, Manabu Okumura<sup>1</sup>, Masaaki Nagata<sup>2</sup>
$^{1}$ Institute of Innovative Research, Tokyo Institute of Technology,
$^{2}$ NTT Communication Science Laboratories, NTT Corporation
{kobayasi@lr., kamigaito@lr., oku@}pi.titech.ac.jp
{tsutomu.hirao.kp, masaaki.nagata.et}@hco.ntt.co.jp
Most of the previous Rhetorical Structure Theory (RST) parsing methods are based on supervised learning such as neural networks, that require an annotated corpus of sufficient size and quality. However, the RST Discourse Treebank (RST-DT), the benchmark corpus for RST parsing in English, is small due to the costly annotation of RST trees. The lack of large annotated training data causes poor performance especially in relation labeling. Therefore, we propose a method for improving neural RST parsing models by exploiting silver data, i.e., automatically annotated data. We create large-scale silver data from an unlabeled corpus by using a state-of-the-art RST parser. To obtain high-quality silver data, we extract agreement subtrees from RST trees for documents built using the RST parsers. We then pre-train a neural RST parser with the obtained silver data and fine-tune it on the RST-DT. Experimental results show that our method achieved the best micro-F1 scores for Nuclearity and Relation at 75.0 and 63.2, respectively. Furthermore, we obtained a remarkable gain in the Relation score, 3.0 points, against the previous state-of-the-art parser.
Rhetorical Structure Theory (RST) (Mann and Thompson, 1987) is one of the most widely used theories for representing the discourse structure of a text as a tree. RST trees are a kind of constituent tree, whose leaves are Elementary Discourse Units (EDUs), i.e., clause-like units, and whose non-terminal nodes cover text spans consisting of either a sequence of EDUs or a single EDU. The label of a non-terminal node represents the attribution of a text span, i.e., nucleus (N) or satellite (S). A discourse relation is also assigned between two adjacent non-terminal nodes.
In most cases, RST parsers have been developed on the basis of supervised learning algorithms
(Wang et al., 2017b; Yu et al., 2018; Kobayashi et al., 2020; Lin et al., 2019; Zhang et al., 2020), which require a high-quality annotated corpus of sufficient size. Generally, they train the following three components of the RST parsing: (1) structure prediction by splitting a text span consisting of contiguous EDUs into two smaller ones or merging two adjacent spans into a larger one, (2) nuclearity status prediction for two adjacent spans by solving a 3-class classification problem, and (3) relation label prediction for two adjacent spans by solving an 18-class classification problem (see Section 3.3 for details). However, it is costly to annotate RST trees for a huge collection of documents, and thus it is difficult to obtain a large amount of human-annotated data for RST parsing. As a result, research on RST parsing has focused on English, with the largest annotated corpus being the RST Discourse Treebank (RST-DT) (Carlson et al., 2001), although even this is still small with only 385 documents.
Many RST parsing methods have recently been developed based on neural models (Ji and Eisenstein, 2014; Li et al., 2014a, 2016; Liu and Lapata, 2017; Braud et al., 2016, 2017). Among them, Kobayashi et al. (2020) is the current state-of-the-art system and is based on the neural top-down method. While its Span and Nuclearity scores achieved the highest level, its Relation score still has room for improvement. One of the reasons for its poor Relation score might be its small amount of training data for solving the 18-class classification problem.
Currently, we can refer to various studies on improving neural models for NLP tasks through acquiring large-scale synthetic training data, sometimes called silver data. Among them, one of the studies on Neural Machine Translation (NMT)
<sup>1</sup>We can find some exceptions for other languages such as Spanish (da Cunha et al., 2011) and German (Stede and Neumann, 2014).
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1600-1612 June 6-11, 2021. ©2021 Association for Computational Linguistics
(Sennrich et al., 2016) introduced a simple learning framework: first pre-train an NMT model with silver data, i.e., pseudo-parallel data generated by automatic back-translation, and then fine-tune it with gold data, i.e., real parallel data, to overcome the data sparseness problem. Since the frameworks successfully improved the NMT systems, it has become a standard approach.
Inspired by the above research, we propose a method for improving a student neural parser by exploiting large-scale silver data, thus generating RST trees using an automatic RST parser. Specifically, we improve the state-of-the-art neural RST parser (Kobayashi et al., 2020), in terms of Relation, by employing another RST parser whose Relation score is also state-of-the-art (Wang et al., 2017b) as a teacher parser to generate the silver data. To yield high-quality silver data, we extract a collection of agreement subtrees (ASTs), which are common subtrees among multiple RST trees automatically parsed by the teacher parser with different seeds. Our method includes an efficient algorithm for extracting the agreement subtrees to handle large-scale data. We first pre-train the student parser by using the obtained silver data. We then fine-tune parameters of the parser on gold data, using the RST-DT. Experimental results on the RST-DT clearly indicate the effectiveness of our silver data. Our method obtained remarkable Nuclearity and Relation  $\mathbf{F}_1$  scores of 75.0 and 63.2, respectively.
Early studies on RST parsing were based on traditional supervised learning methods with handcrafted features and the shift-reduce or CKY-like parsing algorithms (duVerle and Prendinger, 2009; Feng and Hirst, 2012; Joty et al., 2013, 2015; Feng and Hirst, 2014). Recently, Wang et al. (2017b) proposed a shift-reduce parser based on SVMs and achieved the current best results in classical statistical models on the RST-DT. The method first built nuclearity-labeled RST trees and then assigned relation labels between two adjacent spans consisting of a single or multiple EDUs.
Inspired by the success of neural networks in
many NLP tasks, several neural network-based models have been proposed for RST parsing (Ji and Eisenstein, 2014; Li et al., 2014a, 2016; Liu and Lapata, 2017). Yu et al. (2018) proposed a shift-reduce parser based on neural networks and leveraged the information from their neural dependency parsing model within a sentence for RST parsing. The best Relation score on the RST-DT, i.e.,  $\mathrm{F}_1$  of 60.2, was achieved with their method.
Recently, a top-down neural parser was proposed (Lin et al., 2019) for use only at the sentence-level. The method parses a tree in a depth-first manner with a pointer-generator network. Zhang et al. (2020) extended the method and applied it to document-level RST parsing. Kobayashi et al. (2020) proposed another top-down RST parsing method exploiting multiple granularity levels in a document and achieved the best Span and Nuclearity scores on the RST-DT, i.e.,  $\mathrm{F}_1$  of 87.0 and 74.6, respectively.
Since the RST-DT, the largest treebank, contains only 385 documents, several studies have been conducted on overcoming the problem of a limited number of training data. Braud et al. (2016) leveraged multi-task learning not only with 13 related tasks as an auxiliary task but also for multiple views of discourse structures, such as Constituent, Nuclearity, and Relation. Braud et al. (2017) used multilingual RST discourse datasets that share the same underlying linguistic theory. Huber and Carenini (2019) adopted distant supervision with an auxiliary task of sentiment classification to create large-scale training data, i.e., they trained a two-stage RST parser (Wang et al., 2017a) with RST trees automatically built based on attention and sentiment scores from the Multiple-Instance Learning network, which was trained with a review dataset. However, these studies need other annotated corpora than the RST-DT, which means we still face the problem of being dependent on costly annotated corpora. Jiang et al. (2016) proposed a framework for enriching training data based on co-training to improve the performance for infrequent relation labels. However, the method failed to improve the overall Relation score, while they did not aim at improving the Span and Nuclearity scores.
Unsupervised RST parsing methods have also been proposed recently (Kobayashi et al., 2019; Nishida and Nakayama, 2020). Since they are unsupervised, they do not require any annotated corpora. However, they can predict only tree structures and
Nguyen et al. (2020) proposed a similar approach in NMT and introduced a method named data diversification: it diversifies the training data by using multiple forward and backward translation models. We can find some weak supervision approaches for other discourse representation formalisms such as (Badene et al., 2019).
Figure 1: Overview of proposed method. In the subtree extraction step, the teacher RST parsers first annotate trees to unlabeled documents, and then the proposed subtree extraction method constructs large silver data. In the training step, the student parser is trained through pre-training and fine-tuning.
cannot predict nucleus and relation labels. Therefore, the predicted trees cannot be used for learning for predicting relation labels.
We should mention the relationship of our work with semi-supervised learning as a machine learning framework. First, the reason why we do not adopt self-training, where the student and teacher parsers are the same, but instead use two different parsers that rely on different parsing algorithms is that we can acquire instances that the student parser cannot correctly parse yet the teacher parser can parse as the training data. Second, using multiple different RST parsers in a semi-supervised manner in our work might seem reminiscent of co- or tri-training. While co- or tri-training is attractive, it is time consuming to repeat the step of alternately training multiple different neural network-based parsers many times. Thus, previous studies have focused on simplifying the repetition step in constituency and dependency parsing (McClosky et al., 2006; Yu et al., 2015; Pekar et al., 2014; Weiss et al., 2015; Li et al., 2014b).
We believe our method is similar to these simplified version as a semi-supervised framework with two different RST parsers.
3 Neural RST Parsing with Silver Data
3.1 Training Student Parser
Traditional semi-supervised learning frameworks, such as self-, co-, and tri-training, tend to iteratively train a student classifier with the training data that contains human-annotated (gold) data and iteratively added silver data. Since neural network-based models require a large amount of time for training, the iterative procedure is not suitable for training them. Furthermore, the training method may be affected by the bias problem in relation-label distribution because frequent labels in the original training data become yet more frequent in the future training data. For these reasons, we
adopt a simple pre-training and fine-tuning strategy, which is inspired by the NMT research (Sennrich et al., 2016), to train a student RST parser.
Since early statistical RST parsing methods relied on handcrafted features, i.e., sentence-level features obtained from parse trees and document-level features, they require complete documents with complete sentences for their feature extraction. On the other hand, recent neural models do not necessarily need such features. Thus, we can exploit subtrees as training data for the neural networks.
Our method involves the following two steps: First, we extract a collection of ASTs from RST trees for each document in unlabeled data as the silver data. In this step, each document is first parsed using multiple teacher RST parsers with different seeds, trained with a gold dataset, the RST-DT. We then apply our algorithm for extracting the ASTs, which are common subtrees among multiple automatically parsed RST trees. In the second step, we pre-train the student RST parser with the collection of ASTs to complement the amount of training data. The parameters of the student parser are then finetuned on the RST-DT. Figure 1 shows an overview of our proposed method.
3.2 Extracting Agreement Subtrees
A good strategy for obtaining high-quality silver data is to get agreement among the results of multiple RST parsers. However, it is difficult to reach agreement for the entire RST trees at the document-level because their size is big. Thus, we believe we cannot collect enough silver data using agreement for the whole trees. On the other hand, we find that many subtrees agreed among multiple RST trees, even when the whole trees do not agree with each other. Accordingly, we extract ASTs as the silver data.
To create large-scale silver data, we need an efficient algorithm that extracts the ASTs, i.e., com
mon subtrees among multiple RST trees for a document. Note that we need to extract multiple maximal common subtrees among the RST trees. This requires a different algorithm from the maximum agreement subtree problem, which is well-known in bioinformatics (Deepak and Fernandez-Baca, 2014). Thus, we develop the algorithm in Algorithm 1. This algorithm follows a tree-traversal algorithm and works with  $O(n)$ , where  $n$  indicates the number of nodes in an RST tree.
In the algorithm, a tree is represented as a fully-labeled nested span structure (see the example in Figure 2). The function AGREEMENT receives an arbitrary span as the input and returns a Boolean value indicating whether the subtree for the span is an AST. AGREEMENT first counts how many times the input span appears in the set of given RST trees
and checks the status of the left and right children of the input span.  $\mathrm{Len}()$  returns the length of the span and Count() returns the frequency of the fully-labeled span among the trees, which indicates how many trees agree on the subtree. The minimum and maximum values of Count() are 1 and  $k$  respectively, where  $k$  indicates the number of RST trees. The variables  $\mathbf{S}_c$ ,  $\mathbf{S}_l$ , and  $\mathbf{S}_r$  store the Boolean value for the input span and the left and right children of the span, respectively. Here, root, leftChild, and rightChild are functions for returning the root span and the left and right children spans, respectively. To obtain the status of each child, AGREEMENT calls itself with the child span. When the frequency of the input span is  $k$ , indicating all of the trees in the set agree on the span, and the status of the left and right children is True, indicating the left and right children are ASTs, the function returns True. Furthermore, the information regarding which subtrees are ASTs is stored in variable S during the execution of AGREEMENT.
The function FINDROOT returns the list of ASTs, based on the information in variable S, given by AGREEMENT. FINDROOT first checks the S-span), the Boolean value of the span. If it is True, the function appends the span, corresponding to the root node of an AST, to the output. Otherwise, it searches both left and right children for ASTs recursively. The function, therefore, lists all of the maximal ASTs in a depth-first fashion, based on the information in variable S.
In the algorithm,  $l_{\mathrm{min}}$  and  $l_{\mathrm{max}}$  are used to control the size of extracted ASTs. If the trees parsed using the multiple teacher parsers significantly differ from each other, the extracted ASTs tend to be small, which might become noise. To avoid such noise, we do not take into account subtrees with less than  $l_{\mathrm{min}}$  EDUs. Excessively large subtrees are difficult to handle because they need a lot of time and space for training. Therefore, if the size of subtrees exceeds  $l_{\mathrm{max}}$ , the algorithm tries to find smaller ASTs from both their left and right children.
Initially, we call the function AGREEMENT with an arbitrary tree in multiple RST trees. We show an example of extracting ASTs in Figure 2. Assume the two trees at the left are from two RST parsers. The right part represents how the algorithm works with the top tree at the left as the input. In the figure, two subtrees consisting of spans (1,4) and (5,7) are extracted as ASTs since the frequency of
Figure 2: Example of extracting ASTs. Assume the left two trees are from two RST parsers. In the figure, red spans are shared by the two trees. The right part represents how the algorithm works with the top tree in the left as the input. Subtrees whose roots are spans (1,4) and (5,7) are extracted as ASTs. Note that we do not extract a subtree in extracted ASTs as an AST.
these two spans and all their descendant spans is 2, which is the number of given RST trees. Note that, while several spans, such as spans (2,3) and (6,7), are also common subtrees, we do not extract them since they are contained in either span (1,4) or (5,7).
3.3 Span-based Neural Top-down Parser as student Parser
As described in Section 3.1, the advantage of recent neural models is that they can utilize the annotation for partial documents, or subtrees, as training data. Among the neural models, the span-based neural top-down RST parsing method (Kobayashi et al., 2020) achieved the best Span and Nuclearity scores. Thus, we employ it as the student parser.
The method builds a tree by recursively splitting a text span into two smaller ones while predicting the nuclearity status and relation labels. As we explain below, the parser can be trained with arbitrary subtrees for spans consisting of EDUs.
Structure Prediction
For each position  $k$  in a span which consists of  $i$ -th EDU to  $j$ -th EDU, a scoring function,  $s_{\mathrm{split}}(i,j,k)$ , is defined as follows:
$$
s _ {\text {s p l i t}} (i, j, k) = \mathbf {h} _ {i: k} ^ {\top} \mathbf {W} _ {u} \mathbf {h} _ {k + 1: j} + \mathbf {v} _ {\ell} ^ {\top} \mathbf {h} _ {i: k} + \mathbf {v} _ {r} ^ {\top} \mathbf {h} _ {k + 1: j}, \tag {1}
$$
where  $\mathbf{W}_u$  is a weight matrix and  $\mathbf{v}_{\ell}$  and  $\mathbf{V}_r$  are weight vectors corresponding to the left and
right spans, respectively.  $\mathbf{h}_{i:k}$  and  $\mathbf{h}_{k+1:j}$  are defined as follows:  $\mathbf{h}_{i:k} = \mathrm{MLP}_{\mathrm{left}}(\mathbf{u}_{i:k})$ ,  $\mathbf{h}_{k+1:j} = \mathrm{MLP}_{\mathrm{right}}(\mathbf{u}_{k+1:j})$ , where  $\mathrm{MLP}_*$  is the multi-layer perceptron. The vector representation of a span,  $\mathbf{u}_{i:j}$ , is obtained by feeding word embedding vectors into LSTMs. Then, the span is split at position  $k$  that maximizes Eq. (1):
$$
\hat {k} = \operatorname {a r g m a x} _ {k \in \{i, \dots , j - 1 \}} \left[ s _ {\text {s p l i t}} (i, j, k) \right]. \tag {2}
$$
Label Prediction
When splitting a span at position  $k$ , the score of the nuclearity status and relation labels for the two spans is defined as follows:
$$
s _ {\text {l a b e l}} (i, j, k, \ell) = \mathbf {W} _ {\ell} \mathrm {M L P} \left(\left[ \mathbf {u} _ {i: k}; \mathbf {u} _ {k + 1: j}; \mathbf {u} _ {1: i}; \mathbf {u} _ {j: n} \right]\right), \tag {3}
$$
where  $\mathbf{W}_{\ell}$  is a weight matrix and  $\mathbf{u}_{1:i}; \mathbf{u}_{j:n}$  are vector representation of left and right spans that appear outside the current focus. Then, the label that maximizes Eq. (3) is assigned to the spans:
$$
\hat {\ell} = \underset {\ell \in \mathcal {L}} {\operatorname {a r g m a x}} \left[ s _ {\text {l a b e l}} (i, j, k, \ell) \right], \tag {4}
$$
where  $\mathcal{L}$  denotes a set of valid nuclearity status combinations,  $\{\mathrm{N - S,S - N,N - N}\}$ , for predicting the nuclearity, and a set of relation labels,  $\{\mathrm{Elaboration},\mathrm{Condition},\ldots \}$ , for predicting the relation. Accordingly, we solve a 3-class classification problem for the nuclearity labeling and an 18-class classification problem for the relation labeling. Note that the weight parameters  $\mathbf{W}_{\ell}$  and MLP for the nuclearity and relation labeling are separately learned.
All parameters,  $\mathbf{W}_u$ ,  $\mathbf{W}_{\ell}$ ,  $\mathbf{v}_r$ ,  $\mathbf{v}_{\ell}$ , and the parameters for LSTMs are optimized by using margin-based learning. When the correct splitting position  $k^*$  and labels  $\ell^*$  are given, loss functions for splitting and labeling are defined as follows:
$$
\begin{array}{l} \max  (0, 1 + s _ {\text {s p l i t}} (i, j, k ^ {*}) - s _ {\text {s p l i t}} (i, j, \hat {k})), \\ \max  \left(0, 1 + s _ {\text {l a b e l}} (i, j, \hat {k}, \ell^ {*}) - s _ {\text {l a b e l}} (i, j, \hat {k}, \hat {\ell})\right). \tag {5} \\ \end{array}
$$
By minimizing the sum of the losses in each splitting point, the parameters are optimized.
3.4 Two-stage Parser as Teacher Parser
Since the student parser still has room for improvement in Relation, it is desirable to utilize another state-of-the-art parser based on a different parsing algorithm with a good Relation score. While the current best Relation score was achieved by NNDisParser (Yu et al., 2018), we cannot reproduce this score with their official code. Therefore, we employ the two-stage parser (Wang et al., 2017b), which obtained the second-best Relation score, as the teacher parser. This two-stage parser is based on a shift-reduce parsing algorithm and utilizes SVMs to determine actions to build trees. Since their SVMs are optimized by a dual coordinate descent method, we build multiple two-stage parser models with different seeds to obtain enough agreement between teacher parsers, $^3$  and create silver data by the agreement among the parsers.
We used the RST-DT to evaluate the performance of our student RST parser and compared it with state-of-the-art parsers. It is officially divided into 347 documents as the training dataset and 38 documents as the test dataset. Since there is no development dataset, we used a part of the training dataset, 40 documents, as the development dataset by following the previous study (Heilman and Sagae, 2015). By following conventional studies, we used gold EDU segmentation for the RST-DT. The training and development datasets were used as gold data to fine-tune our student parser.
To obtain silver data for pre-training, we used the CNN dataset (Hermann et al., 2015). To parse each document, we split sentences into EDUs by using
Tree type	lmin	# of trees	# of nodes
DT	-	91,536	8,162,114
ADT	-	2,142	57,940
AST	5	534,352	4,087,989
6	387,636	3,501,125
7	290,532	3,015,605
8	223,101	2,611,019
9	175,709	2,279,275
10	140,384	1,996,675
Table 1: Number of trees and nodes in each type of silver data obtained from the CNN dataset. Note that we utilized only 91,536 documents that could go through the pre-processing with the CoreNLP toolkit.
the Neural EDU Segmenter (Wang et al., 2018) $^4$  and applied the two-stage parser.
$l_{\mathrm{min}}$  and  $l_{\mathrm{max}}$  for AST extraction: Since the number of EDUs for a document in the RST-DT is from 7 to 240, we selected  $l_{\mathrm{min}}$  with a range from 5 to 10 and set  $l_{\mathrm{max}}$  to 240. Based on the results for the development dataset,  $l_{\mathrm{min}}$  was fixed to 9 (see Appendix A for details).
Student Parser: We used the official code of the span-based neural top-down parsing method. The dimension of the hidden layers was set to 500. We trained the model in 5 and 10 epochs for pretraining and fine-tuning, respectively. Other parameters of the model and an optimizer were the same as those used by Kobayashi et al. (2020) (see Appendix E for details).