SlowGuess/NAACL_2022_OCR huggingface dataset:
Yifei Zhou
yz639@cornell.edu
Yansong Feng*
WICT, Peking University
MOE Key Lab. of Computational Linguistics
fengyansong@pku.edu.cn
Recent works show that discourse analysis benefits from modeling intra- and inter-sentential levels separately, where proper representations for text units of different granularities are desired to capture both the meaning of text units and their relations to the context. In this paper, we propose to take advantage of transformers to encode contextualized representations of units of different levels to dynamically capture the information required for discourse dependency analysis on intra- and inter-sentential levels. Motivated by the observation of writing patterns commonly shared across articles, we propose a novel method that treats discourse relation identification as a sequence labelling task, which takes advantage of structural information from the context of extracted discourse trees, and substantially outperforms traditional direct-classification methods. Experiments show that our model achieves state-of-the-art results on both English and Chinese datasets. Our code is publicly available<sup>1</sup>.
Discourse dependency parsing (DDP) is the task of identifying the structure and relationship between Elementary Discourse Units (EDUs) in a document. It is a fundamental task of natural language understanding and can benefit many downstream applications, such as dialogue understanding (Perret et al., 2016; Takanobu et al., 2018) and question answering (Ferrucci et al., 2010; Verberne et al., 2007).
Although existing works have achieved much progress using transition-based systems (Jia et al., 2018b,a; Hung et al., 2020) or graph-based models (Li et al., 2014a; Shi and Huang, 2018; Afantenos et al., 2015), this task still remains a challenge. Different from syntactic parsing, the basic components in a discourse are EDUs, sequences of words, which are not trivial to represent in a
straightforward way like word embeddings. Predicting the dependency and relationship between EDUs sometimes necessitates the help of a global understanding of the context so that contextualized EDU representations in the discourse are needed. Furthermore, previous studies have shown the benefit of breaking discourse analysis into intra- and inter-sentential levels (Wang et al., 2017), building sub-trees for each sentence first and then assembling sub-trees to form a complete discourse tree. In this Sentence-First (Sent-First) framework, it is even more crucial to produce appropriate contextualized representations for text units when analyzing in intra- or inter-sentential levels.
Figure 1: An excerpt discourse dependency tree in SciDTB. Each indexed block is an EDU, and the origin of the arrow pointing to a particular EDU is its head.
Figure 1 shows an excerpt discourse dependency structure for a scientific abstract from SciDTB (Yang and Li, 2018). The lengths of EDUs vary a lot, from more than 10 words to 2 words only (EDU 12: tests show), making it especially hard to encode by themselves alone. Sometimes it is sufficient to consider the contextual information in a small range as in the case of EDU 13 and 14, other times we need to see a larger context as in the
<sup>1</sup>https://github.com/YifeiZhou02/Improve-Discourse-Dependency-Parsing-with-Contextualized-Representations
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2250 - 2261
case of EDU 1 and 4, crossing several sentences. This again motivates us to consider encoding contextual representations of EDUs separately on intra- and inter-sentential levels to dynamically capture specific features needed for discourse analysis on different levels.
Another motivation from this example is the discovery that the distribution of discourse relations between EDUs seems to follow certain patterns shared across different articles. Writing patterns are document structures people commonly use to organize their arguments. For example, in scientific abstracts like the instance in Figure 1, people usually first talk about background information, then introduce the topic sentence, and conclude with elaborations or evaluations. Here, the example first states the background of widely used automatic metrics, introduces the topic sentence about their contribution of a significance test followed by evaluation and conclusion. Taking advantage of those writing patterns should enable us to better capture the interplay between individual EDUs with the context.
In this paper, we explore different contextualized representations for DDP in a Sent-First parsing framework, where a complete discourse tree is built up sentence by sentence. We seek to dynamically capture what is crucial for DDP at different text granularity levels. We further propose a novel discourse relation identification method that addresses the task in a sequence labeling paradigm to exploit common conventions people usually adopt to develop their arguments. We evaluate our models on both English and Chinese datasets, and experiments show our models achieve the state-of-the-art results by explicitly exploiting structural information in the context and capturing writing patterns that people use to organize discourses.
In summary, our contributions are mainly twofold: (1) We incorporate the Pre-training and Fine-tuning framework into our design of a Sent-First model and develop better contextualized EDU representations to dynamically capture different information needed for DDP at different text granularity levels. Experiments show that our model outperforms all existing models by a large margin. (2) We formulate discourse relation identification in a novel sequence labeling paradigm to take advantage of the inherent structural information in the discourse. Building upon a stacked BiLSTM architecture, our model brings a new state-of-the
art performance on two benchmarks, showing the advantage of sequence labeling over the common practice of direct classification for discourse relation identification.
A key finding in previous studies in discourse analysis is that most sentences have an independent well-formed sub-tree in the full document-level discourse tree (Joty et al., 2012). Researchers have taken advantage of this finding to build parsers that utilize different granularity levels of the document to achieve the state-of-the-art results (Kobayashi et al., 2020). This design has been empirically verified to be a generally advantageous framework, improving not only works using traditional feature engineering (Joty et al., 2013; Wang et al., 2017), but also deep learning models (Jia et al., 2018b; Kobayashi et al., 2020). We, therefore, introduce this design to our dependency parsing framework. Specifically, sub-trees for each sentence in a discourse are first built separately, then assembled to form a complete discourse tree.
However, our model differs from prior works in that we make a clear distinction to derive better contextualized representations of EDUs from fine-tuning BERT separately for intra- and intersentential levels to dynamically capture different information needed for discourse analysis at different levels. We are also the first to design stacked sequence labeling models for discourse relation identification so that its hierarchical structure can explicitly capture both intra-sentential and inter-sentential writing patterns.
In the case of implicit relations between EDUs without clear connectives, it is crucial to introduce sequential information from the context to resolve ambiguity. Feng and Hirst (2014) rely on linear-chain CRF with traditional feature engineering to make use of the sequential characteristics of the context for discourse constituent parsing. However, they greedily build up the discourse structure and relations from bottom up. At each timestep, they apply the CRF to obtain the locally optimized structure and relation. In this way, the model assigns relation gradually along with the construction of the parsing tree from bottom up, but only limited contextual information from the top level of the partially constructed tree can be used to predict relations. Besides, at each timestep, they sequentially assign relations to top nodes of the partial
tree, without being aware that those nodes might represent different levels of discourse units (e.g. EDUs, sentences, or even paragraphs). In contrast, we explicitly train our sequence labeling models on both intra- and inter-sentential levels after a complete discourse tree is constructed so that we can infer from the whole context with a clear intention of capturing different writing patterns occurring at intra- and inter-sentential levels.
We define the task of discourse dependency parsing as following: given a sequence of EDUs of length  $l$ ,  $(e_1, e_2, \dots, e_l)$  and a set of possible relations between EDUs  $Re$ , the goal is to predict another sequence of EDUs  $(h_1, h_2, \dots, h_l)$  such that  $\forall h_i, h_i \in (e_1, e_2, \dots, e_l)$  is the head of  $e_i$  and a sequence of relations  $(r_1, r_2, \dots, r_l)$  such that  $\forall r_i, r_i$  is the relation between tuple  $(e_i, h_i)$ .
4 Our Model
We follow previous works (Wang et al., 2017) to cast the task of discourse dependency parsing as a composition of two separate yet related subtasks: dependency tree construction and relation identification. We design our model primarily in a two-step pipeline. We incorporate Sent-First design as our backbone (i.e. building sub-trees for each sentence and then assembling them into a complete discourse tree), and formulate discourse relation identification as a sequence labeling task on both intra- and inter-sentential levels to take advantage of the structure information in the discourse. Figure 1 shows the overview of our model.
4.1 Discourse Dependency Tree Constructor
To take advantage of the property of well-formed sentence sub-trees inside a full discourse tree, we break the task of dependency parsing into two different levels, discovering intra-sentential sub-tree structures first and then assembling them into a full discourse tree by identifying the inter-sentential structure of the discourse.
Arc-Eager Transition System Since discourse dependency trees are primarily annotated as projective trees (Yang and Li, 2018), we design our tree constructor as a transition system, which converts the structure prediction process into a sequence of predicted actions. At each timestep, we derive a state feature to represent the state, which is fed into an output layer to get the predicted action. Our
model follows the standard Arc-Eager system, with the action set:  $O = \{Shift, Left - Arc, Right - Arc, Reduce\}$ .
Specifically, our discourse tree constructor maintains a stack  $S$ , a queue  $I$ , and a set of assigned arcs  $A$  during parsing. The stack  $S$  and the set of assigned arcs  $A$  are initialized to be empty, while the queue  $I$  contains all the EDUs in the input sequence. At each timestep, an action in the action set  $O$  is performed with the following definition: Shift pushes the first EDU in queue  $I$  to the top of stack  $S$ ; Left-Arc adds an arc from the first EDU in queue  $I$  to the top EDU in stack  $S$  (i.e. assigns the first EDU in  $I$  to be the head of the top EDU in  $S$ ) and removes the top EDU in  $S$ ; Right-Arc adds an arc from the top EDU in stack  $S$  to the first EDU in queue  $I$  (i.e. assigns the top EDU in  $S$  to be the head) and pushes the first EDU in  $I$  to stack  $S$ ; Reduce removes the top EDU in  $S$ . Parsing terminates when  $I$  becomes empty and the only EDU left in  $S$  is selected to be the head of the input sequence. More details of Arc-Eager transition system can be referred from Nivre (2003).
We first construct a dependency sub-tree for each sentence, and then treat each sub-tree as a leaf node to form a complete discourse tree across sentences. In this way, we can break a long discourse into smaller sub-structures to reduce the search space. A mathematical bound for the reduction of search space of our Sent-First framework for DDP and discourse constituent parsing is also provided in Appendix.
Contextualized State Representation Ideally, we would like the feature representation to contain both the information of the EDUs directly involved in the action to be executed and rich clues from the context from both the tree-structure and the text, e.g. the parsing history and the interactions between individual EDUs in the context with an appropriate scope of text. In order to capture the structural clues from the context, we incorporate the parsing history in the form of identified dependencies in addition to traditional state representations to represent the current state. At each timestep, we select 6 EDUs from the current state as our feature template, including the first and the second EDU at the top of stack  $S$ , the first and the second EDU in queue  $I$ , and the head EDUs for the first and the second EDU at the top of stack  $S$ , respectively. A feature vector of all zeros is used if there is no EDU at a certain position.
Figure 2: An overview of our model. Intra-sentential dependencies are discovered first and inter-sentential dependencies are constructed after that to form a complete dependency tree.
EDU Representations To better capture an EDU in our Sent-First framework, we use pre-trained BERT (Devlin et al., 2018) to obtain representations for each EDU according to different context. We argue that an EDU should have different representations when it is considered in different parsing levels, and thus requires level-specific contextual representations. For intra-sentential tree constructor, we feed the entire sentence to BERT and represent each EDU by averaging the last hidden states of all tokens in that EDU. The reason behind is that sentences are often self-contained sub-units of the discourse, and it is sufficient to consider interactions among EDUs within a sentence for intra-sentential analysis. On the other hand, for inter-sentential tree constructor, we concatenate all the root EDUs of different sentences in the discourse to form a pseudo sentence, feed it to BERT, and similarly, represent each root EDU by averaging the last hidden states of all tokens in each root EDU. In this way, we aim to encourage EDUs across different sentences to directly interact with each other, in order to reflect the global properties of a discourse. Figure 2 shows the architecture for our two-stage discourse dependency tree constructor.
4.2 Discourse Relation Identification
After the tree constructor is trained, we train separate sequence labeling models for relation identification. Although discourse relation identification in discourse dependency parsing is traditionally treated as a classification task, where the common practice is to use feature engineering or neural lan
guage models to directly compare two EDUs involved isolated from the rest of the context (Li et al., 2014a; Shi and Huang, 2018; Yi et al., 2021), sometimes relations between EDU pairs can be hard to be classified in isolation, as global information from the context like how EDUs are organized to support the claim in the discourse is sometimes required to infer the implicit discourse relations without explicit connectives. Therefore, we propose to identify discourse relation identification as a sequence labeling task.
Structure-aware Representations For sequence labeling, we need proper representations for EDU pairs to reflect the structure of the dependency tree. Therefore, we first tile each EDU in the input sequence  $(e_1, e_2, \ldots, e_l)$  with their predicted heads to form a sequence of EDU pairs  $((e_1, h_1), (e_2, h_2), \ldots, (e_l, h_l))$ . Each EDU pair is reordered so that two arguments appear in the same order as they appear in the discourse. We derive a relation representation for each EDU pair with a BERT fine-tuned on the task of direct relation classification of EDU pairs with the [CLS] representation of the concatenation of two sentences.
Position Embeddings We further introduce position embeddings for each EDU pair  $(e_i, h_i)$ , where we consider the position of  $e_i$  in its corresponding sentence, and the position of its sentence in the discourse. Specifically, we use cosine and sine functions of different frequencies (Vaswani et al., 2017) to include position information as:
$$
P E _ {j} = \sin (N o / 1 0 0 0 0 ^ {j / d}) + \cos (I D / 1 0 0 0 0 ^ {j / d})
$$
where  $PE$  is the position embeddings,  $No$  is the position of the sentence containing  $e_i$  in the discourse,  $ID$  is the position of  $e_i$  in the sentence,  $j$  is the dimension of the position embeddings,  $d$  is the dimension of the relation representation. The position embeddings have the same dimension as relation representations, so that they can be added directly to get the integrated representation for each EDU pair.
Stacked BiLSTM We propose a stacked BiLSTM neural network architecture to capture both intra-sentential and inter-sentential interplay of EDUs. After labeling the entire sequence of EDU pairs  $((e_1, h_1), (e_2, h_2), \dots, (e_l, h_l))$  with the first layer of BiLSTM, we select the root EDU for each sentence (namely the root EDU selected from our intra-sentential tree constructor for each setence) to form another inter-sentential sequence. Another separately trained BiLSTM is then applied to label those relations that span across sentences. Note that we will overwrite predictions of inter-sentential relations of the previous layer if there is a conflict of predictions.
Our models are trained with offline learning. We train the tree constructor and the relation labeling models separately. We attain the static oracle to train tree constructors and use the gold dependency structure to train our discourse relation labelling models. Intra- and inter-sentential tree constructors are trained separately. To label discourse relations, we fine-tune the BERT used to encode the EDU pair with an additional output layer for direct relation classification. Sequence labeling models for relation identification are trained on top of the fine-tuned BERT. We use cross entropy loss for training.
Our experiments are designed to investigate how we can better explore contextual representations to improve discourse dependency parsing.
We evaluate our models on two manually labeled discourse treebanks of different language, i.e., Discourse Dependency Treebank for Scientific Abstracts (SciDTB) (Yang and Li, 2018) in English and Chinese Discourse Treebank (CDTB) (Li et al., 2014b). SciDTB contains 1,355 English scientific abstracts collected from ACL Anthology. Averagely, an abstract includes 5.3 sentences, 14.1 EDUs, where an EDU has 10.3 tokens in average.
On the other hand, CDTB was originally annotated as connective-driven constituent trees, and manually converted into a dependency style by Yi et al. (2021). CDTB contains 2,332 news documents. The average length of a paragraph is 2.1 sentences, 4.5 EDUs. And an EDU contains 23.3 tokens in average.
We evaluate model performance using Unlabeled Attachment Score (UAS) and Labeled Attachment Score (LAS) for dependency prediction and discourse relation identification. UAS is defined as the percentage of nodes with correctly predicted heads, while LAS is defined as the percentage of nodes with both correctly predicted heads and correctly predicted relations to their heads. We report LAS against both gold dependencies and model predicted dependencies. We adopt the fine-granularity discourse relation annotations in the original datasets, 26 relations for SciDTB and 17 relations for CDTB.
For both datasets, we trained our dependency tree constructors with an Adam optimizer with learning rate 2e-5 for 3 epochs. Our relation labeling models are all trained with an Adam optimizer until convergence. Learning rate is set to one of  $\{1\mathrm{e} - 5,2\mathrm{e} - 5,4\mathrm{e} - 5\}$ .
Structure Prediction We compare with the following competitive methods for structure prediction. (1) Graph adopts the Eisner's algorithm to predict the most probable dependency tree structure (Li et al., 2014a; Yang and Li, 2018; Yi et al., 2021). (2) Two-stage, which is the state-of-the-art model on CDTB and SciDTB, uses an SVM to construct a dependency tree (Yang and Li, 2018; Yi et al., 2021). (3) Sent-First LSTM is our implementation of the state-of-the-art transition-based discourse constituent parser on RST (Kobayashi et al., 2020), where we use a vanilla transition system with pretrained BiLSTM as the EDU encoder within the Sent-First framework to construct dependency trees. (4) Complete Parser is modified from a state-of-the-art constituent discourse parser on CDTB (Hung et al., 2020), using a transition system with BERT as the EDU encoder to construct a dependency tree. Because of the inherent difference between constituency parsing and dependency parsing, we only adopt the encoding strategy of (4) and (5) into our arc-eager transition system.
We also implement several model variants for
Figure 3: The architecture of our relation labeling stacked BiLSTM model. Hierarchical sequence labeling is used for labeling relations on intra-sentential and inter-sentential levels.
comparison and ablation study. (5) Complete Parser (contextualized) is our modified version of Complete Parser where, instead of encoding each EDU separately, we obtain the EDU representations by encoding the whole sentence with BERT and average the corresponding token representations for the EDU. (6) BERT + Sent-First (shared) incorporate different contextualized embeddings from BERT into the Sent-First framework for parsing at intra- and inter-sentential levels, with the same BERT layer shared across intra-sentential and inter-sentential parsing. (7) BERT + Sent-First fine-tunes separate BERT layers for intra-sentential and inter-sentential parsing independently.
Relation Identification (1) Graph uses an averaged perceptron to classify relations by direct classification (Yi et al., 2021; Yang and Li, 2018). (2) Two-stage exploits careful feature engineering and trains an SVM to classify the relations for pairs of EDUs (Yi et al., 2021; Yang and Li, 2018). (3) Sent-First LSTM uses biLSTM to encode each EDU separately and a feed forward neural network for direct relation classification. (4) BERT is our implementation of the state-of-the-art model from Yi et al. (2021) and Hung et al. (2020), which fine
Model	SciDTB UAS	CDTB
Graph (Cheng21)	57.6	58.5
Two-stage (Cheng21)	70.2	80.3
Sent-First LSTM (Kobayashi20)	63.9	/
Complete Parser (Hung20)	75.4	77.7
Complete Parser (contextualized)	76.1	79.1
BERT + Sent-First (shared)	77.3	81.5
BERT + Sent-First	79.3	82.2
Human	80.2	89.7
Table 1: Model performance of structure prediction on SciDTB and CDTB.
tunes a BERT model with an additional output layer to directly classify both intra-sentential and inter-sentential relations. (5) BERT + BiL formulates dependency discourse relation identification as a sequence labeling task, training an additional layer of BiLSTM on top of the BERT layer finetuned on direct classification. (6) BERT SBiL trains another BiLSTM to label inter-sentential relations on top of the original model BERT + BiL.
Model	SciDTB	CDTB
Gold	Pred.	Gold	Pred.
Graph (Cheng21)	/	42.5	/	41.5
Two-stage (Cheng21)	/	54.5	/	58.7
Sent-First LSTM (Kobayashi20)	52.5	44.6	/	/
BERT (Cheng21)	75.5	63.6	74.9	64.1
BERT + BiL	76.6	64.8	76.5	64.8
BERT + SBiL	77.4	65.0	76.5	64.4
Human	/	62.2	/	77.4