SlowGuess/ACL_2020_OCR huggingface dataset:
Qiankun Fu $^{1,2,3}$ , Yue Zhang $^{2,3}$ , Jiangming Liu $^{4}$  and Meishan Zhang $^{5}$
1. Zhejiang University
2. School of Engineering, Westlake University
3. Institute of Advanced Technology, Westlake Institute for Advanced Study
4. ILCC, University of Edinburgh
5. School of New Media and Communication, Tianjin University, China
fqiankun@gmail.com and zhangyue@westlake.edu.cn
jiangming.liu@ed.ac.uk and mason.zms@gmail.com
Discourse representation tree structure (DRTS) parsing is a novel semantic parsing task which has been concerned most recently. State-of-the-art performance can be achieved by a neural sequence-to-sequence model, treating the tree construction as an incremental sequence generation problem. Structural information such as input syntax and the intermediate skeleton of the partial output has been ignored in the model, which could be potentially useful for the DRTS parsing. In this work, we propose a structural-aware model at both the encoder and decoder phase to integrate the structural information, where graph attention network (GAT) is exploited for effectively modeling. Experimental results on a benchmark dataset show that our proposed model is effective and can obtain the best performance in the literature.
Discourse representation tree structure (DRTS) is a form of discourse structure based on Discourse Representation Theory of Kamp and Reyle (1993), a popular theory of meaning representation (Kamp, 1981; Asher, 1993; Asher and Lascarides, 2003). It is designed to account for a variety of linguistic phenomena, including the interpretation of pronouns and temporal expressions within and across sentences. Correspondingly, as one type of discourse parsing, DRTS parsing (Liu et al., 2018) can be helpful for paragraph or document-level text understanding by converting DRS to tree-style DRTS. (Liu et al., 2019).
Figure 1 shows an example of DRTS, where the leaf nodes are discourse representation units (DRUs), upon which a discourse tree structure built. In particular, a DRU consists of several individual tuples, where each tuple denotes a relation inside the DRU. For example, there is a relationship
Figure 1: Left: Our proposed model with two structure-aware module. Right: The DRTS for a clause in a document: “The letter $_{x_4}$  warns Jewish women $_{x_{16}}$  that they will suffer if they date Arab men.”
"That" between the specific entity  $x_{16}$  and a proposition  $p_4$ . The relationships between the DRUs are organized by a tree skeleton, which includes three types of nodes: the S(DRS) nodes to introduce DRU, the relation nodes for inter-DRU relationship, and the variable nodes, which are used to define S(DRS) (e.g.,  $p_4$ ,  $k_1$  and  $k_4$ ).
There have been only a few existing studies related to DRTS parsing (van Noord et al., 2018a,b). In particular, the end-to-end encoder-decoder model of Liu et al. (2019) gives the state-of-the-art performance, which converts the task into a sequence-to-sequence problem. The input sequence consists of words in paragraphs, encoded by a BiLSTM structure, and the output sequence is top-to-bottom depth-first traversal of the output DRTS tree, which is decoded incrementally with an attention-based LSTM feature representation module. During decoding, Liu et al. (2019) separate the skeleton generation and the DRU producing, as illustrated by Figure 1.
Although highly effective, the above model ig-
6818
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6818-6828 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
nores some useful structure information in both the encoder and the decoder, which can be potentially useful for our task. Specifically, for encoding, syntax-based tree structure information has been demonstrated effective for a number of NLP tasks (Kasai et al., 2019; Li et al., 2018), including several other types of discourse parsing (Yu et al., 2018; Li et al., 2015). For decoding, the skeleton structure of DRTS can be also beneficial for our task. As a two-phase decoding strategy is exploited, the skeleton tree from the first phase could be helpful for DRU parsing of the second phase.
We propose to improve DRTS parsing by making use of the above structure information, modeling dependency-based syntax of the input sentences as well as the skeleton structure to enhance the baseline model of Liu et al. (2019) using Graph Attention Network (GAT) (Veličković et al., 2018), which has been demonstrated effective for tree/graph encoding (Huang and Carley, 2019; Linmei et al., 2019). In particular, we first derive dependency tree structures for each sentence in a paragraph from the Stanford Parser, and then encode them directly via one GAT module, which are fed as inputs for decoding. Second, after the first-state skeleton parsing is finished, we encode the skeleton structures by another GAT module, feeding the outputs for DRU parsing.
Following Liu et al. (2019), we conduct experiments on the Groningen Meaning Bank (GMB) dataset. Results show that structural information is highly useful for our task, bring a significantly better performance over the baseline. In particular, dependency syntax gives an improvement of  $2.84\%$  based on the standard evaluation metrics and the skeleton structure information gives a further improvement of  $1.41\%$ . Finally, our model achieves  $71.65\%$  F1-score for the task,  $4.25\%$  better than the baseline model. Additionally, our model is also effective for sentence-level DRTS parsing, leading to an increase of  $1.72\%$  by the F1-score by our final model. We release our code and best models at http://github.com/seenblank/DRTSparsing for facilitating future research.
2 Discourse Representation Tree (DRT)
Formally, a DRT structure consists of two components according to the function: (1) the leaf nodes and (2) the tree skeleton (non-terminal nodes), respectively. Similar to other types of discourse representation methods, we have minimum semantic
Figure 2: A full DRTS tree for document: “ $k_{1}$ : At least 27 wives of Israeli rabbis have signed a letter urging Jewish women to avoid dating Arab men.  $k_{4}$ : The letter warns Jewish women that they will suffer if they date Arab men.” Red numbers indicate top-down depth-first order traversal of the DRTS skeleton.
units named by DRU, and then a discourse tree is built by the discourse relationships between these minimum units. Figure 2 shows the full tree version of Figure 1 in the introduction.
DRU. DRU serves as terminal nodes of a DRT structure, which is constituted by a set of unordered relation tuples, as shown by the below dashed components of the tree in Figure 1. A relation tuple consists of a relation  $r$  and several arguments  $v_{1} \cdots v_{n}$  in  $r$ , it can be denoted as  $r(v_{1} \cdots v_{n})$ . Variables refer to entities  $x$ , events  $e$ , states  $s$ , time  $t$ , propositions  $p$ , segment  $k$  and constants  $c$ . The relation is used to indicate the discourse connections among the inside variables. A total of 262 relation labels are defined in DRTS. One DRU may include unlimited relation tuples, which are all extracted from the corresponding text pieces.
Skeleton. The skeleton reflects the structural connection between DRUs. Nodes in a skeleton can be divided into three categories, including the (S)DRS nodes, the relation nodes and the variable nodes. In particular, (S)DRS nodes denotes a full semantically-completed node of discourse analysis. The relation node defines a specific discourse relationship over its covered (S)DRS nodes. DRTS has defined six types of DRS relations, including IMP (implication), OR (disjunction), DUP (duplex), POS (possibility), NEC (necessity) and
6819
NOT (negation), respectively, which is orthogonal to the relations inside the DRUs. The variable node assigns one (S)DRS node with a specific symbol. There are two types of variable nodes, namely proposition and segment. For example, in Figure 2, the root is a SDRS node, IMP is a relation nodes and  $k_{1}, p_{4}$  denote the variable nodes.
3 Baseline
We take the multi-step encoder-decoder method of Liu et al. (2019) as the baseline model for DRTS parsing. First, an encoder is used to convert one input paragraph into neural vectors by using word embeddings as well as BiLSTMs, and then a multi-step decoder is exploited to generate a full tree structure in a sequential manner incrementally.
Given a paragraph, we concatenate all the sentences into one sequence, where each sentence is augmented with a start symbol  $\langle s\rangle$  and an end token  $\langle e\rangle$  at the front and end positions, respectively, obtaining a final input sequence for the paragraph  $D = \langle s\rangle ,w_{1,1},\ldots ,w_{1,n_1},\langle e\rangle ,\langle s\rangle ,w_{2,1},\ldots ,w_{m,n_m},\langle e\rangle .$  For simplicity, we use  $D = w_{1},\dots,w_{n}$  to denote the sequence for short.
We use three different embedding representations to denote each word  $w_{i}$ :
$$
\boldsymbol {v} _ {i} = \boldsymbol {e} _ {\text {r a n d}} \left(w _ {i}\right) \oplus \boldsymbol {e} _ {\text {p r e t}} \left(w _ {i}\right) \oplus \boldsymbol {e} _ {\text {l e m}} \left(w _ {i}\right), \tag {1}
$$
where  $e_{\mathrm{rand}}(\cdot)$ , and  $e_{\mathrm{pret}}(\cdot)$  denotes random and pretrained embeddings for current word,  $e_{\mathrm{lem}}(\cdot)$  denotes the random embedding for current word lemma, and  $\oplus$  denotes concatenation,
We then apply MLP over the word representations, and further use BiLSTM to encode the vector sequence:
$$
\begin{array}{l} \boldsymbol {x} _ {1} \dots \boldsymbol {x} _ {n} = \operatorname {M L P} (\boldsymbol {v} _ {1} \dots \boldsymbol {v} _ {n}) \\ \boldsymbol {H} ^ {\text {e n c}} = \boldsymbol {h} _ {1} \dots \boldsymbol {h} _ {n} = \operatorname {B i L S T M} \left(\boldsymbol {x} _ {1} \dots \boldsymbol {x} _ {n}\right), \tag {2} \\ \end{array}
$$
where  $H^{\mathrm{enc}} = h_1 \cdots h_n$  is the encoder output.
We transform the DRTS structure into a sequence of symbols, so that the original DRTS can be restored from the symbol sequence as well. By this transformation, we can apply the sequence-to-sequence architecture for decoding. In particular, a two-stage strategy for the decoding is adapted, first generating the skeleton structure, and then generating the DRUs. The key step is the transformation strategies of the two stages.
Generating the skeleton structure. We define two types of symbols for each skeleton, where the first is the node label conjoined by a left bracket, indicating the start of traversal of the current node, and the second symbol is a right bracket, indicating the end of traversal of the current node. We exploit a top-down depth-first order to traverse the skeleton subtree, finishing a node traversal when all its child nodes have been finished. Figure 2 showed an example to illustrate the transformation. In this way, we can obtain a symbol sequence  $Y^{\mathrm{skt}} = y_1^{\mathrm{skt}}, \dots, y_s^{\mathrm{skt}}$  which is equivalent to the skeleton tree.
Generating the DRUs. After the skeleton is ready, we start the DRU generation process. The DRU nodes are only related to the (S)DRS nodes in the skeleton. Thus we generate DRU nodes one by one according to the (S)DRS nodes in the skeleton structure. For each DRU, we have two types of symbols, one for the relations and the other for the variables. We first generate all the relations and then generate the variables of each relation incrementally. In this way, we can obtain a sequence of  $Y^{\mathrm{dru}} = y_1^{\mathrm{dru}}$ , ...,  $y_t^{\mathrm{dru}}$  for DRU generation.
Sequence decoding. We follow the standard sequence-to-sequence architecture (Liu et al., 2018) to obtain the final sequence  $Y = Y^{\mathrm{skt}}Y^{\mathrm{dru}} = y_1^{\mathrm{skt}}, \ldots, y_s^{\mathrm{skt}}y_1^{\mathrm{dru}}, \ldots, y_t^{\mathrm{dru}}$  incrementally. At each step, we score the candidate next-step symbols based on current observations:
$$
\boldsymbol {o} _ {j} ^ {\mathrm {s k t}} = g ^ {\mathrm {s k t}} (\boldsymbol {H} _ {y _ {<   j} ^ {s k t}}, \boldsymbol {H} ^ {\mathrm {e n c}}),
$$
$$
\boldsymbol {o} _ {k} ^ {\mathrm {d r u}} = g ^ {\mathrm {d r u}} (\boldsymbol {H} _ {y _ {<   k} ^ {\mathrm {s k t}}}, \boldsymbol {H} ^ {\mathrm {s k t}}, \boldsymbol {H} ^ {\mathrm {e n c}}),
$$
where  $H^{\mathrm{enc}}$  refers to the encoder outputs,  $H^{\mathrm{skt}}$  and  $H^{\mathrm{dru}}$  denotes the outputs of skeleton decoder and the DRU decoder uses left-to-right LSTMs over  $Y^{\mathrm{skt}}$  and  $Y^{\mathrm{dru}}$ , respectively, and  $g^{\mathrm{skt}}(\cdot)$  and  $g^{\mathrm{dru}}(\cdot)$  are neural feature extraction functions for predicting skeleton and DRU symbols, respectively. Here we neglect the detailed description for  $g^{\mathrm{skt}}(\cdot)$  and  $g^{\mathrm{dru}}(\cdot)$ , which can be found in Liu et al. (2019).
Training. Given a set of labeled data, the model is trained to minimize average cross-entropy losses
<sup>1</sup>We follow a predefined order for relations. In fact, the order impacts little on the final influence.
2Our description is equivalent to Liu et al. (2019), who split this process into two steps (i.e., relation prediction and variable prediction). We merge the relation and variable predictions for brief.
Figure 3: Model structure with two graph modules over RNN outputs. The hidden state for each word of the encoder is taken as the input node vector of the GAT module using syntax structure, and the output is fed into the skeleton decoder. The output of skeleton decoder is fed into the GAT module with the skeleton structure, and the output of GAT module is used to guide each DRU sequence generation.
over all individual symbol predictions:
$$
L (\theta) = - \frac {1}{N} \sum_ {i} \log p _ {y _ {i} ^ {*}} \tag {4}
$$
where  $\theta$  are the set of model parameters,  $p_{y_i^*}$  denotes the output probability of  $y_i^*$ , which is computed by softmax over  $o_i$ ,  $N$  is the total length of the output sequence.
4 Structure-Aware Seq2Seq
To represent the structure features, we use a GAT module on top of encoder and skeleton decoder stage to enhance the baseline model. The graph module is designed to learn non-local and non-sequential information from structural inputs. In this section, we first describe the GAT in detail and then illustrate its application on our task.
4.1 Graph Attention Network
Given a graph  $G = (V, E)$ , where each node  $v_{i}$  has a initial vectorial representation, the GNN module enriches node representation with neighbor informations derived from the graph structure:
$$
\boldsymbol {H} ^ {l + 1} = \operatorname {G N N} \left(\boldsymbol {H} ^ {l}, \boldsymbol {A}; \boldsymbol {W} ^ {l}\right), \tag {5}
$$
where  $\pmb{H}^{l}\in \mathbb{R}^{n\times d}$  is the stacked hidden outputs for all nodes at layer  $l$  ( $\pmb{H}^{0}$  denotes the input initial representations),  $\pmb{A}\in \mathbb{R}^{n\times n}$  denotes the graph adjacent matrix representation, and  $\pmb{W}^{l}$  is the parameter set of the GNN at layer  $l$ .
Different information aggregation functions lead to different GNN architectures. In particular, GAT uses the attention mechanism (Bahdanau et al., 2014) on graph neighbors, which has been demonstrated more effective than graph convolution neural network (GCN). The aggregation weights in GAT are computed by multi-head attention mechanism (Vaswani et al., 2017).
Specifically, given a node  $i$  with a hidden representation  $h_i^l$  at layer  $l$  and the its neighbors  $\mathcal{N}_i$  as well as their hidden representations, a GAT updates the node's hidden representation at layer  $l + 1$  using multi-head attention:
$$
\boldsymbol {h} _ {i} ^ {l + 1} = \left\| _ {k = 1} ^ {K} \sigma \left(\sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} ^ {k} \boldsymbol {W} ^ {k} \boldsymbol {h} _ {j} ^ {l}\right) \right. \tag {6}
$$
where  $\parallel$  represents concatenation,  $\sigma$  is a sigmoid function, and  $W^{k}$  is the corresponding weight matrix of input linear transformation.  $\alpha_{ij}^{k}$  are normalized attention coefficients computed by the  $k$ -th attention mechanism:
$$
\begin{array}{l} \alpha_ {i j} ^ {k} = \operatorname {S O F T M A X} _ {j} \left(e _ {i j}\right) \\ = \frac {\exp \left(e _ {i j}\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(e _ {i k}\right)} \tag {7} \\ \end{array}
$$
where  $e_{ij}$  is attention coefficient that indicate the importance of node  $j$  to node  $i$  computed by:
$$
e _ {i j} = \text {L e a k y R e L U} \left(f \left[ \boldsymbol {W h} _ {i} \| \boldsymbol {W h} _ {j} \right]\right) \tag {8}
$$
$f(\cdot)$  is a single-layer feed-forward neural network, parameterized by a shared weight,  $\pmb{W}$  denotes a
6821
Section	#Doc	#Sent	AVGsent	AVGword
Train	7843	48599	6.2	135.3
Devel	991	6111	6.2	134.0
Test	1035	6469	6.3	137.2
Table 1: Statistics on GMB document level benchmarks,  $\mathbf{AVG}_{\mathrm{sent}}$  and  $\mathbf{AVG}_{\mathrm{word}}$  denote the average number of sentences and words per document, respectively.
shared linear transformation and LeakyReLU is a non-linearity activation function.
4.2 GAT for the Encoder
On the encoder side, we equip the inputs with dependency syntax structures, which have been demonstrated helpful for closely-related tasks such as RST discourse parsing. A GAT module is used to represent the encoder output as mentioned in Section 4.1. We transform the document into a dependency graph represented by a undirected adjacent matrix using an off-the shelf dependency parser (Chen and Manning, 2014). The hidden states of each node is updated with a multi-layer GAT network on the adjacent matrix  $A$ :
$$
\boldsymbol {H} ^ {\mathrm {g} - \mathrm {e n c}} = \operatorname {G A T} ^ {\mathrm {e n c}} \left(\boldsymbol {H} ^ {\mathrm {e n c}} \oplus \boldsymbol {E} ^ {\mathrm {s y n}}, \boldsymbol {A}; \boldsymbol {W}\right), \tag {9}
$$
where  $E^{\mathrm{syn}}$  is the embedding outputs of the syntactic labels in the dependency tree.
The learned representation  $H^{\mathrm{g - enc}}$  is used to substitute the original  $H^{\mathrm{enc}}$  for predictions.
4.3 GAT for the Decoder
We further enhance the baseline model by exploiting the partial output after skeleton prediction step is finished. On one hand, the skeleton structures can guide for DRU parsing. On the other hand, the joint skeleton and DRU parsing can further help to rerank the skeleton predictions as well, since global skeleton representations are exploited.
Specifically, after all the skeleton nodes are generated, we construct a graph based on the nodes except the right parenthesis as shown in Figure 3. We use a GAT network on top of the hidden states to capture global structure information:
$$
\boldsymbol {H} ^ {\mathrm {g - s k t}} = \operatorname {G A T} ^ {\mathrm {s k t}} \left(\boldsymbol {H} ^ {\mathrm {s k t}} \oplus \boldsymbol {E} ^ {\mathrm {s k t}}, \boldsymbol {A}; \boldsymbol {W}\right), \tag {10}
$$
where  $\pmb{E}^{\mathrm{skt}}$  is the embedding outputs of the node labels in the generated skeleton tree, and the global skeleton-aware representation  $H^{\mathrm{g - skt}}$  is used instead of the original  $H^{\mathrm{skt}}$  for future predictions.
5.1 Data and Settings
Data We conduct experiments on the benchmark GMB dataset, which provides a large collection of English texts annotated with Discourse Representation Structures (Bos et al., 2017). We follow Liu et al. (2019) using the processed tree-based DRTS format, and focus on document-level parsing. The data statistics are shown in Table 1.
Hyperparameters We exploit the same hyperparameters as Liu et al. (2019) for fair comparison. In particular, we use the same pre-trained 100-dimensional word embeddings, which are trained on the AFP portion of the English Gigaword corpus. The sizes of random word and lemma embeddings are set to 300 and 100, respectively. The hidden sizes of BiLSTM modules in encoder and decoder are set to 300 and 600, respectively. In addition, the BiLSTM layer sizes of encoder and decoder are respectively 2 and 1. The hidden size of GAT modules is set to 300 and 600 for encoder and decoder, respectively.
Following Liu et al. (2019), we adopt the COUNTER (van Noord et al., 2018a) tool to evaluate our final experimental results. In particular, we first transform the DRTS into a clause format and then run the standard evaluation script to obtain the F1-scores of our results compared with the gold-standard clause form. Note that COUNTER is computationally expensive, requiring more than 50 hours for the entire test dataset by using more than 100 threads. To facilitate development and analysis experiments, we suggest three alternatives for evaluation particularly for development experiments:
(1) BLEU: a standard BLEU (Papineni et al., 2002) value is adopted as the metric to evaluate the resulting node sequence against the gold-standard output, since we model the task as a sequence-to-sequence task.
(2) Skeleton: The bracket scoring method of constituent parsing is exploited to evaluate the skeleton performance, by regarding terminal DRU nodes as words in comparison with a constituent tree.
3https://nlp.cs.nyu.edu/evalb/
6822
(a) structure labels
Figure 4: Feature ablation experiments.
(b) GAT modules
(3) Tuple: The F1-score of tuple-level matching is exploited to measure the DRU performance, since the basic units inside a DRU are tuples of relation-variable functions. Exact matching is adopted considering variable orders.
The BLEU is used for development and the Skeleton and Tuple are used for analysis.
5.3 Development Experiments
We conduct experiments on the development dataset to understand the key factors of our proposed model.
Impact of structure labels Syntactic arcs and skeleton labels are embedded and concatenated to the embedding of the current node when using GAT to model the tree structure. We conduct a comparison to examine their effectiveness in our model. Figure 4(a) shows the results. We can see that a performance degradation occurs without these label embeddings. In particular, BLEU score drops by 0.4 without syntax label embeddings and 0.93 without skeleton label embeddings, which shows that modeling label information improves unfixed skeleton tree structure even more.
Impact of GAT setting As our proposed modules involve a  $l$ -layer GAT, we investigate the effect of the layer number  $l$  on the dev set as shown in Table 2. In particular, we vary the value of  $l$  in the set  $\{1, 2, 3, 4, 5\}$  and measure the corresponding BLEU scores. The structural-aware model equipped with GAT achieves the best performance when  $l$  is 2, which justifies the selection on the number of layers in the experimental setting section. Moreover, a dropping trend on both metrics is present as  $l$  increases. For a larger  $l$ , the GAT module becomes more difficult to train due to larger amounts of parameters. One intuitive reason is that each layer of the GAT module aggregates
Model	BLEU	Model	BLEU
Head=1	48.76	layers=1	49.11
Head=2	49.48	layers=2	50.04
Head=3	50.01	layers=3	49.72
Head=4	50.04	layers=4	49.01
Head=5	50.04	layers=5	48.54
Table 2: GAT settings results on development set.
Model	BLEU	exact F1
Liu et al. (2019)
(baseline)	46.86	66.56
GAT-encoder	48.24	69.40
GAT-decoder	50.04	70.81
GAT-enc+dec	50.16	71.65
Tree-LSTM	48.36	69.66
GCN	49.88	70.72
Table 3: Final results on the test dataset.
the direct neighbor information of a node. After 2 layers, each node can obtain sufficient information, and further more layers can bring noise.
We make comparison with multi-head attention, varying the heads in the set  $\{1,2,3,4,5\}$  and checking the corresponding BLEU scores. Theoretically, the larger the number of heads, the better the performance of the model. As can be seen in Table 2, when the number of heads exceeds 4, the performance becomes relatively stable. We thus choose the head to be 4 for the remaining experiments.