SlowGuess/ACL_2020_OCR huggingface dataset:
Yang Wei, Yuanbin Wu, and Man Lan
School of Computer Science and Technology
godweiyang@gmail.com {ybwu,mlan}@cs.ecnu.edu.cn
We propose a novel linearization of a constituent tree, together with a new locally normalized model. For each split point in a sentence, our model computes the normalizer on all spans ending with that split point, and then predicts a tree span from them. Compared with global models, our model is fast and parallelizable. Different from previous local models, our linearization method is tied on the spans directly and considers more local features when performing span prediction, which is more interpretable and effective. Experiments on PTB (95.8 F1) and CTB (92.1 F1) show that our model significantly outperforms existing local models and efficiently achieves competitive results with global models.
Constituent parsers map natural language sentences to hierarchically organized spans (Cross and Huang, 2016). According to the complexity of decoders, two types of parsers have been studied, globally normalized models which normalize probability of a constituent tree on the whole candidate tree space (e.g. chart parser (Stern et al., 2017a)) and locally normalized models which normalize tree probability on smaller subtrees or spans. It is believed that global models have better parsing performance (Gaddy et al., 2018). But with the fast development of neural-network-based feature representations (Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017), local models are able to get competitive parsing accuracy while enjoying fast training and testing speed, and thus become an active research topic in constituent parsing.
Locally normalized parsers usually rely on tree decompositions or linearizations. From the perspective of decomposition, the probability of trees can be factorized, for example, on individual spans. Teng and Zhang (2018) investigates such a model
which predicts probability on each candidate span. It achieves quite promising parsing results, while the simple local probability factorization still leaves room for improvements. From the perspective of linearization, there are many ways to transform a structured tree into a shallow sequence. As a recent example, Shen et al. (2018) linearizes a tree with a sequence of numbers, each of which indicates words' syntactic distance in the tree (i.e., height of the lowest common ancestor of two adjacent words). Similar ideas are also applied in Vinyals et al. (2015), Choe and Charniak (2016) and transition-based systems (Cross and Huang, 2016; Liu and Zhang, 2017a). With tree linearizations, the training time can be further accelerated to  $\mathcal{O}(n)$ , but the parsers often sacrifice a clear connection with original spans in trees, which makes both features and supervision signals from spans hard to use.
In this work, we propose a novel linearization of constituent trees tied on their span representations. Given a sentence  $\mathcal{W}$  and its parsing tree  $\mathcal{T}$ , for each split point after  $w_{i}$  in the sentence, we assign it a parsing target  $d_{i}$ , where  $(d_i,i)$  is the longest span ending with  $i$  in  $\mathcal{T}$ . We can show that, for a binary parsing tree, the set  $\{(d_i,i)\}$  includes all left child spans in  $\mathcal{T}$ . Thus the linearization is actually sufficient to recover a parsing tree of the sentence.
Compared with prior work, the linearization is directly based on tree spans, which might make estimating model parameters easier. We also build a different local normalization compared with the simple per-span-normalization in Teng and Zhang (2018). Specifically, the probability  $P(d_{i}|i)$  is normalized on all candidate split points on the left of  $i$ . The more powerful local model can help to further improve parsing performance while retaining the fast learning and inference speed (with a greedy heuristic for handling illegal sequences, we can achieve  $\mathcal{O}(n\log n)$  average inference complexity).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3267-3277 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(a) Original parsing tree.
(b) Right binary tree.
(c) Span table and linearization.
Figure 1: The process of generating the linearization of the sentence "She loves writing code". Given an original parsing tree (a), we firstly convert it to a right binary tree by recursively combining the rightmost two children (b). Then, we represent the tree as a span table, and divide it into five parts according to the right boundaries of the spans (c). Green and red circles represent left and right child spans respectively. Gray circles represent spans which do not appear in the tree. In each part, there is only one longest span (green circles), thus the corresponding value of that part is just the left boundary of the green circle.
We perform experiments on PTB and CTB. The proposed parser significantly outperforms existing locally normalized models, and achieves competitive results with state-of-the-art global models (95.8 F1 on PTB and 92.1 F1 on CTB). We also evaluate how the new linearization helps parse spans with different lengths and types.
- Proposing a new linearization which has clear interpretation (Section 2).
- Building a new locally normalized model with constraints on span scores (Section 3).
- Compared with previous local models, the proposed parser achieves better performance (competitive with global models) and has faster parsing speed (Section 4).
2 Tree Linearization
We first prepare some notations. Let  $\mathcal{W} = (w_1, w_2, \ldots, w_n)$  be a sentence,  $\mathcal{T}$  be its binary constituent tree and  $A_{ij} \to B_{ik} C_{kj}$  be a derivation in  $\mathcal{T}$ . Denote  $(i,j) (0 \leq i < j \leq n)$  to be a span from  $w_{i+1}$  to  $w_j$  (for simplicity, we ignore the label of a span).
Definition 1. Given a sentence  $\mathcal{W}$  and its tree  $\mathcal{T}$ , we call  $\mathcal{D} = (d_1, d_2, \ldots, d_n)$  a linearization of  $\mathcal{T}$ , where  $d_i \in \{0, 1, \ldots, i-1\}$  and  $(d_i, i)$  is the longest span ending with  $i$  in  $\mathcal{T}$ .
Clearly, there is only one such linearization for a tree. We have an equal definition of  $\mathcal{D}$ , which
shows the span  $(d_i, i)$  is a left child span.
Proposition 1. Given a tree  $\mathcal{T}$ , the set of spans  $\{(d_i, i) \mid i = 1, 2, \ldots, n\}$  is equal to the set of left child spans
$$
\mathcal {S} = \{(i, j) \mid \exists A _ {i k} \rightarrow B _ {i j} C _ {j k} \} \cup \{(0, n) \}.
$$
Proof. First, for each  $j$ , there is only one left child span  $(i,j)$  ending with  $j$ , otherwise if  $(i',j)$  is a left child span with  $i' \neq i$  (e.g.  $i' < i$ ),  $(i,j)$  must also be a right child span. Therefore  $|S| = n$ . Similarly, if  $i \neq d_j$ ,  $(i,j)$  should be a right child span of  $(d_j,j)$ .
Thus we can generate the linearization using Algorithm 1. For span  $(i,j)$  and its gold split  $k$ , we can get  $d_{k} = i$ . Then we recursively calculate the linearization of span  $(i,k)$  and  $(k,j)$ . Note that the returned linearization  $\mathcal{D}$  does not contain  $d_{n}$ , so we append zero  $(d_{n} = 0$  for the root node) to the end as the final linearization. Figure 1 is a generation process of sentence "She loves writing code." From the span table, it is obvious that there is only one left child span (green circles) ending with the same right boundary.
In the following discussions, we will use  $\mathcal{D}$  and  $S$  interchangeably. Next, we show two properties of a legal  $\mathcal{D}$ .
Proposition 2. A linearization  $\mathcal{D}$  can recover a tree  $\mathcal{T}$  iff.
$$
I. 0 \leq d _ {i} <   i, \forall 1 \leq i \leq n.
$$
The root node is also regarded as a left child span.
2.  $d_{j}$  is not in the range  $(d_i,i),\forall j > i$
Proof. The necessity is obvious. We show the sufficiency by induction on the sentence length. When  $n = 1$ , the conclusion stands. Assuming for all linearizations with length less than  $n$ , property 1 and 2 lead to a well-formed tree, and now consider a linearization with length  $n$ .
Define  $k = \max \{k' \mid d_{k'} = 0, k' < n\}$ . Since  $d_1 = 0$  (by property 1),  $k$  is not none. We split the sentence into  $(0, k), (k, n)$ , and claim that after removing  $(0, n)$ , the spans in  $\mathcal{D}$  are either in  $(0, k)$  or  $(k, n)$ , thus by induction we obtain the conclusion. To validate the claim, for  $k' < k$ , by property 1, we have  $d_{k'} < k' < k$ , thus  $(d_{k'}, k')$  is in  $(0, k)$ . For  $k' > k$ , by property 2, either  $d_{k'} \geq k$  or  $d_{k'} = 0$ . Since  $k$  is the largest index with  $d_k = 0$ , we have  $d_{k'} \neq 0$ , which means  $(d_{k'}, k')$  is in  $(k, n)$ . Therefore, we show the existence of a tree from  $\mathcal{D}$ . The tree is also unique, because if two trees  $\mathcal{T}$  and  $\mathcal{T}'$  have the same linearization, by Proposition 1, we have  $\mathcal{T} = \mathcal{T}'$ .
Proposition 2 also suggests a top-down algorithm (Algorithm 2) for performing tree inference given a legal linearization. For span  $(i,j)$  (with label  $\ell(i,j)$ ), we find the rightmost split  $k$  satisfying  $d_k = i$ , and then recursively decode the two subtrees rooted at span  $(i,k)$  and  $(k,j)$ , respectively. When  $\mathcal{D}$  does not satisfy property 2 (our model can ensure property 1), one solution is to seek a minimum change of  $\mathcal{D}$  to make it legal. However, it is reduced to a minimum vertex cover problem (regarding each span  $(d_i,i)$  as a point, if two spans violate property 2, we connect an edge between them). We can also slightly modify Algorithm 2 to perform an approximate inference (Section 3.4).
Finally we need to deal with the linearization of non-binary trees. For spans having more than two child spans, there is no definition for their middle child spans whether they are left children or right children, thus Proposition 1 might not stand. We recursively combine two adjacent spans from right to left using an empty label  $\varnothing$ . Then the tree can be converted to a binary tree (Stern et al., 2017a). For a unary branch, we treat it as a unique span with a new label which concatenates all the labels in the branch.
3 The Parser
In this section, we introduce our encoder, decoder and inference algorithms in detail. Then we compare our normalization method with two other methods, globally normalized and existing locally normalized methods.
We represent each word  $w_{i}$  using three pieces of information, a randomly initialized word embedding  $e_{i}$ , a character-based embedding  $c_{i}$  obtained by a character-level LSTM and a randomly initialized part-of-speech tag embedding  $p_{i}$ . We concatenate these three embeddings to generate a representation of word  $w_{i}$ ,
$$
\boldsymbol {x} _ {i} = \left[ \boldsymbol {e} _ {i}; \boldsymbol {c} _ {i}; \boldsymbol {p} _ {i} \right].
$$
To get the representation of the split points, the word representation matrix  $\mathbf{X} = [x_1, x_2, \ldots, x_n]$  is fed into a bidirectional LSTM or Transformer (Vaswani et al., 2017) firstly. Then we calculate the representation of the split point between  $w_i$  and  $w_{i+1}$  using the outputs from the encoders,
$$
\boldsymbol {h} _ {i} = \left[ \vec {h} _ {i}; \vec {h} _ {i + 1} \right]. \tag {1}
$$
Note that for Transformer encoder,  $\vec{h}_i$  is calculated in the same way as Kitaev and Klein (2018a).
Since a split point can play two different roles when it is the left or right boundary of a span, we use two different vectors to represent the two roles inspired by Dozat and Manning (2017). Concretely, we use two multi-layer perceptrons to generate two different representations,
$$
\boldsymbol {l} _ {i} = \operatorname {M L P} _ {l} (\boldsymbol {h} _ {i}), \quad \boldsymbol {r} _ {i} = \operatorname {M L P} _ {r} (\boldsymbol {h} _ {i}). \tag {2}
$$
Then we can define the score of span  $(i,j)$  using a biaffine attention function (Dozat and Manning, 2017; Li et al., 2019),
$$
\alpha_ {i j} = \boldsymbol {l} _ {i} ^ {\top} \mathbf {W} \boldsymbol {r} _ {j} + \boldsymbol {b} _ {1} ^ {\top} \boldsymbol {l} _ {i} + \boldsymbol {b} _ {2} ^ {\top} \boldsymbol {r} _ {j},
$$
where  $\mathbf{W}$ ,  $b_{1}$  and  $b_{2}$  are all model parameters.  $\alpha_{ij}$  measures the possibility of  $(i,j)$  being a left child span in the tree.
Different from Stern et al. (2017a) which does global normalization on the probability of the whole tree and Teng and Zhang (2018) which does local normalization on each candidate span, we do normalization on all spans with the same right boundary  $j$ . Thus the probability of span  $(i,j)$  to be a left child span is defined as,
$$
P (i | j) = \operatorname {S o f t m a x} _ {i} \left(\alpha_ {i j}\right), \forall i <   j. \tag {3}
$$
Finally, we can predict the linearization using the probability  $P(i|j)$ ,
$$
d _ {j} = \underset {i} {\arg \max } P (i | j), \forall i <   j. \tag {4}
$$
For label prediction, we first infer the tree structure from the linearization (Section 3.4). Then we use a multi-layer perceptron to calculate the label probability of span  $(i,j)$ ,
$$
P (\ell | i, j) = \operatorname {S o f t m a x} \left(\operatorname {M L P} _ {\text {l a b e l}} \left([ \boldsymbol {l} _ {i}; \boldsymbol {r} _ {j} ]\right)\right) _ {\ell}.
$$
Final predicted label of span  $(i,j)$  is  $\ell(i,j) = \arg \max_{\ell} P(\ell | i, j)$ .
Given a gold parsing tree  $\mathcal{T}$  and its linearization  $(d_{1},d_{2},\ldots ,d_{n})$  , we can calculate the loss using the negative log-likelihood:
$$
\mathcal {L} = - \frac {1}{n} \left(\sum_ {i = 1} ^ {n} \log P (d _ {i} | i) + \sum_ {(i, j, \ell) \in \mathcal {T}} \log P (\ell | i, j)\right).
$$
The loss function consists of two parts. One is the structure loss, which is only defined on the left child spans. The other one is the label loss, which is defined on all the spans in  $\mathcal{T}$ .
3.4 Tree Inference
To reconstruct the tree structure from the predicted linearization  $(d_{1}, d_{2}, \ldots, d_{n})$ , we must deal with illegal sequences. One solution is to convert an illegal linearization to a legal one, and then use Algorithm 2 to recover the tree. However, the optimal converting algorithm is NP hard as discussed in Section 2. We propose two approximate reconstruction methods, both of which are based on replacing line 5 of Algorithm 2. One is to find the largest  $k$  satisfying  $d_{k} \leq i$ ,
$$
k \gets \max \left\{k ^ {\prime} \mid d _ {k ^ {\prime}} \leq i, i <   k ^ {\prime} <   j \right\}.
$$
The other is to find the index  $k$  of the smallest  $d_{k}$  (if there are multiple choices, we choose the largest one),
$$
k\leftarrow \operatorname *{arg  min}_{k^{\prime}}d_{k^{\prime}}.
$$
Both methods are applicable to legal situations, and they have similar performance in our empirical evaluations. The inference time complexity is  $\mathcal{O}(n^2)$  in the worst-case for unbalanced trees, while in average it is  $\mathcal{O}(n\log n)$  (which is the same as Stern et al. (2017a)).
Finally, instead of reconstructing trees from linearization sequences  $(d_{1}, d_{2}, \ldots, d_{n})$ , we could have an accurate CKY-style decoding algorithm from probabilities  $P(i|j)$  (Equation 3). Specifically, it maximizes the product of left child span probabilities,
$$
\mathcal {G} (i, j) = \max  \left\{P (i | k) \times \mathcal {G} (k, j) \mid i <   k <   j \right\},
$$
where  $\mathcal{G}(i,j)$  represents the highest probability of subtree with root node  $(i,j)$ . We can calculate  $\mathcal{G}(0,n)$  using dynamic programming algorithm and back-trace the tree accordingly. The complexity is  $\mathcal{O}(n^3)$ .
<sup>2</sup>Note that we would perform label prediction without the tree inference step which will train the entire parser in linear time as sequence labelling models (Gómez-Rodríguez and Vilares, 2018), but we empirically find that the tree structure helps improving the label classifier.
(a) Global normalization.
(b) Local normalization.
(c) Our normalization.
Figure 2: Factor graphs of three types of normalization. Green circles represent all potential spans in the span table. Red blocks represent scores of the spans. Blue blocks represent normalization operations and dotted lines connect all the spans involved in the normalization. Global normalization (a) needs to calculate the sum of all span scores in parsing tree  $\mathcal{T}$ . Existing local normalization (e.g. Teng and Zhang (2018)) (b) only calculates the probability of each candidate span. Our method (c) does local normalization on all the spans with the same right boundary.
3.5 More Discussions on Normalization
We can compare our locally normalized model (Equation 3) with other probability factorizations of constituent trees (Figure 2).
Global normalization (Figure 2(a)) performs marginalization over all candidate trees, which requires dynamic programming decoding. As a local model, our parser is a span-level factorization of the tree probability, and each factor only marginalizes over a linear number of items (i.e., the probability of span  $(i,j)$  is normalized with all scores of  $(i',j), i' < j$ ). It is easier to be parallelized and enjoys a much faster parsing speed. We will show that its performance is also competitive with global models.
Teng and Zhang (2018) studies two local normalized models over spans, namely the span model and the rule model. The span model simply considers individual spans independently (Figure 2(b)) which may be the finest factorization. Our model lies between it and the global model.
The rule model considers a similar normalization with our model. If it is combined with the top-down decoding (Stern et al., 2017a), the two parsers look similar. We discuss their differences. The rule model takes all ground truth spans from the gold trees, and for each span  $(i,j)$ , it compiles a probability  $P((i,j) \gets (i,k)(k,j))$  for its ground truth split  $k$ . Our parser, on the other side, factorizes on each word. Therefore, for the
same span  $(i,j)$ , their normalization is constrained within  $(i,j)$ , while ours is over all  $i' < j$ . The main advantage of our parser is simpler span representations (not depend on parent spans): it makes the parser easy to batch for sentences with different lengths and tree structures since each  $d_i$  can be calculated offline before training.
Datasets and Preprocessing All models are trained on two standard benchmark treebanks, English Penn Treebank (PTB) (Marcus et al., 1993) and Chinese Penn Treebank (CTB) 5.1. The POS tags are predicted using Stanford Tagger (Toutanova et al., 2003). To clean the treebanks, we strip the leaf nodes with POS tag -NONE- from the two treebanks and delete the root nodes with constituent type ROOT. For evaluating the results, we use the standard evaluation tool  $^{4}$ .
For words in the testing corpus but not in the training corpus, we replace them with a unique label  $<\mathrm{UNK}>$ . We also replace the words in the training corpus with the unknown label  $<\mathrm{UNK}>$  with probability  $p_{\mathrm{unk}}(w) = \frac{z}{z + c(w)}$ , where  $c(w)$  is the number of time words  $w$  appears in the training corpus and we set  $z = 0.8375$  as Cross and Huang (2016).
Hyperparameters We use 100D GloVe (Pennington et al., 2014) embedding for PTB and 80D structured-skipgram (Ling et al., 2015) embedding
<sup>3</sup>We thank an anonymous reviewer for pointing out the connection. The following discussions are based on his/her detailed reviews.
4http://nlp.cs.nyu.edu/evalb/
Type	NP	VP	S	PP	SBAR	ADVP	ADJP	QP	WHNP
Count	18630	8743	5663	5492	1797	1213	893	490	429
PSN Model	93.15	91.81	91.21	89.73	87.81	86.89	73.01	89.80	97.20
Our Model	93.42	92.62	91.95	89.91	88.93	87.39	75.14	91.63	97.44
Difference	+0.27	+0.81	+0.74	+0.18	+1.12	+0.50	+2.13	+1.83	+0.24
Table 1: Comparison on different phrases types. Here we only list top nine types.
Figure 3: F1 scores against span length. Here the length  $l$  represents lengths between  $[l, l + 4]$ .
for CTB. For character encoding, we randomly initialize the character embeddings with dimension 64.
We use Adam optimizer with initial learning rate 1.0 and epsilon  $10^{-9}$ . For LSTM encoder, we use a hidden size of 1024, with 0.33 dropout in all the feed-forward and recurrent connections. For Transformer encoder, we use the same hyperparameters as Kitaev and Klein (2018a). For split point representation, we apply two 1024-dimensional hidden size feed-forward networks. All the dropout we use in the decoder layer is 0.33. We also use BERT (Devlin et al., 2019) (uncased, 24 layers, 16 attention heads per layer and 1024-dimensional hidden vectors) and use the output of the last layer as the pre-trained word embeddings.
Training Details We use PyTorch as our neural network toolkit and run the code on a NVIDIA GeForce GTX Titan Xp GPU and Intel Xeon E5-2603 v4 CPU. All models are trained for up to 150 epochs with batch size 150 (Zhou and Zhao, 2019).
Table 2 shows the final results on PTB test set. Our models (92.6 F1 with LSTM, 93.7 F1 with Trans
former) significantly outperform the single locally normalized models. Compared with globally normalized models, our models also outperform those parsers with LSTM encoder and achieve a competitive result with Transformer encoder parsers. With the help of BERT (Devlin et al., 2018), our models with two encoders both achieve the same performance (95.8 F1) as the best parser (Zhou and Zhao, 2019). Table 3 shows the final results on CTB test set. Our models (92.1 F1) also significantly outperform local models and achieve competitive result amongst global models.
Compared with Teng and Zhang (2018) which does local normalization on single span, our model increases 0.2 F1 on PTB, which shows that doing normalization on more spans is really better. Our model also significantly outperforms Shen et al. (2018) which predicts the syntactic distance of a tree. This indicates the superiority of our linearization method directly tied on the spans.
To better understand the extent to which our model transcends the locally normalized model which does normalization on a single span described in Teng and Zhang (2018), we do several experiments to compare the performance about different lengths of spans and different constituent types.
In order to make a fair comparison, we implement their model by ourselves using the same LSTM encoder as ours. Besides, we ignore the LSTM for label prediction and complex span representations in their models and use simpler settings. Our own implementation achieves the same result as they report (92.4 F1). For convenience, we call their model per-span-normalization (PSN for short) model in the following.
Influence of Span Length First, we analyse the influence of different lengths of spans and the results are shown in Figure 3. We find that for sentences of lengths between [11, 45], our model significantly outperforms PSN model. For short
The source code for our model is publicly available: https://github.com/AntNLP/ span-linearization-parser
Model	LR	LP	F1
Global Model
Stern et al. (2017a)	90.6	93.0	91.8
Gaddy et al. (2018)	91.8	92.4	92.1
Kitaev and Klein (2018a)♣	93.2	93.9	93.6
Zhou and Zhao (2019)♣	93.6	93.9	93.8
Local Model
Vilares et al. (2019)	-	-	90.6
Liu et al. (2018)	-	-	91.2
Ma et al. (2017)	-	-	91.5
Shen et al. (2018)	91.7	92.0	91.8
Liu and Zhang (2017a)	-	-	91.8
Hong and Huang (2018)	91.5	92.5	92.0
Teng and Zhang (2018)	92.2	92.5	92.4
Dyer et al. (2016)♥	-	-	92.4
Stern et al. (2017b)♥	92.6	92.6	92.6
Our Model	92.3	92.9	92.6
Our Model♣	93.3	94.1	93.7
Pre-training/Ensemble/Re-ranking
Liu et al. (2018)	-	-	92.3
Choe and Charniak (2016)	-	-	93.8
Liu and Zhang (2017a)	-	-	94.2
Fried et al. (2017)	-	-	94.7
Kitaev and Klein (2018a)♣	94.9	95.4	95.1
Kitaev and Klein (2018b)♣	95.5	95.7	95.6
Zhou and Zhao (2019)♣	95.7	96.0	95.8
Our Model (+BERT)	95.6	96.0	95.8
Our Model (+BERT)♣	95.5	96.1	95.8