SlowGuess/NAACL_2021_OCR huggingface dataset:
Thomas Ruprecht and Richard Morbitz
Technische Universität Dresden
01062 Dresden, Germany
{thomas.ruprecht,richard.moerbitz}@tu-dresden.de
We present the first supertagging-based parser for linear context-free rewriting systems (LCFRS). It utilizes neural classifiers and outperforms previous LCFRS-based parsers in both accuracy and parsing speed by a wide margin. Our results keep up with the best (general) discontinuous parsers, particularly the scores for discontinuous constituents establish a new state of the art. The heart of our approach is an efficient lexicalization procedure which induces a lexical LCFRS from any discontinuous treebank. We describe a modification to usual chart-based LCFRS parsing that accounts for supertagging and introduce a procedure that transforms lexical LCFRS derivations into equivalent parse trees of the original treebank. Our approach is evaluated on the English Discontinuous Penn Treebank and the German treebanks Negra and Tiger.
In NLP, constituency parsing is a task that assigns – usually tree-shaped – syntactic structures to sentences. Formalisms such as context-free grammars (CFG) are used in this setting because they are conceptually simple, interpretable, and parsing is tractable (cubic in sentence length).
Discontinuous constituents span non-contiguous sets of positions in a sentence. The resulting phrase structures do not take the shape of a tree anymore, as they contain crossing branches (cf. the left of Fig. 1), and cannot be modeled by CFG. As a countermeasure, many treebanks, e.g. the Penn Treebank (PTB; Marcus et al., 1994), denote these phrase structures as trees nevertheless and introduce designated notations for discontinuity, which is then often ignored in parsing. However, discontinuity occurs in about  $20\%$  of the sentences in the PTB and to an even larger extent in German treebanks such as Negra and Tiger. For parsing discontinuous constituents, so-called
mildly context-sensitive grammar formalisms have been investigated, e.g. tree-adjoining grammars (TAG; Joshi et al., 1975) and linear context-free rewriting systems (LCFRS; Vijay-Shanker et al., 1987). An LCFRS derivation of a discontinuous phrase is shown in the right of Fig. 1. The increased expressiveness of these formalisms comes at the cost of a higher parsing complexity: given a sentence of length  $n$ , parsing is in  $O(n^{6})$  for TAG and  $O(n^{3\cdot \mathrm{fo}(G)})$  for a binary LCFRS  $G$ . The grammar-specific fanout  $\mathrm{fo}(G)$  indicates that  $G$  can parse constituents spanning up to  $n$  non-contiguous sets of positions. TAG have the same expressiveness as LCFRS with fanout 2 (Seki et al., 1991), which accounts for  $96.67\%$  of the sentences in Negra and  $96.83\%$  of the sentences in Tiger (Maier and Søgaard, 2008). Previous publications have established mildly context-sensitive formalisms in the field of statistical constituent parsing, and found methods to tame the high parsing complexity (Evang and Kallmeyer, 2011; Kallmeyer and Maier, 2013; Angelov and Ljunglöf, 2014; van Cranenburgh, 2012).
One approach for making parsing with mildly context-sensitive grammars tractable is supertagging, which was originally introduced for lexical TAG (Bangalore and Joshi, 1999). A TAG is lexical if each rule contains exactly one lexical item, i.e. word in the parsed language. The supertagger is a (often discriminative) classifier that selects for each position of the input sentence a subset of the rules of the TAG; these are the so-called supertags. Parsing is then performed with the much smaller grammar of supertags. Research on supertagging has also been conducted in the context of combinatory categorial grammars (CCG; Clark, 2002), but not yet for LCFRS. The use of recurrent neural networks (RNN) as classifiers in supertagging has improved their accuracy by far (Vaswani et al., 2016; Kasai et al., 2017; Bladier et al., 2018; Kadari et al., 2018).
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2923-2935 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: A discontinuous phrase structure tree of the sentence  $A$  hearing is scheduled on the issue today (left) and a corresponding LCFRS derivation of the discontinuous noun phrase  $A$  hearing on the issue (right).
Recently, Moritz and Ruprecht (2020) introduced a lexicalization procedure for probabilistic LCFRS<sup>1</sup>, paving the way to employ supertagging for parsing with this formalism. Early experiments showed that the approach is infeasible in realistic settings: the set of rules explodes in a step of the construction where new rules are introduced for pairs of terminals in the grammar. To mitigate this problem, we conduct the procedure for single derivations. Consequently, we only have to construct rules for pairs of terminals that occur in sibling nodes of a derivation (cf. step (4) in Section 4). Moreover, we consider unweighted LCFRS, as weights of underlying grammar structures are usually not considered in supertagging-based approaches.
In this paper, we present the first supertagging-based parser for LCFRS. Section 3 extends the usual chart-based parsing approach for LCFRS to account for supertagging with lexical LCFRS. Section 4 adapts the lexicalization procedure by Morbitz and Ruprecht (2020) to efficiently induce a lexical LCFRS from any given treebank. We implemented and evaluated the approach. Section 5 describes the experimental setups of our evaluation using three discontinuous treebanks (one English and two German). Section 6 compares our results to recent LCFRS-based parsers and other state-of-the-art parsers for discontinuous constituents. The implementation of our approach is published on GitHub.
We start by introducing some basic notation that will be used throughout Sections 3 and 4. The set of non-negative (resp. positive) integers is denoted by  $\mathbb{N}$  (resp.  $\mathbb{N}_{+}$ ). We abbreviate  $\{1, \dots, n\}$  by  $[n]$ ; for each  $n \notin \mathbb{N}_{+}$ , the set  $[n]$  is the empty set. An alphabet  $\Sigma$  is a finite and non-empty set; the set of (finite) strings over  $\Sigma$  is denoted by  $\Sigma^{*}$ . The symbol  $\varepsilon$  denotes an empty string or sequence.
Compositions. Linear context-free rewriting systems (LCFRS) extend the rule-based string rewriting mechanism of CFG to string tuples; we describe the generation process by compositions. Let  $k \in \mathbb{N}$  and  $s_1, \ldots, s_k, s \in \mathbb{N}_+$ ; one can think of  $k$  as the number of arguments of a function mapping string tuples of the sizes  $s_1, \ldots, s_k$  to a string tuple of size  $s$ . A  $\Sigma$ -composition is a tuple  $(u_1, \ldots, u_s)$  where each  $u_1, \ldots, u_s$  is a non-empty string over  $\Sigma$  and variables of the form  $x_i^j$  with  $i \in [k]$  and  $j \in [s_i]$ . Each of these variables must occur exactly once in  $u_1 \cdots u_s$  and they are ordered such that  $x_i^1$  occurs before  $x_{i+1}^1$  and  $x_i^j$  occurs before  $x_i^{j+1}$  for each  $i \in [k-1]$  and  $j \in [s_i-1]$ . The set of all such compositions is denoted by  $C_{(s_1 \cdots s_k, s)}^{\Sigma}$ .
As usual in the literature, we will only consider binary compositions (where  $k \leq 2$ ) in the following. Variables of the form  $\mathbf{x}_1^i$  and  $\mathbf{x}_2^j$  are abbreviated by  $\mathbf{x}_i$  and  $\mathbf{y}_j$ , respectively.
We associate with each composition  $(u_{1},\ldots ,u_{s})\in \mathbf{C}_{(s_{1}\dots s_{k},s)}^{\Sigma}$  a function from  $k$  string tuples, where the  $i$  -th tuple is of arity  $s_i$  , to a string tuple of arity  $S$  . This function is denoted by  $\llbracket (u_1,\ldots ,u_s)\rrbracket$  . Intuitively, it replaces each variable of the form  $\mathrm{x}_i$  in  $u_{1},\ldots ,u_{s}$  by the  $i$  -th component of the first argument, and  $\mathrm{y}_j$  by the  $j$  -th component of the second argument. The identity composition  $(\mathrm{x}_1,\ldots ,\mathrm{x}_s)$  is denoted by  $\operatorname {id}_s$
1Their work is an instance of the lexicalization of (unweighted) multiple context-free tree grammars by Engelfriet et al. (2018). 2https://github.com/truprecht/ lcfrs-supertagger
Figure 2: An overview over the supertagging-based parsing procedure. A sequence tagger predicts  $k$  (here,  $k = 1$ ) supertags and one enriched preterminal (cf. step (4) in Section 4) for each sentence position. The supertags are rules of a uni-lexical LCFRS (the annotation of the nonterminals is explained in Section 4). The terminal of each rule is the sentence position it was predicted for (rather than the word at that position). The range of sentence positions is parsed and finally the resulting derivation is transformed into a parse tree. The transformation requires the predicted nonterminals, which are used as preterminals.
Let  $c \in \mathrm{C}_{s_1 \dots s_k, s}^{\Sigma}$  be a composition where  $k \in [2]$ ,  $i \in [k]$  such that  $s_i = 1$ , and  $w \in \Sigma^*$ . We obtain the partial application of  $c$  to  $w$  as  $i$ -th argument, denoted by  $\llbracket c \rrbracket_i(w)$  as follows:
-  $\llbracket c\rrbracket_2(w)\in \mathbf{C}_{(s_1,s)}^\Sigma$  is obtained from  $c$  by replacing  $\mathrm{y}_1$  by  $w$  and
-  $\llbracket c\rrbracket_1(w)$  is obtained from  $c$  by replacing  $\mathbf{x}_1$  by  $w$  and each variable  $\mathrm{y}_j$  by  $\mathbf{x}_j$ . If  $k = 1$ , then  $\llbracket c\rrbracket_1(w) \in \mathbf{C}_{\varepsilon, s}^{\Sigma}$ , otherwise  $\llbracket c\rrbracket_1(w) \in \mathbf{C}_{s_2, s}^{\Sigma}$ .
LCFRS. A (binary) LCFRS is a tuple  $G = (N, \Sigma, S, R)$  where
-  $N$  is a finite set (nonterminals) where each nonterminal  $A \in N$  has a fanout  $\operatorname{fo}(A) \in \mathbb{N}_{+}$ ,
-  $\Sigma$  is an alphabet (terminals),
-  $S \subseteq N$  (initial nonterminals) such that  $\operatorname{fo}(A) = 1$  for each  $A \in S$ , and
-  $R$  is a finite set (rules); each rule in  $R$  is of the form  $A \to c(B_1, \ldots, B_k)$ , where  $k \in \{0, 1, 2\}$ ,  $A, B_1, \ldots, B_k \in N$ , and  $c \in C_{(\mathrm{fo}(B_1) \cdots \mathrm{fo}(B_k), \mathrm{fo}(A))}^{\Sigma}$ . The function  $\llbracket c \rrbracket$  maps  $k$  string tuples (of sizes  $\mathrm{fo}(B_1), \ldots, \mathrm{fo}(B_k)$ ) to a string tuple of size  $\mathrm{fo}(A)$ . We call  $A$  the left-hand side (lhs),  $B_1, \ldots, B_k$  the right-hand side (rhs) and  $c$  the rule's composition. We drop the parentheses around the rhs if  $k = 0$ .
In our examples, whenever the fanout of a nonterminal is greater than 1, the fanout is the subscript of the nonterminal. For instance,  $\mathrm{VP}_2$  denotes a verbal phrase with fanout 2. The fanout of  $G$  is  $\mathrm{fo}(G) = \max_{A\in N}\mathrm{fo}(A)$ .
Rules of the form  $A \to c$ ,  $A \to c(B)$ , and  $A \to c(B_1, B_2)$  are called terminating, monic, and branching, respectively. A rule is called (uni-/double-)lexical, if its composition contains at least one terminal (resp. exactly one terminal/ exactly two terminals). The LCFRS  $G$  is called (uni-)lexical, if each rule is (uni-)lexical.
A derivation in  $G$  (starting with  $A \in N$ ) is a tree over rules  $d = r(d_{1},\ldots ,d_{k})$  such that  $r$  is of the form  $A \to c(B_1,\dots,B_k)$  and each  $d_{i}$  is a derivation in  $G$  starting with  $B_{i}$ . The set of derivations in  $G$  is denoted by  $\mathrm{D}^G$ . The string tuple computed by  $d$  is defined recursively as  $w = [c](w_{1},\ldots ,w_{k})$  where  $w_{1},\ldots ,w_{k}$  are the string tuples computed by  $d_{1},\ldots ,d_{k}$ ; in the following we also call  $d$  a derivation for  $w$ .
3 Supertagging-based parsing
Our parsing model consists of two components: a uni-lexical LCFRS and a discriminative sequence
tagger which we henceforth call supertagger. The LCFRS is induced from a treebank by an adaptation of the construction of Moritz and Ruprecht (2020); the interested reader may find a detailed description of this procedure in Section 4. After the induction, we replace every terminal of the LCFRS by the wildcard symbol “\_”, and we refer to the resulting rules as supertags.
Our parsing pipeline is depicted in Fig. 2.
(1) Given a sentence  $w$ , the supertagger predicts for each position of  $w$  the  $k$  best supertags, where  $k$  constitutes a hyperparameter of our approach.
(2) We combine the predicted supertags to a new LCFRS which we call  $G_{w}$ . In doing so, we replace the wildcard of each supertag by the sentence position it was predicted for.
(3) We employ a usual chart-based parsing algorithm to parse the sequence  $12 \cdots |w|$  of sentence positions with  $G_w$ .
(4) We transform the resulting derivation in  $G_{w}$  into a parse tree of the same form as those in the original treebank.
As  $G_{w}$  only resembles a fraction of all supertags, this approach shifts a huge amount of work from parsing with grammars to predicting the rules. Thus its success is mainly determined by the quality of the supertagger.
4 Inducing Lexical LCFRS
Our lexicalization scheme is based on Moritz and Ruprecht (2020). However, we ignore all weights and perform lexicalization on individual derivations rather than on a grammar induced from the entire treebank. More specifically, we directly read off a set of uni-lexical rules from each tree in the treebank; then the union of these rules forms our uni-lexical LCFRS  $G_{\mathrm{lex}}$ . In contrast to that, Moritz and Ruprecht (2020) first induce an LCFRS  $G$  from the entire treebank and then lexicalize  $G$ . Thus  $G_{\mathrm{lex}}$  may have a different language than the lexicalization of  $G$ .
We obtain a set of uni-lexical rules from each tree  $t$  in the treebank by the following procedure.
(1) Binarize the tree. The symbol  $|$  is appended to constituents that result from binarization (this reflects Markovization with a vertical context of 1 and a horizontal context of 0).
(2) Transform the tree into an LCFRS derivation using the standard technique for induction of LCFRS (Maier and Søgaard, 2008).
(3) Collapse every chain of monic rules; the
nonterminals of each chain are combined to a new nonterminal.
(4) Remove every terminating rule that has a parent and insert the terminal from its composition into the parent.
(5) Propagate terminals from double-lexical terminating rules into non-lexical branching rules. All rules in the resulting derivation are lexical.
(6) Split all remaining double-lexical terminating rules into two uni-lexical rules. All rules in the resulting derivation are uni-lexical. The resulting derivation is called  $d_{\mathrm{lex}}(t)$ .
(7) Read off the rules of  $d_{\mathrm{lex}}(t)$ ; call them  $R(t)$ .
These steps are defined such that in the LCFRS formed by  $R(t)$ ,  $d_{\mathrm{lex}}(t)$  is a derivation for the sentence of  $t$ . Moreover, we are able to reconstruct  $t$  from  $d_{\mathrm{lex}}(t)$  by reverting steps (6) to (1) (we will give the details later).
Finally, we obtain the uni-lexical LCFRS  $G_{\mathrm{lex}}$  by combining the rules  $R(t)$  for each tree  $t$ . The initial nonterminals of  $G_{\mathrm{lex}}$  are all left-hand sides of roots of  $d_{\mathrm{lex}}(t)$ .
Let  $t$  be a tree in the treebank. Steps (1) and (2) and their reversal are standard techniques for trees and LCFRS. After applying them to  $t$ , we obtain an LCFRS derivation in which each occurring rule is either of the form
-  $A \to (\sigma)$ , where  $\sigma$  is a terminal and  $A$  is the part-of-speech tag of  $\sigma$ ,
-  $A \to c(B_1)$  where  $\operatorname{fo}(A) = \operatorname{fo}(B)$  and  $c = \operatorname{id}_{\operatorname{fo}(A)}$ , or
-  $A \to c(B_1, B_2)$  where  $c$  contains no terminals and none of  $B_1, B_2$  is an initial nonterminal.
Let us denote this derivation by  $d$ .
In the following, we describe steps (3) to (6) of the above procedure in more detail (showing examples in Figs. 3 to 6) and also glimpse at how the individual steps are reverted.
Step (3). We repeatedly replace parts in  $d$  of the form  $A \to c(B)(B \to c')$  by  $A + B \to c'$ , and  $A \to c(B)\big(B \to c'(C_1, C_2)\big(\dots\big)\big)$  by  $A + B \to c'(C_1, C_2)\big(\dots\big)$ , until there is no monic rule in  $d$  left. If the occurrence of  $A \to c(B)$  is not the root of  $d$ , then the corresponding nonterminal in the parent's rhs is replaced by  $A + B$ . After this step, there are only branching rules and terminating rules in  $d$ . Figure 3 shows an example for this step.
This step is easy to reverse, as the composition of every removed rule is  $c = \mathrm{id}_{\mathrm{fo}(B)}$ . We give the
Note that root nodes in the derivation may be collapsed, this is why we use LCFRS with multiple initial nonterminals.
(a) A derivation for scheduled today. Gray arrows show how the bottom-most composition is chained with the monic rule on top.
Figure 3: Example for step (3).
(b) The derivation resulting from applying step (3) to the derivation in Fig. 3a.
formaldescription in Appendix A.3.
Step (4). We remove every non-root occurrence  $r$  of a terminating rule  $A \to (\sigma)$  in  $d$ . Let  $r$  be the  $i$ th child of its parent (with  $i \in [2]$ ), then we replace the parent's composition  $c$  by  $[[c]]_i(\sigma)$  and remove the  $i$ th nonterminal in the parent's rhs.
We note that the parent becomes lexical, and after this step, every rule in  $d$  is either branching or lexical. Moreover, every terminal rule in  $d$  is either double-lexical (if both children were removed) or the root of  $d$  (and thus its only node). Figure 4 shows an example for this step.
(a) A derivation for the string tuple (A hearing, on the issue). Gray arrows show the terminals that are put into binary non-lexical rules during step (4).
Figure 4: Example for step (4).
(b) The derivation resulting from applying step (4) to the derivation in Fig. 4a.
Clearly, this step loses information, namely the left-hand sides of the removed rules. These nonterminals are part-of-speech tags (that may be enriched with nonterminals of collapsed monic rules
from the previous step). For the reversal of this step, we opted to predict them along with the supertags as part of the supertagger. The formal description of the reversal is given in Appendix A.2.
Step (5). For each occurrence  $r$  of a branching rule  $A \rightarrow c(A_1, A_2)$  in  $d$ , let us consider the occurrence  $t$  of the leftmost terminating rule (i.e.  $t$  is a leaf) that is reachable via the second successor of  $r$ . For example, in Fig. 5a, the two binary rules  $(r)$  are end points of gray arrows; these arrows start at the mentioned leaves  $(t)$ . Our goal is to remove one terminal from  $t$  and propagate it all the way up to  $r$ . For this, at each node  $s$  on the path from  $t$  to  $r$  (from bottom up):
- If  $s$  is  $t$ , we remove the leftmost terminal in the rule's composition at  $s$ .
- If  $s$  is neither  $t$  nor  $r$ , we insert the last removed terminal right before the variable  $x_1$  and then remove the leftmost terminal in the rule's composition at  $s$ .
We note that if the rule at  $s$  is monic and the variable  $\mathbf{x}_1$  occurs right of the terminal in its composition, then we propagate a different terminal than the one received from the child. In order to be able to reverse this step, we need to remember whether the terminal in the rule's composition stayed the same or was swapped with the terminal received from the child. In the following, we consider this information as part of the rule (cf. the gray annotation swapped in Fig. 5).
- If  $s$  is  $r$ , we insert the last removed terminal right before the variable  $y_1$  in the rule's composition at  $s$ .
If  $s \neq r$ , let  $s'$  be the parent of  $s$  and  $s$  the  $i$ th child of  $s'$ . If, after removal of a terminal at  $s$ , the first component in the composition is empty:
- we annotate the lhs nonterminal at  $s$  and the  $i$ th rhs nonterminal at  $s'$  with  $^{-}$  and remove the empty component, and
- if  $i = 1$  (resp.  $i = 2$ ), we remove  $x_1$  (resp.  $y_1$ ) and replace every other occurrence of  $x_i$  by  $x_{i - 1}$  (resp.  $y_j$  by  $y_{j - 1}$ ) at  $s'$ .
Otherwise, we annotate the nonterminals with  ${}^{ + }$  .
We note that the rule at  $r$  is uni-lexical and branching now, the rule at  $t$  is uni-lexical and terminating, and the number of terminals in each rule between them did not change. After this step, every rule in  $d$  is lexical. Figure 5 shows an example for this step.
There is a suitable leaf  $t$  for every branching rule
$r$ . Intuitively, this holds since (a) after step (4) every leaf of  $d$  is a double-lexical rule and (b) for each branching rule we first go to its second successor and then follow the path of first successors until we reach a leaf. Here, (a) guarantees that there exists a double-lexical rule for each branching rule and (b) guarantees that each double-lexical rule is "assigned" to at most one branching rule, thus at most one terminal is removed from it. We refer the more interested reader to consult the proof of correctness by Engelfriet et al. (2018); this proof also applies to our method.
(a) A derivation for the string tuple (A hearing, scheduled on the issue today). Gray arrows show how terminals will be propagated through the derivation to lexicalize branching rules during step (5).
Figure 5: Example for step (5).
(b) The derivation resulting from applying step (5) to the derivation in Fig. 5a. A gray annotation swapped marks a monic rule whose terminal changed.
The reversal of this step removes all annotation  $(^{+},^{-}$ , and swapped) and restores each composition in  $d$  to its original form. We note that the original composition can be obtained deterministically; the construction is given in Appendix A.1.
Step (6). We replace the rightmost terminal  $\sigma_{2}$  in the composition of each double-lexical terminating rule by a variable and add a new nonterminal  $A^{\mathrm{R}}$  to the rule's right-hand side (making it a uni-lexical monic rule). Then we insert the rule  $A^{\mathrm{R}}\rightarrow (\sigma_{2})$  as a child. After this step, every rule in  $d$  is unilexical. Figures 5b and 6 show an example for this step. The reversal of this step is straightforward.
Figure 6: The derivation resulting from applying step (6) to the derivation in Fig. 5b. Each rule in the derivation contains exactly one terminal.
Implementation. The induction of uni-lexical LCFRS and parsing was implemented by extending disco-dop (van Cranenburgh et al., 2016), from which we could borrow the generic LCFRS extraction and statistical parsing implementation. Moreover, we used the computation of evaluation scores in disco-dop.
The supertagger was implemented using the flair framework (Akbik et al., 2019). We report results for three types of architectures:
- bert - the output of the four topmost layers of a pretrained BERT $^4$  model (Devlin et al., 2019), which is fine-tuned during training,
- flair – the concatenation of language-specific fasttext (Mikolov et al., 2018) and flair embeddings (Akbik et al., 2018), which is fed through a two layered Bi-LSTM (Hochreiter and Schmidhuber, 1997), and
supervised (small/large) - word embeddings (one-hot embeddings and character-based BiLSTM outputs) are trained with the model, and fed through a two layered Bi-LSTM. The small model adopts its size parameters from Stanojevic and Steedman (2020); Coavoux and Cohen (2019) and the large model from Corro (2020).
On top of each of those, there are two linear layers in parallel: one for the supertags and one for the nonterminals that were removed in step (4) of our lexicalization scheme (i.e. part-of-speech tags plus nonterminals from collapsed monic rules). The sequence tagger is trained to predict the gold supertag and the removed nonterminal for each sentence position via the sum of cross-entropy losses. More
4We used language-specific flavors of bert-base that were available in huggingface's transformers library; bert-base-german-cased for Tiger and Negra, and bert-base-cased for DPTB.
details with respect to hyperparameters for all models are shown in Appendix B.