SlowGuess/NAACL_2021_OCR huggingface dataset:
During parsing, the predicted supertags are interpreted as a probabilistic grammar. At each sentence position, the weight of the rules is the softmax of the supertag's score among the  $k$  best scores. The parsing implementation that we borrow from disco-dop supports heuristics and early-stopping to speed-up the parsing process. For each intermediate parse that does not span all sentence positions, we use the best supertag probability for each position that does not belong to the parse as a heuristic to estimate the weight of a complete parse.
We extended the parser with a fallback mechanism that deals with parse fails, i.e. when it is not able to find parse trees for the whole sentence. It picks the largest partial derivations (for parts of the sentence) that it was able to find and combines them as children of artificial NOPARSE nodes. This is especially beneficial in settings with small  $k$  as there are many parse fails (cf. Table 1 column cov.). For example, if we did not use this mechanism, we would obtain prec. = 95.53, rec. = 46.21 and F1 = 62.29 for the development set of Negra and  $k = 1$  (cf. first row in Table 1).
We use only the highest-scoring nonterminal predicted for the reversal of step (4).
Data. Following Coavoux and Cohen (2019), we use three treebanks for discontinuous constituent parsing in our evaluations: Negra (Skut et al., 1998), Tiger (Brants et al., 2004), and a discontinuous version of the Penn treebank (DPTB; Evang and Kallmeyer, 2011). The treebanks were split according to the usual standards into training, development and test sets. During development, the lexicalization, tagging and parsing were mostly tested and optimized using Negra. We binarized each training set before extracting the LCFRS and supertags. Markovization with horizontal context  $\mathrm{h} = 0$  and vertical context  $\mathrm{v} = 1$  has yielded the best results; we thus extracted 3275 supertags from the training set of Negra, 4614 from Tiger and 4509 from DPTB. More context in Markovization lead to a blowup in the number of supertags which proved to be disadvantageous.
Baselines. We report labeled F1-scores, obtained from predicted and gold parse trees us
ing disco-dop (with the usual parameters in proper.prm), for all constituents  $(F1)$  and all discontinuous constituents  $(Dis - F1)$ . Additionally to the scores, parse speed $^6$  is reported in sentences per second  $(sent / s)$ .
Our scores are compared to recent state-of-the-art parsers for discontinuous constituent trees in four categories:
- grammar-based parsers (van Cranenburg et al., 2016; Gebhardt, 2020; Versley, 2016) – that directly rely on an underlying (probabilistic) grammar,
- chart-based parsers (Corro, 2020; Stanojevic and Steedman, 2020) – that share parsing algorithms with LCFRS, but lack an explicit set of rules,
transition systems (Coavoux and Cohen, 2019; Coavoux et al., 2019), and
- neural systems (Fernández-González and Gómez-Rodríguez, 2020; Vilares and Gómez-Rodríguez, 2020) – all other recent parsing approaches using neural classifiers.
Our approach is in the first category, as the supertags are clearly constructed from a grammar that was extracted from the treebank. Therefore, the local relations in the predicted derivations are restricted to those occurring in the treebank. The approaches by Corro (2020) and Stanojevic and Steedman (2020), on the other hand, rank spans in the sentence for occurrence in the predicted parse tree and predict their nonterminal; both independently from previous spans and nonterminals. Hence, they allow any combination of parent/child nonterminals in the resulting derivations.
Table 1 shows statistics of our parser on the development sets for different amounts  $(k)$  of supertags per sentence position. Specifically, we report the parsing speed (sent/s), the rate of sentence positions where the gold supertag was among the  $k$  predicted supertags (tag accuracy), the rate of sentences that was completely parsed (coverage) and parsing scores (labeled precision, recall and F1).
We see that the parsing speed gradually drops with rising  $k$ , but for  $k > 10$  there are barely any gains in terms of parsing scores. As expected, with rising  $k$ , the recall increases drastically. The preci
<sup>5</sup>We use the split for Negra by Dubey and Keller (2003), for Tiger by Seddah et al. (2013), and the standard split for DPTB (sections 2–21 for training, 22 for development, 23 for testing).
<sup>6</sup>We measured the parsing speed on a system with an Nvidia GeForce RTX 2080, two Intel Xeon Silver 4114 (20 cores/40 threads at 2.2 GHz) and 256 GB RAM.
Table 1: Results for different values for  $k$ , i.e. how many supertags for each sentence position are used for parsing, on development sets after training. Includes only the results for our bert model.
k	Negra
sent/s	tag acc.	cov.	prec.	rec.	F1
1	78	87.37	64.60	93.10	57.16	70.83
2	75	93.35	85.80	91.27	78.73	84.54
3	71	94.88	92.10	91.22	84.87	87.93
5	69	96.23	96.80	91.15	89.32	90.23
10	63	97.50	99.20	91.30	91.00	91.15
15	58	98.19	99.80	91.37	91.27	91.32
20	53	98.52	99.90	91.56	91.44	91.50
k	Tiger
1	75	89.38	73.26	94.23	67.14	78.42
2	72	94.96	92.00	92.65	85.18	88.76
3	67	96.33	96.14	92.53	88.83	90.65
5	66	97.45	98.32	92.66	90.79	91.71
10	61	98.48	99.67	92.74	91.79	92.27
15	56	98.83	99.88	92.73	91.88	92.30
20	53	99.04	99.98	92.80	91.97	92.38
k	DPTB
1	95	90.00	58.88	91.95	59.84	72.50
2	90	95.92	90.06	93.89	87.21	90.43
3	82	97.21	95.35	93.92	91.41	92.65
5	79	98.24	97.88	94.07	93.05	93.56
10	72	98.98	99.29	94.03	93.81	93.92
15	66	99.23	99.82	94.09	94.01	94.05
20	61	99.38	99.94	94.11	94.05	94.08
Table 2: Our results compared to other published supertaggers.  $(\dagger)$  Bladier et al. (2018) used a slightly different split of Tiger.  $(\ddagger)$  Tian et al. (2020) use CCGbank instead of DPTB, which is a digest of PTB specifically for CCG parsing. Recent publications for CCG parsing use a slightly different split (sec. 0 instead of sec. 22 for development) for CCGbank than we do for DPTB.
Model	Tiger	DPTB‡
tags	tag acc.	tags	tag acc.
Bladier et al., 2018	3426	88.51†	-	-
Kasai et al., 2017	-	-	4727	89.71
ours (sup., small)	4614	74.35	4509	83.06
ours (sup., large)	4614	78.96	4509	86.73
ours (flair)	4614	81.50	4509	88.55
ours (bert)	4614	85.40	4509	90.14
Tian et al., 2020	-	-	1284	96.39
sion, on the other hand, only changes slightly. The drop in precision using Negra and Tiger may be explained by a significant decrease in parse fails from  $k = 1$  to  $k = 2$ , then the effects of fewer parse fails and considering lower-scored supertags seem to balance each other out. We found  $k = 10$  to be a good parameter for the rest of our experiments.
Table 3 shows the parsing scores and speed of our trained models on the test set compared to the scores reported in other recent publications on discontinuous constituent parsing. The experiments suggest that parsing using LCFRS can greatly benefit from supertagging with respect to both speed and accuracy. This, however, requires a strong discriminative classifier for the sequence tagger to predict useful rules. Most notably, the prediction accuracy for discontinuous constituents seems to strongly benefit from pretrained word embeddings.
Compared to other parsing approaches, we obtain results that are on par with the state of the art; recently, this is rather unusual for grammar-based constituent parsing. We would like to especially highlight our results for discontinuous constituents, which surpass the previous state of the art by a wide margin.
Unfortunately, we can only compare our results to those of other supertagging-based parsers to a very limited extent, as authors seem to either report no parsing scores at all (Bladier et al., 2018), or give attachment scores for dependency relations (Kasai et al., 2017; Tian et al., 2020). However, Table 2 compares the accuracy of our supertagger to some recent publications. The CCG community is very active in the field of neural supertagging, achieving an improvement from  $91.3\%$  (Lewis and Steedman, 2014) to  $96.4\%$  accuracy (Tian et al., 2020) for predicted supertags in the last 6 years. We can not compete with those numbers, but this may be due to the fact that there are far fewer supertags trained in these approaches than in ours. In the case of TAG, the supertagger by Bladier et al. (2018) achieves a better accuracy than ours. But again, there are fewer tags to predict. Compared to Kasai et al. (2017), our models with pretrained embeddings seem to be on par in both the number of tags and the accuracy.
We described an approach to utilize supertagging for parsing discontinuous constituents with LCFRS and evaluated it. Compared to other parsers for the
Table 3: Our results on test sets compared to other published constituent parsers.  $(\dagger)$  Van Cranenburg et al. (2016) use a different split for Tiger.  $(\ddagger)$  Parsing speeds marked with this symbol were measured on our system as processing times were not reported by the authors.
Model	F1	Negra Dis-F1	sent/s	F1	Tiger Dis-F1	sent/s	F1	DPTB Dis-F1	sent/s
Grammar-based systems
van Cranenburg et al., 2016	76.8	-	2‡	78.2†	-	1‡	87.0	-	<1‡
Gebhardt, 2020	81.7	43.5	-	77.7	40.7	-	-	-	-
ours (supervised, small)	77.59	28.28	136	78.38	44.73	103	87.24	64.84	103
ours (supervised, large)	82.72	49.03	136	82.53	55.91	101	90.08	72.87	95
ours (flair)	86.54	61.89	104	85.12	61.00	80	91.77	76.14	86
ours (bert)	90.94	72.58	68	88.34	69.02	60	93.32	80.53	57
Versley, 2016	-	-	-	79.50	-	-	-	-	-
Chart-based systems
Corro, 2020 (supervised)	86.3	56.1	478	85.2	51.2	474	92.9	64.9	355
Corro, 2020 (bert)	91.6	66.1	-	90.0	62.1	-	94.8	68.9	-
Stanojević and Steedman, 2020	83.3	50.7	15‡	83.4	53.5	9‡	90.5	67.7	-
Transition systems
Coavoux and Cohen, 2019	84.0	54.0	-	87.6	52.5	64	91.4	70.9	38
Coavoux et al., 2019	83.2	54.6	-	82.7	55.9	126	91.0	71.3	80
Neural systems
Fernández-G. and Gómez-R., 2020	86.1	59.9	12‡	86.3	60.7	11‡	-	-	-
Vilares and Gómez-R., 2020 (bilstm)	77.1	36.5	715	79.2	40.1	568	89.1	41.8	611
Vilares and Gómez-R., 2020 (bert)	84.2	46.9	81	84.7	51.6	80	91.7	49.1	80
same grammar formalism, we achieve state-of-the-art results, i.e. we are more accurate and also faster (cf. Table 3, Grammar-based systems). In contrast to previous parsers utilizing LCFRS, we can even keep up with other (neural) parsing approaches and establish a new state of the art for discontinuous constituents (cf. Table 3, columns for Dis-F1).
Recent publications by Corro (2020) and Stanojevic and Steedman (2020) address discontinuous constituent parsing using approaches that share an algorithmic foundation with LCFRS parsing, but do not use an underlying grammar. Both of them restrict constituents to two non-contiguous spans (equivalent to an LCFRS with fanout 2), we have no such limitation. Considering the margin between our discontinuous F1-score and theirs, we suppose that this restriction is only benefiting the complexity, not the accuracy.
Future Work. Compared to previous approaches for supertagging, we utilize large sets of supertags. We are confident that the accuracy of the supertagger can be improved by appropriately reducing these sets. The approach how terminals are transported in derivations during step (5) of the extraction is quite technical and chosen such that there is no impact on the fanout of the grammar (Mörbitz and Ruprecht, 2020). Alternative techniques
could conceivably result in smaller sets of supertags and/or improve parsing results.
To validate the benefit of LCFRS (compared to using TAG or CCG) for supertagging-based approaches to constituent paring, we aim for an in-depth comparison of our work to previous approaches. However, currently, these approaches lack of publicly available implementations for constituent parsing.
We thank Alex Ivliev for conducting early experiments during the development of our parser, and our colleague Kilian Gebhardt as well as the anonymous reviewers for their insightful comments on drafts of this paper.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence
labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Krasimir Angelov and Peter Ljunglöf. 2014. Fast statistical parsing with parallel multiple context-free grammars. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 368-376, Gothenburg, Sweden. Association for Computational Linguistics.
Srinivas Bangalore and Aravind K. Joshi. 1999. Supertagging: An approach to almost parsing. Computational linguistics, 25(2):237-265.
Tatiana Bladier, Andreas van Cranenburg, Younes Samih, and Laura Kallmeyer. 2018. German and French neural supertagging experiments for LTAG parsing. In Proceedings of ACL 2018, Student Research Workshop, pages 59-66, Melbourne, Australia. Association for Computational Linguistics.
Sabine Brants, Stefanie Dipper, Peter Eisenberg, Silvia Hansen-Schirra, Esther König, Wolfgang Lezius, Christian Rohrer, George Smith, and Hans Uszkoreit. 2004. TIGER: Linguistic interpretation of a German corpus. Research on language and computation, 2(4):597-620.
Stephen Clark. 2002. Supertagging for combinatory categorial grammar. In Proceedings of the Sixth International Workshop on Tree Adjoining Grammar and Related Frameworks (TAG+6), pages 19-24.
Maximin Coavoux and Shay B. Cohen. 2019. Discontinuous constituency parsing with a stack-free transition system and a dynamic oracle. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 204-217, Minneapolis, Minnesota. Association for Computational Linguistics.
Maximin Coavoux, Benoit Crabble, and Shay B. Cohen. 2019. Unlexicalized transition-based discontinuous constituency parsing. Transactions of the Association for Computational Linguistics, 7:73-89.
Caio Corro. 2020. Span-based discontinuous constituency parsing: a family of exact chart-based algorithms with time complexities from  $\mathrm{O}(\mathrm{n}^{\wedge}6)$  down to  $\mathrm{O}(\mathrm{n}^{\wedge}3)$ . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2753-2764, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),
pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Amit Dubey and Frank Keller. 2003. Probabilistic parsing for German using sister-head dependencies. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 96-103, Sapporo, Japan. Association for Computational Linguistics.
Joost Engelfriet, Andreas Maletti, and Sebastian Maneth. 2018. Multiple context-free tree grammars: Lexicalization and characterization. Theoretical Computer Science, 728:29-99.
Kilian Evang and Laura Kallmeyer. 2011. PLCFRS parsing of English discontinuous constituents. In Proceedings of the 12th International Conference on Parsing Technologies, pages 104-116, Dublin, Ireland. Association for Computational Linguistics.
Daniel Fernández-González and Carlos Gómez-Rodríguez. 2020. Discontinuous constituent parsing with pointer networks. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence.
Kilian Gebhardt. 2020. Advances in using grammars with latent annotations for discontinuous parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 91-97, Online. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Aravind K. Joshi, Leon S. Levy, and Masako Takahashi. 1975. Tree adjunct grammars. Journal of Computer and System Sciences, 10(1):136-163.
Rekia Kadari, Yu Zhang, Weinan Zhang, and Ting Liu. 2018. CCG supertagging via bidirectional LSTM-CRF neural architecture. Neurocomputing, 283:31-37.
Laura Kallmeyer and Wolfgang Maier. 2013. Data-driven parsing using probabilistic linear context-free rewriting systems. Computational Linguistics, 39(1):87-119.
Jungo Kasai, Bob Frank, Tom McCoy, Owen Rambow, and Alexis Nasr. 2017. TAG parsing with neural networks and vector representations of supertags. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1712-1722, Copenhagen, Denmark. Association for Computational Linguistics.
Mike Lewis and Mark Steedman. 2014. Improved CCG parsing with semi-supervised supertagging. Transactions of the Association for Computational Linguistics, 2:327-338.
Wolfgang Maier and Anders Søgaard. 2008. Treebanks and mild context-sensitivity. In Proceedings of the 13th conference on Formal Grammar, pages 61-76, Hamburg, Germany. CSLI Publications.
Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pages 114-119, Plainsboro, New Jersey. Association for Computational Linguistics.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Richard Moritz and Thomas Ruprecht. 2020. Lexicalization of probabilistic linear context-free rewriting systems. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 98-104, Online. Association for Computational Linguistics.
Djame Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito,inho D. Choi, Richard Farkas, Jennifer Foster, lakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Wolinski, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146-182, Seattle, Washington, USA. Association for Computational Linguistics.
Hiroyuki Seki, Takashi Matsumura, Mamoru Fujii, and Tadao Kasami. 1991. On multiple context-free grammars. Theoretical Computer Science, 88(2):191-229.
Wojciech Skut, Thorsten Brants, Brigitte Krenn, and Hans Uszkoreit. 1998. A linguistically interpreted corpus of German newspaper text. In Proceedings of the ESSLLI Workshop on Recent Advances in Corpus Annotation, Saarbrücken, Germany.
Miloš Stanojevic and Mark Steedman. 2020. Span-based LCFRS-2 parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 111–121, Online. Association for Computational Linguistics.
Yuanhe Tian, Yan Song, and Fei Xia. 2020. Supertagging Combinatory Categorical Grammar with
attentive graph convolutional networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6037-6044, Online. Association for Computational Linguistics.
Andreas van Cranenburg. 2012. Efficient parsing with linear context-free rewriting systems. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 460-470, Avignon, France. Association for Computational Linguistics.
Andreas van Cranenburgh, Remko Scha, and Rens Bod. 2016. Data-oriented parsing with discontinuous constituents and function tags. Journal of Language Modelling, 4(1):57-111.
Ashish Vaswani, Yonatan Bisk, Kenji Sagae, and Ryan Musa. 2016. Supertagging with LSTMs. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 232-237, San Diego, California. Association for Computational Linguistics.
Yannick Versley. 2016. Discontinuity (re) $^{2}$ -visited: A minimalist approach to pseudoprojective constituent parsing. In Proceedings of the Workshop on Discontinuous Structures in Natural Language Processing, pages 58-69, San Diego, California. Association for Computational Linguistics.
Krishnamurti Vijay-Shanker, David Jeremy Weir, and Aravind K. Joshi. 1987. Characterizing structural descriptions produced by various grammatical formalisms. In Proceedings of the 25th Annual Meeting on Association for Computational Linguistics, ACL '87, pages 104-111, Stroudsburg, PA, USA. Association for Computational Linguistics.
David Vilares and Carlos Gomez-Rodriguez. 2020. Discontinuous constituent parsing as sequence labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2771-2785, Online. Association for Computational Linguistics.
A Unlexicalizing Derivations
In this Appendix we formally describe the reversal of selected steps of our lexicalization scheme (cf. Section 4). In each instance we assume a derivation like it would be obtained right after applying the corresponding step.
A.1 Reversal of step (5).
This step is applied to each occurrence  $r$  of branching rules of the form  $A \rightarrow c(A_1, A_2)$  from the bottom to the top (i.e. it was already done for branching rules in the subtrees below a node before it is applied to the node itself). Let  $t$  be the leftmost occurrence of a terminating rule that is reachable from the second child of  $r$ . At each node  $s$  on the path from  $r$  to  $t$  (i.e. from top down) we perform three steps. (1) We transform the composition back into the original composition, (2) we remove all annotation  $(^+, -, \text{and swapped})$ , and (3) we pass a terminal to the child (if  $s$  is not  $t$ ).
Obtaining the original composition. The composition at each node  $s$  is transformed back into the original composition depending on the type of the rule. We note that if  $s$  is a branching rule and  $s \neq r$ , the composition at  $s$  was already changed previously in this step and we leave it as it is.
Branching rule Let  $B \to (u_1, \ldots, u_s)(B_1, B_2)$  be the rule at  $s$  and  $\sigma$  be the terminal in  $(u_1, \ldots, u_s)$ .
- If  $B_{2}$  is annotated with  $^{-}$  (i.e., its first component was removed during step (5)), we replace  $\sigma$  with  $y_{0}$ , and replace every occurrence of  $y_{i}$  by  $y_{i + 1}$ .
- Otherwise,  $\sigma$  is removed from  $(u_{1},\ldots ,u_{s})$
Moreover, if  $s$  occurs as a successor of the right child of some other branching rule, then the nonterminals  $B$  and  $B_{1}$  have annotation as well.
- If  $B_{1}$  and  $B$  are annotated with  $^{-}$ , then we replace  $(u_{1},\ldots ,u_{s})$  by  $(\mathrm{x}_0,u_1,\dots ,u_s)$ .
- If  $B_{1}$  is annotated with  $^{-}$  and  $B$  with  $^{+}$ , then we replace  $(u_{1},\ldots ,u_{s})$  by  $(\mathbf{x}_0u_1,\dots ,u_s)$ .
If  $B_{1}$  is annotated with  $\bar{\mathbf{\mu}}^-$ , we replace every occurrence of  $\mathbf{x}_i$  by  $\mathbf{x}_{i + 1}$  afterwards.
Monic rule Let  $B \to c'(B_1)$  be the rule at  $s$  with  $c'$  of the form  $(u_1, \ldots, u_s)$ ,  $\sigma_1$  be the terminal received from the parent, and  $\sigma_2$  be the terminal in  $c'$ .