SlowGuess/ACL_2020_OCR huggingface dataset:
Since BERT is trained on subwords (i.e., substrings of the original token), we take the 768-dimension vector of each subword of an input token and use the average embedding as the final representation  $\mathbf{e}_i^{BERT}$ . Finally, this is directly concatenated to the resulting basic word representation before feeding the BiLSTM-based encoder:
$$
\mathbf {x} _ {i} ^ {\prime} = \mathbf {x} _ {i} \oplus \mathbf {e} _ {i} ^ {B E R T}; \mathbf {h} _ {i} = \mathbf {B i L S T M} (\mathbf {x} _ {i} ^ {\prime})
$$
Higher performances can be achieved by summing or concatenating (depending on the task) several layers of BERT; however, exploring these combinations is out of the scope of this paper and we simply use embeddings extracted from the second-to-last hidden layer (since the last layer is biased to the target objectives used to train BERT's language model).
In order to test the proposed approach, we conduct experiments on the SemEval 2015 Task 18 English datasets (Oepen et al., 2015), where all sentences are annotated with three different formalisms: DELPH-IN MRS (DM) (Flickinger et al., 2012), Predicate-Argument Structure (PAS) (Miyao and Tsujii, 2004) and Prague Semantic Dependencies (PSD) (Hajic et al., 2012). Standard split as in previous work (Almeida and Martins, 2015; Du et al., 2015) results in 33,964 training sentences from Sections 00-19 of the Wall Street Journal corpus (Marcus et al., 1993), 1,692 development sentences from Section 20, 1,410 sentences from Section 21 as in-domain test set, and 1,849 sentences sampled from the Brown Corpus (Francis and Kucera,
Architecture hyper-parameters
CNN window size	3
CNN number of filters	50
BiLSTM encoder layers	3
BiLSTM encoder size	512
LSTM decoder layers	1
LSTM decoder size	512
LSTM layers dropout	0.33
Word/POS/Char./Lemma embedding dimension	100
BERT embedding dimension	768
Embeddings dropout	0.33
MLP layers	1
MLP activation function	ELU
Arc MLP size	512
Label MLP size	128
UNK replacement probability	0.5
Adam optimizer hyper-parameters
Initial learning rate	0.001
β1, β2	0.9
Batch size	32
Decay rate	0.75
Gradient clipping	5.0
Table 2: Model hyper-parameters.
1982) as out-of-domain test data. For the evaluation, we use the official script, $^{2}$  reporting labelled F-measure scores (LF1) (including ROOT arcs) on the in-domain (ID) and out-of-domain (OOD) test sets for each formalism as well as the macro-average over the three of them.
We use the Adam optimizer (Kingma and Ba, 2014) and follow (Ma et al., 2018; Dozat and Manning, 2017) for parameter optimization. We do not specifically perform hyper-parameter selection for SDP and just adopt those proposed by Ma et al. (2018) for syntactic dependency parsing (detailed in Table 2). For initializing word and lemma vectors, we use the pre-trained structured-skipgram embeddings developed by Ling et al. (2015). POS tag and character embeddings are randomly initialized and all embeddings (except the deep contextualized ones) are fine-tuned during training. Due to random initializations, we report average accuracy over 5 repetitions for each experiment. In addition, during a 500-epoch training, the model with the highest labelled F-score on the development set is chosen. Finally, while further beam-size exploration might improve accuracy, we use beam-search decoding with beam size 5 in all experiments.
Table 3 reports the accuracy obtained by state-of-the-art SDP parsers detailed in Section 2 in comparison to our approach. To perform a fair comparison, we group SDP systems in three blocks depending on the embeddings provided to the architecture: (1) just basic pre-trained word and POS tag embeddings, (2) character and pre-trained lemma embeddings augmentation and (3) pre-trained deep contextualized embeddings augmentation. As proved by these results, our approach outperforms all existing transition-based models and the widely-used approach by Dozat and Manning (2018) with or without character and lemma embeddings, and it is on par with the best graph-based SDP parser by (Wang et al., 2019) on average in the fully-supervised scenario. $^3$
In addition, our model achieves the best fully-supervised accuracy to date on the PSD formalism, considered the hardest to parse. We hypothesize that this might be explained by the fact that the PSD formalism is the more tree-oriented (as pointed out by Oepen et al. (2015)) and presents a lower ratio of arcs per sentence, being more suitable for our transition-based approach.
In the semi-supervised scenario, BERT-based embeddings proved to be more beneficial for the out-of-domain data. In fact, while not being a fair comparison since we neither include contextual string embeddings (Flair) (Akbik et al., 2018) nor explore different BERT layer combinations, our new transition-based parser manages to outperform the state-of-the-art system by He and Choi (2019)<sup>4</sup> on average on the out-of-domain test set, obtaining a remarkable accuracy on the PSD formalism.
5.4 Complexity
Given a sentence with length  $n$  whose SDP graph has  $m$  arcs, the proposed transition system requires  $n$  Shift plus  $m$  Attach-  $p$  transitions to parse it. Therefore, since a DAG can have at most  $\Theta (n^2)$  edges (as is also the case for general directed graphs), it could potentially need  $O(n^{2})$  transitions in the worst case. However, we prove that this does not happen in practice and real sentences can be
<sup>3</sup>It is common practice in the literature that systems that only use standard pre-trained word or lemma embeddings are classed as fully-supervised models, even though, strictly, they are not trained exclusively on the official training data.
<sup>4</sup>He and Choi (2019) do not specify in their paper the BERT layer configuration used for generating the word embeddings.
$^{2}$ https://github.com/semantic-dependencyParsing/toolkit
7041
Parser	DM	PAS	PSD	Avg
ID	OOD	ID	OOD	ID	OOD	ID	OOD
Du et al. (2015) TbGb+Ens	89.1	81.8	91.3	87.2	75.7	73.3	85.3	80.8
Almeida and Martins (2015) Gb	88.2	81.8	90.9	86.9	76.4	74.8	85.2	81.2
Peng et al. (2017) Gb	89.4	84.5	92.2	88.3	77.6	75.3	86.4	82.7
Peng et al. (2017) Gb+MT	90.4	85.3	92.7	89.0	78.5	76.4	87.2	83.6
Wang et al. (2018) Tb	89.3	83.2	91.4	87.2	76.1	73.2	85.6	81.2
Wang et al. (2018) Tb+Ens	90.3	84.9	91.7	87.6	78.6	75.9	86.9	82.8
Dozat and Manning (2018) Gb	91.4	86.9	93.9	90.8	79.1	77.5	88.1	85.0
Kurita and Søgaard (2019) Gb	91.1	-	92.4	-	78.6	-	87.4	-
Kurita and Søgaard (2019) Gb+MT+RL	91.2	-	92.9	-	78.8	-	87.6	-
Wang et al. (2019) Gb	93.0	88.4	94.3	91.5	80.9	78.9	89.4	86.3
This work Tb	92.5	87.7	94.2	91.0	81.0	78.7	89.2	85.8
Dozat and Manning (2018) Gb+char+lemma	93.7	88.9	93.9	90.6	81.0	79.4	89.5	86.3
Kurita and Søgaard (2019) Gb+MT+RL+lemma	92.0	87.2	92.8	88.8	79.3	77.7	88.0	84.6
Wang et al. (2019) Gb+char+lemma	94.0	89.7	94.1	91.3	81.4	79.6	89.8	86.9
This work Tb+char+lemma	93.9	89.6	94.2	91.2	81.8	79.8	90.0	86.9
Zhang et al. (2019) Tb+char+BERTLARGE	92.2	87.1	-	-	-	-	-	-
He and Choi (2019) Gb+lemma+Flair+BERTBASE	94.6	90.8	96.1	94.4	86.8	79.5	92.5	88.2
This work Tb+char+lemma+BERTBASE	94.4	91.0	95.1	93.4	82.6	82.0	90.7	88.8
Table 3: Accuracy comparison of state-of-the-art SDP parsers on the SemEval 2015 Task 18 datasets.  $Gb$  and  $Tb$  stand for graph- and transition-based models,  $+char$  and  $+lemma$  for augmentations with character-level and lemma embeddings,  $+Flair$  and  $+BERT_{BASE|LARGE}$  for augmentations with deep contextualized character-level and word-level embeddings, and, finally,  $+MT$ ,  $+RL$  and  $+Ens$  for the application of multi-task, reinforcement learning and ensemble techniques.
parsed with  $O(n)$  transitions instead.
Parsing complexity of a transition-based dependency parsing algorithm can be determined by the number of transitions performed with respect to the number of words in a sentence (Kübler et al., 2009). Therefore, we measure the transition sequence length predicted by the system to analyze every sentence from the development sets of the three available formalisms and depict the relation between them and sentence lengths. As shown in Figure 3, a linear behavior is observed in all cases, proving that the number of Attach-  $p$  transitions evaluated by the model at each step is considerably low (behaving practically like a constant).
This can be explained by the fact that, on average on the training set, the ratio of predicate-argument dependencies per word in a sentence is 0.79 in DM, 0.99 in PAS and 0.70 in PSD, meaning that the transition sequence necessary for parsing a given sentence will need no more Attach- $p$  transitions than Shift ones (which are one per word in the sentence). It is true that one argument can be attached to more than one predicate; however, the amount of words unattached in the resulting DAG (single
tons) $^5$  can be significant in some formalisms (as described graphically in Figure 1): on average on the training set,  $23\%$  of words per sentence in DM,  $6\%$  in PAS and  $35\%$  in PSD. In addition, edge density on non-singleton words, computed by Oepen et al. (2015) on the test sets, also backs the linear behavior shown in our experiments: 0.96 in DM, 1.02 in PAS and 1.01 in PSD for the in-domain set and 0.95 in DM, 1.02 in PAS and 0.99 in PSD for the out-of-domain data. In conclusion, we can state that, on the datasets tested, the proposed transition system executes  $O(n)$  transitions.
To determine the runtime complexity of the implementation of the transition system, we need to consider the following: firstly, at each transition, the attention vector  $\mathbf{a}^t$  needs to be computed, which means that each of the  $O(n)$  transitions takes  $O(n)$  time to run. Therefore, the overall time complexity of the parser, ignoring cycle detection, is  $O(n^2)$ . Note that this is in contrast to algorithms like (Wang et al., 2019), which takes cubic time even though it does not enforce acyclicity.
A singleton is a word that has neither incoming nor outgoing edges.
7042
Figure 3: Number of predicted transitions relative to the length of the sentence, for the three SDP formalisms on the development set from SemEval 2015 Task 18.
If we add cycle detection, needed to forbid transitions that would create cycles and therefore to enforce that the output is a DAG, then the complexity becomes  $O(n^{2} \log n)$ . This is because an efficient implementation of cycle detection contributes an additive factor of  $O(n^{2} \log n)$  to worst-case time complexity, which becomes the dominant factor. To achieve this efficient implementation, we incrementally keep two data structures: on the one hand, we keep track of weakly connected components using path compression and union by rank, which can be done in inverse Ackermann time, as is com
monly done for cycle detection in tree and forest parsers (Covington, 2001; Gómez-Rodríguez and Nivre, 2010). On the other hand, we keep a weak topological numbering of the graph using the algorithm by Bender et al. (2015), which takes overall  $O(n^{2} \log n)$  time over all edge insertions. When these two data structures are kept, cycles can be checked in constant time: an arc  $a \to b$  creates a cycle if the involved nodes are in the same weakly connected component and  $a$  has a greater topological number than  $b$ .
Therefore, the overall expected worst-case running time of the proposed SDP system is  $O(n^{2} \log n)$  for the range of data attested in the experiments, and can be lowered to  $O(n^{2})$  if we are willing to forgo enforcing acyclicity.
Our multi-head transition system can accurately parse a sentence in quadratic worst-case runtime thanks to Pointer Networks. While being more efficient, our approach outperforms the previous state-of-the-art parser by Dozat and Manning (2018) and matches the accuracy of the best model to date (Wang et al., 2019), proving that, with a state-of-the-art neural architecture, transition-based SDP parsers are a competitive alternative.
By adding BERT-based embeddings, we significantly improve our model accuracy by marginally affecting computational cost, achieving state-of-the-art F-scores in out-of-domain test sets.
Despite the promising results, the accuracy of our approach could probably be boosted further by experimenting with new feature information and specifically tuning hyper-parameters for the SDP task, as well as using different enhancements such as implementing the hierarchical decoding recently presented by Liu et al. (2019), including contextual string embeddings (Akbik et al., 2018) like He and Choi (2019), or applying multi-task learning across the three formalisms like Peng et al. (2017).
7043
Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228-238, Sofia, Bulgaria. Association for Computational Linguistics.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Mariana S. C. Almeida and André F. T. Martins. 2015. Lisbon: Evaluating TurboSemanticParser on multiple languages and out-of-domain data. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 970-973, Denver, Colorado. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Hongxiao Bai and Hai Zhao. 2019. SJTU at MRP 2019: A transition-based multi-task parser for cross-framework meaning representation parsing. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 86-94, Hong Kong. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Michael A. Bender, Jeremy T. Fineman, Seth Gilbert, and Robert E. Tarjan. 2015. A new approach to incremental cycle detection and related problems. ACM Trans. Algorithms, 12(2):14:1-14:22.
Wanxiang Che, Longxu Dou, Yang Xu, Yuxuan Wang, Yijia Liu, and Ting Liu. 2019. HIT-SCIR at MRP 2019: A unified pipeline for meaning representation parsing via efficient training and effective encoding. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 76–85, Hong Kong. Association for Computational Linguistics.
Jinho D. Choi and Andrew McCallum. 2013. Transition-based dependency parsing with selectional branching. In Proceedings of the 51st
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1052-1062, Sofia, Bulgaria. Association for Computational Linguistics.
Y. J. Chu and T. H. Liu. 1965. On the shortest arborescence of a directed graph. Science Sinica, 14:1396-1400.
Michael A. Covington. 2001. A fundamental algorithm for dependency parsing. In Proceedings of the 39th Annual ACM Southeast Conference, pages 95-102.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep bioaffine attention for neural dependency parsing. In ICLR. OpenReview.net.
Timothy Dozat and Christopher D. Manning. 2018. Simpler but more accurate semantic dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 484-490, Melbourne, Australia. Association for Computational Linguistics.
Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. Stanford's graph-based neural dependency parser at the CoNLL 2017 shared task. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 20-30, Vancouver, Canada. Association for Computational Linguistics.
Yantao Du, Fan Zhang, Xun Zhang, Weiwei Sun, and Xiaojun Wan. 2015. Peking: Building semantic dependency graphs with a hybrid parser. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 927-931, Denver, Colorado. Association for Computational Linguistics.
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 334-343, Beijing, China. Association for Computational Linguistics.
Jack Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards, 71B:233-240.
7044
Daniel Fernández-González and Carlos Gómez-Rodríguez. 2019. Faster shift-reduce constituent parsing with a non-binary, bottom-up strategy. Artificial Intelligence, 275:559 - 574.
Daniel Fernández-González and Carlos Gómez-Rodríguez. 2019. Left-to-right dependency parsing with pointer networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Volume 1 (Long and Short Papers), pages 710-716, Minneapolis, Minnesota. Association for Computational Linguistics.
Dan Flickinger, Yi Zhang, and Valia Kordoni. 2012. Deepbank: a dynamically annotated treebank of the wall street journal. In Proceedings of the Eleventh International Workshop on Treebanks and Linguistic Theories (TLT11), pages 85-96, Lisbon. HU.
W.N. Francis and H. Kucera. 1982. Frequency Analysis of English Usage: Lexicon and Usage. Houghton Mifflin.
Carlos Gómez-Rodríguez and Joakim Nivre. 2010. A transition-based parser for 2-planar dependency structures. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1492-1501, Uppsala, Sweden. Association for Computational Linguistics.
Jan Hajic, Eva Hajicova, Jarmila Paneová, Petr Sgall, Ondřej Bojar, Silvie Cinková, Eva Fučíková, Marie Mikulová, Petr Pajas, Jan Popelka, Jiří Semecký, Jana Šindlerová, Jan Štepanek, Josef Toman, Zdenka Urešová, and Zdeněk Žabokrtský. 2012. Announcing Prague Czech-English dependency treebank 2.0. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3153-3160, Istanbul, Turkey. European Language Resources Association (ELRA).
Han He and Jinho D. Choi. 2019. Establishing strong baselines for the new decade: Sequence tagging, syntactic and semantic parsing with bert. *ArXiv*, abs/1908.04943.
Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2017. A transition-based directed acyclic graph parser for UCCA. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1127-1138, Vancouver, Canada. Association for Computational Linguistics.
Daniel Hershcovich and Ofir Arviv. 2019. TUPA at MRP 2019: A multi-task baseline system. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 28-39, Hong Kong. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Published as a
conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Sandra Kübler, Ryan T. McDonald, and Joakim Nivre. 2009. Dependency Parsing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
Shuhei Kurita and Anders Søgaard. 2019. Multi-task semantic dependency parsing with policy gradient for learning easy-first strategies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2420-2430, Florence, Italy. Association for Computational Linguistics.
Sunny Lai, Chun Hei Lo, Kwong Sak Leung, and Yee Leung. 2019. CUHK at MRP 2019: Transition-based parser with cross-framework variable-arity resolve action. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 104-113, Hong Kong. Association for Computational Linguistics.
Wang Ling, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. Two/too simple adaptations of Word2Vec for syntax problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1299-1304, Denver, Colorado. Association for Computational Linguistics.
Linlin Liu, Xiang Lin, Shafiq Joty, Simeng Han, and Li-dong Bing. 2019. Hierarchical pointer net parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074. Association for Computational Linguistics.
Xuezhe Ma, Zeong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard H. Hovy. 2018. Stackpointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1403-1414.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313-330.
André Martins, Miguel Almeida, and Noah A. Smith. 2013. Turning on the turbo: Fast third-order nonprojective turbo parsers. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 617-622, Sofia, Bulgaria. Association for Computational Linguistics.
7045