Venue: NAACL 2019

# Viable Dependency Parsing as Sequence Labeling

Michalina Strzyz  David Vilares Carlos Gomez-Rodriguez

Universidade da Coruna, CITIC

FASTPARSE Lab, Lys Research Group, Departamento de Computacion

Campus de Elvina, s/n, 15071 A Coruna, Spain

{michalina.strzyz,david.vilares,carlos.gomez}@udc.es

###### Abstract

We recast dependency parsing as a sequence labeling problem, exploring several encodings of dependency trees as labels. While dependency parsing by means of sequence labeling had been attempted in existing work, results suggested that the technique was impractical. We show instead that with a conventional bilst-based model it is possible to obtain fast and accurate parsers. These parsers are conceptually simple, not needing traditional parsing algorithms or auxiliary structures. However, experiments on the PTB and a sample of UD treebanks show that they provide a good speed-accuracy tradeoff, with results competitive with more complex approaches.

## 1 Introduction

The application of neural architectures to syntactic parsing, and especially the ability of long short-term memories (LSTMs) to obtain context-aware feature representations Hochreiter and Schmidhuber (1997), has made it possible to parse natural language with conceptually simpler models than before. For example, in dependency parsing, the rich feature models with dozens of features used in transition-based approaches Zhang and Nivre (2011) can be simplified when using feedforward neural networks Chen and Manning (2014), and even more with BiLSTM architectures Kiperwasser and Goldberg (2016), where in fact two positional features can suffice Shi et al. (2017). Similarly, in graph-based approaches, Dozat and Manning (2017) have shown that an arc-factored model can achieve state-of-the-art accuracy, without the need for the higher-order features used in systems like Koo and Collins (2010).

In the same way, neural feature representations have made it possible to relax the need for structured representations. This is the case of sequence-to-sequence models that translate sentences into linearized trees, which were first applied to constituent Vinyals et al. (2015) and later to dependency parsing Wiseman and Rush (2016); Zhang et al. (2017); Li et al. (2018). Recently, Gomez-Rodriguez and Vilares (2018) have shown that sequence labeling models, where each word is associated with a label (thus simpler than sequence to sequence, where the mapping from input to output is not one to one) can learn constituent parsing.

ContributionWe show that sequence labeling is useful for dependency parsing, in contrast to previous work Spoustova and Spousta (2010); Li et al. (2018). We explore four different encodings to represent dependency trees for a sentence of length \(n\) as a set of \(n\) labels associated with its words. We then use these representations to perform dependency parsing with an off-the-shelf sequence labeling model. The results show that we produce models with an excellent speed-accuracy tradeoff, without requiring any explicit parsing algorithm or auxiliary structure (e.g. stack or buffer). The source code is available at [https://github.com/mnstrise/dep2label](https://github.com/mnstrise/dep2label)

## 2 Parsing as sequence labeling

Sequence labeling is a structured prediction problem where a single output label is generated for every input token. This is the case of tasks such as PoS tagging, chunking or named-entity recognition, for which different approaches obtain accurate results Brill (1995); Ramshaw and Marcus (1999); Reimers and Gurevych (2017).

On the contrary, previous work on dependency parsing as sequence labeling is vague and reports results that are significantly lower than those provided by transition-, graph-based or sequence-to-sequence models Dyer et al. (2015); Kiperwasser and Goldberg (2016); Dozat and Manning (2017); Zhang et al. (2017). Spoustova and Spousta

(2010) encoded dependency trees using a relative PoS-based scheme to represent the head of a node, to then train an averaged perceptron. They did not provide comparable results, but claimed that the accuracy was between 5-10% below the state of the art in the pre-deep learning era. Recently, Li et al. (2018) used a relative positional encoding of head indexes with respect to the target token. This is used to train Bidirectional LSTM-CRF sequence-to-sequence models Huang et al. (2015), that make use of sub-root decomposition. They compared their performance against an equivalent BiLSTM-CRF labeling model. The reported UAS for the sequence labeling model was 87.6% on the Penn Treebank, more than 8 points below the current best model Ma et al. (2018), concluding that sequence-to-sequence models are required to obtain competitive results.

In this work, we show that these results can be clearly improved if simpler architectures are used.

## 3 Encoding of trees and labels

Given a sentence \(w_{1}\ldots w_{n}\), we associate the words with nodes \(\{0,1,\dots,n\}\), where the extra node \(0\) is used as a dummy root for the sentence. A dependency parser will find a set of labeled relations encoded as edges of the form \((h,d,l)\), where \(h\in\{0,1,\dots,n\}\) is the head, \(d\in\{1,\dots,n\}\) the dependent, and \(l\) a dependency label. The resulting dependency graph must be acyclic and such that each node in \(\{1,\dots,n\}\) has exactly one head, so it will be a directed tree rooted at node \(0\).

Thus, to encode a dependency tree, it suffices to encode the unique head position and dependency label associated with each word of \(w_{1}\ldots w_{n}\). To do so, we will give each word \(w_{i}\) a discrete label of the form \((x_{i},l_{i})\), where \(l_{i}\) is the dependency label and \(x_{i}\) encodes the position of the head in one of the following four ways (see also Figure 1):

1. Naive positional encoding: \(x_{i}\) directly stores the position of the head, i.e., a label \((x_{i},l_{i})\) encodes an edge \((x_{i},i,l_{i})\). This is the encoding used in the CoNLL file format.
2. Relative positional encoding: \(x_{i}\) stores the difference between the head index minus that of the dependent, i.e., \((x_{i},l_{i})\) encodes an edge \((i+x_{i},i,l_{i})\). This was the encoding used for the sequence-to-sequence and sequence labeling models in Li et al. (2018), as well as for the sequence-to-sequence model in Kiperawaser and Ballesteros (2018).
3. Relative PoS-based encoding: \(x_{i}\) is a tuple \(p_{i},o_{i}\). If \(o_{i}>0\), the head of \(w_{i}\) is the \(o_{i}\)th closest among the words to the right of \(w_{i}\) that have PoS tag \(p_{i}\). If \(o_{i}<0\), the head of \(w_{i}\) is the \(-o_{i}\)th closest among the words to the left of \(w_{i}\) that have PoS tag \(p_{i}\). For example, \((V,-2)\) means "the second verb to the left" of \(w_{i}\). This scheme is closer to the notion of valency, and was used by Spoustova and Spousta (2010).
4. Bracketing-based encoding: based on YliJyra (2012); Yli-Jyra and Gomez-Rodriguez (2017). In each label \((x_{i},l_{i})\), the component \(x_{i}\) is a string following the regular expression (<)? ((\(\backslash\)*| (/)*) (>) ? where the presence of character < means that \(w_{i-1}\) has an incoming arc from the right, \(k\) copies of character \(\setminus\) mean that \(w_{i}\) has \(k\) outgoing arcs towards the left, \(k\) copies of / mean that \(w_{i-1}\) has \(k\) outgoing arcs towards the right, and the presence of > means that \(w_{i}\) has an incoming arc from the left. Thus, each right dependency from a word \(i\) to \(j\) is encoded by a \((/,>)\) pair in the label components \(x_{i+1}\) and \(x_{j}\), and each left dependency from \(j\) to \(i\) by a \((<,\setminus)\) pair in the label components \(x_{i+1}\) and \(x_{j}\). Note that the intuition that explains why information related to a word is encoded in a neighboring node is that each \(x_{i}\) corresponds to a fencepost position (i.e., \(x_{i}\) represents the space between \(w_{i-1}\) and \(w_{i}\)), and the character pair associated to an arc is encoded in the most external fencepost positions covered by that arc. These pairs act as pairs of matching brackets, which can be decoded using a stack to reconstruct the dependencies.

The first three encodings can represent any dependency tree, as they encode any valid head position for each node, while the bracketing encoding only supports projective trees, as it assumes that brackets are properly nested. All the encodings are total and injective, but they are not surjective: head indexes can be out of range in the first three encodings, brackets can be unbalanced in encoding 4, and all the encodings can generate graphs with cycles. We will deal with ill-formed trees later.

## 4 Model

We use a standard encoder-decoder network, to show that dependency parsing as sequence label 

ing works without the need of complex models.

EncoderWe use bidirectional LSTMs Hochreiter and Schmidhuber (1997); Schuster and Paliwal (1997). Let LSTM\({}_{\theta}(\mathbf{x})\) be an abstraction of a long short-term memory network that processes the sequence of vectors \(\mathbf{x}=[\mathbf{x}_{1},...,\mathbf{x}_{|\mathbf{x}|}]\), then output for \(\mathbf{x}_{i}\) is defined as \(\mathbf{h}_{i}\) = BiLSTM\({}_{\theta}(\mathbf{x},i)\) = LSTM\({}^{l}_{\theta}(\mathbf{x}_{[1:i]})\circ\) LSTM\({}^{r}_{\theta}(\mathbf{x}_{[|\mathbf{x}|:i]})\). We consider stacked BiLSTMs, where the output \(\mathbf{h}_{i}^{m}\) of the \(m\)th BiLSTM layer is fed as input to the \(m\)+1th layer. Unless otherwise specified, the input token at a given time step is the concatenation of a word, PoS tag, and another word embedding learned through a character LSTM.

DecoderWe use a feed-forward network, which is fed the output of the last BiLSTM. The output is computed as \(P(y_{i}|\mathbf{h}_{i})\) = softmax\((W\cdot\mathbf{h}_{i}+\mathbf{b})\).

Well-formedness(i) Each token must be assigned a head (one must be the dummy root), and (ii) the graph must be acyclic. If no token is the real root (no head is the dummy root), we search for candidates by relying on the three most likely labels for each token.1 If none is found, we assign it to the first token of the sentence. The single-head constraint is ensured by the nature of the encodings themselves, but some of the predicted head indexes might be out of bounds. If so, we attach those tokens to the real root. If a cycle exists, we do the same for the leftmost token in the cycle.

Footnote 1: If single-rooted trees are a prerequisite, the most probable node will be selected among multiple root nodes.

## 5 Experiments

We use the English Penn Treebank (PTB) Marcus et al. (1993) and its splits for parsing. We transform it into Stanford Dependencies De Marneffe et al. (2006) and obtain the predicted PoS tags using Stanford toutanova et al. (2003). We also select a sample of UDv2.2 treebanks Nivre et al. (2018): Ancient-Greekproiel, CzechpDT, ChineseEGSD, EnglishEWT, FinnishTDT, HebrewHTB, KazakhKTB and TamilTTB, as a representative sample, following de Lhoneux et al. (2017). As evaluation metrics, we use Labeled (LAS) and Unlabeled Attachment Score (UAS). We measure speed in sentences/second, both on a single core of a CPU2 and on a GPU3.

Footnote 2: Intel Core i7-7700 CPU 4.2 GHz.

Footnote 3: GeForce GTX 1080.

SetupWe use NCRFpp as our sequence labeling framework Yang and Zhang (2018). For PTB, we use the embeddings by Ling et al. (2015), for comparison to BIST parser Kiperwasser and Goldberg (2016), which uses a similar architecture, but also needs a parsing algorithm and auxiliary structures. For UD, we follow an end-to-end setup and run UDPipe4Straka and Strakova (2017) for tokenization and tagging. We use the pretrained word embeddings by Ginter et al. (2017). Appendix A contains additional hyperparameters.

### Encoding evaluation and model selection

We first examine the four encodings on the PTB dev set. Table 1 shows the results and also compares them against Li et al. (2018), who proposed seq2seq and sequence labeling models that use a relative positional encoding.

As the relative PoS-based encoding and bracketing-based encoding provide the best results, we will conduct the rest of our experiments with these two encodings. Furthermore, we perform a small hyperparameter search involving encoding, number of hidden layers, their dimension and presence of character embeddings, as these parameters influence speed and accuracy. From now on, we write \(P_{x,y}^{z}\) for a PoS-based encoding model and \(B_{x,y}^{z}\) for a bracketing-based encoding

Figure 1: Types of encoding on an example tree.

 

model, where \(z\) indicates whether character representation was used in the model, \(x\) the number of BiLSTM layers, and \(y\) the word hidden vector dimension. We take as starting points (1) the hyperparameters used by the BIST parser Kiperwasser and Goldberg (2016), as it uses a BiLSTM architecture analogous to ours, with the difference that it employs a transition-based algorithm that uses a stack data structure instead of plain sequence labeling without explicit representation of structure, and (2) the best hyperparameters used by Gomez-Rodriguez and Vilares (2018) for constituent parsing as sequence labeling, as it is an analogous task for a different parsing formalism.

From there, we explore different combinations of parameters and evaluate 20 models on the PTB development set, with respect to accuracy (UAS) and speed (sentences/second on a single CPU core), obtaining the Pareto front in Figure 2. The two starting models based on previous literature (\(P_{2,250}\) and \(P^{C}_{2,800}\), respectively) happen to be in the Pareto front, confirming that they are reasonable hyperparameter choices also for this setting. In addition, we select two more models from the Pareto front (models \(P^{C}_{2,400}\) and \(B_{2,250}\)) for our test set experiments on PTB, as they also provide a good balance between speed and accuracy.

### Results and discussion

Table 2 compares the chosen models, on the PTB test set, against state-of-the-art models. Contrary to previous dependency-parsing-as-sequence-labeling attempts, we are competitive and provide a good speed-accuracy tradeoff. For instance, the \(P^{C}_{2,800}\) model runs faster than the BIST parser Kiperwasser and Goldberg (2016) while being almost as accurate (-0.18 LAS). This comes in spite of its simplicity. While our BiLSTM architecture is similar to that of BIST, the sequence labeling approach does not need a stack, a specific transition system or a dynamic oracle. Using the BIST hyperparameters for our model (\(P_{2,250}\)) yields further increases in speed, at some cost to accuracy: 3.34x faster and -0.04 LAS score than the graph-based model, and 3.51x faster and

\begin{table}
\begin{tabular}{l|c c c c c c} \hline  & & \multicolumn{2}{c|}{**PC\({}_{2,800}\)**} & \multicolumn{2}{c}{**KG (transition-based)**} \\
**Treebank** & Pos type & \begin{tabular}{c} **(sems/a)** \\ **CPU** \\ \end{tabular} & \begin{tabular}{c} **LAS** \\ **CAS** \\ \end{tabular} & \begin{tabular}{c} **LS** \\ **CRPU** \\ \end{tabular} & \begin{tabular}{c} **(sems/a)** \\ **CPU** \\ \end{tabular} & 
\begin{tabular}{c} **LAS** \\ **CRPU** \\ \end{tabular} \\ \hline Ancient Greek & XPOS & \(123_{1.7}\) & 73.71 & 70.87 & \(176_{1.9}\) & 69.43 & 64.41 \\ Chinese & UPOS & \(105_{\pm 0}\) & 6.230 & 59.12 & 73.11 & 64.69 & 60.45 \\ Czech & UPOS & \(125_{\pm 1}\) & 89.10 & 86.68 & \(94_{1.3}\) & 89.25 & 86.11 \\ English & UPOS & \(139_{\pm 1}\) & 84.84 & 78.64 & \(120_{\pm 2}\) & 82.22 & 79.00 \\ Finnish & UPOS & \(168_{\pm 0}\) & 80.12 & 76.22 & \(127_{\pm 3}\) & 80.99 & 76.63 \\ Hebrew & equal PoS & \(120_{\pm 0}\) & 63.84 & 86.66 & \(70_{11}\) & 63.56 & 58.80 \\ Kazakh & XPOS & \(283_{\pm 3}\) & 29.33 & 17.07 & \(178_{\pm 5}\) & 23.09 & 12.73 \\ Tamil & UPOS & \(150_{\pm 2}\) & 71.59 & 64.00 & \(127_{\pm 3}\) & 75.41 & 68.58 \\ \hline \end{tabular}
\end{table}
Table 4: Comparison on UD-CoNLL18 test sets.

Figure 2: UAS/speed Pareto front on the PTB dev set.

 

-0.94 LAS score than their transition-based one.

We now extend our experiments to the sample of UD-CoNLL18 treebanks. To this end, we focus on the \(P_{2,800}^{C}\) model and since our PoS tag-based encoding can be influenced by the specific PoS tags used, we first conduct an experiment on the development sets to determine what tag set (UPoS, the universal PoS tag set, common to all languages, or XPoS, extended language-specific PoS tags) produces the best results for each dataset.

Table 3 shows how the number of unique UpoS and XPoS tags found in the training set differs in various languages. The results suggest that the performance of our system can be influenced by the size of the tag set. It appears that a very large tag set (for instance the XPoS tag set for Czech and Tamil) can hurt the performance of the model and significantly slow down the system, as it results into a large number of distinct labels for the sequence labeling model, increasing sparsity and making the classification harder. In case of Ancient Greek and Kazakh, the best performance is achieved with the XPoS-based encoding. In these corpora, the tag set is slightly bigger than the UpoS tag set. One can argue that the XPoS tags in this case were possibly more fine-grained and hence provided additional useful information to the system facilitating a correct label prediction, without being so large as to produce excessive sparsity.

Table 4 shows experiments on the UD test sets, with the chosen PoS tag set for each corpus. \(P_{2,800}^{\mathrm{C}}\) outperforms transition-based BIST in LAS in 3 out of 8 treebanks,8 and is clearly faster in all analyzed languages. We believe that the variations between languages in terms of LAS difference with respect to BIST can be largely due to differences in the accuracy and granularity of predicted PoS tags, since our chosen encoding relies on them to encode arcs. The bracketing-based encoding, which does not use PoS tags, may be more robust to this. On the other hand, finding the optimal granularity of PoS tags for the PoS-based encoding can be an interesting avenue for future work.

Footnote 8: For Ancient Greek, this may be related to the large amount of non-projectivity (BIST is a projective parser). For extra comparison, a non-projective variant of BIST [57] obtains 71.58 LAS with mono-treebank training, but from better segmentation and morphology than used here. UDPipe [60] obtains 67.57 LAS. Czech and Kazakh have a medium amount of non-projectivity.

In this work, we have also examined the impact of the training data size on the performance of our system compared to the performance of BIST parser. The results in Figure 3 suggest that our model requires more data during the training than BIST parser in order to achieve similar performance. The performance is slightly worse when little training data is available, but later on our model reduces the gap when increasing the training data size.

## 6 Conclusion

This paper has explored fast and accurate dependency parsing as sequence labeling. We tested four different encodings, training a standard BiLSTM-based architecture. In contrast to previous work, our results on the PTB and a subset of UD treebanks show that this paradigm can obtain competitive results, despite not using any parsing algorithm nor external structures to parse sentences.

## Acknowledgments

This work has received funding from the European Research Council (ERC), under the European Union's Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150), from the TELEPARES-UDC project (FFI2014-51978-C2-2-R) and the ANSWER-ASAP project (TIN2017-85160-C2-1-R) from MINECO, and from Xunta de Galicia (ED431B 2017/01). We gratefully acknowledge NVIDIA Corporation for the donation of a GTX Titan X GPU.

Figure 3: Impact of the PTB data size available for parsers during training on the results from the test set.

 



## References

* Brill (1995) Eric Brill. 1995. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. _Computational linguistics_, 21(4):543-565.
* Chen and Manning (2014) Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 740-750, Doha, Qatar. Association for Computational Linguistics.
* De Marneffe et al. (2006) Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In _Lrec_, volume 6, pages 449-454.
* Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In _Proceedings of the 5th International Conference on Learning Representations_.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 334-343. Association for Computational Linguistics.
* automatically annotated raw texts and word embeddings. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
* Gomez-Rodriguez and Vilares (2018) Carlos Gomez-Rodriguez and David Vilares. 2018. Constituent parsing as sequence labeling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1314-1324. Association for Computational Linguistics.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. _arXiv preprint arXiv:1508.01991_.
* Kiperwasser and Ballesteros (2018) Eliyahu Kiperwasser and Miguel Ballesteros. 2018. Scheduled multi-task learning: From syntax to translation. _Transactions of the Association for Computational Linguistics_, 6:225-240.
* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _Transactions of the Association for Computational Linguistics_, 4:313-327.
* Koo and Collins (2010) Terry Koo and Michael Collins. 2010. Efficient third-order dependency parsers. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 1-11, Uppsala, Sweden. Association for Computational Linguistics.
* de Lhoneux et al. (2017) Miryam de Lhoneux, Sara Stymne, and Joakim Nivre. 2017. Old school vs. new school: Comparing transition-based parsers with and without neural network enhancement. In _TLT_, pages 99-110.
* Li et al. (2018) Zuchao Li, Jiaxun Cai, Shexia He, and Hai Zhao. 2018. Seq2seq dependency parsing. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 3203-3214, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Ling et al. (2015) Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1299-1304. Association for Computational Linguistics.
* Ma et al. (2018) Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1403-1414. Association for Computational Linguistics.
* Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn treebank. _Computational linguistics_, 19(2):313-330.
* Nivre et al. (2018) Joakim Nivre et al. 2018. Universal dependencies 2.2. LIDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
* Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus. 1999. Text chunking using transformation-based learning. In _Natural language processing using very large corpora_, pages 157-176. Springer.
* Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 338-348. Association for Computational Linguistics.
* Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. _IEEE Transactions on Signal Processing_, 45(11):2673-2681.
* Shi et al. (2017) Tianze Shi, Liang Huang, and Lillian Lee. 2017. Fast(er) exact decoding and global training for transition-based dependency parsing via a minimal feature set. In _Proceedings of the 2017 Conference

on Empirical Methods in Natural Language Processing_, pages 12-23, Copenhagen, Denmark. Association for Computational Linguistics.
* Smith et al. (2018) Aaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, and Sara Stymne. 2018. 82 treebanks, 34 models: Universal dependency parsing with multi-treebank models. In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 113-123. Association for Computational Linguistics.
* Spoustova and Spousta (2010) Drahomira Spoustova and Miroslav Spousta. 2010. Dependency parsing as a sequence labeling task. _The Prague Bulletin of Mathematical Linguistics_, 94(1):7-14.
* Straka and Strakova (2017) Milan Straka and Jana Strakova. 2017. Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 88-99, Vancouver, Canada. Association for Computational Linguistics.
* Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_, pages 173-180. Association for Computational Linguistics.
* Vinyals et al. (2015) Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In _Advances in Neural Information Processing Systems_, pages 2773-2781.
* Wiseman and Rush (2016) Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1296-1306, Austin, Texas. Association for Computational Linguistics.
* Yang and Zhang (2018) Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In _Proceedings of ACL 2018, System Demonstrations_, pages 74-79, Melbourne, Australia. Association for Computational Linguistics.
* Yli-Jyra (2012) Anssi Yli-Jyra. 2012. _On Dependency Analysis via Contractions and Weighted FSTs_, pages 133-158. Springer Berlin Heidelberg, Berlin, Heidelberg.
* Yi-Jyra and Gomez-Rodriguez (2017) Anssi Yli-Jyra and Carlos Gomez-Rodriguez. 2017. Generic axiomatization of families of noncrossing graphs in dependency parsing. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1745-1755, Vancouver, Canada. Association for Computational Linguistics.
* Zhang et al. (2017a) Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2017a. Dependency parsing as head selection. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, volume 1, pages 665-676.
* Zhang and Nivre (2011) Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 188-193, Portland, Oregon, USA. Association for Computational Linguistics.
* Zhang et al. (2017b) Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. 2017b. Stack-based multi-layer attention for transition-based dependency parsing. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1677-1682, Copenhagen, Denmark. Association for Computational Linguistics.

## Appendix A Model parameters

During the training we use Stochastic Gradient Descent (SGD) optimizer with a batch size of 8, and the model is trained for up to 100 iterations. We keep the model that obtains the highest UAS on the development set. Additional hyperparameters are shown in Table 5.

\begin{table}
\begin{tabular}{|l|l|} \hline Word embedding dimension & 100 \\ Char embedding dimension & 30 \\ PoS tag embedding dimension & 25 \\ Word hidden vector dimension & 250,400,600, \\  & 800,1000,1200 \\ Character hidden vector dimension & 50 \\ Initial learning rate & 0.02 \\ Time-based learning rate decay & 0.05 \\ Momentum & 0.9 \\ Dropout & 0.5 \\ \hline \end{tabular}
\end{table}
Table 5: Common hyperparameters for the sequence labeling models.