Venue: ACL 2017

Character Composition Model with Convolutional Neural Networks for Dependency Parsing on Morphologically Rich Languages

Xiang Yu &Ngoc Thang Vu

Institut fur Maschinelle Sprachverarbeitung

Universitat Stuttgart

{xiangyu,thangvu}@ims.uni-stuttgart.de

###### Abstract

We present a transition-based dependency parser that uses a convolutional neural network to compose word representations from characters. The character composition model shows great improvement over the word-lookup model, especially for parsing agglutinative languages. These improvements are even better than using pre-trained word embeddings from extra data. On the SPMRL data sets, our system outperforms the previous best greedy parser Ballesteros et al. (2015) by a margin of 3% on average.1

Footnote 1: The parser is available at [http://www.ims.uni-stuttgart.de/institut/mitarbeiter/xiangyu/index.en.html](http://www.ims.uni-stuttgart.de/institut/mitarbeiter/xiangyu/index.en.html)

+
Footnote †: _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers)_, pages 672–678 Vancouver, Canada, July 30 - August 4, 2017. ©2017 Association for Computational Linguistics

## 1 Introduction

As with many other NLP tasks, dependency parsing also suffers from the out-of-vocabulary (OOV) problem, and probably more than others since training data with syntactical annotation is usually scarce. This problem is particularly severe when the target is a morphologically rich language. For example, in the SPMRL shared task data sets Seddah et al. (2013, 2014), 4 out of 9 treebanks contain more than 40% word types in the development set that are never seen in the training set.

One way to tackle the OOV problem is to pre-train the word embeddings, _e.g._, with word2vec Mikolov et al. (2013), from a large set of unlabeled data. This comes with two main advantages: (1) more word types, which means that the vocabulary is extended by the unlabeled data, so that some of the OOV words now have a learned representation; (2) more word tokens per type, which means that the syntactic and semantic similarities of the words are better modeled than only using the parser training data.

Pre-trained word embeddings can alleviate the OOV problem by expanding the vocabulary, but it does not model the morphological information. Instead of looking up word embeddings, many researchers propose to compose the word representation from characters for various tasks, _e.g._, part-of-speech tagging dos Santos and Zadrozny (2014); Plank et al. (2016), named entity recognition dos Santos and Guimaraes (2015), language modeling Ling et al. (2015), machine translation Costa-jussa and Fonollosa (2016). In particular, Ballesteros et al. (2015) use a bidirectional long short-term memory (LSTM) character model for dependency parsing. Kim et al. (2016) present a convolutional neural network (CNN) character model for language modeling, but make no comparison among the character models, and state that "it remains open as to which character composition model (_i.e._, LSTM or CNN) performs better".

We propose to apply the CNN model by Kim et al. (2016) in a greedy transition-based dependency parser with feed-forward neural networks Chen and Manning (2014); Weiss et al. (2015). This model requires no extra unlabeled data but performs better than using pre-trained word embeddings. Furthermore, it can be combined with word embeddings from the lookup table since they capture different aspects of word similarities.

Experimental results show that the CNN model works especially well on agglutinative languages, where the OOV rates are high. On other morphologically rich languages, the CNN model also performs at least as good as the word-lookup model.

Furthermore, our CNN model outperforms both the original and our re-implementation of the bidirectional LSTM model by Ballesteros et al. (2015) by a large margin. It provides empirical evidence to the aforementioned open question, suggesting that the CNN is the better character composition model for dependency parsing.

 



## 2 Parsing Models

### Baseline Parsing Model

As the baseline parsing model, we re-implement the greedy parser in Weiss et al. (2015) with some modifications, which brings about \(0.5\%\) improvement, outlined below.2

Footnote 2: We only experiment with the greedy parser, since this paper focuses on the character-level input features and is independent of the global training and inference as in Weiss et al. (2015); Andor et al. (2016).

Since most treebanks contain non-projective trees, we use an approximate non-projective transition system similar to Attardi (2006). It has two additional transitions (Left-2 and Right-2) to the Arc-Standard system (Nivre, 2004) that attach the top of the stack to the third token on the stack, or _vice versa_. We also extend the feature templates in Weiss et al. (2015) by extracting the children of the third token in the stack. The complete transitions and feature templates are listed in Appendix A.

The feature templates consist of 24 tokens in the configuration, we look up the embeddings of the word forms, POS tags and dependency labels of each token.3 We then concatenate the embeddings \({\bf E}_{word}(t_{i})\), \({\bf E}_{tag}(t_{i})\), \({\bf E}_{label}(t_{i})\) for each token \(t_{i}\), and use a dense layer with ReLU non-linearity to obtain a compact representation \({\bf f}(t_{i})\) of the token:

Footnote 3: The tokens in the stack and buffer do not have labels yet, we use a special label <NOLABEL> instead.

\[{\bf x}(t_{i})=[{\bf E}_{word}(t_{i});{\bf E}_{tag}(t_{i});{\bf E }_{label}(t_{i})] \tag{1}\] \[{\bf f}(t_{i})=\max\left\{{\bf 0},{\bf W}_{f}{\bf x}(t_{i})+{ \bf b}_{f}\right\}\]

We concatenate the representations of the tokens and feed them through two hidden layers with ReLU non-linearity, and finally into the softmax layer to compute the probability of each transition:

\[{\bf h}_{0}=[{\bf f}(t_{1});{\bf f}(t_{2});...;{\bf f}(t_{24})]\] \[{\bf h}_{1}=\max\left\{{\bf 0},{\bf W}_{1}{\bf h}_{0}+{\bf b}_{1}\right\}\] \[{\bf h}_{2}=\max\left\{{\bf 0},{\bf W}_{2}{\bf h}_{1}+{\bf b}_{2}\right\}\] \[p(\cdot|t_{1},...,t_{24})=\operatorname*{softmax}({\bf W}_{3}{ \bf h}_{2}+{\bf b}_{3})\]

\({\bf E}_{word}\), \({\bf E}_{tag}\), \({\bf E}_{label}\), \({\bf W}_{f}\), \({\bf W}_{1}\), \({\bf W}_{2}\), \({\bf W}_{3}\), \({\bf b}_{f}\), \({\bf b}_{1}\), \({\bf b}_{2}\), \({\bf b}_{3}\) are all the parameters that will be learned through back propagation with averaged stochastic gradient descent in mini-batches.

Note that Weiss et al. (2015) directly concatenate the embeddings of the words, tags, and labels of all the tokens together as input to the hidden layer. Instead, we first group the embeddings of the word, tag, and label of each token and compute an intermediate representation with shared parameters, then concatenate all the representations as input to the hidden layer.

### LSTM Character Composition Model

To tackle the OOV problem, we want to replace the word-lookup table with a function that composes the word representation from characters.

As a baseline character model, we re-implement the bidirectional LSTM character composition model following Ballesteros et al. (2015). We replace the lookup table \({\bf E}_{word}\) in the baseline parser with the final outputs of the forward and backward LSTMs \(\overline{\mathbf{lstm}}\) and \(\overline{\mathbf{lstm}}\). Equation (1) is then replaced with

\[{\bf x}(t_{i})=[\overline{\mathbf{lstm}}(t_{i});\overline{\mathbf{lstm}}(t_{ i});{\bf E}_{tag}(t_{i});{\bf E}_{label}(t_{i})].\]

We refer the readers to Ling et al. (2015) for the details of the bidirectional LSTM.

### CNN Character Composition Model

In contrast to the LSTM model, we propose to use a "flat" CNN as the character composition model, similar to Kim et al. (2016).4

Footnote 4: We do not use the highway networks since it did not improve the performance in preliminary experiments.

Equation (1) is thus replaced with

\[{\bf x}(t_{i})= [{\bf cnn^{l_{1}}}(t_{i});{\bf cnn^{l_{2}}}(t_{i});...;{\bf cnn^{ l_{k}}}(t_{i});\] \[{\bf E}_{tag}(t_{i});{\bf E}_{label}(t_{i})]. \tag{2}\]

Concretely, the input of the model is a concatenated matrix of character embeddings \({\bf C}\in\mathbb{R}^{d,y\times w}\), where \(d_{i}\) is the dimensionality of character embeddings (number of input channels) and \(w\) is the length of the padded word.5 We apply \(k\) convolutional kernels \(\mathcal{K}\in\mathbb{R}^{d_{o}\times d,yl_{k}}\) with ReLU non-linearity on the input, where \(d_{o}\) is the number of output channels and \(I_{k}\) is the length of the kernel. The output of the convolution operation is \({\bf O}_{conv}\in\mathbb{R}^{d_{o}\times(l-k+1)}\), and we apply a max-over-time pooling that takes the maximum activations of the kernel along each channel, obtaining the final output \({\bf O}_{final}\in\mathbb{R}^{d_{o}}\), which corresponds to the most salient n-gram representation of the word, denoted \({\bf cnn^{l_{k}}}\) in Equation (2). We then concatenate the outputs of several such CNNs with different lengths, so that the information from different n-grams are extracted and can interact with each other.





## 3 Experiments

### Experimental Setup

We conduct our experiments on the treebanks from the SPMRL 2014 shared task Seddah et al. (2013, 2014), which includes 9 morphologically rich languages: Arabic, Basque, French, German, Hebrew, Hungarian, Korean, Polish, and Swedish. All the treebanks are split into training, development, and test sets by the shared task organizers. We use the fine-grained predicted POS tags provided by the organizers, and evaluate the labeled attachment scores (LAS) including punctuation.

We experiment with the CNN-based character composition model (CNN) along with several baselines. The first baseline (WORD) uses the word-lookup model described in Section 2.1 with randomly initialized word embeddings. The second baseline (WZV) uses pre-trained word embeddings by word2vec Mikolov et al. (2013) with the CBOW model and default parameters on the unlabeled texts from the shared task organizers. The third baseline (LSTM) uses a bidirectional LSTM as the character composition model following Ballesteros et al. (2015). Appendix C lists the hyper-parameters of all the models.

Further analysis suggests that combining the character composition models with word-lookup models could be beneficial since they capture different aspects of word similarities (orthographic vs. syntactic/semantic). We therefore experiment with four combined models in two groups: (1) randomly initialized word embeddings (LSTM+WORD vs. CNN+WORD), and (2) pre-trained word embeddings (LSTM+WZV vs. CNN+WZV).

The experimental results are shown in Table 1, with Int denoting internal comparisons (with three groups) and Ext denoting external comparisons, the highest LAS in each group is marked in bold face.

### Internal Comparisons

In the first group, we compare the LAS of the four single models WORD, WZV, LSTM, and CNN. In macro average of all languages, the CNN model performs 2.17% higher than the WORD model, and 1.24% higher than the WZV model. The LSTM model, however, performs only 0.9% higher than the WORD model and 1.27% lower than the CNN model.

The CNN model shows large improvement in four languages: three agglutinative languages (Basque, Hungarian, Korean), and one highly inflected fusional language (Polish). They all have high OOV rate, thus difficult for the baseline parser that does not model morphological information. Also, morphemes in agglutinative languages tend to have unique, unambiguous meanings, thus easier for the convolutional kernels to capture.

In the second group, we observe that the additional word-lookup model does not significantly improve the CNN model (from 82.75% in CNN to 82.90% in CNN+WORD on average) while the LSTM model is improved by a much larger margin (from 81.48% in LSTM to 82.56% in LSTM+WORD on average). This suggests that the CNN model has already learned the most important information from the the word forms, while the LSTM model has not. Also, the combined CNN+WORD model is still better than the LSTM+WORD model, despite the large improvement in the latter.

In the third group where pre-trained word embeddings are used, combining CNN with WZV brings an extra 0.75% LAS over the CNN model. Combining LSTM with WZV shows similar trend but with much larger improvement, yet again, CNN+WZV outperforms LSTM+WZV both on average and individually in 8 out of 9 languages.

\begin{table}
\begin{tabular}{c|c c c c c c c c c|c|c} Model & Ara & Baq & Fre & Ger & Heb & Hun & Kor & Pol & Swe & Avg \\ \hline \multirow{10}{*}{Int} & Word & 84.50 & 77.87 & 82.20 & 85.35 & **74.68** & 76.71 & 84.62 & 80.71 & 79.14 & 80.58 \\  & WZV & **85.11** & 79.07 & **82.73** & **86.60** & 74.55 & 78.21 & 85.30 & 82.37 & 79.67 & 81.51 \\  & LSTM & 83.42 & 82.97 & 81.35 & 85.47 & 74.03 & 83.06 & 86.56 & 80.13 & 77.44 & 81.48 \\  & CNN & 84.65 & **83.91** & 82.41 & 85.61 & 74.23 & **83.68** & **86.99** & **83.28** & **80.00** & **82.75** \\ \cline{2-11} \cline{2-11}  & LSTM+WORD & **84.75** & 83.43 & **82.25** & 85.56 & 74.62 & 83.43 & 86.85 & 82.30 & 79.85 & 82.56 \\  & CNN+WORD & 84.58 & **84.22** & 81.79 & **85.85** & **74.79** & **83.51** & **87.21** & **83.66** & **80.52** & **82.90** \\ \cline{2-11}  & LSTM+WZV & 85.35 & 83.94 & 83.04 & 86.38 & **75.15** & 83.30 & 87.35 & 83.00 & 79.38 & 82.59 \\  & CNN+WZV & **85.67** & **84.37** & **83.09** & **86.81** & 74.95 & **84.08** & **87.72** & **84.44** & **80.35** & **83.50** \\ \hline \multirow{10}{*}{Ext} & B15-WORD & **83.46** & 73.56 & **82.03** & **84.62** & **72.70** & 69.31 & 83.37 & **79.83** & **76.40** & 78.36 \\  & B15-LSTM & 83.40 & **78.61** & 81.08 & 84.49 & 72.26 & **76.34** & **86.21** & 78.24 & 74.47 & **79.46** \\ \cline{1-1} \cline{2-11}  & BestPub & 86.21 & 85.70 & 85.66 & 89.65 & 81.65 & 86.13 & 87.27 & 87.07 & 82.75 & 85.79 \\ \end{tabular}
\end{table}
Table 1: LAS on the test sets, the best LAS in each group is marked in bold face.





### External Comparisons

We also report the results of the two models from Ballesteros et al. (2015): B15-WORD with randomly initialized word embeddings and B15-LSTM as their proposed model. Finally, we report the best published results (BestPub) on this data set (Bjorkelund et al., 2013, 2014).

On average, the B15-LSTM model improves their own baseline by 1.1%, similar to the 0.9% improvement of our LSTM model, which is much smaller than the 2.17% improvement of the CNN model. Furthermore, the CNN model is improved from a strong baseline: our WORD model performs already 2.22% higher than the B15-WORD model.

Comparing the individual performances on each language, we observe that the CNN model almost always outperforms the WORD model except for Hebrew. However, both LSTM and B15-LSTM perform higher than baseline only on the three agglutinative languages (Basque, Hungarian, and Korean), and lower than baseline on the other six.

Ballesteros et al. (2015) do not compare the effect of adding a word-lookup model to the LSTM model as in our second group of internal comparisons. However, Plank et al. (2016) show that combining the same LSTM character composition model with word-lookup model improves the performance of POS tagging by a very large margin. This partially confirms our hypothesis that the LSTM model does not learn sufficient information from the word forms.

Considering both internal and external comparisons in both average and individual performances, we argue that CNN is more suitable than LSTM as character composition model for parsing.

While comparing to the best published results (Bjorkelund et al., 2013, 2014), we have to note that their approach uses explicit morphological features, ensemble, ranking, _etc._, which all can boost parsing performance. We only use a greedy parser with much fewer features, but bridge the 6 points gap between the previous best greedy parser and the best published result by more than one half.

### Discussion on CNN and LSTM

We conjecture that the main reason for the better performance of CNN over LSTM is its flexibility in processing sub-word information. The CNN model uses different kernels to capture n-grams of different lengths. In our setting, a kernel with a minimum length of 3 can capture short morphemes; and with a maximum length of 9, it can practically capture a normal word. With the flexibility of capturing patterns from morphemes up to words, the CNN model almost always outperforms the word-lookup model.

In theory, LSTM has the ability to model much longer sequences, however, it is composed step by step with recurrence. For such deep network architectures, more data would be required to learn the same sequence, in comparison to CNN which can directly use a large kernel to match the pattern. For dependency parsing, training data is usually scarce, this could be the reason that the LSTM has not utilized its full potential.

### Analyses on OOV and Morphology

The motivation for using character composition models is based on the hypothesis that it can address the OOV problem. To verify the hypothesis, we analyze the LAS improvements of the CNN and LSTM model on the development sets in two cases: (1) both the head and the dependent are in vocabu

\begin{table}
\begin{tabular}{r|c c c c c c c c c|c} Model & Case & Ara & Baq & Fre & Ger & Heb & Hun & Kor & Pol & Swe & Avg \\ \hline \multirow{2}{*}{CNN} & AIV & 0.12 & 2.72 & -0.44 & 0.13 & -0.35 & 1.48 & 1.30 & 0.98 & 1.39 & 0.81 \\  & AOV & 0.03 & 5.78 & 0.33 & 0.10 & -1.04 & 5.04 & 2.17 & 2.34 & 0.95 & 1.74 \\ \hline \multirow{2}{*}{LSTM} & AIV & -0.58 & 1.98 & -0.55 & -0.08 & -1.23 & 1.62 & 1.12 & -0.49 & 0.21 & 0.22 \\  & AOV & -0.32 & 5.09 & 0.12 & -0.21 & -1.99 & 4.74 & 1.51 & 0.10 & 0.38 & 1.05 \\ \end{tabular}
\end{table}
Table 2: LAS improvements by CNN and LSTM in the IV and OOV cases on the development sets.

\begin{table}
\begin{tabular}{l|c c c c c c c c c|c} Mod & Ara & Baq & Fre & Ger & Heb & Hun & Kor & Pol & Swe & Avg \\ \hline \multicolumn{1}{l}{\(\bullet\)bc} & -1.23 & -1.94 & -1.35 & -1.57 & -0.79 & -3.23 & -1.22 & -2.53 & -1.54 & -1.71 \\ \multicolumn{1}{l}{a\(\bullet\)c} & -3.47 & -3.96 & -2.39 & -2.54 & -1.24 & -4.52 & -3.21 & -4.47 & -4.19 & -3.33 \\ \multicolumn{1}{l}{ab\(\blacktriangle\)} & -1.52 & -15.31 & -0.72 & -1.23 & -0.26 & -13.97 & -10.22 & -3.52 & -2.61 & -5.48 \\ \hline \multicolumn{1}{l}{a\(\blacktriangle\)} & -3.73 & -19.29 & -3.30 & -3.49 & -1.21 & -17.89 & -12.95 & -6.22 & -6.01 & -8.23 \\ \multicolumn{1}{l}{\(\blacktriangle\)**b\(\blacklozenge\)**} & -3.02 & -18.06 & -2.60 & -3.54 & -1.42 & -18.43 & -11.69 & -6.22 & -3.85 & -7.65 \\ \multicolumn{1}{l}{\(\blacktriangle\)**c**} & -5.11 & -7.90 & -4.05 & -4.86 & -2.50 & -9.75 & -4.56 & -6.71 & -6.74 & -5.80 \\ \end{tabular}
\end{table}
Table 3: Degradation of LAS of the CNN model on the masked development sets.



larly or (2) at least one of them is out of vocabulary. Table 2 shows the results, where the two cases are denoted as \(\Delta\texttt{IV}\) and \(\Delta\texttt{OOV}\). The general trend in the results is that the improvements of both models in the OOV case are larger than in the IV case, which means that the character composition models indeed alleviates the OOV problem. Also, CNN improves on seven languages in the IV case and eight languages in the OOV case, and it performs consistently better than LSTM in both cases.

To analyze the informativeness of the morphemes at different positions, we conduct an ablation experiment. We split each word equally into three thirds, approximating the prefix, stem, and suffix. Based on that, we construct six modified versions of the development sets, in which we mask one or two third(s) of the characters in each word. Then we parse them with the CNN models trained on normal data. Table 3 shows the degradations of LAS on the six modified data sets compared to parsing the original data, where the position of \(\blacklozenge\) signifies the location of the masks. The three agglutinative languages Basque, Hungarian, and Korean suffer the most with masked words. In particular, the suffixes are the most informative for parsing in these three languages, since they cause the most loss while masked, and the least loss while unmasked. The pattern is quite different on the other languages, in which the distinction of informativeness among the three parts is much smaller.

## 4 Conclusion

In this paper, we propose to use a CNN to compose word representations from characters for dependency parsing. Experiments show that the CNN model consistently improves the parsing accuracy, especially for agglutinative languages. In an external comparison on the SPMRL data sets, our system outperforms the previous best greedy parser.

We also provide empirical evidence and analysis, showing that the CNN model indeed alleviates the OOV problem and that it is better suited than the LSTM in dependency parsing.

## Acknowledgements

This work was supported by the German Research Foundation (DFG) in project D8 of SFB 732. We also thank our collegues in the IMS, especially Anders Bjorkelund, for valuable discussions, and the anonymous reviewers for the suggestions.

## References

* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 2442-2452. [https://doi.org/10.18653/v1/P16-1231](https://doi.org/10.18653/v1/P16-1231).
* Atardi (2006) Giuseppe Attardi. 2006. Experiments with a multilanguage non-projective dependency parser. In _Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X)_. Association for Computational Linguistics, pages 166-170. [http://aclweb.org/anthology/W06-2922](http://aclweb.org/anthology/W06-2922).
* Ballesteros et al. (2015) Miguel Ballesteros, Chris Dyer, and A. Noah Smith. 2015. Improved transition-based parsing by modeling characters instead of words with lstms. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 349-359. [https://doi.org/10.18653/v1/D15-1041](https://doi.org/10.18653/v1/D15-1041).
* Bjorkelund et al. (2014) Anders Bjorkelund, Ozlem Cetinoglu, Agnieszka Falenska, Richard Farkas, Thomas Muller, Wolfgang Seeker, and Zsolt Szanto. 2014. The imsprocLaw-szeged-cis entry at the spmrl 2014 shared task: Reranking and morphosylntax meet unlabeled data. _Notes of the SPMRL_ .
* Bjorkelund et al. (2013) Anders Bjorkelund, Ozlem Cetinoglu, Richard Farkas, Thomas Mueller, and Wolfgang Seeker. 2013. (re)ranking meets morphosylntax: State-of-the-art results from the spmrl 2013 shared task. In _Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages_. Association for Computational Linguistics, pages 135-145. [http://aclweb.org/anthology/W13-4916](http://aclweb.org/anthology/W13-4916).
* Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 740-750. [https://doi.org/10.3115/v1/D14-1082](https://doi.org/10.3115/v1/D14-1082).
* Costa-jussa and Fonollosa (2016) R. Marta Costa-jussa and R. Jose A. Fonollosa. 2016. Character-based neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, pages 357-361. [https://doi.org/10.18653/v1/P16-2058](https://doi.org/10.18653/v1/P16-2058).
* Santos and Guimaraes (2015) Cicero dos Santos and Victor Guimaraes. 2015. Boosting named entity recognition with neural character embeddings. In _Proceedings of the Fifth Named Entity Workshop_. Association for Computational Linguistics, pages 25-33. [https://doi.org/10.18653/v1/W15-3904](https://doi.org/10.18653/v1/W15-3904) 

Cleero dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In _Proceedings of the 31st International Conference on Machine Learning (ICML-14)_. pages 1818-1826.
* He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In _Proceedings of the IEEE international conference on computer vision_. pages 1026-1034.
* Kim et al. [2016] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA.._ AAAI Press, pages 2741-2749.
* Ling et al. [2015] Wang Ling, Chris Dyer, W. Alan Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 1520-1530. [https://doi.org/10.18653/v1/D15-1176](https://doi.org/10.18653/v1/D15-1176).
* Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26_, Curran Associates, Inc., pages 3111-3119.
* Nivre [2004] Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In _Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together_. [http://aclweb.org/anthology/W04-0308](http://aclweb.org/anthology/W04-0308).
* Plank et al. [2016] Barbara Plank, Anders Sogaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, pages 412-418. [https://doi.org/10.18653/v1/P16-2067](https://doi.org/10.18653/v1/P16-2067).
* Seddah et al. [2014] Djame Seddah, Sandra Kubler, and Reut Tsarfaty. 2014. Introducing the spmrl 2014 shared task on parsing morphologically-rich languages. In _Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages_. Dublin City University, pages 103-109. [http://aclweb.org/anthology/W14-6111](http://aclweb.org/anthology/W14-6111).
* Seddah et al. [2015] Djame Seddah, Reut Tsarfaty, Sandra Kubler, Marie Candito, D. Jinho Choi, Richard Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletbeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepidrowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Wolinski, Alina Wroblewska, and Villemone Eric de la Clergerie. 2013. Overview of the spmrl 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In _Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages_. Association for Computational Linguistics, pages 146-182. [http://aclweb.org/anthology/W13-4917](http://aclweb.org/anthology/W13-4917).
* Weiss et al. [2015] David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 323-333. [https://doi.org/10.3115/v1/P15-1032](https://doi.org/10.3115/v1/P15-1032).

 



## Appendix A Transitions and Feature Templates

## Appendix B Character Input Preprocessing

For the CNN input, we use a list of characters with fixed length to for batch processing. We add some special symbols apart from the normal alphabets, digits, and punctuations: <SOW> as the start of the word, <EOW> as the end of the word, <MUL> as multiple characters in the middle of the word squeezed into one symbol, <PAD> as padding equally on both sides, and <UNK> as characters unseen in the training data.

For example, if we limit the input length to 9, a short word _ein_ will be converted into <PAD>-<PAD>-<SOW>-e-i-n-<EOW>-<PAD>-<PAD>; a long word _prachtiger_ will be <SOW>-p-r-a-<MUL>-g-e-r-<EOW>. In practice, we set the length as 32, which is long enough for almost all the words.

## Appendix C Hyper-Parameters

The common hyper-parameters of all the models are tuned on the development set in favor of the WORD model:

* 100,000 training steps with random sampling of mini-batches of size 100;
* test on the development set every 2,000 steps;
* early stop if the LAS on the development does not improve for 3 times in a row;
* learning rate of 0.1, with exponential decay rate of 0.95 for every 2,000 steps;
* L2-regularization rate of \(10^{-4}\);
* averaged SGD with momentum of 0.9;
* parameters are initialized following He et al. (2015);
* dimensionality of the embeddings of each word, tag, and label are 256, 32, 32, respectively;
* dimensionality of the hidden layers are 512, 256;
* dropout on both hidden layers with rate of 0.1;
* total norm constraint of the gradients is 10.

The hyper-parameters for the CNN model are:

* dimensionality of the character embedding is 32;
* 4 convolutional kernels of lengths 3, 5, 7, 9;
* number of output channels of each kernel is 64;
* fixed length for the character input is 32.

The hyper-parameters for the LSTM model are:

* 128 hidden units for both LSTMs;
* all the gates use orthogonal initialization;
* gradient clipping of 10;
* no L2-regularization on the parameters.

\begin{table}
\begin{tabular}{c|l} Transition & Configurations (Before  After) \\ \hline Shift & (\(\sigma,[ue_{i}|\beta],A)\Rightarrow\) ((\(\sigma|w_{i}|,\beta,A\)) \\ Left & (\([\sigma|w_{i},w_{j}],\beta,A\)) \\  & (\(\sigma|w_{j}\)), \(\beta,A\cup(w_{j}\to w_{i})\)) \\ Right & (\([\sigma|w_{i},w_{j}],\beta,A\)) \\  & \(\Rightarrow\) ((\(\sigma|w_{i}\)), \(\beta,A\cup(w_{i}\to w_{j})\)) \\ Left-2 & (\([\sigma|w_{i},w_{j},w_{k}],\beta,A\)) \\  & \(\Rightarrow\) (\([\sigma|w_{j},w_{k}]\)), \(\beta,A\cup(w_{k}\to w_{i})\)) \\ Right-2 & (\([\sigma|w_{i},w_{j},w_{k}],\beta,A\)) \\  & \(\Rightarrow\) (\([\sigma|w_{i},w_{j}]\)), \(\beta,A\cup(w_{i}\to w_{k})\)) \\ \end{tabular}
\end{table}
Table 4: The transition system in our experiments, where the configuration is a tuple of (stack, buffer, arcs).

\begin{table}
\begin{tabular}{l} \hline \(s_{1}\), \(s_{2}\), \(s_{3}\), \(s_{4}\), \(b_{1}\), \(b_{2}\), \(b_{3}\), \(b_{4}\), \\ \(s_{1}.lc_{1}\), \(s_{1}.lc_{2}\), \(s_{1}.rc_{1}\), \(s_{1}.rc_{2}\), \\ \(s_{2}.lc_{1}\), \(s_{2}.lc_{2}\), \(s_{2}.rc_{1}\), \(s_{2}.rc_{2}\), \\ \(s_{3}.lc_{1}\), \(s_{3}.lc_{2}\), \(s_{3}.rc_{1}\), \(s_{3}.rc_{2}\), \\ \(s_{1}.lc_{1}.lc_{1}\), \(s_{1}.lc_{1}.rc_{1}\), \(s_{1}.rc_{1}.lc_{1}\), \(s_{1}.rc_{1}.lc_{1}\), \(s_{1}.rc_{1}.rc_{1}\) \\ \hline \end{tabular}
\end{table}
Table 5: The list of tokens to extract feature templates, where \(s_{i}\) denotes the \(i\)-th token in the stack, \(b_{i}\) the \(i\)-th token in the buffer, \(tc_{i}\) denotes the \(i\)-th leftmost child, \(rc_{i}\) the \(i\)-th rightmost child.