Venue: NAACL 2019

# Recursive Subtree Composition in LSTM-Based Dependency Parsing

 Miryam de Lhoneux\({}^{\spadesuit}\) Miguel Ballesteros\({}^{\diamondsuit}\) Joakim Nivre\({}^{\clubsuit}\)

\({}^{\spadesuit}\) Department of Linguistics and Philology, Uppsala University

\({}^{\diamondsuit}\) IBM Research AI, Yorktown Heights, NY {miryam.de_lhoneux,joakim.nivre}@lingfil.uu.se miguel.ballesteros@ibm.com

###### Abstract

The need for tree structure modelling on top of sequence modelling is an open issue in neural dependency parsing. We investigate the impact of adding a tree layer on top of a sequential model by recursively composing subtree representations (composition) in a transition-based parser that uses features extracted by a BiLSTM. Composition seems superfluous with such a model, suggesting that BiLSTMs capture information about subtrees. We perform model ablations to tease out the conditions under which composition helps. When ablating the backward LSTM, performance drops and composition does not recover much of the gap. When ablating the forward LSTM, performance drops less dramatically and composition recovers a substantial part of the gap, indicating that a forward LSTM and composition capture similar information. We take the backward LSTM to be related to lookahead features and the forward LSTM to the rich history-based features both crucial for transition-based parsers. To capture history-based information, composition is better than a forward LSTM on its own, but it is even better to have a forward LSTM as part of a BiLSTM. We correlate results with language properties, showing that the improved lookahead of a backward LSTM is especially important for head-final languages.

## 1 Introduction

Recursive neural networks allow us to construct vector representations of trees or subtrees. They have been used for constituency parsing by Socher et al. (2013) and Dyer et al. (2016) and for dependency parsing by Stenetorp (2013) and Dyer et al. (2015), among others. In particular, Dyer et al. (2015) showed that composing representations of subtrees using recursive neural networks can be beneficial for transition-based dependency parsing. These results were further strengthened in Kuncoro et al. (2017) who showed, using ablation experiments, that composition is key in the Recurrent Neural Network Grammar (RNNG) generative parser by Dyer et al. (2016).

In a parallel development, Kiperwasser and Goldberg (2016b) showed that using BiLSTMs for feature extraction can lead to high parsing accuracy even with fairly simple parsing architectures, and using BiLSTMs for feature extraction has therefore become very popular in dependency parsing. It is used in the state-of-the-art parser of Dozat and Manning (2017), was used in 8 of the 10 highest performing systems of the 2017 CoNLL shared task Zeman et al. (2017) and 10 out of the 10 highest performing systems of the 2018 CoNLL shared task Zeman et al. (2018).

This raises the question of whether features extracted with BiLSTMs in themselves capture information about subtrees, thus making recursive composition superfluous. Some support for this hypothesis comes from the results of Linzen et al. (2016) which indicate that LSTMs can capture hierarchical information: they can be trained to predict long-distance number agreement in English. Those results were extended to more constructions and three additional languages by Gulordava et al. (2018). However, Kuncoro et al. (2018) have also shown that although sequential LSTMs can learn syntactic information, a recursive neural network which explicitly models hierarchy (the RNNG model from Dyer et al. (2015)) is better at this: it performs better on the number agreement task from Linzen et al. (2016).

To further explore this question in the context of dependency parsing, we investigate the use of recursive composition (henceforth referred to as _composition_) in a parser with an architecture like the one in Kiperwasser and Goldberg (2016b). This allows us to explore variations of features and isolate the conditions under which composi

tion is helpful. We hypothesise that the use of a BiLSTM for feature extraction makes it possible to capture information about subtrees and therefore makes the use of subtree composition superfluous. We hypothesise that composition becomes useful when part of the BiLSTM is ablated, the forward or the backward LSTM. We further hypothesis that composition is most useful when the parser has no access to information about the function of words in the context of the sentence given by POS tags. When using POS tags, the tagger has indeed had access to the full sentence. We additionally look at what happens when we ablate character vectors which have been shown to capture information which is partially overlapping with information from POS tags. We experiment with a wider variety of languages than Dyer et al. (2015) in order to explore whether the usefulness of different model variants vary depending on language type.

## 2 K&G Transition-Based Parsing

We define the parsing architecture introduced by Kiperwasser and Goldberg (2016b) at a high level of abstraction and henceforth refer to it as K&G. A K&G parser is a greedy transition-based parser.1 For an input sentence of length \(n\) with words \(w_{1},\ldots,w_{n}\), a sequence of vectors \(x_{1:n}\) is created, where the vector \(x_{i}\) is a vector representation of the word \(w_{i}\). We refer to these as _type_ vectors, as they are the same for all occurrences of a word type. Type vectors are then passed through a feature function which learns representations of words in the context of the sentence.

Footnote 1: Kiperwasser and Goldberg (2016b) also define a graph-based parser with similar feature extraction, but we focus on transition-based parsing.

\[x_{i}=e(w_{i})\] \[v_{i}=f(x_{1:n},i)\]

We refer to the vector \(v_{i}\) as a _token_ vector, as it is different for different tokens of the same word type. In Kiperwasser and Goldberg (2016b), the feature function used is a BiLSTM.

As is usual in transition-based parsing, parsing involves taking transitions from an initial configuration to a terminal one. Parser configurations are represented by a stack, a buffer and set of dependency arcs Nivre (2008). For each configuration \(c\), the feature extractor concatenates the token representations of core elements from the stack and buffer. These token vectors are passed to a classifier, typically a Multilayer Perceptron (MLP). The MLP scores transitions together with the arc labels for transitions that involve adding an arc. Both the word type vectors and the BiLSTMs are trained together with the model.

## 3 Composing Subtree Representations

Dyer et al. (2015) looked at the impact of using a recursive composition function in their parser, which is also a transition-based parser but with an architecture different from K&G. They make use of a variant of the LSTM called a stack LSTM. A stack LSTM has push and pop operations which allow passing through states in a tree structure rather than sequentially. Stack LSTMs are used to represent the stack, the buffer, and the sequence of past parsing actions performed for a configuration.

The words of the sentence are represented by vectors of the word types, together with a vector representing the word's POS tag. In the initial configuration, the vectors of all words are in the buffer and the stack is empty. The representation of the buffer is the end state of a backward LSTM over the word vectors. As parsing evolves, the word vectors are popped from the buffer, pushed to and popped from the stack and the representations of stack and buffer get updated.

Dyer et al. (2015) define a recursive composition function and compose tree representations incrementally, as dependents get attached to their head. The composed representation \(c\) is built by concatenating the vector \(h\) of the head with the vector of the dependent \(d\), as well as a vector \(r\) representing the label paired with the direction of the arc. That concatenated vector is passed through an affine transformation and then through a \(tanh\) non-linear activation.

\[c=tanh(W[h;d;r]+b)\]

They create two versions of the parser. In the first version, when a dependent is attached to a head, the word vector of the head is replaced by a composed vector of the head and dependent. In the second version, they simply keep the vector of the head when attaching a dependent to a head. They observe that the version with composition is substantially better than the version without, by 1.3 LAS points for English (on the Penn Treebank (PTB) test set) and 2.1 for Chinese (on the Chinese Treebank (CTB) test set).

 

Their parser uses POS tag information. POS tags help to disambiguate between different functional uses of a word and in this way give information about the use of the word in context. We hypothesise that the effect of using a recursive composition function is stronger when not making use of POS tags.

## 4 Composition in a K&G Parser

The parsing architectures of the stack LSTM parser (S-LSTM) and K&G are different but have some similarities.2 In both cases, the configuration is represented by vectors obtained by LSTMs. In K&G, it is represented by the token vectors of top items of the stack and the first item of the buffer. In the S-LSTM, it is represented by the vector representations of the entire stack, buffer and sequence of past transitions.

Footnote 2: Note that we use S-LSTM to denote the stack LSTM parser, not the stack LSTM as an LSTM type.

Both types of parsers learn vector representations of word types which are passed to an LSTM. In K&G, they are passed to an LSTM in a feature extraction step that happens before parsing. The LSTM in this case is used to learn vectors that have information about the context of each word, a token vector. In the S-LSTM, word type vectors are passed to Stack LSTMs as parsing evolves. In this case, LSTMs are used to learn vector representations of the stack and buffer (as well as one which learns a representation of the parsing action history).

When composition is not used in the S-LSTM, word vectors represent word types. When composition is used, as parsing evolves, the stack and buffer vectors get updated with information about the subtrees they contain, so that they gradually become contextualised. In this sense, those vectors become more like token vectors in K&G. More specifically, as explained in the previous section, when a dependent is attached to its head, the composition function is applied to the vectors of head and dependent and the vector of the head is replaced by this composed vector.

We cannot apply composition on type vectors in the K&G architecture, since they are not used after the feature extraction step and hence cannot influence the representation of the configuration. Instead, we apply composition on the token vectors. We embed those composed representations in the same space as the token vectors.

In K&G, like in the S-LSTM, we can create a composition function and compose the representation of subtrees as parsing evolves. We create two versions of the parser, one where word tokens are represented by their token vector. The other where they are represented by their token vector and the vector of their subtree \(c_{i}\), which is initially just a copy of the token vector (\(v_{i}=f(x_{1:n},i)\circ c_{i}\)). When a dependent word \(d\) is attached to a word \(h\) with a relation and direction \(r\), \(c_{i}\) is computed with the same composition function as in the S-LSTM defined in the previous section, repeated below.3

Footnote 3: Note that, in preliminary experiments, we tried replacing the vector of the head by the vector of its subtree instead of concatenating the two but concatenating gave much better results.

This composition function is a simple recurrent cell. Simple RNNs have known shortcomings which have been addressed by using LSTMs, as proposed by Hochreiter and Schmidhuber (1997). A natural extension to this composition function is therefore to replace it with an LSTM cell. We also try this variant. We construct LSTMs for subtrees. We initialise a new LSTM for each new subtree that is formed, that is, when a dependent \(d\) is attached to a head \(h\) which does not have any dependent yet. Each time we attach a dependent to a head, we construct a vector which is a concatenation of \(h\), \(d\) and \(r\). We pass this vector to the LSTM of \(h\). \(c\) is the output state of the LSTM after passing through that vector. We denote those models with \(+rc\) for the one using an ungated recurrent cell and with \(+lc\) for the one using an LSTM cell.

\[c =tanh(W[h;d;r]+b)\] \[c =\textsc{Lstm}([h;d;r])\]

As results show (see SS 5), neither type of composition seems useful when used with the K&G parsing model, which indicates that BiLSTMs capture information about subtrees. To further investigate this and in order to isolate the conditions under which composition is helpful, we perform different model ablations and test the impact of recursive composition on these ablated models.

First, we ablate parts of the BiLSTMs: we ablate either the forward or the backward LSTM. We therefore build parsers with 3 different feature functions \(f(x,i)\) over the word type vectors \(x_{i}\) in the sentence \(x\): a BiLSTM (\(bi\)) (our baseline), a backward LSTM (\(bw\)) (i.e., ablating the forward LSTM) and a forward LSTM (\(fw\)) (i.e., ablating 

the backward LSTM):

\[bi(x,i) =\textsc{BiLSTM}(x_{1:n},i)\] \[bw(x,i) =\textsc{Lstm}(x_{n:1},i)\] \[fw(x,i) =\textsc{Lstm}(x_{1:n},i)\]

K&G parsers with unidirectional LSTMs are, in some sense, more similar to the S-LSTM than those with a BiLSTM, since the S-LSTM only uses unidirectional LSTMs. We hypothesise that composition will help the parser using unidirectional LSTMs in the same way it helps an S-LSTM.

We additionally experiment with the vector representing the word at the input of the LSTM. The most complex representation consists of a concatenation of an embedding of the word type \(e(w_{i})\), an embedding of the (predicted) POS tag of \(w_{i}\), \(p(w_{i})\) and a character representation of the word obtained by running a BiLSTM over the characters \(ch_{1:m}\) of \(w_{i}\) (BiLSTM\((ch_{1:m})\)).

\[x_{i}=e(w_{i})\circ p(w_{i})\circ\textsc{BiLSTM}(ch_{1:m})\]

Without a POS tag embedding, the word vector is a representation of the word type. With POS information, we have some information about the word in the context of the sentence and the tagger has had access to the full sentence. The representation of the word at the input of the BiLSTM is therefore more contextualised and it can be expected that a recursive composition function will be less helpful than when POS information is not used. Character information has been shown to be useful for dependency parsing first by Ballesteros et al. (2015). Ballesteros et al. (2015) and Smith et al. (2018b) among others have shown that POS and character information are somewhat complementary. Ballesteros et al. (2015) used similar character vectors in the S-LSTM parser but did not look at the impact of composition when using these vectors. Here, we experiment with ablating either or both of the character and POS vectors. We look at the impact of using composition on the full model as well as these ablated models. We hypothesise that composition is most helpful when those vectors are not used, since they give information about the functional use of the word in context.

ParserWe use UUParser, a variant of the K&G transition-based parser that employs the arc-hybrid transition system from Kuhlmann et al. (2011) extended with a Swap transition and a Static-Dynamic oracle, as described in de Lhoneux et al. (2017b)4. The Swap transition is used to allow the construction of non-projective dependency trees Nivre (2009). We use default hyperparameters. When using POS tags, we use the universal POS tags from the UD treebanks which are coarse-grained and consistent across languages. Those POS tags are predicted by UDPipe Straka et al. (2016) both for training and parsing. This parser obtained the 7th best LAS score on average in the 2018 CoNLL shared task Zeman et al. (2018), about 2.5 LAS points below the best system, which uses an ensemble system as well as ELMo embeddings, as introduced by Peters et al. (2018). Note, however, that we use a slightly impoverished version of the model used for the shared task which is described in Smith et al. (2018a): we use a less accurate POS tagger UDPipe) and we do not make use of multi-treebank models. In addition, Smith et al. (2018a) use the three top items of the stack as well as the first item of the buffer to represent the configuration, while we only use the two top items of the stack and the first item of the buffer. Smith et al. (2018a) also use an _extended feature set_ as introduced by Kiperwasser and Goldberg (2016b) where they also use the rightmost and leftmost children of the items of the stack and buffer that they consider. We do not use that extended feature set. This is to keep the parser settings as simple as possible and avoid adding confounding factors. It is still a near-SOTA model. We evaluate parsing models on the development sets and report the average of the 5 best results in 30 epochs and 5 runs with different random seeds.

Footnote 4: The code can be found at [https://github.com/mdelhoneuv/upparser-composition](https://github.com/mdelhoneuv/upparser-composition)

DataWe test our models on a sample of treebanks from Universal Dependencies v2.1 Nivre et al. (2017). We follow the criteria from de Lhoneux et al. (2017c) to select our sample: we ensure typological variety, we ensure variety of domains, we verify the quality of the treebanks, and we use one treebank with a large amount of non-projective arcs. However, unlike them, we do not use extremely small treebanks. Our selection is the same as theirs but we remove the tiny treebanks and replace them with 3 others. Our final set is: Ancient Greek (PROIEL), Basque, Chinese, Czech, English, Finnish, French, Hebrew and Japanese.

 

Results

First, we look at the effect of our different recursive composition functions on the full model (i.e., the model using a BiLSTM for feature extraction as well as both character and POS tag information). As can be seen from Figure 1, recursive composition using an LSTM cell (\(+lc\)) is generally better than recursive composition with a recurrent cell (\(+rc\)), but neither technique reliably improves the accuracy of a BiLSTM parser.

### Ablating the forward and backward LSTMs

Second, we only consider the models using character and POS information and look at the effect of ablating parts of the BiLSTM on the different languages. The results can be seen in Figure 2. As expected, the BiLSTM parser performs considerably better than both unidirectional LSTM parsers, and the backward LSTM is considerably better than the forward LSTM, on average. It is, however, interesting to note that using a forward LSTM is much more hurtful for some languages than others: it is especially hurtful for Chinese and Japanese. This can be explained by language properties: the right-headed languages suffer more from ablating the backward LSTM than other languages. We observe a correlation between how hurtful a forward model is compared to the baseline and the percentage of right-headed content dependency relations (\(R=-0.838\), \(p<.01\)), see Figure 3.5

Footnote 5: The reason we only consider content dependency relations is that the UD scheme focuses on dependency relations between content words and treats function words as features of content words to maximise parallelism across languages de Marneffe et al. (2014).

There is no significant correlation between how hurtful ablating the forward LSTM is and the percentage of left-headed content dependency relations (\(p>.05\)) indicating that its usefulness is not dependent on language properties. We hypothesise that dependency length or sentence length can play a role but we also find no correlation between how hurtful it is to ablate the forward LSTM and average dependency or sentence length in treebanks. It is finally also interesting to note that the backward LSTM performance is close to the BiLSTMs performance for some languages Japanese and French).

We now look at the effect of using recursive composition on these ablated models. Results are given in Figure 4. First of all, we observe unsurprisingly that composition using an LSTM cell

Figure 1: LAS of models using a BiLSTM (\(bi\)) without composition, with a recurrent cell (\(+rc\)) and with an LSTM cell (\(+tc\)). Bar charts truncated at 50 for visualization purposes.

Figure 3: Correlation between how hurtful it is to ablate the backward LSTM and right-headedness of languages.

Figure 2: LAS of models using a BiLSTM (\(bi\)), backward LSTM (\(bw\)) and forward LSTM (\(fw\)).

 

is much better than using a simple recurrent cell. Second, both types of composition help the backward LSTM case, but neither reliably helps the \(bi\) models. Finally, the recurrent cell does not help the forward LSTM case but the LSTM cell does to some extent. It is interesting to note that using composition, especially using an LSTM cell, bridges a substantial part of the gap between the \(bw\) and the \(bi\) models.

These results can be related to the literature on transition-based dependency parsing. Transition-based parsers generally rely on two types of features: _history-based features_ over the emerging dependency tree and _lookahead features_ over the buffer of remaining input. The former are based on a hierarchical structure, the latter are purely sequential. McDonald and Nivre (2007) and McDonald and Nivre (2011) have shown that history-based features enhance transition-based parsers as long as they do not suffer from error propagation. However, Nivre (2006) has also shown that lookahead features are absolutely crucial given the greedy left-to-right parsing strategy.

In the model architectures considered here, the backward LSTM provides an improved lookahead. Similarly to the lookahead in statistical parsing, it is sequential. The difference is that it gives information about upcoming words with unbounded length. The forward LSTM in this model architecture provides history-based information but unlike in statistical parsing, that information is built sequentially rather than hierarchically: the forward LSTM passes through the sentence in the linear order of the sentence. In our results, we see that lookahead features are more important than the history-based ones. It hurts parsing accuracy more to ablate the backward LSTM than to ablate the forward one. This is expected given that some history-based information is still available through the top tokens on the stack, while the lookahead information is almost lost completely without the backward LSTM.

A composition function gives hierarchical information about the history of parsing actions. It makes sense that it helps the backward LSTM model most since that model has no access to any information about parsing history. It helps the forward LSTM slightly which indicates that there can be gains from using structured information about parsing history rather than sequential information. We could then expect that composition should help the BiLSTM model which, however, is not the case. This might be because the BiLSTM constructs information about parsing history and lookahead into a unique representation. In any case, this indicates that BiLSTMs are powerful feature extractors which seem to capture useful information about subtrees.

Figure 4: LAS of models using a BiLSTM (\(bi\)), backward LSTM (\(bw\)) and forward LSTM (\(fw\)), without recursive composition, with a recurrent cell (\(+rc\)) and with a LSTM cell (\(+lc\)). Bar charts truncated at 50 for visualization purposes.

 



### Ablating POS and character information

Next, we look at the effect of the different word representation methods on the different languages, as represented in Figure 5. As is consistent with the literature Ballesteros et al. (2015); de Lhoneux et al. (2017); Smith et al. (2018), using character-based word representations and/or POS tags consistently improves parsing accuracy but has a different impact in different languages and the benefits of both methods are not cumulative: using the two combined is not much better than using either on its own. In particular, character models are an efficient way to obtain large improvements in morphologically rich languages.

We look at the impact of recursive compositions on all combinations of ablated models, see Table 1. We only look at the impact of using an LSTM cell rather than a recurrent cell since it was a better technique across the board (see previous section).

Looking first at BiLSTMs, it seems that composition does not reliably help parsing accuracy, regardless of access to POS and character information. This indicates that the vectors obtained from the BiLSTM already contain information that would otherwise be obtained by using composition.

Turning to results with either the forward or the backward LSTM ablated, we see the expected pattern. Composition helps more when the model lacks POS tags, indicating that there is some redundancy between these two methods of building contextual information. Composition helps recover a substantial part of the gap of the model with a backward LSTM with or without POS tag. It recovers a much less substantial part of the gap in other cases which means that, although there is some redundancy between these different methods of building contextual information, they are still complementary and a recursive composition function cannot fully compensate for the lack of a backward LSTM or POS and/or character information. There are some language idiosyncracies in the results. While composition helps recover most of the gap for the backward LSTM models without POS and/or character information for Czech and English, it does it to a much smaller extent for Basque and Finnish. We hypothesise that arc depth might impact the usefulness of composition, since more depth means more matrix multiplications with the composition function. However, we find no correlation between average arc depth of the treebanks and usefulness of composition. It is an open question why composition helps some languages more than others.

Note that we are not the first to use composition over vectors obtained from a BiLSTM in the context of dependency parsing, as this was done by Qi and Manning (2017). The difference is that they compose vectors before scoring transitions. It was also done by Kiperwasser and Goldberg (2016a) who showed that using BiLSTM vectors for words in their Tree LSTM parser is helpful but they did not compare this to using BiLSTM vectors without the Tree LSTM.

Recurrent and recursive LSTMs in the way they have been considered in this paper are two ways of constructing contextual information and making it available for local decisions in a greedy parser. The strength of recursive LSTMs is that they can build this contextual information using hierarchical context rather than linear context. A possible weakness is that this makes the model sensitive to error propagation: a wrong attachment leads to using the wrong contextual information. It is therefore possible that the benefits and drawbacks of using this method cancel each other out in the context of BiLSTMs.

### Ensemble

To investigate further the information captured by BiLSTMs, we ensemble the 6 versions of the models with POS and character information with the different feature extractors (\(bi\), \(bw\), \(fw\)) with (\(+lc\)) and without composition. We use the (unweighted) reparsing technique of Sagae and Lavie

Figure 5: LAS of baseline, using char and/or POS tags to construct word representations

(2006)6 and ignoring labels. As can be seen from the UAS scores in Table 2, the ensemble _(full)_ largely outperforms the parser using only a BiLSTM, indicating that the information obtained from the different models is complementary. To investigate the contribution of each of the 6 models, we ablate each one by one. As can be seen from Table 2, ablating either of the BiLSTM models or the backward LSTM using composition, results in the least effective of the ablated models, further strengthening the conclusion that BiLSTMs are powerful feature extractors.

Footnote 5: This method scores all arcs by the number of parsers predicting them and extracts a maximum spanning tree using the Chu-Liu-Edmonds algorithm Edmonds (1967).

## 6 Conclusion

We investigated the impact of composing the representation of subtrees in a transition-based parser. We observed that composition does not reliably help a parser that uses a BiLSTM for feature extraction, indicating that vectors obtained from the BiLSTM might capture subtree information, which is consistent with the results of Linzen et al. (2016). However, we observe that, when ablating the backward LSTM, performance drops and recursive composition does not help to recover much of this gap. We hypothesise that this is because the backward LSTM primarily improves the lookahead for the greedy parser. When ablating the forward LSTM, performance drops to a smaller extent and recursive composition recovers a substantial part of the gap. This indicates that a forward LSTM and a recursive composition function capture similar information, which we take to be related to the rich history-based features crucial for a transition-based parser. To capture this information, a recursive composition function is better than a forward LSTM on its own, but it is even better to have a forward LSTM as part of a BiLSTM. We further find that recursive composition helps more when POS tags are ablated from the model, indicating that POS tags and a recursive composition function are partly redundant ways of constructing contextual information. Finally, we correlate results with language properties, showing that the improved lookahead of a backward LSTM is especially important for head-final languages.

 



## Acknowledgments

We acknowledge the computational resources provided by CSC in Helsinki and Sigma2 in Oslo through NeIC-NLPL (www.nlpl.eu). We thank Sara Stymne and Aaron Smith for many discussions about this paper. Joakim Nivre's contributions to this work were supported by grant 2016-01817 of the Swedish Research Council.

## References

* Ballesteros et al. (2015) Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 349-359.
* Bozat and Manning (2017) Timothy Bozat and Christopher Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing. In _Proceedings of the 5th International Conference on Learning Representations_.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 334-343.
* Dyer et al. (2016) Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In _Proceedings of NAACL-HLT_, pages 199-209.
* Edmonds (1967) Jack Edmonds. 1967. Optimum branchings. _Journal of Research of the National Bureau of Standards_, 71B:233-240.
* Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. _arXiv preprint arXiv:1803.11138_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Kiperwasser and Goldberg (2016a) Eliyahu Kiperwasser and Yoav Goldberg. 2016a. Easy-first dependency parsing with hierarchical tree LSTMs. _Transactions of the Association for Computational Linguistics_, 4:445-461.
* Kiperwasser and Goldberg (2016b) Eliyahu Kiperwasser and Yoav Goldberg. 2016b. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _Transactions of the Association for Computational Linguistics_, 4:313-327.
* Kuhlmann et al. (2011) Marco Kuhlmann, Carlos Gomez-Rodriguez, and Giorgio Satta. 2011. Dynamic programming algorithms for transition-based dependency parsers. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 673-682.
* Kulcoro et al. (2017) Adihyana Kulcoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 1249-1258. Association for Computational Linguistics.
* Konctoro et al. (2018) Adhiguna Kunctoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. Lstms can learn syntax-sensitive dependencies well, but modeling structure makes them better. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1426-1436.
* look, no tags! In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 207-217, Vancouver, Canada. Association for Computational Linguistics.
* de Lhoneux et al. (2017b) Miryam de Lhoneux, Sara Storme, and Joakim Nivre. 2017b. Arc-hybrid non-projective dependency parsing with a static-dynamic oracle. In _Proceedings of the 15th International Conference on Parsing Technologies_, pages 99-104, Pisa, Italy. Association for Computational Linguistics.
* de Lhoneux et al. (2017c) Miryam de Lhoneux, Sara Storme, and Joakim Nivre. 2017c. Old school vs. new school: Comparing transition-based parsers with and without neural network enhancement. In _Proceedings of the 15th Treebanks and Linguistic Theories Workshop (TLT)_.
* Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. _Transactions of the Association for Computational Linguistics_, 4:521-535.
* de Marneffe et al. (2014) Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Universal Stanford Dependencies: A cross-linguistic typology. In _Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC)_, pages 4585-4592.
* McDonald and Nivre (2007) Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_, pages 122-131

Ryan McDonald and Joakim Nivre. 2011. Analyzing and integrating dependency parsers. _Computational Linguistics_, pages 197-230.
* Nivre (2006) Joakim Nivre. 2006. _Inductive Dependency Parsing_. Springer.
* Nivre (2008) Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. _Computational Linguistics_, 34:513-553.
* Nivre (2009) Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (ACL-IJCNLP)_, pages 351-359.
* Nivre et al. (2011) Joakim Nivre, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Attuxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, John Bauer, Kepa Benggoetxea, Riyaz Ahmad Bhat, Eckhard Bick, Victoria Bobicev, Carl Borstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Aljocsha Burchardt, Marie Candito, Gauthier Caron, Gulsen Cehrogliu Eryigit, Giuseppe G. A. Celano, Savas Cetin, Fabricio Chalub, Jinho Choi, Silvie Cinkova, Cagn Coltekin, Miriam Connor, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Arantza Diaz de Ilarraza, Peter Dirix, Kaja Dobrovoljo, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Marhaba Eli, Ali Elkahky, Tomaz Erajvec, Richard Farkas, Hector Fernandez Alcalde, Jennifer Foster, Claudia Freitas, Katarina Galdosova, Daniel Galbraith, Marcos Garcia, Moa Gardenfors, Kim Gerdes, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Memduh Gokirmak, Yoav Goldberg, Xavier Gomez Guinovart, Berta Gonzales Saavedra, Matias Grioni, Normunds Grazitis, Bruno Guillaume, Nizar Habash, Jan Hajic, Jan Hajic jr., Linh Ha My, Kim Harris, Dag Haug, Barbara Hladka, Jaroslava Hlavacova, Florinel Hocius, Potter Holbe, Radu Ion, Elena Irimi, Tomas Jelinek, Anders Johannnsen, Fredrik Jorgensen, Huner Kasikara, Hiroshi Kanayama, Jenna Kanerva, Tolga Kayadelen, Vaclava Kettnerova, Jesse Kirchner, Natalia Kotsbyba, Simon Krek, Veronika Laippala, Lorenzo Lamberntino, Tatiana Lando, John Lee, Phutong Le Hong, Alessandro Lenci, Saran Lertpratid, Herman Leung, Cheuk Ying Li, Josie Li, Keying Li, Nikola Ljubesic, Olga Loginova, Olga Lyshevskaya, Teresa Lynn, Vivien Macketaz, Aibek Makazhanov, Michael Mandli, Christopher Manning, Catalina Maranduc, David Marecek, Katrin Marheinesecke, Hector Martinez Alonso, Andre Martins, Jan Masek, Yuji Matsumoto, Ryan McDonald, Gustavo Mendonca, Niko Miekka, Anna Missila, Catalin Mititelu, Yusuke Miyao, Simoneita Montemagnini, Amir More, Laura Moreno Romero, Shinsuke Mori, Bohdan Mosalevskyi, Kadri Muischnek, Kaili Muurisep, Pinkey Nainwani, Anna Nedoluzhko, Gunta Nespore-Berzkalne, Lutong Nguyen Thi, Huyen Nguyan Thih, Vitaly Nikolaev, Hanna Nurmi, Stina Ojala, Petya Osenova, Robert Ostling, Lilja Ovrelid, Elena Pascual, Marco Passarotti, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen, Emily Pitler, Barbara Plank, Martin Popel, Lauma Pretkanilang, Prokopis Prokopidis, Tiina Puolakainen, Sampo Pyysalo, Alexandre Raddemaker, Loganathan Ramasamy, Taraka Rama, Viniti Ravishankar, Livy Real, Siva Reddy, Georg Rehm, Larissa Rinaldi, Laura Rittuna, Mykhailo Romanenko, Rudolf Rosa, Davide Rovati, Benoit Sagot, Shadi Saleh, Tanja Samardzic, Manuela Sanquinetti, Baiba Saullfe, Sebastian Schuster, Djame Seddah, Wolfgang Seeker, Mojgan Seraji, Mo Shen, Atusko Shimada, Dmitry Sichinava, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simko, Maria Simkova, Kirill Simov, Aaron Smith, Antonio Stella, Milan Straka, Jana Strnadova, Alane Suhr, Umut Sulubacak, Zsolt Szanto, Dima Taij, Takaaki Tanaka, Trond Trosterud, Anna Trukhina, Reutz Tsafratty, Francis Tyers, Sumire Uematsu, Zdenika Uscosova, Larralite Uria, Hans Uszkoreit, Sowmya Vajalai, Daniel van Niekerk, Gerjun van Noord, Viktor Varga, Eric Villemonte de la Clergerie, Veronika Vincze, Lars Wallin, Jonathan North Washington, Mats Wuren, Tak-sum Wong, Zhuoran Yu, Zdenek Zhobkrtsy, Amir Zeldes, Daniel Zeeman, and Hanzhi Zhu. 2017. Universal dependencies 2.1. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proc. of NAACL_.
* Qi and Manning (2017) Peng Qi and Christopher D. Manning. 2017. Arc-swift: A novel transition system for dependency parsing. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 110-117. Association for Computational Linguistics.
* Sagae and Lavie (2006) Kenji Sagae and Alon Lavie. 2006. Parser combination by reparsing. In _Proceedings of the Human Language Technology Conference of the NAACL Companion Volume: Short Papers_, pages 129-132.
* Smith et al. (2018a) Aaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, and Sara Stymne. 2018a. 82 treebanks, 34 models: Universal dependency parsing with multi-treebank models. In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 113-123. Association for Computational Linguistics.
* Smith et al. (2018b) Aaron Smith, Miryam de Lhoneux, Sara Stymne, and Joakim Nivre. 2018b. An investigation of the interactions between pre-trained word embeddings, character models and pos tags in dependency parsing.

 

In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2711-2720. Association for Computational Linguistics.
* Socher et al. (2013) Richard Socher, John Bauer, Christopher D Manning, et al. 2013. Parsing with compositional vector grammars. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 455-465.
* Stenetorp (2013) Pontus Stenetorp. 2013. Transition-based dependency parsing using recursive neural networks. In _Deep Learning Workshop at the 2013 Conference on Neural Information Processing Systems (NIPS)_, Lake Tahoe, Nevada, USA.
* Straka et al. (2016) Milan Straka, Jan Hajic, and Jana Strakova. 2016. UDPipe: trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In _Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016)_, Portoroz, Slovenia. European Language Resources Association.
* Zeman et al. (2018) Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_.
* Zeman et al. (2019) Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Vaclava Kettnerova, Zhenka Uresova, Jenna Kanerva, Stina Ojala, Anna Missila, Christopher Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Hector Martinez Alonso, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpratid, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strandova, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_. Association for Computational Linguistics.