Venue: NAACL 2018

# Exploiting Dynamic Oracles to Train Projective Dependency Parsers on Non-Projective Trees

Lauriane Aufrant\({}^{1,2}\), Guillaume Wisniewski\({}^{1}\) and Francois Yvon\({}^{1}\)

\({}^{1}\)LIMSI, CNRS, Univ. Paris-Sud, Universite Paris-Saclay, F-91405 Orsay, France

\({}^{2}\)DGA, 60 boulevard du General Martial Valin, 75 509 Paris, France

{lauriane.aufrant,guillaume.wsiniewski,francois.yvon}@limsi.fr

###### Abstract

Because the most common transition systems are projective, training a transition-based dependency parser often implies to either ignore or rewrite the non-projective training examples, which has an adverse impact on accuracy. In this work, we propose a simple modification of dynamic oracles, which enables the use of non-projective data when training projective parsers. Evaluation on 73 treebanks shows that our method achieves significant gains (+2 to +7 UAS for the most non-projective languages) and consistently outperforms traditional projectivization and pseudo-projectivization approaches.

## 1 Introduction

Because of their efficiency and ease of implementation, transition-based parsers are the most common systems for dependency parsing. However, efficiency comes at a price, namely a loss in expressivity: while graph-based parsers are able to produce any tree spanning the input sentence, many transition-based systems are restricted to projective trees. Informally, a dependency tree is _non-projective_ if at least one dependency crosses another arc (see Figure 1).

The inability to generate non-projective trees is an obvious issue for accuracy: at test time, a projective parser is guaranteed to be wrong for all the non-projective dependencies, a limitation already pointed out several times Nivre (2009); Lacroix and Bechet (2014). In this paper, we show that the impact can also be severe _at training time_. This is because the standard training procedure assumes that the reference tree is within reach of the parser, which is not the case for non-projective examples. Therefore, projective parsers cannot make any use of such samples and common practice is to filter them out, thereby wasting potentially valuable training material. Depending on the annotation schemes and languages, between 5 and 10% of the training set are typically discarded.1

Footnote 1: In UD 1.2, examples with non-projective dependencies represent 4.96% of trees (0.48% of dependencies) for English, 12.45% (0.83%) for French, 8.19% (0.33%) for Arabic. However, the numbers rise for some languages such as Dutch (30.87%, 4.10%) or Ancient Greek (63.22%, 9.78%). See Straka et al. (2015) for complete data on 37 treebanks.

Several strategies have been proposed to overcome the projectivity constraint. One line of research is to sacrifice parsing efficiency and introduce special transition systems capable to build non-projective dependencies Covington (2001); Nivre (2009). Another approach introduces non-projective dependencies by post-processing the output projective trees. This is the case of the pseudo-projectivization method Nivre and Nilsson (2005), which encodes crossings in augmented relation labels and makes all examples projective. The accuracy on projective dependencies alone can also be maximized by projectivizing all training examples prior to training, using Eisner (1996)'s decoder.

In this work, we propose an alternative strategy: we show (SS3) that it is possible, with a small modification of the dynamic oracle of Goldberg and Nivre (2012), to directly train a projective parser with non-projective examples. While our approach remains unable to produce non-projective trees, it still results in signific

Figure 1: An example of non-projectivity: the tree is made non-projective by the dependency \(What^{\prime}do\).



on the overall UAS (SS4), and consistently outperforms the (pseudo-)projectivization approaches.

## 2 Training transition-based parsers

### Parsing with a transition system

In transition-based parsing Nivre (2003), dependency trees are built incrementally, in a shift-reduce manner: to parse a sentence, a sequence of transitions (the _derivation_) is applied on the internal state of the parser (the _configuration_), consisting typically in a stack and a buffer of unprocessed words. In the ArcEager transition system Nivre (2004), four actions are available, one for each possible transition (see Table 1).

At parsing time, each transition to follow is predicted in turn by a classifier, typically an averaged perceptron Collins and Roark (2004) or a neural network Chen and Manning (2014); Dyer et al. (2015); Andor et al. (2016), based on features extracted from the current parser configuration.

Compared to other parsing frameworks, such as graph-based parsing McDonald et al. (2005), a major advantage of transition-based parsing is its computational efficiency: the processing time of a sentence is linear in its length.

### Training with dynamic oracles

The classifier used to predict the actions is typically trained in an online fashion, using a dataset consisting of input sentences and reference parse trees. Various strategies have been envisioned to generate, based on that data, pairs of positive (gold) and negative (predicted) parser configurations with which to update the model. A recent and successful proposal uses so-called _dynamic oracles_Goldberg and Nivre (2012): the training example is parsed by the model, and for each predicted configuration in the resulting derivation, the dynamic oracle computes a reference action, tailored to the current configuration.

In practice, the reference is defined as an action which does not degrade the accuracy on that sentence: if a transition prevents a gold arc from being produced later (such as attaching a token to the wrong head), it is incorrect, but no error is flagged if that arc was already unreachable (for instance if the true head was already removed from the stack).

Formally, if each configuration is associated with a \({\rm UAS}_{max}\), the maximum UAS value that can be achieved by any of its successor derivations, then the _action cost_ is defined as the difference between the current \({\rm UAS}_{max}\) and the future \({\rm UAS}_{max}\) (once the corresponding action has been applied). The best decision in that situation is the one which ensures the best future UAS, ie. which leaves \({\rm UAS}_{max}\) unchanged and has zero cost. By definition of \({\rm UAS}_{max}\), in all configurations at least one action has zero cost; there may even be several in case of spurious ambiguities. Hence, when asked for a reference action, the dynamic oracle simply returns the set of zero-cost actions.

The core of the method is thus the computation of action costs. In order to simplify it, Goldberg and Nivre (2013) introduce the concept of _arc decomposition_: a transition system is _arc-decomposable_ if in every configuration, all the gold dependencies that are still reachable can be reached simultaneously by the same derivation. It ensues that for such systems, the action cost is simply the number of gold arcs that the action explicitly forbids, which are in general straightforward to enumerate. For instance in ArcEager, the Reduce action (which pops the topmost stack element \(s\)) has a cost of 1 for each child of \(s\) still in the buffer, since they no longer can get their true head Goldberg and Nivre (2013).

Arc-decomposability does not always hold, however, in which case there are extra costs to take into account: by definition of non-arc-decomposable systems, some arcs are incompatible (they are not unreachable, they can simply not be reached together). Therefore, at some point, adding a gold arc will imply renouncing to another gold arc, thereby inserting an error. It is however incorrect to assign this cost to the given action, since it is due to a much earlier action which introduced the incompatibility.2 As exemplified by Goldberg et al. (2014) and Gomez-Rodriguez and Fernandez-Gonzalez (2015), it is not impossible to derive dynamic oracles for non-arc-decomposable systems, but taking this kind of incompatibilities into account makes the computation of their action 

costs much more complex.

## 3 Using dynamic oracles to train on non-projective data

The reason why dynamic oracles can help solving the non-projectivity issue is that non-projective examples, in this framework, are not different from projective ones: the cost is well-defined for any action, and by definition there is always at least one zero-cost action. So, all training examples are usable by design, regardless of their projectivity. The issue rather resides in deriving a sound definition of the cost, which covers non-projective cases;3 in the current state of the art, the dynamic oracles derived for projective systems are only sound for projective examples, though (see Figure 2).

Footnote 3: Exhaustive search would be a straightforward strategy to compute exact action costs in any setting, but it is computationally too expensive.

One way to look at non-projective examples is as a set of configurations containing arc incompatibilities: when two crossing edges are reachable, only one can actually belong to the final output. Yet, this is a known setting: with non-arc-decomposable systems, some erroneous configurations face the same issue. Hence, from the oracle point of view, the initial empty configuration already comes with embedded 'past errors' (the incompatibilities due to edge crossings). As in non-arc-decomposable systems, the cost incurred by these incompatibilities is not due to actions to come, but should be attributed to previous actions, taken in a fictive history before the initial configuration. As such, the natural behavior of dynamic oracles is to ignore this cost. Hence, using the same methodology as for non-arc-decomposable systems, it is formally possible to define dynamic oracles for non-projective examples. But it implies enumerating all non-projective arc incompatibilities in an arbitrary parser configuration (and proving exhaustivity), which is a difficult task and remains, to date, an open question.

Instead of deriving exact costs, we propose here a straightforward strategy which _approximates_ the action costs for non-projective examples: using the usual cost computation, but defining the oracles as _minimum-cost_ actions instead of zero-cost ones. Indeed, when the parser ends up in a configuration where all decisions appear erroneous, the part of the cost which is common to all actions should in fact have been taken care of in the past; with this minimum-cost approach, it is ignored. In Figure 2, the Right action is chosen as reference by the minimum-cost criterion, thereby acknowledging the fact that non-projectivity by itself incurs a cost of 1. This oracle generalizes the zero-cost one, as they are equivalent on projective trees.

Compared to an exact oracle, this approximated cost biases the oracle towards delaying the resolution of incompatibilities (like the Shift action in Figure 3). A few updates are consequently unsound, but empirically their impact remains small compared to the benefits of making more examples usable, as will be assessed in the next section.

\begin{table}
\begin{tabular}{|l l c r r r r r r r|} \hline Shift & [S] & \((\sigma,\quad b|\beta,\quad P)\) & \(\Rightarrow\) & \((\sigma|b,\quad\beta,\quad P)\) & if \(b\) is a word \\ Left & [L] & \((\sigma|s,\quad b|\beta,\quad P)\) & \(\Rightarrow\) & \((\sigma,\quad b|\beta,\quad P+(b\to s))\) & if \(s\) is a word and \(s\) is unattached \\ Right & [R] & \((\sigma|s,\quad b|\beta,\quad P)\) & \(\Rightarrow\) & \((\sigma|s|b,\quad\beta,\quad P+(s\to b))\) & & \\ Reduce & [Re] & \((\sigma|s,\quad\beta,\quad P)\) & \(\Rightarrow\) & \((\sigma,\quad\beta,\quad P)\) & if \(s\) is attached \\ \hline \end{tabular}
\end{table}
Table 1: The ArcEager transition system: semantics and preconditions of each action. \(\sigma\) stands for the stack, \(s\) for the topmost stack element, \(\beta\) stands for the buffer, \(b\) for the first buffer element, and \(P\) is the partially built tree.

Figure 3: A configuration where action cost is poorly approximated, for a non-projective reference tree (see dotted edges). All gold arcs are reachable, but at most two can be reached simultaneously. The cost computed for Left is 2, 1 for Right and 0 for Shift, even though all actions lead to a tree with two gold arcs.

Figure 2: A configuration where all actions have non-zero cost (thereby contradicting its definition), for a non-projective reference tree (see dotted edges), when using the standard ArcEager dynamic oracle. The action cost is 2 for Shift (only w\({}_{2}\) can be correctly attached, by applying L+L+L afterwards), 2 for Left (L+S+L correctly attaches w\({}_{1}\) only) and 1 for Right (Re+L+L correctly attaches both w\({}_{2}\) and w\({}_{3}\)).

 



## 4 Experiments

The benefits of non-projective examples for training projective parsers are evaluated on the 73 treebanks of the UD 2.0 (Nivre et al., 2017, 2018). Three methods to exploit non-projective trees (instead of discarding them) are contrasted: learning on the trees projectivized using Eisner (1996)'s algorithm, learning on pseudo-projectivized examples (Nivre and Nilsson, 2005) and learning on the non-projective trees, with the minimum-cost oracle described in SS3. Projectivization is based on Yoav Goldberg's code.4 For pseudo-projectivization, the MaltParser 1.9 implementation is used, with the head encoding scheme. For parsing, we use PanParser(Aufrant and Wisniewski, 2016), our own open source5 implementation of a greedy ArcEager parser (using an averaged perceptron and a dynamic oracle).

Footnote 4: [https://www.cs.hgu.ac.il/~yoavg/software/projectivize](https://www.cs.hgu.ac.il/~yoavg/software/projectivize)

Footnote 5: [https://perso.limsi.fr/aufrant](https://perso.limsi.fr/aufrant)

As shown in Table 2, it is empirically better to handle non-projective sentences with minimum-cost dynamic oracles than to discard them all; but this strategy also outperforms projectivization and pseudo-projectivization. As expected, the gains of all methods increase when the proportion of non-projectivity increases, i.e. when more examples would have been discarded.

The minimum-cost technique is notably effective on the least projective treebanks, which correspond to ancient languages like Ancient Greek (63% of non-projective examples, with a gain of +2.4 UAS, compared to +1.1 and +2.0 for projectivization and pseudo-projectivization) and Latin (41%, +2.0 vs +0.4/+2.8); but it also achieves large improvements for modern languages with less non-projectivity, such as Dutch-LassySmall (30%, +7.0 vs +5.7/+3.3), Belarusian (17%, +5.2 vs +2.2/+4.4) and Turkish (11%, +2.3 vs +1.9/+1.3).

Apart from higher gains on average, the advantage of the minimum-cost strategy is that

\begin{table}
\begin{tabular}{l r r r r r r r} \hline \hline  & & \multicolumn{3}{c}{\% non-projective sentences} & \multicolumn{3}{c}{\# training sentences} \\ \cline{3-8}  & \multirow{2}{*}{\(\mu\)} & \(>50\%\) & 25-50\% & 10-25\% & \(<10\%\) & \(>500\) & \(<500\) \\ \# corresponding treebanks: [73] & [1] & [9] & [32] & [31] & [57] & [16] \\ \hline \multicolumn{8}{c}{PanParser – greedy Arceaeger} \\ \hline Static oracle (only projective snt.) & 78.28 & 56.23 & 76.22 & 75.49 & 82.47 & 81.34 & 67.37 \\ Static oracle + projectivized snt. & +0.31 & +2.12 & +1.54 & +0.15 & +0.06 & +0.50 & -0.36 \\ Dynamic oracle (only projective snt.) & 78.94 & 57.75 & 76.99 & 76.26 & 82.97 & 81.92 & 68.34 \\ Dynamic oracle + projectivized snt. & +0.32 & +1.06 & +1.04 & +0.52 & -0.11 & +0.24 & +0.61 \\ Dynamic oracle + pseudo-proj. snt. & +0.26 & +2.01 & +1.49 & +0.19 & -0.07 & +0.47 & -0.45 \\ Dynamic oracle + non-projective snt. & +0.48 & +2.43 & +1.83 & +0.44 & +0.07 & +0.52 & +0.36 \\ \hline \multicolumn{8}{c}{PanParser –**beam**ArcEager} \\ \hline Global dynamic oracle (only projective snt.) & 79.77 & 57.12 & 78.28 & 76.83 & 83.96 & 83.12 & 67.84 \\ Global dynamic oracle + non-projective snt. & 80.40 & 61.49 & 80.11 & 77.54 & 84.04 & 83.63 & 68.87 \\  & +0.63 & +4.38 & +1.84 & +0.71 & +0.07 & +0.51 & +1.03 \\ \hline \multicolumn{8}{c}{PanParser – greedy ArCHybrid} \\ \hline Static oracle (only projective snt.) & 75.71 & 53.08 & 73.66 & 73.19 & 79.63 & 78.29 & 66.51 \\ Dynamic oracle (only projective snt.) & 76.51 & 54.23 & 74.61 & 73.96 & 80.41 & 79.23 & 66.81 \\ Dynamic oracle + non-projective snt. & +0.55 & +3.08 & +2.16 & +0.33 & +0.22 & +0.53 & +0.61 \\ \hline \multicolumn{8}{c}{**MaltrParser – greedy ArCéager**} \\ \hline Baseline (only projective snt.) & 72.88 & 57.88 & 71.74 & 69.99 & 76.68 & 76.82 & 58.87 \\ + pseudo-projectivized sentences & +0.37 & +5.84 & +1.41 & +0.19 & +0.08 & +0.48 & -0.01 \\ + non-proj. output (pseudo-proj. + deproj.) & +0.45 & +6.83 & +1.69 & +0.25 & +0.09 & +0.59 & -0.04 \\ \hline \multicolumn{8}{c}{**UDPIPe – tuned hyperparameters**} \\ \hline Baseline (proj. and non-proj. parsens) & 79.47 & 66.99 & 81.20 & 75.51 & 83.45 & 83.67 & 64.48 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison on Universal Dependencies 2.0 of various strategies to handle non-projective training examples, depending on the non-projectivity rate and on treebank size. We report the average UAS over the corresponding sets of languages. All UAS gains are computed with respect to their ‘_only projective snt._’ baseline.



it is consistently beneficial, whereas pseudo-projectivization is detrimental for small treebanks. A plausible explanation is that arbitrarily rewriting the trees introduces inconsistencies in the training material, which are only alleviated when data is large enough. In that regard, the opposite effects of projectivization (detrimental with a static oracle, beneficial with a dynamic one) highlight the limited reliability of such transformations.

The minimum-cost strategy is also applied to an improved version of PanParser, using beam search and a dynamic oracle extended to global training Aufrant et al. (2017), with a beam of size 8 and the max-violation strategy. The minimum-cost criterion appears particularly fit for that setting, with even larger gains (+0.63 UAS on average) despite a higher baseline.

Comparison with other parsersFor illustrative purposes, similar experiments are conducted with other parsing systems: the ArchHybrid version of PanParser, MaltParser and UDPipe. MaltParser is the original implementation of the ArcEager system, but differs from ours in several ways, notably feature templates and the oracle (which is not dynamic, but precomputed statically); to help comparison, additional results are reported for PanParser without dynamic oracles. UDPipe is a state-of-the-art neural parser including both projective and non-projective parsing systems; we use version 1.1 Straka and Strakova (2017) with Straka (2017)'s set of tuned hyperparameters, but without their pre-trained word embeddings, for fair comparison.

The ArchHybrid results show that the gains achieved by the minimum-cost criterion are not specific to the ArCEager system: despite different baseline scores, the proposed strategy yields similar improvements.

Compared to MaltParser, our ArcEager baseline appears much stronger (+5.4 UAS) on the downsized datasets; but the gains achieved when exploiting the non-projective trees (with pseudo-projectivization) are similar in both implementations. There is one exception, Ancient Greek (the only treebank with more than 50% non-projective sentences), for which the MaltParser gains are way larger than those of PanParser; but this treebank seems particular in several regards6 and consequently does not question the superiority of the minimum-cost oracle over the pseudo-projectivization strategy, measured even in Ancient Greek for PanParser.

Footnote 6: The baseline MaltParser already behaves differently for that language: it slightly outperforms the baseline PanParser instead of underperforming it by a large margin. The explanation may simply lie in other differences between implementations, for instance MaltParser’s feature templates may be particularly suited to Ancient Greek.

Table 2 also reports the gains achieved by MaltParser when pseudo-projectivization is followed by deprojectivization of the output. Plain comparison of this line with the minimum-cost strategy is delicate, because it does not result from better training only, but also from a gain in expressivity: it is able to retrieve even non-projective dependencies. But it is interesting to see that deprojectivization only marginally improves over pseudo-projectivization alone: most of the gain actually resides in the treebank augmentation rather than in retrieving non-projective dependencies. Besides, the minimum-cost strategy outperforms even the deprojectivized results.

Finally, measures with UDPipe reveal that, even though it benefits a lot from its higher expressivity (as it uses non-projective systems for the most non-projective treebanks), it achieves low accuracies on small treebanks and is thus outperformed on average by the beam version of PanParser (+0.30 UAS) - and the minimum-cost criterion significantly widens that gap (+0.97 UAS).

## 5 Conclusion

This work has addressed the restriction of projective parsers to train only on projective examples. We have explained how the dynamic oracle framework can help overcoming this issue, and shown that a simple modification of the framework (using minimum-cost actions as references instead of zero-cost ones) enables a seamless use of non-projective examples. Compared to the traditional (pseudo-)projectivization approaches, this method provides higher and more reliable improvements over the filtering baseline.

## Acknowledgments

This work has been partly funded by the French _Direction generale de l'armement_ and by the _Agence Nationale de la Recherche_ under ParSiTi project (ANR-16-CE33-0021) and MultiSem project (ANR-16-CE33-0013).

 



## References

* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally Normalized Transition-Based Neural Networks. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2442-2452, Berlin, Germany. Association for Computational Linguistics.
* Aufrant and Wisniewski (2016) Lauriane Aufrant and Guillaume Wisniewski. 2016. PanParser: a Modular Implementation for Efficient Transition-Based Dependency Parsing. Technical report, LIMSI-CNRS.
* Aufrant et al. (2017) Lauriane Aufrant, Guillaume Wisniewski, and Francois Yvon. 2017. Don't Stop Me Now! Using Global Dynamic Oracles to Correct Training Biases of Transition-Based Dependency Parsers. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 318-323, Valencia, Spain. Association for Computational Linguistics.
* Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 740-750, Doha, Qatar. Association for Computational Linguistics.
* Collins and Roark (2004) Michael Collins and Brian Roark. 2004. Incremental Parsing with the Perceptron Algorithm. In _Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL'04), Main Volume_, pages 111-118, Barcelona, Spain.
* Covington (2001) Michael A. Covington. 2001. A Fundamental Algorithm for Dependency Parsing. In _Proceedings of the 39th annual ACM southeast conference_, pages 95-102.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-Based Dependency Parsing with Stack Long Short-Term Memory. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 334-343, Beijing, China. Association for Computational Linguistics.
* Eisner (1996) Jason M Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In _Proceedings of the 16th conference on Computational linguistics-Volume 1_, pages 340-345. Association for Computational Linguistics.
* Goldberg and Nivre (2012) Yoav Goldberg and Joakim Nivre. 2012. A Dynamic Oracle for Arc-Eager Dependency Parsing. In _Proceedings of COLING 2012_, pages 959-976, Mumbai, India. The COLING 2012 Organizing Committee.
* Goldberg and Nivre (2013) Yoav Goldberg and Joakim Nivre. 2013. Training Deterministic Parsers with Non-Deterministic Oxacles. _Transactions of the Association for Computational Linguistics_, 1:403-414.
* Goldberg et al. (2014) Yoav Goldberg, Francesco Sartorio, and Giorgio Satta. 2014. A Tabular Method for Dynamic Oracles in Transition-Based Parsing. _Transactions of the Association for Computational Linguistics_, 2:119-130.
* Gomez-Rodriguez and Fernandez-Gonzalez (2015) Carlos Gomez-Rodriguez and Daniel Fernandez-Gonzalez. 2015. An Efficient Dynamic Oracle for Unrestricted Non-Projective Parsing. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 256-261, Beijing, China. Association for Computational Linguistics.
* Lacroix and Bechet (2014) Ophelie Lacroix and Denis Bechet. 2014. A Three-Step Transition-Based System for Non-Projective Dependency Parsing. In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_, pages 224-232, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
* McDonald et al. (2005) Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. 2005. Non-Projective Dependency Parsing using Spanning Tree Algorithms. In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_, pages 523-530, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
* Nivre (2003) Joakim Nivre. 2003. An Efficient Algorithm for Projective Dependency Parsing. In _Proceedings of the 8th International Workshop on Parsing Technologies_, IWPT 2003, Nancy, France.
* Nivre (2004) Joakim Nivre. 2004. Incrementality in Deterministic Dependency Parsing. In _Proceedings of the ACL Workshop Incremental Parsing: Bringing Engineering and Cognition Together_, pages 50-57, Barcelona, Spain. Association for Computational Linguistics.
* Nivre (2009) Joakim Nivre. 2009. Non-Projective Dependency Parsing in Expected Linear Time. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 351-359, Suntec, Singapore. Association for Computational Linguistics.
* Nivre et al. (2012) Joakim Nivre, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Attuxa, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, John Bauer, Kepa Bengotexea, Riyaz Ahmad Bhat, Eckhard Bick, Cristina Bosco, Gosse Bouma, Sam Bowman, Aljoscha Burchardt, Marie Candito, Gauthier Caron, Gulsen

Cebirolu Eryjit, Giuseppe G. A. Celano, Savas Cetin, Fabricio Chalub, Jinho Choi, Yongseok Cho, Silvie Cinkova, Car Coltekin, Miriam Connor, Marie-Catherine de Marmeffe, Valeria de Paiva, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy Dozat, Kira Droganov, Mariaba Eli, Ali Elkahky, Tomaz Erajvec, Richard Farkas, Hector Fernandez Alcalde, Jennifer Foster, Claudia Freitas, Katarina Gajdosova, Daniel Galbraith, Marcos Garcia, Filip Ginter, Iakes Goenaga, Koldo Gjoenda, Memudh Gokrmak, Yoav Goldberg, Xavier Gomez Guinourart, Berta Gonzalez Saavedra, Matias Grioni, Normunds Grazitis, Bruno Guillaume, Nizar Habash, Jan Hajic, Jan Hajic jr., Linh Ha M, Kim Harris, Dag Haug, Barbora Hladka, Jaroslava Hlavacova, Petter Holhe, Radu Ion, Elena Irimia, Anders Johannsen, Fredrik Jorgensen, Huner Kaszaka, Hiroshi Kanayama, Jenna Kanerva, Tolga Kayadelen, Vaclava Kettenrova, Jesse Kirchner, Natalia Kotsyba, Simon Krek, Sookyoung Kwak, Veronika Liappala, Lorenzo Lambertino, Tatiana Lando, Phng Le Hong, Alessandro Lenci, Saran Lertpratidt, Herman Leung, Cheuk Ying Li, Josie Li, Nikola Ljubesic, Olga Loginova, Olga Lyashesvskaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Catalina Maranduc, David Marecek, Katrin Marinhenecke, Hector Martinez Alonso, Andre Martins, Jan Masek, Yuji Matsumoto, Ryan McDonald, Gustavo Mendonca, Anna Missila, Vergniaci Mititelu, Yusuke Miyao, Simoneit Montemagni, Amir More, Laura Moreno Romero, Shunsuke Mori, Bohdan Mosalevskyki, Kadri Muischnek, Ninna Mustafina, Kaili Muurisep, Pinkey Naunwani, Anna Nedoluzhko, Ling Nguyen Th, Huyn Nguyen Th Minh, Vitaly Nikolaev, Rattima Nitsiaroy, Hanus Nurmi, Stina Ojala, Petya Osnova, Lilja Ovrelid, Elena Pascual, Marco Passarotti, Cenel-Augustto Perez, Guy Perrier, Slav Petrov, Jussi Piittulainen, Emily Pitler, Barbara Plank, Martin Popel, Lauma Pretkalnia, Prokopis Prokopidis, Tiina Poulakinen, Sampo Pyysalo, Alexandre Rademaker, Livy Real, Siva Reddy, Georg Rehm, Larissa Rinaldi, Laura Rutumal, Rudolf Rosa, Davide Rovati, Shadi Saleh, Manuela Sanginetiti, Baiba Sauile, Yanin Sawanakunnon, Sebastian Schuster, Djame Seddah, Wolfgang Seeker, Mojan Seraji, Lena Shakurova, Mo Shen, Atsuko Shimada, Muh Shohibussiri, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simko, Maria Simkova, Kiril Simov, Aaron Smith, Antonio Stella, Jana Str nadoova, Alane Suhr, Umut Suluback, Zsolt Szanto, Dima Taji, Takaaki Tanaka, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Zenika Uresova, Larnatitz Uria, Hans Uszkoreit, Gerylan van Noord, Viktor Varga, Veronika Vincze, Jonathan North Washington, Zhuoran Yu, Zdenek Zabotkrtsky, Daniel Zeman, and Hamzi Zhu. 2017a. Universal Dependencies 2.0 - CoNLL 2017 Shared Task Development and Test Data. LIMIT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University.
* Nivre et al. (2017b) Joakim Nivre, Zeljko Agic, Lars Ahrenberg, et al. 2017b. Universal Dependencies 2.0. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University, Prague.
* Nivre and Nilsson (2005) Joakim Nivre and Jens Nilsson. 2005. Pseudo-Projective Dependency Parsing. In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)_, pages 99-106, Ann Arbor, Michigan. Association for Computational Linguistics.
* UDP Barealine Models and Supplementary Materials. LIINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University.
* Straka et al. (2015) Milan Straka, Jan Hajic, Jana Strakova, and Jan Hajic jr. 2015. Parsing Universal Dependency Tree-banks Using Neural Networks and Search-Based Oracle. In _International Workshop on Treebanks and Linguistic Theories (TLTI14)_, pages 208-220.
* Straka and Strakova (2017) Milan Straka and Jana Strakova. 2017. Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPPipe. In _Proceedings of the ConNL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 88-99, Vancouver, Canada. Association for Computational Linguistics.