Venue: NAACL 2018

# Non-Projective Dependency Parsing with Non-Local Transitions

Daniel Fernandez-Gonzalez

Carlos Gomez-Rodriguez

Universidade da Coruna

FASTPARSE Lab, Lys Research Group, Departamento de Computacion

Campus de Elvina, s/n, 15071 A Coruna, Spain

d.fgonzalez@udc.es, carlos.gomez@udc.es

###### Abstract

We present a novel transition system, based on the Covington non-projective parser, introducing non-local transitions that can directly create arcs involving nodes to the left of the current focus positions. This avoids the need for long sequences of No-Arc transitions to create long-distance arcs, thus alleviating error propagation. The resulting parser outperforms the original version and achieves the best accuracy on the Stanford Dependencies conversion of the Penn Treebank among greedy transition-based parsers.

## 1 Introduction

Greedy transition-based parsers are popular in NLP, as they provide competitive accuracy with high efficiency. They syntactically analyze a sentence by greedily applying transitions, which read it from left to right and produce a dependency tree.

However, this greedy process is prone to error propagation: one wrong choice of transition can lead the parser to an erroneous state, causing more incorrect decisions. This is especially crucial for long attachments requiring a larger number of transitions. In addition, transition-based parsers traditionally focus on only two words of the sentence and their local context to choose the next transition. The lack of a global perspective favors the presence of errors when creating arcs involving multiple transitions. As expected, transition-based parsers build short arcs more accurately than long ones McDonald and Nivre (2007).

Previous research such as Fernandez-Gonzalez and Gomez-Rodriguez (2012) and Qi and Manning (2017) proves that the widely-used projective _arc-eager_ transition-based parser of Nivre (2003) benefits from shortening the length of transition sequences by creating non-local attachments. In particular, they augmented the original transition system with new actions whose behavior entails more than one arc-eager transition and involves a context beyond the traditional two focus words Attardi (2006) and Sartorio et al. (2013) also extended the _arc-standard_ transition-based algorithm Nivre (2004) with the same success.

In the same vein, we present a novel unrestricted non-projective transition system based on the well-known algorithm by Covington (2001) that shortens the transition sequence necessary to parse a given sentence by the original algorithm, which becomes linear instead of quadratic with respect to sentence length. To achieve that, we propose new transitions that affect non-local words and are equivalent to one or more Covington actions, in a similar way to the transitions defined by Qi and Manning (2017) based on the arc-eager parser. Experiments show that this novel variant significantly outperforms the original one in all datasets tested, and achieves the best reported accuracy for a greedy dependency parser on the Stanford Dependencies conversion of the WSJ Penn Treebank.

## 2 Non-Projective Covington Parser

The original non-projective parser defined by Covington (2001) was modelled under the transition-based parsing framework by Nivre (2008). We only sketch this transition system briefly for space reasons, and refer to Nivre (2008) for details.

Parser configurations have the form \(c=\langle\lambda_{1},\lambda_{2},B,A\rangle\), where \(\lambda_{1}\) and \(\lambda_{2}\) are lists of partially processed words, \(B\) a list (called buffer) of unprocessed words, and \(A\) the set of dependency arcs built so far. Given an input string \(w_{1}\cdots w_{n}\), the parser starts at the initial configuration \(c_{s}(w_{1}\ldots w_{n})=\langle[],[],[1\ldots n],\emptyset\rangle\) and runs transitions until a terminal configuration of the

form \(\langle\lambda_{1},\lambda_{2},[],A\rangle\) is reached: at that point, \(A\) contains the dependency graph for the input.1

Footnote 1: Note that, in general, \(A\) is a forest, but it can be converted to a tree by linking headless nodes as dependents of an artificial root node at position 0.

The set of transitions is shown in the top half of Figure 1. Their logic can be summarized as follows: when in a configuration of the form \(\langle\lambda_{1}|i,\lambda_{2},j|B,A\rangle\), the parser has the chance to create a dependency involving words \(i\) and \(j\), which we will call left and right focus words of that configuration. The Left-Arc and Right-Arc transitions are used to create a leftward (\(i\gets j\)) or rightward arc (\(i\to j\)), respectively, between these words, and also move \(i\) from \(\lambda_{1}\) to the first position of \(\lambda_{2}\), effectively moving the focus to \(i-1\) and \(j\). If no dependency is desired between the focus words, the No-Arc transition makes the same modification of \(\lambda_{1}\) and \(\lambda_{2}\), but without building any arc. Finally, the Shift transition moves the whole content of the list \(\lambda_{2}\) plus \(j\) to \(\lambda_{1}\) when no more attachments are pending between \(j\) and the words of \(\lambda_{1}\), thus reading a new input word and placing the focus on \(j\) and \(j+1\). Transitions that create arcs are disallowed in configurations where this would violate the single-head or acyclicity constraints (cycles and nodes with multiple heads are not allowed in the dependency graph). Figure 3 shows the transition sequence in the Covington transition system which derives the dependency graph in Figure 2.

The resulting parser can generate arbitrary non-projective trees, and its complexity is \(O(n^{2})\).

## 3 Non-Projective NL-Covington Parser

The original logic described by Covington (2001) parses a sentence by systematically traversing every pair of words. The Shift transition, introduced by Nivre (2008) in the transition-based version, is an optimization that avoids the need to apply a sequence of No-Arc transitions to empty the list \(\lambda_{1}\) before reading a new input word.

However, there are still situations where sequences of No-Arc transitions are needed. For example, if we are in a configuration \(C\) with focus words \(i\) and \(j\) and the next arc we need to create

Figure 1: Transitions of the non-projective Covington (top) and NL-Covington (bottom) dependency parsers. The notation \(i\rightarrow^{*}j\in A\) means that there is a (possibly empty) directed path from \(i\) to \(j\) in \(A\).

Figure 3: Transition sequence for parsing the sentence in Figure 2 using the Covington parser (LA=Left-Arc, RA=Right-Arc, NA=No-Arc, SH=Shift).

Figure 2: Dependency tree for an input sentence.

 

goes from \(j\) to \(i-k~{}(k>1)\), then we will need \(k-1\) consecutive No-Arc transitions to move the left focus word to \(i\) and then apply Left-Arc. This could be avoided if a non-local Left-Arc transition could be undertaken directly at \(C\), creating the required arc and moving \(k\) words to \(\lambda_{2}\) at once. The advantage of such approach would be twofold: (1) less risk of making a mistake at \(C\) due to considering a limited local context, and (2) shorter transition sequence, alleviating error propagation.

We present a novel transition system called _NL-Covington_ (for "non-local Covington"), described in the bottom half of Figure 1. It consists in a modification of the non-projective Covington algorithm where: (1) the Left-Arc and Right-Arc transitions are parameterized with \(k\), allowing the immediate creation of any attachment between \(j\) and the \(k\)th leftmost word in \(\lambda_{1}\) and moving \(k\) words to \(\lambda_{2}\) at once, and (2) the No-Arc transition is removed since it is no longer necessary.

This new transition system can use some restricted global information to build non-local dependencies and, consequently, reduce the number of transitions needed to parse the input. For instance, as presented in Figure 4, the NL-Covington parser will need 9 transitions, instead of 12 traditional Covington actions, to analyze the sentence in Figure 2.

In fact, while in the standard Covington algorithm a transition sequence for a sentence of length \(n\) has length \(O(n^{2})\) in the worst case (if all nodes are connected to the first node, then we need to traverse every node to the left of each right focus word); for NL-Covington the sequence length is always \(O(n)\): one Shift transition for each of the \(n\) words, plus one arc-building transition for each of the \(n-1\) arcs in the dependency tree. Note, however, that this does not affect the parser's time complexity, which is still quadratic as in the original Covington parser. This is because the algorithm has \(O(n)\) possible transitions to be scored at each configuration, while the original Covington has \(O(1)\) transitions due to being limited to creating local leftward/rightward arcs between the focus words.

The completeness and soundness of NL-Covington can easily be proved as there is a mapping between transition sequences of both parsers, where a sequence of \(k-1\)No-Arc and one arc transition in Covington is equivalent to a Left-Arc or Right-Arc\({}_{k}\) in NL-Covington.

## 4 Experiments

### Data and Evaluation

We use 9 datasets2 from the CoNLL-X Buchholz and Marsi (2006) and all datasets from the CoNLL-XI shared task Nivre et al. (2007). To compare our system to the current state-of-the-art transition-based parsers, we also evaluate it on the Stanford Dependencies de Marneffe and Manning (2008) conversion (using the Stanford parser v3.3.0)3 of the WSJ Penn Treebank Marcus et al. (1993), hereinafter PT-SD, with standard splits. Labelled and Unlabelled Attachment Scores (LAS and UAS) are computed excluding punctuation only on the PT-SD, for comparability. We repeat each experiment with three independent random initializations and report the average accuracy. Statistical significance is assessed by a paired test with 10,000 bootstrap samples.

Footnote 2: We excluded the languages from CoNLL-X that also appeared in CoNLL-XI, i.e., if a language was present in both shared tasks, we used the latest version.

Footnote 3: [https://nlp.stanford.edu/software/lex-parser.shtml](https://nlp.stanford.edu/software/lex-parser.shtml)

### Model

To implement our approach we take advantage of the model architecture described in Qi and Manning (2017) for the _arc-swift_ parser, which extends the architecture of Kiperwasser and Goldberg (2016) by applying a biaffine combination during the featurization process. We implement both the Covington and NL-Covington parsers under this architecture, adapt the featurization process with biaffine combination of Qi and Manning (2017) to these parsers, and use their same training

Figure 4: Transition sequence for parsing the sentence in Figure 2 using the NL-Covington parser (LA=Left-Arc, RA=Right-Arc, SH=Shift).

 

setup. More details about these model parameters are provided in Appendix A.

Since this architecture uses batch training, we train with a static oracle. The NL-Covington algorithm has no spurious ambiguity at all, so there is only one possible static oracle: canonical transition sequences are generated by choosing the transition that builds the shortest pending gold arc involving the current right focus word \(j\), or Shift if there are no unbuilt gold arcs involving \(j\).

We note that a dynamic oracle can be obtained for the NL-Covington parser by adapting the one for standard Covington of Gomez-Rodriguez and Fernandez-Gonzalez (2015). As NL-Covington transitions are concatenations of Covington ones, their loss calculation algorithm is compatible with NL-Covington. Apart from error exploration, this also opens the way to incorporating nonmonotonicity Fernandez-Gonzalez and Gomez-Rodriguez (2017). While these approaches have shown to improve accuracy under online training settings, here we prioritize homogeneous comparability to Qi and Manning (2017), so we use batch training and a static oracle, and still obtain state-of-the-art accuracy for a greedy parser.

### Results

Table 1 presents a comparison between the Covington parser and the novel variant developed here. The NL-Covington parser outperforms the original version in all datasets tested, with all improvements statistically significant (\(\alpha=.05\)).

Table 2 compares our novel system with other state-of-the-art transition-based dependency parsers on the PT-SD. Greedy parsers are in the first block, beam-search and dynamic programming parsers in the second block. The third block shows the best result on this benchmark, obtained with constituent parsing with generative re-ranking and conversion to dependencies. Despite being the only non-projective parser tested on a practically projective dataset,4 our parser achieves the highest score among greedy transition-based models (even above those trained with a dynamic oracle).

Footnote 4: Only 41 out of 39,832 sentences of the PT-SD training dataset present some kind of non-projectivity.

We even slightly outperform the arc-swift system of Qi and Manning (2017), with the same model architecture, implementation and training setup, but based on the projective arc-eager transition-based parser instead. This may be because our system takes into consideration any permissible attachment between the focus word \(j\) and any word in \(\lambda_{1}\) at each configuration, while their approach is limited by the arc-eager logic: it al

\begin{table}
\begin{tabular}{l|c c|c c}  & \multicolumn{2}{c}{Covington} & \multicolumn{2}{c}{NL-Covington} \\ Language & UAS & LAS & UAS & LAS \\ \hline Arabic & 66.67 & 53.24 & **68.69** & **54.59** \\ Basque & 74.31 & 66.18 & **75.45** & **67.61** \\ Catalan & 91.93 & 86.12 & **92.60** & **86.99** \\ Chinese & 83.87 & 76.19 & **85.25** & **77.56** \\ Czech & 84.27 & 77.91 & **86.26** & **79.95** \\ English & 89.94 & 88.74 & **91.51** & **90.47** \\ Greek & 79.91 & 72.65 & **80.61** & **73.41** \\ Hungarian & 76.80 & 65.21 & **78.57** & **67.51** \\ Italian & 82.03 & 75.87 & **83.63** & **78.03** \\ Turkish & 80.29 & 70.68 & **81.30** & **71.28** \\ \hline Bulgarian & 81.78 & 76.23 & **83.65** & **78.40** \\ Danish & 86.56 & 81.18 & **88.40** & **82.77** \\ Dutch & 86.19 & 82.24 & **87.45** & **83.76** \\ German & 85.72 & 82.28 & **87.24** & **83.92** \\ Japanese & 92.20 & 90.41 & **93.63** & **91.65** \\ Portuguese & 86.69 & 82.19 & **87.89** & **83.69** \\ Slovene & 76.07 & 66.81 & **77.83** & **69.74** \\ Spanish & 74.67 & 69.41 & **76.58** & **71.60** \\ Swedish & 74.65 & 64.67 & **75.62** & **65.95** \\ \hline Average & 81.82 & 75.17 & **83.27** & **76.78** \\ \end{tabular}
\end{table}
Table 1: Parsing accuracy (UAS and LAS, including punctuation) of the Covington and NL-Covington non-projective parsers on CoNLL-XI (first block) and CoNLL-X (second block) data-sets. Best results for each language are shown in bold. All improvements in this table are statistically significant (\(\alpha=.05\)).

\begin{table}
\begin{tabular}{l l l l} \multirow{2}{*}{Parser} & \multicolumn{2}{c}{Type} & UAS & LAS \\ \cline{2-4} Chen and Manning (2014) & gs & 91.8 & 89.6 \\ Dyer et al. (2015) & gs & 93.1 & 90.9 \\ Weiss et al. (2015) greedy & gs & 93.2 & 91.2 \\ Ballesteros et al. (2016) & gd & 93.5 & 91.4 \\ Kiperwasser and Goldberg (2016) & gd & 93.9 & 91.9 \\Qi and Manning (2017) & gs & 94.3 & 92.2 \\
* \\ \hline Kumeroo et al. (2017) (constit.) & c & 95.8 & 94.6 \\ \end{tabular}
\end{table}
Table 2: Accuracy comparison of state-of-the-art transition-based dependency parsers on PT-SD. The “Type” column shows the type of parser: _gs_ is a greedy parser trained with a static oracle, _gd_ a greedy parser trained with a dynamic oracle, _b(n)_ a beam search parser with beam size \(n\), _dp_ a parser that employs global training with dynamic programming, and \(c\) a constituent parser with conversation to dependencies.
 

lows all possible rightward arcs (possibly fewer than our approach as the arc-eager stack usually contains a small number of words), but only one leftward arc is permitted per parser state. It is also worth noting that the arc-swift and NL-Covington parsers have the same worst-case time complexity, (\(O(n^{2})\)), as adding non-local arc transitions to the arc-eager parser increases its complexity from linear to quadratic, but it does not affect the complexity of the Covington algorithm. Thus, it can be argued that this technique is better suited to Covington than to arc-eager parsing.

We also compare NL-Covington to the arc-swift parser on the CoNLL datasets (Table 3). For fairness of comparison, we projectivize (via maltparser5) all training datasets, instead of filtering non-projective sentences, as some of the languages are significantly non-projective. Even doing that, the NL-Covington parser improves over the arc-swift system in terms of UAS in 14 out of 19 datasets, obtaining statistically significant improvements in accuracy on 7 of them, and statistically significant decreases in just one.

Footnote 5: [http://www.maltparser.org/](http://www.maltparser.org/)

Finally, we analyze how our approach reduces the length of the transition sequence consumed by the original Covington parser. In Table 4 we report the transition sequence length per sentence used by the Covington and the NL-Covington algorithms to analyze each dataset from the same benchmark used for evaluating parsing accuracy. As seen in the table, NL-Covington produces notably shorter transition sequences than Covington, with a reduction close to 50% on average.

## 5 Conclusion

We present a novel variant of the non-projective Covington transition-based parser by incorporating non-local transitions, reducing the length of transition sequences from \(O(n^{2})\) to \(O(n)\). This system clearly outperforms the original Covington parser and achieves the highest accuracy on the WSJ Penn Treebank (Stanford Dependencies) obtained to date with greedy dependency parsing.

#### Acknowledgments

This work has received funding from the European Research Council (ERC), under the European Union's Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150), from the TELEPARES-UDC (FFI2014-51978-C2-2-R) and ANSWER-ASAP (TIN2017-85160-C2-1-R) projects from MINECO, and from Xunta de Galicia (ED431B 2017/01).

\begin{table}
\begin{tabular}{l|c|r}  & \multicolumn{1}{c}{Covington} & \multicolumn{1}{c}{NL-Covington} \\ Language & trans./sent. & trans./sent. \\ \hline Arabic & 194.80 & **78.22** \\ Basque & 46.74 & **30.13** \\ Catalan & 117.35 & **60.07** \\ Chinese & 19.12 & **14.95** \\ Czech & 60.62 & **33.03** \\ English & 78.01 & **46.75** \\ Greek & 89.23 & **48.77** \\ Hungarian & 68.54 & **37.66** \\ Italian & 63.67 & **40.93** \\ Turkish & 53.53 & **30.08** \\ \hline Bulgarian & 51.35 & **29.81** \\ Danish & 66.77 & **36.34** \\ Dutch & 42.78 & **28.93** \\ German & 61.16 & **31.89** \\ Japanese & 24.30 & **16.11** \\ Portuguese & 76.14 & **40.74** \\ Slovene & 56.15 & **31.79** \\ Spanish & 109.70 & **55.28** \\ Swedish & 48.59 & **29.07** \\ \hline PTB-SD & 81.65 & **46.92** \\ \hline Average & 70.51 & **38.37** \\ \end{tabular}
\end{table}
Table 4: Average transitions executed per sentence (trans./sent.) when analyzing each dataset by the original Covington and NL-Covington algorithms.

\begin{table}
\begin{tabular}{l|c c c c}  & \multicolumn{2}{c}{Arc-swift} & \multicolumn{2}{c}{NL-Covington} \\ Language & UAS & LAS & UAS & LAS \\ \hline Arabic & 67.54 & 53.65 & **68.69*** & **54.59*** \\ Basque & 74.88 & 67.44 & **75.45** & **67.61** \\ Catalan & **92.98** & **87.51*** & 92.60 & 86.99 \\ Chinese & 84.96 & 77.34 & **85.25** & **77.56** \\ Czech & 85.92 & 79.82 & **86.26** & **79.95** \\ English & 91.41 & 90.43 & **91.51** & **90.47** \\ Greek & **81.64* & **74.56* & 80.61 & 73.41 \\ Hungarian & **78.70** & **69.27*** & 78.57 & 67.51 \\ Italian & 83.29 & **78.60*** & **83.63** & **78.03** \\ Turkish & 79.56 & 70.22 & **81.30*** & **71.28*** \\ \hline Bulgarian & 83.28 & 78.19 & **83.65** & **78.40** \\ Danish & 87.86 & 82.58 & **88.40*** & **82.77** \\ Dutch & 83.27 & 80.14 & **87.45*** & **83.76*** \\German & 86.28 & 82.97 & **87.24*** & **83.92*** \\ Japanese & **93.64** & **91.92** & 93.63 & 91.65 \\ Portuguese & 87.01 & 83.09 & **87.89*** & **83.69*** \\ Slovene & **77.89** & 69.37 & 77.83 & **69.74** \\ Spanish & 75.55 & 70.62 & **76.58*** & **71.60*** \\ Swedish & 75.00 & 65.66 & **75.62** & **65.95** \\ \hline Average & 82.67 & 76.49 & **83.27** & **76.78** \\ \hline \end{tabular}
\end{table}
Table 3: Parsing accuracy (UAS and LAS, with punctuation) of the arc-swift and NL-Covington parsers on CoNLL-XI (1st block) and CoNLL-X (2nd block) datasets. Best results for each language are in bold. * indicates statistically significant improvements (\(\alpha=.05\)).

 



## References

* Alberti et al. (2015) Chris Alberti, David Weiss, Greg Coppola, and Slav Petrov. 2015. Improved transition-based parsing and tagging with neural networks. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_. pages 1354-1359. [http://aclweb.org/anthology/D15/D15-1159.pdf](http://aclweb.org/anthology/D15/D15-1159.pdf).
* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. [http://aclweb.org/anthology/P7/P16/P16-1231.pdf](http://aclweb.org/anthology/P7/P16/P16-1231.pdf).
* Attardi (2006) Giuseppe Attardi. 2006. Experiments with a multilanguage non-projective dependency parser. In _Proceedings of the 10th Conference on Computational Natural Language Learning (CoNLL)_. pages 166-170.
* Ballesteros et al. (2016) Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A. Smith. 2016. Training with exploration improves a greedy stack-lstm parser. _CoRR_ abs/1603.03793. [http://arxiv.org/abs/1603.03793](http://arxiv.org/abs/1603.03793).
* Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. _arXiv preprint arXiv:1607.04606_ .
* Buchholz and Marsi (2006) Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In _Proceedings of the 10th Conference on Computational Natural Language Learning (CoNLL)_. pages 149-164. [http://www.aclweb.org/anthology/W06-2920](http://www.aclweb.org/anthology/W06-2920).
* Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Doha, Qatar, pages 740-750. [http://www.aclweb.org/anthology/D14-1082](http://www.aclweb.org/anthology/D14-1082).
* Covington (2001) Michael A. Covington. 2001. A fundamental algorithm for dependency parsing. In _Proceedings of the 39th Annual ACM Southeast Conference_. ACM, New York, NY, USA, pages 95-102.
* de Marneffe and Manning (2008) Marie-Catherine de Marneffe and Christopher D. Manning. 2008. The stanford typed dependencies representation. In _Coling 2008: Proceedings of the Workshop on Cross-Framework and Cross-Domain Parser Evaluation_. Association for Computational Linguistics, Stroudsburg, PA, USA, CrossParser '08, pages 1-8. [http://dl.acm.org/citation.cfm?id=1608858.1608859](http://dl.acm.org/citation.cfm?id=1608858.1608859).
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers_. pages 334-343. [http://aclweb.org/anthology/P7/P15/P15-1033.pdf](http://aclweb.org/anthology/P7/P15/P15-1033.pdf).
* Fernandez-Gonzalez and Gomez-Rodriguez (2012) Daniel Fernandez-Gonzalez and Carlos Gomez-Rodriguez. 2012. Improving transition-based dependency parsing with buffer transitions. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. Association for Computational Linguistics, pages 308-319. [http://aclweb.org/anthology/D/D12/D12-1029.pdf](http://aclweb.org/anthology/D/D12/D12-1029.pdf).
* Fernandez-Gonzalez and Gomez-Rodriguez (2017) Daniel Fernandez-Gonzalez and Carlos Gomez-Rodriguez. 2017. A full non-monotonic transition system for unrestricted non-projective parsing. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, pages 288-298. [http://aclweb.org/anthology/P17-1027](http://aclweb.org/anthology/P17-1027).
* Gomez-Rodriguez and Fernandez-Gonzalez (2015) Carlos Gomez-Rodriguez and Daniel Fernandez-Gonzalez. 2015. An efficient dynamic oracle for unrestricted non-projective parsing. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015). Volume 2: Short Papers_. Association for Computational Linguistics, Beijing, China, pages 256-261. [http://www.aclweb.org/anthology/P15-2042](http://www.aclweb.org/anthology/P15-2042).
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. _Neural Networks_ pages 5-6.
* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _TACL_ 4:313-327. [https://transal.org/ojs/index.php/tacl/article/view/885](https://transal.org/ojs/index.php/tacl/article/view/885).
* Kuncoro et al. (2017) Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. Association for Computational Linguistics, pages 1249-1258. [http://aclanthology.coli.uni-saarland.de/pdf/E/E17/E17-1117.pdf](http://aclanthology.coli.uni-saarland.de/pdf/E/E17/E17-1117.pdf).
* Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated

corpus of English: The Penn Treebank. _Computational Linguistics_ 19:313-330.
* McDonald and Nivre (2007) Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_. pages 122-131.
* Nivre (2003) Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In _Proceedings of the 8th International Workshop on Parsing Technologies (IWPT 03)_. ACL/SIGPARSE, pages 149-160.
* Nivre (2004) Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In _Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together (ACL)_. pages 50-57.
* Nivre (2008) Joakim Nivre. 2008. Algorithms for Deterministic Incremental Dependency Parsing. _Computational Linguistics_ 34(4):513-553. [https://doi.org/10.1162/coli.07-056-R1-07-027](https://doi.org/10.1162/coli.07-056-R1-07-027).
* Nivre et al. (2007) Joakim Nivre, Johan Hall, Sandra Kubler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. The CoNLL 2007 shared task on dependency parsing. In _Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007_. pages 915-932. [http://www.aclweb.org/anthology/D/D07/D07-1096.pdf](http://www.aclweb.org/anthology/D/D07/D07-1096.pdf).
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_. pages 1532-1543. [http://www.aclweb.org/anthology/D14-1162](http://www.aclweb.org/anthology/D14-1162).
* August 4, Volume 2: Short Papers_. pages 110-117. [https://doi.org/10.18653/v1/P17-2018](https://doi.org/10.18653/v1/P17-2018).
* Sartorio et al. (2013) Francesco Sartorio, Giorgio Satta, and Joakim Nivre. 2013. A transition-based dependency parser using a dynamic parsing strategy. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 135-144. [http://aclanthology.coli.uni-saarland.de/pdf/P/P13/P13-1014.pdf](http://aclanthology.coli.uni-saarland.de/pdf/P/P13/P13-1014.pdf).
* Shi et al. (2017) Tianze Shi, Liang Huang, and Lillian Lee. 2017. Fast(er) exact decoding and global training for transition-based dependency parsing via a minimal feature set. _CoRR_ abs/1708.09403. [http://arxiv.org/abs/1708.09403](http://arxiv.org/abs/1708.09403).
* Weiss et al. (2015) David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers_. pages 323-333. [http://aclweb.org/anthology/P/P15/P15-1032.pdf](http://aclweb.org/anthology/P/P15/P15-1032.pdf).

## Appendix A Model Details

We provide more details of the neural network architecture used in this paper, which is taken from Qi and Manning (2017).

The model consists of two blocks of 2-layered bidirectional long short-term memory (BiLSTM) networks Graves and Schmidhuber (2005) with 400 hidden units in each direction. The first block is used for POS tagging and the second one, for parsing. As the input of the tagging block, we use words represented as word embeddings, and BiLTSTMs are employed to perform feature extraction. The resulting output is fed into a multi-layer perceptron (MLP), with a hidden layer of 100 rectified linear units (ReLU), that provides a POS tag for each input token in a 32-dimensional representation. Word embeddings concatenated to these POS tag embeddings serve as input of the second block of BiLSTMs to undertake the parsing stage. Then, the output of the parsing block is fed into a MLP with two separate ReLU hidden layers (one for deriving the representation of the head, and the other for the dependency label) that, after being merged and by means of a softmax function, score all the feasible transitions, allowing to greedily choose and apply the highest-scoring one.

Moreover, we adapt the featurization process with biaffine combination described in Qi and Manning (2017) for the arc-swift system to be used on the original Covington and NL-Covington parsers. In particular, arc transitions are featurized by the concatenation of the representation of the head and dependent words of the arc to be created, the No-Arc transition is featurized by the rightmost word in \(\lambda_{1}\) and the leftmost word in the buffer \(B\) and, finally, for the Shift transition only the leftmost word in \(B\) is used. Unlike Qi and Manning (2017) do for baseline parsers, we do not use the featurization method detailed in Kiperwasser and Goldberg (2016)6 for the original Covington parser, as we observed that this results in lower 

scores and then the comparison would be unfair in our case. We implement both systems under the same framework, with the original Covington parser represented as the NL-Covington system plus the No-Arc transition and with \(k\) limited to 1. A thorough description of the model architecture and featurization mechanism can be found in Qi and Manning (2017).

Our training setup is exactly the same used by Qi and Manning (2017), training the models during 10 epochs for large datasets and 30 for small ones. In addition, we initialize word embeddings with 100-dimensional GloVe vectors (Pennington et al., 2014) for English and use 300-dimensional Facebook vectors (Bojanowski et al., 2016) for other languages. The other parameters of the neural network keep the same values.

The parser's source code is freely available at [https://github.com/danifg/Non-Local-Covington](https://github.com/danifg/Non-Local-Covington)