Venue: ACL 2017

# Arc-swift: A Novel Transition System for Dependency Parsing

Peng Qi Christopher D. Manning

Computer Science Department

Stanford University

{pengqi, manning}@cs.stanford.edu

###### Abstract

Transition-based dependency parsers often need sequences of local shift and reduce operations to produce certain attachments. Correct individual decisions hence require global information about the sentence context and mistakes cause error propagation. This paper proposes a novel transition system, _arc-swift_, that enables direct attachments between tokens farther apart with a single transition. This allows the parser to leverage lexical information more directly in transition decisions. Hence, _arc-swift_ can achieve significantly better performance with a very small beam size. Our parsers reduce error by 3.7-7.6% relative to those using existing transition systems on the Penn Treebank dependency parsing task and English Universal Dependencies.

+
Footnote †: _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers)_, pages 110–117 Vancouver, Canada, July 30 - August 4, 2017. ©2017 Association for Computational Linguistics [https://doi.org/10.18653/v1/P17-2018](https://doi.org/10.18653/v1/P17-2018)

structions. We show that _arc-swift_ parsers reduce errors in attaching prepositional phrases and conjunctions compared to parsers using existing transition systems.

## 2 Transition-based Dependency Parsing

Transition-based dependency parsing is performed by predicting transitions between states (see Figure 1 for an example). Parser states are usually written as \((\sigma|i,j|\beta,A)\), where \(\sigma|i\) denotes the _stack_ with token \(i\) on the top, \(j|\beta\) denotes the _buffer_ with token \(j\) at its leftmost, and \(A\) the set of dependency arcs. Given a state, the goal of a dependency parser is to predict a _transition_ to a new state that would lead to the correct parse. A _transition system_ defines a set of transitions that are sound and complete for parsers, that is, every transition sequence would derive a well-formed parse tree, and every possible parse tree can also be derived from some transition sequence.1

Footnote 1: We only focus on projective parses for the scope of this paper.

_Arc-standard_[35] is one of the first transition systems proposed for dependency parsing. It defines three transitions: shift, left arc (LArc), and right arc (RArc) (see Figure 2 for definitions, same for the following transition systems), where all arc-inducing transitions operate on the stack. This system builds the parse bottom-up, i.e., a constituent is only attached to its head after it has received all of its dependents. A potential drawback is that during parsing, it is difficult to predict if a constituent has consumed all of its right dependents. _Arc-eager_[35] remedies this drawback by defining arc-inducing transitions that operate between the stack and the buffer. As a result, a constituent no longer needs to be complete before it can be attached to its head to the left, as a right arc doesn't prevent the attached dependent from taking further dependents of its own.2[40] propose a hybrid system derived from a tabular parsing scheme, which they have shown both _arc-standard_ and _arc-eager_ can be derived from. _Arc-hybrid_ combines LArc from _arc-eager_ and RArc from _arc-standard_ to build dependencies bottom-up.

Footnote 2: A side-effect of _arc-eager_ is that there is sometimes _spurious ambiguity_ between Shift and Reduce transitions. For the example in Figure 1, the first Reduce can be inserted before the third Shift without changing the correctness of the resulting parse, i.e., both are feasible at that time.

## 3 Non-local Transitions with _arc-swift_

The traditional transition systems discussed in Section 2 only allow very local transitions affecting one or two words, which makes long-distance dependencies difficult to predict. To illustrate the limitation of local transitions, consider parsing the following sentences:

_I__ate fish with ketchup._

_I__ate fish with chopsticks._

The two sentences have almost identical structures, with the notable difference that the prepositional phrase is complementing the direct object in the first case, and the main verb in the second.

For _arc-standard_ and _arc-hybrid_, the parser would have to decide between Shift and RArc when the parser state is as shown in Figure 2(a), where \(\star\) stands for either "_ketchp_" or "_chopistics_".3 Similarly, an _arc-eager_ parser would deal with the state shown in Figure 2(b). Making the correct transition requires information about context words "_ate_" and "_fish_", as well as "\(\star\)".

Figure 2: Transitions defined by different transition systems.

 

Parsers employing traditional transition systems would usually incorporate more features about the context in the transition decision, or employ beam search during parsing (Chen and Manning, 2014; Andor et al., 2016).

In contrast, inspired by graph-based parsers, we propose _arc-swift_, which defines non-local transitions as shown in Figure 2. This allows direct comparison of different attachment points, and provides a direct solution to parsing the two example sentences. When the _arc-swift_ parser encounters a state identical to Figure 3b, it could directly compare transitions RArc[1] and RArc[2] instead of evaluating between local transitions. This results in a direct attachment much like that in a graph-based parser, informed by lexical information about affinity of the pairs of words.

_Arc-swift_ also bears much resemblance to _arceager_. In fact, an LArc\([k]\) transition can be viewed as \(k-1\) Reduce operations followed by one LArc in _arc-eager_, and similarly for RArc\([k]\). Reduce is no longer needed in _arc-swift_ as it becomes part of LArc\([k]\) and RArc\([k]\), removing the ambiguity in derived transitions in _arc-eager_. _arc-swift_ is also equivalent to _arc-eager_ in terms of soundness and completeness.4 A caveat is that the worst-case time complexity of _arc-swift_ is \(O(n^{2})\) instead of \(O(n)\), which existing transition-based parsers enjoy. However, in practice the runtime is nearly linear, thanks to the usually small number of reducible tokens in the stack.

Footnote 4: This is easy to show because in _arc-eager_, all Reduce transitions can be viewed as preparing for a later LArc or RArc transition. We also note that similar to _arc-eager_ transitions, _arc-swift_ transitions must also satisfy certain pre-conditions. Specifically, an RArc\([k]\) transition requires that the top \(k-1\) elements in the stack are already attached; LArc\([k]\) additionally requires that the \(k\)-th element is unattached, resulting in no more than one feasible LArc candidate for any parser state.

## 4 Experiments

### Data and Model

We use the Wall Street Journal portion of Penn Treebank with standard parsing splits (PTB-SD), along with Universal Dependencies v1.3 (Nivre et al., 2016) (EN-UD). PTB-SD is converted to Stanford Dependencies De Marneffe and Manning, 2008) with CoreNLP 3.3.0 (Manning et al., 2014) following previous work. We report labelled and unlabelled attachment scores (LAS/UAS), removing punctuation from all evaluations.

Our model is very similar to that of Kiperwasser and Goldberg, 2016), where features are extracted from tokens with bidirectional LSTMs, and concatenated for classification. For the three traditional transition systems, features of the top 3 tokens on the stack and the leftmost token in the buffer are concatenated as classifier input. For _arc-swift_, features of the head and dependent tokens for each arc-inducing transition are concatenated to compute scores for classification, and features of the leftmost buffer token is used for Shift. For other details we defer to Appendix A. The full specification of the model can also be found in our released code online at [https://github.com/qipeng/arc-swift](https://github.com/qipeng/arc-swift).

### Results

We use static oracles for all transition systems, and for _arc-eager_ we implement oracles that always Shift/Reduce when ambiguity is present (_arc-eager_-_S/R_). We evaluate our parsers with greedy parsing (i.e., beam size 1). The results are shown in Table 1.5 Note that K&G 2016 is trained with a dynamic oracle (Goldberg and Nivre, 2012), Andor 2016 with a CRF-like loss, and both Andor 2016 and Weiss 2015 employed beam search (with sizes 32 and 8, respectively).

Footnote 5: In the interest of space, we abbreviate all transition systems (TS) as follows in tables: _asw_ for _arc-swift_, _asd_ for _arc-standard_, _ae_S/R_ for _arc-eager_-_S/R_, and _ah_ for _arc-hybrid_.

For each pair of the systems we implemented, we studied the statistical significance of their difference by performing a paired test with 10,000 bootstrap samples on PTB-SD. The resulting p-values are analyzed with a 10-group Bonferroni-Holm test, with results shown in Table 2. We note

Figure 3: Parser states that present difficult transition decisions in traditional systems. In these states, parsers would need to incorporate context about “ate”, “fish”, and “\(\star\)” to make the correct local transition.

 

that with almost the same implementation, _arc-swift_ parsers significantly outperform those using traditional transition systems. We also analyzed the performance of parsers on attachments of different distances. As shown in Figure 4, _arc-swift_ is equally accurate as existing systems for short dependencies, but is more robust for longer ones.

While _arc-swift_ introduces direct long-distance transitions, it also shortens the overall sequence necessary to induce the same parse. A parser could potentially benefit from both factors: direct attachments could make an easier classification task, and shorter sequences limit the effect of error propagation. However, since the two effects are correlated in a transition system, precise attribution of the gain is out of the scope of this paper.

Computational efficiency.We study the computational efficiency of the _arc-swift_ parser by comparing it to an _arc-eager_ parser. On the PTB-SD development set, the average transition sequence length per sentence of _arc-swift_ is 77.5% of that of _arc-eager_. At each step of parsing, _arc-swift_ needs to evaluate only about 1.24 times the number of transition candidates as _arc-eager_, which results in very similar runtime. In contrast, beam search with beam size 2 for _arc-eager_ requires evaluating 4 times the number of transition candidates compared to greedy parsing, which results in a UAS 0.14% worse and LAS 0.22% worse for _arc-eager_ compared to greedily decoded _arc-swift_.

### Linguistic Analysis

We automatically extracted all labelled attachment errors by error type (incorrect attachment or relation), and categorized a few top parser errors by hand into linguistic constructions. Results on PTB-SD are shown in Table 3.7 We note that the _arc-swift_ parser improves accuracy on prepositional phrase (PP) and conjunction attachments, while it remains comparable to other parsers on other common errors. Analysis on EN-UD shows a similar trend. As shown in the table, there are still many parser errors unaccounted for in our analysis. We leave this to future work.

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline  & \multicolumn{2}{c}{_aeS_} & \multicolumn{1}{c}{_asd_} & \multicolumn{1}{c}{_ah_} & \multicolumn{1}{c}{_aeR_} \\ \hline _asw_ & ***/** & ***/*** & ***/*** & ***/*** \\ _aaS_ & & -/- & -/- & */- \\ _asd_ & & & -/- & */- \\ _ah_ & & & & -/- \\ \hline \hline \end{tabular}
\end{table}
Table 2: Significance test for transition systems. Each grid shows adjusted test result for UAS and LAS, respectively, showing whether the system on the row is significantly better than that on the column. “***” stands for \(p<0.001\), “*’\(~{}p<0.01\), “*”\(~{}p<0.05\), and “-”\(~{}p\geq 0.05\).

Figure 4: Parser attachment errors on PTB-SD binned by the length of the gold dependency.

 



## 5 Related Work

Previous work has also explored augmenting transition systems to facilitate longer-range attachments. Attardi (2006) extended the _arc-standard_ system for non-projective parsing, with arc-inducing transitions that are very similar to those in _arc-swift_. A notable difference is that their transitions retain tokens between the head and dependent. Fernandez-Gonzalez and Gomez-Rodriguez (2012) augmented the _arc-eager_ system with transitions that operate on the buffer, which shorten the transition sequence by reducing the number of Shift transitions needed. However, limited by the sparse feature-based classifiers used, both of these parsers just mentioned only allow direct attachments of distance up to 3 and 2, respectively. More recently, Sartorio et al. (2013) extended _arc-standard_ with transitions that directly attach to left and right "spines" of the top two nodes in the stack. While this work shares very similar motivations as _arc-swift_, it requires additional data structures to keep track of the left and right spines of nodes. This transition system also introduces _spurious ambiguity_ where multiple transition sequences could lead to the same correct parse, which necessitates easy-first training to achieve a more noticeable improvement over _arc-standard_. In contrast, _arc-swift_ can be easily implemented given the parser state alone, and does not give rise to spurious ambiguity.

For a comprehensive study of transition systems for dependency parsing, we refer the reader to Bohnet et al. (2016), which proposed a generalized framework that could derive all of the traditional transition systems we described by configuring the size of the active token set and the maximum arc length, among other control parameters. However, this framework does not cover _arc-swift_ in its original form, as the authors limit each of their transitions to reduce at most one token from the active token set (the buffer). On the other hand, the framework presented in Gomez-Rodriguez and Nivre (2013) does not explicitly make this constraint, and therefore generalizes to _arc-swift_. However, we note that _arc-swift_ still falls out of the scope of existing discussions in that work, by introducing multiple Reduces in a single transition.

## 6 Conclusion

In this paper, we introduced _arc-swift_, a novel transition system for dependency parsing. We also performed linguistic analyses on parser outputs and showed _arc-swift_ parsers reduce errors in conjunction and adverbial attachments compared to parsers using traditional transition systems.

## Acknowledgments

We thank Timothy Dozat, Arun Chaganty, Danqi Chen, and the anonymous reviewers for helpful discussions. Stanford University gratefully acknowledges the support of the Defense Advanced Research Projects Agency (DARPA) Deep Exploration and Filtering of Text (DEFT) Program under Air Force Research Laboratory (AFRL) contract No. FA8750-13-2-0040. Any opinions, findings, and conclusion or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the DARPA, AFRL, or the US government.

## References

* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. [https://www.aclweb.org/anthology/P16-1231](https://www.aclweb.org/anthology/P16-1231).
* Angeli et al. (2015) Gabor Angeli, Melvin Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015)_. [http://www.aclweb.org/anthology/P15-1034](http://www.aclweb.org/anthology/P15-1034).
* Attardi (2006) Giuseppe Attardi. 2006. Experiments with a multilanguage non-projective dependency parser. In _Proceedings of the Tenth Conference on Computational Natural Language Learning_. Association for Computational Linguistics, pages 166-170. [http://www.aclweb.org/anthology/W06-2922](http://www.aclweb.org/anthology/W06-2922).
* Altrimer et al. (2016)

\begin{table}
\begin{tabular}{l r r r} \hline \hline  & _asw_ & _aeS_ & _asd_ \\ \hline PP attachment & 545 & 569 & 571 \\ Noun/Adjective confusion & 221 & 230 & 221 \\ Conjunction attachment & 156 & 170 & 164 \\ Adverbial attachment & 123 & 122 & 143 \\ \hline Total Errors & 3884 & 4100 & 4106 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Common parser errors on PTB-SD. The top 4 common errors are categorized and shown in this table. Errors not shown are in a long-tail distribution and warrant analyses in future work.



Bernd Bohnet, Ryan McDonald, Emily Pitler, and Ji Ma. 2016. Generalized transition-based dependency parsing via control parameters. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. volume 1, pages 150-160. [https://www.aclweb.org/anthology/P16-1015](https://www.aclweb.org/anthology/P16-1015).
* Bowman et al. (2016) Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)_. [http://www.aclweb.org/anthology/P16-1139](http://www.aclweb.org/anthology/P16-1139).
* Chen and Manning (2014) Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In _EMNLP_. pages 740-750. [http://www.aclweb.org/anthology/D14-1082](http://www.aclweb.org/anthology/D14-1082).
* De Marneffe and Manning (2008) Marie-Catherine De Marneffe and Christopher D Manning. 2008. The Stanford typed dependencies representation. In _COLING 2008: Proceedings of the Workshop on Cross-framework and Cross-domain Parser Evaluation_. pages 1-8. [http://www.aclweb.org/anthology/W08-1301](http://www.aclweb.org/anthology/W08-1301).
* Fernandez-Gonzalez and Gomez-Rodriguez (2012) Daniel Fernandez-Gonzalez and Carlos Gomez-Rodriguez. 2012. Improving transition-based dependency parsing with buffer transitions. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. pages 308-319. [http://www.aclweb.org/anthology/D12-1029](http://www.aclweb.org/anthology/D12-1029).
* Goldberg and Nivre (2012) Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In _COLING_. pages 959-976. [http://www.aclweb.org/anthology/C12-1059](http://www.aclweb.org/anthology/C12-1059).
* Gomez-Rodriguez and Nivre (2013) Carlos Gomez-Rodriguez and Joakim Nivre. 2013. Divisible transition systems and multiplanar dependency parsing. _Computational Linguistics_ 39(4):799-845.
* Gubbins and Vlachos (2013) Joseph Gubbins and Andreas Vlachos. 2013. Dependency language models for sentence completion. In _EMNLP_. volume 13, pages 1405-1410. [http://www.aclweb.org/anthology/D13-1143](http://www.aclweb.org/anthology/D13-1143).
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_ .
* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional lstm feature representations. _Transactions of the Association for Computational Linguistics (TACL)_[https://aclweb.org/anthology/Q16-1023](https://aclweb.org/anthology/Q16-1023).
* Kuhlmann et al. (2011) Marco Kuhlmann, Carlos Gomez-Rodriguez, and Giorgio Satta. 2011. Dynamic programming algorithms for transition-based dependency parsers. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. pages 673-682. [http://www.aclweb.org/anthology/P11-1068](http://www.aclweb.org/anthology/P11-1068).
* Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Association for Computational Linguistics (ACL) System Demonstrations_. pages 55-60. [http://www.aclweb.org/anthology/P/P14/P14-5010](http://www.aclweb.org/anthology/P/P14/P14-5010).
* McDonald et al. (2005) Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. 2005. Non-projective dependency parsing using spanning tree algorithms. In _Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing_. pages 523-530. [http://www.aclweb.org/anthology/H05-1066](http://www.aclweb.org/anthology/H05-1066).
* McDonald and Nivre (2007) Ryan T McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In _EMNLP-CoNLL_. pages 122-131. [http://www.aclweb.org/anthology/D07-1013](http://www.aclweb.org/anthology/D07-1013).
* Nivre (2003) Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In _Proceedings of the 8th International Workshop on Parsing Technologies_. pages 149-160. [http://step.lingfili.uu.se/](http://step.lingfili.uu.se/) nivre/docs/iwpt03.pdf.
* Nivre (2004) Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In _Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together_. pages 50-57. [https://www.aclweb.org/anthology/W04-0308](https://www.aclweb.org/anthology/W04-0308).
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In _Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016)_. pages 1659-1666.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_. pages 1532-1543. [http://www.aclweb.org/anthology/D14-1162](http://www.aclweb.org/anthology/D14-1162).
* Sartorio et al. (2013) Francesco Sartorio, Giorgio Satta, and Joakim Nivre. 2013. A transition-based dependency parser using a dynamic parsing strategy. In _Asociation for Computational Linguistics_. pages 135-144. [http://www.aclweb.org/anthology/P13-1014](http://www.aclweb.org/anthology/P13-1014).
* Schmitz et al. (2012) Michael Schmitz, Robert Bart, Stephen Soderland, Oren Etzioni, et al. 2012. Open language learning for information extraction. In _Proceedings 

of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. pages 523-534. [https://www.aclweb.org/anthology/D12-1048](https://www.aclweb.org/anthology/D12-1048).
* Weiss et al. (2015) David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015)_. [http://www.aclweb.org/anthology/P15-1032](http://www.aclweb.org/anthology/P15-1032).

## Appendix A Model and Training Details

Our model setup is similar to that of (Kiperwasser and Goldberg, 2016) (See Figure 5). We employ two blocks of bidirectional long short-term memory (BiLSTM) networks (Hochreiter and Schmidhuber, 1997) that share very similar structures, one for part-of-speech (POS) tagging, the other for parsing. Both BiLSTMs have 400 hidden units in each direction, and the output of both are concatenated and fed into a dense layer of rectified linear units (ReLU) before 32-dimensional representations are derived as classification features. As the input to the tagger BiLSTM, we represent words with 100-dimensional word embeddings, initialized with GloVe vectors (Pennington et al., 2014).8 The output distribution of the tagger classifier is used to compute a weighted sum of 32-dimensional POS embeddings, which is then concatenated with the output of the tagger BiLSTM (800-dimensional per token) as the input to the parser BiLSTM. For the parser BiLSTM, we use two separate sets of dense layers to derive a "head" and a "dependent" representation for each token. These representations are later merged according to the parser state to make transition predictions.

Footnote 8: We also kept the vectors of the top 400k words trained on Wikipedia and English Gigaword for a broader coverage of unseen words.

For traditional transition systems, we follow (Kiperwasser and Goldberg, 2016) by featurizing the top 3 tokens on the stack and the leftmost token in the buffer. To derive features for each token, we take its head representation \(v_{\text{head}}\) and dependent representation \(v_{\text{dep}}\), and perform the following biaffine combination

\[\begin{split} v_{\text{feat},i}&=[f(v_{\text{ head}},v_{\text{dep}})]_{i}\\ &=\text{ReLU}\big{(}v^{\top}_{\text{head}}W_{i}v_{\text{dep}}+ b^{\top}_{i}v_{\text{head}}\\ &\hskip 28.452756pt+c^{\top}_{i}v_{\text{dep}}+d_{i}\big{)}\end{split} \tag{1}\]

where \(W_{i}\in\mathbb{R}^{32\times 32}\), \(b_{i},c_{i}\in\mathbb{R}^{32}\), and \(d_{i}\) is a scalar for \(i=1,\dots,32\). The resulting 32-dimensional features are concatenated as the input to a fixed-dimensional softmax classifier for transition decisions.

For _arc-swift_, we featurize for each arch-inducing transition with the same composition function in Equation (1) with \(v_{\text{head}}\) of the head token and \(v_{\text{dep}}\) of the dependent token of the arc to be induced. For Shift, we simply combine \(v_{\text{head}}\) and \(v_{\text{dep}}\) of the leftmost token in the buffer with the biaffine combination, and obtain its score by computing the inner-product of the feature and a vector. At each step, the scores of all feasible transitions are normalized to a probability distribution by a softmax function.

In all of our experiments, the parsers are trained to maximize the log likelihood of the desired transition sequence, along with the tagger being trained to maximize the log likelihood of the correct POS tag for each token.

To train the parsers, we use the Adam optimizer (Kingma and Ba, 2014), with \(\beta_{2}=0.9\), an initial learning rate of 0.001, and minibatches of size 32 sentences. Parsers are trained for 10 passes through the dataset on PTB-SD. We also find that annealing the learning rate by a factor of 0.5 for every pass after the 5th helped improve performance. For EN-UD, we train for 30 passes, and anneal the learning rate for every 3 passes after the 15th due to the smaller size of the dataset. For all of the biaffine combination layers and dense layers, we dropout their units with a small probability of 5%. Also during training time, we randomly replace 10% of the input words by an artificial \(\langle\text{UNK}\rangle\) token, which is then used to replace

Figure 5: Illustration of the model.

 

all unseen words in the development and test sets. Finally, we repeat each experiment with 3 independent random initializations, and use the average result for reporting and statistical significance tests.

The code for the full specification of our models and aforementioned training details are available at [https://github.com/qipeng/arc-swift](https://github.com/qipeng/arc-swift).