Comparing State-of-the-art Dependency Parsers on the Italian Stanford Dependency Treebank
Authors: Alberto Lavelli (Fondazione Bruno Kessler, 2016)
Topic: Syntax and Dependency Treebanks

Abstract
In the last decade, many accurate dependency parsers have been made publicly available. It can be difficult for non-experts to select a good off-the-shelf parser among those available. This is even more true when working on languages different from English, because parsers have been tested mainly on English treebanks. Our analysis is focused on Italian and relies on the Italian Stanford Dependency Treebank (ISDT). This work is a contribution to help non-experts understand how difficult it is to apply a specific dependency parser to a new language/treebank and choose a parser that meets their needs.

Introduction
In the last decade, there has been an increasing interest in dependency parsing, witnessed by the organisation of various shared tasks, e.g. AUTHOR, AUTHOR, AUTHOR, AUTHOR. Concerning Italian, there have been tasks on dependency parsing in the first four editions of the EVALITA evaluation campaign AUTHOR. In the 2014 edition, the task on dependency parsing exploited the Italian Stanford Dependency Treebank (ISDT), a treebank featuring an annotation based on Stanford Dependencies AUTHOR. This paper is a follow-up of AUTHOR and reports the experience in applying an updated list of state-of-the-art dependency parsers on ISDT. It can be difficult for non-experts to select a good off-the-shelf parser among those available. This is even more true when working on languages different from English, given that parsers have been tested mainly on English treebanks (and in particular on the WSJ portion of the PennTreebank). This work is a contribution to help practitioners understand how difficult it is to apply a specific dependency parser to a new language/treebank and choose a parser to optimize their desired speed/accuracy trade-off. As in many other NLP fields, there are very few comparative articles where different parsers are directly run by the authors and their performance compared AUTHOR. Most of the papers simply present the results of a newly proposed approach and compare them with the results reported in previous articles. In other cases, the papers are devoted to the application of the same tool to different languages/treebanks. A notable exception is the study reported in AUTHOR, where the authors present a comparative analysis of ten leading statistical dependency parsers on a multi-genre corpus of English. It is important to stress that the comparison presented in this paper concerns tools used more or less out of the box and that the results cannot be used to compare specific characteristics like: parsing algorithms, learning systems, ...

Conclusion
In the paper we have reported on work in progress on the comparison between several state-of-the-art dependency parsers on the Italian Stanford Dependency Treebank (ISDT). We are already working to widen the scope of the comparison including more parsers and to perform an analysis of the results obtained by the different parsers considering not only their performance but also their behaviour in terms of speed, CPU load at training and parsing time, ease of use, licence agreement, \ldots The next step would be to apply the parsers in a multilingual setting, exploiting the availability of treebanks based on Universal Dependencies in many languages AUTHOR}. We thank the authors of the parsers for making them freely available, for kindly answering our questions and for providing useful suggestions. We thank the reviewers for valuable suggestions to improve this article.