Parsing Events: a New Perspective on Old Challenges
Authors: Rachele Sprugnoli, Felice Dell'Orletta, Tommaso Caselli, Simonetta Montemagni, Cristina Bosco (Fondazione Bruno Kessler, CNR-ILC, VU Amsterdam, Università di Torino, 2015)
Topic: Syntax and Dependency Treebanks

Abstract
The paper proposes a new evaluation exercise, meant to shed light on the syntax-semantics interface for the analysis of written Italian and resulting from the combination of the EVALITA 2014 dependency parsing and event extraction tasks. It aims at investigating the cross-fertilization of tasks, generating a new resource combining dependency and event annotations, and devising metrics able to evaluate the applicative impact of the achieved results.

Introduction
Since the '90s, evaluation campaigns organized worldwide have offered to the computational linguistics community the invaluable opportunity of developing, comparing and improving state-of-the-art technologies in a variety of NLP tasks. ACE}, MUC}, CoNLL} and SemEval} are probably the best-known series of evaluation campaigns that covered syntactic and semantic tasks for English as well as for other languages (e.g. Spanish, Arabic, Chinese). For Italian, EVALITA campaigns} have been organized since 2007 around a set of evaluation exercises related to the automatic analysis of both written text and speech. Over the years, many challenging tasks have been proposed %in international campaigns as well as in EVALITA with the aim of advancing state-of-the-art technologies in different NLP areas: to mention only a few, dependency parsing AUTHOR, AUTHOR, textual entailment AUTHOR, frame labeling AUTHOR and cross-document event ordering AUTHOR, all requiring cutting-edge methods and techniques as well as innovative approaches. Following the fact that, in recent years, research is moving from the analysis of grammatical structure to sentence semantics, the attention in evaluation campaigns is shifting towards more complex tasks, combining syntactic parsing with semantically-oriented analysis. %This is the case, for instance, of the SemEval 2014 task ``Broad-Coverage Semantic Dependency Parsing'' % AUTHOR, The interest of composite and articulated tasks built by combining basic tasks also lies at the applicative level, since Information Extraction architectures can realistically be seen as integrating components which carry out distinct basic tasks. Starting from the analysis of the results achieved for individual tasks in EVALITA 2014 and illustrated in AUTHOR, this paper represents a first attempt of designing a complex shared task for the next EVALITA edition, resulting from the combination of the dependency parsing and event extraction tasks for the analysis of Italian texts. Such a complex task is expected to shed new light onto old challenges by: a.) investigating whether and how the cross-fertilization of tasks can make the evaluation campaign more application-oriented, while also improving individual task results; b.) generating a new resource combining dependency and event annotation; and, c.) devising evaluation metrics more oriented towards the assessment of the applicative impact of the achieved results.

Conclusion
For the evaluation of subtask results, participants will be asked to submit different runs, carried out with and without the information from the other subtask: i.e. Dependency Parsing will be carried out with and without event information, and Event Extraction will be carried out with and without dependency information. This type of contrastive evaluation highlights one of the main novelties of the proposed complex task, which is not only aimed at assessing the performance of participating systems and ranking achieved results, but also at investigating impact and role of different types of information on each task depending on the adopted algorithm. A shared task organized along these lines thus creates the prerequisites for a more accurate error analysis and will possibly open up new directions of research in tackling old challenges.