Find Problems before They Find You,with AnnotatorPro's Monitoring Functionalities
Authors: Mohammed R. H. Qwaider, Anne-Lyse Minard, Manuela Speranza, Bernardo Magnini (Fondazione Bruno Kessler, 2017)
Topic: Lexical and Semantic Resources and Analysis

Abstract
We present a tool for annotation of linguistic data. AnnotatorPro offers both complete monitoring functionalities (e.g. inter-annotator agreement and agreement with respect to a gold standard) and highly flexible task design (e.g. token and document level annotation, adjudication and reconciliation procedures). We teste AnnotatorPro in several industrial annotation scenarios, coupled with Active Learning techniques.

Introduction
Driven by the popularity of machine learning approaches, there has been in the last years an increasing need to produce human annotated data for a large number of linguistic tasks (e.g. named entity recognition, semantic role labeling, sentiment analysis, word sense disambiguation, and discourse relations, just to mention a few). Datasets (development, training and test data) are being developed for different languages and different domains, both for research and industrial purposes. A relevant consequence of this is the increasing demand for annotated datasets, both in terms of quantity and quality. This in turn calls for tools with a rich apparatus of functionalities (e.g. annotation, visualization, monitoring and reporting), able to support and monitor a large variety of annotators (i.e. from linguists to mechanical turkers), flexible enough to serve a large spectrum of annotation scenarios (e.g. crowdsourcing and paid professional annotators), and open to the integration of %classification NLP tools (e.g. for automatic pre-annotation and for instance selection based on Active Learning). Although there is a large supply of annotation tools, such as brat AUTHOR, GATE AUTHOR, CAT AUTHOR, and WebAnno AUTHOR, and several functions are included in common crowdsourcing platforms (e.g. CrowdFlower}), we believe that none of the available tool possesses the full range of functionalities for a real and intensive industrial use. As an example, none of the afore mentioned %options tools allows one to implement adjudication rules (i.e. under what condition an item annotated by more than one annotator is assigned to a certain category) or to visualize items with disagreement among annotators. This paper introduces AnnotatorPro, a new annotation tool which was mainly conceived to fulfill the above-mentioned needs. We highlight two main aspects of the tool: (i) a high level of flexibility to design the annotation task, including the possibility to define adjudication and reconciliation procedures; (ii) the rich set of functionalities allowing for constant monitoring of the quality of the data being annotated. The paper is organized as follows. In Section sec:relatedwork we compare AnnotatorPro with some state-of-the-art annotation tools. Section sec:desc provides a general description of the tool. Sections sec:taskDesign and sec:monitoring focus on the task design and on the monitoring functionalities, while Section sec:appl provides a brief overview of the tool's application and future extensions.

Conclusion
We used ANNOTATOR P RO for multiple projects, on different tasks, including named entity recognition AUTHOR, event detection AUTHOR and sentiment analysis. The tool has been successfully exploited both in situations with few experienced annotators as well as with more than 20 non-expert annotators (i.e. high school students) working in parallel. ANNOTATOR PRO has been fully integrated within an Active Learning platform AUTHOR and successfully employed in two industrial projects, resulting in high quality data. As for our next steps, we are working to extend ANNOTATOR PRO to include relations among annotated entities, such as the relation between a verb and its argument/s in semantic role labeling. ANNOTATOR PRO is distributed as open source software under the terms of Apache License 2.0.4 from the web page:.