Inter-Annotator Agreement in linguistica: una rassegna critica
Authors: Gloria Gagliardi (Università di Bologna, 2018)
Topic: Lexical and Semantic Resources and Analysis

Abstract
Agreement indexes are widely used in English. Agreement indexes are widely used in Computational Linguistics and NLP to assess the reliability of annotation tasks. The paper aims at reviewing the literature on the topic, illustrating chance-corrected coefficients and their interpretation.

Introduction
The construction of linguistic resources, and more generally the annotation of data, implies the formulation of subjective judgments. The need to determine to what extent such judgments are reliable and reproducible has become increasingly important, to the point of making validation procedures a consolidated practice. This has happened in computational linguistics (CL) with over 30 years of delay compared to psychometrics: already in 1960, Cohen, in a famous article, wrote: "Because the categorizing of the units is a consequence of some complex judgment process performed by a ‘two-legged meter’ [...], it becomes important to determine the extent to which these judgments are reproducible, i.e., reliable." (Cohen, 1960: 37) It is a fairly widespread belief that a high level of Inter-Annotator Agreement (henceforth: I.A.A.) between annotators is indicative of the goodness and reproducibility of an annotation paradigm. As Di Eugenio underlines: "This raises the question of how to evaluate the ‘goodness’ of a coding scheme. One way of doing so is to assess its reliability, namely, to assess whether different coders can reach a satisfying level of agreement with each other when they use the coding manual on the same data." (Di Eugenio, 2000: 441) The basic assumption, therefore, is that data can be considered "reliable" if two or more annotators agree in identifying a linguistic phenomenon or in assigning a category to the item under analysis. From this perspective, reliability thus becomes a prerequisite for demonstrating the validity of a coding scheme, and broad consensus among annotators is taken as a guarantee of the intrinsic accuracy of the annotation process (Warrens, 2010). "The main reason for the analysis of annotation quality is to obtain a measure of the ‘trustworthiness’ of annotations. [...] Only if we can trust that annotations are provided in a consistent and reproducible manner, can we be sure that conclusions drawn from such data are likewise reliable and that the subsequent usage of annotations is not negatively influenced by inconsistencies and errors in the data. Inter-annotator (or inter-coder) agreement has become the quasi-standard procedure for testing the accuracy of manual annotations." (Bayerl & Paul, 2011: 700) In the computational field, I.A.A. is used as a means to move from annotated material to a gold standard, that is, a set of sufficiently noise-free data that serves for the training and testing of automatic systems. By convention, agreement coefficients are used to ensure the quality of the annotation procedure and the annotated material: a high level of I.A.A. ensures that the phenomenon is considered consistent and systematic, and that the validated resource is suitable for training an automatic system to perform the same task as the linguist. In reality, the idea that I.A.A. can absolutely indicate the quality of the dataset as a reference resource is fallacious: two observers can both be wrong and yet be in perfect agreement in evaluating an event: "However, it is important to keep in mind that achieving good agreement cannot ensure validity: two observers of the same event may well share the same prejudice while still being objectively wrong." (Artstein & Poesio, 2008: 557) It is also important to consider that the agreement typically reached by annotators varies depending on the level of experience: I.A.A. in homogeneous groups is comparable regardless of experience levels, but it decreases when mixed groups of experts and non-experts are formed: "Implicit in discussions of inter-annotator agreement is that coders not only agree on which unit belongs to which category, but that if they agree these decisions are also correct with respect to the phenomenon under scrutiny [...]. In our study, this assumption left us with a dilemma. Our data showed that experts and non-experts could achieve comparable levels of agreement, whereas the average agreement for mixed groups was significantly lower. In other words, experts and novices were equally reliable, yet did not agree with each other." (Bayerl & Paul, 2011: 721) Not all linguistic annotation tasks can be evaluated using the same procedures; from a qualitative point of view, at least two general types can be identified (Mathet, Widlöcher, & Métivier, 2015): • "Unitizing" (Krippendorff, 1980): in this type of task, the annotator, given a written or spoken text, must identify the position and boundaries of linguistic elements (e.g., identification of prosodic or gestural units, topic segmentation). • "Categorization": the annotator must assign a tag to pre-identified linguistic objects (e.g., PoS Tagging, Word Sense Disambiguation). The paper aims to present a brief critical review of the metrics used in this second type of task, particularly focusing on the calculation of coefficients and their interpretation.

Conclusion
As suggested in the initial paragraphs, a high level of I.A.A. does not constitute an end in itself, but only one of the indicators of the actual reliability of the annotation undergoing validation. It is therefore desirable that an increasing amount of data on I.A.A. for different annotation tasks be shared by researchers, in order to facilitate the emergence of benchmark values through comparison.