Inter-Annotator Agreement (IAA). Pair-wise Cohen kappa and group Fleiss’... | by Louis de Bruijn | Towards Data Science

In this story, we’ll explore the Inter-Annotator Agreement (IAA), a measure of how well multiple annotators can make the same annotation decision for a certain category. Supervised Natural Language Processing algorithms use a labeled dataset, that is often annotated by humans. An example would be the annotation scheme for my master’s thesis, where tweets were labeled as either abusive or non-abusive.
IAA shows you how clear your annotation guidelines are, how uniformly your annotators understood it, and how reproducible the annotation task is. It is a vital part of both the validation and reproducibility of classification results.
Accuracy and F1 score don’t take into account the expected chance agreements that are likely to occur when people annotate instances. The measures taking expected chance agreement into account:
Cohen’s κ: two annotators annotating each instance with a category
Fleiss’ κ: each instance was annotated n times with a category
This story covers best practices for annotations and explores the two IAA metrics for qualitative annotations: Cohen and Fleiss’ kappa. Cohen kappa is calculated between a pair of annotators and Fleiss’ kappa over a group of multiple annotators.
Supervised learning is the machine learning task of learning a function that maps an input to an output based on example input-output pairs. It infers this function from labeled training data consisting of a set of training examples.
When annotating data it is preferable to have multiple annotators annotate the same training instances to validate the labels. When multiple annotators annotate the same portion of the data we’re able to compute the inter-observer agreement or IAA.
For my master’s thesis, I worked with 44 Bachelor students, divided over 11 groups. Each individual annotated 100 unique tweets and 50 overlapping tweets that the other three group members also annotated. This resulted in 50 annotated tweets by four different annotators and 400 tweets by a single annotator.
Cohen’s kappa coefficient (κ) is a statistic to measure the reliability between annotators for qualitative (categorical) items. It is a more robust measure than simple percent agreement calculations, as κ takes into account the possibility of the agreement occurring by chance. It is a pairwise reliability measure between two annotators.
Cohen’s kappa statistic is the agreement between two raters where Po is the relative observed agreement among raters (identical to the accuracy), and Pe is the hypothetical probability of chance agreement. Below you will find the programmatic implementation of this evaluation metric.
Note that the annotations lists have to be ordered by the same index. ann1[0] and ann2[0] should represent the same annotated instance (tweet).
Fleiss’ kappa is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to several items or classifying items. It is a generalization of Scott’s pi (π) evaluation metric for two annotators extended to multiple annotators. Whereas Scott’s pi and Cohen’s kappa work for only two raters, Fleiss’ kappa works for any number of raters giving categorical ratings, to a fixed number of items. In addition to that, not all raters are required to annotate all items.
Thanks to Skylander86 for his contribution to the Fleiss kappa implementation in Python. I’ve simply duplicated his code and added comments to explain what is happening.
If the raters are in complete agreement then κ = 1. If there is no agreement among the raters (other than what would be expected by chance) then κ ≤ 0.
Cohen kappa has just two annotators, each annotator annotates every item.
Cohen’s κ is more informative than Scott’s π due to the way the chance agreement is calculated: The latter uses one distribution for each rater while the former uses different distributions.
Fleiss’ kappa is an extension of Scott’s π for two coders (not Cohen’s κ).
Fleiss’ kappa can have any number of annotators, where every item is not necessarily annotated by each annotator!
There are also implementations for Cohen and Fleiss’ kappa statistics available in the following packages, so you don’t have to write separate functions for them (even though it’s good practice!).
import sklearnfrom sklearn.metrics import cohen_kappa_scoreimport statsmodelsfrom statsmodels.stats.inter_rater import fleiss_kappa
Hopefully, this story showed you how to compute the IAA to validate your results and increase the reproducibility of your experiments! The full script can be found here.