Package: prodigy-iaa
✨ Prodigy - Inter-Annotator Agreement Recipes 🤝These recipes calculateInter-Annotator Agreement(aka Inter-Rater Reliability) measures for use withProdigy. The measures include Percent (Simple) Agreement, Krippendorff'sAlpha, and Gwet'sAC2. All calculations were derived using the equations inthis paper[^1], and this includes tests to match the values given on the datasets referenced in that paper.Currently this package supports IAA metrics for binary classification, multiclass classification, and multilabel (binary per label) classification. Span-based IAA measures for NER and Span Categorization will be integrated in the future.Note that you can also use the measures included here w/o directly interfacing with Prodigy, see section onother use cases.Installpip install prodigy-iaaFor devpip install git+https://github.com/pmbaumgartner/prodigy-iaaThis package usesentry pointsso you should just be able to install and run the commands below.RecipesRecipes depend the source data structure:iaa.datasetswill calculate measures assuming you have multiple datasets in prodigy, one dataset per annotatoriaa.sessionswill calculate measures assuming you have multiple annotators, identified typically by_session_id, in a single datasetiaa.jsonloperates the same asiaa.sessions, but on a file exported to JSONL withprodigy db-out.ℹ️Get details on each recipe's arguments withprodigy <recipe> --helpExampleIn this toy example, the command calculates agreement using datasetmy-dataset, which is amulticlassproblem -- meaning it's data is generated using thechoiceinterface, exclusive choices, storing choices in the "accept" key. In this example, there are 5 total examples, 4 of them have co-incident annotations (i.e. any overlap), and 3 unique annotators.$ prodigy iaa.sessions my-dataset multiclass

ℹ Annotation Statistics

Attribute                      Value
----------------------------   -----
Examples                           5
Categories                         3
Co-Incident Examples*              4
Single Annotation Examples         1
Annotators                         3
Avg. Annotations per Example    2.60

* (>1 annotation)

ℹ Agreement Statistics

Statistic                     Value
--------------------------   ------
Percent (Simple) Agreement   0.4167
Krippendorff's Alpha         0.1809
Gwet's AC2                   0.1640Validations & Practical UseAll recipes depend on examples being hashed uniquely and stored under_task_hashon the example. There are other validations involved as well:Checks ifview_idis the same for all examplesChecks iflabelis the same for all examplesChecks that each annotator has not double-annotated the same_task_hashIf any validations fail, or your data is unique in some way,iaa.jsonlis the recipe you want.Export your data, identify any issues and remedy them, and then calculate your measures on the cleaned exported data.TheoryThere is no single measure across all datasets to give a reasonable measurement of agreement - often times the measures are conditional on qualities of the data. The metrics included in these recipes have nice properties that make them flexible to various annotation situations: they can handle missing values (i.e. incomplete overlap), scale to any number of annotators, scale to any number of categories, and can be customized with your own weighting functions. In addition, the choice of metrics available within this package follow the recommendations in the literature[^2][^3], plus theoretical analysis[^4] demonstrating when certain metrics might be most useful.Table 13 inthis paper[^4] highlights systematic issues with each metric. They are as follows:When there islow agreement: Percent (Simple) Agreement can produce high scores.Imagine a binary classification problem with a very low base rate. Annotators can often agree on the negative case, but rarely agree on the positive.When there arehighly uneven sizes of categories:AC2can produce low scores,Alphacan produce high scores.When there areN < 20co-incident annotated examples:Alphacan produce high scores.You probably shouldn't trustN < 100generally.When there are3 or more categories:AC2can produce high scores.Summary: Use simple agreement andAlpha. If simple agreement is high, andAlphais low, verify withAC2[^3]. In general these numbers correlate, if you're getting contradictory or unclear information increase the number of examples and explore your data.Other Use-Cases / Use Outside ProdigyIf you want to calculate these measures in a custom script on your own data, you can usefrom prodigy_iaa.measures import calculate_agreement. See tests intests/test_measures.pyfor an example. The docstrings for each function should indicate the expected data structures.You could also use this, for example, to print out some nice output during anupdatecallback and get annotation statistics as each user submits examples.If you want to calcualte more precise statistics, e.g. comparing two annotators pairwise, you could write a script to do that as well with these existing functions.TestsTests require a working version ofprodigy, so they are not run in CI and must be run locally.References[^1]: K. L. Gwet, “On Krippendorff’s Alpha Coefficient,” p. 16, 2015.
[^2]: J. Lovejoy, B. R. Watson, S. Lacy, and D. Riffe, “Three Decades of Reliability in Communication Content Analyses: Reporting of Reliability Statistics and Coefficient Levels in Three Top Journals,” p. 44.
[^3]: S. Lacy, B. R. Watson, D. Riffe, and J. Lovejoy, “Issues and Best Practices in Content Analysis,” Journalism & Mass Communication Quarterly, vol. 92, no. 4, pp. 791–811, Dec. 2015, doi: 10.1177/1077699015607338.
[^4]: X. Zhao, J. S. Liu, and K. Deng, “Assumptions Behind Intercoder Reliability Indices,” Communication Yearbook, p. 83.