Package: quica
Quick Inter Coder Agreement in PythonQuica (Quick Inter Coder Agreement in Python) is a tool to run inter coder agreement pipelines in an easy and effective ways.
Multiple measures are run and results are collected in a single table than can be easily exported in Latex.
quica supports binary or multiple coders.Quick Inter Coder Agreement in PythonFree software: MIT licenseDocumentation:https://quica.readthedocs.io.Installationpipinstall-UquicaGet Quick AgreementIf you already have a python dataframe you can run Quica with few liens of code! Let’s assume you have two
coders; we will create a pandas dataframe just to show how to use the library. As for now, we support only integer values
and we still have not included weighting.fromquica.quicaimportQuicaimportpandasaspdcoder_1=[0,1,0,1,0,1]coder_3=[0,1,0,1,0,0]dataframe=pd.DataFrame({"coder1":coder_1,"coder3":coder_3})quica=Quica(dataframe=dataframe)print(quica.get_results())This is the expected output:Out[1]:scorenameskrippendorff0.685714fleiss0.666667scotts0.657143raw0.833333mace0.426531cohen0.666667It was pretty easy to get all the scores, right? What if we do not have a pandas dataframe? what if we want to directly get
the latex table to put into the paper? worry not, my friend: it’s easier done than said!fromquica.measures.irrimport*fromquica.dataset.datasetimportIRRDatasetfromquica.quicaimportQuicacoder_1=[0,1,0,1,0,1]coder_3=[0,1,0,1,0,0]disagreeing_coders=[coder_1,coder_3]disagreeing_dataset=IRRDataset(disagreeing_coders)quica=Quica(disagreeing_dataset)print(quica.get_results())print(quica.get_latex())you should get this in output, note that the latex table requires the booktabs package:Out[1]:scorenameskrippendorff0.685714fleiss0.666667scotts0.657143raw0.833333mace0.426531cohen0.666667Out[2]:\begin{tabular}{lr}\toprule{}&score\\names&\\
\midrulekrippendorff&0.685714\\fleiss&0.666667\\scotts&0.657143\\raw&0.833333\\mace&0.426531\\cohen&0.666667\\
\bottomrule\end{tabular}Featuresfromquica.measures.irrimport*fromquica.dataset.datasetimportIRRDatasetfromquica.quicaimportQuicacoder_1=[0,1,0,1,0,1]coder_2=[0,1,0,1,0,1]coder_3=[0,1,0,1,0,0]agreeing_coders=[coder_1,coder_2]agreeing_dataset=IRRDataset(agreeing_coders)disagreeing_coders=[coder_1,coder_3]disagreeing_dataset=IRRDataset(disagreeing_coders)kri=Krippendorff()cohen=CohensK()assertkri.compute_irr(agreeing_dataset)==1assertkri.compute_irr(agreeing_dataset)==1assertcohen.compute_irr(disagreeing_dataset)<1assertcohen.compute_irr(disagreeing_dataset)<1Supported AlgorithmsMACE(Multi-Annotator Competence Estimation)Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., & Hovy, E. (2013, June). Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1120-1130).We define the inter coder agreeement as the average competence of the users.Krippendorff’s AlphaCohens’ KFleiss’ KScotts’ PIRaw Agreement: Standard AccuracyCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template. Thanks to Pietro Lesci and Dirk Hovy
for their implementation of MACE.History0.1.0 (2020-11-08)New API to get the outputFixed test casesExtended documentation on the README file0.1.0 (2020-11-05)First release on PyPI.