Package: agreement
AgreementInter-rater agreementAgreement library provides an implementation of popular metrics used to measure inter-rater agreement.
Inter-rater agreement (know also as a inter-rater reliability) is used to describe the degree of agreement among raters.
It is a score of how much homogeneity or consensus exists in the ratings given by various judges.If you want to learn more about this topic, you can start by reading thisWikipediapage.Implemented metricsThis library provides a pure numpy implementation of an extended formulas for following metrics:Observed agreementBennett et al.'s S scoreCohen's kappaGwet's gammaKrippendorff alphaScott's piAnd extended formulas can be used to measure agreement for;multiple raters- support for two or more raters,multiple categories- support for binary problems, as well as more categories,missing ratings- not all raters provided answers for all the questions.weighted agreement- used to model distance between categories (e.g.dist(5, 4) < dist(5, 1))More information about implemented metrics can be found here: TODOImplemented weights kernelsAgreement provides implementations for eight weight kernels:identity kernellinear kernelquadratic kernelordinal_kernelradical_kernelradio_kernelcircular_kernelbipolar_kernelMore information about implemented weights kernels can be found here: TODOInstallationAgreement can be installed via pip fromPyPI.pipinstallagreementExample usage1. Prepare datasetLet's assume you have a dataset in a format of a matrix with three columns:question id,rater idandanswer.importnumpyasnpdataset=np.array([[1,1,'a'],[1,2,'a'],[1,3,'c'],[2,1,'a'],[2,2,'b'],[2,3,'a'],[3,1,'c'],[3,2,'b'],])2. Transform dataset into matricesIn the next step we want to transform the dataset into matrices in a form accepted by the metrics functions.Most of the matrices require a "questions answers" matrix, which contains a frequency of answers for each question.
So more formally we could sayM = I x A, whereIis a list of all items andAis a list of all possible answers.
Matrix elementM_ijrepresents how many times answerjwas chosen for the questionsi.The second matrix can be required (currently it is only required by the Cohen's kappa metrics) is "users answers" matrix, which
contains a frequency of answers selected by each user.
So more formally we could sayM = U x A, whereUis a list of all users andAis a list of all possible answers.
Matrix elementM_ijrepresents how many times answerjwas chosen for the useri.The library provides a helper functions that can be used to prepare that.fromagreement.utils.transformimportpivot_table_frequencyquestions_answers_table=pivot_table_frequency(dataset[:,0],dataset[:,2])users_answers_table=pivot_table_frequency(dataset[:,1],dataset[:,2])3. Select kernelWeights are used to model situations, where categories are represented as (at least) ordinal data.
Using this approach, the agreement between raters is not binary, but it differs depending on the
weights between chosen categories.There is no formal rule that can be used for deciding which set weights should be used
in a particular study, so it all depends on your problem and the data your are working with.In a default, metrics are using theidentity_kernel, which do not provide any weighting between the answers.
If you want to use an alternative kernel, you can import it from:fromagreement.utils.kernelsimportlinear_kernel4. Compute the metricThe last step is to chose the metric you want to compute and run following code:fromagreement.metricsimportcohens_kappa,krippendorffs_alphakappa=cohens_kappa(questions_answers_table,users_answers_table)weighted_kappa=cohens_kappa(questions_answers_table,users_answers_table,weights_kernel=linear_kernel)alpha=krippendorffs_alpha(questions_answers_table)For more detailed example see: TODOReferenceAll equations are based on the Handbook of Inter-Rater ReLiability, Kilem Li. Gwet, 2014.
This book provides an extensive explanation to all topics related to inter-rater agreement.
The book provides a detailed description of all metrics implemented in this library, as well as
an example datasets that were used to this this implementation.I also recommend taking a look at MatLab implementation of the same metricsmReliability,
which provides a more detailed explanation of metrics' formulas then the one you will find here.