Package: crowdnalysis
crowdnalysisCrowdsourcing Citizen Science projects usually require citizens to classify items (images, pdfs, songs, etc.) into
one of a finite set of categories. Once an image is annotated by contributing citizens, we need to aggregate these
annotations to obtain a consensus classification. Usually, the consensus for an item is achieved by selecting the
most voted category for the item.crowdnalysisallows computing consensus using more advanced techniques beyond the
standard majority voting. In particular, it provides consensus methods that model quality for each of the citizen
scientists involved in the project. This more advanced consensus results in higher quality information for the
Crowdsourcing Citizen Science project, an essential requirement as citizens are increasingly willing and able to
contribute to science.Implemented consensus algorithmsMajority VotingProbabilisticMultinomialDawid-SkeneIn addition to the pure Python implementations above, the following models are implemented in the
probabilistic programming languageStanand used via theCmdStanPyinterface:MultinomialMultinomial EtaDawid-SkeneDawid-Skene Eta Hierarchical~ Eta models impose that the probability of a reported label is higher for the real class in the error-rate
(a.k.a. confusion) matrix.FeaturesImport annotation data from aCSVfile with preprocessing optionSet inter-dependencies between questions to filter out irrelevant annotationsDistinguish real classes for answers from reported labels (e.g., "Not answered")Calculate inter-rater reliability with different measuresFit selected model to annotation data and compute the consensusCompute the consensus with a fixed pre-determined set of parametersFit the model parameters provided that the consensus is already knownGenerate the confusion matrix between a consensus and the ground truthGiven the parameters of a generative model (Multinomial, Dawid-Skene); sample annotations, tasks,
and workers (i.e., annotators)Conduct prospective analysis of the 'accuracy vs. number of annotations' for a given set of modelsVisualize the error-rate matrix for annotatorsVisualize the consensus on annotated images inHTMLformatQuick startcrowdnalysis is distributed via PyPI:https://pypi.org/project/crowdnalysis/You can easily install it just like any other PyPI package:pipinstallcrowdnalysisCmdStanPywill be installed automatically as a dependency.
However, this package requires the installation of theCmdStancommand-line interface too.
This can be done via executing theinstall_cmdstanutility that comes withCmdStanPy.
See relateddocsfor  more information.install_cmdstanUse the package in code:>>> import crowdnalysisCheck available consensus models:>>> crowdnalysis.factory.Factory.list_registered_algorithms()See theTUTORIALnotebook for the usage of
main features.Unit testsWe usepytestas the testing framework. Tests can be
run—at the cloned repo directory—by:pytestIf you want to get the logs of the execution, run:pytest--log-cli-level0LoggingWe use the standardlogginglibrary.Deployment to PyPINote for contributorsFollow these simple steps to have a new release automatically deployed toPyPIby theCD workflow.
The example is given for the versionv1.0.2:Update the version insrc/crowdnalysis/_version.py:__version__="1.0.2"# Note no "v" prefix here.git pushthe changes tooriginand make sure the remotemasterbranch is up-to-date;Create a newtagpreferably with (multiline) annotation:gittag-av1.0.2-m". Upgrade to CmdStanPy v1.0.1"Push the tag toorigin:gitpushoriginv1.0.2And shortly, the new version will be available on PyPI.LicenseThis project is licensed under the GNU Affero General Public License v3.0 - see theLICENSEfile for details.CitationIf you find our software useful for your research, kindly consider citing it using the followingbiblatexentry with
the DOI attached to all versions:@software{crowdnalysis2022,author={Cerquides, Jesus and M{\"{u}}l{\^{a}}yim, Mehmet O{\u{g}}uz},title={crowdnalysis: A software library to help analyze crowdsourcing results},month=jan,year=2022,publisher={Zenodo},doi={10.5281/zenodo.5898579},url={https://doi.org/10.5281/zenodo.5898579}}Acknowledgementscrowdnalysis is being developed within theCrowd4SDGandHumane-AI-netprojects funded by the
European Union’s Horizon 2020 research and innovation programme under grant agreements No. 872944 and No. 952026.ReferenceFor the details of the conceptual and mathematical model of crowdnalysis, see:[1]
Cerquides, J.; Mülâyim, M.O.; Hernández-González, J.; Ravi Shankar, A.; Fernandez-Marquez, J.L.
A Conceptual Probabilistic Framework for Annotation Aggregation of Citizen Science Data.Mathematics2021, 9, 875.https://doi.org/10.3390/math9080875