Package: peerannot
A Python library for managing and learning from crowdsourced labels in image classification tasks—Thepeerannotlibrary was created to handle crowdsourced labels in classification problems.InstallTo installpeerannot, simply runpipinstallpeerannotOtherwise, asetup.cfgfile is located at the root directory.
Installing the library gives access to the Command Line Interface using the keywordpeerannotin a bash terminal. Try it out using:peerannot--helpQuick startOur library comes with files to download and install standard datasets from the crowdsourcing community. Those are located in thedatasetsfolderpeerannotinstall./datasets/cifar10H/cifar10h.pyRunning aggregation strategiesIn python, we can run classical aggregation strategies from the current dataset as followsforstratin["MV","NaiveSoft","DS","GLAD","WDS"]:!peerannotaggregate.-s{strat}This will create a new folder nameslabelscontaining the labels in thelabels_cifar10H_${strat}.npyfile.Training your networkOnce the labels are available, we can train a neural network withPyTorchas follows. In a terminal:forstratin["MV","NaiveSoft","DS","GLAD","WDS"]:!peerannottrain.-ocifar10H_${strat}\-K10\--labels=./labels/labels_cifar-10h_${strat}.npy\--modelresnet18\--img-size=32\--n-epochs=1000\--lr=0.1--scheduler-m100-m250\--num-workers=8End-to-end strategiesFinally, for the end-to-end strategies using deep learning (as CoNAL or CrowdLayer), the command line is:peerannotaggregate-deep.-ocifar10h_crowdlayer\--answers./answers.json\--modelresnet18-K=10\--n-epochs150--lr0.1--optimizersgd\--batch-size64--num-workers8\--img-size=32\-scrowdlayerFor CoNAL, the hyperparameter scaling can be provided as-sCoNAL[scale=1e-4].Peerannot and the crowdsourcing formattingInpeerannot, one of our goals is to make crowdsourced datasets under the same format so that it is easy to switch from one learning or aggregation strategy without having to code once again the algorithms for each dataset.So, what is a crowdsourced dataset? We define each dataset as:dataset
├── train
│     ├── ...
│     ├── data as imagename-<key>.png
│     └── ...
├── val
├── test
├── dataset.py
├── metadata.json
└── answers.jsonThe crowdsourced labels for each training task are contained in theanwers.jsonfile. They are formatted as follows:{0:{<worker_id>:<label>,<another_worker_id>:<label>},1:{<yet_another_worker_id>:<label>,}}Note that the task index in theanswers.jsonfile might not match the order of tasks in thetrainfolder… Thence, each task’s name contains the associated votes file index.
The number of tasks in thetrainfolder must match the number of entry keys in theanswers.jsonfile.Themetadata.jsonfile contains general information about the dataset. A minimal example would be:{"name":<dataset>,"n_classes":K,"n_workers":<n_workers>,}Create you own datasetThedataset.pyis not mandatory but is here to facilitate the dataset’s installation procedure. A minimal example:classmydataset:def__init__(self):self.DIR=Path(__file__).parent.resolve()# download the data needed# ...defsetfolders(self):print(f"Loading data folders at{self.DIR}")train_path=self.DIR/"train"test_path=self.DIR/"test"valid_path=self.DIR/"val"# Create train/val/test tasks with matching index# ...print("Created:")forset,pathinzip(("train","val","test"),[train_path,valid_path,test_path]):print(f"-{set}:{path}")self.get_crowd_labels()print(f"Train crowd labels are in{self.DIR/'answers.json'}")defget_crowd_labels(self):# create answers.json dictionnary in presented format# ...withopen(self.DIR/"answers.json","w")asansw:json.dump(dictionnary,answ,ensure_ascii=False,indent=3)