Improve learning combining crowdsourced labels by weighting Areas Under the Margin
Authors: Tanguy Lefort
Benjamin Charlier
Alexis Joly
Joseph Salmon
Venue: Submitted to ICLR 2023 (2023)
Keywords: crowdsourcing
ambiguity
area under the margin
aggregation
noisy labels

Abstract: In supervised learning -- for instance in image classification -- modern massive datasets are commonly labelled by a crowd of workers. The obtained labels in this crowdsourcing setting are then aggregated for training. The aggregation step generally leverages a per worker trust score. Yet, such worker-centric approaches discard each task ambiguity. Some intrinsically ambiguous tasks might even fool expert workers, which could eventually be harmful for the learning step. In a standard supervised learning setting -- with one label per task and balanced classes -- the Area Under the Margin (AUM) statistic is tailored to identify mislabeled data. We adapt the AUM to identify ambiguous tasks in crowdsourced learning scenarios, introducing the Weighted AUM (WAUM). The WAUM is an average of AUMs weighted by worker and task dependent scores. We show that the WAUM can help discarding ambiguous tasks from the training set, leading to better generalization or calibration performance. We report improvements with respect to feature-blind aggregation strategies both for simulated settings and for the CIFAR-10H crowdsourced dataset.

Under review as a conference paper at ICLR 2023
IMPROVE LEARNING COMBINING CROWDSOURCED LABELS BY WEIGHTING AREAS UNDER THE MARGIN
Anonymous authors Paper under double-blind review
ABSTRACT
In supervised learning for instance in image classification modern massive datasets are commonly labeled by a crowd of workers. The obtained labels this crowdsourcing setting are then aggregated for training. The aggregation step generally leverages a per worker trust score. Yct, such worker-centric approaches discard each task ambiguity. Some intrinsically ambiguous tasks might even fool expert workers, which could eventually be harmful for the learning step. In a stan- dard supervised learning setting with one label per task and balanced classes the Area Under the Margin (AUM) statistic is tailored to identify mislabeled data. We adapt the AUM to identify ambiguous tasks in crowdsourced learning scenar- ios, introducing the Weighted AUM (WAUM). The WAUM is an average of AUMs weighted by worker and task dependent scores. We show that the WAUM can help discarding ambiguous tasks from the training set, leading to better generalization or calibration performance. We report improvements with respect to feature-blind aggregation strategies both for simulated settings and for the CIFAR- 10H crowd- sourced dataset.
INTRODUCTION
Crowdsourcing labels for supervised learning has become quite common in the last two decades, notably for image classification with datasets such CIFAR-10 and Imagenet. Using a crowd of workers is fast, simple (see Figure 1) and less expensive than using experts. Furthermore, aggregat- ing crowdsourced labels instead of working directly with a single one enables modeling the sources of possible ambiguities and directly take them into account in the training pipeline (Aitchison, 2021). With deep neural networks nowadays common in many applications, both the architectures and data quality have direct impact and on calibration (Guo et al., 2017). Yet, depending on the crowd and platform's 's control mecha- nisms, the obtained label quality might vary and harm generalization (Snow et al., 2008).
the model performance (Müller et al., 2019; Northcutt et al., 202
Popular label aggregation schemes take into account the uncertainty related to workers' abilities: for example by estimating confusions between classes, or using latent variable representing each worker trust (Dawid Skene, 1979; Kim & Ghahramani, 2012; Sinha et al., 2018; Camilleri & Williams, 2019). This leads scoring workers without taking into account the inherent difficulty of a task at stake. Inspired by the Item Response Theory (IRT) (Birbbau, 1968), Whitehill et al. (2009) combined both the task difficulty and the worker's ability in feature-blind fashion for label aggregation. They only require labels but not the associated features In the classical supervised learning setting, the labels are said be hard i.e., a Dirac mass on one class. Multiple crowd- sourced labels induce soft labels - probability distributions over the classes - for each task. Our motivation is to identify ambiguous tasks from their associated features, hence discarding hurtful tasks (such as the one illustrated on Figure 2b). Recent work on data-cleaning in supervised learn- ing (Han et al., 2019; Pleiss et al., 2020; Northeutt et al., 2021a) has shown that some images might be too corrupted or too ambiguous be labeled by humans. Hence, one should not consider these tasks for label aggregation and learning since they might be harmful for generalization.
In this work, we combine task difficulty scores with worker abilities scores, but we measure the task difficulty by incorporating feature information. We thus introduce the Weighted Area Under
We use the task interchangeably with the feature in this work

---

Under review as a conference paper at ICLR 2023
Image x proposed
Image proposed labels
Apgropatedd ssafo labels
Crowd of workers
Train
Unlabeled images
2 7
V
Figure I: Crowdsourcing labels scheme, from label collection using a crowd to training neural network on aggregated training labels. High ambiguity from either crowd workers or tasks intrinsic difficulty can lead to mislabeled data and harm generalization performance. To illustrate our notation, here the set of task annotated by worker wy is 7 (wa) = {1.3) while the set of workers annotating the task xa is A(za) (1,3,4).
the Margin (WAUM), a generalization to the crowdsourcing setting of the Area Under the Margin (AUM) (Pleiss et al., 2020). The AUM is a confidence indicator in an assigned label defined for each training task. It is computed as an average of margins over scores obtained along the learning steps, and reflects how a learning procedure struggles to classify a task to an assigned label (see Figures 3 and to visualize how the AUM connected to the classical margin from the kernel literature). The AUM is well suited when training neural network (where the steps are training epochs) or other iterative methods. For instance, it has led to better network calibration (Park & Caragea, 2022) using MixUp strategy (Zhang et al., 2018), i.e., mixing tasks identified as simple and difficult by the AUM. The WAÜM identifies harmful data points in crowdsourced datasets, so one can prune ambiguous tasks that degrade the generalization. It is weighted average of workers AUM, where the weights reflect trust scores based on tasks difficulty and workers ability.
2 RELATED WORK
Inferring learning consensus from crowd is a challenging task. In Table I, we summarize features used by standard strategies to address such a task. In this work we do not consider methods with prior knowledge
workers, since most platforms do not provide this information Likewise, do not rely on ground-truth knowledge for any tasks. Hence, trapping-set or control-items based algorithms like ELICE or CLUBS (Khattak, 2017) do not match our framework. Some algorithms rely on self-reported confidence: they directly ask workers their answering confidence and integrate it in the model (Albert al., 2012: Oyama et al., 2013; Hoang et al., 2021). We discard such cases for several reasons. First, self-reported confidence might not be beneficial without a reject option (Li & Varshney, 2017). Second, workers have a tendency to be under or overconfident, raising questions on how to present self- evaluation and inferring their own scores (Draws et al., 2021).
The most common aggregation step is majority voting (MV), where one selects the label most often answered by workers. MV does not infer any trust score on workers, thus does not leverage workers abilities. MV is also very sensitive to under-performing workers (Gao & Zhou, 2013; Zhou et al., 2015), to biased workers (Kamar et al., 2015), to spammers (Raykar & Yu, 2011), or to lack of experts for hard tasks (James, 1998; Gao & Zhou, 2013; Germain et al., 2015). Closely related to MV, naive soft labeling goes beyond hard labels (also referred to one-hot labels) by computing the frequency of answers per label. In practice, training a neural network with soft labels improves calibration (Guo al., 2017) with respect to using hard labels. However, both methods to spammers (e.g., workers answer all tasks randomly) or workers biases (e.g,, workers who answer some tasks randomly). Hence, the noise induced by workers labeling might not be representative of the actual task difficulty (Jamison & Gurevych, 2015).
sensitive
Another class of methods leverages latent variables, defining a probabilistic model on worker's responses. The most popular one, proposed by Dawid & Skene (1979) (DS) estimates a single con- fusion matrix per worker, as a measure of workers' expertise. The vanilla DS model assumes that a worker answers according to a multinomial distribution, yielding a joint estimation procedure of the error-rates and the soft labels through an Expectation-Maxxiizaaii rithm 2 in Appendix A). Variants on the original DS algorithm include accelerated versions (Sinha et al., 2018), sparse versions (Servajean et al., 2017), clustered versions (Imamura et al., 2018)
(EM) algorithm (see Algo-
FFor instance, by default Amazon Mechanical Turk https: /www. www.kk com/ does not provide it.
2

---

Under review as a conference paper at ICLR 2023
Vines disributoo
1
(a) Label frog is casy to identify. Only worker mislabeled this frog as bird.
(b) Label deer is hard to identify, confused with horse, dog other animals available.
Two images from CIFAR-IOH dataset (Peterson et al., 2019): the deer image is more ambiguous due to the image poor quality. With enough workers, the votes' distribution reflects this ambiguity though in practice there may be only few workers for some tasks.
Figure 2:
among others. Since DS only models workers abilities, Whitehill et al. (2009) have introduced the Generative model of Labels, Abilities, and Difficulties (GLAD) with a task difficulties scores to bet- ter handle confusion factors. While DS estimates matrix of pairwise labels confusion per worker, GLAD estimates (also with EM) a single ability score per worker, and a single difficulty score per task. is inspired by Item Response Theory (IRT) (Birnbaum, 1968), modeling the workers' proba- bility to answer the true label with a logistic transform the product of these scores. Following IRT, the difficulty is inferred task affected to each worker. Other feature-blind aggregation strategies exist using rank-one matrix completion (Ma et al., 2020; Ma & Olshevsky, 2020) also pairwise co-occurences (Ibrahim al., per worker
latent variable given the answers, without ever considering the actual
2019). We propose the WAUM to combine the information from a confusion- and a measure of relative difficulty between tasks. leads to a more accurate judging system and identifies data points harming generalization that should be pruned. Data pruning has been shown to improve generalization by removing mislabeled data (Angelova al., 2005; Pleiss et al., 2020), possibly dynamically along the learning phase (Raju et al., 2021) 1 or by defining forgetfulness score (Paul et al., 2021), I). Finally, Sorscher et al. (2022) have highlighted that data pruning strategies re- quire label information to be successful in the standard supervised setting; confirm its relevance to the crowdsourcing framework.
Table 1: Summary of vote aggregation algorithms: MV, naive soft, DS, GLAD and WAUM (ours). With the WAUM, use tasks features to identify and remove harmful tasks.
Soft label
Worker ability
Task difficulty
Use tasks votes
MV Naive soft (vanilla)
x
x
x
X
GLAD WAUM
/
WEIGHTED AREA UNDER THE MARGIN
3.1 DEFINITION AND CONSTRUCTION
For any set S, write S] for the cardinality, and for any integer n, [n] represents the set {1, n). We consider classical notation from multi-class classification settings with input from X' (e.g. im- ages) and labels in [K] {1...., K). Consider a set {(11, vi). (Insaa Ve }} with neask i.idd tasks and labels. The set of tasks is written Xrain { Inean }. The underlying distribution is denoted by P., and each pair (I,, vf) E X x [K] is a feature/label pair.
I,..., In our context, the true labels for I,...,,taskk are unobserved (except in simulations); only collects the proposed labels given by a crowd of Nyorker workers (named W1, Wn.wrr.. ). As represented in Figure some workers might annotate more tasks than others, and similarly some tasks might receive more annotations than others. For any E [ntaak], we write the annotators set given task r; A(xa) (i [Mwrkke=] worker wy answered task x.). We recover the standard
3

---

Under review as a conference paper at ICLR 2023
supervised setting when |A(z,)] for all i [meask] Similarly, for any j E [ we write
the tasks given a worker Wy as ((w;) = E [ntass] : worker wy answered task z; For each
denote y/ [K] the label answered by the j-th worker, and write Y: an estimation of the (possibly soft) label obtained after aggregation. Note that ýi belongs to the
task x, and each j E A(x,),
standard simplex AK-1 {o E IK E Pk 1, 2 0, k = [K]} of dimension K - 1. Our
training set has task-wise wise or worker wise formulations:
U i=1
{ (on, (Bx,(())))
(v(2)) for j E A(xs)
}
Roozbex
U ==1
(1ss (v( for T(w,))
Pwooker
U j=1
D
Dirain
train
DS model. The Dawid and Skene (DS) model (Dawid & Skene, 1979) aggregates answers and evaluates the workers' confusion matrix to observe where their expertise lies exactly. The confusion matrix of worker w, is denoted by T E R and reflects individual error-rates between pairs of labels. Each individual error-rate Tzk which represents the probability that worker wj gives label k to a task with true label f, is estimated using the EM algorithm (see Appendix A, Algorithm 2; the likelihood it maximizes is available in Appendix A, Equation 6). This algorithm is independent of Nuain and only considers the labels. The model assumes that the probability for a task i to have true label y = l follows a multinomial distribution with probabilities xxa for each worker.
(5)
AUM. Pleiss et al. (2020) have introduced the AUM in the standard learning setting, where one observes a training set D Duans i.e., with [A(x;)] = for all [Meass]] Given a training image and a label (x, y) E Dusin let (z) E RK be the associated score vector at epoch I T when learning neural network on Danin (I' being the number of training epochs). We use the notation (t)
"in (x) 2... ... 2 2(k] (x) for sorting the vector (z) (z)" (x),.... 2 (x)) in decreasing order. Denote softmax (I) (x) softmax(2"" (x)) the softmax output of the scores at epoch t. Sorting the proba-
bilities in decreasing order such that softmax [1] (x) 2 ... 2 softmax [K] (T), the AUM reads:
(1)
T
1 = T [softmax/ (x) - sootmax (f) E R.
AUM (I, yi miii) == AUM (r,
Σ
(2)
Pleiss et al. (2020) uses an average of margins over scores, whereas we instead consider the average over the margin with the softmax output in Equation 2. We have adapted the original AUM by using the softmax to avoid any scaling issue (as advocated by Ju et al. (2018) in ensemble learn- ing). Another difference is that the margin considered is from Yang & Koyejo (2020), since the corresponding Hinge loss has better theoretical properties, especially for top-k settings". than the one used in the original AUM definition (Lapin et al., 2016; Yang & Koyejo, 2020).
During the training phase, the AUM keeps track of the difference between the score assigned to the proposed label and the score assigned to the second largest one. It has been introduced to detect mislabeled observations in a dataset: the higher the AUM, the more confident the prediction is in the assigned label. Hence, the lower the AUM, the more likely the label is wrong. The AUM algorithm is described in Appendix B, Algorithm 4. Finally, note that the AUM computation depends on the chosen neural network and on its initialization: pre-trained architectures could be used, yet any present bias would transfer to the AUM computation.
WAUM. The AUM is defined in a standard supervised setting with (hard) labels: we now adapt it to crowdsourced frameworks. Given (Ii,, () E Dirain D(1) U ... (itaorree) let s0J) (2:)
[0, 1] be trust factor in the answer of worker wy for the task 24. The WAUM is then defined as:
WAUM(x)) = EjEA(H)) (())
Σ
(I.)AUM Mi () (D) wain) E ERR
(3)
JEA(z.)
The WAUM the weighted average of the AUMs over each worker's answer with weighting score أli) for each task based on workers abilities. The WAUM allows to identify potentially too hard tasks and to remove them from the training set and from the worker's confusion estimation.
For kk accuracy, one could consider softmax coftmax/et1jf (x))(()) instead of softmax (1) (x) in Equation (2).
4

---

Under review as a conference paper at ICLR 2023
The scores (x;)) consider the impact of the AUM of one answer. It indeed more informative if an expert worker's 's AUM indicates a potential uncertainty in the label than a potential error from a poor-quality worker. The architecture used in the AUM computation also needs to be reset between workers to avoid mutual influence. For additional details on AUM and WAUM, see Appendix B. The weights şli) are obtained la Servajean et al. (2017): each worker has an estimated confusion matrix (()) E R Note that the vector diag(#(0) E I& represents the probability for the worker Wj to answer correctly to each task. Moreover, with neural network classifier, we estimate the probability for the input x, E Xuain to belong each category by softmax(") (xx,) - iee, the probability estimate obtained at the last epoch. As trust factor, we propose the inner product between the diagonal of the confusion matrix and the softmax vector:
(4) that controls the weight of each worker in the WAUM(a;) formulation given by Equation (3). This choice of weight is also inspired by the bilinear scoring system of the GLAD strategy as further detailed hereafter. The closer to one, the more we trust the worker for the given task.
800) (x.) (diag(#())) softmax (x (z.)) [0.1]
Link with GLAD. In GLAD (Whitehill et al., 2009), the trust score is modeled as the product ay,8, with a E R (resp. B, E (0, +o)) representing worker ability (resp. task difficulty), cf Ap- pendix A, Algorithm 3. Using DS, we can leverage the full confusion matrices using a scalar product between a quantity related to the worker and another to the task. The diagonal of the confusion ma- trix #() represents the worker ability and the softmax probabilities represents the task information. This trust score e(t) (I;)) can thus be seen as a multidimensional version of GLAD's trust score.
3.2 LABEL AGGREGATION USING THE WAUM.
The WAUM metric aims at identifying training samples that are difficult to classify, and that could be discarded either when aggregating workers' labels or at training. The last step relies on training a neural network thanks to the obtained soft labels, whose architecture is adapted to the task at hand".
Dataset pruning, Our proposed procedure (Algorithm 1), proceeds as follows. Confusion matri- ces are estimated for all workers using an estimation step on the full training set Duain- By default we rely on the DS algorithm described in Algorithm 2, but any estimates of workers' confusion matrices can be used instead (see Appendix E). For each worker wy, AUMs are computed for its
labeled tasks using Algorithm 4 on D(1) train and worker-dependent trust scores for each task 2 (a;)) with Equation (4). The WAUM in Equation (3) is then computed for each task. Tasks below the quantile of order a E [0, 1] are then pruned. From the resulting dataset D Prnneed (with tasks Xpruned), we update confusion matrices. We eventually provide soft labels ÿ; for task x; E X Xpruned by weight- ing labels with workers' confidence. A neural network can then be trained on (Iz, ÿt) for E Xpruned- The hyperparameter a (amount of training data points pruned) can be chosen on a validation set, yet choosing a 0.1 or 0.01 has led to satisfactory results. The output soft labels still contain infor- mation regarding human uncertainty, while being less noisy than naive soft labels. They could help improving model calibration (Wen et al., 2021; Zhong et al., 2021), a property often expected for interpretations (Jiang et al., 2012; Kumar et al., 2019).
Datasets stacking. In case of time constraints or with few labels per workers, the AUM and worker trust scores can be modified only train a single network. To do so, one can modify Algo- rithm 1; Lines 2-5, so the network is trained on D Datsek := {(2,, y. ( (ememmaalll of all workers' training sets. We explore stacking in Appendix E.3 for few labels per worker.
the stacking
Refining confusion matrix estimation. DS might suffer from the curse of dimensionality when the number K of classes is large. Indeed, relies on estimating K2 coefficients per worker. Possible the number of workers and how many answers they give are possible. For
adaptation depending instance, one can use clustered (workers) confusion matrices in Algorithm 1 as a remedy (Imamura et al., 2018). Another alternative is to leverage the iterative nature of Algorithm I and estimate confusion matrices 40) only on unambiguous tasks, see Appendix E.2.
We deseribe this choice for each experiment in the following section.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Label aggregation using the WAUM.
Data: Damin: tasks and crowdsourced labels, a E [o, 1]: cut-off proportion
Result: [e(nenasee estimated confusion matrices; (Bt): tasks' aggregated label
Initialization: Get confusion matrix estimation { ) [mmaaaee
for j E [woorer]] do
Train a neural network on D(f) = { ( v! for i E T(w))) for epochs
Compute (UM( V/ ( ((G))
train using Equation (2) for i E ((:s)
Compute trust scores 5660 (II) using Equation (4) for i E T(w;)
for each task E Xerain do
Compute (AUM(()) using Equation (3)
Get ga the WAUM's quantile threshold of order a of (WAUM(z)))iefraaa))
Define Pprazed = { (T). (v())
:WAU((>)) 2 qa for E [meask] )EAtIa))
10 Compute { )E (Nvorsss)) tasks in Promed
I1 Soft labels: ÿJ- (B.e E Ax-1 with di = ( Σ
8i
#(6) 1. (o,0)= KE(R) "k. sEd(1;)
for all x4 E Rigraned
EXPERIMENTS
In our experiments, we investigate simulations and the CIFAR-IOE dataset (Peterson et al., 2019) 19), a crowdsourced dataset with both tasks and workers labels openly accessible. For each aggregation schemes considered, we train a neural network on the soft (except for MV) labels obtained after the aggregation step. We compare our WAUM scheme with MV, Naive Soft, DS and GLAD; see Appendix A for an overview of the methods and Appendix D for details and additional simulations.
Metries investigated After training with the aggregated labels, we report the following perfor- mance metrics on test set Dreat (available for simulations and for CIFAR-1OH): top-1 accuracy and expected calibration error (ECE) (with M = 15 bins as in Guo et al. (2017), see Equation in Ap- |Pmal 1 [arg [aargmax max @== }- i.e. the accuracy of the aggregation method on the training set's true labels (available for simulations and for CIFAR-10H). When dealing with hard labels, the arg max can simply be omitted in the definition of Acciminn For the WAUM, the train accuracy is computed on DD prunee instead of Drain as this method does not label tasks detected ambiguous. All results are averaged over 10 repetitions.
pendix C). We also report the train accuracy: Accurin (y", () = |Puiill
1
Implementation details For simulated datasets, the training is performed with three dense lay- ers' artificial neural network (30, 20, 20) with batch size set to 64. Moreover, workers are simu- lated with scikit-iiiicccccc tlan rn (Pedregosa et al., 2011). For with CIFAR-IOH the Resnet-18 (He et al., 2016) architecture is chosen for simplicity (with batch size also set to 64). To optimize the parameters, we consider an SGD optimizer over 150 training epochs, with initial learning rate of 0.1, decreasing it by a factor 10 at epoch 50 and 100. Other hyperparameters for Pytorch's (Paszke et al., 2019) SGD are moment um=0. and weight..ecay=teeeee generalization performance considering feature-bling We do not use data augmentation since duplicating mislabeled or poorly labeled tasks can artifi- cially worsen performance (Harutyunyan et al., 2020; Nishi et al., 2021), see Appendix D. All experiments are executed on servers equipped with an Nvidia RTX 2080 and Quadro T2000 GPUs.
Our goal is to observe the aggregation strategies against the WAUM.
Simulated binary dataset: two two.circles. We simulate alaak 500 points with scikit- learn's function two_circles (with noise s = 0.2 and scaling factor 0.4). Our Tlorker 3 workers are standard classifiers (see details in Appendix D): a linear Support Vector Machine Classifier (linear SVC), an SVM with RBF kernel (SVC), and gradient boosted classifier (GBM) with five estimators. Data is split between train (70%) and test (30%) and each simulated worker votes for each task, i.e,, for all æ Xrain, AA(z) Tworker- Figure 3 illustrates that the AUM identifies tasks misclassified or close to the decision change, for each worker. The WAUM indicates
*Code is available in the supplementary material; an online repository will be relcased for the conference. /ttpps///oot kit- www.n.org/eeeoeeeeeoo...o. at abl a /moduul as /geeerat /d/sk baann.dataeee makee_tt es.

---

Under review as a conference paper at ICLR 2023
few potentially ambiguous tasks (low value) and peak simple ones (high value), Figure 3, right part. Tasks with large WAUM are located mostly on the west/south-weet of the two circles separation, Le., where workers disagree the most. Thanks to soft labels and DS confusion matrices estimated on pruued, WAUM achieves better test accuracy and expected calibration (in terms of ECE), see Table 2.
Wh Linear SVC
MP2 SVC
Wh
Ground truth
GBM
WAUM
2
1.5
90.1
0.5
AUM
0.2 0.460.8
0.5
0.8
0.6
0.5
0.4
5
0.25
Figure 3: two.circles: simulated workers (w1, and wa) with their associated AUM and normalized trust scores (left) and associated WAUM distributions (right) for 0.1.
Ground truth
MV
Naive soft
DS
GLAD
WAUM
Test Train (hard labels) (hard labels)
Figure 4; two_ci rrleesrcees: label predictions on train/test sets provided by a three dense layers' artificial neural network (30, 20, 20) trained smooth labeled obtained by various aggregation strategies. Points in red pruned from training by WAUM (here 0.1). In this binary case, we display binary labels, but soft alternatives can also be visualized (ef. Appendix subsection D.2).
Table 2: Aggregation schemes on two_ci cccerserrsees............ooo (Neass 500 tasks, [A(x)) Mearker 3. For reference, best worker is ut (GBM) with training accuracy of 0.960 test accuracy of 0.880.
Aggregation
Actiois
Test accuracy
ECE
MV
0.889
0.847 + 0.004
0.120 0.006
Naive soft
0.810 0.017
0.091
0.015
DS
0.871
0.819 = 0.006
0.127 0.010
GLAD
0.889
0.847 1 0.003
0.123 0.006
WAUM(a = 100
0.891
0.855 = 0.001
0.054 0.006
WAUM(a 2)
0.011
WAUM(a 1001)
0.930
0.863 0.012
0.075 0.013
WAUM (a 0.25)
0.962
0.837 1 0.016
0.082 0.012
Simulated multiclass dataset: Carre.circces.. We adapt the previous example for K classes, simulating 250 tasks per class (πeask = 750), and similar workers, each classifying all training tasks. The disagreement area is identified in the north-west seen in Figure 5. Table also shows that pruning too little data (a small) or too much (a large) can mitigate the performance.
of the dataset can be

---

Under review as a conference paper at ICLR 2023
Ground truth
Linear SVC
SVC
GBM
WAUM
2
90.1
1
AUM
0.5
0.5
0.8
0.6
0.4
0.2
0.25
Figure 5: three circles: simulated workers (w), we and wy) with their associated AUM and normalized trust scores (left) and associated WAUM distributions (right) for 0.1.
Ground truth
MV
Naive soft
DS
GLAD
WAUM
Test Train (hard labels) (hard labels)
Figure 6: thres_circles: label predictions train/test sets provided three dense layers' artificial neural network (30, 20, 20) trained on smooth labeled obtained by various aggregation strategies. Points are pruned from training by WAUM (here a 0.1).
Table 3: Aggregation schemes on three c circles (πaa. = 7500 tasks, IA(z)) Nworeer = 3). For reference, the best worker is (GBM) with training accuracy of 0.920 and a test accuracy of 0.836.
Aggregation
Accem
Test accuracy
BCE
0.794
0.727 0.026
Naive soft
0.794
0.697 0.018
0.178 + 0.016
DS
0.747
0.753 0.072
0.220 + 0.077
GLAD
0.549
0.578 0.023
0.356 1 0.024
WAUM(a = 3)
0.891
0.781 4: 0.042
0.179 1 0.014
WAUM(a = 10 2)
0.894
0.783 +: 0.055
0.181 + 0.020
WAUM(a 100
0.936
0.806
0.051
0.186 1 0.033
WAUM(a 0.25)
0.949
0.790 0.042
0.183 1 0.025
CIFAR-1OH. We consider now the CIFAR-10H dataset Peterson et al. (2019). The training part of CIFAR- 10H consists of the 10000 tasks of the test of the original CIFAR-10 dataset (Krizhevsky & Hinton, 2009). In total Nyorker = 2571 workers participated on the Amazon Mechanical Turk platform, each labeling 200 images (20 from each original class), leading to approximately 50 an- swers per task. We have randomly extracted 500 tasks for validation set (hence |Duainl = 9500). The final weights output by the network are the one achieving the lowest cross-entropy loss on this set. We test our model on [Dres 50K tasks of the original (hard labeled) train dataset of CIFAR- 10. This dataset is notoriously more curated (Aitchison, 2021) than common dataset in the field (but openly available with both votes and tasks): most difficult tasks were identified and removed at the creation of the CIFAR-10 dataset, resulting in few ambiguities. Table 4 shows that in this simple setting, our data pruning strategy is still relevant, with the choice 0.01. A visual
is the only
8

---

Under review as a conference paper at ICLR 2023
influence of the hyperparameter is available in Appendix D.2 Figure 10. Also notice that in this case, the GLAD algorithm is highly impacted and performs worse than a majority voting.
Furthermore, the WAUM leads to better generalization performance than vanilla DS model or DS model with spammers identification. However, due to the few ambiguous tasks, using naive soft labels can lead to close results for the WAUM, with slight but consistent gain final calibration error. Note that vanilla DS slightly underperformed compared to other aggregation schemes, but using the WAUM we obtain both confusion matrices from DS and aggregated labels with competitive performance.
side on the
Table 4: Label recovery, generalization performance and calibration error on the CIFAR-1OH dataset by a Resnet-18 (here a 0.01, removing on average 95 tasks).
Aggregation method
accuracy
AcCmii
MV
69.533 = 0.84
0.175 1: 0.00
99.2
Naive soft
72.149 2.74
0.132
0.03
99.2
DS (vanilla)
70.268 = 0.93
0.173 + 0.00
99.3
DS (spam identification)
70.053
= 0.01
99.3
GLAD WAUM
66.569 8.48
0.173
99.2
72.747 1.93
0.124 4: 0.00
99.2
We observe in Table 4 that with labels collected in CIFAR-IOH, simple aggregation methods already perform well. Over the 2571 workers, less than 20 are identified as spammers using Raykar & Yu (2011), but remind that most difficult tasks were removed when creating the CIFAR-10 original dataset. We refer to the CIFAR-10 --0 dataset collection procedure described in the "labeler instruc- tion sheet" of Krizhevsky & Hinton (2009, Appendix C) for more information about the written incentives given to workers.
5 CONCLUSION AND FUTURE WORK
In this paper, we empirically investigate crowdsourcing aggregation models and how judging sys- tems may impact generalization performance. Most models consider the ambiguity from the work- ers' perspective (very few consider the difficulty of the task itself) and evaluate workers on hard tasks that might be too ambiguous to be relevant, leading to performance drop. Using popular model (DS) and small architectures' classifiers, we develop the WAUM, a feature aware metric that improve generalization performance. It also yields fairer evaluation of workers' abilities and sup- ports recent research on data pruning in supervised datasets. Independently of pruning, the WAUM allows identifying early the images that need extra labelling efforts, or that cannot be correctly labeled at all Extension of the WAUM to more general learning tasks (e.g., top-k classification) would be natural, including sequentially labeling tasks. Indeed, the WAUM could help to identify tasks requiring additional expertise, and guide how to allocate more experts/worken for such iden- tified tasks. Future works need to improve computational efficiency of the WAUM the vanilla algorithm requires to train one neural network per worker as initiated with our stacked version (see Appendix E.3). Moreover, adapting the WAUM to imbalanced crowdsourced datasets of interest to identify potentially too ambiguous images that naturally occurs in open platforms like PI@ntNet'. Last but not least, on the dataset side, providing a challenging dataset (such as the one by Garcin et al. (2021) for instance) to learn in crowdsourcing settings is still missing. Indeed, dataset with the following properties could greatly foster future research in the field: 1) a varying number of labels per worker, high number of classes and 3) a subset with ground truth labels to test generalization performance.
REFERENCES
Laurence Aitchison. A statistical theory of cold posteriors deep neural networks. In ICLR, 2021.
Isabelle Albert, Sophie Donnet, Chantal Guinenneuc-Jouyaux, Samantha Low-Choy, Kerrie Mengersen, and Judith Rousseau. Combining expert opinions in prior elicitation. Bayesian Anal- ysis, 7(3):503-532. 2012.
ptttps://plantnet.oo

---

Under review as a conference paper at ICLR 2023
Anelia Angelova, Yaser S. Abu-Mostafa, and Pietro Perona. Pruning training sets for learning of object categories. In CVPR, volume I, pp. 494-501 vol. 1, 2005.
A Lord Birnbau,, Some latent trait models and their use in inferring an examinee's ability. Statis- lical theories of mental test scores, 1968.
Michael PJ Camilleri and Christopher KI Williams. The extended Dawid-Skene model. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 121- 136. Springer, 2019.
AP Dawid and AM Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. J. R. Stat. Soc. Ser. C. Appl. Stat., 28(1))20-28, 1979.
Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. A checklist to combat cog- nitive biases in crowdsourcing. AAAI Conference on Human Computation and Crowdsourcing, volume 9, pp. 48-59, 2021.
Chao Gao and Dengyong Zhou. Minimax optimal convergence rates for estimating ground truth from crowdsourced labels. arXiv preprint arXiv: 2013.
C. Garcin, A. Joly, P. Bonnet, A. Affouard, J.-C. Lombardo, M. Chouet, M. Servajean, T. Lorieul, and J. Salmon, Pl@ #0@ntnet330kk: a plant image dataset with high label ambiguity and a long-tailed distribution. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Pascal Germain, Alexandre Lacasse, Francois Laviolette, Mario Marchand, and Jean-Francis Roy. Risk bounds for the majority vote: from a pac-bayesian analysis to a learning algorithm. J. Mach. Learn. Res., 16:787-860. 2015.
Guo, G Pleiss, Y Sun, and KQ Weinberger. On calibration of modern neural networks. In ICML, pp. 1321, 2017.
Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In ICCV, pp. 5138-5147, 2019.
2019.
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noise information in neural network weights. In ICML, pp. 4071-4081, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Le-Nguyên Hoang, Louis Faucon, Aidan Jungo, Sergei Volodin, Dalia Papuc, Orfeas Liossatos, Ben Crulis, Mariame Tighanimine, Isabela Constantin, Anastasiaa Kucherenko, Alexandre Maurer, Fe- lix Grimberg, Vlad Nitu, Chris Vossen, Sébastien Rouault, and El-Mahdi El-Mhamdi. Tournesol: A quest for large, secure and trustworthy database of reliable human judgments. arXiv preprint arXiv:2107.07334, 2021.
Shahana Ibrahim, Xiao Fu, Nikolaos Kargas, and Kejun Huang. Crowdsoureing via pairwise occurrences: Identifiability and algorithms. Advances in neural information processing systems, 32, 2019.
Hideaki Imamura, Issei Sato, and Masashi Sugiyama. Analysis of minimax error rate for crowd- sourcing and its application worker clustering model. In ICML, pp. 2147-2156, 2018.
Gareth Michael James. Majority vote classifiers: theory and applications. PhD thesis, Stanford University, 1998.
Emily Jamison and Iryna Gurevych. Noise additional information? leveraging crowdsource annotation item agreement for natural language tasks. In Conference on Empirical Methods in Natural Language Processing, pp. 291-297, 2015.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates to support personalized medicine. J. Am. Med. Inform. Assoc., 19(2):263-274, 2012.
10

---

Under review as a conference paper at ICLR 2023
Cheng Ju, Aurélien Bibaut, and Mark van der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. J. Appl. Stat., 45(15):2800- 2818, 2018.
Ece Kamar, Ashish Kapoor, and Eric Horvitz. Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing, 2015.
Faiza Khan Khattak. Toward Robust and Universal Crowd Labeling Framework. PhD thesis, Columbia University, 2017.
Hyun-Chin Ch-u Kim and Zoubin Ghahramani. Bayesian classifier combination. In AISTATS, volume 22, pp. 619-627, 2012.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In NeurlPS, volume 32, 2019.
Maksim Lapin, Matthias Hein, and Bernt Schiele. Loss functions for top-k emoor Analysis and insights. In CVPR, pp. 1468-1477, 2016.
Qunwei Li and Pramod K Varshney. Does confidence reporting from the crowd benefit crowdsoure- ing performance? In International Workshop on Social Sensing, pp. 49-54, 2017.
Qianqian Ma and Alex Olshevsky. Adversarial crowdsourcing through robust rank-one matrix com- pletion. Advances in Neural Information Processing Systems, 33:21 841228822 2020.
Yao Ma, Alex Olshevsky, Venkatesh Saligrama, and Csaba Szepesvari. Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers. The Journal of Machine Learning Research, (1(1):5245-5288) 2020.
Rafael Müller, Simon Korblithh and Geoffrey E Hinton. When does label smoothing help?
NeurlPS, 32, 2019.
Kento Nishi, Yi Ding, Alex Rich, and Tobias Hollerer. Augmentation strategies for learning with noisy labels. In CVPR, pp.
2021.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. J. Artif. Intell. Res., 70:1373-1411, 2021a.
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test destabilize machine learning benchmarks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021b.
Satoshi Oyama, Yukino Baba, Yuko Sakurai, and Hisashi Kashima. Accurate integration of crowd- sourced labels using workers' self-reported confidence scores. In IJCAI, 2013.
Seo Yeon Park and Cornelia Caragea. On the calibration of pre-trained language models using mixup guided by area under the margin and saliency. In ACML, pp. 5364-5374, 2022.
Rebecca J. Passonneau and Bob Carpenter. The benefits of model of annotation. Transactions of the Association for Computational Linguistics, 2:311 1:311-326, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, pp. 8024 8035. 2019.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding important examples early in training. In NeurIPS, volume 34, pp. 20596-20607, 2021.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Couraapeau, M. Brucher, M. Perrot. and E. Duchesnay. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res., 12:2825-288 2830,, 2011.
11

---

Under review as a conference paper at ICLR 2023
JC Peterson, RM Battleday, and TL Griffithsand O Russakovsky. Human uncertainty makes classi- fication more robust. In ICCV, pp. 9617-9626, 2019.
Geoff Pleiss, Tianyi Zhang, Ethan R Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. In NeurIPS, 2020.
Ravi Raju, Kyle Daruwalla, and Mikko H. Lipasti. Accelerating deep learning with dynamic data pruning. CoRR, abs/21 11. 11211111.222111 12621, 2021.
Vikas C Raykar and Shipeng Yu. Ranking annotators for crowdsourced labeling tasks. In NeurlPS, pp. 1809-1817, 2011.
Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. In Proceedings of the AAAI conference on artificial infelligence, volume 32, 2018.
Maximilien Servajean, Alexis Joly, Dennis Shasha, Julien Champ, and Esther Pacitti. Crowdsoure- ing thousands of specialized labels: A bayesian active training approach. IEEE Trans. Multimed., 1966)1137-1139, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Vaibhav B Sinha, Sukrut Rao, and Vineeth N Balasubramanian. Fast Dawid-Skene: A fast vote aggregation scheme for sentiment classification. arXiv preprint 1rXiv:1803.02781 2018.
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Conference on Empirical Meth- ods in Natural Language Processing, pp. 254 263.. Association for Computational Linguistics, 2008.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206. /4486, 2022.
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W. Dusenberry, Jasper Snoek, Balaji Lak- shminarayanan, and Dustin Tran. Combining ensembles and data augmentation can harm your calibration. In ICLR, 2021.
J Whitehill, Wu, Bergsma, J Movellan, and Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NewrlPS, volume 22, 2009.
Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In ICML, pp. 10727- 10735, 2020.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- risk minimization. In ICLR, 2018.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recog- nition. In CVPR, pp. 16489-16498, 2021.
Dengyong Zhou, Qiang Liu, John C Platt, Christopher Meek, and Nihar B Shah. Regularized minimax conditional entropy for crowdsourcing. arXiv preprint 103t::1503.07240. 2015.
12

---

Under review as a conference paper at ICLR 2023
A
POPULAR LABEL AGGREGATION TECHNIQUES
Hereafter, we recall
Dirain
U
Tii (v! for j A(ci))
(1 revisited)
1=1
Thorreer
{
(1, (v!")) for E T(us)
D(1) train
U...
(Pocrkar)
(5)
train
In practice, we usually aggregate the crowdsourced labels from Pusin into a probability distribution (soft labels) over the possible classes to train a classifier. For the i-th task TG, we write ÿn for a soft label associated to zę. The loss used to train a (neural network) classifier must be adapted these soft labels. Here, we have only considered the cross entropy loss:
H( Pc)
Σ kE[K]
(Un)s log(p;))
between the aggregated label 0; and the associated predicted probability p;. In our experiments, we have only investigated p; softmax(I;)). i.e., the softmax output associated to the task z;,-
In this paper, we compare several aggregation strategics described thereafter: Naive soft, Majority Voting (MV), Dawid and Skene (DS), GLAD, and compared it to our WAUM aggregation.
A.1
NAIVE SOFT
The naive soft labeling consists on computing the distribution of answered votes:
with
T
Σ jEA((,))
1 =1) kE[K]
ELE[RY (7:)
A.2 MAJORITY VOTE
Majority voting simply consists on choosing the most answered label (and can be seen as an arg max of the naive soft):
Σ JEA(wEs)
U: = arg max RE []
A.3 DAWID AND SKENE
The Dawid and Skene (DS) (Dawid & Skene, 1979) model aggregates answers and evaluates the workers' confusion matrix to observe where their expertise lies exactly. Let us introduce Pe the prevalence of each label in the dataset, i.e., the probability that task drawn random has label É E [K]. Following standard notations, we also write (Tie, i E [Mrask]} the indicator variables for task that is 1 if the task has label y² = l. Finally, let x ( be the probability for worker j to select label k when y" = L. The model's likelihood is:
The
II iE(nesse EE[K]
II
Pe II
II
! TT
(6)
jE [E[TNarter]]
KE[K]
Using the EM Algorithm 2, obtain maximum likelihood estimations of (1) and pe for j [Twozkez] and & E [K]. The convergence criterion is often chosen as the likelihood's variability between two iterations.
The DS model has been adapted to different settings (sparsity, worker clustering, accelerations) (Servajean et al., 2017; Imamura et al., 2018; Sinha et al., 2018) to overcome limitations mostly due to the estimation the confusion matrices size K² for each worker. We discuss these possible alternatives in Appendix E.
13

---

Under review as a conference paper at ICLR 2023
Algorithm 2: EM algorithm DS model.
Data: vain: crowdsoureed dataset
Result: { ] estimated confusion matrices; (Us (() tasks soft labels
Initialization: Vi E Meaaa] V4 E [K], The [Taa Ejed( 1 =x(
while Convergence not achieved do 11 M-step: Get क and asssmmin I's known v(e,k) E [K]2,
*(f) "Ek
Exelneast Tie EKejK] Edelear Tie
vE E [K] PE 4 1) Dielncnul The
Beaak
1/ E-step: Estimate Ts with current and p
Vi [Reask), vl V [K], Tix = [E [eς] fr +0))
EremmIIyeees Heegg Per (492)
Labels: Vi [Meask], D: Tio R
A.4 GLAD
We recall the GLAD (Whitehill et al., 2009) algorithm in the binary setting A modeling assump- tion is that the j-th worker labels correctly the i-th task with probability given by
Fly!" = yf ay 8,) Ite Ite
with a, E the worker's expertise: aj implies misunderstandinng a, an impossibility to separate the two classes and oj > 0 valuable expertise. The coefficient 1/8; E TR., represents the task's intrinsic difficulty: if 1/ -> the task trivial; on the other side when 1/8 + +00 the task is very ambiguous. Parameters (a:))e[[nozzzs] and (3.)ie[mease) estimated using an EM algorithm as described Algorithm 3.
Algorithm 3: EM algorithm GLAD model.
Data: D crowdsourced dataset
Result:
{ax [Presaeess worker abilities, 8 = (B: (e(aaa) task difficulties,
(ū (ū))etnnaaal aggregated labels
while Convergence not achieved do E-step
Estimate probability of of
Vi [Reask], P(y Kv ).,
a, 0.) xc P())) II, P(y) (1) lu a..
M-step
// Maximization
Maximize auxiliary function ((a, 3) in Equation (7) with respect to a and 3
The auxiliary function for the binary GLAD model is:
(( B) E[oog P ((x (1) Jef, tut }+)
Σ Elog P(t))++
Σ Ellog (o B.)
Following Whitehill al. (2009), denote x² Plyt kl kl Jesr ar B), then one obtain partial
derivatives of the function Q with respect to or and B, as follows:
aQ
((v'v(")
vi + + ) P(u!
vilag,
da,
0Q = Ew''v + + - ( P(v (:)a,,;))aa
0a,
Setting the partial derivatives of the function Q to zero, one can solve iteratively the nonlinear system of equations. See Whitehill al. (2009)]Su material] for additional details.
we provide few details the end of the paragraph for multi-class settings
14

---

Under review as a conference paper at ICLR 2023
An extension to the multiclass setting is given by Whitehill et al. (2009) under the following assump- tion: the distribution over all incorrect labels is supposed uniform. However, this is not verified in many practical cases, can be seen for example in Figure 2 where the deer label is only mistaken with labels referring to other animals and not with vehicles. We have used the implementation from https: / / thub. com/ /nnonntann h/ppythooogggg poon-llad ad evaluate the GLAD performance in our experiments. Instead of setting the partial derivatives to zero and solving the nonlinear system, it maximizes the function Q with respect a and / using the conjugate gradient algorithm.
B ADDITIONAL DETAILS ON AUM AND WAUM
AUM computation in practice. We recall in Algorithm 4 how to compute the AUM in practice for a given training set Duain- This step is used within the WAUM (label aggregation step). Overall, with respect to training model, computing the AUM, requires an additional cost: T training epochs are needed to record margins for each task. This usually represents less than twice the original time budget. We recall that softmax (z:) ) is the softmax output of the predicted scores for the task æ; at iteration t.
Algorithm 4: AUM algorithm.
Data: Duain = (I'; V:)se(n..m)) training set with Ntaak task/label couples, T: total epochs. Result: (AUM( (E) tasks' AUM.
for == 1....,T do
Train the neural network for epoch using Dinin for i [Meask] do
Record softmax output softmax (ars) E AK-1
Compute margin M( (1;; Vi) = softmax i1 (x,) - softmax (1) (3x)
Vi e [Massx], AUM(I, y4: Dtrai) = T Ece(Ty ( (21,
Example (Confusion matrix update). The WAUM procedure estimates each worker's confusion matrix using DS-like model first on Dirain and then on DD Promed- A naturally arising question is then: does pruning these few tasks selected with lowest WAUM really has an impact on the esti- mates matrices 703) ? Let us go back to the three circles simulated dataset from Section 4 and (1) compare the confusion matrices on Drain denoted Troin and on Drraned denotes p for our three simulated workers. We also represent in Figure 7 the variation between the estimated confusion matrices on Dinin and Promed- Figure 7 shows that on the three_ca rcles dataset, removing am- biguous training data with cut-off parameter error rates in the confusion matrices. It decreased the confidence in wy to find the innermost class, and increased the confidence in the best worker W's.
0.1 lead significant changes in the estimated
Example 2 (Trust factors). Assume K 3, Tookker = 2 and confusion matrices:
1
0
0
0.5
0.5
# (11) =
0.5
0.5
0
and * (2)
0.5
0.5
0
0.5
0.5
0
Given a task x answered by both workers and an arbitrary neural network trained with T epochs then (z) = 0.75 and g(²) = 0.25. Thus. the averaged margin with the label proposed by worker wi counts three times more than the one from
such that softmax(") (x) (0.5,
worker We as the network ambiguity only comes from the first two labels where worker wi is more competent.
Example 3 (Extreme cases for WAUM aggregations). Aggregation strategies often return soft la- bels, but in a context with low uncertainty, hard label (say Dirac distribution) might be popular too. With high uncertainty, the estimated label distribution tend to be uniform over the K classes. Such phenomenon naturally arises with WAUM aggregation. In the following we illustrate when these extreme cases occur in the binary setting.
Using Algorithu 1, the aggregated label ğ of task in Dpruned is Dirac distribution (perfect consen- sus) if all workers, independently of their confusions, answer the same label. On the other extreme,
15

---

Under review as a conference paper at ICLR 2023
Powe
H6A
PPo Pawed
Figure 7: Confusion matrices the three simulated workers on the three_ci chree_circles reles datasct before and after pruning tasks selected with lowest WAUM with a = 0.1.
soft labels representing uniform distributions can be obtained by many configurations. In a binary setting (K- = -2), picking Tworker = 3 such that y( =y = 1, y and
# (1) 0.25
0.75
#22) 0.25 0.75
and (3)
0.5
0.5
0.4 0.6
0
1
results in uniform soft labels: ÿ = 22 2) E Aj. Similar compensations occur for > 2.
B.1 AUM AND WAUM VARIATIONS
In this part we show the impact of the choice of the criterion for identifying the difficulty of a task, or the likelihood that label is wrong. For that we compare on CIFAR-IÓEE four variants of AUM for identifying wrong or ambiguous labels. Each line represents one of the 10 classes in CIFAR-1OH, where the images are the 10 worst images by class for each AUM variant.
our proposed AUM as described in Equation (2) following Yang & Koyejo (2020)
the original AUM by as described in Pleiss et al. (2020, Eq. (1))
the WAUM using our proposed formulation of the AUM and following Yang & Koyejo (2020)
the WAUM using the original AUM by described in Pleiss al. (2020, Eq. (1))
The images surrounded by a box Figure 8 are possible wrong or undecidable labels accord- ing to our personal judgment. The conclusion are as follows. First, note that the AUM variants without using the crowdsourced labels are pretty similar for our proposed and the original (only one image detected wrong ambiguous, for the original, cf. Figure 8c, and none for Figure 8a). For the WAUM variants both criterion provided many more discoveries labels (seven and eight respectively for the WAUM using our proposed formulation of the AUM, see Figure 8c and following Yang & Koyejo (2020) and for the one using the original formulation, Figure 8d).
wrong or ambiguous
C REMINDER ON CALIBRATION OF NEURAL NETWORKS
Hereafter, we propose a reminder on neural networks calibration metric defined in Guo et al. (2017). Calibration measures the discrepancy between the accuracy and the confidence of network. In this context, we say that a neural network perfectly calibrated if it is as accurate as is confident. For each task x Xrain {ø1,.... Bresa }. let recall that associated predicted probability distribution is provided by softmax(x) AR A-11- Let us split the prediction interval [0, 1] into
bins J1,...; Jas of size 1/M: I.. (M m Following Guo et al. (2017), we denote B,n
16

---

Under review as a conference paper at ICLR 2023
(a) Our proposed AUM described in Equa-
(b) Original AUM, described in Pleiss al. (2020)
tion (2) following Yang Koyejo (2020)
(c) The WAUM using proposed formulation of the AUM and following Yang & Koyejo (2020)
(d) The WAUM using the original AUM by as descrihed by Pleiss (2020)
Figure 8: CIFAR- 10H: Worst images detected by class, for four variants of AUM. Each line represents for each of the 10 classes, the 10 worst AUM WAUM: a) proposed AUM described in Equation (2) following Yang & Koyejo (2020) b) original AUM by Pleiss al. (2020), e) WAUM using our proposed formulation of the AUM and following Yang & Koyejo (2020) and d) the WAUM using the original AUM by as deseribed in Pleiss et al. (2020, Eq. (1)). The images surrounded by red box are possible wrong or undecidable labels
(x E Ximi sy E Im the task whose predicted probability in the m-th bin?. We recall that the accuracy of the network for the samples in Bm. is given by:
aco (Bm)
Σ
Bm
s
We define the empirical confidence the same B,n by:
(Bm)
Σ
softmax(y((x))
Bm
Remember that with our notation softmax arg max (softmax(z)) with ties broken random.
17

---

Under review as a conference paper at ICLR 2023
Finally, the expected calibration error (ECE) reads:
M
Bm
ECE-
acc((m) - conf(Bm))
(8)
Theak
m==1
neural network said perfectly calibrated ECE = 0, thus if the accuracy equals the confidence for each subset Bm.
D LEARNING ON SIMULATED CROWDSOURCED DATASETS
In this section, we provide some details about the experiments on simulated datasets two_ci rrcles and three.circles described in Section 4. We also consider two extra experiments showing some limitations of the pruning step.
We consider simulated datasets and workers represented by popular classifiers: linear SVC, an SVC with RBF kernel and a gradient boosted classifier (with five estimators). To induce more ambiguity (and avoid too similar workers), one-shot learners are classifiers with maximum iteration set to in the learning phase. Other hyperparameters are set to
's default values
The two.circles dataset is simulated with the make from scikit -eattnleann... The noise 0.2 and the factor is set to 0.4. For the three cireleesssssss 's dataset, we use the same scikit-learn function twice, with different factors: 0.3 and 0.7 creating each circle ring.
D.1 LABEL NOISE AND DATA AUGMENTATION
The classification dataset t two_cirell is also generated using indicating the standard deviation of the noise added to each task. We also factor of 0.4 to scale between the two circles. We generate Reask = 500 tasks addressed by our workers. Workers are simulated as in Section with answers stated in Figure Here, show with the simulation setting of Section 4 that data augmentation with noisy tasks could hurt the test accuracy.
with = 0.2
Each task Xurai is labeled by all workers, and augmented by adding Gaussian noise with probability p E {0.1, 0.25, 0.5, 0.75) as follows (on each batch):
Taug (O0
We then train three dense layers' artificial neural network (30, 20, 20) neural network aggre- gated soft labels from naive soft and WAUM methods and compare the test accuracy in Table 5. Duplicating much noisy labels learning can thus harm the generalization performance.
Table 5: Test accuracy after learning with augmented data the two_circles dataset with Ytack 500
tasks and A(zx)] = Thooree by a three dense layers' artificial neural network (30, 20, 20). Each ir E Xirain is augmented with probability p. Results are averaged over 10 repetitions.
0.1
0.25
0.5
0.75
Naive soft
0.810 0.02
0.619 0.01
0.392 0.02
0.536 + 0.00
0.482
WAUM(a = 0.01)
0.863 = 0.01
0.795 + 0.01
0.782 + 0.01
0.736 - 0.01
0.684 0.01
D.2 OTHER SIMULATED CROWDSOURCED DATASETS
We provided here more datasets to illustrate some aspects of our proposed WAUM, as well as its limits. In all cases, the data split into train (70%) and test (30%) sets.
The two_circles dataset, as introduced in Section 4, highlight the ambiguity visual- ization.
For instance, the squared houree--hnnge loss function is penalized with es regularization with strength set to for the linear SVC and the SVC), the gradient boosted classifier uses the multinomial deviance as loss, and the maximum depth (number of nodes trees) defaults 3.
18

---

Under review as a conference paper at ICLR 2023
The three.cireie dataset, introduced in Section 4, show the visual impact of the parameter a on the WAUM.
The make_clastn f ication dataset, that consists on two clusters. provides a case where pruning is not relevant and all methods achieve similar generalization.
The two_moors dataset with an intrinsic difficulty that is highly relevant for the final decision, this setting, pruning is unadvised as can hurt generalization performance.
The first dataset does not present any significant difficulty thus leading to similar performance aggregated labels. The goal with the tw moons dataset to showcase a limitation of the WAUM: removing data should not be done the cost of removing inherent structure. We recall that the three simulated workers are a Linear Support Vector Classifier (Linear SVC), a one-shot radial basis function SVC (SVC) and gradient boosting classifier using trees with five estimators (GBM).
The two circles dataset: ambiguity visualization. The workers' answers here are un- changed from Figure This part shows the ambiguity in the soft labels estimated with Daain using the different aggregation strategies. We also represent the ambiguity in the output distribution. Each point represents a task H,, and the associated label y, is represented with the color. Indeed, ÿt A1, thus knowing the probability belong in one class, the other probability can easily be inferred.
Ground truth
MV
Naive soft
DS
GLAD
WAUM
(hard labels)
Train
0.8
Train (soft labels)
0.6
Test (soft labels) (hard labels)
Test
0.4
0.2
Figure 9: twe. ccreesrr: soft labels (probabilities) predictions train/test sets provided by three dense layers' artificial neural network (30, 20, 20) trained smooth labeled obtained by various aggregation strate- gies. Points color is the probability to belong in class 1. This lets us visualize the ambiguity in the soft training aggregated lahels, but also in the resulting predictions by the neural network.
are pruned from training by WAUM (here 0.1). Each point represents a task æ,, and its
Figure shows that most training labels with DS and GLAD do not induce much ambiguity. This lack of ambiguity is then reflected in the test output with more confident probabilities. However. WAUM labels have less uncertainty than the Naive soft ones, especially the south-west re- gion. There, the test probabilities also reflect this confidence, while keeping the uncertainty near the decision boundary similar to the one by the Naive soft strategy. Overall, the WAUM aggregation removed much of the unneeded uncertainty and kept the uncertainty very ambiguous tasks.
19

---

Under review as a conference paper at ICLR 2023
The three circles dataset: influence of a. In Table 3, we show on the three circles simulated dataset that we get better performance with the WAUM using hyperparameter a = 0.1. We visually compare the influence of this quantile hyperparameter the pruning, in Figure 10.
Ground truth
10-3
10
10-1
0.25
Train
Test
Figure 10: Influence of a on the pruning step. Red dots indicate data points pruned from the training set, at
level qu in the WAUM (see line in Algorithm 1). We consider (a E {10 10-2, 10-1 0.255)) The closer is to 1, more training tasks are pruned from the training set (and the worse the performance). The neural network used for predictions three dense layers' (30, 20, 20), as other simulated experiments.
The www.:ccaassfffcaaiiiiooo dataset: a case where pruning is useless. We simulate Ataak 500 tasks using make classi fi c on from sci scikit-leas using two clusters per class (here 2) and split the data in train/test with test size of 0.3. We consider class separation factor of 1.5 on the hypercube. With this dataset, all methods achieve similar performance.
Table 6: Training and test accuracy depending on the aggregation method used for the make_classi wwcclasssficatlii ficatien's dataset with Лtaak 500 points used for training three dense layers' artificial neural network (30, 20, 20). For reference the best workers wy and wy with respective training accuracies 0.786 and 0.790 test accuracies of 0.770 0.660.
Aggregation
necuracy
ECE
MV
0.923
0.907 0,000
0.085 + 0.000
Naive soft
0.923
0.906
0.002
0.160 1 0.012
DS
0.920
0.886 +: 0.000
0.108 + 0.002
GLAD
0.893 0.001
0.076 0.004
WAUM(a =003
0.928
0.897 + 0.006
0.078 = 0.013
WAUM(a 10 10-2)
0.933
0.901 + 0.002
0.078 + 0.012
WAUM(a = -10-1
0.965
0.889 0.007
0.084 # 0.016
WAUM(a 0.25)
20

---

Under review as a conference paper at ICLR 2023
Ground truth
Linear SVC
SVC
GBM
WAUM
2
Q0.1
AUM
0.6
0.5
0.8
0.4
0.25
Figure 11:
Simulated workers with associated AUM and normalized trust scores on the
make classifiaino cessiffcciiinn dataset. The hyperparameter a is set to 0.1.
Ground
MV
Naive soft
GLAD
WAUM
(hard labels)
Train
0.8
Train (soft labels)
0.6
Test (soft labels) (hard labels)
0.4
0.2
Test
Figure 12: Training set and test predictions probabilities by three dense layers' artificial neural network (30, 20, 20) on the make_class ietsssifiaaiin. Points red are pruned from the training set in the WAUM aggregation. The a hyperparameter is set to 0.1. Each point represents task and its color is the probability to belong in class 1. This lets visualize the ambiguity in the soft training aggregated labels, but also the resulting predictions by the neural network.
simulated dataset depending on the label aggregation strategy.
21

---

Under review as a conference paper at ICLR 2023
dataset: where pruning is not recommended The toomsonn sim- The framework showcases the difference between relevant ambiguity in a dataset and artificial ulation
one. This dataset is created using make moons function from scikit-learnh We simulate Deask = 500 points, noise 0.2 and use a test split of 0.3. As can observed with Figure 13 and Figure 14, the difficulty this dataset comes from the two shapes leaning into another. However, this intrinsic difficulty is not due to noise, but is actually inherent to the data. In this case, removing the hardest tasks mean removing points the edges of the crescents, and those important in the data's ' structure. From Table 7, we observe that learning on naive soft labeling leads better performance than other aggregations. But with these workers, no aggregation produced label capturing the shape of the data.
fact
U2 SVC
W33 GBM
Ground truth
Linear SVC
WAUM
2
40.1
AUM
0.5
0.20.40660.8
0.5
0.8
0.6
0.25
0.4
Figure 13: Simulated workers with associated AUM and normalized trust scores on the two_moons dataset. The hyperparameter is set to 0.1.
Ground truth
MV
Naive soft
DS
GLAD
WAUM
Thain (hard labels)
0.8
Train (soft labols)
0.6
Test (hard lahels)
0.4
Test (soft lahels)
0.2
Figure 14: Training set and test predictions probabilities by three dense layers' artificial neural network (30, 20, 20) the two_moons simulated dataset depending the label aggregation strategy. Points in red pruned from the training in the WAUM aggregation. The a hyperparameter is to 0.1. Each point represents task 7;, its color the prohability to belong class 1. This lets us visualize the ambiguity the soft training aggregated labels, but also in the resulting predictions by the neural network.
22

---

Under review as a conference paper at ICLR 2023
Table 7: Training and test accuracy depending on the aggregation method used for the two moons's dataset with Reass 500 points used for training three dense layers' artificial neural network (30, 20, 20). For reference, the best worker is wy with a training accuracy of 0.923 and a test accuracy of 0.900.
Aggregation
Ac0pzn
Test accuracy
ECE
MV
0.917
0.894 0.002
0.098 = 0.004
Naive soft
0.917
0.887 : 0.002
0.217 1 0.010
DS
0.871
0.867 0.000
0.126 + 0.001
GLAD
0.006
0.872 + 0.006
0.107 = 0.004
WAUM(a 10
0.917
0.875 0.002
0.088 = 0.012
WAUM(a 1002)
0.919
0.874
WAUM(a =1
0.926
0.870 0.003
0.101 0.020
WAUM(a = 0.25)
0.946
0.829 + 0.006
0.135 + 0.011
E MODELS LIMITATIONS AND POSSIBLE EXTENSIONS
E.I GLAD LIMITATIONS: ILLUSTRATION ON A CLASSICAL DATASET
Our goal in this section is to show the importance to consider both the tasks and the labels answered when assessing task's s difficulty in an aggregation model. Indeed, we show experimentally that removing ambiguous tasks from training leads to significant accuracy improvements for GLAD. In this experiment, we consider the classical crowdsourcing setting where task is available (i.e. no X. We investigate the performance of the GLAD algorithm to retrieve training labels. Recall that GLAD takes into account both tasks' intrinsic diffieulties and workers' abilities.
The three difficulties dataset. We consider the three difficull crowdsourcing example with binary labels we have adapted from Whitehill et al. (2009). In this dataset, crowd of Rorkee = 50 workers has to label Blaak 1000 tasks. Labels are binary and balanced. Tasks are either easy, hard or random indicating their intrinsic difficulty. Tasks difficulties are drawn with probabilities respectively Peasy 0.53, Paard 0.27 and Prandom 0.2. We thus simulate (y, d);) E {0, 1 x feasy, hard, r random]. We model each worker ability by two levels: they independently chosen good (with probability 0.75) or bad (with probability 0.25). The interplay between task difficulty and worker ability is governed by the confusion matrices presented in Table 8. In this context, the r random tasks represent ambiguous tasks that workers labels by flipping fair coin.
dataset,
Table 8: Confusion matrices simulated with three levels of difficulty. Tasks easy induce no error, random ones are answered fair coin while hard tasks induce more confusion to bad workers than to good ones.
hard (27%)
easy (53%)
random (20%)
good worker (75%)
bad worker (25%)
(1 9)
0.75
0.25
(0.55
0.45
0.5 0.5
0.25
0.75
0.44
0.55
0.5
0.5
We chose a varying number of answers per task to reflect that con-controlled experiments, the number of answers per task can fluctuate. Hence, each tasks' number of answers is drawn uniformly random as |A(x)) U([I..... 10)). ). and then [A(z)] workers are chosen uniformly random among the Mworker. We repeat the experiment 40 times and monitor the training label accuracy obtained by GLAD.
Without pruning the random tasks, the training accuracies lays inside [0.838, 0.885] for 95% of the repetitions. When pruning the random tasks, the accuracy improves and lays inside [0.941, 0.965) for 95% of the repetitions. Thus, identifying such tasks is important to improve training since the accuracy is sensitive to ambiguous task (such as r random tasks).
As displayed in Figure 15a, with GLAD, the difficulty estimated 3 only discriminates easy tasks from the others. Besides, when task has only received single label ( ([A(z)) 1, see Figure 15b) the task's 's difficulty cannot be estimated. Thus all these tasks have nearly the same estimated diffi- culty. Identifying such tasks is thus essential to distinguish between ha rd (necessary in the learning
23

---

Under review as a conference paper at ICLR 2023
aricity
1.00
Trar
30
60
Sorted task index
difficulty
(a) Distribution of the log estimated difficulty (5) depend- ing in each simulated difficulty level: random and hard tasks have similar distributions
(b) Sorted tasks by estimated difficulty (3) using GLAD: tasks with single vote all have nearly the same estimated difficulty
Figure 15: The three di difficulties dataset (Nua.s = 1000, Alockes = 50). Difficulty estimated with GLAD. Associated difficulty levels, workers quality and confusion matrices are presented in Table 8.
process) and r random ones (that could be harmful). We have introduced the WAUM to identify such ambiguous tasks. Note that the neural network output probabilities take into account tasks that only have one vote too. This is why using models with the actual tasks (aa) [ and not only the tasks' labels is crucial in crowdsourcing settings with high variations in the number of votes per task.
act control agent and
E.2 CONFUSION MATRICES ESTIMATION IN THE WAUM
To compute the WAUM, we use the confusion matrices { )eee) jje[noceed (estimated with Algo- rithm 1). The vanilla DS model (Dawid Skene, 1979) can be used to estimate these confusion matrices π()) E I Kxk for each worker Wj. The quadratic number of parameters to estimate for each worker can lead to convergence issues with the vanilla DS model. But as stated in Section 3, any model that estimates confusion matrices can be used in the WAUM's computation.
We detail below some possible variants:
Sinha al. (2018), who have accelerated vanilla DS convergence by constraining the esti- mated labels' distribution T:, to be a Dirac mass. Hence, predicted labels are hard labels, leading worse calibration error than vanilla DS while preserving the same accuracy.
Passonneau Carpenter (2014) who have introduced Dirichlet priors on the confusion matrices' rows and the prevalence p.
Servajean et al. (2017) who have exploits the sparsity of the confusion matrices when the number of classes is high.
Imamura et al. (2018) estimates < Aworker clusters of workers thus constraining at most L different confusion matrices using variational inference. This lead estimate K2 x L coefficients for the confusion matrices instead of K2 x nwozker-
These alternatives to the vanilla DS model could help computing faster the confusion matrices used in the WAUM for the trust score computation.
Iterate confusion matrix estimation. As Algorithm is an iterative procedure estimate confu- sion matrices (66) on less ambiguous tasks, natural question is whether one iteration is enough or if repeating this process improves the performance. We denote WAUMiter the strategy that esti- mates the WAUM with confusion matrices obtained from the first dataset D prumedd
We consider the same workers on the three catre-ccize wiw.cciccc........... circles classification task, but vary the number of votes per task. Each tasks' number of answers is drawn uniformly at random such that 1 < [A(x) < 3. Then, [A(r) workers are chosen among the Nworker ones available in the crowd with probability
0.2 for the linear SVC (that worst simulated workers), and 0.4 for the other classifiers. Workers'
24

---

Under review as a conference paper at ICLR 2023
Table 9: Performance metrics by aggregation method for the three circles dataset with 250 tasks per label and I [A(x)] I Ruorker is 0.3.
by a three dense layers' artificial neural network (30, 20, 20). Test size
Aggregation
Naive soft
DS
GLAD
WAUM
WAUMiter
ACCauin
0.729
0.659 0.644 + 0.027
0.583
0.667
0.753 0.753 + 0.039
Test accuracy
0.728 + 0.018
0.581 = 0.003
0.725 = 0.056
ECE
0.146 + 0.023
0.242 + 0.011
0.196 0.004
0.137 = 0.034
0.156 = 0.023
decisions are unchanged from Figure 5, Table 9 reports the train and accuracy for the aggregation methods considered. DS and GLAD underperform WAUM, whose decision boundary is affected nonetheless. Note that, contrary GLAD, the tasks' difficulty considered by the WAUM can be evaluated thanks to the tasks' features even with a single answer.
In short, taking into accounts tasks' features lets us break through one of GLAD limitations estimating the difficulty for tasks with a single vote. We recall possible alternatives to compute faster the confusion matrix used in the WAUM statistic. Finally, iterating the confusion matrix and scores estimation in the WAUM procedure can improve performance in practice.
E.3 STACKING WORKERS ANSWERS IN THE WAUM
As stated in Section the WAUM computation in Algorithm can be costly in scenarios with many workers, as it requires one neural network per worker to compute each AUM independently. To limit this issue, we propose stacked version. consists in computing the WAUM with Daack :=
{(z;, vi ) (. the dataset obtained by stacking all workers' tasks and labels together. We denote this alternate version WAUMstack. The difference between Algorithm 1 and Algorithm resides in the neural network training happening once on Danck instead of one networks trained per D(G)
train' The AUM and trust scores are still evaluated worker-wise and task-wise. Moreover, using Ditnck can break independence between workers. Indeed, for a given task, the trust score associated to worker j depends on the answers given by all other workers. might not be desirable: when many spammers are present in the crowd, the network's softmax output (on which the scores rely) could be corrupted.
Algorithm 5: WAUMstack computation.
Data: Demini tasks and crowdsoureed labels, E [0, 1]: cut-off proportion
Result: (+0) ] estimated confusion matrices; (úi): tasks' aggregated label
Initialization: Get confusion matrix estimation (=))) ]
Train neural network on the stacked dataset Deack {(1,, (1) } for T epochs
for E [Tuorier] do
Compute AUM( ((()) train using Equation (2) for i E T (w;
Compute trust scores 5(50 (Ire) using Equation (4) for é E ((w;)
for each task x E Xmaa do
Compute (AUM(z) using Equation (3)
Get ya the WAUM's quantile threshold of order a of (WAUM(s)))
WAUM(2;) 2 qu for í E [ncsa]] Detine Dpranss- = {(Z... (v(")))AA())))
)EAA))))
10 Compute { [[(oooceer) on tasks Dprumed
Soft labels: ÿt (B) E AA-- with Ø1
Σ (EA(s))
Tki (u (0 m.)) kE(K)
for all x; Xgramed
We simulate Nw = 150 workers who answer tasks from a dataset with K 4 classes function make_class f ficatio; we call this dataset
simulated using make.classifican i www.mmilss.....m All simulated tasks are labeled by up to five work- among Linear SVCs, SVCs or Gradient Boosted Classifiers (GBM) chosen uniformly. In order to simulate multiple workers with some dissimilarities, we randomly assign hyperparameters for each classifier as follows:
25

---

Under review as a conference paper at ICLR 2023
Linear SVC:
the margin C is chosen in linear grid of 20 points from 10-3 to 3, the maximum number of iterations ranges between and 100,
and the loss function is in (hinge, squared.binge)
SVC:
the kernel is in (poly, rbf, si gmoid] (polynomial is with degree 3), and the maximum number of iterations is between and 100.
GBM:
the learning-rate
{0.01, 0.5,
the number of estimators {1, 2, 10, 15, 20, 30, 50, 100), and the maximum number of iterations between and 100.
All simulated workers are also initialized using different seeds. All hyperparameters are drawn uniformly at random from their respective set of possible values.
(a) Number of answers per task
(b) Number of votes per worker
Figure 16: The make classi ficatiiio on mult i triicass ass dataset. Distribution votes per task (a) and number of vntes for the 150 workers (b) in the simulated classification problem with K classes and 250 tasks per class.
Table 10 shows case with many workers and varying number of votes (Nowrkke 150 and up five votes per task, see Figure 16 for the repartition). In this setting, the stacked version of the WAUM has the same performance as the vanilla WAUM, with much lower computational cost (as do not train Towerkr networks but single one).
Table 10:
The wwtteeelasssfiaaaiiiiiooo mww.uutttslssss dataset: method. The number of tasks is Teask 250 tasks per classes and 1 S [A(z) < 5.
Performance metrics by aggregation
Aggregation
Naive soft
DS
GLAD
WAUM
WAUMstack
AcCunin
0,8428
0.820
0.850
0.858
0.883
Test accuracy
0.851 + 0.00
0.849 + 0.004
0.842 = 0.002
0.849 = 0.006
0. 0.861 + 0.007 0.156 + 0.023
ECE
0.146 0.023
0.242 + 0.011
0.196 0.004
0.137 = 0.034
F CIFAR-101 DATASET
Introduced by Peterson et al. (2019), the crowdsourced dataset CIFAR-IOH attempts to recapture human labeling noise present when creating the dataset. CIFAR-IOH train set consists of the test set of CIFAR-I0. The generalization performance is measured on CIFAR- 10's training set:
10000 and |Dusal 50000
The crowdsourcing experimentation involved nwerker 2571 workers Amazon Mechanical
Turk. Workers had to chose one the ten labels of CIFAR-10 10 for each presented image (no alter- native choice available). Each worker labeled 200 tasks (and was paid $1.50 for that): 20 for each
26

---

Under review as a conference paper at ICLR 2023
original category. Answering time was also measured for each worker. Note that attention check occurred every 20 trials on tasks considered obvious: 14 workers failed a 75% accuracy threshold. The CIFAR-10 dataset is balanced with respect to crowd votes each task has been labelled by 50 workers in average. However, due to the original curation of the dataset (Krizhevsky & Hinton, 2009; Aitchison, 2021) this high number of votes does not imply wide disagreements in the pro- posed labels. Figure 17 shows the distribution of the entropy on the soft label distribution of votes per task:
Σ (1(()=k) SEA(2)
Yor E Xerain; En((x)
Σ kE[
log(ūs) with Σ kE[K]
Σ
(16)=))
JEA(Z)
Count
10a
10a
109
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.35
2.00
Entropy
Figure 17: Frequency of task's naive soft labels entropies (in logarithmic scale) in the CIFAR-10H dataset. Only very few tasks show workers disagreeing. For reference, the maximum entropy is log(K) 2.303.
27

---

Under review as a conference paper at ICLR 2023
G IDENTIFICATION AND AGGREGATION USING THE WAUM
As stated in Section 3.2, the WAUM is primarily a statistic used to identify possibly too ambigu- ous tasks (or even mislabelled ones) thanks crowdsourced labels. In Algorithm 1, we present pipeline from identification to learnable labels using the WAUM. After pruning ambiguous tasks, we chose as aggregation scheme a simple weighted distribution of votes with weights based on expertise obtained by the DS model. Yet, it is also possible to run any aggregation strategy after pruning the tasks flagged "ambiguous".
We show in Table 11 that by considering the WAUM's pruning with DS (resp. GLAD) soft labels, performance improves upon simply using DS alone (resp. GLAD alone). This empirically shows that identifying and removing these tasks can improve the learning efficiency, in agreement with already existing experiments on data cleaning and pruning by Pleiss et al. (2020) for standard su- pervised settings. For example, on the simulated three_circles dataset, the vanilla DS reached 0.753 in test accuracy, but using pruning with the WAUM lead to a test accuracy score of 0.804 (competitive with our first proposed aggregation method). A similar improvement can be observed with the GLAD's s strategy.
We also compare results with Crowdlayer (MW), feature-aware strategy proposed by Ro- drigues & Pereira (2018) to learn from crowdsourced labels using an additional layer on the network. This additional layer used to model each worker's confusion with one confusion matrix per worker (as in the DS model). Note that this strategy does not aggregate labels, thus we can not compute the Aca metric. Note other strategies (Ma Olshevsky, 2020; Ibrahim et al., 2019; Ma al., 2020) are not considered in this paper, due to missing open-source implementations.
Table 11: Training and test accuracy depending on the aggregation method used for the three circles's dataset with 250 tasks class used for training a three dense layers' artificial neural network (30, 20, 20). The hyperparameter is to 0.1 for the WAUM computations.
Aggregation
Accuan
Test accuracy
ECE
DS WAUM + DS
0.747
0.753
0.220
0.887
0.804
0.171
GLAD
0.578
WAUM GLAD
0.569
0.605
0.333
Crowdlayer (MW)
0.640
0.234
WAUM
0.928
0.806
0.078
28

---

Under review as a conference paper at ICLR 2023
H
USING THE LABELME DATASET
Another real dataset in the crowdsourced image classification field that can be used is the Labe 1M crowdsourced dataset created by Rodrigues Pereira (2018). This dataset consists of Teask 1000 training images dispatched among K 8 classes. The validation set has 500 images and the test set 1188 images. The whole training tasks have been labelled by feorker 59 workers, each task having between one and three given (crowdsourced) labels. In particular, 42 tasks have been labelled only once, 369 tasks have been labelled twice and 589 received three labels. This way sparser labelling setting than the CIFAR-10H dataset. Following Appendix 1, GLAD's algorithm fails at inferring a difficulty parameter on such tasks.
In our experiment, we train a VGG-16 (Simonyan & Zisserman, 2014) with batch normalization on the Labe 1Me dataset for 100 epochs using the same label aggregation strategies as in Appendix A, with the same hyperparameters as in Section 4. The neural network is initialized with weights from training using ImageNet dataset. The learning rate is multiplied by a factor 0.1 at epochs 50 and 75. The WAUM is thus computed on the first 50 epochs.
WMUM 4.15
WAMM -411
WAUH
WAN 4.15
WAUM 4.34
WAIM -.34
WAUN 4.
VAMMI 40
WAW 021
WKUM 4211
WAMM-
VOUU
VAUM
WAHM
WAUM-AM
WAM--M
WAIM
WAIM-CIT
WAIN--OUT
WAUM
Wilile C18
WilMle
Wiiteen
Figure 18: LabelMe: top-10 worst images detected street. insidecity tallbuilding and highway classes using the WAUM. Overlapping classes lead to labelling confusion and learning difficul- ties for both the workers and the neural network.
The Labe 1M dataset has classes that overlap and thus lead to intrinsic ambiguities. For example, the classes highway, ins i decity ty street and tall Iding (in rows) are overlapping for some tasks. Most cities have streets and tall buildings, and highways are a specific type of street, so the confusion naturally happens can be seen in Figure 18. After training, from Table 12 observe that the proposed feature aware aggregation using the WAUM lead to better performance in test accuracy and calibration.
Table 12: Training and test accuracy depending on the aggregation method used for the LabelMe's dataset with Alakk 1000, training VGG-16 (here with 0.01 for the WAUM). Results are averaged repetitions.
Aggregation
AcCuis
Test accuracy
ECE
MV
0.764
0.662 1.018
0.143 + 0.005
Naive soft
0.761
0.723 = 3.074
0.138 + 0.013
DS
0.797
0.705 + 3.637
1141+0.00
GLAD
0.776
0.736 3.868
0.134
Crowdlayer (MW)
0.733 : 0.078
0.166 + 0.058
WAUM
0.775
0.742 1: 2.917
29


Reviews:
[{"review_id": "tlO3F_ItRkF", "reviewer": "Reviewer_JrVm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel supervised learning model in the scenarios of crowdsourcing based on AUM, which is a confidence indicator for each task. The proposed method generalizes the existing AUM indicator to the crowdsourcing setting as WAUM (the Weighted Area Under the Margin), and the WAUM identifies harmful data and prunes the ambiguous tasks to improve the results of label aggregation. Experimental results show the effectiveness of the proposed method on several simulated and real-world datasets.", "summary_of_the_review": "Overall, the paper has its merits to show a new method introducing a crowdsourcing-specific confidence indicator to prune harmful tasks and improve the performance of the model trained on the pruned dataset. However, it also has  some  weaknesses. Thus, this paper cannot meet the high-quality requirements of ICLR.", "strength_and_weaknesses": "This study is intuitive and technically sound. The proposed method is simple but interesting. In addition, the paper is presented clearly and is easy to follow and reproduce.\n\nHowever, there are several weaknesses existing in this paper:\n\n- First, the main idea of this paper is straightforward and not very impressive, and the novelty of the method is limited. The proposed WAUM seems like a simple combination of existing works, i.e. AUM and confusion matrix (which is mentioned in many EM-based label aggregation methods). I suggest the authors discuss the contribution of this paper in more detail and highlight the innovations.\n- Second, the design of the experiments is somewhat weak. 1) The proposed WAUM-based method measures the task difficulty by incorporating feature information and trains a neural network classifier simultaneously, but the comparison methods such as MV, DS, and GLAD are all feature-blind methods, also the comparison methods are classical algorithms but not the latest research works, so the experimental results are less convincing. I suggest that more algorithms (better to use the feature of tasks) should be compared with the proposed methods, e.g., (Raykar et al, 2010), (Rodrigues et al, 2018), and so on.\n- Furthermore, since the main idea of the proposed method is to identify the harmful tasks and prune them to reconstruct a new sub-dataset, could the sub-dataset be used on other existing methods to show the effectiveness of the pruned dataset?\n\nRaykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010), Learning from crowds. The Journal of Machine Learning Research 11:1297–1322.\nFilipe Rodrigues and Francisco C. Pereira. Deep learning from crowds. AAAI’18, 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qqEtIWY2PDQ", "reviewer": "Reviewer_jfcJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers the crowdsourcing problems where there exist ambiguous tasks. To address this problem, a new framework that used a similar technique in AUM method to identify ambiguous tasks has been proposed. To show the effectiveness of the proposed method, evaluations on both the synthetic datasets and the CIFAR-10H crowdsourced dataset have been implemented.", "summary_of_the_review": "In my opinion, this paper is a borderline paper. The motivation is valid, and the organization is clear. However, the baseline methods used are not very new which is my concern.", "strength_and_weaknesses": "This paper is well-organized, the structure is clean and clear, and some nice graphs are provided to help the readers to understand the proposed framework. Also, the authors provided links for almost all the involved datasets and baseline methods, which brings convenience to the readers and shows respect to the original authors at the same time. However, this work can be further improved in the following aspects:\n\n- The baseline methods used are classic methods but not very new. It would be better to use some more recent methods as baseline methods. \n- Since the proposed algorithm used the idea of AUM method, can you compare these two methods in depth?\n- In this paper, the comparison with some vote aggregation algorithms is provided, however, these methods are not new, can you compare the proposed method with more recent aggregation methods [1][2][3]\n[1] https://jmlr.org/beta/papers/v21/19-359.html\n[2] https://arxiv.org/pdf/1909.12325.pdf\n[3]https://proceedings.neurips.cc/paper/2020/file/f86890095c957e9b949d11d15f0d0cd5-Paper.pdf\n- The format of the tables used in this paper should meet the requirement of the conference.\n- It would be better to give some theoretical insights or analyses why the proposed algorithm works well.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2tr1GEKzN6", "reviewer": "Reviewer_Zduv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a way to combine crowdsourced multiple labels per task by using a weighting areas under the margin method (WAUM), which uses area under the margin method (AUM) used for detecting mislabeled data in the training dataset with a single hard label per task. It is a weighted version of AUM based on the trust factor. The trust factor can be derived as the inner product between the diagonal of the confusion matrix and the softmax vector. Experiments show how the proposed method is helpful in achieving higher test accuracy and lower ECE scores.", "summary_of_the_review": "The paper works on an interesting problem and provides a novel algorithm that aims to achieve better generalization performance. However I had a hard time understanding the main motivation of the paper along with some other questions/weaknesses. More discussions around those questions would be helpful for the reader of the paper.", "strength_and_weaknesses": "\nStrengths\n\n1. Taking into account task ambiguity is an important research topic.\n2. The proposed method can achieve a higher test accuracy for the simulated synthetic datasets and achieve the same performance as the naive soft method in the CIFAR-10H dataset.\n\nQuestions (and Weaknesses)\n\n1. One of the motivation of the paper is to remove intrinsically ambiguous tasks (which might even fool expert workers) because these tasks may be harmful for the learning step. My understanding is that intrinsically ambiguous tasks are helpful for learning, and I was not sure why these tasks may be harmful. For example, the most intrinsically ambiguous task may have p(y=+1|x) = p(y=-1) = 0.5 in binary classification (assuming y \\in {+1, -1}), where p(y|x) is the underlying true class probability. If we collect many hard labels (n labels) for this same task, close to half of them will be positive hard labels and the rest of the half will be negative hard labels. When we naively regard this as n samples of (x,y) pairs (as well as other tasks), and train a classifier, our (probabilistic) classifier will try to learn hatp(y|x) = 0.5 for this task, and perhaps this will lead to improved ECE. It will do no harm for generalization performance, because test samples close to this x is intrinsically hard. On the other hand, I understand that noisy mislabelled samples (which is not related to the intrinsic difficulty of the task) may be harmful for learning. In Section 3 of the AUM paper by Pleiss et al. (2020), the following sentence \"In general, we assume both easy and hard correctly-labeled samples in D_train improve model generalization, whereas mislabeled examples hurt generalization\" is similar to my understanding. It would be helpful if the paper can provide some discussions about why removing intrinsically ambiguous tasks will be beneficial.\n2. The paper explains that the margin in Eq. 2 is modified from the original version, following Yang & Koyejo (2020). I did not have time to fully go through Yang & Koyejo (2020), but I am predicting that this corresponds to using softmax_y - softmax_2 in Eq. 2 instead of softmax_y - max_{j\\neq y} softmax_j which is used in Pleiss et al. (2020), assuming we ignoring the softmax-logit difference. I wasn't sure if the nice theoretical properties discussed in Yang & Koyejo (2020) directly transfer to AUM as a mislabel identification metric. Intuitively, the original margin definition seems to be easier to understand, e.g., if we consider the right example in Fig.2 of Pleiss et al. (2020), the modified margin will be near zero because Dog logit is the 2nd largest value for most of the epochs. It would be helpful if the paper can further discuss the benefits of the modified margin.\n3. Is it also possible to compare the unmodified and modified margins in the experiments?\n4. The paper explains that CIFAR-10 has very few (intrinsically) ambiguous tasks, because of the careful creation of the dataset. Since the goal of the proposed method is to remove ambiguous tasks, is this a good benchmark dataset to use to test the perfomance of the proposed method?\n\nMinor comments, suggestions\n1. Page 4, typo: \"Appendix A\" should be \"Appendix B\".\n2. What is bold in Table 2, 3, and 4? Are they results of a t-test?\n3. Is it possible to provide a practical suggestion for the number of hard labels per task?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2T94LDNkeF", "reviewer": "Reviewer_Q6Gx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine a measure of the difficulty of an example in a supervised classification task (AUM) with the performance of annotators, calculated from their confusion matrix on a crowd-sourcing task. This combined estimator, called WAUM (weighted AUM), allows to take into account both the intrinsect difficulty of an example and the confidence in its label, when the latter comes from a crowd-sourcing campaign. The method is described in detail and compared to four other standard methods in the literature. The experiments are conducted on 2 synthetic datasets and 1 real dataset (CIFAR-10H). The WAUM method achieves slightly better classification results than the other methods on the synthetic data set and similar to the other methods on the real data set. \n\n", "summary_of_the_review": "The proposed metric developed in the paper is well motivated and clearly described. The experimental evaluation is still limited and has not been able to show performances clearly superior to the state of the art. The results described in the article still seem too preliminary.", "strength_and_weaknesses": "* The proposed method is motivated, very clearly described and new as far as I can tell.\n* As far as the performance of the proposed method is concerned, the experiments do not really allow to highlight a major contribution. As indicated by the authors, the CIFAR-10H dataset may already be too clean to show the contribution of the proposed method.\n\nIt seems to me that the results should be confirmed on other real datasets, as indicated by the authors in their conclusion to give more strength to their proposal.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject