Identify Ambiguous Tasks Combining Crowdsourced Labels by Weighting Areas Under the Margin
Authors: Tanguy Lefort
Benjamin Charlier
Alexis Joly
Joseph Salmon
Venue: Accepted by TMLR ()

Abstract: In supervised learning — for instance in image classification — modern massive datasets are commonly labeled by a crowd of workers. The obtained labels in this crowdsourcing setting are then aggregated for training, generally leveraging a per-worker trust score.
Yet, such workers oriented approaches discard the tasks' ambiguity.
Ambiguous tasks might fool expert workers, which is often harmful for the learning step.
In standard supervised learning settings -- with one label per task -- the Area Under the Margin (AUM) was tailored to identify mislabeled data.
We adapt the AUM to identify ambiguous tasks in crowdsourced learning scenarios, introducing the Weighted Areas Under the Margin (WAUM). 
The WAUM is an average of AUMs weighted according to task-dependent scores. 
We show that the WAUM can help discarding ambiguous tasks from the training set, leading to better generalization performance.
We report improvements over existing strategies for learning with a crowd, both on simulated settings, and on real datasets such as CIFAR-10H (a crowdsourced dataset with a high number of answered labels), LabelMe and Music (two datasets with few answered votes).

Published in Transactions on Machine Learning Research (04/2024)
Identify Ambiguous Tasks Combining Crowdsourced Labels by Weighting Areas Under the Margin
Tanguy Lefort
tanguy.leforr@uumnnnpeeleee
IMAG, Univ. Montpellier, CNRS, LIRMM, INRIA
Benjamin Charlier IMAG, Univ. Montpellier, CNRS
Alexis Joly
alexis Micris.joly@inrio..
LIRMM, INRIA
Joseph Salmon
joeeph..ddmon@@mmnntpplleee
IMAG, Univ. Montpellier, CNRS, Institut Universitaire France (IUF)
Reviewed on OpenReview: https: // // openreveew...eo.. net/ forun? id- raD846n122
Abstract
In supervised learning for instance in image classification modern massive datasets are commonly labeled by a crowd of workers. The obtained labels in this crowdsourcing setting are then aggregated for training, generally leveraging per-worker trust score. Yet, such workers oriented approaches discard the tasks' ambiguity. Ambiguous tasks might fool expert workers, which often impacts the learning step. In standard supervised learning set- tings - with one label per task the Area Under the Margin (AUM) was tailored to identify mislabeled data. We adapt the AUM to identify ambiguous tasks in crowdsourced learning scenarios, introducing the Weighted Areas Under the Margin (WAUM). The WAUM is an average of AUMs weighted according to task-dependent scores. We show that the WAUM can help discarding ambiguous tasks from the training set, leading to better generalization performance. We report improvements over existing strategics for learning with crowd, both simulated settings, and on real datasets such CIFAR-10H (a crowdsourced cltasstt with a high number of answered labels), LabelMe and Music (two datasets with few answered votes).
1 Introduction
Crowdsourcing labels for supervised learning has become quite common in the last two decades, notably for image classification datasets. Using a crowd of workers is fast, simple (see Figure [ and less expensive than using experts. Furthermore, aggregating crowdsourced labels instead of working directly with a single one enables modeling the sources of possible ambiguities and directly taking them into account at training (Aitchison] 2021). With deep neural networks nowadays common many applications, both the architectures and data quality have direct impact on the model performance (Müller al. 2019 Northcutt al. 2021a) and on calibration (Guo et al. 2017). Yet, depending the crowd and platform's 's control mechanisms, the quality of the labels might be low, with possibly many mislabeled instances (Müller & Markert 2019 hence,
degrading generalization (Snow et al. 2008
Popular label aggregation schemes take into account the uncertainty related to workers' abilities: for example by estimating confusions between classes, or using a latent variable representing each worker trust (Dawid
& Skene 1979 Kim & Ghahramani] 2012 Sinha et al. 20188 Camilleri Williams 2019). This leads scoring workers without taking into account the inherent difficulty of the tasks at stake. Inspired by the Item

---

Published in Transactions on Machine Learning Research (04/2024)
Image X wocket + propceed Jabel
Image proposed labels
Tasks idmitifccation
Crowd workens
g+
VE
fi
Prine
2
[1)
Vi
Train
Unlabeled
1 x]
2 Fa
7 23
7
imsgges
1/2
Figure 1: Learning with crowdsourcing labels: from label collection with crowd to training a pruned dataset. High ambiguity from either crowd workers or tasks intrinsic difficulty can lead to mislabeled data and harm generalization performance. To illustrate our notation, here the set of tasks annotated by worker Wy is (wss) = 31,3} while the set of workers annotating task T3 is A(æz) = 31,
Response Theory (IRT) introduced in (Birnbaum 1968), the authors of (Whitehill et al. 2009 have combined both the task difficulty and the worker's 's ability in maature-blind fashion for label aggregation. Other feature- blind aggregation ion strategies exist using (rank-one) matrix completion techniques (Ma & Olshevsky] 2020: Ma et al. 2020) or pairwise co-occurrences (Ibrahim et al., 2019 Both rely on the work by Dawid & Skene (1979h and take into account worker abilitics but neglect the task difficulty. All the feature-blind strategies only leverage the labels but discard the associated features to evaluate workers performance. For instance, GLAD (Whitehill et al. 2009) estimates task difficulty without the actual task: its estimation only relies the collected labels and not on the tasks themselves (in clasiiclassiiicttion settings, this means the images are not considered for evaluating the task difficulyy). Neglecting such task difficulty might become critical when the number of labels collected per task is small.
In this work, we aim at identifying ambiguous tasks from their associated features, hence discarding hurtful tasks (such as the ones rllustrated on Figure 2b and Figure 2c). Recent works on data-cleaning in supervised learning (Han et al.) 20199 Pleiss al.] 2020 Northcutt et al.) 2021b) have shown that some images might be too corrupted too ambiguous to be labeled by humans. Hence, one should not consider these tasks for label aggregation or learning since they might reduce generalization power; see for instance (Pleiss el al. 2020p. Throughout this work, we consider the ambiguity of a task with the informal definition proposed by Angelova (2004) that fit standard learning frameworks: "Difficult ezamples are those which obstruct the learning process or mislead the learning algorithm or those which are impossible to reconcile with the rest of the ezamples". ". This definition links back to with how Pleiss et al. (2020) detect corrupted samples using the area under the margin (AUM) during the training steps of machine learning classifier. However, is important to notice that, in this context, the task ambiguity is inherent to the classifier architecture, and thus might not exactly overlap with human-level difficulty.
In this work, we combine task difficulty scores with worker abilities scores, but we measure the task difficulty by incorporating feature information. We thus introduce the Weighted Area Under the Margin (WAUM), a generalization to the crowdsourcing setting of the Area Under the Margin (AUM) by (Pleiss et al., 2020) The AUM is a confidence indicator an assigned label defined for cach training task. It is computed as an average of margins over scores obtained along the learning steps. The AUM reflects how learning procedure struggles to classify task to assigned labe!] The AUM is well suited when training a neural network (where the steps are training epochs) or other iterative methods. For instance, it has led to better network calibration (Park & Caragea, 2022) using MixUp strategy (Zhang et al. 2018). i.e., mixing tasks identified as simple and difficult by the AUM. The WAUM, our extension of the AUM, aims at identifying ambiguous data points in crowdsourced datasets, so one can prune ambiguous tasks that degrade the generalization. It is a weighted average of workers AUM, where the weights reflect trust scores based on task difficulty and workers' ability.
1 SSee the Linear SVC Figure visualize how the AUM connected the classical margin from the kernel literature.
2

---

Published in Transactions on Machine Learning Research (04/2024)
Vone dettribtton
ge
Lmnge #7381 CIRAT- abel sirylen
CIPAR bilel doer
Image (66330
alssue anfonmn
Mtd
der
dos
bocs
trock
CTFAR: label
Iuop #5295
der
fore
(a) Label airplane is casy to identify (unanimity among workers).
(b) Label deer is meaningless here, and workers are confused with all other la- bels.
(c) Label cat often confused with horns of a wild deer
Three images from CIPAR-10H dataset (Peterson et al. 2019), with the empirical distribution of workers' labels (soft labels): the airplane image (a) is easy, while the landscape (b) is ambiguous due to the image's poor quality. The last image (c) looks like black cat face often perceived the horns of deer.
Figure 2:
2 Related Work
Inferring a learning consensus from a crowd is a challenging task. Tn this work, we do not consider methods with prior knowledge on the workers, since most platforms do not provide this information Likewise, we do not rely on ground-truth knowledge for any tasks. Hence, trapping-set or control-items based algorithms like ELICE or CLUBS (Khattak) 2017) do not match our framework. Some algorithms rely on self-reported confidence: they directly ask workers their answering confidence and integrate it into the model (Albert al.
2012 Oyama al. 2013 Hoonn et al. 2021). We discard such cases for several reasons. First, self-reported confidence might not be beneficial without reject option (Li & Varshney] 2017). Second, workers have tendency to be under or overconfident, raising questions on how to present self-evaluation and assossing own
scores Draws et al. 2021
To reach a consensus in the labeling process, the most common aggregation step is majority voting (MV), where one selects the label most often answered. MV does not infer any trust score on workers and does not leverage workers' abilities. MV is also very sensitive under-performing workers (Gao & Zhou 2013 Zhou al., 2015), to biased workers [Kamar et al. 2015), to spammers (Raykar & Yu! 2011), or lack of experts for hard tasks [James 1998 Gao & Zhou 2013 Germain et al. 2015). Closely related to MV, naive soft (NS) labeling goes beyond hard labels (also referred to as one-hot labels) by computing the frequency of answers per label, yield a distribution over labels, often referred to as sofl-labels. In practice, training a neural network with soft labels improves calibration (Guo et al.) 2017) w.r.t. using hard labels. However, both MV and NS are sensitive to spammers (e.g., workers answer all tasks randomly) or workers' biases (e.g., workers who answer some tasks randomly). Hence, the noise induced by workers' labeling might not be representative of the actual task difficulty (Jamison & Gurevych 2015).
Another class of methods leverages latent variables, defining probabilistic model on workers' responses. The most popular one, proposed by (Dawid & Skene 1979) (DS), estimates single confusion matrix per worker, as a measure of workers' expertise. The underlying model assumes that a worker answers according to a multinomial distribution, yielding a joint cstimation procodure of the confusion matrices and the soft (EM). Variants of the DS algorithm include accelerated (Sinha) lt al., 2018), sparse (Servajean al.] (017),, and clustered versions (Imamura et al., 2018)) among others. DS strategy
labels through Exppetationn-Maxxiivalll
also be used to create weights for a weighted MV (WMV) strategy. Indeed, cach worker weight can be considered as how accurate they are at labeling the correct class, i.e. the sum of the diagonal of their confusion matrix.
Since DS only models workers' abilitics, Whitchill et al.) 2009 have introduced the Generative model of Labels, Abilities, and Difficulties (GLAD) to exploit task difficulties to improve confusion estimation. While DS estimates a matrix of pairwise label confusion per worker, GLAD considers also an EM procedure estimate one ability score per worker, and one difficulty score per task. It is inspired by the IRT (Birnbaum)
For instance, by default A Amazon Mechanical Turk https::/t///et wtw.k. does not provide it.
3

---

Published in Transactions on Machine Learning Research (04/2024)
1968), modeling the workers' probability to answer the true label with logistic transform of the product of these scores. Following IRT, the difficulty is inferred as a latent variable given the answers: as for DS, the underlying tasks are discarded. Other methods based on probabilistic models or objective optimization have been developed. Platanios et al. (2014) defines a setting to estimate the accuracy of a set of classifiers (workers) with unlabeled data based on their agreement/erne rales. This could later be used as a set of worker weights in a different weighted majority vote, a setting with dependent workers. Their agroement rate (AR) approach is based on binary classifications output, and the adaptation to multiple classes needs to be split into binary problems to optimize. Each of those optimization problems depends exponentially on the number of workers and thus is limited for our crowdsourced datasets as is. In Platanios et al. (2017), they later proposed a new probabilistíc-logic approach for the same accuracy inference problem. They infer the output based on classifier (workers) outputs and a set of logical comstraints. Multiclass classification fits this setting using mutual exclusion class constraints. They achieve competitive performance against probabilistic strategics and could in practice be combined with the pruning method presented in this paper.
Finally, following deep learning progresses, end-to-end strategies have emerged that do not produce aggre- gated labels but allow to train classifiers from crowdsourced labels. Rodrigues & Pereira (2018) introduced CrowdLayer adding a new layer inside the network mimicking confusion matrices per worker. Later, Chu
ct al. 2021 have generalized this setting with CoNAL, adding an element encoding global confusion.
Here, we propose the WAUM to combine the information from a confusion matrix per worker and a mea- sure of relative difficulty between tasks. It refines the judging system and identifies data points harming generalization that should be pruned. Data pruning has been shown to improve generalization by removing mislabeled data (Angelova. et al. 2005; Pleiss et al.) 2020), possibly dynamically along the learning phase [Raju et al. 2021 by defining forgetfulness score (Paul et al. 2021). Recent work on data pruning in crowdsourcing (Xing et al. 2023) separates label errors (the tasks are useful for learning as their ambiguity is relevant) from label noise (tasks are too ambiguous for learning) in binary classification setting, They also consider the model scores for each task and the given label to estimate the probability for a given label to be the true label. And from these estimates, they rank the lasks and prune the most unreliable ones. In this paper, however, we only consider multiclass classification and can not consider as-is this concurrent strategy. Sorscher et al. (2022) have highlighted that data pruning strategies are highly impacted by the labeling in supervised settings and we confirm its relevance to the crowdsourcing framework. It is also flexible tool that can be combined with most existing methods, using the pruning as a preliminary step.
1.5
1.0
-12
60
0.
-12
WAUSE
WAIM
RAUM
(a) CIFAR- 10H dataset.
(b) LabelMe dataset.
(c) Music dataset.
Figure 3: Entropy of votes vs. WAUM for CIFAR-10H, LabelMe, and Music, each point representing a task/image. When large amounts of votes per task are available, WAUM and entropy ranking coincide well, as in (a). Yet, when votes scarce, as in (b) and (c), entropy becomes irrelevant while introduced WAUM remains useful. Indeed, tasks with few votes can benefit from feedback obtained for similar one. And for the LabelMe dataset in particular, there are only up to three votes available per task, thus only four different values of the entropy possible, making it irrelevant in such cases for modeling task difficulty.
3 Weighted Area Under the Margin
3.1 Definitions and Notation
General notation. We consider classical multi-class learning notation, with input in X' and labels in [K] = (I....,K] K}. The set of tasks is written as Xtraia (..,.... Tramh }. and we assume
4

---

Published in Transactions on Machine Learning Research (04/2024)
{(11 vf)...., (Iress Fieesee )} are ntask i.i.d tasks and labels, with underlying distribution denoted by P. The true labels (v, )ifanaa are unobserved but crowdsourced labels are provided by Rearker workers (w;) [mmummel) worker wy labeled task x the annotators set of a task 2i and T(w;)) = {i E [Meaok] : worker wy answered task m3} the tasks set for a worker Wy. For a task iy and each j E A(xi), we denote yi E [K] the label answered by worker wy. Given an aggregation strat- egy agg (such as MV, DS or GLAD), we call estimated soft label y, the obtained label. Note that for MV, the aggregated label g
We write A(r;) = {j E [Mworkee]
[K] and for other strategies, 0, lies in the standard simplex Ax-1 = {p E Rx Ek= Pk = L. 2 0}. For any set S, we write [S] for its cardinality. Examples of annotators set and tasks set are provided in Figure I formulations:
The training set has task-wise and worker-wise
D Perain
Praak
U
(I (y: ()) for j E Aloo)=
Preazber U
(1%s (I,,(v) (v.")) for i T T(wy)
f=1
D
DS model notation. The Dawid and Skene (DS) model (Dawid & Skene 1979) aggregates answers and evaluates the workers' confusion matrix to observe where their expertise lies. The confusion matrix of worker wy is denoted by x(t)= E k and reflects individual error-rates between pairs of labels: a(4) = (P( kly, - () represents the probability that worker wy gives label to a task whose true label is l. The model assumes that the probability for a task x; to have true label yt follows a multinomial distribution (5) with probabilities T.. for each worker, independently of Eerain (feature-blind)) In practice, DS estimates are obtained thanks to the EM algorithm to output estimated confusion matrices (E)))))(( The full likelihood is given in Equation S Appendix A.3 Once DS confusion matrices are estimated, it is possible to use the diagonal terms as weights in a majority voting strategy. We denote this Weighted DS vote by WDS, and give more details in Appendix A.4 Essentially, the WDS strategy produces soft labels as NS, and also takes into account the estimated worker ability to recognize a task whose true label is indeed the voted one.
3.2 Ambiguous tasks identification with the AUM
To identify labeling errors and evaluate task difficulties, Pleiss et al. 2020 have introduced the AUM
the standard learning sctting (f..., when [A(x;)] = 1 for all i E [eaakk]
). Given a training task and a label
(v, y), let (x) E R.K be the logit score vector at epoch < T when learning a neural network (where T is
(t) the number of training epochs). We use the notation (t) (xr ... 2 (z) for sorting (zl" (z)..... 2 (z))
(t)
in non-incrcasing order. Let us denote (<)) == (2(0)) (r)) the softmax output of the scores at epoch t.
Sorting the probabilities in decreasing order such that [1] (x) .... (0)) (2), the AUM reads:
T
AUM (x., ys D Derain)
-
(x)
(2)
We write AUM (I, instead of AUM (I,, yi Derain) when the training set is clear from the context. Pleiss et al. (0200 use an average of margins over logit scores, while we rather consider the average of margin after a softmax step in Equation 2 to temper scaling issues, as advocated by Ju et al. (2018) in ensemble learning. Moreover, we consider the margin introduced by Yang & Koyejo (2020) since the corresponding hinge loss has better theoretical properties than the one used in the original AUM, especially top-k setting (Lapin) let al.] 2016 Yang & Koyejo| 2020 Garcin al.) 2022). However, one could easily consider the original margin with few differences in practice for top-1 classification (sce Appendix G
During the training phase, the AUM keeps track of the difference between the score assigned to the proposed label and the score assigned to the second-largest one. It has been introduced to detect mislabeled obser- vations in a dataset: The higher the AUM, the more likely the nelwork confirms the given label. And, the lower the AUM, the harder it is for the network to differentiate the given label from (at least) another class.
3 For to consider 1]+1] (x) instead of [ (x:) in equation

---

Published in Transactions on Machine Learning Research (04/2024)
Finally, note that the AUM computation depends the chosen neural network and on its initialization: pre-trained architectures could be used, yet. any present bias would transfer to the AUM computation.
To generalize the AUM from Equation (2) to the crowdsourcing setting, diffcuulty lies in the term dy (z) as, in this context, the label is unknown, one observes several labels per task. A naive adaptation of the AUM would be to use the majority voting strategy in order to recover a hard label to be used in Equation (2. We denote such a strategy by AUMC (AUM for Crowdsourced data). More formally, this writes as:
(0)
AUMC
(Z..
Zti. (2r.(1 )EA(O))) Persin
1
Σ
T
(za) (xt) - -0 (a:)
(3)
T
This naive approach can be refined by taking into account the whole distribution of labels, and not simply its mode (with MV).
3.3 WAUM and data pruning
The AUM is defined in a standard supervised setting with (hard) labels. The naive adaptation AUMC defined at equation g does not take into account the fact that workers may have different abilitics. Wo adapt the AUM to crowdsourced frameworks to improve the identification of difficult tasks. Mirroring the AUM, train a classifier network on Derain for a number T >> O of epochs. Using this classifier, we can compute the following. Let (x;) E [0, be a trust factor in the answer of worker wy for task 2z. The WAUM then defined as:
Σ (z:) (
WAUM(*)) jEA(a,)
(4)
((x)
J'E.(a.)
It is a weighted average of AUMs over each worker's answer with per task weighting score g (x;)) based on workers' abilities. This score considers the impact of the AUM for each answer since it is more informative if the AUM indicates uncertainty for an expert than for a non-expert.
The scores slf) are obtained la Servajean et al. 2017y: ; each worker has an estimated confusion matrix #(i E k Note that the vector diag(#W7) E R represents the probability for worker Wy to answer correctly to each label. With neural network classifier, we estimate the probability for the input H; E Xirain to belong in each category by (a;), i.e., the probability estimate at the last epoch. As a trust factor, we propose the inner product between the diagonal of the confusion matrix and the softmax vector:
(x:) (diag(# ()) ( (ar)) E [0.1)
The scores control the weight of each worker in Equation scoring system of GLAD (Whithiil et al.) 2009), as detailed hereafter. The closer to one, the more trust the worker for the given task. The score s (x:) can be seen as a multidimensionall version of GLAD's trust score. Indeed, in GLAD, the trust score is modeled as the product ayβ, with aj E R (resp. 33 E (0, +o0)) reprosenting worker ability (resp. task difficulty). In Equation (5), the diagonal the confusion matrix #0) represents the worker's ability and the softmax the lask difficulty.
This choice of weight is inspired by the bilinear
Dataset Pruning and hyperparameter tuning. Our procedure (Algorithm D proceeds as follows. We initialize our method by estimating the confusion matrices for all workers. For each worker Wy, the AUM is computed for its labeled tasks, and is its depker-d trust scores (1;) with Equation (5) during the training phase of classifier. The WAUM in Equation 4 is then computed for each task. The most ambiguous tasks, the ones whose WAUM are below a threshold, are then discarded, and the associated pruned dataset D Pprumed is output. We consider for the pruning threshold a quantile order a E [o, 1] of the WAUM scores. The hyperparamcter a (proportion of training data points pruned) calibrated on a validation set, choosing a E 10.1, 0.05, 0.01} has led satisfactory results in all our experiments. In general, dataset annotated by humans have roughly between and 5% of errors (Northeutt et al.] 2021a) and the choice

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm WAUM (Weighted Area Under the Margin).
Input:
Derasn: tasks and crowdsourced labels, a E [0, IJ: proportion of training points pruned, T' E N:
number of epochs, Est: Estimation procedure for the confusion matrices
Output: pruned dataset D Pruued
1: Get confusion matrix )
JjE(naorar! from Est
( Train classifier for T epochs on Tzx Mi
3: for j E [[worker] do
Get AUM( v1 ; :Drrai)) using Equation for i E T(w,) for i E ((w))
5: Get trust. scores (1;) using Equation
6: end for
7: for each lask x E Xirain do
8: Compute WAUM(2) using Equation
4
end for
10: Get Go (WAUM(z)))E(naaa)) a-quantile threshold
11: Prruned = (mis ( JEA(o.)) <WAUM(a) 2 yas Ts E Kerain
of a should reflect that. Note that the same pruning procedure can be applied to AUMC for comparison. Both the AUMC and WAUM inherit the hyperparamcter T > 0 from the original AUM. Following the recommendations from Pleiss et al. (2020), we need T large enough for stability and T not too big to avoid overfitting the data. In practice, guideline given is to train until the first learning rate scheduler drop to only keep the beginning of the scores trajectories without finetuning. The main assumptions to identify ambignous tasks is thus not to over-train the neural network in the WAUM (or AUMC) step, and be able to run DS-like algorithm to recover the diagonal of the confusion matrix for Equation
Refined initialization: estimating confusion matrices. By default, we rely the Est=DS algorithm to get workers' confusion matrices, but other estimates are possible: DS might suffer from the curse of dimensionality when the number K of classes is large (K² coefficients needed per worker).
Training on the pruned dataset Once a pruned dataset Drruned has been obtained thanks to the WAUM, one create soft labels through aggregation step, and use them to train classifier. Aggregated soft labels contain information regarding human uncertainty, and could often be less noisy than NS labels. They can help improve model calibration Wen et al. 2021 Zhong et al.) 2021 property useful for interpretation (Jiang et al., 2012 Kunar et al. 2019). Concerning the classifier training, note that it can differ from the one used to compute the WAUM. We train a neural network whose architecture is adapted dataset per dataset and that can differ from the one used in Algorithm [ (it the case for instance for the LabelMe dataset).. For an aggregation technique agg, we write the full training method on the pruned dataset created from the WAUM: agg + WAUM and instantiate several choices in Section 4 For comparison, we write agg + AUMC the training method on the pruned dataset created from the AUMC.
4 Experiments
Our first experiments focus on multi-class classification datasets with large number of votes per task. We consider first simulated dataset to investigate the WAUM and the pruning hyperparameter Then, with the real CIFAR-10H dataset from Peterson et al. (2019) compare label aggregation- based procedures with and without pruning using the AUMC or the WAUM. Finally, we run our experiments on the LabelMe dataset from Rodrigues & Pereira (2018) and Music dataset from Rodrigues et al. (2014), both real crowdsourced datasets with few labels answered per task. For each aggregation scheme considered, we train a neural network on the soft labels (or hard labels for MV) obtained after the aggregation step. We compare our WAUM scheme with several other strategics like GLAD (feature-blind) or CoNAL (feature-aware) with and without pruning from the AUMC identification step. For CoNAL, two regularization levels are considered:
7

---

Published in Transactions on Machine Learning Research (04/2024)
W22 SVC
Ground truth
Linear SVC
GBM
WAUM
00.1
AUM
0.5
0.5
0.8
0.6
0.4
0.2
0.25
Figure three_ciiee circles: one realization of simulated workers W1, W2, Wa, with their AUM, normalized trust scores ز0o (Ieft) and WAUM distributions (right) for a = 0.1. Worker wi has less impact into the final WAUM in the disagreement area. Note also that for worker w1 (LinearSVC),, the region with low AUM values recovers the usual classifier's margin around the decision boundary.
À 0 and X = 10 (A controls the distance between the global and the individual confusion matrices).
More simulations and overview of the methods compared are available in Appendix D.1
Metrics investigated After training, we report two performance metrics on test set Drest: top-1 accu- racy and expected calibration error (ECE) (with M = 15 bins as in Guo et al. (017)). The ECE measures the discrepancy between the predicted probabilities and the probabilities of the underlying distribution. For ease of reporting results, we display the score ECE (hence, the higher the better, and the closer to 1, the better the calibration); see Appendix [ for more details. Reported errors represent standard deviations over the repeated experiments (10 repetitions simulated datasets and 3 for real datasets).
Implementation details For simulations, the training is performed with three dense layers' artificial neural nelwork (a MLP with three layers) (30, 20, 20) with batch size set to 64. Workers are simulated with scikit-learn (Pedregosa et al. 2011) classical classifiers. For CIFAR-10H the Resnet-18 (He et al. 2016) architecture is chosen with batch size set to 64. We minimize the cross-entropy loss, and use when available a validation step to avoid overfitting. For optimization, we consider an SGD solver with 150 training epochs, an initial learning rate of 0.1, decreasing it by factor at epochs 50 and 100. The WAUM and AUMC are computed with the same parameters for T' 50 epochs. Other hyperparameters for Pytorch's (Paszke et al. 2019) SGD are momentum=0.9 9 and weight_ weight_decay de-ay For the LabelMe and Music datasets, use the Adam optimizer with learning rate set to 0.005 and default hyperparameters. On these two datasets, the WAUM and AUMC' are computed using a more classical Resnet-50 for T = 500 epochs and the same optimization settings. The architecture used for train and test steps is a pretrained VGG-16 combined with two dense layers as described in Rodrigues & Pereira 2018 to reproduce original experiments on the LabelMe dataset. This architecture differs from the one used to recover the pruned set. Indeod, contrary to the modified VGG-16, the Resnet-50 could be fully pre-trained. The general stability of pre-trained Resnets, thanks to the residuals connections, allows epochs (each being also with lower computational cost) compared to VGGs (He et al. 2016 As there are few tasks, we use data augmentation with random flipping, shearing and dropout (0.5) for 1000 epochs. Experiments were executed with Nvidia RTX 2080 and Quadro T2000 GPUs. Appendix B presents more details on the code used with the peerannot library. Source codes are available at peeramot/peerannot] Evaluated strategies are main/poeramot/modolls sorted according whether they are aggregation based, learning- based or only for identification. The WAUM and AUMC sources available in the identification module.
compute the WAUM and AUMC with way fewer
com/
sorted
8

---

Published in Transactions on Machine Learning Research (04/2024)
Ground truth
a = 10-3
a = 10
a = 10
a = 0.25
Train
Test
Figure 5: Influence of a on the pruning step. Red dots indicate data points pruned from the training set, at
level 9a in the WAUM (see line 10 in Algorithm D We consider (a E {10-3, 10 1010 (00-,.))))). The neural network used for predictions is three dense layers' (30, 20, 20), for other simulated experiments. Training labels are from the WDS + WAUM strategy with performance reported in Section 4.1 The more prune data, the worse the neural network can learn from the training dataset. However, removing the tasks with high disagreement. noise helps to generalize.
4.1 Simulated multiclass dataset: faree_circles.
Ground truth
MV
DS
GLAD
WDS
WAUM
Test (hard labels) (hard labels)
Train
Figure three_cireee circles: One realization of Section varying the aggregation strategy. Training labels are provided from Figure and predictions on the Lest set from three dense layers' artificial neural network (30, 20, 20) trained the aggregated soft labels. For ease of visualization, the color displayed for each task represents the most likely class. Red points are pruned from training by WAUM with threshold a = 0.1. Here, we have Teack 525. WAUM method as in Section 4.1 uses WDS labels.
We simulate three cloud points (to represent K = classes) using scikit- function two two__trrcll ciccciere;; see Figure 4. The worker =3 3 workers are standard classifiers: W1 is a linear Support Vector Machine Classifier (lincar SVC), We an SVM with RBF kernel (SVC), and wy is a gradient boosted classifier (GBM). Data is split between train (70%) and test (30%) for total of 750 points and each simulated worker voles for all tasks, i.e., for all E Xersin: [A(x) Rworker 3, leading to naas's 525 tasks (points). We do not use validation set to calibrate the hyperparameter as this experiment is mostly for pedagogical purpose. The performance reported in Section is averaged over 10 repetitions using the peerannot library.
A disagreement area is identified in the northeast area of the dataset (see Figure 4p. Section 4.1 also shows that pruning too little data (a small) or too much (a large) can mitigate the performance. In Figure we show the impact of the pruning hyperparameter a. The closer to 1, the more training tasks are pruned from the training set (and the worse the performance).
9

---

Published in Transactions on Machine Learning Research (04/2024)
Stratiggy
AcCeest
BCF
MV
0.73 = 0.03
0.13 + 0.03
NS
0.70 1 0.02
0.18 0.02
DS
0.75 1 0.07
0.08
GLAD
0.58 + 0.02
0.36
WDS
0,81 = 0.04
0.17 0.03
WDS + AUMC(a 10-1)
0.81 + 0.02
0.01
WDS + WAUM(a = 10
0.80 1 0.04
0.17 1.01
WDS WAUM(a 10-1)
0.83 0.03
0.19
WDS Wa 0.25)
0.69 = 0.02
0.19 0.02
Table 1:
Aggregation and learning performance presented in Figure 6 (Teask = 525 tasks, A(x)] = Roorker = 3, 10 repetitions). Errors represented are standard deviations. Note that the best worker, W3, reaches 0.84 on test accuracy. We vary a E (0.01, 0.25] to visualize the impact of pruning.
4.2 Real datasets
In this section, we investigate three popular crowdsourced datasets: CIFAR-1OH, LabelMe and Music. The first one, CIFAR-10H Peteerson et al., 2019). is a curated dataset with many votes per task while LabelMe [Rodrigues Pereira 2018) and Music (Rodrigues et al. 2014) datasets are more challenging, having fewer labels per task. This low number of votes per task, especially for LabelMe can lead erroneous MV label which then impact the quality of the AUMC. In this context, the label distribution's entropy is also poor choice to identify hard tasks as can be scen in Figure Indeed, with up to three labels, the entropy only lake four different values and thus is no help in ranking the difficulty of 1000 tasks.
To prune only a few tasks, we have = 1% for CIFAR-10H and LabelMe datasets. For the Music dataset, a = 5% lcads to better generalization performance; considering the dataset size and complexity, picking a 0.1 would lead to worse performance. This value for the hyperparameter a has been calibrated as described in Section 3.3 using the accuracy on the available validation set for a E {0.01, 0.05, 0.1]. Ablation studies by architecture are performed on CIFAR-10H and LabelMe datasets in Figure 12 show consistent improvement in performance by using the WAUM to prune ambiguous data.
Strategy
Acceus (%)
ECE
MV
69.53 1 0.84
0.825 0.00
MV + AUMC
71.12 + 1.12
0.836 = 0.01
MV + WAUM
72.34 1.01
0.814 + 0.02
NS
72.14 2.74
0.868 0.03
NS AUMC
71.80
0.838 + 0.00
NS WAUM
=
0.829 + 0.00
DS
7
0.827 + 0.00
DS AUMC
70.43
0.833 0.02
DS WAUM
72.71-0.35
0.814 0.02
GLAD
70.28 0.88
0.838 0.01
GLAD + AUMC
70.42 1.23
0.830 0.01
GLAD + WAUM
1.12
0.812
WDS
72.49 1 0.48
0.868 0.00
WDS AUMC
72.47 1 0.45
0.866 0.00
WDS WAUM
72.67 0.59
0.868 0.00
Table 2: CIFAR-10H: performance of a ResNet-18 by label-aggregation crowdsourcing strategy (a = 0.01). Errors represented are standard deviations.
CIFAR-10H datasct. The training part of CIFAR-10H consists of the 10000 tasks extracted from the test set of the classical CIFAR-10 datasel. (Krizhevsky & Hinton 2009), and K = 10. A total of Boorker 2571 workers participated on the Amazon Mechanical Turk platform, each labeling 200 images (20 from each original class), leading to approximately 50 answers per task. We have randomly extracted 500 tasks for a validation set (hence Neraia 9500). The test set of CIFAR- 10H comprised of the train set of CIFAR-10 10 (see more details in Appendix D.2 This dataset is notoriously more curated (Aitchison 2021 than common
10

---

Published in Transactions on Machine Learning Research (04/2024)
dataset in the field: most difficult tasks were identified and removed at the creation of the CIFAR-10 dataset, resulting in few ambiguities (Krishevsky & Hinton 2009). Section 4.2 shows that in this simple setting, our data pruning strategy is still relevant, with the choice = 0.01. Images with worst WAUM for each class are presented in Figure
Furthermore, the WAUM leads better generalization performance than the vanilla DS model and the pruning with AUMC. Overall, we show that there gain in performance to obtain by using a pruning pre-processing step compared to training the classifier the aggregated labels for the full training set. There is consistently improvement using the WAUM pruning which weights the margins by worker and tasks over the naive AUMC which does not use reweighing.
BMGV
WNVV
WANN 202
WALM
MMUV
WKGM 4711
WMN--
VAMM
MILM
WNJU
RMAV
WNVV 1.45
VAMM 945
WAMM 6.M
MALVV C.
WWW 1.9
VANN 3.9
WAN ON
MALM 071
ARJV-1R
WhV
WMMM 26G
WAMH
MALM
MMUV CCS
WNOV 1.44
WAMM
NALVV C.90
WKJ. 1.00
WOUN-3X
WAN ON
MALM 6.N
AMJV-171
WWWV
WWM
KAULV
WNOV-ETE
YAMN 375
WWW 6.1
MMLV 777
WWWM 172
WAIN 3
WANN OO
MALM 6NN
ARJV-17
WNVY
WWM MLLV-02
Figure 7: CIFAR-10H: 10 worst images for WAUM scores, by labels given in CIFAR-10. The rows represent the labels airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Images in red can be particularly hard to classily as they are not typical examples of their label. Comparison with the AUMC and the AUM are available in Figure 16 Appendix D.2.1
CIFAR-10H a relatively well-curated dataset, and we observe in Scction 4.2 that in this case, simple aggregation methods already perform well, in particular NS. Over the 2571 workers, less than 20 are identified spammers using Raykar & Yu (2011) but note that most difficult tasks were removed when creating
11

---

Published in Transactions on Machine Learning Research (04/2024)
the original CIFAR-10 dataset. We refer to the 'labeler instruction sheet" of Krizhevsky & Hinton (2009 Appendix C) for more information about the directives given to workers.
MV
MV
MV Ø
MV
MV AUMC
MV AUMC
MV + WAUM
MV + WAUM
NS
NS
NS
NS
NS AUMC
NS AUMC
NS + WAUM
NS WAUM
DS
DS
DS
DS + AUMC
DS + AUMC
DS + WAUM
DS + WAUM
GLAD
GLAD
GLAD
GLAD Ø
GLAD AUMO
GLAD AUMC
GLAD WAUM
GLAD WAUM
WDS
WDS
WDS O
WDS Ø
WDS AUMC
WDS AUMC
WDS WALM
WDS WALM
CrowdLayer
CrowdLayer
CrowdLayer Ø
CrowdLayer Ø
CrowdLayer + AUMC
CrowdLayer + AUMC
CrowdLayer WAUM
Crowdlayer WAUM
CoNAL 0)
CoNAL 0)
CONAL (0)
CONAL (0) +
CoNAL (0) AUMC
CoNAL (0) + AUMC
CONAL (0) WALM
CoNAL (0) WAUM
CoNAL le- 4)
CoNAL le- 4)
CONAL (1e-4) D
CONAL (1e-4) Ø
CoNAL (Ie-4) AUMC
CoNAL (1e-4)
AUMC
CONAL (1e-4)
WAUM
CoNAL (1e-4)
WAUM
0.78
0.90
Accuracy
1- ECE
Figure 8: Ablation study on LabelMe using the VGG backbone: 0.01. Errors are Gaussian confidence intervals at 95%. Numeric tables are available in Appendix F
LabelMe dataset. This dataset consists in classifying 1000 images in K = 8 categories. In total 77 workers are reported in the dataset (though only 59 of them answered any lask at all). Each lask has ween 1 and 3 labels. A validation set of 500 images and a test set of 1188 images are available.
We observe in Figure X that the WAUM improves the final test accuracy when combined with the CoNAL network with regularization. Note that the LabelMe dataset. has classes that overlap and thus lead to intrinsic ambiguities. This is the reason why the CoNAL strategy was introduced by Chu et al. (2021): modeling common confusions help the network's decision, so it was expected for the CoNAL to perform well. Combined with our WAUM, additional gains are obtained on both metrics. The vanilla strategy, either for aggregation or learning, can be improved using a pruning preprocessing step. However, between the AUMC and the WAUM, we show a consistent improvement on using the WAUM that considers weights for the workers individually. For example, the classes highway, insidecity, street and tallbuilding (in rows) are overlapping for some tasks: some cities have streets with tall buildings, leading to confusion as shown in Figure 10
Music dataset. This dataset differs from LabelMe and CIFAR-10H as it consists in classifying 1000 record- ings of 30 seconds into K 10 music genres. All the 44 workers involved voted for at least one music, resulting in up to labels per task. Instead of classifying the original audio files, we use the associated Mel spectrograms following the methodology considered by Dong (018)) to retrieve an image classification setting. Though the benefits are not as striking as before on test accuracy, the ECE is slightly improved by combining our WAUM with CoNAL as can be seen in Table G Moreover, we show constant improvement of the test generalization performance using the WAUM preprocessing either in accuracy or in calibration.
Among other interesting discoveries, the WAUM helped us detect that the music Zydeco Honky Tonk by Buckwheat Zydeco was labeled as classical, country or pop by the workers, though it is blues standard.
12

---

Published in Transactions on Machine Learning Research (04/2024)
100%
100%
Votce
100%
2859
209
Salg
tallbun diniiiinn
tall Siiiiiiiee
farres
lagg highoass
pourn furcss
0%
coul
Spreen
bualectly
tall williiiiii
0ood
opencc
tallioi
(a) Label street.
(b) Label tallbuilding:
Figure 9: LabelMe dataset: Worst WAUM for classes (top) and the associated voting distribution for each image (bottom). (a) Label street (b) Label tallbuilding. Even if the two tasks are very similar, because the workers are different the associated proposed labels can differ and add noise during training.
Figure 10: LabelMe: top-10 worst images detected by the WAUM (with labels row-ordered from top bottom: highway, insidecity, street, tallbuilding). Overlapping classes lead to labeling confusion and learning difficulties for both the workers and the neural network.
Another example is Caught in the middle by Dio classified (with the same number of votes) as rock, jazz, or country though is a metal song. One last example detected: the music Patches by Clarence Carter is stored in the disco00020. wav file. The true label is supposed to be disco, while the workers have provided the following labels: two have chosen rock, two blues, pop and another proposed country. The actual genre of this music is country-soul, so both the true label and five out of six workers are incorrect.
WAUM sensitivity to the neural network architecture In the following, we explore the architecture's impact on the genoralization performance using the WAUM preprocessing. We compare three architectures, VGG-16 with two dense layers added from Rodrigues & Pereira (2018), Resnet-18 and Resnet-34. We show in Figure 12 that depending the network used, performance vary, but the WAUM step improves generalization performance in most cases (and does not worsen it.)
Limitations: computing the weights with many classes First, concerning the weights s (reflecting the trust in the
interaction), rely confusion matrices {*(1) jee messe). The DS model (Dawid Skene 1979) can be naturally used estimate such matrices T(1) E R for each worker wy. Yet, the quadratic number of parameters (w.r.t. K) to be estimated for each worker create convergence
13

---

Published in Transactions on Machine Learning Research (04/2024)
MV
MV
MV Ø
HV + Ø
AUMC
MV + AUMC
MV + WAUM
NV WAUN
NS
NS
NS O
NS AUMO
NS AUMC
NS WAUM
NS WALM
DS
DS
DS
DS + AUMO
DS + AUMO
DS + WAUM
DS WAUM
GLAD
GLAD
GLAD
GLAD
GLAD AUMO
GLAD AUMC
GLAD WAUM
GLAD WAUN
WDS
WDS
Ø
WDS AUMC
WDS AUIC
WDS WAUM
WDS WAUN
CrowdLayer
CrowdLayer
CrowdLayer
CrowdLayer
CrowdLayer + AUMC
CrowdLayer
AUMC
CrowdLayer WAUM
CrowdLayer
MAUM
CoNAL (A- 0)
CoNAL (A ===0]
CoNAL (0)
CONAL (0)
Ø
CONAL (0) + AUMC
CoNAL (0)
AUMC
CoNAL (0) WALM
CoNAL (0)
WAUM
CoNAL
CoNAL (A) A--ee 1e 4)
CoNAL (1e.4]
CONAL (1e-4)
2
CONAL (1e-4)
AUMC
CoNAL (1e-4)
AUMC
CoNAL (1c-4)
WAUM
CoNAL (1e-4)
WAUM
57.5
60.0
62.5
65.0
67.5
0.54
0.60
0.66
0.72
0.78
Accuracy
1- ECE
Figure 11: Ablation study on Music using the VGG backbone: = 0.05. Errors are Gaussian confidence intervals at 95%. Numeric tables are available in Appendix
issues for the vanilla DS model when K is large. But as stated in Section 3 any model that can estimate confusion matrices can be considered for the WAUM's computation. We detail below some possible variants, that could help computing the confusion matrices used in the WAUM for the trust score computation.
Sinha et al. 2018 accelerated the vanilla DS by constraining the estimated labels' distribution to be Dirac mass. Hence, predicted labels are hard labels. This leads to worse calibration errors than vanilla DS but preserves the same accuracy.
Passonncau & Carpenter 2014 introduced Dirichlet priors on the confusion matricos' rows and the prevalence p to incorporate previously known information on the workers in the model (e.g., from other experiments).
Servajean et al. (017 exploited the sparsity of the confusion matrices to cope with large K.
Imamura et al. 2018 estimated with variational inference L & Ryorker clusters of workers, con- straining at most T. different confusion matrices. This reduces the number of parameters required from K2 X Rworker to K2 x L.
Pruning and i. assumption For the pruning at preprocessing can induce distortion in the training data distribution. A usual assumption made on learning problems is that the task/label pairs are i.i.d. However, by removing some of the hardest tasks, the new dataset Dpruned contains tasks that are not independent anymorc. We should also keep in mind that Myas et al. (2022) have shown that in the standard datasets, the data not i. to begin with. Moreover, we should not set a too high as in imbalanced settings this might cause even more imbalance.
14

---

Published in Transactions on Machine Learning Research (04/2024)
85.0
0.90
62.5
0.88
0.86
ACCrest
77.5
CIFAR- 10H
72.5
1.80
0.78
67.5
0.76
WDS
HV
DS Strategy
Strategy
90
0.925
0.900
AcCrest
85
0.875
LabelMe
0.825
0.800
0.775
WDS
CoNAL A
CoNAL Am 10
=
A=10-
Strategy
Strategy
With WAUM pruning step
Resnet-18
Resnet-34
VGG-16
Figure 12: Performance obtained by training the pruned dataset from the WAUM preprocessing step on CIFAR-10H and LabelMe. We consider multiple neural network architectures ResNett-1,, ResNet-34 or VGG-16 with batch normalization and two supplementary dense layers. We show that performance in accuracy are improved in most cases. Calibration performance term of ECE fluctuate depending on the architecture considered, especially for the CIFAR-10H datasot. Using the WAUM with CoNAL on the LabelMe dataset, we obtain best performance both in accuracy and calibration.
5 Conclusion
In this paper, we investigate crowdsourcing aggregation models and how judging systems may impact gener- alization performance. Most models consider the ambiguity from the workers' perspective (very few consider the difficulty of the task itself) and evaluate workers on hard tasks that might be too ambiguous to be relevant, leading to performance drop. Using popular model (DS), we develop the WAUM, flexi- ble feature-aware metric that identify hard tasks and improves generalization performance over vanilla strategies and naive pruning AUMC. Tt also yields fair evaluation of workers' abilities and supports recent research on data pruning in supervised datasets. Independently of pruning, the WAUM allows identifying carly the images that need extra labeling efforts or that are impossible to correctly label.
Extension of the WAUM to more general learning tasks (e.g., top-k classification, Appendix would be natural, including sequential label. Indeed, the WAUM could help identify tasks requiring additional expertise and guide how to allocate more experts/workers for such identified tasks. Future works could adapt the WAUM to imbalanced crowdsourced datasets to identily potentially too ambiguous images that naturally occur in open platforms like Pl@ntNeff] And in this case, class-dependent pruning threshold quantile could be used to avoid a learning bias for classes with very few instances.
Last but not least, on the dataset side, we believe that the community would benefit from releasing a challenging datasct (such as the one by Garrin et al. (2021) for instance) tailored to learn crowdsourcing settings. Indeed, dataset with the following properties could greatly foster future research in the field:
https://plantnet..or///
15

---

Published in Transactions on Machine Learning Research (04/2024)
varying number of labels per worker, high number of classes, and subset with ground truth labels to test generalization performance.
Broader Impact Statement
As this work proposes a method to prune tasks from training datasets based on human-derived data, we remind that pruning based on learning difficulty can induce learning bias for the model. To mitigate this, only pruning small portion of the dataset can help avoid any class with small number of representatives to be removed of the dataset. Also, in this paper, only remove tasks that are difficult to classify, do not remove workers from the dataset. In particular, there is no repercussion their pay, and by only evaluating them on tasks that are not detected as ambiguous, we evaluate their abilities on fairer tasks. Finally, during the entire procedure, all anonymity is conserved for workers, no other data than their anonymous identification number is used.
Acknowledgments
This work supported by the ANR through the Chaire IA CaMeLOt (ANR-20-CHIAA-0011-0)
References
L. Aitchison. statistical theory of cold posteriors in deep neural networks. In ICLR, 2021.
I. Albert, S. Donnet, C. Guihenneue-Jouyaux, S. Now-Choy, K. Mengersen, and J. Rousseau. Combining expert opinions in prior elicitation. Bayesian Analysis, 7(3):503 532, 2012.
A. Angelova. Data pruning. PhD thesis, California Institute of Technology, 2004.
Angelova, Y. S. Abu-Mostafa, and P. Perona. Pruning training sets for learning of object categories. Tn CVPR, volume 1, pp. 494 501 vol. 1, 2005.
A. Lord Birnbaum. Some latent trait models and their use in inferring an examinee's ability, Statistical theories of mental test scores, 1968.
M. PJ. Camilleri and C. KI. Williams. The extended Dawid-Skene model. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 121-136. Springer, 2019.
7. Chu, J. Ma, and H. Wang. Learning from crowds by modeling common confusions. In AAAI, pp- 5832 5840, 2021.
AP. Dawid and A.M. Skene, Maximum likelihood estimation of observer error-rates using the EM algorithm. R. Stat. Soc. Ser. C. Appl. Stat., 28(1):20-28 -80 1979.
AP. Dempster, NM. Laird, and DB. Rubin. Maximum likelihood from incomplete data via the em algorithm.
J. R. Stat. Soc. Ser. B Stat. Methodol, 39(1):1-22, 1977.
M. Dong. Convolutional neural network achieves human-level accuracy in music genre classification. arXiv preprint Xiv::11802.00 09697, 2018.
T. Draws, A. Rieger, O. Incl, Y. Gadiraju, and N. Tintarev. chocklist to combat cognitive biases crowdsourcing. In AAAI Conference on Human Computation and Crowdsourcing, volume 9, pp. 48-59, 2021.
G. Gao and D. Zhou. Minimax optimal convergence rates for estimating ground truth from crowdsourced labels. arXiv preprint or::i:::11.0000 2013.
C. Garcin, A. Joly, P. Bonnct, A. Affouard, J.-C. Lombardo, M. Chouct, M. Servajean, Loricul, and J. Salmon. Plontnet-300k: plant image dataset with high label ambiguity and a long-tailed distribution. Tn Proceedings of the Neural Information Processing Systems Track Datasels and Benchmarks, 2021.
16

---

Published in Transactions on Machine Learning Research (04/2024)
C. Garcin, M. Servajean, A. Joly, and J. Salmon. Stochastic smoothing of the top-k calibrated hinge loss for deep imbalanced classification. In ICML, 2022.
P. Germain, A. Lacasse, F. Laviolette, M. Marchand, and JF. Roy. Risk bounds for the majority vote: from pac-bayesian analysis to learning algorithm. J. Mach. Learn. Res., 16:787-860, 2015.
MY. Guan, V. Gulshan, AM. Dai, and GE. Hinton. Who said what: Modeling individual labelers improves classification. CoRR, abs/ his/1703.08774 2017.
C. Guo, G. Pleiss, Y. Sun, and KQ. Weinberger. On calibration of modern neural networks. In ICML, pp. 1321, 2017.
J. Han, Luo, and X. Wang. Deep self-learning from noisy labels. In ICCV, pp. 5138 5147, 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pp- 770-778, 2016.
LN. Hoang, L. Faucon, A. Jungo, S. Volodin, D. Papuc, O, Liossatos, B. Crulis, M. Tighanimine, 1. Con- stantin, A. Kucherenko, A. Maurer, F. Grimberg, V. Nitu, C. Vossen, S. Rouault, and EM El-Mhamdi. Tournesol: A quest for a large, secure and trustworthy database of reliable human judgments. arXiv preprint xXxc:2107.07994 2021.
S. Ibrahim, X. Fu, N. Kargas, and K. Huang. Crowdsourcing via pairwise co-oceurrences: Identifiability and algorithms. Advances in neural information processing systems, 32, 2019.
A. Tyas, SM. Park, L. Engstrom, G. Leclere, and A. Madry. Datamodels: Predicting predictions from training data. arXiv preprini arXiv:2202.000 00622, 2022.
H. Imamura, Sato, and M. Sugiyama. Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In ICML, pp. 2147-2156 2018.
GM. James. Majorily vole classifiers: theory and applications. PhD thesis, Stanford University, 1998.
E. Jamison and I. Gurevych. Noise or additional information? leveraging crowdsource annotation item agreement for natural language tasks. In Conference on Empirical Methods in Natural Language Processing, pp. 291-297, 2015.
X. Jiang, M. Osl, J. Kim, and L. Maco-Maahadoo.. Calibrating predictive model estimates to support person- alized medicine. J. Am. Med. Inform. Assoc., 19(2):263-27) 274, 2012.
C. Ju, A. Bibaut, and M. Van der Laan. The relative performance of ensemble methods with deep convolu- tional neural networks for image classification. J. Appl. Stat., 45(15):2800 2818, 2018.
E. Kamar, A. Kapoor, and E. Horvitz. Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsoureing, 2015.
FK. Khattak. Toward a Robust and Universal Crowd Labeling Framework. PhD thesis, Columbia University, 2017.
H-C. Kim and Z. Ghahramani. Bayesian classifier combination. In AISTATS, volume 22, pp. 619-627, 2012.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
A. Kumar, PS. Liang and T. Ma. Verified uncertainty calibration. In NewrIPS, volume 32, 2019.
M. Lapin, M. Hein, and B. Schiele. Loss functions for top-k error: Analysis and insights. In CVPR, pp. 1478-147,, 2016.
Q. Li and PK. Varshney. Does confidence reporting from the crowd benefit crowdsoureing performance? In International Workshop Social Sensing, pp. 49-50 19-54, 2017.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Q. Ma and A. Olshevsky. Adversarial crowdsourcing through robust rank-one matrix completion. In NeurIPS, volume 33, pp.
2020.
Y. Ma, A. Olshevsky, V. Saligrama, and C. Szepesvari. Gradient descent. for sparse rank-one matrix comple- tion for crowd-sourced aggregation of sparsely interacting workers. J. Mach. Learn, Res., 21(1):5245 5280, 2020.
R. Müller, S. Kornblith, and GE. Hinton. When does label smoothing help? NeurIPS, 32, 2019.
NM. Müller and K. Markert. Identifying mislabeled instances in classification datascts. In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1 8, 2019.
C. Northcutt, A. Athalye, and J. Mueller. Pervasive label errors in test sets destabilize machine Icarn- ing benchrnarks. In Proceediigg of the Neural Information Processing Systems Track on Dalasels and Benchmarks, 2021a.
C. Northcutt, L. Jiang, and I. Chuang. Confident learning: Estimating uncertainty in dataset labels. J. Artif. Intell. Res., 70:1373-1411, 2021b.
S. Oyama, Y. Baba, Y. Sakurai, and H. Kashima. Accurate integration of crowdsourced labels using workers' self-reported confidence scores. In IJCAI, 2013.
SY. Park and C. Caragea, On the calibration of pre-trained language models using mixup guided by area under the margin and salieney. In ACML, pp. 5364 5374, 2022.
RJ. Passonneau and B. Carpenter. The benefits of model of annotation. Trunsactions of the Association for Computational Linguistics, 2:311-326 -26, 2014.
A. Paszke, S. Gross, F. Massa, A. Lerer, Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, T. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Newr/PS, pp. 8024 8035, 2019.
M. Paul, Ganguli, and GK. Dziugaile. Deep learning on a data diet: Finding important examples early in training. In NeurIPS, volume 34, pp. 20596 20596-20607, 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res., 12:2825 2830, 2011.
JC. Peterson, RM. Battleday, TL. Griffiths, and O. Russakovsky. Human uncertainty makes classification more robust. In ICCV, pp. 9617-9626, 2019.
Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric J Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. Advances in neural information processing systems, 30, 2017.
Emmanouil Antonios Platanios, Avrim Blum, and Tom Mitchell. Estimating accuracy from unlabeled data, 2014.
G. Pleiss, T. Zhang, ER. Elenberg, and KQ. Weinberger. Identifying mislabeled data using the area under the margin ranking. In NewrIPS, 2020.
Ravi Raju, Daruwalla K., and Mikko HL. Accelcrating deep learning with dynamic data pruning. CoRR, abs/2111.12622 2021.
VC. Raykar and S. Yu. Ranking annotators for crowdsourced labeling tasks. In NeurIPS, pp. 1809-1817, 2011.
F. Rodrigues and F. Pereira. Deep learning from crowds. In A AAAT, volume 32, 2018.
18

---

Published in Transactions on Machine Learning Research (04/2024)
F. Rodrigues, F. Percira, and B. Ribeiro. Gaussian process classification and active learning with multiple annotators. In ICML, pp. 433-441. PMTR, 2014.
M. Servajean, A. Joly, D. Shasha, J. Champ, and E. Pacitti. Crowdsourcing thousands of specialized labels:
A Bayesian active training approach. IEEE Transactions Mullimedia, 19(6):1376-138)) 2017.
VB. Sinha, S. Rao, and VN. Balasubramanian. Fast Dawid-Skene: A fast vote aggregation scheme for sentiment classification. arXiv preprint drXi::1803.02281 2018.
R. Snow, B. O'Connor, and A. Jurafsky, D.and Ng. Cheap and fast but is it good? evaluating non- expert annotations for natural language tasks. In Conference on Empirical Methods in Natural Language Processing, pp. 254 263. Association for Computational Linguistics, 2008.
B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and AS. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Xiviv preprint or Xiv:22200. 1 14486, 2022.
Y. Wen, G. Jerfel, Muller, WM. Dusenberry, J. Snoek, B. Lakshminarayanan, and D. Tran. Combining ensembles and data augmentation can harm your calibration. In ICLR, 2021.
J. Whitehill, T. Wu, J. Bergsma, Movellan, and P. Ruvolo. Whose vote should count. more: Optimal integration of labels from labelers of unknown expertise. In NewrIPS, volume 22, 2009.
Yue Xing, Ashutosh Pandey, David Yan, Fei Wu, Michale Fronda, and Pamela Bhattacharya. Training with low-label-quality data: Rank pruning and multi-roview, 2023.
F. Yang and S. Koyejo. On the consistency of top-k surrogate losses. In ICML, pp. 10727 10735, 2020.
H. Zhang, M. Cissé, YN. Dauphin, and D. Lopez-Pat Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
A. Zhong, J. Cui, S. Liu, and J. Jia. Improving calibration for long-tailed recognition. In CVPR, pp. 16489-16498, 2021.
D. Zhou, Q. Liu, JC. Platt, C. Moek, and N. Shah. Regularized minimax conditional entropy for crowd- sourcing. arxiv preprint ar 15v: 07240, 2015.
19

---

Published in Transactions on Machine Learning Research (04/2024)
A Popular label aggregation techniques
Several aggregation techniques can transform crowdsourced labels into probability distributions (soft labels). For any d E N and E o0)", let Nzrm(z) E (0, xc) be the vector defined by vi E [d], Norm(=); 2i/2 2//2 1==1 zg.
A.1 Naive soft (NS)
The naive soft (NS) labeling is simply the empirical distribution of the answered votes:
Vxs Derain:
V. Norm( where ÿi = (E
11 [(v()=a)) KE[K]
(6)
jEA(zz)
A.2
Majority voting (MV)
Majority voting (MV) outputs the most answered label:
Vx, E Derain,
arg max kE[K]
Σ JEA(())
1 (()0==1))
(7)
A.3
Dawid and Skene (DS)
The Dawid and Skene (Dawid & Skene 1979 model aggregates answers and evaluates the workers' confusion matrix to observe where their expertise lies exactly. Let us introduce P the prevalence of each label in the dataset (i.e., P(y, = f)), the probability that a task drawn at random is labeled é E [K]. Following standard notations, we also write {Te,, i E [Neasz]} the indicator variables for task i, that is Ti = 1 if the true label for task i is 8 (i.e., V E) and zero otherwise. Finally, let Ti.k be the probability for worker j to sclect label k when L. The model's likelihood reads:
(i)
II
II
Pt
II (1) (g
=x))
(8)
NE (raas)) KE(K]
1E [E(rroenaer
KE[K]
To maximize the likelihood, use the EM algorithm (Dempster al. 1977 to estimate the parameters (?)
"f and Pt, using (T(rase( as latent variables. Our implementation of the FM algorithm is given in Algorithm ? The convergence criterion we use in practice that the likelihood has not decreased more than > between two iterations. By default, e set to 10-6, and the EM algorithm stops at ileration N if Likelihood, Likelihoode41 < E.
A.4 Weighted Dawid and Skene (WDS)
Let us run the DS model to get estimated confusion matrices 40) E RRKK for j E [Nrorkar] Now, remind that for a given worker j [Rworker] and a class k E [K], the term
((3) estimate the probability for worker w to recognize a task whose true label is We use this term as trust. score and define the WDS soft label as
Vay E Diealn:
WDS No),(W), with (=
Σ
a * (9/)=))) KE[K]
(9)
jEA(z.)
A.5 Generative model of Labels, Abilities, and Difficulties (GLAD)
We recall the GLAD Whitchill et al. 2009) algorithm in the binary setting. A modeling assumption is that the j-th worker labels correctly the i-th task with probability given by
P ()) = vf vfla,,3)) 1 Teeaa -
(10)
20

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 2 DS (EM version)
Input: Dtrain: crowdsourced datasct
Output: (i) )iEinese
estimated soft labels and { [Mporarr] estimated confusion matri-
ces
1: Initialization: Vi E [nease] vl E [K], Th, [A(a;) ESEA(E)) 100
1
while Likelihood not converged do Get and I assuming I's are known
Eielmerl Tr (1 =2)
4: k(C,k) E [K]2,
a(3) 2,k + 21lE|/1 Levemnun
Ty. 10 =0))
5: V [K], PE -
Deane Eie(mual) The Estimate Is knowing and P
Iy HeEIn de (x(()) (#(22)"00 (m
V(i, () [Meaak] x [K],
Erei Ibreaw IwEII Per (e.(11t))) (#/22) (off)
8: end while
Algorithm 3 GLAD (EM version)
Input: Dtrain: crowdsourced dataset
Output:a {ag)je(mmmeer? worker abilities,
(eleaala task difficulties,
aggregated la-
bels
1: while Likelihood not converged do
2: Estimate probability of yr
vi E [Mtaek], (yt (0) )i, a, 3.) II, II,
4: Maximization step
5: Maximize auxiliary function ((a,3) in Equation
w.r.t. a and 8
6: end while
with 0j E IR the worker's expertisc: aj < 0 implics misundorstanding, aj O an impossibility to separate the two classes and aj > 0 valuable expertise. The coefficient 1/3: E R4 represents the lask's intrinsic difficulty: if 1/3 - 0 the task is trivial; on the other side when 1/3; - +00 the task is very ambiguous. Parameters (az)je([nunan) and (B:)se(maal are estimated using an EM algorithm as described Algorithm
The auxiliary function for the binary GLAD model is:
Q(a. 3) = Eloog P(1y 5) by tut bol = Σ Bog P P(v))] + Σ Ellog Ply W.
( /0 ag. B.]
(11)
An extension to the multiclass setting given by Whitehill al. (2009) under the following assumption: the distribution over all incorrect labels is supposed uniform. In this setting, the model assumption from Equation 10) still holds and
1 Wk + vi. P(O!) () = kkay, Bi) K-1 1 -1-1111
However, this is not verified in many practical cases, as can be seen for example in Figure 2c where the cat label is only mistaken deer and not with other ones. We have used the implementation from https://github. con//ootani//yytonnggaaa
to evaluate the GLAD performance in our experiments. The maximization of the function Q w.r.t. a and 3 is performed using a conjugate gradient solver. The initial parameters are all set to
A.6 CrowdLayer and its matrix weights strategy (MW)
From Roorigues & Pereira, 2018). CrowdLayer is an end-to-end strategy in the crowdsourcing setting. From the output of neural network, new layer called crowd layer is added to take into account worker
21

---

Published in Transactions on Machine Learning Research (04/2024)
specificities. The main classifier thus becomes globally shared, and the new layer is the only worker-aware layer. As multiple variants of Cyowdlayer can exist, we only considered in this paper the matrix weights (MW) strategy that is akin to the DS model. Denoting : = f(z;) the output of the neural network classifier f for a given task 2; labeled by a worker Wj, the added layer multiplics & by a matrix of weights W E R This matrix of weights per worker lakes inlo account. the local confusion of each worker. In practice, the forward pass F on task 2; annotated by worker wy using CrowdLayer computes F w1) = (f:))
A.7 Common Noise Adaptation Layers (CoNAL)
CrowdLayer takes into account worker-specific confusion matrices. CoNAL (Chu et al. 2021 generalizes this setting by creating a global confusion matrix W E R R addition to the local ones W E R for [[worker] working all together with the classifier f. Given worker W)s the confusion is global with weight wi and local with weight w The final distribution output used to compute the loss is given by:
Pout (Ti, wzy) w W fa:)) + (1- w/) f(x:)
As is, CoNAL local matrices tond to aggregate themselves onto the global matrix. To avoid this phenomenon, a regularization lerm in the loss can be added as leading to the final loss:
L(W9, (W2) ) Teaak Pesiil [Teokkce
Σ
()) Pout (ara, wz)) H HVv
| 1
with À the regularization hyperparameter and H the crossentropy loss. confusion weights are from the shared confusion.
The larger X, the farther local
B AUM and WAUM additional details
Algorithm wisker-wise WAUM.
Input: Derain: tasks and crowdsourced labels, a [0, 1]: proportion of training points pruned, T N: number of epochs, Est: Estimation procedure for the confusion matrices
Initialization: Get confusion matricos { [ from Est (= DS by default)
3: for Moorka]] do
4: for T epochs do
Train a neural net work for T epochs on D(1) = (vi)))") for i T(ws)) end for
7: Get AUM(,;, (2) ( (uain) using Equation
8: Get trust scores (x,) using Equation
9: end for
10: for each task x E Xerain do
11: Compute WAUM(2) using Equation
12: end for
13: Get da the quantile threshold of order a of (WAUM(z;))semall
14: Define Ppruned= = {(112) [2 )eAA))))) -WAUM(2)) 2 qa for i [maeek]
B.1 Unstacking workers answers in the WAUM: the worker-wise WAUM
In Algorithm [ the WAUM requires training a classifier directly from all votes. If the crowdsourcing experiment generates many answers per worker, for example when each worker answers all the tasks, modify Algorithm to train classifier per worker for T epochs instead of single one à la Guan et al.
(2017). This means that each classifier is only trained on DÚ) == {( (1) yf hee(now)) to compute the AUM of the tasks answered. We refer to this the worker-wise WAUM and give the full algorithm in Algorithm
22

---

Published in Transactions on Machine Learning Research (04/2024)
By doing so, the network trained for given worker is not influenced by the answers of the other workers. Hence, the AUM computed by this worker-wise WAUM is independent across workers (assuming workers are answering independently). One downside of this worker-wise application is its training cost that increases drastically. Where the vanilla WAUM adds a cost of epochs before training to identify ambiguous tasks, worker-wise WAUM adds cost of T x Nyokker epochs.
In the simulated examples we propose, we provide the results for the worker-wise WAUM, yet in such simulated cases with many labels per task, the results do not differ much from the WAUM; see for instance Table
B.2 AUM computation in practice.
We recall Algorithm 5 how to compute the AUM in practice for a given training set Derain This step is used within the WAUM (label aggregation step). Overall, w.r.t. training a model, computing the AUM requires additional cost: T training epochs are needed to record the margins' evolution for each task. This usually represents less than twice the original time budget. We recall that (1;)) is the softmax out of the predicted scores for the task at iteration t.
Algorithm AUM algorithm
Input: Drrain = (Ii Vi (E)sE()a))) training set with Deask task task /label couples, T E N: number of epochs for = I.....7 T do
Train the neural network for the tt epoch, using Drran
for [Ntac*] do
Record softmax output (0 E K-1
(t) Compute margin ( (2,,)) v:) = 0% (z;) - [ (x;)
end for end for E [Nraak], AUM(I,, yo Derain) = + EiE(T) ( (II
C Reminder on the calibration of neural networks
Hereafter, we propose reminder on neural networks calibration metric defined in Guo al. (017). Cal- ibration measures the discrepancy between the accuracy and the confidence of network. In this con- text, we say that a neural network is perfectly calibrated if it is as accurate it is confident. For each task x E Xtrai = {II...., Franc let us recall that an associated predicted probability distri- bution is provided by a(x) E AK-1- Let us split the prodiction interyal [0. 1] into M = 15 bins I,...., IM of size 1/M: Im = mm- m where m = I....,M, M. Following Guo et al. (2017). we denote
Bm {= E Xtrain [1)(x)) E Im} the task whose predicted probability in the m-th bir We recall
that the accuracy of the network for the samples in Bn is given by ace(Bm) the empirical confidence by co.)
ace aco(Bm) =
1
Σ B
((an()))))) and conf(Bm))
Σ [u((z;)
Bm
B
B
Finally, the expected calibration error (ECE) reads:
M
Bin |aco Bm) conf(B,..)
ECE
(12)
Deack
m=
A neural network said perfectly calibrated ECE O, thus if the accuracy equals the confidence for each subset Bm-
Remember that with our notation (x) arg mag (x()) with ties broken random.
23

---

Published in Transactions on Machine Learning Research (04/2024)
D Datasets description
D.1 Synthetic dataset
In this soction, we present simulated datascts to showcase the specificitios and possible limitations of the WAUM. Here summary of the experiments detailed in the following sub-sections:
The three_ circles dataset: we explain further how the simulations in Section were conducted
The two mrommooss dataset: we showcase a setting where the ambiguous tasks should be kept and not pruned. No simulated worker was able to get past the intrinsic difficulty of the datasct.
3.
The make_classication, aateaaationmman__eeeee many_workers dataset: showcase a setting with many workers and few labels per task. In this case, it is more relevant to consider the WAUM instead of the worker-wisc WAUM.
D.1.1 The three three_circles circles dataset
This dataset presented in Section we give additional details here. We simulate three cloud points using scikit-learn's function wwo_circles. Each of the Utaak = 525 points represents task. The foorrkr = 3 workers are standard classifiers: Wi is linear Support Vector Machine Classifier (linear SVC), w2 is an SVM with RBF kernel (SVC), and wy is gradient boosted classifier (GBM) with five estimators. To induce more ambiguity (and avoid too similar workers), the SVC has a maximum iteration set to 1 in the learning phase. Other hyperparameters are set scikit-learn's default values Data is split between train (70%) and test (30%) and each simulated worker votes for each lask, i.e., for all Xrain, disagreement area is identified in the northeast area of the dataset as can be seen in Figure [ Section also shows that pruning too little data (a small) too much (a large) can mitigate the performance.
[A(x)] Ylorrar The
D.1.2 The two_moons dataset
This dataset is introduced the worker-wise WAUM method. The two moons simulation framework showcases the difference between relevant ambiguity in dataset and an artificial one. This dataset is created using make __oons function from scikit-learn. We simulate Tlask 500 points, a noise = 0.2 and use a test split of 0.3.
where pruning is not recommended, to illustrate the limitations of
Wi
W122 SVC
Ground truth
Linear SVC
GBM
WAUM worker- wise
20.1
AUM
0.5
0.20.40.60.8
0.5
0.8
0.6
0.25
0.4
Figure 13: two _ro_moons dataset: simulated workers with associated AUM and normalized trust scores. The hyperparameter set to 0.1 for the worker-wise WAUM. Notice that the SVC classifier is mostly wrong (since only train for one epoch for this worker), inducing lower trust score overall.
As can be observed with Figure 13 and Figure 14 the difficulty of this dataset comes from the two shapes leaning into one another. However, this intrinsic difficulty is not due to noise but is inherent to the data.
For instance, the squared-hing penalized with e regularization parameter loss the multinomial deviance, and the maximum depth equals 3 (default).
for linear SVC and SVC, GBM
24

---

Published in Transactions on Machine Learning Research (04/2024)
WAUM
Ground truth
MV
NS
DS
GLAD
worker-wise
Train (hard labels)
0.8
Train (solt labels)
Test (hard labels)
0.4
Test (soft labels)
Figure 14: two two_moome moons dataset: One realization of Table varying the aggregation strategy. Label predictions on train/test sets provided by a three dense layers' artificial neural network (30, 20,,20,2200 20) trained on smooth labeled obtained by after aggregating the crowdsoureed labels (as in Figure from the training set in the worker-wise WAUM aggregation. The a hyperparameter is set to 0.1. Each point represents task Fú, and its color is the probability to belong in class 1. One visualize the ambiguity in the soft training aggregated labels, but also in the resulting predictions by the neural network. Errors represented are standard deviations.
Points in red are pruned
Table 3: Training and test accuracy depending on the aggregation method used for the two crrommoon''s 's dataset with Ntaex = 500 points used for training a three dense layers' artificial neural network (30, 20, 20). For reference, the best worker is W3 with a training accuracy of 0.923 and a test accuracy of 0.900.
Aggregation
AcC
ECE
MV
0.894 + 0.002
0.098 0.004
NS
0.887 + 0.002
0.217 + 0.010
DS
0.867 = 0.000
0.126 0.001
GLAD
0.872
0.107 0.004
worker-wisc WAUM(a 10-3)
0
0.088
worker-wise WAUM(a 10-2)
4.844
0.092
worker-wise WAUM(a = =10-1)
0.870 0.003
0
worker-wisc WAUM( = -0.25)
0.829 + 0.006
+
In this case, removing the hardest tasks means removing points at the edges of the crescents, and those are important in the data's structure. From Table 3 we observe that learning on naive soft labeling leads to better performance than other aggregations. But with these workers, no aggregation produced labels capturing the shape of the data.
D.1.3 The make_ www.sssicaaaiiin.nnmmmmcm _acl___ann______ workers dataset
We simulate nw = 150 workers who answer tasks from a dataset with K = classes simulated using scikit-learn's function cake_claassficcttion In this sctting, the WAUM has the same performance the worker-wise WAUM, with a much lower computational cost (as we do not train Towoker networks but single one). All simulated tasks labeled by up to five workers among Linear SVCs, SVCs Gradient
25

---

Published in Transactions on Machine Learning Research (04/2024)
Boosted Classifiers (GBM) chosen uniformly. To simulate multiple workers with some dissimilarities, randomly assign hyperparameters for each classifier as follows.
Each Linear SVC has a margin C chosen in a linear grid of 20 points from 10-3 to 3, a maximum number of iterations between and 100, and either hinge or squared hinge as loss function. Each SVC has a poly (with degree 3), rbf or sigmoid kernel and a maximum number of iterations between and 100. Finally, each GBM has learming rate of 0.01, 0.1 0.5, given number of base estimators {1, 2, 5, 10, 15, 20, 30, 50, 100} and maximum number of iterations between 1 and 100. All simulated workers are also initialized using different seeds. All hyperparameters drawn uniformly at random from their respective set of possible values.
Table 4: The make_ classification many_ workers dataset: Performance metrics by aggregation method.
The number of tasks Reack = 250 tasks per classes and 1 < A(xx) S 5. Errors represented are standard deviations.
Aggregation
ACCrest
ECE
NS
0.851 0.00
0.146 0.023
DS
0.849 + 0.004
0.242 + 0.011
GLAD
0.842 + 0.002
0.196 + 0.004
worker-wise WAUM(a 10-1)
0.849 + 0.006
0.137 + 0.034
WAUM(a= = 10MM((==00)) 10-1)
0.861 + 0.007
0.156 + 0.023
D.2 Real datasets
The datasets we consider are all decomposed into three parts: train (Dtrain,, validation (Dvai), and test (Dret)) They are described in the following subsections. In particular, we provide for the training set of each dataset (see Figures 15 17 and 18 three visualizations: the feedback effort per task distribution (IA(x))), the load per worker distribution ([()))), and the naive soft labels entropy distribution, i.e., the entropy distribution for each task in the training set, defined by: Vz, E Xusinn Ent(z,) ΣxE[K] N log((1))) )x).
We have conducted experiments three real datasets. The CIFAR-10H dataset has been proposed to reflect human ual uncertainty in (a subpart of) the classical CIFAR-10 dataset. Each worker has annotated a large number of (seemingly easy) tasks, thus leading to few disagreements. The LabelMe and Music datasets have very few votes per task, leading to more ambiguous votes distributions.
D.2.1
The CIFAR- 10H dataset
304
62%5
363
285
47.5
205
455
15%
085
6038
100
-50
12%
Nusilor
such
Nutiler
(a) Peodback effort per task distribu-
(b) Load per worker distribution
(c) Naive soft labels, entropy distribu- tion
tion
Figure 15: CIFAR-10H: dataset visualization
Introduced by Peterson al. (2019), the crowdsourced dataset CIFAR-10F attempts recapture the human labeling noise present when creating the dataset. We have transformed this dataset, mainly by creating
26

---

Published in Transactions on Machine Learning Research (04/2024)
validation set. Hence, the training set for version of CIFAR-10H consists of the first 9500 test images from CIFAR-10, hence [Derain| = 9500. The validation set is then composed of the last 500 images from the training set of CIFAR-10 meaning [Drett = 500. The test set consists of the whole training set from CIFAR-10, so Dtest| = 50000. The crowdsourcing experimentation involved Noorker = 2571 workers on Amazon Mechanical Turk. Workers had choose one label for each presented image among the K = labels of CIFAR-10: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. Each worker labeled 200 tasks (and was paid $1.50 for that): 20 for each original category. Answering time was also measured for each worker The CIFAR-10H annotating effort is balanced: each task has been labeled by 50 workers on average.
ACAP-99
1SAM-
MN-40
KM
J6N:
AM
AUMCHSUL
AUNC-6.12
AUMC-6-19
NUNG-6J0
NUNG-GAE
MAM--MM
ABK-4O
1M
WM--1
KUNC A.
AUNC-CW
NUNC-AM
AUNC-SIR
MM0- 11
VAAM
NaP-oR
(a) WAUM crowdsourced identifica- tion
AUMC crowdsourced identifica-
(c) AUM ground truth identification
tion
Figure 16: Comparison of the worse images detected by the WAUM, AUMC and classical AUM preprocessing step. Identification was computed with a ResNet-18 for 50 epochs using the parameters described in Section Each row represents the class given by the unobserved ground truth label from the CIFAR-10 datasct. Only the AUM uses the ground truth label, other methods are based on the crowdsourced labels only. Images framed in red can be hard to classify.
D.2.2
The LabelMe dataset
Another real dataset in the crowdsourced image classification field that can be used is the LabelMe crowd- sourced dataset created by Rodrigues & Pereiral 2018 This dataset consists of Baakk 1000 training images
Note that attention checks occurred every trial each worker, for tasks whose labels known. They have been removed from the dataset since the corresponding images not available.
27

---

Published in Transactions on Machine Learning Research (04/2024)
30/
40.5
305
203
20.5
125
06
Nuntee mustaling
(a) Feedback effart per task
(b) Load per worker distribution
(c) Naive soft labels, entropy distribu-
tion
Figure 17: LabelMe: dataset. visualization
dispatched among K classes: highway, insidecity, tallbuilding, street, forest, coast, mountain open country. The validation set has 500 images and the test set has 1188 images. The whole training tasks have been labeled by Yoorker = 59 workers, each task having between one and three given (crowd- sourced) labels. In particular, 42 tasks have been labeled only once, 369 tasks have been labeled twice and 589 received three labels. This is a way sparser labeling setting than the CIFAR-10H dataset.
Also, note that the LabelMe dataset has classes that overlap and thus lead to intrinsic ambiguities. This is the reason why the CoNAL strategy was introduced by Chu et al. (2021), see details in Appendix AA.7 For example, the classes highway, insidecity, street and tallbui Simuiilding lding (in rows) are overlapping for some tasks: some cities have streets with tall buildings, leading to confusion as shown in Figure [0] The proposed feature aware aggregation using the WAUM leads to better performance in test accuracy and calibration as illustratd Table
D.2.3
The Music dataset
363
375
1150
15%
202
15%
183
10%
100
Number sooken
Number
(a) Feedback effort per task distribu-
(b) Load per worker distribution
(c) Naive soft labels, entropy distribu- tion
tion
Figure 18: Music: dataset visualization
Rodrigues et al. (2014) relcased crowdsourced dataset of audio files. The goal of this classification task to decide the genre of 30 seconds musical excerpts. Number of tasks is Utask 700. The Dearker = 44 workers had K = 10 possible labels: blues, classical, country, disco, hiphop, jazz, metal, pop and reggae. Each audio file was labeled by between and workers. To test the results, dataset of 299 labeled clips is used (originally 300, but one file is known to be corrupted). Instead of working with the original audio files, we have used Mel spectrograms, openly available to rely on standard neural net works architecture for image classification.
28

---

Published in Transactions on Machine Learning Research (04/2024)
Strategy
ACCrest (%)
ECE
Strategy
AcCrest (%)
1- FCR
MV
85.4 1.0
0.864 + 0.01
MV
59.9 + 1.23
0.631 0.01
MV + AUMC
86.0 1 1.1
0.859 1 0.01
MV + AUMC
62.0 1 1.23
0.650 1 0.02
MV WAUM
86.1 0.9
0.858 1 0.02
MV WAUM
61.1 1 2.35
0.624 0.02
NS
86.1 1.0
0.862 + 0.01
NS
59.9 = 1.40
0.624 0.02
NS + AUMC
87.2 0.8
0.882 = 0.01
NS + AUMC
59.9 = 1.41
0.640 = 0.02
NS + WAUM
88.1 1.0
0.890 1 0.02
NS + WAUM
62.1 1: 2.18
0.642 i 0.01
DS
86.8
0.877 1 0.01
DS
62.9 1.72
0.661 1 0.01
DS AUMC
86.3
0.841 1 0.02
DS AUMC
1.222
0.659 0.01
DS + WAUM
8..
0.862 = 0.02
DS WAUM
62.1 + 2.81
0.640 0.02
GLAD
8.9
0.881 = 0.01
GLAD
61.5 1.72
0.639 0.01
GLAD AUMC
87.6
0.861 + 0.03
GLAD + AUMC
61.6 0.93
0.664 0.01
GLAD WAUM
88.2 0.8
0.885 1 0.02
GLAD WAUM
61.5 1.23
0.645 0.01
WDS
0.7
0.838 0.02
WDS
60.2
0.652
WDS + AUMC
0.7
0.862 0.02
WDS AUMC
62.9
0.647
WDS + WAUM
87.1
0.871 + 0.01
WDS WAUM
63.1 0.91
0.660 0.02
CrowdLayer
4.2
0.858 + 0.04
CrowdLayer
63.2
0.02
CrowdLayer + AUMC
3.5
0.809 0.05
CrowdLayer AUMC
0
0.617 0.04
CrowdLayer + WAUM
3.2
0.860 + 0.03
CrowdLayer WAUM
6.42
0.680 + 0.03
CoNAL(A = 0)
8.
0.881 = 0.01
CoNAL(A = 0)
6.92
0.660 0.02
CoNAL(0) AUMC
8..1
0.903 1 0.02
CoNAL(0) 1 AUMC
64.3
0.735 1001
CoNAL(0) 1 WAUM
8
0.892 + 0.01
CoNAL(0) WAUM
6..6
0.735 1011
CoNAL(A = 10-4)
6.4
0.863 + 0.06
CoNAL(X=
64.2
0.639 0.06
CoNAT(1004)) + AUMO
88.4 2.3
0.884 + 0.04
CoNAT,(10-)) + AUMC
64.1 + 0.74
0.745 1 0.02
CoNAL(10-4) WAUM
90.0 1:00:000 0.8
0.901 1 0.01
CoNAL(10 WAUM
64.4 1 0.78
0.726 0.02
Table 5: Ablation study on LabelMe using the VGG backbone: a 0.01. Errors represented are standard deviations.
Table 6: Ablation study on Music using the VGG backbone: a 0.05. Errors represented are standard deviations.
E Algorithmic details on the neural network training
Experiments can be reproduced using the code available at htttes//gthhu..cco/pprrmo///ppa from the peerannot library which briefly described below:
The identification module is used to explore datasets tasks and workers. Tasks can be explored thanks to the entropy of the label distribution, the WAUM or the AUMC. Workers can be evaluated thanks to the Spam-score of Raykar Yu (0011), the trace of the DS estimated matrices, GLAD's parameters among other.
The aggregate module is used to produce aggregated labels from multiple answered labels. The labels can then be used for training a neural network archilecture from Pytorch using the train module.
The deep module is used for the CoNAL and CrowdLayer strategies. A neural network is directly learning from the crowdsourcod tasks and labels without the aggregation step.
Multiple datasets are ready use, including CIFAR-10H, LabelMe and Music.
The documentation of the library is at https::/paerannot.ggie 10/
F Results on LabelMe and Music datasets
In this section, we provide the raw results tables of the forest plots (Figure g and Figure 11 the real datasets Table 5 the LabelMe and in Table for the Music datasets. The errors presented are standard deviations.
29

---

Published in Transactions on Machine Learning Research (04/2024)
G Margin comparison
In the AUM, AUMC and WAUM formulac, we consider a margin from Yang & Koyejo 2020 (denoted d/s in the original article) that has better theoretical properties for top-k classification but that is not the margin proposed in Pleiss et al. 2020 (11)) Indeed, our margin in the AUM is written as:
0 ( (z.) [2 (xa)
instead of
(t) (2:) max ox (1)
(x:)
91
Using the CIFAR-10H datasel, library used (and briefly described in Appendix D switching from the original margin to the top-k based margin is executed with the argument use _se_pleiss-True or use use_plee _ee_pess==aalse with the WAUM, AUM and AUMC. comparison of the images with lowest AUM is provided in Figure 21 Similar visual comparison on the CIFAR-10H dataset provided in Figure 24
compare the identified tasks using each margin. Note that in the
Figure 19: Lowest AUM with th margin
Figure 20: Lowest AUM with tho margin
Figure 21: Comparison of the images with lowest AUM in CIFAR-10H dataset using the margin from Pleiss et al. 2020 (#1) or the margin for top-1 classification from Yang & Koyejo (0020) (1s). Both margins yield
similar results.
Furthermore, we provide ablation study top-2 accuracy scores using the WAUM with the ths margin the LabelMe dataset in Table 7 We use the crossentropy loss during the training phase. With ths, the top-2 margin writes o (x;) o (xx) as indicated in Section 3.1 We compute the top-2 accuracy f.e. the accuracy in recovering the true label as the first or second predicted label by the classifier. However, note that this dataset has K classes, hence we do not report top-5 accuracy all strategies perform similarly. We notice that performance this dataset similar for most strategies between the two margins used for pruning.
(t)
30

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 22: Lowest WAUM with db
Figure 23: Lowest WAUM with s/s
Figure 24: Comparison of the images with lowest WAUM in CIFAR-10H dataset using the margin from Pleiss et al. lead to similar results.
(2020) (wh) the margin for top-1 classification from Yang & Koyejo (220)) (v/s). Both margins also
Table 7: Top-2 Accuracy comparison the LabelMe dataset using the modified VGG-16 backbone and
the same hyperparameters in Section ! deviations.
Results are averaged over 5 repetitions, errors are standard
Strategy
Top-2 no pruning
Top-2 WAUM(*)
Top-2 WAUM(
MV
91.25 + 2.01
91.17 + 2.12
92.02 + 2.08
NS
90.92 + 1.53
90.41 + 2.77
89.91 + 1.08
DS
89.98 + 1.12
90.24 + 0.92
91.41 + 0.99
GLAD
90.78 0.98
91.34 1.59
90.49 - 0.38
WDS
89.56 1.76
90.82 1.81
91.16 2.78
CrowdLayer
87.45 2.03
88.33 1 1.49
88.57 1 2.51
CoNAL(A- 0)
92.34 0.74
89.49 + 0.53
94.30 + 1.32
CoNAL(A= = 10
91.68 + 1.01
94.10 0.9
94.93 + 0.76
31


Reviews:
[{"review_id": "v4frqOSf87", "reviewer": "Reviewer_r8Ez", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- A novel method is proposed that empirically works better than baselines (and also another proposed naive baseline.)\n- Experiments are conducted for both synthetic ones and benchmark datasets. Synthetic datasets provide insights into the behavior of the proposed method. Benchmark datasets include a scenario with higher number of labels per data (CIFAR-10H) and fewer number of labels per data (LabelMe, Music).\n- Well-documented code is provided in the submission. \n- It is interesting that the method is applicable not only for label noise and out-of-distribution scenarios, but also for class-overlap type of situations (shown with the experiments with LabelMe dataset.)\n\nWeaknesses\n- I may have missed it, but the details about the experimental setup were not clear. For example, what is the test dataset in CIFAR-10H? (I understand the training and validation dataset which is explained in page 10)\n- A related comment is that, it would be more appropriate if we can check the accuracy with the test dataset, for the best hyper-parameter alpha chosen based on the validation dataset. (For example in Section 4.1, it seems there is only train and test dataset (but no validation dataset.) This may be helpful to avoid overfitting on the evaluation dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A novel extension of the Area Under the Margin (AUM) method for crowdsourcing data with multiple labels called Weighted Areas Under the Margin (WAUM) is proposed by using task-dependent scores. It can be used to identify harmful data points in the dataset, so as a step 2, the user can prune the dataset based on the WAUM score of each data point. Also a naive baseline called AUMC is proposed. Experiments show the behavior of the proposed methods. Additionally, experiments show comparison with many baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I wrote some comments in the \"weaknesses\" in the previous section. I believe it would be beneficial for the paper to take these two points into consideration.\n\nA very minor comment: In the rebuttal letter, the authors wrote they updated the paper to avoid using the term \"harmful\". It seems that the version I am reading still uses this term 3 times.", "broader_impact_concerns": "I have no societal concerns with respect to this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OTMLimQrvw", "reviewer": "Reviewer_GkhP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper tackles an interesting problem.\n- The paper is well-written and easy to follow.\n- The proposed method makes sense intuitively.\n\n### Weaknesses\n\n- The experiment results do not seem statistically strong, though I'm not sure if I'm assessing this correctly based on the information provided in the paper.\n- The paper is lacking comparison to simple baselines for pruning the underlying dataset. A very simple baseline could be to prune based on the variance of the crowdworker labels for each task.\n- The assumptions that the proposed method makes are not discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method for identifying ambiguous tasks in datasets where each task has been performed by multiple crowdworkers. The proposed method, Weighted Areas Under the Margin (WAUM), is an adapted version of Area Under the Margin (AUM), which averages AUMs weighted by task-dependent scores. They perform experiments on both synthetic and real datasets showing that the proposed method is able to identify ambiguous tasks and subsequently improve the performance of classifiers by training them on cleaned versions of the underlying dataset (i.e., with the ambiguous tasks removed).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Algorithm 1 also requires training a classifier as part of step 4 right (i.e., obtaining the trust scores). If that's the case, then you should clarify that in the algorithm and also in the \"dataset pruning\" paragraph. Relatedly, this approach to computing trust scores assumes that the classifier that you train is good. How good does it need to be? You should elaborate a bit more on this in Section 3.3 in my opinion.\n\n2. In the \"dataset pruning\" section you mention that you are pruning based on a quantile of the WAUM scores. Why did you choose to use a quantile instead of a threshold on the values themselves? I am a bit concerned that using a quantile means you will always throw away some data even if it is good. I would at least clarify that in the paper and also provide some justification for why you are using a quantile instead of a threshold on the WAUM value itself.\n\n3. In the implementation details section of your experiments, you may want to clarify that by \"three dense layers' artificial network\" what you actually mean a three-layer perceptron (i.e., a three-layer MLP). There is also a typo in the penultimate sentence of that paragraph (\"wether\" -> \"whether\").\n\n4. In the experiments section when you present tables with results you have some error bars in \"+-\" values. You should say what those quantities are (i.e., are they standard deviations, standard errors, etc.), ideally in the caption of each table. In those same captions you should also explain what the bolded values represent, and especially so for tables 2, 3, and 4 where they are the largest value per section. I would also recommend you separate those sections in those tables with lines/borders to make it even more clear.\n\n5. Related to those same tables, a lot of the results do not seem statistically significant. I would use a visual cue for representing those that are statistically significant (e.g., bold and red or something to differentiate it from just bold).\n\n6. In Section 3.2 you say \"... the higher the AUM, the more confident the prediction is in the assigned label. Hence, the lower the AUM, the more likely the label is wrong ...\" This is confusing \"confidence\" with \"correctness\" which is a strong assumption. I think you should not make that statement.\n\n7. In the introduction, you sometimes use the word \"harmful\" to characterize the instances of the data that you prune. I think this can be misinterpreted and a more accurate characterization would be to use \"ambiguous\" throughout.\n\n8. There is some related work that you might find interesting and pertinent to this work, starting with Platanios, et al. \"Estimating Accuracy from Unlabeled Data\" (2014), that is followed by 2-3 more papers by the same authors exploring variations of this problem.", "broader_impact_concerns": "I found the Broader Impact Statement included in the paper to be sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lKlQEelYBq", "reviewer": "Reviewer_p4kV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Straightforward combination of existing work\n- Well-organized, with clear notation\n- Thorough experiments show clear improvements when combined with multiple aggregation baselines\n\n### Weaknesses\n\n- It is unclear why the authors define AUM with $\\psi_5$ in Table 1 of Yang & Koyejo (2020) rather than with the much more standard margin definition $\\psi_1$. The argument in Section 3.2 about top-$k$ settings is not convincing because the main metric considered is top-1 accuracy. The authors should justify this, report top-$k$ accuracy, and consider performing an ablation with $\\psi_1$ margin.\n\n- Typo: \"which is weights the margins\" should be \"which weights the margins\"\n\n\nF. Yang and S. Koyejo. On the consistency of top-k surrogate losses. In ICML, pp. 10727–10735, 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of classification with crowdsourced labels. Each annotation task is to label an example in the dataset, and some tasks may have noisier labels than others. The authors propose to leverage the Area Under the Margin (AUM) ranking to filter out tasks with high ambiguity, in order to improve generalization on the test set. Experiments show that two variants (hard label AUMC and soft label WAUM) are compatible with standard crowdsourced data aggregation methods, leading to improved classification accuracy and calibration error on synthetic and standard image classification datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Critical\n- Justify the use of non-standard margin in Equation (2)\n- Report top-$k$ accuracy\n\n\n### Non-critical\n- Ablation with standard margin in place of Equation (2).\n- Include AUMC method in Table 1\n- Comment on whether methods in Section 4 are common/simple baselines or state-of-the-art methods for crowdsourced label aggregation\n- Cite and discuss/compare with recent work which combines data pruning and crowdsourcing, Xing et al. (2023).\n\nXing et al. [Training with Low-Label-Quality Data: Rank Pruning and Multi-Review](https://dmlr.ai/assets/accepted-papers/7/CameraReady/Improving_Scammer_Detection__Algorithm_and_Multiple_Reviews%20(3).pdf), in ICML Workshop on Data-centric Machine Learning Research, 2023.", "broader_impact_concerns": "Concerns are addressed sufficiently at the end of the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were satisfied with the updates of the paper made during the review phase and all recommended acceptance of the paper. Hence, as the paper also tackles a relevant and interesting problem, I am recommending acceptance in line with the reviewers' recommendations.
Decision: Accept (poster)