AQuA: A Benchmarking Tool for Label Quality Assessment
Authors: Mononito Goswami
Vedant Sanil
Arjun Choudhry
Arvind Srinivasan
Chalisa Udompanyawit
Artur Dubrawski
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Label errors
evaluation
design space
benchmarking
TL;DR: We propose a benchmark to evaluate methods that enable machine learning in the presence label noise. We also introduce a design space for labeling error detection models.

Abstract: Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. _ImageNet_, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark to evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance in the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment _AQuA_ to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation of machine learning tools facing mislabeled data.

AQuA: A Benchmarking Tool for Label Quality Assessment
Mononito Goswami", Vedant Sanil, Arjun Choudhry',
Arvind Srinivasan! Chalisa Udompanyawit, Artur Dubrawski
Auton Lab, School of Computer Science
Carnegie Mellon University
{mgoswami, vsanil, arjuncho, arvindsr, cudompan, awd) awd)@cs.cmu.edee edu
www www.githube wtw.tthw........ttottomoo wwt.//atoooaaaaauuu.....
Abstract
Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. ImageNet, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation machine learning tools facing mislabeled data.
Introduction
lot of machine learning (ML. research is devoted to making efficient and effective use of available data to learn accurate, high-fidelity, and interpretable models, with little focus on the quality of the data they are trained and evaluated on. Nonetheless, it is widely recognized that ML models are only good the data they rely on, i.e., the quality of data imposes practical limits what ML models can achieve. Not only are datasets used to train ML models; they also serve as benchmarks to measure the state-of-the-me art and validate theoretical findings. Thus, high quality large labeled datasets are the cornerstone of progress in supervised machine learning. However, the data is rarely free of noise, which can both manifest in the features of the data (feature noise) and labels that categorize them (label noise). Between feature and label noise, the former has been found to be much more harmful to machine learning models 11 3 To make matters worse, label noise is prevalent in popular MI benchmarks. A recent study estimated an average of at least 3.3% label errors across 10 datasets commonly used for benchmarking computer vision, natural language, and audio classification algorithms [. Consequently, growing body of research is devoted to understanding the harms of label noise and to developing techniques to identify and mitigate labeling errors.
MG and VS contributed equally. MG is the corresponding author.
1 AC AS have equal contribution.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Datasets
Synthetic Noise Injection
Detection
Error
Classification
Evaluation
Predictive
ALM
Appranezi
Cos Rud--
Ara under MOC
ancnp DIPNCCONTT
Contiert Learnigg
Then
Single labels
Das Ceal-bbBEITTa
Falmena Party
ReoNee-1
Diswnting Label
Generalisation
Gharpennu-baee
Crowet Vojorty
infffes
X2
X
Multi-annolator
Robustness
Figure 1: Overview the AQuA benchmark framework. AQuA comprises of datasets from 4 modalities, 4 single-label and 3 multi-annotator label noise injection methods, 4 statc-of-the-a label error detection models, classification models, and several evaluation metrics beyond metrics of predictive accuracy. We are in the process of integrating several faimess, generalization, and robustness metrics into AQuA. The red and blue arrows show two example experimental pipelines for image data and time-series data, respectively.
In recent years, over 50 papers have been written this topic, including 6 surveys, yet the literature lacks a comprehensive benchmark to evaluate the available methods. The evaluation of existing methods is lacking along the following dimensions:
Arbitrary choice of datasets and limited data modalities. To the best of our knowledge, relevant studies have used over 40 datasets (e.g., ImageNet (5)) and their variations (e.g., Imagenette [6]. ImageNet-100 7 for evaluation, but mostly on computer vision related tasks, with less than 15 studies using text data, 7 using tabular data and only paper using time-series data.
Arbitrary choice of classification models. The ultimate goal of identifying labeling errors is to learn classification model using training data with clean labels. Much like the datasets, relevant studies have used over 47 different classification architectures (e.g., ResNet [ MobileNet (}]
ResNeXt IO. BERT I XLM-RoBERTa [2], etc) to measure the impact of label cleaning.
Inconsistent evaluation protocols and metries. Different studies conduct different experiments measure the efficacy of their proposed methods (e.g., the accuracy of the label cleaning method, or performance of the downstream model before and after label cleaning, etc.) and use various measures of success (e.g., high accuracy, -cooe,, or low error rate).
With such diversity and inconsistency in the way in which these methods are evaluated, it is hard to measure the state the art. To bridge this gap, we propose the Annotation Quality Assessment, AQuA, the first benchmark framework evaluate machine learning methods in the presence of label noise (Fig. [ We also elucidate the design space for such models, with the hope that will not only foster future research on detecting labeling errors, but also enable ML practitioners to choose the appropriate label cleaning tools for their specific data and tasks. We run a large-scale experiment (> 1000 unique experiments) and make several interesting observations, demonstrating AQuA's 's efficacy in benchmarking machine learning models in the presence of label noise.
2 Background and Problem Formulation
Sources of labeling errors. Labeling errors can arise from automated labeling processes such as crowd-sourcing [3], programmatic weak supervision [14 15). and human error (e.g., due to lack of
expertise or low confidence in expert assessment) [16 Errors may also stem from idiosyncrasies
'The root cause of labeling errors crowd-soureing is different from human expert annotation. For instance. errors during nowddsourcing have been shown to arise from other factors such as gaming the system maximize monetary gains []]

---

of the annotation procedure and the corresponding guidelines themselves 17 labels may also become inconsistent with prevailing knowledge due to constantly evolving problem definitions and domain knowledge leading to concept drif
Finally, existing
Impact of labeling errors. At training time, labeling errors can cripple an ML model's abil- ity to generalize and introduce undesirable bi- ases in its hypothesis space 19 20 Mislabeled training data especially problematic for over- parameterized deep neural networks, which can achieve zero training error even on randomly- assigned labels (20). At test time, labeling errors can lead to noisy model evaluations and inval- idate common model selection strategies. In safety-critical settings, models trained, evalu- ated, and selected using mislabeled data can be ineffective at best and can lead to disastrous outcomes at worst, Finally, recent studies in the context of fairness have shown that naively enforcing parity constraints based noisy labels can harm groups that are unaffected by label noise 21 22).
"Buser uueer just modelling Planty of women make of money just for looking good"
cat
kniwwar
normal
not offonslve
frog
dress
tarmythmia
otfensive
Figure 2: Labeling errors in widely used bench- marks: CIFAR-10, Clothing- 100hing-100K,, MIT-BIH, and TweetEval Hate Speech datasets. Observed labels are in red and true labels are in green.
Problem formulation. Due to the far-reaching consequences that labeling emrrrs can have on model training and evaluation, the literature has attacked multiple different but related problems, for example: (1) label error detection, identify which data points have erroneous labels (23) 24 (2) label noise estimation, estimate the proportion of data with noisy labels (25), (3) label noise robust learning, learn models robust to label noise 22 27 and (4) noise transition matrix estimation, estimate the parameters of the noisy label generation process
28
In this work, we focus on the label error detection problem, because (a) it is the most general of the above problem types, i.e., with knowledge of labeling errors, we can estimate the noise rate, parameters of the noise generation process and train ML models free from label noise, (b) it provides practitioners greater visibility of issues that plague their data, and (c) allows them directly rectify these errors.
Label error detection problem: Assume a dataset 2 = [(x,, ) E (X.))). where x; and y denote the features and labels, respectively. In practice, we do not have access to D but instead observe noisy dataset D- {(x1, y: ) (X We call yı labeling erron") if yı * ye. and correctly labeled, otherwise. Our goal identify all labeling errors in D.
"We assume that we observe the true features since we are interested in identifying labeling errors and isolating their impact downstream model performance.
AA note on terminology: In this paper, we will sometimes refer to labeling ermors as noisy labels or label noise, and the process of identifying them as label error detection, or loosely as label cleaning.
3 A Design Space of Labeling Error Detection Models
In this section, we seek to align the dimensions along which label error detection models vary, with dimensions that can facilitate model selection for ML practitioners. We provide brief overview of these dimensions below and defer detailed discussions to Appendix A.T
What inputs do you have? All label error detection models take features and noisy labels as input. In most datasets, data points are labeled by multiple experts, but their individual annotations are seldom available. When available, mulii-annotator labels can be used to identify data points that are inherently ambiguous [2 or to model individual annotators to estimate their expertise and propensity for mislabeling examples 7 and using these to identify likely labeling errors. While most methods identify labeling errors and automatically remove or correct them, a few rely on a human
*For example, sepsis one of the most sought-afice after clinical conditions to predict. However, with the constantly evolving definition of sepsis, the labeling process is frequently affected, causing many annotations in legacy benchmark data to become inconsistent with the latest guidelines [8] very dangerous risk to take in the particular type of application area
3

---

Some other methods assume access data points called anchor points, which most certainly belong particular class B1 32] The number of anchor points required is generally proportional to the number of classes, and quickly becomes prohibitive for multi-class classification problems, and in more complicated noise settings Finally, vast majority of methods assume access to classification models, and primarily differ in their number (model-free [ one or multiple models
expert who can be queried to relabel suspicious data points 29 30
33
67 38) ), nature of access versus access logits [24], gradients 30) etc.), and extent of pre-training (no
35 3G
(prediction-only 23
pre-training
2 2
versus large-scale pre-training e.g. large language models
What modeling assumptions can you make?
Different studies use different assumptions on
What have?
data (noise structure and clusterability). heuris-
tics (model self-confidence and perceptual un- certainty), and modeling decisions (whether to explicitly model the transition matrix and multi- network training). Most studies in the literature explicitly assume some form of structure in the noise present in the data 2X
Frog
Cat
Frog
Frog
Gat
What modeling sesumptions
40
PX Most
Call Frgg
early studies assumed class-dependen noise, ie., the likelihood of eπror is only dependent on the latent true class, not on the data [23 (28].
Krovidge
Noitee diauiare
Heurekk
Vodkling Deditoon
There is growing interest in more realistic forms of noise where the probability of error also de- pends the features of data point (instance- dependent noise)
What outputs de went?
What would you do with the cubbus??
To this end, some recent studies have shown promising results by leveraging natural notions of similarity between data points and their labels. For example, Zhu et al. 44 assume that examples with similar features should have similar labels.
Cat
Figure 3: Design space of labeling error detection
models to delineate concrete design choices.
Many studies treat a trained model's low confidence that a data point belongs to its observed label as In similar vein, recent study used the
heuristic likelihood to identify labeling errors [24] 23 [R]
loss of pre-trained large language model on each data point to identify mislabeled examples [39]. When multi-annotator labels are available, as discussed before, some studies have also used them to model the perceptual uncertainty in the annotators to identify labeling errors.
Finally, studies differ in their modeling decisions. While some explicitly estimate a data structure called the noise transition matrix, which encodes the joint probability latent true and observed 8O 14] Finally, there is body of work on label noise
noisy labels 2
3
2
others do not 24
robust learning using multiple model instances either using knowledge distillation 35 36 44 or meta-learning 33
The key idea is to a cooperative game between models to identify labeling errors and ensure that the eventually deployed model only learns from clean data.
37
What outputs do you want, and what would you do with them? All labeling error detection models identify data points that are likely to labeling errors. With knowledge of the potentially mislabeled data points, most studies simply remove them from consideration 23 24 be practical for large datasets, where only a small fraction of data is found to be mislabeled and domain experts are unavailable for supervision. We use this strategy by default in AQuA. A smaller number methods predict the alternate class that the data point is most likely to belong to B8 [E] and even provide explanations for their predictions 30 46 CINCER B80 is one of the few methods which not only finds labeling errors but also identifies counter-example
42 This strategy may 45
in the training data to serve as expl explanations for its suspicion. Some studies use the label predicted by these models and perform loss re- re-weighting or correction to learn robust classification models 27
47 48
When domain experts
available, some studies also leverage their insight to re-label mislabeled data point

---

Mooialit
Datawet
Train/
Ammotators/aamll
Label Source
Chocolcaation Task
Sumple
Usagee
TOK
ausotation
Objest
Image
CIFAR-10H
1o0h
1OK
47-63
Huma ausotation
reb-labeeled
Object
Imuge
NoayCKR
26K
XX
Homan expert ammoration
Pseumonia
1024 1024
IMDD>
25K/
Huma ansotation
Scatimen
TweetEvel
10K
Human anzotation
Hate speech
Crdis Carl Fraue
384K
Human imnotation
Crodit card frou
A [X]
45K
Rule based extraction
Salary
Dry Dean
13K.
Visios systems based innotation
Bean varieyy
Car Evallatton 2M
IK
Ileearchical decision model
KH
Car conditoo
Mustroom: 14
SK
Muskmon edibiiity
22
K50
COMPAS 74
6K
Recidivism
28
A1
Crop [
TK/ lok
Fiersnhiicll k-meats Inee with dynamic time warping 5e
Crop cover
46 x
Time Seris
FleisidDDvvis 55
9K/ 7K
Haman annotation
Applianczttpe
96.6
MITTBIH K0
23K 14K
Human expesi ammitation Homan iumotations
Achhyhhmia
256
PeaDiglt 50
7K/3K
Harabriiten digit
16:x
WhaleCalle
X3
11K/2K
Whole call
4,000 x
Table 1: Summary of datasets. AQuA currently includes a variety of datasets for different classifi-
cation problems, varying in the number of classes, sources of annotations, and data modalities. All datasets except those marked with 3 are multi-class.
4 Benchmark Design
4.1 Real-world, Popular Datasets, and Downstream Classification Models
Datasets. AQuA currently comprises of a collection of 17 popular real-world public datasets from 4 prevalent data modalities: image, text, time-series and tabular. To evaluate label error detection models across various practical scenarios,
carefully choose datasets with diversity in the following characteristics: (1) classification problems (e.g., sentiment classification vs. hate speech detection), (2) number of classes (binary vs multi-class classification), (3) relative prevalence of classes (c.g.. skewed datasets like Credit Card Fraud (55) and balanced ones like IMDb 5 ) (4) sources of annotations (e.g., human
mase- annotation), and (5) number of annotations per example (e.g..
CIFAR- 10N labeled by 3 annotators). Table I summarizes the key characteristics of datasets included part of AQuA. In particular, to make comparison with prior work easier while maintaining diversity across practical scenarios, we try to include datasets that have been used frequently by prior work (see usage in Table D and preprocess them in manner consistent with those works. We do not any data augmentation during training. Apple A.3 provides detailed descriptions of the datasets.
Classification models. The ultimate goal of label cleaning is train accurate downstream classifiers, but different studies use different classification models to measure the efficacy of their proposed label cleaning methods. To provide level playing field for all cleaning methods, include mul- tiple classification model architectures for each data modality. Specifically, we include ResNet- 18 6 for image datasets, al1-iistirroberaa all-MiniLM-L6-ve v2 [22 for text datasets, ResNet-1D, PatchTST (7
M MobileNet
9 and FastViT-T8
ZO
ril and
and LSTM Fully Convolu- tional Network 77 for time-series datasets, and Tranffrrorrr 5 and Multi-Layer Perceptron for tabular datasets. While choosing classification models we prioritized performant methods with (1) different architectures and inductive biases, (2) ideally pre-trained using different strategies, and (3) previously-used either by label cleaning methods or task-relevant papers. App. A.4 and App. A.5 provide a detailed description of classification models and their hyperparameters, respectively.
4.2 Advanced Label Error Detection Methods
AQuA provides easy-to-use Application Programmer Interfaces (Fig. 4 for 4 ofute-of-the-art label error detection methods, namely Area Under Margin ranking (AUM) 2 Confident Learning 23 Contrastive and Influent Counter Example Strategy (CINCER) BC and Model-free Label Error Detection (SimiFeat)
54 Below, we provide a brief overview of these methods and their key ideas.
Area Under the Margin Ranking (AUM) [4]] Given noisy data and access to the logits of a deep learning model, AUM exploits differences in training dynamics of clean and mislabeled samples to identify labeling errors. The key idea is to identify data points that do not contribute to the generalization of model as labeling errors by leveraging the delicate tension between the label of data point (via memorization) and its predicted label (via gradient updates), measured as the margin between the logits of a sample's ' assigned class and its highest unassigned class.
5

---

Confident Learning (CON) 23
Given noisy from qusa.oddele inport TrainkgKodel, Convilet from aqua. data import Aqdata, liad_ from rqua. inport generate_repart
data, confident learning estimates data struc- ture called confident joint, which is the joint probability distribution of observed noisy and latent true labels. The key idea is to leverage a model trained on held-out data drawn from the same (or similar) distribution to predict the probability that an example x; belongs to its ob- served label y, A low probability is then used as heuristic-likelee likelihood of y; being a label error. The confident joint can then be used to identify labeling errors and estimate the noise rate.
# Load CIPAB-10 and Reellet-18
clf Convilet ('resnet18') data (ead_cifer() data. ddta.add aad__noiee(ooissratteee noise
/ Add uniform
Instantiate cleaning wethod claseifier cleaner TrainkgKodel (elf, nethod=' label_errore cleaner iemu
# Remove data with label issues aeaa.clean_dat/le (aaaaallttaaaaliiiaaaaaa _itaee))
# Train dounstrean model cleaned data y-prads (rafs) fit_predicct
Contrastive and Influent Counter Example Strategy (CINCER or CIN) [30]. CINCER treats the problem of identifying labeling errors as a sequential decision making problem where a domain expert can be queried to relabel suspi- cious examples. CINCER uses the same heuris- tie as AUM to identify labeling errors, but also identifies exampess in the data to serve as explanations of the model's 's suspicion.
Figure 4: AQuA makes identifying label issues, and evaluating new and existing label error detection models simple.
Model-free Label Error Detection (SimiFeat) (3). Unlike other methods, SimiFFat does not need a ppre-)trained model to identify labeling errors. Instead, it utilizes labels of the l nearest neighbors to identify labeling errors based on the clusterability assumption, i.e. data points with similar features should have the same true label with high probability.
There are many methods to detect labeling errors, but we choose these methods as a starting point because they are recent, state-of-the-are and have different inputs and core assumptions. While all these methods have existing public implementations, through AQuA, our goal is to create a one-stop shop for using and evaluating open-source label error detection models.
4.3 Evaluation
There is significant variance in the ways that label cleaning methods are evaluated. To rigorously, fairly, and systematically assess these models, we unify the breadth of experimental settings through the following three dimensions of evaluation.
Supervision. Identifying labeling errors in practice is an unsupervised problem since do not know which data points are mislabeled. Hence, evaluating these methods is a challenging endeavor. Most studies in the literature gather noise labels either from human experts (human-in-the-loww evaluation) or by introducing synthetic label noise by design (synthetic label noise).
In human-in -imman----thh-loopp evaluation, true labels of data points identified as having erroneous labels 39 4. While this is a straightforward and precise evaluation method, it is in general unscalable, expensive, cime-consuming, and limited to only measuring the precision of models (and not recall), because the experts are typically only shown data points which model considers erroneous.
or more human experts are asked independently assess the
much more common and scalable way of evaluating these methods to introduce various kinds of synthetic label noise and measure model's ability to detect them. There are many ways of introducing label noise, but injected noise may not always be reflective of the true noise that occurs in natural datasets, and hence identifying realistic noise injection strategies is active area of research 33 3
76 77. Moreover, model evaluation may still be noisy because there may be mislabeled examples for which our pscudo-noise labels are negative (or correctly labeled).
Hypotheses. In general, existing studies evaluate two hypotheses: (I) cleaning labels on the train set improves the performance of the downstream classifier on the test set, and (2) cleaning methods can accurately identify mislabeled data on the train set. Hypothesis is practical since the primary goal of identifying labeling errors is to train accurate and unbiased classifiers. However, appropriately regularized deep learning models are known to be naturally robust to some label noise. Hence, hypothesis 2 allows researchers to directly measure the efficacy of label cleaning techniques.

---

Measures of goodness. Different studies use different measures of predictive accuracy. While some measure error rate [24]. others report the accuracy ES] or ROC-AUC (2 of their classification models. Similarly, for their cleaning methods, some studies report the F₁ score while others report
the precision or recall 23 24
More gaps in evaluation. In addition to the lack of consistency, we believe that the experimental settings in many studies are occasionally (1) unrealistic, e.g., adding label noise to more than half and (2) uni-dimensional, c.g. reporting only one (sometimes up to 80%) of the data points 24 23 metric of predictive performance.
AQuA's design. To enable a realistic, multi-faccted and holistic evaluation of label error detec- tion models, implement 7 popular label noise injection techniques and multiple metrics of predictive performance. Specifically, for single-label datasets, we implement asymmetric [34), class- dependent (76), instance-dependent [3), , and uniform (76) noise, and for datasets with labels from multiple annotators, we implement dissenting label, dissenting worker, and crowd majority B9 In terms of metries of predictive accuracy, we implement Fi, accuracy, (weighted) precision, recall, area under ROC curve (ROC-AUC), average precision (PR-AUC), and error rate. We are in the process of implementing some other metries beyond predictive accuracy, such as generalization [78 and robustness 779] of models. Our hope is that AQuA's 's config-driven design will allow non-technical users to integrate into their labeling workflows and researchers to add new models, datasets, and evaluation pipelines seamlessly. Our choice of datasets and downstream classifiers ensures that the computational complexity of running experiments is not prohibitive. Finally, we make all code. pre-trained models, and experimental logs open-source to enable rigorous and fair evaluation of models.
5 Experiments, Results and Discussion
Datasets
Uniform
Asymmetric
Clas--eependent
Instance-eppendent
AUX
CTM
CDN
STM
AUX
CTM
CDR
STM
AUX
CTM
CDN
STM
AUX
CTM
CDN
STM
CIFAR-10
73.3
74.1
45.6
76.7
74.3
70.8
47.7
75.5
93.5
80.5
42.6
93.6
68.0
69.9
44.8
70.9
Ckithing 100K
70.0
73.6
71.6
NossyCXR
75.2
74.4
43.2
74.5
73.7
71.5
39.5
73.5
84.7
78.7
31.4
88.4
68.0
69.8
43.3
72.1
IMDb
75.6
73.3
58.4
78.5
75.7
74.3
59.5
7M.7
92.1
91.0
62.8
95.01
69.7
70.2
56.4
74.5
TweetEval
75.3
75.2
58.9
77.7
75.8
76.0
57.4
77.6
69.2
67.9
52.4
70.2
69.6
62.4
73.2
Credit Fraud
75.8
75.8
73.3
78.1
75.7
75.8
$0.0
76.7
63.3
63.0
87.2
72.0
69.5
69.4
74.3
73.5
Adult
25.7
75.8
72.9
75.8
75.8
66.9
63.6
64.6
61.2
64.9
70.2
72.4
Dry Bean
75.7
91.6
42.1
82.2
75.7
84.9
39.0
80.3
87.2
95.0
35.4
92.1
69.5
83.1
35.8
77.5
Car Evaluation
75.3
83.5
77.4
84.1
75.6
80.2
75.7
81.6
77.3
87.5
83.2
81.2
70.1
78.8
78.5
77.0
Mustrooms
82.5
75,4
74,3
COMPAS
75.8
74.9
63.2
75.9
75.8
74.8
64.6
76.5
55.5
57.1
52.9
$7.7
69.5
69.4
61.0
73.1
Crup
76.0
79.0
16.3
73.1
75.8
73.6
16.2
70.1
29.1
40.8
51.2
63.7
69.5
63.2
163
63.8
Electric Devices
75.8
82.2
35.0
79.3
75.7
78.6
35.3
75.8
37.8
50.5
55.9
68.3
69.9
71.5
31.7
69.2
MIT-BIH
88.4
49.7
75.7
51.3
68.2
75.7
45.4
80.6
69.6
78.4
75.2
Penliigits
75.8
89.0
23.1
73.4
75.7
83.1
23.4
72.7
46.7
44.9
53.5
78.4
69.9
76.0
19.8
68.1
WholeCatls
75.6
74.9
60.3
77.3
75.7
75.5
61.8
77.2
42.3
44.7
52.4
47.1
69.6
69.1
59.2
71.2
Table 2: Performance evaluation of cleaning methods to detect erroneous labels across different types
of synthetic noise added to the train set in terms of weighted F1, averaged across noise rates and downstream models.
(
(ii)
(iv)
(v)
(vi)
Figure 5: Critical difference diagrams representing rankings of cleaning methods across: (7) all datasets, (iil) only image or (iv) only text datasets. (v) also shows the ranking of cleaning methods across all datasets when accuracy is measured instead of weighted Fi (c.f. i). Finally, (ii) represents the performance of downstream models trained using cleaned labels, and (vi) performance of all cleaning methods disaggregated by noise type.
7

---

Datrett
Neise Injected
Uniferm
Asymmnetric
deo--deemmentt
Instunex- depecllett
C3M
0ΣR
01R
8.3
231
3IM
E3RAR-10
76.5
76.1
73.0
46.0
73-5
63-2
67.6
65.0
16.9
163-4
15.1
6311
67.3
16.3
63.1
71.0
T1.7
70.5
46.7
67.5
91.1
141.2
62.1
34.0
56.8
Clohing 100%.
50.9
00.7
90.5
50.5
90.2
92.5
70.5
83.2
85.4
83.2
$2.8
$1.A
79.2
79.4
83.1
90.6
83.3
82.1
83.5
86.1
75.9
74.6
71.7
$1.6
$1.6
NasyCRR
56.0
156.5
50.7
25.2
57.8
49.6
49.4
53.3
19.1
48.2
49.6
48.93
59.6
16.0
47.9
54.2
54.8
55.8
18.5
53.8
46.4
146.7
49.3
15.9
46.7
65.1
Cledin Froud
100
59.99
19.9
59.1
09.9
99.9
99.99
99.9
55.5
99.9
99.0
99.9
99.9
99.8
99.6
06.6
199.
99.9
66.6
99.9
99.8
99.9
15.7
99.
Adult
50.0
80.9
10.00
79.0
34.0
12.1
K2.1
185.0
75.0
51.9
83.5
8311
81.0
77.9
3310
81.8
12.3
83.1
31.9
14.00
12.5
5016
81.5
73:1
$1.1
Dry Hore
91.9
141.
01:3
57.2
14.0
19.5
91.4
00.6
78.6
15.5
870
W9.3
15.1
87.
91.4
91.0
M0.5
79.
04.33
81:1
63.6
84-1
43.0
87.5
Clr Evaluatos
93.4
02.5
85.9
57.6
92.0
$3.8
52.5
90.0
66.5
$1.1
96.1
K5.2
74.1
62.7
52.6
$6.9
63.2
78.0
57.5
$3.9
$2.6
$1.5
75.3
60.2
$0.5
Mishroors
99.7
99.9
99.5
59.7
99.9
98.3
10.2
97.6
$9.0
98.7
97.9
97.5
95.4
57.3
96.5
99.5
100
99.3
59.1
99.9
55.3
96.9
95.
$1.1
95.8
CUMPAS
67.2
67.3
66.2
63.7
66.6
05.6
02.1
65.9
58.6
05.3
66.1
68.8
66.2
46.9
05.6
54.3
65.1
63.5
35.5
66.2
61.6
53.0
01.8
45.6
63.7
Elctric Devices
BIH
12.7
81.3
74.4
78.0
709
PeiDighs
64.3
64.5
64.4
57.8
06.4
65.
54.7
Whaletalls
682
50.9
51.0
47.4
47.5
Table 3:
Impact of label noise and each cleaning method on weighted F1 ScOre of a downstream model
for each modality on the test set, averaged across noise rates and downstream models. Highlighted cells indicate better performance than that obtained without label cleaning (NON).
We conduct several experiments to support AQuA's 's design choices and demonstrate its utility in providing comprehensive and holistic evaluation of machine learning models in the presence of label noise.
Experimental Setup and Hyper-Parameter Tuning. We run experiments for all combinations of cleaning methods (AUM (AUM), confident learning (CON), CINCER (CIN) and SimiFeat (SIM), including no label cleaning (NON), noise types (asymmetriic, class-dependent, instance-dependent and uniform); for four different noise rates (0%, 2%, 10% and 40%), for a total of 2400 unique experiments. We conduct experiments using three distinct classification architectures for image and time-series data, and two different architectures for text and tabular data. To account for class imbalance in some datasets, all other evaluation metrics can be found in App. A.8 We also adopt critical difference diagrams &C to succinctly represent comparisons between multiple cleaning methods and other independent variables (e.g., data modality and noise type) on multiple datasets. These diagrams represent the average ranks of methods across datasets while grouping those with insignificant difference We tuned hyper-parameters of all the classification and cleaning methods till they performed reasonably well on average on all the datasets using hyper-parameter grids used by prior work and reported in App. A.5 Finally, all our experiments were carried out on a computing cluster, with a typical machine having 128 AMD EPYC 7502 CPUs, 503 GB of RAM, and 8 NVIDIA RTX A6000 GPUs.
report the Fi weighted by the support of each class. Results for
Research Questions. We aim to answer the following research questions through our experiments:
Which is the best cleaning method in terms of (i) its ability to identify synthetically injected label noise, and (ii) performance of the downstream classifier trained its cleaned labels?
Do the rankings of cleaning methods differ across different (i) types of synthetic label noise, (fi)
data modalities, and ( evaluation metrics (weighted Fi versus accuracy)?
5.1 Insights from Large-scale Experiments using AQuA
Tables and Fig. report results from all our experiments aggregated by noise rate, and down- stream c classification models. Below we highlight some of our key findings. Due to lack of space, we defer finer grained results App. A.8
Best cleaning method. Overall, we found SimiFeat (SIM) to be the best cleaning method in terms of its ability to identify synthetically injected label noise. closely followed by CINCER (CIN) BC (Fig. 5tt).. However, these differences shrink when evaluating cleaning methods using the performance of the downstream model trained using their cleaned labels (Fig. (i)). Confident learning (CON) 2 consistently performed the worst among all the evaluated methods.
"To form cliques, we abandon the posthoc test in favor of pairwise tests with Holm's 's correction for multiple testing based prior work [1] 32
"We deliberately did not perform extensive hyper- parameee tuning to not overfit to already existing label noise in the original datasets. Also, in practice it is unclear how to tune these cleaning methods well, without explicit knowledge of where the label emrrr are.
8

---

Deep learning models are inherently robust to label noise. Perhaps unsurprisingly, we found that most downstream classifiers were reasonably robust to synthetic label noise, as can be seen from the insignificant difference between the setting where datasets were not explicitly cleaned (NON), compared to when they were cleaned using SIM, CIN and AUM. These results also illustrate the importance of measuring both hypotheses (performance of cleaning methods versus downstream models) when evaluating the performance of ML models in the presence of label noise.
Adding label noise can sometimes improve model performance. In the context of class-dependent or uniform noise, label noise serves as regularization to prevent models from overfitting. This phenomenon is not specific any one modality, but happens for multiple modalities, datasets, and noise types too, for example Electric Devices serees) under uniform noise, MIT-BIH time--sries), and Dry Bean (tabular) for class- class-dependent noise, in Table T6] Moreover, deep learning optimization is highly non-convex, so adding some noise might help the model reach the global minima by traversing an alternative path within the loss landscape.
Impact of AQuA's design choices. We found that cleaning methods perform differently for different data modalities, For instance, all cleaning methods barring CON perform on par on image datasets (iii), but on tabular data (iv), AUM performs significantly worse than CIN and SIM. This may be due to a variety of reasons beyond cleaning methods: size and nature of datasets, inductive biases of downstream classifier, and the quality of feature representations [4 We also observed that some types of label noise are easier to detect than others. For example, uniform noise and asymmetric noise were the easiest to detect, cleaning methods found it much hard to detect instance and class-dependen noise (vi). Finally, we noticed differences in model rankings when measuring different evaluation metrics. As an example, the difference between CIN and AUM vanishes when we measure the accuracy (v) of the cleaning methods instead of their weighted F1 (i). These findings highlight the need to evaluate label error detection methods across multiple datasets from different modalities, noise types and evaluation metrics.
6 Conclusion and Future Work
We propose the first benchmark designed to rigorously evaluate machine learning models in the presence of label noise. We also elucidate the design space of these methods to not only enable ML practitioners to choose the right label cleaning tool for their data, but also foster academic research on the label noise problem. We demonstrate AQuA's 's utility by running large-scale experiments glean several interesting findings. We believe that, as a benchmarking toolkit, AQuA would benefit from more cleaning methods, datasets, synthetic label noise injection strategies, and evaluation metries.
Our short-term goals include experimenting with multi-annotator label noise, measuring the impact of feature noise on time-series and image data in comparison to label noise, incorporating several metrics for model generalization, robustness and fairness, and including audio datasets. While other types of noise are beyond the scope of this work, we believe that noise in regression problems are exciting avenues of future work, and AQuA's modular design will enable researchers to experiment with both multi-annotator and multi-class multi-label classification problems easily. We restrict ourselves to multi-class but single-label classification (as opposed to multi-label classification).
multi-class multi-label and
We believe that future work on label error detection should address label issues in the multi-label classification and regression settings. We believe that our work on AQuA can both hamess and facilitate the development of foundation models in the two ways: (1) foundation models can be used to identify labeling errors, without explicit supervision, and (2) methods within AQuA can be use to identify labeling errors which can affect foundation model pre-training and fine-tuning. We also believe that future work shoul
7 Limitations, Biases, and Social Impacts
We acknowledge the potential adverse impact of large-scale experimentation on the environment, but believe that our publicly accessible code and experimental findings can significantly reduce resource consumption for ML practitiioone in this field. Label enror detection models might perpetuate existing biases and impact the fairness of models. We included the Adult dataset, that is frequently used in the fairness literature, in AQuA, to evaluate the impact of label errors on the faimess of models. We

---

would also like to acknowledge that our experiments were carried without extensive parammeeeeeeer tuning. Moreover, papereparaeetes for cleaning methods and downstream classifiers were chosen based on model performance on the observed training set and fixed throughout the training process. We futher discuss these design choices and their limitations in Appendix A.6
Acknowledgments and Disclosure of Funding
We would like to thank Cherie Ho and Jack H. Good for their useful comments initial drafts of the paper. This work was partially supported by the National Institutes of Health under awards R01HL141916, IROINS124442-01, and IRO1DK 186-00. and by the U.S. Army Research Office and the U.S. Army Futures Command under Contract No. 191 The content of the information does not necessarily reflect the position the policy of the government and no official endorsement should be inferred.
References
[1] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions neural networks and learning systems, 25(5):845- 100884556699.0 2013.
[2] Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study. The Artificial Intelligence Review, 22(3):177, 2004.
[3] José A Sáez, Mikel Galar, Julián Luengo, and Francisco Herrera. Analyzing the presence of noise in multi-class problems: alleviating its influence with the one-vs-one decomposition. Knowledge and information systems, 38:179 18:1799006, 2014.
[4] Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, December 2021.
[5] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC),, 115(3) 2015. doi:
[6] Jeremy Howard. imagenette. URL
[7] Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Mare Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S Boning. Learning from multiple annotator noisy labels via sample-wise label fusion. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 407-422. Springer, 2022, ISBN 978-3-031-20053-3.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[9] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017. URL URL htttp://rxiiv org/abs//704..453 04861
[10] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5995, 2017. doi: 00.1109/CVPR.201..6544
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, 2019.
10

---

[12] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zet,lemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4450-8451, Online, July 2020. Association for Computational Linguistics. doi: 1221 747. URL httpe://aalaanthologg org wttp://g///.......o acl main.ainn.....
[13] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. A survey of crowdsourcing systems. In 2011 IEEE third international conference on privacy, security, risk and trust and 201 IEEE third international conference social computing, pages 766-773. IEEE, 2011.
[14] Jieyu Zhang, Cheng-Yu Yu Hsich, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433, 2022.
[15] Mononito Goswami, Benedikt Boecking, and Artur Dubrawski. Weak supervision for affordable modeling of electrocardiogram data. In AMIA Annual Symposium Proceedings, volume 2021, page 536. American Medical Informatics Association, 2021.
[16] Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Hu- man uncertainty makes classification more robust. In Proceedings of the IEEE/CVE In- ternational Conference on Computer Vision, pages 9617-9626, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10.1 10.1109//CV.2219..00 19.979... URL httpss://ooi Leeecomputersociety. org/10. 1109/ICCV 2019. 2019.
[17] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07/59 2020.
[18] Daniele Roberto Giacobbe, Alessio Signori, Filippo Del Puente, Sara Mora, Luca Carmisciano, Federica Briano, Antonio Vena, Lorenzo Ball, Chiara Robba, Paolo Pelosi, et al. Early detection of sepsis with machine learning techniques: a brief clinical perspective. Frontiers in medicine, 8:617486 2021.
[19] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning.
pages 233-242. PMLR, 2017.
[20] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
[21] Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 526-536, 2021.
[22] Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance-dependent label noise. In Conference on Causal Learning and Reasoning, pages 927-943 PMLR, 2022.
[23] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
[24] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the area under the margin ranking. In Proceedings of the 34th International Confer- ence on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN
[25] Curtis G. Northcutt, Tailin Wu, and Isaac L. Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. In Proceedings of the Thiriy-Third Conference on Uncertainty Artificial Intelligence, 1A AUAI Press, 2017. URLhttp hwtiii/ogguaaai11.... //auai org/ www./1///11///// htw.22017//occeeinnng wtwpiiinggs/papp//////. /appps/35.ppp
11

---

[26] Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and An- drew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL. attp://arxiv.orr abs////s////// 1412. 6596
[27] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944-1952, 2017. doi: 10.1 10110CVRR.2011.2444
(28) Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in neural information processing systems, 32, 2019.
[29] Melanic Bernhardt, Daniel Coelho de Castro. Ryutaro Tanno, Anton Schwaighofer, Kerem Tezcan, Miguel Monteiro, Shruthi Bannur, Matthew Lungren, Aditya Nori, Ben Glocker, Javier Alvarez- and Ozan Oktay. Active label cleaning for improved dataset quality under resource constraints. Nature Communications, 13, 03 2022. doi: 10. 00.1088/44146--022-255
[30] Stefano Teso, Andrea Bontempelli, Fausto Giunchiglia, and Andrea Passerini. Interactive label cleaning with example-based explanations. In Neural Information Processing Systems, 2021.
[31] T. Liu and D. Tao. Classification with noisy labels by importance reweighting. IEEE Transac- tions on Pattern Analysis & Machine Intelligence, 38(03):447-461, mar 2016. ISSN 1939-3539. doi: 10.1 T09M 22556999.
[32] Clayton Scott. A Rate of Convergence for Mixture Proportion Estimation, with Application to Learning from Noisy Labels. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 838 846, San Diego, California, USA, 09-12 May 2015. PMLR. URL https:ttp::///e p/tp://d////ccceoesseeoe mlr press/ https/////scoottt05 html
[33] Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as alternative to anchor points when learning with noisy labels. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12912- 12923. PMLR, 18-24 Jul 2021. URL https
//proceedings.nit mlr press/ v139/ v139/zhu21e zhu21e. html
[34] Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training a model to predict. In International Conference on Machine Learning, pages 27412-27427. PMLR, 2022.
[35] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 23133.... PMLR, 10-15 Jul 2018. URL httptttt///ccceddiiiii
mlr ptress/v80/jianee
html
[36] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, 18, page 8536-8546, Red Hook, NY, USA, 2018. Curran Associates Inc.
[37] Junnan Li. Yongkang Wong. Qi Zhao, and Mohan S. Kankanhalli. Learning learn from noisy labeled data. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5046-5054, 2019, doi: 1011099CVPR2001..00199
[38] Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12): 11053-11011, May 2021. doi: 10. wtw.9///aai.35512.11.. URL httpps://///:o saajjsaaaaaa aaai org/
index. Dicex.ppp/AAAe wtwp/AAI///////aa// wwwaaiieeeeiaeee wtttaiclleeeeeeees...11
12

---

[39] Derek Chong, Jenny Hong, and Christopher Manning. Detecting label errors by using pre- trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Longuage Processing, pages 9074-9091, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htt.httppss//cclaaaooooo.ooorrrp emn mmnlp-main... main. 618
[40] Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 5601-5610, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781 51886606699
[41] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance label noise. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7597-7610. Curran Associates, Inc., 2020. URL. https: / httt:p:://prcceediigs.eeeeee iewpppeeeeeeiiiineeeeiiiirispepp neurips. cc/paper__eeeeeeeeee /
[42] Hao Cheng. Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependentt label noise: A sample sieve approach. In International Conference on Learning Representations, 2021.
[43] Golara Javadi, Samareh Samadi, Sharareh Bayat, Samira Sojoudi, Antonio Hurtado, Silvia Chang. Peter Black, Parvin Mousavi, and Purang Abolmaesumi. Characterizing the uncertainty of label noise in systematic ultrasound-guided prostate biopsy. In 2021 IEEE 18th International doi: 10. 10.1109/1S114811.
Symposium on Biomedical Imaging (ISBI), pages 424-244 2021. 2021 1933765.
[44] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? Kamalika Chaudhuri and Ruslan Machine Learning,
Salakhutdinov, editors. Proceedings of the 36th International Conference volume 97 of Proceedings of Muchine Learning Research, pages 7164-7173. PMLR, 09-15 Jun 2019. URL htRphhtts:///pooeeeeee
mlr wall..press//e
peress/v97/yee vv7 9ttp//99///1//bb/ html
[45] Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-NN for noisy labels. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume I 119 of Proceedings of Machine Learning Research, pages 540-550. PMLR, 13-18 Jul 2020. URL https: https:://proceedigg mlr. press/vi 1ee1119/ahhhh htt//bahr222e hhtm
[46] Michael Desmond, Catherine Finegan-Dollate Jeff Boston, and Matt Arnold. Label noise in context. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics: System Demonstrations, pages 157-186, Online, July 2020, Associa- tion for Computational Linguistics. doi: 12186653///20200clleleme //aclanthology.. org/ 2220/2200002 acl-
URL https
[47] Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 31 (1), Feb. 2017, doi: 10. v3..
URL htttps///jss..... aaai: .rgg/ index. php/
AAAI/ htll/article/vat wit./////www//eeee 10894
[48] Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6234-6243. PMLR, 09-15 Jun 2019. URL https:: htttpp://proceeding.. mlr. pww.s/eeeseee... v97 tt7/thhllasidasann html
[49] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations 2022. URL
id=TBBAA6LLJI(Qm
[50] Johnson Kuan and Jonas Mueller. agnssicgi label quality scoring to detect real-world label errors. In ICML DataPerf Workshop, 2022.
13

---

[51] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 2691-26999 2015. doi: 10. 20155.2288......
[52]
George Shih, Carol C. Wu, Safwan S. Halabi, Mare D. Kohli, Luciano M. Prevedello, Tessa S. Cook, Arjum Sharma, Judith K. Amorosa, Veronica Arteaga, Maya Galperin-Aizenlon Ritu R. Gill, Mymn C.B. Godoy, Stephen Hobbs, Jean Jeudy, Archana Laroia, Palmi N. Shah, Dharshan Vummidi, Kavitha Yaddanapudi, and Anouk Stein. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1(1) 2019. doi: 10.1 10. 2411148ryai.2001 20191ii00110000000 URL. httpss
/ //doi. 1 1148 //ar//ra...// 2019180041
PMID: 33937785.
[53] Andrew I.. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142- 150, Portland, Oregon, USA, June 1. Association for Computational Linguisties. URL
http::/www..cch www.ww..wwwbb....... www.wwwwwwww.eeeeeeeeee.eooeeeee otgg/an/hologg//11100000 1atplllyyyyyy1111111101
[54] Francesco Barbieri, Jose Camacho-Collados Luis Espinosa Anke, and Leonardo Neves. Tweet- Eval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online, November 2020. Association for Computational Linguistics. doi: 10. /0/886633/1122000 htt.://5ww//0000...dinnn-mmmmmmo www.iinsneennl........ 148. URL httppp://acllaanooooggooott
findings-
148
[55] Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson, and Gianluca Bontempi. Calibrating probability with undersampling for unbalanced classification. In 2015 IEEE Symposium Series on Computational Intelligence, pages 159-166, 2015. doi: 10. 109/SSCC.2015..33..
[56] Chang Yue and Niraj K. Jha. Ctrl: Clustering training losses for label error detection, 2022.
[57] Zahra Salekshahrezaee, Joffrey Leevy, and Taghi Khoshgoftaar. A reconstruction error- based framework for label noise detection. Journal of Big Data, 8, 04 2021. doi: 10.118 00.1186/440537-02-004477777
[58] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL hRtp:/:///arcei ics. mici.edu/ml
[59] Murat Koklu and liker Ali Ozkan. Multiclass classification of dry beans using computer vision and machine learning techniques. Computers and Electronics in Agriculture, 174: 105507, 2020. ISSN 1699. doi:
1055000
105507. URL
[60] Marko Bohanec and Vladislav Rajkovi. Knowledge acquisition and explanation for multi- attribute decision making. In 8th Intl Workshop on Expert Systems and their Applications, 1988.
[61] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' Red Hook, NY, USA, 2020. Curran Associates Inc ISBN 9781713829546.
[62] J.S. Schlimmer. Concept Acquisition Through Representational Adjustment (Technical Report 87-19). PhD thesis, Department of Information and Computer Science, University of California, Irvine, 1987.
[63] Lauren Kirchner Jeff Larson, Surya Mattu and Julia Angwin. How we analyzed the compas recidivism algorithm, 05 2016.
[64] Chang Wei Tan, Geoffrey I. Webb, and Francois Petitjean. Indexing and classifying gigabytes of time series under time warping. SIAM International Conference on Data Mining, pages 1-10, 2017.
14

---

[65] Jason Lines, Anthony Bagnall, Patrick Caiger-Smith, and Simon Anderson. Classification of household devices by electricity usage profiles. In Hujum Yin, Wenjia Wang, and Victor Rayward-Smice editors, Intelligent Data Engineering and Automated Learning IDEAL 2011, pages 403-412, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-644-23877- 9.
[66] George B. Moody and Roger Mark. The impact of the MIT-BIH arrhythmia datahase. IEEE Engineering in Medicine and Biology Magazine, 20:45-50, 2001.
[67] Alimoglu and E. Alpaydin. Combining multiple representations and classifiers for pen- based handwritten digit recognition. In Proceedings of the Fourth International Conference on Document Analysis and Recognition, volume 2, pages 637-640 vol.2, 1997. doi: 10.1 109/
ICDAR. DIDAR.1997.65 6990558.......
[68] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018, 2018.
[69] Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization, 2023.
[70] Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Com- putational Linguistics, pages 1218-1227, Huhhot, China, August 2021. Chinese Information Processing Society of China. URL h org/ wwt...ccc.........
[71] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter, 2020.
[72] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 978171 1771293899
[73] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. International Conference on Learning Representations, 2023.
[74] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Shun Chen. LSTM fully con- volutional networks for time series classification. IEEE Access, 6:1662-1669, 2018. doi: 10.1 C0.100/AACEESS.217..7799
[75] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings, 2020.
[76] Görkem Algan and Ilka Ulusoy. Label noise types and their effects on deep learning. arXiv preprint arXiv:2003. 10471, 2020.
[77] Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. International conference on machine learning, pages 1804-4815. PMLR, 2020.
[78] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpps://ppenrrevieet net wet.://ErrruiidddgggPP......
[79] Nicholas Gisolfi. CedellCCnttric Verification of Artificial Intelligence. PhD thesis, Carnegie
Mellon University, 2021.
[80] Janez Demăar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
15

---

[81] Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoe tests based on mean-ranks? The Journal of Machine Learning Research, 17(1):152-661, 2016.
[82] Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Kcogh. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data mining and knowledge discovery, 31:606-660, 2017.
[83] Michael A. Hedderich, Zhu, and Dietrich Klakow, Analysing the noise model error for realistic noisy label data. In AAAI Conference on Artificial Intelligence, 2021.
[84] Dawei Zhu, Michael A. Hedderich, Fangzhou Zhai, David Adelani, and Dietrich Klakow. Is BERT robust to label noise? a study on learning with noisy labels in text classification. In Proceedings of the Third Workshop Insights from Negative Results in NLP, pages 62-67, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 insights- URL htt.ittt///cclaaaooooooooooo org/ htw.://.......222............... insiinnsssisii
[85] Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 322-330, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10. 1109/CCCV.019.00044 htRpsttp::///ot weeeccoppteraaccietoo....o org 11..0..19/ICC
[86] Varun Gulshan, Lily Peng, Mare Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. Devel- opment and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA, 316(2) 12 2016. ISSN 0098-7484. doi: 10. 00.1001/ama.2206.111 17211 URL httph:tt///tt http:::/dii..rrr/0000000//////// 2016 2016. 2016.17216
[87] Jieyu Zhang, Cheng- Yu Hsich, Yue Yu, Chao Zhang, and Alexander Ratner. A survey programmatic weak supervision, 2022.
[88] Mononito Goswami, Benedikt Boecking, and Artur Dubrawski. Weak supervision for affordable modeling of clectrocardiogram data, 2022.
[89] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: review. Data mining and knowledge discovery, 33(4):917-963, 2019.
[90] Milton Friedman. comparison of alternative tests significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86-92, 1940. ISSN 00034851. URL
http://www. jstorr wtt.o..org/ttalleeee..o.o. attoooorr///aaabe2255/7777
[91] Frank Wilcoxon. Individual comparisons by ranking methods, Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987. URL http:://tt::: www.:.sstrrooooo stble//0091/
[92] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979. ISSN 03036898, 14679469. URL http::/ :ttp::///ww..../ jstor org/ stable/4615733
16


Reviews:
[{"review_id": "pf04AOTkGa", "reviewer": "Reviewer_iCgr", "summary": "", "questions": "", "limitations": "To their credit, the authors do run a large number of experiments. However, there is no analysis or discussion of which experiments can be grouped together to increase their statistical power and which can not. Given that the conclusions about which cleaner performs the best changes depending on whether only image or only text data is considered, does an overall comparison make sense? The samples for the comparison are each dataset+architecture run. In which case, removing some text datasets and adding some new image datasets and architectures would change the distribution of the modalities in the overall data and hence the final conclusions?\n\nThe authors use three image datasets which have multiple annotators, but do not make use of the multiple annotations. This limitation is acknowledged in the paper, and the authors say it is left as future work. Confusingly though, the appendix (L957 etc) refers to methods that were not used in the main paper, nor the appendix.\n> for datasets with labels from multiple annotators, we implement dissenting label, dissenting worker, and crowd majority...", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors perform experiments with a sufficiently large number of datasets, and across a range of noise injection methods. They do not just measure the ability of the cleaning methods to identify samples with noisy labels, but also measure the impact of the cleaning methods on the performance of downstream models trained from scratch on cleaned versions of the datasets.\n\nBy combining the results from two noise levels and two architectures, the main results tables show results with n=4 (though these are not like-for-like repetitions), which is acceptable", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "WIsidZCIjb", "reviewer": "Reviewer_Q72y", "summary": "", "questions": "", "limitations": "I am not familiar with this domain, but I think it could be better if the author discuss more about the label noise in other settings such as multimodal and regression.", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. **Comprehensive Problem Addressing:** The paper tackles the significant issue of labeling errors in widely used ML datasets, emphasizing their impact on model generalization and evaluation.\n\n2. **Novel Benchmark Framework:** The introduction of the Automated Label Quality Assessment (AQuA) benchmark provides a valuable contribution to the field, enabling rigorous evaluation of methods dealing with label noise.\n\n3. **Practical Relevance:** The paper's practical implications are noteworthy, as it empowers ML practitioners to select appropriate label cleaning tools for improving model performance when faced with mislabeled data.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "IG2PCqcROn", "reviewer": "Reviewer_AcXH", "summary": "", "questions": "", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Unified Evaluation. this benchmark provides a standardized and objective evaluation platform for various label error detection models.\n\nThe motivation of this paper is very clear.  The paper addresses the prevalent issue of erroneous labels in widely-used datasets, which has significant implications for model generalization and evaluation. By focusing on real-world challenges, the paper's contributions have practical relevance for ML practitioners and researchers.\n\nI appreciate the effort of the authors for adding a section of Design Space for Label Error Detection Models. The introduction of a design space allows for concrete delineation of design choices for label error detection models. This framework facilitates the understanding and comparison of various approaches, aiding in the selection of appropriate methods to improve label quality.\n\nThe benchmark contains datasets from multiple domains and includes baselines from recent works. The wide range of datasets and state-of-the-art baselines make this benchmark a valuable resource for researchers and practitioners in the field.\n\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Infa92ydSF", "reviewer": "Reviewer_RN59", "summary": "", "questions": "", "limitations": "The authors have discussed the limitations of their work and have provided recommendations for future work. They have also discussed the potential negative societal impacts of their work and have ensured that their paper conforms to ethics review guidelines. However, there are some areas where the authors could provide more detailed recommendations for mitigating the risks associated with their work. \n\nFor example, the authors could discuss how to ensure that label error detection models are not used to perpetuate existing biases in the data. They could also discuss how to ensure that the models are not used to discriminate against certain groups of people. Additionally, the authors could provide more detailed recommendations for ensuring that the benchmark datasets are representative of the broader population and are not biased in any way. \n\nOverall, the authors have done a good job of addressing the limitations and potential negative societal impact of their work. However, there is always room for improvement, and the authors could provide more detailed recommendations for mitigating the risks associated with their work.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The submission has several strengths. Firstly, AQuA is a significant contribution to the research community as it provides a standardized evaluation framework, benchmark datasets, and evaluation metrics for assessing the performance of label error detection models. This will enable researchers to compare the performance of different models and identify the most effective methods for detecting and correcting labeling errors in their data. \n\nSecondly, the proposed design space and benchmark design are well thought out and provide a comprehensive approach to evaluating label error detection models. The authors have conducted a large-scale experiment (>1000 unique experiments) to demonstrate the efficacy of AQuA in benchmarking machine learning models in the presence of label noise. \n\nThirdly, the paper discusses the limitations of the proposed approach and provides recommendations for future work. This demonstrates the authors' commitment to improving the quality of research in this area and ensuring that the proposed approach is continually refined and improved. \n\nFinally, the authors have discussed the potential negative societal impacts of their work and have ensured that their paper conforms to ethics review guidelines. This demonstrates their commitment to ethical and responsible research practices. \n\nOverall, the submission is of high quality and has significant relevance to the broader research community. The proposed approach has the potential to improve the accuracy and reliability of machine learning models, which has important implications for a wide range of applications, including healthcare, finance, and transportation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper's scores are 7-7-6-6, where reviewers are generally positive. The paper introduces the Automated Label Quality Assessment (AQuA) benchmark, a tool designed for evaluating label quality in machine learning datasets. Reviewers acknowledge that the benchmark addresses the issue of label errors and their impact on model generalization and offers a standard framework for assessing label error detection methods. Reviewers have generally expressed positive opinions about the paper's contributions, practical relevance, and the clear motivation behind the research. During rebuttal, authors did a good job in addressing reviewer questions such as hyperparameter tuning, the use of stronger baseline models, etc. Considering all these aspects, AC recommends to accept this paper.
Decision: Accept (Poster)