SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mononito Goswami*, Vedant Sanil*, Arjun Choudhry†, Arvind Srinivasan†, Chalisa Udompanyawit, Artur Dubrawski
Auton Lab, School of Computer Science
{mgoswami, vsanil, arjuncho, arvindsr, cudompan, awd}@cs.cmu.edu
www.github.com/autonlab/aqua
Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. ImageNet, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark to evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance in the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation of machine learning tools facing mistrabeled data.
A lot of machine learning (ML) research is devoted to making efficient and effective use of available data to learn accurate, high-fidelity, and interpretable models, with little to no focus on the quality of the data they are trained and evaluated on. Nonetheless, it is widely recognized that ML models are only as good as the data they rely on, i.e., the quality of data imposes practical limits to what ML models can achieve. Not only are datasets used to train ML models; they also serve as benchmarks to measure the state-of-the-art and validate theoretical findings. Thus, high quality large labeled datasets are the cornerstone of progress in supervised machine learning. However, the data is rarely free of noise, which can both manifest in the features of the data (feature noise) and in labels that categorize them (label noise). Between feature and label noise, the former has been found to be much more harmful to machine learning models [1, 2, 3]. To make matters worse, label noise is prevalent in popular ML benchmarks. A recent study estimated an average of at least  $3.3\%$  label errors across 10 datasets commonly used for benchmarking computer vision, natural language, and audio classification algorithms [4]. Consequently, a growing body of research is devoted to understanding the harms of label noise and to developing techniques to identify and mitigate labeling errors.
*MG and VS contributed equally. MG is the corresponding author.
$\dagger$  AC and AS have equal contribution.
Figure 1: Overview of the AQuA benchmark framework. AQuA comprises of datasets from 4 modalities, 4 single-label and 3 multi-annotator label noise injection methods, 4 state-of-the-art label error detection models, classification models, and several evaluation metrics beyond metrics of predictive accuracy. We are in the process of integrating several fairness, generalization, and robustness metrics into AQuA. The red and blue arrows show two example experimental pipelines for image data and time-series data, respectively.
In recent years, over 50 papers have been written on this topic, including 6 surveys, yet the literature lacks a comprehensive benchmark to evaluate the available methods. The evaluation of existing methods is lacking along the following dimensions:
Arbitrary choice of datasets and limited data modalities. To the best of our knowledge, relevant studies have used over 40 datasets (e.g., ImageNet [5]) and their variations (e.g., Imagenette [6], ImageNet-100 [7]) for evaluation, but mostly on computer vision related tasks, with less than 15 studies using text data, 7 using tabular data and only 1 paper using time-series data.
Arbitrary choice of classification models. The ultimate goal of identifying labeling errors is to learn a classification model using training data with clean labels. Much like the datasets, relevant studies have used over 47 different classification architectures (e.g., ResNet [8], MobileNet [9], ResNeXt [10], BERT [11], XLM-RoBERTa [12], etc) to measure the impact of label cleaning.
Inconsistent evaluation protocols and metrics. Different studies conduct different experiments to measure the efficacy of their proposed methods (e.g., the accuracy of the label cleaning method, or performance of the downstream model before and after label cleaning, etc.) and use various measures of success (e.g., high accuracy,  $F_{1}$ -score, or low error rate).
With such diversity and inconsistency in the way in which these methods are evaluated, it is hard to measure the state of the art. To bridge this gap, we propose the Annotation Quality Assessment, AQuA, the first benchmark framework to evaluate machine learning methods in the presence of label noise (Fig. 1). We also elucidate the design space for such models, with the hope that it will not only foster future research on detecting labeling errors, but also enable ML practitioners to choose the appropriate label cleaning tools for their specific data and tasks. We run a large-scale experiment (> 1000 unique experiments) and make several interesting observations, demonstrating AQuA's efficacy in benchmarking machine learning models in the presence of label noise.
2 Background and Problem Formulation
Sources of labeling errors. Labeling errors can arise from automated labeling processes such as crowd-sourcing [13], programmatic weak supervision [14, 15], and human error (e.g., due to lack of expertise or low confidence in expert assessment) [16]. Errors may also stem from idiosyncrasies
3The root cause of labeling errors in crowd-sourcing is different from human expert annotation. For instance, errors during crowd-sourcing have been shown to arise from other factors such as gaming the system to maximize monetary gains [13].
of the annotation procedure and the corresponding guidelines themselves [17]. Finally, existing labels may also become inconsistent with prevailing knowledge due to constantly evolving problem definitions and domain knowledge leading to concept drift.
Impact of labeling errors. At training time, labeling errors can cripple an ML model's ability to generalize and introduce undesirable biases in its hypothesis space [19, 20]. Mislabeled training data is especially problematic for overparameterized deep neural networks, which can achieve zero training error even on randomly-assigned labels [20]. At test time, labeling errors can lead to noisy model evaluations and invalidate common model selection strategies. In safety-critical settings, models trained, evaluated, and selected using mislabeled data can be
cat frog
knitwear dress
normal arrhythmia
not offensive offensive
Figure 2: Labeling errors in widely used benchmarks: CIFAR-10, Clothing-100K, MIT-BIH, and TweetEval Hate Speech datasets. Observed labels are in red and true labels are in green.
ineffective at best and can lead to disastrous outcomes at worst. Finally, recent studies in the context of fairness have shown that naively enforcing parity constraints based on noisy labels can harm groups that are unaffected by label noise [21, 22].
Problem formulation. Due to the far-reaching consequences that labeling errors can have on model training and evaluation, the literature has attacked multiple different but related problems, for example: (1) label error detection, identify which data points have erroneous labels [23, 24], (2) label noise estimation, estimate the proportion of data with noisy labels [25], (3) label noise robust learning, learn models robust to label noise [26, 27], and (4) noise transition matrix estimation, estimate the parameters of the noisy label generation process [28].
In this work, we focus on the label error detection problem, because (a) it is the most general of the above problem types, i.e., with knowledge of labeling errors, we can estimate the noise rate, parameters of the noise generation process and train ML models free from label noise, (b) it provides practitioners greater visibility of issues that plague their data, and (c) allows them to directly rectify these errors.
Label error detection problem: Assume a dataset  $\mathcal{D}^* = \{(\mathbf{x}_i, y_i^*)\}_{i=1}^N \in (\mathcal{X}, \mathcal{Y})$ , where  $\mathbf{x}_i$  and  $y_i^*$  denote the features and labels, respectively. In practice, we do not have access to  $\mathcal{D}^*$ , but instead observe a noisy dataset  $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N \in (\mathcal{X}, \mathcal{Y})^{[a]}$ . We call  $y_i$  a labeling error if  $y_i \neq y_i^*$ , and correctly labeled, otherwise. Our goal is to identify all labeling errors in  $\mathcal{D}$ .
<sup>a</sup>We assume that we observe the true features since we are interested in identifying labeling errors and isolating their impact on downstream model performance.
${}^{b}$  A note on terminology: In this paper,we will sometimes refer to labeling errors as noisy labels or label noise, and the process of identifying them as label error detection, or loosely as label cleaning.
3 A Design Space of Labeling Error Detection Models
In this section, we seek to align the dimensions along which label error detection models vary, with dimensions that can facilitate model selection for ML practitioners. We provide a brief overview of these dimensions below and defer detailed discussions to Appendix A.1
What inputs do you have? All label error detection models take features and noisy labels as input. In most datasets, data points are labeled by multiple experts, but their individual annotations are seldom available. When available, multi-annotator labels can be used to identify data points that are inherently ambiguous [29], or to model individual annotators to estimate their expertise and propensity for mislabeling examples [7], and using these to identify likely labeling errors. While most methods identify labeling errors and automatically remove or correct them, a few rely on a human
For example, sepsis is one of the most sought-after clinical conditions to predict. However, with the constantly evolving definition of sepsis, the labeling process is frequently affected, causing many annotations in legacy benchmark data to become inconsistent with the latest guidelines [18], a very dangerous risk to take in the particular type of application area
expert who can be queried to relabel suspicious data points [29, 30]. Some other methods assume access to data points called anchor points, which most certainly belong to a particular class [31, 32]. The number of anchor points required is generally proportional to the number of classes, and quickly becomes prohibitive for multi-class classification problems, and in more complicated noise settings [33]. Finally, a vast majority of methods assume access to classification models, and primarily differ in their number (model-free [34], one or multiple models [35, 36, 37, 38]), nature of access (prediction-only [23] versus access to logits [24], gradients [30] etc.), and extent of pre-training (no pre-training [23, 24] versus large-scale pre-training e.g. large language models [39]).
What modeling assumptions can you make?
Different studies use different assumptions on data (noise structure and clusterability), heuristics (model self-confidence and perceptual uncertainty), and modeling decisions (whether to explicitly model the transition matrix and multi-network training). Most studies in the literature explicitly assume some form of structure in the noise present in the data [23, 34, 40, 28]. Most early studies assumed class-dependent noise, i.e., the likelihood of error is only dependent on the latent true class, not on the data [23, 28]. There is growing interest in more realistic forms of noise where the probability of error also depends on the features of a data point (instance-dependent noise) [41, 42]. To this end, some recent studies have shown promising results by leveraging natural notions of similarity between data points and their labels. For example, Zhu et al. [34] assume that examples with similar features should have similar labels.
Figure 3: Design space of labeling error detection models to delineate concrete design choices.
Many studies treat a trained model's low confidence that a data point belongs to its observed label as a heuristic likelihood to identify labeling errors [24, 23, 43]. In a similar vein, a recent study used the loss of a pre-trained large language model on each data point to identify unlabeled examples [39]. When multi-annotator labels are available, as discussed before, some studies have also used them to model the perceptual uncertainty in the annotators to identify labeling errors.
Finally, studies differ in their modeling decisions. While some explicitly estimate a data structure called the noise transition matrix, which encodes the joint probability of latent true and observed noisy labels [23, 33, 27], others do not [24, 30, 14]. Finally, there is a body of work on label noise robust learning using multiple model instances either using knowledge distillation [35, 36, 44] or meta-learning [38, 37]. The key idea is to use a cooperative game between models to identify labeling errors and ensure that the eventually deployed model only learns from clean data.
What outputs do you want, and what would you do with them? All labeling error detection models identify data points that are likely to be labeling errors. With knowledge of the potentially mislabeled data points, most studies simply remove them from consideration [23, 24, 45, 42]. This strategy may be practical for large datasets, where only a small fraction of data is found to be mislabeled and domain experts are unavailable for supervision. We use this strategy by default in AQuA. A smaller number of methods predict the alternate class that the data point is most likely to belong to [38, 34] and even provide explanations for their predictions [30, 46]. CINCER [30] is one of the few methods which not only finds labeling errors but also identifies counter-examples in the training data to serve as explanations for its suspicion. Some studies use the label predicted by these models and perform loss re-weighting or correction to learn robust classification models [27, 47, 48]. When domain experts are available, some studies also leverage their insight to re-label mislabeled data point [30, 29].
Modality	Dataset	# Train / Test	# Annotators/sample	Label Source	Classification Task	Sample Size	Usage
Image	CIFAR-10N [49]	50K / 10K	3	Human annotation	Object	32 × 32 × 3	50.134
CIFAR-10H [16]	0 / 10K	47–63	Human annotation	Object	32 × 32 × 3	29
Clothing100K [51]	100K	1	Web-labeled	Image	256 × 256 × 3	24.4134
NoisyCXR [52]	26K / 3K	1–XX	Human expert annotation	Pneumonia	1024 × 1024 × 1	29
Text	IMDbβ [53]	25K / 25K	1	Human annotation	Sentiment	-	27.474
TweetEval [54]	10K	1	Human annotation	Hate speech	-	-
Tabular	Credit Card Fraudβ [55]	284K	1	Human annotation	Credit card fraud	28	56.57
Adultβ [58]	48K	1	Rule-based extraction	Salary	14	30.2122
Dry Bean [59]	13K	1	Vision system-based annotation	Bean variety	17	-
Car Evaluation [60]	1K	1	Hierarchical decision model [60]	Car condition	6	61
Mushroomβ [62]	8K	1	-	Mushroom edibility	22	56
COMPASβ [63]	6K	1	-	Recidivism	28	21
Time Series	Crop [61]	7K / 16K	1	Hierarchical k-means tree with dynamic time warping [64]	Crop cover	46 × 1	-
ElectricDevices [65]	9K / 7K	1	Human annotation	Appliance-type	96 × 1	-
MIT-BIH [66]	23K / 4K	1	Human expert annotation	Arrhythmia	256 × 2	-
PenDigits [67]	7K / 3K	1	Human annotations	Handwritten digit	16 × 1	-
WhaleCalls [68]	11K / 2K	1	-	Whale call	4,000 × 1	-
Table 1: Summary of datasets. AQuA currently includes a variety of datasets for different classification problems, varying in the number of classes, sources of annotations, and data modalities. All datasets except those marked with  $\beta$  are multi-class.
4 Benchmark Design
4.1 Real-world, Popular Datasets, and Downstream Classification Models
Datasets. AQuA currently comprises of a collection of 17 popular real-world public datasets from 4 prevalent data modalities: image, text, time-series and tabular. To evaluate label error detection models across various practical scenarios, we carefully choose datasets with diversity in the following characteristics: (1) classification problems (e.g., sentiment classification vs. hate speech detection), (2) number of classes (binary vs multi-class classification), (3) relative prevalence of classes (e.g., skewed datasets like Credit Card Fraud [55] and balanced ones like IMDb [53]), (4) sources of annotations (e.g., human vs rule-based annotation), and (5) number of annotations per example (e.g., CIFAR-10N labeled by 3 annotators). Table summarizes the key characteristics of datasets included as a part of AQuA. In particular, to make comparison with prior work easier while maintaining diversity across practical scenarios, we try to include datasets that have been used frequently by prior work (see usage in Table) and preprocess them in a manner consistent with those works. We do not use any data augmentation during training. App [A.3] provides detailed descriptions of the datasets.
Classification models. The ultimate goal of label cleaning is to train accurate downstream classifiers, but different studies use different classification models to measure the efficacy of their proposed label cleaning methods. To provide a level playing field for all cleaning methods, we include multiple classification model architectures for each data modality. Specifically, we include ResNet-18 [8], MobileNet [9] and FastViT-T8 [69] for image datasets, a11-distilroberta-v1 [70, 71] and a11-MiniLM-L6-v2 [72] for text datasets, ResNet-1D, PatchTST [73] and LSTM Fully Convolutional Network [74] for time-series datasets, and TabTransformer [75] and a Multi-Layer Perceptron for tabular datasets. While choosing classification models we prioritized performant methods with (1) different architectures and inductive biases, (2) ideally pre-trained using different strategies, and (3) previously-used either by label cleaning methods or task-relevant papers. App. A.4 and App. A.5 provide a detailed description of classification models and their hyperparameters, respectively.
4.2 Advanced Label Error Detection Methods
AQuA provides easy-to-use Application Programmer Interfaces (Fig. 4) for 4 state-of-the-art label error detection methods, namely Area Under Margin ranking (AUM) [24], Confident Learning [23], Contrastive and Influent Counter Example Strategy (CINCER) [30], and Model-free Label Error Detection (SimiFeat) [34]. Below, we provide a brief overview of these methods and their key ideas.
Area Under the Margin Ranking (AUM) [24]. Given noisy data and access to the logits of a deep learning model, AUM exploits differences in training dynamics of clean and mislabeled samples to identify labeling errors. The key idea is to identify data points that do not contribute to the generalization of a model as labeling errors by leveraging the delicate tension between the label of a data point (via memorization) and its predicted label (via gradient updates), measured as the margin between the logits of a sample's assigned class and its highest unassigned class.
Confident Learning (CON) [23]. Given noisy data, confident learning estimates a data structure called confident joint, which is the joint probability distribution of observed noisy and latent true labels. The key idea is to leverage a model trained on held-out data drawn from the same (or similar) distribution to predict the probability that an example  $\mathbf{x}_i$  belongs to its observed label  $\mathbf{y}_i$ . A low probability is then used as a heuristic-likelihood of  $\mathbf{y}_i$  being a label error. The confident joint can then be used to identify labeling errors and estimate the noise rate.
Contrastive and Influent Counter Example Strategy (CINCER or CIN) [30]. CINCER
treats the problem of identifying labeling errors as a sequential decision making problem where a domain expert can be queried to relabel suspicious examples. CINCER uses the same heuris
tic as AUM to identify labeling errors, but also identifies counter-examples in the data to serve as explanations of the model's suspicion.
Model-free Label Error Detection (SimiFeat) [34]. Unlike other methods, SimiFeat does not need a (pre-)trained model to identify labeling errors. Instead, it utilizes labels of the  $k$  nearest neighbors to identify labeling errors based on the clusterability assumption, i.e. data points with similar features should have the same true label with high probability.
There are many methods to detect labeling errors, but we choose these methods as a starting point because they are recent, state-of-the-art, and have different inputs and core assumptions. While all these methods have existing public implementations, through AQuA, our goal is to create a one-stop shop for using and evaluating open-source label error detection models.
from aqua.models import TrainAqModel, ConvNet  
from aqua.data import Aqdata, load_cifar  
from aqua.report import generate_report
Load CIFAR-10 and ResNet-18  
clf = ConvNet('resnet18')  
data = load_cifar()  
data.add_noise(noise_rate=0.2) # Add uniform
InstANTIate a cleaning method and classifier cleaner  $=$  TrainAqModel(clf,method  $\equiv$  'aum') label errors  $=$  cleaner.find_label_issues(data)
Remove data with label issues  
data.clean_data.label Issues)  
# Train a downstream model on cleaned data  
y_preds = TrainAqModel(clf).fit.predict(data)
Figure 4: AQuA makes identifying label issues, and evaluating new and existing label error detection models simple.
4.3 Evaluation
There is significant variance in the ways that label cleaning methods are evaluated. To rigorously, fairly, and systematically assess these models, we unify the breadth of experimental settings through the following three dimensions of evaluation.
Supervision. Identifying labeling errors in practice is an unsupervised problem since we do not know which data points are mislabeled. Hence, evaluating these methods is a challenging endeavor. Most studies in the literature gather noise labels either from human experts (human-in-the-loop evaluation) or by introducing synthetic label noise by design (synthetic label noise).
In human-in-the-loop evaluation, one or more human experts are asked to independently assess the true labels of data points identified as having erroneous labels [39, 4]. While this is a straightforward and precise evaluation method, it is in general unscalable, expensive, time-consuming, and limited to only measuring the precision of models (and not recall), because the experts are typically only shown data points which a model considers erroneous.
A much more common and scalable way of evaluating these methods is to introduce various kinds of synthetic label noise and measure a model's ability to detect them. There are many ways of introducing label noise, but injected noise may not always be reflective of the true noise that occurs in natural datasets, and hence identifying realistic noise injection strategies is an active area of research [33, 34, 39, 41, 76, 77]. Moreover, model evaluation may still be noisy because there may be mislabeled examples for which our pseudo-noise labels are negative (or correctly labeled).