SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hypotheses. In general, existing studies evaluate two hypotheses: (1) cleaning labels on the train set improves the performance of the downstream classifier on the test set, and (2) cleaning methods can accurately identify unlabeled data on the train set. Hypothesis 1 is practical since the primary goal of identifying labeling errors is to train accurate and unbiased classifiers. However, appropriately regularized deep learning models are known to be naturally robust to some label noise. Hence, hypothesis 2 allows researchers to directly measure the efficacy of label cleaning techniques.
Measures of goodness. Different studies use different measures of predictive accuracy. While some measure error rate [24], others report the accuracy [33] or ROC-AUC [29] of their classification models. Similarly, for their cleaning methods, some studies report the  $F_{1}$  score while others report the precision or recall [23, 24].
More gaps in evaluation. In addition to the lack of consistency, we believe that the experimental settings in many studies are occasionally (1) unrealistic, e.g., adding label noise to more than half (sometimes up to  $80\%$ ) of the data points [24, 23]; and (2) uni-dimensional, e.g., reporting only one metric of predictive performance.
AQuA's design. To enable a realistic, multi-faceted and holistic evaluation of label error detection models, we implement 7 popular label noise injection techniques and multiple metrics of predictive performance. Specifically, for single-label datasets, we implement asymmetric [34], class-dependent [76], instance-dependent [33], and uniform [76] noise, and for datasets with labels from multiple annotators, we implement dissenting label, dissenting worker, and crowd majority [39]. In terms of metrics of predictive accuracy, we implement  $F_{1}$ , accuracy, (weighted) precision, recall, area under ROC curve (ROC-AUC), average precision (PR-AUC), and error rate. We are in the process of implementing some other metrics beyond predictive accuracy, such as generalization [78] and robustness [79] of models. Our hope is that AQuA's config-driven design will allow non-technical users to integrate it into their labeling workflows and researchers to add new models, datasets, and evaluation pipelines seamlessly. Our choice of datasets and downstream classifiers ensures that the computational complexity of running experiments is not prohibitive. Finally, we make all code, pre-trained models, and experimental logs open-source to enable rigorous and fair evaluation of models.
5 Experiments, Results and Discussion
Datasets	Uniform	Asymmetric	Class-dependent	Instance-dependent
AUM	CIN	CON	SIM	AUM	CIN	CON	SIM	AUM	CIN	CON	SIM	AUM	CIN	CON	SIM
CIFAR-10	73.3	74.1	45.6	76.7	74.3	70.8	47.7	75.5	93.5	80.5	42.6	93.6	68.0	69.9	44.8	70.9
Clothing-100K	75.0	70.0	76.6	76.5	74.2	68.4	73.6	75.7	76.3	71.2	74.0	81.2	69.4	65.1	72.9	71.6
NoisyCXR	75.2	74.4	43.2	74.5	73.7	71.5	39.5	73.5	84.7	78.7	31.4	88.4	68.0	69.8	43.3	72.1
IMDb	75.6	73.3	58.4	78.5	75.7	74.3	59.5	78.7	92.1	91.0	62.8	95.0	69.7	70.2	56.4	74.5
TweetEval	75.3	75.2	58.9	77.7	75.8	76.0	57.4	77.6	69.2	67.9	52.4	70.2	69.6	69.6	62.4	73.2
Credit Fraud	75.8	75.8	73.3	78.1	75.7	75.8	80.0	76.7	63.3	63.0	87.2	72.0	69.5	69.4	74.3	73.5
Adult	75.7	75.8	72.9	78.5	75.8	75.8	66.9	77.5	63.6	64.6	61.2	64.9	69.6	70.2	68.7	72.4
Dry Bean	75.7	91.6	42.1	82.2	75.7	84.9	39.0	80.3	87.2	95.0	35.4	92.1	69.5	83.1	35.8	77.5
Car Evaluation	75.3	83.5	77.4	84.1	75.6	80.2	75.7	81.6	77.3	87.5	83.2	81.2	70.1	78.8	78.5	77.0
Mushrooms	76.0	82.5	62.7	85.2	75.7	80.7	66.3	83.0	99.3	100	75.5	99.8	69.5	75.4	64.1	74.3
COMPAS	75.8	74.9	63.2	75.9	75.8	74.8	64.6	76.5	55.5	57.1	52.9	57.7	69.5	69.4	61.0	73.1
Crop	76.0	79.0	16.3	73.1	75.8	73.6	16.2	70.1	29.1	40.8	51.2	63.7	69.5	63.2	16.3	63.8
Electric Devices	75.8	82.2	35.0	79.3	75.7	78.6	35.3	75.8	37.8	50.5	55.9	68.3	69.9	71.5	32.7	69.2
MIT-BIH	75.6	88.4	49.7	83.3	75.7	83.0	51.3	78.4	68.2	75.7	45.4	80.6	69.6	78.4	48.1	75.2
PenDigits	75.8	89.0	23.1	73.4	75.7	83.1	23.4	72.7	46.7	44.9	53.5	78.4	69.9	76.0	19.8	68.1
WhaleCalls	75.6	74.9	60.3	77.3	75.7	75.5	61.8	77.2	42.3	44.7	52.4	47.1	69.6	69.1	59.2	71.2
Table 2: Performance evaluation of cleaning methods to detect erroneous labels across different types of synthetic noise added to the train set in terms of weighted  $F_{1}$ , averaged across noise rates and downstream models.
(i)
$(iv)$
(ii)
(v)
(vi)
Figure 5: Critical difference diagrams representing rankings of cleaning methods across: (i) all datasets, (iii) only image or (iv) only text datasets. (v) also shows the ranking of cleaning methods across all datasets when accuracy is measured instead of weighted  $F_{1}$  (c.f. i). Finally, (ii) represents the performance of downstream models trained using cleaned labels, and (vi) performance of all cleaning methods disaggregated by noise type.
Datasets	No Noise Injected	Uniform	Asymmetric	Class-dependent	Instance-dependent
NON	AUM	CIN	CON	SIM	NON	AUM	CIN	CON	SIM	NON	AUM	CIN	CON	SIM	NON	AUM	CIN	CON	SIM	NON	AUM	CIN	CON	SIM
CIFAR-10	74.3	74.1	73.0	46.0	73.5	63.2	62.6	65.0	36.9	63.4	58.1	63.1	62.2	38.3	63.1	71.0	71.7	70.5	46.2	67.5	57.7	60.2	62.1	34.0	56.8
Clothing-100K	90.9	90.7	90.5	90.8	90.8	82.5	79.5	83.2	85.4	83.2	82.8	81.4	79.2	79.4	83.1	80.8	83.3	82.1	83.8	86.1	78.9	74.6	71.7	81.6	81.6
NoisyCXR	56.0	56.5	56.7	25.2	57.0	49.6	49.4	52.2	19.1	48.2	49.6	48.8	50.6	18.0	47.9	54.2	54.8	55.8	18.6	53.8	46.4	46.7	48.3	18.9	46.7
IMDb	84.9	87.5	89.2	69.6	90.3	69.3	65.6	68.2	64.5	70.8	74.9	67.7	76.8	66.4	80.6	87.1	85.5	89.1	84.4	87.4	65.3	62.3	69.2	64.4	64.5
TweetEval	73.6	73.6	77.1	65.1	76.8	71.3	72.4	74.2	53.8	73.2	71.3	68.4	71.9	61.8	72.4	77.7	74.8	70.6	49.8	67.4	68.5	69.4	71.7	71.5	63.1
Credit Fraud	100	99.9	99.9	99.9	99.9	99.9	99.9	99.9	88.8	99.9	99.9	99.9	99.9	99.8	99.9	99.6	66.6	99.8	99.9	66.6	99.9	99.8	99.9	88.7	99.8
Adult	84.0	84.0	84.0	79.9	84.0	82.1	82.1	83.0	75.0	81.9	83.3	83.0	83.0	77.9	83.0	81.8	82.3	83.1	81.9	80.4	82.5	80.6	81.5	73.1	81.1
Dry Bean	91.9	90.9	91.2	57.2	90.6	89.5	90.9	91.4	60.6	78.6	85.6	87.0	89.3	55.1	87.4	91.4	91.0	90.5	29.8	90.4	87.3	83.6	84.5	40.0	87.5
Car Evaluation	93.4	92.5	85.9	57.6	92.0	83.8	82.5	80.0	66.5	81.1	86.1	85.2	74.1	62.7	82.6	86.9	83.2	78.0	57.6	83.9	82.6	81.5	75.3	60.2	80.5
Mushrooms	99.7	99.9	99.6	99.7	99.9	98.3	98.2	97.8	89.0	98.7	97.9	97.5	98.4	87.3	96.5	99.5	100	99.3	99.1	99.9	95.3	96.9	96.4	81.1	95.8
COMPAS	67.2	67.3	66.2	63.7	66.6	65.6	62.1	65.9	58.6	65.3	66.1	64.4	66.2	46.9	65.6	54.3	65.1	63.8	35.5	66.2	61.6	63.0	61.8	48.6	63.7
Crop	39.1	38.7	35.5	8.4	37.8	33.1	37.2	36.2	7.3	37.9	34.1	31.5	32.8	7.2	33.4	32.3	31.2	29.5	7.3	28.9	27.7	27.8	29.9	5.7	34.5
Electric Devices	45.3	48.0	48.0	29.8	46.7	41.8	42.6	44.8	27.3	42.1	42.5	41.3	41.3	26.9	42.7	30.9	30.9	32.1	24.2	31.7	39.3	36.6	38.3	23.1	40.4
MIT-BIH	72.7	65.1	81.2	55.7	72.5	73.2	70.1	80.1	61.7	74.7	71.3	68.4	69.2	46.3	69.6	72.6	73.9	74.4	56.9	78.0	63.6	68.1	70.9	52.2	71.5
PenDigits	64.8	65.2	64.3	39.5	64.5	62.6	64.7	64.4	24.6	64.3	58.1	59.1	57.8	22.9	59.0	43.9	46.5	46.4	15.3	45.3	59.2	56.4	57.7	14.8	59.7
WhaleCalls	68.2	34.3	50.9	52.7	53.0	48.7	44.5	51.0	43.7	50.4	48.8	53.6	47.4	45.3	47.2	42.5	43.3	47.1	41.6	42.4	48.5	50.9	58.5	44.5	47.5
Table 3: Impact of label noise and each cleaning method on weighted  $F_{1}$  score of a downstream model for each modality on the test set, averaged across noise rates and downstream models. Highlighted cells indicate better performance than that obtained without label cleaning (NON).
We conduct several experiments to support AQuA's design choices and demonstrate its utility in providing a comprehensive and holistic evaluation of machine learning models in the presence of label noise.
Experimental Setup and Hyper-Parameter Tuning. We run experiments for all combinations of cleaning methods (AUM (AUM), confident learning (CON), CINCR (CIN) and SimiFeat (SIM), including no label cleaning (NON), noise types (asymmetric, class-dependent, instance-dependent and uniform); for four different noise rates (0%, 2%, 10% and 40%), for a total of 2400 unique experiments. We conduct experiments using three distinct classification architectures for image and time-series data, and two different architectures for text and tabular data. To account for class imbalance in some datasets, we report the  $F_{1}$  weighted by the support of each class. Results for all other evaluation metrics can be found in App. A.8. We also adopt critical difference diagrams to succinctly represent comparisons between multiple cleaning methods and other independent variables (e.g., data modality and noise type) on multiple datasets. These diagrams represent the average ranks of methods across datasets while grouping those with insignificant difference. We tuned hyper-parameters of all the classification and cleaning methods till they performed reasonably well on average on all the datasets using hyper-parameter grids used by prior work and reported in App. A.5. Finally, all our experiments were carried out on a computing cluster, with a typical machine having 128 AMD EPYC 7502 CPUs, 503 GB of RAM, and 8 NVIDIA RTX A6000 GPUs.
Research Questions. We aim to answer the following research questions through our experiments:
- Which is the best cleaning method in terms of (i) its ability to identify synthetically injected label noise, and (ii) performance of the downstream classifier trained its cleaned labels?
- Do the rankings of cleaning methods differ across different (i) types of synthetic label noise, (ii) data modalities, and (iii) evaluation metrics (weighted  $F_{1}$  versus accuracy)?
5.1 Insights from Large-scale Experiments using AQuA
Tables 3.2 and Fig. 5 report results from all our experiments aggregated by noise rate, and downstream classification models. Below we highlight some of our key findings. Due to lack of space, we defer finer grained results to App. A.8.
Best cleaning method. Overall, we found SimiFeat (SIM) [34] to be the best cleaning method in terms of its ability to identify synthetically injected label noise, closely followed by CINCER (CIN) [30] (Fig. 5(i)). However, these differences shrink when evaluating cleaning methods using the performance of the downstream model trained using their cleaned labels (Fig. 5(ii)). Confident learning (CON) [23] consistently performed the worst among all the evaluated methods.
5To form cliques, we abandon the posthoc test in favor of pairwise tests with Holm's correction for multiple testing based on prior work [81, 82]
<sup>6</sup>We deliberately did not perform extensive hyper-parameter tuning to not overfit to already existing label noise in the original datasets. Also, in practice it is unclear how to tune these cleaning methods well, without explicit knowledge of where the label errors are.
Deep learning models are inherently robust to label noise. Perhaps unsurprisingly, we found that most downstream classifiers were reasonably robust to synthetic label noise, as can be seen from the insignificant difference between the setting where datasets were not explicitly cleaned (NON), compared to when they were cleaned using SIM, CIN and AUM. These results also illustrate the importance of measuring both hypotheses (performance of cleaning methods versus downstream models) when evaluating the performance of ML models in the presence of label noise.
Adding label noise can sometimes improve model performance. In the context of class-dependent or uniform noise, label noise serves as regularization to prevent models from overfitting. This phenomenon is not specific any one modality, but happens for multiple modalities, datasets, and noise types too, for example Electric Devices (time-series) under uniform noise, MIT-BIH (time-series), and Dry Bean (tabular) for class-dependent noise, in Table 16. Moreover, deep learning optimization is highly non-convex, so adding some noise might help the model reach the global minima by traversing an alternative path within the loss landscape.
Impact of AQuA's design choices. We found that cleaning methods perform differently for different data modalities. For instance, all cleaning methods barring CON perform on par on image datasets (iii), but on tabular data  $(iv)$ , AUM performs significantly worse than CIN and SIM. This may be due to a variety of reasons beyond cleaning methods: size and nature of datasets, inductive biases of downstream classifier, and the quality of feature representations [34]. We also observed that some types of label noise are easier to detect than others. For example, uniform noise and asymmetric noise were the easiest to detect, cleaning methods found it much hard to detect instance and class-dependent noise  $(\nu i)$ . Finally, we noticed differences in model rankings when measuring different evaluation metrics. As an example, the difference between CIN and AUM vanishes when we measure the accuracy  $(\nu)$  of the cleaning methods instead of their weighted  $F_{1}(i)$ . These findings highlight the need to evaluate label error detection methods across multiple datasets from different modalities, noise types and evaluation metrics.
We propose the first benchmark designed to rigorously evaluate machine learning models in the presence of label noise. We also elucidate the design space of these methods to not only enable ML practitioners to choose the right label cleaning tool for their data, but also foster academic research on the label noise problem. We demonstrate AQuA's utility by running large-scale experiments to glean several interesting findings. We believe that, as a benchmarking toolkit, AQuA would benefit from more cleaning methods, datasets, synthetic label noise injection strategies, and evaluation metrics.
Our short-term goals include experimenting with multi-annotator label noise, measuring the impact of feature noise on time-series and image data in comparison to label noise, incorporating several metrics for model generalization, robustness and fairness, and including audio datasets. While other types of noise are beyond the scope of this work, we believe that multi-annotator, multi-class multi-label, and noise in regression problems are exciting avenues of future work, and AQuA's modular design will enable researchers to experiment with both multi-annotator and multi-class multi-label classification problems easily. We restrict ourselves to multi-class but single-label classification (as opposed to multi-label classification).
We believe that future work on label error detection should address label issues in the multi-label classification and regression settings. We believe that our work on AQuA can both harness and facilitate the development of foundation models in the two ways: (1) foundation models can be used to identify labeling errors, without explicit supervision, and (2) methods within AQuA can be used to identify labeling errors which can affect foundation model pre-training and fine-tuning. We also believe that future work should
7 Limitations, Biases, and Social Impacts
We acknowledge the potential adverse impact of large-scale experimentation on the environment, but believe that our publicly accessible code and experimental findings can significantly reduce resource consumption for ML practitioners in this field. Label error detection models might perpetuate existing biases and impact the fairness of models. We included the Adult dataset, that is frequently used in the fairness literature, in AQuA, to evaluate the impact of label errors on the fairness of models. We
would also like to acknowledge that our experiments were carried without extensive hyper-parameter tuning. Moreover, hyper-parameters for cleaning methods and downstream classifiers were chosen based on model performance on the observed training set and fixed throughout the training process. We further discuss these design choices and their limitations in Appendix A.6
We would like to thank Cherie Ho and Jack H. Good for their useful comments on initial drafts of the paper. This work was partially supported by the National Institutes of Health under awards R01HL141916, 1R01NS124642-01, and 1R01DK131586-01, and by the U.S. Army Research Office and the U.S. Army Futures Command under Contract No. W911NF-20-D-0002. The content of the information does not necessarily reflect the position or the policy of the government and no official endorsement should be inferred.
[1] Benoit Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845-869, 2013.
[2] Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study. The Artificial Intelligence Review, 22(3):177, 2004.
[3] José A Sáez, Mikel Galar, Julián Luengo, and Francisco Herrera. Analyzing the presence of noise in multi-class problems: alleviating its influence with the one-vs-one decomposition. Knowledge and information systems, 38:179-206, 2014.
[4] Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, December 2021.
[5] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.
[6] Jeremy Howard. imagenette. URL https://github.com/fastai/imagenette/
[7] Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Marc Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S Boning. Learning from multiple annotator noisy labels via sample-wise label fusion. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 407-422. Springer, 2022. ISBN 978-3-031-20053-3.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[9] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017. URL http://arxiv.org/abs/1704.04861
[10] Saining Xie, Ross Girshick, Piotr Dólár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5995, 2017. doi: 10.1109/CVPR.2017.634.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, 2019.
[81] Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoc tests based on mean-ranks? The Journal of Machine Learning Research, 17(1):152-161, 2016.
[82] Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data mining and knowledge discovery, 31:606-660, 2017.
[83] Michael A. Hedderich, D. Zhu, and Dietrich Klakow. Analysing the noise model error for realistic noisy label data. In AAAI Conference on Artificial Intelligence, 2021.
[84] Dawei Zhu, Michael A. Hedderich, Fangzhou Zhai, David Adelani, and Dietrich Klakow. Is BERT robust to label noise? a study on learning with noisy labels in text classification. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 62-67, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.insights-1.8. URL https://aclanthology.org/2022.insights-1.8.
[85] Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 322-330, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10.1109/ICCV.2019.00041. URL https://doi.ieeecomputersociety.org/10.1109/ICCV.2019.00041.
[86] Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA, 316(22):2402-2410, 12 2016. ISSN 0098-7484. doi: 10.1001/jama.2016.17216. URL https://doi.org/10.1001/jama.2016.17216.
[87] Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision, 2022.
[88] Mononito Goswami, Benedikt Boecking, and Artur Dubrawski. Weak supervision for affordable modeling of electrocardiogram data, 2022.
[89] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917-963, 2019.
[90] Milton Friedman. A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86-92, 1940. ISSN 00034851. URL http://www.jstor.org/stable/2235971
[91] Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987. URL http://www.jstor.org/stable/3001968.
[92] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979. ISSN 03036898, 14679469. URL http://www.jstor.org/stable/4615733