Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI
Authors: Nabeel Seedat
Fergus Imrie
Mihaela van der Schaar
Venue: ICLR 2024 poster (2024)
Keywords: data-centric AI
hardness characterization
data-centric ML
DMLR
benchmarking
Primary area: datasets and benchmarks
TL;DR: A fine-grained analysis of Hardness Characterization Methods across multiple hardness types

Abstract: Characterizing samples that are difficult to learn from is crucial to developing highly performant ML models. This has led to numerous Hardness Characterization Methods (HCMs) that aim to identify ''hard'' samples. However, there is a lack of consensus regarding the definition and evaluation of ''hardness''. Unfortunately, current HCMs have only been evaluated on specific types of hardness and often only qualitatively or with respect to downstream performance, overlooking the fundamental quantitative identification task. We address this gap by presenting a fine-grained taxonomy of hardness types. Additionally, we propose the Hardness Characterization Analysis Toolkit (H-CAT), which supports comprehensive and quantitative benchmarking of HCMs across the hardness taxonomy and can easily be extended to new HCMs, hardness types, and datasets. We use H-CAT to evaluate 13 different HCMs across 8 hardness types. This comprehensive evaluation encompassing over 14K setups uncovers strengths and weaknesses of different HCMs, leading to practical tips to guide HCM selection and future development. Our findings highlight the need for more comprehensive HCM evaluation, while we hope our hardness taxonomy and toolkit will advance the principled evaluation and uptake of data-centric AI methods.

Published as a conference paper at ICLR 2024
DISSECTING SAMPLE HARDNESS: A FINE-GRAINED ANALYSIS OF HARDNESS CHARACTERIZATTON METHODS FOR DATA-CENTRIC AI
Nabeel Scedat University of Cambridge ns 7 7410 cacmamm.m ac. uk
Fergus Imric UCLA imrie@ucla.edu
Mihaela van der Schaar University of Cambridge mv4720 cam. ac. uk
ABSTRACT
Characterizing samples that are difficult to learn from is crucial to developing highly performant ML models. This has led to numerous Hardness Characterization Methods (HCMs) that aim to identify "hard" samples. However, there is lack of consensus regarding the definition and evaluation "hardness". Unfortunately, current HCMs have only been evaluated specific types of hardness and often only qualitatively or with respect to downstream performance, overlooking the fundamental quantitative identification task. We address this gap by presenting fine- tree-graied taxonomy of hardness types. Additionally, we propose the Hardness Characterization Analysis Toolkit (H-CAT), which supports comprehensive and quantitative benchmarking of HCMs across the hardness taxonomy and can easily be extended to new HCMs, hardness types, and datasets. We use H-CAT evaluate 13 different HCMs across hardness types. This comprehensive evaluation encompassing over 14K setups uncovers strengths and weaknesses of different HCMs, leading to practical tips to guide HCM selection and future development. Our findings highlight the need for more comprehensive HCM evaluation, while we hope hardness taxonomy and toolkit will advance the principled evaluation and uptake of data-centric AI methods.
INTRODUCTION
Data quality, an important MI problem. Data quality is crucial to the performance and robustness of machine learning (ML) models (Jain et al., 2020; Gupta et al., 2021; Renggli et al., 2021;
Sambasivan et al., 2021; et al., 2021). Unfortunately, challenges arise in real-world data that make samples "hard" for ML models to learn from effectively, including but not limited to mislabeling, outliers, and insufficient coverage (Chen et al., 2021: Li et al., 2021). These "hard" samples or data points can significantly hamper the performance of ML models, creating barrier ML adoption in practical applications (Bedi et al., 2019; West, 2020). For instance, model trained on mislabeled samples lead inaccurate predictions (Krishnan et al., 2016; Gupta et al., 2021). Outliers can bias the model to learn suboptimal decision boundaries (Liu et al., 2022; Eduardo et al., 2022; Krishnan et al., 2016), harming model performance. Long tails of samples can result poor model performance for these cases (Feldman, 2020; Hooker et al., 2019; Hooker, 2021; Agarwal et al., 2022). Consequently, "hard" samples can pose serious challenges for training and the performance of ML models, making it crucial to identify these samples. This is especially important in where manually identifying "hard" samples is expensive, impractical, or time-consuming given the scale.
Characterizing hardness, growing area. Recent interest in data-centric AI, which aims to ensure and improve data quality, has led to the development systematic methods to characterize the data used to train ML models (Liang et al., 2022; Seedat al., 2023b; 2022b). Data characterization typically assigns scores to each sample based on its learnability and utility for an ML task, thereby facilitating the identification of "hard" samples. We collectively refer to methods that perform the characterization as Hardness Characterization Methods (HCMs).
After samples are characterized, how and for what purpose they used can differ. For exam- ple: (1) curating datasets via sample selection to improve model performance (Maini et al., 2022; Swayamdipta et al., 2020: Seedat al., 2022a; Northeutt al., 2021a; Pleiss al., 2020; Agarwal

---

Published as a conference paper at ICLR 2024
et al., 2022; Seedat et al., 2023a), (2) sculpting the dataset reduce computational requirements while maintaining performance (Toneva et al., 2019; Paul et al., 2021; Sorscher et al., 2022; Mindermann al., 2022). (3) guiding acquisition of additional samples (Zhang al., 2022), or (4) understanding learning behavior from a theoretical perspective (Baldock et al., 2021; Shwartz et al., 2022; Jiang al., 2021).
Challenges in definition and evaluation. A fundamental and overlooked aspect is that while different HCMs tackle the issue of "hardness", it remains a vague and ill-defined term in the literature. The lack of a clear definition of hardness types has led HCMs, seemingly tackling the same problem, to unintentionally target and assess different aspects of hardness (Table 1). The lack of clarity is further exacerbated by: (1) qualitative evaluation: a significant focus on post hoc qualitative assessment and downstream improvement, instead of the fundamental hardness identification task, and (2) narrow and unrepresentative scope: even when quantitative evaluation has been performed, it has typically focused on a single hardness type, neglecting different manifestations of hardness. The lack of comprehensive and quantitative evaluation means we do not know how different HCMs perform on different hardness types and whether they indeed identify the correct samples of interest.
Can we define sample hardness manifestations and then comprehensively and systematically evaluate the capabilities of different HCMs to correctly detect the hard samples?
Unified taxonomy and benchmarking framework. To answer this question, we begin by defining taxonomy of hardness types across three broad categories: (a) Mislabeling, (b) OoD/Outtier, (e) Atypical. We then introduce the Hardness-Charaateen Characterization Analysis Toolkit (H-CAT), which, to the best of our knowledge, the first unified data characterization benchmarking framework focused on hardness. Using H-CAT, we comprehensively and quantitatively benchmark 13 statc-of-the oftt---r-tee-rtttt HCMs across various hardness types. doing so, we address recent calls for more rigorous benchmarking (Guyon, 2022) and understanding of existing ML methods (Lipton & Steinhardt, 2019; Snoek et al., 2018). We make the following contributions:
Contributions: ! Hardness taxonomy: we formalize a systematic taxonomy of sample-level hardness types, addressing the current literature's ad hoc and narrow scope. By defining the different dimensions of hardness, our taxonomy paves the way for more rigorous evaluation of HCMs. ② Benchmarking framework: we propose H-CAT, which is both (i) a benchmarking standard to evaluate the strengths of different HCMs across the hardness taxonomy and (ii) a unified software tool integrating 13 different HCMs. With extensibility in mind, H-CAT can easily incorporate new HCMs, hardness types, and datasets, thus enhancing its utility for both researchers and practitioners. Systematic & Quantitative HCM evaluation: we use H-CAT to comprehensively benchmark and evaluate 13 different HCMs across 8 different hardness types, comprising over 14K experimental setups. 4 Insights: our benchmark provides novel insights into the capabilities of different HCMs when dealing with different hardness types and offers practical usage tips for researchers and practitioners. The variability in HCM performance across hardness types underscores the importance of multi-dimensione evaluation, exposing gaps and opportunities in current HCMs. We hope H-CAT will promote rigorous HCM evaluations and inspire new advances in data-centric AI.
2 HARDNESS CHARACTERIZATION AND TAXONOMY
We now outline the hardness characterization problem and formalize hardness taxonomy for HCMs.
Learning problem. Consider the typical supervised learning setting, with X and y input and output spaces, respectively. We assume k-class classification problem, i.e. ) = [k], where [k] (1,..., ......, k), with a training dataset D = {(x;, y:) E [N]] with N E N+ samples, where I; X and y1 E y. The goal is to learn model fo X + Y. parameterized by 0 E e.
Hardness problem. To understand the intricacies of data hardness, we start by providing a broad definition of the hardness problem common to all HCMs. As starting point, let us frame the hardness problem in general terms: specifically, some samples or data points are easier for a model to learn from, whilst others may either be harder to learn from or harm the model's performance.
Formally, this assumes that the training dataset can be decomposed as D = D. Dr. where D. are easy samples and Dr are hard samples. We denote the corresponding joint distributions as PxY. P and p Going beyond this general definition to different manifestations of hardness requires more rigorous characterization. However, what constitutes "hard" sample has not been rigorously
2

---

Published as a conference paper at ICLR 2024
Label:
Label:
Label: &
Label: 3
Label:
Label:
Label: 3
Label: 3
3
3
3
3
(i) Uniform (ii) Asymmetric
(iii) Instance
(i) Near OOD (ii) Near OOD (iii) Far OOD
(i) Shift
(ii) Zoom
(Covariate)
(Domain)
(a) Mislabeling
(b) OOD/Outlier
(c) Atypical
Figure I: Examples of the hardness types included within our taxonomy and supported by H-CAT. HCMs need
to be comprchensively assessed across different dimensions to quantify their ability to handle different hardness types we might expect in practice. See Sec. 2.1 for precise definitions of hardness types.
defined in the literature. To address this gap, we first formalize a taxonomy of hardness in Sec. 2.1, providing systematic and formal definition of differrnnt types of sample-level hardness.
Data characterization. The goal of data characterization is to assign a scalar "hardness" score to each sample in D, thereby allowing to order samples in D according to their scores. Typically, selector function applies a threshold T and then assigns a hardness label g E G. where ç = [Easy, H Hard], to each sample (z), i.e. assign samples in D to either D, Dr.- We refer to these methods as Hardness Characterization Methods (HCM), having the following input-output paradigm, where methods differ based on their scoring mechanism:
Inputs: (i) Dataset = { {I;; } drawn from both Pxy. (ii) Learning algorithm fo- Outputs: Assign score 8; to sample (zi, sı). Apply threshold to assign hardness label g E G. where G = (Easy, H ard], which is then used to partition D = D. U UDr
We group the HCMs into broad classes (see Table 1) determined based the core metric or approach each method uses to characterize example hardness, namely (1) Learning dynamics- basai: relying on metries computed during the training process itself characterize example hardness; (2) Distance- based: using the distance or similarity of examples in an embedding space; (3) Statistical- based: using statistical metrics computed over the data characterize example hardness.
2.1 TAXONOMY OF HARDNESS
Hardness taxonomy formalism. The term "hardness" is broad and manifest in various ways, as demonstrated by the different types of hardness previously examined (see Table 1). Therefore, we must first formalize a taxonomy representative of the different types of hardness we might expect in practice. Each type of hardness is characterized by a latent or unseen hardness perturbation function that creates hard samples Dj, from samples in D. We denote hardness perturbations on X as X + X and Y as Y + Y The effect of each hardness perturbation is explained in terms of the relationship between the joint probability distributions y (z. y) and Ph (2.
The taxonomy deals with three broad types (and corresponding subtypes) of hardness: (1) Mislabeling, (2) OoD/Outlier, and (3) Atypical, as illustrated in Figure 1. We anchor the different hardness manifestations with respect to relevant literature for each subtype. We define the various types next.
Samples where the true label is replaced with an incorrect label, such that sample (x. + (.). The main distinction between easy and hard samples lies in the label space, leading to different conditional probability distributions: (xlx + Phh P(ix(ylx)) Note, the marginal
Mislabeling:
probability distributions are the same P (z) = (x).
We consider three subtypes of mislabeling: (i) Uniform, (ii) Asymmetric, and (iii) Instance. The HCM literature primarily focuses on evaluation with a uniform noise model (Paul et al., 2021; Swayamdipta et al., 2020; Pleiss et al., 2020; Toneva et al., 2019; Maini et al., 2022; Jiang et al., 2021; Mindermann al., 2022: Baldock et al., 2021), with equal probability of mislabeling across classes. However, in reality. mislabeling is often asymmetric, where mislabeling is label-dependent (Northcutt et al., 2021a; Sukhbaatar et al., 2015) or cettacc--spcifiee where certain mislabeling is more likely given the sample (Jia et al., 2022; Han et al., 2020; Hendrycks et al., 2018; Song et al., 2022). For example, mislabeling an image of a car as truck more likely than mislabeling a car as a dog.
Formally. the subtypes differ in their noise models to perturb the true labels, defined by the probabili- ties = = i), where and j are the true and perturbed labels, respectively.
Uniform:
P(Y* y = 1) 1/k-1
Vi,j E [R), Vi,j E [k],
Instance Independent
Asymmetric:
P(Y" j = =1) =i)=
Instance:
P(Y* = = =i,X x) = (z)(x) Vi,j E [k], $ } IInstance Dependent
3

---

Published as a conference paper at ICLR 2024
Table I: HCMs, even in similar classes, often (1) quantitatively evaluated and (2) assess different hardness types. The tick and cross denote quantitative and brackets qualitative evaluation was performed. not quantitative (qualitative). For HCM descriptions. see Appendix A.
e.g. X
Mislaheling
OoDiOutlier
Atypical
HCM Class
HCM Name
Uniform
Asymmetric
Instance
Near Ool
Far OoD
Long-tail
(Margin)
Leaming haved
AUM (Pleiss et al. 2020)
V(v)
V(v)
X(v)
X(0)
Learning-baed
Data Maps (Swayamdigtaa et al., 2020)
X()
X(3)
XXX
X(3
X(3)
X(X)
(Uncertainty)
Duta (Seedat et al., 2022a)
Xv)
X00)
X00
X0X)
X(v)
Leurning based
Small:1 los (Xia et al., 2021)
X03
X(X)
X(
(Loss)
Action scores (Arringa et al. 2023)
v(v)
X00)
XXX
XO0
X(X)
X(
RHO-Lass (Mindermann al., 2022)
X()
X00
XX0
XU9
X00
X(
Learing hased
Grail et al., 2021)
K()
x(0)
X01
XV)
x00)
X(
(radient)
(Agarwal al., 2022)
X00
X(0
V(V)
X00
X(
bamiin-based
Forgetting Scores (Toneva ot al., 2019)
X()
X(X)
XXX
X(/
X00
X(
(Forgttingg)
SSFT (Maini al., 2022)
V(v)
X00)
X09
X00)
X(v)
Leaming-haee hased
Detector (Jia et al., 2022)
V(v)
X(X)
XXI
X0X
X(X
XQX)
(Statistics)
EL22 (Paul et 2021)
X()
X(X
XXX
X(
X(X)
X((
Distance hased
Prototypicality (Sorscher al., 2022)
X(0
X(X)
XXX
X((
X00)
XUX
Information theory
PVI (Ethnyarajj et al,, 2022)
KV)
X00
X(X
X00
X03)
X(A)
Cleanlah (Nothhurt et A021a)
A(0)
((()
V(
X03
X00
X(X)
Statistical-based
ALLSII (Zhang et al., 2022)
X00
X(0)
X(0
X00
X00
X(X)
Agreement (Carlini al., 2019)
X00
X00
X00
X(V)
Data Shapley (Ghochami & Zou, 2019)
v(V)
X(0)
X00
V(V)
X00
X(X)
OoD/Outlier: Samples where the covariates undergo a transformation/shiff, such that sample (ar, y) + (x" y)- The distinction between easy and hard samples lies in the feature space, leading to different marginal probability distributions Px (x) * (z). Further, for any subset S within the support of Px, P = O. Note, conditional probability distributions remain consistent where ( = Pa (1)zz). We consider two subtypes differing in degree of shift, for clarity denoted by an arbitrary distance measure between distributions dist(:,-).
Near OoD (Anirudh & Thiagarajan, 2023; Mu Gilmer, 2019; Hendrycks & Dietterich, 2018; Sun et al., 2023b; Yang et al., 2023; Tian et al., 2021): samples which have their features transformed or shifted such that they remain proximal to the original samples in D, e.g, introducing noise, pixclating an image, or adding subtle texture changes. In this case, dist Pxx is positive but relatively small, indicating the nearness of the perturbed distribution to the original. We can represent this bounded distance such that 0 < (P Px) c, with 0.
Winkens et al., 2020; Graham et al., 2022: Yang et al., 2023): samples which are distinet and likely unrelated samples in D, often not belonging to the same data mata-generating process. This could be by sampling from a different dataset or a perturbation s.t. d Px) significantly large, i.c. (Ptt x. Px) > e. For example, for dataset of digits, images of dogs or cats are distinctly different and unrelated. They are not just rare occurrences but images of dogs or cats represent a different data generation process compared to the digits.
Far OoD (Mukhoti et al., 2022:
Atypical: Samples that, although rare, are still valid instances deviating from common patterns (Yuksekgonul et al., 2023). Atypical samples are inherently part of the primary data distribution, but located in its long tail or less frequent regions (Feldman, 2020; Hooker et al., 2019; Hooker, 2021; Agarwal et al., 2022). The distinction between easy and hard samples leads to different marginal probability distributions P (x) * Pt (z), where P (z) is very small, highlighting their rarity or infrequency. Here, for any subset S within the support of Px, (S) > 0. This signifies that the long-tail samples, though rarer in occurrence, are still within the bounds of the primary data-generating process. For example, these could be images with atypical variations or vantage points compared to the standard pattern (Agarwal et al., 2022).
Contrasting OoD/Outlier and Atypical. Both have different marginal probability distributions (x) # + (z). The difference is that OoD/Outliers come from a shifted or completely different distribution than the original data, falling outside the support of Px. In contrast, atypical samples are rare samples from the tails and could naturally arise, falling within the support of Px-
From a practitioner's standpoint, these distinctions can dictate different courses of action. OoD/out- liers represent likely anomalies or errors that we should detect for potential sculpting or filtering from the dataset. In contrast, atypical samples are rare cases deviating from the "norm", with no or limited similar examples. They are still valid points and should not necessarily be discarded; in fact, there might be need to gather more of such samples. The goal of surfacing atypical samples is both for dataset auditing and understanding edge cases. We provide additional example images to provide further intuition of the difference between Ool and Atypical in Appendix A, Fig. 3.
4

---

Published as a conference paper at ICLR 2024
Dataloader
Mislabeling
AUPRC
CSV
AUROC
(g.)
Dataframe
OODiOutlier
Dara Maps
Accuracy
Drrain
Atypical
(i) Dataloader module
Hardness Module
(iii)
Module
Evalnator Module
Thalner
module
MAuhfoormeeeeen oederiitttuutwwe detaowl
puiker Pa Trainsnnimmsinttinn "paans)
nom Louaao bainne nardhasa, (ectoos))
un/omr,
eval - Enalattrrhonniiis dicenttonn scoves,
"parama)
dag starlablle pmo)
datt, lcader - deget cotskader)
cvatoonpen
Figure 2: H-CAT facilitates comprehensive benchmarking of HCMs for multiple hardness types. Examples of the single-chain workflow is shown below - with example usage in Appendix C. The modules described Sec. 3 are easily extended to new HCMs, datasets, or evaluation metrics
2.2 GAPS AND LIMITATIONS OF CURRENT HCMS WITHIN OUR TAXONOMY
To provide a more comprehensive perspective, we critically analyze various HCMs within our proposed taxonomy. Alarmingly, we find inconsistent and diverse definitions of "hardness", even within the same class of methods (refer to Table 1). Such discrepancies, as evidenced by their experimental evaluations, point challenge: HCMs, despite appearing to measure similar constructs, in fact, evaluate different "hardness" dimensions. Table highlights two critical deficiencies discussed below, underscoring the urgent need for systematic evaluation framework that accurately captures the scope and applicability of each HCM across different hardness types.
Issue 1: Qualitative indirect measures. Many HCMs limit their evaluation to (i) qualitative analyses: merely showcasing flagged samples, or (ii) indirect measures: for example, showing downstream performance improved when removing flagged samples without directly quantifying what the HCM captures. This overlooks the necessity for an objective, quantitative evaluation.
Issue 2: Narrow and unrepresentative scope. HCMs that conduct quantitative evaluation focus on a single hardness type and often target the simplest manifestation. An example of this shortfall is seen in the handling of mislabeling. Many HCMs only focus on uniform mislabeling, thereby failing to account for the more realistic and complex scenarios of asymmetric or instance- mislabeling. Beyond this, many HCMs only test on mislabeling, overlooking other types of hardness.
These limitations emphasize the value of our fine-graine ame-grrined taxonomy in categorizing and evaluating various hardness types. By laying a solid foundation for the systematic evaluation and comparison of HCMs, our taxonomy enables the design and selection of HCMs that are tailored to specific hardness challenges, thereby promoting the development of more robust ML systems.
3 H-CAT: A BENCHMARKING FRAMEWORK FOR HCMS
To address the aforementioned limitations and facilitate benchmarking of HCMs, we propose the Hardness Characterization Analysis Toolkit (H-CAT). H-CAT serves two purposes: (1) empirical benchmarking standard: supporting comprehensive and quantitative benchmarking of HCMs on different hardness types within the taxonomy across multiple data modalities and (2) software toolkit: H-CAT unifies multiple HCMs under a single unified interface for easy usage by practitioners.
H-CAT Design. H-CAT has four modules described below, which called sequentially (Fig- ure 2). The framework follows widely adopted abjected-orientee paradigms with fit-predict interfaces (here, update-score). The workflow is simple with single-chain API calls. The stepwise composability aims to facilitate easy benchmarking and allows H-CAT to be used outside of benchmarking as a data characterization tool by practitioners.
Dataloader module: loads a variety of data types for ease of use, including Torch datasets, NumPy arrays, and Pandas DataFrames, allowing users to easily use H-CAT with their chosen datasets. Hardness module: generates controllable hardness for different hardness types in the taxonomy.
HCM module: provides unified HCM interface with 13 HCMs implemented. It wraps the trainer module which is a conventional Py PyTTrch training loop.
Evaluator module: computes the HCM evaluation metrics to correctly identify the ground-truth hard samples using the scores provided by each HCM. User-specified metrics can easily be included by operating on the raw HCM scores.
Extensibility. H-CAT is easily extendable to include new HCMs, hardness types, or datasets by defining simple wrapper class. For details and step-by-step code examples, refer to Appendix C.
5

---

Published as a conference paper at ICLR 2024
4
BENCHMARKING FRAMEWORK SETUP
We now describe three key aspects of the benchmarking setup the implementation of the Hardness Module, HCM Module and Evaluator Module.
4.1 HARDNESS MODULE
We describe the implementation of our hardness perturbations h. We focus on image data here, as most HCMs (10/13) have been developed for this modality. However, we discuss the corresponding hardness perturbations for tabular data in the Appendix.
Mislabeling: (i) Uniform: random mislabeling, drawn uniformly from all possible class labels. (ii) Asymmetric: random mislabeling. drawn asymmetrically via Dirichlet distribution (Zhang et al., 2021; Bae et al., 2022; Zhu al., 2022). We denote a special case of asymmetric applicable to ordinal labels, namely adjacent. Here, the mislabeling is to the nearest numerical class, e.g. an MNIST digit 3 mislabeled as or 4. (iii) Instance: mislabeling probability is conditioned on an instance (reflecting human mislabeling). This can often be determined by domain/user knowledge, e.g. for MNIST, is likely mislabeled as 7; for CIFAR- 10, an automobile could be mislabeled as a truck.
OoD/Outlier: (i) Near OoD: pertubed data is different but related. Covariate Shift via Gaus- sian noise as performed in MNIST-C (Mu & Gilmer, 2019) or Domain Shift: image texture from the original photographic image is changed via edge detection and smoothing (Median fil- ter). (ii) Far OoD: perturbed data is distinctly different and unrelated. We replace a subset of data with unrelated data, e.g. for MNIST replace with CIFAR-10 10 images and vice versa.
Easy
Near-OOD
Far-OOD
Atypical
BUS
aiIN
Figure 3: Examples providing intuition on the differ- ence between OoD and Atypical.
Atypical: (i) Shift: translate and shift the image, causing portions to cut off in an atypical manner. (ii) Zoom: create an atypical perspective by magnifying (X2) features usually seen at a smaller scale.
4.2 HCM MODULE
We include 13 widely used HCMs applicable to supervised classification under a unified interface.
The HCMs span the range of HCM classes, at least one per class from Table 1: Learning-baan (Uncertainty): Data Maps (Swayamdipta et al., 2020) and Data-IQ (Seedat et al., 2022a); Learning- based (Loss): Sample-Loss (Xia et al., 2021; Arriaga et al., 2023): Learning-based (Margin): Area-under-the-maal
(AUM) (Pleiss et al., 2020); Learning- based (Gradient): GraNd (Paul al., 2021) and VoG (Agarwal et al., 2022); Learning-based (Statistics): EL2N (Paul et al., 2021) and Noise detector (Jia et al., 2022). Learning- based (Forgetting): Forgetting scores (Toneva al., 2019); Statistical measures: Cleanlab (Northcutt al., 2021a), ALLSH (Zhang et al., 2022), Agreement (Carlini et al., 2019); Distance-based: Prototypicality (Sorscher et al., 2022).
Specifically, focus on HCMs that plug into the training loop and do not: (i) alter training, (ii) require repeated training. (iii) need additional datasets beyond the training set, or (iv) require training additional models. Consequently, we exclude the following: RHO-Loss (Mindermann et al., 2022) requires an additional irreducible loss model, SSFT (Maini et al., 2022) requires fine-tuning on validation dataset, PVI (Ethayarajh et al., 2022) requires training a Null model. We also do not consider Data Shapley (Ghorbani & Zou, 2019) and variants (e.g. Beta Shapley (Kwon & Zou, 2022)), which have been shown to be computationally infeasible with numerical instabilities for higher dimensional data such as MNIST and CIFAR-10 with > 1000 samples (Wang & Jia, 2023).
4.3 EVALUATOR MODULE
We directly assess the HCM's capability to detect the hard samples. Recall that HCMs assign a
score s to each sample (z, 3) and then apply a threshold to assign samples a group g E G, where
g = [Easy, Hard). Many HCMs do not explicitly state how to define т; hence to account for this compute two widely used metrics: AUPRC (Area Under Precision-Res Recall Curve) and AUROC (Area Under Receiver Operating Curve) - for hard sample detection performance, which we denote D-AUPRC and D-AUROC 2, User-specified metrics easily computed raw HCM scores.
'I -.aaming-baase generally refers to learning/raining dynamics based HCMs
to distinguish them from the typical downstream performance metrics.

---

Published as a conference paper at ICLR 2024
5 COMPREHENSIVE EVALUATION OF HCMS USING H-CAT
We evaluate 13 different HCMs (spanning range of techniques) across 8 distinct hardness types. To the best of our knowledge, this represents the first comprehensive HCM evaluation, encompassing over 14K experimental setups (specific combination of HCM, hardness type, perturbation proportion, dataset, model, and seed).
We primarily focus on image datasets, as this the modality for which the majority of the HCMs (10/13) have been developed. We use the MNIST (LeCun et al., 2010) and CIFAR-10 (Krizhevsky et al.) datasets, as their use well-established in the HCM literature (Paul al., 2021; Swayamdipta al., 2020; Pleiss et al., 2020; Toneva et al., 2019; Maini et al., 2022; Jiang et al., 2021; Mindermann et al., 2022; Baldock et al., 2021). Importantly, they are realistic images yet contain almost no/little mislabeling (<0.5%) (Northcutt al., 2021b). This contrasts other common image datasets like ImageNet that contain significant mislabeling (over 5%), hence we cannot perform controlled experiments. Furthermore, show generalizability across modalities, also evaluate tabular datasets "Covertype" and "Diabetes " benchmark datasets (Grinsztajn et al., 2022) from OpenML (Vanschoren et al., 2014) (see Appendix D.6).
To assess HCM sensitivity to the backbone model, we use two different models for our image experiments with different degrees of parameterization, LeNet and ResNet-18. All experiments repeated with random seeds and for varying proportions of hard samples.
We present aggregated results Figs. 4-7, with more granular results in Appendix D along with additional experiments. The main paper shows results for 6 out of 8 hardness types. We include results for other sub-types not covered in the main paper in Appendix D including: Domain shift (a type of Near OoD), Zoom shift (a type of Atypical), and Adjacent (a special case of Asymmetric mislabeling), offering similar conclusions. We investigate three aspects HCMs (A-C), distilling the results into benchmarking takeaways and practical tips.
A. Hardness detection performance. Directly evaluate HCM capabilities to detect the hard samples for different hardness types, for varying perturbation proportions see Fig. 4.
0.1
1.0
2011
1.0
0.1
1.0
03
0.5
0.8
0.6
0.6
0.6
prodd
forpett
lus
suni
detirtor
athar
prodd
you
forpet
ause
dctirde
clearint
grann
dated
dransen
detietar
wtr
clearina
and
ussend
(a) Uniform
(b) Asymmetric
(c) Instance
0.1
1.0
0.1
1.0
033
0.8
0.8
2.4
0.6
0.6
prodd
clesctint uygee
crt
alles
clearnal
corl.
(d) OoD (Covariate)
(c) Far-OoD
Atypical
Figure 4: D-AUPRC for different HCMs for different hardness types aggregated across setups. We vary the proportion perturbed, i.e. the proportion hard examples. Blue better, red worse. We see variability of HCM capabilities across hardness types and proportions.
Comprehensive testing is vital. Many HCMs are assessed on a single hardness type. The reality is hardness manifests in many ways. We show HCM performance varies across hardness types and across proportions of hardness, with some more challenging than others, c.g. instance is harder than uniform. This result shows the critical need for comprehensive HCM evaluation.
Takeaway A1:
Takeaway A2: Hardness types vary in difficulty. We find that different types of hardness are easier or harder characterize. For instance, uniform mislabeling Far-OoD are much easier than data-specific hardness like instance and Atypical. Given the performance differences of HCMs on different hardness types, it becomes important to understand the hardness type expected in practice.
7

---

Published as a conference paper at ICLR 2024
Takeaway A3: Learning dynamics-based methods with respect to output confidence are effective general-purposee HCMs. In selecting a general-purooss HCM, we find that HCMs that characterize samples using learning dynamics on the confidence uncertainty- -basee methods, which use probabilities (DataMaps, Data-IQ) or logits (AUM), are the best performing in terms of AUPRC across the board.
Takeaway A4: HCMs typically used for computational efficiency are surprisingly uncompeti- tive. We find that HCMs that leverage gradient changes (e.g. GraNd), typically used for selection to improve computational efficiency, fare well at low p. However, at higher p. they become notably less competitive compared to simpler and computationally cheaper methods.
Practical Tip A1: HCMs should only be used when hardness proportions are low. In general, different HCMs have significantly reduced performance higher proportions of hardness, This is expected as we get closer to 0.5 since it's harder to identify a clear difference between samples.
B. Rankings and Statistical Significance. Compare the ranking of methods, as well as assess statistical significance of performance differences using critical difference diagrams (CD diagram) (Demšar, 2006) based on the Siegel-Friedman method (p S 0.05) - see Figs. and
Rank Rank Scores Ho-ow
Scores
gank COoO
Pank
grand
ing
Rits
eut
decary
derestin
atda
creantal cord
vigg
Emryee
lire
Bum
dataly
estamsni
aitah
chantail corll agr
amió grontng
lise
dataly
det aema
dlaanii cord pgg
(a) Uniform
(b) Asymmetric
(c) Instance
Pank Scores
pank
Pank
prote
arand
forpet
anp
dazomsart
cieastab covtl
pinto
foopt lats
som
astal
alih creyntab cort apor
proto
furger
sur
avali Joomsart
alti diarraa cord
(d) Noar-OoD (Covariate)
(c) Far-OoD
(f) Atypical
Figure Performance rankings of HCMs vary depending on the hardness type.
arttropkey
(a) Uniform
(b) Asymmetric
(c) Instancc
Creed Salciaa
prmoppaaly
(d) Near-OoD (Covariate)
(c) Far-OoD
(f) Atypical
Figure 6: Critical difference diagrams highlight that similar categories/classes of HCMs do not have statistically significant difference in their performance, indicated by the horizontal black lines linking HCMs which are not statistically different. The numbers in brackets denote mean rank.
Individual HCMs within a broad "class" of methods are NOT statistically different. We find from the critical difference diagrams that methods falling into the same class of characterization are not statistically significant from one another (based on the black connected lines), despite the minor performance differences between them. Hence, practitioners should select an HCM within the broad HCM class most suitable for the application.
Takeaway B1:
Practical Tip B1: Selecting an HCM based on the hardness is useful. We find that confidence is good general- -urposse tool if one does not know the type of hardness. However, if one knows the hardness, one can better select the HCM. For example, Prototypicality, as expected, is very strong on instance hardness as are able to match samples via similarity of classes in embedding space.
8

---

Published as a conference paper at ICLR 2024
Practical Tip B2: Divergence and distance-based methods suitable primarily for distribu- tional changes. Divergence and eestance-basel methods such as ALLSH and Prototypicality should primarily be used if the hardness is with respect to shift of the data itself rather than mislabeling.
C. Ceabiitt//onnsisteeeyy The rank ordering of samples is important in data characterization (Maini et al., 2022; Wang & Jia, 2023; Seedat et al., 2022a). Hence, desire HCM scores to be stable to ensure consistent insights. As is standard (Maini et al., 2022: Seedat et al., 2022a), we compute the Spearman rank correlation across multiple runs see Fig. 7. We also assess HCM stability/consisteney across backbone models and parameterizations in Appendix D.
Spearma coneation
maoe comeatton
0.6-
06
D.
8.4
(a) Uniform
(b) Asymmetric
(c) Instance
1.0
coneaatin
0.75
Spasmarn
(d) Near-OoD (Covariate)
(e) Far-OoD
(f) Atypical
Figure 7: Certain classes HCMs are more stable and consistent than others (retaining rank order),
with higher Spearman rank correlations for multiple runs of the HCM on the same data
Takeaway C1:
Learning dynamics-based HCMs using output metrics or distance-based HCMs are most stable and consistent. We find across all hardness types that the Spearman rank correlation is highest for HCMs that learning dynamics on the outputs measures - specifically Uncertainty (Probabilities), Margins (Logits), Loss, or Prototypicality. The low correlation for other HCMs highlights sensitivity to the run itself when characterizing data, which would lead to inconsistent ordering.
distance
Takeaway C2: Insights consistent across backbone models & parameterizations. As shown in Appendix D, we find similar results (as above) for the Spearman rank correlation of HCM scores across different backbone models and parameterizations. As the scores are consistent, this indicates that the findings and insights computed on those scores will also be consistent, i.e. those HCMs which were the most stable and consistent remain the most stable and consistent.
Practical Tip Cl: Select a stable HCM. Certain HCMs are more sensitive to randomness. Hence, it advised to select stable and consistent HCMs (higher Spearman correlation) to avoid such effects.
DISCUSSION
We introduce H-CAT, a comprehensive benchmarking framework for hardness characterization in data-centric AI. We analyzed 13 HCMs spanning a range of techniques for variety of setups over 14K. We hope our framework and insights addressing calls for rigorous benchmarking (Guyon, 2022) and understanding of existing ML methods (Lipton & Steinhardt, 2019; Snock al., 2018) will spur advancements in data-centric AI and that H-CAT helps practitioners better evaluate and use HCMs.
Limitations & Future Work. No benchmark can exhaustively test all the possible hardness manifes- tations and this work is no different. However, we cover multiple instances from the three fundamental types of hardness, which is significantly broader than any prior work. Building on this work, future research could investigate cases where multiple types of "hardness" manifest simultaneously (e.g. Near-OoD and mislabeling together) or where hardness continuous rather than binary. To spur this, we provide an example of simultaneous hardness in Appendix D. From usage perspective, future work could also look into the best way hardness scores could be used to guide better model training highlight that HCMs, and by extension H-CAT, cannot tell you
(e.g. data curriculum). Finally, which hardness type exists in a dataset; rather, H-CAT serves to benchmark the capabilities of HCMs unified HCM interface.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors would like to thank Nicolas Huynh, Alicia Curth and the anonymous reviewers for their helpful comments and discussions earlier drafts of this paper and Robert Davis for discussions on the code. NS and FI gratefully acknowledge funding from the Cystic Fibrosis Trust and NSF grant (1722516), respectively. This work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab.
ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics Statement. HCMs that accurately identify hard samples can help make models more robust and reliable. This paper highlights the importance of rigorously evaluating HCMs to better understand their capabilities and guide better usage. This paper aims to enable the community to conduct a more systematic hardness characterization through the proposed taxonomy and benchmarking framework.
Reproducibility Statement. We include implementation details about our benchmark Sec. and 4, as well as in Appendix B. The code for the H-CAT framework can be found at https: github. cwm/ httr//eeeatttabbeeee H/H CAT or https: ht///gtthub com www./ooo//oooeeooeeoo /wwaddaaaacaaaaaaaaee abb/ H-CAT. Step-by-step code examples can be found in the repository and in Appendix C along with guide on how to extend H-CAT to new HCMs, hardness types, and datasets.
REFERENCES
Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10368-10378, 2022.
Rushil Anirudh and Jayaraman Thiagarajan. Out of distribution detection via neural network anchoring. In Asian Conference Machine Learning, pp. 32 47. PMLR, 2023.
Octavio Arriaga, Sebastian Palacio, and Matias Valdenegro- Toro. Difficulty estimation with action scores for computer vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 245-253, 2023.
HeeSun Bae, Seungjae Shin, Byeonghu Na, JoonHo Jang, Kyungwoo Song, and I1-Chul Moon. From noisy prediction true label: Noisy prediction calibration via generative model. International Conference Machine Learning, pp. 1277-1297. PMLR, 2022.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10889, 2021.
Chris Bedi, Edward Cone, Brian Foster, Riva Froymovich, Colleen Haikes, Michael Hubbard, Brian Klapac, Alan Marks, Debbie O' O'Brien, Kathy O' O'Connell, Gaby Pacini-Pinto, Kevin Palmer, Rob Pickering. Chris Pope, Dan Rogers, Julia Smith, and Dave Wright. The Global CIO Point of View The New Agenda for Trunsformative Leadership: Reimagine Business for Machine Learning. Oxford Econometric Service Now, 2019.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:/9/0) 13427, 2019,
Haihua Chen, Jiangping Chen, and Junhua Ding. Data evaluation and enhancement for quality improvement of machine learning. IEEE Transactions on Reliability, 70(2):831-847, 2021.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
Simao Eduardo, Kai Xu, Alfredo Nazabal, and Charles Sutton. Repairing systematic outliers by learning clean subspaces in VAEs. arXiv preprint arXiv:2207. 200002200.000000 2022.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with o-usable information. In International Conference on Machine Learning, pp. 0088-59880000 PMLR, 2022.
10

---

Published as a conference paper at ICLR 2024
Vitaly Feldman. Does learning require memorization? A short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In Interaational Conference on Machine Learning, pp. 2242-2251. PMLR, 2019.
Mark Graham, Saaru-Daieel Tudosiu, Paul Wright, Walter Hugo Lopez Pinaya, U Marie,, Yee H Mah, James T Teo, Rolf Jager, David Werring, Parashkev Nachev, et al. Transformer- based -iitii-ddittitttioon detection for clinically safe segmentation. In International Conference on Medical Imaging with Deep Learning, pp. 457-476. PMLR, 2022.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35: 507-520, 2022.
Nitin Gupta, Hima Patel, Shazia Afzal, Naveen Panwar, Ruhi Sharma Mittal, Shanmukha Guttula, Abhinav Jain, Lokesh Nagalapatti, Sameep Mehta, Sandeep Hans, al. Data quality toolkit: Automatic assessment of data quality and remediation for machine learning datasets. arXiv preprint arXiv:2108. 1005552000.00355,, 2021.
Isabelle Guyon. The data-centric era: How ML is becoming an experimental science. Keynote: Advances in Neural Information Processing Systems, 2022.
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint 1rXiv:2001.044000 2020.
Lasse Hansen, Nabeel Seedat, Mihaela van der Schaar, and Andrija Petrovic. Reimagining synthetic tabular data generation through data-centric Al: A comprehensive benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. Advances neural information processing systems, 31, 2018.
Elisabeth RM Heremans, Nabeel Seedat, Bertien Buyse, Dries Testelmans, Mihaela van der Schaar, and Maarten De Vos. U-PASS: An uncertainty-guided deep learning pipeline for automated sleep staging. Computers in Biology and Medicine, pp. 108205, 2024.
Sara Hooker. Moving beyond "algorithmic bias is data problem".. Patterns, 104))1 2021.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint urXiv:1911.085
2019.
Abhinav Jain, Hima Patel, Lokesh Nagalapatti, Nitin Gupta, Sameep Mehta, Shanmukha Guttula, Shashank Mujumdar, Shazia Afzal, Ruhi Sharma Mittal, and Vitobha Munigala. Overview and importance of data quality for machine learning tasks. In Proceedings of the 26th ACM SIGKDD 2020.
International Conference Knowledge Discovery & Data Mining, pp.
Mohammad Hossein Jarrahi, Ali Memariani, and Shion Guha. The principles of data-centric AI (DCAD). arXiv preprint 1rXiv:2211.161111 2022.
Qingrui Jia, Xuhong Li, Lei Yu, Jiang Bian, Penghao Zhao, Shupeng Li, Haoyi Xiong, and Dejing Dou. Learning from training dynamics: Identifying mislabeled data beyond manually designed features. arXiv preprint arXiv:2212.0932. 2022.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. In International Conference on Machine Learning, pp.
PMLR, 2021.
11

---

Published as a conference paper at ICLR 2024
Sanjay Krishnan, Jiannan Wang, Eugene Wu, Michael J Franklin, and Ken Goldberg. ActiveClean: Interactive data cleaning for statistical modeling. Proceedings of the VLDB Endowment, 9(12): 948-959 2016.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton, CIFAR-I0 (Canadian Institute for Advanced Research). URL http: /www. , cs torooto......................... wdw.eedu///eee //iiffiss far. html..
Yongchan Kwon and James Zou. Beta Shapley: A unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 8780-8802. PMLR, 2022.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann. www.://wwww...........oo wtwpwuw..oo//ee///nniissst 2, 2010.
Peng Li, Xi Rao, Jennifer Blase, Yue Zhang. Xu Chu, and Ce Zhang. CleanML: A study for evaluating the impact of data cleaning on ML classification tasks. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 13-24. IEEE, 2021.
Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho. L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy Al. Nature Machine Intelligence, 4(8):669-46) (77699677,,70 2022.
Zachary C Lipton and Jacob Steinhardt. Research for practice: Troubling trends in machine-learning
scholarship. Communications of the ACM, 62(6))45-53, 2019.
Zifan Liu, Zhechun Zhou, and Theodoros Rekatsinas. Picket: Guarding against corrupted data in tabular data during learning and inference. The VLDB Journal, pp. 1-29, 2022.
Pratyush Maini, Saurabh Garg, Zachary Chase Lipton, and Zico Kolter. Characterizing datapoints via second-split forgetting. In Advances in Neural Information Processing Systems, 2022.
Anna Majkowska, Sid Mittal, David F Steiner, Joshua J Reicher, Scott Mayer McKinney, Gavin E Duggan, Krish Eswaran, Po-Hsuan Cameron Chen, Yun Liu, Sreenivasa Raju Kalidindi, et al. Chest radiograpp interpretation with deep learning models: assessment with hawioooogiss--jjdicati reference standards and ippulation-adjuster evaluation. Radiology, 294(2)422143L. 2020.
Sören Mindermann, Jan Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are Icarnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649, PMLR, 2022.
Norman Mu and Justin Gilmer. MNIST-C: A robustness benchmark for computer vision. arXiv preprint arXiv: 02337, 2019.
Jishnu Mukhoti, Tsung- Yu Lin, Bor-Chun Chen, Ashish Shah, Philip HS Torr, Puneet K Dokania, and Ser-Nam Lim. Raising the bar on the evaluation of -istr--distribttion detection. arXiv preprint
vrXiv:2209.11960 2022.
Afshan Nabi, Berke Dilekoglu, Ogun Adebali, and Oznur Tastan. Discovering misannotated Incrnas using deep learning training dynamics. Bioinformatics, 39(1) 39(t):btac821, 2023.
Andrew Ng, Lora Aroyo, Cody Coleman, Greg Diamos, Vijay Janapa Reddi, Joaquin Vanschoren, Carole-Jean Wu, and Sharon Zhou. NeurIPS data-centric AI workshop. 2021. URL https: aatatt ciriiiiii.. orq/.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021a.
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round l), 2021b.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 90596-20607 2021.
12

---

Published as a conference paper at ICLR 2024
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33: 17044-17056, 2020.
Cedric Renggli, Luka Rimanic, Nezihe Merve Gürel, Bojan Karlas. Wentao Wu, and Ce Zhang, A data quality-driven view of MLOps. IEEE Data Engineering Bulletin, 2021.
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "Everyone wants to do the model work, not the data work": Data cascades in high-stakes AI. In proceedings of the 2021 CHI Conference Human Factors in Computing Systems, pp. 1-15, 2021.
Nabeel Seedat, Jonathan Crabbé, Ioana Bica, and Mihaela van der Schaar. Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data. In Advances in Neural Information Processing Systems, 2022a.
Nabeel Seedat, Jonathan Crabbé, and Mihaela van der Schaar. Data-SUITE: Data-centric identification of in-distribution incongruous examples. In International Conference on Machine Learning, pp. 19467-19496. PMLR, 2022b.
Nabeel Seedat, Jonathan Crabbé, Zhaozhi Qian, and Mihaela van der Schaar. TRIAGE: Characterizing and auditing training data for improved regression. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. Navigating data-centric artificial intelli- gence with DC-Check: Advances, challenges, and opportunities. IEEE Transactions Artificial Intelligence, 2023b.
Daniel Shwartz, Uri Stern, and Daphna Weinshall. The dynamic of consensus in deep networks and the identification of noisy labels. arXiv preprint arXiv:2210.0053: 2022.
Jasper Snoek, Alex Wiltschko, and Ali Rahimi. Winner's curse? On pace, progress, and empirical rigor. In International Conference on Learning Representations Workshop, 2018.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: Beating power law scaling via data pruning. In Advances in Neural Information Processing Systems, 2022.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Train- ing convolutional networks with noisy labels. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Hao Sun, Alex J Chan, Nabeel Seedat, Alihan Hüyük, and Mihaela van der Schaar. When is off-policy evaluation useful? a data-centric perspective. arXiv preprint arXiv:2311. 1,1 2023a.
Hao Sun, Boris van Breugel, Jonathan Crabbé, Nabeel Scedat, and Mihacla der Schaar. What is flagged in uncertainty quantification? latent density models for uncertainty categorization. Advances in Neural Information Processing Systems, 36, 2023b.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannanch Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275-9293, 2020.
Junjiao Tian, Yen- Yen-Chang Hsu, Yilin Shen. Hongxia Jin, and Zsolt Kira. Exploring covariate and concept shift for detection and confidence calibration of distrributoonn data. arXiv preprint 1rXiv:2/10.1 15231, 2021.
13

---

Published as a conference paper at ICLR 2024
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):99600, 2014.
Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In Interaational Conference on Artificial Intelligence and Statistics, pp. 6388 6421. PMLR, 2023.
Amanda West. Machine Learning Models for the Future: The rise of the data scientist. Refinitiv Labs, 2020.
Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collection and quality challenges deep learning: A data-centric AI perspective. The VLDB Journal, pp. 1-23, 2023.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566. 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In International Conference on Learning Representations, 2021.
Jingkang Yang, Kaiyang Zhou, and Ziwei Liu. Full-spectrum out-of-distribution detection. Interna- tional Journal of Computer Vision, pp. 1-16, 2023.
Mert Yuksekgonul, Linjun Zhang, James Zou, and Carlos Guestrin. Beyond confidence: Reliable models should also consider atypicality, arXiv preprint arXiv:2305.18262, 2023.
Shujian Zhang, Chengyue Gong, Xingehao Liu, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. ALLSH: Active learning guided by local sensitivity and hardness. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1328-1342, 2022.
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In International Conference on Machine Learning, pp. 12501-12512. PMLR, 2021.
Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. In International Conference Machine Learning, pp. 27633-27653, PMLR, 2022.
14

---

Published as a conference paper at ICLR 2024
Appendix: Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI
Table of Contents
A Additional background and details
17
A.1
Description of HCMs
17
A.1.I
Learning-based (Margin)
18
A.1.2
Learning-based (Uncertainty)
A.1.3
Learning-basee (Loss)
A.1.4
Learning-based (Gradient)
19
A.1.5
Learning-based (Forgetting)
19
A.1.6
Learning- based (Statistics)
19
A.1.7
Distance-based Statistical-based
20
A.1.8
20
A.2
Practical applications of HCMs.
20
A.3
Hardness laxonomy for tabular data.
21
B Experimental details
23
B.I
Computational resources
23
B.2
Datasets
................................
23
Backbone models
................................
23
B.4 HCMs
................................
24
B.5
Implementation details
................................
25
C
Using H-CAT
26
C.I H-CAT Benchmarking Framework Repository
i.pr............................
26
Tutorial on using H-CAT
26
C.3 Extending H-CAT
29
C.3.1 Adding new HCM
29
C.3.2 Using different dataset
30
D Additional experiments
31
D.1 Multiple hardness types simultancously
simulancosly....................
31
Stability/Connistenney: Backbone model
ie..............................
32
D.3 Stability/Consistenncy: Parameterizations
33
D.4 Additional aggregated results
34
D.5 Individual results Images
35
D.S.I
Mislabeling: Adjacent
35
D.5.2
Mislabeling: Asymmetric......................
Witeea..........................
38
D.5.3
Mislabeling: Instance
41
D.5.4
Mislaheling: Uniform
44
D.5.5
Near OOD: Covariate
47
D.5.6
Near OOD: Domain
50
D.5.7
Far OoD
53
D.5.8
................................
Shift
56
D.5.9
Atypical: Zoom Shift
59
D.6
Individual results Tabular
62
15

---

Published as a conference paper at ICLR 2024
D.6.1
Mislabeling: Uniform
62
D.6.2
Mislaheling: Asymmetric
63
D.6.3
Mislabeling: Instance
64
D.6.4
Near OOD: Covariate
65
D.6.5
Far OOD
66
D.6.6
Atypical
D.7
Additional results Medical Ray data
68
D.8
Insights: effect of sevenity of perturbation hardness scores
70
16

---

Published as a conference paper at ICLR 2024
A ADDITIONAL BACKGROUND AND DETAILS
This appendix provides additional details and background on the different HCMs.
A.1 DESCRIPTION OF HCMs
We now provide a detailed description of the HCMs supported by H-CAT grouped by their class. We briefly describe how they characterize data and the modalities to which the HCMs have been previously applied.
We group the HCMs into broad classes determined based on the core metric or approach each method uses to characterize example hardness, namely
Learning dynamics-based based: These HCMs rely on metries computed during the training process itself to characterize example hardness. As we describe in the paper, we further divide these into sub-categories based on the specific training dynamic metric used: (i)
Margin, (ii) Uncertainty, (iii) Loss, (iv) Gradient, (v) Forgetting.
Distance-based. These HCMs characterize hardness based on the distance similarity of examples in an embedding space.
cadistical-basede These methods use statistical metrics computed over the data to character- ize example hardness.
Recall HCMs generally follow the following input-output paradigm:
Inputs: (i) Dataset = {(r;; }i)} drawn from both Pxy. (ii) Learning algorithm So- Outputs: Assign score s; to sample (æ,, y1) Apply threshold to assign hardness label g G. where G = [ass, H ard), which is then used to partition D = De U Dn.
Consequently, we define the scoring function S as the general notation applicable to each HCM. Note that whether a high score or low score indicates a "hard" sample is method-dependee HCMs included in H-CAT with * and formalize those specifically with notation.
We highlight
We provide a Table below to clarify whether high or low scores mean "hard" for specific HCMs.
Table 2: Overview of HCMs and the meaning of their scoring functions.
Method
Description
Score
Area Under Marinn (AUM)
Characterizes data examples based on the margin classificr ie., the difference between the logit values of the correct class and the next class.
Hard - low scores.
Conlident Learning
Contident Learning estimates the joint distribution moisy and
Hard - low
true labels - characterizing data as casy and hard for mislabeling.
Conl Agree
Agrvement meusures the agreement predictions the sume
Hard low
example.
Hard low confidence scores. Low Alentoric Uncertainty scores
Data IQ
Data-1Q computes the alcatoric uncertainty and confidence to charcterrie the data into easy, amhigunus, and hard examples.
Hard low confidence scores. Low Epistemic Uncertaiaty scores
Data Maps
Data Maps focuses on measuring variability (epistemic uncer-
lainty) and confidence characterize data easy, ambigu-
ous, and hard examples.
Gradient Norm (GraNd)
GraNd measures the gradient norm churacterize data.
Hard high scores Hard high scvres
Emoor L2 Norm (EL.2N)
EI..N calculates the 1.2 norm of ervor over tramnnn in order to
characterize for computational purposes.
Forgetting
Forgetting scores analyze example transitions through training. ie., time sample correctly leamed one epoch then
Hard - liggh scores
forgotten.
Large Loss
Large Loss characterizes data based on sample-leve tudes.
magni-
Hard high scores
Prototypicalilty
Prototypicality calculates the latent space clustering distance of the sampls to the class centroid as the metric to characterize data.
Hard high scores
Varisnce of Gradients (VOG)
YoG Variiance of gradients) estimates the variance of gradients for each sample over training
Hard high scores
ALL.SH
A.S. computes the KI. divergence of sofimax patputs between original and augmented samples churacterize data.
Hard - high scores
17

---

Published as a conference paper at ICLR 2024
We now outline different HCMs. Specifically, formalize the HCMs included in H-CAT in unified notation of S(r
A.1.1 LEARNING-BASED (MARGIN)
AUM (Pleiss et al., 2020)* AUM (Area under the margin) characterizes data examples based on the margin of a classifier - i.c. the difference between the logit values of the correct class and the next class. It has been applied to mislabeling for computer vision (CV).
assigned logit
largest other logit
M (zx v)
(0) max b (x)
Over all epochs the AUM is then computed as the average of these margin calculations. We define in terms of S.
S(z,y) = AUM((, - + Et M (z,
assigned logit
largest other logit
S(x, v) + Et'( (t)
maxiş 2, (x))
A.1.2 LEARNING-BASED (UNCERTAINTY)
Data-IQ (Seedat al., 2022a)* Data-IQ computes the aleatoric uncertainty and confidence to characterize the data into easy, ambiguous, and hard examples and has been studied for the tabular domain.
The alcatoric uncertainty is computed as (z) # Σt= P(r, 0.) P(x, 0.)) and confidence over epochs as (xx,6 0.).
For hard samples, the score for an input (x.y) is computed primarily on the confidence with uncertainty used delineate ambiguous:
1
S(x,y)
P(x, 02)
T
T P(r, 0c)
C(æ,v)
(5)
T
Data Maps (Swayamdipta et al., 2020)* Data Maps focuses on measuring variability (epistemic uncertainty) and confidence characterize the data into easy, ambiguous, and hard examples and has been studied for natural language processing (NLP) tasks.
The epistemic uncertainty is computed as Vep (x) + [P(x, [(()))))p())) and confidence over epochs as P(x,0)).
For hard samples, the score for an input (x.y) is computed primarily on the confidence with uncertainty used to delineate ambiguous:
5(r. y) = (P(x.
A.1.3 LEARNING-BASED (Loss)
Small Loss (Xia al., 2021)* Small Loss characterized data based on sample-level loss magnitudes and has been studied for computer vision (CV) tasks.
The small loss for an input æ, y computed as:
S(x, y)
(7)
1=1
where
4 4=;(((w;)); v).)))
18

---

Published as a conference paper at ICLR 2024
RHO-Loss (Mindermann et al., 2022) RHO-Loss estimates the sample loss holdout and has been applied in CV tasks. We do not include it in H-CAT as it requires training an irreducible loss model.
A.1.4 LEARNING-BASED (GRADIENT)
GraNd (Paul et al., 2021)* GraNd measures the gradient norm to characterize data, particularly for computational reasons in computer vision (CV) tasks.
The gradient norm at epoch for an input T, is computed as
Szx v) E
forl (fe(o,))) v) w/th (x) (x)
wherey (x) V wt (x).
(8)
VoG (Agarwal et al., 2022)* VoG (Variance of gradients) estimates the variance of gradients for each sample over training, particularly in computer vision (CV) tasks.
The VoG at epoch / for an input is computed as
N S(x,y) Σ
(VoGG
(9)
K
VoGp(x, =
Σ
(M, (z, y) -x)2.
(3)
\
K
where μ = 1 ΣK M₁
Note Mj is the gradient matrix.
A.1.5 LEARNING-BASER (FORGETTING)
Forgetting scores (Toneva et al., 2019)* Forgetting scores analyze example transitions through training. i... the time sample correctly learned at epoch is then forgotten. It is commonly used in CV tasks.
The Forgetting score input z, y is computed as S (x. 3) < S(z, y) + 1 (prev_ serev_ace__a > acc(2, y))
(10)
SSFT (Maini al., 2022) SSFT (Second Split Forgetting Time) measures the time of forgetting when fine-tuning models on a second split. It has been applied in CV tasks. We do not include it in H-CAT as it requires fine-tuning validation dataset,
A.1.6 LEARNING-BASHE (STATISTICS)
EL2N (Paul et al., 2021)* EL2N (L2 norm of error over training) calculates the L2 norm of error over training in order characterize data for computational purposes. predominantly applied in CV tasks.
The EL2N at epoch t for an input x, y is computed as S(x, v) [llp((,, x) y1|2-
(11)
Noise detector (Jia et al., 2022)* Noise detector utilizes a trained detector model to predict noisy samples. The detector is applied to training dynamics and has been used in CV tasks.
Noise detector uses a trained detector g to score x.y as:
S(z,, w/) = gl Ly. (i))
(12)
where T(i,t)
dynamic]
19

---

Published as a conference paper at ICLR 2024
A.1.7 DISTANCE-BASE BASED
Prototypicality (Sorscher al., 2022)* Prototypicality calculates the latent space clustering distance to the centroid as the metric to characterize data. It has been applied to CV tasks.
Prototypicality score for x.y computed as:
S(a, y) [|((x)) - Myll2
(13)
Here, (xx) represents the mapping of the latent space, and My is the centroid of the class y in the latent space. The norm ||2 denotes the Euclidean distance.
A.1.8 STATISTICAL--ASED
Cleanlab (Northcutt et al., 2021a)" Cleanlab estimates the joint distribution of noisy and true labels characterizing data as easy and hard for mislabeling. can be applied to any modality.
The Cleanlab score for an input v; y is computed as:
S(z. v) = f E X y" + j} for r from the off-diagonals of C yy.y"
(14)
ALLSH (Zhang al., 2022)*
ALLSH computes the KL divergence of softmax outputs between
original and augmented samples to characterize data.
The Allsh score for an input v, y is computed as:
S(z, y) ((po(-1))),
(15)
where x' augmented and D a distance measure (e.g KL-Divergence).
Agreement (Carlini et al., 2019)* Agreement measures the agreement of predictions the same example. is is commonly used in CV tasks.
The Agreement for input 2; y is computed as
S(x = N max f((x)
(16)
Data Shapley (Ghorbani & Zou, 2019) Data Shapley computes the Shapley value to characterize the value of including data point. It is commonly used in CV tasks. We also do not consider Data Shapley in H-CAT and variants (e.g. Beta Shapley (Kwon & Zou, 2022)). which have been shown be computationally unfeasible with numerical instabilities for higher dimensional data such as MNIST and CIFAR-10 10 with 1000 samples (Wang & Jia, 2023).
A.2 PRACTICAL APPLICATIONS OF HCMs.
To illustrate the practical value and need for HCMs in creating high-quality datasets, we highlight the following application domains:
The Cleanlab HCM was used audit ML benchmark datasets for mislabeling (Northcutt et al., 2021b).
The Data Maps HCM was used to audit and create high-quality RNA data in biology (Nabi et al., 2023).
The Data-IQ HCM was used to audit and curate high-quality sleep staging data (Heremans et al., 2024).
The Data-IQ and Cleanlab HCMs have been used to guide synthetic data generation (Hansen et al., 2023).
The Data Maps and Data-IQ HCMs where re- ce-purposed to assess samples for data-centric off-policy evaluation (Sun al., 2023a).
20

---

Published as a conference paper at ICLR 2024
A.3 HARDNESS TAXONOMY FOR TABULAR DATA.
In the main paper, we primarily discussed our manifestation of the hardness taxonomy with respect to images. We describe below how this isn't confined to images and how these hardness types could manifest, for instance, in tabular data.
Mislabeling in tabular data can occur when the label attributed to a particular row (representing a sample) is incorrect. For instance, in a medical dataset, this could occur when a patient with certain condition wrongly labeled as not having the condition (or vice versa). Note mislabeling is no different from the image domain as the perturbation is applied on y.
Mislabeling:
Hence, the various sub-types of mislabeling can still apply. Uniform mislabeling means any class of disease can be mislabeled as any other with equal probability. Asymmetric mislabeling would mean certain classes of disease are more likely to be confused with others, for instance, certain diseases might be more commonly misdiagnosed each other. Consequently, in implementation uniform and asymmetric do not change from images. Adjacent mislabeling (special case of asymmetric) could refer to situations where labels are ordinal and a higher value might be mislabeled as lower one (or vice versa). mestance-speciffe mislabeling refers to instances where certain individual observations are more likely to be mislabeled based on their specific attributes. We could still use "human knowledge" of similar classes for tabular data. We demonstrate the use of PCA to embed the data and then find the nearest class for specific label in emhedding space the instance-wis mislabeling.
Near-OoD in tabular data could refer to rows of data where the feature values are still within the same domain but differ in some way from typical observations. For example, in a financial dataset, a Near OoD sample might be an account with an unusually high number of transactions, which, although not completely unusual, stands apart from typical transaction behavior. This could represent a Covariate Shift where the distribution of the predictors changes, or Domain Shift where the rules of data generation change slightly.
Near-OoD:
Alternatively, in dataset of regular health screenings, a Near OoD instance might be health record of patient who shows high values in their lab results, which although still plausible, stand apart from typical health metrics. For instance, patient with very high cholesterol levels would be Near OoD example in this context, indicating a Covariate Shift where the distribution of predictors changes. In terms of implementation, the continuous variables of our tabular data might be noisy or covariate shifted data. For example, Gaussian noise to continuous features.
Far-OoD: Far Out-of of-Distribution in tabular data refers to rows that are dramatically different from typical observations in a dataset, or might even represent entirely new categorics of data. Using the financial dataset example, a Far OoD sample might be transactions in new foreign currency not previously seen in the dataset. This type of data is distinctly different and unrelated to the rest of the dataset.
In a medical dataset, Far-OoD might refer to patient records that are substantially different from typical observations that fall into entirely new categories. For instance, imagine dataset mainly consisting of adult patients, and suddenly you get data from pediatric patients. Children have different medical characteristics and range for many medical metrics (like heart rate, blood pressure, etc.), so pediatric patient data would be Far OoD in this context. Another example could be the inclusion of a new disease class not previously part of the dataset.
Practically, for a given tabular dataset, this could be data from a different distribution (e.g. shuffled data) or feature relationships that don't match - e.g. older people having a specific feature linked to a disease, but these relationships are now broken. For example, within the framework this could be implemented as follows: for binary covariates, we can swap the features (i.e. flip male to female etc.) to break the feature relationships such that they drawn from completely different distribution. For categorical variables, we can flip the category of a subset of variables in order to break the feature relationships.
Atypical: Atypical data in a tabular dataset can refer to observations that, while not incorrect or out of distribution, deviate from common patterns in the data, i.e. they are rare but valid rows of the table (e.g. outliers, extremes- - samples from tails of marginal). For example, in a dataset of real estate properties, an atypical sample could be property with an unusually large number of rooms or an
21

---

Published as a conference paper at ICLR 2024
unusually low price for its size. While such property could exist (so it's not necessarily mislabeled), it deviates from the common pattern in the data and might be found in the long tail of the distribution.
In medical dataset, it might refer to patient records that, while not incorrect out of distribution, deviate from common patterns in the data. For instance, consider a dataset of prostate cancer patients, where naturally, most patients are older. An atypical example might be young adult with prostate cancer, which would represent an unusual instance, as they fall outside the common pattern of older patients with prostate cancer. From framework implementation perspective, we could sample and replace with values from the tails of the marginals for the most predictive long-tailed feature (via leature-label correlation).
22

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL DETAILS
This appendix describes our experimental details, computational resources, HCMs, and datasets below.
B.1 COMPUTATIONAL RESOURCES
Experiments are parallelized to run different setups using three NVIDIA RTX A4000s, with an 18-Core Intel Core 19 and 16 gb of RAM. Each experimental setup be run single GPU.
B.2 DATASETS
We rum experiments MNIST and CIFAR-10 datasets. However, these experiments can be run on alternative datasets, as shown in Appendix C.
MNIST (LeCun et al., 2010) consists of 70,000 small square 28x28 pixel grayscale images of digits from 0 to 9. Each image a handwritten digit, with 60,000 of the images dedicated for training data and the remaining 10,000 for testing data.
CIFAR-10 (Krizhevsky et al.) consists of 60,000 32x32 color images distributed among 10
different classes such as airplanes, dogs, cats, and trucks. The dataset is divided into 50,000 training images and 10,000 testing images. Each class has an equal representation in the dataset with 6,000 images per class.
Note on selection of MNIST & CIFAR-10. We chose the MNIST and CIFAR-10 10 datasets as benchmarks because they are well-established datasets in the literature. Notably, many HCMs employ MNIST and CIFAR-10 10 within the context of their limited quantitative evaluation. Further, the noisy label literature frequently uses them for controlled experiments. This prevalence underscores their appropriateness for controlled benchmarking.
Let's compare potential alternatives.
Completely synthetic data: This would be 100% clean, but too simple and toy-like.
Large and highly complicated datasets like ImageNet: These contain significant levels of mislabeling (over 5%) (Northcutt et al., 2021b), hence cannot do controlled experiments.
MNIST and CIFAR-10: These are real image datasets. But they have been shown to contain almost no/little mislabeling (under 0.5%) (Northcutt al., 2021b)..
We conducted analogous experiments to the image experiments on tabular datasets, specifically, the OpenML (Vanschoren al., 2014) datasets 'cover' and Biabetes130US Note these datasets are included in a recent NeurlPS benchmark for tabular models (Grinsztajn et al., 2022). Additionally, this allows us flexibly use the H-CAT framework with any tabular dataset as long as it conforms to the OpenML structure.
Tabular data
B.3 BACKBONE MODELS
We use the following backbone models: LeNet (7-layer convolutional neural network) and ResNet-18 (18-layer deep convolutional neural network with "skip" or "residual" connections). We train the backbones with cross-entropy loss using the Adam optimizer. Our learning rate of 1e-3 and batch size of 32 are kept fixed as they showed good convergence. As discussed, we explore different training epochs of 10 and 20 and show consistency irrespective, in Appendix D. This is reasonable as has been shown by (Seedat et al., 2022a) that the learning captured by HCMs (especially in training dynamics) happens in the early epochs. Note, fix the backbone models in line with principles of Data-Cena Centric Al of systematically assessing the data for fixed ML model (Liang et al., 2022; Seedat al., 2023b; Whang al., 2023; Ng et al., 2021; Jarrahi al., 2022).
23

---

Published as a conference paper at ICLR 2024
B.4 HCMS
We outline the implementation of the 13 HCMs supported by H-CAT below. We have described the HCMs in Appendix A.
1. Learning-based (Margin)
AUM (Pleiss et al., 2020) Our implementation is based on 3.
2. Learning-based (Uncertainty)
Data-IQ (Seedat et al., 2022a) Our implementation is based on 4.
Data Maps (Swayamdipta et al., 2020)
Our implementation based on
3. Learning-based (Loss)
Large Loss (Xia et al., 2021) Our implementation computes the loss per sample individually.
Learning-based (Gradient)
GraNd (Paul et al., 2021) Our implementation is based the Pytorch implementation from which is based on the original JAX 7
VoG (Agarwal al., 2022)
Our implementation is based
S.L.erring-based (Forgetting)
Forgetting scores (Toneva et al., 2019) Our implementation based on the original paper tracks the learning and forgetting times per sample in order to compute the forgetting score.
6. Learning-based (Statistics)
EL,2N (Paul et al., 2021) Our implementation is based on the Pytorch implementation from which is based on the original JAX
Noise detector (Jia et al., 2022) Our implementation based on We use the noise detector training scripts in the repo
7. Distance-based
Prototypicality (Sorscher al., 2022) Our implementation follows the original paper and computes prototypicality as follows. We embed the image as low-dimensional embedding using the backbone network. We then compute the Cosine distances to the mean embedding of the specific label for that sample. The cosine distances being smaller are more prototypical.
Statistical-based
} http:://gthhu.ccm//aapprrssseeee
https///ghhbb.ommmccccttaabbbba datadict
hhttss///thhbbcccmmmmnsseeeeaaaa diet
hhuts://gihhub.com//chhng11///
10 http:://gtthu..co/BBlacccCCCCCC
_t datadict
24

---

Published as a conference paper at ICLR 2024
Cleanlab (Northeutt et al., 2021a)
Our implementation is based on
ALLSH (Zhang et al., 2022) Our implementation follows the original paper. We compute the KL divergence the model predictive probabilities for the pairs containing: (1) original input and (2) augmentation of the input. We perform the augmentation using AugLY 13 The augmentations are selected from Horizontal flip, Random brightness, Random Noise and Random Pixelization.
Agreement (Carlini et al., 2019) Our implementation of confidence agreement produces multiple predictions using MC-Dropout to create 10 pseudo-ensemble, We then compute the mean of the confidences measure of agreement, per the original paper.
B.5 IMPLEMENTATION DETAILS
We outline the experimental implementation details next. Details on the H-CAT and its usage can be found in Appendix C.
Hardness detection & Ranking. In this experiment, set a different random seed for each of the three runs. For each run, we resample the dataset and change the samples that we perturb (i.e. which samples are hard). We also repeat the experiments for multiple proportions p.
We execute all experiments with 'p" values of [0.1, 0.2, 0.3, 0.4, 0.5], except for Far-OoD and Atypical experiments, where 'p' values are restricted to [0.05, 0.1, 0.15, 0.2, 0.25]. The rationale behind this distinction is that, in reality, Far-OoD and Atypical conditions do not typically occur in high proportions. For instance, sample can't 't be considered Atypical if it constitutes 50% of occurrences. Therefore, for these conditions, we limit 'p" to a maximum of 0.25.
In this experiment, we maintain consistent random data split and the of perturbed samples, which represent the difficult examples, is also kept constant across different runs. The element of randomness is brought in solely through the training of the backbone model and by extension, the characterization of the HCM. i.e. we have different seed for the model. This approach taken because our aim to evaluate the Stability/Connisteneyy of the HCM across multiple runs when presented with the same data.
Stability/Consssttnnyy
25

---

Published as a conference paper at ICLR 2024
C
USING H-CAT
This appendix discusses how to use the H-CAT framework. We link to the codebase, provide tutorial on using H-CAT (see Sec. C.2), and also show how H-CAT can be extended new HCMs and datasets (see Sec. C.3).
C.I H-CAT BENCHMARKING FRAMEWORK REPOSITORY
To foster case of use and engagement, we release the H-CAT Benchmarking Framework's ' codebase with Apache 2.0 License. It can be found at: https: github. com/ weeeattnaeeel/--ee
C.2 TUTORIAL ON USING H-CAT
Below, we illustrate a tutorial demonstrating how to use H-CAT, highlighting the composable nature and the object-oriented interface making it easy use. We also provide this in the form of Jupyter notebook called tutorial.ipynb.
From trainer import FyTorchTrainer
from sre.dataloader import MutiiForataateiiaeer
3 From src.models import
from src.evaluator import Evaluator
Listing 1: Step 1: Necessary imports
hardness "uniform"
p=0.1
dataset "mnist"
model_namc "lenet"
epochs 20
seed
Defined by prior or domain knowledge if haroness ==""nstance";
if cataset == "mnist":
11
rule_mate maxe_matrix
13
1: [7],
13
2: [7],
3: [8],
4: [4],
16
5: [6],
17
6: [5],
18
7: [1, 21,
8: [3],
20
[7],
21
0: [0]
22
if dataset - "cifar":
25
rule matrix
26
0: 121,
airplane (unchanged)
27
1: [9], autcmobile -> truck
2: [9],
bird (unchanged) automobile A deer (unchanged)
[5],
30
4: 71,
31
5: [3, 4],
# dog -> cat frog (unchanged)
6: [6],
[5],
horse -> dog
34
[7],
ship (unchanged)
35
9: 19],
truck -> horse
36
26

---

Published as a conference paper at ICLR 2024
rule matrix
None
Listing 2: Step 2: Define experimental parameters
charactoriaation
"aum", "data_uncert", for both Data-IQ and Data-Maps "e12n",
"grand",
"cleanlab",
"forgetting",
"vog",
"prototypicalityy"
"allsh",
11
"loss", "conf_agree",
12
13
"detector"
Listing 3: Step 3: Define HCMs evaluate if excluded we evaluate all
import terch
import torch.nn
import torch. optim as optim
from torchvision import datasets, transforms
if dataset -- 'cifar':
Define transforms for the dataset
transform cooussforms.Cmppseeele.
transforms transforms.Totenserr (),
10
Ne (0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
12
13
Load the CIFAR-10 dataset
14
train_dataset CIFASS CFFF (root=", /data", train=True, download-
True, transform-transforr)) test_dataset ceittasets.CFAALLL (root=", /data", train=False, dod- True, transform-transfomm)
16
elff cataset =='mnist'
Define transforms for the dataset
19
transform transforms Compose
20
T ToTensor (),
21
transforms Moansforms.Noralll Normalize (0.5,), (0.5,))3)
24 Load the MNIST dataset train_dataset datasets. troot="./data"" train=True, download=
25
True, transform=t.rnsffomml test_dataset = datasets MNIST (rcot="./datt train=False download-
True, transform-transffrmm
27
28
total samples Len (train dataset) classes 10
32 Set device to use
33 device torch. device ""cuda" if tcrch. cuda. is_available () else "cpu")
Listing 4: Step 4: Load datasets
27

---

Published as a conference paper at ICLR 2024
Allows importing data in multiple formats
dataloader_class MulliForraaDDttaacccddeer baaaaaaceeettttttttttaaaaadttttt
target_columm-Nonn
tota_ttpe="oorch_dataeee
batch_size==44
shuffle=True,
num_workerg=0,
transform=None,
image e
perturaatiioummtbhchhahhhheee p-P, rule_matrix-rul__ea
12
13
16
17 dataloader, datallader_unhhuffldd dataloader_claso lellass.get_datalooeeer ()
18
flag_ids = atwpaadt____elsseee___aaaaoooooo
Listing 5: Step 5: Call H-CAT Dataloader Module (with Hardness module parameterized)
Instantiate the neural network and optimizer
if dataset == 'cifar';
if model_nane == 'LeNet':
model LeNet (num classes=10) to (device)
model_name
RResNet'
model ResNet18 () to (device)
elif dataset -- m
if model_ _ane == 'LeNet'
model LeNetMNIST (num_claaa classes=10) .to (device)
10
if model_name name 'ResNet's
11
model RcsNet18MNIST to (device)
12
criterion roosooot
optimizer optim. Adam (model (arame..arramttrrss))) 1r=0.001)
15
16
17 Instantiate the PyTorchTrainer class
trainer PyTorchTrainer (model=model,
19
criterionscriterion,
20
optimileer=optimiizet,,
21
1r-0.001,
epochs-epochs,
24
num characterizatiion_mehhoe
num_classesmmun_classss,
25
characterizationnnttbooe device=device)
27
28 Train the model
29
trainer. fit (dataloades @attaloader_unshoffies
hardness dict trainer. get hardness methods
Listing 6: Step 6: Call H-CAT Trainer Module (with HCM module parameterized)
eval = Evaluator hbardiess_dicct-hardeesss
flag_ids=flag p-p)
eval_diet raw_scores_diatt eval eval.compute_resule
print (eval_dict))
Listing 7: Step 7: Call H-CAT Evaluator Module
28

---

Published as a conference paper at ICLR 2024
C.3 EXTENDING H-CAT
C.3.1 ADDING A NEW HCM
New HCMs can easily be added to H-CAT. We describe the process below.
We take an object-oriented approach and hence new HCMs can be included by inheriting from our base class Hardness Base defined below.
Base class for BCMs class Hardness_Base:
def -iiiint__ (self, name): nale name self self._scores = None
def updates
pass
def compute__corresselll (self)
pass
12
13
@property
def scores(self):
if self. scores is None: self. self.compee scores ()
16
17
return self._score
Listing 8: HCM Base class
To create new HCM, you would create a new class that inherits from Hardness Base. In this new subclass, override the updates and compute scores methods with the specific logic for this new hardness calculation method. We show an example below.
class (yXewHCM(Hardnee BBaseadddddssassa
def ieiiiintelett_ name):
super ().ert)._nnttt iet.__nni___a_____
def updates (self):
$ logic to perform updates goes here pass
def compute_ccores (self):
10
Logic to compute scores goes here
scores = "some computed value" Replace this with your
computation
Listing 9: Example New HCM
The HCM then needs to be added to trainerpy in the correct part of the training loop and can be used as below
hcm
MyNewHCM some name')
hom. updates () print (hem. scores)
Listing 10: Usage of the new defined HCM
Note, one needs to think for the HCM if it gets updated every epoch or at the end of training to define placement the training loop.
29

---

Published as a conference paper at ICLR 2024
C.3.2 USING DIFFERENT DATASET
We use different datasets with H-CAT quite simply.
Before delving further, we show below where the new dataset needs to be passed to the Dataloader module. i.e. We need to pass train rain_datasel.
Allows importing data in multiple formats
estaloader_caase = MultiFormatDataaoozeer (data=train datasst,
target_ Barget__columm=None,,
atta_typo=ttorcchaatteeeee
batch_sizo=64,
shuffle=True, num_workers-0,
transform-None,
10
11
12
p=p, rule mwwriairuteeeeeeaeeeearaaawweeee _attii
Listing 11: Example of passing data to the Dataloader module
The easiest way is to define your dataset a standard torch dataset outside of H-CAT and simply to pass that into the MuliFFormuiDatllaoudd and specify a lurget_colummn-None
If one has the data as NumPy arrays torch lensors, would need pass tuple data (X.y) and specify a target_column.
Other formats are also possible with H-CAT. For instance, if you have images in folders. One could pass data as a CSV file, JSON, or Python dictionary - wherein there is column with paths to each sample and a column with the target. One can then pass the path the CSV file JSON file or pass the Python dict itself. In addition, one should specify the cargel_colunne The rest is handled internlly within H-CAT to convert the data an appropriate Torch dataset for training.
That said, ideally if the dataset can be converted to Torch dataset before passing it to H-CAT that is preferred.
30

---

Published as a conference paper at ICLR 2024
D ADDITIONAL EXPERIMENTS
This appendix presents additional experiments and full results across all setups using H-CAT to evaluate different HCMs.
D.I MULTIPLE HARDNESS TYPES SIMULTANEOUSLLY
Overview. As discussed in the main paper, we primarily focussed on single-source hardness. We might also expect cases where multiple hardness types manifest simultancously. For instance, an image is Near-OoD (e.g. blurry different texture), might be more likely also to mislabel it. As demonstration of the usage of H-CAT for this purpose, we hence assess the case where both Near-Oo OoD (Covariate and Domain) paired with Instance-specific mislabeling.
We want to understand the effect of this on HCM capabilities.
Results. We show results for both Near-OoD (Covariate) + Instance and Near-OoD (Domain) + Instance below in Figure 8
Proportin papprtennrruueee penunedd
Proprtion peruubed
0.1
0.1
0.2
0.2
0.81
0.8
0.6
0.6
0.5
0.5
pord
grand
vay
uum
dhald
etter
Aldii
Ckrenta agr
Bun
dhald
ettir
Ciiental
(a) Near-OoD (Covariate) Instance
(b) Near-OoD (Domain) Instance
Scores
Pank
alsh
amn
dleantan
dabii
deamara
eish
rorger
prito vou
alch
amm
dreantab
dabii
deamapa
rorges
pri
voug
carl
cont
(c) Near-OoD (Covariate) Instance
(d) Near-OoD (Domain) Instance
5
10
cleanaap 12
protelypoatl
defertor
(e) Near-OoD (Covariate) Instance
Near-OoD (Domain) Instance
Figure 8: Results using H-CAT for multiple hardness types manifesting simultaneously.
Takeaway. Our results indicate that H-CAT can be used to assess multiple hardness types, which manifest simultancously. We find that HCMs which perform well individually on both types of hardness - still perform well in the simultaneous setting.
31

---

Published as a conference paper at ICLR 2024
D.2 STABLITY/CONSISTENCY BACKBONE MODEL TYPE
Overview. We assessed HCMs using two different backbone models LeNet and ResNet-18 18. We wish assess the stabiliyy/coniisennyy of the backbone model. As discussed in the main paper, can quantify this based on the Spearman rank correlation - i.e., are the HCMs ordering the samples consistently in stable manner?
In the main paper, we assessed, within model stability/consistennyy the randomness of run. However, here we assess the rank correlation of models head-to-head for the same hardness type.
Results. Figure 9 shows the results across multiple hardness types.
Spanmnn corcatton
1.00
Speamar coreeation
1:00
1.00
0.75
0.75
0.75
0.50
0.50
0.25
(a) Uniform
(b) Asymmetric
(c) Adjacent
Spearran comrbzin
1.00
Spearmon correttton
1.00
1.00
0.75
0.75
0.75
0.0
0.50
0.50
025-
Oet LbGre Oheer dubu
Lidres Suntete
Dut Grt Uhikker
L A but
La Shate
Der LB Gree Uoline decomee
(d) Instance
(c) Near-OoD (Covariate)
(f) Near-OoD (Domain)
Spearman seressie
1.00
Spesemnn cereartin
1.00
1.00
0.75
0.75
0.50
0.50
(g) Far-OoD)
(h) Atypical (Shift)
(i) Atypical (Zoom)
Figure 9: Stability/consistency model types: higher Spearman rank correlations indicate greater consistency and stability.
Takeaway. Of course, the LeNet and ResNet-18 are significantly different in model type and size. However, we find that compared to the results in the main paper, we see minimal degradation in the Spearman rank correlation. This indicates that the backbone model do the hardness characterization is minimally impacted by the backbone model. i.e. those HCMs which were the most stable and consistent remain the most stable and consistent.
32

---

Published as a conference paper at ICLR 2024
D.3 STABLLITY/CONSISTENCY PARAMETERIZATIONS
Overview. We can train the backbone models differently, with different parameterizations. For instance, with different numbers of epochs, 10 20. We wish to assess stability/consistencyy with respect to different training parameterizations when performing the HCM evaluation. As discussed in the main paper, we can quantify this based on the Spearman rank correlation i.c. are the HCMs ordering the samples consistently in stable manner?
We make head-to-head comparison of scores obtained from the HCMs applied backbone models trained for 10 and 20 epochs. Ideally, we still would want a high Spearman correlation indicating sample order is maintained.
Results. Figure 10 shows the results across multiple hardness types.
Spearman concbtiion
1.0
Spearman conraaaioon
1.0
1.0
0.8
0.8
0.6-
0.4
desssia
dan
Uctred Ueiteer tho
Lidits Sutstel
thdee thtter LBoute Sitt
tee Leove Looper Dutt US Statated
(a) Uniform
(b) Asymmetric
(c) Adjacent
Spcaman conelation
1.0
Spcorm correaatio
1.0
0.8
0.0
0.50
D. Meooaa MACRne Laid Leatas Maiiol
Li Loctbe Di bias
Dit Lete aatter
trais stalanad
(d) Instance
(e) Near-OoD (Covariate)
(f) Near-OoD (Domain)
Spcaman conciation
1.00
Spearmn coreeatioon
1.00
1.00
0.75
0.75
0.25
0.50
0.50
0.50
025
(g) Far-OoD
(h) Atypical (Shift)
(i) Atypical (Zoom)
Figure 10: Stability/consistencyy to parameterizations: higher Spearman rank correlations indicate greater consistency and stability.
Takeaway. We investigated the stability/consistency of the model parameterization in terms of the number of epochs used to train the backbone models. We find the Spearman correlations are not as affected by the epoch number as potentially expected. We see the magnitudes of the Spearman correlation for the different HCMs are similar to that of the main paper. Moreover, the HCMs, which were the most stable and consistent, remain the most stable and consistent.
This result is understandable, it shown in (Seedat et al., 2022a) that the majority of variation comes from the early epochs (before epochs), hence after the training dynamics on which most of the metrics are computed stabilize. This means that it won't 't affect we continue training the backbone till 20 epochs. This result simply mirrors this result from the literature and highlights for benchmarking purposes that don't 't need train the backbone model for too many epochs in order to have a good assessment of the HCMs capabilities.
HCM characterization whether
33

---

Published as a conference paper at ICLR 2024
D.4 ADDITIONAL AGGREGATED RESULTS
Overview. In the main paper, we presented aggregate results for 6 out of 9 hardness manifestations types. We present the results for the remaining three Adjacent mislabeling, Near-OOD (Domain) and Atypical (Zoom), below in Figures 11-14.
1.0
1.0
03
0.8
8.0
0:1
0.6
0.6
proto grand
voa
Bist
assai
deca decraat
detentoo
cost agrior
0.5
Proto
vog forger bist
stont
dicamaayy diennte
desumtaa
025
(a) Adjacent
(b) Near- OOD (Domain)
(c) Atypical (Zoom)
Figure 1I: AUPRC for different HCMs for different hardness types aggregated across setups. We
vary the proportion perturbed. iee. proportion of hard examples. Blue is better, red worse. We see the variability of HCM capabilities across hardness types and proportions.
Park Sconss
(a) Adjacent
(b) Near-OO OOD (Domain)
(c) Atypical (Zoom)
Figure 12: Performance rankings of HCMs vary depending on the hardness type.
10
decit
diac
dacds
FAST
cved
protetyccaa
(a) Adjacent
(b) Ncar-OOD (Domain)
(c) Atypical (Zoom)
Figure 13: Critical difference diagrams highlight that similar categories/classes of HCMs do have a statistically significant difference their performance, indicated by the horizontal black lines
linking HCMs which are not statistically different. The numbers in brackets denote mean rank.
Sparman comclaiin
Spermmnn cometation
2.8
0.8
0.6-
3.6
0.6
(a) Adjacent
(b) Near-OOD (Domain)
(c) Atypical (Zoom)
Figure 14: Certain classes of HCMs are more stable and consistent than others (retaining rank order), with higher Spearman rank correlations for multiple runs of the HCM the same data
34

---

Published as a conference paper at ICLR 2024
D.5
INDIVIDUAL RESULTS IMAGES
D.5.1 MISLABELING: ADIACENT
Heatmaps.
1.0
Proportion perturbed
0.1
10
0.1
0.2
0.8
0.3
Proportion
0.6
0.4
0.5
proto
grand
vig
torget
Barard
Satamats
detecto
eaan
alels
ckeaniat
cont.9g
grand
vog
lorget
loss
aum
dataig
datamaps
detecio
e22n
alton-
cleantab
cont Ag)
Ds: Uedrre Uiither
ouguut
Statsarca
1KCrod L8.i
LEsus
Statsticl
(a) MNIST LeNet
(b) MNIST ResNet
Propartinn perturbed
1.0
0.3
0.2
0.2
0.8
Proportion
0.3
0.6
0.6
0.4
0.5
proto
grand
veg
forae
sum
datadd
datamaas
detecior
e2nn
sibl
Cleantab
cont 90)
grand
vag
forget
losa
aum
dataiy
datamapss
detector
e22n
allot'
diantabi
cont ag
Dic LtCree Lh:Sther
LB ouput
dub
Stataticd
LECred
L2:0ta
Locteb
Statatial
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 15: Adjacent mislabeling AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse
Proportion perturhed
0.2
0.8
0.3
Poportion
0.6
0.5
0.6
0.5
proto
grand
vog
aum
dataig
Le dittanass
detector
elzn
slish
dhanlab
cant sgr
grand
vog
forget
losa
dataig
detamaps
detector
e2zn
alluh
cleantah
cont agy
Statiaica
1etndd
Statstaal
(a) MNIST LeNet
(b) MNIST ResNet
Progarton partu Raad
parturbeed
0.8
0.2
0.3
Proportion
0.7
0.6
0.5
0.5
0.5
proto
grand
vag
forget
lass
aum
dataig
Batamans
detector
e2n
atsh
deantab
cont.agr
grand
lorget
loss
sum
dataig
datamass
detector
e/2n
alts
cleantah
cont. agg
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 16: Adjacent mislabeling - AUROC. We vary the proportion perturbed. i.e. proportion of
hard examples. Blue is better, red worse
35

---

Published as a conference paper at ICLR 2024
Rankings.
7
Rank
Rank
with
claaiee
candt 3op
daraig
detemspy
discertor
Forper
Koad
alth
axm
chaaiab
cont. ag9
datall
detemides
disaccor
farget
grand
(a) MNIST LeNet
(b) MNIST ResNet
9
dleaeitt
cant Kaaa
daramapa
detectev
elsh
rogger
grand
Provo
aum
ciantaa
cont apr
ditail
datamaro
detircoor
rorget
mrnnd
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 17: Adjacent mislabeling AUPRC. Ranking of HCMs.
adjacent roc
adjacent roc
Scores
13
allsh
aum
cleanta
datamops
derector, gg
lorgening
grand
protosty picaltg
loss
alish
aum
cleantah
cont. agrse
datamaps detector
eizn
forgetting
grand
less
vog
cont
(a) MNIST LeNet
(b) MNIST ResNet
adjacent roc
adjacent roc
Scores
allsh
aum
cleantah
daramaps cont.
derectosn
logettng
loss
prototypicalty
alish
aum
cleaniabi
agresin
datamaps
detectot
eizn
lorgetting grand
less
vog
CD diagrams.
cond
10
10
dataia
loss LIgh
dalaig aUm
(12) el2n grand lorgeting
proschypesil Cear
cleate conf,
(a) MNIST LeNet
(b) MNIST ResNet
10
dataig (1 aum
elzn dutactor
TOES agres
protctes
conf agrce
forgetting
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 19: Adjacent mislabeling AUPRC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
36

---

Published as a conference paper at ICLR 2024
10
10
dataig protetypic cieania0
Polector
plish
agree
eizn 13
D dctoctor
Lagre
dia nd
cleantab
2 vog
MNIST LoNet
(b) MNIST ResNet
10
datak 8.
alsh
datamaps
OO) forgstting
dataminn
loss
R] vag
(C) CIFAR LeNet
(d) CIFAR ResNett
Figure 20: Adjacent mislabeling AUROC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
Matrix compare
2.0
1.0
mmm
0.8
clenniab
0.8
ag
cont.agr
5.6
00ta/0
0.6
dolector
forget
d2n
0.4
c2n
0.4
grand
forget
loss
0-2
grand
0.2
proto
vog
mmm
Card bg
detsnopp
V90
0.0
0.0
H2N
(a) MNIST LeNet
(b) MNIST ResNet
alch
alsh
sum
cleariat
conl
cont.agr agr
delais
datalq
dutector
detector
farget
e2n
0.2
grand
0.2
loss
corl oot
000mp9
v00
0.0
coambb
cont 20r
v00
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 21: Adjacent mislabeling. Head-to-head comparison matrix assessing for runs when a certain
HCM outperforms another. i.e. when y-axis (HCM 1) beats x-axis competitor (HCM 2).
37

---

Published as a conference paper at ICLR 2024
D.5.2 MISLABELING: ASYMMETRIC Heatmaps.
Proportion parturbed
1.0
0.2
0.8
0.8
Proportion
0.6
0.4
0.5
0.5
0.4
proto
grand
vop
forgur
losp
aunr
dataly
dstamare
detector
ela
alsii
Clantab
ag
grand
lorget
aum
datag
datamaps
detector
e
allar
chent atr
Cont
(a) MNIST LeNet
(h) MNIST ResNet
Propertion gurturbed
1.0
0.1
0.2
0.8
0.8
Proportion
0.3
0.6
0.4
0.4
0.5
0.5
0.4
proto
grand
vog
torper
losg
aun
dataid
datamans
detector
e
albi-
daanta
grand
vog
toget
loss
Bum
dataig
datamaps
detector
ezzv
afatt
dicontal
cont ag
cont
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 22: Asymmetric mislabeling AUPRC. We vary the proportion perturbed. i.e. proportion of
hard examples. Blue is better, red worse.
Proportion parturbed
1.0
0.1
0.2
0.2
0.8
0.8
Proportion
0.3
0.4
0.4
0.5
proto
grand
vag
forgas
Ianis
aum
dataig
Edatamaps
detedior
e0nn
albji
claanaa conf. agr
grand
vog
lorget
aum
dataig
datamaps
detedor
e2n
altsh
dleataah
ront agr
Dtteee
Sais
LKGrd 10:Cther
Cutbut
Statetical
(a) MNIST LeNet
(b) MNIST ResNet
Proportion perturbed
0.9
0.1
0.2
0.8
0.2
0.3
0.7
0.4
0.6
0.4
0.5
0.5
proto
grand
vog
forgat
less
aum
dataiq
idtaamapp
detector
enn
albn
chandab
cont. .gg
grand
vog
forget
lose
Bum
dataiq
datamape
Setector
e2n
adsh
cleantal
agr
(B.5me 1n.0teer
Suie
[KGMd (8:0/
18 Sa
Statetial
(c) CIFAR LeNet
CIFAR ResNet
Figure 23: Asymmetric mislabeling.. AUROC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
38

---

Published as a conference paper at ICLR 2024
Rankings.
3
7
9
Midi
oun
createas
card K.ga
davag
datemaro
detactor
eltn
regger
geranca
looel
Provo
mith
aum
diantad
cont apr
daumaja
denaccoo
etth
rarger
mrand
Yead
(a) MNIST LeNet
(b) MNIST ResNet
a466
creantan
cant. Jun
dbtagg
dedemara
derector
torget
grand
loud
proi
althi
aom
cent.
cent.gge
datail
datemari
defeecto
esf
forgee
prand
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 24: Asymmetric mislabeling AUPRC. Ranking of HCMs.
asymmetric roc
asymmetric.ro roc
allsh
cleanisb
cont. diibig
datamaps
forgettng
grand
prototypicalli
alts
aum
cleanlab
dataid
datamaps detecter
et2n
forgetting grand
inss
vog
cont.
(a) MNIST LoNet
(b) MNIST ResNet
asymmetric roc
asymmetric roc
13
allsh aum
cleantab
daramops defecior dlan
forgenting grated loss
protoryplcaltt
allsh
aum
cleaniahi
datamass cont.
datula
destecto
elzn
forgetting
grand
lnss
veg
cont.
CD diagrams.
10
datamraps
CURS Correttnng
cont.
10
vog
MNIST LeNet
(b) MNIST ResNet
10
AIDO ROREE agree
037310
crototystc
forg Cergeliing
conf. agrce
(c) CIFAR LeNet
(d) CIFAR ResNett
Figure 26: Asymmetric mislabeling AUPRC. Critical difference diagrams. Black lines connect methods that not statistically significantly different.
39

---

Published as a conference paper at ICLR 2024
10
streetting ONIee
dataid Sum
forgeitting
CODY agree grand
prococyply REn
cicaniao
deaninb
(a) MNIST LeNet
(b) MNIST Ret
10
10
pgree BeRRiin prototypicaity
dataig
forghtingg
datamig
clean
cleaniãD
grand
(c) CIFAR LeNet
(d) CIFAR -eF
Figure 27: Asymmetric mislabeling AUROC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
Matrix compare
1.0
1.0
alke
mum
claaraab
clenni4b
cmn
005210
datamaps
forget
grand
forget
loss
grand
proto
vog
vop
cold ap
Galar
oaammpp
octoctte
coonbo
001 30
gra
HOM
HEW
(a) MNIST LeNet
(b) MNIST ResNet
1.0
1.0
aum
0.8
cleanisb
0.8
20
cont_agr
detnis
0.6
dutalq
0.5
datamaps
delector
0-4
e2n
0.4
farget
grand
fonget
Ioss
0.2
proto
vog
mmm
000
0.0
deantb
100
HON
HCM
(c) CIFAR LeNet
(d) CIFAR RosNet
Figure 28: Asymmetric mislabeling. Head-to-head comparison matrix assessing runs when certain HCM outperforms another. i.e. when y-axis (HCM 1) beats x-axis competitor (HCM 2).
40

---

Published as a conference paper at ICLR 2024
D.5.3 MISLABELING: INSTANCE
Heatmaps.
Proportion perturbed
0.2
0.8
0.3
Proportion
0.6
0.5
0.5
0.4
Prote
grand
veg
toger
less
aun
darald
ditamaps
detector
ein
atsif
cleania
cant sg
grand
vog
torget
losg
dntaiy
datamaps
detectes
e/2n
alt
glantad
cont aon
Statiaica
Statstial
(a) MNIST LeNet
(h) MNIST ResNet
Progarton parturaa Rard
0.2
0.9
0.3
Proportin
0.6
0.5
0.5
proto
grand
vag
forgus
less
aum
dataia
détamaps
detector
el2n
albni
clanntab
cont. [gg
grand
vog
torges
loss
sum
detelo
datamags
detector
e2n
aliat
cheadaat
cont. .ay
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 29: Instance mislabeling AUPRC. We vary the proportion perturbed. i.e. proportion hard examples. Blue is better, red worse.
Proportion partubeed
0.9
0.8
0.2
0.3
Proportion
0.7
0.4
0.6
0.4
0.6
0.5
0.5
0.5
proto
grand
vag
forgat
loss
aum
dataig
detector
e22n
sibh
cleantab
cont agr
grand
forget
lnta
aum
dataig
datamopa
eDn
allat
dieanfat
cont agg
(a) MNIST LeNet
(b) MNIST ResNet
Propertion parturbed
0.8
0.1
0.2
0.7
0.7
Proportion
0.3
0.4
0.6
0.4
0.5
0.5
Proto
grand
vog
forget
less
sum
datalq
mans
detector
nnn
clsantab
cont agr
grand
vog
Forget
sum
dataiq
daramaps
detector
e2n
atsh
cleantab
conf Agg
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 30: Instance mislabeling AUROC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
41

---

Published as a conference paper at ICLR 2024
Rankings.
7
aidh aun
clantan
cort agr
datag
datemapa
Intertor
reggee
grand
loof
Bum
crantaa
cont. agr
datail
dazaaayo
detector
etif
rangee
grand
leed
MNIST LeNet
(b) MNIST ResNet
Rank
Rank
aish
cleartan
cord aay
dabii
determaa
dereector
forpen
pro
alan
cleaniat
dataig
deammsps
dereccor
eco
forget grand
cont
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 31: Instance mislabeling AUPRC. Ranking of HCMs.
instance roc
instance roc
ellsh
aum
dlaaniab
datamaps detector
eizn
lorgering
protorypicattg
loss
alish
aum
cleantab
datamaps
detecta
ezzn
forgetting grand
inss
vag
cont.
cont.
MNIST LeNet
(b) MNIST ResNet
Instance roc
instance roc
allsh
aum
cisaniah
dataraps cont
defector 1
prototypicaitg
ailsh
aum
cleanlati
dntamaps
detectot eizn
Iargetting grand less
cont.
CD diagrams.
10
cleatii COOOO
SUm datamage
groind
agres
(a) MNIST LoNet
(b) MNIST ResNet
10
10
datail datamaps
e2tn grand SU cOYy agges
Solal
dajamee cont
4E3A (5.6)
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 33: Instance mislabeling- AUPRC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
42

---

Published as a conference paper at ICLR 2024
5
10
2.5
5.0
10.0
prototypealhy
cizn (.6)
catala
alsh
datai 12.8
11) detector
datamags grand
forgettin
vog Lagree
cleaniab
lozs (4)
(8.8) forgetting
_agre
aum (5:4)
(.22 grand
(a) MNIST LeNet
(b) MNIST ResNet
10
2.5
5.0
7.5
10.0
protocypl
Soree Wrirggeiing
dajamaps deance
e22h
derector
agrea 4) vog
loss
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 34: Instance mislabeling AUROC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
Matrix compare
aum
dearlab
0.8 cleaniab
0.8
conl_gg
cont.agr
datakg
0.6
dateig
0.5
detector
forget
#2n
0.4
c2n
0.4
grand
0.2
0.2
poto
loss
vog
eUm
dlenna
cent so
#2t
forge
greed
100
0.0
deanbb
cont 20
mand
wo
0.9
(a) MNIST LeNet
(b) MNIST ResNet
wish
dearlab
conl 401
conf. agr
d3ta/9
datamaps
dutector
detector
e2n
grand
loss
cart QUERE
wog
0.0
cesnab
cont 3gr
330 mand
N09 gron
wo
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 35: Instance mislabeling. Head-to-head comparison matrix assessing for runs when a certain
HCM outperforms another. i.c. when y-axis (HCM beats x-axis competitor (HCM 2).
43

---

Published as a conference paper at ICLR 2024
D.5.4 MISLABELING: UNIFORM
Heatmaps.
1.0
Proportion parturbed
0.1
0.1
0.2
0,8
0.2
0.
0.3
0.4
0.6
prots
grand
vol
loss
sum
daradd
datamaps
detector
e/en
aleif
ckantat
cont.9g agg
proto
grand
voy
targer
losr
aum
datamaps
aatali
detector
acn
allor cleantal:
cant agr
DS: Ledrra Unitter ouruut
Statiazca
Pe: Untred La othor
Lesaas
Staiitical
(a) MNIST LeNet
(b) MNIST ResNet
1.0
Progartion garturbed
0.1
1.00
0.2
0.8
0.75
Proportion
0.3
0.6
0.3
0.50
0.4
0.4
0.25
0.5
Proto
grand
vog
forger
long
aum
dntaig
datamawo
detector
e0n
elln
dhanlab
ag)
grand
vay
forget
lots
aunf
datadd
datamaps
detector
el
aili
deantal
ag
LB:0thrr
ouputt
33b Statiaical
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 36: Uniform mislabeling AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
0.9
Proportion perturbed
0.8
0.8
Proportion
0.4
0.5
0.5
0.5
Proto
grand
vag
torget
loss
aum
datalg
Gd datamaps
detector
e22n
alsh
dleaniab
cont. agr
Proto
grand
vog
forget
loes
sum
dataig
dalamapa
detector
eCn
akch
deaniab
cont. agr
Unstteer
stus
15.00ha
Lestas
(a) MNIST LeNet
(b) MNIST ResNet
Proparion perturbed
0.9
0.8
0.2
Proportion
0.4
0.7
0.4
0.6
0.5
0.5
0.5
prato
grand
vog
forgat
Iosg
aum
dataid
datamaus
detector
elnn
alsh
diantab
cont agr
grand
torget
lass
Bum
dataiq
datamups
detector
e2zn
alihh
chantaab
cont. agr
Stare
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 37: Uniform mislabeling AUROC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
44

---

Published as a conference paper at ICLR 2024
Rankings.
3
7
11-
wids
deerrian
cant Lgy
datal
dattenago
wish
Borger
Food
Probo
wish
auni
cant agy
datala
datterago
deteeroo
wlsn
torget
grand
reed
prodo
(a) MNIST LeNet
(b) MNIST -
daan
cleantad
cant. agr
datain
datemoges
deserrce
aldn
forges
grandl
last
allen
creaniad
cont agr
datag
datamopei disteroor econ
Aogert grand les
(c) CIFAR LeNet
CIFAR ResNet
Figure 38: Uniform mislabeling AUPRC. Ranking of HCMs.
uniform roc
uniform,
Rank Scores
altsh
aum
cleantab
data dataraps
defectos
forgetting grand
prototypccltg
allsh aum cleantab
conf. WiSf
Ratamaps dete
torgegiiga
prototypicang)
loss
cont.
(a) MNIST LeNet
(b) MNIST ResNet
uniform
uniform roc
11
13
allsh
aum
cleanish
deramaps cont.
lorgettig
protottplcalttg
loss
allsh
sum
cleantati
datala
datamaps detector elzn
forgetting grand loss
veg
CD diagrams.
cont.
10
Josp grend
prote dacamise
de
grans
cont ao
conT agree
(a) MNIST LoNet
(b) MNIST ResNet
10
5
10
33 IoAe PedeCttr Conreerre
3Um 1L dat
forgetting PiRlactor
prototyskia V0A
IIS agres PIZI
forgen
vog
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 40: Uniform mislabeling AUPRC, Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
45

---

Published as a conference paper at ICLR 2024
update uniform, roc
5
10
10
forgetiig grand
prototypicg 120
ccon
BRand
cicanla
protatypica
(a) MNIST LeNet
(b) MNIST ResNet
10
10
3UT
Tergetting Blarand
dstamags cleanisn
clean
prototypica
(c) CIFAR LeNet
(d) CIFAR -ResNet
Figure 41: Uniform mislabeling AUROC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
Matrix compare
1.0
1.0
dbartab
9.8
cleaniab
0.6
datamaps
farget
fongel
grans
grand
loss
loss
proto
prote
vog
vel
HON
HCM
(a) MNIST LeNet
(b) MNIST ResNet
mum
0.8
cleanab
0.8
agr
cont.agr
1btomaps
datak
0.6
00t3/0
0.6
datector
farget
d2n
0-4
c2n
0.4
grand
fonget
l8ss
0.2
0.2
proto
vog
emm
0.0
prndd
wo
0.0
HON
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 42: Uniform mislabeling. Head-to-head comparison matrix assessing for runs when a certain
HCM outperforms another. i.c. when y-axis (HCM 1) beats x-axis competitor (HCM 2).
46

---

Published as a conference paper at ICLR 2024
D.5.5
NEAR OOD: COVARIATE
Heatmaps.
Proportion perturbed
0.2
0.8
0.3
Proportion
0.5
0.4
0.5
0.4
Prote
grand
veg
togerr
less
aun
darald
ditamaps
dstector
ein
atsff
cleania
cantt sgr
grand
vog
torget
losg
dntaiy
datamaps
detectes
e2nn
alt
glantad
cont con
Statiaica
Statstial
(a) MNIST LeNet
(h) MNIST ResNet
Progarton parturaa Raad
0.2
0.8
0.8
0.3
Proportin
0.6
0.5
0.5
0.4
proto
grand
vog
forgut
less
aum
dataig
détamaps
detecto
el2n
albni
clansiab
cont. [gg
grand
vog
torges
loss
sum
datelo
datamaps
detector
e2n
aliat
cheadaat
cont. .ay
Statiticl
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 43: Near OOD: Covariate AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
0.9
Proporion perturbed
0.70
0.1
0.2
0.8
0.2
0.65
Proportion
0.3
0.7
0.3
0.60
0.6
0.55
0.5
0.5
0.50
proto
grand
vog
fogat
lost
aum
dataig
etatamaps
detertor
ei2n
atsh
cuannab
Lagr
grand
vog
farget
lose
bum
dataig
tammap
detector
el2n
alsh
deaniab
cont. agr
Pa: .2:Cre LB:0ther
1Rdra
Par eral
(a) MNIST LeNet
(b) MNIST ResNet
Proportion parturbed
0.8
0.9
0.1
0.1
0.2
0.3
0.3
0.6
0.4
0.5
0.5
0.5
proto
grand
vog
forget
lass
dataiq
Ratamaass
detestor
eizn
alln
deartab
conf.ag Lagr
grand
vog
forget
hosd
aum
dataiq
datamapi
detector
allih
cisantah
cont. Agy
Dst (Bdrr Uiitteer
stb
Staiaica
LEcred 18tit
Statskill
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 44: Near OOD: Covariate- AUROC. We vary the proportion perturbed. i.c. proportion of
hard examples. Blue is better, red worse.
47

---

Published as a conference paper at ICLR 2024
Rankings.
9-
earh
card 2.3
daval
datemaapi
detectoy
elan
forger
groand
love
alin
aum
dhantad)
cont sgr
ebai
forger
grand lass
(a) MNIST LeNet
(b) MNIST ResNer
2856
creantan
card. aur
dbtagg
dedemarc
detectos
torjest
grand
loud
propp
aom
cent age dit
datamaria
detector eofi
ferger prand
(c) CIFAR LeNet
CIFAR ResNet
Figure 45: Near OOD: Covariate AUPRC. Ranking of HCMs.
ood covarlate roc
ood covarooteeeeeeee
13
alfsh
cleantsis
cont. "dstala
datamaps
detector
822n
forgentng
prototypicailty
graft
allsh
aum
cleanlab
dataig
datamans
detectos
elan
forgetting
grand
lnss
vog
cont.
(a) MNIST LoNet
(b) MNIST ResNet
ood covariate roc
ood covariate_re roc
Scores
altsh aum
cleantab
agifaig
datamas defector etznn
loggnting graned loos
protoxypicaati
allsh
aum
cleanlali
datula
datamaps
detectoo
eizn
forgetting
grand
less
veg
cont.
cont.
CD diagrams.
10
datang
aum
vop
datamapp
eizn
prototypic
detector
8.8) alish
MNIST LeNet
(b) MNIST ResNet
10
tuní Bum
VRO grand deteccor
Aatala alsh
prototspicall
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 47: Near OOD: Covariate AUPRC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
48

---

Published as a conference paper at ICLR 2024
12) detector
z0m
forgetting
datamaps
8.4 Cieanla
1055 (5:4)
6.5) eizn
(a) MNIST LeNet
(b) MNIST ResNet
5
10
10
CECONEI ZAAD 32n
cont_agree
8um
[11 forgetting
data/o datamaps
0) loss
grand
(c) CIFAR LeNet
CIFAR -RssNet
Figure 48: Near OOD: Covariate AUROC. Critical difference diagrams. Black lines connect
methods that are statistically significantly different.
Matrix compare
1.0
1.0
alke
mum
duardab
0.8
cleani4b
5.6
005210
detamaps
dotector
forget
grand
forget
loss
grand
proto
vog
vop
cont ogr
Sttsmaps
lange
ceundd
HOV
HEW
(a) MNIST LeNet
(b) MNIST ResNet
1.0
1.0
aum
0.8
cleanisb
0.8
con 4g7
cont_agr
detnis
aetamaps
0.6
dataiq
0.6
datamaps
delector
farget
0-4
e2n
0.4
grand
forget
0.2
proto
loss
vog
emm
rimentab
cord og
Quta0
6nampp
60000C0
#inn
Tony
100
0.0
dleanbbb
100
0.0
HON
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 49: Near OOD: Covariate. Head-to-head comparison matrix assessing for runs when a certain
HCM outperforms another. i.e. when y-axis (HCM 1) beats x-axis competitor (HCM 2).
49

---

Published as a conference paper at ICLR 2024
D.5.6
NEAR OOD: DOMAIN
Heatmaps.
Proportion parturbed
1.0
0.2
0.8
0.8
Proportion
0.6
0.4
0.5
0.5
0.4
proto
grand
vop
forgur
losp
auni
dataly
dstamare
detector
elan
alsii
Clantab
agr
grand
lorget
aum
datag
datamaps
detector
e
allar
clean atl
Cont agr
(a) MNIST LeNet
(h) MNIST ResNet
Propertion gurturbed
1.0
0.1
0.2
0.8
0.8
Proportion
0.3
0.4
0.6
0.4
0.5
0.4
proto
grand
vog
torper
losg
aun
dataid
datamans
detector
albi-
dlanta
0.4
grand
vig
toget
losr
Bum
dataig
datamaps
detector
ezzvi
afatt
clenntal
cont ag
cont
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 50: Near OOD: Domain AUPRC. We vary the proportion perturbed. i.e. proportion hard examples. Blue is better, red worse.
Proportion parturbed
1.0
0.1
0.2
0.2
0.8
0.8
Proportion
0.3
0.4
0.4
0.5
proto
grand
vog
forgas
Ianis
aum
dataig
Edatamaps
detedior
e0nn
albyi
claanaa conf. agr
grand
vog
lorget
Bum
dataig
datamaps
detecoo
e2n
altsh
dleataah
ront agr
Lttttee
LRCrd 10:Cther
Statetical
(a) MNIST LeNet
(b) MNIST ResNet
Proportion perturbed
0.9
0.1
0.2
0.8
0.2
Proportion
0.3
0.7
0.4
0.6
0.4
0.5
0.5
proto
grand
vog
forgat
less
aum
dstaiq
ddtaamapp
detector
enn
albn
chandab
cont. .gg
grand
vog
forget
lose
Bum
dataiq
datamape
Setector
e/2
adsh
cleantal
agr
(B.5me 1B.00 Beteee
Ctrpur
Suix
Saniiia
[RG (8:0w
Statetial
(c) CIFAR LeNet
CIFAR ResNet
Figure 51: Near OOD: Domain AUROC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
50

---

Published as a conference paper at ICLR 2024
Rankings.
cant apr
datalo
datamages
debertoy
forgerr
graand
almn
aum
craandd
agr
datadl
datamapo
desarooor
elan
fogeet
grand
(a) MNIST LeNet
(b) MNIST ResNet
dlsataab
cant. aup
daralg
dasemago
desartiy
eloo
foroest
grand loud
pri
alla
axmr
clantab
cont. agr
dataiy
datanago
detacror
fergert
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 52: Near OOD: Domain AUPRC. Ranking of HCMs.
semantic shift,
semantic
alfsh aum
cleantab
dacanaps
defector elan
forgening
prototyplcaltg
allsh
cleantals
dataiq
datamaps
deteecte
ezn
forgetting
grand
lnss
vog
cont.
cont.
MNIST LeNet
(b) MNIST ResNet
semantic shift roc
semantic shift roc
altch aum
cleanish
daranaps cont.
"dabain
detectogn 82nn
lorgetting
grand
prototypicaltg
loss
altsh
sum
cleantab
datatg
datamags detector
elzn
forget ting grand Inss
veg
cont
CD diagrams.
10
datal
datarragg
datala
VOO
derector
Cleatiie
derector
MNIST LeNet
(b) MNIST ResNet
10
cleanlab (D)
el2n 6I grand detector
sum
dalm
prototypical
datamops
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 54: Near OOD: Domain AUPRC. Critical difference diagrams. Black lines connect methods
that are not statistically significantly different.
51

---

Published as a conference paper at ICLR 2024
10
5
10
(221
dataie 22
blish
datamaps
forgetting
Rizh Cleanlab
(a) MNIST LeNet
(b) MNIST ResNet
Invorriccaity PAEKK ROoeeing tng
gleaniab 11 ConT
dotector SItc datarmaps grand
COnT
V3g
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 55: Near OOD: Domain AUROC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
Matrix compare
aum
dearlab
clennabb
Lgg
cont.agr
datakg
de Lempap
0.6
dateiq
0.5
HCR
delector
forget
#2n
0.4
c2n
0.4
grand
0.2
grand
0.2
poto
loss
vog
canl sar
ditamann
forge
0.0
dharta
cont 30
0.9
(a) MNIST LeNet
(b) MNIST ResNet
aizh
wish
aUT
dearlat
conl 4g1
conf. agr
d3ta/t
aatamaps
datamaps
detector
detectar
e2n
fonget
grand
loss
cart s0
datalo
mand
6etocto
wog
0.0
dinna
cont 39
(c) CIFAR LeNet.
CIFAR ResNet
Figure 56: Near OOD: Domain. Head-to-head comparison matrix assessing for runs when a certain
HCM outperforms another. i.c. when y-axis (HCM beats x-axis competitor (HCM 2).
52

---

Published as a conference paper at ICLR 2024
D.5.7
FAR OoD
Heatmaps.
Proportion parturbed
Proportion perturbed
1.0
0.05
0.8
0.1
0.8
0.6
0.25
0.4
0.25
0.4
proto
Brand
yog
forget
loss
sum
datalg
datamaa
detector
eftr
alal
dkantaai
Cont agr
grand
vog
forpet
loss
aum
dataig
datanapy
detector
allot
dleattai
Cont. ago
Der LBsos inlmmee
La
18 Shat
Seretral
1RGmd 1h:0
LB Sam
(a) MNIST LeNet
(h) MNIST ResNet
Proporton perturbed
1.0
Proportion pertubeed
1.0
0.05
0.15
0.6
0.6
0.2
proto
grand
terget
less
Bun
datai
datamaps
detector
etr
allet
cleantas
cont ogg
0.4
grand
vog
forgat
bas
aum
dalaid
Batanops
detector
elin
alloh
cleantali
cont. agr
Outsut
Lecrnd
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 57: Far OoD AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
Propartion perturbed
1.0
0.05
0.05
0.1
0.1
0.8
0.8
Proportion
0.15
0.2
0.2
0.25
0.6
0.25
proto
grand
vog
forest
loss
ditaig
Batamags
dereector
deartab
cany. agr
grand
forget
Joss
dataia
datamaps
deraedor
alish
cleantab
cont. agr
LiGel Jaoteer
Stttticl
L8Gnd
(a) MNIST LeNet
(b) MNIST ResNet
Proportion perturbed
Proportion partu parturfhadd
Radd
0.1
0.8
0.15
0.6
0.6
0.25
prote
grand
vog
forget
less
aum
dataig
sdntamops
for
eon
alish
clanttal
cont. ogr
grand
wog
forget
bist
mum
dalaiq
datamaps
deteclor
e/
allsh
deartaa
cont. agr
Dat UBirred .2:0teer
L5 Sttu
Statataal
L3:3te 15:0tbr
LB e5bb
Retakaa
(c) CIFAR LeNet
CIFAR ResNet
Figure 58: Far OoD AUROC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue better, red worse.
53

---

Published as a conference paper at ICLR 2024
Rankings.
3
earh
cant apr
datamages
debertor
forgert
grandd
almh
aum
crtasidd
agr
datadl
datamapo
desarooor
elan
fogeet
grand
losd
vog
(a) MNIST LeNet
(b) MNIST ResNer
clsantab
canl, aup
daralg
dasemage
forgest
grond
foud
prii
alla
anm
claatab
cont. ag
dataiy
datanagor
detactor
ferger prand
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 59: Far OoD AUPRC. Ranking of HCMs.
ood
far
alfsh aum
cleantab
dacanaps
defecios dlan
forgening
prototyplcaltg
allsh
cleantal
dataia
datamaps deteccer
ezn
forgetting
grand
lnss
vog
cont.
cont.
(a) MNIST LeNet
(b) MNIST ResNet
far ood
far ood roc
Rank
9
altch
aum
cleanish
aa detectogn 82n
lorgetting grand
prototypicaltg
lons
alssh
sum
cleantab
datatg
datamags detector
elzn
forget ting grand Inss
veg
cont.
cont
CD diagrams.
alish aotald dal
(12) IcEs
conf
grand
protonyarca CIN
aum
VOg
MNIST LeNet
(b) MNIST ResNet
10
ront Agree
TOEE NHD grand
datomisor alio
datamine
BUTI
(c) CIFAR LLNeet
(d) CIFAR ResNet
Figure 61: Far OoD AUPRC. Critical difference diagrams. Black lines connect methods that are not
statistically significantly different.
54

---

Published as a conference paper at ICLR 2024
10
5
10
agrs Sira
10 forgetinn
(2) detector
proto tamaee
datarmaps
forOptting gramd
cleaniab
(a) MNIST LeNet
(b) MNIST ResNet
10
cOnT
Octecior PIC grand
2121 detector forgeming grahd
Gatanco da IARERE
datamaps
dearce
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 62: Far OoD AUROC. Critical difference diagrams. Black lines connect methods that are
not statistically significantly different.
Matrix compare
1.0
1.0
aum
0.8
claanibb
0.5
detamaps
0.6
0.5
datamaps
delector
d2n
0.4
forpet
grand
fonget
los
pota
vog
0.0
0.0
cheorke
Con
HON
HCW
(a) MNIST LeNet
(b) MNIST ResNet
alsh
4um
3um
dewnlat
conl_agr
canf Agr
Satemaps
datakt
0.6
data1o
0.5
detector
HEW
torpet
d2r
0.4
#2n
0.4
grand
0.2
grand
0.2
loss
2.0
diarbo
oint 30
1000000
100
0.0
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 63: Far OoD. head-to-head comparison matrix assessing for runs when a certain HCM outperforms another. i.e. when y-axis (HCM 1) beats x-axis competitor (HCM 2).
55

---

Published as a conference paper at ICLR 2024
D.5.8 ATYPICAL: CROP SHIFT
Heatmaps.
Proportion parturbed
1.0
0.05
0.1
0.8
Proportion
0.6
0.6
0.25
0.25
prato
grand
yog
forget
loss
dum
datalg
datamag
detector
eßt
aldi
diantali
Cont agr
grand
vog
forpet
loss
aum
dataig
datanapy
detector
allot
cleartaii
Cont. ago
Der LBsos inlmmee
La Outue
18 Stat
Seretral
1R0me 12:0
LD Sam
Sheiiira
(a) MNIST LeNet
(h) MNIST ResNet
Proporton perturbed
0.9
Proportion pertubeed
0.9
0.05
0.8
0.8
0.15
0.7
0.2
0.6
0.6
0.5
proto
grand
vag
ferget
loss
Bun
dataiy
datamaps
detector
etr
allet
cleantas
cont agr
0.5
grand
vog
forgat
bat
aum
dalaid
Batanops
detector
elinn
alloh
cleantali
cont. ago
Ouisst
Sotttial
Lecrnd L5
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 64: Atypical: Crop Shift - AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
Propartion perturbed
1.0
0.05
0.05
0.1
0.1
0.7
Proportion
0.15
0.2
0.2
0.6
0.25
0.6
0.25
proto
grand
vog
forest
loss
ditaig
Batamags
deftector
deartab cany agr
grand
forget
Joss
dataia
dstamaps
deracdor
alish
cleantab
cont. agr
UiGel Jaoteer
Stas
Sttstial
L8Gnd
(a) MNIST LeNet
(b) MNIST ResNet
Proportion perturbed
0.70
Proportion perurredd
0.600
0.05
0.1
0.65
0.1
0.575
0.35
0.60
0.15
0.550
0.2
0.2
0.25
0.25
proto
grand
vog
farjet
Inst
Rum
datarq
eetamape
detector
ezn
adsh
dleaninb
Conr.agr
grand
vog
forget
lots
danig
Astamapt
detecter
etzn
alfah
cleaniab
D. Uir LB Cala
Resu
4e Swisied
LEGt 18:00a
125
(c) CIFAR LeNet
CIFAR ResNet
Figure 65: Atypical: Crop Shift AUROC. We vary the proportion perturbed. i.c. proportion of hard examples. Blue better, red worse.
56

---

Published as a conference paper at ICLR 2024
Rankings.
andh and
cleantan
cort agr
datag
detemarrs dntortty
tngaet
grand
craataa
cont ogr
datad)
desamago
detector
etif
imgget
grand
Heaf
MNIST LeNet
(b) MNIST ResNet
Rank Scores
Rank
alsh
deariea
cant, agr
estaig
dedemigaa disertor
lorper
loud
prop
alan
anm
(itania
cond agr
datalg
detamagi
direectoo
eco
forget
grand
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 66: Atypical: Crop Shift AUPRC. Ranking of HCMs.
shipt roc
crop_sto crop_shift roc
11
altsh
aum
cleantab
ditammas
detector
elan
prototypicalty
loss
allsh
aum
cleantab
dataig
datamaps delector
ezn
forgetting
grand
less
vog
cont.
cont.
MNIST LeNet
(b) MNIST ResNet
shift crop_shift_ro
crop shift crop_shift.com
allsh aum
cleantab
datamaps
detector
lorgering
prototypicatig
ailsh
aum
cleanlab
dataiq
datamaps
detecer
622
lergetiing
grand
less
cont.
cont.
CD diagrams.
10
10
protetypicalit.co
d3n
Seariab
2) Torgetting
20g
COn deramann
sordetting
arandd
allsh
(a) MNIST LeNet
(b) MNIST ResNet
10
5
10
dest prototypic
cont agree
TOEE LOU ohand
conl dataat vog (6
sum cicaniab
elZn
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 68: Atypical: Crop Shift AUPRC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
57

---

Published as a conference paper at ICLR 2024
10
10
prototypicality (1
detector aish agree
detecter 11 sibn
eizn 12
catale C
VO3
cleantab
d3tain
veg
grand
cont Sgree
5.00 datamaps
(a) MNIST LeNet
(b) MNIST ResNet
10
10
conT 2gree
cont Forge
cleanlab
detector WOO D Tofguttnn
datamaga
probtyyaality
datarq
datamaps
(c) CIFAR LeNet
(d) CIFAR ResNett
Figure 69: Atypical: Crop Shift AUROC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
Matrix compare
alzh
dearlab
cleaniab
conl
conf agr
detsmaps
dotectar
forget
grand
fongel
grand
proto
ung
vog
dewn Ne
cant s0
denao
Cont 3g
330 maas
gono
HOV
H
(a) MNIST LeNet
(h) MNIST ResNet
1.0
1.0
8um
0.8
cleaniab
0.5
Jamass
0.6
HC41
detector
datamaps
0.4
forpet
grand
longe
lsss
prooo
vog
doanso
conT 30r
HOV
HCM
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 70: Atypical: Crop Shift. Head-to-head comparison matrix assessing for runs when a certain
HCM outperforms another. i.e. when y-axis (HCM 1) beats x-axis competitor (HCM 2).
58

---

Published as a conference paper at ICLR 2024
D.5.9
ATYPICAL: ZOOM SHIFT
Heatmaps.
Proportion parturbed
Proportion perturbed
1.0
0.05
0.1
0.8
0.6
0.6
0.25
0.25
prato
Brand
yog
forget
loss
dum
datalg
datamap
detector
eßt
aldi
diantali
Cont agr
grand
vog
forpet
loss
aum
datai
datanapy
ditector
allot
cleartai
Cont. ago
Der LB inlmmee
La Outue
18 Shet
Seretral
1R0me 1h:0
LD Sam
Sheiiira
(a) MNIST LeNet
(h) MNIST ResNet
Proporton perturbed
1.0
Proportion perturbed
1.0
0.05
0.15
0.6
0.2
0.4
proto
grand
terget
loss
Bun
dataiy
datamaps
detector
etr
allet
cleantas
cont agg
0.4
grand
vog
forgat
bat
aum
dalaid
Batanops
detector
elinn
alloh
cleantali
cont. agr
Ouisut
Sottticl
Lecrnd L3
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 71: Atypical: 7oom Shift - AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
Proportion partubbed
0.05
0.8
0.1
Proportion
0.7
0.6
0.55
0.25
0.5
0.50
Proto
grand
vog
forges
long
Burn
dataiq
Hilamips
detector
eßn
alteh
cleaniab
grand
vog
forget
leas
Bun
dataig
dattmmps
detector
e22
attel
Cranlab
cord. agr
conf
Satetcal
13:6nd 1i0teer
LRS
MNIST LeNet
(b) MNIST ResNet
Proportion parturbed
Proportion penurbed
0.05
0.05
0.7
0.15
0.2
0.25
0.55
0.25
0.50
lnis
dataig
idisannsps
detector
ednn
alinn
deaniab
conT. agr
grand
forget
loss
dalain
satamaps
detector
el2n
allsh
cleantali
Conf
L8Gnd Ls
Stactiicll
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure Atypical: Zoom Shift AUROC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse
59

---

Published as a conference paper at ICLR 2024
Rankings.
aiso and
dlantia
cart age
detamago
desertoy
graand Andd
prodt
artii
doo
crantab
age
datad
detamaga
elif
Iarget
srand
MNIST LeNet
(b) MNIST ResNet
mnla 6005
cleantet
cant. Jup
dataig
dasamago
deste for
forget
pro
allo
anm
creantab
cont apr
dataly
daaamge
den con
ferget
grand
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 73: Atypical: Zoom Shift AUPRC. Ranking of HCMs.
zoom
Scores
3-
allsh
aum
cleaniab
dalai
datamsps detector
lorgeting grahd
prototypicailg
allsh
aum
cleantah
datamaps
derectos
elzn
Inrgetting grand
less
vog
cont.
cont.
MNIST LeNet
(b) MNIST ResNet
shift roc
shift
alsh
cleanish
datamaps cont.
distalg
detectos
forgentng
grand
prototypl.
allsh
aum
cleanlab
dolald
datamaps
detectos
e22
forgetting
grand
Isss
vog
cont.
CD diagrams.
10
2.5
5.0
7.5
10.0
protatypicality
Sur
alis d
less.
cleaniab
grand
CorF capee
longtting
datamaps
cont. Sprce
detactor
MNIST LeNet
(b) MNIST ResNet
10
deanlib.
cleatise
RrAnd denictor
TOOOR pretclpiicaltyy
dATaMIAR allsh
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 75: Atypical: Zoom Shift AUPRC. Critical difference diagrams. Black lines connect methods that are not statistically significantly different.
60

---

Published as a conference paper at ICLR 2024
5,0
7,5
10.0
12.5
5
100
MISh PEPP Myree
VOG datamaps eum
forgetting
datamagg descaa
grand
cleafill
(a) MNIST LeNet
(b) MNIST ResNet
10
cleanlab 12
(12) detector
vog
[11] alish
forgetting
dalamhat
al2n
31 lose
datamaps
dataig
(6.4) cont jagnee
(c) CIFAR LeNet
(d) CIFAR ResNeet
Figure 76: Atypical: Zoom Shift AUROC. Critical difference diagrams. Black lines connect
methods that statistically significantly different.
Matrix compare
1.0
1.0
aum
0.8
cleaniab
0.5
Satmmas
0.6
cotalg
0.6
datamaps
delector
d2n
0.4
forpet
grand
longe
Isss
9.2
proto
vop
HON 2
H
(a) MNIST LeNet
(b) MNIST ResNet
deartat
cleanibb
conf. 3g
carf Agr
d3bb
de mmmaa
0.6
dateig
0.5
detector
octoctar
forget
#2n
0.4
#2n
0.4
0.2
grand
0.2
poto
loss
vog
cant sar
650000
forge
0.0
dbarto
cont. 3g
5000000
000
0.5
(c) CIFAR LeNet
(d) CIFAR ResNet
Figure 77: Atypical: Zoom Shift. Head-to-heal comparison matrix assessing for runs when a certain
HCM outperforms another. i.e. when y-axis (HCM 1) beats x-axis competitor (HCM 2).
61

---

Published as a conference paper at ICLR 2024
D.6
INDIVIDUAL RESULTS TABULAR
D.6.1 MISLABELING: UNIFORM
Heatmaps.
0.1
1.0
Proportion perturbed
0.1
0.9
0.8
Proportion
0.3
0.6
0.3
0.7
0.4
0.6
0.5
0.5
proto
grand
vog
lorget
loss
aum
dateiq
datamaps
detecto
e2m
deanteb
conf .gr
proto
grand
vap
forget
loss
aum
dataig
datana
detestor
e/2n
cleantab
cont Agr
2st LKCred 200
outup
Lasta staiiiial
Us: LB 15,00e
L5 Surjut
LB 5ts
Staiitkl
(a) Cover
(b) Diabetes
Figure 78: TABULAR: Uniform mislabeling AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse
Rankings.
Rank Scores U
Scores
aum
deannbbi
cont. agr
dataig
dacamaps
detector
ech
forget
grand
mar
vog
Bum
cleaniab
dataig
datamaps
detector
egh
forget
grand
lost
proto
cant
Cover
(b) Diahetes
Figure 79: TABULAR: Uniform mislabeling AUPRC. Ranking of HCMs.
CD diagrams.
10
5
10
deanleb
prototyplc5Mf
Sum [1]
vOg agree grand
datamant
detoctor grant fongetting
Sbcaag 2.2
prototypically
datariags
(a) Cover
(b) Diahctes
Figure 80: TABULAR: Uniform mislabeling AUPRC. Critical difference diagrams. Black lines
connect methods that are not statistically significantly different.
62

---

Published as a conference paper at ICLR 2024
D.6.2
MISLABELING: ASYMMETRIC
Heatmaps.
1.0
Proportion parturbed
0.2
0.8
0.2
0.3
0.4
0.4
0.5
0.5
proto
grand
lorger
sum
daaag
datamupe
detecto
elzn
cleaniab
proto
grand
veg
forger
law
auni
dataiq
datamaps
delector
et2r
chantat
Cont
(a) Cover
(b) Diahetes
Figure 81: TABULAR: asymmetric mislabeling AUPRC. We vary the proportion perturbed. i.c.
proportion of hard examples. Blue is better, red worse.
Rankings.
Rank Scores
Scores
mum
cleaniab
Cort. agr
datai
da datamans
detector
forget
grand
proto
veg
aum
cisardab
canf agr
dataig
datanaps
detector
elth
forent
grand
lons
proto
vag
(a) Cover
Diabetes
Figure 82: TABULAR: asymmetric mislabeling AUPRC. Ranking of HCMs.
CD diagrams.
10
30
dlaa
bum
el2n octactor araaaad forgetting
prototypgcalft
36m (1.4
(12) cont agroo
10 grand
prttoypicatty
#gree
daralg datamioes
Rorgetting
(a) Cover
(b) Diabetes
Figure 83: TABULAR: asymmetric mislabeling AUPRC. Critical difference diagrams. Black lines
connect methods that are not statistically significantly different.
63

---

Published as a conference paper at ICLR 2024
D.6.3
MISLABELING: INSTANCE
Heatmaps.
1.0
Proportion parturbed
0.2
0.2
0.3
0.6
0.4
0.5
0.5
prote
grand
lorger
sum
daaay
datamupe
detecto
elzn
cleaniab
egr
proto
grand
veg
forger
law
auni
dataiq
datamaps
delector
et2r
chantat
Cont
(a) Cover
(b) Diahctes
Figure 84: TABULAR: instance mislabeling AUPRC. We vary the proportion perturbed. i.c.
proportion of hard examples. Blue is better, red worse.
Rankings.
Rank Scores
Scores
mum
cleaniab
Cord agr
dakai
da Batamaps
detector
forget
grand
proto
veg
aum
cisardab
canf agr
dataig
datanaps
detector
elth
forett
grand
lons
proto
vag
(a) Cover
Diabetes
Figure 85: TABULAR: instance mislabeling AUPRC. Ranking of HCMs.
CD diagrams.
10
cleanlab (1.4)
dataig 12.2
27 al2n
loss
aum
detamags
protatypicaiity Setector
prototypica
Graatting
vdg
Sgree
(a) Cover
(b) Diabetes
Figure 86: TABULAR: instance mislabeling AUPRC. Critical difference diagrams. Black lines
connect methods that are not statistically significantly different.
64

---

Published as a conference paper at ICLR 2024
D.6.4
NEAR OOD: COVARIATE
Heatmaps.
Proportion parturbed
0.2
0.8
0.2
0.7
0.3
0.8
0.4
0.5
0.5
0.5
prote
grand
lorger
Rase
sum
dnaay
datamupe
detecto
elzn
deantal
proto
grand
veg
forger
law
auni
dataiq
datamaps
delector
et2r
chantat
Cont
(a) Cover
(b) Diahetes
Figure 87: TABULAR: Near OOD AUPRC. We vary the proportion perturbed. i.c. proportion of hard examples. Blue is better, red worse.
Rankings.
Rank Scores
Scores
mum
cleaniab
Cord agr
ditai
da aatamaps
detector
eztn
forget
grand
proto
vog
aum
cisardab
canf agr
dataig
datanaps
detector
elth
forget
grand
lons
prote
vab
(a) Cover
(b) Diabetes
Figure 88: TABULAR: Near OOD- AUPRC. Ranking of HCMs.
CD diagrams.
10
pratotypicalkky Ity
deanlab (4
grand 2) el2n
cleaniab (1
(12) conf agreo
conf agree
yag detector
Betector
locs forgetting
(a) Cover
(b) Diabetes
Figure 89: TABULAR: Near OOD AUPRC. Critical difference diagrams. Black lines connect
methods that are not statistically significantly different.
65

---

Published as a conference paper at ICLR 2024
D.6.5
FAR OOD
Heatmaps.
perturbed
Proportion parturbed
0.05
0.1
0.8
0.1
0.15
0.7
0.15
0.6
0.25
0.25
0.5
0.5
proto
grand
veg
forget
lasa
aunt
dataiy
datanaps
delnsto
diartta
agr
proto
grand
forgey
las
aurr
datalry
datamopa
detactor
etn
deartaai
Cont. agr
Die 1n:3nd 13:0me
1B Sans Statetical
Dir G3ddnd Lnore
LRtt Statetes
(a) Cover
(b) Diabctes
Figure 90: TABULAR: far ood AUPRC. We vary the proportion perturbed. i.e. proportion of hard examples. Blue is better, red worse.
Rankings.
Rank Scores
Scores
aum
cleaniab
Cond agr
datakg
datamaps
detector
ecnn
forget
grand
paotto
vog
aum
cleardab
cant agr
dataig
datamaps
detecter
eiah
forget
grand
losa
proto
(a) Cover
(b) Diabetes
Figure 91: TABULAR: far ood - AUPRC. Ranking of HCMs.
CD diagrams.
5
10
5
10
grand
cleaplab datamaap
conf agree protatypicality loss Rirgltting
araating (
datala detector Is
21 sum) vog
datamaps
(a) Cover
(b) Diabctes
Figure 92: TABULAR: far ood AUPRC. Critical difference diagrams. Black lines connect methods
that are not statistically significantly different.
66

---

Published as a conference paper at ICLR 2024
D.6.6 ATYPICAL
Heatmaps.
Proportion parturbed
0.2
0.2
0.8
0.3
0.4
0.6
0.5
0.5
proto
grand
lorger
Jase
sum
dnaag
datamupe
detector
elzn
cleaniab
proto
grand
veg
forger
law
auni
dataiq
datamaps
delector
et2r
chantat
Cont
(a) Cover
(b) Diahetes
Figure 93: TABULAR: atypical AUPRC, We vary the proportion perturbed. i.c. proportion hard examples. Blue is better, red worse.
Rankings.
Rank Scores
Scores
aum
cleaniab
Cort. agr
datai
da Batamaps
detector
forget
grand
proto
veg
aum
cisardab
canf agr
dataig
datanaps
detector
elth
forget
grand
lons
proto
104
(a) Cover
(b) Diabetes
Figure 94: TABULAR: atypical AUPRC. Ranking of HCMs.
CD diagrams.
10
conf agree (1.6)
dlearlab
31 forgetting
cleaniab
0 cont agree
prototypicality (4.6)
9.6) ootector
datar Lald
@rtttotypicaitty lo Rargetting
grand (6.2)
6.2) vog
e1.2
Cover
(b) Diabctes
Figure 95: TABULAR: atypical AUPRC. Critical difference diagrams. Black lines connect methods
that are not statistically significantly different.
67

---

Published as a conference paper at ICLR 2024
D.7 ADDITIONAL RESULTS MEDICAL X-RAY DATA
Overview. Our main experiments are conducted on the MNIST and CIFAR-I0 datasets. The reason is that it is important have "clean" datasets for the purposes of ensuring a good simulation environment. e.g. we do not want the original data to already have mislabeling. We now extend our analysis of HCMs to an additional medical imaging dataset from the NIH (Majkowska et al.. 2020). These contain chest x-rays which have been audited by multiple expert radiologists to ensure they are "clean". We repeat our experiments from the main paper and show the results below.
We now analyze the results and show the findings about HCMs are retained for the X- Ray dataset.
(a) Uniform
(b) Near-OoD (Covariate)
(c) Far-OoD
(d) Atypical (Zoom)
(c) Atypical (Crop)
Figure 96: AUPRC on medical X-Ray for different HCMs for different hardness types aggregated across setups. We vary the proportion perturbed, i.e. the proportion of hard examples. Blue is better, red worse. We see variability of HCM capabilities across hardness types and proportions.
(a) Uniform
(b) Near-Ool OoD (Covariate)
(c) Far-OoD
Sopec Goope
(d) Atypical (Zoom)
(c) Atypical (Crop)
Figure 97: Performance rankings of HCMs vary depending on the hardness type.
30
(a) Uniform
(b) Near-Oole (Covariate)
(c) Oor
10
2.5
40g
paooppiatty
(d) Atypical (Zoom)
(c) Atypical (Crop)
Figure 98: Critical difference diagrams highlight that similar categories/classes of HCMs do not have a statistically significant difference in their performance, indicated by the horizontal black lines linking HCMs which are not statistically different. The numbers in brackets denote mean rank.
68

---

Published as a conference paper at ICLR 2024
We highlight the following main takeaways which maintained from the main paper experiments, showing the findings indeed are similar across datasets:
Hardness types vary in difficulty. Similarly to the main text, we find that different types of hardness are easier harder to characterize. For instance, uniform mislabeling Far- OoD are much easier than data-specific hardness like Atypical. Hence, we still see the performance differences.
Learning dynamics-based methods with respect to output confidence effective general-purpose HCMs. In selecting a general-purpose HCM, we find similar to the main text that HCMs that characterize samples using learning dynamics on the confidence uncertainty-basee methods, which use probabilities (DataMaps, Data-IQ) or logits (AUM), are the best performing in terms of AUPRC across the board.
HCMs typically used for computational efficiency surprisingly uncompetitive. Similarly to the main text, we find that HCMs that leverage gradient changes (e.g. GraNd), typically used for selection improve computational efficiency, fare well at low p. However, at higher p, they become notably less competitive compared to simpler and computationally cheaper methods.
HCMs should only be used when hardness proportions are low. In general, different HCMs have significantly reduced performance at higher proportions of hardness. This is expected as get closer 0.5 since it's harder identify a clear difference between samples.
Individual HCMs within a broad "class" of methods are NOT statistically different. We find from the critical difference diagrams that methods falling into the same class of characterization are not statistically significant from one another (based on the black con- nected lines), despite the minor performance differences between them. Henee, practitioners should select an HCM within the broad HCM class most suitable for the application.
Selecting an HCM based on the hardness is useful. We find that confidence a good general- purposse tool if one doesn't know the hardness. However, if one knows the hardness, one can better select the HCM.
Of course, there are subtle differences given the difference in datasets. As an example, the performance rankings exhibit higher variability. That said, this is a function of running fewer seeds compared to the main paper. The most important compared to raw ranking is the CD Diagram which provides an analysis of the statistical significance of the performance differences which the raw scores don't provide. In that case, despite the rankings being different, we see very many of the same HCMs from the main paper are similarly connected by black lines, indicating the performance differences are not statistically different.
69

---

Published as a conference paper at ICLR 2024
D.8 INSIGHTS: EFFECT OF SEVERITY OF PERTURBATION ON HARDNESS SCORES
Overview. We ask what is the effect of the severity of the perturbation on the hardness score. Specifically, are samples with greater perturbations considered "harder" than others by HCMs based on their scores.
We conducted an experiment for zoom shift, crop shift, and Near-OOD (covariate), where have two severities of perturbation namely small and large. Specifically, for zoom (2x vs. 10x), crop shift (5 pixels vs. 20 pixels), and Near-OOD (std deviation of 0.5 vs. 2).
We report the results in Fig 99. The results show the percentage changes in hardness scores (in the direction of hardness, e.g. increase or decrease) for different HCMs for the different hardness types considered. In all cases, we see increase in hardness score for increased severity, with some HCM scores showcasing greater sensitivity and changes than others. particular, we see the gradient and loss-based approaches have the greatest changes compared to the methods computing scores based on probabilities.
These results shed interesting insights into the sensitivities of different HCMS.
aum
dataig
datamaps
+lnn cicantab
<n
large
cont_agrce
50
ao
hardness:
20
20
200m chif
shi
near ond
Hardness types
Figure 99: Effect hardness scores with respect to perturbation severity from small large.
70


Reviews:
[{"review_id": "zLTcipG4Dh", "reviewer": "Reviewer_SFiV", "summary": "The paper presents a benchmark for evaluating methods that make supervised learning models robust to hard data examples. The paper introduces a taxonomy of hard cases and defines each category in terms of where the corruption happens and how the data distributions differ. The authors developed a toolkit to automatically evaluate models and include 13 algorithms in their study. The paper summarizes the results and presents the main highlights, together with an extensive supplementary material with more details.", "questions": "Unfortunately, I don't find this study very compelling as a full research article. As it is currently, the paper seems to be more appropriate for a Datasets & Benchmarks track at NeurIPS, for instance, rather than bringing in novel insights to advance hardness characterization methodology. Maybe I'm missing what the most important conclusion is, which is what I'd like the authors to clarify.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The organization of the taxonomy is sound and formalizes observations from prior literature.\n* The systematic experimental design is strong and the evaluations are exhaustive.\n* The toolkit for evaluating and comparing performance across HCM seems well designed.", "weaknesses": "* The conclusions of the study seem limited. Despite the extensive experimental work, it is unclear how the hardness characterization field can make progress. While the paper claims to have practical tips, there is the underlying assumption that the hardness type is known and coming from only one type. This is not realistic in practice and undermines the value of the recommendations for practitioners.\n* The paper insists that studying data hardness is critical for advancing data-centric AI, but it is unclear what data-centric AI is, and why HCMs are the key to advance it. A more fundamental question is, do we really need HCMs at all or we only need to create high quality datasets that eliminate corrupted examples?\n* Following the need for HCMs, a baseline of training models without HCMs is missing. There are two scenarios that should be considered as baselines: 1) training a model without HCM with the same level of corruption introduced as the other models. 2) Training a model without HCM and with only the easy examples, i.e. removing the portion of corrupted samples. This would be informative to evaluate what cases in the taxonomy really need to be investigated further with HCMs and which ones can be addressed by simply cleaning the dataset.\n* The taxonomy is manually defined based on the interpretation of previous literature, and the datasets used in the study are intentionally clean (MNIST and CIFAR) to simulate hard samples. Perhaps data hardness should be evaluated in a data-driven way, i.e., using a real world dataset and using data science, statistics, and machine learning to find out the nature of hard samples on the dataset. This reviewer finds the approach of defining the hardness of data artificial, and perhaps not reflecting the challenges of current problems and datasets.\n* The paper points to the use of indirect measures, such as improvement in downstream performance, as an issue (Sec. 2.2). However, it seems like the metric that they use to evaluate methods and create the benchmark is downstream performance in the corresponding classification task. It was not clear what the proposed solution to this issue is in the benchmark.\n* The writing style and paper formatting is too flashy and very distracting. Instead of having a natural flow, the paper has sentences in bold, boxes in colors, and other decorations that ask for too much attention, giving the impression that there is something more important to read than the argument the reader is focusing on. I found this style very hard to follow. In addition, the paper reads as if it was trying hard to sell a toolkit rather than presenting a study with solid conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ag71tp6RAk", "reviewer": "Reviewer_5UHg", "summary": "This paper did the following:\n- A taxonomy of hardness -- mislabel, OOD, atypical\n- Provide a library of modules for dataloading, hardness measurements across different types of hardness, unified interface and evaluator\n- Propose that hardness is ill-defined, indeed\n- Propose that hardness is usually being measured by downstream, which should not be\n- Give a list of takeaways at the end\n- Experimentd ran on cifar and mnist", "questions": "See weaknesses.", "limitations": "", "rating": 1, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper dedicated a significant portion to describing why hardness is not well-defined anywhere. It also provides a taxonomy of hardness that are split into mislabeling, OOD and atypical. It also provides a unified interface for evaluation.", "weaknesses": "The authors stated that hardness should not depend on downstream improvement. I find that to be very controversial. One model's hardness may be another model's easy. Maybe I have not understood this completely and would like clarification. Moreover, in the end you run classification on cifar and mnist to obtain a list of takeaways iiuc -- did the authors not just use downstream tasks (cifar and mnist classifications) to inform us of their takeaways? If I understood incorrectly, please provide clarification. Also the categorization of hardness into mislabel, ood and atypical is quite unsatisfying. Why mislabel? Also in this era of self-supervision, LLM autoregressive, etc., I find this paper a little not keeping up with time. What is hardness in this new day and age?\n\nExperiments on cifar and mnist only --- authors claimed that they did not select imagenet because imagenet has less than 5% mislabel. Can you not artificially mislabel yourself then?\n\nA large part of the paper reads like an engineering document. I appreciate that, but I think scientific insights are more important while the engineering information can be delegated to appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cXVLn6kymZ", "reviewer": "Reviewer_zsTV", "summary": "This paper proposes a new framework called H-CAT for evaluating HCM methods that measure how difficult it is for a classification model to learn from certain examples. The paper provides three possible reasons for an example to be 'hard':\n- The example is mislabeled\n- It is OOD or outlier\n- The example is valid but is atypical\n\nNow the methods that are used to estimate the any of hardness types above are called HCMs:\n- Learning dynamics-based\n- Gradient-based\n- Distance-based\n\nThe paper presents a codebase which allows to evaluate the hardness types as well as the HCMs. The main takeaways are:\n- Many HCMs disagree with each other.\n- Some types of hardness are characterize to measure than others.\n- Learning dynamics-based methods are generally more effective.\n- The best HCM to use depends on the specific problem.", "questions": "I would like to know authors opinion regarding the limitations mentioned above, particularly 1) not knowing the type of hardness and 2) evolving hardness.\n\nTwo recommendations:\n- I would suggest to reorganize the content to ensure a logical flow of information, placing Figure 7, for example, and some details to the main body can help grasping key concepts.\n\n- Add detailed equations in a unified notation for the HCMs explored in section A.2.*. This would make your paper a go-to paper whenever someone wants to learn about hardness literature.\n\n--------------------\nPost rebuttal note:\nThank you very much for your response and applying the changes I suggested. I respectfully disagree with Reviewer 5UHg- the fact that the community's focus is now on LLMs doesn't make this type of works worthless. I believe that this paper has significant potential to advance our understanding of hardness in machine learning and hence I vote towards acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a novel codebase for the systematic analysis of HCMs. The categorization done in section 2.1 is nice to read. It is great to define hardness more formally and identify different underlying reasons for the difficulty a classification model in learning from certain examples.\n\n- The paper have sufficient experiments. The authors evaluated 13 HCMs, providing a robust validation of their findings.\n\n- The takeaways are generally valuable. The authors offer practical advice on how to choose HCMs based on the type of data hardness, and they highlight the advantages of learning dynamics-based methods.\n\n- Overall, the paper is well-written. However, reorganizing the content and moving some parts to the appendix and placing back some key figures and explanations to the main text could improve its readability.", "weaknesses": "- The paper could be structured better. Introducing and clearly defining \"hardness types\" and \"HCMs\" earlier would help readers understand the paper better. Some important explanations and figures that support the main concepts are in the appendix, which makes it difficult for readers to access them quickly and easily\n\n- The paper would be more technically rigorous if it included detailed equations and unified notation for the various HCMs in the appendix. Providing a more detailed explanation of methods, AUM for example, in the main text would also make the paper more instructive.\n\n- While the authors acknowledge certain limitations, two are particularly significant in my opinion:\n\n1. Before evaluating HCMs, you need to know what type of hardness is present in the data. This may not always be possible in real-world scenarios.\n2. The paper does not address the fact that hardness can change during the training process. A hard example at epoch 1 might not be hard at epoch 10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eHLpYDgMXf", "reviewer": "Reviewer_kMBG", "summary": "The paper proposes a new taxonomy for quantifying hardness of samples and then creates a framework on top of it (H-CAT) which uses existing hardness evaluation techniques to predict hardness categorized into different hardness types from their taxonomy. The paper then goes on to conduct hardness evaluation experiments on 2 image and 2 tabular datasets and highlights several key takeaways and recommendations for practitioners using their toolkit.", "questions": "1. As the severity of perturbations increases, do the methods used in HCMs also show an increase in the hardness score? Does this vary for learning-integrated methods vs others?\n2. What is the effect of using augmentations during training for these models? I'm guessing the hardness predictions will change for some methods but not others.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The taxonomy proposed by the paper is simple, clearly explicated, and practically useful. Relating it in detail to previous methods and frameworks and what's missing in existing HCMs based on this taxonomy helps readers create a good mental model of the space.\n2. The comprehensive benchmarking of more than 13 HCMs across multiple datasets, on robust evaluation criteria is useful for the community.\n3. Some takeaways like the efficacy of learning dynamics-based HCMs, lack of efficacy of computational-efficiency motivated methods, importance of significance testing and the variation in stability of the HCMs across seeds, backbones and parameterizations, all provide relevant, empirical information to practitioners using these methods.", "weaknesses": "1. Sticking to just MNIST and CIFAR10 limits the usefulness of the takeways. The dataset sizes and complexity we currently operate with are very different from MNIST/CIFAR10 and I'm afraid these learnings might not generalize to larger, more complex datasets.\n2. The Near OOD, far OOD, and the atypical perturbations used in the experiments are oversimplified in my opinion. In the real world, defining near/far OOD and atypical have a complicated notion of support of the sample distribution, but the use of Gaussian noise, texture changes for near OOD and exchanging MNIST with CIFAR (which are drastically different) for far OOD, and using simple invariant functions like shift and zoom for atypical does not do justice to these concepts.\n3. [Minor] The bulk of paper is focussed on creating a taxonomy for hardness evaluation and then building a software toolkit to evaluate existing methods for its quantification. Some initial discussions during the taxonomy creation and some of the non-obvious takeaways are instructive for the researchers in this field, but most of this work is geared towards practitioners. This isn't necessarily a negative thing, but limits the research novelty/impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U2n8lUlDOn", "reviewer": "Reviewer_u8mR", "summary": "The author claims that existing works on Hardness Characterization Methods (HCMs) fail to address comprehensive and quantitative evaluation of HCMs. Also, the author points out the absence of an integrated and practical software tool that can simulate and evaluate various types of HCMs. In this context, the author proposes a fine-grained taxonomy for categorizing multiple hardness types and introduces the Hardness Characterization Analysis Toolkit (H-CAT), which supports over 14K setups with 13 different HCMs across 8 hardness types. The author reports the performance of each HCM in various settings using H-CAT and provides invaluable takeaways and practical tips that were not seen in previous works.", "questions": "1) In the Introduction, it is stated that \"These 'hard' samples or data points can significantly hamper the performance of ML models.\" However, from an active learning perspective, can't we consider hard samples to be more informative and better samples with higher annotation efficiency?\n2) On page 3, in the section \"Data characterization,\" it is mentioned that difficult or easy samples are classified into separate easy or hard subsets. However, instead of categorizing sample difficulty into two binary sets, wouldn't it be better to dynamically control the learning process by assigning weights or using other methods based on continuous difficulty values? Additionally, excluding OOD/outlier and atypical samples from model training could potentially enhance the model's generalization capability. So, couldn't excluding them altogether from training be a suboptimal choice?\n3) The author mentioned that existing works have a disadvantage of conducting indirect evaluation in downstream tasks. In that case, how does the proposed framework perform direct evaluation?\n4) In Figure 6, Spearman rank correlation scores for HCMs are reported. As far as I know, Spearman rank correlation calculates the correlation between two variables. How was the correlation computed from multiple runs in this case?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) This paper diligently considered all the recently proposed HCM algorithms within the past 3-4 years and developed H-CAT, reporting experimental results using it.\n2) Through graphical illustration (Figure 1) and table analysis (Table 1), the need for a systemic evaluation framework for HCMs was effectively underscored.\n3) Figures 3, 4, 5, and 6 analyzed the behavior of popular HCMs under various hardness settings from different perspectives. Additionally, the experiments conducted using the toolkit developed by the authors hold greater significance.\n4) The author provided detailed instructions for use of H-CAT and additional experimental results through a voluminous appendix.", "weaknesses": "Hardness characterization is an important concept in various fields such as computer vision works like object detection and segmentation, as well as natural language processing and reinforcement learning. However, the proposed HCM evaluation framework in this paper can only handle image classification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)