Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmark
Authors: Thomas George
Pierre Nodet
Alexis Bondu
Vincent Lemaire
Venue: Accepted by TMLR ()

Abstract: Mislabeled examples are ubiquitous in real-world machine learning datasets, advocating the development of techniques for automatic detection. We show that most mislabeled detection methods can be viewed as probing trained machine learning models using a few core principles. We formalize a modular framework that encompasses these methods, parameterized by only 4 building blocks, as well as a Python library that demonstrates that these principles can actually be implemented. The focus is on classifier-agnostic concepts, with an emphasis on adapting methods developed for deep learning models to non-deep classifiers for tabular data. We benchmark existing methods on (artificial) Completely At Random (NCAR) as well as (realistic) Not At Random (NNAR) labeling noise from a variety of tasks with imperfect labeling rules. This benchmark provides new insights as well as limitations of existing methods in this setup.

Published in Transactions on Machine Learning Research (09/2024)
Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmark
Thomas George Orange Innovation
thomes.gonge@@rrngee.com
Pierre *odet Orange Innovation
picrv.noleettooragg.ccom
Alexis Bondu Orange Innovation
alezis. Secrissbondd@@@orggee.ccmm
Vincent Lemaire
wincent.lemaiiee@Brannggccc
Orange Innovation
Reviewed on OpenReview: https: // openreview. net/ forum? id- 7BHka
Abstract
Mislabeled examples are ubiquitous in real-world machine learning datasets, advocating the development of techniques for automatic detection. We show that most mislabeled detec- tion methods can be viewed as probing trained machine lcarning models using few core principles. We formalize a modular framework that encompasses these methods, parame- terized by only building blocks, as well as a Python library that demonstrates that these principles can actually be implemented. The focus on classifier- concepts, with an emphasis on adapting methods developed for deep learning models to non-deep classifiers for tabular data. We benchmark existing methods on (artificial) Completely At Random (NCAR) well as (realistic) Not At Random (NNAR) labeling noise from variety of tasks with imperfect labeling rules. This benchmark provides new insights as well as limitations of existing methods in this setup.
1 Introduction
In supervisod machine learning, the performance of learned algorithms crucially depends on the quality of the dataset of examples used during training: how many examples do have access lo, are these examples the feature space, and were the training examples correctly
representative of the actual distribution labeled. We focus on the latter subject. Indeed, many actual use cases include some amount of labeling errors. For example, this is typically the case in tasks that involve human supervision since labeling large datasets requires pool of annotators that possess a mix of expert knowledge (which is costly), and willingness to perform repetitive tasks (which is dull). This also known to be the case for widely used benchmark datascts such as CIFAR10/100 or MNIST (Northcutt et al.] 2021b). Therefore, cleansing datasets offers the promise of better performance, but at the cost of additional ellorts. Since the early days of machine learning, it has been widely believed that this could be achieved through automated methods, eliminating the need for further human intervention. This has led to number of methods for automatic detection of mislabeled examples using classical machine learning methods (Guan & Yuan 2013). With the success of deep learning methods in applications ranging from image recognition to language models, new mislabeled detection methods have also been proposed that exploit its specific training dynamics.
equul contribution

---

Published in Transactions on Machine Learning Research (09/2024)
The aim of this paper is to offer new perspective existing mislabeled detection methods, well practical recommendations in actual use cases in the presence of labeling noise. Rather than learning a model that captures the structure of the labeling noise, our approach is to blindly evaluate existing methods on real-world datascts, with no prior knowledge. We survey mislabeled detection methods regardless of whether they were designed to work with deep learning models or other classical machine learning algorithmss and we highlight a few common principles. We also focus on tabular and text data, a type of data that is prevalent in the industry (e.g. in logs, in customer databases, etc) but that has recently received less attention than datasets that more amenable to deep learning methods such images, sound or language.
This paper is organized as follows: In section 2 we suggest a definition of the problem of detecting mislabeled examples. We discuss its limitations and the inherent ambiguity of what can be considered a mislabeled example in the statistical learning framework. We also describe existing strategies for dealing with mislabeled examples. In section 3 we highlight the concepts behind a majority of mislabeled detection methods found in the litcrature. We show that most methods can be described using a few core principles with distinct components. We survey large amount of existing mislabeled detection methods and show how they fit in this framework. The modularity of this framework can also be leveraged to design new methods by recombining existing components. We review a number of extensions to existing methods that address some specific issues encountered in real-world use cases. In section 4 we describe our contributed library that implements the framework of the previous section, showing that it is not just a theoretical view but that it can actually be implemented. Most existing methods in the literature can be readily implemented using this library. In section 3 we perform a large-scale experiment by varying detectors, handling strategies, and noise structure on a. large number of actual datasets. This leads to new insights regarding the behavior of mislabeled detection methods in different setups, well as practical guidance aimed at future practitioners. Section 6 is dedicated to other related works that were not included in previous sections, and we conclude in section
List of contributions
Concepts We provide a fresh view on a majority of existing mislabeled deteccion methods by showing that they can be described in a modular framework that uses a few components. This allows us to provide a large survey the highlights the similarities and differences of these methods. We identify a set of probes that provide the base signal used to discriminate between genuine and mislabeled examples.
Survey We review large number of existing detection methods, as well as other strategies to address specific issues. We survey 3 common strategies in weakly supervised learning.
Implementation We contribute a library that allows instantiating a large number of existing mislabeled detection methods, as well as designing and experimenting with new ones. Rather than packaging a number of different detection methods, our library focuses on implementing the core principles of our framework, then existing specific methods can be instantiated in a few lines of code and are readily available, as well as the possibility to benchmark new methods. We also package helpers to load existing weakly supervised datasots from the literature using a common interface order to improve the reproducibility of experiments in the weakly supervised setup.
Empirical evaluation On a large benchmark of text and tabular datasets, we evaluate series of detectors in different setups, where we vary the type of noise (weak supervision using labeling functions or uniform noise), the hyperparameter selection method (using clean validation set a noisy validation set), and the strategy for handling detected mislabeled examples (filtering or relabeling). We identify a number of practical questions for which our experiments provide new insights. We also share raw data for our experimental results as we believe they can be used to answer some future questions. Using our classifieragnosttcc implementation, we propose the first comparison of different mislabeled detection methods on the exact same task and using the same features and base machine learning model.
2

---

Published in Transactions on Machine Learning Research (09/2024)
2 Supervised learning and mislabeled examples: concepts and strategies
2.1 Definitions and problem statement
Training datasets that contain mislabeled examples are prevalent in real-world machine learning applications. Our main aim is to detect these mislabeled examples. A first challenge that we found while reviewing mislabeled detection methods is that it is difficult to come up with general formal definition of what it means for an example to be mislabeled.
For some obvious cases, human annotator can easily tell if an example is correctly labeled or mislabeled, but in some others the definition of mislabeled is more ambignous such as when an instance can be considered to lie in 2 different classes. As an attempt at giving a precise definition, we distinguish between 2 theoretical frameworks in the next sections. We found no precedence of such an attempt.
2.1.1 Deterministic case: the true concept is function
We aim at estimating (deterministic) function from imput space X' to an output space V. Ideally we would observe a finite sample nosislesss of n examples and their corresponding labels ((r,, W/ = where the z; are sampled from an unknown distribution P (X) over the input space. Typically, in classifica- tion, is the class of the instance, whereas in regression, y/ is a scalar value.
In real life, the data labeling process is often imperfect: our act observed dataset Dirain contains examples {(r,, ū:) Dg)gg that can undergo a corruption process and get a different label yi * ys. In this case, the definition is straightforward: an example considered mislabeled if ÿt + f (zz).
The approach of seeing machine learning as estimating unknown function has permitted many theoretical and practical successes in the carly days of computational learning theory. It, however, falls short of formal- izing the more general case where each point in the input space corresponds to several different targets with non-zero (but possibly unbalanced) probability. This is the setup studied in statistical learning theory (see Luxburg & Schölkopi 2011 for a concise but comprehensive overview of statistical learning theory).
2.1.2 Stochastic case: the true concept is defined as probability distribution
A more general case consists in defining the true underlying concept P (X,Y)) = P (X) P (Y At fixed x E X, the mass of P [Y||X = x) is not necessarily concentrated into a single mode. Put differently, there are (possibly infinitely) many possible values for y with non-zero probability. This happens for instance when æ does not contain all necessary information to predict y and there remains some aleatoric uncertainty. Typically, in classification, we would like to learn an estimator that returns probabilities of belonging to each class (x), = IP (Y = c| = =x), while in regression, would return the expected value f (z) = E[Y|X = z].
joint probability distribution
Similarly to the deterministic case, contrarily to the independent and identically distributed assumption often used in statistical learning theory, there is some discrepancy between the true concept that we want to learn, and the actual distribution of collected examples during the data labeling process: whereas we would ideally get sample Dideal of examples {xx;, Ji) from IP (X, Y), we actually observe training dataset Drrain of examples (z. 5) sampled from a corrupted distribution P (X.
In this case, the definition of a mislabeled example is ambiguous since we do not have a single ground truth value, but instead a probability distribution over the output space y of many possible ones. This happens, for instance, when the true concept is modeled as mixture of possible classes. As an example, suppose input æ where P (YIX- = x) is mixture of a majority class with probability 99% and a minority class with probability 1 1%, and suppose an example (r, v) where is the minority class. Do we consider this example to be mislabeled?
In the rest, we suppose that we have access to a sensible threshold E 1]: we consider examples (z, y) with probability under the true concept IP (Y y|X x) < to be mislabeled and other examples to be genuine. Note that this is the true concept PP (Y])),, which is unknown in general, not the output of an
3

---

Published in Transactions on Machine Learning Research (09/2024)
estimator trained using finite set of examples. This definition also covers the deterministic setting special case.
This choice is further motivated in data pipelines (section 2.6 where the output of detection stage is fed to a filtering procedure that produces a dataset of most trusted examples used to train a machine learning estimator. In this case, we are often interested in recovering only the majority classes since they are also the most likely ones in our evaluation (test) set. However, we emphasize that in some other contexts, this definition might not be well suited, such as when are more interested in predicting correctly for underrepresented instances in fairness-related related tasks.
2.2 Detecting mislabeled examples using trust scores
Estimating the conditional probability is by itself a difficult problem, which e.g. requires proper calibration of machine learning models. Since we are only interested in splitting the dataset into genuine examples and mislabeled ones, it sufficient. to solve the relaxed problem of estimating a proxy of the conditional probability, hereafter called trust score.
Definition 2.1 (Trust score). Any scoring function s (z, v) that is correlated with the conditional probability such that for any examples (x1, V1) and (222, 122), it preserves the ranking bet ween conditional probabilities: (21. (1) S s (I2, (2) + P PY 01|X x1) S IP (Y = 02 = x2).
Equipped with this trust score, we can split the training dataset in 2 distinct parts: trusted examples with high trust scores and untrusted examples with low trust scores. For an ideal trust score method and threshold, the trusted and untrusted datasets are equal to the genuine and mislabeled examples sets. In section 3.2
we give comprchensive review of modcl-probing methods for computing trust scores.
2.3 Assumptions regarding the noise generating process
A common approach to detect mislabeled examples within a dataset is to design detectors based on explicit assumptions regarding the structure of the underlying noise-generative process. widely favored structure is known as the noise transition matrir denoted by T. Here, V(i.j) E [I, x [1, K], where K is the number of noisy classes and K the number of true classes, Tú represents the probability P(Ý = = j) of an example from the class j to have been assigned a noisy label from class i (Van Rooyen & Williamson 2017). This concept holds the advantage of generalizing over class-dependent label noise in multi-class classification setting.
In the instance- deeendent label noise scenario, the noise transition matrix becomes a function of the example T: + MM (R), and in the uniform label noise scenario, it is reduced to a constant corresponding to the overall noise rate. A series of work has been dedicated to the estimation of the noise transition matrix for
the class-dependent (Liu & Tao 2015; Patrini et al.) 2017 Xia et al., 2019 Yao et al., 2020) and instance- dependent Xia et al. 2020 Yang et al. 2022) case.
often necessary to estimate the noise transition matrix. Anchor point- based hethods assume the existence and identifiability of high-confidence samples within the dataset (Liu & Tao 2015 Patrini et al. 2017). Alternatively, some techniques require prior knowledge of class distributions or specific noise ratios (Wang et al. 2017), or they exploit the structural characteristics of the noise transition matrix, such as its tendency to form clusters in the feature space (Liuu et al. 2023). Mixture proportion estimation approaches infer noise ratios or the noise transition matrix by asscssing the contamination level
However, additional assumptions
of one class's 's feature distribution by others (Vandermeulen & Scott] (016).. However, these approaches presuppose that class distributions are mutually irreducible, implying distinct and non-overlapping patterns among classes (Scott) 2015).
In contrast, we assume no prior knowledge of the underlying noise structure in this paper. Instead, we chose to focus on detectors that do not explicitly depend on structural aspects the noise-generating process, specifically focusing on the family of model-probing detectors. The succoss of these detectors relies on an implicit assumption concerning the base model's behavior, which should exhibit significant differences when applied to noisy versus clean data. In practice, model-probing detectors may fail in extreme cases where the
4

---

Published in Transactions on Machine Learning Research (09/2024)
base model struggles to discern regularities in the data, e.g. due to excessive noise inductive bias in the base model that cannot capture the structure of the noise. In such situations, expecting them to identify mislabeled examples correctly may be unrealistic.
Furthermore, we note that more complex scenarios, such as concept drift (Lu et al. 2018 or data poisoning
attacks (Tian et al. 2022). are out of the soope of this study but remain as open and interesting problems tackle.
2.4 A taxonomy of data regions
In practical machine learning applications, we do not have exact knowledge of the true concept (X, (), but instead, we only have access to a limited sample of data points. Depending on the availability of data, by looking at the training set examples only, we distinguish between cases (pictured in figure for 2-classes toy example):
Fig We have access to many examples and all are from the same class. In this case unambiguously consider this class the true class: any example from another class lying in this region would be considered mislabeled.
Fig (2)
We have access many examples, but they are equally spread into 2 diffcrent classes. this case, no example from these 2 classes should be considered mislabeled.
Fig 1
(3) We only have access to a few examples but it looks like they all come from the same class. Does this mean that this the true class, or just that data is too scarce in this region so we are just unsure about the true class?
Fig We only have access a few examples, and they come from different classes. Does that mean that close to boundary of the true underlying concept with 2 separate regions from 2 different classes, or is it region where examples from the 2 classes are sampled with equal probability from the true underlying concept?
alcatoric
epistemic
ply pz)
3
rare
p(x)
densc
Figure 1: Illustaation of a ground truth distribution P (XX, Y) decomposed as IP (X) on the y-axis and P (Y|X) on the x-axis and a sample of 100 data points from this distribution. In this toy distribution, we distinguish 4 different cases represented as quadrants:
(1) (X) dense, P (Y is low entropy: we are pretty confident that the ground truth class is so any example would be mislabeled (2) IP (X) is dense, IP PY]j) is high entropy, cannot distinguish between classes * and thus would be unable to tell correctly labeled from mislabeled examples (3) P (X) is scattered, but IP (Y[X) is low entropy so can assume that the ground truth class is * and any example should be deemed mislabeled (4) Since IP (X) is scattered, it is more difficult to detect that P (Y[X) is high entropy by looking at the data only, is likely that mislabeled detection methods would fail in the absence of further assumptions.
5

---

Published in Transactions on Machine Learning Research (09/2024)
In the last case, by looking the data alone, the main difficulty resides in distinguishing between rare examples which provide useful information regarding some specific part of the input space, and mislabeled examples. These rare useful examples are termed "exceptions" in Brodley & Friedl (1999), and are the ones that provide the last few percents in test accuracy in state-of-the-art classifiers (Feldman 2020), or that carry most. information regarding underrepresented subgroups in the population lairness-related lasks [Liu et al. 2021).
A related concept is the distinction between alcatoric and epistemic uncertainty, where alcatoric uncertainty is high in regions where the true underlying concept IP (Y[X) has high entropy, whereas epistemic uncer- tainty comes from a lack of knowledge due to not enough data points in specific regions (see Hüllermeier & for an introduction).
Waegeman 2021
2.5
Use cases
Detecting mislabeled examples is of practical interest in real-world scenarios where obtaining the ground truth label of an example is an imperfect process. The causes of these imperfections are diverse and sometimes even intended in order to automatize much as possible machine learning operations. We now highlight some imperfect labeling processes and the role that mislabeled example detection can play in these scenarios.
Weak supervision Properly labeling example sometimes requires a costly and non-scalable procedure prohibiting annotation of the whole training dataset. Fraud detection and cyber security are both fields where the labeling process requires scarce domain experts who need to conduct time-consuming forensic analysis. One of the most popular solutions is to distill expert knowledge into hand-engincered labeling rules to automatize and scale the labeling process to the entire training dataset. These rules form a new form of supervision called weak supervision (Ratner et al., 2016), and instcad of the usual strong supervision, these labeling rules might produce incorrect labels that could harm the efficiency of machine learning model trained from these rules. Mislabeled example detection could potentially help experts design better rules to strengthen the weak supervision. Our experiments in section 5 include benchmark of mislabeled detection methods in this setting.
Crowd labeling When the labeling task is achievable by human annotators, outsourcing the labeling process to decentralized annotators is common approach to annotate webscale datasets. This technique called crowdsouree labeling (Yuen et al., 2011) has been employed extensively in computer vision to create the first large image datasets such as ImageNet (Deng et al., 2009 Nowadays, it is employed to fine- tune large language models from human feedback (Ouyang el al. 2022 Bai et al. (022). However, with crowdsourced labeling, the effectiveness of each annotator in following the labeling guidelines can vary and be unreliable. Mislabeled example detection provides way to evaluate the annotators' ability to systematically follow the guidelines and even detect malicious annotators.
Web scraping In order to quickly assemble supervised web-scale datasets free of human intervention, automatically crawling the web gathering data and labels from querying engines is popular approach. Web scraping has been used in the natural language processing community to design sentiment analysis datasets (Maas et al. 2011) from movie reviews, or in the computer vision community to study label noise at scale (Xiao et al.] 2015). As shown in the latter, the oracle used to label examples, here the search engine, sometimes diverges from the ground truth and provides noisy annotations. In the former, the data in itself was wrong because of human error, resulting in wrongly labeled data. More recently, web scraping has been extensively used to train large language models, in self but still supervised fashion where the label to predict is the token next to the input sequence. However, using non-curated web data has been shown to severely hinder their capacities to produce non-toxic or hallucinations-free text (Wang et al. 2023). In these situations, mislabeled examples detection can provide insights on the quality of the data source used to automatically construct. datasets.
In light of these scenarios, the data understanding part of the data mining methodology (Shaarer 2000) seems more prevalent than ever, caused by the ambition automatize every part of the machine learning operations.
6

---

Published in Transactions on Machine Learning Research (09/2024)
2.6 Fully automated learning in the presence of mislabeled examples: detect + handle strategies
Strategies to address learning with noisy labels include (Frénay & Verleysen 2013]: (i) using algorithuss that are naturally robust to label noise; (ii) using algorithms that explicitly model label noise during training; (iii) assigning a trust score to cach training example to then mamually inspoct low trust training examples, or use an automated downstream method that can leverage this additional metadata. All three familics of methods are useful depending on the context. In the rest. of the paper, we focus on the latter strategy: The detection of mislabeled instances is not only valuable for the sole purpose of curating dataset that more accurately reflects the underlying concept, but it is also critical preliminary step in comprehensive pipeline, illustrated in figure 2 The ultimate goal is to train an estimator on a smaller dataset, free of mislabeled instances, with the expectation of getting more accurate predictions. It is achieved using a detection method that provides trust scores that are then fed to a splilting strategy that separates a trusted part of the training examples from untrusted ones, A final stage handles the 2 datasets in order to provide a trained estimator. Here we distinguish between 3 strategies:
The simplest approach for handling untrusted examples is to discard them from the training dataset, thereby training using only trusted data. This approach can also be found in the literature under the name of data cleaning or editing (Wilson & Martinez 2000). From this perspective, it is not harmful to accidentally flag some correctly labeled examples as being unirust unirustworthy as long as enough represent ative correctly labeled examples remain in the trusted dataset. To the contrary, some correctly labeled examples that lic in underrepresented regions might be accidentally flagged as mislabeled, which would degrade the performance of the final estimator in these regions of the feature space. Therefore the effectiveness of this approach is bound to the underlying detection method being able to distinguish between rare (thus difficult) and mislabeled examples and the precise tuning of the threshold used to split the dataset into trusted and untrusted examples.
Filtering
Semi-supervised learning Although filtering is a direct approach, it may be overly dismissive of the in- formation contained within untrusted data. Since this study is restricted to labeling noise, training instances are considered free of feature space noise. Thus, a more reasonable approach is to retain the instances while disregarding their labels, thus casting the handle step into a semi-supervised problem (Li et al.. 2020).
This semi-supervised approach has the advantage of maintaining the entirety of the training dataset, thus preserving the original data distribution. However, it inherits the filtering method's intrinsic limitation of discarding some information from the untrustworthy examples (here, their labels), which could be partially correct or beneficial for the learning process.
Biquality learning The ideal approach would involve retaining the full training dataset while incorporat- ing metadata about the quality of each example, allowing the learning algorithm to leverage this auxiliary information. A particular case of this scenario is called biqualily data, where two datasets are available at training time, a trusted and an untrustworthy dataset, which falls under the biquality learning framework [Nodet et al. 2021). Algorithms within this framework are able to make more granular docisions regarding how untrusted examples are handled, potentially reweighting or relabeling individually each example. How- ever, this approach assumes a high confidence in the trusted dataset: Any mislabeled example misclassified as trusted could significantly undermine the effectiveness of these algorithms.
3
Model-probing detection of mislabeled examples
3.1 Using trained models to detect mislabeled examples
Machine learning consists in identifying regularities in data sets and exploiting these regularities in order to predict the oulcome on new examples. To the contrary, mislabeled examples are instead the ones that deviate from these regularities. The underlying concept behind model--pobing mislabeled detection methods is that these irregular examples are treated somchow differently from genuine examples by the machine lcarning model. Informally, a good base model should find mislabeled examples difficult to learn and genuine examples easier to learn. In practice, quantifying how regular or irregular every example is, is done by probing trained
7

---

Published in Transactions on Machine Learning Research (09/2024)
Detect
Spore
Spont
XT
Yr
Y
Y
Trasted
X
Yu
Unrrusted
Original data
XT
YT
XT
YT
Pr
Pr
Pr
X
X
Yu
Du
Du Semi-superviool learning
Du
Filtering
Higuality
Handle
Figure 2: Data pipeline for different learning strategies when in the presence of labeling noise, where intermediary step uses detection method to assign trust scores to every example, then splits the dalaset into a trusted and an untrusted part.
To this end, similar to the fact that machine learning methods may be more suited than others some particular tasks (i.e. they get better generalization performance). accurately detecting mislabeled examples crucially depends on the choice of machine learning method and proper luning of hyperparameters.
machine learning models (figure 3 3
learning
Data
Model
diagnostics
Figure 3: Whereas machine learning consists in choosing a model that best fits the data (from left to right), model-probing mislabeled detection methods follow the opposite direction and probe a trained model order to give diagnostics on set of examples (from right to left).
3.2 A general framework for pooelnggobing methods with examples
We now discuss one of the contributions our work, which is to provide general framework that en- compasses most methods for the identification of mislabcled examples, with few exceptions discussed subsequent sections. We also survey existing methods found in the literature and show that they fit in this framework. The framework is pictured in figure 4 The final outcome of these methods is scalar trust score (defined in Section 2.1 for each example, used to rank examples from most likely to be mislabeled to most trusted. The framework is composed of 4 components that we shortly describe next. We then go in more depth in subsequent sections, where we show that most surveyed methods correspond to an instance of this framework using specific choices each component.
1/ base model: All model-probing detection methods rely on fitting a machine learning model to the training set examples (or subset thereof, depending on an optional ensemble strategy). This model should incorporate some robustness to mislabeled examples so that they are treated differently from genuine exam- ples.
8

---

Published in Transactions on Machine Learning Research (09/2024)
2/ model probe: We then probo this model (possibly at different checkpoints during training, depending on the ensemble strategy), in order to score every example using scalar metric used to discriminate bet genuine and mislabeled examples.
3/ ensemble strategy: Optionally, the base model is trained multiple times using an ensemble strategy such as bootstrapping or boosting, where each learner of the ensemble provides a slightly different value of the probed scores.
4/ aggregation method: These scores are then aggregated to provide single scalar trust score for each example.
As an example, the Area Under the Margin (AUM) method (Pleiss et al. 2020 fits in this framework by considering the base model to be a deep network, the probe be the margin at every iteration during training, the ensemble strategy is the consecutive iterates, and the aggregation method is just the sum of these margins.
Table I summarizes all surveyed methods that fit in the framework. This unified picture suggests that we can automatically design new methods by simply replacing one or several of the components in already existing methods. This also advocates for transferring ideas developed for deep lcarning modols to classical ones (section (.3) and vice versa, where for instance the method of observing the fluctuations of the per-example accuracy as training progresses has been independently discovered in deep learning (Toneva et al. 2018) and with AdaBoost (Chen et al. 2022).
3.2.1 Base model
The base model is the core component of this framework, which is trained on the training examples (or a subset of the training examples depending on the ensemble strategy, see section 3.2.3) and then probed (section (.2.2) to give a scalar score to every example. Intuitively, good candidate base model should treat genuine and mislabeled examples differently, so that mislabeled examples are more difficult to learn than genuine ones. This form of robustness against learning mislabcled examples is for instance found in doep learning models (Arpit el al., 2017 Feldman & Zhang. 2020) where mislabeled examples have been shown to be handled differently (Krueger al. 2017), in particular depending on the training regime (George et al., 2022), or in decision trees where mislabeled examples often end-up single instance of different class in otherwise pure leaves. In general, every off-the-shelf machine learning method can be chosen, depending on the task (data and output types, and performance metric). Some detection methods however require specific characteristics for the base model, such as being differentiable with respect to their input (Agarwal et al. 2022 or their parameters (Koh & Liang 2017).
The popularity of base models used for mislabeled examples is directly linked to the general popularity of machine learning models, where nearest neighbors methods used to be more popular in the early days of machine learning (Wilson 1972 Tomek 1976), then kernelized linear methods (Thongkam et al. 2008
Ekambaram et al.) 2016) and more recently decision tree ensembles (Verbaeten & Van Assche 2003: Chen ct al.) 2022). With the success of deep lcarning methods in image or text related tasks, new mislabelod detection methods have followed that exploit the specific features of neural networks such as their training dynamics (Toneva et al. 2018: Pleiss et al. 2020 Agarwal et al. 2022 Pruthi et al., 2020 Koh & Liang, 2017 Jiang al. (2021), as well as classical machine learning methods on top of features extracted from
deep networks representations (c.g. k-NNs in Bahri et al. 2020 Zhu et al. 2022 2024h.
As summary, any supervised learning method can be used as base model.
3.2.2 Model probe
Fitted models are then probed in order to get scalar scores that are used to discriminate between genuine and mislabeled examples (figure 3h. Probes produce intermediate values that are then aggregated (section
"We use the word probe throughout which generic enough encompass variety of different methods that follow the same purpose of scoring each example by means some measurement. on trained model.

---

Published in Transactions on Machine Learning Research (09/2024)
3. Ensemble strategy
Probe
Probe
Probe
scores
BOO00S
SCOCOS
Aguregate
Progressive Ensemble
Agpregate
Aggregate
1. Base model
Probe
Asgregate
Probe
Sports
Aggrerata
some
Aptregate
Probe
sporcs
2. Probe
4. Aggreg. method
Indepemdent Easemble
Probe
No Eusemblle
Figure& 4: Schematic summary of model-probing detection methods, with their 4 components.
3.2.4 to single scalar trust score for each example. The most naive way of probing model is simply to use its prediction, but it can also be done using more convoluted methods, some that are similar to the concept of uncertainty in active learning (Settles] 2011), some others which are more specific to the machine learning model used such as gradient of logits with respect to input pixels in Agarwal al. 2022
An early series of detection methods simply use the predicted class by comparing the prediction given by neighbor examples with k-NN classifier (Wilson] 1972 Tomek 1976 Brodley & Friedl 1999) SVM
(Segata et al. 2010 Thongkam et al. 2008) and flag examples mislabeled if the prediction does not match the dataset label. Following a different motivation but similarly using the predicted class as the probe, forget scores Toneva et al. 2018) in deep learning and fluctuation scores (Chen et al.) (022) in AdaBoost measure how much the per-example accuracy oscillates as training progresses, where more oscillations indicate a more difficult. thus potentially mislabeled example.
AdaBoost example weights (Verbaeten & Van Assche 2003) have been used as probes, with larger weights indicating more difficult examples. This is very similar to the method of small loss popularized by series of methods in deep learning (Amiri et al. 2018 Jiang et al. 2018 Here, examples with small loss are considered genuine and larger loss are sign of suspicions examples. In the context of learning with the cross-entropy loss, this is also similar to using the margin (Dligach & Palmer] 2011 Pleiss el al. 2020). where a small or negative margin indicates a potential mislabeled example, or to consider support vector examples (Ekambaram et al. 2016) as suspicious as these are the examples closer the decision boundary in support vector machines. In Paul al. (2021), the l distance between the one-hot encoded label and the softmax output of deep network defines the EL2N score which is generally higher for noise instances.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Following the principle that perfectly fitting mislabeled examples would require more complex models, lamily of methods estimates the additional complexity of the base model required to learn an example compared to removing the example from the training set (Gamberger et al. 2000). A proxy measure for complexity is used in Chen et al. 2022) as the number of weak learners required to learn the label. In Ma et al. 20188) a high value of the local intrinsic dimensionality of the last layer representation for some examples as training progresses indicates the need for a higher compression in order to learn these examples. In [Blldokk et al. (2021) in deep learning, classifiers are trained using the representations extracted from layers of varying depth, the smallest depth able to correctly classify data point measures how easy it is, thus higher prediction depths correspond to potential mislabeled examples. With different motivation,
Agarwal et al. (2022) propose to measure the gradient of the target logit of individual dataset examples with respect to the input space, which is a measure of the smoothness of the prediction function. This builds on the idea that perfectly fitting a mislabeled example in a region with otherwise genuine examples requires a very localized spike in the decision function, which will be reflected in a larger gradicnt.
A popular lool in statistics, influence functions (Hampel 1974) have been adapted to machine learning diagnostic tool to estimate the effect of adding removing example in the training dataset. Informally, they are an estimate of the effect of an infinitesimal change in the weight given to a datapoint, to some value of interest, such as the parameters of the optimal model learned from this new weighting of examples, or the prediction on other examples (also see Cook 1977 for closed-form expressions of several variants applied to linear models). In deep learning, they are estimated using linearization of the predictor near an optimum (Koh & Liang 2017 Barshan et al., 2020 Kong al. 2021 Bac et al. 2022). Self-influence is defined as the influence of an example on its own prediction. High values of self-influence indicate that the prediction on an example is only influenced by itsclf, whercas low self-influence is hint that other examples carry similar features and larget, indicating a more trustworthy example. In TracIn (Pruthi et al., 2020), the influence of examples is instead estimated using the gradient of the per-example loss at several checkpoints during training, also similar to GraNd (Paul et al. (021)) which use the averaged gradient norm as we vary the initial parameters of a neural network. Inspired by the representer theorem for functions obtained by minimizing the empirical risk over a reproducing kernel Hilbert space (RKHS), representer values (Yeh et al.] 2018) aim at explaining& a trained deep model's prediction by means of the contribution from each training point. Similar to self-inftuence values, these coefficients can be used to diagnose mislabeled examples.
In Sedova et al. 2023). the cosine similarity between the gradients of the loss of individual examples, and the average gradient of the loss estimated on minibatch that does not contain these examples, is used as an indication that this example disagrecs with other examples, in that it would push the Icarned model towards a different direction than that of the majorily of examples. Low or negative cosine similarity indicates potential mislabeled examples.
While most probes discussed so far specialized to classification tasks, the same model-probing framework also applies when dealing with regression by just choosing appropriate probes. This is e.g. done in Zhou ct al. (2023) where the model is probed using the f distance between the prediction of the trained model and the dataset larget.
Designing new ways of probing trained models can lead to improved detection of mislabeled examples. This is for instance explored in Kuan & Mueller (202))
Summary of probes: predicted class, boosting weights, loss, margin, la distance one-hot, support vectors, number of weak learners, local intrinsic dimensionality, prediction depth, input//otputt gradient, influence function, representer values, é distance, similarity between gradients.
3.2.3 Ensemble strategy
Whereas some detection methods only rely on a point estimate (e.g. Wilson 1972 Segata el al. 2010).
mislabeled detection methods can be improved by leveraging an ensemble strategy [Verbaeten & Van Assche 2003). We distinguish between the independent and progressive familics of ensemble methods, that differ by the nature of the members of the ensemble, and how we leverage the differences in weak learners in the context of detecting mislabeled examples.
11

---

Published in Transactions on Machine Learning Research (09/2024)
Independent ensembles are built using bootstrapping, cross-validation leave-one-out, where models are trained on subsets of the original training set. This produces a variety of measurements of the probe. In these methods, all weak learners are trained on independentll drawn subsamples of the training set, they can be considered to be on equal terms. The disagroement or inconsistency of different models within the ensemble can be leveraged as an indication of a potential mislabeled example (Jiang el al.; 2021 Northeut et al. 2021a). Instead of varying the subset of examples on which ensemble members are trained, it also possible train using different models (Smith & Martinez 2014], and leverage their diversity.
By contrast, in progressive ensembles, there is natural ranking between weak learners that come from the consccutive iterations. In boosting, a predictor is progressively built as a sum of weak learners, cach of which is learned from the prediction of the previous boosting iteration. In the context of the detection of mislabeled examples, boosting models can be probed at every iteration during training, which produces a series of different values (Chen et al., 2022 Deep learning models are also trained an iterative fashion (Plciss et al., 2020), where cach step in the parameter space incurs a change in function space that can be assimilated to weak learner] Because of this analogy, methods designed to work with deep networks can be readily adapted to boosted models (section 4.3/. In progressive ensemble strategies, early iterations learn a prominent pattern (ideally corresponding to the clean examples) whercas late iterations are required to learn spiked decision boundaries as well as mislabeled examples. In this case, the complexity of the decision function learned increases as training progresses, which can be leveraged as an extra signal to detect clean and mislabeled examples.
Summary of ensemble strategies: bootstrapping, cross-validation, leave-onc-out, different models, boosting/deep learning.
3.2.4 Aggregation method
We obtain a series of probe scores from each model of the ensemble. In order to summarize them to a single scalar trust score, we now need to aggregate all these measurements. This is achieved by choosing an aggregation method. The simplest one is just to average the probed scores (or equivalently, take their sum) as done c.g. in Northcutt et al. (2021a): Pleiss et al. (2020): Pruthi et al. (2020). or compute a majority vote or consensus as in Guan et al. (2011): Verbaeten & Van Assche (2003).
A natural extension is instead to measure some form of spread of probed scores, with a large spread indicating that ensemble members disagree thus a potential mislabeled example. This is e.g. achieved using the & variance in Agarwal et al.] (2022) Seedat et al. (2022), also reminiscent of the idea of uncertainty in active learning (Chang et al.) 2017). Going a step further, the difference in probe scores between members of the ensemble that include a given example in their training subset and members that do not (e.g., yaan Halteren 2000) can also be leveraged, where larger difference suggests potential mislabeled example, whereas a prototypical example will probably have less influence on the predictor since similar examples are likely included in the training subset. For instance C-scores (Jiang et al. 2021] are measure of how likely a datapoint is to be correctly classified by model trained on a subset that does not include it. Similarly, DataShapley values (Ghorbani & Zou 2019; Kwon & Zou 2022) measure the individual value of each example on a utility function such as the test loss, with a negative value indicating a potentially mislabeled example.
For progressive ensemble strategies, there is a natural ordering of the members of the ensemble, which are the consccutive iterations of the algorithm. This is exploited by several techniques. Forget scores in deep learning (Toneva et al. 2018) and fluctuation (Chen et al. 2022 in AdaBoost. count how many times the per-example accuracy between consecutive iterations changed from not predicting the dataset label to correctly predicting it, with larger values indicating that the example is more difficult to learn. In Cordeiro et al. (2023), examples are considered clean if their individual loss is smaller than a threshold 5 for Ç epochs in row where 7 and Ç are hyperparameters, and in Yuan et al. (2023). trustworthy examples are those
At the end of training, fwr Swo + Σt Sw. swe can be viewed as an ensemble of weak learners (he (haheeu mhs
stemming from parameters updates Wt- W
12

---

Published in Transactions on Machine Learning Research (09/2024)
Base model
Probe
Ensemble strategy
Aggregation
k-NN
accuracy
leave-one-out
OOB value n/a majority vote
Wilson 1972
k-NN
accuracy
no ensemble bootstrapping no ensemble no ensemble no ensemble no ensemble different models no ensemble no ensemble learning iterations no ensemble no ensemble no ensemble no ensemble
Tomek 1976
various
accuracy
Brodle Fried] (1999
AdaBoost
example weights
n/a n/a
Verbaeten & Van Assche
2003
SVM
accuracy
Thongkam et al.
(2008
Local SVM
accuracy
n/a n/a sum n/a n/a change count n/a n/a n/a n/a
Segata et al. (2010)
MaxEnt
margin
Dligach & Palmer
2011
various
self confidence
Smith & Martinez
2014
SVC
support vectors influence accuracy
Ekambaram et al.
2016
deep network
Koh & Liang
(2017
deep network
Toneva al.
2018
deep network
loss local intrinsic dim.
Amiri al.
2018
deep network
Ma ct al.
2018
deep network
representer value
Yeh el al. 2018
deep network
loss margin loss gradient accuracy l distance one-hot accuracy prediction depth self confidence influence imput/ /output gradient accuracy # weak learners accuracy
Jiang et al. 2018
deep network
learning iterations learning iterations no ensemble no ensemble bootstrapping no ensemble bootstrapping no ensemble learning iterations boosting iterations ensemble no ensemble cross-validation
sum
Pleiss et al.
2020
deep network
sum
Pruthi et al.
2020
k-NN
n/a
Bahri et al.
2020
deep network
n/a
Paul et al.
2021
deep network
sum
Jiang ct al.
202)
deep network
n/a
Baldock et al.
[202]
deep network
mean
Northcutt et a.
2222
deep network deep network decision stump
n/a variance change count n/a n/a
Kong et al.
(02
Agarwal al. [202
Chen et al. 2022
AdaBoost
Chen et al. 2022
k-NN
Zhu et al. 2022
various
& distance
OOB value
Zhou et al.
2023
Table 1: Taxonomy of modcl-probing detection methods
that are correctly classified for & epochs in a row earlier during training, where also a hyperparameter. The lg variance of the input/output gradient of individual examples across iterations used Agarwal lt al. (0022), with the idea that the training dynamics of neural network will fluctuate around mislabeled examples as it will require decision function that spikes around single example in otherwise uniform region.
Instead of aggregating different scores from different members of an ensemble, it is also possible to compute k different probes for a single trained model, as done in Lu et al. (2023) for & = 2. Here, the scores are aggregated a single trust score using a Gaussian mixture model (GMM), but in general, we could expect any k-dimensional clustering method work.
Summary of aggregation methods average/sum, majority vote, consensus, variance, difference in out, DataShaplcy, difference between iterates, stability for Ç epochs in a row, clustering in higher dimension
3.3 Bag of (clever) tricks
The framework proposed section 3.2 is the backbone for many reviewed detection methods. In addition, we now survey some additional techniques proposod in the literature, which we consider as plugins that aim solving specific problems that arise when dealing with mislabeled examples. These methods are applied top of the framework and agnostic to the model-probing detection strategy.
13

---

Published in Transactions on Machine Learning Research (09/2024)
3.3.1 Iterative refinement
The pipeline for learning in the presence of mislabeled examples detailed in section 2.6 is a 2-stage approach with detect and handle stages applied once. natural way to improve its elficiency to do many-passes over the training examples by probing base models fitted on sequence of refined datasets iteratively, where the base models of iteration T are trained using clean examples only filtered by iteration T T-1 (Tomck 1976 Chen et al. 2019).
The iterative refinement approach can be integrated directly into the training procedure of the machine learning model. As training progresses, the sets of beneficial and detrimental examples may change. By incorporating the detection stage into each iteration of the training procedure, the model can be updated increenntaly with the suitable refined dataset given its current progress its curriculum (Sedova et al. 2023).
3.3.2 Surely mislabeled pseudo-class
The trust scores generated by most detection methods are often on an arbitrary scale. Splitting a training set into a trusted and untrusted part thus requires a carefully chosen threshold. This threshold depends on the detection method (and scale of the trust scores), as well as on the noise level and structure of noise. When used in a detect + filter pipeline, choosing It can be achioved by treating it a hyperparameter in cross-validation sctup (which likely requires noise-free validation set, see discussion in section 5.3).
appropriate value of the threshold is of crucial importance.
Alternatively, Pleiss et al. (2020) proposed to artificially introduce examples that are purposely mislabeled (we assign them a wrong class) and measure their trust scores so that we get a distribution of trust scores corresponding to surely mislabeled examples. This is achieved by introducing an extra pseudo-class and assigning it to random subset of the training data. It, however, assumes that the noise introduced by this additional class has similar properties the noise in the original data.
3.3.3 Class Mlassbblanccing mechanisms
A limitation of detection methods arises when dealing with Mlss-imbalanced datasets. Given that minority class examples are scarce, they are often more difficult to correctly predict than other examples, thus they are more prone to being flagged as potentially mislabcled examples by most detection methods. In the meantime, they often the most useful ones given their scarcity: we would not be able to correctly predict the minority class if there were to few examples from the minority class in the training data.
In order to alleviate this issue, reasonable approach is to detect mislabeled examples in a one vs. rest fashion by selecting trusted examples class per class (Northeutt et al. 2021a Wang et al. 2022 Karim et al. 2022).
An alternative approach to normalize scores across classes allowing the splitting step to be done parsi-
moniously for all classes, which can be done by simple scaling (Kim et al., 2023), done under the peered
prediction framework (Miller et al. 2005) using peer examples (Lu & Guo
2020 Cheng et al.) 2020
An alternative could revolve around the calibration of the trust scores compared to the true conditional probability, which remains an under-explored area. The only works we are aware of that attempt to tackle this problem proposes to adjust the predicted probabilities of training examples by the average predicted probability for each class while probing the model (Northcutt el al., 2021a Kuan & Mueller 2022
3.3.4 Reducing epistemic uncertainty
Another interpretation of the problem of differentiating hard but clean examples from noisy examples is through the field of uncertainty quantification, specifically in the distinction between epistemic and aleatoric uncertainty (Hüllermeier Waegeman 2021). Epistemic uncertainty corresponds to uncertain predictions of base model that can be reducod with more training data (rare and hard examples). Alcatoric uncortainty corresponds to irreducible uncertainty, where the information from the features of a sample alone cannot predict its label (e.g. when the true underlying concept is mix of several classes).
14

---

Published in Transactions on Machine Learning Research (09/2024)
When probing the base model, both of these categories of examples will be assigned a low trust score (Hooker et al. 2019). To better differentiate these types of uncertainty, data augmentation can be used to artificially create more training data (D'souza et al. 2021). This way, trust scores of examples with epistemic uncertainty will increase, while trust scores of examples with alcatoric uncertainty will remain the same.
An alternative strategy involves designing detectors that combine probes that respond differently to aleatoric and epistemic uncertainty. In Kuan & Mueller (2022); Zhou et al. (2023), it is done by re-weighting label- noise probes (respectively the self-confidence and the ε distance) by out-of-distribution probes (respectively the prediction entropy and the prediction variance). In Lu et al. (2023), label-noise probe and an out-of- distribution probe are computed separately and aggregated through clustering.
Furthermore, framing the problem of mislabeled examples delection as an outlier detection task is another way to disambiguate epistemic and aleatoric uncertainty, yet out of scope of the model-probing framework (soc scction for more thorough discussion regarding detection of outliers).
4 Library
To further emphasize that the proposed framework in section 3.2 is not only of theoretical interest but also of practical one, we now present another important contribution of our work in the form of a Python library that matorializes the components (basc model, probe, ensemble strategy, aggregation method) of the framework into modular approach with 4 blocks that can readily be customized. Implementing an existing method of the literature from table | then amounts to just specifying the value of each column of the table, and we can invent new methods as new combinations of already existing components.
4.1 Detection of mislabeled examples by computing trust scores
The core of the library is a versatile ModelPrrobingDetector object that uses arguments:
BaseModel which can be any estimator using scikil-learn's API Pedregosa et al. 2011
an EnsembleStrategy that defines the logic on how to fit and probe the base model,
the probe that returns score for every training example given fitted model,
and the aggregator that defines how we aggregate scores over multiple probes and/or multiple fitted models.
Trust scores for training examples and corresponding, possibly corrupted, labels y are then computed by calling the perustsscores(X. (x,y) method of ModelProbingDetector which closoly follows scikit-learn's design so that most machine learning practitioners should already feel familiar. The method returns scalar trust score for each example.
For example, the AreaUnderMargin detection method (AUM, Pleiss et al. 2020) can be defined to work with gradient boosted trees using scikit-learn's implementation GBT with the following code snippet:
AreaUnderMargin ModellProbingDetector base (om.mmdeelBMM,,,.. eneemblesPoogressiveeeembbbee probe="nargin", aggregate="sunm
Figure 5: This code reads * "consider a gradient boosted tree model (GBM) as progressive ensemble, com- pute margins for all ezamples at every iteration during training, them up to obtain scalar trust scores".
Noteworthy, we can readily perform ablation studies with respect to any of the 4 components by keeping all others fixed. For instance, implementing the same detector but using itcrates of a logistic regression model trained with gradient descent can be done by replacing GBM with scikit-learn's LogisticRegressicn istimator. Similarly, instead of using the margin, could design different probe for specific need, and
15

---

Published in Transactions on Machine Learning Research (09/2024)
e.g. imagine alternative implementation of AUM for regression that would instead use the & distance to the target as its probe (see section D.1
The library comes with a series of helpers for defining most detectors found in the literature, so that all detectors benchmarked in section are readily available with a simple Python import.
4.2 A versatile API
We designed our library so that we can easily exlend it with new ideas (e.g. new ways of probing a base model), as long as each component follows the following block contracts:
The BaseModel contract follows the widely used API of scikit-learn's estimators (Buitinck et al. 2013). It allows using scikit-learn's suite of already implemented estimators, as well as estimators from other librarics that follow the same widely usod API.
The EnsembleStrategy contract is a single method probe_model _adee,, X, y, probe) that takes as an input a non-initialized base model, the features, the labels, and the probing method, and outputs the computed probes as an iterator of length n a_models yielding NumPy arrays of shape (n_ in_samples, n_probes) and potential metadata, such as an iterator of boolean masks indicating if sample was part of the training set of the probed model (e-g. in the case of bootstrapping it allows to distinguish in-the-bag from out-of-bar bag examples).
The probe contract callable that takes as an input fitted model, features, and labels of training samples and outputs one-dimensional NumPy array of length n_ sammpess.
The aggregator is a callable that defines how we summarize a series of probe scores viewed as an iterator yielding NumPy arrays of shape (n (n_samples, n_probes) to a one-dimension NumPy array of trust scores of length n samples.
Thanks to the modularity of the API as well as the case of adding new components, exploring new detectors uncovered in table ! (unknown region of a 4D cube) is as easy changing one string when instantiating ModelProbingDetectorr We hope that our library can foster the design and understanding of future mislabeled detection methods.
4.3 A common API for progressive ensembles
We propose novel API to unify all incremental machine learning approaches into single contract named staged_fit fit that produces stream of machine learning models from dataset. As streams are lazy data structures, it allows flexible implementations of this contract for different families of machine learning models. For deep networks, to reduce memory cost, only a single model is kept in memory, and the network is trained incrementally between each iteration. For gradient boosting machines, all trees are trained at once and then copied and dispatched into smaller GBT for each iteration. Moreover, the concept iteration can be changed dynamically and independently for different model families. The provided implementation for deep networks uses an epoch as the reference for an iteration, but batch version could be used instead. As long notion of increment in complexity exists for family of machine learning models, a staged_ fitt can be implemented. For example, decision trees are treated as progressive models, from decision stumps to fully grown trees with pure leaves.
4.4 Full pipelines
In addition to the detection API using NodelProbingDetecton objects, we also provide way of building full pipelines following delect + handle strategies as described in section 2 Splitter objects define stralegies to split a training set in a trusted and untrusted part using the trust scores (e.g. using a specified threshold or by keeping a specified top quantile of the trust scores). Handler objects implement connectors learning strategies such as filtering, supervise learning, or biquality learning so that all necessary tools create fully automated data pipelines are readily available.
16

---

Published in Transactions on Machine Learning Research (09/2024)
5 Benchmarks
A classical approach to benchmarking detection methods is to evaluate them on synthetic tasks where noisy labels are injected artificially into otherwise clean datasets. Tt. allows to conduct. a post-mortem analysis the accuracy of detecting mislabeled examples since both the noisy and ground truth labels of all examples are known. Yet, using only synthetic corruptions in experimental protocols might lead to wrong conclusions on the actual performance of detection methods as they might not be representative of real-world corruptions. In our benchmark, we use text and tabular datasets with noisy labels generated from imperfect labeling rules, corresponding to more realistic scenarios, thanks to the growing availability of such datasets. On these tasks, we evaluated multiple surveyed detection methods on different criteria, most notably in the case of the fully automated weakly-supervised pipeline described in section 2.6
Overall, the purpose of this benchmark is not to provide definitive ranking between detection methods. As we surveyed a large spectrun of existing and adapted methods, we could not exhaustively fine-tune each method individually, so it is likely that some methods were not evaluated at their best capacity. Rather, this benchmark serves to highlight a few recommcndations for practitioners, as it provides data for a meta-analysis on a large number of actual dataasess
5.1 Benchmark design
We now outline the features of the benchmark. A more detailed presentation with all specifics is included in appendix
Tabular data We choose to put the emphasis of our benchmark on tabular tasks. Arguably (Grinsztajn et al. 2022), these tasks do not benefit from the representation learning properties of deep models, thus our evaluation only relics on the regularization properties of the machine learning models used instead of how good they are at learning useful representations. Tabular data lasks also often include ambiguous mixing
regions, where the true underlying concept is a mix of several classes (Seedat et al. 2022 For illustration, we also include an experiment on an image dataset in appendix D.2
Datasets We use weakly supervised datasets from the Wrench benchmark (Zhang et al. 2021 supple- mented by other datasets from the literature (Rühling Cachay et al. 2021 Hedderich et al. 2020), for which we have access to weak labels from a set of automatic labeling rules as well as ground truth labels. A summary of tasks statistics is available in table 2
Sources of noise in benchmarked datasets
We experiment with the following setups
Artificial uniform noise: with probability 30%, an example is assigned a random label uniformly between existing classes, independently of its fcature or true class. This creates dataset with Completely At Random (NCAR) noise. In practice, this form of noise is often used when experi- menting since it can easily be artificially introduced in existing noise free datasets. It is, however, very different (and simpler to deal with) from the actual structure of noise encountered in real-world datasets.
Imporfect labeling rules: a set of automatic labeling rules are appliod to every example and then aggregated as single label. These labeling rules are imperfect, rendering the assigned labels noisy. In this case, noisy examples are more frequent in regions that are not correctly covered by the labeling rules or when several labeling rules disagree, thus the probability for examples to be mis- labeled depends on their features æ. The structure of the noise is Not At Random (NNAR). Some examples might not be covered by any labeling rule This more general form of noise is often more representative actual use cases, but also more difficult to tackle.
In these experiments, we chose exclude examples that not covered by any labeling rules from the training set. Alternatively, one could assign them a random label, but it would likely he noisy one.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Datasets used to benchmark detectors. Columns: dataset size n, number of raw features d, number of encoded features ((d), number of classes K, histogram of class priors p(y), number of labeling rules LRs, noise transition matrix T, noise ratio p(ū + v), percentage of examples for which at least one labeling rule gave a label coverage.
Benchmark
Modality
Datasct
d
((d)
K
plu)
LRs
T
p(ù # v)
coverage
waln
text
hausa
2.92K
4.82K
750
5
18.7K
0.50
97%
yoruba
1.91K
6.95K
539
7
20.3K
0.40
93%
weasel
text
amazon
200K
160K
3.68K
2
175
0.25
65%
professor-teacher
24.6K
113K
4.06K
99
0.18
81%
wrench
tabular
Dank-marketing
45.2K
16
78
2
20
0.26
93%
basketball
20.3K
2.05K
2.05K
L
4
0.25
100%
bioresponse
3.75K
1.78K
10.4K
2
20
0.46
99%
31.9K
105
105
L
83
0.19
99%
commercial
81.1K
2.05K
2.05K
4
0.10
100%
mushroom
8.12K
22
108
2
20
0.13
99%
phishing
11.1K
0.21
97%
spambase
4.6K
57
57
2
15
0.25
97%
tennis
8.8K
2.05K
2.05K
6
0.13
100%
text
agnews
120K
145K
3.36K
4
9
0.19
69%
imdb
25K
74K
10.1K
2
5
0.26
87%
sms
5.57K
13.5K
1.37K
2
73
0.03
40%
trec
6.03K
9.25K
946
6
68
0.46
95%
yelp
38K
200K
5.35K
8
0.28
82%
youtube
2.06K
7.08K
423
2
10
0.15
89%
Detection methods We evaluate several detection methods surveyed in section choosing different detectors with the right diversity of components: Some originate from the deep learning literature, using progressive ensemble: Variance of Gradients (VoG, Agarwal et al. 2022), , Area Under the Margin (AUM,
Pleiss et al. 2020), Forget Scores (Toneva et al. 2018), TracIn (Pruthi et al.) 2020). Small Losses (Amiri ct al. 2018 Jiang et al. 2018) and AGRA (Sedova et al. 2023), others are not specific to deep lcarning: Consensus Consistency (Jiang et al., 2021 and CleanLab (Northcutt el al.] (021a), finally some methods come from the influence literature in linear models: Self-Influence (Koh Liang 2017 and Self- Relf-Representer
Yeh ct al. 2018). Their respective position in our framework is described in table
Evaluation of detection methods Evaluating detection methods is often task specific. We choose to evaluate surveyed methods using the following critoria:
Predictive power of the trust scores to detect mislabeled examples. We use the Area under the Receiver Operator Curve (AUROC) ranking quality metric.
Representativeenes of the filtered dataset. We use class-balance as proxy representativeness. To measure class-balance in multi-class classification, we use the ratio of the prior of the minority class over the prior of the majority class.
Performance in a fully automated weakly supervised pipeline with no additional supervision. We use the test loss of an estimator trained after filtering of the less trusted examples given by method's trust scores (detect filter).
18

---

Published in Transactions on Machine Learning Research (09/2024)
4. Performance in semi-automated pipeline with additional supervision. We use the test loss of an estimator trained after relabeling of the 10% less trusted examples given by method's trust scores (detect + relabel).
These pipelines provide a practical method for evaluating mislabeled detectors: an efficient detector must at least be able to somewhat distinguish between genuine and mislabeled examples, but it would be too restrictive to assess the quality of a detector by only measuring its accuracy. In particular, some examples are more important than others (e.g. because they are rare instances, or they belong to a minority class or an underropresented pattern), and this should be reflected in the evaluation of detection methods. Instcad, we employ these fully automated pipelines some classification metric (here the test loss) is measured for classifiers trained on filtered datasets. This approach allows for the disparate value of different examples to be taken into account: two detectors that have the same error rate but on different instances will result in different classifiers trained on filtered data.
methodology for the evaluation of detection methods, whereby
The performance mislabeled detection pipelines depends on the value of the follow- ing perrpaarametes::
Hyperparametent
Hyperparameters of the detection method (This includes hyperparameters of the base model such as e.g. the l regularization coefficient logistic regression, as well as the hyperparameters of the probe, ensemble and aggregation strategies).
Hyperparameters of the final estimator.
Threshold for splitting the training dataset between trusted and untrusted examples.
Hyperparameters the base model of the detector, well as hyperparamoters of the final estimator are sampled by random search (12 limes 12 times). For each sampled couple of hyperparameters, threshold values are then chosen from the grid 0.10.1.000 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9.
Approzimately millions models have been trained in the making of this benchmark.
Choice of hyperparancter values Hyperparameter tuning is done using cross-validation, where an holdout split of the training dataset is kept apart from training examples, and used only to compute an estimate of the test loss. We distinguish between the following cases:
Noisy: the validation set follows the same distribution as the training set. In particular, it contains potential noisy examples
Noise free: the validation set only contains clean examples. An example use-case is when an addi- tional effort is made on labeling of the validation set: since it is typically smaller than the training set, it is not prohibitively costly to review these particular instances more carefully.
the test set is used validation set. This answers the question 'how would my detection method perform had I had access to an oracle that would give me perfect hyperparameters! Even if not useful in practical ical applications, this hyperparameter selection method gives us some indications on the behavior of detection methods.
Oracle:
Baselines and normalization In order to properly evaluate the surveyed detection methods, we use the following baselines:
None:
No filtering step is performed, the training set is used as a whole including mislabeled exam- ples.
Random: Filtering or relabeling (depending on the experiment) uses random trust scores.
Silver (perfect filtering): The training step only includes examples that have genuine labels.
19

---

Published in Transactions on Machine Learning Research (09/2024)
Gold: The whole training set is used, mislabeled examples are assigned their gemuine label.
Since we compare detection methods across tasks varying difficulties, we normalize by scaling the observed metrics (i.e. the test loss) linearly between 100 and 200 so that the performance of the none baseline gets 200 and the silver baseline gets 100.
Machine learning models There are 2 different machine learning models involved in benchmarked pipelines: the base model used at the detection stage, and the final estimator of the pipeline (Figure 2 For both stages, we experiment with 2 different machine learning models: a kernelized linear model (KLM) trained with stochastic gradient descent and a gradient boosting model (GBT).
Reproducibility Both detectors and helpers to download the datasets used in the benchmark are avail- able in the open-sourced library described section 4 available on the repository https://github. com/ hatpee/ppppnourrre/iiiaaabbblble The benchmark code spanning from feature pre-processing to detector evaluation is available in a separate open-source repository lnttps://iithub. attp::///gihabb...mo/raaaaaggppp ccm//rrange--ppsssurrel with fixed seeds for random number generators. The entire benchmark results are also available in public archive https://githi: wtw.ttuub.com/tffggrgegggo jomore/geerrg//miii.. htt.geerrgeemmseeechhheeee _e_s so that reproducing figures and tables can be done without re-running the benchmark. We hope that all provided code and raw results will help foster research in weakly supervised learning.
5.2 Benchmark observations
This large scale benchmark allows us to ask series of questions and observe some trends that we now highlight. For completeness, additional experiments with different setups (different final estimator), different noise structures (NCAR instead of NNAR), and different hyperparameter selection strategy (using a clean or noisy validation set) are deferred to appendix C
Overall performance of detection methods We start by evaluating detoction methods in the detect + relabel pipeline (Figure G The experiment consists in relabeling the 10% less trusted examples as pointed by mislabeled detection methods. We use random trust scores as baseline that every sctup is given the same number of training examples, and the same budget of relabeling. On most datasets, we observe an improvement in test loss compared to the random baseline, which confirms that mislabeled example detectors provide a useful signal. This also gives us a ranking between detectors on these particular tasks, where AGRA shows consistent performance compared to other methods.
Overall performance of filtering pipelines We turn to detect + filter pipelines, and ask the question whether we can get an improvement in performance by using such a pipcline (which is fully automated and does not require additional human supervision) compared to just using carefully regularized model on noisy dataset. This is not trivial machine learning methods are known to already embed some form of robustness to noisy examples: by playing with hyperparameters that reduce its capacity, we can tune any machine learning method to focus on more salient features while trading off some flexibility to fit mislabeled examples. Furthermore, there is a trade-off between a filtered training set with only trusted examples, or keeping as many training examples as possible in order to have a bigger training dataset but at the cost of including potential mislabeled examples. In our experiments on NNAR, we observe that models trained the subset of training examples that have gemuine labels (the silver baseline) consistently get better gencralization performance than models trained using the whole training set (the none bascline) including observe that
mislabeled examples (figure 7] even if there is less examples overall. More interestingly, using a pipeline detect + filter with hyperparameters tuned using clean validation set allows to improve on generalization performance most of the time. For a few detectors, however, the detect + filter pipelines do not improve compared to the none baseline or the random baseline. We believe this could be improved by spending more effort to tune individual detectors.
coteet/nonee capacity In figure g we compare the regularization hyperparameter given by the oracle in the classifier of the none baseline to the same hyperparameter in the classifier of detect filter pipelines.
20

---

Published in Transactions on Machine Learning Research (09/2024)
NNAR 10% relabeling
Linear Classifier
HP tuned on noise free validation
300
250
normalized test loss
200
150
100
50
0
Forget scores (KLM)
random
Consensus (GBM)
Influence (KLM)
AUM (KLM)
TracIn (KLM)
VoSG (CBM)
Forget scores (GBM)
Cleanlab (GBM)
Cleanlab (KLM)
Small loss (GBM)
AUM (GBM)
Representer (KLM)
VoSG (KLM))
Consensus (KLM)
Small loss (KLM)
AGRA (KLM))
Figure 6: Distribution (boxplot) the normalized (base 100=training on correctly labeled examples only, base 200=training on all examples including mislabeled oncs) test loss relabeling 10% less trusted examples with varying detectors using a linear model estimator tasks (dols) corrupted by NNAR. Hyperparame- ters are tuned using clean validation set. Detector names include the detection method well as the base model. Respective colors assigned to cach detector are consistent across figures in the rest of this section.
We observe that most of the time, less regularization is needed in detect + filter pipelines, even if the trusted training set is smaller since untrusted examples have been removed. This is aligned with the intuition that noisy datasets require more robust machine learning models (i.e. with larger regularization).
Clean noisy validation set In our survey of mislabeled detection methods, we found that the question of the validation set was often overlooked: as for any machine learning application, the final performance crucially depends on set of hyperparameter, among which the threshold used to filter untrusted examples is paramount. We found that choosing hyperparameters on a noisy validation set gave no improvement compared lo the none baseline (figure Intuitively, this can be understood as the fact that since the noisy training and noisy validation set, what is actually noise does not look like noise. In practice, the threshold for splitting the training set often chosen to be (no filtering at all, see figure [O]. This questions the practical utility of mislabeled detection methods comparatively with the biquality setup (Nodet et al. 2021 ): in biquality learning, clean examples are used to actually learn the parameters of a model and simultaneously provide weak supervision on other (by default untrusted) examples whereas here, they are just used to choose hyperparameters, losing some useful signal.
follow the same (noisy) distribution, from the perspective of the validation
Detection performance final performance In previous experiments, evaluated mislabeled detec- tion methods by observing their performance in full pipclines (detect + filter or detect + relabel). This is the most relevant. metric in practice since this is how detectors will be used in most cases. In this experiment, we instead evaluate detection methods by measuring the predictive power trust scores to distinguish between genuine and mislabeled examples. We expect detectors that rank examples correctly would also lead to good classifiers trained on their most trusted examples. However, figure [1 only mild correlation is found between the two quantities, which suggests that good detectors possess other intrinsic qualities than their
21

---

Published in Transactions on Machine Learning Research (09/2024)
NNAR Filtering Linear Classifier
HP tuned on noise free validation
300
250
normalized test loss
200
150
100
0..
none
Consensus (GBT)
random
Forget scores (KLM)
Consensus (KLM)
Small loss (GBT)
Cleanlab (GBT)
Forget scores (CBT)
VoSG (GBT)
TracIn (KLM)
AUM (KLM)
Representer (KLM)
Small loss (KLM)
Influence (KLM)
Cleanlab (KLM)
VoSG (KLM)
AUM (GBT)
AGRA (KLM)
silver
gold
Figure 7: Distribution (boxplot) of the normalized (base 100=training on correctly labeled examples only, basc 200-training on all examples including mislabeled oncs) test loss detect + filter pipclines with varying detectors with linear final estimator lasks (dots) corrupted by NNAR. Hyperparamelers clean validation set. Detector names include the detection method well as the base model.
tuned using
75
97 72
132 below
82 regularization of detect + filter
10
Figure 8: For each detector pivector/dataan /daaset pair (a circle), we com- pare the oracle regularization (l2 regularization in a lincar model) chosen when using the whole corrupted training set on the x-axis, to the oracle regularization chosen in detect + filter pipeline on the y-axis. Most of the time, pipelines obtained smaller regularization (dots are below the y line).
10
10
10
10
10
82 regularization of none baseline
ranking capacity, such as their capacitics to seloct prototype examples or balance datascts that are otherwise class-imbalanced.
Weak is more difficult than noise We compare two sets of experiments on the same datasets: examples corrupted using artificial uniform noise (NCAR) and imperfect labeling rules (NNAR) in figure 12 Perhaps unsurprisingly, we observe that detect + filter pipelines perform worse on NNAR corruption than NCAR corruption. This expected as NNAR is notorionsly more difficult (Frénay & Verleysen 2013): indced, the patterns in the noise often indistinguishable from the patterns in the non-corrupted data from the
22

---

Published in Transactions on Machine Learning Research (09/2024)
Figure For each detector/datasel pair (a cross or cir- cle), compare the classifier obtained by detect + filter pipeline (y-axis) to the classifier trained on all examples, including noisy examples as bascline (x-axis). Hyperpa- rameters are tuned either on a clean validation set (crosses) or on noisy validation set (circles) for both the baseline and the pipeline. We observe a trend where detect fil- ter pipelines tuned on the noisy validation set give worse performance than the baseline (circles are above the y æ line), whereas pipelines tuned on the clean validation set give better performance than the baseline (crosses are be- low the y = x line).
test log loss (detect + filter)
*
clean valid.
noisy valid.
0.1
0.1
test log loss (no filtering)
1.0
Tuned using noisy valid. set
0.8
Filtering quantile
For each detector/dataset pair (a blue circle), we compare the split quantile obtained by cross-validation on a noisy validation set (y-axis) to the oracle (best) split quan- tile (x-axis). Choosing the split threshold using a noisy val-
Figure 10:
0.6
0.4
idation set consistently underestimates the optimal thresh- old. In fact, the threshold (no filtering all) is chosen most of the time.
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Filtering quantile Oracle
perspective of the learning algorithm in the absence of further hypotheses. More noteworthy, our experiments show no clear correlation between performance on NNAR and performance on NCAR. As a corollary, this questions the choice of testing mislabeled detection methods on NCAR-corrupted datasets: even if it is often more convenient to experiment on prevalent benchmarks and artificially introduce uniform label corruption, it might translate to actual use cases with more intricate NNAR corruption.
Adapting deep learning methods to classical machine learning algorithms A contribution of this benchmark also to evaluate mislabeled detection methods that were initially designed to work with deep learning models and that we adapted work generically with any model that is learned sequentially (more details in appendix A.. This is the case for AUM, Forget Scores, VoSG and TracIn. We evaluate them both with the gradient boosting algorithm and linear model learned by stochastic gradient descent (SGD), which our library allows to treat as progressive learning natively. Empirically, we report mixed results: whereas some methods did not produce any significant improvement in test loss, VoSG with a linear model is among the best- pers-reeformin methods. This demonstrates the feasibility of such an approach, which fosters future work on the similarities and differences of training dynamics between deep learning algorithms and other machine learning algorithms.
Choice of basc model and additive robustness We hypothesize that in detect + filter pipelines, using different machine learning models as base model in the detection stage and the final classifier could improve the final performance. The rationale is that different machine learning models include some form
23

---

Published in Transactions on Machine Learning Research (09/2024)
normalized test log loss
200
Figure 11: For cach detector/dataset pair (a circle), we com- pare the normalized (none=200, silver=100) test loss of de- tect + filter pipelines on the x-axis, to the ranking quality of the trust scores on the y-axis. Detectors with better ranking tends to produce filtered dataset that allows the training of better classifiers (the black line is robust. linear regression).
100
0.5
ranking quality
test log loss (NNAR dataset)
Figure 12: For each detector /dataset pair (a circle), we com- pare the task with NNAR corrupted labels (x-axis) to the same training examples but corrupted using NCAR labels (y-axis). This confirms that NNAR tasks are more difficult than NCAR most of the time (circles are above the y = æ line). This also shows that there is no clear correlation be- tween NCAR performance and NNAR performance.
0.1
0.1
test log loss (NCAR dataset)
of robustness different examples: we could benefit from the combined robustness by using 2 different methods. In figure 13 we compare the performance of detect + filter pipelines where both stages use either lincar model (KLM) or gradient boosted troe (GBT) model. We expect to see two point clouds pulled away from the x line, where KLM + GBT and GBT + KLM would perform better than KLM + KLM or GBT + GBT. We actually do not see such a pattern, which invalidates the hypothesis, at least in the current setup. Overall, is clear from this figure that KLM is the best choice as a base model in these experiments.
Representativeness of filtered data Being able to accurately sort out mislabeled examples from a dataset is a fundamental property that detectors should possess, which they do (Figure 11 However, their ranking capacity does not explain by itself the performance of model trained on a filtered dataset. We the quality of the filtered dataset, matters. We experiment
think that the representativity, in addition with proxy of representativity, the class balance. We expect that the class balance of the filtered dataset to be closer to the test one than the noisy one. Figure 14 shows that most of the time, detectors tend to favor examples from the majority class, or in other words, filter out more aggressively examples from the minority class. We propose two polential reasons for this bias. Firstly, detectors may struggle to distinguish between mislabeled instances and those of the minority class, as the latter are inherently scarcer and more challenging learn from. Secondly, the value among examples might not be evenly distributed. We speculate that failing detect mislabeled instance from the minority class could be more detrimental to the downstream taskthan failing to detect from the majority class.
24

---

Published in Transactions on Machine Learning Research (09/2024)
test log loss (KLM base model)
Figure 13: For each detector /datase pair (a blue orange circle), we compare (lower is better) the best classifier ob- tained when using & detector with a GBT model on the x-axis to the best classifier obtained when using a detector with KLM model on the y-axis. KLM detectors (orange circles) seem produce better performance most of the time (circles below the y = x line), and no clear pattern emerges to whether mixing models show trend (blue and orange circles do not show different patterns).
KLM cstimator
GBT estimator
0.1
0.1
test log loss
(GBM base model)
filtered class balance
Figure 14: For each detector/ /ataset pair (a circle), we com- pare the class balance of the original training dataset on the x-axis to the class balance of the filtered train dataset on the y-axis. Orange circles correspond to datasets where train- ing is less balanced than test, and blue circles correspond to datasets where training is more balanced than test. Even though the ratio of above and below circles varies by color, detectors tend to introduce more class imbalance than orig- inally found in the training dataset (circles are below the y line).
noisy class balance
Filtering by class So far, we have studied detect + filter pipelines where the filtering step performed regardless of the (potentially noisy) observed class of the example. As seen in the previous experiment, this can change the distribution of the class in the filtered dataset. Indeed, in class-ibbalaneed dataset, examples in the minority class often end up being less trusted than examples of the majority class. We thus experiment with the alternative strategy of filtering example class-by-class, where e.g. the top 50% most trusted examples of each class are kept for training. This ensures that the distribution between classes in the filtered dataset does not depart too much from that the original (noisy) dataset, thus avoiding a situotion where a minority class fully disappears from the training dataset. Figure 5.2, we observe no such trend: for most lasks does not make much difference, for some other lasks tiny difference in performance is observed, either in favor of filtering by class, or in favor of filtering all classes at once.
Detection performance vs base model performance Similar to the fact that the inductive bias of machine learning algorithms is paramount to their classification performance on unobserved examples, we study how much this inductive bias relates to their performance at detecting mislabeled examples. Figure 16 shows that on datasots where the base model gets its better classification performance (measured as the test loss the same model trained on the whole noisy training set), also better at detecting mislabeled examples, The trend is consistent across detectors.
25

---

Published in Transactions on Machine Learning Research (09/2024)
test log loss (filtering byclass)
Figure 15: For each detector pattetoordatasett dataset pair (a circle), we com- pare (lower is better) the best classifier obtained when filter- ing examples blass-b--clase
the x-axis to the best classifier obtained when filtering all classes at once on the y-axis. No clear picture emerges: is sometimes better to filter class- by-class, sometimes better to filter all classes at once, and sometimes it does not make much difference.
0.1
0.1
test log loss (fittering allclass)
1.0
0.9
Figure 16: For each detector across all datasets, we compare the performance at detecting mislabeled examples on the y-axis (AUROC for the task of detecting mislabeled exam- ples, higher is better) base model (measured using the test loss, lower is better) when trained using the whole training set including misla- beled examples. For each detector, we also plot a linear regression across datasets. We observe a trend where good classification performance correlates with good performance detecting mislabeled examples
ranking quality
0.8
the performance of the underlying
0.7
0.6
0.5
10
test loss of none baseline
5.3 Lessons learned
Our benchmark offers a critical view on the approach filtering the context of machine learning with mislabeled examples when working with labeling functions or uniform noise: fundamental flaw of such a methodology lies in the fact that it requires access clean validation set in order select hyperparameters (the most important one being the threshold for splitting into trusted and untrusted examples). With this in mind, it seems like waste of resources to use this clean data only for hyperparameter selection, rather than using it directly as training set, or in a biquality learning setup. Filtering also imposes hard threshold between trusted and untrusted examples, whereas in some cases, examples that are on the verge of the splitting threshold could also carry out some useful learning signal.
However, the results show some intoresting trends that could inspire future rescarch. In particular, it shows that some of the methods developed for deep learning algorithms (more specifically VoSG and AUM) also show promising results with other classical machine learning algorithms (here linear model and gradient boosting machine). On the contrary, Forget Scores do not seem to work very well with our current implementation.. Since we could not afford to spend too much effort on every method, it could simply mean that we did not find appropriate hyperparameters, or that the training dynamics of deep learning on which Forget Scores rely are different from the training dynamics of GBTs and linear models.
Our experiments show encouraging results for applying probing methods to text and tabular datasets. For small adnitional implementation cost, computing trust provides useful information about which
26

---

Published in Transactions on Machine Learning Research (09/2024)
examples look genuine and which examples require additional reviewing. However, there is no clear winner among the detectors. Even though AGRA appears to perform better on this series of datasets, the distribu- tions of normalized test log loss largely overlap, suggesting that the best performing detectors vary between datasots. This also provides room for improvement: since trust scores catch slightly different signals for different detectors, a natural extension might be to try and summarize several detectors into a single trust. score.
6 Other related works
This paper is part of a series of surveys the weakly supervised learning literature, specifically on learning with noisy labels (Frénay & Verleysen 2013 [an al.) 2020; Song al.) 2022). However, it sets itself apart from other surveys by focusing on the task of identilying mislabeled examples instead of studying more broadly the literature of learning algorithms robust to noisy labels. It also provides more modern view the mislabeled example detection literature (Guan & Yuan 2013) by proposing an encompassing framework closing the gap between approaches that were specifically developed for deep networks and classical machine learning algorithms.
Identifying mislabeled examples topic also found in the data cleaning literature [yyys & Chu 2019) and has recently been successfully applied to the growing text datasets used to train or line-tune large language models (e.g- Zhuu et al. 2024). Data cleaning surveys (Côté al.] 2023) also have broader scope than the one studied in this survey and are more comparable to the detect + filter pipeline but extended to other forms of data corruption such as feature noise, missing data, and outliers detection.
Furthermore, outlier detection is an important field related to mislabeled examples detection. These met are designed to identify outliers in the sense P(X), whereas mislabeled detection methods seek to find outliers in the sense X.) Outlier detection approaches have been applied to split non-scalar trust scores, for example, when using the output of multiple detectors (Lu al. 2023) or multiple probes where no apparent aggregation cxists. Another use of outlier detection is to find outliers in P(X(Y) instead of P(Y by training one outlier detection algorithm per class, assuming that outliers for a given class are mislabeled examples (Rebbapragada & Brodley 2007). We did not include these methods in the survey, as they were out of scope.
Finally, we omit a series of methods that jointly optimize the two sleps of the detect + handle pipeline. Most notably, these approaches work iteratively, akin to the expectation expectation-maxiiizaii maximization algorithm, where the detect and handle steps are optimized alternatively to minimize a global objective, usually getting the best possible classifier out of the handle step (Taakka et al. 2018 Zeng et al. 2022). Contrary to the iterative refinement from section 3.3.1 where proper trust scores can be explicitly retrieved at every iteration, joint methods use implicit trust scores. As they only serve the role to guide the optimization procedure, they lack intrinsic significance, defcating our primary goal of mislabeled example detection.
7 Conclusion
The tremendous size of training datasets in modern tasks advocates for cheaper labeling strategies (e.g. crowdsourced annotation or automatic labeling rules), at the cost of some degree of labeling error. Methods for detecting mislabeled examples offer the promise of being able to diagnose training datasets and review ex- amples in a sccond step, cither automatically (using filtering, scmi-supervised learning or biquality Icarning) or by manual relabeling. In this paper, we lake fresh look at past and recent detection methods, and show that most of them can be understood using a framework consisting of 4 components and few principles. Notably this includes recent methods that exploit the particular training dynamics of deep networks, which we extended to be classifier-gnnostic (in particular, in the empirical evaluation, we experimented with linear models and gradient boosted trees).
We proposed an implementation of this framework, that follows the scikit-loarn API which familiar to every machine learning practitioner. Our implementation focuses on the core mechanics of the framework, which then allows the implementation of the existing methods in the literature by only passing specific
27

---

Published in Transactions on Machine Learning Research (09/2024)
values for the 4 components. This demonstrates that this framework is not just abstract but can also be actually implemented. Using this framework, we proposed benchmark on a large number of tabular and text datasets, with some amount of labeling noise that comes either from uniform noise (NCAR noise) or from imperfect automatic labeling rules (NNAR noise). This benchmark is made available for reuse in the weakly supervised machine learning communityy with helpers for automatically fetching datasets with train/van main/validation// sattiioo//ttttttt test splits and fixed sceds for pscudo-randon generators.
This benchmark allowed us to provide set of new insights for machine learning in the presence of mislabeled examples. We also empirically verify common folklore in the field, specifically the difference between NCAR and NNAR setups, or the fact that cleansing datascts from mislabeled examples allows to use less regularized models. We highlight the often overlooked issue of the role of a clean validation set free from any source of labeling noise in automatic pipelines, questioning their usefulness in real use cases: if have access to clean examples, why not use them directly to learn the parameters of my model using a semisupervised algorithm or biquality learning?
Perspectives
The framework presented in section B. and the implementation that we distribute, suggest that there is room to improve the detection of mislabeled examples by experimenting with combinations of base model, probe, ensemble strategy, and aggregation methods that have not yet been explored. We encourage a systematic comparison of probes by looking specifically at which examples they score differently. Depending the context, it may be interesting to use mix of probes in order to improve the robustness of detection methods. More generally, the idea of using trained machine learning models in order to diagnose datasels of examples extends to other subfields of machine learning, such as active learning, example-base based explainability methods, or conformal prediction. This urges for cross-fortilization between communitics for a toolbox of methods that exlend machine learning algorithms order to obtain not only a single prediction (e.g. class or real value) but also additional information about that prediction.
Finally, advocate for more fino-grained score of cach instance than just being correctly mislaboled. As we discussed briefly in section 2.4 training examples can be categorized depending on whether they belong to a region of the input space with low or high aleatoric or epistemic uncertainty, and recent works aim at capturing this distinction (e.g. Javanmardi et al. (2024) using conformal prediction). Moreover, some examples might be more useful or harmful, either because they belong to a minority class or an underrep- dataset, or because they ward against some undesired bias
resented pattern in an otherwise class-balande of the training data, In some cases, these are the examples that are the most important ones, for instance, in fairness sensitive lasks. A more relevant metric would then be to use a measure of how useful is any given example to the prediction of learned classifier test examples of this minority group, such as the DataShapley value (Ghorbani & Zou 2019) using custom utility function.
References
Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating Example Difficulty Using Variance of Gradients. pp. 10368-10378, 2022. URL htttss:/ wtt.///..oo//ooeeeetttittitoteee cctp://coonteeee////////////a Agarwal Estimating Example Difficulty Using Variance_ of "___drrdddnns.... Stt___cVV___pppppee html
Hadi Amiri, Timothy Miller, and Guergana Savova. Spotting Spurious Data with Neural Networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2006 2016, New Or- leans, Louisiana, 2018. Association for Computational Linguistics. doi:
URL
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. Tn International conference on machine learning, pp. 233-242. 242. PMLR, 2017.
28

---

Published in Transactions on Machine Learning Research (09/2024)
Juhan Bac, Nathan Ng, Alston Lo, Marzych Ghassemi, and Roger Grosse. If Influence Functions are the Answer, Then What is the Question?, September 2022. URL http: //arxiv org/ wwwp/bgg////eeeeeee.. 22w. wttp:/bb///2...... arXiv:2209.03364 (cs, stat].
Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-NN for noisy labels. In Hal Daumé TTT and Aarti Singh (ods.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learwing Research, pp. 540 550. PMLR, 13 18 Jul 2020. URL htttps://proceeddnng mlr.
press/v119/barr1200 html
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprint orXi:2204.05866 2022.
Robert Baldock, Hartmut Macnnel, and Behnam Neyshabur. Doep learning through the lens of example difficulty, Advances in Neurol Information Processing Systems, 94:10876-1088) 2021.
Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory
training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pp. 1899-1909. PMLR, 2020.
Carla E Brodley and Mark A Friedl. Tdentifying mislabeled training data. Journal of artificial intelligence research, 11:131 167, 1999.
D.S. Broomhead and D. Lowe. Multivariable functional interpolation and adaptive networks Compler Systems, 2:321-355, 1988.
Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Ar- naud Joly, Brian Holt, and Gaël Varoquaux, API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pp. 108 122, 2013.
Haw-Shiuan Chang, Erik Learnod-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advonces in Neural Information Processing Systems, 30, 2017.
Bowen Chen, Yun Sing Koh, and Ben Halstead. Measuring Difficulty of Learning Using Ensemble Methods. Laurence A. F. Park, Heitor Murilo Gomes, Maryam Doborjoh, Yee Ling Boo, Yun Sing Koh, Yanchang Zhao, Graham Williams, and Simcon Simoff (eds.), Data Mining, Communications in Computer and Information Science, pp. 28 42, Singapore, 2022. Springer Nature. ISBN 978-981-19884-665. doi: 10.1007/ 178-981-19-8746-3 3.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang- Understanding and utilizing deep neural networks trainod with noisy labels. In International Conference on Machine Learning, pp. 1062- 1070. PMLR, 2019.
Hao Chong, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dopendent label noise: A sample sieve approach. arXiv preprint or 2i::2000002347, 2020.
R. Dennis Cook.
Delection of Influential Observation in Linear Regression. Technomelrics, 19(1):15 18, 1977. ISSN 0040-1706. doi: 10.2307 01.2307/126820 11668249. URL TTallor & Francis, Ltd., American Statistical Association, American Society for Quality).
org/stablee/268249 Publisher:
Filipe R Cordeiro, Ragav Sachdeva, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Longremix: Robust learning with high confidence samples in a noisy label enviromment. Pattern Recognition, 133: 109013, 2023.
Pierro-Olivier Côté, Amin Nikanjam, Nafisa Ahmed, Dmytro Humeniuk, and Foutse Khomh. Data cleaning and machine learning: A systematic literature review. ar Niv preprint orXim:2310. 0.01166, 2023.
29

---

Published in Transactions on Machine Learning Research (09/2024)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pallern recognilion, pp. Teee, 2009.
Dmitriy Dligach and Martha Palmer. Reducing the need for double annotation. In Proceedings of the 5th Linguistic Annotation Workshop, pp. 65-73, 2011.
Daniel D'souza, Zach Nussbaum, Chirag Agarwal, and Sara Hooker. A tale of two long tails. aix preprint arXiv:2107.1300 2021.
Rajmadhan Ekambaram, Sergiy Fefilatyev, Matthew Shreve, Kurt Kramer, Lawrence O Hall, Dmitry B Goldgof, and Rangachar Kasturi. Active eleaning of label noisc. Pattern Recognition, 51:463-488 480,, 2016.
Vitaly Feldman. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, pp. 954-959, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369794. doi: 10.1145/3357713. 3384290. URL httpp:://ai..org// 1145/3357713 3384290
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Benoît Frénay and Michel Verleysen, Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845 869, 2013.
Dragan Gamberger, Nada Lavrac, and Saso Dzeroski. Noise detection and elimination in data preprocessing: Experiments in medical domains. Applied Artificial Intelligence, ArtiffciiaIInteelligennee 14(2):205 223, February 2000. ISSN 0883- Publisher:
9514. doi: 1100000883951001100 Taylor & Francis
URL https:://oi. org/ 10. 1080/068395100117224
eprint: https://doi.orr//1 100000000000003550011111121
Thomas George, Guillaume Lajoie, and Aristide Baratin. Lany vs hasty: linearization in deep networks impacts learning schedule based on example difficulty. Transactions on Machine Learning Research, 2022.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pp. 2242-2251. PMTR, 2019.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35:507-522 520, 2022.
Donghai Guan and Weiwei Yuan. A Survey of mislabeled training data detection techniques for pattern classification. IETE Technical Review, 30(6):524 530, November 2013. ISSN 0256-4602. doi: 10.4103/ 1256-4602.1256888 URL https://www http://wwww..tddfnnnll..o www.aaannonnnnnocmmmmoo com/ httpee/eeae/.//////////0///////0
4wtp://000225444222252222.22 125689 Pub-
lisher: Taylor & Francis
eprint: http:///wwwww..dettoaaaa///////a
Donghai Guan, Weiwei Yuan, Young-Koo Lee, and Sungyoung Lee. Identifying mislabeled training data with the aid of unlabeled data. Applied Intelligence, (3) 358, December 2011. ISSN 1573-7497, doi: -100-22--0.--4444 URL. htttss://doi htt:://doo.orgg//// h1tp://0000000000000000000000000
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, (0(346):383-338 -33,, 1974.
Bo Han, Quamming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint trXiv:2011.044000 2020.
Michael A. Hedderich, David Adelani, Dawci Zhu, Jesujoba Alabi, Udia Markus, and Dictrich Klakow. Transfer learning and distant supervision for multilingual transformer models: A study on African lan- guages. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2580-2501, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/224 /2020.emmmlppp..m mailp..aai.aaam 204. URL https:///ccllatholoog..o////// emnlp main. 204
30

---

Published in Transactions on Machine Learning Research (09/2024)
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressed deep neural networks forget? xiviv preprinl or arXin:191 N1111911.05244 1.05248. 2019.
Eyke Hillermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110(3):457- March 2021. ISSN 1573-0565. URL
doi: 00.1007/110994-021-0555444
Thab F Tyas and Xu Chu. Data cleaning. Morgan Claypool, 2019.
Alireza Javanmardi, David Stutz, and Eyke Hüllermeier. Conformalized credal set predictors. orXiv preprint arXiv:2402.0 10723, 2024.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304 2313. PMLR, 2018.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. In International Conference on Machine Learning, pp. 5034 5044. PMLR, 2021.
Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Uni- Con: Combating Label Noise Through Uniform Selection and Contrastive Learning. pp. 9676 9686, 2022. URL /stpss:/oppeaccesss: thecvf. com/ http://ontntt//Paa/2////////aaaa
UniCon Combating.
Label Noise_ Through, Uniform Selection and Contrastive Learansiniie_LLLrnnngg.nnng Cesaarnnnn_CRRR,,, 2022_ 2012__aperr_.__ .hml
Jihye Kim, Aristide Baratin, Yan Zhang, and Simon Lacoste-Julien. CrossSplit: Mitigating Label Noise Memorization through Data Splitting, April 2023. URL http://arxiv, http://axxiv.oggabbb/////s/ 2tt.//222........ arXiv:2212.01674 [cs].
Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. In Proceedings of the 34th International Conference on Machine Learning, pp. 1885 1894. PMLR, July 2017. URL
Dttps://proceediee
mlr At....prres//0//ooo1//// html ISSN: 2640-3498.
Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving Training Biases via Influence-based Data
Relabeling. October 2021. URL http:://pppeeereeeeeeeeetoororrr
David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don't learn via memorization. 2017.
Johnson Kuan and Jonas Mueller. Model-agnostic label quality scoring to detect real-world label errors. In ICML DataPerf Workshop, 2022.
Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 3780-8802 PMLR, 2022.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels semi-supervised learning. In International Conference on Learning Representations, 2020.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6781-6792. PMLR, 18-24 Jul 2021. URL ptstp:://proceedngg..... wwwpiiins.mlr..r.ss/eeee wtt.pprres////11122222 .tmml
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Trans- actions on pullern analysis and machine intelligence, 38(3):447-446 461, 2015.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference on machine learning, pp. 6226 6236. PMT.R, 2020.
31

---

Published in Transactions on Machine Learning Research (09/2024)
Yang Liu, Hao Cheng, and Kun Zhang. Identifiability of label noise transition matrix. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.); Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of mlr.
Machine Learning Research, pp. 21475-21496. PMLR, 23-29 Jul 2023. URL httpt:t/tttscceeeeeeeeee press/v202/liu2gg.tr
Jie Lu, Anjin Lüu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12):2346 2363, 2018.
Yang Lu, Yiliang Zhang, Bo Han, Yiu ming Cheung, and Hanzi Wang. Label-noise learning with intrinsically long-taile tailed data. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), October 2023.
Ulrike von Luxburg and Bernhard Schölkopf. Statistical Learning Theory: Models, Concepts, and Results. In Dov M. Gabbay, Stephan Hartmann, and John Woods (eds.), Handbook of the History of Logic, volume 10 of Induclive Logic, pp. 651 706. North-Holland, January 2011. doi: 10. URL https: /:::::/wwwwt htwp//wwwcccco www.sscceecedirecce http:///tt.ceee.aaaa.a//////////
Xingiun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijew- ickrema, and James Bailey. Simensionality-Drivee Learning with Noisy Labels. In Proceedings of the 35th International Conference on Machine Learning, pp. 3355-3364. PMI.R, July 2018. URL https://proceeeding.. mlr. wwwps//w0mmmaaaaee./ee html
html ISSN: 2640-3498.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 140, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http::///wr http://ww.wwwwwb............. aclweb. org//athhologr//111-001
Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peer-prediction method. Management Science, 51(9):1359-133 1373, 2005.
Pierre Nodet, Vincent Lemaire, Alexis Bondu, Antoine Cormuéjols, and Adam Ouorou. From weakly super- vised learning to biquality learning: an introduction. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1 10. IEEE, 2021.
Curtis Northeutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimaling uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021a.
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Track on Dalasels and Benchmarks, December 2021b.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730 27744, 2022.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEFE conference. on computer vision and pattern recognition, pp. 1944-1952, 2017.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596 20607, 2021.
Fabian Pedregosa, Gaöl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825 2830,,880,-0 2011.
32

---

Published in Transactions on Machine Learning Research (09/2024)
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying Mislabeled Data using the Area Under the Margin Ranking. In Advances in Neural Information Processing Systems, volume 33, pp. 17044 17056. Curran Associates, Inc., 2020. URL hta1:/66000777771166611611111111 Abstract.
Liudmila Prokhorenkova, Gleb Gusev, Alcksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin, Catboost: unbiased boosting with categorical features. Advances in newral information processing systems, 31, 2018.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
Ali Rahimi and Benjamin Rocht. Random features for largo-scale kernel machines. Advances in neural informalion processing systems, 20, 2007.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29, 2016.
Umaa Rebbapragada and Carla E. Brodley, Class Noise Mitigation Through Instance Weighting. In Joost N. Kok, Jacek Koronacki, Raomon Lopez De Mantaras, Stan Matwin, Dunja Mladenič, and Andrzej Skowron (eds.). Machine Learning: ECML 2007, volume 4701, pp. 708-715. Springer Berlin Heidelberg, Berlin, Heidelberg, 2007. ISBN 978-3-540--4957--8 978-3-540--4956855 doi: 10.1007/9788 990099977------4955888-- 71. URL http://link.sprr/// com/10..cc0.c/ http:/001100/9778---5----------0 5_71
ISSN: 0302-9743, 1611-3344 Series Title:
Lecture Notes in Computer Science.
Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. End-to-end weak supervision. Advances Neural Information Processing Systems, 2021.
Hinrich Schütze, Christopher D) Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge, 2008.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In Artificial Intelligence and Statistics, pp. 838-846, 846. PMLR, 2015.
Anastasiaa Sedova, Lena Zellinger, and Benjamin Roth. Learning with noisy labels by adaptive gradient- based outlier removal. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi (eds.), Machine Learning and Knowledge Discovery in Databases: Research Track, pp. 237 253, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-43412-9,
Nabeel Seedat, Jonathan Crabbé, Ioana Bica, and Mihaela van der Schaar. Data-IQ: Characterizing sub- groups with heterogeneous outcomes in tabular data. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (ods.), Advances in Newral Information Processing Systems, volume 35, pp. 23660- 23674. Curran Associates, Inc., 2022. URL http::////prreddnggggn.rr.p...pp https//ccpppppfflle/eeeeeeeeeeee fiteerfflllepppp/////
Nicola Segata, Enrico Blanzieri, Sarah Jane Delany, and Pádraig Cunningham. Noise reduction for instance- based lcarning with a local maximal margin approach. Journal of Intelligent Information Systems, 35: 301-331, 2010.
Burr Settles. From theories to queries: Active learning in practice. In Active learning and experimental design workshop in conjunction with ATSTATS 2010, pp. 1-18. JMLR Workshop and Conference Proceedings, 2011.
Colin Shearer. The crisp-dm model: the new blueprint for data mining. Journal of data warehousing, 5(4): 13-22, 2000.
Michael R. Smith and Tony Martinez. Becoming More Robust to Label Noise with Classifier Diversity, March 2014. URL.http::/eee http://rxxxi..orr////////00.0. 1893 wxXiv:1403.1895 [es, stat).
33

---

Published in Transactions on Machine Learning Research (09/2024)
Hwanjun Song, Minseok Kim, and Jac-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International conference on machine learning, pp. 5907-5915. PMT.R, 2019.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with doep noural networks: survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint Optimization Framework for Learning with Noisy Labels, March 2018. URL http:://rxiyy wwwggbas//03...// 11364 arXiv:1803.1136
(cs, stat].
Jaree Thongkam, Guandong Xu, Yanchun Zhang, and Fuchun Huang. Support Vector Machine for Outlier Detection in Brcast Cancer Survivability Prediction. In Yoshiharu Ishikawa, Jing He, Guandong Xu, Yong Shi, Guangyan Huang, Chaoyi Pang, Qing Zhang, and Guoren Wang (eds.), Advanced Web and Network Technologies, and Applications, Lecture Notes in Computer Science, pp. 99-109, Berlin, Heidelberg, 2008. Springer. ISBN 978-3-540-89376-9. doi:
10.
Zhiyi Tian, Lei Cui, Jie Liang, and Shui Yu. A comprehensive survey on poisoning attacks and countermea- sures in machine learning. ACM Computing Surveys, 55(:):1-35 35, 2022.
Ivan Tomek. An Experiment with the Edited Nearest-Neighbbor Rule. IEEE Transactions Systems, Man, and Cybernetics, SMC-6(6):448-4P5 1976. doi:
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Ge- offrey Gordon. An Empirical Study of Example Forgetting during Deep Neural Network Learning. September 2018. URL httpps://openreviev http://perrriiee..eeef..oor///// .ett/fforrmiiiccB3mm0m........
Hans van Halteren. The detection of incousistency in manually tagged text. In Proceedings of the COLING- 2000 workshop linguistically interpreted corpoma, pp. 48 05,55,- 2000.
Brendan Van Rooyen and Robert C Williamson. A theory of learning with corrupted labels. J. Mach. Learn. Res., 18(1):8501-855 8850, 2017.
Robert A Vandermeulen and Clayton D Scott. An operator theoretic approach to nonparametric mixture models. arXiv preprint arXiu: Nfl::667700071. 2016.
Sofie Verbaeten and Anneleen Van Assche. Ensemble Methods for Noise Elimination in Classification Problems. In Terry Windcatt and Fabio Roli (ods.), Multiple Classifier Systems, Lecture Notes in Computer Science, pp. 317-325, Berlin, Heidelberg, 2003. Springer. ISBN 978-3-540-41938-6 doi: 32.
Haobo Wang, Ruixuan Xiao, Yiwen Dong, Lei Feng, and Junbo Zhao. Promix: combating label noise via maximizing clean sample utility. arxiv preprinl 2xXi:2220.1027% 2022.
Ruxin Wang, Tongliang Liu, and Dacheng Tao. Multiclass learning with partially corrupted labels. JEEE transactions on neural networks and learning systems, (9(6)22568 2580, 2017.
Zige Wang, Wanjun Zhong, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Lifeng Shang, Xin Jiang, and Qun Liu. Data management for large language models: A survey. arXiv preprint arXiv:2312.01 01288122.17000 700, 2023.
D Randall Wilson and Tony R Martinez. Reduction techniques for instance-based learning algorithms. Machine learning, 38:257-285 286, 2000.
Dennis Wilson. Asymptotic properties of nearest neighbor rules using edited data. IEFF Transactions on Systems, Man, and Cybernetics, (3):408442)) 1972.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Arc anchor points really indispensable in label-noise learning? Advances in neural information processing systems, 32, 2019.
34

---

Published in Transactions on Machine Learning Research (09/2024)
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Aduances in Neural Information Processing Systems, 33:7597- 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang- Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 2691 2691-2699, 2015.
Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance- dependent bayes-label transition matrix using deep neural network. In International Conference Machine Learning, pp. 25302-25312. PMLR, 2022.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing cstimation crror for transition matrix in label-noisc learning. Advances in neural information processing systems, 33:7260 7271, 2020.
Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in newral information processing systems, 31, 2018.
Suqin Yuan, Lei Feng, and Tongliang Liu. Late stopping: Avoiding confidently learning from mislabeled examples, 2023.
Man- Maan-Ching Yuen, Irwin King, and Kwong-San Sak Leung. A survey of crowdsourcing systems. In 2011 IEEE third international conference on privacy, security, risk and trust and 2011 TEEE third international conference on social computing, pp. 766 773. IEEE, 2011.
Yi Zeng, Minzhou Pan, Himanshu Jahagirdar, Ming Jin, Lingjuan Lyu, and Ruoxi Jia. How to sift out a clean data subset in the presence of data poisoning? arXiv preprint arXiv:2210 06516, 2022.
Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. WRENCH: A comprehensive benchmark for weak supervision. In Thirty-fifth Conference on Neural In- formation Processing Systems Datasets and Benchmarks Track, 2021. URL
net.
Hang Zhou, Jonas Mueller, Mayank Kumar, Jane-Ling Wang, and Jing Lei. Detecting errors in numerical data via any regression model, 2023.
Zhaowei Zhm, Zihao Dong, and Yang Liu. Detecting corrupted labels without training model to prodict. In International conference on machine learning, pp. 27412 27427. PMLR, 2022.
Zhaowei Zhu, Jialu Wang, Hao Cheng, and Yang Liu. Unmasking and improving data crodibility: A study with datasets for training harmless language models. In The Twelfth International Conference Learning Representations, 2024. URL http:::://peeeeeee..eeeeeeereeee
35

---

Published in Transactions on Machine Learning Research (09/2024)
A Implementation details and comments regarding specific detection methods
A.1 Variance of gradients
): the toy experiment involves MIP with no non-lincarities, which computes a lincar mapping of the imput vector. In this case, the gradient of the logit w.r.t. the input vector only depends on the equivalent weight matrix W = WWW.... WW2...Wy Wi, and not on the input. Put differently, the proposed VoG statistics is the same for every example, which means that it cannot used to rank examples. We thus think that there is an inconsistency in the results presented in the toy experiment section. This is in contrast to the larger scale experiment with ResNet architectures, where the mapping from the input space to the logits is non-linear since it involves non-linearities (here ReLU activations).
We found small inconsistency in the experiments in Agarwal et al. (022
When working with linear models in our experiments, instead implemented slightly different version where we differentiate the probability given by the softmax, and not the logit. This mapping from input space to softmax output is non-lincar, and it depends on the example contrarily to the mapping from input space to logit. We found the resulting statistics (i.e. the variance gradients of the probability given by the softmax) be useful detecting mislabeled examples.
A.2 Finite differences
During our survey, we reviewed some detectors which where fundamentally tailored work on differentiable machine learning models. For example, the Variance Of Gradients detector probes the model by looking the derivative of the pre-soflmax layer of neural network with respect to the inpul features. We proposed in the library to use the finite difference approach for non- non-differentiable models such as decision trees. On the same 2D toy-dataset used in the original paper (Agarwal et al. 2022), the finite difference method showed to reasonably approximate the exact method for a kernelized linear model:
Try Dataset trained decision boundary
Distonce vs V&G (Pintte Dif!) score
score
Distance vo VDG (Uineai) score
3.0
30-
255
2.5
2.0-
3.5
1.0-
-3-
0.5-
-10002-0 000002--d00----- dope--.dioo losz
-0.00s -odoee -odase -edom tocos 1o0acamo
Figure 17: We reproduce the experiments from figure in Agarwal al. (2022) with kernelized linear model and finite difference approximation.
Thus it can be instantiated with the user's favorite progressive ensemble such as gradient boosted trees.
B Benchmark details
B.1 Generating noisy label from labeling rules
The datascts used in the benchmark provides, top of ground truth labels for cach examples, a set of weak labels given by labeling rules. Given an example, a labeling rule outpul a weak label, that may be class label the rule matched, or nothing. These weak labels are aggregated thanks to a majority vote. However two odge cases may arisc, the first the votes are tied between two classos, the second when no labeling rules matched for an example. In the first case we pick class completely at random among tied winners, for the latter we chose to drop these examples from the training dataset.
36

---

Published in Transactions on Machine Learning Research (09/2024)
Dropping unmatched examples is not consensus among the weakly supervised learning literature, the usual approach is to assign them a random label. We think dropping is more sensible approach, akin to what practitioners would do in practice, than the random assignment.
It should be noted that such noisy labels generate a non-squared transition matrix as the set noisy classes is equal to the set of original classes plus the unlabeled class. In the figures from table the unlabeled noisy class corresponds the last row of the transition matrix T.
For example, the youtube dataset, the noise transition matrix is the following:
0.79
0.22
0.04
0.74
0.17
0.04
The columns corresponds to true labels and the rows corresponds to noisy labels. The element in the second row and first column means that 4% of examples from class have been assigned the noisy labels 1. The last element of the last row means that 4% of examples from class 1 have been assigned no noisy labels.
The implementation of weak label encoding available in the open-source library, see section
B.2 Feature engineering
In order to have a fair starting ground between the different machine learning models used in section 5 datasets have been preemptively encoded to only contain numerical attributes.
all
Two different feature engineering pipelines are applied to each dataset, depending if it's text or tabular dataset.
For text datasets, TF-IDF features Schütze et al. 2008 are generated and l2 normalized. The size of
vocabulary is chosen on a per dataset basis to balance between accuracy and compute time. For tabular datasets, categorical features are one-hot encoded and numerical features are normalized.
Each component of the feature engineering pipeline uses scikil-learn's implementation.
B.3 Details regarding machine learning models
Two families of models are used through the benchmark, kernelized linear models (KLM) and gradient boosted trees (GBT), two of the most popular approaches for machine learning on tabular data.
To have scalable KLMs, we chose to use the Random Kitchen Sinks approach (Rahimi Recht, 2007) approximate kernel computations on large-scale datasets. We used the Gaussian RBF kernel (Broomhead & Lowe 1988) for tabular datasets and lincar (or no) kernel for text datasets. Then, the lincar model is trained by minimizing the log-loss training samples thanks to Stochastic Gradient Descent. All KLMs components use scikit-learn's implementation.
For GBTs, we chose the CatBoost (Prokhorenkova et al. 2018 implementation mainly because of its fast training time on GPUs.
B.4 Hyperparameters Sampling
We summarize the search space used in random search of the main hyperparameters of each family of models described in section B.3 in the following table:
On top of that models from both families are trained for a maximum a thousand iteration (mumber of epochs for KLM, number of trees for GBT), ensuring convergence in most cases. Yet, models can be early stopped if their log-loss on an holdout dataset does not decrease for more than iterations. This specific holdout dataset corresponds to 10% of the training data.
37

---

Published in Transactions on Machine Learning Research (09/2024)
Hyperparameter
Search space
kernel bandwith
(ad))((()))
KLM
lz regularization learning rate
log-uniform [le 5, le 1]
log-uniform [Te 3, 1]
GBT
& regularization learning rate
uniform [0, 100]
log-uniform [le - 5, le - 1]
Table Table of hyperparameters.
C
Additional results
C.1
Relabeling
NNAR 10%
relabeling
GBT Classifier
HP tuned noise free validation
300
250
normallzed test loss
200
f000
150
100
random
AUM (GBT)
VoSG (GBT)
Forget scores (GBT)
Cleanlab (GBT)
Forget scores (KLM)
Consensus (GBT)
Small loss (GBT)
VoSG (KLM)
Cleanlab (KLM)
Influence (KLM)
Consensus (KLM)
Representer (KLM)
TracIn (KLM)
AUM (KLM)
Small loss (KLM)
AGRA (KLM)
Figure 18: Distribution (boxplot) the normalized (base 100=training on correctly labeled examples only, basc 200=training on all examples inchuding mislabeled ones) test loss relabeling 10% less trusted examples with varying detectors using GBT model estimator on tasks (dots) corrupted by NNAR. Hyperparam- eters tuned using clean validation set. Same figure f but with GBT estimator.
38

---

Published in Transactions on Machine Learning Research (09/2024)
NCAR 10% relabeling
Linear Classifier
HP tuned on noise free validation
300
250
normalized test loss
200
150
8
100
50
random
gb vosg
gb forget
klm forget
klm vosg
gb. consensus
klm. consensus
gl. cleanlab
gb aum
agra
klm. num
klm representer
tracin
klm cleanlah
Figure 19: Distribution (boxplot) of the normalized (base 100=training on correctly labeled examples only, base 200-training on all examples including mislabeled ones) test loss relabeling 10% less trusted examples with varying detectors using a linear model as estimator on tasks (dots) corrupted by NCAR. Hyperparam- eters are tuned using clean validation set. Same figure but with NCAR noise.
C.2 All detectors
NCAR
Filtering Gradient Boosting Classifier
HP tuned on noise free validation
300
250
nomaalized test loss
200
150
100
foo
to
50
wood
none
random
gb. vosg
gb. fonget
klm forget
klm vosg
gb. consensuus
gb aum
gb cleanlab
klm cleanlab
agra
klm amm
klm. representer
silver
tracin
klm consensus
gold
39

---

Published in Transactions on Machine Learning Research (09/2024)
Figure 20: Normalized test loss of detect filter pipelines with varying detectors with GBT final estimator tasks corrupted by NCAR. Hyperparameters are tuned using a clean validation set. (same as figure but using& a GBT final classifier)
NNAR Filtering
Linear Classifier
HP tuned on noisy validation
300
250
6loe @
normallzed test loss
200
80
150
100
0
VoSG (GBT)
Representer (KLM)
Consensus (KLM)
Forget scores (GBT)
AUM (GBT)
VoSG (KLM)
Consensus (GBT)
Cleanlab (GBT)
Cleanlab (KLM)
Small loss (GBT)
Small loss (KLM)
AGRA (KLM)
AUM (KLM)
TracIn (KLM)
Forget scores (KLM)
Influence (KLM)
none
random
silver
gold
Figure 21:
Normalized test loss of detect + filter pipelines with varying detectors with linear final estimator on tasks corrupted by NNAR. Hyperparameters are tuned using a noisy validation set. (same as figure but using a noisy validation set for hyperparameter selection)
C.3 Filtering threshold noisy validation set vs oracle
1.0
Tunod using noisy valid. sot
0.8
Filtering quantile
For each /ddtattordatasct pair (a blue circle), we compare the split quantile obtained by cross-validation on a noisy validation set (y-axis) to the oracle (best.) split quan-
Figure 22:
0.6
0.4
tile (x-axis). Choosing the split threshold using noisy val-
idation set consistently underestimates the optimal thresh- old. In fact, the threshold (no filtering at all) is chosen most of the time. (same as figure 10 but using a GBT final estimator)
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Filtering quantile Oracle
40

---

Published in Transactions on Machine Learning Research (09/2024)
C.4 Regularization of none pipelines
88
52
164
~
below
regularization of detect + filter
100
Figure 23: For each detector/datasel pair, compare the oracle regularization (62 regularization in GBT model) chosen in detect + filter pipeline, to the oracle regulariza- tion chosen when using the whole corrupted training set. Most of the time, detect + filter pipelines obtained smaller regularization, meaning that filtering noisy examples allows for less regularized classifiers. (same figure using dif- ferent final estimator)
80
60
40
20
50
100
8g regularization none baseline
Other data modalities
While we mainly focus on text and tabular data. as these represent an important use case of machine learning techniques, include the following additional experiments, where showcase our library applied to different data types.
D.1 Results on a regression task on tabular data
We experiment on the California Housing regression task, that consists in predicting the price in the housing market (a real number) using several feat of the sold house. Here the same framework of section 3 can used provided that we use a regression probe. We use a model-probing detector with a random forest regressor base model, bootstrapping as the ensemble strategy, the & loss as the probe and the mean across out-of-bag examples the aggregation method. As illustrated by the bottom-5 trusted examples circled in red in figure D.1 the detector correctly identifies some examples with suspicious labels.
41

---

Published in Transactions on Machine Learning Research (09/2023)
5
8
6
AveOccup
3
Price
4
3
2
1
0.0
2.5
5.0
7.5
10.0
12.5
15.0
Medinc
Figure 24: Bottom-ō trust scores on the California housing dataset.
D.2 Results an image classification task with features from a pre-trained ResNet
We use the 50.000 images of the CIFAR10 classification task. We extract features before the classification layer of a ResNet50 model, pre-trained on the ImageNet dataset. Using these features, we choose a Logis- ticRegression model from scikit-loarn with defanlt hyperparameters as our base model, and TracIn as our detector.
In figure 25 for each class, show the less trusted image (in red), well as 4 other images of the same class. As expected, the most untrusted image often looks less representative of the observed class, or even mislabeled (e.g. windsurfer in class "ship", a dog that looks like a cat, or a zoomed-in image of the front of a truck whereas other images of trucks often display full trucks with their trailer).
automoooe e
airplane
bird
cat
dog
42

---

Published in Transactions on Machine Learning Research (09/2024)
horse
truck
Figure 25: On CIFAR10, for each class we display the less trusted training set image (red frame), compared to other representative images from the same class.
D.3 Results on an image classification task with confusing classes
We use 10.000 images from the Animal-10N classification task (Song et al. 2019 The setup is similar to the previous section D.2, where we extract features before the classification layer of a ResNet50 model, pre-trained on the ImageNet dataset. Using these features, we choose a LogisticRegression model from scikit-learn with default hyperparaneters as our base model, and TracIn as our detector.
In figure 26 for each class, we show the less trusted image (in red), as well as other images of the same class. Each row represents very similar classes (c.g. cats often look like lynxes, wolves like coyotes, etc). In this setup, less trusted images correspond to very unusual pictures, such drawings of a cheetah and wolf. In contrast, other images are real photographs, a coyote seating in a bus whereas other coyote images have more usual backgrounds.
Woll
coyote
cheetah
chimpanzee
GREAT
hamster
Figure 26: On Animal-1ON, for each class we display the less trusted training set image (red frame), compared 4 other representative images from the same class. Left and right classes correspond looling classes that more likely to be confused.
43


Reviews:
[{"review_id": "k7nK2aDyhC", "reviewer": "Reviewer_tFQi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes a general framework for noisy label detection that can integrate various existing techniques. The Python implementation provides valuable support for research in this field.\n2. The empirical studies are thorough, usding various benchmark datasets, and the findings are convincing.\n\nWeaknesses:\n1. The proposed framework follows a basic procedure for noisy label detection. While the empirical findings are valuable, the theoretical contribution is somewhat limited. Additionally, the ensemble module is primarily an engineering solution rather than a novel theoretical advancement.\n2. The experiments are primarily conducted on benchmark datasets with artificially generated label noise. The performance of the proposed method with different modules remains unclear on real-world noisy labeled datasets, such as the Animal10N dataset [1].\n\n[1] Song, H., Kim, M., and Lee, J., \"SELFIE: Refurbishing Unclean Samples for Robust Deep Learning,\" In Proc. 36th Int'l Conf. on Machine Learning (ICML), Long Beach, California, June 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of detecting mislabeled data and proposes a general framework composed of four modular components that can integrate various existing techniques. The main contribution is a Python library that precisely implements these principles and benchmarks existing methods against both random and non-random label noise across a series of datasets with automated labeling rules. The experiments are conducted fairly to me, and the key findings offer valuable insights for advancing research in this area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is recommended to test the proposed framework on real-world datasets with noisy labels, such as the Animal10N dataset, to strengthen the applicability of this work in practical scenarios.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vWGNYoRU3g", "reviewer": "Reviewer_rwvM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents a unified, modular framework for detecting mislabeled examples in datasets, and releases a library for fast implementation and benchmarking.\n\n2. The paper offers a thorough review of existing mislabeled detection methods, providing valuable insights into their strengths and weaknesses.\n\n3. The empirical results reveal important findings, such as the limitations of existing methods in handling complex noise structures and the potential benefits of certain detection strategies. These insights are valuable for guiding future research in this area.\n\n**Weaknesses**\n\n1. Dependency on Clean Validation Sets: The effectiveness of the detection methods often hinges on the availability of clean validation sets for hyperparameter tuning. In real-world scenarios where such clean data may not be accessible, the practical utility of these methods could be significantly constrained.\n\n2. Omission of Recent Works: The paper's survey, as illustrated in Table 1, primarily focuses on methods developed no later than 2023, with most benchmarked detection methods dating before 2022. To offer a truly comprehensive survey, it is essential to include more recent works, such as the detection method for text data discussed in [R1]. Furthermore, given the paper's emphasis on text data, benchmarking \"natural\" label noise in popular RLHF datasets, as examined in [R2], would be particularly relevant.\n\n3. Information Loss in Stochastic Case Handling: In Section 2.1.2, titled \"Stochastic Case: The True Concept is Defined as a Probability Distribution,\" the paper discusses using a threshold to convert a probability into a hard label. This approach results in significant information loss, potentially limiting the effectiveness of the method.\n\n[R1] Unmasking and improving data credibility: A study with datasets for training harmless language models. ICLR 2024.\n\n[R2] Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The paper introduces a novel modular framework for detecting mislabeled examples in datasets. This framework is parameterized by four core components, providing a unified perspective that encompasses most existing mislabeled detection methods.\n\n2. A comprehensive survey of existing mislabeled detection methods is provided, where these methods are categorized and compared within the proposed framework. The survey includes methods designed for both deep learning models and classical machine learning algorithms.\n\n3. The authors contribute a Python library that implements the modular framework. \n\n4. The paper includes an extensive empirical evaluation, benchmarking a variety of mislabeled detection methods on datasets with different types of labeling noise.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please revise the paper according to the comments above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vYtmb1FfZA", "reviewer": "Reviewer_AptK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Clear theoretical derivation and formula presentation.\n2. A large number of experiments and visualizations of charts for clear visualization of methods and results.\n3. Providing a Python library, contributing to the community, and future work.\n\nWeaknesses:\n1. Limited discussion on vision data: The article mainly focuses on tabular and text data, with limited discussion on other modalities data, for example, vision data.\n2. Insufficient discussion of mislabeled example detection applications. For example, data used to train large language models and data generated by large language models which are current research hotspots.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Comprehensive survey for mislabeled examples detection in text and tabular data, including model, model probe, ensemble strategy, and aggregation method.\n2. Providing a python library that integrates mislabeled examples detection framework, making the whole framework clear.\n3. Providing a benchmark on text and tabular data, and conducting comprehensive experiments and analysis on the benchmark.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Expand the discussion on mislabeled examples in other modality data, especially vision data, including semi-supervised visual methods and their relation to fully supervised methods. Also, there are some interesting tasks, such as concealed object detection, that may provide more insights. This would make the survey more comprehensive and informative.\n2. More discussion about mislabeled examples' influences on different tasks. For example, the use of unfiltered train data in large language models and multimodal large language models, and the potential issues arising from data generated by large language models and multimodal large language models. This will provide a more in-depth understanding of the challenges and opportunities of mislabeled example detection for various tasks. For example, halluciation in large language models and multimodal large language models now seriously affects models' performance.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers were unanimously in favour of acceptance. Following the discussion, one remaining comment was that the paper is rather long; the authors may wish to consider whether some material can be relegated to the Appendix, to aid the reader.
Decision: Accept (poster)