Regretful Decisions under Label Noise
Authors: Sujay Nagaraj
Yang Liu
Flavio Calmon
Berk Ustun
Venue: ICLR 2025 Poster (2025)
Keywords: Uncertainty Quantification
Fairness
Model Multiplicity
Clinical Decision Support
Classification
Label Noise
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Machine learning models are routinely used to support decisions that affect individuals -- be it to screen a patient for a serious illness or to gauge their response to treatment. In these tasks, we are limited to learning models from datasets with noisy labels. In this paper, we study the instance-level impact of learning under label noise. We introduce a notion of regret for this regime, which measures the number of unforeseen mistakes due to noisy labels. We show that standard approaches to learning under label noise can return models that perform well at a population-level while subjecting individuals to a lottery of mistakes. We present a versatile approach to estimate the likelihood of mistakes at the individual-level from a noisy dataset by training models over plausible realizations of datasets without label noise. This is supported by a comprehensive empirical study of label noise in clinical prediction tasks. Our results reveal how failure to anticipate mistakes can compromise model reliability and adoption -- we demonstrate how we can address these challenges by anticipating and avoiding regretful decisions.

Published as a conference paper at ICLR 2025
REGRETFUL DECISIONS UNDER LABEL NOISE
Sujay Nagaraj University of Toronto
Yang Liu UC Santa Cruz
Flavio P. Calmon
Berk Ustun UC San Diego
Harvard SEAS
ABSTRACT
Machine learning models are routinely used to support decisions that affect indi- viduals - be it to screen a patient for a serious illness or to gauge their response to treatment. In these tasks, we are limited to learning models from datasets with noisy labels. In this paper, we study the instance-level impact of learning under label noise. We introduce a notion regret for this regime, which measures the number of unforeseen mistakes due to noisy labels. We show that standard ap- proaches to learning under label noise can return models that perform well at a population-level while subjecting individuals a versatile approach to estimate the likelihood of mistakes at the individual-level from noisy dataset by training models over plausible realizations of datasets without label noise. This is supported by a comprehensive empirical study of la- bel noise in clinical prediction tasks. Our results reveal how failure to anticipate mistakes can compromise model reliability and adoption we demonstrate how we can address these challenges by anticipating and avoiding regretful decisions.
lottery of mistakes. We present
INTRODUCTION
Machine learning models are routinely used to support or automate decisions that affect individuals be it to screen patient for mental illness [52]. assess their risk for adverse treatment response [3]. In such tasks, we train models with labels that reflect noisy observations of the true outcome wish to predict. In practice, such noise may arise due measurement error [e.g., 23, 39]. human anmotation [30], or inherent ambiguity [39]. In all these cases, label noise can have detrimental effects model performance [11]. Over the past decade, these issues have led to extensive work on learning from noisy datasets [see e.g., 11, 32, 40, 44, 49]. As a result, we have developed foundational results that characterize when label noise can be ignored and algorithms to mitigate its detrimental effects.
By and large, this work has focused on the impact of label noise the population-level. In contrast, studying the effects of label noise at the instance-level has received limited attention. This oversight reflects the fact that we cannot provide meaningful guarantees on individual predictions under label noise [32]. In best-case scenario, where we have perfectly specified distributional assumptions on label noise, we can learn a model that performs well on average, but we cannot identify where makes mistakes; as result, individuals are subject to a "lottery of mistakes."
These effects undermine the utility of models in major real-world applications, as label noise arises in many settings where models are used to support automate individual decisions [see, e.g., 55, for a meta-review of 72 cases in medicine]. mistakes can lead to overreliance, where physicians rely predictions that may be incorrect [7, 29]. In automation tasks, our failure to assess the confidence of predictions can prevent us from reaping broader benefits e.g., by abstention [10, 18).
medical decision support tasks, our inability to identify
In this work, we study how label noise affects individual predictions. Our motivation stems from the fact that, even if we cannot fully resolve the effects of label noise at the instance-level, we can mitigate harm by anticipating regretful predictions through uncertainty quantification. To this end, our main contributions are:
1. We introduce notion of regret for learning from noisy datasets, capturing how label uncertainty affects individual predictions. We show that learning under label noise leads to inevitable regret, characterizing key limitations in wide class of methods for learning from label noise.

---

Published as a conference paper at ICLR 2025
Noiey Labels
y
Predictions
[in))
B/ ED Noise Draw
"Mistakee"
f(z))
True Labels
Regret
1/ I Noise Drow
Noley Labels
Predictions
f(n))
"Mistakes"
far.) #
0
0
Ragrot
Figure Datasets with noisy labels only contain single draw label noise. such settings, we can learn a model that performs well at a population-level but cannot anticipate its mistakes. We characterize the number of individuals who are subjected to lottery of mistakes in terms of regret i.e. the difference between anticipated mistakes and actual mistakes. Here, show stylized classification task with 5 points, where each point with positive label may he flipped with probability of 30%. In this case, 4 points are subject to lottery of mistakes and our model assigns regretful predictions to points, highlighted in yellow.
We develop a method to flag regretful predictions by training models on plausible realizations of clean dataset. Our approach can measure the sensitivity of individual predictions under label noise and incorporates common noise assumptions while controlling for plausibility.
We conduct a comprehensive empirical study on clinical prediction tasks. Our findings highlight the instance-level impact of label noise, and demonstrate how our approach support safer inference by flagging potential mistakes.
Related Work Our work is related to a stream of research on learning from noisy labels. We focus on applications where we cannot resolve label noise by acquiring clean labels [see e.g. 11, 49, for surveys]. Many methods learn models by hedging for uncertainty in labels [33, 40, 44]. As we show in Section 2, such approaches are robust to label noise at population-Ievel while subjecting individuals to lottery of mistakes, Our work highlights the limitations of this regime. In this sense, our results complement the work Oyen et al. [43], who characterize the lack of robustness to label noise under general distributional assumptions.
We propose to mitigate these issues through a principled approach to uncertainty quantification. Our approach relates to recent work model multiplicity, which shows how changes in the machine learning pipeline can produce models that assign conflicting predictions [see e.g., 4, 8, 20, 35, 38, 42, 53) and lead downstream effects fairness, explanations, and recourse [5, 17, 27,36) 36). With respect to the literature on label noise, our approach is similar to the work of Reed et al. [47], who propose training an ensemble of deep neural networks by sampling alternative realizations of clean labels. In contrast, our procedure samples plausible realizations of clean labels and retrains plausible models to quantify uncertainty at an individual-leww rather than predict.
2 PRELIMINARIES
We consider classification task where we wish to learn a model f X' + 3 to predict a label y from a feature vector æ X' C Rd In standard regime, would be given dataset D {1) where each (æ; M/) is drawn from a joint distribution of random variables, X and Y. Given the dataset, we would learn model that performs well in deployment ie.. that minimizes the true risk R(f) Ex,y (XX) # Y]].
We consider a variant of this task where we learn model from noisy dataset D { ū.) D.) where each noisy label y represents a potentially corrupted true label y:- In what follows, we refer to this corruption labels in terms of true labels as y := Du, and vice-versa as y; == y Ou, Here, ab = a+b-2al 2ab the XOR operator. Given noisy dataset, we represent all flips as vector called the noise draw.
fip and denote it u; [य4 * 5t]- Given the flip u;
express noisy
2

---

Published as a conference paper at ICLR 2025
Definition 1. Given binary classification task with examples, the noise draw [u,,.... Uny S {0, 1)" is a realization of n random variables [U1,..., Un] S {0,1)"
Given an example (a:;, Vi), each flip u, is drawn from a Bernoulli distribution with parameters Pul Vely, = Pr(U; = n.. Y yc). Thus, the noise is generated by the random process:
U: Bernouilli (Pllg....)
y: y U2
In what follows, we assume that the values Puly: a. determined by generic noise model that can take on different forms e.g., uniform, class-level, or instance-level as shown in Table 1. We write Pa instead of Puly..w, when conditioning terms irrelevant clear from context. We assume that the model is correctly specified and that pa. < 0.5 for all points to ensure there are more clean than noisy labels [c.f., 40, 44].
Given a noisy dataset, we denote the noise draw over all instances as the true draw w [uTue ..... u liue]. In practice, the true draw tree fixed but unknown. From the practitioner's perspective, u mee could be any realization of random variables U. If they knew u the they could recover the true labels as y/i Oi ① t!te and learn without label noise. As this is infeasible, given the noise model and set of priors, practitioners can estimate the posterior noise model Pr(U, = X Ti Ý ūi) to infer clean labels from observed noisy labels.
Zu ú.
Naioe Type
PGM
Naise Model
Postcior Modd
Inference Requirenents
Sample Use Case
Uniften
- (07
Pt (7 -
Screzting teso with a fiod failure ca2e [eg. COVID lods 2].
Chea X-my daggaes wisre lunel make Y
PoUU (1(x))
PV(Y
chanzes base the dacasd Y
DOVID 15).
Diggnnai: tsls
incidence libel clanges sulptpuiation
Grap-LLael
Poiss PP(U
Nea P:l Ro
[ea.. recial is diagzsii 14, (4.1)..
Data drive discovery tisks where is
Level
Tox Pr/Y Pr(P p. aa)) Piss is - P Aro--B((t-11)) 5.x.
esperimenta muscome conttmed by
hepothesis test with type Vil enoe 116, 39
Table Common noise models that we consider in this work. We represent each model as a probability distribution with parameters Puly. and show corresponding probabilistic graphical model (PGM). Given noisy dataset, noise model, and prior distribution Ty, we infer noise draws from a posterior distribution with parameters aulj.w-
3 REGRETFUL DECISIONS
Consider a practitioner who learns a model f X' + y from noisy dataset. In practice, they may learn a model that performs well on average. However, they cannot determine where makes mistakes. In such tasks, individuals are subject to lottery of mistakes. We characterize this effect in terms of regret.
Definition 2. Given a classification task where we learn a model f X - y from a noisy dataset, we define the regret for an instance (æs, Te) as:
Regret( (xx;) Br, U.) = pereed (f(x;). 01) # c (f(x))
Here:
(f(aa)) (UU)) == Iff();) # (U.)) indicates actual mistake with respect to the true
label. We write the true label as (².U.) ãi @ U, to show that it a random variable.
(f((((()))) 7.) indicates the model has made anticipated mistake i... that it appears to have made a mistake based on what we can tell during training.
In practice, medd (.) determined by how we account for noise, if at all. If we ignore label noise and fit model via standard ERM on the noisy dataset, then epred (f (mx), ūz) = T [ʃ (ax) # Dz). If we fit
model via noise-tolerant ERM [e.g., 40, 44], then epred (f(x,) 7.) = ln (xx,) Te) where for
an unbiased loss defined such that Eer (2);, ū:) = foi [x: ()))
3

---

Published as a conference paper at ICLR 2025
Regret captures the irreducible error incur due to randomness. In online learning, regret arises because we cannot foresee randomness in the future. In learning from noisy labels, regret arises because cannot infer randomness from the past. In this case, randomness undermines ity to determine which predictions are correct. In this regime, as individual predictions cannot be assumed to be accurate even the training data. As result, cannot rely on predictions to sup- port individual decisions, Moreover, we cannot rely on any downstream applications that depend on the correctness of individual predictions - e.g., model explanations [6, 48] or post-hoc analyses [25, 26, 34]. In Prop. 3, we explore the relationship between these effects and label noise.
abil-
Proposition 3. a classification task where we learn a classifier f from a noisy dataset D:
E ((X))t(f((X),XX) U)]
Pr(U XY.).
Prop. 3 provides an opportunity to highlight several implications of learning from label noise the instance-level. On the one hand, this result implies that regret is unavoidable when learning under label noise. In practice, we can only avoid it by "predicting less" (e.g.. via selective classification) or by "removing noise" (e.g., via relabeling). On the other hand, the result also implies that we can estimate the expected number of regretful predictions in terms of the posterior noise rate. In practice, however, we cannot tell how these mistakes are distributed over all instances.
One of the key issues in this regime is that the value of a prediction may be compromised, as each instance where Tu á 0 is subject to lottery of mistakes. Consider screening for rare disease using a diagnostic test. In such cases, we can view the presence of the disease as a clean label yi and the test outcome as a noisy label Dt. Given a disease that affects 10% of patients and class- level noise model that flips 10% of positive cases, an average draw of label noise would affect I 1% of predictions.
practice, such conditions would undermine the value of screening because any patient with a negative test may have the disease. We characterize these effects by measuring the proportion of instances in a dataset that are susceptible to regret i.e., that are subject to the lottery
of mistakes. Given noisy dataset D and posterior noise model Pr (U 1 x, x) the number of points susceptible to regret is:
Susceptibility(D)
I
(U (
x4 Y = 31 > >0
(1)
On the Regret of Hedging One of the benefits of studying regret in this regime is that we can characterize when learning is feasible at both the population and instance-levels. Many algorithms for learning from noisy labels are designed to hedge against label noise [46]. Given a noisy dataset and noise model, hedging minimizes the expected risk over all possible noise draws. In some cases, algorithms may implement this strategy explicitly via ERM with a modified loss [see e.g., 37, 40). In others, algorithms may hedge implicitly e.g., by assigning sample weights to training instances and setting their values to minimize expected risk over all possible draws [see e.g., 33, 44, 54].
In best-case scenario, where we correctly specify the noise model and fit model that minimizes the average number of mistakes over all noise draws, we would still incur regret. Formally, we would expect Exjx, [AError D, U)] = 0 where:
AErrri[J ((Tor((.D;U)=
gred (f(x))
ine (f(z)).
Predicted Training Emar
True Training Eroo
However, the resulting model f would still incur regret ExixY Regret(f. > In Prop. 4, we show that the classical hedging algorithm of Natarajan et al. [40] exhibits this behavior.
Proposition 4. Consider training model ʃ X - y on noisy dataset via ERM with modified loss function É: y x y + RR,, such that Ey [(([(()) 7)] E( (((()),)) y) for all (ar, W) In this case, the model minimizes risk for an implicit noise draw u mile = |uT ..... mile] where unile corresponds to most likely outcome under the posterior noise model Qulg..
Prop. 4 implies that hedging will incur regret unless the implicit noise draw mle matches the true noise in the dataset Tme In practice, this event is unlikely as lim Pr mle = u me) = (see Appendix A).
4

---

Published as a conference paper at ICLR 2025
4 ANTICIPATING MISTAKES WITH PLAUSIBLE MODELS
Our results in Section 2 show how model we learn under label noise will output regretful pre- dictions. this section, we develop methods to estimate the likelihood an individual instance yielding regretful prediction.
4.1 MOTIVATION
Our goal is evaluate the correctness of individual predictions for models learned from noisy data. In a standard classification task, we apply an algorithm for ERM to a clean dataset, recover the model E argmin JEF n Ex= (x:)) * vil, and evaluate the correctness of each prediction on the training data terms of mistakes. When we learn from a noisy dataset Î {(rr, Di) H=1+ the corresponding measure is no longer deterministic:
Mistake( (zi, Y., - ((x.) + Y2
In this case, the randomness stems from: (1) the true label Y,, which is a random variable that can only be inferred from the observed noisy label ÿi and the posterior noise model; (2) the model F F X - y, which is the output of a learning algorithm on the noisy dataset.
Our proposed measure, which we call ambiguity, quantifies the expected likelihood of a learning algorithm making mistake on the training data i.e., the expected value of (3).
Ambiguity (æ; T.) == Flō Mistake æ., Y, () = Eu-UD [T] + (Th @ U.)]
(4)
Ambiguity uses all the information have at hand: a noisy dataset and noise model. In Prop. 5, we show how ambiguity corresponds to regret as it correctly ranks the instances based on the likeli- hood of experiencing regret.
Proposition 5. Given classification task, denote the clean label risk of model F(x.) an instance as that is e == Pr ((..) * # When < 0.5 that is, model makes more correct than incorrect predictions a higher label noise rate for instance æ; corresponds to higher ambiguity((:;:22 (1).
Since Prop. 3 establishes that regret corresponds to the posterior noise rate, Prop. 5 suggests that ambiguity serves as viable measure regret, given its correspondence the posterior noise rate.
4.2 ESTIMATION
We can construct unbiased estimates of ambiguity using Algorithm Given a noisy dataset and noise model, this procedure generates plausible realizations of clean dataset, and then trains set of plausible models that can be used to estimate ambiguity. In what follows, we describe this procedure in greater detail.
Sampling Plausible Draws Given a noisy dataset D. class-level noise model pa, and prior distri- bution Tg. := Pr (Y y). we can sample noise draws from the posterior distribution:
(1 -1-38 (1))g) Pal1
Gulp
(5)
Pulū (1 nT)
Pulj) Kg
This generalizes to different types of noise models (see e.g., Table 1). We can use these samples from the posterior distribution to estimate ambiguity directly. In practice, however, it may lead to biased estimates by returning atypical draws - unlikely noise draws under given noise model (c.g., noise draw that flips 30% of labels under uniform noise model with a noise rate of 10%). In settings where we wish to estimate ambiguity using a limited number of draws, an atypical draw can bias our estimates and undermine their utility. Although we could moderate this bias by increasing the number of draws, this would require training a separate model for each draw. We address these issues by sampling from set of plausible draws.
5

---

Published as a conference paper at ICLR 2025
Algorithm Generate Plausible Draws, Datasets, and Models
Input noisy datasct (2t; Bc) noise model Pulyr number of models m 2 1, atypicality e [0, 1]]
Initialize F: f-
1: repeat
2: U, Bernouilli Bernouilli(uuij.a.) for i [n]
generate posterior inference
3: [U1, Un E Ue then
check if drow is plausible using Def. 6
4:
D: < Y1 ① us for E [xc]
D - ((T,, 81)
consrruct plausible clean dataset
f +- argmin FEF R(f:D)
train plausible model
7:
fu +- friium {f}
update plausible models
8:
end if
9:
until [F
Output Fe sample of models from the set of plausible models Fillos
Definition 6. Given noise draw {0, 1)" denote its true posterior noise rate 9ulğ Pr(U 1 Ý = 7)) and empirical noise rate as duly == --== [u, ÿ = y]. For any E [0, 1]. the of plausible draws contains all draws whose empirical noise rate is within of the posterior rate:
( = tu E {0, 1) s.t. Mdul - Eutol < Gulg for all u {0,1))
The set of plausible draws a strongly typical set [see 9]. In classification task where is large, we can expect most (but not all) draws to concentrate in U, [see Theorem 3.1.2 in 9]. We can limit atypical draws by setting the atypicality parameter e, which represents the relative deviation between the true noise rate qulğ and the noise rate of sampled draws. Given uniform noise model where duly = 0.1, we would set = 0.2 to only consider draws that flip between 8% 12% instances. Alternatively,
ensure that (ū) will include particular noise draw u²* J.llaus with high probability (see Prop. 9 in Appendix A). By default, we set e = 0.1 to consider draws within 10% of what would expect.
Estimating Ambiguity Given plausible noise draw u E (ū), we construct plausible clean dataset by pairing each æ; with a plausible value of true label 9 u + it: Definition 7. The set of e-plausible models contains all models trained using e-plausible datasets:
{
F.plass ==
argmin R(,,DD) D ((x,, 11 a EU. fEF
In an ideal case, where we recover plausible draw that matches the true draw u² tuee our procedure returns a plausible dataset E and model f that perfectly flags all regretful pre- dictions. Seeing how tre is unknown, we repeat this process times and use the m plau- sible models to get an unbiased estimate of ambiguity for each point our noisy dataset as: i(x ū) = 1 nh Exe(m]]
(}(())) 5t
In practice we can use ambiguity as confidence score to operationalize techniques to learn or predict reliably. We propose few examples and demonstrate how these perform in Section 6:
ambiguity to flag regretful instances in training dataset to drop or relabel. Given the correspondence between regret and noise (Prop. 3), this approach can be used to "de-noise" dataset to train models that generalize better on clean test data.
Data Cleaning: We
Selective Prediction: We can use ambiguity to abstain from potentially regretful predictions at test time via selective prediction [12]. This approach can be used in instances such clinical decision support, where we only show sufficiently reliable predictions and defer uncertain predictions to clinician.
The main limitation of this approach is that we assume access to a correctly specified noise model. This assumption is practical limitation and can be validated by comparing it against
Discussion

---

Published as a conference paper at ICLR 2025
distributions estimated from noisy dataset [see e.g., 31, 33, 44]. When working with simple noise models (e.g., uniform or class-level), we may be conservative and assume higher noise rate. Al- ternatively, can hedge against misspecification by setting capture larger set of plausible draws. The set of plausible models can also be used in ways to construct uncertainty measures, as demonstrated in Sections 5 and 6.
We also note that our estimates of ambiguity assume that the true noise draw, u = is typical. In practice, although uheee is unknown, most draws can be shown be typical this follows from standard application of Chernoff bound [9].
5 EXPERIMENTS
In this section, we present an empirical study on clinical prediction tasks. Our goals are to document the effects of label noise on individual-level predictions. Supporting material and code can be found in Appendix and GitHub.
Setup We work with 5 classification datasets from clinical applications where models support individual medical decisions (see Table 3). We treat the labels in each dataset as true labels. We create noisy datasets by corrupting the labels using noise draw sampled according three class- level noise models with noise rates 15% 20%, 40%] where label noise only affects positive instances (v: 1). We split each dataset into training sample (80%), which we use train a logistic regression model (LR) and neural network (DNN) using noisy labels, and a test sample (20%) ), which we use to measure out-of-sample performance using true labels. We train these models using the following methods:
1. Ignore, where we ignore label noise and fit a model predict noisy training labels; and Hedge where we hedge against label noise using the method Natarajan et [40].
This yields 12 models for each dataset (3 noise regimes x 2 model classes x 2 training procedures).
Metric
Definition
Description
TrueEmoon/.D D)
f cleae training labels.
AnticipatedErron D)
Error of f the noisy labels.
SuscepübilitytD)
| -1-111X-2..P >
Proportion of instances in D subject regret.
Mean regret across all instances D. We expect
Regret(S.D) D)
Σ (rtr))))) ((
Regret( /, D) ss >x 9-1y By under class-level label noise.
Proportion of predictions D that incorrectly perceived accurate.
Overreliancol,, D)
Σ (fo)))))))))))) (.
Table 2: Overview of summary statistics in Table 3. We report these metrics for models that we train from noisy labels using specific training procedure, model class, noise model, and dataset. We evaluate all models trained given dataset and noise model using fixed noise draw.
We characterize the accuracy and reliability of predictions from each model using the measures in Table 2. We report our results for LA models in Table 3 and results for DNN models in Appendix B. In what follows, we discuss all results.
On Label Noise, Regret, and Hedging Our results in Table 3 highlight several implications of learning under label noise. We confirm that our result in Prop. 3 holds empirically i.e., the ex- pected prevalence of regretful predictions corresponds to the effective noise rate in each dataset. We observe similar effects across all datasets, model classes, and noise regimes, underscoring the need to quantify the effect of label noise on individual predictions.
Existing approaches to handle label noise (i.e., hedging) can learn models that are robust to noise at population-level but still experience regret. As shown in Table 3, we observe that Hedge can moderate the impact of label noise at a population-Ieve by reducing the true (clean label) error compared to Ignore. Even with more noise robustness, regret is unchanged, and remains high across

---

Published as a conference paper at ICLR 2025
all experimental conditions. On the mortality dataset, for example, Hedge reduces the error rate by over 13% compared to Ignore for a LR model under 40% label noise. However, regret is unchanged and continues to affect 19.5% of instances. It is interesting to note that Hedge can moderate the effects of overrelianceby redistributing unforeseen mistakes from instances that lead to overreliance instances where epood (f (a;)) ūt) = and e (f(:a)) y1) 0 i.e., where practitioner may fail to reap the benefits of a correct prediction because it appears to be incorrect.
On the Lottery of Mistakes Our results highlight how a small amount of label noise can undermine common use cases for predic- tion by subjecting a far greater number of in- stances to a lottery of mistakes. In Table 3, for example, we consider a noise model where only positive instances (y- = 1) are subject to label noise. Thus, every instance with a nega- tive noisy label (ū- 0) is subject to a lottery of mistakes. We report the proportion of points that take part in the lottery using the suscepti- bility metric in Eq. (1). In this case, we can see that in task where the label noise is low as 5%, over half of instances are subject to lot- tery across all five datasets. For example, in the lungcancer dataset, even a small misdiagno- sis (i.e.. label noise) rate, which inevitable, can compromise the reliability of half of all di-
Paly
Pirx 1
Pils 1
Eusr
23.5%
ahoor_esez
Reght
(45)
Seocpthilly
135
09.39
Asticipatd Euse
22.18
Malsso (E3)
101
Secopihiity
Oxemdionc
Raget
57.98
0.88
2.5%
57.4%
0.05
2.5%
00.75
185
01.25
10.25
09.85
18.8%
19.8%
1933
5.59
Jusgusnces
60, 916
40
Articip tirer
Overilince
Reget
hiror
31.99
13.25
2.5%
1.5
50.899
2.59
1.355
10.0%
17.75
5.15
10.05
31.6
3.0%
21.9%
19.79
5105
11.3%
19.75
Secepiiilty
57.79
57.7%
60.25
00.25
09.95
69 995
moctalloy
Le Gaa at. (24)
ni.. 201 2010
af 81
Articipaee liree
Ovenelaace
"Ine tirne
Regit
20.75
19.55
0.6%
3.25
21.4%
19.15
0.5%
3.25%
25.75
21.25
4.8%
4.9%
25.15
19.15
9.8
2.65
51.2%
27.7%
19.5%
13.3%
31.9
19.05
3.85
19.95
Sescepaniity
52.29
True Euso
33.19
33.79
36.79
33.76
44.2%
33.95
Artcipannd time
31.85
54.15
56.15
29.7%
114
Ocomdlance
2.8%
1.89
2,85
1.7%
9.68
10.75
6.0%
19.8%
19885
agnoses.
Succptizilt
Table 3: Accuracy and reliability of predictions for LR models trained on noisy datasets where we flip 5%, 20% and 40% of positive instances. We defer results for DNN models to Appendix B for clarity.
On the Consequences of Blindness Our re- sults highlight the importance of considering the effect of label noise in instance-level pre- dictions. Many real-world practitioners assume that their training data is clean and ignore label noise, but most real-world datasets are not perfectly labeled this inevitably leads to regretful pre- dictions on individuals.
To demonstrate how regretful predictions can negatively impact individuals, we consider a partic- ular flavor of regret- overreliance the fraction of instances where practitioner would incor- rectly assume that a model assigned a correct prediction - i.e. where eped (f(x)) ūt) = 0 and (f yı) 1. From Table 3, we consider overreliance on the lungcancer dataset, under 40% noise. We observe that up to 23.4% of instances are assigned this type of prediction. In prac- tice, such instances correspond to patients with cancer but who are classified as cancer-free based on the prediction of seemingly accurate model. These are patients where the model is making a mistake, however, the practitioner cannot tell as it would not appear to be a mistake based on the noisy label. This highlights the importance of looking at the distribution of regretful instances across predictions. By analyzing the distribution of regretful predictions, we can adjust our reliance on model predictions ensuring that practitioners do not blindly trust or explain away incorrect model decisions.
6 DEMONSTRATIONS
In this section, we show how the machinery developed in Section 4 can be used to promote safety at critical parts of the machine learning lifecycle in real-world applications where noisy labels are inevitable.
Data Cleaning Our approach in Section 4 can clean noisy datasets by using ambiguity to drop noisy instances from a training dataset. Given the "denoised" dataset, we can then train models that perform better in deployment. In Fig. 2. we demonstrate the effectiveness of this approach on the shock borocck_miiiiim c dataset. Here, we drop training examples using a confidence- based threshold rule of the form TI [conf(x:) (x:) T], where is threshold set to control the number of instances to drop. We
8

---

Published as a conference paper at ICLR 2025
compare the performance of this strategy using confidence scores that can compute training data: (1) conf(x;) = 1 x; Di), which is a measure based on the estimated ambiguity that we recover using Algorithm 1; and (2) conf((z.)
p(y; æ;), which is the predicted probability of final model. As shown, removing instances with high ambiguity from the training dataset prior to training final model the cleaned data leads improved test using ambiguity to drop uncertain instances reduces test error by 14.9% when dropping only 20% of noisy training data compared baseline approach.
clean labels. Specifically,
Selective Classification with Cheap Labels We use results to highlight how the machinery in Sec- tion 4 can promote safer predictions. Consider the mimic dataset in Fig. 3 here, use the same confidence-basee I [conf(a::) S 7] where conf(æ;;) is confidence and r is a threshold value. We consider confidence scores based standard predicted probabilities and ambigu- ity, where ambiguity can be measured using cheaply ac- quired test instances (e.g., noisy test data). We show how the selective test error clean labels and selective re- gret change we vary the confidence threshold value (0, 1). Specifically, in regime where 20% of the labels are noisy, abstaining on 40% of instances using ambiguity reduces selective error by -6.6% and selective regret by -5.9% compared to the standard approach on eshock_minn
threshold rule the form
Confidence
30%
Eror
25%
-14.9%
Anbiguity
10%
Clean Rate
Figure 2: Clean test error for a LR model on the shock_misse mimic dataset with 40% class- level label noise when dropping training instances using different confidence based threshold rules. We show the clean test error vs percent of instances dropped from train- ing using confidence measures hased on pre- dicted probabilities conf(æ:) p(ū w;) ambiguity conf(æ.) = pl.., V.)
Selective Classification for Scientific Discovery We demonstrate how our approach can support a modern sci- entific discovery task biotechnology. In such tasks, researchers perform in-vitro experiments to identify in- stances with desired properties [e.g. identifying new an- tibiotics 50J. Given dataset of successful and unsuc- cessful experiments and their characteristics, we can train model to predict which future experiments are likely to succeed, thereby accelerating discovery by prioritizing high-yield experiments.
We use the enhancer dataset from Gschwind et al. [16] to predict the outcome of experiments to discover enhancers i.e., segments of DNA that regulate gene expression. The dataset contains 992 noisy instances (I;; fk), each with 13 features (e.g., gene location, cell type, etc.).
12%
25%
Selective Regret
8%
Confidence
Selective Error
Confidenca
15%
-6.6%
4%
2%
-5.9%
5%
Anbiguuity
Ambiguity
0%
0%
20%
40%
60%
80%
100%
0%
20%
40%5
50%
80%
100%
Abstention Rate
Abstention
Rate
Figure 3: Selective classification frontiers for a LR model on the shock mimic dataset under 20% class-level noise when abstaining from uncertain predictions at test time using confidence-basee threshold rule. We plot the selective regret (left) and selective error (right) vary the percent of abstained predictions for confidence measures based on predicted probabilities conf(x; p(k æ5;) and ambiguity conf(æ:;) f(x:)).

---

Published as a conference paper at ICLR 2025
In this setting, noisy label y/ = 1 indicates statistically significant experiment (i.e., reject null hypothesis: Hg = "no effect"). Here, label noise arises from the Type error for each experiment:
Pr (U: 1 y: = 0) Pr (reject Ho Ho holds) = p-value for experiment
We fit LR model f and its predictions identify experiments that likely succeed in the test sample. To avoid low-confidence predictions, we use a thresholding rule of the form I [c,nf(xx) < r] where T is threshold value that we can control the number experiments to perform. Confidence for an instance is measured with a score conf(æ,) == 1- Disagreement( (z.) where Disagreement(z:) measures the disagreement between the predictions of f and the plau- sible models from Algorithm 1:
Disagreement(a:)) == m7
1
Σ (a:) S(())) {=(mm]
(6)
Fig. 4 shows that disagreement can reliably predict which experiments will be successful. We use two strategies for abstention: (1) a standard approach thresholding according to p(ù; æx), or (2) using disagreement rates (6). Performance is measured using test hit rate (i.c., the number of successful experiments divided by the number total experiments that we run). Our approach improves the hit rate (+10.7%) compared to standard confidence-based abstention, with modest 20% abstention rate (Fig. 4). This demonstrates that we can optimize laboratory resource allocation and increase the discovery of enhancers by forgoing 20% of experiments.
Disggrrement
70%
30%
60%
25%
Hit Rate
+1.79%
Enro
50%
20%
-8,5%
Confidence
Disag
15%
30%
10%
20%
20%
30%
Abstention Rate
Abstentian Rate
Figure 4: Selective classification frontiers for an LR model on the enhancer dataset when abstaining from uncertain predictions using confidence-basse based threshold rule. We plot the selective hit rate (left) and selective emror (right) increase the proportion of abstained predictions for confidence measures based predicted probabilities conf(2:, p( æ.) and disagreement conf(æ:) (x.)
Disagreement((:)))
CONCLUDING REMARKS
Learning under label noise is a major challenge in practice. While models may perform well on average, model that is 99% accurate can inadvertently misclassify anyone, as label noise can subject each individual prediction to a lottery of mistakes. In this work, we studied these effects through the lens of regret and highlighted the inherent limits of learning in this regime.
Our results show that, even as regret is inevitable when learning from label noise, we can opera- tionalize simple techniques predict safely by quantifying the uncertainty in individual predictions ee... by estimating ambiguity individual predictions and using this to flag predictions where we should abstain or examples that should re-label.
Our use of regret extends beyond label noise - into any setting where models are trained on datasets with single draw of noise [c.g., for probabilistic classification 13]. In such settings, regret can explicitly reveal the impact predictions on individuals, and estimating it can act as a safeguard or signal to collect more data avoid prediction altogether.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by funding from the National Science Foundation IIS 2040880, IIS 2313105, IIS-20079,,, IIS-2143895, and the NIH Bridge2AI Center Grant U54HG0125 Sujay Nagaraj was supported by CIHR Vanier Scholarship. Resources used in preparing this research were provided. in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
REFERENCES
[1] Dana Angluin and Philip Laird. Learning from noisy examples. Machine learning, 2:343-370, 1988.
[2] Mayara Lisboa Bastos, Gamuchirai Tavaziva, Syed Kunal Abidi, Jonathon Campbell, Louis-Patrict Haraoui, James C Johnston, Zhiyi Lan, Stephanie Law, Emily MacLean, Anete Trajman, et al. Diagnostic accuracy of serological tests for covid-19 19: systematic review and meta antaaa-sayyss... bmj. 370, 2020.
[3] Atharva M Bhagwat, Kadija S Ferryman, and Jason B Gibbons. Mitigating algorithmic bias in opioid risk-score modeling to ensure equitable access pain relief. Nature medicine, 29(4):769-770, 2023.
[4] Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concems, and solutions. In 2022 ACM Conference on Fairness, Accountability, and Transparency. FAceT '22, pp. 850- 863, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1 145/3331 146.3533149,
URL https::////......... / /doi /tt.//o..oo/////////////o 11.50111133333 146 3533149.
[5] Marc-Etienne Brunet, Ashton Anderson, and Richard Zemel. Implications of model indeterminacy for explanations of automated decisions. Advances in Neural Information Processing Systems, 35:7810- 7823, 2022.
[6] Seung Hyun Cheon, Anneke Wernerfelt, Sorelle Friedler, and Berk Ustun. Feature responsiveness scores: Model-agnostic explanations for recourse. In The Thirteenth International Conference on Learning Rep- resentations, 2025.
[7] Chun- Wei Chiang and Ming Yin. You'd 'd better stop! understanding human reliance on machine learning models under covariate shift. In Proceedings of the 13th ACM Web Science Conference 2021, pp. 120- 129, 2021.
[8] Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. Characterizing fairness over the set of good models under selective labels. CoRR, abs/2101.00352, 2021. URL https:///xxxxxxorrg abs / /2101.110 00352.
[9] Thomas M Cover. Elements of Information Theory. John Wiley & Sons, 1999.
[10] Vojtech Franc, Daniel Prusa, and Vaclay Voracek. Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24(11):1-44 2023.
[11] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE Transactions on neural networks and learning systems, 25(5):845-869, 2013.
[12] Yonatan Geifman and Ran El-va. Selective classification for deep neural networks, Advances in neural information processing systems, 30, 2017.
[13] Erin George, Deanna Needell, and Berk Ustun. Observational multiplicity and regret. 2025 Joint Mathemutics Meetings. American Mathematical Society.
[14] Kan Z Gianattasio, Christina Prather, M Maria Glymour, Adam Ciarleglio, and Melinda C Power. Racial disparities and temporal trends in dementia misdiagnosis risk in the united states. Alzheimer's 's & Demen- tia: Translational Research & Clinical Interventions, 5:891 898, 2019.
[15] Athanasios Giannakis, Dorottya Móré. Stella Erdmann, Laurent Kintzelé, Ralph Michael Fischer, Monika Nadja Vogel, David Lukas Mangold, Oyunbileg von Stackelberg, Paul Schnitzler, Stefan Zim- mermann, et al. Covid-19 pncumonia and its lookalikes: How radiologists perform diffcrentiating atypical pncumonias. European Journal of Radiology, 144:1 10002, 2021.
[16] Andreas R Gschwind, Kristy S Mualim, Alireza harbalayghared Maya U Sheth, Kushal K Dey, Evelyn Jagoda, Ramil N Nurtdinov, Wang Xi, Anthony S Tan, Hank Jones, al. An encyclopedia of enhancer- gene regulatory interactions in the human genome. bioRxiv, 2023.
11

---

Published as a conference paper at ICLR 2025
[17] Faisal Hamman, Erfaun Noorani, Saumitra Mishra, Daniele Magazzeni, and Sanghamitra Dutta. Robust algorithmic recourse under model multiplicity with probabilistic guarantees. IEEE Journal on Selected Areas in Information Theory, 2024.
[18] Kilian Hendrickx. Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with reject option: A survey. Machine Learning, pp. 1-38, 2024.
[19] SM Hollenberg. Cardiogenic shock. In Intensive Care Medicine: Annual Update 2003, pp. 447-458. Springer, 2003.
[20] Hsiang Hsu and Flavio du Pin Calmon. Rashomon capacity: A metric for predictive multiplicity in probabilistic classification, 2022. URL https::p///// //assiv////vvv// org/ abs/220s00 2ttp/226..122900 C1295.
[21] Hailey James, Chirag Nagpal, Katherine A Heller, and Berk Ustun. Participatory personalization in classification. In shirty-seventh Conference on Neural Information Processing Systems, 2023.
[22] Alistair EW Johnson, Tom Pollard. Lu Shen, Lehman Li- Lei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):199, 2016.
[23] Jamil N Kanji, Nathan Zelyas, Clayton MacDonald, Kanti Pabbaraju, Muhammad Nacem Khan, Abhaya Prasad, Jia Hu, Mathew Diggle, Byron M Berenger, and Graham Tipples. False negative rate of covid-19 per testing: discordant testing analysis. Virology journal, 18:1-6, 2021.
[24] William A Knaus, Frank E Harrell, Joanne Lynn, Lee Goldman, Russell Phillips, Alfred F Connors, Neal V Dawson, William Fulkerson, Robert M Califf, Norman Desbiens, et al. The support prognostic model: Objective estimates of survival for seriously ill hospitalized adults. Annals of internal medicine, 122(3):191 203, 1995.
[25] Pang Wei Koh and Percy Liang. Understanding black-hox predictions via influence functions. In Inter- national conference on machine learning, pp. 1885-1894, PMLR, 2017.
[26] Avni Kothari, Bogdan Kulynych, Tsui-Wei Weng, and Berk Ustun. Prediction without preclusion: Re- course verification with reachable sets. In The Twelfth International Conference on Learning Represen- tations, 2024.
[27] Bogdan Kulynych. Hsiang Hsu, Carmela Troncoso, and Flavio P Calmon. Arbitrary decisions are a hidden cost of diffcrentially private training. In Proceedings of the 2023 ACM Conference on Fairness, Accountabiliry, and Transparency, pp. 1609-1623, 2023.
[28] Jean- Jean-Roger Le Gall, Stanley Lemeshow, and Fabienne Saulnier. A new simplified acute physiology score (saps ii) based on european/north american multicenter study. Jama, (70(24):2957-2965 1993.
[29] John D Lee and Katrina See. Trust in automation: Designing for appropriate reliance. Human factors, 46(1):50-80) 2004.
[30] Dana Li, Lea Marie Pehrson, Lea Tettrup, Marco Fraccaro, Rasmus Bonnevie, Jakob Thrane, Peter Jagd Sørensen, Alexander Rykkje, Tobias Thostrup Andersen, Henrik Steglich- observer agreement when using a diagnostic labeling scheme for annotating findings on chest x-rays- an early step in the development of a deep learning-based decision support system. Diagmostics, 12(12): 3112, 2022.
Steglich-Arnholm. Arnholm, Inter-and intra-
[31] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International conference muchine learning, pp. 6403-6413, PMLR, 2021.
[32] Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In International Conference Machine Learning, pp. 6725-6735. PMLR, 2021.
[33] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. ICMI, 2020.
[34] Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint 1rXiv:1802.03888 2018.
[35] Charles Marx, Flavio P. Calmon, and Berk Ustun. Predictive Multiplicity in Classification, 2019.
[36] Charles Marx, Youngsuk Park, Hilaf Hasson, Yuyang Wang, Stefano Ermon, and Luke Huan. But are you sure? an uncertainty-aware perspective on explainable ai. In International Conference on Artificial Intelligence and Statisties, pp. 7375-7391. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
[37] Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class-probabillty estimation. In International Conference on Machine Learning, pp. 125-134,22 2015.
[38] Anna Meyer, Aws Albarghouthi, and Loris D' Antoni. The dataset multiplicity problem: How unreliable data impacts predictions. Proceedings of the 2023 ACM Conference on Faimess, Accountability, and Transparency, pp. 193-204, 2023.
[39] Sujay Nagaraj, Sarah Goodday, Thomas Hartvigsen, Adrien Boch, Kopal Garg, Sindhu Gowda, Luca Foschini, Marzyeh Ghassemi, Stephen Friend, and Anna Goldenberg. Dissecting the heterogeneity of "in the wild" stress from multimodal sensor data. NPJ Digital Medicine, 6( 6(1):237, 2023.
[40] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
[41] Surveillance Research Program NCI, DCCPS. Surveillance, epidemiology, and end results (scer) program research data (1975-2016), 2019. URL www. seer, cancer gov.
[42] Sejoon Oh, Berk Ustun, Julian McAuley, and Srijan Kumar. Rank list sensitivity of recommender systems to interaction perturbations. In Proceedings of the 31st ACM International Conference on Information & Knowledge Monagement, pp. 1584-1594, 2022.
[43] Diane Oyen, Michal Kucer, Nicolas Hengartner, and Har Simrat Singh. Robustness to label noise depends the shape of the noise distribution. Advances in Neural Information Processing Systemx, 35:35645- 35656, 2022.
[44] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944-1952, 2017.
[45] Tom Pollard, Alistair FWW Johnson, Jesse D) Raffa, Ieo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, freely available multi-center database for critical care research. Scientific data, 5(1):1-13, 2018
[46] Ramamoorthi Ravi and Amitabh Sinha. Hedging uncertainty: Approximation algorithms for stochastic optimization problems. In International Conference on Integer Programming and Combinatorial Opti- mization, pp. 101-115. Springer, 2004.
[47] Scott Reed, Honglak Lec, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv: 1412.6566, 2014.
[48] Marco Tulio Riberro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretabilitty of machine lcarning, arXiv preprint arXiv:1666.055 1606.0386. 2016
[49] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on neural networks and learning systems, 2022.
[50] Jonathan M Stokes. Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell, 180(4)6888 702, 2020.
[51] Vinith Menon Suriyakumar, Marzych Ghassemi, and Berk Ustun, When personalization harms perfor- mance: reconsidering the use of group attributes in prediction. In International Conference on Machine Learning, pp. 30209-333333 33228. PMLR, 2023.
[52] Berk Ustun, Lenard A Adler, Cynthia Rudin, Stephen V Faraone, Thomas Spencer, Patricia Berglund, Michael J Gruber, and Ronald C. Kessler. The world health organization adult attention- deficishhyperactiiity disorder self-report screening scale for dsm-5. Jama psychiatry. 74(5):520-52,, 2017.
[53] Jamelle Watson Daniels, Flavio du Pin Calmon, Alexander D' Amour, Carol Long, David C Parkes, and Berk Ustun. Predictive churn with the set of good models. arXiv preprint arXiv.2402.07745, 2024.
[54] Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In International Conference on Machine Learning 2022.
[55] Yishu Wei, Yu Deng, Cong Sun, Mingquan Lin, Hongmei Jiang, and Yifan Peng. Deep learning with noisy labels in medical prediction problems: a scoping review. arXiv preprint (03::2003111111 2024.
13

---

Published as a conference paper at ICLR 2025
[56] Eric Yamga, Sreekar Mantena, Darin Rosen, Emily M. Bucholz, Robert W. Yeh, Leo A. Celi, Berk Ustun, and Neel M. Butala. Optimized risk score to predict mortality in patients with cardiogenic shock the cardiac intensive care unit. Journal of the American Heart Association, 12(13), July 2023, ISSN 2047-9980. doi: 10.1/61/jaha 122.029232. URL http: / wttp:/oo..oggg.....oo 111//11.111111 61 JAHA 122. 029232.
14

---

Published as a conference paper at ICLR 2025
A OMITTED PROOFS
A.1 RESULTS FROM SECTION 3
Proof of Prop. 3. Consider any classification task with label noise. Given a point with (X, P), let denote the posterior noise rate and for (f()), V) T [(x) *
Px,y- Pr U = 1 v)
its zero-one loss.
By definition, hedging algorithm [see e.g., 40] is designed learn classifier f such that: E (f(x) = "ue ().
We observe that f will achieve zero error in expectation result of the unbiasedness property of hedging algorithms.
Ex pred (f(X), - e tue (f(X), Ex,y Eujx,Y [ (f(X), - m (f(X),
The last line follows from the fact that Y is deterministic function of U given Y. We now show that will still incur regret in this regime. We begin by expressing the expected regret for any point (X, Y) and any noise draw U as:
Ex v [eegret(X, 8,
xxy (1 - 2 (e (J()) + (oi(1(X), + 2(gu - 1) ([()) (x V) + 4u
E Ex,9, [Regret[X[ X, XXY, = /1
Ex,s T (1
= Ex |1,,,,00,0011110000==-------,,,0
(1 J (00-
= Ex 17 U) -
(1xx) x] (1-
1(x)
[[(]] # 1 8J0)
(1 ((X)) ((x) + 8 (1 (1- U) + [(x) # 1 8 T|1]] U1]
Letting 4n = Pr X1, and En (I()), (Xx # 8] we have:
= Ex,r - -0-
+
Ex.y. [Regret(X, X, X,Y,
= Ex.y [11 - 2qw) (6 (f(X), tor ())
2(g 1) good (f(X), 8) for (f(X), Pu].
When there is no label noise, we have that qu = 0 and med (f(x), 8) = Los (oi for all X, Y. Because they are binary terms, in this regime, we have:
E U Regret(X,Y,,U) Ex, / [0] =
When there is label noise, we have that qu > 0 for some X, Y. In this regime, we have:
E E Regrett(X, EogrettX,Y,00 Y.U) Ex,9 [q.] > 0.
15

---

Published as a conference paper at ICLR 2025
The proof for Prop. 4 uses the following lemma. Lemma 8. Minimizing the expected risk under the clean label distribution is equivalent to minimiz- ing a noise-corrected (hedged) risk under the noisy label distribution.
[ qu [((X) + r] + qo
((x))1- +1 -+]
Ex,y (X)) #
Y]] x
(7)
Here:
Pul1
qu Puly. (1 (1oPull)=)
Ti.= = Pr = 5/ X = x) is the clean class prior an observed noisy label,
Pa Pr = 1 y. x) the class-level noise probability.
The result is analogous to Lemma in Natarajan et al. [40]. In what follows, we include an additional proof for the sake of completeness.
Proof.
ExpectedRisk(J) = Ex [[(()*
EX [[[(()) + +(110)+00--]]]
Euix, [ [(x) [1=)++[-=0)
Ex..BUxXx T [=-0)=+||1-))+||/)))+-------1 r|0]
1xx,
[Bexx ( f(x) * 1 u|
(1-))+
Ex,y
Pr -U= [U= [xx) +Pr=+
x =
U=
[(x) x1 8]]
Ex,y
Pr Pr(Y=P11
[(x) # 8/ Pr Y ,8 x) [(x) + - $]]
= E x (1- qull [(x) *
8 + qul [(x) #1
We can recover the statement of Lemma 8 by applying Bayes theorem to write in terms of the clean class priors and class-level noise probabilities.
Proof of Prop. 4. We define u mie as the noise draw for instance (X, Y), such that using u mle to minimize the expected risk implicitly coincides with the true minimizer of the expected risk (defined in Lemma 8). That is:
argmin E x [t [F(() f(1 2 mlo + um (1-)
fer
-argmin Ex qw) qu)d F(x) * 8 + qull [((x) r]]
JEF
We can express the minimizer of the LHS as:
f' E argmin x [I [((X * 8 mle + mle r
(8)
feF
argmin E x,y (1 mlo)
m )I [(x) x r/ + u mlo [f(x) = I1]
(9)
JEF
We can denote the minimizer of the RHS:
f E argmin Ex,ý [1- gm)il [(x) 8 qud [(x) -(x) $]]
(10)
JEF
Observe that:
Guly, < 0.5 = f(x) =
2u1yx > 0.5 f(x) -
Thus, have that mle TI 14m > 0.5) + f = f', as desired.
16

---

Published as a conference paper at ICLR 2025
A.2
RESULTS FROM SECTION 4
Proof of Prop. 5. Denote the noise rate of F(x,,) e, that is Pr (F(z.) # v1)
PID
I ((
Yi]
FID
F(x;)) Yi F(x;) V/
(11-
I [(x,) # Y F(x;)) # u1]
+ By,
I Vr + v1 uilp.e
FID
Y vel
(1-c) (1 Fx,
(1 2e) BID
II
Yr # v1
te
When e < 0.5, we can claim that the higher the FID [1 # vi
the higher the AID I ((x.) * v.] the ambiguity measure. If we assume that Ey AID
Y + y/ is
monotonic in the noise rates in u, which is intuitively true, then establish that the higher noise, the higher the ambiguity measure.
the
A.3 ON CHOOSING AN ATYPICALITY PARAMETER
In Prop. 9, we present an additional bound that can be used to set atypicality parameter to guarantee that the set of plausible draws F includes reference noise draw with high probability. Proposition 9. Given set of Mp instances (æ,, 5) subject to noise rate Pu. we determine the mini- mum to ensure that reference noise draw u belongs the set plausible draws F.llas with high probability. That is, with probability at least 5, u E (ū) if obeys:
1
(2) 2mp
+ Ip.. yuls
Gulö
Here np represents the number of instances whose labels are corrupted by the same noise model. For example, under class-level noise, this bound would need be evaluated separately using the number of instances for each class.
For example, given a dataset with n = 10, 000 instances under 20% uniform label noise, a practi- tioner must set 2 6% to ensure that u E plaaus with probability at least 90%.
Proof of Prop. 9. Our goal is to show that Pr (u" E()(O) 2 6. for any given 0 < 8 < 1.
The uncertainty set (ğ) defined on Pulj strongly typical set (see [9]) where the true mean Pulv
and the empirical mean is pu = 1 - [[u = 1]. Thus,
u E u. - 1 Pulal S Pulg
(11)
We will derive conditions to satisfy the incquality Eq. (11) Observe that can write
P
Pujøl ((fu Pu)) + (pu. Pujsil
1 p:u pul 1pa Putjl
(by the triangle inequality)
We require liu. -Palgl S Pulg c. Therefore we need liv pul lp. - -Pulgl $ Pulg e which implies
that 1p.. Pul Pulg Pu Putøl
17

---

Published as a conference paper at ICLR 2025
We observe that u" is sequence of bounded, independently sampled random variables. Thus, can apply Hoeffding's ' inequality to see that:
Pr (léu Pul 2 a)) S exp(-2nac
Here, Pulg Ipu. - -Pujal Rearranging, have that:
Pr Pr(u" E4) = Pr (lλ.. p.. - a) 2 1 - 2
= - 2 exxp( (m y < lp= Pule 12)
We invert the bound to obtain the following statement: with probability at least 5, u² (1) the number of samples n obeys:
In (g)
2 [Pul IPu Putg))
To conclude the proof, we rearrange for e, that is, given dataset of instances, u" E (.( if obeys:
In (3)
E 2
+ 1Pu Pulpl
Pulø
2n
18

---

Published as a conference paper at ICLR 2025
B SUPPORTING MATERIAL FOR SECTION 5 AND SECTION 6
Here we include further details about the datasets used in our experimental results.
B.1 DATASETS
lungeancer We used a cohort of 120,641 lung cancer patients diagnosed between 2004-2016 who were monitored in the National Cancer Institute SEER study [41] and processed the dataset to match the processing in James et al. [21]. The outcome variable is death within five years from any cause, with 16.9% dying within this period. The cohort includes patients across the USA (California, Georgia, Kentucky, New Jersey, and Louisiana), excluding those lost to follow-up. Features include measures of tumor morphology and histology (e.g., size, metastasis, stage, node count and location), as well as clinical interventions at the time of diagnoses (e.g., surgery, chemotherapy, radiology).
shock_eies _icc & shock_mimic mimic Cardiogenic shock is acute cardiac condition where the heart fails to sufficiently pump enough blood [19] leading to seder-perfusion of vital organs. These datasets are designed build algorithms to predict cardiogenic shock ICU patients as described in Yamga et al. [56]. Both datasets contain identical features, group attributes, and outcome variables they capture different patient populations. The shock_eion dataset includes records from dataset includes records from the MIMIC-III database [22]. The target variable is whether a patient with cardiogenic shock will die in the ICU. Features include vital signs and routine lab tests (c.g., systolic BP, heart rate, hemoglobin count) collected within 24 hours before the onset of cardiogenic shock.
EICU Collaborative Research Database V2.0 [45], while the shock_mime
mortality The Simplified Acute Physiology Score (SAPS II) score is risk-score designed to predict the risk of death in ICU patients collected in [28] and used in [51]. The data contains records of 7.797 patients from 137 medical centers in 12 countries. The outcome variable indicates whether patient dies in the ICU, with 12.8% patient of patients dying. Similar to the other datasets, mortality contains features reflecting comorbidities, vital signs, and lab measurements.
support This dataset comprises 9,105 ICU patients from five U.S. medical centers, collected dur- ing 1989-1991 and 1992-1994 [24]. Each record pertains to patients across nine disease categories: acute respiratory failure, chronic obstructive pulmonary disease, congestive heart failure, liver dis- ease, coma, colon cancer, lung cancer, multiple organ system failure with malignancy, and multiple organ system failure with sepsis. The aim is determine the individual-level 2- and 6-month sur- vival rates based on physiological, demographic, and diagnostic data.
B.2 ADDITIONAL RESULTS FOR DNN MODELS
In Section 5, we include results for models. In this section, we include additional results for DNN models trained on the same unique noise draw as in Section 5,
19

---

Published as a conference paper at ICLR 2025
Puly=1
5%
Puly=1
20%
Puly=1
40%
Dataset
Metrics
Ignore
Hedge
Ignore
Hedge
lgnore
Hedge
True Error
13.3%
12.8%
18.6%
19.2%
37.5%
26.2%
shock_eion
Anticipated Error
14.4%
14.0%
20.3%
22.0%
25.1%
26.7%
n n= 3, 456
Regret 3.0%
3.0%
10.1%
10.1%
19.7%
19.7%
d = 104
Overreliance
1.1%
1.0%
5.3%
4.7%
21.4%
13.1%
Pollard et al. [45]
Susceptibility
52.6%
52.6%
59.7%
59.7%
69.3%
69.3%
True Error
15.6%
15.9%
18.8%
16.8%
32.7%
22.1%
shock mimic
Anticipated Error
17.4%
17.5%
23.9%
23.2%
26.6%
25.9%
15, 254
Regret 2.5%
2.5%
10.2%
10.2%
19.8%
19.8%
d= 104
Overreliance
0.4%
0.5%
3.4%
2.5%
17.7%
10.8%
Johnson [22]
Susceptibility
52.5%
52.5%
60.2%
60.2%
69.8%
69.8%
True Error
29.8%
29.7%
31.5%
30.0%
37.7%
29.5%
lungeancer n 62, 916
Anticipated Error
30.4%
30.4%
31.8%
33.4%
29.7%
36.7%
Regret 2.5%
2.5%
10.0%
10.0%
19.7%
19.7%
= 40
Overreliance 1.4%
1.3%
7.1%
5.0%
19.8%
9.9%
NCI [41]
Susceptibility
52.7%
52.7%
60.2%
60.2%
69.9%
69.9%
True Error
17.7%
17.9%
19.2%
18.3%
24.0%
18.9%
mortality
Anticipated Error
19.1%
19.4%
23.4%
24.0%
26.2%
29.5%
n 20, 334
Regret 2.2%
2.2%
9.8%
9.8%
19.5%
19.5%
d = 84
Overreliance
0.6%
0.5%
3.7%
2.7%
11.7%
6.3%
Le Gall al. [28]
Susceptibility
52.2%
52.2%
59.8%
59.8%
69.5%
69.5%
True Error
28.4%
28.6%
31.0%
30.3%
39.4%
35.7%
support 9,,6699 696
Anticipated Error
28.2%
28.2%
28.6%
28.7%
25.2%
27.8%
Regret 2.6%
2.6%
10.0%
10.0%
19.6%
19.6%
d 114
Overreliance 2.0%
2.1%
8.7%
8.1%
22.6%
19.1%
Knaus et al. [24]
Susceptibility 52.6%
52.6%
60.0%
60.0%
69.6%
69.6%
Table 4: Overview of performance and regret for DNN model trained on all datasets and training procedures.
20


Reviews:
[{"review_id": "MoIXoiiEVh", "reviewer": "Reviewer_tJN2", "summary": "The authors introduce the notion of regret when learning from a dataset that is subject to label noise. The authors point out that standard learning approaches typically target a notion of “average” loss or “average” risk over the population and cannot provide instance level guarantees. One way to identify that the model may make a mistake is to have access to clean labels, but this is often infeasible in practice. As a result, the authors propose to simulate “clean” datasets by assuming a noise model, simulating noise from that noise model, and then backing out a clean dataset from the noisy dataset and the sampled noise. Then the authors define a notion of ambiguity based on models trained on various plausible clean datasets.", "questions": "Why do we write $y_{i}(U_{i})$ in the definition of $U_{i}$, from what I recall, $U_{i}$ is generated given $y_{i}$, so it is a bit confusing to think of $y_{i}$ as a function of $U_{i}$.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper emphasizes that standard machine learning methods target some notion of “average loss” or “average risk” but that does not provide guarantees on performance for an individual instance, which is an important point. In particular, I enjoyed lines 186-188 “we cannot anticipate how [mistakes] will be assigned over instances that are subject to label noise. In this case, each instance where [there is a nonzero probability of a label flip] is subjected to a lottery of mistakes.”\n\nThe idea of constructing multiple plausible clean datasets from a noisy one is interesting, and seems very reminiscent of distributionally robust optimization (the idea constructing the set of plausible noise draws seems related to constructing a robustness set over distributions). It might be worthwhile to consider what connections there are between constructing the set of plausible noise draws and a robustness set.", "weaknesses": "- It would be helpful if the authors could provide additional explanation on how the notion of regret differs from standard classification accuracy, and why it is useful.\n\n- A key limitation of the approach is that it requires the machine learning practitioner to specify a reasonable noise model. \n\n- The justification for restricting the sampled noise draws to a set of “plausible” noise draws is not clear to me. Why can’t we just account for the fact that each noise draw has a different likelihood? \n\n- The ambiguity quantity is not well-motivated. Why is it defined as the fraction of misclassifications across the cleaned datasets? How do the authors intend this ambiguity quantity to be used? Under what conditions is ambiguity equal to zero?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9CJprA9Ub4", "reviewer": "Reviewer_gw9a", "summary": "This paper examines the problem of learning from noisy labels by addressing instance-level noise. The main contribution is the insight that a method performing well over the population can still lead to errors at the instance level. The paper introduces the concept of \"regret\" to characterize this phenomenon and proposes a method to mitigate the regret caused by randomness sampling multiple plausible noisy label draws. Theoretical analysis and experiments are presented to validate the proposed approach.", "questions": "- Could you elaborate further on the benchmark algorithm discussed in the paper and clarify its relationship with [37]?\n- Could you provide an additional explanation regarding the proof of Proposition 4, particularly addressing the concerns mentioned in the weaknesses above?\n- Could you include a performance comparison with other methods in the literature?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ This paper addresses an important yet challenging task: learning from instance-level label noise.\n+ Theoretical analysis and experiments are conducted to validate the proposed method.", "weaknesses": "- **Unclear Benchmark Algorithm**: One of my main concerns is the paper's clarity, particularly regarding the introduction of the benchmark algorithm critiqued in Section 2. It appears the paper intends to use a noise-tolerant method, such as that of Natarajan et al. [37], as a benchmark. However, by Proposition 5, the algorithm under discussion seems to actually refer to a different approach (let's call it Benchmark 2). In Benchmark 2, an implicit noise draw $\\mathbf{u}^{\\mathrm{mle}}$ is generated, $y_i$ is recovered using this noise draw, and then ERM is performed to train the model. To me, this algorithm (Benchmark 2) differs from the method in [37], especially in terms of instance-level performance. Therefore, it is less convincing that the criticisms for Benchmark 2 are applicable to the noise-tolerant method in [37]\n\n- **Clarity on Notation**: I find the notation in Section 2 somewhat confusing, particularly in distinguishing which variables are random and which are deterministic. Based on the discussion in lines 130-135, it appears that $ y_i $ is deterministic, while $ U_i $ and $ \\tilde{y}_i $ are random variables generated based on $ y_i $. However, I struggle to interpret the equation in line 173, as it seems $ y_i(U_i) $ is simply a deterministic value, making it challenging to see how it could be compared in inequality to a random variable.\n\n- **Regarding Proposition 4 and its Proof**: It would be helpful if the authors provided a clearer explanation of which random variables the expectation is taken over. In the proof, it appears that the expectation is taken over $ X, \\tilde{Y} $, and $ U $ while this is not mentioned in the main text. Additionally, I find it difficult to follow the reasoning in lines 734-744; the conclusion seems to rely on $ E_{X, \\tilde{Y}, U}[\\text{Regret}] $, yet the analysis is conducted for $ E_{X, Y, U}[\\text{Regret}] $. Finally, the last lines indicate only that $ E_{X, \\tilde{Y}, U}[\\text{Regret}] > 0 $, but it is unclear how this leads to the conclusion stated in Proposition 4.\n\n- **Strong Assumption**: The proposed method in Section 3 requires knowledge of $ P(U = 1 \\vert X, Y) $. This is somewhat a strong assumption to me, as accurately estimating this value is generally challenging.\n\n- **Insufficient Experiments**: It appears that the paper does not compare the proposed method with others in the literature. At a minimum, it would be beneficial to include a comparison with the noise-tolerant method [37] under the condition that $ P(U = 1 \\vert X, Y) $ is known. Although [37] is designed for class-dependent noise, with knowledge of  $P(U = 1 \\vert X, Y)$, extending it to handle instance-level noise should not be too challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJggV1IaS2", "reviewer": "Reviewer_mXWQ", "summary": "The authors tackle the situation where observations come with label noise. They introduce a criterion (regret) which measures when the prediction errors disagree when the model is computed with  noisy observation \\tile{y} and not noisy y. They develop a new method that estimate the posterior distribution of the possible noise and try to sample these observations. Hence if the distribution of the noise is well chosen, it becomes possible to construct set of plausible models and thus detect zones for which the uncertainty is above a certain level. The paper develop a new theory and provides sound mathematical proofs and simulations.", "questions": "can you define \\tile{l}_{0,1}", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The point of view which is developed is interesting and is a valuable contribution.\nThe ideas are straightforward once the frame is set : defining the regret and then plausible sets minimizing the regret on epsilon-plausible datasets.\nExperiments are convincing.\nA whole section is devoted to the theoretical analysis of the results. Proposition 12 proposes the statistical guarantees of the methos.", "weaknesses": "I found the paper sometimes difficult to read and some sentences are difficult to understand \nl159 : \"a practitioner may be able they expect ... \" I can not understand what the authors mean.\nl162 : the definition of the regret is fuzzy with some words that are not properly defined . \"anticipated\" mistake. What does it mean since you compare the error with labels with noise and labels without ?\nl172 the paper should be self-contained if possible, so epxlain the comparison with \\tilde{l}_{0,1}\nl182 : what is \" :-= \" ?\nl215: sometimes you use words that have a mathematical meaning : \"most likely to flip\" for instance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wLnINwLGrV", "reviewer": "Reviewer_cRDp", "summary": "This work proposes an evaluation framework for noisy label learning methods in terms of \"regret,\" as quantified by the discrepancy between model errors with respect to noisy labels vs. errors with respect to true labels. But regret is not distributed equally in the data: in their own words, \"even if we can limit the number of mistakes, we cannot anticipate how they will be assigned over instances that are subject to label noise.\" The proposed approach takes a generative model of noise to train a set of models on plausible (under the distribution induced by the generative model) clean realizations, and estimates instance-level \"regret\" accordingly. Empirical results show that a common noisy-label learning baseline and naive approaches (ignore noise) exhibit non-zero regret consistently. A case study on a genomics dataset demonstrates the practical utility of the approach by leveraging an instance-level ambiguity measure derived from regret to abstain from low-confidence predictions.", "questions": "* Re: [W1] — I would love to hear any thoughts on the robustness of the proposed approach to noise model misspecification from a theoretical perspective.\n* Re: [W2] — I would love to hear any commentary on how high-variance in the noise posterior could negatively affect the proposed approach. \n* Re: [W3] — Are small violations of the $\\mathbf{u}\\_{mle} = \\mathbf{u}\\_{true}$ condition (Prop. 5) truly \"problematic?\" Is there an example to demonstrate this? \n\n**Other questions/suggestions**\n* Did the authors consider looking at metrics beyond expected regret/ambiguity (e.g., worst-case over $\\varepsilon$-plausible models)?\n* The noisy label evaluated in the experiments is >10 years old; while the value of the approach isn't based on which underlying noisy label learning method is under evaluation, it might be more salient to the noisy-label learning community to test a more recent suite of methods + different noise models. For example, [some](https://arxiv.org/abs/1809.03207) [methods](https://arxiv.org/abs/2406.18865) specify a full generative model and cast the clean label as a latent variable, while [other](https://arxiv.org/abs/2002.07394) [approaches](https://arxiv.org/abs/1910.01842) filter out examples flagged as noisy (according to some rule) in the learning process. Given the plethora of assumptions/noise models in the literature, I wouldn't be shocked if there is systematic variation in errors across methods. \n\n**Minor Suggestions**\n* In Table 2, $\\hat{\\mu}(x)$ is defined as the median, but in Eq. (8), it is defined as the mean — I suggest making the definition consistent. \n* The proof of Prop. 4 in Appendix A is a little unclear: there are also some typographical inconsistencies (math mode vs. regular font), and I think $f(X)$ is mistakenly written as $X$ in one of the loss terms as L725-726. I was unable to replicate the final step, but this is likely since I had a hard time following the parentheses/whether each line was a continuation of the previous. Could this be clarified? I believe the result, since it seems to be essentially a result of the form E_{noise}[estimand] = estimand as is common in the noisy-label learning literature. \n* Is Prop. 9 (Appendix A only) not simply a restatement of Lemma 1 from [Learning with Noisy Labels, Natarajan et al., NeurIPS '13](https://proceedings.neurips.cc/paper_files/paper/2013/file/3871bd64012152bfb53fdf04b401193f-Paper.pdf)? If so, the proof can be omitted and replaced with the relevant citation. \n* Prop 10. and 11 appear to be standard applications of a weak law of large numbers + Hoeffding. If they're not referenced in the main text, consider removal.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This is a very well-written paper. The prose is clear and the technical aspects of the problem motivation are well-defined and explained concisely. \n* The proposed approach is very simple and the theoretical results are intuitive, but backed by rigorous theoretical and empirical analyses. \n* Rather than assuming completely random models of noise, the proposed approach is adaptable to arbitrary generative models of noise.", "weaknesses": "* [W1 — knowing the true noise model]: The proposed approach requires knowledge of a full generative noise model of $(U, X, Y)$. It is not clear where this would come from in practice. This weakness is somewhat mitigated by the discussion at the end of Section 3 and empirical results showing robustness of the proposed approach to noise model misspecification, but building more formal machinery to characterize the sensitivity of the approach to noise model misspecification would strengthen the paper.  \n* [W2] Proposition 4 provides the motivation for the proposed approach — using the generative model of noise, sample plausible realizations of the clean dataset. But the variance of the posterior could be extremely high — even with the $\\varepsilon$-plausibility constraint (Def. 7), this could still yield high-variance regret/ambiguity averages. \n* [W3] I'm unsure about the usefulness of Prop. 5, which \"implies that we can only expect hedging to learn a model that does not assign\nunanticipated mistakes when $\\mathbf{u}\\_{mle} = \\mathbf{u}\\_{true}$. I read this as \"models will overfit in finite samples to the observed noise draw rather than the true noise draw,\" which is intuitive. But if regret grows very, very slowly in $|\\mathbf{u}\\_{mle} - \\mathbf{u}\\_{true}|$ (any measure of distance between the two, to abuse some notation) — then it seems like this effect is not an issue. \n* [W4 — minor] The presentation of empirical results could be improved. Table 3 is very large, and it's hard for me to parse what I'm looking for. Similarly, Figures 3 and 4 could be designed a little more informatively — specifically, the caption should include a statement about why the proposed approach is \"better\" (e.g., our approach has X property, while the standard approach ... ).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)