Resurfacing the Instance-only Dependent Label Noise Model through Loss Correction
Authors: Mustafa Enes Aydın
Maarten De Vos
Alexander Bertrand
Venue: ICLR 2026 Poster (2026)
Keywords: label noise
loss correction
instance-dependence
risk equivalence
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We resurrect the instance-only dependent label noise model via loss correction that connects the empirical-noisy-risk with the true-clean-risk.

Abstract: We investigate the label noise problem in supervised binary classification settings and resurface the underutilized instance-_only_ dependent noise model through loss correction. On the one hand, based on risk equivalence, the instance-aware loss correction scheme completes the bridge from _empirical noisy risk minimization_ to _true clean risk minimization_ provided the base loss is classification calibrated (e.g., cross-entropy). On the other hand, the instance-only dependent modeling of the label noise at the core of the correction enables us to estimate a single value per instance instead of a matrix. Furthermore, the estimation of the transition rates becomes a very flexible process, for which we offer several computationally efficient ways. Empirical findings over different dataset domains (image, audio, tabular) with different learners (neural networks, gradient-boosted machines) validate the promised generalization ability of the method.

Published as a conference paper at ICLR 2026
RESURFACING THE INSTANCE-ONLY DEPENDENT LABEL NOISE MODEL THROUGH Loss CORRECTION
Mustafa Enes Aydm', Maarten De Vos & Alexander Bertrand
Department of Electrical Engineering, KU Leuven, Belgium
Department of Development and Regeneration, KU Leuven, Belgium maarten. (mustafaenes maaitn, devos, alexander beeraaddeeeeetttrreeeeeeeeeeee
ABSTRACT
We investigate the label noise problem in supervised binary classification settings and resurface the underutilized instance-only dependent noise model through loss correction. the one hand, based on risk equivalence, the instance-aware loss correction scheme completes the bridge from empirical noisy risk mininization to true clean risk minimization provided the base loss is classification calibrated (e.g., cross-entropy). On the other hand, the instance-only dependent modeling of the label noise at the core of the correction enables us to estimate single value per instance instead of a matrix. Furthermore, the estimation of the transition rates be- comes a very flexible process, for which we offer several computationally efficient ways. Empirical findings over different dataset domains (image, audio, tabular) with different learners (neural networks, gradient- boosted machines) validate the promised generalization ability of the method.
INTRODUCTION
Label noise is one of the culprits for low-quality data in supervised classification settings where some of the labels deviate from their true values. While it relatively easy to collect data, is as laborious to label them correctly in time-efficient manner, which inevitably results in errors in annotations. The machines are prone to overfitting these mislabels which results in poor generalization (Arpit
et al. 2017 Belkin et al., 2018). The need for tackling noisy labels has thereby arisen.
Surrogate loss functions are one prominent way of handling label noise in a machine-agnostie man- ner: given a possibly noise intolerant (Manwani & Sastry] (001)) loss function, "correct" it by devis- ing a new loss which is robust to label noise. For example, Ma et al. (2020) suggest a normalization trick to symmetrize any loss function, which is a sufficient condition for noise tolerance (Ghosh jet al. 2015). A main advantage of these loss correction methods is their ease in implementation: usually the correction of the base loss takes a few lines of code with very little computational over- head, compared to, say, modifying an entire machinery.
To this end, direct attention to the ultimate aim in machine learning: generalization (Mohri et al. 2012). In classification with label noise, one seeks a classifier trained on the noisy data that generalizes well with respect to the clean data. True risk quantifies the generalization error of hypothesis under a loss function; the mathematical formulation of the desideratum is therefore the following risk equivalence:
Ex,9 [(()(X), F)] = Exx (for (k(X),
where E X' is the instance random variable, Y. E {+1 are the (unknown) clean and (observed) noisy label random variables, respectively; H 3 h: X + IR is any scorer function from the hypoth- esis space H: for (h(#),)) y) (1 sgn(y h(==))) is the 0-1 loss function and É. R {+1) IR our target loss function that is minimized during the training of h. Ideally, one wants = lor- which is in fact noise-tolerant loss function (Manwani & Sastry 2013). Nonetheless, optimiza- tion against is intractable in general (Ben- et al.) 2003) and the proposed direct optimizers constrain the hypothesis space to be linear functions (Sastry et al. 2010 Nguyen & Sanner 2013)
Therefore, classification calibrated (Bartlett et al. (006) convex surrogate loss functions, e.g., lo- gistic and hinge losses, used instead. These loss functions (here denoted by E) not necessarily

---

Published as a conference paper at ICLR 2026
noise tolerant, however. In fact, Manwani & Sastry (2013) showed that neither the logistic loss nor the hinge loss is label-noise tolerant whether the noise is instance- ceetence-deeendentt or not. This eliminates the possibility e l, and devising a noise tolerant loss function is in order: noise tolerance, com- bined with classification calibratedness, form path from noisy empirical risk w.r. i to clean true risk w.r.t Lo1.
Risk equivalence in equation I iinvllves the transition probabilities to make a connection between Y and Y. This transition is an unknown process in general whose modeling roughly falls into four categories (Menon et al. 2018): ; RCN (random classification noise), CCN (class conditional noise), IDN (instance dependent noise) and ILDN (instance and label dependent noise). They are essentially distinguished by what they put on the right side of the conditional probability P(Y + 1 .): RCN leaves it empty; CCN puts Y; IDN puts X; and ILDN puts X, Y. RCN is rather unrealistic as it assumes a uniform rate of transition independent of X and Y. CCN the most widely studied noise model that assumes y LL X Y. i.e., the label noise depends only on the class but not on the specific instance. The more natural (Chen et al. 202T and powerful yet challenging assumption involves a conditioning on X by assuming IDN or ILDN, which continues to attract attention in recent research (Bae et al., 2024 Li al. 2025).
The distinction between IDN and ILDN is worth emphasizing. When "instance dependent noise" is used, what is usually meant is instance and label dependent noise, i.e., ILDD and not IDN. IDN is driven by the plausible assumption that Y, being an aggregate statistic of X, does not convey any more information than on the transition rate. That IDN involves less parameters might imply underfitting in modeling the noise; however, as empirically shown, is not less powerful than ILDN while being much more computationally efficient. Interestingly, there are only a few works that model the noise in this instance-only dependent form (Bylander) 1998 Du & Cail 2015), which both focused on linear machines. We generalize this underutilized noise model to be employable with any learning machine and provide practical ways to estimate its value per instance. This model complements the risk equivalence starting point to devise a noise tolerant surrogate loss function.
We summarize main contributions follows.
We introduce an instance-aware loss correction mechanism based on risk equivalence to fortify a given loss function to combat instance-dependent label noise effectively (theory- wise) and efficiently (computation-wiss).
We resurface the underutilized instance-only dependent noise model to estimate the tran- sition probability of each instance to complement the surrogate loss function design. We also provide computationally efficient ways to model the transition rates.
Empirically, validate our approach with moderate and high noise rates over four datasets of different domains with neural networks and tree-based models, showing compara- ble/better results against the current approaches while being dataset- and machine-agnostic.
2 RELATED WORK
Noise models. Angluin & Laird 1988) formalized the label noise problem and showed learnability under RCN where the transition rate is the same for all æ, y pairs. The more realistic CCN model appoints a distinct rate for each possible transition among classes, which is the most widely studied noise model: Liu & Tao (2016) employed data reweighting: [Zhang & Sabuncu (2018), Patrini al. (017)) and Wang et al. 2019) focused on loss correction; and Han et al. 2018). Yu et al. (2019) and Li et (2020) introduced effective neural network machineries based on the difference of loss suffered among samples. The more natural noise model where labeling errors depend on the instance itself has also been heavily studied: Xia et al. (2020) and YYan et al. (0022) learn the transition matrix for ILDN via three stage training where the former learns partial matrices to be combined and the latter uses distillation to acquire Bayes-optimal labels; Zhang et al. (202 and Cheng et al. (2021) progressively correct suspicious data pairs; Bac et al. (2024) perform Dirichlet- based resampling using the transition matrix; Du & Cai (2015) reinterpret linear classifiers from a distance-to toouunddr---buudrryyy perspective under IDN, which BBlander]] 1998) also studied under monotonic noise for perceptron learning; and Yao et al. (202T) and Li ct al. (2025) took a causal graph viewpoint in ILDN modeling.
2

---

Published as a conference paper at ICLR 2026
Loss correction. Many waachine-agnosiie loss correction methods have been proposed to deal with label noise. Natarajan et al. (2013) proposed a CCN-based surrogate loss. Zhang & Sabuncu (2018) and Wang et al.J (2119) specifically modified the cross-entropy loss for robust learning based on CCN; the former used a -ox--Cx- transformation and the latter introduced an additive term based on the (reverse) KI. divergence. [Goldberger & Ben- (2017) introduced noise- adaptation layer to given neural network to make cross entropy more tolerant to label noise. Ma al. (2020) proposed normalization trick symmetrize any loss function for noise tolerance and further improved it by weighted combinations of losses, which was generalized by (Zhou et al. (2021)
as asymmetric loss functions. Xu et al. (2019) introduced an information theory-centric loss involv-
ing the determinant of the transition matrix of a CCN noise model. Rooyen et al. (2015) proposed
modified hinged loss that is provably noise tolerant to combat RCN. Patrini et al. (2017) introduced
two ways to correct given loss function via CCN-based transition rates. Liu & Guo (2020) intro-
duced peer losses without transition rates directly involved, which achieves risk equivalence up to a constant involving the transition rates.
Perhaps the closest studies to our work are Natarajan et al. (2013), Patrini et al. (2017); Bylanddr [1998) Du & Cai (215). The critical difference from Natarajan an et al. (2013) is their lack of ex- pectation over the latent label variable Y. i.e., we would argue that it does not properly achieve risk equivalence (in fact, the expectation is present in Patrini et al. (017)) Furthermore, it is a CCN- based model, i.e,, the label noise is not instance-dependent, and the transition rates are assumed known or otherwise estimated via validation which not scalable IDN. Patrini et al. (2017) is also based on CCN and the estimation of the transition rates assumes the existence of anchor points, i.e., points whose annotations are almost surely correct, and involves a separate training stage, which incurs computational load and disconnection in learning. As for the latter IDN-based works of Byy lander (1998) and Du Cai (2015), they are focused on linear machines (former pereeptron; latter legistic/probit regressors) with one way of estimating the noise (distance-based); we, however, not only generalize the learning to nonlinear machines with a generie loss correction mechanism but also formalize and propose several computationally efficient ways estimate the noise rates.
METHODOLOGY
In this section, we first present the notation used, then introduce the inste loss correction and lastly propose estimation approaches for the instance-only dependent noise.
3.1 PRELIMINARIES
We consider the supervised binary classification problem. X' denotes the feature space, y = ÿ {+1} the label space (clean and noisy, respectively). X, Y, Y denote the random variables and x, y, ÿ their specific samples associated with X, 3 and Di respectively. H 3 X + IR is any scorer function from the hypothesis space H; 0-1 loss is given by for (h(x), v) = (1- sgn(y (x)))/22 Given loss function E IR y - R. (h) = Ex, [(()(X), Y)] is the true clean
risk; (h) x [(), Y)] is the true noisy risk; and (h) = [S] E ((((x;), 7.) the empirical noisy risk, where S = {xx } is realization of the noisy data pairs of cardinality |S]. When the subscript l on the risk is omitted, it implies Lo Par = IP(Y + Ý X = x) denotes the label flip probability of a given instance x. We assume Ex [pa] < 0.5 for learning to take place
3.2 LOSS CORRECTION VIA RISK EQUIVALENCE
We aim to form path from the empirical noisy risk mininization w.r.t. surrogate loss ? to the true clean risk minimization w.r.t. fa1- Since for is hard to optimize, a classification calibrated loss function is used instead, c.g., logistic loss: Soggh((),, y) = log(1 + e --vh(ar));; it is well-known that risk minimization w.r.t. the logistic loss parallels the minimization of the true risk w.r.t. to ie., R(h) - Rd (h) S (1) - Ri (h)), where R' denotes the minimum possible risk, i.e.,
'The machine would be indifferent to this assumption heing invalidated, which amounts to the semantic flip of the positive and negative lahels; its predictions on unseen data would simply need to be inverted.
3

---

Published as a conference paper at ICLR 2026
the Bayes-optimal one. As the true distribution of æ's never known, empirical/structurll risk minimization is instead used as a proxy backed up by the law large numbers. Once the label noise
comes into the picture, however, the connection between the Re, (h) and R(h) is lost: logistic loss not noise tolerant even when the noise rate is uniform, i.e., Pv: = p Vax (Ghosh et al. 2015). To this end, we propose to devise an é based on l, which can be any classification calibrated loss function (e.g., logistic, hinge), such that the path is complete:
Rios (h)
Re(h)
R2(h)
Law R2)
Classification Calitratedness
Our design
Largo Numbers
We now propose our loss correction scheme. (All proofs are in Appendix A.I Proposition 1. Let & be any loss function. Define
((h(z).0) IP(Y = T7 x)) - x) (a)e((((z))))) - IP(Y - p)) -ū)
PP = x>) =- x)) Pre
Then we have Ry(h) Re(h), i.e. x [(h(()), F)] Ex,y (e(((e))()))))
(The proof of Proposition in Appendix A.1 also explains the rationale behind this choice for modified loss function.)
In equation there are three unknown probabilities: PY 2)). PP x)) and Px. We now explain how estimate them in practice.
3.2. MODELING IP a)) AND IP(Y x))
PPY x) can be estimated directly from the data using any learning machine we have direct access to (æ: 5) pairs. This is not the case for IP(Y x:) as we have no access to the clean labels. However, since the scorer h also models IP(Y æ²), we set IP(Y az) in iquationntto the standard logistic over the scorer, i.e., IP(Y = = æ)) = (1 h((=))
-==hh(=))- The rationale is that,
ideally, when the training with finishes, the machine has converged to learn PP(Y) x)), i.e., the clean label probability for generalization. Since clean data pairs (despite being unknown) dominate the learning in the initial stage (Han et al. 2025), we spare few epochs at the beginning (warm- up) where we only the (unmodified) base loss to form strong baseline for IP(Y) x). As for P(Y (x), one might train a normal model for Y. and then use its predictions inside the modified loss to train h (and hence implicitly also P|YY x)). This disjoint training for IP(Y x)) and IP x) h, however, is not only time consuming but turns out to also lack in performance (as empirically shown in Appendix A.4 Therefore, in practice, we use (2 h(x)) for P(Y x)) also; as the machine is trained with the noisy labels to mimic the annotator's 's brain, it is expected to model IP( x)) in the process (including and after warm-up) up)) We provide detailed argument for this modeling choice in Appendix 1.4.1]] and an analysis on the effect of the warm-up period in Appendix A.5)
With these estimates in place, the loss in equation 3 becomes
{(h()))))) a h(=))(((-)) h(x)) Pm (((((r)))))))) - -o(-ī- (((---h()))Pprr (h(e))
(4)
o. h(x))o(-- h(x)) Px
and we move to the modeling of the last unknown, Pa-
3.2.2 MODELING Pao
The transition rate Pa = P(Y + Ý X = x)) is only conditioned on X and not on Y. Therefore, per instance, we do not estimate transition matrix but instead a single value. Furthermore, as Pa does not depend on the true label, there is nothing preventing the estimation of P.. 's in an unsupervised manner, giving flexibility. Before providing ways to estimate Pa. we formalize it over the notion of difficulty the more difficult an instance x: is, the higher the label flip probability Pa.
Definition 1 (px).. Let X be a "difficulty" mapping such that the more difficult (to label) ar the higher label flip probability Pre = IP # Y X == n)), i.e. Pm d(=(x));
4

---

Published as a conference paper at ICLR 2026
where Ø : + [0, 1] is monotonically increasing function (not necessarily differentiable), which, together with 2, also satisfies Ex [0(2(2))) < 0.5 so that there is more signal than noise for learning to take place.
obvious choice would be to use the model h itself within the difficulty map =(.). .). as the model has some notion of which labels are harder to classify than others. As such, z(-) can be learned online during training of h (see below). this case, we impose that x.) is
increasing when hxx) and decreasing when h(z)) 0. and
3pm differentiable and Lipschits]suchh that -o0 < inf Zh(m)
< < suPweR ah(m)
Den
op.
The definition necessitates the design of two functions: the difficulty metric and é that turns that difficulty into probability. We now provide several ways to model them (a nonexhaustive list).
How to find the difficulty of an instance, i.e., =<x)?
1. Offline (unsupervised)
(a) Clustering, e.g., K-means Lloyd 1982
2(x) is the (inverted) distance of to its
cluster center.
(b) Representation learning, c.g., sparse autoencoder (Lee et al. 2007 : <()) is the reconstruction error. The learned representations then used for the downstream classification task.
2. Online (while training h))
(a) Distance to decision boundary: the further away the instance from the decision boundary formed by h(x), the less error prone it is for label flips.
(b) Proximity to uniform distribution of number of votes in an ensemble: the closer the vote distribution 50/50, the higher the probability of label flip. KL divergence can be used to compute the desired proximity (inverted), for example.
How turn difficulties into probabilities, i.e., o?
1. B-logistic function: (1 + e -==2 with 3 > 0.
2. Distribution functions: exponential PDF: ] je= with B 2 1 (when the range of 2(-) is
R+); Gaussian CDF: + ferf(= with R
We found empirically that distance to decision boundary for difficulty modeling and 3-logistic func- tion for turning them into probabilities work well (these also used in the experiments in Section 4p. Finding the distance to the decision boundary is straightforward if /a is linear, i.e., h(x)) w x For nonlinear models, however, this no longer applies. One perform
(it is [//]) clever random perturbations on the sample until the decision under h changes, e.g., with DeepFool MMosavi-Dezzood et al. 2016) to approximate this distance. However, these approaches are time consuming given the distance needs calculating every iteration. Instead, we observe that if the net- work was a perfect classifier, then the embedded data its last layer must be linearly separable. In fact, Li et al.] (2019) showed that the last layer is solving an SVM problem in such network.
Building on this observation, we use [x(( as a proxy for the distance of x to the decision boundary of h. whether h is linear not (note that this choice satisfies the conditions imposed Pr earlier). As h(x) already gets computed while learning, this approach is computationally highly efficient (a sample implementation in code is in Appendix A. We also observed that this proxy also works well for nonlinear machines that are not neural networks, e.g., LightGBM (Ke et al. 2017). as shown in Section 4.
On the theoretical side, we provide a mathematical ground for this design choice, due to Menon et al. (018). They show that "boundary-consisten consistent noise" model, of which boundary design is special case, is not only consistent between noisy and clean domains for AU- ROC maximization (Proposition 1 in Menon et al. (2018) but also lends itself an explicit excess AUROC risk bound to quantify the said consistency (Theorem 2 in Menon et al. (2018)). Here, we restate the theorem by adapting to our notation for convenience:
-iraaccetto-ecisionn
"hhe reason for this requirement will be clear in Section 3.2.4 (L.mma in particular).
Cooe is also available https://gthub..om/uussaaaaaaay
5

---

Published as a conference paper at ICLR 2026
Theorem 1 (Menon al. 2018 adapted). Given Pz (dd(2z)) where o: R 10, is monotonically decreasing function and (2) is the distance of x to the decision boundary of h, suppose that Pmax max,ex Pm
:/. Then, for any scorer h,
म. (1-
Brank (h)
Byank
(Rrank Byank)
π x)-x) 2- Puax
where ãे p( = +1), P(Y==1) +1); Branke (h) denotes the true clean ranking risk of h i.e. - h(X',, 1)] and (h) the true noisy ranking risk of h, defined x similarly, and starred risks represent the Bayes optimal ones in the respective domains.
Theorem lays down theoretical ground for this particular choice of Pe along with its empirical support. We note that the assumption Pmax 2 satisfiable trivially via, e.g., halving the output of the sigmoid or using an exponential PDF with rate < 1/2.
3.2.3 INSIGHTS INTO THE MODIFIED LOss FUNCTION
With our models for the unknown probabilities in place, we can analyze several limit cases, which give some insight in the behavior of the modified loss function:
Per - 0, i.e., label flip seems highly unlikely: E(., 3) + l(, M). i.e, suffers what would normally suffer as the given label is (most likely) the true label.
Par [0.5, 1], i.e., high probability for label flip: E(, D) is a (nontrivial) weighted combina- tion of f( v) and ((., - which is reasonable as it is these "gray" areas that the machine should be careful about not leaning towards one side, which in turn making more robust for generalization.
T h(()) + oo, i.e. the machine is strongly agrecing with the given annotation: E(: 9) ((:,0) = 0, i.e., it suffers no loss as it would not normally.
T -((x)) + -00 i.c., the machine is strongly disagrecing with the given annotation: ((-,9) ((---))
0: this is significantly different behavior from label noise- intolcrant loss function, c.g., llag. which would make the network suffer the maximal loss. Here, though, are facing "obvious" mislabel. Instead of insisting on blindly agreeing with the annotation, the modified loss trusts the machine and moves on.
3.2.4 STABILIZING
The division in the formulation of in equation 3 poses danger- we have no mathematical control over the denominator and for some instances æ, can become arbitrarily close to O leading to instability in practice. The same worry carries to the gradient, possibly exploding it. Therefore, we employ regularization trick to address this: we approximate the division with repeated subtraction, and propose the regularized form of f, called ह. as follows:
7R (R(((z), Ø7 = aaumerator Aedenominator
o h(z)) T h(x)) - pc))(((())))))) - o(- (())))
(5)
- -((((ū h(x)) h(x)) Pp)) Pw).
where À > O is a hyperparameter. Even though the risk equivalence is hurt, the generalization ability with 2 remains intact under sufficient conditions on X, which we show next with high probability generalization bound. To this end, we first establish the Lipschitz continuity of j..
Lemma 1. Let ((h(ar)y)) y) be an L-Lipschitz (w.r.. h(x))) loss function with a finite upper and lower
bound, i.e. there exists an loo such that |((-,-) < Loo < CC. Assume -o inf Bh(a) dos < 00. Then, ? (H(()z)),)) is Lipschitz w.r.t. h((x)) with the
Opa
Onmin < 0 < O'max = supxEX Dh(
constant
in== : (1 +1+- (Omax Omin)) lo L (3) + Ommax ('mmin X.
We note that the assumption Ommin 2 S Ormax is reasonable as O'min > 0 would imply Par as a monotonic function of h(z)), which is not realistic since the confidence (and the distance of the

---

Published as a conference paper at ICLR 2026
decision boundary) of the machine is at its maximum when both h(x) - too and h(x) - -00. We address the boundedness imposition on (and E) after the next proposition.
Proposition 2. Given a noisy training sample S (.),, base loss function E, any hypothesis H 3 X + IR, the vectors nig and dg denote the evaluations of the numerator and the denominator of over S, respectively. Now define Ag == {ngg ds)² 1 |3] dsll2 and assume
|((;))) S Exo < 00. For X > 0, provided that \dsli] S i 1/ds(i) Vi E [1,15]]. and 2/5/ <
(mg.ds) VAs+4|811lds || 8 for any E (0,
-As ||ssl
41S1, the following inequality holds with
1|
probability at least
Re(h) S 2 +
+ 22 +3
30
log 2/5
(6)
2|S|
where LR is given by Lemma Aand (H) is the empirical Rademacher complexity of over S.
Although the practical utility of this bound is limited, serves as theoretical sanity check for replacing the numerically unstable modified loss in equation O with the more stable regularized loss ("t Indeed, Proposition 2 implies there exist settings for which there is learning guarantee through empirical noisy risk minimization w.r.f. the regularized loss e towards true clean risk minimization w.r.t. e, i.e., as if the training was performed using the clean labels. If the base loss is classification calibrated (Bartlett et al. 2006), c.g., flog. the path in 2 is completed reach R(h). We note that while no common loss function is bounded (e.g.. logistic. hinge, squared), it is not uncommon to clip the losses (see, e-g., Rooyen et al. (2015); Wang et al. (2019), which does not necessarily hinder the classification calibratedness (Rooyen et al. 2015). Then, if the base loss l is hooper-bounde (say by 6o by thresholding and then possibly downscaling), then i can be upper-bounded well; an example is given in Appendix A.6
4 EXPERIMENTS
Here, we first validate the proposed theory using synthetic noise over datasets from the image, audio and tabular domains using neural networks and decision tree-based models. We then experiment with a real-world dataset, i.e., a naturally noisy one. We note that we also perform sanity check study to show the effectiveness of the loss correction scheme in Appendix A.2]
4.1 SETTINGS
We use the following datasets: on the synthetic noise side, CIFAR-10 Krrihevskyy 2009) for images; Speakers (Rimi) (023) for audio signals (to feature a time series dataset); and Adult (Becker & Kohavi 1996), Diabetes (Bennett et al. 1971). Heart (Janosi et al. 1989). Splice (Towell et al. 1991) and Segmentation (Brodley] 1990) for tabular (to use a machine other than neural network). For naturally noisy dataset, we use Clothing IM (Xiao et al.) 2015). As our method is aimed at binary classification, we split the multiclass-aimed datasets into arbitrary 2-class sub-datasets. The ratio of the positive class sub-datasets is provided in Appendix A.7] most the datasets are balanced. We use six-layer ReLU CNNs for CIFAR-10 and Clothing IM, and three-layer ReLU MLP for the other datasets. For CIFAR- 10 and Clothing MM, the provided training-test splits are used (no clean training data is used for the latter); for others, 80%-20% split is done, where only in the test set are the clean labels used. In all datasets, 10% of the (noisy) training split is spared for validation. In all the experiments, the base loss function correct is the logistic loss fog and the noise rates refer to the fraction of labels that are actually flipped. Details of the datasets, networks, preprocessing, optimization and hyperparameter tuning can be found in Appendix A.8 a thyerrparmmeter sensitivity study of our method is presented in Appendix A.3
4.2 METHODS FROM THE LITERATURE
We compare our model with methods of different characteristics from the literature, on both syn- thetic and real-world experiments. While the details of these methods are given in Appendix A.9
7

---

Published as a conference paper at ICLR 2026
CIFAR- 10
Dvl
2846
44%
28%
4498
2816
44%
28%6
44%
2856
4496
Sormal
76.12
64.23
53.42
50,00
61,56
50000
64.50
52.73
68.38
57,00
BCN
56.82
50.00
56.43
51,33
53.92
52.58
70,28
55.78
57.98
52.73
UB
76.72
62.62
00.92
55.23
67:15
02.43
76.52
02.33
75.72
61.45
65.43
61.87
5600
60.75
69:97
65.82
Poor
66.18
69.35
61.88
57.90
57.22
50.02
66:45
63.42
65:07
67.22
APL
79.82
70:37
54.48
64-77
70.42
67.53
PTD
76,88
56.85
66.88
5543
70.17
57.55
78.07
60.90
76.65
68.98
RL.TM
77.13
33.00
55.00
33.00
67.90
21.72
78.58
19.67
70.28
21,28
78.3
54.05
50.00
06.40
50.00
78.45
5000
75.77
53.23
Cocenching+
80.22
67.23
64.58
55.00
23,57
65.17
82.43
63.53
80.57
65.40
Forward
75.00
71,90
80.77
50.9
50,00
PLC 75.87
00.48
54.73
32,97
08.88
65.32
70.75
08.78
09:18
03.95
NDX
83,43
79.33
72.38
36,45
76.77
66.70
83.27
70.23
82,96
64.76
Table Mean test accuracy (%) comparisons on CIFAR-10's five different binary sub-datasets with varying noise levels over three trials. Scores within 2% of the maximum (relative) are highlighted in bold.
we list their names here: BCN (Du & Cai 2015); UB (Natarajan et al. 2013 DMI (u et al. 2019);
Peer (Liu & Guo 2020); APL (Ma al., 2020;; PTD (Xia et al.] (020); BLTM Yang et al.
2022
Coteaching+ (Yu et al. 2019): Forward & Backward (Patrini et al.) 2017); GCE (Zangg & Sabuncu 2018) and PLC (Zhang et al. 2021). We also compare against the Normal model trained with the logistic loss, which does nothing special for label noise.
4.3 SYNTHETIC LABEL NOISE EXPERIMENTS
We artificially inject label noise to the training sets of CIFAR-I0. Speakers and Tabular datasets to simulate the problem. The injection is done by following the procedure in Xia et al. (2020) in an instance-dependent manner (that is not matched with the lesance-to-decistee
boundary model we are using for Pe). We experiment with two noise rates: 28% and 44%, which are representatives of moderate and high noise rates for binary classification, respectively. Three independent trials per noise rate are made such that possibly different labels are flipped in each trial.
Image Results. We form 5 binary sub-datasets of CIFAR-10. Results in Table D suggest that not only the test set accuracy of NDX is overall better or comparable other methods, it is also more "stable" - the performance does not sweep much across datasets as much as other models. is also featuring in high noise regime- while some models go astray under 44% noise, NDX manages to learn from the signal present. The rationale for achieving these can be attributed to the promised risk equivalence accompanying better generalization provided that the noise model is fine. While knowing the exact flip rates is practically impossible, the flexibility in modeling it via the instance- only dependent model opens doors for good approximations, combined with the risk equalizer loss correction.
Audio Results. Over 5 binary sub-datasets of the Speakers dataset, Table 2 shows that NDX is always a top-2 performing method regardless of the sub-dataset or the noise level, which speaks for its robustness. Furthermore, since the method has no dataset-specific assumptions (e.g., image- based), it is effectively applicable to dataset of audio signals.
Tabular Results. Since a loss correction approach generally machine-agnostic by design, here experiment with several tabular datasets with LightGBM (Ke et al., 2017), decision tree- based boosting machine, being the underlying learner h for our method. Since 7 with Pa (<()))((=))))
wiiee-differentiable w.r.t. (xx),, the gradient and Hessian values are available to customize the loss function used for LightGBM. We still use neural network for the other meth- ods since some of them are not compatible with machines other than neural networks (c.g., PTD, Coteaching+), and others only exposed implementations with neural networks (e.g., GCE, PLC).
The results are shown in Table 3 where NDX (NN) and NDX (LGBM) are our models with the learning machine being neural network and LightGBM model, respectively. We observe that NDX models perform quite well compared to the other methods across datasets and noise levels. GBM models are known for their effectiveness in tabular datasets (Shwartz-Ziv & Armon 2022);
8

---

Published as a conference paper at ICLR 2026
Speniters
Dvl
2846
44%
28%
4498
28%
44%
25%
44%
2856
4496
Normal
69.29
36.68
62.91
53.13
69.09
54.17
58.47
51.85
52.83
81,40
60.30
62.70
44.41
45.22
45.23
67.90
58.25
52.25
52.25
UB
69.54
00.00
64.92
56.04
57.48
57.35
07.34
50.79
60.32
55.71
55.39
$1.59
52.71
Poor
74.16
72.16
63.53
69.70
58.35
62.38
71.16
69.25
56.00
67.01
APL
82.65
78.09
09.98
60.05
79.69
56.92
57.55
83.65
67.42
79.14
31.63
60.91
50.00
76.99
60.72
39.63
RL.TM
79.78
63.67
77.04
55.94
60.85
51.35
72.73
53.42
56.63
57.21
77.40
53887
Cocenching+
79.03
67.04
72.14
49.65
68.26
50.78
77.55
56.12
65.97
62,86
Bockward
77.15
71.10
PLC 15,03
64.29
25.29
58.16
68.50
01.22
73.85
34.10
61.48
56.52
NDX
83.77
72.16
80.30
64.10
74.51
66.11
79.35
69.70
72.32
65,40
Table 2: Mean test accuracy (%) comparisons on the Speakers dataset's five different binary sub- sub-datasets with varying noise levels over three trials. Scores within 2% of the maximum (relative) are highlighted in bold.
Tabular Dalnseis
Method
Adull
Disheles
Segmentation
Splice
2876
45%
2876
44%
25%
44F%
25%
40%
28%
82.72
49.18
65.01
BCN
83.86
80.05
58.96
38.83
63.39
55.47
81.89
68.18
81.84
67.45
UB
84.29
71.10
00.06
74.39
691015
76.
25.64
68.40
49.73
66.88
70.34
67.72
Peer
78.91
79.89
68.18
68.83
59.50
59.02
85.50
76.26
73.28
72,077
51.91
PTD
78.00
76.39
66.45
67.75
59.56
$4.10
78.07
47.11
62.31
55.33
BLTM
84,05
64.94
49.18
45,85
84.38
76.38
61.1
65.94
50.82
5083
37.79
37,38
54.17
34.17
Costochingg
83,65
78.22
71.00
67.75
68.85
46.99
87.81
66.23
78.79
62.36
57.79
PLC
76.38
77.85
6154
66.23
74.86
5032
72.87
70.35
78.79
68.66
NDX
84.59
NDX (GBM)
8248
82.23
73.34
60.33
77.05
73.77
85.06
72.78
88.35
78.90
Table 3: Mean test accuracy (%) comparisons on five different tabular datasets with varying noise levels over three trials. Scores within 2% of the maximum (relative) are highlighted bold.
here we also see it in action: in the Heart dataset with 44% label noise, for example, LightGBM performed more than 10% better than the runner-up up in absolute terms. Overall, we were able to use an entirely different learning machine, a decision based one, instead of neural network thanks the machine-agnostic nature of the loss correction.
4.4 REAL-WORLD LABEL NOISE EXPERIMENT
Here we experiment with a naturally noisy dataset, Clothing (Xiao et al. 2015). We use the ~ ~1,000,000 noisy training pairs for training, and 100,00 clean testing pairs for testing the models. Note that the dataset also comes with clean training and validation splits but we discard them for all models. We form 10 binary sub-datasets of Clothing e.g., "6v8" in the header represents the task "Wind-breaker versus Down coat". Results in T that our model has also competitive performance in real-world scenario by achieving comparable mostly better testing accuracy metries in comparison to the baseline models.
5 CONCLUSION
We introduced risk-equivalence based instance-aware loss correction approach to address label noise supervised binary classification settings with the underutilized instance-only dependent noise model. We showed that when the base loss is classification calibrated, the bridge from empirical noisy risk minimization to true clean risk minimization is complete, i.e., training ma- chine with the new loss on the noisy labels promises a generalization accuracy as if it was trained on the clean labels. The instance-only dependent modeling of the transition rates is at the correction scheme, which is highly flexible in how is approximated, for which we offered several computationally efficient ways. The performance of the corrected loss is empirically validated over
of the

---

Published as a conference paper at ICLR 2026
Cliching
Normal
62.79
74.85
09:03
73.53
67.10
05.07
68.42
71.97
7228
71.46
BCN
63.24
13.10
61.56
60.46
61.99
55,42
69.50
70.20
61.21
65.65
70.29
61.90
65.33
58.06
63.56
57.74
57.71
71.06
62.77
DMI
64.33
67.95
64.44
66.33
58,00
61.59
57,89
59.27
08.30
62.51
68.52
API.
76.06
78.95
76.75
79.73
80.37
59.97
77.35
79.23
79.42
77.34
PTD
71.89
72.34
64.57
73,80
59.14
67.50
53.27
58.13
56.35
72.78
63.97
GCE
70.54
75.20
71.13
71.78
72.63
67.64 64
72.49
72.46
75.78
75.31
Coneachinge
73:17
7782
Backward
69.59
80.70
77.43
83.08
80.83
65.96
74,80
76.83
78.56
76.88
NDX
78.10
79,88
78.37
81.57
80.92
72.96
78.71
79.80
78.78
78.25
Table 4: Test accuracy (%) comparisons on Clothing ten different binary sub-datasets. Scores within 2% of the maximum (relative) are highlighted in bold.
a variety of datasets of different domains as well as different underlying machines. A natural step forward the multi-class/label generalization of the instance-only dependent noise model.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors acknowledge the financial support of the Flemish Government (AI Research Program), METHUSALEM: METH/260003; the FWO (Research Foundation Flanders) for projects G026026N, G081722N and G046925N; Internal Funds KU Leuven (projects C3/25/017, IDN/23/006 and C14/25/108); and
STA-HLT under EU Grant Agree-
ment #101080581.
A. Bertrand, M. De Vos and Mustafa Enes Aydin are affiliated to Leuven.Al KU Leuven institute for AI, B-3000, Leuven, Belgium.
REFERENCES
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, April 1988.
Devansh Arpit, Stanistaw Jastrzundeffnedbski. Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In International Conference Machine Learning, 2017.
HeeSun Bae, Seungjae Shin, Byeonghu Na, and II chul Moon. Dirichlet-based per-sample weighting by transition matrix for noisy label learning. In International Conference on Learning Represen- tations, 2024.
Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe and. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):188-155, 2006.
Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to under-
stand kernel learning. In International Conference on Machine Learning, 2018.
Shai Ben-David, Nadav Eiron, and Philip M. Long. On the difficulty of approximately maximizing agreements. Journal of Computer and System Sciences, 66(3):496-514, May 2003.
Peter H. Bennett, Thomas A. Burch, and Max Miller. Diabetes mellitus in american (pima) indians. The Lancet, 1971.
Carla Brodley. Image Segmentation. UCI Machine Learning Repository, 1990.
Tom Bylander. Learning noisy linear threshold functions. Technical report, The University of Texas at San Antonio, 1998.
Pengfei Chen, Junjie Ye. Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Beyond class- conditional assumption: A primary attempt to combat instance- deeundddde label noise. In Pro- ceedings of the AAAI Conference Artificial Intelligence, 2021.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Rep- resentations, 2021.
Jun Du and Zhihua Cai. Modelling class noise with symmetric and asymmetric distributions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2015.
Aritra Ghosh, Naresh Manwani, and P.S. Sastry. Making risk minimization tolerant to label noise. Neurocomputing, (60:93-107, 2015.
Jacob Goldherger and Ehud Ren-Reuven. Training deep neural-networks using noise adaptation layer. In International Conference Learning Representations, 2017.
Andi Han, Wei Huang, Zhanpeng Zhou, Gang Niu, Wuyang Chen, Junchi Yan. Akiko Takeda, and Taiji Suzuki. On the role of label noise in the feature learning process. In International Conference Machine Learning, 2025.
11

---

Published as a conference paper at ICLR 2026
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: robust training deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, 2018.
Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart Disease. UCI Machine Learning Repository, 1989.
Guolin Ke, Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area v2. In Advances in Neural Information Processing Systems, 2007.
Jiahui Li, Tai-Wei Chang, Kun Kuang, Ximing Li, Long Chen, and Jun Zhou. Learning causal transition matrix for instance- dependent label noise. Proceedings of the AAAI Conference Artificial Intelligence, 2025.
Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi-
supervised learning. In International Conference on Learning Representations, 2020.
Yu Li. Lizhong Ding, and Xin Gao. On the decision boundary of deep neural networks. arXiv preprint IrXiv:88066 05385, 2019.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. In IEEE Transactions on pattern analysis and machine intelligence, 2016.
Yang Liu and Hongyi Guo. Peer loss functions: learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, 2020.
Stuart Lloyd. Least squares quantization in PCM. IEEE transactions on information theory, 1982.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In International Conference on Ma- chine Learning, 2020.
Naresh Manwani and P. S. Sastry. Noise tolerance under risk minimization. IEEE Transactions on Cybernetics, 43(3): 146-1141.111 2013.
Aditya Krishna Menon, Brendan Rooyen, and Nagarajan Natarajan. Learning from binary labels with mnstance-depeneen noise. Machine Learning, 10)6--0)) September 2018.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. The MIT Press, 2012.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method fool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in Neural Information Processing Systems, 2013.
Tan T. Nguyen and Scott Sanner. Algorithms for direct 0-1 loss optimization in binary classification. In International Conference on Machine Learning, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
12

---

Published as a conference paper at ICLR 2026
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Aklima Akter Rimi. Audio Classifier Dataset. Kaggle, 2023.
Brendan van Rooyen, Aditya Krishna Menon, and Robert C. Williamson. Learning with symmetric label noise: the importance of being unhinged. In Advances in Neural Information Processing Systems, 2015.
S. Sastry, G. D. Nagendra, and Naresh Manwani. A team of continuous-action learning automata for noise-tolerant learning of half-spaces. IEEE Transactions Systems, Man, and Cybernetics, Part B (Cybernetics), 40(1):19-28, February 2010.
Ravid Shwartz-Ziv -iv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 2022.
G. Towell, M. Noordewier, and Shavlik. Molecular Biology (Splice-junction Gene Sequences). UCT Machine Learning Repository, 1991.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In IEEE International Conference on Computer Vision, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama, Part-dependent label noise: Towards instance-dependent label noise. In Advances in Neural Information Processing Systems, 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. dmi: a novel information-thhorrticc loss function for training deep nets robust label noise. In Advances in Neural Information Process- ing Systems, 2019.
Shuo Yang, Erkun Yang. Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-deeendent Bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning, 2022.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. In Advances Neural Information Process- ing Systems, 2021.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning. 2019.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In International Conference on Learning Representations, 2021.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, 2018
Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. In International Conference on Machine Learning, 2021.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PROOFS
Proof of Proposition
We start from the desired risk equivalence:
E [x F)] = Ex,r V)
Ex Ex xx ((, Y) X]] = Ex (e(, Y) X]]
(2)
J B X PY x = m] dpx (ar) = J Eyxx Y) x = a:] dp
We now expand the integrands and require them to be equal for all x E X, which is a sufficient condition for equation
P( = x x) E(, +1) - p(Ý = -1 xx)
PY -1| x = x) ((., +1) + P(Y - =1) (4)
To make use of ;= P( + Y X = x), we employ the following trick based on a (trivial)
equality: P(Ý = +1 æ))
P Y +1 x)) Pw) + Pas and similar for the Ý case.
Substituting these into equation
4 we get
(P(( +1 X a) pm E(. +1) + Pn. p= +1)
+ (PP( = -1 = Pa 0( -1) + Poel(, -1) PP =11 x)) ((, + IP = -1 1 (x:))((;->))
(5)
We now split equation into parts and form the following linear system, which is sufficient condition for equation
(P(Ý +1 X = x) Pa + + Pa Pae(( +1) + - - Assuming a unique solution, equation yields
P(Y +1 X x x=))
-
-1) - P(Y = = x-)) -1). (6)
+1,+1) = IP(Y = +1 ax) (IP(Y = =1 (x) - Pm) e(:, +1) - IP(Y = 1 f(.--1)
P( = =11 x) P( = -1 x)) Pr
{( -1) PP(Y = - m)) (IP(Y +1 .r) Pm) ((,--1) -1) - PPYY +1 2x e(,
PP-- +1 1)) PP = x:) Pm
We unite the individual loss terms in equation 7 to arrive at
E(.0)- = PPY = 5 (PP( = - ar) - Px))((:))) - IP(Y = - -
(8)
IP( = - IPP -ūi xx) - Pr:
We make the following remarks about the proof.
Why subtract Pa in the trick and not anything else? In CCN label noise model for a binary classification, we would have the two noise rates as p+ == PPÝ = -1 -1
y = +1) and p_ = P(Ý- = +1 y== -1). In Natarajan et al. (2013), the following relationship was then proposed to construct a modified loss for the CCN setting:
(1- - 1--p)) +1) +1 ((.,+++
p- p-( + (1-p)))(11 =(( ((:-
We argue that equation 6 forms natural extension of equation to the IDN setting, including some additional corrections to ensure risk equivalence in equation I First, on the RHS we add probabilities on Y take into account the expectation Y in the RHS of equation [ Similarly,
14

---

Published as a conference paper at ICLR 2026
on the LHS, the non-flip rates (set to I in equation 9 should take the uncertainty on Y into account Finally, the single probability of flip Pe is in lieu of two
according to the LHS of equation [ different ones in the CCN. The resulting system hosts a weighted combination of & for a "robust" loss correction the label noise sense, i.c., the weights are the label flip and non-flip probabilities in each of the +1 cases.
What if there is no solution the linear system? That happens and only if P( x) - Ø) = P. for given æ, y pair. Mathematically, there is nothing preventing this (though, certain measures be taken in generating P= satisfy, e.g., IP y x x)) -- x) Pe for any æ, W) However, since the existence of the division leads to numeric issues to begin with (e.g., being close to 0 for some instances x),, go for the regularized version of the loss function and transfer the worry to the regularization coefficient's selection, which an easier issue to handle.
Proof of Lemma 7
Since ( is differentiable w.r.t. h(x:), it suffices to seek upper bound to (the absolute value of) its derivative w.r.t h(x). We consider four cases of {ū = =1} x {h(x) $ 0} and take the maximum of the bounds found therein.
OPR (h(x). 7) (- -a) (a)(()()))
@h(x))
0p= ((h(z)) + (oor(-h(a)) Px)) @l(h(z)))) ah(x)
+ o(Th(x)))
(o(h(=))(
ah(x)
0pm ((((x), 17) Pa 0h(x)
ol(h(z)),-))
+(1-o(-;M=))
0h(x)
0pm
-Modbee)ll--dd0)))))-4-4aaaaa-0- -a))
0h(x))
(x)(
(1)
Case 1:
OēR (e)()()),+
max {3 ammn
33
Oh(x))
fo + L
+
8
+ amax
X,
(2)
3
1
2
+
0max
Lo
+
2
+
8
amin
Case 2: = -1,(xxx) > O.
Oe (h(x), -1)
1
< max
amin
lx +
= 1
+
Omax
A
0h(x))
4
2
2
8
(3)
+ O
loo + I+
Ommn
4
Case 3: 5 +1, <
Oe (h(x). +1)
< max
Omin + Omuxx
loo
+
3
++ (1
Omin )2,
0h(x))
8 2
2
(4)
1
3
3
3
{
+
O'ax
lxx +
L
+
ammi
4
2
8
Case 4: -11-((1(2) <
OēR (h(x), +1)
{
1
1
max
Ommin
loc +
-
Olmax
X,
dh(x)
(5)
1
3
3
3
}
O max
loo +
L
Omin
8
Combining the upper bounds in 2-5 with "element-wise" maximums yields the given T.n.
15

---

Published as a conference paper at ICLR 2026
Proof of Proposition (Mohri et al. 2012
The generic empirical Rademacher complexity- based generalization bound
log 2/6 2|S]
R2( S R₂(N) 2 36 V
(1)
Since is derived from the risk equivalence between noisy and clean domains, we have R7(h)
Re(h). To replace the empirical risk w.r.t with that w.n.t &R, we need À to satisfy R2(h) E
Rix (h) Vh E H. i... [ >. e(h(a:), 3h) s 1ST E (h(((z)), V.). We note that element- wise inequality is sufficient condition for this to hold, i.e., ((h(zx)) Ui) (R(())))) B.) Vi [S].) Writing the loss terms in terms of the numerator and denominator evaluations of l, we have hssti/dssb & [ Ads[i] Ví. Rearranging this gives the first condition on À as Adsli] < ns [i] (1 (t] vi [S]]. To "denoise" Rs(E H), i.c., be left with the complexity of the function space only, we first show that upper-boundess by Rs(RR H)) under some sufficient conditions on X, which is further upper-bounded by a constant multiple of (?) using the Lipschitz composition property of the Rademacher complexity. To this end, recall the definition of the empirical Rademacher complexity:
S1
1
Rs( H)
E
sup NEW
of (.E(h(2r)).
|
1 |
Eg
sup hEh
(o, Sh
(3)
where og are i.idd. Rademacher random variables, [(o:)) and En = [(((h(x))) T.) Note
that og and x, are mutually independent. We first lower-bound R.g (RR H) as follows:
R o H!
Eg
sup KEH
(a,
1 |
B [1o s
(4)
[
1
Egg
||o||
2
+ 11
--lo &R,ss
(5)
2|55|
1
2|5||
for any given h E H. where we used the definition of the supremum, polarization identity, in- dependence of from h,st monotonicity of expectation, the fact || = SS], and imposed |12 < S). We now upper-bound ( H.) as follows:
Rs(E H)
1
Eg sup (a, {nss) S
||0
sup NEM
||en,sl22
[
NEH
|Ss
|k S 8000
sup heH
S|
where we used the Sauchy-Schward inequality and the independence assumption. Now we further impose À such that the upper-bound in equation g is to bound the lower-bound in equation X from below for any h E H, i.e.,
7,0 S
Adsl12
2
Combining this with the imposed assumption for equation 6 we have 2| n
xds 113 <
4 S| This is set of second degree polynomial inequalities over À and the following are sufficient for it to hold:
(ms, ds) |ds 2
As + 4|s|dsss1
-As
21
<|
(10)
|dsll
dg
1
16

---

Published as a conference paper at ICLR 2026
1 VS
2 vs
4 VS
7 vs 8
100
60
Test
NDX
10
20
30
40
45
10
20
30
40
45
0
10
20
30
40 45
0
10
20
30
40
45
(%)
Figure 1: Test accuracy comparisons between the proposed method (NDX; solid) and the Normal model (dashed), i... eft versus bao on random sub-datasets the CIFAR-10 dataset when the noise process is favorable. The horizontal dashed line is the 50% level, i.e. the test accuracy of random guess model since the datasets are all balanced. The proposed method is quite stable and performs better than the model with the noise intolerant loss even when the noise rate is close to 50%, where the Normal model performs worse than the random guess.
log
With the Lipschitz composition property of Rademacher complexity, we have Rs (ER o H) s &RRs(H) The chain is then completed, i.e.,
( H) 2 S 2,5) ns Adsl13 Rs H) S LRRs(H))
(11)
and the proposition follows.
A.2 SANITY CHECK OF THE LOsS CORRECTION WITH MATCHED NOISE MODEL
To demonstrate the efffcctiveness of the loss correction when the flip rates well-modeled, purposefully inject synthetic label noise to a given dataset the way we are estimating it, i.c., based on
proposed in the last paragraph of Section 3.2.2] To this end, first train a three-layer ReLU MLP over the dataset with ling without any consideration of the label noise and compute the transition rates based on the distances. We then flip the labels with these rates and train MI.Ps with fing (named Normal) and eR for comparison.
log (named NDX for "noise depending only on X*)
We pick four binary classification sub-datasets of CIFAR-10 randomly, Means of the test set accu- racy over three trials are plotted in Figure against the noise rates ranging from 0% 45%. Across all four datasets, while the network trained with llog experiences dramatic decrease in performance as the noise rate goes up (so much to go below 50% at times, i.c., worse than random guessing for a balanced dataset such as the ones used here), the one with Lit 'oog
is much more stable, better at each rate and manages to learn from the signal even when the flip rate is close to 50%, which underlines the claimed generalization ability when the noise process is well-modeled.
Furthermore, the leftmost points of subfigures in Figure TaareeToooee correspond to 0% noise rate experiments on CIFAR-10 10. We observe that our model's performance is not unnecessarily degraded and practi- cally the same as the Normal model's ' on all four subdatasets.
A.3 HYPERPARAMETER SENSITIVITY
The proposed method has two hyperparameters to tune: helps approximate the original division in the loss function with repeated subtraction:
the regularization coefficient that
T ? mumcratorr Xe
and 3 > 0 the logistic function's scale parameter, which function we use to turn distances-to- bousddnyb into pprobdbiiipbbabiiitiis P. = o (=))h where h X - is the learning machine. To check the sensitivity of the method with respect to the hyperparameters, we present the frequency table the selected B combinations over the 5 sub-datasets of the CIFAR-10 dataset and the Speakers dataset (results of which were both presented in Section 4.2 Note that there
17

---

Published as a conference paper at ICLR 2026
X
Frequency
À
B
Frequency
10.00
1.0
5
0.01
20.0
10
1.00
1.0
4
0.50
20.0
3
0.01
1.0
3
0.50
7.0
3
0.01
7.0
3
Table 5: Frequency of selected hyperparam- eters of NDX over sub-datasets of CIFAR-10 over 30 sub-experiments.
Table 6: Frequency selected hyperparam- eters of NDX over sub-datasets of the Speak- ers dataset over 30 sub-experiments.
30 exteebxeerments for each dataset: 5 binary sub-datasets 2 noise levels 3 trials, and the validation set is 10% of the noisy training set as mentioned in Section [4] Tables (nd and show that 3 and À combinations, respectively, already account for 50% of the experiments done, i.e, the distribution to the chosen - 3 values are far from uniform and instead cluster around 2-3 unique values of the parameters, showing the rather insensitive nature of the method with respect to its hyperparameters. Note that the need for validation is not relinquished as the chosen values are different across the image and audio datasets.
A.4 DISIOINT LEARNING OF IP(Y x) AND IP x))
Here, compare setups for estimating P(Y X) and IP(Y X):
1. The disjoint estimation where we first train a network with 4log (and no label noise specific adjustment) to gather IP(Y X) values per instance, and then train different network with
ZR while using the frozen P(Y X) values in the loss correction.
2. Joint training where both P(Y X) and IP(Y) X) approximated the way described in
Section 3.2.
Subset
Noise Level
Disjoint
Joint
20%
79.08
92.34
Ov4
40%
64.42
68.25
20%
68.38
79.90
2v7
40%
60.1
62.87
20%
67.11
73.09
5v6
40%
61.30
61.79
Table 7: Test accuracy (%) comparisons on three different sub-datasets of Speakers dataset with varying noise levels.
We perform the experiment on three random sub-datasets of the Speakers dataset of Section 4.3 the experimental setup is the same described therein. As shown in Table somewhat surprisingly, the disjoint training performs considerably worse than the proposed approximation. also takes more time to train while being memory heavy requires saving N floating point numbers into memory. We attribute the empirical success to the warm-up period we employ before the loss correction kicks in, during which the "clean" Y's dominate and later, the correction refrains the learner from overfitting to the noisy ones such that two disjoint stages are blended into one in that sense. We thereby have an empirical support for our way of modeling clean and noisy label probabilities.
A.4.1 ON USING THE SAME APPROXIMATION FOR BOTH IP(Y x)) AND (Y x))
We first note that IP(Y) at's estimation in statistically consistent (i.e., training if with the clean labels as the sample size grows) or probabilistic models in general requires design choices. We exemplify from the literature as follows.
"Learning from Massive Noisy Labeled Data for Image Classification" (Xiao et al. 2015):
they concurrently fit two models to model Y and Z (the label noise kind latent variable)
18

---

Published as a conference paper at ICLR 2026
and maximize the likelihood of Y with Enpectation-Maximizaa That is, they exploit the factorization (of Y over Y) the class-only dependent label noise model allows, and maximize the (incomplete) likelihood. However, the resulting framework has 3 models in it (one for Y. one for Z and one for the transition probabilities) and the optimization via EM gets complex (e.g., requires careful initialization) and requires an identified set of clean labels.
"Learning with Noisy Labels" (Natarajan et al. 2013): when they develop their "Method of Unbiased Estimators", they require H[e(Y, f())]] = {(Y, f(x)) for all Y, f(-) values. Please note that there is no expectation on the right hand side. Therefore IP(Y æ)) is nonexistent in their formulation, making it not achieve risk equívalence (we also point to this fact in the manuscript (end of related work)).
"Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach" (Patrini et al. 2017): This is multi-class generalization of the above work of Natarajan et al. (2013): this time we see the expectation on the right hand side. Building on the factorization IP æ) = P(Y T))) x) and their assumption of y-only dependent label noise, they aim to estimate P(Y Y)) i.e., a K x K transition matrix (K: number of classes). Their (one) design choice is to train a separate neural network to model this matrix by assuming the existence of perfect samples (i.e., those having almost surely clean labels). Therefore, they first learn a transition matrix, freeze it and then use it in the second phase of learning a new machine on Yss to uncover PY æ). This modeling of the transition matrix turned out to perform really well, Forward & Backward loss correction methods from this paper is still of high relevance in label noise research (theory- and performance-wise)
Our instance-dependen label noise model, while more natural/powerful (Chen et al. 2021), does not lend itself to the factorization above because in Pz P(Y + Y x). Y and Y are the left side of the condition together. Nevertheless, we could still learn the "annotator's 's brain" " in a separate machine, i.e., model IP( x)), then freeze its in-sample predictions. Then, in a second phase, while training an h with e (on the same dataset still), use the frozen predictions of the former machine as proxy for PY X), and the current output of h as proxy for P(Y x) (since with é, this phase's 's machine / is expected to uncover P(Y x) by Proposition 2). In the previous section (Appendix A.4.I). we present a detailed empirical comparison of this disjoint approach with what instead do do-the the difference in performance (and naturally also the computation time) was significantly worse in this disjoint way of modeling those probabilities, providing empirical evidence for our design choice.
While that empirical evidence suggests one can do better than disjoint modeling, we directly justify our assumption as follows. As noted above, in Patrini et al.'s loss correction design (Patrini et al.) 2017]. they assume the existence of perfectly clean examples. on/with which they first train a net- work to model IP Y) (and similarly, also in Xiao et al. 2015)) In our setup (or in any noisy label learning setup for that matter), while the identification of the clean labels is not assumed, their existence is, e.g., our Elpz] < 0.5 assumption. In fact, in a K -lllass scenario, one needs at least 1/K clean samples to exist so that learning is even possible (Menon et al. 2018). (Actually, training is possible either way, e.g., as an extreme case of 100% label noise in binary classification, a (good enough) model's predictions flipped, at which point the semantic labels are matched. So the assumption of high ignai-to-noise ratio is to preserve consistency of the semantic meaning of the labels between training and testing sets.) What's more, it has been demonstrated by Han et al. (2025) that the neural network first "fo- cuses" on these clean samples in the early stages of the training. But this means we can elevate the disjoint modeling idea of Patrini et al. (017)) of IP(Y a>) and IP(Y a>) with the dominance of clean samples in early stages of training to make the modeling joint: a number of warm-up epochs at start with the normal (uncorrected) loss. Once the "groundwork" of establishing a decision boundary by the machine in the warm-up period is done with the usual loss function through the dominant clean samples, our loss correction mechanism kicks in to make the machine more aware of the pitfalls due to label noise (this "awareness" is examined mathematically where we present similarities and differences between i and E).
the unseen data will always be worse than random until they
several edge cases in Section 3.2.3
19

---

Published as a conference paper at ICLR 2026
28% noise
44% noise
Testing Accuracy (%)
84.50
Testing Accuracy (%)
81
80
84.25
79
84.00
78
83.75
77
83.50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#WarmUp / #Total Epochs
#WarmUp #Tota Epochs
Observing (approximately) U-shapes with rather flat maxima while sweeping the rate of number of warm-up epochs from 0 to 1, against test accuracy. This supports the need for the warm-up period as well as the (in)sensitivity against it. Note that the rightmost points represent full warm-up without any loss correction, i.e., the Normal model.
Figure 2:
A.5 EFFECT OF THE NUMBER WARM-UP PERIODS
Here, we perform an experiment on the effect of the number of warm-up epochs to model's gener- alization ability by sweeping it the need for it well the sensitivity against it. We sweep the ratio "#Warm-Up Cup Epochs/ #Total Epochs" from 0 to 1, both ends inclusive. Ideally, we expect reverse U-shape with rather flat maxima w.r.t. test accuracy represent:
1.
The trade-off: when the ratio is 1, model reduces to the "Normal" model, i.e., does nothing special about the label noise and has an inferior performance than the other rates; when the ratio is O, dominance of the clean labels is not sufficiently utilized to form a reasonable baseline to correct;
2. The insensitivity against the number of warm-up epochs in the middle region rather flat maxima.
We use the Tabular Adult dataset (Becker & Kohavi 1996) with 28% and 44% label noise in Section 4.3 We train and validate our model as done in the usual experiments while sweeping the warm-up rate. We plot the warm-up rate versus test accuracy with different noise rates in Figure We roughly observe the mentioned trade-off and insensitivity over the number of warm-up epochs (while not being perfect U-shapes), supporting the warm-up strategy. The insensitivity also allows for not treating the number of warm-up epochs as hyperparameter that needs heavy tuning (also case in, e.g., et al. (202)), Yang et al, (2022) and /Zhang al.) (202T)).
2
A.6 BOUNDING ?
Even though there is nothing mathematically restraining from going to infinity, it can still be bounded loss function provided that the base loss é is also bounded and Pr satisfies certain condition. For example, if, for a given / E H and a finite M,
Ps (((h()))o((M((z) min
((h(x). D) +( -7)
((h(z).)) M
} Vx. V.
+ +((i() -D) M
then [0(---)] max AM = T.c. An exemplary Pa would be max(0, (1(=)))((-tt s) for small ε > with lx = 50/40 and M = 1.5 loc. i.c., there exists a P.r configuration that preserves boundedness of f.
20

---

Published as a conference paper at ICLR 2026
Note that the threshold the original unbounded loss function for finite upper bound cannot be chosen arbitrarily small, as the gradient will vanish for even small activation values close to O, severely hindering learning. For example, while the raw input in absolute value to the logistic loss barely crosses 10 in practice (making it candidate upper bound), would be too stringent to, say, clamp the loss 1.5 only to get lower gap the RHS of the bound, Furthermore, while scaling down any loss function has no effect for the optimization problem (all other things being equal), again, in practice, heavy downscaling to get a lower gap matter of trade-off since numerical issues may arise (e.g., underflow) and ii) it requires extra care in tuning the optimization related hyperparameters, e.g.. the learning rate case of gradient- baatieett-aseddd methods. More importantly, though, the risk values themselves in the bound (equation () are scaled down since they involve averaging the losses suffered, so the interpretation of the bound stays the same up to a constant factor.
A.7 TEST SET DISTRIBUTION OF THE DATASETS
Since we report accuracy scores on the (clean) test sets, here we present the ratios of the positive (latter) class in each (sub- Jdttsee used the experiments:
Subset CIFAR-10 10
Ov1
2v3
4v5
6v7
8v9 50%
50%
50%
50%
50%
Speakers
43%
56%
55%
49%
48%
Adult 24%
Diabetes
Heart 49%
Segmentation 58%
Splice 46%
Tabular
35%
Subset
6v8
6v7
6v9
1v6
2v6
0v2
2v9
2v11 51%
1v7
0v11
Clothing M
48%
50%
52%
48%
48%
50%
51%
48%
51%
Table 8: Ratios of the positive class in the testing sets of the datasets.
We observe that the sets are mostly balanced with the exception Adult and Diabetes tabular datasets, for which baseline (a model predicting 0 regardless of the instance) would achieve 76% and 65% testing accuracy, respectively. We see from the results Section 4.3 that the best perform- ing models considerably pass these thresholds (e.g.. 84.53% and 73.34%, respectively (under 28% noise)).
A.8 DETAILS OF THE SETUP OF THE EXPERIMENTS
Datasets.
The well-known 10-class classification of 32x32x3 RGB images; 50,000 training and 10,000 test samples.
CIFAR-10 (Krizhevsky 2009):
Speakers (Rimil 2023): : This is times series-basee based dataset aimed for 11-class classification of You clips of famous motivational speakers. Each clip five second signal and the corresponding label is the name of the speaker. 6,204 training and 1,551 test samples.
Tabular datasets.
1. Adult (Becker Kohavi 1996): predict whether the income an individual exceeds
a certain threshold; 48,842 instances (11,687 positive), 14 features.
2. Diabetes (Bennett al.
1971): predict whether given patient has diabetes; 768 instances (268 positive), features.
3. Heart (Janosi et al.
1989): predict whether given patient has a heart disease; 303
instances (165 positive), 13 features.
Splice (Towell et al.
1991): predict whether a given boundary at the splice junction of DNA is acceptor donor; 3,175 instances (1,527 positive), 60 features.
21

---

Published as a conference paper at ICLR 2026
5. Segmentation (Brodley 1990): assign group number to hand-segmented 3x3 areas of various outdoor images; 2,310 instances (1,320 positive), 18 features.
Clothing M (Xiao et al. 2015): This dataset is a large-scale real-world dataset of cloth images that is naturally noisy. It provides ~~1,000,000 (noisy) training pairs and 10,0000 (clean) testing pairs. There are 14 classes and the images are 64x64x3 RGB. We note that we do not use the clean training and validation sets the dataset also provides.
Machines. We use neural networks and LightGBM as the underlying machines in the experiments.
MLP: three-layer feed-forward neural network with ReLU as the hidden activations. Number of hidden neurons are 128, 64 and 32 towards the (single) output. Output acti- vation is identity.
CNN (for CIFAR-10): two convolution ReLU ReLu-maxpoo) layers (kernel sizes 6x6 and 16x16, respectively with stride of 5 and no padding for both layers) followed by 128- 64-32 fully connected layers towards the (single) output. Output activation is identity.
CNN (for Clothing MM): a four convolution-Ke Reeioon-Re-----aaaa- dellU-mappol layers (kernel sizes 32x32, 32x32, 64x64 and 64x64 respectively with stride of I and no padding for all layers) fol- lowed by 128-sized fully connected layer towards the (single) output. Output activation is identity.
LightGBM: default parameters are used except for the number of trees and the learning rate, which are tuned.
Preprocessing. All datasets undergo standardization (subtract the mean, divide by the standard de- viation; both statistics are obtained from the training split) after the following specific preprocessing steps are applied:
CIFAR-10: random crops and random horizontal flips.
Speakers: Mel-frequency cepstrum with 128 mel bands such that inputs are akin to a 2D image of shape 128 x 157.
Tabular datasets: For the Adult dataset, categorical features are one-hot encoded yielding 100 features in effect.
Optimization. For all experiment runs, 20 epochs of SGD with a batch size of 128 (32 for Speakers)
used (first 4 epochs are for warm-up in NDX). 10% of the training is spared for the validation of hyperparameters with grid search: learning rate, regularization coefficient (A) and sigmoid scale (3) for NDX; learning rate for other models. Noncritical hyperparameters, any, are taken from the respective papers/source code as is, against which most of the methods assert to be rather insensitive anyway (c.g.. 'iteration nmf" of PTD (Xia et al. 2020) is taken to be 20; "forget_rate" of Coteach- ing+ (Yu et al. 2019) (which is actually not mentioned in the paper) is set to 0.2). Validation is done for more prominent hyperparameters, e.g., P and p_ of UB (Natarajan et al., 2013) are searched in
(0.1, 0.3, 0.4]; of Peer (Liu & Guo 2020) in [0.1, 1., 5. and and 3 of APL (M et al. 2020)
in [0.1, 1., 10.]. The center learning rate (call v)) is 0.01 for all experiments; 1/1000, 7// 10, 10y, 100n the learning rate search space. For NDX, X E (0.01, 0.10, 0.50, 1, 10) and 3 (1, 20, 50) are searched. As an exception, Adam optimizer is used in Coteaching+'s training as advised in its paper (Yu et al., (2019). For LightGBM, number of trees are searched in (10, 25, 50, 100, 150) and the learning rate in (0.01, 0.05, 0.10, 0.50, 1).
A.9 DETAILS OF THE METHODS FROM THE LITERATURE USED IN THE EXPERIMENTS
Here give brief details on the methods used in the Experiments in Section 4.2 for comparison over the real-life datasets.
BCN (Du & Cai 2015) is modified logistic regression model with boundary-consistent instance- only dependent noise model. UB is the "method unbiased estimators" of Natarajan et al. (2013) with CCN-based noise model. DMI (Xu et al.) 2019) is CCN-based loss correction method from an information theoretic view that involves the determinant of the transition matrix. Peer (Liu & Guo 2020) represents the peer loss that achieves risk equivalence up constant involving noise
22

---

Published as a conference paper at ICLR 2026
rates. APL (Ma al. 2020) introduced normalization trick to make any loss function noise toler- ant and uses combinations of un/normalized (active/passive) loss functions for better generalization. PTD (Xia et al., 2020) is based part-dependent ILDN modeling of the noise which, for each part of given instance, first learns representations and then transition matrix by utilizing almost surely non-noisy samples. BL7M (Yang et al. 2022) is also IL.DN-based model which brings in the Bayes-optimal labels through which transition matrix is learned and revision is lastly made as in PTD. Coteaching+ (Yu et 2019) is an improvement over the Coteaching model (Han al. 2018) that uses two neural networks in parallel and proposes small-loss and disagreement based cross update rule to deal with the noise. Forward and Backward (Patrini al. 2017) are CCN-based loss correction mechanisms where first normal network is trained to estimate the transition matrix and is then fed to another one to correct predictions or the loss itself, respectively. GCE the general- ized cross entropy loss function proposed by /Zhang & Sabuncu (2018), which uses a Box-Cox-liic transformation to correct the logistic loss. PLC (Zhang et al., 2021) progressively corrects suspi- cious data pairs with ILDN- -DN-based noise model. Lastly, note that also experimented with DivideMix (Li et al. 2020). neural network machinery with distillation of small-loss examples via Gaussian mixture model; however, it did not attain reasonable ably above 50% test accuracy) for any of the binary classification tasks presented in Section and therefore is excluded from the results.
with validation (consider-
A.10 CODE FOR THE LOsS CORRECTION
As mentioned in Section T loss correction methods are generally very easy to implement. Our method with probabilities calculated through distances to the decision boundary requires little extra computation thanks to the observation mentioned at the end of Section 3.2.2 i.e. distances are approximated with |h(z)] where h(x) already calculated in the training loop as an input gets feed-forwarded. Here, we share sample implementation of this loss correction mechanism in code using PyTorch (Paszke et al. 2019) where we correct Eiog.
import torch
from torch import exp, log, sigmoid
def loss loss_fun (self, X, y)
Given (batch) pairs x and y (noisy), return "1"R
## Feed-forwaz forward
h_x x self (X) squeeze (-1) (-1)
f_x x = sigmoid (h_x)
## Calculate distance-based probabilities of label flips 'self. seelf.sigmolse gigoid_sccllle is \beta
distances h_x.abs
rho_x = (1 exp (self. sigmoid distances)) )
# Compute the corrected loss
# Base loss (Nell; logistic) w.r.t. "y and -"y
nornal_lose
torch.where (y +1, -log (f_x), x) -log (1 f_x))
opposite_loss =
(y == +1, -log (1 - f_x), -log (f_x))
# Approximate ( X) and P X), form the modified loss pyy_x torch. where (y = +1, f_x, worrhhwweeee/e
f_x)
numerator ((1 pyy_x rho_x) pyy_x normal_los loss rho_x (1 pYy_x) opposite_loss) denominator Pyy_x (1 pyy_x) rho_x
'self. reg..rriaaaiiiiiion.. scale is \lambda
loss = numerator - rew.laaauuii scale + denominator
return moss.mean/ ()
23


Reviews:
[{"review_id": "drreYT5atY", "reviewer": "Reviewer_5k6V", "summary": "This paper tackles label noise in binary classification by reviving the instance-only dependent (IDN) noise model, where label corruption depends solely on the input instance. The authors derive a theoretically consistent loss correction scheme ensuring risk equivalence between noisy and clean risks, and stabilize it via a regularized variant with generalization guarantees. The instance-level noise rate ρₓ is efficiently estimated from model confidence (|h(x)|), enabling online learning. Experiments on image, audio, and tabular datasets show strong robustness under moderate-to-high noise.", "questions": "Have you tried to training two model approach, one for P(Y|X) and one for rho_x ?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper offers a solid theoretical foundation: Proposition 1 rigorously establishes risk equivalence, while Lemma 1 and Proposition 2 ensure stability and generalization.\n\nExperiments across image, audio, and tabular domains with diverse models (CNN, MLP, LightGBM) confirm robustness and broad applicability.", "weaknesses": "Justification for using P(Y|X)=P(Y'|X) : This is a critical step in the method that needs justification. Without proper justification, it seems that there is a significant gap between the analysis and the actual proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "neZH5OARcp", "reviewer": "Reviewer_nFiM", "summary": "This paper addresses learning under label noise in supervised binary classification by introducing a risk-equivalence-based loss correction that ensures robustness when label flips depend only on the input instance. The method derives a corrected loss so that minimizing the empirical noisy risk aligns with minimizing the true clean risk for any classification-calibrated base loss (e.g., logistic).\n\nThe authors propose efficient ways to estimate the per-instance noise rate $\\eta_x$\t​without requiring matrix-valued transition models, extending CCN and ILDN approaches with lower complexity. Experiments on image, audio, and tabular datasets show strong robustness to moderate and high noise, matching or exceeding prior methods.\n\nOverall, the paper offers a principled and efficient framework for instance-dependent noise modeling. While conceptually incremental, it combines theoretical rigor and empirical breadth to provide a valuable step toward scalable, noise-tolerant learning.", "questions": "- Can the proposed framework be extended to multi-class classification tasks?\n- How sensitive is the method to the duration of the warm-up stage used for estimating ($P(Y|x)$?\n- How does the estimation of $\\eta_x$ influence performance when no label noise is present? Including results for a 0% noise setting (e.g., in Table 1) would clarify this.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Provides a rigorous derivation of a corrected loss guaranteeing risk equivalence between noisy and clean risks, with solid theoretical grounding (Propositions 1, 2; Lemma 1).\n- Introduces a stable regularized variant with Lipschitz and generalization-bound analysis.\n- Revives the under-studied **instance-only dependent noise (IDN)** model, offering a simpler yet effective alternative to ILDN.\n- Proposes an elegant, computationally efficient estimation of $ \\rho_x$ using $ |h(x)| $ as a difficulty proxy, avoiding complex multi-stage procedures.\n- Demonstrates machine-agnostic applicability across neural networks and gradient-boosted trees.\n- Achieves strong and stable performance across image, audio, and tabular datasets, outperforming or matching competitive baselines under moderate and high noise.", "weaknesses": "- Theoretical contributions mainly extend prior loss-correction frameworks (e.g., Natarajan et al., Patrini et al.), with novelty centered on the instance-only dependency rather than new statistical insights.\n- The warm-up assumption for accurate noise estimation is heuristic and may break under severe noise.\n- The formulation $\\rho_x = f(|h(x)|$ is empirically motivated without strong theoretical grounding.\n- Experiments are limited to binary classification; multi-class extension is left for future work.\n- Evaluation relies on relatively small or subsampled datasets, lacking results on large-scale real-world noise (e.g., WebVision).\n- The drastic accuracy drop below 50% in Figure 1 suggests potential tuning or early-stopping issues in baseline comparisons", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HA4EaJH8LB", "reviewer": "Reviewer_joei", "summary": "This paper investigates the label noise problem in binary classification. The authors revisit the **Instance-Only Dependent Noise (IDN)** model, a rarely explored framework where the label corruption depends solely on the instance features. The authors also propose an instance-aware loss correction method based on risk equivalence. At its core, the method models a per-instance noise rate $\\rho_x = P(Y \\neq \\tilde{Y} \\mid X=x)$ instead of a full transition matrix. Several ways for the estimation of transition rates are proposed. Experiments on diverse datasets (image, audio, and tabular) are conducted to verify the proposed methods.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces an unbiased risk that incorporates the per-instance noise rate $\\rho_x = P(Y \\neq \\tilde{Y} \\mid X = x)$.\n2. The authors conducted experiments on several datasets to verify the proposed method.", "weaknesses": "1. The proposed method is currently restricted to binary classification, and its extension to multi-class settings remains unexplored. This limits the applicability of the approach to broader real-world scenarios.\n2. The paper does not include experiments on large-scale real-world noisy datasets (e.g., WebVision or Clothing1M). Evaluating on such datasets would provide stronger evidence of the method's scalability and practical robustness.\n\n3. The estimation of the per-instance noise rate $\\rho_x$ relies on heuristic proxies (e.g., distance to the decision boundary). The paper does not provide an in-depth analysis of how the choice of proxy or mapping function affects performance. Different proxies may exhibit different levels of robustness.\n\n4. The paper lacks a clear discussion of its relationship with prior loss-correction methods, such as Natarajan et al. (2013). A more explicit theoretical analysis or empirical comparison would help clarify the novelty and contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "68WeOiiskr", "reviewer": "Reviewer_Hbdw", "summary": "This paper looks at label noise in binary classification. It suggests a new way to fix errors using an instance-aware loss correction scheme. This is based on the instance-only dependent (IDN) noise model. The main idea is to model the chance of a label being wrong as $\\rho_x = \\mathbb{P}(Y \\neq \\tilde{Y} | X = x)$. This depends only on the instance $X$, not the true label $Y$. This is different from the instance-label dependent noise model (ILDN). The authors create a new loss function $\\tilde{\\ell}$ that matches the risk of noisy data with clean data, as long as the base loss is classification-calibrated, like cross-entropy. They estimate the transition rates $\\rho_x$ using the distance to the decision boundary to show how hard an instance is. A regularized version $\\tilde{\\ell}_R$ is also introduced to avoid numerical problems, with theoretical support using Rademacher complexity bounds. Main contributions: 1. A loss correction method that works well with instance-dependent noise 2. Highlighting the efficient IDN model (one value per instance instead of a matrix in ILDN) 3. Different ways to estimate $\\rho_x$, like distance-based, clustering, and ensemble methods 4. Testing in different areas (image, audio, tabular) and with different learners (CNNs, MLPs, GBMs) The tests show that this method works as well as or better than 12 other methods on CIFAR-10, and 5 tabular datasets with moderate (28%) and high (44%) noise levels.", "questions": "1. (Section 3.2.1): - You mention that $h(x)$ models both $\\mathbb{P}(Y|x)$ and $\\mathbb{P}(\\tilde{Y}|x)$. How can one function represent both clean and noisy label distributions? - Can you give a theoretical or practical analysis of when this approximation works? - What if you use the separate model for the whole training, not just the warm-up? Appendix A.3 shows worse results, but the separate model used a fixed $\\mathbb{P}(\\tilde{Y}|x)$—what if both networks are updated together?\n\n\n\n2. Can you explain when $|h(x)|$ is a good way to measure distance to the decision boundary? - In neural networks with ReLU activations, the decision boundary can be made of straight-line segments. Does your method still work well in areas far from the training data? - Have you thought about using real distance calculations (like DeepFool) for some samples to check how good the approximation is during training?\n\n\n\n3. Why is the 4-epoch warm-up chosen? Does it affect performance? - Can you show results for 0, 2, 4, 8, and 16 warm-up epochs? - Is there a method to decide the warm-up length, like using the clean sample ratio or when $h$ converges?\n\n\n\n4. Your main experiments use ILDN noise injection. Can you also test with true IDN noise to check the model's assumptions? - When would IDN not be enough, and ILDN be needed? Can you describe the task or data features that decide this? - Tables 1-3 show PTD and BLTM (ILDN methods) sometimes do better than NDX. Does this mean ILDN is better for these cases?\n\n\n\n5. The conditions in Proposition 2 are hard to check in real situations. Can you give practical tips or simple methods for setting $\\lambda$ other than grid search? - How does the best $\\lambda$ depend on dataset features like size, noise, and dimension? - Have you seen cases where the denominator in Eq. 3 gets close to zero during training even with regularization?\n\n6. How does the method work with larger datasets like full ImageNet and more complex problems? - Can you explain how to extend it to multi-class classification? Is it as simple as one-vs-rest, or are there major challenges? - For the tabular LightGBM results, how does the method work with GBM's way of handling noisy data?\n\n7. Are there times when NDX does much worse than the usual methods? What are these situations like? - What happens to performance when the noise model is very wrong (for example, the real noise depends on the class, but you think it is IDN)?\n\n8. In Section 3.2.3, you show that $\\tilde{\\ell}(\\cdot, \\tilde{y})$ becomes $\\ell(\\cdot, -\\tilde{y})$ when $\\tilde{y} \\cdot h(x)$ goes to negative infinity. This means the model trusts itself more than the label. Could this cause the model to repeat its mistakes? How does this compare to methods that reduce the importance of doubtful samples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper connects theory and practice by creating a loss correction method (Proposition 1) and a practical version with guarantees (Proposition 2), even with some limits. \n\n\n\n2. Estimating one flip probability per instance instead of a full matrix is a big practical benefit. The paper argues well for when this simpler model works, challenging the trend of more complex ILDN methods. \n\n\n\n3. Using $|h(x)|$ as a stand-in for distance to decision boundary is smart and costs nothing extra (already done in forward pass). Though not fully explored theoretically, tests (like the matched-noise experiment in Fig. 1) show it works well. \n\n\n\n4. Applied to CNNs, MLPs, and gradient-boosted trees - Tested in different areas (vision, audio, tabular) - Consistently good performance (often top-2) across 10 datasets and different noise levels - Stability across datasets (noted in Section 4.2) is very useful for users \n\n\n\n5. Two noise levels (28%, 44%) for moderate and high noise - Three trials per setup - Compared with 12 baseline methods with different noise assumptions - Study (Appendix A.3) explains joint vs. separate probability estimation - Analysis (Section 3.2.3) gives insight into the changed loss behavior - Code (Appendix A.8) shows it's easy to use (~20 lines) - Hyperparameter analysis (Appendix A.2) shows it's robust", "weaknesses": "1. The paper talks about using the same function $h(x)$ to estimate both $\\mathbb{P}(Y|x)$ and $\\mathbb{P}(\\tilde{Y}|x)$ (see Section 3.2.1). This is based on warm-up epochs, not detailed analysis. When is this estimation accurate? How much does the performance depend on the length of the warm-up? The paper says 4 epochs but does not test this. - The idea that $|h(x)|$ shows the true distance to a boundary is not backed by theory. It works in practice, but we don't know when it is accurate. For very complex boundaries or early in training, it might not work.\n\n2. Proposition 2's conditions on $\\lambda$ depend on dataset-specific factors like $\\Delta_S$ and norms, which are hard to check. The paper uses grid search but doesn't explain how to choose search ranges or give any tips. - There is no advice on which $\\phi$ (probability transformation) or $z$ (difficulty metric) to use for new areas. The paper shows that distance-based + $\\beta$-logistic works well but doesn't explain when other methods (like clustering or autoencoders) might be better. - The warm-up period of 4 epochs seems random, with no explanation or analysis of its impact.\n\n\n\n3. The main experiments (Section 4.2) use ILDN noise injection, not IDN noise, as per Xia et al. 2020. This checks how well the model handles mismatches but doesn't prove that real-world noise is like IDN. - The matched-noise experiment (Section 4.1) is artificial and repetitive—noise is added and then removed using the same model. - There is no real-world data study to see if IDN is enough compared to ILDN. Are there areas where ignoring label dependence affects performance? - The statement that IDN is \"not less powerful than ILDN\" (Abstract) lacks strong support. Tables 1-3 show mixed results when compared to ILDN methods (PTD, BLTM).\n\n\n\n4. Authors recognize this issue, but it limits how useful the method is. There is no plan for extending it to multiple classes, even though it seems like a logical next step. - Small dataset size: Using parts of CIFAR-10 and small tables doesn't show if it works on large datasets. How does it handle big data like ImageNet or millions of examples? - Fairness issues with baseline methods - Some methods for multiple classes (GCE, DMI) might not work well - DivideMix was left out because it didn't perform well, but it's unclear why (was it a problem with how it was done or a limit of the method for two-class tasks?) - Using different optimizers for some methods (like Adam for Coteaching+) makes it hard to compare them.\n\n5. There is no explanation of when $\\mathbb{E}_X[\\rho_x] &lt; 0.5$ is true. There is no talk about how complex the calculations are or how long they take. There is little error analysis: what if $\\rho_x$ is estimated wrong? Sensitivity analysis is only done for hyperparameters $\\lambda, \\beta$, not for the noise model itself. There is no study of when the method fails or does worse than other methods.\n\n6. Equation 4 is hard to understand and needs more steps to explain it. The main difference (IDN vs ILDN) is not highlighted early enough. Some important details (like warm-up and approximations) are mentioned briefly without explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)