Enhancing Learning with Noisy Labels via Rockafellian Relaxation
Authors: Louis Chen
Bobbie Chern
Eric Eckstrand
Amogh Mahapatra
Johannes O. Royset
Venue: ICLR 2026 Poster (2026)
Keywords: Noisy Labels
Loss-reweighting
Neural Networks
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Labeling errors in datasets are common, arising in a variety of contexts, such as human labeling and weak labeling. Although neural networks (NNs) can tolerate modest amounts of these errors, their performance degrades substantially once the label error rate exceeds a certain threshold. We propose the Rockafellian Relaxation Method (RRM) -- an architecture-independent, loss reweighting approach to enhance the capacity of neural network methods to accommodate noisy labeled data. More precisely, it functions as a wrapper, modifying any methodology's training loss - particularly, the supervised component. Experiments indicate RRM can provide an increase to accuracy across classification tasks in computer vision and natural language processing (sentiment analysis). This observed potential for increase holds irrespective of dataset size, noise generation (synthetic/human), data domain, and adversarial perturbation.

Published as a conference paper at ICLR 2026
ENHANCING LEARNING WITH NOISY LABELS VIA ROCKAFELLIAN RELAXATION
Louis L Chen & Eric Eckstrand Department of Operations Research Naval Postgraduate School
Bobbie Chern
Meta Menlo Park, CA 94025, USA bgcherne @eeta com
Monterey, CA 93943, USA (louis. chen, eric. eckstrand @nps edu
Johannes Royset
Amogh Mahapatra
Department of Industrial and Systems Engineering University of Southern California Los Angeles, CA 90089, USA royset@usc.edu
lucky5devi 3e com
ABSTRACT
Labeling errors in datasets are common, arising in variety of contexts, such as human labeling and weak labeling. Although neural networks (NNs) can tolerate modest amounts of these errors, their performance degrades substantially once the label error rate exceeds a certain threshold. We propose the Rockafellian Relax- ation Method (RRM) an architecture-indped enhance the capacity of neural network methods to accommodate noisy labeled wrapper, modifying any methodology'
loss reweighting approach to
data. More precisely, it functions training loss particularly, the supervised component. Experiments indicate RRM provide increase accuracy across classification tasks in computer vision and natural language processing (sentiment analysis). This observed potential for increase holds irrespective of dataset size, noise generation (synthetic/human)) data domain, and adversarial perturbation.
INTRODUCTION
Labeling errors are systematic in practice, stemming from various sources. For example, the reliability of human-gent meeaa-geeeerted labels can negatively impacted by incomplete information, or the subjectivity of the labeling task as is commonly seen in medical contexts, in which experts can often disagree on matters such as the location electrocardiogram signal boundaries Frenay & Verleysen (2014), prostate tumor region delineation, and tumor grading (Nr et al. 2018). As well, labeling systems, such as Mechanical Turk Turk"oofen often find expert labelers being replaced with unreliable non-experts (Snow et al. 2008). The impact of these errors classification performance be severe for machine learning approaches (Zhu & Wu 2004), including, most notably, NNs (Krause et al., 2016 Zhang al. 2017).
In this paper, we propose a loss-reweighting methodology for the task of training a classifier on datasets with labeling errors. More precisely, we provide meta-algorithm that may "wrap" loss-minimization methodology to enhance performance. The method relates to optimistic and robust distributional optimization formulations aimed at addressing adversarial training (AT), which underscores our numerical experiments on NNs that suggest this method of training can provide test performance robust to high levels of labeling error, and some extent, feature perturbation as well. Overall, we tackle the prevalent challenge of label contamination in training data, which is critical obstacle for deploying robust machine learning models. The approach utilizes Rockafellian Relaxations (Royset et al. 2023) in addressing contaminated labels without the need for clean validation sets or sophisticated paprr-paraetters. common constraints current methodologics.
http://muukkccom

---

Published as a conference paper at ICLR 2026
This distinct capability represents key contribution, making approach more practical for handling industrial datasets.
We discuss related works and our contributions in Section 2 In Section 3 we discuss our methodology
and provide theoretical justifications. We conclude with numerical experiments/results in section
2 RELATED WORK
In loss adjustment methods, individual training example losses are typically adjusted multiple times throughout the training process prior to NN updates. These methods can be further grouped into loss correction, loss reweighting, label refurbishment, and meta-learning (Song et al., 2020). Our approach has some resemblance to loss reweighting methods wherein each training example assigned a unique weight, where smaller weights are assigned to examples that have likely been contaminated. For example, Ren et al. (2018); [hh et al. (2019) learn sample weights through the use a moise-free validation set. [Chang et al. (017) assign sample weights based prediction variances, and /Zhan let al.] (202T) examine the structural relationship among labels to assign sample weights. However, we view the need by these methods for clean dataset, or least one with sufficient class balance, as shortcoming, and our method, in contrast, makes no assumption on the availability of such a dataset.
Recent state-of-the-art methods explore training-time interventions based on sample selection Wang et al. (2023), training dynamics, and hybrid -up-evueeee methods. Xia et al. (2023) select
high-discrepancy samples based on disagreement between multiple models. Kim et al. (2021)
use feature-space neighborhood consistency to identify those likely to be informative/uncoruptee Bai et al. (202T) analyze early stopping noisy settings, arguing for patience- based mechanisms balancing memorization and underfitting. Nishi et al. (2021) use data augmentations for robustness and find that stronger augmentations, when paired with consistency objectives, significantly improve generalization in noisy-label regimes. ProMix (Xiao et al. 2023), DivideMix (Li et al. 2020) and CC (Zhao et al., 2022), treats noisy-label training as a semi- supervised learning problem, combining loss modeling with consistency regularization. |Wang et al. (2022) propose scalable penalized regression to detect noise using sparsity-awar optimization, offering a fully unsupervised and efficient alternative to traditional clean-set-dependent approaches. While many of these methods achieve tatt-off-he--ar, we show that some of their performances can he enhanced by using Rockafellian Relaxation during training.
There exist many other approaches to mitigate the effect of label noise (oogg et al., 2020: Frenay
& Verleysen, 2014), and can be categorized into: (1) robust architectures (Lee et al. 20 9 Han
et Goldberger & Ben- 2017 Sukhbaatar et al. 2015 Chen & Gupta, 2015);
robust regularization (Shorten & Khoshgoftaar 2019; Krogh Hertz] Srivastaa al.) 201 4
Toffe & Szegedy 2015); and (3) robust loss functions. Robust loss functions including robust mean absolute error (MAH) (Ghosh et al. 2017), early learning regularization (ELR) (Liu et al. 2020). and generalized cross entropy (GCE) (Zhang & Sabuncu 2018) are more robust than categorical cross entropy (CCE); however, our method is not dependent on a particular loss function, and it is possible that arbitrary loss functions, including robust MAE and ELR, can be wrapped by our method with ease.
3 METHODOLOGY
3.1 MISLABELING
Let X' denote a feature space, with y a corresponding label space. Then 2 X y will be a collection of feature-label pairs, with an unknown probability distribution D. Throughout the forthcoming discussions, ((.,,,))) y)) will denote sample of N feature-label pairs, for which some pairs will be mislabeled. More precisely, we begin with a collection (I;, (z) drawn i.i.d. from D. but there is some unknown set C C {1..... NN) denoting (contaminated) indices for which Y/ Ys if and only if & C. For indices i C, y: is some incorrect label the particular choice of noisc/contamination matters not. Indeed, we experiment with human human-generater generated, as well as uniform label noise in Section 4 We also consider non-uniform noise in Appendix Section D
2

---

Published as a conference paper at ICLR 2026
3.2 ROCKAFELLIAN RELAXATION METHOD (RRM)
Let there be given an NN architecture, for which when the parameter 0 (to be optimized/learne) is set, takes input any feature x and outputs prediction ý. Towards optimizing the choice of 0, prediction loss [0; xt y) is employed to evaluate the prediction ÿ with respect to More precisely, training typically (e.g., Li et al. (2020); Xiao al. (2023): Zhao et al. (2022)) involves assembling. from (noisily) labeled data, a loss (()) given by
1 N Σ
(())
J(0 vi) + +r0),,
where a regularization term r(0) is added to, effectively, an expected loss in which each member i is drawn with probability p. = 1/N. Without loss of generality, we will proceed by assuming that r(0) = 0; otherwise, any prediction loss J(0; z, y) could be redefined as J [0; y) + r(0).
However, we have noisily-labeled data, we might desire to remove those members that are affected; in other words, C & f1...., N is the of contaminated observations, then might desire instead to have p; = 0 when é E C, and p; = 1 those methods that optimize a loss C resembling equation | with the intention of aligning the pa values closer to these desired (but unknown) p. We call this procedure the Rockafellian Relaxation Method (RRM), for its use of a Rockafellian Relaxation (Royset et al. (023). More precisely, with ((N) C R denoting the standard probability simplex, and pd denoting the uniformly weighted (empirical) distribution {p, JEE[N] = (1/N), we consider the problem
N-IC otherwise. In this work, we consider modification of
min
(( (O; v)1 + + - (p .)) PEA(N)
min
(2)
+ uz) J(0 + 2 2 1|ull1.
Lanu (0)=
min WEU
N
where U = {u E R.V E E, 1p, -pll.
0,
0 Vi 1,...,,,,, r 0, and drv(w" .p)
In words, given a method that optimizes some training loss L. we can "wrap" this method with the RRM procedure by instead optimizing LRAM- In doing so, in addition to 9, we obtain via equation 2 an alternative distribution p that may now replace the empirical, at the cost of y per unit of total variation. Equivalently, any such alternative distribution is a loss-reweighting u E U; indeed, any " E U yields a nd E A(N) via p. = 1/N + uz so that 2 ||u]]] is the total variation distance between the empirical distribution and p".
3.3 ANALYSIS AND INTERPRETATION OF ROCKAFELLIAN RELAXATION
Although problem 2 is nonconvex in general, it is amenable to a block coordinate descent approach that iteratively cycles between updating and u. Towards this, we hegin by demonstrating that the computation of (RRM(() for any fixed 0, i.e., the minimiiiiii (over E U)) in equation 2 amounts to a tractable linear program. The following result characterizes the complete set of solutions to this linear program, and in doing so, provides an interpretation of the role that y plays in the loss-reweighting action of RRM.
Theorem 3.1. Let ? > O and {c= J(H; vi, (4, W:) for some 6, with Cminn = min; Cr and Cmaz max; Ci. Write Imin := ti lbi = (i: Gi = Cmin + 7), and for any S₁ C Imin, S2 C lbig define the polytope
Cmin), Imid ;= ii E (Cmins Cmin + })}.
u² E U
US = u² ut 20 Vi I
" =0
0 vi S1 U Tinid: ut 5 Vi E Tbig
ui
1 Vi Sz,
" 1
Vi G > Cmin + 7%
N
Then
N
arg min wEU
+ u.) :4 + 2 ||ll1 = conv
U US S1.S2
(3)
N
2
1=1
3

---

Published as a conference paper at ICLR 2026
The theorem explains that the construction of any optimal solution u² essentially reduces to cat- egorizing each of the losses among {c, = J(0; 24, 34) ]= as "small" or "big", according to their position in the partitioning of [Cmin, 00) = [Cmin: Cynán + +) U [Cmin + 20). For losses that occur at the break points of Gmin and Cmin + 7, this classification can be arbitrary hence, the use of Si and S2 set configurations to capture this degree of freedom. In particular, those points with losses oa exceeding Cmán + 7 are down-weighted to zero and effectively removed from the dataset. And in the event that Cmaz
Cmin < 7, no loss reweighting occurs.
Although Theorem 3.1 establishes that there are a plethora of valid loss-reweightings, one particular solution is intuitive, simple (efficient) to execute, and offers tunable control over the reweighting. is obtained by setting S = S2 = 0 in Theorem 3.1
Corollary 3.1.1. Let > 0 and 0 be given, yielding losses {c== = J(; 24, y:) Define x(0) == (i: C. Cmin +7} and let u" (0) given by {u, (O) =I
[x100 (/ni J. {uī (0) ( = {0}.
and (0) N for all i E x()). Then u² (f) solves the linear program Lanas equiv, the inner minimization of equation
In words, given any 0, there is always an optimal reweighting (for problem CRRA (0)) in which [x(0) - many samples are removed from training; specifically, transfers [x(2) probability mass away from the group of highest-cost examples and uniformly re-distributes among Imin- Further, as crucially determines x(), we see that by tuning 7 we can control the fraction of samples that are pruned, method that we expand upon in the forthcoming Section 3.5.1 and for which executed
2v
in many experiments of Section
If over the course of an iterative algorithmic scheme, e.g. Algorithm I set, then this set is effectively removed even if the training of & might proceed. The experiments of Section 4.2.2
(0) converges to some
(see Table display such convergent behavior.
3.4 RRM AND OPTIMISTIC WASSERSTEIN DISTRIBUTIONALLY ROBUST OPTIMIZATION
In this section, we discuss RRM's relation to distributionally robust and optimistic optimization formulations. Indeed, (2) 's formulation
min-min problem bears resemblance to optimistic formu- lations of recent works, c.g., Nguyen et al. (2019). We will see as well that the minimization in u, as considered in Theorem 3.1 relates an approximation of data-driven Wasserstein Distributionally
Robust Optimization (DRO) formulation (Staib & Jegelka 2017).
3.4.1 LOSS-REWEIGHTING VIA
WASSERSTEIN FORMULATION
For this discussion, as it relates to reweighting, we will lift the feature-label space Z = X' x V. More precisely, we let W : R2, denote a space of weights. Next, we say W x Z has an unknown probability distribution 'D such that n DD = D and Dw()((1) 1. In words, all possible (w.r.t. D) feature-label pairs have a weight of 1. Finally, we define an auxiliary loss E W x 2 x O (w, :- w [xx y, 0), for any
(I, E Z.
Given sample {(1, us) ) just as Section 3.2 we can opt not to take as granted the resulting empirical distribution Dx because of the possibility that |Cl- have incorrect labels (i.e., + ús) Instead, we will admit alternative distributions obtained by shifting the Dx 's probability mass off "contaminated" tuples (1, vee to possibly (0,z,, z., v1). (1, V.). even some other tuple
(I, 21, M) with j & C for example equivalently, climinating, correcting, or replacing the sample, respectively. In order to admit such favorable corrections to DN, we can consider the optimistic (Nguyen et al. 2019 Staib & Jegelka 2017) data-driven problem
min UN ( = min Ep [[ 2;]]
(4)
(Pw.
in which for each parameter tuning 0, uN (0) measures the expected auxiliary loss with respect to the most favorable distribution within an c- e-rescribed W1 (1- W-sssersttin) distance of Dx. It turns out
that a budgeted deviation of the weights alone (and not the feature-labe pairs) can approximate (up to an error diminishing in N) (y) More precisely, we derive the following approximation along similar lines to Staib & Jegelka 2017).
4

---

Published as a conference paper at ICLR 2026
Proposition 3.1. Let e O, and suppose for any 8, (ax(r 1J < co. Then there exists R 2 such that for any 8, the following problem
1
MIX M1
=
min
uz) J(0; Fz, (1, + r0llu|1
th1 M.N
s.t. U
N 20
M (0) uN (0). In particular, -20 S min; J(0; 31, yı), and, for any u² solving U MTX (0), holds that u. for such that J(:: z4. y1) > 70--
satisfies un (0) N UN
In summary, while the optimistic Wasserstein formulation would permit correction to Dx with combination of reweighting and/or feature-label revision, the above indicates that process focused on reweighting alone could accomplish reasonable approximation; further, upon comparison to (2) we see that RRM a constrained version of this approximating problem, that is,
Caass (0) Z UN
MIX (0) Z (0).
Hence, RRM is optimistic but not as much as the data Gata-driven Wasserstein approach.
3.5 A- RRM/RRR ALGORITHM
Towards solving problem (2) the two decisions 0 and u, we proceed iteratively with a block- coordinate descent heuristic outlined in Algorithm [] whereby we update the two separately in cyclical fashion. In other words, we update while holding u fixed, and we update u whilst holding 8 fixed. The update of 9 is an SGD step batch of s-many samples. The update of u is linear program, but can also be executed instantaneously via Corollary 3.1.1
Algorithm 1 (Adversarial) Rockafellian Relaxation Algorithm (A-RRM/RRM)
Require: Loss Function J(0; z, y), training perturbation e [o. 1]. Number of epochs a, Batch size 2 1, learning rate 17 > 0, threshold y > 0, reweighting step μ E (0, 1). (optional) contamination estimate C" E [0, 1].
<--00 5-0c N
Iter +
repeat
Iter + Iter +1
f t Gradient Grwdiiettteesss
if C" provided then
f + max{f: [0:10 2 C²}
7 l1-0 min, J v0: / 1- end if u - Re wee-weiiht(yy (u) until Desired Validation Accuracy Loss, Iter is sufficiently large Return 8
Apart from the learning rate y that is (industry) standard in gradient- based algorithms, Algorithm can be parameter-less; more precisely, the step-size μ and threshold 7 parameters can in fact be auto-tuned, in a manner that follows from Corollary [3..1] so as to precisely control how many examples pruned in each re-weighting step, might be desired guided by a contamination estimate C".
3.5.1
AUTO-TUNING PRECISE SAMPLE PRUNING WITH A CONTAMINATION ESTIMATE C
As outlined in Algorithm I an estimate C' of the true contamination C is on hand, then by setting 7 f1-C min; (0; I,, y:), where c- is approximately the (1 1 - th quantile of the costs,
'A minimizer can he obtained via linear program solver or explicitly via the solution of Corollary 3.1.1
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 Craddienttttepss0...:/
Require: E 8 Σ21, U, 0,c [0, 11.0
for e = 1...., do
for = 1, ,....
do {<x), (/)ii draw batch from {(z,, y:) yi)
for i l...,, do
x <- 1i + sign (V [0; (x).)())
end for
0 <- 1) Eisi ( + u.) VoJ (1) 9)
end for end for Return 0
Algorithm 3 Re-weight(2;.41
Require: threshold y > 0, re-weighting step (0, 1). u² min Σ
N ( + u.) J(0; æ4, 91) >||u||1 11)
Return u pm"
or equivalently, max{](() s. y6) (i:(0aa) (: ((:z..)))
C²}, then the fraction of observations
(B: )2-.)))24-mmmm w(s)l that the reweighting of Corollary 3.1.1 "prunes" is at least
experiments implement this form of RRM, and we examine how RRM enhancement scales with the quality of the estimate C" in Appendix Section experiments.
Sections 4.1 4.2.
3.5.2 RRM 0) AND A-RRM (E>0) 0)
We will refer to Algorithm | as RRM or A-RRM, when e = or e > 0, respectively, The only difference lies in the execution of (Algorithm s) Gradient
When e = 0, GradientSteps simply executes a iterations of SGD. But e > O, then StadientSteps additionally executes Fast Gradient Sign Method (FGSM) (Goodfellow et al.) 2015) adversarial attack, whereby each time a batch (x!, v²') is drawn, every x is perturbed via I + sign (0, .rf, ))) We note that other adversarial perturbation methods can substituted in place of this choice of attack. Then Stochastic gradient descent (SGD) step is taken.
3.5.3 ON COMPLEXITY
Each iteration of Algorithm T comprised of two different tasks: 1.) Gradient.Stess and 2.) Re-weight Task I's gradient step calculations are entirely standard practice. Task amounts to solving the polynomial-sized (in training data) linear program equation 3 for which general theoretical efficiency is well-established and either commercial solver (e.g. CPLEX 12.8.0) would suffice, or, even quicker, the solution of Corollary 3.I.1 could be derived with single pass over the list of losses (see experiments of Sections 4.1
4.2.1 We especially note that Re-weight is only executed after all the σ- many epochs of maddinttterp have concluded. In summary, the scaling of computation with larger datasets is not limiting in nature. We refer the reader Section 4.2.T and Appendix Section [oor experiments on MNIST-3 and CIFAR-10 with average computation times of Algorithm (ke-weighh) reported.
4 EXPERIMENTS AND RESULTS
4.1 RRM ENIIACCEMENT IN REAL-WORLD NOISE EXPERIMENTS
We experimented on the CIFAR-N Wei et al. 2022), Clothing 1M (iao et al. 2015). and Food- 101N (Lee et al. 2018) datasets which all exhibit real-world noise. Specifically, for each dataset, identified a handful of top performing methods to wrap, and then compared the resulting performance against the leaderboard performers for that dataset.

---

Published as a conference paper at ICLR 2026
CIFAR- 10N/100N furnishes CIFAR- 10/100 with human-annotin Mechanical Turk, and the top two performers for this dataset are ProMix and Divide-Mix. Clothing IM is a dataset composed of 1 million images scraped from online shopping websites, with labels derived from accompanying text, and the top two performers for this dataset are L.A--difusion and CC. Food-1011 consists of 310009 noisily-labeled training images collected from the web, and the top two performers for this dataset are ARA--iffusion and SURE.
noisy labels from Amazon
We chose wrap ProMix, Divide-Mix, CC, and CCE to compare against the three leaderboards.
ProMix, Divide-Mix, and CC all exhibit a training loss of the form L()) = (0) + r(0), where r()) == (0) Lauz(0). (0). In words, the loss combines supervised component Lx, (Au- weighted) superviiee component xu Cu, and some additional auxiliary loss Lauæ. In particular, these methods use some procedure select subset X of labeled data for which formulate an average loss of the form Lx (x()) |x] Σvex J(0; 2i, Vi) for some classification loss J. While these methods endeavor for X to be clean of noisily -labeled data, this cannot be guaranteed in practice.
In the wrapping of these methods by RRM. we replace the training loss & with equation 2 CRRM- We do similarly for CCE with r()) 0. More precisely, after every gradient steps (see Algorithm 2 for 0, we fix the current I and execute a reweighting by solving for u (see Algorithm 3 For 10. For CCE, Section 3.5.1Js ss use of conservative estimate C" of label error rate 0.5, 0.5, 0.4, and 0.2 for datasets CIFAR- ION, LIFAR-IOON, Clothing and Food respectively.
ProMix. Divide-Mix, and CC,
1. As for % this was set by employing
As for any hyperparameter of ProMix, Divide-Mix, or CC, in our experiments we simply re-used those reported in their respective papers' (iiao al. 2023 Li et al. 2020 Zhao et al., 2022 experimental
setup.
Table lists the top published accuracies for comparison, with asterisks marking state-of-the-art. On CIFAR-1OON, we achieved state-of-the-art accuracy with both the wrappings of ProMix and Divide-Mix. On CIFAR-ION, wrapping enhanced Mride-MMx while ProMix's accuracy was virtually unaffected.
CIFAR-N Results:
Table CIFAR-N Test Accuracies.
MethodDattaet ProMix (+ RRM)
CIFAR-ION (Worst)
CIFAR- 10ON (Noisy Fine)
96.34% (96.32)
73.79 (74.19%)
Divide-Mix (+ RRM)
92.56 + 0.42 (94.75** + 0.08)
71.13 + 0.48 (73.98** 0.05)
SOP
93.24 + 0.21
67.87 + 0.23
ELR+
91.09 +'2 1.60
66.72 + 0.07
To-Teaching+
83.26 + 0.17
57.88 + 0.24
GCE
80.66 + 0.35
56.73 = 0.30
ClothingIM and Food-101N Results: Tables (ibles[[an]] and list the top published accuracies by methods on ClothingIM IM and Food- 1OIN respectively for comparison, with asterisks marking state-of-the-arr. On Clothing M, enhanced top-performer in CC to perform virtually well the best performing LRA-diffusion method. We also examined to what extent conventional training with CCE loss, a common baseline, could be enhanced by RRM accommodate noisy-labeling. Along these lines, upon wrapping CCE, we found that it now outperformed some noisy label methods (JoCoR and GCE) on Clothing M. On Food- 10IN, we achieved a top accuracy by wrapping the traditional CCE method.
In Tables 2 and wherever confidence intervals are omitted, this is because they are in fact not reported by the authors of that method.
4.2 RRM ENHANCEMENT ACROSS LEVELS OF LABEL NOISE AND ADVERSARIAL PETURBATION
Unlike the previous section's 's comparisons with state-of-the-ar methods, here the focus is how RRM enhancement of various methods scales with C' through synthetically generated label noise experiments CIFAR-10 (Alex 2009) and MNIST-10 (LeCun & Cortes 2010). We refer to
7

---

Published as a conference paper at ICLR 2026
Table 2: Clothing IM Test Accuracies.
Table 3: Food-101N Test Accuracies.
Method\Dataset
ClothingIM
CC (+ RRM)
75,4 (75.69**) 68.94 (71.48 + 0.25)
Mataset
Food-101N
CCE (+ RRM)
CCE (+RRM)
81.67 (84.21)
LRA-diffusion
75.7*
LRA-diffusion
93.42*
ELR+
74.81
SURE
88.0**
DivideMix
74.76
LongReMix
87.39
Knockoffs-SPPe
75.2
CleanNet
83,95
Meta- Beta-Weight-Net
73.72
Appendix (Section for further experimentation on Toxic Comments, IMDb and Tissue Necrosis datasets.
4.2.1 SYNTHETIC EXPERIMENTS WITH ONLY LABEL NOISE
Implementation Details: We employ symmetric noise at rate of C. 50000 CIFAR-10 training labels. The 10000 testing labels are unperturbed. A ResNet-34 architecture is utilized as in Zhang Saauuncu (0188) along with their data preprocessing and augmentation scheme. Two sets of experiments are performed. As a baseline, 180 epochs are performed for CCE, MAE and MSE loss functions without RRM wrapping. Training label contamination settings of 10%, 20%, and 30% are utilized. In the other set of experiments, CCE, MAE and MSE loss functions are wrapped with RRM and executed iterations with gradient descent (SGD) with learning rate (7) of 10.0, momentum of 0.9 with Nesterov momentum enabled, and a weight decay of 10-5 are employed for both sets experiments. Each experiment executed 5 times and the average test set accuracy and standard deviation is reported in Table 4 with the RRM results placed in parentheses.
epochs per iteration, also for a total of 180 epochs. Stochastic
Analysis: As Table indicates, both CCE and MSE methods greatly benefit from RRM approach across contamination levels, whereas the benefit of RRM for MAE is less definitive. The additional computation time incurred from the Re-we i ght((%, (b) step is average of 3.88 seconds per execution of every iteration, for a total of 46.56 seconds.
Table 4: CIFAR-10 Test accuracy (%) comparison between CCE, MAE, and MSE loss functions across contamination levels. RRM-wrapped accuracy results in parentheses. Values marked with (t) are from Zhang & Sabuncu 2018). Values marked with (*) are the best.
Contamination
Method
10%
20%
30%
CCE (+RRM)
89.94 + 0.29 (92.20 0.26)
86,98 + 0.44 (90.44 + 0.31)
81,90 + 0,86 (88 49 + 0,33)
MAE (+RRM)
92.04 0.25
(90. 3.56)
83.72 + 3.84 (87.82 + 4.07)
88.28
3.52 (82.98 + 6.55)
MSE (+RRM)
91.83 0.25 (93.04 0.17)
89.43 0.37 (91.3* 0.32)
86.34
0.22 (89,95* 0.55)
We refer the reader to Appendix Section C for experiments with higher levels of contamination.
4.2.2 SYNTHETIC EXPERIMENTS WITH BOTH LABEL NOISE AND ADVERSARIAL PERTURBATION
In the following experiment we evaluate RRM in a setting of both adversarial feature perturbation and label contamination.
Adversarial Training (AT) Baseline: As baseline, we execute standard adversarial training (AT) approach, in which execute A-RRM, omitting the Re-we w-ihtttht step. Indeed, it is common to perform training on data augmented with adversarial examples crafted through FGSM (Madry et al. 2018). Further, using this benchmark, may highlight the value of re-weighting.
Test-set Perturbation Etest: In the experiments, upon conclusion of the A-RRM training for which trained 6* is obtained, x Etest. v.] : x. y). .). In this way, we explore the harm to performance that misalignment between training perturbation and true test-set perturbation étest could have.
executed FGSM attack each member (z, y) of the test set via
8

---

Published as a conference paper at ICLR 2026
Implementation Details: We employ symmetric noise at rate of C on 60000 MNIST-I0 10 training labels. The 10000 testing labels are unperturbed. Twenty percent of the training data is set aside for validation purposes. We adopt basic CNN architecture with a few convolutional layers. The first layer has a depth of 32, and the next two layers have a depth of 64. Each convolutional layer employs kernel of size three and the ReLU activation function followed by max-pooling layer employing a kernal of size 2. The last convolutional layer is connected to a classification head consisting of 100-unit dense layer with ReLU activation, followed by a 10-unit dense layer with softmax activation. Categorical cross-entropy is employed for the loss function. Using Tensorflow 2.10 10 (Abadi et al.) 2015), 50 iterations of A-RRM are executed with o 10 epochs per iteration for a total of 500 epochs for given hyperparameter setting. For A- the hyperparameter settings of μ and 7 at 0.5 and 2.0, respectively, are based on search to optimize validation set accuracy. Separately, we perform comparable 500 epochs using AT only. Both AT and A-RRM employ stochastic gradient descent (SGD) with a learning rate (7) of 0.1. An E = 1.0 is used for all training image perturbations.
Performance Analysis: For each of the 0%, 5%, 10%, 20%, and 30% training label contamination levels, we compare adversarial training (AT) and A-RRM performance under various regimes of test set perturbation (Crest E {0.0, 0.1, 0.25, 0.5, 1.0)). In Table 5 we show the test set accuracy achieved when validation set accuracy peaks. We can see that training with an c = 1.0 and testing with lower étest levels of 0.00, 0010, and 0.25, results in drastic degradation in accuracy for AT for contamination levels greater than 0%. This performance collapse is not observed when using A-RRM. Given that the employed during training may not match test/production environment (Etest), our findings suggest that A-RRM can confer a greater benefit than AT in these scenarios,
Convergence of u: Table 6 tracks the evolution of the u, v,lvel across 49 iterations of Algorithm for 9600 contaminated and 38400 clean MNIST training samples at 20% contamination. After the first iteration, nearly all u, 0. By iteration 10, many contaminated samples already exhibit negative valuue, while most clean samples remain near zero. By iteration 49, 9286 of the 9600 contaminated samples fall in (-2.08, 1156 10-5 effectively canceling their nominal probability 1/N = 2.08- 10-5 so they are pruned from training. In contrast, most clean samples (35246 of 38400) retain their nominal probability 1/N = 2.08 10 55-5, This selective downweighting explains A-RRM's advantage over AT: A-RRM suppresses contaminated data in situ, critical property when test-time perturbations are unknown or milder than those used during training. Interestingly, some u,- values presented "close-calls", i.e., were such that N + u, R 0 at some iteration before then recovering by the conelusion of training to have a positive sample weight 1 + u 2 1 suggesting RRM's 's ability to recover borderline but ultimately learnable sample that was assigned to near-zero weight early. We refer the reader to Appendix F for details.
2N
Table 5: MNIST-10 Test accuracy (%) for AT and A-RRM under different levels of contamination C, training perturbation 1.0, and test-set adversarial perturbation étstt
C
0%
5%
10%
20%
30%
AT
A-RRM
AT
A-RRM
AT
A-RRM
AT
A-RRM
AT
A-RRM
0.00
97
96
63
95
57
97
58
96
26
86
0.10
95
93
64
92
71
94
61
93
20
82
0.25
93
90
83
91
88
92
84
90
74
81
0.50
91
88
94
94
90
90
88
97
80
1.00
83
95
90
94
86
88
98
5
CONCLUSION
RRM is a novel loss reweighting procedure that enhances learning with noisy labels, as our ex- periments demonstrate across datasets and noise schemes. In fact, we achieve state-of-the-art performance on CIFAR-100N when wrapping ProMix and DivideMix. Further, our experiments indi- cate improvements that persist across model architectures and applications. The method is virtually hyperparameter-fiie free, upon using auto-tuning feature to control the Re-wei ght pruning action with an estimate of label contamination. Our experiments further show that RRM boosts the test accuracy of both standard and noise-robust methods, and that A-RRM shows promise in extending

---

Published as a conference paper at ICLR 2026
Table 6: MNIST-10 Evolution of across |C| 9600 contaminated data points and -- 38400
clean data points. Note that I/N = 2.08 10 -
1. iteration
iteration
49.
iteration
C
N
C
N
C
C
[V]
C
>> 0
0
0
25
Ms value (10
R2 0
8844
38385
2058
37524
91
35246
(-0.52, 0.00)
0
36
146
1655
(-1.04, -0.52]
0
(-1.56, -1.04]
756
14
415
174
34
168
(-2.08, 11.56]
0
7079
617
9286
1131
these gains settings with combined adversarial perturbations and label noise, yielding models more resilient to both. For more experiments showcasing enhancement, please see the Appendix.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: //tp://wwwwww/.... wewworensort low, org/ Software available from tensorflow.org.
Krizhevsky Alex. Learning multiple layers of features from tiny images. https://www. cs. toronto. Saaulkri/eaamiinng-eetuu-----
pdf. 2009.
Mohamed Amgad, Habiba Elfandy, Hagar Hussein, Lamees A Atteya, Mai A T Elsebaie, Lamia S Abo Elnasr, Rokia Sakr, Hazem S Salem, Ahmed Ismail, Anas M Saad, Joumana Ahmed, Maha A T Elsebaie, Mustafijur Rahman, Inas A Ruhban, Nada M Elgazar, Yahya Alagha, Mo- hamed H Osman, Ahmed M Alhusseiny, Mariam M Khalaf, Abo-Alela Younes, Ali Abdulkarim, Duaa M Younes, Ahmed M Gadallah, Ahmad M Elkashash, Salma Y Fala, Basma M Zaki, Jonathan Beezley, Deepak R Chittajallu, David Manthey, David Gutman, and Lee A D Cooper. Structured crowdsurriing enables convolutional segmentation of histology images. Bioinformal- ics, 13x33511):33461344777 02 2019. ISSN 1367-4803. doi: 10. 100109//iiinfommacccs/rr..:..: URL htt.://gw...........ooooooaaaaa.
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. ciates, Inc., 2021. URL U/LLhttpes//ppoocedins...eetteee newrpps.........sree........... ccppaper fi files/paper Paper.r.
Curran Asso-
Haw-Shiuan Chang, Erik Learned-Mill and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30, 2017.
Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In Proceed- ings of the IEEE international conference on computer vision, pp. 1431-1439, 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1...1865/1111111111 1423. URL
orq/ N1 Brq/N19-1424 1423//
Benoit Frenay and Michel Verleysen. Classification in the presence of label noise: A survey. JEEE Transactions on Neural Networks and Learning Systems, 25(5):845-869, 2014. doi: 10.1 WUNLS.2013.22991
Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 48(2):603 655, 2023. doi: 12.. h0tp://mrrr.2222...22.222 wrg/ /0. 128. 1020////or..000 2022.
1275. URL httthhhtss:///t
Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the Thirty-Fire AAAI Conference on Artificial Intelligence, AAAI 17, pp. 1925. AAAI Press, 2017.
Jacob Goldberger and Ehud Ben- Ben-Reuven. Training deep neural- networks using a noise adaptation layer. In International conference learning representations, 2017.
11

---

Published as a conference paper at ICLR 2026
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015. Conference Track Proceedings, 2015. URL
/11 6572
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. Advances in neural information processing systems, 31, 2018.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: http::s///ppnzeveew..ro forrm?
[id=nZeVKeeFYf9
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Taehyeon Kim, Jongwoo Ko, Sangwook Cho, JinHwan Choi, and Se- Young Yun. FINE samples for learning with noisy labels. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htttpa://operrwweeww net/fooum: dir.ummmddd22px2200554
Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei. The unreasonable effectiveness of noisy data for fine-grained recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pp. 301-320. Springer, 2016.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http::/yaan
lecun com/ eetp///bbb/mm//////////////////t
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International conference on machine learning, pp. 3763-3772, PMLR, 2019.
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Junnan Li, Richard Socher, and Steven Hoi. Dividemix: Learning with noisy labels as semi-supervised learning, 02 2020.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-larniing regularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331-20344 2020.
Andrew I.. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http!
// wwww.aclwwed.. www.wwwwwww.............e.eeeaae wttp:///aaaooooooo/ooo------1-1.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver: BC, Canada, April 30 May 3. 2018. Conference net forum?icic
Track Proceedings. OpenReview.net, 2018. URL
FJZIBfZAb
12

---

Published as a conference paper at ICLR 2026
Viet Anh Nguyen, Soroosh Shafieezadeh Abadeh, Man-Chung Yue, Daniel Kuhn, and Wolfram Wiesemann. Optimistic distributionally robust optimization for nonparametric likelihood ap- proximation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https httptttss://prooeeeeeeeeeeeee wwwr ccpppper.pppp.p..ppppee files/paper)
Guy Nir, Soheil Hor, Davood Karimi, Ladan Fazli, Brian F. Skinnider, Peyman Tavassoli, Dmitry Turbin, Carlos F. Villamil, Gang Wang, R. Storey Wilson, Kenneth A. Iczkowski, M. Scott Lucia, Peter C. Black, Purang Abolmaesumi, S. Larry Goldenberg, and Septimiu E. Salcudean. Automatic grading of prostate cancer in digitized histopathology images: Learning from multiple experts. Medical Image Analysis, 10:1671180, 2018. ISSN 1361-8415. doi: hutp:://dii.or//00.0000
j.media.2018 09. 005. URL TRRLhttps:/ wwww//wwwoo...../...w/ www.ssciecediireeeoo com wtt.ccooccc..i.. htmm//cc/eciicclll/
Kento Nishi, Yi Ding, Alex Rich, and Tobias Höllerer. Augmentation strategies for learning with noisy labels. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 8022-8031. Computer Vision Foundation / IEEE, 2021. doi: thecch com/ comm/ttnttt// 111322221/tmm//iis//tts Augmentation Strategies for Learning With Noisy Loisy_Labels_o Labels_aeaa__ CVPR 2021 paper .html
10.1 10.1109/CVPR464377.00 220MM/PRR4437.2211...... 100993. URL https: htttps::/ppeaacceeetttte//e///
Dominika Petríková and Ivan Cimrák. Survey of recent deep neural networks with strong annotated
supervision in histopathology. Computation, 11(4), 2023. ISSN 2079-3197. doi: 10.3390/
computation| 1040081. RRLhttps:// betep::/wweed /wwp mdpi. wwm.coo//////// 2ttm//279-222 3197/11////1
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, 2018.
Johannes O. Royset and Roger J-B Wets. An Optimization Primer. Springer, 2021.
Johannes O. Royset, Louis L. Chen, and Eric Eckstrand. Rockafellian relaxation and stochastic optimization under perturbations. Mathematics of Operations Research (to appear), 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL hRRl/tttpsss//rxxv..cco/////// 01108
Connor Shorten and Taghi M Khoshgoftaar. survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.
Rion Snow, Brendan O' C'Conor, Daniel Jurafsky, and Andrew Ng. Cheap and fast - but is it good? evaluating non con-expert annotations for natural language tasks. Mirella Lapata and Hwee Tou Ng (eds.), Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. pp.
Honolulu, Hawaii, October 2008 Association for Computational Linguistics. URL https: /tttps://aclaiiii antho wow.oooogg...................... wwtp:oggg/pD000000
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. CoRR, abs/2007.08 007770819999 199, 2020. URL
Abbs2207.0811
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1988 2014.
Matthew Staib and Stefanie Jegelka. Distributionally robust deep learning generalization of adversarial training. In NIPS workshop on Machine Learning and Computer Security, volume 3, pp. 4, 2017.
13

---

Published as a conference paper at ICLR 2026
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Train- ing convolutional networks with noisy labels. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Yikai Wang, Xinwei Sun, and Yanwei Fu. Scalable penalized regression for noise detection in learning with noisy labels. pp. 346-355, 06 2022. doi: 10.1 10.1009/C2
Yikai Wang, Yanwei Fu, and Xinwei Sun. Knockoffs-spr: Clean sample selection in learning with noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3242-322 2023.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference Learning Representations, 2022.
Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, and Tongliang Liu. Combating noisy labels with sample selection by mining high-discrepancy examples. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1833-1843, 2023. doi: 10.1 10.11000CCC3111000223..01111
Ruixuan Xiao, Yiwen Dong, Haobo Wang, Lei Feng, Runze Wu, Gang Chen, and Junbo Zhao. Promix: combating label noise via maximizing clean sample utility. In Proceedings of the Thirty-Second International Joint Conference Artificial Intelligence, ICAI 22, 2023. ISBN 978-1-956792-03- 2023/494]
4. doi: 00096///jcai.2223//3333
URL doi org 211.:/.....999/1
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6268. Curran Asso- ciates, Inc., 2020. URLAttoss www.:eedd...........e......... papper files/ Lepp//pprr/////////p pdf
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL htttts:: wtwp:errreieeeeeeeeooouuummmma
net Eorwmmidddiioooo.... Sy
Hai Yang Zhang, XiMing Xing, and Liang Liu. Dualgraph: A graph-based method for reasoning about label noise. In 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9649-9658, 2021. doi: 10.1109/CVPR44447.2000
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 8792-8802, Red Hook, NY, USA, 2018. Curran Associates Inc.
Ganlong Zhao, Guanbin Li, Yipeng Qin, Feng Liu, and Yizhou Yu. Centrality and consistency: Two-stage clean samples identification for learning with instance-dependent noisy labels. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 21-37, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031--1886699
Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study, Artificial intelligence review, 22:177-210, 2004.
14

---

Published as a conference paper at ICLR 2026
A
APPENDIA/SUPPLEEENTTA MATERIAL
A.1 SECTION 3 PROOFS
Theorem 3.1. Let n > 0 and {or J(;; 24 Bi)l for some 0. with Cmin == ming C₁ and
Cmaz = max; Ci. Write Imin == li = Cmin h. Lmid == {i : G. E (Cmins Cmin + 7)}.
Tbig = {i: G. Cmin + 7h, and for any S1 S min: S2 C Ibig define the polytope
E EU
U $1
ut 20 Vi min,
" = 0 Vi E S1 UTiid un S0 Vi Thig
N
vi E S2, u N
Vi Q
> Cmin 7%
Then
arg min WEU
+ u.) a + 2 [1llll = conv N
UUS US 81.82
(3)
Proof. For any set C, let (c()) and (c(x) oo otherwise. We recognize that w a solution of the minimization problem if and only if is a minimizer of the function h given by
h(u) = Σ (c/N + usc + Iu + 40,00) (1/N u:)) 400) 1=1
Thus, because h(u) -00 for all RN and h is convex, u² is solution of the minimization problem if and only i E Oh(u*). We proceed by characterizing dh.
Consider the univariate function h; given by
h,(ws)= = c/N + u.e + |usl (10.co + u;).
For u, 2 11/N, the Rockaal-o sum rule reveals that
(2)
if u, > 0
2.
u, = 0
ah,(u:) = 4 +
if - 1/N < < 0
00,
3]
if 1/N.
For (U1,..., E [-1/N, N. we obtain by Proposition 4.63 in Royset & Wets (2021) that
ΣΣ h4 (u) 0hi (ui) x ... @hx(uN).
Let ho be the function given by ho(u) fop (E u.). Again invoking the Moreau-Rockafellar sum rule while recognizing that the interior of the domain of E h4 intersects with the domain of hg, we obtain
@h(u) = 0
Σ h4 (1) + aho(u) (1)( Ohx(uN)
(1)
R
for any u = (uI...., uN) with u, 2 11/N 1...., optimal if and only if for some X R.
......, N. and ΣN Us = 0. Hence, u" E U is
(c: + }}
if u > 0 if of
( 3. +]
AE
(5)
if u E (-1/N,0) (-00, 3] if o = 1/N.
{4 - 3}
follows that any such must then satisfy < Cmin + 3. as the above list of cases indicate for arbitrary i and matter the value of ut, for any optimal U.
15

---

Published as a conference paper at ICLR 2026
We proceed to show that in fact, u² E U solves equation if and only if A Cmin satisfies the conditions of equation B with respect to u- for all i. Suppose u E is optimal and that there exists corresponding À satisfying equation | for all i, and that it satisfies X < Cmin 3. Then for any optimal u* E U, there exists no for which u > 0, otherwise G =
< Cmin, an impossibility. This means any optimal u² E U must satisfy u² S 0, for which then we would conclude that in fact u" = O, and is the unique optimal solution, which in turn yields that À - S S. S + < Cman + for all This last inequality reveals that X = Cmin + = is compatible with the unique optimal solution un = 0 E U in such a case. In summary, if À < Cmin + is compatible with all optimal solutions u* E U, then X == Cman + 2 is well.
It follows that u² E U solves equation 3 if and only if A = Cmin + satisfies the conditions of equation 5
with respect u, for all i; hence, the result follows.
Corollary 3.1.1. Ler > and be given, yielding losses {ci = J(;; His y:) Define x(0) == [x(0]] }. {i:ce q: > Cmin + and let u² (0) given by {O) = -N
= {0). N for all i E x(0). Then u² (0) solves the linear program LRRM (f), equiv, the inner minimization of equation
and un (0) =1
Proof. Clearly, u² E U. Let = J0"; nz, 11) for i =- 1,..., N. In the proof of Theorem 3.1 it was
shown that u E U solves equation 3 if and only if A Cmii + satisfies the conditions of equation
with respect to u.* for all i. Therefore, with S₁ = S2 = 0, Theorem 3. indicates that u² U S1,S) and hence must solve equation
Proposition 3.1. Let e >> O, and suppose for any 0, max().)) [[00 10;00.11: v\1 < co. Then there exists such that for any the following problem
N Σ 1=1
MIX = min UN
N
uz) J(0; xi, x1; + r0||u|]1
S.t. u
20 N
satisfies uN (0) + $ 2 UN U (0) 2 uN (0). In particular, -7o S min; J(0; vi, vi), and, for any u² solving MIX (0), it holds that i = N for such that [0; F4. U1) > 70-
Proof. Fix 0. Then for any (x, 3) E Z, the function ((- 2, 0) is linear, and hence Lipschitz with constant 2(1,,,0) 2) (0; 3) max(r,y) [J(0; x, y)
By Lemma 3.1 of Staib & Jegelka (011)) and/or Corollary of Gao & Kleywegt (203))
N
MIX UN
(0) =
min
l(vo, x 0)
N
s.t.
wi S
provides the stated approximation of r0())
Upon introducing the change variable u, = 1a budget constraint (any convex dual optimal multúplier,, recover
N N. and applying a Lagrange multiplier "9 to the
1 21: + 70
min M
El(
N
s.t. + N
20 i i= i N
16

---

Published as a conference paper at ICLR 2026
B CASE OF CONSERVATIVE ESTIMATE C C
We repeated the experimental setup Section 4.2. to evaluate the benefit of RRM for MSE, but now with an estimate C' > C provided to RRM.
Analysis: Upon comparing Table with Table we confirm that as the estimate C' is made larger and larger than C, conferred benefit by RRM is reduced and is generally not high as when C" = C. However, despite this, RRM-wrapping still generally outperforms MSE alone.
Table 7: CIFAR-10 Test accuracy (%) of RRM-wwapped MSE with Conservative Estimate C' of C on classification task.
Contamination C'
10%
20%
30%
C 10%
91.73 = 0.28
90.60 + 0.40
88.47 + 0.73
C' 5%
92.55 + 0.15
91.29 + 0,41
89.43 + 0.64
C ENHANCING (ROBUST) Loss FUNCTION METHODOLOGIES WITH RRM
RRM enhance the performance of common loss functions that do not account for label contamina- tion, including categorical cross-entropy (CCE), mean absolute error (MAE), and mean squared error (MSE). In fact, the following experiments indicate RRM with robustness to noisy labels, such as early-learning regularization (ELR).
enhance those methods devised
Table 8: MNIST-3 Test accuracy (%) comparison of CCE, MAE, and MSE against ELR across contamination levels. ERM-wrapped accuracy results are in parentheses.
Contamination Level
Method
55%
60%
65%
ELR (+ RRM)
98 (99)
97 (98)
82 (87)
CCE (+ RRM)
90 (98)
77 (96)
46 (67)
MAE (+ RRM)
98 (98)
96 (98)
62 (68)
MSE (+ RRM)
90 (98)
74 (97)
45 (87)
Dataset: this experiment, we divide MNIST-3 into a training set 18623 examples (5923 O's, 6742 I's, and 5958 2's), and 3147 testing examples.
Architecture: We used a simple fully-connected architecture consisting of a few layers. The first two layer consists of 320 units, and the third layer consists of 200 units. Each of these dense layers employs the ReLU activation function. The output layer consists of a 3-unit dense layer with softmax activation. In total, there are 417880 trainable parameters.
Using Tensorflow 2010 (Abai et al. 2015), 100 epochs of training are performed for each of the baseline loss functions (ELR, CCE, MAE, and MSE) and training label contamination levels of 55%, 60%, and 65% for total 12 baseline experiments. In order to assess the benefit of our method, the RRM algorithm executed for each of the baseline loss functions and contamination levels. More specifically, 10 iterations of RRM are executed with total of 100 epochs. For RRM, the hyperparameter settings of μı and y are set to 0.5 and 0.4, although auto-tuning could have been implemented as in Section 4.2 All methods employ stochastic gradient descent (SGD) with a learning rate (7)) of 0.1. The test set accuracy cach experiment is shown in Table 8 with the RRM experimental results placed in parentheses.
Experiment:
10 epochs per iteration for
For every method tested, the wrapped approach confers a test set performance bene- fit over the baseline approach. The additional computation time incurred from the Re- Re-weight(( www.ghhtt..... ()) step is an average of 2.86 seconds per execution of every iteration, for a total of 28.6 additional seconds.
Takeaway:
17

---

Published as a conference paper at ICLR 2026
D NON-UNIFORM LABEL CONTAMINATION EXPERIMENT
Although we explicitly state the use of uniform label noise in Section 3.1 which is indeed a very common scheme in the literature,
analysis in fact did not rely this assumption. Towards providing insight into the non-uniform case, we have repeated the experiments Section [42.2 that produced Table []] but now with non-uniform label noise. More precisely, after uniformly at random selecting C' percent of the training pairs, we proceed to contaminate the label in each pair (xi, Dt) in the following non-uniform manner, outlined below in the transition kernel matrix of (True Label, Contaminated Label) entries. For example, if the true label 5, then instead of uniformly random drawing an alternative digit ys from among {0, ....., 9} \ {5] we have
0 w.p.0.051
1 w.p.0.017
2 w.p.O.
3 w.p.0.627
Table Contamination Kernel
Contaminated
3
4
6
0.0769
0.0769
0.1538
0
0.0769
0.3846
0.1538
0.0769
0
0.3333
0.1111
0
0.1111
0.1111
0.3333
0
0.0968
0.0645
0.2581
0.0323
0
0.0968
0. 0.1935
0.2581
0
Original
0. 0.1250
0
0.1250
0.5050
0.6250
0.1111
0.0370
0.0741
0.0741
0
0.0741
0.2222
0.0370
0.1111
0.2593
0.0508
0.0169
0.6271
0.0169
0
0.1525
0.1017
0.0339
0.2353
0.1765
0.0588
0.0588
0.0588
0.1765
0
0.2353
0
0.0500
0.2250
0.2000
0.1250
0
0
0.2000
0.2000
0.1071
0.0357
0.1071
0.3571
0.1071
0.0714
0.0714
0.1071
0.0357
0.0638
0.1702
0.2128
0.1702
0.1702
0.0213
0.0851
0.1064
0
These entries were generated by the confusion matrix of an imperfect MNIST classifier. The results from this new experiment confirm the performance benefits that were observed (compare to Table under conditions of uniform label contamination.
Table 10: MNIST-10 Test accuracy (%) for AT and A-RRM under different levels of contamination C, training perturbation 1.0, and test-set adversarial perturbation Étest
C
0%
5%
10%
20%
30%
AT
A-RRM
AT
A-RRM
AT
A-RRM
A-RRM
AT
A-RRM
0.00
96.5
97.3
93.6
95.6
60.4
87.8
32.2
92.4
58.3
89.2
0.10
93.4
95.2
89.3
92.4
63.2
84.7
89,5
36.1
81.7
Ctest
0.25
92.4
93.1
87.9
90.6
86.9
86.3
80.6
89.3
69.0
79.8
0.50
92.0
90.9
90.3
89.8
94.4
91.2
92.9
89.2
85.2
81.6
1.00
89.4
85.5
90.3
86.8
94.9
92.6
93.9
86.6
81.8
77.8
E ADDITIONAL DATA EXPERIMENTS
E TOXIC COMMENTS
Dataset: Toxic Comments is a multi-label classification problem from JIGSAW that consists of Wikipedia comments labeled by humans for toxic behavior. Comments can be any number (including zero) of six categories: toxic, severe toxic, obscene, threat, insult, and identity hate. We convert this into binary classification problem by treating the label as either none of the six categories least one of the six categories. This dataset is public dataset used as part of the Kaggle Toxic Comment Classification Challenge.
18

---

Published as a conference paper at ICLR 2026
Architecture: We use simple model with only single convolutional layer. A pretrained embedding from FastText is first used to map the comments into a 300 dimension embedding space, followed by single convolutional layer with kernel size of two with ReLU activation layer followed by a max-pooling layer. We then apply a 36-unit dense layer, followed by a 6 unit dense layer with sigmoid activation. Binary cross-entropy is used for the loss function.
Experiment: We use the Toxic Comments dataset to test the efficacy of RRM on low prevalence text data. The positive (toxic) comments consist of only 3% of the data and contaminate anywhere from 1% to 20% of the labels. There are a total of 148,000 samples, and we set aside 80% for 2 with 3 iterations of the heuristic algorithm results in a total of
training and 20% for test. epochs, and ERM is run for a total of 6 epochs to make the results comparable. Since the data is highly imbalanced, we look at the area under the curve of the precision/recall curve to assess the performance of the models. Unsurprisingly, as the noise increase, the model performance decreases. We note that RRM outperforms ERM across all noise levels tested, though as the noise increase, the gap between RRM and ERM decreases.
Table II: Toxic Comments Comparison of training and test area under the precision/recall curve for ERM and RRM at noise levels ranging from 1 1% to 20%.
Method
Percentage Contaminated Training Data
1%
5%
7%
10%
15%
20%
ERM (train)
0.2904
0.2006
0.1589
0.1302
0.1073
0.0920
RRM (train)
0.6875
0.4458
0.3805
0.3087
0.2438
0.1966
ERM (test)
0.5861
0.3970
0.3246
0.2550
0.2013
0.1717
RRM (test)
0.6705
0.4338
0.3619
0,2824
0.2208
0.1861
Takeaway: The Toxic Comment example presents another challenging classification problem, characterized by a low prevalence target class amidst label noise. Our experiments demonstrate that as the amount of label noise increases, standard methods become increasingly ineffective. However, RRM remains reasonably robust under varying degrees of label contamination. Therefore, RRM could be a valuable addition to the set of tools being developed to enhance the robustness of Al-based decision engines.
E.2 IMDB
Dataset (Maas et al., 2011): A binary elassification dataset consisting of 50000 movie reviews each assigned positive negative sentiment label. 25000 reviews are selected randomly for training and the remaining are used for testing. 25%. 30%, 40%, and 45% of the labels of the training reviews randomly selected and swapped from positive sentiment to negative sentiment, and vice versa, to achieve four training datasets of desired levels of label contamination.
Architecture: Transformer architectures have achieved SOA performance the IMDb dataset sentiment analysis task (Devlin et al. [009 Xie et al., 2020). As such, we a adopt a reasonable trans- former architecture assess RRM. We utilize the DistiBBERT (Sanh al., 2020) architecture with low-rank adaptation (LoRA) Hu et al. 2022) for large language models, which reduces the number of trainable weights from 67584004 628994. In this manner. we reduce the computational burden, while maintaining excellent sentiment analysis performance. Binary cross-entropy is employed for the loss function.
Experiment: Twenty percent of the training data is set aside for validation purposes. Using Pytorch 2.1.0, 30 iterations of RRM are executed, with o = 10 epochs per iteration for total of 300 epochs for a given hyperparameter setting. For RRM, the hyperparameter settings of μu and y at 0.5 and 0.4, respectively, are based on search to optimize validation set accuracy. For contrast, we perform a comparable 300 epochs using ERM. Both ERM and RRM employ stochastic gradient descent (SGD) with a learning rate (1)) of 0.001. In Table 12 we record both the test set accuracy achieved when validation set accuracy peaks, well as the maximum test set accuracy. At these high levels of contamination RRM consistently achieves a better maximum test set accuracy.
We demonstrate that RRM can confer benefits to the sentiment analysis classification task using pre-trained large models under conditions of high label contamination. The success of fine-tuning in L.MMs depends, in large part, on access to high quality training examples. We have
Takeaway:
19

---

Published as a conference paper at ICLR 2026
Table 12: IMDb Test accuracy (%) for ERM and RRM under different levels of contamination, Test set accuracy at peak validation accuracy and maximum test set accuracy are recorded.
Method
Percentage Contaminated Training Data
25%
30%
40%
45%
ERM
90.2, 90.2
89.5, 89.6
86.4, 86.6
80.7, 81.1
RRM
90.4
90.4
90.2, 90.4
88.4, 88.7
76.9, 82.6
shown that RRM can mitigate this need by allowing effective training in scenarios of high training data contamination. As such, resource allocation dedicated dataset curation may be lessened by the usage of RRM.
E.3 TISSUE NECROSIS
Dataset: A binary classification dataset consisting of 7874 256x256-pixxt hematoxylin and eosin (H&E) stained RGB images derived from (Amgad et al. 2019). The training dataset consists of 3156 images labeled non-necrotic, as well as 3156 images labeled necrotic. The training images labeled non-necrotic contain no necrosis. However, only 25% of the images labeled necrotic contain necrotic tissue. This type of label error can be expected in cases of weakly-labeled Whole Slide Imagery (WSI). Here, expert pathologist will provide slide-level label for a potentially massive slide consisting of gigapixels, but they lack time or resources to provide granular, segmentation-level annotations the location of the pathology in question. Also, the diseased tissue often occupies small portion of the WSI, with the remainder consisting of normal tissue. When the gigapixel-sized WSI is subsequently divided into sub-images of manageable size for typical machine-learning workflows, many of the sub-images will contain no disease, but will be assigned the "weak" label chosen by the expert for the WSI. The test dataset consists of 718 necrosis and 781 non-necrosis 256x256 pixel H&E images. which were also derived from (Amgad et al., 2019). For both the training and test images, (Amgad) jet al. 2019) provide segmentation- necrosis annotations, so we are able to ensure a pristine test set, and, in the case of the training set, we were able to identify the contaminated images for the purpose of algorithm evaluation.
Consistent with the computational histopathology literature (Petríková & Cimrák 2023), we employ a convolutional neural network (CNN) architecture for this classification task. In particular, ResNet-50 architecture with pre- pre-trained ImageNet weights is harnessed. The classifi- cation head is removed and replaced with a dense layer 512 units and ReLU activation function, followed by an output layer with single unit using a sigmoid activation function. All weights, with the exception of the new classification head are frozen, resulting in 10501 14 trainable parameters out of 24637826. Binary cross-entropy employed for the loss function.
Architecture:
Experiment: Twenty percent of the training data is set aside for validation purposes, including hyperparameter selection. 60 iterations of RRM are executed, with 10 epochs per iteration, for total of 600 epochs for a given hyperparameter setting. For RRM, the hyperparameter settings of and y at 0.5 and 0.016, respectively, are based on a search to optimize validation set accuracy. For contrast, we perform comparable 600 epochs using ERM. Both ERM and RRM employ stochastic gradient descent (SGD) with learning rate (1)) of 5.0 and 1.0, respectively. RRM achieves a test set accuracy at peak validation accuracy of 74.6, and maximum test set accuracy 77.2, whereas ERM achieves 71.7 and 73.2, respectively. RRM appears to confer performance benefit under this regime of weakly labeled data.
Takeaway: In the Tissue Necrosis example, we demonstrate that RRM also confers accuracy benefits to the necrosis identification task provided weakly labeled WSIs. Again, RRM can mitigate the need for expert-curated, detailed pathology annotations, which are costly and time-consuming to generate.
F SUPPLEMENT TO TABLE EXPERIMENTS
In this section, we supplement the experiment of Table a follow-up investigation into individual u-value trajectories.
20

---

Published as a conference paper at ICLR 2026
F.1 CLLSE-CALLS
In Table 6) is shown that the set of samples down-weighted to zero is not precisely C and some (Type-1) errors made. Upon closer examination, interesting behavior observed namely, that some samples initially down-weighted to zero were later restored to nonzero weight. In other words, these samples were close to being erroneously removed from training.
We re-ran the results of our Table 6 experiment, this time tracking all u-value trajectories. We found that of the 38400 clean-labeled samples, 37058 of them were never lown-weighted zero any time during training. Of the remaining 1342 clean- labeled samples, 626 had sample weights N + u4 ~ O upon conclusion of training, i.e, were down-wen -eightee to zero (Type errors), whereas 716 of them were "close-calls", i.e., had sample weights of + R O at some iteration before then recovering by the conclusion of training have positive sample weight + u, 2N 0.
We find these results interesting because the observed behavior indicates RRM can recover a border- line but ultimately learnable sample that assigned to near-zero weight early.
One clean-labeled training sample's u-value Rerations
56
Average "close-call" u-value trajectories
0.0
Mean uvalue
-.0
-10
-15
-2.0
20 Berations
50
10
terabions
(a) The evolution of clcan-labeled training ple's 'e's u-value across training iterations. We note that its sample weight N 24 s O at around iteration 30 before then recovering N. + us R3 by the conclusion of training.
(b) Average trajectory (over 716 "close-call" clean- labeled samples). Shading indicates one standard de- viation.
Figure 1: Behavior of u-values during training.
21


Reviews:
[{"review_id": "wTK3QNa7ZV", "reviewer": "Reviewer_XbZp", "summary": "The authors propose Rockafellian Relaxation (RRM), a method for learning with noisy labels that jointly optimizes model parameters $\\theta$ and per-sample weights $u$ under a total-variation style constraint. For fixed $\\theta$, RRM solves a linear program to obtain weights $(1/N + u_i)$ on each training example $i$, minimizing $\\sum(1/N + u_i) \\cdot J(\\theta; x_i, y_i)$ $+ (\\gamma / 2) \\lvert u \\rvert_{1}$ subject to $1/N + u_i \\geq 0$ and $\\sum u_i = 0$. The optimal $u$ places nearly zero weight on high-loss samples whose supervised loss exceeds $c_{min} + \\gamma$ and reallocates their probability mass to the lowest-loss set. Training alternates between: (i) taking $\\sigma$ steps of SGD on $\\theta$ using these per-sample weights, and (ii) re-solving the linear program using updated losses to recompute $u$. The authors present the method as a wrapper for existing noisy-label pipelines such as DivideMix, ProMix, and CC. They report accuracy gains on CIFAR-10N/100N, Clothing1M, Food-101N, and two NLP datasets.", "questions": "1. When a sample is assigned $(1/N + u_i) \\approx 0$ because its supervised loss exceeds $c_{min} + \\gamma$, do you continue to use that sample in the semi-supervised component $r(\\theta)$ (for example, as unlabeled data in a consistency or pseudo-label objective), or is it excluded from training until it regains positive weight?\n2. During $\\theta$-updates in your wrapped DivideMix/ProMix/CC runs, are gradients from $r(\\theta)$ also scaled by $(1/N + u_i)$, or are they left unweighted?\n3. More broadly, what representational insight does RRM offer that is of direct interest to the ICLR community? How does this work advance our understanding of learned representations, beyond improving robustness to noisy labels?\n\nI would be happy to further increase my score if the authors can answer these questions and revise the manuscript accordingly.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The optimization view of noisy-label learning is compelling. RRM reframes sample selection as an explicit joint optimization over $\\theta$ and a perturbed empirical distribution, rather than as a heuristic small-loss filter. The inner reweighting problem is convex in $u$ for fixed $\\theta$ and exhibits a closed-form thresholding structure: all samples above $c_{min} + \\gamma$ are effectively suppressed, and their mass is reassigned to the minimum-loss set.\n2. The method seems lightweight and practical. The algorithm alternates two inexpensive steps ($\\sigma$ steps of weighted SGD, then one reweight solve), and the contamination prior $C'$ optionally sets $\\gamma$ through a loss quantile so that pruning aggressiveness is tied directly to an estimated corruption rate.\n3. The method functions as a wrapper around already strong noisy-label learners that combine supervised and semi-supervised components. The authors convincingly demonstrate consistent improvements on realistic label noise benchmarks (CIFAR-N, Clothing1M, Food-101N).", "weaknesses": "1. The integration of RRM with semi-supervised pipelines is somewhat underspecified.\n    * The authors model existing noisy label methods as having a supervised term $J(\\theta; x_i, y_i)$ and an additional term $r(\\theta)$, then state that they replace the training loss with $L_{RRM}$ (i.e., $\\theta$ updates weight each sample's gradient by $(1/N + u_i)$).\n    * What is not made explicit is how $r(\\theta)$ participates in those $\\theta$ updates once RRM is active. Many noisy label methods (e.g., DivideMix, ProMix, CC) employ two-branch training: one branch treats some samples as labeled and optimizes a supervised loss on them, while the other uses the remaining samples in a semi-supervised objective (consistency, pseudo-labeling, auxiliary regularizers).\n    * In RRM, samples whose current supervised loss $J(\\theta; x_i, y_i)$ exceeds $c_{min} + \\gamma$ can be assigned weight $(1/N + u_i) \\approx 0$, which zeroes their gradient contribution in the weighted SGD phase. The critical question is whether these samples still contribute through the semi-supervised branch $r(\\theta)$ as unlabeled/consistency signal, or whether they are effectively removed from training altogether.\n    * I point this out because methods like DivideMix are designed so that even suspected-noisy samples influence training through their semi-supervised branch rather than being discarded. If RRM can fully suppress those samples in the supervised branch when RRM and the inner wrapped method disagree on the noisy/clean partition, the optimization may not be maximally using all of the available data.\n2. The authors do not fully characterize the trajectory of sample weights over time.\n    * The authors report that the set of pruned samples converges and stabilizes during training.\n    * However, to my knowledge, the paper does not show trajectories of individual samples. Interesting behavior might arise: for example, some samples initially downweighted to $(1/N + u_i) \\approx 0$ may or may not later be restored to nonzero weight. Such behavior would show whether the method can recover a borderline but ultimately learnable sample that was assigned to near-zero weight early.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yb9aLLP3IZ", "reviewer": "Reviewer_myha", "summary": "The paper proposes the Rockafellian Relaxation Method (RRM), a simple, architecture-agnostic loss reweighting scheme for learning with noisy labels. RRM formulates training as a min-min problem over model parameters and example weights with a TV-style penalty, yielding a linear program with a closed-form solution that prunes high-loss samples beyond a threshold cmin + γ and redistributes their mass to lowest-loss samples. The method can be auto-tuned via an estimated contamination rate C′ and can be used with or without adversarial training (A-RRM). Experiments on CIFAR-N, Clothing1M, Food-101N, CIFAR-10, MNIST, IMDb, Toxic Comments, and histopathology show improvements over baselines and, in some cases, state-of-the-art results.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Simple, general “wrapper” that can enhance diverse methods and losses without architectural changes.\n2. Clear optimization view with an explicit, efficient solution; connection to optimistic DRO/Wasserstein formulations is insightful.\n3. Auto-tuning strategy to control pruning via an estimate of noise level.\n4. Broad experimental sweep (vision, NLP, medical imaging) with competitive or improved results; SOTA on CIFAR-100N when wrapping strong baselines.\n5. Useful analysis of u’s convergence showing selective downweighting of noisy points; practical relevance when test-time perturbations differ from training.", "weaknesses": "1. Novelty is moderate: the resulting rule effectively implements trimmed/quantile ERM focused on smallest-loss samples; closely related to self-paced/trimmed risk/reweighting/DRO literature (e.g., self-paced learning, MentorNet, trimmed loss), but comparisons to these specific baselines are missing.\n2. Dependence on contamination estimate C′: despite some ablations, guidance for estimating C′ without clean data is thin; sensitivity analysis is limited.\n3. Experimental fairness and reporting: some baselines lack confidence intervals; training schedules and adversarial settings are not always matched; improvements are sometimes small or inconsistent (e.g., MAE).\n4. The method of weighting samples is widely used in the field of noisy label learning, such as [1][2]. However, the authors did not conduct a comparison. Furthermore, the baseline used by the author is quite outdated. It would be better to incorporate some more recent related works and baselines, such as those from 2024 and 2025.\n5. The author should include a comparison of the time spent. I think this is very valuable for reference.\n\n[1] Knockoffs-SPR: Clean Sample Selection in  Learning with Noisy Labels.\n\n[2] Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ipeefmbr01", "reviewer": "Reviewer_yHgs", "summary": "This paper introduces the Rockafellian relaxation method to enhance the capacity of neural networks to tolerate noisy labels.", "questions": "NA", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "- The authors make an effort to propose a general framework for improving robustness in learning under noisy labelling conditions.\n\n- The paper attempts to provide both theoretical analysis and empirical results, showing the authors’ intention to bridge formulation and practice.", "weaknesses": "This submission contains a significant formatting violation that should be desk-rejected. Specifically, the appendix section appears before the bibliography, causing the main text to exceed the 9-page limit.\n\nRegarding the work itself, the current version is very difficult to follow, and the overall presentation lacks clarity. The writing requires substantial improvement to enhance readability and logical flow.\n\nIn the next revision, the authors should at least provide a clear introduction to the concept of Rockafellian relaxation and explain the motivation for applying this technique to address label noise. The paper should also avoid overcomplicating simple concepts. For example, Equations (1) and (2) are described in an overly complex manner.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)