Learning to Clean: Reinforcement Learning for Noisy Label Correction
Authors: Marzi Heidari
Hanping Zhang
Yuhong Guo
Venue: NeurIPS 2025 poster ()
Keywords: Learning with Noisy Labels
Reinforcement Learning
Primary area: general_machine_learning

Abstract: The challenge of learning with noisy labels is significant in machine learning, as it can severely degrade the performance of prediction models if not addressed properly. This paper introduces a novel framework that conceptualizes noisy label correction as a reinforcement learning (RL) problem. The proposed approach, Reinforcement Learning for Noisy Label Correction (RLNLC), defines a comprehensive state space representing data and their associated labels, an action space that indicates possible label corrections, and a reward mechanism that evaluates the efficacy of label corrections. RLNLC learns a deep feature representation based policy network to perform label correction through reinforcement learning, utilizing an actor-critic method. The learned policy is subsequently deployed to iteratively correct noisy training labels and facilitate the training of the prediction model. The effectiveness of RLNLC is demonstrated through extensive experiments on multiple benchmark datasets, where it consistently outperforms existing state-of-the-art techniques for learning with noisy labels.

Learning to Clean: Reinforcement Learning for Noisy Label Correction
Marzi Heidari', Hanping Zhang', Yuhong Guo
School of Computer Science, Carleton University, Ottawa, Canada 2CIFAR Al Chair, Amii, Canada
(marzihheidariicmail,, jagzhang@cmail, aahong-guo888 www.eton....... ca.
Abstract
The challenge of learning with noisy labels is significant in machine learning, it can severely degrade the performance of prediction models not addressed properly. This paper introduces novel framework that conceptualizes noisy label correction as a reinforcement learning (RL) problem. The proposed approach, Reinforcement Learning for Noisy Label Correction (RLNLC), defines compre- hensive state space representing data and their associated labels, an action space that indicates possible label corrections, and reward mechanism that evaluates the efficacy of label corrections. RENLC learns a deep feature representation based policy network perform label correction through reinforcement learning, utilizing an actor-critic method. The learned policy is subsequently deployed to iteratively correct noisy training labels and facilitate the training of the prediction model. The effectiveness of RLNLC is demonstrated through extensive experi- ments on multiple benchmark datasets, where consistently outperforms existing state-of-the-arr techniques for learning with noisy labels.
Introduction
Deep neural networks have made significant strides in various domains of machine learning 14 These models predominantly utilize supervised learning, which depends extensively on the avail- ability of large datasets with high-quality labeled data. Unfortunately, in real-world applications, the quality of labels is frequently compromised by various issues, including subjective errors from indistinguishable samples and objective mistakes in label recording, making noisy labels a common challenge. The presence of such label noise has been shown to significantly undermine the general- ization ability of deep neural networks. Therefore, the development of robust methods for learning from noisy labels of paramount importance [5].
Label noise has been categorized into two primary types: class-conditional noise (CCN) and instance- dependent noise (IDN) (6), while many recent works have focused the more common IDN, where noise distribution correlates with feature similarities among samples 77 Deep neural networks tend to learn from clean data initially and subsequently adapt noisy data, which underpins several sample selection methods that utilize training dynamics, such as prediction loss and confidence levels, to isolate and manage noisy labels [10] IT] To enhance the resilience of models trained noisy datasets, other methods have evolved to include sophisticated training regimens that incorporate label filtering [12]. label correction 15, robust loss formulations [16), and semi-supervised learning frameworks [8). However, these methods typically lack the ability to actively explore different possibilities or learn from long-term consequences, which can limit their effectiveness in more complex or highly noisy scenarios.
To address this drawback, we propose to formulate the task of noisy label rectification as reinforce- ment learning problem. Reinforcement learning can dynamically adapt its noisy label correction 39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

strategy based on feedback from its actions, optimizing performance through sequence of decisions that maximize a cumulative, long-term reward. This ability to make sequential, non-myopic decisions is particularly well-suited for addressing label noise in complex and instance-dependent environ- ments. Specifically, we devise a novel policy gradient method, named as Reinforcement Learning for Noisy Label Correction (RLNLC), to learn stochastic policy function, which determines the label correction actions in a given state and consequently separates the training data into a clean subset without label corrections and noisy subset with corrections, by maximizing the expected cumulative reward. A key component of this RLNLC method is the reward function, which evaluates the impact of actions and the resulting states. We design the reward function to enhance both label consistency across the entire dataset and the inter-subset alignment of corrected labels with clean labels. This is achieved through k-nearest neighbor prediction mechanisms, aiming enhance the robustness of label correction, maintain instance-dependent and prediction-aware label smoothness, and support the performance of downstream label prediction tasks. This RL. formulation allows for dynamic adap- tation to the evolving data characteristics, offering a robust method for noise correction. We evaluate RLNLC through rigorous experiments on benchmark datasets, demonstrating its effectiveness in various noisy settings. The contributions of this work are summarized as follows:
We innovatively formulate noisy label correction as an RL, problem and propose RLNLC, a method that leverages the adaptive capabilities of RL to effectively address label noise.
We design a tailored policy function based on a deep representation network, which adap- tively determines label correction actions by considering the complete state of the data.
We develop an informative reward function that encourages effective label correction by aligning with both local data structures and confident clean labels through k-nearest neighbor prediction mechanisms.
We devise an efficient encoding scheme for the critic network to enable effective deployment of the actor-critic framework for optimizing the policy function.
Extensive experiments demonstrate that RLNLC outperforms existing state-of tfae-off-hhe-at meth- ods, highlighting the effectiveness of this innovative RL approach.
2 Related Work
2.1 Learning with Noisy Labels
Deep neural networks are highly susceptible to overfitting when trained on datasets with noisy labels and thus perform poorly clean test datasets (5). Previous studies have primarily focused on two types of label noise, class-conditional label noise (CCN) and instance-dependen label noise (IDN).
Class- Cuss-Conditional Label Noise For the CCN label noise, the probability of label being noisy is conditional on the class but not on individual instances. To address CCN, MentorNet [10 and Co-teaching [11] use networks to select low-loss, reliable samples through guidance. Reweighting techniques adjust the training influence of each sample based on estimated noise levels, thus enhanc- ing robustness [19] Decoupling (20) updates model parameters only on instances with classifier disagreement, thus mitigating the reinforcement of noisy labels. Nested Dropout (21) integrates dropout with curriculum learning, increasing sample difficulty progressively to enhance model re- silience to noise. Semi-supervised learning strategies, e.g., DivideMix (17), use loss distribution modeling to differentiate between clean and noisy data, treating noisy labels as unlabeled to leverage semi-supervised techniques.
mustance-Dependee Label Noise In the IDN scenario, the label noise probability varies with instance features, presenting unique challenges. To address this problem, CleanNet 22] leverages clean validation set to assess and correct noisy labels accurately. Advanced strategies like Lon- gReMix [18] combine consistency regularization with data augmentation to effectively handle the variability in noise. Pseudo-Label Correction (PLC) [15] refines labels during training using the model's predictions, enhancing label accuracy. Adaptive reweighting methods such as BARE [23 dynamically adjust the weights of samples based on their likelihood of being clean, addressing the complexities of IDN. SSR (24) employs statistical methods to estimate and correct noise rates, facilitating more effective learning from noisy datasets by adapting training strategies to the specific

---

State
P Puiiay
Adtion
Rycu
Policy
Rewerd
So
plo 1)
R(s ,a)
Q(ot,a") (- Qo(v(11)
Dalan
ReLa
Actor-Critic
Figure I: Overview the proposed RLNLC. Each data point x; is associated with an initial label y: that is potentially noisy. The policy network тa is constructed over deep feature extraction network fe. and it determines actions based on the current state of the data s² = {(x,, (x), Hy) resulting in label corrections. The updated labels subsequently lead to the next state. The reward function designed to evaluate the labels in an instance-dependee manner, capturing dataset-wide label consistency and the inter-subset alignment of the noisy labels with clean labels. The policy function learned using an actor-critic method.
characteristics of the noise. SURE 25 enhances uncertainty estimation by integrating techniques from model regularization like Mixup cosine similarity classifier, and sharpness aware minimiza- tion tackle IDN.
2.2 Reinforcement Learning
Reinforcement learning (RL) is a machine learning framework in which an intelligent agent learns to make optimal decisions by interacting with environment and achieving goals through sequential srcision-making [26]. RL is known for its ability to explore effectively [27] 28
and to learn Leveraging these strengths, RI. has seen domains over the past few years 31
optimized decisions for long-term outcomes 229 30
extensive application in decision-make
32 In particular,
the remarkable success of ChatGPT has provoked significant interest in Reinforcement Learning from Human Feedback (RLHF) (33H35], where RLHF fine-tunes complex LLMs by leveraging reward model trained on human-provided feedback. Recent work has applied RLHF fine-tune text-to-image diffusion models, achieving superior alignment with user preferences (36]. Another study reformulated semi-supervised learning (SSL) as a bandit problem to guide classifier training using weighted rewards [37]. More recently, finisoffiiiiinggg tuning has been framed as goal- conditioned RL problem, allowing iterative optimization of pipeline parameters guided by a goal image (38). These advancements highlight RL's adaptability in fine-tuning and optimizing diverse machine learning systems.
Policy Gradient Methods Policy gradient is class of RL methods designed directly learn an optimal policy that maximizes cumulative rewards, REINFORCE [39], foundational policy gradient method, uses Monte Carlo simulations to estimate the value function and calculate the policy gradient. Actor-Crite Critic methods 40) extend this by incorporating value function (the Critic) to evaluate the policy (the Actor), providing feedback to assist updating the policy. One of the key advantages of policy gradient methods is their ability to learn stochastic policies, making them effective in complex environments requiring exploration. Additionally, they are well-suited for high dige--immnsional
or complex action spaces, driving their adoption across diverse applications [1 36 37
3 Method
3.1 Problem Setup
We assume noisy classification training dataset D P) D:)) where each input instance X; E X' paired with an observed label vector y; E y that potentially a corrupted noisy version of the true label vector y.. The objective is to learn an effective prediction model, defined as

---

the composition hy o fa X + y/ where fo denotes the feature extractor with parameters 8, and h represents the classifier with parameters s. We aim develop an effective label cleaning technique to correct noisy labels and enable efficient training of prediction models without being hindered by the challenges posed by label noise.
3.2 RL for Noisy Label Correction
Reinforcement Learning (RL) problem is often modeled as a Markov Decision Process (MDP) 126] characterized by the tuple M = A, R), where S denotes the state space, A represents the action space, P(s'| 18, a) defines the transition dynamics for 8, 8' E S and a E A. R: S x A + R a reward function, and y E (0. 1) is the discount factor. The primary objective is determine an optimal policy πt' + AA that maximizes the expected discounted cumulative reward J, (x) E E ('R(("
In this work, we formulate the challenge of noisy label cleaning as an RL problem by properly designing and defining the key components of the MDP. We then deploy an actor-critic method with state encoding to learn an optimal policy function for noisy label correction. The overall framework of the proposed RLNLC is illustrated in Figure with the approach detailed in the remaining section.
3.2.1 State
We define the state as the observed data and corresponding (corrected) labels, expressed as s = D' = (X, Pt) {x) ) at any time-step t. The provided training dataset D = (X, F) can be treated as static initial state sy. To enhance the exploratory aspect of our RL methodology, we initiate each RL process by randomly altering small subset of labels in d to establish an initial state s". The state encapsulates the current knowledge about the environment and serves as the input for the policy function и. As the model operates, the policy и selects actions based on the current state s'. These actions primarily involve modifying the labels, which leads to a transition
new state, st+1 = (X, p(+1). Ideally, we expect to reach an optimal goal state where the labels are "clean".
3.2.2 Action and Policy Function
We define the action space A as the possible binary valued vectors of label correction decisions on all the data instances in any state, such that for each action vector a = [a,, .a,," ... aN A, a, E {0, } indicates whether the current label for instance x; needs to be corrected (a, = 1).
At the core of RLNLC is a probabilistic policy function, Te, which maps a given state, s actions in stochastic manner. Given the state and action definitions above, the policy function is designed to make probabilistic decisions regarding whether to apply label corrections each instance x; in the dataset (i.e., setting a, = 1) given the current state s' {x, ) To effectively identify and rectify noisy labels, we define a parametric policy function Ty based on the label consistency of k-nearest neighbors, calculated within the embedding space generated by a deep feature extraction network for which initially pre-trained on the given training dataset D using standard cross- eross-nntoppy loss. The underlying assumption is that a label inconsistent with those of its k-nearest neighbors is more likely to be noisy and need of correction.
Let (x:)) denotes the indices of the k-nearest neighbors of instance x; within the dataset DD based on the Euclidean distances calculated in the embedding space, (xX), extracted by the current fa- For each instance x,, we then aggregate the labels of its k-nearest neighbors in the current state s' using an attention mechanism to generate a new label prediction. y,, that aligns with the local data structures encoded by fo- Specifically, y, is computed as follows:
y: EjEN(H) 0i
The attention weights { } are computed using similarities of instance pairs in the embedding space, such that:
exp (sim fo(x)) Otg Σ Ex))(()) exp (sim (fo (x,), (x,;)))
(2)
where T is a temperature hyperparameter, and sim(..-) -) denotes the cosine similarity. Finally the probability of taking each element action af 1 for instance x; in state andand thus the policy
4

---

function- is computed by comparing the k-nearest neighbor predicted label vector y: with the current label vector y/. such that:
1= (V. > Vij.) Vy.
p(a) = Σj=1 1 2 Viy.) Vy
(3)
where C' denotes the length of the label vector-the the number of classes for classification, Di arg max, Yiy denotes the original predicted class index for instance x, in state s².
The probability p(a = 1) defined above quantifies the level of label the extent to which the class prediction from the k-nearest neighbors disagrees with the original class prediction Dr. thereby indicating the likelihood of noise and the probability of applying label correction to x,. To interpret, the probability p(aद = 1) is proportional to the sum of probabilities of classes that are more likely than the original predicted class 3/ in the k-nearest neighbor prediction y. This sum is then normalized by the sum of probabilities of classes whose probabilities are no less than that the original label Di in ys. This normalization ensures that p(a = 1) scales appropriately relative to the level of disagreement in class predictions, without being affected by disagreement- irrelevant classes whose probabilities in y, are lower than that of y.. Note, the probability p(a = 1) becomes zero when all the other classes except Yi become disagreement iereenntirreeevat--aa ie.. class y) has the largest probability in y, and there is no prediction disagreement.
Deterministic Transition with Stochastic Policy Given state 8", we determine the action vector a' using the stochastic policy function ve introduced above, enhancing the exploratory behavior of the learning process. Specifically, randomly sample the value for each binary-valued action element at from a Bernoulli distribution, Bernoulli(p)) with p; = p(a = 1). Given (s a²), , we then deploy a deterministic transition model to induce the next state g++1 = (xx, )) such that:
Jizi
if a = O, if 1.
(4)
This mechanism maintains soft label distribution in vector space, ensuring that label corrections occur probabilistically based on the likelihood of a label being noisy. By introducing randomness, the learning process explores broader range of state-action pairs, increasing the chances of discov- ering better strategies while avoiding suboptimal solutions. Additionally, it mitigates disruptions from abrupt label changes, allowing the model to adaptively learn from the evolving environment characterized by different states, ultimately enhancing the overall robustness of the learning system.
3.2.3 Reward Function
In RL, the reward function R(s², a') provides feedback on the quality of an action a' taken in given state s', guiding the learning process toward its objective. To achieve the goal of cleaning label noise for the proposed RINLC, we design the reward function evaluate the quality of an action through the labels produced from taking the action. Our reward function R consists of two sub-reward evaluation functions: label consistency reward function, RECR. and a noisy label alignment reward function, RNLA-
Label Consistency Reward With the deterministic transition mechanism, the next state s1+1 is obtained in a fixed manner by taking action a' in the given state s'. Consequently, the quality of the labels ( 1= in sl+1 directly reflects the quality of the state-action pair (s* aa).. We design the label consistency reward (LCR) function, RLCK. to assess the label quality in si+1_i.e., how well each label fits the dataset-based on the k-nearest neighbor label prediction mechanism. To separate the impact of the policy function from label quality evaluation, we employ a fixed backbone model, fur pre-trained on the original dataset to extract embedding features for the data instances in g!+11 supporting k-nearest neighbor label prediction. Specifically, the label consistency reward function quantifies the negative Kullback-Leible (KL) divergences between the given labels in gl+1 and the k-nearest neighbor predicted labels across all the instances, such that:
RLCR (s', a²) -Exe[L:N] KL. r141
aGsst11 (x.)
Σ
(5)
5

---

Here, (x;) denotes the indices of the global k-nearest neighbors of instance within s calculated using embeddings extracted by fw. The attention weight a;j computed similar manner as in Eg. (2), but based on the feature extractor Sw- This reward function evaluates the statistical smoothness of the labels based on local data structures from a prediction perspective, aligning with the ultimate goal of supporting prediction model training.
Noisy Label Alignment Reward In addition, to evaluate the stability and robustness of the label correction action a', we divide the data in state g into subsets: clean subset D that contains the indices of instances whose labels are changed-i.e., a 0, and a noisy subset D that contains the indices of instances whose labels are corrected -ereetteddde...... of = 1. We design the noisy label alignment (NLA) reward function, RNLA, to assess how well the noisy labels in 1 align with the clean labels in cle based on an inter-subset k-nearest neighbor label prediction mechanism, such that:
as' = =-F JED KL.
Σ
0ig
(6)
(x.)
Here, .Nle (x,) represents the k-nearest neighbors identified in the clean subset for an instance x; from the noisy subset. Both (x,) and the attention weights {aiz} are again computed using embeddings extracted by fu. The aligmment- negatt-v KL-divergence- brrrgenceebetween each noisy label yi+ and the antention-based aggregation from its k-nearest clean neighbors reflects the statistical consistency of the label correction applied on x; by the action a'.
The composite reward function R(s a²) integrates the two sub-reward functions introduced above as follows:
R(s", a²) exp (PLCRR a') XRNLA ,a))
where À is a trade-off hyper-paramene that balances the contributions of the two sub-reward functions. Since the value of negative KI.-divergence is non-positive and unbounded, we deploy the exponential function, exp(.). ). to rescale the reward values to the range of (0, 1]. This scaling mechanism is crucial for ensuring that the rewards remain bounded and normalized, facilitating a stable learning process.
3.2.4 Actor-Critic Method
Based on the RL formulation of the noisy label correction problem, defined through the key MDP components above, we adopt an actor-critic framework to learn the policy function by maximizing the expected cumulative reward.
In this framework, the policy function ma is considered as an "actor", while an action-value function (8 a) parameterized by &i is introduced as ((s, a) Ex, |E o RR((s", ; a²)(s" s s, a a). which represents the expected discounted cumulative reward from taking action a in state s. The learning objective for the policy function in the actor-critic method can be written
"critic" to directly estimate the Q-value, defined
J(TO) = I Ea-Pas
Σ. mo(a E.mo(a Q(s,
(8)
where PTo denotes the stationary state distribution. The gradient of the objective over 0 can be expressed as:
VoJ( Eprergie>rptt) [ log (a(aa)))(ss)) a)).
The Q-values involved in the objective above are estimated using the critic network Qp. To learn the critic network, we use the SARSA method to compute the Temporal Difference (TD) error [42 26
for making on-policy updates the critic function. The TD error at time-step - 1 given by:
8لع R(s ) + QQ(s", a') Q=(s" aaa
(10)
The parameters of the critic network, o. are updated via the following stochastic gradient step using the TD error 1.
ー + + 38 VoQo(st 1
(11)
where 3 is the learning rate for the critic.

---

Input Encoding for the Critic The critic network, Qg. requires both the state and action as inputs. Given the potentially large dimensions of the states and actions corresponding to the size of the training dataset, an efficient input encoding scheme is essential for reducing the computational cost. To this end, we devise a simple yet effective two step encoding scheme. First, given that our proposed RLNLC framework utilizes deterministic transition mechanism, the next state s is uniquely determined by the current state-action pair (s', a²). Consequently, we can use g to replace the corresponding input pair (s*, a²). We calculate r (xx;,((+) as reward for a single pair of instance (x,y) y !f+! with the same principle as Label Consistency Reward:
rxx s(+)) exp (-
KL [KLL 6441 Eangz
(12)
(EN
where (x;) is the indices of neighbor set of (x,) in D, and {aiz} are calculated using Eq. ( The exp(.) function is used to normalize the reward and bound it in (0, 1j. Next, we employ binning
strategy to encode the state s = {(x; {( )) based on the reward evaluations, substantially
reduces its dimensionality. Specifically, we consider No (N₂ < N) number of bins and allocate each
instance-label pair (x1, + in g one bin based on the following rule:
J (5-1) i
(x. 1 E By if S(x,, E
(13)
No
No.
where Bj denotes the j-th bin with j E [] No). and S is the exponential of label consistency reward computed on the single given instance using Eq. (12). After binning, we construct a vector v with length N₂ to encode the state s with each entry representing the proportion of instances allocated to the corresponding bin, such that:
r+1 |B,I/N
(14)
where [By] denotes the number of instances in the j-th bin. This encoding process is deployed as part of the critic network Qo to transform the inputs (s', a²) into a simple vector v facilitating subsequent learning and promoting computational efficiency.
3.3 Label Cleaning for Prediction Model Training
After learning the policy function To using the actor-critic method, deploy the trained policy function for T" time-steps to perform label cleaning on the noisy training dataset DD This creates a trajectory with length 7' to progressively correct the noisy labels, starting from the initial state sg- The labels obtained in the last state sT ((x, y: > are treated "cleaned" labels. To promote efficiency, the prediction model h2 fe is pre-trained on the given dataset D with noisy labels using standard cross-entropy loss. The feature extraction network fo then is further trained as part of the policy function learning. The final prediction model has fo is obtained by further fine-tuning it on the "cleaned" data in s with the standard cross-entropy loss. The learning process of the proposed RLNLC method is summarized in Algorithm]]
4 Experiments
4.1 Experimental Setup
Datasets In the experiments, we employ four benchmark datasets to evaluate RLNLC under diverse noise conditions. CIFARIO-IDN and 50,000 training and 10,000 test images across 10 and 100 classes, respectively. Following prior work 77, instance-dependent label noise injected into their training sets to mimic realistic corruption. Animal- 1ON [47 includes ten visually similar animal classes with 50,000 training and 5,000 test images, featuring roughly 8% noisy labels. Food-101N 22 consists of 310,009 web-sourced images over 101 categories with about 20% label noise, evaluated using the clean Food-101 test split of 25,250 manually verified images. Together, these datasets encompass experimental scenarios with different levels of label noise complexity.
adapted from CIFAR datasets 46), contain
7

---

Algorithm RLNLC Training Algorithm
Input: Initialized policy network my and critic network Qo: static initial state s8. Output: Trained policy network parameters 0 and critic network parameters Φ. for each epoch do
Randomly modify some labels in sq to create the initial state s
for 0, .... do
Form (N;)) Si= for all x, E D.
Compute {ass [==1 f E (xx, using Eq.
1.
Compute the predicted labels /1 using Eq.
Sample action a' xo(-|s²) based the probabilities computed using Eq.
Transition to the next state s g++1 by taking action a' using Eq. 4
Compute RLCR and RNLA and combine them store reward R a²) using Eq. 5 Eq. 6
and Eq. 7
Compute and store Q-value: Q(st, a²) Qolst Ca(ss,aa)) aa). 0 < + 3y V log To(a" (a'(ss))(s'" a²).
if > then
8th R(s a ) (ss a²) Q0(s"1 a
é - á + 38 V Ro a
end if end for end for
Table 1: Test accuracy (%) of different methods CIFARIO-IDN and CIFARIO0-IDN under various IDN noise rates. Standard deviations are shown as subscripts in parentheses. Columns correspond to different label noise ratios. denotes results reproduced using publicly available source code.
TO-NNN
CIFARTOO-IDN
Method
0.20
0.30
0.40
0.48
0.50
0.20
0.30
0.40
0.45
0.50
CE
0
75.8
69.2
62.5
51.7
39.4
30.4
24.2
21.5
15.2
14.4
Mixup
481
73.2
72.0
61.6
56.5
49.0
32.9
29.8
25.9
23.1
21.3
Forward
45
74.6
69.8
60.2
48.8
46.3
36.4
33.2
26.8
21.9
19.3
Reweight
76.2
70.1
62.6
51.5
45.5
36.7
31.9
28.4
20.2
Decoupling
20
78.7
75.2
61.7
58.6
50.4
36.5
30.9
27.9
23.8
19.6
Co-teaching
81.0
78.6
73.4
71.6
45.9
38.0
33.4
28.0
25.6
24.0
MentorNet 10
81.0
77.2
71.8
66.2
47.9
38.9
34.2
31.9
27.5
24.2
DivideMix
17
94.8
94.6
94.5
94.1
93.0
77.1
76.3
70.8
57.8
58.6
CausaINL
6
81.4
80.3
77.3
78.6
67.3
41.4
40.9
34.0
33.3
32.1
SSRT 24
96.5
96.5
96.3
95.9
94.1
78.8
78.6
77.0
75.0
72.8
RLNLC (Ours)
97.1 (7.3
(o.1)
(6.9
(6.6
95.
(0.4))
(0.5
80.1(a.7)
(8.5
77.2
74.7
Implementation Details In line with previous research 18
48
our experiments employed a and Food-101N, For
ResNer-34 backbone for CIFAR10-IDN and ResNet-50 for Animal-1ON, a VGG-19 with batch normalization, as outlined in prior work (47) is utilized. We also used ResNet-18 for experiments with symmetric noise CIFAR10 and CIFAR 100. Our training methodology involved stochastic gradient descent (SGD) with momentum of 0.9 and a batch size of 128. We also implemented L2 regularization with coefficient of 5 10- We started with an initial learning rate of 0.01, which was reduced to 0,001 at the halfway point of the training duration. Additionally, preliminary warmup phase was conducted for the first 50 epochs. We train the policy network on the CIFAR 10-IDN, CIFAR 100-I000,, IDN, and Animal- 10N datasets for 500 epochs and set 10. Subsequently, we evaluate the trained policy on the training data with trajectory length of T' = 25. Using the corrected labels, we fine-tune the prediction model for an additional 100 epochs. The parameters т, 7, No. and i: are set 0.5, 0.5, 0.9, 100, and 10, respectively.
Comparison Results
We compare the proposed RLNLC with set of methods, including CE 66, Mixup 43 Forward 44]
Reweight [19], Decoupling 200,, Co-teaching [[]), Co-teaching+ 11]. MentorNet 110]
Div
17 CausaINI. (6. SSR (25. SSR+ 25 Nested-
(21]
(21) SELFIE 47
PL. 15). Nested-CI (2T. CleanNet [22] BARE 23, DeepSelf 45 PI.C [15), LongReMix
18
and SURE [25]. We record the average results of RLNLC over five independent runs.

---

Table 2: Test accuracy (%, mean and standard deviation) on Animal-1ON. denotes results reproduced using publicty available source code.
Method
CE
N N-Dropout CE+Dropout SELFIE
PLC
Nested-CE SSR SSR+ SURE
RLNLC (Ours)
(9.4
81.8
81.3
(1.8 83.4(0.4)
84.1
87.7 88.5 89.0
90.2
Table 3: Test accuracy (%, mean and standard deviation) on Food-101N
Method
CF]6]
[2
BARE[23
DeepSelf[45]
PI.CIIS] LongReMix [18] RLNLC (Ours)
81.6
83,9
84.1
85.1
85.2(0.0)
87.3
89.2(0.1)
Table presents the performance of our method compared existing techniques with various noise rates on the CIFAR I0-IDN and CIFAR 100-IDN datasets, utilizing ResNet-34 and ResNer-50 as backbone networks, respectively. Our proposed method, RLNIC,, consistently produces the best results. On CIFARIO- IO-NR, RINLC achieves a noteworthy improvement of 1.7% over the second best method, SSR, with a large noise rate of 0.50, highlighting our method's ability to maintain high accuracy under elevated noise conditions. On CIFAR 100-IDN, our RLNLC consistently outperforms the nearest competitor, SSR, by margins of 1.7%, 1.5%, 1.5%. 2.2%, and 1.9% across noise rates of 0.20, 0.30, 0.40, 0.45, and 0.50, respectively. This sustained superiority across various noise intensities demonstrates the robustness and adaptability of our approach, affirming its effectiveness in handling complex noisy label scenarios.
The results in Table showcase the comparative performance of our proposed RLNLC the Animal-10N dataset, with VGG-19 as the backbone network. Our method achieves a leading test accuracy of 90.2%, marking a significant improvement of 1.2% over the next best method, SURE. This enhancement underscores the advantage our approach provides over established techniques.
Table presents the comparative results the Food-101N dataset employing ResNet-50 as the backbone network. RLNLC achieves the best test accuracy of 89.2%, outperforming all competing methods by significant margin. Specifically, we observe an improvement of approximately 1.9% over the second best- performeffnng method, LongReMix.
We also conducted experiments on CIFARIO and CIFARIOO with symmetric noise 44 type of class-conditioned noise, using ResNet- as the backbone network. The comparison results across different noise rates are reported in Table 4 Our RINLC exhibits superior performance on both CIFARIO and CIFARI00 under high symmetric noise conditions. Notably, at a 90% noise level on CIFARI00, RLNLC achieved an impressive test accuracy of 44.2%, significantly surpassing DivideMix, which recorded only 31% This 13.2 percentage point increase emphasizes the robustness and effectiveness of RLNLC in handling extreme noisy scenarios. Similarly, on CIFARIO with the same noise level, RLNLC reached an accuracy of 82.1 %, outperforming DivideMix by 6.7 percentage points. Such quantitative enhancements validate the superior capabilities of our method in maintaining accuracy despite high levels of label noise.
4.3 Ablation Study
We conducted an ablation study to investigate the impact of different components in our proposed RLNLC framework the overall performance. The study was carried out on the CIFAR100 10NN-DD dataset, with noise rates ranging from 0.20 to 0.50. Specifically, we considered four ablation variants: (1) "-w/o RNLaA" " removes the noisy label alignment reward; (2) "-w/o RL,RR" " drops the label consistency reward; (3) "-w/o random. g drops initial state randomization and starts each epoch's 's training from the fixed initial state s + = D; (4) "Su < so" uses the policy network feature extractor for reward evaluation. The comparison results are reported in Table 5 We can see that removing any key component from RLNLC leads to a noticeable decrease in performance across all noise levels. Specifically, the removal of RNLA results in lower accuracy, highlighting its crucial role in leveraging clean data to guide the correction of noisy labels within the dataset. Similarly, excluding the label consistency reward, RICH, diminishes the system's 's ability to maintain local label consistency, which is essential for the model's 's robust performance across various noise conditions. The variant without initial state randomization shows a smaller decline in performance compared to the first two ablated conditions but still underperforms relative to the full RLNLC model. This suggests that slight initial state randomization helps the model in exploring more diverse corrective strategies. Finally, the variant "fw +- fo" leads to a clear performance drop, demonstrating that decoupling the policy and reward networks is essential for stable reinforcement learning.

---

Table 4: Test accuracy (%) of different methods on CIFARIO and CIFARI00 under various symmetric
noise rates. Columns correspond to different label noise ratios. Bold values indicate the best results.
CIFARIO
CIFARI00
Method
0.50
0.80
0.90
0.50
0.80
0.90
CB 17
57.9
26.1
16.8
37.3
8,8
3.5
Co-teaching+
12
84.1
45.5
30.1
45.3
15.5
8.8
Mixup 43
77.6
46.7
43,9
46.6
17.6
8.1
DivideMix
94.4
92.9
75.4
74.2
59.6
31.0
RLNLC (Ours)
97.4 (7.4(..1)
95.8(u.2)
82.1
81.2(0.3)
70.6
(u.3
Table 5: Results of ablation study in terms of test accuracy on CIFARIO0-IDN, Top row presents the label noise ratios. Bold font indicates the best results.
Method
0.20
0.30
0.40
0.45
0.50
RINI C (Ours)
(0.5
(0.7)
78.5
77.2
74.7
-Wo RNLA
78.4(0.4)
77.9(0.8)
76.2(o.6)
76.3(o.8)
(2.0
w/o RECR
79.3(p.6)
78.5(0.7)
(6.1
(6.1
73.9(0.8)
w/o random. 8
(9.9(0.6)
79.5(0.0)
77.8(a.6)
76.8(0.9)
73.1(0.9)
[w Se
78.4(0.8)
(6.9
75.2
74.3(0.s)
73.8
Accuracy
Accuracy
Sxte mep-114) 4)
(a) CIFARIO-IDN
(b) CIFARI00-IDA
Figure 2: Label correction accuracy on the training set by deploying the trained policy function for T time-steps. Results on CIFAR1Ó-IDN and CIFAR100-IDN with various noise rates are plotted.
4.4 Label Correction Accuracy
We tested the label correction accuracy CIFARIO-IDN and CIFARIO0-IDN, examining how varying noise rates in {0.2, 0.3, 0.4, 0.45, 0.5) impacts the label correction performance during the T" noise correction steps on the training set using the trained policy function. The results are presented Figure 2] The DIFARIO-IDE. dataset reveals a clear distinction in label correction accuracy improvement across different noise rates; lower noise levels (0.2, 0.3, 0.4) exhibit rapid convergence, achieving over 90% accuracy by T" = Even with the highest noise rate (0.50) our method yields 90% accuracy by 7" 10, indicating
approach's 's robustness to high noise rates. On the more complex CIFARIO0 100-IDN dataset, our label correction strategy adeptly handles the increased class count and intrinsic dataset complexities. albeit with slightly lower accuracies. This is consistent with expectations given the heightened difficulty of the task. Achieving around 90% accuracy in lower noise conditions and maintaining over 79% accuracy even in high noise settings on CIFAR 100-IDN showcases RLNLC's adaptability.
5 Conclusion
In this paper, we innovatively formulated noisy label correction as reinforcement learning problem and developed a novel RL approach, named RLNLC, to address the problem of learning with noisy labels. RLNLC integrates stochastic policy-driven actions for label correction with a carefully crafted reward function that fosters both label consistency and noisy label alignment. Its capability to make sequential, non-myopic decisions is well-suited for handling label noise in complex environments, thereby enhancing the robustness of label corrections. Experiments are conducted on multiple benchmark datasets, and the experimental results demonstrated that RLNLC consistently outperforms existing state-of-the-e methods. These findings underscore the potential of reinforcement learning to transform the landscape of learning in noisy environments.
10

---

References
[1] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., "Imagenet large scale visual recognition challenge," Interna- tional journal of computer vision (IJCV), 2015.
[2] K. He, X. Zhang. S. Ren, and J. Sun, "Deep residual learning for image recognition," in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[3] J. D. M.-W. C. Kenton and L. K. Toutanova, "Bert: Pre-training of deep bidirectional transform- ers for language understanding," in Proceedings of naacL-HLT, 2019.
[4] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," JEEE Signal processing magazine, 2012.
[5] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, "Understanding deep learning (still) requires rethinking generalization," Communications of the ACM, 2021.
[6] Y. Yao, T. Liu, M. Gong, B. Han, G. Niu, and K. Zhang, "Instance-dependent label-noise learning under structural causal model," Advances in Neural Information Processing Systems (NeurIPS), 2021.
[7] X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama, "Part-dependent label noise: Towards instance-dependent label noise." Advances in Neural Information Processing Systems (NeurIPS), 2020.
[8] A. Berthon, B. Han, G. Niu, T. Liu, and M. Sugiyama, "Confidence scores make instance- dependent label-noise learning possible," International conference on machine learning (ICML), PMLR, 2021.
[9] D. Cheng, T. Liu, Y. Ning, N. Wang, Han, G. Niu, Gao, and M. Sugiyama, "Instance- dependent label-noise learning with manifold-regularized transition matrix estimation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[10] L. Jiang. Z. Zhou, T. Leung. L.-J. Li, and L. Fei-Fei, "Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels," in International conference on machine learning (ICML), 2018
[11] B. Han, Q. Yao, X. Yu, G. NIu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, "Co-teaching: Robust training of deep neural networks with extremely noisy labels," in Conference on Neural Information Processing Systems (NeurIPS), 2018.
[12] X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama, "How does disagreement help generalization against label corruption?," in International conference on machine learning (ICML), PMLR, 2019,
[13) Y. Wang, W. Liu, X. Ma, Bailey, H. Zha, L.. Song, and S.-T. Xia, "Iterative learning with open-set noisy labels." in Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), 2018
[14] S. Zheng, P. Wu, A. Goswami. M. Goswami, D. Metaxas, and C. Chen, "Error-bounded correction of noisy labels," in International Conference on Machine Learning (ICML), PMLR. 2020.
[15] Y. Zhang, S. Zheng, P. Wu. M. Goswami, and C. Chen, "Learning with feature-dependent label noise: A progressive approach," in International Conference on Learning Representations (ICLR), 2021.
[16] 7. Zhang and M. Sabuneu, "Generalized cross entropy loss for training deep neural networks with noisy labels." Advances in neural information processing systems (NeurIPS), 2018.
11

---

[17] J. Li, R. Socher, and S. C. Hoi, "Dividemix: Learning with noisy labels as semi-supervised learning," in International Conference on Learning Representations (ICLR), 2020.
[18] R. Cordeiro, R. Sachdeva, V. Belagiannis, I. Reid, and G. Carneiro, "Longremix: Robust learning with high confidence samples in noisy label environment," Pattern recognition, 2023.
[19] Liu and D. Tao, "Classification with noisy labels by importance reweighting," IEEE Transac- tions on pattern analysis and machine intelligence, 2015.
[20] E. Malach and S. Shalev-Shward "Decoupling"
when to update" from" how to update"." Advances neural information processing systems (NeurIPS), 2017.
[21] Y. Chen, X. Shen, X. Hu, and J. A. Suykens, "Boosting co-teaching with compression regularization for label noise," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[22] K.-H. Lee, X. He, L. Zhang, and L. Yang, "Cleannet: Transfer learning for scalable image classifier training with label noise," in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
[23] D. Patel and P. Sastry, "Adaptive sample selection for robust learning under label noise," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
[24] C. Feng, G. Tzimiropoulos, and I. Patras, "Ssr: An efficient and robust framework for learning with unknown label noise," in 33rd British Machine Vision Conference 2022 (BMVC), 2022.
[25] Y. Li, Y. Chen, X. Yu, D. Chen, and X. Shen, "Sure: Survey recipes for building reliable and robust deep networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[26] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
[27] S. Amin, M. Gomrokchi, H. Satija, H. Van Hoof, and D. Precup, "A survey of exploration methods in reinforcement learning," arXiv preprint -XXx:2209.00117, 2021.
[28] J. C. Balloch, R. Bhagat, G. Zollicoffer, R. Jia, J. Kim, and M. O. Riedl, "Is exploration all you need? effective exploration characteristics for transfer in reinforcement learning," arXiv preprint 24XXv:240040223333 2024.
[29] T. M. Moerland, Broekens, and C. M. Jonker, "A framework for reinforcement learning and planning," arXiv preprint arXiv:2006. 15009, vol. 127, 2020.
[30] R. Xu, J. Bhandari, D. Korenkevych, F. Liu, Y. He, A. Nikulkov, and Z. Zhu, "Optimizing long-term value for auction-based recommender systems via on-policy reinforcement learning," in Proceedings of the 17th ACM Conference on Recommender Systems, pp. 955-962, 2023.
[31] J. Kober, J. A. Bagnell, and J. Peters, "Reinforcement learning in robotics: A survey," The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238-1274, 2013.
[32] D. Lee, H. Seo, and M. W. Jung, "Neural basis of reinforcement learning and decision making," Annual review of neuroscience, vol. 35, no. 1, pp. 287-308, 2012.
[33] F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg. and D. Amodei, "Deep reinforce- ment learning from human preferences." Advances in Neural Information Processing Systems (NeurIPS), 2017.
[34] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. "Fine-tuning language models from human preferences," arXiv preprint 1XX.:1999.005933 2019.
[35] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, "Learning to summarize with human feedback," Advances in Neural Information Processing Systems (NeurIPS), 2020.
12

---

[36] Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu. C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee. "Dpok: Reinforcement learning for fine-tuning to-iage diffusion models," Advances in Neural Information Processing Systems (NeurIPS), 2024.
[37] M. Heidari, H. Zhang, and Y. Guo, "Reinforcement learning- semi-supervised learning," Advances in Neural Information Processing Systems (NeurIPS), 2024.
[38] J. Wu, Y. Wang. L. Li, Z. Fan, and T. Xue, "Goal conditioned reinforcement learning for photo finishing tuning," in The ehirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[39] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, "Policy gradient methods for rein- forcement learning with function approximation." Advances in Neural Information Processing Systems (NeurIPS), 1999.
[40] V. Konda and J. Tsitsiklis, "Actor-critic algorithms," Advances in Neural Information Processing Systems (NeurIPS), 1999.
[41] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems (NeurIPS), 2022.
[42] G. Tesauro el al., "Temporal difference learning and td-gammon," Communications of the ACM, 1995.
[43] H. Zhang, "mixup: Beyond empirical risk minimization," arXiv preprint arXiv:1710.09411 2017.
[44] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, "Making deep neural networks robust to label noise: A loss correction approach," " in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
[45] J. Han, P. Luo, and X. Wang. "Deep self-learning from noisy labels," in Proceedings of the JEEE/CVF international conference on computer vision (ICCV), 2019.
[46] A. Krizhevsky, G. Hinton, el al., "Learning multiple layers of features from tiny images," " 2009.
[47] H. Song, M. Kim, and J.-G. Lee, "Selfie: Refurbishing unclean samples for robust deep learning." International conference on machine learning (ICML), 2019.
[48] Lv, J. Liang. S. Li, Zang, C. H. Liu, Z. Wang, and Liu, "Causality inspired representation learning for domain generalization," in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Yes, the abstract and introduction accurately reflect the paper's contributions and scope. They clearly state the central idea of formulating noisy label correction
reinforcement learning problem and propose novel actor-critic method (RLNLC) that dynamically corrects labels through a learned policy guided by tailored reward function. These claims align with the detailed methodology and are validated by the experimental results, which demonstrate consistent improvements over existing approaches.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Limitation Section in Appendix C Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper includes detailed descriptions of the experimental setup and parame- ters. Additionally, the paper includes algorithms to aid reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
The code is not shared.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
Codeeobmiiiie SanSSuubissss sebbbbisssiooi ssisssoicccyyy for more details. 7/nips cc/ wtc/publicc/ /tsp://guudeeeeeeeuubiaiiiii
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yes)
Justification:
See Section 4.1 for details about training settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes)
Justification: All experiments were conducted five times, with the average results and standard deviations reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information about computer resources are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Broader Impacts Section in Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in our research are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer[[NA) [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs where used in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
20

---

17.5
71
Accuracy
AccuI ocy
-2
281
50
500
300
200
520
2034
96
67
11.
50
350
150
200
21
Figure 3: Sensitivity analysis for four papper-arammters, h, No. A, and T, on CIFARIO0-IDN with 0.50 noise rate.
A Hyper-parameter Sensitivity Analysis
We conducted a detailed sensitivity analysis of four key hyperparamettrss within the RLNLC frame- work on CIFAR IO0-IDN: (1) k b-bhyperparameter for number of neighbors k-nearest neighbors, (2) No hy-hhyperrarmmtter for the number of bins for state encoding, (3) -rade-o--off coefficient for RNLA. and (4) T trajectryy length in the training procedure, revealing their influence on the performance. The experimental results for various hyper-parameter values plotted in Figure
We can that the test accuracy performance improves increases from 10, highlighting the benefits of a broader contextual basis for label refinement. However, further increases beyond 10 yield diminishing returns, with accuracy plateauing around 74.9% for values of and 30. This suggests that while larger contextual window is beneficial up to a point, excessively wide windows offer no extra benefits, likely due to less informative examples. The granularity of state encoding, controlled by No. shows similar trend where increasing N, from 10 to 100 leads improved performance. Further increases up to 500 continue to marginally enhance the model's accuracy, This improvement shows that finer state space discretization enables gains diminish with increasing Ng. The optimal range for is between 0.4 and 0.6, achieving a peak accuracy of 74.9%. Outside this range, effectiveness drops, indicating imbalance that may reduce learning efficiency. Varying the training trajectory length T shows an initial increase in performance as T increases from to 10. However, extending T further to 25 leads decline in performance. This reduction may stem from overfitting to noisy data or excessive refinement causing label drift.
precise state encoding, though
B Computer resources
Our experiments were performed using computing systems equipped with 8-core Intel Core proces- sors, 64 GB of system memory, and NVIDIA GeForce RTX 3060 GPUs, each providing 12 GB of dedicated video memory.
C Limitation
RLNLC targets the standard seamiingwwtt-noisyy-e tribution, without explicitly modeling domain shift or class imbalance. Nonetheless, the proposed formulation general and flexible enough to be extended to more complex scenarios. These exten- sions would build upon the core principles of RLNLC, and thus represent natural and promising directions for future work.
setting under single, stationary data dis-
D Broader Impacts
This work proposes a reinforcement learning-based framework for learning with noisy labels, which can improve the reliability and robustness of machine learning models trained imperfect datasets. By reducing the reliance on clean labels, RLNLC can make it more feasible to leverage large-scale datasets domains where annotation is expensive error-prone, such as medical imaging, remote sensing, crowd-sourced labeling. However, automatic label correction methods also introduce potential risks, such as reinforcing existing biases in the data or misclassifying minority or rare samples if not carefully validated. Future deployments should therefore include safeguards, such as human-in numan-in-the-lon in-the-loop verification or fairness audits, ensure responsible use.
21


Reviews:
[{"review_id": "UnGejcv4FD", "reviewer": "Reviewer_uqZQ", "summary": "This paper presents a framework for correcting noisy labels through a reinforcement learning approach. It incorporates two reward functions to assess the quality of actions: a label consistency reward function and a noisy label alignment reward function. Experimental results on four public datasets demonstrate the effectiveness of the proposed method, showing marginally better performance compared to other approaches.", "questions": "Please see my questions in the weakness section. My main concerns are the constraint on the noise levels and whether the proposed method can be used in multi-label classification tasks.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Addressing the challenge of learning from noisy labels is a critical topic in the machine learning community. This paper introduces a novel approach by employing a reinforcement learning framework to correct noisy labels.\n\n2. The paper proposes two informative reward functions that promote label correction by leveraging local data structures and clean labels.\n\n3. Experimental results on four public datasets demonstrate that the proposed method outperforms the compared approaches.\n\nWeaknesses:\n1. More details are needed to understand Equation (3), specifically why it only considers the classes where the new prediction is equal to or greater than the original predicted class. What about cases where the new prediction has a smaller probability?\n\n2. Regarding the feature extraction network ( f_\\theta ), it is unclear which data was used to pretrain the model. Were they label-noisy data? Additionally, information on the pretraining performance, such as accuracy, is needed.\n\n3. The proposed method is only evaluated on multi-class classification problems, where a single label is predicted for a given image. Could you explain whether the proposed method can be applied to multi-label classification problems, where multiple labels are predicted?\n\n4. It is important to consider whether there is a constraint on the noise level that could negatively impact the proposed method's performance. Although the experiment section shows different levels of noise, it would be beneficial to indicate the threshold beyond which the proposed method fails to handle label noise effectively.", "was_revised": true, "final_justification": "Based on the authors rebuttal, which solves my concern in this paper, I keep my original rating.", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n**About Eq (3)**: Equation (3) is intentionally designed to focus on classes whose predicted probability from the k-nearest neighbor aggregation is equal to or greater than that of the original label because these are the classes that actively compete with the current label in terms of likelihood. Our goal is to estimate the likelihood that the current label is incorrect, and classes with lower predicted probabilities are irrelevant to that decision, as they do not offer sufficient evidence to challenge the existing label. Including classes with lower scores would dilute the signal and introduce noise into the correction probability. By normalizing over only the competitive classes, the policy precisely quantifies the relative disagreement strength and produces sharper, more meaningful gradients for learning.\n\n**About pretraining $f_\\theta$** : The feature extractor $f_\\theta$ is initially pretrained on the same label-noisy dataset using standard cross-entropy (CE) loss, without access to any clean supervision. This design choice reflects a realistic and widely adopted assumption in noisy-label learning, where no trusted labels are available at initialization. Despite the presence of label noise, this pretraining stage enables $f_\\theta$ to learn useful semantic representations that serve as a strong foundation for the subsequent reinforcement learning-based label correction policy. The performance of this pretrained model matches the 'CE' baseline reported in our experimental tables. For clarity and convenience, we include those results below:\n\nPretraining Accuracy of $f_\\theta$ (CE Baseline) on different datasets and noise rates\n\n| Dataset        | Noise Rate | Accuracy (%) |\n|----------------|------------|---------------|\n| CIFAR10-IDN    | 0.20       | 75.8          |\n|                | 0.30       | 69.2          |\n|                | 0.40       | 62.5          |\n|                | 0.45       | 51.7          |\n|                | 0.50       | 39.4          |\n| CIFAR100-IDN   | 0.20       | 30.4          |\n|                | 0.30       | 24.2          |\n|                | 0.40       | 21.5          |\n|                | 0.45       | 15.2          |\n|                | 0.50       | 14.4          |\n| ANIMAL-10N     | --         | 79.4          |\n| Food-101N      | --         | 81.6          |\n\n\n\n\n**About multi-label prediction:** Our proposed method is specifically designed to address the standard learning with noisy labels setting under multi-class classification, where each instance is associated with a single label. Extending RLNLC to the multi-label setting, where multiple, potentially noisy labels may be associated with a single instance, introduces new challenges, such as modeling inter-label dependencies and defining appropriate action and reward formulations for partially correct label sets. While our current formulation does not directly support this scenario, we view multi-label noisy label correction as a promising direction for future work..\n\n**About constraints on the noise level:** There is no fixed or universal failure threshold for RLNLC, as its performance depends not just on the overall noise rate, but on the interplay between class separability and the local structure of label noise, specifically, how noise affects neighborhood consistency in feature space.  RLNLC leverages local agreement in the embedding space and optimizes label corrections via long-term reward feedback. This enables it to remain effective as long as a non-trivial fraction of each sample’s neighbors retain correct labels. Empirically, we observe that even at 90% CCN on CIFAR-100 in Table 5, RLNLC degrades gracefully and continues to outperform strong baselines."}, {"review_id": "qnMgGjrB6t", "reviewer": "Reviewer_FMfs", "summary": "This paper introduces RLNLC, a novel reinforcement learning framework for noisy label correction, modeling the problem as a Markov Decision Process (MDP). By defining a state space of data and labels, an action space of label corrections, and a reward function combining label consistency and subset alignment, RLNLC uses an actor-critic method to learn a policy network for iterative label cleaning. Extensive experiments on benchmark datasets demonstrate its effectiveness.", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The RL-based approach to deal with label correction is a fresh idea, offering dynamic, sequential decision-making that traditional methods (e.g., sample selection or loss regularization) lack.\n\n2. This work develops a customized policy function grounded in a deep representation network. \n\n3. This work designs an effective reward function. Through k-nearest neighbor prediction mechanisms, it promotes efficient label correction by matching local data structures with reliable clean labels.\n\n\nWeaknesses:\n1. In Line 56, the paper states that its designed policy function determines label correction actions by considering the state of the entire dataset. However, as observed in Section 3.2.2, it appears to only consider data within a single batch or subset, which may constitute an overclaim. If my understanding is incorrect, please provide further clarification.\n\n2. The paper does not compare with baselines under another noise setting (CCN). If such comparisons are omitted, it is necessary to add explanations, preferably highlighting that the algorithm is specifically designed for the more practical IDN noise setting.\n\n3. In the paper, the figures need to be polished up. Figure 1 is too small and could be enlarged. The font in Figure 2 is slightly distorted, and the figure appears not to be in vector format. It is recommended to use vector graphics for a more professional look.\n\n4. The iterative RL training and multi-step label correction may incur higher computational costs than non-iterative methods (e.g., CleanNet), though hardware details (e.g., training time on RTX 3060) are missing.\n\n5. The related work section discusses and cites too few works.", "was_revised": true, "final_justification": "The author's response adequately addresses the raised concerns, and their work, with its innovative RL-based label correction approach and well-designed components, shows promise, warranting a positive assessment.", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n**About state**: The policy **does in fact operate on a global view of the dataset**. The k-NN aggregation in Section 3.2.2  is performed globally, leveraging the full feature space of the entire dataset at each policy step. This ensures that the correction probabilities incorporate the relative similarity of each sample to all others, not just the local batch context. Furthermore, to make the global structure tractable and informative for the policy network, we introduce a **state encoding mechanism (in Section 3.2.4)** that summarizes the dataset as a fixed-length representation via a binning strategy . This allows the policy to condition on a **compressed but global encoding of the full dataset state**, not just a partial view.\n\n**About CCN noise:** The comparison results under class-conditional noise (CCN) are provided in **Appendix C, Table 5** of our manuscript, demonstrating that RLNLC maintains strong performance even in this simpler noise setting.\n\n**About Figure**: We thank the reviewer for the helpful suggestion regarding the figure. We will incorporate the feedback and improve the figure in the revision.\n\n**About Related works**: We have discussed the most relevant and state-of-the-art works in learning with noisy labels, as well as the application of reinforcement learning in related problems. We appreciate the feedback and will include additional related works in the revision.\n\n**About training time**: Our method (RLNLC) has a training time lower than most SOTA methods. The table shows the training time in seconds per epoch on CIFAR100-IDN using an RTX 3060 GPU.\n\n| Method         | Time  |\n|----------------|-------|\n| SSR            | 51.37 |\n| DivideMix      | 119.91|\n| LongRemix      | 140.80|\n| RLNLC (Ours)   | 59.51 |\n\n\n."}, {"review_id": "vmjbAszRjQ", "reviewer": "Reviewer_Kkod", "summary": "The paper proposes a framework that learns a policy for noisy label correction via reinforcement learning, with comprehensive formulations of the problem setup, state space, action space, reward functions, and an actor-critic method that is used to learn the desired policy function. The proposed framework addresses several drawback of prior works. Extensive experiments are also conducted to show the effectiveness of it.", "questions": "1. Line 33. The paper claims that the prior works lack the ability to learn from long-term consequences. What are considered \"lone-term consequences\" in this context, and how does the proposed method in this paper addresses the problem?\n2. Line 42-43. The policy function separates the training set into a \"clean\" subset and a \"noisy\" subset. Here by saying \"clean\" and \"noisy\", does it mean whether the labels in the subsets are clean or noisy by nature, or does it mean whether the labels in the subsets are \"predicted to be clean or noisy by the reinforcement model\"?\n3. Does the proposed RLNLC applies only on IDN or both IDN and CCN?\n4. From my understanding of the framework of RLNLC, $f_\\theta$ is to be trained, while $f_\\omega$ is fixed for computing the reward function? If so, then what guarantees the performance of the fixed model itself? What guarantees that the reward function computed using $f_\\omega$ indeed reflects the true \"goodness\" of the corrected labels?\n5. Line 226. I suppose that by saying \"trade-off\" you are implying that if one of the sub-reward functions gets better, the other one will get worse? If so, can you explain this trade-off more concretely or intuitively?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Clear chain of logics.\n2. The experiments are well conducted and sufficiently detailed。\n3. The review on the related works is comprehensive.\n4. The formulation of the proposed method is thorough and detailed.\nWeaknesses:\n1. The fixed backbone model itself doesn't have a performance guarantee.", "was_revised": true, "final_justification": "Most of my questions have been addressed. I have maintained my original rating.", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n**About long-term consequences**: In the context of learning with noisy labels, \"long-term consequences\" refer to the downstream effects that early label correction decisions can have on the final performance of the prediction model, especially under instance-dependent noise, where errors can compound over time. Most prior works, such as Co-teaching [1] and DivideMix [2], rely on short-term heuristics (e.g., small-loss filtering) without accounting for how corrections influence future learning dynamics. In contrast, RLNLC frames label correction as a sequential decision-making problem using reinforcement learning. By modeling corrections as actions and using a reward function that evaluates dataset-wide label consistency and inter-subset alignment, RLNLC explicitly learns a policy that optimizes over future label quality and model behavior, not just immediate improvements. The actor-critic framework allows the system to assess whether a series of corrections improves the overall learning trajectory, thereby mitigating the risk of early missteps cascading into degraded final performance. \n\n**About clean and noisy in policy**: The terms \"clean\" and \"noisy\" refer to the policy’s predicted status of each sample’s label during training, i.e., whether the reinforcement learning policy believes the label is likely correct (clean) or incorrect (noisy). Since the true label correctness is unknown during learning, these designations reflect the policy’s internal belief rather than the ground truth. We will clarify this distinction more explicitly in the revision.\n\n**About noise type:** Our method is general and applies to class-conditional noise (CCN) as well as IDN. To demonstrate this, we include CCN results in Appendix C, Table 5, where RLNLC shows competitive performance, further confirming its robustness across different noise models.\n\n**About reward function network $f_\\omega$**:  The role of $f_\\omega$ is to provide stable embeddings for the reward function. Using a separate, fixed model avoids coupling the label correction policy with the reward signal, reducing potential bias. We emphasize three key points:\n\n1. **Empirical Competence of $f_\\omega$**:  \n   Although trained on noisy labels, $f_\\omega$ produces sufficiently informative representations, as reflected in its performance (CE rows in Tables 1–3 in our manuscript; for convince also included below). These features enable meaningful similarity estimation via k-NN, allowing the policy to reason about label quality based on local structure.\nAccuracy of $f_\\omega$ (CE Baseline)\n\n| Dataset        | Noise Rate | Accuracy (%) |\n|----------------|------------|---------------|\n| CIFAR10-IDN    | 0.20       | 75.8          |\n|                | 0.30       | 69.2          |\n|                | 0.40       | 62.5          |\n|                | 0.45       | 51.7          |\n|                | 0.50       | 39.4          |\n| CIFAR100-IDN   | 0.20       | 30.4          |\n|                | 0.30       | 24.2          |\n|                | 0.40       | 21.5          |\n|                | 0.45       | 15.2          |\n|                | 0.50       | 14.4          |\n| ANIMAL-10N     | --         | 79.4          |\n| Food-101N      | --         | 81.6          |\n\n\n\n2. **Policy Learning with Feedback**:  \n   The policy is not blindly steered by $f_\\omega$’s outputs. Instead, it learns through reinforcement feedback to calibrate its reliance on neighborhood signals. The reward captures both short- and long-term consistency, encouraging corrections that lead to better downstream fine-tuning. This dynamic loop allows RLNLC to iteratively refine labels beyond the limitations of the fixed model.\n\n3. **Empirical Justification**:  \n   The table below shows that using a separate $f_\\omega$ for the reward yields better performance than reusing the policy model $f_\\theta$. Results are reported as test accuracy (with standard deviation) on CIFAR-100 IDN.\n\n\n| Method             | 0.20           | 0.30          | 0.40          | 0.45              | 0.50              |\n|--------------------|----------------|---------------|---------------|-------------------|-------------------|\n| **RLNLC (Ours)**   | **80.5** (0.7) | **80.1** (0.7)| **78.5** (0.8)| **77.2** (0.8)    | **74.7** (0.9)    |\n| - w $f_\\theta$   | 78.4 (0.4)     | 76.9 (0.8)    | 75.2 (0.6)    | 74.3 (0.8)        | 72.8 (0.9)        |\n\n\n\n\n\n**About trade-off**: We did not mean a zero-sum relationship between the reward terms. By “trade-off,” we refer to a **weighting hyperparameter** that balances the contributions of $\\mathcal{R}\\_{\\text{NLA}}$ and $ \\mathcal{R}\\_{\\text{LCR}}$ in the total reward. The effect of different values of $\\lambda$ is illustrated in **Figure 3, Appendix D** of our manuscript. We will clarify this in the revision.\n\n\n[1] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching:\nRobust training of deep neural networks with extremely noisy labels,” NeurIPS, 2018.\n\n[2] J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning”, ICLR, 2020"}, {"review_id": "6rMqhs6hEM", "reviewer": "Reviewer_un5g", "summary": "This paper proposes a novel framework for addressing the problem of label noise by formulating label correction as a reinforcement learning problem. The authors introduce the RLNLC method, which constructs a Markov Decision Process (MDP) that includes a state space, action space, and reward mechanism. A policy network is trained using an actor-critic framework to correct potentially mislabeled data. For the reward function, the authors design two submodules—label consistency and clean-label alignment—to guide the policy network in generating effective label correction sequences. Experiments conducted on multiple benchmark datasets demonstrate the strong generalization ability and robustness of RLNLC.", "questions": "1. Noted that the k-means method is used in the reward function. During training state, there may be cases where the number of samples with certain labels in a batch is fewer than k. In such situations, the computed reward may deviate from the intended objective, potentially leading to suboptimal guidance for the policy network. Will this method be affected by this situation?\n\n2. Could the repeated correction of labels introduce new errors or cause the model to diverge from plausible semantics (especially in datasets with inherent label ambiguity)?\n\n3. Can the authors provide a comparison of the number of parameters and computational cost between RLNLC and other baseline methods?", "limitations": "yes", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The idea of casting noisy label correction as a reinforcement learning problem is both original and compelling.\n2. The MDP elements states, actions, and rewards are clearly defined and grounded in label consistency principles. Key components such as the reward terms and randomized initialization are empirically validated.\nWeaknesses\n1. While the method is empirically effective, there is little theoretical analysis of convergence properties, stability of the policy under label noise, or sample complexity in the RL setting.\n2. The actor-critic framework, kNN-based reward computation, and multiple label correction steps may introduce significant computational costs.", "was_revised": true, "final_justification": "For the first issue, the authors offered a clear and thorough explanation regarding the effect of the hyper parameter k on the model, including how different values of k impact the reward function. This resolved my main concern.\n\nFor the second issue, the authors responded from multiple perspectives and noted that no such problem was observed in their experiments. This helped alleviate the concern I raised.\n\nFor the third issue, the authors responded directly with runtime comparisons against different baseline methods, highlighting the efficiency of their approach.\n\nMy concerns have been adequately addressed. Since the first issue was my primary focus, and the authors’ response was meaningful and improved my perception of the paper’s clarity, I have reassessed the manuscript. Additionally, the response to the third issue strengthened my view of the paper’s significance by demonstrating a clear advantage in computational efficiency.\n\nTaking all aspects into account, I have adjusted my score accordingly.", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n**About insufficient samples**: We clarify that the reward function uses k-NN, not k-means. Additionally, the state representation in RLNLC is built from the **entire dataset**, not just a mini-batch, so k-NN is computed globally over all feature embeddings. This ensures that each sample has access to a sufficiently large context, thereby avoiding issues arising from limited local information.\n\n\n**About the effect of repeated corrections**:RLNLC mitigates this risk of diversion through several design choices. First, label corrections are guided by a stochastic policy trained to maximize long-term reward, which explicitly encodes both local consistency and alignment with confidently clean labels via k-NN mechanisms. This ensures that updates are not arbitrary but grounded in dataset-wide structure. Second, corrections are applied softly and probabilistically, avoiding hard deterministic overwrites that could amplify noise. Finally, the actor-critic framework evaluates the downstream impact of corrections via reward feedback, discouraging degenerate or unstable policies that would introduce semantic drift. Empirically, we observe stable convergence and improved performance across diverse datasets, indicating the model does not diverge but instead learns semantically coherent label refinements.\n\n**About number of parameters**:  We emphasize that RLNLC uses the exact same backbone architecture as all baselines, ensuring a fair comparison in terms of model capacity. The only additional component is a lightweight critic network composed of a few MLP layers, which introduces roughly 3K additional trainable parameters, negligible relative to modern backbones such as ResNet, which has millions of parameters. In contrast, several widely used LNL baselines introduce substantially more overhead. For example, Co-teaching [1] and DivideMix [2] both maintain two separate networks during training, effectively doubling the number of backbone parameters. Despite its minimal overhead, RLNLC delivers significantly stronger results, showing that performance gains stem not from model size but from the structured, reward-driven label correction process. \n\n**About training time**: Our method (RLNLC) has a training time lower than most SOTA methods. The table shows the training time in seconds per epoch on CIFAR100-IDN using an RTX 3060 GPU.\n\n| Method         | Time  |\n|----------------|-------|\n| SSR            | 51.37 |\n| DivideMix      | 119.91|\n| LongRemix      | 140.80|\n| RLNLC (Ours)   | 59.51 |\n\n\n[1] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching:\nRobust training of deep neural networks with extremely noisy labels,” NeurIPS, 2018.\n\n[2] J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning”, ICLR, 2020"}]

Author rebuttal:

Meta-review: This paper proposes a reinforcement learning framework for noisy label correction, modeling the task as an MDP with well-defined state, action, and reward components. The method features a novel reward design based on label consistency and neighborhood alignment, and is trained via an actor-critic strategy.

The paper received mostly positive reviews. Reviewers appreciated the originality of modeling label correction as a sequential decision problem, the clarity in defining the MDP components, and the experimental validation. All reviewers acknowledged the potential impact of 

Most concerns were adequately addressed during the discussion phase, and the overall contribution remains technically sound, original, and practically relevant. I recommend acceptance.
Decision: Accept (poster)