Title: Learning to Clean: Reinforcement Learning for Noisy Label Correction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning with Noisy Labels

Reinforcement Learning
Abstract: The challenge of learning with noisy labels is significant in machine learning, as it can severely degrade the performance of prediction models if not addressed properly. This paper introduces a novel framework that conceptualizes noisy label correction as a reinforcement learning (RL) problem. The proposed approach, Reinforcement Learning for Noisy Label Correction (RLNLC), defines a comprehensive state space representing data and their associated labels, an action space that indicates possible label corrections, and a reward mechanism that evaluates the efficacy of label corrections. RLNLC learns a deep feature representation based policy network to perform label correction through reinforcement learning, utilizing an actor-critic method. The learned policy is subsequently deployed to iteratively correct noisy training labels and facilitate the training of the prediction model. The effectiveness of RLNLC is demonstrated through extensive experiments on multiple benchmark datasets, where it consistently outperforms existing state-of-the-art techniques for learning with noisy labels.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for addressing the problem of label noise by formulating label correction as a reinforcement learning problem. The authors introduce the RLNLC method, which constructs a Markov Decision Process (MDP) that includes a state space, action space, and reward mechanism. A policy network is trained using an actor-critic framework to correct potentially mislabeled data. For the reward function, the authors design two submodules—label consistency and clean-label alignment—to guide the policy network in generating effective label correction sequences. Experiments conducted on multiple benchmark datasets demonstrate the strong generalization ability and robustness of RLNLC.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The idea of casting noisy label correction as a reinforcement learning problem is both original and compelling.\n2. The MDP elements states, actions, and rewards are clearly defined and grounded in label consistency principles. Key components such as the reward terms and randomized initialization are empirically validated.\nWeaknesses\n1. While the method is empirically effective, there is little theoretical analysis of convergence properties, stability of the policy under label noise, or sample complexity in the RL setting.\n2. The actor-critic framework, kNN-based reward computation, and multiple label correction steps may introduce significant computational costs.\n\nQUESTIONS:\n1. Noted that the k-means method is used in the reward function. During training state, there may be cases where the number of samples with certain labels in a batch is fewer than k. In such situations, the computed reward may deviate from the intended objective, potentially leading to suboptimal guidance for the policy network. Will this method be affected by this situation?\n\n2. Could the repeated correction of labels introduce new errors or cause the model to diverge from plausible semantics (especially in datasets with inherent label ambiguity)?\n\n3. Can the authors provide a comparison of the number of parameters and computational cost between RLNLC and other baseline methods?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework that learns a policy for noisy label correction via reinforcement learning, with comprehensive formulations of the problem setup, state space, action space, reward functions, and an actor-critic method that is used to learn the desired policy function. The proposed framework addresses several drawback of prior works. Extensive experiments are also conducted to show the effectiveness of it.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Clear chain of logics.\n2. The experiments are well conducted and sufficiently detailed。\n3. The review on the related works is comprehensive.\n4. The formulation of the proposed method is thorough and detailed.\nWeaknesses:\n1. The fixed backbone model itself doesn't have a performance guarantee.\n\nQUESTIONS:\n1. Line 33. The paper claims that the prior works lack the ability to learn from long-term consequences. What are considered \"lone-term consequences\" in this context, and how does the proposed method in this paper addresses the problem?\n2. Line 42-43. The policy function separates the training set into a \"clean\" subset and a \"noisy\" subset. Here by saying \"clean\" and \"noisy\", does it mean whether the labels in the subsets are clean or noisy by nature, or does it mean whether the labels in the subsets are \"predicted to be clean or noisy by the reinforcement model\"?\n3. Does the proposed RLNLC applies only on IDN or both IDN and CCN?\n4. From my understanding of the framework of RLNLC, $f_\\theta$ is to be trained, while $f_\\omega$ is fixed for computing the reward function? If so, then what guarantees the performance of the fixed model itself? What guarantees that the reward function computed using $f_\\omega$ indeed reflects the true \"goodness\" of the corrected labels?\n5. Line 226. I suppose that by saying \"trade-off\" you are implying that if one of the sub-reward functions gets better, the other one will get worse? If so, can you explain this trade-off more concretely or intuitively?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RLNLC, a novel reinforcement learning framework for noisy label correction, modeling the problem as a Markov Decision Process (MDP). By defining a state space of data and labels, an action space of label corrections, and a reward function combining label consistency and subset alignment, RLNLC uses an actor-critic method to learn a policy network for iterative label cleaning. Extensive experiments on benchmark datasets demonstrate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The RL-based approach to deal with label correction is a fresh idea, offering dynamic, sequential decision-making that traditional methods (e.g., sample selection or loss regularization) lack.\n\n2. This work develops a customized policy function grounded in a deep representation network. \n\n3. This work designs an effective reward function. Through k-nearest neighbor prediction mechanisms, it promotes efficient label correction by matching local data structures with reliable clean labels.\n\n\nWeaknesses:\n1. In Line 56, the paper states that its designed policy function determines label correction actions by considering the state of the entire dataset. However, as observed in Section 3.2.2, it appears to only consider data within a single batch or subset, which may constitute an overclaim. If my understanding is incorrect, please provide further clarification.\n\n2. The paper does not compare with baselines under another noise setting (CCN). If such comparisons are omitted, it is necessary to add explanations, preferably highlighting that the algorithm is specifically designed for the more practical IDN noise setting.\n\n3. In the paper, the figures need to be polished up. Figure 1 is too small and could be enlarged. The font in Figure 2 is slightly distorted, and the figure appears not to be in vector format. It is recommended to use vector graphics for a more professional look.\n\n4. The iterative RL training and multi-step label correction may incur higher computational costs than non-iterative methods (e.g., CleanNet), though hardware details (e.g., training time on RTX 3060) are missing.\n\n5. The related work section discusses and cites too few works.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for correcting noisy labels through a reinforcement learning approach. It incorporates two reward functions to assess the quality of actions: a label consistency reward function and a noisy label alignment reward function. Experimental results on four public datasets demonstrate the effectiveness of the proposed method, showing marginally better performance compared to other approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Addressing the challenge of learning from noisy labels is a critical topic in the machine learning community. This paper introduces a novel approach by employing a reinforcement learning framework to correct noisy labels.\n\n2. The paper proposes two informative reward functions that promote label correction by leveraging local data structures and clean labels.\n\n3. Experimental results on four public datasets demonstrate that the proposed method outperforms the compared approaches.\n\nWeaknesses:\n1. More details are needed to understand Equation (3), specifically why it only considers the classes where the new prediction is equal to or greater than the original predicted class. What about cases where the new prediction has a smaller probability?\n\n2. Regarding the feature extraction network ( f_\\theta ), it is unclear which data was used to pretrain the model. Were they label-noisy data? Additionally, information on the pretraining performance, such as accuracy, is needed.\n\n3. The proposed method is only evaluated on multi-class classification problems, where a single label is predicted for a given image. Could you explain whether the proposed method can be applied to multi-label classification problems, where multiple labels are predicted?\n\n4. It is important to consider whether there is a constraint on the noise level that could negatively impact the proposed method's performance. Although the experiment section shows different levels of noise, it would be beneficial to indicate the threshold beyond which the proposed method fails to handle label noise effectively.\n\nQUESTIONS:\nPlease see my questions in the weakness section. My main concerns are the constraint on the noise levels and whether the proposed method can be used in multi-label classification tasks.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}