Title: Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: computer vision

learning with noisy labels

robustness
Abstract: Recent studies indicate that deep neural networks degrade in generalization performance under noisy supervision. Existing methods focus on isolating clean subsets or correcting noisy labels, facing limitations such as high computational costs, heavy hyperparameter tuning process, and coarse-grained optimization. To address these challenges, we propose a novel two-stage noisy learning framework that enables instance-level optimization through a dynamically weighted loss function, avoiding hyperparameter tuning. To obtain stable and accurate information about noise modeling, we introduce a simple yet effective metric, termed $\textit{wrong event}$, which dynamically models the cleanliness and difficulty of individual samples while maintaining computational costs. Our framework first collects $\textit{wrong event}$ information and builds a strong base model. Then we perform noise-robust training on the base model, using a probabilistic model to handle the $\textit{wrong event}$ information of samples. Experiments on six synthetic and real-world LNL benchmarks demonstrate our method surpasses state-of-the-art methods in performance, achieves a nearly 75\% reduction in storage and computational time, strongly improving model scalability. Our code is available at https://github.com/iTheresaApocalypse/IDO.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes IDO (Instance-level Difficulty Modeling and Dynamic Optimization), a two-stage training framework for learning with noisy labels. The core contribution is a novel metric called “wrong event”, which tracks the frequency of model misclassification during training to distinguish clean, noisy, and hard samples without requiring extra hyperparameters. A Beta Mixture Model (BMM) is fitted to the distribution of wrong event scores to probabilistically model both sample cleanliness and difficulty. The method dynamically adjusts loss contributions based on these measures, achieving strong generalization performance while being computationally efficient and scalable.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper proposes instance-level dynamic loss weighting that captures both sample difficulty and label cleanliness. It\navoids the extensive hyperparameter tuning required by prior methods like DivideMix or ELR and provides clear theoretical rationale and empirical validation for the stability of the proposed metric.\n\nWeeknesses: \nIn the first stage of this framework, the paper assumes that \"prior knowledge, i.e., coarse distribution of wrong event, is obtained\". However,  the prior knowledge cannot be easily obtained in reality.\n\n In this paper, the performance improvement may be from different components/terms of the loss function. Ablation Study is missing.\n\nQUESTIONS:\nIn the caption of Figure 2, the authors mentioned \"Since wrong events are monotonically increasing based on historical statistics instead of current model prediction, when model overfits the dataset, wrong event values for all samples do not change, rather than converging to zero as loss values typically do.\". It is not clear about what is the \"wrong event values\". Why when wrong events are monotonically increasing, wrong event values do not change?\n\nLIMITATIONS:\nThe authors didn't compare the results with very recent work defencing against label noises, e.g., \"Noise attention learning: Enhancing noise robustness by gradient scaling, NeurIPS 2022\",", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of label noise learning in the lense of empirical observation. In particular, the paper proposes a metric, coined as *wrong event*, which count the number of times the model mispredict the given label after each training epoch. Such a newly-observed metric has a profound impact in identifying the clean, noisy and difficult samples. That distincs the proposed method with previous methods which may lump both the noisy and difficult samples together. Based on the *wrong event* metric, a mixture of 2 Beta distributions is inferred to divide samples into two groups, each with their own nature (i.e., clean or noisy) and their difficulty. These factors are then taken into account to design a loss function consisting of three components: clean loss, noisy loss and consistency loss between two different data augmentations. Empirical evaluation achieves state-of-the-art results compared to previous methods in several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The motivation of separating or weighting noisy and difficult samples is sound. It is actually the main weakness of the majority methods in label noise learning, which often rely on small loss hypothesis. Tackling such a problem is one of the major strengths of the paper.\n- The paper proposes a simple metric to quantify both the cleanness and difficulty of each training samples. This metric has empirically shown working well in practice.\n\n## Weaknesses\nThe paper is mainly a pure-observation based approach. There is limited justification for what is proposed in the paper, and in particular the *wrong event*. Although it has been demonstrated through several empirical results, it is unclear whether the metric is effectiveness in identifying label noise. In addition, despite being claimed with theoretical analysis, such an analysis is mostly incorrect and very limited to shed any insights into the proposed *wrong event*.\n\nThe claim of the theoretical analysis at line 132 and in the Appendix C2: $\\frac{w\\_{i}^{T + 1} - w_{i}^{T}}{w_{i}^{T}} \\in [0, 1/T]$ is incorrect. The left hand side term should be in $[0, +\\infty]$. A slightly similar concern on the loss. Since the loss is positive, $\\frac{l\\_{i}^{T + 1} - l_{i}^{T}}{l_{i}^{T}} \\in (-\\infty, +\\infty)$.\n\nQUESTIONS:\nMy main concern is the limited theoretical justification and analysis on the proposed metric *wrong event*. In its current form, it is unclear how that would result in something clean, or noisy with their own difficulty. At the end of the day, such a metric largely depends on the model used to estimate it. Then, if the model is bias, the metric is also bias, rendering its ineffectiveness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of Learning with Noisy Labels (LNL) by introducing both a metric, Wrong Event, and a framework, Instance-level Difficulty Modeling and Dynamic Optimization (IDO), and custom loss function. The main contribution is the development and usage of the Wrong Event metric. The Wrong Event metric is an indicator of label cleanliness (whether a label is correct or not), and is computed as the cumulative sum (over epochs) that a given training example is incorrectly classified, as per its as-labelled class. The idea is that easy & noisy samples will consistently be incorrectly predicted and thus have a high Wrong Event score, and that difficult & noisy (or clean) samples will have a medium Wrong Event score. With this insight, the Wrong Event metric feeds into the IDO framework as a parameter from which cleanliness and difficulty (at the per-sample-level) are estimated (using a two-component beta mixture model). The IDO framework itself works in two stages (first stage is used to generate prior distributions for the Wrong Event distributions, and the second is for actual robust model training, making use of the custom loss function and Wrong Event metric). The paper evaluates its method on several synthetically corrupted datasets (CIFAR-10, CIFAR-100, Tiny ImageNet), and real-world noisy datasets (CIFAR-100N, Clothing1M, WebVision), with compelling results. Thought classification accuracy is only marginally improved vs. existing LNL methods, the improvements seem to persist across all datasets and rates of label noise.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n\n-\tThe paper seems  to be technically sound, but some claims are not well supported:\n  - “wrong event can clearly separate noisy and clean data at all training stages” – unclear how this is demonstrated, can this be shown in fig 2, can this work after just one epoch, is there evidence for fitting to noisy samples?;  \n  - comparing the variance of loss vs wrong event – these have different units, do you mean relative variance?; \n  - line 129 “with their outcomes frequently flipping between similar classes” – it would be nice to see evidence of this, and would most of these flipped-between classes not also be incorrect?; \n  - line 205 “With more accurate model output, the clean and noisy distribution gradually separate.” – is this true in terms of being able to discern which distribution a sample is from? Please demonstrate.\n-  L_{SIM} is only the squared error (you need to do this over all samples to get MSE). Because of how it is used in the overall training objective (8), this is NOT MSE.\n-\tResults are from 5 random trials, but only means are reported. Please include error bars (std).\n-\tMethods are appropriate, several datasets with several types of label noise (symm, asymm, instance-dept) are used.\n-\tAuthors have not touched on weaknesses of their work\n\n\nClarity:\n\n-\tI found this paper difficult to read, follow, and understand. I spent considerable time to carefully read this paper to the best of my ability. There are several notational issues that lead to difficulties in understanding: \n  - Line 115 says f() is a model. Does f output a list of predicted probabilities or does it output the predicted class directly? In Eq (1), it seems that f(x) outputs a categorical label (should this be argmax of f?). In Eq (10), the output from f on two augmentations is averaged, but how can we average two categories (should the output be lists of probabilities?)? Also in Eq (10), it would seem that f is defined recursively based on the right hand side of the equation (is the left hand side the definition of f, or is it just a new pseudo-label and should be given a new variable). In Eq (11), it is unclear how the mathematical operations are performed on either a categorical or list of predicted probabilities output from f (is log evaluated on all probs of all classes?). \n  - Is noise rate given a variable of \\mu or r?\n-\tIssues with Figures: \n  - Fig 2 axes - Why is horizontal axis \"Loss / Wrong event Distribution\", What is the horizontal axis meant to be?, In (d), why is the vertical axis on the right?; \n  - Fig 2 - says Mean but caption says median, which is it?; \n  - Fig 4 - Why use line chart here? The methods on the horizontal axis are categorical and not interpolatable. Is the vertical axis meant to be number of examples, or accuracy?\n-\tMinor grammatical issues are found throughout (use of singular vs plural, “Following setting in [12,13].”, etc.) are distracting.\n\n\nSignificance:\n\n-\tThis paper improves upon existing LNL work in a meaningful way, given the comprehensive outperformance of existing methods on all datasets and noise rates.\n\n\nOriginality:\n\n-\tThis paper is original in its presentation of the Wrong Event metric, framework, and multi-part custom loss function.\n\nQUESTIONS:\n- My interpretation was that the main contribution is the Wrong Event metric; however, a rather elaborate framework (and loss function) are also developed here. What is the main takeaway here? Is it this metric (such that other creative frameworks may be used) or the framework (whereby other creative metrics may be used) that is the main contribution?\n- Why does the method use only 10 or 15 epochs when the earlier figures show 100 epochs? Does the intuition described in the earlier figures still hold in this case?\n- What is the meaning of f() in your paper? Is this the model? What does f() output - the list of predicted probabilities or the predicted class?\n\nLIMITATIONS:\n- No error bars even though the checklist answers item 7 as \"Yes\". \n- Limitations and Broader Impacts are discussed in Appendix F according to justification in checklist item 2. However, as reviewer, I am not obligated to review the appendices. It is my understanding that limitations should be discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage framework, IDO (Instance-level Difficulty Modeling and Dynamic Optimization), for robust training on noisy labeled data. It introduces wrong event, a metric counting incorrect predictions per sample, which effectively distinguishes clean, noisy, and hard samples throughout training. Experiments on synthetic and real-world datasets show that IDO outperforms state-of-the-art methods in accuracy, efficiency, and scalability.\n\nSTRENGTHS AND WEAKNESSES:\nThe method is well-motivated and empirically sound. The paper is clearly written, with thorough explanations, visualizations, and ablation studies. In addition, the proposed approach is thoughtfully designed to reduce reliance on hyperparameter tuning.\n\nHowever, the paper does not sufficiently discuss potential failure cases, such as performance under a wider range of noise levels, especially in instance-dependent noise settings. Moreover, although obtaining prior knowledge in Stage 1 is crucial—particularly in capturing the label wave number—there is a lack of detailed analysis on its impact and sensitivity.\n\nQUESTIONS:\n1. How does the method perform when class boundaries are highly entangled? \n\n2. Similar to question 1, additional experiments across a wider range of noise levels—especially under instance-dependent noise—would strengthen the paper. These results would offer a more realistic understanding of the method’s robustness and practical applicability. \n\n3. According to Table 5, the similarity loss seems to have a larger effect in more challenging noise settings (e.g., Asym. 40%, Inst. 40%). Does this imply that the method relies more heavily on consistency learning under difficult conditions? What happens if the difficulty score is fixed to a high value—i.e., treating all samples as difficult? Is it desirable for easy samples to receive a low weight for similarity loss? A deeper analysis of the behavior of the difficulty-based weighting would be helpful.\n\n4. Including more baselines, such as InstanceGM, would provide a more comprehensive comparison and help contextualize the proposed method’s performance more clearly.\n  - Garg, Arpit, et al. \"Instance-dependent noisy label learning via graphical modelling.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2023.\n\nLIMITATIONS:\nAs the paper mentioned, the situation with a high noise ratio differs from the actual data environment, but analyzing performance will help in evaluating the robustness of the methodology.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}