Title: Enhancing Sample Selection Against Label Noise by Cutting Mislabeled Easy Examples
Venue: NeurIPS 2025
Accepted: yes
Keywords: label noise

learning with noisy labels

sample selection
Abstract: Sample selection is a prevalent approach in learning with noisy labels, aiming to identify confident samples for training. Although existing sample selection methods have achieved decent results by reducing the noise rate of the selected subset, they often overlook that not all mislabeled examples harm the model's performance equally. In this paper, we demonstrate that mislabeled examples correctly predicted by the model early in the training process are particularly harmful to model performance. We refer to these examples as Mislabeled Easy Examples (MEEs). To address this, we propose Early Cutting, which introduces a recalibration step that employs the model's later training state to re-select the confident subset identified early in training, thereby avoiding misleading confidence from early learning and effectively filtering out MEEs. Experiments on the CIFAR, WebVision, and full ImageNet-1k datasets demonstrate that our method effectively improves sample selection and model performance by reducing MEEs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a sample selection strategy called Early Cutting, aimed at improving robustness against label noise by identifying and eliminating a specific group of samples termed Mislabeled Easy Examples (MEEs)—i.e., mislabeled data points that are learned early and confidently by the model. The method leverages the model state at a later training epoch to re-evaluate these \"early-learned confident samples\", applying a joint criterion of high loss, high confidence, and low gradient norm to isolate and remove MEEs. Experimental results on CIFAR-10, CIFAR-100, WebVision, and full ImageNet-1K demonstrate promising performance improvements over state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \\\n1.The paper highlights an overlooked yet important issue: not all noisy labels are equally harmful, and some “easy” mislabeled examples can be particularly misleading.\\\n2.Experimental results are extensive and evaluated across diverse datasets and noise settings.\\\n3.The method is simple yet principled, with clearly defined criteria (loss, confidence, gradient norm).\\\n4.The authors provide visual and empirical evidence to support the existence and impact of MEEs (e.g., Figure 1, Figure 4).\\\n\nWeaknesses: \\\n1.Overstatement of Novelty / Lack of Clear Contribution Boundaries\\\nThe authors introduce the concept of Mislabeled Easy Examples (MEEs) and claim that these are particularly harmful mislabeled samples that are confidently learned by the model early in training. They state (e.g., Line 61):\n“We discover that mislabeled samples correctly predicted by the model early in training disproportionately harm model’s performance…”\nHowever, this phenomenon has been studied in prior works, including:\na. Me-Momentum (Bai & Liu, CVPR 2021), which proposes a momentum-based mechanism to identify memorized noisy samples early in training.\\\nb. Late Stopping (Yuan et al., ICCV 2023), which explicitly analyzes the impact of early-learned noisy examples and designs a strategy to mitigate their negative effect.\\\nAlthough these works are briefly acknowledged in Appendix A, they are not properly discussed or contrasted in the main text—particularly in Sections 2.2 and 3 where MEEs and the proposed Early Cutting strategy are introduced. The current manuscript presents the identification of MEEs as a novel discovery without clearly delineating the incremental contributions beyond these prior works.\nThe authors should explicitly cite and compare these related works in the main body and clearly articulate the difference between MEEs and previously defined early-learned noisy examples. If the contribution lies in improved detection or filtering of such examples (e.g., via gradient stability), that should be clearly emphasized as the novel component. Without this clarification, the claim of novelty appears overstated.\\\n2. Ambiguity in Method Motivation\\\nThe use of the gradient norm to distinguish MEEs from hard samples (Equation 5) lacks solid theoretical justification. Why does low gradient norm necessarily imply \"false patterns memorized\"? The connection is heuristic and may not generalize.\nSee Lines 200–204: “MEEs tend to have low gradient norms because the model has confidently mislearned them…”\nThis needs more analysis or ablation to support.\\\n3. Lack of Baseline Comparisons\\\nWhile experiments include many methods, the paper omits direct comparisons with methods that also target early-learned noisy samples—e.g., “Late Stopping” or “Me-Momentum”，which are representative state-of-the-art methods targeting similar early-stopping-based noisy label issues. This omission makes it difficult to convincingly demonstrate that the proposed method outperforms existing approaches in the same problem setting.\\\n4. Limited Novelty Beyond Gradient-Based Criterion\\\nIf we set aside the motivational premise—i.e., that early-learned mislabeled samples are especially harmful—since this has already been discussed in prior work (e.g., Me-Momentum, Late Stopping), then the core novelty of this paper is mainly the incorporation of the gradient norm as an additional filtering criterion. However, the use of combined loss and confidence thresholds for filtering mislabeled data is not new and has been explored in various existing methods. Therefore, the paper's overall innovation may appear limited unless the unique contribution of the gradient-based criterion is more clearly justified and empirically highlighted.\n\nQUESTIONS:\n1. Can the authors clearly distinguish how MEEs differ from early-learned noisy samples in Late Stopping (Yuan et al., ICCV 2023)? Are they essentially the same, and if not, what distinguishes them?\n\n2. Why does the gradient norm reliably filter out MEEs rather than hard clean samples? Could the authors provide further evidence or ablations to justify this?\n\n3. How sensitive is the method to the percentile thresholds (e.g., top 10% loss, bottom 20% gradient)? While a small ablation is given, it could be expanded.\n\n4. Could the authors include comparisons against Me-Momentum and other dynamics-based selection methods targeting early-stage noise?\n\n5. In the introduction, the authors state:\n\"Specifically, even with a low noise rate in the selected subset, the presence of certain mislabeled samples can still significantly impair the model’s generalization performance.\"\nHowever, it is unclear what is meant by “low noise rate” in quantitative terms. Have the authors conducted any analysis to determine how low the noise rate needs to be for the impact on generalization to become negligible? In the experiments (e.g., Section 2.1), each subset contains 4,000 mislabeled samples among 34,000 total, which is still a relatively high proportion (~11.7%).\nWould the proposed conclusions still hold in more extreme settings—for instance, if the number of mislabeled examples was fewer than 1,000 among 30,000+ clean samples (i.e., <3%)? A discussion or additional experiment under such conditions would help validate the generality of the claim.\n\n6. When integrating the proposed Early Cutting method into the MixMatch semi-supervised learning framework (as described in Section 4.3), it is unclear whether the labeled and unlabeled subsets are fixed after the initial selection, or dynamically updated at each epoch during training.\nCould the authors clarify whether the confident (labeled) and unconfident (unlabeled) sets are re-evaluated iteratively, and if so, how often the update occurs? This detail is crucial for understanding the training dynamics and the consistency regularization effect.\n\nLIMITATIONS:\nThe paper would benefit from a more thorough discussion of the limitations regarding the generality of its claims. In particular, the authors argue that even with a low noise rate, harmful mislabeled samples (MEEs) can significantly degrade generalization. However, no quantitative analysis is provided to determine how low the noise rate must be for the harmful effects to become negligible. For instance, if only 500 or 1000 mislabeled samples are mixed into 30,000+ clean examples (i.e., <3%), would the same degradation still occur?\nIn addition, the Early Cutting method may mistakenly eliminate a small number of clean samples—especially borderline or ambiguous cases—due to its reliance on high-loss, high-confidence, and low-gradient criteria. While the authors argue that such early-learned samples are redundant, this might not hold in low-data or long-tail scenarios where each clean instance carries significant value. A more balanced discussion of this trade-off would improve the paper's robustness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper shows that not all incorrect labels are equally bad for training models. Mislabeled data learned early, which the authors call Mislabeled Easy Examples (MEEs), are especially harmful. The paper introduces a method named Early Cutting to fix this issue. This technique uses a more mature model from later in the training process to find and remove these MEEs from the initially selected data. Experiments on datasets like CIFAR, WebVision, and ImageNet confirm that this strategy successfully improves model performance by filtering out these specific harmful examples.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper identifies a new problem: mislabeled examples learned early during training cause disproportionate harm to the model.

2. This work proposes a counter-intuitive but effective method that uses a later model to correct the initial confident dataset.

3. The method is thoroughly validated with strong results across various datasets, noise types, and strong competitor methods.

Weaknesses:
1. The use of similar notations like “$T_i$” on Line 98, “$LT_{i}$” in Eq. (1), and the noise transition matrix “$T$” on Line 444 can easily confuse readers.

2. Citations for some important literature are missing, for example, the SGD and AdamW optimizers.

QUESTIONS:
1. Please use different notations for time ($T_i$), learning time ($LT_{i}$), and the noise matrix ($T$).

2. Consider adding citations for fundamental methods like the SGD and AdamW optimizers you used.

3. Please clarify the “Early Cutting Rate” hyperparameter and its selection in the main paper.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSample selection in learning with noisy labels typically identifies a confident subset of training\nexamples by reducing the noise rate within the selected data. However, such a strategy often overlooks\nthe fact that not all mislabeled examples are equally harmful to model performance. In contrast,\nthrough numerical experiments, this manuscript aims to demonstrate that predicted mislabeled\nexamples in the early stage of training, referred to as Mislabeled Easy Examples (MEEs), can be\nparticularly detrimental to performance. Building on this observation, the authors propose a method\ncalled Early Cutting to filter out MEEs. The method is evaluated on several benchmark datasets,\nincluding CIFAR, WebVision, and the full ImageNet-1k, to demonstrate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n\n    1. This manuscript explores an important and often overlooked issue in sample selection strategies for learning with label noise.\n\n     2.  Specifically, it aims to distinguish the varying negative impacts of mislabeled examples on model performance, which is valuable.\n\n    3. The experimental results support the intended merit of the proposed method.\n\n\nWeaknesses:\n\n    1. The proposed methods is merely evaluated by numerical experiments, and there is no theoretical guarantee.\n\n    2. Some concepts or terms are introduced without sufficient motivation or justification regarding their rationale or effectiveness.\n\n    3.  Some notations are inconsistently defined or used. For example, the number of classes is denoted by $K$ in the main text (Lines 184–185), but by $C$ in \nAppendix A on Related Work.\n\n    4. Certain key statements lack clarity.\n\n    5. Repetitive descriptions seem to appear a bit too frequently throughout the manuscript. While repetition can help emphasize important points, its overuse may affect the overall conciseness.\n\nQUESTIONS:\nBelow, I list some of my questions in the order of their appearance in the manuscript, rather than by significance. Please note that the list is not exhaustive, but is intended to illustrate key concerns.\n\n\\begin{itemize}\n    \\item  [1.]  Lines 39–43:\n\nThe statement ``As shown in Figure 1(a), we demonstrate that mislabeled samples which are correctly predicted by the model early in the training process disproportionately degrade performance\" lacks clarity. Specifically:\n\nA. There is no description of how Figure 1 was generated. As such, it is  unconvincing to support the authors’ claim.\n\nB. Clarify the meaning of ``mislabeled samples which are correctly predicted by the model...\".\n\n\nC. The phrase \"early in the training process\" needs clarification. How early is considered ``early\"? Does this vary across training algorithms or settings?\n\nD. The phrase ``disproportionately degrade performance\" is vague. Disproportionate to what? And whose performance is being degraded?\n\n\n\\item [2.] Lines 40-43:\n\n\nIn the statement ``In our analysis (see Section 2.2), we find that MEEs are often closer to the centers of their mislabeled classes in the feature space of classifiers trained in the early stages\", how are the ``centers” defined? In what sense are MEEs considered closer to these centers, e.g., in terms of Euclidean distance, cosine similarity, or other metric?\n\n\n\n\\item [3.] Lines 190-213:\n\n\n   The key idea is to introduce a set of suspicious samples ${\\cal S}$\n    in (3) and its refined version ${\\cal S}^\\prime$ in (5). While the rationale of using the scale of the cross-entropy loss, the prediction confidence, and  the gradient stability  seems reasonable, some fundamental issues are not addressed or fully discussed:\n\n   A. the predicted label $\\hat y_i$  is determined by the conditional probability $p_i^{(k)}$, which essentially requires the information on the clean label $y_i$ (BTW, there seems a typo in the definition of $p_i^{(k)}$: should $y$  be $y_i$?).\n   In the presence of label noise, the true label $y_i$ is typically unknown.\n     Then how do we determine $L_i$ and $c_i$\n   in order to construct ${\\cal S}$ and ${\\cal S}^\\prime$?\n\n\n\n\n   B. Assuming the issues in A is resolved, how do we  ensure that ${\\cal S}$ in (3) is not an empty   set? The constraints $L_i > \\delta$ and $c_i> \\tau$ can be equivalent written as\n   $$\n   p_i^{(y_i)} < \\exp(-\\delta) \\ \\text{and} \\ p_i^{(\\hat y_i)}> \\tau.\n   $$\n   To make ${\\cal S}$ mathematically meaningful and practically useful, appropriate constrains should be imposed on the thresholds $\\delta$ and $\\tau$. This important aspect seems to be neglected.\n\n    Although the author(s) state  \"Specifically, for all settings, we target the top 10\\% for loss, top 20\\% for confidence, and bottom 20\\% for gradient norm. These percentile-based selections are intentionally kept fixed across all experimental settings\n   to underscore the method’s general applicability and robustness, thereby obviating the need for dateset-specific hyperparameter tuning\", this treatment of the threshold values $\\delta$ and\n   $\\tau$ appears overly simplistic and does not adequately account for the diversity and  complexity of real-world data.\n\n\n\n\\item [4.] Lines 445 (in Appendix A):\n\n      A setting of instance-independent label noise is presented in (7). However, this key assumption does not appear to be clearly indicated in the main text. How\n        does this type of label noise relate to the construction of\n       ${\\cal S}$ and ${\\cal S}^\\prime$ in Section 3? Shouldn't these sets be constructed\n        based on the noisy labels and the cross-entropy loss defined in equation (10), rather than using $L_i$ as defined in Line 189?\n\n\\end{itemize}\n\nLIMITATIONS:\nThey do not seem to make limitations clear. Addressing the questions or concerns above could help improve the clarity of the work.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper has proposed a re-selection way of the confident subset identified in the former iteration. To support this, the authors have tried to show some aspects of explaining why this operation is effective. Experimental tests have shown the performance of the selection way. But, in my view, the paper spent a considerable amount of space on meaningless analysis and definitions, resulting in the details of the core algorithm not being provided in the main text. Without so much analysis, the method is still intuitive and effective enough.

STRENGTHS AND WEAKNESSES:
Strengths
1. From the experiments, the proposed algorithm has achieved the SOTA performance.
2. The paper is clear.

Weaknesses
1. Definition of MEE makes little sense. Its mathematical expression is not given.
2. The authors employed around 3 pages to describe a kind of phenomenon that not all mislabeled examples harm the model’s performance equally, and then derived the MEE. But the phenomenon is intuitive. The redundant details did not uncover any significant insight of the phenomenon.
3. In the Methodology section. What is the eventual method? What’s the specific objectives of the method? Why these important details are all in the appendix?

QUESTIONS:
See weaknesses.

LIMITATIONS:
Limitation is not clear.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01