Jump-teaching: Ultra Robust and Efficient Learning with Noisy Labels

Authors: Kangye Ji, Fei Cheng, Zeqing Wang, Qichang Zhang, Bohu Huang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: learning with noisy labels, machine learning, classification

Abstract:
Sample selection is the most straightforward technique to combat noisy labels, aiming to prevent mislabeled samples from degrading the robustness of neural networks. However, compounding selection bias and redundant selection operations have always remained challenging in robustness and efficiency. To mitigate selection bias, existing methods utilize disagreement in partner networks or additional forward propagation in a single network. For selection operations, they involve dataset-wise modeling or batch-wise ranking. Any of the above methods yields sub-optimal performance. In this work, we propose $\textit{Jump-teaching}$, a novel framework for optimizing the typical workflow of sample selection.  Firstly, Jump-teaching is the $\textit{first}$ work to discover significant disagreements within a single network between different training iterations. Based on this discovery, we propose a jump-manner strategy for model updating to bridge the disagreements. We further illustrate its effectiveness from the perspective of error flow. 
Secondly, Jump-teaching designs a lightweight plugin to simplify selection operations. It creates a detailed yet simple loss distribution on an auxiliary encoding space, which helps select clean samples more effectively. In the experiments, Jump-teaching not only outperforms state-of-the-art works in terms of robustness, but also reduces peak memory usage by $0.46\times$ and boosts training speed by up to $2.53\times$. Notably, existing methods can also benefit from the integration with our framework.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730679588275

thread_type:
official_review

<thread_object>
creator: reviewer_fvym

<content>
summary:
This paper proposes the Jump-Teaching methodology for learning with noisy labels. 

Specifically, it investigates an efficient approach that requires only a single network. 

To achieve this, the authors introduce two key techniques: a Jump-update Strategy to mitigate selection bias and Semantic Loss Decomposition to simplify the selection operation. 

The effectiveness of the proposed approach is demonstrated through experiments on three benchmark datasets: CIFAR-10, CIFAR-100, and Clothing-1M.

strengths:
Research on sample selection methods using a single network to reduce computational costs is an interesting research topic.

weaknesses:
The academic novelty of this approach appears limited. It is unclear whether updating the model based on selections from the previous step offers any theoretical advantage over the naive approach of updating the model at every iteration. Additionally, is there theoretical support that using data from the previous step effectively addresses the noisy label problem? Moreover, extracting useful information from models at different training epochs has already been extensively explored in the literature. A seminal work in this area, for example, is Snapshot Ensembles: Train 1, get M for free (ICLR ’17).


The experimental results are not convincingly state-of-the-art. In particular, several recent relevant papers (a, b, c) are missing from the references. Their result tables show significantly better performance on CIFAR-10 and CIFAR-100 compared to the results presented in this work.
 
(a) Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels (NeurIPS’21)

(b) DISC: Learning From Noisy Labels via Dynamic Instance-Specific Selection and Correction (CVPR’23)

(c) Sample-wise Label Confidence Incorporation for Learning with Noisy Labels (ICCV’23)

questions:
It is not clear why using data selection results from previous iterations for model updates would be beneficial. Specifically, why would the set of samples selected by the model in the previous step yield better results than the samples selected in the current step? Is the main advantage simply that it avoids sequential updates, thereby reducing the amplification of error accumulation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691391543

thread_type:
official_review

<thread_object>
creator: reviewer_5v7y

<content>
summary:
To mitigate compounding selection bias and redundant selection operations in existing methods, the authors of this paper propose a novel framework for optimizing the typical workflow of sample selection, called Jump-teaching. Jump-teaching focuses on discovering significant disagreements within a single network between different training iterations by employing a jump-manner strategy for model updating to bridge the disagreements. Besides, Jump-teaching designs a lightweight plugin to simplify selection operations to help select clean samples more effectively. Finally, experimental results on synthetic and real-world noisy datasets, demonstrate the robustness of Jump-teaching.

strengths:
1. This idea and motivation for discovering significant similarities within a single network between different training iterations are interesting and fascinating.

2. This paper has carried out a lot of formula derivation and proved the effectiveness of the proposed method from the theoretical knowledge level.

3. Figures 1, 2, and 3 in this paper simply and clearly express the main ideas and innovations of the paper.

weaknesses:
1. Authors need to further provide the results of J-Co-teaching and J-DivideMix on Clothing1M in Table 3 to prove the reliable performance in real scenarios.

2. There are errors with the experimental results data. J-Co-teaching does not achieve optimal performance under some settings in Table 4, such as Sym-0.5 of CIFAR-10 and Sym-0.5 and Asym-0.4 of CIFAR100.

3. The content of the semantic loss decomposition part of this paper is not strongly related to the main motivation of this paper, Jump-update Strategy, and is more like an auxiliary trick.

4. There is no clear definition of I_{detection} in Eq. (8).

5. The names of the citation methods are not uniform, such as ' JoCoR ' in the relevant work section and ' JoCor ' in the experimental section.

6. It is recommended to compare the proposed method with 2024 SOTAs.

questions:
See above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693377785

thread_type:
official_review

<thread_object>
creator: reviewer_zgnd

<content>
summary:
The paper proposes "Jump-teaching," a novel framework for robust and efficient learning with noisy labels. By introducing a jump-update strategy and a Semantic Loss Decomposition plugin, the method reduces sample selection bias and enhances efficiency. Experiments show Jump-teaching improves performance over state-of-the-art methods, particularly under high noise conditions, with notable gains in memory efficiency and processing speed.

strengths:
- Proposes an innovative jump-update strategy that significantly reduces selection bias in single-network training.  
- Semantic Loss Decomposition provides a lightweight yet effective way to distinguish clean and noisy samples.  
- Empirically validated with improved accuracy, efficiency, and robustness across various noise levels and datasets.

weaknesses:
- I disagree with the claim that this work is the first to identify disagreements across different iterations within a single network. Prior studies [1][2] have leveraged these disagreements to distinguish clean samples from corrupted data in training sets. I recommend that the authors revise this claim and include comparisons with these two works.

- In Figure 2, the authors introduce the IoU metric to measure disagreements. Although an explanation of IoU is provided in the appendix, could the authors illustrate what range of IoU values is considered preferable? Because I notice that the IoU value of Jump-update is between the values of self-update and cross-update, the performance of Jump-update is the best (see Figure 2(c,d)).

- Property 1.   (1) I have a question regarding the assumption in Property 1, namely that $N_A$ equals $N_{iterations}$. From past experience, the model often generates biased selections initially, then gradually corrects this bias as performance improves, given moderate noise rates (10%, 20%). Therefore, error accumulation may not persist in later iterations.  (2) Additionally, the results in Figure 4(a) do not align well with the conclusion of Property 1. The highest test accuracy occurs at r = 50% rather than r = 10%. 

- There are some concerns regarding whether the jump-update is a more effective strategy for selecting clean samples. (1) In Table 4, at typical noise ratios (e.g., CIFAR-10/100 sym. 50%, CIFAR-100 asym. 40%), J-Co-teaching does not outperform standard Co-teaching (2 networks). (2) While non-trivial improvements are observed in Table 1, these gains do not carry over to a semi-supervised learning setting (see Table 5). In some settings, J-DivideMix is worse than DivideMix.

- The compared methods in Table 1 are outdated. Comparing with more recent works is necessary; for example, ProMix (IJCAI'23).


[1] Late Stopping: Avoiding Confidently Learning from Mislabeled Examples. ICCV'23  
[2] Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization. ECCV'22  
[3] ProMix: Combating Label Noise via Maximizing Clean Sample Utility. IJCAI'23

questions:
see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713700652

thread_type:
official_review

<thread_object>
creator: reviewer_ysnh

<content>
summary:
The authors introduce the concept of identifying significant disagreements within a single neural network across different training iterations. This discovery leads to the proposal of a "jump-manner" strategy for model updates, effectively bridging the gaps caused by these disagreements. Jump-Teaching simplifies the sample selection process through a lightweight plugin that generates a clear loss distribution in an auxiliary encoding space. This approach enhances the ability to select clean samples more effectively, addressing selection bias and redundancy. The framework demonstrates substantial improvements in both robustness and efficiency compared to state-of-the-art methods, specifically reducing peak memory usage by 46% and increasing training speed by up to 253%. The paper highlights that current methods can benefit from integrating with the Jump-Teaching framework, suggesting that it enhances the overall approach to learning with noisy labels.

strengths:
1. The method achieves improved performance under high noise rates on CIFAR datasets.
2. The method demonstrates better computational and storage efficiencies during testing.

weaknesses:
1. The experiments conducted on CIFAR-10 with 90% symmetric noise lack meaningful insight, as this setting results in random labels for each sample, effectively reducing the task to an unsupervised learning scenario.
2. The presentation needs improvement. Suggested changes include:
   - The methodology section incorporates experimental analysis (Figure 4), making it difficult to discern insights related to debiasing.
   - The connection among the four subsections in Section 3.2 is unclear.
   - The framework presented in Figure 1 contains excessive details that are not explained in the introduction; these should either be removed or relocated to the methodology section.
3. The authors claim that “Jump-Teaching is the first work to discover significant disagreements within a single network between different training iterations.” However, the concept of leveraging disagreements across different training iterations has been previously studied (see [1]).

[1] Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization, ECCV 2022.

questions:
1. As indicated in Table 5, the accuracy under 90% symmetric noise on CIFAR-10 exceeds 75%, which corresponds to random labels for the training samples. This scenario can be classified as an unsupervised learning task rather than weakly supervised learning. We need to reconsider the implications of generalization in learning with noisy labels using semi-supervised learning methods, given the lack of supervision.
2. It seems unreasonable to separate the updates of the neural network parameters in steps 9-10. Combining \( L^{BCE} \) and \( L^{CE} \) and updating the neural network with respect to the total loss could be more efficient.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732293512903

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers (Part 2)

comment:
**[Experiments]**

R.ZGND and R.5v7y recommended comparing our work with recent studies. In response, we expand the following experiments to support our paper further:
- **Supplementing baselines in Table 1.** To compare with the latest method, we include the results of RML [4] in Table 1 to showcase the up-to-date effectiveness of Jump-teaching.
- **Statistical analysis of labels under 90% symmetric noise.** We conduct experiments to calculate the ratio and number of selected labels under 90% symmetric noise. The results confirm that the selected labels are not random, thereby validating the rationality of our experimental setup.
- **Conducting additional experiments on real-world datasets.** We conduct further experiments using J-Co-teaching and J-DivideMix on the Clothing1M dataset to demonstrate the reliable performance of the Jump-update strategy in real-world scenarios.

[1] Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization \
[2] Late stopping: Avoiding confidently learning from mislabeled examples \
[3] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels \
[4] Regroup Median Loss for Combating Label Noise (AAAI'24)


**[Presentation]**

We have improved our presentation by:
- Simplify Figure 1 to better align with the introduction. 
- Enhance the connection between Figure 1 and the text in lines 98 and 99.
- Add necessary explanations, such as disagreement and the notation $I_{detection}$.
- Revise Property 1 by incorporating a hypothesis.
- Correct a few typos.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732293642581

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers (Part 1)

comment:
We sincerely thank all the reviewers for their valuable time and insightful suggestions. In this 'global' rebuttal, we clarify our technical contributions and present new experimental results. Furthermore, we will address other concerns in our responses to individual reviews. A revised version of our paper has been uploaded, with all the modifications marked in blue to address the concerns. 

**[Novelty]**

R.Ysnh and R.ZGND expressed concerns regarding our claim that "We are the first work to discover significant disagreement within a single network," as prior studies [1, 2] have utilized predictions of different epochs. This concern can be addressed with three key points:

1)  The concept of "disagreement" in our paper essentially differs from counterparts in [1, 2] in terms of definition, described objects, application, calculation, and focus.
  - **Definition.** The definition of "disagreement" is different from these terms, such as "Fluctuation" [1] and "First-time k-epoch Learning (FkL)" [2]. As stated in lines 75 to 76, the "disagreement" we refer to is derived from [3]. It denotes the differences in selection behaviors within networks. In contrast, [1] defines a similar concept as "fluctuation," which refers to a sample being classified correctly at one moment but misclassified in the subsequent learning step.  "FkL" in [2] refers to the minimum index of the training epoch that the instance has been predicted to its given label for $k$ consecutive epochs.
  - **Described Object.** The object described by "disagreement" is entirely distinct from others. "Fluctuation" or "FkL" refers to the characteristics of samples, while "disagreement" relates to the characteristics of networks. For example, in [3], the object is two networks, whereas in this paper, the object is the network across different iterations.  "Fluctuation" or "FkL" describes variations in judgments within a single sample.
  - **Application.** "Disagreement" is applied in a completely different way from others. "Disagreement" as a network characteristic, is used to design model update strategies, whereas "fluctuation" and "FkL" as sample characteristics define selection criteria.
  - **Calculation and Focus.** The calculation of each term differs significantly, which stems from their distinct focus. From the focus perspective,  "fluctuation" and "FkL" concern whether a sample is clean, while "disagreement" addresses the potential for correcting selection bias in model updates. Therefore, "Fluctuation" and "FkL" emphasize and calculate consistency between model predictions and labels, whereas "disagreement" is independent of labels and is calculated as the Intersection Over Union (IoU) between selected data sets.  

2) The meaning we emphasized through "first" has not been discovered and utilized.
  - We are the first to discover that disagreement within a single network not only exists but also persists during the training, shown in Figure 2(a)(b).
  - We are the first to discover that disagreement within a single network is significant, which is even larger than two networks, as shown in Figure 2(a)(b).
  - We are also the first to quantify and visualize "disagreement" by the IoU metric. 
3) We sincerely thank all the reviewers for their dedicated efforts. We have made the following revisions for further clarity:
  - Revise the claim by emphasizing its beneficial attributes for the mitigation of bias in line 107.
  - Add the definition of "disagreement" clearly in lines 75 and 76. "Disagreement" refers to differences of networks in the selection behaviors.
  - Add discussions on [1, 2] in Related Work in lines 151 and 152.
  - Add discussions to compare our paper and the related papers in Appendix 2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733131514225

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Submission and Discussion

comment:
We sincerely thank the reviewers and chairs for their valuable feedback. During the discussion period, we made every effort to address all of the reviewers' concerns, particularly regarding the novelty and effectiveness of the proposed method. Additionally, we have conducted the suggested experiments and made several revisions, all of which are highlighted in blue in the uploaded PDF file. Although we are still awaiting responses from some reviewers, we would be happy to provide a clear summary of the paper and the points discussed.

## Motivation
This paper focuses on combating noisy labels by selecting clean samples for training. The motivation behind Jump-teaching is to optimize the outdated sample selection workflow by mitigating selection bias within a single network and simplifying redundant selection operations through the incorporation of a lightweight plugin.
## Contributions
- **Novelty of Discovery.** We are the first work to discover the disagreement for correcting selection bias within a single network. Notably, this has been clarified in the [Novelty] section of the general response and in specific responses to Reviewer #1 and Reviewer #2.
- **Technical Contribution of Jump-update Strategy.** Jump-update strategy proposes a simple and cost-free solution to bridge disagreement and split sequential error flows, leading to a significantly better trade-off between efficiency and robustness. 
- **Technical Contribution of Semantic Loss Decomposition.** The proposed plugin virtually eliminates redundancy in selection operations, occupying only 2.2% of the total training time, by exploiting the memorization effects in the distribution of the decomposed single loss.
## Experiments
Extensive experimental results confirm the effectiveness of our method. 
- **Superior Performance of Jump-teaching.** To address the concerns regarding up-to-dateness, we compare 2024 SOTA in Table 1. Jump-teaching outperforms the latest baselines across various noise ratios and types, including symmetric noise (Table 1), asymmetric noise (Table 1), pairflip noise (Table 9), IDN (Table 10), and real-world noise (Table 3). Furthermore, it reduces peak memory usage by $0.46\times$ and accelerates training speed by up to $2.53\times$ (Table 2), achieving an extremely low overhead for sample selection (2.2% in training speed and 2.8% in peak memory).
- **Effective Integration of Jump-update Strategy.** We successfully apply the Jump-update strategy to improve the performance of mitigating bias in two representative settings: supervised-only and semi-supervised methods. The results demonstrate impressive improvements under extreme noise conditions (Table 4 and Table 5) and real-world noise (Table 13).  The newly added results in Table 13 also address Reviewer #3's concerns regarding the performance of J-co-teaching and J-DivideMix under real-world noise.
We also summarize the newly conducted experiments addressing these concerns in the [Experiments] section of the general response and respond to specific reviewers in detail.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel technique called Jump-teaching for learning with noisy labels. Specifically, Jump-teaching aims to discover significant disagreements within a single network between different training iterations. Based on this discovery, this paper proposes a jump-manner strategy for model updating to bridge the disagreements. The authors further illustrate the effectiveness from the perspective of error flow. Moreover, Jump-teaching designs a lightweight plugin to simplify selection operations. It creates a detailed yet simple loss distribution on an auxiliary encoding space, which helps select clean samples more effectively. 

Although the authors claim that their method is novel, the reviewers consider that utilizing the disagreements for sample selection across different iterations within a single network or across different networks is not new, so all of them show negative scores.