DivideMix: Learning with Noisy Labels as Semi-supervised Learning

Abstract: Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Code is available at https://github.com/LiJunnan1992/DivideMix .

Reviews and author responses:

The authors have responded to my questions, and I have no other comments to make.

We thank Reviewer 2 for the insightful comments. We believe that our work can inspire new research ideas to explore the intersection between the areas of learning with noisy labels and semi-supervised learning.

We appreciate Reviewer 1 for the very helpful and constructive suggestions.  Following the suggestions, we have improved the paper to (1) emphasize the difference between the two types of loss corrections in Section 2.1, (2) include [1-2] into the related work section, and (3) add a training time analysis in Appendix D. Next we provide the computation time analysis.
 
In terms of inference time, our method does not introduce extra computation if a single model (e.g. theta1) is used for test, whereas the inference time is doubled if we use the ensemble of both models.
In terms of training time, Appendix D shows a detailed analysis. We first compare the total training time of DivideMix to several SOTA methods, using a single Nvidia V100 GPU. DivideMix (5.2 h) is slower than Co-teaching+ (4.3 h) but faster than P-correction (6.0 h) and Meta-Learning (8.6 h) which involve multiple training iterations. We also break down the computation time per-epoch for each operation in DivideMix, Co-Divide (Alg. 1, line 4-8) takes 17.2 seconds, MixMatch of data (Alg. 1, line 12-24) takes 16.0 seconds, whereas the model’s forward-backward computation (Alg. 1, line 25-27) takes 12.5 seconds.

We appreciate Reviewer 3 for the recognition of this paper and the valuable comments. Next we response to the questions raised by the reviewer. 
 
Question 1: What is the value of \tau in other experiments?
Response: As explained in Appendix B, we set the value of \tau as 0.5 for other experiments.
 
Question 2: I’m also wondering if \tau needs some decay during training, since deep NNs are gradually fitting noisy features during training.
Response: We find that the proposed DivideMix can prevent the network from fitting to label noise and keep higher loss for noisy samples. This is supported by Figure 3 in Appendix, which shows that the GMM improves in distinguishing clean and noisy samples as training proceeds. Therefore, we can keep a constant \tau during training.
 
Question 3: I’m a bit concerned about efficiency. So how about the computation time?
Response: In terms of inference time, our method does not introduce extra computation if a single model (e.g. theta1) is used for test, whereas the inference time is doubled if we use the ensemble of both models.
In terms of training time, we have added a training time analysis in Appendix D. We first compare the total training time of DivideMix to several SOTA methods, using a single Nvidia V100 GPU. DivideMix (5.2 h) is slower than Co-teaching+ (4.3 h) but faster than P-correction (6.0 h) and Meta-Learning (8.6 h) which involve multiple training iterations. We also break down the computation time per-epoch for each operation in DivideMix, Co-Divide (Alg. 1, line 4-8) takes 17.2 seconds, MixMatch of data (Alg. 1, line 12-24) takes 16.0 seconds, whereas the model’s forward-backward computation (Alg. 1, line 25-27) takes 12.5 seconds.
 
Question 4: More details on experimental protocol may be needed: What kind of hyperparameter tuning was done? How many repeated runs? 
Response: As explained in Section 4.1 and Appendix B, we keep most hyperparameters fixed across experiments and do light tuning on \lambda_u (weight for unsupervised loss) on a per-experiment basis using a small validation set. We report results on single runs due to time concerns with the large number of experiments that we have conducted. Since all methods use the same training data, we believe that the results are fair for comparison. We thank the reviewer for this suggestion and would perform repeated runs in the future.

This paper proposed a method named DivideMix for learning with noisy labels, on top of the recent semi-supervised learning method MixMatch from Google. The idea is to model the per-sample loss distribution with a mixture model to dynamically DIVIDE the training data into (a labeled set with clean samples) and (an unlabeled set with noisy samples) and trains the model on both the labeled and unlabeled data in a semi-supervised manner.

The novelty is borderline. In the area of learning with noisy labels, it is known that SSL can work under this problem setting for several years, for example, the famous method "virtual adversarial training" from ICLR 2016 and a recent method "smooth neighbors on teacher graphs" from CVPR 2018. As a consequence, it is not surprising that the latest MixMatch can work as well, since MixMatch comes from mixup, virtual adversarial training and entropy minimization. This makes the novelty borderline. However, the significance may still be high according to the reported experimental results, and thus we may accept it to let more deep learning practitioners see the promising results.

This paper proposes an algorithm that learns with noisy labels that achieves state-of-the-art results. Their algorithm tries to exploit the noisy samples by assigning a ‘correct’ label through MixMatch. It borrows the idea from both semi-supervised learning and learning with label noise. 

Suggestions:

1. When the author talked about “correct the loss function”, there are in fact two types of corrections: the first type tries to correct the loss function by equally treating all the samples, e.g., classical Huber loss, or F-correction. The second type tries to either re-weight samples or separate clean and noisy samples explicitly, which also results in correcting the loss function. It would be more clear if Section 2.1 can emphasize the difference between the two. 

2. Also, there are other papers providing different but insightful ideas to label noise problem. The authors addressed the idea of using co-learning to avoid confirmation bias. However, it should be noted that this is not the only way to avoid confirmation bias, and their are other methods without using two networks [1-2], both providing some theoretical insights to the problem. It would be good to include them in the related work as well. 

3. I would like to see a comparison of running time other than the accuracy, understanding the efficiency of each algorithm is important from a practical perspective. 

[1] Learning with Bad Training Data via Iterative Trimmed Loss Minimization, Yanyao Shen, Sujay Sanghavi, ICML 2019.
[2] Robust Learning from Untrusted Sources,Nikola Konstantinov, Christoph H. Lampert, ICML 2019



This paper proposes a DivideMix framework for learning with noisy labels, where they first Co-Divide the training data into a labeled clean set and an unlabeled noisy set by modeling the per-sample loss distribution with GMM and using the small loss trick, then they exploit MixMatch to train the model on those labeled and unlabeled data in a semi-supervised manner. Experiments and comparisons with SOTA are provided, together with an ablation study.

Pros:
-The paper bridges the area of learning with noisy labels with semi-supervised learning and proposes an interesting method to learn with noisy labels in a semi-supervised manner. The treatment proceeds from analyzing good unsupervised loss functions, improving the MixMatch and implementing thorough experiments.

-The paper is clear and flows smoothly. The treatment is thorough, proceeding from designing algorithms, implementing experiments and an ablation study.

-The impact of the method is a clear asset. Most existing label noise works focus on certain noise models, but this paper is more general and proposes an interesting method to learn with noisy labels in a semi-supervised manner, and the experimental results are promising.

-The effort made on designing the confidence penalty for asymmetric noise is interesting.

Remarks:
-Sec 3.1: it seems that \tau is an important hyperparameter for dividing the noisy data into labeled and unlabeled sets, but in Sec 4 I only see that \tao is set to be 0.5 or 0.6 for CIFAR, what about other experiments? I’m also wondering if \tao needs some decay during training? For example, it may be 0.5 at early training epochs, but may be smaller at last, since deep NNs are gradually fitting noisy features during training.

-Algorithm: the proposed method needs to train two networks simultaneously. During each epoch, it firstly divides the noisy data by modeling the per-sample loss distribution with GMM, and then do MixMatch with label co-refinement and co-guessing. I’m a bit concerned about efficiency. So how about the computation time?

-Experiments: more details on experimental protocol may be needed: what kind of hyperparameter tuning was done? How many repeated runs? It would be helpful to report the means and standard deviations based on repeated samplings.

Overall take: this paper proposes a thorough treatment of learning with noisy labels in a semi-supervised manner, designing the algorithm and testing it empirically, which is an interesting and important contribution. My only concern is about the novelty since the small loss trick in label noise and the MixMatch approach in SSL are already explored by many recent studies, but to the best of my knowledge, this paper is the first to unify them to solve label noise problems.

Thanks for your comments! 
We have indeed analyzed the divergence of the two networks using the networks' prediction discrepancy (on the same images) as an indicator. Our result shows that the two networks can stay sufficiently diverged because of multiple factors: different parameter initialization, different training data division, different mini-batch sequence, and different training targets. The iou of the labeled set is one of the causes, rather than a direct indicator of the network divergence. The iou would get smaller as training proceeds because both networks would get better at dividing clean and noisy samples.

Thank you for your well written and thorough study. The joint training was specially interesting to me. Have you looked at the intersection over union of the samples in one set (labeled) during training of the two networks? 

This could be a divergence indicator of the two networks. If iou is high, maybe one can enforce (guarantees even) divergence with aux losses. But if analyzing the iou plot shows they are already sufficiently diverged there would be no point in it. 

Thanks

Meta-review: This paper proposes an algorithm for noisy labels by adopting an idea in the recent semi-supervised learning algorithm.

As two problems of training noisy labels and semi-supervised ones are closely related, it is not surprising to expect such results as pointed out by reviewers. However, reported thorough experimental results are strong and I think this paper can be useful for practitioners and following works. 

Hence, I recommend acceptance.

Decision: accept-poster