Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a DivideMix framework for learning with noisy labels, where they first Co-Divide the training data into a labeled clean set and an unlabeled noisy set by modeling the per-sample loss distribution with GMM and using the small loss trick, then they exploit MixMatch to train the model on those labeled and unlabeled data in a semi-supervised manner. Experiments and comparisons with SOTA are provided, together with an ablation study.

Pros:
-The paper bridges the area of learning with noisy labels with semi-supervised learning and proposes an interesting method to learn with noisy labels in a semi-supervised manner. The treatment proceeds from analyzing good unsupervised loss functions, improving the MixMatch and implementing thorough experiments.

-The paper is clear and flows smoothly. The treatment is thorough, proceeding from designing algorithms, implementing experiments and an ablation study.

-The impact of the method is a clear asset. Most existing label noise works focus on certain noise models, but this paper is more general and proposes an interesting method to learn with noisy labels in a semi-supervised manner, and the experimental results are promising.

-The effort made on designing the confidence penalty for asymmetric noise is interesting.

Remarks:
-Sec 3.1: it seems that \tau is an important hyperparameter for dividing the noisy data into labeled and unlabeled sets, but in Sec 4 I only see that \tao is set to be 0.5 or 0.6 for CIFAR, what about other experiments? I’m also wondering if \tao needs some decay during training? For example, it may be 0.5 at early training epochs, but may be smaller at last, since deep NNs are gradually fitting noisy features during training.

-Algorithm: the proposed method needs to train two networks simultaneously. During each epoch, it firstly divides the noisy data by modeling the per-sample loss distribution with GMM, and then do MixMatch with label co-refinement and co-guessing. I’m a bit concerned about efficiency. So how about the computation time?

-Experiments: more details on experimental protocol may be needed: what kind of hyperparameter tuning was done? How many repeated runs? It would be helpful to report the means and standard deviations based on repeated samplings.

Overall take: this paper proposes a thorough treatment of learning with noisy labels in a semi-supervised manner, designing the algorithm and testing it empirically, which is an interesting and important contribution. My only concern is about the novelty since the small loss trick in label noise and the MixMatch approach in SSL are already explored by many recent studies, but to the best of my knowledge, this paper is the first to unify them to solve label noise problems.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes an algorithm that learns with noisy labels that achieves state-of-the-art results. Their algorithm tries to exploit the noisy samples by assigning a ‘correct’ label through MixMatch. It borrows the idea from both semi-supervised learning and learning with label noise. 

Suggestions:

1. When the author talked about “correct the loss function”, there are in fact two types of corrections: the first type tries to correct the loss function by equally treating all the samples, e.g., classical Huber loss, or F-correction. The second type tries to either re-weight samples or separate clean and noisy samples explicitly, which also results in correcting the loss function. It would be more clear if Section 2.1 can emphasize the difference between the two. 

2. Also, there are other papers providing different but insightful ideas to label noise problem. The authors addressed the idea of using co-learning to avoid confirmation bias. However, it should be noted that this is not the only way to avoid confirmation bias, and their are other methods without using two networks [1-2], both providing some theoretical insights to the problem. It would be good to include them in the related work as well. 

3. I would like to see a comparison of running time other than the accuracy, understanding the efficiency of each algorithm is important from a practical perspective. 

[1] Learning with Bad Training Data via Iterative Trimmed Loss Minimization, Yanyao Shen, Sujay Sanghavi, ICML 2019.
[2] Robust Learning from Untrusted Sources,Nikola Konstantinov, Christoph H. Lampert, ICML 2019


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposed a method named DivideMix for learning with noisy labels, on top of the recent semi-supervised learning method MixMatch from Google. The idea is to model the per-sample loss distribution with a mixture model to dynamically DIVIDE the training data into (a labeled set with clean samples) and (an unlabeled set with noisy samples) and trains the model on both the labeled and unlabeled data in a semi-supervised manner.

The novelty is borderline. In the area of learning with noisy labels, it is known that SSL can work under this problem setting for several years, for example, the famous method "virtual adversarial training" from ICLR 2016 and a recent method "smooth neighbors on teacher graphs" from CVPR 2018. As a consequence, it is not surprising that the latest MixMatch can work as well, since MixMatch comes from mixup, virtual adversarial training and entropy minimization. This makes the novelty borderline. However, the significance may still be high according to the reported experimental results, and thus we may accept it to let more deep learning practitioners see the promising results.
review_title: Official Blind Review #2

Meta-review: This paper proposes an algorithm for noisy labels by adopting an idea in the recent semi-supervised learning algorithm.

As two problems of training noisy labels and semi-supervised ones are closely related, it is not surprising to expect such results as pointed out by reviewers. However, reported thorough experimental results are strong and I think this paper can be useful for practitioners and following works. 

Hence, I recommend acceptance.
Decision: Accept (Poster)