Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a training strategy for robustness against label noise. The training strategy is simple and straightforward. The neural network will first be trained on the entire dataset with all the noisy labels. After obtaining the network with lowest validation error, the network will be used to make a prediciton on the original training set and select a subset of it to construct a maximal safe set. Finally, the network will be findtuned on this maximal safe set. The training strategy is very similar to tradictional  self-training in semi-superivsed learning and co-training for domain adaptation ([Co-training for domain adaptation, NIPS 2011]), except that the proposed prestopping only iterate the procedure once.

The paper discusses two important questions for the method: (1) when to early stop the training; (2) how to constuct a maximal safe set. The authors' responses to these questions are very natual but less interesting. Using the lowest validation error to early stop the training could be suboptimal, since the small validation set can not fully capture the data distribution and could make the network empirically overfit to this validation set. The criterion to contruct a maxial safe set is also conventional, and is similar to what a number of papers are doing, for example,
[1] Co-training for domain adaptation, NIPS 2011
[2] Self-ensembling for visual domain adaptation, ICLR 2018
[3] A dirt-t approach to unsupervised domain adaptation, ICLR 2018
[4] Iterative learning with open-set noisy labels, CVPR 2018

In experiments, the results are not very surprising. There are some baselines that adopt a similar (iterative) pipeline (learning the network - selecting a subset of the training samples - re-learning the network):
[1] Iterative Learning with Open-set Noisy Labels, CVPR 2018
[2] Dimensionality-Driven Learning with Noisy Labels, ICML 2018
[3] Symmetric Cross Entropy for Robust Learning with Noisy Labels, ICCV 2019
The authors can consider to compare to some of these baselines, especially [1] and [2]. The difference between the paper and [1,2] is basically the criterion to construct the maximal safe subset.

Besides, I suggest the authors to conduct large-scale experiments on ImageNet or even a subset of ImageNet, since the difficulty of detecting label noise is much higher when the resolution of images become bigger. CIFAR-10 and CIFAR-100 only contain 32x32 images, which is far less challenging.

Overall, I think the paper is well written, the idea is clearly presented, and the experiments also seem convinceing. However, the contribution of this paper is very incremental.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents a training approach on label noise datasets and outperforms state-of-art methods. It defines the samples whose average probability on assigned label in recent q iterations is largest among all labels as memorized samples, in the sense of the network memorize these samples. Then authors proposed two stage method which firstly early-stops at minimum validation error (or $\tau$ memorized rate), and then trains on maximal safe set that gathers memorized samples. The experiments compared several state-of-art approaches and showed that the proposed method benefits from early-stopping and safe set. Authors also showed that the prestopping idea can also be used to improve other approaches.

Pros:

The proposed method achieves better performance than state-of-art methods.

Authors have good experiments which evaluate on multiple datasets and algorithms.

Authors also investigate the relation between model complexity and performance of co-teaching+

Cons:

Many recent papers indicate the “error-prone period”, authors should include related works about early-stopping on label noise training.
https://arxiv.org/pdf/1901.09960.pdf fig1
https://arxiv.org/pdf/1903.11680.pdf fig5
https://arxiv.org/pdf/1906.05392.pdf fig3

Although the method achieves good performance, since the idea is a bit straightforward especially after exploring above papers, I am slightly worried about novelty of the ideas.


review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a two-phase training method for learning with label noise. 

On the positive side, this paper focuses on the idea of prestopping and proposes several relevant definitions to formalize their idea and come up with a heuristic algorithm. 

However, I believe the paper has missed several very relevant papers that provides very similar ideas. Both [2] & [3] provide theoretical analysis to why early stopping matters in learning with label noise for DNNs. Before these two papers, [1] also observed that the learning trajectories for clean and noisy samples are different in label noise problem, and they used early stopping in their experiments to address this issue. Given these existing literatures, the contribution of this paper should be considered more properly. 

[1] Learning with Bad Training Data via Iterative Trimmed Loss Minimization, Yanyao Shen, Sujay Sanghavi, ICML 2019.
[2] Hu, Wei, Zhiyuan Li, and Dingli Yu. "Understanding Generalization of Deep Neural Networks Trained with Noisy Labels." arXiv preprint arXiv:1905.11368 (2019).
[3] Li, Mingchen, Mahdi Soltanolkotabi, and Samet Oymak. "Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks." arXiv preprint arXiv:1903.11680 (2019).

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes to study how early stopping in optimization helps find confident examples. Overall, the paper is well-organized and easy to read. Although there is some parallel study regarding the theoretical aspect of how early stopping help finds confident examples (i.e., Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks, which has unfortunately not been cited), the paper focuses on the empirical perspective. A thorough empirical study illustrating how early stop works would interest the label noise community.

The authors claim that early stopping is efficient to find a maximal safe set. I think it would be necessary to illustrate the maximal safe set for all the datasets. The authors only did this for one case of CIFAR-100, which is not convincing. The small loss based learning has shown the confident examples extracted. It would be essential to compare the proposed method with those methods.

The authors are suggested to compare the proposed method with more baselines. There are lots of algorithms exploiting the transition matrix and with statistically consistent estimators. The authors have ignored all those methods. 

It is unclear from the paper that if the baselines have used the clean validation sets. For fair comparison, those clean data should be used in the training procedures of the baselines.

Clothing1M is a more challenging dataset with real-world label noise. The dataset also includes some clean data for validation use.  The authors should verify the effectiveness of the proposed method on this dataset.

The authors are suggested to make it clear why noise rates are sometimes available for use.
review_title: Official Blind Review #2

Meta-review: This paper focuses on avoiding overfitting in the presence of noisy labels. The authors develop a two phase method called pre-stopping based on a combination of early stopping and a maximal safe set. The reviewers raised some concern about illustrating maximal safe set for all data sets and suggest comparisons with more baselines. The reviewers also indicated that the paper is missing key relevant publications. In the response the authors have done a rather through job of addressing the reviewers comments. I thank them for this. However, given the limited time some of the reviewers comments regarding adding new baselines could not be addressed. As a result I can not recommend acceptance because I think this is key to making a proper assessment. That said, I think this is an interesting with good potential if it can outperform other baselines and would recommend that the authors revise and resubmit in a future venue.
Decision: Reject