Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper reads well, the topic is interesting, and the connection between early stopping and fitting true label distributions before noisy ones, if generally true, is of potential impact. I do not feel the paper is ready for publication, though: 

Missing context/references:
	⁃	Noisily collected labels are standard elsewhere, e.g., in sentiment analysis (self-ratings), in discourse parsing (explicit discourse markers), in weakly supervised POS tagging (crowdsourced dictionaries), in NER (Wikidata links), and in machine translation from mined parallel corpora or comparable corpora. 
	⁃	Learning a ground truth from a population of turkers (MACE and subsequent work). It is well-known that the noise in such silver standard or non-adjudicated annotations is also “more structured” and to a large extent predictable (Plank et al., EACL 2014 and subsequent work on learning/predicting inter-annotator disagreements). 
	⁃	Connection between L2 and early stopping. Can you replicate your results with simple L2 regularisation?
	⁃	Cost-sensitive learning of systematic disagreements (use class coherence scores for cost-sensitive learning). 
	⁃	The observation that “clean examples are fitted faster than noise” is obviously related to baby steps training regimes (training on easy examples first), including active learning. 

Experimental details and flaws: 
	⁃	Using a black box search engine to collect labels is problematic for a few reasons: (a) Search engines change, so results are hard to reproduce. (b) Search engines are biased toward certain types of categories. Such biases will be reinforced by any model trained on this data. 
	⁃	I’m always a little worried about papers that only report results for a single dataset. 
	⁃	I would have liked to see some more error analysis. Are the improvements on classes with more or less support in the original dataset, for example?
	⁃	It seems to be that it would have been relatively straight-forward to construct synthetic datasets that would more directly evaluate the hypothesis that the true labels are learned first. I realize you’re interested in “real” noise rather than “random” noise - but synthetic noise doesn’t have to be random. 
	⁃	That “the noise generated through the label collection process is not nearly as adversarial for learning as the noise generated by randomly flipping labels” is no surprise, and I would not present this as a finding. It would be interesting to describe the bias, e.g., by presenting confusion matrices (see Plank et al., EACL 2014). 

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Summary

The paper exploits the training dynamics of deep nets: gradient descent fits clean labels much faster than the noisy ones, therefore early stopping resembles training on the clean labels. The paper shows that early stopping the training on datasets with noisy labels can achieve classification performance higher than when training on clean labels (on condition that the total number of clean labels in the noisy training set is sufficiently large).

The paper also makes the point that noise introduced during data collection are different from artificially generated noise through randomly flipping the labels of a clean dataset. The latter is often done in the literature. The “real” noise is more structured and therefore it is easier to fit and less harmful to the classification performance.

Strengths

The paper is well written. The results are very interesting even though they are very intuitive and simple.

Weaknesses

The idea seems to be known. For example,
Better Generalization with On-the-fly Dataset Denoising
https://openreview.net/forum?id=HyGDdsCcFQ

The paper talks about early stopping. As shown by the above paper, it is also a function of the learning rate. Please comment on what happens in the case of small learning rate and early stopping.

It would have been great to prove the theorem for deep learning. The result is limited to linear models with large number of random features.

The paper does not make it clear under what conditions early stopping prevents the model from memorizing bad labels.

The paper focuses on classification. Will the claims hold for other task types such as object detection, segmentation, etc?

review_title: Official Blind Review #3

Meta-review: The authors present an approach to learning from noisy labels. The reviews were mixed and several issues remain unresolved. I do not accept the following as a valid response: "We fully agree that noisily collected labels are common for many problems other than image classification. However, the focus of our paper is image classification, and we thus concentrate on classification problems related to the widely popular CIFAR-10 and ImageNet classification problems." ICLR is a conference on theoretical and applied ML, and the fact that a technique has not been used for image classification before, does not mean you bring something to the table by doing so. The NLP literature is abundant with interesting work on label noise and should obviously be considered related work. That said, there's also missing references directly related to the connection between early stopping/regularization and label bias correction, including: 

[0] https://arxiv.org/pdf/1904.11238.pdf
[1] https://arxiv.org/pdf/1705.03419.pdf
[2] http://proceedings.mlr.press/v80/ma18d/ma18d.pdf

See also this paper submitted to this conference: https://openreview.net/forum?id=SJldu6EtDS
Decision: Reject