Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new method of detecting label noise through calculating the Area Under the Loss statistic, which is based on the learning trajectory.  

I believe this paper is NOT the first one to point out the different training behavior of clean and noisy samples in the label noise problem. While the author mentioned that Shen & Sanghavi used the training losses for selecting data, Shen & Sanghavi also observed the training loss for good and bad samples are different across time. Therefore, it would be good to give correct credit to previous work, and not over-emphasizing the contribution. 

Please proofread the paper, and correct the typos. For example, on page 3, there are 2 typos in the last three lines in the paragraph “rate of loss reduction indicates memorization need”. 

Can you explain why you normalize the loss of each batch by c ? If z_i s are all small within a batch, this loss function does not reweight the bad samples in the correct way.

Can you explain why you use Pearson’s correlation instead of Spearman’s correlation to evaluate the consistency of the ranking? This sounds surprising to me. 

Also, the cited paper “Understanding generalization of deep neural networks trained with noisy labels. “ by Hu et al. is not in NeurIPS 2019. Please correct this citation error if possible. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed methodologies to identify mislabeled training examples. Specifically, the authors proposed to examine the training loss of each example to identify examples with high training losses. To mitigate the randomness in the training loss, the authors proposed an AUL (area under the loss) metric, which is less noisy that directly examining the loss. The authors proposed to use Gaussian mixture to approximate the AUL distribution, and then estimate the probability of mislabeling. The authors proposed to downweight training examples with high likelihood of being mislabeled. The proposal is examined in numerical studies.

While I agree the method has a good potential to be significant, in its current form, I have several concerns that prevent me from recommending acceptance.
1. How can we be sure that examples with large training loss are mislabeled examples, rather than examples that are inherently difficult to classify? For example, if the dataset consists of 99% bright images and 1% dark images, then it could be possible that dark image training examples have a higher loss. This question becomes more relevant when the model does not have sufficient complexity, or does not have the correct form of complexity.
2. If we cannot distinguish mislabeled examples with examples that are hard to classify, then it seems to me that a simple downweighting based on AUL could potential lead to the opposite outcome. Intuitively, we should upweight those difficult to classify examples to improve generalizability.
3. It is somewhat surprising to me that the AUL distribution is a nicely looking bimodal shape. Could it be due to the fact that the wrong labels are randomly generated with equal probabilities from other classes? What it we have a different error distribution, e.g., the label has p probability of being correct, and (1-p) probability of being k, where k is a constant?
4. Given the above concerns, I feel the numerical studies are insufficient. The authors may consider 1) including more noise distributions, 2) presenting the AUL distributions to check whether they are truly bimodal, and 3) trying more datasets and more model architectures.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Pros:
1. This work suggests a surprisingly elegant approach to identify data with corrupted labels. The proposed AUL (area under the loss curve) makes intuitive sense, and exploiting the fact that noisy examples are hard to learn at the early stage of training, leading to higher AULs than clean examples.
2. The paper is well-written, with plenty of visualization to aid the understanding.
3. Experimental results suggest advantage of the proposal. Evaluation seem to suggest the using AULs is more robust against large corruptions compared with SOTA.

Cons:
1. The biggest concern is the generalizability of AULs to datasets that are more challenging than CIFAR10 and CIFAR100. This has been the concern not only for this work, but also for this general research area. For instance, on Tiny-ImageNet dataset (e.g., used in SELFIE: Refurbishing Unclean Samples for Robust Deep Learning (ICML19), which could btw be used in the benchmark), a clean dataset only gives ~55% test error, compared with about ~10% on CIFAR10 and ~30% on CIFAR100. The robustness of AULs needs to be tested on those harder problems as well. 
2. AULs are based on the assumption of a clear separation of two types of examples -- clean and noisy. It ignores the difficulty of training examples. This raises the question of where to place hard-clean examples between those two modes.
3. It usually helps to include a real-world noisy datasets where its noisy corruption is unknown. One would be curious to know the noisy level estimate.

Questions:
1. It seems from the experiments that MoGs are fit per training example in order to compute weights for the cost?
2. The computational cost of AUL-based training is not mentioned. Is it 2x, 3x, considering there are distinct stages of training before and after AUL estimation?
review_title: Official Blind Review #1

Meta-review: The paper proposes a new, stable metric, called Area Under Loss curve (AUL) to recognize mislabeled samples in a dataset due to the different behavior of their loss function over time. The paper build on earlier observations (e.g. by Shen & Sanghavi) to propose this new metric as a concrete solution to the mislabeling problem. 

Although the reviewers remarked that this is an interesting approach for a relevant problems, they expressed several concerns regarding this paper. Two of them are whether the hardness of a sample would also result in high AUL scores, and another whether the results hold up under realistic mislabelings, rather than artificial label swapping / replacing. The authors did anecdotally suggest that neither of these effects has a major impact on the results. Still, I think a precise analysis of these effects would be critically important to have in the paper. Especially since there might be a complex interaction between the 'hardness' of samples and mislabelings (an MNIST 1 that looks like a 7 might be sooner mislabeled than a 1 that doesn't look like a 7). The authors show some examples of 'real' mislabeled sentences recognized by the model but it is still unclear whether downweighting these helped final test set performance in this case. 

Because of these issues, I cannot recommend acceptance of the paper in its current state. However, based on the identified relevance of the problem tackled and the identified potential for significant impact I do think this could be a great paper in a next iteration. 
Decision: Reject