Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a label correction approach based on a likelihood ratio test, for robust training of deep neural networks against label noise. First, this paper introduces the LRT-Correction procedure, which is the main component of the proposed label correction approach. LRT-Correction uses the current model prediction to run a likelihood ratio test and flip labels when they are rejected. The decision is made by comparing the likelihood test results with a predefined value \Delta. Then they introduce the full algorithm, AdaCorr, where the LRT-Correction procedure serves as an inner loop for the label correction. Lastly, there are experiments done on four datasets to conclude the superior performance of the proposed AdaCorr in contrast to several existing methods. 

Overall, this paper proposes a new label correction approach based on a likelihood ratio test. Standard experiments show that the proposed AdaCorr is superior to several existing methods. 

The following questions are expected to be addressed during rebuttal:
1.	The LRT-Correction procedure introduces additional computation costs. What is the difference in computation costs between Standard and the proposed label correction approach? Is the extra computation cost significant? 

2.	The LRT-Correction procedure is introduced based on a binary setting. The main theorem (Theorem 1) also only supports the binary setting. There is a statement in Corollary 1 that “LRT-Correction can be generalized to multiclass classification tasks, by flipping \tilde{y} to be the best prediction of f when the null hypothesis is rejected. Theorem 1 can be generalized to multiclass classification tasks, by considering all pairs of class values.” How exactly did you do for that? Please provide more details.

3.	This paper uses the ablation study to demonstrate that the proposed AdaCorr is robust to several important hyper-parameters, e.g. the number of epochs m for the burn-in stage, the predefined value \Delta for likelihood ratio test. How did you exactly choose the optimal value for these hyper-parameters? These is a statement “We choose m=20 in this data set (CIFAR10) and “similarly” in other datasets.” Did you use the same m(=20) for all datasets? Does this also hold for the hyper-parameter \Delta ?

4.	The experiments are too standard. Any results on real-world datasets, e.g. Clothing1M [1]？

Minor comments:
1.	The summarization of the existing related work is not consistent throughout the paper. In Introduction section, this paper believes that the existing methods mainly follow two directions, i.e. probabilistic reasoning and data selecting; while in Related Work section, they are classified into three categories. Please clarify your arguments. 

2.	Page 5: In Corollary 1, “LRT-Correctioncan” -> “LRT-Correction can”

3.	Page 7: In Table 2, MINIST -> MNIST. 

4.	Page 7: In Experiment Setup, please provide more training details, e.g. learning rate. 


[1] Xiao, Tong, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. "Learning from massive noisy labeled data for image classification." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699. 2015. 


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new method for correcting training label noise, using a likelihood ratio test on the predicted probability of the classifier trained on the noisy labels, and then proposes an algorithm for iteratively cleaning training labels and retraining a model.  The paper also provides a theoretical guarantee for the probability of correctly re-labeling the training set, and provides empirical results showing that the proposed method significantly outperforms existing approaches for handling noisy training labels.

Overall, I think the empirical results appear very strong, but think this paper is below the acceptance threshold due to three factors, in ranked order:
- (1) The theoretical guarantee, which is positioned as a core contribution of the paper (and in fact claims it as "the first to correct labels with theoretical guarantees", which is not true), is based on assumptions that seem overly strong; these are somewhat relaxed in a "Remark", but this seems unproven and is a confusing presentation regardless.
- (2) The theoretical bound itself is somewhat vacuous as it contains several constant factors that seem very material to the bound, but totally opaque to the reader.
- (3) The experiments are very strong overall, which is a major plus for the paper; however, there are some questions about the hyperparameter tuning and some other points where more clarity could improve the strength of the empirical results

Regarding (1):
As a reader, my first natural reaction was to worry about circularity/degeneracy in the proposed method: basically, we are using the confidence (as a ratio of predicted cond. probabilities) of the model trained on the corrupted labels to correct those labels... if the labels are so corrupted that the model is also way off, then intuitively, this method should not work.  I was wondering about how this situation would be bounded / handled.

It turns out in Thm 1 that an incredibly strong assumption is made, namely that the model trained on the corrupted labels, f, is a linear function of the true model, with constants a, b known to some small degree of error epsilon (note that the theorem statement says that these constants are unknown- but it then assumes that \Delta, which is set based on a and b, is known up to \epsilon error).  This seems like an incredibly strong assumption- and no context / motivation is given about why it should be taken as reasonable.

Then, immediately after the Theorem, "Remark 2" states that this condition is not actually needed at all- but (a) then why not just strike it from Thm 1 statement, and (b) there does not seem to be any proof of this Remark in the appendix (where the proof of the main theorem itself is closer to a sketch than a standard proof...).

Regarding (2):
The bound produced in Thm 1 seems somewhat vacuous: letting \tau_{01} = \tau_{10}, then the probability of the label correction being erroneous is bounded by 8C(O(\epsilon))^\lambda.  This quantity is presumably in (0,1/2], so it's a small range to start... but it seems hard to get anything from this bound without some idea of what the constant factors (C, and those hidden in O(\epsilon)) and \lambda are.  In particular, as presented, it seems implausible that \epsilon- the error in specifying the \Delta threshold- gets that small, in which case these constants become very important to know!  Another way of phrasing this remark: many theoretical bounds have lots of unknown constant factors, but are ultimately just trying to expose some scaling with respect to one parameter, e.g. number of data points, and therefore the constants don't need to be known that well for the statement to have some value.  This doesn't exactly seem to be the case here- therefore it seems hard to extract something from this statement (even ignoring the strong assumptions it is predicated on).

Regarding (3):
Overall, I think the empirical performance reported in Table 2, and overall thoroughness of the ablation in Section 4, are major strong points for the paper- the performance is very impressive!  However I have a few questions, clarification of which would be very helpful in my mind:
- (i) A major issue that seems to be raised in the earlier sections is that there is a hyperparameter \Delta--the threshold for the likelihood ratio test--that everything depends on, and must be chosen empirically.  Table 5 shows that the effect of choosing it is not crazy, but also clearly not insignificant.  My question is: how is it chosen?  On the validation dataset?  And is this validation dataset also corrupted in the same way as the training dataset?  If not, that seems like a major whole in the setup.
- (ii) I also have a high level question for understanding: how is it possible for the various approaches to do so well with 0.6 and 0.8 noise level of uniform flipping?  In the p=0.8 noise model, for example, the probability of a data point getting flipped to *any individual wrong label* is *greater than that of it being the correct label*.  How is it possible to learn a model based on such a dataset?  Was there some kind of pre-training?  Was the validation set not corrupted?  I don't conceptually understand how the results shown are possible...?

Overall, I think points in (1) and (3) could be helped with additional clarification and contextualization, and possibly (2) as well.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Label noise widely exists in the large-scale data sets. This paper proposes a novel approach that directly cleans labels in order to train a high quality model. The proposed method leverages statistical principles to correct data labels and has a theoretical guarantee of the correctness. In particular, a likelihood ratio test (LRT) to flip the labels of training data is used, and it can prove that the LRT label correction algorithm is guaranteed to flip the label so it is consistent with the true Bayesian optimal classifier with high probability. The experimental results on several benchmark data sets show that the proposed method is promising. Overall, this paper could be a significant algorithmic contribution. But I also have some minor concerns:
[1] The theoretical analysis and the experimental results are both well organized in the paper. How about the time complexity of the proposed method. If the authors can show the time cost in the paper, I will much more agree with the paper.
[2] In the experimental parts, the convergence curve of the proposed method during the training epochs may be better to prove the theoretical analysis.
[3]The details of the compared methods should be given, and it will be better to give the results without any noise labels. In this way, the confidence of the paper will be further improved.

review_title: Official Blind Review #3

Meta-review: This paper addresses a very interesting topic, and the authors clarified various issues raised by the reviewers.
However, given the high competition of ICLR2020, this paper is unfortunately still below the bar.
We hope that the detailed comments from the reviewers help you improve the paper for potential future submission.
Decision: Reject