Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors discuss potential weakness of previous sample selection criteria and propose new selection criteria based on the uncertainty of losses (not losses themselves).  Some theoretical analysis on the criteria is provided.  Then, they experimentally validate the proposed algorithms (CNLCU-S and CNLCU-H), showing that the proposed algorithms outperform some baselines in learning with noisy labels.
main_review: Distinguishing between mislabeled examples and hard examples is a fundamentally difficult problem and worthwhile to investigate.

The key idea is to incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses.  This idea is definitely reasonable, but is not totally new.  Although the details are different, batch selection based on uncertainty has been actively discussed in this field.  For example, refer to ActiveBias (NeurIPS 2017).

The assumption that the loss conforms to a Markov process is not reasonable.  Thus, Theorem 2 based on this assumption may not work well with a real loss function.

My main concern is on the experiments.  The authors argue that they did not compare the proposed algorithms with the state-of-the-art algorithms (e.g., SELF and DivideMix), because they focus on sample selection mechanisms.  I understand the authors' intention.  However, I believe that this is not acceptable for this top-tier conference, because the proposed algorithms should be proven to improve the cutting edge performance.  Sample selection alone may not be very convincing, because combining various techniques is a clear trend these days.

According to a nice survey, https://arxiv.org/abs/2007.08199, there are recent sample selection methods, which are not covered in this paper.  Please see Section III.E.  Even though the authors want to focus on sample selection, it would be necessary to cover recent advances more comprehensively.

Also, it is common to include at least two real-world noisy datasets, e.g., WebVision, Clothing1M.

Typo:
Line 217, selectionin

===================

After the author feedback: the authors have successfully addressed my concern on the experiments (lack of the comparison with the state-of-the-art algorithms). Thus, I am willing to raise my rating to 6.
limitations_and_societal_impact: I think that there is no potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces the uncertainty of losses into the sample selection approach for learning with noisy labels. The authors discuss how to effectively exploit underlying clean examples that are less selected during training. Experimental results verify the effectiveness of the proposed method. 
main_review: The sample selection approach is popular and useful in learning with noisy labels. Motivated by the fact that deep networks first memorize the training data with clean labels, the training data with small losses can be seen as clean data and used for optimization. This paper considers the uncertainty of losses in the sample selection procedure, which consists of two aspects: the uncertainty of small-loss data and the uncertainty of large-loss data. The authors use the concentration inequalities and confidence bounds to study these uncertainties at the same time. The experiments in this paper are comprehensive. The results are also promising, especially when the training data are noisy and imbalanced. 

Strength
- Strong motivation. This paper is motivated by experimental results, which show that sometimes the loss values cannot fully reflect the purity of an example. When the data are noisy and imbalanced, the losses of mislabeled data and clean imbalanced data are almost equal and hence cannot be used directly for sample selection. Besides, the loss achieved at a certain epoch may not be trusted. The results are clear and well motivate this paper. 
- Great writing. The logic of this paper is smooth. The authors first reveal the issues of prior effects and then present the proposed method step by step. Most of the technical details are described clearly. 
- Theory. This paper provides theoretical analyses for two robust mean estimators, which could guide the process of sample selection during training. 
- Experiments. The experiments of this paper are comprehensive. The authors consider five types of synthetic label noise and real-world label noise, following good experimental results. Detailed experimental analyses and ablation study are also provided. 

Weakness/Question
- Learning with noisy imbalanced data is challenging. Expect for the results (Figure 1 Right) on the MNIST dataset, would the results on the other datasets be similar to the case on MNIST?
- The methods CNLCU-S (H) are built on Co-teaching. It seems that the proposed methods can also build on other sample selection methods. Could the authors further add such discussions and results?
- The proposed method relies on a noisy validation set to find suitable hyperparameters, i.e., $\sigma$ and $\tau$, which makes the methods more complex. 
- The authors use ResNet-18 for the experiments on Clothing1M. I agree with the claims of the authors. The proposed method cannot be competitive with DivideMix. However, most of the existing methods use a ResNet-50 network. It would be better if the results with ResNet-50 can be provided. 
- The paper discusses the weaknesses of existing methods on sample selection at a high level (Section 1). Could the authors provide a more detailed algorithm flow like Algorithm 1 to explain this more clearly?

It is expected that the above issues/concerns could be addressed. 


limitations_and_societal_impact: The proposed method introduces extra hyperparameters into training, which cannot be determined adaptively. Although the results look very promising, it is better if such issues can be addressed. 

The authors should discuss the negative societal impacts of this work like the possible model bias from the data.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a CNLCU strategy that incorporates the uncertainty of losses by adopting interval estimation.  In learning with noisy labels, the sample selection approach regards small-loss data as correctly labeled during training.  This paper gives large-loss data a try to explore underrepresented data that are correctly labeled but seem to be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types. This paper opens up new possibilities in the topics of using sample selection to handle noisy labels.
main_review: In learning with noisy labels, the sample selection based on small-loss criteria is the most common method. This paper reveals the limitation of small-loss criteria and proposes a new selection criterion based on uncertainty to give large-loss data a try. The theoretical analyses are detailed and the claims are well supported by experimental results.  This paper adopts interval estimation instead of point estimation, where the lower bounds are used for sample selection. In this paper, the writer focuses on promoting the prior sample selection. However, this paper is not well organized and there is quite a bit of room for improvement in the representation. The overall writing is not fluent and the theorems are lack of interpretation. Besides, the penultimate paragraph in the introduction is a little bit confusing and figure 2 is worthy of improvement.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The submission considers the problem of learning from noisy labels, and propose to incorporate an interval estimate of training loss to the sample selection approach. The intuition is to distinguish between mislabelled data and under-represented data. Theoretical and empirical results shows the promise of the method.

Three reviewers have carefully considered the strengths and weaknesses of the paper. During the author rebuttal period, the authors put in significant effort to address some concerns about empirical results raised by the reviewers. The reviewers appreciated the clarifications, and one reviewer increased their score. There was unfortunately no substantive discussion among the reviewers post rebuttal.

There are many high quality papers submitted to NeurIPS each year, and this year is no exception. Many papers are on the borderline, with reviewers finding the problems important, but also found weaknesses in the paper. Unfortunately this paper could not be accepted into this year's program.
Decision: Reject