Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper identifies label errors in test sets of popular vision, text, and audio datasets using confident learning. The detected label errors are validated by crowdsourced workers from Amazon MTurk. It is important to ensure all the test labels in benchmark datasets are correct. The core finding in this paper challenges fundamentally the reported benchmark results in the literature so far. The finding that, higher capacity models generalize well on the original incorrect test labels but generalize worse on the corrected test labels than simpler counterparts, is interesting.
strengths: 1. It is interesting to find and validate the label errors in test sets of popular benchmarks. The open-source corrected test labels would be impactful.
2. Correcting all the test labels by brute force is expensive. Thus implementing pre-selection algorithmically as employed in this paper is important. Besides, it is worthwhile to deal with real-world human-level label noise rather than synthetic label noise.
3. The paper corrects test labels for many popular benchmark datasets that are heavily used in the literature. 

weaknesses: 1. Only a subset of the label errors had been corrected by humans. It is unclear how many label errors are missed. A complete set of corrected labels should include all label errors.
2. The author only validates the label errors guessed by confident learning, making the finding on model capacity not convincing enough. This is because the label errors guessed by confident learning may not be independent. It is unclear whether we can get the same finding if we have a subset of corrected labels independently sampled from the whole set with wrong annotations. Thus, the currently used corrected labels in Fig. 3b may not represent the property of the other undetected label errors. An experiment to show the difference between the real label errors and the current subset may be necessary.

correctness: Most of the claims are correct. But it is unclear whether the selected subset of label errors can represent the other undetected label errors. Please check the details in weakness.
clarity: Yes. The paper is well-organized and easy to follow.

relation to prior work: The paper well positioned in the literature of learning with noisy labels. The authors also did a great job motivating the current study and justified why this is an important missing piece in the literature. 
additional feedback: 1. The paper would be better presented if the above weakness could be fixed.
2. It would be helpful if the author can provide more intuitions on why a larger model performs worse in the corrected label set.
3. Other label error detection methods may also be helpful.
review_title: Solid work with interesting findings and potential high impacts

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: summary and contributions: This paper, for the first time, investigates the pervasive existence and important consequences of test-data errors in 10 popular ML datasets, including ImageNet, MNIST, CIFAR-10, etc. The paper uses confident learning techniques to propose test-data error candidates and then asked mechanical turkers to confirm or verify the data errors. The contribution of test-data error correction is very important to the ML/vision/NLP fields as these datasets are so popularly used. This paper also produces a significantly important observation that some simpler networks (ResNet-18) sometimes work better than complicated ones (ResNet-50) over data with error corrections, which reveals an important fact that the big-capacity models may fit too well to the training split, even to the data errors there. Overall, this is a very strong submission (and very impactful to the field) that I would even consider as a strong candidate competing for the best paper award of this track.
strengths: 1) This paper, for the first time, studied the pervasive existences and important consequences of test-data errors in 10 very popular datasets, including ImageNet, CIFAR-10, MNIST, etc.
2) The paper finally produced data-error corrections in these ten important datasets, which could benefit the ML/CV/NLP communities very much.
3) This paper uses the error-data corrections to reveal some important consequences, including a) big-capacity networks may fit too much to the training data-errors; b) some simpler network, e.g. ResNet-18, may work better than ResNet-50, over the corrected error-data. All of these conclusions are super important to the field.
4) This paper is very well-written, easy-to-read, ready-for-publication.
5) The authors also provided the codebase for reproducing the results, a website for data browsing, detailed statistics/data annotation process, etc..
weaknesses: I don't think this paper has major weaknesses. Below are some small suggestions:
1) For QuickDraw and AmazonReviews, the authors can discuss how do you plan to finish them? or, any other ways to scale up the verification?
2) The information in the current Sec. 5 is too dense. To make it easier to read, I suggest the authors break the materials into some sub-sections or paragraphs.
3) Is it planned to also fix the annotation errors in the training split? Is the same CL framework scalable to fix errors in the training set?
correctness: yes
clarity: yes
relation to prior work: yes
additional feedback: no

Post-rebuttal: I want to keep my rating.
review_title: An excellent submission that may compete for the best paper of this track

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This is an important and interesting work concerning label errors in commonly used benchmarks.

Contributions

* analysis of label errors in test sets of 10 commonly-used datasets
* release of cleaned tests for the data sets
* analysis of implications of label errors in benchmarking neural models
strengths: The problem of label errors in test sets has not been given enough attention so far and the authors are filling an important gap here: they use confident learning to find errors in test sets. In paper, it is shown that such errors can have important implications when models are compared.
weaknesses: The main weakness of the paper is that the "dark matter" of hard-to-detect label errors is not considered and discussed. It might be the case that such errors have different impact for simpler and complicated models. I think the authors should run their procedure for the manual identification of label errors on random samples as an additional step to compare the statistics with the errors found with the help of confident learning.

Minor issue: The label "flying saucer" in an example given in Figure 1 is quite arbitrary, I'd say. I think it merit some discussion, problems like this.

Another issue is that the variance of error rate across the 10 test sets is quite large (10% for Quick Draw vs <2% for 3 data sets), this should be discussed.
correctness: In general, the submission is correct, though see the discussion of confounders in Weaknesses.
clarity: The paper is very well written, no remarks here.
relation to prior work: I did not find any issues here.
additional feedback: Minor remark:

* 178: spelling "practioners"
review_title: Label Errors in Test Sets

Meta-review: This dataset concerns label errors in test sets for ML benchmarks; the reviewers found it well-documented, motivated, and overall a compelling contribution.
Decision: Accept