Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides an extensive study on 1) the difference between human annotations and synthetic ones and 2) the impact on the classifier learning from human annotations and synthetic ones. To this end, the paper crowdsources the human annotations for CIFAR10 and CIFAR100 from Amazon Mechanical Turk. The observations show that human annotations are more challenging to learn, and human annotations result in a different memorization scheme for neural networks.
main_review: Overall, the paper provides great value to the research community in not just learning from noisy labels, but also data labeling [1]. The paper is easy to follow, and the observations are comprehensive.

**Strength**
- The paper systematically compares human noise and synthetic noise (including class-dependent label noise and instance-dependent label noise) thoroughly.
- The paper compares human and synthetic labels in terms of the label distribution and the corresponding model.

**Weakness**
- (minor) The paper can make some connections with the worker simulations in data labeling. They also adopt a similar class-dependent scheme for worker simulations. [1,2, 3]


**Question**
- In the bottom row of figure 7, are the predictions compared to the given label (wrong labels) or the ground truth labels?

**Other**

Some observations (3 and 5) coincide with the observations in prior work [1]. Prior work [1] considers class-dependent label noise but initializes the label noise with crowdsourced human noise. It also observes that using the human label noise is more challenging in data labeling.

[1] Y. Liao, A. Kar, and S. Fidler. Towards good practices for efficiently annotating large-scale image classification datasets, CVPR2021

[2] G. Hua, C. Long, M. Yang, and Y. Gao. Collaborative active learning of a kernel machine ensemble for recognition, CVPR2013

[3] C. Long and G. Hua. Multi-class multi-annotator active learning with robust gaussian process for visual recognition, CVPR2015

summary_of_the_review: The paper provides great value in the research communities that focus on the impact of label noise. The paper is well written and easy to follow. The observations shown in the paper are supported by extensive analysis. There are only some parts that require more explanation, but it does not harm the overall readability.
To make the paper more complete, I suggest the author make more connections toward a broader community that focus on label noise.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper collects additional human annotations for Cifar10 and Cifar100. These annotations are also compared to synthetic label noise alternatives.
main_review: Strengths:
- The paper is clear and well-written.
- The annotation collection procedure was clearly described.
- A large number of baselines were implemented and compared in Section 5.1.

Weaknesses:
- How does the annotation collection procedure differ from the original CIFAR10 & CIFAR100? Are there any important differences or modifications to the process the authors made? These points should be addressed.
- The "qualitative" section on instance-dependent noise comparisons is confusing and does not help the overall flow of the paper. Figure 5 is more confusing than helpful, and there is no real "conclusion" from the subjective analysis until we get to the quantitative section with the hypothesis test.

-- UPDATE after rebuttal --
The authors have sufficiently addressed my concerns regarding the differences with respect to CIFAR-10H. Thus, I will raise my score from 6 to 8 with the understanding that the authors will update the paper to include the information they provided in their rebuttal reply.
summary_of_the_review: The paper collects additional manual annotations for CIFAR10 and CIFAR100, which will be of great value to the research community, for ex. helping practitioners study the robustness of their algorithms to label noise.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two smaller-sized datasets with real-world label noise. The datasets contain CIFAR-10 and CIFAR-100 images with human-annotated labels. The paper analysis the noise pattern, and benchmarks existing methods on these datasets.
main_review: Strengths: The paper highlights an important issue in the literature of label-noise-learning: synthetic noise is prevalently used to evaluate methods. This paper shows that real-world noise pattern is different from synthetic noise, and the datasets proposed could be useful as a smaller-scale benchmark for future researchers. The paper is also well-written in general.

Weaknesses: 
1. Even though the proposed datasets offer an effective benchmark for researchers to conduct smaller-scale experiments, its significance is not enough. There already exists larger-scale real-world noisy datasets such as Food-101 and WebVision. Because the ultimate goal of label-noise-learning is to learn from weakly-labeled web data, these existing web datasets are sufficient for model evaluation. The paper argues that these datasets do not have clean training labels. However, it should not be a problem because they do have a clean test set for fair evaluation. Therefore, in my opinion, the proposed datasets do not address a research gap.
2. Besides the dataset, the paper does not offer other significant contributions. It can be expected that real-world noise has different distribution compared to synthetic noise. As a dataset paper without any technical novelty, I do not find this paper to provide much new insights that could guide future research directions in this field.
summary_of_the_review: While I find this paper to have a clear strength, the contribution is not enough to justify acceptance to ICLR. The dataset proposed is good but not crucial, and the paper does not provide enough new insights. This paper might have a better chance at other venues specific for dataset papers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on learning from noisy labels, which is a very important research topic in the ML community. One of the main challenges in this area has been the absence of controllable real-world noisy data. The authors present the variation of CIFARs data with human annotations via MTurk.  By comparing the results on the original CIFARs and the newly proposed CIFAR-N, four observations are detailed quantitatively or qualitatively. In addition, they provide the performance comparison of more than 10 approaches on CIFAR-10 and -100N. The analysis of the memorization effect in human-annotated noisy data is also discussed.
main_review: This paper has several strengths although there is no technical novelty. In particular, the analysis with human-annotated label noise (using CIFAR-10N and CIFAR-100N) provides some insightful information. However, I have some critical concerns below:

First of all, I agree with the absence of controllable real noisy data, which can be used popularly for fair comparison or in-depth analysis (satisfying the three conditions: resolution, clean labels, and no interventions). For this purpose, CIFAR-10N and CIFAR-100N will be good reference data for research. However, I wonder if the data with such a small resolution (32x32) can properly reflect real noise. In real-world applications, the resolution of training images is mostly larger than 32x32, and the corrupted labels happen even with high-resolution images (e.g., medical images or data with fine-grained classes). I think that the restriction of the low resolution (actually, 32x32 is not moderate resolution) can generate label noise attributed to ambiguity in the scene (e.g., color information could be dominant than appearance information in labeling). Therefore, the collected noisy labels may be different from the noise label in a real real-world scenario.

Second, the observation of imbalanced annotation is quite interesting. Owing to this imbalanced annotation, I think CIFAR-10N and CIFAR-100N have class imbalance problems. Can authors report the imbalance ratio of the two proposed datasets? If the ratio is low, the results in performance evaluation are convincing. However, if the ratio is high, the low performance of robust approaches can be due to the more complicated scenario of 'class imbalance + label noise'. Similarly, in all other analyses, I am wondering if the class imbalance factor was properly handled. 

Third, many readers will be interested in why the existing approaches show poor performance on human noise. The table summarizes many values but I feel hard to reason why ELR achieves the best performance consistently. In addition, why does the DivideMix shows the worst performance in the CIFAR-10N aggregate dataset? There is no discussion and analysis on these points. I know, that human noise is like a feature-dependent noise, thus it is much harder than the synthetic noise setup. But, why do all methods show inconsistent trends on that?

These are some minor comments:
- Clothing1M data has a 38% noise level (see the paper [1], the authors reported the clean accuracy of labeling was 61.54%)
[1] Learning from massive noisy labeled data for image classification, CVPR'15
- The explanation of Figure 7 is very confusing. e.g., what is the 'wrong' prediction on 'wrong 'labels? and there is no f_M in the paper (see the caption for Figure 7) Is it f_H?

summary_of_the_review: As a paper that proposes insightful analysis and some datasets, there is room for further improvement. In particular, resolving the above concerns will add more clarity and novelty to the paper. I will decide my final rating after the rebuttal.

> [rebuttal] The score increases 5 $\rightarrow$ 6 since the response addressed my concerns properly.
review_title: null

Meta-review: The authors propose two new benchmark datasets CIFAR-10-N and CIFAR-100-N, variants of CIFAR-10 and CIFAR-100 with real-world human annotation noise. The benchmark datasets are more realistic (e.g. instance-dependent noise) than some existing synthetic benchmarks for label noise. The authors also benchmark several popular baselines on the proposed benchmark

All the reviewers thought that this is an useful contribution to the community and appreciated the detailed author response. The consensus decision leaned towards accept. I recommend acceptance & encourage the authors to address any remaining concerns in the final version. 
Please clarify the license (e.g. MIT license) when you release the dataset.
Decision: Accept (Poster)