Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a more realistic model of label noise, in which pseudo-labels are generated by models trained on a subset of data. Typically label noise is modelled by simply randomly flipping the label, which does not actually match real-world label noise (e.g., semantically similar classes are more likely to be confused). They also suggest a "Label Quality Model" that conditions on the quality of a labeler and can be used to generate more robust labels, and find that it helps over a variety of settings. (Note: I am an "emergency reviewer" for this paper, and this is not my area of expertise.)
strengths: The Label Quality Model seems like a useful general purpose tool.

Interesting connection between the study of label noise and semi-supervised learning, which hadn't occurred to me before.

I didn't know label noise was interesting to study (my thought was "label noise sucks, but what can you do about it?")---this paper has convinced me that it is worthwhile to assume that your dataset has label noise and to try applying LQM and seeing if results improve.

Nice paper, well written.

Code open-sourced.
weaknesses: "Training machine learning models typically requires a large amount of labeled data" --> this is a pretty broad/very often not correct statement

The LQM is presented in a generic probabilistic way, but I think there are some important cases where it might not make sense, like variable-length labels in speech recognition or machine translation, where it is not possible to "interpolate" between two possible labels. Could be interesting to extend the model to this case.

Setting the interpolation hyperparameter for the LQM uses the validation set labels---but how do we know that these don't have noise too? Would it be worthwhile or possible to study the impact of label noise on the validation set as well?
correctness: Experiments are very careful and sensible; everything seems correct.
clarity: Yes.
relation to prior work: The paper carefully describes related work and how theirs improves upon it, though I don't know enough about this topic to say whether they accurately report prior work.
additional feedback: n/a

EDIT: I'm actually not sure this submission is in scope for the Datasets and Benchmarks track? "* Papers that mainly introduce a new method and don't really address any of the above are a better fit for the main NeurIPS track." Still, I'll leave my score as-is and let the ACs decide.
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This work focuses on creating realistic benchmark datasets with noisy labels. Noisy labeled datasets are generated by combining predictions from multiple models (trained with different hyper-parameter settings) and using these predictions as noisy labels. The paper claims this process produces realistic noise labels similar to the noisy labels multiple human annotators might generate. Evidence from many prior publications suggests that **this claim may sometimes be false** for datasets with noisy labels (see *Weaknesses section* below).  Further, no comparison of their methods with prior works for finding noisy labels in datasets or weak supervision (two different tasks), is included. Without some kind of comparison, it is difficult to quantify how the contributions of this work sit in the context of prior work.

Also, the background/prior work covered is lacking (perhaps severely lacking). For example, one might summarize the primary contributions of Section 1 and Section 2 as simple saving the labels of ensembles [3], yet no coverage of ensembles is mentioned. This is also related to distillation (training a single model to distill the knowledge of ensembles) [1], [2], but distillation is similarly not mentioned in this context. There was also no mention of Dawid & Skene's seminal work on multiple annotations [4]. Further, the coverage of learning with noisy labels and data cleaning approaches is lacking (specific details provided in sections below).

On the positive side, I found the contributions in Section 3 and Section 4 enlightening, namely that noisy labels are more detrimental under class imbalanced settings, when pretraining is not used, and on tasks that are easier to learn with clean labels (Section 3) and their LQM model that strengthens the ties between prior work with human annotations and their work on model-synthetic annotations (Section 4). Sections 3 and 4 were the highlights of the paper from my perspective.


[1] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).

[2] Cristian Buciluˇa, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings ofthe 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 535–541.

[3] Thomas G Dietterich. 2000. Ensemble methods in machine learning. In International workshop on multiple classifier systems. Springer, 1–15.

[4] A. P. Dawid and A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, 28(1):20–28, 1979
strengths: The strength of this paper lies in Section 3 and Section 4. Section 3 is the first I've seen to study the effects of both class imbalance and pretraining on learning with labels generated by psuedo-label learning approaches. I thoroughly enjoyed these sections of the paper. Similarly, Section 4 pushes the field of synthetic-annotations by drawing additional ties between multiple synthetic annotators (from rater models) versus multiple human annotators (in terms of history and quality). The results of Section 4 seem useful contributions to dealing with underspecification (D'amour et al. (2020)) and looking at empirical examples of model-based item response theory (versus student/human-based). Both Section 3 and Section 4 provide contributions which seem useful to the broader research community.
weaknesses: A central claim of this paper is the ability to generate realistic noisy labels, similar to multiple-annotated human generated noisy labels for a given dataset (Section 2). I am concerned that the evidence presented in the paper is insufficient to infallably validate this claim.

A possible immediate issue is that CIFAR-10 is assumed to be clean in this paper (L79 - c.f., "ground truth labels") but CIFAR-10 has been shown to contain label errors (see: https://labelerrors.com/?dataset=CIFAR-10). CIFAR-10 includes errors in the test set which is especially problematic given L89 in the paper. 

Specifically, I am concerned that realistic/human-like noise generation claim in the paper (c.f., Figure 1 (b) which shows that the predictions from a large-parameter ResNet can represent a more-expert human rater and a small-parameter MLP's predictions can represent a more-novice human rater) may sometimes be false for CIFAR-10 because CIFAR-10 contains noisy labels. Several prior works suggest this might be the case, for example, (Advani, Saxe, & Sompolinsky (Neural Networks, 2020) and (Belkin, Hsu, Ma, & Mandal (2019)) show that smaller models (resnet-18) **actually outperform** larger models (resnet-50) on datasets with noisy labels because the smaller models regularize against label noise during training. Similarly (Zhang, Bengio, Hardt, Recht, & Vinyals (ICLR, 2017)) showed that larger-parameter models are better at over-fitting to the label noise (and can exactly predict random labels with near-perfect accuracy). Similarly, we've seen evidence that smaller models can outperform larger models if the test set is sufficiently noisy when trained on CIFAR-10 with no noisy labels added (Northcut et al., ICLR (2021)). These works suggest simple models may sometimes perform on par with complex models when the dataset contains noisy labels, which goes against the claims of realistic data because of the expert/novice human rater analogy argument in this paper. I think this needs to be addressed (even briefly) for the paper to be accepted.

Some ways the authors might reconcile these issues are to: (1) use the corrected test set release for CIFAR-10 and also correct errors in the train set; (2) ignore error in CIFAR-10 and justify why that's a reasonable thing to do in terms of issues raised above; (3) possibly use your LQM approach to remove the errors in both the train and test sets.

Figure 3 seemed like a good way to substantiate the claim that the noisy-label generation process is human-like with empirical evidence, but in Figure 3, the off-diagonals of these matrices (which quantify the class-conditional label noise) appear significantly different in distribution in terms of real noise versus synthetic noise. If the paper wants to lean on empirical evidence to corroborate the claim of generating synthetic but realistic noise, that seems fine, but Figure 3 would need to more closely match real versus synthetic across a few diverse settings. Figure 3 is also missing the absolute difference of subfigure (a - human annotators) and subfigure (b - synthetic annotators) which is needed to quantify how close to 'realistic' this noise generation process is. Because no theoretical justification is provided and the main claim is supported empirically, it is also necessary to compare more than just the CIFAR-10 dataset to ensure that the empirical findings in Figure 3 generalize and the method presented can be used to synthetically produce noisy labeled benchmark data for other datasets.

For minimal acceptance the authors must provide empirical and/or theoretical justification to infallibly demonstrate that the predictions from multiple models trained with different parameter settings is a reasonable proxy for the labels assigned by human raters of various levels of expertise.

For a strong acceptance, guarantees about the conditions when the rater model predictions actually produce realistic label noise is needed. (along with a clear description of a realistic noise model)

Because a central claim of the paper (and the title) is that the generated noisy annotations are realistic/human-like, the above issues seem (to me) a major impedance to acceptance.

A (very important) weakness is the paper is the missing a comparison of the authors' noise generation approach vs prior baseline approaches for realistic label noise generation (some of which are easy to compare against via open-source packages, e.g. cleanlab). See "Relation to Prior Work" below.

A precise mathematical description of the noisy label generation process (in terms of assumptions and conditions) is missing. This is necessary since the purpose of creating these datasets is benchmarks and researchers need to know which methods to benchmark, e.g. symmetric noise? asymmetric noise? etc? Various methods address various settings. Which settings can this paper cover and can the authors make it precise?
correctness: I mention several issues above. Additionally, there seems to be an error in nomenclature used in the paper regarding prior work. The authors user the term "instance-dependent" to describe the label noise studied. But throughout the paper, the results emphasize asymmetric class-conditional label noise. This occurs in Figure 3, Criteria (b) on L107, etc. Clarification is needed and a mathematical description of the noise process studied would greatly improve the paper. However, one cannot study "instance dependent" label noise without making assumptions to clarify aleatoric (label noise) from epistemic label noise (inherent model noise in predicted probabilities). I agree that the authors' multiple-rater pseudo-label noisy label generation process doesn't inherently make a class-conditional assumption and so I understand why the authors may call it 'instance dependent', but the analysis in the paper is not instance-dependent, so I think the authors should either (1) state the asymmetric class-conditional noise assumption of the paper used for the figures and claims or (2) clarify the assumptions used for instance-dependent noise and precise/mathematically define the label noise process.
clarity: The paper is extremely clear and well-written. 
relation to prior work: This aspect of the paper needs significant improvement. Figure 3, Criteria (b) on L107, etc. refer to assymmetric class-conditonal noise processes, but no prior works are mentioned (in fact, the terms 'assymmetric' and 'class-conditional' are never mentioned in the paper). Some related works include: SCE-Loss (Wang et al., 2019), Mixup (Zhang et al., 2018), MentorNet (Jiang et al., 2018), Co-Teaching (Han et al., 2018), etc.

The biggest issue here for the paper is the missing comparison of the authors' noise generation approach vs prior baseline approaches for realistic label noise generation (some of which are easy to compare against via open-source packages, e.g. cleanlab). The authors instead compare LQM versus a vanilla model which avoids this much-needed comparison with prior art. There are several straightforward baselines for generating realistic noisy labeled datasets (the problem this paper solves) which would strengthen the paper by explicating the contributions of LQM compared to prior art. Three suggestions for baselines:
(1) Perhaps the simplest (and easy to implement) class-conditional is to compute the confusion matrix of predictions versus given labels for a dataset and use that as an approximation of the joint distribution of noisy and true labels. Several prior works consider this baseline (e.g. Hendrycks et al. (NeurIPS, 2020), Sukhbaatar et al. (ICLR, 2015), Chen et al., (ICML, 2019)).
(2) The confident learning (Northcutt, Jiang, & Chuang (2021)) approaches provide a standard benchmark comparison (see cleanlab python package). They model the joint distribution of noisy and true labels for a given dataset which can increase/decrease the amount of label noise proportionally to the distribution of real-world label noise in the dataset (by generating the noisy labels based on this noise-increased/noise-decreased joint distribution of noisy and true labels). Confident learning approaches have some theoretical guarantees for exact estimation of the distribution of label noise.
(3) A comparison with Lu Jiang, Huang, Liu, & Yang (ICML, 2020) realistic synthetic noise generation approaches. These folks are also at Google, so it might make collaboration easier for the authors.

The LQM approach used in this paper is very similar to Item Response Theory, which is not discussed. Item Response Theory (IRT) is a latent-trait model useful for estimating learner-proficiency and example hardness/difficulty. IRT is often overlooked in machine learning papers (probably because it originally was developed in psychometrics and measurement theory); I think the authors will be particularly interested in the principled nature of IRT for using multiple raters/learners to estimate the hardness/difficulty of an instance-dependent labeled-example similar to this paper which uses multiple rater models to estimate/generate noisy labels.

Why is there no mention of Dawid & Skene's work on multiple annotators? The goal of Section 4 is to show that the authors' LQM model leverages synthetic-annotator information. Dawid & Skene were the first to do this for human-annotator information.

That (1) Dawid & Skene's work is not mentioned, (2) the nomenclature on label noise processes has issues, and (3) prior work on synthetic noise generation is not compared with; may suggest that this work needs to mature further before it is ready for publication.
additional feedback: Specific feedback:

L17 - "The first step of research on label noise is to identify the appropriate datasets." -  Remove this claim as it does not strengthen the paper and its not necessarily true. Most people would say the first step is to define a noise model (in your case, instance-dependent). I would recommend starting this paragraph without a claim and instead summarize the main message of the paragraph in the first sentence (e.g., use L22 to start the paragraph, which makes the main (and interesting!) claim)

L30 - " we often have additional features of the raters, such as tenure, historical biases, and expertise level." - Citations to support this claim would help support the contributions that follow.

L64 - TYPO: "We find that the behavior of these techniques on the our synthetic datasets"

L103 - Why is this set of four criteria the right criteria? Is there a principled approach to coming up with these criteria?

L132 - Very glad to see the inclusion of the CIFAR10-H dataset here! Evaluating this dataset against your method is a highlight of this paper and might be greater emphasized. I would consider mentioning this result in the abstract.

L160 - Careful here... these papers assume symmetric uniform class-conditional label noise. They don't show that deep learning is robust to asymmetric class-conditional label noise.

Fig 3 - We need to see the diff between (LEFT) and (MIDDLE). Its too hard to compare by eye. They should be similar enough to support the claim of generating realistic or human-like noisy labels.

L215 - precisely define 'learnability'

L221 - TYPO: change 'by' to 'with' (or 'using' or 'via')


How does this work related to underspecification (D'amour et al. (2020))? Did the authors study how their various rater models tended to agree/disagree on predictions and why their approach is reasonable? Is it possible that one of the rater models is simply poorly configured such that it produces systematic error (e.g., always predicts only a few of the classes)? How do the authors control for this to ensure their generated datasets are useful?


This new datasets and benchmarks track is an important (and much needed) step taken by the NeurIPS committee to encourage a deeper understanding of data-centric methods, datasets, and benchmarks. I think an analysis of how the noise generation process in this paper alters distribution of label errors (the actual data, not just performance in models) would be very appropriate for this track. Some examples of works that focus on the dataset perspective include Shankar et al. (2020), Beyer et al. (2020), Recht et al. (2019), Tsipras et al., (2020), Taori et al. (2021), Liu, Niles-Weed, et al. (2020) ). More time spent on understanding how their generation methodology characterizes the data and an analysis of how dataset characterization from their generation method is linked to benchmarks could be helpful.
review_title: Good direction, but needs more work: no comparison with prior work, potential issues with assumptions, lacks justification that label noise is realistic

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper proposes to produce label noise by aggregating the pseudo-labels of different models. The main idea is that this form of label noise is a better model for the label noise generated by human raters as it correlates with the difficulty of the instances.
Using this partially controllable form of label noise several hypothesis on the impact of label noise are checked in experiments. Finally, a Label Quality Model is proposed to predict from given noisy labels and "rater information" the true label.

I appreciate the study in Section 2 about the possibility to model human label noise via classifiers the presentation has to be more detailed to really see the correspondence, the findings of Section 3 are known or seem quite obvious, regarding the Label Quality model the final results for CIFAR10 and CIFAR100 are worse than what one can achieve with Semi-supervised Learning Techniques e.g. Fixmatch, UDA etc with only 250 labels. 
strengths: - I appreciate Section 2 which shows to some extent (see below for more comments) that the pseudo-labels from a set of models is in certain characteristic close to that of a set of human raters using the CIFAR10-H dataset. A more closer look would help how close these relations really are

- extensive experiments on the impact of label noise in Section 3 regarding impact of imbalance of classes, pretraining, sensitivity to label noise according to the class difficulty

- the label quality model seems to improve over the standard baseline
weaknesses: - While I agree  that the aggregation of the results of different models is more correlated with human label noise rather than random label noise, it is not clear on a more individual level if this is true or not e.g. one could have checked the agreement of the entropy of predictions on test points by comparing the overall ranking of the test according to the entropy of predictions. As far as I understand the Peterson et al paper they did this only for a very strong single model on CIFAR10. It remains also unclear if the aggregation of different models is a better predictor for human label noise than the best individual model (the numbers in the confusion table for CIFAR-10 Synthetic are significantly worse than that of a good standard model on CIFAR10. It would be very interesting if there were particular classes or even subsets of classes on which human raters and models agree resp. disagree.
If the authors can provide this stronger evidence then it would be very nice, if not then the authors should avoid suggesting like in Figure 1 that the aggregation of deep models is the same as aggregating human raters.

- the y-axis label in Figure 2 is missing, the x-axis should be the same for all three plots to make the comparison easy. For the y-axis I recommend a logarithmic scale as otherwise the small counts are not visible

- Figure 4 shows curves with different k-alpha but the k-alpha seems to be computed for the synthetic dataset before subsampling, but if the disagreement is depending on the classes then k-alpha after subsampling can be quite different?
Also curves of the same k-alpha or very similar one show quite different behavior already which questions a bit the general statements made

- it seems rather obvious that pre-trained model on a similar task helps and does not help if the task is completely different, the same seems true with the finding that label noise is affecting easier tasks more than already difficult ones

- in Figure 6 (right) the linear fit looks quite noisy

- the result of the LQM with or without improvement seem to be worse than what SSL techniques also relying on pseudo-labeling resp. self-learning like Fixmatch, UDA achieve with only 250 labels on CIFAR10 resp. 2500 labels on CIFAR100 (which seems less than the number of clean labels used in the experiments, even though N seems nowhere specified). 
This part requires a much better comparison to also other similar techniques like noisy student SSL
Self-training with Noisy Student improves ImageNet classification, Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le, CVPR 2020

- the suggestion to use in the end a convex combination of LQM and noisy label is unmotivated and introduces another hyperparameter

- it remains completely unclear what the LQM model really learns, there is also no discussion of this anyhwere



correctness: Apart from the questions and concerns mentiond in other paragraphs, everything seems fine.
clarity: - I guess that just LQM in Figure 8 means LQM+Bootstrap
- Why is then LQM+Bootstrap improving so strongly for high label noise?

- N is unspecified

- In the appendix it remains unclear what kind of data augmentation is used also the performance of the individual models is not provided - in particular what do these ones achieve just training on the clean data?



relation to prior work: There is a huge literature on this topic and I am not super-familiar with the literature, in particular not the one regarding the aggregation of human labels. Thus it seems ok but it could be that I missed important related work.
additional feedback: If the authors can emphasize more the part in Section 2, then the paper would be better suited for this benchmark/dataset track. The LQM method seems a bit off-topic and there remains the missing comparison to self-training/pseudo-labeling SSL techniques.
review_title: Nice study but contribution remains unclear

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the task of performing meta-learning based on the unsupervised dataset is considered. The high-level idea is to mimic the way that human raters annotate real-world data via using a diverse pool of rater models to generate pseudo-annotations. The proposed method is straightforward and the synthetic datasets exhibit some important features of realistic label noise. The empirical results reflect the usefulness of the proposed simulation framework. The writing was clear and easy to follow. I like the experimental design and the comparison analysis with independent random label flipping. 

strengths: 1. The proposed simulation framework is technically sound and similar to how human raters annotate real-world data.
2. The experimental design for evaluating the generated dataset is thorough and clean. The experimental results in section 3 reveal the impacts of realistic label noise on different deep learning models. 
3. The proposed label quality model (LQM) is demonstrated to largely improve model performance with rater features. I like the idea of using extra information from rater models to enhance the model's performance and this can motivate the future algorithm design on learning with noisy data.
weaknesses: 1. Simulating human errors in data annotation is good, but it would be better to enhance the current framework with other types of errors. 
2. The quality of generated pseudo-labels will be highly dependent on how the pool of rater models is constructed. It would be better to dive deeper into what types of models / how many models are needed to create a high-quality rater models' pool.
correctness: The proposed approach and constructed synthetic datasets are sound. The empirical experimental analysis is thorough.

clarity: The text and figures are clear. The paper is easy to follow.
relation to prior work: A detailed related work section is provided in the paper. The authors clearly distinguish their work from related noisy label simulation frameworks.
additional feedback: 1. One benefit of independent random label flipping is that the simulator can generate pseudo-labels with arbitrary user-defined noisy ratios. It seems to be hard to control the noisy ratio in the proposed simulation framework as the quality of rater models are depending on their architecture designs. 
2. Typos: line 322: data are may not / line 177: characteristic -> characteristics / line 64: on the our
review_title: Interesting idea with well carried and thorough empirical experiments.

Meta-review: This paper proposes a simulation framework for learning with instance-dependent label noise, which they do via a pseudo-labeling paradigm, evaluating on the CIFAR10-H dataset. The authors also propose a modeling step based on "annotator" features.  While there were several strong points to this paper, the reviewers were not convinced that the central claim of the label noise being accurate was sufficiently proven, or all the label noise/modeling techniques appropriately described.
Decision: Reject