Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies how different forms of label noise affect the performance of ML classifiers. It is argued that the most benign form of label noise is uniform noise independent of features and labels, and the most harmful is feature dependent label noise. For feature independent label noise, classifiers retain good accuracy until some level of noise, after which there is a sharp drop-off. On the other hand, for feature dependent label noise, what matters is whether the flips occur close to or far away fromt he decision boundary. This is illustrated through theory and experiments. 
strengths_and_weaknesses: For the theory, the authors study how the Bayes optimal classifiers with and without label noise compare for simple label flipping attacks. While studying the Bayes optimal classifiers is quite limiting, it is nonetheless a useful starting point that seems to not have been considered before in the literature. The theory is simple, but leads to the following insights:
- Feature-independent label flipping is benign until a large value of epsilon, after which there is a sudden drop in performance. In particular, for class-dependent label flipping, the point of drop in performance depends on the 'spread' of the flip (the support size of the flip targets).
- Feature dependent label flipping can hurt even at a small value of epsilon, especially if the flipped labels are closed to the optimal decision boundary. On the other hand, flips away from the decision boundary have a smaller effect. 

Despite the simplicity of the theoretical analysis, it matches very well with synthetic experiments. 

The results on CIFAR-10 and CIFAR-100 (Tables 2 and 3) seemed more of a mixed bag. Shouldn't the drop in performance from class noise to feature noise be more based on the theory? Why is this not the case? 

The paper was easy to read and follow, although I felt that the authors could have been more transparent about the limited scope of the theoretical study (only Bayes optimal classifiers). 

Overall, I found the paper informative, but lacking in a final punchline message. The existing findings are concrete and well-supported, but not too surprising (at least to me): I would have expected that feature based label noise is a more powerful attack than class based or uniform noise. The work makes no methodological contribution, which for me slightly increases the bar of acceptance. (Perhaps this research could help engineers seeking annotations to take appropriate steps if label noise seems higher than a threshold? Not sure.)
questions: I have asked a few questions in the strengths and weaknesses section, as well as the limitations section. 

The \eta_{ki} values in eq. 8 do not seem to add up to 1? Could you please clarify. 

I like the paper and would be happy to increase my score from 6 to 7 if the response is strong. 

limitations: I feel that this paper does not address the 'so-what' question: how do I use this theory in a practical setting where I don't know the extent of label noise, or the type of label noise? I encourage the authors to think and elaborate on this.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors analyze the problem of the robustness of the machine learning classifiers to the label noise under different noise types: 1) uniform (independent of features and classes), 2) class-dependent, and 3) feature-dependent. In their analysis, the authors focus on analyzing the "typing point" - the noise ratio value below which classifiers suffer a significant reduction in predictive performance. The authors show that typing point can be much lower when noise is feature dependent than in two other cases, suggesting that one needs to be careful when assuming that the noise is independent of features. The theoretical results are later confirmed by empirical evaluation on both synthetic and real datasets with introduced noise. 
strengths_and_weaknesses: Strengths:
- Although this isn't a methodology paper, and while the main claim is not surprising, the paper helps to understand and provide a sound analysis of the problem. This is the type of paper that is interesting to read as opposed to many papers that demonstrate yet another epsilon improvement in an experimental evaluation.
- The theoretical results are confirmed by empirical experiments, on different datasets and with different types of introduced noise.
- The experiment with methods for learning with noise labels is a nice addition.

Weaknesses:
- Most of the presented results come from previous works.
- The main conclusions are not surprising and, as the authors acknowledge, observed in previous empirical studies.

NITs:
- In eq 14. I think double equal signs are not needed.
- In line 180, P Y|X are missing brackets.
questions: I don't have questions right now.
limitations: Since this isn't a methodology paper, there are not many limitations to discuss. Also, there are no potential negative social impacts of this work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This manuscript studies the robustness of instance-dependent label noise. The authors provide a theoretical framework that generalizes beyond a typical assumption that label noise is independent of the features by modeling label noise as a distribution over feature space. They also show that the shape of noise distribution has a strong impact on classification performance if the noise is concentrated in feature space. 
 
strengths_and_weaknesses: The theories (Lemma3.2 and Lemma3.5) presented in this paper simply derived the results from [2] under both uniform label noise and class-dependent label noise, while theoretical analysis on instance-dependent label noise is not provided, the novelty is limited.

This paper focus on the instance-dependent label noise studied, but two-thirds of the paragraph are describing instance-independent label noise.

[2] Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels.


questions: In Lemma 3.2, what is the definition of a Bayes optimal classifier, or what is the property of a Bayes optimal classifier? Please formulate it since it is important to understand the following Eq.(6).

Corollary 3.2.1 "The minimum noisy accuracy of $\frac{1}{c}$ occurs at $\epsilon=\frac{c-1}{c}$". This claim is not exactly correct since it misses the condition of uniform assumption. Besides, this claim does no significance since it is common sense that the training labels are useless if $\epsilon=\frac{c-1}{c}$.

Corollary 3.3.1 "Clean accuracy drops by half at $\epsilon=\frac{c-1}{c}$". This claim is not exactly correct. what is the meaning of "Clean accuracy drops by half", is $\epsilon<\frac{c-1}{c}$, and the clean accuracy does not drop by half?

In Fig.3(a), why the greater the $\epsilon$ (noise rate) the better the noisy accuracy? 

Eq.(8) is problematic. Since $\sum_{i\neq k}|t_{ki}|^0=s$ and  $\sum_{i\neq k}t_{ki}=1$, $\sum_{i=1}^{k} \eta_{ki} \neq 1$ if $s\geq 2$.

From the process of generating feature-dependent noise (line186-188), it seems a class-dependent label noise rather than feature-dependent noise.

limitations: Please see the comment above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper suggest difference of generalization ability of Machine Learning considering the scale and shape of the noise distribution. Theoretically and empirically, Machine learning is known to be robust to uniform label noise. However, no study has been researched for the case of class-dependent or instance-dependent label noise setting, and this paper supports theoretical background for those cases. The paper also shows the degradation of noisy label managing algorithms after tipping point empirically, supporting the theoretical result they show.
strengths_and_weaknesses: <<Strengths>>
- Originality : The paper theoretically proves the robustness and fragility of ML under trained with noisy labeled dataset setting with different noisy label condition. It is quite interesting. Not only showing the robustness of uniform label noise, the paper also shows the weakness of class- and instance- dependent label noise, which is novel.

- Quality  : The submission is technically sound and the claims are well supported both by theoretical analysis and the expermental results.

- Clarity : Sentences are easy to read. Mathematical expressions are understandable. Well organized.

- Significance : The results are important suggesting two points.
1) The existence of tipping point 
2) The model performance degrades suddenly after the tipping point, not gradually.

======================================================================================================

<<Weaknesses>>
- Quality :  Maybe want to see more experimental results?(e.g. Clothing 1M, a well-known benchmark real dataset for noisy label problem, would it also have tipping point?)

- Clarity : (typo) Py|x for section 4.

questions: 1) How did you generate featrure-dependent label noise for CIFAR10 and CIFAR100 for experiments?
2) What will happen if two or more types of noisy label generation algorithm are simultaneously applied? e.g. a noisy labeled dataset with symmetric noise 20% and asymmetric noise 20%.
3) Not to face sudden model performance degradation, is it the only way to get dataset clean enough (not having noisy label as many as the tipping point)?

limitations: The authors have not addressed ther limitations.
review_title: null

Meta-review: metareview: The authors provide an interesting perspective on robustness by looking at how different forms of label noise affect performance. Although some of the results are unsurprising given the literature, providing theoretical justification for empirically observed phenomena is an important contribution that could promote new ideas. The paper is also well-written and was appreciated by most reviewers.
confidence: Certain
Decision: Accept