Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: When learning with noisy labels under certain robust loss designs, the authors focus on addressing two open questions: (1) Why do robust losses underfit? and (2) What affects the robustness of a loss? The two questions are explained through the view of sample-weighting curriculum perspective for certain robust losses. To overcome the underfitting issue, the authors propose to adapt the sample-weighting curriculums. To make loss more robust, the authors further show that adjusting the learning rate schedule helps.
strengths_and_weaknesses: $\textsf{Strengths:}$
1. This work connects several popular robust loss designs to sample-weighting curriculum.

2. Empirically, the authors explain the two open questions in the literature of learning-with-noisy-labels. The introduce of marginal effective learning rate looks interesting and helps with explaining the underfitting issue. And the shifting of soft-margin mitigates the underfitting, especially when the number of classes is large.

$\textsf{Weaknesses:}$

1. Presentation could be further improved, i.e., notation definitions, and argument statements  (please refer to Section "Questions" for more details.)

2. A potential weakness of this work is that all the analysis focuses on the class-dependent label noise, although it enables easiness of theoretical analysis, it was shown that real-world label noise tends to be instance-dependent [1]. And the majority section of this work falls into empirical explorations. It would be interesting to see whether the conclusions still hold for real-world noisy data, either small-scaled ones (CIFAR-10N, CIFAR-100N real-world noisy CIFAR [2]) or the large-scale Clothing 1M dataset.

$\textsf{References:}$

1. Part-dependent Label Noise: Towards Instance-dependent Label Noise. [NeurIPS 2020]

2. Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. [ICLR 2022]

3. Learning From Massive Noisy Labeled Data for Image Classification. [CVPR 2015]
questions: My concerns/questions/suggestions are listed below:

1. In Line 62, a pointer/reference to the argument "With $\eta<(k-1)<k, L$ is robust against symmetric label noise" would be appreciated.

2. In Line 64, $\ell$ is never defined, also for $\Delta p$ in Eqn. (2).

3. Typo: Line 108 "a sampled is learnt".

4. In Table 3, although it seems that MAE has pretty bad performances on the mentioned tasks, would fine-tuning influence much on the performance of MAE? Since 1.00 accuracy on CIFAR-100 is merely random guessing or predicting all samples to be the same class.
limitations: See Weakness 2.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper offers a curriculum perspective in analyzing robust loss functions, which argues that each loss function implicitly induces a sample weighting curriculum. This analysis reveals that underfitting of robust losses can be explained by marginal effective learning rate and marginal initial sample weight. The paper proposes to address underfitting by adapting the sample-weighting curriculums.
strengths_and_weaknesses: Strengthes:

1. The paper is well-motivated from the existing problems of robust loss functions.
2. The analysis on the sample weighting curriculum offers an insightful perspective on the underfitting effect of robust loss functions.
3. Motivated by the sample weighting analysis, the paper proposes sample-weighting curriculums that can address the underfitting issue.

Weaknesses:
My main concern is that it is unclear how general the findings would hold. The experiments are conducted in a posthoc manner where it is first known that CIFAR100 suffer from the underfitting issue. Given that CIFAR100 is the only dataset with underfitting, it is unclear whether the findings can generalize to other datasets with similar issues. How to ensure that the sample weight curriculum is not a spurious correlation imposed by the CIFAR100 dataset?
questions: Please refer to my weakness section.
limitations: No
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In learning with noisy labels, designing certain loss functions is a promising direction to achieve noise robustness. This paper aims to use a new perspective (curriculum) to understand the key issues of robust loss functions. Typically, there are two issues in existing loss functions. 1). noise robustness, which is usually analyzed by symmetric condition; 2). underfitting problem, which is an open question to be explored.

Overall, this paper has following contributions: 
1. This paper claims that the robust loss functions are implicitly weighted. Based on this finding, the authors explain the reason of underfitting problem (i.e. either due to small initialized weight or weight diminishing during training) and noise robustness (i.e., weights are different for clean and noisy samples).
2. This paper addresses underfitting problem by shifting or scaling the initialized weights.

strengths_and_weaknesses: Strengths:
1. Using sample-weighting curriculum to understand underfitting problem is interesting.
2. The overall paper is well-written and makes sense.

Weaknesses or concerns:
1. Since this paper attempts to explain some open questions, no SOTA method is proposed. The overall novelty is limited.
2. The performance improvement of the proposed techniques (shifting and scaling) is marginal, especially at high noise rate. Compared to the existing methods (NCE+AUL or NCE+RCE), the performance of MAE+rescale does not outperform them much. 
3. It seems that the proposed approaches are not practical and dependent on empirical observations. For example, it has to estimate the expectation of $\Delta_{y}$ in Eq. (9) based on several assumptions.  In Eq. (10), it seems that the hyperparameter $a$ also depends on complexity of data, which can be hard to find in practice. I am curious about the performance of MAR+rescale on real-world noisy datasets, such as Clothing1M and Webvision.
4. In Table 2, it shows that both AUL and AGCE suffer from severe underfitting problem. The paper only provides the results on MAE. Do shift or rescale work for AUL or AGCE? More experiment results should be conducted on these complex loss.
5. One of the proposed claim that "cross entropy can appear robust by adjusting learning rate" is not new idea. [1] has observed the similar results and adjusts the learning rate cyclically to achieve noise robustness.
6. In Eq. (5), the loss can be represented as a form of $w(\Delta_{y})$ and $\Delta_{y}$. This paper only considers the impact of $w(\Delta_{y})$ but does not further investigate the effect of $\Delta_{y}$ distribution (only provide the $\Delta_{y}$ distribution at at initialization in Figure 1). Will the $\Delta_{y}$ distribution change in training?

In summary, this paper proposes an interesting perspective to understanding key problems of noise robust loss functions. However, some claims are "hand-waving" to me and the theoretical analysis is incomplete. The experimental results are not fully supporting the advantage of the proposed methods when compared with recent noisy-label learning methods. I hope that the authors can clarify these points in the rebuttal.

[1] Huang, Jinchi, et al. "O2u-net: A simple noisy label detection approach for deep neural networks." Proceedings of the IEEE/CVF international conference on computer vision. 2019.  
questions: 1. In Table 3, settting $a=2.6$ achieves optimal results for MAE. How to find this optimal hyperparameter? Does it depend on the complexity of dataset?

2. Does shitting or rescaling also work for other robust loss functions? such as NCE or AGCE?

3. Existing works explain the underfitting problem by gradient analysis (gradient coefficients are smoothed) or regularization theory (fitting ability is limited). How will you compare this work with other works?

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper provides a new understanding for two open questions in the learning with noisy labels literature: why robust loss functions can underfit the data distribution and why robust functions can be robust. The analyses are delivered by decoupling the equivalent sample weight from each loss function, which is further viewed as an implicit sample-weighting curriculum. Based on the proposed analytical framework, the authors experimentally studied the relationship between the underfitting of robust loss functions and the average effective learning rate (due to equivalent sample weights), and proposed solutions to mitigate underfitting. Besides, by putting the equivalent sample weights and training dynamics together, the authors showed the robust loss functions assign lower weights to wrong labels compared with CE. 
strengths_and_weaknesses: **Originality**

The idea and analytical framework are original and novel.

**Quality, clarity**

Both the theoretical part and the experiments are clear and of high-quality.

**Significance**

The reviewer believes this new finding could contribute sufficiently to the noisy learning literature, and guide the design of robust loss functions.
questions: There are some minor questions:
1. The considered loss functions are relatively limited due to the fast-moving literature. There are several recent robust loss functions, such as peer loss and its extensions [R1-R2], Taylor cross-entropy [R3], label smoothing (positive or negative) [R4-R5]. It would be interesting to provide a library for the equivalent weights for these loss functions.
2. In addition to the loss functions that do not require estimating the noise transition matrix, is it possible to also find the equivalent weights for the loss correction/reweighting based methods, such as [40], [R6]?
3. It would also make the paper more convincing if Figure 4 could also be plotted for the above loss functions.
4. Current experiments only consider the synthetic label noise with simple noise generation regime. It is interesting to see whether the findings hold on real-world human annotations, e.g., CIFAR-N [R7].

[R1] Liu Y, Guo H. Peer loss functions: Learning from noisy labels without knowing noise rates. ICML 2020.

[R2] Cheng H, Zhu Z, Li X, Gong Y, Sun X, Liu Y. Learning with instance-dependent label noise: A sample sieve approach. ICLR 2021.

[R3] Feng L, Shu S, Lin Z, Lv F, Li L, An B. Can cross entropy loss be robust to label noise? IJCAI 2021.

[R4] Lukasik M, Bhojanapalli S, Menon A, Kumar S. Does label smoothing mitigate label noise? ICML 2020.

[R5] Wei J, Liu H, Liu T, Niu G, Sugiyama M, Liu Y. To Smooth or Not? When Label Smoothing Meets Noisy Labels. ICML 2022.

[R6] Liu T, Tao D. Classification with noisy labels by importance reweighting. IEEE TPAMI 2015.

[R7] Wei J, Zhu Z, Cheng H, Liu T, Niu G, Liu Y. Learning with noisy labels revisited: A study using real-world human annotations. ICLR 2022.
limitations: They discussed the limitations in Sec. 3.1 Potential negative societal impacts are not applicable.
review_title: null

Meta-review: metareview: The paper focuses on investigating the underfitting issue of certain robust loss functions and why functions deviating from theoretical robustness conditions can still be robust. A key vehicle to support the analysis is a standard form with equivalent gradients that enables the investigation of different robust loss functions from a curriculum learning perspective. In this standard form, each loss function is factorized into a sample weight and an implicit loss. Empirical studies have been conducted to show the interaction between sample-weighing curricula and different training dynamics to gain deeper insights on the underfitting issue and why loss combination can mitigate this. A shifting/rescaling strategy is also developed to adapt the sample-weighing curricula so that the underfitting issue can be addressed. 

The paper contributes to an important area in machine learning given the labeling errors commonly exist in most real-world datasets. The proposed work complements most existing research that focuses on the design of robust loss functions and bounding the risk minimizer over these functions. The derivation of the standard form with equivalent gradients is new, which nicely connects a set of robust loss functions with a sample-weighting curriculum. The analysis afterwards also shows some interesting insights on the open questions being considered. 

Major concerns from reviewers include whether some of the important findings will hold in general. These concerns are related to the evaluation on limited datasets (e.g., only CIFAR100 suffers from underfitting) and only MAE is tested for the effectiveness of the shifting/rescaling strategy. While the authors’ rebuttal addresses some of these critical comments, a more comprehensive evaluation may still be needed to make a convincing case. Furthermore, while the analysis is interesting, in order to justify it is truly useful, it is still necessary to show how it helps to inform the design of better robust loss functions or lead to a better learning process that can further improve the current models. The proposed shifting/rescaling strategy shows some promise, but it is designed in a rather heuristic way. It heavily relies on a critical hyper-parameter that is dataset dependent and may be quite sensitive and thus difficult to be set properly in practice. 

The Senior AC and AC discussed the paper and the authors concerns and feel that the paper is not yet above the NeurIPS publication bar. The authors are encouraged to take the reviewer feedback into account to prepare an improved draft which they can submit to an upcoming conference.

confidence: Certain
Decision: Reject