Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a novel view to examine robustness against label noise and its under-fitting issue. It first points out previous arguments could not sufficiently address causes of the under-fitting issue. In contrast with the previous approaches, it factorizes a loss function with the sample-weighting function and the distribution of implicit losses, which provides a novel view to see it as a type of curriculum learning. It provides several experimental results showing training dynamics along with several measures based on the factorized terms, and makes arguments about the under-fitting issue and robustness.
strength_and_weaknesses: ## Strength
This paper provides an interesting view for under-fitting issues in loss functions known to be robust against noisy labels. Various loss functions with various parameter settings are considered to be examine and observations are well aligned with our intuition. 

## Weakness
- The experiments are mainly focusing on image classification, and on convolutional layer based models. The authors should discuss in the paper if they want to claim their findings can be generalized to any other tasks, and model architectures. Or it should clearly mention to what extent was this experiment covers in the main text. It is important because the paper's main claims are mainly supported my experimental results. 
- The main manuscript is not sufficiently self-inclusive. Readers should check the supplementary material to check important experimental settings, and term definitions. I understand this is mostly because this paper have to introduce many terms, measures, and acronyms. But such a paper should be more cautious in clarity. See comments in Clarity for actionable feedbacks.
clarity,_quality,_novelty_and_reproducibility: ### Novelty
Its novelty comes from providing a novel view (supported by experimental evidences) about an existing, well-known problem. I have no big concern about novelty.

### Quality
Their experiments cover various types of loss functions and different hyperparameter settings. The experiments support the claims well, but they have to mention to what extent the claims can be generalized (as I mentioned in the Weakness).

### Clarity
Several minor improvements are needed. 
- Pg4. Are those Wei et al.'s human noisy labels and WebVision's noisy dataset much closer to the symmetric or asymmetric?
- Pg5. section 4.1. It is not clear what 'Robust loss function' means and what are those. Does it indicate loss functions known to mathematically satisfy the condition (1)? Or loss functions showing a good performance in Table 2? 
- Pg5. section 4.1. It is not clear what 'parameter update size' means. 
- Fig 1. Can you provide more detail how you get the initial distribution of CIFAR10 and CIFAR100?

### Reproducibility
The authors said their code will be available at github, but not provided in this submission. So I cannot evaluate its reproducibility.
summary_of_the_review: This paper provides a novel, and potentially impactful view about under-fitting problems of robust loss functions. They made several sub-claims based on their observations, which is intuitively understandable. Experiments are well designed, and the claims are sound in some extent, but they should clearly state that to what extent they want to generalize the claims. Also there are some minor issues including writing clarity and reproducibility. I am willing to raise the score upon the authors response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Learning with noisy labels is an important task in weakly supervised learning, which has received extensive attention in recent years. This paper aims to answer two questions: why robust loss functions can underfit and why loss functions deviating from theoretical robustness conditions can appear robust? Specifically, the authors rewrite the loss function into a standard form with equivalent gradients and show that most robust loss functions differ only in the sample-weighting curriculums they implicitly define. Thus, they propose to scale or shift the sample-weighting functions to address underfitting. Additionally, they also show that training schedules can affect noise robustness. 

strength_and_weaknesses: Strengths:
1. This paper explains the robustness of the loss function against label noise from a new perspective and shows most robust loss functions differ only in the sample weighting. Therefore, the robustness of some loss would be improved by adjusting the implicit sample-weighting function.
2. This paper not only analyzes why some robust losses underfit but also gives two effective solutions, scaling or shifting the sample-weighting functions.
3. Comprehensive experimental results in different cases clearly demonstrate the effectiveness of the proposed methods.
4. This paper is well-written and well-organized.

Weaknesses:
1. There is a question, that is, does robust loss (e.g. MAE) with scaled or shifted sample-weighting function still satisfy the conditions for noise robustness?
2. In nature, the methods proposed in this paper aim to adjust the sample weights. However, the authors neither compare the proposed methods with existing sample reweighting methods nor provide details and discussions for the differences between the proposed methods and existing sample reweighting methods.
3. The authors claim that robust loss functions are vulnerable to label noise with extended training and provide the result of MAE on MNIST as an example. However, such observation does not mean that all robust losses would be vulnerable to label noise with extended training. They should provide more results of other robust losses to support this statement.
4. I found some typos in this paper, so I would suggest that the authors could carefully check the manuscript repeatedly. The following are some typos I found:
- "Although the training dynamics of NCE is complicated by ..." might be "Although the training dynamics of NCE are complicated by ..." in section 3.1.
- The word "distribution" in the title of Figure 3 and Figure 5 may be in the plural.


clarity,_quality,_novelty_and_reproducibility: Novelty: Good. The paper makes non-trivial advances over the current state-of-the-art.
Quality: Good. The paper appears to be technically sound. The proofs appear to be correct, but I have not carefully checked the details. The experimental evaluation is adequate and the results convincingly support the main claims.
Clarity: Good. The paper is well organized but the presentation has minor details that could be improved.
Reproducibility: Good. Key resources (e.g., proofs, code, data) are available and sufficient details (e.g., proofs, experimental setup) are described such that an expert should be able to reproduce the main results.

summary_of_the_review: Overall, I think this paper is a relatively good work on learning with noisy labels because it provides a new perspective to explain the robustness of loss function against label noise. However, there are some typos in this paper and some details of the proposed methods are missing. So, I think the authors should carefully check the manuscript repeatedly and provide the necessary details and discussions to further polish this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies why robust loss in the literature of learning with noisy labels may suffer from underfitting. Even though some early papers have already studied this question, authors provide a different perspective by showing that most robust loss functions differ only in the sample-weighting curriculums they implicitly define. Inspired this observation, authors explain the under-fitting phenomenon from the weighting term and propose an adjustment of weighting funciton to improve the model performance. Experiments are conducted on CIFAR-10, CIFAR-100 and Webvision datasets.
strength_and_weaknesses: **Strength:**

- By writting the losses into the form of Equation (6) is intersting, which enables us to examine the effectiveness of each loss by observing the dynamics of  $w(\delta y)$ and $\delta y$. The explanation is supported by the extensive experiments.

- The proposed shifted and scaled MAE significantly improve the performance comapred to vanilla MAE.

**Weakness:**

- Even though the explanation and claims are interesting. The performance of adjusted loss is not competetive to SOTA. For example, the performance in this paper is even not stronger than [1] which also improves the robust losses. 

- [2] also observes that CE with adjusted learning rate can improve performance. Authors should cite and discuss the difference with [2].


[1] Normalized loss functions for deep learning with noisy labels

[2] O2U-Net: A Simple Noisy Label Detection Approach for Deep Neural Networks
clarity,_quality,_novelty_and_reproducibility: - This paper is well-motivated and organized. 
- The explanation of why robut losses suffer from underfitting is interesting but the performance of adjusted loss is not competitive.
- The paper does not provide the code currently.
summary_of_the_review: - The paper has merits on explaining the underfitting phenomenon of certain robust losses in the literature of learning with noisy labels. Even though
the performance is somehow limited, I think the claims and experiments may inspire further research in this area. Thus, I initially give the score of 6.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents work on learning with noisy labels. Interestingly, the paper provides a new curriculum perspective based on robust loss functions. The perspective aims to explain (1) why robust loss functions can underfit; (2) why loss functions deviating from theoretical robustness conditions can appear robust. Theoretical analyses and empirical evaluations are provided to justify the paper's claims.
strength_and_weaknesses: **Strengths** 

- The research problem is realistic and important. It is popular to design robust loss functions to handle noisy labels. A new perspective on the robust loss function could provide insights into the research community.
- Experimental results are detailed with extensive ablation studies. 

**Weaknesses**

- The contributions of this paper are overclaimed. 
- The writing and organization of this paper should be improved. It is hard to follow the implementation and claims. 
- Theoretical results and discussions do not correspond to intuition and are often contradictory. 

**Specification of weaknesses**

1. The paper claims that one of its main contributions is to understand why robust loss functions can underfit, and argue that prior works fail to comprehensively characterize the performance of robust loss functions. However, existing works, such as GCE, have analyzed the model robustness in training from the perspectives of gradients. It is not agnostic to training dynamics. Besides, it seems that the theoretical results of this paper are not related to a specific deep model. 

2. How to understand "the weights are marginal"? Although I carefully check this paper, experimental parts, in particular, there is still no clear understanding of "marginal weights". 

3. The definition of the sample weights should be given before Table 1 since $w$ appears in Table 1. 

4. From Section 2.2, the paper argues that the MAE loss is passive with the first specification but active with the second. The conclusion is completely contradictory to the definitions of active and passive losses in [1]. 

5. The paper claims that most loss functions in Table 1 can be written as a function of the target softmax probability, which is a bit rough. From this claim, some loss functions cannot be written as this function. Therefore, theoretical results cannot be applied to them?

6. The derivation of Eq. (6) is important. More details should be provided. 

7. How do Eq. (8) and the following discussions help analyze the underfitting issue? It is not clear. Besides, the theoretical results of peer loss functions are different from GCE and SCE. There is no clear explanation for relating two types of theoretical results.

8. The experimental results in Table 2 are confusing. Specifically, there are moderate underfitting for NCE and severe underfitting for MAE. Then, there is no underfitting for the combination of NCE and MAE. This is very strange. Can the underfitting issues of two loss functions be addressed by each other?

9. Eq. (9) gives the explanation that $\Delta_{y}$ is related to the class number $k$. However, it is still unclear how $\Delta_{y}$ causes underfitting. 

10. Both "scale" and "shift" methods make the initial sample weights agnostic to the number of classes. From Table 4, there is a large gap between the performance of the two methods. Could the paper provide more discussions about the results? Besides, it seems that MAE totally fails in this case. 

11. Does extended learning mean training with a fixed learning rate?

12. There is no clear understanding of why slowing down the learning pace can improve the robustness of a loss function. Intuitively, a slow learning pace still makes the model overfitting to mislabeled data. 

----
[1] Xingjun Ma et al. Normalized loss functions for deep learning with noisy labels. ICML 2020.
clarity,_quality,_novelty_and_reproducibility: For clarity, there are some unclear explanations at the present stage (see the above comments). For quality, although the new curriculum perspective is interesting, the quality of this paper could be further enhanced. For novelty, the technical novelty is good. However, the concerns about contributions are not addressed. For reproducibility, the experimental settings are detailed in the main paper and appendix. 
summary_of_the_review: This paper provides a new perspective on learning with noisy labels. However, as mentioned, there are still many unclear explanations in the present version. Therefore, before rebuttal, the reviewer is negative about this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper analyzes two aspects of robust loss functions -- why robust loss functions can underfit; and why loss functions deviating from theoretical robustness conditions can appear robust. Authors show that most robust loss functions differ only in the sample-weighting curriculums they implicitly define. They propose to scale or shift the sample-weighting functions to address underfitting and empirically show that this scale/shift operation is able to improve the performance of robust losses for noisy labels. Reviewers were split on the paper with some placing the paper above the acceptance bar due to a novel perspective (implicit weighting) for analyzing robust losses and suggesting a fix with shifting/scaling the weights. However reviewers still have following outstanding concerns -- 

1. Robust loss functions (e.g., MAE) with the proposed scaled or shifted sample-weighting function may not satisfy the conditions for noise robustness, and it may require further theoretical discussions on this fix. 
2. Since this paper involves the sample-weighting perspective, empirical comparisons against simple sample-weighting methods (and the simple small-loss trick) will be good to have in the paper. Can something be said about explicit sample reweighting methods?

Meta-reviewer thinks the paper will indeed benefit from addressing these points and it falls below the acceptance bar in current form. 



justification_for_why_not_higher_score: Reviewers still have following outstanding concerns:

1. Robust loss functions (e.g., MAE) with the proposed scaled or shifted sample-weighting function may not satisfy the conditions for noise robustness, and it may require further theoretical discussions on this fix. 
2. Since this paper involves the sample-weighting perspective, empirical comparisons against simple sample-weighting methods (and the simple small-loss trick) will be good to have in the paper. Can something be said about explicit sample reweighting methods?
Decision: Reject