Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
The authors first analyze and answer the question: What training examples should be focused and how large the emphasis spread should be? Then, they proposed the gradient rescaling framework serving as emphasis regularization. 

Strengths:
1. The paper is well organized except the reference citation (read difficultly)
2. The proposed method is very simple and effective. 
3. Experiments show the improvements over SOTA. 

Weakness:
1. The experiments lack the recent important baseline "symmetric cross entropy for robust learning with noisy labels, ICCV2019", which are the current SOTA. Maybe the author should check the above paper and show the results.  
2. The experiments are only conducted on symmetric noise. Actually, asymmetric noise is also important. The author should conduct at least some experiments on asymmetric noise. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary
This paper presents Gradient Rescaling (GR) for robust learning to combat label noise. They propose to treat each data sample with different significance scores: some samples are important to learning, and some examples are insignificant (or even detrimental) to learning. So they desire to weight each samples according to their significance. They propose the notion of emphasis focus (When learning, whether we should put emphasis on learning “hard” examples or “easy” examples) and emphasis spread (the variance of these significance weights). The authors propose that this “difficulty” of samples are proportional to their network output logit values.
The authors examine the analytical forms of the gradients of popular loss functions such as Categorical Cross Entropy, Mean Absolute Error and Generalized Cross Entropy. They find that the formulas for the gradient are of similar family with varying hyperparameters. Authors claim that tweaking these hyperparameters result in tuning the emphasis focus and spread.
The authors conduct Experiments on CIFAR10, CIFAR100 with simulated symmetric noise. Also, they conduct experiments on real-world noisy datasets: Clothing 1M dataset and MARS video dataset. The authors claim that the performance of GR exceeds various baselines.


Significance/Novelty/Clarity
Significance: Low-Medium. The performance increase exhibited in the experiments are a bit underwhelming (when considering the fact that benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are missing).
Novelty: Medium. The paper is interesting in the sense that the authors integrated (and allegedly generalized) the gradient formulas for several losses into one family, and tried to integrate and tweak their postulation of  “Emphasis focus” and “emphasis spread” into the framework. However, the theoretical ground and convincing reasoning for their claim seems a bit lackluster.
Clarity: Low. The overall flow of the paper is a bit fuzzy - exhibiting a stream-of-consciousness style flow.


Pros and Cons in Detail
Pros:
1.The authors try to unify the analytical forms of the gradients of various loss functions into a single family equipped with hyperparameters that control emphasis focus and spread.
2.Conducted experiments show that GR achieved increased performance when compared to the baselines.
Cons:
My major concern is about tuning newly introduced hyperparameters in practical settings. How can we guarantee to have intact validation set?  Can we get any improvement via GR even with corrupted validation set for tuning hyperparameters? 
1. The arguments of the authors are grounded in the premise that “difficult” samples will exhibit small logit values, and “easy” samples high logit values.
2. No justifications (both theoretical and experimental) are provided on the claim that controlling emphasis focus/spread will result in more robust learning. 
3. This algorithm introduces 2 additional hyperparameters that are correlated with each other. This introduces additional labor.
4. By changing the loss function, the outputs of the network might lose its interpretation as a probability distribution.
5. No confidence intervals are shown except for the CIFAR-100 experiment.
6. Experiments are only conducted on vision tasks.
7. The baseline menagerie also changes when the authors change the target dataset.
8. Additional benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are required.


Questions
1. Is it always the case that “difficult” samples exhibit small logit values, and “easy” samples high logit values?
2. If not, GR’s emphasis manipulation might result in neglecting samples containing valuable information.
3. Can GR be used simultaneously with other noise-robust learning methods to further boost the performance?
4. Technically, GR aims to rescale the gradients of the logits. How will it interact with optimizers  other than SGD such as Adam?
5. Does GR still work well on small datasets(#points < 5000)?


Misc. Comments
Page 3-> inside L1 norm, no differentiation sign in the denominator.
Around eq 2 and 4: missing derivative symbol w.r.t. z
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:
The paper proposes a method for noise robustness based on scaling gradients of examples. By choosing the proper scaling parameters (alpha and beta), the method recovers standard losses such as CCE, MAE, and GCE, while also recovering other losses. The method is strongly related to reweighting training examples, where alpha and beta define the shape of this weighting as a function of the model's prediction (i.e., p_i). Experiments show that the proposed method achieves competitive results on several standard benchmarks for noisy-labelled data.

Comments:
- The main drawback of the proposed approach is that there is no clear way of choosing alpha and beta, other than a hyper-parameter search, which is not very practical and can lead to overfitting the test set.
- The paper in general is easy to follow, but the paper is not very rigorous or clear on some important concepts. For example: 
     * No clear mathematical definition of emphasis focus and spread
     * The term "semantically abnormal examples" should be defined in the main text.
     * It is not so clear what it means to "babyset" emphasis focus and spread.
     * I don't understand what Eq. 6 is supposed to tell.
     * What are the \dots in equation 
- The experiments are very thorough and the results are very good, but I have few clarifying questions:
     * The procedure for choosing beta/gamma is not clear, and I see that for every experiment those values change.
     * It would be nice if the CIFAR-10 Table 4 results are performed using the exact same setting to prior work to make sure the comparison is fair. For example, GCE results in (Zhang & Sabuncu 2018) are much that the reported ones. While it seems that you're using GoogLeNet V1 architecture similar to Jiang et al. 2018, it's not clear which experimental setting you are comparing against.
     * Can you be more specific what do you mean by "with a little effort for optimizing beta and gamma" in caption of Table 5?

Minor:
     * Grammer mistake: "what training examples...focused *on*..."
     * Citations should be done with parentheses

review_title: Official Blind Review #1

Meta-review: The paper proposes a gradient rescaling method to make deep neural network training more robust to label noise. The intuition of focusing more on easier examples is not particularly new, but empirical results are promising. On the weak side, no theoretical justification is provided, and the method introduces extra hyperparameters that need to be tuned. Finally, more discussions on recent SOTA methods (e.g., Lee et al. 2019) as well as further comprehensive evaluations on various cases, such as asymmetric label noise, semantic label noise, and open-set label noise, would be needed to justify and demonstrate the effectiveness of the proposed method. 
Decision: Reject