Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper tackles the problem of learning with noisy labels and proposes a novel method CDR which is inspired by the lottery ticket hypothesis. In particular, the proposed method categorizes the parameters into two parts, including critical parameters and non-critical parameters, and applies different update rules to these parameters. Using comprehensive experiments on synthetic datasets and real-world datasets, the authors verify that the proposed method can improve the robustness of the classifiers against noisy labels.

Pros.
1. The proposed method is interesting in its design. The authors provide an alternative interpretation for the optimality criterion, which reveals that the value of the parameter can also be used to check the optimality. It is novel and of significance. Meanwhile, the proposed method is easy to implement. This method can also be applied to existing algorithms to further improve their robustness.
2. Overall, this paper is very well written and well organized. The technical details are easy to follow, and Algorithm 1 helps understand the procedures. 
3. Extensive experiments are performed to verify that the proposed method indeed helps over the baselines at fighting noisy labels. I like the details of experimental settings, which really can help reproduce these experimental results.

Cons.
1. The new interpretation for the optimality criterion is important, but lacks detailed explanation. I find that there is only simple analysis. Perhaps the authors could add some details or citations for better understanding.
2. Though the baseline S2E uses AutoML, it seems that S2E is an improvement on co-teaching or co-teaching+? It is not suitable to place it at the end. 
3. For some baselines such GCE and Joint, there are hyperparameters to consider. The authors should explain how to set their values for a fair comparison. For APL, there are multiple combinations of loss functions. In this experiment, which one did you choose? I suggest that the authors add such explanation in Section 4.2, which will make the results more convincing. 
4. The proposed method implicitly exploits the memorization effects of deep models, and can reduce the side effect of noisy labels before early stopping. After early stopping, noisy labels still affect the performance. How to reduce the side effect during the whole training by using this idea? I personally think this is an interesting and meaningful direction. The authors can regard this as future work to improve this paper. 

Overall, I think this paper makes sense in learning with noisy labels. The proposed method is novel and effective. I recommend to accept this paper, and hope that the authors can address the above issues carefully.
review_title: A novel and effective method for learning with noisy labels

final_score: rating: 7: Good paper, accept
review_content: This paper aims to exploit the early stopping method to solve the problem of learning with noisy labels. Specifically, this paper finds that only partial parameters (critical parameters) are important for fitting clean labels and generalize well; while the other parameters (non-critical parameters) tend to fit noisy labels and cannot generalize well. Based on this observation, this paper proposes to divide all parameters into the critical parameters and non-critical ones, and perform different update rules for the two types of parameters, in each iteration. Extensive experiments on benchmark-simulated and real-world label-noise datasets demonstrate the effectiveness of the proposed method.

This paper has the following advantages:
1.	This paper is well-written and the motivation is very clear. This paper has clearly explained the two types of parameters, i.e., critical parameters and non-critical parameters.
2.	This paper proposes a novel method with an interesting idea. Unlike many existing methods that are aggregations of multiple techniques, this paper only focuses on one concept. It is simple but effective. The view of updating different parameters by different rules is quite novel for learning with noisy labels. I think this view may bring some new insights to the area of learning with noisy labels.
3.	Experiments are quite thorough and results on both synthetic and real-world datasets validate the effectiveness of the proposed method.

This paper also has some minor issues:
1.	I would suggest the authors to carefully check the notations used in the paper. For example, $\mathbf{w}$ denotes all the learnable parameters of the model in the paper, while $\mathbf{w}$ usually means a vector. So I would suggest the authors to use another symbol to denote the set of all the learnable parameters of the model, e.g., $\mathcal{W}$.
2.	It is not well justified why this paper chooses the $\ell_1$ regularizer. Is there any consideration to use the $\ell_1$ regularizer except that could be associated with weight decay?
3.	The hyper-parameter $\tau$ (noise rate) needs to be known. It may not be a big issue as the ablation study in this paper demonstrates that the proposed method is insensitive to this hyper-parameter.

Overall, I think this is a good paper with an interesting idea and convincing experimental performance. So I prefer to accept it.

review_title: Interesting idea with convincing experimental performance

final_score: rating: 7: Good paper, accept
review_content: ------Overall------
This paper utilizes the memorization effects of deep models and aims to improve their robustness to noisy labels before early stopping. As the deep models fit training data with clean labels in the early stage of training, the authors propose a novel method to identify those more important parameters for fitting clean labels. They then deactivate the unimportant parameters to reduce side effects brought by noisy labels, which enhances the fitting to clean labels implicitly. I think this paper is interesting and makes sense. The major comments and issues are as follows: 

------Major comments------
1. Different from other complex methods for learning with noisy labels , this work discusses that standard cross entropy loss can achieve competitive performance with early stopping. We can therefore focus the training stage before early stopping to handle noisy labels. The authors skillfully allow the optimality to be checked by a scalar, and then judge the importance of the parameters by analyzing the influence of the parameters on this scalar. The idea of this paper is novelty and meaningful. 

2. The paper is very well-written. The description of its motivation and technical details is clear and flows smoothly, which makes it easy for readers to understand the core idea of this paper and follow its implementation details.

3. The experimental results are convincing. The authors provide a very detailed description of experimental settings. Besides, this paper exploits multiple methods for comparison and considers various noise settings to verify the effectiveness of the proposed method. The experimental results on synthetic and real-world datasets are convincing. The authors also perform an ablation study to present the proposed method is insensitive to the estimation of noise rate. 

------Issues------
1. I only find the illustration of comparison between CE and CDR in the case of noisy CIFAR-100. This paper aims to reduce the side effect of noisy labels before early stopping, thus CE is an importance baseline in this paper. Can the authors add illustrations of the experimental results in other cases like Figure.2? 

2. The baselines and experimental results are sufficient. Could the authors add some introduction for the baselines and more detailed discussion for experimental results. 

3. The authors may need add some explanation for Eq.(2) and Eq.(5). The proposed method makes use of the memorization effects of deep models. The authors directly write \tilde{S} rather than S in Eq.(5). However, this may be easy to misunderstand. I suggest that the authors can emphasize it or change it. 

4. Some typos need to be corrected. (1) “The underlying issue of directly using the gradient of......”; (2) “Robust positive update uses the gradients to update the critical ones......”.

5. Some minor comments. (1) The experimental results in Table 1 are too dense. (2) The figures are not readable, especially the title is small for me. This makes it a little hard to match the figures with specific cases. (3) The parameter (noise rate) $\tau$ still needs to be estimated, which may be challenging. It will be promising to automatically set this parameter during training. Thus, a more in-depth analysis is worthy of further learning. 

I hope the authors can address these issues carefully to improve this work. 
review_title: Review 2 for "Robust early-learning: Hindering the memorization of noisy labels"

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method for deep learning with noisy labels, which distinguishes the critical parameters and non-critical parameters for fitting clean labels and updates them by different rules. The method is easy to implement and the empirical results are promising. Experiments on both simulated and real-world datasets show it reaches new state-of-the-arts results.

Questions:
- I don’t think the ablation is convincing – why does the proportion of non-critical parameters is assumed to be the same as the noise rate? The ablation only shows the method is insensitive when the estimation of the noise rate is not precise. However, why do we need to estimate the noise rate? What if the proportion of non-critical parameters is assumed to be a constant number? In other words, what will the performance be like if the estimation is largely different from the real noise rate? 
- Since the validation set is also noisy, why does the early stopping criterion adopts the minimum classification error on it? 
review_title: New technique for learning with noisy labels

Meta-review: The paper presents a novel method for learning with noisy labels based on an interesting insight into the learning dynamics of deep neural networks. 

Reviewers unanimously vote for acceptance. I agree with their assessment, and it is my pleasure to recommend the paper for acceptance. 

If I can draw attention to one comment, I strongly agree with R1 that the criterion in Eq. (3) is somewhat poorly motivated. I believe the paper would benefit from a clearer exposition of this part. 

Please make sure to address all reviewers' remarks in the camera-ready version. Thank you for submitting your work to ICLR.
Decision: Accept (Poster)