Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Summary
The paper proposes a joint pseudo-labelling and curriculum learning strategy. It addresses the problem of training with noisy labels, especially robustly correcting noisy labels.

# Recommendation
Borderline paper, with a lot of strong points, as given below. My major concern, is that the improvement is e.g. due to more advanced data augmentation (Cubuk et al. (2020)). I think it should be easy for the authors to provide more details here, i.e. an ablation study, that could help in deciding. So far I am unfortunately not conviced, that the progress is due to the proposed method alone.

# Strong/Weak points
## Pros
Coupling the acquisition of pseudo-labels and selection of clean labels and especially a smooth curriculum using both is an interesting idea.
The proposed usage of the exponential moving average is reasonably motivated by the oscillating patterns of the instantaneous loss values.
The proposed algorithm is well embedded in recent publications and well motivated.
Multiple noise rates and multiple datasets are considered, proving the methods applicability.

## Cons
The origin of Equation 6 is not as clear as it should be. The description of the terms in eq. 6 could be improved, especially the fact, of "abuse notation".
While the coupling of $p_t(i)$ and $q_t(i)$ seems to be a good idea, the theoretical justification is not as convincing to me. Further theoretical background or experimental verification would be good to support the claim of this coupling being meaningful.
This and the fact, that a ablation study is missing makes it hard to judge the methods contribution.

# Questions to the authors
A more detailed comparison e.g. experimental between using $p_t(i)$ and $q_t(i)$ separatly, vs. using the jointly would be very interesting.

What data augmentation was used for all other methods, except RoCL? If all other results are taken from other papers, the aim should be to excatly reproduce the setting used there, or at least prove via an ablation study, the impact of each part of your training.

What happens if you do not use data augmentation at all?

Could you provide more details on $\tau_{1,2}$ and $\lambda$, so far do not see a justification for their respective values. Simple hyperparameter search? Is there a theoretical interpretation, or limits you could derive?

# Detailed comments
Eq. 2 is referenced before it is stated, consider rearranging.

Eq. 6 please clarify the description, e.g. what is "abuse notation", why are you replacing by instantaneous counterparts here?

Minor comment, the text in Figure 1 could be larger


Minor typo "Simply removing noisy data from training discards important information about data distribution." --> "the data distribution", some more typos in that same paragraph, please take some time to correct them.
review_title: Interesting method, with many experiment and good performance, but some details missing

final_score: rating: 6: Marginally above acceptance threshold
review_content: This article is concerned with the problem of training models under noisy data. The authors first adopt the loss and output consistency for data selection. EMA method is used for smoothing to obtain more accurate clean label detection. Meanwhile, through the introduction of temperature hyperparameters, the model gradually completes the transition from supervised learning using clean labels to self-supervised learning using noisy labels. 

Strength:
1.	The authors use both loss function and prediction invariance for sample selection. The proposed model adjusts the sample selection strategy for different training stages to obtain a more informative training sample.
2.	The authors used the EMA algorithm to smooth the sample selection metrics, resulting in better clean sample detection performance.
3.	The change of hyperparameters allows the model to accomplish the change of sample selection strategy and transform the training strategy of the model from fully supervised to self-supervised.
4.	The authors provide a detailed theoretical analysis and experimental demonstration of the proposed method, which achieves SOTA performance in both the CIFAR10/100 and WebVision datasets

Weakness:
1.	As several modifications mentioned in Section 3.4 were used, it would be better to provide some ablation experiments of these tricks to validate the model performance further. 
2.	The model involves many hyperparameters. Thus, the selection of the hyperparameters in the paper needs further explanation. 
3.	A brief conclusion of the article and a summary of this paper's contributions need to be provided.
4.     Approaches that leveraging noisy label noise label regularization and multi-label co-regularization were not reviewed or compared in this paper.
review_title: This article focuses on model training tasks under noisy labels. The authors combine classification loss and prediction inconsistency and complete the shift the training strategies from supervision to self-supervision with the hyperparameters' schedule.

final_score: rating: 5: Marginally below acceptance threshold
review_content: #Summary

This paper proposes a robust curriculum learning method that interpolates a regular loss and a consistency loss, aiming at a smooth transition from learning from clean data and then to noisy data with pseudo labels.

#Pros
- The paper is well-written, and the results over several benchmark datasets seem to be strong.
- Some of the insights provided in this paper, seem rather interesting, like by transitioning from supervised learning, to self-supervised learning of noisy data, can better benefit the learning process.

#Cons
- The authors should perform an ablation study of the RoCL method. Currently the final proposed method mixes too many components, and it is hard to disentangle the true contribution of each component. For example, in section 3.4, the authors mentioned additional techniques were added, like class-balance regularization, label-smoothing, and mix-up, a further analysis is required to understand the true contribution for each individual part.

- Similarly, for all the baselines used, the authors should do a better categorization of each baseline method, to ensure a fairer comparison. For example, does any of the baselines use model averaging, mix-up, label-smoothing, or data augmentations? How is RoCL without mix-up compared to baselines that don't use mix-up? 

- If I understand correctly, one of the key contributions is the interplay between the regular loss and the consistency loss, but the scheduling part is not super principled and seems to involve a lot of ad-hoc tuning of the balancing parameter $\lambda$ and the temperatures. Is there a principled way to balance the two losses?

- The RoCL algorithm seems to involve a lot of parameters. The averaging parameter $\gamma$ for EMA, $\lambda$ for the trade-off between loss and consistency, and tempature $\tau_1, \tau_2$ (and the additional params required for scheduling the temperatures properly). In practice, how are those parameters picked? Is there a lot of careful tuning required?

- The paper seems to have combined a lot of existing techniques. The paper would be stronger if the authors can provide further analysis to better understand how/why RoCL works. E.g., how important is EMA/data augmentation/data sampling respectively?


#Overall recommendation

Overall I'm on the fence but tend to reject. I think this paper can benefit a lot from better organizing of the methods and results, with a clearer focus on its major contributions. Currently the experimental result is a mixture of multiple existing approaches and the proposed RoCL method, it is hard to know what the role of each approach is and whether the proposed RoCL method indeed improves curriculum learning. I think this paper has the potential of providing some great insights, but the current set of results are rather noisy.

#Minor comments
- Figure 1 is really small and hard to read.
- The paper needs to be better organized. Currently it seems like the authors run out of space and rushed through the experimental results.
- Can you clarify if the $\gamma$ is the same for Eq. 1, Eq. 3, and for computing $\bar{\theta}_t$? If they are different please use different notations.
- I don't quite get Eq. 9, why is the temperature defined in this way?
- How important is the sampling part in Algorithm 1? It involves another parameter $b_k$ and how sensitive is RoCL to the choice of that parameter?
- The consistency loss over augmented examples is also a commonly-adopted technique in semi-supervised learning, citations to the use of those methods in existing literature are missing, e.g., [1] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. NeuIPS 2016.
[2] Samuli Laine and Timo Aila. Temporal ensembling for semisupervised learning. ICLR 2017.
review_title: Interesting work on Robust Curriculum Learning via interplay between loss and consistency

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a curriculum learning method to handle noisily labeled data. The idea is to introduce a consistency measure instead of directly apply a loss function for the typical supervised learning, where the specific consistency is measure for both temporal dimension along neighboring steps and spatial dimension over different data augmentation samples for a given real sample. The consistency measure is applied for self-supervision while the loss function is applied for supervised learning. The final optimization is managed between the two components through weighting parameters, such that the training is made through a migration from a supervised learning to self-supervision by gradually adapting the weighting parameters. Evaluations are reported on Cifar10/100, WebVision, and ILSVRC2012.

The motivation and rationale of this work makes sense, and the proposed method appears to be correct, at least at the conceptual level though I did not check in detail. After the discussions and the interactions with the authors, it came to my attention that one of my comments was wrong - I looked at a different paper that led me to have the conclusion that the authors cited a wrong performance number from a competitor in the literature; it turns out that I was wrong and I apologize to the authors.

The paper at the initial submission version read very rough, with a lot of grammatical errors, typos, or misleading/incorrect statements. In the experiment section, it is stated that three datasets were used for evaluations but in Table 1 it appears there is the fourth dataset ILSVRC2012 used, which never mentioned in the text, nor is mentioned in the paper on how that dataset is used (a portion like WebVision or the full). The whole paper ended up with no conclusion or discussion. After the discussions with the authors, it became clearer that the mentioned three datasets in the text were for training and ILSVRC2012 was used for evaluation. But still it would be a lot clearer to have such a statement in the text.

I had the comment that the title of the paper was misleading. The title reads: ROBUST CURRICULUM LEARNING: FROM CLEAN LAEL DETECTION TO NOISY LABEL SELF-CORRECTION. However, the proposed method, together with all the reported evaluations, focuses on learning the noise (in the labels); neither clean label detection nor noisy label correction is addressed. The authors disagreed with me but I was still not convinced by their argument. For noisy label self-correction, it may be relevant; but for clear label detection, I don’t think so.

I had a comment regarding the scale of the datasets used in the evaluations. But after having read the competitors’ work such as MentorMix, I now took it back and agreed with the authors. On the other hand, I agree with the comments raised by the other reviewers on lacking the ablation studies. I appreciated the authors’ efforts to report back the ablation studies, though only in part, and the results appeared to be convincing to me.

So overall, after the discussions and the revision provided by the authors, I am convinced that the paper is above the acceptance threshold. The paper does have presentation issue, and lacks extensive ablation studies. 

review_title: After the discussions I found that one of my comments was wrong and now I revised the review

Meta-review: This paper has been thoroughly evaluated by four expert reviewers and it had received one public comment. The authors provided extensive explanations and added technical updates to the contents of their submission in response to constructive critiques from the reviewers. Even though some minor issues have not been fully resolved in the discussion between the authors and the reviewers, I consider this paper worthy of inclusion in the program of ICLR 2021 since, albeit marginally, the apparent strengths outweigh its outstanding limitations.  
Decision: Accept (Poster)