Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a two-stage approach to learning with noisy labels (LNL).
1. a. Clean sample selection based on cosine similarity with k nearest neighbors in embedding space: the average of class distribution of those neighbors should be consistent with the label for sample to be selected.  
    b. Noisy sample relabeling using temporal self-ensemble: the prediction is defined is average over last L epochs. The sample is relabeled if the confidence of the prediction is higher than some threshold.
2. Training with self-consistency regularization: the regular training with mixup regularization is performed on selected and relabeled samples, with consistency regularization in form of cosine similarity.
The paper tests the performance of method on multiple datasets both with synthetic and real-life noise.
main_review: **Strength**
- State-of-the art results in different settings.
- The methods works both with open- and closed-set noise.
- While the ideas of neighborhood-based sample selection, sample relabeling, and consistency regularization are well-known, the particular implementation is novel and interesting.
**Weaknesses**
- The presentation should be improved. The authors explain well-known concepts like softmax gradient but some details of the proposed methods are explained briefly or not at all.
- There is a number of existing methods that you don't compare to: C2D (Zheltonozhskii et al., 2021), which you cite but do not include in comparison and which outperforms you at least in some setups (e.g., CIFAR-100 90% noise -- 58.45±0.30% as opposed to 56.8% you report); FaMUS (Xu et al., 2021). Similarly AugDesc (Nishi et al., 2021) is mention in Tab. 1 but not in main comparison.
**Comments and questions**
- Most of existing papers report top-1 and top-5 accuracy for WebVision training, both on ImageNet and WebVision validation sets. Can you provide those numbers?
- Comparison with other self-consistency approaches could be helpful (e.g., L2 loss is often used in semi-supervised learning or contrastive loss)
- Different parameters are compared on different datasets, which makes the comparison hard. You should use same settings (or multiple settings, but for every experiment). Also, you show dependence on $\theta_r$ but not $\theta_s$. Why?
- Using average of at least two runs is better than nothing, but in some cases differences are clearly not significant, e.g. Tab. 1 and Tab. 3. Showing standard deviation would make clear which improvements are significant.
- Probably balancing would be more effective in unbalanced settings, either synthetic or real-life (maybe Clothing1M, Xiao et al., 2015)?
- For ablation study, what happens if you use only one of sample selection or sample relabeling?
- Tab. 3: balanced selection refers to weighting $p'$ by $\pi^{-1}$?


**Minor comments and typos**
- Section 3.3 "are the inverses of the entries of the vector $\pi$ of the class probabilities in the whole dataset" is hard to read. I assume you meant something like "The vector $\pi$ contains the class probabilities in the whole dataset, and we denote by $\pi^{-1}$ its elementwise inverse".
- Section 4.4 has wrong citation format. It should be "DivideMix (Li et al., 2020a)" and not "DivideMix Li et al., (2020a)" (there are more of those in this paragraph and possibly elsewhere).
- Section 4.2 "Genrally->Generally"
- Table 7 caption "ANIMAl->ANIMAL"


**Post-rebuttal/revision**
The authors have addressed many concerns, especially experiment-wise. New comparisons were added (C2D still missing from Tab. 6 though -- the results for WebVision appear in table 4 in the paper), as well as Clothing1M experiments. The presentation still can be improved, in particular, I suggest to proofread the newly-written part.

**References**

Nishi, Kento, et al. "Augmentation strategies for learning with noisy labels." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Xiao, Tong, et al. "Learning from massive noisy labeled data for image classification." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

Xu, Youjiang, et al. "Faster meta update strategy for noise-robust deep learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Zheltonozhskii, Evgenii, et al. "Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels." arXiv preprint arXiv:2103.13646 (2021).
summary_of_the_review: The proposed method is novel and interesting and shows promising results in various settings.  My main problem is quality of writing and presentation. if the authors can address my comments and questions and improve the general presentation quality, I'll happily increase the recommendation to accept.

**Post-rebuttal/revision**
The authors have done good job addressing my concerns (as well as other reviewers'). The experimental results look interesting and while presentation (in particular writing) can be improved, I think the paper should be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes "S3" framework for learning with noisy labels. Specifically, S3 consists of two stages. In the first stage, a relabelling approach and normalized neighboring voting are utilized to guide efficient sample selection; in the second stage, supervised loss (Mixup) and self-consistency loss are used to train networks on selected samples. S3 can be applied to both close-set label noise and open-set label noise and exhibit good performance on several benchmark datasets. 
main_review: Strength:

1.  The paper is easy to follow. The proposed "S3"  framework achieves very good performance on both close-set label noise and open-set label noise. 

2. The ablation studies are well designed. Each part of S3 along with the hyper-parameters are analyzed. 

Concerns:

1. I do not think "S3" is a simple framework or simple method. It contains two stages with multiple techniques such as KNN voting, relabelling, sample selection, mixup and consistency loss, which make it hard to understand which part exactly contributes most to the performance. Further, I am worried that the techniques such as KNN voting may increase the training time. Authors are encouraged to calculate the training time of each module in S3. 

2. KNN voting-based relabelling, mixup and consistency loss have already been explored in the literature of learning with noisy labels [A1, A2, A3].  I do admire the performance gain brought by S3 in the experiments. However, without deeper analyses, I think the technical novelty is a little limited since it seems that S3 simply combines existing approaches with little modifications. To improve the paper, authors are encouraged to elaborate the main difference among S3 and other approaches in [A1, A2, A3] and state why the combination or the modification is important. 

3.  From Figure 2 (a), the network still has good performance when $\theta_{r}$ approximates 1. Does it mean that relabelling is not very necessary? I think it will be much better if authors calculate the F-score to represent the quality of sample selection and relabelling especially when open-set label noise involved. 



Reference:

A1: Multi-Objective Interpolation Training for Robustness to Label Noise. CVPR 2021

A2: Unsupervised Label Noise Modeling and Loss Correction. ICML 2019

A3: Consistency Regularization Can Improve Robustness to Label Noise. ICML 2021


summary_of_the_review: **Pre-rebuttal:** S3 achieves very good performance but the technical novelty is limited. In the current version, it seems that S3 is more like a combination work. Thus I give the initial score of 5. I will consider increasing my score if authors can well clear my concerns.


**Post-rebuttal & Final recommendation:**

I thank the authors for their further reply and respect to other works. 

It is relatively a hard decision for me. Experimentally, this paper does achieve promising performance. The authors did a good literature review in the paper 
and elaborate the difference among S3 and other methods in the rebuttal. The designed S3 framework only has few hyper-parameters which are shown robust. To this, I appreciate the efforts and contribution.  However, I still think the combination of components of S3 lacks enough novelty since each component is not new in the literature. 
For the DivideMix that the authors mention, I admit that DivideMix is also a combination paper. However, the performance of DivideMix 
consistently outperforms all the previous methods by a large margin especially when the noise rate is high (at that time). While it seems S3 only has little improvement compared to C2D, which is also a work that uses self-supervised learning. In some cases, C2D is even better. Further, considering the supervised, self-supervised training framework is also explored in other works (Co-learning) and S3 currently does not have theoretical justifications such as why KNN voting-based relabelling, or other component, works on both close-set and open-set label noise, I remain my final score as 5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on learning with noisy labels problems, which constructs an iterative learning framework to refine labeling set and train the model parameters.  It demonstrates its advantages by comparing with current baselines on a range of datasets.
main_review: The novelty of this paper is ad-hoc, which stacks the benefits of sample selection and contrastive learning. Considering several previous works have explored them [1, 2, 3] and the proposed method in this paper has minor difference with previous works, it is below the bar of ICLR for the area of learning with noisy labels.

[1] Label Distribution for Learning with Noisy Labels. IJCAI 2020
[2] Contrastive Learning Improves Model Robustness Under Label Noise. CVPRW 2021
[3] Learning from Noisy Data with Robust Representation Learning. ICCV 2021
summary_of_the_review: This paper focuses on learning with noisy labels problems, which constructs an iterative learning framework to refine labeling set and train the model parameters.  It demonstrates its advantages by comparing with current baselines on a range of datasets. However, the novelty of this paper is ad-hoc, which stacks the benefits of sample selection and contrastive learning. Considering several previous works have explored them [1, 2, 3] and the proposed method in this paper has minor difference with previous works, it is below the bar of ICLR for the area of learning with noisy labels. Another problem is k-NN search for neighborhood voting is computational-expensive especially for large-scale datasets.

[1] Label Distribution for Learning with Noisy Labels. IJCAI 2020
[2] Contrastive Learning Improves Model Robustness Under Label Noise. CVPRW 2021
[3] Learning from Noisy Data with Robust Representation Learning. ICCV 2021
review_title: null

Meta-review: This work describes a two-stage method for learning with noisy labels. The crux of the reviews, discussions with the authors and post-rebuttal discussions between reviews (and myself) was related to the novelty of this work. The main concern is that while this body of work presents a relatively solid method (from an empirical point of view), the underlying components are not altogether that novel, and have been used in the context of learning with noisy labels before. Fundamentally, the proposed S3 method did not feel *convincingly* better, given its relative lack of novel technical insights. I appreciate that this is a frustrating reasoning to get -- after all, much of what we do in empirical ML is combinations of existing things. Ultimately, there was consensus amongst the reviewers that the work did not have sufficient insights or such outstanding empirical results so as to overcome this relative lack of technical novelty. 

All the reviewers have engaged meaningfully in discussions, provided constructive feedback and I hope that this will make subsequent iterations of this work better in many dimensions.
Decision: Reject