Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper combines three branches of approaches (1. sample selection, 2. noise robust loss, 3. label correction) to address label noise in classification in a single framework. Specifically, the method includes (1) warm up phase, (2) a novel label-free sample selection, (3) noise aware loss (as a standard technique) and (4) self-training with pseudo labels along with the given labels. The proposed method outperforms prior arts in CIFAR-10/100, especially in high noise regimes (80-90%). More interestingly, in the small sample training regime, gains by the proposed method increase significantly in evaluations with CIFAR-10/100.
main_review: Strengths
- S1: A novel combination of the existing successful approaches with a combination of novel contributions in some components (e.g., sample selection)
- S2: Good empirical gains over LNL SOTA methods (Table 3)

Weaknesses
- W1: Fig 1 is not clear. As the method seems having the order of operations, the figure without the order of operations is not clear. In addition, it is not clear how the pseudo-label generator is used to label the noisy labeled data.
- W2: Empirical gain in small noise regime (SYM 20%-50% in CIFAR-10/100) are small.
- W3: No empirical gain against SOTA in large dataset (Clothing1M)
- W4: There are large gains in large noise regime (more than 80%). But this set-up may not be very realistic.
summary_of_the_review: The proposed method is a novel combination of existing successful components for LNL. Although the proposed method is somewhat novel, the empirical gain does not assert the benefit of the proposed approach. In addition, some of the presentation (e.g., Fig 1) is not clear to understand the proposed method as it omits the order of the procedure.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hybrid framework PARS for Learning with Noisy Labels (LNL) task. The framework jointly leverages the original noisy labels and estimated pseudo labels of all samples for model training. Specifically, for samples whose maximum classification probabilities are higher than a threshold, their original/pseudo labels are used in robust/positive learning, while for the remaining samples, their original/pseudo labels are used in negative learning. When using pseudo labels, strong augmentations are also applied to the samples. Experiments conducted on three public datasets with the traditional and a new low-resource semi-supervised LNL settings show certain improvements over existing methods.
main_review: Strengths:
1. This paper proposes an easy-to-implement method PARS for LNL task and it obtains certain improvements under the traditional LNL setting especially with a large noise ratio and a low-resource semi-supervised LNL setting. The ablation study also shows the contributions of each of the two main improvements in this paper.
2. The paper is well organized and easy to follow, with clear explanations of the algorithm details.

Weaknesses:
1. This paper mainly makes use of two existing techniques, i.e., negative learning and FixMatch (a semi-supervised learning method) to build a better framework. While the motivation of adopting negative learning is relatively clear, I think the authors should further explain the motivation of applying the pseudo-labeling for the entire datasets instead of only the noisy data, the reason of using strong data augmentation in this step, and the difference compared to FixMatch. Since the pseudo-labeling contributes a lot to the results as shown in Table 4, further discussions and analyses are needed to figure out why the pseudo-labeling is so important in PARS? Are the pseudo labels more accurate than original labels or does the consistency between the model predictions of a sample with weak and strong augmentations matter? I think some in-depth analyses lack here. Besides, the authors could try replacing the strong augmentation with weak augmentation to show its advantage with experimental results. 
2. In my opinion, when the noise ratio becomes large, from the perspective of PARS, the LNL task becomes more similar to a semi-supervised learning task, thus PARS (part of which inherits from FixMatch) would achieve a high performance as shown in Table 1. However, when the noise ratio is small, i.e., 20% or 50%, PARS doesn’t perform the best on CIFAR-100 dataset. I think the reason of this situation should be further discussed. Also, on Clothing1M dataset, is the fact that PARS doesn’t perform the best possibly related to the noise ratio in this dataset? 
3. Some claims or components of PARS are not evaluated. (1) In Equation 6, a label-free conﬁdence-based thresholding is proposed and the authors claim that it would reduce the self-conﬁrmation bias. Are there any empirical results that would support this claim? (2) I think an additional ablation study is needed for validating the effectiveness of Equation 10.

summary_of_the_review: This paper unifies several well-performed methods for LNL task to build a better framework, which achieves certain improvements especially under high-noise scenarios. However, some in-depth analyses about the effectiveness of the pseudo-labeling in PARS and the inferior performances under some settings still lack in this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a unified approach to handle noisy labels for training neural networks and utilize all the training data to learn effectively when noise is present. The authors utilize the assumed correct and noisy labels with different loss functions adjusted using different weights in order to adjust the impact on training. The authors show how their approach improves over CIFAR-10 and CIFAR-100 with high noise ratio and demonstrate competitive results in Clothing1M dataset. The authors also show strong results in semi-supervised setting, outperforming FixMatch significantly.
main_review: __Strengths__
-	The paper address problems arising from the three main approaches used to tackle noisy labels. They discuss about the shortcomings of sample selection algorithms, loss functions designed for noisy distribution and label correction/pseudo-labelling algorithms. While all of them reduce noisy labels in successive iterations, a common problem stems from confirmation bias of the network due to higher noise in labels. The authors study and explain various ways to address this issue and propose a noise aware loss mechanism that adjust the weight for ambiguous and noisy labels, reducing the training error from prior approaches.
-	For sample selection mechanism, the authors use the confidence of all classes for thresholding and decide based on the highest probability value of all present classes. This is different from prior works which use the probability value of raw label (which could be noisy pseudo-label) class for thresholding and deciding if the sample is noisy or clean. The authors’ proposed mechanism only uses the network’s output for separating the samples into ambiguous and noisy set and can reduce noise propagation from one iteration to the other.
-	The use of different loss functions for different sample set based on the noise level is also intuitive to adjust the impact they have during the training process. Dealing with ambiguous and noisy labels with independent losses allows adjustment of feedback from these labels better. This is one of the valid methods to handle learning from noisy labels which this paper sheds more light on and their results in CIFAR-10/100 validates the claim. Clothing1M scores are similar to prior work which hints that this loss formulation is equally valid as previous methods.
-	The proposed method shows better resilience to noisy labels when learning in a low-resource semi-supervised setting and significantly outperforms prior works. Compared to FixMatch which is dedicated to learn using semi-supervised approach on low resource, the proposed approach is able to handle unlabelled data better and reduce impact from noise.

__Weakness__
-	The authors specify they use pseudo-labels from both D_A and D_N in Eq 9 to train the network. Is this pseudo-label based self-training done together with noise aware training with raw labels? Or is it trained in alternating iterations with either of those losses? 
-	Continuing on previous point, a large gain is seen with the use of pseudo-labels in training which suggests the model’s prediction after warm-up is better than the raw labels. In that case, the raw labels might differ from the pseudo-labels significantly. The small performance gap by removing negative learning in both pseudo and raw loss shows its more dependent on the positive learning using perceived labels. In this case, the network is being trained using both raw label and pseudo-label for the same sample at the same time, which could be different from each other. Is there a detailed explanation for this occurrence?
-	Have the authors tried not using raw labels after warm-up and only use the pseudo-labels for all loss formulations? This is interesting as biggest gain comes from pseudo-labels after the initial warm-up.
-	From appendix A.4, using noise robust loss on pseudo-labels performs lower than using positive learning from Eq 9. This finding is also interesting as the pseudo-label would also be assumed to have certain ambiguity to it that could benefit from the noise robust loss. Is the model underfit caused by pseudo-labels being very accurate that positive learning has better feedback than a toned-down robust loss?
-	Since the approach shows more impact in semi-supervised setting, perhaps including more study on that similar to FixMatch would benefit the paper and the readers more.

summary_of_the_review: The paper provides a unified algorithm to learn from noisy label and shows convincing results on various datasets. The results on semi-supervised setting are also appreciated and can be useful for future research in this direction.
review_title: null

Meta-review: This paper a framework of learning with noisy labels named PARS that combines three types of approaches, i.e., sample selection, noise robust loss, and label correction.  The framework leverages both original noisy labels and estimated pseudo labels of all samples for improving the training performance, and the empirical studies demonstrated competitive results on CIFAR datasets especially in high-noise and low-resource settings. 

Reviewers raised some major concerns about the weaknesses. For example, empirical gain in small noise regime are small or negligible, and no empirical gain against SOTA in large dataset with real-world noise (Clothing1M).  While large gains in large noise regime (more than 80%), such setting may not be very realistic and there also  lack of in-depth analysis on the sources of the gain (e.g., it is unknown if the gain is mainly because of using a better SSL or other factors since LNL becomes more similar to SSL when noise is very high). For technical novelty perspective, while the proposed approach is new, the overall novelty may not be very significant as this paper mainly combines existing techniques, e.g., negative learning and FixMatch (a semi-supervised learning method) in the proposed learning approach. 

Authors have made great efforts for addressing the reviewers’ concerns partly, but some major concerns on the technical novelty and empirical studies remain.  Therefore, the paper is not recommended for acceptance in its current form. I hope authors found the review comments and discussions useful and constructive, and like to see it accepted in the near future after these issues are fully addressed.
Decision: Reject