Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a training-free solution by using only good representations to detect noisy labels. The author designed two detection methods of voting and ranking to filter instances that are likely to be corrupted. The major contribution is proposing a training-free solution to efficiently detect noisy labels, which is different from most current methods. Also, theoretical analysis is conducted on the worst-case error bound and the choice of K.
main_review: The perspective of this paper is somewhat novel compared with existing training-based methods. The presented solution would be beneficial in detecting noisy labels. The authors present their algorithm clearly and the theoretical analyses are reasonable.
The whole method is based on the assumption that good representations can distinguish the noisy and clean labels. But the representations of hard samples are likely to be similar to the noisy ones. Although SimiRep achieves high F1-scores, the filtered hard samples can have a significant impact on model robustness. It might be necessary to conduct follow-up experiments on the model performance without filtered samples of SimiRep.
There are some mistakes in this paper. In the third line from the last of section 3.1, “the noisy labels are referred in representation learning”. It’s not true and should be “are not referred”. In the first line of page 5, the example of y_n3 is written into “0.0.33”.
summary_of_the_review: The perspective is novel, and the presented solution would be beneficial in detecting noisy labels. But the significance of detecting noisy labels is uncertain and needs more experimental proof.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to detect noisy labels given good representations (e.g., ones pre-trained by contrastive learning approaches). 
The proposed noisy-label detection method uses  the neighborhood information defined by a good set of representations in two ways: 

1) checks the noisy label consensuses of nearby representations

2) scores each instance by its likelihood of being clean and filters out a guaranteed percentage of instances with low scores as corrupted ones 

The work provide definitions for good representation and further proves a worst-case error bound for the ranking-based method given a 'good enough' representation. It also provides empirical results for its proposed method. 
main_review: Strengths: the paper proposes a new perspective to detect noisy labels
Weaknesses: my main concern is on the good representation assumption. To be more specific,

1. Though it is possible to obtain some good representations via pre-training on other datasets or self-supervised learning on the noisy datasets, it is still unclear how the obtained representations could be guaranteed to help with noisy label detection. It would be helpful to include experiments evaluating the goodness of representations obtained from various approaches based on the proposed criteria (e.g., Definition 1 and 2).

2. In Table 1 and 2, as the proposed method uses representations pre-trained by SimCLR, a self-supervised learning method, it seems unfair to compare with other noise detection methods, which do not have access to the good representations from SSL.

3. The paper claims the proposed method as training-free, which may be misleading and a bit overclaim since the good representation assumed by the proposed method still requires learning. Supervision-free maybe a better choice.
summary_of_the_review: This work proposes an interesting perspective to analyze how the representations could help with noise detection. Yet, the major assumption of this work that a good representation is given still needs further justifications. Additional experimental results are also needed to fully support the claim.

===Update after rebuttal===
Thanks to the authors' response, which resolved some of my confusions. I am increasing my score for the additional discussions and results
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a training-free approach to detect samples with noisy labels by leveraging representations learned from pre-trained models. The models can be obtained by either supervised or self-supervised pre-training. The authors then argue that samples in the pre-trained manifold should be closer if they share the same clean label, therefore one can use (a) local voting or (b) ranking methods to detect samples with corrupted labels. Experiments on CIFAR10/100 show improvement over other learning-based approaches (CORES, CL, TracIn).
main_review: Strengths
* The idea of using a pre-trained model to help detect noisy samples is interesting. The training-free approach is also largely different from existing works in literature.
* The finding in Sec 5.3 is particularly interesting as well. Table 3 in this section shows that contrastive pre-training shows better generalization ability than supervised pre-training for the task being addressed here. 

Weaknesses
* The major concern of this work is about the experimental setup. CIFAR10/100 has a clear class definition across only 10 or 100 classes. By using a well-trained model pre-trained on a large amount of data, the representations of the samples from CIFAR are already well separated. That is also the reason the proposed K-NN approach works well compared to learning-based methods that do not have the advantage of large-scale pre-training. To justify the effectiveness of the proposed method in practice, it is crucial to test the algorithm on more complex and larger datasets. 
* The theoretical analyses are highly appreciated in the submission. However, the analyses are based on a strong assumption that nearby representations should belong to the same true class. This might be true for simple CIFAR10 with a balanced training sample across 10 classes. In reality, the training set is usually imbalanced and nearby samples could have completely different labels. 
* Figure 1 can be improved. The green circle in the figure is not defined.
* Typo: page 5 y_n3=[0.34, 0.033, 0.33] -> [0.34, 0.33, 0.33]
summary_of_the_review: Although I appreciate the idea of using pre-trained models to provide guidance in selecting corrupted samples. It is also important to ensure the proposed method works well in practice, especially for imbalance or more complex large-scale settings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a training-free instancewise noise label detection method. The main motivation of this paper is the observation that deep models generalize poorly because memorizing noisy labels in supervised training while using only representations may avoid this issue.  The authors suppose a good representation extractor is given and generate the initial soft labels based on the clusterability of representations using kNN. Followed by that,  they perform a local voting and global ranking-based scoring system to detect the corrupted labels. The main contribution of this paper is: It introduces a representation-based method instead of directly training a deep model on the corrupted data, which is more efficient and may avoid overfitting noisy labels. 

main_review: The paper is well organized and easy to follow. The intuition of the paper, which lies in utilizing proper representation instead of training a deep inference model to avoid memorizing corrupted data, also sounds somewhat interesting and seem to work. Besises, the proposed method can detect label noise in the instancewise manner. However, my concerns mainly lie in the methodology and the experiment.

First, what is the specific definition and criterion of good representation emphasized in this paper? It seems that the prerequisite of the proposed method is “given good representation”. The authors have done some basic tests in section 5.3 (table 3) that indicates the representations lead to accurate inference result is a good representation. However, it is the post-inference analysis, is there any way to select a good representation extractor g(·) in advance? Besides, the dependence on good representation may become the bottleneck of the proposed method since visual representation itself is an important and challenging topic.
 
Second, this paper takes the corrupted CIFAR-10 and CIFAR-100 (corrupted by automatic labeling tool) as the real-world noisy dataset for experiments, which is not very convincing. The acknowledged noisy-label dataset such as Webvision and Cothing1M could be more helpful in verifying the effectiveness and universality of the proposed method. Though these datasets may not involve the ground truth labels, there are still many ways to achieve the quantitative comparison test. For example, you can filter the wrong labels detected by the proposed method, and compare the overall inference accuracy between the proposed method and the SOTAs. 

In addition, though the authors claim that their method is universal, they adopt clustering methods like k-NN to generate the soft labels and resolve the discrimination task. I thus wonder if the increasing number of object categories may result in a significant drop in noise detection accuracy. That is also one of the reasons why I wonder about the convincingness of the datasets used in this paper.

There are also some minor concerns：
1. The related work is too brief to reflect the association and difference between the proposed method and the existing methods.
2. The ambiguous meaning of words. Eg, why do the authors call the representation extractor g(·)  k perfect extractor if it induces k-NN label clusterability in Definition 2? Why perfect?  
3. There are some writing issues to fix, the authors may need to proofread the papers more carefully. eg. Property 1, "With in"  in the sentence " With in the same instance, bla bla".
summary_of_the_review: The paper proposes a training-free instancewise noise label detection method. It supposes a good representation extractor is given and generates the initial soft labels based on the clusterability of representations using kNN. Followed by that,  the authors perform a local voting and global ranking-based scoring system to detect the corrupted labels. In general, the paper is well written and easy to follow. The main idea of utilizing good representation instead of training a deep inference model to avoid memorizing corrupted data sounds interesting and seems to work on CIFAR10 and CIFAR100 datasets.  However, I have some concerns about the definition and criterion of good representation which is the prerequisite of the method. As the proposed method uses kNN to analyze the clusterability of representations, I also wonder the increase of object categories may cause a significant accuracy drop in discrimination. Besides, the author only perform experiments on CIFAR10 and CIFAR100 instead of the acknowledged noisy-label datasets, which is not convincing enough for me. There are also some minor issues in the paper to be taken care of.
review_title: null

Meta-review: This paper proposed a training-free method to detect noisy labels based a given pretrained representation. The author suggested two methods based on either voting or ranking to filter noisy instances with corrupted labels without training. Reviewers generally agree that the technical novelty and contributions are only limited or marginally significant. Also experiments are not very convincing as there lacks of extensive comparisons with many existing methods for either noisy label removal or learning with noisy labels and the datasets are somewhat simple and not complex enough.
Decision: Reject