A Good Representation Detects Noisy Labels
Authors: Zhaowei Zhu
Zihao Dong
Hao Cheng
Yang Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: noisy labels
sample selection
training-free
representation
noise dection

Abstract: Label noise is pervasive in real-world datasets, which encodes wrong correlation patterns and impairs the generalization of deep neural networks (DNNs). It is critical to find efficient ways to detect the corrupted patterns. Current methods primarily focus on designing robust training techniques to prevent DNNs from memorizing corrupted patterns. This approach has two outstanding caveats: 1) applying this approach to each individual dataset would often require customized training processes; 2) as long as the model is trained with noisy supervisions, overfitting to corrupted patterns is often hard to avoid, leading to performance drop in detection.  In this paper, given good representations, we propose a universally applicable and training-free solution to detect noisy labels. Intuitively, good representations help define "neighbors" of each training instance, and closer instances are more likely to share the same clean label. Based on the neighborhood information, we propose two methods: the first one uses "local voting" via checking the noisy label consensuses of nearby representations. The second one is a ranking-based approach that scores each instance and filters out a guaranteed number of instances that are likely to be corrupted, again using only representations. Given good (but possibly imperfect) representations that are commonly available in practice, we theoretically analyze how they affect the local voting and provide guidelines for tuning neighborhood size. We also prove the worst-case error bound for the ranking-based method. Experiments with both synthetic and real-world label noise demonstrate our training-free solutions are consistently and significantly improving over most of the training-based baselines.

Under review as a conference paper at ICLR 2022
A GOOD REPRESENTATION DETECTS NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
Label noise is pervasive in real-world datasets, which encodes wrong correlation patterns and impairs the generalization of deep neural networks (DNNs). It is critical find efficient ways detect the corrupted patterns. Current methods primarily focus on designing robust training techniques to prevent DNNs from memorizing corrupted patterns. This approach has two outstanding caveats: 1) applying this approach to each individual dataset would often require customized training processes; 2) as long as the model is trained with noisy supervisions, overfitting to corrupted patterns is often hard to avoid, leading to performance drop in detection. this paper, given good representations, we propose a uni- versally applicable and training-free solution to detect noisy labels. Intuitively, good representations help define "neighbors" of each training instance, and closer instances are more likely to share the same clean label. Based on the neighbor- hood information, we propose two methods: the first one uses "local voting" via checking the noisy label consensuses of nearby representations. The second one is a ranking-based approach that scores each instance and filters out a guaranteed number of instances that are likely to be corrupted, again using only represen- tations. Given good (but possibly imperfect) representations that are commonly available practice, we theoretically analyze how they affect the local voting and provide guidelines for tuning neighborhood size. We also prove the worst-case error bound for the ranking-basee based method. Experiments with both synthetic and real-word weal-world label noise demonstrate our training-free free solutions are consistently and significantly improving over most of the training-baa based baselines.
INTRODUCTION
The generalization of deep neural networks (DNNs) depends on the quality and the quantity of the data. Nonetheless, in practice real-world datasets often contain label noise that challenges the above assumption (Krizhevsky et al. 2012) Zhang al. 2017 Agarwal et al. 2016). Employing human workers to cleaning annotations is one rehabbe way to improve the label quality, but it is too expensive and time-consuming for large-scale dataset. One promising way automatically cleaning up label errors is to first algorithnically detect possible label errors from a large-scale dataset (Cheng al. 2021 Northcutt et al.) 2021a Pruthi al. 2020 Bahri al. 2020), and then correct them using either algorithm or crowdsourcing (Northcutt et al. 2021b).
Almost all the algorithmic detection approaches focus on designing customized training processes to learn with noisy labels, where the idea is to train DNNs with noisy supervisions and then make decisions based on the output (Northcutt al. 2021a) or gradients (Pruthi et al., 2020p of the last logit layer of the trained model. The high-level intuition of these methods is the memorization effects (Han et al. 2020), i.e., instances with label errors, a.k.a.. corrupted instances, tend be harder to be learned by DNNs than clean instances (Xia et al. 2021 Liu et al. 2020). By setting appropriate hyperparameters to utilize the memorization effect, corrupted instances could be identified.
The above methods suffer from two major limitations: 1) the customized training processes are task-specific and may require fine-tuning hyperparameters for different datasets/noise; 2) as long as the model is trained with noisy supervisions, the memorization of corrupted instances exists. The model will "subjectively" and wrongly treat the memorized/lvverfited corrupted instances as clean. For example, some low-frequencyffare clean instances may be harder to memorize than high- frequency/common corrupted instances. Memorizing these corrupted instances lead to unexpected and disparate impacts (Liu 2021). One way avoid memorizing/overfiting is to drop the depen-

---

Under review as a conference paper at ICLR 2022
dency training using the noisy supervisions, which motivates us to design training free method to find label errors. Intuitively. we can carefully use the information from nearby representations to infer whether one instance is corrupted or not. This observation inspired our solution of using a good representation to first identify "neighbor" instances for each training example, whose noisy labels jointly would help us perform the detection. Note the representation extractor could he fully independent of noisy supervisions by referring to unsupervised learning or self-supervised learning, or even adapting from other tasks.
Our training-free free method enables more possibilities beyond better detection result. For example, the concerns of the required assumptions and hyperparameter tuning in those training- rraining-based methods will now be released due to our training-free free property. The complexity of such solution will also be much lower, again due to the removal of the possibly involved training processes. This light detection solution also has the potential to serve pre-processing module to prepare data for other sophisticated tasks (e.g., semi-supervised learning (Xie et al.] 2019 (Berthelot et al.. (019)).
Our main contributions are: 1) New perspective: Different from current methods that train cus- tomized models on noisy datasets, we proposed a training-free solution to efficiently detect noisy labels given good representations. We believe this is the first attempt of the same kind to the best of our knowledge. 2) Efficient algorithms: Based on the neighborhood information, we propose two methods: voting-based local detection method that only requires checking the noisy label con- sensuses of nearby representations, and a ranking-based global detection method that scores each instance by its likelihood of being clean and filters low scores as corrupted ones. 3) Theoretical analyses: We theoretically analyze how a good rep- resentation (but possibly imperfect in practice) affects the local voting and provide guidelines for tuning neighborhood size. We also prove the worst-case error bound for the ranking-based method. 4) Numerical findings: Our numerical experiments show three important messages: in label noise detection, i) training with noisy supervisions may not be necessary; ii) representation layers tend to be more useful than the logit layers; iii) representations from other tasks or domains are helpful.
guaranteed percentage of instances with
1.1 RELATED WORKS
Learning with noisy labels There are many other works that can detect corrupted instances (a.k.a. sample selection) in the literature, e.g., (Jiang et al., 2018 Han et al. 2018 Yu et al., 2019 Yao
al. 2020 Wei et al., 2020 Jiang et al. 2020; Zhang et al. 202 p. and combination with
semi supervised learning (Nguyen et al.) 2019 Li et al. |020b] Cheng et al. (021). Another line of
works focus on designing robust loss functions to mitigate the effect of label noise, such as numerical
methods (Ghosh et al. 2017, Zhang & Sabuncu 2018 Gong et al.] 2018 Amid et al., 2019 Wang
ctt al.
2019: Shu et al., 2020) and statistical methods (Natarajan et al. 2003 Liu & Tao 2015 Patrini
let al.
2017 Liu & Guo 2020; Xia et al. 2019 2021 Wei & Liu 2021 Zhu et al. 202 Li et al.
2021
Liu et al. 2020). All these methods require training DNNs with noisy supervisions.
Pre-training or Self-supervised Learning Training DNNs from scratch may be time-consumin
(Goyal et al. 2017). A common strategy to alleviate this issue is initializing DNN with a pre-trained model when training
new tasks (Krizhevsky et al. 2012 Zeiler & Fergus 2014 Sermanet al., 2013). The pre-trained model can be used for down-strean tasks due to the great transfer-ability of DNNs (Tan et al.) 2018). Popular tre-trained models include BERT (Devlin al.) 2018) for
language tasks or CLIP (Radford et al. 202T) for vision-language tasks. Recently, unsupervised
learning et al. 2019 and self-supervised learning (Jaiswal et al. 202 el. 2021 He al.
2020 Chen et al.
2020 also exhibit great power when tine-tuning DNN on down-stream tasks.
Label aggregation Our work is also relevant the literature of crowdsourcing that focuses on label
aggregation (to clean the labels) et al. 2012; Karger et al. 2011 2013: Liu & Liu 2015 Zhang
al. 2014). Most of these works can access multiple reports (labels) for the same input feature, while our real-world datasets usually have only one noisy label for each feature.
PRELIMINARIES
Instances Traditional classification problems with perfect supervisions build on clean dataset D = [(In Mm) (ecrnnn))eeetee JNe[N], where [N] := {1, 2, ... ..,,,,, Jj. Each clean instance (In, Bn) includes feature In and clean label yn. which is drawn according to random variables (X, Y) ~ D. In a more practical case, the clean labels are likely be unavailable and the learner could only observe a noisy
2

---

Under review as a conference paper at ICLR 2022
K-N Labels
Represemanicns
[0.82.0.38]
Vating Ranióng
[0.67, 0.33
[0.24, 0.766
Figure 1: Detect noisy labels with similar representations. Orange circle: instance with noisy label
1. Blue square: instance with noisy label 2. Green dashed circle: A k-NN example.
dataset denoted by D ; (Enn BnEIN where (In, Bn.) is a noisy instance and the noisy label Un may or may not be identical yn- We call Dn is corrupted if yn + yn and clean otherwise. The instance (In: Th.) is a corrupted instance if ih is corrupted. The noisy data distribution corresponds to D is Y) D. We focus on the closed-set label noise that Y and Y are assumed to be in the [K] for a K-class classification task. Explorations on open-set data 202T are deferred to future works.
same label space, e.g.. Y. Y
(Xia et al. 2020a Luo et al.,
Representations The representation of feature zn denoted by ā == g(z,). where g(-) denotes representation extractor. Feature T, could be in any shape and representation Fs is generally a high-dimensional vector. In this paper, we focus on setting where the distances between two representations should be comparable or clusterable (Zhu et al., 202Ib), i.e. nearby representations should belong to the same true class with a high probability (Gao et al. 2016), which could be defined as (ke, 5x) label clusterability in Definition!! [
Definition 1 ((k, 5&) label clusterability). Given a representation extractor g(-), dataset D sat- isfies (Ic, 5&) label clusterability if: Vn [N], the representation F, := g(z.) and its k-Nearest- Neighbors (k-NN) Em.. Inn belong to the same true class with probability at least 1 8y.
Note d2 captures two types of randomnesses: one comes from probabilistic Y|X, i.e., 3i E [K]. = x|X) & {0. 1]: the other depends on the quality of representation and the value of k. which will be further illustrated in Figure 2 The (k, 0) label clusterability is also known as k-NN label clusterability (Zhu et al., 2021b). It has been shown in (Zhu et al., 202Ib) that k-NN label clusterability can often be satistied for reasonable representation when k moderate small. We in Definition
define k perfect extractors/heprrsenttaie
Definition (k perfect extractoo/herresennatton))) representation extractor g(-) is called k perfect extractor if induces k-NN label clusterability. Its outputs are called k perfect representations.
Label noise detection Our paper aims improve the performance of the label noise detection (a.k.a, detecting noisy/corrupted labels, finding label errors) which measured by the Fi-score of the detected corrupted instances, which is the harmonie mean of the precision and recall, i.e. F, 2/PPrciiiion + Recall-1). Let I(-) be the indicator function that takes value when the specified condition is satisfied and 0 otherwise. Let On indicate that Vn is detected corrupted label, and = 0 if In is detected to be clean. Then the precision and recall can be calculated as Precision Σne(N) 1(v,,= fy..
.Recall ΣnElM) (0,,11.f.. Eneini 1 (un))))
EneIM [(v,=1)
LABEL NOISE DETECTION USING SIMILAR REPRESENTATIONS
Different from most methods that detect noisy labels based on the logit layer model predictions Northcutt et al. 2021a Cheng et al. |021] Pruthi et al. (020), we focus on representations of features. Particularly way, once given good representations. In this section, we will first introduce intuitions, and then provide two efficient algorithms detect label noise with similar representations.
are interested the possibility of detecting noisy labels in training-free
3.1 INTUITIONS
The training-based detection methods often make decisions by comparing model predictions with noisy labels (Cheng et al. 2021 Northcutt et al., 2021a). However, the representation g(x,) cannot be directly compared with the noisy label yn since (( is not directly categorical, i.e.. the output of a single g(x,)) rarely corresponds to any label class. Thus the first step of our training-fre solution should be establishing auxiliary categorical information using only representations.
3

---

Under review as a conference paper at ICLR 2022
As illustrated in Figure 1 the high-level intuition is to check label consensuses of nearby represen- tations. With &o perfect representations as defined in Definition 2 we know the true labels of En and its k--N En..... .n, should be the same. If the label noise across these instances are group- dependent (Wang et al. 202T). we can treat their noisy labels as k + independent observations of PY|| Zn. Y Vr), then estimate the probability by counting the frequency or weighted frequeney of each class in the k-NN label estimator, and get k-NN labels yn. The i-th element yn [1] can be interpreted as the estimated probability of predicting class-i.
Representations could be better than model predictions During supervised training, memorizing noisy labels makes the model generalizes poorly (Li et al.) 2020al Han et al. 2020), while using only representations may effectively avoid this issue. This is because representations often come from techniques beyond memorization such as unsupervised learning or self-supervised learning where the noisy labels are not referred in representation learning. On the other hand, the extractor g(-) has great potential in domain adaption transfer, which makes representations more flexible than the specific model predictions. We will demonstrate this point in Section
3.2 VOTING-BASED LOCAL DETECTION
Inspired by the idea implemented in model decisions, i.e., selecting the most likely class as the true class, we can simply "predict" the index that corresponds to the largest element in 9n with random ne-brraking, i.e., yn gote arg [[ In [i]. To further detect whether Un, is corrupted or not, we only need to check vn == I (y * ). Recall en = indicates a corrupted label. This voting method relies only the local information within each k-NN label yn. which may not be robust with imperfect representations. Intuitively, when the gap between the true class probability and the wrong class probability is small, the majority vote will be likely to make mistakes due sampling errors in Dn. Thus only using local information within each yn may not be sufficient. It is important to leverage more information such as some global statisties, which will be discussed later.
3.3 RANKING-BASED GLOBAL DETECTION
From a global perspective, if the likelihood for each instance being clean could be evaluated by some scoring functions, we can sort the scores in an increasing order and filter out the low- sow-score instances as corrupted ones. Based on this intuition, there are two critical components: the scoring function and the threshold to differentiate the low-score part (corrupted) and the high-score part (clean).
Scoring function A good scoring function should be able to give clean instances higher scores than
9_ e, corrupted instances. We adopt cosine similarity defined as: (Un, = la where ej is the one-hot encoding of label j. To evaluate whether the soft label Dn informs us a clean instance or not, we compare Score Ún: V.) with other instances that have the same noisy label. This scoring function captures more information than majority votes, which is summarized as follows.
Property (Relative score). Within the same instance, the score of the majority class is higher than the others, i.e. Score Gn, yy xyate)) > Score(ý,; f), Vj + "y j E [K], Vn E [N).
Property 2 (Absolute score). Sco is jointly determined by both i.l and Dnb + j.
The first property guarantees that the corrupted labels would have lower scores than clean labels for the same instance when the vote is correct. However, although solely relying on Property [ may work well in the voting-based based method which makes decisions individually for each instance, it is not sufficient to be trustworthy in the ranking- based global detection. The main reason is that, across different instances, the non-majority classes of some instances may have higher absolute scores than the majority classes of the other instances, which is especially true for general instance-dependent label noise with heterogeneous noise rates (Cheng et al. 2021). Property 2 helps make it less likely to happen. Consider an example as follows.
Example Suppose Un, = Ong [0.6, 0.4, 0.0] 9ng [0.34, 0.33, 0.33] Yay Ung Ura 1, Un Uns 1, Una = 2. We can use the majority vote to get perfect detection in this case, i.e., yoote = Vig Weste = yns vate = 1 = yn,, since the first class of each instance has the largest value.
However, if we directly use a single value in soft label yn to score them, e.g., Score' (Dn.
OOne can weight instances by the similarity the center instance.
4

---

Under review as a conference paper at ICLR 2022
Algorithm Detection with Similar Representations (The SimiRep Detector)
Input: Number of epochs: M. k--N parameter: k. Noisy dataset: D (ans In Fn) Ine
Representation extractor: 9(-). Method: Vote or Rank. Epoch counter m = 0.
2: repeat
# Initialize Stannaad data augmentations # Extract representations with g(-) 5: yn + KNNLabel n. (nN],, k) # Get soft labels based on the clusterability representations
3: in + RandPreProcess
Vn;
4: En < g(x²,), Vn;
6:
if Vote then
7:
Un + arg [ Un [i];
Apply local majority vote # Treat as coruutted if majority votes agree with noisy labels
8:
Un + 1y Note # Un), Vn E [M];
9:
else
10:
+ (( In) InE(N))
Estimate clean priors P(Y) and noise transitions P((IY) by the HOC estimator # Estimate thresholds by Bayes' rule
P(Y (11)==((Y[] =((PY) PPY))
11:
12:
for j in [K] do
13:
N3 = {nl Un =j}; j]:
# Detect corrupted labels in each set Ng # I. records the raw index of each sorted value
14:
T argsort (u:
U/ 1 (Loc(nn s [(1 P(Y j ; ()) N2,);
# Select low-score (head) instances corrupted ones
end for end if
17:
18:
Vm = [UNJnElNNi
Record detection results in the m-th epoch
19:
until M times
20:
V =Vooe(VV Vm E [M]];
# Do majority vote based on results from M epochs
21: Output: [N] V.
In [i], we will have Go, [ūn,]] = 0.6 > [7ng] 0.4 Ins [mg]] 0.33, where the ranking is my < n2 < n1- Ideally, we know instance n2 is corrupted and the true ranking should be n2 X ni < ni or n2 x n1 < n3- To mitigate this problem, we choose the cosine similarity as scoring function. The three instances could be scored as 0.83, 0.55, 0.59, corresponding to an ideal ranking n22 ng K n4. We formally introduce the detailed ranking approach as follows.
Ranking Suppose we have a group of instances with the same noisy class j. i.e. {(Im, To) Jae,, where N₁ [nlin j} are the set of indices that correspond to noisy class j. Let N, be the number of indices in Nj (counted from noisy labels). Intuitively, we can first sort all instances in N, in an increasing order by argsort and obtain the original indices for the sorted scores as: I= argsort: ( (Score(@r)))) JheEN,, where the low-score head is supposed to consist of corrupted instances Northcutt al. 2021a). Then we can simply select the first N, instances with low scores as corrupted instances: Ux 1 (Loc(n, I) s Nj), where Loc I) returns the index of n in I. Instead of manually tuning Nj. we discuss how determine it algorithmically.
Threshold The number corrupted instances in N2 is approximately IP(Y * = j) N₁ when N2 is sufficiently large. Therefore if all the corrupted instances have lower scores than any clean instance, we can set N, = -P(Y * j|V = j)- N, to obtain the ideal division. Note Nj can be obtained by directing counting the number of instances with noisy label j. To calculate the probability PPYY + j )
P(Y j|Ý² j), borrow the results from the HOC estimator (Zhu et al. 2021b), where the noise transition probability P(Ý = j|Y = j) and the marginal distribution of clean label IP(Y j) can be estimated with only representations and the corresponding noisy labels. Then we can calculated our needed probability by Bayes' rule P - j = j) = P(Ý j|Y = j) P.Y = -} = j), where P(Ý = j) can be estimated by counting the frequency of noisy label, in D. Technically other methods exist in the literature to estimate Y() (Lu & Tao 2015 Patrini et al. 2017 Northeutt et al. 2021a: Li et al. 2021 But they often require training a model to fit the data distribution, which conflict with our goal of a fraening-free solution; instead, HOC fits us perfectly.
5

---

Under review as a conference paper at ICLR 2022
3.4 ALGORITHM: DETECTION WITH SIMILAR REPRESENTATION (SIMIREP)
Algorithm [ summarizes our solution. The main computation complexity is obtaining representa- tions with extractor g()), which is less than the cost of evaluating the model compared with the training-based methods. Thus SimiRep can filter out corrupted instances efficiently. In Algo- rithm 1 we run either voting-based local detection as Lines 7 or ranking- banking-based global detection The detection is run multiple times with random standard data augmentations to reduce the variance of estimation. The majority of results from different epochs adopted as the final detection output as Line 20 i.e., flag as corrupted if en = in more than half of the epochs.
as Lines 14 T5
4 HOW Do REPRESENTATIONS AFFECT OUR SOLUTION?
In this section, we will first show how good but imperfect representation affects the selection of the hyperparameter k. We then offer an analysis of the error upper bound for the ranking- based method given representation extractor g()).
4.1 How Do REPRESENTATION AFFECT THE CHOICE OF k?
Recall & is used as illustrated in Figure | On one hand, the k-NN label estimator will be more accurate if there is stronger clusterability that more neighbor representations belong the same true class (Liu & Liu 2015) Zhu et al. 20211), which helps improve the performance of later algorithms. On the other hand, with good but imperfect representations, stronger clusterability with larger k is less likely to satisfy, thus the violation probability 6ş increases with l for given extractor g(-). We take the voting- based method an example and analyze this tradeoff. For a clean presentation, we focus on binary classification with instance-dependen label noise where P(Y 1) = p,
P(Ý- 2| 1) e1 (X), P(Ý 1|X,Y = 2) (x)) Suppose the ristance-dependee noise rate is upper-bounded by e, i.e., (X) S ex) < e. With dg as in Definition | we
calculate the lower bound of the probability that the vote is correct in Proposition
Proposition 1. The lower bound for the probability of getting true detection with mujority vote is
P(Vote is correct k) 2 (1 - 6x) Te-e (k + 1- ---KKK ,,K + 1),
where K' = [[k: + 1)/2 1, 1 1-e, (k +1 k kv' + +1 the regularized incomplete beta function defined as 11 (k- + 1- K², + 1) = (A: + k²) (*xz So gk p ( t) dt.
Proposition [ shows the tradeoff between a reliable k-NN label and an accurate vote. When k is increasing, Term-1 (1 5u) (quality of representations) decreases but Term-2 Ti (k + 1 k", k' 1) (result of pure majority vote) increases. With Proposition
[
we are ready
to answer the question: when do need more labels? See Remark
Remark 1. Consider the lower bounds with kn and k2 (k) < k2). Supposing the first lower bound lower than the second lower bound, based on Proposition ! increasing k by compuring two bounds and gel
we roughly study the trend with an
1 - 6bz
11 T1-c(kz + M2. 1/g + 1)
<
1 - 8k,
11 I1-c(kı + 1--1-ki k4, K + +1
For example, when kn = 5, 12 = 20, e 0.4, we can calculate the incomplete beta function and 1-62. 1.52. Supposing 85 = 0.2, we have ò20 0..47 This indicates increasing from 5 to 20 I-
would not improve the lower bound with imperfect representations with ô20 0.47. This observa- tion helps us set lc with practical and imperfect g(-). We set k = 10 in all of our experiments.
Remark Mindicates that: with imperfect representations, a small may achieve the best (highest) probability lower bound. To further consolidate this claim, we numerically calculate ó, with differ- ent g(-) and the corresponding probability lower bound in Figure 2 We find most of the probability lower bounds first increase then decrease except for the perfect g()) which is trained using ground- truth labels. Note the perfect g()) has memorized all clean instances so that 5z + 0 since k < 5000 (the number of instances in the same label class).
Note the voting-based method achieves Fi-score of given perfect representations, k +00.

---

Under review as a conference paper at ICLR 2022
0.8
0.1
R34
R18mmg
0.6
Lowedd bound
R34-C1D-SSL
R504mng
0.4
0.4
VIT CUP
134 C100-5SL-
R3K-CID-CWan
0.2
R34-C100-Clean
0.0
0.0
Figure 2: The trends of 0g and probability lower bounds on CIFAR- 10 (Krizhessyy et al. 2009) with different g(-). The outputs of the last convolution layer adopted. R18/34/50: ResNet ResNeel8734/50. Img: Pre-trained on ImageNet (Deng et al. 2009). C10/100-Ceann: Pre-trained on clean CIFAR- 10/100. C10/100-SSI. Pre-trained on CIFAR-TO7TO0 without labels by SimCLR (Chen al. 2020
ViT-B/32-CLIP: CLIP Pre-trained vision transformer (Radford et al. 2021)
4.2
How Do REPRESENTATION AFFECT Fi-SCORE?
We next prove the probability bound for the performance of the ranking-based method. Consider K-class classification problem with informative instance-dependent label noise (Cheng al. 202 Denote random variable S by the score of each instance being clean. A higher score indicates the instance is more likely to be clean. Then for instances in N₁, we have two set of random variables Sqme = {Suue/ E N, In yn and Sfabe = (Stabeej I N₁, Un * yo Intuitively, the score Sque should be greater than Stakse Suppose their means, which depend on noise rates, are bounded, i.c., B[symm] 2 11 E[Stfabe < 1 falee. Assume there exists feasible v such that both Stme and Stalse follow sub-Gaussian distributions with variance proxy A et al. 2021c) such that: P(u STme 2) S (()/AA) 20
[Buldygin & Kozachenko 1980 Zhu
and PP 1 false 2 (
where 1/A is the "height" of both distributions, i.e., P( = n P(S = 11g ( 1/A
is the decay rate of tails. Let N (Nj+) be the number of indices in Stalse (Stme)) Theorem summarizes the performance bound of the ranking- ranking-based method. See Appendix for the proof.
Theorem 1. With probability at least p. the Fi-score of detecting corrupted instances in N₂ with the N
rank-based method by threshold N2IP(Y = -|ا = -) is at least
max(N "=xxxx"NNN )N+ where p =
1 - A f(t)dt, f(t) is the probability density function of the difference of two independent beta distributed random variables 31 - 32, where 31 Beta(N 1), 32 Beta(a + 1, a).
Theorem shows the performance of detection depends on 1) the concentration of Sy and stals (denoted by variance proxy - 2) the distance between Stme and Stale (denoted by n μ fale Intuitively, with proper scoring function and good representations, we have small variance proxy (small A and large v) and Fi-score approximates to 1.
5 EMPIRICAL RESULTS
We present experimental evidence in this section. The performance is measured by the Fi-score of the detected corrupted labels as defined in Section 2 Note there is no any training procedure in method. The only hyperparameters in methods parameter k. Intuitively, larger M returns a collective result from more times of detection, which should be more accurate. The hyperparameter & cannot be set too large as demonstrated in Figure In CIFAR Krizhevsky et al. 2009) experiments, rather than fine-tune M and & for different settings, fix M 21 and ls 10. We also test on Clothing IM (Xiao al. 2015). Detailed experiment settings on Clothing M are in Appendix C
the number of epochs M and the k-NN
Synthetic label noise We experiment with three popular synthetic label noise models: the sym- metric label noise, the asymmetric label noise, and the instance-dependee
label noise. Denote the ratio of instances with corrupted labels in the whole dataset by 7). Both the symmetric and the asymmetric noise models follow the class-dependent assumption (Liu & Tao 2015), i.e., the label noise only depends only the clean class: X. Y) P(Ý Y). Specially, the symmetric noise is generated by uniform flipping, i.e., randomly flipping a true label to the other possible classes w.p. 7) (Cheng et al. 202T). The asymmetric noise is generated by pair-wise flipping, i.e., randomly

---

Under review as a conference paper at ICLR 2022
Table I: Comparisons of Fi-scores (%). CORES, CL. TracIn: Train with noisy supervisions. SimiRep-V V and SimiRep-R: Get g(.) without any supervision. Top 2 are bold.
Method
CIFARIO
CHEARTOD
Homsoe
Symou
Arym,
Iust
Human
Symum.
Asyan
Iost. 0.4
CDRES
65.00
87.43
3.52
0.012
Ca.
55.85
80,59
7645
62.89
78.98
52.96
50.08
Trxcln
55.02
76.94
73.47
58,85
61.75
76.74
48.42
49.89
SimiRep-V V
82.30
93.21
82.52
81.09
73.19
84.48
65.42
74.26
SimiRop-8
83,28
95.56
83.58
82.26
74.67
62,89
73.53
Table 2: Comparisons of Fi-scores (%). CORES, CL, TracIn: Use logit layers. SimiRep- V/R: Use only representations. All methods use the same extractor from CLIP. Top 2 are bold.
Method
Homan
Symou.
CIARIO
Arym, 0.3
Just
Human
Sywm.
CIEARGO
Asyon
Jnst. 0.4
Sieve CORES
67:21
14:36
K.41
16.24
58.55
83.18
96.94
12.05
88.89
38.52
92.33
7,02
N5.52
Cl.
69.76
95.11
77.14
62.91
67.64
85.67
62.58
61.53
Trxcin
$1.85
95.96
80.75
64,97
79.32
91,03
63.12
64..31
SimiRep-V
SimiRop-8
87.45
96.74
91.14
79.21
90.54
68.14
77.37
flipping true label i to the next class (i mod K) + 1. Denote by d the dimension of features. The instance-deenndent label noise is synthesized by randomly generating a d K projection matrix w for each class and project each incoming feature with true class y/, onto each column of Wyn (iaa eet al. 2020b). Instance n is more likely to be flipped to class j if the projection value of Fy, on the j-th column of Wy. is high. See Appendix B in Xia et al. (2020b) and Appendix D.1 in /Zhu et al. (2021b) for more details. We use symmetric noise with 17 0.6 (Symm. 0.6), asymmetric noise with 1 0.3 (Asym. 0.3), and instance-dependent noise with 7 = 0.4 (Inst. 0.4) in experiments.
Real-world label noise The real-world label noise comes from human annotations or weakly la- beled web data. We use the 50, 000 noisy training labels (17 R 0.16) for CIFAR-I0 collected by Zhu itt al. Ib2) and our self-collected 50, 000 noisy training labels (17 A 0,40) for CIFAR- 100AR-100 (data will be released in the non-anonymous version). Both sets of noisy labels are crowd-sourced from Amazon Mechanical Turk. For Clothing IM (Xiao et al. 2015), we could not calculate the Fi-scores due to the lack of ground -trut labels. We firstly perform noise detection on I million noisy training instances then train only with the selected clean data to check the effectiveness.
5.1 SUPERVISION MAY NOT BE NECESSARY
Our first experiment aims to show that training with noisy supervisions may not be necessary in detecting corrupted labels. To this end, we compare our methods, i.e., voting-based local detection global detection (SimiRep-R), with three recent noise detection works: CORES (Cheng et al., (02T). confident learning (CI.) (Northeutt et al., 2021a), and TracIn (Pruthi et al.] 2020). We use ResNer34 as the backbone network in this experiment.
(SimiRep-V) V) and ranking-basce
Baseline settings All these three baselines require training model with the noisy supervision. Specifically, CORES (Cheng al. 202I) trains ResNet34 on the noisy dataset and uses its proposed sample sieve to filter out the corrupted instances. We adopt its default setting during training and calculate the Fi-score of the sieved out corrupted instances. Confident learning (CL) (Northeutt ctt al. 202 detects corrupted labels by firstly estimating probabilistic thresholds to characterize label noise, ranking instances based on model predictions, then filtering out corrupted instances based on ranking and thresholds. We adopt its default harem-prr setting to train ResNet34. TracIn (Pruthi al. (2020) detects corrupted labels by evaluating the self-influence of each instance, where the corrupted instances tend to have high influence score. The influence scores are calculated based on gradients of the last layer of ResNet34 at epoch 40, 50, 60, 100, where the model is trained with a batch size of 128. The initial learning rate is 0.1 and decays to 0.01 at epoch 50. Note TracIn only provides ranking for instances. To exactly detect corrupted instances, thresholds are required. For a fair comparison, we refer to the thresholds learned by confident learning (Northcutt) et al., 2021a). Thus the corrupted instances selected by TracIn are based on the ranking from its self- influence and thresholds from CL. To highlight that solutions work well without any supervision, our representation extractor g(-) comes from the ResNet34 pre-trained by SimCLR (Chen et al, 2020) where contrastive learning is applied and no supervision is required. Extractor g(-) is obtained with only in-distribution features, e.g., for experiments with CIFAR-10, g(-) is pre-trained with features only from CIFAR-10.
Performance Table Dallpareerrre the results obtained with or without supervisions. We can see both the voting-based and the ranking- method achieve overall higher Fi-scores compared with the
8

---

Under review as a conference paper at ICLR 2022
Table 4: Experiments on Clothing M. None: Standard training with 1M noisy data. R50-Img (or VTT-B/32-CLIP, R50-Img Warmup-1); D): Apply our method with ResNet50 pre-trained on ImageNet (or ViT-B/32 pre-trained by CLIP, R50-Ing with 1-epoch warmup). Top-1 is bold.
Data Selection
Training Samples
Best Epoch
Last 10 Epochs
Last Epoch
None (Standard Baseline)
IM (1006)
70.32
09:44 = 0.13
69.53
R50-Ing
770k (77.0%)
72.37
71.95 4: 0.08
71,89
NIT-B/32-CLA
700k (70.0%)
72.51
72.23 4: 0.17
72.11
150 Warmup-1
(76.7%)
73.64
0.18
73.41
other three results that require learning with noisy supervisions. Moreover, in detecting the real- world human-level noisy labels, our solution outperforms baselines around 20% on CIFAR-10 and 10% on CIFAR- -0F, which indicates the training-free solution are more robust to complicated noise patterns. One might also note that CORES achieves exceptionally low Pi-scores on CIFAR-10/100 with asymmetric noise and CIFAR-100 with human noise. This observation also informs us that customized training processes might not always be universally applicable.
5.2 REPRESENTATION LAYERS MAY PERFORM BETTER THAN LOGIT LAYERS
Table 2 shows the detection using either representation layers or logit layers of DNNs. In addition to the baselines compared in Section [. we also compare CE Sieve (Cheng al. 2021) which follows the same sieving process as CORES but uses CE loss without regularizer. All methods adopt ViT-B/32 pre-trained by CLIP (Raford et al. 202T). Specifically, the training-based methods further fine-tune a linear logit layer on noisy supervisions, while our solution directly sets this pre- trained model as g()). Other settings are the same as those in Section 5.1 By counting the frequency of reaching top-2 Pi-cccrss, we find SimiRep-R wins 1st place, SimiRep-V V and CORES are tied for 2nd place. However, similar to Table 2. we find the training process of CORES to be unstable. For instance, it almost fails for C10AR-100 with asymmetric noise. It is therefore reasonable to believe both methods with only representations achieve an overall higher Fi-score than other methods with logit layers. In other words, compared with our methods, the extra fine-tuning of logit layers with noisy supervisions cannot always help improve the performance of detecting corrupted labels.
5.3 WHAT ARE GOOD REPRESENTATIONS
Previous experiments demonstrate methods overall outperform baselines given representations from contrastive pre-training- It is interesting to see how other repre- sentations perform. We summarize results of SimiRep-R in Table 3 There are several interesting find- ings: 1) The ImageNet pre-trained models perform well, indicating the Iraditional supervised training out-of-distribution data helps; 2) For CIFAR-100, extractor g(-) obtained with only features from CIFAR-10 (R34-C10-SSL) performs better than the extractor with clean CIFAR- 10 (R34-c10-Clean,, indicating that contrastive pre-training has better general- ization ability to cuttt--distributionn data than supervised learning; 3) The Fi-scores achieved by g(-) trained with the corresponding clean dataset are close to 1, indicating our solution can give perfect detection with ideal representations. Besides, we test the performance of training only with the clean instances selected by our approach in Table 4 Standard training with Cross-Entropy loss is adopted. The only difference between the first row and other rows of Table that some training instances are filtered out by our approach. Table 4 shows simply filtering out corrupted instances based on our approach distinctively outperforms the baseline. We also observe that slightly tuning g(-) the fine-grained Clothing M dataset would be helpful, See more details in Appendix
Table 3: Comparisons of Fi -ccores (%) using g()) with differ- ent O₂ (%). Model names are the same as Figure 2
CIAATO Human
CIPARTOU
Pre-trained Model
Inst. 0.4
-
Humss
0.4
RTB-Img
22
11.30
74.91
71.99
R34-Img
48.13
79.52
$2.43
16.17
76.88
74,00
R50-Img
45.77
78.10
$2.06
15.81
76.55
73,51
WTBB-2-CLIP
64.12
87.45
91.14
19.94
79.21
77.37
R34-C10-8SI.
69.31
2.59
68.03
65.94
R3LC1D-Clemn
99.41
98.39
98.59
0.22
60.90
60.75
R34-C100-SSI.
18.59
59.96
74.99
22/46
74.67
73.53
R31-C100-Clan
60.17
76.41
89.07
92.87
95.29
6 CONCLUSIONS
This paper proposed a new and universally applicable training-free -free solution to detect noisy labels by using the neighborhood information defined by good set of representations. We have also demon- strated that good representations are reasonable assumptions with pre- pre-training or self-supervised learning. Future works will explore other tasks that could benefit from label cleaning.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Vibhu Agarwal, Tanya Podchiyska, Juan M Banda, Veena Goel, Tiffany Leung, Evan P Minty, Timothy Sweeney, Elsie Gyang, and Nigam Shah. Learning statistical models of phenotypes using noisy labeled training data. Journal of the American Medical Informaties Association, 23
19):116-1173, 2016.
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. In Advances in Neural Information Processing Systems, pp- 14987-14996, 2019.
Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Confer- on Machine Learning, pp. 540-550. PMLR, 2020.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint
1:X.::9905.02249 2019.
Valerii V Buldygin and V Kozachenko. Sub-gaussian random variables. Ukrainian Mathematical Journal, 32(6):483 489, 1980.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Rep- resentations, 2021. URL https: /ttp::::////eeereeeeeeeeeee
net/ forum??d-22VXyy2m
J. Deng. W. Dong. R. Socher, L.-. Li. K. Li, and L. Fei-Fei. ImageNet: A Lcrge-Scale Hicrarchical Image Database. In CVPRO9, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810008800 2018.
Wei Gao, Bin-Bin Yang, and Zhi- Zhi-Hua/ae Zhou, On the resistance of nearest neighbor to random noisy 2016
labels. arXiv preprint 1r0iv:1607.000
Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2703-2708, 2021.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Maoguo Gong, Hao Li, Deyu Meng, Qiguang Miao, and Jia Liu. Decomposition- based evolutionary multiobjective optimization to self-paced learning. IEEE Transactions on Evolutionary Compu- tation, (3)2):288 302, 2018.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 706.02677, 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:201 I.04406, 2020.
Kaiming He, Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729- 2020.
10

---

Under review as a conference paper at ICLR 2022
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2021.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9865-9874, 2019.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven eurriculum for very deep neural networks corrupted labels. In International Conference on Machine Learning, pp. 2304-234 2313. PMLR, 2018.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on con- trolled noisy labels. In International Conference Machine Learning, pp. 4804-4815. PMLR, 2020.
David R Karger, Sewoong Oh, and Devavrat Shah. Efficient crowdsourcing for multi-class label- ing. In Proceedings of the ACM SIGMETRICS/mmematiooni conference Measurement and modeling of computer systems, pp. 81-92, 2013.
D.R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In Neural Information Processing Systems, NIPS 11, 2011.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 15: 2012.
Jingling Li, Mozhi Zhang, Keyulu Xu, John Dickerson, and Jimmy Ba. Noisy labels can induce good representations. arXiv preprini arXiv:2012.1 200002..2288 /2896, 2020a.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020b. URL
https::/oppereviier net http:/oommmmdidueeaaaaae
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label- noise learning without anchor points. arXiv preprint 1rXiv:2102.02444 2021.
Andy T Liu, Shang- Wen Li, and Hung-yi yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. JEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:2351-2366, 2021.
Qiang Liu, Jian Peng, and Alexander Ihler. Variational inference for crowdsourcing, In Proceedings of the 25th International Conference Neural Information Processing Kestems-Volume 1, pp- 692-700, 2012.
Sheng Liu, Jonathan Wiles-Weed. Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. arXiv preprint arXiv:2007.00151, 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE
Transactions on pattern analysis and machine intelligence, 38(3):447 2015.
Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. Interna-
lional Conference on Machine Learning, pp. 6725-6735. PMLR, 2021.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In Proceedings of the 37th International Conference on Machine Learning, ICML '20, 2020.
Yang Liu and Mingyan Liu. An online learning approach improving the quality of crowd- sourcing. ACM SIGMETRICS Performance Evaluation Review, 43(1):217-230, 2015.
Huixiang Luo, Hao Cheng, Yuting Gao, Ke Li, Mengdan Zhang, Fanxu Meng, Xiaowei Guo, Feiyue Huang, and Xing Sun. On the consistency training for open-set semi-supervised learning, 2021.
11

---

Under review as a conference paper at ICLR 2022
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-
ensembling. arXiv preprint 1910:01910 2019.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 1111333311111 2021a.
Curtis Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2/03. 14749, 2021b.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the JEEE Conference on Computer Vision and Pattern Recognition, pp. 1 1944-1952, 2017.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pp. 19920-19930, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0080 2021.
Pierre Sermanet, David Eigen. Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXivv1312.620 2013.
Jun Shu, Qian Zhao, Keyu Chen, Zongben Xu, and Deyu Meng. Learning adaptive loss for robust learning with noisy labels. arXiv preprint arXiv:2002.0644 2020.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018.
Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. FAccT, pp. 526-536, New York, NY, USA, 2021.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross en- tropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vision, pp. 322-330, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13726-13735. 2020.
Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise. International Conference on Learning Representations, 2021.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Advances in Neural Information Processing Systems, pp. 6838-6849 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, and Yinian Mao. Extended T: Learning with mixed closed-set and open-set noisy labels. arXiv preprint 2rXi::2012200933. 2020a.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instanne-dependent label noise. In Advances in Neural Information Processing Systems, volume 33, pp. 7597-7610, 2020b.
12

---

Under review as a conference paper at ICLR 2022
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang- Robust larly-leamning Hindering the memorization noisy labels. In International Conference Learning Representations, 2021.
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification.. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2691-2699 2015.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh- Thah-Thang Luong, and Quoc Le. Unsupervised data augmentation. arXiv preprint urxiv:19044 12848, 2019.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James T Kwok. Searching exploit memo- rization effect in learning with noisy labels. In Proceedings of the 37th International Conference Machine Learning, ICML '20, 2020.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 7164-7173. PMLR, 09-15 Jun 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818 9033333.55 Springer, 2014.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In International Conference on Learning Representations, 2018. URL
https::/cpenrev/ net forum?id=r11ddpl
Jing Zhang, Victor S Sheng, Tao Li, and Xindong Wu. Improving crowdsourced label quality using noise correction. IEEE transactions on neural networks and learning systems, 29(5):1675-1688 2017.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. arXiv preprint arXiv:2103.07756, 2021.
Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael Jordan. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. Advances in neural information processing sys- tems, 27: 27:1260-12666 1268, 2014.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 777888778, 2018.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021a.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as alternatiee anchor points when learning with noisy labels. In Proceedings of the 38th International Conference on Machine Learning. ICML 22, 2021b.
Zhaowei Zhu, Jingxuan Zhu, Liu, and Yang Liu. Federated bandit: gossiping approach. In Abstract Proceedings of the 2021 ACM SICMMrrCCSSheeeraiiinnd Conference on Measurement and Modeling of Computer Systems, pp. 3-4, 2021c
13

---

Under review as a conference paper at ICLR 2022
The omitted proofs are provided as follows.
A THEORETICAL ANALYSES
A.1 PROOF FOR PROPOSITION I
Now we derive lower bound for the probability of getting true detection with majority vote:
T(6+12/21-
h 1
F(Vote is correct|k) 2(1 02))
P
Σ
ef(1 (1 e1)
1(++1)/21-- -1
k +1
+ (1 -p)
Σ
e (1 - -2y)
=(1 6k)) Ii-ei (k +1+1- k + p) (k+ K. +1]
where 7 1-a (k:
k,, ; k' 1) is the regularized incomplete beta function defined
[k+1] k'
11-0(k + 1 K' k' + 1) = (k: + 1 x3)
* (1- t) b)hdt dt.
and k/ = =[(k- + 1)/2] 1.
B PROOF FOR THEOREM
Proof. Now we derive the worst-case error bound. We first repeat the notations defined in Sec- tion a.2 follows.
Denote random variable S by the score of each instance being clean. A higher score indicates the instance is more likely to be clean. Then for instances in N₁. we have two set of random variables Sqme {Suuee In E N2 In ya and Stals {Stabe nn Nj, Un * yn Intuitively, the score S should greater than Stalse Suppose their means, which depend on noise rates, are bounded, i.e..
false Assume there exists feasible v such that both S and Stake E[ 2 (1) B[[Stabee < 1
follow sub-Gaussian distributions with variance proxy 20 (Buldygin & Kozachenko 1980 7hu al. 2021c) such that: P(u - STJe 2 S "*t/AA)² and PC 11
fale 2 t) S Al)
where I/A is the "height" of both distributions, i.e., PP(STue = p1)0)) = P(Sfalse = 11 =
1/A, is the decay rate of tails. Let N, (N be the number of indices in Stakee (Sy For case of notations, we omit the subscript j in this proof since the detection is perfórmed on each j individually.
where Stake is the small- est order statistic and Staale is the largest order statistic. The following lemma motivates the perfor- mance of the rank-based method.
Denote the order statistics of random variables in Stalse by (1) . . (N-
Stibe Stalse
Lemma 1. The Fi-score of detecting corrupted labels in N2 by the rank-hased method will be no less than a/N when the true probability P(Y = -| = =) is known and Sfake Stree (a+1y)
Lemma D connects the upper bound for the number of wrongly detected corrupted instances with order statistics. There are two cases that can cause detection errors: Case-1:
0 < thee Strre A and <
gralse
false A at most a errors when Stalee (N-)
Strue (a+
and Case-2:
true
Sirue A or glabse Laleo > at most max(N-, N4) errors
We analyze each follows.
14

---

Under review as a conference paper at ICLR 2022
Case-1:
When Case-1 holds, have
P( Suru = [0,
and
P( - =) = 1 [0,
The above two inequalities show that the left tail of 5uw and the right tail of Stalso can be up- per bounded by uniform distributions. Denote the corresponding uniform distribution by Urtue Unif(w A, m and (fake Unif(ulale fale fae A).
With true P(Y = j|Y j), the detection emrors only exist in the cases when the left tail of S and the right tail of Stalse are overlapped. When the tails are upper bounded by uniform distributions, we have
((
(N-) < Strue
<S(att)
2 ((U
U (a+11)
(N-)
=P
(1rtiiee
(N-) + jule
[[tinve
(11 - A)] (art1) + (y - 4))
=P (Lufa
11 false
(N-) U
(He A))(ot)) < uueee 1 false - A)
Note
[Uralse [abo]]
(N BeBtaaaa,
and
T (4 A)] (a+1) Beta + 1, aa),
where Beta denotes the Beta distribution. Both variables are independent. Thus the PDF of the difference is
B(N n. 11p=~-00
F(I N + -
0 0.6 SSP
r(p)
BLN N o)(-p) 1 -11+p) +N -a-1--t--------
N N - 1
-1 ppp0
BIN- +a, N+ aalAA
p=
where A B(N 1)B(( + 1, + a), B(a, [0 - (a-1(11 1 dt
F(a, b2g r(a)l((c- a) ('' ( a) zt) (1 yt) yt) dt.
r(c)
Therefore, we have
<((att)))
(p))
Case-2 The other part, we have no more than e max max(N- (N-, _N+) corrupted instances that may have higher scores than one clean instance.
Wrap-up From the above analyses, we know, w.p. at least 1" "h -h f(p) there are most e max(N- N+) + a errors in detection corrupted instances. Note Precision Recall we
detect with the best threshold N2P(Y j|Ý j). Therefore, the corresponding F -coore would be at least 1 em* max(N ++
N
C EXPERIMENT SETTINGS ON CLOTHINGIM
We firstly perform noise detection 1 million noisy training instances then train only with the selected clean data to check the effectiveness. Particularly, in each epoch the noisy detection, we batch size of 32 and sample 1,000 miti-batches from 1M training instances while ensuring the (noisy) labels are balanced. We repeat noisy detection for 600 epochs to ensure a full coverage million training instances. Parameter & is set to 10.
15

---

Under review as a conference paper at ICLR 2022
(Xiao et al., 2015) with or without balanced sampling. None: Standard training with 1M noisy data. R50-Img (or VT--//2-CLIP, R50-Img Warmup-11 1): Apply our method with ResNet50 pre-trained on ImageNet (or ViT-B/32 pre-trained by CLIP, R50- Img with 1-epoch warmup).
Table 5:
Experiments on Clothing IM
Training Samples
Epoch
Epochs
Epoch
None (Siandard Baseline) (Unbalanced)
(100%)
70.32
09:44
0.13
69.53
None (Standued Baseline) (Ralenced)
(1015)
R50-Img (Unbalmced)
220k: (77.0%)
72.37
71. 11.95-
+ 0.088
71.89
R50-Img (Balanced)
770k: (27.0%))
72.42
72.06
0.16
72.24
VIT-8/32-411 (Unhalancst)
700K (70.05)
VIT-B/3-CCLIP (Balanced)
200k (70.05))
72.91
R50-Ing Warmup-1 (Unbalancd))
767K (76.75)
73.64
73.28
1: 0.18
73.41
850-ing Warmup-1
(Ralaneed)
767k (76.75))
73.97
73.37
1.133
73.35
Representation Extractor: We tested three different representation extractors in Table 4 R50- Img, ViT-B/32-CLIP, and R50-Img Warmup-1. The former two representation extractors are the same the ones used in Table 3 Particularly, R50-Img means the representation extractor is the standard ResNet50 encoder (removing the last linear layer) pre-trained on ImageNet (Deng et al. 2009). ViT-B/32-CLIP indicates the representation extractor a vision transformer pre-trained by CLIP (Radford et al. 202T). Noting that Clothing M is fine-grained dataset. To get better domain-specific fine-grained visual representations, we slightly train the ResNet50 pre-trained with ImageNet for one epoch, i.e., 1,000 ing instances while ensuring the (noisy) labels are balanced. The learning rate 0.002.
(batch size 32) randomly sampled from IM train-
Training with the selected clean instances: Given the selected clean instances from ap- proach, we directly apply the Cross-Entropy loss to train a ResNet50 initialized by standard Im- ageNet pre-trained parameters. We did not apply any sophisticated training techniques, e.g., mixup (Zhang et al. (2018). dual networks (Li et al. [2020b Han et al. 2018). loss-correction (Liu & Tao,
2015 Natarajan et al. 2013 Patrini al. [2017] 7, and robust loss functions (Liu & Guo 2020 Cheng
et al. 2021 Zhu et al. 202 Wei & Liu (021). We train the model for SO epochs with batch
size 32. We sample e 000 mini-batches per epoch randomly selected from 1M training instances.
4 does not apply balanced sampling. Only the pure cross-entropy loss is applied. We also test the performance with balanced training, i.e., in each epoch, ensure the noisy labels from each class are balanced. Our approach can be consistently benefited by balanced training, and achieves an accuracy of 73.97 in the best epoch, outperforming many baselines such as HOC 73.39% (Zhu ct al. 2021b), GCE+SimCLR 73.35% (Ghosh & Lan 2021). CORES 73.24%% (Cheng et al. 2021 GCE 69.75% (Zhang & Sabuncu 2018). We believe the performance could be further improved by using sophisticated training techniques mentioned above.
Note Table
16


Reviews:
[{"review_id": "sQgeNuDP8KB", "reviewer": "Reviewer_39dB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: the paper proposes a new perspective to detect noisy labels\nWeaknesses: my main concern is on the good representation assumption. To be more specific,\n\n1. Though it is possible to obtain some good representations via pre-training on other datasets or self-supervised learning on the noisy datasets, it is still unclear how the obtained representations could be guaranteed to help with noisy label detection. It would be helpful to include experiments evaluating the goodness of representations obtained from various approaches based on the proposed criteria (e.g., Definition 1 and 2).\n\n2. In Table 1 and 2, as the proposed method uses representations pre-trained by SimCLR, a self-supervised learning method, it seems unfair to compare with other noise detection methods, which do not have access to the good representations from SSL.\n\n3. The paper claims the proposed method as training-free, which may be misleading and a bit overclaim since the good representation assumed by the proposed method still requires learning. Supervision-free maybe a better choice.", "summary_of_the_paper": "This paper proposes a method to detect noisy labels given good representations (e.g., ones pre-trained by contrastive learning approaches). \nThe proposed noisy-label detection method uses  the neighborhood information defined by a good set of representations in two ways: \n\n1) checks the noisy label consensuses of nearby representations\n\n2) scores each instance by its likelihood of being clean and filters out a guaranteed percentage of instances with low scores as corrupted ones \n\nThe work provide definitions for good representation and further proves a worst-case error bound for the ranking-based method given a 'good enough' representation. It also provides empirical results for its proposed method. ", "summary_of_the_review": "This work proposes an interesting perspective to analyze how the representations could help with noise detection. Yet, the major assumption of this work that a good representation is given still needs further justifications. Additional experimental results are also needed to fully support the claim.\n\n===Update after rebuttal===\nThanks to the authors' response, which resolved some of my confusions. I am increasing my score for the additional discussions and results", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tg6Hmtmx4tJ", "reviewer": "Reviewer_1co7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well organized and easy to follow. The intuition of the paper, which lies in utilizing proper representation instead of training a deep inference model to avoid memorizing corrupted data, also sounds somewhat interesting and seem to work. Besises, the proposed method can detect label noise in the instancewise manner. However, my concerns mainly lie in the methodology and the experiment.\n\nFirst, what is the specific definition and criterion of good representation emphasized in this paper? It seems that the prerequisite of the proposed method is “given good representation”. The authors have done some basic tests in section 5.3 (table 3) that indicates the representations lead to accurate inference result is a good representation. However, it is the post-inference analysis, is there any way to select a good representation extractor g(·) in advance? Besides, the dependence on good representation may become the bottleneck of the proposed method since visual representation itself is an important and challenging topic.\n \nSecond, this paper takes the corrupted CIFAR-10 and CIFAR-100 (corrupted by automatic labeling tool) as the real-world noisy dataset for experiments, which is not very convincing. The acknowledged noisy-label dataset such as Webvision and Cothing1M could be more helpful in verifying the effectiveness and universality of the proposed method. Though these datasets may not involve the ground truth labels, there are still many ways to achieve the quantitative comparison test. For example, you can filter the wrong labels detected by the proposed method, and compare the overall inference accuracy between the proposed method and the SOTAs. \n\nIn addition, though the authors claim that their method is universal, they adopt clustering methods like k-NN to generate the soft labels and resolve the discrimination task. I thus wonder if the increasing number of object categories may result in a significant drop in noise detection accuracy. That is also one of the reasons why I wonder about the convincingness of the datasets used in this paper.\n\nThere are also some minor concerns：\n1. The related work is too brief to reflect the association and difference between the proposed method and the existing methods.\n2. The ambiguous meaning of words. Eg, why do the authors call the representation extractor g(·)  k perfect extractor if it induces k-NN label clusterability in Definition 2? Why perfect?  \n3. There are some writing issues to fix, the authors may need to proofread the papers more carefully. eg. Property 1, \"With in\"  in the sentence \" With in the same instance, bla bla\".", "summary_of_the_paper": "The paper proposes a training-free instancewise noise label detection method. The main motivation of this paper is the observation that deep models generalize poorly because memorizing noisy labels in supervised training while using only representations may avoid this issue.  The authors suppose a good representation extractor is given and generate the initial soft labels based on the clusterability of representations using kNN. Followed by that,  they perform a local voting and global ranking-based scoring system to detect the corrupted labels. The main contribution of this paper is: It introduces a representation-based method instead of directly training a deep model on the corrupted data, which is more efficient and may avoid overfitting noisy labels. \n", "summary_of_the_review": "The paper proposes a training-free instancewise noise label detection method. It supposes a good representation extractor is given and generates the initial soft labels based on the clusterability of representations using kNN. Followed by that,  the authors perform a local voting and global ranking-based scoring system to detect the corrupted labels. In general, the paper is well written and easy to follow. The main idea of utilizing good representation instead of training a deep inference model to avoid memorizing corrupted data sounds interesting and seems to work on CIFAR10 and CIFAR100 datasets.  However, I have some concerns about the definition and criterion of good representation which is the prerequisite of the method. As the proposed method uses kNN to analyze the clusterability of representations, I also wonder the increase of object categories may cause a significant accuracy drop in discrimination. Besides, the author only perform experiments on CIFAR10 and CIFAR100 instead of the acknowledged noisy-label datasets, which is not convincing enough for me. There are also some minor issues in the paper to be taken care of.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DiFLSNXqBsb", "reviewer": "Reviewer_fRLd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The perspective of this paper is somewhat novel compared with existing training-based methods. The presented solution would be beneficial in detecting noisy labels. The authors present their algorithm clearly and the theoretical analyses are reasonable.\nThe whole method is based on the assumption that good representations can distinguish the noisy and clean labels. But the representations of hard samples are likely to be similar to the noisy ones. Although SimiRep achieves high F1-scores, the filtered hard samples can have a significant impact on model robustness. It might be necessary to conduct follow-up experiments on the model performance without filtered samples of SimiRep.\nThere are some mistakes in this paper. In the third line from the last of section 3.1, “the noisy labels are referred in representation learning”. It’s not true and should be “are not referred”. In the first line of page 5, the example of y_n3 is written into “0.0.33”.", "summary_of_the_paper": "This paper proposed a training-free solution by using only good representations to detect noisy labels. The author designed two detection methods of voting and ranking to filter instances that are likely to be corrupted. The major contribution is proposing a training-free solution to efficiently detect noisy labels, which is different from most current methods. Also, theoretical analysis is conducted on the worst-case error bound and the choice of K.", "summary_of_the_review": "The perspective is novel, and the presented solution would be beneficial in detecting noisy labels. But the significance of detecting noisy labels is uncertain and needs more experimental proof.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5OW6vv3pO83", "reviewer": "Reviewer_fWAV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* The idea of using a pre-trained model to help detect noisy samples is interesting. The training-free approach is also largely different from existing works in literature.\n* The finding in Sec 5.3 is particularly interesting as well. Table 3 in this section shows that contrastive pre-training shows better generalization ability than supervised pre-training for the task being addressed here. \n\nWeaknesses\n* The major concern of this work is about the experimental setup. CIFAR10/100 has a clear class definition across only 10 or 100 classes. By using a well-trained model pre-trained on a large amount of data, the representations of the samples from CIFAR are already well separated. That is also the reason the proposed K-NN approach works well compared to learning-based methods that do not have the advantage of large-scale pre-training. To justify the effectiveness of the proposed method in practice, it is crucial to test the algorithm on more complex and larger datasets. \n* The theoretical analyses are highly appreciated in the submission. However, the analyses are based on a strong assumption that nearby representations should belong to the same true class. This might be true for simple CIFAR10 with a balanced training sample across 10 classes. In reality, the training set is usually imbalanced and nearby samples could have completely different labels. \n* Figure 1 can be improved. The green circle in the figure is not defined.\n* Typo: page 5 y_n3=[0.34, 0.033, 0.33] -> [0.34, 0.33, 0.33]", "summary_of_the_paper": "The authors propose a training-free approach to detect samples with noisy labels by leveraging representations learned from pre-trained models. The models can be obtained by either supervised or self-supervised pre-training. The authors then argue that samples in the pre-trained manifold should be closer if they share the same clean label, therefore one can use (a) local voting or (b) ranking methods to detect samples with corrupted labels. Experiments on CIFAR10/100 show improvement over other learning-based approaches (CORES, CL, TracIn).", "summary_of_the_review": "Although I appreciate the idea of using pre-trained models to provide guidance in selecting corrupted samples. It is also important to ensure the proposed method works well in practice, especially for imbalance or more complex large-scale settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a training-free method to detect noisy labels based a given pretrained representation. The author suggested two methods based on either voting or ranking to filter noisy instances with corrupted labels without training. Reviewers generally agree that the technical novelty and contributions are only limited or marginally significant. Also experiments are not very convincing as there lacks of extensive comparisons with many existing methods for either noisy label removal or learning with noisy labels and the datasets are somewhat simple and not complex enough.
Decision: Reject