FINE Samples for Learning with Noisy Labels
Authors: Taehyeon Kim
Jongwoo Ko
Sangwook Cho
JinHwan Choi
Se-Young Yun
Venue: NeurIPS 2021 Poster (2021)
Keywords: Learning with noisy labels
Image Classification
Deep Neural Networks
Eigen Decomposition

Abstract: Modern deep neural networks (DNNs) become frail when the datasets contain noisy (incorrect) class labels. Robust techniques in the presence of noisy labels can be categorized into two folds: developing noise-robust functions or using noise-cleansing methods by detecting the noisy data. Recently, noise-cleansing methods have been considered as the most competitive noisy-label learning algorithms. Despite their success, their noisy label detectors are often based on heuristics more than a theory, requiring a robust classifier to predict the noisy data with loss values. In this paper, we propose a novel detector for filtering label noise. Unlike most existing methods, we focus on each data's latent representation dynamics and measure the alignment between the latent distribution and each representation using the eigen decomposition of the data gram matrix. Our framework, coined as filtering noisy instances via their eigenvectors (FINE), provides a robust detector with derivative-free simple methods having theoretical guarantees. Under our framework, we propose three applications of the FINE: sample-selection approach, semi-supervised learning approach, and collaboration with noise-robust loss functions. Experimental results show that the proposed methods consistently outperform corresponding baselines for all three applications on various benchmark datasets.

FINE Samples for Learning with Noisy Labels
Tachyeon Kim KAIST Al KAIST
Jongwoo Ko KAIST KAIST
Daejeon, South Korea
Daejeon, South Korea jongwoo. ko.kaist.
potter328kaist.ac.
Sangwook Cho
Jinhwan Choi KAIST AI KAIST
Se- Young Yun KAIST AI KAIST
KAIST AI KAIST
Daejeon, South Korea sangwookcho@kaist..c ac.kr
Daejeon, South Korea jinhwanchoi @kaist ac.kr
Daejeon, South Korea yunseyoungdkaist.cmm
Abstract
Modern deep neural networks (DNNss become weak when the datasets contain noisy (incorrect) class labels. Robust techniques in the presence of noisy labels can be categorized into two types: developing noise-robust functions or using noise- cleansing methods by detecting the noisy data. Recently, noise-cleansing methods have been considered as the most competitive noisy-label learning algorithms. De- spite their success, their noisy label detectors are often based on heuristics more than a theory, requiring a robust classifier to predict the noisy data with loss val- ues. this paper, we propose a novel detector for filtering label noise. Unlike most existing methods, we focus on each data point's latent representation dy- namics and measure the alignment between the latent distribution and each repre- sentation using the eigen decomposition of the data gram matrix. Our framework, coined as filtering noisy instances via their eigenvectors (FINE), provides robust detector using derivative-free simple methods with theoretical guarantees. Under our framework, we propose three applications of the FINE: sample-selection ap- proach, semi-supervised learning (SSL) approach, and collaboration with noise- robust loss functions. Experimental results show that the proposed methods con- sistently outperform corresponding baselines for all three applications on various benchmark datasets
Introduction
Deep neural networks (DNNs) have achieved remarkable success in numerous tasks as the amount of accessible data has dramatically increased [21 [5]. On the other hand, accumulated datasets are typically labeled by a human, a labor-intensive job or through web crawling [48] so that they may be easily corrupted (label noise) in real-world situations. Recent studies have shown that deep neu- ral networks have the capacity to memorize essentially any labeling of the data [49]. Even small amount of such noisy data can hinder the generalization of DNNs owing to their strong memorization of noisy labels [49 [29] Hence, it becomes crucial to train DNNs that are robust to corrupted labels. As label noise problems may appear anywhere such robustness increases reliability in many appli- cations such the e-commerce market [9]. medical fields 45), on-device Al [46). and autonomous
driving systems [I]]
To improve the robustness against noisy data, the methods for learning with noisy labels (LNL) have been evolving in two main directions 11 (1) designing noise-robust objective functions or regular-
Equua contribution
Code available
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

LABEL: 900
CLEAN
FIT
UNEAR
REATURE DXTRACTOR
VECTOR
NO4SY
Hoot
NOIY
BIGEN DECOMPOSSION
GAUSSIAN MIXTURE MODEL
(a) Noise- Saiie-Cleansing--ee based Approach
(b) FINE
Figure 1: Illustration of (a) basic concept of this work and (b) proposed detection framework, FINE. Noise-cleansing learning generally separates clean data from the original dataset by using prediction outputs. We propose novel derivative free detector based on an unsupervised clustering algorithm on the high-order topological space. FINE measures the alignment of pre-logits (i.e., penultimate layer representation vectors) toward the class-representatiie vector that is extracted through the eigen decomposition of the gram matrix of data representations.
izations and (2) detecting and cleansing the noisy data. In general, the former noise-robust direction 50 or robust loss functions 38
51 but their performance is far from state-of-the-an 49 26 on datasets with severe noise rates. Recently, re- searchers have designed noise-cleansing algorithms focused on segregating the clean data (i.e., sam- 42 One of the popular 32
uses explicit regularization techniques 6 52
ples with uncorrupted labels) from the corrupted data [99 14 47 18
criteria for the segregation process is the loss value between the prediction of the noisy classifier and its noisy label, where it is generally assumed that the noisy data have large loss [19 [4] 47 [8] or the magnitude of the gradient during training [51 40]. However, these methods may still be bi- ased by the corrupted linear classifier towards label noise because their criterion (e.g.. loss values or weight gradient) uses the posterior information such a linear classifier [24]. Maennel et al. [31 analytically showed that the principal components of the weights of neural network align with the randomly labeled data; this phenomenon can yield more negative effects on the classifier as the number of randomly labeled classes increases. Recently, Wu al. 44 used an inherent geo- metric structure induced by nearest neighbors (NN) in latent space and filtered out isolated data in such topology, and its quality was sensitive its hyperparameters regarding NN clustering in the presence of severe noise rates.
To mitigate such issues for label noise detectors, provide novel yet simple detector frame- work, filtering noisy labels via their eigenvectors (FINE) with theoretical guarantees to provide a high-quality splitting of clean and corrupted examples (without the need estimate noise rates). Instead of using the neural network's ' linear classifier, FINE utilizes the principal components of latent representations made by eigen decomposition which is of the most widely used unsuper- vised learning algorithms and separates clean data and noisy data by these components (Figure Ia). To motivate our approach, as Figure 1b shows, we find that the clean data (blue points) mainly aligned on the principal component (black dotted line), whereas the noisy data (orange points) are not; thus, the dataset is well clustered with the alignment of representations toward the principal component by fitting them into Gaussian mixture models (GMM). We apply our framework to var- ious LNL methods: the sample selection approach, supervupe learning (SSL) approach, and collaboration with noise-robust loss functions. The key contributions of this work are summarized as follows:
We propose a novel framework, termed FINE (filtering noisy labels via their eigenvectors), for detecting clean instances from noisy datasets. FINE makes robust decision boundary for the high-order topological information of data in latent space by using eigen decomposition of their gram matrix.
We provide provable evidence that FINE allows a meaningful decision boundary made by eigenvectors in latent space. We support theoretical analysis with various experimental results regarding the characteristics of the principal components extracted by our FINE detector.
We develop a simple sample-selection method by replacing the existing detector method with FINE. We empirically validate that sample-selection learning with FINE provides consistently superior detection quality and higher test accuracy than other existing alterna- tive methods such as the Co-teaching family [4 47). TopoFilter 42), and CRUST [32).

---

We experimentally show that our detection framework can be applied in various ways to existing LNL methods and validate that ours consistently improves the generalization in the presence of noisy data: sample-selection approach [14 47). SSI. approachh [25] and
collaboration with noise-robust loss functions 51 29
Organization. The remainder of this paper is organized as follows. In Section 2
we discuss the
recent literature on LNL solutions and meaningful detectors. In Section we address our motivation for creating noisy label detector with theoretical insights and provide our main method, filtering the noisy labels via their eigenvectors (FINE). In Section 4 we present the experimental results. Finally, Section 5 concludes the paper.
2 Related Works
Zhang al. [49] empirically showed that any convolutional neural networks trained using stochastic gradient methods easily fit a random labeling of the training data. To tackle this issue, numerous works have examined the classification task with noisy labels. We do not consider the works that assumed the availability of small subsets of training data with clean labels [1 36 39 53 3].
Noise- Seeeee-leaannnnnnnne based Approaches. Noise-cleansing methods have evolved following the im- provement of noisy detectors. Han et al. [14] suggested a noisy detection approach, named co- teaching, that utilizes two networks, extracts subsets of instances with small losses from each net- work, and trains each network with subsets of instances filtered by another network. Yu et al. [47] combined a disagreement training procedure with co-teaching, which only selects instances pre- dicted differently by two networks. Huang et al. [88] provided a simple noise-cleansing framework, waatinnggiieettttraaaaa training; the empirical efficacy was improved by first finding label errors, then training the model only on data predicted as clean. Recently, new noisy detectors with theoretical support have been developed. Wu et al. [4] proposed a method called TopoFilter that filters noisy data by utilizing the k- k-nearest neighborhood algorithm and Euclidean distance between pre-logits. Mirzasoleiman al. 2 introduced an algorithm that selects subsets of clean instances that provide an approximately low-rank Jacobian matrix and proved that gradient descent applied to the subsets prevents overfitting noisy labels. Pleiss et al. 34 proposed an area under margin (AUM) statistic that measures the average difference between the logit values of the assigned class and its high- est non-assigned class to divide clean and noisy samples. Cheng et. al [8] progressively filtered out corrupted instances using a novel confidence regularization term. The noise-cleansing method was also developed in a semi-supervised learning (SSL) manner. Li et al. (5 modeled the per-sample loss distribution and divide it into a labeled set with clean samples and an unlabeled set with noisy samples, and they leverage the noisy samples through the well-known SSL technique MixMatch [4].
Woise-Robuse Models. Noise-robust models have been studied in the following directions: robust- loss functions, regularizations, and strategies. First, for robust-loss functions, Ghosh et al. [3] showed that the mean absolute error (MAE) might be robust against noisy labels. Zhang & Sabuncu et al. (51) argued that MAE performed poorly with DNNs and proposed a GCE loss function, which can be seen
generalization of MAE and cross-entropy (CE). Wang al. 40 introduced the re- verse version of the cross-entropy term (RCE) and suggested that the SCE loss function is a weighted sum of the CE and RCE. Some studies have stated that the early-stopped model can prevent the memorization phenomenon for noisy labels (2 49 and theoretically analyzed it 126.. Based on this hypothesis, Liu al. [29] proposed an early-lcarning regularization (ELR) loss function to prohibit memorizing noisy data by leveraging the semi- semi-supervised learning techniques. Xia et al. 4 clari- fied which neural network parameters cause memorization and proposed a robust training strategy for these parameters. Efforts have been made to develop regularizations on the prediction level by smoothing the one-hot vector [30], using linear interpolation between data instances (50), and dis- tilling the rescaled prediction of other models (0).. However, these works have limitations in terms of performance as the noise rate of the dataset increases.
Dataset Resampling. Label-noise detection may be a category of data resampling which is com- mon technique in the machine learning community that extracts a "helpful" dataset from the distri- bution of the original dataset to remove the dataset bias. In class-imbalance tasks, numerous studies have conducted over-sampling of minority classes [ or undersampling the majority classes [5] to balance the amount of data per class. Li & Vasconcelos et al. (27] proposed a resampling procedure to reduce the representation bias of the data by learning a weight distribution that favors difficult instances for a given feature representation. Le Bras et al. 22 suggested an adversarial filtering- based approach to remove spurious artifacts in a dataset. Analogously, in anomaly detection and
3

---

Algorithm 1:
FINE Algorithm for Sample Selection
INPUT Noisy training data D, feature extractor g, number of classes K, clean probability threshold ç, set of FINE scores for class ls Fk
OUTPUT Colllected clean data C
1: Initialize C f 0, D + D, Σ + 0 for all k = I....,
1* Update the convariance matrices for all classes */
2: for (24,, 9:) E D do
3: 2; < g(z;))
4: Update the gram matrix Ey. By, + z,2,
5: end for
1* Generate the principal component with eigen decomposition */
6: for k = .,..., K do
7: UR, A4 EIGEN DECOMPOSITION Σn
8: Ug: + THE FIRST COLUMN OF U
9: end for
1* Compute the alignment score and get clean subset C */
for (æt, Bi) E D do
Compute the FINE score fa (ugg x and Fy, + Fy (fi)
12: end for
1 Finding the samples whose clean probability larger than Ç *
13: C + CU GMM (Fr,S) for all k I....,
-ittii--isttitttion detection problems [66 23]. the malicious data are usually detected by exam- ining the loss value negative behavior in the feature representation space. While research is motivated by such previous works, this paper focuses on the noisy image classification task.
Method
In this section, we present our detector framework and the theoretical motivation behind using the detector in high-dimensional classification. To segregate the clean data, we utilize the degree of alignment between the representations and the eigenvector of the representations' gram matrices for all classes, called FINE (Flltering Noisy instnaces via their Eigenvectors). Our algorithm is as follows (Algorithm D. FINE first creates gram matrix of the representation in the noisy training dataset for each class and conducts the eigen decomposition for those gram matrices. Then, FINE finds clean and noisy instances using the square of inner product values between the representations this manner, we treat the data as clean
and the first eigenvector having the largest eigenvalue. aligned onto the first eigenvector, while most of the noisy instances not. Here, formally define "alignment" and "alignment clusterability' in Definition T and Definition]] respectively.
Definition 1. (Alignment) D
Definition 2. (Alignment Clusterability) For all features labeled as class k in dataset DD let fit Gaussian Mixture Model (GMM) on their alignment (Definition (7] distribution samples into clean set and a noisy set; the set having larger mean value is treated as clean set, and another one a noisy set. Then, we say a dataset D satisfies alignment clusterability if the representation labeled as the same true class belongs to the clean set.
divide current
As an empirical evaluation, the quality of our detector for noisy data is measured with the F-score, a widely used criterion in noisy label detection, anomaly detection and out-of- -iit-ri-iitiittionn detection 88 [6, 28 23). We treat the selected clean samples as the positive class and the noisy samples as negative class. The F-score is the harmonic mean of the precision and the recall; the precision indicates the fraction of clean samples among all samples that are predicted as clean, and the recall indicates the portion of clean samples that are identified correctly.
3.1 Alignment Analysis for Noisy Label Detector
To design robust label noise filtering framework, we explore the linear nature of the topological space of feature vectors for data resampling techniques and deal with the classifier contamination due to random labels. Recent studies on the distribution of latent representations in DNNs provide

---

insight regarding how correctly the outlier samples can be filtered with the hidden space's geomet- rical information. For instance, in [23] 24 the authors proposed frameworks for novelty detection using the topological information of pre-logit based on the Mahalanobis distance, and, in [42], the authors filtered the noisy data based on the Fuclidean distance between pre-logits. Maennel et al. analytically showed that an alignment between the principal components of network parameters and those of data takes place when training with random labels. This finding points out that random labels can corrupt classifier, and thus building robust classifier is required.
Motivated by these works, we aim to design a novel de- tector using the principal components of latent features to satisfy Definition 2 However, it is intractable to find the optimal classifier to maximize the separation of alignment clusterability because clean data distribution and noisy data distribution are inaccessible. To handle this issue, we attempt to approximate the clean eigenvector to maximize the alignment values of clean data rather than to maximize the separation; the algorithm utilizes the eigenvector of the data for each class (Tigure 2). Below, we provide the upper bound for the perturbation toward the clean data's ' eigenvector under simple problem settings with noisy la- bels referred to in other studies [2). We first introduce notations. Next, establish the theoretical evidence that our FINE algorithm approximates the clean data's eigen- vectors under some assumptions for its analytic tractabil- ity (Theorem D. We mainly present the theorem and its interpretation; details of the proofs can be found in the Appendix.
Clean
Perturbation
Noisy
Figure 2: Illustration for the prob- lem setrings and [Theorem The per- turbation (green shade) is the an- gle between the first eigenvector of clean instances (blue line) and the es- timated first eigenvector (green line) which is perturbed by that of noisy in- stances (orange line). Note that blue and orange points are clean instances and noisy instances, respectively.
Notations. Consider binary classification task. As- sume that the data points and labels lie in X' x V, where the feature space X' C. Rd and label space y [-1,+11, A single data point x and its true label y follow a dis- tribution (x. y) Pxxy. Denote by ÿ the observed la- bel (potentially corrupted). Without loss of generality, we focus on the set of data points whose observed label is ÿ +1.
Let X C X be the finite set of features with clean instances whose true label is y = +1. Similarly, let X C X be the set of noisy instances whose true label = 1. establish our theorem, assume the following reasonable conditions referred to other works using linear discriminant analysis (LDA) assumptions [24;
Assumption 1. The feature distribution is comprised of two Gaussians, each identified clean cluster and noisy cluster.
Assumption 2. The features of all instances with y = +1 are aligned on the unit vector with the white noise, i.e., Emex (x]] = v. Similarly, features of all instances with y = are aligned on the unit vector w, i.e., F "xE [z] un.
Theorem 1. (Upper bound for the perturbation towards the clean data's eigenvector v) Let N and N_ be the number of clean instances and noisy instances, respectively, and u be the FINE's eigenvector which is the first column of U from the eigen decomposition of the whole data's 's matrix E. For any ö E (0, 1), its perturbation towards the v in assumption fi.... 2-norm for difference of projection matrices; left hand side of Eq. 77) holds the following with probability 8:
37 cos 0 +0(22
d+1og((4/8)
|uuu" -vv" 112
N+
1 1 (sin + 3 cos 9) - O(o2 d+log(1/3)
where w is the first eigenvector of noisy instances, T is the fraction between noisy and clean in- stances ( (N,) ), is L(w, v), and d² is a variance of white noise.
Theorem states that the upper bound for the perturbation toward u are dependent on both the ratio and the angle 0 between and v; small upper bound can be guaranteed as the number of clean data
5

---

63
20%
22
033
033
-1
0m
(a) CIFARIO
(b) CIFARI00
(c) Perturbation
(d) Sculability
Figure 3: (a), (b): Heatmaps of Eq. 2 values unit circle in random hyperplane. We evaluate this visualization on the ResNet34 model trained with common cross-entropy loss on CIFAR-10 with asymmetric noise 40% and CIFAR-I00 with symmetric noise 80%, respectively. Colors closer to yellow indicate larger the values; (c): comparison of perturbations of Eq. T on CIFAR-10 with symmetric noise 20%; (d): comparison of cosine similarity values between FINE's principal compo- nents and approximated principal components using fraction of data on CIFAR-10 10 with symmetric noise 80%.
increases, T decreases, and & approaches 5. We also derive the lower bound for the precision and the recall when using the eigenvector u. in Appendix. In this theoretical analysis, we can ensure that such lower bound values become larger as e and w become orthogonal to each other. To verify these assumptions, we provide various experimental results for the separation of alignment clusterability, the perturbation values, the scalability to the number of samples, the quality of our detector in the application of sample selection approach, and the comparison with an alternative clustering- estimator 24
Validation for our Estimated Eigenvector. To validate our FINE's principal components, we first propose a simple visualization scheme based on the following steps: (1) Pick the first eigenvector (u) extracted by FINE algorithm, (2) Generate random hyperplane spanned by such eigenvector (u) and random vector, (3) Calculate the value the following Eq. (2) on any unit vectors (a) such hyperplane and plot heatmap with them:
aa.
x1
1
Σ xa, Ø,EX
(2)
X
X
2
Eq. (2) is maximized when the unit vector a not only maximizes the FINE scores of clean data for the first term in Eq. (2). but also minimizes those of noisy data for the second term in Eq. (2). This visualization shows in 2-D how FINE's first eigenvector (u) optimizes such values in the presence of noisy instances (Figure 3a and 3b). As the figures show, the FINE's eigenvector u (red dotted line) has almost maximum value of Eq. 2 Furthermore, we empirically evaluate the perturbation values in Theorem as the noise rate changes (Figure 3b); FINE has small perturbation values even severe noise rate.
Scalability to Number of Samples. Despite FINE's superiority, it may require high computational costs if the whole dataset is used for eigen decomposition. To address this issue, approximate the eigenvector with a small portion of the dataset and measure the cosine similarity values between the approximated term and the original eigenvector is computed even using 1% data (i.e., a cosine similarity value is 0.99), and thus the eigenvector can be accurately estimated with little computation time.
(u) (Figure 3(t). Interestingly, we verify that far accurate
Validation for Dynamics of Sample-selection Approach. We evaluate the F-score dynamics of every training epoch on the symmetric and the asymmetric label noise in Figure 4 We compare FINE with the following sample-selection approaches: Co-teaching [14] and Topol-ilter [42]. In Fig- ure 4 during the training process, F-scores of FINE becomes consistently higher on both symmetric noise and asymmetric noise settings, while Co-teaching and TopoFilter achieve lower quality. Un- like TopoFilter and FINE, Co-teaching even performs the sample-sclection with the access of noise selection approaches.
rate, This evidence show that FINE also applicable to the naive sample-so sample-selection

---

CIFAR 10 Sym 80%.
CIFAR 100 Sym 80%.
CIFAR 100 Asym 40%.
0.8
0.6
0.9
0.6
0.5
0.8
0.4
0.3
0.7
0.2
0.2
0.6
1B6
Go-saacing
Tepeltbr
0.1
0.0
200
0.0
0.5
40
100
120
140
(a) Sym 80% (C10)
(b) Sym 80% (C100)
(c) Asym 40% (C100)
Figure 4: Comparisons of F-scores on CIFARIO and CIFARI00 under symmetric and asymmetric
label noise. C1O and C100 denote CIFAR-10 and CIFAR-100, respectively.
CIFAR
CIFAR 100
Comparison for Mahalanobis Distance Es- timator Under similar conditions, Lee et al. (2 measured the Mahalanobis distance of pre-logits using the minimum covariance de- terminant (MCD) estimator and selected clean samples based on this distance. While they also utilized the LDA assumptions on pre-logits, FINE consistently outperforms MCD in both precision and recall, thus yielding better F- score (Figure 5). The experimental results jus- tify our proposed detector, in comparison with similar alternative.
1.0
8.
0.8
6.2
0.2
20 50
$ 88
A 40
50
50
80
40
MCD
FINE
Figure 5: Comparisons of F-scores CIFAR-10 and CIFAR-I00 under symmetric (S) and asym- metric noise (A) settings.
Experiment
In this section, demonstrate the effectiveness of our FINE detector for three applications: sample selection approach, SSL, and collaboration with noise-robust loss functions.
4.1 Experimental Settings
we artificially generate two types of random noisy labels: injecting uniform randomness into a fraction of labels (symmetric) and corrupting label only to specific class (asymmetric). For example, we generate noise by mapping TRUCK + AUTOMOBILE, BIRD -> AIRPLANE, DEER + HORSE, CAT +> DOG to make asymmetric noise for CIFAR-10. For CIFAR- 100, we create 20 five-size super-classes and generate asymmetric noise by changing each class to next class within super-classes circularly. For a real-world dataset, Clothing MM [44] containing inherent noisy labels is used. This dataset contains million clothing images obtained from online shopping websites with classe The dataset provides 50k, 14k, and 10k verified as clean data for training, validation, and testing. Instead of using the 50k clean training data, we use a randomly sampled pseudo-balanced subset as a training set with 120k images. For evaluation, compute the classification accuracy the 10k clean dataset.
Noisy Benchmark Dataset. Following the previous setups 25
29
Networks and Hyperparameter Settings. We use the architectures and hyperparameter settings for all baseline experiments following the setup of Liu al. [29] except with SSL approaches. For SSL approaches, we follow the setup of Li et al. 25 We set the threshold as 0.5.
4.2 Application of FINE
4.2.1 Sample Selection-Basee Approaches
We apply our FINE detector for various sample selection algorithms. In detail, after warmup train- ing, at every epoch, FINE selects the clean data with the eigenvectors generated from the gram matrices data predicted to be clean in the previous round, and then the neural networks are trained with them. We compare
proposed method with the following sample selection approaches: (1)
Bootstrap 33), (2) Forward (33, (3) Co-teaching 14] (4) Co-teaching+ [47]: (5) TopoFilter [42];
(6) CRUST 32 We evaluate these algorithms three times and report error bars.
'T-shirt, Shirt, Knitwear, Chiffon, Sweater, Hoodie, Windbreaker, Jacket, Down Coat, Suit, Shaw], Dress, Vest, and Underwear. The labels in this dataset are extremely noisy (estimated noisy level is 38.5% (37].
7

---

Table 1: Test accuracies (%) on CIFAR-10 and CIFAR-100 under different noisy types and frac- tions. All comparison methods are reproduced with publicly available code, while the results for Bootstrap 35 and Forward [33 are taken from [29]. For CRUST [2], we experiment without mix- up to compare the intrinsic sample selection effect of each method. The average accuracies and standard deviations over three trials are reported. Here, we substitute the sample selection method of Co-teaching [14 47 with FINE (i.e., F-Co-teaching) The best results sharing the noisy fraction and method are highlighted in bold.
Datasct
CIFAR-10
CIFAR-I00 100
Noisy Type
Sym
Asym
Sym
Asym
Noise Ratio
20
50
80
40
20
50
80
40
Standard
87.0 +000011.
78.2 = 0.8
53.8 + 1.0
85.0 + 0.0
58.7 + 0.3
42.5 + 0.3
18.1 + 0.8
42.7 = 0.6
Bootstrap 33
86.2 + 0.2
54.1
- 1.3
81.2
58.3 0.2
216 1.0
45.1
Forward
88.0 0.4
54.6
0.4
83.6
39.2
0.0
34.4 1.9
Co-teaching
44
89.3 0.3
83.3 = 0.6
66.3
+ 1.5
88.4 0.44220 28
63.4
49.1 0.4
20.5
47.7 1.2
Co-teaching+
77
89.1
0.4
86.5
5.
2022
Tupofiilerr
42
90.4 1: 0.2
86.8 1 0.3
46,8 1: 1.0
87.5
1: 0.4
66.9 1 0.4
$3.4 1: 1.8
18.3 1: 1.7
56.6 0.5
CRUST
89.4
0.1
82.4
69.3 4
62.3 0.2
21.7
56.1
FINE 91.0
87.3 = 0.2
69.4 = 1.1
89,5 0.1
70.3 + 0.2
64.2 = 0.5
25.6 = 1.2
61.7
= 1.0
F-Cotcaching
92.0
87.5 0.1
74.2
0.8
90.5
71.1
0.3
1.0
64,8
Table summarizes the performances of different sample selection approaches on various noise dis- tribution and datasets. We observe that our FINE method consistently outperforms the competitive methods over the various noise rates. Our FINE methods can filter the clean instances without losing essential information, leading to training the robust network.
To go further, we improve the performance Co-teaching [4] by substituting its sample selection state with our FINE algorithm. To combine FINE and the Co-teaching family, unlike the original methods that utilize the small loss instances train with clean labels, we train one model with extracted samples by conducting FINE on another model. The results of the experiments are shown in the eighth and ninth rows of Table
4.2.2 SSL-Based Approaches
SSL approaches 25 10 41 divide the training data into clean instances as labeled instances and noisy instances as unlabeled instances and both the labeled and unlabeled samples to train the networks in SSL. Recently, methods belonging to this category have shown the best performance among the various LNL methods, and these methods can train robust networks for even extremely high noise rates. We compare the perfor- mances of the existing supervsse approaches and that in which the sample selection state of DivideMix (25) is substi- tuted with our FINE algorithm (i.e., F-DivideMix). The results of the experiments are shown Table 2 We achieve consis- tently higher performance than DivideMix by utilizing FINE instead of its loss-based filtering method and show compa- rable performance to the state-of-the-art SSL methods such as DST [4] and LongReMix [10]. Interestingly, as Figure shows, clean and noisy data well classified in F-DivideMix under extreme noise cases.
DivideMix
F-DivideMix
0.9
F-score
0.8
0.7
0.6
0.5
50
100
150
200
250
Epochs
Figure 6: Comparisons of F-scores CIFAR-10 under symmetric 90% noise. Blue line indicates the error bar of two networks' F-score used in Dividemix (25), and Or- ange line indicates those replaced by our FINE detector.
4.2.3 Collaboration with Noise-Robus. Loss Functions
The goal of the noise-robust loss function is to achieve small risk for unseen clean data even when noisy labels exist in the training data. There have been few collaboration studies of the noise-robust loss function methodology and dynamic sample selection. Most studies have selected clean and noisy data based on cross-entropy loss.
Here, we state the collaboration effects of FINE with various noise-robust loss functions: gener- alized cross entropy (GCE) (5I, symmetric cross entropy (SCF) 44), and early-learning regulariza- tion (ELR) (29] Figure shows that FINE facilitates generalization in the application of nobse-robust.

---

Table 2: Comparison of accuracies (%) for FINE collaborating with DivideMix and existing semi-supervised approaches on CIFAR-10 and CIFAR 100 under different noisy types and fractions. The results for all comparison methods are taken from their original works.
Dataset
CIFAR-10
CIFAR-100
Noisy Type
Sym
Asym
Sym
Noise Ratio
20
50
80
90
40
20
50
80
90
DivideMix 25
Best
96.1
94.6
93.2
76.0
93.4
77.3
74.6
60.2
31.5
92.9
75.4
76.9
74.2
59.6
Best
96.1
95.2
92.9
94.3
78.0
74.3
57.8
DST
Last
95.9
94.7
92.6
92.3
77.4
73.6
55.3
Lonx T0
95.0
93.9
82.0
77.8
75.6
62.9
33.8
Last
96.0
94.7
93.4
81.3
94.3
77.5
75.1
62.3
33.2
DivideMix
74.6
asst
96.0
94.5
93.2
89.6
92.8
78.8
74.3
60.1
31.2
CIFAR10 Sym 80%
CIFAR10 Asym 40%.
CIFAR100 Sym 80%
CIFAR100 Asym 40%.
95
40
30
60-
25
50
70
15
ELR 40
SCE
SCE
ELR
(a) Sym 80% (C10)
(b) Asym 40% (C10)
(c) Sym 80% (C100)
(d) Asym 40% (C100)
Figure 7: Test accuracies (%) on CIFAR-10 and CIFAR- under different noisy types and fractions for noise-robust loss approaches. Note that the blue and orange bars results for without and with FINE, respectively. The average accuracies and standard deviations over three trials are reported.
loss functions on severe noise rate settings. The detailed results are reported in the Appendix. Un- like other methods, it is still theoretically supported because FINE extracts clean data with a robust classifier using representation.
Experiments on Real-World Dataset
As Table shows, FINE and M-DivideMix work fairly well the Clothing 1M dataset compared to other approaches when we reproduce the experimental results under the same settings.
Table 3: Test accuracy Clothing MM dataset
Methed
Standard
OCB 51
SCE 50
ELR 29
DivideMix 25
CORES g
FINE
F-DivideMix
Accuracy
68
71.02
72.87
74.30
73.24
72.91
74.37
5 Conclusion
This paper introduces FINE for detecting label noise by designing robust noise detector. Our main idea is utilizing the principal components latent representations made by eigen decomposition. Most existing detection methods are dependent on the loss values, while such losses may be biased by corrupted classifier (1]. Our methodology alleviates this issue by extracting key information from representations without using explicit knowledge of the noise rates. We show that the FINE detector has an excellent ability to detect noisy labels theoretical and experimental results. We propose three applications of the FINE detector: sample-selection approach, SSL approach, and collaboration with noise-robust loss functions. FINE yields strong results on standard benchmarks and wolld dataset for various LNL approaches.
We believe that our work opens the door to detecting samples having noisy labels with explainable results. It is non-trivial task and of social significance, and thus, our work will have a substantial social impact on DI. practitioners because avoids the labor-intensive job of checking data label quality. As future work, hope that our work will trigger interest in the design of new label-

---

noise detectors and bring fresh perspective for other resampiigg approaches (e.g., anomaly detection and novelty detection). The development of robustness against label noise even leads to an improvement in the performance of network trained with data collected through web crawling. We believe that our contribution will lower the barriers to entry for developing robust models for DL practitioners and greatly impact the internet industry. On the other hand, we are concerned that it can be exploited to train robust models using data collected illegally and indiscriminately on the dark (e.g. web crawling), and thus it may raise privacy concerns (e.g., copyright).
Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No.2019-0-0007,, Artificial In- telligence Graduate School Program (KAIST)) and [No. 2021-0-00907, Development Adaptive and Lightweight adge-Collaborative Analysis Technology for Enabling Proactively Immediate Re- sponse and Rapid Learning]. We thank Seongyoon Kim for discussing about the concept pertur- bation.
References
[1] Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbal- anced data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 770-785. Springer, 2017.
[2] Devansh Arpit, Stanislaw Jastrzæbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pages 243. PMLR, 2017.
[3] Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Conference on Machine Learning, pages 540-550. PMLR, 2020.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, pages 5049-5059, 2019.
[5] Mateusz Buda, Atsuto Maki, and Maciej Mazurowski. systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
[6] Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Het- croskedastic and imbalanced deep learning with adaptive regularization. arXiv preprint arXiv:2006. 2020.
[7] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321- 357, 2002.
[8] Hao Cheng. Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. arXiv preprint arXiv:2010.02347, 2020.
[9] Charles Corbiere, Hedi Ben- Younes, Alexandre Ramé, and Charles Ollion. Leveraging weakly annotated data for fashion image retrieval and label prediction. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages
2017.
[10] Filipe R Cordeiro, Ragav Sachdeva, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Longremix: Robust learning with high confidence samples in noisy label environment. arXiv preprint wxXiv:2103.04177 2021.
[11] Di Feng, Christian Haase-Schute Lars Rosenbaum, Heinz Hertlein, Claudius Glaser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and se- mantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Trans- actions on Intelligent Transportation Systems, 22(3):1341-136) Mar 2021.
10

---

[12] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugen- ics, 7(2):179-118, 1936.
[13] Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, 2017.
[14] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pages 8527-8537, 2018.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 770-778, 2016.
[16] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. arXiv preprint arXiv:1610.02136 2016.
[17] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. arXiv preprint arXiv:1802.05300 2018.
[18] Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. O2u-net: A simple noisy label detection approach for deep neural networks. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 3326-3334, 2019.
[19] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks corrupted labels, 2018.
[20] Taehyeon Kim, Jachoon Oh, NakYil Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback-leibler divergence and mean squared error loss knowledge distillation, 2021.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1105, 2012.
[22] Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In International Conference Machine Learning, pages 1078-1088. PMLR, 2020.
[23] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for de- tecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[24] Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, pages 3763-3772. PMLR, 2019.
[25] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
[26] Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stop- ping is provably robust to label noise for overparameterized neural networks. In International Conference Artificial Intelligence and Statistics, pages 4313-4324. PMLR, 2020.
[27] Yi Li and Nuno Vasconcelos. Repair:
Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9572 9581, 2019.
[28] Shiyu Liang. Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. arXiv preprint PrXiv:1706.0266 2017.
[29] Sheng Liu, Jonathan Neles-Weed, Narges Razavian, and Carlos Frrnanezz Farly-learning regularization prevents memorization of noisy labels. arXiv:2007.00151, 2020.
arXiv preprint
11

---

[30] Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smooth- ing mitigate label noise? In International Conference on Machine Learning, pages PMLR, 2020.
[31] Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert JN Baldock, Olivier Bous- quet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? arXiv preprint 2XXi::0006.105555 2020.
[32] Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of neural networks against noisy labels. arXiv preprint arXiv:2011.0745/ 2020.
[33] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1944-1952, 2017.
[34] Geoff Pleiss, Tianyi Zhang, Ethan R Elenberg, and Kilian Q Weinberger. Identifying misla- beled data using the area under the margin ranking. arXiv preprint arXiv:2001.10526 2020.
[35] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. arXiv preprint 1rXiv:1412.6596, 2014.
[36] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pages 4334 4343. PMLR, 2018.
[37] Hwanjun Song, Minseok Kim, Dongmin Park, and Jae- Bei Lee. Prestopping: How does early stopping help generalization against label noise?, 2020.
[38] Brendan Van Rooyen, Aditya Krishna Menon, and Robert C Williamson. Learning with sym- metric label noise: The importance of being unhinged. arXiv preprint 1505 07634, 2015.
[39] Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 839 847, 2017.
[40] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 322-330, 2019.
[41] Yi Wei, Xue Mei, Xin Liu, and Pengxiang Xu. Dst: Data selection and joint training for learning with noisy labels. arXiv preprint 1xXiv:2103.008// 2021.
[42] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. arXiv preprint arxiv:20/2.04835, 2020.
[43] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust carly-learning: Hindering the memorization of noisy labels. In International Confer- ence on Learning Representations, 2021.
[44] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 2691-2699, 2015.
[45] Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L. A novel information thhorrtic loss function for training deep nets robust to label noise. In NewrlPS, pages 6222-6233, 2019.
[46] Seunghan Yang, Hyoungseob Park, Junyoung Byun, and Changick Kim. Robust federated learning with noisy labels, 2020.
[47] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? arXiv preprint arXiv: 1901.042155 15, 2019.
12

---

[48] Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased comple- mentary labels. In Proceedings of the European conference on computer vision (ECCV), pages 68-83, 2018.
[49] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. arXiv preprint arX.v:1611.03530, 2016.
[50] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
[51] Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv:1805.07830 2018.
[52] Zizhao Zhang, Han Zhang. Sercan O Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 93033, 2020.
[53] Zizhao Zhang, Han Zhang, Sercan O. Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
13


Reviews:
[{"review_id": "nKTJTXhAQGh", "reviewer": "Reviewer_YKHM", "summary": "Deep neural networks become inefficient when the datasets contain noisy class labels. There are two types of robust techniques that are applied in the presence of the noisy labels: a) noise-robust loss functions; b) removing noisy data by detecting them. Most existing detection methods are dependent on the loss values, while such losses may be biased by corrupted classifier. This paper proposes a method that attempts to alleviate this issue by extracting key information from representations, but without using explicit knowledge of the noise rates.\n\nThe paper proposes a new method for filtering noisy data. It focuses on each data’s latent representation dynamics and measure the alignment between the latent distribution and each representation using the eigen decomposition of the data covariance matrix. This framework is coined as filtering noisy instances via their eigenvectors (FINE). It provides a detector with derivative-free simple methods with some theoretical guarantees. It provides a provable evidence that FINE allows meaningful decision boundary made by eigenvectors in latent space. It supports the theoretical analysis with various experimental results regarding the characteristics of the principal components extracted by the FINE detector.\n\nIt proposes three applications of the FINE: sample-selection approach, semi-supervised learning approach, and collaboration with noise-robust loss functions. It empirically validates that a sample-selection learning with FINE provides consistently more superior detection quality and higher test accuracy than other existing alternative methods such as Co-teaching families, TopoFilter, and CRUST. Further, experimental results show that the proposed methods consistently outperform corresponding baselines for all three applications on various benchmark datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed technique for detecting noisy data based on the degree of alignment between the representations and the eigenvector of the representations' covariance matrices for all classes is new and meaningful. The theoretical guarantees given in Theorem 1 are not very innovative but nonethless help understand the proposed approach. Further, numerical results for the case of \"Sample-selection\" shows that the proposed FINE approach consistently outperforms the baseline existing approaches for all the symmetric/assymetric noise settings on CIFAR-10 and CIFAR-100 datasets. These numerical results are the strongest part of the paper. Experiments for the setting of Semi-supervised learning show little gain over the baseline approaches. For all the numerical experiments, authors do compare their approach with multiple state-of-the-art competitive approaches. \n\nThe major limitation is that theoretical contribution of the paper is not very innovative and does not take into account the fact that the representations are learned dynamically using a noisy dataset. So, the original claim of the paper that their approach is different from the existing methods as in the existing methods are biased by the corrupted classifier is not true. Their proposed approach FINE suffers from the same issue as the representations that are used to compute the degree of alignment are latent representations of a corrupted classifier (a classifier leanred using noisy dataset). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "gLx75rjTG2K", "reviewer": "Reviewer_EK8d", "summary": "This paper is about learning with noisy labels. The authors propose a method which selects clean and noisy examples by first computing a covariance matrix (for each class separately) based on the learned representations. They then compute the first eigenvector corresponding to the largest eigenvalue from the covariance matrix. The \"cleanness\" measurement then becomes the inner product between the original representation and the eigenvector. The authors show multiple applications of their approach, such as sample selection, semi-supervised learning etc., and show that their method consistently improves the classification accuracy.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a simple but effective method based on the eigenvectors of representations. As far as I know, this is a novel approach, and is also supported by theoretical analysis in the paper. The authors also show that it can scale up to the large datasets by computing the eigenvectors on a smaller subset. \n\nThe paper is also well written and easy to understand. I also appreciate the fact that it can be easily combined with various existing learning with noisy labels approaches to further improve the accuracy. Overall, it's a nice paper to read with convincing results. However, I have some minor comments:\n\n- I think the paper can benefit from one baseline to show why the covariance matrix and eigenvectors are needed. More specifically, instead of computing the first eigenvectors, one can compute the class mean vector v_c, where v_c = 1/n_c \\sum z_i where all z_i belongs to class C, and n_c is the number of elements in class C. One can then compute the FINE score based on this vector, i.e. f_i = <v_c, z_i>. This comparison would make it more convincing to show why we need to compute the eigenvectors.\n\n- The proposed method brings minimal improvements in the real-world dataset experiments. Can the authors comment on why this is the case? Have they tried other real-world noisy datasets, e.g. Webvision, Food-101N?\n\n- Nitpick: I was initially confused by the terminology \"covariance matrix\" used throughout the paper. The confusion was mainly due to the lack of notation in the paper, i.e. the reader does not know if z_i is considered as a dx1 vector or 1xd. I assumed that it was it was dx1, which would make the \"covariance\" matrix dxd, as in PCA. However, after looking at the code (in supplementary material), it seems like that the \"covariance matrix\" is actually nxn, where n is the number of vectors in a class. Can the authors clarify this? If it is indeed nxn, perhaps calling it a Gram matrix would be better?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "Olj3Eu_yuZZ", "reviewer": "Reviewer_utGV", "summary": "This paper proposes FINE detector to detect samples with noisy labels. Specifically, authors first construct data covariance matrix by using the latent representation of each training sample, then perform eigen-decomposition on the data covariance matrix to generate the first eigenvector for each class. The authors state that the representation of clean samples have larger alignment score with eigenvector than noisy ones. The authors also provide theoretical results to analyze the robustness of FINE detector and shows this detector can be applied to multiple approaches including sample-selection, semi-supervised learning and collaboration with noise-robust loss functions. Experiments are conducted on CIFAR-10, CIFAR-100 and Clothing1M.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "***Strength***\n\n**1:** The noisy label detector is simple and novel. Necessary theoretical results are also provided to verify its robustness. \n\n**2:** I especially like the analysis part in Figure 3. It coincides with the theorem to help better understand the FINE detector.\n\n**3:** Experimental results on three applications are impressive. It shows that FINE can be combined with other approaches to further improve network performance.\n\n***Concerns***\n\n**1:** Since the noise rate is unknown, I am worried that FINE is sensitive to the hyper-parameter $\\zeta$ in Algorithm 1.  Authors should clarify the value of $\\zeta$  for each experiment setting and perform sensitivity analysis on $\\zeta$.\n\n**2:** The noise types in the paper are symmetric and asymmetric label noise which are assumed to be feature-independent. I am wondering how FINE performs on a more challenging setting such as feature-dependent label noise.  \n\n**3:** FINE only uses the first eigenvector. If it is possible to use multiple eigenvectors to calculate alignment scores. Is the result better than using only one single eigenvector?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "7jaPLzbSE4D", "reviewer": "Reviewer_VZE7", "summary": "This paper proposes a new method for learning with noisy labels. The proposed method belongs to the research direction of data-selection, and aims to better select the clean data out of noisy samples. The key component of the proposed method is a novel operation, named _filtering noisy labels via their eigenvectors_ (FINE). FINE first computes the covariance matrix of the feature representation for the noisy samples. Then the principal eigenvector is extracted from the covariance matrix via eigen decomposition. Afterwards, the similarity score between the feature vector of each sample and the eigenvector is computed. Finally, the computed similarly scores are used to fit a Gaussian Mixture Model, which predicts the probability that a sample is clean or noisy. The proposed FINE approach is simple and computationally feasible. Theoretically it is shown that FINE is able to approximate the eigenvector of clean data with guaranteed accuracy under a few reasonable assumptions. In practice, FINE is demonstrated to be effective by integrating it into three different kinds of label noise learning approaches. The experimental results show that FINE brings (much) performance gain over previous methods across several synthetic and real-world noisy datasets.\n\nTo summarize, the contributions of this paper include:\n\n1. A new method, named FINE, is developed to better select the clean data out of noisy dataset. Theoretically it is shown that FINE is able to approximate the true eigenvector of clean data with guaranteed upperbound. Also, it is shown that the data selection accuracy of FINE has guaranteed lower bound in terms of precision and recall.\n\n2. This paper also includes a few illustrative experiments to explain the characteristics of the proposed method. These experiments are helpful.\n\n3. In practice, FINE is integrated into three different kinds of label noise learning approaches, and (significant) performance gain is observed over different datasets, involving the synthetic and real-world noisy data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper is well motivated and has a clear organization. I think this paper is novel in terms of technical contribution and experimental evaluation:\n\n1. The proposed data-selection method is reasonably designed, with theoretical analysis and insights. The proof the theory is complete and illustrates the key ideas (yet with some typos and unclear definitions, though). \n\n2. Besides, to illustrate the characteristics of the proposed method, a series of experiments are conducted (on Page 6). These experiments are interesting, reasonably designed, and helpful in understanding the behavior of the method. In particular, I find Figure 3(a)(b) particularly useful and interesting. Also, the authors show that the proposed FINE can be efficiently computed via a simple approximate yet accurate strategy (i.e., using a subset of the noisy data). Such an experiment validation is useful since it shows the feasibility of FINE on large-scale datasets. \n\n3. Finally, to show the effectiveness and generality of FINE, the paper investigates the performance gain by integrating FINE into three different kinds of noisy label learning methods. The experimental results on synthetic datasets are impressive. In particular, significant performance gain is achieved under high noise ratios (80% symmetric noise and 40% asymmetric noise).\n\nBesides the above merits of this paper, it also has some weaknesses as follows:\n\n1. The literature review is inadequate. In particular, this paper only mentions and discusses two directions for handling noisy labels: data selection and designing noise-robust loss. Another important direction, i.e., label correction, is ignored. Label correction tries to correct the wrong labels and thus is different from the noise-robust loss and data-selection. Some existing works along this direction include: Li _et al_ (ICCV 2017), Tanaka _et al_ (CVPR 2018), Yi _et al_ (CVPR 2019), Zheng _et al_ (ICML 2020).\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Reference:\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Li _et al_. Learning from noisy labels with distillation. ICCV, 2017.\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Tanaka _et al_. Joint optimization framework for learning with noisy labels. CVPR, 2018.\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Yi _et al_. Probabilistic end-to-end noise correction for learning with noisy labels. CVPR, 2019.\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Zheng _et al_. Error-bounded correction of noisy labels. ICML, 2020.\n\n2. The accuracy of the proposed method is heavily subject to the degree of data imbalance. More specifically, to achieve good precision and recall for data-selection, the number of clean samples (i.e., $N_+$) should be much larger than the number of noisy samples (i.e., $N_-$). This phenomenon can be observed from Line 13 of Algorithm 1, Definition 2, Figure 2, and Eq.(21)(22) in the Appendix. More specifically, when $N_- \\geq N_+$, the precision and recall would be low, as indicated by the numerator of Eq.(21) and denominator of Eq.(22). This can also be visually inspected in Figure 2. For example, when $N_- \\geq N_+$, the principal eigenvector would be skewed towards the noisy samples; as a result, the F-score will be largely hurt because GMM would mistakenly predict many noisy samples as clean. Therefore, only in cases where $N_- \\leq N_+$ the proposed method would achieve a good performance. However, in practice this condition may not always hold.\n\n3. While the theoretical results are informative, they are for the binary case. It is unclear if the results would be extended to multi-class scenario.\n\n4. The Definition 2 is not very clear. In particular, \"... if the representation $z$ labeled as the same true class belongs to the clean set\" is a bit confusing. Does this sentence mean for all the samples whose noisy labels are the same with their underlying clean labels, they are successfully moved to the clean set? I think this sentence should be refined to make it clearer.\n\n5. Figure 3(a)(b) are illustrative. However, it is better to also provide the heatmaps for ground-truth principal eigenvector. This is because in some cases when the computed $u$ is deviated from the ground-truth eigenvector, the heatmaps of $u$ would still appear as Figure 3(a)(b). By providing the heatmaps of ground-truth principal eigenvector, it is clearer to see if the heatmaps of computed $u$ are as bright as those of ground-truth eigenvector, and thereby check if the computed $u$ is accurate enough.\n\n6. A minor concern: In Figure 3(c), although the perturbation values are small for low noise ratio (20%), they increase a lot for a few categories under high noise ratio (80%). Can we still say \"FINE has small perturbation values even in a severe noise rate\"?\n\n7. For the combination of FINE and Co-teaching, does it mean both the two networks use FINE for data-section within a batch? And one network uses the selected data from another network for training itself? If so, does the batch size need to be large for effectively computing FINE score? Do the two networks still need warmup? Overall I find the description \"we train one model with extracted samples by conducting FINE on another model\" is unclear. Further clarification is necessary.\n\n8. In Section 4.2.3, it is unclear how to combine FINE with noise-robust loss functions. No descriptions are provided. Does it mean Algorithm 2 is performed here, but the loss function $\\mathcal{L}$ is replaced with the noise-robust loss (such as GCE)?\n\n9. In Table 3, it is reported that DivideMix [26] achieves 74.30% test accuracy on Clothing1M. However, after checking the paper of DivideMix, I find it achieved 74.76% as reported in the original paper [26]. Is this a mistake? Overall the performance gain of FINE is marginal on the Clothing1M dataset.\n\n10. There are some typos and grammatical errors in the paper. Besides, some descriptions are unclear. The writing should be further improved. For example:\n\n-  Line 37-38: \"loss value between ... and its noisy label\". I think this description is confusing. A better one could simply be \"the loss value of the noisy classifier\". Line 39, \"the magnitude\", where \"the\" should be removed. Line 53: \"and separates\" -> \", and separates\" to break the long sentence. Line 92: \", then\" -> \", and then\". Line 113: \"Base on this\" -> \"Based on this\". Line 152: \"presented along\" -> \"presented with\". \"explore\" -> \"exploit\"? The description \"maximize the separation of alignment clusterability\" is confusing, and need to be refined. Line 201, the \"$v$\" should be italic. Line 241: \"F-scores of FINE becomes\" -> \"F-scores of FINE become\". \"the access of noise rate\" -> \"the access to noise rate\". \"and10k\"->\"and 10k\". \"methods belong to this\" -> \"methods belonging to this\". \"clean and noisy data is\" -> \"clean and noisy data are\".\n\n- For Lemma 3, I think it is more commonly to call it \"David-Kahan sin($\\theta$) Theorem\".\n\n- One typo in Eq.(20). $(\\frac{1}{N_+} + \\frac{1}{N_+})$ should be $(\\frac{1}{N_-} + \\frac{1}{N_+})$.\n\n- Line 659, \"extension on\" -> \"extension to\".\n\n- Line 699, \"there remain little difference as the number of instances is differently used for eigen decomposition\". This is misleading, because the time cost changes a lot from 20.1s to 1.1s, when switching from \"FINE\" to \"FINE using 1% dataset\".\n\n- Overall, **the writing of this paper should be further improved**.\n\n\nTo summarize, although this paper has several issues, especially those about unclear descriptions and typos, this paper still has a few interesting points. In terms of data-selection, it achieves better performance than the previous works (e.g., CRUST, TopoFilter, etc.) on synthetic datasets. Therefore, I think this paper would potentially meet the standard of NeurIPS after addressing my concerns, and could be **conditionally** accepted.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed some of the limitations. Another limitation of this paper is that, the proposed method is subject to the degree of data imbalance.\n\nThis work does not have potential negative societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "18"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for identifying noisy examples by looking at their alignment with top eigenvector of the class specific feature covariance matrix. Reviewers have appreciated the simplicity of the method. Author responses were able to address the reviewers' concerns well and the paper is suitable to appear at the conference. 
Decision: Accept (Poster)