FINE Samples for Learning with Noisy Labels

1 Introduction . Deep neural networks ( DNNs ) have achieved remarkable success in numerous tasks as the amount of accessible data has dramatically increased [ 21 , 15 ] . On the other hand , accumulated datasets are typically labeled by a human , a labor-intensive job or through web crawling [ 48 ] so that they may be easily corrupted ( label noise ) in real-world situations . Recent studies have shown that deep neural networks have the capacity to memorize essentially any labeling of the data [ 49 ] . Even a small amount of such noisy data can hinder the generalization of DNNs owing to their strong memorization of noisy labels [ 49 , 29 ] . Hence , it becomes crucial to train DNNs that are robust to corrupted labels . As label noise problems may appear anywhere , such robustness increases reliability in many applications such as the e-commerce market [ 9 ] , medical fields [ 45 ] , on-device AI [ 46 ] , and autonomous driving systems [ 11 ] . To improve the robustness against noisy data , the methods for learning with noisy labels ( LNL ) have been evolving in two main directions [ 18 ] : ( 1 ) designing noise-robust objective functions or regular- ∗Equal contribution 1Code available at https : //github.com/Kthyeon/FINE_official 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . izations and ( 2 ) detecting and cleansing the noisy data . In general , the former noise-robust direction uses explicit regularization techniques [ 6 , 52 , 50 ] or robust loss functions [ 38 , 13 , 40 , 51 ] , but their performance is far from state-of-the-art [ 49 , 26 ] on datasets with severe noise rates . Recently , researchers have designed noise-cleansing algorithms focused on segregating the clean data ( i.e. , samples with uncorrupted labels ) from the corrupted data [ 19 , 14 , 47 , 18 , 32 , 42 ] . One of the popular criteria for the segregation process is the loss value between the prediction of the noisy classifier and its noisy label , where it is generally assumed that the noisy data have a large loss [ 19 , 14 , 47 , 18 ] or the magnitude of the gradient during training [ 51 , 40 ] . However , these methods may still be biased by the corrupted linear classifier towards label noise because their criterion ( e.g. , loss values or weight gradient ) uses the posterior information of such a linear classifier [ 24 ] . Maennel et al . [ 31 ] analytically showed that the principal components of the weights of a neural network align with the randomly labeled data ; this phenomenon can yield more negative effects on the classifier as the number of randomly labeled classes increases . Recently , Wu et al . [ 42 ] used an inherent geometric structure induced by nearest neighbors ( NN ) in latent space and filtered out isolated data in such topology , and its quality was sensitive to its hyperparameters regarding NN clustering in the presence of severe noise rates . To mitigate such issues for label noise detectors , we provide a novel yet simple detector framework , filtering noisy labels via their eigenvectors ( FINE ) with theoretical guarantees to provide a high-quality splitting of clean and corrupted examples ( without the need to estimate noise rates ) . Instead of using the neural network ’ s linear classifier , FINE utilizes the principal components of latent representations made by eigen decomposition which is one of the most widely used unsupervised learning algorithms and separates clean data and noisy data by these components ( Figure 1a ) . To motivate our approach , as Figure 1b shows , we find that the clean data ( blue points ) are mainly aligned on the principal component ( black dotted line ) , whereas the noisy data ( orange points ) are not ; thus , the dataset is well clustered with the alignment of representations toward the principal component by fitting them into Gaussian mixture models ( GMM ) . We apply our framework to various LNL methods : the sample selection approach , a semi-supervised learning ( SSL ) approach , and collaboration with noise-robust loss functions . The key contributions of this work are summarized as follows : • We propose a novel framework , termed FINE ( filtering noisy labels via their eigenvectors ) , for detecting clean instances from noisy datasets . FINE makes robust decision boundary for the high-order topological information of data in latent space by using eigen decomposition of their gram matrix . • We provide provable evidence that FINE allows a meaningful decision boundary made by eigenvectors in latent space . We support our theoretical analysis with various experimental results regarding the characteristics of the principal components extracted by our FINE detector . • We develop a simple sample-selection method by replacing the existing detector method with FINE . We empirically validate that a sample-selection learning with FINE provides consistently superior detection quality and higher test accuracy than other existing alternative methods such as the Co-teaching family [ 14 , 47 ] , TopoFilter [ 42 ] , and CRUST [ 32 ] . • We experimentally show that our detection framework can be applied in various ways to existing LNL methods and validate that ours consistently improves the generalization in the presence of noisy data : sample-selection approach [ 14 , 47 ] , SSL approach [ 25 ] , and collaboration with noise-robust loss functions [ 51 , 40 , 29 ] . Organization . The remainder of this paper is organized as follows . In Section 2 , we discuss the recent literature on LNL solutions and meaningful detectors . In Section 3 , we address our motivation for creating a noisy label detector with theoretical insights and provide our main method , filtering the noisy labels via their eigenvectors ( FINE ) . In Section 4 , we present the experimental results . Finally , Section 5 concludes the paper . 2 Related Works . Zhang et al . [ 49 ] empirically showed that any convolutional neural networks trained using stochastic gradient methods easily fit a random labeling of the training data . To tackle this issue , numerous works have examined the classification task with noisy labels . We do not consider the works that assumed the availability of small subsets of training data with clean labels [ 17 , 36 , 39 , 53 , 3 ] . Noise-Cleansing-based Approaches . Noise-cleansing methods have evolved following the improvement of noisy detectors . Han et al . [ 14 ] suggested a noisy detection approach , named coteaching , that utilizes two networks , extracts subsets of instances with small losses from each network , and trains each network with subsets of instances filtered by another network . Yu et al . [ 47 ] combined a disagreement training procedure with co-teaching , which only selects instances predicted differently by two networks . Huang et al . [ 18 ] provided a simple noise-cleansing framework , training-filtering-training ; the empirical efficacy was improved by first finding label errors , then training the model only on data predicted as clean . Recently , new noisy detectors with theoretical support have been developed . Wu et al . [ 42 ] proposed a method called TopoFilter that filters noisy data by utilizing the k-nearest neighborhood algorithm and Euclidean distance between pre-logits . Mirzasoleiman et al . [ 32 ] introduced an algorithm that selects subsets of clean instances that provide an approximately low-rank Jacobian matrix and proved that gradient descent applied to the subsets prevents overfitting to noisy labels . Pleiss et al . [ 34 ] proposed an area under margin ( AUM ) statistic that measures the average difference between the logit values of the assigned class and its highest non-assigned class to divide clean and noisy samples . Cheng et . al [ 8 ] progressively filtered out corrupted instances using a novel confidence regularization term . The noise-cleansing method was also developed in a semi-supervised learning ( SSL ) manner . Li et al . [ 25 ] modeled the per-sample loss distribution and divide it into a labeled set with clean samples and an unlabeled set with noisy samples , and they leverage the noisy samples through the well-known SSL technique MixMatch [ 4 ] . Noise-Robust Models . Noise-robust models have been studied in the following directions : robustloss functions , regularizations , and strategies . First , for robust-loss functions , Ghosh et al . [ 13 ] showed that the mean absolute error ( MAE ) might be robust against noisy labels . Zhang & Sabuncu et al . [ 51 ] argued that MAE performed poorly with DNNs and proposed a GCE loss function , which can be seen as a generalization of MAE and cross-entropy ( CE ) . Wang et al . [ 40 ] introduced the reverse version of the cross-entropy term ( RCE ) and suggested that the SCE loss function is a weighted sum of the CE and RCE . Some studies have stated that the early-stopped model can prevent the memorization phenomenon for noisy labels [ 2 , 49 ] and theoretically analyzed it [ 26 ] . Based on this hypothesis , Liu et al . [ 29 ] proposed an early-learning regularization ( ELR ) loss function to prohibit memorizing noisy data by leveraging the semi-supervised learning techniques . Xia et al . [ 43 ] clarified which neural network parameters cause memorization and proposed a robust training strategy for these parameters . Efforts have been made to develop regularizations on the prediction level by smoothing the one-hot vector [ 30 ] , using linear interpolation between data instances [ 50 ] , and distilling the rescaled prediction of other models [ 20 ] . However , these works have limitations in terms of performance as the noise rate of the dataset increases . Dataset Resampling . Label-noise detection may be a category of data resampling which is a common technique in the machine learning community that extracts a “ helpful ” dataset from the distribution of the original dataset to remove the dataset bias . In class-imbalance tasks , numerous studies have conducted over-sampling of minority classes [ 7 , 1 ] or undersampling the majority classes [ 5 ] to balance the amount of data per class . Li & Vasconcelos et al . [ 27 ] proposed a resampling procedure to reduce the representation bias of the data by learning a weight distribution that favors difficult instances for a given feature representation . Le Bras et al . [ 22 ] suggested an adversarial filteringbased approach to remove spurious artifacts in a dataset . Analogously , in anomaly detection and Algorithm 1 : FINE Algorithm for Sample Selection INPUT : Noisy training data D , feature extractor g , number of classes K , clean probability threshold ζ , set of FINE scores for class k Fk OUTPUT : Collected clean data C 1 : Initialize C ← ∅ , D̂ ← D , Σk ← 0 for all k = 1 , . . . , K / * Update the convariance matrices for all classes * / 2 : for ( xi , yi ) ∈ D do 3 : zi ← g ( xi ) 4 : Update the gram matrix Σyi ← Σyi + ziz > i 5 : end for / * Generate the principal component with eigen decomposition * / 6 : for k = 1 , . . . , K do 7 : Uk , Λk ← EIGEN DECOMPOSITION OF Σk 8 : uk ← THE FIRST COLUMN OF Uk 9 : end for / * Compute the alignment score and get clean subset C * / 10 : for ( xi , yi ) ∈ D do 11 : Compute the FINE score fi = 〈uyi , zi〉 2 and Fyi ← Fyi ∪ { fi } 12 : end for / * Finding the samples whose clean probability is larger than ζ * / 13 : C ← C∪ GMM ( Fk , ζ ) for all k = 1 , . . . , K out-of-distribution detection problems [ 16 , 28 , 23 ] , the malicious data are usually detected by examining the loss value or negative behavior in the feature representation space . While our research is motivated by such previous works , this paper focuses on the noisy image classification task .

Summary: This paper proposes a new method for learning with noisy labels. The proposed method belongs to the research direction of data-selection, and aims to better select the clean data out of noisy samples. The key component of the proposed method is a novel operation, named _filtering noisy labels via their eigenvectors_ (FINE). FINE first computes the covariance matrix of the feature representation for the noisy samples. Then the principal eigenvector is extracted from the covariance matrix via eigen decomposition. Afterwards, the similarity score between the feature vector of each sample and the eigenvector is computed. Finally, the computed similarly scores are used to fit a Gaussian Mixture Model, which predicts the probability that a sample is clean or noisy. The proposed FINE approach is simple and computationally feasible. Theoretically it is shown that FINE is able to approximate the eigenvector of clean data with guaranteed accuracy under a few reasonable assumptions. In practice, FINE is demonstrated to be effective by integrating it into three different kinds of label noise learning approaches. The experimental results show that FINE brings (much) performance gain over previous methods across several synthetic and real-world noisy datasets.  To summarize, the contributions of this paper include:  1. A new method, named FINE, is developed to better select the clean data out of noisy dataset. Theoretically it is shown that FINE is able to approximate the true eigenvector of clean data with guaranteed upperbound. Also, it is shown that the data selection accuracy of FINE has guaranteed lower bound in terms of precision and recall.  2. This paper also includes a few illustrative experiments to explain the characteristics of the proposed method. These experiments are helpful.  3. In practice, FINE is integrated into three different kinds of label noise learning approaches, and (significant) performance gain is observed over different datasets, involving the synthetic and real-world noisy data.