A Good Representation Detects Noisy Labels

1 INTRODUCTION . The generalization of deep neural networks ( DNNs ) depends on the quality and the quantity of the data . Nonetheless , in practice real-world datasets often contain label noise that challenges the above assumption ( Krizhevsky et al. , 2012 ; Zhang et al. , 2017 ; Agarwal et al. , 2016 ) . Employing human workers to cleaning annotations is one reliable way to improve the label quality , but it is too expensive and time-consuming for a large-scale dataset . One promising way to automatically cleaning up label errors is to first algorithmically detect possible label errors from a large-scale dataset ( Cheng et al. , 2021 ; Northcutt et al. , 2021a ; Pruthi et al. , 2020 ; Bahri et al. , 2020 ) , and then correct them using either algorithm or crowdsourcing ( Northcutt et al. , 2021b ) . Almost all the algorithmic detection approaches focus on designing customized training processes to learn with noisy labels , where the idea is to train DNNs with noisy supervisions and then make decisions based on the output ( Northcutt et al. , 2021a ) or gradients ( Pruthi et al. , 2020 ) of the last logit layer of the trained model . The high-level intuition of these methods is the memorization effects ( Han et al. , 2020 ) , i.e. , instances with label errors , a.k.a. , corrupted instances , tend to be harder to be learned by DNNs than clean instances ( Xia et al. , 2021 ; Liu et al. , 2020 ) . By setting appropriate hyperparameters to utilize the memorization effect , corrupted instances could be identified . The above methods suffer from two major limitations : 1 ) the customized training processes are task-specific and may require fine-tuning hyperparameters for different datasets/noise ; 2 ) as long as the model is trained with noisy supervisions , the memorization of corrupted instances exists . The model will “ subjectively ” and wrongly treat the memorized/overfitted corrupted instances as clean . For example , some low-frequency/rare clean instances may be harder to memorize than highfrequency/common corrupted instances . Memorizing these corrupted instances lead to unexpected and disparate impacts ( Liu , 2021 ) . One way to avoid memorizing/overfitting is to drop the depen- dency on training using the noisy supervisions , which motivates us to design a training-free method to find label errors . Intuitively , we can carefully use the information from nearby representations to infer whether one instance is corrupted or not . This observation inspired our solution of using a good representation to first identify “ neighbor ” instances for each training example , whose noisy labels jointly would help us perform the detection . Note the representation extractor could be fully independent of noisy supervisions by referring to unsupervised learning or self-supervised learning , or even adapting from other tasks . Our training-free method enables more possibilities beyond a better detection result . For example , the concerns of the required assumptions and hyperparameter tuning in those training-based methods will now be released due to our training-free property . The complexity of such a solution will also be much lower , again due to the removal of the possibly involved training processes . This light detection solution also has the potential to serve as a pre-processing module to prepare data for other sophisticated tasks ( e.g. , semi-supervised learning ( Xie et al. , 2019 ; Berthelot et al. , 2019 ) ) . Our main contributions are : 1 ) New perspective : Different from current methods that train customized models on noisy datasets , we proposed a training-free solution to efficiently detect noisy labels given good representations . We believe this is the first attempt of the same kind to the best of our knowledge . 2 ) Efficient algorithms : Based on the neighborhood information , we propose two methods : a voting-based local detection method that only requires checking the noisy label consensuses of nearby representations , and a ranking-based global detection method that scores each instance by its likelihood of being clean and filters out a guaranteed percentage of instances with low scores as corrupted ones . 3 ) Theoretical analyses : We theoretically analyze how a good representation ( but possibly imperfect in practice ) affects the local voting and provide guidelines for tuning neighborhood size . We also prove the worst-case error bound for the ranking-based method . 4 ) Numerical findings : Our numerical experiments show three important messages : in label noise detection , i ) training with noisy supervisions may not be necessary ; ii ) representation layers tend to be more useful than the logit layers ; iii ) representations from other tasks or domains are helpful . 1.1 RELATED WORKS . Learning with noisy labels There are many other works that can detect corrupted instances ( a.k.a . sample selection ) in the literature , e.g. , ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ; Yao et al. , 2020 ; Wei et al. , 2020 ; Jiang et al. , 2020 ; Zhang et al. , 2021 ) , and its combination with semi-supervised learning ( Nguyen et al. , 2019 ; Li et al. , 2020b ; Cheng et al. , 2021 ) . Another line of works focus on designing robust loss functions to mitigate the effect of label noise , such as numerical methods ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Gong et al. , 2018 ; Amid et al. , 2019 ; Wang et al. , 2019 ; Shu et al. , 2020 ) and statistical methods ( Natarajan et al. , 2013 ; Liu & Tao , 2015 ; Patrini et al. , 2017 ; Liu & Guo , 2020 ; Xia et al. , 2019 ; 2021 ; Wei & Liu , 2021 ; Zhu et al. , 2021a ; Li et al. , 2021 ; Liu et al. , 2020 ) . All these methods require training DNNs with noisy supervisions . Pre-training or Self-supervised Learning Training DNNs from scratch may be time-consuming ( Goyal et al. , 2017 ) . A common strategy to alleviate this issue is initializing DNN with a pre-trained model when training on new tasks ( Krizhevsky et al. , 2012 ; Zeiler & Fergus , 2014 ; Sermanet et al. , 2013 ) . The pre-trained model can be used for down-stream tasks due to the great transfer-ability of DNNs ( Tan et al. , 2018 ) . Popular pre-trained models include BERT ( Devlin et al. , 2018 ) for language tasks or CLIP ( Radford et al. , 2021 ) for vision-language tasks . Recently , unsupervised learning ( Ji et al. , 2019 ) and self-supervised learning ( Jaiswal et al. , 2021 ; Liu et al. , 2021 ; He et al. , 2020 ; Chen et al. , 2020 ) also exhibit great power when fine-tuning DNN on down-stream tasks . Label aggregation Our work is also relevant to the literature of crowdsourcing that focuses on label aggregation ( to clean the labels ) ( Liu et al. , 2012 ; Karger et al. , 2011 ; 2013 ; Liu & Liu , 2015 ; Zhang et al. , 2014 ) . Most of these works can access multiple reports ( labels ) for the same input feature , while our real-world datasets usually have only one noisy label for each feature . 2 PRELIMINARIES . Instances Traditional classification problems with perfect supervisions build on a clean dataset D : = { ( xn , yn ) } n∈ [ N ] , where [ N ] : = { 1 , 2 , · · · , N } . Each clean instance ( xn , yn ) includes feature xn and clean label yn , which is drawn according to random variables ( X , Y ) ∼ D. In a more practical case , the clean labels are likely to be unavailable and the learner could only observe a noisy dataset denoted by D̃ : = { ( xn , ỹn ) } n∈ [ N ] , where ( xn , ỹn ) is a noisy instance and the noisy label ỹn may or may not be identical to yn . We call ỹn is corrupted if ỹn 6= yn and clean otherwise . The instance ( xn , ỹn ) is a corrupted instance if ỹn is corrupted . The noisy data distribution corresponds to D̃ is ( X , Ỹ ) ∼ D̃ . We focus on the closed-set label noise that Y and Ỹ are assumed to be in the same label space , e.g. , Y , Ỹ ∈ [ K ] for a K-class classification task . Explorations on open-set data ( Xia et al. , 2020a ; Luo et al. , 2021 ) are deferred to future works . Representations The representation of feature xn is denoted by x̄n : = g ( xn ) , where g ( · ) denotes a representation extractor . Feature xn could be in any shape and representation x̄n is generally a high-dimensional vector . In this paper , we focus on a setting where the distances between two representations should be comparable or clusterable ( Zhu et al. , 2021b ) , i.e. , nearby representations should belong to the same true class with a high probability ( Gao et al. , 2016 ) , which could be defined as ( k , δk ) label clusterability in Definition 1 . Definition 1 ( ( k , δk ) label clusterability ) . Given a representation extractor g ( · ) , a dataset D satisfies ( k , δk ) label clusterability if : ∀n ∈ [ N ] , the representation x̄n : = g ( xn ) and its k-NearestNeighbors ( k-NN ) x̄n1 , · · · , x̄nk belong to the same true class with probability at least 1− δk . Note δk captures two types of randomnesses : one comes from a probabilistic Y |X , i.e. , ∃i ∈ [ K ] , P ( Y = i|X ) /∈ { 0 , 1 } ; the other depends on the quality of representation and the value of k , which will be further illustrated in Figure 2 . The ( k , 0 ) label clusterability is also known as k-NN label clusterability ( Zhu et al. , 2021b ) . It has been shown in ( Zhu et al. , 2021b ) that k-NN label clusterability can often be satisfied for a reasonable representation when k is moderate to small . We define k perfect extractors/representations in Definition 2 . Definition 2 ( k perfect extractor/representation ) . A representation extractor g ( · ) is called k perfect extractor if it induces k-NN label clusterability . Its outputs are called k perfect representations . Label noise detection Our paper aims to improve the performance of the label noise detection ( a.k.a , detecting noisy/corrupted labels , finding label errors ) which is measured by the F1-score of the detected corrupted instances , which is the harmonic mean of the precision and recall , i.e . F1 = 2/ ( Precision −1 + Recall−1 ) . Let 1 ( · ) be the indicator function that takes value 1 when the specified condition is satisfied and 0 otherwise . Let vn = 1 indicate that ỹn is detected as a corrupted label , and vn = 0 if ỹn is detected to be clean . Then the precision and recall can be calculated as Precision = ∑ n∈ [ N ] 1 ( vn=1 , ỹn 6=yn ) ∑ n∈ [ N ] 1 ( vn=1 ) , Recall = ∑ n∈ [ N ] 1 ( vn=1 , ỹn 6=yn ) ∑ n∈ [ N ] 1 ( ỹn 6=yn ) .

Summary: The authors propose a training-free approach to detect samples with noisy labels by leveraging representations learned from pre-trained models. The models can be obtained by either supervised or self-supervised pre-training. The authors then argue that samples in the pre-trained manifold should be closer if they share the same clean label, therefore one can use (a) local voting or (b) ranking methods to detect samples with corrupted labels. Experiments on CIFAR10/100 show improvement over other learning-based approaches (CORES, CL, TracIn).