A Good Representation Detects Noisy Labels

1 INTRODUCTION . The generalization of deep neural networks ( DNNs ) depends on the quality and the quantity of the data . Nonetheless , in practice real-world datasets often contain label noise that challenges the above assumption ( Krizhevsky et al. , 2012 ; Zhang et al. , 2017 ; Agarwal et al. , 2016 ) . Employing human workers to cleaning annotations is one reliable way to improve the label quality , but it is too expensive and time-consuming for a large-scale dataset . One promising way to automatically cleaning up label errors is to first algorithmically detect possible label errors from a large-scale dataset ( Cheng et al. , 2021 ; Northcutt et al. , 2021a ; Pruthi et al. , 2020 ; Bahri et al. , 2020 ) , and then correct them using either algorithm or crowdsourcing ( Northcutt et al. , 2021b ) . Almost all the algorithmic detection approaches focus on designing customized training processes to learn with noisy labels , where the idea is to train DNNs with noisy supervisions and then make decisions based on the output ( Northcutt et al. , 2021a ) or gradients ( Pruthi et al. , 2020 ) of the last logit layer of the trained model . The high-level intuition of these methods is the memorization effects ( Han et al. , 2020 ) , i.e. , instances with label errors , a.k.a. , corrupted instances , tend to be harder to be learned by DNNs than clean instances ( Xia et al. , 2021 ; Liu et al. , 2020 ) . By setting appropriate hyperparameters to utilize the memorization effect , corrupted instances could be identified . The above methods suffer from two major limitations : 1 ) the customized training processes are task-specific and may require fine-tuning hyperparameters for different datasets/noise ; 2 ) as long as the model is trained with noisy supervisions , the memorization of corrupted instances exists . The model will “ subjectively ” and wrongly treat the memorized/overfitted corrupted instances as clean . For example , some low-frequency/rare clean instances may be harder to memorize than highfrequency/common corrupted instances . Memorizing these corrupted instances lead to unexpected and disparate impacts ( Liu , 2021 ) . One way to avoid memorizing/overfitting is to drop the depen- dency on training using the noisy supervisions , which motivates us to design a training-free method to find label errors . Intuitively , we can carefully use the information from nearby representations to infer whether one instance is corrupted or not . This observation inspired our solution of using a good representation to first identify “ neighbor ” instances for each training example , whose noisy labels jointly would help us perform the detection . Note the representation extractor could be fully independent of noisy supervisions by referring to unsupervised learning or self-supervised learning , or even adapting from other tasks . Our training-free method enables more possibilities beyond a better detection result . For example , the concerns of the required assumptions and hyperparameter tuning in those training-based methods will now be released due to our training-free property . The complexity of such a solution will also be much lower , again due to the removal of the possibly involved training processes . This light detection solution also has the potential to serve as a pre-processing module to prepare data for other sophisticated tasks ( e.g. , semi-supervised learning ( Xie et al. , 2019 ; Berthelot et al. , 2019 ) ) . Our main contributions are : 1 ) New perspective : Different from current methods that train customized models on noisy datasets , we proposed a training-free solution to efficiently detect noisy labels given good representations . We believe this is the first attempt of the same kind to the best of our knowledge . 2 ) Efficient algorithms : Based on the neighborhood information , we propose two methods : a voting-based local detection method that only requires checking the noisy label consensuses of nearby representations , and a ranking-based global detection method that scores each instance by its likelihood of being clean and filters out a guaranteed percentage of instances with low scores as corrupted ones . 3 ) Theoretical analyses : We theoretically analyze how a good representation ( but possibly imperfect in practice ) affects the local voting and provide guidelines for tuning neighborhood size . We also prove the worst-case error bound for the ranking-based method . 4 ) Numerical findings : Our numerical experiments show three important messages : in label noise detection , i ) training with noisy supervisions may not be necessary ; ii ) representation layers tend to be more useful than the logit layers ; iii ) representations from other tasks or domains are helpful . 1.1 RELATED WORKS . Learning with noisy labels There are many other works that can detect corrupted instances ( a.k.a . sample selection ) in the literature , e.g. , ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ; Yao et al. , 2020 ; Wei et al. , 2020 ; Jiang et al. , 2020 ; Zhang et al. , 2021 ) , and its combination with semi-supervised learning ( Nguyen et al. , 2019 ; Li et al. , 2020b ; Cheng et al. , 2021 ) . Another line of works focus on designing robust loss functions to mitigate the effect of label noise , such as numerical methods ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Gong et al. , 2018 ; Amid et al. , 2019 ; Wang et al. , 2019 ; Shu et al. , 2020 ) and statistical methods ( Natarajan et al. , 2013 ; Liu & Tao , 2015 ; Patrini et al. , 2017 ; Liu & Guo , 2020 ; Xia et al. , 2019 ; 2021 ; Wei & Liu , 2021 ; Zhu et al. , 2021a ; Li et al. , 2021 ; Liu et al. , 2020 ) . All these methods require training DNNs with noisy supervisions . Pre-training or Self-supervised Learning Training DNNs from scratch may be time-consuming ( Goyal et al. , 2017 ) . A common strategy to alleviate this issue is initializing DNN with a pre-trained model when training on new tasks ( Krizhevsky et al. , 2012 ; Zeiler & Fergus , 2014 ; Sermanet et al. , 2013 ) . The pre-trained model can be used for down-stream tasks due to the great transfer-ability of DNNs ( Tan et al. , 2018 ) . Popular pre-trained models include BERT ( Devlin et al. , 2018 ) for language tasks or CLIP ( Radford et al. , 2021 ) for vision-language tasks . Recently , unsupervised learning ( Ji et al. , 2019 ) and self-supervised learning ( Jaiswal et al. , 2021 ; Liu et al. , 2021 ; He et al. , 2020 ; Chen et al. , 2020 ) also exhibit great power when fine-tuning DNN on down-stream tasks . Label aggregation Our work is also relevant to the literature of crowdsourcing that focuses on label aggregation ( to clean the labels ) ( Liu et al. , 2012 ; Karger et al. , 2011 ; 2013 ; Liu & Liu , 2015 ; Zhang et al. , 2014 ) . Most of these works can access multiple reports ( labels ) for the same input feature , while our real-world datasets usually have only one noisy label for each feature . 2 PRELIMINARIES . Instances Traditional classification problems with perfect supervisions build on a clean dataset D : = { ( xn , yn ) } n∈ [ N ] , where [ N ] : = { 1 , 2 , · · · , N } . Each clean instance ( xn , yn ) includes feature xn and clean label yn , which is drawn according to random variables ( X , Y ) ∼ D. In a more practical case , the clean labels are likely to be unavailable and the learner could only observe a noisy dataset denoted by D̃ : = { ( xn , ỹn ) } n∈ [ N ] , where ( xn , ỹn ) is a noisy instance and the noisy label ỹn may or may not be identical to yn . We call ỹn is corrupted if ỹn 6= yn and clean otherwise . The instance ( xn , ỹn ) is a corrupted instance if ỹn is corrupted . The noisy data distribution corresponds to D̃ is ( X , Ỹ ) ∼ D̃ . We focus on the closed-set label noise that Y and Ỹ are assumed to be in the same label space , e.g. , Y , Ỹ ∈ [ K ] for a K-class classification task . Explorations on open-set data ( Xia et al. , 2020a ; Luo et al. , 2021 ) are deferred to future works . Representations The representation of feature xn is denoted by x̄n : = g ( xn ) , where g ( · ) denotes a representation extractor . Feature xn could be in any shape and representation x̄n is generally a high-dimensional vector . In this paper , we focus on a setting where the distances between two representations should be comparable or clusterable ( Zhu et al. , 2021b ) , i.e. , nearby representations should belong to the same true class with a high probability ( Gao et al. , 2016 ) , which could be defined as ( k , δk ) label clusterability in Definition 1 . Definition 1 ( ( k , δk ) label clusterability ) . Given a representation extractor g ( · ) , a dataset D satisfies ( k , δk ) label clusterability if : ∀n ∈ [ N ] , the representation x̄n : = g ( xn ) and its k-NearestNeighbors ( k-NN ) x̄n1 , · · · , x̄nk belong to the same true class with probability at least 1− δk . Note δk captures two types of randomnesses : one comes from a probabilistic Y |X , i.e. , ∃i ∈ [ K ] , P ( Y = i|X ) /∈ { 0 , 1 } ; the other depends on the quality of representation and the value of k , which will be further illustrated in Figure 2 . The ( k , 0 ) label clusterability is also known as k-NN label clusterability ( Zhu et al. , 2021b ) . It has been shown in ( Zhu et al. , 2021b ) that k-NN label clusterability can often be satisfied for a reasonable representation when k is moderate to small . We define k perfect extractors/representations in Definition 2 . Definition 2 ( k perfect extractor/representation ) . A representation extractor g ( · ) is called k perfect extractor if it induces k-NN label clusterability . Its outputs are called k perfect representations . Label noise detection Our paper aims to improve the performance of the label noise detection ( a.k.a , detecting noisy/corrupted labels , finding label errors ) which is measured by the F1-score of the detected corrupted instances , which is the harmonic mean of the precision and recall , i.e . F1 = 2/ ( Precision −1 + Recall−1 ) . Let 1 ( · ) be the indicator function that takes value 1 when the specified condition is satisfied and 0 otherwise . Let vn = 1 indicate that ỹn is detected as a corrupted label , and vn = 0 if ỹn is detected to be clean . Then the precision and recall can be calculated as Precision = ∑ n∈ [ N ] 1 ( vn=1 , ỹn 6=yn ) ∑ n∈ [ N ] 1 ( vn=1 ) , Recall = ∑ n∈ [ N ] 1 ( vn=1 , ỹn 6=yn ) ∑ n∈ [ N ] 1 ( ỹn 6=yn ) .

Summary: This paper proposes a method to detect noisy labels given good representations (e.g., ones pre-trained by contrastive learning approaches).  The proposed noisy-label detection method uses  the neighborhood information defined by a good set of representations in two ways:   1) checks the noisy label consensuses of nearby representations  2) scores each instance by its likelihood of being clean and filters out a guaranteed percentage of instances with low scores as corrupted ones   The work provide definitions for good representation and further proves a worst-case error bound for the ranking-based method given a 'good enough' representation. It also provides empirical results for its proposed method.