Class2Simi: A New Perspective on Learning with Label Noise

1 INTRODUCTION . It is expensive to label large-scale data accurately . Therefore , cheap datasets with label noise are ubiquitous in the era of big data . However , label noise will degenerate the performance of trained deep models , because deep networks will easily overfit label noise ( Zhang et al. , 2017 ; Zhong et al. , 2019 ; Li et al. , 2019 ; Yi & Wu , 2019 ; Zhang et al. , 2019 ; 2018 ; Xia et al. , 2019 ; 2020 ) . In this paper , we propose a new perspective on handling label noise called “ Class2Simi ” , i.e. , transforming training examples with noisy class labels into pairs of examples with noisy similarity labels . A class label shows the class that an instance belongs to , while a similarity label indicates whether or not two instances belong to the same class . This transformation is motivated by the observation that the noise rate becomes lower , e.g. , even if two instances have incorrect class labels , their similarity label could be correct . In the label-noise learning community , a lower noise rate usually results in higher classification performance ( Han et al. , 2018b ; Patrini et al. , 2017 ) . Specifically , we illustrate the transformation and the robustness of similarity labels in Figure 1 . Assume we have eight noisy examples { ( x1 , ȳ1 ) , . . . , ( x8 , ȳ8 ) } as shown in the upper part of the middle column . Their labels are of four classes , i.e. , { 1 , 2 , 3 , 4 } . The labels marked in red are incorrect labels . We transform the 8 examples into 8× 8 example-pairs with noisy similarity labels as shown in the bottom part of the middle column , where the similarity label 1 means the two instances have the same class label and 0 means the two instances have different class labels . We present the latent clean class labels and similarity labels in the left column . In the middle column , we can see that although the instances x2 and x4 both have incorrect class labels , the similarity label of the example-pair ( x2 , x4 ) is correct . Similarity labels are robust because they further consider the information on the pairwise relationship . We prove that the noise rate in the noisy similarity labels is lower than that of the noisy class labels . For example , if we assume that the noisy class labels in Figure 1 are generated according to the latent clean labels and the transition matrix shown in the upper part of the right column ( the ij-th entry of the matrix denotes the probability that the clean class label i flips into the noisy class label j ) , the noise rate for the noisy class labels is 0.5 while the rate for the corresponding noisy similarity labels is 0.25 . Note that the noise rate is the ratio of the number of incorrect labels to the number of total examples , which can be calculated from the noise transition matrix combined with the proportion of each class , i.e. , 1/6× 3/4 + 1/2× 1/4 = 0.25 . It is obvious that Class2Simi suffers information loss because we can not recover the class labels from similarity labels . However , since the similarity labels are more robust to noise than the class labels , the advantage of the reduction of noise rate overweighs the disadvantage of the loss of information . Intuitively , in the learning process , it is the signal in the information that enhances the performance of the model , while the noise in the information is harmful to the model . Through Class2Simi , although the total amount of information is reduced , the signal to noise ratio is increased , and so would be the total amount of signals . Thus , we can benefit from the transformation and achieve better performances . Theorem 2 and the experimental results will verify the effectiveness of this transformation . It remains unsolved how to learn a robust classifier from the data with transformed noisy similarity labels . To solve this problem , we first estimate the similarity noise transition matrix , a 2 × 2 matrix whose entries denote the flip rates of similarity labels . Note that the transition matrix bridges the noisy similarity posterior and the clean similarity posterior . The noisy similarity posterior can be learned from the data with noisy similarity labels . Then , given the similarity noise transition matrix , we can infer the clean similarity posterior from the noisy similarity posterior . Since the clean similarity posterior is approximated by the inner product of the clean class posterior ( Hsu et al. , 2019 ) , the clean class posterior ( and thus the robust classifier ) can thereby be learned . We will empirically show that Class2Simi with the estimated similarity noise transition matrix will remarkably outperform the baselines even given with the ground-truth class noise transition matrix . The contributions of this paper are summarized as follows : • We propose a new perspective on learning with label noise , which transforms class labels into similarity labels . Such a transformation reduces the noise level . • We provide a way to estimate the similarity noise transition matrix by theoretically establishing its relation to the class noise transition matrix . We show that even if the class noise transition matrix is inaccurately estimated , the induced similarity noise transition matrix still works well . • We design a deep learning method to learn robust classifiers from data with noisy similarity labels and theoretically analyze its generalization ability . • We empirically demonstrate that the proposed method remarkably surpasses the baselines on many datasets with both synthetic noise and real-world noise . The rest of this paper is organized as follows . In Section 2 , we formalize the noisy multi-class classification problem , and in Section 3 , we propose the Class2Simi strategy and practical implementation . Experimental results are discussed in Section 4 . We conclude our paper in Section 5 . 2 PROBLEM SETUP AND RELATED WORK . Let ( X , Y ) ∈ X × { 1 , . . . , C } be the random variables for instances and clean labels , where X represents the instance space and C is the number of classes . However , in many real-world applications ( Zhang et al. , 2017 ; Zhong et al. , 2019 ; Li et al. , 2019 ; Yi & Wu , 2019 ; Zhang et al. , 2019 ; Tanno et al. , 2019 ; Zhang et al. , 2018 ) , the clean labels can not be observed . The observed labels are noisy . Let Ȳ be the random variable for the noisy labels . What we have is a sample { ( x1 , ȳ1 ) , . . . , ( xn , ȳn ) } drawn from the noisy distribution Dρ of the random variables ( X , Ȳ ) . Our aim is to learn a robust classifier that could assign clean labels to test data by exploiting the sample with noisy labels . Existing methods for learning with noisy labels can be divided into two categories : algorithms that result in statistically inconsistent or consistent classifiers . Methods in the first category usually employ heuristics to reduce the side-effect of noisy labels , e.g. , selecting reliable examples ( Yu et al. , 2019 ; Han et al. , 2018b ; Malach & Shalev-Shwartz , 2017 ) , reweighting examples ( Ren et al. , 2018 ; Jiang et al. , 2018 ; Ma et al. , 2018 ; Kremer et al. , 2018 ; Tanaka et al. , 2018 ; Reed et al. , 2015 ) , employing side information ( Vahdat , 2017 ; Li et al. , 2017 ; Berthon et al. , 2020 ) , and adding regularization ( Han et al. , 2018a ; Guo et al. , 2018 ; Veit et al. , 2017 ; Vahdat , 2017 ; Li et al. , 2017 ) . Those methods empirically work well in many settings . Methods in the second category aim to learn robust classifiers that could converge to the optimal ones defined by using clean data . They utilize the noise transition matrix , which denotes the probabilities that the clean labels flip into noisy labels , to build consistent algorithms ( Goldberger & Ben-Reuven , 2017 ; Patrini et al. , 2017 ; Thekumparampil et al. , 2018 ; Yu et al. , 2018 ; Liu & Guo , 2020 ; Zhang & Sabuncu , 2018 ; Kremer et al. , 2018 ; Liu & Tao , 2016 ; Northcutt et al. , 2017 ; Scott , 2015 ; Natarajan et al. , 2013 ; Yao et al. , 2020b ) . The idea is that given the noisy class posterior probability and the noise transition matrix , the clean class posterior probability can be inferred . Note that the noisy class posterior and the noise transition matrix can be estimated by exploiting the noisy data , where the noise transition matrix additionally needs anchor points ( Liu & Tao , 2016 ; Patrini et al. , 2017 ) . Some methods assume anchor points have already been given ( Yu et al. , 2018 ) . There are also methods showing how to identify anchor points from the noisy training data ( Liu & Tao , 2016 ; Patrini et al. , 2017 ) . 3 CLASS2SIMI MEETS NOISY SUPERVISION . In this section , we propose a new strategy for learning from noisy data . Our core idea is to transform class labels to similarity labels first , and then handle the noise manifested on similarity labels . 3.1 TRANSFORMATION ON LABELS AND THE TRANSITION MATRIX . As in Figure 1 , we combine every 2 instances in pairs , and if the two instances have the same class label , we assign this pair a similarity label 1 , otherwise 0 . If the class labels are corrupted , the generated similarity labels also contain noise . We denote the clean and noisy similarity labels of the example-pair ( xi , xj ) by Hij and H̄ij respectively . The definition of the similarity noise transition matrix is similar to the class one , denoting the probabilities that clean similarity labels flip into noisy similarity labels , i.e. , Ts , mn = P ( H̄ij = n|Hij = m ) . The dimension of the similarity noise transition matrix is always 2× 2 . Since the similarity labels are generated from class labels , the similarity noise is also determined and , thus can be calculated , by the class noise transition matrix . Theorem 1 . Assume that the dataset is balanced ( each class has the same amount of samples , and c classes in total ) , and the noise is class-dependent . Given a class noise transition matrix Tc , such that Tc , ij = P ( Ȳ = j|Y = i ) . The elements of the corresponding similarity noise transition matrix Ts can be calculated as Ts,00 = c2 − c− ( ∑ j ( ∑ i Tc , ij ) 2 − ||Tc||2Fro ) c2 − c , Ts,01 = ∑ j ( ∑ i Tc , ij ) 2 − ||Tc||2Fro c2 − c , Ts,10 = c− ||Tc||2Fro c , Ts,11 = ||Tc||2Fro c . A detailed proof is provided in Appendix A . Remark 1 . Theorem 1 can easily extend to the setting where the dataset is unbalanced in classes by multiplying each Tc , ij by a coefficient ni . ni is the number of examples from the i-th class . Note that the similarity labels are only dependent on class labels . If the class noise is class-dependent , the similarity noise is also “ class-dependent ” ( class means similar and dissimilar ) . Under classdependent label noise , a binary classification is learnable as long as T00 + T11 > 1 ( Menon et al. , 2015 ) , where T is the corresponding binary transition matrix ; a multi-class classification is learnable if the corresponding transition matrix Tc is invertible . For Class2Simi , in the most general sense , i.e. , Tc is invertible , Ts,00 + Ts,11 > 1 holds . Namely , the learnability of the pointwise classification implies the learnability of the reduced pairwise classification . However , the latter can not imply the former . A proof and a counterexample are provided in Appendix F. Theorem 2 . Assume that the dataset is balanced ( each class has the same amount of samples ) , and the noise is class-dependent . When the number of classes c ≥ 81 , the noise rate for the noisy similarity labels is lower than that of the noisy class labels . A detailed proof is provided in Appendix B . When dealing with label noise , a low noise rate has many benefits . The most important one is that the noise-robust algorithms will consistently achieve higher performance when the noise rate is lower ( Bao et al. , 2018 ; Han et al. , 2018b ; Xia et al. , 2019 ; Patrini et al. , 2017 ) . Another benefit is that , when the noise rate is low , the complex instance-dependent label noise can be well approximated by class-dependent label noise ( Cheng et al. , 2020 ) , which is easier to handle . After the Class2Simi transformation , the number of dissimilar pairs is ( c − 1 ) times as much as that of similar pairs . Meanwhile , compared with the original noise rate of class labels , the noise rate of similar pairs ( the ratio of the number of mislabeled similar pairs to the number of total real similar pairs ) is higher and the noise rate of dissimilar pairs is lower , while the overall noise rate of pairwise examples is lower , which partially reflects that the impact of the label noise is less bad . Moreover , the flip from dissimilar to similar should be more adversarial and thus more important . In practice , it is common that one class has more than one clusters , while it is rare that two or more classes are in the same cluster . If there is a flip from similar to dissimilar and based on it we split a ( latent ) cluster into two ( latent ) clusters , we still have a high chance to label these two clusters correctly later . If there is a flip from dissimilar to similar and based on it we join two clusters belonging to two classes into a single cluster , we nearly have zero chance to label this cluster correctly later . As a consequence , the flip from dissimilar to similar is more adversarial , and thus more important . To sum up , considering the reduction of the overall noise rate is meaningful .

Summary: This paper presented a working framework for learning a robust classifier with noisy labels. It proves that if the number of the classes is more than 8 then the noise rate in the similarity matrix is less than that in the noise rate in labels. Hence after learning classifiers from noisy labels, it updates the classifier using entropy loss between predicted similarity matrix and actual similarity matrix calculated from similarity matrix from data multiplied with noisy transition matrix derived using existing techniques(Xia et al., 2019; Patrini et al., 2017).