Estimating Instance-dependent Label-noise Transition Matrix using DNNs

In label-noise learning , estimating the transition matrix is a hot topic as the matrix plays an important role in building statistically consistent classifiers . Traditionally , the transition from clean labels to noisy labels ( i.e. , clean label transition matrix ) has been widely exploited to learn a clean label classifier by employing the noisy data . Motivated by that classifiers mostly output Bayes optimal labels for prediction , in this paper , we study to directly model the transition from Bayes optimal labels to noisy labels ( i.e. , Bayes label transition matrix ) and learn a classifier to predict Bayes optimal labels . Note that given only noisy data , it is ill-posed to estimate either the clean label transition matrix or the Bayes label transition matrix . But favorably , Bayes optimal labels have less uncertainty compared with the clean labels , i.e. , the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not . This enables two advantages to estimate the Bayes label transition matrix , i.e. , ( a ) we could theoretically recover a set of noisy data with Bayes optimal labels under mild conditions ; ( b ) the feasible solution space is much smaller . By exploiting the advantages , we estimate the Bayes label transition matrix by employing a deep neural network in a parameterized way , leading to better generalization and superior classification performance . 1 INTRODUCTION . The study of classification in the presence of noisy labels has been of interest for three decades ( Angluin & Laird , 1988 ) , but becomes more and more important in weakly supervised learning ( Thekumparampil et al. , 2018 ; Li et al. , 2020b ; Guo et al. , 2018 ; Xiao et al. , 2015 ; Zhang et al. , 2017a ; Yang et al. , 2021b ; a ) . The main reason behind this is that datasets are becoming bigger and bigger . To improve annotation efficiency , these large-scale datasets are often collected from crowdsourcing platforms ( Yan et al. , 2014 ) , online queries ( Blum et al. , 2003 ) , and image engines ( Li et al. , 2017 ) , which suffer from unavoidable label noise ( Yao et al. , 2020a ) . Recent researches show that the label noise significantly degenerates the performance of deep neural networks , since deep models easily memorize the noisy labels ( Zhang et al. , 2017a ; Yao et al. , 2020a ) . Generally , the algorithms for combating noisy labels can be categorized into statistically inconsistent algorithms and statistically consistent algorithms . The statistically inconsistent algorithms are heuristic , such as selecting possible clean examples to train the classifier ( Han et al. , 2020 ; Yao et al. , 2020a ; Yu et al. , 2019 ; Han et al. , 2018b ; Malach & Shalev-Shwartz , 2017 ; Ren et al. , 2018 ; Jiang et al. , 2018 ) , re-weighting examples to reduce the effect of noisy labels ( Ren et al. , 2018 ) , correcting labels ( Ma et al. , 2018 ; Kremer et al. , 2018 ; Tanaka et al. , 2018 ; Reed et al. , 2015 ) , or adding regularization ( Han et al. , 2018a ; Guo et al. , 2018 ; Veit et al. , 2017 ; Vahdat , 2017 ; Li et al. , 2017 ; 2020b ; Wu et al. , 2020 ) . These approaches empirically work well , but there is no theoretical guarantee that the learned classifiers can converge to the optimal ones learned from clean data . To address this limitation , algorithms in the second category aim to design classifier-consistent algorithms ( Yu et al. , 2017 ; Zhang & Sabuncu , 2018 ; Kremer et al. , 2018 ; Liu & Tao , 2016 ; Northcutt et al. , 2017 ; Scott , 2015 ; Natarajan et al. , 2013 ; Goldberger & Ben-Reuven , 2017 ; Patrini et al. , 2017 ; Thekumparampil et al. , 2018 ; Yu et al. , 2018 ; Liu & Guo , 2020 ; Xu et al. , 2019 ; Xia et al. , 2020b ) , where classifiers learned on noisy data will asymptotically converge to the optimal classifiers defined on the clean domain . The label transition matrix T ( x ) plays an important role in building statistically consistent algorithms . Traditionally , the transition matrix T ( x ) is defined to relate clean distribution and noisy distribution , where T ( x ) = P ( Ỹ | Y , X = x ) andX denotes the random variable of instances/features , Ỹ as the variable for the noisy label , and Y as the variable for the clean label . The above matrix is denoted as the clean label transition matrix , which is widely used to learn a clean label classifier by employing the noisy data . The learned clean label classifier is expected to predict a probability distribution over a set of pre-defined classes given an input , i.e . clean class posterior probability P ( Y | X ) . The clean class posterior probability is the distribution from which clean labels are sampled . However , Bayes optimal labels Y ∗ , i.e. , the class labels that maximize the clean class posteriors Y ∗ | X : = argmaxY P ( Y | X ) , are mostly used as the predicted labels and for computing classification accuracy . Motivated by this , in this paper , we propose to directly model the transition matrix T ∗ ( x ) that relates Bayes optimal distribution and noisy distribution , i.e. , T ∗ ( x ) = P ( Ỹ | Y ∗ , X = x ) , where Y ∗ denotes the variable for Bayes optimal label . The Bayes optimal label classifier can be learned by exploiting the Bayes label transition matrix directly . Studying the transition between Bayes optimal distribution and noisy distribution is considered advantageous to that of studying the transition between clean distribution and noisy distribution . The main reason is due to that the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not . Two advantages can be introduced by this to better estimate the instance-dependent transition matrix : ( a ) We can collect a set of examples with theoretically guaranteed Bayes optimal labels out of noisy data . The intrinsic reason that Bayes optimal labels can be inferred from the noisy data while clean labels can not is that Bayes optimal labels are the labels that maximize the clean class posteriors while clean labels are sampled from the clean class posteriors . In the presence of label noise , the labels that maximize the noisy class posteriors could be identical to those that maximize the clean class posteriors ( Bayes optimal labels ) under mild conditions ( Cheng et al. , 2020 ) . Therefore some instances ’ Bayes optimal labels can be inferred from their noisy class posteriors while their clean labels are impossible to infer since the clean class posteriors are unobservable , as shown in Figure 1 . ( b ) The feasible solution space of the Bayes label transition matrix is much smaller than that of the clean label transition matrix . This is because that Bayes optimal labels have less uncertainty compared with the clean labels . The transition matrix defined by Bayes optimal labels and the noisy labels is therefore sparse and can be estimated more efficiently with the same amount of training data . These two advantages naturally motivate us to collect a set of examples with their theoretically guaranteed Bayes optimal labels out of the noisy data to learn to approximate the Bayes label transition matrix T ∗ ( x ) . Due to the high complexity of the instance-dependent matrix T ∗ ( x ) , we simplify its estimation by parameterizing it using a deep neural network . The collected examples , inferred Bayes optimal labels , and their noisy labels are served as data points to optimize the deep neural network to approximate the T ∗ ( x ) . Compared with the previous method ( Xia et al. , 2020a ) , which made assumptions and leveraged hand-crafted priors to approximate the instance-dependent transition matrices , we train a deep neural network to estimate the instance-dependent label transition matrix with a reduced feasible solution space , which achieves lower approximation error , better generalization , and superior classification performance . 2 RELATED WORK . Noise model . Currently , there are several typical label noise models . Specifically , the random classification noise ( RCN ) model assumes that clean labels flip randomly with a constant rate ( Biggio et al. , 2011 ; Manwani & Sastry , 2013 ; Natarajan et al. , 2013 ) . The class-conditional label noise ( CCN ) model assumes that the flip rate depends on the latent clean class ( Patrini et al. , 2017 ; Xia et al. , 2019 ; Ma et al. , 2018 ) . The instance-dependent label noise ( IDN ) model considers the most general case of label noise , where the flip rate depends on its instance/features ( Cheng et al. , 2020 ; Xia et al. , 2020a ; Zhu et al. , 2020 ) . Obviously , the IDN model is more realistic and applicable . For example , in real-world datasets , an instance whose feature contains less information or is of poor quality may be more prone to be labeled wrongly . The bounded instance dependent label noise ( BIDN ) ( Cheng et al. , 2020 ) is a reasonable extension of IDN , where the flip rates are dependent on instances but upper bounded by a value smaller than 1 . However , with only noisy data , it is a non-trivial task to model such realistic noise without any assumption ( Xia et al. , 2020a ) . This paper focuses on the challenging BIDN problem setting . Learning clean distributions . It is significant to reduce the side effect of noisy labels by inferring clean distributions statistically . The label transition matrix plays an important role in such an inference process , which is used to denote the probabilities that clean labels flip into noisy labels . We first review prior efforts under the class-dependent condition ( Patrini et al. , 2017 ) . By exploiting the class-dependent transition matrix T , the training loss on noisy data can be corrected . The transition matrix T can be estimated in many ways , e.g. , by introducing the anchor point assumption ( Liu & Tao , 2016 ) , by exploiting clustering ( Zhu et al. , 2021 ) , by minimizing volume of T ( Li et al. , 2021 ) , and by using extra clean data ( Hendrycks et al. , 2018 ; Shu et al. , 2020 ) . To make the estimation more accurately , a slack variable ( Xia et al. , 2019 ) or a multiplicative dual T ( Yao et al. , 2020b ) can be introduced to revise the transition matrix . As for the efforts on the instance-dependent transition matrix , existing methods rely on various assumptions , e.g. , the noise rate is bounded ( Cheng et al. , 2020 ) , the noise only depends on the parts of the instance ( Xia et al. , 2020a ) , and additional valuable information is available ( Berthon et al. , 2020 ) . Although the above advanced methods achieve superior performance empirically , the introduction of strong assumptions limit their applications in practice . In this paper , we propose to infer Bayes optimal distribution instead of clean distribution , as Bayes optimal distribution is less uncertain and easy to be inferred under mild conditions . Other approaches . Other methods exist with more sophisticated training frameworks or pipelines , including but not limited to robust loss functions ( Zhang & Sabuncu , 2018 ; Xu et al. , 2019 ; Liu & Guo , 2020 ) , sample selection ( Han et al. , 2018b ; Wang et al. , 2019 ; Lyu & Tsang , 2020 ) , label correction ( Tanaka et al. , 2018 ; Zhang et al. , 2021 ; Zheng et al. , 2020 ) , ( implicit ) regularization ( Xia et al. , 2021 ; Zhang et al. , 2017b ; Liu et al. , 2020 ) , and semi-supervised learning ( Li et al. , 2020a ; Nguyen et al. , 2020 ) .

Summary: The transition matrix plays a vital role in modeling label noise. Current methods focus on modeling the transition from clean labels to noisy labels. While this paper alternatively models the transition from Bayes optimal labels to noisy labels. Since we usually use the Bayes optimal labels for prediction. This transformation will not affect the practical use but makes the estimation of the matrix much easier.  Specifically, this paper designs a DNN to estimate the transition matrix. During training, the DNN can be optimized with the classifier simultaneously in an end-to-end manner.  Extensive experiments are conducted to support the proposed method.