An Information Fusion Approach to Learning with Instance-Dependent Label Noise

1 INTRODUCTION . Data labeling annotated from human efforts , such as crowdsourcing ( Yan et al. , 2014 ; Chen et al. , 2017 ) and online queries ( Divvala et al. , 2014 ) , may be heavily noisy in practice . To make it worse , the label noise stemmed from the human annotations is often instance-dependent . For example , the images closing to the decision boundary are usually prone to be mislabeled ( Zhang et al. , 2021 ; Zhu et al. , 2021b ) . On the other hand , the remarkable success of deep neural networks ( DNNs ) on supervised learning tasks heavily relies on the expressive power and a large number of data with accurate labels . Unfortunately , deep neural networks memorizes noisy labels leading to poor generalization ( Zhang et al. , 2017 ) . It is challenging to learn with practical instance-dependent label noise ( IDN ) due to the hidden and complicated label noise properties ( Zhu et al. , 2021a ; Liu , 2021 ) . The methods of dealing with noisy labels falls in two lines , including heuristically identifying noisy samples or statistical label noise modeling . The training of deep neural network often learns the clean labels first ( Arpit et al. , 2017 ) and memorizes the noisy labels gradually , which is recognized as the memorization effect . Based on the general memorization effects , the heuristical methods are all designed via following the anomaly detection strategy : identifying the noisy sample based on different behaviors ( e.g. , loss values ) between clean and noisy samples during training . The typical methods contain sample selection ( Yu et al. , 2019 ; Han et al. , 2018b ) , reweight samples ( Cheng et al. , 2021 ; Jiang et al. , 2018 ; Ren et al. , 2018 ) , label correction ( Ma et al. , 2018 ; Tanaka et al. , 2018 ) , and regularization ( Han et al. , 2018a ) . Although these algorithms empirically work well , without modeling the label noise explicitly , the reliability can not be guaranteed . Another line of works relies on noise transition matrix ( NTM ) to model the label noise statistically ( Xia et al. , 2019 ; 2020 ; Patrini et al. , 2017 ) . Namely , the NTM quantifies the probabilities that clean labels flip into noisy labels . Although the NTM-based methods possess theoretical guarantee , under IDN , the NTM estimation for each instance is pretty challenging . To ease the estimation , some unrealistic assumptions have to be posed the on NTM , including instance-independent transition matrix ( Liu & Guo , 2020 ; Wei & Liu , 2021 ; Li et al. , 2021 ) , symmetric transition matrix ( Menon et al. , 2018 ) , upper bounded noise rate ( Cheng et al. , 2020 ) , and part-dependent label noise ( Xia et al. , 2020 ) . However , under the complex IDN , the empirical noise distribution is highly different with underlying noise distribution . For example , in Figure 1 , the underlying and empirical noisy distribution for long-tail instance is different since the empirical noisy label is either the same or different with clean label . Additionally , observed noisy label provide inductive bias toward label corruption . In other word , the genuine problem causing from IDN is the empirical , instead of underlying , clean and noisy distribution mismatch problem . To mitigate empirical distribution mismatch problem , we propose the posterior transition matrix ( PTM ) , posteriorly modeling label noise given the observed noisy labels . We adopt PTM to provably bridge the gap between clean and noisy underlying data distribution , and anchor points ( i.e. , data points that belong to a specific class almost surely ) empirical distribution simultaneously . We also provide an easy-to-compute PTM estimation under specific condition of low label noise . To extend the applicability to a wide range of label noise , motivated by Kalman filtering ( Kalman , 1960 ) , we proposed the information fusion ( IF ) method linearly combining the estimated noise and PTM , which achieves lower transition matrix estimation error . We empirically show that proposed method IF can achieve higher accuracy and more stable training . The main contributions of this work are summarized below . • We propose a new concept , named PTM , achieving consistent classifier for underlying distribution and anchor point empirical distribution mismatch simultaneously . • We propose the simple PTM estimation method based on observed noisy label under low label noise . To extend the applicability , we propose IF method , linearly combining the estimated NTM and PTM , to achieve lower estimation error with theoretical and experimental justification . • We experimentally demonstrate the superiority of proposed IF method achieving higher accuracy and more stable training under synthetic and real-world label noise . 2 PRELIMINARIES . Throughout this paper , we adopt uppercase letters to denote random variables , and lowercase letters to denote particular realization of the random variables . We target the c-class classification problem with instance-dependent ( label ) noise ( IDN ) . Let P ( X , Y ) be the underlying clean distribution of the random variables ( X , Y ) , where X and Y represent the instance and clean label . However , in real-world scenario , the clean label is corrupted and thus only noisy label , denoted as Ỹ , is observed . Given a set of N training instances denoted by D̃ : = { ( xn , ỹn ) } Nn=1 , where xn is the instance vector of the n-th sample , ỹn ∈ [ c ] : = { 1 , · · · , c } is the corresponding observed but noisy label , our goal is to predict the clean label yn for any given instance xn . Data distribution We assume that the unobserved clean samples ( xn , yn ) and available noisy samples ( xn , ỹn ) are drawn from unknown underlying clean distribution P ( X , Y ) and underlying noisy distribution P ( X , Ỹ ) , respectively . For the noisy samples data D̃ , we may approximate underlying noisy distribution by the empirical noisy distribution , i.e. , P̂D̃ ( X , Ỹ ) = 1 N ∑N n=1 δ ( X = xn , Ỹ = ỹn ) , where δ ( X = xn , Ỹ = ỹn ) is a Dirac mass centered at ( xn , ỹn ) . Since the clean data sample is unobserved , we define posterior empirical clean distribution P̂ ( X , Y |D̃ ) as the inference posterior empirical clean distribution from noisy samples data D̃ . Based on Bayes ’ s rule , the posterior empirical clean distribution is given by P̂ ( X , Y |D̃ ) = 1N ∑N n=1 ∑c yn=1 δ ( X = xn , Y = ỹn ) P ( Y = yn|Ỹ = ỹn , X = xn ) . Noise transition matrix T ( x ) To describe the corruption process of the label , the NTM T ( x ) ∈ Rc×c is defined as Ti , j ( x ) = P ( Ỹ = j|Y = i , X = x ) , which represents the transition probability of the instance x labeling from clean i-th class to noisy j-th class . The underlying noisy label Ỹ probability given the instance x satisfies : P ( Ỹ = j|X = x ) = c∑ i=1 P ( Ỹ = j|Y = i , X = x ) P ( Y = i|X = x ) = c∑ i=1 Tij ( x ) P ( Y = i|X = x ) . The NTM T ( x ) hence bridges the gap between the underlying clean labels probability P ( Y|X = x ) = [ P ( Y = 1|X = x ) , · · · , P ( Y = c|X = x ) ] > and underlying noisy labels probability P ( Ỹ|X = x ) = [ P ( Ỹ = 1|X = x ) , · · · , P ( Ỹ = c|X = x ) ] > given instance x , i.e. , P ( Ỹ|X = x ) = T ( x ) > P ( Y|X = x ) . Loss Correction method Let function f ( · ) represents a neural network and f ( x ) denotes the c-dimensional output probability for instance x , where the ith index of the output yf , i ( x ) represents the predicted probability for class i . The common approach is to minimize the cross-entropy ( CE ) loss l ( f ( x ) , y ) : = − log ( yf , y ( x ) ) to force the output yf , y ( x ) approximate 1 . However , the label noise may mislead deep learning model . The existing NTM based methods first estimate NTM T ( x ) and then adopt it to correct the loss function . For example , in the forward correction procedure ( Patrini et al. , 2017 ) , the estimated NTM is adopted to corrupt the output predicted probability f ( x ) , i.e. , the corrupted predicted probability is f̃ ( x ) = T ( x ) > f ( x ) , and then compares it with the noisy label ỹ . Suppose T ( x ) is non-singular and the loss function is proper and composite , the forward loss correction can achieve consistent classifier , i.e. , the optimal classifier for the corrected loss with respect to the underlying noisy distribution is the same as that for the CE loss with respect to the underlying clean distribution : arg min f E ( X , Ỹ ) ∼P ( X , Ỹ ) [ l ( Ỹ , T ( X ) > f ( X ) ) ] = arg min f E ( X , Y ) ∼P ( X , Y ) [ l ( Y , f ( X ) ) ] . ( 1 ) 3 LEARNING WITH INFORMATION FUSION . Even though the existing loss correction methods could achieve consistent classifier theoretically , their performances are still undesirable in practice . Since the deep learning model is often trained on the empirical distribution with limited samples , the NTM can not correctly bridge the empirical clean distribution and empirical noisy distribution , i.e. , P̂ ( Ỹ = j|X = x ) 6= ∑c i=1 Tij ( x ) P̂D̃ ( Y = i|X = x ) . In other words , the NTM-based method may not work well in practice since due to the empirical clean and noisy distribution mismatch problem in IDN . Aiming to utilize the observed noisy label , we propose a concept , named posterior transition matrix ( PTM ) , to describe the transition probability given the observed noisy labels ( formally defined in Section 3.1 ) . The motivation for PTM stems from a simple thought experiment as shown in Figure 1 , which shows that PTM can model the empirical label noise better than NTM . Figure 2 illustrates the overview framework of IF , including NTM and PTM estimations ( Section 3.2 ) , information fusion via linear combination ( Section 3.3 ) , and posterior loss function ( Section 3.1 ) . 3.1 THE LOSS CORRECTION METHOD . The main goal is to train a c-class neural network classifier f ( x , ω ) to predict the clean label probability P ( Y |X ) . Since only the noisy labels are observed , there is a gap between the clean and noisy label , describing via NTM ( Goldberger & Ben-Reuven , 2017 ) . Motivated by the observed noisy labels ( posterior information ) , we define the PTM W ( x ) to describe the posterior clean label probability given noisy label , where Wi , j ( x ) = P ( Y = i|Ỹ = j , X = x ) . We provide the relationship between the PTM W ( x ) and NTM T ( x ) via Bayes ’ rulei.e. , Wi , j ( x ) = P ( Y = i , Ỹ = j|X = x ) P ( Ỹ = j|X = x ) = P ( Y = i|X = x ) Tij ( x ) ∑c i=1 P ( Y = i|X = x ) Tij ( x ) . ( 2 ) Notice that only the summation of any column is 1 for PTM W ( x ) , while the summation of any rows are 1 for NTM T ( x ) . Subsequently , we provide posterior reweight loss correction method via NTM . Definition 1 ( Posterior reweight loss ) Assume the model prediction is f ( x ) for noisy sample ( x , ỹ ) and W ( x ) is the NTM associated with the noisy sample . The posterior reweight loss is defined as lp−rew ( ỹ , f ( x ) ) = c∑ i=1 Wi , ỹ ( x ) l ( i , f ( x ) ) , ( 3 ) We next analyze the property of the posterior reweight loss and provide the theoretical justification . Specifically , we analyze the expected risk RP ( X , Ỹ ) ( f ) = E ( x , ỹ ) ∼P ( X , Ỹ ) [ lp−rew ( ỹ , f ( x ) ) ] and empirical risk R̂D̂ ( f ) = 1 N ∑N n=1 lp−rew ( ỹn , f ( xn ) ) under noisy samples dataset . Theorem 3.1 ( Statistically Consistent Classifier ) The posterior reweight loss can achieve unbiased classifier for underlying distribution and empirical distribution . ( i ) For underlying distribution , the expected risk satisfies arg min f E ( x , ỹ ) ∼P ( X , Ỹ ) [ lp−rew ( ỹ , f ( x ) ) ] = arg min f E ( x , y ) ∼P ( X , Y ) [ l ( y , f ( x ) ) ] , ( 4 ) ( ii ) For the anchor points samples ( xap , ỹap ) with underlying clean probability P ( Y = yap|X = xap ) , the empirical risk satisfies lp−rew ( ỹap , f ( xap ) ) = l ( yap , f ( xap ) ) . For the empirical distribution , the empirical risk satisfies arg min f E ( x , ỹ ) ∼P̂D̃ ( X , Ỹ ) [ lp−rew ( ỹn , f ( xn ) ) ] = arg min f E ( x , y ) ∼P̂ ( X , Y |D̃ ) [ l ( yn , f ( xn ) ) ] . ( 5 ) We also show that the PTM can be adopted in the forward manner . The definition of posterior reweight loss is given as follows , Definition 2 ( Posterior forward loss ) Assume the model prediction is f ( x ) for noisy sample ( x , ỹ ) and W ( x ) is the NTM associated with the noisy sample . The posterior forward loss is defined as lp−fw ( ỹ , f ( x ) ) = l ( ỹ , c∑ i=1 Wi , ỹ ( x ) fi ( x ) ) , ( 6 ) We next analyze the property of the posterior forward loss and provide the theoretical justification . Lemma 3.2 The posterior forward loss is not larger than posterior reweighted loss lp−rew ( ỹ , f ( x ) ) ≥ lp−fw ( ỹ , f ( x ) ) . Theorem 3.3 ( Statistically Consistent Classifier ) Suppose the loss function l ( y , f ) is convex with respect to f ( the convex condition is commonly satisfies , e.g. , cross entropy loss ) , and the minimum expected risk RP ( X , Y ) ( f ) can achieve 0 , then the posterior forward loss can achieve unbiased classifier for underlying distribution , arg min f E ( x , ỹ ) ∼P ( X , Ỹ ) [ lp−fw ( ỹ , f ( x ) ) ] = arg min f E ( x , y ) ∼P ( X , Y ) [ l ( y , f ( x ) ) ] . ( 7 ) Compare the posterior forward loss with forward loss , it is seen that both of them correct the output of the neural network with statistically consistent risk guarantee , while the difference falls in the different transition matrix in the loss correction . In addition , the statistically consistent risk for forward loss and posterior forward loss both require the accurate transition matrix estimation . However , the label/PTM estimation is pretty challenging since the label noise is instance-dependent in real world . Such observation motivates us to combine the NTM T ( x ) and PTM W ( x ) to correct the loss . Motivated by Kalman filtering ( Kalman , 1960 ) , a famous estimation method combining the prior knowledge and measurement information , we propose an information approach to tackle instance-dependent label noise via loss correction . Specifically , we firstly claim that there are prior knowledge information and measurement information on the label noise , where prior knowledge information and measurement information represent the estimated NTM and PTM .

Summary: This paper provides new insight on estimating noise-transition matrix in the setting of instance-dependent label noise. Specifically, the authors observe that the traditional noise-transition matrix cannot well bridge the gap between underlying distribution and empirical distribution. Inspired by this observation, the authors propose to estimate the posterior noise-transition matrix. Further, authors use information fusion (IF) to linearly combine noise-transition matrix and posterior noise-transition matrix to obtain more accurate transition matrix estimation with theoretical guarantees. Experiments are conducted on CIFAR-10, SVHN, F-MNIST, and real-world dataset Clothing-1M.