An Information Fusion Approach to Learning with Instance-Dependent Label Noise

1 INTRODUCTION . Data labeling annotated from human efforts , such as crowdsourcing ( Yan et al. , 2014 ; Chen et al. , 2017 ) and online queries ( Divvala et al. , 2014 ) , may be heavily noisy in practice . To make it worse , the label noise stemmed from the human annotations is often instance-dependent . For example , the images closing to the decision boundary are usually prone to be mislabeled ( Zhang et al. , 2021 ; Zhu et al. , 2021b ) . On the other hand , the remarkable success of deep neural networks ( DNNs ) on supervised learning tasks heavily relies on the expressive power and a large number of data with accurate labels . Unfortunately , deep neural networks memorizes noisy labels leading to poor generalization ( Zhang et al. , 2017 ) . It is challenging to learn with practical instance-dependent label noise ( IDN ) due to the hidden and complicated label noise properties ( Zhu et al. , 2021a ; Liu , 2021 ) . The methods of dealing with noisy labels falls in two lines , including heuristically identifying noisy samples or statistical label noise modeling . The training of deep neural network often learns the clean labels first ( Arpit et al. , 2017 ) and memorizes the noisy labels gradually , which is recognized as the memorization effect . Based on the general memorization effects , the heuristical methods are all designed via following the anomaly detection strategy : identifying the noisy sample based on different behaviors ( e.g. , loss values ) between clean and noisy samples during training . The typical methods contain sample selection ( Yu et al. , 2019 ; Han et al. , 2018b ) , reweight samples ( Cheng et al. , 2021 ; Jiang et al. , 2018 ; Ren et al. , 2018 ) , label correction ( Ma et al. , 2018 ; Tanaka et al. , 2018 ) , and regularization ( Han et al. , 2018a ) . Although these algorithms empirically work well , without modeling the label noise explicitly , the reliability can not be guaranteed . Another line of works relies on noise transition matrix ( NTM ) to model the label noise statistically ( Xia et al. , 2019 ; 2020 ; Patrini et al. , 2017 ) . Namely , the NTM quantifies the probabilities that clean labels flip into noisy labels . Although the NTM-based methods possess theoretical guarantee , under IDN , the NTM estimation for each instance is pretty challenging . To ease the estimation , some unrealistic assumptions have to be posed the on NTM , including instance-independent transition matrix ( Liu & Guo , 2020 ; Wei & Liu , 2021 ; Li et al. , 2021 ) , symmetric transition matrix ( Menon et al. , 2018 ) , upper bounded noise rate ( Cheng et al. , 2020 ) , and part-dependent label noise ( Xia et al. , 2020 ) . However , under the complex IDN , the empirical noise distribution is highly different with underlying noise distribution . For example , in Figure 1 , the underlying and empirical noisy distribution for long-tail instance is different since the empirical noisy label is either the same or different with clean label . Additionally , observed noisy label provide inductive bias toward label corruption . In other word , the genuine problem causing from IDN is the empirical , instead of underlying , clean and noisy distribution mismatch problem . To mitigate empirical distribution mismatch problem , we propose the posterior transition matrix ( PTM ) , posteriorly modeling label noise given the observed noisy labels . We adopt PTM to provably bridge the gap between clean and noisy underlying data distribution , and anchor points ( i.e. , data points that belong to a specific class almost surely ) empirical distribution simultaneously . We also provide an easy-to-compute PTM estimation under specific condition of low label noise . To extend the applicability to a wide range of label noise , motivated by Kalman filtering ( Kalman , 1960 ) , we proposed the information fusion ( IF ) method linearly combining the estimated noise and PTM , which achieves lower transition matrix estimation error . We empirically show that proposed method IF can achieve higher accuracy and more stable training . The main contributions of this work are summarized below . • We propose a new concept , named PTM , achieving consistent classifier for underlying distribution and anchor point empirical distribution mismatch simultaneously . • We propose the simple PTM estimation method based on observed noisy label under low label noise . To extend the applicability , we propose IF method , linearly combining the estimated NTM and PTM , to achieve lower estimation error with theoretical and experimental justification . • We experimentally demonstrate the superiority of proposed IF method achieving higher accuracy and more stable training under synthetic and real-world label noise . 2 PRELIMINARIES . Throughout this paper , we adopt uppercase letters to denote random variables , and lowercase letters to denote particular realization of the random variables . We target the c-class classification problem with instance-dependent ( label ) noise ( IDN ) . Let P ( X , Y ) be the underlying clean distribution of the random variables ( X , Y ) , where X and Y represent the instance and clean label . However , in real-world scenario , the clean label is corrupted and thus only noisy label , denoted as Ỹ , is observed . Given a set of N training instances denoted by D̃ : = { ( xn , ỹn ) } Nn=1 , where xn is the instance vector of the n-th sample , ỹn ∈ [ c ] : = { 1 , · · · , c } is the corresponding observed but noisy label , our goal is to predict the clean label yn for any given instance xn . Data distribution We assume that the unobserved clean samples ( xn , yn ) and available noisy samples ( xn , ỹn ) are drawn from unknown underlying clean distribution P ( X , Y ) and underlying noisy distribution P ( X , Ỹ ) , respectively . For the noisy samples data D̃ , we may approximate underlying noisy distribution by the empirical noisy distribution , i.e. , P̂D̃ ( X , Ỹ ) = 1 N ∑N n=1 δ ( X = xn , Ỹ = ỹn ) , where δ ( X = xn , Ỹ = ỹn ) is a Dirac mass centered at ( xn , ỹn ) . Since the clean data sample is unobserved , we define posterior empirical clean distribution P̂ ( X , Y |D̃ ) as the inference posterior empirical clean distribution from noisy samples data D̃ . Based on Bayes ’ s rule , the posterior empirical clean distribution is given by P̂ ( X , Y |D̃ ) = 1N ∑N n=1 ∑c yn=1 δ ( X = xn , Y = ỹn ) P ( Y = yn|Ỹ = ỹn , X = xn ) . Noise transition matrix T ( x ) To describe the corruption process of the label , the NTM T ( x ) ∈ Rc×c is defined as Ti , j ( x ) = P ( Ỹ = j|Y = i , X = x ) , which represents the transition probability of the instance x labeling from clean i-th class to noisy j-th class . The underlying noisy label Ỹ probability given the instance x satisfies : P ( Ỹ = j|X = x ) = c∑ i=1 P ( Ỹ = j|Y = i , X = x ) P ( Y = i|X = x ) = c∑ i=1 Tij ( x ) P ( Y = i|X = x ) . The NTM T ( x ) hence bridges the gap between the underlying clean labels probability P ( Y|X = x ) = [ P ( Y = 1|X = x ) , · · · , P ( Y = c|X = x ) ] > and underlying noisy labels probability P ( Ỹ|X = x ) = [ P ( Ỹ = 1|X = x ) , · · · , P ( Ỹ = c|X = x ) ] > given instance x , i.e. , P ( Ỹ|X = x ) = T ( x ) > P ( Y|X = x ) . Loss Correction method Let function f ( · ) represents a neural network and f ( x ) denotes the c-dimensional output probability for instance x , where the ith index of the output yf , i ( x ) represents the predicted probability for class i . The common approach is to minimize the cross-entropy ( CE ) loss l ( f ( x ) , y ) : = − log ( yf , y ( x ) ) to force the output yf , y ( x ) approximate 1 . However , the label noise may mislead deep learning model . The existing NTM based methods first estimate NTM T ( x ) and then adopt it to correct the loss function . For example , in the forward correction procedure ( Patrini et al. , 2017 ) , the estimated NTM is adopted to corrupt the output predicted probability f ( x ) , i.e. , the corrupted predicted probability is f̃ ( x ) = T ( x ) > f ( x ) , and then compares it with the noisy label ỹ . Suppose T ( x ) is non-singular and the loss function is proper and composite , the forward loss correction can achieve consistent classifier , i.e. , the optimal classifier for the corrected loss with respect to the underlying noisy distribution is the same as that for the CE loss with respect to the underlying clean distribution : arg min f E ( X , Ỹ ) ∼P ( X , Ỹ ) [ l ( Ỹ , T ( X ) > f ( X ) ) ] = arg min f E ( X , Y ) ∼P ( X , Y ) [ l ( Y , f ( X ) ) ] . ( 1 ) 3 LEARNING WITH INFORMATION FUSION . Even though the existing loss correction methods could achieve consistent classifier theoretically , their performances are still undesirable in practice . Since the deep learning model is often trained on the empirical distribution with limited samples , the NTM can not correctly bridge the empirical clean distribution and empirical noisy distribution , i.e. , P̂ ( Ỹ = j|X = x ) 6= ∑c i=1 Tij ( x ) P̂D̃ ( Y = i|X = x ) . In other words , the NTM-based method may not work well in practice since due to the empirical clean and noisy distribution mismatch problem in IDN . Aiming to utilize the observed noisy label , we propose a concept , named posterior transition matrix ( PTM ) , to describe the transition probability given the observed noisy labels ( formally defined in Section 3.1 ) . The motivation for PTM stems from a simple thought experiment as shown in Figure 1 , which shows that PTM can model the empirical label noise better than NTM . Figure 2 illustrates the overview framework of IF , including NTM and PTM estimations ( Section 3.2 ) , information fusion via linear combination ( Section 3.3 ) , and posterior loss function ( Section 3.1 ) . 3.1 THE LOSS CORRECTION METHOD . The main goal is to train a c-class neural network classifier f ( x , ω ) to predict the clean label probability P ( Y |X ) . Since only the noisy labels are observed , there is a gap between the clean and noisy label , describing via NTM ( Goldberger & Ben-Reuven , 2017 ) . Motivated by the observed noisy labels ( posterior information ) , we define the PTM W ( x ) to describe the posterior clean label probability given noisy label , where Wi , j ( x ) = P ( Y = i|Ỹ = j , X = x ) . We provide the relationship between the PTM W ( x ) and NTM T ( x ) via Bayes ’ rulei.e. , Wi , j ( x ) = P ( Y = i , Ỹ = j|X = x ) P ( Ỹ = j|X = x ) = P ( Y = i|X = x ) Tij ( x ) ∑c i=1 P ( Y = i|X = x ) Tij ( x ) . ( 2 ) Notice that only the summation of any column is 1 for PTM W ( x ) , while the summation of any rows are 1 for NTM T ( x ) . Subsequently , we provide posterior reweight loss correction method via NTM . Definition 1 ( Posterior reweight loss ) Assume the model prediction is f ( x ) for noisy sample ( x , ỹ ) and W ( x ) is the NTM associated with the noisy sample . The posterior reweight loss is defined as lp−rew ( ỹ , f ( x ) ) = c∑ i=1 Wi , ỹ ( x ) l ( i , f ( x ) ) , ( 3 ) We next analyze the property of the posterior reweight loss and provide the theoretical justification . Specifically , we analyze the expected risk RP ( X , Ỹ ) ( f ) = E ( x , ỹ ) ∼P ( X , Ỹ ) [ lp−rew ( ỹ , f ( x ) ) ] and empirical risk R̂D̂ ( f ) = 1 N ∑N n=1 lp−rew ( ỹn , f ( xn ) ) under noisy samples dataset . Theorem 3.1 ( Statistically Consistent Classifier ) The posterior reweight loss can achieve unbiased classifier for underlying distribution and empirical distribution . ( i ) For underlying distribution , the expected risk satisfies arg min f E ( x , ỹ ) ∼P ( X , Ỹ ) [ lp−rew ( ỹ , f ( x ) ) ] = arg min f E ( x , y ) ∼P ( X , Y ) [ l ( y , f ( x ) ) ] , ( 4 ) ( ii ) For the anchor points samples ( xap , ỹap ) with underlying clean probability P ( Y = yap|X = xap ) , the empirical risk satisfies lp−rew ( ỹap , f ( xap ) ) = l ( yap , f ( xap ) ) . For the empirical distribution , the empirical risk satisfies arg min f E ( x , ỹ ) ∼P̂D̃ ( X , Ỹ ) [ lp−rew ( ỹn , f ( xn ) ) ] = arg min f E ( x , y ) ∼P̂ ( X , Y |D̃ ) [ l ( yn , f ( xn ) ) ] . ( 5 ) We also show that the PTM can be adopted in the forward manner . The definition of posterior reweight loss is given as follows , Definition 2 ( Posterior forward loss ) Assume the model prediction is f ( x ) for noisy sample ( x , ỹ ) and W ( x ) is the NTM associated with the noisy sample . The posterior forward loss is defined as lp−fw ( ỹ , f ( x ) ) = l ( ỹ , c∑ i=1 Wi , ỹ ( x ) fi ( x ) ) , ( 6 ) We next analyze the property of the posterior forward loss and provide the theoretical justification . Lemma 3.2 The posterior forward loss is not larger than posterior reweighted loss lp−rew ( ỹ , f ( x ) ) ≥ lp−fw ( ỹ , f ( x ) ) . Theorem 3.3 ( Statistically Consistent Classifier ) Suppose the loss function l ( y , f ) is convex with respect to f ( the convex condition is commonly satisfies , e.g. , cross entropy loss ) , and the minimum expected risk RP ( X , Y ) ( f ) can achieve 0 , then the posterior forward loss can achieve unbiased classifier for underlying distribution , arg min f E ( x , ỹ ) ∼P ( X , Ỹ ) [ lp−fw ( ỹ , f ( x ) ) ] = arg min f E ( x , y ) ∼P ( X , Y ) [ l ( y , f ( x ) ) ] . ( 7 ) Compare the posterior forward loss with forward loss , it is seen that both of them correct the output of the neural network with statistically consistent risk guarantee , while the difference falls in the different transition matrix in the loss correction . In addition , the statistically consistent risk for forward loss and posterior forward loss both require the accurate transition matrix estimation . However , the label/PTM estimation is pretty challenging since the label noise is instance-dependent in real world . Such observation motivates us to combine the NTM T ( x ) and PTM W ( x ) to correct the loss . Motivated by Kalman filtering ( Kalman , 1960 ) , a famous estimation method combining the prior knowledge and measurement information , we propose an information approach to tackle instance-dependent label noise via loss correction . Specifically , we firstly claim that there are prior knowledge information and measurement information on the label noise , where prior knowledge information and measurement information represent the estimated NTM and PTM .

Summary: This paper proposes to estimate an input-dependent noise transition matrix (PTM). The authors make a strong assumption that the neural network output approximates the underlying clean probability and estimate the PTM by extracting the output of each training sample using the network. To enforce the network to have better probabilities, PTM estimation is applied iteratively and warm-up is used during the training. The experiments are conducted in CIFAR-10 with different noise ratio, as well as a real world dataset, i.e. Clothing-1M.