Year: 2021

Abstract:
Instance-dependent label noise (IDN) widely exists in real-world datasets and usually misleads the training of deep neural networks. Noise transition matrix (NTM) (i.e., the probability that clean labels flip into noisy labels) is used to characterize the label noise and can be adopted to bridge the gap between clean and noisy underlying data distribution. However, most of instances are long-tail, i.e., the number of appearance for each instance is usually limited, which leads to the gap between underlying distribution and empirical distribution. Therefore, the genuine problem caused by IDN is empirical, instead of underlying, data distribution mismatch during training. To directly tackle empirical distribution mismatch problem, we propose posterior transition matrix (PTM) to posteriorly model label noise given limited observed noisy labels achieving statistically consistent classifiers. Note that even if the instance is corrupted by the same NTM, the intrinsic randomness incurs to different noisy labels, and thus requires different correction methods. Motivated by this observation, we propose an Information Fusion (IF) approach to fine-tune the NTM based on estimated PTM. Specifically, we adopt the noisy labels and model predicted probability to estimate PTM and then correct the NTM in forward propagation. Empirical evaluations on synthetic and real-world datasets demonstrate that our method is superior to the state-of-the-art approaches, and achieves more stable training for instance-dependent label noise.

heading: 1 INTRODUCTION
text: Data labeling annotated from human efforts, such as crowdsourcing (Yan et al., 2014; Chen et al., 2017) and online queries (Divvala et al., 2014), may be heavily noisy in practice. To make it worse, the label noise stemmed from the human annotations is often instance-dependent. For example, the images closing to the decision boundary are usually prone to be mislabeled (Zhang et al., 2021; Zhu et al., 2021b). On the other hand, the remarkable success of deep neural networks (DNNs) on supervised learning tasks heavily relies on the expressive power and a large number of data with accurate labels. Unfortunately, deep neural networks memorizes noisy labels leading to poor generalization (Zhang et al., 2017). It is challenging to learn with practical instance-dependent label noise (IDN) due to the hidden and complicated label noise properties (Zhu et al., 2021a; Liu, 2021).
The methods of dealing with noisy labels falls in two lines, including heuristically identifying noisy samples or statistical label noise modeling. The training of deep neural network often learns the clean labels first (Arpit et al., 2017) and memorizes the noisy labels gradually, which is recognized as the memorization effect. Based on the general memorization effects, the heuristical methods are all designed via following the anomaly detection strategy: identifying the noisy sample based on different behaviors (e.g., loss values) between clean and noisy samples during training. The typical methods contain sample selection (Yu et al., 2019; Han et al., 2018b), reweight samples (Cheng et al., 2021; Jiang et al., 2018; Ren et al., 2018), label correction (Ma et al., 2018; Tanaka et al., 2018), and regularization (Han et al., 2018a). Although these algorithms empirically work well, without modeling the label noise explicitly, the reliability cannot be guaranteed.
Another line of works relies on noise transition matrix (NTM) to model the label noise statistically (Xia et al., 2019; 2020; Patrini et al., 2017). Namely, the NTM quantifies the probabilities that clean labels flip into noisy labels. Although the NTM-based methods possess theoretical guarantee, under IDN, the NTM estimation for each instance is pretty challenging. To ease the estimation, some
unrealistic assumptions have to be posed the on NTM, including instance-independent transition matrix (Liu & Guo, 2020; Wei & Liu, 2021; Li et al., 2021), symmetric transition matrix (Menon et al., 2018), upper bounded noise rate (Cheng et al., 2020), and part-dependent label noise (Xia et al., 2020). However, under the complex IDN, the empirical noise distribution is highly different with underlying noise distribution. For example, in Figure 1, the underlying and empirical noisy distribution for long-tail instance is different since the empirical noisy label is either the same or different with clean label. Additionally, observed noisy label provide inductive bias toward label corruption. In other word, the genuine problem causing from IDN is the empirical, instead of underlying, clean and noisy distribution mismatch problem.
To mitigate empirical distribution mismatch problem, we propose the posterior transition matrix (PTM), posteriorly modeling label noise given the observed noisy labels. We adopt PTM to provably bridge the gap between clean and noisy underlying data distribution, and anchor points (i.e., data points that belong to a specific class almost surely) empirical distribution simultaneously. We also provide an easy-to-compute PTM estimation under specific condition of low label noise. To extend the applicability to a wide range of label noise, motivated by Kalman filtering (Kalman, 1960), we proposed the information fusion (IF) method linearly combining the estimated noise and PTM, which achieves lower transition matrix estimation error. We empirically show that proposed method IF can achieve higher accuracy and more stable training.
The main contributions of this work are summarized below.
• We propose a new concept, named PTM, achieving consistent classifier for underlying distribution and anchor point empirical distribution mismatch simultaneously.
• We propose the simple PTM estimation method based on observed noisy label under low label noise. To extend the applicability, we propose IF method, linearly combining the estimated NTM and PTM, to achieve lower estimation error with theoretical and experimental justification.
• We experimentally demonstrate the superiority of proposed IF method achieving higher accuracy and more stable training under synthetic and real-world label noise.

heading: 2 PRELIMINARIES
text: Throughout this paper, we adopt uppercase letters to denote random variables, and lowercase letters to denote particular realization of the random variables. We target the c-class classification problem with instance-dependent (label) noise (IDN). Let P(X,Y ) be the underlying clean distribution of the random variables (X,Y ), where X and Y represent the instance and clean label. However, in real-world scenario, the clean label is corrupted and thus only noisy label, denoted as Ỹ , is observed. Given a set of N training instances denoted by D̃ := {(xn, ỹn)}Nn=1, where xn is the instance vector of the n-th sample, ỹn ∈ [c] := {1, · · · , c} is the corresponding observed but noisy label, our goal is to predict the clean label yn for any given instance xn.
Data distribution We assume that the unobserved clean samples (xn, yn) and available noisy samples (xn, ỹn) are drawn from unknown underlying clean distribution P(X,Y ) and underlying noisy distribution P(X, Ỹ ), respectively. For the noisy samples data D̃, we may approximate underlying noisy distribution by the empirical noisy distribution, i.e., P̂D̃(X, Ỹ ) = 1 N ∑N n=1 δ(X = xn, Ỹ = ỹn), where δ(X = xn, Ỹ = ỹn) is a Dirac mass centered at (xn, ỹn). Since the clean data sample is unobserved, we define posterior empirical clean distribution P̂(X,Y |D̃) as the inference posterior empirical clean distribution from noisy samples data D̃. Based on Bayes’s rule, the posterior empirical clean distribution is given by P̂(X,Y |D̃) = 1N ∑N n=1 ∑c yn=1
δ(X = xn, Y = ỹn)P(Y = yn|Ỹ = ỹn, X = xn).
Noise transition matrix T (x) To describe the corruption process of the label, the NTM T (x) ∈ Rc×c is defined as Ti,j(x) = P(Ỹ = j|Y = i,X = x), which represents the transition probability of the instance x labeling from clean i-th class to noisy j-th class. The underlying noisy label Ỹ probability given the instance x satisfies: P(Ỹ = j|X = x) = c∑ i=1 P(Ỹ = j|Y = i,X = x)P(Y = i|X = x) = c∑ i=1 Tij(x)P(Y = i|X = x).
The NTM T (x) hence bridges the gap between the underlying clean labels probability P(Y|X = x) = [ P(Y = 1|X = x), · · · ,P(Y = c|X = x) ]> and underlying noisy labels probability
P(Ỹ|X = x) = [ P(Ỹ = 1|X = x), · · · ,P(Ỹ = c|X = x) ]> given instance x, i.e., P(Ỹ|X =
x) = T (x)>P(Y|X = x).
Loss Correction method Let function f(·) represents a neural network and f(x) denotes the c-dimensional output probability for instance x, where the ith index of the output yf,i(x) represents the predicted probability for class i. The common approach is to minimize the cross-entropy (CE) loss l(f(x), y) := − log(yf,y(x)) to force the output yf,y(x) approximate 1. However, the label noise may mislead deep learning model. The existing NTM based methods first estimate NTM T (x) and then adopt it to correct the loss function. For example, in the forward correction procedure (Patrini et al., 2017), the estimated NTM is adopted to corrupt the output predicted probability f(x), i.e., the corrupted predicted probability is f̃(x) = T (x)>f(x), and then compares it with the noisy label ỹ. Suppose T (x) is non-singular and the loss function is proper and composite, the forward loss correction can achieve consistent classifier, i.e., the optimal classifier for the corrected loss with respect to the underlying noisy distribution is the same as that for the CE loss with respect to the underlying clean distribution:
arg min f
E(X,Ỹ )∼P(X,Ỹ )[l(Ỹ , T (X) >f(X))] = arg min f E(X,Y )∼P(X,Y )[l(Y, f(X))]. (1)

heading: 3 LEARNING WITH INFORMATION FUSION
text: Even though the existing loss correction methods could achieve consistent classifier theoretically, their performances are still undesirable in practice. Since the deep learning model is often trained on the empirical distribution with limited samples, the NTM cannot correctly bridge the empirical clean distribution and empirical noisy distribution, i.e., P̂(Ỹ = j|X = x) 6= ∑c i=1 Tij(x)P̂D̃(Y = i|X = x). In other words, the NTM-based method may not work well in practice since due to the empirical clean and noisy distribution mismatch problem in IDN.
Aiming to utilize the observed noisy label, we propose a concept, named posterior transition matrix (PTM), to describe the transition probability given the observed noisy labels (formally defined in Section 3.1). The motivation for PTM stems from a simple thought experiment as shown in Figure 1, which shows that PTM can model the empirical label noise better than NTM. Figure 2 illustrates the overview framework of IF, including NTM and PTM estimations (Section 3.2), information fusion via linear combination (Section 3.3), and posterior loss function (Section 3.1).

heading: 3.1 THE LOSS CORRECTION METHOD
text: The main goal is to train a c-class neural network classifier f(x, ω) to predict the clean label probability P(Y |X). Since only the noisy labels are observed, there is a gap between the clean and noisy label, describing via NTM (Goldberger & Ben-Reuven, 2017).
Motivated by the observed noisy labels (posterior information), we define the PTM W (x) to describe the posterior clean label probability given noisy label, where Wi,j(x) = P(Y = i|Ỹ = j,X = x). We provide the relationship between the PTM W (x) and NTM T (x) via Bayes’ rulei.e.,
Wi,j(x) = P(Y = i, Ỹ = j|X = x)
P(Ỹ = j|X = x) = P(Y = i|X = x)Tij(x)∑c i=1 P(Y = i|X = x)Tij(x) . (2)
Notice that only the summation of any column is 1 for PTM W (x), while the summation of any rows are 1 for NTM T (x). Subsequently, we provide posterior reweight loss correction method via NTM.
Definition 1 (Posterior reweight loss) Assume the model prediction is f(x) for noisy sample (x, ỹ) and W (x) is the NTM associated with the noisy sample. The posterior reweight loss is defined as
lp−rew
( ỹ, f(x) ) = c∑ i=1 Wi,ỹ(x)l(i, f(x)), (3)
We next analyze the property of the posterior reweight loss and provide the theoretical justification. Specifically, we analyze the expected risk RP(X,Ỹ )(f) = E(x,ỹ)∼P(X,Ỹ )[lp−rew ( ỹ, f(x) ) ] and
empirical risk R̂D̂(f) = 1 N ∑N n=1 lp−rew ( ỹn, f(xn) ) under noisy samples dataset.
Theorem 3.1 (Statistically Consistent Classifier) The posterior reweight loss can achieve unbiased classifier for underlying distribution and empirical distribution.
(i) For underlying distribution, the expected risk satisfies
arg min f
E(x,ỹ)∼P(X,Ỹ ) [ lp−rew ( ỹ, f(x) )] = arg min
f E(x,y)∼P(X,Y )[l(y, f(x))], (4)
(ii) For the anchor points samples (xap, ỹap) with underlying clean probability P(Y = yap|X = xap), the empirical risk satisfies lp−rew ( ỹap, f(xap) ) = l(yap, f(xap)). For the empirical
distribution, the empirical risk satisfies
arg min f E(x,ỹ)∼P̂D̃(X,Ỹ ) [ lp−rew ( ỹn, f(xn) )] = arg min f E(x,y)∼P̂(X,Y |D̃) [ l(yn, f(xn)) ] . (5)
We also show that the PTM can be adopted in the forward manner. The definition of posterior reweight loss is given as follows,
Definition 2 (Posterior forward loss) Assume the model prediction is f(x) for noisy sample (x, ỹ) and W (x) is the NTM associated with the noisy sample. The posterior forward loss is defined as
lp−fw ( ỹ, f(x) ) = l ( ỹ, c∑ i=1 Wi,ỹ(x)fi(x) ) , (6)
We next analyze the property of the posterior forward loss and provide the theoretical justification.
Lemma 3.2 The posterior forward loss is not larger than posterior reweighted loss lp−rew ( ỹ, f(x) ) ≥ lp−fw ( ỹ, f(x) ) .
Theorem 3.3 (Statistically Consistent Classifier) Suppose the loss function l(y, f) is convex with respect to f (the convex condition is commonly satisfies, e.g., cross entropy loss), and the minimum expected risk RP(X,Y )(f) can achieve 0, then the posterior forward loss can achieve unbiased classifier for underlying distribution,
arg min f
E(x,ỹ)∼P(X,Ỹ ) [ lp−fw ( ỹ, f(x) )] = arg min
f E(x,y)∼P(X,Y )[l(y, f(x))]. (7)
Compare the posterior forward loss with forward loss, it is seen that both of them correct the output of the neural network with statistically consistent risk guarantee, while the difference falls in the different transition matrix in the loss correction. In addition, the statistically consistent risk for forward loss and posterior forward loss both require the accurate transition matrix estimation. However, the label/PTM estimation is pretty challenging since the label noise is instance-dependent in real world. Such observation motivates us to combine the NTM T (x) and PTM W (x) to correct the loss.
Motivated by Kalman filtering(Kalman, 1960), a famous estimation method combining the prior knowledge and measurement information, we propose an information approach to tackle instance-dependent label noise via loss correction. Specifically, we firstly claim that there are prior knowledge information and measurement information on the label noise, where prior knowledge information and measurement information represent the estimated NTM and PTM.

heading: 3.2 PTM ESTIMATION
text: The posterior loss correction requires knowing PTM W (x) given any instance x. However, the PTM is unknown and needs to be estimated. In this subsection, we provide a simple PTM estimation method under the condition that the neural network output f(X) probability can well approximate the underlying probability P(Y |X), i.e., f(x) ≈ W (X = x)P̂(Ỹ|X = x). Intu-
itively, this condition is strong since neural network is prone to overfit noisy data. To achieve this condition, the warm-up training strategy and iterative PTM estimation are adopted. Warm-up training to make "good" neural network output approximating the empirical clean distribution since the network fits clean samples in the beginning. Subsequently, we iteratively estimate the PTM for loss correction during training.
For the case that the number of appearance for instance x is large, the empirical noisy distribution convergences to the underlying noisy distribution, i.e., P̂(Ỹ|X = x) −→ P(Ỹ|X = x). For the long-tail instance with only one appearance (x, ỹ), the empirical noisy labels distribution is always one-hot. Thus, this condition requires that the output of neural network f(x) should approximate to the underlying posterior clean probability, which is quite strong.
Under this condition, we propose a PTM estimation method based on the noisy labels and the model predicted probability. For the long-tail instance with only one appearance (x, ỹ), the observed noisy labels distribution is the one-hot. Compared with the NTM estimation (Xia et al., 2019; 2020), the proposed estimation method does not require the anchor point assumption that data points belong to a specific class almost surely. Specifically, the PTM can be estimated as follows,
Ŵi,j(x) = { fi(x), if ỹ = j; 0, if ỹ 6= j. (8)
For the case that the number of appearance is larger than one, the PTM estimation is ill-posed since the number of unknown variable c2 is larger the number of constraint c. Thus, we choose the PTM with minimum Frobenius norm, which is consistent with that for the case of only one appearance instance. Specifically, we have the following theorem:
Theorem 3.4 (Posterior transition matrix estimation) We aim to find the solution with minimal norm solution, i.e.,
Ŵ (x) = arg min W ||W ||2F
s.t.W>(x)P̂(Ỹ |x) = f(x), for ∀x Then, we have the optimal solution as follows,
Ŵ (x) = f(x)P̂(Ỹ |x)>
||P̂(Ỹ |x)||22 . (9)
Theorem 3.4 provides the PTM estimation for general empirical noisy labels distribution. For the case of only one appearance instance x, the empirical noisy distribution satisfies ||P̂(Ỹ |x)||2 = 1, and achieve consistent estimated PTM in equation (9).

heading: 3.3 INFORMATION FUSION
text: Subsection 3.2 introduces the PTM estimation methods based on the observed noisy label, the condition that neural network approximates clean label is quite strong even though the warm-up strategy and iterative estimation are adopted. Thus PTM estimation error could be large for large IDN rate. To further reduce the estimation error, motivated by Kalman filtering, we propose the information fusion (IF) to obtain more accurate transition matrix estimation via weighted average PTM estimation and NTM estimation with intuitive and theoretical justification. 1 Intuitively, for each instance, the estimated NTM and PTM may have different estimation accuracy. Via adaptively and linearly combine these two matrices, it is possible to obtain a more accurate transition matrix estimation. Specifically, we quantify the estimation uncertainty and lower uncertainty incurs higher weight. In this way, more accurate estimated transition matrix can be provided. Before the illustration of information fusion algorithm, we need to quantify the transition matrix estimation error without ground-truth. Suppose the (posterior) forward loss is adopted. For the estimated NTM T̂ (x), the corrupted model predicted probability f̃(x) = T̂>(x)f(x). Suppose the corrupted model predicted probability is accurate, the noisy label Ỹ satisfies c-dimension Bernoulli distribution with parameter f̃(x), i.e., Ỹ ∼ Bernoulli ( T̂>(x)f(x) ) , whereBernoulli(·) represents multi-dimension Bernoulli
distribution. Furthermore, we define the uncertainty σT̂ for transition matrix T̂ (x) as the trace of the covariance matrix for Ỹ , i.e., σT̂ (x) = Tr(KỸ (x)), where Tr(·) means the trace of matrix, KỸ (x) represents the covariance matrix of random variable Ỹ . For the multi-dimension Bernoulli distribution, we have the covariance matrix as follows,
KỸ = E[(Ỹ − E[Ỹ ])(Ỹ − E[Ỹ ]) >] = diag(T̂>(x)f(x))− T̂>(x)f(x)f(x)>T̂ (x)
Taking trace operation on the covariance matrix, we have the uncertainty for NTM as σT̂ (x) = ||T̂>(x)f(x)||1 − ||T̂>(x)f(x)||2. Similarly, the uncertainty for the PTM estimation is given by σŴ (x) = ||Ŵ>(x)f(x)||1− ||Ŵ>(x)f(x)||2. Once the uncertainty for estimated transition matrix has been established, we further integrate the two estimated transition matrices into Kalman transition matrix, defined as Wkm(x), via weighted average operation. Mathematically, Kalman transition matrix is given by
Wkm(x) = (1− λ(x))T̂ (x) + λ(x)Ŵ (x) (10) where the Kalman gain λ(x) is carefully selected to minimize the estimation error. Suppose the true PTM is W ∗(x), where W ∗ij(x) = 1 if y = i and ỹ = j, otherwise 0. Subsequently, we define the reconstruction error for NTM and PTM as eT (x) = W ∗(x)− T̂ (x) 2 and eW (x) = W ∗(x)−Ŵ (x), respectively. The parameter λ(x) is determined via minimization of the mean square reconstruction error. We theoretically justify the superiority of IF for a general scenario.
1The core idea for more accurate estimation is similar to statistical efficiency (Gong et al., 2020) via generating two “auxiliary" estimators. The key difference falls on the generation manner. IF generates NTM and PTM from the prior and posterior perspectives, while statistical efficiency generates two different noisy labels.
2The ideal estimated transition matrix is to approximate the empirical (actual) transition matrix W ∗(x), instead of the expectation of label transition matrix. Therefore, the estimation error is defined to characterize the bias–variance tradeoff for transition matrix estimation
Theorem 3.5 (Theoretical justification of IF) Suppose the correlation coefficient between reconstruction error eT (x) and eW (x) is defined as cov(x) ∈ [−1, 1] and the trace of covariance matrix for error eT̂ (x) and eŴ can be quantified by σT̂ (x) and σŴ (x), i.e., Tr(eT̂ (x)e > T̂ (x)) = σT̂ (x) and Tr(eŴ (x)e > Ŵ (x)) = σŴ (x). For the Kalman transition matrix Wkm, we define the reconstruc-
tion error as Lmse(λ(x)) = Tr ( (W ∗(x) −Wkm(x))(W ∗(x) −Wkm(x))> )
, then the optimal kalman gain to minimize the reconstruction error is given by
λ∗(x) = arg min λ Lmse(λ) =
σT̂ (x)− cov(x) √ σT̂ (x)σŴ (x)
σT̂ (x) + σŴ (x)− 2cov(x) √ σT̂ (x)σŴ (x)
(11)
and the minimum reconstruction error satisfies Lmse(λ∗(x)) ≤ min(σT̂ (x), σŴ (x)).
Theorem 3.5 implies that the Kalman transition matrix Wkm(x) provably achieve lower estimation error for general eT (x) and eW (x), either correlated or independent.

heading: 4 EXPERIMENTS
text: In this section, we empirically evaluate the effectiveness and stability of information fusion on synthetic and real-world noisy datasets. We aim to answer two questions as follows. Q1: Compared with the state-of-the-art transition matrix based methods, can information fusion achieve higher accuracy? Q2: Can information fusion leads to more accurate and robust transition matrix estimation and more stable training?

heading: 4.1 EXPERIMENTAL SETTINGS
text: Datasets We verify the superiority of IF on the three manually corrupted datasets, i.e., F-MNIST, SVHN, CIFAR-10, and one real-world noisy dataset Clothing1M. The first three datasets contain clean data and we manually corrupted the label of training dataset following from (Xia et al., 2020). More details about this instance-dependent label noise generation approach can be found in Appendix. IDN-τ means that the controlled noise rate is τ . All experiments on those datasets with synthetic instance-dependent label noise are repeated five times. The real-world dataset Clothing1M has 1M images with real-world noisy labels and 10k images with clean labels for testing. In experiments, we leave out 10% of the noisy training samples as a noisy validation set for model selection.
Baselines We compare the proposed IF method with (i) CE, which trains the standard deep network with the cross entropy loss on noisy datasets. (ii) DMI, a novel information-theoretic robust loss function for instance-independent label noise (Xu et al., 2019). (iii) Forward (Patrini et al., 2017), Reweight (Liu & Tao, 2015), and T-Revision (Xia et al., 2019). (iv) Part-dependent transition matrix (PTD) (Xia et al., 2020). (v) instance-level forward correction (ILFC) (Berthon et al., 2021). These approaches utilize the NTM to correct the loss function.
Implementations We choose the standard neural network and optimizer. We use a ResNet-18 network for F-MNIST, a ResNet-34 network for SVHN and CIFAR-10. For the optimization, we first use SGD with 0.9 momentum, 10−4 weight decay, 128 batch size, 50 epochs and an initial learning rate of 10−2 to initialize the network. Then, we adopt Adam optimizer and 5× 10−7 learning rate to learn the NTM following PTD (Xia et al., 2020). Once the NTM is obtained, we retrain the neural network. In the forward propagation, Kalman transition matrix is obtained and then adopted to correction the loss function. In addition, the revision trick (Xia et al., 2019) is also adopted for our proposed IF method, which introduces the slack variables ∆W for the estimated Kalman transition matrix. Specifically, the slack variable ∆W is initialized as all zero elements in the experiments and can be optimized during the training. Aiming to guarantee valid transition matrix, We first project their negative entries of Wkm(x) + ∆W to zero and then normalize the summation of each row as 1. Following (Xia et al., 2020), we do not use any data augmentation technique in the experiments. For Clothing1M, we use a ResNet-50 pre-trained model on ImageNet. We do not use any clean data to learn the transition matrices and classifiers. After the NTM T̂ (x) is obtained according to (Xia et al., 2020), we use SGD with 0.9 momentum, 10−3 weight decay, 32 batch size, and run with 10−3 learning rate and 10 epochs. For the evaluation, we adopt the test accuracy of the epoch with best accuracy on validation dataset.

heading: 4.2 EXPERIMENTAL RESULTS
text: Accuracy on synthetic and real-world noisy datasets To answer Q1, Figrue 3 shows the estimation error of PTD, PTM and IF methods with noise rate IDN-10% to IDN-40%, where the estimation error is defined as the average difference of the transition probability err = 1N ∑N n=1(1− Ŵynŷn). It is seen that IF can achieve the lowest estimation error over all noise ratio.3 Table 1 reports the classification accuracy on the SVHN, CIFAR-10 and F-MNIST with noise rate IDN-10% to IDN-50%, where the highest accuracy is bold faced. It is seen that information fusion achieves better performance compared with all baselines across the three datasets and five noise rate. For example, in CIFAR-10, the improvements over the strongest baseline is 1.07%, 0.83%, 4.03%, 9.22%, 9.89% for noise rate IDN-10% to IDN-50%, respectively. The higher accuracy implies that the posterior information is more important in higher noise rate. In addition, the methods IF-F and IF-F-V is comparable across the three datasets and noise rate, which demonstrates that the revision on the transition matrix is not significant beneficial compared with the method PTD. This observation validates the effectiveness of posterior information on transition matrix modification. For the real-world dataset Clothing1M, IF outperforms the baselines as shown in Table 2. Our IF method achieves 0.5% improvements over the strongest baseline early-learning regularization (ELR) (Liu et al., 2020).
3The transition matrix estimation for instance-dependent label noise is more challenging than the counterpart for class-dependent label noise (Li et al., 2021).
Transition matrix estimation evaluation during We study transition matrix estimation error in the training datat for IF and PTM to answer Q2. Figures 4 shows the training accuracy and estimation error of each epoch on CIFAR10 dataset with noise rate IDN-10% to IDN-40%, respectively. The shadow area represents the standard deviation. It is seen that, on all IDN rate, IF can achieve higher training accuracy and lower estimation error compared with IF.
Stability on synthetic noisy datasets We study the training stability via inspecting the test accuracy over epochs to answer Q2. Figures 5 shows the test accuracy with respect to epochs on CIFAR10 dataset with noise rate IDN-10% to IDN-40%, respectively. The shadow area represents the standard deviation of the accuracy on the same epoch across five experiments. We can clearly observe that, on both low-level and high level noise, IF shows higher accuracy in a more stable way since the posterior information of each instance provides reliable guideness on the loss correction.

heading: 5 CONCLUSIONS
text: This paper presents PTM to characterize label noise given observed noisy label, and a simple but effective learning paradigm named information fusion, which trains deep neural networks under instance-dependent label noise. The main goal of PTM is to characterize the label noise via limited observed noisy labels for each instance, which can bridge the empirical noisy and posterior empirical clean distribution. Aiming to effectively combine NTM and PTM, we propose PTM to characterize label noise given observed noisy label. Furthermore, we proposed an information method integrating both transition matrix to correct the loss function. Experiments on synthetic noisy and real-world dataset show that IF can achieve higher accuracy and more stable training compared with state-of-theart methods, especially for high noise rate. In the future, we can extend to incorporate more prior knowledge of the transition matrix, e.g., sparsity or low rank, into the end-to-end learning algorithm.

heading: A THEORETICAL ANALYSIS

heading: A.1 PROOF OF THEOREM 3.1
text: Based on the definition of posterior reweight loss, we have lp−rew ( ỹ, f(x) ) = ∑c y=1 P(Y =
y|Ỹ = ỹ, X = x)l(y, f(x)). For noisy underlying distribution, the expected risk can be calculated as follows,
E(x,ỹ)∼P(X,Ỹ ) [ lp−rew ( ỹ, f(x) )] =
∫ x c∑ ỹ=1 P(X = x, Ỹ = ỹ)lp−rew ( ỹ, f(x) ) dx
= ∫ x c∑ ỹ=1 P(X = x, Ỹ = ỹ) c∑ y=1 P(Y = y|Ỹ = ỹ, X = x)l ( y, f(x) ) dx
= ∫ x c∑ ỹ=1 c∑ y=1 P(X = x, Y = y, Ỹ = ỹ)l ( y, f(x) ) dx
= ∫ x c∑ y=1 P(X = x, Y = y)l ( y, f(x) ) dx = E(x,y)∼P(X,Y ) [ l ( y, f(x) )] , (12)
which implies that posterior reweight loss over noisy underlying distribution equals to cross entropy loss over clean underlying distribution and thus posterior reweight loss induces to statistically consistent classifier.
For the anchor points samples (xap, ỹap) with underlying clean probability P(Y = yap|X = xap), based on equation (2), the PTM value satisfies Wyap,ỹap(x) = 1 and Wk,ỹap = 0 for any k 6= yap, which implies that empirical distribution mismatch on anchor points samples can be completely mitigated. As for empirical distribution, the empirical risk satisfies
lp−rew ( ỹap, f(xap) ) = l(yap, f(xap)). (13)
For noisy empirical distribution, the expected risk satisfies E(x,ỹ)∼P̂D̃(X,Ỹ ) [ lp−rew ( ỹ, f(x) )] = 1
N N∑ n=1 lp−rew ( ỹi, f(xi) ) = 1
N N∑ n=1 c∑ y=1 P(Y = y|Ỹ = ỹn, X = xn)l(y, f(x))
= E(x,y)∼P̂(X,Y |D̃) [ l(yn, f(xn)) ] . (14)
which implies that posterior reweight loss over noisy empiricial distribution equals to cross entropy loss over posterior clean empirical distribution.

heading: A.2 PROOF OF LEMMA 3.2
text: Note that the loss function l(y, f) is convex with respect to f and ∑c i=1Wi,ỹ(x) = ∑c i=1 P(Y = i|Ỹ = ỹ, X = x) = 1 for any ỹ and x, according to Jensen’s inequality, we have
lp−rew
( ỹ, f(x) ) = c∑ y=1 Wi,ỹ(x)l(y, f(x))
≥ l ( ỹ, c∑ i=1 Wi,ỹ(x)f(x) ) = lp−fw ( ỹ, f(x) ) , (15)
which shows the relation between posterior reweight loss and posterior forward loss.

heading: A.3 PROOF OF THEOREM 3.3
text: Based on Lemma 3.2 and Theorem 3.1, we have the inequality on the expected risk as follows, E(x,ỹ)∼P(X,Ỹ ) [ lp−fw ( ỹ, f(x) )] ≤ E(x,ỹ)∼P(X,Ỹ ) [ lp−rew ( ỹ, f(x) )] = E(x,y)∼P(X,Y )[l(y, f(x))],
Define the optimal classifier f∗ to minimize expected risk for underlying clean distribution and cross entropy loss as
f∗ = arg min f RP(X,Y )(f) = arg min f E(x,y)∼P(X,Y )[l(y, f(x))], (16)
then RP(X,Y )(f∗) = 0 and for lp−rew ( ỹ, f∗(x) ) = 0 for any x. Note that the expected risk for posterior forward loss is non-negative and not larger than that for posterior reweight loss, we also have lp−fw ( ỹ, f∗(x) ) = 0 for any x. In addition, the difference of expected risk for two different classifiers satisfies
E(x,ỹ)∼P(X,Ỹ ) [ lp−fw ( ỹ, f∗(x) )] − E(x,ỹ)∼P(X,Ỹ ) [ lp−fw ( ỹ, f(x) )] = −E(x,ỹ)∼P(X,Ỹ ) [ lp−fw ( ỹ, f(x) )] ≤ 0, (17)
which implies that the classifier f∗ can also minimize expected risk for underlying noisy distribution and posterior reweight loss. This completes the proof.

heading: A.4 PROOF OF THEOREM 3.4
text: Aiming to standardize the constraint, we vectorize W (x) as c2 × 1 dimension wvec(x) := [W11(x), · · · ,Wc1(x), · · · ,W1c(x), · · · ,Wcc(x)]> and define the coefficient matrix A(x) as follows,
A(x) =  P̂>(Ỹ|x) · · · 0 · · · 0 0 P̂>(Ỹ|x) 0 · · · 0 ... ... . . . ... ...
0 0 0 · · · P̂>(Ỹ|x)  c×c2
(18)
where P>(Ỹ|x) = [P(Ỹ = 1|x), · · · ,P(Ỹ = c|x)]1×c. Therefore, the constraintW>(x)P̂(Ỹ |x) = f(x) can be transformed asA(x)wvec(x) = f(x), which is a least-norm problem with undetermined equations constraint. Note that matrix A(x) is full column rank, we can prove that the least-norm solutions is
w∗vec(x) = A >(x)
( A(x)A>(x) )−1 f(x). (19)
Considering any possible solution wvec(x) satisfying the constraint A(x)wvec(x) = f(x), then A(x)[wvec(x)−w∗vec(x)] = 0 and
[wvec(x)−w∗vec(x)]>w∗vec(x) = [wvec(x)−w∗vec(x)]>A>(x) ( A(x)A>(x) )−1 f(x)
= [ A(x)[wvec(x)−w∗vec(x)] ]>( A(x)A>(x) )−1 f(x) = 0,
which implies that the vectors wvec(x) − w∗vec(x) and w∗vec(x) are perpendicular. Thus, the L2 norm of wvec(x) satisfies
||wvec(x)||2 = ||w∗vec(x) + wvec(x)−w∗vec(x)||2
= ||w∗vec(x)||2 + ||wvec(x)−w∗vec(x)||2 ≥ ||w∗vec(x)||2. (20) which shows that the equation (19) is the optimal solution. Via direct calculation, it is easy to obtain that A(x)A>(x) = ∑c j=k P̂2(Ỹ = k|x)Ic×c. Therefore, the element of PTM satisfies
W ∗ij(x) = c∑ k=1 P̂2(Ỹ = k|x)P̂(Ỹ = i|x)fj(x), (21)
i.e., W ∗(x) = f(x)P̂(Ỹ |x) >
||P̂(Ỹ |x)||22 and completes the proof.

heading: A.5 PROOF OF THEOREM 3.5
text: Considering the general reconstruction error relationship, the trace of the covariance matrix for reconstruction error satisfies Tr(eT̂ (x)e > Ŵ (x)) = cov(x) √ σT (x)σW (x). For Kalman transition matrix, the reconstruction deviation satisfies W ∗(x)−Wkm(x) = (1− λ)eT (x) + λeW (x). For simplicity, we remove the instance x to abbreviate the notations. Therefore, the reconstruction error is given by
Lmse(λ) = E [ Tr (( (1− λ)eT + λeW )( (1− λ)eT + λeW )>)]
= (1− λ)2σT + λ2σW + 2λ(1− λ)cov √ σTσW , (22)
For such quadratic function, it is easy to obtain the optimal parameter via the local minimal
∂Lmse ∂λ = 2(1− λ)σT + 2λσW + 2(1− 2λ)cov √ σTσW = 0, (23)
and thus the optimal weight is given by λ∗ = σT−cov √ σTσW
σT+σW−2cov √ σTσW . Plugging in the optimal weight into the reconstruction error, we obtain the minimal reconstruction error is given by
Lmse(λ∗) = 1
[σT + σW − 2cov √ σTσW ]2
{ σT [ σW − cov √ σTσW ]2 +σW [ σT − cov √ σTσW
]2 +2 [ σW − cov √ σTσW ][ σT − cov √ σTσW ] cov √ σTσW } (24)
Aiming to investigate the relation between minimal reconstruction error for IF, PTM and PTD, we adopt variable substitution t = σT + σW − 2cov √ σTσW ≥ 0 to further simplify minimal reconstruction error Lmse(λ∗), then we have
Lmse(λ∗) = σT [(σW − σT ) + t] 4t2 + σW [(σT − σW ) + t] 4t2
+ [(σT − σW ) + t][(σW − σT ) + t][(σW + σT )− t]
4t2
= σW + σT 2 − [ t 4 + (σW − σT )2 4t ]
≤ σW + σT 2
− 2 √ t
4 · (σW − σT ) 2 4t = min{σW , σT } (25)
where the inequality holds according to arithmetic mean-geometric mean inequality. Therefore, the linear combination can provably achieve lower estimation error for general eT (x) and eW (x), either correlated or independent.

heading: B REPRODUCIBILITY

heading: B.1 DATASET STATISTICS
text: For fairly comparison with previous work, we performance the image classification task on the three manually corrupted datasets, i.e., F-MNIST, SVHN, CIFAR-10, and one real-world noisy dataset Clothing1M. They have been widely adopted to study label noise problem. The detailed statistics are listed in Table 3.

heading: B.2 RUNNING ENVIRONMENT
text: All baselines and IF approaches are implemented in PyTorch, and tested on a machine with AMD EPYC 7282 16-core processors, 4 GeForce GTX-3090 Ti GPUs with 24GB memory size.

heading: B.3 ALGORITHMS
text: We summarize the IF algorithm to correct the loss and provide the pseudo codes in Algorithm 1. For instance-dependent Label noise generation, we provide pseudo codes with controllable noise rate are provided in Algorithm 2. This algorithm follows the state-of-the-art method (Xia et al., 2020; Zhu et al., 2021a), where the overall noise rate is τ .
Algorithm 1: Information Fusion Algorithm Input :Noisy dataset D̃ = (xn, ỹn)Nn=1; Noisy validation data; tolerant epochs t. Output :The robust neural network over noisy label.
1 Estimate the NTM T̂ (x) according to (Xia et al., 2020) ; 2 while Validation accuracy has increases in the last t epochs do 3 Calculate the PTM Ŵ (x) based on Equation (9) ; 4 Obtain the uncertainty for noise and PTM ; 5 Calculate the Kalman gain and Kalman transition matrix based on Equations (10) and (11) ; 6 Correct the loss function as posterior forward loss with Kalman transition matrix Wkm(x) ; 7 end
Algorithm 2: Instance-Dependent Label Noise Generation Input :Clean samples (xn, yn)Nn=1; Overall noise rate τ ; Number of classes c; Size of input features: 1× d. Output :Noisy samples (xi, ỹn)Nn=1.
1 Sample flip rate qi based on truncated normal distribution N (τ, 0.12, [0, 1]) ; 2 Sample the projection matrix W ∈ Rd×K based on normal distribution N (0, 12); 3 for n = 1 to N do 4 p = xnW // Genrate instance dependent flip rate ; 5 pyn = −∞ // Control the diagonal elements of the transition matrix ; 6 p = qn·softmax(p) // Make the sum of the off-diagonal elements to be qn ; 7 pyn = 1− qn // Set the diagonal element to be 1− qn ; 8 end

heading: B.4 LEARNING NOISE TRANSITION MATRIX VIA DECOMPOSITION
text: For the instance-dependent label noise, the NTM may be different for different samples. Thus, the complexity of NTM estimation is O(Nc2), which leads to ill-posed estimation problem. (Xia et al., 2020) introduces a part-dependent decomposition on the NTM, i.e., the noise of an instance depends only on its parts. Specifically, given L part-dependent transition matrix, e.g, P l ∈ [0, 1]c×c, where j = 1, · · · , L, the instance-dependent matrix can be approximated by a combination of partdependent transition matrix and the combination coefficients, defined as h(x) ∈ [0, 1]L, depend on the feature vectors. Then the instance-dependent transition matrix W (x) can be approximated as follows
T̂ (x) = L∑ l=1 hj(x)P l. (26)
where hj(x) represents the j-th elements of h(x) and satisfies the normalization condition ||h(x)|| = 1 for any features vector to maintain that the summation of column is 1 for NTM.
In this work, we follow (Xia et al., 2020) to learn the NTM in three steps: (i) learning combination coefficients h(x) via parts-based features representation (Lee & Seung, 1999; Tao et al., 2017); (ii)
learning the instance-dependent transition matrix by exploiting anchor points (Liu & Tao, 2015; Xia et al., 2019); (iii) learning the part-dependent transition matrices via minimization of reconstruction error for instance-dependent transition matrix.
(Xia et al., 2020) justifies the part-dependent decomposition assumption via parts-based representations. For example, (Hosseini-Asl et al., 2016) shows that the input features rely on parts-based representations in object recognition. Thus, it is natural to approximate the label noise in the part level. In display advertising, the delay feedback of the label (Ktena et al., 2019; Yasui et al., 2020) introduces label noise. Earlier the data is generated, the less probability the feedback exists. Thus, the label noise highly depends on part of features.
Since the instance-dependent NTM may be poorly learned, the slack variable trick (Xia et al., 2019; 2020) is also adopted to modify the instance-dependent transition matrix, which significantly improve the accuracy. However, the slack variable trick ignores the posterior information for each instance, which limits the accuracy. In addition, the variance of accuracy in (Xia et al., 2020) is large since there is not any guide information to modify the instance-dependent NTM except minimization of the corrected loss. In this paper, we claim that the measurement information is useful to modify the NTM. We will introduce how to estimate the PTM and information fusion in the next two subsections.
B.5 IMPLEMENTATION DETAILS
Warm-up Training Strategy Aiming to estimate the transition matrix, we adopt warm-up training strategy that training the neural network via cross entropy loss with noisy data. we use SGD as the optimizer with 0.9 momentum, 128 batch size, and 10 epochs. With the benefit of memorization effect (Arpit et al., 2017), warm-up training with appropriate epochs can guarantee that the predicted probability of neural network is reliable and can approximate underlying clean probability.
Training with Corrected Loss Our proposed PTM estimation relies on the neural network predicted probability. Although the warm-up training can guarantee the prediction accuracy for clean instances with the benefits of memorization effect, the prediction of the PTM estimation may still be not reliable. To ease the dilemma, we adopt sample selection to choose the reliable prediction and thus guarantee the reliable PTM estimation for specific instance. Similar to (Cheng et al., 2021), the small-loss criteria is adopted to select the instances with small training loss for training. Specifically, we select the instance via the weighted loss if the instance loss is less than the average loss over all classes.
For the instance satisfying the small loss criterion, we have confidence that the instance is clean, which means the PTM is identity. To put it simply, we sum the estimated PTM and identity and then adopt column normalization. It is worth noting that the noisy labels are adopted in forward propagation, which is highly different with previous transition matrix-based works (Xia et al., 2019; Liu & Tao, 2015; Patrini et al., 2017; Xia et al., 2020). Compared with hard sample selection based methods (Cheng et al., 2021), our method achieves the soft sample selection and adaptively correct the loss for “clean" instance.

heading: C ADDITIONAL EXPERIMENTAL RESULTS

heading: C.1 STABILITY ON SYNTHETIC NOISY DATASETS
text: We study the training stability via inspecting the test accuracy over epochs. Figures 6 and 7 show the test accuracy with respect to epochs on SVHN and F-MNIST datasets with noise rate IDN-10% to IDN-40%. It is seen that our proposed method IF can achieve better accuracy and stable training compare with baseline PTD, which implies that the PTM fusion can decrease the transition matrix estimation error and improve the training stability. T-Revision trick (-V method) proposed in (Xia et al., 2019) aims to compensate the NTM and thus improve the accuracy performance. It is seen that PTD-F-V can achieve better performance than PTD-F, which verifies that the learned NTM is not very accurate. For our proposed method, IF-V can achieve comparable performance than IF-F and demonstrates that transition matrix revision trick can not improve the performance since PTM has already been employed to compensate the transition matrix error.
Compared with different noise rates, it is seen that the accuracy of PTD method may decrease with respect to epochs, especially for large noise rate. Under large noise rate, the number of clean samples is limited and the neural network may already overfit noisy samples during warm-up training and thus hurt the transition matrix estimation. In this scenario, PTD method does not have the capability to adjust such poor NTM estimation and thus leads to the accuracy drop during training. Our proposed method IF and T-Revision trick both adjust the estimated transition matrix and mitigate the accuracy drop.

heading: C.2 EXPERIMENTAL RESULTS ON HIGH NOISE RATIO
text: We investigate the performance of IF for a high noise ratio compared with PTM and PTD. Table. 4 shows the test accuracy for high noise ratio IDN-70%, IDN-80% and IDN-90% in CIFAR 10 dataset. We have two observations: (1) Comparing with PTD at the noise ratios of IDN-80% and IDN-90%, our PTM could still outperform PTD. Although the network output is not perfect, the performance deterioration of PTD is still higher than PTM. (2) Our IF consistently delivers the most superior performance no matter under the low or high noise. That is because IF adaptively and linearly combines NTM and PTM in instance level and thus achieves more accurate estimation.

heading: C.3 EXPERIMENTAL RESULTS ON MORE BASELINES
text: Although many transition matrix-based methods are compared in Table. 1, we are also interested in several more baselines, including label smoothing (LS) (Szegedy et al., 2016), early-learning regularization (ELR) (Liu et al., 2020), (Berthon et al., 2021), progressive label correction (PLC) (202, 2021), confidence regularized sample sieve (CORES) (Cheng et al., 2021). It could be observed that our method can still achieve state-of-the-art performance compared with all other baselines. Table. 5 demonstrates the test accuracy over IDN-10% to IDN-50% label noise ratio in CIFAR 10. It could be observed that our method can still achieve state-of-the-art performance, based on which we can confidently validate the effectiveness of our method.

heading: D RELATED WORK
text: Label Noise Model There are three types of label noise model, including the random classification noise (RCN) model (Manwani & Sastry, 2013; Natarajan et al., 2013), the class-conditional label noise (CCN) model (Patrini et al., 2017; Xia et al., 2019; Zhang & Sabuncu, 2018; Liu & Guo, 2020) and the e instance-dependent label noise (IDN) model (Cheng et al., 2020; Xia et al., 2020; Cheng
et al., 2021; Berthon et al., 2021; Zhu et al., 2021a). Specifically, RCN means that each label is flipped independently with a constant probability, CCN assumes that the flip probability (noise rates) are the same for the instance with the same clean labels. IDN is the most general case, where the flip probability depends on its instance.
Loss Correction via Transition Matrix The transition matrix bridges the gap between the model predicted probability for noisy and clean data, which can achieve risk-consistent classifier with label noise. The transition matrix can be estimated via cross-validation method (Natarajan et al., 2013), anchor points assumption (Xia et al., 2019; Yu et al., 2018; Yao et al., 2020). Subsequently, the estimated transition matrix can be adopted to correct the loss function via forward correction (Patrini et al., 2017), backward correction (Patrini et al., 2017) and reweight (Xia et al., 2019; Liu & Tao, 2015).
Loss Correction via Sample Selection Due to the experiments observation– the neural network learns clean instances first (Arpit et al., 2017) and memorizes the instances gradually, there are lots of methods attempting to identify mislabeled training examples and then filter them out (Brodley & Friedl, 1999; Jiang et al., 2018; Han et al., 2018b; Yu et al., 2019; Angelova et al., 2005; Huang et al., 2019; Li et al., 2019). The core idea is to select the “small loss" samples as clean one.

Review summaries:
This paper provides new insight on estimating noise-transition matrix in the setting of instance-dependent label noise. Specifically, the authors observe that the traditional noise-transition matrix cannot well bridge the gap between underlying distribution and empirical distribution. Inspired by this observation, the authors propose to estimate the posterior noise-transition matrix. Further, authors use information fusion (IF) to linearly combine noise-transition matrix and posterior noise-transition matrix to obtain more accurate transition matrix estimation with theoretical guarantees. Experiments are conducted on CIFAR-10, SVHN, F-MNIST, and real-world dataset Clothing-1M. 

This paper proposes a new information fusion approach to deal with the instance-dependent label noise (IDN). Specifically, the authors claim that the essential problem caused by IDN is empirical, instead of underlying, data distribution mismatch during training, which I think is new and interesting. Therefore, the authors fuse posterior transition matrix (PTM) and noise transition matrix (NTM) and design an unbiased risk estimator. 

This paper proposes to estimate an input-dependent noise transition matrix (PTM). The authors make a strong assumption that the neural network output approximates the underlying clean probability and estimate the PTM by extracting the output of each training sample using the network. To enforce the network to have better probabilities, PTM estimation is applied iteratively and warm-up is used during the training. The experiments are conducted in CIFAR-10 with different noise ratio, as well as a real world dataset, i.e. Clothing-1M.