An Information Fusion Approach to Learning with Instance-Dependent Label Noise
Authors: Zhimeng Jiang
Kaixiong Zhou
Zirui Liu
Li Li
Rui Chen
Soo-Hyun Choi
Xia Hu
Venue: ICLR 2022 Poster (2022)
Keywords: Instance-dependent label noise
posterior transition matrix
statiscally consistent classifier

Abstract: Instance-dependent label noise (IDN) widely exists in real-world datasets and usually misleads the training of deep neural networks. Noise transition matrix (NTM) (i.e., the probability that clean labels flip into noisy labels) is used to characterize the label noise and can be adopted to bridge the gap between clean and noisy underlying data distributions. However, most instances are long-tail, i.e., the number of occurrences of each instance is usually limited, which leads to the gap between the underlying distribution and the empirical distribution. Therefore, the genuine problem caused by IDN is \emph{empirical}, instead of underlying, \emph{data distribution mismatch} during training. To directly tackle the empirical distribution mismatch problem, we propose \emph{posterior transition matrix} (PTM) to posteriorly model label noise given limited observed noisy labels, which achieves \emph{statistically consistent classifiers}. Note that even if an instance is corrupted by the same NTM, the intrinsic randomness incurs different noisy labels, and thus requires different correction methods. Motivated by this observation, we propose an \textbf{I}nformation \textbf{F}usion (IF) approach to fine-tune the NTM based on the estimated PTM. Specifically, we adopt the noisy labels and model predicted probabilities to estimate the PTM and then correct the NTM in \emph{forward propagation}. Empirical evaluations on synthetic and real-world datasets demonstrate that our method is superior to the state-of-the-art approaches, and achieves more stable training for instance-dependent label noise. 

Published as a conference paper at ICLR 2022
AN INFORMATION FUSION APPROACH TO LEARNING WITH
LABEL NOISE
Zhimeng Jiang', Kaixiong Zhou?, Zirui Liu², Li", Rui Soo-Huun Choi Hu² Texas A&M University, "Rice University, "Samsung Research America, 'Samsung Electronics
ABSTRACT
Instance-dependent label noise (IDN) widely exists in real-world datasets and usu- ally misleads the training of deep neural networks. Noise transition matrix (NTM) (i.e.. the probability that clean labels flip into noisy labels) is used to characterize the label noise and can be adopted to bridge the gap between clean and noisy un- derlying data distributions. However, most instances are long-tail, i.e., the number of occurrences of each instance is usually limited, which leads to the gap between the underlying distribution and the empirical distribution. Therefore, the genuine problem caused by IDN is empirical, instead of underlying. data distribution mismatch during training. To directly tackle the empirical distribution mismatch problem, we propose posterior transition matrix (PTM) to posteriorly model label noise given limited observed noisy labels, which achieves statistically consistent classifiers. Note that even if an instance corrupted by the same NTM, the intrinsic randomness incurs different noisy labels, and thus requires different correction methods. Motivated by this observation. we propose an Information Fusion (IF) approach to fine-tune the NTM based on the estimated PTM. Specifically, we adopt the noisy labels and model predicted probabilities to estimate the PTM and then correct the NTM in forward propagation. Empirical evaluations on synthetic and real-world datasets demonstrate that our method is superior to the state-of-the-art approaches, and achieves more stable training for instance-dependent label noise.
INTRODUCTION
Data labels annotated from human efforts, such as crowdsourcing Yan et al. 2014 Chen et al. 2017) and online queries (Divvala et 20.44 may be heavily noisy practice (Wei et al. (022) To make it worse, the label noise stemmed from human annotations is often instance-dependent
For example, the images close to the decision boundary are usually prone to be mislabeled (Zhang
et al. 2021b Zhu et al. 20216). On the other hand, the remarkable success of deep neural networks (DNNs) on supervised learning tasks heavily relies on the expressive power and a large number of
data with accurate labels. Unfortunately, deep neural networks memorizes noisy labels leading to
poor generalization (Zhang et al. 2017). is challenging to learn with practical insennne 7hu et al.
label noise (IDN) due to the hidden and complicated label noise properties LLiu 2021
2022b Cheng et al. 20216 Zhu et al. 2022a).
The methods dealing with noisy labels fall into two lines, including heuristically identifying noisy samples and statistical label noise modeling. The training of deep neural networks often learns clean labels first (Arpit et al.) 2017) and then gradually memorizes noisy labels, which is recognized as the memorization effect. Based on the general memorization effect, the heuristic methods are all designed by following the anomaly detection strategy: identify noisy samples based different behaviors (e.g., loss values) berween clean and noisy samples during training Cheng et al.) (2021a).
The typical methods contain sample selection (Yu al. 2019 Han al. 2018b). reweight samples (Cheng et al., 202 Jiang et al. 2018 Ren et al.) 2018), label correction (Ma et al.) 2018)) Tanaka ct al. 2018), and regularization (Han et a)) Although these algorithms empirically work well.
the reliability cannot be guaranteed explicitly without modeling label noise.
Another line of works relies on noise transition matrix (NTM) to model label noise statistically (Xia et al.] 2019 2020 Patrini et al. 2017) by quantifying the probabilities that clean labels flip into noisy
"Corresponding author (sonhyunc gmmalllcmmmmlll.ommmm

---

Published as a conference paper at ICLR 2022
labels. Although the NTM-based methods possess theoretical guarantee, NTM estimation for each instance under IDN is pretty challenging. To ease the estimation, some unrealistic assumptions have been made NTM, including instance-independent transition matrix (Liu & Guo 2020; Wei & Liu 2021 et al. 2021), symmetric transition matrix (Menon et al. 2018), upper bounded noise rate (Cheng et al. 2020), and part-dependent label noise (Xia al., 2020). However, under the complex IDN, the empirical noise distribution could be highly different from the underlying noise distribution. For example, in Figure I the underlying and empirical noisy distributions for long-tail instances are different since the empirical noisy label can be either the same as or different from the clean label. Additionally, observed noisy labels provide inductive bias toward label corruption. other words, the genuine problem arising from IDN is the empirical, instead of underlying, clean and noisy distribution mismatch problem.
To mitigate the empirical distribution mismatch problem, we propose the posterior transition matrix (PTM) to model label noise given the observed noisy labels. We adopt PTM to provably bridge the gap between clean and noisy underlying data distributions, and empirical distribution of all anchor points (i.e., data points that belong to a specific class almost surely (Xia et al. 2020 simultaneously, We also provide an eesy-to-comuute PTM estimation method under the low label noise condition. To further extend the applicability. motivated by Kalman filtering (Kalman 1960). we propose the information fusion (IF) method linearly combine the estimated NTM and PTM, which achieves lower transition matrix estimation error. We empirically show that the proposed IF method can achieve higher accuracy and more stable training. The main contributions of this work are summarized below.
We propose new concept of PTM achieve statistically consistent elassifiers for underlying distribution mismatch and anchor point empirical distribution mismatch simultaneously.
We propose simple yet effective PTM estimation method based observed noisy labels under the condition low label noise ratio. To extend the applicability, we propose an IF method, which combines the estimated NTM and PTM to achieve lower estimation error with both theoretical and experimental justifications.
We experimentally demonstrate that the proposed IF method achieves higher accuracy and more stable training under synthetic and real-world label noise.
Clean Distribution
Noisy Distribution
Underlying
0
0.3
0.7
D
0.7 0.3 Noise TM
0.7
Cise
Empirical
0
Case
Cleam label
Noiey Tabel
Clean Lahel/Prob
Noisy Label/Prob
Noise
Posterior
Figure 1: Illustration of the thought experiment. Assume we use identified noise transition matrix (NTM) to corrupt the instance with clean label [o, 1]. From the underlying distribution perspective, the noisy underlying distribution is [0.3, 0.7]. From the empirical distribution perspective, the label of this sample is statistically either the same as (case 1) or different from (case 2) the clean label, leading to different PTM. In other words, the observed noisy labels (posterior information) provide inductive bias of label correction, which motivates the notion of PTM.
PRELIMINARIES
Throughout this paper, we adopt uppercase letters to denote random variables, and lowercase letters to denote particular realization of the random variables. We target the c-class classification problem with instance-dependent label noise (IDN). Let P(X, Y' be the underlying clean distribution of the random variables (X, Y), where X and Y represent the instance and clean label, respectively. In real-world scenarios, clean labels are usually corrupted, and thus only noisy labels, denoted as Y. observed. Given set of N training instances denoted by Î (( Fn) ] n=1. where x,, is the
2

---

Published as a conference paper at ICLR 2022
instance vector of the n-th sample and yn [c] == {1.... .c} is the corresponding observed noisy label, our goal is to predict the clean label Un for any given instance nn.
Data distribution. We assume that the unobserved clean samples (æ:,; Un) and available noisy samples (z,, 9n) are drawn from the unknown underlying clean distribution P(X, Y) and the under- lying noisy distribution P(X, Y), respectively. For the noisy samples D, we may approximate the underlying noisy distribution by the empirical noisy distribution, i.e., P Y)
Σ 6(X nn, y = in where 8(X- Fn Ý = Un) is a Dirac mass centered (In:: Dn Since clean data samples are unobserved, we define the posterior empirical clean distribution P(Xx, YID) as the inferred posterior empirical clean distribution from the noisy samples D. Based on Bayes' rule, the posterior empirical clean distribution is given by P(X,YID) = N > 2 5(X = In. Y =
G. JPY = yn|l IÝ = Dn. X = x.n)
Noise transition matrix T(x)). To describe the corruption process of the clean label, the NTM T(x) gexe is defined as Ti P( = =j i æ). which represents the transition probability of flipping the instance x' label from the clean i-th class to the noisy j-th class. The
probability of the underlying noisy label Y given the instance satisfies:
P(Ý =j = =x) Σp(
p(y - j|y = i,
x) = i|X x)
Σ Tisla)(( i|X xx)
The NTM T(x) hence bridges the gap between the underlying clean label probability P(Y
2) [P(Y = 1|X -x....,," c|X = =3)] and underlying noisy label probability
P( X x) = -[P(Y- = 1|X = x)... .... = c|X = 20] given the instance æ. i.e.,
P( z)= P
Loss correction method. Let function f(-) represent neural network and f fax) denote the c- dimensional output probability for instance æ. where the 7th index of the output (i(x)) represents the predicted probability for class i. The common approach is to minimize the cross-entropy (CE) loss ([f())). y) == (z)( to force the output fy (z) to approximate 1. However, the label noise may mislcad a deep learning model. The existing NTM-based methods first estimate NTM T(x) and then adopt it correct the loss function. For example, in the forward correction procedure PPtrinn et al. (017). the estimated NTM adopted to corrupt the predicted probability f f(x) i.e., the corrupted predicted probability is f(x) = T(x) f(x), and then the corrupted predicted probability enforced to approximate the noisy label ÿ. Suppose T(x) non-singular and the loss function is proper and composite. The forward loss correction can achieve a consistent classifier, i.e., the optimal classifier for the corrected loss with respect to the underlying noisy distribution is the same as that for the CE loss with respect to the underlying clean distribution:
arg min E y (x. F,),,, (((x)))))T) [(X)) = arg min E Bx....eaaaan)
(1)
3 LEARNING WITH INFORMATION FUSION
Even though the existing loss correction methods could achieve consistent classifiers theoretically, their performances are still undesirable in practice. Since a deep learning model is often trained on
the empirical distribution with limited samples, the NTM cannot correctly bridge the empirical clean distribution and empirical noisy distribution, i.c., P(Y j|X x) + Di=1 Tey (x) Y = i|XX
x). In other words, the baseb method may not work well in practice due to the empirical clean and noisy distribution mismatch problem in IDN.
With the goal of utilizing the observed noisy labels, we propose a concept, named posterior transition matrix (PTM), to describe the transition probabilities given the observed noisy labels (formally defined in Section (.1).. The motivation for PTM stems from a simple thought experiment as shown in Figure [[] which shows that PTM model the empirical label noise better than NTM. Figure 2 illlsstrates the
3

---

Published as a conference paper at ICLR 2022
xx,
NTM Estimation
T(x)
Neural Network Training
Noisy
Softman
Training
(x)
PTM
W(x
Linear Combination
W
km
Posterior
Samples
Estimation
Loss
Neural Network Update
Figure 2: The overview of IF. NTM can be estimated beforehand based on the previous work (Xia ct al. 2020). PTM estimation is iteratively obtained based on model prediction and observed label. Subsequently, IF adopts linear combination for NTM and PTM to reduce the estimation emror. Finally, a posterior loss function is proposed to tackle IDN.
overall framework of IF, including NTM and PTM estimations (Section 3.2 information fusion via linear combination (Section 3.3). and posterior loss function (Section 3.1.
3.1 THE Loss CORRECTION METHON
The main goal to train c-class neural network classifier f f(x w) to predict the clean label probability P(Y Since only the noisy labels are observed, there is gap between the clean and noisy label, described via NTM (Goldberger Ben-Reuven (017).
Motivated by the observed noisy labels (i.e., posterior information), we define the PTM W(x) to describe the posterior clean label probability given noisy labels, where (x) IP(Y i j, x). We provide the relationship between the PTM W (a:) and NTM (x) via Bayes' rule:
(ay(#) PP(Y = i =
PP(Y x|= = T)T;; E P(Y = T))
(2)
PP(Y =|===(x))
Notice that the summation of any column is for PTM (x), while the summation of any row is 1 for NTM Twx). Subsequently, provide posterior reweight loss correction method via NTM.
Definition (Posterior reweight loss) Assume the model prediction is f(x) for noisy sample (xx, Ū)
and (xx) is the PTM associated with the noisy sample. The posterior reweight loss is defined as
1p-rew G. f(x))))))))))())))))))))))))ff
(()(())((1,((z)))
We next analyze the property of the posterior reweight loss and provide the theoretical justification. Specifically, we analyze the expected risk (f) = (,,,,))7(x;;,) Ú. / (xx) and
empirical risk Ro(f) = Em lo-rew (Jr. f under the noisy samples.
Theorem 3.1 (Statistically Consistent Classifier) The posterior reweight loss can achieve an con- sistent classifier for the underlying distribution and empirical distribution.
(i) For the underlying distribution, the expected risk satisfies
arg min EE (e (g-rew (ū, [(x)) arg min
(ii) For the anchor point samples (Zaps Dap) with the underlying clean probability P(Y yap = Dap), the empirical risk satisfies le-rew (Baps If (xxp)) = U(yaps fx For the empirical distribution, the empirical risk satisfies
arg min E (2.))) (X.8) [pp-rew (Tnn [2w))) = arg min E ((x,)))( [[y, ((w))) (5)
We also show that the PTM can be adopted in the forward manner. The definition of the posterior reweight loss is given below.
4

---

Published as a conference paper at ICLR 2022
Definition 2 (Posterior forward loss) Assume the model prediction is f(x) for noisy sample (xx D) and (x)) is the PTM associated with the noisy sample. The posterior forwardd loss is defined as
(U. = (ū. Wes(<)fi(z)))
We next analyze the property of the posterior forward loss and provide the theoretical justification.
Lemma 3.2 The posterior forward loss is not larger than the posterior reweight loss
1p-rew T. ((+()))) 2 [1.
Theorem 3.3 (Statistically Consistent Classifier) Given that the loss function My f) is convex with respect to f (the convex condition can be commonty satisfied, e.g., the cross-entropy loss) and that the minimum expected risk X (f) can achieve O, the posterior forward loss can achieve an consistent classifeer for the underlying distribution,
arg min y ((a.))) In- (()) = arg min B.a.)--pX..,
By comparing the posterior forward loss with the forward loss (Patrini et al. 2017). it can be seen that both of them correct the output of neural network with statistically consistent risk guarantee, while the difference falls into the different transition matrices used in the loss correction. In addition, the statistically consistent risks for the forward loss and posterior forward loss both require accurate transition matrix estimation. However, the NTM and PTM estimation is highly challenging since label noise is instance-dependen in reality. Such an observation motivates us to combine both NTM T(x) and PTM W (x) to correct the loss.
Motivated by Kalman filtering (Kalman| 1960). a famous estimation method combining prior knowl- edge and measurement information, propose an Information Fusion (IF) approach to tackle instance-dependent label noise via loss correction. Specifically, we first claim that there is prior knowledge information and measurement information in the label noise, where prior knowledge information and measurement information correspond to the estimated NTM and PTM, respectively.
3.2 PTM ESTIMATION
PTD PTD-F PTM
PTMM- MIF IIF-
Essimatonn Liror
The posterior loss correction requires knowing PTM (x) given any instance æ. However, the PTM is unknown and needs to be estimated, In this subsection, we provide a simple yet effec- tive PTM estimation method under the condition that the neural network output fXX probability well approximate the underlying probability X())X), i.e., f(x) W(X x)) (Y))
8 8 8 B
IDN Ratio
Figure 3: Estimation error for CIFAR10
æ). Intuitively, this condition seems strong since
neural network is prone to overfit noisy data. To achieve this condition, the warm-up training strategy and iterative PTM estimation are adopted. Warm-up training can make "good" neural network outputs approximating the empirical clean distribution since the network fits clean samples in the beginning of training (Liu et al.) 2020). Subsequently, we iteratively estimate the PTM for loss correction during training.
For the case that the number of occurrences of instance is large, the empirical noisy distribution converges to the underlying noisy distribution, i.c., P(Y x) - P X x). For a long-tail instance with only a single occurrence (x, ū). the empirical noisy label distribution is always one-hot. Thus, this condition requires that the output of neural network f(x) should approximate to the underlying posterior clean probability, which is quite strong.
Under this condition, we propose PTM estimation method based the noisy labels and the model predicted probability. Compared with the NTM estimation method (Xia et al. 2019 2020). the proposed estimation method does not require the anchor point assumption that data points
5

---

Published as a conference paper at ICLR 2022
belong to specific class almost surely. Specifically, the PTM can be estimated ( (xx i = J. otherwise is O. For the case that the number of occurrences is larger than one, the PTM estimation is ill-posed since the number of unknown variables c² is larger than the number of constraint e (i.e., number of classes). Thus, we choose the PTM with the minimum Frobenius norm, which is consistent with the case of having only single occurrence. Specifically, we have the following theorem.
Theorem 3.4 (Posterior transition matrix estimation) The optimal solution to finding the PTM with the minimal Frobenius norm, i.e.,
((x) arg min ||W]|
s. s. (x))((())) = 100=ffee f(æ), for dx
given below:
W(x) f( |P 1
Theorem 3.4 provides the PTM estimation for general empirical noisy label distributions. For the case of instance æ with only a single occurrence, the empirical noisy distribution satisfies ||P((|>)) and achieves consistent estimated PTM in Equation (8).
3.3 INFORMATION FUSION
Section 3.2 introduces the PTM estimation method based on the observed noisy labels. However, the condition that neural network approximates clean labels could still be strong even after the warm-up strategy and iterative estimation are adopted, and PTM estimation error could be large for large IDN rates. To further reduce the estimation error, motivated by Kalman filtering, we propose the information fusion (IF) approach to obtain more accurate transition matrix estimation via weighted average of PTM estimation and NTM estimation with both intuitive and theoretical justifications] Intuitively, for each instance, the estimated NTM and PTM may have different estimation accuracy, and, therefore, possible to obtain more accurate transition matrix estimation by adaptively and linearly combine these two matrices. Specifically, we first quantify the estimation uncertainty and assign higher weight for the estimation with lower uncertainty. this way, more accurate estimated transition matrix can be generated.
Before illustrating the information fusion algorithm, we need to quantify the transition matrix estimation eтror without ground truth. Suppose posterior forward loss is adopted. For the estimated NTM (xx the corrupted model predicted probability is f(x) fT (xx) If the corrupted model predicted probability is accurate, the noisy label Y satisfies e-dimension Bernoulli distribution with parameter f(x), i.c., Y ~ -ernouuli ( (x)f())) where Bernoulli(-) represents multi- dimension Bernoulli distribution. Furthermore, we define the uncertainty af for transition matrix f(x) the trace of the covariance matrix for Ý, i.e., dx(x) Tr(Ky(x)), where Tr(.) means the trace of a matrix, Kş(x) represents the covariance matrix of random variable Y. For the multi-dimension Bernoulli distribution, we have the covariance matrix as follows,
Ky = B[(" - E())) B[E)) T] diag(? (x))())) - AT (x){())((a)) T(()).
Taking trace operation on the covariance matrix, we have the uncertainty for NTM as (T(2)) I (z))((=) [ (x) Similarly, the uncertainty for the PTM is given by (w(x) = f=))(l (x) Once the uncertainty has been established, the two estimated transition matrices into a Kalman transition matrix, defined as (x),,, via weighted average operation. Mathematically, the Kalman transition matrix is given by
further integrate
Wem (x) (1 - X(=))((())) + A(x)
'The core idea for more accurate estimation is similar to statistical efficiency (Gogg et al. 2020 via generating two "auxiliary" estimators. The key difference is the generation manner: IF generates NTM and PTM from the prior and posterior perspectives, while statistical efficiency generates different noisy labels.

---

Published as a conference paper at ICLR 2022
where the Kalman gain X(x) is carefully selected to minimize the estimation error. Let the true PTM be (x)), where Wij (z) = 1 if y and ÿ = j. otherwise O. Subsequently, we define the reconstruction error for NTM and PTM as et (xx (*(x) (a)) and (w) (+))
respectively. The parameter X(x) is determined via minimizing the mean square reconstruction error. We theoretically justify the superiority of IF for a general scenario.
Theorem 3.5 (Theoretical justification of IF) Let the correlation coefficient between reconstruc- tion error er(z) and (w) be con(x:) E [- The trace of the covariance matrix for error ex(x) and ew can be quantified by 0z((x) and (x)), i.e., [r(eg(z)e) (x)) = (f(x) and Tr(ew (x)e (x)) = =ow(x) For the Kalman transition matrix Wkm let the reconstruction error be Emoel (xx))) Tr ((W)((x))-
-hcc)-WWhaa Wnm (
- km (x))) Then the optimal Kalman gain
minimize the reconstruction error is given by
(x((x) - cou(2) (r
(x))) = arg min Liuse (A) (2(2)) (a)
(10)
and the minimum reconstruction error satisfies Susse (X* (x)) S mi)(a,
Theorem 3.5 implies that the Kalman transition matrix Wkm (a) provably achieves lower estimation error for general eT(x) and ew (2)), either correlated or independent.
4 EXPERIMENTS
In this section, empirically evaluate the effectiveness and stability of our IF approach synthetic and real-world noisy datasets. We aim to answer two questions as follows. Q1: Compared with the state-of-the-ar art transition matrix based methods, can IF achieve higher accuracy? Q2: Can IF lead to more accurate and robust transition matrix estimation and more stable training?
4.1 EXPERIMENTAL SETTINGS
Datasets. We verify the superiority of IF on three manually corrupted datasets, i.e., F-MNIST, SVHN, 10, and one real-world noisy dataset Clothing M. M. The first three datasets contain clean data, and we manually corrupt the labels of the training datasets by following (Xia et al. 2020). IDN-T means that the controlled noise rate is T. All experiments on those datasets with synthetic instance-dependen label noise are repeated five times. The real-world dataset Clothing MM has 1M images with world noisy labels and 10k images with clean labels for testing. In the experiments, we leave out 10% of the noisy training samples as a noisy validation set for model selection.
Baselines. We compare the proposed IF method with (i) CE, which trains a standard deep network with the cross- loss on noisy datasets; (ii) DMI, novel information-theoree robust loss function for instance-independent label noise (Xu et al. 2019); (iii) Forward (Patrini et al., 2017). Reweight (Liu & Tao 2015), and T-Revision (Xia et al. 2019); (iv) part-dependent transition matrix
(PTD) (iia al.| 2020); (v) instance-level forward correction (ILFC) (Berthon et al. 2021 All
these approaches utilize the NTM to correct the loss function.
Implementations. We choose standard neural networks and optimizers. Specifically, we use R8sNet-18 network for F-MNIST, ResNet-34 network for SVHN and CIFAR-10. For the optimization, we first use SGD with 0.9 momentum, 10 wwight decay, 128 batch size, 50 epochs and an initial learning rate of 10 to initialize the network. Then, we adopt Adam optimizer and 5 x 10-7 learning rate to learn the NTM following PTD (Xia et al. 2020). Once the NTM is obtained, we retrain the neural network.
the forward propagation, the Kalman transition matrix obtained and then adopted correct the loss function. In addition, the revision trick (Xia et al.) 2019) is also adopted for our proposed IF method, which introduces the slack variables AW for the estimated
"The ideal estimated transition matrix is to approximate the empirical (actual) transition matrix W*(z), instead of the expectation of label transition matrix. Therefore, the estimation error is defined to characterize the hias-variance tradeoff for transition matrix estimation.
7

---

Published as a conference paper at ICLR 2022
Table 1: Means and standard deviations (percentage) of classification accuracy with different instance- dependent label noise levels. Methods with "-F" adopt the Forward correction loss; methods with ". "-V" mean that the transition matrices are revised via the slack variable trick.
Dataset
Method
IDN-
IDN-200
IDN-0%
IDN-40%
IDN-50%.
CE
90.77 1 0.45
90.23 1 0.62
86.33 1 1.34
65.66 1 1.65
48.01 1 4.59
DMI
93.51 + 1.09
93.22 + 0.62
91.78 + 1.54
69.34 + 2.45
48.93 + 2.34
Forward
90.89 - 0.60
90.65 + U.27
87.32 + 0.59
78.46 1: 2.58
46.27 :: 3,90
Reweight
92.49 0.44
91.09 0.34
90.25 0.77
84.48 + 0.86
45.46 +: 3.56
T-Revision
94.24 0.53
94.00 = 0.88
93.01 - 0.83
88.63 + 1.37
49.02 4.33
SVIN
ILFC
92.08 0.12
91.67 0.16
90.80 1 0.15
89.16 0.67
65.69
6.54
PTD-F
91.92 0.87
90.21
87.11 1.43
81.10 2.78
67.81 5.41
PTD-E-V
92.64 0.81
9.25
92.71 0.44
90.71 0.91
80.35 5.46
PTM-F
93.78 0.12
89.25 = 0.09
81.89 2.82
67.30 + 3.07
PTM-F-V
93.89 + 0.10
92.72 1 0.18
88.97 + 0.26
81.14 1: 2.06
68.95 +: 3.02
IF-
94.70 0.14
94.01
+ 0.31
93.33 + 0.15
91.60 0.29
85.57 + 3.17
IF-F-
94.63 0.12
93.92 + 0.26
93.21 + 0.20
91.66 + 0.23
86.24 + 2.61
0.29
68.21 + 0.72
60.48 + 0.62
49.84 1.27
38,86 2.71
DMI
75.02
69.89 1 0.33
61.88 1 0.64
$1.23
1.18
41.45 1.97
Forward
68.99 0.62 0.24
60.21 + 0.75
47.17 2.96
40.73 2.09
Reweight
68.42 0.23
62.58 1 0.46
50.12 0.96
41.08 2.44
T-Revision
74.61
69.32 + 0.64
64.09 + 0.37
50.38
0.87
42.57 +
CIFARIO
ILFC
80.22
0.09
73.27 = 0.24
57.00 4.07
PTD-F
79.77
74.96 0.71
70.68 + 0.81
61.92 + 1.59
45.34 + 4.67
PTD-EV
80.08 0.86
74.67 0.36
71.66 = 1.05
62.45 + 1.73
46.16 1
PTM-F
0.36
74.81 +: 0.81
70.03 + 0.38
63.48 1 0.38
51.03 3.08
PTM-F-V
78.58 0.31
75.06 + 0.56
69.83 + 0.58
62.69 + 0.69
50.53 + 3.41
IFF
80.92 0.28
19.5810.55
74.34 1 0.86
68.21 2.21
50.07 3.95
IF-FV
80.94 0.43
79.54 0.45
74.67 + 0.92
68.03 2.90
52.34 1.31
CE
88.54 0.31
88.38 1 0.42
84.22 + 0.35
68.86 0.78
51.42 0.66
DMI
91.98 = 0.62
90.33 + 0.21
84.81 + 0.44
69.01 + 1.87
51.64 + 1.78
Forward
89.05 1 0.43
88.61 1 0.43
84.27 1 0.46
70.25 1.28
57.33 1: 3.75
Rewcight
90.33 + 0.27
89.70 + 0.35
87.04 + 0.35
80.29 + 0.89
65.27 + 1.33
T-Revision
91.56 1 0.31
90.68 - 0.66
89.46 + 0.45
87.21 + 1.20
74.22 1 0.81
F-NNIST
ILFC
91.84 + 0.04
90.47 = 0.29
87.07 + 2.89
86.68 0.36
65.93 0.29
PTD-F
91.13
19.511..6
89.01 1 0.40
89.18 0.20
75.37
3.63
PTD-E-V
91.71
0.28
91.18 + 0.30
90.62 + 0.08
89.38 0.69
77.80 7.29
PIM-F
91.39 + 0.45
90.66
88.48 + 0.09
83.20 1.02
63.85 3.92
PTM-F-V
91.73 + 0.32
91.07 + 0.14
88.87 + 0.62
82.20 + 0.92
64.23 + 4.73
IFF
91.58 -
91.23 1 0.09
90.10 = 0.74
89.37 + 0.26
82.78 = 0.21
IF-F-
92.08 0.06
91.73
1 0.19
90.82 1 0.33
90.32 1 0.12
83.33 1: 0.73
Kalman transition matrix. Specifically, the slack variable AW initialized as zero elements in the experiments and can be optimized during the training. To guarantee valid transition matrices, We first project their negative entries of W Kmm (x) + AW to zero and then normalize the summation of each row be 1. Following (Xia et al. 2020
do not any data augmentation technique in the experiments. For ClothingIM, we use a ResNet-50 pre-trained model on ImageNet. We do not use any clean data to learn the transition matrices and classifiers. After the NTM ((x) is obtained SGD with 0.9 momentum, 10-3 weight decay, 32 batch size,
according to (Xia et al. 2020), and run with 10-3 learning rate and 10 epochs. For the evaluation, we adopt the test accuracy of the epoch with best accuracy on validation datasets.
4.2 EXPERIMENTAL RESULTS
Accuracy on synthetic and real-world noisy datasets. To answer Q1, Figure 3 shows the esti- mation errors PTD. PTM (i.c., only adopt PTM in loss correction) and IF methods with noise rate IDN-10% to IDN-40% where the estimation error is defined the average difference of the transition probability err N Σm=
WT, W.,,),) ). It can be observed that IF can achieve the lowest estimation eтror over all noise ratios") Table I reports the classification accuracy on the SVHN, CIFAR-10 and F-MNIST with noise rate IDN-10% to IDN-50%, where the highest accuracy is bold faced. We can observe that IF achieves better performance than all baselines across all the three datasets and five noise rates. For example, in CIFAR-10, the improvements over the strongest baseline
Thhe transition matrix estimation for instance-dependent label noise is more challenging than the counterpart for class-dependent label noise et al. 202
8

---

Published as a conference paper at ICLR 2022
Table 2: Classification accuracy on ClothingIM. In the experiments, only noisy samples are exploited to train and validate the deep model.
CE
Forward
Reweight
T-Revision
PTD-F
PTD-F-V
ELR
IF-F
IF-F-V 72.29
68.94
69.98
70.82
71.27
70.37
70.62
71.86
72.04
1.07%, 0.83%, 4.03%, 9.22%. 9.89% for noise IDN-10% IDN-50%, respectively. The higher accuracy implies that the posterior information is more important for higher noise rates. In addition, the methods IF-F and IF-F-V are comparable across the three datasets and noise rates, which demonstrates that the revision on the transition matrix is not significantly beneficial compared with the method PTD. This observation validates the effectiveness of posterior information on transition matrix modification. For the real-world dataset Clothing M, IF outperforms the baselines as shown in 2 Our IF method achieves 0.5% improvements over the strongest baseline early-learning regularization (ELR) (Liu al. 2020).
Transition matrix estimation evaluation. We study the transition matrix estimation error in the training data for IF and PTM to answer Q2. Figure shows the training accuracy and estimation error of each epoch on the CIFAR 10 dataset with noise rate IDN-10% to IDN-40%, respectively. The shadow area represents the standard deviation. For all IDN rates, IF can achieve higher training accuracy and lower estimation error compared with PTM.
Figure 4: The training accuracy and transition matrix estimation error on the CIFAR-10 10 dataset with IDN noise 0.1, 0.2, 0.3, and 0.4.
Stability synthetic noisy datasets. We study the training stability via inspecting the test accuracy over epochs to further answer Q2. Figure 5 shows the test accuracy with respect to epochs on the CIFARIO dataset with noise rate 10N-10% IDN-40%. respectively. The shadow the standard deviation of the accuracy on the same epoch across five experiments. We can clearly observe that, both low-level and high-level noise, IF shows higher accuracy in more stable way since the posterior information of each instance provides reliable guidance on loss correction.
represents
Figure 5: The test accuracy on the CIFAR-10 dataset with IDN noise 0.1, 0.2, 0.3, and 0.4.
5 CONCLUSIONS
This paper presents a novel notion of PTM and simple yet effective learning paradigm named information fusion, which trains deep neural networks under eetance-deeenden label noise. The main goal of PTM is to characterize the label noise via limited observed noisy labels, which can bridge the empirical noisy and posterior empirical clean distributions. To effectively combine the benefits of NTM and PTM, propose an information fusion approach integrate both transition matrices to correct the loss function. Experiments on synthetic and real-world noisy datasets show that IF can achieve higher accuracy and more stable training compared with state-of-the- methods, especially for high noise rates. In the future, we can extend to incorporate more prior knowledge of the transition matrix, e.g., sparsity low rank, into the end-to-end learning algorithm.

---

Published as a conference paper at ICLR 2022
REFERENCES
Anelia Angelova, Yaser Abu-Mostafam, and Pietro Perona. Pruning training sets for learning of object categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OS). volume 1, pp. 494-501. IEEE, 2005.
Devansh Arpit. Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 2017.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependee label-noise learning possible. In International Conference Machine Learning. PMLR, 2021.
Carla E Brodley and Mark A Friedl. Identifying mislabeled training data. Journal of artificial intelligence research, 11:131-167, 1999.
Guangyong Chen, Shengyu Zhang, Di Lin, Hui Huang, and Pheng Ann Heng. Learning to aggregate ordinal labels by maximizing separating width. In International Conference on Machine Learning, pp. 787-796. PMLR, 2017.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In In Proceedings of the 9th International Conference on Learning Representation, 2021a.
Hao Cheng, Zhaowei Zhu, Xing Sun, and Yang Liu. Demystifying how self-supervised features improve training from noisy labels. arXiv preprint arXiv:2110.0902 2021b.
Jiacheng Cheng. Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning, pp. 1789-1799. PMLR, 2020.
Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly- supervised visual concept learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3270-3277, 2014.
Jacob Goldberger and Ehud Ben-Reuven Training deep neural-networks using noise adaptation layer. 2017.
Chen Gong, Jian Yang, Jane J You, and Masashi Sugiyama. Centroid estimation with guaranteed efficiency: general framework for weakly supervised learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2020.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor W Tsang, Ya Zhang, and Masashi Sugiyama. Masking: new perspective of noisy supervision. Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. -5551, 2018a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama. Co-teaching: robust training of deep neural networks with extremely noisy labels. In Proceedings the 32nd International Conference on Neural Information Processing Systems, pp. 8536-8546, 2018b.
E. Hosseini-Are Asl,, J. M. Zurada, and O. Nasraoui. Deep learning of part-based representation of data using sparse autoencoders with nonnegativity constraints. IEEE Transactions on Neural Networks and Learning Systems, 27(12))2486-244 24988 2016.
Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. 02u-net: A simple noisy label detection approach for deep neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3326-3334, 2019.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference Machine Learning, pp. 2304-2313, PMLR, 2018.
10

---

Published as a conference paper at ICLR 2022
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Huszár, Steven Yoo, and Wenzhe Shi. Addressing delayed feedback for continuous training with neural networks in ctr prediction. In Proceedings of the 1 13th ACM Conference on Recommender Systems, pp. 187-195, 2019.
Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factoriza- tion. Nature, 401(6755):788-79,, 1999.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-
supervised learning. In International Conference on Learning Representations, 2019.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end
label-noise learning without anchor points. 2021.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda Early-learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on paltern analysis and machine intelligence, 38(3)447-461, 2015.
Yang Liu. The importance of understanding instance-level noisy labels. In International Conference on Machine Learning. PMLR, 2021.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, pp. 6226-6236. PMLR, 2020.
Xingjun Ma. Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijew- ickrema, and James Bailcy. Dimensionality-drive learning with noisy labels. In International Conference on Machine Learning, pp. 3355-3364. PMLR, 2018
Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics, ((13):146-11551 2013.
Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. Learning from binary labels with instance-dependent noise. Machine Learning, 107(8): 1561-1555, 2018.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. NIPS, volume 26, pp. 1196-12044 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2233-2241. IEEE, 2017.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pp. 4334-4343. PMLR, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-28888 18-822 2016.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018
Dapeng Tao, Dacheng Tao, Xuelong Li, and Xinbo Gao. Large sparse cone non-negative matrix factorization for image annotation. ACM Transactions on Intelligent Systems and Technology (TIST), 8(3):1211, 2017.
Jiaheng Wei and Yang Liu. When optimizing f-divergence robust with label noise. In In Proceedings of the 9th International Conference Learning Representation, 2021.
11

---

Published as a conference paper at ICLR 2022
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference Learning Representations, 2022.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33, 2020.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L_dmi: A novel information-theone loss function for training deep nets robust label noise. In NeurlPS, pp. 6222 0233,, 2019.
Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine learning, 95(3):291-327, 2014.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Towards mixture proportion estimation without irreducibility. arXiv preprint arXiv:2002.03673, 2020.
Shota Yasui, Gota Morishita, Fujita Komei, and Masashi Shibata. A feedback shift correction in predicting conversion rates under delayed feedback. In Proceedings of The Web Conference 2020, pp. 2740-2746, 2020.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pp. 7164-7173. PMLR, 2019.
Xiyu Yu, Tongliang Liu, Mingming Gong, Kayhan Batmanghelich, and Dacheng Tao. An efficient and provable approach for mixture proportion estimation using linear independence assumption. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4480-4489, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature dependent label noise: a progressive approach. 2021a.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In In Proceedings of the 9th International Conference on Learning Representation, 2021b.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8792-888 8802, 2018.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10113-10123, 2021a.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative anchor points when learning with noisy labels. In International Conference on Machine Learning. PMLR, 2021b.
Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semi-supervised learning. 2022a.
Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. arXiv preprint arXiv:2202.0127. 2022b.
12

---

Published as a conference paper at ICLR 2022
In this Appendix, we provide A) theoretical analysis, B) Reproducibility, C) Additional Experimental Results, and D) Related Works.
A THEORETICAL ANALYSIS
A.1 1 PROOF OF THEOREM 3.1
Based the definition of posterior reweight loss, have 1p-rem (1))()) E
y/Y = x. = x)((y, fx)).. For noisy underlying distribution, the expected risk can be calculated as follows,
E ((x.))=((X.1) ( (1. (3.((x)
P(X 2 peoo (ū, f(x))
dx
Σ
P(X x. y Σ
P(Y y/. x.X==
= x:)1 2)) ((x)(
dx
x
y=1
P(X = x, y, Y T)|l 8)( f(x))
=
j=1
Σ =(X= y) f()) do = E(ax)) [V)
(11)
which implies that posterior reweight loss over noisy underlying distribution equals to cross entropy loss over clean underlying distribution and thus posterior reweight loss induces to statistically consistent classifier.
For the anchor point samples (Fops Dap) with underlying clean probability P(Y Yap |X Top). based on equation (2), the PTM value satisfies W Wap (x) = I and Wk = 0 for any k # Yop. which implies that empirical distribution mismatch on anchor points samples can be completely mitigated. As for empirical distribution, the empirical risk satisfies
1p-rew (Tap: f (xpp)) (yaps (xap))
(12)
For noisy empirical distribution, the expected risk satisfies
N
E ((m.i))P [p-rew (1. =
p-ree Ui. f(:))
N
n=1
N
P(Y = ul. = -in, X (v,
N
(ax)) [(0... f(z))
(13)
which implies that posterior reweight loss over noisy empiricial distribution equals to cross entropy loss over posterior clean empirical distribution.
A.2 PROOF OF LEMMA 3.2
Note that the loss function 1(y, f') is convex with respect to f and Σi Win(#)= Ei P(Y y X x) 1 for any and æ, according to Jensen's inequality, we have
1p-rew U. ((x)))
W Wes()))(y,, f(x:)
8=1
2 1 v. Σ Was(2))(x)) = (ū.-)(()
(14)
which shows the relation between posterior reweight loss and posterior forward loss.
13

---

Published as a conference paper at ICLR 2022
A.3 PROOF OF THEOREM 3.3
Based on Lemma 3.2 and Theorem 3.1, we have the inequality on the expected risk as follows,
E ((m,)))P(xx)) fp- (ū. f())) S lE ((.))))(XX))) (p- @ (ū. f(x)) -(a.y))-(xx,) (y,, f(x)))
Define the optimal classifier f" minimize expected risk for underlying clean distribution and cross entropy loss as
f = arg min Re(x,)) = arg min (xxx,)) (())))
(15)
then R (f)) 0 and for 1p-rew f (x)) 0 for any a. Note that the expected risk for posterior forward loss is non-negative and not larger than that for posterior reweight loss, we also have fp-fw (g. f (x))
for any æ. addition, the difference expected risk for two different
classifiers satisfies
E ((..)) (o- (ū. (xx)
[ (b- (f. f())
- ( (w. f())) S O,
(16)
which implies that the classifier f can also minimize expected risk for underlying noisy distribution and posterior reweight loss. This completes the proof.
A.4 PROOF OF THEOREM 3.4
Aiming to standardize the constraint, we vectorize W(x) as 2² x 1 dimension Ween( = [W (a), Wei ... Wio(x), ... [W) and define the coefficient matrix A(x) as fol- lows,
PT (Y (m)
PT(Y(() x
A(x)
(17)
0
IP (Y(z) exc2
where PT f(x) can be transformed as A(x)Weeol = f(x). which a least-norm problem with undetermined
( P( = 1|x).... P(Ý = c|x)]ixe- Therefore, the constraint W (x))((\)))
equations constraint. Note that matrix A(x) is full column rank, prove that the least-norm solutions is
8co (x) A
f(x).
(18)
Considering any possible solution W (x) satisfying the constraint A(x) f(x), then A(x) ((<x)Weeo(()) w (x)] = 0 and
(Wee((x)) (x)) "w' (x))
(we) - Wx:)((:)) A (a)) (A(z)A A(x))A(<x)) f(x) (A(z)A (xx) -1 f(x) = 0,
= [A(x)) w wiee (x))
which implies that the vectors Weec(x) wiec(x) (x)) and w wi (x) perpendicular, Thus, the L2 norm of Weee(1) (2) satisfies
(ar)((:)) (ree(x:) Weec(()) (2=
|Weeo (x)] l/Wuee(w) w (x)c()))
(x))1
(19)
which shows that the equation
18 is the optimal solution. Via direct calculation, it is easy to obtain 2 Y k k Therefore, the element of PTM satisfies
that A E
Wi (z) = p2(8
(.))
(20)
Km
ie., Wxxx)= = f(=)P(Y(a)) T and completes the proof. [[(())]]]] [m]]
14

---

Published as a conference paper at ICLR 2022
A.5 PROOF OF THEOREM 3.5
Considering the general reconstruction error relationship, the trace of the covariance matrix for reconstruction error satisfies (x)) = cov(2) (or (æ)). For Kalman transition matrix, the reconstruction deviation satisfies W - Wkm (x) = (1 - X)er(z) + xew(x) For simplicity, we remove instance a to abbreviate the notations. Therefore, the reconstruction error is given by
Liss(() = E
AjeT + Xew) ((1 Ajer + dew)
= (1 2) + + + 2X(1 - Nocouvoroww
(21)
For such quadratic function, is easy to obtain the optimal parameter via the local minimal
dCmse 2(1 x)or + 2dow + 2(1 2A)eou = 0,
(22)
and thus the optimal weight is given by X into the reconstruction error, we obtain the minimal reconstruction error is given by
CT -on / forow
Plugging in the optimal weight
Lusse (X*)
ØT [ow con oroww
/oow/orow /orow (or ow 2000
tow oor- ferr-ouyoyy cowworow
+2[ow cou orrow oT con cou/Drow
(23)
Aiming to investigate the relation between minimal reconstruction error for IF, PTM and PTD, we adopt variable substitution oT ow 2000 2 to further simplify minimal reconstruction error (mns(X"), then we have
Cmoe(A') orl(ow - oT) + +t]
owl(or Fww) +
+
4f2
412
|(TT )w) + &lllow σT) + Hlllow oT)
4t2
dW +
(ow (ow-oT)) -OT)²
4
4t
dw + OT
2/ (aw - or)
min min(aw.ar)
(24)
4t
where the inequality holds according to arithmetic mean-geometrie mean inequality. Therefore, the linear combination can provably achieve lower estimation error for general eT and ew (x), either correlated or independent.
B REPRODUCIBILITY
B. DATASET STATISTICS
For fairly comparison with previous work, we performance the image classification task on the three manually corrupted datasets, i.e., F-MNIST, SVHN, CIFAR-10, and one real-worke noisy dataset Clothing M. They have been widely adopted to study label noise problem. The detailed statistics are listed in Table 3
Table 3: Dataset statistics on F-MNIST, SVHN, CIFAR-10, and Clothing L.
F-MNIST
SVHN
CIFAR-10
Clothing IM
training/noisy images
60,000
73,257
50,000
1,000,000
# test/clean images
10,000
26,032
10,000
10,000
label noise
synthetic
synthetic
synthetic
real-world
15

---

Published as a conference paper at ICLR 2022
B.2 RUNNING ENVIRONMENT
All baselines and IF approaches are implemented in PyTorch, and tested on a machine with AMD EPYC 7282 16-core processors, 4 GeForce GTX-3090 Ti GPUs with 24GB memory size.
B.3 ALGORITIMMS
We summarize the IF algorithm to correct the loss and provide the pseudo codes in Algorithm [ For instance-dependent Label noise generation, we provide pseudo codes with controllable noise rate are provided in Algorithm 2 This algorithm follows the state-of-the-art method (Xia et al. 2020 Zhu al. 202 Ia), where overall noise rate T.
Algorithm I: Information Fusion Algorithm
Input Noiss dataset 1) (m Da ) Noisy validation data; tolerant epochs t. Output: : The robust neural network over noisy label.
Estimate the NTM (x) according to (Xia et al. 2020)
while Validation accuracy has increases
last epochs do
Calculate the PTM (xx) based on Equation
Obtain the uncertainty for noise and PTM;
Calculate the Kalman gain and Kalman transition matrix based on Equations and
Correct the loss function posterior forward loss with Kalman transition matrix W Jom
end
Algorithm 2: Instance-Dependent Label Noise Generation
Input :Clean samples (In: Un) Overall noise rate r; Number classes e Size of input features: x d.
Output: Noisy samples (æ,, Un)
Sample flip rate qs based truncated normal distribution (( 0.12, 2,,,, [0, [])
Sample the projection matrix W E dd based normal distribution (/ 12); for π to N do
p x, W // Genrate instance dependent flip rate;
Py. = - 00 // Control the diagonal elements of the transition matrix p = On -softmax(()))) (p) Make the sum of the off-diagonal elements be On ;
Pyn = 1 9r. // Set the diagonal element to be 1 - qn
end
B.4 LEARNING NOISE TRANSITION MATRIX VIA DECOMPOSITION
For the instance-dependent label noise, the NTM may be different for different samples. Thus. the complexity of NTM estimation is O(Ne²), which leads ill- ill-posed estimation problem. (Xia et al. 2020) introduces a part-dependent decomposition on the NTM, i.e., the noise of an instance depends only its parts. Specifically, given part-dependent transition matrix, e.g, pl [0, 1 1Jexe, where j = -.... L. the instance- sestance-depeneent matrix can be approximated by combination of part- dependent transition matrix and the combination coefficients, defined as hxx) E [0, 1] depend on the feature vectors. Then the instance-dependent transition matrix W(x) can be approximated as follows
T(x) = Σ px'
(25)
where hy (x: represents the j-th elements of 1x(x) and satisfies the normalization condition |/((x)] = 1 for any features vector to maintain that the summation of column is 1 for NTM.
In this work, we follow (Xia et al. 2020) to learn the NTM in three steps: (i) learning combination
coefficients h(x)) via basts features representation (Lee & Seung 1999 Tao al. 2017); (ii)
16

---

Published as a conference paper at ICLR 2022
learning the instance- deeeeneedd transition matrix by exploiting anchor points (Liu & Tao 2015 Xia
it al.] 2019): (iii) learning the part-dependent transition matrices via minimization of reconstruction error for nistancedeeeeddent transition matrix.
(Xia et al. 2020) justifies the part-dependent decomposition assumption via parts-based represen-
tations. For example, (Hosseini-Asl et al., 2016) shows that the input features rely on parts-based based
representations in object recognition. Thus, it is natural to approximate the label noise in the part
level. In display advertising, the delay feedback of the label (Ktena et al. 2019 Yasui et al. 2020) introduces label noise. Earlier the data is generated, the less probability the feedback exists. Thus, the label noise highly depends on part of features.
Since the mustance-dependee NTM may be poorly learned, the slack variable trick (Xia et al. 2019 2020) also adopted to modify the instance-dependent transition matrix, which significantly improve the accuracy. However, the slack variable trick ignores the posterior information for each instance, which limits the accuracy. In addition, the variance of accuracy in (Xia et al., (020) is large since there is not any guide information to modify the instance-dependent NTM except minimization of the corrected loss. In this paper, we claim that the measurement information is useful to modify the NTM. We will introduce how to estimate the PTM and information fusion in the next two subsections.
B.5 IMPLEMENTATION DETAILS
Warm-up Training Strategy Aiming to estimate the transition matrix, we adopt warm-up training strategy that training the neural network via cross entropy loss with noisy data. we use SGD as the optimizer with 0.9 momentum, 128 batch size, and 10 epochs. With the benefit of memorization effect (Arpit et al. 2017), warm-up training with appropriate epochs can guarantee that the predicted probability of neural network is reliable and can approximate underlying clean probability.
Training with Corrected Loss probability. Although the warm-up training can guarantee the prediction accuracy for clean instances with the benefits of memorization effect, the prediction of the PTM estimation may still be not reliable.
Our proposed PTM estimation relies on the neural network predicted
To ease the dilemma, we adopt sample selection to choose the reliable prediction and thus guarantee the reliable PTM estimation for specific instance. Similar to (Cheng et al.] 2021a), the small-loss criteria is adopted select the instances with small training loss for training. Specifically, we select the instance via the weighted loss if the instance loss is less than the average loss over all classes.
For the instance satisfying the small loss criterion, we have confidence that the instance is clean, which means the PTM is identity. To put it simply, we sum the estimated PTM and identity and then adopt column normalization. is worth noting that the noisy labels are adopted in forward propagation, which is highly different with previous transition matrix-based works (Xia et al. 2019) Liu & Tao 2015: Patrini et al. 2017] Xia et al. 2020). Compared with hard sample selection based methods (Cheng al.] [2221a), our method achieves the soft sample selection and adaptively correct the loss for "clean" instance.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I STABILITY ON SYNTHETIC NOISY DATASETS
We study the training stability via inspecting the test accuracy over epochs. Figures and 7 show the test accuracy with respect to epochs on SVHN and F-MNIST datasets with noise rate IDN- to IDN-40%. It is seen that our proposed method IF can achieve better accuracy and stable training compare with baseline PTD, which implies that the PTM fusion can decrease the transition matrix estimation error and improve the training stability. T-Revision trick (-V method) proposed in (Xia) al.) 2019) aims compensate the NTM and thus improve the accuracy performance. It is seen that PTD-F-V can achieve better performance than PTD-F, which verifies that the learned NTM is not very accurate. For our proposed method, IF- V achieve comparable performance than IF-F and demonstrates that transition matrix revision trick can not improve the performance since PTM has already been employed to compensate the transition matrix error.
Compared with different noise rates, it is seen that the accuracy of PTD method may decrease with respect epochs, especially for large noise rate. Under large noise rate, the number of clean samples
17

---

Published as a conference paper at ICLR 2022
(a) c = 0.1
(b) E == 0.2
(c) 0.3
(d) c 0.4
Figure 6: The test accuracy SVHN datasets with different levels of IDN noise.
(a) 0.1
(b) e 0.2
(c) c 0.3
(d) c 0.4
Figure 7: The test accuracy F-MNIST datasets with different levels of IDN noise.
is limited and the neural network may already overfit noisy samples during warm-up training and thus hurt the transition matrix estimation. In this scenario, PTD method does have the capability to adjust such poor NTM estimation and thus leads to the accuracy drop during training. Our proposed method IF and T-Revision trick both adjust the estimated transition matrix and mitigate the accuracy drop.
C.2 EXPERIMENTAL RESULTS ON HIGH NOISE RATIO
We investigate the performance of IF for high noise ratio compared with PTM and PTD. Table. shows the test accuracy for high noise ratio IDN-70%, IDN-80% and IDN-90% CIFAR 10 dataset. We have observations: (1) Comparing with PTD the noise ratios of IDN-80% and IDN-90%,
18

---

Published as a conference paper at ICLR 2022
our PTM could still outperform PTD. Although the network output is not perfect, the performance deterioration of PTD is still higher than PTM. (2) Our IF consistently delivers the most superior performance matter under the low or high noise. That is because IF adaptively and linearly combines NTM and PTM in instance level and thus achieves more accurate estimation.
Table 4: Means and standard deviations (percentage) of classification accuracy with high instance- dependent label noise levels for CIFAR 10 dataset. Methods with "-FF" adopts the Forward correction loss; methods with "- means that the transition matrices are revised via slack variable trick.
Method PTD-F
IDN-70%
(DN-80%
IDN-90%
20.25 + 1.56
13.48 1 0.81
9.22 1 0.23
PTD-F-V
20.35 1.56
13.58 + 0.80
9,44 + 0.24
PTM-F
16.79 3.86
14.13 0.34
0.34
10.34 0.71
PTM-F-V
18.95 + 2.89
13.89 + 0.42
10,57 = 0.82
IF-F
21.26 = 1.46
15.78 + 1.65
10.58 = 0.53
IF-F-V
21.09 0.45
16.72
0.61
+ 0.61
10.86 0.40
C.3 EXPERIMENTAL RESULTS ON MORE BASELINES
Although many transition matrix based methods are compared in Table. I we are also interested in several more baselines, including label smoothing (LS) (Szegedy et al. 2016), early-learning regularization (ELR) (Liu et al., (020). (Berthon et al. 2021 progressive label correction (PI.C (Zhang al. 2021a), confidence regularized sample sieve (CORES) (Cheng et al. (021a). It could be observed that our method can still achieve state-of- ofcc--r-tte-rttttt performance compared with all other baselines. Table. 5 demonstrates the test accuracy over IDN-10% IDN-50% label noise ratio in CIFAR 10. It could be observed that our method can still achieve state-of of of-the-arr art performance, based on which we can confidently validate the effectiveness of our method.
Table 5: Means and standard deviations (percentage) of classification accuracy with more baselines for CIFAR 10 dataset. Methods with "-F" adopts the Forward correction loss; methods with "-V" means that the transition matrices revised via slack variable trick.
Method LS
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
63.19 0.25
S7.67 1 0.10
49.50.10.11
41.57
1.85
34.89 1 2.47
ELR
66.20 = 0.09
60.58 1 0.75
51.65 + 0.33
43.83
1.61
34.44 0.12
PLC
66.38 = 0.61
60.42 + 0.65
51.60 + 0.58
42.77 1 0.50
36.00 + 1.73
CORES
67.39 = 0.44
60.85 0.38
51.47 0.80
43.96 = 0.68
34.85 + 1.66
PTD-F
79.77 0.91
74.96 0.71
70.68 + 0.81
61.92 = 1.59
45.34 4.67
PTD-F-V
80.08 0.86
74.67 - 0.36
71.66 + 1.05
62.45 1.73
46.16 4.48
PTM-F
78.16 0.36
74.81 + 0.81
70.03 0.38
63.48 + 0.38
$1.03 + 3.08
PTM-F-V
78.58 0.31
75.06 - 0.56
69.83 + 0.58
62.69 = 0.69
50.53 + 3,41
IFF
80.92 0.28
79.58
0.52
74.34
0.86
68.21 2.21
50.07 3.95
IF-F-V
80.94 0.43
79.54 0.45
74.67 + 0.92
68.03 2.90
52.34 1.31
D RELATED WORK
Label Noise Model
There are three types of label noise model, including the random classification
noise (RCN) model
(Manwani & Sastry 2013 3, Natarajan et 2013), the canss-ooniiional label
noise (CCN) model (Patrini et al. 2017: Xia al. 2019; 9 Zhang & Sabuncu 2018 Liu & Guo 2020) and the instance-dependent label noise (IDN) model (Cheng et al.) 2020 Xia et al. 2020
Cheng et al., 2021a Berthon et al. 2021 ?). Specifically, RCN means that each label is flipped
independently with constant probability. CCN assumes that the flip probability (noise rates) are the same for the instance with the same clean labels. IDN is the most general case, where the flip probability depends on its instance.
Loss Correction via Transition Matrix The transition matrix bridges the gap between the model predicted probability for noisy and clean data, which can achieve risk-consistent classifier with label
19

---

Published as a conference paper at ICLR 2022
noise. The transition matrix can be estimated via cross volssvvalio method (Natarajan al. 2013),
anchor points assumption (Xia et al. 2019; Yu et al. [018) Yao et al. 2020). Subsequently, the estimated transition matrix can be adopted to correct the loss function via forward correction (Patrini
et al. 2017), backward correction (Patrini et al. 2017) and reweight (Xia et al. 2019 Liu & Tao
2015
Loss Correction via Sample Selection Due to the experiments observation- the neural network
learns clean instances first (Arpit et al., 2017) and memorizes the instances gradually, there are lots of methods attempting identify mislabeled training examples and then filter them out (Brodley &
Friedl 1999 Diang et al. 2018: Han et al. 2018b; Yu et al. 2019; Angelova et al.) 2005 |Huang et 2019
Li et al. 2019). The core idea is to select the "small loss" samples as clean onc.
20


Reviews:
[{"review_id": "xhLPOWazTuc", "reviewer": "Reviewer_cR7o", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The argument in this paper, namely IDN is indeed caused by empirical distribution rather than underlying real distribution, is interesting.\n2. The proposed estimation method for PTM and the fusion of PTM and NTM via Kalman filtering are new.\n3. Theoretical analyses in this paper are sufficient.\n\nCons:\n1. The title of this paper can be more concise.\n2. More related baseline methods are suggested to be added.\n\n", "summary_of_the_paper": "This paper proposes a new information fusion approach to deal with the instance-dependent label noise (IDN). Specifically, the authors claim that the essential problem caused by IDN is empirical, instead of underlying, data distribution mismatch during training, which I think is new and interesting. Therefore, the authors fuse posterior transition matrix (PTM) and noise transition matrix (NTM) and design an unbiased risk estimator. ", "summary_of_the_review": "Generally, I feel that the claim in this paper is reasonable, the model proposed is also new. The paper is written in a clear way. I only have some minor concerns:\n1. In the title, the authors use \"information fusion\", which I think is not concise. Note that ``information'' is a rather wide topic, so after reading the title, I cannot see what are fused in this paper. \n2. Since this paper studies instance-dependent label noise, I hope to see more comparisons with existing methods for handling IDN, such as \"A Second-Order Approach to Learning with Instance-Dependent Label Noise\" (CVPR 21); Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model (AAAI 21);  Learning with Bounded Instance- and Label-dependent Label Noise (ICML 20). In the current experiment, only one method, namely  PDT (NeurIPS 20) is designed for IDN. Other methods such as Forward, T-revision are not specifically designed for IDN. Therefore, I feel that the comparison is not that fair.\n3. Is the solution for Posterior transition matrix in Thm 3.4 guaranteed to satisfy the constraint that the sum of every column is 1?\n4. From Thm 3.5 and Eq. 13, I feel that the solution is also related to statistical efficiency? See Section 2.4 and Section 3 of \"Centroid Estimation with Guaranteed Efficiency: A General Framework for Weakly Supervised Learning\" (TPAMI 20). If yes, the authors may analyze the connections or relationship between them.\n5.  In Fig. 1, I suggest the authors clearly annotating PTM in the figure. Note that the authors use \"noise transition matrix (PTM)\" in the caption, but write \"posterior TM\" in the figure. Such inconsistency in the figures should be avoided.\n6. Some language issues, such as \"a easy-to compute\" should be \"an easy-to-compute\"; \"is the instance vector of n-th sample\"->\"is the instance vector of the n-th sample\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nAQ1nb74GhE", "reviewer": "Reviewer_d7yz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\n1. The paper defines a new loss-correction approach, which uses the posterior transition matrix, to achieve the statistically consistent classifier.\n\n2. The authors also propose the corresponding approach to estimate the posterior transition matrix based on model predictions.\n\n3. The high-level intuition of using the Kalman filter to reduce the estimation error is insightful. \n\n### Weakness:\n\n1. It is not clear why the proposed information fusion approach works. Section 3.3 presents an information fusion method based on Kalman filtering, where the Kalman transition matrix is given by a convex combination of NTM and PTM. Although it can be proved that carefully assigning the combination weights makes the reconstruction error lower than both $\\sigma_T(x)$ and $\\sigma_W(x)$, the errors indicate the variance of the estimation. Only reducing the variance may not be helpful due to the estimation bias. In others words, a convex combination of NTM and PTM would introduce estimation bias, which makes $E[W_{km}(x)] \\ne E[W^*(x)]$. With the existence of bias, it is hard to conclude the proposed information fusion approach is helpful.\n\n2. It would make the claim stronger if the authors provide more intuitions, examples, or theoretical analyses to show why PTM could be easier to be estimated. According to the Bayes rule, we have\n\\\\[\nP(\\tilde Y=i|Y=j,X=x) = P(Y=i|\\tilde Y=j,X=x) \\cdot \\frac{P(\\tilde Y=j|X=x)}{P(Y=i|X=x)},\n\\\\]\nwhere $P(\\tilde Y=j|X=x)$ could be estimated from the noisy dataset and $P(Y=i|X=x)$ could be estimated by marginalizing $P(Y=i,\\tilde Y=j|X=x)$ w.r.t. $\\tilde Y$. From this aspect, estimating both matrices should have the same degree of difficulty.\n\n3. It is not clear how estimation error is defined in Figure 2. Recent methods to estimate NTM (e.g., end-to-end approach [R1], using clusterability of representations (Zhu et al., 2021b cited in the paper) can also achieve relatively low estimation errors according to their papers. Figure 2 would be more trustworthy if recent works are compared.\n\n\n4. In Page 6, $e_T(x)=W^*(x)-\\hat T(x)$ looks confusing. Should it be $e_T(x)=T^*(x)-\\hat T(x)$? Besides, typo \"nose\" in the caption of Figure 3 and Figure 4. Typo in Table 1, CIFAR10 PTM-F-V IDN-20\\%.\n\n[R1] Li, Xuefeng, et al. \"Provably end-to-end label-noise learning without anchor points.\" ICML 2021.\n", "summary_of_the_paper": "This paper proposes to do loss correction by a posterior transition matrix (PTM).\nThe authors first define the loss function using PTM which achieves the statistically consistent classifier, then propose an information fusion approach to reduce the variance in estimating the transition matrix.", "summary_of_the_review": "The idea is interesting but the paper is not well-organized or presented. The information fusion approach, which should be the key novelty of this paper, is not demonstrated sufficiently. The main concern is that the proposed information fusion approach reduces the variance at the cost of introducing bias in estimation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hrmPauylw_H", "reviewer": "Reviewer_oWz1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting direction of estimating \"instance-based\" noise transition matrix, rather than estimating a class-based transition matrix. The authors first provide a brief introduction of noise transition matrices in the existing work, and motivate the need for estimating an instance-based transition matrix.\n\nThe presentation and the writing of the actual method can be improved to make it easier for the reader to understand different steps of the pipeline. For example, the interaction of different components (the network, PTM estimation IF correction etc. ) is not very clear, and an overview figure can be used to visualize their interaction. Figure 1 is not clear and does not help reader understand the interaction of different components.\n\nMy biggest concern is the assumption of \"good\" network outputs to estimate the PTM. The network is known to overfit the noisy labels, and this assumption may not always work well, especially if the noise ratio is high. For example, what happens when the noise ratio is above 50%. Existing work usually report noise ratio up to 80%-90% on CIFAR, and it would be interesting to see how PTM behaves under such setting.\n\nThe experiments would also benefit from additional baselines. For example, Label smoothing (Szegedy et al., 2016) adds a uniform distribution to the assigned label. Mixup (Zhang et al., 2018) creates a combination of labels. These methods do not correct the label noise through an instance-based transition matrix like this paper, however the way the \"corrupt\" the assigned label adds a regularization effect and help denoise the noisy labels.\n\nThe Clothing1M results in Table 2 are missing better-performing comparisons, such as Divide-Mix (Li et al., 2019), ELR+ (Liu et al., 2020) etc.\n\n\nChristian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pp. 2818–2826, 2016.\n\nHongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018.\n\nJunnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semisupervised learning. In ICLR, 2019.\n\nSheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. NeurIPS, 2020.", "summary_of_the_paper": "This paper proposes to estimate an input-dependent noise transition matrix (PTM). The authors make a strong assumption that the neural network output approximates the underlying clean probability and estimate the PTM by extracting the output of each training sample using the network. To enforce the network to have better probabilities, PTM estimation is applied iteratively and warm-up is used during the training. The experiments are conducted in CIFAR-10 with different noise ratio, as well as a real world dataset, i.e. Clothing-1M.\n", "summary_of_the_review": "The paper proposes an interesting and original idea. However, I think a few important experiments missing (higher noise ratio, additional baselines such as label smoothing and mixup), and the Table 2 needs to be updated with more recent work. I would be willing to upgrade my evaluation if the authors address my concerns during the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3Qz7sUoZxHv", "reviewer": "Reviewer_8i2r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths：\n1, I think the posterior noise-transition matrix and information fusion (IF) approach are new and interesting. Theoretical analyses are also provided to prove the robustness of such a transition matrix. \n\n2, The paper is well written and organized. The authors did a good literature review on the previous methods.\n\nWeaknesses:\n\n1: In Theorem 3.5, authors assume that $e_T(x)$ and $e_W(x)$ are independent. I think it is impractical. By the definition of $e_T(x)$ and $e_W(x)$, these two terms may be strongly dependent. Thus Theorem 3.5 can not well explain the experiments. \n\n2: From experiments in Table 1, PTM does not show much performance gain compared to PTD. Thus the applicability of the posterior transition matrix may be limited.\nBesides, it is not clear how PTM and IF behave on the dataset with more categories such as CIFAR100.\n\n3:  Many methods have been proposed in the literature to deal with IDN (A1, A2, A3, A4, A5). However, the authors only compare PTD in the experiments. I think more comparisons are needed to further validate the effectiveness of IF. \n\n\nA1: Learning with bounded instance- and label-dependent label noise. ICML 2020\n\nA2: Part-dependent label noise: Towards instance-dependent label noise. NeurlPS 2020\n\nA3: Confidence scores make instance-dependent label-noise learning possible. ICML2021\n\nA4: Learning with feature-dependent label noise: A progressive approach. ICLR2021\n\nA5: Learning with instance-dependent label noise: A sample sieve approach. ICLR2021", "summary_of_the_paper": "This paper provides new insight on estimating noise-transition matrix in the setting of instance-dependent label noise. Specifically, the authors observe that the traditional noise-transition matrix cannot well bridge the gap between underlying distribution and empirical distribution. Inspired by this observation, the authors propose to estimate the posterior noise-transition matrix. Further, authors use information fusion (IF) to linearly combine noise-transition matrix and posterior noise-transition matrix to obtain more accurate transition matrix estimation with theoretical guarantees. Experiments are conducted on CIFAR-10, SVHN, F-MNIST, and real-world dataset Clothing-1M. ", "summary_of_the_review": "Overall, the quality of this paper is good. The proposed approach is supported by the theorem. The experiment shows that information fusion (IF) outperforms PTD (Part-dependent transition matrix) by a large margin when the noise rate is high.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: To tackle the problem of classification under input-dependent noise, the authors proposed the posterior transition matrix (PTM) to achieve statistically consistent classification. Specifically the information fusion approach was developed to fine-tune the noise transition matrix. Experiments demonstrated the effectiveness of the proposed approach.

I would like to thank the authors for the detailed feedback to the initial reviews and also further feedback to the reviewers' additional questions. Many concerns were clarified by the feedback, and the additional experiments still demonstrate the effectiveness of the proposed method.

The issue of data augmentation still remains, which should be at least experimentally investigated,
but the contribution of the current manuscript is still valuable to be presented as ICLR2022.
Decision: Accept (Poster)