Estimating Instance-dependent Label-noise Transition Matrix using DNNs
Authors: Shuo Yang
Erkun Yang
Bo Han
Yang Liu
Min Xu
Gang Niu
Tongliang Liu
Venue: ICLR 2022 Submitted (2022)

Abstract: In label-noise learning, estimating the transition matrix is a hot topic as the matrix plays an important role in building statistically consistent classifiers. Traditionally, the transition from clean labels to noisy labels (i.e., clean label transition matrix) has been widely exploited to learn a clean label classifier by employing the noisy data. Motivated by that classifiers mostly output Bayes optimal labels for prediction, in this paper, we study to directly model the transition from Bayes optimal labels to noisy labels (i.e., Bayes label transition matrix) and learn a classifier to predict Bayes optimal labels. Note that given only noisy data, it is ill-posed to estimate either the clean label transition matrix or the Bayes label transition matrix. But favorably, Bayes optimal labels have less uncertainty compared with the clean labels, i.e., the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not. This enables two advantages to estimate the Bayes label transition matrix, i.e., (a) we could theoretically recover a set of noisy data with Bayes optimal labels under mild conditions; (b) the feasible solution space is much smaller. By exploiting the advantages, we estimate the Bayes label transition matrix by employing a deep neural network in a parameterized way, leading to better generalization and superior classification performance.

Under review as a conference paper at ICLR 2022
ESTIMATING INSTANCE-DEPENDENT LABEL-NOISE TRANSITION MATRIX USING DNNs
Anonymous authors Paper under double-blind review
ABSTRACT
In label-noise learning, estimating the transition matrix is a hot topic as the matrix plays an important role in building statistically consistent classifiers. Tradition- ally, the transition from clean labels noisy labels (i.e., clean label transition matrix) has been widely exploited to learn a clean label classifier by employing the noisy data. Motivated by that classifiers mostly output Bayes optimal labels for prediction, in this paper, we study to directly model the transition from Bayes opti- mal labels to noisy labels (i.c., Bayes label transition matrix) and learn a classifier to predict Bayes optimal labels. Note that given only noisy data, it is ill-posed to estimate either the clean label transition matrix or the Bayes label transition ma- trix. But favorably, Bayes optimal labels have less uncertainty compared with the clean labels, i.e., the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not. This enables two advantages to estimate the Bayes label transition matrix, i.e., (a) we could theoretically recover a set of noisy data with Bayes optimal labels under mild conditions; (b) the feasible solution space is much smaller. By exploiting the advantages, we estimate the Bayes label transition matrix by employing a deep neural network in parameterized way, leading to better generalization and superior classification performance.
INTRODUCTION
The study of classification in the presence of noisy labels has been of interest for three decades (Angluin Laird 1988). but becomes more and more important in weakly supervised learning (Thekumprrmmpi et al. 2018 Li et al., 2020b; Guo et al. 2018 Xiao et al.) 2015 Zhang et al., 2017a] Yang et al. 2021b a). The main reason behind this that datasets are becoming bigger and bigger. To improve annotation efficiency, these large-scale datasets are often collected from crowdsourcing platforms (Yan et al. 2014), online queries (Blum et al. 2003), and image engines (Li et al. 2017). which suffer from unavoidable label noise (Yao et al. 2020a). Recent researches show that the label noise significantly degenerates the performance of deep neural networks, since deep models easily memorize the noisy labels (Zhang al.. 2017a Yao et al. 2020a).
Generally, the algorithms for combating noisy labels can be categorized into statistically inconsis- tent algorithms and statistically consistent algorithms. The statistically inconsistent algorithms heuristic, such as selecting possible clean examples to train the classifier (Han et al. 2000 Yao et al.,
2020a Yu et al., 2019; [Han al. 2018b Malach & Shalev-Shwartz) 2017 Ren al. 2018 Jiang et al.] 2018), re- weighigt examples to reduce the effect of noisy labels (Ren et al. 2018). correcting
labels (Ma et al. 2018 Kremer et al. 2018 Tanaka et al. 2018 Reed al. 2015). ) or adding regu-
larization (Han et al. 2018a] Guo et al. 2018: Veit et al. [017]] Vahdat 2017 Li et al. [017]] 2020b) Wu et al. 2020). These approaches empirically work well, but there is no theoretical guarantee that the learned classifiers can converge to the optimal ones learned from clean data. To address this lim-
itation, algorithms the second category aim to design classifier-consister algorithms (Yu et al.)
2017
Zhang & Sabuncu, 2018: Kremer et al. 2018 Liu & Tao 2016 Northcutt et al.) 2017 Scott
2015
Natarajan et al. Goldberger & Ben- Reuven |017]] Patrini et al. [00 7 ( Thekumparampil
et al.
2018: Yu et al.
2018 Liu & Guo 2020 Xu al. 2019 Xia et al. 2020b). where classifiers
learned on noisy data will asymptotically converge to the optimal classifiers defined on the clean
domain.

---

Under review as a conference paper at ICLR 2022
The label transition matrix Tx) plays important role in building statistically consistent algo- rithms. Traditionally, the transition matrix T(x) is defined to relate clean distribution and noisy dis- tribution, where T(x) = P( | = x) and X denotes the random variable of instances/features, Y as the variable for the noisy label, and Y as the variable for the clean label. The above ma- trix is denoted as the clean label transition matrix, which is widely used to learn a clean label classifier by employing the noisy data. The learned clean label classifier is expected to predict a probability distribution over a set of pre-defined classes given an input, i.e. clean class posterior probubility P(Y X). The clean class posterior probability is the distribution from which clean labels are sampled. However, Bayes optimal labels Y". i.c., the class labels that maximize the clean class posteriors Y X arg maxy P(Y | X), are mostly used as the predicted labels and for computing classification accuracy. Motivated by this, in this paper, we propose to directly model the transition matrix T (x) that relates Bayes optimal distribution and noisy distribution, i.e., T (x) P( Y X x), where Y denotes the variable for Bayes optimal label. The Bayes optimal label classifier can be learned by exploiting the Bayes label transition matrix directly.
Studying the transition between Bayes optimal distribution and noisy distribution is consid- ered advantageous to that of studying the transition between clean distribution and noisy dis- tribution. The main reason is due to that the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not. Two advantages can be intro- duced by this better estimate the instance-dependent transition matrix: lect a set of examples with theoretically guaranteed Bayes optimal labels out of noisy data.
(a) We can col-
The intrinsic reason that Bayes optimal labels can be inferred from the noisy data while clean labels cannot is that Bayes optimal labels are the labels that maximize the clean class posteriors while clean labels are sampled from the clean class posteriors. In the presence of label noise, the labels that maximize the noisy class posteriors could be identical to those that maximize the clean class posteriors (Bayes optimal labels) under mild conditions (Cheng et al., 2020). Therefore some instances' Bayes optimal labels can be inferred from their noisy class posteriors while their clean labels are impossible to infer since the clean class pos- teriors are unobservable, as shown in F (b) The feasible solution space of the Bayes label transition matrix is much smaller than that of the clean label transition matrix. This is because that Bayes optimal la- bels have less uncertainty compared with the clean labels. The transition matrix defined by Bayes optimal labels and the noisy labels therefore spurse and can be estimated more efficiently with the same amount of training data.
.05
.85
.05
.02
.03
clean class postorior
.25
.55
.05
.10
.10
naisy class posterior
1
Bayes class posterior
Figure 1:
The noisy class posterior is learned from noisy data. Bayes opti- mal label can be inferred from the noisy class pos- terior if the noisy rate is controlled. Also, the Bayes optimal label is less uncertain since the Bayes class posterior one-hot vector.
These two advantages naturally motivate to collect set of examples with their theoretically guaranteed Bayes optimal labels out of the noisy data to learn approximate the Bayes label transition matrix (x).. Due to the high complexity of the instance-dependent matrix (x), we simplify its estimation by parameterizing it using a deep neural network. The collected examples, inferred Bayes optimal labels, and their noisy labels are served as data points to optimize the deep neural network approximate the T (x). Compared with the previous method (Xia et al.) (2000)), which made assumptions and leveraged hand-crafted priors to approximate the instance-dependent transition matrices, we train a deep neural network to estimate the instance -ettance-deeedeent label transition matrix with reduced feasible solution space, which achieves lower approximation error, better generalization, and superior classification performance.
RELATED WORK
Noise model. Currently, there are several typical label noise models. Specifically, the random clas- sification noise (RCN) model assumes that clean labels flip randomly with constant rate (Biggio et al. 2011 Manwani & Sastry] 2013, Natarajan et al., 2013). The cladsiion label noise (CCN) model assumes that the flip rate depends on the latent clean class (Patrini et al., 2017 Xia ctt al. 2019: Ma et al. 2018). The instance-dependdet label noise (IDN) model considers the most
general case of label noise, where the flip rate depends on its instance/features (Cheng et al. 2020
2

---

Under review as a conference paper at ICLR 2022
Xia et al. 2020a. Zhu et al. 2020). Obviously, the IDN model is more realistic and applicable. For example, in world datasets, an instance whose feature contains less information or is of poor quality may be more prone to be labeled wrongly. The bounded instance dependent label noise (BIDN) (Cheng et al. 2020) is a reasonable extension of IDN, where the flip rates are dependent on instances but upper bounded by a value smaller than 1. However, with only noisy data, it is non-trivial task to model such realistic noise without any assumption (Xia et al. 2020a). This paper focuses on the challenging BIDN problem setting.
Learning clean distributions. It is significant to reduce the side effect of noisy labels by inferring clean distributions statistically. The label transition matrix plays an important role in such an infer- ence process. which is used to denote the probabilities that clean labels flip into noisy labels. We first review prior efforts under the dependen condition (Patrini et al., 2017). By exploiting the class-dependent transition matrix T, the training loss on noisy data can be corrected. The transition matrix T can be estimated in many ways, e.g., by introducing the anchor point assumption (Liu &
Tao 2016), by exploiting clustering (Zhu et al., 2021b. by minimizing volume of T (Li et al. 2021), and by using extra clean data (Hendrycks et al. 2018 Shh et al. 2020), To make the estimation more accurately, slack variable (Xia et | 20199 or a multiplicative dual T (Yao et al.) 2020b) can be introduced to revise the transition matrix. As for the efforts on the instance-dependent transition matrix, existing methods rely on various assumptions, e.g., the noise rate is bounded (Cheng al.) 2020), the noise only depends on the parts of the instance (Xia et al.] 2020a), and additional valu- able information is available (Berthon al. 2020). Although the above advanced methods achieve superior performance empirically, the introduction of strong assumptions limit their applications in practice. In this paper, we propose infer Bayes optimal distribution instead of clean distribution, as Bayes optimal distribution is less uncertain and easy to be inferred under mild conditions.
Other approaches. Other methods exist with more sophisticated training frameworks or pipelines, including but not limited to robust loss functions (Zhang & Sabuncu 2018 Xu et al. 2019 Liu
& Guo 2020), sample selection (Han et al. 2018b Wang et al. 2019, Lyu & Tsang 2020). label
correction (Taaaaa et al. 2018: et al. [021] et al. (020), (implicit) regularization Xia
ctt al. 2021: Zhang et al. 2017b Liu et al. 2020), and semi-supervised learning (Li ct al. 2020a
Nguyen et al. 2020).
3 PRELIMINARIES
We introduce the problem setting, some important definitions, and the formulation of the proposed Bayes label transition matrix in this section.
Problem setting. This paper focuses on classification task given training dataset with Instance Dependent Noise (IDN), which is denoted by $ = {(x,, mi: }m We consider that training examples {(x;, U:) 1= are drawn according to random variables (X, Y D. where D is noisy distribution. The noise rate for class y is defined as (x) P(Y y Y + y,,). This paper focuses on reasonable IDN setting that the noise rates have upper bounds Pmoz as in (Cheng et al. 2020), i.e.. Note that the problem in (Cheng et al. 2020) defined on binary classification task while we extend the problem setting to multi-class classifcation. Our aim to learn a robust classifier only from the noisy data, which could assign accurate labels for test data.
v(x) E X, 0 < Py I Pmaz
Clean distribution. For the observed noisy training examples, all of them have corresponding clean labels, which are unobservable. The clean training examples are denoted by S = [(x,, y:)) J which are considered to be drawn according to random variables (X, Y) D. The term D denotes the underlying clean distribution.
Bayes optimal distribution. Given X, its Bayes optimal label is denoted by Y Y | X == arg maxy P(Y | X), (X ~ D. The distribution of Y") is denoted by D Note the Bayes optimal distribution D* is different from the clean distribution D when P(Y & {0, 1). Like clean labels, Bayes optimal labels are unobservable due to the information encoded between features and labels is corrupted by label noise (Zhu et al.) 2020). Note that it is a non-trivial task to infer D only with the noisy training dataset S. Also, the noisy label ÿ. clean label y, and Bayes optimal label y", for the same instance x may disagree with each other (Cheng et al. 2020).
3

---

Under review as a conference paper at ICLR 2022
Other definitions. The classifier is defined stance and label spaces respectively. Let [[]] be the indicator function. Define the 0-1 risk of f as (((()) & [x( * Y]. Define the Bayes optimal classifier f" as f - arg miny ((((),, ). Note that there is NP-hardness of minimizing the 0-1 risk, which neither convex nor smooth (Bartlett et al., 2006). We can use the softmax cross entropy loss as the surro- gate loss function to approximately learn the Bayes optimal classifier (Bartlett al., 2006 Cheng al. 2020). We aim to learn a classifier f from the noisy distribution D which also approximately minimizes (XX,,
X - V where X' and y denote the in-
Bayes label transition matrix. Traditional instance-dependent label transition matrix encodes the probabilities that clean labels flip into noisy labels given input instances. However, due to the reasons that clean labels have more uncertainty, estimating the clean label transition matrix is relatively harder. In this paper, we focus on studying the transition from Bayes optimal labels to noisy labels. We define the Bayes label transition matrix that bridges the Bayes optimal distribution and noisy distribution as follows.
(x) = P(a))=''''''"==-
(I)
where Thy (X) denotes the (i,j)-th element of the matrix (X)),, indicating the probability of a Bayes optimal label i flipped to noisy label j for input X. Given the noisy class posterior probability
P(Y X = x) = [P(Ý = 11 X x),..., P(Ý C X x)] (which can be learned
from noisy data) and the Bayes label transition matrix T = P( = j|Y i, x). the Bayes class posterior probability P(Y*IX x) can be inferred, i.e., P(Y X x)
(X"
xx)).
4 METHOD
The feasible solution space of the Bayes label transition matrix is much smaller since Bayes optimal labels are deterministic. Therefore, we propose to estimate a Bayes label transition matrix for each input instance in a parameterized way. We firstly collect a distilled dataset (is defined in Defina- tion D with theoretically guaranteed Bayes optimal labels out of the noisy dataset (Section 4.1 Then, we can train a deep neural network (Bayes label transition network) on the collected distilled dataset to learn the transition between Bayes optimal labels and noisy labels (Section 4.2) The learned Bayes label transition network is then fixed and used to train a classification network on the noisy dataset in a F-Correction Patrini al. 2017 fashion (Section 4.3)
4.1 COLLECTING DISTILLED EXAMPLES
In this subsection, we show how to construct a distilled dataset consists of distilled examples. We formally introduce the concept of distilled examples first and then present how to collect distilled examples automatically. The collected distilled examples can be used for training the Bayes label transition network.
Definition 1 (Distilled examples (Cheng et al. 2020)). An example (x. G. y²) is defined to be distilled example if y² is identical to the one assigned by the Bayes optimal classifier under the clean data, i.e., y = / (x).
The distilled examples can be collected of noisy examples automatically with the following guarantee,
Theorem (CCheng et al. 2020)). Denote by (x((x) the noisy class posterior probability pp( = u|X x) and my (x) PD y|X = x) the clean class posterior probability. v(x:) E X, we have
1 + Pmax
1
1 - (x,
(x)
>
Tly ())()))))))))))>))
= is distilled;
(2)
2
2
where Pmaz is the noise rate upper bound. Theorem I can be proved in a similar way as did in (Cheng et al. 2020) (Theorem 2 therein). Note the original theorem in (Cheng al.) (020) was built on binary-classffiaan task, we extend it to the multi-class classification problem. the extension is
straightforward.
4

---

Under review as a conference paper at ICLR 2022
According to Theorem 1 obtain distilled examples by collecting all noisy examples (x. ÿ) whose x satisfies f etwee and then assigning the label y to it as its Bayes optimal label y" After that, we obtain an instance of distilled example (xdistilled, D. 1 the I indicates the noisy label while the y indicates the inferred Bayes optimal label. The y may disagree with the y. The conditional probability fy can be estimated as fly by several methods, such as the probabilistic classification methods (logistic regression and deep neural networks).
4.2 ESTIMATING BAYES LABEL TRANSITION MATRICES USING DISTILLED EXAMPLES
As discussed in Section 4. we can collect a set of dis-
tilled examples (xtsstitled V/ v²") from noisy data. Now we proceed to train a Bayes label transition network parameterized by 0 to estimate the instance-dependent label-noise (IDN) transition matrices, which model the probability of observing a noisy label ÿ given input im- age x and its Bayes optimal label y":
distiiled
Bayes Label
Transition Network
(x, (xddiiisstlledd () = P( j|Y i, i distilled, :0),
(3)
Specifically, the Bayes label transition network takes ddistilled as input and output an estimated Bayes label
t (x distillldd
transition matrix 7.r (xdistilled, 0) E RCCC" where C' is
the number of classes. Note T (xddiatiliedd 0) is an approximation to T (xdistilleed) and it might be not exactly the same as the definition of Bayes label transition matrix. We can use the collected Bayes labels y² and the estimated Bayes label transition matrix T (xqiitilledd; 0) to infer the noisy labels. The following empirical risk on the inferred noisy labels and the ground-truth noisy labels are minimized to learn the network's parameter 0:
(0) - Σ Uz log(2?
Tx (xdistsiledd :)),
(4)
where J: and D/ are 0J and yi in the form of one-hot vectors, 0: E RIxC and y E R respectively. Note that if we have a distilled example for the i-th class, we can only make use of to learn the i-th row of the transition matrix. For the other rows, they will not contribute to calculating the loss of the current training example. However, it does not mean that they will be random or not learnable. Their information will be learned by exploiting distilled examples from the non-i-th classes. More specifically, the parameters of the network can be divided into row row-specific parameters and commonly shared parameters. By assuming that have distilled examples for each class, both the row-specific parameters and commonly shared parameters will be optimized.
4.3 TRAINING CLASSIFICATION NETWORK WITH BAYES LABEL TRANSITION MATRICES
Our goal is train classification network ((]w) that predicts Bayes class posterior probability P(Y = i|x: w) parameterized by w. In the training stage, we cannot observe the Bayes optimal label Y Instead, we only have access to noisy label Y. The probability of observing a noisy label Y given input image x is:
&
P(ý = =
= | w),
(5)
With the trained Bayes label transition network, we can get T (x: = -(-== for each input x. We exploit F-Correction (Patrini et al. 2017). which typical classifier- consistent algorithm, to train the classification network. To be specific, fix the learned Bayes label transition network parameter 0, we minimize the empirical risk as follows to optimize the classifi- cation network parameter w:
Rz(w) =
Σ Ts [x; T- (xx; 0)),
(6)
5

---

Under review as a conference paper at ICLR 2022
Algorithm Bounded Instance-dependeni Label Noise Generation.
Require: Clean examples [xx,,,))] 1:
Require: Noise rate
Require: Noise rate upper bound pmazi
1:
Sample instance flip rates qs from the truncated normal distribution N (p, 0.1², [0, pmax]]
//mean 7, variance 0.1 range [0.pmar] 2: Independently sample wi, w2,..., from the standard normal distribution N(0, 1²);
3: for 1,2,...,7. do
/fgenerate instance- Instance-eeenddet flip rates Houly consider entries that are different from the true label /maae the sum of the off uff-diagonal entries of the yi-th row to be //set the diagonal entry be -qi 8: Randomly choose label from the label space according to the possibilities p as noisy label iG:
p x; x Wy.
5: Ps: 00;
7: Po: soflmaxe(p):
6: p=
Qi
9: end for
10: return Noisy samples [(x,, 9:))==
where f(x;; E RIxC. The F-Correction has been proved to be classifier-consiseeni algorithm, the minimizer of [2(ww) under the noisy distribution is the same as the minimizer of the original cross-entropy loss under the
(x)
y
Classification Network
9
Bayes optimal distribution (Patrini et al. 2017), the transition matrix 7 is estimated unbi- ased. Note the Bayes label transition network
Bayes Label
Transition Network
is trained on a biased set, i.e., the set of distilled examples. The network will generalize to the non con-distilled examples if they share the same pattern with the distilled examples which causes la- bel noise. A recent study (Xia et al.) (2020a) empirically verified that the patterns that cause label noise are commonly shared. Our empirical experiments further show that the network ~' generalizes well to unseen examples and thus helps achieve superior classification performance.
EXPERIMENTS
In this section, we first introduce the experiment setup (Section 5.1 including the datasets used (Section 5.1.1 and the compared methods (Section [51.2)). Next, we analyse the hhyer-paramede sensitivity in Section 5.2 Finally, we present and analyze the experimental results on synthetic and real-world noisy datasets show the effectiveness of the proposed method (Section 5.3). The
implementation details and more ablation studies are included in the Appendix.
5.1
EXPERIMENT SETUP
In this section, we introduce the datasets used and the bascline methods compared with.
5.1.1 DATASETS
We conduct the experiment on four datasets to verify the effectiveness of our proposed method, where three of them are manually corrupted, i.e., F-MNIST, CIFAR-1O, and SVHN, one of them is real-world noisy datasets, i.c.. Clothing M.. F-MNIST has 28 x 28 grayscale images of 10 classes including 60,000 training images and 10,000 test images. CIFAR- 10 dataset contains 50,000 color images from 10 classes for training and 10,000 color images from 10 classes for testing both with shape of 32 X 32 3. SVHN has 10 classes images with 73,257 training images and 26,032 test images. We manually corrupt the three datasets, i.e., F-MNIST, CIFAR-1O and SVIN with bounded instance-dependent label noise according to Algorithm D which is modified from (Xia et al. (020a). In noise generation, the noise rate upper bound Pmoz in Algorithm i set 0.6 for all experiments. All experiments on those datasets with synthetic instanceddependen label noise are repeated five times to guarantee reliability. The Clothing M has 1M images with real- world noisy labels for

---

Under review as a conference paper at ICLR 2022
1.00
Class-dependent Revision
85
10N-10%
Ours
80
IDN-30%
0.90
Ours
IDN-40%
75
0.85
accuncy
0.80
0.75
0.65
0.60
0.1
0.2
0.3
0.4
0.5
0.6
0.1
0.2
0.3
0.4
0.5
0.6
Distiliation threshold
Distillation threshold
(a)
(b)
Figure 2: Illustration of the transition matrix approximation errоr and the hyperparameter sensitivity. Figure (a) illustrates how the distillation threshold p affects the approximation error for the instance- dependent transition matrix. Figure (b) illustrates how the distillation threshold affects the test classification performance. The error bar for standard deviation in each figure has been shaded.
training and 10k images with the clean label for testing. 10% of the noisy training examples of all datasets are left out as noisy validation set for model selection.
5.1.2 COMPARISON METHOOSS
We compare the proposed method with several state-of-the-at approaches: (1) CE, which trains the classification network with the standard cross-entropy loss on noise datasets. (2) GCE (Zhang & Sabuncu 2018) 8), which unites the mean absolute error loss and the cross-entropy loss combat noisy labels. (3) APL MM et al., 2020), which combines two mutually reinforeing robust loss functions,
(4) Decoupling (Malach & Shalev- Shwartz 2017). , which trains two networks on samples whose predictions from two networks are
employ its combination of NCE and RCE for comparison.
different. (5) MentorNet (Jiang al. 2018), Co-teaching (Han et al. 2018b), and Co-teaching+ YYu jtt al, 2019) mainly handle noisy labels s by training networks on instances with small loss values. (6) Joint (Tanaka et al., 2018), which jointly optimizes the network parameters and the sample labels. The hyperparameters a and 3 are set to 1.2 and 0.8, respectively.
(7) DMI (Xu et al. (019).. which proposes novel informationntheoretti loss function for training neural networks robust label noise. (8) Forward (Patrini et al.) 2017), Reweight (Liu & Tao 2016). and T-Revision (Xia et al., 2019 utilize class-dependent transition matrix T to correct the loss function. (9) PTD (Xia al., 2020a), estimates instance instance-dependee dependent transition matrix by combing part-dependent transition matri ces, which the most related work to our proposed method. We also provide comparison results between our method and DivideMix(Li et al., 2020a), which is hybrid algorithm that combines multiple powerful techniques, c.g. Gaussian Mixture Model, MixMatch, MixUp. regularization and asymmetric noise penalty in Appendix. As for our method, we simply model the instance-dependent matrix by employing a neural network.
5.2 HYPER-PAREE PARAMETER SENSITIVITY
The quality of the distilled dataset relies on the choice of distillation threshold Amas (denoted as in the following paragraph) in Theorem [ To further explore the effect of f. we conduct hyper- parameter sensitivity studies CIFAR-10 10 in this section.
In Figure 2(a) we show the instance-dependen transition matrix approximation error when em- ploying the class-dependent transition matrix, the revised class-dependent transition matrix, and our proposed instance-dependentt transition matrix estimation method. The error measured by E norm between the ground-truth transition matrix and the estimated transition matrix. For each instance, we only analyze the approximation error of a specific row because the noisy label is generated by of the instance-dependent transition matrix. The "Class-dependent" represents the class- dependent transition matrix learning methods (Patrini et al.) 2017). the 'T-Revision' indicates the class-dependent transition matrix is revised by learnable slack variable (Xia et al. 2019). Our pro-
7

---

Under review as a conference paper at ICLR 2022
Table 1:
Means and standard deviations (percentage) of classification accuracy on F-MNIST with different label noise levels. *-V' indicates matrix revision Xia et al.
2019p.
IDN- 10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
88,65 = 0.45
88.31 = 0.37
85.22 = 0.56
76.56 + 2.50
67.42 = 3,91
GCE
90.86 + 0.38
88.59 + 0.26
86.64 0.76
76.93 1.64
66.69
1.07
APL
86.46 = 0.27
85.32 0.88
85.59 0.85
74.66 +
62.82 + 0.44
Decoupling
89.83 = 0,45
86.01 1.01
78.78 0.53
67.33 + 1.33
MentorNet
90.35 = 0.64
87.92
87.24 0.99
79.01 2.30
66.44 2
Co To-teaching
90.65 = 0.58
8.41
06.98
78.92 1.36
67.66 2.42
Co-teaching+
90.47 = 0.98
89.15
86.15
79.23
63.49 2.94
Joint
80.19 0.99
78.46 1.24
7
65.93 2.08
50.93
DMI
91.58 + 0.46
90.33 0.66
85.96 +
77.77 2.15
68.02 1.59
Forward
89.65 + 0.24
88.61 0.77
85.01 0.43
78.59 + 0.38
67.11 + 1.46
Reweight
90.33 = 0.27
0.44
76.07 1.93 0.42
67.66 1.65
S2E
91.04 = 0.92
89.93 1.08
76.12 1.21
70.24 2.64
T-Revision
91.36 = 0.39
90.24 1.01
78.24 1.12
69.04 = 2.92
PTD
92.03 = 0.33
90.78 0.64
87.86 - 0.78
79.46 1.58
73.38 = 2.25
Ours
96.06 + 0.71
94.97 0.33
91.47 1.36
82.88 + 2.72
76.35 + 3.79
Ours-V V
96.93 = 0.31
95.55 + 0.59
92.24 1.87
83.43 + 1.72
76.89 + 4.26
Table 2: Means and standard deviations (percentage) of classification accuracy on CIFAR-10 with different label noise levels. *V'' indicates matrix revision (Xia al. 2019).
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
73.54 + 0.14
71.49 + 1.35
67.52 1.68
58.63
4.92
51.54 + 2.70
GCE
74.24 = 0.89
72.11 0.43
69.31 = 0.18
56.86 0.92
53.44 # 1.28
APL
71.12 0.19
65.17 0.35 0.27
53.22
47.31 # 1.41
Decoupling
73.91 = 0.37
7
70.85 1.88
54.73
52.04 2.09
MentorNet
74.93 = 1.37
72.32 1.04
57.85 +
52.96 1.98
Co-teaching
75.49 + 0.47
75.93
74.86 00.42
59.07 + 1.03
55.62 3.93
Co-teaching+
74.77 + 0.16
0.61
71.92 2.13
59.15 0.87
0.87
53.02 + 3.34
Joint
75.97 0.98
0.45
75.93 = 1.65
63.22
45.843312
DMI
74.65 = 0.13
0.88
73.93 0.34
60.22 3.47
54.35 + 2.28
Forward
72.35 + 0.91
0.32
66.53 1.96
58.63 1.25
52.33 1.65
Reweight
73.55 + 0.32
0.57
60.32 + 1.03
52.03 + 1.70
S2E
75.93 + 1.01
75.53
11.112.5
64.62 0.68
56.03 1.07
T-Revision
74.01 + 0.45
4.64
71.15
0.43 75.42
59.93 + 1.33
55.67 -: 2.07
PTD
76.33 + 0.38
76.05 + 1.72
1.33
65.92 2.33
56.63 1.88
Ours
81.73 + 0.56
80.26 + 0.63
77.69
1.37
71.96
2.27
59.15 + 3.11
Ours-V
82.16 + 1.01
80.37 + 1.98
78.82
1.07
72.93 + 4.00
60.33 + 5.29
posed method estimates an nestance-dependent transition matrix for each input. can be observed that proposed method can achieve much lower approximation error. Figure 2(b) shows the classification performance our proposed method when choosing various distillation threshold ős. When I is not too large too small, our method is not sensitive the choice of More experi- mental results on the quality of distilled dataset when applying different values ő are included in the appendix B We manually
0.3, decent trade-off between distillation accuracy and the number distilled examples, all experiments later to avoid laborious aeper-parameter tuning and access the true noise rate.
5.3 COMPARISON WITH THE STATE-OF-TIEE-ATE
Results on synthetic noisy datasets. Table 2 and report the classification accuracy on the datasets of F-MNIST, CINNST,,IFARRIO, and SVHN, respectively.
For F-MNIST, our method surpasses all the baseline methods by large margin. Equipping the transition matrix revision (-V) (Xia al. 2019) can further boost the performance of our method.
8

---

Under review as a conference paper at ICLR 2022
Table 3:
Means and standard deviations (percentage) of classification accuracy on SVHN with dif- ferent label noise levels. *-V' indicates matrix revision Xia et al. 2019).
IDN-10% 10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
90.39 = 0.13
89.04 = 1.32
85.65 = 1.84
79.94 + 2.71
61.01 = 5,41
GCE
90.82 + 0.15
89.35 + 0.94
86.43 0.63
81.66 1.58
54.77 + 0.25
APL
71.78 + 0.76
89.48 1.67
83.46 + 2.17
77.90
55.25 3.77
Decoupling
90.55 = 0.83
0.77
85.03 1.63
2.73
56.76 1.87
MentorNet
90.28 = 0.52
85.89 0.73
82.63
55.27 4.14
Co fo-teaching
91.05 = 0.33
$
59.56
+ 2.34
Co-teaching+
92.83 = 0.87
90.73 1.39
86.37 1.66
75.24
54.58 3.46
Joint
= 0.62
$5.37:004
78.98 2.98
59.14 + 3.22
DMI
92.11 0.49
91.63 0.87
86.98 + 0.36
81.11 + 0.68
63.22 3.97
Forward
90.01 + 0.78
89.77 + 1.54
86.70 1.44
80.24 2.77
57.57
+ 1.45
Reweight
91.06 = 0.19
9
83.79
55.08
1.25
S2E
92.70 = 0.51
92.02
88.77 1
2.19
65.39 2
T-Revision
93.07 = 0.79
92.67 0.88
88.49
1.44
8.77
67.64 2.57
PTD
93.77 = 0.33
92.59 1.07
89.64 = 1.98
83.56
71.57 = 3.32
Ours
96.05 + 0.32
94.97 0.58
93.99
1.24
87.67 1.29
78.13 + 4.62
Ours-V
96.37 = 0.77
95.12 + 0.40
94.69 = 0.24
88.13 + 3.23
78.71 + 4.37
For SVHN and CIFAR-IO, the superiority of our method is gradually revealed along with the noise rate increase, which shows that our method can handle the extremcly hard situation much better. Specifically, the classification accuracy of our method is 5.83% higher than PTD (the best statisti- cally consistent baseline) on CIFAR-IO in the IDN-10% case, and the performance gap is enlarged to 7.01% in the IDN-40% case. On the SVHN, the classification accuracy of our method is 2.60% higher than PTD the IDN-10% case, 5.05% higher than PTD in the IDN-30% case, and 7.14% higher than PTD in the most challenging IDN-50% case.
5.3.1 RESULTS ON ROAL-WORLLD DATASETS
The noise model of real-world datasets is more likely be instance-dependent. Our proposed method also performs favorably on the challenging Clothing dataset (Table 4). which proves that our method is more flexible to handle such real-world noise problem.
Table 4: Classification accuracy on Clothing/M. In the experiments, only noisy samples are ex- ploited train and validate the deep model.
CE
Decoupling
MentorNet
Co-teaching
Co-teaching+
Joint
DMI
68.88
54.53
56.79
60.15
65.15
70.88
70.12
Forward
Reweight
T-Revision
PTD
PTD-V
Ours
Ours-V
69.91
70.40
70.97
70.07
70.26
73.33
73.39
6 CONCLUSION
In this paper, we focus on training the robust classifier with the challenging instance-dependent label noise. To address the issues of existing clean label transition matrix, we propose to directly build the transition between Bayes optimal labels and noisy labels. By reducing the feasible solution of the transition matrix estimation, we prove that the instance-dependent label transition matrix that relates Bayes optimal labels and noisy labels can be directly learned using deep neural networks. The main limitation of our method comes from that the distilled examples are collected out of noisy data leading unavoidable data distribution bias to the transition matrix estimation. Experimental results demonstrate that the proposed method is more superior in dealing with instance-dependent label noise, especially for the case of high-level noise rates.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
This paper doesn't t raise any ethics concerns. This study doesn't involve any human subjects, prac- tices data releases, potentially harmful insights, methodologies and applications, pontential conflicts of interest and sponsorship, diseriminnaioon/iasffiiinn concerns, privacy and security is- sues, legal compliance, and research integrity issues.
REFERENCES
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
Peter L Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138 156, 2006.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores
make instance-dependent label-noise learning possible. arXiv preprini urXiv:2001.03777 2020.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Support vector machines under adversarial label noise. In ACML, 2011.
Avrim Blum, Adam Kalai, and Hal Wasserman. Noise-tolerant learning, the parity problem, and the statistical query model. Journal of the ACM, 50(4):506-51, 2003.
Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance-and label-dependent label noise. In ICML, 2020.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In ECCV, pp. 135-150, 2018.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In NewrIPS, pp. 5836-5846, 2018a.
Bo Han, Quanming Yao. Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 3527-8537, 2018b.
Bo Han, Gang Niu, Xingrui Yu, Quamming Yao, Miao Xu, Ivor W Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In ICML, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. In NewrIPS, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICMI, pp. 2309-2318, 2018.
Jan Kremer, Fei Sha, and Christian Igel. Robust active label correction. In AISTATS, pp. 308-316, 2018.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020a.
Mingchen Li. Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparamettrrize neural networks. In AISTATS, 2020b.
10

---

Under review as a conference paper at ICLR 2022
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label- noise learning without anchor points. arXiv preprint arXiwv2102.002400, 2021.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, pp. 1910-1918, 2017.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda Early-learning regularization prevents memorization of noisy labels. In NeurIPS, 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting, IEEE Transactions on pattern analysis and machine intelligence, 38(3):44-445 2016.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In ICML, 2020.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label corruption. In ICLR, 2020.
Xingjun Ma, Yisen Wang, Michael Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-driee learning with noisy labels. In ICML., pp. 3361-3370, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In ICML, 2020.
Eran Malach and Shai Shwaar-Swwrrt.. Decoupling" when to update" from" how update". In NewrlPS, pp. 960-970, 2017.
Naresh Manwani and P.S. Sastry, Noise tolerance under risk minimization. IEEE Transactions Cybernetics, 2013.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurlPS, pp. 1196-1204, 2013.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox Self: Learning to filter noisy labels with self- ensembling. In ICLR, 2020.
Curtis G Northcutt, Tailin Wu, and Isaac L Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. In UAI, 2017.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: loss correction approach. CVPR, pp. 1944-1952, 2017.
Scott E Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 2015.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331 5331-434, 2018.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, pp. 838-846, 2015.
Jun Shu, Qian Zhao, Zengben Xu, and Deyu Meng. Meta transition adaptation for robust deep learning with noisy labels. arXiv preprint arXiv:2006.0569) 2020.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, 2018.
Kiran K Thekumparampil. Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. In NeurIPS, pp. 10271-10282, 2018
Arash Vahdat, Toward robustness against label noise in training deep discriminative neural networks. In NewrIPS, pp. 5596-5605, 2017.
11

---

Under review as a conference paper at ICLR 2022
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, pp. 849, 2017.
Xiaobo Wang, Shuo Wang, Jun Wang, Hailin Shi, and Tao Mei. Co-mining: Deep face recognition with noisy labels. In ICCV, pp. 9358-9367, 2019.
Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A new perspective on learning with label noise. arXiv preprint arXiv:2006.07831, 2020.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in nooie learning? In NeurlPS, pp. 6838-688866 0849, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS, 2020a.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Parts-dependent label noise: Towards instance-dependent label noise. In NeurIPS, 2020b.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2021.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L_dmi: A novel information--hheortie loss function for training deep nets robust label noise. NeurIPS, pp. 6222-6233, 2019.
Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine learming, 95(3):291-377, 2014.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2021a.
Shuo Yang, Songhua WU, Tongliang Liu, and Min Xu. Bridging the gap between few-shot and many-shot learning via distribution calibration, 2021b.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James T Kwok. Searching to exploit memo- rization effect learning with noisy labels. In ICML, 2020a.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix in label-noise learning. In NeurlPS, 2020b.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement benefit co-teaching? In ICML, 2019.
Xiyu Yu, Tongliang Liu, Mingming Gong, Kun Zhang, Kayhan Batmanghelich, and Dacheng Tao. 2017.
Transfer learning with label noise. arXiv preprint
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. ECCV, pp. 68-83, 2018.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017b.
Yikai Zhang, Songzhu Zheng. Pengxiang Wu, Mayank Goswami. and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In ICLR, 2021.
12

---

Under review as a conference paper at ICLR 2022
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS. pp. 3778-8788. 2018.
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML, pp. 11447-11457, 2020.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. arXiv preprint arXiv:2012.11885 2020.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability alternative to anchor points when learning with noisy labels. arXiv preprint 2rXiv:2100205291, 2021.
13

---

Under review as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
We use ResNet-18 (He et al. 2016) for F-MNIST, ResNet-34 networks (He et al. (016) for CIEAR- 10 and SVHN. We first SGD with momentum 0.9, batch size 128, and initial learning rate of 0.01 to warm the network for five epochs on the noisy dataset. For Clothing IM, we use a ResNer50 pretrained on ImageNet, and the learning rate is set as le-3. Then, we use the warm- upped network to collect distilled examples from noisy datasets according to Section 4.1 After distilled examples collection, we train the ristance-dependee the distilled dataset for epochs. The Bayes label transition network is the same architecture as the classification network, but the last linear layer is modified according to the transition matrix shape. The optimizer of the Bayes label transition network is SGD, with momentum of 0.9 and learning rate of 0.01. Then, we fix the trained Bayes label transition network to train the classification network. The Bayes label transition network is used to generate a transition matrix for each input image; the transition matrix is used to correct the outputs of the classification network to bridge the Bayes posterior and the noisy posterior. The classification network is trained on the noisy dataset for 50 epochs for F-MNIST, CIFAR-IO and SVHN and for 10 epochs for Clothing 1M using Adam optimizer with learning rate of 5e and weight decay of le 4. We also apply the transition matrix revision technique (Xia et al. 2019) to boost the performance. Note for fair comparison, we do not use any data augmentation technique in all experiments as in (Xia al.) 2020a). All the codes are implemented in Py PoTorc 1.6.0 with CUDA 10.0, and run on NVIDIA Tesla V100 GPUs.
transition matrix estimator network on
B THE QUALITY OF DATASET DISTILLATION
The distillation threshold أ controls how many examples can be collected out of noisy dataset and the quality of the distilled dataset. We analyse the effect of f the CIFAR-10 dataset in Table. [ The distillation accuracy is computed by counting how many inferred Bayes optimal labels are consistent with their corresponding clean labels among all distilled examples.
ò 0.3
0.5
Noise rate
distill. acc.
# of distilled examples
distill. acc.
# of distilled examples
IDN-
98%
27983 50000
99%
19983 50000
IDN-30%
96%
17673 50000
99%
10673 / 50000
IDN-50%.
94%
8029 / 50000
98%
5098 / 50000
Table 5: Distillation quality analysis on CIFAR-10, with total 50,000 examples the original non- distilled dataset.
C ABLATION ON BAYES LABEL TRANSITION MATRIX
To verify the effectiveness of the estimated Bayes label transition matrix, compare method with some ablated variants, c.g. directly train a classifier on the distilled dataset and relabel the noisy dataset using the classifier trained distilled dataset.
CIFAR-10 10 IDN-10%
CIFAR-10 IDN-30%
Clothing M
Training classifier on distilled dataset Relabeling noisy dataset
74.56
67.42
62.37
76.68
70.73
64.98
Ours
82.16
78.82
73.39
D COMPARISION WITH DIVIDEMIX
DivideMix has a much more complicated pipeline than us and is not statistically consistent al- gorithm. We compare our method with DivideMix further show the effectiveness and flexibility of our proposed method. Compared with DivideMix, our method exhibit competitive performance when noise rate is low and surpass DivideMix by a large margin the worst noise cases (3.22%
14

---

Under review as a conference paper at ICLR 2022
performance improvement CIFAR-I0 10 and 4.38% SVHN, both under IDN-50% ), with much simpler and flexible algorithm design.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
DivideMix
96.37 + 0.72
95.92 + 0.73
90.37 + 0.83
80.92 + 2.32
74.63 + 3.76
Ours
96.93 + 0.31
95.55 + 0.59
92.24 1.87
83.43
1.72
76.89
4.26
Table 6: F-MNIST
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
DivideMix
83.31 + 0.23
81.42 + 0.28
80.73 + 1.28
70.29
1.97
57.11 3.64
Ours
82.16 + 1.01
80.37 + 1.98
78.82 + 1.07
72.93 + 4.00
60.33 + 5.29
Table 7: CIFARIO
IDN-10%
IDN-20%
IDN-30%
IDN-400
IDN-50%
DivideMix
96.02 + 0,45
95.73 + 0.48
92.07 + 1.47
85.69 2.47
74.33 4.07
Ours
96.37 + 0.77
95.12 + 0.40
94.69 + 0.24
88.13 + 3.23
78.71 + 4.37
Table SVHN
15


Reviews:
[{"review_id": "md1XodR_x_g", "reviewer": "Reviewer_M4SW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper mainly applies the idea in Cheng et al. (2020) to multiclass problems to identify \"distilled examples\". The extension is straightforward, but is problematic. The use of *Bayes label transition matrix* seems novel, but lacking justifications make it a minor novelty. Finally, using DNNs for estimations has nothing new. Overall this paper is not novel nor significant. The correctness and clarity are also of concern.\n\n---\n\nStrengths:\n\n- The use of *Bayes label transition matrix* to learn from instance-dependent noisy labels seems novel.\n- The experiments seem comprehensive.\n\n---\n\nWeaknesses:\n\n1. The idea to identify \"distilled examples\" is neither novel nor significant. This paper simply applies the setting and the idea in Cheng et al. (2020), the bounded instance-dependent label noise paper. Although the authors claimed they extended Cheng et al. (2020) from binary to multiclass, this extension is neither novel nor significant. Instead, it is quite problematic. First, Cheng et al. (2020) already has showed that their framework can be easily extended to multiclass setting in the supplementary material. Second, in this paper, the authors used the same theorem (Theorem 1) to identify distilled examples as that in Cheng et al. (2020). This is problematic: although it is true that $\\eta_y(x) > 0.5$ implies $y$ is the Bayes label in multiclass, this is only a sufficient condition. In fact, requiring $\\eta_y(x) > 0.5$ is the so-called \"dominant label\" condition [1], and is generally considered as a quite strong assumption. Third, Cheng et al. (2020) also considered covariate shift and the case when $\\rho_{max}$ is not known, but such extensions are missing in this work. To summarize, this work only trivially applies some of Cheng et al. (2020)'s results to multiclass to identify \"distilled examples\". This is not a meaningful contribution.\n\n2. The use of *Bayes label transition matrix* seems novel, but lacks justifications. The paper claims (second paragraph in page 2, (b)) that \"The feasible solution space of the Bayes label transition matrix is much smaller than that of the clean label transition matrix.\", but no justifications are given for the claimed \"sparsity\". Recall that \"sparsity\" means that entries of a matrix are mostly zeros. Such claim is not true for the *Bayes label transition matrix*, even if the Bayes class posterior is sparse. In Section 4.2, the authors acknowledged that \"if we have a distilled example for the i-th class, we can only make use of it to learn the i-th row of the transition matrix.\", but the explanation for why the other rows are not random nor learnable is far from satisfactory. More theoretical and empirical work is needed to understand and justify the use of *Bayes label transition matrix*.\n\n3. The use of DNNs to estimate *Bayes label transition matrix* has nothing new. No insights about how to design such nets are provided.\n\n4. The noise generating process (Algorithm 1) does not reflect the bounded noise assumption. At the beginning of Section 3, it is assumed that $\\forall x, 0 \\le \\rho_y(x) \\le \\rho_{max} < 1$, where $\\rho_y(x)  = P(\\tilde{Y} = y | Y \\ne y, x)$. However, in Algorithm 1, it is only guaranteed that for each $i$, $P(\\tilde{Y} \\ne y_i | Y = y_i, x_i) \\le \\rho_{max}$. Therefore, the proposed noise generating process does not satisfy the bounded noise assumption.\n\n5. In experiments, specifically Section 5.2, the baseline does not make sense, because both \"class-dependent\" and \"T-Revision\" are for CCN setting (the label noise only depends on the label, but not on the instance). The authors should have compared with methods that are designed for instance-dependent label noise.\n\n6. The clarity of this paper is of concern. The writing quality is below the requirements of this conference. For example, the paper used a long paragraph in page 1 to discuss about *statistical consistency*, but such information is not directly related to this work since this work does not have consistency results. BTW, Xia et al. (2020b) is not classifier-consistent.\n\n---\n\nOther minor issues:\n\n1. In Figure 1, the noisy class posterior does not sum to 1.\n \n---\nRefs:\n\n[1] Tong Zhang:\nStatistical Analysis of Some Multi-Category Large Margin Classification Methods. J. Mach. Learn. Res. 5: 1225-1251 (2004)", "summary_of_the_paper": "This work proposes to use *Bayes label transition matrix* (instead of the common *clean label transition matrix*) to learn from instance-dependent noisy labels. The Bayes label transition matrix is estimated by DNNs.", "summary_of_the_review": "This paper has no novel and significant contribution. The correctness and clarity are also of serious concern. Therefore I vote for a reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "atUHboC_dit", "reviewer": "Reviewer_6uGG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis paper transforms the original clean transition matrix estimation problem to the estimation of transition from Bayes optimal labels to noisy labels, which is quite interesting and novel for the noisy label learning.  A DNN is used to estimate the transition matrix, which can be optimized simultaneously with the classifier in an end-to-end manner. This is also a major contribution in this area. Extensive experiments are conducted to demonstrate the method. And the results have shown great improvements. This can be considered as another highlight of this paper.\n\nMinor weaknesses:\n\nThe authors introduced a parametric Bayes label transition network, which is a DNN. Thus, the authors should compare the extra parameter quantity of the proposed method with those of related methods.\n\nIt would be good if the author can provide an example of the learned transition matrix from Bayes optimal labels to noisy labels. Or there should be some investigation about how the Bayes-transition matrix was being estimated, empirically or theoretically. \n\n\n\n\n\n", "summary_of_the_paper": "The transition matrix plays a vital role in modeling label noise. Current methods focus on modeling the transition from clean labels to noisy labels. While this paper alternatively models the transition from Bayes optimal labels to noisy labels. Since we usually use the Bayes optimal labels for prediction. This transformation will not affect the practical use but makes the estimation of the matrix much easier.  Specifically, this paper designs a DNN to estimate the transition matrix. During training, the DNN can be optimized with the classifier simultaneously in an end-to-end manner.  Extensive experiments are conducted to support the proposed method.", "summary_of_the_review": "The method has made certain theoretical contributions to the label noise learning community.  The algorithm also clearly outperforms the current SotA. Therefore, I would like to lean on the positive side of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NmSc2oWvAEB", "reviewer": "Reviewer_ieRv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n(1) The view of dealing with Bayes class posterior instead of clean class posterior is new, and seems provides benefits.\n\n(2) Explicitly parameterizing the transition matrix is good, and might have advantages in task generalization consideration.\n\n\nWeakness:\n\n(1) No theoretical guarantee that dealing with Bayes class posterior transition is better than conventional noise transition matrix estimation methods, either in robustness or generalization error.\n\n(2) It seems that final method is not intrinsically related with Bayes optimal labels, except that soft labels were replaced by hard ones.\n\n(3) Experiments were only conducted on datasets with relatively few classes, such as CIFAR-10  and SVHN with only 10 classes.", "summary_of_the_paper": "This paper proposed a new noise-transition-matrix based label correction method for robust deep learning against label noise. Different from conventional methods, that estimating noise transition matrix from clean class posterior to the noisy one, the proposed method try to estimation the transition matrix from Bayes class posterior to the noisy one, which is expected to reduce the feasible solution space, and lead to better performance.", "summary_of_the_review": "The idea is new and interesting, but final solution is not satisfactory, especially for theoretical guarantee.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JBGrWBkfREO", "reviewer": "Reviewer_dRUQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To the best of my knowledge, this is the first paper that introduces to model Bayes label transition rather than clean label transition. The paper is well written and easy to follow. Results are very promising compared to other SOTA methods.\n\nMy main concern is on the novelty of the contribution. Given that the idea of modelling Bayes label transition is a new and interesting idea, the techniques are mostly based on existing works:\n- Step 1: The Bayes label collection closely follows the framework proposed by Cheng et al. (2020). Note that the extension to multiclass setting, despite being straightforward as agreed by the authors, was also presented in the Supplementary Section B in Cheng et al. (2020) already.\n- Step 2: By replacing clean labels with Bayes labels, the authors model transition matrix with a DNN. This seems to be the main contribution (as it is included in the paper title too?) but in experiments the authors simply did straightforward adaptation of standard DNN architecture for this task. Some ablation or other experiments to compare to non-parametric models for (Bayes) label transition could also support the paper.\n- Step 3: The final classification algorithm uses F-correction proposed by Patrini et al. (2017).\n\nGiven the relatively weak methodological contribution, the current paper is also a bit weak in terms of theoretical contributions. For instance,\n- As stated in Sec 1, one of the motivations for modelling Bayes label transition is that Bayes label has a smaller feature solution space hence easier to model because it is one-hot, but it would support the paper if the authors can include a formal definition and/or theoretical justification on this claim.\n- Besides, I would argue that Bayes label transition being easier to model does not mean that using Bayes labels to learn a weakly supervised model is preferred to using inferred clean labels in practice, unless the impact of error propagation from label transition to weakly supervised model is carefully studied. Again theoretical justification may strongly improve the paper.\n- In Sec 4.3, the authors acknowledged that the Bayes transition estimation through distilled examples breaks the assumption that F-correction requires an unbiased estimate of the transition matrix, but then only briefly mentioned that the network can generalise well to non-distilled examples empirically. This is only a claim at best, and also breaks the motivation that the authors are looking for classifier-consistent algorithms.\n\nOther minor concerns/suggestions:\n- Since the paper significantly relies on the distilled example collection from Cheng et al. (2020), Cheng et al. (2020) spent quite some space discussing the known issues and mitigation strategies of their distilled example collection (e.g., the automatically-collected distilled examples only is not statistically consistent, covariate shift correction, how to collect without knowledge on noise rate bound). The current paper only has ablation on the impact of noise rate bound (Sec 5.2), it would be worth discussing the impact of any other known issues of their distilled example collection on the currently proposed methods.", "summary_of_the_paper": "The paper proposes to estimate an Instance-Dependent Noise (IDN) label transition matrix. Instead of modelling the clean label transition as typically done in previous literature, the authors propose to estimate the Bayes label transition using a DNN, motivate by several advantages including theoretically guaranteed Bayes label collection and smaller feasible solution space, hence empirically easier to model. Controlled experiments show consistent improvement over other SOTA methods in noisy label transition.", "summary_of_the_review": "This paper introduces an interesting idea to model Bayes label transition rather than clean label transition based on several existing literatures. However, the overall contribution of the paper is a bit weak for ICLR standards IMHO, and some theoretical justification are currently missing to support most of their claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received a majority voting of rejection. In the internal discussion, one reviewer updated his/her score from 1 to 3 according to the author response. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the recommendation on this paper, *rejection*. Here are the comments that I summarized, which include my opinion and evidence.

**Interesting Idea**

Every reviewer including me agree that the idea of modelling Bayes label transition is novel and interesting. 

**The motivation lacks of supportive evidence**

The second motivation that "the feasible solution space of the Bayes label transition matrix is much smaller than that of the clean label transition matrix" is not well supported. The authors should theoretically or empirically demonstrate this point. The current description on uncertainty is not strong enough. Moreover, if so, the benefits are not illustrated. The feasible solution space, even with a small coverage area is continuous with infinite solutions. 

**A new concept**

The authors tried to sell the concept of a new transition matrix, but failed. I believe it might result from the organization and presentation. The authors spent too much pages introducing others' work. At least, a formal definition of the new concept should be given. In the current version, Definition 1 is from Cheng et al., 2020 on distilled examples.

**Title**

Literally from title, I guess DNN is a key component or a selling point of this paper. Actually no. We expect the authors could provide the insights on what benefits are using DNN over other techniques and how to apply DNN to estimate the transition matrix. If this is not a selling point, this word might be removed from the title.

**Algorithm 1**

I am a little surprised that the only algorithm listed in this paper is label noise generation. Instead the proposed algorithm of this paper is expected.

**Experimental Evaluation**

The experimental results look much better than other baselines. It is a little confusing that some best results are bold, some not. 

**Presentation**

Although I did not notice obvious grammar errors, some sentences are very long (3 lines). They made difficulties to follow the idea. I have to read these sentences several times. In my eyes, this is the biggest one! Presentation means how to sell the idea to audience (not only reviewers, but also future readers) in an easy understood way. The current version spent much space introducing others' work; on the contrary, the original or key part is not well illustrated. 

Although this paper has a novel idea and good experimental support, other issues listed above demonstrate the current version is not ready for a top-tier conference. No objection from reviewers was raised to again this recommendation.
Decision: Reject