Year: 2021

Abstract:
In label-noise learning, estimating the transition matrix is a hot topic as the matrix plays an important role in building statistically consistent classifiers. Traditionally, the transition from clean labels to noisy labels (i.e., clean label transition matrix) has been widely exploited to learn a clean label classifier by employing the noisy data. Motivated by that classifiers mostly output Bayes optimal labels for prediction, in this paper, we study to directly model the transition from Bayes optimal labels to noisy labels (i.e., Bayes label transition matrix) and learn a classifier to predict Bayes optimal labels. Note that given only noisy data, it is ill-posed to estimate either the clean label transition matrix or the Bayes label transition matrix. But favorably, Bayes optimal labels have less uncertainty compared with the clean labels, i.e., the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not. This enables two advantages to estimate the Bayes label transition matrix, i.e., (a) we could theoretically recover a set of noisy data with Bayes optimal labels under mild conditions; (b) the feasible solution space is much smaller. By exploiting the advantages, we estimate the Bayes label transition matrix by employing a deep neural network in a parameterized way, leading to better generalization and superior classification performance.

text: In label-noise learning, estimating the transition matrix is a hot topic as the matrix plays an important role in building statistically consistent classifiers. Traditionally, the transition from clean labels to noisy labels (i.e., clean label transition matrix) has been widely exploited to learn a clean label classifier by employing the noisy data. Motivated by that classifiers mostly output Bayes optimal labels for prediction, in this paper, we study to directly model the transition from Bayes optimal labels to noisy labels (i.e., Bayes label transition matrix) and learn a classifier to predict Bayes optimal labels. Note that given only noisy data, it is ill-posed to estimate either the clean label transition matrix or the Bayes label transition matrix. But favorably, Bayes optimal labels have less uncertainty compared with the clean labels, i.e., the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not. This enables two advantages to estimate the Bayes label transition matrix, i.e., (a) we could theoretically recover a set of noisy data with Bayes optimal labels under mild conditions; (b) the feasible solution space is much smaller. By exploiting the advantages, we estimate the Bayes label transition matrix by employing a deep neural network in a parameterized way, leading to better generalization and superior classification performance.

heading: 1 INTRODUCTION
text: The study of classification in the presence of noisy labels has been of interest for three decades (Angluin & Laird, 1988), but becomes more and more important in weakly supervised learning (Thekumparampil et al., 2018; Li et al., 2020b; Guo et al., 2018; Xiao et al., 2015; Zhang et al., 2017a; Yang et al., 2021b;a). The main reason behind this is that datasets are becoming bigger and bigger. To improve annotation efficiency, these large-scale datasets are often collected from crowdsourcing platforms (Yan et al., 2014), online queries (Blum et al., 2003), and image engines (Li et al., 2017), which suffer from unavoidable label noise (Yao et al., 2020a). Recent researches show that the label noise significantly degenerates the performance of deep neural networks, since deep models easily memorize the noisy labels (Zhang et al., 2017a; Yao et al., 2020a).
Generally, the algorithms for combating noisy labels can be categorized into statistically inconsistent algorithms and statistically consistent algorithms. The statistically inconsistent algorithms are heuristic, such as selecting possible clean examples to train the classifier (Han et al., 2020; Yao et al., 2020a; Yu et al., 2019; Han et al., 2018b; Malach & Shalev-Shwartz, 2017; Ren et al., 2018; Jiang et al., 2018), re-weighting examples to reduce the effect of noisy labels (Ren et al., 2018), correcting labels (Ma et al., 2018; Kremer et al., 2018; Tanaka et al., 2018; Reed et al., 2015), or adding regularization (Han et al., 2018a; Guo et al., 2018; Veit et al., 2017; Vahdat, 2017; Li et al., 2017; 2020b; Wu et al., 2020). These approaches empirically work well, but there is no theoretical guarantee that the learned classifiers can converge to the optimal ones learned from clean data. To address this limitation, algorithms in the second category aim to design classifier-consistent algorithms (Yu et al., 2017; Zhang & Sabuncu, 2018; Kremer et al., 2018; Liu & Tao, 2016; Northcutt et al., 2017; Scott, 2015; Natarajan et al., 2013; Goldberger & Ben-Reuven, 2017; Patrini et al., 2017; Thekumparampil et al., 2018; Yu et al., 2018; Liu & Guo, 2020; Xu et al., 2019; Xia et al., 2020b), where classifiers learned on noisy data will asymptotically converge to the optimal classifiers defined on the clean domain.
The label transition matrix T (x) plays an important role in building statistically consistent algorithms. Traditionally, the transition matrix T (x) is defined to relate clean distribution and noisy distribution, where T (x) = P (Ỹ | Y,X = x) andX denotes the random variable of instances/features, Ỹ as the variable for the noisy label, and Y as the variable for the clean label. The above matrix is denoted as the clean label transition matrix, which is widely used to learn a clean label classifier by employing the noisy data. The learned clean label classifier is expected to predict a probability distribution over a set of pre-defined classes given an input, i.e. clean class posterior probability P (Y | X). The clean class posterior probability is the distribution from which clean labels are sampled. However, Bayes optimal labels Y ∗, i.e., the class labels that maximize the clean class posteriors Y ∗ | X := argmaxY P (Y | X), are mostly used as the predicted labels and for computing classification accuracy. Motivated by this, in this paper, we propose to directly model the transition matrix T ∗(x) that relates Bayes optimal distribution and noisy distribution, i.e., T ∗(x) = P (Ỹ | Y ∗, X = x), where Y ∗ denotes the variable for Bayes optimal label. The Bayes optimal label classifier can be learned by exploiting the Bayes label transition matrix directly.
Studying the transition between Bayes optimal distribution and noisy distribution is considered advantageous to that of studying the transition between clean distribution and noisy distribution. The main reason is due to that the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not. Two advantages can be introduced by this to better estimate the instance-dependent transition matrix: (a) We can collect a set of examples with theoretically guaranteed Bayes optimal labels out of noisy data.
The intrinsic reason that Bayes optimal labels can be inferred from the noisy data while clean labels cannot is that Bayes optimal labels are the labels that maximize the clean class posteriors while clean labels are sampled from the clean class posteriors. In the presence of label noise, the labels that maximize the noisy class posteriors could be identical to those that maximize the clean class posteriors (Bayes optimal labels) under mild conditions (Cheng et al., 2020). Therefore some instances’ Bayes optimal labels can be inferred from their noisy class posteriors while their clean labels are impossible to infer since the clean class posteriors are unobservable, as shown in Figure 1. (b) The feasible solution space of the Bayes label transition matrix is much smaller than that of the clean label transition matrix. This is because that Bayes optimal labels have less uncertainty compared with the clean labels. The transition matrix defined by Bayes optimal labels and the noisy labels is therefore sparse and can be estimated more efficiently with the same amount of training data.
These two advantages naturally motivate us to collect a set of examples with their theoretically guaranteed Bayes optimal labels out of the noisy data to learn to approximate the Bayes label transition matrix T ∗(x). Due to the high complexity of the instance-dependent matrix T ∗(x), we simplify its estimation by parameterizing it using a deep neural network. The collected examples, inferred Bayes optimal labels, and their noisy
labels are served as data points to optimize the deep neural network to approximate the T ∗(x). Compared with the previous method (Xia et al., 2020a), which made assumptions and leveraged hand-crafted priors to approximate the instance-dependent transition matrices, we train a deep neural network to estimate the instance-dependent label transition matrix with a reduced feasible solution space, which achieves lower approximation error, better generalization, and superior classification performance.

heading: 2 RELATED WORK
text: Noise model. Currently, there are several typical label noise models. Specifically, the random classification noise (RCN) model assumes that clean labels flip randomly with a constant rate (Biggio et al., 2011; Manwani & Sastry, 2013; Natarajan et al., 2013). The class-conditional label noise (CCN) model assumes that the flip rate depends on the latent clean class (Patrini et al., 2017; Xia et al., 2019; Ma et al., 2018). The instance-dependent label noise (IDN) model considers the most general case of label noise, where the flip rate depends on its instance/features (Cheng et al., 2020;
Xia et al., 2020a; Zhu et al., 2020). Obviously, the IDN model is more realistic and applicable. For example, in real-world datasets, an instance whose feature contains less information or is of poor quality may be more prone to be labeled wrongly. The bounded instance dependent label noise (BIDN) (Cheng et al., 2020) is a reasonable extension of IDN, where the flip rates are dependent on instances but upper bounded by a value smaller than 1. However, with only noisy data, it is a non-trivial task to model such realistic noise without any assumption (Xia et al., 2020a). This paper focuses on the challenging BIDN problem setting.
Learning clean distributions. It is significant to reduce the side effect of noisy labels by inferring clean distributions statistically. The label transition matrix plays an important role in such an inference process, which is used to denote the probabilities that clean labels flip into noisy labels. We first review prior efforts under the class-dependent condition (Patrini et al., 2017). By exploiting the class-dependent transition matrix T , the training loss on noisy data can be corrected. The transition matrix T can be estimated in many ways, e.g., by introducing the anchor point assumption (Liu & Tao, 2016), by exploiting clustering (Zhu et al., 2021), by minimizing volume of T (Li et al., 2021), and by using extra clean data (Hendrycks et al., 2018; Shu et al., 2020). To make the estimation more accurately, a slack variable (Xia et al., 2019) or a multiplicative dual T (Yao et al., 2020b) can be introduced to revise the transition matrix. As for the efforts on the instance-dependent transition matrix, existing methods rely on various assumptions, e.g., the noise rate is bounded (Cheng et al., 2020), the noise only depends on the parts of the instance (Xia et al., 2020a), and additional valuable information is available (Berthon et al., 2020). Although the above advanced methods achieve superior performance empirically, the introduction of strong assumptions limit their applications in practice. In this paper, we propose to infer Bayes optimal distribution instead of clean distribution, as Bayes optimal distribution is less uncertain and easy to be inferred under mild conditions.
Other approaches. Other methods exist with more sophisticated training frameworks or pipelines, including but not limited to robust loss functions (Zhang & Sabuncu, 2018; Xu et al., 2019; Liu & Guo, 2020), sample selection (Han et al., 2018b; Wang et al., 2019; Lyu & Tsang, 2020), label correction (Tanaka et al., 2018; Zhang et al., 2021; Zheng et al., 2020), (implicit) regularization (Xia et al., 2021; Zhang et al., 2017b; Liu et al., 2020), and semi-supervised learning (Li et al., 2020a; Nguyen et al., 2020).

heading: 3 PRELIMINARIES
text: We introduce the problem setting, some important definitions, and the formulation of the proposed Bayes label transition matrix in this section.
Problem setting. This paper focuses on a classification task given a training dataset with Instance Dependent Noise (IDN), which is denoted by S̃ = {(xi, ỹi)}ni=1. We consider that training examples {(xi, ỹi)}ni=1 are drawn according to random variables (X, Ỹ ) ∼ D̃, where D̃ is a noisy distribution. The noise rate for class y is defined as ρy(x) = P (Ỹ = y | Y 6= y,x). This paper focuses on a reasonable IDN setting that the noise rates have upper bounds ρmax as in (Cheng et al., 2020), i.e., ∀(x) ∈ X , 0 ≤ ρy(x) ≤ ρmax < 1. Note that the problem in (Cheng et al., 2020) is defined on binary classification task while we extend the problem setting to multi-class classification. Our aim is to learn a robust classifier only from the noisy data, which could assign accurate labels for test data.
Clean distribution. For the observed noisy training examples, all of them have corresponding clean labels, which are unobservable. The clean training examples are denoted by S = {(xi, yi)}ni=1, which are considered to be drawn according to random variables (X,Y ) ∼ D. The term D denotes the underlying clean distribution.
Bayes optimal distribution. Given X , its Bayes optimal label is denoted by Y ∗, Y ∗ | X := argmaxY P (Y | X), (X,Y ) ∼ D. The distribution of (X,Y ∗) is denoted by D∗. Note the Bayes optimal distribution D∗ is different from the clean distribution D when P (Y |X) /∈ {0, 1}. Like clean labels, Bayes optimal labels are unobservable due to the information encoded between features and labels is corrupted by label noise (Zhu et al., 2020). Note that it is a non-trivial task to infer D∗ only with the noisy training dataset S̃. Also, the noisy label ỹ, clean label y, and Bayes optimal label y∗, for the same instance x may disagree with each other (Cheng et al., 2020).
Other definitions. The classifier is defined as f : X → Y , where X and Y denote the instance and label spaces respectively. Let 1[·] be the indicator function. Define the 0-1 risk of f as 1(f(X), Y ) , 1[f(X) 6= Y ]. Define the Bayes optimal classifier f∗ as f∗ , argminf 1(f(X), Y ). Note that there is NP-hardness of minimizing the 0-1 risk, which is neither convex nor smooth (Bartlett et al., 2006). We can use the softmax cross entropy loss as the surrogate loss function to approximately learn the Bayes optimal classifier (Bartlett et al., 2006; Cheng et al., 2020). We aim to learn a classifier f from the noisy distribution D̃ which also approximately minimizes E[1(f(X), Y )].
Bayes label transition matrix. Traditional instance-dependent label transition matrix encodes the probabilities that clean labels flip into noisy labels given input instances. However, due to the reasons that clean labels have more uncertainty, estimating the clean label transition matrix is relatively harder. In this paper, we focus on studying the transition from Bayes optimal labels to noisy labels. We define the Bayes label transition matrix that bridges the Bayes optimal distribution and noisy distribution as follows, T ∗i,j(X) = P (Ỹ = j | Y ∗ = i,X), (1) where T ∗i,j(X) denotes the (i, j)-th element of the matrix T
∗(X), indicating the probability of a Bayes optimal label i flipped to noisy label j for inputX . Given the noisy class posterior probability P (Ỹ | X = x) = [P (Ỹ = 1 | X = x), . . . , P (Ỹ = C | X = x)] (which can be learned from noisy data) and the Bayes label transition matrix T ∗ij(x) = P (Ỹ = j|Y ∗ = i,X = x), the Bayes class posterior probability P (Y∗|X = x) can be inferred, i.e., P (Y∗ | X = x) =( T ∗(X = x)> )−1 P (Ỹ | X = x).

heading: 4 METHOD
text: The feasible solution space of the Bayes label transition matrix is much smaller since Bayes optimal labels are deterministic. Therefore, we propose to estimate a Bayes label transition matrix for each input instance in a parameterized way. We firstly collect a distilled dataset (is defined in Defination 1) with theoretically guaranteed Bayes optimal labels out of the noisy dataset (Section 4.1). Then, we can train a deep neural network (Bayes label transition network) on the collected distilled dataset to learn the transition between Bayes optimal labels and noisy labels (Section 4.2). The learned Bayes label transition network is then fixed and used to train a classification network on the noisy dataset in a F-Correction (Patrini et al., 2017) fashion (Section 4.3).

heading: 4.1 COLLECTING DISTILLED EXAMPLES
text: In this subsection, we show how to construct a distilled dataset consists of distilled examples. We formally introduce the concept of distilled examples first and then present how to collect distilled examples automatically. The collected distilled examples can be used for training the Bayes label transition network. Definition 1 (Distilled examples (Cheng et al., 2020)). An example (x, ỹ, y∗) is defined to be a distilled example if y∗ is identical to the one assigned by the Bayes optimal classifier under the clean data, i.e., y = f∗D(x).
The distilled examples can be collected out of noisy examples automatically with the following guarantee,
Theorem 1 ((Cheng et al., 2020)). Denote by η̃y(x) the noisy class posterior probability PD̃(Ỹ = y|X = x) and ηy(x) = PD(Y = y|X = x) the clean class posterior probability. ∀(x) ∈ X , we have
η̃y(x) > 1 + ρmax
2 =⇒ ηy(x) >
1 2 =⇒ (x, ỹ, Y ∗ = y) is distilled; (2)
where ρmax is the noise rate upper bound. Theorem 1 can be proved in a similar way as did in (Cheng et al., 2020) (Theorem 2 therein). Note the original theorem in (Cheng et al., 2020) was built on binary-classification task, we extend it to the multi-class classification problem, the extension is straightforward.
According to Theorem 1, we can obtain distilled examples by collecting all noisy examples (x, ỹ) whose x satisfies η̃y(x) > 1+ρmax2 and then assigning the label y to it as its Bayes optimal label y∗. After that, we obtain an instance of distilled example (xdistilled, ỹ, ŷ∗), the ỹ indicates the noisy label while the ŷ∗ indicates the inferred Bayes optimal label. The ŷ∗ may disagree with the ỹ. The conditional probability η̃y can be estimated as ˆ̃ηy by several methods, such as the probabilistic classification methods (logistic regression and deep neural networks).

heading: 4.2 ESTIMATING BAYES LABEL TRANSITION MATRICES USING DISTILLED EXAMPLES
text: Bayes Label Transition Network𝑥 !"#$"%%&!
𝜃
#𝑦𝑦∗
∗(𝑥!"#$"%%&!)%
As discussed in Section 4.1, we can collect a set of distilled examples (xdistilled, ỹ, ŷ∗) from noisy data. Now we proceed to train a Bayes label transition network parameterized by θ to estimate the instance-dependent label-noise (IDN) transition matrices, which model the probability of observing a noisy label ỹ given input image x and its Bayes optimal label y∗:
ˆT ∗i,j(x distilled; θ) = P (Ỹ = j|Y ∗ = i,xdistilled; θ),
(3) Specifically, the Bayes label transition network takes xdistilled as input and output an estimated Bayes label transition matrix T̂ ∗(xdistilled; θ) ∈ RC×C , where C is the number of classes. Note T̂ ∗(xdistilled; θ) is an approximation to T ∗(xdistilled) and it might be not exactly the same as the definition of Bayes label transition matrix. We can use the collected Bayes labels ŷ∗ and the estimated Bayes label transition matrix T̂ ∗(xdistilledi ; θ) to infer the noisy labels. The following empirical risk on the inferred noisy labels and the ground-truth noisy labels are minimized to learn the network’s parameter θ:
R̂1(θ) = − 1
n n∑ i=1 ỹi log(ŷ∗i · T̂ ∗(x distilled i ; θ)), (4)
where ỹi and ŷ∗i are ỹi and ŷ ∗ i in the form of one-hot vectors, ỹi ∈ R1×C and ŷ∗i ∈ R1×C , respectively. Note that if we have a distilled example for the i-th class, we can only make use of it to learn the i-th row of the transition matrix. For the other rows, they will not contribute to calculating the loss of the current training example. However, it does not mean that they will be random or not learnable. Their information will be learned by exploiting distilled examples from the non-i-th classes. More specifically, the parameters of the network can be divided into row-specific parameters and commonly shared parameters. By assuming that we have distilled examples for each class, both the row-specific parameters and commonly shared parameters will be optimized.

heading: 4.3 TRAINING CLASSIFICATION NETWORK WITH BAYES LABEL TRANSITION MATRICES
text: Our goal is to train a classification network f(·|w) that predicts Bayes class posterior probability P (Y ∗ = i|x;w) parameterized by w. In the training stage, we cannot observe the Bayes optimal label Y ∗. Instead, we only have access to noisy label Ỹ . The probability of observing a noisy label Ỹ given input image x is:
P (Ỹ = j | x;w, θ) = k∑ i=1 P (Ỹ = j | Y ∗ = i,x; θ)P (Y ∗ = i | x;w), (5)
With the trained Bayes label transition network, we can get ˆT ∗i,j(x; θ) = P (Ỹ = j | Y ∗ = i,x; θ) for each input x. We exploit F-Correction (Patrini et al., 2017), which is a typical classifierconsistent algorithm, to train the classification network. To be specific, fix the learned Bayes label transition network parameter θ, we minimize the empirical risk as follows to optimize the classification network parameter w:
R̂2(w) = − 1
n n∑ i=1 ỹi log(f(xi;w) · T̂ ∗(xi; θ)), (6)
Algorithm 1 Bounded Instance-dependent Label Noise Generation. Require: Clean examples {(xi, yi)}ni=1; Require: Noise rate ρ; Require: Noise rate upper bound ρmax; 1: Sample instance flip rates qi from the truncated normal distributionN (ρ, 0.12, [0, ρmax]);
//mean γ, variance 0.12, range [0,ρmax] 2: Independently sample w1, w2, . . . , wc from the standard normal distributionN (0, 12); 3: for i = 1, 2, . . . , n do 4: p = xi × wyi ; //generate instance-dependent flip rates 5: pyi = −∞; //only consider entries that are different from the true label 6: p = qi × softmax(p); //make the sum of the off-diagonal entries of the yi-th row to be qi 7: pyi = 1− qi; //set the diagonal entry to be 1-qi 8: Randomly choose a label from the label space according to the possibilities p as noisy label ỹi; 9: end for
10: return Noisy samples {(xi, ỹi)}ni=1
Bayes Label Transition Network
𝜃
"𝑦Classification Network
𝑤
𝑥 𝑦∗ 𝑇∗(𝑥)where f(xi;w) ∈ R1×C . The F-Correction
has been proved to be a classifier-consistent algorithm, the minimizer of R̂2(w) under the noisy distribution is the same as the minimizer of the original cross-entropy loss under the Bayes optimal distribution (Patrini et al., 2017), if the transition matrix T̂ ∗ is estimated unbiased. Note the Bayes label transition network is trained on a biased set, i.e., the set of distilled examples. The network will generalize to the non-distilled examples if they share the same pattern with the distilled examples which causes label noise. A recent study (Xia et al., 2020a) empirically verified that the patterns that cause label noise are commonly shared. Our empirical experiments further show that the network T̂ ∗(x; θ) generalizes well to unseen examples and thus helps achieve superior classification performance.

heading: 5 EXPERIMENTS
text: In this section, we first introduce the experiment setup (Section 5.1) including the datasets used (Section 5.1.1) and the compared methods (Section 5.1.2). Next, we analyse the hyper-parameter sensitivity in Section 5.2. Finally, we present and analyze the experimental results on synthetic and real-world noisy datasets to show the effectiveness of the proposed method (Section 5.3). The implementation details and more ablation studies are included in the Appendix.

heading: 5.1 EXPERIMENT SETUP
text: In this section, we introduce the datasets we used and the baseline methods we compared with.

heading: 5.1.1 DATASETS
text: We conduct the experiment on four datasets to verify the effectiveness of our proposed method, where three of them are manually corrupted, i.e., F-MNIST, CIFAR-10, and SVHN, one of them is real-world noisy datasets, i.e., Clothing1M. F-MNIST has 28 × 28 grayscale images of 10 classes including 60,000 training images and 10,000 test images. CIFAR-10 dataset contains 50,000 color images from 10 classes for training and 10,000 color images from 10 classes for testing both with shape of 32× 32× 3. SVHN has 10 classes of images with 73,257 training images and 26,032 test images. We manually corrupt the three datasets, i.e., F-MNIST, CIFAR-10 and SVHN with bounded instance-dependent label noise according to Algorithm 1, which is modified from (Xia et al., 2020a). In noise generation, the noise rate upper bound ρmax in Algorithm 1 is set as 0.6 for all experiments. All experiments on those datasets with synthetic instance-dependent label noise are repeated five times to guarantee reliability. The Clothing1M has 1M images with real-world noisy labels for
training and 10k images with the clean label for testing. 10% of the noisy training examples of all datasets are left out as a noisy validation set for model selection.

heading: 5.1.2 COMPARISON METHODS
text: We compare the proposed method with several state-of-the-art approaches: (1) CE, which trains the classification network with the standard cross-entropy loss on noise datasets. (2) GCE (Zhang & Sabuncu, 2018), which unites the mean absolute error loss and the cross-entropy loss to combat noisy labels. (3) APL (Ma et al., 2020), which combines two mutually reinforcing robust loss functions, we employ its combination of NCE and RCE for comparison. (4) Decoupling (Malach & ShalevShwartz, 2017), which trains two networks on samples whose predictions from two networks are different. (5) MentorNet (Jiang et al., 2018), Co-teaching (Han et al., 2018b), and Co-teaching+ (Yu et al., 2019) mainly handle noisy labels by training networks on instances with small loss values. (6) Joint (Tanaka et al., 2018), which jointly optimizes the network parameters and the sample labels. The hyperparameters α and β are set to 1.2 and 0.8, respectively. (7) DMI (Xu et al., 2019), which proposes a novel information-theoretic loss function for training neural networks robust to label noise. (8) Forward (Patrini et al., 2017), Reweight (Liu & Tao, 2016), and T-Revision (Xia et al., 2019) utilize a class-dependent transition matrix T to correct the loss function. (9) PTD (Xia et al., 2020a), estimates instance-dependent transition matrix by combing part-dependent transition matrices, which is the most related work to our proposed method. We also provide comparison results between our method and DivideMix(Li et al., 2020a), which is a hybrid algorithm that combines multiple powerful techniques, e.g. Gaussian Mixture Model, MixMatch, MixUp, regularization and asymmetric noise penalty in Appendix. As for our method, we simply model the instance-dependent matrix by employing a neural network.

heading: 5.2 HYPER-PARAMETER SENSITIVITY
text: The quality of the distilled dataset relies on the choice of distillation threshold ρ̂max (denoted as ρ̂ in the following paragraph) in Theorem 1. To further explore the effect of ρ̂, we conduct hyperparameter sensitivity studies on CIFAR-10 in this section.
In Figure 2(a), we show the instance-dependent transition matrix approximation error when employing the class-dependent transition matrix, the revised class-dependent transition matrix, and our proposed instance-dependent transition matrix estimation method. The error is measured by `1 norm between the ground-truth transition matrix and the estimated transition matrix. For each instance, we only analyze the approximation error of a specific row because the noisy label is generated by one row of the instance-dependent transition matrix. The ”Class-dependent” represents the classdependent transition matrix learning methods (Patrini et al., 2017), the ‘T-Revision’ indicates the class-dependent transition matrix is revised by a learnable slack variable (Xia et al., 2019). Our pro-
posed method estimates an instance-dependent transition matrix for each input. It can be observed that our proposed method can achieve a much lower approximation error. Figure 2(b) shows the classification performance of our proposed method when choosing various distillation threshold ρ̂s. When ρ̂ is not too large or too small, our method is not sensitive to the choice of ρ̂. More experimental results on the quality of distilled dataset when applying different values of ρ̂ are included in the appendix B. We manually set ρ̂ = 0.3, a decent trade-off between distillation accuracy and the number of distilled examples, in all experiments later to avoid laborious hyper-parameter tuning and access to the true noise rate.

heading: 5.3 COMPARISON WITH THE STATE-OF-THE-ARTS
text: Results on synthetic noisy datasets. Table 1,2 and 3 report the classification accuracy on the datasets of F-MNIST,CIFAR-10, and SVHN, respectively.
For F-MNIST, our method surpasses all the baseline methods by a large margin. Equipping the transition matrix revision (-V) (Xia et al., 2019) can further boost the performance of our method.
For SVHN and CIFAR-10, the superiority of our method is gradually revealed along with the noise rate increase, which shows that our method can handle the extremely hard situation much better. Specifically, the classification accuracy of our method is 5.83% higher than PTD (the best statistically consistent baseline) on CIFAR-10 in the IDN-10% case, and the performance gap is enlarged to 7.01% in the IDN-40% case. On the SVHN, the classification accuracy of our method is 2.60% higher than PTD in the IDN-10% case, 5.05% higher than PTD in the IDN-30% case, and 7.14% higher than PTD in the most challenging IDN-50% case.

heading: 5.3.1 RESULTS ON REAL-WORLD DATASETS
text: The noise model of real-world datasets is more likely to be instance-dependent. Our proposed method also performs favorably on the challenging Clothing1M dataset (Table 4), which proves that our method is more flexible to handle such real-world noise problem .

heading: 68.88 54.53 56.79 60.15 65.15 70.88 70.12

heading: 6 CONCLUSION
text: In this paper, we focus on training the robust classifier with the challenging instance-dependent label noise. To address the issues of existing clean label transition matrix, we propose to directly build the transition between Bayes optimal labels and noisy labels. By reducing the feasible solution of the transition matrix estimation, we prove that the instance-dependent label transition matrix that relates Bayes optimal labels and noisy labels can be directly learned using deep neural networks. The main limitation of our method comes from that the distilled examples are collected out of noisy data leading to unavoidable data distribution bias to the transition matrix estimation. Experimental results demonstrate that the proposed method is more superior in dealing with instance-dependent label noise, especially for the case of high-level noise rates.

heading: 7 ETHICS STATEMENT
text: This paper doesn’t raise any ethics concerns. This study doesn’t involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, pontential conflicts of interest and sponsorship, discrimination/bias/fairness concerns, privacy and security issues, legal compliance, and research integrity issues.

heading: B THE QUALITY OF DATASET DISTILLATION
text: The distillation threshold ρ̂ controls how many examples can be collected out of noisy dataset and the quality of the distilled dataset. We analyse the effect of ρ̂ on the CIFAR-10 dataset in Table. 5. The distillation accuracy is computed by counting how many inferred Bayes optimal labels are consistent with their corresponding clean labels among all distilled examples.

heading: C ABLATION ON BAYES LABEL TRANSITION MATRIX
text: To verify the effectiveness of the estimated Bayes label transition matrix, we compare our method with some ablated variants, e.g. directly train a classifier on the distilled dataset and relabel the noisy dataset using the classifier trained on distilled dataset.
CIFAR-10 IDN-10% CIFAR-10 IDN-30% Clothing1M
Training classifier on distilled dataset 74.56 67.42 62.37 Relabeling noisy dataset 76.68 70.73 64.98 Ours 82.16 78.82 73.39

heading: D COMPARISION WITH DIVIDEMIX
text: DivideMix has a much more complicated pipeline than us and is not a statistically consistent algorithm. We compare our method with DivideMix to further show the effectiveness and flexibility of our proposed method. Compared with DivideMix, our method exhibit competitive performance when noise rate is low and surpass DivideMix by a large margin on the worst noise cases (3.22%
performance improvement on CIFAR-10 and 4.38% on SVHN, both under IDN-50% ), with a much simpler and flexible algorithm design.

Review summaries:
The paper proposes to estimate an Instance-Dependent Noise (IDN) label transition matrix. Instead of modelling the clean label transition as typically done in previous literature, the authors propose to estimate the Bayes label transition using a DNN, motivate by several advantages including theoretically guaranteed Bayes label collection and smaller feasible solution space, hence empirically easier to model. Controlled experiments show consistent improvement over other SOTA methods in noisy label transition.

The transition matrix plays a vital role in modeling label noise. Current methods focus on modeling the transition from clean labels to noisy labels. While this paper alternatively models the transition from Bayes optimal labels to noisy labels. Since we usually use the Bayes optimal labels for prediction. This transformation will not affect the practical use but makes the estimation of the matrix much easier.  Specifically, this paper designs a DNN to estimate the transition matrix. During training, the DNN can be optimized with the classifier simultaneously in an end-to-end manner.  Extensive experiments are conducted to support the proposed method.