Title: CONFIDENCE SCORES MAKE INSTANCE-DEPENDENT LABEL-NOISE LEARNING POSSIBLE

Abstract: Learning with noisy labels has drawn a lot of attention. In this area, most of recent works only consider class-conditional noise, where the label noise is independent of its input features. This noise model may not be faithful to many real-world applications. Instead, few pioneer works have studied instance-dependent noise, but these methods are limited to strong assumptions on noise models. To alleviate this issue, we introduce confidence-scored instance-dependent noise (CSIDN), where each instance-label pair is associated with a confidence score. The confidence scores are sufficient to estimate the noise functions of each instance with minimal assumptions. Moreover, such scores can be easily and cheaply derived during the construction of the dataset through crowdsourcing or automatic annotation. To handle CSIDN, we design a benchmark algorithm termed instance-level forward correction. Empirical results on synthetic and real-world datasets demonstrate the utility of our proposed method.

1. INTRODUCTION:

The recent success of deep neural networks has increased the need for high-quality labeled data. However, such a labelling process can be time-consuming and costly. A compromise is to resort to weakly-supervised annotations, using crowdsourcing platforms or trained classifiers that annotate the data automatically. These weakly-supervised annotations tend to be low-quality and noisy, which negatively affects the accuracy of high-capacity models due to memorization effects (Zhang et al., 2017) . Thus, learning with noisy labels has often drawn a lot of attention.

Early works on noisy labels studied random classification noise (RCN) for binary classification (Angluin & Laird, 1988; Kearns, 1993) . In the RCN model, each instance has its label flipped with a fixed noise rate ρ ∈ [0, 1 2 ). A natural extension of RCN is class-conditional noise (CCN) for multiclass classification (Stempfel & Ralaivola, 2009; Natarajan et al., 2013; Scott et al., 2013; Menon et al., 2015; van Rooyen & Williamson, 2015; Patrini et al., 2016) (Appendix A). In the CCN model, each instance from class i has a fixed probability ρ i,j of being assigned to class j. Thus, it is possible to encode some similarity information between classes. For example, we can expect that the image of a "dog" is more likely to be erroneously labelled as "cat" than "boat".

To handle the CCN model, a common method is the loss correction, which aims to correct the prediction or the loss of the classifier using an estimated noise transition matrix (Patrini et al., 2017; Sukhbaatar et al., 2015; Goldberger & Ben-Reuven, 2017; Ma et al., 2018) . Another common approach is the label correction, which aims to improve the label quality during training. For example, Reed et al. (2015) introduced a bootstrapping scheme. Similarly, Tanaka et al. (2018) proposed to update the weights of a classifier iteratively using noisy labels, and use the updated classifier to yield more high-quality pseudo-labels for the training set. Although these methods have theoretical guarantees, they are unable to cope with real-world noise, e.g., instance-dependent noise (IDN).

The IDN model considers a more general noise (Manwani & Sastry, 2013; Ghosh et al., 2014; Menon et al., 2016; Cheng et al., 2017; Menon et al., 2018) , where the probability that an instance is mislabeled depends on both its class and features. Intuitively, this noise is quite realistic, as poorquality or ambiguous instances are more likely to be mislabeled in real-world datasets. However, it is much more complex to formulate the IDN model, since the probability of a mislabeled instance is a function of not only the label space but also the input space that can be very high dimensional. As a result, several pioneer works have considered stronger assumptions on noise functions. However, stronger assumptions tend to restrict the utility of these works (Table 1 ). For instance, the boundary-consistent noise model considers stronger noise for samples closer to the decision boundary of the Bayesian optimal classifier (Du & Cai, 2015; Menon et al., 2018) . However, such a model is restricted to binary and cannot estimate noise functions. Cheng et al. (2017) recently studied a particular case of the IDN model, where noise functions are upper-bounded. Nonetheless, their method is limited to binary classification and has only been tested on small datasets.

Instead of simplifying assumptions on noise functions, we propose to tackle the IDN model from the source, by considering confidence scores to be available for the label of each instance. We term this new setting confidence-scored instance-dependent noise (CSIDN, Figure 1c ). The confidence scores denote how likely an instance is to be correctly labeled. Assuming that (i) confidence scores are available for each instance, (ii) transitions probabilities to other classes are independent of the instance conditionally on the assigned label being erroneous and (iii) a set of anchor points is available, we derive an instance-level forward correction algorithm which can fully estimate the transition probability for each instance, and subsequently train a robust classifier with a loss-correction method similarly to Patrini et al. (2017) .

It is noted that confidence scores can be easily and cheaply derived during the construction of the dataset. For example, in crowdsourcing platforms, simply counting how many annotators agree on a given instance can give a notion of how confident a label is. Besides, many real-world datasets are automatically annotated using a trained classifier, such as web-scraped datasets (Tong Xiao et al., 2015) and physiological features inferred from medical records (Agarwal et al., 2016) . In these cases, the class-probabilities of the labels assigned by the classifier can be seen as confidence scores, provided that the classifier is well calibrated (Guo et al., 2017) .

To sum up, we first formulate instance-dependent noise in Section 2.1, and expose its robustness challenge in Section 2.2. Then, we explain our motivation to use confidence scores, and propose the confidence-scored instance-dependent noise (CSIDN) model in Section 2.3. Lastly, to handle this new noise model, we present the first practical algorithm termed instance-level forward correction in Section 3, and validate the proposed algorithm through extensive experiments in Section 4.

2. TACKLING INSTANCE-DEPENDENT NOISE FROM THE SOURCE:

In this section, we present the IDN model along with the limitations of existing approaches, and introduce the CSIDN model as a tractable instance-dependent noise model.

2.1. NOISE MODELS: FROM CLASS-CONDITIONAL TO INSTANCE-DEPENDENT NOISE:

We formulate the problem of learning with noisy labels in this section. Let D be the distribution of a pair of random variables (X, Y ) ∈ X × Y, where X ∈ R d , Y = {1, 2, . . . , K} and K is the number of classes. In the classification task with noisy labels, we hope to train a classifier while having only access to samples from a noisy distribution D of random variables (X, Ȳ ) ∈ X × Y.

Given a point x sampled from X, Ȳ is derived from the random variable Y via a noise transition matrix T (x) = (T i,j (x)) Each noise function T i,j : X → [0, 1] is defined as T i,j (x) = P ( Ȳ = j|Y = i, X = x). In the class-conditional noise (CNN) model (Figure 1a ), the transition matrix does not depend on the instance x and the noise is entirely characterized by the K2 constants T i,j . However, in the instancedependent noise (IDN) model (Figure 1b ), the transition matrix depends on the actual instance. This tremendously complicates the problem, as the noise is now characterized by K 2 functions over the latent space X , which can be very high dimensional (e.g., d ∼ 10 4 -10 6 for an object recognition dataset).

EQUATION

2.2. CHALLENGES FROM INSTANCE-DEPENDENT NOISE:

Limitation of existing CCN methods. Due to the complexity of the IDN model, most recent works in learning with noisy labels have focused on the CCN model (Figure 1a ), and the CCN model can be seen as a simplified IDN model (Figure 1b ) free of feature information.

In addition to loss correction and label correction mentioned before, another method for the CCN model is sample selection, which aims to find reliable samples during training, such as the smallloss approaches (Jiang et al., 2018; Han et al., 2018) . Inspired by the memorization in deep learning (Arpit et al., 2017) , those methods first run a standard classifier on a noisy dataset, then select the small-loss samples for reliable training.

However, all approaches cannot handle the IDN model directly. Specifically, loss correction considers the noise model to be characterized by a fixed transition matrix, which does not include any instance-level information. Meanwhile, label correction is vulnerable to the IDN model, since the classifier will be much weaker on noisy regions and labels corrected by the current prediction would likely be erroneous. Similarly, sample selection is easily affected by the IDN model.

For example, in the small-loss approaches, instance-dependent noise functions can leave partial regions of the input space clean and other regions very noisy (e.g., in an object recognition dataset, poor-quality pictures will tend to receive more noisy labels than high-quality ones). Since clean regions will tend to receive smaller losses than noisy regions, the small-loss approaches, which only trains on points with the smallest-losses, will focus on clean regions and neglect harder noisy regions. Then, since the distribution of clean regions will subsequently be different from the global distribution, this will introduce a covariate-shift (Shimodaira, 2000) , which greatly degrades performances. Moreover, it is hard to use importance reweighting (Sugiyama et al., 2007) for alleviate the issue, since importance reweighting would require estimating the clean posterior probability that is intractable for the IDN model.

To validate this fact, we generate a 3-class distribution of concentric circles (cf. Figure 2a ), with then train a network on the top R(T ) small-loss instances at each epoch T based on the losses of the previous epoch, with R(T ) decreasing in T as described in Han et al. (2018) . Figure 2c shows the density of the top 50% small-loss instances selected after 10 epochs: since noisy regions are associated to higher losses, the network eventually tends to select instances from the clean region and neglect the noisy region. This leads to covariate-shift, which is associated with decreased performances (Shimodaira, 2000) .

∀(x, y) ∈ R 2 × {1, 2, 3}, P (ȳ = y|x) = 1

Limitation of pioneer IDN methods. The main challenge of the IDN model is the wide range of possible noise functions included in its formulation. Since each T i,j (•) is a function of the highdimensional input space X , it is challenging for a model to be flexible enough to fit any real-world noise function while being trainable on corrupted datasets, let alone derive theoretical results. Instead, various recent works have considered stronger assumptions on noise functions.

For instance, boundary-consistent noise (BCN), first introduced by (Du & Cai, 2015) and generalized in Menon et al. (2018) , considers stronger noise for samples closer to the decision boundary of the Bayesian optimal classifier. This is a reasonable model for noise from human annotators, since "harder" instances (i.e., instances closer to the decision boundary) are more likely to be corrupted. Moreover, it is simple enough to derive some theoretical guarantees, as done in Menon et al. (2018) . Additionally, an extension of the BCN model was studied in Bootkrajang & Chaijaruwanich (2018) , where the noise function is a Gaussian mixture of the distance to the Bayesian optimal boundary. However, the BCN model and its extension are restricted to binary classification, and their geometry-based assumption becomes difficult to fathom for high-dimensional input spaces.

Furthermore, Cheng et al. (2017) recently studied a particular case of the IDN model, where the probabilities that the true labels of samples flip into corrupted ones have upper bounds. They proposed a method based on distilled samples, where noisy labels agree with the optimal Bayesian classifier on the clean distribution. However, their method is limited to binary classification and has only been tested on small UCI datasets. Table 1 summarizes the characteristics of those approaches.

2.3. CONFIDENCE-SCORED INSTANCE-DEPENDENT NOISE:

Instead of simplifying assumptions on noise functions, we propose to tackle the IDN model from the source. Namely, we consider that, for each instance, we have access to a measure of confidence in the assigned label. As most of noisy datasets arise from crowdsourcing or automatic annotation, such confidence scores can be easily derived during the dataset construction, often with no extra cost. This allows for a good approximation of noise functions with weaker assumptions.

Before introducing our proposed noise model confidence-scored instance-dependent noise (CSIDN, Figure 1c ), we first define what are the confidence scores, and explain why the confidence scores are available in real-world applications.

Definition of confidence scores. For any data point (x, ȳ) sampled from the joint distribution (X, Ȳ ), we define the confidence score r x as follows.

EQUATION

Namely, the probability that the assigned label is correct.

Availability of confidence scores. Our rationale is that in tasks involving instance-dependent noise, the confidence information can be easily derived with no extra cost.

Firstly, in crowdsourcing platforms, when multiple workers manually annotate datasets, an aggregation step is often took to aggregate answers of different workers for each instance (e.g., majority vote). An estimation of r x could then be derived by taking the ratio of votes for the assigned label on the total number of workers. Moreover, since this estimation would of course be less reliable as the number of workers decreases, an alternative could be to directly ask workers for self-reported confidence scores of their responses (Cosmides & Tooby, 1996; Oyama et al., 2013) .

Secondly, the confidence information can also be available in automatic annotation via a softmax output layer of deep neural networks. This layer outputs an estimation of the probability that each class is the true label: when a model outputs a given class with probability 0.9, we expect the predicted class to be true 9 times out of 10 on average. A model that estimates the accurate probability is well-calibrated. Therefore, in the case of labels generated by a well-calibrated model, the softmax probability of the assigned label can be directly interpreted as a confidence measure that the label is correct. Even though Guo et al. (2017) showed that recent deep neural networks are not usually well-calibrated (whereas early shallower networks were, as shown in Niculescu-Mizil & Caruana (2005) ), model calibration can be achieved in a relatively straightforward way at the validation time, e.g., using temperature scaling (Section 4.2 in Guo et al. ( 2017)).

CSIDN: a tractable instance-dependent noise model. Recall the intrinsic difficulty of the IDN model: to fully characterize this noise, one would need to estimate K 2 functions T i,j (•) over the input space X . This is of course intractable with a finite noisy dataset. This is why pioneer solutions to the IDN model have been so far limited by very strong assumptions.

However, considering additional confidence scores, one can wonder whether such information would make the IDN model tractable with less restrictive assumptions. Hence, we introduce a new and tractable instance-dependent noise model: confidence-scored instance-dependent noise (CSIDN, Figure 1c ). In this noise model, the training data takes the form S := {(x i , ȳi , r xi ), i = 1, . . . , N }, where {(x i , ȳi )} i i.i.d.

∼ D and r xi = P (Y = ȳi | Ȳ = ȳi , X = x i ) is the previously defined confidence scores in the assigned label of a given instance (Eq. ( 2)). The confidence information r x is decisive for robustness to instance-dependent noise, as it provides a proxy for the noise functions T i,j of the training data that are often intractable otherwise.

3. BENCHMARK SOLUTION FOR HANDLING THE CSIDN MODEL:

To tackle the CSIDN model, we propose a benchmark solution. Inspired by forward correction (Patrini et al., 2017) for the CCN model, we want to correct each prediction P (ȳ|x) with the noise transition matrix T (x). However, the transition matrix for the CSIDN model is instance-dependent, and has to be estimated for each instance x. We term our solution instance-level forward correction.

3.1. ESTIMATING INSTANCE-DEPENDENT TRANSITION MATRIX:

Using the confidence scores, we will first estimate the diagonal terms (T i,i (•)) K i=1 of the transition matrix, and then estimate the non-diagonal ones.

Diagonal terms. The diagonal terms of the transition matrix correspond to the probabilities that assigned labels are equal to true labels. However, the confidence scores available are only relevant to the class corresponding to the observed label. Therefore, we need to proceed differently whether the confidence scores are available for the considered class or not.

First, note that for each sample (x, ȳ, r x ) ∈ S i := {(x, ȳ, r x ) ∈ S|ȳ = i}, T i,i (x) can be derived for the most part from the confidence scores alone:

EQUATION

where β i (x) = P ( Ȳ =i|X=x) P (Y =i|X=x) . In practice, we use an iterative procedure to estimate in turn β i (•) and T i,i (•) (see Section 3.2 for details). Then, for the rest of samples (x, ȳ, r x ) ∈ S\S i , r x does not give any direct information on T i,i (•). Hence, we simply set each function T i,i (•) as its empirical mean µ i estimated using samples from S i at the current epoch:

EQUATION

where |S| denotes the cardinality of S.

Non-diagonal terms. For non-diagonal terms, we have:

EQUATION

where α i,j (x) = P ( Ȳ = j| Ȳ = i, Y = i, X = x).

In Eq. ( 4), α i,j (x) refers to the probability that an instance x with true label i has an observed label j, once we know that the observed label is different from the true one. Then, a reasonable assumption is that ∀i = j, ∀x ∈ X , α i,j (x) = α i,j : conditionally on the observed label being erroneous, the class transitions are not influenced by the instance x. In other words, the dependence in x of the noise function only impacts the "magnitude" of the noise and not the class transitions.

To illustrate this assumption, consider a crowdsourcing task of object recognition with adjacent classes which annotators can only differentiate with details that can be more or less visible depending on the instance. For example, objects from a given class may have distinctive traits, but those can be more or less visible in the pictures. When those traits are present, the annotators can confidently predict the right class. Otherwise, they will make errors towards adjacent classes. In this case, the probability that the assigned label is wrong highly depends the instance (with distinctive traits being visible or not). Nonetheless, conditionally on the instance being corrupted, i.e., because those traits were not visible enough on the image, the transition probabilities to the adjacent classes are not influenced by the instance itself.

With the previous assumption, we obtain ∀i = j, ∀x ∈ X , T i,j (x) = α i,j (1 -T i,i (x)) with α i,j ∈ [0, 1]. This allows us to estimate the K(K -1) constants (α i,j ) i =j once, and derive the non-diagonal noise functions of T (x) directly from our estimates of the diagonal noise functions (Eq. ( 5)).

3.2. OVERALL ALGORITHM: INSTANCE-LEVEL FORWARD CORRECTION:

Estimating T i,i and β i . To train a classifier h with the instance-level forward correction method, we need to estimate both T i,i (x) and β i (x) = P ( Ȳ =i|X=x) P (Y =i|X=x) from Eq. ( 3), for all x ∈ S i . Firstly, the noisy posterior P ( Ȳ = i|X = x) can be easily estimated by training a naive classifier on the noisy dataset. Secondly, the true posterior P (Y = i|X = x) can be estimated using the output of the classifier h(x) = P (Y = i|X = x) at the previous epoch.

Therefore, we iteratively update β and T with the following steps: 1) ∀x ∈ X , initialize βi (x) = 1 and train a naive classifier h noisy on the noisy data D to obtain h noisy (x) = P ( Ȳ |X = x). 2) ∀i ∈ [1, K], for each sample (x, ȳ, r x ) ∈ S i , compute Ti,i (x) = r x βi (x) and train classifier h for one epoch. 3) ∀i ∈ [1, K], for each sample (x, ȳ, r x ) ∈ S i , update βi (x) = hnoisy(x)i h(x)i . Then, we repeat steps 2) and 3) through training. In this way, for every epoch, each function T i,i (•) is estimated for the samples from S i . Lastly, for the rest of samples with noisy label j = i, T i,i (•) is estimated at each epoch using Eq. ( 4):

EQUATION

)

Computing α i,j . The computation of α i,j boils down to approximating non-diagonal terms of the transition matrix in the CCN model. As ∀i = j, ∀x ∈ X , T i,j (x) = α i,j (1 -T i,i (x)), we have:

E x [T i,j (x)] = α i,j (1 -E x [T i,i (x)]) ⇔ α i,j = E x [T i,j (x)] 1 -E x [T i,i (x)]

.

A simple and reliable way is to use anchor points, i.e., points for which we can know the true class almost surely. These points may be directly available when some training data has been curated, or they can be identified either theoretically as in Liu & Tao (2015) or heuristically as in Patrini et al. (2017) . Having S * i := {(x, ȳ, r x ) ∈ S|P (Y = i|X = x) ≈ 1} a set of class i anchor points, we simply need compute:

∀(x, ȳ, r x ) ∈ S * i , ∀j = i, T i,i (x) = r x P ( Ȳ = i|X = x) and T i,j (x) = P ( Ȳ = j|X = x).

Two noisy posteriors can be estimated using the same classifier h noisy trained on the noisy distribution h noisy (x) = P ( Ȳ |X = x) aforementioned. Thus, α i,j can be estimated as follows:

EQUATION

Summary of the training procedure. Given samples S and K sets of anchor points (S * i ) K i=1 , we want to train a classifier h(•) equipped with a loss l. For any loss l : y, ŷ → l(y, ŷ), we define the T -corrected loss as l T : y, ŷ → l(y, T ŷ). The overall procedure is in Algorithm 1 (Appendix C).

4. EXPERIMENTS:

We compare our instance-level forward correction (ILFC) method with four representative baselines: forward correction (FC) (Patrini et al., 2017) , mean absolute error (MAE) (Ghosh et al., 2017) , L qnorm (LQ) (Zhang & Sabuncu, 2018) and co-teaching (CT) (Han et al., 2018) . Details are shown in Appendix D. Note that the pioneer IDN methods cannot work for multi-class cases.

4.1. SYNTHETIC DATASET:

Generation process. We generate a synthetic dataset (Appendix E) consisting in three classes of concentric circles (Figure 6a ). We then apply the following instance-dependent noise to each label:

P ( Ȳ = Y |X = x) = ρ w•x

w x + 1 /2 with w = (0, 1) and ρ controlling the mean noise rate. If corrupted, each label is flipped to another class uniformly.

Empirical results. Figure 3 shows the test accuracy of different methods on the synthetic dataset. Each experiment is repeated 5 times and we plot the confidence intervals of each curve. On low-level noise, all methods show good performances (Figure 3a ). On mild-level noise, both Co-teaching and ILFC show good performances and outperform other baselines (Figure 3b ). On high-level noise, the performance of all the baselines collapse, whereas ILFC constantly maintains good performances (Figures 3c and 3d ). More experiments are shown in Appendix B and F. 

4.2. REAL-WORLD DATASET:

Generation process. In order to corrupt labels from clean datasets such as SVHN and CIFAR10, we adopt the following procedure: (1) train a classifier h : x → σ(g(x)) on a small subset of the clean dataset; (2) using a small validation set, calibrate the classifier by selecting the temperature t that maximizes the expected calibration error as in Guo et al. (2017) ; (3) for each instance x, set: ȳ = argmax i h t (x) i and r x = max i h t (x) i . With this process, we attempt to emulate the construction of a real-world dataset (Appendix G).

Empirical results. Figures 4a and 4b show the test accuracy on SVHN with 25% and 45% instance-dependent noise, respectively. We can clearly observe that, on both low-level and highlevel noise, ILFC shows good performances with a fast convergence rate, and outperforms other baselines. Figures 4c and 4d show the test accuracy on CIFAR10 with 25% and 45% instancedependent noise, respectively. On low-level noise, all methods show good performances. However, on high-level noise, ILFC shows a fast convergence rate and outperforms other baselines.

5. CONCLUSION:

In this paper, we give an overview of label-noise learning from class-conditional noise (easier) to instance-dependent noise (harder). We explain why existing approaches cannot handle instancedependent noise well, and try to address this challenge via confidence scores. Thus, we formally propose the confidence-scored instance-dependent noise (CSIDN) model. To tackle the CSIDN model, we design a practical algorithm termed instance-level forward correction (ILFC). Our ILFC method robustly outperforms existing methods, especially in the case of high-level noise. In future works, we would like to extend label correction and sample selection approaches with the confidence scores from the CSIDN model.

A RELATED WORKS:

Besides the works aforementioned, we survey other approaches to learning with noisy labels.

Robust losses. Various approaches propose to use a provably robust loss function in the learning process. In the case of class-dependent label noise, Natarajan et al. (2013) constructed an unbiased estimator of any loss function under the noisy distribution. Masnadi-shirazi & Vasconcelos (2009) introduced a robust non-convex loss. Recently, works on symmetric losses showed that such loss offer theoretical robustness results to various types of noise (Ghosh et al., 2017; Charoenphakdee et al., 2019) . Motivated by the robustness to noise of the mean absolute error loss (MAE) shown in Ghosh et al. (2017) , Zhang & Sabuncu (2018) introduced generalized cross entropy loss that allows for a trade-off between the efficient learning properties of the CCE loss and the noise-robustness of MAE. Shen & Sanghavi (2019) introduced a trimmed loss with an iterative minimization process that allows for theoretical guarantees in the simpler setting of generalized linear models.

Annotator-level modelling. Another recent line of related works attempts to model labels and worker's quality directly during the crowdsourcing annotation process, in order to produce more accurate labels efficiently. Branson et al. (2017) modeled the annotators' skill and instances difficulty while incrementally training a computer vision model during the annotation process, effectively reducing the time burden of the annotation process as well as the error rate in the assigned labels. Guan et al. ( 2018) modeled each annotator individually in order to better aggregate labels based on each worker's skill and area of expertise. Khetan et al. (2018) introduced a method that allows to learn each workers' skill even when each example is only annotated once, by jointly modelling the assigned labels and the workers during the annotation process.

Learning with multiple noisy labels. A closely related setting is learning from multiple noisy labels, where the aim is to predict an unknown ground-truth label from (X, (Y j ) j ), each Y j referring to a noisy annotation. This setting can arise for example from crowdsourcing tasks; Snow et al. (2008) showed that using multiple non-expert annotators to train a classifier can be as effective as using gold standard annotations from experts. In Raykar et al. (2009) , the authors derive a Bayesian approach to jointly learn the expertise of each annotator, the actual true label and the classifier. Yan et al. (2010) extends this Bayesian approach by considering that each annotator's expertise varies across the input space. This setting differs from ours as it takes place before the aggregation of multiple annotations, which, for CSIDN, is only a way among others to obtain a confidence score for each noisy label.

Explicit/implicit regularizers. Recently, several other regularization techniques have shown good robustness in weakly-supervised settings. Temporal Ensembling (TE) (Laine & Aila, 2017 ) method labels some additional unlabeled instances using a consensus of predictions from models from previous epochs and with different regularizations and input augmentation conditions. Mean-teacher (MT) (Tarvainen & Valpola, 2017) instead uses predictions from a model obtained by averaging the weights of a set of models similar to TE, as using the prediction from a unique model is more efficient when a large amount of unlabeled data is available. Virtual Adversarial Training (Miyato et al., 2018) regularizes the network using a measure of local smoothness of the conditional label distribution given the input, defined as the robustness of the prediction to local adversarial perturbations in the input space. Introduced in Zhang et al. (2018) , mixup trains a neural network on convex combinations of instance pairs and their respective labels, and has been shown to reduce the memorization of corrupted labels.

B SENSITIVITY ANALYSIS:

In practice, the confidence scores obtained may not be accurate. Therefore, we run a sensitivity analysis to assess the robustness of ILFC: similarly to Ishida et al. (2018) , we add a zero-mean Gaussian noise with standard deviation σ ∈ {0.0, 0.3, 0.6} to each confidence score and clip the values between 0 and 1. Figure 5 shows the resulting performances on the synthetic dataset. ILFC shows good robustness to inaccurate confidence scores even with high standard deviation on a highly noisy dataset. 

C ALGORITHM:

We present Algorithm 1 here, which can be referred to Section 3.2 in details.

D BASELINES:

Forward correction. Introduced in Patrini et al. (2017) , forward correction estimates a fixed transition matrix T before training, and trains a classifier with the corrected loss l T : (y, ŷ) → l(y, T ŷ).

Mean absolute error loss. Due to its symmetric property, the Mean Absolute Error (MAE) has been theoretically justified to be robust to label noise under assumptions (Ghosh et al., 2017) . However, this loss is more difficult to train, especially on complex datasets.

L q norm. Introduced in Zhang & Sabuncu (2018) , L q norm attempts to bring the best of both worlds between the CCE and the MAE loss: the CCE is easy to train, while the MAE is robust to label noise. The authors therefore define this loss using the negative box-cox transformation:

L q (h(x), e j ) = (1 -h j (x) q ) q , so that the L q tends to the CCE q → 0 and to the MAE when q = 1. In the following experiments, we set q = 0.7, suggested by authors.

Co-teaching (Han et al., 2018) . Co-teaching algorithm is a small-loss approach where two classifiers are trained in parallel. At each epoch, each classifier selects the instances with the smallest modelling will degenerate around the most noisy region of the input space. On the other hand, our model successfully accounts for the high noise in this region and is able to keep consistent predictions.

G EXAMPLES OF REAL-WORLD DATASETS For example, the method would be similar to constructing a dataset with images scraped from the web, and automatically labelling them from neighbouring text fields using a classifier such as a recurrent neural network. Then, a small subset of curated images could be used at the beginning of the process to calibrate the classifier, in order to make the predictions of the softmax output faithful to the confidence in each label. This way, we could construct a very large dataset for a very low-cost that, while involving some instance-dependent noise, would be equipped with confidence information and therefore could be tackled with our proposed algorithm. In the presence of highly noisy regions, a classifier that does not include any instance-level information will degenerate in those regions, while the ILFC approach stays consistent with the clean distribution.

:

w•x w x + 1 with w = (0, 1) (cf. Figure2b). We

Figures and Tables:

Figure 1: Illustration of different noise models. Each color represents an observed class ȳ: circles indicate ȳ = y, while crosses indicate ȳ = y. The size of each point represents the confidence scores in the label ȳ: the bigger the point is, the more confident it is. In the CCN model, the noise function only depends on the label of each instance. In the IDN and CSIDN models, the noise function depends on the observed instance x. To illustrate the IDN model, we show a special case called boundary-consistent noise, i.e., points that lie close to the decision boundary are more likely to be mislabelled. Note the CSIDN model varies from the IDN model via confidence scores (Section 2.3).

Figure 2: The limitation of the small-loss approaches in the IDN model. (a) Clean distribution. (b) instance-dependent noise in the direction w = (0, 1) with an average corruption rate of 40%: points towards the upper region are more likely to be corrupted than points towards the bottom region. (c) Density map of the instances selected by a small-loss approach at epoch 10. The sample selection gets biased towards clean regions. Since the clean and noisy regions have different distributions, selecting most instances from clean regions creates a covariate-shift between the training and test distributions, which can greatly degrades performances.

Figure 3: The test accuracy on synthetic datasets with different levels of IDN noise.

Figure 4: The test accuracy on real-world datasets with different levels of IDN noise.

Figure 5: Sensitivity analysis on the synthetic dataset: a zero-mean Gaussian noise of standard deviation σ ∈ {0.0, 0.3, 0.6} is added to each confidence score before running ILFC with the noisy confidence scores.

Figure7: Decision boundaries of the learned classifier for the ILFC model (left column) and the L q norm model (right column). In the presence of highly noisy regions, a classifier that does not include any instance-level information will degenerate in those regions, while the ILFC approach stays consistent with the clean distribution.

Comparisons between baselines and our work for handling the IDN model. Rate identifiability denotes whether the transition matrix is identifiable.

Table content: <table><tr><td>Approaches</td><td>Multi-class Rate-identifiability Unbounded-noise</td></tr><tr><td>Du &amp; Cai (2015)</td><td/></tr><tr><td>Menon et al. (2018)</td><td/></tr><tr><td>Bootkrajang &amp; Chaijaruwanich (2018)</td><td/></tr><tr><td>Cheng et al. (2017)</td><td/></tr><tr><td>Our work</td><td/></tr></table>

Peer review (iclr2020):
--- Overall --- I think the problem of incorporating information about labeling uncertainty (when such information is available) is an interesting and important problem; however, I think this paper contains a crucial misunderstanding of the definition of calibration (as defined in Guo et al. 2017), is missing some important parts of the literature, and does not provide adequate convergence guarantees for the proposed method. --- Major comments --- 1. In section 3.1, the authors substitute the confidence score r_x for the conditional probability P(Y=y|_x0008_ar{Y}=y,X=x); however, even if the confidence scores are perfectly calibrated, these quantities are not necessarily equal (or even particularly close). As defined in Guo et al. (2017), a confidence score is well calibrated if the probability that a prediction is correct given the confidence score is equal to the confidence score. Using their notation: P(Y=hat{Y}|hat{P}=p) = p. Importantly, this is not conditioned on hat{Y} or X. One way to see that these two quantities are not equal is to observe that there are many possible confidence score functions that satisfy the above definition whereas the conditional probability defined above is a unique function. In particular, if the confidence score function is a constant equal to the accuracy of the model, then the confidence score is well calibrated, but clearly not equal to the conditional probability above. 2. The other scenario considered by the authors is when there are multiple annotators; however, there is substantial literature on learning from multiple noisy annotations which the authors do not review. I suggest the authors start with *Modeling annotator expertise: Learning when everybody knows a bit of something* by Yan et al. (2010) (and the citations therein) which also considers the instance dependent noise case. 3. Under what conditions does the proposed algorithm converge and to what does it converge to? 4. The proposed method relies on several assumptions that are scattered throughout the description of the algorithm. I highly recommend making these assumptions clear near the beginning of the paper. Specifically, my understanding is that the main assumptions are: i. Confidence scores r_x are available for each instance and r_x approx P(Y=y|_x0008_ar{Y}=y,X=x). ii. Y _|_ X | Y
eq_x0008_ar{Y}, _x0008_ar{Y} = y iii. Anchor points are available on some portion of the data. Beyond the presentation, I find this to be a fairly strong set of assumptions, particularly the first assumption. --- Minor comments --- 1. I really appreciated the synthetic example demonstrating the potential pitfalls of the small loss approach; however, I would spend a bit more time clearly explaining the small loss approach so that readers understand why it fails and how you are solving it*s problems. 2. Also in the synthetic example, the authors state that covariate shift leads poor accuracy, however, I think this point would be stronger if demonstrated instead of just asserted.