Confidence Scores Make Instance-dependent Label-noise Learning Possible
Authors: Antonin Berthon
Bo Han
Gang Niu
Tongliang Liu
Masashi Sugiyama
Venue:  (2020)
Keywords: Instance-dependent label noise
Deep learning

Abstract: Learning with noisy labels has drawn a lot of attention. In this area, most of recent works only consider class-conditional noise, where the label noise is independent of its input features. This noise model may not be faithful to many real-world applications. Instead, few pioneer works have studied instance-dependent noise, but these methods are limited to strong assumptions on noise models. To alleviate this issue, we introduce confidence-scored instance-dependent noise (CSIDN), where each instance-label pair is associated with a confidence score. The confidence scores are sufficient to estimate the noise functions of each instance with minimal assumptions. Moreover, such scores can be easily and cheaply derived during the construction of the dataset through crowdsourcing or automatic annotation. To handle CSIDN, we design a benchmark algorithm termed instance-level forward correction. Empirical results on synthetic and real-world datasets demonstrate the utility of our proposed method.

Under review as a conference paper at ICLR 2020
CONFIDENCE SCORES MAKE INSTANCE-DEPENDENT LABEL-NOISE LEARNING POSSIBLE
Anonymous authors Paper under double-blind review
ABSTRACT
Learning with noisy labels has drawn a lot of attention. In this area, most of recent works only consider class-conditioaal noise, where the label noise is independent of its imput features. This noise model may not be faithful many real-world ap- plications. Instead, few pioneer works have studied instance-dependee dependent noise, but these methods are limited to strong assumptions on noise models. To alleviate this issue, we introduce confidence-scome scoreed instance-dependent noise (CSIDN), where each instance-label pair associated with a confudence score. The confidence scores are sufficient estimate the noise functions of each instance with minimal assumptions. Moreover, such scores can be easily and cheaply derived during the construction of the dataset through crowdsoureing handle CSIDN, we design a benchmark algorithm termed instance-level forward correction. Empirical results synthetic and real-world datasets demonstrate the utility of our proposed method.
automatic annotation. To
INTRODUCTION
The recent success of deep neural networks has increased the need for high-quality labeled data. However, such labelling process can be consuminn and costly. A compromise is to resort to weakly-supervised annotations, using crowdsourcing platforms or trained classifiers that annotate the data automatically. These suple--suereisee annotations tend to be low-quality and noisy, which negatively affects the accuracy of high-capacity models due to memorization effects (Zhang et al., 2017]. Thus, learning with noisy labels has often drawn a lot of attention.
Early works on noisy labels studied random classification noise (RCN) for binary classification (Angluin & Laird 1988 Kearns 1993). In the RCN model, each instance has its label flipped with a fixed noise rate p 10, 5) A natural extension of RCN is class-conditional noise (CCN) for multi- class classification (Stempfel & Ralaivola 2009 Natarajan et al. 2013 Scott et al. 2013 Menon eet al.] [2015 |van Rooyen & Williamson
Patrini et al. (2016) (Appendix/A Ap. In the CCN model, each instance from class ý has a fixed probability Pi.j of being assigned to class j. Thus, is possible to encode some similarity information between classes. For example, we can expect that the image of a "dog" is more likely to be erroneously labelled as "cat" than "boat".
To handle the CCN model, a common method is the loss correction, which aims correct the pre- diction or the loss of the classifier using an estimated noise transition matrix (Patrini et al.) 2017] Sukhbaatar et al., 2013 Goldberger & Ben-Reuven 2017; Ma et al., 2018). Another common ap- proach is the label correction, which aims to improve the label quality during training. For example, Reed et al. (0015) introduced bootstrapping scheme. Similarly, (anaka al./ (2018) proposed to update the weights of a classifier iteratively using noisy labels, and use the updated classifier to yield more high-quality pseudo-labels for the training set. Although these methods have theoretical guarantees, they are unable cope with real-world noise, e.g., instance-dependers noise (IDN).
The IDN model considers a more general noise MManwani & Sastry 2013 Ghosh et al. 2014 Menon et al. 2016 Cheng et al. 2017 Menon al. 2018), where the probability that instance is mislabeled depends on both its class and features. Intuitively, this noise is quite realistic, as poor- quality ambiguous instances
datasets. However, it much more complex to formulate the IDN model, since the probability of a mislabeled instance is function of not only the label space but also the input space that can be very high dimensional.
more likely to be mislabeled in

---

Under review as a conference paper at ICLR 2020
Table 1: Comparisons between baselines and our work for handling the IDN model. Rate identifia-
bility denotes whether the transition matrix is identifiable.
Approaches
Multi-class
Rate- Rate-identifiabiiity
Unbounded-noise
Du & Cai (2015
X
Menon et al. 2018
X
Bootkrajang & Cha
2018
X
Cheng al. [2017]
X
x
Our work
As a result, several pioneer works have considered stronger assumptions on noise functions. How- ever, stronger assumptions tend restrict the utility of these works (Table [ For instance, the boundary-consiste
noise model considers stronger noise for samples closer to the decision bound- ary of the Bayesian optimal classifier (Du & Cai 2015 Menon al. 2018). However, such a model is restricted to binary and cannot estimate noise functions. Cheng et al. (2017) recently studied partieular case of the IDN model, where noise functions are upper-bounded. Nonetheless, their method is limited to binary classification and has only been tested on small datasets.
Instead of simplifying assumptions on noise functions, we propose tackle the IDN model from the source, by considering confidence scores to be available for the label of each instance. We term this new setting confidence-scoree
setrnce-ddpendenn noise (CSIDN, Figure lcl. The confidence scores denote how likely an instance is to be correctly labeled. Assuming that (i) confidence scores are available for each instance, (ii) transitions probabilities to other classes are independent of the instance conditionally on the assigned label being erroneous and (iii) set of anchor points is avail- able, we derive an instance-level forward correction algorithm which can fully estimate the transi- tion probability for each instance, and subsequently train robust classifier with loss-correction method similarly to Patrini et al. (2017).
It is noted that confidence scores can be easily and cheaply derived during the construction of the dataset. For example, in crowdsourcing platforms, simply counting how many annotators agree on given instance can give notion of how confident label is. Besides, many real-world datasets automatically annotated using a trained classifier, such as web-scraped datasets (Tong Xiao et al.) 2013) and physiological features inferred from medical records (Agarwal et al. (0166). In these cases, the class- panss-rrbabbiltiess of the labels assigned by the classifier can be seen as confidence scores, provided that the classifier is well calibrated (Guo al. 2017).
To sum up. we first formulate instance-dependent noise in Section 2.1 and expose its robustness challenge in Section 2.2 Then, we explain our motivation to use confidence scores, and propose the confidence-scou scored castance-dependers noise (CSIDN) model in Section 2.3 Lastly, to handle this new noise model, we present the first practical algorithm termed instance-level forward correction in Section 3 and validate the proposed algorithm through extensive experiments in Section 4
TACKLING INSTANCE-DEPENDENT NOISE FROM THE SOURCE
In this section, we present the IDN model along with the limitations of existing approaches, and introduce the CSIDN model as tractable instance-dependent noise model.
2.1 NOISE MODELS: FROM CLASS-CONDITIONAL TO DEEENCEED NOISE
We formulate the problem of learning with noisy labels in this section. Let D be the distribution of a pair of random variables (X, Y) E X x y, where X' E y = {1, 2......., K} and K is the number of classes. In the classification task with noisy labels, having only access to samples from a noisy distribution D of random variables (X, Y) E X x y Given a point x sampled from X, Y is derived from the random variable Y via noise transition E [0, KxK;
hope to train a classifier while
matrix T(x)= (=) 1==1 (
K
v1 S S P(Ý = j|x = x)= = Σ (Y = x|X =|-= x).
(1)
2

---

Under review as a conference paper at ICLR 2020
(a) Class-conditioall noise
(b) Instance-dependent noise
(c) Confidence-scored dependent noise
instance-
Figure 1: Illustration of different noise models. Each color represents an observed class j; circles indicate y = y. while crosses indicate y * y. The size of each point represents the confidence scores in the label y; the bigger the point is, the more confident it is. In the CCN model, the noise function only depends on the label each instance. In the IDN and CSIDN models, the noise function depends on the observed instance x. To illustrate the IDN model, we show special case called boundary-consistent noise, i.c., points that lie close to the decision boundary are more likely to be mislabelled. Note the CSIDN model varies from the IDN model via confidence scores (Section 2.3
Each noise function Ti, : X + [0, 1] is defined as (x) = P(Ý = -|Y = i 2). In the class-conditional noise (CNN) model (Figure Tap, the transition matrix does not depend on the instance x and the noise is entirely characterized by the K² constants Tı.j- However, in the instance- dependent noise (IDN) model (Ib) the transition matrix depends on the actual instance. This tremendously complicates the problem, as the noise is now characterized by K functions over the latent space X, which be very high dimensional (e.g.. d 10--106 for object recognition dataset).
2.2 CHALLENGES FROM INSAACE-DEPENDEER NOISE
Limitation of existing CCN methods. Due to the complexity of the IDN model, most recent works in learning with noisy labels have focused on the CCN model (Figure [aa,, and the CCN model can be seen as a simplified IDN model (Figure lb) free of feature information.
In addition to loss correction and label correction mentioned before, another method for the CCN model is sample selection, which aims to find reliable samples during training, such as the small- loss approaches (Jiang et al.] al.]|2115 2018; Han et al.] 2018). Inspired by the memorization in deep learning (Arpit al. 2017). those methods first run standard classifier on noisy dataset, then select the small-loss samples for reliable training.
However, all approaches cannot handle the IDN model directly. Specifically, loss correction con- siders the noise model to be characterized by a fixed transition matrix, which does not include any instance-level information. Mcanwhile, label correction is vulnerable the IDN model, since the classifier will be much weaker on noisy regions and labels corrected by the current prediction would likely be erroneous. Similarly, sample selection is easily affected by the IDN model.
For example, in the small-loss approaches, instance-dependent noise functions can leave partial re- gions of the input space clean and other regions very noisy (e.g., in object recognition dataset, poor-quality pictures will tend to receive more noisy labels than high-quality ones). Since clean regions will tend to receive smaller losses than noisy regions, the small-loss approaches, which only trains on points with the smallest-losses, will focus on clean regions and neglect harder noisy re- gions. Then, since the distribution of clean regions will subsequently be different from the global distribution, this will introduce covariate-shife (Shimodaira] 2000). which greatly degrades perfor- mances. Moreover, it hard to use importance reweighting (Sugiyama al. (007) for alleviate the issue, since importance reweighting would require estimating the clean posterior probability that is intractable for the IDN model.
To validate this fact, we generate a 3-class distribution of concentric circles (cf. Figure 2ah with
v(a, v) E R² x (1,,,3), P(ū + y/x) =
|
+ with w (0, 1) (cf. Figure 2b
We

---

Under review as a conference paper at ICLR 2020
Clean distribution
Insance-dopendent noise distribution
Top selected instances densily
rightn
-2
-4
2
(a) Clean data distribution
(b) Data distribution with IDN
(c) Density of selected small-loss in- stances at epoch 10
Figure 2: The limitation of the small-loss approaches in the IDN model. (a) Clean distribution. (b) instance-dependens noise in the direction w: (0, 1) with average corruption rate of 40%: points towards the upper region are more likely to be corrupted than points towards the bottom region. (c) Density map of the instances selected by small-loss approach at epoch 10. The sample selection gets biased towards clean regions. Since the clean and noisy regions have different distributions, selecting most instances from clean regions creates covariate-shift between the training and test distributions, which can greatly degrades performances.
then train network on the top R(T) small-loss instances each epoch T based on the losses of the previous epoch, with R(T") decreasing in T as described in Han et al. (2018). Figure 22 shows the density of the top 50% small-loss instances selected after 10 epochs: since noisy regions are associated to higher losses, the network eventually tends to select instances from the clean region and neglect the noisy region. This leads to covariate-shift, which is associated with decreased per- formances (Shimodaira] 2000).
Limitation of pioneer IDN methods. The main challenge of the IDN model is the wide range of possible noise functions included in its formulation. Since each T- is a function of the high- dimensional input space X, it is challenging for a model to be flexible enough to any real-world noise function while being trainable corrupted datasets, let alone derive theoretical results. In- stead, various recent works have considered stronger assumptions on noise functions.
For instance, boundary-comsistenn noise (BCN), first introduced by (Du & Cai 2015) and general- ized in Menon et al. (2018), considers stronger noise for samples closer to the decision boundary of the Bayesian optimal classifier. This is reasonable model for noise from human annotators, since "harder" instances (i.e., instances closer to the decision boundary) are more likely to be cor- rupted. Moreover, is simple enough derive some theoretical guarantees, as done Menon al. 2018). Additionally, an extension of the BCN model was studied in BBookrajjng & Chaijaruwanich 2018). where the noise function boundary. However, the BCN model and its extension are restricted to binary classification, and their geometry-basen assumption becomes difficult to fathom for high-dimensional input spaces.
Gaussian mixture of the distance to the Bayesian optimal
Furthermore, Cheng et al.] (017)) recently studied a particular case of the IDN model, where the probabilities that the true labels of samples flip into corrupted ones have upper bounds. They pro- posed method based on distilled samples, where noisy labels agree with the optimal Bayesian classifier on the clean distribution. However, their method is limited to binary classification and has only been tested small UCI datasets. summariess the characteristics of those approaches.
2.3 CONFIDENCE-SCOREE INSTANCE- NOISE
Instead of simplifying assumptions on noise functions, we propose to tackle the IDN model from the source. Namely, we consider that, for each instance, we have access to a measure of confidence in the assigned label. As most of noisy datasets arise from crowdsourcing automatic annotation, such confidence scores can be easily derived during the dataset construction, often with no extra cost. This allows for good approximation of noise functions with weaker assumptions.
4

---

Under review as a conference paper at ICLR 2020
Before introducing our proposed noise model confidence-scorel instance-dependen noise (CSIDN, Figure Ie]. we first define what are the confidence scores, and explain why the confidence scores are available in real-world applications.
Definition of confidence scores. For any data point (a, 3) sampled from the joint distribution (X,Y) Y), we define the confidence score Ta follows.
re - P y|Y =
Namely, the probability that the assigned label is correct.
Availability of confidence scores. Our rationale is that in tasks involving instance-dependent noise, the confidence information can be easily derived with no extra cost.
Firstly, in crowdsoureing platforms, when multiple workers manually annotate datasets, aggre- gation step often took to aggregate answers of different workers for each instance (e.g., majority vote). An estimation of Tx could then be derived by taking the ratio of votes for the assigned label on the total number of workers. Moreover, since this estimation would of course be less reliable as the number of workers decreases, an alternative could be directly ask workers for self-reported confidence scores of their responses (Cosmides & Tooby 1996. Oyama et al. 2013).
Secondly, the confidence information also be available in automatic annotation via softmax output layer of deep neural networks. This layer outputs an estimation of the probability that each class is the true label: when model outputs given class with probability 0.9, expect the pre- dicted class to be true 9 times out of 10 on average. A model that estimates the accurate probability callbcabibr.edd Therefore, in the case of labels generated by well-calibrated model, the softmax probability of the assigned label can be directly interpreted as a confidence measure that the label correct. Even though Guo al. (2017) showed that recent deep neural networks are not usually well-calibrated (whereas early shallower networks were, as shown in Niculescu- & Caruana (2005), model calibration can be achieved in a relatively straightforward way at the validation time, e.g., using temperature scaling (Section 4.2 in Guo et al. (2017))
CSIDN: a tractable instance-dependent noise model. Recall the intrinsic difficulty of the IDN
model: to fully characterize this noise, one would need to estimate K2 functions Tzy(-) over the input space X. This is of course intractable with finite noisy dataset. This is why pioneer solutions to the IDN model have been so far limited by very strong assumptions.
However, considering additional confidence scores, one can wonder whether such information would make the IDN model tractable with less restrictive assumptions. Hence, introduce new and tractable instance-dependent noise model: confidence-scored instance-dependent noise (CSIDN, Figure lch. In this noise model, the training data takes the form S == {(rri, Di, Tx;);
1,...,Ny N). where {3r,, W iid. D and rz; P(Y V Di, X xzx) is the previously defined confidence scores in the assigned label of a given instance (Eq. 2 The confidence infor- mation rs is decisive for robustness to instance-deee nesennee-eependett noise, provides proxy for the noise functions Ti of the training data that are often intractable otherwise.
BENCHMARK SOLUTION FOR HANDLING THE CSIDN MODEL
To tackle the CSIDN model, we propose a benchmark solution. Inspired by forward correction (Patrini et al.) 2017) for the CCN model, we want to correct each prediction P(j]æ) with the noise transition matrix Tr). However, the transition matrix for the CSIDN model is instance-dependent, and has to be estimated for each instance We term our solution instance-level forward correction.
3.1 ESTIMATING DNSTANCE-DEPENDER TRANSITION MATRIX
Using the confidence scores, we will first estimate the diagonal terms (Ti,())) of the transition matrix, and then estimate the non-diagonal ones.
Diagonal terms. The diagonal terms of the transition matrix correspond to the probabilities that assigned labels are equal to true labels. However, the confidence scores available only relevant
5

---

Under review as a conference paper at ICLR 2020
to the class corresponding to the observed label. Therefore, need to proceed differently whether the confidence scores are available for the considered class or not.
First, note that for each sample (z, ra) S [(x, (.22) E S i}. (z)(2) can be derived
for the most part from the confidence scores alone:
V,) E Si, T2,(x) P(Y = =i|Y = x = xx)
= P(Y = i|y = = x) P(| = = x)
P(Y = =i = x)
rx (1)
(3)
where B((x)= -(X=i|X=x) P(Y=IX==))
In practice, we use an iterative procedure to estimate in turn 3-(-) and T.x(-) (see Section 3.2 for details). Then, for the rest of samples (Ir, y. rx)) E SY, Tz does not give any direct information on T.,(-). Hence, we simply set each function Ti,(-) as its empirical mean 14 estimated using samples from S; at the current epoch:
v.r2) SYS, (x)
Σ
T2,(z') pi:
(4)
SS
(wr irss
where |S| denotes the cardinality of S.
Non-diagonal terms. For non-diagonal terms, we have:
j.vz X, (z)(2) = P(Ý = =j(Y -,
= -P( = =
= =P( = =
=) + 2)
(1) (x),
(5)
where (1)( = 1 -ix= z).
In Eq. (4), (z) refers to the probability that an instance z with true label i has an observed label j, once we know that the observed label is different from the true one. Then, reasonable assumption is that vi * Vo E X; (x) Oiiz conditionally on the observed label being erroneous, the class transitions are not influenced by the instance æ. In other words, the dependence x of the noise function only impacts the "magnitude" of the noise and not the class transitions.
To illustrate this assumption, consider a crowdsourcing task of object recognition with adjacent classes which annotators can only differentiate with details that can be more less visible depending on the instance. For example, objects from a given class may have distinctive traits, but those can be more or less visible in the pictures. When those traits are present, the annotators can confidently predict the right class. Otherwise, they will make errors towards adjacent classes. In this case, the probability that the assigned label is wrong highly depends the instance (with distinctive traits being visible or not). Nonetheless, conditionally on the instance being corrupted, i.c., because those traits were not visible enough on the image, the transition probabilities to the adjacent classes are not influenced by the instance itself.
With the previous assumption, we obtain Vi # V. E Xx)),(z))= = a1 (x)) with asj [0. 1]. This allows us to estimate the K (K -1) constants (as,,) Vişs once, and derive the non non-diagonal noise functions of (xx) directly from our estimates of the diagonal noise functions (Eq. (5))
3.2 OVERALL ALGORITHM: INSTANCE-LEVEL FORWARD CORRECTION
Estimating Tr,; and 8,. To train classifier h with the instance-level forward correction method, =i X==)
we need to estimate both (x) and 3x(x) = P(Y=i x x=x) from Eq. 3. for all x E S.. Firstly, the noisy posterior P(Y = i|X = x) can be easily estimated by training a naive classifier on the noisy dataset. Secondly, the true posterior P(Y = i| z) can be estimated using the output of the classifier h(x) P(y x|X- x>) at the previous epoch.

---

Under review as a conference paper at ICLR 2020
Therefore, iteratively update / and T with the following steps: 1) V/ X, initialize {.
and train naive classifier Mosss the noisy data D obtain monssy (x) P( x). 2)
vi E K], for each sample (x, Tx) E Si, compute T(,(2) = (x) and train classifier for epoch. Vi [1, for each sample (I, S. update (x(x) Aumr (*), Then,
b(z),
we repeat steps 2) and 3) through training. In this way, for every epoch, each function (..(.) is
estimated for the samples from S₁. Lastly, for the rest of samples with noisy label j # TTs(:) is
estimated at each epoch using Eq. (4)
V(,,U,T)) E S' T2,(z)
Σ
(r = pr.
Si
Computing as,y The computation of a4, boils down to approximating non non-diagonal terms of the transition matrix in the CCN model. As vi # V.v E X, (z)(2) =0 (z,())),, we have:
Ez [Tay(x)] 0s. (1 - (T),()))) + 0z Eg [T)
Fz [T),(x)]
simple and reliable way is use anchor points, i.e., points for which can know the true class almost surely. These points may be directly available when some training data has been curated, or they can be identified either theoretically as in Liu & Tao (2015) heuristically as Patrini et al.) 2017). Having ST == ((r, rx) E P(Y = i = ~ I a set of class anchor points, we simply need compute:
V(x, T2) ST, Vj Ai, (2) - P(Y i|X x) and T()(s) P(Y = =|
Two noisy posteriors be estimated using the same classifier noois trained the noisy distribu- tion mooss (x) P( X æ) aforementioned. Thus, az, can be estimated as follows:
Σ
Dmnoisy (x))
( EST
VI S i K.jj O6
Σ
T2 (2)) (x);
(z.0.r.)ES
Summary of the training procedure. Given samples S and K sets of anchor points (S; ) we want to train classifier h(.) equipped with a loss 7. For any loss 1 y. + I(y, 8). we define the T-corrected loss as lT y, + 1(y, Tğ). The overall procedure is in Algorithm (Appendix C
4 EXPERIMENTS
We compare our instance-level forward correction (ILFC) method with four representative baselines: forward correction (FC) (Patrini et al. 2017). mean absolute error (MAE) (Ghosh et al. (2017), T.y- norm (LQ) (Zhang Sabuncu 2018) and co-teaching (CT) (Han et al., 2018). Details are shown in Appendix D Note that the pioneer IDN methods cannot work for multi-class cases.
4.1 SYNTHETIC DATASET
Generation process. We generate synthetic dataset (Appendix consisting in three classes of concentric circles (Figure 6ap. We then apply the following instance-dependent noise to each label: P(Y # Y x) p corrupted, each label is flipped to another class uniformly.
| + 1) /2 with (0, 1) and p controlling the noise rate. If
Empirical results. Figure Jshows the test accuracy of different methods on the synthetic dataset. Each experiment is repeated 5 times and we plot the confidence intervals of each curve. On low-level noise, all methods show good performances (Figure Bay. On mild-level noise, both Co-teaching and ILFC show good performances and outperform other baselines (Figure On high-level noise, the performance of all the baselines collapse, whereas ILFC constantly maintains good performances (Figures 3c and 3d. More experiments shown in Appendix and

---

Under review as a conference paper at ICLR 2020
ILFC
MAR
PC
LQ
CT
IDN 25%
IDN
IDN 45%
IDN 50%
30.00
8000
30.30
$2.00
00.30
09:00
50.30
00.30
30.00
3030
Epocts
Epoch
Epoch
Epoch
(a) p 0.25
(b) 0.35
(c) p 0.45
(d) p 0.50
Figure 3: The test accuracy on synthetic datasets with different levels of IDN noise.
ILFC
MAE
(SVHNN IDN 25%)
(SVIEN, IDN 85%)
(CEARIO, IDN 25%)
(CIFARIO, IDN 45%)
300
600
50.0
Tel: Accurity
500
800
R33
Ted
304
200
Epoch
Epoch
Epoch
Epoch
(a) SVHN, IDN-25%
(b) SVHN, IDN-45%
(c) CIFARIO. IDN-25% (d) CIFARIO, IDN-45%
Figure 4: The test accuracy on real-world datasets with different levels of IDN noise.
4.2 REAL-WORLD DATASET
Generation process. In order corrupt labels from clean datasets such as SVHN and CIFARIO. we adopt the following procedure: (1) train a classifier h x + (g(2)) on a small subset of the clean dataset; (2) using a small validation set, calibrate the classifier by selecting the temperature t that maximizes the expected calibration error as in Guo et al. (2017); (3) for each instance x, set: ū argmax; he(x);; and r- = max; (x); With this process, we attempt to emulate the construction of real-world dataset (Appendix G
Empirical results. Figures 4a and 4b show the test accuracy on SVHN with 25% and 45% instance-dependent noise, respectively. We clearly observe that, both low-level and high- level noise, ILFC shows good performances with a fast convergence rate, and outperforms other baselines. Figures 4c] and 44 show the test accuracy on CIFARIO with 25% and 45% instance- dependent noise, respectively. On low-level noise, all methods show good performances. However, on high-level noise, ILFC shows a fast convergence rate and outperforms other baselines.
CONCLUSION
In this paper, we give an overview of label-noise learning from class- candicoondii noise (easier) to restance-dependee noise (harder). We explain why existing approaches cannot handle instance- dependent noise well, and try address this challenge via confidence scores. Thus, we formally propose the confidence-scored instance-dependet noise (CSIDN) model. To tackle the CSIDN model, we design a practical algorithm termed instance-level forward correction (ILFC). Our ILFC method robustly outperforms existing methods, especially in the case of high-level noise. In future works, we would like to extend label correction and sample selection approaches with the confidence scores from the CSIDN model.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vibhu Agarwal, Tanya Podchiyska, Juan M Banda, Veena Goel, Tiffany Leung, Evan P Minty, Timothy Sweeney, Elsie Gyang, and Nigam Shah. Learning statistical models of phenotypes using noisy labeled training data. Journal of the American Medical Informatics Association, 23 (6):1166-1173, 1173, 2016.
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look memorization in deep networks. In ICMI,, 2017.
Jakramate Bootkrajang and Jeerayut Chaijaruwanich. Towards instance-dependent label noise- tolerant classification: a probabilistic approach. Patlern Analysis and Applications, pp. 1-17, 2018.
Steve Branson, Grant Van Horn, and Pietro Perona. Lean crowdsoureing: Combining humans and machines in online system. In CVPR, 2017.
Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On Symmetric Losses for Learning from Corrupted Labels. ICML, 2019.
Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance-and label-dependent label noise. stat, 1050:12, 2017.
Leda Cosmides and John Tooby. Are humans good intuitive statisticians after all? rethinking some conclusions from the literature on judgment under uncertainty. Cognition, 58(1):1-73, 1996.
Jun Du and Zhihua Cai. Modelling class noise with symmetric and asymmetric distributions. In AAAI, 2015.
Aritra Ghosh, Naresh Manwani, and Sastry. Making risk minimization tolerant to label noise. Neurocomputing, 160, 2014.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, 2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.
Melody Guan, Varun Gulshan, Andrew M Dai, and Geoffrey Hinton. Who said what: Modeling individual labelers improves classification. In AAAI, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.
Takashi Ishida, Gang Niu, and Masashi Sugiyama. Binary classification from positive-confidence data. In NeurIPS, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
Michael Kearns. Efficient noise-tolerant learning from statistical queries. Proceedings of the twenty- fifth annual ACM symposium on Theory of computing STOC 93, 1993.
Ashish Khetan, Zachary C. Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. In ICLR, 2018.
Samuli Laine and Timo Aila. Temporal Ensembling for Semi-Supervised Leaming. ICL.R, 2017.

---

Under review as a conference paper at ICLR 2020
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447 461, 2015.
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wi- jewickrema, and James Bailey, Dimensionality-driven learning with noisy labels. In ICML, 2018.
Naresh Manwani and P. S. Sastry. Noise tolerance under risk minimization. IEEE Transactions on Cybernetics, 43:1 146-1 151, 2013.
Hamed Masnadi-shiradt and Nuno Vasconcelos. On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost. NeurlPS. 2009.
Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from cor- rupted binary labels via closs-pro estimation. In ICML, pp. 125-134, 2015.
Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. Learning from binary labels with instance-dependent corruption. arXiv preprint arXiv:1605.0075:: 2016.
Aditya Krishna Menon, Brendan van Rooyen, and Nagarajan Natarajan. Learning from binary labels with instance-dependent noise. Machine Learning, 107(8- 10):1561-1599 September 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1999-1995 2018.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with Noisy Labels. In NeurIPS. 2013.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learn- ing. In ICML, 2005.
Satoshi Oyama, Yukino Baba, Yuko Sakurai, and Hisashi Kashima. Accurate integration of crowd- sourced labels using workers' self-reported confidence scores. In ICCAI, 2013.
Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Loss factorization, weakly supervised learning and label noise robustness. In ICMI, 2016.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
Vikas C Raykar, Shipeng Yu, Linda Zhao, Anna Jerebko, Charles Florin, Gerardo Hermosillo Valadez, Luca Bogoni, and Linda Moy. Supervised learning from multiple experts: whom to trust when everyone lies a bit. In ICML, 2009.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. ICLR, 2015.
Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. COLT, pp. 489-511, 2013.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. In ICML, 2019.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244 2000.
Rion Snow, Brendan O' O'Connor, Daniel Jurafsky, and Andrew Ng. Cheap and fast - but is it good? evaluating non non-expert annotations for natural language tasks. EMNLP. 2008
Guillaume Stempfel and Liva Ralaivola. Learning SVMs from sloppily labeled data. International Conference Artificial Neural Networks, pp. 884 893, 2009.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert MÅžller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(May): -180 2007.
10

---

Under review as a conference paper at ICLR 2020
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. ICLR workshop, 2015.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averagen consis- tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
Brendan van Rooyen and Robert C. Williamson. Learning in the Presence of Corruption. arXiv e-prints, arXiv: 150.400099,, Mar 2015.
Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. Modeling annotator expertise: Learning when everybody knows bit of something. AISTATS, 2010.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized Cross Entropy Loss for Training Deep Neural Net- works with Noisy Labels. In NeurlPS. 2018.

---

Under review as a conference paper at ICLR 2020
A
RELATED WORKS
Besides the works aforementioned, we survey other approaches to learning with noisy labels.
Robust losses. Various approaches propose to use provably robust loss function in the learning process. In the case of class-dependent label noise, Natarajan et al. (2013) constructed an unbiased estimator of any loss function under the noisy distribution. Masnadi-shirazi & Vasconcelos (2009) introduced a robust non-convex loss. Recently, works on symmetric losses showed that such loss offer theoretical robustness results various types of noise (Ghosh et al. 2017] Charoenphakdee itt al. (2019). Motivated by the robustness to noise of the mean absolute error loss (MAE) shown in Ghosh et al. (2017), /Zhang & Sabuncu (2018) introduced generalized cross entropy loss that allows for a trade-off between the efficient learning properties of the CCE loss and the noise-robustness of MAR. (Shen& & Sanghavi (2019) introduced trimmed loss with iterative minimization process that allows for theoretical guarantees in the simpler setting of generalized linear models.
Annotator-level modelling. Another recent line of related works attempts to model labels and worker's ' quality directly during the crowdsoureing annotation process, in order to produce more ac- curate labels efficiently. [Branson et al. (2017) modeled the annotators' skill and instances difficulty while incrementally training computer vision model during the annotation process, effectively re- ducing the time burden of the annotation process as well as the error rate in the assigned labels. Guan et al. (2018) modeled each annotator individually in order to better aggregate labels based on each worker's s skill and area of expertise. Khetan et al. (2018) introduced method that allows to learn each workers' skill even when each example is only annotated once, by jointly modelling the assigned labels and the workers during the annotation process.
Learning with multiple noisy labels. A closely related setting is learning from multiple noisy labels, where the aim is to predict an unknown ground-truth label from (X, (Y²)g), each YJ referring to a noisy annotation. This setting can arise for example from crowdsourcing tasks; [Snow et al. (2008) showed that using multiple non-expert annotators to train a classifier can be as effective as using gold standard annotations from experts. In Raykar et al. (2009), the authors derive a Bayesian approach to jointly learn the expertise of each annotator, the actual true label and the classifier. Yan al. (2010) extends this Bayesian approach by considering that each annotator's expertise varies across the input space. This setting differs from ours as it takes place before the aggregation of multiple annotations, which, for CSIDN, is only a way among others to obtain a confidence score for each noisy label.
Explicit/implict regularizers. Recently, several other regularization techniques have shown good robustness in reakly-superviied settings. Temporal Ensembling (TE) (Laine & Aila, (017) method labels some additional unlabeled instances using consensus of predictions from models from pre- vious epochs and with different regularizations and input augmentation conditions. Mean-teacher (MT) (Tarvainen & Valpola| 2017) instead uses predictions from model obtained by averaging the weights of a set of models similar to TE, as using the prediction from a unique model is more efficient when large amount of unlabeled data available. Virtual Adversarial Training (Miyato ct al. 2018) regularizes the network using measure of local smoothness of the conditional label distribution given the input, defined the robustness of the prediction to local adversarial pertur- bations in the input space. Introduced in Zhang et al. (2018). mixup trains neural network on convex combinations of instance pairs and their respective labels, and has been shown to reduce the memorization of corrupted labels.
B SENSITIVITY ANALYSIS
In practice, the confidence scores obtained may not accurate. Therefore, we run a sensitivity analysis to assess the robustness of ILFC: similarly to Ishida et al. (0118), we add a zero-mean Gaussian noise with standard deviation a E {0.0, 0.3, 0.6} to each confidence score and clip the values between 0 and 1. Figure shows the resulting performances shows good robustness to inaccurate confidence scores even with high standard deviation on a highly noisy dataset.
the synthetic dataset. ILFC
12

---

Under review as a conference paper at ICLR 2020
IDN-25%
IDN-35%
80.0
80.0
Test Accuracy
Test Accuracy
70.0
Std
Std
60.0
0.3
50,0
0.3
0.6
0.6
50.0
40.0
Epoch
Epoch
(a) p 0.25
(h) 0.35
IDN-45%
IDN-50%
80.0
80.0
Test Accuracy
Test Accuracy
70.0
Std
60.0
Std
50.0
0.0
0.0
0.3
0.3
40.0
0.6
0.6
30.0
0
8
0
2
Epoch
Epoch
(c) p 0.45
(d) 0.50
Figure 5: Sensitivity analysis on the synthetic dataset: a zero-mean Gaussian noise of standard deviation e {0.0, 0.3, 0.6] is added to each confidence score before running ILFC with the noisy confidence scores.
C ALGORITHM
We present Algorithm here, which can be referred Section 3.2 in details.
D BASELINES
Forward correction. Introduced in Patrini al. (2017).forward forward correction estimates fixed tran- sition matrix T' before training, and trains classifier with the corrected loss 4T (y, 9) + U(y TQ).
Mean absolute error loss. Due its symmetric property, the Mean Absolute Error (MAE) has been theoretically justified to be robust to label noise under assumptions (Ghosh et al.) 2017). How- ever, this loss more difficult to train, especially on complex datasets.
Lg norm Introduced in Zhang & Sabuncu (2018), Lg norm attempts to bring the best of both worlds between the CCE and the MAE loss: the CCE is easy to train, while the MAE is robust to label noise. The authors therefore define this loss using the negative box-cox transformation:
by (M(z)), (1 (x)
4
so that the Lg tends to the CCE when q + and to the MAE when q In the following experiments,
0.7, suggested by authors.
Co-teaching (Han et al. 2018). Co-teaching algorithm is a small-loss approach where two clas- sifiers are trained in parallel. At each epoch, each classifier selects the instances with the smallest
13

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Instance-level Forward Correction
Input confidence-annotated samples = [(r., Ma, Tx;), 1...., N.. any loss classifier l(.), anchor points sets (S: )
(1) Train naive classifier havisy on samples {(zi, 14)1:
(2) VI < i,j S K, compute [t] from Eq. 7 with anchor points set S
(3) LI
K, initialize = =
for epoch = 1,..., Nmax do
/ Update diagonal constants
(4) v1 < i < K, compute sli] from Eq. 6
for (z, y. ra)
do
Set =
// Compute diagonal terms
(5) = (z)r(z) and Ak (1, 14k;;
1/ Compute non-diagonal terms
(6) Set i,j & K, i
1/ Train classifies with Enstance-leved corrected loss
(7) Train h(-) on sample (z,
(. (2) with loss lz;
1/ Update density ratio estimate
Masiy,
(8) Update Sis K, Va E St, ((()) (x(
end
end
Output classifier l(-).
loss, and feed them to the other network as a training set for the next iteration. This recent work has proved be a leading benchmark in the field of noisy labels.
E
SYNTHETIC DATASET
Figure 6 shows three synthetic datasets, which cover clean, IDN and CSIDN models.
Clean distribution
Mustace-DDppendee Noise
Iostance Dependent Noise Confidence
(a) Clean
(b) IDN
(c) CSIDN
Figure 6: Synthetic dataset. The clean distribution (a) consists in three classes of concentric circles.
T + +1) /2 with (0, 1) of being corrupted, where is a parameter controlling the mean noise rate. Therefore the noise the strongest towards the direction (0, 1) and the weakest in the direction (0, 10) If corrupted, the label is flipped to another class uniformly. The CSIDN setting (c) is similar to the IDN setting, but each point is associated with measure the confidence in the assigned label. A lower confidence is represented by lower opacity in the figure.
In the IDN setting (b), each point has probability P(Ý + Y|z)
F DECISION BOUNDARIES
Figure 7 shows the decision boundaries of our approach versus the ones a benchmark model, for different levels of noise. With high levels of noise, model that does not include any instance-level
14

---

Under review as a conference paper at ICLR 2020
modelling will degenerate around the most noisy region of the imput space. On the other hand, our model successfully accounts for the high noise in this region and is able to keep consistent predictions.
G EXAMPLES OF REAL-WORLD DATASETS
For example, the method would be similar to constructing dataset with images scraped from the web, and automatically labelling them from neighhouring text fields using classifier such as recurrent neural network. Then, small subset of curated images could be used the beginning of the process to calibrate the classifier, in order to make the predictions of the softmax output faithful to the confidence in each label. This way, we could construct a very large dataset for very low-cost that, while involving some instance-dependee information and therefore could be tackled with our proposed algorithm.
noise, would be equipped with confidence
ILFC Average noise 30.0%
Lg Average noise -3.00%
(a) ILFC, 0.30
(b) LQ. P = 0.30
ILKC Average noise 40.0%
Average noise 40 0%
cc) ILFC, P 0.40
(d) LQ, P = 0.40
ILFC Average noise 50.0%
Lg Average noise 50.0%
(e) ILFC, P 0.50
(1) LQ. P 0.50
Figure 7: Decision boundaries of the learned classifier for the ILFC model (left column) and the Lg norm model (right column). In the presence of highly noisy regions, a classifier that does not include any instance-level information will degenerate in those regions, while the ILFC approach stays consistent with the clean distribution.
15


Reviews:
[{"review_id": "r1xGNNd2FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on instance-dependent label noise problem, which is a new and important area in learning with noisy labels. The authors propose confidence-scored instance-dependent noise (CSIDN) to overcome strong assumptions on noise models. They clearly define confidence scores and justify their availability. To solve CSIDN model, they propose instance-level forward correction with theoretical guarantees. Their experiments on both synthetic and real-world datasets show the advantage of this algorithm.\n\nPros:\n\n1. This paper is clearly written and well-structured in logic. For example, in Section 2, they introduce from class-conditional noise to instance-dependent noise first, which paves the way for confidence-scored instance-dependent noise. This make readers easy to follow the main contribution, namely the new noise model.\n\n2. This paper pushes the knowledge boundary of learning with noisy labels, since it focuses on more realistic and challenge topic \"instance-dependent label noise\". The authors leverage the idea of confidence scores, and propose confidence-scored instance-dependent noise (CSIDN). Compared to previous solutions, CSIDN is a tractable instance-dependent noise model, which enjoys several benefits, such as multi-class classification, rate-identifiability and unbound-noise.\n\n3. This paper proposes an algorithm to solve CSIDN inspired by forward correction called instance-level forward correction (ILFC). Their algorithm has been verified in both synthetic datasets and real-world datasets. The empirical results show the advantage of ILFC.\n\n(Minor) cons:\n\n1. Section 3 is a bit dense in understanding the estimation of transition matrix. The authors are encouraged to polish this section. \n\n2. Although ILFC outperform CT and LQ in real-world datasets, the authors need to add the reasults of MAE and FC to more thoroughly verify the performance of ILFC.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bye6SmyRYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Overall ---\n\nI think the problem of incorporating information about labeling uncertainty (when such information is available) is an interesting and important problem; however, I think this paper contains a crucial misunderstanding of the definition of calibration (as defined in Guo et al. 2017), is missing some important parts of the literature, and does not provide adequate convergence guarantees for the proposed method.\n\n--- Major comments ---\n\n1. In section 3.1, the authors substitute the confidence score r_x for the conditional probability P(Y=y|\\bar{Y}=y,X=x); however, even if the confidence scores are perfectly calibrated, these quantities are not necessarily equal (or even particularly close). As defined in Guo et al. (2017), a confidence score is well calibrated if the probability that a prediction is correct given the confidence score is equal to the confidence score. Using their notation: P(Y=\\hat{Y}|\\hat{P}=p) = p. Importantly, this is not conditioned on \\hat{Y} or X. One way to see that these two quantities are not equal is to observe that there are many possible confidence score functions that satisfy the above definition whereas the conditional probability defined above is a unique function. In particular, if the confidence score function is a constant equal to the accuracy of the model, then the confidence score is well calibrated, but clearly not equal to the conditional probability above.\n\n2. The other scenario considered by the authors is when there are multiple annotators; however, there is substantial literature on learning from multiple noisy annotations which the authors do not review. I suggest the authors start with \"Modeling annotator expertise: Learning when everybody knows a bit of something\" by Yan et al. (2010)  (and the citations therein) which also considers the instance dependent noise case.\n\n3. Under what conditions does the proposed algorithm converge and to what does it converge to?\n\n4. The proposed method relies on several assumptions that are scattered throughout the description of the algorithm. I highly recommend making these assumptions clear near the beginning of the paper. Specifically, my understanding is that the main assumptions are:\n\ni. Confidence scores r_x are available for each instance and r_x \\approx P(Y=y|\\bar{Y}=y,X=x).\nii. Y _|_ X | Y\\neq\\bar{Y}, \\bar{Y} = y\niii. Anchor points are available on some portion of the data.\n\nBeyond the presentation, I find this to be a fairly strong set of assumptions, particularly the first assumption. \n\n--- Minor comments ---\n\n1. I really appreciated the synthetic example demonstrating the potential pitfalls of the small loss approach; however, I would spend a bit more time clearly explaining the small loss approach so that readers understand why it fails and how you are solving it's problems.\n\n2. Also in the synthetic example, the authors state that covariate shift leads poor accuracy, however, I think this point would be stronger if demonstrated instead of just asserted.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxXPiqHqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Learning with noise labels is a hot topic now due to the reason that deep learning algorithms often require large-scale supervised training samples and labelling a large amount of data is costly. However, almost all of the existing methods assume that the label noise is instance-independent. It either depends on the clean classes or is completely random. This paper studies the instance-dependent label noise, which is more realistic and applicable, but difficulty to address. The authors target to solve this problem. A feasible solution would contribute to the community a lot.\n\nThe challenging part for dealing with instance-dependent label noise is to learn the instance-dependent label flip function, which is hard to learn by only exploiting noisy data without any assumption. Few existing papers proposed assumptions to make the flip function learnable. This paper also introduces an assumption that the confidence score for each data is given. To me, this is also a strong assumption. Although the authors have provided examples of how to collect confidence score, collecting confidence scores may not be easy for many specific problems. \n\nGiven the confidence score, the authors proposed to learn the flip function. The clean class posterior can be inferred by using the noisy class posterior and the label flip function. So, how to learn the instance-dependent flip function is an essential step. Some technical contribution has been made to learn the flip function. In this section, the authors further assume that the off-diagonal entries of the flip matrix are independent of instance. After seeing the explanation, I personally agree that the assumption is reasonable for some cases. However, I found that in the experiments, the authors synthesized label noise where the off-diagonal entries depend on the instance. The proposed method also shows its advantages on this case. Can you explain this?\n\nThe experiment parts show the effectiveness of the proposed method both on synthetic data and real-world data. Overall, the paper is well-motivated and well-written. \n\nI have another concern that the obtained confidence score may not accurate. Is the proposed method sensitive to the confidence scores?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While two reviewers  rated this paper as an accept, reviewer 3 strongly believes there are unresolved issues with the work as summarized in their post-rebuttal review. This work seems very promising and while the AC will recommend rejection at this time, the authors are strongly encouraged to resubmit this work.
Decision: Reject