AutoCleansing: Unbiased Estimation of Deep Learning with Mislabeled Data
Authors: Koichi Kuriyama
Venue:  (2021)
Keywords: Automatic Data Cleansing
Incorrect Labels
Multiple Objects

Abstract: Mislabeled samples cause prediction errors. This study proposes a solution to the problem of incorrect labels, called AutoCleansing, to automatically capture the effect of incorrect labels and mitigate it without removing the mislabeled samples. AutoCleansing consists of a base network model and sample-category specific constants. Both parameters of the base model and sample-category constants are estimated simultaneously using the training data. Thereafter, predictions for test data are made using a base model without the constants capturing the mislabeled effects. A theoretical model for AutoCleansing is developed and showing that the gradient of the loss function of the proposed method can be zero at true parameters with mislabeled data if the model is correctly constructed.  Experimental results show that AutoCleansing has better performance in test accuracy than previous studies for CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.

Under review as a conference paper at ICLR 2021
AUTOCLEANSING: UNBIASED ESTIMATION OF DEEP LEARNING WITH MISLABELED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Mislabeled samples cause prediction errors. This study proposes solution the problem of incorrect labels, known as AutoCleansing, to automatically cap- ture the effect of incorrect labels and mitigate it without removing the mislabeled samples. AutoCleansing consists of a base network model and sample-category specific constants. Both parameters of the base model and sample-category con- stants are estimated simultancously using the training data. Thereafter, predictions for test data are made using base model without the constants capturing the mis- labeled effects. A theoretical model for AutoCleansing is developed and showing that the gradient of the loss function of the proposed method can zero true parameters with mislabeled data if the model is correctly constructed. Experimen- tal results show that AutoCleansing has better performance in test accuracy than previous studies for CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.
INTRODUCTION
The prediction performance of supervised machine learning depends on the quality of the training data. For classification tasks, the dataset is assumed to have correct label for each object. However, real-world datasets may contain some mislabeled samples. For instance, Pleiss et al. (2020) analyzed incorrect labels in the CIFAR-10 and CIFAR- --0 datasets (Krizhevsky & Hinton 2009).. They reported that the mislabeled sample was of CIFAR-10 and 13 of the CIFAR- 100 datasets.
#1
#2
#3
Image
10-
Original label
DOG
TRUCK PERSON
DEER DEER PERSON
Alternative label
CAT
Example of incorrect labels in CIFAR-10. The original label is the corresponding label of each image in the dataset. Alternative label is the possibly correct label of each image.
Figure 1:
Figure shows typical examples of incorrect labels in the CIFAR-10 dataset. It consists of 60,000 images 10 category classes. Each image was assigned one of 10 classes. this figure, the original label of #1 is DOG; however, it appears to be the image of CAT. As the category set of CIFAR-10 10 includes both DOG and CAT, #1 is considered to be an example of an incorrect label within the category set. The image of #2 has TRUCK as the original label. However, it shows the image of PERSON, which does not belong to the category set. Thus, this is an example of an incorrect label outside the category set. For the image of #3, there two objects in this image; however, it has only one label DEER. It can be considered as an example of an incorrect label with multiple objects.

---

Under review as a conference paper at ICLR 2021
Incorrect labels in the training dataset may cause prediction errors. The most intuitive way to ad- dress the problem of incorrect labels is by removing mislabeled samples from the training dataset. However, to identify mislabeled samples, it is necessary to measure the correctness of labels and de- fine the threshold determining whether the label correct or not. Deleting excess data may reduce the efficiency of estimation by decreasing the sample size. Finding optimal threshold requires several runs of learning by removing mislabeled samples with different thresholds.
This study proposes an alternative solution the problem of incorrect labels, called AutoCleansing to automatically capture the effect of incorrect labels and mitigate it without removing mislabeled samples. AutoCleansing consists of base network model and sample-category specific constants. Both parameters of the base model and cunple-category constants are estimated simultancously using the training data. Thereafter, predictions for test data are made using a base model without the constants capturing the mislabeled effects.
Figure shows the concept of AutoCleansing. Let x be the input, y be the output, and y = m(x. 0) be the base network model, where 0 denotes the parameter of the base model. Consider five obser- vations of A, B, ...., E. The red line is the true model defined as, y = mı( 0*) where 0* denotes the true parameter. B is a mislabeled sample, as the observed label B differs significantly from the true label B" The dotted line is the estimated model y m(x, 0) using incorrect data, where A denotes the estimated parameter. As can be observed, overfitting occurs owing to the mislabeled sample. In this figure, denotes the prediction for x= 3 using the estimated model; however, the true label is y². Thus, the incorrect label causes the prediction error. Consider the cleansing model, y m (x, 0) + a, where denotes the constant parameter for each observation. If the constant captures the effect of an incorrect label, as shown in this figure, removing the constant from the cleansing model may mitigate the overfitting problem.
Correct
Incorrect
A,
B
B (Incorrect)
Output (y)
Cl
y = mm(x)= + a
(4)
(3)
B*
y
True
= m(x)
α
Observed
Prediction
1
2
3
5
6
Input (x)
Figure 2: Concept of AutoCleansing. (Left) A to E are the observations. B is the incorrect label. Red line represents the true model. Black dot line represents the estimated model using incorrect data. y denotes the predicted label using the overfitting model. y² denotes the true label. Auto- Cleansing consists of the base network model and constant a. The constant O: captures the effect of the incorrect label for B. Thus, removing mitigates the overfitting effect due to the incorrect la- bel. (Right) (1) Training data has correct and incorrect labels. (2) Constructing the cleansing model consists of base network model m(x) and sample-category specific constant a. Learning with the training data using cleansing, (3) Deleting the constant å, and (4) Testing with the validation data using the cleansed network model.
As shown in the section of the theoretical analysis, the proposed AutoCleansing can address the pre- diction errors due to the incorrect labels within the category set, outside the category set, and with multiple objects. AutoCleansing tion method. For example, the experimental section presents the estimation results of AutoCleansing with base models of ResNet (He et al. 2016), WideResNet (Zagoruyko & Komodaki 2016), Shake-
any network model the base model with any augmenta-
2

---

Under review as a conference paper at ICLR 2021
Shake (Gastaldi 2017). and PyramidNet- SraamidNet-SSakeDroop (Yamada et al. 2018) using AutoAugment (Cubuk et al. 2018). Furthermore, AutoCleansing does not require iterative runs to identify the incorrect labels because the effects of the incorrect labels are automatically captured in a single run by the sample-category specific constant.
The contribution of this study can be summarized follows:
It provides theoretical model for AutoCleansing. The incorrect labels in the training data cause a prediction error. The proposed method can capture the biased effects of incorrect labels automatically and address the problem of prediction error due to incorrect labels.
The proposed method can be implemented with any network model or augmentation method. This study considers experiments of AutoCleansing with ResNet, NeteResNet, Shake-Shake, and SrrramddNet+-SakeDDin using AutoAugment.
Experimental results show that the proposed AutoCleansing method can improve the vali- dation accuracy for the CIFAR-10, 10, CIFAR-100, SVHN, and ImageNet datasets.
The additional training cost of AutoCleansing relative to the base network model can be negligible. AutoCleansing can remove the biased effect of incorrect labels automatically in a single run of learning. For example, the additional learning time of AutoCleansing with CIFAR-10/100 datasets is only 0.5 % of that of the base network models.
2 RELATED WORKS
There are several studies noisy datasets in the literature machine learning. Frnay & Verleysen (014)) provide a comprehensive review of label noise classification. Algan & Ulusoy (2019) provide a complete overview of deep learning with noisy datasets. There are three approaches to dealing with mislabeled datasets as follows: (1) robust learning with label noise, (2) identification of mislabeled data, and (3) utilization of small dataset without incorrect labels.
For robust learning with label noise, after the early works of Reed et al. (2015) and Azadi et al. 2015), several algorithms using deep neural networks have been proposed including the S-model Goldberger & Ren-Reeuen] 2016). MentorNet (Jiang et al. 2018), decoupling (Malach & Shalev-
Shwartz 2017), F-correction (Patrini al.] (017), Open-set (Wang et al.; 20188) Bi-level-model (Jenni & Favaro 2018). Lq (Zhang & Sabuncu 2018), co-teaching (Han et al. (018). random reweighting (Ren et al. 20188) joint optimization (Tnaka et al. 20188). DAC (Thulasidasan al., 2019), SELF (Nguyen et al. (0020), dynamic bootstrapping (Arazo et al. 2019). and DivideMix (Li al. 2020). Goldberger & Ben-Reuver (2016) and Patrini et (2017) estimated a noise transition matrix to correct for the loss function. However, it was difficult to correctly estimate the transition matrix. Jiang al. (2018) and et al. (2018) proposed weighted samples to adapt the noisy sam- ples. However, estimation of correctly weighted samples was also challenging. Arazo et al. (2019) proposed a beta mixture model of the cross-entropy loss of each sample and modeled the label noise. Their approach showed outstanding performance for high-level noise. For linear regression, consistent robust regression was proposed for the corrupted data (Bhatia et al. 2017). However, the consistency of robust learning method of nonlinear estimation using the classification model was not clear. These studies of robust learning with label noise used datasets having synthetic-label noise added. Label noise is generated by replacing one label with another category set. These studies showed good performance for artificially generated label noise. How- ever, most did not find mislabeled samples in real-world datasets. On the other hand, the approach proposed in the this study considers incorrect labels within and outside the category set as well as multiple objects in real-world datasets.
given probability within
For the identification of mislabeled data, some studies found incorrect labels in famous datasets for deep learning. For instance, [Fkambaram et al. (2017) found 92 mislabeled examples in 18 of the image classes, after reviewing more than 15 % of the images in the ImageNet dataset. [AL] Rawi & Karatzasu (2018) reported incorrect samples on CIFARIO and 15 mislabeled samples on CIFAR100, Müller & Markert 2019) reported 4 mislabeled samples on MNIST, 7 samples on CIFAR100, and 64 samples on -NIIST,-MMIITT.. Pleiss et al. (2020) identified incorrect labels using the area under the margin (AUM) statistic. They showed that incorrect labels are 3 % on CIFARIO, 13 % on CIFAR100, and 24 % on Tiny ImageNet. On CIFAR100, they reported that removing 13 % of the data leads to 1.2 % drop in error. Identifying mislabeled data requires some criteria
3

---

Under review as a conference paper at ICLR 2021
to determine whether it is correct incorrect. Several runs of learning may be required to search for optimal criteria for the identification of incorrect labels. The method proposed in this study, in contrast, allows capture and remove the effects of incorrect labels in single run of learning.
For the utilization of small dataset without incorrect labels, it is assumed that we have a small set of clean data, namely, free of mislaheled samples. (Sukhbaatar et al. (2015) and Hendrycks et al. (2018) used clean data to estimate the noise-transition matrix for incorrect labels. Other studies on the utilization of clean data include Ren al. (2018), . [i al. (2017), and Zhang al. (2019b). However, it is difficult to find a small set of clean data in real-world datasets. Meanwhile, the method proposed in this study does not require clean data.
AUTOCLEANSING
Consider the classification task with K categories for the N training data points X (21,22,-.1 .. ,2N and labels Y = 13/,,92. yN J. Let the learning network model be, M (z, 0) {m1 (2. ...... m (z. })} that assigns an input x to an output m with a given parameter 8. The predicted probability of output y, given the input using this model, is assumed to be calculated by the following softmax function:
e
P Pyy= i|z)
Σ JEX
em
where m, = (2, 8) denotes the ith element of output and K (1...., KK} the category set.
If the training data has mislabeled samples, the estimated parameter might be biased, which could cause a prediction error. To address the problem of incorrect labels, consider the cleansing network model, C' (xr, 8) M (x, () + a, where a E R.NK x is parameter of the sample-category specific constant. Let ar be the constant for input of the sample and category k. Therefore, the prediction probability pC with the constant can be expressed follows:
pC (y -(x)
Σ JEK
tm
If the effect of the mislabeled samples is captured by the catple--attgor--speeiiii constant a of the cleansing network model (I, 0, ax) M (x. a, the base network model M (v, 0) could avoid the biased problem due to incorrect labels. The learning process with AutoCleansing is as follows: (1) Learning with the training data using a cleansing network model C (ax 8 a)) M (r, 0) + a, (2) deleting the sample-cateoor--speiiff constant â estimated in the learning process, and (3) testing with the validation data using cleansed network model, C(x, ((), á, where â denotes the estimated parameter of the base model.
3.1 THEORETICAL ANALYSIS OF AUTOCLEANSING
To investigate the performance of the proposed AutoCleansing for mislabeled data, we consider the following definition of a general case:
Definition (Incorrect labels and outside of the category set). Let K" be a full set of true categories and K. C K' be the category set of the model. Let m m y²x" x)) be the probability such that the input x with the true category y" E K" has label ÿ E K. Let P (y" x NrK,,, "K*) be the true probability of the category y" from the category probability of category jEK for the input x with incorrect labels within and outside of the category set is defined as follows:
IK* with input x given true parameter 0*. Thus, the observed
Q (0'x,K',0')
Σ DEKK"
[T (0lp" x) PV 12,'
1x.KK,07)
This definition includes incorrect labels, outside the category set, and multiple objects. If = K, this definition is equivalent to that within the category set. If incorrect labels occur outside the category set, namely æ (ğ ,) = O&ğ # y" E K and я (gly". x) 2 O'y K" K, defines the observed probability outside the category set. Let S be the combination of categories in multiple
4

---

Under review as a conference paper at ICLR 2021
objects (for example, DEER and PERSON) and K+ = KUS be the true category set including the original categories and combination of categories in multiple objects. If If = K", definition describes multiple objects.
If the sample data has incorrect labels within the category set, incorrect labels outside of the category set, or multiple objects, the estimated parameter using the minimum loss function does not converge to the true parameter: thus, the incorrect labels cause the prediction error. However, the following theorem shows that AutoCleansing can address the biased estimation due to mislabeled data:
Theorem 1 (AutoCleansing for incorrect labels within and outside the category set). Let IK* be
full set of true categories and K. be the category sel of the model. Let a (yly" 2)) be probability such that the input x with the true category y² E 'K' has label ÿ E K. Assume that the sample has incorrect labels and the outside of the category set defined as Definition 1. Let L be the expected loss function of AutoCleansing and 6+ be the set of the solution to 0L /00 = 0. Furthermore assume that the model is correctly constructed and the probability distribution of the output the softmax function. Then the gradient of the expected loss function with AutoCleansing is zero at the true purameter value 0', Namely, 0* E0+
The proof can be found in the Appendix[A.. A.1 Note that, although this theorem states that the stochas- tic gradient descent using the loss function of the correct model with AutoCleansing can be stopped at true values, it does not guarantee that minimization of the loss function will converge to the true value, if the loss function has more than one local minimum. Theorem shows that the implemen- tation of the sample-category specific constant can capture the biased effect of incorrect labels. This suggests that the value of a may reflect the effects of incorrect labels. The following theorem confirms this:
Theorem 2 (Sample-categor specific constants and incorrect labels). Assume that the sample has incorrect labels and outside of the calegory set defined as Definition 1. Consider that the true label t is assigned a false label f. Let ân and a be parameters of the sumple-category-speiiii of the true and false labels, respectively, which are estimated by the minimum loss function with AutoCleansing. Assume that the probability of observing a false label is greater than or equal to that of the true label, namely (f x. K" .0)) 2 (t) x, ,,,6"',,, ( Furthermore assume that the model is correctly constructed, the probability distribution of the output is the softmax function, and the loss function is minimized at true parumeter values. Then, as N oc, the estimated parameter of the sumple-categon specific constants â using the minimum loss function with AutoCleansing has the following properties:
constants
1. General case: The sumple-category-spette constants of the true label are equal to or less than that of the false label: ôg S ā.j-
2. Symmetric case: Assume x (f(t, x)) is symmetric and independent of x, namely,
x (f)t,z)= = (f(t) π (t) For this case, the sample-category specific constants of the true label are equal to or less than zero: dy S O.
3. Single symmetric case: Assume π (J(I,, x) is symmetric, independent of x, and incorrect labels occur between t and f only. Consequently, æ xf)t, = π (ftt)= (t) and
x (jlj,z) 1, vj # t. f. For this case, the sample-category-specfff constants of labels except the true and false labels are equal to zero: àj = 0, vj + f.
The Appendix A.2 provides the proof of this theorem. Table I provides numerical examples of the theorem. Assume that the category set has three categories K 3.). Example (A) is the incorrect label within the category set. Assume the true model output is {mi, my, mī} {0.1, 0.8). Namely, the correct category Category 3. having the highest output value. If the first label is observed, the incorrect label causes biased learning: {c), c22 c3] = {0.8, 0.1,0.1,, The estimated
constants are assumed to be optimal such that a = m² - c. The constant of the true label (0' -0.7) is less than that of the false label (a) 0.7), such that the output value of the biased model (c = m + a) of the observed category (0.8) is larger than the one of the true category (0.1). Example (B) ) is the incorrect label outside the category set. Consider
that the true category set is K- = {1, 2, 3, 4}. Assume the true category is Category 4 outside of the observed category set. The optimal constant of observed label (a3 = 0.7) has the largest value. Note that the constant of true category (a4 = -0.7) is not estimated, because this category is the outside of category set. Example (C) is the multiple objects. Assume the observed label is 1, whereas the
5

---

Under review as a conference paper at ICLR 2021
true categories are 1 and 3. The constant of unobserved true category (as -0.7) is less than that of the observed true label (a1 = 0.0),
Table I: Neumerical example of AutoCleansing. Obs. is the observed label and True is the true category. Outside is the outside of the category set. is the output of biased model, m" is the output of true model, and a is the biased effects estimated by AutoClcansing.
(A) Incorrect label
(B) Outside of category
(C) Multiple objects
Label
2
3
2
3
(outside)
Obs.
True
Obs.
True
Obs. and True
True
mn*
0.1
0.1
0.8
0.1
0.1
0.1
0.7
0.8
0.1
0.8
c
0.8
0.1
0.1
0.8
0.1
0.1
0.0
0.8
0.1
0.1
Ch
0.7
0.0
-0.7
0.7
0.0
0.0
-0.7
0.0
0.0
-0.7
4 EXPERIMENTS
This section provides the experiments investigating the performance of the proposed AutoCleansing on the CIFAR-10 and CIFAR- 100 (Krizhevsky & Hinton 2009), SVHN (Netzer et al. 2011), and ImageNet (Russakovsky et al.) 2015) datasets. The proposed AutoCleansing method requires base network models. In these experiments, the base network models are Wide-ResNet 40-2 and Wide- ResNet 28-10 (Zagoruyko & Komodaki 2016). Shake-Shake 26 2x32d, 26 2x96d and 26 2x112d [astaldii 2017), and PyramidNet SSakke-r (Yamada et al.] 2018). All hyperparameters of base network models are the same as those used in AutoAugment (Cubuk et al. 2018), FugmnntoAugment (Lim et al. 2019), and PBA (Ho et al. 2019 L A cosine learning decay with one annealing cycle was applied all models except ResNet. For AutoCleansing. the sample-categon specific con- stants could not converge without regularization. The weight decay of the sample-category specific constant is x10 except for PyramidNet, which uses x10-5
AutoCleansing has learning parameters of the sample-category specific constant that consists of variables for each sample. Note that K variables cannot be identified; therefore, the constants for the first category are set zero for all samples (a1 0). Thus, the estimated sample-category specific constant a E IR X has N (K- - 1) estimable parameters. constant of N(K- 1) parameters is called as AC1. However,
this study, the AutoCleansing
with the sample-category-sseeiii might be difficult to estimate all parameters of a for large datasets. For example, ImageNet has more than 1.2 million images with 1,000 categories for training data. Therefore, AC1 needs to estimate more than 1.2 billion parameters. Instead, consider the sample specific constant a E RV such that all categories except the observed label set to zero for all samples (a, = 0 Vjj + ÿ); that is, has N estimable parameters. The AutoCleansing with the sample-specific constant of N parameters called as AC2. Note that AC2 corresponds to the special case of the single symmetric case of Theorem 2. For the single symmetric case, Onj = 0 Vn, Vj * t, f. If the true category belongs to the outside of the category set, we cannot estimate the ant of the true category, therefore, all categories except the observed label have zero values of anj.
The experiments compare the results with baseline preprocessing, Cutout (DeVries Taylor 2017). AutoAAument (AA). FastAutoAAgmenn (FAA), and Population Based Augmentation (PBA). The baseline preprocessing is conventional augmentation as follows: standardizing the data, horizontal flipping with 50 % probability, zero- pero-padding, and randomly cropping. In the proposed AutoCleans- ing. we follow the procedure of AutoAugment, which first applics the bascline preprocessing, then applies the AutoAugment policy, and finally applies the Cutout.
4.1 EXPERIMENTAL RESULTS
The CIFAR-10 10 dataset has a total of 60,000 images, including 50,000 for training set and 10,000 for test sets. The number of categories was 10. Thus, the sample-category specific constant for AC1 has 0.45 million estimable parameters, whereas the sample specific constant for AC2 has 0.05 million
See Table 7 in Appendix

---

Under review as a conference paper at ICLR 2021
estimable parameters. Table 2 shows the results of the test accuracy for different network models using the CIFAR- 10 dataset. For all models, the proposed AutoCleansing with AutoAugment can achieve better performance compared to previous model
Table 2: Test accuracy (%) CIFAR-10. ACI+AA the results of the proposed AutoCleansing with sample-category specific constants and AutoAugment. All experiments in this study replicate the results of Baseline, Cutout, and AutoAugmen methods from Cubuk et al. (2018), FAA from Lim et al. (2019), and PBA from Ho et al. (2019). Averages of five runs are reported.
Baseline
Cutout
AA
FAA
PBA
ACI+AA
Wide-ResNet-40-22 Wide-ResNet-28-10
94.70
95.90
96.30
96.30
96.56
#0.11
96.13
96.92
97.32
97.30
97.42
97.53
#0.08
Shake-Shake (26 2x32d)
96.45
96.98
97.53
97.50
97.46
97.60
+0.07
Shake-Shake (26 2x96d)
97.14
97.44
98.01
98.00
97.97
98.12
#0.11
Shake-Shake (26 2x 12d)
97.18
97.43
98.11
98.10
97.97
98.27
#0.05
PyramidNett-haakDDropp
97.33
97.69
98.52
98.30
98.54
98.59
#0.05
Table 3: Test accuracy (%) on CIFAR-100. Averages of five runs are reported.
Baseline
Cutout
AA
FAA
PBA
ACI+AA
40se-RReNNe-40-2
74.00
74.80
79.30
79.40
79.94
+0.20
Mode-ResNNet-2-111
81.20
81.59
82.91
82.70
83.27
84.07
10.18
Shake-Shake (26 2x96d)
82.95
84.00
85.72
85.40
84.69
86.19
#0.12
PyramidNet--SakeeDrop
86.01
87.81
89.33
88.30
89.06
89.40
#0.11
Table 4: Test accuracy (%) on SVHN
Baseline
Cutout
AA
FAA
PBA
ACI+AA
Wide-ResNett-8--1
98.50
98.70
98.93
98.90
98.82
98.96
The CIFAR-100 dataset also has total of 60,000 images, including 50,000 for training set and 10,000 for test sets. The number of categories was 100. Thus, the sample-category specific constant for AC1 has 4.95 million estimable parameters, whereas the sample specific constant for AC2 has 0.05 million estimable parameters. Table provides the results for the CIFAR-100 dataset. Simi- larly, for CIFAR-10, the proposed model has better accuracy than previous models.
The SVHN dataset has 73,257 digit images for the core training set, 531,131 for the additional training set, and 26,032 for the test set. In this experiment, both core and additional training sets were used. The number of categories was 10. Thus, the sample-categon ACI has 5.44 million estimable parameters, whereas the sample specific constant for AC2 has 0.53 million estimable parameters. Table reports the results for the SVHN dataset. For the SVHN dataset, the proposed model has better accuracy than previous models.
specific constant for
The ImageNet dataset has more than 1.2 million images for the training set and 0.15 million images for the validation and test sets. The number of categories was 1,000. Thus, the sample-category specific constant for AC1 has more than 1.28 billion parameters that may not be feasible esti- mate. Therefore, this experiment uses the sample specific constant for AC2, which has 1.28 million estimable parameters. Table shows the results for the ImageNet dataset. The proposed model of AC2 has better Top1 accuracy than previous models, whereas the Top5 accuracy of AC2 is less than that of AA and FAA.
Pleiss et al. (2020) proposed the the area under the margin (AUM) statistic for robust learning with label noise. They provided experiments of label noise using real-world datasets and the artificially generated noise. Their experimental results showed that the AUM had better performance than
TTable 8 in Appendix provides a comparison between AC1 and AC2. Note that the test accuracy of AC1+AA is close that of AC2+AA

---

Under review as a conference paper at ICLR 2021
Table 5: Top Top test accuracy (%) on ImageNet
Baseline
AA
FAA
AC2+AA
ResNet-50 75.30 / 92.20 77.63 93.82
77.60 93.70
77.71 / /93.58
Table AutoCleansing and Area Under the Margin (AUM), Base network model ResNet32. This table replicates the results of Baseline and AUM from Pleiss et al. (2020). Averages of five runs are reported.
Baseline
AUM
AC1
CIFAR-10
91.9
92.1
92.75
#0.06
CIFAR-100
67.0
68.2
69.13
#0.24
those the previous studies for datasets having synthetic label noise added. Table shows the test accuracy of AutoCleansing and AUM using the ResNet 32 model. All hyperparameters of the base network models were the same as those used by Pleiss et al. (2020). AutoCleansing demonstrates
outperforming the AUM on both CIFAR-I0 and CIFAR-I00.
4.2 DETECTION OF INCORRECT LABELS USING AUTOCLEANSING
As shown in the theoretical analysis section, AutoCleansing can capture the effect of incorrect labels using the sample-categon specific constant a. If there is no mislabeled sample, a + 0, as N + co. large value of |a| indicates the existence of mislabeled samples in the data. Therefore, might be possible to identify incorrect labels using the estimated sample-categon category specific constant &
Figure in Appendix show the sample-category specific constants for the example of mis- labeled images in CIFAR-10 and CIFAR-100. 'Tee estimation model is Wide-ResNet 40-2 with AutoCleansing. Because Ô1 fixed zero, these figures show the standardized values of âg Meanfāi, áxk. Let a be the maximum value and a be the minimum value of the standardized caaple-categoyy-seeiife constants for nth data. MaxRank is the percentile rank of sorted a:Maz in descending order, and MinRank is sorted MMMn in ascending order.
#1 in Figure 4 is an example of incorrect labels within the category set. The original label of #1 is DOG and alternative to this image is CAT. The estimated of original label of #1 is 0.233, whereas á of an alternative label of 1 is -0.227. Similarly, the original labels are positive â, whereas alternative labels are negative à for images #2-#5 in this figure.
#6 in F an example of incorrect labels outside the category set. The original label of #6 TRUCK, whereas the correct label might be PERSON, which does belong to the category set for the CIFAR-10 dataset. For this image, á of the original labels have positive. For these examples of #6-#10, the estimated à of the original labels are positive.
#11 in F an example of multiple objects. The original label of #11 is DEER; however, this image includes an additional object of PERSON that does not belong the category of the CIFAR-I0 10 dataset. Images of # #15 are examples of multiple objects in the CIFAR-100 datasets. For example, the original label of #12 is PLANE; however, this image also has the object of SEA. Both PLANE and SEA belong to the category set. For these images, á of the original labels have positive, å and additional labels have negative labels.
Note that the values of MaxRank MinRank are less than 0.1 % for all images in these figures. This suggests that the mislaheled samples can identified using the high or low value of the sample- category-spee -spocifcc constants & estimated by AutoCleansing. To identify the mislabeled samples, must specify the threshold criteria between the correct and incorrect labels. Let 7 be the percentage of mislabeled samples in the dataset. Algorithm in Appendix A.4 provides the procedure for searching for mislabeled samples given r using AutoCleansing.
remove the mislabeled data from the sample and run the learning models using the trimmed sample. Figure F shows the test accuracies of the base model using trimmed data with different threshold criteria of incorrect labels the CIFAR-10
After searching for the mislabeled samples,
8

---

Under review as a conference paper at ICLR 2021
and CIFAR-100 datasets. The base model is Wide-ResNet 40-2 model with AutoAuumentation. As can be observed, the test accuracies are highest when 0.2 of incorrect labels are dropped in both datasets. This figure shows that dropping mislabeled samples improve the classification accuracy. However, is necessary to repeat learning several times with different criteria determine the optimum drop of mislaheled samples. If an excessively small sample is dropped, the effects of bias due to incorrect labels might remain. If excess samples are removed, the estimation efficiency could be reduced because of the decreasing sample size.
appropriate drop rate
Notably, the maximum test accuracies using trimmed data are very close to the accuracies of Auto- Cleansing in this figure. This suggests that AutoCleansing can remove the biased effects of incorrect labels without dropping the mislabeled samples from the datasets. Furthermore, AutoCleansing does not need to repeat learning because it does not require the threshold criteria of the drop rate for the mislabeled samples. Instead of dropping the mislabeled samples that requires the threshold crite- ria of incorrect labels, AutoCleansig drops the sample-category specific constantsby capturing the mislabeled bias.
CIFAR-10
CIFAR-100
96.60
80.00
96.55
79.90
79.80
96.50
test accuracy
fest accuracy
79.70
96.45
79.60
96.40
79.50
96.35
79.40
AutoCleansing
AutoCleansing
96.30
79.30
Trimmed Data
Trimmed Data
96.25
79.20
0.0%
0.2%
0.4%
0.6%
0.8%
1.0%
0.0%
0.2%
0.4%
0.6%
0.8%
1.0%
percent of mislabeled samples
percent of mislabeled samples
Figure 3: Test accuracies of the base model with trimmed data and AutoCleansing. Averages of five runs are reported.
CONCLUSION
This study introduces AutoCleansing to address the biased problem due to incorrect labels. The proposed method is appealing in that it can automatically capture the effect of incorrect labels and mitigate it without removing mislabeled samples. As shown in the theoretical analysis, if the model is correctly constructed, the gradient of the expected loss function of AutoCleansing is equal to zero true parameter values using mislabeled samples for incorrect labels within outside the category set as well as multiple objects. Furthermore, AutoCleansing can be implemented with any network model and any augmentation method. Experimental results show that the proposed AutoCleans- ing has better performance than previous studies on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. Additional topics for future investigation into AutoCleansing include applications to artifi- cial label noise (Algan & Ulusoy 2019), the use of other network models such as EfficientNet (Tan 2019), and the use of recent augmentation methods such as adversarial AutoAugment (Zhang et al. 2019a).
REFERENCES
Mohammed Al-Rawi and Dimosthenis Karatzasu. On the labeling correctness in computer vision datasets. In Proceedings of the Workshop Interactive Adaptive Learning, pp. 1-23, 2018.

---

Under review as a conference paper at ICLR 2021
Grkem Algan and Iikay Ulusoy. Image classification with deep learning in the presence of noisy labels: A survey, arXiv preprint, arXiv:1912.05770,222 2019.
Takeshi Amemiya. Advanced Econometrics. Harvard University Press, 1985.
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, pp. 312-321, 2019.
Samaneh Azadi, Jiashi Feng, Stefanie Jegelka, and Trevor Darrell. Auxiliary image regularization for deep cnns with noisy labels. arXiv preprint, arXiv:/5/ 1.07069, 2015.
Kush Bhatia, Prateck Jain, Parameswaran Kamalaruban, and Purushottam Kar. Consistent robust regression. In NeurIPS, pp. 2110-2119, 2017.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2018.
Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint, arXiv:
2017.
Rajmadhan Ekambaram, Dmitry B. Goldgof, and Lawrence O. Hall. Finding label noise examples in large scale datasets. 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2424,, 2017.
Benot Frnay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5):845-869, 2014.
Xavier Gastaldi. Shake-shake regularization. In CoRR, abs/l 705.07485, 2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2016.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NewrlPS, pp. 4536-8546, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NeurIPS, pp. 10477-10486, 201 88
Daniel Ho, Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. In JCML, 2019.
Simon Jenni and Paolo Favaro. Deep bilevel learning. In ECCV, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. ICMI, pp. 2309-2318. 2018.
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, pp. 1928-1929 1936, 2017.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NewrIPS, 2019.
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In NIPS, pp. 960-970, 2017.
10

---

Under review as a conference paper at ICLR 2021
Nicolas Michael Müller and Karla Markert. Identifying mislabeled instances in classification datasets. In IJCN,, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling, In ICLR,, 2020.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the area under the margin ranking. arXiv preprint, arXiv:2001.10528, 2020.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 2015.
Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331-4340, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. In IJCV, 2015.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. In ICLR, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint, 19Xiv:190..11946. 2019.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, pp. 5552-5560, 2018
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In ICMI, pp. 6234 6243, 2019.
Yisen Wang, Weiyang Liu, Xingiun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pp. 8688 8696, 2018.
Yoshihiro Yamada, Masakazu Iwamura, and Koichi Kisei. Shakedrop regularization. In CORR, abs/1802.02375, 2018.
Sergey Zagoruyko and Nikos Komodaki. Wide residual networks. British Machine Vision Confer- ence, 2016.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. In ICLR, 2019a.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, pp. 8778-8788, 2018.
Zizhao Zhang, Han Zhang, Sercan O. Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. arXiv preprint, 11Xiv:1910.00701, 2019b.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 PROOF OF THEOREM 1
Consider following loss function:
1 N
(()) Σ
[ym = In P (Un =i 0)
KEK
where 1/yx = i] is an indicator function equal to one when y,, = i and zero otherwise. Let L (0) be the probability limit of the loss function & (0). Consider that the sample has incorrect labels and the outside the category set defined as definition 1. If the model is correctly constructed, by the strong law of large numbers, N co, I. (0) is follows:
C (0) 0.3, L (0) (x)
Σ
[i|,, 0*) In (2 0)
OEK
-EEppp(X Σ
Σ
P ((k x))
11, 0*)] In P 2 N,
ieK LAEK*
The derivative of L (8) is as follows:
( -Exxp(X)
Σ
x(ikk, x) P (k x, ,0) OP (i a. K. 0)
00
Pi x. K.
00
EER
AEK-
If the distribution function is the softmax function follows:
the derivative of L (0) can be expressed as
Σ
E
OL (0)
JEK
(i)kk,)) emi] .m)) KEK
OP @ x K.
00
em
00
EKK
where m = m& (I, 0*) denotes the kth element of the output for the base model given the true pa-
rameter and the true category set. First, consider the case of incorrect label, namely, (ili, x)) for all E K and K = K" For this case, the derivative of L (0) is equal to zero at the true parameter f 8", as follows:
Σ E
AT.
Expp(X)
Σ iEK
JEK
OP
(1 x. K, 0*)
08
Σ em
0A
JEK'
Σ m
Σ
P
K, 8r)
= -xExxp(X)
JEK
iEK
0
Σ
00
jEK-
Note that Σ P x, = 1. From the assumption of identification for the parameter, L (0) * iEK L(O*) for all 0 + 0r Thus, if there is no incorrect label and the model is constructed correctly, the estimated parameter á using the minimum loss function is consistent as follows: 0" as N - 00. This is well-known property of consistency in the maximum likelihood estimator of the logit model (Amemiya 1985).
However, in general, the derivative of L (0) is not always equal to zero at the true parameter value if the sample has an incorrect label. If there are mislabeled samples, the estimated parameter 0 using the minimum loss function may not converge to the true value.
Consider the estimation using the AutoCleansing model, çj (x. θ) = m, 0) + aj where θ = {0, a.. The probability limit of the loss function for the AutoCleansing model is as follows:
C (0) -Ex
Σ TEK
(ilk,,)) ((k xr K", 8²)] In PC 21 K, +)
LAEK
12

---

Under review as a conference paper at ICLR 2021
Assume that the probability distribution is softmax function 2 Let 0 be the set of the solution to O /00 = The derivative of L (+) can be expressed as follows:
Σ
e
taj
Σ REK
x)
aL (0) -xx YEK
dpc (i| K, 9)
E
JEK
00
Σ
e
e
00
jeK
jEK
emito; apc
(i| KK +))
--Eupp(X)
Σ m
e
00
iEK
jeK
Thus, the derivative of L (9) is equal to zero at where aj in Σ (ilk; x)) c
m.
AKEK
-¹ {0", .'). Namely, H" a+
A.2 PROOF OF THEOREM 2
From the assumption, Q (f (f 2 (t|22, K") If the model is correctly constructed, the output of the model for the true category is higher than that of the other category. Therefore, m, 2 my, Vj. For the general case, from Theorem I, the difference between the sample-category- specific constants of the false label and that of the true label is follows:
a - 0z =nn Σ (flk,z)) e -m NEEK"
In Σ KEEK
(1),, ()k,,.)-e)) - mi
Σ x (flkiz)
kgk
Σ
= In
JEK
m
Σ x)) KEK
SCK-
Q 2 K"
= in Q x. Kr, 0*)
+ 0.
For the symmetric case, (t/ks, 2²) = (klt, x)) for all & Therefore, the sample-category-ppciii stants of the true label are as follows:
a = Im Σ
x (t)k, (ilk,zz)-0 e -m-=In In
Σ
()) m² m
KEKK
KEEK
S <ln
Σ a (k]t, x) . - 0.
kEK-
For the single-symmetric case, x =11))=11) # ftt. Therefore, the sample-category-speiff con- stants of category + f. are as follows:
= in Σ x x))k,x) m²
-mj=In Σ
(klj,.) =² m.
MEK
kER
13

---

Under review as a conference paper at ICLR 2021
A.3 ADDITIONAL TABLES
Table 7: Hyperparameters for the Experiment. LR represents the learning rate, whereas WD rep-
resents the weight decay. Multi steps schedule decays the learning rate by 10-fold at epochs (150, 225) for CIFAR and (90, 180, 240) for ImageNet.
Dataset
Model
LR
LR Schedule
WD
Batch Size
Epoch
CIFAR-10
ResNet-32
0.1
multi steps
0.0001
256
300
CIFAR-10
Wide-ResNett40-2
0.1
cosine
0.0002
128
200
CIFAR-10
0.1
cosine
0.0005
128
200
CIFAR-10
Shake (26 2x32d)
0.01
cosine
0.001
128
1800
CIFAR-10
Shake-Shake (26 2x96d)
0.01
cosine
0.001
128
1800
CIFAR-10
Shake-Shake (26 2x1 12d)
0.01
cosine
0.001
128
1800
CIFAR-10
PyramidNet++ShakeDDog
0.05
cosine
5.00E-05
64
1800
CIFAR-100
ResNet-32
0.1
multi steps
0.0001
256
300
CIFAR-100
Wide-ResNet-28--
0.1
cosine
0.0005
128
200
CIFAR-100
(26 2x96d)
0.01
cosine
0.0025
128
1800
CIFAR-100 100
PyramidNet++hakkeDrpp
0.025
cosine
0.0005
64
1800
SVHN
MiteeRRssNet-255
0.005
cosine
0.001
128
200
ImageNet
Resnet-50
0.1
multi steps
0.0001
256
270
Table 8: AutoCleansing with the sample-category-speiif constant (ACI) and the sample specific constant (AC2). Averages of five runs are reported.
AA
AC1+AA
AC2+AA
CIFAR-10
Wide-ResNet-40-22
96.30
96.56
+0.11
96.54
#0.11
Wide-ResNet-288
97.32
97.53
+0.08
97.48
+0.06
CIFAR-100
40---422
79.30
79.94
+0.20
79.86
10.19
Wiet-ResNet-22-10
82.91
84.07
+0.18
84.12
#0.17
14

---

Under review as a conference paper at ICLR 2021
A.4 ALGORIHHMS
Algorithm 1 AutoCleansing
Require: Initial parameter a E R of the sample-category specific constant. N: The number of samples. K: The number of categories. B: The number of minibatches. Learning:
for l..... do
Sample a minibatch of 2(0) from training data set X. Let d(b) be a vector of index of the sampling such that (b)) X[d
Compute gradient using the cleansing model for the minibatch C(x(0) a) M(x(), 0)+
(h)), where (+) a afd N.* is the constant for the minibatch and N₂ is the batch size.
Update the parameter = {0.a).
end for
Testing: Predict the category using the cleansed model M (xx 0) C(x, 0, a) a
Algorithm 2 Detection of mislaheled samples using AutoCleansing
Require: & E RNNK the sample-category specific constants nth sample and kth category, N: the number of samples, K: the number of categories, and т: the percent of mislabeled samples.
Require: Standardize ånk ânk Mean(āni,""" _nx) } for all (1, N). Require: Set ā Mar Max {ān1. ". &nnk) } and ā Min Min {āx1, _ } for all n
{1.-.. ...
Require: Set Timiss = I: mislabeled list. Nmiss T N/110: the number of mislabeled samples.
while the size of Lmiis Nisss do
Mar = Max(á Delete á from Max, Add the Find the sample index n such as á sample index
the mislabeled list Liiis Liniss
if the size Lmiss Nisse do
Find the sample index such 6,Min Min(a Delete Mini Min from & Add the sample index n to the mislabeled list Liniss if Liniss
end if
end while
Return: the mislabeled list Linies
15

---

Under review as a conference paper at ICLR 2021
A.5
ADDITIONAL FIGURES
Origianl labels af
Alternative labels at
Dataset
Image
MaxRank
MinRank
ID
CIFAR-10 10
dog
cat
10
#1
2.002%
0.028%
38775
0.233
-0.227
CIFAR-10
cat
dog
10
#2
0.438%
0.020%
25684
0.274
-0.235
10
CIFAR-100
leopard
tiger
#3
0.020%
0.002%
10178
0.337
-0.300
CIFAR-100
tulip
rose
10
#4
1.328%
0.038%
7019
0.312
-0.240
20
CIFAR-100
cloud
can
10
#5
0.072%
0.268%
24900
0.334
-0.182
Figure 4: Example images of incorrect labels within category set and sample-category specific con- stants a. MaxRank is the percentile rank of sorted á Maz in descending order, and MinRank is sorted Min in ascending order. See text for more details.
16

---

Under review as a conference paper at ICLR 2021
Origianl labels of
Alternative labels
Dataset
Image
MaxRank
MinRank
ID
CIFAR-10 10
truck
person
10
#6
0.002%
1.242%
20
18310
0.332
30
CIFAR-100
lion
giraffe
10-
#7
0.044%
39.838%
19011
0.335
CIFAR-100
plate
cutlery
#8
0.054%
13.646%
1578
0.335
CIFAR-100
cup
glass
10
#9
0.016%
43.730%
29783
0.338
CIFAR-100
motorcycle
raincoat
#10
0.002%
7.970%
48760
0.345
Figure 5: Example images of incorrect labels outside category set and sample-category specific constants a.
17

---

Under review as a conference paper at ICLR 2021
Origianl Alternative labels af
Dataset
labels at
Image
MaxRank
MinRank
ID
CIFAR-10
deer
person
10
#11
0.024%
0.734%
20
3879
0.317
CIFAR-100
plain
#12
0.112%
0.006%
16305
0.332
-0.281
10
20
CIFAR-10
cat
dog
#13
1.356%
0.060%
9145
0.247
-0.212
CIFAR-100
bus
road
10
#14
0.450%
0.052%
33026
0.323
-0.231
CIFAR-100
house
road
10
#15
5.558%
0.088%
780
0.284
-0.218
Figure 6: Example images of incorrect labels with multiple objects and sample-categor, specific constants a.
18


Reviews:
[{"review_id": "tT4Lpcdqsyz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This study introduces AutoCleansing to address the biased problem due to incorrect labels. This framework can automatically capture the effect of incorrect labels and mitigate it without removing mislabeled samples. There is some improvement in performance, but not much difference.\n\nWhere to improve:\n\nThe challenges of solving the problem of incorrect labels must be explained in-depth, but they are not introduced at all.\nSection 2 leaves me with an idea of incompleteness, and I would request the authors to make it a critical review and not just a list of methods.\nSection 3 for the proposed method AutoCleansing. The method is primarily described through equations. The authors may want to consider adding more descriptive text (and possibly figures) that would make the paper more accessible to readers without and extensive mathematics background.\nthe section Results should be substantially expanded. The authors should be explicit about the difference between the model proposed here and the models implemented by previous studies and how their model works compared to other methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "niUrvrnrtNx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach for handling noisy labels in predictive models without removing them. The approach is based on a base network and a category dependent constant. At test time the prediction is done using the base network. The paper is well-written and the ideas are explained clearly. I have the following comments and questions on the empirical and theoretical aspects of the work:\n\nAs far as I understand, theoretical analysis is only applicable to AC1 model but the more practical approach which is applicable to datasets such as imagenet is AC2. What portion of your results can be extended to AC2? \n\nThe experiments on CIFAR-10, CIFAR-100 and SVHN do not show a statistically significant improvement over the baselines; specifically after considering the confidence interval for the AA results (from table 2 of the AutoAugment paper). Adding experiments with synthetic datasets with different levels of noise can be helpful in understanding the advantages of AC1/AC2 over other methods for handling noisy labels. \n\nExperiments in Section 4.2 are only trimming the incorrect training labels; ideally, you also want to remove the noisy labels from the test set too. However, as far as I understand it’s not straightforward to apply AC1 to the test set and trim the noisy labels. Is this correct?\n\nOther methods for learning with noisy labels such as the ones that are mentioned in the Section 2: ‘Related Works’ should be added in order to provide a better picture of the pros and cons of the method. In the current version of the paper, the emphasis in the experiments is on augmentation methods which may not be the best choice.  For instance, “Unsupervised Label Noise Modeling and Loss Correction” by Arazo et al. has a similar approach to AC1 as it also models each sample and doesn’t require a matrix based noise model. This could be a good candidate baseline. \n\nOverall, I like the simplicity of the approach but I believe with the current state of the paper, it’s hard to judge the value of the approach over other methods for learning with noisy labels. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BeTHTeZWNiC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper Summary\n\nIn this paper, the authors proposed to train high quality classifiers from datasets with some mislabels.\n\nFor this purpose, the authors considered adjusting the softmax prediction using an additional term $\\alpha$ as follows:\n$$\nP^C(i | x; \\theta) = \\frac{\\exp(m_i(x; \\theta) + \\alpha_i(x))}{\\sum_j \\exp(m_j(x; \\theta) + \\alpha_j(x))}\n$$\nwhere $m(\\cdot; \\theta)$ is a model and $\\theta$ is its parameter.\nThe authors claimed that, by adjusting $\\alpha$ through training, the trained model $m(\\cdot; \\hat{\\theta})$ with an optimal parameter $\\hat{\\theta}$ is asymptotically consistent with the model trained on a dataset with clean labels, i.e.,  the trained model without $\\alpha$ performs well on clean test data\n$$\nP(i | x; \\theta) =\\frac{\\exp(m_i(x; \\theta))}{\\sum_j \\exp(m_j(x; \\theta))}\n$$\nIn the proposed method, for the training set $D = \\\\{x_n, y_n \\\\}$, we first train the model by minimizing the following loss function:\n$$\n\\hat{\\theta}, \\hat{\\alpha} = \\arg\\min_{\\theta \\in \\Theta, \\alpha \\in \\mathbb{R}^{N \\times K}} -\\frac{1}{N} \\sum_{n=1}^N \\sum_{i=1}^K 1[y_n = i] \\log \\frac{\\exp(m_i(x_n; \\theta) + \\alpha_{ni})}{\\sum_j \\exp(m_j(x_n; \\theta) + \\alpha_{nj})}\n$$\nwhere $K$ is the number of classes. We then classify the new instance $x$ by $\\hat{y} =\\arg\\max_i m_i(x; \\hat{\\theta})$.\n\nIn Theorem 1, the authors claimed that the above estimator $\\hat{\\theta}$ converges to the *true* parameter $\\theta^*$.\n\n\n\n### Pros & Cons\n\n[Pros]\n\nThe experimental results indicate that the proposed method is effective on several datasets.\n\n[Cons]\n\nThe paper contains a serious flaw in its problem formulation and the subsequent theorems. The proposed problem formulation has a trivial solution which is completely useless. The effectiveness reported in the experiments seems to be just an artifact caused by the tunings of hyperparameters. See my comments in \"Quality\" below for the detail.\n\n\n\n### Quality\n\nThe paper contains a serious flaw in its problem formulation.\n\nRecall the training problem:\n$$\n\\hat{\\theta}, \\hat{\\alpha} = \\arg\\min_{\\theta \\in \\Theta, \\alpha \\in \\mathbb{R}^{N \\times K}} -\\frac{1}{N} \\sum_{n=1}^N \\sum_{i=1}^K 1[y_n = i] \\log \\frac{\\exp(m_i(x_n; \\theta) + \\alpha_{ni})}{\\sum_j \\exp(m_j(x_n; \\theta) + \\alpha_{nj})}\n$$\nThis problem has a trivial solution that $\\alpha_{n y_n} \\to +\\infty$ for $\\forall n$, which leads to\n$$\n\\frac{\\exp(m_i(x_n; \\theta) + \\alpha_{ni})}{\\sum_j \\exp(m_j(x_n; \\theta) + \\alpha_{nj})} \\to\n\\delta(y_n = i)\n$$\nwhere $\\delta(y_n = i) = 1$ if $y_n=i$ and 0 otherwise.\nNote that this trivial solution does not depend on the model $m(\\cdot; \\theta)$. Thus, any parameter $\\theta$ can be an optimal solution $\\hat{\\theta}$ as long as $m(\\cdot ;\\theta)$ is finite.\n\nThe above observation indicates that the proposed method do not work as expected if the training problem is solved appropriately. Thus, I conjecture that the good performances reported in the experiments are the artifact caused by the tuning of hyperparameters, e.g., the training converged to local optima that occasionally performed well.\n\nNote that the above observation on the training problem also suggests that the claim of Theorem 1 (the estimator $\\hat{\\theta}$ converges to the *true* parameter $\\theta^*$) is not correct.\n\nIn the proof, the authors considered the following objective function:\n$$\nL^C(\\theta, \\alpha) = - \\mathbb{E} \\sum_{i=1}^K \\left[ \\sum_{k=1}^K \\pi(i | k, x) P(k | x, \\theta^*) \\log P^C(i | x, \\theta) \\right]\n$$\nLet $U(i | x, \\theta^*) = \\sum_{k=1}^K \\pi(i | k, x) P(k | x, \\theta^*)$. We then have\n$$\nL^C(\\theta, \\alpha) = - \\mathbb{E} \\sum_{i=1}^K U(i | x, \\theta^*) \\log \\frac{\\exp(m_i(x; \\theta) + \\alpha_i(x))}{\\sum_j \\exp(m_j(x; \\theta) + \\alpha_j(x))}\n$$\nBy taking the derivative with respect to $\\omega \\in \\\\{\\theta, \\alpha\\\\}$, we have\n$$\n\\frac{\\partial L^C(\\theta, \\alpha)}{\\partial \\omega} = - \\mathbb{E} \\sum_{i=1}^K U(i | x, \\theta^*)\\left( \\frac{\\partial (m_i(x; \\theta) + \\alpha_i(x))}{\\partial \\omega} - \\sum_{k=1}^K \\frac{\\exp(m_k(x; \\theta) + \\alpha_k(x))}{\\sum_j \\exp(m_j(x; \\theta) + \\alpha_j(x))} \\frac{\\partial (m_k(x; \\theta) + \\alpha_k(x))}{\\partial \\omega} \\right) \\\\\n= - \\mathbb{E} \\sum_{i=1}^K \\left( U(i | x, \\theta^*) - \\frac{\\exp(m_i(x; \\theta) + \\alpha_i(x))}{\\sum_j \\exp(m_j(x; \\theta) + \\alpha_j(x))} \\right) \\frac{\\partial (m_i(x; \\theta) + \\alpha_i(x))}{\\partial \\omega}\n$$\nThus, any $\\theta, \\alpha$ that satisfy $U(i | x, \\theta^*) = \\frac{\\exp(m_i(x; \\theta) + \\alpha_i(x))}{\\sum_j \\exp(m_j(x; \\theta) + \\alpha_j(x))}$ are optimal.\n\nIn the proof of Theorem 1, the authors only considered a specific $\\alpha$, and overlooked the existence of other $\\alpha$ that are equally optimal, which led to the wrong claim that $\\hat{\\theta}$ converges to $\\theta^*$.\n\n\n\n### Clarity\n\nApart from the serious flaw above, I think the paper is clearly written and the main claim of the paper is easy to follow.\n\n\n\n### Originality\n\nThe use of the adjustable parameters for fitting noisy data is studied in the literature of robust learning. I would like to suggest the authors to see [Ref1] and references therein. In [Ref1], an additional penalty is imposed on the adjustable parameter to avoid the trivial solution I raised above.\n\n[Ref1] Consistent Robust Regression, NeurIPS17.\n\n\n\n### Significance\n\nBecause of the flaw I raised above, I think the contribution of this paper is not significant.\n\n\n----------\n### Feedback after discussion\n\nI would like to clarify my thought here. Recall that using the weight decay is equivalent to adding the L2 regularization to the training objective. An important observation here is that the addition of the L2 regularization to the proposed objective will make the global optima non-trivial (apart from the trivial ones I raised), and there might be a hope that the new global optima has some useful properties. What this observation indicates is that the use of the L2 regularization (or weight decay) is an essential factor for the proposed method to output something meaningful. This fact also implies that the analysis of the objective function alone (without the regularization, in Section3) is no longer meaningful. Moreover, because the L2 regularization (or weight decay) is an essential factor, the tuning of its weight should have a major impact to the resulting model. I therefore think it will be important to investigate the effect of such a weight in the experiments, instead of just using a standard weight.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-LfNiaz3CiU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper seems to be a useful contribution to the literature on deep learning with noisy datasets, showing a good improvement over the state of the art. \nThis work presents a theoretical model formulation to capture the biased effects of incorrect labels automatically \nThe paper is generally well-written and structured clearly. However, there are few small changes or suggestions to improve are as follows:\n•\tThe authors have not rationalized enough the performance of the AutoCleansing over the other methods and on the data sets in detail for better readability.\n•\tSummary of the data sets as a table provides better visibility and readability. \n•\tDetection of incorrect labels using the proposed method has been described in detail but why AutoCleansing does not require the threshold criteria of the drop rate needs further discussion. Please add additional details.\n•\tIn addition to the learning rate, weight decay, Epoch etc. the authors can add the time units for each of the datasets helps the future researchers. Please add.\n\nThe paper and the supplementary provided describes the theoretical formulation and other objective functions in full detail and provides enough information for an expert reader to understand and interpret the results.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses learning with noisy labels, by detecting and correcting samples with noisy labels. Reviewers had concerns about the empirical evaluations, specifically about comparing to additional methods, about hyperparameter tuning, and about the improvements being vey small. There was also a concern that the analysis of the objective does not take into account explicitly the L2 regularization induced by weight decay. Based on these concerns the paper is not ready yet for publication.



Decision: Reject