Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
Authors: Jiaheng Wei
Zhaowei Zhu
Hao Cheng
Tongliang Liu
Gang Niu
Yang Liu
Venue: ICLR 2022 Poster (2022)
Keywords: Learning with noisy labels
benchmark
real-world label noise
human annotations

Abstract: Existing research on learning with noisy labels mainly focuses on synthetic label noise. The synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model the real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, e.g., Food-101N, WebVision, and Clothing1M. Yet the existing efforts suffer from two caveats: firstly, the lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise. Secondly, these efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is important to establish controllable, easy-to-use, and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets, which we name as CIFAR-10N, CIFAR-100N (jointly we call them CIFAR-N), equipping the training datasets of CIFAR-10 and CIFAR-100 with human-annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g.,  class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using  CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions. The corresponding datasets and the leaderboard are available at http://noisylabels.com. 

Published as a conference paper at ICLR 2022
LEARNING WITH NOISY LABELS REVISITED: A STUDY USING REAL- WORLD HUMAN ANNOTATIONS
Jiaheng Wei"t, Zhaowei Zhu"i Hao Cheng', Tongliang Liu', Gang Niu and Yang Liu University of California, Santa Cruz, 'TML Lab, University of Sydney, RRIKEN
f jjahhngwei,, zwzhu, haocheng, yangliu) (ucsc. edu, tongliang. liuu sydney........ edu.au, gang niu, ml @gmail com.
ABSTRACT
Existing research on learning with noisy labels mainly focuses on synthetic label noise. The synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model the real-world noise patterns. The recent literature has observed several efforts offer real-world noisy datasets, e.g., Food- 10IN, WebVision, and Clothing M. Yet the existing efforts suffer from two caveats: firstly, the lack of ground-truth verification makes it hard theoretically study the property and treatment of real-world label noise. Secondly, these efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is important to establish controllable, easy-to-use and moderate-sizer real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets, which we name as CIFAR-1ON, CIFAR- 100N (jointly call them CIFAR-N), equipping the training datasets of CIFAR-10 and 10FAR-100 with human-annotatee annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real- world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class- daass-depnndent label noise). We then initiate an effort to benchmarking subset of the existing solutions using CIFAR- ION and CIFAR-IOON. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependee synthetic noise. We show indeed the real-world noise patterns impose and outstanding challenges compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these datasets would facilitate the development and evaluation future learning with noisy label solutions. The corresponding datasets and the leaderboard are available at hnttp://noisylabelle: com
INTRODUCTION
Image classification task in deep learning requires assigning labels to specific images. Annotating labels for training use often requires tremendous expenses on the payment for hiring human annotators.
The pervasive noisy labels from data annotation present significant challenges to training a quality
machine learning model. The problem of dealing with label noise has been receiving increasing
attentions. Typical approaches include unbiased estimators and weighted loss functions (Natarajan)
et al. 2013 Liu & Tao 2015). loss correction (Patrini al.) 2017; Liu & Guo 2020). sample
selection aided (Jiang et al. 20 Han et al. 2018; Yu et al. 2019), etc. The majority of existing solutions are often developed under stylish synthetic noise model, where the noise rates are either class-dependent or homogeneous across data instances. However, real-world supervision biases may come from humans (Peterson al.) 2019). sensors (Wang al. 2021b). or models (Zhu et al. 2022). which are likely to be instance- -eeeedeeed Recent works on instance-dependent settings (Cheng al. 2021 Jiang et al. 2002) also have some structural assumptions, e.g., the noise transition differs
"Equal contributions in alphabetical ordering.
1 "ooresponniig author; Yang Liu <yanglius iu@ucsc.

---

Published as a conference paper at ICLR 2022
Table 1: Summarized information of existed noisy-laber benchmarks: the "estimated" noisy levels are obtained through subset of the dataset with verified clean labels. the max image width pixel is less than 250.
means
Datuset
Train/Test Size
Classes
Noise level
Moderate resolution
Cleun label
No Interventions
Food-101 (Bossadd at al. 2014
75.75K / 25K
101
N/A Estimated 38%
Clothing|T (Xioo
2015
VisVVssoon cl nl. 20171
522,44M/ / TOOK
1900
NIA
X
Food-101N
20118
Animal-10N Sting ei al. 2019
/ 5K
10
8%
/
Mini-ImageSet Daang
Red Stanford Cars Frinng di th (0200
8K /SK
196
USS-M0%
/
/
CIFARTOHT (Peserson et all 2019
50K7 JOK
10
NIA
CIPAR-16N. aggggggggtt (Ours)
5
9.03%
CIFAR- 10N rundom (Ours)
S0K7 I0K
10
no 18%
CIFAR-IQN-woree (Ours)
S0K/
40.211%
CIFAR-100 100N-coarse (Ours)
50K TOK
20
25. 6078
CIEAR- 100N-tine (Ours)
50K / 10K
40.200%
in different parts of features (Xia et al. 2020b) or sub- sub-populations (Wang et al., 202 Zhu et al. 2021a). Although these statistical assumptions facilitate the derivation of theoretical solutions, it is unclear how the existing models captured the real-world noise scenario.
To empirically validate the robustness of proposed methods, synthetic noisy labels on CIFAR-10 and CIFAR-100 (Krizhevsky et al.] (009) are the most widely accepted benchmarks. The literature has also observed approaches to the simulation of human annotators in data labeling (Hua al.)
2013 Long & Hua, 2015: Liao et al. 2021), and real-world label noise benchmarks, including
Food-10 10ld-101@ (Bossard al. (014)). Clothing- IM (Xiao et al.] 2013), Web Vision (Li et al. 2017). etc.
We summarize the above real-world noisy label datasets in Table While a more detailed description
and discussion of the existing datasets can be found in the related works, we want to highlight several outstanding issues in existing benchmarks and evaluations. As noted in Table except for CIFAR
related noisy label datasets, all other datasets suffer from at least one of the three caveats:
Complex task (High-resolution)): when learning with large-scale and relative high-resolution data, the complex data pattern, various augmentation strategies (Xiao et al. 2015). the use of extra train or clean data (Bossard al., [014] Xiao et al.) 2015 Lee et 2018). different computation power (for hyper- parameter tuning such as batch-size, learning rate, etc) jointly contribute to the model performance and then result in unfair comparison.
Missing clean labels: the lack of clean labels for verification in most existed noisy- datasets makes the evaluation of robust methods intractable.
Interventions: human interventions in data generation (Jiang et al. 2020) and non-representative
data collection process (Song al. 2019) might disturb the original noisy label pattern.
In addition, despite synthetically labeled CIFAR datasets are popular and highly used benchmarks for evaluating the robustness of proposed methods, there exists no publicly available human annotated labels for CIFAR training datasets perform either validation of existing methods verification of popular noise models] A annatttnd version of CIFAR datasets would greatly facilitate the evaluations of existing and future solutions, due to the already standardized procedures for experimenting with CIFAR datasets. All above issues motivate us to revisit the problem of learning with noisy labels and establish accessible and easy-to-use, verifiable datasets that would be broadly usable to the research community. Our contributions can be summarized as follows:
We present two new benchmarks CIFAR-ION CIFAR- CIFAR-1OON which provide CIFAR-10 and CIFAR- 100 with human annotated noisy labels. Jointly we call our datasets CIFAR-N. Our efforts built upon the CIFAR datasets and provide easily usable benchmark data for the weakly supervised learning community (Section 3. We expect to continue to maintain the datasets to facilitate future development of results.
We introduce new observations for the distribution of human annotated noisy labels on tiny images, i.e., imbalanced annotations, the flipping of noisy labels among similar features, co-existence of multiple clean labels for CIFAR- 100 train images (which leads to a new pattern of label noise), etc. We further distinguish noisy labels in CIFAR-1OX and CIFAR- with synthetic class- dependent label noise, from the aspect of noise transitions for different features qualitatively and quantitatively (via hypothesis testing) (Section
CIFAR 10H
Peterssn al. 2019) only provides test images with noisy human annotations.
2

---

Published as a conference paper at ICLR 2022
We empirically compare the robustness of comprehensive list of popular methods when learning with CIF,R-ION, CIFAR-100M 10ON. We observe consistent performance gaps between human noise and synthetic noise. The different memorization behavior further distinguishes the human noise and synthetic noise (Section 5). The corresponding datasets and the leaderboard are publicly available at httt:://noiii sy abe 1s....... com
1.1 RELATED WORKS
Learning from noisy labels Earlier approaches for learning from noisy labels mainly focus on loss adjustment techniques. To mitigate the impact of label noise, line of approaches modify the loss of image samples by multiplying an estimated noise transition matrix (Patrini et al. 2017; Hendrycks al. 2018 Xia et al., [2199] Yao al. 2020), re-weight the loss to encourage deep neural nets fit
on correct labels (Liu & Tao 2015), propose robust loss functions (Natarajan et al. 2013 Ghosh et al.
2017. Zhang Sabuncu| 201 8 Amid et al. 2019 Wang et al. 2019; Liu & Guo 2020), or introduce
robust regularizer (Liu et al. [020]] Xia et al. [020a]] Cheng et al. [021] Wei et al. [022]. Another
line of popular approaches behaves like a semi- superiupereieed manner which begins with a clean sample selection procedure, then makes use of the wrongly-labeled
samples. For example, several methods (Jiang et al. 2018 Han et al. [0188] Yu et al.) 2019; Wei et al. (020)p adopt a mentor/peer network "clean" ones for the student/peer network. To further explore the benefits of wrongly-labeled samples and improve the model performance, et al. (2020) chose the
to select small-loss samples
Mix.atthh (Berthelot et al. 2019) technique which has shown success in semi-supervised learning.
Benchmarks noisy labels datasets Food-101 (Bossard et al. 2014), Clothing- IM (Xiao al.) 2015), WebVision (Li al.) 2017) are three large-scale noisy labeled web-image databases which consist of food images, clothes images or other web images, respectively. However, the majority of images in these three datasets do not have a corresponding clean label to perform controlled verification (e.g., verifying the noise levels). Later, much larger-scale food dataset is collected by Lee et al. (0118), which contains exactly the same classes as F01 (Bossard et al.) 2014). More recently. Peterson et al. (2019) present noisily labeled benchmarks on CIFAR- 10 test dataset where each test image has 51 human annotated labels in average. Jiing et al. 2020)) construct noisily labeled Mini- Mini-ImageNet (Vinyals et al. (016)) and Stanford Cars datasets (Krause al., 2013) with controlled
noise levels by substituting human annotated incorrect labels for synthetic wrong labels.
2 SYNTHETIC LABEL NOISE
In this section, we discuss a few popular synthetic models for generating noisy labels. We focus on K K-class classification task. Denote by D == ((x,. V/ )/)) ]ne[N]] the training samples where [N] = "I,,,..,NN; N). (In, yn )s are given by random variables (X, Y) E X' y drawn from the joint distribution D, where X' y can be viewed as the space of feature and label, respectively. In real-world scenarios, a classifier f only has access to noisily labeled training sets D = ((r Gn IN([N) We assume the noisy samples (Ing, Is are given by random variables (X, Y) E X' x 5 which are drawn from the joint distribution D. Clearly, there may exist n E [N] such that yn o Hn. The flipping from clean to noisy label is usually formulated by a noise transition matrix T(X), with elements: (X) == P(Ý = jJY = i, X). We shall specify different modeling choices of T(X) below.
2.1 CLASS-DEPENDENT LABEL NOISE
The first family of noise transition matrix is the class-dependent noise where the label noise is assumed to be conditionally independent of the feature X. Mathematically, T(X) = T and
Tx5(X) = P( = -iy = o), Vi, j E [K].
Symmetric T The symmetric noise transition matrix (Natarajan et al. 2013) describes the scenario where an amount of human labelers maliciously assign a random label for the given task. It assumes that the probability of randomly flipping the clean class to the other possible class with probability e. Assume the noise level is e, the diagonal entry of the symmetric T is denoted as T = 1 - e. For any other off diagonaa entry Tú, where * j. the corresponding element is Tıj -K--
3

---

Published as a conference paper at ICLR 2022
Asymmetric T The asymmetric noise transition matrix (Patrini et al., 2017) simulates the where there exists ambiguity classes, ie, human labelers may wrongly annotate the truck as auto- mobile due to the low-resolution images. There two types of widely adopted asymmetric T. The assymetric-next 7 assumes that the clean label flips to the next class with probability 6, i.e, i (i + 1) mod K for E [K]. The assymetric-pair T considers [ disjoint class pairs (fc; Jc) where la < je. For e E [7], Tesie Tasd = c, and the diagonal entries are 1 -e.
2.2 DETANCE-DEPENDET LABEL NOISE
Beyond the feature-inen independent assumption, recent works pay more attention challenging case where the label noise is jointly determined by feature X and clean label Y. There are some techniques for synthesizing the instance-dependent label noise, such as the polynomial margin diminishing label noise (Zhagg et al. 2021b) where instances near decision boundary are easier to be mislabeled, the part-dependent label noise (Xia et al., 2020b) where different parts of feature may contribute different noise transition matrices, and the group-dependent label noise (Wang et al. 2021a Zhu al. 2021a) where different cub-populations may have different noise rates. All of these noise models are proposed with some statistical assumptions, which facilitate the derivation of theoretical solutions.
HUMAN ANNOTATED NOISY LABELS ON CIFAR-10, CIFAR-100
In this section, we introduce two new benchmark datasets for learning with noisy labels: CIFAR- 1ON and 110N.-10ON. Both datasets are built using human annotated labels collected on Amazon Mechanical Turk (M-Turk): we post CIFAR- 10 and CIFAR-100 training images as the annotation Human Intelligence Tasks (HITs), and workers receive payments by completing HIT's.
3.1 CIFAR-1ON REAL-WORLD NOISY LABEL BENCHMARK
CIFAR-10 10 (Krizhessky et al. 2009) dataset contains 60k 32 x 32 color images, 50k images for training and 10k images for testing. Each image belongs to of ten completely mutually exclusive classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
Dataset collection We randomly split the training dataset of CIFAR-10 10 without replacement into ten batches. In the Mturk interface, each batch contains 500 HITs with 10 images per HIT. The training images and test dataset remain unchanged. Each HIT is then randomly assigned to three independent workers. Workers gain base reward $0.03 after submitting the answers of each HIT. We reward workers with huge bonus salary if the worker contributes more HITs than the averaged number of submissions. We did not make use of any ground-truie clean labels to approve or reject submissions. We only block and reject workers who submit answers with fixed/regular distribution patterns. We defer more details of the dataset collection to Appendix A
Dataset statistics For CIFAR-ION dataset, each training image contains one clean label and three human annotated labels. We provide five nabsy-label sets as follows.
Aggregate: aggregation of three noisy labels by majority voting. If the submitted three labels different for an image, the aggregated label will be randomly selected among the three labels. Random i {1, 2, 3}): the i-th submitted label for each image. Note our collection procedure ensures that one image cannot be repeatedly labeled by the same worker.
Worst: dataset with the highest noise rate. For each image, there exist any wrongly annotated labels in three noisy labels, the worst label is randomly selected from wrong labels. Otherwise, the worst label is equal to the clean label.
In CIFAR-ION, 60.27% of the training images have received unanimous label from three indepen- dent labelers. The noise rates of prepared five noisy label sets are 9.03% (Aggregate), 17.23 % (Random 1), 18.12% (Random 2), 17.64% (Random 3) and 40.21 % (Worst). A complete dataset comparison among existing benchmarks and ours are given in Table I We defer the noise level of each batch to Table (Appendix). Aggregating the annotated labels sigmificantly decreases the noise rates. All three random sets have 18% noise level. To provide challenging noisy setting, also prepare a worst label set which serves to cover highly possible mistakes from human annotators on 100.R-10.
4

---

Published as a conference paper at ICLR 2022
3.2 CIFAR-10ON REAL-WORLD NOISY LABEL BENCHMARK
CIFAR-101 100 (Krizhevsky et al. 2009) dataset contains 60K 32 x 32 color images of 100 fine classes, 50000 images for training and 10000 images for testing. Each (fine) class contains 500 training images and 100 test images. The 100 classes are grouped into 20 mutually exclusive super- super-classes.
Data collection We split the training dataset of CIFAR- 100 without replacement into ten batches with five images per HIT. Only one worker is assigned to each HIT. We group the 100 classes into 20 disjoint super-classes (see Table 5 in the Appendix) which are slightly different from the 20 "coarse" categories named in CIFAR- 100. The fine label in each super-class is summarized in Table (Appendix). Workers are instructed to firstly select the super-class for each image. And will then be re- re-directed to the best matched fine label. Since some super-classes are hard to recognize without prior knowledge in biology, we provide workers with easy access to re-select the super-class, and every fine class has an example image for references. The rejecting rule and the bonus policy are the same as those in CIFAR-ION -ON.. We defer more details of the dataset collection to Appendix B
Dataset statistics For CIFAR-100N dataset, each image contains a coarse label and fine label given by a human annotator. Most batches have approximately 40% noisy fine labels and 25 % noisy coarse labels. The overall noise level of coarse and fine labels are 25.60 % and 40.20 %. respectively, We defer the noise level of each batch to Table 7 (Appendix).
PRELIMINARY OBSERVATIONS ON CIFAR-ION, CIFAR-10ON
In this section, we analyze the human annotated labels for CIFAR-10 10 and CIFAR-100. We will empirically compare the noisy labels in CIFAR- -ON and CIFAR- -OOAR-10ON with class-dependent label noise both qualitatively and quantitatively.
4.1 THE NOISY LABEL DISTRIBUTION
Observation 1: Imbalanced annotations Our first observation is the imbalanced contribution of labels. Note that while the number of images are the same for each clean label, across all the five noisy label sets of CIFAR-1ON, we observe that human annotators have different preferences for similar classes. For instance, they are more likely to annotate an image to be an automobile rather than the truck, be the horse rather than the deer (see Figure 8 in the Appendix). The aggregated labels appear more frequently in automobile and ship, and less frequently in deer and cat. This gap of frequency becomes more clear the worst label set. CIFAR- -1ONN, human annotators annotate frequently on classes which are outside of the clean-coarse, i.e., 25% noisy labels fall outside of the super-class and 15% inside the super-class. And the phenomenon of imbalanced annotations also appears substantially as shown in Figure which presents the distribution of noisy labels for each selected fine class. "Man" appears 2
750 times, while "Strectcar" only has Rs 200 annotations.
Count
400
Count
Count
800
Count
500
Count
400 2m
400
200
000
0990
SAT TOWK
people
mediatt-ee
other
Figure 1: Categorical distribution of noisy labels on CIFAR-10OM (most imbalanced 6 super-classes): the red line in each subplot indicates that the number of each clean fine class is 500.
Observation 2: Noisy label flips to similar features In CIFAR- 100N, most fine classes are more likely to be mislabeled into less than four fine classes. In Figure 2 we show top three wrongly annotated fine labels for several fine classes that have relative large noise rate. Due to the low- resolution of images, a number of noisy labels are annotated in pair of classes, i.c. Re 20% of "snake" and "worm" images mislabeled between each other, similarly for "rockroack"""beetd "wolve", etc. While some other noisy labels are more frequently annotated within more classes, such as "baa"""""b"""""""""""""" "mmm""""""" "haake"""hale"" "dolphin""" ""truut"" etc, which share similar features.
"fox"-
The pattern of noise transition matrices In the class-dependent label noise setting, suppose the label noise is conditional independent of the feature, the noise transition matrices of CIFAR- 10N and CIFAR- 1OON are best described by a mixture of symmetric and asymmetric T. For CIFAR-1ON. we heatmap the aggregated noisy labels, randoml noisy labels and worst noise
Observation 3:
5

---

Published as a conference paper at ICLR 2022
equarum finh Willow Trep
pine
willow True foreet
dopriin STOLE
hamstor rable
ban house
my
pin Jree
Iorest
100 fastchh
100
100
100
60 shark
100
100 strcetcar
maple_rrep
toke
pin_trpc
strow
Figure 2: Top 3 wrongly annotated fine labels in selected fine classes. For "pine tree", "shrew". "streetcar", the dominant class is the wrong class. The corresponding number of correct annotations are highlighted with red lines.
0.00
631
SCO
840
821
8CT
520
831
030
(a) Aggregated noisy labels
(b) Random noisy labels
(c) Worst noisy labels
Figure 3: Transition matrix of CIFAR-101 1ON noisy labels (color bar is log- log-norm transformed).
labels w.r.t. the clean labels. In Figure the three noisy label sets share common pattern: the clean label flips into one or more similar classes more often. The remaining classes largely follow the symmetric noise model with low noise rate. For example, "truck" and "automobile" flip between each other more often (= 25% 30% percentage), which is much larger than that of all other classes. Besides, in the central area of each transition matrix, it is quite obvious that the clean label of animal classes flips more often to other animals. Similar observations hold in CIFAR- R-00N,, where each class flips to few misleading classes with much higher probability than that of remaining ones (see Figure Ti the Appendix). Apparently, current synthetic class-dependent noisy settings are not as complex as the real-world human annotated label noise.
Observation 4: Label noise: bad news or good news? During the label collection, there exist a non-negligible amount of the wrongly annotated classes that indeed co-exist in the corresponding images. In other words, training images of CIFAR-106 may contain multiple labels rather than a single We select several exemplary training images of CIFAR-100 where multiple labels appear (in Figure 4). The annotated class also appears in the corresponding image while is deemed as wrong annotation by referring the officially provided clean label. The most frequent case is best described by the scenario where man holding a flatfish in hands. The clean label usually comes "flatfish", while human annotators are more likely categorize these images into "man". We conjecture that with the increasing label dimension, the phenomenon of multiple clean labels might be a more common issue. We leave more explorations for the future work.
I
lawn_ wwn_mower man
bicycle
plain
bee sunflower
butterfly poppies
flatfish man
dolphin
mountain n clowd
beaver man
table
clowd
Figure 4: Exemplary CIFAR- 100 training images with multiple labels. The text below each picture denotes the CIFAR-100 clean label (first row) and the human annotated noisy label (second row).
4.2 HUMAN NOISY LABELS V.S. SYNTHETIC NOISY LABELS
Recall that, for the class-dependent label noise, we have P P P(" indicating the noise transitions identical for different features. For general instance- cettnce-deeendent label noise, the above equality may not hold. In this subsection, we explore to what degree this -eauuee-dependeccy holds by checking whether the equality is satisfied not for different features. In the following, by checking

---

Published as a conference paper at ICLR 2022
cas à (airatone)
dbss 2 [iaamoblee
coss (birs)
Glos 4 (cot)
Class Weee
date
50
33
(a) Human noise (Random 1): noise level RJ 17.28%
Class (MFrolenne)
Clss (wotomsbit))
class (ain)
dlas LOK!)
Cleas (dwer)
2
(b) Synthetic class- ceas--ependent noise with the same T
Figure 5: Illustration of noise transitions of human-level label noise and the synthetic version. We divide the representations of images from the same true class into 5 clusters by k-means. The representations come from the output before the final fully-connected layer of ResNet34. The model is trained on clean CIFAR-10. Negative cosine similarity measures the distance between features.
the equality for different features, we will show the feature-dependency from both qualitative aspect and a quantitative aspect.
4.2.1 A QUALITATIVE ASPECT
We first visualize the noise transitions for different features from a qualitative aspect. Taking CIFAR- 1ON as an example, each image in CIFAR-10 only appears once. Without additional assumptions, only have three noisy labels for each individual feature X, which makes difficult accurately estimate the noise transition probability T(X), VX, even though the ground-truth clean label is available. To make the estimation tractable, we consider estimations on locally homogeneous label noise, i.e., nearby features share the same T(X). See the rigorous definition as follows.
Definition (M-NN noise clusterability) (hhu al. 2021b) We call D, satisfies M-NN (M- Hearest-Neighhon noise clusterability if the M-NN of x,, have the same noise transition matrix as En, Le., T( T (I Vi [M] where (ELM) denote the M-NN of zn.
estimate T(X) each subset D.,. In visualization (e.g., Figure 5). rather than manually fix a particular M, we use the k-means algorithm to separate features belonging to the same true class to 5 clusters and adaptively find a suitable M for each cluster. Denote by It the set of instance indices from the y-th cluster of clean class i. Then for label noise It,,, we assume it is -nnture-indeenndentt and denote the corresponding transition vector by Pi,e, where each element [r. is expected to be P(Ý = j| n E IT,,, Y = i). The transition vector Pi can be estimated by counting the frequency of each noisy class given noisy labels in Ii For example, in Figure 5a each row of the top-right subfigure titled "Class 5 (deer)" shows
With M-NN noise clusterability,
P(Y)X,Y 5) for different clusters of X, i.e., transition vectors P5. {1.... .5). Before putting up a more formal testing, we clearly observe that different transition vectors across different feature clusters, signaling the fact that the noise transitions |( 5) are feature- ceature-dependent. For controlled comparison, we synthesize the class-dependent label noise with the same expected noised transition matrix T := [X)) as our human-level label noise and illustrate it in Figure (5b) We can find that different rows of each matrix in Figure 55 are very similar, showing the synthetic noise is ceature-independen and our verification method is valid.
4.2.2 A QUANTITATIVE ASPECT
In this section, we quantitatively compare human noise and synthetic class-dependent noise through hypothesis testing.
Following the clustering results in Section 4.2.1 the human-noise is feature-dependent or not. For CIFAR- ION, the null hypothesis Ho and the corresponding alternate hypothesis H2 are defined as:
Formulation
further statistically test whether
Ho Human annotated label noise in CIFAR-1ON is feature-indeppndent:;
H Human annotated label noise in CIFAR-1ON is feature-dependent

---

Published as a conference paper at ICLR 2022
Note that the synthetic label noise illustrated in Figure 5b supposed to be feature- ceature-nddpenndett, the above hypotheses are converted to:
Ho Human annotated label noise in CIFAR-ION is the same as the corresponding synthetic one;
H1 Human annotated label noise in CIFAR-1ON is different from the corresponding synthetic one.
From Figure 5 one measure of the difference between human noise and synthetic noise is the distance between transition vectors pi across different noise, e.g., df == IIp Ilphummnn P.,
p.ppthetic 1
As contrast, we need to compare d with d := llp Syymheetic - Pi,, 113. where Piv
symthetic' denotes the transition vector from the same synthetic noise but different clustering result (caused by random
synihetic
data augmentation). Intuitively, if af is much greater than d/2)) we should accept H₁. The above hypotheses are then equivalent to:
Ho: : ([i,)E][)))EE come from the same distribution as {d (di.hettk))eEE
H1 : {d NE(K), come from different distributions from {d he(
We repeat the generation of di,y for 10 times, where the images are modified with different data augmentations each time. We choose the significance level a = 0.05 and perform a two-sided t-test w.r.t [E]K),EE and ([(e, Hypothesis testing results show that p 1.8e for the whole data. Thus, the null hypothesis is rejected with the significance value hypothesis "human annotated label noise in CIFAR-ION is feaaure-dependent" is accepted. Similar conclusion can be reached for CIFAR-10ON we defer more details to Appendix D
5 LEARNING WITH CIFAR- ION AND CIFAR- 100N
We reproduce several popular and sffe-oo--th-aan robust methods on synthetic noisy labels (using calculated transition matrices from our collected data) as well as our collected human annotated labels. Most of the selected methods fall into loss correction, loss re-weighting, and loss regularization related methods, etc.
5.1 PERFORMANCE COMPARISONS ON CIFAR-ION AND CIFAR-10ON
For fair comparison, adopt ResNet-34 He al. 2016), the same training procedure and batch- size for all implemented methods. More experiment details are deferred to the Appendix H In Table 2. note that both ELR+ (Liu al. 2020) and Divide-Mix (Li et al. 2020) adopt two networks with advanced strategies such as mix-up data augmentation, their performances on CIFAR- 100N largely outperforms all other methods. We also empirically test the performances of the above methods under the class- class-dependent noise settings which follow exactly the same T as appeared in CIFAR- -IFAR-1ON and CIFAR-10ON. details are deferred to Table 9 in the Appendix.
Table 2: Comparison of test accuracies (%) on CIFAR-ION and CIFAR-IO0 (fine-label) using different methods. Top 3 performances are highlighted in bold (ttdard deviation of 5 runs). More methods are included in the Appendix as well as http: http:://noisylaaaa ls com
Method
Clese
Aggregute
Randow
CIFAR-TON
Random 2
Rondon
Worst
Cleon
CIFAR 1ONN
Noisy
CE:Somnarrh
92.9
0.11
87.77 0.36
0.65
16.46
1.79
85
77.69
1.35
16,70
0.74
55,50
0.66
Foeward 7 [atrinn al.
2017
93002
0.12
88.24
0.22
85.33
0.50
86.14
0.24
87.04
:: 0.55
79.79
0.46
76.18
= 0.37
57.01
1.03
Co Co-seachnng=
2019
92.41
0.20
90.61
1: 0.22
89.70
0.27
89.47
1: 0.18
89.54
0.22
83.26
1: 0.17
70.99 -: 0.22
57.88
1.8880024 0.24
T-Revision
Rirer
2019
93.35
0.25
88.52
+ 0.17
88.33
0.32
87.71
1.02
87.79
0.67
80145
= 1.20
72.85
= 0.21
0.33
Peer Loss
93.99
0.13
90.75
1: 0.25
89:06
0.11
88.76
0.19
88.57
0.09
82.00
0.60
74.67
: 0.36
57.59.1000
BLR+
95.39
94.83
0,10
94.20
0.24
94.34
91.09
1.60
78.57
66.72
0.07
Positive
91.77
91.57
82.76
55.81
F-DRv
Wer
91.64
89
89.79
0.12
89.55
82.53
57.10 0.65
Divide-
Yurd)
95.37
0.14
95.01
0.71
95.16
0.19
V5.23
0.07
95.21
0.14
92.56
0.42
76.94
0.22
71.13
0.48
Negative-L.
LS
or 2002
2021
94.22
0.25
91.97
0.45
90.29
0.32
90.37
0.12
90.13
1: 0.19
82.99
0.36
77.06 = 0.73
58,59
0.9
CORES.
(Chpao call 2024
94.16
0.11
95.25
0.09
94.45
0.14
94.88
0.31
94,74
0.03
91.66
0.09
73.87
1 0.16
55.72
0.42
VolMinNar La de 2021
92.12
0.30
$9.70
0.21
88.30
0.12
88.27
0.09
88.19
0.41
SI.53
0.20
70.61
0.88
$7.80
0.31
CAL (hhi
94
0.31
91:97
0.32
90.91
8 0.31
95
0.30
90.74
0.24
85.36
0.16
75.67
= 0.25
61.73
1:3:::033 0.42
PES (Setzi
94.66
0.18
95.06
95.19
0.23
95.23
0.13
92
0.22
77.92
0.4
70.36
0.3
Observation 5: Performance gap (human noise v.s. synthetic noise) Continuing the reported
observations in Section 4.1, in Table we highlight that for most selected methods, class-dependent synthetic noise is much easier to learn on CIFAR- 10, especially when the noise level is high. The gap is less obvious for CIFAR-100. However, to work well in low noise regime. Besides, ELR (Liu et al., (020)) performs even slight better when learning with real-world human noise than that the synthetic class-dependent noise settings.
also observe that Divide-Mix (Li et al. (2020) fails
8

---

Published as a conference paper at ICLR 2022
Table 3: Performance gap between human noise and class-dependent noise: test accuracy (trained on synthetic noise) test accuracy (trained on human noise). Negative gaps are highlighted in red.
Method
Agamguate
Rundow
CHEAR
Rundon
Rundom
CIFAR: 100
CB (Staodard)
4.35
6.01
4.50)
5.82
9.00
1.20
Forward
Parini al
2017
A.30
4.82
4.86
4.27
7.08
-0.14
Co-taccin
0.86
1,05
-0.51
F-Dly
Wo
1.33
1.65
Divide-ri Stx
2100
-001
0.44
0.42
0.28
B.29
0.65
Negatved 1
0.77
1.31
1.08
1.36
4,00
1.26
JoCok
Welcell 2020
0.35
0.78
0.68
1.01
2.43
00.48
CORES
10:1
1.52
1,66
0.72
0.04
0.09
5.2 MEMORIZATION EFFECTS
When learning with noisy labels on CIFAR- 10 and CIFAR-100 datasets, empirical observations (Arpit et al., 2017] Liu et al. 2020 Xia al. 2020a /hang et al., 2021a) on synthetic noise settings suggest that deep neural networks firstly fit on samples with clean labels, then gradually over-fit and memorize samples with wrong/noisy labels (Xie et al. (202I). We next explore the memorization of clean and noisy labels on CIFAR-IÓN and CIFAR-IO0)
Definition 2 (Memorized feature) In K-class classification task, given trained classifier f. feature x and confidence threshold r), x is memorized by f if E [K] s.t. P(f(() = 7/.
In Figure 6 we train CE loss with ResNet-34 (He et 2016) neural network on three noisy label sets of CIFAR-1ON, aggre-label (left column), random- -aaeell-labell (middle column) and worst-label (right column). While visualizing the memorization (1) 0.95) on training samples, we split the train data into two parts: images with clean labels (the annotation matches the clean label) and wrong labels (the rest). We observe that: deep neural nets memorize features more easily when learning with real-world human annotations than synthetic ones. This is attributed to the fact that, compared with synthetic label noise, human annotators are prone to providing wrong labels on more misleading/ambiiuuon images complex patterns. Given the same noise level, learning with human noise labels is more challenging and deep neural nets over-fit on features of wrong annotations inevitably.
Fraztoo samples
Clcan labels (aggre_ (abere_
Frantion of samoles
Clean labels (rand)
Frection of sampies
Clcan lab (warse Jab el
Synthetic Real
15
386
130
25
20
76
109
322
75 Sporh
300 1b
Epach
Epach
labal)
Frecionc 23 mampls
Synthetic Roal
Fevetioon
25
Epach
Epoch
Epoch
Figure 6: Memorization of clean/correct and wrong labels on CIFAR- 1ON and synthetic noise with same T: red line denotes the percentage of memorized (wrongly predicted) samples, blue line denotes that of correctly predicted ones.
CONCLUSIONS
Building upon CIFAR datasets, we provide the weakly supervised learning community with two accessible and easy-to-use benchmarks: CIFAR-1ON and CIFAR-100N. We introduce observa- tions from human annotations such as imbalanced annotations, the flipping of noisy labels among similar features, co-existing labels in CIFAR- etc. From the perspective of noise transitions, we qualitatively show that human noise is indeed (eature-dependent and differs substantially from synthetic class-dependent label noise using hypothesis testing. We empirically compare the robustness of large quantity of popular methods when learning with CIFAR- ION, CIFAR- 10ON and synthetic noisy CIFAR datasets. We also consistently observe the large performance gap between human noise and synthetic noise, as well as the different memorization behavior on training samples.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is supported by a University of California, Santa Cruz startup fund, the National Sci- Foundation (NSF) under grant IIS-2007951 and IIS-2143895. TL partially supported by Australian Research Council Projects DE-190101473 and DP-220102121.
LICENSE
The released datasets CIFAR-N are publicly available at
com, under 4.0 International (CC BY-NC 4.0) license. You are free to share and adapt, while under the terms of attribution and non-commercial use.
OPEN COMPETITION
Moving forward, we would like to invite researchers openly compete using our CIFAR-N datasets to advance the field. Correspondingly, we will actively update our leaderboards. Featured competi- tions will include building better and more accurate models, estimating noise transition matrix, and detecting corrupted labels. This year (2022), we will host the inaugural public CIFAR-N competi- tion with UCAI-ECAI 2022. We will be actively maintaining http: http:///oosylllae com to disseminate future information.
ETHICS STATEMENT
This paper does not raise any ethics concerns. Our work contains the human subject study which involves only simply image annotation tasks in Amazon Mechanical Turk. The study has been carefully reviewed and received Institutional Review Board (IRB) exempt approval. We have followed the outlined protocol perform the data collection. Our implemented human subject studies:
Do not have negative consequences, i.e., leaking the private information that would identify human annotators Amazon Mechanical Turk.
Do not misrepresent work that might be competing or related.
Do not present misleading insights. Our work does not present applications that can lead to misuse.
Do not introduce bias or fairness concerns, and research integrity issues.
We make the collected datasets (CIFAR-N) and the leaderboard publicly available at http:// moisylate laylabelss... com A starter code is provided in httpt:t///// http:::///tthb..// com/ SCL///CCCC-CCC REAL/
cifar-10-100n
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based bregman divergences. In Advances Neural Information Processing Systems, pp. 14987-14996, 2019.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning- Scarning-Volume 70, pp. 233-242. JMLR. org, 2017.
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. Mixmatch: A holistic approach semi-supervised learning. Advances in Neural Information Processing Systems, 32:5049-5059, 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-10l-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446 461. Springer, 2014.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. International Conference on Learning Representations, 2021.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. Advances Neural Information Processing Systems, 11110616114555, 2018.
Gang Hua, Chengjiang Long, Ming Yang, and Yan Gao. Collaborative active learning of kernel machine ensemble for recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1209-1200 1216, 2013.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentoreet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp. 2304-2313. PMLR, 2018.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: controlled noisy labels. In International Conference on Machine Learning, pp. 4804-4815. PMLR, 2020.
Deep learning on
Zhimeng Jiang, Kaixiong Zhou, Zirui Liu, Li Li, Rui Chen, Soo-Hyun Choi, and Xia Hu. information fusion approach to learning with instance-dependent label noise. In International Conference on Learning Representations, 2022. URL httpt:tt:/// httttps://openreview.nt feeforrumm 1d=ecH2FKaARUp
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.

---

Published as a conference paper at ICLR 2022
Kuang- Mueni Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5447-5456, 2018.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020. URL
/ttp:::///peeeve//e eew net Forum? IdHHJGEEX.VWW.W
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 02862, 2017.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. arXiv preprint arXiv:2102.02400 2021.
Ktan-Hong Liao, Amlan Kar, and Sanja Fidler. Towards good practices for efficiently annotating large-scale image classification datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4350-4359 2021.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez-GGranda. Farly-learning regularization prevents memorization of noisy labels. Advances Neural Information Processing Systems, 33, 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. arXiv preprint urXiv:220..14022 2022.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting, JEEE
Transactions pattern analysis and machine intelligence, 38(3)44774411, 2015.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, pp. 6226-6236. PMLR, 2020.
Chengjiang Long and Gang Hua. Multi-class multi-annotator active learning with robust gaussian process for visual recognition. In Proceedings of the IEEE international conference computer vision, pp. 2839-2847, 2015.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp. 6448-6458. PMLR, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952 2017.
Joshua Peterson, Ruairidh M Battleday, Thomas Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedingy of the IEEE/CVE International Conference on Computer Vision pp. 9617-9626, 2019.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep
learning. International Conference on Machine Learning, pp. 5907-5915, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 19:3630-3638, 2016.
Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 526-536. 2021a.
Jingkang Wang, Hongyi Guo, Zhaowei Zhu, and Yang Liu. Policy learning using weak supervision. Advances in Neural Information Processing Systems, 34, 2021b.
12

---

Published as a conference paper at ICLR 2022
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference Computer Vision, pp. 322-330, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization, In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 13726-13735, 2020.
Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise. In International Conference on Learning Representations, 2020.
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, and Yang Liu. Understanding (generalized) label smoothing whenlearning with noisy labels. arXiv preprint (tfi::2006.0411 2021.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Advances in Neural Information Processing Systems, pp. 6838 6849, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-leamnigg Hindering the memorization of noisy labels. In International Conference on Learning Representations, 2020a.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In Advances in Neural Information Processing Systems, volume 33, pp. 7597-7610, 2020b.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2691-2699 2015.
Zeke Xie, Fengxiang He, Shaopeng Fu, Issei Sato, Dacheng Tao, and Masashi Sugiyama. Artificial neural variability for deep learning: on overfitting, noise memorization, and catastrophic forgetting. Neural computation, (3(8):2163-2122 2021.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T: Reducing estimation error for transition matrix in label-noise learning. In Advances in Neural Information Processing Systems, volume 33, pp. 7260-7271, 2020.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? arXiv preprint 1901:09 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021a.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In International Conference on Learning Representations, 2021b. URL http:tttss///ppereeeeee
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 8778-87787788 8788, 2018
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach learning with instance- dependent label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10113- 2021a.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative anchor points when learning with noisy labels. In International Conference on Machine Learning, pp. 12912-12923, PMLR, 2021b.
Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semi-supervised learning. In International Conference on Learning Representations, 2022. URL https://
openreview wewrriiwweeeteeeee forumm wtt.rrummmmdxxppffnnff.em.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
The Appendix is organized follows:
Section A: the details dataset collection, processing and information of CIFAR-ION.
Section B: the details of dataset collection, processing and information of CIFAR-I0ON.
Section C: the comparison of label collection procedure among CIFAR, CIFAR-IOH, and CIFAR-
N.
Section D: more detailed hypothesis testing results of CIFAR-N.
Section additional experiment results and details.
Before proceeding the appendix, want highlight our boarder impacts follows. Our observations and contributions may have following potential broader im-
Broader Impacts pacts:
Crowd-sourcing of data annotations in computer vision: CIFAR-N may be further used for
studying/proposing simulations of human annotations in crowd-sourcing, where the expenses of obtaining human annotations are often tremendous.
A template for hypothesizing label noise patterns: Our hypothesis testing method of instance- dependent label noise may provide a quantitative tool for testing the simulated human labels.
Benchmarking effort is important: Although learning from noisy labels has witnessed thriving developments, we often observed conflicting comparisons due to the randomness in the synthetic noisy labels. While there exist several datasets with real human noise, we view contribution as complementary to existing ones, due to the elaborated reasons above. We believe benchmarking existing and population solutions is important technical contribution the community.
Understanding real-world label noise: Our observations and the provided human-annotated labels help with understanding real-world label noise. Besides, our observations of the multi-label issues in CIFAR- 100 impose new label noise pattern that is largely neglected.
Motivations for real-world label noise solutions: our observations, especially the memorizing effects of woal-world label noise may provide the literature with motivations for addressing real- world label noise.
A CIFAR- 1ON WORLD NOISY LABEL BENCHMARK
In this section, we introduce the preparation for data collection, collection procedure of CIFAR-ION, the workers' behaviors, and more detailed statistics of the obtained labels.
A. CASE STUDIES BEFORE THE FORMAL COLLECTION
To make the collection procedure reasonable and efficient, we firstly upload few batches (500 images / batch) to test the behaviors of workers. Our observations show that the workers on image classification tasks may possibly incur following phenomenons:
with the appearance of bots, the accepted HIT may result in low-quality meaningless responses if the bot is able to pick answers and maliciously/randomly submit them. Otherwise, the bot accepts the HIT but could not submit annotations. The accepted HIT may have be re-assigned to another work after this HIT becomes expired which results in inefficient data collection.
Bots:
A large variance of the workers' contribution: empirical observations show that large amount of workers contribute too few HITs, while some professional workers upload with much less time. Thus, there exists a large variance in the number of the worker's submitted HITs and the noise label pattern substantially controlled by only few workers.
Incomplete submission: when there more than one image per HIT, worker would possibly neglect to click the label of one or more images, for example, the worker may only choose easy tasks annotate and skip tough ones. The incomplete submission complexes the reassignment
14

---

Published as a conference paper at ICLR 2022
of images with missing annotations, the processing of annotated results well the individual payment.
A.2 DATASET COLLECTION
To alleviate the impacts of aforementioned phenomenons, we randomly split the training dataset of CIFAR-10 without replacement into ten batches. Each batch contains 500 HITs. To ensure most workers do not contribute too few to the annotation task, we include ten 32 32 images per HIT. Each HIT is then randomly assigned to three independent workers. The worker gains base reward $0.03 after submitting all annotations of one HIT within 2 minutes. The worker can not submit the annotations unless all appeared images in the assigned HIT are labeled. The averaged number of submitted HITss per worker that we constrain the time duration for each assignment and re-design the interface, bots are less likely to finish our task either on time or under the procedure. What is more, we didn't make use of any ground-truth clean labels to approve or reject submissions. We only block and reject workers who submit answers with fixed/regular distribution patterns.
Workers with no less 7 submissions share $200 bonus rewards. Note
A.3 THE WORKERS' BELAAIIRSS
There are 747 independent workers contribute the construction of CIFAR-1ON. As depicted in Figure 7a most workers can submit the labels for ten images within one minute. Although we do observe that a small amount of assignments (S 0.2%) are finished within 10 seconds, which are likely to be low-quality responses. The work time in seconds have the mean 46.7, 7, the standard deviation 21.2 and the interquartile (25th percentile 75th percentile) range [31, 58). Among these 747 workers, most of them annotated more than 80 images. The number of annotated images per work has the mean 201, the standard deviation 329 and the interquartile range is (30, 220].
35.00
500
Number areggmmnns
3000
Number of workers
2500
2000
1500
200
Number submitted 2000
Work time per assignment
(a) Distribution of work time in seconds per HIT.
(b) Distribution of the amount of submitted labels.
Figure 7: The behaviors of workers in the collection of CIFAR-
A.4 MORE DETAILED DATASET STATISTICS
Table includes the summarized statisties of CIFAR-ION for each batch. In Table 4 the statistics "Consensus" means the three labelers have a consensus on the label of the same image. We do observe that several batches tend out to be more challenging for human workers annotate, i.e, the noise rate appeared in Batch3 is clearly higher than those of Batch4 and Batch5. The difference of noise rate especially significant on the "Worst" label set. We conjecture that there might exist more human annotators who malicious submit low-quality annotations when working on Batch3.
Note that while the number of images are the same for each clean label, across all the five noisy label sets of CIFAR-ION, we observe that human annotators have different preferences for similar classes, ie, they are more likely to annotate an image to be an automobile rather than the truck, to be the horse rather than the deer (see Figure 8). The aggregated labels appear more frequently in automobile and ship, and less frequently in deer and cat. This gap of frequency becomes more clear in the worst label set.
15

---

Published as a conference paper at ICLR 2022
Table 4: Consensus and noise levels (%) of each noisy label set in 10 batches (CIFAR-LON).
Statisties
Batch1
Batch2
Batch3
Batch4
Batch$
Batchó
Batch?
Batch8
Batch?
BatchI0
Overall
Consensus
53.32
64.26
40.20
68.42
70.04
62,90
58.10
66,98
58.00
60.52
60.27
Aggregate
10.20
7.92
10.76
7.90
7.14
7.72
10.12
8.42
10.80
9.30
9.03
Random
20.40
15.22
23.74
14.00
13.44
15.56
18.92
14.96
18.36
17.74
17.23
Random
21.54
28.
14.80
12.92
16.08
18.70
15.18
20.74
17.70
18.12
Random
19.24
17.54
23.24
13.84
13.62
16.84
18.34
15.46
20.24
18.04
17.64
Worst
47.06
36.40
39.44
32,20
30.44
37,50
42.54
33,56
42.88
40.06
40.21
6000
5750
6600
5250
Clean distribution
Count
6000
Aggregale
Random
Rendom
Random
4500
Worst
4250
4000
airpiane
bird
truck
automobile
doer Label distribution
horse
cat
dog
ship
frog
Figure 8: Categorical distribution of noisy label sets in CIFAR-1ON.
B CIFAR-10ON REAL-WORLD NOISY LABEL BENCHMARK
In this section, we introduce the preparation for data collection, collection procedure of CIFAR- 100N, and more detailed statistics of the obtained labels.
B. CASE STUDIES BEFORE THE FORMAL COLLECTION
To make the collection procedure reasonable and efficient, we firstly upload a few batches (500 images / batch) to test the behaviors of workers. Our observations show that the workers on CIFAR- 100 classification tasks may not only have mentioned phenomenons in A but have following issues as well:
Hard and time consuming: directly let workers to find the best matched label for each image from 100 possible classes is time assuming. Our case study shows that finding the label per image takes averaged 5-6 minutes. The work load as well as the difficulty level stop many workers from contributing or more submissions.
Lack of background knowledge: distinguishing several classes require some preliminary knowl- edge in biology, for example, it is common for workers to select a wrong super-class, especially for animal related ones: "aquatic mammals" and "fish". And the differences among some fine labels are hard to recognize, i.e, "trees" (oak, palm, pine), "medium-sized mammals" (porcupine, possum, raccoon, skunk), etc.
Label aggregation has few effeets: our empirical observations show that the the decrease of noise rates from aggregated labels given by 3 independent workers is less significant than the results on CIFAR-1ON.
B.2 DATASET COLLECTION
To deal with above mentioned issues, we firstly split the training dataset CIFAR-100 without replacement into ten batches. Each batch contains 1000 HIT's We include five 96 x 96 images per HIT which are reshaped from 32 x 32 ones in CIFAR- 100 train images. Only one worker is assigned with each HIT. Instead of requesting workers find the best matched label from 100 labels directly, we group the 100 classes into 20 super-classes which are slightly different from the 20 raw "coarse" categories given by Krizhevsky et al. (2009). The 20 newly defined super-classes summarized Table [5] And the new division in each super-class stated in Table 6 In order to reduce the workload of workers, they instructed to firstly select the super-class for each image.
16

---

Published as a conference paper at ICLR 2022
They will then re-directed to the corresponding 4-6 fine labels. Note that some super-classes hard to recognize without some prior knowledge in biology, we provide workers with easy access to re-select the super-class for the current image, and every fine label has an example image from Google images for references.. If the worker luckily finds the most suitable fine label from her point of view, also supply the jumping button so that the worker efficiently goes to the next image or the final submission of this HIT. A worker gains base reward $0.07 after submitting the answers of each HIT within minutes (averaged working time less than 90 seconds). Similar to the setting in the collection of CIFAR-ION huge bonus applied to workers who contribute more HIT's than the averaged number of submissions. Workers who submit answers with fixed/regular distribution patterns will be blocked and rejected all submitted results.
Table 5: Newly defined super-classes in CIFAR-100N.
(I) Aquatic mammals
Fi)Fish
Fish
Flowers
(4) Food containers
(5) Pruit, vegetables and mushrooms
(6) Household electricul devices
(7) Household furniture
(8) Insects
(9) Large carnivores and bear
(10) Large man-made outdoor things
(11) Large natural outdoor scenes
(12) Large omnivores and herbivores
(13) Medium-sized mammals
(14) Nou-insect invertebrates
(15) People
(16) Reptiles
(17) Small mammals
(18) Trees
(19) Transportation vehicles
(20) Other vehicles
Table 6: Division of each super-class in CIFAR-1OON.
Super-class
Fine-class
Aquatic mammals
beaver, dolphin, otter, seal, whale aquarium fish, flatfish, ray, shark, trout
Fish
Flowers
orchids, poppies, roses, sunflowers, tulips
Food containers
bottles, bowls, cans, cups, plates
Fruit, vegetables and mushrooms
apples. mushrooms, oranges, pears, sweet peppers
Household electrical devices
clock, computer keyboard, lamp, telephone, television bed, chair, couch, table, wardrohe
Houschold furniture
Insects
bee, beetle, butterfly, caterpillar, cockroach bear, Icopard, lion, tiger, wolf bridge, castle, house, road, skyscraper cloud. forest, mountain, plain, sea camel, cattle, chimpanzee, elephant, kangaroo fox. porcupine, possum, raccoon, skunk crab, lohster, snail, spider, worm baby, boy, girl, man, woman crocodile, dinosaur, lizard, snake, turtle hamster, mouse, rabbit, shrew, squirrel maple, oak, palm, pine, willow bicycle, bus, motorcycle, pickup truck, train, streetear lawn-mower, rocket, tank, tractor
Large carnivores and bear
Large man-made outdoor things
Large natural outdoor scenes
Large omnivores and herbivores
Medium-sized mammals
Non-insect invertebrates
People
Reptiles
Small mammals
Trees
Transportation vehicles
Other vehicles
B.3 MORE DETAILED DATASET STATISTICS
For CIFAR-10ON dataset, each image contains a coarse label and fine label given by a human annotator. Most batches have approximately 40% noisy fine labels and 25 % noisy coarse labels. The overall noise level of coarse and fine labels are 25.60% and 40.20% respectively. A detailed summary of noise level available in Table 7 which covers the statisties of each batch. Human annotators annotate frequently on classes which are outside of the clean-coarse, i.c., 25% noisy labels fall outside of the super-class and 15% inside the super-class.
Table 7: Noise level (%) on 1IFAR-100M
Statistics
Batch1
Batch2
Batch3
Batch4
Batch5
Batch6
Batch?
Batch8
batch9
Batch10
Overall
100-class
40.30
40.76
40.84
40.16
42.50
34,44
43.26
40.34
43.66
35.84
40.20
20-class
26.82
28.02
25.54
25.76
27.02
20.80
28.62
25.56
26.92
20.98
25.60
Imbalanced annotations. The phenomenon of imbalanced annotations also appears substantially as shown in Figure S which presents the distribution of noisy labels for each fine class. "Man" appears > 750 times, while "Streetcar" only has x 200 annotations.
17

---

Published as a conference paper at ICLR 2022
Count 4
Couat
Count
Counr
apple
WOT
poroup no
large comvvoeee and bear
modium size--izdddd
smtell MATETHIE
frall veggeaable mushroom
Count
Oorrt
Count
Nu-Sah
17y
lamp
hause
(boter
non-insact live Seoratee
housebrrld Mechical devices
large man-made outiooor thinga
boos
Count
100
Cuant
Count
Couatt
1002
1.1p
chal
C0031
100
people
household fureeturo
large natural outidoee
vehicles
Count
50
100
O
1000
1990
bwstt
CHE
000 OFF
roplless
Herge crniroonss and hertivores
other vehises
Figure 9: Categorical distribution of noisy labels on CIFAR-10ON: the red line in each subplot indicates that the number of each clean fine class is 500.
Noisy label flips to similar features. In -OFAR-10ON, most fine classes are more likely to be mislabeled into less than four fine classes. In Figure I0 we show top three wrongly annotated fine labels for several fine classes that have relative large noise rate. Due to the low-resolution of images, a number of noisy labels are annotated in pair of classes, i.e, v 20% of "snake" and "worm" images mislabeled between each other, similarly for "oockroack"""beettee some other noisy labels are more frequently annotated within more classes, such as "boy"""baby" "owhale"-"dolphinn"
"fox"-"wolye" etc. While
"girl"-"man",
etc, which share similar features.
<tar
oder
spide
ma1
52
bocka
SAE
bootu
C/ Bide
loxll can
pist:
coud
card
ap
MEARMA
lavg
30
100
75
22 gf
45
40
coppin
fbel
farsew
truses
osk.
NBooy Frwe
Fox
path
FM Yee
02
20
53
380)
doad
Hulp
enchal
otter
poln, Free
popsy
ekyecoger
hemriarr
harohr
coue
M2
pld1
bear
20 40 40 possum
20
40
20
40
63
150
60
porcuptr
rodue
20a
heads
Yout
move
75
100
sisceuper
aquarum Bay
2000
6d
cob Jas D0c
1eato suptin
1000001
Figure 10: Top 3 wrongly annotated fine labels for each selected fine classes. For "pine tree", "shrew". "streetcar", the dominant class is the wrong class. The corresponding number of correct annotations are highlighted with red lines.
18

---

Published as a conference paper at ICLR 2022
Transition matrices of CIFAR-10ON In the class-dependent label noise setting, suppose the label noise is conditional independent of the feature, the noise transition matrices of CIFAR-IOON are best described by mixture of symmetric and asymmetrie T. For CIFAR-1001 coarse and fine noisy labels w.r.t. the clean labels. In Figure [T] the clean label flips into one or more similar classes more often. And the remaining classes follow the symmetric noise model with low noise rate. Apparently, current synthetic class-dependent noisy settings are not as complex as the real-world human annotated label noise.
heatmap the
vepetables runteaurr
houstt ductricl device
lige rar zilgg
lage rakin ouldor scones
Lagn wesveess reriivess
obe votiidea
people
(a) Noisy coarse labels
(b) Noisy fine labels
Figure 11: Transition matrix of CIFAR-100H noisy labels: coarse labels (left) and fine labels (right).
C COMPARISONS OF THE LABEL COLLECTION PROCEDURE
C.I COMPARISONS BETWEEN CIFAR AND CIFAR-N
We summarize the label collection procedure of CIFAR dataset as below:
Images sources: the images of CIFAR come from down-scaled images from 80 million color images obtained by various search engines. The corresponding searching terms are viewed as noisy source label/class;
Remove irrelevant images: the designer provided each student (labeller) with class. Students were asked verify all the images which were found with that class as the search term. And reject images that are outside of the assigned class.
Payment: the payment for each student/labeller per hour is fixed.
Noise rate: with additional personal verification, the noise rate of each class is negligible, espe- cially for CIFAR-10.
We highlight the differences of label collection procedure between CIFAR and CIFAR-N as be- low:
Annotation method: CIFAR adopted search engines such that they obtained images by specifying the class name (conditioned on the label, from label to feature); while CIFAR-N makes use of CIFAR images and pays workers for annotating labels (from feature to label).
Payment and incentives: CIFAR fixed the payment for student workers (self-collected), there is no incentive to rush. As for CIFAR-N, we did not have any constraints on the workers via Amazon Mechanical Turk, i.c. the salary, the education level, etc. Besides, we time limits and bonuses for each annotation task, there are incentives to rush and meanwhile provide high-quality labels.
Noise rate: for CIFAR, the noise rate can be viewed as O. While for CIFAR-N, we provide several noisy label sets which are of various noise rates.
19

---

Published as a conference paper at ICLR 2022
C.2 COMPARISONS BETWEEN CIFAR- 10H AND CIFAR-N
We firstly summarize the main collection procedure of CIFAR-1OH, and then move to discuss the difference in the label collection procedure.
The label collection procedure of CIFAR- 10H dataset given below:
No time limit: participants were asked select the label of each image from the surrounding text with no time limit.
Controlled workload: Each participant is assigned 200 images (20 per class);
Attention check (intervention): Participants with a low accuracy (475%) on (easy-to- sasy-to-recognizz))
images were removed.
Other minor differences: Label positions are shuffled among participants:47-63 annotations per image; payment: 0.0075/10 images.
We want to highlight that the central query for CIFAR- 10H is to understand the benefits of uncertainty in human annotations to improve the generalization power of the trained model. Therefore, a number control the human noise
of controls and interventions were applied when building CIFAR-1OH rates to be not excessive to disturb the above benefit study (details below). We believe this aspect renders the dataset not super appropriate for the relevant studies reported in Section and 5. Our detailed reasons come as follows:
Intervened real-world human noise pattern
Purpose of the dataset construction: CIFAR- 10H targets to identify the benefits from increas- ing the richness of label distributions (hard label - soft label) for image classification tasks. The soft labels are constructed by human uncertainty. CIFAR- -OH may not fully reveal the real-world human annotation noise due to the check and removal procedure as we described above in atlention check. While we aim to study the real-world label noise pattern: we only reject uninformative and spamming annotation patterns (c.g.. labeling every task as class I) and we do not restrict the number of annotations required from different workers with different working efficiency. We accept submissions even if a worker has moderate accuracy (e.g.. <60%) and meanwhile reward workers that contribute large number of annotations.
Noise rate: we randomly select the i-th (e.g.. 1,2...,, annotated label for each test image in CIFAR- 10-H,, and there are approximately 5% wrong labels in the annotation. In CIFAR- ION, the random noise rate is around 18%. For CIFAR- 1O0N, the noise rate increased to 40.20%. CIFAR-IOK has a much smaller noise rate due to the control intervention, and due to different objects the collection. We believe that very low noise rate may deviate from real-world human noise.
Therefore, we think our collection might have better captured the real-world noise patterns.
Training data V.S. Test data
Training on CIFAR- 10 test data may lead to a model performance drop. It is reported that, when trained using the much smaller test data, the generalization accuracy on training data is only about 83% (Peterson et al. (019). Note the standard training and testing on CIFAR-10 has an accuracy about 93%. With added label noise, the substantial drop of the number of training data limits the possibility of fully evaluating the potentials and properties of the competing benchmark methods (c.g., learning and showing some of the established thcoretical properties of particular method might require sufficient number of training data).
Looking forward, we think it might be beneficial to let the have an option of training using 50k training data and testing on the 10k test data, the same and standard way as other learning communities have developed and evaluated algorithms using CIFAR-10 10 data. As we benchmarked in Table 2, most of the existing works are tuned (e.g., pre-trained models for representation extraction for CIFAR-10, etc) for the training with 50k training images. This can help the community better align and calibrate the progress, as compared other learning tasks (e.g., supervised learning, semi-supervised learning, etc).
lahhe community
20

---

Published as a conference paper at ICLR 2022
D HYPOTHESIS TESTING OF CIFAR-N
D.1 HYPOTHESIS TESTING OF CIFAR-1ON
In this section, we include the hypothesis testing results for each class as. As described in Figure 12 the p-value of the human noise label w.r.t each clean class (except for "automobile") in CIFAR- ION is less than 0.05 (*). Most of the classes (except for three of them) achieve a p-value that is smaller than 0.01 (***). Since there exists classes in CIFAR- 10N such that the null hypothesis is rejected with the significance value at, hypothesis "human annotated label noise in CIFAR-1ON feature-dependen" is accepted.
Noise synthetic
n5 0.12971
= --7
1.26-00
5.4e-12 1-100-05 p 1.44-06 0.01864 p 0.00018
0.00077 p p 0.00153
30
Detaanc
20
10
airplane
automablle
cal
dog
frog
horse
ship
Class
Figure 12: Hypothesis testing results: we adopt two samples (( NE(K) and [d. we(s) student t-tests. The p-value and the significance level are shown for each class. Significance levels
are denoted 'ns": p > 0.05; **" p S 0.05; '#*" S 0.01; *#**; p S 0.001; '$****; p S 0.0001. When removed the class constraint, we obtained p = 1.8e 36 for the whole data.
D.2 HYPOTHESIS TESTING OF CIFAR-10ON
We further statistically test whether the human-noise is feature-dependent not in CIFAR-10ON. The null hypothesis Ho and the corresponding alternate hypothesis H, are defined as:
Ho: Human annotated label noise in CIFAR-10ON is feature-indeppnneen::
H, Human annotated label noise in CIFAR- 100N is feature-dependen.
Note that the synthetic label noise is supposed be feature-independent, the above hypotheses are converted to:
Ho Human annotated label noise in CIFAR-100N the same as the corresponding synthetic one;
H, Human annotated label noise in CIFAR-10ON is different from the corresponding synthetic one.
As implemented for CIFAR-ION, we adopt the distance between transition vectors Pi, across different noise measure of the difference between human noise and synthetic noise, e.g., d
human syntheric
Pi [2. As contrast, we need to compare d! with d? == 1p synthetic' -i, 2
synthetic
synthetic' denotes the transition vector from the same synthetic noise but different clustering where P,
result (caused by random data augmentation). Intuitively, if a(1) much greater than d(2) accept H1, The above hypotheses are then equivalent to:
we should
Ho : heEK),ee come from the same distribution as (2) (d[phet(k))eess
()) FrEK),eee come from different distributions from (d See H : fa
We repeat the generation of di,, for 10 times, where the images are modified with different data augmentations each time. We choose the significance level a = 0.05 and perform a two-sided t-test
w.r. {d(.t) (e(Ss) and {d( "iseeieejjeeee ]iE] As described in Figure 13 the p-value of approximately 50 classes is less than a is rejected with the significance value a. And we accept the hypothesis:
0.05. Thus, the null hypothesis for the human noisy labels in CIFAR-IOON
Human annotated label noise in CIFAR-10OX is depeedeee-
21

---

Published as a conference paper at ICLR 2022
10a
WW
p-value
Count
10-14
10-19
10-24
20
60
100
*.. Significance level
Class index
(a) p-valuc of each fine class
(b) Count-ple plot of cach significance level
Figure 13: Hypothesis testing results of CIFAR-1ONN: we adopt two samples (fd (iE(K)) and
{dw (d()) (ej student t-tests. The (a) p-value and the (b) significance level are shown for each class. Significance levels are denoted as 'ns': p > 0.05; *# p I 0.05; *#** p 0.01; *$*8* p S 0.001; *****, 0.0001, When removed the class constraint, = 5.2e-10 16 for the whole data.
shown Figure (b), although the overall noisy fine labels in CIFAR- 100N are feature dependent, the noise transition vectors of around 50 classes indeed be viewed as class dependent by referring to their significance levels. Thus, we can conclude that real-world human annotated noisy labels may be feature independent (class dependent) for certain classes
Beyond feature dependency
E ADDITIONAL EXPERIMENT RESULTS
E. PERFORMANCE COMPARISONS ON CIFAR-N DATASETS
We include larger family of robust methods when learning with CIFAR-N in the Table 8 Note that both ELR+ (Liu et al. 2020) and Divide-Mix (Li et al. 2020) adopt two networks with advanced strategies such as mix-up data augmentation, their performances all other methods. The performance gap between ELR and ELR+ becomes much larger when the noise level is high.
CIFAR-10ON largely outperforms
Table 8: Comparison of test accuracies (%) on CIFAR-1ON and CIFAR-10ON (fine-label) using different methods. Top performances are highlighted in bold (meandsstaddard deviation of runs). SOP (Liu et al. (0222) trained on Pre-act ResNet-18 architecture, while all other methods are reproduced on ResNet-34.
Method
Cleas
Aggrepate
Rondow
CIFAR-TON
Rondom
Rondon
Worst
Clean
CIFAR- TOON
Noiey
Forward
Patrini
Rackwrnd
86
85.86
77.61
57.14
GCE (Zhang & Strunem
2015
92.83
0.16
87.85
0.70
87.61
0.28
87.70
0.55
87.58
0.29
80.66
0.35
76.35
0.48
56.73
Co-tekinng
0115
93.35
0.14
91.20
0.13
90:33
0.13
90.30
0.17
90.15
83.83
0.13
73.46
0.09
60.37
0.27
Co Co-tsaching
2019
92.41
0.20
90.61
0.22
89.70
0.27
89.47
0.18
89.54
0.22
N3.25
0.17
70.99
0.22
57,88
0.24
T-Revision
Xin
2019
93.35
0:23
88.52 + 0.17
83.33
0.32
87.71
1.02
87.79
0.67
S014S
1.20
72.85 - 0.21
51.55
+:3.1 0.31
Peer
88.76
0.6
Positive
1020
91.77
0.17
91.57
0.07
89.50
0.28
89.35
0.33
89.82
82.76
0.53
76.25
0.35
55.84
0.48
F-DiY Mer
1:ll
2314
94
0.12
91.64
0.54
89.70
0.40
89.799
0.12
89.55
0.49
82.55
0.52
76.14
57.10
0.6
Divide-biry Nin
95.37
0.14
95.01
:: 0.71
95.16
0.19
95.23
0.07
95.21
0.14
92.56
0.42
76.94
71.13
0
Negative L$
94.92
0.25
91.97
1 0.95
90.29
0.32
90.37 1: 0.12
90.13
0.19
$2.99
0.36
77.06
0.73
58,59
0.98
JoCaR
We
2010
93.40
0.24
91.44
0.05
90.30
90.21
0.19
90.11
83.37
0.30
74,07
0.33
39.97
0221
CORES
55.72
80.70
88.30
88.10
30.61
$7.80
CAL
(2h0 puree)
94.50
91.97
0.32
90.9
90.75
0.30
90.74
0.24
85.36
0.16
75.67
0.25
61.73
0.42
PES (Seni hesteuee suel
94.76
+ 0.2
94
0.18
95.06
0.15
95.19
0.23
95.22
0.13
92.68
0.22
77.92
0M
70.36
0.33
SOP Litoeie 2004
N/A
95.61 1 0.13
95.28
0.13
95.31
0.10
95.39
0.11
93.24
0.21
NIA
67,81 1 0.23
E.2 PERFORMANCE COMPARISONS ON SYNTHETIC CIFAR DATASETS
Continuing the empirical observations Section 5.1 in this section, we adopt ResNet-34 (He et al. 2016), the same training procedure and batch-size for all implemented methods. The synthetic CIFAR datasets generate synthetic noisy labels by using the same -lass-dependent noise transition matrices in CIFAR-1ON and CIFAR-10ON. The comparisons of test accuracies summarized in Table For
22

---

Published as a conference paper at ICLR 2022
most methods, class deeendet synthetic noise is much easier to learn CIFAR- - especially when the noise level is high. The performance difference between human noise and the synthetic noise is less obvious for CIFAR-100.
Table 9: Comparison of test accuracies (%) on CIFAR-IO and CIFAR-100 (fine-label) with synthetic noisy labels using different methods. Top 3 performances are highlighted in bold (mean-tstandard deviation runs).
Method
Cleas
Aggmggote
Rondow
CIPAR-TO
syatactic Randim
Rondow
Worst
Clron
CIFAR-IOO symthene
Nokcy
(Sannaddidd)
Forward
Patrin et
0:12
92.54
0.20
91.70
01.00
0.21
0.44
0.37
56.87
Rackward
201
91
0.05
92.31
+ 0.28
91:34
0.16
91.14
0.41
91.58
0.14
86.50
0.42
76.79
0.60
$7.68
1.90
GCE (2hang
SErOIco
2015
92.3
0.16
92.44
0.21
91.38
0.17
91.17 1: 0.07
91.49
1 0.19
87.12
0.16
76.35
0.48
57.17
1.51
Co-keazting
2013
93.35
0.14
91.57
+ 0.32
90.99
0.27
90.97
0.24
91.31
0:14
85.74
0.36
73.46
0.09
59.63
0.22
Co-koschngg-
Mr
2019
92.41
0.20
91.50
0.13
0.16 9066
0:46 88.56
90.33
90.59
0.06
85.89
0.25
70,99
= 0.23
57.27
+:0.3
T-Revision
Loss
ELK
0.24
81.94
ELR+
Li
RUAU
99
0.05
94.98
0.15
94.77
0.18
94,60
0.05
94,69
0.14 91.30
87.38
2.66
78.57
0.12
68.46
0.07
Positvo-LLS
0
2020
94.7
0.17
92.13
0.15
91.02
0.53
91.15
0.16
86.71
0.34
76.25
0.35
56.51
+071
[-Die Wet X0
94.88
0.12
92.36
0.41
91:32
0.29
91.12
0.46
91.20
01.20.1100
0.10
86.67
0.38
76.14
0.26
58.41
0.90
Divide Anx
2020
95
0.09
95.45 =: 0.09
95.58
0.07
0.08 95.51
0.23 95.50
0.10
0.14 93.55
0.40
76,94
0.22
71.78
Negative
92.74
0.25
0.28 91.45
59.85 1:15
(Werer
0.21
92.72
VolMinNaT (Freen R0MA
90
0.09
89.54
89.1
0.13
0.15
83.58
0.34
70.61
59.65
CAL (2hu Bell b02401
94.50 0.31
92.22 0.21
90.97
0.35
90.79
0.24
90.83 0.31
85,80
0.33
73.67
0.00
62.20 1 0.67
E.3 EXPERIMENT DETAILS ON CIFAR- 1ON AND CIFAR- 100N
The basic hyper-parameters settings for 1ON and CIFAR-10ON are listed as follows: mini- batch size (128), optimizer (SGD), initial learning (0.1), momentum (0.9), weight decay (0.0005),
number of epochs (100) and learning rate decay (0.1 1 at 50 epochs). Standard data augmentation is applied to each dataset.
Special treatments In the reproduced experiments, we use the default setting for Cores* (Cheng itt al. 2021). ELR+ (Liu et al., (020) and DivideMix since either advanced data augmentation strategies two deep neural networks adopted. And we fix same pre-trained model for methods that require a CE warm-up or a pre-trained model.
E,4 COMPUTING INFRASTRUCTURE
All our experiments run on a GPU cluster (500 GPUs of kinds, mainly use 2080 Ti) for training and evaluation.
23


Reviews:
[{"review_id": "W-KO3Ulc8u", "reviewer": "Reviewer_7fRY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper provides great value to the research community in not just learning from noisy labels, but also data labeling [1]. The paper is easy to follow, and the observations are comprehensive.\n\n**Strength**\n- The paper systematically compares human noise and synthetic noise (including class-dependent label noise and instance-dependent label noise) thoroughly.\n- The paper compares human and synthetic labels in terms of the label distribution and the corresponding model.\n\n**Weakness**\n- (minor) The paper can make some connections with the worker simulations in data labeling. They also adopt a similar class-dependent scheme for worker simulations. [1,2, 3]\n\n\n**Question**\n- In the bottom row of figure 7, are the predictions compared to the given label (wrong labels) or the ground truth labels?\n\n**Other**\n\nSome observations (3 and 5) coincide with the observations in prior work [1]. Prior work [1] considers class-dependent label noise but initializes the label noise with crowdsourced human noise. It also observes that using the human label noise is more challenging in data labeling.\n\n[1] Y. Liao, A. Kar, and S. Fidler. Towards good practices for efficiently annotating large-scale image classification datasets, CVPR2021\n\n[2] G. Hua, C. Long, M. Yang, and Y. Gao. Collaborative active learning of a kernel machine ensemble for recognition, CVPR2013\n\n[3] C. Long and G. Hua. Multi-class multi-annotator active learning with robust gaussian process for visual recognition, CVPR2015\n", "summary_of_the_paper": "The paper provides an extensive study on 1) the difference between human annotations and synthetic ones and 2) the impact on the classifier learning from human annotations and synthetic ones. To this end, the paper crowdsources the human annotations for CIFAR10 and CIFAR100 from Amazon Mechanical Turk. The observations show that human annotations are more challenging to learn, and human annotations result in a different memorization scheme for neural networks.", "summary_of_the_review": "The paper provides great value in the research communities that focus on the impact of label noise. The paper is well written and easy to follow. The observations shown in the paper are supported by extensive analysis. There are only some parts that require more explanation, but it does not harm the overall readability.\nTo make the paper more complete, I suggest the author make more connections toward a broader community that focus on label noise.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TYB2yGHkGDd", "reviewer": "Reviewer_EZud", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has several strengths although there is no technical novelty. In particular, the analysis with human-annotated label noise (using CIFAR-10N and CIFAR-100N) provides some insightful information. However, I have some critical concerns below:\n\nFirst of all, I agree with the absence of controllable real noisy data, which can be used popularly for fair comparison or in-depth analysis (satisfying the three conditions: resolution, clean labels, and no interventions). For this purpose, CIFAR-10N and CIFAR-100N will be good reference data for research. However, I wonder if the data with such a small resolution (32x32) can properly reflect real noise. In real-world applications, the resolution of training images is mostly larger than 32x32, and the corrupted labels happen even with high-resolution images (e.g., medical images or data with fine-grained classes). I think that the restriction of the low resolution (actually, 32x32 is not moderate resolution) can generate label noise attributed to ambiguity in the scene (e.g., color information could be dominant than appearance information in labeling). Therefore, the collected noisy labels may be different from the noise label in a real real-world scenario.\n\nSecond, the observation of imbalanced annotation is quite interesting. Owing to this imbalanced annotation, I think CIFAR-10N and CIFAR-100N have class imbalance problems. Can authors report the imbalance ratio of the two proposed datasets? If the ratio is low, the results in performance evaluation are convincing. However, if the ratio is high, the low performance of robust approaches can be due to the more complicated scenario of 'class imbalance + label noise'. Similarly, in all other analyses, I am wondering if the class imbalance factor was properly handled. \n\nThird, many readers will be interested in why the existing approaches show poor performance on human noise. The table summarizes many values but I feel hard to reason why ELR achieves the best performance consistently. In addition, why does the DivideMix shows the worst performance in the CIFAR-10N aggregate dataset? There is no discussion and analysis on these points. I know, that human noise is like a feature-dependent noise, thus it is much harder than the synthetic noise setup. But, why do all methods show inconsistent trends on that?\n\nThese are some minor comments:\n- Clothing1M data has a 38% noise level (see the paper [1], the authors reported the clean accuracy of labeling was 61.54%)\n[1] Learning from massive noisy labeled data for image classification, CVPR'15\n- The explanation of Figure 7 is very confusing. e.g., what is the 'wrong' prediction on 'wrong 'labels? and there is no f_M in the paper (see the caption for Figure 7) Is it f_H?\n", "summary_of_the_paper": "This paper focuses on learning from noisy labels, which is a very important research topic in the ML community. One of the main challenges in this area has been the absence of controllable real-world noisy data. The authors present the variation of CIFARs data with human annotations via MTurk.  By comparing the results on the original CIFARs and the newly proposed CIFAR-N, four observations are detailed quantitatively or qualitatively. In addition, they provide the performance comparison of more than 10 approaches on CIFAR-10 and -100N. The analysis of the memorization effect in human-annotated noisy data is also discussed.", "summary_of_the_review": "As a paper that proposes insightful analysis and some datasets, there is room for further improvement. In particular, resolving the above concerns will add more clarity and novelty to the paper. I will decide my final rating after the rebuttal.\n\n> [rebuttal] The score increases 5 $\\rightarrow$ 6 since the response addressed my concerns properly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HPOC_LeYnPU", "reviewer": "Reviewer_q5d1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The paper highlights an important issue in the literature of label-noise-learning: synthetic noise is prevalently used to evaluate methods. This paper shows that real-world noise pattern is different from synthetic noise, and the datasets proposed could be useful as a smaller-scale benchmark for future researchers. The paper is also well-written in general.\n\nWeaknesses: \n1. Even though the proposed datasets offer an effective benchmark for researchers to conduct smaller-scale experiments, its significance is not enough. There already exists larger-scale real-world noisy datasets such as Food-101 and WebVision. Because the ultimate goal of label-noise-learning is to learn from weakly-labeled web data, these existing web datasets are sufficient for model evaluation. The paper argues that these datasets do not have clean training labels. However, it should not be a problem because they do have a clean test set for fair evaluation. Therefore, in my opinion, the proposed datasets do not address a research gap.\n2. Besides the dataset, the paper does not offer other significant contributions. It can be expected that real-world noise has different distribution compared to synthetic noise. As a dataset paper without any technical novelty, I do not find this paper to provide much new insights that could guide future research directions in this field.", "summary_of_the_paper": "This paper proposes two smaller-sized datasets with real-world label noise. The datasets contain CIFAR-10 and CIFAR-100 images with human-annotated labels. The paper analysis the noise pattern, and benchmarks existing methods on these datasets.", "summary_of_the_review": "While I find this paper to have a clear strength, the contribution is not enough to justify acceptance to ICLR. The dataset proposed is good but not crucial, and the paper does not provide enough new insights. This paper might have a better chance at other venues specific for dataset papers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6Xu98NBQ9qi", "reviewer": "Reviewer_PLPg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is clear and well-written.\n- The annotation collection procedure was clearly described.\n- A large number of baselines were implemented and compared in Section 5.1.\n\nWeaknesses:\n- How does the annotation collection procedure differ from the original CIFAR10 & CIFAR100? Are there any important differences or modifications to the process the authors made? These points should be addressed.\n- The \"qualitative\" section on instance-dependent noise comparisons is confusing and does not help the overall flow of the paper. Figure 5 is more confusing than helpful, and there is no real \"conclusion\" from the subjective analysis until we get to the quantitative section with the hypothesis test.\n\n-- UPDATE after rebuttal --\nThe authors have sufficiently addressed my concerns regarding the differences with respect to CIFAR-10H. Thus, I will raise my score from 6 to 8 with the understanding that the authors will update the paper to include the information they provided in their rebuttal reply.", "summary_of_the_paper": "This paper collects additional human annotations for Cifar10 and Cifar100. These annotations are also compared to synthetic label noise alternatives.", "summary_of_the_review": "The paper collects additional manual annotations for CIFAR10 and CIFAR100, which will be of great value to the research community, for ex. helping practitioners study the robustness of their algorithms to label noise.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose two new benchmark datasets CIFAR-10-N and CIFAR-100-N, variants of CIFAR-10 and CIFAR-100 with real-world human annotation noise. The benchmark datasets are more realistic (e.g. instance-dependent noise) than some existing synthetic benchmarks for label noise. The authors also benchmark several popular baselines on the proposed benchmark

All the reviewers thought that this is an useful contribution to the community and appreciated the detailed author response. The consensus decision leaned towards accept. I recommend acceptance & encourage the authors to address any remaining concerns in the final version. 
Please clarify the license (e.g. MIT license) when you release the dataset.
Decision: Accept (Poster)