Year: 2021

Abstract:
Existing research on learning with noisy labels mainly focuses on synthetic label noise. Synthetic label noise, though has clean structures which greatly enabled statistical analyses, often fails to model the real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, e.g., Food101N, WebVision, and Clothing1M. Yet the existing efforts suffer from two caveats: firstly, the lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise. Secondly, these efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is important to establish controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets, which we name as CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10 and CIFAR-100 with human-annotated real-world noisy labels that we collect from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions.

heading: 1 INTRODUCTION
text: Image classification task in deep learning requires assigning labels to specific images. Annotating labels for training use often requires tremendous expenses on the payment for hiring human annotators. The pervasive noisy labels from data annotation present significant challenges to training a quality machine learning model. The problem of dealing with label noise has been receiving increasing attentions. Typical approaches include unbiased estimators and weighted loss functions (Natarajan et al., 2013; Liu & Tao, 2015), loss correction (Patrini et al., 2017; Liu & Guo, 2020), sample selection aided (Jiang et al., 2018; Han et al., 2018; Yu et al., 2019), etc. The majority of existing solutions are often developed under stylish synthetic noise model, where the noise rates are either class-dependent or homogeneous across data instances. Recently, there has been efforts for establishing results for the more realistic instance-dependent setting, i.e, the noise transition differs in different parts of features (Xia et al., 2020b) or sub-populations (Wang et al., 2021; Zhu et al., 2021a), etc. All of these noise models are proposed with some statistical assumptions, which facilitate the derivation of theoretical solutions. Yet, it is unclear how the existing models captured the real-world scenario.
To empirically validate the robustness of proposed methods, synthetic noisy labels on CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) are the most widely accepted benchmarks. The literature has also observed approaches to the simulation of human annotators in data labeling (Hua et al., 2013; Long & Hua, 2015; Liao et al., 2021), and real-world label noise benchmarks, including Food-101 (Bossard et al., 2014), Clothing-1M (Xiao et al., 2015), WebVision (Li et al., 2017), etc. We summarize the above real-world noisy label datasets in Table 1. While a more detailed description
and discussion of the existing datasets can be found in the related works, we want to highlight several outstanding issues in existing benchmarks and evaluations. As noted in Table 1, except for CIFAR related noisy label datasets, all other datasets suffer from at least one of the three caveats:
• Complex task (High-resolution): when learning with large-scale and relative high-resolution data, the complex data pattern, various augmentation strategies, the use of extra train or clean data, different computation power (for hyper-parameter tuning such as batch-size, learning rate, etc) jointly contribute to the model performance and then result in unfair comparison.
• Missing clean labels: the lack of clean labels for verification in most existed noisy-label datasets makes the evaluation of robust methods intractable.
• Interventions: human interventions in data generation (Jiang et al., 2020) and non-representative data collection process (Song et al., 2019) might disturb the original noisy-label pattern.
In addition, despite synthetically labeled CIFAR datasets are popular and highly used benchmarks for evaluating the robustness of proposed methods, there exists no publicly available human annotated labels for CIFAR training datasets to perform either validation of existing methods or verification of popular noise models 1. A human-annotated version of CIFAR datasets would greatly facilitate the evaluations of existing and future solutions, due to the already standardized procedures for experimenting with CIFAR datasets. All above issues motivate us to revisit the problem of learning with noisy labels and establish accessible and easy-to-use, verifiable datasets that would be broadly usable to the research community. Our contributions can be summarized as follows :
• We present two new benchmarks CIFAR-10N, CIFAR-100N which provide CIFAR-10 and CIFAR100 with human annotated noisy labels. Our efforts built upon the CIFAR datasets and provide easily usable benchmark data for the weakly supervised learning community (Section 3). We expect to continue to maintain the datasets to facilitate future development of results.
• We introduce new observations for the distribution of human annotated noisy labels on tiny images, i.e., imbalanced annotations, the flipping of noisy labels among similar features, co-existence of multiple clean labels for CIFAR-100 train images (which leads to a new pattern of label noise), etc. We further distinguish noisy labels in CIFAR-10N and CIFAR-100N with synthetic classdependent label noise, from the aspect of noise transitions for different features qualitatively and quantitatively (via hypothesis testing) (Section 4).
• We empirically compare the robustness of a comprehensive list of popular methods when learning with CIFAR-10N, CIFAR-100N. We observe consistent performance gaps between human noise and synthetic noise. The different memorization behavior further distinguishes the human noise and synthetic noise (Section 5).

heading: 1.1 RELATED WORKS
text: Learning from noisy labels Earlier approaches for learning from noisy labels mainly focus on loss adjustment techniques. To mitigate the impact of label noise, a line of approaches modify the loss of image samples by multiplying an estimated noise transition matrix (Patrini et al., 2017; Hendrycks et al., 2018; Xia et al., 2019; Yao et al., 2020), re-weight the loss to encourage deep neural nets to
1CIFAR10H (Peterson et al., 2019) only provides test images with noisy human annotations.
fit on correct labels (Liu & Tao, 2015), propose robust loss functions (Natarajan et al., 2013; Ghosh et al., 2017; Zhang & Sabuncu, 2018; Amid et al., 2019; Wang et al., 2019; Liu & Guo, 2020), or introduce a robust regularizer (Liu et al., 2020; Xia et al., 2020a; Cheng et al., 2021; Wei et al., 2021). Another line of popular approaches behaves like a semi-supervised manner which begins with a clean sample selection procedure, then makes use of the wrongly-labeled samples. For example, several methods (Jiang et al., 2018; Han et al., 2018; Yu et al., 2019) adopt a mentor/peer network to select small-loss samples as “clean” ones for the student/peer network. To further explore the benefits of wrongly-labeled samples and improve the model performance, Li et al. (2020) chose the Mix-match (Berthelot et al., 2019) technique which has shown success in semi-supervised learning.
Benchmarks noisy labels datasets Food-101 (Bossard et al., 2014), Clothing-1M (Xiao et al., 2015), WebVision (Li et al., 2017) are three large-scale noisy labeled web-image databases which consist of food images, clothes images or other web images, respectively. However, the majority of images in these three datasets do not have a corresponding clean label to perform controlled verification (e.g., verifying the noise levels). Later, a much larger-scale food dataset is collected by Lee et al. (2018), which contains exactly the same classes as Food-101 (Bossard et al., 2014). More recently, Peterson et al. (2019) present a noisily labeled benchmarks on CIFAR-10 test dataset where each test image has 51 human annotated labels in average. Jiang et al. (2020) construct noisily labeled Mini-ImageNet (Vinyals et al., 2016) and Stanford Cars datasets (Krause et al., 2013) with controlled noise levels by substituting human annotated incorrect labels for synthetic wrong labels.

heading: 2 SYNTHETIC LABEL NOISE
text: In this section, we discuss a few popular synthetic models for generating noisy labels. We focus on a K-class classification task. Denote by D := {(xn, yn)}n∈[N ] the training samples where [N ] := {1, 2, ..., N}. (xn, yn)s are given by random variables (X,Y ) ∈ X ×Y drawn from the joint distributionD, where X ,Y can be viewed as the space of feature and label, respectively. In real-world scenarios, a classifier f only has access to noisily labeled training sets D̃ := {(xn, ỹn)}n∈[N ]. We assume the noisy samples (xn, ỹn)s are given by random variables (X, Ỹ ) ∈ X ×Ỹ which are drawn from the joint distribution D̃. Clearly, there may exist n ∈ [N ] such that yn 6= ỹn. The flipping from clean to noisy label is usually formulated by a noise transition matrix T (X), with elements: Ti,j(X) := P(Ỹ = j|Y = i,X). We shall specify different modeling choices of T (X) below.

heading: 2.1 CLASS-DEPENDENT LABEL NOISE
text: The first family of noise transition matrix is the class-dependent noise where the label noise is assumed to be conditionally independent of the feature X . Mathematically, T (X) ≡ T and
Ti,j(X) = P(Ỹ = j|Y = i),∀i, j ∈ [K].
Symmetric T The symmetric noise transition matrix (Natarajan et al., 2013) describes the scenario where an amount of human labelers maliciously assign a random label for the given task. It assumes that the probability of randomly flipping the clean class to the other possible class with probability . Assume the noise level is , the diagonal entry of the symmetric T is denoted as Ti,i = 1− . For any other off-diagonal entry Ti,j where i 6= j, the corresponding element is Ti,j = /(K − 1). Asymmetric T The asymmetric noise transition matrix (Patrini et al., 2017) simulates the case where there exists ambiguity classes, i.e, human labelers may wrongly annotate the truck as automobile due to the low-resolution images. There are two types of widely adopted asymmetric T . The assymetric-next T assumes that the clean label flips to the next class with probability , i.e, i→ (i+ 1) mod K for i ∈ [K]. The assymetric-pair T considers [K/2] disjoint class pairs (ic, jc) where ic < jc. For c ∈ [K/2], Tic,jc = Tjc,ic = , and the diagonal entries are 1− .

heading: 2.2 INSTANCE-DEPENDENT LABEL NOISE
text: Beyond the feature-independent assumption, recent works pay more attention to a challenging case where the label noise is jointly determined by featureX and clean label Y . There are some techniques for synthesizing the instance-dependent label noise, such as the polynomial margin diminishing label noise (Zhang et al., 2021b) where instances near decision boundary are easier to be mislabeled,
the part-dependent label noise (Xia et al., 2020b) where different parts of feature may contribute different noise transition matrices, and the group-dependent label noise (Wang et al., 2021; Zhu et al., 2021a) where different sub-populations may have different noise rates. All of these noise models are proposed with some statistical assumptions, which facilitate the derivation of theoretical solutions.

heading: 3 HUMAN ANNOTATED NOISY LABELS ON CIFAR-10, CIFAR-100
text: In this section, we introduce two new benchmark datasets for learning with noisy labels: CIFAR10N and CIFAR-100N. Both datasets are built using human annotated labels collected on Amazon Mechanical Turk (M-Turk): We post CIFAR-10 and CIFAR-100 training images as the annotation Human Intelligence Tasks (HITs), and workers receive payments by completing HITs.

heading: 3.1 CIFAR-10N REAL-WORLD NOISY LABEL BENCHMARK
text: CIFAR-10 (Krizhevsky et al., 2009) dataset contains 60k 32 × 32 color images, 50k images for training and 10k images for testing. Each image belongs to one of ten completely mutually exclusive classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.
Dataset collection We randomly split the training dataset of CIFAR-10 without replacement into ten batches. In the Mturk interface, each batch contains 500 HITs with 10 images per HIT. The training images and test dataset remain unchanged. Each HIT is then randomly assigned to three independent workers. Workers gain base reward $0.03 after submitting the answers of each HIT. We reward workers with bonus salary if the worker contributes more HITs than the averaged number of submissions. We did not make use of any ground-truth clean labels to approve or reject submissions. We only block and reject workers who submit answers with fixed/regular distribution patterns. We defer more details of the dataset collection to Appendix A.
Dataset statistics For CIFAR-10N dataset, each training image contains one clean label and three human annotated labels. We provide five noisy-label sets as follows. • Aggregate: aggregation of three noisy labels by majority voting. If the submitted three labels are
different for an image, the aggregated label will be randomly selected among the three labels. • Random i (i ∈ {1, 2, 3}): the i-th submitted label for each image. Note our collection procedure
ensures that one image cannot be repeatedly labeled by the same worker. • Worst: dataset with the highest noise rate. For each image, if there exist any wrongly annotated
labels in three noisy labels, the worst label is randomly selected from wrong labels. Otherwise, the worst label is equal to the clean label.
60.27% of the training images have received unanimous label from three independent labelers. The noise rates of prepared five noisy label sets are 9.03% (Aggregate), 17.23% (Random 1), 18.12% (Random 2), 17.64% (Random 3) and 40.21% (Worst). A complete dataset comparison among existing benchmarks and ours are given in Table 1. We defer the noise level of each batch to Table 4 (Appendix). Aggregating the annotated labels significantly decreases the noise rates. All three random sets have ≈ 18% noise level. To provide a challenging noisy setting, we also prepare a worst label set which serves to cover highly possible mistakes from human annotators on CIFAR-10.

heading: 3.2 CIFAR-100N REAL-WORLD NOISY LABEL BENCHMARK
text: CIFAR-100 (Krizhevsky et al., 2009) dataset contains 60K 32× 32 color images of 100 fine classes, 50000 images for training and 10000 images for testing. Each (fine) class contains 500 training images and 100 test images. The 100 classes are grouped into 20 mutually exclusive super-classes.
Data collection We split the training dataset of CIFAR-100 without replacement into ten batches with five images per HIT. Only one worker is assigned to each HIT. We group the 100 classes into 20 disjoint super-classes (see Table 5 in the Appendix) which are slightly different from the 20 “coarse” categories named in CIFAR-100. The fine label in each super-class is summarized in Table 6 (Appendix). Workers are instructed to firstly select the super-class for each image. And will then be re-directed to the best matched fine label. Since some super-classes are hard to recognize without prior knowledge in biology, we provide workers with easy access to re-select the super-class, and every fine class has an example image for references. The rejecting rule and the bonus policy are the same as those in CIFAR-10N. We defer more details of the dataset collection to Appendix B.
Dataset statistics For CIFAR-100N dataset, each image contains a coarse label and a fine label given by a human annotator. Most batches have approximately 40% noisy fine labels and 25 % noisy coarse labels. The overall noise level of coarse and fine labels are 25.60% and 40.20%, respectively. We defer the noise level of each batch to Table 7 (Appendix).

heading: 4 PRELIMINARY OBSERVATIONS ON CIFAR-10N, CIFAR-100N
text: In this section, we analyze the human annotated labels for CIFAR-10 and CIFAR-100. We will empirically compare the noisy labels in CIFAR-10N and CIFAR-100N with class-dependent label noise both qualitatively and quantitatively.

heading: 4.1 THE NOISY LABEL DISTRIBUTION
text: Observation 1: Imbalanced annotations Our first observation is the imbalanced contribution of labels. Note that while the number of images are the same for each clean label, across all the five noisy label sets of CIFAR-10N, we observe that human annotators have different preferences for similar classes. For instance, they are more likely to annotate an image to be an automobile rather than the truck, to be the horse rather than the deer (see Figure 9 in the Appendix). The aggregated labels appear more frequently in automobile and ship, and less frequently in deer and cat. This gap of frequency becomes more clear in the worst label set. In CIFAR-100N, human annotators annotate frequently on classes which are outside of the clean-coarse, i.e., 25% noisy labels fall outside of the super-class and 15% inside the super-class. And the phenomenon of imbalanced annotations also appears substantially as shown in Figure 1, which presents the distribution of noisy labels for each selected fine class. “Man” appears ≥ 750 times, while “Streetcar” only has ≈ 200 annotations.
Observation 3: The pattern of noise transition matrices In the class-dependent label noise setting, suppose the label noise is conditional independent of the feature, the noise transition matrices of CIFAR-10N and CIFAR-100N are best described by a mixture of symmetric and asymmetric T . For CIFAR-10N, we heatmap the aggregated noisy labels, random1 noisy labels and worst noise labels w.r.t. the clean labels. In Figure 3, the three noisy label sets share a common pattern: the clean label flips into one or more similar classes more often. The remaining classes largely follow the symmetric noise model with a low noise rate. For example, “truck” and “automobile” flip between each other more often (≈ 25%− 30% percentage), which is much larger than that of all other classes. Besides, in the central area of each transition matrix, it is quite obvious that the clean label of animal classes flips more often to other animals. Similar observations hold in CIFAR-100N, where each class flips to a few misleading classes with much higher probability than that of remaining ones (see Figure 12 in the Appendix). Apparently, current synthetic class-dependent noisy settings are not as complex as the real-world human annotated label noise.
Observation 4: Label noise: bad news or good news? During the label collection, there exist a non-negligible amount of the wrongly annotated classes that indeed co-exist in the corresponding images. In other words, training images of CIFAR-100 may contain multiple labels rather than a single one. We select several exemplary training images of CIFAR-100 where multiple labels appear (in Figure 4). The annotated class also appears in the corresponding image while is deemed as a wrong annotation by referring to the officially provided clean label. The most frequent case is best described by the scenario where a man holding a flatfish in hands. The clean label usually comes to “flatfish”, while human annotators are more likely to categorize these images into “man”. We conjecture that with the increasing label dimension, the phenomenon of multiple clean labels might be a more common issue. We leave more explorations for the future work.

heading: 4.2 HUMAN NOISY LABELS V.S. SYNTHETIC NOISY LABELS
text: Recall that, for the class-dependent label noise, we have P(Ỹ |X,Y ) = P(Ỹ |Y ), indicating the noise transitions are identical for different features. For general instance-dependent label noise, the above equality may not hold. In this subsection, we explore to what degree this feature-dependency holds by checking whether the equality is satisfied or not for different features. In the following, by checking the equality for different features, we will show the feature-dependency from both a qualitative aspect and a quantitative aspect.

heading: 4.2.1 A QUALITATIVE ASPECT
text: We first visualize the noise transitions for different features from a qualitative aspect. Taking CIFAR10N as an example, each image in CIFAR-10 only appears once. Without additional assumptions, we only have three noisy labels for each individual feature X , which makes it difficult to accurately estimate the noise transition probability T (X),∀X , even though the ground-truth clean label is available. To make the estimation tractable, we consider estimations on locally homogeneous label noise, i.e., nearby features share the same T (X). See the rigorous definition as follows.
Definition 1 (M -NN noise clusterability) (Zhu et al., 2021b) We call D̃n satisfies M -NN (M - Nearest-Neighbor) noise clusterability if the M -NN of xn have the same noise transition matrix as xn, i.e., T (xn) = T (xni),∀i ∈ [M ] where {xni}i∈[M ] denote the M -NN of xn.
With M -NN noise clusterability, we can estimate T (X) on each subset D̃n. In our visualization (e.g., Figure 5), rather than manually fix a particular M , we use the k-means algorithm to separate features belonging to the same true class to 5 clusters and adaptively find a suitable M for each cluster. Denote by Ii,ν the set of instance indices from the ν-th cluster of clean class i. Then for label
1 2 3 4 5 6 7 8 9 10
0
2
4 clu
st er
class 1 (airplane)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 2 (automobile)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 3 (bird)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 4 (cat)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 5 (deer)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
clu st
er
class 6 (dog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 7 (frog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 8 (horse)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 9 (ship)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 10 (truck)
1 2 3 4 5 6 7 8 9 10
0
2
4
clu st
er
class 1 (airplane)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 2 (automobile)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 3 (bird)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 4 (cat)
1 2 3 4 5 6 7 8 9 10
0.0
2.5
class 5 (deer)
0
1
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
clu st
er
class 6 (dog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 7 (frog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 8 (horse)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 9 (ship)
1 2 3 4 5 6 7 8 9 10 noisy class
0.0
2.5
class 10 (truck)
0
1
(a) Human noise (Random 1): noise level ≈ 17.23%
1 2 3 4 5 6 7 8 9 10
0
2
4
clu st
er
class 1 (airplane)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 2 (automobile)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 3 (bird)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 4 (cat)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 5 (deer)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
clu
st
er
class 6 (dog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 7 (frog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 8 (horse)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 9 (ship)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 10 (truck)
1 2 3 4 5 6 7 8 9 10
0
2
4
clu st
er
class 1 (airplane)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 2 (automobile)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 3 (bird)
1 2 3 4 5 6 7 8 9 10
0
2
4
class 4 (cat)
1 2 3 4 5 6 7 8 9 10
0.0
2.5
class 5 (deer)
0
1
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
clu st
er
class 6 (dog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 7 (frog)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 8 (horse)
1 2 3 4 5 6 7 8 9 10 noisy class
0
2
4
class 9 (ship)
1 2 3 4 5 6 7 8 9 10 noisy class
0.0
2.5
class 10 (truck)
0
1
(b) Synthetic class-dependent noise with the same T
Figure 5: Illustration of noise transitions of human-level label noise and the synthetic version. We divide the representations of images from the same true class into 5 clusters by k-means. The representations come from the output before the final fully-connected layer of ResNet34. The model is trained on clean CIFAR-10. Negative cosine similarity measures the distance between features.
noise in Ii,ν , we assume it is feature-independent and denote the corresponding transition vector by pi,ν , where each element pi,ν [j] is expected to be P(Ỹ = j|xn, n ∈ Ii,ν , Y = i). The transition vector pi,ν can be estimated by counting the frequency of each noisy class given noisy labels in Ii,ν . For example, in Figure 5a, each row of the top-right subfigure titled “Class 5 (deer)” shows P(Ỹ |X,Y = 5) for different clusters of X , i.e., transition vectors p5,ν , ν ∈ {1, · · · , 5}. Before putting up a more formal testing, we clearly observe that different transition vectors across different feature clusters, signaling the fact that the noise transitions P(Ỹ |X,Y = 5) are feature-dependent. For a controlled comparison, we synthesize the class-dependent label noise with the same expected noised transition matrix T := E[T (X)] as our human-level label noise and illustrate it in Figure 5b. We can find that different rows of each matrix in Figure 5b are very similar, showing the synthetic noise is feature-independent and our verification method is valid.

heading: 4.2.2 A QUANTITATIVE ASPECT
text: In this section, we quantitatively compare human noise and synthetic class-dependent noise through hypothesis testing.
Formulation Following the clustering results in Section 4.2.1, we further statistically test whether the human-noise is feature-dependent or not. For CIFAR-10N, the null hypothesis H0 and the corresponding alternate hypothesis H1 are defined as:
H0 : Human annotated label noise in CIFAR-10N is feature-independent; H1 : Human annotated label noise in CIFAR-10N is feature-dependent.
Note that the synthetic label noise illustrated in Figure 5b is supposed to be feature-independent, the above hypotheses are converted to:
H0 : Human annotated label noise in CIFAR-10N is the same as the corresponding synthetic one; H1 : Human annotated label noise in CIFAR-10N is different from the corresponding synthetic one.
From Figure 5, one measure of the difference between human noise and synthetic noise is the distance between transition vectors pi,ν across different noise, e.g., d (1) i,ν := ‖phumani,ν − p synthetic i,ν ‖22. As contrast, we need to compare d(1)i,ν with d (2) i,ν := ‖p synthetic′ i,ν − p synthetic i,ν ‖22, where p synthetic′
i,ν denotes the transition vector from the same synthetic noise but different clustering result (caused by random data augmentation). Intuitively, if d(1)i,ν is much greater than d (2) i,ν , we should accept H1. The above hypotheses are then equivalent to:
H0 : {d(1)i,ν }i∈[K],ν∈[5] come from the same distribution as {d (2) i,ν }i∈[K],ν∈[5];
H1 : {d(1)i,ν }i∈[K],ν∈[5] come from different distributions from {d (2) i,ν }i∈[K],ν∈[5].
We repeat the generation of di,ν for 10 times, where the images are modified with different data augmentations each time. We choose the significance level α = 0.05 and perform a two-sided t-test w.r.t {d(1)i,ν }i∈[K],ν∈[5] and {d (2) i,ν }i∈[K],ν∈[5]. As described in Figure 6, the p-value of the human noise
label w.r.t each clean class (except for “automobile") in CIFAR-10N is less than 0.05 (*). Most of the classes (except for three of them) achieve a p-value that is smaller than 0.01 (***). Since there exists classes in CIFAR-10N such that the null hypothesis is rejected with the significance value α, hypothesis “human annotated label noise in CIFAR-10N is feature-dependent” is accepted. Similar conclusion can be reached for CIFAR-100N (details in Appendix D).

heading: 5 LEARNING WITH CIFAR-10N AND CIFAR-100N
text: We reproduce several popular and state-of-the-art robust methods on synthetic noisy labels (using calculated transition matrices from our collected data) as well as our collected human annotated labels. Most of the selected methods fall into loss correction, loss re-weighting, and loss regularization related methods, etc.

heading: 5.1 PERFORMANCE COMPARISONS ON CIFAR-10N AND CIFAR-100N
text: For a fair comparison, we adopt ResNet-34 (He et al., 2016), the same training procedure and batch-size for all implemented methods. More experiment details are deferred to the Appendix E. In Table 2, ELR+ (Liu et al., 2020) maintains its state-of-the-art in most noise settings. Note that both ELR+ (Liu et al., 2020) and Divide-Mix (Li et al., 2020) adopt two networks with advanced strategies such as mix-up data augmentation, their performances on CIFAR-100N largely outperforms all other methods. The performance gap between ELR and ELR+ becomes much larger when the noise level is high. We also empirically test the performances of the above methods under the class-dependent noise settings which follow exactly the same T as appeared in CIFAR-10N and CIFAR-100N, details are deferred to Table 8 in the Appendix.
Observation 5: Performance gap (human noise v.s. synthetic noise) Continuing the reported observations in Section 4.1, in Table 3 we highlight that for most selected methods, class-dependent synthetic noise is much easier to learn on CIFAR-10, especially when the noise level is high. The gap is less obvious for CIFAR-100. However, we also observe that Divide-Mix (Li et al., 2020) fails to work well in low noise regime. Besides, ELR (Liu et al., 2020) performs even slight better when learning with real-world human noise than that on the synthetic class-dependent noise settings.

heading: 5.2 MEMORIZATION EFFECTS
text: When learning with noisy labels on CIFAR-10 and CIFAR-100 datasets, empirical observations (Arpit et al., 2017; Liu et al., 2020; Xia et al., 2020a; Zhang et al., 2021a) on synthetic noise settings suggest that deep neural networks firstly fit on samples with clean labels, then gradually over-fit and memorize samples with wrong/noisy labels. We next explore the memorization of clean and noisy labels on CIFAR-10N and CIFAR-100N. Definition 2 (Memorized feature) In a K-class classification task, given a trained classifier f , a feature x and confidence threshold η, x is memorized by f if ∃i ∈ [K] s.t. P(f(x) = i) > η. In Figure 7, we train CE loss with a ResNet-34 (He et al., 2016) neural network on three noisy label sets of CIFAR-10N: aggre-label (left column), random-label1 (middle column) and worst-label (right column). While visualizing the memorization (η = 0.95) on training samples, we split the train data into two parts: images with clean labels (the annotation matches the clean label) and wrong labels (the rest). Denote fH and fS as models trained on human noisy labels and synthetic noisy labels (with the same T ), respectively. fS gradually memorizes both correct and wrong predictions, while fH only memorizes correct predictions on clean labels and wrong predictions on wrong labels.
Why memorizing effects differ Compared with synthetic label noise, human annotators are prone to providing wrong labels on more misleading/ambiguous images or complex patterns. Therefore we conclude that, given the same noise level, learning with human noise labels is more challenging and deep neural nets over-fit on features of wrong annotations inevitably.

heading: 6 CONCLUSIONS
text: Building upon CIFAR datasets, we provide the weakly supervised learning community with two accessible and easy-to-use benchmarks: CIFAR-10N and CIFAR-100N. We introduce new observations from human annotations such as imbalanced annotations, the flipping of noisy labels among similar features, co-existing labels in CIFAR-100N, etc. From the perspective of noise transitions, we qualitatively show that human noise is indeed feature-dependent and differs substantially from synthetic class-dependent label noise using hypothesis testing. We empirically compare the robustness of a large quantity of popular methods when learning with CIFAR-10N, CIFAR-100N and synthetic noisy CIFAR datasets. We also consistently observe the large performance gap between human noise and synthetic noise, as well as the different memorization behavior on training samples.

heading: ETHICS STATEMENT
text: This paper does not raise any ethics concerns. Our work contains the human subject study which involves only simply image annotation tasks in Amazon Mechanical Turk. The study has been carefully reviewed and received Institutional Review Board (IRB) approval. We have followed the outlined protocol to perform the data collection. Our implemented human subject studies:
• Do not have negative consequences, i.e., leaking the private information that would identify human annotators on Amazon Mechanical Turk.
• Do not misrepresent work that might be competing or related. • Do not present misleading insights. Our work does not present applications that can lead to misuse. • Do not introduce bias or fairness concerns, and research integrity issues. • Will publish our collected datasets once there are no conflicts of interest/sponsorship and accepted.

heading: APPENDIX
text: The Appendix is organized as follows:
• Section A: the details of dataset collection, processing and information of CIFAR-10N. • Section B: the details of dataset collection, processing and information of CIFAR-100N. • Section C: the comparison of label collection procedure between CIFAR and CIFAR-N. • Section D: hypothesis testing of CIFAR-100N. • Section E: additional experiment results and details.
Before proceeding to the appendix, we want to highlight our boarder impacts as follows.
Broader Impacts Our observations and contributions may have following potential broader impacts:
• Crowd-sourcing of data annotations in computer vision: CIFAR-N may be further used for studying/proposing simulations of human annotations in crowd-sourcing, where the expenses of obtaining human annotations are often tremendous.
• A template for hypothesizing label noise patterns: Our hypothesis testing method of instancedependent label noise may provide a quantitative tool for testing the simulated human labels.
• Benchmarking effort is important: Although learning from noisy labels has witnessed thriving developments, we often observed conflicting comparisons due to the randomness in the synthetic noisy labels. While there exist several datasets with real human noise, we view our contribution as complementary to existing ones, due to the elaborated reasons above. We believe benchmarking existing and population solutions is an important technical contribution to the community.
• Understanding real-world label noise: Our observations and the provided human-annotated labels help with understanding real-world label noise. Besides, our observations of the multi-label issues in CIFAR-100 impose a new label noise pattern that is largely neglected.
• Motivations for real-world label noise solutions: our observations, especially the memorizing effects of real-world label noise may provide the literature with motivations for addressing realworld label noise.

heading: A CIFAR-10N REAL-WORLD NOISY LABEL BENCHMARK
text: In this section, we introduce the preparation for data collection, collection procedure of CIFAR-10N, the workers’ behaviors, and more detailed statistics of the obtained labels.

heading: A.1 CASE STUDIES BEFORE THE FORMAL COLLECTION
text: To make the collection procedure reasonable and efficient, we firstly upload a few batches (500 images / batch) to test the behaviors of workers. Our observations show that the workers on image classification tasks may possibly incur following phenomenons:
• Bots: with the appearance of bots, the accepted HIT may result in low-quality or meaningless responses if the bot is able to pick answers and maliciously/randomly submit them. Otherwise, the bot accepts the HIT but could not submit annotations. The accepted HIT may have to be re-assigned to another work after this HIT becomes expired which results in inefficient data collection.
• A large variance of the workers’ contribution: empirical observations show that a large amount of workers contribute too few HITs, while some professional workers upload with much less time. Thus, there exists a large variance in the number of the worker’s submitted HITs and the noise label pattern is substantially controlled by only a few workers.
• Incomplete submission: when there are more than one image per HIT, a worker would possibly neglect to click the label of one or more images, for example, the worker may only choose easy tasks to annotate and skip tough ones. The incomplete submission complexes the reassignment of images with missing annotations, the processing of annotated results as well as the individual payment.

heading: A.2 DATASET COLLECTION
text: To alleviate the impacts of aforementioned phenomenons, we randomly split the training dataset of CIFAR-10 without replacement into ten batches. Each batch contains 500 HITs. To ensure most workers do not contribute too few to the annotation task, we include ten 32 × 32 images per HIT. Each HIT is then randomly assigned to three independent workers. The worker gains a base reward $0.03 after submitting all annotations of one HIT within 2 minutes. The worker can not submit the annotations unless all appeared images in the assigned HIT are labeled. The averaged number of submitted HITs per worker is 7. Workers with no less 7 submissions share $200 bonus rewards. Note that we constrain the time duration for each assignment and re-design the interface, bots are less likely to finish our task either on time or under the procedure. What is more, we didn’t make use of any ground-truth clean labels to approve or reject submissions. We only block and reject workers who submit answers with fixed/regular distribution patterns.

heading: A.3 THE WORKERS’ BEHAVIORS
text: There are 747 independent workers contribute to the construction of CIFAR-10N. As depicted in Figure 8a, most workers can submit the labels for ten images within one minute. Although we do observe that a small amount of assignments (≤ 0.2%) are finished within 10 seconds, which are likely to be low-quality responses. The work time in seconds have the mean 46.7, the standard deviation 21.2 and the interquartile (25th percentile — 75th percentile) range is [31, 58]. Among these 747 workers, most of them annotated more than 80 images. The number of annotated images per work has the mean 201, the standard deviation 329 and the interquartile range is [30, 220].

heading: A.4 MORE DETAILED DATASET STATISTICS
text: Table 4 includes the summarized statistics of CIFAR-10N for each batch. In Table 4, the statistics “Consensus" means the three labelers have a consensus on the label of the same image. We do observe that several batches tend out to be more challenging for human workers to annotate, i.e, the noise rate appeared in Batch3 is clearly higher than those of Batch4 and Batch5. The difference of noise rate is especially significant on the “Worst" label set. We conjecture that there might exist more human annotators who malicious submit low-quality annotations when working on Batch3.
Note that while the number of images are the same for each clean label, across all the five noisy label sets of CIFAR-10N, we observe that human annotators have different preferences for similar classes, i.e, they are more likely to annotate an image to be an automobile rather than the truck, to be the horse rather than the deer (see Figure 9). The aggregated labels appear more frequently in automobile and ship, and less frequently in deer and cat. This gap of frequency becomes more clear in the worst label set.

heading: B CIFAR-100N REAL-WORLD NOISY LABEL BENCHMARK
text: In this section, we introduce the preparation for data collection, collection procedure of CIFAR-100N, and more detailed statistics of the obtained labels.

heading: B.1 CASE STUDIES BEFORE THE FORMAL COLLECTION
text: To make the collection procedure reasonable and efficient, we firstly upload a few batches (500 images / batch) to test the behaviors of workers. Our observations show that the workers on CIFAR-100 classification tasks may not only have mentioned phenomenons in A, but have following issues as well:
• Hard and time consuming: directly let workers to find the best matched label for each image from 100 possible classes is time assuming. Our case study shows that finding the label per image takes averaged 5-6 minutes. The work load as well as the difficulty level stop many workers from contributing two or more submissions.
• Lack of background knowledge: distinguishing several classes require some preliminary knowledge in biology, for example, it is common for workers to select a wrong super-class, especially for animal related ones: “aquatic mammals" and “fish". And the differences among some fine labels are hard to recognize, i.e, “trees" (oak, palm, pine), “medium-sized mammals" (porcupine, possum, raccoon, skunk), etc.
• Label aggregation has few effects: our empirical observations show that the the decrease of noise rates from aggregated labels given by 3 independent workers is less significant than the results on CIFAR-10N.

heading: B.2 DATASET COLLECTION
text: To deal with above mentioned issues, we firstly split the training dataset of CIFAR-100 without replacement into ten batches. Each batch contains 1000 HITs. We include five 96 × 96 images per HIT which are reshaped from 32 × 32 ones in CIFAR-100 train images. Only one worker is assigned with each HIT. Instead of requesting workers to find the best matched label from 100 labels directly, we group the 100 classes into 20 super-classes which are slightly different from the 20 raw “coarse" categories given by Krizhevsky et al. (2009). The 20 newly defined super-classes are summarized in Table 5. And the new division in each super-class is stated in Table 6. In order to reduce the workload of workers, they are instructed to firstly select the super-class for each image.
They will then re-directed to the corresponding 4-6 fine labels. Note that some super-classes are hard to recognize without some prior knowledge in biology, we provide workers with easy access to re-select the super-class for the current image, and every fine label has an example image from Google images for references.. If the worker luckily finds the most suitable fine label from her point of view, we also supply the jumping button so that the worker efficiently goes to the next image or the final submission of this HIT. A worker gains base reward $0.07 after submitting the answers of each HIT within 6 minutes (averaged working time is less than 90 seconds). Similar to the setting in the collection of CIFAR-10N, huge bonus applied to workers who contribute more HITs than the averaged number of submissions. Workers who submit answers with fixed/regular distribution patterns will be blocked and rejected all submitted results.

heading: B.3 MORE DETAILED DATASET STATISTICS
text: For CIFAR-100N dataset, each image contains a coarse label and a fine label given by a human annotator. Most batches have approximately 40% noisy fine labels and 25 % noisy coarse labels. The overall noise level of coarse and fine labels are 25.60% and 40.20%, respectively. A detailed summary of noise level is available in Table 7 which covers the statistics of each batch. Human annotators annotate frequently on classes which are outside of the clean-coarse, i.e., 25% noisy labels fall outside of the super-class and 15% inside the super-class.

heading: Statistics Batch1 Batch2 Batch3 Batch4 Batch5 Batch6 Batch7 Batch8 batch9 Batch10 Overall
text: Imbalanced annotations. The phenomenon of imbalanced annotations also appears substantially as shown in Figure 10, which presents the distribution of noisy labels for each fine class. “Man” appears ≥ 750 times, while “Streetcar” only has ≈ 200 annotations.
Transition matrices of CIFAR-100N In the class-dependent label noise setting, suppose the label noise is conditional independent of the feature, the noise transition matrices of CIFAR-100N are best described by a mixture of symmetric and asymmetric T . For CIFAR-100N, we heatmap the coarse and fine noisy labels w.r.t. the clean labels. In Figure 12, the clean label flips into one or more similar classes more often. And the remaining classes follow the symmetric noise model with a low noise rate. Apparently, current synthetic class-dependent noisy settings are not as complex as the real-world human annotated label noise.

heading: C COMPARISONS OF LABEL COLLECTION PROCEDURE BETWEEN CIFAR AND
text: CIFAR-N
We summarize the label collection procedure of CIFAR dataset as below:
• Images sources: the images of CIFAR come from down-scaled images from 80 million color images obtained by various search engines. The corresponding searching terms are viewed as a noisy source label/class;
• Remove irrelevant images: the designer provided each student (labeller) with a class. Students were asked to verify all the images which were found with that class as the search term. And reject images that are outside of the assigned class.
• Payment: the payment for each student/labeller per hour is fixed. • Noise rate: with additional personal verification, the noise rate of each class is negligible, espe-
cially for CIFAR-10.
We highlight the differences of label collection procedure between CIFAR and CIFAR-N as below:
• Annotation method: CIFAR adopted search engines such that they obtained images by specifying the class name (conditioned on the label, from label to feature); while CIFAR-N makes use of CIFAR images and pays workers for annotating labels (from feature to label).
• Payment and incentives: CIFAR fixed the payment for student workers (self-collected), there is no incentive to rush. As for CIFAR-N, we did not have any constraints on the workers via Amazon Mechanical Turk, i.e., the salary, the education level, etc. Besides, we set time limits and bonuses for each annotation task, there are incentives to rush and meanwhile provide high-quality labels.
• Noise rate: for CIFAR, the noise rate can be viewed as 0. While for CIFAR-N, we provide several noisy label sets which are of various noise rates.

heading: D HYPOTHESIS TESTING OF CIFAR-100N
text: We further statistically test whether the human-noise is feature-dependent or not in CIFAR-100N. The null hypothesis H0 and the corresponding alternate hypothesis H1 are defined as:
H0 : Human annotated label noise in CIFAR-100N is feature-independent; H1 : Human annotated label noise in CIFAR-100N is feature-dependent.
Note that the synthetic label noise is supposed to be feature-independent, the above hypotheses are converted to:
H0 : Human annotated label noise in CIFAR-100N is the same as the corresponding synthetic one; H1 : Human annotated label noise in CIFAR-100N is different from the corresponding synthetic one.
As implemented for CIFAR-10N, we adopt the distance between transition vectors pi,ν across different noise as measure of the difference between human noise and synthetic noise, e.g., d(1)i,ν := ‖phumani,ν − p synthetic i,ν ‖22. As contrast, we need to compare d (1) i,ν with d (2) i,ν := ‖p synthetic′ i,ν − p synthetic i,ν ‖22, where psynthetic ′
i,ν denotes the transition vector from the same synthetic noise but different clustering
result (caused by random data augmentation). Intuitively, if d(1)i,ν is much greater than d (2) i,ν , we should accept H1. The above hypotheses are then equivalent to:
H0 : {d(1)i,ν }i∈[K],ν∈[5] come from the same distribution as {d (2) i,ν }i∈[K],ν∈[5];
H1 : {d(1)i,ν }i∈[K],ν∈[5] come from different distributions from {d (2) i,ν }i∈[K],ν∈[5].
We repeat the generation of di,ν for 10 times, where the images are modified with different data augmentations each time. We choose the significance level α = 0.05 and perform a two-sided t-test w.r.t {d(1)i,ν }i∈[K],ν∈[5] and {d (2) i,ν }i∈[K],ν∈[5]. As described in Figure 13, the p-value of approximately 50 classes is less than α = 0.05. Thus, the null hypothesis for the human noisy labels in CIFAR-100N is rejected with the significance value α. And we accept the hypothesis:
Human annotated label noise in CIFAR-100N is feature-dependent.
Beyond feature dependency As shown in Figure 13 (b), although the overall noisy fine labels in CIFAR-100N are feature dependent, the noise transition vectors of around 50 classes can indeed be viewed as class dependent by referring to their significance levels. Thus, we can conclude that real-world human annotated noisy labels may be feature independent (class dependent) for certain classes.

heading: E ADDITIONAL EXPERIMENT RESULTS

heading: E.1 PERFORMANCE COMPARISONS ON SYNTHETIC CIFAR DATASETS
text: Continuing the empirical observations in Section 5.1, in this section, we adopt ResNet-34 (He et al., 2016), the same training procedure and batch-size for all implemented methods. The synthetic CIFAR datasets generate synthetic noisy labels by using the same class-dependent noise transition matrices in CIFAR-10N and CIFAR-100N. The comparisons of test accuracies are summarized in Table 8. For most methods, class-dependent synthetic noise is much easier to learn on CIFAR-10, especially when the noise level is high. The performance difference between human noise and the synthetic noise is less obvious for CIFAR-100.

heading: E.2 EXPERIMENT DETAILS ON CIFAR-10N AND CIFAR-100N
text: The basic hyper-parameters settings for CIFAR-10N and CIFAR-100N are listed as follows: minibatch size (128), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (100) and learning rate decay (0.1 at 50 epochs). Standard data augmentation is applied to each dataset.
Special treatments In the reproduced experiments, we use the default setting for Cores* (Cheng et al., 2021), ELR+ (Liu et al., 2020) and DivideMix since either advanced data augmentation strategies or two deep neural networks are adopted. And we fix a same pre-trained model for methods that require a CE warm-up or a pre-trained model.

heading: E.3 COMPUTING INFRASTRUCTURE
text: All our experiments run on a GPU cluster (500 GPUs of all kinds, mainly use 2080 Ti) for training and evaluation.

Review summaries:
This paper proposes two smaller-sized datasets with real-world label noise. The datasets contain CIFAR-10 and CIFAR-100 images with human-annotated labels. The paper analysis the noise pattern, and benchmarks existing methods on these datasets.

This paper focuses on learning from noisy labels, which is a very important research topic in the ML community. One of the main challenges in this area has been the absence of controllable real-world noisy data. The authors present the variation of CIFARs data with human annotations via MTurk.  By comparing the results on the original CIFARs and the newly proposed CIFAR-N, four observations are detailed quantitatively or qualitatively. In addition, they provide the performance comparison of more than 10 approaches on CIFAR-10 and -100N. The analysis of the memorization effect in human-annotated noisy data is also discussed.

The paper provides an extensive study on 1) the difference between human annotations and synthetic ones and 2) the impact on the classifier learning from human annotations and synthetic ones. To this end, the paper crowdsources the human annotations for CIFAR10 and CIFAR100 from Amazon Mechanical Turk. The observations show that human annotations are more challenging to learn, and human annotations result in a different memorization scheme for neural networks.