Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

1 INTRODUCTION . Image classification task in deep learning requires assigning labels to specific images . Annotating labels for training use often requires tremendous expenses on the payment for hiring human annotators . The pervasive noisy labels from data annotation present significant challenges to training a quality machine learning model . The problem of dealing with label noise has been receiving increasing attentions . Typical approaches include unbiased estimators and weighted loss functions ( Natarajan et al. , 2013 ; Liu & Tao , 2015 ) , loss correction ( Patrini et al. , 2017 ; Liu & Guo , 2020 ) , sample selection aided ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ) , etc . The majority of existing solutions are often developed under stylish synthetic noise model , where the noise rates are either class-dependent or homogeneous across data instances . Recently , there has been efforts for establishing results for the more realistic instance-dependent setting , i.e , the noise transition differs in different parts of features ( Xia et al. , 2020b ) or sub-populations ( Wang et al. , 2021 ; Zhu et al. , 2021a ) , etc . All of these noise models are proposed with some statistical assumptions , which facilitate the derivation of theoretical solutions . Yet , it is unclear how the existing models captured the real-world scenario . To empirically validate the robustness of proposed methods , synthetic noisy labels on CIFAR-10 and CIFAR-100 ( Krizhevsky et al. , 2009 ) are the most widely accepted benchmarks . The literature has also observed approaches to the simulation of human annotators in data labeling ( Hua et al. , 2013 ; Long & Hua , 2015 ; Liao et al. , 2021 ) , and real-world label noise benchmarks , including Food-101 ( Bossard et al. , 2014 ) , Clothing-1M ( Xiao et al. , 2015 ) , WebVision ( Li et al. , 2017 ) , etc . We summarize the above real-world noisy label datasets in Table 1 . While a more detailed description and discussion of the existing datasets can be found in the related works , we want to highlight several outstanding issues in existing benchmarks and evaluations . As noted in Table 1 , except for CIFAR related noisy label datasets , all other datasets suffer from at least one of the three caveats : • Complex task ( High-resolution ) : when learning with large-scale and relative high-resolution data , the complex data pattern , various augmentation strategies , the use of extra train or clean data , different computation power ( for hyper-parameter tuning such as batch-size , learning rate , etc ) jointly contribute to the model performance and then result in unfair comparison . • Missing clean labels : the lack of clean labels for verification in most existed noisy-label datasets makes the evaluation of robust methods intractable . • Interventions : human interventions in data generation ( Jiang et al. , 2020 ) and non-representative data collection process ( Song et al. , 2019 ) might disturb the original noisy-label pattern . In addition , despite synthetically labeled CIFAR datasets are popular and highly used benchmarks for evaluating the robustness of proposed methods , there exists no publicly available human annotated labels for CIFAR training datasets to perform either validation of existing methods or verification of popular noise models 1 . A human-annotated version of CIFAR datasets would greatly facilitate the evaluations of existing and future solutions , due to the already standardized procedures for experimenting with CIFAR datasets . All above issues motivate us to revisit the problem of learning with noisy labels and establish accessible and easy-to-use , verifiable datasets that would be broadly usable to the research community . Our contributions can be summarized as follows : • We present two new benchmarks CIFAR-10N , CIFAR-100N which provide CIFAR-10 and CIFAR100 with human annotated noisy labels . Our efforts built upon the CIFAR datasets and provide easily usable benchmark data for the weakly supervised learning community ( Section 3 ) . We expect to continue to maintain the datasets to facilitate future development of results . • We introduce new observations for the distribution of human annotated noisy labels on tiny images , i.e. , imbalanced annotations , the flipping of noisy labels among similar features , co-existence of multiple clean labels for CIFAR-100 train images ( which leads to a new pattern of label noise ) , etc . We further distinguish noisy labels in CIFAR-10N and CIFAR-100N with synthetic classdependent label noise , from the aspect of noise transitions for different features qualitatively and quantitatively ( via hypothesis testing ) ( Section 4 ) . • We empirically compare the robustness of a comprehensive list of popular methods when learning with CIFAR-10N , CIFAR-100N . We observe consistent performance gaps between human noise and synthetic noise . The different memorization behavior further distinguishes the human noise and synthetic noise ( Section 5 ) . 1.1 RELATED WORKS . Learning from noisy labels Earlier approaches for learning from noisy labels mainly focus on loss adjustment techniques . To mitigate the impact of label noise , a line of approaches modify the loss of image samples by multiplying an estimated noise transition matrix ( Patrini et al. , 2017 ; Hendrycks et al. , 2018 ; Xia et al. , 2019 ; Yao et al. , 2020 ) , re-weight the loss to encourage deep neural nets to 1CIFAR10H ( Peterson et al. , 2019 ) only provides test images with noisy human annotations . fit on correct labels ( Liu & Tao , 2015 ) , propose robust loss functions ( Natarajan et al. , 2013 ; Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Amid et al. , 2019 ; Wang et al. , 2019 ; Liu & Guo , 2020 ) , or introduce a robust regularizer ( Liu et al. , 2020 ; Xia et al. , 2020a ; Cheng et al. , 2021 ; Wei et al. , 2021 ) . Another line of popular approaches behaves like a semi-supervised manner which begins with a clean sample selection procedure , then makes use of the wrongly-labeled samples . For example , several methods ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ) adopt a mentor/peer network to select small-loss samples as “ clean ” ones for the student/peer network . To further explore the benefits of wrongly-labeled samples and improve the model performance , Li et al . ( 2020 ) chose the Mix-match ( Berthelot et al. , 2019 ) technique which has shown success in semi-supervised learning . Benchmarks noisy labels datasets Food-101 ( Bossard et al. , 2014 ) , Clothing-1M ( Xiao et al. , 2015 ) , WebVision ( Li et al. , 2017 ) are three large-scale noisy labeled web-image databases which consist of food images , clothes images or other web images , respectively . However , the majority of images in these three datasets do not have a corresponding clean label to perform controlled verification ( e.g. , verifying the noise levels ) . Later , a much larger-scale food dataset is collected by Lee et al . ( 2018 ) , which contains exactly the same classes as Food-101 ( Bossard et al. , 2014 ) . More recently , Peterson et al . ( 2019 ) present a noisily labeled benchmarks on CIFAR-10 test dataset where each test image has 51 human annotated labels in average . Jiang et al . ( 2020 ) construct noisily labeled Mini-ImageNet ( Vinyals et al. , 2016 ) and Stanford Cars datasets ( Krause et al. , 2013 ) with controlled noise levels by substituting human annotated incorrect labels for synthetic wrong labels . 2 SYNTHETIC LABEL NOISE . In this section , we discuss a few popular synthetic models for generating noisy labels . We focus on a K-class classification task . Denote by D : = { ( xn , yn ) } n∈ [ N ] the training samples where [ N ] : = { 1 , 2 , ... , N } . ( xn , yn ) s are given by random variables ( X , Y ) ∈ X ×Y drawn from the joint distributionD , where X , Y can be viewed as the space of feature and label , respectively . In real-world scenarios , a classifier f only has access to noisily labeled training sets D̃ : = { ( xn , ỹn ) } n∈ [ N ] . We assume the noisy samples ( xn , ỹn ) s are given by random variables ( X , Ỹ ) ∈ X ×Ỹ which are drawn from the joint distribution D̃ . Clearly , there may exist n ∈ [ N ] such that yn 6= ỹn . The flipping from clean to noisy label is usually formulated by a noise transition matrix T ( X ) , with elements : Ti , j ( X ) : = P ( Ỹ = j|Y = i , X ) . We shall specify different modeling choices of T ( X ) below . 2.1 CLASS-DEPENDENT LABEL NOISE . The first family of noise transition matrix is the class-dependent noise where the label noise is assumed to be conditionally independent of the feature X . Mathematically , T ( X ) ≡ T and Ti , j ( X ) = P ( Ỹ = j|Y = i ) , ∀i , j ∈ [ K ] . Symmetric T The symmetric noise transition matrix ( Natarajan et al. , 2013 ) describes the scenario where an amount of human labelers maliciously assign a random label for the given task . It assumes that the probability of randomly flipping the clean class to the other possible class with probability . Assume the noise level is , the diagonal entry of the symmetric T is denoted as Ti , i = 1− . For any other off-diagonal entry Ti , j where i 6= j , the corresponding element is Ti , j = / ( K − 1 ) . Asymmetric T The asymmetric noise transition matrix ( Patrini et al. , 2017 ) simulates the case where there exists ambiguity classes , i.e , human labelers may wrongly annotate the truck as automobile due to the low-resolution images . There are two types of widely adopted asymmetric T . The assymetric-next T assumes that the clean label flips to the next class with probability , i.e , i→ ( i+ 1 ) mod K for i ∈ [ K ] . The assymetric-pair T considers [ K/2 ] disjoint class pairs ( ic , jc ) where ic < jc . For c ∈ [ K/2 ] , Tic , jc = Tjc , ic = , and the diagonal entries are 1− . 2.2 INSTANCE-DEPENDENT LABEL NOISE . Beyond the feature-independent assumption , recent works pay more attention to a challenging case where the label noise is jointly determined by featureX and clean label Y . There are some techniques for synthesizing the instance-dependent label noise , such as the polynomial margin diminishing label noise ( Zhang et al. , 2021b ) where instances near decision boundary are easier to be mislabeled , the part-dependent label noise ( Xia et al. , 2020b ) where different parts of feature may contribute different noise transition matrices , and the group-dependent label noise ( Wang et al. , 2021 ; Zhu et al. , 2021a ) where different sub-populations may have different noise rates . All of these noise models are proposed with some statistical assumptions , which facilitate the derivation of theoretical solutions .

Summary: The paper provides an extensive study on 1) the difference between human annotations and synthetic ones and 2) the impact on the classifier learning from human annotations and synthetic ones. To this end, the paper crowdsources the human annotations for CIFAR10 and CIFAR100 from Amazon Mechanical Turk. The observations show that human annotations are more challenging to learn, and human annotations result in a different memorization scheme for neural networks.