Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

1 INTRODUCTION . Image classification task in deep learning requires assigning labels to specific images . Annotating labels for training use often requires tremendous expenses on the payment for hiring human annotators . The pervasive noisy labels from data annotation present significant challenges to training a quality machine learning model . The problem of dealing with label noise has been receiving increasing attentions . Typical approaches include unbiased estimators and weighted loss functions ( Natarajan et al. , 2013 ; Liu & Tao , 2015 ) , loss correction ( Patrini et al. , 2017 ; Liu & Guo , 2020 ) , sample selection aided ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ) , etc . The majority of existing solutions are often developed under stylish synthetic noise model , where the noise rates are either class-dependent or homogeneous across data instances . Recently , there has been efforts for establishing results for the more realistic instance-dependent setting , i.e , the noise transition differs in different parts of features ( Xia et al. , 2020b ) or sub-populations ( Wang et al. , 2021 ; Zhu et al. , 2021a ) , etc . All of these noise models are proposed with some statistical assumptions , which facilitate the derivation of theoretical solutions . Yet , it is unclear how the existing models captured the real-world scenario . To empirically validate the robustness of proposed methods , synthetic noisy labels on CIFAR-10 and CIFAR-100 ( Krizhevsky et al. , 2009 ) are the most widely accepted benchmarks . The literature has also observed approaches to the simulation of human annotators in data labeling ( Hua et al. , 2013 ; Long & Hua , 2015 ; Liao et al. , 2021 ) , and real-world label noise benchmarks , including Food-101 ( Bossard et al. , 2014 ) , Clothing-1M ( Xiao et al. , 2015 ) , WebVision ( Li et al. , 2017 ) , etc . We summarize the above real-world noisy label datasets in Table 1 . While a more detailed description and discussion of the existing datasets can be found in the related works , we want to highlight several outstanding issues in existing benchmarks and evaluations . As noted in Table 1 , except for CIFAR related noisy label datasets , all other datasets suffer from at least one of the three caveats : • Complex task ( High-resolution ) : when learning with large-scale and relative high-resolution data , the complex data pattern , various augmentation strategies , the use of extra train or clean data , different computation power ( for hyper-parameter tuning such as batch-size , learning rate , etc ) jointly contribute to the model performance and then result in unfair comparison . • Missing clean labels : the lack of clean labels for verification in most existed noisy-label datasets makes the evaluation of robust methods intractable . • Interventions : human interventions in data generation ( Jiang et al. , 2020 ) and non-representative data collection process ( Song et al. , 2019 ) might disturb the original noisy-label pattern . In addition , despite synthetically labeled CIFAR datasets are popular and highly used benchmarks for evaluating the robustness of proposed methods , there exists no publicly available human annotated labels for CIFAR training datasets to perform either validation of existing methods or verification of popular noise models 1 . A human-annotated version of CIFAR datasets would greatly facilitate the evaluations of existing and future solutions , due to the already standardized procedures for experimenting with CIFAR datasets . All above issues motivate us to revisit the problem of learning with noisy labels and establish accessible and easy-to-use , verifiable datasets that would be broadly usable to the research community . Our contributions can be summarized as follows : • We present two new benchmarks CIFAR-10N , CIFAR-100N which provide CIFAR-10 and CIFAR100 with human annotated noisy labels . Our efforts built upon the CIFAR datasets and provide easily usable benchmark data for the weakly supervised learning community ( Section 3 ) . We expect to continue to maintain the datasets to facilitate future development of results . • We introduce new observations for the distribution of human annotated noisy labels on tiny images , i.e. , imbalanced annotations , the flipping of noisy labels among similar features , co-existence of multiple clean labels for CIFAR-100 train images ( which leads to a new pattern of label noise ) , etc . We further distinguish noisy labels in CIFAR-10N and CIFAR-100N with synthetic classdependent label noise , from the aspect of noise transitions for different features qualitatively and quantitatively ( via hypothesis testing ) ( Section 4 ) . • We empirically compare the robustness of a comprehensive list of popular methods when learning with CIFAR-10N , CIFAR-100N . We observe consistent performance gaps between human noise and synthetic noise . The different memorization behavior further distinguishes the human noise and synthetic noise ( Section 5 ) . 1.1 RELATED WORKS . Learning from noisy labels Earlier approaches for learning from noisy labels mainly focus on loss adjustment techniques . To mitigate the impact of label noise , a line of approaches modify the loss of image samples by multiplying an estimated noise transition matrix ( Patrini et al. , 2017 ; Hendrycks et al. , 2018 ; Xia et al. , 2019 ; Yao et al. , 2020 ) , re-weight the loss to encourage deep neural nets to 1CIFAR10H ( Peterson et al. , 2019 ) only provides test images with noisy human annotations . fit on correct labels ( Liu & Tao , 2015 ) , propose robust loss functions ( Natarajan et al. , 2013 ; Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Amid et al. , 2019 ; Wang et al. , 2019 ; Liu & Guo , 2020 ) , or introduce a robust regularizer ( Liu et al. , 2020 ; Xia et al. , 2020a ; Cheng et al. , 2021 ; Wei et al. , 2021 ) . Another line of popular approaches behaves like a semi-supervised manner which begins with a clean sample selection procedure , then makes use of the wrongly-labeled samples . For example , several methods ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ) adopt a mentor/peer network to select small-loss samples as “ clean ” ones for the student/peer network . To further explore the benefits of wrongly-labeled samples and improve the model performance , Li et al . ( 2020 ) chose the Mix-match ( Berthelot et al. , 2019 ) technique which has shown success in semi-supervised learning . Benchmarks noisy labels datasets Food-101 ( Bossard et al. , 2014 ) , Clothing-1M ( Xiao et al. , 2015 ) , WebVision ( Li et al. , 2017 ) are three large-scale noisy labeled web-image databases which consist of food images , clothes images or other web images , respectively . However , the majority of images in these three datasets do not have a corresponding clean label to perform controlled verification ( e.g. , verifying the noise levels ) . Later , a much larger-scale food dataset is collected by Lee et al . ( 2018 ) , which contains exactly the same classes as Food-101 ( Bossard et al. , 2014 ) . More recently , Peterson et al . ( 2019 ) present a noisily labeled benchmarks on CIFAR-10 test dataset where each test image has 51 human annotated labels in average . Jiang et al . ( 2020 ) construct noisily labeled Mini-ImageNet ( Vinyals et al. , 2016 ) and Stanford Cars datasets ( Krause et al. , 2013 ) with controlled noise levels by substituting human annotated incorrect labels for synthetic wrong labels . 2 SYNTHETIC LABEL NOISE . In this section , we discuss a few popular synthetic models for generating noisy labels . We focus on a K-class classification task . Denote by D : = { ( xn , yn ) } n∈ [ N ] the training samples where [ N ] : = { 1 , 2 , ... , N } . ( xn , yn ) s are given by random variables ( X , Y ) ∈ X ×Y drawn from the joint distributionD , where X , Y can be viewed as the space of feature and label , respectively . In real-world scenarios , a classifier f only has access to noisily labeled training sets D̃ : = { ( xn , ỹn ) } n∈ [ N ] . We assume the noisy samples ( xn , ỹn ) s are given by random variables ( X , Ỹ ) ∈ X ×Ỹ which are drawn from the joint distribution D̃ . Clearly , there may exist n ∈ [ N ] such that yn 6= ỹn . The flipping from clean to noisy label is usually formulated by a noise transition matrix T ( X ) , with elements : Ti , j ( X ) : = P ( Ỹ = j|Y = i , X ) . We shall specify different modeling choices of T ( X ) below . 2.1 CLASS-DEPENDENT LABEL NOISE . The first family of noise transition matrix is the class-dependent noise where the label noise is assumed to be conditionally independent of the feature X . Mathematically , T ( X ) ≡ T and Ti , j ( X ) = P ( Ỹ = j|Y = i ) , ∀i , j ∈ [ K ] . Symmetric T The symmetric noise transition matrix ( Natarajan et al. , 2013 ) describes the scenario where an amount of human labelers maliciously assign a random label for the given task . It assumes that the probability of randomly flipping the clean class to the other possible class with probability . Assume the noise level is , the diagonal entry of the symmetric T is denoted as Ti , i = 1− . For any other off-diagonal entry Ti , j where i 6= j , the corresponding element is Ti , j = / ( K − 1 ) . Asymmetric T The asymmetric noise transition matrix ( Patrini et al. , 2017 ) simulates the case where there exists ambiguity classes , i.e , human labelers may wrongly annotate the truck as automobile due to the low-resolution images . There are two types of widely adopted asymmetric T . The assymetric-next T assumes that the clean label flips to the next class with probability , i.e , i→ ( i+ 1 ) mod K for i ∈ [ K ] . The assymetric-pair T considers [ K/2 ] disjoint class pairs ( ic , jc ) where ic < jc . For c ∈ [ K/2 ] , Tic , jc = Tjc , ic = , and the diagonal entries are 1− . 2.2 INSTANCE-DEPENDENT LABEL NOISE . Beyond the feature-independent assumption , recent works pay more attention to a challenging case where the label noise is jointly determined by featureX and clean label Y . There are some techniques for synthesizing the instance-dependent label noise , such as the polynomial margin diminishing label noise ( Zhang et al. , 2021b ) where instances near decision boundary are easier to be mislabeled , the part-dependent label noise ( Xia et al. , 2020b ) where different parts of feature may contribute different noise transition matrices , and the group-dependent label noise ( Wang et al. , 2021 ; Zhu et al. , 2021a ) where different sub-populations may have different noise rates . All of these noise models are proposed with some statistical assumptions , which facilitate the derivation of theoretical solutions .

Summary: This paper focuses on learning from noisy labels, which is a very important research topic in the ML community. One of the main challenges in this area has been the absence of controllable real-world noisy data. The authors present the variation of CIFARs data with human annotations via MTurk.  By comparing the results on the original CIFARs and the newly proposed CIFAR-N, four observations are detailed quantitatively or qualitatively. In addition, they provide the performance comparison of more than 10 approaches on CIFAR-10 and -100N. The analysis of the memorization effect in human-annotated noisy data is also discussed.