S3: Supervised Self-supervised Learning under Label Noise

1 INTRODUCTION . It is now commonly accepted that supervised learning with deep neural networks can provide excellent solutions for a wide range of problems , so long as there is sufficient availability of labeled training data and computational resources . However , these results have been mostly obtained using well-curated datasets in which the classes are balanced and the labels are of high quality . In the realworld , it is often costly to obtain high quality labels especially for large-scale datasets . A common approach is to use semi-automatic methods to obtain the labels ( e.g . “ webly-labeled ” images where the images and labels are obtained by web-crawling ) . While such methods can greatly reduce the time and cost of manual labeling , they also lead to low quality noisy labels . To deal with noisy labels , earlier approaches tried to improve the robustness of the model using robust loss functions ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Wang et al. , 2019 ) or robust regularizations ( Srivastava et al. , 2014 ; Zhang et al. , 2017 ; Pereyra et al. , 2017 ) . Goldberger & BenReuven ( 2016 ) tried to model the noise transition matrix between classes while Han et al . ( 2019 ) ; Patrini et al . ( 2017 ) ; Hendrycks et al . ( 2018 ) proposed to correct the losses of noisy samples . More recently , sample selection methods became perhaps the dominant paradigm for learning with noisy labels . Most of the recent sample selection methods do so , by relying on the predictions of the model classifier , for example on the per-sample loss ( Arazo et al. , 2019 ; Li et al. , 2020a ) or model prediction ( Song et al. , 2019 ; Malach & Shalev-Shwartz , 2017 ) . By separating clean samples and noisy samples and subsequently performing supervised training on the clean set , or semi-supervised training on both , sample selection methods achieved the state-of-the-art results in synthetic and realworld noisy datasets . However , there are three main issues with current sample selection methods . Firstly , the sample selection will be inevitably biased if the models ( classifier and feature extractor ) are trained with noisy labels – this is immediately apparent in the case that the sample selection is based on the loss of the classifier itself ( Arazo et al. , 2019 ; Li et al. , 2020a ; Yu et al. , 2019 ; Han et al. , 2018 ) . Second , in supervised classification problems , noisy samples usually come from two main categories : closedset noise where the true labels belong to one of the given classes ( Set B in Fig . 1 ) and open-set noise where the true labels do not belong to the set of labels of the classification problem ( Set C in Fig . 1 ) . Most of the works in the literature , including works that estimate the probabilities of label-exchange between pairs of classes ( Goldberger & Ben-Reuven , 2016 ; Patrini et al. , 2017 ) , that do relabeling based on the model ’ s predictions ( Song et al. , 2019 ; Han et al. , 2019 ) or works that adopt semisupervised approaches ( Li et al. , 2020a ; Ortego et al. , 2021 ) deal with the former and not directly address the latter . However , this is a considerable source of noise in real world scenarios , e.g. , when training from web-crawled data , where there is less control over the collection of the dataset . Crucially , those works ( implicitly or explicitly ) relabel all samples and do training based on the new labels of all samples . Those are bound to be wrong for all samples in set C , but also are bound to be wrong for several samples of A and B that are ( implicitly or explicitly ) relabeled in the early stages of training . For the latter reason , those works do not work well even under heavy close-set noise . Finally , current approaches usually require extensive hyperparameters tuning , often even on a perdataset basis – this is unrealistic in scenarios where there is little knowledge about the types of noise . This is partly due to the complexity of the applied semi-supervised learning method , and partly because of the complicated methods that are employed , such as model pretraining ( Zheltonozhskii et al. , 2021 ) and model cotraining ( Han et al. , 2018 ; Yu et al. , 2019 ; Li et al. , 2020a ) , so as to deal with self-confirmation bias . In this paper , we address the problem of training under different types of noise with a simple method–namely Supervised , Self-Supervised learning ( S3 ) , with two major components that are clearly separated : a selection/relabelling mechanism that selects/relabels samples so as to construct a clean and a noisy set ( Section 3.3 ) , and a training framework that aims at learning a strong feature extractor f and a classification head g [ Fig . 2 ] from both noisy and clean samples ( Section 3.4 ) . In the training stage we use off-the-shelf learners and a ) train the feature extractor and the classification head using a classical Supervised cross-entropy loss applied only on the clean samples – this avoids treating samples identi- fied as noisy as if they belonged to one of the given classes as most methods ( Arazo et al. , 2019 ; Li et al. , 2020a ; Ortego et al. , 2021 ; Wu et al. , 2021 ) implictly or explicitly do ; and b ) train the feature extractor using a Self-Supervised loss , namely the consistency loss between the representations of augmented version of the sample ( as in ( Chen & He , 2021 ) ) applied on all samples – this avoids false-negatives that are inherent in contrastive-learning with instance discrimination and different to works that apply the consistency on the label predictions that are unreliable at the first iterations or in the presence of open-set noise . The noisy sample selection mechanism relies on a measure of confidence what we define using the ground truth label of the sample in question and an estimate of distribution of the labels of its neighbours – in order to deal with noisy samples , we adopt a scheme in which the distribution is calculated based both on the ground truth labels and on consistently ( over subsequent iterations ) confident estimates of the labels . Our method is embedded into a standard MixUp and data augmentation framework , and without bells and whistles , such as co-training of multiple models , it achieves state-of-the-art results in both synthetic and realistic noise patterns in CIFAR10 , CIFAR100 , ANIMAL-10N , Clothing1M and WebVision datasets . 2 RELATED WORK . Leaning with noisy data by sample selection Some works focused on sample selection to filter out noisy samples . Jiang et al . ( 2018 ) introduced a pretrained mentor network to guide the selection of a student network . Song et al . ( 2019 ) evaluates the per-sample losses and identify as clean the top r % of the samples – the precise ratio r % depends is either predefined , or is an estimate of the noise level in the specific dataset . Arazo et al . ( 2019 ) proposed to model per-sample losses with a Beta Mixture Model ( BMM ) and split the dataset according to which of the components of the mixture each sample belongs . In a very similar approach , Li et al . ( 2020a ) extended upon Arazo et al . ( 2019 ) by introducing semi-supervised learning to fully utilize the dataset . Related to our work , Bahri et al . ( 2020 ) ; Ortego et al . ( 2021 ) ; Wang et al . ( 2018 ) also utilized the feature space for sample selection . Bahri et al . ( 2020 ) applied KNN for sample selection for closed-set noisy dataset while Ortego et al . ( 2021 ) further proposed to relabel samples based on the KNN voting . Wang et al . ( 2018 ) proposed to reweight samples based on its probability of being outliers in open-set noisy dataset . Self-supervised learning Self-supervised methods attempt to learn good representation without human annotations . In the recent years , the dominant method is contrastive learning with instance discrimination task . MoCo ( He et al. , 2020 ) is an important baseline for current contrastive learning methods , which reuses the memory bank since samples in a single mini-batch may lead to insufficient negative pairs , and proposes a momentum encoder to update the memory bank in real-time to avoid outdated data representation . SimCLR ( Chen et al. , 2020 ) is another important baseline which found that setting mini-batch size to be large enough can eliminate the need for memory bank . More recently , SimSiam ( Chen & He , 2021 ) and BYOL ( Grill et al. , 2020 ) proposed a non-contrastive learning framework which enforce the perturbation consitency between different views , and avoid mode collapse by applying stop-gradient and an extra predictor between two representation vectors . 3 METHOD . 3.1 PROBLEM FORMULATION . Let us denote with X = { xi } Ni=1 , xi ∈ Rd , a training set with the corresponding one-hot vector labels Y = { yi } Ni=1 , yi ∈ { 0 , 1 } K , where K is the number of classes and N is the number of samples . For convenience , let us also denote the index where the one-hot vector yi is one as the label li ∈ { 0 , ... , K } . Finally , let us denote the true labels with Y ′ = { y′i } Ni=1 . Clearly , for an open-set noisy label it is the case that y′i 6= yi , y′i /∈ { 0 , 1 } K , while for closed-set noisy samples y′i 6= yi , y′i ∈ { 0 , 1 } K . 3.2 OVERVIEW OF PROPOSED METHOD . Aiming to deal with potential concurrence of both open-set and close-set noise , we view the classification network as an encoder f that extracts a feature representation and a classification head g that deals with the classification problem in question . The proposed method , named Supervised Self-Supervised ( S3 ) learning , attempts to decouple their training so as to deal with possible noise in the labels , by adopting a two stage , iterative scheme , as outlined in Fig . 2 . In the first stage , we utilize a novel sample selection and a novel relabeling mechanism ( top block in Fig . 2 ) that prepares the set based on which the classifier g should be trained in Stage 2 . The selection mechanism is based on the assumption of smoothness of labels in the feature space , and more specifically , on a consistency measure that we defined based on the annotated label of the sample in question and the distribution of the labels in its neighborhood in the feature space . Relabeling is performed on samples for which the classifier gives confident predictions consistently across subsequent iterations . Clearly , the mechanism relies on the quality of the features extracted by the encoder f and should reject samples whose true labels are not in the class set ( open-set noise ) . This stage is explained in Section 3.3 . In the second stage ( bottom block in Fig . 2 ) , training is performed with two objectives/losses . First , a cross entropy loss on the output of the classifier g , ( i.e. , on g ( f ( . ) ) ) on the samples selected in Stage 1 , that updates both the encoder f and the classifier head g. Second , a self-supervision loss that enforces consistency between the representations of different augmentations of the same sample , and which utilizes all samples , that is both noisy and clean – this updates the encoder f and helps learning a strong feature space on which the selection mechanism of Stage 1 can rely . By contrast to other methods that , either by using a noise transition matrix or in their semisupervised scheme , implicitly relabel all samples ( e.g . DivideMix ( Li et al. , 2020a ) , MOIT ( Ortego et al. , 2021 ) ) and use the new labels to learn , in our method the labels in the noisy set are not used at all . This stage is explained in Section 3.4 .

Summary: This paper proposes "S3" framework for learning with noisy labels. Specifically, S3 consists of two stages. In the first stage, a relabelling approach and normalized neighboring voting are utilized to guide efficient sample selection; in the second stage, supervised loss (Mixup) and self-consistency loss are used to train networks on selected samples. S3 can be applied to both close-set label noise and open-set label noise and exhibit good performance on several benchmark datasets.