S3: Supervised Self-supervised Learning under Label Noise

1 INTRODUCTION . It is now commonly accepted that supervised learning with deep neural networks can provide excellent solutions for a wide range of problems , so long as there is sufficient availability of labeled training data and computational resources . However , these results have been mostly obtained using well-curated datasets in which the classes are balanced and the labels are of high quality . In the realworld , it is often costly to obtain high quality labels especially for large-scale datasets . A common approach is to use semi-automatic methods to obtain the labels ( e.g . “ webly-labeled ” images where the images and labels are obtained by web-crawling ) . While such methods can greatly reduce the time and cost of manual labeling , they also lead to low quality noisy labels . To deal with noisy labels , earlier approaches tried to improve the robustness of the model using robust loss functions ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Wang et al. , 2019 ) or robust regularizations ( Srivastava et al. , 2014 ; Zhang et al. , 2017 ; Pereyra et al. , 2017 ) . Goldberger & BenReuven ( 2016 ) tried to model the noise transition matrix between classes while Han et al . ( 2019 ) ; Patrini et al . ( 2017 ) ; Hendrycks et al . ( 2018 ) proposed to correct the losses of noisy samples . More recently , sample selection methods became perhaps the dominant paradigm for learning with noisy labels . Most of the recent sample selection methods do so , by relying on the predictions of the model classifier , for example on the per-sample loss ( Arazo et al. , 2019 ; Li et al. , 2020a ) or model prediction ( Song et al. , 2019 ; Malach & Shalev-Shwartz , 2017 ) . By separating clean samples and noisy samples and subsequently performing supervised training on the clean set , or semi-supervised training on both , sample selection methods achieved the state-of-the-art results in synthetic and realworld noisy datasets . However , there are three main issues with current sample selection methods . Firstly , the sample selection will be inevitably biased if the models ( classifier and feature extractor ) are trained with noisy labels – this is immediately apparent in the case that the sample selection is based on the loss of the classifier itself ( Arazo et al. , 2019 ; Li et al. , 2020a ; Yu et al. , 2019 ; Han et al. , 2018 ) . Second , in supervised classification problems , noisy samples usually come from two main categories : closedset noise where the true labels belong to one of the given classes ( Set B in Fig . 1 ) and open-set noise where the true labels do not belong to the set of labels of the classification problem ( Set C in Fig . 1 ) . Most of the works in the literature , including works that estimate the probabilities of label-exchange between pairs of classes ( Goldberger & Ben-Reuven , 2016 ; Patrini et al. , 2017 ) , that do relabeling based on the model ’ s predictions ( Song et al. , 2019 ; Han et al. , 2019 ) or works that adopt semisupervised approaches ( Li et al. , 2020a ; Ortego et al. , 2021 ) deal with the former and not directly address the latter . However , this is a considerable source of noise in real world scenarios , e.g. , when training from web-crawled data , where there is less control over the collection of the dataset . Crucially , those works ( implicitly or explicitly ) relabel all samples and do training based on the new labels of all samples . Those are bound to be wrong for all samples in set C , but also are bound to be wrong for several samples of A and B that are ( implicitly or explicitly ) relabeled in the early stages of training . For the latter reason , those works do not work well even under heavy close-set noise . Finally , current approaches usually require extensive hyperparameters tuning , often even on a perdataset basis – this is unrealistic in scenarios where there is little knowledge about the types of noise . This is partly due to the complexity of the applied semi-supervised learning method , and partly because of the complicated methods that are employed , such as model pretraining ( Zheltonozhskii et al. , 2021 ) and model cotraining ( Han et al. , 2018 ; Yu et al. , 2019 ; Li et al. , 2020a ) , so as to deal with self-confirmation bias . In this paper , we address the problem of training under different types of noise with a simple method–namely Supervised , Self-Supervised learning ( S3 ) , with two major components that are clearly separated : a selection/relabelling mechanism that selects/relabels samples so as to construct a clean and a noisy set ( Section 3.3 ) , and a training framework that aims at learning a strong feature extractor f and a classification head g [ Fig . 2 ] from both noisy and clean samples ( Section 3.4 ) . In the training stage we use off-the-shelf learners and a ) train the feature extractor and the classification head using a classical Supervised cross-entropy loss applied only on the clean samples – this avoids treating samples identi- fied as noisy as if they belonged to one of the given classes as most methods ( Arazo et al. , 2019 ; Li et al. , 2020a ; Ortego et al. , 2021 ; Wu et al. , 2021 ) implictly or explicitly do ; and b ) train the feature extractor using a Self-Supervised loss , namely the consistency loss between the representations of augmented version of the sample ( as in ( Chen & He , 2021 ) ) applied on all samples – this avoids false-negatives that are inherent in contrastive-learning with instance discrimination and different to works that apply the consistency on the label predictions that are unreliable at the first iterations or in the presence of open-set noise . The noisy sample selection mechanism relies on a measure of confidence what we define using the ground truth label of the sample in question and an estimate of distribution of the labels of its neighbours – in order to deal with noisy samples , we adopt a scheme in which the distribution is calculated based both on the ground truth labels and on consistently ( over subsequent iterations ) confident estimates of the labels . Our method is embedded into a standard MixUp and data augmentation framework , and without bells and whistles , such as co-training of multiple models , it achieves state-of-the-art results in both synthetic and realistic noise patterns in CIFAR10 , CIFAR100 , ANIMAL-10N , Clothing1M and WebVision datasets . 2 RELATED WORK . Leaning with noisy data by sample selection Some works focused on sample selection to filter out noisy samples . Jiang et al . ( 2018 ) introduced a pretrained mentor network to guide the selection of a student network . Song et al . ( 2019 ) evaluates the per-sample losses and identify as clean the top r % of the samples – the precise ratio r % depends is either predefined , or is an estimate of the noise level in the specific dataset . Arazo et al . ( 2019 ) proposed to model per-sample losses with a Beta Mixture Model ( BMM ) and split the dataset according to which of the components of the mixture each sample belongs . In a very similar approach , Li et al . ( 2020a ) extended upon Arazo et al . ( 2019 ) by introducing semi-supervised learning to fully utilize the dataset . Related to our work , Bahri et al . ( 2020 ) ; Ortego et al . ( 2021 ) ; Wang et al . ( 2018 ) also utilized the feature space for sample selection . Bahri et al . ( 2020 ) applied KNN for sample selection for closed-set noisy dataset while Ortego et al . ( 2021 ) further proposed to relabel samples based on the KNN voting . Wang et al . ( 2018 ) proposed to reweight samples based on its probability of being outliers in open-set noisy dataset . Self-supervised learning Self-supervised methods attempt to learn good representation without human annotations . In the recent years , the dominant method is contrastive learning with instance discrimination task . MoCo ( He et al. , 2020 ) is an important baseline for current contrastive learning methods , which reuses the memory bank since samples in a single mini-batch may lead to insufficient negative pairs , and proposes a momentum encoder to update the memory bank in real-time to avoid outdated data representation . SimCLR ( Chen et al. , 2020 ) is another important baseline which found that setting mini-batch size to be large enough can eliminate the need for memory bank . More recently , SimSiam ( Chen & He , 2021 ) and BYOL ( Grill et al. , 2020 ) proposed a non-contrastive learning framework which enforce the perturbation consitency between different views , and avoid mode collapse by applying stop-gradient and an extra predictor between two representation vectors . 3 METHOD . 3.1 PROBLEM FORMULATION . Let us denote with X = { xi } Ni=1 , xi ∈ Rd , a training set with the corresponding one-hot vector labels Y = { yi } Ni=1 , yi ∈ { 0 , 1 } K , where K is the number of classes and N is the number of samples . For convenience , let us also denote the index where the one-hot vector yi is one as the label li ∈ { 0 , ... , K } . Finally , let us denote the true labels with Y ′ = { y′i } Ni=1 . Clearly , for an open-set noisy label it is the case that y′i 6= yi , y′i /∈ { 0 , 1 } K , while for closed-set noisy samples y′i 6= yi , y′i ∈ { 0 , 1 } K . 3.2 OVERVIEW OF PROPOSED METHOD . Aiming to deal with potential concurrence of both open-set and close-set noise , we view the classification network as an encoder f that extracts a feature representation and a classification head g that deals with the classification problem in question . The proposed method , named Supervised Self-Supervised ( S3 ) learning , attempts to decouple their training so as to deal with possible noise in the labels , by adopting a two stage , iterative scheme , as outlined in Fig . 2 . In the first stage , we utilize a novel sample selection and a novel relabeling mechanism ( top block in Fig . 2 ) that prepares the set based on which the classifier g should be trained in Stage 2 . The selection mechanism is based on the assumption of smoothness of labels in the feature space , and more specifically , on a consistency measure that we defined based on the annotated label of the sample in question and the distribution of the labels in its neighborhood in the feature space . Relabeling is performed on samples for which the classifier gives confident predictions consistently across subsequent iterations . Clearly , the mechanism relies on the quality of the features extracted by the encoder f and should reject samples whose true labels are not in the class set ( open-set noise ) . This stage is explained in Section 3.3 . In the second stage ( bottom block in Fig . 2 ) , training is performed with two objectives/losses . First , a cross entropy loss on the output of the classifier g , ( i.e. , on g ( f ( . ) ) ) on the samples selected in Stage 1 , that updates both the encoder f and the classifier head g. Second , a self-supervision loss that enforces consistency between the representations of different augmentations of the same sample , and which utilizes all samples , that is both noisy and clean – this updates the encoder f and helps learning a strong feature space on which the selection mechanism of Stage 1 can rely . By contrast to other methods that , either by using a noise transition matrix or in their semisupervised scheme , implicitly relabel all samples ( e.g . DivideMix ( Li et al. , 2020a ) , MOIT ( Ortego et al. , 2021 ) ) and use the new labels to learn , in our method the labels in the noisy set are not used at all . This stage is explained in Section 3.4 .

Summary: The paper proposes a two-stage approach to learning with noisy labels (LNL). 1. a. Clean sample selection based on cosine similarity with k nearest neighbors in embedding space: the average of class distribution of those neighbors should be consistent with the label for sample to be selected.       b. Noisy sample relabeling using temporal self-ensemble: the prediction is defined is average over last L epochs. The sample is relabeled if the confidence of the prediction is higher than some threshold. 2. Training with self-consistency regularization: the regular training with mixup regularization is performed on selected and relabeled samples, with consistency regularization in form of cosine similarity. The paper tests the performance of method on multiple datasets both with synthetic and real-life noise.