Demystifying How Self-Supervised Features Improve Training from Noisy Labels

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have achieved remarkable performance in many areas including speech recognition ( Graves et al. , 2013 ) , computer vision ( Krizhevsky et al. , 2012 ; Lotter et al. , 2016 ) , natural language processing ( Zhang & LeCun , 2015 ) etc . The high-achieving performance often builds on the availability of quality-annotated datasets . In real world scenario , data annotation inevitably brings in label noise which degrades the performance of the network , primarily due to DNNs ’ capability in “ memorizing ” noisy labels ( Zhang et al. , 2016 ) . In the past few years , a number of methods have been proposed to tackle the problem of learning with label noise including robust loss design ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Liu & Guo , 2020 ) , sample selection ( Han et al. , 2018 ; Yu et al. , 2019 ; Cheng et al. , 2021 ) and noise transition matrix estimation ( Patrini et al. , 2017 ; Zhu et al. , 2021b ) . Among all these methods , arguably the most efficient treatment is to adopt robust losses , since sample selection and noise transition matrix estimation always involve training multiple networks or need multi-stage training . Nonetheless , though the designed losses are theoretically proven robust , they often suffer from significant performance drop when noise rate is high ( Wang et al. , 2019 ; Ma et al. , 2020 ; Cheng et al. , 2021 ; Zhu et al. , 2021a ) , hinting that the ability of converging to the optimal classifier is also important . Very recent works ( Zheltonozhskii et al. , 2021 ; Nodet et al. , 2021 ; Ghosh & Lan , 2021 ; Yao et al. , 2021 ; Tan et al. , 2021 ) started applying self-supervised learning to solving the problem of learning from noisy labels . The experiments show that methods built on the self-supervised features can achieve exceptional performance even when the noise rate is high and largely outperform previously reported SOTA approaches . Despite the empirical observations , the reasons why self-supervised features lead to significant performance improvement are not well understood . In this paper , we provide theoretical insights to understand how self-supervised features improve classification with label noise and perform extensive experiments to support our theory . Our analysis provides a new understanding on learning with noisy labels from the perspective of self-supervised learning . We summarize our main contributions below : • We theoretically and experimentally show that by using self-supervised features to fine-tune the network on noisy datasets , Cross Entropy itself is robust to label noise ( Theorem 1–2 ) . The theory also answers the question of whether or not to fix the encoder when performing fine-tuning . • We theoretically and experimentally show that by using self-supervised features , a regularizer commonly used in knowledge distillation ( Hinton et al. , 2015 ) ( where the dataset does not contain label noise ) can greatly alleviate over-fitting problem of DNN on noisy datasets ( Theorem 3–4 ) . 1.1 RELATED WORKS . Learning with Noisy Labels : Due to the over-fitting problem of DNN , many works design robust loss to improve the robustness of neural networks . ( Ghosh et al. , 2017 ) proves MAE is inherently robust to label noise . However , MAE has a severe under-fitting problem . ( Zhang & Sabuncu , 2018 ) propose a loss which can combine both the advantage of MAE and CE , exhibiting good performance on noisy datasets . ( Liu & Guo , 2020 ) introduces peer loss , which is proven statistically robust to label noise without knowing noise rate . The extension of peer loss also shows good performance on instance-dependent label noise ( Cheng et al. , 2021 ; Zhu et al. , 2021a ) . Another efficient approach to combat label noise is by sample selection ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ; Northcutt et al. , 2021 ; Yao et al. , 2020 ; Wei et al. , 2020 ; Zhang et al. , 2020 ) . These methods regard “ small loss ” examples as clean ones and always involve training multiple networks to select clean samples . Semi-supervised learning is also popular and effective on learning with noisy labels in recent years . Some works ( Li et al. , 2020 ; Nguyen et al. , 2020 ) first perform clustering on the sample loss and divide the samples into clean ones and noisy ones . Then drop the labels of the ” noisy samples ” and perform semi-supervised learning on all the samples . Self-Supervised Learning : The goal of self-supervised learning ( SSL ) is to learn good presentation without using the information of the labels . Generally , the methods of SSL can be divided into two categories : designing pretext tasks or designing loss functions . The designed tasks or losses do not involve any labels . Some popular tasks include patch orderings ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ) , tracking ( Wang & Gupta , 2015 ) or clustering features ( Caron et al. , 2018 ; 2019 ) . However , the SSL performance of pretext tasks is limited . Recent SOTA methods for SSL is by designing contrastive loss functions . The representative works include Moco ( He et al. , 2020 ) and SimCLR ( Chen et al. , 2020 ) which train neural networks based on InfoNCE loss ( Oord et al. , 2018 ) . In our paper , we also adopt InfoNCE for performing self-supervised training to get SSL pre-trained features . The first part of our paper relates to the works that apply SSL features to perform finetuning on noisy dataset ( Nodet et al. , 2021 ; Ghosh & Lan , 2021 ) and our goal is to build theoretical understanding on this aspect . Knowledge Distillation : The second part of our paper is very related to the research field of knowledge distillation ( KD ) . The original idea of KD can be traced back to model compression ( Buciluǎ et al. , 2006 ) , where authors demonstrate the knowledge acquired by a large ensemble of models can be transferred to a single small model . ( Hinton et al. , 2015 ) generalize this idea to neural networks and show a small , shallow network can be improved through a teacher-student framework . Due to its great applicability , KD has gained more and more attention in recent years and numerous methods have been proposed to perform efficient distillation ( Mirzadeh et al. , 2020 ; Zhang et al. , 2018 ; 2019 ) . However , the dataset used in KD is assumed to be clean . Thus it is hard to connect KD with learning with noisy labels . In this paper , we theoretically and experimentally show that a regularizer generally used in KD ( Park et al. , 2019 ) can alleviate the over-fitting problem on noisy data by using SSL features which offers a new alternative for dealing with label noise . 2 PRELIMINARY . We introduce preliminaries and notations including definitions and problem formulation . Problem Formulation : Consider a classification problem on a set of N training examples denoted by D : = { ( xn , yn ) } n∈ [ N ] , where [ N ] : = { 1 , 2 , · · · , N } is the set of example indices . Examples ( xn , yn ) are drawn according to random variables ( X , Y ) from a joint distribution D. The classification task aims to identify a classifier C that maps X to Y accurately . Our theoretical analyses focus on binary classifications thus Y ∈ { 0 , 1 } . In real-world applications , the learner can only observe noisy labels . For instance , human annotators may wrongly label some images containing cats as ones that contain dogs accidentally or irresponsibly . The label noise of each instance is assumed to be class-dependent ( Liu & Tao , 2015 ) , i.e. , P ( Ỹ |Y ) = P ( Ỹ |X , Y ) . Thus the error rates are defined as e+ = P ( Ỹ = 0|Y = 1 ) , e− = P ( Ỹ = 1|Y = 0 ) . The corresponding noisy dataset and distribution are denoted by D̃ : = { ( xn , ỹn ) } n∈ [ N ] and D̃ . Define the expected risk of a classifier C as R ( C ) = ED [ 1 ( C ( X ) 6= Y ) ] . The goal is to learn a classifier C from the noisy distribution D̃ which also minimizes R ( C ) , i.e. , learn the Bayes optimal classifier such that CBayes ( x ) = arg maxi∈ { 0,1 } P ( Y = i|X = x ) . For better presentation , we define the following notations : X+ = X|Y = 1 , X− = X|Y = 0 , and Xclean = X|Y = Ỹ , Xnoisy = X|Y 6= Ỹ . Evaluation of SSL ( Self-Supervised Learning ) : SSL is usually evaluated by two steps : First , use SSL to train an encoder f with only unlabeled data X , then add a linear classifier g following the pre-trained encoder f and only fine-tune g on ( X , Y ) with a fixed f . The high-level intuition is that , if the encoder f is well learned by SSL , only fine-tuning linear classifier g is often sufficient to achieve good performance on test data . If the test performance is comparable to SL ( Supervised Learning ) , we call the gap between SSL and SL is small ( Chen et al. , 2020 ) . Denote by G the space of linear classifier g. Fine-tuning linear layer g on ( X , Y ) ∼ D can be represented as : min g∈G ED [ CE ( g ( f ( X ) ) , Y ) ] , where CE denotes the Cross-Entropy loss . Note the dimension of f ( X ) is determined by the network structure , e.g. , 512 for ResNet34 . In binary classifications , g ( f ( X ) ) ∈ [ 0 , 1 ] where g ( f ( X ) ) < 0.5 indicates predicting class-0 and g ( f ( X ) ) > 0.5 corresponds to class-1 . g ( f ( X ) ) is supposed to predict the same label as CBayes ( x ) . 3 ROBUSTNESS OF CROSS-ENTROPY WITH SSL FEATURES . We will analyze the robustness of Cross-Entropy with SSL features by comparing three different learning paths as illustrated in Figure 1 . Path-1 is the traditional learning path that learns both encoder f and linear classifier g at the same time . Path-2 is the strategy applied in ( Ghosh & Lan , 2021 ) that firstly pre-trains encoder f with SSL , then treats the pre-trained model as a network initialization and jointly fine-tunes f and g. Path-3 is an alternate SSL-based path that first learns the encoder f then only fine-tunes the linear classifier g with fixed f . 3.1 THEORETICAL TOOLS . We prepare some theoretical tools for our analyses . Our first theorem focuses on demonstrating the effectiveness of only fine-tuning linear classifier g as in Path-3 . We present Theorem 1 below . Theorem 1 Let g1 = arg ming∈G ED [ CE ( g ( f ( X ) ) , Y ) ] , g2 = arg ming∈G ED̃ [ CE ( g ( f ( X ) ) , Ỹ ) ] . Then if e+ = e− < 0.5 , we have : Round ( g1 ( f ( X ) ) ) = Round ( g2 ( f ( X ) ) ) ( 1 ) where f is fixed encoder and g is the linear classifier , g ( f ( · ) ) denotes the output whose value ranges from 0 to 1 . Round ( p ) is a predictor function that outputs 1 if p > 0.5 and outputs 0 otherwise . Theorem 1 shows with balanced error rates , simply fine-tuning a linear classifier g on the noisy data distribution D̃ can achieve the same decision boundary as the optimal linear classifier obtained from the corresponding clean distribution D . i.e. , g1 ( f ) and g2 ( f ) have the same predictions for all the samples . Theorem 1 can be generalized to the case with an arbitrary classifier beyond linear . However , admittedly with limited data , training complicated classifiers is hard to converge to the optimal decision boundary . We defer more details to the next subsection . We then evaluate the former part of Path-3 , i.e. , the performance of SSL . Recall in Section 2 , SSL is usually evaluated by performance gap between f ◦ gBayes and CBayes , where gBayes is the optimal linear classifier trained on D , f ◦ gBayes denotes the joint model given by gBayes ( f ( X ) ) . We consider a tractable case in Assumption 1 . Assumption 1 The encoder outputs f ( X+ ) and f ( X− ) follow Gaussian distribution with parameters ( µ1 , Σ ) and ( µ2 , Σ ) , where Σ = σ2 · I , I is the identity matrix . Assumption 1 states that the self-supervised features for each class follow simple Gaussian distributions . We check the effectiveness of this assumption by Figure 2 . It can be observed that the features of each class may have overlaps , but a good SSL method is supposed to return features with good separations ( small overlaps ) . In Assumption 1 , we use ||µ1 − µ2|| and σ to capture the overlapping area of two classes . If ||µ1 − µ2|| is large and σ is small , then there exists small overlapping . Based on this assumption , we show the performance of SSL in Theorem 2 . Theorem 2 If P ( Y = 1 ) = P ( Y = 0 ) , the risk ( error rate ) of Bayes optimal classifier f ◦ gBayes follows as : R ( f ◦ gBayes ) = 1− Φ ( ||µ1 − µ2|| 2 · σ ) ( 2 ) where Φ is the cumulative distribution function ( CDF ) of the standard Gaussian distribution . Wrap-up With Theorem 1 , we know CE is robust , the performance of which is subject to f . Theorem 2 implies that if SSL features learned by f exhibit good property , i.e. , when ||µ1 − µ2|| is large and σ is small , only fine-tuning g can approach the Bayes optimal classifier CBayes . Therefore , good SSL features induce high performance . In summary , Theorem 1 and Theorem 2 connect SSL features with robustness and generalization ability of CE loss , providing an insight on why SSL features improve classification with label noise .

Summary: This paper studies the usefulness of self-supervised features when encountering data with noisy labels. It presents an array of ideas on this topic: including the question of fine-tuning pre-trained representations, using ideas from distillation as regularisation to improve generalisation. Some theoretical and empirical results are relating to the authors arguments.