Demystifying How Self-Supervised Features Improve Training from Noisy Labels

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have achieved remarkable performance in many areas including speech recognition ( Graves et al. , 2013 ) , computer vision ( Krizhevsky et al. , 2012 ; Lotter et al. , 2016 ) , natural language processing ( Zhang & LeCun , 2015 ) etc . The high-achieving performance often builds on the availability of quality-annotated datasets . In real world scenario , data annotation inevitably brings in label noise which degrades the performance of the network , primarily due to DNNs ’ capability in “ memorizing ” noisy labels ( Zhang et al. , 2016 ) . In the past few years , a number of methods have been proposed to tackle the problem of learning with label noise including robust loss design ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Liu & Guo , 2020 ) , sample selection ( Han et al. , 2018 ; Yu et al. , 2019 ; Cheng et al. , 2021 ) and noise transition matrix estimation ( Patrini et al. , 2017 ; Zhu et al. , 2021b ) . Among all these methods , arguably the most efficient treatment is to adopt robust losses , since sample selection and noise transition matrix estimation always involve training multiple networks or need multi-stage training . Nonetheless , though the designed losses are theoretically proven robust , they often suffer from significant performance drop when noise rate is high ( Wang et al. , 2019 ; Ma et al. , 2020 ; Cheng et al. , 2021 ; Zhu et al. , 2021a ) , hinting that the ability of converging to the optimal classifier is also important . Very recent works ( Zheltonozhskii et al. , 2021 ; Nodet et al. , 2021 ; Ghosh & Lan , 2021 ; Yao et al. , 2021 ; Tan et al. , 2021 ) started applying self-supervised learning to solving the problem of learning from noisy labels . The experiments show that methods built on the self-supervised features can achieve exceptional performance even when the noise rate is high and largely outperform previously reported SOTA approaches . Despite the empirical observations , the reasons why self-supervised features lead to significant performance improvement are not well understood . In this paper , we provide theoretical insights to understand how self-supervised features improve classification with label noise and perform extensive experiments to support our theory . Our analysis provides a new understanding on learning with noisy labels from the perspective of self-supervised learning . We summarize our main contributions below : • We theoretically and experimentally show that by using self-supervised features to fine-tune the network on noisy datasets , Cross Entropy itself is robust to label noise ( Theorem 1–2 ) . The theory also answers the question of whether or not to fix the encoder when performing fine-tuning . • We theoretically and experimentally show that by using self-supervised features , a regularizer commonly used in knowledge distillation ( Hinton et al. , 2015 ) ( where the dataset does not contain label noise ) can greatly alleviate over-fitting problem of DNN on noisy datasets ( Theorem 3–4 ) . 1.1 RELATED WORKS . Learning with Noisy Labels : Due to the over-fitting problem of DNN , many works design robust loss to improve the robustness of neural networks . ( Ghosh et al. , 2017 ) proves MAE is inherently robust to label noise . However , MAE has a severe under-fitting problem . ( Zhang & Sabuncu , 2018 ) propose a loss which can combine both the advantage of MAE and CE , exhibiting good performance on noisy datasets . ( Liu & Guo , 2020 ) introduces peer loss , which is proven statistically robust to label noise without knowing noise rate . The extension of peer loss also shows good performance on instance-dependent label noise ( Cheng et al. , 2021 ; Zhu et al. , 2021a ) . Another efficient approach to combat label noise is by sample selection ( Jiang et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ; Northcutt et al. , 2021 ; Yao et al. , 2020 ; Wei et al. , 2020 ; Zhang et al. , 2020 ) . These methods regard “ small loss ” examples as clean ones and always involve training multiple networks to select clean samples . Semi-supervised learning is also popular and effective on learning with noisy labels in recent years . Some works ( Li et al. , 2020 ; Nguyen et al. , 2020 ) first perform clustering on the sample loss and divide the samples into clean ones and noisy ones . Then drop the labels of the ” noisy samples ” and perform semi-supervised learning on all the samples . Self-Supervised Learning : The goal of self-supervised learning ( SSL ) is to learn good presentation without using the information of the labels . Generally , the methods of SSL can be divided into two categories : designing pretext tasks or designing loss functions . The designed tasks or losses do not involve any labels . Some popular tasks include patch orderings ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ) , tracking ( Wang & Gupta , 2015 ) or clustering features ( Caron et al. , 2018 ; 2019 ) . However , the SSL performance of pretext tasks is limited . Recent SOTA methods for SSL is by designing contrastive loss functions . The representative works include Moco ( He et al. , 2020 ) and SimCLR ( Chen et al. , 2020 ) which train neural networks based on InfoNCE loss ( Oord et al. , 2018 ) . In our paper , we also adopt InfoNCE for performing self-supervised training to get SSL pre-trained features . The first part of our paper relates to the works that apply SSL features to perform finetuning on noisy dataset ( Nodet et al. , 2021 ; Ghosh & Lan , 2021 ) and our goal is to build theoretical understanding on this aspect . Knowledge Distillation : The second part of our paper is very related to the research field of knowledge distillation ( KD ) . The original idea of KD can be traced back to model compression ( Buciluǎ et al. , 2006 ) , where authors demonstrate the knowledge acquired by a large ensemble of models can be transferred to a single small model . ( Hinton et al. , 2015 ) generalize this idea to neural networks and show a small , shallow network can be improved through a teacher-student framework . Due to its great applicability , KD has gained more and more attention in recent years and numerous methods have been proposed to perform efficient distillation ( Mirzadeh et al. , 2020 ; Zhang et al. , 2018 ; 2019 ) . However , the dataset used in KD is assumed to be clean . Thus it is hard to connect KD with learning with noisy labels . In this paper , we theoretically and experimentally show that a regularizer generally used in KD ( Park et al. , 2019 ) can alleviate the over-fitting problem on noisy data by using SSL features which offers a new alternative for dealing with label noise . 2 PRELIMINARY . We introduce preliminaries and notations including definitions and problem formulation . Problem Formulation : Consider a classification problem on a set of N training examples denoted by D : = { ( xn , yn ) } n∈ [ N ] , where [ N ] : = { 1 , 2 , · · · , N } is the set of example indices . Examples ( xn , yn ) are drawn according to random variables ( X , Y ) from a joint distribution D. The classification task aims to identify a classifier C that maps X to Y accurately . Our theoretical analyses focus on binary classifications thus Y ∈ { 0 , 1 } . In real-world applications , the learner can only observe noisy labels . For instance , human annotators may wrongly label some images containing cats as ones that contain dogs accidentally or irresponsibly . The label noise of each instance is assumed to be class-dependent ( Liu & Tao , 2015 ) , i.e. , P ( Ỹ |Y ) = P ( Ỹ |X , Y ) . Thus the error rates are defined as e+ = P ( Ỹ = 0|Y = 1 ) , e− = P ( Ỹ = 1|Y = 0 ) . The corresponding noisy dataset and distribution are denoted by D̃ : = { ( xn , ỹn ) } n∈ [ N ] and D̃ . Define the expected risk of a classifier C as R ( C ) = ED [ 1 ( C ( X ) 6= Y ) ] . The goal is to learn a classifier C from the noisy distribution D̃ which also minimizes R ( C ) , i.e. , learn the Bayes optimal classifier such that CBayes ( x ) = arg maxi∈ { 0,1 } P ( Y = i|X = x ) . For better presentation , we define the following notations : X+ = X|Y = 1 , X− = X|Y = 0 , and Xclean = X|Y = Ỹ , Xnoisy = X|Y 6= Ỹ . Evaluation of SSL ( Self-Supervised Learning ) : SSL is usually evaluated by two steps : First , use SSL to train an encoder f with only unlabeled data X , then add a linear classifier g following the pre-trained encoder f and only fine-tune g on ( X , Y ) with a fixed f . The high-level intuition is that , if the encoder f is well learned by SSL , only fine-tuning linear classifier g is often sufficient to achieve good performance on test data . If the test performance is comparable to SL ( Supervised Learning ) , we call the gap between SSL and SL is small ( Chen et al. , 2020 ) . Denote by G the space of linear classifier g. Fine-tuning linear layer g on ( X , Y ) ∼ D can be represented as : min g∈G ED [ CE ( g ( f ( X ) ) , Y ) ] , where CE denotes the Cross-Entropy loss . Note the dimension of f ( X ) is determined by the network structure , e.g. , 512 for ResNet34 . In binary classifications , g ( f ( X ) ) ∈ [ 0 , 1 ] where g ( f ( X ) ) < 0.5 indicates predicting class-0 and g ( f ( X ) ) > 0.5 corresponds to class-1 . g ( f ( X ) ) is supposed to predict the same label as CBayes ( x ) . 3 ROBUSTNESS OF CROSS-ENTROPY WITH SSL FEATURES . We will analyze the robustness of Cross-Entropy with SSL features by comparing three different learning paths as illustrated in Figure 1 . Path-1 is the traditional learning path that learns both encoder f and linear classifier g at the same time . Path-2 is the strategy applied in ( Ghosh & Lan , 2021 ) that firstly pre-trains encoder f with SSL , then treats the pre-trained model as a network initialization and jointly fine-tunes f and g. Path-3 is an alternate SSL-based path that first learns the encoder f then only fine-tunes the linear classifier g with fixed f . 3.1 THEORETICAL TOOLS . We prepare some theoretical tools for our analyses . Our first theorem focuses on demonstrating the effectiveness of only fine-tuning linear classifier g as in Path-3 . We present Theorem 1 below . Theorem 1 Let g1 = arg ming∈G ED [ CE ( g ( f ( X ) ) , Y ) ] , g2 = arg ming∈G ED̃ [ CE ( g ( f ( X ) ) , Ỹ ) ] . Then if e+ = e− < 0.5 , we have : Round ( g1 ( f ( X ) ) ) = Round ( g2 ( f ( X ) ) ) ( 1 ) where f is fixed encoder and g is the linear classifier , g ( f ( · ) ) denotes the output whose value ranges from 0 to 1 . Round ( p ) is a predictor function that outputs 1 if p > 0.5 and outputs 0 otherwise . Theorem 1 shows with balanced error rates , simply fine-tuning a linear classifier g on the noisy data distribution D̃ can achieve the same decision boundary as the optimal linear classifier obtained from the corresponding clean distribution D . i.e. , g1 ( f ) and g2 ( f ) have the same predictions for all the samples . Theorem 1 can be generalized to the case with an arbitrary classifier beyond linear . However , admittedly with limited data , training complicated classifiers is hard to converge to the optimal decision boundary . We defer more details to the next subsection . We then evaluate the former part of Path-3 , i.e. , the performance of SSL . Recall in Section 2 , SSL is usually evaluated by performance gap between f ◦ gBayes and CBayes , where gBayes is the optimal linear classifier trained on D , f ◦ gBayes denotes the joint model given by gBayes ( f ( X ) ) . We consider a tractable case in Assumption 1 . Assumption 1 The encoder outputs f ( X+ ) and f ( X− ) follow Gaussian distribution with parameters ( µ1 , Σ ) and ( µ2 , Σ ) , where Σ = σ2 · I , I is the identity matrix . Assumption 1 states that the self-supervised features for each class follow simple Gaussian distributions . We check the effectiveness of this assumption by Figure 2 . It can be observed that the features of each class may have overlaps , but a good SSL method is supposed to return features with good separations ( small overlaps ) . In Assumption 1 , we use ||µ1 − µ2|| and σ to capture the overlapping area of two classes . If ||µ1 − µ2|| is large and σ is small , then there exists small overlapping . Based on this assumption , we show the performance of SSL in Theorem 2 . Theorem 2 If P ( Y = 1 ) = P ( Y = 0 ) , the risk ( error rate ) of Bayes optimal classifier f ◦ gBayes follows as : R ( f ◦ gBayes ) = 1− Φ ( ||µ1 − µ2|| 2 · σ ) ( 2 ) where Φ is the cumulative distribution function ( CDF ) of the standard Gaussian distribution . Wrap-up With Theorem 1 , we know CE is robust , the performance of which is subject to f . Theorem 2 implies that if SSL features learned by f exhibit good property , i.e. , when ||µ1 − µ2|| is large and σ is small , only fine-tuning g can approach the Bayes optimal classifier CBayes . Therefore , good SSL features induce high performance . In summary , Theorem 1 and Theorem 2 connect SSL features with robustness and generalization ability of CE loss , providing an insight on why SSL features improve classification with label noise .

Summary: This paper studies how self-supervised pre-training impacts the resistance of the neural network to noisy labels. The paper provides both theoretical analyses and numerical experiments of their study.  The main contribution of the provided study is (i) given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise (ii) providing insights for how knowledge distilled from SSL features can alleviate the over-fitting problem.