Noise against noise: stochastic label noise helps combat inherent label noise

The noise in stochastic gradient descent ( SGD ) provides a crucial implicit regularization effect , previously studied in optimization by analyzing the dynamics of parameter updates . In this paper , we are interested in learning with noisy labels , where we have a collection of samples with potential mislabeling . We show that a previously rarely discussed SGD noise , induced by stochastic label noise ( SLN ) , mitigates the effects of inherent label noise . In contrast , the common SGD noise directly applied to model parameters does not . We formalize the differences and connections of SGD noise variants , showing that SLN induces SGD noise dependent on the sharpness of output landscape and the confidence of output probability , which may help escape from sharp minima and prevent overconfidence . SLN not only improves generalization in its simplest form but also boosts popular robust training methods , including sample selection and label correction . Specifically , we present an enhanced algorithm by applying SLN to label correction . Our code is released1 . 1 INTRODUCTION . The existence of label noise is a common issue in classification since real-world samples unavoidably contain some noisy labels , resulting from annotation platforms such as crowdsourcing systems ( Yan et al. , 2014 ) . In the canonical setting of learning with noisy labels , we collect samples with potential mislabeling , but we do not know which samples are mislabeled since true labels are unobservable . It is troubling that overparameterized Deep Neural Networks ( DNNs ) can memorize noise in training , leading to poor generalization performance ( Zhang et al. , 2017 ; Chen et al. , 2020b ) . Thus , we are urgent for robust training methods that can mitigate the effects of label noise . The noise in stochastic gradient descent ( SGD ) ( Wu et al. , 2020 ) provides a crucial implicit regularization effect for training overparameterized models . SGD noise is previously studied in optimization by analyzing the dynamics of parameter updates , whereas its utility in learning with noisy labels has not been explored to the best of our knowledge . In this paper , we find that the common SGD noise directly applied to model parameters does not endow much robustness , whereas a variant induced by controllable label noise does . Interestingly , inherent label noise is harmful to generalization , while we can mitigate its effects using additional controllable label noise . To prevent confusion , we use stochastic label noise ( SLN ) to indicate the label noise we introduce . Inherent label noise is biased and unknown , fixed when the data is given . SLN is mean-zero and independently sampled for each instance in each training step . Our main contributions are as follows . • We formalize the differences and connections of three SGD noise variants ( Proposition 1- 3 ) and show that SLN induces SGD noise that is dependent on the sharpness of output landscape and the confidence of output probability . ∗Corresponding to : gy.chen @ siat.ac.cn , junjie.ye @ vivo.com . 1https : //github.com/chenpf1025/SLN • Based on the noise covariance , we analyze and illustrate two effects of SLN ( Claim 1 and Claim 2 ) : escaping from sharp minima2 and preventing overconfidence3 . • We empirically show that SLN not only improves generalization in its simplest form but also boosts popular robust training methods , including sample selection and label correction . We present an enhanced algorithm by applying SLN to label correction . In Fig . 1 , we present a quick comparison between models trained with/without SLN on CIFAR-10 with symmetric/asymmetric/instance-dependent/open-set label noise . Throughout this paper , we use CE to indicate a model trained with standard cross-entropy ( CE ) loss without any robust learning techniques , while the standard CE loss is also used by default for methods like SLN . In Section 4 , we will provide more experimental details and more results that comprehensively verify the robustness of SLN on different synthetic noise and real-world noise . Here , the test curves in Fig . 1 show that SLN avoids the drop of test accuracy , with converged test accuracy even higher than the peak accuracy of the model trained with CE . The right two subplots in Fig . 1 show the average loss on clean and noisy samples . When trained with CE , the model eventually memorizes noise , indicated by the drop of average loss on noisy samples . In contrast , SLN largely avoids fitting noisy labels . 2 RELATED WORK . 2.1 SGD NOISE AND THE REGULARIZATION EFFECT . The noise in SGD ( Wu et al. , 2020 ; Wen et al. , 2019 ; Keskar et al. , 2016 ) has long been studied in optimization . It is believed to provide a crucial implicit regularization effect ( HaoChen et al. , 2020 ; Arora et al. , 2019 ; Soudry et al. , 2018 ) for training overparameterized models . The most common SGD noise is spherical Gaussian noise on model parameters ( Ge et al. , 2015 ; Neelakantan et al. , 2015 ; Mou et al. , 2018 ) , while empirical studies ( Wen et al. , 2019 ; Shallue et al. , 2019 ) demonstrate that parameter-dependent SGD noise is more effective . It is shown that the noise covariance containing curvature information performs better for escaping from sharp minima ( Zhu et al. , 2019 ; Daneshmand et al. , 2018 ) . On a quadratically-parameterized model ( Vaskevicius et al. , 2019 ; Woodworth et al. , 2020 ) , HaoChen et al . ( 2020 ) prove that in an over-parameterized regression setting , SGD with label perturbations recovers the sparse groundtruth , whereas SGD with Gaussian noise directly added on gradient descent overfits to dense solutions . In the deep learning scenario , HaoChen et al . ( 2020 ) present primary empirical results showing that SGD noise - induced by Gaussian noise on the gradient of the loss w.r.t . the model ’ s output - avoids performance degeneration of large-batch training . Xie et al . ( 2016 ) discuss the implicit ensemble effect of random label perturbations and demonstrate better generalization performance . In this paper , we provide new insights by analyzing SGD noise variants and the effects , and showing the utility in learning with noisy labels . 2Around sharp minima , the output changes rapidly ( Hochreiter & Schmidhuber , 1997 ; Keskar et al. , 2017 ) . 3The prediction probability on some class approaches 1 . 2.2 ROBUST TRAINING METHODS . Mitigating the effects of label noise is a vital topic in classification , which has a long history ( Ekholm & Palmgren , 1982 ; Natarajan et al. , 2013 ) and attracts much recent interest with several directions explored . 1 ) Malach & Shalev-Shwartz ( 2017 ) ; Han et al . ( 2018b ) ; Yu et al . ( 2019 ) ; Chen et al . ( 2019a ) ; Wei et al . ( 2020 ) propose sample selection methods that train on trusted samples , identified according to training loss , cross-validation or ( dis ) agreement between two models . 2 ) Liu & Tao ( 2015 ) ; Jiang et al . ( 2018 ) ; Ren et al . ( 2018 ) ; Shu et al . ( 2019 ) ; Li et al . ( 2019a ) develop sampleweighting schemes that aim to add higher weights on clean samples . 3 ) Sukhbaatar et al . ( 2015 ) ; Patrini et al . ( 2017 ) ; Hendrycks et al . ( 2018 ) ; Han et al . ( 2018a ) apply loss-correction based on an estimated noise transition matrix . 4 ) Reed et al . ( 2015 ) ; Tanaka et al . ( 2018 ) ; Arazo et al . ( 2019 ) ; Zheng et al . ( 2020 ) ; Chen et al . ( 2020a ) propose label correction based on the model ’ s predictions . 5 ) Ghosh et al . ( 2017 ) ; Zhang & Sabuncu ( 2018 ) ; Xu et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Lyu & Tsang ( 2020 ) ; Ma et al . ( 2020 ) study robust loss functions that have a theoretical guarantee for noisy risk minimization , typically with the assumption that the noise is class-conditional ( Scott et al. , 2013 ; Natarajan et al. , 2013 ) . 6 ) Chen et al . ( 2019b ) ; Menon et al . ( 2020 ) ; Hu et al . ( 2020 ) ; Harutyunyan et al . ( 2020 ) ; Lukasik et al . ( 2020 ) apply regularization techniques to improve generalization under label noise , including explicit regularizations such as manifold regularization ( Belkin et al. , 2006 ) and virtual adversarial training ( Miyato et al. , 2018 ) , and implicit regularizations such as dropout ( Srivastava et al. , 2014 ) , temporal ensembling ( Laine & Aila , 2017 ) , gradient clipping ( Pascanu et al. , 2012 ; Zhang et al. , 2019 ; Menon et al. , 2020 ) and label smoothing ( Szegedy et al. , 2016 ) . 7 ) One can combat label noise with refined training strategies ( Li et al. , 2019b ; 2020 ; Nguyen et al. , 2020 ) that potentially incorporate several techniques , including sample selection/weighting , label correction , meta-learning ( Li et al. , 2019b ) and semi-supervised learning ( Tarvainen & Valpola , 2017 ; Berthelot et al. , 2019 ) . Among these methods , regularization techniques are closely related to the essence of training networks , and studying robustness under label noise provides a new lens of understanding the regularization apart from the optimization lens . 3 METHOD . 3.1 THE DIFFERENCES AND CONNECTIONS OF SGD NOISE VARIANTS . Notations . LetD = { ( x ( i ) , y ( i ) ) } ni=1 be a dataset with noisy labels . For each sample ( x , y ) , its label y may be incorrect and the true label is unobservable . Let f ( x ; θ ) be the neural network model with trainable parameter θ ∈ Rp . For a c-class classification problem , we have the output f ( x ; θ ) ∈ Rc . We use a softmax function S ( f ( x ; θ ) ) ∈ [ 0 , 1 ] c to obtain the probability of each class . The loss on a sample is denoted as ` ( f , y ) . For classification , we use the cross-entropy ( CE ) loss by default . In parameter updates , a sample contributes ∇θ ` ( f , y ) to the gradient descent . With SGD noise , the model is trained with a noisy gradient ∇̃θ ` ( f , y ) . Following the standard notation of the Jacobian matrix , we have ∇θ ` ∈ R1×p , ∇f ` ∈ R1×c , ∇θf ∈ Rc×p , ∇θif ∈ Rc and∇θfi ∈ R1×p . Gaussian noise on the gradient of loss w.r.t . parameters . The most common SGD noise is the spherical Gaussian noise directly added to the gradient to parameters ( Neelakantan et al. , 2015 ) as follow , ∇̃θ ` ( f , y ) = ∇θ ` ( f , y ) + σθzθ , ( 1 ) where σθ > 0 , zθ ∈ R1×p and zθ ∼ N ( 0 , Ip×p ) . Gaussian noise on the gradient of loss w.r.t . the model output . Taking a step further , HaoChen et al . ( 2020 ) study SGD noise induced by label noise on a quadratically-parameterized regression model , whereas for classification , they add mean-zero noise to∇f ` ( f , y ) as follow , ∇̃θ ` ( f , y ) = ( ∇f ` ( f , y ) + σfzf ) · ∇θf , ( 2 ) where σf > 0 , zf ∈ R1×c and zf ∼ N ( 0 , Ic×c ) . Noise induced by SLN . The label perturbation is a common technique ( Xie et al. , 2016 ) , while we provide new insights by analyzing the effects from the lens of SGD noise . Our SLN adds mean-zero Gaussian noise to the one-hot labels , where the noise is independently sampled for each instance in each training step . ∇̃θ ` ( f , y ) = ∇θ ` ( f , y + σyzy ) , ( 3 ) where σy > 0 , zy ∈ Rc and zy ∼ N ( 0 , Ic×c ) . Here , σyzy is the SLN on the label y . In above Eq . ( 1-3 ) , the standard deviation σ is a hyperparameter . Since the SGD noise we introduce is i.i.d for each sample , we consider each single sample ( x , y ) independently in the following propositions . For convenience , we use f , S to indicate the model output f ( x ; θ ) and the softmax output S ( f ( x ; θ ) ) on a sample . The proofs are presented in Appendix A . Proposition 1 . Compared with Eq . ( 1 ) , Eq . ( 2 ) induces noise z ∼ N ( 0 , σ2fM ) on ∇θ ` ( f , y ) , s.t. , M ∈ Rp×p and Mi , j = ( ∇θif ) T∇θjf , ∀i , j ∈ { 1 , · · · , p } . Note that the standard deviation of noise on the i-th parameter θi is σf‖∇θif‖2 , where ‖·‖2 denotes the L2 norm . Proposition 2 . For the cross-entropy loss , compared with Eq . ( 2 ) , Eq . ( 3 ) induces noise z ∼ N ( 0 , σ2yM ) on ∇f ` ( f , y ) , s.t. , M ∈ Rc×c , Mi , i = c ( Si − 1/c ) 2 + ( c − 1 ) /c and Mi , j = c · SiSj − Si − Sj , if i 6= j . Note that the standard deviation of noise on the i-th entry is σy √ c ( Si − 1/c ) 2 + ( c− 1 ) /c . Proposition 3 . For the cross-entropy loss , compared with Eq . ( 1 ) , Eq . ( 3 ) induces noise z ∼ N ( 0 , σ2yM ) on∇θ ` ( f , y ) , s.t. , M ∈ Rp×p and Mi , j = ( ∇θiS S ) T ∇θjS S , ∀i , j ∈ { 1 , · · · , p } , where · S denotes the element-wise division . Note that the standard deviation of noise on the i-th parameter θi is σy ∥∥∥∇θiSS ∥∥∥ 2 .

Summary: The paper tackles the problem of training under noisy labels. It proposes adding random  zero-mean Gaussian noise to the labels during training. It is shown that such a noise induces a variable gradient noise which adaptively increases 1) when the learnt network function has higher curvature around training points and 2) when the output prediction has lower entropy (higher confidence) for the training points. It is claimed that the former property helps avoiding sharper minima which generally improves generalization. The second property avoids overfitting to noise in similar fashion to label smoothing.