Implicit Regularization Effects of Unbiased Random Label Noises with SGD

1 INTRODUCTION . Stochastic Gradient Descent ( SGD ) has been widely used as an effective way to train deep neural networks with large datasets ( Bottou , 1991 ) . While the mini-batch sampling strategy was firstly proposed to lower the cost of computation per iteration , it has been consider to incorporate an implicit regularizer preventing the learning process from converging to the local minima with poor generalization performance ( Zhang et al. , 2017 ; Zhu et al. , 2019 ; Jastrzebski et al. , 2017 ; Hoffer et al. , 2017 ; Keskar et al. , 2017 ) . To interpret such implicit regularization , one can model SGD as gradient descent ( GD ) with gradient noises caused by mini-batch sampling ( Bottou et al. , 2018 ) . Studies have demonstrated the potentials of such implicit regularization or gradient noises to improve the generalization performance of learning from both theoretical ( Mandt et al. , 2017 ; Chaudhari & Soatto , 2018 ; Hu et al. , 2019b ; Simsekli et al. , 2019 ) and empirical aspects ( Zhu et al. , 2019 ; Hoffer et al. , 2017 ; Keskar et al. , 2017 ) . In summary , gradient noises keep SGD away from converging to the sharp local minima that generalizes poorly ( Zhu et al. , 2019 ; Hu et al. , 2019b ; Simsekli et al. , 2019 ) and would select a flat minima ( Hochreiter & Schmidhuber , 1997 ) as the outcome of learning . In this work , we aim at investigating the influence of random label noises to the implicit regularization under mini-batch sampling of SGD . To simplify our research , we assume the training dataset as a set of vectorsD = { x1 , x2 , x3 , . . . , xN } .The label ỹi for every vector xi ∈ D is the noisy response of the true neural network f∗ ( x ) such that ỹi = yi + εi , yi = f ∗ ( xi ) , and E [ εi ] = 0 , var [ εi ] = σ2 , ( 1 ) where the label noise εi is assumed to be an independent zero-mean random variable . In our work , the random label noises can be either ( 1 ) drawn from probability distributions before training steps ( but dynamized by mini-batch sampling of SGD ) or ( 2 ) realized by the random variables per training iteration ( Han et al. , 2018 ) . Thus learning is to approximate f ( x , θ̂ ) that beats f∗ ( x ) , such that θ̂ ← argmin ∀θ∈Rd { 1 N N∑ i=1 L̃i ( θ ) : = 1 N N∑ i=1 ( f ( xi , θ ) − ỹi ) 2 } . ( 2 ) Inspired by ( Hochreiter & Schmidhuber , 1997 ; Zhu et al. , 2019 ) , our work studies how unbiased label noises εi ( 1 ≤ i ≤ N ) would affect the “ selection ” of θ̂ from possible solutions , in the viewpoint of learning dynamics ( Saxe et al. , 2014 ) of SGD under mini-batch sampling ( Li et al. , 2017 ; Wu et al. , 2020 ; Hu et al. , 2019b ) . Contributions . Our analysis shows that under mild conditions , with gradients of label-noisy losses , SGD might incorporate an additional data-dependent noise term , complementing with the stochastic gradient noises ( Li et al. , 2017 ; Wu et al. , 2020 ) of label-noiseless losses , through resampling the samples with label noises ( Li et al. , 2018 ) or dynamically adding noises to labels over iterations ( Han et al. , 2018 ) . We consider such noises as an implicit regularization caused by unbiased label noises , and interpret the effects of such noises as a solution selector of learning procedure . More specifically , this work has made unique contributions as follow . ( 1 ) Implicit Regularizer . We reviewed the preliminaries ( Li et al. , 2017 ; Ali et al. , 2019 ; Hu et al. , 2019b ; Wu et al. , 2020 ) and extent the analytical framework in ( Li et al. , 2017 ) to interpret the effects of unbiased label noises as an additional implicit regularizer on top of the continuous-time dynamics of SGD . Through discretizing the continuous-time dynamics of label-noisy SGD , we write discrete-time approximation to the learning dynamics , denoted as θULNk for k = 1 , 2 , . . . , as θULNk+1 ← θULNk − η N N∑ i=1 ∇L∗i ( θULNk ) + ξ∗k ( θULNk ) + ξULNk ( θULNk ) , ( 3 ) where L∗i ( θ ) = ( f ( xi , θ ) − f∗ ( xi ) ) 2 refers to the label-noiseless loss function with sample xi and the true ( noiseless ) label yi , the noise term ξ∗k ( θ ) refers to the stochastic gradient noise ( Li et al. , 2017 ) of label-noiseless loss function L∗i ( θ ) , then we can obtain the new implicit regularizer caused by the unbiased label noises ( ULN ) for ∀θ ∈ Rd , which can be approximated as follow ξULNk ( θ ) ≈ √ η B ( σ2 N N∑ i=1 ∇θf ( xi , θ ) ∇θf ( xi , θ ) > ) 1 2 zk , and zk ∼ N ( 0d , Id ) , ( 4 ) where zk refers to a random noise vector drawn from the standard Gaussian distribution , θk refers to the parameters of network in the kth iteration , ( · ) 1/2 refers to the Chelosky decomposition of the matrix , ∇θf ( xi , θ ) = ∂f ( xi , θ ) /∂θ refers to the gradient of the neural network output for sample xi over the parameter θk , and B and η are defined as the learning rate and the batch size of SGD respectively . Obviously , the strength of such implicit regularizer is controlled by σ2 , B and η . ( 2 ) Effects to Linear Regression . To understand behaviors of the implicit regularizer ξULNt ( θt ) to the learning dynamics , we studied SGD over Ordinary Least Square ( OLS ) .With the proposed model , we could easily obtain the implicit regularizer such that ξULNk ( β ) ≈ √ η/B ( σ2Σ̄N ) 1/2 zk and zk ∼ N ( 0d , Id ) ( 5 ) where Σ̄N = 1N ∑N i=1 xix > i referring to the sample covariance matrix of the training dataset . Our theoretical elaboration suggests that SGD with unbiased random label noises would converge to a distribution of Gaussian-alike centered at the optimal solution of OLS , where the span and shape of the distribution would controlled by σ2 and Σ̄N when η and B are constant . We conducted the extensive experiments using SGD with various σ2 and Σ̄N , and successfully obtain the results that coincide our theories and directly visualize the effects of the implicit regularizer over the path of learning and converging distribution of SGD for noisy linear regression . ( 3 ) Inference Stabilizer . The regularization effects of unbiased random label noises should be Ezk ∥∥ξULNk ( θk ) ∥∥22 ≈ ησ2BN N∑ i=1 ‖∇θf ( xi , θk ) ‖22 = ησ2 BN N∑ i=1 ∥∥∥∥ ∂∂θf ( xi , θk ) ∥∥∥∥2 2 , ( 6 ) where ∇θf ( x , θ ) refers to the gradient of f over θ and the effects is controlled by the batch size B and the variance of label noises σ2 . We extend above results to understand the newly-fashioned noisy self-distillation ( Zhang et al. , 2019a ; Kim et al. , 2020 ) paradigms , where a well-trained model is supposed to be further improved through learning from its noisy outputs . Our analysis showed that , when the new convergence achieved , the noisy self-distillation strategy would prefer to re-select a model with a lower neural network gradient norm 1N ∑N i=1 ‖∇θf ( xi , θ ) ‖22 , where the gradient norm characterizes the variation/instability of neural network inference results ( over perturbations ) around parameters of the model . We carry out extensive experiments while results backup our theories . Note that while the earlier work ( Bishop , 1995 ) found training with input noises can also bring regularization effects , our work focuses on the observational noises on labels . 2 PRELIMINARIES AND RELATED WORK . SGD Dynamics and Implicit Regularization We follow settings in ( Li et al. , 2017 ) and consider SGD as an algorithm that , in the kth iteration with the estimate θk , it randomly picks up a B-length subset of samples from the training dataset i.e. , Bk ⊂ D , and estimates the mini-batch stochastic gradient 1b ∑ ∀xi∈Bk ∇Li ( θk ) , then updates the estimate for θk+1 based on θk , as follow θk+1 ← ( θk − η B ∑ ∀xi∈Bk ∇Li ( θk ) ) = ( θk − η N ∑ ∀xi∈D ∇Li ( θk ) + √ ηVk ( θk ) ) , ( 7 ) where η refers to the step-size of SGD , and Vk ( θk ) refers to a stochastic gradient noise term caused by mini-batch sampling . The noise would converge to zero with increasing batch size , as follow Vk ( θk ) = √ η ( 1 N ∑ ∀xi∈D ∇Li ( θk ) − 1 B ∑ ∀xi∈Bk ∇Li ( θk ) ) → 0d , as B → N . ( 8 ) Let define ΣSGDN ( θk ) as the sample covariance matrix of loss gradients ∇Li ( θk ) for 1 ≤ i ≤ N , where we follow ( Li et al. , 2017 ) and do not make low-rank assumptions on ΣSGDN ( θk ) . Under mild conditions ( Li et al. , 2017 ; Chaudhari & Soatto , 2018 ) , one can approximate SGD as θ̄k such that θ̄k+1 ← θ̄k− η N ∑ ∀xi∈D ∇Li ( θ̄k ) + ξk ( θ̄k ) , ξk ( θ̄k ) = √ η B ( ΣSGDN ( θ̄k ) ) 1 2 zk , zk ∼ N ( 0 , Id ) . ( 9 ) The implicit regularizer of SGD could be considered as ξk ( θ̄k ) = √ η/B ( ΣSGDN ( θ̄k ) ) 1 2 zk which is data-dependent and controlled by the learning rate η and batch size B ( Smith et al. , 2018 ) . ( Mandt et al. , 2017 ; Chaudhari & Soatto , 2018 ; Hu et al. , 2019b ) discussed SGD for varational inference and enabled novel applications to samplers ( Zhang et al. , 2019b ; Xiong et al. , 2019 ) . To understand the effect to generalization performance , ( Zhu et al. , 2019 ; Smith et al. , 2018 ) studied the escaping behavior from the sharp local minima ( Keskar et al. , 2017 ) and convergence to the flat ones . Finally , ( Gidel et al. , 2019 ) studied regularization effects to linear DNNs and ( Wu et al. , 2020 ) proposed new multiplicative noises to interpret SGD and obtain stronger theoretical properties . SGD Implicit Regularization for Ordinary Least Square ( OLS ) The most recent and relevant work in this area is ( Ali et al. , 2019 ; 2020 ) , where the same group of authors studied the implicit regularization of gradient descent and stochastic gradient descent for OLS . They investigated an implicit regularizer of ` 2-norm alike on the parameter , which regularizes OLS as a Ridge estimator with decaying penalty . Prior to these efforts , F. Bach and his group have studied the convergence of gradient-based solutions for linear regression with OLS and regularized estimators under both noisy and noiseless settings in ( Dieuleveut et al. , 2017 ; Marteau-Ferey et al. , 2019 ; Berthier et al. , 2020 ) . Self-Distillation and Noisy Students Self-distillation ( Zhang et al. , 2019a ; Xie et al. , 2020 ; Xu et al. , 2020 ; Kim et al. , 2020 ) has been examined as an effective way to further improve the generalization performance of well-trained models . Such strategies enable knowledge distillation using the well-trained ones as teacher models and optionally adding noises ( e.g. , dropout , stochastic depth , and label smoothing or potentially the label noises ) onto training procedure of student models . Discussion on the Relevant Work Though tremendous pioneering studies have been done in this area , we still make contributions in above three categories . First of all , this work characterizes the implicit regularization effects of label noises to SGD dynamics . Compared to ( Ali et al. , 2019 ; 2020 ) working on linear regression , our model interpreted general learning tasks . Even from linear regression perspectives ( Ali et al. , 2019 ; 2020 ; Berthier et al. , 2020 ) , we precisely measured the gaps between SGD dynamics with and without label noises using the continuous-time diffusion.Compared to ( Lopez-Paz et al. , 2016 ; Kim et al. , 2020 ) , our analysis emphasized role of the implicit regularizer caused by label noises for model selection , where models with high inferential stability would be selected . ( Li et al. , 2020 ) is the most relevant work to us , where authors studied the early stopping of gradient descent under label noises via neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) approximation . Our work made the analyze for SGD without assumptions for approximation such as NTK . To best of our knowledge , this work is the first to understand the effects of unbiased label noises to SGD dynamics , by addressing technical issues including implicit regularization , OLS , self-distillation , model selection , and the stability inference results .

Summary: This paper studies the implicit regularization effect that arises from using stochastic gradient descent with label noise and squared loss. They derive the expression for the implicit regularization term and show that it favors solutions which are stable against perturbations of the parameters. This paper validates their empirical findings using SGD on linear regression task with label noise. This paper also uses their results to study the self distillation technique.