Shape Matters: Understanding the Implicit Bias of the Noise Covariance

The noise in stochastic gradient descent ( SGD ) provides a crucial implicit regularization effect for training overparameterized models . Prior theoretical work largely focuses on spherical Gaussian noise , whereas empirical studies demonstrate the phenomenon that parameter-dependent noise — induced by minibatches or label perturbation — is far more effective than Gaussian noise . This paper theoretically characterizes this phenomenon on a quadratically-parameterized model introduced by Vaskevicius et al . and Woodworth et al . We show that in an over-parameterized setting , SGD with label noise recovers the sparse groundtruth with an arbitrary initialization , whereas SGD with Gaussian noise or gradient descent overfits to dense solutions with large norms . Our analysis reveals that parameter-dependent noise introduces a bias towards local minima with smaller noise variance , whereas spherical Gaussian noise does not . 1 INTRODUCTION . One central mystery of deep artificial neural networks is their capability to generalize when having far more learnable parameters than training examples Zhang et al . ( 2016 ) . To add to the mystery , deep nets can also obtain reasonable performance in the absence of any explicit regularization . This has motivated recent work to study the regularization effect due to the optimization ( rather than objective function ) , also known as implicit bias or implicit regularization Gunasekar et al . ( 2017 ; 2018a ; b ) ; Soudry et al . ( 2018 ) ; Arora et al . ( 2019 ) . The implicit bias is induced by and depends on many factors , such as learning rate and batch size Smith et al . ( 2017 ) ; Goyal et al . ( 2017 ) ; Keskar et al . ( 2016 ) ; Li et al . ( 2019b ) ; Hoffer et al . ( 2017 ) , initialization and momentum Sutskever et al . ( 2013 ) , adaptive stepsize Kingma and Ba ( 2014 ) ; Neyshabur et al . ( 2015 ) ; Wilson et al . ( 2017 ) , batch normalization Ioffe and Szegedy ( 2015 ) and dropout Srivastava et al . ( 2014 ) . Among these sources of implicit regularization , the SGD noise is believed to be a vital one ( LeCun et al. , 2012 ; Keskar et al. , 2016 ) . Previous theoretical works ( e.g. , Li et al . ( 2019b ) ) have studied the implicit regularization effect from the scale of the noise , which is directly influenced by learning rate and batch size . However , people have empirically observed that the shape of the noise also has a strong ( if not stronger ) implicit bias . For example , prior works show that mini-batch noise or label noise ( label smoothing ) – noise in the parameter updates from the perturbation of labels in training – is far more effective than adding spherical Gaussian noise ( e.g. , see ( Shallue et al. , 2018 , Section 4.6 ) and Szegedy et al . ( 2016 ) ; Wen et al . ( 2019 ) ) . We also confirm this phenomenon in Figure 1 ( left ) . Thus , understanding the implicit bias of the noise shape is crucial . Such an understanding may also apply to distributed training because synthetically adding noise may help generalization if parallelism reduces the amount of mini-batch noise ( Shallue et al. , 2018 ) . In this paper , we theoretically study the effect of the shape of the noise , demonstrating that it can provably determine generalization performance at convergence . Our analysis is based on a nonlinear quadratically-parameterized model introduced by ( Woodworth et al. , 2020 ; Vaskevicius et al. , 2019 ) , which is rich enough to exhibit similar empirical phenomena as deep networks . Indeed , Figure 1 ( right ) empirically shows that SGD with mini-batch noise or label noise can generalize with arbitrary initialization without explicit regularization , whereas GD or SGD with spherical Gaussian noise can not . We aim to analyze the implicit bias of label noise and Gaussian noise in the quadraticallyparametrized model and explain these empirical observations . We choose to study label noise because it can replicate the regularization effects of minibatch noise in both real and synthetic data ( Figure 1 ) , and has been used to regularize large-batch parallel training ( Shallue et al. , 2018 ) . Moreover , label noise is less sensitive to the initialization and the optimization history than mini-batch noise , which makes it more amenable to theoretical analysis . For example , in an extreme case , if we happen to reach or initialize at a solution that overfits the data exactly , then mini-batch SGD will stay there forever because both the gradient and the noise vanish ( Vaswani et al. , 2019 ) . In contrast , label noise will not accidentally vanish , so the analysis is more tractable . Understanding label noise may lead to understanding mini-batch noise or replacing it with other more robust choices . In our setting , we prove that with a proper learning rate schedule , SGD with label noise recovers a sparse ground-truth classifier and generalizes well , whereas SGD with spherical Gaussian noise generalizes poorly . Concretely , SGD with label noise biases the parameter towards the low sparsity regime and exactly recovers the sparse ground-truth , even when the initialization is arbitrarily large ( Theorem 2.1 ) . In this same regime , noise-free gradient descent quickly overfits because it trains in the NTK regime ( Jacot et al. , 2018 ; Chizat and Bach , 2018 ) . Adding Gaussian noise is insufficient to fix this , as this algorithm would end up sampling from a Gibbs distribution with infinite partition function and fail to converge to the ground-truth ( Theorem 2.2 ) . In summary , with not too small learning rate or noise level , label noise suffices to bias the parameter towards sparse solutions without relying on a small initialization , whereas Gaussian noise can not . Our analysis suggests that the fundamental difference between label or mini-batch noise and Gaussian noise is that the former is parameter-dependent , and therefore introduces stronger biases than the latter . The conceptual message highlighted by our analysis is that there are two possible implicit biases induced by the noise : 1. prior work ( Keskar et al. , 2016 ) shows that by escaping sharp local minima , noisy gradient descent biases the parameter towards more robust solutions ( i.e , solutions with low curvature , or “ flat ” minima ) , and 2. when the noise covariance varies across the parameter space , there is another ( potentially stronger ) implicit bias effect toward parameters where the noise covariance is smaller . Label or mini-batch noise benefits from both biases , whereas Gaussian noise is independent of the parameter , so it benefits from the first bias but not the second . For the quadratically-parameterized model , this first bias is not sufficient for finding solutions with good generalization because there is a large set of overfitting global minima of the training loss with reasonable curvature . In contrast , the covariance of label noise is proportional to the scale of the parameter , inducing a much stronger bias towards low norm solutions which generalize well . 1.1 ADDITIONAL RELATED WORKS . Closely related to our work , Blanc et al . ( 2019 ) and Zhu et al . ( 2019 ) also theoretically studied implicit regularization effects that arise due to shape , rather than scale , of the noise . However , they only considered the local effect of the noise near some local minimum of the loss . In contrast , our work analyzes the global effect of noise . For a more detailed comparison with ( Blanc et al. , 2019 ) , see Section 2.2 . Woodworth et al . ( 2020 ) ; Vaskevicius et al . ( 2019 ) analyze the effect of initialization for the same model that we study , showing that large initialization trains in the NTK regime ( shown to generalize poorly ( Wei et al. , 2019 ; Ghorbani et al. , 2019 ) ) whereas small initialization does not . We show that when the initialization is large , adding noise helps avoid the NTK regime ( Li and Liang , 2018 ; Jacot et al. , 2018 ; Du et al. , 2018b ; Woodworth et al. , 2020 ) without explicit regularization . Several previous works have studied generalization bounds and training dynamics of SGD with statedependent noises for more general models . Hardt et al . ( 2015 ) derived stability-based generalization bounds for mini-batch SGD based on training speed . Cheng et al . ( 2019 ) proved that SGD with statedependent noises has iterate distribution close to the corresponding continuous stochastic differential equation with the same noise covariance . Meng et al . ( 2020 ) ; Xie et al . ( 2020 ) showed that SGD with state-dependent noises escapes local minimum faster than SGD with spherical Gaussian noise . There has been a line of work empirically studying how noise influences generalization . Keskar et al . ( 2016 ) argued that large batch training will converge to “ sharp ” local minima which do not generalize well . Hoffer et al . ( 2017 ) argued that large batch size doesn ’ t hurt generalization much if training goes on long enough and additional noise is added with a larger learning rate . Goyal et al . ( 2017 ) and Shallue et al . ( 2018 ) showed large batch training with proper learning rate and additional label noise can achieve similar generalization as small batch . Wei and Schwab ( 2019 ) ; Chaudhari and Soatto ( 2018 ) ; Yaida ( 2018 ) ( heuristically ) suggested that SGD may encourage solutions with smaller noise covariance . Martin and Mahoney ( 2018 ) used random matrix theory to analyze implicit regularization effects of noises . The noise induced by dropout has been shown to change the expected training objective , hence provides a regularization effect ( Mianjy et al. , 2018 ; Mianjy and Arora , 2019 ; Wei et al. , 2020 ; Arora et al. , 2020 ) . Wei et al . ( 2020 ) showed that there also exists an implicit bias induced by dropout noise . Langevin dynamics or the closely-related stochastic gradient descent with spherical Gaussian noise has been studied in previous works Welling and Teh ( 2011 ) ; Teh et al . ( 2016 ) ; Raginsky et al . ( 2017 ) ; Zhang et al . ( 2017 ) ; Mou et al . ( 2017 ) ; Roberts et al . ( 1996 ) ; Ge et al . ( 2015 ) ; Negrea et al . ( 2019 ) ; Neelakantan et al . ( 2015 ) ; Mou et al . ( 2018 ) . In particular , Raginsky et al . ( 2017 ) and Li et al . ( 2019a ) provided generalization bounds for SGLD using algorithmic stability . Several works have theoretically analyzed other types of implicit biases in simplified settings ( Soudry et al. , 2018 ; Gunasekar et al. , 2018b ; Ji and Telgarsky , 2018a ) . Gunasekar et al . ( 2017 ) and Li et al . ( 2017 ) showed that gradient descent finds low rank solutions in matrix completion . Gradient descent has also been shown to maximize the margin in linear and homogeneous models ( Soudry et al. , 2018 ; Ji and Telgarsky , 2018b ; Nacson et al. , 2018 ; Lyu and Li , 2019 ; Gunasekar et al. , 2018a ; Nacson et al. , 2019 ; Poggio et al. , 2017 ) . Du et al . ( 2018a ) showed that gradient descent implicitly balances the layers of deep homogeneous models . Other works showed that it may not always be possible to characterize implicit biases in terms of norm ( Arora et al. , 2019 ; Razin and Cohen , 2020 ) . Gissin et al . ( 2019 ) showed that gradient descent dynamics exhibit different implicit biases based on depth . Li et al . ( 2019b ) studied the implicit regularization effect of a large initial learning rate . Guo et al . ( 2018 ) studies the notion of “ elimination singularities ” in RBF networks , where optimization runs into a regime with small weight and small gradient , therefore the training can be slowed down . Our paper also involves training trajectory with small weight norm , but instead focuses on its influnece on the generalization performance .

Summary: This paper considers the effect of label noise on stochastic gradient descent. The setup is that there is a vector $v \in R^d$. We observe samples from $v^2\cdot x$. We only have $n < d$ samples but $v$ is $r$-sparse for $r < n < d$ which makes recovery possible information theoretically. The main result is that stochastic gradient descent with label noise, and without any explicit regularization will recover the ground truth. whereas adding spherical Gaussian noise does not.