SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs

p n ) , where n is the number of samples . In this paper , we consider the problem of SCO and explore the role of implicit regularization , batch size and multiple epochs for SGD . Our main contributions are threefold : 1 . We show that for any regularizer , there is an SCO problem for which Regularized Empirical Risk Minimzation fails to learn . This automatically rules out any implicit regularization based explanation for the success of SGD . 2 . We provide a separation between SGD and learning via Gradient Descent on empirical loss ( GD ) in terms of sample complexity . We show that there is an SCO problem such that GD with any step size and number of iterations can only learn at a suboptimal rate : at least e⌦ ( 1/n5/12 ) . 3 . We present a multi-epoch variant of SGD commonly used in practice . We prove that this algorithm is at least as good as single pass SGD in the worst case . However , for certain SCO problems , taking multiple passes over the dataset can significantly outperform single pass SGD . We extend our results to the general learning setting by showing a problem which is learnable for any data distribution , and for this problem , SGD is strictly better than RERM for any regularization function . We conclude by discussing the implications of our results for deep learning , and show a separation between SGD and ERM for two layer diagonal neural networks . 1 Introduction . We consider the problem of stochastic optimization of the form : Minimize F ( w ) ( 1 ) where the objective F : Rd 7 ! R is given by F ( w ) = Ez⇠D [ f ( w ; z ) ] . The goal is to perform the minimization based only on samples S = { z1 , . . . , zn } drawn i.i.d . from some distribution D. Standard statistical learning problems can be cast as stochastic optimization problems , with F ( w ) being the population loss and f ( w , z ) being the instantaneous loss on sample z for the model w. Stochastic Convex Optimization ( SCO ) . Perhaps the most well studied stochastic optimization problem is that of SCO . We define a problem to be an instance of a SCO problem if , Assumption I : Population loss F is convex . ( 2 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Notice above that we only require the population loss to be convex and do not impose such a condition on the instantaneous loss functions f . Algorithms like Stochastic Gradient Descent ( SGD ) , Gradient Descent on training loss ( GD ) , and methods like Regularized Empirical Risk Minimization ( RERM ) that minimize training loss with additional penalty in the form of a regularizer are all popular choices of algorithms used to solve the above problem and have been analyzed theoretically for various settings of Stochastic Optimization problems ( convex and non-convex ) . We discuss below a mix of recent empirical and theoretical insights about SGD algorithms that provide motivation for this work . SGD and Implicit Regularization . A popular theory for why SGD generalizes so well when used on large over-parameterized models has been that of implicit regularization . It has been oberved that in large models , often there are multiple global minima for the empirical loss . However not all of these empirical minima have low suboptimality at the population level . SGD when used as the training algorithm often seems to find empirical ( near ) global minima that also generalize well and have low test loss . Hence while a general Empirical Risk Minimization ( ERM ) algorithm might fail , the implicit bias of SGD seems to yield a well-generalizing ERM . The idea behind implicit regularization is that the solution of SGD is equivalent to the solution of a Regularized Empirical Risk Minimizer ( RERM ) for an appropriate implicit regularizer . The idea of implicit regularization of SGD has been extensively studied in recent years . In Gunasekar et al . [ 2018a ] , Zou et al . [ 2021 ] , the classical setting of linear regression ( with square loss ) is considered and it was shown that when considering over-parameterized setting , the SGD algorithm is equivalent to fitting with a linear predictor with the smallest euclidean norm . In Soudry et al . [ 2018 ] , Ji and Telgarsky [ 2018 ] linear predictors with logistic loss are considered and it was noted that SGD for this setting can be seen as having an implicit regularization of ` 2 norm . Gunasekar et al . [ 2018b ] considered multi-layer convolutional networks with linear activation are considered and showned that SGD for this model can be seen as having an implicit regularization of ` 2/L norm ( bridge penality for depth L networks ) of the Fourier frequencies corresponding to the linear predictor . Gunasekar et al . [ 2018c ] considered matrix factorization and showed that running SGD is equivalent to having a nuclear norm based regularizer . More recent work of Arora et al . [ 2019 ] , Razin and Cohen [ 2020 ] shows that in particular in the deep matrix factorization setting , SGD can not be seen as having any norm based implicit regularizer but rather a rank based one . However , in all these cases , the behavior of SGD corresponds to regularizers that are independent of the training data ( e.g . rank , lp-norm , etc ) . One could surmise that a grand program for this line of research is that for problems where SGD works well , perhaps there is a corresponding implicit regularization explanation . That is , there exists a regularizer R such that SGD can been seen as performing exact or approximate RERM with respect to this regularizer . In fact , one can ask this question specific to SCO problems . That is , for the problem of SCO , is there an implicit regularizer R such that SGD can be seen as performing approximate RERM ? In fact , a more basic question one can ask : is it true that SCO problem is always learnable using some regularized ERM algorithms ? We answer both these questions in the negative . SGD vs GD : Smaller the Batch-size Better the Generalization . It has been observed that in practice , while SGD with small batch size , and performing gradient descent ( GD ) with empirical loss as the objective function both minimize the training error equally well , the SGD solution generalizes much better than the full gradient descent one [ Keskar et al. , 2016 , Kleinberg et al. , 2018 ] . However , thus far , most existing literature on theorizing why SGD works well for over-parameterized deep learning models also work for gradient descent on training loss [ Allen-Zhu and Li , 2019 , Allen-Zhu et al. , 2018a , b , Arora et al. , 2018 ] . In this work , we construct a convex learning problem where single pass SGD provably outperforms GD run on empirical loss , which converges to a solution that has large excess risk . We thus provide a problem instance where SGD works but GD has strictly inferior sample complexity ( with or without early stopping ) . Multiple Epochs Help . The final empirical observation we consider is the fact that multiple epochs of SGD tends to further continually decrease not only the training error but also the test error [ Zhang et al. , 2016 , Ma et al. , 2018 , Bottou and Bousquet , 2011 ] . In this paper , we construct an SCO instance for which multiple epochs of single sample mini-batched SGD significantly outperforms single pass SGD , and for the same problem , RERM fails to converge to a good solution , and hence , so does GD when run to convergence . 1.1 Our Contributions . We now summarize our main contributions in the paper : 1 . SGD and RERM Separation . In Section 3 , we demonstrate a SCO problem where a single pass of SGD over n data points obtains a 1/ p n suboptimality rate . However , for any regularizer R , regularized ERM does not attain a diminishing suboptimality . We show that this is true even if the regularization parameter is chosen in a sample dependent fashion . Our result immediately rules out the explanation that SGD is successful in SCO due to some some implicit regularization . 2 . SGD and GD Separation . In Section 4 , we provide a separation between SGD and GD on training loss in terms of sample complexity for SCO . To the best of our knowledge , this is the first1 such separation result between SGD and GD in terms of sample complexity . In this work , we show the existence of SCO problems where SGD with n samples achieves a suboptimality of 1/ p n , however , irrespective of what step-size is used and how many iterations we run for , GD can not obtain a suboptimality better than 1/ ( n5/12 log2 ( n ) ) . 3 . Single-pass vs Multi-pass SGD . In Section 5 , we provide an adaptive multi-epoch SGD algorithm that is provably at least as good as single pass SGD algorithm . On the other hand , we also show that this algorithm can far outperform single pass SGD on certain problems where SGD only attains a rate of 1/ p n. Also , on these problems RERM fails to learn , indicating that GD run to convergence fails as well . 4 . SGD and RERM Separation in the Distribution Free Agnostic PAC Setting . The separation result between SGD and RERM introduced earlier was for SCO . However , it turns out that the problem is not agnostically learnable for all distributions but only for distributions that make F convex . In Section 6 , we provide a learning problem that is distribution-free learnable , and where SGD provably outperforms any RERM . 5 . Beyond Convexity ( Deep Learning ) . The convergence guarantee for SGD can be easily extended to stochastic optimization settings where the population loss F ( w ) is linearizable , but may not be convex . We formalize this in Section 7 , and show that for two layer diagonal neural networks with ReLU activations , there exists a distribution for which the population loss is linearizable and thus SGD works , but ERM algorithm fails to find a good solution . This hints at the possibility that the above listed separations between SGD and GD / RERM for the SCO setting , also extend to the deep learning setting . 1.2 Preliminaries . A standard assumption made by most gradient based algorithms for stochastic optimization problems is the following : Assumption II : 8 < : F is L-Lipschitz ( w.r.t . ` 2 norm ) 9w ? 2 argminw F ( w ) s.t . kw1 w ? k B supw Ez⇠D krf ( w , z ) rF ( w ) k2 2 . ( 3 ) In the above and throughout this work , the norm denotes the standard Euclidean norm and w1 is some initial point known to the algorithm . Next , we describe below more formally what the regularized ERM , GD and SGD algorithms are . ( Regularized ) Empirical Risk Minimization . Perhaps the simplest algorithm one could consider is the Empirical Risk Minimization ( ERM ) algorithm where one returns a minimizer of training loss ( empirical risk ) FS ( w ) : = 1 n nX t=1 f ( w ; zt ) . That is , wERM 2 argminw2W FS ( w ) . A more common variant of this method is one where we additionally penalize complex models using a regularizer function R : Rd 7 ! R. That is , a Regularized Empirical Risk Minimization ( RERM ) method consists of returning : wRERM = argmin w2W FS ( w ) +R ( w ) . ( 4 ) 1Despite the way the result is phrased in Amir et al . [ 2021 ] , their result does not imply separation between SGD and GD in terms of sample complexity but only number of iterations . Gradient Descent ( GD ) on Training Loss . Gradient descent on training loss is the algorithm that performs the following update on every iteration : wGDi+1 wGDi ⌘rFS ( wGDi ) , ( 5 ) where ⌘ denotes the step size . After t rounds , we return the point bwGDt : = 1t Pt i=1 w GD i . Stochastic Gradient Descent ( SGD ) . Stochastic gradient descent ( SGD ) has been the method of choice for training large over-parameterized deep learning models , and other convex and non-convex learning problems . Single pass SGD algorithm runs for n steps and for each step takes a single data point and performs gradient update with respect to that sample . That , is on round t , wSGDi+1 wSGDi ⌘rf ( wSGDi ; zi ) ( 6 ) Finally , we return bwSGDn = 1n Pn i=1 w SGD i . Multi-epoch ( also known as multi-pass ) SGD algorithm simply cycles over the dataset multiple times continuing to perform the same update specified above . It is well know that single pass SGD algorithm enjoys the following convergence guarantee : Theorem 1 ( Nemirovski and Yudin [ 1983 ] ) . On any SCO problem satisfying Assumption I in ( 2 ) and Assumption II in ( 3 ) , running SGD algorithm for n steps with the step size of ⌘ = 1/ p n enjoys the guarantee ES [ F ( bwSGDn ) ] inf w2Rd F ( w ) O ⇣ 1p n ⌘ , where the constant in the order notation only depends on constants B , L and in ( 3 ) and is independent of the dimension d. In fact , even weaker assumptions like one-point convexity or star convexity of F w.r.t . an optimum on the path of the SGD suffices to obtain the above guarantee . We use this guarantee of SGD algorithm throughout this work . Up until Section 6 we only consider SCO problems for which SGD automatically works with the above guarantee .

Summary: This paper considers the sample complexity comparison between one-pass SGD vs. ERM/RERM/GD/multi-pass SGD in the setting of SCO (with Lipschitz, finite $\ell_2$-norm solution, bounded gradient variance). It shows that, for any problem instances in the considered SCO class, one-pass SGD always achieves good, $1/\sqrt{n}$ rate; in contrast, ERM, RERM (that is distribution-agnostic but is allowed to tune a multiplicative hyperparameter), GD (with any number of passes) could be arbitrarily bad in the worse-case instance in the considered problem class. Moreover, when used with a hold-out validation set, multi-pass SGD can outperform (at least no worse than) one-pass SGD. In conclusion, the paper reveals an important message, that (one-pass) SGD does have its own implicit bias, but it cannot be explained by a simple explicit regularizer.