SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs

p n ) , where n is the number of samples . In this paper , we consider the problem of SCO and explore the role of implicit regularization , batch size and multiple epochs for SGD . Our main contributions are threefold : 1 . We show that for any regularizer , there is an SCO problem for which Regularized Empirical Risk Minimzation fails to learn . This automatically rules out any implicit regularization based explanation for the success of SGD . 2 . We provide a separation between SGD and learning via Gradient Descent on empirical loss ( GD ) in terms of sample complexity . We show that there is an SCO problem such that GD with any step size and number of iterations can only learn at a suboptimal rate : at least e⌦ ( 1/n5/12 ) . 3 . We present a multi-epoch variant of SGD commonly used in practice . We prove that this algorithm is at least as good as single pass SGD in the worst case . However , for certain SCO problems , taking multiple passes over the dataset can significantly outperform single pass SGD . We extend our results to the general learning setting by showing a problem which is learnable for any data distribution , and for this problem , SGD is strictly better than RERM for any regularization function . We conclude by discussing the implications of our results for deep learning , and show a separation between SGD and ERM for two layer diagonal neural networks . 1 Introduction . We consider the problem of stochastic optimization of the form : Minimize F ( w ) ( 1 ) where the objective F : Rd 7 ! R is given by F ( w ) = Ez⇠D [ f ( w ; z ) ] . The goal is to perform the minimization based only on samples S = { z1 , . . . , zn } drawn i.i.d . from some distribution D. Standard statistical learning problems can be cast as stochastic optimization problems , with F ( w ) being the population loss and f ( w , z ) being the instantaneous loss on sample z for the model w. Stochastic Convex Optimization ( SCO ) . Perhaps the most well studied stochastic optimization problem is that of SCO . We define a problem to be an instance of a SCO problem if , Assumption I : Population loss F is convex . ( 2 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Notice above that we only require the population loss to be convex and do not impose such a condition on the instantaneous loss functions f . Algorithms like Stochastic Gradient Descent ( SGD ) , Gradient Descent on training loss ( GD ) , and methods like Regularized Empirical Risk Minimization ( RERM ) that minimize training loss with additional penalty in the form of a regularizer are all popular choices of algorithms used to solve the above problem and have been analyzed theoretically for various settings of Stochastic Optimization problems ( convex and non-convex ) . We discuss below a mix of recent empirical and theoretical insights about SGD algorithms that provide motivation for this work . SGD and Implicit Regularization . A popular theory for why SGD generalizes so well when used on large over-parameterized models has been that of implicit regularization . It has been oberved that in large models , often there are multiple global minima for the empirical loss . However not all of these empirical minima have low suboptimality at the population level . SGD when used as the training algorithm often seems to find empirical ( near ) global minima that also generalize well and have low test loss . Hence while a general Empirical Risk Minimization ( ERM ) algorithm might fail , the implicit bias of SGD seems to yield a well-generalizing ERM . The idea behind implicit regularization is that the solution of SGD is equivalent to the solution of a Regularized Empirical Risk Minimizer ( RERM ) for an appropriate implicit regularizer . The idea of implicit regularization of SGD has been extensively studied in recent years . In Gunasekar et al . [ 2018a ] , Zou et al . [ 2021 ] , the classical setting of linear regression ( with square loss ) is considered and it was shown that when considering over-parameterized setting , the SGD algorithm is equivalent to fitting with a linear predictor with the smallest euclidean norm . In Soudry et al . [ 2018 ] , Ji and Telgarsky [ 2018 ] linear predictors with logistic loss are considered and it was noted that SGD for this setting can be seen as having an implicit regularization of ` 2 norm . Gunasekar et al . [ 2018b ] considered multi-layer convolutional networks with linear activation are considered and showned that SGD for this model can be seen as having an implicit regularization of ` 2/L norm ( bridge penality for depth L networks ) of the Fourier frequencies corresponding to the linear predictor . Gunasekar et al . [ 2018c ] considered matrix factorization and showed that running SGD is equivalent to having a nuclear norm based regularizer . More recent work of Arora et al . [ 2019 ] , Razin and Cohen [ 2020 ] shows that in particular in the deep matrix factorization setting , SGD can not be seen as having any norm based implicit regularizer but rather a rank based one . However , in all these cases , the behavior of SGD corresponds to regularizers that are independent of the training data ( e.g . rank , lp-norm , etc ) . One could surmise that a grand program for this line of research is that for problems where SGD works well , perhaps there is a corresponding implicit regularization explanation . That is , there exists a regularizer R such that SGD can been seen as performing exact or approximate RERM with respect to this regularizer . In fact , one can ask this question specific to SCO problems . That is , for the problem of SCO , is there an implicit regularizer R such that SGD can be seen as performing approximate RERM ? In fact , a more basic question one can ask : is it true that SCO problem is always learnable using some regularized ERM algorithms ? We answer both these questions in the negative . SGD vs GD : Smaller the Batch-size Better the Generalization . It has been observed that in practice , while SGD with small batch size , and performing gradient descent ( GD ) with empirical loss as the objective function both minimize the training error equally well , the SGD solution generalizes much better than the full gradient descent one [ Keskar et al. , 2016 , Kleinberg et al. , 2018 ] . However , thus far , most existing literature on theorizing why SGD works well for over-parameterized deep learning models also work for gradient descent on training loss [ Allen-Zhu and Li , 2019 , Allen-Zhu et al. , 2018a , b , Arora et al. , 2018 ] . In this work , we construct a convex learning problem where single pass SGD provably outperforms GD run on empirical loss , which converges to a solution that has large excess risk . We thus provide a problem instance where SGD works but GD has strictly inferior sample complexity ( with or without early stopping ) . Multiple Epochs Help . The final empirical observation we consider is the fact that multiple epochs of SGD tends to further continually decrease not only the training error but also the test error [ Zhang et al. , 2016 , Ma et al. , 2018 , Bottou and Bousquet , 2011 ] . In this paper , we construct an SCO instance for which multiple epochs of single sample mini-batched SGD significantly outperforms single pass SGD , and for the same problem , RERM fails to converge to a good solution , and hence , so does GD when run to convergence . 1.1 Our Contributions . We now summarize our main contributions in the paper : 1 . SGD and RERM Separation . In Section 3 , we demonstrate a SCO problem where a single pass of SGD over n data points obtains a 1/ p n suboptimality rate . However , for any regularizer R , regularized ERM does not attain a diminishing suboptimality . We show that this is true even if the regularization parameter is chosen in a sample dependent fashion . Our result immediately rules out the explanation that SGD is successful in SCO due to some some implicit regularization . 2 . SGD and GD Separation . In Section 4 , we provide a separation between SGD and GD on training loss in terms of sample complexity for SCO . To the best of our knowledge , this is the first1 such separation result between SGD and GD in terms of sample complexity . In this work , we show the existence of SCO problems where SGD with n samples achieves a suboptimality of 1/ p n , however , irrespective of what step-size is used and how many iterations we run for , GD can not obtain a suboptimality better than 1/ ( n5/12 log2 ( n ) ) . 3 . Single-pass vs Multi-pass SGD . In Section 5 , we provide an adaptive multi-epoch SGD algorithm that is provably at least as good as single pass SGD algorithm . On the other hand , we also show that this algorithm can far outperform single pass SGD on certain problems where SGD only attains a rate of 1/ p n. Also , on these problems RERM fails to learn , indicating that GD run to convergence fails as well . 4 . SGD and RERM Separation in the Distribution Free Agnostic PAC Setting . The separation result between SGD and RERM introduced earlier was for SCO . However , it turns out that the problem is not agnostically learnable for all distributions but only for distributions that make F convex . In Section 6 , we provide a learning problem that is distribution-free learnable , and where SGD provably outperforms any RERM . 5 . Beyond Convexity ( Deep Learning ) . The convergence guarantee for SGD can be easily extended to stochastic optimization settings where the population loss F ( w ) is linearizable , but may not be convex . We formalize this in Section 7 , and show that for two layer diagonal neural networks with ReLU activations , there exists a distribution for which the population loss is linearizable and thus SGD works , but ERM algorithm fails to find a good solution . This hints at the possibility that the above listed separations between SGD and GD / RERM for the SCO setting , also extend to the deep learning setting . 1.2 Preliminaries . A standard assumption made by most gradient based algorithms for stochastic optimization problems is the following : Assumption II : 8 < : F is L-Lipschitz ( w.r.t . ` 2 norm ) 9w ? 2 argminw F ( w ) s.t . kw1 w ? k B supw Ez⇠D krf ( w , z ) rF ( w ) k2 2 . ( 3 ) In the above and throughout this work , the norm denotes the standard Euclidean norm and w1 is some initial point known to the algorithm . Next , we describe below more formally what the regularized ERM , GD and SGD algorithms are . ( Regularized ) Empirical Risk Minimization . Perhaps the simplest algorithm one could consider is the Empirical Risk Minimization ( ERM ) algorithm where one returns a minimizer of training loss ( empirical risk ) FS ( w ) : = 1 n nX t=1 f ( w ; zt ) . That is , wERM 2 argminw2W FS ( w ) . A more common variant of this method is one where we additionally penalize complex models using a regularizer function R : Rd 7 ! R. That is , a Regularized Empirical Risk Minimization ( RERM ) method consists of returning : wRERM = argmin w2W FS ( w ) +R ( w ) . ( 4 ) 1Despite the way the result is phrased in Amir et al . [ 2021 ] , their result does not imply separation between SGD and GD in terms of sample complexity but only number of iterations . Gradient Descent ( GD ) on Training Loss . Gradient descent on training loss is the algorithm that performs the following update on every iteration : wGDi+1 wGDi ⌘rFS ( wGDi ) , ( 5 ) where ⌘ denotes the step size . After t rounds , we return the point bwGDt : = 1t Pt i=1 w GD i . Stochastic Gradient Descent ( SGD ) . Stochastic gradient descent ( SGD ) has been the method of choice for training large over-parameterized deep learning models , and other convex and non-convex learning problems . Single pass SGD algorithm runs for n steps and for each step takes a single data point and performs gradient update with respect to that sample . That , is on round t , wSGDi+1 wSGDi ⌘rf ( wSGDi ; zi ) ( 6 ) Finally , we return bwSGDn = 1n Pn i=1 w SGD i . Multi-epoch ( also known as multi-pass ) SGD algorithm simply cycles over the dataset multiple times continuing to perform the same update specified above . It is well know that single pass SGD algorithm enjoys the following convergence guarantee : Theorem 1 ( Nemirovski and Yudin [ 1983 ] ) . On any SCO problem satisfying Assumption I in ( 2 ) and Assumption II in ( 3 ) , running SGD algorithm for n steps with the step size of ⌘ = 1/ p n enjoys the guarantee ES [ F ( bwSGDn ) ] inf w2Rd F ( w ) O ⇣ 1p n ⌘ , where the constant in the order notation only depends on constants B , L and in ( 3 ) and is independent of the dimension d. In fact , even weaker assumptions like one-point convexity or star convexity of F w.r.t . an optimum on the path of the SGD suffices to obtain the above guarantee . We use this guarantee of SGD algorithm throughout this work . Up until Section 6 we only consider SCO problems for which SGD automatically works with the above guarantee .

Summary: This paper studies the generalization error bound of multi-epoch, small-batch, SGD for learning over-parameterized models. In particular, the authors compare the generalization performance of SGD to GD and regularized empirical risk minimization (RERM). For RERM, the authors show that there is an SCO problem for which RERM will not give vanishing generalization error while SGD enjoys a $O(1/\sqrt{n})$ rate. For GD, the authors also show that for some SCO problems GD with any step size and number of iteration can only achieve a $\Omega(1/n^{5/12})$ rate, which is also worse than SGD. Lastly, the authors show that multi-epoch SGD can perform better than single-pass SGD.