The Benefits of Implicit Regularization from SGD in Least Squares Problems

1 Introduction . Deep neural networks often exhibit powerful generalization in numerous machine learning applications , despite being overparameterized . It has been conjectured that the optimization algorithm itself , e.g. , stochastic gradient descent ( SGD ) , implicitly regularizes such overparameterized models [ 29 ] ; here , ( unregularized ) overparameterized models could admit numerous global and local minima ( many of which generalize poorly [ 29 , 21 ] ) , yet SGD tends to find solutions that generalize well , even in the absence of explicit regularizers [ 22 , 29 , 19 ] . This regularizing effect due to the choice of the optimization algorithm is often referred to as implicit regularization [ 22 ] . ∗Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Before moving to the non-convex regime , we may hope to start by understanding this effect in the ( overparameterized ) convex regime . At least for linear models , there is a growing body of evidence suggesting that the implicit regularization of SGD is closely related to an explicit , ` 2-type of ( ridge ) regularization [ 25 ] . For example , ( multi-pass ) SGD for linear regression converges to the minimum-norm interpolator , which corresponds to the limit of the ridge solution with a vanishing penalty [ 29 , 14 ] . Tangential evidence for this also comes from examining gradient descent , where a continuous time ( gradient flow ) analysis shows how the optimization path of gradient descent is ( pointwise ) closely connected to an explicit , ` 2-regularization [ 24 , 1 ] . Similar results [ 2 ] have been further extended to SGD , where a ( early-stopped ) continuous-time SGD is demonstrated to perform similarly to ridge regression with certain regularization parameters . However , as of yet , a precise comparison between the implicit regularization afforded by SGD and the explicit regularization of ridge regression ( in terms of the generalization performance ) is still lacking , especially when the hyperparameters ( e.g. , stepsize for SGD and regularization parameter for ridge regression ) are allowed to be tuned . This motivates the central question in this work : How does the generalization performance of SGD compare with that of ridge regression in least square problems ? In particular , even in the arguably simplest setting of linear regression , we seek to understand if/how SGD behaves differently from using an explicit ` 2-regularizer , with a particular focus on the overparameterized regime . Our Contributions . Due to recent advances on sharp , instance-dependent excess risks bounds of both ( single-pass ) SGD and ridge regression for overparameterized least square problems [ 26 , 30 ] , a nearly complete answer to the above question is now possible using these tools . In this work , we deliver an instance-based risk comparison between SGD and ridge regression in several interesting settings , including one-hot distributed data and Gaussian data . In particular , for a broad class of least squares problem instances that are natural in high-dimensional settings , we show that • For every problem instance and for every ridge parameter , ( unregularized ) SGD , when provided with logarithmically more samples than that provided to ridge regularization , generalizes no worse than the ridge solution , provided SGD uses a tuned constant stepsize . • Conversely , there exist instances in our problem class where optimally-tuned ridge regression requires quadratically more samples than SGD to achieve the same generalization performance . Quite strikingly , the above results show that , up to some logarithmic factors , the generalization performance of SGD is always no worse than that of ridge regression in a wide range of overparameterized least square problems , and , in fact , could be much better for some problem instances . As a special case ( for the above two claims ) , our problem class includes a setting in which : ( i ) the signal-to-noise is bounded and ( ii ) the eigenspectrum decays at a polynomial rate 1/iα , for 0 ≤ α ≤ 1 ( which permits a relatively fast decay ) . This one-sided near-domination phenomenon ( in these natural overparameterized problem classes ) could further support the preference for the implicit regularization brought by SGD over explicit ridge regularization . Several novel technical contributions are made to make the above risk comparisons possible . For the one-hot data , we derive similar risk upper bound of SGD and risk lower bound of ridge regression . For the Gaussian data , while a sharp risk bound of SGD is borrowed from [ 30 ] , we prove a sharp lower bound of ridge regression by adapting the proof techniques developed in [ 26 , 7 ] . By carefully comparing these upper and lower bound results ( and exhibiting particular instances to show that our sample size inflation bounds are sharp ) , we are able to provide nearly complete conditions that characterize when SGD generalizes better than ridge regression . Notation . For two functions f ( x ) ≥ 0 and g ( x ) ≥ 0 defined on x > 0 , we write f ( x ) . g ( x ) if f ( x ) ≤ c · g ( x ) for some absolute constant c > 0 ; we write f ( x ) & g ( x ) if g ( x ) . f ( x ) ; we write f ( x ) h g ( x ) if f ( x ) . g ( x ) . f ( x ) . For a vector w ∈ Rd and a positive semidefinite matrix H ∈ Rd×d , we denote ‖w‖H : = √ w > Hw . 2 Related Work . In terms of making sharp risk comparisons with ridge , the work of [ 10 ] shows that OLS ( after a PCA projection is applied to the data ) is instance-wise competitive with ridge on fixed design problems . The insights in our analysis are draw from this work , though there are a number of technical challenges in dealing with the random design setting . We start with a brief discussion of the technical advances in the analysis of ridge regression and SGD , and then briefly overview more related work comparing SGD to explicit norm-based regularization . Excess Risk Bounds for Ridge Regression . In the underparameterized regime , the excess risk bounds for ridge regression has been well-understood [ 16 ] . In the overparameterized regime , a large body of works [ 12 , 15 , 28 , 27 ] focused on characterizing the excess risk of ridge regression in the asymptotic regime where both the sample size N and dimension d go to infinite and d/N → γ for some finite γ . More recently , Bartlett et al . [ 7 ] developed sharp non-asymptotic risk bounds for ordinary least square in the overparameterized setting , which are further extended to ridge regression by Tsigler and Bartlett [ 26 ] . These bounds have additional interest because they are instance-dependent , in particular , depending on the data covariance spectrum . The risk bounds of ridge regression derived in Tsigler and Bartlett [ 26 ] is highly nontrivial in the overparameterized setting as it holds when the ridge parameter equals to zero or even being negative . This line of results build one part of the theoretical tools for this paper . Excess Risk Bounds for SGD . Risk bounds for one-pass , constant-stepsize ( average ) SGD have been derived in the finite dimensional case [ 4 , 9 , 17 , 18 , 11 , 1 ] . Very recently , the work of [ 30 ] extends these analyses , providing sharp instance-dependent risk bound applicable to the overparameterized regime ; here , Zou et al . [ 30 ] provides nearly matching upper and lower excess risk bounds for constant-stepsize SGD , which are sharply characterized in terms of the full eigenspectrum of the population covariance matrix . This result plays a pivotal role in our paper . Implicit Regularization of SGD vs . Explicit Norm-based Regularization . For least square problems , multi-pass SGD converges to the minimum-norm solution [ 22 , 29 , 14 ] , which is widely cited as ( one of ) the implicit bias of SGD . However , in more general settings , e.g. , convex but non-linear models , a ( distribution-independent ) norm-based regularizer is no longer sufficient to characterize the optimization behavior of SGD [ 3 , 8 , 23 ] . Those discussions , however , exclude the possibility of hyperparameter tuning , e.g. , stepsize for SGD and penalty strength for ridge regression , and are not instance-based , either . Our aim in this paper is to provide instance-based excess risk comparison between the optimally tuned ( one-pass ) SGD and the optimally tuned ridge regression . 3 Problem Setup and Preliminaries . We seek to compare the generalization ability of SGD and ridge algorithms for least square problems . We use x ∈ H to denote a feature vector in a ( separable ) Hilbert spaceH . We use d to refer to the dimensionality ofH , where d =∞ ifH is infinite-dimensional . We use y ∈ R to denote a response that is generated by y = 〈x , w∗〉+ ξ , where w∗ ∈ H is an unknown true model parameter and ξ ∈ R is the model noise . The following regularity assumption is made throughout the paper . Assumption 3.1 ( Well-specified noise ) . The second moment of x , denoted by H : = E [ xx > ] , is strictly positive definite and has finite trace . The noise ξ is independent of x and satisfies E [ ξ ] = 0 , and E [ ξ2 ] = σ2 . In order to characterize the interplay between w∗ and H in the excess risk bound , we introduce : H0 : k : = ∑k i=1λiviv > i , and Hk : ∞ : = ∑ i > kλiviv > i , where { λi } ∞i=1 are the eigenvalues of H sorted in non-increasing order and vi ’ s are the corresponding eigenvectors . Then we define ‖w‖2 H−10 : k = ∑ i≤k ( v > i w ) 2 λi , ‖w‖2Hk : ∞ = ∑ i > k λi ( v > i w ) 2 . The least squares problem is to estimate the true parameter w∗ . Assumption 3.1 implies that w∗ is the unique solution that minimizes the population risk : L ( w∗ ) = min w∈H L ( w ) , where L ( w ) : = 1 2 E ( x , y ) ∼D [ ( y − 〈w , x〉 ) 2 ] . ( 3.1 ) Moreover we have that L ( w∗ ) = σ2 . For an estimation w found by some algorithm , e.g. , SGD or ridge regression , its performance is measured by the excess risk , L ( w ) − L ( w∗ ) . Constant-Stepsize SGD with Tail-Averaging . We consider the constant-stepsize SGD with tailaveraging [ 4 , 17 , 18 , 30 ] : at the t-th iteration , a fresh example ( xt , yt ) is sampled independently from the data distribution , and SGD makes the following update on the current estimator wt−1 ∈ H , wt = wt−1 + γ · ( yt − 〈wt−1 , xt〉 ) xt , t = 1 , 2 , . . . , w0 = 0 , where γ > 0 is a constant stepsize . AfterN iterations ( which is also the number of samples observed ) , SGD outputs the tail-averaged iterates as the final estimator : wsgd ( N ; γ ) : = 2 N N−1∑ t=N/2 wt . In the underparameterized setting ( d < N ) , constant-stepsize SGD with tail-averaging is known for achieving minimax optimal rate for least squares [ 17 , 18 ] . More recently , Zou et al . [ 30 ] investigate the performance of constant-stepsize SGD with tail-averaging in the overparameterized regime ( d > N ) , and establish instance-dependent , nearly-optimal excess risk bounds under mild assumptions on the data distribution . Notably , results from [ 30 ] cover underparameterized cases ( d < N ) as well . Ridge Regression . Given N i.i.d . samples { ( xi , yi ) } Ni=1 , let us denote X : = [ x1 , . . . , xN ] > ∈ RN×d and y : = [ y1 , . . . , yN ] > ∈ Rd . Then ridge regression outputs the following estimator for the true parameter [ 25 ] : wridge ( N ; λ ) : = arg min w∈H ‖Xw − y‖22 + λ‖w‖22 , ( 3.2 ) where λ ( which could possibly be negative ) is a regularization parameter . We remark that the ridge regression estimator takes the following two equivalent form : wridge ( N ; λ ) = ( X > X + λId ) −1X > y = X > ( XX > + λIN ) −1y . ( 3.3 ) The first expression is useful in the classical , underparameterized setting ( d < N ) [ 16 ] ; and the second expression is more useful in the overparameterized setting ( d > N ) where the empirical covariance X > X is usually not invertible [ 20 , 26 ] . As a final remark , when λ = 0 , ridge estimator reduces to the ordinary least square estimator ( OLS ) [ 13 ] . Generalizable Regime . In the following sections we will make instance-based risk comparisons between SGD and ridge regression . To make the comparison meaningful , we focus on regime where SGD and ridge regression are “ generalizable ” , i.e , the SGD and the ridge regression estimators , with the optimally-tuned hypeparameters , can achieve excess risk that is smaller than the optimal population risk , i.e. , σ2 . The formal mathematical definition is as follows . Definition 1 ( Generalizability ) . Consider an algorithm Alg and a least squares problem instance P. Let Alg ( n , θ ) be the output of the algorithm when provided with n i.i.d . samples from the problem instance P , and a set of hyperparameters θ ( that could be a function on n ) . Then we say that the algorithm Alg with sample size n and hyperparameters configuration θ is generalizable on problem instance P , if EAlg , P [ L ( Alg ( n , θ ) ) ] − L ( w∗ ) ≤ σ2 , where the expectation is over the randomness of Alg and data drawn from the problem instance P. Clearly , the generalizable regime is defined by conditions on both the sample size , hyperparameter configuration , the problem instance , and the algorithm . For example , in the d-dimensional setting with ‖w∗‖2 = O ( 1 ) , the ordinary least squares ( OLS ) solution ( ridge regression with λ = 0 ) , i.e. , wridge ( N ; 0 ) hasO ( dσ2/N ) excess risk , then we can say that the ridge regression with regularization parameter λ = 0 and sample size N = ω ( d ) is in the generalizable regime on all problem instances in d-dimension with ‖w∗‖2 = O ( 1 ) . Sample Inflation vs. Risk Inflation Comparisons . This work characterizes the sample inflation of SGD , i.e. , bounding the required sample size of SGD to achieve an instance-based comparable excess risk as ridge regression ( which is essentially the notion of Bahadur statistical efficiency [ 5 , 6 ] ) . Another natural comparison would be examining the risk inflation of SGD , examining the instancebased increase in risk for any fixed sample size . Our preference for the former is due to the relative instability of the risk with respect to the sample size ( in some cases , given a slightly different sample size , the risk could rapidly change . ) .

Summary: This paper compares the excess risk of averaged SGD with that of ridge regression. The comparison is in the setting where the expected noise is zero and also the excess risks of the optimally tuned algorithms are smaller than the optimal population risk. The overall conclusion is that the performance of SGD is close to ridge regression in every problem in the class of interest and can even be better than ridge regression based on the eigenspectrum of the input and the signal-to-noise ratio.