Implicit Sparse Regularization: The Impact of Depth and Early Stopping

In this paper , we study the implicit bias of gradient descent for sparse regression . We extend results on regression with quadratic parametrization , which amounts to depth-2 diagonal linear networks , to more general depth-N networks , under more realistic settings of noise and correlated designs . We show that early stopping is crucial for gradient descent to converge to a sparse model , a phenomenon that we call implicit sparse regularization . This result is in sharp contrast to known results for noiseless and uncorrelated-design cases . We characterize the impact of depth and early stopping and show that for a general depth parameter N , gradient descent with early stopping achieves minimax optimal sparse recovery with sufficiently small initialization w0 and step size ⌘ . In particular , we show that increasing depth enlarges the scale of working initialization and the early-stopping window so that this implicit sparse regularization effect is more likely to take place . 1 Introduction . Motivation . Central to recent research in learning theory is the insight that the choice of optimization algorithms plays an important role in model generalization [ 1 , 2 , 3 ] . A widely adopted view is that ( stochastic ) gradient descent — the most popular optimization algorithm in machine learning — exhibits some implicit form of regularization . Indeed for example , in the classical under-determined least squares setting , gradient descent ( with small step size ) starting from the origin converges to the model with minimum Euclidean norm . Similar implicit biases are also observed in deep neural network training in which the networks typically have many more parameters than the sample size . There , gradient descent without explicit regularization finds solutions that not only interpolate the training data points but also generalize well on test sets [ 2 , 4 , 5 , 6 , 7 ] . This insight , combined with the empirical success stories of deep learning , has sparked significant interest among theoretical researchers to rigorously understand implicit regularization . The majority of theoretical results focus on well-understood problems such as regression with linear models [ 8 , 9 , 10 , 11 , 12 , 13 ] and matrix factorization [ 14 , 15 , 16 , 17 ] , and show that the parametrization ( or architecture ) of the model plays a crucial role . For the latter , Gunasekar et al . [ 15 ] conjectured that gradient descent on factorized matrix representations converges to the solution with minimum nuclear norm . The conjecture was partially proved by Li et al . [ 16 ] under the Restricted Isometry Property ( RIP ) and the absence of noise . Arora et al . [ 17 ] further show the same nuclear-norm implicit bias using depth-N linear networks ( i.e. , the matrix variable is factorized into N components ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Parallel work on nonlinear models and classification [ 5 , 9 ] has shown that gradient descent biases the solution towards the max-margin/minimum ` 2-norm solutions over separable data . The scale of initialization in gradient descent leads to two learning regimes ( dubbed “ kernel ” and “ rich ” ) in linear networks [ 18 ] , shallow ReLU networks [ 19 ] and deep linear classifiers [ 20 ] . Li et al . [ 21 ] showed that depth-2 network requires an exponentially small initialization , whereas depth-N network ( N 3 ) only requires a polynomial small initialization , to obtain low-rank solution in matrix factorisation . Woodworth et al . [ 18 ] obtained a similar result for high dimensional sparse regression . The trend in the large majority of the above works has been to capture implicit regularization of gradient descent using some type of norm with respect to the working parametrization [ 18 , 22 , 23 , 24 ] . On the other hand , progress on understanding the trajectory of gradient descent has been somewhat more modest . [ 11 , 12 ] study the sparse regression problem using quadratic and Hadamard parametrization respectively and show that gradient descent with small initialization and careful early stopping achieves minimax optimal rates for sparse recovery . Unlike [ 16 , 18 ] that study noiseless settings and require no early stopping , [ 11 , 12 ] mathematically characterize the role of early stopping and empirically show that it may be necessary to prevent gradient descent from over-fitting to the noise . These works suggest that the inductive bias endowed by gradient descent may be influenced not only by the choice of parametrization , but also algorithmic choices such as initialization , learning rate , and the number of iterations . However , our understanding of such gradient dynamics is incomplete , particularly in the context of deep architectures ; see Table 1 for some comparisons . Contributions . Our focus in this paper is the implicit regularization of ( standard ) gradient descent for high dimensional sparse regression , namely implicit sparse regularization . Let us assume a groundtruth sparse linear model and suppose we observe n noisy samples ( xi , yi ) , such that y = Xw⇤+⇠ ; a more formal setup is given in Section 2 . Using the samples , we consider gradient descent on a squared loss kXw yk2 with no explicit sparsity regularization . Instead , we write the parameter vector w in the form w = uN vN with N 2 . Now , the regression function f ( x , u , v ) = hx , uN vN i can be viewed as a depth-N diagonal linear network [ 18 ] . Minimizing the ( now non-convex ) loss over u and v with gradient descent is then analogous to training this depth-N network . Our main contributions are the following . We characterize the impact of both the depth and early stopping for this non-convex optimization problem . Along the way , we also generalize the results of [ 11 ] for N > 2 . We show that under a general depth parameter N and an incoherence assumption on the design matrix , gradient descent with early stopping achieves minimax optimal recovery with sufficiently small initialization w0 and step size ⌘ . The choice of step size is of order O ( 1/N2 ) . Moreover , the upper bound of the initialization , as well as the early-stopping window , increase with N , suggesting that depth leads to a more accessible generalizable solution on gradient trajectories . Techniques . At a high level , our work continues the line of work on implicit bias initiated in [ 11 , 12 , 18 ] and extends it to the deep setting . Table 1 highlights key differences between our work and [ 11 , 13 , 18 ] . Specifically , Woodworth et al . [ 18 ] study the interpolation given by the gradient flow of the squared-error loss function . Vaskevicius et al . [ 11 ] analyze the finite gradient descent and characterize the implicit sparse regularization on the recovery of true parameters with N = 2 . Lastly , Gissin et al . [ 13 ] discover the incremental learning dynamic of gradient flow for general N but in an idealistic model setting where u ⌫ 0 , v = 0 , ⇠ = 0 , uncorrelated design and with infinitely many samples . At first glance , one could attempt a straightforward extension of the proof techniques in [ 11 ] to general settings of N > 2 . However , this turns out to be very challenging . Consider even the simplified case where the true model w ? is non-negative , the design matrix is unitary ( i.e. , n 1XTX = I ) , and the noise is absent ( ⇠ = 0 ) ; this is the setting studied in [ 13 ] . For each entry wi of w , the tth iterate of gradient descent over the depth-N reparametrized model is given by : wi , t+1 = wi , t ⇣ 1 + w 1 2N i , t ( w ? i wi , t ) ⌘N , which is no longer a simple multiplicative update . As pointed out in [ 13 ] ( see their Appendix C ) , the recurrence relation is not analytically solvable due to the presence of the ( pesky ) term w1 2 N i , t when N > 2 . Moreover , this extra term w1 2 N i , t leads to widely divergent growth rates of weights with different magnitudes , which further complicates analytical bounds . To resolve this and rigorously analyze the dynamics for N > 2 , we rely on a novel first order , continuous approximation to study growth rates without requiring additional assumptions on gradient flow , and carefully bound the approximation error due to finite step size ; see Section 4 . 2 Setup . Sparse regression/recovery . Let w ? 2 Rp be a p-dimensional sparse vector with k non-zero entries . Assume that we observe n data points ( xi , yi ) 2 Rp⇥R such that yi = hxi , w ? i+⇠i for i = 1 , . . . , n , where ⇠ = ( ⇠1 , . . . , ⇠n ) is the noise vector . We do not assume any particular scaling between the number of observations n and the dimension p. Due to the sparsity of w ? , however , we allow n ⌧ p. The linear model can be expressed in the matrix-vector form : y = Xw ? + ⇠ , ( 1 ) with the n⇥p design matrix X = [ x > 1 , . . . , x > n ] > , where xi denotes the ith row of X . We also denote X = [ X1 , . . . , Xp ] , where Xi denotes the ith column of X . The goal of sparse regression is to estimate the unknown , sparse vector w ? from the observations . Over the past two decades , this problem has been a topic of active research in statistics and signal processing [ 25 ] . A common approach to sparse regression is penalized least squares with sparsityinduced regularization such as ` 0 or ` 1 penalties/constraints , leading to several well-known estimators [ 25 , 26 , 27 ] and algorithms [ 28 , 29 ] . Multiple estimators enjoy optimal statistical and algorithmic recovery guarantees under some conditions of the design matrix X ( e.g. , RIP [ 30 ] ) and the noise ⇠ . We deviate from the standard penalized least squares formulation and instead learn w⇤ via a polynomial parametrization : w = uN vN , u , v 2 Rp , where N 2 and zN = [ zN1 , . . . , zNp ] > for any z = [ z1 , . . . , zN ] > 2 Rp . The regression function f ( x , u , v ) = hx , uN vN i induced by such a parametrization is equivalent to a N -layer diagonal linear network [ 18 ] with 2p hidden neurons and the diagonal weight matrix shared across all layers . Given the data { X , y } observed in ( 1 ) , we analyze gradient descent with respect to the new parameters u and v over the mean squared error loss without explicit regularization : L ( u , v ) = 1 n X ( uN vN ) y 2 2 , u , v 2 Rp . Even though the loss function yields the same value for the two parametrizations , L ( u , v ) is nonconvex in u and v. Unlike several recent studies in implicit regularization for matrix factorization and regression [ 16 , 18 , 13 ] , we consider the noisy setting , which is more realistic and leads to more insights into the bias induced during the optimization . Because of noise , the loss evaluated at the ground truth ( i.e. , any u , v such that w ? = uN vN ) is not necessarily zero or even minimal . Gradient descent . The standard gradient descent update over L ( u , v ) reads as : u0 = v0 = ↵1 , ( ut+1 , vt+1 ) = ( ut , vt ) ⌘ @ L ( ut , vt ) @ ( ut , vt ) , t = 0 , 1 , . . . . ( 2 ) Here , ⌘ > 0 is the step size and ↵ > 0 is the initialization of u , v . In general , we analyze the algorithm presented in ( 2 ) , and at each step t , we can estimate the signal of interest by simply calculating wt = uNt vNt . We consider constant initialization for simplicity sake . Our results apply for random initialization concentrating on a small positive region with a probabilistic statement . Vaskevicius et al . [ 11 ] establish the implicit sparse regularization of gradient descent for N = 2 and show minimax optimal recovery , provided sufficiently small ↵ and early stopping . Our work aims to generalize that result to N > 2 and characterize the role of N in convergence . Notation . We define S = { i 2 { 1 , . . . , p } : w ? i 6= 0 } and Sc = { 1 , . . . , p } \S . The largest and smallest absolute value on the support is denoted as w ? max = maxi2S |w ? i | and w ? min = mini2S |w ? i | . We use 1 to denote the vector of all ones and 1S denotes the vector whose elements on S are all one and 0 otherwise . Also , denotes coordinate-wise multiplication . We denote st = 1S wt and et = 1Sc wt meaning the signal part and error part at each time step t. We use ^ and _ to denote the pointwise maximum and minimum . The coordinate-wise inequalities are denoted as < . We denote inequalities up to multiplicative absolute constants by . , which means that they do not depend on any parameters of the problem . Definition 1 . Let X 2 Rn⇥p be a matrix with ` 2-normalized columns X1 , . . . , Xp , i.e. , kXik2 = 1 for all i . The coherence µ = µ ( X ) of the matrix X is defined as µ : = max 1i 6=jp |hXi , Xji| . The matrix X is said to be satisfying µ-incoherence . The coherence is a measure for the suitability of the measurement matrix in compressive sensing [ 31 ] . In general , the smaller the coherence , the better the recovery algorithms perform . There are multiple ways to construct a sensing matrix with low-incoherence . One of them is based on the fact that sub-Gaussian matrices satisfy low-incoherence property with high probability [ 32 , 33 ] . In contrast to the coherence , the Restricted Isometry Property ( RIP ) is a powerful performance measure for guaranteeing sparse recovery and has been widely used in many contexts . However , verifying the RIP for deterministically constructed design matrices is NP-hard . On the other hand , coherence is a computationally tractable measure and its use in sparse regression is by now classical [ 33 , 34 ] . Therefore , in contrast with previous results [ 11 ] ( which assumes RIP ) , the assumptions made in our main theorems are verifiable in polynomial time .

Summary: In this paper the authors study implicit sparse regularization for diagonal linear networks. Specifically, the authors extend the previous result of Vaskevicius et al. [9] to the case of depth>2, showing that for noisy regression under a relaxed assumption on the inputs, increasing depth enlarges the scale of working initialization and the early-stopping window.