Implicit Sparse Regularization: The Impact of Depth and Early Stopping

In this paper , we study the implicit bias of gradient descent for sparse regression . We extend results on regression with quadratic parametrization , which amounts to depth-2 diagonal linear networks , to more general depth-N networks , under more realistic settings of noise and correlated designs . We show that early stopping is crucial for gradient descent to converge to a sparse model , a phenomenon that we call implicit sparse regularization . This result is in sharp contrast to known results for noiseless and uncorrelated-design cases . We characterize the impact of depth and early stopping and show that for a general depth parameter N , gradient descent with early stopping achieves minimax optimal sparse recovery with sufficiently small initialization w0 and step size ⌘ . In particular , we show that increasing depth enlarges the scale of working initialization and the early-stopping window so that this implicit sparse regularization effect is more likely to take place . 1 Introduction . Motivation . Central to recent research in learning theory is the insight that the choice of optimization algorithms plays an important role in model generalization [ 1 , 2 , 3 ] . A widely adopted view is that ( stochastic ) gradient descent — the most popular optimization algorithm in machine learning — exhibits some implicit form of regularization . Indeed for example , in the classical under-determined least squares setting , gradient descent ( with small step size ) starting from the origin converges to the model with minimum Euclidean norm . Similar implicit biases are also observed in deep neural network training in which the networks typically have many more parameters than the sample size . There , gradient descent without explicit regularization finds solutions that not only interpolate the training data points but also generalize well on test sets [ 2 , 4 , 5 , 6 , 7 ] . This insight , combined with the empirical success stories of deep learning , has sparked significant interest among theoretical researchers to rigorously understand implicit regularization . The majority of theoretical results focus on well-understood problems such as regression with linear models [ 8 , 9 , 10 , 11 , 12 , 13 ] and matrix factorization [ 14 , 15 , 16 , 17 ] , and show that the parametrization ( or architecture ) of the model plays a crucial role . For the latter , Gunasekar et al . [ 15 ] conjectured that gradient descent on factorized matrix representations converges to the solution with minimum nuclear norm . The conjecture was partially proved by Li et al . [ 16 ] under the Restricted Isometry Property ( RIP ) and the absence of noise . Arora et al . [ 17 ] further show the same nuclear-norm implicit bias using depth-N linear networks ( i.e. , the matrix variable is factorized into N components ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Parallel work on nonlinear models and classification [ 5 , 9 ] has shown that gradient descent biases the solution towards the max-margin/minimum ` 2-norm solutions over separable data . The scale of initialization in gradient descent leads to two learning regimes ( dubbed “ kernel ” and “ rich ” ) in linear networks [ 18 ] , shallow ReLU networks [ 19 ] and deep linear classifiers [ 20 ] . Li et al . [ 21 ] showed that depth-2 network requires an exponentially small initialization , whereas depth-N network ( N 3 ) only requires a polynomial small initialization , to obtain low-rank solution in matrix factorisation . Woodworth et al . [ 18 ] obtained a similar result for high dimensional sparse regression . The trend in the large majority of the above works has been to capture implicit regularization of gradient descent using some type of norm with respect to the working parametrization [ 18 , 22 , 23 , 24 ] . On the other hand , progress on understanding the trajectory of gradient descent has been somewhat more modest . [ 11 , 12 ] study the sparse regression problem using quadratic and Hadamard parametrization respectively and show that gradient descent with small initialization and careful early stopping achieves minimax optimal rates for sparse recovery . Unlike [ 16 , 18 ] that study noiseless settings and require no early stopping , [ 11 , 12 ] mathematically characterize the role of early stopping and empirically show that it may be necessary to prevent gradient descent from over-fitting to the noise . These works suggest that the inductive bias endowed by gradient descent may be influenced not only by the choice of parametrization , but also algorithmic choices such as initialization , learning rate , and the number of iterations . However , our understanding of such gradient dynamics is incomplete , particularly in the context of deep architectures ; see Table 1 for some comparisons . Contributions . Our focus in this paper is the implicit regularization of ( standard ) gradient descent for high dimensional sparse regression , namely implicit sparse regularization . Let us assume a groundtruth sparse linear model and suppose we observe n noisy samples ( xi , yi ) , such that y = Xw⇤+⇠ ; a more formal setup is given in Section 2 . Using the samples , we consider gradient descent on a squared loss kXw yk2 with no explicit sparsity regularization . Instead , we write the parameter vector w in the form w = uN vN with N 2 . Now , the regression function f ( x , u , v ) = hx , uN vN i can be viewed as a depth-N diagonal linear network [ 18 ] . Minimizing the ( now non-convex ) loss over u and v with gradient descent is then analogous to training this depth-N network . Our main contributions are the following . We characterize the impact of both the depth and early stopping for this non-convex optimization problem . Along the way , we also generalize the results of [ 11 ] for N > 2 . We show that under a general depth parameter N and an incoherence assumption on the design matrix , gradient descent with early stopping achieves minimax optimal recovery with sufficiently small initialization w0 and step size ⌘ . The choice of step size is of order O ( 1/N2 ) . Moreover , the upper bound of the initialization , as well as the early-stopping window , increase with N , suggesting that depth leads to a more accessible generalizable solution on gradient trajectories . Techniques . At a high level , our work continues the line of work on implicit bias initiated in [ 11 , 12 , 18 ] and extends it to the deep setting . Table 1 highlights key differences between our work and [ 11 , 13 , 18 ] . Specifically , Woodworth et al . [ 18 ] study the interpolation given by the gradient flow of the squared-error loss function . Vaskevicius et al . [ 11 ] analyze the finite gradient descent and characterize the implicit sparse regularization on the recovery of true parameters with N = 2 . Lastly , Gissin et al . [ 13 ] discover the incremental learning dynamic of gradient flow for general N but in an idealistic model setting where u ⌫ 0 , v = 0 , ⇠ = 0 , uncorrelated design and with infinitely many samples . At first glance , one could attempt a straightforward extension of the proof techniques in [ 11 ] to general settings of N > 2 . However , this turns out to be very challenging . Consider even the simplified case where the true model w ? is non-negative , the design matrix is unitary ( i.e. , n 1XTX = I ) , and the noise is absent ( ⇠ = 0 ) ; this is the setting studied in [ 13 ] . For each entry wi of w , the tth iterate of gradient descent over the depth-N reparametrized model is given by : wi , t+1 = wi , t ⇣ 1 + w 1 2N i , t ( w ? i wi , t ) ⌘N , which is no longer a simple multiplicative update . As pointed out in [ 13 ] ( see their Appendix C ) , the recurrence relation is not analytically solvable due to the presence of the ( pesky ) term w1 2 N i , t when N > 2 . Moreover , this extra term w1 2 N i , t leads to widely divergent growth rates of weights with different magnitudes , which further complicates analytical bounds . To resolve this and rigorously analyze the dynamics for N > 2 , we rely on a novel first order , continuous approximation to study growth rates without requiring additional assumptions on gradient flow , and carefully bound the approximation error due to finite step size ; see Section 4 . 2 Setup . Sparse regression/recovery . Let w ? 2 Rp be a p-dimensional sparse vector with k non-zero entries . Assume that we observe n data points ( xi , yi ) 2 Rp⇥R such that yi = hxi , w ? i+⇠i for i = 1 , . . . , n , where ⇠ = ( ⇠1 , . . . , ⇠n ) is the noise vector . We do not assume any particular scaling between the number of observations n and the dimension p. Due to the sparsity of w ? , however , we allow n ⌧ p. The linear model can be expressed in the matrix-vector form : y = Xw ? + ⇠ , ( 1 ) with the n⇥p design matrix X = [ x > 1 , . . . , x > n ] > , where xi denotes the ith row of X . We also denote X = [ X1 , . . . , Xp ] , where Xi denotes the ith column of X . The goal of sparse regression is to estimate the unknown , sparse vector w ? from the observations . Over the past two decades , this problem has been a topic of active research in statistics and signal processing [ 25 ] . A common approach to sparse regression is penalized least squares with sparsityinduced regularization such as ` 0 or ` 1 penalties/constraints , leading to several well-known estimators [ 25 , 26 , 27 ] and algorithms [ 28 , 29 ] . Multiple estimators enjoy optimal statistical and algorithmic recovery guarantees under some conditions of the design matrix X ( e.g. , RIP [ 30 ] ) and the noise ⇠ . We deviate from the standard penalized least squares formulation and instead learn w⇤ via a polynomial parametrization : w = uN vN , u , v 2 Rp , where N 2 and zN = [ zN1 , . . . , zNp ] > for any z = [ z1 , . . . , zN ] > 2 Rp . The regression function f ( x , u , v ) = hx , uN vN i induced by such a parametrization is equivalent to a N -layer diagonal linear network [ 18 ] with 2p hidden neurons and the diagonal weight matrix shared across all layers . Given the data { X , y } observed in ( 1 ) , we analyze gradient descent with respect to the new parameters u and v over the mean squared error loss without explicit regularization : L ( u , v ) = 1 n X ( uN vN ) y 2 2 , u , v 2 Rp . Even though the loss function yields the same value for the two parametrizations , L ( u , v ) is nonconvex in u and v. Unlike several recent studies in implicit regularization for matrix factorization and regression [ 16 , 18 , 13 ] , we consider the noisy setting , which is more realistic and leads to more insights into the bias induced during the optimization . Because of noise , the loss evaluated at the ground truth ( i.e. , any u , v such that w ? = uN vN ) is not necessarily zero or even minimal . Gradient descent . The standard gradient descent update over L ( u , v ) reads as : u0 = v0 = ↵1 , ( ut+1 , vt+1 ) = ( ut , vt ) ⌘ @ L ( ut , vt ) @ ( ut , vt ) , t = 0 , 1 , . . . . ( 2 ) Here , ⌘ > 0 is the step size and ↵ > 0 is the initialization of u , v . In general , we analyze the algorithm presented in ( 2 ) , and at each step t , we can estimate the signal of interest by simply calculating wt = uNt vNt . We consider constant initialization for simplicity sake . Our results apply for random initialization concentrating on a small positive region with a probabilistic statement . Vaskevicius et al . [ 11 ] establish the implicit sparse regularization of gradient descent for N = 2 and show minimax optimal recovery , provided sufficiently small ↵ and early stopping . Our work aims to generalize that result to N > 2 and characterize the role of N in convergence . Notation . We define S = { i 2 { 1 , . . . , p } : w ? i 6= 0 } and Sc = { 1 , . . . , p } \S . The largest and smallest absolute value on the support is denoted as w ? max = maxi2S |w ? i | and w ? min = mini2S |w ? i | . We use 1 to denote the vector of all ones and 1S denotes the vector whose elements on S are all one and 0 otherwise . Also , denotes coordinate-wise multiplication . We denote st = 1S wt and et = 1Sc wt meaning the signal part and error part at each time step t. We use ^ and _ to denote the pointwise maximum and minimum . The coordinate-wise inequalities are denoted as < . We denote inequalities up to multiplicative absolute constants by . , which means that they do not depend on any parameters of the problem . Definition 1 . Let X 2 Rn⇥p be a matrix with ` 2-normalized columns X1 , . . . , Xp , i.e. , kXik2 = 1 for all i . The coherence µ = µ ( X ) of the matrix X is defined as µ : = max 1i 6=jp |hXi , Xji| . The matrix X is said to be satisfying µ-incoherence . The coherence is a measure for the suitability of the measurement matrix in compressive sensing [ 31 ] . In general , the smaller the coherence , the better the recovery algorithms perform . There are multiple ways to construct a sensing matrix with low-incoherence . One of them is based on the fact that sub-Gaussian matrices satisfy low-incoherence property with high probability [ 32 , 33 ] . In contrast to the coherence , the Restricted Isometry Property ( RIP ) is a powerful performance measure for guaranteeing sparse recovery and has been widely used in many contexts . However , verifying the RIP for deterministically constructed design matrices is NP-hard . On the other hand , coherence is a computationally tractable measure and its use in sparse regression is by now classical [ 33 , 34 ] . Therefore , in contrast with previous results [ 11 ] ( which assumes RIP ) , the assumptions made in our main theorems are verifiable in polynomial time .

Summary: In this work, the authors study the implicit bias of gradient descent for sparse linear regression when parameterizing the ground truth signal as a depth-$N$ diagonal linear network. Previous works have shown that when utilizing a Hadamard parameterization $w = u \odot u - v \odot v$, gradient descent exhibits an implicit bias towards sparse solutions, without any explicit regularization encouraging sparsity. Follow-up work has also considered the depth-$N$ case in simplified settings (e.g., without noise, positive signals, and idealized measurement setups). The authors extend this analysis to the general depth-$N$ case $w = u^N - v^N$ in the presence of noise. They prove that when the measurement matrix is $\mu$-incoherent with sufficiently small $\mu$ and noise is present, there is an early stopping window depending on the problem parameters where any iterate during this time is close in $\ell_{\infty}$ norm to the underlying ground truth sparse signal. These results hold for any depth $N \geqslant 2$ and showcase that depth allows for larger initialization sizes and that the early stopping window grows with deeper parameterizations. Experimental evidence on toy examples is also provided corroborating several elements of the proposed theory.