Last iterate convergence of SGD for Least-Squares in the Interpolation regime.

1 Introduction . As soon as large-scale statistics and optimization need to work together , stochastic gradient descent ( SGD ) is the core algorithm everybody tries to build upon [ 7 ] . Its versatility , practicability and adaptability make it the workhorse of almost every supervised machine learning problem . Yet , its outstanding efficiency remains mysterious , or at least surprising on certain aspects . Furthermore , the recent successes of deep neural networks ( DNN ) brought a new paradigm to the classical supervised learning setting with the ability to fit the data perfectly and to generalize well [ 5 ] . Following this idea , the old statistical modelling where the model suffers from problem-dependent noise has to be revisited : there is a need of analyzing stochastic algorithms in this new light [ 18 ] . Whether we call it over-parameterization to put emphasis on the large number of neurons needed in DNNs , interpolation as in approximation theory or noiseless model to stress the absence of noise in this statistical model , all these terminologies refer to the same idea . This regime brings with it new insights that reflect better the current machine learning setup . Hence , the main question : how would SGD profit from this noiseless model ? At first glance , the story seems clear : the old problem of variance at optimum making the SGD iterates oscillate asymptotically now disappears . Thus , should also disappear techniques that prevent from this , namely , averaging and decaying step sizes : one should be able to study the convergence of the SGD final iterate with constant step-size . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the study of the last iterate of SGD has always caused some technical problems preventing from a clear theory in this case [ 26 ] . Indeed , the convergence of the final iterate is much more difficult to prove than the convergence of the average of the iterates [ 27 , 13 , 15 ] . This counter-intuitive difficulty can be explained by the fact that the interactions coming from the sampling noise of SGD prevent the loss of the final iterates from decreasing . Therefore standard Lyapounov strategies often failed in such a setting . Besides , even if averaged SGD has shown theoretically some good convergence properties , the final iterate is commonly used in practice . Finally , averaging techniques always suffer from saturation coming from the slow forgetting of initial conditions . To tackle these questions , we consider the simplest setting of the linear regression over features φ ( X ) in a Hilbert space H. In this context , the setup corresponds to the existence of linear relationship between the output and the input : Y = 〈θ∗ , φ ( X ) 〉 . Note that this setting is rich as the features can be a non-linear transformation of the inputs , as it is commonly the case when they are defined through a positive-definite kernel K ( X , X ′ ) [ 25 , 28 ] . Note also that our analysis will , unless stated explicitly , be conducted in a non-parametric and dimensionless fashion , enabling the features to come from a infinite dimensional reproducing kernel Hilbert space . In this perspective , the problem we are considering is not strongly convex . Main contributions . The aim of the present article is to answer at once the two following problems : ( i ) from a ( stochastic ) optimization perspective , the goal is to exhibit an archetypal problem where we can show explicitly the convergence of SGD final iterate for a non-strongly convex problem with constant step-size and ( ii ) from a statistical/machine learning perspective , the aim is to push deeper the study of the over-parameterized setting for the non-parametric least-squares problem . Contrary to the noisy setting , where ( almost ) everything is known , the noiseless model suffers from a certain lack of understanding . More precisely , our contributions are the following : • We show that the final iterate of constant step-size SGD achieves a convergence rate of O ( ln ( T ) /T ) under minimal assumptions . With a slightly stricter hypothesis we improve this convergence rate to O ( 1/T ) . • Going further , we assume the usual non-parametric capacity and source conditions respectively on the spectrum of the covariance and the optimum , and derive bounds for this fine-grained model . In this setup , we show the SGD fast rates of O ( 1/T 1+α ) . • We derive an explicit recursion on the eigenspaces of the covariance matrix that is of its own interest . Indeed , it is the cornerstone of our analysis and could be very useful in the future to understand important properties of SGD . From a technical standpoint , going from the average to the last iterate is far from anodyne for constant step-size SGD even in the interpolation regime . Indeed , in a similar setting , D.Aldous considered it an open problem [ 1 , Sec . 3.3 , Open problem ( i ) ] . Prior to our result , how to directly deal with the fluctuations induced by the stochasticity of the gradients without any variance reduction was not known . Our work presents a direct Lyapunov technique that handles them without any explicit variance reduction methods like averaging or step-size decay . 1.1 Related Work . As recalled earlier , it is often easier to show convergence for the averaged iterates of SGD than for the final one . However , there has been a huge effort by the optimization and machine learning communities to work on the final iterate . We report here the different works and contexts for which such results are shown . All the results stated below are for convergence in function value . Last versus averaged iterates . First , for non-smooth functions ( and variance of the SGD-gradients uniformly bounded ) , it is easy to show , with proper decrease of the step sizes , that the averaged iterates converge at rate O ( 1/ √ T ) in the non-strongly convex case and O ( 1/T ) when the function is strongly convex . On the other side , convergence of the last iterate has been first addressed by [ 35 , 27 ] who showed convergence rates of O ( log ( T ) / √ T ) for non-strongly convex objectives and O ( log ( T ) /T ) for strongly convex objectives . These rates have recently been shown to be tight by [ 13 ] ( the log term can not be suppressed for the last iterate ) . Secondly , for smooth functions ( gradient Lipschitz and variance bounded only at optimum ) , [ 2 ] showed in the non-strongly convex case that smoothness does not help ; results on the averaged iterate are the same O ( 1/ √ T ) and results on the last iterate are actually worse : O ( T−1/3 ) . In the strongly convex setting , the optimal rate O ( 1/T ) rate is obtained both by the averaged and last iterates . However the averaged iterates are preferred since they lead to the optimal covariance and the step size is independent of the strong convexity constant [ 23 ] . Finally , the final iterate of SGD , despite its ubiquitous use in machine learning , never theoretically performs as well as the averaged , except when used with a geometrically-decaying learning-rate [ 15 , 12 ] . Over-parameterized setting . As recalled earlier , SGD in the over-parameterization regime corresponds to the assumption that all the function gradients vanish at optimum . It has been first studied by [ 24 ] who assumed furthermore a strong growth condition ( SGC ) ( first introduced by [ 29 , 32 ] ) but often too stringent to match the machine learning practical set-up . Up to our knowledge , [ 18 ] was the first to point out that this interpolation regime was particularly relevant in the recent deep learning framework and to show linear convergence in the strongly-convex setting . In the non-strongly convex case , O ( 1/T ) -convergence has been shown by [ 33 ] for the averaged iterates . Consequently , there has been many papers discussing this setting , proving convergence rates in different contexts : Polyak-Lojasiewicz [ 4 ] , accelerated [ 33 , 17 ] , second-order [ 19 ] , line-search [ 34 ] . However , no rate for the final iterate has been shown in the non-strongly convex setting for the last iterate as the only convergence rates achieved , O ( 1/T ) , corresponds in all these works to the averaged iterates . The Quadratic Case . More specifically , the quadratic case has been widely studied as a central model in the machine learning literature . As explained above , on the one hand , in the usual noisy setting , averaging or decaying step sizes have been considered , showing well-known O ( 1/T ) convergence rates [ 3 ] . The final iterate convergence of SGD for the noisy quadratic case is studied in [ 12 ] where they show a O ( d log T/T ) rate and advocate the primacy of exponential step decay . On the other hand , the SGD in over-parameterized setting has been considered only lately by [ 18 , 6 , 36 ] . The first work corresponds to the strongly convex case , where the final iterate provably converges whereas our work is more aligned to [ 6 , 36 ] where the non-strongly convex case is considered . In [ 21 ] , the authors also studied the noiseless setting and provide worst and average cases asymptotic rates for the non-strongly convex case in the n , d→∞ regime . In [ 6 ] , among other results , it is shown that only the minimum of the function value iterates converge at rate O ( 1/T ) and up to O ( 1/T 1+α ) for α > 0 when usual capacity and source conditions are assumed for the problem [ 8 , 30 ] . A part of our work can be seen as a continuation of [ 6 ] , strengthening its result by proving that the final iterate converges instead of the min or some average . 2 Problem Set-up : Stochastic Gradient Descent on the Least-squares Problem . The setting is classical for stochastic gradient descent for linear least-squares in a Hilbert spaceH . The function we would like to minimize over θ ∈ H is R ( θ ) = 1 2 Eρ ( 〈θ , φ ( X ) 〉 − Y ) 2 , ( 1 ) when ρ is the joint law over input/outputs ( X , Y ) ∈ X × R and φ is a feature map from X toH . For the sake of simplicity , the reader can assume thatH is a finite-dimensional Euclidean space . However , we take a special care that all the quantities and definitions could be valid whenever H is infinite dimensional . Hence , all the theorems are valid in a Reproducing Kernel Hilbert Spaces ( RKHS ) framework . Without any loss of generality , and to avoid heavy notations , we assume that φ ( X ) = X , and as soon as we refer the the finite dimensional case , we setH = Rd . Covariance matrix . We define the covariance operator on H : H : = Eρ [ X ⊗ X ] . This positive semi-definite operator is diagonalizable and plays a central role in our analysis . Hence , let us denote λi its non-negative eigenvalues sorted in non-increasing order and vi its corresponding eigenvectors . In finite dimension , the covariance operator is simply a d× d matrix H = Eρ [ XX > ] but , once again , notice that the analysis can also afford being infinite dimensional . In either case we denote λmax the largest eigenvalue of H . In the finite dimensional case λmin > 0 is always correctly defined but can be arbitrarily small . On the opposite , a standard and important consequence of the infinite dimension is that , in most cases ( especially when TrH < +∞ ) , the infinite sequence of eigenvalues is no longer lower bounded as it converges to 0 : H is no longer invertible and the problem is non-strongly convex . Noiseless model . We assume that the model does not suffer from any noise : there exists θ∗ ∈ H such that , ρ-almost surely , 〈θ∗ , x〉 = y . This means that the model is well specified and that there is no noise at optimum . Due to this noiseless condition , we expect our iterates to go to the optimum without decaying step-sizes or averaging . We can also rewrite the Risk in Eq . ( 1 ) : R ( θ ) = 1 2 ( θ − θ∗ ) > H ( θ − θ∗ ) = 1 2 Tr [ ( θ − θ∗ ) ( θ − θ∗ ) > H ] . ( 2 ) Link between Noiseless model and Interpolation . Interpolation corresponds to the case where our predictor can fit a finite set n of training data i.e. , 〈θ∗ , xi〉 = yi for i ≤ n. Note that this setting is naturally included in our model replacing the test risk R ( θ ) with the training loss whose finite sum structure can be rewritten Eρ̂ ( 〈θ , x〉 − Y ) 2 where ρ̂ is the empirical distribution of the data . Consequently our result on the final iterate convergence holds in this model for the training loss . SGD with constant step-size . To minimize the functionR defined in Eq . ( 1 ) , we do not have access to the distribution ρ but to a stream of i.i.d . observations ( xt , yt ) t > 1 sampled from it . We hence perform a gradient descent in the direction given by one sample at a time with constant stepsize γ > 0 . Throughout all this paper , the initial condition is always set to θ0 = 0 and for t > 1 , θt+1 = θt − γ ( 〈θt , xt〉 − yt ) xt . ( 3 ) The impossibility of linear rates . We stress that the least-squares setup we consider is a nonstrongly convex problem . Indeed , in finite dimension d , λmin > 0 , and SGD converges at linear rate ∼ e−γλminT [ 18 , Theorem 1 ] , this asymptotic regime occurring after a time scale τ ∼ 1/ ( γλmin ) . However , this apparent strong convexity is a lure , since the large ( or infinite ) dimension makes this convergence rate vacuous for an arbitrarily small λmin . Hence we focus on the non-parametric non-strongly convex setup where we prove non-asymptotic polynomial rates . Yet in finite dimension , it is always possible to see the linear regime after time 1/ ( γλmin ) as shown in Figure 1 , even if this time can be arbitrarily long .

Summary: The paper explores the last iterate convergence of SGD on the least squares problem in the interpolation regime (noiseless setting). The authors work in an abstract Hilbert space and thus consider (1) the non-strongly convex setting and (2) the streaming or online setting. There are three contributions of the paper:  (1). The final iterate of constant step-size SGD achieves a convergence rate of $O(ln(T)/T)$ under the assumption that the data matrix $X$ has (in expectation) bounded fourth moments.   (2). Under more assumption about the behavior of the eigenvalues of the covariance matrix of the data, that is $\lambda_i \le O(1/i \log(i))$, the authors can improve the last iterate convergence rate to $O(1/T)$. This matches the rate for the averaged iterates of SGD.  (3). Under stronger assumptions of the eigenvalues of the covariance matrix at the optimum (source) and capacity condition, one can get an even faster rate of convergence.