Last iterate convergence of SGD for Least-Squares in the Interpolation regime.
Authors: Aditya Vardhan Varre
Loucas Pillaud-Vivien
Nicolas Flammarion
Venue: NeurIPS 2021 Poster (2021)
Keywords: Interpolation
Least-Squares
SGD
Last iterate
Non-parametric rates.

Abstract: Motivated by the recent successes of neural networks that have the ability to fit the data perfectly \emph{and} generalize well, we study the noiseless model in the fundamental least-squares setup. We assume that an optimum predictor perfectly fits the inputs and outputs $\langle \theta_* , \phi(X) \rangle = Y$, where $\phi(X)$ stands for a possibly infinite dimensional non-linear feature map. To solve this problem, we consider the estimator given by the last iterate of stochastic gradient descent (SGD) with constant step-size. In this context, our contribution is two fold: (i) \emph{from a (stochastic) optimization perspective}, we exhibit an archetypal problem where we can show explicitly the convergence of SGD final iterate for a non-strongly convex problem with constant step-size whereas usual results use some form of average and (ii) \emph{from a statistical perspective}, we give explicit non-asymptotic convergence rates in the over-parameterized setting and leverage a \emph{fine-grained} parameterization of the problem to exhibit polynomial rates that can be faster than $O(1/T)$. The link with reproducing kernel Hilbert spaces is established.

Last iterate convergence of SGD for Least-Squares in the Interpolation regime
Aditya Varre
Loucas Pillaud-Vivien
EPFL aditya. var.epaaa.reeeepl..co
EPFL
loucas www.a..ppllladdd...aaeael.....
Nicolas Flammarion
EPFL
nicolas www.mamioo@epplllc...m
Abstract
Motivated by the recent successes of neural networks that have the ability to the data perfectly and generalize well, we study the noiseless model in the fundamental least-squares setup. We assume that an optimum predictor perfectly fits the inputs and outputs (0., (()))())) = Y. where ((X) stands for a possibly infinite dimensional non-linear feature map. To solve this problem, we consider the estimator given by the last iterate of stochastic gradient descent (SGD) with constant step-size. In this context, our contribution is twofold: (i).rrmm a (stochastic) optimization perspective, we exhibit an archetypal problem where we can show explicitly the convergence of SGD final iterate for non-strongly convex problem with constant step-size whereas usual results use some form of average and (ii) from statistical perspective, give explicit non-asymptotic convergence rates in the over- peer-parametrizedd setting and leverage a fine-grained parameterization of the problem exhibit polynomial rates that can be faster than (1/TT). The link with reproducing kernel Hilbert spaces is established.
Introduction
As soon as large-scale statistics and optimization need to work together, stochastic gradient descent (SGD) is the core algorithm everybody tries to build upon [7]. Its versatility, practicability and adaptability make the workhorse of almost every supervised machine learning problem. Yet, its outstanding efficiency remains mysterious, recent successes of deep neural networks (DNN) brought a new paradigm to the classical supervised learning setting with the ability fit the data perfectly and generalize well [5]. Following this idea, the old statistical modelling where the model suffers from problem-depence
least surprising certain aspects. Furthermore, the
noise has to be revisited: there is a need of analyzing stochastic algorithms in this new light [18]. Whether call over-parameterization to put emphasis on the large number of neurons needed in DNNs, interpolation as in approximation theory or noiseless model stress the absence of noise this statistical model, all these terminologies refer to the same idea. This regime brings with it new insights that reflect better the current machine learning setup.
Hence, the main question: how would SGD profit from this noiseless model? At first glance, the story seems clear: the old problem of variance optimum making the SGD iterates oscillate asymptotically now disappears. Thus, should also disappear techniques that prevent from this, namely, averaging and decaying step sizes: one should be able study the convergence of the SGD final iterate with constant size-
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

However, the study of the last iterate of SGD has always caused some technical problems preventing from a clear theory in this case [26]. Indeed, the convergence of the final iterate is much more difficult to prove than the convergence of the average of the iterates [27, 13, 15]. This counter-intuitive difficulty can be explained by the fact that the interactions coming from the sampling noise of SGD prevent the loss of the final iterates from decreasing. Therefore standard Lyapounov strategies often failed in such a setting. Besides, even if averaged SGD has shown theoretically some good convergence properties, the final iterate is commonly used in practice. Finally, averaging techniques always suffer from saturation coming from the slow forgetting of initial conditions.
To tackle these questions, we consider the simplest setting of the linear regression over features ((X) in a Hilbert space H. In this context, the setup corresponds to the existence of linear relationship between the output and the input: Y = (f., ((X)). Note that this setting is rich as the features can be non-linear transformation of the inputs, as it commonly the case when they are defined through a positive-definite kernel K(X, X") [25, 28]. Note also that our analysis will, unless stated explicitly, be conducted in a non-parametric and dimensionless fashion, enabling the features to come from a infinite dimensional reproducing kernel Hilbert space. In this perspective, the problem we are considering is not strongly convex.
Main contributions. The aim of the present article is to answer at once the two following problems: (i) from a (stochastic) optimization perspective, the goal is exhibit an archetypal problem where we can show explicitly the convergence of SGD final iterate for a non-strongly convex problem with constant step-size and (ii) from a statistical/machine learning perspective, the aim is to push deeper the study of the weeepparaeteriized setting for the non-parametric least-squares problem. Contrary to the noisy setting, where (almost) everything is known, the noiseless model suffers from a certain lack of understanding. More precisely, our contributions are the following:
We show that the final iterate of constant step-size SGD achieves a convergence rate of OIn(T)/T) under minimal assumptions. With a slightly stricter hypothesis we improve this convergence rate to O(L/T).
Going further, we assume the usual non- non-parametric capacity and source conditions respectively on the spectrum of the covariance and the optimum, and derive bounds for this fine- fine-grained model. In this setup, we show the SGD fast rates of O()
We derive an explicit recursion on the eigenspaces of the covariance matrix that is of its own interest. Indeed, it is the cornerstone of our analysis and could be very useful in the future to understand important properties of SGD.
From a technical standpoint, going from the average to the last iterate is far from anodyne for constant step-size SGD even in the interpolation regime. Indeed, in a similar setting, D.Aldous considered it an open problem [1. Sec. 3.3, Open problem (i)). Prior to our result, how to directly deal with the fluctuations induced by the stochasticity of the gradients without any variance reduction was not known. Our work presents a direct Lyapunov technique that handles them without any explicit variance reduction methods like averaging or step-size decay.
1.1 Related Work
As recalled earlier, it is often easier to show convergence for the averaged iterates of SGD than for the final one. However, there has been a huge effort by the optimization and machine learning communities to work on the final iterate. We report here the different works and contexts for which such results are shown. All the results stated below are for convergence in function value.
Last versus averaged iterates. First, for non-sh functions (and variance of the SGD -GD-gradients uniformly bounded), it is easy to show, with proper decrease of the step sizes, that the averaged iterates converge at rate O(1/
in the non-strongly convex case and O11TT) when the function is strongly convex. On the other side, convergence of the last iterate has been first addressed by [35, 27] who showed convergence rates of P[log(T)/TT for non-strongly convex objectives and (llog(T)/) /T) for strongly convex objectives. These rates have recently been shown to be tight by [13] (the log term cannot be suppressed for the last iterate). Secondly, for smooth functions (gradient Lipschitz and variance bounded only at optimum), [2] showed in the non-strongly convex case that smoothness does not help; results on the averaged iterate are the same O(1/VT) and results on the last iterate are actually worse: OT331/3). In the strongly convex setting, the optimal rate ((1/T) rate is obtained both by the averaged and last iterates. However the averaged iterates are preferred

---

since they lead to the optimal covariance and the step size is independent of the strong convexity constant [23]. Finally, the final iterate of SGD, despite its ubiquitous use in machine learning, never theoretically performs as well as the averaged, except when used with geometrically-decayll learning-rate [15, 12].
Over-parameterized setting, As recalled earlier, SGD in the veeetparamttrizztionn regime corre- sponds the assumption that all the function gradients vanish at optimum. It has been first studied by [24] who assumed furthermore a strong growth condition (SGC) (first introduced by [29, 32]) but often too stringent to match the machine learning practical set-up. Up to our knowledge, [18] was the first to point out that this interpolation regime was particularly relevant in the recent deep learning framework and show linear convergence in the strongly-conwe setting. In the non-strongly convex case, O(1/T)) convergence has been shown by [33] for the averaged iterates. Consequently, there has been many papers discussing this setting, proving convergence rates in different contexts: Polyak-I Lojjssewijaiiiiicz [4), accelerated [33, 17], second-order [19], lene-search [34]. However, no rate for the final iterate has been shown in the non-strongly convex setting for the last iterate as the only convergence rates achieved, OL/TT), corresponds in all these works to the averaged iterates.
The Quadratic Case. More specifically, the quadratic case has been widely studied as a central model in the machine learning literature. As explained above, on the one hand, in the usual noisy setting, averaging or decaying step sizes have been considered, showing well-knowwn O(1/T) convergence rates [3]. The final iterate convergence of SGD for the noisy quadratic case is studied in [12] where they show (d log T/T) rate and advocate the primacy of exponential step decay. On the other hand, the SGD in werrparamettrizze setting has been considered only lately by [18, 6, 36). The first work corresponds to the strongly convex case, where the final iterate provably converges whereas our work is more aligned to [6, 36] where the non-strongly convex case is considered. In [21], the authors also studied the noiseless setting and provide worst and average cases asymptotic rates for the non-strongly convex case in the n, d + oo regime. In [6], among other results, it is shown that only the minimum of the function value iterates converge at rate (1/TT) and up to 011/T2+ for 0 when usual capacity and source conditions are assumed for the problem [8, 30J. A part of our work can be seen as a continuation of [6], strengthening its result by proving that the final iterate converges instead of the min some average.
Problem Set-up: Stochastic Gradient Descent on the Least-squares Problem
The setting is classical for stochastic gradient descent for linear least-squares in a Hilbert space H. The function we would like to minimize over 0 E H
1 E. (18, ((X)) - -Y)² R0) = 2
(1)
when p is the joint law over input/outputs (X, Y) E X R and & is feature map from X' to H. For the sake of simplicity, the reader can assume that H is a finite-dimensional Huclidean space. However, we take a special care that all the quantities and definitions could be valid whenever H is infinite dimensional. Hence, all the theorems are valid in a Reproducing Kernel Hilbert Spaces (RKHS) framework. Without any loss of generality, and to avoid heavy notations, we assume that ((X) = X, and as soon refer the the finite dimensional case, we set 7 Rdd
Covariance matrix. We define the covariance operator on H: H := E, @ X]. This positive semi-definin operator is diagonalizable and plays central role in our analysis. Hence, let us denote A its non- -oganeettvve eigenvalues sorted in non-increasing order and U, its corresponding cigenvectors. In finite dimension, the covariance operator is simply a d x d matrix H E, but, once again, notice that the analysis can also afford being infinite dimensional. In either case we denote Amax the largest eigenvalue of H. In the finite dimensional case Xmiin 0 is always correctly defined but can be arbitrarily small. On the opposite, a standard and important consequence of the infinite dimension is that, in most cases (especially when TrH < -0)), the infinite sequence of eigenvalues is no longer lower bounded as it converges to 0: H is no longer invertible and the problem is non-strongly convex.
Noiseless model. We assume that the model does not suffer from any noise: there exists 0, E H such that, p-almost surely, (0., .) = y. This means that the model is well specified and that there is no noise at optimum. Due to this noiseless condition, we expect our iterates to go to the optimum
3

---

without decaying step-sizes or averaging. We can also rewrite the Risk in Eq. (1):
1 (0 8.) H (e - 2 Tr (0 - 0.) 0.)* H R(0) =
(2)
Link between Noiseless model and Interpolation. Interpolation corresponds to the case where our predictor fit a finite set n of training data i.e., (6.. 22.) = yr for S n. Note that this setting is naturally included in our model replacing the test risk R(0) with the training loss whose finite sum structure can be rewritten B. ((0, . y)²² where ý is the empirical distribution of the data. Consequently our result on the final iterate convergence holds in this model for the training loss.
SGD with constant step-size. To minimize the function R defined in Eq. (I), we do not have access to the distribution p but to stream of i.i.d. observations (3a,,e)) Ve sampled from it. We hence perform a gradient descent in the direction given by one sample at a time with constant stepsize y 0. Throughout all this paper, the initial condition is always set 0g and for > 1,
dc+1 0 - 7 ((01, yc) Zt.
(3)
The impossibility of linear rates. We stress that the least-squares setup consider is non- strongly convex problem. Indeed, in finite dimension d, Amin > O, and SGD converges at linear rate -- [18, Theorem 1], this asymptotic regime occurring after time scale 7 ~ 1/(7>min). However, this apparent strong convexity is a lure, since the large (or infinite) dimension makes this convergence rate vacuous for arbitrarily small Amin- Hence we focus on the non-parametric non-strongly convex setup where we prove non-asymptotic polynomial rates. Yet in finite dimension, it is always possible see the linear regime after time 1/(77min) as shown in Figure 1, even if this time can be arbitrarily long.
3 Convergence rates of the final iterate of SGD
In large dimension, two quantities govern principally the rates of convergence of least-squares estimators: (i) the spectrum of the covariance matrix, and (ii) how the solution, 0., projects on the eigenbasis of the covariance matrix. In this section, we state refined assumptions on the spectrum of the covariance matrix and the decomposition of 0, over its eigenbasis. Note that in finite-dimension, all these quantities are always finite, but be extremely large compared to the sample size T. This is why the reader can see these more as a fine- fine-grained parameterization of the problem rather than restrictive assumptions. The assumptions we make always go in pairs, (i) one for the features through the covariance matrix (Assumptions I, 3 and 5) and (ii) one for the target solution (Assumptions 2, 4 and 6).
Summary on the results. As assumptions go stricter, the convergence rates go faster. Every theorem is a bound on the expected risk given by the last iterate of the SGD recursion with constant step size y, started at fg = 0. Only for Theorem 1, for which the assumptions are the weakest, we allow the the finite horizon T (through its logarithm). All the theorems respect to finite constants defined thanks to the different assumptions. The reader can refer to Table 1 for concise summary of them. Note also that we put particular effort for the clarity of the bounds, and hence, some numerical constants might appear large. These are simple artifacts on the proofs and could be lowered, but the price of less clear results. A more detailed summary of our results can be found in Appendix 7, All the proofs are deferred to Appendix 9.
step-size to depend
stated with
3.1 Standard Least-Squares setting
As often, when we analyze SGD, we make a 4-th order assumption on the distribution of the features.
Assumption (Fourth Moment Condition)
There exists a finite constant 2 O, such that
E
| xx_] $ RH
(4)
The assumption holds in the case of bounded features, i.e. when ||X|| < R. px-almost surely. also holds more generally for features with infinite supports, such sub- Gaussaassi data, and canonical basis distributions (i.e. x = e, with probability p.). This is a standard assumption when analysing SGD for least squares [3, 14, 12] which is weaker than what is assumed in [10, 11, 36].

---

Theorem
Assumption
Condition
Rate
A.
E
X
XX
x RH
In(T)
Theorem 1
A. 2
110.1
to0
T
ft
A.
E
[X, In (H (H- xx x Rn
Theorem 2
A. 4
Tr In(H-1)
to0
T
ft
E H 0/2 x/2x| || Xx 5 RaH, a E (0,1) Mr F) +o0
A. 5
Theorem 3
A.
300
+
Table 1: Table showing the main results of the article: different upper- upper-bounde soper-bounds for the convergence of the SGD final iterate determined by different assumptions.
Assumption (Attainable case) The target solution 0, lies in the space H. This ensures that has finite norm ||0 ll < +00.
While this is always true in finite dimension, this assumption draws the attention on the fact that the norm of the optimum 0- could be very large in high-dimension. As limit, in infiite-dimensionall spaces, 9. could not belong to H and hence would have an infinite norm. This is why we refer to this assumption as the attainable case. Under these two assumptions we have the following result.
Theorem Assume Assumptions 1, 2. Then, for T > 2, if we set (nT) following bound for the expected risk of the estimator given by the Tth iterate of SGD:
have the
ER(Or) < 3 (T) T
(5)
Let us comments upon this result proven in Appendix 9.1. The theorem above states that, under mild assumptions, if we allow the step-size to depend the time horizon, have (In(T)/T) convergence rate for the final iterate. First, removing the dependence on finite horizon for the step-size by considering decaying step-sizes "/ x 1 In(t) could be done, but decided to keep this way as we have focused on constant step-size in this paper. Furthermore. Theorem of [6] shows that this bound is optimal up to log (T') for SGD, This rate can also be compared the classical optimization results for the non-strongly convex objective (our case). is well known that gradient descent and averaged SGD (even for non-quadratic objectives) achieve (1/TT) rate with constant step size. Similarly [6] achieves a convergence rate of (1//T) rate with constant step size for the min of the function value along the iterates. However, the rate of convergence of SGD last iterate was open problem [see 6, Remark 1]. Note however that our bound suffers from a log(T") both in function value and for the step-size. Whether this term is necessary an open question for us. The purpose of the following development is to first remove these log(T) dependence at the price of log-scale refinement of the assumptions.
3.2 A logarithm-scale refinement
This second sequence of assumptions is slightly stronger. They reinforce assumptions 1 and 2 at the log-scale. Once again, there is one on the features and the other one on the target.
Assumption (log-regularity of the features) There exists constant 7o O and Rin such that the covariance matrix H satisfies the following condition:
E [[X, (No XX R. H.
Note that A is a just reference which lets us present cleaner results. We choose A, such that 7. Ag. This is always possible Ri, scales linearly with In X, (see details in 7.1). This condition implies in fact Tr In(2,H-1)) & Rin which consequently implies the following eigenvalue decay: X & ((1((in i)) (see details 7.1). This is the reason why we say that this assumption refines Assumption at log-scale.
5

---

Assumption rlog-regllrrity of the optimum) The covariance matrix at optimum Mo = 0.6. satisfies the following condition:
Tr (Mo in (1I +00.
We define Cn :
: E, In(.//X)))) with the same reference Ao To give order of magnitude in finite dimensions, when the spectrum of the covariance matrix lower bounded by Amin > 0, Cin can be estimated Cin ||0,11 1/ In Ao/ Amin all0, ll In d even if Amin is small 1/d". When the dimension is not too big (even at the log-scale), this constant Cin is comparable with ||0. ll up to log factors. Under these two assumptions we have the following convergence result.
Theorem 2 Assume Assumptions 3,4. Then, for T T> 3, if we set (14Rka) we have the following bound for the expected risk of the estimator given by the T iterate of SGD:
ER(0,) 1
This theorem, proven in Appendix 9.2, states that at the price of a log-scale refinement on the features and optimum, the GGD-eonverge rate (1/TT). This naturally restricts the class of problems that suffers from a log(T") (see Theorem 1) to a very small class: roughly speaking, it is the class of problems for which the eigenvalue decreasing rate is strictly squeezed between: (i1/(i In < À < O1/i). The role of the assumptions is fundamental here: Assumption 3 allows to remove the In(T) for the step-size and Assumption allows to remove the (T( for the convergence rate. The proof technique is the same as for the previous theorem, the only difference is that the assumptions allow to control more precisely the bias and the variance in the SGD recursion.
3.3 A fine-grained parameterization of the problem: capacity and source conditions
The final set of assumptions have been introduced by [8]. They are in the same vein as above assumptions and are often called capacity and source conditions the reproducing kernel Hilbert spaces community.
Assumption (capacity condition: a-regularity of the features)) The covariance matrix H is such that there exists > 0 and finite constant R. > O verifying
xx & R.H. E XX,
Note that Assumption is strictly more demanding than Assumption 3 and has been named regularity of the features in [6, Remark 3]. Note also that a + 0 corresponds to Assumption I and the larger the a the stricter the assumption is. The capacity condition stated above implies 'r( < Rg- which consequently implies an eigenvalue decay as a power law for the sequence of eigenvalue of H: A; O(1/ir==) (see details in 7.1). It is also often related to the effective dimension of the problem [8]. Finally, as a limiting case, when & -> 1, R. - Tr(Id) = d and the assumption is only valid in finite dimension. Hence, is expected that the bound would blow in large dimension when & grows to one. For this reason, in the literature [8, 30), the bound is often reparameterized as a' = (1 a) -1 push this singularity at infinity, We decide keep our parameterization for the clarity of the result. The same type of assumption can be stated for the optimum.
Assumption (source condition: 8-regularity of the optimum)) The covariance matrix at opti- mum Mo 0_0__ is such that there exists -1 such that Tr(MoH t00. Here, define
C3 = Tr TT = Σ > m m:
Here important remarks this assumption. First note that
Tr,M, 8) = Tr H 0. - -B
Hence, 3 = 0 corresponds to Assumption 2 which is the attainable case. It is worth noting that for 3 E (-1, 0), this assumption takes into account the fact that O, does not necessarily belong to H. In such case, it is still possible to define properly 0, the infimum over H of the risk R defined in

---

(q. However, for sake of clarity and to avoid cumbersome technicalities, we refer to [9, p. 1.1355] for the extension of the analysis is this case. As before, the larger the 3 the stricter the assumption is. Finally as a limiting case, / - +00 corresponds to the fact that 0. belongs to finite dimensional space. This hypothesis is often called the source condition in the literature because quantifies the complexity of the optimum [see 8, 30, 20, for further details]. Once again, parameterization has not yet been fixed by the literature and one often uses the parameter r that corresponds to 2+1 We 2
have the following theorem under these capacity and source conditions.
Theorem 3 Assume Assumptions 5, 6 with constants a E (0, 1) and ß > -1 respectively. Then, for T 3 3, we have the following bound for the expected risk of the Tth iterate of SGD:
1
+
3
1+8
1
BR(OT) & 2Cg
?
TI+aA8:
where 71-a aa-a & (32EaRa) and Ea-
Σ n²l
1
2+a
Let us comment this result. Its proof can be found in Appendix 9.3.
Discussion on the limit cases for a and 3. Let us note a few observations on the two parameters a and B. Firstly, it is observed in [6] that the rate 1/T is optimal in terms of power law for SGD. Second, the convergence rate depends on the minimum of the two. It implics that either the regularity of the features, or the one of the optimum, is bottleneck for the convergence of SGD. More precisely if a < B, the features are not regular enough to counter the multiplicative noise of SGD. Conversely, when the optimum is the bottleneck, there is no difference between SGD and Gradient Descent (GD), as +1 is the rate of convergence of GD. Note that a - 0 corresponds to the setting of Theorem 1 where a log(T)) appears. Hence, it is normal that our bound does not hold in this limit: indeed, we remark that the step-size shrinks to 0 as &,x goes to infinity. Another interesting limit is the case where - 1: as said before this cannot occur in infinite (or large) dimension bound blows up in this limit also. The fact that the step-size depends on a is weakness of our result and is due the mixing power of the covariance eigenspaces. This dependence could be eliminated with an extra assumption on a lower bound on the decrease rate of the spectrum of the covariance matrix as made by [8]. Finally note that the rate of convergence is always strictly better than 1/T for B >> 0 and is remarkably adaptive to the possible misspecification of the problem 33 E (-1,0).
R. +"d" Therefore the
Comparison with the literature. We can first compare to the results on the noisy setting studied by [9]. Naturally, when some additive noise is assumed, averaging is necessary, and the results are weaker: 7 is not adaptive to the problem, depends on the time-horizon and the rates are always slower than ((/TT). However, when the noise is 0, the averaged iterates achieves the exact same convergence rate as in our theorem. The closest results to our theorem are in [6]. where the same rates are shown. The important difference is that their results are given for the min of the function value and not the final iterate: our theorem solves an open problem stated in the aforementioned paper. Remark however that one superiority of [6] is that the step-size does not depend on a showing some adaptivity with this parameter. As noted in the previous paragraph, we could fix this difference with an extra assumption on the spectrum of the covariance matrix. Finally, rates of convergence for the noiseless setting have been addressed by [16] with some truncated version of the kernel ridge estimator. It was an open problem stated by the author to understand whether SGD could achieve these par-mmrmetrric rates. As for the min, this problem seems to be properly solved now. A main difference is that the bounds of [16] suffer from some saturation in the well specified setting, when > O, whereas our estimator does not, When a - 1 & B & O. then the bounds match, but when the problem is really misspecified, for 3 & a 1, the bound of [16] is strictly better than ours. As we know that our result is optimal for SGD, it raises the interesting question whether some form of acceleration [10] multiple passes over the data [22] could reach these rates.
Link with kernel regression. These capacity and source conditions are classical in the reproducing kernel Hilbert spaces community. For sake of clearness, we did not mention any specificity on the features. However, as has been done [31, 9], SGD can "kernelized" and the descent becomes an optimization algorithm in the RKHS space of functions. In this context, the capacity and source conditions take on their full meaning: when RKHS are Sobolev spaces, they represent a smoothness condition on the features and the optimum respectively. The coefficient would be the degree of
7

---

regularity that we choose for the kernel, and /8 the prior we have on the optimum solution. For more details, we refer to section 4 of [22] or section 3.1 of [6].
Development of the SGD recursions and proof outline
In this section we provide an overview of the arguments that comprise the proof of our results.
Recursions for multiplicative noise. We can rewrite the SGD recursion Eq. (3) for the deviation to the optimum 7)t == O2 02 as a descent on the risk plus a multiplicative noise term. For > 1,
M++ = (1 - -) the = (1 - -1) n + + (H - x(x7] 1)1
(8)
We can also write the recursion satisfied by the covariance of the iterates, M, == EE [nm
M41 = (1 - H) M, (T 7H + 7 [(H - ).Fe M2 (H - x... }]
Note a main difference between the two recursions, Eq. (9) is a deterministic recursion over operators whereas Eq. (8) a stochastic recursion on vectors.
Deviations sequence and initial conditions. We emphasis that and M, represent deviations to the optimum 0.. This is why proving that 0, goes to O. corresponds to the and M, going 0. As the initial point of the SGD recursion is 0g = O, the initial conditions o/- and Mz-o represent, in fact, respectively the optimum and the covariance at optimum.
Eigenspaces of the covariance. The core of the analysis rests on a reformulation of our problem in the eigenspaces of the covariance operator. Recall that (Aa, 04); are the sorted eigenclements of H and define the decomposition of M, in the basis of t.uo T. M, := E; mmv,a, + Eins miyu We can write the expected risk of the estimator given by the t-th iterate of SGD, that we denote by fŞ:
== (R( Tr (MH) Exm, Amp.
(10)
It is quite remarkable that the (mag)is) j do not play any role in the recursions. The aim now is to write recursion for (m²); that leads to a recursion for the function value f, through Eq. (10). This is the purpose of the following and central lemma, whose proof can be found in Appendix 8.1.
Lemma 4 (Recursion on the covariance of iterates) Define f == E (oo, X)² X M.X]
Fort t > 1, we have the following recursion on the covariance of the iterates of SGD, Vi,
m
[+1 = m! - 27),,m4 + 2
(11)
m ++1 = -1 (1-2 m 2 Σ (1 27A,) $7
(12)
k=0
Let us make comments on this important lemma. First, note that the only difference with the deterministic recursion (gradient descent) is the presence of the "mixing terms" (f)),. In fact, these terms affect dramatically the dynamics. Indeed, there is no reason that the iterates decrease along the iterations: this is what makes the analysis more tedious for the final iterate. On the contrary, the usual convergence rate for the averaged iterates easily obtained by summing Eq. (11) and comparing the terms. Note finally that the strength of Eq. (11) lies in the fact that the different eigenspaces (e.g. mț and m for * j) interact only through ft and fz. By summing Eq. (12), and appropriately controlling sums of f! with Assumptions 1, or 5, we can get recursive inequalities on f. These show that f, is in fact a Lyapunov function; going further, inequalities to finally upper-bound it. To exemplify this reasoning, we present, in the following lemma, Lyapunov control on f, using Assumptions 1. Its proof can be found in Appendix 8.2.
use discrete versions of Gronwall-type
Lemma 5 (Recursion on f) Under Assumption I, assume 7y & (4Amax) -1 For t we have the following recursion on the function value fr. For all i,
Tr(Mo) 4tt
fp t
f, <
7
(13)
k=U
8

---

71
10t d = 300 a = 0.5 3 =
n1
10t
300 a = 0.75 3
10
10
R
R
10
1
10
Last iterate
iterate
Averaged iterate
Averaged Iterate
10
10
10
10*
10
100
102
10*
10
Number of iterations
Number of iterations
Figure 1: Least-squares regression. Left: 0.5 and / The vertical dashed line marks the transition to the linear convergence regime. Right: 0.75 and 3 1. The orange dashed line
represents the curve 1/T1+anß predicted by Theorem 3.
The decrease of the function value ft is controlled by the sum of a bias term -characterizing how fast the initial conditions m. are forgotten-, and variance term -characterizing how the noise reverberates through the iterates. All the theorems are proven using the same technique: the aim to use the inequality recursively to control the variance term. The different Assumptions 3, lead to different variance term. Trying to factorize the proof' in one general bound does not allow for a clear presentation of the results. This why, despite the apparent redundancy of the proof technique, for the sake of clarity and easy reading, preferred to split the different proofs of the theorem and factorize only certain technical lemmas. All the proofs of Theorems 1, 2, can be found in Appendix 9, respectively in Sections 9.2, 9.3.
5 Experiments
We illustrate our theoretical results on a synthetic least-squares problem using the SGD algorithm defined in Eq. (3). For d 300 consider a stream of normally distributed inputs (Ir,), whose covariance matrix H has random eigenvectors i; and eigenvalues 1 1/²1/(1-a) for = 1,.... d. The
optimum is chosen randomly: 0, = E 1//1i-/ 0y(1-a; the coefficient a and of the capacity and source conditions are perfectly controlled. The outputs (yn),, are generated through Yn = rx)) We take step-size 7 = 2TTH- All results are averaged over ten repetitions. We compare the performance of the last iterate and the averaged iterate of SGD on two different problems: one corresponding to 0.5 and B = 0 and the other to a = 0.75 and 1. For those problems, the predicted convergence given by Theorem ( ( Hence we expect 1/T convergence in the first example and fast 1/7 rates for the second.
t. This allows to reproduce the setting where
First note that the bound of Theorem 3 is perfectly matched by these two examples. Second, can see that, the averaged SGD and the final iterate show the same behavior quite accordingly to the theory (see comments on Theorem 3) even if we can notice slight but real- better performance (result are shown in log-scale) from the final iterate. However, the main difference is that, as the averaged iterates show some saturation, the final iterate meets a point where it changes convergence regime (left plot in Figure 1). This adaptivity of the final iterate appearing after time scale 1/(7>min) represents true asset in choosing the final iterate versus the averaged one. Finally notice that the linear rate regime cannot be seen in the right plot as the maximum number of iterations shown, n 10%, is negligible when compared to d!((1-)) 3000/(1-0000 ~ 10e0
linear
6 Conclusion and Perspectives
In this paper, we proceed to a detailed analysis of convergence rates of SGD the noiseless least-squares setting. We derive systematic study of the SGD last iterate that leverages a sequence of fine-gran ene-ggained assumptions allowing us to exhibit fast polynomial rates. The absence of additive noise changes dramatically the behavior of SGD: no decaying step sizes averaging are needed for

---

convergence as constant step-size SGD naturally adapts to the problem. The development of the SGD recursion as an interacting particle systems open many perspectives: can be used to analyze SGD accelerations? multiple passes over the data? Another significant result is that, from an optimization perspective, we give a prototypal example where the last iterate of constant step-size SGD provably converges in the non-strongly convex case. This raises the following fundamental question: can we extend our method to show the convergence of SGD last iterate the general non-strongly convex case?
References
[1] D. Aldous and D. Lanoue. A lecture on the averaging process. Probability Surveys, (none):90
- -102, 2012.
[2] F. Bach and E. Moulines, Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in Neural Information Processing Systems, pages 2011.
[3] F. Bach and E. Moulines. Non-strongly-convee smooth stochastic approximation with conver- gence rate o(Inn). Advances in Neural Information Processing Systems, 2013.
[4] R. Bassily, M. Belkin, and S. Ma. On exponential convergence of sgd non-convex over- parametrized learning. arXiv preprint arXiv: 1188/.0222 .00564, 2018.
[5] M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical varianee trade-off. Proceedings of the National Academy of Sciences, 116 (32):15849-155 15854, 2019.
[6] R. Berthier, F. Bach, and P. Gaillard. Tight nonparametric convergence rates for stochastic gradient descent under the noiseless linear model. Advances Neural Information Processing Systems, 2020.
[7] L. Bottou and O. Bousquet. The tradeoff's of large scale learning. In Advances in Neural Information Processing Systems, 2008
[8] A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-368, 2007.
[9] A. Dieuleveut and F. Bach. Nonparametric stochastic approximation with large step-sizes. Annals of Statistics, 44(4):1363-1399) 2016.
[10] A. Dieuleveut, N. Flammarion, and F. Bach. Harder, better, faster, stronger convergence rates for least-squares regression. The Journal of Machine Learning Research, 18(1):5520-3570 2017.
[11] N. Flammarion and F. Bach. Stochastic composite least-squares regression with convergence rate (1/m.). Conference Learning Theory, pages 831-875 2017
[12] R. Ge, S. M. Kakade, R. Kidambi, and P. Netrapalli. The step decay schedule: near opti- mal, geometrically decaying learning rate procedure for least squares. Advances in Neural Information Processing Systems, 32: 14977-19988, 2019.
[13] N. A. Harvey, C. Liaw, Y. Plan, and S. Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory, pages 1579-1613, PMLR, 2019.
[14] P. Jain, S. M. Kakade, R. Kidambi, P. Netrapalli, and A. Sidford. Accelerating stochastic gradient descent for least squares regression. Conference On Learning Theory, pages 545 455, 2018
[15] Jain, D. Nagaraj, and P. Netrapalli. Making the last iterate of sgd information theoretically optimal. In Conference on Learning Theory, pages 1752-1755 PMLR, 2019.
[16] K. Jun, A. Cutkosky, and F. Orabona. Kernel truncated randomized ridge regression: optimal rates and low noise acceleration. Advances in neural information processing systems, 32, 2019.
[17] C. Liu and M. Belkin. Accelerating sgd with momentum for over-parameterized learning. In International Conference on Learning Representutions, 2020.
[18] S. Ma, R. Bassily, and M. Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern peraparametrized learning. International Conference on Machine Learning, pages 3334.3334. PMLR, 2018.
10

---

[19] S. Y. Meng, S. Vaswani, I. H. Laradji, M. Schmidt, and S. Lacoste-Julien. Fast and furious convergence: stochastic second order methods under interpolation. In International Conference on Artificial Intelligence and Statistics, pages 1375-1386. PMLR, 2020.
[20] N. Mücke and E. Reiss. Stochastic gradient descent in Hilbert scales: smoothness, precondi- tioning and earlier stopping. CoRR, abs/2006. 10840, 2020.
[21] C. Paquette. K. Lee, F. Pedregosa, and E. Paquette. SGD in the large: Average-case analysis. asymptotics, and stepsize criticality. In Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder. Colorado, USA, volume 134, pages 1548-3626. PMLR, 2021.
[22] I.. Pillaud-Vivien A. Rudi, and F. Bach. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems, 31:81 14 8124, 2018.
[23] B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging, SIAM journal on control and optimization, 30(4):838- 855, 1992.
[24] M. Schmidt and N. L. Roux. Fast convergence of stochastic gradient descent under strong growth condition. arXiv preprint 138.63300 2013.
[25] B. Schölkopf and A. Smola. Learning with Kernels. MIT Press, 2002.
[26] O. Shamir. Open problem: is averaging needed for strongly convex stochastic gradient descent? In Proceedings of the 25th Annual Conference on Learning Theory, volume 23 of Proceedings of Machine Learning Research, pages 47.1-47.3, JMLR Workshop and Conference Proceedings. 25-27 Jun 2012.
[27] O. Shamir and T.. Zhang. Stochastic gradient descent for non-smooth optimization: convergence results and optimal averaging schemes. In International Conference on Machine Learning, pages 71-79, 2013.
[28] Thhwee-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
[29] M. V. Solodov. Incremental gradient algorithms with stepsizes bounded away from zero. Computational Optimization and Applications, 11(1):23-35, 1998.
[30] I. Steinwart, D. R. Hush, and C. Scovel. Optimal rates for regularized least squares regression. In Conference On Learning Theory. PMLR, 2009.
[31] P. Tarres and Y. Yao. Online learning as stochastic approximation of regularization paths: optimality and almost-sure convergence. IEEE Trans. Inf. Theory, (99:5516--5733 2014.
[32] P. Tseng, An incremental stepsize rule. SIAM Journal on Optimization, 8(2):506-555 531, 1998.
method with momentum term and adaptive
[33] S. Vaswani, F. Bach, and M. Schmidt. Fast and faster convergence of sgd for paramearmeteriize models and an accelerated perceptron. In International Conference on Artificial Intelligence and Statistics, pages 1194-1204. PMLR, 2019.
[34] S. Vaswani, A. Mishkin, I. Laradji, M. Schmidt, G. Gidel, and S. Lacoste-Julien. Painless stochastic gradient: interpolation, line-search, and convergence rates. In Advances in Neural Information Processing Systems, pages 3727-3740, 2019.
[35] T. Zhang. Solving large scale linear prediction problems using stochastic gradient descent algorithms. International Conference on Machine Learning, 2004.
[36] D. Zou, J. Wu, V. Braverman, Q Gu, and M. Kakade, Benign overfitting of constant-stepsize sgd for linear regression. COLT, 2021.
11


Reviews:
[{"review_id": "pvBSopsbAgL", "reviewer": "Reviewer_Mkow", "summary": "The paper explores the last iterate convergence of SGD on the least squares problem in the interpolation regime (noiseless setting). The authors work in an abstract Hilbert space and thus consider (1) the non-strongly convex setting and (2) the streaming or online setting. There are three contributions of the paper:\n\n(1). The final iterate of constant step-size SGD achieves a convergence rate of $O(ln(T)/T)$ under the assumption that the data matrix $X$ has (in expectation) bounded fourth moments. \n\n(2). Under more assumption about the behavior of the eigenvalues of the covariance matrix of the data, that is $\\lambda_i \\le O(1/i \\log(i))$, the authors can improve the last iterate convergence rate to $O(1/T)$. This matches the rate for the averaged iterates of SGD.\n\n(3). Under stronger assumptions of the eigenvalues of the covariance matrix at the optimum (source) and capacity condition, one can get an even faster rate of convergence. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the details of the exact assumptions are clearly stated and there are enough details to follow the main ideas of the results. Although I did not check the proofs in detail, I am confident that they are correct. One concern about this paper is that it is fully devoted to trying to remove a log factor, which I understand is of interest theoretically, but of practical interest one may not even see. It is not entirely clear that one could even remove the log factor in the general setting. Can one show in simulated data that the $1/T$ is achieved?\n\nI would also suggest that the authors elucidate on the $\\alpha$ and $\\beta$. While I understand that these assumptions have been used in previous works, it is not clear what are \"natural choices\" for these parameters. For instance in the case of $X$ having i.i.d. Gaussian entries of mean $0,1$ and $\\theta^*$ pure noise, I believe one could compute the values of $\\alpha$ and $\\beta$. Could the authors comment on this? \n\nThere is also some recent work on last iterate convergence in the multi-pass setting (see Paquette, et al. \"SGD in the Large\") and could the authors comment on comparisons with this work (to match the streaming setting $r = \\infty$)?\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the results are clearly stated. This result only applies to the least-squares problem and in the streaming setting. The results are purely theoretically. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "PG5zJQOjvY", "reviewer": "Reviewer_ttkk", "summary": "This paper studies the problem of minimizing a non-strongly convex Least-Squares using stochastic gradient descent (SGD). The authors show that when the interpolation assumption holds, then under mild assumptions, the last iterate of SGD converge to the optimal solution with rate $\\mathcal{O}(\\frac{\\log(T)}{T})$. And under stronger assumptions, the convergence rate can be improved to $\\mathcal{O}(\\frac{1}{T})$. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the manuscript to be clearly written and technically sound.  However, there are still a few points need further clarification:\n1. One important setting in this paper is that the covariance operator $H = \\mathbb{E}[X \\times X]$ is not positive definite, namely the least square is not strongly convex. However, in Assumption 1, it assumes that $H \\succeq \\frac{\\mathbb{E}[\\|X\\|^2 XX^T]}{R}$. Does it mean that the covariance operator is actually positive definite? Same question for Assumption 3. \n2. In the discussion after Assumption 4, you have the condition $\\lambda_{\\min} > 0$. I think this is also equivalent to assuming the problem is strongly convex.\n\nBased on the discussion above, I think although there is some novelty in the problem setting and the corresponding theoretical result. The contribution may not be enough provided there are already some result about the convergence of last iterate from the SGD on strongly convex functions. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no negative societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hrs"}, "extra_text": {}}, {"review_id": "9fURmmRF8u7", "reviewer": "Reviewer_vd3S", "summary": "This paper investigates the excess risk bounds of constant-stepsize SGD last iterate in the setting of noise-less least squares (i.e., only considering the bias error). The presented bounds could be applied to overparameterized cases, but are also limited to attainable cases. The presented bounds for the last iterate nearly match the established ones for the averaged iterates. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Pros:\n+ The presented bounds for the last iterates of SGD are the first results that can be applied in the overparameterized setting.\n+ Some of the presented bounds are (nearly) tight and (partly) solve an open question for the rate of SGD last iterate.\n+ Lemma 5 is interesting in its own for bounding the SGD (bias) iterates.\n+ Overall the paper is organized well, but the writing could be improved here and there.\n\n# Cons:\n- The considered least square problems are noiseless, i.e., the variance error bound is not considered. In fact I am thinking that using a large and constant stepsize and without averaging, the variance term cannot be decreased. In this sense the presented results and techniques are somehow less satisfactory.\n- The presented bound, e.g., Thm1, requires the true parameter to have a finite $\\ell_2$-norm (attainable case), which is limited and could potentially be relaxed to an even more interesting version like $(\\theta^*)^\\top H \\theta^* < \\infty$. \n- The writing could be improved in several places. See below.\n\n# Small issues:\n* The paper/section/paragraph title capitalization style should be made consistent. \n* l.7. two folds.\n* l.25. This sentence is quite confusing. Note that a noisy model could also interpolate. See e.g., [bartlett et al 2020].\n* l.37. To be more clear: \"... a decreasing sequence in terms of loss\"\n* l.47. an infinite..\n* l.48. This is also confusing, as strongly convex cases are covered by the presented results. Better to write, e.g., \"the problem we are considering is not restricted to strongly convex or finite-dim cases\".\n* l.49. the following two.\n* l.55. Everything is known for noisy setting (which covers noiseless setting)? Perhaps you mean the finite-dim setting?\n* l.68. This is too confusing... Lots of papers listed in paragraph l.76 explicitly or implicitly discussed how to control SGD last iterate. I agree this paper has some new results for SGD last iterate but the claim here is too exaggerative!\n* related works section is quite messy. Perhaps the authors could discuss papers for offline/multi-pass SGD and online/one-pass SGD separately? These two are quite different in terms of generalization bounds. Also, there are big paper overlaps in the three paragraphs & could find better ways to organize.\n* l.144. an arbitrarily...\n* l.149. Not only in large dim case, but in any case. \n* l.168. a 4th...\n* l.173. at least some results in [10] only assume eq (4) instead of something stronger.\n* l.203. $C_{ln}$ is not defined. I guess it refers to left hand side of the above inequality?\n* l.227. an eigenvalue...\n* l.321. remove the empty space before ?\n\n# Overall\nBased on my knowledge on papers for SGD last iterate, I think the presented results have their spirits and could be a good reading for controlling SGD last iterate. I would love to see further discussions/results to relax the finite $\\ell_2$-norm (of the true parameter) condition, as the $\\ell_2$-norm easily blows up in infinite-dim cases. Discussions on how to control the variance error could also be useful. As for the writing, there are plenty places need to revised though. My current evaluation for this paper is neutral and slightly lean to weak acceptance.\n\n\n\n[bartlett et al 2020] Bartlett, Peter L., et al. \"Benign overfitting in linear regression.\" Proceedings of the National Academy of Sciences 117.48 (2020): 30063-30070.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "517A9zMpndW", "reviewer": "Reviewer_cbwB", "summary": "This paper studies the least-squares problem in the interpolation regime, solved by the last iterate of the Stochastic Gradient Descent. It characterizes the performance of SGD under 3 sets of assumptions and shows the explicit impact of the conditions on the convergence rate.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not very well familiar with the literature on the infinite-dimensional least-squares problem. My background is mainly in the field of convex optimization, so I will make the comments mainly from this perspective.\nFirstly, the assumptions seem strange and hardly interpretable, and I think that the paper can benefit from drawing connections to general convex stochastic optimization. At least it would be helpful for non-experts in this area because, for now, it seems targeted at a very narrow audience.\n\nCould you please tell me what is the purpose of Assumption 2, if the noiseless problem statement includes the existence of $\\theta_* \\in \\mathcal{H}$ such that $\\<\\theta_*, x> = y$.\n\nHow can you use the $H^{-1}$ in Assumption 3 and further, if on page 3 (line 131) it is said that *“$H$ is no longer invertible”*?\n\nThe paper is clearly written, and the “related work” section seems complete. Though, excessive details ultimately make it harder to get the broad picture, distinguish the paper's contributions, and compare it to the previous results (especially in terms of technical novelty).\n\nTo be honest, I do not quite get the point of considering the infinite-dimensional case, and the motivation presented by the authors does not seem convincing to me. That is why it is hard to see if the results are valuable for the machine learning community.\n\nFrom the practical viewpoint, the paper seems quite irrelevant, as I can not see any helpful insights for practitioners. The synthetic experiments are very basic, although this is clearly not the main focus of the work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable due to theoretical character of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proves the risk bound of the last iterate for constant step size SGD in the interpolation regime. The main concern from the reviewers is that the linear regression model is assumed to be noiseless, which makes the results less interesting. After the author response and reviewer discussion, the paper gathers enough support from the reviewers. Thus, I recommend acceptance. 
Decision: Accept (Poster)