Title: The Sample Complexity of Gradient Descent in Stochastic Convex Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Stochastic Convex Optimization

Learning Theory
Abstract: We analyze the sample complexity of full-batch Gradient Descent (GD) in the setup of non-smooth Stochastic Convex Optimization. We show that the generalization error of GD, with common choice of hyper-parameters, can be $\tilde \Theta(d/m+1/\sqrt{m})$, where d is the dimension and m is the sample size. This matches the sample complexity of \emph{worst-case} empirical risk minimizers. That means that, in contrast with other algorithms, GD has no advantage over naive ERMs. Our bound follows from a new generalization bound that depends on both the dimension as well as the learning rate and number of iterations. Our bound also shows that, for general hyper-parameters, when the dimension is strictly larger than number of samples, $T=\Omega(1/\epsilon^4)$ iterations are necessary to avoid overfitting. This resolves an open problem by Schlisserman et al.23 and Amir er Al.21, and improves over previous lower bounds that demonstrated that the sample size must be at least square root of the dimension.

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization

Abstract

We analyze the sample complexity of full-batch Gradient Descent (GD) in the setup of non-smooth Stochastic Convex Optimization. We show that the generalization error of GD, with common choice of hyper-parameters, can be ˜ Θ ( 𝑑 / 𝑚 + 1 / √ 𝑚 ) , where 𝑑 is the dimension and 𝑚 is the sample size. This matches the sample complexity of worst-case empirical risk minimizers. That means that, in contrast with other algorithms, GD has no advantage over naive ERMs. Our bound follows from a new generalization bound that depends on both the dimension as well as the learning rate and number of iterations. Our bound also shows that, for general hyper-parameters, when the dimension is strictly larger than number of samples, 𝑇 = Ω ( 1 / ε 4 ) iterations are necessary to avoid overfitting. This resolves an open problem by Amir, Koren, and Livni [3], Schliserman, Sherman, and Koren [20], and improves over previous lower bounds that demonstrated that the sample size must be at least square root of the dimension.

1 Introduction

Stochastic Convex Optimization (SCO) is a theoretical model that depicts a learner that minimizes a (Lipschitz) convex function, given finite noisy observations of the objective [22]. While often considered simplistic, in recent years SCO has become a focus of theoretical research, partly, because of its importance to the study of first-order optimization methods. But, also, it has become focus of study because it is one of few theoretical settings that exhibit overparameterized learning . In more detail, classical learning theory often focuses on the tension between number of samples, or training data, and the complexity of the model to be learnt. A common wisdom of classical theories [1, 7, 14, 24] is that, to avoid overfitting, the complexity of a model should be adjusted in proportion to the amount of training data. However, recent advances in Machine Learning have challenged this viewpoint. Evidently [18, 25], state-of-the-art algorithms generalize well but without, explicitly, controlling the capacity of the model to be learnt. In turn, today, it is one of the most emerging challenges, for learning theory, to understand learnability when the number of parameters in a learnt model exceeds the number of examples, and when, seemingly, nothing withholds the algorithm from overfitting.

Towards this, we look at SCO. In SCO, Shalev-Shwartz, Shamir, Srebro, and Sridharan [22] showed how algorithms can overfit with dimension dependent sample size. But, at the same time, it was known [8, 26] that there are algorithms that provably avoid overfitting with far fewer examples than dimensions. As such, SCO became a canonical model to study how a well-designed algorithm can avoid overfitting even when the number of examples is too small to guarantee generalization by an algorithmic-independent argument [2-5, 11, 12, 16, 20-22]. A step towards understanding what induces generalization is to identify which algorithms generalize. Then, we can ask what yields the separation. Surprisingly, for many well-studied algorithms this question is not always answered.

Perhaps the simplest algorithm, whose sample complexity is not yet understood, is Gradient Descent (GD). And we turn to the basic question of the sample complexity of gradient descent.

While this question remained open, there have been several advancements and intermediate answers: The first, dimension independent, generalization bound was given by Bassily, Feldman, Guzmán, and Talwar [6] that provided stability bounds [8]. The result of Bassily et al. demonstrated that, GD can have dimension-independent sample complexity rate. However, to achieve that, one has to use non-standard choice of hyperparameters which affects the efficiency of the algorithm. In particular, the number of rounds becomes quadratic in the size of the sample (as opposed to linear, with standard choice). On the other hand, a classical covering argument shows that linear dependence in the dimension is the worst possible, for any empirical risk minimizer, irrespective of properties such as stability.

In terms of lower bounds, Amir, Koren, and Livni [3] were the first to show that GD may have a dimension dependence in the sample complexity. They showed that, with natural hyperparameters, the algorithm must observe number of samples that is at least logarithmic in the dimension. This result was recently improved by Schliserman, Sherman, and Koren [20] that showed that at least square root of the dimension is required. Taken together, so far it was shown that either the algorithm's hyperparameters are tuned to achieve stability, at a cost in running time, or the algorithm must suffer some dimension dependence, linear at worst square root at best.

Here, we close the gap and show that linear dependence is necessary. Informally, we provide the following generalization error bound, in terms of dimension 𝑑 , sample size, 𝑚 , and hyperparameters of the algorithm, η and 𝑇 (the learning rate and number of iterations). We show that when 𝑇 is at most cubic in the dimension (see Theorem 1 for a formal statement):

The first factor in the RHS describes the linear dependence of the generalization error in the dimension, and corresponds to the optimal sample complexity of empirical risk minimizers, as demonstrated by Carmon, Livni, and Yehudayoff [11]. The second term lower bounds the stability of the algorithm [6], and played a similar role in previous bounds [3, 20]. Each factor is optimal at a certain regime, and cannot be improved. Most importantly, for a standard choice of η = 𝑂 ( 1 / √ 𝑇 ) , the first term is dominant, and the aformentioned lower bound is complemented with the upper bound of Carmon et al. [11]. Our result implies, then, a sample complexity of ˜ Θ ( 𝑑 / 𝑚 + 1 / √ 𝑚 ) . When 𝑑 ≥ 𝑚 , the second factor is dominant. When running time is at most quadratic in number of examples, this term also governs the stability of the algorithm, hence the result of Bassily et al. [6] provides a complementary upper bound (see further discussion in Section 3.1).

2 Background

We consider the standard setup of Stochastic Convex Optimization (SCO) as in [22]. Set W = { 𝑤 : ∥ 𝑤 ∥ ≤ 1 } , and let Z be an arbitrary, finite domain (our main result is a lower bound, hence finiteness of Z is without loss of generality). We assume that there exists a function 𝑓 ( 𝑤, 𝑧 ) that is convex and 𝐿 -Lipschitz in 𝑤 ∈ W for every choice of 𝑧 ∈ Z . Recall that a function 𝑓 is convex and 𝐿 -Lipschitz if for any 𝑤 1 , 𝑤 2 ∈ W and 0 ≤ λ ≤ 1:

First order optimization Algorithmically we require further assumptions concerning any interaction with the function to be optimized. Recall [19] that, for fixed 𝑧 , the sub-gradient set of 𝑓 ( 𝑤, 𝑧 ) at point 𝑤 is the set:

Afirst order oracle for 𝑓 is a mapping O 𝑧 ( 𝑤 ) such that O 𝑧 ( 𝑤 ) ∈ 𝜕 𝑓 ( 𝑤, 𝑧 ) . Our underlying assumption is that a learner has a first order oracle access. In other words, given a function 𝑓 ( 𝑤, 𝑧 ) , we will assume that there is a procedure O 𝑧 that calculates and returns a subgradient at every 𝑤 for every 𝑧 . Recall [10, 19] that when | 𝜕 𝑓 ( 𝑤, 𝑧 )| = 1, the function is differentiable, at 𝑤 , and in that case, the unique subdifferential is the gradient ∇ 𝑓 ( 𝑤, 𝑧 ) .

Learning

Alearning algorithm 𝐴 , in SCO, is any algorithm that receives as input a sample 𝑆 = { 𝑧 1 , . . . , 𝑧 𝑚 } ∈ Z 𝑚 of 𝑚 examples, and outputs a parameter 𝑤 𝑆 . An underlying assumption in learning is that there exists a distribution 𝐷 , unknown to the learner 𝐴 , and that the sample 𝑆 is drawn i.i.d from 𝐷 . The goal of the learner is to minimize the population loss:

More concretely, We will say that the learner has sample complexity 𝑚 ( ε ) if, assuming | 𝑆 | ≥ 𝑚 ( ε ) , then w.p. 1 / 2 (Again, because we mostly care about lower bounds, fixing the confidence will not affect the generality of our result):

Empirical Risk Minimization A natural approach to perform learning is by Empirical Risk Minimization (ERM). Given a sample 𝑆 , the empirical risk is defined to be:

An ε -ERM is any algorithm that, given sample 𝑆 , returns a solution 𝑤 𝑆 ∈ 𝑊 that minimizes the empirical risk up to additive error ε > 0. Recently, Carmon et al. [11] showed that any ε -ERM algorithm has a sample complexity bound of

The above rate is optimal up to logarithmic factor [12]. Namely, there exists a construction and an ERM that will fail, w.p. 1 / 2, unless 𝑚 = Ω ( 𝑑 / ε ) examples are provided 1 . Importantly, though, there are algorithms that can learn with much smaller sample complexity. In particular SGD [26], stable-GD [6] and regularized ERMs [8].

Gradient Descent

We next depict Gradient Descent whose sample complexity is the focus of this work. GD depends on hyperparameters 𝑇 ∈ ℕ and η ≥ 0 and operates as follows on the empirical risk. The algorithm receives as input a sample 𝑆 = { 𝑧 1 , . . . , 𝑧 𝑚 } , defines 𝑤 0 = 0, and operates for 𝑇 iterations according to the following recursion:

where Π is the projection onto the unit ball, and O 𝑧 ( 𝑤 𝑡 ) is a subgradient of the loss function 𝑓 ( 𝑤, 𝑧 ) at 𝑤 𝑡 . The final output, 𝑤 𝐺𝐷 𝑆 , of the algorithm is the averaged iterate (our result, though, can be generalized to other possible suffix-averages such as, say, outputting the last iterate, see Theorem 10). GD constitutes an ε -ERM. Concretely, it is known [10, 17] that GD minimizes the empirical risk and its optimization error is given by:

The above bound is tight irrespective of the dimension 2 . The population loss have also been studied, and Bassily et al. [6] demonstrated the following learning guarantee:

The last two terms in the RHS follow from a stability argument, provided in [6], and the first term follows from the optimization error of GD as depicted in Eq. (5). Notice that there is always an 𝑂 ( η √ 𝑇 ) gap between the generalization error and empirical error of gradient descent.

3 Main Result

Theorem 1. For every 𝑑 ≥ 4096 , 𝑇 ≥ 10 , 𝑚 ≥ 1 and η > 0 , there exists a distribution 𝐷 , and a 4 -Lipschitz convex function 𝑓 ( 𝑤, 𝑧 ) in ℝ 𝑑 + 1 , such that for any first order oracle of 𝑓 ( 𝑤, 𝑧 ) , with probability 1 / 2 , if we run GD with η as a learning rate then:

We remark, that the above theorem is true for any suffix averaging (e.g. last iterate), and not restricted to the averaged iterate (see Theorem 10). We now specialize our bound for two interesting regimes. First, we improve previous dependence in the dimension in [3, 20] and obtain a generalization error bound for 𝑑 = Ω ( 𝑚 + 𝑇 1 / 3 ) :

Corollary 2. Fix η , and suppose 𝑑 = Ω GLYPH<0> 𝑚 + 𝑇 1 / 3 GLYPH<1> . There exists a distribution 𝐷 , and an 𝑂 ( 1 ) -Lipschitz convex function 𝑓 ( 𝑤, 𝑧 ) in ℝ 𝑑 , such that for any first order oracle of 𝑓 ( 𝑤, 𝑧 ) , with probability 1 / 2 , if we run GD for 𝑇 iterations, then:

The first term follows from Theorem 1, the second term follows from the optimization error in Eq. (5). Equation (7) does not hold for 𝑑 < 𝑚 , and the linear improvement over [2, 20] is tight. This can be seen from Eq. (5) that shows that GD achieves ε empirical excess error when η = 𝑂 ( 1 / √ 𝑇 ) and 𝑇 = 𝑂 ( 1 / ε 2 ) . Equation (7) becomes vacuous for such choice of parameters, but Carmon et al. [11] showed that the sample complexity of any ERM is bounded by ˜ 𝑂 (( 𝑑 + √ 𝑚 )/ 𝑚 ) . However, as depicted next, this upper bound becomes tight and GD does not improve over a worst-case ERM:

Corollary 3. Suppose 𝑇 = 𝑂 ( 𝑚 1 . 5 ) , and η = Θ ( 1 / √ 𝑇 ) . There exists a distribution 𝐷 , and an 𝑂 ( 1 ) -Lipschitz convex function 𝑓 ( 𝑤, 𝑧 ) in ℝ 𝑑 , such that for any first order oracle of 𝑓 , with probability 1 / 2 , if we run GD with η , for 𝑇 iterations:

Corollary 3 complements Carmon et al. [11] upper bound, and improves over Feldman [12] lower bound that only showed existence of some ERM with the aforementioned sample complexity. To see that Corollary 3 follows from Theorem 1, notice that when 𝑑 ≤ √ 𝑚 , then 𝑑 / 𝑚 < 1 / √ 𝑚 and the bound is dominated by the second term, which is a well known-information theoretic lower bound for learning. When 𝑑 > √ 𝑚 , and 𝑇 < 𝑚 1 . 5 we have that 𝑇 ≤ 𝑑 3 , plugging η = 𝑂 ( 1 / √ 𝑇 ) yields the bound.

3.1 Discussion

Theorem 1 provides a new generalization error bound for GD. It shows that the worst case sample complexity for ERMs, derived by Feldman [12], is in fact applicable also to a very natural first order algorithm and not just to abstract ERMs. This Highlights the importance of choosing the right algorithm for learning in SCO. As discussed, the bound is tight in several regimes, nevertheless still there are unresolved open problems.

Stability in low dimension When GD is treated as a naive empirical risk minimizer, and η = 𝑂 ( 1 / √ 𝑇 ) , 𝑇 = 𝑂 ( 𝑚 ) , there is no improvement, when using GD, over a worst-case ERM. In the other direction, for dimension that is linear in 𝑚 , one cannot improve over the Ω ( η √ 𝑇 ) term that governs stability. Our bound, though, provide a hope that stability in low dimension can yield an improved bound. In particular, consider the case where η = 1 / 𝑇 1 / 4 and 𝑑 < 𝑚 . This is a case where we apply a stable algorithm in small dimensions. Our bound does not negate the possibility of an improved generalization bound. That would mean that, at least at some regime, GD can improve over the worst-case ERM behaviour. We leave it as an open problem for future study

Open Question 4. Is there a generalization bound for GD such that

Alternatively, can we prove an improved generalization error bound such that:

Late stopping Another regime where there is a gap between known upper bound and lower bound appears when 𝑇 = Ω ( 𝑚 2 ) . Specifically, the stability upper bound for GD by Bassily et al. [6] gives

By Corollary 2, for large enough dimension:

When 𝑇 = 𝑂 ( 𝑚 2 ) , the two bounds coincide. Indeed, for the η 𝑇 / 𝑚 term to dominate the η √ 𝑇 term, we must have 𝑇 = Ω ( 𝑚 2 ) . One has to take at least 𝑇 = 𝑂 ( 𝑚 2 ) iterations in order to generalize with GD (in fact, any full batch method [2]), however 𝑇 = 𝑂 ( 𝑚 2 ) iterations are sufficient. Nevertheless, the above gap does yield the possibility of an unstable GD method that does generalize. Particularly, if we just regulate the term η √ 𝑇 , but allow η 𝑇 / 𝑚 = Ω ( 1 ) , then this may yield a regime where GD is unstable (and ERM bounds do not apply) and yet generalize.

Open Question 5. Are there choices of η and 𝑇 (that depend on 𝑚 ) such that η 𝑇 / 𝑚 ∈ Ω ( 1 ) , but GD has dimension indpendent sample complexity?

Notice that the η 𝑇 / 𝑚 term also governs stability in the smooth convex optimization setup [13]. Recall that a function 𝑓 ( 𝑤, 𝑧 ) is said to be β -smooth if for all 𝑧 , 𝑓 ( 𝑤, 𝑧 ) is differentiable, and the gradient is an β -Lipschitz mapping [10, 15]. For smooth optimization, even if η √ 𝑇 = Ω ( 1 ) , GD is still stable. Hardt, Recht, and Singer [13] showed that the stability of GD in the smooth case is governed by 𝑂 GLYPH<16> η 𝑇 𝑚 GLYPH<17> for η < 1 / β . Therefore, the question of generalization when η 𝑇 / 𝑚 ∈ Ω ( 1 ) remains open, even under smoothness assumptions:

Open Question 6. Assume that 𝑓 ( 𝑤, 𝑧 ) is Θ ( 1 ) -smooth. What is the sample complexity of GD, when η and 𝑇 are chosen so that η + 1 η 𝑇 = 𝑜 ( 1 ) , but η 𝑇 𝑚 = Ω ( 1 ) .

4 Technical Overview

We next provide a high level overview of our proof technique. For simplicity of exposition we begin with the case 𝑇 = 𝑚 = 𝑑 . We begin by a brief overview of previous construction by Amir et al. [3] that demonstrated Corollary 2 when 𝑚 = Ω ( log 𝑑 ) . The construction in [3] can be decomposed into three terms:

The function 𝑔 has the property that an ERM may fail to learn, unless dimension dependent sample size is considered. Amir et al. [3] incorporated Shalev-Shwartz et al. [22] construction. Later, [20] used Feldman's function [12] to construct 𝑔 . The shift from the construction depicted in Shalev-Shwartz et al. [22] to Feldman's function is the first step that allows to move from logarithmic to polynomial dependence in the dimension. In both constructions an underlying property of 𝑔 is that there exists a distribution 𝐷 such that, for small samples, there are overfitting minima. Concretely, there exists a 𝑤 𝑆 ∈ { 0 , 1 / √ 𝑑 } 𝑑 such that

The challenge is then, to make gradient descent's trajectory move towards the point 𝑤 𝑆 . The idea can be decomposed into two parts:

Simplifying with an adversarial subgradient:

To simplify the problem, let us first ease the challenge and suppose we can choose our subgradient oracle in a way that depends on the observed sample. Let 𝑁 0 be the Nemirovski function [9]:

Notice that 𝑁 0 is not differentiable and the choice of subgradient at certain points is not apriori determined. For example, notice that every standard basis vector -𝑒 𝑖 ∈ 𝜕𝑁 0 ( 0 ) . More generally, given a sample 𝑆 , let 𝐼 = 𝑖 1 , . . . , 𝑖 𝑑 ′ be exactly the set of indices such that 𝑤 𝑆 , from Eq. (8), 𝑤 𝑆 ( 𝑖 ) ≠ 0. Now assume by induction that 𝑤 𝑡 ( 𝑖 ) > 0 exactly for 𝑖 = 𝑖 1 , . . . 𝑖 𝑡 , then one can show that we can define the subgradient oracle of 𝑁 0:

In that case 𝑤 𝑡 + 1 will satisfy our assumption for 𝑖 𝑡 + 1 and we can continue to follow this dynamic for 𝑇 steps.

√

Notice that, in this case, GD will converge to 𝑤 𝑆 (if η = 1 / 𝑑 which we assume now for concreteness). One can also show that the output of GD (the averaged iterate) will overfit. The caveat is that our subgradient oracle depends on the sample 𝑆 . In reality, the sample is drawn independent of the subgradient oracle. and all previous constructions, as well as ours need to handle this. This is discussed in the next section. But before that, let us review another challenge which is when 𝑇 ≠ 𝑑 :

When 𝑑 ≪ 𝑇 Another challenge we face with the construction above is that it works when we assume that 𝑇 ≈ 𝑑 . That is because, in Nemirovski's function, the number of iterates we can perform is bounded by the dimension. After 𝑑 iterations we will end up with the vector 𝑣 = P 𝑑 𝑡 = 1 η 𝑒 𝑖 𝑡 . If 𝑇 = ω ( 𝑑 ) then η = 𝑜 ( 1 / √ 𝑑 ) , and the dynamic will end up with a too small norm vector to induce a sizeable population loss. This strategy will provide, at best, with a factor of the form Ω GLYPH<16> η √︁ min { 𝑑, 𝑇 } GLYPH<17> . Such a factor may be unsatisfactory in a very natural setting where, say, 𝑇 = 𝑂 ( 𝑚 ) , η = 𝑂 ( 1 / √ 𝑚 ) , and 𝑑 = Ω ( √ 𝑚 ) . To obtain the 𝑑 3 dependence, we perform the following alternation over the Nemirovski function. Consider the function:

And suppose that at each iteration we return a subgradient as follows:

If there is 𝑖 ≤ 𝑑 , such that 𝑤 ( 𝑖 ) = 𝑤 ( 𝑖 + 1 ) > η , return subgradient 𝑒 𝑖 + 1 -𝑒 𝑖 and 𝑤 is updated by 𝑤 𝑡 + 1 = 𝑤 𝑡 -η 𝑒 𝑖 + 1 + η 𝑒 𝑖 .

If there is no such 𝑖 , then take the minimal 𝑖 (if exists) such that 𝑤 ( 𝑖 ) = 0, and return subgradient -𝑒 𝑖 and update 𝑤 𝑡 + 1 = 𝑤 𝑡 + η 𝑒 𝑖 .

When non of the above is met, return subgradient 0.

The dynamic of the above scheme is depicted in Fig. 1, and solves the problem when 𝑇 ≈ 𝑑 3 . One can show that GD will run for at least 𝑑 3 ≈ 𝑇 iterations, and will increase 𝑂 ( 𝑑 ) coordinates, each, on average, by an order of 𝑂 ( η 𝑑 ) . This is better than the increase of η in each coordinate that we get from Nemirovski's function. In this way we obtain the improved result of η √ 𝑇 , even when 𝑇 ≈ 𝑑 3 .

When 𝑇 ≪ 𝑑 , when the number of iterations is smaller than 𝑑 we face a different challenge. The immediate solution is to embed in ℝ 𝑑 a construction from ℝ 𝑇 , this will provide us with the Ω ( η √ 𝑇 ) term but, on the other hand, such a construction will not yield a Ω ( 𝑑 / 𝑚 ) term. A different approach, that exploits the dimension to its fullest, is to consider blocks of coordinates and operate on those instead of single coordinates.

The conclusive outcome incorporates both ideas together, and we replace the Nemirovski function with a version of Eq. (9) that operates on 𝑂 ( 𝑇 1 / 3 ) blocks of coordinates. And this concludes our construction. We next move on to the challenge of replacing the data dependent oracle with a standard first order oracle.

a=-e1

a=e2-e1

a=-e2

a=-e2

a=-e3

a=e3一

ez

a=0

a=e2-e1

Reduction to sample dependent oracle:

As discussed, the construction above does not yield a lower bound as it relies on a subgradient oracle that is dependent on the whole sample. To avoid such dependence of the oracle on the sample, we observe that if we can infer the sample 𝑆 from the trajectory, i.e. if the state 𝑤 𝑡 'encodes" the sample, then formally the subgradient is allowed to 'decipher" the sample from the point 𝑤 𝑡 . In that way we achieve this behaviour of sample dependent subgradient oracle. This part becomes challenging and may depend on the way we choose 𝑔 , and 𝑁 . The simplest case, studied by Amir et al. [3], introduced the third function, ℎ , which was a small perturbation function that elevates coordinates in 𝐼 and inhibits coordinates not in 𝐼 . The function ℎ depends on 𝑧 and not on 𝑆 , hence it cannot know apriori 𝐼 . But, an important observation is that, in Shalev-Shwartz et al. [22] construction, if 𝑖 ∉ 𝐼 , there exists 𝑧 ∈ 𝑆 that 'certifies" that. In fact, each 𝑧 can be thought of as a subset of indices, and if an index appears in 𝑧 , then it cannot be in 𝐼 . So we can build the perturbation in a way that every coordinate is elevated, unless 𝑧 certifies 𝑖 ∉ 𝐼 : In that case we define ℎ ( 𝑤, 𝑧 ) so that its gradient will radically inhibit 𝑖 .

The last observation is what becomes challenging in our case. As discussed, to achieve improved rate, we need to use Feldman's function. When using Feldman's function the coordinates cannot be ruled out, or identified, by a single 𝑧 but one has to look at the whole sample to identify 𝐼 . While Schliserman et al. [20] tackle a similar problem, we take a slightly different approach described next: For each 𝑧 assign a random, positive, number α ( 𝑧 ) . We can think of this α as a hash function. Let us add another coordinate to the vector 𝑤 , 𝑤 ( 𝑑 + 1 ) . Consider the function

Then 𝜕ℎ ( 𝑤, 𝑧 ) = γα ( 𝑧 ) 𝑒 𝑑 + 1. Write α ( 𝑆 ) = 1 | 𝑆 | P 𝑧 ∈ 𝑆 α ( 𝑧 ) then in turn:

If γ , α ( 𝑧 ) are chosen correctly, α ( 𝑆 ) is a one to one mapping from samples to real numbers, and small γ ensures that the overall addition of ℎ has negligible affect on the outcome. Then, we may define the subgradient oracle to be dependent on coordinate 𝑑 + 1 which encodes the whole sample. Our final construction will take a different ℎ , which adds small strong convexity in this coordinate, for reasons next explained:

Working with any first order oracle Notice that our statement is slightly stronger than what we so far illustrated. Theorem 1 states that, for any subgradient oracle, GD will fail. For that, we need to be a little bit more careful, and we want to replace our function with a function that leads to the

same guaranteed trajectory, but at the same time it should be differentiable at visited points. This will ensure a unique derivative, making the construction independent of the choice of (sub)gradient oracle.

Towards this goal, we start with the construction depicted so far and consider the set of all values, gradients, and points { 𝑓 𝑗 , 𝑔 𝑗 , 𝑤 𝑗 } 𝑗 ∈ 𝐽 that our algorithm may visit, for any possible time step and any possible sample, with our construction. Notice that, while this set may be big and even exponential, it is nevertheless finite. What we want is to interpolate a new function through these triplets. In contrast with our original construction, we require a differentiable function at the designated points. Notice, that such an interpolation will have the exact same behaviour when implementing GD on it (with the added feature that the oracle is well defined and unique).

The problem of convex interpolation is well studied, for example Taylor et al. [23] shows sufficient and necessary conditions for interpolation of a smooth function. Our case is slightly easier as we do not care about the smoothness parameter. On the other hand we do require Lipschitzness of the interpolation. We therefore provide an elementary, self-contained, proof to the following easy to prove Lemma, (proof is provided in Appendix B)

Lemma 7. Let 𝐺 = { 𝑓 𝑗 , 𝑔 𝑗 , 𝑤 𝑗 } 𝑗 ∈ 𝐽 ⊆ ℝ × ℝ 𝑑 × ℝ 𝑑 be a triplet of values in ℝ , and gradients and points in ℝ 𝑑 , such that ∥ 𝑔 𝑗 ∥ ≤ 𝐿 . Suppose that for every 𝑖, 𝑗 ∈ 𝐽 :

and let

Then there exists a convex 𝐿 -Lipschitz function ˆ 𝑓 such that for all 𝑗 ∈ 𝐽 : ˆ 𝑓 ( 𝑤 𝑗 ) = 𝑓 𝑗 , and for all 𝑖 ∈ 𝐼 diff , ˆ 𝑓 is differentiable at 𝑤 𝑖 and:

With Lemma 7 at hand, consider the function

The above function encodes in 𝑤 ( 𝑑 + 1 ) the sample and time-step as before. Moreover, because it is slightly strongly convex (in coordinate 𝑑 + 1), 𝑤 1 ( 𝑑 + 1 ) ≠ 𝑤 2 ( 𝑑 + 1 ) ensures that

Then the term ℎ in 𝑓 ensures that the triples { 𝑓 𝑗 , 𝑔 𝑗 , 𝑤 𝑗 } along the trajectory generate gradient vectors that satisfy strict inequality in Eq. (10) and in turn, our interpolation from Lemma 7 will be differentiable at these points. There's some technical subtlety because the interpolation needs to also take the averaged iterate into account, but this is handled in a similar fashion.

In the next two sections we provide more formal statements of the two main ingredients: First, we define a setup of optimization with a sample-dependent first order Oracle and state a lower bound for the generalization error in this setup. The second ingredient is a reduction from the standard setup of first order optimization.

4.1 Sample-dependent Oracle

As discussed, the first step in our proof is to consider a slightly weaker setup where the first-order oracle may depend on the whole sample. Let us formally define what we mean by that. Define

the set of all subseqences of samples of size at most 𝑚 . Given a function 𝑓 ( 𝑤, 𝑧 ) , a sample dependent oracle, O S , is a finite sequence of first order oracles

that each receive as input a finite sample 𝑆 , as well as 𝑤 and returns a subgradient:

The sequence of samples can be thought of as the past samples that were observed by the algorithm. In the case of full-batch GD these will be the whole sample, and for SGD, each 𝑆 provided to O ( 𝑡 ) will be all previously observed samples. Given S ∈ S 𝑇 𝑚 let us also denote

«

‹

where we let S 1:0 = ∅ , and 𝑆 1: 𝑡 -1 = ( 𝑆 1 , . . . , 𝑆 𝑡 -1 ) is the concatenated subsample of all previously observed samples in the sequence. As discussed, working with a sample-dependent oracle is easier (for lower bounds). And indeed, our first result shows that, if the subgradient can be chosen in a way that depends on the sample, we can provide the desired lower bound. For fixed and known η > 0, 𝑇 , a sample dependent first order oracle O S , and a sequence of samples S = ( 𝑆 1 , 𝑆 2 , . . . , 𝑆 𝑇 ) , define 𝑤 0 = 0 and inductively:

Lemma8. For every 𝑚, 𝑑, 𝑇 ≥ 18 and η > 0 there are a distribution 𝐷 , a 3 -Lipschitz convex function 𝑓 ( 𝑤, 𝑧 ) in ℝ 𝑑 , as well as a sample dependent first order oracle O S such that: if S = ( 𝑆, 𝑆, . . . 𝑆 ) ∈ S 𝑇 𝑚 for 𝑆 ∼ 𝐷 𝑚 i.i.d, then w.p. 1 / 2 , for every suffix averaging s :

The proof of Lemma 8 is provided in Appendix A.1. We next move to describe the second ingredient of our proof.

4.2 Reduction to sample-dependent oracles

As discussed, the second ingredient of our proof is a reduction to the sample-dependent setup. Instead of using a perturbation function as in [3], we take a more black box approach and show that, given a sample dependent first order oracle, there exists a function that basically induces the same trajectory. Proof is provided in Appendix A.2.

Lemma 9. Suppose 𝑞 ∈ ℝ 𝑇 , ∥ 𝑞 ∥∞ ≤ 1 . And suppose that 𝑓 ( 𝑤, 𝑧 ) is a convex, 𝐿 -Lipschitz, function over 𝑤 ∈ ℝ 𝑑 , let η > 0 , let O S be a sample dependent first order oracle, and for every sequence of samples S = ( 𝑆 1 , 𝑆 2 , . . . , 𝑆 𝑇 ) define the sequence { 𝑤 S 𝑡 } 𝑇 𝑡 = 1 as in Eq. (12) .

Then, for every ε > 0 there exists an 𝐿 + 1 Lipschitz convex function 3 ¯ 𝑓 (( 𝑤, 𝑥 ) , 𝑧 ) over ℝ 𝑑 + 1 (that depends on 𝑞, 𝑓 , 𝑇, η , 𝑚, O S , ε ).

such that for any first order oracle O 𝑧 for ¯ 𝑓 , define 𝑢 0 = 0 ∈ ℝ 𝑑 and 𝑥 0 = 0 ∈ ℝ , and:

then if we define:

then, we have that 𝑢 𝑞 = 𝑤 S 𝑞 and:

and,

3 i.e. 𝑤 ∈ ℝ 𝑑 and 𝑥 ∈ ℝ

and for every suffix s < 𝑇

Ackgnoweledgments The author would like to thank Tamar Livni for creating Figure 1. Tamar holds all copyrights to the artwork. The author would also like to thank Tomer Koren and Yair Carmon for several discussions. This research was funded in part by an ERC grant (FOG, 101116258), as well as an ISF Grant (2188 \ 20).

References

N. Alon, S. Ben-David, N. Cesa-Bianchi, and D. Haussler. Scale-sensitive dimensions, uniform convergence, and learnability. Journal of the ACM (JACM) , 44(4):615-631, 1997.

I. Amir, Y. Carmon, T. Koren, and R. Livni. Never go full batch (in stochastic convex optimization). Advances in Neural Information Processing Systems , 34:25033-25043, 2021.

I. Amir, T. Koren, and R. Livni. Sgd generalizes better than gd (and regularization doesn't help). In Conference on Learning Theory , pages 63-92. PMLR, 2021.

I. Amir, R. Livni, and N. Srebro. Thinking outside the ball: Optimal learning with gradient descent for generalized linear stochastic convex optimization. Advances in Neural Information Processing Systems , 35:23539-23550, 2022.

I. Attias, G. K. Dziugaite, M. Haghifam, R. Livni, and D. M. Roy. Information complexity of stochastic convex optimization: Applications to generalization and memorization. arXiv preprint arXiv:2402.09327 , 2024.

R. Bassily, V. Feldman, C. Guzmán, and K. Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems , 33:4381-4391, 2020.

A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the vapnikchervonenkis dimension. Journal of the ACM (JACM) , 36(4):929-965, 1989.

O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

S. Bubeck, Q. Jiang, Y.-T. Lee, Y. Li, and A. Sidford. Complexity of highly parallel non-smooth convex optimization. Advances in neural information processing systems , 32, 2019.

S. Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends ® in Machine Learning , 8(3-4):231-357, 2015.

D. Carmon, R. Livni, and A. Yehudayoff. The sample complexity of erms in stochastic convex optimization. arXiv preprint arXiv:2311.05398 , 2023.

V. Feldman. Generalization of erm in stochastic convex optimization: The dimension strikes back. Advances in Neural Information Processing Systems , 29, 2016.

M. Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

D. Haussler and M. Warmuth. The probably approximately correct (pac) and other learning models. The Mathematics of Generalization , pages 17-36, 2018.

E. Hazan et al. Introduction to online convex optimization. Foundations and Trends ® in Optimization , 2(3-4):157-325, 2016.

R. Livni. Information theoretic lower bounds for information theoretic upper bounds. Advances in Neural Information Processing Systems , 36, 2023.

Y. Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2013.

B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

R. T. Rockafellar. Convex Analysis:(PMS-28) . Princeton university press, 2015.

M. Schliserman, U. Sherman, and T. Koren. The dimension strikes back with gradients: Generalization of gradient methods in stochastic convex optimization. arXiv preprint arXiv:2401.12058 , 2024.

A. Sekhari, K. Sridharan, and S. Kale. Sgd: The role of implicit regularization, batch-size and multiple-epochs. Advances In Neural Information Processing Systems , 34:27422-27433, 2021.

S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Stochastic convex optimization. In COLT , volume 2, page 5, 2009.

A. B. Taylor, J. M. Hendrickx, and F. Glineur. Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Mathematical Programming , 161:307-345, 2017.

V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. In Measures of complexity: festschrift for alexey chervonenkis , pages 11-30. Springer, 2015.

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928-936, 2003.

A Proof of Theorem 1

The proof is an immediate corollary of Lemmas 8 and 9, which we prove in Appendices A.1 and A.2 respectively. To see that Theorem 1 indeed follows from these Lemmas, start with η , 𝑑, 𝑚, 𝑇 that satisfy the conditions. Let 𝑓 ( 𝑤, 𝑧 ) be the function and O S the sample dependent first order oracle, whose existence follows from Lemma 8 with suffix s = 0. And let ¯ 𝑓 be the function whose existence follows from Lemma 9 to some arbitrarily small ε 0, with 𝑞 ( 𝑡 ) = 1 𝑇 for all 𝑡 . It is easy to see that if we apply GD on ¯ 𝑓 and define its output ( 𝑢 𝐺𝐷 , 𝑥 𝐺𝐷 ) then: ( 𝑢 𝐺𝐷 , 𝑥 𝐺𝐷 ) = ( 𝑢 𝑞 , 𝑥 𝑞 ) , and 𝑤 S 𝑞 = 𝑤 𝐺𝐷 S .

Then, we have that w.p. 1 / 2:

Where in the last equation, we assume ε 0 to be sufficiently small. Finally, note that ¯ 𝐹 ( 0 ) ≥ min 𝑤 ∈ W ¯ 𝐹 ( 𝑤 ) .

Notice, that by the same argument, by taking any suffix s < 𝑇 , and setting 𝑞 ( 𝑡 ) = 0 for 𝑡 ≤ s , and 𝑞 ( 𝑡 ) = 1 𝑇 -s for 𝑡 ≥ s + 1, we can obtain the following stronger result for any suffix averaging:

Theorem 10. For every 𝑑 ≥ 4096 , 𝑇 ≥ 10 , 𝑚 ≥ 1 and η > 0 and suffix s < 𝑇 , there exists a distribution 𝐷 , and a 4 -Lipschitz convex function 𝑓 ( 𝑤, 𝑧 ) in ℝ 𝑑 , such that for any first order oracle of 𝑓 ( 𝑤, 𝑧 ) , with probability 1 / 2 , if we run GD with η as a learning rate then:

A.1 Proof of Lemma 8

For simplicity we will assume that 𝑑 = 2 𝑛 for some 𝑛 ∈ ℕ , the final result will be obtained by embedding a construction in a subspace of size at least half the original dimension.

We start by recalling Feldman's construction [12]: There exists a set V ⊆ { 0 , 1 } 𝑑 , such that:

Indeed, suppose we pick randomly 𝑤 ∈ { 0 , 1 } 𝑑 according to probability 𝑃 where each coordinate 𝑃 ( 𝑤 ( 𝑖 )) = 1 with probability 1 / 2 (independently). Then, by Hoeffding's inequality for two 𝑤 1 , 𝑤 2 ∼ 𝑃 independently:

Thus, picking V elements i.i.d according to 𝑃 , randomly, of size | V | ≥ 𝑒 𝑑 / 258 we can show by union bound that with positive probability, all | V | 2 pairs in V will satisfy the requirement. Next, we define a distribution 𝐷 ε supported on subsets of V such that for a random variable 𝑉 ⊆ V each 𝑣 ∈ 𝑉 w.p. ε independently. We start by assuming that 𝑇 ≤ 1 17 𝑑 3 (the case 𝑇 > 𝑑 3 is handled at the end), and let 𝑘 ∈ ℕ be such that:

One can show that without loss of generality we can assume 𝑘 is also a power of 2 (in particular, 𝑑 is divisble by 𝑘 for large enough 𝑇 ). We next follow the idea depicted in Section 4, but we want to handle the case 𝑘 ≫ 1. For that, we redefine the function in Eq. (9), and take blocks of coordinates. To simplify notations, let us define for two set of indices 𝐼, 𝐽 of elements in [ 𝑑 ] : 𝐼 = { 𝑖 1 , . . . , 𝑖 𝑘 } , 𝐽 = { 𝑗 1 , . . . , 𝑖 𝑘 } , 𝐼 ≺ 𝐽 if max { 𝑖 ∈ 𝐼 } < min { 𝑗 ∈ 𝐽 } . and we will also write:

then we define our final function as:

Define α = min { 1 η √ 2 𝑇 , 1 } , and let:

where 𝑁 is defined in Eq. (14), and 𝑔 is defined to be Feldman's function with a suitable choice of threshold:

Notice that 𝑁 is 2-Lipschitz, and 𝑔 is 1-Lipschitz.

To obtain the trajectory, we next define a sample dependent oracle. We only define it for samples S such that there exists 𝑣 ★ ∉ 𝑉 𝑖 for all 𝑉 𝑖 ∈ 𝑆 (define it arbitrarily to any other type of sample). Let I = { 𝑖 1 , . . . , 𝑖 𝑑 ′ } be a set of 7 𝑑 16 indices such that 𝑣 ★ ( 𝑖 𝑗 ) ≠ 0. Divide the elements of I into 𝑑 ′ / 𝑘 subsets. Namely, let

We start by defining only an oracle for the function α 𝑁 . We will later show that the trajectory induced by this oracle stays in the minima of 𝑔 , and that will show that, for our purposes, we can choose the

same oracle for the whole function 𝑓 . We denote by O ( 𝑡 ) α 𝑁 the sample dependent oracle for α 𝑁 , and we first define:

Next, we define for 𝑡 > 1. For any 𝑤 such that 0 ∉ 𝜕𝑁 ( 𝑤 ) , define it arbitrarily. If 0 ∈ 𝜕𝑁 ( 𝑤 ) we define it as follows:

If there is a multi-index 𝐼 𝑗 that 𝑤 ( 𝐼 𝑗 ) = 𝑤 ( 𝐼 𝑗 + 1 ) > ηα : Then:

If there is no such multi-index, and if 𝐼 𝑗 is the minimal multiindex such that 𝑤 ( 𝐼 𝑗 ) = 0, set:

If both conditions cannot be met, then:

The trajectory of this dynamic is depicted in Fig. 1, for the case 𝑘 = 1. In the general case, we can think of each coordinate in Fig. 1 as a block of size 𝑘 . Now we assume that { 𝑤 S 𝑡 } 𝑇 𝑡 = 1 follows the trajectory depicted in Eq. (12) with that choice of Oracle. It can be seen that the update step is such that after 𝑇 ′ = 1 + P 𝑑 ′ / 𝑘 𝑡 ′ = 1 P 𝑡 ′ 𝑡 ′′ = 1 𝑡 ′′ rounds, we will have that for every 𝑖 ∈ 𝐼 𝑡 :

and that for 𝑡 > 𝑇 ′ : 𝑤 S 𝑡 = 𝑤 S 𝑇 ′

Moreover, one can show that 𝑤 𝑡 is non zero only in coordinates 𝑖 ∈ I , and that for any subset 𝐼 𝐵 ⊆ 𝐼 such that | 𝐼 | = 𝐵 :

A formal proof is provided in Appendix D. Notice that:

Next, for any 𝑣 ≠ 𝑣 ★ , we have that:

and,

So far we assume that 𝑑 = 2 𝑛 , notice that if 𝑇 < 1 8 · 17 𝑑 3 , we can find a subspace of size 𝑑 1 > 𝑑 2 so that 𝑇 < 1 8 · 17 𝑑 3 1 , and we obtain our final result by embedding our construction in this subspace.

.

As such 0 ∈ 𝜕𝑔 ( 𝑤 S 𝑡 , 𝑉 𝑖 ) for all 𝑉 𝑖 , and we can define O ( 𝑡 ) so that

And we have that, { 𝑤 S 𝑡 } is the trajectory obtained from Eq. (12) with respect to this oracle of 𝑔 also. We also have:

And because 𝑇 ′ ≤ 1 17 𝑇 , for every suffix s ∈ [ 𝑇 ] :

Recall that we assume 136 𝑑 3 𝑘 3 ≥ 𝑇 :

Eq. (18) lower bounds the generalization under the event that there exists 𝑣 ∈ V such that 𝑣 ∉ 𝑉 𝑖 for every 𝑖 = 1 , . . . , 𝑚 . Now assume ε = min { 𝑑 516 𝑚 , 1 4 } , then for every 𝑣 , using the inequality ( 1 -ε ) ≤ 𝑒 -2 ε for ε < 1 / 2:

When 𝑇 > 1 17 · 8 𝑑 3 Notice that if we take the construction with 𝑇 = ⌊ 1 17 · 8 𝑑 3 ⌋ then:

hence we can use the above construction for any 𝑇 ′ > 𝑇 , and the subdifferential oracle is defined for every iteration above 𝑇 as returning 0, and we obtain a similar analysis to the case that 𝑇 = ⌊ 𝑑 3 / 136 ⌋ , and our bounds yield as in Eq. (18):

A.2 Proof of Lemma 9

The main ingredient in our proof is the following claim whose proof is provided in Appendix C. To state the claim we will need a few notations. First, for any two tuples of sample-sequences in S 𝑇 𝑚 , and 𝑡 > 0 ( S , 𝑡 ) and ( S ′ , 𝑡 ) , let us denote ( S , 𝑡 ) ≡ ( S ′ , 𝑡 ′ ) if 𝑡 = 𝑡 ′ and 𝑆 1: 𝑡 = 𝑆 ′ 1: 𝑡 , namely the prefixes of the sample sequences agree up to point 𝑡 ′ = 𝑡 . Second, for a mapping α : Z → [ 0 , 1 ] , from sample points to real numbers, and a sample 𝑆 , let us denote

Claim 1. For every η , γ and α define for every sequence of samples S = ( 𝑆 1 , . . . , 𝑆 𝑇 ) , and 𝑡 inductively: 𝑥 S 0 = 0 and

Then, for any ε > 0 , there is a choice of γ < ε and α such that:

For ( S , 𝑡 ) . ( S ′ , 𝑡 ′ ) : 𝑥 S 𝑡 ≠ 𝑥 S ′ 𝑡 ′ .

For every S , 𝑡 , such that 𝑡 < max { 𝑡 ′ : 𝑞 ( 𝑡 ′ ) ≠ 0 } and S ′ : 𝑥 S 𝑡 ≠ P 𝑇 𝑡 = 1 𝑞 ( 𝑡 ) 𝑥 S ′ 𝑡 .

The 𝑥 S 𝑡 represents the different states the trajectory can be in. 𝑥 S 𝑞 represent the output of the trajectory which can be an aggreagated sum. We require that each provide a signature for the state, and this will allow us to 'code" the state of the trajectory along GD.

We now continue with the proof of Lemma 8. For every sample dependent first order oracle O ( 𝑡 ) , we have that:

To simplify notations, let us denote:

as S , 𝑡, 𝑧 completely determine the output. We next define

where γ > 0 is arbitrarily small 4 .

and observe that, if 𝑥 S 𝑡 is defined as in Eq. (19):

and define:

To simplify notation, let us denote:

and for all 𝑡 ≤ 𝑇 ,

and assume without loss of generality that max { 𝑞 ( 𝑖 ) ≠ 0 } = 𝑇 (Otherwise, we look only at the sequence up to point max { 𝑞 ( 𝑖 ) ≠ 0 } ). Consider now the sets of triplets:

where O S ,𝑇 + 1 ,𝑧 ∈ 𝜕 ( 𝑓 ( 𝑤 S 𝑞 ) + ℎ 𝑧 ( 𝑥 S 𝑞 )) is chosen arbitrarily.

Convexity of 𝑓 + ℎ 𝑧 ensure that the triplets in 𝐺 ( 𝑧 ) satisfy Eq. (10) for all 𝑡 ≤ 𝑇 + 1, as in Lemma 7. To apply the Lemma, we also want to achieve differentiability at points such that 𝑡 < 𝑇 . Therefore, take any two triplets

where 𝑡 1 < 𝑇 and 𝑡 2 ≤ 𝑇 + 1, and suppose 𝑔 1 ≠ 𝑔 2. To simplify notations, let us write 𝑤 S 𝑡 𝑖 = 𝑤 𝑖 and 𝑥 S 𝑡 𝑖 = 𝑥 𝑖 .

First, by convexity of 𝑓 we have that:

Next, because 𝑔 1 ≠ 𝑔 2, either ∇ ℎ 𝑧 ( 𝑥 1 ) ≠ ∇ ℎ 𝑧 ( 𝑥 2 ) , which implies 𝑥 1 ≠ 𝑥 2, or O S 1 ,𝑡 1 ,𝑧 ≠ O S 2 ,𝑡 2 ,𝑧 which implies ( S 1 , 𝑡 1 ) . ( S 2 , 𝑡 2 ) which again implies 𝑥 1 ≠ 𝑥 2 by Claim 1. In other words, if 𝑔 1 ≠ 𝑔 2 then 𝑥 1 ≠ 𝑥 2:

We showed then, that 𝑣 1 -𝑣 2 + 𝑔 ⊤ 2 ( 𝑢 2 -𝑢 1 ) = 0, implies 𝑔 1 = 𝑔 2. We obtain, by Lemma 7, that there exists a function ¯ 𝑓 (( 𝑤, 𝑥 ) , 𝑧 ) such that for all 𝑡 and S :

This proves Lemma 9. Indeed. By the Lipschitzness of ℎ in the unit ball, we have that | 𝑥 𝑡 | ≤ γη 𝑇 . For sufficiently small γ , from Eq. (23), since {( 𝑓 ( 0 , 𝑧 ) + ℎ 𝑧 ( 0 ) , ( 0 , ∇ ℎ 𝑧 ( 0 )) , ( 0 , 0 ))} ∈ 𝐺 ( 𝑧 ) , we obtain that

Further, if we assume by induction that for every 𝑡 ′ ≤ 𝑡 -1, 𝑢 𝑡 ′ = 𝑤 S 𝑡 ′ , and 𝑥 𝑡 ′ = 𝑥 S 𝑡 ′ , then from Eq. (24) we have that for any first order oracle:

which proves, by linearity, that 𝑢 𝑞 = 𝑤 S 𝑞 .

B Proof of Lemma 7

We choose

On the other hand,

ˆ 𝑓 is indeed convex as it is the maximum of linear functions. Further, it is known [19] that at each point 𝑤 :

It follows that, ˆ 𝑓 is 𝐿 -Lipschitz. Next, for any 𝑤 𝑖 , notice that our assumption implies:

Hence ˆ 𝑓 ( 𝑤 𝑖 ) = 𝑓 𝑖 . Finally, to see the function is differentiable at designated points, take any 𝑖 ∈ 𝐼 𝑑𝑖 𝑓 𝑓 and consider 𝑤 𝑖 . By Eq. (25), it is enough to show that if 𝑔 𝑗 ∈ 𝜕 𝑓 ( 𝑤 𝑖 ) then 𝑔 𝑖 = 𝑔 𝑗 , but this clearly follows from our assumption, and the fact that ˆ 𝑓 ( 𝑤 𝑖 ) = 𝑓 𝑖 .

C Proof of Claim 1

We first prove by induction that 𝑥 S 0 = 0 and, for 𝑡 ≥ 1:

«

‹

Indeed,

Now, for every S , 𝑡, 𝑧 , define a polynomial:

and let 𝑟 be a rational point, sufficinetly small, so that 1 -𝑟 is not the root of any polynomial of the form 𝑃 S ,𝑡,𝑧 -𝑃 S ′ ,𝑡 ′ ,𝑧 ′ that is distinct from 0. In other words, we choose 𝑟 so that

and we also require that

Notice that there are only finitely many polynomials of the above form, hence we can choose such 𝑟 in any interval ( 0 , ε ) for any ε > 0. Rewriting Eq. (26) we have:

Now, suppose we choose { α ( 𝑧 )} to be reals, independent over the rationals, and suppose we choose γ = 𝑟 / η .

Proof of Item 1 Assume that 𝑥 S 𝑡 = 𝑥 S ′ 𝑡 ′ . Because α ( 𝑧 ) are independent over the rationals, and because 𝑃 ( 1 -γη ) are always rationals, we have that 𝑃 S ′ ,𝑡 ′ ,𝑧 ( 1 -𝑟 ) = 𝑃 S ′ ,𝑡 ′ ,𝑧 ( 1 -𝑟 ) for every 𝑧 . But then,

by choice of 𝑟 . But then, 𝑡 = 𝑡 ′ . Indeed, assume by contradiction and w.l.og assume 𝑡 < 𝑡 ′ : then for any 𝑧 ∈ 𝑆 ′ 1 𝑃 S ,𝑡,𝑧 is a 𝑡 ′ -1-degree polynomial, on the other hand 𝑃 S ,𝑡,𝑧 is at most of degree 𝑡 -1 < 𝑡 ′ -1. Next, if 𝑆 𝑖 ≠ 𝑆 ′ 𝑖 , for 𝑖 ≤ 𝑡 , then we can assume (w.l.o.g) that there is 𝑧 ∈ 𝑆 𝑖 such that 𝑧 ∉ 𝑆 ′ 𝑖 it follows that, by looking at the coefficient of the two polymials of the monomial 𝑋 𝑡 -𝑖 we have that:

Overall then, we obtain that if 𝑥 S 𝑡 = 𝑥 S ′ 𝑡 ′ then ( S , 𝑡 ) ≡ ( S ′ , 𝑡 ′ ) . .

Proof of Item 2: The proof of Item 2 is similar to how we proved 𝑡 = 𝑡 ′ . Notice that:

Hence 𝑥 S 𝑡 = 𝑥 S ′ implies for all 𝑧 ∈ 𝑍 :

For 𝑧 ∈ 𝑆 ′ 1 we have that P 𝑇 𝑡 = 1 𝑞 ( 𝑡 ) 𝑃 S ,𝑡,𝑧 is a max { 𝑡 : 𝑞 ( 𝑡 ) ≠ 0 } -degree polynomial, but on the other hand we assume that 𝑡 < max { 𝑡 : 𝑞 ( 𝑡 ) ≠ 0 } , hence 𝑃 S ,𝑡,𝑧 is a lower degree polynomial.

D Proof of Eqs. (16) and (17)

D.1 Proof of Eq. (16)

To avoid cumbersome notations, we will supress dependence on S and write 𝑤 𝑡 instead of 𝑤 S 𝑡 .

As a first step, observe that at each iteration no projection is performed. Indeed, let us show by induction that:

The definition of α then implies that 𝑤 𝑡 are restricted to the unit ball without projections.

To see the above is true, let us consider the case where the first type of update is performed:

And if the second type of update is performed:

We now move on to prove that Eq. (16) holds by induction. Specifically, we will show that for 𝑑 0 ≤ ⌊ 𝑑 ′ / 𝑘 ⌋ that at time 𝑇 𝑑 0 = 1 + P 𝑑 0 𝑑 = 0 P 𝑑 𝑘 = 0 𝑘 , for any 𝑡 ≤ 𝑑 0:

Eq. (16) then follows by plugging 𝑑 0 = 𝑑 ′ / 𝑘 , and noting that at every step we have that if 𝑖 ∈ 𝐼 𝑡 then 𝑤 𝑇 𝑑 0 ( 𝑖 ) = √ 𝑘𝑤 𝑇 𝑑 0 ( 𝐼 𝑡 ) . Therefore, we are left with proving Eq. (27).

For 𝑑 0 = 0, 𝑇 0 = 1 and we have, indeed, 𝑤 1 = 0. Next assume we proved the statement for 𝑑 0, and we will prove it for 𝑑 0 + 1. Here too, we will use induction, and we prove that, for 𝑑 1 ≤ 𝑑 0 + 1, at time

we have that:

we have that:

We start the induction with the case 𝑑 2 = 1, in that case notice that 𝑇 𝑑 0 ,𝑑 1 ,𝑑 2 = 𝑇 𝑑 0 ,𝑑 1 + 1, and by induction hypothesis:



For the case 𝑑 1=0, 𝑇 𝑑 0 ,𝑑 1 = 𝑑 0, and it follows from our outer-induction step. Assume the statement is true for 𝑑 1 and we will prove it for 𝑑 1 + 1, here, yet again, we use induction. And we will show that for 1 ≤ 𝑑 2 < 𝑑 0 + 2 -𝑑 1 we have at time





In this case, note that there are no two consecutive coordinates that are equal, hence our choice of oracle is defined so that O ( 𝑡 ) = -𝑒 𝐼 𝑑 0 + 1 . Hence, by our update rule (and the lack of projections which we proved at the beginning):



Which satisfies Eq. (28). Now assume that Eq. (28) holds for 𝑑 2, and take 𝑑 2 + 1 < 𝑑 0 + 2 -𝑑 1 (otherwise, we are done). Notice that 𝑇 𝑑 0 ,𝑑 1 ,𝑑 2 + 1 = 𝑇 𝑑 0 ,𝑑 1 ,𝑑 2 + 1. Observe that 𝑤 𝑇 0 ,𝑑 1 ,𝑑 2 ( 𝑑 0 + 2 -𝑑 2 ) = 𝑤 𝑇 0 ,𝑑 1 ,𝑑 2 ( 𝑑 0 + 1 -𝑑 2 ) (notice that 𝑑 0 + 1 -𝑑 2 > 𝑑 1), and our update rule is such that O ( 𝑡 ) = 𝑒 𝐼 𝑑 0 + 2 -𝑑 2 -𝑒 𝐼 𝑑 0 + 1 -𝑑 2 and we obtain then:



The most inner induction step is now complete. We now notice that 𝑇 𝑑 0 ,𝑑 1 ,𝑑 0 + 1 -𝑑 1 = 𝑇 𝑑 0 ,𝑑 1 + 1, which proves the middle-induction step. And we notice that 𝑇 𝑑 0 ,𝑑 0 + 1 = 𝑇 𝑑 0 + 1, which proves the whole induction argument.

D.2 Proof of Eq. (17)

We only need to show that the following quantity is increasing in 𝑡

But, as shown in Appendix D, the update rule is such that we don't perform projections. It then follows easily from our update step. Indeed if we increase a set of coordinates by αη , then clearly the 𝑋 𝑡 only increases. Also, if we perform update of the form:

for two consecutive and equal coordinates, then for any 𝐼 𝐵 : If 𝐼 𝐵 includes same number of coordinates from 𝐼 𝑗 as in 𝐼 𝑗 + 1 then the magnitude doesn't change. If 𝐼 𝐵 contains more 𝐼 𝑗 then it increases, and if 𝐼 𝐵 contains more from 𝑖 𝑗 + 1 then consider 𝐼 𝐵 ′ that swaps coordinates in 𝐼 𝑗 with 𝐼 𝑗 + 1 then we clearly have:

E Dimension independent lower bound for GD

In this section we prove that the optimization error of GD in Eq. (5) is optimal. The lower bound is an optimization error for first order methods and is well established (see [10]). The point here is to show that the bound is valid in any dimension, for GD.

Claim 2. For every choice of η , 𝑇 , there exists a convex and 1 Lipschitz function 𝑓 ( 𝑥 ) : ℝ → ℝ such that, if we run GD on 𝑓 , then

Proof. We divide the proof into two cases:

case1: η ≥ 1 / η 𝑇 :

In this case we choose 𝑓 ( 𝑥 ) = | 𝑥 -γ | , where γ > 0 is a arbitrarily small (may depend on η ). It can be seen that for every even iteration, we have that:

As such, for every even iteration we have that 𝑥 2 𝑡 = 0, and at every odd iteration we have that 𝑥 2 𝑡 + 1 = η . We thus have that 𝑥 𝐺𝐷 = η 2 and

case1: η ≤ 1 / η 𝑇 ≤ 1 : In this case choose 𝑓 ( 𝑥 ) = α 𝑥 , where α = 1 ( 𝑇 + 1 ) η ≤ 1 . Then, one can show that GD outputs

at at every odd iteration and:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should state which version of the asset is used and, if possible, include a URL.

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows a new lower bound for the generalization error of gradient descent for Lipschitz functions. The bound shows a linear dependency on the dimension that closes the gap between lower and upper bounds in the sample complexity of GD under several regimes. The construction of the function relies on a block version of a variation of Nemirovski function and a reduction from a sample-independent oracle to a sample-dependent oracle. \nThe authors also propose a set of open problems related to the generalization error of GD.\n\nSTRENGTHS:\n- The paper shows a clever way to make changes and leverage Nemirovski and Feldman's function so as to control the dynamics of the sub-differentials\n- The authors try to give the best intuition for the construction of the function in the presentation of the paper\n- The paper is technically challenging and the proofs have been explained well. I have read some parts of the proofs and didn't see any problem.\n\nWEAKNESSES:\n- I think the main weakness in the paper is the presentation of the results and improvements. I know that these have all been written in the paper, but I found myself lost and rereading many times to see the significance of the results. This could be rewritten in a more concise way with all the improvements under which regimes spelled out explicitly (maybe a table would make sense).\n- There are a few typos: Line 294: sequences. Line 296: receive(s). Line 300: $S_{1:0}$, should S be bolded? Line 320: punctuation and line break.\n- It should be mentioned that Eq 16 will be proven in Appendix D.\n\nQUESTIONS:\n- I'm really confused by the fact that all bounds and the proofs use $F(0)$ for the gap. Why is this the case? Why isn't it $\\min_w F(w)$? If the algorithm starts at $w_0=0$, doesn't it mean the function value increases?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
The paper proved a tight lower bound for the sample complexity of full-batch gradient descent. The authors also presented some open questions in this area.

STRENGTHS:
I think the paper is not ready.

WEAKNESSES:
- The structure of the paper is not standard. There isn't any conclusion, and it seems that the paper is written in a very rushed manner.
- The notation section is not well-written.
- It seems that the main contribution of the paper is in Theorem 1, but it is not clear where its proof is located. The proof in the appendix is very short, and it appears that some parts of the proof are in the main body of the paper and some in the appendix. It is not organized very well.

QUESTIONS:
I couldn't understand the paper very well, so I do not have any questions. Please refer to the weaknesses section.

LIMITATIONS:
I think the paper is not ready for this conference, and I am not sure how important the result is.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the sample complexity of full-batch gradient descent under stochastic convex optimization problem. The main result of this work provides a lower bound of the generalization gap of GD of order $\\Omega(\\min\\\\{\\frac{d}{m},1\\\\} \\cdot \\min\\\\{\\eta d^{3/2}, \\eta\\sqrt{T},1\\\\})$, where $d$ denotes dimension, $m$ denotes sample size, and $\\eta,T$ denotes learning rate and number of iterations of GD. Notably, this lower bound, together with upper and lower bounds of GD or general ERM established in prior works, implies meaningful results under different regime. In the over-parametrized regime where $d=\\Omega(m+T^{1/3})$, the main result, together with lower bound of empirical risk of GD, yields a lower bound of $\\Omega(\\min\\\\{\\eta\\sqrt{T}+\\frac{1}{\\eta T}, 1\\\\})$. Moreover, when $T=O(m^{3/2})$ and $\\eta=\\Theta(1/\\sqrt{T})$, the main results yields a lower bound of $\\Omega(\\min\\\\{\\frac{d}{m}+\\frac{1}{\\sqrt{m}},1\\\\})$, matching the known upper bound of general ERM algorithms.\n\nSTRENGTHS:\nThis paper provides an improved lower bound on the sample complexity of full-batch gradient descent for SCO. Under several regimes, the lower bound match existing upper bounds of GD or general ERM and becomes optimal. Moreover, it implies that the worst case sample complexity of GD is the same as that of general ERM. These results contribute to better understanding the sample complexity of the SCO problem.\n\nWEAKNESSES:\nThe technical results are solid, but I think the writing could be improved. In particular, the connection between the high-level intuition provided in Sec 4 and the actual constructions in Sec 4.1 is not apparent. For example, it's not clear to me why the oracle $\\mathcal{O}^{(t)}$ defined in Eq. 11 satisfies the zero-chain property depicted in Figure 1.\n\nQUESTIONS:\n- From Thm 1 to Cor 3 when $\\eta=\\Theta(1/\\sqrt{T})=\\Omega(m^{-3/4})$ and $d\\le\\sqrt{m}$: I agree that the first term in Thm 1 is lower bounded by $\\min\\\\{\\frac{d}{m},1\\\\} \\ge \\min\\\\{\\frac{1}{\\sqrt{m}},1\\\\}$, but the scond term $\\eta d^{3/2} \\gtrsim d^{3/2}m^{-3/4}$ could potentially be smaller than 1. If so, should the lower bound actually be $\\min\\\\{\\frac{1}{\\sqrt{m}},1\\\\}\\cdot \\min\\\\{\\frac{d^{3/2}}{m^{3/4}}, 1\\\\}$?\n\n- line 219: could the authors elaborate why this new oracle defined in 214-218 only activates $O(d)$ coordinates in $d^3$ iterations? From my intuition, to activate a new coordinate, say $k+1$, the oracle needs to increase all previous coordinates to $k$. Then this would only require $d^2$ iterations to activate $O(d)$ coordinates. I think I miss something here.\n\nMinor comments:\n- It would be clear to clarify early in the paper that $w(i)$ denotes the $i$-th coordinate of a vector $w$.\n- 194: $\\partial N(0)$ => $\\partial N_0(0)$\n- 299: Let => let\n- 300: $\\mathbf{S}\\_{1:0}$ => $S\\_{1:0}$\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
### Summary:
 

The authors study the generalization error and sample complexity of GD for GD in stochastic CO. Their results show that one can achieve the generalization error of $\tilde{\Theta}(d/m + 1/\sqrt{m)}$ which is the same as the generalization error of ERM. Indeed, they prove that the linear dependence to dimension $d$ in unavoidable. Moreover, they show that $1/\epsilon^4$ steps are needed for GD to avoid overfitting where $\epsilon$ is the optmality gap.

STRENGTHS:
### Pros:

- interesting theoretical problem/results
- excellent citation to related work
- extremely well-written

WEAKNESSES:
### Cons:

- some discussion about SGD is missing

QUESTIONS:
### Questions/Comments:


This is an excellent paper, I recomment acceptance as I found the paper well-written and the contributions/motivations are clear. 

- Can you explain how this analysis restricts to full-batch GD and what makes it impossible to obtain results for SGD? I recommend adding a bit of discussion to the paper regarding this.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-24