Title: Tight Risk Bounds for Gradient Descent on Separable Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Convex optimization

Gradient Descent

separable data

generalization bounds

Stochastic Gradient Descent.
Abstract: We study the generalization properties of unregularized gradient methods applied to separable linear classification---a setting that has received considerable attention since the pioneering work of Soudry et al. (2018).
We establish tight upper and lower (population) risk bounds for gradient descent in this setting, for any smooth loss function, expressed in terms of its tail decay rate.
Our bounds take the form $\Theta(r_{\ell,T}^2 / \gamma^2 T + r_{\ell,T}^2 / \gamma^2 n)$, 
where $T$ is the number of gradient steps, $n$ is size of the training set, $\gamma$ is the data margin, and $r_{\ell,T}$ is a complexity term that depends on the tail decay rate of the loss function (and on $T$).
Our upper bound greatly improves the existing risk bounds due to Shamir (2021) and Schliserman and Koren (2022), that either applied to specific loss functions or imposed extraneous technical assumptions, and applies to virtually any convex and smooth loss function.
Our risk lower bound is the first in this context and establish the tightness of our general upper bound for any given tail decay rate and in all parameter regimes.
The proof technique used to show these results is also markedly simpler compared to previous work, and is straightforward to extend to other gradient methods; we illustrate this by providing analogous results for Stochastic Gradient Descent.

Tight Risk Bounds for Gradient Descent on Separable Data

Abstract

We study the generalization properties of unregularized gradient methods applied to separable linear classification-a setting that has received considerable attention since the pioneering work of Soudry et al. [14]. We establish tight upper and lower (population) risk bounds for gradient descent in this setting, for any smooth loss function, expressed in terms of its tail decay rate. Our bounds take the form Θ ( 𝑟 2 ℓ,𝑇 / 𝛾 2 𝑇 + 𝑟 2 ℓ,𝑇 / 𝛾 2 𝑛 ) , where 𝑇 is the number of gradient steps, 𝑛 is size of the training set, 𝛾 is the data margin, and 𝑟 ℓ,𝑇 is a complexity term that depends on the tail decay rate of the loss function (and on 𝑇 ). Our upper bound greatly improves the existing risk bounds due to Shamir [13] and Schliserman and Koren [12], that either applied to specific loss functions or imposed extraneous technical assumptions, and applies to virtually any convex and smooth loss function. Our risk lower bound is the first in this context and establish the tightness of our general upper bound for any given tail decay rate and in all parameter regimes. The proof technique used to show these results is also markedly simpler compared to previous work, and is straightforward to extend to other gradient methods; we illustrate this by providing analogous results for Stochastic Gradient Descent.

1 Introduction

Recently, there has been a marked increase in interest regarding the generalization capabilities of unregularized gradient-based learning methods. One specific area of attention in this context has been the setting of linear classification with separable data, where a pioneering work by Soudry et al. [14] showed that, when using plain gradient descent to minimize the empirical risk on a linearly separable training set with an exponentially-tailed classification loss (such as the logistic loss), the trained predictor will asymptotically converge in direction to the max-margin solution. As a result, standard margin-based generalization bounds for linear predictors suggest that, provided the number of gradient steps ( 𝑇 ) is sufficiently large, the produced solution will not overfit, despite the lack of explicit regularization and the fact that its magnitude (i.e., Euclidean norm) increases indefinitely with 𝑇 . This result has since been extended to incorporate other optimization algorithms and loss functions [3, 4, 8, 9, 5].

Despite the high interest in this problem, the tight finite-time (population) risk performance of unregularized gradient methods, and even just of gradient descent, have not yet been fully understood. The convergence to a high margin solution exhibited by Soudry et al. [14] (for the logistic loss) occurs at a slow logarithmic rate, thus, the risk bounds for the trained predictors only become effective when 𝑇 is at least exponentially large in comparison to the size of the training set 𝑛 and the margin 𝛾 . In a more recent work, Shamir [13] established risk bounds of the form e 𝑂 ( 1 / 𝛾 2 𝑇 + 1 / 𝛾 2 𝑛 ) for several gradient methods in this setting (still with the logistic loss), that apply for smaller and more realistic values of 𝑇 .

a The lower bound is shown for the empirical (logistic) risk.

c ℓ is ( 𝑐 1 , 𝑐 2 ) -quadratically-bounded if, for every 𝑥 and 𝑧 , | ℓ ′ ( 𝑥, 𝑧 )| ≤ 𝑐 1 + 𝑐 2 (∥ 𝑥 ∥ + ∥ 𝑧 ∥) .

Later Schliserman and Koren [12] gave a more general analysis that extends the bounds of Shamir [13] to a wide range of smooth loss functions satisfying a certain 'self-boundedness' condition, and Telgarsky [16] provided a test-loss analysis of stochastic mirror descent on 'quadratically-bounded' losses. However, all of these works either assume a specific loss (e.g., the logistic loss), impose various conditions on the loss function (beyond smoothness), or do not establish the tightness of their bounds in all regimes of parameters ( 𝑇, 𝑛 and 𝛾 ), and especially in the regime where the number of gradient steps 𝑇 is larger than the sample size 𝑛 (see Table 1 for details).

1.1 Our contributions

In this work, we close these gaps by showing nearly matching upper and lower risk bounds for gradient descent, which hold essentially for any smooth loss function, and for any number of steps 𝑇 and sample size 𝑛 . Compared to recent prior work in this context [13, 16, 12], our results do not require any additional assumptions on the loss function besides its smoothness, and they strictly improve upon the existing bounds by their dependence on 𝑇 . Further, to the best of our knowledge, risk lower bounds were not previously explored in this context, and the lower bounds we give establish, for the first time, the precise risk convergence rate of gradient descent for any smooth loss function and in all regimes of 𝑇 and 𝑛 .

In some more detail, our results assume the following form. Let C 𝜙,𝛽 be the class of nonnegative, convex and 𝛽 -smooth loss functions ℓ ( 𝑢 ) that decay to zero (as 𝑢 →∞ ) faster than a reference 'tail function' 𝜙 : [ 0 , ∞) → ℝ . The function 𝜙 is merely used to quantify how rapidly the tails of loss functions in C 𝜙,𝛽 decay to zero. Then, the risk upper and lower bounds we prove for gradient descent are of the following form: 2

The bounds depend on the tail function 𝜙 through the term 𝑟 𝜙,𝑇 that, roughly, equals 𝜙 -1 ( 𝜀 ) for 𝜀 chosen such that 𝜀 ≈ ( 𝜙 -1 ( 𝜀 )) 2 GLYPH<14> 𝛾 2 𝑇 (for the precise statements refer to Theorems 1 and 2, and for concrete examples of implied bounds, see Table 2).

The form of Eq. (1) resembles the bounds given in recent work by Schliserman and Koren [12]. However, the latter work imposed an extraneous 'self-boundedness' assumption (stronger than smoothness, see Table 1) that we do not require, and they did not establish the tightness of their bounds, as we do in this paper by providing matching lower bounds. On the flip side, their upper bounds apply in a broader stochastic convex optimization setup, whereas our bounds are specialized to generalized linear models for classification.

In terms of techniques, our proof methodology is also distinctly (and perhaps somewhat surprisingly) simple compared to previous work. Our upper bounds rely on two elementary properties that gradient methods admit when applied to a smooth and realizable objective: low training loss and low norm of the produced solutions. Both properties are obtained from fairly standard arguments and convergence bounds for smooth gradient descent, when paired with conditions implied by the decay rate of the loss function. Finally, to bound the gap between the population risk and the empirical risk, we employ a classical result of Srebro et al. [15] that bounds the generalization gap of linear models in the so-called 'low-noise' (i.e., nearly realizable) smooth regime using local Rademacher complexities. Somewhat surprisingly, this simple combination of tools already give sharp risk results that improve upon the state-of-the-art [13, 12, 16] both in terms of tightness of the bounds and the light set of assumptions they rely on.

We remark here that the proof scheme summarized above can be generalized to essentially any gradient method for which one can prove simultaneously bounds on the optimization error and the norm of possible solutions produced by the algorithm. In the sequel, we focus for concreteness on standard gradient descent, but in Appendix B we also give bounds for Stochastic Gradient Descent (SGD), which is shown to admit both of these properties with high probability over the input sample.

An interesting conclusion from our bounds pertains to the significance of early stopping. We see that gradient descent, even when applied on a smooth loss functions that decay rapidly to zero, might overfit with respect the the surrogate loss (i.e., reach a trivial Θ ( 1 ) risk) as the number of steps 𝑇 grows. The time by which gradient descent starts overfitting depends on the tail decay rate of the loss function: the slower the decay rate, the shorter the time it takes to overfit. Interestingly, a similar phenomenon may not occur for the zero-one loss of the trained predictor. For example, Soudry et al. [14] show that with the logistic loss, gradient descent does not overfit in terms of the zero-one as 𝑇 approached infinity-whereas, at the same time, it does overfit as 𝑇 →∞ in terms of the logistic loss itself, as seen from our lower bounds.

Summary of contributions. To summarize, the main contribution of this paper are as follows:

Our first main result (in Section 3) is a high probability risk upper bound for gradient descent in the setting of separable classification with a convex and smooth loss function. Our bound matches the best known upper bounds [13, 12] and greatly extend them to allow for virtually any convex and smooth loss function, considerably relaxing various assumptions imposed by prior work.

Our second main result (in Section 4) is a nearly matching lower bound for the risk of gradient descent, establishing the tightness of our analysis given the smoothness and tail-decay conditions. The tightness of our bounds holds across tail decay rates and different regimes of the parameters 𝑇, 𝑛 and 𝛾 . To our knowledge, these constitute the first (let alone tight) risk lower bounds in this context, despite a long line of work on linear classification with separable data.

Wealso provide analogous results for Stochastic Gradient Descent with replacement (in Appendix B), mainly to emphasize that that our analysis uses only two elementary properties of the optimization algorithm: low optimization error and low norm of the produced solution. The same analysis can be generalized to any gradient method that admits these two properties.

1.2 Additional related work

Unregularized gradient methods on separable data. The most relevant work to ours is of Schliserman and Koren [12], who used algorithmic stability and two simple conditions of selfboundedness and realizability which the loss functions hold, to get generalization bounds for gradient methods with constant step size in the general setting of stochastic convex and smooth optimization. Then, they derived risk bounds which hold in expectation for the setting of linear classification with separable data for every loss function which decays to 0. The exact bound was depend in the rate of decaying to 0 of the function. For the Lipschitz case their risk bound with respect to the loss function ℓ is 𝑂 ( ℓ -1 ( 𝜀 ) 2 GLYPH<14> 𝛾 2 𝑇 + ℓ -1 ( 𝜀 ) 2 GLYPH<14> 𝛾 2 𝑛 ) for any choice of 𝜀 such that 𝜀 / ℓ -1 ( 𝜀 ) 2 ≤ 1 / 𝛾 2 𝑇 . For example, for the logistic loss, the bound translates to 𝑂 ( log 2 ( 𝑇 ) GLYPH<14> 𝛾 2 𝑇 + log 2 ( 𝑇 ) GLYPH<14> 𝛾 2 𝑛 ) .

In another work, Telgarsky [16], showed a high probability risk bound for 𝑇 ≤ 𝑛 for Batch Mirror Decent with step size 𝜂 ≃ 1 / √ 𝑇 in linear models, using a reference vector, which when selected properly, can be translated to a risk bound of 𝑂 ( ℓ -1 ( 𝜀 ) GLYPH<14> 𝛾 √ 𝑇 ) for gradient descent applied on the loss function ℓ , and to a 𝑂 ( log 𝑇 GLYPH<14> 𝛾 √ 𝑇 ) for the logistic loss.

Fast rates for smooth and realizable optimization. The problem of smooth and realizable optimization, also known as the 'low-noise' regime of stochastic optimization, is a very well researched problem. Srebro et al. [15] showed that stochastic gradient descent achieved risk bound of 𝑂 ( 1 / 𝑛 ) in this setting. For linear models, they also showed that ERM achieve similar fast rates by using local Rademacher complexities. Later [10] showed that SGD converges linearly when the loss function is also strongly convex. In more recent works, [7] used stability arguments to show that SGD with replacement with 𝑇 = 𝑛 achieve risk of 𝑂 ( 1 / 𝑛 ) .

Lower bounds. A lower bound related to ours appears in Ji and Telgarsky [4]. In this work, the authors showed a lower bound of ∥ 𝑤 ′ 𝑡 -𝑤 ∗ ∥ ≥ log ( 𝑛 )/ log ( 𝑇 ) . In our work, however, we get lower bound directly for the loss itself and not for this objective. More recently, Shamir [13] showed a lower bound of Ω ( 1 / 𝛾 2 𝑇 ) for the empirical risk of GD when applied on the logistic loss which is tight up to log factors. When generalizing this technique for other objectives, e.g., functions that decay polynomially to zero, the log factors become polynomial factors and the bound becomes not tight, even in the regime 𝑇 ≪ 𝑛 where the 1 / 𝑇 term in the bounds is dominant. In contrast, we establish nearly tight bounds for virtually any tail decay rate, and in all regimes of 𝑇 and 𝑛 .

2 Problem Setup

We consider the following typical linear classification setting. Let D be distribution over pairs ( 𝑥, 𝑦 ) , where 𝑥 ∈ ℝ 𝑑 is a 𝑑 -dimensional feature vector and 𝑦 ∈ ℝ is a real number that represents the corresponding label. We assume that data is scaled so that ∥ 𝑥 ∥ ≤ 1 and | 𝑦 | ≤ 1 (with probability one with respect to D ). We focus on the setting of separable , or realizable , linear classification with margin. Formally, we make the following assumption.

Assumption 1 (realizability) . There exists a unit vector 𝑤 ∗ ∈ ℝ 𝑑 and 𝛾 > 0 such that 𝑦 ( 𝑤 ∗ · 𝑥 ) ≥ 𝛾 almost surely with respect to the distribution D .

Equivalently, we will identify each pair ( 𝑥, 𝑦 ) with the vector 𝑧 = 𝑦𝑥 , and realizability implies that 𝑤 ∗ · 𝑧 ≥ 𝛾 with probability 1. Our assumptions then imply that ∥ 𝑧 ∥ ≤ 1 with probability 1.

Given a nonnegative loss function ℓ : ℝ → ℝ + , the objective is to determine a model 𝑤 ∈ ℝ 𝑑 that minimizes the (population) risk, defined as the expected value of the loss function over the distribution

D , namely

2.1 Loss functions

The loss functions ℓ considered in this paper are nonnegative, convex and 𝛽 -smooth. 3 We also require that ℓ is strictly monotonically decreasing and lim 𝑢 →∞ ℓ ( 𝑢 ) = 0. The vast majority of loss functions used in supervised learning for classification satisfy these conditions; these include, for example, the logistic loss ( ℓ ( 𝑢 ) = log ( 1 + 𝑒 -𝑢 ) ), the probit loss ( ℓ ( 𝑢 ) = -log ( 1 2 -1 2 erf ( 𝑢 )) ), and the squared hinge loss ( ℓ ( 𝑢 ) = ( max { 1 -𝑢, 0 }) 2 ).

Amain goal of this paper is to quantify how do the achievable bounds on the risk depend on properties of the loss function ℓ used, and most crucially on the rate in which ℓ decays to zero as its argument approaches infinity. To formalize this, we need a couple of definitions.

Definition 1 (tail function) . We say that 𝜙 : [ 0 , ∞) → ℝ is a tail function if 𝜙

(i) is a nonnegative, 1-Lipschitz and 𝛽 -smooth convex function;

(ii) is strictly monotonically decreasing such that lim 𝑢 →∞ 𝜙 ( 𝑢 ) = 0;

(iii) satisfies 𝜙 ( 0 ) ≥ 1 2 and | 𝜙 ′ ( 0 )| ≥ 1 2 .

Every tail function 𝜙 defines a class of loss functions characterized by the rate 𝜙 decays to zero.

Definition 2 ( 𝜙 -tailed class) . For a tail function 𝜙 , the class C 𝜙,𝛽 is the set of all nonnegative, convex, 𝛽 -smooth and monotonically decreasing loss functions ℓ : ℝ → ℝ + such that ℓ ( 𝑢 ) ≤ 𝜙 ( 𝑢 ) for all 𝑢 ≥ 0.

We detail several examples for tail functions in Appendix A.

2.2 Gradient Descent

The algorithm that we focus in this paper is standard gradient descent (GD) with a fixed step size 𝜂 > 0 applied to the empirical risk b 𝐿 ; this method is initialized at 𝑤 1 = 0 and at each step 𝑡 = 1 , . . . , 𝑇 performs an update

The algorithm returns the final model, 𝑤 𝑇 .

We remark however that most of the results we present in the sequel can be straightforwardly adapted to other gradient methods; we include results for Stochastic Gradient Descent (SGD) in Appendix B, and the same proof techniques can be used to analyze multi-epoch and/or mini-batched SGD, gradient flow, and more.

3 Risk Upper Bounds

We begin by giving a general upper bound for the risk of gradient descent, when the loss function ℓ is taken from the class C 𝜙,𝛽 . Our main result in this section is the following.

Theorem 1. Let 𝜙 be a tail function and let ℓ be any loss function from the class C 𝜙,𝛽 . Fix 𝑇 , 𝑛 and 𝛿 > 0 . Then, with probability at least 1 -𝛿 (over the random sample 𝑆 of size 𝑛 ), the output of GD applied on b 𝐿 with step size 𝜂 ≤ 1 / 2 𝛽 initialized at 𝑤 1 = 0 has

for any 𝜀 ≤ 1 2 such that 𝜂𝛾 2 𝑇 ≤ ( 𝜙 -1 ( 𝜀 )) 2 / 𝜀 , where 𝐾 < 10 5 is a numeric constant.

For finding such a model, we use a set of training examples 𝑆 = { 𝑧 1 , ..., 𝑧 𝑛 } which drawn i.i.d. from D and an empirical proxy, the empirical risk , which is defined as

The expression ( 𝜙 -1 ( 𝜀 )) 2 / 𝜀 increases indefinitely as 𝜀 approaches 0; therefore, for any 𝑇 , there exists an 𝜀 > 0 that satisfies the theorem's condition. For several examples of how this bound is instantiated for different tail decay functions 𝜙 , see Table 2 and Appendix A.

In the remainder of this section we prove Theorem 1. The structure of the the proof will be as follows: First, we bound the norm of the GD solution; by smoothness and realizability, we get that the norm will remain small compared to a reference point with small loss value. Second, we get a bound on the optimization error in this setting, the relies on the same reference point. Finally, we use a fundamental result due to Srebro et al. [15] (reviewed in the subsection below) together with both bounds to derive the risk guarantee. As discussed broadly in the introduction, this proof scheme can be generalized to other gradient methods which satisfy the properties of model with low norm and low optimization error.

3.1 Preliminaries: Uniform Convergence Using Rademacher Complexity

One property of linear models is that in this class of problems is that we have dimension-independent and algorithm-independent uniform convergence bounds, that enables to bound the difference between the empirical risk and the population risk of a specific model. A main technical tool for bounding this difference is the Rademacher Complexity [2]. The worst-case Rademacher complexity of an hypothesis class 𝐻 for any sample size 𝑛 is given by:

We are interested in models that achieve low empirical risk on smooth objectives. A fundamental result of Srebro et al. [15] bounds the generalization gap under such conditions:

Proposition 1 (15, Theorem 1) . Let 𝐻 be a hypothesis class with respect to some non negative and 𝛽 -smooth function, ℓ ( 𝑡 · 𝑦 ) , such that for every 𝑤 ∈ 𝐻, 𝑥, 𝑦 , | ℓ ( 𝑤𝑥 · 𝑦 )| ≤ 𝑏 . Then, for any 𝛿 > 0 we have, with probability at least 1 -𝛿 over a sample of size 𝑛 , uniformly for all ℎ ∈ 𝐻 ,

«

«

where 𝐾 < 10 5 is a numeric constant.

3.2 Properties of Gradient Descent on Smooth Objectives

In this section we prove that GD satisfies the two desired properties- low norm and low optimization error. We begin with showing that the norm of 𝑤 𝑇 , the output of GD after 𝑇 iterations, is low, as stated in the following lemma,

Lemma 1. Let 𝜙 be a tail function and let ℓ ∈ C 𝜙,𝛽 . Fix any 𝜀 > 0 and a point 𝑤 ∗ 𝜀 ∈ ℝ 𝑑 such that b 𝐿 ( 𝑤 ∗ 𝜀 ) ≤ 𝜀 (exists due to realizability). Then, the output of 𝑇 -steps GD, applied on b 𝐿 with stepsize 𝜂 ≤ 1 / 𝛽 initialized at 𝑤 1 = 0 has,

Now, we bound the optimization error of GD on every function ℓ ∈ C 𝜙,𝛽 , by using a variant of Lemma 13 from [12]. The proof is fairly standard and appears in Appendix C.

Lemma 2. Let 𝜙 be a tail function and let ℓ ∈ C 𝜙,𝛽 . Fix any 𝜀 > 0 and a point 𝑤 ∗ 𝜀 ∈ ℝ 𝑑 such that b 𝐿 ( 𝑤 ∗ 𝜀 ) ≤ 𝜀 . Then, the output of 𝑇 -steps GD, applied on b 𝐿 with stepsize 𝜂 ≤ 1 / 𝛽 initialized at 𝑤 1 = 0 has,

‹

‹

3.3 Proof of Theorem 1

We now turn to prove Theorem 1. The proof is a simple consequence of the properties proved above and Proposition 1. We first claim that there exists a model 𝑤 ∗ 𝜀 with low norm such that b 𝐿 ( 𝑤 ∗ 𝜀 ) ≤ 𝜀 ,

which implies, through Lemmas 1 and 2, that 𝑤 𝑇 of gradient descent has both low optimization error and it remains bounded within a ball of small radius. Then, we use Proposition 1 to translate the low optimization error to low risk.

Proof of Theorem 1. First, we show that there exists a model 𝑤 ∗ 𝜀 with low norm such that b 𝐿 ( 𝑤 ∗ 𝜀 ) ≤ 𝜀 . Let 𝜀 ≤ 1 2 and ℓ ∈ C 𝜙,𝛽 . By separability, there exists a unit vector 𝑤 ∗ such that 𝑤 ∗ · 𝑧 𝑖 ≥ 𝛾 for every 𝑧 𝑖 in the training set 𝑆 . Moreover, ℓ is monotonic decreasing. Then, for 𝑤 ∗ 𝜀 = GLYPH<0> 𝜙 -1 ( 𝜀 )/ 𝛾 GLYPH<1> 𝑤 ∗ and every 𝑧 𝑖 ∈ 𝑆 ,

Then, by the fact that 𝜙 -1 ( 𝜀 ) ≥ 0, we have ℓ ( 𝑤 ∗ 𝜀 · 𝑧 𝑖 ) ≤ ℓ ( 𝜙 -1 ( 𝜀 )) ≤ 𝜙 ( 𝜙 -1 ( 𝜀 )) = 𝜀 for all 𝑖 , hence

Now, for 𝜀 such that 𝜂𝛾 2 𝑇 ≤ ( 𝜙 -1 ( 𝜀 )) 2 / 𝜀 , we get by Lemma 1,

Denote 𝐵 𝜀 = { 𝑤 : ∥ 𝑤 ∥ ≤ 𝑟 𝜀 } , where for brevity 𝑟 𝜀 = 4 𝜙 -1 ( 𝜀 ) GLYPH<14> 𝛾 . We have, by Lemma 10, 𝑓 ( 𝑥 ) ≤ 2 𝑓 ( 𝑦 ) + 𝛽 ∥ 𝑥 -𝑦 ∥ 2 for all 𝑥, 𝑦 ∈ ℝ 𝑑 (see proof in Appendix C). Then, together with the fact that ∥ 𝑧 ∥ , ∥ 𝑧 ′ ∥ ≤ 1 and choosing 𝜀 such that 𝜀 ≤ 𝜙 -1 ( 𝜀 ) 2 GLYPH<14> 𝛾 2 𝜂𝑇 , with probability 1,

Moreover, 𝐵 𝜀 is hypothesis class of linear predictors with norm at most 𝑟 𝜀 . We know that the norm of the examples is at most 1, thus, it follows that the Rademacher complexity of 𝐵 𝜀 is 𝑅 𝑛 ( 𝐵 𝜀 ) = 𝑟 𝜀 GLYPH<14> √ 𝑛 [e.g., 6, Theorem 3].

Now, by the choice of 𝜀 , we have b 𝐿 ( 𝑤 𝑇 ) ≤ 3 𝑟 2 𝜀 GLYPH<14> 16 𝜂𝑇 . Thus, Proposition 1 implies that with probability at least 1 -𝛿 , for every 𝑤 ∈ 𝐵 𝜀 and any 𝜀 such that 𝜀 ≤ 𝜙 -1 ( 𝜀 ) 2 GLYPH<14> 𝛾 2 𝜂𝑇 ,

« « ‹ ‹ Plugging in the bound on 𝑏 , dividing by 𝑟 2 𝜀 and using twice the fact that 𝑥𝑦 ≤ 1 2 𝑥 2 + 1 2 𝑦 2 for all 𝑥, 𝑦 ,

The theorem follows by rearranging the inequality.

□

For the same 𝜀 , by Lemma 2,

4 Risk Lower Bounds

In this section we present our second main result: a lower bound showing that the bound we proved in Section 3 for gradient descent is essentially tight for loss functions in the class C 𝜙,𝛽 , for any given tail function 𝜙 and any 𝛽 > 0. Formally, we prove the following theorem.

Theorem 2. There exists a constant 𝐶 such that the following holds. For any tail function 𝜙 , sample size 𝑛 ≥ 35 and any 𝑇 , there exist a distribution D and a loss function ℓ ∈ C 𝜙,𝛽 , such that for 𝑇 -steps GD over a sample 𝑆 = { 𝑧 𝑖 } 𝑛 𝑖 = 1 sampled i.i.d. from D , initialized at 𝑤 1 = 0 with stepsize 𝜂 ≤ 1 / 2 𝛽 , it holds that

We remark that the right-hand side of the bound is well defined, as we restrict 𝜀 to be sufficiently small so as to ensure that all arguments to 𝜙 -1 are at most 1 2 (recall that 𝜙 admits all values in [ 0 , 1 2 ] due to our assumptions that 𝜙 ( 0 ) ≥ 1 2 ). Further, the lower bound above matches the upper bound given in Theorem 1 up to constants, unless the tail function 𝜙 decays extremely slowly, and slower than any polynomial (at this point, however, the entire bound becomes almost vacuous).

To prove Theorem 2, we consider two different regimes: the first is where 𝑇 ≫ 𝑛 , when the first term in the right-hand side of the bound is dominant; and the 𝑇 ≪ 𝑛 regime where the second term is dominant. We begin by focusing on the first regime, and prove the following.

Lemma 3. There exists a constant 𝐶 1 such that the following holds. For any tail function 𝜙 , sample size 𝑛 ≥ 35 and any 𝛾 and 𝑇 , there exist a distribution D with margin 𝛾 , a loss function ℓ ∈ C 𝜙,𝛽 such that for GD over a sample 𝑆 = { 𝑧 𝑖 } 𝑛 𝑖 = 1 sampled i.i.d. from D , initialized at 𝑤 1 = 0 with stepsize 𝜂 ≤ 1 / 2 𝛽 , it holds that

To prove the lemma, we construct a learning problem for which the risk of GD can be lower bounded. We design a loss function ℓ which exhibits a decay to zero at a rate similar to that of the function 𝜙 for 𝑥 > 0, and behaves as a quadratic function for 𝑥 ≤ 0. The distribution D is constructed such that there are two fixed examples in its support, denoted 𝑧 1 and 𝑧 2, that are both sampled with constant probability and are almost opposite in direction to each other: 𝑧 2 is aligned with -𝑧 1 except for a small component, denoted 𝑣 , which is orthogonal to 𝑧 1. Crucially, since 𝑧 1 and 𝑧 2 point in nearly opposite directions, minimizing optimization error requires the GD iterate to have a substantial component aligned with the vector 𝑣 (otherwise, the loss for 𝑧 2 will be large). Finally, we define another example, denoted as 𝑧 3, that has a significant component in the opposite direction -𝑣 . We arrange the distribution D so that 𝑧 3 is sampled with with probability roughly 1 / 𝑛 . The lower bound now follows by observing that, if 𝑧 3 is sampled at test time, but did not appear in the training set (this happens with probability ≈ 1 / 𝑛 ), the test error is quadratically large in the magnitude of the GD iterate along -𝑣 .

We now turn to formally proving Lemma 3.

Proof. Given 𝛾 ≤ 1 8 , let us define the following distribution D :

First, note that the distribution is separable: for 𝑤 ∗ = ( 𝛾, 1 2 , 1 4 ) it holds that 𝑤 ∗ 𝑧 𝑖 = 𝛾 for every 𝑖 ∈ { 1 , 2 , 3 } . Moreover, Lemma 11 in Appendix D ensures that indeed ℓ ∈ C 𝜙,𝛽 .

and loss function:

Next, let 𝑆 be a sample of 𝑛 i.i.d. examples from D and let 𝑧 ′ ∼ D be a validation example independent from 𝑆 . Denote by 𝛿 2 ∈ [ 0 , 1 ] the fraction of appearances of 𝑧 2 in the sample 𝑆 , and by 𝐴 1 , 𝐴 2 the following events;

In Lemma 12 (found in Appendix D), we show that

Furthermore, as in the proof of Theorem 1, there exists a vector 𝑤 ∗ 𝜀 which holds ∥ 𝑤 ∗ 𝜀 ∥ ≤ 𝜙 -1 ( 𝜀 ) GLYPH<14> 𝛾 . Then by Lemma 2 and the choice of 𝜀 ,

For the remainder of the proof, we condition on the event 𝐴 1 ∩ 𝐴 2. First, we show that 𝑤 𝑡 · 𝑧 2 ≥ 0. Indeed, if it were not the case, then ℓ ( 𝑤 𝑇 · 𝑧 2 ) > 𝜙 ( 0 ) ; together with Eq. (6) we obtain

which is a contradiction to 𝜙 ( 0 ) ≥ 1 2 . Moreover, 𝑤 𝑇 ( 1 ) ≥ 0. Again, we show this by contradiction. Conditioned on 𝐴 2, we have 𝛿 1 > 7 8 . Then, if 𝑤 𝑇 ( 1 ) < 0, ℓ ( 𝑤 𝑇 · 𝑧 1 ) > 𝜙 ( 0 ) , and

which is a contradiction. In addition, we notice that 𝑧 3 is the only possible example whose third entry is non zero. Given the event 𝐴 1, we know that 𝑧 3 is not in 𝑆 . Equivalently, for every 𝑧 ∈ 𝑆 , 𝑧 ( 3 ) = 0. As a result,

Then, we get that,

Then, using the fact that 𝑤 𝑇 · 𝑧 2 ≥ 0, ℓ ( 𝑤 𝑇 · 𝑧 2 ) = 𝜙 ( 𝑤 𝑇 · 𝑧 2 ) , and conditioned on 𝐴 2, we have

Therefore, by combining Eq. (8) with the fact that 𝑤 𝑡 ( 1 ) ≥ 0,

which implies, 𝑤 𝑇 ( 2 ) ≥ 1 3 𝛾 𝜙 -1 ( 32 b 𝐿 ( 𝑤 𝑇 )) . By Eq. (7),

We therefore see that for every 𝜀 such that 𝜀 ≥ ( 𝜙 -1 ( 𝜀 )) 2 GLYPH<14> 𝛾 2 𝑇𝜂 ,

where in the final inequality we again used Eq. (6). We conclude the proof using Eq. (5) and the law of total expectation,

(Expectations here are taken with respect to both the sample 𝑆 and the validation example 𝑧 ′ .) □

which implies

Second, we focus the second expression in the lower bound of Theorem 2, which is dominant in the early stages of optimization.

Lemma 4. There exists a constant 𝐶 2 such that the following holds. For any tail function 𝜙 , and for any 𝑛, 𝑇 and 𝛾 , there exist a distribution D with margin 𝛾 , a loss function ℓ ∈ C 𝜙,𝛽 such that for GD initialized at 𝑤 1 = 0 with stepsize 𝜂 ≤ 1 / 2 𝛽 over an i.i.d. sample 𝑆 = { 𝑧 𝑖 } 𝑛 𝑖 = 1 from D and 𝑤 1 = 0 holds

for any 𝜀 ≤ 1 16 such that 𝜂𝛾 2 𝑇 ≥ 𝜙 -1 ( 𝜀 ) 2 GLYPH<14> 𝜀 .

The argument for proving Lemma 4 is similar to that of Lemma 3. Here we define a 1-Lipschitz loss function ℓ that decays to zero at the same rate as 𝜙 , and a distribution such that there is a possible example 𝑧 1 which is sampled with high probability and an almost 'opposite' example 𝑧 2, that with constant probability, appears rarely in the training set 𝑆 . The lower bound follows from the fact that although 𝑧 2 appears in the dataset, the gradients of the loss function are bounded and thus not large enough so as to make the trained predictor classify 𝑧 2 correctly. The full proof can be found in Appendix D.2.

Finally, we derive Theorem 2 directly from Lemmas 3 and 4:

Proof of Theorem 2. Let 𝐶 = 1 2 min { 𝐶 1 , 𝐶 2 } , where 𝐶 1 and 𝐶 2 are the constants from Lemmas 3 and 4, respectively. If ( 𝜙 -1 ( 8 𝜀 )) 2 GLYPH<14> 𝛾 2 𝑇𝜂 ≥ 𝛽 ( 𝜙 -1 ( 128 𝜀 )) 2 GLYPH<14> 𝛾 2 𝑛 , the theorem follows from Lemma 4; otherwise, it follows from Lemma 3. □

Acknowledgments

Funded by the European Union (ERC, OPTGEN, 101078075). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. This work received additional support from the Israel Science Foundation (grant number 2549/19), from the Len Blavatnik and the Blavatnik Family foundation, and from the Adelis Foundation.

References

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300 , 2018.

Ziwei Ji and Matus Telgarsky. A refined primal-dual analysis of the implicit bias. Journal of Environmental Sciences (China) English Ed , 2019.

Ziwei Ji, Miroslav Dudík, Robert E. Schapire, and Matus Telgarsky. Gradient descent follows the regularization path for general losses. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 2109-2136. PMLR, 09-12 Jul 2020.

Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2008.

Yunwen Lei and Yiming Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning , pages 5809-5819. PMLR, 2020.

Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3420-3428. PMLR, 2019.

MorShpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3051-3059. PMLR, 2019.

Deanna Needell, Rachel Ward, and Nati Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003.

Matan Schliserman and Tomer Koren. Stability vs implicit bias of gradient methods on separable data and beyond. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 3380-3394. PMLR, 02-05 Jul 2022.

Ohad Shamir. Gradient methods never overfit on separable data. The Journal of Machine Learning Research , 22(1):3847-3866, 2021.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1):2822-2878, 2018.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems , volume 23. Curran Associates, Inc., 2010.

Matus Telgarsky. Stochastic linear optimization never overfits with quadratically-bounded losses on general data. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 5453-5488. PMLR, 02-05 Jul 2022.

Peer reviews:
comments: SUMMARY:
This paper considers training a convex linear model with gradient descent on linearly separable data. The paper improves the upper bound of the population risk and proves a matching lower bound.

STRENGTHS:
The paper closes the gap for training convex linear models with smooth loss functions on linearly separable data, by showing matching upper and lower risk bounds. I believe this can be an important work in this field.  Also, the paper is well-written and the logic is easy to follow. The proof is a bit heavy but the authors managed to make it accessible.

WEAKNESSES:
I didn't find a major flaw in this paper. The proof looks correct to me. My only concern is that there seems to have little novelty in techniques as a theoretical work. Especially the improvement on the upper bound of the test risk relies on the results in Srebro et al. [15]. But I believe the proof of the lower bound has some nice intuition, which probably should be further explained.

QUESTIONS:
1. For the lower bound in Theorem 1, is there a way you can derive the minimum of the RHS with respect to the epsilon? 
2. In the proof of Theorem 2, you consider two different regimes where $T \gg n$ and $n \gg T$. How to see that in Lemma 3 and 4?

LIMITATIONS:
As a theory paper, I think it's fine to consider linear models. Therefore, I don't see an important limitation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the problem of learning linearly separable data with margin using gradient descent (GD) and focuses on establishing a population risk bound on the GD output. Previous research in this area has primarily concentrated on understanding the implicit bias and population error associated with solving this task using logistic regression. In contrast, this study introduces a broader class of loss functions known as C_{\\pi,\\beta}, which are non-negative smooth loss functions that converge to zero more rapidly than a reference loss function, \\pi.\n\nThe main contribution of this paper is the derivation of both upper and lower bounds on the population risk of GD (including stochastic gradient descent, SGD) for this class of loss functions. The authors employ a straightforward and elegant proof technique that revolves around controlling the norm of the GD output and its empirical risk. In order to establish a population error guarantee, the authors rely on uniform convergence results specifically tailored for linear models.\n\nSTRENGTHS:\nThis paper is extremely well-written and easy to follow. Also, this paper shows that bounds on the empirical error of (S)GD and the norm of the solution suffice to provide a sharp analysis of the population error. Compared to other work in this line, proofs in this work are more straightforward. Also, the population risk guarantee in this paper can be seen as an algorithmic-dependent uniform convergence which is also interesting.\n\nWEAKNESSES:\nIn classification tasks, the ultimate goal is often to minimize the 0-1 loss function, which directly measures the accuracy of classification. However, when employing gradient descent (GD) optimization, it is common to use surrogate loss functions such as logistic loss or hinge loss. These surrogate loss functions approximate the 0-1 loss and are more amenable to optimization.\n\nIn this paper, all the results focus on controlling the population risk of the surrogate loss functions. It is important to note that many surrogate loss functions can provide an upper bound on the 0-1 loss function. However, for the specific case of logistic loss, there appears to be a discrepancy between the upper bound on the logistic loss presented in this paper and the upper bound on the 0-1 loss. This discrepancy suggests that overfitting may occur, and the concept of early stopping becomes relevant.\n\nSomething that I fail to understand is why the authors do not focus on providing classification error bound for C_{pi,beta}.\n\nQUESTIONS:\n1- In Corollary 1, the authors instantiate their result for the logistic regression. It shows that when T is exponentially large in the number of samples, then we have overfitting. How does this result compare to Shamir 2021?\n\n\n2- Can the lower bound be extended to the case that it holds for every phi and “every” loss in C_{\\pi,beta}? The current statement is different.\n\nLIMITATIONS:\nOne of the limitations is that this paper does not provide any evidence of \"implicit bias\".", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization properties of gradient methods for convex, smooth and decreasing losses (e.g., logistic and polynomial losses) over linearly separable data distributions.  The contributions are 1) a generalization bound based on Rademacher complexity which does not require the self-boundedness assumption for the loss and 2) a lower bound on the test loss which nearly matches the upper-bound. Overall, the results indicate that Rademacher complexity (or algorithmic stability) analyses for data-dependent generalization of GD are rate-optimal, as the lower bounds match the upper bounds in $n,T,\\gamma$.\n\nSTRENGTHS:\nThe paper is in general well-written and clear, although few inconsistencies exist throughout the main body.\n\n The studied problem is interesting as understanding generalization of GD on separable data is foundational to machine learning. This paper takes a step in this direction by studying linear models. The paper's results imply that the commonly studied margin-based generalization bounds which usually take the form $1/(\\gamma^2 n)$ cannot be improved in $\\gamma,n$, or other parameters related to tail behavior, unless additional assumptions on data are imposed. The paper's claims are new and are also applicable to a broad class of smooth losses.\n\nThe main contribution of the paper which is the lower test loss bound for a certain class of convex loss functions involves the construction of a new data distribution which can be useful for future works.\n\nWEAKNESSES:\nweaknesses and some questions:\n\n1- The derived upper-bounds are barely an improvement over known results in literature. The authors claim in abstract and also throughout the introduction that their results apply to any smooth loss function. however this is not the case, as the loss in this work must be monotonically decreasing. This excludes the commonly used square-loss whereas previous results by Lei and Ying 2020 can be applied to square loss. I also do not find the condition of self-boundedness existing in previous works limiting as it includes almost all commonly used losses that are also used in this work. Can the authors elaborate on that? \n\n2- Also regarding the training and generalization upper bounds, I am not convinced of the novelty of the approach. The technique based on Srebro et al. is rather simple and well-known. I assume the challenge is in characterizing the rademacher complexity for linearly separable data, but similar analyses have been already done even for more complex models such as neural networks, for example see the papers titled \"Feature selection and low test error in shallow low-rotation ReLU networks\" and \"Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks\".\n\nCan the authors please comment on the challenges in optimization and generalization analyses and highlight novel steps related to general tail behaviors?\n\n3- How do the bounds look like if $w_1\\neq 0$? Particularly, it is interesting to know how the bounds on generalization behave with respect to initialization. \n\n4- It might be helpful if a new column is added to table 2 explaining what $r_{\\phi,T}$ looks like for each loss. \n\n5- The specific data distribution used for obtaining the lower bound seems very interesting. It can be helpful for readers if the authors provide more discussion on high-level steps of the proof and the their intuition on the designed distribution. \n\n6- It seems the bound of Theorem 1 contains a $\\log ^3(n)$ factor which is missing In Table 2.\n\n7- Table 2 shows that tailes of the form $\\exp(x^{-\\alpha})$ are slightly better in test loss performance than exponential tails. Does it leads to faster convergence for misclassification test error or generally for real-world experiments?\n\n8- I think the previous works on the lower generalization bounds of SVM can be discussed here. e.g.,  \"Optimality of SVM: novel proofs and tighter bounds\" by S Hanneke, A Kontorovich.\n\nQUESTIONS:\nplease see the section above.\n\nLIMITATIONS:\nThe paper does not have a conclusions section. It can be insightful if the authors discuss some limitations of the methodology and data assumptions. There is no potential negative societal impact associated with this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper establishes tight upper and lower bounds on the population risk of linear models trained with gradient descent (GD) on linearly separable data. In contrast to previous work, the paper's result only requires a smoothness assumption on the loss function, and the upper bounds are adaptive to the loss function's tail decay. Surprisingly, the authors achieve this result using standard techniques, with the generalization bound of [Srebro et al., 2010] based on local Rademacher complexity playing a pivotal role.\n\n**References**\n\n[Srebro et al., 2010]: Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. *NIPS* 2010.\n\nSTRENGTHS:\n**Minimal assumptions and standard proof technique.** The only assumptions on the loss function for the population risk upper bound are convexity, monotonicity, and smoothness, which leads to a clean theorem statement. Moreover, the proof only involves standard tools; one shows upper bounds the norm of the GD iterate and the empirical risk, and combines these two using the generalization bound from Srebro et al. (2010). In addition, the authors construct a lower bound which shows that their upper bound is nearly optimal for essentially all tail decay rates of the loss function. The minimality of assumptions, tightness of upper bounds, and simplicity of proof techniques showcases the paper’s strength.\n\nWEAKNESSES:\n**Motivation for studying losses with diverse tail decays?** Having a more general theorem is beneficial, especially if it can be achieved without much sophistication. Still, I am curious about the utility of the upper bound’s generality. Were there important examples of losses that previous works failed to capture? Are there examples of loss functions with polynomial tails that have been used in the context of classification?\n\nQUESTIONS:\n- What were some important losses (either from a theoretical or practical point of view) that previous works failed to cover?\n- When does it make sense to use loss functions with slow (e.g., polynomial) tail decay? It seems that loss functions with slow tails may encourage the predictor to increase the margin of already correct answers at the cost neglecting misclassified samples.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}