SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matan Schliserman $^{1*}$  Tomer Koren $^{12}$
<sup>1</sup>Blavatnik School of Computer Science, Tel Aviv University
2Google Research, Tel Aviv
We study the generalization properties of unregularized gradient methods applied to separable linear classification—a setting that has received considerable attention since the pioneering work of Soudry et al. [14]. We establish tight upper and lower (population) risk bounds for gradient descent in this setting, for any smooth loss function, expressed in terms of its tail decay rate. Our bounds take the form  $\Theta(r_{\ell,T}^2/\gamma^2T + r_{\ell,T}^2/\gamma^2n)$ , where  $T$  is the number of gradient steps,  $n$  is the size of the training set,  $\gamma$  is the data margin, and  $r_{\ell,T}$  is a complexity term that depends on the tail decay rate of the loss function (and on  $T$ ). Our upper bound greatly improves the existing risk bounds due to Shamir [13] and Schliserman and Koren [12], that either applied to specific loss functions or imposed extraneous technical assumptions, and applies to virtually any convex and smooth loss function. Our risk lower bound is the first in this context and establishes the tightness of our general upper bound for any given tail decay rate and in all parameter regimes. The proof technique used to show these results is also markedly simpler compared to previous work, and is straightforward to extend to other gradient methods; we illustrate this by providing analogous results for Stochastic Gradient Descent.
Recently, there has been a marked increase in interest regarding the generalization capabilities of unregularized gradient-based learning methods. One specific area of attention in this context has been the setting of linear classification with separable data, where a pioneering work by Soudry et al. [14] showed that, when using plain gradient descent to minimize the empirical risk on a linearly separable training set with an exponentially-tailed classification loss (such as the logistic loss), the trained predictor will asymptotically converge in direction to the max-margin solution. As a result, standard margin-based generalization bounds for linear predictors suggest that, provided the number of gradient steps  $(T)$  is sufficiently large, the produced solution will not overfit, despite the lack of explicit regularization and the fact that its magnitude (i.e., Euclidean norm) increases indefinitely with  $T$ . This result has since been extended to incorporate other optimization algorithms and loss functions [3, 4, 8, 9, 5].
Despite the high interest in this problem, the tight finite-time (population) risk performance of unregularized gradient methods, and even just of gradient descent, have not yet been fully understood. The convergence to a high margin solution exhibited by Soudry et al. [14] (for the logistic loss) occurs at a slow logarithmic rate, thus, the risk bounds for the trained predictors only become effective when  $T$  is at least exponentially large in comparison to the size of the training set  $n$  and the margin  $\gamma$ . In a more recent work, Shamir [13] established risk bounds of the form  $\widetilde{O}(1 / \gamma^2 T + 1 / \gamma^2 n)$  for several gradient methods in this setting (still with the logistic loss), that apply for smaller and more realistic values of  $T$ .
*Corresponding author; schliserman@mail.tau.ac.il.
Table 1: Comparison between the bounds established in this work and bounds established in previous work for gradient descent on  $\gamma$ -separable data; here,  $T$  is the number of gradient steps and  $n$  is the size of the training set. Logarithmic factors other than  $\log T$  factors are suppressed from the bounds. Examples of how our general bounds are instantiated for some concrete loss functions are presented in Table 2 below.
Paper	Class of losses	Type of bound	Upper bound	Lower bound
Shamir [13]	Logistic loss	0-1 loss (population)	log2T/γ2T + 1/γ2n	1/γ2T(a)
Schliserman and Koren [12]	General convex; smooth; Lipschitz; (c,δ)-self-bounded;(b) Decay rate of φ	Risk (population)	r2φ,T/γ2T + Tδr2φ,T/γ2n	-
General convex; smooth; (c,δ)-self-bounded;(b) Decay rate of φ	Risk (population)	r2φ,T/γ2T + T2δr4φ,T/γ4n1+2δ	-
Telgarsky [16]	Convex linear models; quadratically-bounded;(c) Decay rate of φ	Risk (population)	rφ,T/γ√T(d)	-
This paper	Convex linear models; smooth; Decay rate of φ	Risk (population)	r2φ,T/γ2T + r2φ,T/γ2n	r2φ,T/γ2T + r2φ,T/γ2n
a The lower bound is shown for the empirical (logistic) risk.
${}^{\mathrm{b}}f$  is  $(c,\delta)$ -self-bounded if for every  $x$ , it holds that  $\| \nabla f(x)\| \leq cf(x)^{1 - \delta}$ ; for  $\delta < 1 / 2$ , this property is stronger than smoothness of  $f$ .
c  $\ell$  is  $(c_{1},c_{2})$ -quadratically-bounded if, for every  $x$  and  $z$ ,  $|\ell '(x,z)|\leq c_1 + c_2(\| x\| +\| z\|)$
d The upper bound is valid when  $T\leq n$
Later Schliserman and Koren [12] gave a more general analysis that extends the bounds of Shamir [13] to a wide range of smooth loss functions satisfying a certain "self-boundedness" condition, and Telgarsky [16] provided a test-loss analysis of stochastic mirror descent on "quadratically-bounded" losses. However, all of these works either assume a specific loss (e.g., the logistic loss), impose various conditions on the loss function (beyond smoothness), or do not establish the tightness of their bounds in all regimes of parameters  $(T,n$  and  $\gamma)$ , and especially in the regime where the number of gradient steps  $T$  is larger than the sample size  $n$  (see Table 1 for details).
In this work, we close these gaps by showing nearly matching upper and lower risk bounds for gradient descent, which hold essentially for any smooth loss function, and for any number of steps  $T$  and sample size  $n$ . Compared to recent prior work in this context [13, 16, 12], our results do not require any additional assumptions on the loss function besides its smoothness, and they strictly improve upon the existing bounds by their dependence on  $T$ . Further, to the best of our knowledge, risk lower bounds were not previously explored in this context, and the lower bounds we give establish, for the first time, the precise risk convergence rate of gradient descent for any smooth loss function and in all regimes of  $T$  and  $n$ .
In some more detail, our results assume the following form. Let  $C_{\phi,\beta}$  be the class of nonnegative, convex and  $\beta$ -smooth loss functions  $\ell(u)$  that decay to zero (as  $u \to \infty$ ) faster than a reference "tail function"  $\phi : [0,\infty) \to \mathbb{R}$ . The function  $\phi$  is merely used to quantify how rapidly the tails of loss functions in  $C_{\phi,\beta}$  decay to zero. Then, the risk upper and lower bounds we prove for gradient descent are of the following form:2
$$
\frac {\beta r _ {\phi , T} ^ {2}}{\gamma^ {2} T} + \frac {\beta r _ {\phi , T} ^ {2}}{\gamma^ {2} n}. \tag {1}
$$
2For simplicity, we specialize the bounds here to gradient descent with stepsize  $\eta = \Theta (1 / \beta)$  , but the bounds hold more generally to any stepsize smaller than  $O(1 / \beta)$
Table 2: Examples of tight population risk bounds established in this paper for gradient descent on  $\gamma$ -separable data, instantiated for several different loss tail decay rates. Details on the derivation of the bounds from the general Theorems 1 and 2 can be found in Appendix A.
Tail decay (φ)	rφ,T	Risk bound
exp(-x)	log(T)	Θ(log2(T)/γ2T + log2(T)/γ2n)
x-α	(γ2T)1/α+2	Θ((1/γ)2/2+α(1/T2/α + T2/2+α/n))
exp(-xα)	log1/α(T)	Θ(log2/α(T)/γ2T + log2/α(T)/γ2n)
The bounds depend on the tail function  $\phi$  through the term  $r_{\phi, T}$  that, roughly, equals  $\phi^{-1}(\varepsilon)$  for  $\varepsilon$  chosen such that  $\varepsilon \approx (\phi^{-1}(\varepsilon))^2 / \gamma^2 T$  (for the precise statements refer to Theorems 1 and 2, and for concrete examples of implied bounds, see Table 2).
The form of Eq. (1) resembles the bounds given in recent work by Schliserman and Koren [12]. However, the latter work imposed an extraneous "self-boundedness" assumption (stronger than smoothness, see Table 1) that we do not require, and they did not establish the tightness of their bounds, as we do in this paper by providing matching lower bounds. On the flip side, their upper bounds apply in a broader stochastic convex optimization setup, whereas our bounds are specialized to generalized linear models for classification.
In terms of techniques, our proof methodology is also distinctly (and perhaps somewhat surprisingly) simple compared to previous work. Our upper bounds rely on two elementary properties that gradient methods admit when applied to a smooth and realizable objective: low training loss and low norm of the produced solutions. Both properties are obtained from fairly standard arguments and convergence bounds for smooth gradient descent, when paired with conditions implied by the decay rate of the loss function. Finally, to bound the gap between the population risk and the empirical risk, we employ a classical result of Srebro et al. [15] that bounds the generalization gap of linear models in the so-called "low-noise" (i.e., nearly realizable) smooth regime using local Rademacher complexities. Somewhat surprisingly, this simple combination of tools already give sharp risk results that improve upon the state-of-the-art [13, 12, 16] both in terms of tightness of the bounds and the light set of assumptions they rely on.
We remark here that the proof scheme summarized above can be generalized to essentially any gradient method for which one can prove simultaneously bounds on the optimization error and the norm of possible solutions produced by the algorithm. In the sequel, we focus for concreteness on standard gradient descent, but in Appendix B we also give bounds for Stochastic Gradient Descent (SGD), which is shown to admit both of these properties with high probability over the input sample.
An interesting conclusion from our bounds pertains to the significance of early stopping. We see that gradient descent, even when applied on a smooth loss functions that decay rapidly to zero, might overfit with respect to the surrogate loss (i.e., reach a trivial  $\Theta(1)$  risk) as the number of steps  $T$  grows. The time by which gradient descent starts overfitting depends on the tail decay rate of the loss function: the slower the decay rate, the shorter the time it takes to overfit. Interestingly, a similar phenomenon may not occur for the zero-one loss of the trained predictor. For example, Soudry et al. [14] show that with the logistic loss, gradient descent does not overfit in terms of the zero-one as  $T$  approached infinity—whereas, at the same time, it does overfit as  $T \to \infty$  in terms of the logistic loss itself, as seen from our lower bounds.
Summary of contributions. To summarize, the main contribution of this paper are as follows:
- Our first main result (in Section 3) is a high probability risk upper bound for gradient descent in the setting of separable classification with a convex and smooth loss function. Our bound matches the best known upper bounds [13, 12] and greatly extend them to allow for virtually any convex and smooth loss function, considerably relaxing various assumptions imposed by prior work.
- Our second main result (in Section 4) is a nearly matching lower bound for the risk of gradient descent, establishing the tightness of our analysis given the smoothness and tail-decay conditions. The tightness of our bounds holds across tail decay rates and different regimes of the parameters  $T, n$  and  $\gamma$ . To our knowledge, these constitute the first (let alone tight) risk lower bounds in this context, despite a long line of work on linear classification with separable data.
- We also provide analogous results for Stochastic Gradient Descent with replacement (in Appendix B), mainly to emphasize that our analysis uses only two elementary properties of the optimization algorithm: low optimization error and low norm of the produced solution. The same analysis can be generalized to any gradient method that admits these two properties.
1.2 Additional related work
Unregularized gradient methods on separable data. The most relevant work to ours is of Schliserman and Koren [12], who used algorithmic stability and two simple conditions of self-boundedness and realizability which the loss functions hold, to get generalization bounds for gradient methods with constant step size in the general setting of stochastic convex and smooth optimization. Then, they derived risk bounds which hold in expectation for the setting of linear classification with separable data for every loss function which decays to 0. The exact bound was dependent in the rate of decaying to 0 of the function. For the Lipschitz case their risk bound with respect to the loss function  $\ell$  is  $O(\ell^{-1}(\varepsilon)^{2} / \gamma^{2}T + \ell^{-1}(\varepsilon)^{2} / \gamma^{2}n)$  for any choice of  $\varepsilon$  such that  $\varepsilon /\ell^{-1}(\varepsilon)^{2}\leq 1 / \gamma^{2}T$ . For example, for the logistic loss, the bound translates to  $O(\log^2 (T) / \gamma^2 T + \log^2 (T) / \gamma^2 n)$ .
In another work, Telgarsky [16], showed a high probability risk bound for  $T \leq n$  for Batch Mirror Decent with step size  $\eta \simeq 1 / \sqrt{T}$  in linear models, using a reference vector, which when selected properly, can be translated to a risk bound of  $O(\ell^{-1}(\varepsilon) / \gamma \sqrt{T})$  for gradient descent applied on the loss function  $\ell$ , and to a  $O(\log T / \gamma \sqrt{T})$  for the logistic loss.
Fast rates for smooth and realizable optimization. The problem of smooth and realizable optimization, also known as the "low-noise" regime of stochastic optimization, is a very well researched problem. Srebro et al. [15] showed that stochastic gradient descent achieved risk bound of  $O(1/n)$  in this setting. For linear models, they also showed that ERM achieve similar fast rates by using local Rademacher complexities. Later [10] showed that SGD converges linearly when the loss function is also strongly convex. In more recent works, [7] used stability arguments to show that SGD with replacement with  $T = n$  achieve risk of  $O(1/n)$ .
Lower bounds. A lower bound related to ours appears in Ji and Telgarsky [4]. In this work, the authors showed a lower bound of  $\| w_t' - w^* \| \geq \log(n) / \log(T)$ . In our work, however, we get lower bound directly for the loss itself and not for this objective. More recently, Shamir [13] showed a lower bound of  $\Omega(1/\gamma^2 T)$  for the empirical risk of GD when applied on the logistic loss which is tight up to log factors. When generalizing this technique for other objectives, e.g., functions that decay polynomially to zero, the log factors become polynomial factors and the bound becomes not tight, even in the regime  $T \ll n$  where the  $1/T$  term in the bounds is dominant. In contrast, we establish nearly tight bounds for virtually any tail decay rate, and in all regimes of  $T$  and  $n$ .
We consider the following typical linear classification setting. Let  $\mathcal{D}$  be distribution over pairs  $(x,y)$ , where  $x\in \mathbb{R}^d$  is a  $d$ -dimensional feature vector and  $y\in \mathbb{R}$  is a real number that represents the corresponding label. We assume that data is scaled so that  $\| x\| \leq 1$  and  $|y|\leq 1$  (with probability one with respect to  $\mathcal{D}$ ). We focus on the setting of separable, or realizable, linear classification with margin. Formally, we make the following assumption.
Assumption 1 (realizability). There exists a unit vector  $w^{*} \in \mathbb{R}^{d}$  and  $\gamma > 0$  such that  $y(w^{*} \cdot x) \geq \gamma$  almost surely with respect to the distribution  $\mathcal{D}$ .
Equivalently, we will identify each pair  $(x,y)$  with the vector  $z = yx$ , and realizability implies that  $w^{*} \cdot z \geq \gamma$  with probability 1. Our assumptions then imply that  $\|z\| \leq 1$  with probability 1.
Given a nonnegative loss function  $\ell : \mathbb{R} \to \mathbb{R}^+$ , the objective is to determine a model  $w \in \mathbb{R}^d$  that minimizes the (population) risk, defined as the expected value of the loss function over the distribution
$\mathcal{D}$  ,namely
$$
L (w) = \mathbb {E} _ {z \sim \mathcal {D}} [ \ell (w \cdot z) ]. \tag {2}
$$
For finding such a model, we use a set of training examples  $S = \{z_1,\dots,z_n\}$  which drawn i.i.d. from  $\mathcal{D}$  and an empirical proxy, the empirical risk, which is defined as
$$
\widehat {L} (w) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(w \cdot z _ {i}\right). \tag {3}
$$
2.1 Loss functions
The loss functions  $\ell$  considered in this paper are nonnegative, convex and  $\beta$ -smooth. We also require that  $\ell$  is strictly monotonically decreasing and  $\lim_{u\to \infty}\ell (u) = 0$ . The vast majority of loss functions used in supervised learning for classification satisfy these conditions; these include, for example, the logistic loss  $(\ell (u) = \log (1 + e^{-u}))$ , the probit loss  $(\ell (u) = -\log (\frac{1}{2} -\frac{1}{2}\operatorname {erf}(u)))$ , and the squared hinge loss  $(\ell (u) = (\max \{1 - u,0\})^2)$ .
A main goal of this paper is to quantify how do the achievable bounds on the risk depend on properties of the loss function  $\ell$  used, and most crucially on the rate in which  $\ell$  decays to zero as its argument approaches infinity. To formalize this, we need a couple of definitions.
Definition 1 (tail function). We say that  $\phi :[0,\infty)\to \mathbb{R}$  is a tail function if  $\phi$
(i) is a nonnegative, 1-Lipschitz and  $\beta$ -smooth convex function;
(ii) is strictly monotonically decreasing such that  $\lim_{u\to \infty}\phi (u) = 0$
(iii) satisfies  $\phi (0)\geq \frac{1}{2}$  and  $|\phi^{\prime}(0)|\geq \frac{1}{2}$
Every tail function  $\phi$  defines a class of loss functions characterized by the rate  $\phi$  decays to zero.
Definition 2 ( $\phi$ -tailed class). For a tail function  $\phi$ , the class  $C_{\phi, \beta}$  is the set of all nonnegative, convex,  $\beta$ -smooth and monotonically decreasing loss functions  $\ell: \mathbb{R} \to \mathbb{R}^+$  such that  $\ell(u) \leq \phi(u)$  for all  $u \geq 0$ .
We detail several examples for tail functions in Appendix A.
2.2 Gradient Descent
The algorithm that we focus in this paper is standard gradient descent (GD) with a fixed step size  $\eta > 0$  applied to the empirical risk  $\widehat{L}$ ; this method is initialized at  $w_{1} = 0$  and at each step  $t = 1, \dots, T$  performs an update
$$
w _ {t + 1} = w _ {t} - \eta \nabla \widehat {L} (w _ {t}). \tag {4}
$$
The algorithm returns the final model,  $w_{T}$ .
We remark however that most of the results we present in the sequel can be straightforwardly adapted to other gradient methods; we include results for Stochastic Gradient Descent (SGD) in Appendix B, and the same proof techniques can be used to analyze multi-epoch and/or mini-batched SGD, gradient flow, and more.
3 Risk Upper Bounds
We begin by giving a general upper bound for the risk of gradient descent, when the loss function  $\ell$  is taken from the class  $C_{\phi,\beta}$ . Our main result in this section is the following.
Theorem 1. Let  $\phi$  be a tail function and let  $\ell$  be any loss function from the class  $C_{\phi, \beta}$ . Fix  $T, n$  and  $\delta > 0$ . Then, with probability at least  $1 - \delta$  (over the random sample  $S$  of size  $n$ ), the output of  $GD$  applied on  $\widehat{L}$  with step size  $\eta \leq 1/2\beta$  initialized at  $w_1 = 0$  has
$$
L (w _ {T}) \leq \frac {4 K (\phi^ {- 1} (\varepsilon)) ^ {2}}{\gamma^ {2} \eta T} + \frac {3 2 K \beta (\phi^ {- 1} (\varepsilon)) ^ {2} \left(\log^ {3} n + 4 \log \frac {1}{\delta}\right)}{\gamma^ {2} n} + \frac {4 K (\phi^ {- 1} (\varepsilon)) ^ {2} \log \frac {1}{\delta}}{\gamma^ {2} \eta T n}
$$
for any  $\varepsilon \leq \frac{1}{2}$  such that  $\eta \gamma^2 T \leq (\phi^{-1}(\varepsilon))^2 / \varepsilon$ , where  $K < 10^5$  is a numeric constant.
3A differentiable function  $\ell : \mathbb{R} \to \mathbb{R}$  is said to be  $\beta$ -smooth over  $\mathbb{R}$  if  $\ell(v) \leq \ell(u) + \ell'(u) \cdot (v - u) + \frac{1}{2} \beta(v - u)^2$  for all  $u, v \in \mathbb{R}$ .
The expression  $(\phi^{-1}(\varepsilon))^2 /\varepsilon$  increases indefinitely as  $\varepsilon$  approaches 0; therefore, for any  $T$ , there exists an  $\varepsilon >0$  that satisfies the theorem's condition. For several examples of how this bound is instantiated for different tail decay functions  $\phi$ , see Table 2 and Appendix A.
In the remainder of this section we prove Theorem 1. The structure of the proof will be as follows: First, we bound the norm of the GD solution; by smoothness and realizability, we get that the norm will remain small compared to a reference point with small loss value. Second, we get a bound on the optimization error in this setting, the relies on the same reference point. Finally, we use a fundamental result due to Srebro et al. [15] (reviewed in the subsection below) together with both bounds to derive the risk guarantee. As discussed broadly in the introduction, this proof scheme can be generalized to other gradient methods which satisfy the properties of model with low norm and low optimization error.
3.1 Preliminaries: Uniform Convergence Using Rademacher Complexity
One property of linear models is that in this class of problems is that we have dimension-independent and algorithm-independent uniform convergence bounds, that enables to bound the difference between the empirical risk and the population risk of a specific model. A main technical tool for bounding this difference is the Rademacher Complexity [2]. The worst-case Rademacher complexity of an hypothesis class  $H$  for any sample size  $n$  is given by:
$$
R _ {n} (H) = \sup  _ {z _ {1}, \dots z _ {n}} \mathbb {E} _ {\sigma \sim \operatorname {U n i f} (\{\pm 1 \} ^ {n})} \left[ \sup  _ {h \in H} \frac {1}{n} \left| \sum_ {i = 1} ^ {n} h (z _ {i}) \sigma_ {i} \right| \right].
$$
We are interested in models that achieve low empirical risk on smooth objectives. A fundamental result of Srebro et al. [15] bounds the generalization gap under such conditions:
Proposition 1 (15, Theorem 1). Let  $H$  be a hypothesis class with respect to some non-negative and  $\beta$ -smooth function,  $\ell(t \cdot y)$ , such that for every  $w \in H, x, y, |\ell(wx \cdot y)| \leq b$ . Then, for any  $\delta > 0$  we have, with probability at least  $1 - \delta$  over a sample of size  $n$ , uniformly for all  $h \in H$ ,
$$
L (h) \leq \widehat {L} (h) + K \left(\sqrt {\widehat {L} (h)} \left(\sqrt {\beta} \log^ {1. 5} (n) R _ {n} (H) + \sqrt {\frac {b \log \frac {1}{\delta}}{n}}\right) + \beta \log^ {3} (n) R _ {n} ^ {2} (H) + \frac {b \log \frac {1}{\delta}}{n}\right).
$$
where  $K < 10^{5}$  is a numeric constant.
3.2 Properties of Gradient Descent on Smooth Objectives
In this section we prove that GD satisfies the two desired properties- low norm and low optimization error. We begin with showing that the norm of  $w_{T}$ , the output of GD after  $T$  iterations, is low, as stated in the following lemma,
Lemma 1. Let  $\phi$  be a tail function and let  $\ell \in C_{\phi, \beta}$ . Fix any  $\varepsilon > 0$  and a point  $w_{\varepsilon}^{*} \in \mathbb{R}^{d}$  such that  $\widehat{L}(w_{\varepsilon}^{*}) \leq \varepsilon$  (exists due to realizability). Then, the output of  $T$ -steps  $GD$ , applied on  $\widehat{L}$  with stepsize  $\eta \leq 1 / \beta$  initialized at  $w_{1} = 0$  has,
$$
\left\| w _ {T} \right\| \leq 2 \left\| w _ {\varepsilon} ^ {*} \right\| + 2 \sqrt {\eta \varepsilon T}.
$$
Now, we bound the optimization error of GD on every function  $\ell \in C_{\phi,\beta}$ , by using a variant of Lemma 13 from [12]. The proof is fairly standard and appears in Appendix C.