SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 2. Let  $\phi$  be a tail function and let  $\ell \in C_{\phi, \beta}$ . Fix any  $\varepsilon > 0$  and a point  $w_{\varepsilon}^{*} \in \mathbb{R}^{d}$  such that  $\widehat{L}(w_{\varepsilon}^{*}) \leq \varepsilon$ . Then, the output of  $T$ -steps  $GD$ , applied on  $\widehat{L}$  with stepsize  $\eta \leq 1 / \beta$  initialized at  $w_{1} = 0$  has,
$$
\widehat {L} (w _ {T}) \leq \frac {\| w _ {\varepsilon} ^ {*} \| ^ {2}}{\eta T} + 2 \varepsilon .
$$
We now turn to prove Theorem 1. The proof is a simple consequence of the properties proved above and Proposition 1. We first claim that there exists a model  $w_{\varepsilon}^{*}$  with low norm such that  $\widehat{L}(w_{\varepsilon}^{*}) \leq \varepsilon$ ,
which implies, through Lemmas 1 and 2, that  $w_{T}$  of gradient descent has both low optimization error and it remains bounded within a ball of small radius. Then, we use Proposition 1 to translate the low optimization error to low risk.
Proof of Theorem 1. First, we show that there exists a model  $w_{\varepsilon}^{*}$  with low norm such that  $\widehat{L}(w_{\varepsilon}^{*}) \leq \varepsilon$ . Let  $\varepsilon \leq \frac{1}{2}$  and  $\ell \in C_{\phi, \beta}$ . By separability, there exists a unit vector  $w^{*}$  such that  $w^{*} \cdot z_{i} \geq \gamma$  for every  $z_{i}$  in the training set  $S$ . Moreover,  $\ell$  is monotonic decreasing. Then, for  $w_{\varepsilon}^{*} = (\phi^{-1}(\varepsilon) / \gamma) w^{*}$  and every  $z_{i} \in S$ ,
$$
\ell \left(w _ {\varepsilon} ^ {*} \cdot z _ {i}\right) = \ell \left(\frac {\phi^ {- 1} (\varepsilon)}{\gamma} w ^ {*} \cdot z _ {i}\right) \leq \ell \left(\frac {\phi^ {- 1} (\varepsilon)}{\gamma} \cdot \gamma\right) \leq \ell \left(\phi^ {- 1} (\varepsilon)\right).
$$
Then, by the fact that  $\phi^{-1}(\varepsilon) \geq 0$ , we have  $\ell(w_{\varepsilon}^{*} \cdot z_{i}) \leq \ell(\phi^{-1}(\varepsilon)) \leq \phi(\phi^{-1}(\varepsilon)) = \varepsilon$  for all  $i$ , hence
$$
\widehat {L} \left(w _ {\varepsilon} ^ {*}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(w _ {\varepsilon} ^ {*} \cdot z _ {i}\right) \leq \varepsilon .
$$
Now, for  $\varepsilon$  such that  $\eta \gamma^2 T\leq (\phi^{-1}(\varepsilon))^2 /\varepsilon$ , we get by Lemma 1,
$$
\left\| w _ {T} \right\| \leq 2 \left\| w _ {\varepsilon} ^ {*} \right\| + 2 \sqrt {\eta \varepsilon T} \leq \frac {4 \phi^ {- 1} (\varepsilon)}{\gamma}.
$$
For the same  $\varepsilon$ , by Lemma 2,
$$
\widehat {L} (w _ {T}) \leq \frac {\| w _ {\varepsilon} ^ {*} \| ^ {2}}{\eta T} + 2 \varepsilon \leq 3 \frac {\phi^ {- 1} (\varepsilon) ^ {2}}{\gamma^ {2} \eta T}.
$$
Denote  $B_{\varepsilon} = \{w: \| w \| \leq r_{\varepsilon}\}$ , where for brevity  $r_{\varepsilon} = 4\phi^{-1}(\varepsilon) / \gamma$ . We have, by Lemma 10,  $f(x) \leq 2f(y) + \beta \| x - y \|^2$  for all  $x, y \in \mathbb{R}^d$  (see proof in Appendix C). Then, together with the fact that  $\| z \|, \| z' \| \leq 1$  and choosing  $\varepsilon$  such that  $\varepsilon \leq \phi^{-1}(\varepsilon)^2 / \gamma^2 \eta T$ , with probability 1,
$$
b = \max  _ {w \in B _ {\varepsilon}} | \ell (w \cdot z) | \leq 2 \ell \left(w _ {\varepsilon} ^ {*} z\right) + 4 \beta r _ {\varepsilon} ^ {2} \leq 2 \varepsilon + 4 \beta r _ {\varepsilon} ^ {2} \leq \frac {r _ {\varepsilon} ^ {2}}{8 \eta T} + 4 \beta r _ {\varepsilon} ^ {2}.
$$
Moreover,  $B_{\varepsilon}$  is a hypothesis class of linear predictors with norm at most  $r_{\varepsilon}$ . We know that the norm of the examples is at most 1, thus, it follows that the Rademacher complexity of  $B_{\varepsilon}$  is  $R_{n}(B_{\varepsilon}) = r_{\varepsilon} / \sqrt{n}$  [e.g., 6, Theorem 3].
Now, by the choice of  $\varepsilon$ , we have  $\widehat{L}(w_T) \leq 3r_{\varepsilon}^2 / 16\eta T$ . Thus, Proposition 1 implies that with probability at least  $1 - \delta$ , for every  $w \in B_{\varepsilon}$  and any  $\varepsilon$  such that  $\varepsilon \leq \phi^{-1}(\varepsilon)^2 / \gamma^2\eta T$ ,
$$
L \left(w _ {T}\right) \leq \frac {3 r _ {\varepsilon} ^ {2}}{1 6 \eta T} + K \left(\sqrt {\frac {3 r _ {\varepsilon} ^ {2}}{1 6 \eta T}} \left(\sqrt {\beta} \log^ {1. 5} n \frac {r _ {\varepsilon}}{\sqrt {n}} + \sqrt {\frac {b \log \frac {1}{\delta}}{n}}\right) + \beta \log^ {3} n \frac {r _ {\varepsilon} ^ {2}}{n} + \frac {b \log \frac {1}{\delta}}{n}\right).
$$
Plugging in the bound on  $b$ , dividing by  $r_{\varepsilon}^{2}$  and using twice the fact that  $xy \leq \frac{1}{2} x^{2} + \frac{1}{2} y^{2}$  for all  $x, y$ ,
$$
\begin{array}{l} \frac {L \left(w _ {T}\right)}{r _ {\varepsilon} ^ {2}} \leq \frac {3}{1 6 \eta T} + K \left(\sqrt {\frac {3}{1 6 \eta T}} \left(\frac {\sqrt {\beta} \log^ {1 . 5} n}{\sqrt {n}} + \sqrt {\frac {\left(\frac {1}{8 \eta T} + 4 \beta\right) \log \frac {1}{\delta}}{n}}\right) + \frac {\beta \log^ {3} n}{n} + \frac {\left(\frac {1}{8 \eta T} + 4 \beta\right) \log \frac {1}{\delta}}{n}\right) \\ \leq \frac {3}{1 6 \eta T} + K \left(\sqrt {\frac {3}{1 6 \eta T}} \left(\frac {\sqrt {\beta} \log^ {1 . 5} n}{\sqrt {n}} + \sqrt {\frac {\log \frac {1}{\delta}}{8 \eta T n} + \frac {4 \beta \log \frac {1}{\delta}}{n}}\right) + \frac {\beta \log^ {3} n}{n} + \frac {\log \frac {1}{\delta}}{8 \eta T n} + \frac {4 \beta \log \frac {1}{\delta}}{n}\right) \\ \leq \frac {3}{1 6 \eta T} + K \left(\frac {3}{3 2 \eta T} + \frac {1}{2} \left(\frac {\sqrt {\beta} \log^ {1 . 5} n}{\sqrt {n}} + \sqrt {\frac {\log \frac {1}{\delta}}{8 \eta T n} + \frac {4 \beta \log \frac {1}{\delta}}{n}}\right) ^ {2} + \frac {\beta \log^ {3} n}{n} + \frac {\log \frac {1}{\delta}}{8 \eta T n} + \frac {4 \beta \log \frac {1}{\delta}}{n}\right) \\ \leq \frac {3}{1 6 \eta T} + K \left(\frac {3}{3 2 \eta T} + \frac {\beta \log^ {3} n}{n} + \frac {\log \frac {1}{\delta}}{8 \eta T n} + \frac {4 \beta \log \frac {1}{\delta}}{n} + \frac {\beta \log^ {3} n}{n} + \frac {\log \frac {1}{\delta}}{8 \eta T n} + \frac {4 \beta \log \frac {1}{\delta}}{n}\right) \\ \leq \frac {7 K}{3 2 \eta T} + \frac {2 K \beta \left(\log^ {3} n + 4 \log \frac {1}{\delta}\right)}{n} + \frac {K \log \frac {1}{\delta}}{4 \eta T n} \\ \end{array}
$$
The theorem follows by rearranging the inequality.
4 Risk Lower Bounds
In this section we present our second main result: a lower bound showing that the bound we proved in Section 3 for gradient descent is essentially tight for loss functions in the class  $C_{\phi, \beta}$ , for any given tail function  $\phi$  and any  $\beta > 0$ . Formally, we prove the following theorem.
Theorem 2. There exists a constant  $C$  such that the following holds. For any tail function  $\phi$ , sample size  $n \geq 35$  and any  $T$ , there exist a distribution  $\mathcal{D}$  and a loss function  $\ell \in C_{\phi, \beta}$ , such that for  $T$ -steps  $GD$  over a sample  $S = \{z_i\}_{i=1}^n$  sampled i.i.d. from  $\mathcal{D}$ , initialized at  $w_1 = 0$  with stepsize  $\eta \leq 1/2\beta$ , it holds that
$$
\mathbb {E} \left[ L (w _ {T}) \right] \geq C \frac {\beta (\phi^ {- 1} (1 2 8 \varepsilon)) ^ {2}}{\gamma^ {2} n} + C \frac {(\phi^ {- 1} (8 \varepsilon)) ^ {2}}{\gamma^ {2} \eta T}.
$$
for any  $\varepsilon <  \frac{1}{256}$  such that  $\eta \gamma^2 T\geq (\phi^{-1}(\varepsilon))^2\big / \varepsilon$
We remark that the right-hand side of the bound is well defined, as we restrict  $\varepsilon$  to be sufficiently small so as to ensure that all arguments to  $\phi^{-1}$  are at most  $\frac{1}{2}$  (recall that  $\phi$  admits all values in  $[0, \frac{1}{2}]$  due to our assumptions that  $\phi(0) \geq \frac{1}{2}$ ). Further, the lower bound above matches the upper bound given in Theorem 1 up to constants, unless the tail function  $\phi$  decays extremely slowly, and slower than any polynomial (at this point, however, the entire bound becomes almost vacuous).
To prove Theorem 2, we consider two different regimes: the first is where  $T \gg n$ , when the first term in the right-hand side of the bound is dominant; and the  $T \ll n$  regime where the second term is dominant. We begin by focusing on the first regime, and prove the following.
Lemma 3. There exists a constant  $C_1$  such that the following holds. For any tail function  $\phi$ , sample size  $n \geq 35$  and any  $\gamma$  and  $T$ , there exist a distribution  $\mathcal{D}$  with margin  $\gamma$ , a loss function  $\ell \in C_{\phi, \beta}$  such that for  $GD$  over a sample  $S = \{z_i\}_{i=1}^n$  sampled i.i.d. from  $\mathcal{D}$ , initialized at  $w_1 = 0$  with stepsize  $\eta \leq 1/2\beta$ , it holds that
$$
\mathbb {E} \left[ L (w _ {T}) \right] \geq C _ {1} \frac {\beta \phi^ {- 1} (1 2 8 \varepsilon) ^ {2}}{\gamma^ {2} n},
$$
To prove the lemma, we construct a learning problem for which the risk of GD can be lower bounded. We design a loss function  $\ell$  which exhibits a decay to zero at a rate similar to that of the function  $\phi$  for  $x > 0$ , and behaves as a quadratic function for  $x \leq 0$ . The distribution  $\mathcal{D}$  is constructed such that there are two fixed examples in its support, denoted  $z_{1}$  and  $z_{2}$ , that are both sampled with constant probability and are almost opposite in direction to each other:  $z_{2}$  is aligned with  $-z_{1}$  except for a small component, denoted  $\nu$ , which is orthogonal to  $z_{1}$ . Crucially, since  $z_{1}$  and  $z_{2}$  point in nearly opposite directions, minimizing optimization error requires the GD iterate to have a substantial component aligned with the vector  $\nu$  (otherwise, the loss for  $z_{2}$  will be large). Finally, we define another example, denoted as  $z_{3}$ , that has a significant component in the opposite direction  $-\nu$ . We arrange the distribution  $\mathcal{D}$  so that  $z_{3}$  is sampled with with probability roughly  $1/n$ . The lower bound now follows by observing that, if  $z_{3}$  is sampled at test time, but did not appear in the training set (this happens with probability  $\approx 1/n$ ), the test error is quadratically large in the magnitude of the GD iterate along  $-\nu$ .
We now turn to formally proving Lemma 3.
Proof. Given  $\gamma \leq \frac{1}{8}$ , let us define the following distribution  $\mathcal{D}$ :
$$
\mathcal {D} = \left\{ \begin{array}{l l} z _ {1} := (1, 0, 0) & \text {w i t h p r o b .} \frac {5 9}{6 4} (1 - \frac {1}{n}); \\ z _ {2} := (- \frac {1}{2}, 3 \gamma , 0) & \text {w i t h p r o b .} \frac {5}{6 4} (1 - \frac {1}{n}); \\ z _ {3} := (0, - \frac {1}{8}, 4 \gamma + \frac {1}{4}) & \text {w i t h p r o b .} \frac {1}{n}, \end{array} \right.
$$
and loss function:
$$
\ell (x) = \left\{ \begin{array}{l l} \phi (x) & x \geq 0; \\ \phi (0) + \phi^ {\prime} (0) x + \frac {\beta}{2} x ^ {2} & x <   0. \end{array} \right.
$$
First, note that the distribution is separable: for  $w^{*} = (\gamma, \frac{1}{2}, \frac{1}{4})$  it holds that  $w^{*}z_{i} = \gamma$  for every  $i \in \{1, 2, 3\}$ . Moreover, Lemma 11 in Appendix D ensures that indeed  $\ell \in C_{\phi, \beta}$ .
Next, let  $S$  be a sample of  $n$  i.i.d. examples from  $\mathcal{D}$  and let  $z' \sim \mathcal{D}$  be a validation example independent from  $S$ . Denote by  $\delta_2 \in [0,1]$  the fraction of appearances of  $z_2$  in the sample  $S$ , and by  $A_1, A_2$  the following events;
$$
A _ {1} = \left\{z ^ {\prime} = z _ {3} \wedge z _ {3} \notin S \right\}, \quad A _ {2} = \left\{\delta_ {2} \in \left[ \frac {1}{3 2}, \frac {1}{8} \right] \right\}.
$$
In Lemma 12 (found in Appendix D), we show that
$$
\Pr \left(A _ {1} \cap A _ {2}\right) \geq \frac {1}{1 2 0 e n}. \tag {5}
$$
Furthermore, as in the proof of Theorem 1, there exists a vector  $w_{\varepsilon}^{*}$  which holds  $\| w_{\varepsilon}^{*}\| \leq \phi^{-1}(\varepsilon)\big / \gamma$ . Then by Lemma 2 and the choice of  $\varepsilon$ ,
$$
\widehat {L} \left(w _ {T}\right) \leq 2 \varepsilon + \frac {2 \phi^ {- 1} (\varepsilon) ^ {2}}{\gamma^ {2}} \leq 4 \varepsilon . \tag {6}
$$
For the remainder of the proof, we condition on the event  $A_{1}\cap A_{2}$ . First, we show that  $w_{t}\cdot z_{2}\geq 0$ . Indeed, if it were not the case, then  $\ell (w_T\cdot z_2) > \phi (0)$ ; together with Eq. (6) we obtain
$$
\frac {1}{6 4} > 4 \varepsilon \geq \widehat {L} (w _ {T}) \geq \delta_ {2} \ell (w _ {T} \cdot z _ {2}) \geq \frac {1}{3 2} \phi (0).
$$
which is a contradiction to  $\phi(0) \geq \frac{1}{2}$ . Moreover,  $w_{T}(1) \geq 0$ . Again, we show this by contradiction. Conditioned on  $A_{2}$ , we have  $\delta_{1} > \frac{7}{8}$ . Then, if  $w_{T}(1) < 0$ ,  $\ell(w_{T} \cdot z_{1}) > \phi(0)$ , and
$$
\frac {1}{6 4} > 4 \varepsilon \geq \widehat {L} (w _ {T}) \geq \delta_ {1} \ell (w _ {T} \cdot z _ {1}) > \frac {7}{8} \phi (0) \geq \frac {7}{1 6}.
$$
which is a contradiction. In addition, we notice that  $z_{3}$  is the only possible example whose third entry is non zero. Given the event  $A_{1}$ , we know that  $z_{3}$  is not in  $S$ . Equivalently, for every  $z \in S$ ,  $z(3) = 0$ . As a result,
$$
w _ {t} (3) = - \eta \sum_ {s = 1} ^ {t - 1} \nabla \widehat {L} (w _ {t}) = - \eta \sum_ {s = 1} ^ {t - 1} \frac {1}{n} \sum_ {z \in S} z (3) \ell^ {\prime} (w _ {s} z) = 0.
$$
Then, we get that,
$$
w _ {T} \cdot z _ {3} = - \frac {1}{8} w _ {t} (2). \tag {7}
$$
Then, using the fact that  $w_{T} \cdot z_{2} \geq 0$ ,  $\ell(w_{T} \cdot z_{2}) = \phi(w_{T} \cdot z_{2})$ , and conditioned on  $A_{2}$ , we have
$$
\ell \left(w _ {T} \cdot z _ {2}\right) = \phi \left(w _ {T} \cdot z _ {2}\right) \leq 3 2 \widehat {L} \left(w _ {T}\right)),
$$
$$
w _ {T} \cdot z _ {2} \geq \phi^ {- 1} \left(3 2 \widehat {L} \left(w _ {T}\right)\right). \tag {8}
$$
Therefore, by combining Eq. (8) with the fact that  $w_{t}(1) \geq 0$ ,
$$
3 \gamma w _ {T} (2) \geq - \frac {w _ {T} (1)}{2} + 3 \gamma w _ {T} (2) = w _ {T} \cdot z _ {2} \geq \phi^ {- 1} \left(3 2 \widehat {L} \left(w _ {T}\right)\right).
$$
which implies,  $w_{T}(2)\geq \frac{1}{3\gamma}\phi^{-1}(32\widehat{L} (w_{T}))$  . By Eq. (7),
$$
w _ {T} \cdot z _ {3} = - \frac {1}{8} w _ {T} (2) \leq - \frac {1}{2 4 \gamma} \phi^ {- 1} \left(3 2 \widehat {L} \left(w _ {T}\right)\right).
$$
We therefore see that for every  $\varepsilon$  such that  $\varepsilon \geq (\phi^{-1}(\varepsilon))^2 / \gamma^2 T\eta$ ,
$$
\ell \left(w _ {T} \cdot z _ {3}\right) \geq \frac {\beta}{2} \left(w _ {T} \cdot z _ {3}\right) ^ {2} \geq \frac {\beta}{2} \left(\frac {1}{2 4 \gamma} \phi^ {- 1} \left(3 2 \widehat {L} \left(w _ {T}\right)\right)\right) ^ {2} \geq \frac {\beta}{1 1 5 2 \gamma^ {2}} \left(\phi^ {- 1} (1 2 8 \varepsilon)\right) ^ {2},
$$
where in the final inequality we again used Eq. (6). We conclude the proof using Eq. (5) and the law of total expectation,
$$
\mathbb {E} \left[ L \left(w _ {T}\right) \right] = \mathbb {E} \left[ \ell \left(w _ {T} \cdot z ^ {\prime}\right) \right] \geq \mathbb {E} \left[ \ell \left(w _ {T} \cdot z ^ {\prime}\right) \mid A _ {1} \cap A _ {2} \right] \Pr \left(A _ {1} \cap A _ {2}\right).
$$
(Expectations here are taken with respect to both the sample  $S$  and the validation example  $z'$ .)
Second, we focus the second expression in the lower bound of Theorem 2, which is dominant in the early stages of optimization.
Lemma 4. There exists a constant  $C_2$  such that the following holds. For any tail function  $\phi$ , and for any  $n, T$  and  $\gamma$ , there exist a distribution  $\mathcal{D}$  with margin  $\gamma$ , a loss function  $\ell \in C_{\phi, \beta}$  such that for  $GD$  initialized at  $w_1 = 0$  with stepsize  $\eta \leq 1/2\beta$  over an i.i.d. sample  $S = \{z_i\}_{i=1}^n$  from  $\mathcal{D}$  and  $w_1 = 0$  holds
$$
\mathbb {E} [ L (w _ {T}) ] \geq C _ {2} \frac {(\phi^ {- 1} (8 \varepsilon)) ^ {2}}{\gamma^ {2} T \eta},
$$
for any  $\varepsilon \leq \frac{1}{16}$  such that  $\eta \gamma^2 T \geq \phi^{-1}(\varepsilon)^2 / \varepsilon$ .
The argument for proving Lemma 4 is similar to that of Lemma 3. Here we define a 1-Lipschitz loss function  $\ell$  that decays to zero at the same rate as  $\phi$ , and a distribution such that there is a possible example  $z_{1}$  which is sampled with high probability and an almost "opposite" example  $z_{2}$ , that with constant probability, appears rarely in the training set  $S$ . The lower bound follows from the fact that although  $z_{2}$  appears in the dataset, the gradients of the loss function are bounded and thus not large enough so as to make the trained predictor classify  $z_{2}$  correctly. The full proof can be found in Appendix D.2.
Finally, we derive Theorem 2 directly from Lemmas 3 and 4:
Proof of Theorem 2. Let  $C = \frac{1}{2} \min \{C_1, C_2\}$ , where  $C_1$  and  $C_2$  are the constants from Lemmas 3 and 4, respectively. If  $(\phi^{-1}(8\varepsilon))^2 / \gamma^2 T \eta \geq \beta (\phi^{-1}(128\varepsilon))^2 / \gamma^2 n$ , the theorem follows from Lemma 4; otherwise, it follows from Lemma 3.
Funded by the European Union (ERC, OPTGEN, 101078075). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. This work received additional support from the Israel Science Foundation (grant number 2549/19), from the Len Blavatnik and the Blavatnik Family foundation, and from the Adelis Foundation.
[1] Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning, pages 1638-1646. PMLR, 2014.
[2] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
[3] Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018.
[4] Ziwei Ji and Matus Telgarsky. A refined primal-dual analysis of the implicit bias. Journal of Environmental Sciences (China) English Ed, 2019.
[5] Ziwei Ji, Miroslav Dudík, Robert E. Schapire, and Matus Telgarsky. Gradient descent follows the regularization path for general losses. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 2109–2136. PMLR, 09–12 Jul 2020.
[6] Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008.
[7] Yunwen Lei and Yiming Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning, pages 5809-5819. PMLR, 2020.
[8] Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3420-3428. PMLR, 2019.
[9] Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3051-3059. PMLR, 2019.
[10] Deanna Needell, Rachel Ward, and Nati Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[11] Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
[12] Matan Schliserman and Tomer Koren. Stability vs implicit bias of gradient methods on separable data and beyond. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 3380-3394. PMLR, 02-05 Jul 2022.
[13] Ohad Shamir. Gradient methods never overfit on separable data. The Journal of Machine Learning Research, 22(1):3847-3866, 2021.
[14] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-2878, 2018.
[15] Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[16] Matus Telgarsky. Stochastic linear optimization never overfits with quadratically-bounded losses on general data. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 5453-5488. PMLR, 02-05 Jul 2022.