SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 4.1 (The implicit bias of GD for logistic regression). Suppose that Assumptions 1, 2, and 3 hold. Consider  $(\mathbf{w}_t)_{t\geq 0}$  produced by (GD) with initialization $^2$ $\mathbf{w}_0 = 0$  and constant stepsize  $\eta >0$ . Then there exist positive constants  $c_{1},c_{2},c_{3} > 0$  that are upper bounded by a polynomial of  $\left\{e^{\eta},e^{n},e^{1 / b},1 / \eta ,1 / (\theta -\gamma),1 / \gamma ,e^{\theta /\gamma}\right\}$  but are independent of  $t$ , such that:
(A) The risk is upper bounded by
$$
L (\mathbf {w} _ {t}) \leq c _ {1} / t, \quad t \geq 3.
$$
(B) In the max-margin subspace,
$$
\mathcal {P} (\mathbf {w} _ {t}) \geq \log (t) / \gamma + \log (\eta \gamma^ {2} / 2) / \gamma , \quad t \geq 1.
$$
$(C)$  In the non-separable subspace,
$$
\left\| \bar {\mathcal {P}} (\mathbf {w} _ {t}) \right\| _ {2} \leq c _ {2}, \quad t \geq 0.
$$
$(D)$  In addition, in the non-separable subspace,
$$
G \big (\bar {\mathcal {P}} (\mathbf {w} _ {t}) \big) - \min  G (\cdot) \leq c _ {3} / \log (t), \quad t \geq 3,
$$
where  $G(\cdot)$  is a strongly convex potential defined by
$$
G (\mathbf {v}) := \sum_ {i \in \mathcal {S}} \exp \left(- y _ {i} \cdot \langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \mathbf {v} \rangle\right), \quad \mathbf {v} \in \mathbb {R} ^ {d - 1}.
$$
(a) Empirical risk
(b) Sharpness
Figure 2: The behaviors of GD for logistic regression. We randomly sample 1,000 data with labels "0" and "8" from the MNIST dataset and then use GD to perform logistic regression on those data. The sub-figures (a) and (b) report the risk (i.e., the logistic loss) and sharpness (i.e.,  $\| \nabla L(\mathbf{w}_t)\| _2$ ) along the GD trajectories, respectively. For GD with stepsizes  $\eta$  larger than or equal to 0.1, the training losses oscillate locally and the sharpnesses can exceed  $2 / \eta$ . For GD with a small stepsize  $\eta = 0.01$ , the training losses decrease monotonically and the sharpnesses are always below  $2 / \eta$ . More details of the experiments can be found in Appendix D.
Note that Theorem 4.1 applies to GD with any positive constant stepsize, therefore allowing GD to be in the EoS regime. We next discuss the implications of Theorem 4.1 in detail.
Risk minimization. Theorem 4.1(A) guarantees that the GD iterates minimize the logistic loss at a rate of  $\mathcal{O}(1 / t)$  for any constant stepsize, even for those large step sizes that cause local risk oscillations. This result explains the risk convergence of GD in the EoS regime, as illustrated in Figure 2, and is also consistent with the observations in neural network experiments (see Figure 1).
Margin maximization. Theorem 4.1(B) shows that the GD iterates, when projected to the max-margin direction, tend to infinity at a rate of  $\mathcal{O}(\log (t))$ . Moreover, Theorem 4.1(C) shows that the GD iterates, when projected to the non-separable subspace, are uniformly bounded. These two results together imply that the direction of the GD iterates will tend to a max-margin direction, i.e., the hard-margin SVM direction, at a rate of  $\mathcal{O}(1 / \log (t))$ . Therefore, the implicit bias of GD that maximizes the  $\ell_2$ -margin is consistent in both the EoS regime and the small-stepsize regime [Soudry et al., 2018, Ji and Telgarsky, 2018].
Iterate convergence in the non-separable subspace. Theorem 4.1(D) shows that the GD iterates, when projected to the non-separable subspace, converge to the minimizer of a strongly convex potential  $G(\cdot)$ . Here,  $G(\cdot)$  measures the exponential loss of a parameter on the support vectors with their non-separable features. This provides a more precise characterization of the implicit bias of GD: the direction of the GD iterates converges to the hard-margin SVM direction, moreover, the limit of the projections of the GD iterates to the orthogonal complement to the hard-margin SVM direction minimizes the exponential loss on the non-separable features of the support vectors.
Comparison to Theorem 9 in Soudry et al. [2018]. Theorem 9, in particular, equation (18), in Soudry et al. [2018] indirectly characterizes the convergence of GD iterates in the non-separable subspace. It reads in our notations that:  $\tilde{\mathbf{w}} := \lim_{t\to \infty}\left(\mathbf{w}_t - \hat{\mathbf{w}}\log (t)\right)$  exists and satisfies
for every  $i\in \mathcal{S}$  ，  $\eta \cdot \exp (-y_i\cdot \langle \mathbf{x}_i,\tilde{\mathbf{w}}\rangle) = \alpha_i$  ，where  $\alpha_{i}$  is defined in Assumption 3. (1)
In Appendix A, we show that Theorem 4.1(D) is equivalent to condition (1) in terms of describing  $\bar{\mathcal{P}}(\mathbf{w}_{\infty})$ . Despite their equivalence, (1) is less interpretable than Theorem 4.1(D), as (1) entangles an effect of  $\mathcal{P}(\mathbf{w}_{\infty})$  with  $\bar{\mathcal{P}}(\mathbf{w}_{\infty})$ , while Theorem 4.1 completely decouples  $\mathcal{P}(\mathbf{w}_{\infty})$  and  $\bar{\mathcal{P}}(\mathbf{w}_{\infty})$ . In particular, (1) seems to suggest  $\bar{\mathcal{P}}(\mathbf{w}_{\infty})$  to be a function of stepsize  $\eta$  since  $\tilde{\mathbf{w}}$  depends on  $\eta$ . However, this is only an illusion brought by the lack of interpretability of (1); it is clear that  $\bar{\mathcal{P}}(\mathbf{w}_{\infty})$  is independent of  $\eta$  according to Theorem 4.1(D).
Exponential loss. Until now, our theory for GD is consistent for large and small stepsizes. However, this is a particular benefit thanks to the design of the logistic loss, and may not hold for other losses. Our next result suggests that, in the EoS regime where the step sizes are large, GD can diverge catastrophically under the exponential loss.
2The theorem can be easily extended to allow any  $\mathbf{w}_0$  that has a bounded  $\ell_2$ -norm.
Theorem 4.2 (The catastrophic divergence of GD under the exponential loss). Consider a dataset of two samples, where
$$
\mathbf {x} _ {1} = (\gamma , 1), \quad y _ {1} = 1; \qquad \mathbf {x} _ {2} = (\gamma , - 1), \quad y _ {2} = 1.
$$
It is clear that  $(\mathbf{x}_i, y_i)_{i=1,2}$  is linearly separable and  $(1,0)$  is the max-margin direction. Consider a risk defined by the exponential loss:
$$
L (w, \bar {w}) := \exp (- y _ {1} \langle \mathbf {x} _ {1}, \mathbf {w} \rangle) + \exp (- y _ {2} \langle \mathbf {x} _ {2}, \mathbf {w} \rangle) = e ^ {- \gamma w} \cdot \left(e ^ {- \bar {w}} + e ^ {\bar {w}}\right), \quad \text {w h e r e} \mathbf {w} = (w, \bar {w}).
$$
Let  $(w_{t},\bar{w}_{t})_{t\geq 0}$  be the iterates produced by GD with constant stepsize  $\eta$  for optimizing  $L(w,\bar{w})$ . If
$$
0 \leq w _ {0} \leq 2, \quad | \bar {w} _ {0} | \geq 1, \quad 0 <   \gamma <   1 / 4, \quad \eta \geq 4,
$$
(A)  $L(w_{t},\bar{w}_{t})\to \infty$
(B)  $w_{t}\to \infty$
(C) For every  $t \geq 0$ ,  $|\bar{w}_t| \geq 2\gamma w_t$ .
$(D)$  Moreover, the sign of  $\bar{\psi}_t$  flips every iteration.
As a consequence,  $(w_{t},\bar{w}_{t})_{t\geq 0}$  diverge in terms of either magnitude or direction; in particular, the direction of  $(w_{t},\bar{w}_{t})_{t\geq 0}$  cannot converge to the max-margin direction (which is  $(1,0)$ ).
Theorem 4.2 shows that with a large constant stepsize, the GD iterates no longer minimize the risk defined by the exponential loss and no longer converge along the max-margin direction. In fact, the directions of the GD iterates flip every step, thus the direction of the GD iterates necessarily diverges, resulting in no meaningful implicit bias at all.
In the EoS regime, large-stepsize GD still behaves nicely under the logistic loss (Theorem 4.1) but can behave catastrophically under the exponential loss (Theorem 4.2). From a mathematical standpoint, this difference is rooted in the fact that the gradient of the logistic loss is uniformly bounded while the gradient of the exponential loss could be extremely large. From a practical standpoint, it provides insights into why the logistics loss (and its multi-class version, the cross-entropy loss) is preferable to the exponential loss in practice.
The different behaviors of large-stepsize GD under the logistic and exponential losses also sharply contrast the EoS regime with the small-stepsize regime. Because in the small-stepsize regime, the convergence and implicit bias of GD are known to be similar under any exponentially-tailed losses, including the logistic and exponential losses [Soudry et al., 2018, Ji and Telgarsky, 2018].
5 Techniques Overview
The proofs of Theorems 4.1 and 4.2 are deferred to Appendix C. In this section, we explain the proof ideas of Theorem 4.1 by analyzing a simple dataset considered in Theorem 4.2 (the treatment to the general datasets can be found in Appendix B). But this time we work with the logistic loss instead of the exponential loss, that is,
$$
L (w, \bar {w}) = \log (1 + e ^ {- \gamma w - \bar {w}}) + \log (1 + e ^ {- \gamma w + \bar {w}}).
$$
Then the GD iterates can be written as
$$
w _ {t + 1} = w _ {t} - \eta \cdot g _ {t}, \quad \bar {w} _ {t + 1} = \bar {w} _ {t} - \eta \cdot \bar {g} _ {t},
$$
$$
g _ {t} := - \gamma \cdot \left(\frac {1}{1 + e ^ {\gamma w _ {t} + \bar {w} _ {t}}} + \frac {1}{1 + e ^ {\gamma w _ {t} - \bar {w} _ {t}}}\right), \quad \bar {g} _ {t} := - \left(\frac {1}{1 + e ^ {\gamma w _ {t} + \bar {w} _ {t}}} - \frac {1}{1 + e ^ {\gamma w _ {t} - \bar {w} _ {t}}}\right).
$$
For simplicity, assume that
$$
w _ {0} = 0, \quad | \bar {w} _ {0} | > 0.
$$
Different from Soudry et al. [2018], Ji and Telgarsky [2018], our approach begins with showing the implicit bias (despite that the risk may oscillate). The long-term risk convergence is then simply a consequence of the implicit bias results.
Step 1:  $(\bar{w}_t)_{t \geq 0}$  is uniformly bounded. Observe that  $\bar{g}_t$  and  $\bar{w}_t$  always share the same sign and that  $|\bar{g}_t| \leq 1$ , so we have
$$
\left| \bar {w} _ {t + 1} \right| = \left| \left| \bar {w} _ {t} \right| - \eta \cdot \left| \bar {g} _ {t} \right| \right| \leq \max  \left\{\left| \bar {w} _ {t} \right|, \eta \cdot \left| \bar {g} _ {t} \right| \right\} \leq \max  \left\{\left| \bar {w} _ {t} \right|, \eta \right\}.
$$
By induction, we get that  $\left(|\bar{w}_t|\right)_{t\geq 0}$  is uniformly bounded by  $\max \{| \bar{w}_0 |, \eta \} = \Theta(1)$ .
Step 2:  $w_{t} \approx \log (t) / \gamma$ . We turn to study the max-margin subspace. It is clear that  $g_{t} \leq 0$  for every  $t \geq 0$ . So we have  $w_{t} \geq 0$  by induction. Moreover, we have
$$
- \frac {g _ {t}}{\gamma} = \frac {e ^ {- \gamma w _ {t} - \bar {w} _ {t}}}{1 + e ^ {- \gamma w _ {t} - \bar {w} _ {t}}} + \frac {e ^ {- \gamma w _ {t} + \bar {w} _ {t}}}{1 + e ^ {- \gamma w _ {t} + \bar {w} _ {t}}} \leq e ^ {- \gamma w _ {t}} \cdot e ^ {- \bar {w} _ {t}} + e ^ {- \gamma w _ {t}} \cdot e ^ {\bar {w} _ {t}} \leq e ^ {- \gamma w _ {t}} \cdot \Theta (1),
$$
where the last inequality is because  $|\bar{w}_t|$  is uniformly bounded. We also have
$$
\begin{array}{l} - \frac {g t}{\gamma} = \frac {e ^ {- \gamma w _ {t} - \bar {w} _ {t}}}{1 + e ^ {- \gamma w _ {t} - \bar {w} _ {t}}} + \frac {e ^ {- \gamma w _ {t} + \bar {w} _ {t}}}{1 + e ^ {- \gamma w _ {t} + \bar {w} _ {t}}} \geq 0. 5 \cdot \min  \{1, e ^ {- \gamma w _ {t}} e ^ {- \bar {w} _ {t}} \} + 0. 5 \cdot \min  \{1, e ^ {- \gamma w _ {t}} e ^ {\bar {w} _ {t}} \} \\ \geq 0. 5 \cdot \min  \left\{1, e ^ {- \gamma w _ {t}} e ^ {- \bar {w} _ {t}} + e ^ {- \gamma w _ {t}} e ^ {\bar {w} _ {t}} \right\} \geq 0. 5 \cdot \min  \left\{1, e ^ {- \gamma w _ {t}} \right\} = 0. 5 \cdot e ^ {- \gamma w _ {t}}, \\ \end{array}
$$
where the third inequality is because  $e^{-\bar{w}_t} + e^{\bar{w}_t} \geq 1$  and the last equality is because  $w_t \geq 0$ . Putting these together, we have
$$
g _ {t} \approx - \gamma \cdot e ^ {- \gamma w _ {t}} \cdot \Theta (1) \Rightarrow w _ {t + 1} \approx w _ {t} - \eta \gamma \cdot e ^ {- \gamma w _ {t}} \cdot \Theta (1) \Rightarrow w _ {t} = \log (t) / \gamma \pm \Theta (1). (2)
$$
Step 3:  $\bar{g}_t\approx \exp (-\gamma w_t)\cdot \nabla G(\bar{w}_t)$ . We turn back to the non-separable subspace. Note that  $\bar{g}_t$  is an odd function of  $\bar{w}_t$ . Without loss of generality, let us assume  $\bar{w}_t\geq 0$  in this part. Notice that
$$
\text {f o r e v e r y f i x e d} a > 1, f (t) := \frac {1}{t + 1 / a} - \frac {1}{t + a} \text {i s a d e c r e a s i n g f u n c t i o n o f} t \geq 0. \tag {3}
$$
$$
\bar {g} _ {t} = e ^ {- \gamma w _ {t}} \cdot \left(\frac {1}{e ^ {- \gamma w _ {t}} + e ^ {- \bar {w} _ {t}}} - \frac {1}{e ^ {- \gamma w _ {t}} + e ^ {\bar {w} _ {t}}}\right) \leq e ^ {- \gamma w _ {t}} \cdot \left(\frac {1}{e ^ {- \bar {w} _ {t}}} - \frac {1}{e ^ {\bar {w} _ {t}}}\right) =: e ^ {- \gamma w _ {t}} \cdot \nabla G (\bar {w} _ {t}),
$$
where the inequality is by (3), and  $G(\bar{w}) \coloneqq e^{\bar{w}} + e^{-\bar{w}}$  is defined as in Theorem 4.1(D). On the other hand, since  $|\bar{w}_t|$  is bounded and  $w_t$  is increasing (and tends to infinity), there must exist a time  $t_0$  such that  $e^{-\gamma w_t} \leq e^{-|\bar{w}_t|}$  for every  $t \geq t_0$ . Then for  $t \geq t_0$  we have
$$
\begin{array}{l} \bar {g} _ {t} = e ^ {- \gamma w _ {t}} \cdot \left(\frac {1}{e ^ {- \gamma w _ {t}} + e ^ {- \bar {w} _ {t}}} - \frac {1}{e ^ {- \gamma w _ {t}} + e ^ {\bar {w} _ {t}}}\right) \geq e ^ {- \gamma w _ {t}} \cdot \left(\frac {1}{2 e ^ {- \bar {w} _ {t}}} - \frac {1}{e ^ {- \bar {w} _ {t}} + e ^ {\bar {w} _ {t}}}\right) \\ = e ^ {- \gamma w _ {t}} \cdot \frac {e ^ {\bar {w} _ {t}} - e ^ {- \bar {w} _ {t}}}{2 e ^ {- 2 \bar {w} _ {t}} + 2} \geq e ^ {- \gamma w _ {t}} \cdot \frac {e ^ {\bar {w} _ {t}} - e ^ {- \bar {w} _ {t}}}{4} =: \frac {1}{4} \cdot e ^ {- \gamma w _ {t}} \cdot \nabla G (\bar {w} _ {t}), \\ \end{array}
$$
where the first inequality is by (3) and  $e^{-\gamma w_t} \leq e^{-\bar{w}_t}$ , and the last inequality is because we assume  $\bar{w}_t \geq 0$ . Putting these together, and using (2), we obtain that
$$
f o r e v e r y t \geq t _ {0}, \bar {w} _ {t + 1} = \bar {w} _ {t} - \eta_ {t} \cdot \nabla G (\bar {w} _ {t}), \text {w h e r e} \eta_ {t} \approx \eta \cdot e ^ {- \gamma w _ {t}} \cdot \Theta (1) \approx \Theta (1) / t. \tag {4}
$$
Step 4: a modified descent lemma. Using (4) and Taylor's expansion, we have
$$
\text {f o r e v e r y} t \geq t _ {0}, G (\bar {w} _ {t + 1}) \leq G (\bar {w} _ {t}) - \eta_ {t} \cdot \| \nabla G (\bar {w} _ {t}) \| ^ {2} + \frac {\beta}{2} \cdot \eta_ {t} ^ {2} \cdot \| \nabla G (\bar {w} _ {t}) \| ^ {2} \leq G (\bar {w} _ {t}) + \frac {\Theta (1)}{t ^ {2}},
$$
where  $\beta \coloneqq \sup_{|\bar{v} |\leq \max \{| \bar{w}_0|,\eta \}}\| \nabla^2 G(\bar{v})\| _2 = \Theta (1)$ . Taking a telescoping sum from  $t$  to  $T$ , we have
$$
\text {f o r e v e r y} T \geq t \geq t _ {0}, \quad G (\bar {w} _ {T}) \leq G (\bar {w} _ {t}) + \Theta (1) / t. \tag {5}
$$
Step 5: the convergence of  $\bar{w}_t$ . What remains is adapted from classic convergence arguments. Choose  $\bar{w}_{*} = \arg \min G(\cdot)$ , then
$$
\begin{array}{l} \left\| \bar {w} _ {t + 1} - \bar {w} _ {*} \right\| _ {2} ^ {2} = \left\| \bar {w} _ {t} - \bar {w} _ {*} \right\| _ {2} ^ {2} - 2 \eta_ {t} \cdot \left\langle \bar {w} _ {t} - \bar {w} _ {*}, \nabla G (\bar {w} _ {t}) \right\rangle + \eta_ {t} ^ {2} \cdot \left\| \nabla G (\bar {w} _ {t}) \right\| _ {2} ^ {2} \\ \leq \left\| \bar {w} _ {t} - \bar {w} _ {*} \right\| _ {2} ^ {2} - 2 \eta_ {t} \cdot \left(G (\bar {w} _ {t}) - G (\bar {w} _ {*})\right) + \Theta (1) / t ^ {2}, \quad t \geq t _ {0}, \\ \end{array}
$$
where the equality is by (4), and the inequality is because of the convexity of  $G(\cdot)$ ,  $|\bar{w}_t| \leq \Theta(1)$ , and (4). Taking a telescoping sum, we have
$$
\sum_ {t = t _ {0}} ^ {T} 2 \eta_ {t} \cdot (G (\bar {w} _ {t}) - G (\bar {w} _ {*})) \leq \| \bar {w} _ {t _ {0}} - \bar {w} _ {*} \| _ {2} ^ {2} - \| \bar {w} _ {T + 1} - \bar {w} _ {*} \| _ {2} ^ {2} + \sum_ {t = t _ {0}} ^ {T} \Theta (1) / t ^ {2} \leq \Theta (1).
$$
Combining the above with (5) and using  $\eta_t \approx \Theta(1)/t$  from (4), we get
$$
\sum_ {t = t _ {0}} ^ {T} \eta_ {t} \cdot \left(G (\bar {w} _ {T}) - G (\bar {w} _ {*})\right) \leq \sum_ {t = t _ {0}} ^ {T} \eta_ {t} \cdot \left(G (\bar {w} _ {t}) - G (\bar {w} _ {*})\right) + \sum_ {t = t _ {0}} ^ {T} \eta_ {t} \cdot \Theta (1) / t \leq \Theta (1).
$$
Finally, since  $\sum_{t = t_0}^T\eta_t\geq \Theta (1)\cdot (\log (T) - \log (t_0))$  according to (4), we get that  $G(\bar{w}_T) - G(\bar{w}_*)\leq \Theta (1) / (\log (T) - \log (t_0))$ .
Step 6: risk convergence. The long-term risk convergence result can be easily established by making use of the implicit bias results we have obtained so far.
We consider constant-stepsize GD for logistic regression on linearly separable data. We show that with any constant stepsize, GD minimizes the logistic loss; moreover, the GD iterates tend to infinity when projected to a max-margin direction and tend to a fixed minimizer of a strongly convex potential when projected to the orthogonal complement of the max-margin direction. We also show that GD with a large stepsize may diverge catastrophically if the logistic loss is replaced by the exponential loss. Our theory explains how GD minimizes a risk non-monotonically.
We thank the anonymous reviewers for their helpful comments and Alexander Tsigler for pointing out several typos. VB is partially supported by the Ministry of Trade, Industry and Energy(MOTIE) and Korea Institute for Advancement of Technology (KIAT) through the International Cooperative R&D program. JDL acknowledges the support of the ARO under MURI Award W911NF-11-1-0304, the Sloan Research Fellowship, NSF CCF 2002272, NSF IIS 2107304, NSF CIF 2212262, ONR Young Investigator Award, and NSF CAREER Award 2144994.
Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the" edge of stability". arXiv preprint arXiv:2212.07469, 2022a.
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning, pages 247-257. PMLR, 2022b.
Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pages 903-925. PMLR, 2023.
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning, pages 948-1024. PMLR, 2022.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Peter L Bartlett, Philip M Long, and Olivier Bousquet. The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima. arXiv preprint arXiv:2210.01513, 2022.
Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normalization layers: Sharpness reduction. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=xp5V0BxTxZ.
Chao Ma, Daniel Kunin, Lei Wu, and Lexing Ying. Beyond the quadratic approximation: The multiscale structure of neural network loss landscapes. Journal of Machine Learning, 1(3):247-267, 2022. ISSN 2790-2048. doi: https://doi.org/10.4208/jml.220404. URL http://global-sci.org/intro/article_detail/jml/21028.html.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning, pages 4683-4692. PMLR, 2019a.
Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3420-3428. PMLR, 2019b.
Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3051-3059. PMLR, 2019c.
Yurii Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Skz_WfbCZ.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. *The Journal of Machine Learning Research*, 19(1): 2822-2878, 2018.
Yuqing Wang, Minshuo Chen, Tuo Zhao, and Molei Tao. Large learning rate tames homogeneity: Convergence and balancing effect. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum?id=3tbDrs77LJ5.
Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along GD trajectory: Progressive sharpening and edge of stability. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022b. URL https://openreview.net/forum?id=thgItcQrJ4y.
Lei Wu, Chao Ma, et al. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31, 2018.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018.
Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding edge-of-stability training dynamics with a minimalist example. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=p7EagBsMAEO.
A On the Equivalence between Theorem 4.1(D) and (1)
Note that  $\tilde{\mathbf{w}}$  in (1) contains components in both the max-margin and non-separable subspaces, and we need to disentangle those two components.
Under the coordinate system that defines  $\mathcal{P}$  and  $\bar{\mathcal{P}}$ , we can represent a vector  $\mathbf{v} \in \mathbb{R}^d$  as
$$
\mathbf {v} := \left(\mathcal {P} (\mathbf {v}), \bar {\mathcal {P}} (\mathbf {v})\right).
$$
Then for  $i\in S$  , we have
$$
\begin{array}{l} y _ {i} \cdot \left\langle \mathbf {x} _ {i}, \tilde {\mathbf {w}} \right\rangle = y _ {i} \cdot \left\langle \left(\mathcal {P} (\mathbf {x} _ {i}), \bar {\mathcal {P}} (\mathbf {x} _ {i})\right), \left(\mathcal {P} (\tilde {\mathbf {w}}), \bar {\mathcal {P}} (\tilde {\mathbf {w}})\right) \right\rangle \\ = y _ {i} \cdot \mathcal {P} (\mathbf {x} _ {i}) \cdot \mathcal {P} (\tilde {\mathbf {w}}) + y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \bar {\mathcal {P}} (\tilde {\mathbf {w}}) \right\rangle \quad \text {s i n c e} \mathcal {P} \text {a n d} \bar {\mathcal {P}} \text {a r e o r t h o g o n a l} \\ = \gamma \mathcal {P} (\tilde {\mathbf {w}}) + y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \bar {\mathcal {P}} (\tilde {\mathbf {w}}) \right\rangle . \qquad \text {s i n c e} y _ {i} \mathcal {P} (\mathbf {x} _ {i}) = \gamma \text {f o r} i \in \mathcal {S} \\ \end{array}
$$
So (1) is equivalent to
$$
\text {f o r e v e r y} i \in \mathcal {S}, \quad \eta \exp (- \gamma \bar {\mathcal {P}} (\tilde {\mathbf {w}})) \cdot \exp (- y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \bar {\mathcal {P}} (\tilde {\mathbf {w}}) \right\rangle) = \alpha_ {i}.
$$
Recall that  $\sum_{i\in S}\alpha_i = 1 / \gamma^2$  according to Definition 1. So focusing on  $\bar{\mathcal{P}}$ , the above is equivalent to the following condition on  $\bar{\mathcal{P}} (\tilde{\mathbf{w}})$ :
$$
\alpha_ {i} \propto \exp \left(- y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \bar {\mathcal {P}} (\tilde {\mathbf {w}}) \right\rangle\right), i \in \mathcal {S}. \tag {6}
$$
Here we ignore a shared normalization factor.
Now, recall from Assumption 3(B) that  $(\alpha_{i})_{i\in S}$  are such that