SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abernethy, J., Lai, K. A., Levy, K. Y., and Wang, J.-K. Faster rates for convex-concave games. In Proceedings of the 31st Annual Conference on Learning Theory, pp. 1595-1625, 2018.
Azizan, N. and Hassibi, B. Stochastic gradient/mirror descent: Minimax optimality and implicit regularization. In the 7th International Conference on Learning Representations, 2019.
Boyd, S. and Vandenberghe, L. Convex Optimization. Cambridge University Press, 2004.
Bubeck, S. et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning, 8(3-4):231-357, 2015.
Chen, S. S., Donoho, D. L., and Saunders, M. A. Atomic decomposition by basis pursuit. SIAM review, 43(1):129-159, 2001.
Cutkosky, A. and Orabona, F. Black-box reductions for parameter-free online learning in banach spaces. In Proceedings of the 31st Annual Conference on Learning Theory, 2018.
Daskalakis, C. and Panageas, I. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In 10th Innovations in Theoretical Computer Science Conference, 2019.
Daskalakis, C., Ilyas, A., Syrgkanis, V., and Zeng, H. Training gans with optimism. In the 6th International Conference on Learning Representations, 2018.
Gentile, C. A new approximate maximal margin classification algorithm. Advances in Neural Information Processing Systems 13, 2000.
Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Characterizing implicit bias in terms of optimization geometry. In Proceedings of the 35th International Conference on Machine Learning, pp. 1832-1841, 2018a.
Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems 31, 2018b.
Hazan, E. Introduction to online convex optimization. Foundations and Trends in Optimization, 2 (3-4):157-325, 2016.
Ji, Z. and Telgarsky, M. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018.
Ji, Z. and Telgarsky, M. Gradient descent aligns the layers of deep linear networks. In the 8th International Conference on Learning Representations, 2020a.
Ji, Z. and Telgarsky, M. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems 33, 2020b.
Ji, Z. and Telgarsky, M. Characterizing the implicit bias via a primal-dual analysis. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, pp. 772-804, 2021.
Ji, Z., Dudík, M., Schapire, R. E., and Telgarsky, M. Gradient descent follows the regularization path for general losses. In Proceedings of the 33rd Annual Conference on Learning Theory, pp. 2109-2136. PMLR, 2020.
Ji, Z., Srebro, N., and Telgarsky, M. Fast margin maximization via dual acceleration. In Proceedings of the 38th International Conference on Machine Learning, pp. 4860-4869, 2021.
Lai, K.-W. and Muthukumar, V. General loss functions lead to (approximate) interpolation in high dimensions. arXiv preprint arXiv:2303.07475, 2023.
Wang, J.-K., Abernethy, J., and Levy, K. Y. No-regret dynamics in the fenchel game: A unified framework for algorithmic convex optimization. arXiv preprint arXiv:2111.11309, 2021b.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Zhang, L., Wang, G., Tu, W.-W., and Zhou, Z.-H. Dual adaptivity: A universal algorithm for minimizing the adaptive regret of convex functions. ArXiv e-prints, arXiv:1906.10851, 2019.
Zhang, M., Zhao, P., Luo, H., and Zhou, Z.-H. No-regret learning in time-varying zero-sum games. In Proceedings of the 39th International Conference on Machine Learning, 2022.
As discussed in the introduction, the implicit bias of GD-based methods on linear spreadable data with exponentially-tailed loss has been extensively studied (Soudry et al., 2018; Ji & Telgarsky, 2018; Nacson et al., 2019; Ji & Telgarsky, 2021; Ji et al., 2021; Wang et al., 2022b). For  $\ell_1$ -norm, Telgarsky (2013) provides an  $O\left(\frac{1}{\sqrt{T}}\right)$  margin maximization rate, and an  $O\left(\frac{1}{T}\right)$  convergence rate to a sub-optimal margin. The idea of using the regularized game framework for analyzing the implicit bias of Nesterov-accelerated GD is firstly considered in Wang et al. (2022b). This idea is motivated by the smooth perceptron analysis of Soheili & Pena (2012), and the Fenchel game framework by Wang et al. (2021b). The idea also inspired by the primal-dual analysis of Ji & Telgarsky (2021), who discovered the relationship between the normalized gradient descent for the exponential loss and a smooth update of a distribution over data. Apart from the setting above, the implicit bias of GD-based optimization methods have also been studied in other cases, such as minimizing more general loss functions (Ji et al., 2020; Ji & Telgarsky, 2021; Lai & Muthukumar, 2023), and deep neural networks (Gunasekar et al., 2018b; Ji & Telgarsky, 2020a,b; Lyu & Li, 2020; Vardi, 2022).
Compared with GD-based methods, the implicit bias of generic optimization methods for linear classification is less understood. Apart from the work summarized in the introduction (Gunasekar et al., 2018a; Nacson et al., 2019; Li et al., 2021; Sun et al., 2022), the implicit bias of mirror descent is also studied in regression problems (Gunasekar et al., 2018a; Azizan & Hassibi, 2019). We note that, as indicated in Gunasekar et al. (2018a) and Sun et al. (2022), the analysis of implicit bias for classification and regression are "fundamentally different". Apart from steepest and mirror descent, the implicit bias of other generic methods, such as Adagrad and Adam are also studied (Gunasekar et al., 2018a; Wang et al., 2022a)
Discussion on large enough  $T$  Our Theorem 1 indicates that to ensure the margin can be maximized, the number of iterations  $T$  has to be large enough such that  $C_T \leq \frac{\gamma^2}{4}$ . We note that, similar conditions are also required (explicitly or implicitly) in other work for analyzing the margin maximization rates for generic optimization methods. For example, in the proof of Theorem 8 of Nacson et al. (2019), in order to combine the upper bound in (53) and lower bound in (54), one need to make sure the LHS of (53), i.e., the margin, is non-negative, which essentially hinge on the condition that  $\sqrt{T} = \Omega\left(\frac{\log n\log T}{\gamma^2}\right)$ , similar to what is required in our Corollary 1. In Sun et al. (2022), page 21, to make sure the term  $1 + \frac{\log n}{m_t}$  be a constant,  $T$  has to be large enough such that  $m_t$ , the margin, goes to  $\infty$ . In Li et al. (2021), the requirements for a large enough  $T$  is stated in the main theorem (e.g., Theorem 4.2).
Comparison with Wang et al. (2022b) As discussed in the introduction, our work is motivated by Wang et al. (2022b). Compared with Wang et al. (2022b), we note that: 1) Wang et al. (2022b) draws the connection between Nesterov-accelerated GD for ERM and solving the bilinear game through an online dynamic. However, it was unclear whether this kind of analysis suits other gradient-descent-based methods, and generic optimization methods such as mirror descent/steepest descent was not addressed at all. We observe in this work that the non-linearity of the mirror map in generic optimization methods, such as mirror descent and steepest descent, makes the analysis particularly challenging. In this paper, we reveal that the game framework can in fact encompass implicit bias analysis for a range of generic optimization methods, and offer a more streamlined and unified analysis. 2) Wang et al. (2022b) also proposed an accelerated p-norm perceptron problem. However, they only demonstrated that the algorithm could achieve a non-negative margin, leaving open questions regarding whether the margin can be maximized, and if so, what the margin maximization rate would be; 2) They only presented the online dynamic, without its equivalent optimization form under ERM.
Define  $m(\mathbf{w}) = \min_{\mathbf{p} \in \Delta^n} g(\mathbf{p}, \mathbf{w})$ ,  $\overline{\mathbf{w}}_T = \frac{1}{\sum_{t=1}^T \alpha_t} \sum_{t=1}^T \alpha_t \mathbf{w}_t = \frac{1}{\sum_{t=1}^T \alpha_t} \widetilde{\mathbf{w}}_T$ , and we introduce following lemma, which shows that using online learning for solving the game defined in (2) maximizes  $m(\mathbf{w})$ .
Lemma 1 (Abernethy et al. (2018)). Consider solving the game defined in (2) with the online learning dynamic defend in Protocol 1. We have that  $\forall \mathbf{w} \in \mathbb{R}^d$ ,
$$
m (\mathbf {w}) - m (\overline {{\mathbf {w}}} _ {T}) \leq \frac {\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}}{\sum_ {t = 1} ^ {T} \alpha_ {t}}.
$$
Based on Lemma 1 and the definition of  $m(\cdot)$ , we have
$$
\begin{array}{l} m (\overline {{\mathbf {w}}} _ {T}) = m \left(\frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}}\right) = \min _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} - \frac {1}{2} \left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \right\| ^ {2} \\ \geq m \left(\left\| \frac {\tilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \right\| \mathbf {w} _ {\parallel \cdot \parallel} ^ {*}\right) - \frac {\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \tag {9} \\ = \gamma \left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \right\| - \frac {1}{2} \left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \right\| ^ {2} - \frac {\mathrm {R e g} _ {T} ^ {\mathbf {p}} + \mathrm {R e g} _ {T} ^ {\mathbf {w}}}{\sum_ {t = 1} ^ {T} \alpha_ {t}}, \\ \end{array}
$$
$$
\frac {\underset {\mathbf {p} \in \Delta^ {n}} {\min } \mathbf {p} ^ {\top} \mathbf {A} \widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \|} \geq \gamma - \frac {\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}}{\| \widetilde {\mathbf {w}} _ {T} \|}. \tag {10}
$$
The above proof follows the main idea in Wang et al. (2022b). Next, we turn to lower bound  $\|\widetilde{\mathbf{w}}_T\|$ . Note that since  $\mathbf{w}_T$  (and also  $\widetilde{\mathbf{w}}_T$ ) does not have a simple explicit form, the technique for lower bounding the norm in (Wang et al., 2022b) fails and we need to find a new approach. Let  $(\mathbf{x}, y) \in \{(\mathbf{x}^{(i)}, y^{(i)})\}_{i=1}^n$  be a data point. We have
$$
\left\| \widetilde {\mathbf {w}} _ {T} \right\| \geq \left\| y \mathbf {x} \right\| _ {*} \left\| \widetilde {\mathbf {w}} _ {T} \right\| \geq y \mathbf {x} ^ {\top} \widetilde {\mathbf {w}} _ {T} \geq \min  _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \widetilde {\mathbf {w}} _ {T}, \tag {11}
$$
where the first inequality is due to assumption that  $\| \mathbf{x}\|_{*}\leq 1$ , the second inequality is derived from the Cauchy-Schwarz inequality. To proceed, we need a lower bound on the unnormalized margin of  $\widetilde{\mathbf{w}}_T$ . We have
$$
\begin{array}{l} m \left(\overline {{\mathbf {w}}} _ {T}\right) = m \left(\frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}}\right) = \min _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} - \frac {1}{2} \left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \right\| ^ {2} \\ \geq m \left(\gamma \mathbf {w} _ {\| \cdot \|} ^ {*}\right) - \frac {\operatorname {R e g} _ {T} ^ {\mathbf {P}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \tag {12} \\ = \min  _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \mathbf {w} _ {\| \cdot \|} ^ {*} - \frac {1}{2} \| \gamma \mathbf {w} _ {\| \cdot \|} ^ {*} \| ^ {2} - \frac {\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \\ = \frac {\gamma^ {2}}{2} - \frac {\mathrm {R e g} _ {T} ^ {\mathbf {p}} + \mathrm {R e g} _ {T} ^ {\mathbf {w}}}{\sum_ {t = 1} ^ {T} \alpha_ {t}}, \\ \end{array}
$$
where for the first inequality we apply Lemma 1 and compare  $m(\widetilde{\mathbf{w}}_T)$  with that of  $\gamma \mathbf{w}_{\| \cdot \|}^*$ , and the last equality is derived based on Assumption 1 (the margin of  $\mathbf{w}_{\| \cdot \|}^*$  is  $\gamma$ , and  $\| \mathbf{w}_{\| \cdot \|}^* \| = 1$ ). (12) suggests that
$$
\min  _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \widetilde {\mathbf {w}} _ {T} \geq \underbrace {\frac {1}{2} \frac {\left\| \widetilde {\mathbf {w}} _ {T} \right\| ^ {2}}{\sum_ {t = 1} ^ {T} \alpha_ {t}}} _ {\geq 0} + \frac {\gamma^ {2}}{2} \sum_ {t = 1} ^ {T} \alpha_ {t} - \left(\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}\right) \geq \frac {\gamma^ {2}}{2} \sum_ {t = 1} ^ {T} \alpha_ {t} - \left(\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}\right). \tag {13}
$$
Note that, to plug in the lower bound of  $\widetilde{\mathbf{w}}_T$ , we need to ensure the RHS of (13) is positive. Notice that when  $\frac{\gamma^2}{2}\sum_{t=1}^{T}\alpha_t \geq 2\left(\mathrm{Reg}_T^{\mathbf{P}} + \mathrm{Reg}_T^{\mathbf{W}}\right)$ ,
$$
\| \widetilde {\mathbf {w}} _ {T} \| \geq \frac {\gamma^ {2}}{2} \sum_ {t = 1} ^ {T} \alpha_ {t} - \left(\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}\right) \geq \frac {\gamma^ {2}}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} + \left[ \frac {\gamma^ {2}}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} - \left(\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}\right) \right] \geq \frac {\gamma^ {2}}{4} \sum_ {t = 1} ^ {T} \alpha_ {t}.
$$
Combining (10), (11), and (13), we have
$$
\frac {\min  _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \|} \geq \gamma - \frac {4 \left(\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}\right)}{\gamma^ {2} \sum_ {t = 1} ^ {T} \alpha_ {t}} = \frac {4 C _ {T}}{\gamma^ {2}}. \tag {14}
$$
Note that to apply (13), we need  $\frac{\gamma^2}{2}\sum_{t=1}^{T}\alpha_t - 2\left(\mathrm{Reg}_T^{\mathbf{p}} + \mathrm{Reg}_T^{\mathbf{w}}\right) \geq 0$ .
Finally, we focus on the distance between  $\frac{\bar{\mathbf{w}}_T}{\|\bar{\mathbf{w}}_T\|}$  and  $\mathbf{w}_{\parallel \cdot \parallel}^*$  for the case where  $\Phi (\mathbf{w})$  is strongly convex wrt  $\| \cdot \|$ . This part of the proof is motivated by Theorem 4 of Ramdas & Pena (2016), who show that a variant of the perceptron algorithm can converge to the  $\ell_2$ -maximum margin classifier in an  $O(1 / \sqrt{t})$  convergence rate. We have
$$
\begin{array}{l} \left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \|} - \mathbf {w} _ {\| \cdot \|} ^ {*} \right\| = \left\| \frac {\overline {{\mathbf {w}}} _ {T}}{\| \overline {{\mathbf {w}}} _ {T} \|} - \mathbf {w} _ {\| \cdot \|} ^ {*} \right\| = \frac {\| \overline {{\mathbf {w}}} _ {T} - \| \overline {{\mathbf {w}}} _ {T} \| \mathbf {w} _ {\| \cdot \|} ^ {*} \|}{\| \overline {{\mathbf {w}}} _ {T} \|} \\ = \frac {\left\| \overline {{\mathbf {w}}} _ {T} - \gamma \mathbf {w} _ {\parallel \cdot \parallel} ^ {*} + \gamma \mathbf {w} _ {\parallel \cdot \parallel} ^ {*} - \left\| \overline {{\mathbf {w}}} _ {T} \right\| \mathbf {w} _ {\parallel \cdot \parallel} ^ {*} \right\|}{\left\| \mathbf {w} \right\|} \\ \leq \frac {\left\| \overline {{\mathbf {w}}} _ {T} - \gamma \mathbf {w} _ {\parallel \cdot \parallel} ^ {*} \right\| + | \gamma - \| \overline {{\mathbf {w}}} _ {T} \| |}{\left\| \overline {{\mathbf {w}}} _ {T} \right\|} \tag {15} \\ = \frac {\| \overline {{\mathbf {w}}} _ {T} - \gamma \mathbf {w} _ {\| \cdot \|} ^ {*} \| + | \| \gamma \mathbf {w} _ {\| \cdot \|} ^ {*} \| - \| \overline {{\mathbf {w}}} _ {T} \| |}{\| \overline {{\mathbf {w}}} _ {T} \|} \\ \leq \frac {2 \| \overline {{\mathbf {w}}} _ {T} - \gamma \mathbf {w} _ {\| \cdot \|} ^ {*} \|}{\| \overline {{\mathbf {w}}} _ {T} \|} \\ \end{array}
$$
where the first inequality is based on the Minkowski inequality and the fact that  $\| \mathbf{w}_{\parallel \cdot \parallel}^{*}\| = 1$ . Next, note that  $m(\mathbf{w})$  is  $\lambda$ -strongly concave with respect to  $\| \cdot \|$ , and  $\gamma \mathbf{w}_{\parallel \cdot \parallel}^{*}$  maximize  $m(\mathbf{w})$ . This is because it is easy to see that the optimal solution of  $m(\mathbf{w})$  always lies in the direction of  $\mathbf{w}_{\parallel \cdot \parallel}^{*}$ , and we only need to decide the norm. Let  $c > 0$  be some constant, we have  $m\left(c\mathbf{w}_{\parallel \cdot \parallel}^{*}\right) = c\gamma - \frac{1}{2} c^2$ . The function is maximized when  $c = \gamma$ , which implies that the optimal solution is  $\gamma \mathbf{w}_{\parallel \cdot \parallel}^{*}$ . Combining these facts with Lemma 1, we have
$$
\frac {\lambda}{2} \left\| \overline {{\mathbf {w}}} _ {T} - \gamma \mathbf {w} _ {\parallel \cdot \parallel} ^ {*} \right\| ^ {2} \leq m \left(\gamma \mathbf {w} _ {\parallel \cdot \parallel} ^ {*}\right) - m (\overline {{\mathbf {w}}} _ {T}) \leq \frac {\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}}{\sum_ {t = 1} ^ {T} \alpha_ {t}}. \tag {16}
$$
Finally, combining the lower bound of  $\overline{\mathbf{w}}_T$  proved in (11), we have when  $\frac{\gamma^2}{2}\sum_{t=1}^{T}\alpha_t - 2\left(\mathrm{Reg}_T^{\mathbf{p}} + \mathrm{Reg}_T^{\mathbf{w}}\right) \geq 0$ ,
$$
\| \overline {{\mathbf {w}}} _ {T} \| = \frac {1}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \| \widetilde {\mathbf {w}} _ {T} \| \geq \frac {1}{\sum_ {t = 1} ^ {T} \alpha_ {t}} \left[ \frac {\gamma^ {2}}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \right] = \frac {\gamma^ {2}}{4},
$$
so, combining the equation above with (15) and (16), we get
$$
\left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \|} - \mathbf {w} _ {\| \cdot \|} ^ {*} \right\| \leq \frac {8 \sqrt {2 \left(\operatorname {R e g} _ {T} ^ {\mathbf {p}} + \operatorname {R e g} _ {T} ^ {\mathbf {w}}\right)}}{\gamma^ {2} \sqrt {\lambda \sum_ {t = 1} ^ {T} \alpha_ {t}}}. \tag {17}
$$
C Omitted Proof in Section 4.1
In this section, we provide the omitted proof of Section 4.1. Here, we present a more general algorithm framework (given in Algorithm 5) which allows different step sizes. In the following, we first state a general theorem for this algorithm, and Theorems 2 and 3 is then given as Corollaries 2 and 3.
Algorithm 5 Mirror Descent (General Version)
1: for  $t = 1,\dots ,T$  do
2:  $\nabla \Phi (\mathbf{v}_t) = \nabla \Phi (\mathbf{v}_{t - 1}) - \eta_t\nabla L(\mathbf{v}_{t - 1})$
3: end for
4: Output:  $\widetilde{\mathbf{v}}_T = \sum_{t=1}^T \frac{\alpha_t}{\sum_{i=1}^t \alpha_i} \mathbf{v}_t$
w-player:  $\mathbf{w}_t = \underset {\mathbf{w}\in \mathbb{R}^d}{\mathrm{argmin}}\sum_{j = 1}^t\alpha_jh_j(\mathbf{w})$
Theorem 7. Suppose Assumption 1 holds wrt  $\| \cdot \|_q$ -norm for  $q \in (1,2]$ . For the left box of Algorithm 5, let  $\eta_t = \frac{\alpha_t}{L(\mathbf{v}_{t-1})}$ . For the right box, let  $\beta_t$  be  $\frac{\alpha_t}{\sum_{i=1}^{t-1} \alpha_i}$  for  $t > 1$ ,  $\beta_1 = \alpha_1$ . Then the methods in the two boxes of Algorithm 5 are identical, in the sense that  $\widetilde{\mathbf{v}}_T = \widetilde{\mathbf{w}}_T$ , and  $\mathbf{v}_t = \mathbf{w}_t \cdot \sum_{i=1}^t \alpha_i$ . Moreover, the regret upper bound
$$
\operatorname {R e g} _ {T} ^ {\mathbf {p}} = 2 \sum_ {t = 2} ^ {T} \frac {\alpha_ {t} ^ {2}}{\sum_ {j = 1} ^ {t - 1} \alpha_ {j} (q - 1)} + \sum_ {t = 2} ^ {T} \frac {\sum_ {j = 1} ^ {t - 1} \alpha_ {j} (q - 1)}{. 2} \| \mathbf {w} _ {t} - \mathbf {w} _ {t - 1} \| _ {q} ^ {2} + 2 \log n \sum_ {t = 1} ^ {T} \beta_ {t} + \alpha_ {1},
$$
$$
\operatorname {R e g} _ {T} ^ {\mathbf {w}} = - \sum_ {t = 2} ^ {T} \left(\frac {(q - 1) \sum_ {s = 1} ^ {t - 1} \alpha_ {s}}{2}\right) \| \mathbf {w} _ {t} - \mathbf {w} _ {t - 1} \| _ {q} ^ {2}.
$$
Thus,  $\widetilde{\mathbf{v}}_T$  achieves a positive margin (no smaller than  $\gamma^2 /4$ ) for sufficiently large  $T$  such that
$$
\frac {\gamma^ {2}}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \geq \left(2 \sum_ {t = 2} ^ {T} \frac {\alpha_ {t} ^ {2}}{\sum_ {j = 1} ^ {t - 1} \alpha_ {j} (q - 1)} + 2 \log n \sum_ {t = 2} ^ {T} \frac {\alpha_ {t}}{\sum_ {i = 1} ^ {t - 1} \alpha_ {i}}\right) + \alpha_ {1} (1 + 2 \log n). \tag {18}
$$
After (18) is satisfied, the margin of  $\widetilde{\mathbf{v}}_T$  is lower bounded by
$$
\frac {\underset {\mathbf {p} \in \Delta^ {n}} {\min} \mathbf {p} ^ {\top} A \widetilde {\mathbf {v}} _ {T}}{\| \widetilde {\mathbf {v}} _ {T} \| _ {q}} \geq \gamma - \frac {4 \left[ \left(2 \sum_ {t = 2} ^ {T} \frac {\alpha_ {t} ^ {2}}{\sum_ {j = 1} ^ {t - 1} \alpha_ {j} (q - 1)} + 2 \log n \sum_ {t = 2} ^ {T} \frac {\alpha_ {t}}{\sum_ {i = 1} ^ {t - 1} \alpha_ {i}}\right) + \alpha_ {1} (1 + 2 \log n) \right]}{\gamma^ {2} \sum_ {t = 1} ^ {T} \alpha_ {t}},
$$
and the directional error is
$$
\begin{array}{l} \left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \| _ {q}} - \mathbf {w} _ {\| \cdot \|} ^ {*} \right\| _ {q} \\ \leq \frac {8}{\gamma^ {2} \sqrt {q - 1}} \sqrt {\frac {2 \left[ \left(2 \sum_ {t = 2} ^ {T} \frac {\alpha_ {t} ^ {2}}{\sum_ {j = 1} ^ {t - 1} \alpha_ {j} (q - 1)} + 2 \log n \sum_ {t = 2} ^ {T} \frac {\alpha_ {t}}{\sum_ {i = 1} ^ {t - 1} \alpha_ {i}}\right) + \alpha_ {1} (1 + 2 \log n) \right]}{(q - 1) \sum_ {t = 1} ^ {T} \alpha_ {t}}}. \\ \end{array}
$$
Next, we show that different step sizes (decided by  $\alpha_{t}$ ) lead to different implicit bias convergence rates. Firstly, we consider setting the step size as  $\eta_t = \frac{1}{\sqrt{t}L(\mathbf{v}_{t - 1})}$ , which leads to a slow  $\widetilde{O}\left(\frac{1}{\gamma^2\sqrt{T}(q - 1)}\right)$  bound.
Corollary 1. Let  $\alpha_{t} = \frac{1}{\sqrt{t}}$ . Then the margin is lower bounded by
$$
\frac {\min  _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \widetilde {\mathbf {v}} _ {T}}{\| \widetilde {\mathbf {v}} _ {T} \| _ {q}} \geq \gamma - \frac {4 \left(\left(\frac {2}{q - 1} + \log n\right) \log T + 4 \log n + \frac {4}{q - 1}\right)}{\gamma^ {2} \sqrt {T}} = \gamma - \widetilde {O} \left(\frac {1}{\gamma^ {2} \sqrt {T} (q - 1)}\right),
$$
$$
\left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \| _ {q}} - \mathbf {w} _ {\| \cdot \|} ^ {*} \right\| _ {q} \leq \frac {8 \sqrt {2}}{\gamma^ {2} \sqrt {q - 1}} \sqrt {\frac {\left(\frac {2}{q - 1} + \log n\right) \log T + 4 \log n + \frac {4}{q - 1}}{\sqrt {T}}} = \widetilde {O} \left(\frac {1}{(q - 1) \gamma^ {2} T ^ {1 / 4}}\right),
$$
when  $T$  is sufficiently large such that  $\sqrt{T} \geq \frac{4\left(\left(\frac{2}{q-1} + \log n\right) \log T + 4 \log n + \frac{4}{q-1}\right)}{\gamma^2} = \widetilde{\Omega}\left(\frac{1}{(q-1)\gamma^2}\right)$ .
Next, we show that a faster rate can be obtained with a constant  $\alpha_{t}$ , which implies Theorem 2.
Corollary 2 (Theorem 2). Let  $\alpha_{t} = 1$ . Then the margin is lower bounded by
$$
\frac {\min  _ {\mathbf {p} \in \Delta^ {n}} \mathbf {p} ^ {\top} \mathbf {A} \widetilde {\mathbf {v}} _ {T}}{\| \widetilde {\mathbf {v}} _ {T} \| _ {q}} \geq \gamma - \frac {4 \left(\frac {2}{q - 1} + 2 \log n\right) (\log T + 2)}{\gamma^ {2} T} = \gamma - \widetilde {O} \left(\frac {1}{(q - 1) \gamma^ {2} T}\right),
$$
$$
\left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \| _ {q}} - \mathbf {w} _ {\| \cdot \|} ^ {*} \right\| _ {q} \leq \frac {8 \sqrt {2}}{\gamma^ {2} \sqrt {(q - 1)}} \sqrt {\frac {\left(\frac {2}{q - 1} + 2 \log n\right) (\log T + 2)}{T}} = \widetilde {O} \left(\frac {1}{\gamma^ {2} (q - 1) \sqrt {T}}\right).
$$
when  $T$  is sufficiently large such that  $T \geq \frac{4\left(\frac{2}{q - 1} + 2\log n\right)(\log T + 2)}{\gamma^2}$ .