SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guanghui Wang $^{1}$ , Zihao Hu $^{1}$ , Vidya Muthukumar $^{2,3}$ , Jacob Abernethy $^{1,4}$
<sup>1</sup>College of Computing, Georgia Institute of Technology
$^{2}$ School of Electrical and Computer Engineering, Georgia Institute of Technology
$^{3}$ School of Industrial and Systems Engineering, Georgia Institute of Technology
<sup>4</sup>Google Research, Atalanta
{gwang369,zihaohu,vmuthukumar8}@gatech.edu, abernethyj@google.com
First-order optimization methods tend to inherently favor certain solutions over others when minimizing a given training objective with multiple local optima. This phenomenon, known as implicit bias, plays a critical role in understanding the generalization capabilities of optimization algorithms. Recent research has revealed that gradient-descent-based methods exhibit an implicit bias for the  $\ell_2$ -maximal margin classifier in the context of separable binary classification. In contrast, generic optimization methods, such as mirror descent and steepest descent, have been shown to converge to maximal margin classifiers defined by alternative geometries. However, while gradient-descent-based algorithms demonstrate fast implicit bias rates, the implicit bias rates of generic optimization methods have been relatively slow. To address this limitation, in this paper, we present a series of state-of-the-art implicit bias rates for mirror descent and steepest descent algorithms. Our primary technique involves transforming a generic optimization algorithm into an online learning dynamic that solves a regularized bilinear game, providing a unified framework for analyzing the implicit bias of various optimization methods. The accelerated rates are derived leveraging the regret bounds of online learning algorithms within this game framework.
The training objective in the optimization of modern over-parametrized ML models typically presents various local optima with low training error. Despite this, empirical studies have demonstrated that first-order optimization methods generally converge to the solution with strong generalization, even without any explicit regularization (Neyshabur et al., 2014; Zhang et al., 2021). This observation has spurred interest in the study of the implicit bias of the algorithm. In other words, among all potential parameter choices with low training error, which ones are inherently favored by optimization methods?
For the classical linear classification problem with separable data, the pioneering works (Soudry et al., 2018; Ji & Telgarsky, 2018) reveal that minimizing the (unregularized) empirical risk with exponential loss by the classical gradient descent (GD) automatically maximizes the  $\| \cdot \| _2$ -margin, meaning its margin converges to that of the best classifier within an  $\ell_2$ -norm ball (referred to as the  $\| \cdot \| _2$ -maximal margin classifier). This finding implies that GD exhibits an implicit bias towards the  $\| \cdot \| _2$ -maximal margin classifier, which helps account for its favorable generalization performance. However, these works show that GD only maximizes the  $\| \cdot \| _2$ -margin at a slow  $O\left(\frac{\log n}{\log T}\right)$  rate, where  $T$  is the time horizon, and  $n$  is the cardinality of the data set. Since then, several faster margin
maximization rates have been reported. Nacson et al. (2019) revealed that for the exponential loss, GD with an aggressive step size attains an  $O\left(\frac{\log n + \log T}{\sqrt{T}}\right)\| \cdot \| _2$ -margin maximization rate. This result was later improved to  $O\left(\frac{\log n}{T}\right)$  by Ji & Telgarsky (2021), via an elegant primal-dual analysis. Recent work (Ji et al., 2021; Wang et al., 2022b) further proved that momentum-based GD and Nesterov-accelerated GD can maximize the  $\| \cdot \| _2$ -margin at an  $O\left(\frac{\log n}{T^2}\right)$  rate.
As the implicit bias of gradient-descent-based methods becomes better understood, it is natural to explore similar characterizations for other optimization methods. Note that, since gradient-descent-based methods are biased towards the  $\| \cdot \| _2$ -maximal margin classifier, they might generalize poorly when the data does not adhere to the  $\ell_2$ -geometry, as shown in Gentile (2000); Chen et al. (2001). This limitation suggests that it is essential to study the implicit bias of alternative optimization methods that could potentially be biased in different directions. Two such methods include steepest descent with respect to different norms and mirror descent with different potentials. For instance, Gunasekar et al. (2018a) demonstrate that for the exponential loss, the steepest descent algorithm with respect to a general norm  $\| \cdot \|$  asymptotically converges to the corresponding  $\| \cdot \|$ -maximal margin classifier. This result implies that the steepest descent algorithm can adapt to different data geometries by changing the norm used in the algorithm. On the other hand, Sun et al. (2022) show that the mirror descent algorithm with the potential  $\| \cdot \| _q^q$  for  $q > 1$  can maximize the  $\| \cdot \| _q$ -margin at a rate of  $O\left(\frac{1}{(\log T)^{q - 1}}\right)$ .
While the asymptotic directional convergence of these generic optimization methods is well- understood, a natural question remains: can generic optimization methods (e.g., mirror descent and steepest descent) achieve faster margin maximization rates? Several papers have contributed partial answers to this question. Li et al. (2021) show that mirror descent with an aggressive step size maximizes the margin in an  $O\left(\frac{\log n}{T^{1/4}}\right)$  rate. However, their analysis assumes the potential function is both strongly-convex and smooth with respect to some norm, and thus is mainly limited to the  $\ell_2$ -geometry. For steepest descent with respect to a general norm  $\|\cdot\|$ , Nacson et al. (2019) prove that an  $O\left(\frac{\log n + \log T}{\sqrt{T}}\right)\| \cdot \|$ -margin maximization rate can be achieved with an appropriately chosen step size, but it is unclear whether it can be further improved. In this paper, we provide the fastest known rates for margin maximization for generic optimization methods, through the following contributions:
- First, we study a weighted-average version of mirror descent with the squared  $\ell_q$ -norm  $\frac{1}{2} \| \cdot \|_q^2$  as the potential for  $q \in (1, 2]$ . This potential function is strongly convex but not smooth. We show that, with an appropriately chosen step size, the algorithm achieves a faster  $\| \cdot \|_q$ -margin maximization rate on the order of  $O\left(\frac{\log n \log T}{(q - 1)T}\right)$ . We also further improve the rate to  $O\left(\frac{1}{T(q - 1)} + \frac{\log n \log T}{T^2}\right)$  with a more aggressive step size. When  $q = 2$ , the algorithm reduces to average GD, and our rate  $O\left(\frac{1}{T} + \frac{\log n \log T}{T^2}\right)$  is a log  $n$ -factor tighter than the  $O\left(\frac{\log n}{T}\right)$  rate of the last-iterate GD (Ji & Telgarsky, 2021).
- Next, for the steepest descent algorithm with respect to the  $\ell_q$ -norm for  $q \in (1,2]$ , we show the margin maximization rate can be improved from  $O\left(\frac{\log n + \log T}{\sqrt{T}}\right)$  to  $O\left(\frac{\log n}{T(q - 1)}\right)$ .
- Finally, we demonstrate that a even faster  $O\left(\frac{\log n}{T^2(q - 1)}\right)\| \cdot \|_q$ -margin maximization rate can be achieved in two ways: a) mirror descent with Nesterov acceleration, or b) steepest descent with extra gradient and momentum.
The essential premise for our approach is that minimizing empirical risk (ERM) with generic optimization methods can be equivalently viewed as solving a regularized bilinear game with online learning dynamics. Within this framework, we design new pairs of online learning methods whose outputs (and, by extension, the outputs of the corresponding generic optimization methods) automatically maximize the margin. The convergence rates are determined by the time-averaged regret bounds of these online learning algorithms when played against each other, which turn out to be much faster than the worst-case  $O(1 / \sqrt{T})$  rate.
Wang et al. (2022b) were the first to draw parallels between Nesterov-accelerated  $GD$  for ERM and solving the bilinear game through an online dynamic. However, it was still open that whether this kind of analysis suits other GD-based methods. Moreover, the non-linearity of the mirror map in generic optimization methods makes analysis particularly challenging. In this paper, we reveal that the game framework can in fact encompass implicit bias analysis for a range of generic optimization methods, and offer a more streamlined and unified analysis. Within this game framework, we derive several other results beyond those mentioned above:
- By selecting suitable online learning algorithms, we obtain a momentum-based data-dependent MD algorithm with an  $O\left(\frac{\mathcal{V}_T}{T^2(q - 1)} + \frac{\log n\log T}{T^2}\right) \parallel \cdot \parallel_q$ -margin maximization rate, where  $\mathcal{V}_T = \sum_{t=2}^{T} \|\mathbf{p}_t - \mathbf{p}_{t-1}\|_1^2$  is the path-length of a series of distributions on the training data  $\mathbf{p}_t$ . In the worst case, this reduces to the margin maximization rate of MD, but this can be much tighter if  $\mathcal{V}_T$  is sublinear in  $T$ .
- Apart from margin maximization rates, we also demonstrate the corresponding directional error, i.e., the bound on the  $\ell_q$ -distance between the maximal margin classifier and the normalized output of the generic methods, which are also controlled by the regret bounds of two-players against each other. This kind of convergence rates are new for most of the generic methods. In general, we prove that the directional errors are typically a square-root factor worse than the margin maximization rates.
- For our steepest descent, by setting the norm to the general norm  $\|\cdot\|$  and the  $\ell_2$ -norm respectively, we can recover the algorithms and theoretical guarantees in Nacson et al. (2019); Ji & Telgarsky (2021) under the game framework. This implies that these algorithms can also be viewed as solving a regularized bilinear game using online learning algorithms, offering a deeper understanding of the role of implicit bias in optimization methods.
Additional related work The strategy of solving a zero-sum game using online learning algorithms playing against each other has been extensively studied, primarily through the lens of independent learning agents (e.g., Rakhlin & Sridharan, 2013; Daskalakis et al., 2018; Wang & Abernethy, 2018; Daskalakis & Panageas, 2019; Zhang et al., 2022). In contrast, our central motivation and challenge lies in identifying the exact equivalent forms of generic optimization algorithms under the regularized bilinear game dynamic. Our framework is also motivated by the line of research that employs the Fenchel-game to elucidate commonly used convex optimization methods (Abernethy et al., 2018; Wang & Abernethy, 2018; Wang et al., 2021b). However, our framework diverges significantly from these approaches. These works focus on the convergence of the optimization problem itself, while our framework emphasizes that the choice of optimization algorithm, which solely targets the minimization of empirical risk, has a significant impact on maximizing the margin, which we might view as an "algorithmic externality." Max-margin guarantees can not arise from convergence of the ERM objective alone, as there are typically multiple global minima in ERM minimization. Our analysis also considers an entirely different min-max problem than that of the Fenchel game (Wang et al., 2021b). Consequently, the correspondences we establish between optimization algorithms and online dynamics also differ. Finally, we note that previous work has also analyzed the implicit bias through direct primal optimization analysis (e.g., Nacson et al., 2019; Sun et al., 2022) or using a dual perspective (e.g., Ji & Telgarsky, 2021; Ji et al., 2021). For the former analysis, it is unclear whether and how faster rates can be obtained. For the latter, it remains an open question how to extend the framework beyond  $\ell_2$ -geometry, which in some sense was the motivation for the present work. For more related work, we refer the reader to Appendix A.
In this section, we present our basic setting along with some standard assumptions and definitions.
Notation We use lower case bold face letters  $\mathbf{x},\mathbf{y}$  to denote vectors, lower case letters  $a,b$  to denote scalars, and upper case bold face letters  $\mathbf{A},\mathbf{B}$  to denote matrices. For a vector  $\mathbf{x}\in \mathbb{R}^d$ , we use  $x_{i}$  to denote the  $i$ -th component of  $\mathbf{x}$ . For a matrix  $\mathbf{A}\in \mathbb{R}^{n\times d}$ , let  $\mathbf{A}_{(i,:)}$  be its  $i$ -th row,  $\mathbf{A}_{(:,j)}$  the  $j$ -th column, and  $\mathbf{A}_{(i,j)}$  the  $i$ -th element of the  $j$ -th column.  $\forall \mathbf{x}\in \mathbb{R}^d$ , we use  $\| \cdot \|$  to denote a general norm in  $\mathbb{R}^d$ ,  $\| \cdot \| _*$  its dual norm,  $\| \mathbf{x}\| _q$  the  $q$ -norm of  $\mathbf{x}$ , defined as  $\| \mathbf{x}\| _q = (\sum_{i = 1}^d |x_i|^q)^{1 / q}$  and  $q\in (1,2]$ . We use  $\| \cdot \| _p$  to denote the dual norm of  $q$ -norm, where  $p\in [2,\infty)$ ,  $\frac{1}{p} +\frac{1}{q} = 1$ .
Figure 1: Illustration of the game framework for implicit bias analysis. In Section 3, we show that solving a regularized bilinear game with online learning algorithms (top box) can directly maximize the margin, and the convergence rate is on the same order of the averaged regret  $C_T$  (right box); In Sections 4, we prove that minimizing the empirical risk with a series of generic optimization methods (left box) is equivalent to using online learning algorithms to solve the regularized bilinear game. Thus, the implicit bias rates can be directly obtained by plugging in the regret bounds.
We denote  $\mathcal{B}_{\| \cdot \|}$  the  $\| \cdot \|$ -ball, defined as  $\mathcal{B}_{\| \cdot \|} = \{\mathbf{x} \in \mathbb{R}^d \mid \| \mathbf{x} \| \leq 1\}$ .  $\forall \mathbf{x}, \mathbf{x}' \in \mathbb{R}^d$ , we define the Bregman divergence between  $\mathbf{x}$  and  $\mathbf{x}'$  with respect to a strictly convex potential function  $\Phi(\mathbf{x})$  as  $D_{\Phi}(\mathbf{x}, \mathbf{x}') = \Phi(\mathbf{x}) - \Phi(\mathbf{x}') - \nabla \Phi(\mathbf{x}')^\top (\mathbf{x} - \mathbf{x}')$ . For a positive integer  $n$ , we denote  $\{1, \dots, n\}$  as  $[n]$ , and the  $(n-1)$ -dimensional simplex as  $\Delta^n$ . Let  $E: \Delta^n \mapsto \mathbb{R}$  be the negative entropy function, defined as  $E(\mathbf{p}) = \sum_{i=1}^{n} p_i \log p_i$ ,  $\forall \mathbf{p} \in \Delta^n$ .
Basic setting Consider a set of  $n$  data points  $\mathcal{S} = \{(\mathbf{x}^{(i)},y^{(i)})\}$ , where  $\mathbf{x}^{(i)}\in \mathbb{R}^d$  is the feature vector for the  $i$ -th example, and  $y^{(i)}\in \{-1, + 1\}$  the corresponding binary label. We are interested the optimization trajectory of first-order methods for minimizing the following unbounded and.   
unregularized empirical risk:
$$
\min  _ {\mathbf {w} \in \mathbb {R} ^ {d}} L (\mathbf {w}) = \frac {1}{n} \sum_ {i = 1} ^ {n} r \left(\mathbf {w} ^ {\top} \mathbf {x} ^ {(i)}; y ^ {(i)}\right), \tag {1}
$$
where  $\mathbf{w} \in \mathbb{R}^d$  is a linear classifier,  $r: \mathbb{R} \times \{\pm 1\} \mapsto \mathbb{R}$  is the loss function. Following previous work, we focus on the exponential loss, given by  $r(\mathbf{w}^\top \mathbf{x}; y) = \exp(-y\mathbf{x}^\top \mathbf{w})$ . We introduce the following standard assumption and definitions.
Definition 1 ( $\| \cdot \|$ -margin). For a linear classifier  $\mathbf{w} \in \mathbb{R}^d$  and a norm  $\| \cdot \|$ , we define its  $\| \cdot \|$ -margin as
$$
\widetilde {\gamma} (\mathbf {w}) = \frac {\underset {i \in [ n ]} {\min} y ^ {(i)} \mathbf {w} ^ {\top} \mathbf {x} ^ {(i)}}{\| \mathbf {w} \|} = \frac {\underset {\mathbf {p} \in \Delta^ {n}} {\min} \mathbf {p} ^ {\top} \mathbf {A} \mathbf {w}}{\| \mathbf {w} \|},
$$
where  $\mathbf{A} = [\ldots ;y^{(i)}\mathbf{x}^{(i)^{\top}};\ldots ]\in \mathbb{R}^{n\times d}$  is the matrix that contains all data.
Assumption 1. Assume  $\mathcal{S}$  is linearly separable and bounded with respect to some norm  $\| \cdot \|$ . More specifically, we assume  $\exists \mathbf{w}_{\parallel \cdot \parallel}^{*} \in \mathcal{B}_{\parallel \cdot \parallel}$ , s.t.,  $\mathbf{w}_{\parallel \cdot \parallel}^{*} = \operatorname{argmax}_{\| \mathbf{w} \| \leq 1} \min_{i \in [n]} y^{(i)} \mathbf{x}^{(i)^{\top}} \mathbf{w}$ , whose margin  $\widetilde{\gamma}(\mathbf{w}_{\parallel \cdot \parallel}^{*}) = \gamma > 0$ . We refer to  $\mathbf{w}_{\parallel \cdot \parallel}^{*}$  as the  $\| \cdot \|$ -maximal margin classifier. Note that, for any  $\mathbf{w} \in \mathbb{R}^d$ , if  $\widetilde{\gamma}(\mathbf{w}) = \gamma$ ,  $\mathbf{w}$  and  $\mathbf{w}_{\parallel \cdot \parallel}^{*}$  are at the same direction. Finally, we also assume  $\mathcal{S}$  is bounded wrt some dual norm  $\| \cdot \|_*$ , i.e.,  $\forall i \in [n]$ ,  $\| \mathbf{x}^{(i)} \|_* \leq 1$ .
Definition 2 ( $\|\cdot\|$ -Margin maximization rate and  $\|\cdot\|$ -directional error). Suppose Assumption 1 is satisfied. We consider a sequence of solutions  $\mathbf{w}_1, \ldots, \mathbf{w}_t, \ldots$ , and state that  $\mathbf{w}_t$  converges to  $\mathbf{w}_{\|\cdot\|}^*$  if either  $\lim_{t \to \infty} \widetilde{\gamma}(\mathbf{w}_t) \to \gamma$ , or  $\lim_{t \to \infty} \|\frac{\mathbf{w}_t}{\|\mathbf{w}_t\|} - \mathbf{w}_{\|\cdot\|}^*\| \to 0$ . We define the upper bound on  $|\gamma - \widetilde{\gamma}(\mathbf{w}_t)|$  the  $\|\cdot\|$ -margin maximization rate, and  $\|\frac{\mathbf{w}_t}{\|\mathbf{w}_t\|} - \mathbf{w}_{\|\cdot\|}^*\|$  the  $\|\cdot\|$ -directional error.
3 A Game Framework for Maximizing the Margin
In this section, we present a general game framework and demonstrate that solving this game with online learning algorithms can directly maximize the margin and minimize the directional error.
Protocol 1 No-regret dynamics with weighted OCO for solving  $g(\mathbf{p},\mathbf{w})$
1: Initialization:  $\mathsf{OL}^{\mathbf{w}}$ ,  $\mathsf{OL}^{\mathbf{p}}$ . // The online algorithms for choosing w and p.  
2: for  $t = 1, \dots, T$  do  
3:  $\mathbf{w}_t \gets \mathsf{OL}^{\mathbf{w}}$ ;  
4:  $\mathsf{OL}^{\mathbf{p}} \gets \alpha_t, \ell_t(\cdot)$ ; // Define  $\ell_t(\cdot) = g(\mathbf{w}_t, \cdot)$   
5:  $\mathbf{p}_t \gets \mathsf{OL}^{\mathbf{p}}$ ;  
6:  $\mathsf{OL}^{\mathbf{w}} \gets \alpha_t, h_t(\cdot)$ ; // Define  $h_t(\cdot) = -g(\cdot, \mathbf{p}_t)$   
7: end for  
8: Output:  $\widetilde{\mathbf{w}}_T = \sum_{t=1}^{T} \alpha_t \mathbf{w}_t$ .
Then, in Section 4, we show that many generic optimization methods can be considered as solving this game with different online dynamics. As a result, the margin maximization rate (and also the directional error) of these optimization methods are exactly characterized by the regret bounds of the corresponding online learning algorithms. We illustrate this procedure in Figure 1. The game objective is defined as follows:
$$
\max  _ {\mathbf {w} \in \mathbb {R} ^ {d}} \min  _ {\mathbf {p} \in \Delta^ {n}} g (\mathbf {p}, \mathbf {w}) = \mathbf {p} ^ {\top} \mathbf {A} \mathbf {w} - \Phi (\mathbf {w}), \tag {2}
$$
where  $\Phi (\mathbf{w}) = \frac{1}{2}\| \mathbf{w}\|^2$  is a regularizer and  $\| \cdot \|$  denotes some general norm in  $\mathbb{R}^d$ . Following previous work (Wang et al., 2021b, 2022b), we apply a weighted no-regret dynamic protocol (summarized in Protocol 1) to solve the game. We first give a brief introduction of Protocol 1, and then present the theorem about the margin of its output.
Description of Protocol 1 In Protocol 1, the players of the zero-sum game try to find the equilibrium by applying online learning algorithms. In each round  $t$ , the p-player first picks a decision  $\mathbf{p}_t$ , and passes a weighted loss function to the w-player, defined as
$$
\alpha_ {t} h _ {t} (\mathbf {w}) = - \alpha_ {t} (\mathbf {p} _ {t} ^ {\top} \mathbf {A} \mathbf {w} - \Phi (\mathbf {w})) = - \alpha_ {t} g (\mathbf {p} _ {t}, \mathbf {w}).
$$
Then, the  $\mathbf{w}$ -player observes the loss, picks a decision  $\mathbf{w}_t$ , and passes a weighted loss function
$$
\alpha_ {t} \ell_ {t} (\mathbf {p}) = \alpha_ {t} \left(\mathbf {p} ^ {\top} \mathbf {A} \mathbf {w} _ {t} - \Phi (\mathbf {w} _ {t})\right) = \alpha_ {t} g (\mathbf {p}, \mathbf {w} _ {t}),
$$
to the  $\mathbf{p}$ -player. Note that the order of the two players can also be reversed. After  $T$  iterations, the algorithm outputs the weighted sum of the  $\mathbf{w}$ -player's decisions:  $\widetilde{\mathbf{w}}_T = \sum_{t=1}^{T} \alpha_t \mathbf{w}_t$ . Under this framework, we define the weighted regret upper bound of both players respectively as
$$
\sum_ {t = 1} ^ {T} \alpha_ {t} \ell_ {t} (\mathbf {p} _ {t}) - \min  _ {\mathbf {p} \in \Delta^ {n}} \sum_ {t = 1} ^ {T} \alpha_ {t} \ell_ {t} (\mathbf {p}) \leq \operatorname {R e g} _ {T} ^ {\mathbf {p}}, \text {a n d} \sum_ {t = 1} ^ {T} \alpha_ {t} h _ {t} (\mathbf {w} _ {t}) - \min  _ {\mathbf {w} \in \mathbb {R} ^ {d}} \sum_ {t = 1} ^ {T} \alpha_ {t} h _ {t} (\mathbf {w}) \leq \operatorname {R e g} _ {T} ^ {\mathbf {w}}.
$$
Denote the upper bound on the average weighted regret by  $C_T = (\mathrm{Reg}_T^{\mathbf{P}} + \mathrm{Reg}_T^{\mathbf{w}}) / \sum_{t=1}^{T} \alpha_t$ . We have the following conclusion on the margin and directional error of  $\widetilde{\mathbf{w}}_T$ . The proof of this theorem can be found in Appendix B.
Theorem 1. Suppose Assumption 1 holds with respect to some general norm  $\| \cdot \|$ . Consider solving the two-player zero-sum game defined in (2) by applying Protocol 1. Then  $\widetilde{\mathbf{w}}_T$  will have a positive margin on round  $T$  if  $C_T \leq \frac{\gamma^2}{4}$ . Moreover, as long as  $C_T \leq \frac{\gamma^2}{4}$ , we have
$$
\frac {\underset {\mathbf {p} \in \Delta^ {n}} {\min } \mathbf {p} ^ {\top} \mathbf {A} \widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \|} \geq \gamma - \frac {4 C _ {T}}{\gamma^ {2}}. \tag {3}
$$
If  $\Phi (\mathbf{w})$  is  $\lambda$ -strongly convex with wrt  $\| \cdot \|$ , we have
$$
\left\| \frac {\widetilde {\mathbf {w}} _ {T}}{\| \widetilde {\mathbf {w}} _ {T} \|} - \mathbf {w} _ {\| \cdot \|} ^ {*} \right\| \leq \frac {8 \sqrt {2}}{\gamma^ {2} \sqrt {\lambda}} \sqrt {C _ {T}}.
$$
Theorem 1 shows that the output of Protocol 1, denoted as  $\widetilde{\mathbf{w}}_T$ , achieves a positive margin when the average regret  $C_T \leq \frac{\gamma^2}{4}$ . In the following sections, we demonstrate that with appropriately chosen
online learning algorithms  $C_T$  always decreases with respect to  $T$ ; in fact  $C_T \to 0$  as  $T \to \infty$ . Therefore, once the condition  $C_T \leq \frac{\gamma^2}{4}$  is met for a particular value  $T_0$ , it will also be met for all  $T \geq T_0$ . Thereafter,  $\widetilde{\mathbf{w}}_T$  continues to increase the  $\| \cdot \|$ -margin and converges to the maximum  $\| \cdot \|$ -margin classifier, and the rate is directly characterized by  $C_T$ . Since  $C_T$  is the average regret of the online learning algorithms, better bounds on  $C_T$  lead to a less stringent condition on large enough  $T$ . Finally, we note that the condition on sufficiently large  $T$  is also (explicitly or implicitly) required in all previous work on the non-asymptotic margin maximization rates of generic methods (Nacson et al., 2019; Li et al., 2021; Sun et al., 2022). We refer to Appendix A for more details.
4 Implicit Bias of Generic Methods