Title: Faster Margin Maximization Rates for Generic Optimization Methods
Venue: NeurIPS 2023
Accepted: yes
Keywords: Implicit bias

margin maximization

zero-sum game

online learning
Abstract: First-order optimization methods tend to inherently favor certain solutions over others when minimizing a given training objective with multiple local optima. This phenomenon, known as \emph{implicit bias}, plays a critical role in understanding the generalization capabilities of optimization algorithms. Recent research has revealed that gradient-descent-based methods exhibit an implicit bias for the $\ell_2$-maximal margin classifier in the context of separable binary classification. In contrast, generic optimization methods, such as mirror descent and steepest descent, have been shown to converge to maximal margin classifiers defined by alternative geometries. However, while gradient-descent-based algorithms demonstrate fast implicit bias rates, the implicit bias rates of generic optimization methods have been relatively slow. To address this limitation, in this paper, we present a series of state-of-the-art implicit bias rates for mirror descent and steepest descent algorithms. Our primary technique involves transforming a generic optimization algorithm into an online learning dynamic that solves a regularized bilinear game, providing a unified framework for analyzing the implicit bias of various optimization methods. The accelerated rates are derived leveraging the regret bounds of online learning algorithms within this game framework.

Faster Margin Maximization Rates for Generic Optimization Methods

Abstract

First-order optimization methods tend to inherently favor certain solutions over others when minimizing a given training objective with multiple local optima. This phenomenon, known as implicit bias , plays a critical role in understanding the generalization capabilities of optimization algorithms. Recent research has revealed that gradient-descent-based methods exhibit an implicit bias for the ℓ 2 -maximal margin classifier in the context of separable binary classification. In contrast, generic optimization methods, such as mirror descent and steepest descent, have been shown to converge to maximal margin classifiers defined by alternative geometries. However, while gradient-descent-based algorithms demonstrate fast implicit bias rates, the implicit bias rates of generic optimization methods have been relatively slow. To address this limitation, in this paper, we present a series of stateof-the-art implicit bias rates for mirror descent and steepest descent algorithms. Our primary technique involves transforming a generic optimization algorithm into an online learning dynamic that solves a regularized bilinear game, providing a unified framework for analyzing the implicit bias of various optimization methods. The accelerated rates are derived leveraging the regret bounds of online learning algorithms within this game framework.

1 Introduction

The training objective in the optimization of modern over-parametrized ML models typically presents various local optima with low training error. Despite this, empirical studies have demonstrated that first-order optimization methods generally converge to the solution with strong generalization, even without any explicit regularization (Neyshabur et al., 2014; Zhang et al., 2021). This observation has spurred interest in the study of the implicit bias of the algorithm. In other words, among all potential parameter choices with low training error, which ones are inherently favored by optimization methods?

For the classical linear classification problem with separable data, the pioneering works (Soudry et al., 2018; Ji & Telgarsky, 2018) reveal that minimizing the (unregularized) empirical risk with exponential loss by the classical gradient descent (GD) automatically maximizes the ∥ · ∥ 2 -margin , meaning its margin converges to that of the best classifier within an ℓ 2 -norm ball (referred to as the ∥ · ∥ 2 -maximal margin classifier). This finding implies that GD exhibits an implicit bias towards the ∥ · ∥ 2 -maximal margin classifier, which helps account for its favorable generalization performance. However, these works show that GD only maximizes the ∥ · ∥ 2 -margin at a slow O ( log n log T ) rate, where T is the time horizon, and n is the cardinality of the data set. Since then, several faster margin

maximization rates have been reported. Nacson et al. (2019) revealed that for the exponential loss, GD with an aggressive step size attains an O ( log n +log T √ T ) ∥ · ∥ 2 -margin maximization rate. This result was later improved to O ( log n T ) by Ji & Telgarsky (2021), via an elegant primal-dual analysis. Recent work (Ji et al., 2021; Wang et al., 2022b) further proved that momentum-based GD and Nesterov-accelerated GD can maximize the ∥ · ∥ 2 -margin at an O ( log n T 2 ) rate.

As the implicit bias of gradient-descent-based methods becomes better understood, it is natural to explore similar characterizations for other optimization methods. Note that, since gradient-descentbased methods are biased towards the ∥ · ∥ 2 -maximal margin classifier, they might generalize poorly when the data does not adhere to the ℓ 2 -geometry, as shown in Gentile (2000); Chen et al. (2001). This limitation suggests that it is essential to study the implicit bias of alternative optimization methods that could potentially be biased in different directions. Two such methods include steepest descent with respect to different norms and mirror descent with different potentials. For instance, Gunasekar et al. (2018a) demonstrate that for the exponential loss, the steepest descent algorithm with respect to a general norm ∥ · ∥ asymptotically converges to the corresponding ∥ · ∥ -maximal margin classifier. This result implies that the steepest descent algorithm can adapt to different data geometries by changing the norm used in the algorithm. On the other hand, Sun et al. (2022) show that the mirror descent algorithm with the potential ∥ · ∥ q q for q > 1 can maximize the ∥ · ∥ q -margin at ( 1 )

While the asymptotic directional convergence of these generic optimization methods is wellunderstood, a natural question remains: can generic optimization methods (e.g., mirror descent and steepest descent) achieve faster margin maximization rates? Several papers have contributed partial answers to this question. Li et al. (2021) show that mirror descent with an aggressive step size maximizes the margin in an O ( log n T 1 / 4 ) rate. However, their analysis assumes the potential function is both strongly-convex and smooth with respect to some norm, and thus is mainly limited to the ℓ 2 -geometry. For steepest descent with respect to a general norm ∥ · ∥ , Nacson et al. (2019) prove that an O ( log n +log T √ T ) ∥ · ∥ -margin maximization rate can be achieved with an appropriately chosen step size, but it is unclear whether it can be further improved. In this paper, we provide the fastest known rates for margin maximization for generic optimization methods, through the following contributions:

First, we study a weighted-average version of mirror descent with the squared ℓ q -norm 1 2 ∥ · ∥ 2 q as the potential for q ∈ (1 , 2] . This potential function is strongly convex but not smooth. We show that, with an appropriately chosen step size, the algorithm achieves a faster ∥ · ∥ q -margin maximization rate on the order of O ( log n log T ( q -1) T ) . We also further improve the rate to O ( 1 T ( q -1) + log n log T T 2 ) with a more aggressive step size. When q = 2 , the algorithm reduces to average GD, and our rate O ( 1 T + log n log T T 2 ) is a log n -factor tighter than the O ( log n T ) rate of the last-iterate GD (Ji & Telgarsky, 2021).

Next, for the steepest descent algorithm with respect to the ℓ q -norm for q ∈ (1 , 2] , we show the margin maximization rate can be improved from O ( log n +log T √ T ) to O ( log n T ( q -1) ) .

Finally, we demonstrate that a even faster O ( log n T 2 ( q -1) ) ∥ · ∥ q -margin maximization rate can be achieved in two ways: a) mirror descent with Nesterov acceleration, or b) steepest descent with extra gradient and momentum.

The essential premise for our approach is that minimizing empirical risk (ERM) with generic optimization methods can be equivalently viewed as solving a regularized bilinear game with online learning dynamics . Within this framework, we design new pairs of online learning methods whose outputs (and, by extension, the outputs of the corresponding generic optimization methods) automatically maximize the margin. The convergence rates are determined by the time-averaged regret bounds of these online learning algorithms when played against each other, which turn out to be much faster than the worst-case O (1 / √ T ) rate.

Wang et al. (2022b) were the first to draw parallels between Nesterov-accelerated GD for ERM and solving the bilinear game through an online dynamic. However, it was still open that whether this kind of analysis suits other GD-based methods. Moreover, the non-linearity of the mirror map in generic optimization methods makes analysis particularly challenging. In this paper, we reveal that the game framework can in fact encompass implicit bias analysis for a range of generic optimization methods, and offer a more streamlined and unified analysis. Within this game framework, we derive several other results beyond those mentioned above:

By selecting suitable online learning algorithms, we obtain a momentum-based datadependent MD algorithm with an O ( V T T 2 ( q -1) + log n log T T 2 ) ∥ · ∥ q -margin maximization rate, where V T = ∑ T t =2 ∥ p t -p t -1 ∥ 2 1 is the path-length of a series of distributions on the training data p t . In the worst case, this reduces to the margin maximization rate of MD, but this can be much tighter if V T is sublinear in T .

Apart from margin maximization rates, we also demonstrate the corresponding directional error, i.e., the bound on the ℓ q -distance between the maximal margin classifier and the normalized output of the generic methods, which are also controlled by the regret bounds of two-players against each other. This kind of convergence rates are new for most of the generic methods. In general, we prove that the directional errors are typically a square-root factor worse than the margin maximization rates.

For our steepest descent, by setting the norm to the general norm ∥ · ∥ and the ℓ 2 -norm respectively, we can recover the algorithms and theoretical guarantees in Nacson et al. (2019); Ji & Telgarsky (2021) under the game framework. This implies that these algorithms can also be viewed as solving a regularized bilinear game using online learning algorithms, offering a deeper understanding of the role of implicit bias in optimization methods.

Additional related work The strategy of solving a zero-sum game using online learning algorithms playing against each other has been extensively studied, primarily through the lens of independent learning agents (e.g., Rakhlin & Sridharan, 2013; Daskalakis et al., 2018; Wang & Abernethy, 2018; Daskalakis & Panageas, 2019; Zhang et al., 2022). In contrast, our central motivation and challenge lies in identifying the exact equivalent forms of generic optimization algorithms under the regularized bilinear game dynamic. Our framework is also motivated by the line of research that employs the Fenchel-game to elucidate commonly used convex optimization methods (Abernethy et al., 2018; Wang & Abernethy, 2018; Wang et al., 2021b). However, our framework diverges significantly from these approaches. These works focus on the convergence of the optimization problem itself, while our framework emphasizes that the choice of optimization algorithm, which solely targets the minimization of empirical risk, has a significant impact on maximizing the margin, which we might view as an 'algorithmic externality.' Max-margin guarantees can not arise from convergence of the ERM objective alone, as there are typically multiple global minima in ERM minimization. Our analysis also considers an entirely different min-max problem than that of the Fenchel game (Wang et al., 2021b). Consequently, the correspondences we establish between optimization algorithms and online dynamics also differ. Finally, we note that previous work has also analyzed the implicit bias through direct primal optimization analysis (e.g., Nacson et al., 2019; Sun et al., 2022) or using a dual perspective (e.g., Ji & Telgarsky, 2021; Ji et al., 2021). For the former analysis, it is unclear whether and how faster rates can be obtained. For the latter, it remains an open question how to extend the framework beyond ℓ 2 -geometry, which in some sense was the motivation for the present work. For more related work, we refer the reader to Appendix A.

2 Preliminaries

In this section, we present our basic setting along with some standard assumptions and definitions.

Notation We use lower case bold face letters x , y to denote vectors, lower case letters a, b to denote scalars, and upper case bold face letters A , B to denote matrices. For a vector x ∈ R d , we use x i to denote the i -th component of x . For a matrix A ∈ R n × d , let A ( i, :) be its i -th row, A (: ,j ) the j -th column, and A ( i,j ) the i -th element of the j -th column. ∀ x ∈ R d , we use ∥ · ∥ to denote a general norm in R d , ∥ · ∥ ∗ its dual norm, ∥ x ∥ q the q -norm of x , defined as ∥ x ∥ q = ( ∑ d i =1 | x i | q ) 1 /q , and q ∈ (1 , 2] . We use ∥ · ∥ p to denote the dual norm of q -norm, where p ∈ [2 , ∞ ) , 1 p + 1 q = 1 .

Solving the regularized bilinear game with online learning:

w

∈

R

d

min

p

∈

∆

n

p

⊤

A

w

-

Φ(

w

)

Margin maximization rate:

γ

-

O

(

C

T

)

directional error:

O

(

√

C

T

)

Empirical risk minimization

with generic optimization methods

Plug in regret bounds

Identify equivalent forms

We denote B ∥·∥ the ∥ · ∥ -ball, defined as B ∥·∥ = { x ∈ R d |∥ x ∥ ≤ 1 } . ∀ x , x ′ ∈ R d , we define the Bregman divergence between x and x ′ with respect to a strictly convex potential function Φ( x ) as D Φ ( x , x ′ ) = Φ( x ) -Φ( x ′ ) -∇ Φ( x ′ ) ⊤ ( x -x ′ ) . For a positive integer n , we denote { 1 , . . . , n } as [ n ] , and the ( n -1) -dimensional simplex as ∆ n . Let E : ∆ n ↦→ R be the negative entropy function, defined as E ( p ) = ∑ n i =1 p i log p i , ∀ p ∈ ∆ n .

Basic setting Consider a set of n data points S = { ( x ( i ) , y ( i ) ) } , where x ( i ) ∈ R d is the feature vector for the i -th example, and y ( i ) ∈ {-1 , +1 } the corresponding binary label. We are interested the optimization trajectory of first-order methods for minimizing the following unbounded and unregularized empirical risk:

where w ∈ R d is a linear classifier, r : R ×{± 1 } ↦→ R is the loss function. Following previous work, we focus on the exponential loss, given by r ( w ⊤ x ; y ) = exp( -y x ⊤ w ) . We introduce the following standard assumption and definitions.

Definition 1 ( ∥·∥ -margin) . For a linear classifier w ∈ R d and a norm ∥·∥ , we define its ∥·∥ -margin as

where A = [ . . . ; y ( i ) x ( i ) ⊤ ; . . . ] ∈ R n × d is the matrix that contains all data.

Assumption 1. Assume S is linearly separable and bounded with respect to some norm ∥ · ∥ . More specifically, we assume ∃ w ∗ ∥·∥ ∈ B ∥·∥ , s.t., w ∗ ∥·∥ = argmax ∥ w ∥≤ 1 min i ∈ [ n ] y ( i ) x ( i ) ⊤ w , whose margin ˜ γ ( w ∗ ∥·∥ ) = γ > 0 . We refer to w ∗ ∥·∥ as the ∥ · ∥ -maximal margin classifier. Note that, for any w ∈ R d , if ˜ γ ( w ) = γ , w and w ∗ ∥·∥ are at the same direction. Finally, we also assume S is bounded wrt some dual norm ∥ · ∥ ∗ , i.e., ∀ i ∈ [ n ] , ∥ x ( i ) ∥ ∗ ≤ 1 .

Definition 2 ( ∥ · ∥ -Margin maximization rate and ∥ · ∥ -directional error) . Suppose Assumption 1 is satisfied. We consider a sequence of solutions w 1 , . . . , w t , . . . , and state that w t converges to w ∗ ∥·∥ if either lim t →∞ ˜ γ ( w t ) → γ , or lim t →∞ ∥ w t ∥ w t ∥ -w ∗ ∥·∥ ∥ → 0 . We define the upper bound on | γ -˜ γ ( w t ) | the ∥ · ∥ -margin maximization rate, and ∥ w t ∥ w t ∥ -w ∗ ∥·∥ ∥ the ∥ · ∥ -directional error.

3 A Game Framework for Maximizing the Margin

In this section, we present a general game framework and demonstrate that solving this game with online learning algorithms can directly maximize the margin and minimize the directional error.

Then, in Section 4, we show that many generic optimization methods can be considered as solving this game with different online dynamics. As a result, the margin maximization rate (and also the directional error) of these optimization methods are exactly characterized by the regret bounds of the corresponding online learning algorithms. We illustrate this procedure in Figure 1. The game objective is defined as follows:

where Φ( w ) = 1 2 ∥ w ∥ 2 is a regularizer and ∥·∥ denotes some general norm in R d . Following previous work (Wang et al., 2021b, 2022b), we apply a weighted no-regret dynamic protocol (summarized in Protocol 1) to solve the game. We first give a brief introduction of Protocol 1, and then present the theorem about the margin of its output.

Description of Protocol 1 In Protocol 1, the players of the zero-sum game try to find the equilibrium by applying online learning algorithms. In each round t , the p -player first picks a decision p t , and passes a weighted loss function to the w -player, defined as

Then, the w -player observes the loss, picks a decision w t , and passes a weighted loss function

to the p -player. Note that the order of the two players can also be reversed. After T iterations, the algorithm outputs the weighted sum of the w -player's decisions: ˜ w T = ∑ T t =1 α t w t . Under this framework, we define the weighted regret upper bound of both players respectively as

Denote the upper bound on the average weighted regret by C T = ( Reg p T + Reg w T ) / ∑ T t =1 α t . We have the following conclusion on the margin and directional error of ˜ w T . The proof of this theorem can be found in Appendix B.

Theorem 1. Suppose Assumption 1 holds with respect to some general norm ∥ · ∥ . Consider solving the two-player zero-sum game defined in (2) by applying Protocol 1. Then ˜ w T will have a positive margin on round T if C T ≤ γ 2 4 . Moreover, as long as C T ≤ γ 2 4 , we have

If Φ( w ) is λ -strongly convex with wrt ∥ · ∥ , we have

Theorem 1 shows that the output of Protocol 1, denoted as ˜ w T , achieves a positive margin when the average regret C T ≤ γ 2 4 . In the following sections, we demonstrate that with appropriately chosen

online learning algorithms C T always decreases with respect to T ; in fact C T → 0 as T → ∞ . Therefore, once the condition C T ≤ γ 2 4 is met for a particular value T 0 , it will also be met for all T ≥ T 0 . Thereafter, ˜ w T continues to increase the ∥ · ∥ -margin and converges to the maximum ∥ · ∥ -margin classifier, and the rate is directly characterized by C T . Since C T is the average regret of the online learning algorithms, better bounds on C T lead to a less stringent condition on large enough T. Finally, we note that the condition on sufficiently large T is also (explicitly or implicitly) required in all previous work on the non-asymptotic margin maximization rates of generic methods (Nacson et al., 2019; Li et al., 2021; Sun et al., 2022). We refer to Appendix A for more details.

4 Implicit Bias of Generic Methods

In this section, we show that average mirror descent and steepest descent can find their equivalent online learning forms under Protocol 1. Thus, their margin maximization rates can be directly characterized by the corresponding average regret C T . For clarity, we use v t to denote the classifier updates in the original methods, and w t the update under the game framework. Note that Theorem 1 clearly implies that the convergence rate of the directional error is always a square-root worse than that of the margin maximization rate. Due to space limitations, we only present the margin maximization rates, while the corresponding rates on directional error are presented in the appendix.

4.1 Mirror-Descent-Type of Methods

First, we consider minimizing (1) by applying the following mirror descent algorithm:

where D Φ ( v , v t -1 ) is the Bregman divergence between v and v t -1 , and Φ( v ) is a strongly convex potential function that defines the mirror map. Note that since the feasible domain in (4) is unbounded, we can rewrite the algorithm in the following form:

In this paper, we consider weighted-average mirror descent with the squared q -norm, i.e., Φ( w ) = 1 2 ∥ w ∥ 2 q , where q ∈ (1 , 2] , and demonstrate that this optimization algorithm can enable faster ∥ · ∥ -margin maximization rates. The detailed update rule is summarized in the left box of Algorithm 1. It is worth noting that this type of regularizer is ( q -1) -strongly convex with respect to ∥ · ∥ q , and can be updated efficiently in closed form as below 1 : for each coordinate i ∈ [ d ] , we have

We make a few final observations about this algorithm: 1) Instead of using the weighted sum ˜ v T , we could output the weighted average ˜ v t ∑ t s =1 α s without altering the margin or directional convergence rate. This is attributed to the scale-invariance of the margin, i.e., ∀ c > 0 , w ∈ R d , ˜ γ ( w ) = ˜ γ ( c w ) . The same argument applies to the directional error. 2) The use of the weighted average is standard in the analysis of mirror descent (e.g., Section 4.2 of Bubeck et al. (2015)). This paper shows that using non-uniform weights is advantageous for achieving rapid margin maximization rates; 3) The per-round computational complexity of (5) is O ( d ) , which is similar to that of p -mirror-descent of Sun et al. (2022). However, we note that the p -MD algorithm of Sun et al. (2022) does not need to compute the norm of the decision at each round, which could be more efficient in real-world applications where parallel or distributed computation is desired.

For Algorithm 1, we have the following theorem. We present its proof in Appendix C, along with a more general theorem that allows a general configuration of the parameters η t , α t and β t .

Theorem 2. Suppose Assumption 1 holds wrt ∥ · ∥ q -norm for q ∈ (1 , 2] . For the left box of Algorithm 1, let η t = 1 L ( v t -1 ) . For the right box, let α t = 1 , and β 1 = 1 , β t = 1 t -1 . Then the methods in the two boxes of Algorithm 1 are identical, in the sense that ˜ v T = ˜ w T . Moreover, we have the average

Algorithm 1 Mirror Descent [Recall ℓ t ( p ) = g ( p , w t ) , and h t ( w ) = -g ( p t , w ) ]

regret upper bound C T = ( 2 q -1 +2log n ) (log T +2) T . Therefore, the algorithm achieves a positive margin when T is sufficiently large such that T ≥ 4 ( 2 q -1 +2log n ) (log T +2) γ 2 . We have the convergence rate

and

The first part of Theorem 2 indicates that the mirror descent algorithm can be described as two players using certain cleverly designed online learning algorithms to solve the regularized bilinear game in (2). More specifically, for the p -player, we propose a new and unusual online learning algorithm, which we call regularized greedy .

Essentially, in round t , the p -player minimizes the previous round's loss function, ℓ t -1 , plus a regularizer at round t , and the two terms are balanced by the parameters α t and β t . On the other hand, we select the follow-the-leader + algorithm for the w -player:

which returns the solution that minimize the cumulative loss so far. The + sign in the name is because the algorithm can pick the decision w t after seeing its loss function. This is an interesting and unusual design because the regularized greedy algorithm will clearly suffer a worst-case linear regret for the p -player. Fortunately, for our specific case we are able to prove a sharper data-dependent regret bound for the p -player as below:

Therefore, the dominating term (i.e., the first term above) of the p -player's regret bound can be canceled by the w -player's regret bound, given by:

Note that the w -player's regret bound is negative as the corresponding algorithm used is clairvoyant , i.e. can see the current loss ℓ t before making a decision at round t . This ensures that sublinear (and more generally fast) rates are possible. Note that β t and α t will influence both the regret bound and the algorithm equivalence analysis, so finding the right parameter configuration that works for both is a non-trivial task. We make the choice β t = α t ∑ t -1 i =1 α i , which ensures both algorithmic equivalence and sublinear regret bounds.

The second part of Theorem 2 shows that the average regret C T of Algorithm 1 is on the order of O ( log n log T ( q -1) γ 2 T ) . Therefore, by plugging in Theorem 1, we observe that the margin shrinks on the order of γ -O ( log n log T γ 2 ( q -1) T ) , and the implicit bias convergence rate is O ( log n log T γ 2 ( q -1) √ T ) . Next, we show an improved rate with a more aggressive step size on the order of O ( t L ( v t ) ) instead of O ( 1 L ( v t ) ) . The proof of this result is given in Appendix C.

Theorem 3. Suppose Assumption 1 holds wrt ∥ · ∥ q -norm for q ∈ (1 , 2] . For the left box of Algorithm 1, let η t = t L ( v t -1 ) , and let the final output be ˜ v T = ∑ T t =1 2 t +1 v t . For the right box, let α t = t , and β 1 = 1 , β t = 2 . Then the two algorithms are identical, in the sense that v T = w T

t -1 ˜ ˜ . Moreover,

and

Observe that the margin maximization rate in Theorem 3 is O ( 1 ( q -1) γ 2 T ) + O ( log n log T γ 2 T 2 ) . Compared to (6), it has a better dependence on log n and log T .

Finally, we focus on a momentum-based mirror descent algorithm, which is given in Algorithm 2. For this algorithm, we have the following guarantee.

Theorem 4. Suppose Assumption 1 holds wrt ∥ · ∥ q -norm for q ∈ (1 , 2] . For the left box of Algorithm 2, let η t = t L ( v t -1 ) , and ̂ η t = t -1 L ( v t -1 ) . For the second box, let α t = t , and β t = 2 t +1 . Then the methods in the two boxes of Algorithm 2 are identical, in the sense that ˜ v T = ˜ w T . Moreover, when T ≥ √ 8 ( 4 log n log T + 2 T q -1 ) γ , we have

and

The above theorem shows that, for sufficiently large T , the margin maximization rate can be datadependent. Note that ∑ T t =1 ∥ p t -p t -1 ∥ 2 1 ≤ 2 T , so in the worst case, the bound reduces to the results in Theorem 3, but it can become significantly better when ∑ T t =1 ∥ p t -p t -1 ∥ 2 1 is small. We expect that when T is very large, p T will change very slowly as we already know that the direction of ˜ v T will converge - however, turning this into a precise faster rate dependent on the original training data geometry, i.e. A , is an intriguing open question.

4.2 Steepest Descent

Next, we consider the steepest descent method under a given general norm ∥ · ∥ . For a succinct description of this algorithm see (Boyd & Vandenberghe, 2004). For completeness, we have also described this algorithm in the left box of Algorithm 3. In each iteration t , the algorithm first identifies the steepest direction with respect to the norm ∥ · ∥ (Step 2). It then adjusts the decision towards this direction using a specific step size η t (Step 3). After T iterations, the algorithm yields the final iteration v T . In the following, we show that an O ( λ +log n γ 2 λT ) ∥ · ∥ -margin maximization rate can be achieved when the squared-norm, i.e. 1 2 ∥ · ∥ 2 is λ -strongly convex (e.g., 1 2 ∥ · ∥ 2 q is ( q -1) -strongly convex wrt ∥ · ∥ q ). The proof of this result provided in Appendix D. Moreover, we recover the slower O ( log n +log T √ T ) of Nacson et al. (2019) rate as a special case for norms that are not necessarily strongly convex.

Theorem 5. Suppose Assumption 1 holds wrt a general norm ∥ · ∥ , and 1 2 ∥ · ∥ 2 is λ -strongly convex wrt ∥ · ∥ . Let η t = α t ∥∇ L ( w t ) ∥ L ( w t ) , and δ t -1 = α t -1 . Then the methods in the two boxes of Algorithm 3 are are equivalent, in the sense that v T = ˜ w T . Moreover, let α t = λ 2 . Then C T = λ 4 +log n Tλ . Therefore, when T ≥ λ +4log n λγ 2 , we have

and

The first part of Theorem 5 elucidates the equivalent online dynamic of the steepest descent algorithm, which is also depicted in the right box of Algorithm 3. The w -player employs the standard online mirror descent (OMD) algorithm (Hazan, 2016), while the p -player utilizes FTRL + , i.e., p t is selected by minimizing the cumulative loss observed so far, coupled with a regularization term. The crux of our algorithm equivalence analysis lies in evaluating the output of the w -player. For this, we initially prove that given δ t = 1 α t , the OMD algorithm condenses to best-response (BR), that is, w t = argmin w ∈ R d α t h t -1 ( w ) . We then prove that BR's output coincides with the steepest descent direction. The second part of Theorem 5 shows that the average regret of this online learning dynamic is O ( λ +log n γ 2 λT ) , which leads to the corresponding fast margin maximization/small direction error. We note that the favorable average regret is made possible by allowing the two players to play against each other, rather than plugging in worst-case regret bounds.

4.3 Even Faster Rates with Accelerated Generic Methods

In the preceding subsections, we showed that, with suitable step sizes, steepest descent and average mirror descent can achieve an O ( log n log T T ) margin maximization rate. We now aim to derive even faster rates using two approaches, as illustrated in the top two boxes of Algorithm 4. The left box introduces a Nesterov-acceleration-based mirror descent (Nesterov, 1988; Tseng, 2008): In each iteration t , the algorithm initially performs an extra update to yield v t (Step 2), then executes a mirror descent step with the gradient at v t (Step 3), and finally calculates a moving average (Step 4). On the other hand, the right box depicts a momentum-based steepest descent algorithm: In each iteration,

w

-player:

p

-player:

Output:

w

p

t

w

˜

T

t

= argmin

w

∈

R

d

+

D

= argmin

∑

p

∈

∆

n

=

T

t

=1

∑

t

-

1

t

2

i

=1

α

t

w

1

α

∥·∥

α

t

.

t

2

-

(

i

1

∇

w

ℓ

,

h

w

i

(

p

t

-

t

-

1

(

1

) +

w

t

)

D

KL

⟨

δ

-

1

)

(

p

,

,

w

1

)

n

⟩

1: for t = 1 , . . . , T do

2: v t = β t, 1 ˜ v t -1 + β ′ t, 1 z t -1

3: ∇ Φ( z t ) = ∇ Φ( z t -1 ) -η t ∇ L ( v t )

4: ˜ v t = β t, 2 ˜ v t -1 + β ′ t, 2 z t

5:

end for

6: Output : ˜ v T

1: for t = 1 , . . . , T do

2:

g

t

=

β

t,

3

g

t

-

1

+

β

′

t,

3

∇

L

(

3: s t = argmin ∥ s ∥≤ 1 s ⊤ g t

4: v t = v t -1 + η t s t

5: end for

6: Output : v T

the method maintains a momentum term g t with an additional gradient (Step 2), then identifies the steepest direction with respect to g t (Step 3), and applies this direction to update the decision (Step 4). At first glance, these two algorithms appear markedly different. However, we show that with appropriately chosen parameters, they are actually equivalent, in the sense that they both correspond to the online dynamic in the bottom box of Algorithm 4. More specifically, we provide the following theoretical guarantee. The proof is given in Appendix E.

Theorem 6. Suppose Assumption 1 holds wrt a general norm ∥ · ∥ , and 1 2 ∥ · ∥ 2 is λ -strongly convex wrt ∥ · ∥ . For the left box, let β t, 1 = λ 4 , β ′ t, 1 = λ 2( t -1) , β t, 2 = 1 , β ′ t, 2 = 2 t +1 , and η t = t L ( v t ) . For the right box, let β t, 3 = t -1 t +1 , β t, 4 = λ 4 , β ′ t, 4 = λt ∥ g t -1 ∥ ∗ 4 , β ′ t, 3 = 2 ( t +1) L ( β t, 4 v t -1 + β ′ t, 4 s t -1 ) , and η t = t ∥ g t ∥ ∗ . For the bottom box, let c = λ 4 , α t = t . Then all three methods in Algorithm 4 are identical, in the sense that ˜ v T = v T = ˜ w T . Moreover, when T ≥ 4 √ 2 log n √ λγ , we have

and

Remark Theorem 6 reveals that the two strategies implemented in Algorithm 4 yield an optimal O (log n/ [ γ 2 T 2 ]) rate. It is worth noting that a similar online dynamic to the one detailed in the bottom box of Algorithm 4 was also considered by (Algorithm 5, Wang et al., 2022b). Nonetheless, there are some crucial distinctions: 1) Their work only demonstrated that this dynamic could achieve a positive margin, leaving open questions regarding whether the margin can be maximized (i.e., converge to γ ), and if so, what the margin maximization rate would be; 2) They only presented the online dynamic, without its equivalent optimization form. Finally, we note that, Theorem 6 requires the norm to be strongly convex, which is satisfied for, e.g. the q -norm when q ∈ (1 , 2] .

5 Conclusion and Future Work

This paper examines the implicit bias of generic optimization methods, delivering accelerated margin maximization and directional errors for average mirror descent and steepest descent. Our approach translates generic optimization methods for ERM into online learning dynamics for a regularized bilinear game , offering a simpler analysis and a fresh perspective on implicit bias. Despite the effectiveness of the game framework in handling generic methods and accelerated techniques, it presently holds some limitations: 1) the framework is currently operational only for exponential loss, making its extension to handle more general losses a vital area for future research; 2) identifying algorithmic equivalence is nuanced and non-trivial, and it is as yet unresolved whether this framework can elucidate other methods, such as the last-iterate of MD; 3) Finally, it remains to be seen whether more advanced online learning algorithms are beneficial under our framework, such as parameter-free online learning (Orabona & Pál, 2016; Cutkosky & Orabona, 2018) or adaptive online learning methods (van Erven & Koolen, 2016; Wang et al., 2019; Zhang et al., 2019; Wang et al., 2021a).

β

t,

4

v

t

-

1

+

β

′

t,

4

s

t

-

1

)

Acknowledgments. GWwas supported by a ARC-ACO fellowship provided by Georgia Tech. VM was supported by the NSF (through CAREER award CCF-2239151 and award IIS-2212182), an Adobe Data Science Research Award, an Amazon Research Award and a Google Research Colabs Award. JA was supported by the AI4OPT Institute, as part of NSF Award 2112533, and NSF through Award IIS-1910077.

References

Abernethy, J., Lai, K. A., Levy, K. Y., and Wang, J.-K. Faster rates for convex-concave games. In Proceedings of the 31st Annual Conference on Learning Theory , pp. 1595-1625, 2018.

Azizan, N. and Hassibi, B. Stochastic gradient/mirror descent: Minimax optimality and implicit regularization. In the 7th International Conference on Learning Representations , 2019.

Boyd, S. and Vandenberghe, L. Convex Optimization . Cambridge University Press, 2004.

Bubeck, S. et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Chen, S. S., Donoho, D. L., and Saunders, M. A. Atomic decomposition by basis pursuit. SIAM review , 43(1):129-159, 2001.

Cutkosky, A. and Orabona, F. Black-box reductions for parameter-free online learning in banach spaces. In Proceedings of the 31st Annual Conference on Learning Theory , 2018.

Daskalakis, C. and Panageas, I. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In 10th Innovations in Theoretical Computer Science Conference , 2019.

Daskalakis, C., Ilyas, A., Syrgkanis, V., and Zeng, H. Training gans with optimism. In the 6th International Conference on Learning Representations , 2018.

Gentile, C. A new approximate maximal margin classification algorithm. Advances in Neural Information Processing Systems 13 , 2000.

Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Characterizing implicit bias in terms of optimization geometry. In Proceedings of the 35th International Conference on Machine Learning , pp. 18321841, 2018a.

Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems 31 , 2018b.

Hazan, E. Introduction to online convex optimization. Foundations and Trends in Optimization , 2 (3-4):157-325, 2016.

Ji, Z. and Telgarsky, M. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300 , 2018.

Ji, Z. and Telgarsky, M. Gradient descent aligns the layers of deep linear networks. In the 8th International Conference on Learning Representations , 2020a.

Ji, Z. and Telgarsky, M. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems 33 , 2020b.

Ji, Z. and Telgarsky, M. Characterizing the implicit bias via a primal-dual analysis. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory , pp. 772-804, 2021.

Ji, Z., Dudík, M., Schapire, R. E., and Telgarsky, M. Gradient descent follows the regularization path for general losses. In Proceedings of the 33rd Annual Conference on Learning Theory , pp. 2109-2136. PMLR, 2020.

Ji, Z., Srebro, N., and Telgarsky, M. Fast margin maximization via dual acceleration. In Proceedings of the 38th International Conference on Machine Learning , pp. 4860-4869, 2021.

Lai, K.-W. and Muthukumar, V. General loss functions lead to (approximate) interpolation in high dimensions. arXiv preprint arXiv:2303.07475 , 2023.

Li, Y., Ju, C., Fang, E. X., and Zhao, T. Implicit regularization of bregman proximal point algorithm and mirror descent on separable data. arXiv preprint arXiv:2108.06808 , 2021.

Lyu, K. and Li, J. Gradient descent maximizes the margin of homogeneous neural networks. In the 8th International Conference on Learning Representations , 2020.

Nacson, M. S., Lee, J., Gunasekar, S., Savarese, P. H. P., Srebro, N., and Soudry, D. Convergence of gradient descent on separable data. In Proceeding of the 22nd International Conference on Artificial Intelligence and Statistics , pp. 3420-3428, 2019.

Nesterov, Y. On an approach to the construction of optimal methods of minimization of smooth convex functions. Ekonomika i Mateaticheskie Metody , 24(3):509-517, 1988.

Neyshabur, B., Tomioka, R., and Srebro, N. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

Orabona, F. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Orabona, F. and Pál, D. Coin betting and parameter-free online learning. Advances in Neural Information Processing Systems 29 , 2016.

Rakhlin, S. and Sridharan, K. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems 26 , pp. 3066-3074, 2013.

Ramdas, A. and Pena, J. Towards a deeper geometric, analytic and algorithmic understanding of margins. Optimization Methods and Software , 31(2):377-391, 2016.

Soheili, N. and Pena, J. A smooth perceptron algorithm. SIAM Journal on Optimization , 22(2): 728-737, 2012.

Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1):2822-2878, 2018.

Sun, H., Ahn, K., Thrampoulidis, C., and Azizan, N. Mirror descent maximizes generalized margin and can be implemented efficiently. In Advances in Neural Information Processing Systems 35 , 2022.

Telgarsky, M. Margins, shrinkage, and boosting. In International Conference on Machine Learning , pp. 307-315. PMLR, 2013.

Tseng, P. On accelerated proximal gradient methods for convex-concave optimization. submitted to SIAM Journal on Optimization , 2(3), 2008.

van Erven, T. and Koolen, W. M. MetaGrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29 , pp. 3666-3674, 2016.

Vardi, G. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591 , 2022.

Wang, B., Meng, Q., Zhang, H., Sun, R., Chen, W., Ma, Z.-M., and Liu, T.-Y. Does momentum change the implicit regularization on separable data? Advances in Neural Information Processing Systems 35 , 2022a.

Wang, G., Lu, S., and Zhang, L. Adaptivity and optimality: A universal algorithm for online convex optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence , 2019.

Wang, G., Wan, Y., Yang, T., and Zhang, L. Online convex optimization with continuous switching constraint. Advances in Neural Information Processing Systems , 34:28636-28647, 2021a.

Wang, G., Hanashiro, R., Guha, E., and Abernethy, J. On accelerated perceptrons and beyond. arXiv preprint arXiv:2210.09371 , 2022b.

Wang, J.-K. and Abernethy, J. D. Acceleration through optimistic no-regret dynamics. In Advances in Neural Information Processing Systems 32 , pp. 3824-3834, 2018.

Wang, J.-K., Abernethy, J., and Levy, K. Y. No-regret dynamics in the fenchel game: A unified framework for algorithmic convex optimization. arXiv preprint arXiv:2111.11309 , 2021b.

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Zhang, L., Wang, G., Tu, W.-W., and Zhou, Z.-H. Dual adaptivity: A universal algorithm for minimizing the adaptive regret of convex functions. ArXiv e-prints , arXiv:1906.10851, 2019.

Zhang, M., Zhao, P., Luo, H., and Zhou, Z.-H. No-regret learning in time-varying zero-sum games. In Proceedings of the 39th International Conference on Machine Learning , 2022.

A Additional Related Work

As discussed in the introduction, the implicit bias of GD-based methods on linear spreadable data with exponentially-tailed loss has been extensively studied (Soudry et al., 2018; Ji & Telgarsky, 2018; Nacson et al., 2019; Ji & Telgarsky, 2021; Ji et al., 2021; Wang et al., 2022b). For ℓ 1 -norm, Telgarsky (2013) provides an O ( 1 √ T ) margin maximization rate, and an O ( 1 T ) convergence rate to a sub-optimal margin. The idea of using the regularized game framework for analyzing the implicit bias of Nesterov-accelerated GD is firstly considered in Wang et al. (2022b). This idea is motivated by the smooth perceptron analysis of Soheili & Pena (2012), and the the Fenchel game framework by Wang et al. (2021b). The idea also inspired by the primal-dual analysis of Ji & Telgarsky (2021), who discovered the relationship between the normalized gradient descent for the exponential loss and a smooth update of a distribution over data. Apart from the setting above, the implicit bias of GD-based optimization methods have also been studied in other cases, such as minimizing more general loss functions (Ji et al., 2020; Ji & Telgarsky, 2021; Lai & Muthukumar, 2023), and deep neural networks (Gunasekar et al., 2018b; Ji & Telgarsky, 2020a,b; Lyu & Li, 2020; Vardi, 2022).

Compared with GD-based methods, the implicit bias of generic optimization methods for linear classification is less understood. Apart from the work summarized in the introduction (Gunasekar et al., 2018a; Nacson et al., 2019; Li et al., 2021; Sun et al., 2022), the implicit bias of mirror descent is also studied in regression problems (Gunasekar et al., 2018a; Azizan & Hassibi, 2019). We note that, as indicated in Gunasekar et al. (2018a) and Sun et al. (2022), the analysis of implicit bias for classification and regression are 'fundamentally different'. Apart from steepest and mirror descent, the implicit bias of other generic methods, such as Adagrad and Adam are also studied (Gunasekar et al., 2018a; Wang et al., 2022a)

Discussion on large enough T Our Theorem 1 indicates that to ensure the margin can be maximized, the number of iterations T has to be large enough such that C T ≤ γ 2 4 . We note that, similar conditions are also required (explicitly or implicitly) in other work for analyzing the margin maximization rates for generic optimization methods. For example, in the proof of Theorem 8 of Nacson et al. (2019), in order to combine the upper bound in (53) and lower bound in (54), one need to make sure the LHS of (53), i.e., the margin, is non-negative, which essentially hinge on the condition that √ T = Ω ( log n log T γ 2 ) , similar to what is required in our Corollary 1. In Sun et al. (2022), page 21, to make sure the term 1 + log n m t be a constant, T has to be large enough such that m t , the margin, goes to ∞ . In Li et al. (2021), the requirements for a large enough T is stated in the main theorem (e.g., Theorem 4.2).

Comparison with Wang et al. (2022b) As discussed in the introduction, our work is motivated by Wang et al. (2022b). Compared with Wang et al. (2022b), we note that: 1) Wang et al. (2022b) draws the connection between Nesterov-accelerated GD for ERM and solving the bilinear game through an online dynamic. However, it was unclear whether this kind of analysis suits other gradientdescent-based methods, and generic optimization methods such as mirror descent/steepest descent was not addressed at all. We observe in this work that the non-linearity of the mirror map in generic optimization methods, such as mirror descent and steepest descent, makes the analysis particularly challenging. In this paper, we reveal that the game framework can in fact encompass implicit bias analysis for a range of generic optimization methods, and offer a more streamlined and unified analysis. 2) Wang et al. (2022b) also proposed an accelerated p-norm perceptron problem. However, they only demonstrated that the algorithm could achieve a non-negative margin, leaving open questions regarding whether the margin can be maximized, and if so, what the margin maximization rate would be; 2) They only presented the online dynamic, without its equivalent optimization form under ERM.

B Proof of Theorem 1

Define m ( w ) = min p ∈ ∆ n g ( p , w ) , w T = 1 ∑ T t =1 α t ∑ T t =1 α t w t = 1 ∑ T t =1 α t ˜ w T , and we introduce following lemma, which shows that using online learning for solving the game defined in (2) maximizes m ( w ) .

Lemma1 (Abernethy et al. (2018)) . Consider solving the game defined in (2) with the online learning dynamic defiend in Protocol 1. We have that ∀ w ∈ R d ,

Based on Lemma 1 and the definition of m ( · ) , we have

which implies that

The above proof follows the main idea in Wang et al. (2022b). Next, we turn to lower bound ∥ ˜ w T ∥ . Note that since w T (and also ˜ w T ) does not have a simple explicit form, the the technique for lower bounding the norm in (Wang et al., 2022b) fails and we need to find a new approach. Let ( x , y ) ∈ { ( x ( i ) , y ( i ) ) } n i =1 be a data point. We have

where the first inequality is due to assumption that ∥ x ∥ ∗ ≤ 1 , the second inequality is derived from the Cauchy-Schwarz inequality. To proceed, we need a lower bound on the unormalized margin of ˜ w T . We have

where for the first inequality we apply Lemma 1 and compare m ( ˜ w T ) with that of γ w ∗ ∥·∥ , and the last equality is derived based on Assumption 1 (the margin of w ∗ ∥·∥ is γ , and ∥ w ∗ ∥·∥ ∥ = 1 ). . (12) suggests that

Note that, to plug in the lower bound of ˜ w T , we need to ensure the RHS of (13) is positive. Notice that when γ 2 2 ∑ T t =1 α t ≥ 2 ( Reg p T + Reg w T ) ,

Combining (10), (11), and (13), we have

Note that to apply (13), we need γ 2 2 ∑ T t =1 α t -2 ( Reg p T + Reg w T ) ≥ 0 .

Finally, we focus on the distance between ˜ w T ∥ ˜ w T ∥ and w ∗ ∥·∥ for the case where Φ( w ) is strongly convex wrt ∥ · ∥ . This part of the proof is motivated by Theorem 4 of Ramdas & Pena (2016), who show that a variant of the perceptron algorithm can converge to the ℓ 2 -maximum margin classifier in an O (1 / √ t ) convergence rate. We have

where the first inequality is based on the Minkowski inequality and the fact that ∥ w ∗ ∥·∥ ∥ = 1 . Next, note that m ( w ) is λ -strongly concave with respect to ∥ · ∥ , and γ w ∗ ∥·∥ maximize m ( w ) . This is because it is easy to see that the optimal solution of m ( w ) always lies in the direction of w ∗ ∥·∥ , and we only need to decide the norm. Let c > 0 be some constant, we have m ( c w ∗ ∥·∥ ) = cγ -1 2 c 2 . The function is maximized when c = γ , which implies that the optimal solution is γ w ∗ ∥·∥ . Combining these facts with Lemma 1, we have

Finally, combining the lower bound of w T proved in (11), we have when γ 2 2 ∑ T t =1 α t -2 ( Reg p T + Reg w T ) ≥ 0 ,

so, combining the equation above with (15) and (16), we get

C Omitted Proof in Section 4.1

In this section, we provide the omitted proof of Section 4.1. Here, we present a more general algorithm framework (given in Algorithm 5) which allows different step sizes. In the following, we first state a general theorem for this algorithm, and Theorems 2 and 3 is then given as Corollaries 2 and 3.

Algorithm 5 Mirror Descent (General Version)

Theorem 7. Suppose Assumption 1 holds wrt ∥ · ∥ q -norm for q ∈ (1 , 2] . For the left box of Algorithm 5, let η t = α t L ( v t -1 ) . For the right box, let β t be α t ∑ t -1 i =1 α i for t > 1 , β 1 = α 1 . Then the methods in the two boxes of Algorithm 5 are identical, in the sense that ˜ v T = ˜ w T , and v t = w t · ∑ t i =1 α i . Moreover, the regret upper bound

Thus, ˜ v T achieves a positive margin (no smaller than γ 2 / 4 ) for sufficiently large T such that

After (18) is satisfied, the margin of ˜ v T is lower bounded by

and the directional error is

Next, we show that different step sizes (decided by α t ) lead to different implicit bias convergence rates. Firstly, we consider setting the step size as η t = 1 √ tL ( v t -1 ) , which leads to a slow ˜ O ( 1 γ 2 √ T ( q -1) ) bound.

Corollary 1. Let α t = 1 √ t . Then the margin is lower bounded by

and

when T is sufficiently large such that √ T ≥ 4 (( 2 q -1 +log n ) log T +4log n + 4 q -1 ) γ 2 = ˜ Ω ( 1 ( q -1) γ 2 ) .

Next, we show that a faster rate can be obtained with a constant α t , which implies Theorem 2.

Corollary 2 (Theorem 2) . Let α t = 1 . Then the margin is lower bounded by

and

when T is sufficiently large such that T ≥ 4 ( 2 q -1 +2log n ) (log T +2) γ 2

Remark Note that, by setting α t = 1 , according to Theorem 7, we get η t = 1 L ( v t -1 ) , β 1 = 1 , and β t = 1 ∑ t -1 j =1 α j = 1 t -1 , which recovers the parameter configuration in Theorem 2.

Corollary 2 shows that, with a step size of order O ( 1 L ( v t -1 ) ) , Algorithm 1 enjoys a faster ˜ O ( 1 ( q -1) γ 2 t ) margin maximization rate when t = Ω ( log n log t ( q -1) γ 2 ) . Next, we also considered using a even larger step size, η t = t L ( v t -1 ) , and recover Theorem 3.

T 2

Note that, when α t = t , we have η t = t L ( v t -1 ) , which recovers Theorem 3. Finally, we note that, if T is fixed and known before, we can set α t as T and get rid of the log T term.

C.1 Proof of Theorem 7

We first focus on the algorithm equivalence, and start from the online learning framework. For w -player, we have

which implies that

To proceed, note that

Thus, ∀ c > 0 , c ∇ Φ( w ) = ∇ Φ( c w ) . Then, multiplying c = ∑ t j =1 α j on both sides, we get

On the other hand, for the p -player, we have

Based on a standard argument on the relationship between OMD with the negative entropy regularizer on the simplex (see, e.g., Section 6.6 of Orabona, 2019), it is easy to verify that ∀ i ∈ [ n ] , t ∈ [ T ] ,

where p t,i is the i -th element of p t . Moreover, based on the definition of L ( w ) , for any w ∈ R d ,

which implies that

Combining the above equations and the definition of β t = α t ∑ t -1 i =1 α i , we get

Substituting v t = w t · ∑ t j =1 α j , we get

and ˜ w T = ∑ T t =1 α t w t = ∑ T t =1 α t ∑ t j =1 α j v t . The proof is finished by replacing α t L ( v t -1 ) with η t .

Next, we focus on bounding the regret of the two players. For the p -player, let p ∗ ,ℓ = min p ∈ ∆ n ∑ T t =1 α t p ⊤ Aw t be the best decision in hindsight for the online leanring problem. We have

Here, inequality ( a ) is based on the optimality of p t , inequality ( b ) is due to the fact that the negative entropy regularizer is upper bounded, inequality ( c ) is because of the Hölder's inequality, inequality ( d ) is derived from p t , p ∈ ∆ n , inequality ( e ) is based on the Hölder's inequality and ∥ x ( i ) ∥ p ≤ 1 for all i ∈ [ n ] , and inequality ( f ) is based on Young's inequality:

where we pick ∑ t -1 j =1 α i ( q -1) 2 as the constant of Young's inequality. Finally, note that w 1 = [ ∇ Φ] -1 ( A ⊤ p 1 ) , so based on the property of p -norm, we have

For the w -player, note that h t ( w ) is ( q -1) -strongly convex wrt the ∥ · ∥ q -norm. Thus, based on Lemma 3, we have

C.2 Proof of Corollary 1

Note that C T ≤ γ 2 4 is equivalent to γ 2 4 ∑ T t =1 α t ≥ ( Reg p T + Reg w T ) . We have γ 2 4 ∑ T t =1 α t = γ 2 4 ∑ T t =1 1 √ t ≥ γ 2 4 √ T , and

Therefore, C T ≤ γ 2 4 when

and in this case

C.3 Proof of Corollary 2

We have γ 2 4 ∑ T t =1 α t = γ 2 4 T , and

Therefore, C T ≤ γ 2 4 when

and in this case and in this case

C.5 Proof of Theorem 4

We first focus on the algorithm equivalence. We have

So we have

Therefore, we have

and in this case

C.4 Proof of Corollary 3

We have γ 2 4 ∑ T t =1 α t = γ 2 4 T ( T +1) 2 ≥ γ 2 8 T 2 , and

Therefore, C T ≤ γ 2 4 when

On the other hand, for the p -player, we have

Based on the relationship between OMD with the negative entropy regularizer on the simplex (Hazan, 2016), it is easy to verify that ∀ i ∈ [ n ] , t ∈ [ T ] ,

Combining the equations above and replace ∑ t k =1 α k w k with v t , we get

The proof is finished by setting α t = t .

Next, we focus on the regret. For the w -player, Note that h t ( w ) is ( q -1) -strongly convex wrt the ∥ · ∥ -norm. Let ̂ w t = argmin w ∈ R d ∑ t -1 i =1 α i h i ( w ) . Then, based on Lemma 4, we have

where the first inequality is based on Hölder's inequality, the second inequality is based on Young's inequality, i.e.,

which implies that

For the p -player, we have

Finally, we focus on the margin and implicit bias. Since α t = t , for the w -player's regret, we have

and for the p -player, we have

Following Theorem 1, we have the margin and implicit bounds when

since the RHS is exactly γ 2 4 ( Reg p T + Reg w T ) .

D Omitted Proof in Section 4.2

In this section, we provide the proof related to the steepest descent algorithm. We first restate Algorithm 3, which is presented in Algorithm 6. Here, we provide two online dynamic under the game framework. They both are equivalent to the steepest descent algorithm in the left box, in the sense that v T = ˜ w T . The left one is good for recovering the results in Nacson et al. (2019), while the bottom one is more suitable for analysing our accelerated rates.

We first recover the results of Nacson et al. (2019) in Theorem 8, and then proof our Theorem 5.

Algorithm 6 Steepest Descent [Recall ℓ t ( p ) = g ( p , w t ) , and h t ( w ) = -g ( p t , w ) ]

Theorem 8. Suppose Assumption 1 holds wrt a general norm ∥ · ∥ . Let η t = α t ∥∇ L ( w t -1 ) ∥ ∗ L ( w t -1 ) . Then, the methods in the top two boxes of Algorithm 6 are equivalent, in the sense that v T = ˜ w T . Moreover, let α t = 1 √ t . Then C T = log n +2log T +2 √ T . Therefore, when T is sufficiently large such that √ T ≥ 4(log n +2log T +2) γ 2 , we have

The first part of Theorem 8 shows that the steepest descent algorithm can be seen as an online learning dynamic where the in each round p -player performs the standard follow-the-regularized-leader algorithm (Hazan, 2016), while the w -player uses best-response + (Wang et al., 2021b), meaning it picks the decision by directly minimizing the loss of the round. The second part of the theorem shows that the margin convergence rate, which recovers the γ -O ( log n +log T √ T ) rate of Nacson et al. (2019), while the convergence in terms of distance is new. Note that the strongly convex condition is not required for the margin maximization analysis, but is needed for the distance convergence analysis. Next, we restate Theorem 5, where an improved margin maximization rate when the strong convexity condition is met.

Theorem 9 (Restate of Theorem 5) . Suppose Assumption 1 holds wrt a general norm ∥ · ∥ , and 1 2 ∥ · ∥ 2 is λ -strongly convex wrt ∥ · ∥ . Let η t = α t ∥∇ L ( w t ) ∥ L ( w t ) . Then the methods in the first and third boxes of Algorithm 6 are are equivalent, in the sense that v T = ˜ w T . Moreover, let α t = λ 2 . Then C T = λ 4 +log n Tλ . Therefore, when T ≥ λ +4log n λγ 2 , we have

D.1 Proof of Theorem 8

We first focus on the algorithm equivalence between the top two boxes. Based on the relationship between FTRL and EWA (Orabona, 2019), one can verify that

Combining with the definition of L and ˜ w t , it implies that

That is, ∇ L ( ˜ w t -1 ) = -L ( ˜ w t -1 ) A ⊤ p t . Let ˜ s t = argmin ∥ s ∥≤ 1 s ⊤ ∇ L ( ˜ w t -1 ) L ( ˜ w t -1 ) . Note that, on one hand, we have

where the second equality is because because the argmin does not change if we scale the objective functions. On the other hand, we have

To proceed, we introduce the following lemma.

Lemma 2. Let ∥ · ∥ be any norm in R d . Let a ∈ R d , and

Then

Proof. We first focus on (40). Note that the objective has two terms. For the first term, based on Hölder's inequality, we have

Let ∥ x ∥ = c , where c > 0 is a constant, then the equality is achieved (and thus the first term of the objective function is minimized) when

In this case, for the objective function of (40), we have -a ⊤ x + 1 2 ∥ x ∥ 2 = -c ∥ a ∥ ∗ + 1 2 c 2 . It's easy to see that the objective function is minimized when c = ∥ a ∥ ∗ . The proof is finished by combining (39) and (41).

This lemma shows that the best response direction (under our game) is the steepest direction. Thus, we have

where the first equality is based on the updete rule of w t at the right box of Algorithm 6, the final inequality is based on Lemma 2 and (38). Note that, for the first equality, we dropped α t as it is a scaling parameter and does not have a influence on argmin . Thus, combining with (37), we have

Finally, we have

We can finish the first part of the proof by replacing ˜ w t with v t , α t ∥∇ L ( ˜ w t -1 ) ∥ ∗ L ( ˜ w t -1 ) with η t , and noticing argmin ∥ s ∥≤ 1 s ⊤ ∇ L ( ˜ w t -1 ) is s t -1 .

Next, we study the regret bound. Note that the p -player plays the FTRL algorithm on the simplex with a 1-strongly convex regularizer D KL ( p , 1 n ) (wrt ℓ 1 -norm). Therefore, based on Lemma 5, the regret can be upper bounded by

where the equality is because

On the other hand, the w -player uses the best response + algorithm, and one can easily observe that the regret is upper bounded by 0. Finally, we have ∑ T t =1 α t = ∑ T t =1 1 √ t ≥ √ T .

D.2 Proof of Theorem 9

We first focus on the bottom box. For the w -player, we show that, if we set δ t -1 = 1 α t -1 , then this OMDalgorithm is equivalent to the best response algorithm.

Specifically, note that Φ( w ) = 1 2 ∥ w ∥ 2 is now λ -strongly convex. Thus the corresponding mirror map is well-defined and unique, and the function ∇ Φ( · ) is invertible. Therefore, the solution for best response is

On the other hand, since the w -player uses OMD, and the decision set is unbounded, we have

Note that δ t -1 α t -1 = 1 . Combining (44) and (45), we can draw the conclusion that w t and ̂ w t are identical, which shows OMD (with δ t -1 = 1 α t -1 ) and BR (at round t -1 ) here are the same. We use the BR form the algorithm equivalence analysis, and OMD form for the regret analysis.

Next, we prove the algorithm equivalence of the left and bottom boxes in Algorithm 6. The arguments are similar to that in Appendix D.1. The difference is that the order of the w -player and the p -player is switched. Firstly, For the p -player, based on the connection between FTRL and EWA, we have

Combining with the definition of L , it implies that

That is, ∇ L ( ˜ w t ) = -L ( ˜ w t ) A ⊤ p t . Let

Combining the first equality in (46) and Lemma 2, we have

Thus,

Finally, we have

We can finish the first part of the proof by replacing ˜ w t with v t , α t ∥∇ L ( ˜ w t -1 ) ∥ ∗ L ( ˜ w t -1 ) with η t -1 , and argmin ∥ s ∥≤ 1 s ⊤ ∇ L ( ˜ w t -1 ) with s t .

Next, we focus on regret. For the w -player, it uses the OMD algorithm, and we set the initial point w 0 = 0 . Note that we fixed α t = λ 4 for all t , and thus step size δ t -1 = 1 α t -1 = 4 λ is also fixed.

where the second-to-last inequality is derived using triangle inequality and the assumption that ∥ x ( i ) ∥ ∗ is upper bounded by 1. Next, for u , note that

Based on Lemma 2, we have where

Therefore,

where the inequality is based on the triangle-inequality and the assumption that the dual norm of data is upper bounded by 1. Finally, for the p -player, since it uses FTRL + , based on Lemma 6, we have

To summarize, and let α t = λ 2 , we have

E Proof of Theorem 6

We first focus on the equivalence between the left and bottom boxes of Algorithm 4. For the w -player, similar to the proof of Theorem 7, we have

which implies that and thus

On the other hand, for the p -player, since it performs Optimistic FTRL on a simplex with the negative entropy regularizer, we have which implies that

Let z t = w t ∑ t i =1 α i , then we have

Finally, notice that ˜ w t = ˜ w t -1 + α t w t = ˜ w t -1 + α t ∑ t i =1 α i z t . The proof is finished by replacing ˜ w t with ˜ v t , w t with z t ∑ t i =1 α i , c ˜ w t -1 + cα t w t -1 with v t , configuring β t, 1 = c = λ 4 , β ′ t, 1 = cα t ∑ t -1 i =1 α i = λ 2( t -1) , β 2 ,t = 1 , β ′ 2 ,t = α t ∑ t i =1 α t = 2 t +1 , η t = t L ( v t ) .

Next, we focus on the equivalence between the right and bottom boxes. Note that for the w -player, we also have

Let

Based on Lemma 2, we have

Next, let g t = -1 ∑ t j =1 α j ∑ t j =1 α j A ⊤ p j , we know

For the p -player, it is clear that due to the optimistic term, we have

To summarize, and let α t = t , we can conclude the proof by the following algorithm:

Finally, we focus on the regret bound. For the w -player, note that Φ( w ) is λ -strongly convex with respect to ∥ · ∥ . Thus, based on Lemma 3, we have

On the other hand, note that c = λ 4 , so based on Lemma 8, we have

It is easy to verify that t 2 8 ≤ t ( t -1) 4 for t ≥ 2 . So to summarize we get

The proof can be finished by plugging in Theorem 1.

F Regret Bounds for OCO Algorithms

In this section, we provide standard regret bounds for Follow-The-Leader + (FTL + ), Optimistic Follow-The-Leader (OptimisticFTL), Follow-The-Regularized-Leader (FTRL), Follow-TheRegularized-Leader + (FTRL + ), and Optimistic Follow-The-Regularized-Leader (Optimistic FTRL). Lemmas 3, 4, 5 and 6 are based on Lemma of 3 of Wang et al. (2021b), and Lemmas 7 and 8 comes from Theorems 6.8 and 7.35 of Orabona (2019).

Lemma 3. Consider a weighted online learning problem with a series of λ -strongly functions f 1 , . . . , f T , and a series of corresponding parameters α 1 , . . . , α T . The FTL + algorithm, given by

achieves the following regret bound:

Lemma 4. Consider a weighted online learning problem with a series of λ -strongly functions f 1 , . . . , f T , and a series of corresponding parameters α 1 , . . . , α T . Let ̂ z t = argmin z ∈ R d ∑ t -1 i =1 α i h i ( z ) . Then, the Optimistic FTL algorithm, given by

achieves the following regret ∀ z ∈ R d :

Lemma 5. Consider a weighted online learning problem with a series of convex functions f 1 , . . . , f T , and a series of corresponding parameters α 1 , . . . , α T . Let R ( z ) be a 1 -strongly convex regularizer wrt ∥ · ∥ . The FTRL algorithm, given by

achieves the following regret bound:

Lemma 6. Consider a weighted online learning problem with a series of convex functions f 1 , . . . , f T , and a series of corresponding parameters α 1 , . . . , α T . Let R ( z ) be a 1 -strongly convex regularizer wrt ∥ · ∥ . The FTRL + algorithm, given by

achieves the following regret bound:

Lemma 7. Consider a weighted online learning problem with a series of convex functions f 1 , . . . , f T , and a series of corresponding parameters α 1 , . . . , α T . Let Φ( z ) be a β -strongly convex regularizer wrt ∥ · ∥ . Let z 0 ∈ R d be the initial point. The OMD algorithm, given by

achieves the following regret bound:

Lemma 8. Consider a weighted online learning problem with a series of convex functions f 1 , . . . , f T , and a series of corresponding parameters α 1 , . . . , α T . Let R ( z ) be a 1 -strongly convex regularizer wrt ∥ · ∥ , and ψ t ( z ) be the optimism term in round t . The Optimistic FTRL algorithm, given by

achieves the following regret bound:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies efficient algorithms for margin maximization with respect to a general geometry. The problem of margin maximization is interesting because it has been shown that common optimization algorithm such as gradient descent prefers such solutions through a well-known phenomenon called \"implicit bias.\" This paper proposed a novel analytical framework by re-casting the problem as a bilinear game. The main results show that techniques from online learning can be applied to show fast convergence rates for various optimization algorithms such as mirror descent and steepest descent.\n\nSTRENGTHS:\nThis paper brings an interesting angle of studying the properties of optimization algorithms through the lens of online learning and games. In particular, Theorem 1 cleanly encapsulates the reduction from optimization to online learning. I feel that this result has a lot of potential in quickly deriving new optimization guarantees by reusing tools from online learning.\n\nI also like the presentation of this paper that it clearly describes prior literature and makes conscious effort in recovering those results through the more general framework of this paper. This allows me to clearly understand the contribution of this paper and appreciate its versatility.\n\nWEAKNESSES:\nNotation is quite dense and can be difficult to parse. For example,\n1. In section 4.1, it is stated that the mirror descent potential is restricted to $q$-norm for $q \\in (1, 2]$. I think it would be better to bring this up sooner in the Preliminaries section.\n2. $D_E$ is not a standard notation, why not just call it the KL-divergence?\n3. I found that the terms $p$-norms and $q$-norms are used almost interchangeably, which might be confusing to readers.\n\nIn Section 4.3, does the mirror descent potential suffer the same limitation as the results in Section 4.1? I have a hard time figuring this out.\n\nThe theorem statements are very long and thus look unnecessarily intimidating. In particular, the notion of directional errors is almost entirely ignored in Section 4. I personally would love to see more discussion on the directional error.\n\nThe term \"rate\" is used imprecisely in the introduction. Some of the prior literature measure their rate in terms of margin maximization, while some others in directional error. The author should do more to clarify this.\n\nRegarding the remark on eq. (5)'s computational efficiency. While it is true that its total amount of computation compares similarly to algorithm such as GD, I don't think taking norm of the entire weight vector is practical in a lot of cases, especially for deep learning where parallel or distributed computation is desired.\n\nQUESTIONS:\nI would like to see some proof sketch and strongly recommend the author to add it to the revision. In particular, for Section 4.1, I still don't understand where the $q \\in (1, 2]$ limitation comes into play. Does the same limitation apply to Section 4.3?\n\nLIMITATIONS:\nThis is purely a theoretical paper, so I don't think this paper has any immediate negative societal impact. I find the proofs to be generally easy to follow, but I did not follow all of them line-by-line.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work gives a unified perspective on maximal margin problems realized by a wide range of optimization algorithms. It mainly covers three cases: (i) steepest descent under a general norm, (ii) mirror descent, and (iii) momentum-based acceleration. The authors collectively refer to them as generic optimization problems. The essential point of the theory is that they translate these generic optimization problems into a regularized bilinear game with online learning. This enables us to derive relatively tight margin maximization rates and to characterize the implicit bias of each algorithm.\n\nSTRENGTHS:\nThis work is based on a concrete and rigorous theoretical analysis of a wide range of optimization problems. One interesting point is that all of them can be mapped in the regularized bilinear game formulation. It gives us a unified perspective of various optimization problems. In addition, this unified formulation enables us to obtain tight convergence rates compared to the previous work.\n\nWEAKNESSES:\nWhile this work covers various optimization problems and an interesting formulation with the bilinear game, it might be possible to view this as a collection of results that are not particularly novel. The convergence rates of the problems (for example, (i) steepest descent under a general norm, (ii) mirror descent, and (iii) momentum-based acceleration) have been studied although they seem looser than those obtained in the current work. The game formulation has been also provided by some previous work ([Wang et al 2021, 2022b]) in a limited situation.\n\nQUESTIONS:\n**Difference from previous work**\n\nSince this work covers various topics of optimization problems, it seems hard for beginners in this research area to judge at what points it is significantly novel compared to the previous work. In particular, the difference from Wang et al 2022b seems to need more clarification. Did this previous work not address the max-margin rate of all of (i) steepest descent under a general norm, (ii) mirror descent, and (iii) momentum-based acceleration problems?  \n\n**Exetention to other generic loss functions**\n\nIt is quite curious whether the idea of using the bilinear game can be extended to other loss functions except for the exponential loss. Is there any work or evidence that the current proof approach could potentially be applied to other loss functions? Otherwise, is it quite specific to the exponential function? \n\n**Tightness compared to numerical experiments**\n\nAlthough I agree that the theoretical evaluation of the bounds itself contributes to enriching our understanding of the problems, it remains unclear how tight the obtained convergence rate is.  I mean, there would be the possibility that the obtained bounds could be much loose compared to the real optimization in experiments. The authors show no empirical confirmation and this makes a bit unclear the superiority of the fast convergence rates obtained in this work. It is not a major flaw, though.\n\nLIMITATIONS:\nAs I mentioned in Questions, one limitation is that there is no comparison with numerical experiments and tightness (or \"true\" implicit bias) seems unclear. The other is the exponential loss as is mentioned in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the implicit bias of generic optimization methods, which plays a key role in understanding their generalization capabilities in settings with multiple solutions.\n\nThe authors propose a new game framework to derive margin and directional error rates, which consists of transforming the optimization method into an equivalent instance of a 2-player margin-maximization game. Afterwards, margin and directional error rates can be derived directly from the players’ average online regret.\n\nThey show equivalent transformations (under exp loss) from weighted MD with a squared p-norm potential and SD under a general norm to instances of the proposed 2-player game, yielding convergence rates from each transformation’s induced regret. More aggressive learning rates are also studied and shown to be able to improve convergence rates.\n\nThe paper shows, through their game framework, that even faster rates can be achieved with Nesterov MD and a form of momentum SD.\n\nSTRENGTHS:\nThe paper is clear and well-written, adopting consistent, well-defined notation / objects and presenting assumptions, definitions, and results in a local and organized fashion.\n\nIts novelty and contribution seem significant:\n\nThe proposed game framework seems to be a novel approach to study implicit biases (margin / directional error rates). By reducing rate analyses to the task of finding equivalent transformations to the 2-player game, it might facilitate the study of new method’s implicit biases and hence prove useful for future research.\n\nIt provides a fresh perspective and, as shown in the paper, accommodates different forms of MD and SD, suggesting that it might be able to capture different first-order methods as well. The transformations, although technical and non-trivial, provide valuable insights on the behavior of MD and SD.\n\nThe use of clairvoyance for player w to show form equivalence is also interesting and might be useful for new proof techniques, but I cannot assess its novelty since I am not very familiar with the related work.\n\nThe improved margin rates and new accelerated methods given in Section 4.3 are also impactful and suggest that momentum might be crucial to speed up (general) margin maximization – from what I understand the community only had evidence of this for $p=2$ (Ji et al’21). \n\nFrom my understanding, the paper shows the first $1/T^2$ margin rate for $p \\in (1,2)$, which seems to be a significant contribution.\n\nWEAKNESSES:\nThe last page or two seem to have been written a bit in a hurry, and little space has been allocated to Section 4.3 compared to its significance. However, since these can be easily fixed in a revision I will not account for that when rating the paper.\n\nAlthough I believe the paper has enough contributions, it would be interesting to have some small (even if synthetic) experiments with margin by time curves for different methods and learning rates regimes.\n\nMinor fixes:\n\nL51: log t -> log T?\n\nL{276, 285, L291, L293, L295}: Algorithm 4.3 -> Algorithm 4\n\nL516 proof -> prove\n\nShould state in Section 4.3 that proof of Theorem 6 is in Appendix E\n\nEq 6 and, Eq 7: period instead of comma after equation\n\nEq 8: missing period after equation\n\nEquation at end of page 6: colon instead of period before eq, period after eq.\n\nQUESTIONS:\nIs the potential in the left box of Algorithm 4 the general norm squared? This seems to be the case but I think it is not specified in Section 4.3.\n\nLIMITATIONS:\nLimitations are properly acknowledged, e.g. applicability of the framework only to exp loss and showing method-game equivalence might take effort and be non-trivial.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-15"}

comments: SUMMARY:
This paper studies the implicit bias of the generic optimization method such as mirror descent and steepest descent. By transforming the generic optimization algorithm into an online learning dynamic, this paper shows the accelerated rate and offers a new perspective.

STRENGTHS:
1. This paper is well-written.
2. The theoretical analysis is solid and explicitly explained

WEAKNESSES:
The major weakness is that this paper only focuses on exponential loss.
In addition, choosing the hyperparameters is sometimes difficult.

QUESTIONS:
1. Can the author provide some simple experiments to verify the correctness of the theoretical results? For example, using the synthetic dataset with its max-margin solution known.
2. For the P play, will solving the subproblem lead to a larger complexity?
3. What is the meaning of $D_E$ in Algorithm 3?


Typos:
Algorithm 4.3 should be Algorithm 4.

LIMITATIONS:
OK
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a novel method to derive margin maximization and directional error rates for generic optimization methods. The method consists in finding a reformulation of the regular optimization as a minmax bilinear game. Rates can then be derived using online learning techniques. The authors demonstrate (Thm. 1) how solving a min-max regularized bilinear objective using online learning is maximizing the margin. They also derive (Thm. 2,3,5) the minmax formulation of mirror descent and steepest descent. Using their method, several new margin-maximization and directional error rates for mirror descent (average iterate) and steepest descent are derived. Moreover, rates are also derived for accelerated methods (Thm. 4 and 6).\n\nSTRENGTHS:\nOriginality & Significance: While not being entirely familiar with all the prior works, I find the main idea presented in this paper novel. The proposed rates are improving upon prior works and the approach seems to be able encompass a wide range of optimization techniques, making it a significant contribution in my opinion. \n\nClarity: I found the paper easy to read despite its density. Some small typos: in Figure 1, this should be max min and not max max.\n\nWEAKNESSES:\nThe authors ask the question whether generic optimization methods can achieve faster rates than GD. It seems the proposed rate for steepest descent is matching the GD rate, but the question is still open for mirror-descent. It is unclear how to conclude for mirror descent as you are deriving an average iterate rate that is difficult to compare to existing last iterate rates. Usually average iterate rates are better than last iterate rates. \n\nIt is unclear how good those rates can become. Is there a lower-bound you could comment on? \n\nThis work provide a unifying method to obtain margin maximization rates for linear models with exponential loss. It is unclear how the approach could be extended to different losses.\n\nQUESTIONS:\nWhat would it take to remove the $\\log(T)$ term in the $\\mathcal{O}(\\frac{\\log n \\log T}{(q-1) T})$ for mirror descent? \nFor mirror descent, what is preventing the analysis to be extended to the last iterate? \n\nWhat would a comparison between your mirror descent rate and existing GD rates look like in the average iterate regime?\n\nDo you believe a similar approach could be used beyond the exponential loss?\n\nLIMITATIONS:\nLimitations have been discussed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}