Title: Fast Last-Iterate Convergence of Learning in Games Requires Forgetful Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: Last-Iterate Convergence

Zero-Sum Games

Optimistic Multicaptive Weights Update

Optimistic Gradient
Abstract: Self play via online learning is one of the premier ways to solve large-scale zero-sum games, both in theory and practice. Particularly popular algorithms include optimistic multiplicative weights update (OMWU) and optimistic gradient-descent-ascent (OGDA). While both algorithms enjoy $O(1/T)$ ergodic convergence to Nash equilibrium in two-player zero-sum games, OMWU offers several advantages, including logarithmic dependence on the size of the payoff matrix and $\tilde{O}(1/T)$ convergence to coarse correlated equilibria even in general-sum games. However, in terms of last-iterate convergence in two-player zero-sum games, an increasingly popular topic in this area, OGDA guarantees that the duality gap shrinks at a rate of $(1/\sqrt{T})$, while the best existing last-iterate convergence for OMWU depends on some game-dependent constant that could be arbitrarily large. This begs the question: is this potentially slow last-iterate convergence an inherent disadvantage of OMWU, or is the current analysis too loose? Somewhat surprisingly, we show that the former is true. More generally, we prove that a broad class of algorithms that do not forget the past quickly all suffer the same issue: for any arbitrarily small $\delta>0$, there exists a $2\times 2$ matrix game such that the algorithm admits a constant duality gap even after $1/\delta$ rounds. This class of algorithms includes OMWU and other standard optimistic follow-the-regularized-leader algorithms.

Fast Last-Iterate Convergence of Learning in Games Requires Forgetful Algorithms

Abstract

Self-play via online learning is one of the premier ways to solve large-scale twoplayer zero-sum games, both in theory and practice. Particularly popular algorithms include optimistic multiplicative weights update (OMWU) and optimistic gradient-descent-ascent (OGDA). While both algorithms enjoy O (1 /T ) ergodic convergence to Nash equilibrium in two-player zero-sum games, OMWU offers several advantages including logarithmic dependence on the size of the payoff matrix and ˜ O (1 /T ) convergence to coarse correlated equilibria even in generalsum games. However, in terms of last-iterate convergence in two-player zero-sum games, an increasingly popular topic in this area, OGDA guarantees that the duality gap shrinks at a rate of (1 / √ T ) , while the best existing last-iterate convergence for OMWUdepends on some game-dependent constant that could be arbitrarily large. This begs the question: is this potentially slow last-iterate convergence an inherent disadvantage of OMWU, or is the current analysis too loose? Somewhat surprisingly, we show that the former is true. More generally, we prove that a broad class of algorithms that do not forget the past quickly all suffer the same issue: for any arbitrarily small δ > 0 , there exists a 2 × 2 matrix game such that the algorithm admits a constant duality gap even after 1 /δ rounds. This class of algorithms includes OMWUand other standard optimistic follow-the-regularized-leader algorithms.

1 Introduction

Self-play via online learning is one of the premier ways to solve large-scale two-player zero-sum games. Major examples include super-human AIs for Go, Poker [Brown and Sandholm, 2018], and human-level AI for Stratego [Perolat et al., 2022] and alignment of large language models [Munos et al., 2023]. In particular, Optimistic Multiplicative Weights Update (OMWU) and Optimistic Gradient Descent-Ascent (OGDA) are two of the most well-known online learning algorithms. When applied to learning a two-player zero-sum game via self-play for T rounds, the average iterates of both algorithms are known to be an O (1 /T ) -approximate Nash equilibrium [Rakhlin and Sridharan, 2013, Syrgkanis et al., 2015], while other algorithms, such as vanilla Multiplicative Weights Update

(MWU) and vanilla Gradient Descent-Ascent (GDA), have a slower ergodic convergence rate of O (1 / √ T ) .

For multiple practical reasons, there is growing interest in studying the last-iterate convergence of these learning dynamics [Daskalakis and Panageas, 2019, Golowich et al., 2020b, Wei et al., 2021, Lee et al., 2021]. In this regard, existing results seemingly exhibit a gap between OGDA and OMWU -the duality gap of the last iterate of OGDA is known to decrease at a rate of O (1 / √ T ) [Cai et al., 2022, Gorbunov et al., 2022], with no dependence on constants beyond the dimension and the smoothness of the players' utility functions of the game. 1 In contrast, the existing convergence rate for OMWU depends on some game-dependent constant that could be arbitrarily large, even after fixing the dimension and the smoothness constant of the game [Wei et al., 2021]. 2 Given the fundamental role of OMWU in online learning and its other advantages over OGDA (such as its logarithmic dependence on the number of actions), it is natural to ask the following question:

Is the potentially slow last-iterate convergence an inherent disadvantage of OMWU? (*)

Main Results. In this work, we show that the answer to this question is yes, contrary to a common belief that better analysis and better last-iterate convergence results similar to those of OGDA are possible for OMWU. More specifically, we show the following.

Theorem (Informal) . For OMWU with constant step size, there is no function f such that the corresponding learning dynamics { ( x t , y t ) } t ≥ 1 in two-player zero-sum games [0 , 1] d 1 × d 2 has a last-iterate convergence rate of f ( d 1 , d 2 , T ) . 3 More specifically, no function f can satisfy

lim T →∞ f ( d 1 , d 2 , T ) → 0 .

Our findings show that, despite the significantly superior regret properties of OMWU compared to OGDA, its last-iterate convergence properties are remarkably worse. In turn, this counters the viewpoint that 'Follow-the-Regularized-Leader (FTRL) is better than Online Mirror Descent (OMD)' [van Erven, 2021]: crucially, while OMWU is an instance of (optimistic) FTRL, OGDA is an instance of optimistic OMD that cannot be expressed in the FTRL formalism.

We further show that similar negative results extend to several other standard online learning algorithms, including a close variant of OGDA. More concretely, our main results are as follows.

We identify a broad family of Optimistic FTRL (OFTRL) algorithms that do not forget about the past quickly. We prove that, for any sufficiently small δ > 0 , there exists a 2 × 2 two-player zero-sum game such that, even after 1 /δ iterations, the duality gap of the iterate output by these algorithms is still a constant (Theorem 1). This excludes the possibility of showing a game-independent last-iterate convergence rate similar to that of OGDA.

We prove that many standard online learning algorithms, such as OFTRL with the entropy regularizer (equivalently, OMWU), the Tsallis entropy family of regularizers, the log regularizer, and the squared Euclidean norm regularizer, all fall into this family of non-forgetful algorithms and thus all suffer from the same slow convergence. Also note that Optimistic OMD (OOMD), another well-known family of algorithms, is equivalent to OFTRL when given a Legendre regularizer. Therefore, OOMD with the entropy, Tsallis entropy, and log regularizer also suffer the same issue. 4

Finally, we also generalize our negative results from 2 × 2 games to 2 n × 2 n games for any positive integer n , strengthening our message that forgetfulness is generally needed in order to achieve fast last-iterate convergence.

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

y

t

[1]

Entropy (OMWU)

10

1

10

3

Iteration

10

-

2

10

-

1

Equilibrium gap

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Log regularizer

10

1

10

3

Iteration

10

-

2

10

-

1

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sq. Euclidean norm

10

1

10

3

Iteration

10

-

2

10

-

1

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

OGDA

10

1

10

3

Iteration

10

-

4

10

-

3

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Iteration

10

-

2

10

-

1

Equilibrium gap

δ

= 0

.

05

10

0

10

1

10

2

10

3

10

4

Iteration

δ

= 0

.

01

10

0

10

1

10

2

10

3

10

4

Iteration

δ

= 0

.

005

Main Ideas. Intuitively, we trace the poor last-iterate convergence properties of OFTRL to its lack of forgetfulness . The high-level idea of our hard 2 × 2 game instance, parametrized by δ > 0 , is as follows. First, it has a unique Nash equilibrium at which one player is O ( δ ) close to the boundary of the simplex. We refer to the first row of plots in Figure 1, where the equilibrium is noted by a blue dot (note that we can plot only x [1] , y [1] for each player, since x [2] = 1 -x [1] and y [2] = 1 -y [2] ). As can be seen, the iterates of OGDA and all three OFTRL variants initially have a two-phase structure. In the first phase, they converge to the lower-right area denoted by a red star in Figure 1. Then, from there all algorithms start moving towards the equilibrium. In particular, y [1] increases. However, once they enter the vicinity of the equilibrium, the behavior depends on the algorithms. For OGDA, the dynamics start spiraling closer and closer to the equilibrium. On the other hand, for the OFTRL algorithm, the x player has built up a lot of 'memory" of x [1] being better than x [2] , and for this reason, x [1] will stay very close to 1 for a long time. During the time when x [1] is close to 1 , y [1] keeps increasing since the y player receives gradients that indicates y [1] is better than y [2] . As a

result, the dynamics cannot 'stop' near the equilibrium but start to move away from the equilibrium. The dynamics reach a point (denoted by a green square) whose duality gap is a constant and enter a new cycle where they move out towards the starting point of the learning process. This cycle repeats in smaller and smaller semi-ellipses that slowly converge to equilibrium. Note that the semi-ellipses correspond to the seesaw pattern in the equilibrium gap (second row of plots). OFTRL overshoots the equilibrium as it has built up a lot of "memory" of x [1] being better than x [2] along the phase from the red star to the blue circle, and it requires many iterations to "forget" this fact. We show that as we make δ , the parameter defining the nearness to the boundary, smaller and smaller, it takes longer and longer for these semi-ellipses to get close to the equilibrium along the entire path, as illustrated in Figure 2.

Our results are related to numerical observations made in the literature on solving large-scale extensiveform games. There, algorithms based on the regret-matching + (RM + ) algorithm [Tammelin et al., 2015], combined with the counterfactual regret minimization [Zinkevich et al., 2007], perform by far the best in practice. In contrast, the classical regret matching algorithm [Hart and Mas-Colell, 2000] performs much worse, in spite of similar regret guarantees. It was later discovered that RM + corresponds to OGD, while RM corresponds to FTRL [Farina et al., 2021, Flaspohler et al., 2021]. It was hypothesized that RM builds up too much negative regret at times, and thus is slow to adapt to changes in the learning dynamics related to the strategy of the other player. These numerical results and the hypothesis are consistent with our theoretical findings: FTRL (and thus RM) is not able to 'forget,' whereas OGD and OGDA can forget and thereby quickly adapt to changes in which actions should be played.

1.1 Related Work

The literature on last-iterate convergence of online learning methods in games is vast. In this section, we will cover key contributions focusing on the case of interest for this paper: discrete-time dynamics for two-player zero-sum normal-form games.

Convergence of OGDA. Average-iterate convergence of OGDA has been studied for minimax optimization problems in both the unconstrained [Mokhtari et al., 2020] and constrained settings [Hsieh et al., 2019]. Last-iterate convergence of OGDA in unconstrained saddle-point problems has been shown in [Daskalakis et al., 2018, Golowich et al., 2020a]. In the (constrained) game setting, Wei et al. [2021], Anagnostides et al. [2022] showed best -iterate convergence to the set of Nash equilibria in any two-player zero-sum game with payoff matrix A at a rate of O ( poly ( d 1 , d 2 , max i,j | A i,j | ) / √ T ) using constant learning rate, where d 1 and d 2 are the number of actions of the players. A stronger result was shown by Cai et al. [2022], who showed that the same rate applies to the last iterate.

Convergence of OMWU. Optimistic multiplicative weights update (also known as optimistic hedge) is often regarded as the premier algorithm for learning in games. Unlike OGDA, it guarantees sublinear regret with a logarithmic dependence on the number of actions, and it is known to guarantee only polylogarithmic regret per player when used in self play even for general-sum games [Daskalakis et al., 2021]. It can be applied with similar strong properties beyond normal-form games in several important combinatorial settings [Takimoto and Warmuth, 2003, Koolen et al., 2010, Farina et al., 2022]. The work by Daskalakis and Panageas [2019] established asymptotic last-iterate convergence for OMWU in games using a small learning rate under the assumption of a unique Nash equilibrium. Similar asymptotic results without the unique equilibrium assumption were also given by Mertikopoulos et al. [2019], Hsieh et al. [2021]. Wei et al. [2021] were the first to provide nonasymptotic learning rates for OMWU. Specifically, they showed a linear rate of convergence in games with a unique equilibrium, albeit with a dependence on a condition number-like quantity that could be arbitrarily large given fixed d 1 , d 2 , and max i,j | A i,j | .This result was later extended by Lee et al. [2021] to extensive-form games. Unlike OGDA, no last-iterate convergence result for OMWU with a polynomial dependence on only the natural parameters of the game ( i.e. , d 1 , d 2 , and max i,j | A i,j | ) is known. As we show in this paper, perhaps surprisingly, this is no coincidence: in general, OMWU does not exhibit a last-iterate convergence rate that solely depends on these parameters, whether polynomial or not.

FTRL vs. OMD. While the last-iterate convergence of instantiations of Optimistic Online Mirror Descent has been observed before, the properties of Follow-the-Regularized-Leader dynamics remain mostly elusive. The present paper partly explains this vacuum: all standard instantiations of optimistic FTRL cannot hope to converge in iterates with only a polynomial dependence on the natural parameters of the game, unlike optimistic OMD. Complications in obtaining last-iterate convergence

results for continuous-time FTRL instantiations were already reported by Vlatakis-Gkaragkounis et al. [2020], who showed the necessity of strict Nash equilibria.

Exploiting a no-regret learner. The forgetfulness property that we identify is closely related to the concept of mean-based learning algorithms from Braverman et al. [2018]. Intuitively, mean-based algorithms are ones such that if the mean reward for action a is significantly greater than the mean reward for action b , then the algorithm selects b with negligible probability. They show that MWU is mean-based, along with Follow-the-Perturbed-Leader and the Exp3 bandit algorithm. Braverman et al. [2018] shows that "mean-based" algorithms are exploitable when learning to bid in first-price auctions, whereas Kumar et al. [2024] shows that OGD does not suffer from this exploitability issue.

2 Preliminaries and Problem Setup

We consider the standard setting of no-regret learning in a zero-sum game A ∈ [0 , 1] d 1 × d 2 . In each iteration t ≥ 1 , the x -player chooses x t ∈ X := ∆ d 1 while the y -player chooses y t ∈ Y := ∆ d 2 . Then the x -player receives loss vector ℓ t x = Ay t while the y -player receives loss vector ℓ t y = -A ⊤ x t . The goal is to find or approximate a Nash equilibrium ( x ∗ , y ∗ ) to the game such that x ∗ ∈ argmin x ∈X max y ∈Y x ⊤ Ay and y ∗ ∈ argmax y ∈Y min x ∈X x ⊤ Ay . The approximation error of a strategy pair ( x, y ) is measured by its duality gap, defined as DualityGap( x, y ) = max y ′ ∈Y x ⊤ Ay ′ -min x ′ ∈X x ′ ⊤ Ay , which is always non-negative.

Popular no-regret algorithms for solving the game include the Optimistic Follow-the-RegularizedLeader (OFTRL) algorithm and the Optimistic Online Mirror Descent (OOMD) algorithm, both defined in terms of a certain regularizer R : ∆ d → R (for some general dimension d ). The corresponding Bregman divergence of R is D R ( x, x ′ ) = R ( x ) -R ( x ′ ) -⟨∇ R ( x ′ ) , x -x ′ ⟩ , and the regularizer is 1 -strongly convex if D R ( x, x ′ ) ≥ 1 2 ∥ x -x ′ ∥ 2 2 for all x, x ′ ∈ ∆ d .

Optimistic Online Mirror Descent (OOMD) Starting from an initial point ( x 1 , y 1 ) = ( ̂ x 1 , ̂ y 1 ) , the OOMD algorithm with regularizer R and steps size η > 0 updates in each iteration t ≥ 2 ,

In particular, we call OOMD with a squared Euclidean norm regularizer, that is, R ( x ) = 1 2 ∑ d i =1 x [ i ] 2 optimistic gradient-descent-ascent (OGDA). When R is the negative entropy, that is, R ( x ) = ∑ d i =1 x [ i ] log x [ i ] , we call the resulting OOMD algorithm optimistic multiplicative weights update (OMWU). OGDA and OMWU have been extensively studied in the literature regarding their lastiterate convergence properties in zero-sum games. Specifically, both OMWU and OGDA guarantee that ( x t , y t ) approaches to a Nash equilibrium as t →∞ .

Optimistic Follow-the-Regularized-Leader (OFTRL) Define the cumulative loss vectors L t x := ∑ t k =1 ℓ k x and L t y := ∑ t k =1 ℓ k y . The update rule of OFTRL with regularizer R is for each t ≥ 1 ,

Throughout the paper, we consider the following regularizers:

Negative entropy ( R ( x ) = ∑ d i =1 x [ i ] log x [ i ] ): the resulting OFTRL algorithm coincides with OMWUdefined by the OOMD framework previously.

Squared Euclidean norm ( R ( x ) = 1 2 ∑ d i =1 x [ i ] 2 ): note that the resulting algorithm is different from OGDA since the squared Euclidean norm is not a Legendre regularizer. As we will show, the two algorithms behave very differently in terms of last-iterate convergence.

Log barrier ( R ( x ) = ∑ d i =1 -log( x [ i ]) ): we also call it the log regularizer.

Negative Tsallis entropy regularizers ( R ( x ) = 1 -∑ d i =1 ( x [ i ]) β 1 -β parameterized by β ∈ (0 , 1) ).

The 2-dimension case We denote x ∈ ❘ 2 as x = [ x [1] , x [2]] ⊤ . For d 1 = 2 , finding x t of OFTRL reduces to the following 1-dimensional optimization problem:

where we slightly abuse the notation and denote R ( x ) = R ([ x, 1 -x ]) for x ∈ [0 , 1] . We introduce two notations (the case for the y -player is similar): let e t x = ℓ t x [1] -ℓ t x [2] be the difference between the losses of the two actions, and E t x = ∑ t k =1 e k x be the cumulative difference between the losses of the two actions. For OFTRL, it is clear that the update of x t only depends on the differences E t -1 x , e t -1 x , the step size η , and the regularizer R . For this reason, we define F η,R : R → [0 , 1] as follows:

We assume the function F η,R is well-defined, i.e. , the above optimization problem admits a unique solution in [0 , 1] . This is a condition easily satisfied, for example, when the regularizer R is strongly convex. Then the OFTRL algorithm can be written as

The following lemma shows that the function F η,R is non-increasing (we defer missing proofs in the section to Appendix A).

Lemma 1 (Monotonicity of F η,R ) . The function F η,R ( · ) : R → [0 , 1] defined in (1) is non-increasing.

We present some blanket assumptions on the regularizer, which are satisfied by all the regularizers introduced before.

Assumption 1. We assume that the regularizer R satisfies the following properties: the function F η,R : R → [0 , 1] defined in (1) is,

Unbiased: F η,R (0) = 1 2 .

Rational: lim E →-∞ F η,R ( E ) = 1 and lim E → + ∞ F η,R ( E ) = 0 .

Lipschitz continuous: There exists L ≥ 0 such that F 1 ,R is L -Lipschitz.

Item 1 in Assumption 1 shows that the initial strategy is the uniform distribution over the two actions, which is standard in practice. The rational assumption (item 2 in Assumption 1) is natural since otherwise, the algorithm could not even converge to a pure Nash equilibrium. The Lipschitzness (item 3 in Assumption 1) is implied when the regularizer is strongly convex over [0 , 1] 2 (see Lemma 4), and it further implies Lipschitzness of F η,R for any η as shown in the following proposition.

Proposition 1. The function F η,R satisfies F η,R ( E/η ) = F 1 ,R ( E ) . If F 1 ,R is L -Lipschitz, then F η,R is ηL -Lipschitz for any η > 0 .

3 Slow Convergence of OFTRL: A Hard Game Instance

We give negative results on the last-iterate convergence properties of OFTRL by studying its behavior on a surprisingly simple 2 × 2 two-player zero-sum games. The game's loss matrix A δ is parameterized by δ ∈ (0 , 1 2 ) and is defined as follows:

3.1 Basic Properties

We summarize some useful properties of A δ in the following proposition (missing proofs of this section can be found in Appendix B).

Proposition 2. The matrix game A δ satisfies:

A δ has a unique Nash equilibrium x ∗ = [ 1 1+ δ , δ 1+ δ ] and y ∗ = [ 1 2(1+ δ ) , 1+2 δ 2(1+ δ ) ] .

For a strategy pair ( x t , y t ) , the loss vectors (i.e., gradients) for the two players are respectively:

Moreover,

In particular, we notice that e t y ≥ -δ . It implies that if the cumulative differences between the losses of the two actions E t y is large, then it takes Ω( 1 δ ) iterations to make E t y small (close to 0 ). This has important implications for non-forgetful algorithms like OFTRL that look at the whole history of losses. Since OFTRL chooses the strategy y t based on E t y , it could be trapped in a bad action for a long time even if the current gradients suggest that the other action is better. This is the key observation for our main negative results on the slow last-iterate convergence rates of OFTRL.

The following lemma shows that in a particular region of ( x, y ) , the duality gap is a constant.

Lemma 2. Let δ, ϵ ∈ (0 , 1 2 ) . For any x, y ∈ ∆ 2 such that x [1] ≥ 1 1+ δ and y [1] ≥ 1 2 + ϵ , the duality gap of ( x, y ) for game A δ (defined in (2) ) satisfies DualityGap( x, y ) ≥ ϵ.

3.2 Slow Last-Iterate Convergence

We further require the following assumption on the regularizer R (and thus the function F 1 ,R ).

Assumption 2. Let L be the Lipschitness constant of F 1 ,R in Assumption 1. Denote constant c 1 = 1 2 -F 1 ,R ( 1 20 L ) . There exist universal constants δ ′ , c 2 > 0 and c 3 ∈ (0 , 1 2 ] such that for any 0 < δ ≤ δ ′ ,

For any E that satisfies F 1 ,R ( E ) ≥ 1 1+ δ , we have F 1 ,R ( -c 2 1 30 Lδ + E ) ≥ 1+ c 3 1+ c 3 + δ

Although Assumption 2 is technical, the idea is simple. Item 1 in Assumption 2 states that if a loss difference E < 0 already makes F 1 ,R ( E ) ≥ 1 1+ δ , then the loss difference E ′ = E -Ω( 1 δ ) is able to make F 1 ,R ( E ′ ) greater than F 1 ,R ( E ) by a margin of Ω( δ ) . Item 2 in Assumption 2 states that if a loss difference E already makes F 1 ,R ( E ) ≥ 1 2(1+ δ ) ≈ 1 2 , then the loss difference E ′ = E -Ω(1) is able to make F 1 ,R ( E ′ ) greater than 1 2 by a constant margin c 2 . In Appendix C, we verify that Assumption 2 holds for the negative entropy, squared Euclidean norm, the log barrier, and the negative Tsallis entropy regularizers.

Now we present the main result of the section showing that even after Ω(1 /δ ) iterations, the duality gap of the iterate output by OFTRL is still a constant.

Theorem 1. Assume the regularizer R satisfies Assumption 1 and Assumption 2. For any δ ∈ (0 , ˆ δ ) , where ˆ δ is a constant depending only on the constants c 1 and δ ′ defined in Assumption 2, the OFTRL dynamics on A δ (defined in (2) ) with any step size η ≤ 1 4 L satisfies the following: there exists an iteration t ≥ c 1 3 ηLδ with a duality gap of at least c 2 , a strictly positive constant defined in Assumption 2.

Proof Sketch: We decompose the analysis into three stages as illustrated in Figure 3. We describe the three stages and the high-level ideas of our proof below and defer the full proof to Appendix B.2.

y

t

[1]

x

t

[1]

0

1

1 +

δ

1

3

4

1

2(1 +

δ

)

1

2

-

c

1

2

1

2

-

c

1

1

2 +

c

2

1

z

∗

Stage I

Stage II

Ω(

1

ηδ

)

iterations

Stage III

t

=

T

s

t

=

T

h

t

=

T

1

t

=

T

2

Stage I [1 , T 1 -1] : Recall that x 1 [1] = y 1 [1] = 1 2 by Assumption 1. We show that x t [1] increases and denote T 1 the first iteration that x T 1 [1] ≥ 1 1+ δ . During the time [1 , T 1 -1] , since x t [1] is always smaller than 1 1+ δ , we know from Proposition 2 action 1 has larger loss than action 2 for the y -player, i.e., e t y = ℓ t y [1] -ℓ t y [2] ≥ 0 . Thus y t [1] decreases during stage I and we show that y T 1 [1] ≤ 1 2 -c 1 with c 1 defined in Assumption 2.

Stage II [ T 1 , T 2 ] : Recall that y T 1 [1] ≤ 1 2 -c 1 . We define T 2 > T 1 as the first iteration where y T 2 [1] ≥ 1 2(1+ δ ) > 1 2 -c 1 . We remark that for y t [1] to increase, the loss vector must satisfy e t y < 0 . However, the game matrix A δ guarantees that e t y ≥ -δ no matter what the x -player's strategy is (Proposition 2). Thus by the ηL -Lipschitzness of F η,R (Proposition 1), the per-iteration increase in y t [1] is at most ηLδ . Therefore, we know T 2 -T 1 = Ω( c 1 ηLδ ) . As a result, e t x < 0 during [ T 1 , T 2 ] and the cumulative loss for the x -player decreases to E T 2 x ≤ E T 1 x -Ω( 1 ηLδ ) . Recall x T 1 [1] ≥ 1 1+ δ . Thus x T 2 [1] > x T 1 [1] is much closer to 1 .

Stage III [ T 2 , T 3 ] : Recall that y T 2 [1] ≥ 1 2(1+ δ ) . Moreover, y t [1] could keep increasing if x t [1] ≥ 1 1+ δ since that implies e t y ≤ 0 . Now the question is how long would the x -player stay close to the boundary, i.e, x t [1] ≥ 1 1+ δ . Since OFTRL-type algorithms are not forgetful, this happens only when E t x ≥ E T 1 x (recall x T 1 [1] ≥ 1 1+ δ ). But we have at the end of stage II, E T 2 x ≤ E T 1 x -Ω( 1 ηLδ ) . Since the per-iteration loss is bounded by 1 , it requires at least Ω( 1 ηLδ ) iterations to cancel the cumulative loss of Ω( 1 ηLδ ) . Define T 3 = T 2 + Ω( 1 ηLδ ) . During [ T 2 , T 3 ] , the y -player always receives loss such that e t y ≤ 0 and we prove that in the end y T 3 [1] ≥ 1 2 + c 2 for some constant c 2 .

Conclusion: Finally, we get one iteration T 3 ≥ Ω( 1 ηLδ ) with x T 3 [1] ≥ 1 1+ δ and y T 3 [1] ≥ 1 2 + c 2 . Using Lemma 2, the duality gap of ( x T 3 , y T 3 ) is at least c 2 > 0 .

Theorem 1 immediately implies the following (proof deferred to Appendix B.3).

Theorem 2. For optimistic FTRL with any regularizer satisfying Assumption 1 and Assumption 2 and constant steps size η ≤ 1 4 L ( L is defined in Assumption 1), there is no function f such that the corresponding learning dynamics { ( x t , y t ) } t ≥ 1 in two-player zero-sum games [0 , 1] d 1 × d 2 has a last-iterate convergence rate of f ( d 1 , d 2 , T ) . More specifically, no function f can satisfy

DualityGap( x T , y T ) ≤ f ( d 1 , d 2 , T ) for all [0 , 1] d 1 × d 2 and for all T ≥ 1 .

Theorem 1 and Theorem 2 provide impossibility results for getting a last-iterate convergence rate for OFTRL that solely depends on the bounded parameters, even in two-player zero-sum games. Moreover, they show the necessity of forgetfulness for fast last-iterate convergence in games since OGDA has a last-iterate convergence rate of O ( poly( d 1 ,d 2 ) √ T ) [Cai et al., 2022, Gorbunov et al., 2022].

4 Extension to Higher Dimensions

In this section, we extend our negative results from 2 × 2 matrix games to games with higher dimensions. We start by showing an equivalence result for a single player (say, the first player). We assume that a decision maker is using OFTRL with a 1-strongly convex (w.r.t. the ℓ 2 norm) and separable regularizer R ( x ) = R 1 ( x 1 ) + R 2 ( x 2 ) to choose decisions. At a given time time t , they see a loss ℓ t ∈ [0 , 1] 2 .

Now consider the following 2 n -dimensional decision problem: The player uses OFTRL using the regularizer ˆ R (ˆ x ) = ∑ n i =1 R 1 (ˆ x i ) + ∑ 2 n i = n +1 R 2 (ˆ x i ) , i.e. , they use R 1 on the first half of actions, and R 2 on the second half. This is again a 1-strongly convex regularizer (w.r.t. the ℓ 2 norm). Suppose the decision maker sees the rescaled and duplicated version of the losses ℓ 1 , . . . , ℓ T from the 2dimensional case: ˆ ℓ t i = 1 n α ℓ t 1 if i ≤ n , and ˆ ℓ t i = 1 n α ℓ t 2 if i > n . The parameter α will be chosen later based on the regularizer.

Now we wish to show that by choosing α in the right way, we get that the decisions for the 2 -dimensional and 2 n -dimensional OFTRL algorithms are equivalent. Let x 1 , . . . , x T be the 2dimensional OFTRL decisions, and let ˆ x 1 , . . . , ˆ x T be the 2 n -dimensional OFTRL decisions. Then, we want to show that ∑ n i =1 ˆ x t i = x t [1] and ∑ 2 n i = n +1 ˆ x t i = x t [2] for all t .

Lemma 3. Let the losses ˆ ℓ 1 , . . . , ˆ ℓ T satisfy the duplication procedure given in the preceding paragraph. Then for any time t , we have ˆ x t 1 = · · · = ˆ x t n and ˆ x t n +1 = · · · = ˆ x t 2 n .

̸

Proof. Suppose not and let ˆ x t be the corresponding solution. Then the optimal solution is such that ˆ x t i = ˆ x t k for some i, k both less than n , or both greater than n . But then, by symmetry, we have that there is more than one optimal solution to the OFTRL optimization problem at time t : the objective is exactly the same if we create a new solution where we swap the values of ˆ x t i and ˆ x t k . This is a contradiction due to strong convexity.

From lemma 3, we have that the OFTRL decision problem in 2 n dimensions can equivalently be written as a 2 -dimensional decision problem: Since the first n entries must be the same, we can simply optimize over that one shared value, say x t [1] , which we use for all n entries, and similarly we use x t [2] for the second half of the entries. Let Dupl : ∆ 2 → ∆ 2 n be a function that maps the two-dimensional solution into the corresponding duplicated 2 n -dimensional solution. The equivalent 2 -dimensional problem is then:

The next theorem shows that we can choose α for different regularizers and construct 2 n × 2 n loss matrices whose learning dynamics are equivalent to the learning dynamics in 2 × 2 games given in the preceding sections. We defer the proof to Appendix D.

Theorem 3. For any loss matrix A ∈ [0 , 1] 2 × 2 , there exists a loss matrix ˆ A ∈ [0 , n -α ] 2 n × 2 n such that for the Euclidean ( α = 1 ), entropy ( α = 0 ), Tsallis ( β ∈ (0 , 1) and α = -1 + β ), and log ( α = -1 ) regularizers, the resulting OFTRL learning dynamics are equivalent in the two games.

Combining Theorem 1 and Theorem 3, we have the following:

Corollary 1. In the same setup as Theorem 3, under Assumption 1 and Assumption 2, there exists a game matrix ˆ A δ ∈ [0 , n -α ] 2 n × 2 n such that the OFTRL learning dynamics with any step size η ≤ 1 4 L satisfies the following: there exists an iteration t ≥ c 1 3 ηLδ with a duality gap at least c 2 n -α .

Since α = 0 for the entropy regularizer, the same results hold more generally for games where one player has more actions than the other. In particular, we can create a 2 n × 2 m game such that the resulting dynamics are equivalent to those in a 2 × 2 game. This does not work for the Euclidean and log regularizers because the rescaling factors would be different for the row and column players.

5 Conclusion and Discussions

In this paper, we study last-iterate convergence rates of OFTRL algorithms with various popular regularizers, including the popular OMWU algorithm. Our main results show that even in simple 2 × 2 two-player zero-sum games parametrized by δ > 0 , the lack of forgetfulness of OFTRL leads to the duality gap remaining constant even after 1 /δ iterations (Theorem 1). As a corollary, we show that the last-iterate convergence rate of OFTRL must depend on a problem-dependent constant that can be arbitrarily bad (Theorem 2). This highlights a stark contrast with OOMD algorithms: while OGDA with constant step size achieves a O ( 1 √ T ) last-iterate convergence rate, such a guarantee is impossible for OMWU or more generally OFTRL. We now discuss several interesting questions regarding the convergence guarantees of learning in games and leave them as future directions.

Best-Iterate Convergence Rates While we focus on the last-iterate ( i.e. , DualityGap( x T , y T ) ), the weaker notion of best-iterate ( i.e. , min t ∈ [ T ] DualityGap( x t , y t ) ) is also of both practical and theoretical interest. By definition, we know the best-iterate convergence rate is at least as good as the last-iterate convergence rate and could be much faster. This raises the following question:

What is the best-iterate convergence rate of OMWU/OFTRL?

To our knowledge, there are no concrete results on the best-iterate convergence rates of OMWU or other OFTRL algorithms. It is thus interesting to extend our negative results to the best-iterate convergence rates or develop fast best-iterate convergence rates of OMWU/OFTRL.

Dynamic Step Sizes Our negative results hold for OFTRL with fixed step sizes. We conjecture that the slow last-iterate convergence of OFTRL persists even with dynamic step sizes. In particular, we believe our counterexamples still work for OFTRL with decreasing step sizes. This is because decreasing the step size makes the players move even slower, and they may be trapped in the wrong direction for a longer time due to the lack of forgetfulness. In Appendix E, we include numerical results for OMWU with adaptive stepsize akin to Adagrad [Duchi et al., 2011], which supports our intuition. We observe the same cycling behavior as for fixed step size. While the cycle is smaller than that of fixed step sizes, the dynamics take more steps to finish each cycle. Investigating the effect of dynamic step sizes on last-iterate convergence rates is an interesting future direction.

Slow Convergence due to Lack of Forgetfulness Our work shows that various OFTRL-type algorithms do not have fast last-iterate convergence rates for learning in games. Our proof and hard game instance build on the intuition that these algorithms lack forgetfulness: they do not forget the past quickly. This intuition is also utilized in [Panageas et al., 2023]. In particular, they give an d × d potential game where the last-iterate convergence rate of the Fictitious Play algorithm, which is equivalent to the Follow-the-Leader (FTL) algorithm, suffers exponential dependence in the dimension d . One natural future direction is to formalize the intuition of non-forgetfulness further and give a general condition for algorithms under which they suffer slow last-iterate convergence. It is also interesting to show other lower-bound results for learning in games.

References

Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On last-iterate convergence beyond zero-sum games. In International Conference on Machine Learning , pages 536-581. PMLR, 2022.

James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 321-338, 2018.

Mark Braverman, Jieming Mao, Jon Schneider, and Matt Weinberg. Selling to a no-regret buyer. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 523-538, 2018.

Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Yun Kuen Cheung and Georgios Piliouras. Vortices instead of equilibria in minmax optimization: Chaos and butterfly effects of online learning in zero-sum games. In Conference on Learning Theory , pages 807-834. PMLR, 2019.

Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. Advances in neural information processing systems (NeurIPS) , 2018.

Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In 10th Innovations in Theoretical Computer Science Conference (ITCS) , 2019.

Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. In International Conference on Learning Representations (ICLR) , 2018.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 5363-5371, 2021.

Gabriele Farina, Chung-Wei Lee, Haipeng Luo, and Christian Kroer. Kernelized multiplicative weights for 0/1-polyhedral games: Bridging the gap between learning in extensive-form and normal-form games. In International Conference on Machine Learning (ICML) , pages 6337-6357, 2022.

Genevieve E Flaspohler, Francesco Orabona, Judah Cohen, Soukayna Mouatadid, Miruna Oprescu, Paulo Orenstein, and Lester Mackey. Online learning with optimism and delay. In International Conference on Machine Learning , pages 3363-3373. PMLR, 2021.

Noah Golowich, Sarath Pattathil, and Constantinos Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games. Advances in neural information processing systems (NeurIPS) , 2020a.

Noah Golowich, Sarath Pattathil, Constantinos Daskalakis, and Asuman Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. In Conference on Learning Theory (COLT) , 2020b.

Eduard Gorbunov, Adrien Taylor, and Gauthier Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In Advances in Neural Information Processing Systems , 2022.

Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68(5):1127-1150, 2000.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. Advances in Neural Information Processing Systems , 32, 2019.

Yu-Guan Hsieh, Kimon Antonakopoulos, and Panayotis Mertikopoulos. Adaptive learning in continuous games: Optimal regret bounds and convergence to nash equilibrium. In Conference on Learning Theory , pages 2388-2422. PMLR, 2021.

Wouter M Koolen, Manfred K Warmuth, Jyrki Kivinen, et al. Hedging structured concepts. In COLT , pages 93-105. Citeseer, 2010.

Rachitesh Kumar, Jon Schneider, and Balasubramanian Sivan. Strategically-robust learning algorithms for bidding in first-price auctions. In Proceedings of the 2024 ACM Conference on Economics and Computation , 2024.

Chung-Wei Lee, Christian Kroer, and Haipeng Luo. Last-iterate convergence in extensive-form games. Advances in Neural Information Processing Systems , 34:14293-14305, 2021.

Haipeng Luo. Lecture note 2, Introduction to Online Learning. 2022. URL https://haipeng-luo. net/courses/CSCI659/2022_fall/lectures/lecture2.pdf .

Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the twenty-ninth annual ACM-SIAM symposium on discrete algorithms , pages 2703-2717. SIAM, 2018.

Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In International Conference on Learning Representations (ICLR) , 2019.

Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of O (1 /k ) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization , 30(4):3230-3251, 2020.

Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886 , 2023.

Ioannis Panageas, Nikolas Patris, Stratis Skoulakis, and V olkan Cevher. Exponential lower bounds for fictitious play in potential games. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=tkenkPYkxj .

Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science , 378(6623):990-996, 2022.

Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems , 2013.

Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. Fast convergence of regularized learning in games. Advances in Neural Information Processing Systems (NeurIPS) , 2015.

Eiji Takimoto and Manfred K Warmuth. Path kernels and multiplicative updates. The Journal of Machine Learning Research , 4:773-818, 2003.

Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit texas hold'em. In Twenty-fourth international joint conference on artificial intelligence , 2015.

Tim van Erven. Why FTRL is better than online mirror descent. https://www.timvanerven.nl/ blog/ftrl-vs-omd/ , 2021. Accessed: 2024-05-22.

Emmanouil-Vasileios Vlatakis-Gkaragkounis, Lampros Flokas, Thanasis Lianeas, Panayotis Mertikopoulos, and Georgios Piliouras. No-regret learning and mixed nash equilibria: They do not mix. Advances in Neural Information Processing Systems , 33:1380-1391, 2020.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations (ICLR) , 2021.

Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systems , 20, 2007.

Contents

D Proof of Theorem 3

E Numerical Experiments with Adaptive Stepsizes

A Missing Proofs in Section 2

A.1 Proof of Lemma 1

Proof. Let e 1 < e 2 . Denote x 1 = F η,R ( e 1 ) and x 2 = F η,R ( e 2 ) . By definition, we have

Since e 1 < e 2 , we have x 2 ≤ x 1

A.2 Proof of Proposition 1

Proof. By definition,

B Missing Proofs in Section 3

B.1 Proof of Lemma 2

Proof. We have

B.2 Proof of Theorem 1

Proof. Recall that c 1 = 1 2 -F 1 ,R ( 1 20 L ) defined in Assumption 2. We fix any δ < min { 1 15 , c 1 6 , c 2 1 300 , δ ′ } . Since δ < δ ′ , Assumption 2 holds. We will prove that there exists an iteration t ≥ c 1 3 ηLδ with duality gap c 2 .

Proof Plan: We decompose the analysis into three stages as shown in Figure 3. Below, we describe the three stages and the high-level ideas in our proof.

Stage I: Recall that x 1 [1] = y 1 [1] = 1 2 . In Stage I, we show that x t [1] will increase and denote T 1 ≥ 1 the first iteration where x t [1] ≥ 1 1+ δ . The existence of T 1 can be proved by contradiction (Claim 1). Since before the end of Stage I, x t [1] < 1 1+ δ , the loss vector for the y -player satisfies e t y = ℓ t y [1] -ℓ t y [2] ≥ 0 meaning action 1 is worse than action 2 . We will prove that finally y T 1 [1] ≤ 1 2 -c 1 .

22

22

Stage II: Now we have that y T 1 [1] ≤ 1 2 -c 1 , we denote T 2 > T 1 the first iteration where y T 2 [1] ≥ 1 2(1+ δ ) > 1 2 -c 1 . We remark that in order to increase y t [1] , the loss vector must satisfy e t y < 0 . However, the game matrix A δ guarantees that e t y ≥ -δ no matter what the x -player is playing. Thus by the ηL -Lipschitzness of F η,R (Lemma 4), the increase in y t [1] is at most ηLδ . Therefore, we know T 2 -T 1 = Ω( c 1 ηLδ ) . But during [ T 1 , T 2 ] , for the x -player, we have e t x < 0 which implies its cumulative loss E T 2 x ≤ E T 1 x -Ω( 1 ηLδ ) . In other words, x t [1] is very close to 1 and the cumulative loss for action 1 is much smaller than that of action 2.

Stage III: Now we have y T 2 [1] ≥ 1 2(1+ δ ) and that y t [1] could keep increasing if x t [1] ≥ 1 1+ δ since then the loss satisfies e t y ≤ 0 . Now the question is how long would the x -player stay close to the boundary, i.e, x t [1] ≥ 1 1+ δ . Since OFTRL-type algorithms are not forgetful, this happens only when E t x ≥ E T 1 x (recall x T 1 [1] ≥ 1 1+ δ ). But we have at the end of stage II, E T 2 x ≤ E T 1 x -Ω( 1 ηLδ ) . Since e t x is bounded by a constant, we know x t [1] ≥ 1 1+ δ even after Ω( 1 ηLδ ) iterations. Define T 3 = T 2 +Ω( 1 ηLδ ) . During [ T 2 , T 3 ] , the y -player always receives loss such that e t y ≤ 0 and we prove that y T 3 [1] ≥ 1 2 + c 2 for some constant c 2 .

Conclusion Finally we get one iteration T 3 ≥ Ω( 1 ηLδ ) with x T 3 [1] ≥ 1 1+ δ and y T 3 [1] ≥ 1 2 + c 2 , Using Lemma 2, the duality gap of ( x T 3 , y T 3 ) is at least c 2 .

Stage I: We know x 1 [1] = y 1 [1] = 1 2 . We define (i) T s > 1 to be the smallest iteration such that x T s [1] ≥ 3 4 and (ii) T 1 > T s to be the smallest iteration such that x T 1 [1] ≥ 1 1+ δ . Both T s and T 1 must exist, and the reason will become clear in the following analysis. We postpone the proof of this fact in Claim 1 at the end of this paragraph.

Notice from Proposition 2, the difference e t x is lower bounded: e t x ≥ -1 2 for any t . Thus E t -1 x + e t -1 x ≥ -t 2 for any t ≥ 1 . Since x T s [1] ≥ 3 4 > 1 2 , we know that E T s -1 x + e T s -1 x < 0 . As F η,R is ηL -Lipschitz,

This implies

Since x t [1] < 3 4 for all 1 ≤ t ≤ T s -1 , we know that e t y = ℓ t y [1] -ℓ t y [2] = 1 -(1 + δ ) x t [1] > 1 -3 δ 4 ≥ 1 5 (as δ ≤ 1 15 ) for all 1 ≤ t ≤ T s -1 . Moreover, for all 1 ≤ t ≤ T 1 -1 , we know that e t y ≥ 0 as x t [1] ≤ 1 1+ δ . Since the difference e t y is at least 1 / 5 for all t ≤ T s -1 and remains non-negative for all t ∈ [ T s , T 1 -1] , we can conclude that for all T s ≤ t ≤ T 1

and moreover

This completes the proof of Stage I, where x T 1 [1] ≥ 1 1+ δ and y T 1 [1] ≤ 1 2 -c 1 . Before we proceed to the next stage, we prove the existence of T s and T 1 .

Claim 1. T s and T 1 exist.

Proof. It suffices to prove that T 1 exists as it implies the existence of T s . Assume for the sake of contradiction that T 1 does not exist, i.e. , x t [1] < 1 1+ δ for all t ≥ 1 . By the same analysis as for Stage I, we get y t [1] ≤ 1 2 -c 1 for all t ≥ 1 2 ηL . This implies e t x = -1 2 +(1+ δ ) y t [1] ≤ δ 2 -c 1 ≤ -c 1 2 for

all t ≥ 1 2 ηL . Then E t x + e t x →-∞ as t → + ∞ . As a consequence, x t [1] = F η,R ( E t -1 x + e t -1 x ) → 1 as t → + ∞ by item 2 in Assumption 1. But this contradicts with the assumption that x t [1] < 1 1+ δ for all t ≥ 1 . This completes the proof.

Stage II We define

where the lower bound on T holds since c 1 6 Lηδ ≥ c 1 6 δ ≥ 1 . We note that T = Ω( 1 δ ) since ηL ≤ 1 4 .

In Stage I, we have proved that y T 1 [1] ≤ 1 2 -c 1 . Define T h = T 1 + T . We claim that for all t ∈ [ T 1 , T h -1] , y t [1] ≤ 1 2 -c 1 2 . To prove the claim, we first notice that -δ ≤ e t y ≤ 1 for all t ≥ 1 . Then by the monotonicity and the ηL -Lipschitzness of F η,R (Lemma 1 and Lemma 4), we get for all t ∈ [ T 1 , T h -1] ,

where, in the second-to-last inequality, we use t -T 1 +1 ≤ T ≤ c 1 2 ηLδ by Equation (4).

Now we denote T 2 ≥ T h the smallest iteration when y T 2 [1] ≥ 1 2(1+ δ ) . The existence of T 2 will become clear in the following analysis, and we postpone the proof to Claim 2 at the end of the discussion. Then for all t ∈ [ T s , T 2 -1] , we have y t [1] ≤ 1 2(1+ δ ) , which implies e t x ≤ 0 . Moreover, for all

Then for any T + T ≤ t ≤ T , we have

where in the last inequality, we use the fact that c 1 T 20 ≥ c 2 1 60 ηLδ ≥ 1 .

Claim 2. T 2 exists.

Proof. Assume for the sake of contradiction that T 2 does not exist, i.e. , y t [1] < 1 2(1+ δ ) for all t ≥ T 1 (since we know y t [1] ≤ 1 2 -c 1 2 for all t ∈ [ T 1 , T 1 + T -1] ). Then by the analysis of Stage II and Equation (5), we have x t [1] ≥ 4 4+ δ for all t ≥ T 1 . This implies e t y ≤ -3 δ 5 for all t ≥ T 1 . As a result, we have E t -1 y + e t -1 y → -∞ as t → ∞ . By item 2 in Assumption 1, we get y t [1] = F η,R ( E t -1 y + e t -1 y ) ≥ 1 2 as t → ∞ . But this contradicts with the assumption that y t [1] < 1 2(1+ δ ) for all t ≥ T 1 . This completes the proof.

Stage III Recall that we have argued in State I that F η,R ( E T 1 -1 x + e T 1 -1 x ) = F 1 ,R ( η ( E T 1 -1 x + e T 1 -1 x )) = x T 1 [1] ≥ 1 1+ δ . By item 1 in Assumption 2, we have that

where the first inequality follows from the definition of T and the monotonicity of F η,R (Lemma 1). Now denote T 3 = T 2 + ⌊ c 1 T 10 ⌋ -2 . For any T 2 ≤ t ≤ T 3 , we know that

Note that 1+ c 3 + δ ≤ 2 . This implies e t y = 1 -(1+ δ ) x t [1] = -c 3 δ 1+ c 3 + δ ≤ -c 3 δ 2 for all T 2 ≤ t ≤ T 3 . Moreover, we know that e t y ≥ -δ for any t . Then

Recall that F 1 ,R ( η ( E T 2 -1 y + e T 2 -1 y )) = F η,R ( E T 2 -1 y + e T 2 -1 y ) = y T 2 [1] ≥ 1 2(1+ δ ) . By item 2 in Assumption 2, we have F 1 ,R ( η ( E T 2 -1 y + e T 2 -1 y ) -c 2 1 120 L + δ 4 L ) ≥ 1 2 + c 2 for some absolute constant c 2 > 0 . Thus, we have y T 3 [1] ≥ 1 2 + c 2 . Recall that x T 3 [1] ≥ 1+ c 3 1+ c 3 + δ ≥ 1 1+ δ . Then by Lemma 2 we can conclude that the duality gap of ( x T 3 , y T 3 ) is at least c 2 > 0 . This completes the proof as T 3 ≥ T 2 ≥ T ≥ c 1 3 ηLδ .

B.3 Proof of Theorem 2

Proof. Assume for the sake of contradiction that there is a function that satisfies both conditions. Then for any A ∈ [0 , 1] 2 × 2 , we have the OFTRL learning dynamics over A satisfies

DualityGap( x T , y T ) ≤ f (2 , 2 , T ) for all T .

lim T →∞ f (2 , 2 , T ) → 0

Since lim T →∞ f (2 , 2 , T ) → 0 , we know there exists T 0 > 0 such that for any t ≥ T 0 , DualityGap( x t , y t ) ≤ f (2 , 2 , t ) < c 2 . Now let δ ≤ min { ˆ δ, c 1 3 ηLT 0 } . Then by Theorem 1, we know there exists an iteration t ≥ c 1 3 ηLδ ≥ T 0 such that DualityGap( x t , y t ) ≥ c 2 . This completes the proof.

C Verifying Assumption 2 for Different Regularizers

Lemma 4. If the regularizer R is 1 -strongly convex, then F 1 ,R is 1 2 -Lipschitz.

Proof. Notice that R ( x ) + R (1 -x ) is 2 -strongly convex. Thus by standard analysis (see e.g., Luo [2022, Lemma 4]) we know F 1 ,R is 1 2 -Lipschitz.

By Lemma 4, we can choose L = 1 2 for any 1 -strongly convex regularizer in Assumption 1.

C.1 Negative Entropy

Lemma5 (Assumption 2 holds for the entropy regularizer) . Consider the negative entropy regularizer R defined as R ( x ) = x log x +(1 -x ) log(1 -x ) . Then F 1 ,R is L = 1 2 -Lipschitz. We have c 1 and Assumption 2 holds with δ ′ = c 2 1 480 L , c 2 = F 1 ,R ( -c 2 1 480 L ) -1 2 , and c 3 = 1 2 .

Proof. It is easy to verify that F 1 ,R ( x ) has a closed-form representation

Thus L = 1 2 and c 1 = 1 2 -F 1 ,R ( 1 20 L ) is a universal constant. We also choose c 3 = 1 2 .

1 1

If F 1 ,R ( E ) ≥ 1+ δ ≥ 1+ δ , then we have E ≤ -log(1 /δ ) . We note that

Thus δ ≤ δ 1 = c 2 1 30 L log(1+ c 3 )) = c 2 1 30 log( 3 2 )) L suffices for item 1 in Assumption 2.

If F 1 ,R ( E ) ≥ 1 2(1+ δ ) = 1 1+1+2 δ , we have E ≤ log(1 + 2 δ ) . Note that since log(1 + 2 y ) ≤ 2 y for y > 0 , we have

Thus item 2 in Assumption 2 holds for any δ ≤ δ 2 = c 3 c 2 1 480 L = c 2 1 960 L and c 2 = F 1 ,R ( -c 3 c 2 1 240 L ) -1 2 = F 1 ,R ( -c 2 1 480 L ) -1 2 .

Combining the above, we know Assumption 2 holds for the negative entropy regularizer with δ ′ = c 2 1 960 L and c 2 = F 1 ,R ( -c 2 1 480 L ) -1 2 .

C.2 Squared Euclidean Norm Regularizer

Lemma 6 (Assumption 2 holds for the Euclidean regularizer) . Consider the Euclidean regularizer R defined as R ( x ) = 1 2 ( x 2 +(1 -x ) 2 ) . We have L = 1 2 and c 1 = 1 20 . We also have Assumption 2 holds with δ ′ = c 2 1 , c 2 = c 2 1 , and c 3 = 1 .

Proof. It is easy to verify that F 1 ,R ( x ) has a closed-form representation

Thus F 1 ,R is L -Lipschitz with L = 1 2 . Moreover, c 1 = 1 2 -F 1 ,R ( 1 20 L ) = 1 20 . We choose c 3 = 1 2 .

Fix any E such that F 1 ,R ( E ) ≥ 1 1+ δ . We have E ≤ -1 -δ 1+ δ < 0 . We note that for any δ ≤ c 2 1 30 L = c 2 1 15 ,

Thus δ ≤ δ 1 = c 2 1 30 L suffices for item 1 in Assumption 2.

Fix any E such that F 1 ,R ( E ) ≥ 1 2(1+ δ ) = 1 2(1+ δ ) . We have E ≤ δ 1+ δ ≤ δ . The for any δ ≤ c 3 c 2 1 240 L , we have

Thus item 2 in Assumption 2 holds for any δ ≤ δ 2 = c 2 1 480 L and c 2 = c 2 1 960 L .

Combining the above, we know Assumption 2 holds for the negative entropy regularizer with δ ′ = min { δ 1 , δ 2 } = c 2 1 480 L and c 2 = c 2 1 960 L .

C.3 Log Barrier

Lemma 7 (Assumption 2 holds for the log barrier) . Consider the log barrier regularizer R defined as R ( x ) = -log( x ) -log(1 -x ) . Then Assumption 2 holds with the following choices of constants:

Proof. By setting the gradient of x · E + R ( x ) to 0 , we get a closed-form expression of F 1 ,R :

For x ∈ (0 , 1) , the F 1 ,R function admits an inverse function defined as

Thus we know E 0 := F -1 1 ,R ( 1 1+ δ ) = -1 -δ 2 δ satisfies F 1 ,R ( E 0 ) = 1 1+ δ . Moreover, we can calculate

Thus we can choose c 3 = c 2 1 60 L so that

Thus we have F 1 ,R ( E 0 -c 2 1 30 Lδ ) ≥ F 1 ,R ( E 0 -c 3 δ -c 3 δ 1+ c 3 ) ≥ 1+ c 3 1+ c 3 + δ .

We calculate E 1 := F -1 1 ,R ( 1 2(1+ δ ) ) = 4( δ + δ 2 ) 1+2 δ ≤ 8 δ . Then we can choose δ ≤ δ ′ := c 3 c 2 1 2160 L . Then we have

where c 2 = √ 1 4 +( c 3 c 2 1 240 L ) 2 -c 3 c 2 1 240 L > 0 by the closed-form expression of F 1 ,R .

C.4 Negative Tsallis Entropy

For x ∈ [0 , 1] , the negative Tsallis entropy is a family of regularizers parameterized by β ∈ (0 , 1) :

The corresponding F 1 ,R is defined as

For x ∈ (0 , 1) , we note that F 1 ,R has an inverse function

Lemma 8 (Assumption 2 holds for Tsallis entropy) . Consider Tsallis entropy parameterized by β ∈ (0 , 1) . Then L = 1 2 β and Assumption 2 holds with the following choices of constants:

Proof. We choose c 3 = 1 2 . We have c 1 = 1 2 -F 1 ,R ( 1 20 L ) is a constant. We note that

satisfies F 1 ,R ( E 0 ) = 1 1+ δ . Similarly, we calculate

where in the first inequality we use the fact that (1 + δ ) 1 -β ≤ 2 since δ ≤ 1 ; the second inequality we use the inequality ( x + y ) 1 -β ≤ x 1 -β + y 1 -β . We note that

Thus for any δ ≤ δ 1 , we have for any E such that F 1 ,R ( E ) ≥ 1 1+ δ ,

The above implies F 1 ,R ( -c 2 1 30 Lδ + E ) ≥ 1+ c 3 1+ c 3 + δ and the first item in Assumption 2 is satisfied. We define E 2

where in the first inequality we use (2 + 2 δ ) 1 -β ≤ 4 since 0 ≤ δ ≤ 1 and β ∈ (0 , 1) ; in the second inequality we use the basic inequality (1 -x ) r ≤ 1 -x for r, x ∈ (0 , 1) . We define

Then for any δ ≤ δ 2 and E such that F 1 ,R [ E ] ≥ 1 2(1+ δ ) , we have

Thus we know F 1 ,R ( -c 3 c 2 1 120 L + δ 4 L + E ) ≥ F 1 ,R ( -c 3 c 2 1 240 L ) and item 2 in Assumption 2 is satisfied by c 2 = F 1 ,R ( -c 3 c 2 1 240 L ) -1 2 > 0 .

Combining the above, we can choose ˆ δ = min { δ 1 , δ 2 } so that both items in Assumption 2 hold for δ ≤ ˆ δ .

D Proof of Theorem 3

Recall the equivalent 2 -dimensional problem:

Euclidean regularizer : this regularizer is homogeneous of degree two. Choosing α = 1 , the inner minimization problem is exactly the same as the one solved by OFTRL in two dimensions.

Entropy regularizer : we set α = 0 to get equivalence:

Now we have equivalence because the last term is a constant that does not affect the argmin .

Log regularizer: we set α = -1 to get equivalence, using similar logic as for entropy:

Tsallis entropy regularizer: we set α = -1 + β to get equivalence, using similar logic as for entropy:

E Numerical Experiments with Adaptive Stepsizes

In this section we present our numerical results when OFTRL and OOMD are instantiated with adaptive stepsize [Duchi et al., 2011]: η t = 1 / √ ϵ + ∑ t -1 k =1 ∥ ℓ k ∥ 2 k with some constant ϵ > 0 . We present our numerical experiments in Figure 4, where we choose ϵ = 0 . 1 .

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

y

t

[1]

Entropy (OMWU)

10

2

10

5

Iteration

10

-

3

10

-

2

10

-

1

Equilibrium gap

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Log regularizer

10

2

10

5

Iteration

10

-

3

10

-

2

10

-

1

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sq. Euclidean norm

10

2

10

5

Iteration

10

-

3

10

-

2

10

-

1

0

.

6

0

.

8

1

.

0

x

t

[1]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

OGDA

10

1

10

3

Iteration

10

-

4

10

-

3

10

-

2

10

-

1

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction match both the theoretical and experimental results of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The extension to the general m × n game does not work for the Euclidean and log regularizers because the rescaling factors would be different for the row and column players.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 2, Section 3 and the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Figure 1 and Python codes in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Python code in the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Python code in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: There is no randomness in our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our simulations were done within a few hours on an average consumer laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics. The research conducted in this paper conforms with it in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is mostly theoretical.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present work studies the last iterate convergence of learning algorithms in zero sum games. Existing results have shown that although OMWU and OGDA both enjoy the $O(1/T)$ ergodic convergence rate, their best existing last iterate convergence rates exhibit a nontrivial gap. This paper constructs a particular hard example showing that such a gap is due to a fundamental limitation of the OMWU algorithm, rather than the sub-optimality of existing analysis. From this hard example, the key insight is that fast last iterate convergence requires learning algorithms to forget the past.\n\nSTRENGTHS:\nThis paper is very cool. Last iterate convergence is an important problem for learning in games, and this paper naturally motivates it by the existing theoretical and empirical gap between OMWU and ODGA. To tackle this problem, the paper then presents a particularly nontrivial analysis on the learning dynamics of Optimistic FTRL in games, despite the appeared simplicity of the constructed example. The takeaway message on the importance of forgetting is elegant, which might inspire future works on algorithm design. I also appreciate the graphical illustration on the learning dynamics, which makes the very technical proof intuitively easier to understand. Related works seem to be thoroughly discussed. \n\nFrankly speaking, the analysis in this paper is quite involved, so I only have a high level understanding of the proof while struggling on the details. However, as far as I can see, the high level intuition makes sense, and the numerical experiments are quite helpful.\n\nWEAKNESSES:\nThis paper is already strong in my opinion, so I don't have any major complaint. One possible aspect to improve is the presentation. There are certain parts in the analysis that are particularly involved, so it might be better to write less quantitatively in the main paper, and leave those parts to the appendix. For example, Assumption 2 and the discussion right after it. I would also appreciate a further simplified proof sketch, specifically, more words and less math.\n\nQUESTIONS:\nSome minor comments: \n\n- In line 51-51 it is quoted from (van Erven, 2021) that FTRL is conventionally believed to be better than OMD. From the perspective of online convex optimization, I feel this is actually a bit more nuanced, as the examples I'm aware of (where FTRL beats OMD) concern time varying learning rates, which is different from the constant learning rates considered in this paper. \n\n- The wording of Theorem 2 could be improved.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the last-iterate convergence properties of several classes of algorithms employed in zero-sum matrix games. As a core contribution, it is shown that a large class of iteration rules can not exhibit instance-independent convergence in zero-sum games of dimension 2. The result is further generalized to larger dimensions.\n\nSTRENGTHS:\nThe paper tackles an important problem and makes a potentially significant contribution to our understanding of learning dynamics in zero-sum matrix games.\n\nI think the results are rigorously developed, and while I did not verify the details of all the proofs, they seem to be correct.\n\nWEAKNESSES:\nThe concept of a \"forgetful algorithm\" could be made more explicit, perhaps as a definition. It is not clear to me what property makes an algorithm forgetful by construction. An explicit definition (if it is possible at all) could greatly clarify the work.\n\nAs a possible weakness, the stated negative results apply to a particular class of algorithms (OFTRL) and potentially might be sidestepped.\n\nThe formulation of the main theorems could be clarified or perhaps simplified. For instance, theorem 2 has a rather complicated statement, which to my understanding just means either the duality gap can not go to zero (in which case it can be stated as such directly), or the duality gap can not go to zero uniformly (in which case item 1 should have a statement \"over all possible loss matrices\").\n\nI think one weakness of the paper is the presentation/organization. I found it difficult to follow the main ideas and the main contribution of the paper. For instance, the generalization to higher dimensions, while interesting, seems to be a minor contribution from a theoretical perspective, while significant space is dedicated to its proof ideas.\n\nQUESTIONS:\nIs it possible to extract a \"forgetfulness\" property for an arbitrary learning algorithm that must imply the algorithm has slow last iterate convergence?\n\nAre there significant differences between Theorems 1 and 2 to warrant stating them separately? Upon first read, it was difficult to understand what the main difference was, and Theorem 2 seems to be a corollary of Theorem 1 looking at the proof in the appendix.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows a limitation of optimistic multiplicative weights (OMWU) with a fixed learning rate: the last-iterate convergence rate can be arbitrarily large, depending on a game-dependent constant in normal form games. This demonstrates that the current upper bounds on convergence are not loose and that there is a real barrier to this dynamic.\n\nBroader Context:\nOptimistic online learning algorithms play an important role in solving games. The average iterates of OMWU have been shown to converge in zero-sum games at a rate of 1/T, which is better than the non-optimistic counterpart. Another important algorithm in this context is the online gradient descent ascent (OGDA), which has been shown to have last-iterate convergence of O(1/sqrt{T}), where the constant hides polynomial dependence on the problem parameters. Last-iterate convergence is a desirable property in the context of games. Since OMWU is a central tool for solving games, understanding its limitations is a fundamental question.\n\nThis paper reveals an inherent limitation of OMWU (and more generally, for some algorithms within the optimistic FTRL framework), showing that OGDA might sometimes enjoy better last-iterate convergence. The main property responsible for this limitation is the \"lack of forgetfulness.\"\n\nSTRENGTHS:\nThe paper answers a fundamental question in the \"learning in games\" community (solving games using online learning dynamics). \nI like also the presentation of the paper. For example, the authors made an effort to explain the intuition behind the construction of the hard game.\n\nWEAKNESSES:\nMy only criticism is the following:\nIt might be the case that OMWU can have good last-iterate convergence when using changing learning rates. These are sometimes used to avoid such an issue of forgetfulness.\n\n(But I believe that analyzing fixed learning rate is a significant contribution on its own)\n\nQUESTIONS:\nDo you think using time-dependent learning rates might improve the last-iterate convergence of OMWU?\nIf so, maybe it's worth discussing it or writing it as a limitation.\n\nLIMITATIONS:\nSee the above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The authors, through some hard instances of games study the fundamental differences in convergence of broadly two classes of algorithms which are OOMD and OFTRL. Particular algorithms of interest include the OGDA and the OMWU algorithm respectively. They show that OFTRL (in particular OMWU) necessarily exhibits slow last-iterate convergence and this leads to it having an undesirable dependence on the underlying game.

STRENGTHS:
This paper seems to capture an important property that differentiates the likes of algorithms such as OGDA and OMWU, two extremely popular optimistic algorithms known to achieve last-iterate convergence in zero-sum games. Now this paper mounts further evidence of ``slowness" of OWMU and that it is unavoidable to have game dependent constants appearing in the guarantees! I think this is an important finding for this area.

WEAKNESSES:
This is a focused study on showing certain properties of a class of algorithms such as OFTRL and OOMD. In general it is not clear if some of the results apply to every regularizer satisfying the assumptions or to the particular ones of interest such as Euclidean, Entropic etc. Beyond a few clarifications which I have asked in the next section, I do not think there are major weaknesses.

QUESTIONS:
1) Please clarify whether Assumptions 1 and 2 are satisfied by the specific regularizers such as entropic, euclidean and log or general regularizers that are 1-strongly convex wrt to $l_2$ norm.
2) Does the hard example still hold with vanishing step sizes?
3) Is the high-dimensional example applicable only for the specific regularizers mentioned.

LIMITATIONS:
yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-14