Title: Regret Matching+: (In)Stability and Fast Convergence in Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Regret Matching

Predictive algorithms

Extensive-Form Games
Abstract: Regret Matching$^+$ (RM$^+$) and its variants are important algorithms for solving large-scale games.
However, a theoretical understanding of their success in practice is still a mystery.
Moreover, recent advances on fast convergence in games are limited to no-regret algorithms such as online mirror descent, which satisfy stability.
In this paper, we first give counterexamples showing that RM+ and its predictive version can be unstable, which might cause other players to suffer large regret. 
We then provide two fixes: restarting and chopping off the positive orthant that RM$^+$ works in.
We show that these fixes are sufficient to get $O(T^{1/4})$ individual regret and $O(1)$ social regret in normal-form games via RM$^+$ with predictions.
We also apply our stabilizing techniques to clairvoyant updates in the uncoupled learning setting for RM$^+$ and prove desirable results akin to recent works for Clairvoyant online mirror descent. 
Our experiments show the advantages of our algorithms over vanilla RM$^+$-based algorithms in matrix and extensive-form games.

Regret Matching + : (In)Stability and Fast Convergence in Games

Abstract

Regret Matching + (RM + ) and its variants are important algorithms for solving large-scale games [35]. However, a theoretical understanding of their success in practice is still a mystery. Moreover, recent advances [34] on fast convergence in games are limited to no-regret algorithms such as online mirror descent, which satisfy stability . In this paper, we first give counterexamples showing that RM + and its predictive version [12] can be unstable, which might cause other players to suffer large regret. We then provide two fixes: restarting and chopping off the positive orthant that RM + operates in. Combined with RM + with predictions, we show that restarting is sufficient to get O ( T 1 / 4 ) individual regret and that chopping off achieves O (1) social regret in normal-form games. We also apply our stabilizing techniques to clairvoyant updates in the uncoupled learning setting for RM + , introduced Extragradient RM + , and prove desirable results akin to recent works for Clairvoyant online mirror descent [31, 14]. Our experiments show the advantages of our algorithms over vanilla RM + -based algorithms in matrix and extensive-form games.

1 Introduction

Regret minimization is an important framework for solving games. Its connection to game theory provides a practically efficient way to approximate game-theoretic equilibria [16, 19]. Moreover, it provides a scaleable way to solve large-scale sequential games, for example using the Counterfactual Regret Minimization (CFR) decomposition [37]. Consequently, regret minimization algorithms are a central component in recent superhuman poker AIs [2, 28, 3]. Regret Matching + (RM + ) [35] is the most prevalent regret minimizer in these applications. In theory, it guarantees an O (1 / √ T ) convergence rate after T iterations, but its practical performance is usually significantly faster.

On the other hand, a line of recent works show that regret minimizers based on follow the regularized leader (FTRL) or online mirror descent (OMD) enjoy faster convergence rates in theory when combined with the concept of optimism/predictiveness [32, 34]. The result was originally proven

Julien Grand-Cl´ ement

Chung-Wei Lee

in matrix games [32], and later extended to multiplayer normal-form games [34, 6, 7], extensiveform games [8, 10, 15, 1], and general convex games [22, 13]. However, despite their favorable properties in theory, optimistic algorithms based on FTRL/OMD are usually numerically inferior to RM + when applied to solving large-scale sequential games. It remains a mystery whether some optimistic variant of RM + enjoys a theoretically faster convergence rate, considering the strong empirical performance of RM + . It is also an open question whether there exists an algorithm that has both favorable theoretical guarantees similar to FTRL/OMD algorithms and practical performance comparable to RM + . Inspired by recent work on the connection between OMD and RM + [12], we provide new insights on the theoretical and empirical behavior of RM + -based algorithms, and we show that the analysis of fast convergence for OMD can be extended to RM + with some simple modifications to the algorithm. Specifically, our main contributions can be summarized as follows.

We provide a detailed theoretical and empirical analysis of the potential for slow performance of RM + and predictive RM + . We start by showing that, in stark contrast to FTRL/OMD algorithms that are stable inherently, there exist loss sequences that make RM + and its variants unstable, leading to cycling between very different strategies. The key reason for such instability is that the decisions of these algorithms are chosen by normalizing an aggregate payoff vector ; thus, in a region close to the origin, two consecutive aggregate payoffs may point in very different directions, despite being close, resulting in unstable iterations. Surprisingly, note that this can only happen when the aggregate payoff vectors, which essentially measure the algorithm's regret against each action, are small, so instability can only happen when one's regret is small and thus is seemingly not an issue. However, in a game setting, such instability might cause other players to suffer large regret because they have to learn in an unpredictable environment. Indeed, we identify a 3 × 3 matrix game where this is the case and both RM + and predictive RM + converge slowly at a rate of O (1 / √ T ) (Fig. 1). We emphasize that very little is known about the properties of (predictive) RM + and we are the first to show concrete examples of stability issues in matrix games and in the adversarial setting.

Motivated by our counterexamples, we propose two methods to stabilize RM + : restarting , which reinitializes the algorithms when the aggregate payoffs are all below a threshold, and chopping off the origin from the nonnegative orthant to smooth the algorithms. When applying these techniques to online learning with RM + , we show improved regret and fast convergence similar to predictive OMD: we obtain O ( T 1 / 4 ) individual regrets for Stable Predictive RM + (which uses restarting) and O (1) social regret for Smooth Predictive RM + (which chops off the origin). We also consider conceptual prox and extragradient versions of RM + for normal-form games. We show that our stabilizing ideas also provide the required stability in these settings and thus give strong theoretical guarantees: Conceptual RM + achieves O (1) individual regrets (Theorem 5.3) while Extragradient RM + achieves O (1) social regret (Theorem 5.6). See Table 1 for a summary of our results for normal-form games. We further extend Conceptual RM + to extensive-form games (EFG), yielding O (1) regret in T iterations with O ( T log( T )) gradient computation. The key step here is to show the Lipschitzness of the CFR decomposition (Lemma J.1).

We apply our algorithms to solve matrix games and EFGs. For the 3 × 3 matrix game instability counterexample, our algorithms indeed perform significantly better than (predictive)

RM + . For random matrix games, we find that Stable and Smooth Predictive RM + have very strong empirical performance, on par with (unstabilized) Predictive RM + , and greatly outperforming RM + in all our experiments; Extragradient RM + appears to be more sensitive to the choice of step sizes and sometimes performs only as well as RM + . Our experiments on 4 different EFGs show that our implementation of clairvoyant CFR outperforms predictive CFR in some, but not all, instances.

2 Preliminaries

Notations. For d ∈ N , we write 1 d ∈ R d the vector with 1 on every component. The simplex of dimension d -1 is ∆ d = { x ∈ R d + | ⟨ x , 1 d ⟩ = 1 } . The vector 0 has 0 on every component and its dimension is implicit. For x ∈ R , we write [ x ] + for the positive part of x : [ x ] + = max { 0 , x } , and we overload this notation to vectors component-wise. For two vectors a and b , a ≥ b means a is at least b component-wise. We write ∥ · ∥ ∗ for the dual norm of a norm ∥ · ∥ .

Online Linear Minimization. In online linear minimization, at every decision period t ≥ 1 , an algorithm chooses a decision x t from a convex decision set X . Aloss vector ℓ t is chosen arbitrarily and an instantaneous loss of ⟨ ℓ t , x t ⟩ is incurred. The regret of an algorithm generating the sequence of decisions x 1 , ..., x T is defined as the difference between the cumulative loss generated and that of any fixed strategy ˆ x ∈ X : Reg T (ˆ x ) = ∑ T t =1 ⟨ ℓ t , x t -ˆ x ⟩ . A regret minimizer guarantees that Reg T (ˆ x ) = o ( T ) for any ˆ x ∈ X .

Online Mirror Descent. A famous regret minimizer is Online Mirror Descent (OMD) [30], which generates the decisions x 1 , ..., x T as follows (with a learning rate η > 0 ):

where for any x ∈ X , and any loss ℓ , the proximal operator ℓ ↦→ Π x , X ( ℓ ) is defined as Π x , X ( ℓ ) = arg min ˆ x ∈X ⟨ ℓ , ˆ x ⟩ + D (ˆ x , x ) where D is the Bregman divergence associated with φ : X → R , a 1 -strongly convex regularizer (with respect to some norm ∥ · ∥ ): D (ˆ x , x ) = φ (ˆ x ) -φ ( x ) -⟨∇ φ ( x ) , ˆ x -x ⟩ , ∀ ˆ x , x ∈ X . OMD guarantees that the worst-case regret against any ˆ x grows as O ( √ T ) (omitting other dependence for simplicity; the same below). Other popular regret minimizers include Follow-The-Regularized-Leader (FTRL), and adaptive variants of OMD and FTRL; we refer the reader to [20] for an extensive survey on regret minimizers.

Regret Matching and Regret Matching + . Regret Matching (RM) and Regret Matching + (RM + ) are two regret minimizers that achieve O ( √ T ) worst-case regret when X = ∆ d . RM[19] maintains a sequence of aggregate payoffs ( R t ) t ≥ 1 : R 1 = R 0 1 d , and for t ≥ 1 ,

where R 0 ≥ 0 specifies an initial point and 0 / 0 is defined as the uniform distribution for convenience. The original RM sets R 0 = 0 , making the algorithm completely parameter-free , a very appealing property in practice. RM + is a simple variation of RM, where the aggregate payoffs are thresholded at every iteration [35]. In particular, RM + only keeps track of the non-negative components of the aggregate payoffs to compute a decision: R 1 = R 0 1 d , and for t ≥ 1 ,

We highlight that very little is known about the theoretical properties of RM + , despite its strong empirical performances: [36] show that RM + is a regret minimizer (and enjoys the stronger K -tracking regret property), and [4] show that it can safely be combined with alternation ([18] prove strict improvement when using alternation). Farina et al. [12] show an interesting connection between RM + and Online Mirror Descent: the update R t +1 = [ R t + ⟨ x t , ℓ t ⟩ 1 d -ℓ t ] + of RM + can be rewritten as

for X = R d + , φ = 1 2 ∥ · ∥ 2 2 , η = 1 , and f ( x t , ℓ t ) defined as f ( x t , ℓ t ) = ℓ t -⟨ x t , ℓ t ⟩ 1 d . Therefore, RM + generating a sequence of decisions x 1 , ..., x T facing a sequence of losses ( ℓ t ) t ≥ 1 , is closely connected to OMD instantiated with the non-negative orthant as the decision set and facing a sequence of losses ( f ( x t , ℓ t )) t ≥ 1 . We have the following relation for the regret in x 1 , ..., x T and the regret in R 1 , ..., R T (the proof follows [12] and is deferred to the appendix).

Lemma 2.1. Let x 1 , ..., x T ∈ ∆ d be generated as x t = R t / ∥ R t ∥ 1 for some sequence R 1 , ..., R T ∈ R d + . The regret Reg T (ˆ x ) of x 1 , ..., x T facing a sequence of losses ℓ 1 , ..., ℓ T is equal to Reg T ( ˆ R ) , the regret of R 1 , ..., R T facing the sequence of losses f ( x 1 , ℓ 1 ) , ..., f ( x T , ℓ T ) , compared against ˆ R = ˆ x : Reg T ( ˆ R ) = ∑ T t =1 〈 f ( x t , ℓ t ) , R t -ˆ R 〉 .

Since OMD is a regret minimizer guaranteeing Reg T ( ˆ R ) = O ( √ T ) , Lemma 2.1 directly shows that RM + is also a regret minimizer: Reg T (ˆ x ) = O ( √ T ) .

Multiplayer Normal-Form Games. In a multiplayer normal-form game, there are n ∈ N players. Each player i has d i strategies and their decision space ∆ d i is the probability simplex over the d i strategies. We denote ∆ = × n i =1 ∆ d i as the joint decision space of all players and d = d 1 + · · · + d n . The utility function for player i is a concave function u i : ∆ → [ -1 , 1] that maps every joint strategy profile x = ( x 1 , ..., x n ) ∈ ∆ to a payoff. We assume bounded gradients and L u -smoothness for the utilities of the players: there exists B u > 0 , L u > 0 such that for any x , x ′ ∈ ∆ and any player i ,

The function mapping joint strategies to negative payoff gradients for all players is a vector-valued function G : ∆ → R d such that G ( x ) = ( -∇ x 1 u 1 ( x ) , . . . , -∇ x n u n ( x )) . It is well known that running a regret minimizer for ( x t 1 , ..., x t n ) ∈ ∆ = × n i =1 ∆ d i facing the loss G ( x t ) = ( ℓ t 1 , . . . , ℓ t n ) leads to strong game-theoretic guarantees (e.g., the average iterate being an approximate coarse correlated equilibrium). However, in light of Lemma 2.1, we will instead perform regret minimization on ( R t 1 , ..., R t n ) ∈ X = × n i =1 R d i + with the losses ( f ( x t 1 , ℓ t 1 ) , . . . , f ( x t n , ℓ t n )) . For conciseness, we thus define the operator F : X → R d as, for z = ( R 1 , ..., R n ) , F ( z ) = ( f ( x 1 , ℓ 1 ) , ..., f ( x n , ℓ n )) where x i = R i / ∥ R i ∥ 1 , ∀ i = 1 , ..., n, ( ℓ i ) i ∈ [ n ] = G ( x ) .

Predictive OMD and Its RVU Bounds. The predictive version of OMD proceeds as follows:

When setting m t = ℓ t -1 , predictive OMD satisfies Reg T (ˆ x ) ≤ D (ˆ x , ˜ x 1 ) η + η ∑ T t =1 ∥ ℓ t -ℓ t -1 ∥ 2 ∗ -

1 8 η ∑ T -1 t =1 ∥ x t +1 -x t ∥ 2 . This regret bound satisfies the RVU (regret bounded by variation in utilities) condition, introduced in [34]. The authors show that this type of bound guarantees that the social regret (i.e., sum of the regrets of all players) is O (1) when all players apply this special instance of predictive OMD. Syrgkanis et al. [34] further prove that each player has improved O ( T 1 / 4 ) individual regret by the stability of predictive OMD. Specifically, they show that predictive OMD guarantees ∥ x t +1 -x t ∥ = O ( η ) against any adversarial loss sequence, i.e., the algorithm is stable in the sense that the change in the iterates can be controlled by choosing η appropriately.

Predictive RM + Similar to OMD, we can generalize RM + to Predictive Regret Matching + [12]: define R 1 = m 1 = R 0 1 d (with R 0 = 0 by default), and for t ≥ 1 ,

We call the algorithm predictive RM + (PRM + ) when m t = -f ( x t -1 , ℓ t -1 ) , and it recovers RM + when m t = 0 . Aregret bound with a similar RVU condition is attainable for predictive RM + by its connection to predictive OMD [12], but only in the non-negative orthant space instead of the actual strategy space. To make a connection between them, stability is required as we show later. A natural question is then whether (predictive) RM + is also always stable. We show that the answer is no by giving an adversarial example in the next section.

3 Instability of (Predictive) Regret Matching +

We start by showing that there exist adversarial loss sequences that lead to instability for both RM + and predictive RM + . Our construction starts with an unbounded loss sequence ℓ t so that x t alternates between (1 / 2 , 1 / 2) and (0 , 1) : we set ℓ t = ( ℓ t , 0) , where ℓ 1 = 2 , and for t ≥ 2 , ℓ t = -2 ( t -2) / 2 if t is even and ℓ t = 2 ( t -1) / 2 if t is odd. Our proof is completed by normalizing the losses to [ -1 , 1] given a fixed time horizon (see Appendix B for details).

0

25

50

75

100

Last 100 iterations (out ot

10

7

)

0.00

0.02

0.04

0.06

Norm

||

x t

-

x t

+1

||

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

0

2

4

Regret

Regret (x-player)

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

1

10

-

2

10

-

3

10

-

4

Duality gap

RM

+

linear fit

0

25

50

75

100

Last 100 iterations (out ot

10

7

)

10

-

6

10

-

5

Norm

||

y t

-

y t

+1

||

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

0

10

1

10

2

Regret

Regret (y-player)

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

1

10

-

2

10

-

3

10

-

4

Duality gap

PRM

+

linear fit

Theorem 3.1. There exist finite sequences of losses in R 2 for RM + and its predictive version such that x t = ( 1 2 , 1 2 ) when t is odd and x t = (0 , 1) when t is even.

This is in stark contrast to OMD which always ensures ∥ x t +1 -x t ∥ = O ( η ) and is thus inherently stable. However, a somewhat surprising property about (predictive) RM + is that instability actually implies low regret . To see this, we first present the following Lipschitz property of the normalization function g : x ↦→ x / ∥ x ∥ 1 for x ∈ R d + .

This proposition shows that the normalization step has a reasonable Lipschitz constant ( √ d ) as long as its input is not too close to the origin, which further implies the following corollary.

Put differently, the corollary states that instability can happen only when the cumulative regret vector R t is small. For example, if ∥ x t +1 -x t ∥ = Ω(1) , then we must have ∥ R t ∥ 1 = O ( dB u ) and thus the regret at that point is at most O ( dB u ) . A similar argument holds for predictive RM + as well. Therefore, instability is in fact not an issue for these algorithms' own regret.

However, when using these algorithms to play a game, what could happen is that such instability leads to other players learning in an unpredictable environment with large regret. We show this phenomenon via an example of a 3 × 3 matrix game max x ∈ ∆(3) min y ∈ ∆(3) ⟨ x , Ay ⟩ , where A = ((3 , 0 , -3) , (0 , 3 , -4) , (0 , 0 , 1)) . The first column of Fig. 1 shows the squared ℓ 2 norm of the consecutive difference of the last 100 iterates of Predictive RM + for the x player (top) and the y player (bottom). The iterates of the x player are rapidly changing in a periodic fashion while the iterates of the y player are stable with changes on the order of 10 -5 . In the center plots where we show the individual regret for each player, we indeed observe that the cumulative regret of the x player is near zero as implied by instability, but it causes large regret (close to T 0 . 5 empirically) for the y player. (We show the same plots for RM + in Fig. 4 in Appendix B; there, the iterates of both players are stable, but since RM + lacks predictivity, it still leads to larger regret for one player.)

The right column of Fig. 1 shows the duality gap achieved by the linear average (¯ x t , ¯ y t ) = ( 2 T ( T +1) ∑ T t =1 t x t , 2 T ( T +1) ∑ T t =1 t y t ) , when the iterates are generated by RM + with alternation (top) and predictive RM + (bottom). For both algorithms the convergence rate slows down around 10 4 iterations. A linear regression estimate on the rate for the last 10 6 iterates shows rates of -0 . 497 and -0 . 496 for RM + and predictive RM + respectively. To the best of our knowledge, this is the

first known case of empirical convergence rates on the order of T -0 . 5 for either RM + or predictive RM + ; the worst prior instance for RM + was T -0 . 74 in Farina et al. [10]; no hard instance was known for predictive RM + .

4 Stabilizing RM + and Predictive RM + .

Based on the discussions in the previous section, we aim to make every player stable despite the fact that being an unstable player may actually be good for that particular player. By Corollary 3.2, it suffices to make sure that ∥ R t ∥ 1 is never too small. We provide two approaches to ensure this property and thereby stabilize (predictive) RM + .

Stable Predictive RM + . One way to maintain the required distance to the origin is via restarting : Weinitialize the algorithm with the cumulative regret vector equal to some non-zero amount, instead of the usual initialization at zero. Then, when the cumulative regret vector gets below the initialization point, we restart the algorithm from the initialization point. Applying this idea to predictive RM + yields Algorithm 1. Player i starts with R 1 i = R 0 1 d i , runs predictive RM + , and restarts whenever R t i ≤ R 0 1 d i . In the algorithm we write ( R t 1 , ..., R t n ) compactly as w t (similarly for z t ). Note, though, that the updates are decentralized for each player, as in vanilla predictive RM + .

Given this modification, Stable PRM + achieves improved individual regret in multiplayer games, as stated in Theorem 4.1. We defer the proof to the appendix. One key step in the analysis is to note that by definition, the regret against any action is negative when the restarting event happens, so it is sufficient to consider the regret starting from the last restart. Thanks to the stability enforced by the restarts, the regret from the last restart is also well controlled and the results follow by tuning η and R 0 optimally. In fact, since the algorithm is scale-invariant up to the relative scale of the two parameters, it is without loss of generality to always set R 0 = 1 .

Algorithm 1 Stable Predictive RM +

1:

Input

:

R

0

>

0

, step size

η >

0

2:

Initialization

:

w

0

=

R

0

1

d

3:

for

t

= 1

, . . . , T

do

4:

z

t

= Π

w

t

-

1

,

X

(

η

m

t

)

5:

w

t

= Π

w

t

-

1

,

X

(

ηF

(

z

t

))

6:

(

x

t

1

, . . . ,

x

t

n

) = (

g

(

z

t

1

)

, . . . ,

g

(

z

t

n

))

7:

for

i

= 1

, ..., n

do

8:

if

w

t

i

R

0

1

d

i

then

9:

w

t

i

=

R

0

1

d

i

Algorithm 2 Smooth Predictive RM +

1:

Input

: Step size

η >

0

2:

Initialization

:

w

0

∈ X

3:

for

t

= 1

, . . . , T

do

4:

z

t

= Π

w

t

-

1

,

X

(

η

m

t

)

5:

w

t

= Π

w

t

-

1

,

X

(

ηF

(

z

t

))

6:

(

x

t

1

, . . . ,

x

t

n

) = (

g

(

z

t

1

)

, . . . ,

g

(

z

t

n

))

Theorem 4.1. Let η = ( d 2 T ) -1 / 4 and R 0 = 1 . Let ( f t i ) i ∈ [ n ] = F ( z t ) for t ≥ 1 . For each player i , set the sequence of predictions m t i = 0 when t = 0 or restart happens at t -1 ; otherwise, m t i = f t -1 i , ∀ t ≥ 1 . Then Algorithm 1 guarantees that the individual regret Reg T i (ˆ x i ) = ∑ T t =1 ⟨∇ x i u i ( x t ) , ˆ x i -x t i ⟩ of each player i is bounded by O ( d 3 / 2 T 1 / 4 ) in multiplayer normal-form games.

Although the restarting idea successfully stabilizes the RM + algorithm, the discontinuity created by asynchronous restarts causes technical difficulty for bounding the social regret by O (1) . Next we introduce an alternative stabilization idea to fix this issue.

Smooth Predictive RM + . Our second stabilization idea is to restrict the decision space to a subset where we 'chop off' the area that is too close to the origin, that is, project the vector R t i onto the set ∆ d i ≥ = { R ∈ R d i + | ∥ R ∥ 1 ≥ 1 } . Wedenote the joint chopped-off decision space as X ≥ = × n i =1 ∆ d i ≥ . We call the resulting algorithm smooth predictive RM + (Algorithm 2). Besides a similar result to Theorem 4.1 on the individual regret (omitted for simplicity), Algorithm 2 also guarantees that the social regret is bounded by a game-dependent constant, as shown in Theorem 4.2.

Theorem 4.2. Let η = ( 2 √ 2( n -1) max i { d 3 / 2 i } ) -1 . Using the sequence of predictions m 0 = 0 , m t = F ( z t -1 ) , ∀ t ≥ 1 , Algorithm 2 guarantees that the so-

cial regret ∑ n i =1 Reg T i (ˆ x i ) = ∑ n i =1 ∑ T t =1 ⟨∇ x i u i ( x t ) , ˆ x i -x t i ⟩ is upper bounded by O ( n 2 max i =1 ,...,n { d 3 / 2 i } max i =1 ,...,n {∥ w 0 i -ˆ x i ∥ 2 2 ) in multiplayer normal-form games.

Algorithm 2 dominates Algorithm 1 in terms of our theoretical results so far, but it has one drawback: it requires occasional projection onto X ≥ . In Appendix K we show that this can be done with a sorting trick in O ( d log d ) time, whereas the restarting procedure is implementable in linear time.

5 Conceptual Regret Matching +

In this section, we depart from the predictive OMD framework and develop new smooth variants of RM + from a different angle. Instead of using predictive OMD to compute the iterates ( R t i ) t ≥ 1 , we consider the following regret minimizer that we call cheating OMD , defined for some arbitrary closed decision set Z and an arbitrary sequence of losses ( ℓ t ) t ≥ 1 : z t = Π z t -1 , Z ( η ℓ t ) for t ≥ 1 , and z 0 ∈ X ≥ . Cheating OMD is inspired by the Conceptual Prox method for solving variational inequalities associated with monotone operators [5, 23, 29]. We call it cheating OMD because at iteration t , the decision z t is chosen as a function of the current loss ℓ t , which is revealed after the decision z t has been chosen. It is well-known that cheating OMD yields a sequence of decisions with constant regret; we show it for our setting in the following lemma.

To instantiate RM + with Cheating OMD as a regret minimizer for the sequence ( R t i ) t ≥ 1 of each player i , we need to show the existence of a vector z t ∈ X ≥ such that

Equation (2) can be interpreted as a fixed-point equation for the map z ↦→ Π z t -1 , X ≥ ( ηF ( z )) . For any z ′ ∈ X ≥ , the map z ↦→ Π z ′ , X ≥ ( ηF ( z )) is ηL -Lipschitz continuous as long as F is L -Lipschitz continuous. Therefore, it is a contraction when η < 1 /L , and then the fixed-point equation z = Π z ′ , X ≥ ( ηF ( z )) has a unique solution. Recall that for z = ( R 1 , ..., R n ) ∈ X ≥ , the operator F is defined as F ( z ) = ( f ( x 1 , ℓ 1 ) , . . . , f ( x n , ℓ n )) where x i = g ( R i ) and ℓ i = -∇ x i u i ( x ) , for all i ∈ { 1 , ..., n } . We now show the Lipschitzness of F over X ≥ for normal-form games.

Lemma 5.2. For a normal-form game, the operator F is L F -Lipschitz continuous over X ≥ , with L F = (max i d i ) √ 2 B 2 u +4 L 2 u with B u , L u defined in (1) .

For L F defined as in Lemma 5.2 and η < 1 /L F , the existence of the fixed-point z t = Π z t -1 , X ≥ ( ηF ( z t )) is guaranteed. This yields Conceptual RM + , defined in Algorithm 3. In the following theorem, we show that Conceptual RM + ensures constant regret for each player.

Theorem 5.3. Let L F > 0 be defined as in Lemma 5.2. For η < 1 /L F , Algorithm 3 guarantees that the individual regret Reg T i (ˆ x i ) = ∑ T t =1 ⟨∇ x i u i ( x t ) , ˆ x i -x t i ⟩ of each player i is bounded by 1 2 η ∥ z 0 i -ˆ x i ∥ 2 2 in multiplayer normal-form games.

Note that the requirement of η < 1 /L F in Theorem 5.3 and Algorithm 3 is only needed in order to ensure existence of a fixed-point. If the fixed-point condition holds for some larger η , then the algorithm is still well-defined and the same convergence guarantee holds.

Remark 5.4. Piliouras et al. [31] propose the clairvoyant multiplicate weights updates (MWU) algorithm, based on the classical MWU algorithm, but where the rescaling at iteration t involves the payoff of the players at iteration t . The connection with the conceptual prox method is made explicit by [14], where they show how to extend clairvoyant MWU for normal-form games to clairvoyant OMD for general convex games. Our algorithm uses the same idea but for RM + .

For z ′ ∈ X ≥ , we can approximate the fixed-point of z ↦→ Π z ′ , X ≥ ( ηF ( z )) by performing k ∈ N fixed-point iterations. This results in Algorithm 4. We give the guarantees for Algorithm 4 below.

Theorem 5.5. Let L F > 0 be defined as in Lemma 5.2 and η < 1 /L F . Assume that in Algorithm 4, we ensure ∥ w k -Π z t -1 , X ≥ ( ηF ( w k ) ) ∥ 2 ≤ ϵ ( t ) , for all t ≥ 1 . Then Algorithm 4 guarantees that the individual regret Reg T i (ˆ x i ) = ∑ T t =1 ⟨∇ x i u i ( x t ) , ˆ x i -x t i ⟩ of each player i is bounded by 1 2 η ∥ z 0 i -ˆ x i ∥ 2 2 +2 B u √ d i ∑ T t =1 ϵ ( t ) in multiplayer normal-form games.

By Theorem 5.5, if we ensure error ϵ ( t ) = 1 /t 2 in Algorithm 4 then the individual regret of each player is bounded by a constant. Since w ↦→ Π z t -1 , X ≥ ( ηF ( w )) is a contraction for η < 1 /L F , this only requires k = O (log( t )) fixed-point iterations at each time t . If the number of iterations T is known in advance, we can choose k = O (log( T )) , to ensure ϵ ( t ) = O (1 /T ) and therefore that the individual regret of each player i is bounded by the constant 1 2 η ∥ z 0 i -ˆ x i ∥ 2 2 + O ( 2 B u √ d i ) .

Recall that the uniform distribution over a sequence of strategy profiles { x t } T t =1 is a (max i Reg T i ) /T -approximate coarse correlated equilibrium (CCE) of a multiplayer normal-form game (see e.g. Theorem 2.4 in Piliouras et al. [31]). Therefore, Algorithm 3 guarantees O (1 /T ) convergence to a CCE after T iterations. With the setup from Theorem 5.5 and k = O (log( T )) , Algorithm 4 guarantees O (log( T ) /T ) convergence to a CCE after T evaluations of the operator F .

Extragradient RM + . We now consider the case of Algorithm 4 but with only one fixedpoint iteration ( k = 1 ). This is similar to the mirror prox algorithm [29] or the extragradient method [24]. We call this algorithm extragradient RM + (ExRM + , Algorithm 5). We show that one fixed-point iteration ( k = 1 ) at every iteration ensures constant social regret.

Algorithm 5 Extragradient RM + (ExRM + )

1: Input : Step size η > 0 with η < 1 /L F

2: Initialization : z 0 ∈ X ≥

3: for t = 1 , . . . , T do

4: w t = Π z t -1 , X ≥ ( ηF ( z t -1 ) )

5: z t = Π z t -1 , X ≥ ( ηF ( w t ))

6: ( x t 1 , . . . , x t n ) = ( g ( w t 1 ) , . . . , g ( w t n ))

Theorem 5.6. Define L F as in Lemma 5.2 and let η = ( √ 2 L F ) -1 . Algorithm 5 guarantees that the social regret ∑ n i =1 Reg T i (ˆ x i ) = ∑ n i =1 ∑ T t =1 ⟨∇ x i u i ( x t ) , ˆ x i -x t i ⟩ is bounded by 1 2 η ∑ n i =1 ∥ z 0 i -ˆ x i ∥ 2 2 in multiplayer normal-form games.

We now apply Theorem 5.6 to the case of matrix games, where the goal is to solve

for A d 1 × d 2 . The operator F is defined as

and X ≥ = ∆ d 1 ≥ × ∆ d 2 ≥ . The next lemma gives the Lipschitz constant of the operator F in the case of matrix games.

̸

Lemma 5.7. For matrix games, the operator F is L F -Lipschitz over X ≥ , with L F = √ 6 ∥ A ∥ op max { d 1 , d 2 } with ∥ A ∥ op = sup {∥ Av ∥ 2 / ∥ v ∥ 2 | v ∈ R d 2 , v = 0 } .

Combining Lemma 5.7 with Theorem 5.6, ExRM + for matrix games with X ≥ as a decision set and η = ( √ 2 L F ) -1 guarantees constant social regret, so that the average of the iterates computed by ExRM + converges to a Nash Equilibrium at a rate of O (1 /T ) [16].

Extensive-form games Our convergence results for Conceptual RM + apply beyond normal-form games, to EFGs. Briefly, a EFG is a game played on a tree, where each node belongs to some player, and the player chooses a probability distribution over branches. Moreover, players have information sets , which are groups of nodes belonging to a player such that they cannot distinguish among them, and thus they must choose the same probability distribution at all nodes in an information set. As is standard, we assume that each player never forgets information. Below, we describe the main ideas behind the extension; details are given in Appendix J.

In order to extend our results, we use the CFR regret decomposition [37, 9]. CFR defines a notion of local regret at each information set, using so-called counterfactual values . By minimizing the regret incurred at each information set with respect to counterfactual values, CFR guarantees that the overall regret over tree-form strategies is minimized. Importantly, counterfactual values are multilinear in the strategies of the players, and therefore they are Lipschitz functions of the strategies of the other players. Hence, using Algorithm 4 at each information set with counterfactual value and applying Theorem 5.5 begets a smooth-RM + -based algorithm that computes a sequence of iterates with regret at most ϵ in at O (1 /ϵ ) iterations and using O (log(1 /ϵ ) /ϵ ) gradient computations.

6 Numerical experiments

Matrix games. We compute the performance of ExRM + , Stable and Smooth PRM + on the 3 × 3 matrix game instance from Section 2 (with step size η = 0 . 1 ) and on 30 random matrix games of size ( d 1 , d 2 ) = (30 , 40) with normally distributed coefficients of the payoff matrix and with step sizes η ∈ { 0 . 1 , 1 , 10 } . We initialize our algorithms at (1 /d 1 ) 1 d , all algorithms use linear averaging, and all algorithms (except ExRM + ) use alternation. The results are shown in Figure 2. Our new algorithms greatly outperform RM + and PRM + in the 3 × 3 matrix game; linear regression finds an asymptotic convergence rate of O (1 /T 2 ) . More detailed results for this instance are given in Appendix K.1. For random matrix games, our algorithms ExRM + , Smooth PRM + and Stable PRM + all outperform RM + for stepsize η = 0 . 1 . ExRM + performs on par with RM + for larger values of η , while Stable PRM + and Smooth PRM + remain very competitive, performing on par with the unstabilized version of PRM + . We note that we use step sizes that are larger than the theoretical ones since the latter may be overly conservative [10, 25].

10

1

10

2

10

3

10

4

10

5

10

6

10

7

10

-

11

10

-

8

10

-

5

10

-

2

3x3 matrix game

10

1

10

2

10

3

10

4

10

-

5

10

-

3

10

-

1

random (

η

= 0

.

1

)

10

1

10

2

10

3

10

4

10

-

5

10

-

3

10

-

1

random (

η

= 1

)

10

1

10

2

10

3

10

4

10

-

5

10

-

3

10

-

1

random (

η

= 10

)

Number of iterations

Duality gap

RM

+

PRM

+

+

Stable PRM

+

Smooth PRM

+

Extensive-form games. We implemented and evaluated our CFR-based clairvoyant algorithm (henceforth 'Clairvoyant CFR') for extensive-form games. To our knowledge, it is the first time that clairvoyant algorithms are evaluated in extensive-form games. Overall, we were unable to observe the same strong performance observed in normal-form games (Figure 2), for a combination of reasons. First, we observe that the stepsize η calculated in Appendix J to make the operator F a contraction in extensive-form games is prohibitively small in the games we test on, each of which has a number of sequences on the order of tens of thousands. At the same time, we observe that ignoring the issue by setting a large constant stepsize in practice often leads to non-convergence of the fixed point iterations. To sidestep both issues, we considered a variant of the algorithm which only performs a single fixed-point iteration, and uses a stepsize hyperparameter η , where we pick the best from the set { 1 , 10 , 20 } . We remark that this variant of the algorithm is clairvoyant only in spirit, and while it is a sound regret-minimization algorithm, we expect that the strong theoretical

250

500

750

1000

Gradient computatations

10

-

1

10

0

CCE gap

2-player Sheriff

Cvynt CFR (

η

: 1.0)

Predictive CFR

250

500

750

1000

Gradient computatations

10

-

2

10

-

1

3-player Leduc poker

Cvynt CFR (

η

: 10.0)

Predictive CFR

250

500

750

1000

Gradient computatations

10

-

3

10

-

2

10

-

1

4-player Kuhn poker

Cvynt CFR (

η

: 10.0)

Predictive CFR

250

500

750

1000

Gradient computatations

10

-

17

10

-

16

10

-

15

4-player Liar's dice

Cvynt CFR (

η

: 10.0)

Predictive CFR

guarantees of constant per-player regret do not apply. Nevertheless, in Fig. 3 we show that we are able to sometimes observe superior performance to (non-clairvoyant) predictive CFR in the four games we tried, which are described in the appendix. For both algorithms, we ignore the first 100 iterations, in which the iterates are very far from convergence. To compensate for the increased amount of computation needed at each iteration by our clairvoyant algorithm, we plot on the x-axis not the number of iterations but rather the number of gradients of the utility functions computed for each player. On the y-axis, we measure the gap to a coarse correlated equilibrium, which is equal to the maximum regret across the players, divided by the number of iterations.

7 Conclusion

We initiated the study of stability for RM + , and showed that both RM + and predictive RM + suffer from stability issues that can lead to slow convergence in games. We introduced two simple ideas, restarting and chopping off , that ensure stability. Consequently, we introduced stable/smooth Predictive RM + , conceptual RM + and Extragradient RM + , all with strong regret guarantees. Our results yield the first RM + -based algorithms with better than O ( √ T ) regret guarantees, thus partially resolving the open question of whether optimism can yield theoretical speedup for RM + . Future directions include understanding whether our stability observations can be leveraged more directly in RM + without adding our stability tricks, extending our results to general convex games, for which a regret minimizer based on Blackwell approachability similar to RM + has been proposed recently [17], and combining clairvoyant updates with alternation.

Funding. J. Grand-Cl´ ement is supported by the Agence Nationale de la Recherche [Grant 11LABX-0047] and by Hi! Paris. Christian Kroer is supported by the Office of Naval Research awards N00014-22-1-2530 and N00014-23-1-2374, and the National Science Foundation awards IIS-2147361 and IIS-2238960. Haipeng Luo and Chung-Wei Lee are supported by National Science Foundation award IIS-1943607.

References

Bai, Y., Jin, C., Mei, S., Song, Z., and Yu, T. Efficient phi -regret minimization in extensiveform games via online mirror descent. arXiv preprint arXiv:2205.15294 , 2022.

Bowling, M., Burch, N., Johanson, M., and Tammelin, O. Heads-up limit hold'em poker is solved. Science , 347(6218), January 2015.

Brown, N. and Sandholm, T. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science , pp. eaao1733, Dec. 2017.

Burch, N., Moravcik, M., and Schmid, M. Revisiting cfr+ and alternating updates. Journal of Artificial Intelligence Research , 64:429-443, 2019.

Chen, G. and Teboulle, M. Convergence analysis of a proximal-like minimization algorithm using bregman functions. SIAM Journal on Optimization , 3(3):538-543, 1993.

Chen, X. and Peng, B. Hedging in games: Faster convergence of external and swap regrets. 2020.

Daskalakis, C., Fishelson, M., and Golowich, N. Near-optimal no-regret learning in general games. Advances in Neural Information Processing Systems , 34:27604-27616, 2021.

Farina, G., Kroer, C., Brown, N., and Sandholm, T. Stable-predictive optimistic counterfactual regret minimization. 2019.

Farina, G., Kroer, C., and Sandholm, T. Online convex optimization for sequential decision processes and extensive-form games. In AAAI Conference on Artificial Intelligence , 2019.

Farina, G., Kroer, C., and Sandholm, T. Optimistic regret minimization for extensive-form games via dilated distance-generating functions. In Conference on Neural Information Processing Systems , 2019.

Farina, G., Ling, C. K., Fang, F., and Sandholm, T. Correlation in extensive-form games: Saddle-point formulation and benchmarks. In Conference on Neural Information Processing Systems (NeurIPS) , 2019.

Farina, G., Kroer, C., and Sandholm, T. Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent. In Proceedings of the AAAI Conference on Artificial Intelligence , pp. 5363-5371, 2021.

Farina, G., Anagnostides, I., Luo, H., Lee, C.-W., Kroer, C., and Sandholm, T. Near-optimal no-regret learning dynamics for general convex games. In Advances in Neural Information Processing Systems , 2022.

Farina, G., Kroer, C., Lee, C.-W., and Luo, H. Clairvoyant regret minimization: Equivalence with nemirovski's conceptual prox method and extension to general convex games. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) , 2022.

Farina, G., Lee, C.-W., Luo, H., and Kroer, C. Kernelized multiplicative weights for 0/1polyhedral games: Bridging the gap between learning in extensive-form and normal-form games. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research . PMLR, 2022.

Freund, Y. and Schapire, R. E. Adaptive game playing using multiplicative weights. Games and Economic Behavior , 29(1-2):79-103, 1999.

Grand-Cl´ ement, J. and Kroer, C. Conic blackwell algorithm: Parameter-free convex-concave saddle-point solving. Advances in Neural Information Processing Systems , 34:9587-9599, 2021.

Grand-Cl´ ement, J. and Kroer, C. Solving optimization problems with blackwell approachability. arXiv preprint arXiv:2202.12277 , 2022.

Hart, S. and Mas-Colell, A. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68(5):1127-1150, 2000.

Hazan, E. et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Hoda, S., Gilpin, A., Pena, J., and Sandholm, T. Smoothing techniques for computing nash equilibria of sequential games. Mathematics of Operations Research , 35(2):494-512, 2010.

Hsieh, Y., Antonakopoulos, K., and Mertikopoulos, P. Adaptive learning in continuous games: Optimal regret bounds and convergence to nash equilibrium. In Belkin, M. and Kpotufe, S. (eds.), Conference on Learning Theory, COLT 2021 , volume 134 of Proceedings of Machine Learning Research , pp. 2388-2422. PMLR, 2021.

Kiwiel, K. C. Proximal minimization methods with generalized bregman functions. SIAM journal on control and optimization , 35(4):1142-1168, 1997.

Korpelevich, G. M. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

Kroer, C., Waugh, K., Kılınc ¸-Karzan, F., and Sandholm, T. Faster algorithms for extensiveform game solving via improved smoothing functions. Mathematical Programming , 179(1): 385-417, 2020.

Kuhn, H. W. A simplified two-person poker. In Kuhn, H. W. and Tucker, A. W. (eds.), Contributions to the Theory of Games , volume 1 of Annals of Mathematics Studies, 24 , pp. 97-103. Princeton University Press, Princeton, New Jersey, 1950.

Lis` y, V ., Lanctot, M., and Bowling, M. H. Online monte carlo counterfactual regret minimization for search in imperfect information games. 2015.

Moravˇ c´ ık, M., Schmid, M., Burch, N., Lis´ y, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowling, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science , May 2017.

Nemirovski, A. Prox-method with rate of convergence o (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Nemirovski, A. and Yudin, D. Problem complexity and method efficiency in optimization. 1983.

Piliouras, G., Sim, R., and Skoulakis, S. Optimal no-regret learning in general games: Bounded regret with unbounded step-sizes via clairvoyant mwu. arXiv preprint arXiv:2111.14737 , 2021.

Rakhlin, A. and Sridharan, K. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems , pp. 3066-3074, 2013.

Southey, F., Bowling, M., Larson, B., Piccione, C., Burch, N., Billings, D., and Rayner, C. Bayes' bluff: Opponent modelling in poker. July 2005.

Syrgkanis, V., Agarwal, A., Luo, H., and Schapire, R. E. Fast convergence of regularized learning in games. Advances in Neural Information Processing Systems , 28, 2015.

Tammelin, O. Solving large imperfect information games using cfr+. arXiv preprint arXiv:1407.5042 , 2014.

Tammelin, O., Burch, N., Johanson, M., and Bowling, M. Solving heads-up limit texas hold'em. In Twenty-fourth international joint conference on artificial intelligence , 2015.

Zinkevich, M., Johanson, M., Bowling, M., and Piccione, C. Regret minimization in games with incomplete information. Advances in neural information processing systems , 20, 2007.

A Proof of Lemma 2.1

Proof of Lemma 2.1. Let us write ˆ R = ˆ x . Note that

where (3) follows from ˆ x ⊤ 1 d = 1 and the definition of the map f ( · , · ) , (4) follows from ⟨ R t , f ( x t , ℓ t ) ⟩ = 0 because x t = R t / ∥ R t ∥ 1 (note that this is also trivially true when R t = 0 ), and (5) follows from the definition of Reg T ( ˆ R ) .

B Instability of RM + and predictive RM +

B.1 Proof of Theorem 3.1

Proof of Theorem 3.1. Wefirst prove the case for RM + . Since we consider x t ∈ R 2 , we can express x t = ( p t , 1 -p t ) for some scalar p t ∈ [0 , 1] (starting with p 1 = 1 / 2 ). In our counterexample, we set ℓ t = ( ℓ t , 0) for some scalar ℓ t to be specified. Consequently, we have

To make the algorithm highly unstable, we first provide an unbounded sequence of ℓ t so that the resulting R t alternates between vectors with the same value on both entries and vectors with only the first entry being 0, which means p t by definition alternates between 1 / 2 and 0 . Noting that RM+ is scale-invariant to the loss sequence, our proof is completed by normalizing the losses so that they all lie in [ -1 , 1] .

Specifically, we set ℓ 1 = 2 , which gives f ( x 1 , ℓ 1 ) = (1 , -1) , R 2 = (0 , 1) , and p 2 = 0 . Then for t ≥ 2 we set ℓ t = -2 ( t -2) / 2 when t is even and ℓ t = 2 ( t -1) / 2 when t is odd. By direct calculation it is not hard to verify that

p t +1 = 1 when t is even, and

p t +1 = 0 when t is odd, completing the counterexample for RM + .

It remains to prove the case for predictive RM + , where m t = f ( x t -1 , ℓ t -1 ) . Initially, let ℓ 1 = 4 , ℓ 2 = -1 . Recall that

By direct calculation, we have

Thereafter, we set ℓ t = 2 ( t +1) / 2 when t is odd and ℓ t = -2 ( t -2) / 2 when t is even. The updates for the next 4 steps are:

It is not hard to verify (by induction) that

when t is odd and f ( x t , ℓ t ) = ( -2 ( t -2) / 2 , 0) , R t +1 = (2 ( t -2) / 2 , 2 t/ 2 ) , ˆ R t +1 = (2 t/ 2 , 2 t/ 2 ) , p t +1 = 1 2 when t is even. This completes the proof.

Remark B.1. The losses are unbounded in the examples, but note that the update rules for the algorithms imply that all the algorithms remain unchanged after scaling the losses, so we can rescale them accordingly. Specifically, if we have a loss sequence ℓ 1 , . . . , ℓ T , we can define L T = max {| ℓ 1 | , . . . , | ℓ T |} and consider another loss sequence ℓ 1 /L T , . . . , ℓ T /L T , which is bounded in [ -1 , 1] and will make the algorithms produce the same outputs.

B.2 Counterexample on 3 × 3 matrix game for RM +

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

6

10

-

4

10

-

2

Norm

||

x t

-

x t

+1

||

2

2

(a)

∥

x

t

+1

-

x

t

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

7

10

-

5

10

-

3

Norm

||

y t

-

y t

+1

||

2

2

(b)

∥

y

t

+1

-

y

t

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

0

10

1

10

2

Regret

Regret (x-player)

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

-300

-200

-100

0

Regret

Regret (y-player)

(a) Regret of the first player

(b) Regret of the second player

C Proof of Proposition 1 and Corollary 3.2

We start with a couple of technical lemmas.

Lemma C.1. Given any y ∈ R d + and x ∈ R d such that 1 ⊤ x = 0 ,

Proof. If x = 0 the claim is trivial, so we focus on the other case. Let ξ be the (Euclidean) projection of y onto the orthogonal complement of span { x , 1 } . Since by hypothesis 1 ⊥ x , it holds that

and therefore

Using the hypothesis that y ≥ 0 , we can bound

where we used the well-known inequality between the ℓ 1 -norm and the ℓ 2 -norm. Substituting the previous inequality into (6), and using the fact that ∥ 1 ∥ 2 2 = d ,

Rearranging the terms yields the statement.

Lemma C.2. For any ˆ y ∈ R d + such that ∥ ˆ y ∥ 2 = 1 , 1 ⊤ ˆ y = 0 and for any x ∈ R d such that 1 ⊤ x = 1 ,

̸

Proof. The main idea of the proof is to introduce

Note that 1 ⊤ z = 1 ⊤ x -1 = 0 . Furthermore,

Substituting the previous equality in the statement, we obtain

where we used the hypothesis that ∥ ˆ y ∥ 2 2 = 1 in the third equality. Using the inequality of Lemma C.1 concludes the proof.

We are now ready to prove Proposition 1.

Proof of Proposition 1. If y = 0 , the statement holds trivially. Hence, we focus on the case y = 0 . Let ˆ y := y / ∥ y ∥ 2 be the direction of y ; clearly, ∥ ˆ y ∥ 2 = 1 . Note that

̸

̸

where we used the hypothesis that 1 ⊤ x ≥ 1 in the last step. On the other hand, using Lemma C.2 (note that 1 ⊤ ˆ y = 0 since y = 0 by hypothesis),

̸

Combining (7) and (8), we obtain the statement.

Proof of Corollary 3.2. The condition means that 1 ⊤ R t R 0 ≥ 1 and

D Proof of Theorem 4.1

Proof of Theorem 4.1. When the algorithm restarts, the accumulated regret is negative to all actions, so it is sufficient to consider the regret from T 0 , the round when the last restart happens to the end. In that case, we can analyze the algorithm as a normal predictive regret matching algorithm. By Proposition 5 in [12], we have that the regret for player i is bounded by

where z T 0 i = ( R 0 , . . . , R 0 ) and ( f t -1 i ) i ∈ [ n ] = F ( z t -1 ) . When setting m t i = f t -1 i , then ∥ f t i -m t i ∥ can be bounded by

̸

where the last-but-one inequality is because

and we bound each of RHS of the first inequality using a restatement of Corollary 3.2, shown in Lemma D.1. Therefore, we can further bound (9) it by dropping the negative terms and bounding the rest by

Choosing R 0 = 1 and η = ( d 2 T ) -1 / 4 finishes the proof.

Lemma D.1. Let z = Π w , X ( η f ( x , ℓ )) for x ∈ ∆ d , ℓ ∈ R d , ∥ ℓ ∥ 2 ≤ B u . Suppose ∥ w ∥ 1 ≥ R 0 , then we have

Proof. The proof is essentially the same as Corollary 3.2. The condition means that 1 ⊤ w R 0 ≥ 1 and

E Proof of Theorem 4.2

We write ( R t 1 , ..., R t n ) = z t . Let us consider the regret Reg T i (ˆ x i ) of player i ∈ { 1 , ..., n } . Lemma 2.1 shows that

with Reg T i ( ˆ R i ) the regret against ˆ R i = ˆ x i incurred by a decision-maker choosing the decisions ( R t i ) t ≥ 1 and facing the sequence of losses ( f t i ) t ≥ 1 , with f t i = ℓ t i -⟨ x t i , ℓ t i ⟩ 1 d i :

Note that R 1 i , ..., R T i is computed by Predictive OMD with ∆ d i ≥ as a decision set, f 1 i , ..., f T i as the sequence of losses and m 1 i , ..., m T i as the sequence of predictions. Therefore, Proposition 5 in [12] applies, and we can write the following regret bound:

Since we maintain R t i ∈ ∆ d i ≥ at every iteration, using Proposition 1 we find that

Plugging this into (11), we obtain

Using ∥ · ∥ 2 ≤ ∥ · ∥ 1 ≤ √ d i ∥ · ∥ 2 , we obtain

with α = ∥ w 0 i -ˆ x i ∥ 2 2 2 η , β = d i η, γ = 1 8 d 2 i η . To conclude as in Theorem 4 in [34] we need β ≤ γ/ ( n -1) 2 , i.e., η = 1 2 √ 2( n -1) d 3 / 2 i . Therefore, using η = 1 2 √ 2( n -1) max i { d 3 / 2 i } , we conclude that the sum of the individual regrets is bounded by

F Proof of Theorem 5.3

Proof of Lemma 5.1. The first-order optimality condition gives

Rearranging gives that for any ˆ z ∈ Z , we have

Multiplying by -1 and summing over all t = 1 , ..., T gives the regret bound:

Proof of Lemma 5.2. Let x , x ′ ∈ ∆ and i ∈ { 1 , ..., n } . Let us write ℓ i = -∇ x i u i ( x ) , ℓ ′ i = -∇ x i u i ( x ′ ) . We have, for i ∈ { 1 , ..., n } ,

where the last inequality follows from Cauchy-Schwarz inequality. Now from (1) and the definition of ℓ i , ℓ ′ i , we have

This yields

Since the function g is √ d i -Lipschitz continuous over each decision set ∆ d i ≥ (Proposition 1), we have shown that the Lipschitz constant of F is L F = (max i d i ) √ 2 B 2 u +4 L 2 u .

We are now ready to prove Theorem 5.3. We write ( R t 1 , ..., R t n ) = z t .

Proof of Theorem 5.3. We use the fact that ( z t ) t ≥ 1 is computed following the Cheating OMD update with ℓ t = F ( z t ) at every iteration t ≥ 1 . Therefore, the first-order optimality condition in z t = Π z t -1 , X ≥ ( ηF ( z t )) yields

Similarly as for the proof of Lemma 5.1, we obtain that for any ˆ z ∈ X ≥ , we have

̸

Let i ∈ { 1 , ..., n } . We apply the inequality above to the vector ˆ z ∈ X ≥ = × n i =1 ∆ d i ≥ defined as ˆ z j = z t j for j = i and ˆ z i = ˆ R i for some ˆ R i ∈ ∆ d i ≥ . This yields, for any ˆ R i ∈ ∆ d i ≥ , for x t j = g ( R t j ) and ( ℓ t 1 , ..., ℓ t n ) = G ( x t ) for all j ∈ { 1 , ..., n } ,

Summing the above inequality for t = 1 , ..., T , we obtain our bound on the individual regrets of each player: for any ˆ R i ∈ ∆ d i ≥ , we have

Note that from Lemma 2.1, we have that the individual regret of player i

against a decision ˆ x i ∈ ∆ d i is equal to ∑ T t =1 ⟨ f ( x t , ℓ t ) , R t i -ˆ R i ⟩ for ˆ R i = ˆ x i . Therefore, we conclude that

This concludes the proof of Theorem 5.3.

G Proof of Theorem 5.5

Proof of Theorem 5.5. At iteration t ≥ 1 , let w t ∈ X ≥ such that ∥ w t -Π z t -1 , X ≥ ( ηF ( w t )) ∥ 2 ≤ ϵ ( t ) . Then the first order optimality condition gives, for z t = Π z t -1 , X ≥ ( ηF ( w t )) ,

̸

Let us fix a player i ∈ { 1 , ..., n } . We apply the inequality above with ˆ z j = z t j for j = i . This yields, for x p = g ( w t p ) , ∀ p ∈ { 1 , ..., n } and ℓ t i = -∇ x i u i ( x ) ,

We now upper bound the left-hand side of the previous inequality. Note that

Cauchy-Schwarz inequality ensures that

Note that Π z t -1 , X ≥ ( ηF ( w t )) = z t , so that ∥ w t i -z t i ∥ 2 ≤ ϵ ( t ) . To bound ∥ f ( g ( w t i ) , ℓ t i ) ∥ 2 , we note that by definition,

This gives

Overall, we have obtained that for all ˆ z i ∈ ∆ d i ≥ , we have

We sum the previous inequality for t = 1 , ..., T to obtain that for all ˆ z i ∈ ∆ d i ≥ , we have

Overall, we conclude that

From Lemma 2.1 the left-hand side is equal to Reg T i (ˆ x i ) for ˆ x i = ˆ z i . This concludes the proof of Theorem 5.5.

H Proof of Theorem 5.6

Proof of Theorem 5.5. We will show that for any ˆ w ∈ X ≥ , we have

Since x t i = w t i , ∀ t ≥ 1 , ∀ i ∈ { 1 , ..., n } , this is enough to prove Theorem 5.5. Note that

We will independently analyze each term of the right-hand side of the above equality.

For the first term, we note that the first-order optimality condition for z t = Π z t -1 , X ≥ ( ηF ( w t )) gives, for any ˆ w ∈ X ≥ ,

For the second term, we will prove the following lemma.

√

Lemma H.1. Let η > 0 such that w ↦→ ηF ( w ) is 1 / 2 Lipschitz continuous over X ≥ . Then

Proof of Lemma H.1. We write

We will bound independently each term in the above equation. From w t = Π z t -1 , X ≥ ( ηF ( z t -1 ) ) we have

which gives

From Cauchy-Schwarz inequality, we have

Recall that

Since the proximal operator is 1 -Lipschitz continuous, and since w ↦→ ηF ( w ) is 1 / √ 2 -Lipschitz continuous, we obtain

We can now sum (14) and (15) to obtain

We have shown in Lemma 5.2 that F is L F -Lipschitz continuous for normal-form games. Our choice of step size η = 1 L F √ 2 ensures that that ω ↦→ ηF ( w ) is 1 / √ 2 -Lipschitz continuous as in the assumptions of Lemma H.1.

Combining (12) with (13) yields

Summing this inequality for t = 1 , ..., T and telescoping, we obtain

which directly yields

Overall, for any ( ˆ R 1 , ..., ˆ R n ) ∈ X ≥ we obtain that ∑ T i =1 Reg T i ( ˆ R i ) is upper bounded by ∑ n i =1 1 2 η ∥ w 0 i -ˆ R i ∥ 2 2 . Now from Lemma 2.1, for any (ˆ x 1 , ..., ˆ x n ) ∈ ∆ , we conclude that

This concludes the proof of Theorem 5.6.

I Proof of Lemma 5.7

Proof of Lemma 5.7. The proof of Lemma 5.7 follows the lines of the proof of Lemma 5.2. Clearly, for matrix games we have F = h ◦ g with h : R d 1 × R d 2 → R d 1 × R d 2 defined as Proposition 1.

The function g is Lipschitz continuous over ∆ d 1 ≥ (Proposition 1), with a Lipschitz constant of L g = √ d 1 . Let us now compute the Lipschitz constant of h . Observe that:

Similarly, we have that ∥ f ( y , -A ⊤ x ) -f ( y ′ , -A ⊤ x ′ ) ∥ 2 is upper bounded by and thus

2 2 d 2 ∥ A ∥ 2 op ∥ y -y ′ ∥ 2 2 +4 d 2 ∥ A ∥ 2 op ∥ x -x ′ ∥ 2 2 ,

Therefore, the Lipschitz constant of h is L h = ∥ A ∥ op √ 6 max { d 1 , d 2 } .

Since F = h ◦ g , we obtain that the Lipschitz constant L F of F is L F = L h × L g = √ 6 ∥ A ∥ op max { d 1 , d 2 } .

J Extensive-Form Games

In this section we show how to extend our convergence results for Conceptual RM + from normalform games to EFGs. Briefly, an EFG is a game played on a tree, where each node belongs to some player, and the player chooses a probability distribution over branches. Moreover, players have information sets , which are groups of nodes belonging to a player such that they cannot distinguish among them, and thus they must choose the same probability distribution at all nodes in an information set. When a leaf h is reached, each player i receives some payoff v i ( h ) ∈ [0 , 1] . In order to extend our results, we will use the CFR regret decomposition [37, 9], and then show how to run the Conceptual RM + algorithm on the resulting set of strategy spaces (which will be a Cartesian product of positive orthants). The CFR regret decomposition works in the space of behavioral strategies , which represents the strategy space of each player as a Cartesian product of simplices, with each simplex corresponding to the set of possible ways to randomize over actions at a given information set for the player. Formally, we write the polytope of behavioral-form strategies as

where D i is the set of information sets for player i and n j is the number of actions at information set j . Let P = ∑ i ∈ [ n ] ,j ∈D i n j be the dimension of X . In EFGs with perfect recall , meaning that a player never forgets something they knew in the past, the sequence-form is an equivalent representation of the set of strategies, which allows one to write the payoffs for each player as a multilinear function. This in turn enables optimization and regret minimization approaches that exploit multilinearity, e.g. bilinearity in the two-player zero-sum setting [21, 25, 10]. Instead of working on this representation, the CFR approach minimizes a notion of local regret at each information set, using so-called counterfactual values . The weighted sum of counterfactual regrets at each information set is an upper bound on the sequence-form regret [37], and thus a player in an EFG can minimize their regret by locally minimizing each counterfactual regret. Informally, the counterfactual value is the expected value of an action at a information set, conditional on the player at the information set playing to reach that information set and then taking the corresponding action. The counterfactual value associated to each tuples of player i , information set j ∈ D i , and action a ∈ A j is G ija ( x ) := ∑ h ∈L ja ∏ ( ˆ j, ˆ a ) ∈P j ( h ) x [ ˆ j, ˆ a ] v i ( h ) , where L ja is the set of leaf nodes reachable from information set j after taking action a , and P j ( h ) is the set of pairs of information sets and actions ( ˆ j, ˆ a ) on the path from the root to h , except that information sets belonging to player i are excluded, unless they occur after j, a .

Wewill be concerned with the counterfactual regret, given by the operator H : X → R ∑ i ∈ [ n ] ,j ∈D i n j defined as H ija ( x ) := G ija ( x ) - ⟨ G ij ( x ) , x j ⟩ . Now we can show that the counterfactual regret operator H is Lipschitz continuous. Intuitively, this should hold since H is multilinear. √

Proof. We start by showing a bound for G . We first analyze the change in a single coordinate of G for a given i ∈ [ n ] , j ∈ D i , a ∈ A j . We focus on how G ija changes with respect to the change in | x [ ˆ j, ˆ a ] -x ′ [ ˆ j, ˆ a ] | for some arbitrary information set-action pair ( ˆ j, ˆ a ) ∈ P j ( h ) for some h ∈ L ja . To alleviate inline notation, let P ˆ j, ˆ a j ( h ) = P j ( h ) \ { ( ˆ j, ˆ a ) } .

Now let us bound the error term by noting that v i ( h ) ≤ 1 for all h by assumption:

where the last inequality is because the sum of reach probabilities on leaf nodes in L ˆ j ˆ a ∩ L ja after conditioning on player i playing to reach ( j, a ) and ( ˆ j, ˆ a ) being played with probability one, is less than or equal to one.

By iteratively applying this argument to each ( ˆ j, ˆ a ) ∈ P j ( h ) , we get

Repeating the same argument for x ′ gives

Secondly, we bound the difference in the inner product terms.

where the second-to-last line is by Eq. (18). Again we can start from x ′ instead to get

Putting together all our bonds and applying norm equivalence, we get that

Taking square roots completes the proof.

Since we want to run smooth RM + , we will need to consider the lifted strategy space for each decision point. Let Z be the Cartesian product of the positive orthants for each information set, i.e. Z = × i ∈ [ n ] ,j ∈D i R n j + . Now let ˆ g : Z → X be the function that normalizes each vector from the positive orthant to the simplex such that we get a behavioral strategy, i.e. ˆ g j ( z ) = g ( z j ) , where z j is the slice of z corresponding to information set j . The function ˆ g is also Lipschitz continuous.

Lemma J.2. Suppose that z , z ′ ∈ Z satisfy ∥ z j ∥ 1 ≥ R 0 ,j , ∥ z j ′ ∥ 1 ≥ R 0 ,j for all i ∈ [ n ] , j ∈ D i . Then, ∥ ˆ g ( z ) -ˆ g ( z ′ ) ∥ 2 ≤ max i ∈ [ n ] ,j ∈D i √ n j /R 0 ,j ∥ z -z ′ ∥ 2 .

Proof. We have from Proposition 1 that

Now let us introduce the operator F : Z → R ∑ i ∈ [ n ] ,j ∈D i n j for EFGs. For a given z ∈ Z , the operator will output the regret associated with the counterfactual values for each decision set j . F will be composed of two functions, first ˆ g maps a given z to some behavioral strategy x = ˆ g ( z ) , and then the operator H : X → R ∑ i ∈ [ n ] ,j ∈D i n j outputs the regrets for the counterfactual values.

Now we can apply our bounds on the Lipschitz constant for ˆ g and H to get that F is Lipschitz continuous with Lipschitz constant 2 P max i ∈ [ n ] ,j ∈D i √ n j /R 0 ,j . Combining our Lipschitz result with our setup of X and F , we can now run Algorithm 4 on X and F and apply Theorem 5.5 to get a smooth-RM + -based algorithm that allows us to compute a sequence of iterates with regret at most ϵ in at O (1 /ϵ ) iterations and using O (log(1 /ϵ ) /ϵ ) gradient computations.

K Details on the Numerical Experiments

Efficient orthogonal projection on ∆ n ≥ . Recall that ∆ n ≥ = { R ∈ R n | R ≥ 0 , 1 ⊤ n R ≥ 1 } . Let y ∈ R n and let us consider

Introducing a Lagrange multiplier µ ≥ 0 for the constraint 1 -1 ⊤ n x ≤ 0 , we arrive at

Let us call ( x , µ ) ∈ R n + × R + an optimal solution to the above saddle-point problem. Stationarity of the Lagrangian function shows that x i = [ y i + µ ] + , ∀ i ∈ [ n ] . Therefore, we could simply use binary search to solve the following univariate concave problem:

Let us use the Karush-Kuhn-Tucker conditions. Complementary slackness gives µ · ( 1 -1 ⊤ n x ) = 0 . If µ = 0 , then x = [ y ] + , and by primal feasibility we must have 1 ⊤ n x ≥ 1 , i.e., 1 ⊤ n [ y ] + ≥ 1 . If that is not the case, then we can not have µ = 0 , and we must have 1 -1 ⊤ n x = 0 , i.e., x ∈ ∆ n . In this case, we obtain that x is the orthogonal projection of y on ∆ n . Overall, we see that x is always either [ y ] + , the orthogonal projection of y on R n + , or x is the orthogonal projection of y on ∆ n . Since ∆ n ≥ ⊂ R n + , we can compute the orthogonal projection on ∆ n ≥ as follows:

Compute x = [ y ] + . If 1 ⊤ n x ≥ 1 , then we have found the orthogonal projection of y on ∆ n ≥ . Else, return the orthogonal projection of y on the simplex ∆ n .

K.1 Performances of ExRM + , Stable PRM + and Smooth PRM + on our small matrix game example

In this section we provide detailed numerical results for ExRM + , Stable PRM + , and Smooth PRM + on our 3 × 3 matrix-game counterexample. All algorithms use linear averaging and Stable and Smooth PRM + use alternation. We choose a step size of η = 0 . 1 for our implementation of these algorithms. The results are presented in Figure 6 for ExRM + , in Figure 7 for Stable PRM + and in Figure 8 for Smooth PRM + .

K.2 Extensive-form game used in the experiments

We used the following games in the experiments:

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

11

10

-

8

10

-

5

10

-

2

Duality gap

+

(a) Duality gap

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

21

10

-

15

10

-

9

10

-

3

Norm

||

x t

-

x t

+1

||

2

2

(b)

∥

x

t

-

x

t

+1

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

21

10

-

16

10

-

11

10

-

6

Norm

||

y t

-

y t

+1

||

2

2

(c)

∥

y

t

-

y

t

+1

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

1.50

1.75

2.00

2.25

2.50

Regret

Regret (x-player)

(d)

Reg

T

x

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

-0.8

-0.6

-0.4

-0.2

0.0

Regret

Regret (y-player)

(e)

Reg

T

y

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

11

10

-

8

10

-

5

10

-

2

Duality gap

Stable PRM

+

(a) Duality gap

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

26

10

-

19

10

-

12

10

-

5

Norm

||

x t

-

x t

+1

||

2

2

(b)

∥

x

t

-

x

t

+1

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

26

10

-

19

10

-

12

10

-

5

Norm

||

y t

-

y t

+1

||

2

2

(c)

∥

y

t

-

y

t

+1

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

1

2

3

Regret

Regret (x-player)

(d)

Reg

T

x

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

-1.0

-0.5

0.0

0.5

1.0

Regret

Regret (y-player)

(e)

Reg

T

y

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

11

10

-

8

10

-

5

10

-

2

Duality gap

Smooth PRM

+

(a) Duality gap

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

22

10

-

16

10

-

10

10

-

4

Norm

||

x t

-

x t

+1

||

2

2

(b)

∥

x

t

-

x

t

+1

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

10

-

22

10

-

16

10

-

10

10

-

4

Norm

||

y t

-

y t

+1

||

2

2

(c)

∥

y

t

-

y

t

+1

∥

2

2

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

1.4

1.6

1.8

2.0

Regret

Regret (x-player)

(d)

Reg

T

x

10

1

10

2

10

3

10

4

10

5

10

6

10

7

Number of iterations

-0.5

0.0

0.5

Regret

Regret (y-player)

(e)

Reg

T

y

2-player Sheriff is a two-player general-sum game inspired by the Sheriff of Nottingham board game. It was introduced as a benchmark for correlated equilibria by Farina et al. [11]. The variant of the game we use has the following parameters:

-maximum number of items that can be smuggled: 10

-maximum bribe amount: 3

-number of bargaining rounds: 3

-value of each item: 5

-penalty for illegal item found in cargo: 1

-penalty for Sheriff if no illegal item found in cargo: 1

The number of nodes in this game is 9648.

3-player Leduc poker is a 3-player version of the standard benchmark of Leduc poker [33]. The game has 15659 nodes.

4-player Kuhn poker is a 4-player version of the standard benchmark of Kuhn poker [26]. We use a larger variant the standard one, to assess the scalability of our algorithm. The variant we use has six ranks in the deck. The game has 23402 nodes.

4-player Liar's dice is a 4-player version of the game Liar's dice, already used as a benchmark by Lis` y et al. [27]. We use a variant with 1 die per player, each with two distinct faces. The game has 8178 nodes.

Peer reviews:
comments: SUMMARY:
The paper pushed RM+ based algorithm better than $O(\sqrt T)$ theoretically the first time. Specifically, the authors proposed smooth RM+, conceptual RM+ and extragradient RM+ with theoretical regret guarantee.

STRENGTHS:
To the best of my knowledge, this paper is the first one to establish RM+ based algorithm with better theoretical gurantee than $O(\sqrt T)$. Before, although people observe $O(1)$ regret guarantee of RM+ in *most* games (there exist counterexamples that make it around $O(T^{0.7})$), there's no variant of RM+ that has better than $O(\sqrt T)$ regret bound.

However, this paper proposed a really simple and straight-forward version of RM+ and proved better regret bound. This may partially reveal why RM+ and CFR+ have good performance in many games.

WEAKNESSES:
- The main text of the paper does not reveal much about the proof intuition.

QUESTIONS:
I'm a bit confused about the CFR part. Does CFR equipped with the new RM+ has constant social regret?

LIMITATIONS:
The authors discussed limitations in the conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study $\\text{RM}^+$. They show that there are instances of loss sequences in variants of $\\text{RM}^+$ unstable. The authors point out that the decisions of $\\text{RM}^+$-based algorithms are performed by normalizing an aggregate payoff vector. Hence, if one inputs an instance that is \"close\" to the origin, two consecutive aggregate payoffs can point in different directions in spite of being relatively close, which could make the algorithm cycle between two strategies. The authors exhibit an example establishing that the potential instability mentioned does indeed occur. \n\nThe authors then present two methods to the instability discussed dealing with having two consecutive aggregate payoffs being too close to the origin. In one, they simply re-initialize the regret vector too some non-zero amount. The other solution is simply to ignore an area of the space near the origin and projecting onto that space. Not surprisingly, the second method seems to have a better theoretical guarantee but it's more expensive to implement.\n\nSTRENGTHS:\nThe paper is very-well written. It was a pleasure to read. I had some very minor comments about the draft in regards to exposition and writing. \n\nI also like how natural their solutions are to the issue of instability. The proofs seem to be correct and everything feels very natural, which is probably due to the authors' writing. The experiments also seem reasonable and establish that their methods perform well with random synthetic data.\n\nWEAKNESSES:\nWould the authors please put their contribution in context? How \"important\" it is to the overall community? They have already addressed some of this in the draft, but I was hoping for longer explanation in the rebuttal period. \n\nSome small suggestions \n1. Generally, I think citations are not appropriate for abstracts since abstracts are often read independently of the full paper, so readers may not have access to the cited sources at that stage. If you want to include a specific reference or highlight previous related work in the abstract, I recommend paraphrasing the information or briefly mention the authors' names and year of publication without the specific citation. This maintains the abstract's clarity and conciseness while still giving credit to the relevant work. I realize adding the references simply as numbers might have done intentionally to hide the identity of the authors. \n2. I'd recommend combining the sentences for on lines 45 to 48 \"However, in a game setting...\" and \"Indeed, we identify...\" to avoid using the word \"this\" which is a bit ambiguous.\n3. Would it be possible to list the references in increasing order?\n4. For the sake of completeness, it would be useful if T were defined as the horizon or something like that before being used.\n5. Notation-wide, I think $\\Delta(3)$ should probably be $\\Delta^3$ on line 169.\n6. The definition of $\\mathbf{z}^t$ in Theorem 4.1 was a bit unclear to me until I saw it defined in Algorithm, 1. Maybe declare it before like you did for Theorem 5.1 in expression (2)\n7. On line 346, when you say \"$\\text{RM}^+$-based algorithms\" I would add in parenthesis to refer to which algorithms are meant (either by number or name)\n\nQUESTIONS:\nQ1: More the smooth predictive RM^+, was there anything essentially different if one uses a different threshold for $||R_1||$? Now, 1 is being used, which seems very reasonable. However, how ``far away\" does one need to know from 0 to help stabilization? Like if I were to change 1 for a constant C, how does this affect the theoretical guarantees? \n\nQ2: I must be missing something, but I don't fully understand why clairvoyant CFR doesn't seem to perform as well with EFG as it does with matrix games. If you were to use the strategic game equivalent of the EFGs tested, what is the performance. I apologize because there is probably something I'm missing.\n\nQ3: In your experiments, it seems the data was generated randomly. This is good to know. However, did you try to create data that would be more \"difficult\" like maybe a space that is an annulus?\n\nLIMITATIONS:\nThe authors make an effort of what they have consider and the limitations of their work in terms of experimental results obtained for EFGs\n\nI also would like to see how the algorithm performs in \"harder\" spaces than simply random ones.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper studies variants of the RM+ algorithm for learning in games. They show that RM+ does not satisfy stability, which is an important property for proving good regret bounds, if all players play according to the same algorithm. The paper "fixes" the instability of RM+ by considering a restarting and chopping variant which stabilise RM+ and then prove O(T^{1/4}) and O(1) individual and social regret bounds respectively.

STRENGTHS:
For the most part paper is well written and easy to follow.
The paper points to the importance of stability for proving sub-sqrt regrets bounds, which I find interesting.

WEAKNESSES:
In my opinion Sec 5 too dense and thus it lacks clarity. I would suggest to put only the main technical result there , or to give a skimmed version, in which the more technical assumptions are deferred to the appendix. I think that 4 statements with little discussion, cannot convey clarity.

QUESTIONS:
1. It seems that for the restarting trick you cannot prove O(1) social regret. Is this because restarting may happen at different times for different players? If so I think you should include a larger discussion on this. Moreover in the abstract it seems like both restarting and chopping would give O(1) social regret. I think you should state explicitly in the abstract that only holds for the chopping variant.
2. Do you conjecture that O(1) social is only hard to prove for the restarting method or that is not reached?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the RM+ algorithm and its variants in the context of learning in games.\n- They first show that RM+ and predictive RM+ can be unstable in certain environments. Although the unstable player benefits from instability, the regret of other players may blow up because they are forced to make predictions in an unstable environment.\n- Then they provide several approaches to stabilize RM+.\n\t- The first approach, called **stable predictve RM+**, requires each player to (asynchronously) restart whenever $\\mathbf{R}^t_i$ becomes small. Although this stabilizes the algorithm and guarantees $O(T^{1/4})$ regret for each player, it's hard to bound the social regret by $O(1)$ .\n\t- The second approach is called **smooth predictive RM+**. It chops off the area of decision space that is too close to the origin by adding projection steps. Besides individual regret bounds, this approach also guarantees $O(1)$ social regret.\n\t- They also propose **Conceptual RM+** (and the one fixed-point approximation of it) that achieves $O(1)$ individual regret and **Extragradient RM+** that achieves $O(1)$ social regret.\n- The authors conduct experiments on matrix games and extensive-form games.\n\nSTRENGTHS:\n- The paper contributes to the theoretical understanding of the RM+ and its variants. It is the first one to show the instability of RM+ and predictive RM+ by concretely constructing a hard case that exhibits the $T^{-0.5}$ convergence rate.\n- The paper proposes several variants of RM+ that not only have appealing theoretical guarantees on both individual regret and social regret, but also perform well in experiments.\n- The paper is very well-written. The authors clearly explain the intuition behind the theoretical results.\n\nWEAKNESSES:\n- In section 3, the authors construct a hard case for RM+ and predictive RM+ where the empirical convergence rate are on the order of $T^{-0.5}$. However, the residue errors of the linear fit in Figure 1 are quite large. Providing a theoretical proof that the asymptotic convergence rate is indeed $T^{-0.5}$ would strengthen the result.\n- Given the connection between RM+ and OMD in [12] and the fast convergence properties of OMD in games, the performance guarantee for the stabilized variants of RM+ do not seem surprising. Nonetheless, as mentioned by the authors, RM+-based algorithms are not inherently stable, so they need extra considerations.\n- The experiments show the strong performances of the proposed algorithms compared to RM+ and PRM+. It would be better if the authors also compared their performance with the optimistic/predictive FTRL/OMD algorithms that enjoy fast convergence rates in theory.\n\nQUESTIONS:\nIn the proof of Theorem 3.1, the authors constructed an example where the precision of the losses increases with $T$ (the minimum value is exponentially small), a scenario that may not practically appear.\n- Is this phenomenon of increasing precision the fundamental reason of instability, or is it merely an artifact of the proof?\n- Does this same phenomenon appear in the subsequent experiments involving the $3\\times3$ matrix game?\n- If a bounded precision assumption were introduced, could this potentially stabilize RM+ or PRM+?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on investigating the stability of the Regret Matching+ based algorithm. It demonstrates that both RM+ and predictive RM+ algorithms exhibit instability, leading to significant regret for other players within a game setting. To address this issue, the authors propose two methods: restarting and chopping off. By incorporating these methods, the authors introduce several variants of RM+-based algorithms, including stable/smooth Predictive RM+, conceptual RM+, and Extragradient RM+. The paper also provides theoretical regret bounds for these proposed algorithms, further establishing their efficacy in mitigating instability and reducing regret.\n\nSTRENGTHS:\nOne of the key contributions of this paper is the theoretical speedup provided for the RM+ based algorithm, which is crucial for achieving fast convergence in games. By offering improved convergence guarantees, the proposed advancements in the algorithm can significantly enhance its practical utility.\n\nThe writing in the paper is well-organized and clear, effectively conveying the technical details. The strength of the paper's technical aspects further adds to its overall quality, making it more accessible and understandable for readers.\n\nThe comprehensive experimental results presented in the paper further enhance its credibility. Notably, the authors also consider the application of the algorithm in extensive form games, which is important for real-world scenarios. This consideration highlights the practical relevance and versatility of the proposed techniques.\n\nOverall, the paper provides valuable theoretical insights, demonstrates technical proficiency, and showcases the practical applicability of the proposed algorithm, making it a strong contribution to the field.\n\nWEAKNESSES:\nJust a small point about presentation. It would be beneficial for the authors to consider organizing the algorithms discussed in the paper by creating a table. This table can help provide a clear overview of the different algorithms, highlighting their similarities and differences.\n\nQUESTIONS:\nThe thoerem need $\\eta$ to be small. However, in the matrix game experiment, $\\eta$ is chose as $10$. Is $\\eta = 10$ too big in this setting?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}