Title: Uncoupled and Convergent Learning in Two-Player Zero-Sum Markov Games with Bandit Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: two-player zero-sum Markov game

last-iterate convergence

path convergence

learning in games
Abstract: We revisit the problem of learning in two-player zero-sum Markov games, focusing on developing an algorithm that is *uncoupled*, *convergent*, and *rational*, with non-asymptotic convergence rates to Nash equilibrium. We start from the case of stateless matrix game with bandit feedback as a warm-up, showing an $\tilde{\mathcal{O}}(t^{-\frac{1}{8}})$ last-iterate convergence rate. To the best of our knowledge, this is the first result that obtains finite last-iterate convergence rate given access to only bandit feedback. We extend our result to the case of irreducible Markov games, providing a last-iterate convergence rate of $\tilde{\mathcal{O}}(t^{-\frac{1}{9+\varepsilon}})$ for any $\varepsilon>0$. Finally, we study Markov games without any assumptions on the dynamics, and show a *path convergence* rate, a new notion of convergence we defined, of $\tilde{\mathcal{O}}(t^{-\frac{1}{10}})$. Our algorithm removes the synchronization and prior knowledge requirement of Wei et al. (2021), which pursued the same goals as us for irreducible Markov games. Our algorithm is related to Chen et al. (2021) and Cen et al. (2021)'s and also builds on the entropy regularization technique. However, we remove their requirement of communications on the entropy values, making our algorithm entirely uncoupled.

Uncoupled and Convergent Learning in Two-Player Zero-Sum Markov Games with Bandit Feedback

Abstract

We revisit the problem of learning in two-player zero-sum Markov games, focusing on developing an algorithm that is uncoupled , convergent , and rational , with nonasymptotic convergence rates to Nash equilibrium. We start from the case of stateless matrix game with bandit feedback as a warm-up, showing an O ( t -1 8 ) last-iterate convergence rate. To the best of our knowledge, this is the first result that obtains finite last-iterate convergence rate given access to only bandit feedback. We extend our result to the case of irreducible Markov games, providing a lastiterate convergence rate of O ( t -1 9+ ε ) for any ε > 0 . Finally, we study Markov games without any assumptions on the dynamics, and show a path convergence rate, a new notion of convergence we define, of O ( t -1 10 ) . Our algorithm removes the coordination and prior knowledge requirement of [WLZL21a], which pursued the same goals as us for irreducible Markov games. Our algorithm is related to [CMZ21, CWC21] and also builds on the entropy regularization technique. However, we remove their requirement of communications on the entropy values, making our algorithm entirely uncoupled.

1 Introduction

In multi-agent learning, a central question is how to design algorithms so that agents can independently learn (i.e., with little coordination overhead) how to interact with each other. Additionally, it is desirable to maximally reuse existing single-agent learning algorithms, so that the multi-agent system can be built in a modular way. Motivated by this question, decentralized multi-agent learning emerges with the goal to design decentralized systems, in which no central controller governs the policies of the agents, and each agent learns based on only their local information - just like in a single-agent algorithm. In recent years, we have witnessed significant success of this new decentralized learning paradigm. For example, self-play , where each agent independently deploys the same single-agent algorithm to play against each other without further direct supervision, plays a crucial role in the training of AlphaGo [SSS + 17] and AI for Stratego [PDVH + 22].

Despite the recent success, many important questions remain open in decentralized multi-agent learning. Indeed, unless the decentralized algorithm is carefully designed, self-play often falls short of attaining certain sought-after global characteristics, such as convergence to the global optimum or stability as seen in, for example, [MPP18, BP18].

In this work, we revisit the problem of learning in two-player zero-sum Markov games, which has received extensive attention recently. Our goal is to design a decentralized algorithm that resembles

standard single-agent reinforcement learning (RL) algorithms, but with an additional crucial assurance, that is, guaranteed convergence when both players deploy the algorithm. The simultaneous pursuit of independence and convergence has been advocated widely [BV01, AY16, WLZL21a, SZL + 21], while the results are still not entirely satisfactory. In particular, all of these results rely on assumptions on the dynamics of the Markov game. Our paper takes the first step to remove such assumptions.

More specifically, our goal is to design algorithms that simultaneously satisfy the following three properties (the definitions are adapted from [BV01, DDK11]):

Uncoupled : Each player i 's action is generated by a standalone procedure P i which, in every round, only receives the current state and player i 's own reward as feedback (in particular, it has no knowledge about the actions or policies used by the opponent). There is no communication or shared randomness between the players.

Convergent : The policy pair of the two players converges to a Nash equilibrium.

Rational : If P i competes with an opponent who uses a policy sequence that converges to a stationary one, then P i converges to the best response of this stationary policy.

The uncoupledness and rationality property capture the independence of the algorithm, while the convergence property provides a desirable global guarantee. Interestingly, as argued in [WLZL21a], if an algorithm is uncoupled and convergent, then it is also rational, so we only need to ensure that the algorithm is uncoupled and convergent. Regarding the notion of convergence, the standard definition above only allows last-iterate convergence. Considering the difficulty of achieving such convergence, in the related work review (Section 2) and in the design of our algorithm for general Markov games (Section 6), we also consider weaker notions of convergence, including the best-iterate convergence, which only requires that the Cesaro mean of the duality gap is convergent, and the path convergence, which only requires the convergence of the Cesaro mean of the duality gap assuming minimax/maximin policies are followed in future steps . The precise definitions of these convergence notions are given at the end of Section 3.

1.1 Our Contributions

The main results in this work are as follows (see also Table 1 for comparisons with prior works):

As a warm-up, for the special case of matrix games with bandit feedback, we develop an uncoupled algorithm with a last-iterate convergence rate of O ( t -1 8 ) under self-play (Section 4). To the best of our knowledge, this is the first algorithm with provable last-iterate convergence rate in the setting.

Generalizing the ideas from matrix games, we further develop an uncoupled algorithm for irreducible Markov games with a last-iterate convergence rate of O ( t -1 9+ ε ) for any ε > 0 under self-play (Section 5).

Finally, for general Markov games without additional assumptions, we develop an uncoupled algorithm with a path convergence rate of O ( t -1 10 ) under self-play (Section 6).

Our algorithms leverage recent advances on using entropy to regularize the policy updates [CWC21, CMZ21] and the Nash-V-styled value updates [BJY20]. On the one hand, compared to [CWC21, CMZ21], our algorithm has the following advantages: 1) it does not require the two players to exchange their entropy information, which allows our algorithm to be fully uncoupled; 2) it does not require the players to have coordinated policy updates, 3) it naturally extends to general Markov games without any assumptions on the dynamics (e.g., irreducibility). On the other hand, our algorithm inherits appealing properties of Nash-V [BJY20], but additionally guarantees path convergence during execution.

2 Related Work

The study of two-player zero-sum Markov games originated from [Sha53], with many other works further developing algorithms and establishing convergence properties [HK66, PAI69, VDW78, FT91]. However, these works primarily focused on solving the game with full knowledge of its parameters (i.e., payoff function and transition kernel). The problem of learning in zero-sum games was first formalized by [Lit94]. Designing a provably uncoupled, rational, and convergent algorithm

is challenging, with many attempts [SL99, BV01, HW03, CS07, AY16, SPO22] falling short in one aspect or another, often lacking either uncoupledness or convergence. Moreover, these works only establish asymptotic convergence without providing a concrete convergence rate.

2.1 Non-asymptotic convergence guarantees

Recently, a large body of works on learning two-player zero-sum Markov games use regret minimization techniques to establish non-asymptotic guarantees. They focus on fast computation under full information of payoff and transitions [CWC21, CCDX23, ZLW + 22, SLY23, YM23], though many of their algorithms are decentralized and can be viewed as the first step towards the learning setting.

With rationality and uncoupledness satisfied, [DFG20] established one-sided policy convergence for players using independent policy gradient with asymmetric learning rates. Such an asymmetric update rule is also adopted by [ZTLD22, AVHC22] to establish one-sided policy convergence guarantees. When using a symmetric update rule, [SZL + 21] developed a decentralized-Q learning algorithm. However, the convergence is only shown for the V -function maintained by the players instead of the policies being used, so the policies may still cycle and are not provably convergent in our definition. [ELS + 23] studied regret minimization in general-sum Markov games and provided an algorithm with sublinear regret under self-play and average-iterate convergence rates to equibria, while our work focuses on last-iterate convergence rates to Nash equilibria.

To our knowledge, [WLZL21a] first provided an uncoupled, rational, and convergent algorithm with non-asymptotic convergence guarantee, albeit only for irreducible Markov game. They achieved this via optimistic gradient descent/ascent . Despite satisfying all our criteria, their algorithm still has unnatural coordination between the players and a requirement on some prior knowledge of the game such as the maximum revisiting time of the Markov game. Our algorithm removes all these extra requirements. A follow-up work by [CMZ21] improved the rate of [WLZL21a] using entropy regularization; however, this requires their players to inform the opponent about the entropy of their own policy, making the algorithm coupled again. We show that such an exchange of information is unnecessary under entropy regularization.

2.2 Further handling exploration

The algorithms introduced above all require full information or some assumption on the dynamics of the Markov game. To handle exploration, some works design coupled learning algorithms

which guarantee that the player's long-term payoff is at least the minimax value [BT02, WHL17, XCWY20, HLWY22, JLY22, JJJN21, XZS + 22]. Interestingly, as shown in [WHL17, HLWY22, JLY22, XZS + 22], if the player is paired with an optimistic best-response opponent (instead of using the same algorithm), the first player's strategy can converge to the minimax policy. [XCWY20, BJ20, LYBJ21, CZG22] developed another coupled learning framework to handle exploration, but with symmetric updates on both players. In each round, the players need to jointly solve a generalsum equilibrium problem due to the different exploration bonus added by each player. Hence, the execution of these algorithms is more similar to the Nash-Q algorithm by [HW03].

So far, exploration has been handled through coupled approaches that are also not rational. To our knowledge, the first uncoupled and rational algorithm that handles exploration is the Nash-V algorithm by [BJY20]. Nash-V can output a nearly-minimax policy through weighted averaging [JLWY21]; however, it is not provably convergent during execution. A major remaining open problem is whether one can design a natural algorithm that is provably rational, uncoupled, and convergent with exploration capability. Our work provides the first progress towards this goal.

2.3 Other works on last-iterate convergence

Uncoupled Learning dynamics in normal-form games with provable last-iterate convergence rate receives extensive attention recently. Most of the works assume that the players receive gradient feedback, and convergence results under bandit feedback remain sparse. Linear convergence is shown for strongly monotone games or bilinear games under gradient feedback [Tse95, LS19, MOP20, WLZL21b] and sublinear rates are proven for strongly monotone games with bandit feedback [BLM18, HIMM19, LZBZ21, TK22, DFR22, HH23]. Convergence rate to strict Nash equilibrium is analyzed by [GVGM21]. For monotone games that includes two-player zero-sum games as a special case, the last-iterate convergence rate of no-regret learning under gradient feedback has been shown recently [GPD20, COZ22, GTG22, CZ23]. With bandit feedback, [MPS20] showed an impossibility result that certain algorithms with optimal O ( √ T ) regret do not converge in last-iterate. To the best of our knowledge, there is no natural uncoupled learning dynamics with provable last-iterate convergence rate in two-player zero-sum games with bandit feedback.

3 Preliminaries

Basic Notations Throughout the paper, we assume for simplicity that the action set for the two players are the same, denoted by A with cardinality A = |A| . 1 We usually call player 1 the x -player and player 2 the y -player. The set of mixed strategies over an action set A is denoted as ∆ A := { x : ∑ a ∈A x a = 1;0 ≤ x a ≤ 1 , ∀ a ∈ A} . To simplify notation, we denote by z = ( x, y ) the concatenated strategy of the players. We use ϕ as the entropy function such that ϕ ( x ) = -∑ a ∈A x a ln x a , and KL as the Kullback-Leibler (KL) divergence such that KL ( x, x ′ ) = ∑ a ∈A x a ln x a x ′ a . The all-one vector is denoted by 1 = (1 , 1 , · · · , 1) .

Matrix Games In a two-player zero-sum matrix game with a loss matrix G ∈ [0 , 1] A × A , when the x -player chooses action a and the y -player chooses action b , the x -player suffers loss G a,b and the y -player suffers loss -G a.b . A pair of mixed strategy ( x ⋆ , y ⋆ ) is a Nash equilibrium for G if for any strategy profile ( x, y ) ∈ ∆ A × ∆ A , it holds that ( x ⋆ ) ⊤ Gy ≤ ( x ⋆ ) ⊤ Gy ⋆ ≤ x ⊤ Gy ⋆ . Similarly, ( x ⋆ , y ⋆ ) is a Nash equilibrium for a two-player zero-sum game with a general convex-concave loss function f ( x, y ) : ∆ A × ∆ A → R if for all ( x, y ) ∈ ∆ A × ∆ A , f ( x ⋆ , y ) ≤ f ( x ⋆ , y ⋆ ) ≤ f ( x, y ⋆ ) . The celebrated minimax theorem [vN28] guarantees the existence of Nash equilibria in two-player zero-sum games. For a pair of strategy ( x, y ) , we use duality gap defined as GAP ( G,x,y ) ≜ max y ′ x ⊤ Gy ′ -min x ′ x ′⊤ Gy to measure its proximity to Nash equilibria.

Markov Games A generalization of matrix games, which models dynamically changing environment, is Markov games . We consider infinite-horizon discounted two-player zero-sum Markov games, denoted by a tuple ( S , A , ( G s ) s ∈S , ( P s ) s ∈S , γ ) where (1) S is a finite state space; (2) A is a finite action space for both players; (3) Player 1 suffers loss G s a,b ∈ [0 , 1] (respectively player 2 suffers

loss -G s a,b ) when player 1 chooses action a and player 2 chooses action b at state s ; (4) P is the transition function such that P s a,b ( s ′ ) is the probability of transiting to state s ′ when player 1 plays a and player 2 plays b at state s ; (5) γ ∈ [ 1 2 , 1) is a discount factor.

A stationary policy for player 1 is a mapping S → ∆ A that specifies player 1's strategy x s ∈ ∆ A at each state s ∈ S . We denote x = ( x s ) s ∈S . Similar notations apply to player 2. We denote z s = ( x s , y s ) as the concatenated strategy for the players and z = ( x, y ) . The value function V s x,y denotes the expected loss of player 1 (or the expected payoff of player 2) given a pair of stationary policy ( x, y ) and initial state s :

The minimax game value on state s is defined as V s ⋆ = min x max y V s x,y = max y min x V s x,y . We call a pair of policy ( x ⋆ , y ⋆ ) a Nash equilibrium if it attains minimax game value of a state s (such policy pair necessarily attains the minimax game value over all states). The duality gap of ( x, y ) is max s (max y ′ V s x,y ′ -min x ′ V s x ′ ,y ) . The Q -function on state s under policy pair ( x, y ) is defined via Q s x,y ( a, b ) = G s a,b + γ · E s ′ ∼ P s a,b ( · ) [ V s ′ x,y ] , which can be rewritten as a matrix Q s x,y such that V s x,y = x s Q s x,y y s . We denote Q s ⋆ = Q s x ⋆ ,y ⋆ the Q -function under a Nash equilibrium ( x ⋆ , y ⋆ ) . It is known that Q s ⋆ is unique for any s even when multiple equilibria exist.

Uncoupled Learning with Bandit Feedback We assume the following uncoupled interaction protocol: at each round t = 1 , . . . , T , the players both observe the current state s t , and then, with the policy x t and y t in mind, they independently choose actions a t ∼ x s t t and b t ∼ y s t t , respectively. Both of them then observe σ t ∈ [0 , 1] with E [ σ t ] = G s t a t ,b t , and proceed to the next state s t +1 ∼ P s t a t ,b t ( · ) . Importantly, they do not observe each other's action.

Notions of Convergence For Markov games with the irreducible assumption (Assumption 1), given players' history of play ( s t , x t , y t ) t ∈ [ T ] , the best-iterate convergence rate is measured by the average duality gap 1 T ∑ T t =1 max s,x,y ( V s x t ,y -V s x,y t ) , while the stronger last-iterate convergence rate is measured by max s,x,y ( V s x T ,y -V s x,y T ) , i.e., the duality gap of ( x T , y T ) . For general Markov games, we propose the path convergence rate, which is measured by the average duality gap at the visited states with respect to the optimal Q -function: 1 T ∑ T t =1 max x,y ( x s ⊤ t t Q s t ⋆ y s t -x s ⊤ t Q s t ⋆ y s t t ) . We remark that the path convergence guarantee is weaker than the counterpart of the other two notions of convergence in general Markov games, but still provides meaningful implications (see detailed discussion in Section 6.1 and Appendix F).

4 Matrix Games

In this section, we consider two-player zero-sum matrix games. We propose Algorithm 1 for decentralized learning of Nash equilibria. We only present the algorithm for the x -player as the algorithm for the y -player is symmetric.

Algorithm 1 Matrix Game with Bandit Feedback

1: Define: η t = t -k η , β t = t -k β , ϵ t = t -k ϵ where k η = 5 8 , k β = 3 8 , k ϵ = 1 8 . Ω t = { x ∈ ∆ A : x a ≥ 1 At 2 , ∀ a ∈ A } . 2: Initialization: : x 1 = 1 A 1 . 3: for t = 1 , 2 , . . . do 4: Sample a t ∼ x t , and receive σ t ∈ [0 , 1] with E [ σ t ] = G a t ,b t . 5: Compute g t where g t,a = 1 [ a t = a ] σ t x t,a + β t + ϵ t ln x t,a , ∀ a ∈ A . 6: Update x t +1 ← argmin x ∈ Ω t +1 { x ⊤ g t + 1 η t KL ( x, x t ) } .

7: end for

The algorithm is similar to the EXP3-IX algorithm by [Neu15] that achieves a high-probability regret bound for adversarial multi-armed bandits, but with several modifications. First (and most

importantly), in addition to the standard loss estimators used in [Neu15], we add another negative term ϵ t ln x t,a to the loss estimator of action a (see Line 5). This is equivalent to the entropy regularization approach in, e.g., [CWC21, CMZ21], since the gradient of the negative entropy -ϕ ( x t ) is (ln x t,a +1) a ∈A and the constant 1 takes no effect in Line 6. Like [CWC21, CMZ21], the entropy regularization drives last-iterate convergence; however, while their results require fullinformation feedback, our result holds in the bandit feedback setting. The second difference is that instead of choosing the players' strategies in the full probability simplex ∆ A , our algorithm chooses from Ω t , a subset of ∆ A where every coordinate is lower bounded by 1 At 2 . The third is the choices of the learning rate η t , clipping factor β t , and the amount of regularization ϵ t . The main result of this section is the following last-iterate convergence rate of Algorithm 1.

Theorem 1 (Last-Iterate Convergence Rate) . Algorithm 1 guarantees with probability at least 1 -O ( δ ) , for any t ≥ 1 ,

Algorithm 1 also guarantees O ( t -1 8 ) regret even when the other player is adversarial. If we only target at an expected bound instead of a high-probability bound, the last-iterate convergence rate can be improved to O ( √ A ln 3 / 2 ( At ) t -1 6 ) . The details are provided in Appendix C.

4.1 Analysis Overview

We define a regularized zero-sum game with loss function f t ( x, y ) = x ⊤ Gy -ϵ t ϕ ( x ) + ϵ t ϕ ( y ) over domain Ω t × Ω t , and denote by z ⋆ t = ( x ⋆ t , y ⋆ t ) its unique Nash equilibrium since f t is strongly convex-strongly concave. The regularized game is a slight perturbation of the original matrix game G over a smaller domain Ω t × Ω t , and we prove that z ⋆ t is an O ( ϵ t ) -approximate Nash equilibrium of the original matrix game G (Lemma 9). Therefore, it suffices to bound KL ( z ⋆ t , z t ) since the duality gap of z t is at most O ( √ KL ( z ⋆ t , z t ) + ϵ t ) .

Step 1: Single-Step Analysis We start with a single-step analysis of Algorithm 1, which shows:

where we define v t = KL ( z ⋆ t +1 , z t +1 ) -KL ( z ⋆ t , z t +1 ) (see Appendix B for definitions of λ t , ξ t , ζ t ) The instability penalty comes from some local-norm of the gradient estimator g t . The estimation error comes from the bias between the gradient estimator g t and the real gradient Gy t . We pay the last term v t since the Nash equilibrium z ∗ t of the regularized game f t is changing over time.

Step 2: Strategy Convergence to NE of the Regularized Game Expanding the above recursion up to t 0 , we get

where w i t ≜ ∏ t j = i +1 (1 -η j ϵ j ) . To upper bound term 1 -term 4 , we apply careful sequence analysis (Appendix A.1) and properties of the EXP3-IX algorithm with changing step size (Appendix A.2). The analysis of term 5 uses Lemma 13, which states v t = KL ( z ⋆ t +1 , z t +1 ) -KL ( z ⋆ t , z t +1 ) ≤ O (ln( At ) ∥ z ⋆ t +1 -z ⋆ t ∥ 1 ) = O ( ln 2 ( At ) t ) and is slightly involved as Ω t and ϵ t are both changing. With these steps, we conclude that with probability at least 1 -O ( δ ) , KL ( z ⋆ t , z t ) = O ( A ln 3 ( At/δ ) t -1 4 ) .

5 Irreducible Markov Games

We now extend our results on matrix games to two-player zero-sum Markov games. Similarly to many previous works, our first result makes the assumption that the Markov game is irreducible with bounded travel time between any pair of states. The assumption is formally stated below:

Assumption 1 (Irreducible Game) . We assume that under any pair of stationary policies of the two players, and any pair of states s, s ′ , the expected time to reach s ′ from s is upper bounded by L .

We propose Algorithm 2 for uncoupled learning in irreducible two-player zero-sum games, which is closely related to the Nash-V algorithm by [BJY20], but with additional entropy regularization. It can also be seen as players using Algorithm 1 on each state s to update the policies ( x s t , y s t ) whenever state s is visited, but with σ t + γV s t +1 t as the observed loss to construct loss estimators. Importantly, V s 1 , V s 2 , . . . is a slowly changing sequence of value estimations that ensures stable policy updates [BJY20, WLZL21a, SZL + 21]. Note that in Algorithm 2, the updates of V s t only use players' local information (Line 8).

Algorithm 2 Irreducible Markov Game

1: Define: η t = (1 -γ ) t -k η , β t = t -k β , ϵ t = 1 1 -γ t -k ϵ , α t = t -k α with k α , k ϵ , k β , k η ∈ (0 , 1) , Ω t = { x ∈ ∆ A : x a ≥ 1 At 2 , ∀ a ∈ A } .

2: Initialization: x s 1 ← 1 A 1 , n s 1 ← 0 , V s 1 ← 1 2(1 -γ ) , ∀ s .

3: for t = 1 , 2 , . . . , do

5: Draw a t ∼ x s t t , observe σ t ∈ [0 , 1] with E [ σ t ] = G s t a t ,b t , and observe s t +1 ∼ P s t a t ,b t ( · ) .

4: τ = n s t t +1 ← n s t t +1 (the number of visits to state s t up to time t ).

6: Compute g t where g t,a = 1 [ a t = a ] ( σ t + γV s t +1 t ) x s t t,a + β τ + ϵ τ ln x s t t,a , ∀ a ∈ A .

7: Update x s t t +1 ← argmin x ∈ Ω τ +1 { x ⊤ g t + 1 η τ KL ( x, x s t t ) } .

8:

Update

9:

For all

V

s

10: end for

Comparison to Previous Works Although Algorithm 2 shares similarity with previous works that also use entropy regularization, we believe that both the design and the analysis of our algorithm are novel and non-trivial. To the best of our knowledge, all previous entropy regularized two-player zero-sum Markov game algorithms are coupled (e.g., [CWC21, CMZ21, CCDX23]), while ours is the first that achieves uncoupledness under entropy regularization. We further discuss this by comparing our algorithm to those in [CCDX23], highlighting the new technical challenges we encounter.

The entropy-regularized OMWU algorithm in [CCDX23] is tailored to the full-information setting. Moreover, in the value function update step both players need to know the entropy value of the other player's policy, which is unnatural. Indeed, the authors explicitly present the removal of this information sharing as an open question. We answer this open question affirmatively by giving a fully decentralized algorithm for zero-sum Markov games with provable last-iterate convergence rates. In Algorithm 2 (Line 8), the update of the value function V is simple and does not require any entropy information: V s t t +1 ← (1 -α τ ) V s t t + α τ ( σ t + γV s t +1 t ) . This modification results in a discrepancy between the policy update and the value update. While the policy now incorporates a regularization term, the value function does not. Such a mismatch is unprecedented in earlier studies and necessitates a non-trivial approach to resolve. Additionally, Algorithm 2 operates on bandit feedback instead of full-information feedback, presenting further technical challenges.

Algorithm 2 also offers improvement over the uncoupled algorithm of [WLZL21a]. The algorithm of [WLZL21a] requires coordinated policy update where the players interact with each other using the current policy for several iterations to get an approximately accurate gradient (the number of iterations required depends on L as defined in Assumption 1), and then simultaneously update the policy pair on all states. We do not require such unnatural coordination between the players or prior knowledge on L .

Our main result is the following theorem on the last-iterate convergence rate of Algorithm 2.

Theorem 2 (Last-Iterate Convergence Rate) . For any ε, δ > 0 , Algorithm 2 with k α = 9 9+ ε , k ϵ = 1 9+ ε , k β = 3 9+ ε , and k η = 5 9+ ε guarantees, with probability at least 1 -O ( δ ) , for any time t ≥ 1 ,

s

t

t

+1

̸

=

s

t

,

(1

s

x

t

-

+1

α

)

τ

x

V

s

t

s

t

,

n

t

+

s

t

α

+1

τ

(

σ

t

n

t

+

,

γV

V

s

s

t

t

+1

t

+1

)

.

V

t

s

s

.

5.1 Analysis Overview

We introduce some notations for simplicity. We denote by E s ′ ∼ P s [ V s ′ t ] the A × A matrix such that ( E s ′ ∼ P s [ V s ′ t ]) a,b = E s ′ ∼ P s a,b [ V s ′ t ] . Let t τ ( s ) be the τ -th time the players visit state s , and define ˆ x s τ = x s t τ ( s ) and ˆ y s τ = y s t τ ( s ) . Then, define the regularized game for each state s via the loss function f s τ ( x, y ) = x ⊤ ( G s + γ E s ′ ∼ P s [ V s ′ t τ ( s ) ]) y -ϵ τ ϕ ( x ) + ϵ τ ϕ ( y ) . Furthermore, let ˆ z s τ⋆ = (ˆ x s τ⋆ , ˆ y s τ⋆ ) be the equilibrium of f s τ ( x, y ) over Ω τ × Ω τ . In the following analysis, we fix some t ≥ 1 .

Step 1: Policy Convergence to NE of Regularized Game Using similar techniques to Step 1 and Step 2 in the analysis of Algorithm 1, we can upper bound KL (ˆ z s τ +1 ⋆ , ˆ z s τ +1 ) like Eq. (1) with similar subsequent analysis for term 1 -term 4 . The analysis for term 5 where v s i = KL (ˆ z s i +1 ⋆ , ˆ z s i +1 ) -KL (ˆ z s i⋆ , ˆ z s i +1 ) is more challenging compared to the matrix game case since here V s t i ( s ) is changing between two visits to state s . To handle this term, we leverage the following facts for any s ′ : (1) the irreducibility assumption ensures that t i +1 ( s ) -t i ( s ) ≤ O ( L ln( St/δ )) thus the number of updates of the value function at state s ′ is bounded; (2) until time t i ( s ) ≥ i , state s ′ has been visited at least Ω( i L ln( St/δ ) ) times thus each change of the value function between t i ( s ) and t i +1 ( s ) is at most O (( i L ln( St/δ ) ) -k α ) . With these arguments, we can bound term 5 by O ( ln 4 ( SAt/δ ) Lτ -k α + k η +2 k ϵ ) . Overall, we have the following policy convergence of NE of the regularized game (Lemma 17): KL (ˆ z s τ⋆ , ˆ z s τ ) ≤ O ( A ln 4 ( SAt/δ ) Lτ -k ♯ ) , where k ♯ = min { k β -k ϵ , k η -k β , k α -k η -2 k ϵ } .

Step 2: Value Convergence Unlike matrix games, policy convergence to NE of the regularized game is not enough for convergence in duality gap. We also need to bound | V s t -V s ⋆ | since the regularized game is defined using V s t , the value function maintained by the algorithm, instead of the minimax game value V s ⋆ . We use the following weighted regret quantities as a proxy: Reg s τ ≜ max x,y ( ∑ τ i =1 α i τ ( f s i (ˆ x s i , ˆ y s i ) -f s i ( x s , ˆ y s i )) , ∑ τ i =1 α i τ ( f s i (ˆ x s i , y s i ) -f s i (ˆ x s i , ˆ y s i )) ) , where α i τ = α i ∏ τ j = i +1 (1 -α j ) . We can upper bound the weighted regret Reg s τ using a similar analysis as in Step 1 (Lemma 19). We then show a contraction for | V s t τ ( s ) -V s ⋆ | with the weighted regret quantities: | V s t τ ( s ) -V s ⋆ | ≤ γ ∑ τ i =1 α i τ max s ′ | V s ′ t i ( s ) -V s ′ ⋆ | + ˜ O ( ϵ τ + Reg s τ ) . This leads to the following convergence of V s t (Lemma 20): | V s t -V s ⋆ | ≤ ˜ O ( t -k ∗ ) , where k ∗ = min { k η , k β , k α -k β , k ϵ } .

Obtaining Last-Iterate Convergence Rate Fix any t and let τ be the number of visits to s before time t . So far we have shown (1) policy convergence of KL (ˆ z s τ⋆ , ˆ z s τ ) in the regularized game; (2) and value convergence of | V s t -V s ⋆ | . Using the fact that the regularized game is at most O ( ϵ τ + | V s t -V s ⋆ | ) away from the minimax game martrix Q ⋆ and appropriate choices of parameters proves Theorem 2.

6 General Markov Games

In this section, we consider general two-player zero-sum Markov games without Assumption 1. We propose Algorithm 3, an uncoupled learning algorithm that handles exploration and has path convergence rate. Compared to Algorithm 2, the update of value function in Algorithm 3 uses a bonus term bns τ based on the optimism principle to handle exploration.

Theorem 3 below implies that we can achieve 1 t ∑ t τ =1 max x,y ( x s ⊤ τ τ Q s τ ⋆ y s τ -x s ⊤ τ Q s τ ⋆ y s τ τ ) = O ( t -1 10 ) path convergence rate if we use the doubling trick to tune down u at a rate of t -1 10 .

Theorem 3. For any u ∈ [ 0 , 1 1 -γ ] and T ≥ 1 , there exists a proper choice of parameters ϵ, β, η such that Algorithm 3 guarantees with probability at least 1 -O ( δ ) ,

Algorithm 3 General Markov Game

1: Input: η ≤ β ≤ ϵ and T .

2: Define: Ω = { x ∈ ∆ A : x a ≥ 1 AT , ∀ a ∈ A } , α τ = H +1 H + τ , where H = ln( T ) 1 -γ .

bns τ = κA ln 2 ( SAT/δ )( β + η -1 α τ ) / (1 -γ ) 2 for a sufficiently large absolute constant κ > 0 s s s 1 .

4: for t = 1 , 2 , . . . , do

3: Initialization: V 1 , n 1 ← 0 , x 1 ← A 1 , ∀ s

5: τ = n s t t +1 ← n s t t +1 .

7: Compute g t where g t,a = 1 [ a t = a ] ( σ t + γV s t +1 t ) x s t t,a + β + ϵ ln x s t t,a , ∀ a ∈ A .

6: Sample a t ∼ x s t t , observe σ t ∈ [0 , 1] with E [ σ t ] = G s t a t ,b t , and observe s t +1 ∼ P s t a t ,b t ( · ) .

8: Update x s t t +1 ← argmin x ∈ Ω { x ⊤ g t + 1 η KL ( x, x s t t ) } .

9: Update ∼ V s t t +1 ← (1 -α τ ) ∼ V s t t + α τ ( σ t + γV s t +1 t -bns τ ) and V s t t +1 ← max { ∼ V s t t +1 , 0 } .

̸

10: For all s = s t , x s t +1 ← x s t , V s t +1 ← V s t , ∼ V s t +1 ← ∼ V s t , n s t +1 ← n s t .

11: end for

6.1 Path Convergence

Path convergence has multiple meaningful game-theoretic implications. By definition, It implies that frequent visits to a state bring players' policies closer to equilibrium, leading to both players using near-equilibrium policies for all but o ( T ) number of steps over time.

Path convergence also implies that both players have no regret compared to the game value V s ⋆ , which has been considered and motivated in previous works such as [BT02, TWYS20]. To see this, we apply the results to the episodic setting, where in every step, with probability 1 -γ , the state is redrawn from s ∼ ρ for some initial distribution ρ . If the learning dynamics enjoys path convergence, then E [ ∑ T t =1 x s ⊤ t t G s t y s t t ] = (1 -γ ) E s ∼ ρ [ V s ⋆ ] T ± o ( T ) . Hence the one-step average reward is (1 -γ ) E s ∼ ρ [ V s ⋆ ] and both players have no regret compared to the game value. A more important implication of path convergence is that it guarantees stability of players' policies, while cycling behaviour is inevitable for any FTRL-type algorithms even in zero-sum matrix games [MPP18, BP18]. We defer the proof and more discussion of path convergence to Appendix F.

Finally, we remark that our algorithm is built upon Nash V-learning [BJY20], so it inherits properties of Nash V-learning, e.g., one can still output near-equilibrium policies through policy averaging [JLWY21], or having no regret compared to the game value when competing with an arbitrary opponent [TWYS20]. We demonstrate extra benefits brought by entropy regularization regarding the stability of the dynamics.

6.2 Analysis Overview of Theorem 3

For general Markov games, it no longer holds that every state s is visited often, and thus the analysis is much more challenging. We first define two regularized games based on V s t and the corresponding quantity V s t for the y -player. Define t τ ( s ) , ˆ x s τ , ˆ y s τ the same way as in the previous section. Then define f s ( x, y ) ≜ x ⊤ ( G s + γ E s ′ ∼ P s [ V s ′ ]) y -ϵϕ ( x )+ ϵϕ ( y ) , f s ( x, y ) ≜ x ⊤ ( G s

τ t τ ( s ) τ + γ E s ′ ∼ P s [ V s ′ t τ ( s ) ]) y -ϵϕ ( x ) + ϵϕ ( y ) and denote J t = max x,y ( x s ⊤ t t ( G s t + γ E s ′ ∼ P s t [ V s ′ t ] y s t -x s ⊤ t t ( G s + γ E s ′ ∼ P s t [ V s ′ t ]) y s t t ) . We first bound the 'path duality gap" as follows

Value Convergence: Bounding V s t -V s ⋆ and V s ⋆ -V s t This step is similar to Step 2 in the analysis of Algorithm 2. We first show an upper bound of the weighted regret (Lemma 23): ∑ τ i =1 α i τ ( f s i (ˆ x s i , ˆ y s i ) -f s i ( x s , ˆ y s i )) ≤ 1 2 bns τ , where α i τ = α i ∏ τ j = i +1 (1 -α j ) . Note that the value function V s t is updated using σ t + γV s t +1 t -bns τ . Thus when relating | V s t -V s ⋆ | to the regret, the regret term and the bonus term cancel out and we get V s t ≤ V s ⋆ + O ( ϵ ln( AT ) 1 -γ ) (Lemma 26). The analysis for V s ⋆ -V s t is symmetric. By proper choice of ϵ , both terms are bounded by 1 8 u . Combining

the above with Eq. (3), we can upper bound the left-hand side of the desired inequality Eq. (2) by ∑ T t =1 1 [ J t ≥ 3 4 u ] , which is further upper bounded in Eq. (29) by

Policy Convergence to NE of Regularized Games To bound the first two terms, we show convergence of the policy (ˆ x s τ , ˆ y s τ ) to Nash equilibria of both games f s τ and f s τ . To this end, fix any p ∈ [0 , 1] , we define f s τ = pf s τ + (1 -p ) f s τ and let ˆ z s τ⋆ = (ˆ x s τ⋆ , ˆ y s τ⋆ ) be the equilibrium of f s τ ( x, y ) . The analysis is similar to previous algorithms where we first conduct single-step analysis (Lemma 22) and then carefully bound the weighted recursive terms. We show in Lemma 27 that for any 0 < ϵ ′ ≤ 1 : ∑ s ∑ n T +1 ( s ) τ =1 1 [ KL (ˆ z s τ⋆ , ˆ z s τ ) ≥ ϵ ′ ] ≤ O ( S 2 A ln 5 ( SAT/δ ) ηϵ 2 ϵ ′ (1 -γ ) 3 ) . This proves policy convergence: the number of iterations where the policy is far away from Nash equilibria of the regularized games is bounded, which can then be translated to upper bounds on the first two terms.

Value Convergence: Bounding | V s t -V s t | It remains to bound the last term in Eq. (4). Define c t = 1 [ x s t t ( E s ′ ∼ P s t [ V s ′ t -V s ′ t ]) y s t t ≥ ˜ ϵ ] where ˜ ϵ = u 4 . Then we only need to bound C ≜ ∑ T t =1 c t . We use the weighted sum P T ≜ ∑ T t =1 c t x s t t ( E s ′ ∼ P s t [ V s ′ t -V s ′ t ]) y s t t as a proxy. On the one hand, P T ≥ C ˜ ϵ . On the other hand, in Lemma 25, by recursively tracking the update of the value function and carefully choosing η and β , we upper bound P T by ≤ C ˜ ϵ 2 + O ( AS ln 4 ( AST/δ ) η (1 -γ ) 3 ) . Combining the upper and lower bound of P T gives C ≤ O ( AS ln 4 ( AST/δ ) ηu (1 -γ ) 3 ) (Corollary 2). Plugging appropriate choices of ϵ , η , and β in the above bounds proves Theorem 3 (see Appendix E).

7 Conclusion and Future Directions

In this work, we study decentralized learning in two-player zero-sum Markov games with bandit feedback. We propose the first uncoupled and convergent algorithms with non-asymptotic last-iterate convergence rates for matrix games and irreducible Markov games, respectively. We also introduce a novel notion of path convergence and provide algorithm with path convergence in Markov games without any assumption on the dynamics. Previous results either focus on average-iterate convergence or require stronger feedback/coordination or lack non-asymptotic convergence rates. Our results contribute to the theoretical understanding of the practical success of regularization and last-iterate convergence in multi-agent reinforcement learning.

Settling the optimal last-iterate convergence rate that is achievable by uncoupled learning dynamics is an important open question. The following directions are promising towards closing the gap between current upper bounds O ( T -1 / 8 ) and O ( T -1 / (9+ ε ) ) and lower bound Ω( T -1 2 ) , The impossibility result by [MPS20] demonstrates that certain algorithms with O ( √ T ) regret diverge in last-iterate. Their result indicates that the current Ω( 1 √ T ) lower bound on convergence rate may not be tight. On the other hand, our algorithms provides insights and useful templates to potential improvements on the upper bound. For instance, instead of using EXP3-IX update, adapting optimistic policy update or other accelerated first-order methods to the bandit feedback setting is an interesting future direction.

Acknowledgement We thank Chanwoo Park and Kaiqing Zhang for pointing out a mistake in our previous proof. We also thank the anonymous reviewers for their constructive feedback. HL is supported by NSF Award IIS-1943607 and a Google Research Scholar Award.

References

[AVHC22] Ahmet Alacaoglu, Luca Viano, Niao He, and Volkan Cevher. A natural actor-critic framework for zero-sum markov games. In International Conference on Machine Learning , pages 307-366. PMLR, 2022.

[AY16] Gürdal Arslan and Serdar Yüksel. Decentralized q-learning for stochastic teams and games. IEEE Transactions on Automatic Control , 62(4):1545-1558, 2016.

[BJ20] Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. In International conference on machine learning , pages 551-560. PMLR, 2020.

[BJY20] Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. Advances in neural information processing systems , 33:2159-2170, 2020.

[BLM18] Mario Bravo, David Leslie, and Panayotis Mertikopoulos. Bandit learning in concave n-person games. Advances in Neural Information Processing Systems , 31, 2018.

[BP18] James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Proceedings of the 2018 ACM Conference on Economics and Computation , 2018.

[BT02] Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213-231, 2002.

[BV01] Michael Bowling and Manuela Veloso. Rational and convergent learning in stochastic games. In Proceedings of the 17th international joint conference on Artificial intelligence-Volume 2 , pages 1021-1026, 2001.

[CCDX23] Shicong Cen, Yuejie Chi, Simon Shaolei Du, and Lin Xiao. Faster last-iterate convergence of policy optimization in zero-sum markov games. In International Conference on Learning Representations , 2023.

[CLW21] Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Conference on Learning Theory , pages 1216-1259. PMLR, 2021.

[CMZ21] Ziyi Chen, Shaocong Ma, and Yi Zhou. Sample efficient stochastic policy extragradient algorithm for zero-sum markov game. In International Conference on Learning Representations , 2021.

[COZ22] Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

[CS07] Vincent Conitzer and Tuomas Sandholm. Awesome: A general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents. Machine Learning , 67(1-2):23-43, 2007.

[CWC21] Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems , 34:27952-27964, 2021.

[CZ23] Yang Cai and Weiqiang Zheng. Doubly optimal no-regret learning in monotone games. In International Conference on Machine Learning , 2023. to appear.

[CZG22] Zixiang Chen, Dongruo Zhou, and Quanquan Gu. Almost optimal algorithms for twoplayer zero-sum markov games with linear function approximation. In International Conference on Algorithmic Learning Theory , 2022.

[DDK11] Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal noregret algorithms for zero-sum games. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms , pages 235-254. SIAM, 2011.

[DFG20] Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems , 33:5527-5540, 2020.

[DFR22] Dmitriy Drusvyatskiy, Maryam Fazel, and Lillian J Ratliff. Improved rates for derivative free gradient play in strongly monotone games. In IEEE 61st Conference on Decision and Control (CDC) , 2022.

[ELS + 23] Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games. In International Conference on Machine Learning , pages 9343-9373. PMLR, 2023.

[FT91] Jerzy A Filar and Boleslaw Tolwinski. On the algorithm of pollatschek and avi-ltzhak. 1991.

[GPD20] Noah Golowich, Sarath Pattathil, and Constantinos Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games. Advances in neural information processing systems , 2020.

[GTG22] Eduard Gorbunov, Adrien Taylor, and Gauthier Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In Advances in Neural Information Processing Systems , 2022.

[GVGM21] Angeliki Giannou, Emmanouil-Vasileios Vlatakis-Gkaragkounis, and Panayotis Mertikopoulos. On the rate of convergence of regularized learning in games: From bandits and uncertainty to optimism and beyond. Advances in Neural Information Processing Systems , 34:22655-22666, 2021.

[HH23] Yuanhanqing Huang and Jianghai Hu. Zeroth-order learning in continuous games via residual pseudogradient estimates. arXiv preprint arXiv:2301.02279 , 2023.

[HIMM19] Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. In Advances in Neural Information Processing Systems , 2019.

[HK66] Alan J Hoffman and Richard M Karp. On nonterminating stochastic games. Management Science , 1966.

[HLWY22] Baihe Huang, Jason D. Lee, Zhaoran Wang, and Zhuoran Yang. Towards general function approximation in zero-sum markov games. In International Conference on Learning Representations , 2022.

[HW03] Junling Hu and Michael P Wellman. Nash q-learning for general-sum stochastic games. Journal of machine learning research , 4(Nov):1039-1069, 2003.

[JJJN21] Mehdi Jafarnia-Jahromi, Rahul Jain, and Ashutosh Nayyar. Learning zero-sum stochastic games with posterior sampling. arXiv preprint arXiv:2109.03396 , 2021.

[JLWY21] Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 , 2021.

[JLY22] Chi Jin, Qinghua Liu, and Tiancheng Yu. The power of exploiter: Provable multi-agent rl in large state spaces. In International Conference on Machine Learning , pages 10251-10279. PMLR, 2022.

[LH14] Tor Lattimore and Marcus Hutter. Near-optimal pac bounds for discounted mdps. Theoretical Computer Science , 558:125-143, 2014.

[Lit94] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994 , pages 157-163. Elsevier, 1994.

[LS19] Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In The 22nd International Conference on Artificial Intelligence and Statistics , 2019.

[LYBJ21] Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning , pages 7001-7010. PMLR, 2021.

[LZBZ21] Tianyi Lin, Zhengyuan Zhou, Wenjia Ba, and Jiawei Zhang. Doubly optimal no-regret online learning in strongly monotone games with bandit feedback. Available at SSRN 3978421 , 2021.

[MOP20] Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of o (1 /k ) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization , 30(4):3230-3251, 2020.

[MPP18] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Annual ACMSIAM Symposium on Discrete Algorithms , pages 2703-2717. SIAM, 2018.

[MPS20] Vidya Muthukumar, Soham Phade, and Anant Sahai. On the impossibility of convergence of mixed strategies with no regret learning. arXiv preprint arXiv:2012.02125 , 2020.

[Neu15] Gergely Neu. Explore no more: Improved high-probability regret bounds for nonstochastic bandits. Advances in Neural Information Processing Systems , 28, 2015.

[PAI69] MAPollatschek and B Avi-Itzhak. Algorithms for stochastic games with geometrical interpretation. Management Science , 1969.

[PDVH + 22] Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science , 378(6623):990-996, 2022.

[Put14] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

[Sha53] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences , 1953.

[SL99] Csaba Szepesvári and Michael L Littman. A unified analysis of value-function-based reinforcement-learning algorithms. Neural computation , 1999.

[SLY23] Zhuoqing Song, Jason D. Lee, and Zhuoran Yang. Can we find nash equilibria at a linear rate in markov games? In International Conference on Learning Representations , 2023.

[SPO22] Muhammed O Sayin, Francesca Parise, and Asuman Ozdaglar. Fictitious play in zerosum stochastic games. SIAM Journal on Control and Optimization , 60(4):2095-2114, 2022.

[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

[SSS + 17] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature , 2017.

[SZL + 21] Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized q-learning in zero-sum markov games. Advances in Neural Information Processing Systems , 34:18320-18334, 2021.

[TK22] Tatiana Tatarenko and Maryam Kamgarpour. On the rate of convergence of payoffbased algorithms to nash equilibrium in strongly monotone games. arXiv preprint arXiv:2202.11147 , 2022.

[Tse95] Paul Tseng. On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics , 60(1-2):237-252, 1995.

[TWYS20] Yi Tian, Yuanhao Wang, Tiancheng Yu, and Suvrit Sra. Provably efficient online agnostic learning in markov games. arXiv preprint arXiv:2010.15020 , 2020.

[VDW78] J Van Der Wal. Discounted markov games: Generalized policy iteration method. Journal of Optimization Theory and Applications , 1978.

[vN28] J v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen , 100(1):295320, 1928.

[WDCW20] Yuanhao Wang, Kefan Dong, Xiaoyu Chen, and Liwei Wang. Q-learning with ucb exploration is sample efficient for infinite-horizon mdp. In International Conference on Learning Representations , 2020.

[WHL17] Chen-Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Online reinforcement learning in stochastic games. In Advances in Neural Information Processing Systems , pages 4987-4997, 2017.

[WLZL21a] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games. In Conference on learning theory , pages 4259-4299. PMLR, 2021.

[WLZL21b] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations (ICLR) , 2021.

[XCWY20] Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zerosum simultaneous-move markov games using function approximation and correlated equilibrium. In Conference on learning theory , pages 3674-3682. PMLR, 2020.

[XZS + 22] Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, and Tong Zhang. A self-play posterior sampling algorithm for zero-sum markov games. In ICLR 2022 Workshop on Gamification and Multiagent Solutions , 2022.

[YM23] Yuepeng Yang and Cong Ma. O ( T -1 ) convergence of optimistic-follow-theregularized-leader in two-player zero-sum markov games. In International Conference on Learning Representations , 2023.

[ZLW + 22] Runyu Zhang, Qinghua Liu, Huan Wang, Caiming Xiong, Na Li, and Yu Bai. Policy optimization for markov games: Unified framework and faster convergence. In Advances in Neural Information Processing Systems , 2022.

[ZTLD22] Yulai Zhao, Yuandong Tian, Jason Lee, and Simon Du. Provably efficient policy optimization for two-player zero-sum markov games. In International Conference on Artificial Intelligence and Statistics , pages 2736-2761. PMLR, 2022.

A Auxiliary Lemmas

A.1 Sequence Properties

Lemma 1. Let 0 < h < 1 , 0 ≤ k ≤ 2 , and let t ≥ ( 24 1 -h ln 12 1 -h ) 1 1 -h . Then

We first show that s ≤ t 2 . Suppose not, then we have

and thus t 1 -h < 4( k +1)ln t ≤ 12 ln t . However, by the condition for t and Lemma 3, it holds that t 1 -h ≥ 12 ln t , which leads to contradiction.

Proof. Define

Then the sum can be decomposed as

Lemma 2. Let 0 < h < 1 , 0 ≤ k ≤ 2 , and let t ≥ ( 24 1 -h ln 12 1 -h ) 1 1 -h . Then

Proof.

where in ( a ) we use Lemma 3. Combining the two inequalities finishes the proof.

Lemma 3. Let 0 < h < 1 and t ≥ ( 24 1 -h ln 12 1 -h ) 1 1 -h . Then t 1 -h ≥ 12 ln t .

Proof. By the condition, we have

Applying Lemma 12, we get

Lemma 4 (Lemma A.1 of [SSBD14]) . Let a > 0 . Then x ≥ 2 a ln( a ) ⇒ x ≥ a ln( x ) .

Lemma5 (Freedman's Inequality) . Let F 0 ⊂ F 1 ⊂ · · · ⊂ F n be a filtration, and X 1 , . . . , X n be real random variables such that X i is F i -measurable, E [ X i |F i -1 ] = 0 , | X i | ≤ b , and ∑ n i =1 E [ X 2 i |F i -1 ] ≤ V for some fixed b > 0 and V > 0 . Then with probability at least 1 -δ ,

A.2 Properties Related to EXP3-IX

In Lemma 6 and Lemma 7, we assume that F 0 ⊂ F 1 ⊂ F 2 ⊂ · · · is a filtration, and assume that x i , ℓ i are F i -1 -measurable, where x i ∈ ∆ A , ℓ i ∈ [0 , 1] A . Besides, a i ∈ [ A ] and σ i are F i -measurable with E [ a i = a |F i -1 ] = x i,a and E [ σ i |F i -1 ] = ℓ i . Define ̂ ℓ i,a = σ i,a 1 [ a i = a ] x i,a + β i where β i is non-increasing.

Lemma6 (Lemma 20 of [BJY20]) . Let c 1 , c 2 , . . . , c t be fixed positive numbers. Then with probability at least 1 -δ ,

Lemma 7 (Adapted from Lemma 18 of [BJY20]) . Let c 1 , c 2 , . . . , c t be fixed positive numbers. Then for any sequence x ⋆ 1 , . . . , x ⋆ t ∈ ∆ A such that x ⋆ i is F i -1 -measurable, with probability at least 1 -δ ,

Proof. Lemma 18 of [BJY20] states that for any sequence of coefficients w 1 , w 2 , · · · , w t such that w i ∈ [0 , 2 β i ] A is F i -1 -measurable, we have with probability 1 -δ ,

Since x ⋆ i ∈ ∆ A and β i is decreasing, we know 2 β t · x ⋆ i ∈ [0 , 2 β i ] . Thus we can apply Lemma 18 of [BJY20] and get with probability 1 -δ ,

Lemma8 (Lemma 21 of [BJY20]) . Let c 1 , c 2 , . . . , c t be fixed positive numbers. Then with probability at least 1 -δ , for all x ⋆ ∈ ∆ A ,

Lemma 9. Let ( x 1 , y 1 ) and ( x 2 , y 2 ) be equilibria of f 1 ( · , · ) in the domain Z 1 and f 2 ( · , · ) in the domain Z 2 respectively. Suppose that Z 1 ⊆ Z 2 , and that sup ( x,y ) ∈Z 1 | f 1 ( x, y ) -f 2 ( x, y ) | ≤ ϵ . Then for any ( x, y ) ∈ Z 2 ,

where d = max z ∈Z 2 min z ′ ∈Z 1 ∥ z -z ′ ∥ 1

Proof. Since ( x 1 , y 1 ) is an equilibrium of f 1 , we have for any ( x ′ , y ′ ) ∈ Z 1 ,

For any ( x, y ) ∈ Z 2 , we can find ( x ′ , y ′ ) ∈ Z 1 such that ∥ ( x, y ) -( x ′ , y ′ ) ∥ 1 ≤ d . Therefore, for any ( x, y ) ∈ Z 2 ,

which implies

A.3 Markov Games

Lemma 10 ([WLZL21a]) . For any policy pair x, y , the duality gap on a two player zero-sum game can be related to duality gap on individual states:

A.4 Online Mirror Descent

Lemma 11. Let

for some convex set Ω ⊆ ∆ A , ℓ ∈ [0 , ∞ ) A , and ϵ ∈ [0 , 1 η ] A . Then

for any u ∈ Ω , where ϵ ln x denotes the vector ( ϵ a ln x a ) a ∈A .

Proof. By the standard analysis of online mirror descent, we have for any u ∈ Ω

Below, we abuse the notation by defining KL (˜ x, x ) = ∑ a (˜ x a ln ˜ x a x a -˜ x a + x a ) without restricting ˜ x to be a probability vector. Then following the analysis in the proof of Lemma 1 of [CLW21], we have

Lemma 12. For x ∈ (0 , 1) and y > 0 , we have x 1 -y -x ≤ -yx 1 -y ln x .

Proof.

where the second equality is by the mean value theorem.

B Last-Iterate Convergence Rate of Algorithm 1

Proof of Theorem 1. The proof is divided into three parts. In Part I, we establish a descent inequality for KL ( z ⋆ t , z t ) . In Part II, we give an upper bound KL ( z ⋆ t , z t ) by recursively applying the descent inequality. Finally in Part III, we show last-iterate convergence rate on the duality gap of z t = ( x t , y t ) . In the proof, we assume without loss of generality that t ≥ t 0 = ( 24 1 -k η -k ϵ ln( 12 1 -k η -k ϵ )) 1 1 -kη -kϵ = (96 ln(48)) 4 since the theorem holds trivially for constant t .

Part I.

Rearranging the above inequality, we get

where v t ≜ KL ( x ⋆ t +1 , x t +1 ) -KL ( x ⋆ t , x t +1 ) . Similarly, since the algorithm for the y -player is symmetric, we have the following:

KL

(

y

⋆

t

+1

, y

t

+1

)

≤ (1 -η t ϵ t ) KL ( y ⋆ t , y t ) + η t ( f t ( x t , y t ) -f t ( x t , y ⋆ t )) + 10 η 2 t A ln 2 ( At ) + 2 η 2 t Aλ t + η t ξ t + η t ζ t + v t where

Adding the two inequalities above up and using the fact that f t ( x ⋆ t , y t ) -f t ( x t , y ⋆ t ) ≤ 0 , we get

Part II. Expanding the recursion in Eq. (6), and using the fact that 1 -η 1 ϵ 1 = 0 , we get

where w i t ≜ ∏ t j = i +1 (1 -η j ϵ j ) . We can bound each term as follows.

By Lemma 1 and the fact that that t ≥ t 0 , we have

Using Lemma 7 with x ⋆ = 1 A 1 , ℓ i = 1 for all i , and c i = w i t η 2 i , we have with probability 1 -δ t 2 ,

where in ( a ) we use Lemma 2 with the fact that t ≥ t 0 .

Using Lemma 6 with c i = w i t η i , we have with probability at least 1 -δ t 2 ,

Using Lemma 7 with c i = w i t η i , we get with probability at least 1 -δ t 2 ,

where ( a ) is by Lemma 2 and t ≥ t 0 . By Lemma 13 and Lemma 1,

Combining all terms above, we get that with probability at least 1 -3 δ t 2 ,

Using an union bound over t , we see that Eq. (7) holds for all t ≥ t 0 with probability at least 1 -O ( δ ) .

Part III. Using Lemma 9 with f t ( x, y ) and x ⊤ Gy with domains Ω t × Ω t and ∆ A × ∆ A , we get that for any ( x, y ) ∈ ∆ A × ∆ A ,

Further using Eq. (7), we get that with probability at least 1 -3 δ , for any t and any ( x, y ) ∈ ∆ A × ∆ A ,

where ( a )

is by Pinsker's inequality. This completes the proof of Theorem 1.

Proof.

Proof.

Similarly, KL ( x 2 , x ) -KL ( x 1 , x ) ≤ O (ln( At ) ∥ x 1 -x 2 ∥ 1 ) .

Lemma 14. Let x, x 1 , x 2 ∈ Ω t . Then

Proof. Notice that the feasible sets for the two time steps are different. Let ( x ′ t +1 , y ′ t +1 ) be such that x ′ t +1 = p t +1 A 1 +(1 -p t +1 ) x ⋆ t +1 and y ′ t +1 = p t +1 A 1 +(1 -p t +1 ) y ⋆ t +1 where p t +1 = min { 1 , 2 t -3 } . Since ( x ∗ t +1 , y ∗ t +1 ) ∈ Ω t +1 × Ω t +1 , we have that for any a , x ′ t +1 ,a ≥ p t +1 A +(1 -p t +1 ) 1 A ( t +1) 2 ≥ 1 At 2 . Hence, ( x ′ t +1 , y ′ t +1 ) ∈ Ω t × Ω t .

Because ( x ⋆ t +1 , y ⋆ t +1 ) is the equilibrium of f t +1 in Ω t +1 × Ω t +1 , we have that for any ( x, y ) ∈ Ω t +1 × Ω t +1 ,

where the first inequality is due to the following calculation:

where we use ∇ x f t +1 ( x ⋆ t +1 , y ⋆ t +1 ) ⊤ ( x -x ⋆ t +1 ) ≥ 0 since x ⋆ t +1 is the minimizer of f t +1 ( · , y ⋆ t +1 ) in Ω t +1 . Specially, we have

Similarly, because ( x ⋆ t , y ⋆ t ) is the equilibrium of f t in Ω t × Ω t , we have

which implies

In the first inequality, we use the fact that f t ( x, y ) is convex in x and concave in y and Hölder's inequality. In the second inequality, we use the triangle inequality, ∥∇ x f t ( x, y ) ∥ ∞ ≤ max a { ( Gy ) a + ln( x a ) } ≤ O (ln( At )) , and ∥∇ y f t ( x, y ) ∥ ∞ ≤ max b { ( G ⊤ x ) b +ln( y b ) } ≤ O (ln( At )) . In the second and third inequality, we use ∥ z ′ t +1 -z ⋆ t +1 ∥ 1 = O ( 1 t 3 ) by the definition of z ′ t +1 .

Combining Eq. (8) and Eq. (9), we get

Solving the inequality, we get

C Improved Last-Iterate Convergence under Expectation

In this section, we analyze Algorithm 4, which is almost identical to Algorithm 1 but does not involve the parameter β t . The choices of stepsize η t and amount of regularization ϵ t are also tuned differently to obtain the best convergence rate.

Algorithm 4 Matrix Game with Bandit Feedback

1: Define: η t = t -k η , ϵ t = t -k ϵ where k η = 1 2 , k ϵ = 1 6 . Ω t = { x ∈ ∆ A : x a ≥ 1 At 2 , ∀ a ∈ A } . 2: Initialization: : x 1 = 1 A 1 . 3: for t = 1 , 2 , . . . do 4: Sample a t ∼ x t , and receive σ t ∈ [0 , 1] with E [ σ t ] = G a t ,b t . 5: Compute g t where g t,a = 1 [ a t = a ] σ t x t,a + ϵ t ln x t,a , ∀ a ∈ A .

6: Update x t +1 ← argmin x ∈ Ω t +1 { x ⊤ g t + 1 η t KL ( x, x t ) } .

7: end for

Proof. With the same analysis as in Part I of the proof of Theorem 1, we have

where

Unlike in Theorem 1, here these three terms all have zero mean. Thus, following the same arguments that obtain Eq. (6) and taking expectations, we get

where v t = KL ( z ⋆ t +1 , z t +1 ) -KL ( z ⋆ t , z t +1 ) and E t [ · ] is the expectation conditioned on history up to round t . Then following the same arguments as in Part II of the proof of Theorem 1, we get

Finally, following the arguments in Part III, we get

D Last-Iterate Convergence Rate of Algorithm 2

D.1 On the Assumption of Irreducible Markov Game

Proposition 1. If Assumption 1 holds, then for any L ′ = 2 L log 2 ( S/δ ) consecutive steps, under any (non-stationary) policies of the two players, with probability at least 1 -δ , every state is visited at least once.

̸

Proof. We first show that for any pair of states s ′ , s ′′ , under any non-stationary policy pair, the expected time to reach s ′′ from s ′ is upper bounded by L . For a particular pair of states ( s ′ , s ′′ ) , consider the following modified MDP: let the reward be r ( s, a ) = 1 [ s = s ′′ ] , and the transition be the same as the original MDP on all s = s ′′ , while P ( s ′′ | s ′′ , a ) = 1 (i.e., making s ′′ an absorbing state). Also, let s ′ be the initial state. By construction, the expected total reward of this MDP is the travelling time from s ′ to s ′′ . By Theorem 7.1.9 of [Put14], there exists a stationary optimal policy in this MDP. The optimal expected total value is then upper bounded by L by Assumption 1. Therefore, for any (possibly sub-optimal) non-stationary policies, the travelling time from s ′ to s ′′ must also be upper bounded by L .

̸

Divide L ′ steps into log 2 ( S/δ ) intervals each of length 2 L , and consider a particualr s . Conditioned on s not visited in all intervals 1 , 2 , . . . , i -1 , the probability of still not visiting s in interval i is smaller than 1 2 (because for any s ′ , Pr[ T s ′ → s > 2 L ] ≤ E [ T s ′ → s ] 2 L ≤ L 2 L = 1 2 , where T s ′ → s denotes the travelling time from s ′ to s ). Therefore, the probability of not visiting s in all log 2 ( S/δ ) intervals is upper bounded by 2 -log 2 ( S/δ ) = δ S . Using a union bound, we conclude that with probability at least 1 -δ , every state is visited at least once within L ′ steps.

Corollary 1. If Assumption 1 holds, then with probability 1 -δ , for any t ≥ 1 , players visit every state at least once in every 6 L ln( St/δ ) consecutive iterations before time t .

Proof. First, we fix time t ≥ 1 and define t ′ = 3 L ln( St 3 /δ ) . Let us consider the following time intervals: [1 , t ′ ] , [ t ′ , 2 t ′ ] , . . . , [ t -t ′ , t ] . Using Proposition 1, we known for each interval, with probability at least 1 -δ t 3 , players visit every state s . Using a union bound over all intervals, we have with probability at least 1 -δ t 2 , in every interval, players visit every state s . Since every 2 t ′ consecutive iterations must contain an interval of length L ′ , we have with probability at least 1 -δ t 2 , players visit every state s in every 2 t ′ consecutive iterations until time t . Applying union bound over all t ≥ 1 completes the proof.

According to Corollary 1, in the remaining of this section , we assume that for any t ≥ 1 , players visit every state at least once in every 6 L ln( St/δ ) iterations until time t .

D.2 Part I. Basic Iteration Properties

Lemma 16. For any x s ∈ Ω τ +1 ,

(see the proof for the definitions of λ s τ , ξ s τ , ζ s τ ( · ) )

Proof. Consider a fixed s and a fixed τ , and let t = t τ ( s ) be the time when the players visit s at the τ -th time.

where we omit some calculation steps due to the similarity to Eq. (5).

D.3 Part II. Policy Convergence to the Nash of Regularized Game

Lemma 17. With probability at least 1 -O ( δ ) , for all s ∈ S , t ≥ 1 and τ ≥ 1 such that t τ ( s ) ≤ t , we have

Proof. In this proof, we abbreviate ζ s i (ˆ x s i⋆ ) as ζ s i . By Lemma 16, for all i ≤ τ we have

Similarly, for all i ≤ τ , we have

Adding the two inequalities up, and using f s i (ˆ x s i⋆ , ˆ y s i ) -f s i (ˆ x s i , ˆ y s i⋆ ) ≤ 0 because (ˆ x s i⋆ , ˆ y s i⋆ ) is the equilibrium of f s i , we get for i ≤ τ

Expanding Eq. (11), we get

These five terms correspond to those in Eq. (6), and can be handled in the same way. For term 1 to term 4 , we follow exactly the same arguments there, and bound their sum as with probability at least 1 -O ( δ Sτ 2 ) ,

To bound term 5 , by Lemma 14 and Lemma 18, we have

Combining all the terms with union bound over s ∈ S and τ ≥ 1 finishes the proof.

Proof. The bound holds trivially when τ ≤ 2 L . Below we focus on the case with τ > 2 L . By exactly the same arguments as in the proof of Lemma 15, we have an inequality similar to Eq. (10):

Since t τ ( s ) ≤ t and we assume that every state is visited at least once in 6 L log( St/δ ) steps (Corollary 1), we have that for any state s ′ , n s ′ t τ ( s ) ≥ t τ ( s ) 6 L log( St/δ ) -1 . Thus, whenever V s ′ t updates between t τ ( s ) and t τ +1 ( s ) , the change is upper bounded by 1 1 -γ ( t τ ( s ) 6 L log( St/δ ) -1) -k α . Besides, between t τ ( s ) and t τ +1 ( s ) , V s ′ t can change at most 6 L log( St/δ ) times. Therefore,

where the last inequality holds since k α < 1 . Combining Eq. (12) and Eq. (13) with the fact that ϵ τ = 1 1 -γ τ -k ϵ finishes the proof.

D.4 Part III. Value Convergence

For positive integers τ ≥ i , we define α i τ = α i ∏ τ j = i +1 (1 -α j ) .

Lemma 19 (weighted regret bound) . With probability 1 - O ( δ ) , for any s , any visitation count τ ≥ τ 0 , and any x s ∈ Ω τ +1 ,

where k ′ = min { k η , k β , k α -k β } .

Proof. We will be considering a weighted sum of the instantaneous regret bound established in Lemma 16. However, notice that for f s i , Lemma 16 only provides a regret bound with comparators in Ω i +1 . Therefore, for a fixed x s ∈ Ω τ +1 , we define the following auxiliary comparators for all i = 1 , . . . , τ :

where p i ≜ ( τ +1) 2 -( i +1) 2 ( i +1) 2 [( τ +1) 2 -1] . Since x s ∈ Ω τ +1 , we have that for any a , ˜ x s i,a ≥ p i A + 1 -p i A ( τ +1) 2 = 1 A ( i +1) 2 , and thus ˜ x s i ∈ Ω i +1 .

Applying Lemma 16 and considering the weighted sum of the bounds, we get

where ( a ) is by Lemma 14 and the following calculation:

We proceed to bound other terms as follows: with probability at least 1 -O ( δ Sτ 2 )

Combining all terms, we get

Finally,

Adding up Eq. (14) and Eq. (15) and applying union bound over all s ∈ S and τ finish the proof. Lemma 20. With probability at least 1 -O ( δ ) , for any state s ∈ S and time t ≥ 1 , we have

where k ∗ = min { k η , k β , k α -k β , k ϵ } .

Proof. Fix an s and a visitation count τ . Let t i be the time index when the players visit s for the i -th time. Then with probability at least 1 -δ Sτ 2 ,

Similar inequality can be also obtained through the perspective of the other player: with probability at least 1 -δ Sτ 2

which, combined with the previous inequality and union bound over s ∈ S and τ ≥ 1 , gives the following relation: with probability at least 1 -O ( δ ) , for any s ∈ S and τ ≥ 1 ,

Before continuing, we first some auxiliary quantities. For a fixed t , define

for fixed ( τ, t ) we further define

Now we continue to prove a bound for | V s t -V s ⋆ | . Suppose that Eq. (16) can be written as

for a universal constant C 1 ≥ 1 . Below we use induction to show that for all t ,

This is trivial for t = 1 .

Suppose that Eq. (18) holds for all time 1 , . . . , t -1 and for all s . Now we consider time t and a fixed state s . We denote L ′ = 6 L ln( St/δ ) . Let τ = n s t +1 and let 1 ≤ t 1 < t 2 < · · · < t τ ≤ t

be the time indices when the players visit state s . If t ≤ L ′ ( u ( t ) + 1) , then Eq. (18) is trivial. If t ≥ L ′ ( u ( t ) + 1) , we have τ ≥ t L ′ -1 ≥ u ( t ) . Therefore,

| V s t -V s ⋆ | = | V s t τ -V s ⋆ | ( t τ is the last time up to time t when V s t is updated) ≤ γ τ ∑ i =1 α i τ max s ′ ∣ ∣ ∣ V s ′ t i -V s ′ ⋆ ∣ ∣ ∣ + C 1 A ln 3 ( SAτ/δ ) τ -k ∗ 1 -γ (by Eq. (17)) ≤ γ v ( τ,t ) ∑ i =1 α i τ max s ′ ∣ ∣ ∣ V s ′ t i -V s ′ ⋆ ∣ ∣ ∣ + γ τ ∑ i = v ( τ,t )+1 α i τ max s ′ ∣ ∣ ∣ V s ′ t i -V s ′ ⋆ ∣ ∣ ∣ + C 1 A ln 3 ( SAτ/δ ) τ -k ∗ 1 -γ ( a ) ≤ γτ × (1 -γ ) 3 t 3 × 1 1 -γ + γ τ ∑ i = v ( τ,t )+1 α i τ Φ( t i ) + C 1 A ln 3 ( SAτ/δ ) τ -k ∗ 1 -γ ≤ γ τ ∑ i = v ( τ,t )+1 α i τ Φ( t i ) + 2 C 1 A ln 3 ( SAτ/δ ) τ -k ∗ 1 -γ (induction hypothesis) ≤ γ τ ∑ i = v ( τ,t )+1 α i τ Φ( t ) × ( t t i ) k ∗ + 2 C 1 A ln 3 ( SAτ/δ ) τ -k ∗ 1 -γ (by the definition of Φ and that u ( · ) is an increasing function) ( b ) ≤ γ ( 1 + 1 -γ 2 ) τ ∑ i = v ( τ,t )+1 α i τ Φ( t ) + 2 C 1 A ln 3 ( SAτ/δ ) τ -k ∗ 1 -γ ≤ γ ( 1 + 1 -γ 2 ) Φ( t ) + 2 C 1 A ln 3 ( SAt/δ ) ( t 2 L ′ ) -k ∗ 1 -γ ( t ≥ τ ≥ t L ′ -1 ≥ t 2 L ′ since t ≥ L ′ ( u ( t ) + 1) ≥ 2 L ′ ) ≤ γ ( 1 + 1 -γ 2 ) Φ( t ) + 1 -γ 2 Φ( t ) = Φ( t ) .

In ( a ) we use the following property: if τ ≥ u ( t ) and i ≤ v ( τ, t ) , then

In ( b ) we use the following calculation:

where the first inequality is due to the fact that at time t , state s has only been visited for τ times; the second inequality is because for any k > j , we have t j ≥ j and t k -t j ≤ L ′ ( k -j ) ; the third inequality is by i ≥ v ( τ, t ) ; the fourth inequality is by the definition of v ( τ, t ) ; the fifth inequality is because 4 τ k α -1 ln t 1 -γ ≤ 1 -γ 4 L ′ since τ ≥ u ( t ) , and 1 τ < 1 u ( t ) ≤ 1 -γ 16 L ′ since u ( t ) ≥ 16 L ′ 1 -γ ; the last inequality is because 1+ 1 16 a 1 -1 4 a ≤ 1 + 1 2 a for a ∈ [0 , 1] .

D.5 Part IV. Combining

In this subsection, we combine previous lemmas to show last-iterate convergence rate of Algorithm 2 and prove Theorem 2.

Lemma 21. With probability at least 1 -O ( δ ) , for any time t ≥ 1 ,

Proof. Using Lemma 10, we can bound the duality gap of the whole game by the duality gap on an individual state:

With probability at least 1 -O ( δ ) , for any s, x s , y s , and t ≥ 1 , denote τ the number of visitation to state s until time t , then

Combing the above two inequality with Lemma 17 and Lemma 20 and the choice of parameters k α = 9 9+ ε , k ε = 1 9+ ε , k β = 3 9+ ε , and k η = 5 9+ ε , we have k ♯ = min { k β -k ϵ , k η -k β , k α -k η -2 k ϵ } = 2 9+ ε , k ∗ = min { k η , k β , k α -k β , k ϵ } = 1 9+ ε , and

E Convergent Analysis of Algorithm 3

E.1 Part I. Basic Iteration Properties

Definition 1. Let t τ ( s ) be the τ -th time the players visit state s . Define ˆ x s τ = x s t τ ( s ) , ˆ y s τ = y s t τ ( s ) , ˆ a s τ = a t τ ( s ) , ˆ b s τ = b t τ ( s ) ,. Furthermore, define

Lemma 22. For any x s ∈ Ω ,

where

Proof. The proof is exactly the same as that of Lemma 16.

E.2 Part II. Value Convergence

Lemma 23 (weighted regret bound) . There exists a large enough universal constant κ (used in the definition of bns τ ) such that with probability 1 -O ( δ ) , for any state s , visitation count τ , and any x s ∈ Ω ,

Proof. Fix state s and visitation count τ ≤ T . Applying Lemma 22 and considering the weighted sum of the bounds, we get

We proceed to bound other terms as follows: wiht probability at least 1 -δ Sτ 2

Combining all terms and applying a union bound over s ∈ S and τ , we get with probability 1 -O ( δ ) such that for any s ∈ S , visitation count τ , and x s ∈ Ω ,

This implies the conclusion of the lemma.

Lemma 24. For all t, s , V s t ≥ V s t .

Proof. We prove it by induction on t . The inequality clearly holds for t = 1 by the initialization. Suppose that the inequality holds for 1 , 2 , . . . , t -1 and for all s . Now consider time t and state s . Let τ = n s t , and let 1 ≤ t 1 < t 2 < . . . < t τ < t be the time indices when the players visit state s . By the update rule,

where the inequality is by the induction hypothesis. Therefore,

In the last inequality we also use the fact that ∼ V s t ≤ H and ∼ V s t ≥ 0 . Note that by the induction hypothesis and the update rule of V s t and V s t , we have 0 ≤ V s i < V s i ≤ H for all s and 1 ≤ i ≤ t -1 . Thus ∼ V s t = ∑ τ i =1 α i τ ( γV s t i +1 t i -bns i ) ≤ H and similarly ∼ V s t ≥ 0 .

Lemma 25. Let c = ( c 1 , . . . , c T ) be any non-negative sequence with c i ≤ c max ∀ i and ∑ T t =1 c t = C . Then

Proof.

Note that c ′ t is another sequence with

since ∑ τ i =1 α i τ = 1 for any τ ≥ 1 . Thus, we can unroll the inequality Eq. (19) for H times, which gives

where in the inequality we use that (1 + 1 H ) H ≤ e and γ H = (1 -(1 -γ )) H ≤ e -(1 -γ ) H = 1 T .

Corollary 2. There exists a universal constant C 1 > 0 such that for any ˜ ϵ ≥ C 1 A ln 3 ( AST/δ ) β (1 -γ ) 3 , with probability at least 1 -O ( δ ) ,

Proof. We apply Lemma 25 with the following definition of c t :

and which gives

for some universal constant C 2 and C = ∑ T t =1 c t . By Azuma's inequality, for some universal constant C 3 > 0 , with probability 1 -δ ,

Combining Eq. (20) and Eq. (21), we get

By the definition of c t , the left-hand side above is lower bounded by ˜ ϵ ∑ T t =1 c t = C ˜ ϵ . Define C 1 = 2( C 2 + C 3 ) . Then by the condition on ϵ ′ , the right-hand side above is above inequality is bounded by

by the condition on ˜ ϵ . Combining the upper bound and the lower bound, we get

Lemma 26. With probability at least 1 -O ( δ ) , for any t ≥ 1 ,

Proof. Fix a t and s , let τ = n t ( s ) , and let t i be the time index in which s is visited the i -th time. With probability at least 1 -δ ST , we have

Therefore, using a union bound over s and t , we have with probability 1 -δ , for all s and t ,

for some universal constant C 4 . Next, we use induction to show the first inequality. Suppose that

for all s and t ′ < t . Then by Eq. (22),

which proves the first desired inequality. The other inequality can be proven in the same way.

E.3 Part III. Policy Convergence to the Nash of the Regularized Game

Lemma 27. Let 0 ≤ p ≤ 1 be arbitrarily chosen, and define

Furthermore, let ˆ z s τ⋆ = (ˆ x s τ⋆ , ˆ y s τ⋆ ) be the equilibrium of f s τ ( x, y ) , and define z s t⋆ = ˆ z s τ⋆ where τ = n t ( s ) . Then with probability at least 1 -O ( δ ) , the following holds for any 0 < ϵ ′ ≤ 1 :

if η and β satisfy the following

with sufficiently small universal constant C 5 , C 6 > 0 .

Proof. In this proof, we write ζ s i (ˆ x s i⋆ ) as ζ i . By Lemma 22, we have

Similarly,

Adding the two inequalities up, we get

⋆

where v s i = KL (ˆ z s i +1 ⋆ , ˆ z s i +1 ) -KL (ˆ z s i⋆ , ˆ z s i +1 ) and □ s = □ s + □ s . By Lemma 24, we have f s i ( x, y ) ≤ f s i ( x, y ) for all x, y , and thus f s i (ˆ x s i⋆ , ˆ y s i ) -f s i (ˆ x s i , ˆ y s i⋆ ) ≤ f s i (ˆ x s i⋆ , ˆ y s i ) -f s i (ˆ x s i , ˆ y s i⋆ ) ≤ 0 . Therefore, Eq. (25) further implies

KL

(ˆ

z

s

i

+1

,

ˆ

z

s

i

+1

)

where ∆ s i = f s i (ˆ x s i , ˆ y s i ) -f s i (ˆ x s i , ˆ y s i ) and in the last step we use a ≤ [ a -b ] + + b .

Unrolling the recursion, we get with probability at least 1 -O ( δ ) , for all s and τ (we show that the inequality holds for any fix s and τ with probability 1 -O ( δ ST ) and then apply the union bound over s and τ ),

where in ( a ) we use the following calculation:

and in ( b ) we use the conditions Eq. (23) and Eq. (24).

We continue to bound the sum of term 5 and term 6 over t . Note that

where in the last inequality we use the following calculation:

and that

From Eq. (26), we have

where in the second-to-last inequality we use Eq. (27) and Eq. (28). This finishes the proof.

E.4 Part IV. Combining

Theorem 5. For any u ∈ [ 0 , 1 1 -γ ] , there exists a proper choice of parameters ϵ, β, η such that

with probability at least 1 -O ( δ ) .

Proof. We will choose ϵ such that u ≥ C 7 ϵ ln( AT ) 1 -γ with a sufficiently large universal constant C 7 . By Lemma 26, we have

Therefore, we can upper bound the left-hand side of the desired inequality by

For the first term in Eq. (29), we can bound it by

(here we choose (ˆ x s , ˆ y s ) to be the equilibrium under f ( x, y ) )

The third term in Eq. (29) can be bounded in the same way. The second term in Eq. (29) can be bounded using Corollary 2 by

Overall, we have

Notice that the parameters ϵ, β, η needs to satisfy the conditions specified in this lemma and Lemma 27, with which we apply ϵ ′ = Θ ( u 2 (1 -γ ) 2 ln 2 ( SAT/δ ) ) . The constraints suggest the following parameter choice (under a fixed u ):

Using these parameters in Eq. (30), we get

F Discussions on Convergence Notions for General Markov Games

In general Markov games, learning the equilibrium policy pair on every state is impossible because some state might have exponentially small visitation probability under all policies. Therefore, a reasonable definition of convergence is the convergence of the following quantity to zero:

which is similar to the best-iterate convergence defined in Section 3, but over the state sequence visited by the players instead of taking max over s . It is also a strict generalization of the sample complexity bound for single-player MDPs under the discounted criteria (see e.g., [LH14, WDCW20]).

The path convergence defined in our work is, on the other hand, that the following quantity converges to zero:

Since max y ( x s ⊤ Q s ⋆ y s ) ≤ max y ( x s ⊤ Q s x,y y s ) = max y V s x,y for any x , the convergence of Eq. (31) is stronger than Eq. (32).

Implications of Path Convergence Although Eq. (32) does not imply the more standard best-iterate guarantee Eq. (31), it still has meaningful implications. By definition, It implies that frequent visits to a state bring players' policies closer to equilibrium, leading to both players using near-equilibrium policies for all but o ( T ) number of steps over time.

Path convergence also implies that both players have no regret compared to the game value V s ⋆ , which has been considered and motivated in previous works such as [BT02, TWYS20]. To see this more clearly, we apply the results to the episodic setting, where in every step, with probability 1 -γ , the state is redrawn from s ∼ ρ for some initial distribution ρ (every time the state is redrawn from ρ , we call it a new episode). We can show that if Eq. (32) vanishes, then every player's long-term average payoff is at least the game value. First, notice that if Eq. (32) converges to zero, then

Now fix an i and let t i be time index at the beginning of episode i . Let E t = 1 indicate the event that episode i has not ended at time t . Then

Combining this with Eq. (33), we get

Hence the one-step average reward is at least (1 -γ ) E s ∼ ρ [ V s ⋆ ] . A symmetric analysis shows that it is also at most (1 -γ ) E s ∼ ρ [ V s ⋆ ] . This shows that both players have no regret compared to the game value. Notice that this is only a loose implication of the path convergence guarantee because of the loose second inequality in Eq. (33).

Remark on the notion of 'last-iterate convergence' in general Markov games While Eq. (31) corresponds to best-iterate convergence for general Markov games, an even stronger notion one can

pursue after is 'last-iterate convergence.' As argued above, it is impossible to require that the policies on all states to converge to equilibrium. To address this issue, we propose to study this problem under the episodic setting described above, in which the state is reset after every trajectory whose expected length is 1 1 -γ . In this case, last-iterate convergence will be defined as the convergence of the following quantity to zero when i →∞ :

where we recall that i is the episode index and ( x t i , y t i ) are the policies used by the two players at the beginning of episode i . While last-iterate convergence seems reasonable and possibly achievable, we are unaware of such results even for the degenerated case of single-player MDPs - the standard regret bound corresponds to best-iterate convergence, while the techniques we are aware of to prove last-iterate convergence in MDPs require additional assumptions on the dynamics.

Peer reviews:
comments: SUMMARY:
The paper studies the problem of designing uncoupled learning dynamics that provably converge to Nash equilibria in two-player zero-sum Markov games. As a preliminary result, the paper introduces the first dynamics that converge last iterate in self play in matrix games with bandit feedback. Then, the paper shows how to extend such last-iterate-convergent learning dynamics to the case of irreducible Markov games. Finally, the paper studies the case of general Markov games, proving that a variation of the previously-introduced learning dynamics achieve convergence according to a newly introduced definition of convergence (called path convergence).

STRENGTHS:
The problem studied in the paper is interesting and it has received considerable attention over the last years. The paper is well written and easy to follow.

WEAKNESSES:
A major weakness that I see is that the results presented in the paper seem minor adaptations and different analysis of already known techniques/tools. The authors should focus more on discussing the novelty of their approach.

QUESTIONS:
Please resolve my concerns in the weaknesses part.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new algorithm for learning in two-player zero-sum Markov games based on prior work that  is uncoupled (agents only need their own reward as feedback), convergent (to NE) and rational. The result is an algorithm that is similar in concept to a single agent RL algorithm, but guarantees convergence if both players use the same algorithm. For their initial algorithm, there is an additional assumption of irreducibility on the Markov games, and it achieves last iterate convergence to the Nash with rate O(t^{-1/9+eps}). Meanwhile, for general 2 player zero sum Markov games without additional assumptions, a modification of the algorithm that applies optimism achieves O(t^{-1/10}) path convergence rate.\n\nSTRENGTHS:\nThe algorithm proposed takes inspiration from prior works and makes useful modifications that give practical and theoretical advantages. The structure of the results and explanation of analysis also flows well and is easy to read despite the technical complexity. The authors are also very clear in their motivation and I find the literature review to be quite comprehensive.\n\nWEAKNESSES:\nThe paper does a good job of setting up motivation and prior ideas, but in my opinion the key takeaway from this work is that there is now a truly uncoupled and convergent algorithm for general Markov games, which is in my opinion the paper's most significant contribution. However, there is too much space dedicated to matrix and irreducible Markov games, when in my view, the section on general Markov games should be greatly expanded. In addition to the overview of the analysis, I am interested to know how optimism can be leveraged in this type of algorithm, and the thought process behind the modifications to Algorithm 3 compared to Algorithm 2. As a suggestion, perhaps part of the analysis overview for Thm 3 can be repurposed and moved forward to add to the explanation about Alg 3, which would improve the flow of the section in my opinion. Finally, there is a lack of experimental results which would make help frame the results better in comparison to prior work.\n\nQUESTIONS:\nHave any experiments been run using Algorithms 1, 2 and 3 to compare them to current SOTA methods? If so, how does the empirical performance compare? It seems that your algorithms would scale well with larger games since players only use local information, how does this compare to existing methods?\n\nLIMITATIONS:\nThe authors have adequately addressed the paper's limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the last iteration convergence of uncoupled learning in two player zero-sum markov game, with bandit feedback. The paper provide the first finite last-iterate convergence guarantee under the bandit feedback.\n\nThe paper studies the problem of two-player zero-sum Markov game and designs a new uncoupled learning algorithm that enjoys last iteration convergence, under bandit feedback. Along the way, the paper derive a couple of results, summarized below:\n\n(1) Even for the standard matrix game (where there is no Markov transition), the paper provides an algorithm with $O(T^{-1/8})$ convergence rate\n\n(2) For irreducible Markov game, the paper provides an algorithm with $O(T^{-1/9})$ convergence rate\n\n(3) For general irreducible Markov game, the paper provides an algorithm with $O(T^{-1/10})$ convergence rate, but only under the notion of path convergence.\n\nIt is worth noting that there is a large body of literature, but previous work [WLZL21] are either not uncoupled, or [BJY'20 ] not last iterate convergent.\n\nThe technical part seems novel, from a high level, the results are obtained by adding an entropy regularizer, but there are many subtle details to make it really work and the analysis is involved.\n\n------------------\nI have read the author response and want to keep my positive evaluation.\n\nSTRENGTHS:\nThe paper studies a fairly popular topic and provide the first finite last-iterate convergence guarantee for Markov game (two-player, zero-sum). The technical contribution seems to be novel. The paper is also well-written and the technical part is well explained.\n\nWEAKNESSES:\nNo.\n\nQUESTIONS:\nI have a few minor questions:\n\n(1) Can you provide some intuition on the number of $T^{-1/8}$ or $T^{-1/9}$, ideally, it would be nice if one can written down a short explanation on how these magic exponent comes from.\n\n(2) The following paper seems very relevant?\n\n[1] Regret Minimization and Convergence to Equilibria in General-sum Markov Games\n\nLIMITATIONS:\nno.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper studies the problem of learning a Nash equilibrium in two-player zero-sum Markov games with bandit feedback.
The proposed algorithm introduces the entropy regularization technique into online mirror descent.
First, the author proves that the proposed algorithm converges to an equilibrium in two-player normal-form games.
Then, the last-iterate convergence rate for irreducible Markov games is provided.
Finally, the paper presents a path convergence rate for Markov games without the irreducible assumption.

STRENGTHS:
* The problem is well-motivated. In many scenarios, the last-iterate convergence property under bandit feedback is more suitable than the average-iterate convergence property.
* It seems novel to derive the last-iterate convergence rates under bandit feedback.
* The proof sketch of Theorem 1 is intuitive and easy to follow.

WEAKNESSES:
* Since action probabilities of strategies in $\Omega_t$ are lower bounded by $\frac{1}{At^2}$, line 6 in Algorithm 1 seems to have no closed-form solution. I wonder how much computational cost it will take to update the strategy.
* The term of $\ln^{20}(SAT/\delta)$ in Theorem 3 depends heavily on $T$. This term would be dominant practically and would not be able to be ignored.
* I could not completely follow the sketch of Theorem 3. I would have appreciated more detail.

QUESTIONS:
* How much computational cost will it take to update the strategy (e.g., line 6 in Algorithm 1)?
* Does the path convergence imply the average-iterate convergence?
* What prior knowledge of the game is required for setting of $\epsilon,\beta$, and $\eta$ in Theorem 3?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

comments: SUMMARY:
This paper studies algorithms for two-player zero-sum Markov games that are uncoupled, convergent, and rational. Previous attempts at designing such algorithms failed short in one aspect or the other. This work uses recent advances in entropy-based regularization to design new algorithms that overcome the inherent challenges. The other main contribution of this work is deriving the convergent rates of the proposed algorithms in Matrix, Markov, and general Markov games.

STRENGTHS:
1. The proposed algorithms do not require the two players to exchange information (including that related to entropy)
2. The proposed ideas extend to general Markov games without any assumptions on the dynamics.
3. The work seems technically sound. The algorithms follow easy-to-implement modifications of existing methods.
4. The problem setup, the algorithms, and the proofs are written in an easy-to-digest format.

WEAKNESSES:
See Questions.

QUESTIONS:
1. How tight are the obtained convergence rates?
2. My understanding of the proposed algorithm is that at time step $t,$ player 1 plays action $a _t$ and player 2 action $b_t.$ Thereafter, the local update steps are carried out. If this is correct, then why do the authors claim that their proposed algorithm does not require the two players to have synchronized policy updates?

LIMITATIONS:
The authors do address the limitations in their work adequately.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-30