Uncoupled and Convergent Learning in Two-Player Zero-Sum Markov Games with Bandit Feedback
Authors: Yang Cai
Haipeng Luo
Chen-Yu Wei
Weiqiang Zheng
Venue: NeurIPS 2023 poster (2023)
Keywords: two-player zero-sum Markov game
last-iterate convergence
path convergence
learning in games
TL;DR: This paper presents uncoupled algorithms for learning in two-player zero-sum Markov games with non-asymptotic convergence rates under bandit feedback.

Abstract: We revisit the problem of learning in two-player zero-sum Markov games, focusing on developing an algorithm that is *uncoupled*, *convergent*, and *rational*, with non-asymptotic convergence rates to Nash equilibrium. We start from the case of stateless matrix game with bandit feedback as a warm-up, showing an $\tilde{\mathcal{O}}(t^{-\frac{1}{8}})$ last-iterate convergence rate. To the best of our knowledge, this is the first result that obtains finite last-iterate convergence rate given access to only bandit feedback. We extend our result to the case of irreducible Markov games, providing a last-iterate convergence rate of $\tilde{\mathcal{O}}(t^{-\frac{1}{9+\varepsilon}})$ for any $\varepsilon>0$. Finally, we study Markov games without any assumptions on the dynamics, and show a *path convergence* rate, a new notion of convergence we defined, of $\tilde{\mathcal{O}}(t^{-\frac{1}{10}})$. Our algorithm removes the synchronization and prior knowledge requirement of Wei et al. (2021), which pursued the same goals as us for irreducible Markov games. Our algorithm is related to Chen et al. (2021) and Cen et al. (2021)'s and also builds on the entropy regularization technique. However, we remove their requirement of communications on the entropy values, making our algorithm entirely uncoupled.

Uncoupled and Convergent Learning in Two-Player Zero-Sum Markov Games with Bandit Feedback
Yang Cai Yale University yang. cai @yale edu
Haipeng Luo University of Southern California haipengl@usc.om edu
Chen-Ye Yu Wei University of Virginia chenyu. wwwpeergiiiiiiiii.....mmm
Weigiang Zheng
Yale University weiqiang weigiang.zheng@yale wwwenghalnnnnecccoo edu
Abstract
We revisit the problem learning in two-player zero-sum Markov games, focusing on developing an algorithm that is uncoupled, convergent, and rational, with non- asymptotic convergence rates to Nash equilibrium. We start from the case stateless matrix game with bandit feedback as a warm-up, showing an O(t 3) last-iterate convergence rate. To the best of our knowledge, this is the first result that obtains finite last-iterate convergence rate given access to only bandit feedback. We extend our result the of irreducible Markov games, providing a last- iterate convergence rate of O -ी)) for any 0. Finally, we study Markov games without any assumptions the dynamics, and show path convergence rate, notion of convergence we define, of O(t to Our algorithm removes the coordination and prior knowledge requirement of [WLZL21a)] which pursued the goals to [CMZ21, CWC21] and also builds on the entropy regularization technique. However, we remove their requirement of communications the entropy values, making our algorithm entirely uncoupled.
for irreducible Markov games. Our algorithm is related
Introduction
In multi-agent learning, central question is how to design algorithms that agents can independently learn (i.e., with little coordination overhead) how to interact with each other. Additionally, is desirable to maximally reuse existing single-agent learning algorithms, so that the multi-agent system can be built in modular way. Motivated by this question, decentralized multi-agent learning emerges with the goal to design decentralized systems, in which no central controller governs the policies of the agents, and each agent learns based on only their local information - just like in a single-agent algorithm. In recent years, we have witnessed significant success of this new decentralized learning paradigm. For example, self-play, where each agent independently deploys the same single-agent algorithm to play against each other without further direct supervision, plays crucial role in the training of AlphaGo ISSS* 17] and AI for Stratego [PDVH 22].
Despite the recent success, many important questions remain open in decentralized multi-agent learning. Indeed, unless the decentralized algorithm is carefully designed, pelf-play often falls short of attaining certain sought-after global characteristics, such as convergence to the global optimum or stability as seen in, for example, [MPP18, BP18].
In this work, we revisit the problem of learning in two-player zero-sum Markov games, which has received extensive attention recently. Our goal is to design a decentralized algorithm that resembles
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

standard single-agent reinforcement learning (RL) algorithms,, but with an additional crucial assurance, that is, guaranteed convergence when both players deploy the algorithm. The simultaneous pursuit of independence and convergence has been advocated widely [B [BV01, AY16, WI.7122a, SZI.- while the results are still not entirely satisfactory. In particular, all of these results rely on assumptions on the dynamics of the Markov game. Our paper takes the first step to remove such assumptions.
More specifically, our goal is to design algorithms that simultaneously satisfy the following three properties (the definitions are adapted from [BV01, DDKI
Each player i's action is generated by a standalone procedure P, which, in every round, only receives the current state and player i's own reward as feedback (in particular, it has no knowledge about the actions or policies used by the opponent). There is communication or shared randomness between the players.
Uncoupled:
Convergent: The policy pair of the two players converges to a Nash equilibrium.
Rational: If P. competes with an opponent who uses policy sequence that converges stationary one, then P; converges to the best response of this stationary policy.
The uncoupledness and rationality property capture the independence of the algorithm, while the convergence property provides desirable global guarantee. Interestingly, as argued in [WLZL2Ia]] if an algorithm uncoupled and convergent, then also rational, we only need ensure that the algorithm is uncoupled and convergent. Regarding the notion of convergence, the standard definition above only allows last-iterate convergence. Considering the difficulty of achieving such convergence, in the related work review (Section 2) and in the design of our algorithm for general Markov games (Section 6), we also consider weaker notions of convergence, including the best-iterate convergence, which only requires that the Cesaro mean of the duality gap is convergent, and the path convergence, which only requires the convergence of the Cesaro mean of the duality gap assuming minimax/maximin policies are followed in future steps. The precise definitions of these convergence notions are given at the end of Section 3.
1.1 Our Contributions
The main results in this work are as follows (see also Table 1 for comparisons with prior works):
As a warm-up, for the special case of matrix games with bandit feedback, develop an uncoupled algorithm with last-iterate convergence rate of (tt=)) under self-play (Section 4). To the best of our knowledge, this is the first algorithm with provable last-iterate convergence rate in the setting.
Generalizing the ideas from matrix games, we further develop an uncoupled algorithm for irre- ducible Markov games with last-iterate convergence rate of Ott गोर) ) for any e 0 under self-play (Section 5).
Finally, for general Markov games without additional assumptions, we develop an uncoupled algorithm with a path convergence rate Ot 1b) under self-play (Section 6).
Our algorithms leverage recent advances on using entropy to regularize the policy updates (CWC21, CM721] and the Nash-V-styles value updates [BJY20]. On the one hand, compared to [CWC21, CM our algorithm has the following advantages: 1) it does not require the two players to exchange their entropy information, which allows our algorithm to be fully uncoupled: 2) it does not require the players to have coordinated policy updates, 3) it naturally extends to general Markov games without any assumptions on the dynamics (e.g., irreducibility). On the other hand, our algorithm inherits appealing properties of Nash-V V [BJY20), but additionally guarantees path convergence during execution.
2 Related Work
The study of two-player zero-sum Markov games originated from [Sha53], with many other works further developing algorithms and establishing convergence properties [HK66, PAI69, VDW78, FT91]. However, these works primarily focused solving the game with full knowledge of its parameters (i.e., payoff function and transition kernel). The problem of learning zero-sum games was first formalized by [Lit94]. Designing a provably uncoupled, rational, and convergent algorithm

---

Table 1: (Sample-based) Learning algorithms for finding NE in two-player zero-sum games. Our results are shaded. A halfcheck "X"" in the convergent column means that the policy convergence proven only for one player (typically this is result of asymmetric updates). (L) and (B) stand for last-iterate convergence and best-iterate convergence, respectively. (P) stands for path convergence, weaker convergence notion we introduce (see Section 3, 6.1).
*; While [WIZL21a] also proposes an uncoupled and convergent algorithm for irreducible Markov games, their algorithm requires coordinated updates and some prior knowledge of the game, while ours does not. See Section 2.1 for more detailed discussion.
Setting
Algorithm
Uncoupled?
Converegent?
EXP3 vs. EXP3
Matrix Game
Algorithm
v (L)
[DFG20]
x(B)
ZZLLD22, AVHC22]
x(L)
Markov game + assumptions dynamics
[11
[CMZ21]
X
v VL)
WWLAL21a]
v (L.
Algorithm
/ (L)
[WHL17, JJJN21, HLWY22]
X
<(B)
[LY22, X2S+22]
BJ20, XCWY20]
X
V(B
LYB/21, CZG22]
Markov Game
BJ2 JLWY2I]
Algorithm
V (P)
challenging, with many attempts [SL99, BV01, HW03, CS07, AY16, SPO22] falling short in one aspect or another, often lacking either uncoupledness or convergence. Moreover, these works only establish asymptotic convergence without providing a concrete convergence rate.
2.1 Non-asymptotic convergence guarantees
Recently, a large body of works on learning two-player zero-sum Markov games use regret minimiza- tion techniques establish non-asymptotic guarantees. They focus fast computation under full information of payoff and transitions [CWC21, CCDX23, ZLW+ SLY23, YM23), though many of their algorithms are decentralized and can be viewed as the first step towards the learning setting.
With rationality and uncoupledness satisfied, [DFG20] established one-sided policy convergence for players using independent policy gradient with asymmetric learning rates. Such asymmetric update rule also adopted by (ZTLD22, AVHC22] to establish one-sided policy convergence guarantees. When using symmetric update rule, [SZL* 21] developed a decentralized-O learning algorithm. However, the convergence is only shown for the V-function maintained by the players instead of the policies being used, so the policies may still cycle and are provably convergent in our definition. [ELS+ 23] studied regret minimization in general-sum Markov games and provided an algorithm with sublinear regret under self-play and average-iterate convergence rates equibria, while work focuses on last-iterate convergence rates to Nash equilibria.
To our knowledge, [WLZL21a] first provided an uncoupled, rational, and convergent algorithm with non-asymptotic convergence guarantee, albeit only for irreducible Markov game. They achieved this via optimistic gradient descent/uscent. Despite satisfying all our criteria, their algorithm still has unnatural coordination between the players and a requirement on some prior knowledge of the game such the maximum revisiting time of the Markov game. Our algorithm removes all these extra requirements. follow-up work by [CMZ21] improved the rate of WLZZ22al using entropy regularization; however, this requires their players to inform the opponent about the entropy of their own policy, making the algorithm coupled again. We show that such an exchange of information is unnecessary under entropy regularization.
2.2 Further handling exploration
The algorithms introduced above all require full information or some assumption on the dynamics of the Markov game. To handle exploration, some works design coupled lcarning algorithms

---

which guarantee that the player's long-term payoff is at least the minimax value [BT02, WHL17, XCWY20, HLWY22, JLY22, JJIN21, XZS 22]. Interestingly, as shown in [WHL17. HLWY22. JLY22, 22J, if the player is paired with an optimistic best-response opponent (instead of using the same algorithm), the first player's 's strategy can converge to the minimax policy. [XCWY20, BJ20, LYBJ21, CZG22] developed another coupled learning framework to handle exploration, but with symmetric updates on both players. In each round, the players need to jointly solve a general- sum equilibrium problem due to the different exploration bonus added by each player. Hence, the execution of these algorithms is more similar to the Nash-Q algorithm by [HW03].
So far, exploration has been handled through coupled approaches that are also not rational. To our knowledge, the first uncoupled and rational algorithm that handles exploration is the Nash-V V algorithm by [BJY20] Nash-V V can output [JLWY21]; however, it is not provably convergent during execution. A major remaining open problem is whether one can design a natural algorithm that is provably rational, uncoupled, and convergent with exploration capability. Our work provides the first progress towards this goal.
marimmm policy through weighted averaging
2.3 Other works on last-iterate convergence
Uncoupled Learning dynamics in normal-form games with provable last-iterate convergence rate receives extensive attention recently. Most of the works assume that the players re- ceive gradient feedback, and convergence results under bandit feedback remain sparse. Lin- ear convergence is shown for strongly monotone games or bilinear games under gradient feed- back [Tse95, LS19, MOP20, WLZL21b] and sublinear rates are proven for strongly monotone games with bandit feedback [BLMI8, HIMM19, LZBZ21, TK22, DFR22, HH23]. Convergence rate to strict Nash equilibrium is analyzed by [GVGM21J. For monotone games that includes two-player zero-sum games as a special case, the last-iterate convergence rate of no-regret learning under gradient feedback has been shown recently [GPD20, COZ22, GTG22, CZ23]. With bandit feedback, [MPS20] showed an impossibility result that certain algorithms with optimal (VT)) regret do not converge in last-iterate. To the best of our knowledge, there is no natural uncoupled learning dynamics with provable last-iterate convergence rate in two-player zero-sum games with bandit feedback.
3 Preliminaries
Basic Notations Throughout the paper, we assume for simplicity that the action set for the two players are the same, denoted by A with cardinality A [A]. We usually call player the a--layer and player? 2 the y-player. The set of mixed strategies over an action set A is denoted
as AA : {x Enes Na = 1;0 S Fa S Va E A). To simplify notation, we denote by 2 = (z,y) the concatenated strategy of the players. We use 6 as the entropy function such that <lx) = - Eae A Za In, and KL as the Kullback-Liibler (KL) divergence such that KL
E.SA To In Za The all-one vector is denoted by 1 = (1, ,,,.,,,.........
two-player zero-sum matrix game with a loss matrix G E [0, 1|AXA, when the r-player chooses action a and the y-player chooses action b, the x-player suffers loss Go.b and the y-player suffers loss -Ga.b- b. A pair of mixed strategy (a*, "e*"")) is a Nash equilibrium for G if for any strategy profile (r. y) E A x A, holds that (x*) Gy S (x*)TGy* S x. Gy- Similarly, (e*,)") is a Nash equilibrium for two-player zero-sum game with a general convex-concave loss function fx, y) A x A -> if for all (zx y) E A x A f(x", y) S f(=",0 vy) S f(x, The celebrated minimax theorem [vN28] guarantees the existence of Nash equilibria in two-player zero-sum games. For a pair of strategy (x. v). we use duality gap defined as GAP(G, x. v) A max, x T Gy' - ming' x' Gy to measure its proximity to Nash equilibria.
Matrix Games
Markoy Games A generalization of matrix games, which models dynamically changing environ- ment, is Markov games. We consider -ornitee-orizoon discounted two-player zero-sum Markov games, denoted by tuple (S,A, (G*)ess, (P*) (P*),ES.?) ?) where (1) S is finite state space; (2) A a finite action space for both players; (3) Player 1 suffers loss G E [0. 1] (respectively player 2 suffers
'We make this assumption only to simplify notations; our proofs can be easily extended to the case where the action sets of the two players different.
4

---

loss -Ga,) when player I chooses action and player 2 chooses action b at state s; (4) P is the transition function such that Pa (s') is the probability of transiting to state s' when player plays and player plays at state s; (5) [ 1) a discount factor.
stationary policy for player a mapping S -> A that specifies player I's strategy x² E A each state E S. We denote
(a:*) xes. Similar notations apply player 2. We denote 2 = (x", ( as the concatenated strategy for the players and 2 = (x, y)- The value function V denotes the expected loss of player (or the expected payoff of player 2) given pair of stationary policy (I, y) and initial state s:
00
V 2.y
IE
G Cai,,, = 8,0 xt b 1 8841 pd: (),
The minimax game value on state s defined V ming maxy V2.y max, ming Va.y We call pair of policy (r., Nash equilibrium if it attains minimax game value of state (such policy pair necessarily attains the minimax game value over all states). The duality gap of (z, y) max, (maxy ".y/ ming' V v. The Q-function on state s under policy pair (I, y) is defined via Qi b) G + Eg [V,,) x,ll which can be rewritten
matrix Og such that V2,, x² "* We denote Q Qr.... the Q-function under Nash equilibrium (I+, V-). is known that Q: is unique for any even when multiple equilibria exist.
Uncoupled Learning with Bandit Feedback We assume the following uncoupled interaction
protocol: at each round = 1,...,,, T, the players both observe the current state 8t, and then, with the policy 3y and y in mind, they independently choose actions a₂ x and by ~ yi" respectivel, Both of them then observe o, E [0, 1) with E[aa] = G ar and proceed to the next state 81+1 ~ per (-)- Importantly, they do not observe each other's action.
Notions of Convergence For Markov games with the irreducible assumption (Assumption 1), given players' history of play (st, Fe, (T), the best-iterate convergence rate is measured by the average duality gap + Σi= max.,a.y (V= sL. V2.g.): ). while the stronger last-iterate convergence rate is measured by maxs,r.y (V*, V ). i.e., the duality gap of (æT, yr). For general Markov games, we propose the path convergence rate, which is measured by the average duality gap the
visited states with respect the optimal Q-function: + ET= maX (z Q"1y" xt, Q*yy"") y²²) We remark that the path convergence guarantee is weaker than the counterpart of the other two notions of convergence in general Markov games, but still provides meaningful implications (see detailed discussion in Section 6.1 and Appendix F).
Matrix Games
In this section, we consider two-player zero-sum matrix games. We propose Algorithm for decentralized learning of Nash equilibria. We only present the algorithm for the x-player as the algorithm for the y-player is symmetrie.
Algorithm 1 Matrix Game with Bandit Feedback
1: Define: 7)- t- RM 3r t Ki t t where ki 8. 8. h.
5 hi
AR2 Va E A}. 2: Initialization:: I 11
2 {x E A
3: for ? 2, do
4: Sample 0r Tr, and receive o 10, 1] with [ = Gorde
5: Compute 9t where It.a = 1/a, =al: + E In Va E A. 23,
6: Update r++1 f- argmin { 9e + [1)
7:
end for
The algorithm similar to the EXP3-IX algorithm by [Neu15] that achieves high-probability regret bound for adversarial multi-armed bandits, but with several modifications. First (and most
5

---

importantly), in addition to the standard loss estimators used in [Neu15], we add another negative term & In 2t,a the loss estimator of action a (see Line 5). This equivalent to the entropy regularization approach in, e.g., [CWC21, CM721J, since the gradient of the negative entropy -((x)) is (In Zt,a + A and the constant takes no effect Line 6. Like [CWC21, CMZ21], the entropy regularization drives last-iterate convergence; however, while their results require full- information feedback, our result holds in the bandit feedback setting. The second difference that instead of choosing the players' strategies in the full probability simplex A our algorithm chooses from Ω2 a subset of A where every coordinate lower bounded by TP The third is the choices of A
the learning rate Th, clipping factor Be, and the amount of regularization 6,. The main result of this section is the following last-iterate convergence rate of Algorithm 1.
Theorem 1 (Last-Iterate Convergence Rate). Algorithm 1 guarantees with probability least 1 (6), any & 2 1,
max (Gy-xxGy)) = = (VAIN A,EAA
Algorithm 1 also guarantees (t--)) b) regret when the other player is adversarial. If we only target at an expected bound instead of a high-probability bound, the last-iterate convergence rate can be improved to O (/ +). The details are provided in Appendix C.
4.1 Analysis Overview
We define regularized zero-sum game with loss function f( y) Gy Gy- + cv((v) over domain Q2 x 24 and denote by 2 = (z) yr its unique Nash equilibrium since ft is strongly convex-strongly concave. The regularized game is slight perturbation of the original matrix game G over a smaller domain 24 x Ω2 and we prove that 2 is an O(er) -approxmio Nash equilibrium of the original matrix game G (Lemma 9). Therefore, it suffices bound KI,(2,, zx) since the duality gap of 2, is at most ( ((VKL(Z) (2t 2)) + G))
Step 1: Single-Step Analysis We start with a single-step analysis of Algorithm 1, which shows:
KL(Z(+1:+41) s (1- K= z)) + 2 Al? (At) + 2n AA, + MrSt + nist +
instability penalty
ostimation crror
where we define v KL(=i+1, 1++1) , (see Appendix B for definitions of Å & G+) The instability penalty comes from some local-norm of the gradient estimator gt- The estimation error comes from the bias between the gradient estimator 9i and the real gradient Gyt- We pay the last term 2 since the Nash equilibrium 2f of the regularized game fi is changing over time.
Step 2: Strategy Convergence to NE of the Regularized Game up to fo, we get
Expanding the above recursion
KL F0+1) S O
Σ win?
+ 22
win A
wins& +
wimss +
Σ wpus ). (1)
1=1
term
term2
termng
term.
term;
where 3i 2 Iii (1 n) To upper bound term -term,, we apply careful sequence analysis (Appendix A.1) and properties of the EXP3-IX algorithm with changing step size (Appendix A.2). The analysis of term; uses Lemma 13, which states v KL( #++1) KL( 2441 S (In(At)] =
=f11,) = O (111 (4t)) and is slightly involved as 24 and e are both changing. With
t
these steps, we conclude that with probability at least 1 - (6), KL 21) O (AIn (At/8)(t)
Irreducible Markov Games
We now extend our results on matrix games to two-player zero-sum Markov games. Similarly to many previous works, our first result makes the assumption that the Markov game is irreducible with bounded travel time between any pair of states. The assumption is formally stated below:

---

Assumption 1 (Irreducible Game), We assume that under any pair of stationary policies of the two players, and any pair of states s, s', the expected time reach s' from is upper bounded by L.
We propose Algorithm 2 for uncoupled learning in irreducible two-player zero-sum games, which is closely related the Nash-V algorithm by [BJY20], but with additional entropy regularization. It can also be seen as players using Algorithm 1 on each state to update the policies (zi vt) whenever state s visited, but with o, + +V, as the observed loss to construct loss estimators. Importandly Vi, V22 is a slowly changing sequence of value estimations that ensures stable policy updates 21J. [BJY20, WLZL2Ia, la. -ZL_ Note that in Algorithm 2, the updates of V2² only use players' local information (Line 8).
Algorithm 2 Irreducible Markov Game
1: Define: 1/2 (1 (1- kn, 3r t-Ra, 6: I-7 k. ar tho ko with kos ki, ka, kin E (0, 1),
24 = f E Aa Fa 2 1 Va E A}.
AR2
2: Initialization: xf + 1 1, nf + 0. Vp + 2[1-)]' As.
3: for t-rt- === 1,2, ................................ do
4: T = nit1 f ni + (the number of visits to state 8t up to time t).
5: Draw a, ~ zz', observe or E [0. with E [o,] = G and observe 8141 Pai (-)
6: Compute Ie where Gt.a = [o + In In Va E A. tt.+h.
7: Update rff1 t argminzEa {
x 9t + (I( x²")}
8: Update V44 t (1-a))V; Vv" + ar too+>Ve +2Vett))
9: For all 8 + St, x2+1 t ai, nit1 f ni, Vit t Vi-
10: end for
Comparison to Previous Works Although Algorithm 2 shares similarity with previous works that also use entropy regularization, we believe that both the design and the analysis of our algorithm are novel and non-trivial. To the best of our knowledge, all previous entropy regularized two-player zero-sum Markov game algorithms are coupled (e.g.. ICWC21, CMZ21, CCDX23]), while ours is the first that achieves uncoupledness under entropy regularization. We further discuss this by comparing our algorithm to those in [CCDX23], highlighting the new technical challenges we encounter.
The entropy-regularized OMWU algorithm in [CCDX23] is tailored to the full-information Moreover, the value function update step both players need to know the entropy value of the other player's 'ss policy, which is unnatural. Indeed, the authors explicitly present the removal of this information sharing as an open question. We answer this open question affirmatively by giving a fully decentralized algorithm for zero-sum Markov games with provable last-iterate convergence rates. In Algorithm 2 (Line 8), the update of the value function is simple and does not require any entropy information: Vea < (1 a, V. + a (o, + <V=++1) This modification results in discrepancy between the policy update and the value update. While the policy now incorporates a regularization term, the value function does not. Such mismatch is unprecedented in carlier studies and necessitates a non-trivial approach to resolve. Additionally, Algorithm 2 operates on bandit feedback instead of full-information feedback, presenting further technical challenges.
setting.
Algorithm also offers improvement over the uncoupled algorithm of [WLZLL1aa]. The algorithm of [WLZL21a] requires coordinated policy update where the players interact with each other using the current policy for several iterations to get an approximately accurate gradient (the number of iterations required depends on L as defined in Assumption 1), and then simultaneously update the policy pair on all states. We not require such unnatural coordination between the players or prior knowledge on L.
Our main result the following theorem on the last-iterate convergence rate of Algorithm 2.
Theorem 2 (Last-Iterate Convergence Rate). For any E > 0, Algorithm 2 with k. = 9
545'
ke 9tc kig g46. t > 1.
5 3 and kon 9+c guarantees, with probability at least 1 - (6)), (,),, for any time
8,2,y 21.9 Vin.) s O AL 2 411/ (t (e(t/(1 7)) rote)
max (V2
- 2++1//
7

---

5.1 Analysis Overview
We introduce some notations for simplicity. We denote by BA' [V'1] the A x A matrix such that (E.--= (V. Da. - [. Let (s) be the 7-th time the players visit state and define # = 27, (s) and D? vi, (4) Then, define the regularized game for each state via the loss function (z, x (G* + >B IV (o (x)(x) + +))(v). Furthermore, let T++ (#*.. D++) be the equilibrium of (x, y) over O 2,. the following analysis, we fix some 2
Step 1: Policy Convergence NE of Regularized Game Using similar techniques to Step
and Step 2 in the analysis of Algorithm 1, we can upper bound KI. +++1) like Eq. (1) with similar subsequent analysis for term, -term4. The analysis for term, where vf
i+:1+: 2441)
KL( 1)+1) is more challenging compared to the matrix game case since here (is)) changing
between two visits state s. To handle this term, we leverage the following facts for any s': (1) the irreducibility assumption ensures that (++ - (s) < OL (St/(o)) thus the number of updates of the value function state s' is bounded; (2) until time (s) 2 6, state s' has been visited at times thus each change of the value function between (s)) and 4, (s) is most least D 1n
O(( in(tt/8) -R.). With these arguments, we bound term; by O (In" ATt + 2%.). Overall, we have the following policy convergence of NE of the regularized game (Lemma 17): KL (2 2 2.) s O (A k)), where kg min(kg ke: kin kg. Ki kn 2k.
Step 2: Value Convergence Unlike matrix games, policy convergence NE of the regular- ized game is not enough for convergence in duality gap. We also need to bound [V/- - V/1 since the regularized game is defined using Vi*, the value function maintained by the algorithm, in- stead of the minimax game value V We use the following weighted regret quantities as a proxy: Reg, 4max, (E at (f) 0t) - ff((z, ((=",)))) (()), Σ a² ( v2) (f),, where af a; [[j=++1 1 (1 - az). We can upper bound the weighted regret Reg, using similar analysis as in Step 1 (Lemmaa 19). We then show contraction for (V) V.l with the weighted regret quanti- ties: (v) V S 7 Ei= a² max,' Ve((s) V+ Reg:). This leads to the following convergence of Ve (Lemma 20):/V V < Ó( ), where k. = min (ke,, ka, ki kg. ki
-
Obtaining Last-Iterate Convergence Rate Fix any / and let r be the number of visits to before time L. So far we have shown (1) policy convergence of (L &5) in the regularized game; (2) and value convergence of VV, V.l. Using the fact that the regularized game is at most +1,,+ -V away from the minimax game martrix Q" and appropriate choices of parameters proves Theorem 2.
6 General Markov Games
In this section, we consider general two-player zero-sum Markov games without Assumption 1. We propose Algorithm 3, an uncoupled learning algorithm that handles exploration and has path convergence rate. Compared to Algorithm 2, the update of value function in Algorithm uses bonus term bns, based on the optimism principle to handle exploration.
Theorem below implies that we can achieve 1 E maxx,y (1 Q: x² (?)))))
vtt) =
O(t it) path convergence rate if we use the doubling trick to tune down u at rate of t to
Theorem 3. For any u E [ and T 2 1, there exists proper choice of parameters e, B. such that Algorithm 3 guarantees with probability at least 1 O
T
20 (SAT/5)
max 2,91
""Q0""" -
Q2 -
u 50
(2)
n0(1 7
8

---

Algorithm 3 General Markov Game
1: Input: 7 B S and T.
2: Define: ! { E A va 2 AT: Va A), #+ where = ln(T)
bns, In (SAT//8 (B a-) ()) for sufficiently large absolute constant >
3: Initialization: vj.nf t 0, rं + 11,48.
4: for t = -,1, ...., ........ do
5: т = 1+1 < Ti +
8
6: Sample 0r ~ It observe o, E [0, with [o = aa,b, and observe 51+1 ~ Pay (-).
7: Compute Ie where 9t.a = [[a [V + telnx Va E A.
2/.0
8: Update 2241 < argminzed {ir + (L, }
9: Update Vi41 - (1 - (kr) We + at (o, + +V bbs,) and V + max (Vi41 0}.
For all 8 + St. ri+1 f- zúj Vit1 t Vi, Vi + -Va'. mit1 t- ni.
11: end for
6.1 Path Convergence
Path convergence has multiple meaningful game-theoretic implications. By definition, It implies that frequent visits to a state bring players' policies closer to equilibrium, leading to both players using near-equilibrium policies for all but o(T) number of steps over time.
Path convergence also implies that both players have no regret compared to the game value V which has been considered and motivated in previous works such as [BT02, TWYS20J. To this, we apply the results to the episodic setting. where in every step, with probability 1 y-y, the state is redrawn from ~ p for some initial distribution p. If the learning dynamics enjoys path convergence.
E> [V:] + o(T). Hence the one-step average reward is (1 (1-7) Escup [V*) and both players have no regret compared to the game value. A more important implication of path convergence is that it guarantees stability of players' policies, while cycling behaviour is inevitable for any FTRL-type algorithms even in zero-sum matrix games [MPP18, BP18J. We defer the proof and more discussion of path convergence Appendix F.
then E. EI 2t G
Finally, we remark that our algorithm built upon Nash V-learning [BJY20], so it inherits proper- ties of Nash V-learning, e.g., can still output equiiiiuulirrimm policies through policy averag- ing [JLWY21] Y21]. or having no regret compared to the game value when competing with an arbitrary opponent [TWYS20]. We demonstrate extra benefits brought by entropy regularization regarding the stability of the dynamics.
6.2 Analysis Overview Theorem
For general Markov games, it no longer holds that every state is visited often, and thus the analysis is much more challenging. We first define two regularized games based on V and the corresponding quantity V. for the y-player. Define (=()). 45 of the same way as in the previous section. Then define (xx, A x TG +G (i) (x)x) + (e)())) (Mr, v) A x T(
T Vi - cœ(r) + cΦ(y) and denote J. = max (x; (G + 7E.wPa VV. 1y
It. (G tBs-Essu [V Dive") We first bound the "path duality gap" as follows
ot Q. I.. Qi:ii) < Jr + 0 (max (v.' V'.'-- v:'))
max 2.9
Value Convergence: Bounding Vi V. and V V. This step is similar to Step 2 in the analysis of Algorithm 2. We first show an upper bound of the weighted regret (Lemma 23): EL of (f. (66, 0f) (=" o()) I bbns,. where of a, IT
ay). Note that the value function V is updated using + bns.. Thus when relating VV: V to the regret, In(ATT)
the regret term and the bonus term cancel out and we get V V O T-y (Lemma 26). The analysis for V - -V. is symmetric. By proper choice of both terms bounded by f.. Combining

---

To bound the first two terms, we show con- vergence of the policy ( fff)) to Nash equilibria of both games S and fq. To this end, fix any be the equilibrium of the above with Eq. (3), we can upper bound the left-hand side of the desired inequality Eq. (2) by 11=1 1 Je Fu). which is further upper bounded in Eq. (29) by
++) 1 max (1 (+ (x
1
Σ
Σ
Σ
1 [
min fx .
8
y
[V- v2 yi
4
Policy Convergence to NE of Regularized Games
P E [0, 1], we define f pf + (1 p)f, and let 24, =
ff The analysis is similar to previous algorithms where we first conduct single-step analysis (Lemma 22) and then carefully bound the weighted recursive terms. We show in Lemma 27 that for any 0 c<e 1: E, Σx=1(() 1 ( 24) e'I S O 5 An This proves policy convergence: the number of iterations where the policy is far away from Nash equilibria of the regularized games is bounded, which can then be translated to upper bounds on the first two terms.
me²e(1->))
Value Convergence: Bounding V - Viil It remains to bound the last term in Eq. (4). Define
= 1|=²" (EEg- V - V Dvt" 2 é] where é = 7. Then we only need to bound C' 2 Σl= Ct. We use the weighted sum Pr & ΣT qz2,' (E V -V DyE" a proxy. On the one hand, Pr 2 Ce. On the other hand, in Lemma 25, by recursively tracking the update of the value function and carefully choosing 77 and 3, we upper bound Pr by S
Ci
+ O AS (1) Combining the (1-7)²
2
upper and lower bound of Pr gives C S O AS (AST/O (6) (Corollary 2). Plugging appropriate
yud( (m(1--)3
choices of 17, and / in the above bounds proves Theorem 3 (see Appendix E).
7 Conclusion and Future Directions
In this work, we study decentralized learning in two-player zero-sum Markov games with bandit feedback. We propose the first uncoupled and convergent algorithms with non-asymptotic last-iterate convergence rates for matrix games and irreducible Markov games, respectively. We also introduce a novel notion of path convergence and provide algorithm with path convergence in Markov games without any assumption on the dynamics. Previous results either focus on average-iterate convergence or require stronger cooobacc/cooriiaation or lack non-asymptotic convergence rates. Our results contribute to the theoretical understanding of the practical success of regularization and last-iterate convergence in multi-agent reinforcement learning.
Setting the optimal last-iterate convergence rate that is achievable by uncoupled learning dynamics is an important open question. The following directions are promising towards closing the gap between current upper bounds OTT-1/5) and (TT-94()+)) and lower bound 2( 3), The impossibility result by [MPS20] demonstrates that certain algorithms with O ((VT) regret diverge in last-iterate. Their result indicates that the current Q , lower bound on convergence rate may not be tight. On
the other hand, our algorithms provides insights and useful templates to potential improvements on the upper bound. For instance, instead of using EXP3-IX update, adapting optimistic policy update or other accelerated first-order methods to the bandit feedback setting is interesting future direction.
We thank Chanwoo Park and Kaiqing Zhang for pointing out a mistake in our previous proof. We also thank the anonymous reviewers for their constructive feedback. HL is supported by NSF Award IIS 1943607 and a Google Research Scholar Award.
Acknowledgement
10

---

References
[AVHC22] Ahmet Alacaoglu, Luca Viano, Niao He, and Volkan Cevher. A natural actor-critic framework for zero-sum markov games. In International Conference Machine Learning, pages 307-366. PMLR, 2022.
[AY16] Gürdal Arslan and Serdar Yüksel. Decentralized q-learning for stochastic teams and games. IEEE Transactions on Automatic Control, 6254)):555-15558 2016.
[BJ20] Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. International conference on machine learning, pages 551-560. PMLR, 2020.
[BJY20] Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play playpplay Advances neural information processing systems, 33:2159-2113 2020.
[BLM18] Mario Bravo, David Leslie, and Panayotis Mertikopoulos. Bandit learning in concave n- n-person games. Advances in Neural Information Processing Systems, 31, 2018.
[BPI8] James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Proceedings of the 2018 ACM Conference on Economics and Computation, 2018.
[BTO2] Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(0ct):213-23) 2002.
[BV01] Michael Bowling and Manuela Veloso. Rational and convergent learning in stochas- tic games. Proceedings of the 17th international joint conference on Artificial intelligence-Volume 2, pages 1021-1026, 2001.
[CCDX23) Shicong Cen, Yuejie Chi, Simon Shaolei Du, and Lin Xiao. Faster last-iterate conver- gence of policy optimization in zero-sum markov games. In International Conference on Learning Representations, 2023.
[CLW21] Liyu Chen, Haipeng Luo, and Chen- Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Conference on Learning Theory, pages 1216-1259. PMLR, 2021.
[CMZ21] Ziyi Chen, Shaocong Ma, and Yi Zhou. Sample efficient stochastic policy extragradi- ent algorithm for zero-sum markov game. In International Conference on Learning Representations, 2021.
[C0722) Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate con- vergence for learning in multi-player games. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[CS07] Vincent Conitzer and Tuomas Sandholm. Awesome: A general multiagent learning algorithm that converges in self-play and learns best response against stationary opponents. Machine Learning,
2007.
[CWC21] Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems, 34:27952-27988 27964, 2021.
[CZ23] Yang Cai and Weiqiang Zheng. Doubly optimal no-regret learning in monotone games. In International Conference Machine Learning, 2023. to appear.
[C7G22) Zixiang Chen, Dongruo Zhou, and Quanquan Gu. Almost optimal algorithms for two- player zero-sum markov games with linear function approximation. In International Conference on Algorithmic Learning Theory, 2022.
[DDK11] Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no- regret algorithms for zero-sum games. Proceedings of the twenty-eccond annual ACM-SIAM symposium Discrete Algorithms, pages 235-254. SIAM, 2011.
11

---

[DFG20] Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gra- dient methods for competitive reinforeement learning. Advances in neural information processing systems, 33:5527-5540, 2020.
[DFR22] Dmitriy Drusvyatskiy, Maryam Fazel, and Lillian J Ratliff. Improved rates for derivative free gradient play in strongly monotone games. In IEEE 61st Conference on Decision and Control (CDC), 2022.
[ELS Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Re- gret minimization and convergence to equilibria in general-sum markov games. In International Conference on Machine Learning, pages 9343-9373. PMLR, 2023.
[FT91] Jerzy A Filar and Boleslaw Tolwinski. On the algorithm of pollatschek and avi-Itzhak. 1991.
[GPD20] Noah Golowich, Sarath Pattathil, and Constantinos Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games. Advances in neural information processing systems, 2020.
[GTG22) Eduard Gorbunov, Adrien Taylor, and Gauthier Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In Advances in Neural Information Processing Systems, 2022.
[GVGM21] Angeliki Giannou, Emmanouil- Vasiiliis Vlatakis Satukis-Gaaragkouniii and Panayotis Mer- tikopoulos. On the rate of convergence of regularized learning in games; From bandits and uncertainty to optimism and beyond. Advances in Neural Information Processing Systems, 34:22655-226666 2021.
[HH23) Yuanhanqing Huang and Jianghai Hu. Zeroth-order learning in continuous games via residual pscudogradient estimates. arXiv preprint arXiv:2301.02279 2023.
[HIMМ19] Yu-Guan Hsieh, Franck lutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. In Advances in Neural Information Processing Systems, 2019.
[HK66] Alan J Hoffman and Richard M Karp. On nonterminating stochastic games. Manage- ment Science, 1966.
[HLWY22] Baihe Huang, Jason Lee, Zhaoran Wang, and Zhuoran Yang. Towards general function approximation in zero-sum markov games. In International Conference Learning Representations, 2022.
[HW03] Junling Hu and Michael P Wellman.
Nash q-learning for general-sum stochastic games. Journal of machine learning research, 4(Nov): 2003.
[JJNN21] Mehdi Jafamia--ahrrmmi, Rahul Jain, and Ashutosh Nayyar. Learning zero-sum stochas- tic games with posterior sampling. arXiv preprint arXiv:2109.03396 2021.
[LWY21]] Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-Icarning simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:21 2021.
[JLY22] Chi Jin, Qinghua Liu, and Tiancheng Yu. The power of exploiter: Provable multi-agent rl in large state spaces. In International Conference Machine Learning, pages 10251-10279. PMLR, 2022.
[LH14] Tor Lattimore and Marcus Hutter. Near-optimal pac bounds for discounted mdps. Theoretical Computer Science, 558: 188:122-110 1255-133,00 2014.
[Lit94] Michael L Littman. Markov games framework for multi-agent reinforcement learning, In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[LS19] Tengyuan Liang and James Stokes. Interaction matters: A note non-asymptotic local convergence of generative adversarial networks. The 22nd International Conference on Artificial Intelligence and Statistics, 2019.
12

---

[LYBJ21] Qinghua Liu, Tiancheng Yu. Yu Bai, and Chi Jin. A sharp analysis of model-based re- inforcement learning with self-play. In International Conference on Machine Learning. pages 7001-7010. PMLR, 2021.
[LZBZ21] Tianyi Lin, Zhengyuan Zhou, Wenjia Ba, and Jiawei Zhang. Doubly optimal no-regret online learning strongly monotone games with bandit feedback. Available at SSRN 3978421, 2021.
[MOP20] Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of o(1/k) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization, 30(4):3230-33)) -221,, 2020.
[MPP18] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Annual ACM- SIAM Symposium on Discrete Algorithms, pages 2703-2717. SIAM, 2018.
[MPS20] Vidya Muthukumar, Soham Phade, and Anant Sahai. On the impossibility of conver- gence of mixed strategies with regret learning. arXiv preprint arXiv:2012.02125, 2020.
[Neu15] Gergely Neu. Explore no more: Improved high- bigh-probability regret bounds for non- stochastic bandits. Advances in Neural Information Processing Systems, 28, 2015.
[PAI69] MA Pollatschek and B Avi-ltzhak. Algorithms for stochastic games with geometrical interpretation. Management Science, 1969.
[2DVH+22] Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990-999) 2022.
[Put14] Martin L Puterman. Markov decision processes: discrete stochastic dynamic program- ming. John Wiley & Sons, 2014.
[Sha53] Lloyd Shapley. Stochastic games. Proceedings of the national academy of sciences, 1953.
[SL99] Csaba Szepesvári and Michael L Littman. A unified analysis of calue-function-ie ciinforeeeent-eerrmig algorithms. Neural computation, 1999.
[SLY23] Zhuoqing Song, Jason D. Lee, and Zhuoran Yang. Can find nash equilibria at linear rate in markov games? In International Conference on Learning Representations, 2023.
[SPO22] Muhammed O Sayin, Francesca Parise, and Asuman Ozdaglar. Fictitious play in zero- sum stochastic games. SIAM Journal on Control and Optimization, 60(4):2095-2114, 2022.
[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[SSS+ 17] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Master- ing the game of go without human knowledge. Nature, 2017.
[S]]+ Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized q gelarringg in zero-sum markov games. Advances in Neural Information Processing Systems, 34: 11330--83344 2021.
[TK22] Tatiana Tatarenko and Maryam Kamgarpour. On the rate of convergence of payoff- based algorithms to nash equilibrium in strongly monotone games. arXiv preprint arXiv:2202. 1/147, 2022.
[Tse95] Paul Tseng On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics, 60(1-2):237-252 1995.
13

---

[TWYS20] Yi Tian, Yuanhao Wang, Tiancheng Yu, and Suvrit Sra. Provably efficient online agnostic learning in markov games. arXiv preprint arXiv:2010.15020, 2020.
[VDW78] Van Der Wal. Discounted markov games: Generalized policy iteration method. Journal of Optimization Theory and Applications, 1978.
[vN28] J v. Neumann. Zur theorie der gesellschaftsspieee. Mathematische annalen, 100(1):295- 320, 1928.
[WDCW20] Yuanhao Wang. Kefan Dong, Xiaoyu Chen, and Liwei Wang. Q-learning with ueb exploration is sample efficient for infinite-horizon mdp. In International Conference on Learning Representations, 2020.
[WHL17] Chen-Yr Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Online reinforcement learning in stochastic games. Advances in Neural Information Processing Systems, pages 4987-4997, 2017.
[WLZL21a] Chen-Ye Yu Wei, Chung-Wel Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent infinite-horizon competitive markov games. In Conference on learning theory, pages 4259-4299. PMLR, 2021.
[WLZL21b] Chen-Yu Yu Wei, Chung- Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations (ICLR), 2021.
[XCWY20] Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero- sum simultaneous- move markov games using function approximation and correlated equilibrium. In Conference on learning theory, pages 3674-3682. PMLR, 2020.
[2] Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, and Tong Zhang. A self-play posterior sampling algorithm for zero-sum markov games. In ICLR 2022 Workshop on Gamification and Multiagent Solutions, 2022.
[YM23] Yuepeng Yang and Cong Ma. O(T-1) convergence of aptimistic-follow--oeo regularized-leader in two-player zero-sum markov games. In International Conference on Learning Representations, 2023.
22] Runyu Zhang, Qinghua Liu, Huan Wang, Caiming Xiong, Na Li, and Yu Bai. Pol- icy optimization for markov games: Unified framework and faster convergence. In Advances Neural Information Processing Systems, 2022.
[ZTLD222] Yulai Zhao, Yuandong Tian, Jason Lee, and Simon Du. Provably efficient policy optimization for two-player zero-sum markov games. In International Conference on Artificial Intelligence and Statistics, pages 2736-2761, PMLR, 2022.
14

---

A Auxiliary Lemmas
15
A.1 Sequence Properties
15
A.2 Properties Related to EXP3-IX
17
A.3 Markov Games
18
A.4 Online Mirror Descent
18
Last-Iterate Convergence Rate of Algorithm
19
Improved Last-Iterate Convergence under Expectation
23
Last-Iterate Convergence Rate of Algorithm
24
D.1 On the Assumption of Irreducible Markov Game
24
D.2 Part Basic Iteration Properties
25
D.3 Part II. Policy Convergence to the Nash of Regularized Game
25
D.4 Part III. Value Convergence
27
D.5 Part IV. Combining
Convergent Analysis of Algorithm 3
32
E.1 Part Basic Iteration Properties
32
E.2 Part II. Value Convergence
32
E.3 Part III. Policy Convergence to the Nash of the Regularized Game
37
E,4 Part IV. Combining
40
F Diseussions Convergence Notions for General Markov Games
41
A Auxiliary Lemmas
A.1 Sequence Properties
Lemma 1. Let < h and lert 2
24 In 1-h
12 1-h
Then
II (1-3-b) jmi+1
<91n(()(*th
Proof. Define
- (kk 1)t" Int
We first show that s < $. Suppose not, then we have
(k: + 1)th Int
12 4
(because 2 12 > 4)
and thus t! 4(k + lnt lnt. However, the condition for and Lemma 3, it holds that gl-h > 12 In which Icads to contradiction.
15

---

Then the sum can be decomposed as
Σ II (1-1- Σ th (1-5")
==1
j=it1
i-t=
s=it1
Stx (1 t hh + s - 1+
-k
stx y + s x
2
(kte-( lat +8 x 2* 1
si* + ((k+ 1)
In + 1) x 2 x x
{9In(()( hth
--m
Lemma 2. Let < 1, < 2, and let t2 2
(22 In T-h
Then
i-k
II (1 (1-559)
s st
max 15ist
j=
Proof.
(1-5)) $ (1) *
=(1)
i-
S <22t = =44
max $ Sigt
-=1+1
i- II (1-
5 5(1-ti) S (exp (-r-h) _ exp ( (-1a1-n)
max isis
j=i+1
exp (-2 2 x 12 = トト
where in (a) we use Lemma 3. Combining the two inequalities finishes the proof.
Lemma 3. Let < < and 2
24 In 1-h
I Then 11 2 12 1.
I-h
Proof. By the condition, we have
12 In 12
0 h
2
2
1 h
1- h
Applying Lemma 12, we get
1-hh 12 In(tl-h) 1n
Lemma 4 (Lemma A.1 of [SSBD14)) Let > 0. Then x 2 2a In(a) + x 2 aln(x)
Lemma 5 (Freedman's 's Inequality). Let Fo c C... C F, be a filtration, and XI,... X1 be real random variables such that X; F-measurable. E[X,[FF-I] = 0. |X: < b. and Di= E[X V for some fixed > 0 and V > 0. Then with probability at least - 5,
n
xi 2 V (1)) 118)).
16

---

A.2 Properties Related to EXP3-IX
In Lemma 6 and Lemma 7, we assume that Fo C Fi C22 c.. is filtration, and assume that 2,, l are Fees-ueasurable, where E As. & [0. Besides, E [A]] and are F; Fe-mesuurable with Ela; a F[11] Ti,a and Elo Fi-1] l. Define li Lemma 6 (Lemma 20 of [BJY20]), Let c1, ....... ct be fixed positive numbers. Then with probability at least 6,
1a;=[ where 3 non-increasing. Zi.a +i
21, l - =0
Σ Biq + In(A/o) Σ c
Ci
Lemma (Adapted from Lemma 18 of [BJY20], Let ci2 c2,...,, CC be fixed positive numbers. Then for any sequence uis-..;;0 24 A such that x is Fi 1-easurabllee with probability at least
G cIn(1/8)
Ea Cat - 4.)
44)=0 = =0
max i5t
3.
Proof. Lemma 18 of [BJY20] states that for any sequence of coefficients w1, W2, :. we such that wi E 23,14 is Fi- we have with probability 6,
Σ Ca E. t.) I max c log(1/8).
iSt
Since o E A and 3; is decreasing, we know 25g x E [0, 28:). Thus we can apply Lemma 18 of [BJY20] and get with probability 1- 6,
a ( 41)
Σ
C2
x7 Pi l,) < max 23,
S log(1/0)
23
SEt B.
1=1
Lemma 8 (Lemma 21 of [BJY20]). Let C1, 02,...,... be positive numbers. Then with probability at least 6, for all x* A
Ea -1))
(In(A/8)
max iSt
Br
Lemma 9. Let (I1, 91) and (12, (2) be equilibria of fi())) in the domain 21 and fa(..) in the
domain Z2 respectively. Suppose that Z₁ S Zz and that (up(z.)) [f, y) f2(2, 3) S e. Then for any (x,y) E Zz,
fa(01.y) - f2(2, (1) S 2e + 2d sup
[Vf2(E,
(
where d = max min || ='|li
Proof. Since (x1, 91) is an equilibrium of fi, we have for any (x', 21,
fi(21,))) - fi v1) s 0,
which implies
f2(21, v)) f2(2', vn) s 2c. For any (x,v) E Z2, can find (0',2/) E Z such that [| v) (x', < d. Therefore, for any (x,, y) E Zz,
/2 - f2(x,
$ fe((1,0/) - fe(2', (la,,, y1) + / + 11 --1l Vy,z/z v)
2c + 2d sup f2 2l
(a-E222
17

---

A.3 Markov Games
Lemma 10 (WLLZL21a)). For any policy pair x, y, the duality gap on a two player zero-sum game can be related to duality gap on individual states:
max (V
V2 v) S
max -0'00//=''000
6,2 y
7
e. v¹
A.4
Online Mirror Descent
Lemma 11. Let
Za (La t (al)) 1 xL(
x' argmin 2EB
aEA
for some convex set Ω C A., E [0, 00) and E [0, H Then
(e= $ KL(u, KL(U, x²)
+ 1)
Σ
Fa (la)) +n Σ
Ee ln Ta.
aEA
aEA
for any u E S2 where in denotes the vector (Ea In TalaEA.
Proof. By the standard analysis of online mirror descent, we have for any u E Ω
(x u) ( + cl+))))) (l(u,x)) (L(u,x)))
-x--x')
x') clnx) KL(x', xx)
"
Below, we abuse the notation by defining KI. .)) = Ea (5o Le Fo + x.)) without restricting be a probability vector. Then following the analysis the proof of Lemma of [CLW21], we have
(x (x-x) (e-x')T(e + cln - - (K(x'',))
17
(a 4) (x- cl k
$ max VERA
77
La +6 - + +e -oleotra
Σ Va (Nea lu +1 nta the + e (e
(2-15 = - for 0)
Σ
Σ (neatia In + t nla (xd - xxo))
+
n
S1 1
Ta
2
+
Σ (nea dia + (5) (1t-nta va))
(nfa 2 0 and x1-06a ve- Fo 2 0)
7
51
+
1
Σ (neatta In nea meow uea In
(by Lemma 12)
7
11
Σ
3e
e
+
Σ (ne, In x.) 2x 2
(by Lemma 12)
7
<
17
Σ
2
1
Σ
(n/c 22)
(n S 1 and v. (0. 1))
17
Lemma 12. For (0, 1) and y > 0, we have --- 5 -yxt-v In
18

---

Proof.
1- y x cnn (2- 1) en (-(1))
y
In x.
(In 1
(for some ÿ E [-11 y - 1)
S y
In
(In + (-1+))
= y x: -
where the second equality is by the mean value theorem.
B Last-Iterate Convergence Rate of Algorithm
Proof of Theorem 1. The proof divided into three parts. In Part I, we establish descent inequality for KL(2ī, 2t). In Part II, give upper bound KL( 2)) by recursively applying the descent inequality. Finally in Part III, we show last-iterate convergence rate on the duality gap of 2 = (2), yr).
In the proof, we assume without loss of generality that to 1- In 1-k, k-k 1> (96 In (4(48)) since the theorem holds trivially for constant t.
Part I.
fe(xe, y1) - fe (((t,3))
= -xx - 27)) Gyy + c
Σ Fta In Tta Σ
tta In In
a
= -xx-22 -
Gyy + a
Σ (Fr,a - vt,a In Ieaa
- c Σ ziaa (I=t,a
In x1,a)
(x, (eil= 9: c(KL(I), + Σ 2zaa (Guuio
1 1/a = =olor
1lu aloz.
+ Σ Fiaa
(Gye)a)
It,a + Be
It,a +
$ E
= (by the definition of g:)
(L((x;50t)) KL( 2141) +11 E Ftaa
1aat =a]
+ me Σ
o In (22,a) cKLL + + +
17c
F2,0 + Bg
(by Lemma 11)
(1 KLL (2 xx)
KL F(41)
+ 2np Σ
1(a: al
tet InIn(()ta)) +
"1
Ft,a + B
(1 (1-mec))Ke (x KL 2141)
Tt
+ 2m A x
1
Σ
1 [aa = a)
-1) 1) +2 + 2mp Ac? In? (A12) + +5, + So
A
Tta Be
A),
(1 (1- KL (x 21) - KL x(41) + 10 (At) 2m, + + + +
(5)
Rearranging the above inequality, we get
KL(
< (1 -, KL xZ) + (f( vi) - fe(xe. v.)) + 1077 Am? (Att) 27 + +ng + +755, + +
19

---

where v - KL( 2(+1) (6+1), Similarly, since the algorithm for the y-player is symmetric, have the following:
KL (14+1:
(1 ()ee))L((() ve) + me(f((ee vo) fe(Fe, 107 (At) + 2n Axe + mI.. + +n55 +
where
d A 1
Wr.h + Ar. -1)
1bb
A
(-(GTaaz) ær)b +1 1|b, 61(-or + 1)
3. -
Σ
Vt.b
yt + Br
1 b)(-0 +1) - (-(GTzz)o+1)
3, -
Σ Vi.b
21)
yt,b + Bt
A (L(Wi+1:Mtt)) KL Me++1).
Adding the two inequalities above up and using the fact that fa y1) fe( vi) S 0, we get
(2 1 S (1 7),,,) KL ze) 2012 Al (At) 2nt AA: ncE + ncCe + us
(6)
where
for Xe; St. Sei Ut.
Part II. Expanding the recursion in Eq. (6), and using the fact that 1 - 71 = 0, we get
KL 2411 S 20 (AL) Σ whth 2A
Σ wint +
wimes
Σ
winiee +
w
(ermi
ferma
termy
term
where w T (1 ()) We can bound each term as follows.
By Lemma and the fact that that & 2 to, we have
term; S 50 (Aln In(t)( ( (( = -0 (Aln (AIIn(AA)) #).
Using Lemma with x² 11, 1 for all i, and e; whi have with probability 1- -
term O ln(At/0) max ai) (g) o (A th, O (Aln(At/0)= $
(AAln(AA//
where in (a) we use Lemma with the fact that t 2 to- Using Lemma with e; = win., we have with probability at least
5 7
termy 0
A
3,c+
In(At/5) Σ cf
i=1
A
i
II (1 - -hn
+ In(At/0) Σ
i 2kg
(1 (1-j j k
#=#+1
j=it1
0 A In(t)) My + t log(At/8))
(by Lemma and t t to)
0 (log((At/8)++
Using Lemma with c; win, we get with probability at least 2.
term; =0 [n(At//) max Ci
(a)
O (In(At/0)--kuttt = (I(((At/8))=
Br
20

---

where (a) is by Lemma and t 2 to-
By Lemma 13 and Lemma 1,
term; = =0 (n
Σ w w ==1
(II - (n
Combining all terms above, we get that with probability at least 1
38
KL( 4+11 = O ( (A118(11)))
(7)
Using an union bound over I, we see that Eq. (7) holds for all t 2 to with probability at least 1 (6).
Part III. Using Lemma 9 with (2, v) and x GTy with domains Ω4 x Sz and Aa As, we get that for any (x.y) AA A
z Gy - x Gyt S (0 In(A) +)) =
Further using Eq. (7), we get that with probability least 38, for any and any (x,y) DA AAXDA.
24 Gy x Gy S o (In(A)t+ + 1| 211b1) (a) o (In(A)t /KL(2)
0 An (A118))
where (a) is by Pinsker's inequality. This completes the proof of Theorem 1.
Lemma 13. |ua] = O (In?(At)(-))
Proof.
|ve [L 241) AL(=7,20+1)
< O (In(At) 22tz 211l1)
(by Lemma 14)
= (In? (At)t
(by Lemma 15)
Lemma 14. Let x. 21,22 E Ωr Then
(K( KL x) S (In(At)] x2ll1)-
Proof.
(L( (L(x2,2))
= =Σ 31,a In
F1.0
F2a0
Fa
Σ (Di,a x2,a)
In
31,0 + Σ
32,a
In
I1,a
In
22
Ni
Fa
< O (n((t)]) =22l1)) (L(
s O (In(At)llas - =22l1)-
Similarly, kL( - KL(x), x) S 50 (ln(AA)) - x2ll1).
Lemma 15. || - =f+1ll1 = O [n((at)
21

---

Proof. Notice that the feasible sets for the two time steps are different. Let (2++1.8/+1) be such that
T/-1- = P++1 1 (1 Pt-1 ) and Ve+1 2 + (1 Pe+1) Vé+1 where Pt+1 = min (1, 21 21- }.
Since (zi411 21+1) E Se+1 x we have that for any T141, 2 p++1 + (1-P++1) イ(+ 2 APa
Hence, (2/+1, v6+1) 92 Ωz.
Because (rī-1: Vi+1) is the equilibrium of fe+1 in Se+1 x S4+1, we have that for any ().v) E
Pe+1 X 9241,
fet1 (2, (1
S4t1 Vi+1) Jet1 (*i+1 V641) + Set1 (42:1111: (
2 4+1 x.41) + cet1 (LL vi+1)
1
2 f++ (I| 1itilli bly y lli)
(Pinsker's inequality)
(+1ll 11?-
where the first inequality is due to the following calculation:
E1+1 (L = fi+1(1, vet1) 1) - fo- (1741: 1o41) - V xi+1) s f141 vi+1) f--1 vi+1)
where we use (x 11+111 T z(+1) ) 2 0 since 27+1 is the minimizer of fet1('s Vit1) in 4.+1. Specially, we have
Set1 vi+1) - Set1 (4f41 ut) 2 1 t 1| 11
(8)
4
Similarly, because (zt vt is the equilibrium of in S2 Ge. we have
1
f6(441116) Se(xt, v641) 2 7 fellz//11 =1.1/
which implies
f(( f((2t,
f((1/-11 vi) - fe(x;, (((1),b(41) + (x fe(2441.08 ) + (6447,44.11 felll, vet1)
1
2 7
7 f011 - -1b sup
|V v/ ) 11o0 11 - xt+1ll 111
sup
[VVyfe wllooloo Ve-
sER
1
21+1l1 113 O (In(At) 1
0012441 - 2 7
Tellz
8
/3
1 8 - o
[nn (In(At)
13
In the first inequality, we use the fact that f( v) is convex in and concave in y and Hölder's inequality. In the second inequality, we use the triangle inequality, || [/ a)ll S max, (Gy).. In(x.)) & O(In(A)) At)), and V[Veee vlll & max, {(G +n(m)) & O(I(AA)). In the second and third inequality, we use | 21,,l,, 1l, O by the definition of 2/41- Combining Eq. (8) and Eq. (9), we get
3 c011l2 - 214+1l?
8
[I(At)
(1e1(07, Sel07, v(41) fe41(1741.15 1 + fe(# v.) O
t3
In(Att
= - (fe+1 f0) vi) + +
13
(In(At) 13
sup
[V - V.i.r.. vi+1) (x141, + O
22

---

[I((At)
sup z.e x
V 2411l1 + +0
t
Solving the inequality, we get
(V, Vie(,, + In
- 11 00
sup
(10)
(++ 2.gEf xBr+1
(1 - (+11 (+1))l(At)
In 1/2 (At))
0
Ver+1/3/2
Et+1
In(At)
In 1/3 (At)
+
thk
In (nt)t)
C Improved Last-Iterate Convergence under Expectation
In this section, we analyze Algorithm 4, which is almost identical to Algorithm but does not involve the parameter 3r The choices stepsize Y) and amount regularization e, are also tuned differently to obtain the best convergence rate.
Algorithm 4 Matrix Game with Bandit Feedback
1: Define: 7/ = kn, & ki where kn 2. ke 6.
24 = fo E A Fa 2 A Va E A}.
2: Initialization:: 21 1.
3: for 1,2, 2..... ..... do
4: Sample 02
and receive of E [0, 1] with E [or] = Gar
5: Compute Ie where It.a 1(u.=aj0: + ce In rt,a: Va A. 2ia
6: Update T++1 <- argmingea x IT ge m 21)
7: end for
Theorem Algorithum 4 guarantees mmaxryeaa (x) Gy x. T.GT Gy:)] O (VAIn
for any
Proof. With the same analysis as in Part of the proof of Theorem I, we have
fi (Z Ve) - fe(2t (=ttve)) ve)
M/ (L 21) KL F141) + A0m (At) + 21, + + S,-
where
Σ Tta
(Gut)a
1aa =aoo))
G _ Σ sia
[1aa aloz
(Gy.)a)
It.a
Ft,a
10r 10]= -1).
A
Ft.a
Unlike in Theorem here these three terms all have zero mean. Thus, following the same arguments that obtain Eq. (6) and taking expectations, we get
E, (KL 2411) < (1 KL 2)) + 20m, An2 (At) + E.[o]]
23

---

+ (Att)) t
$ (1 nx m.) + +0 112 An? (At)
(by Lemma 13)
(L +++1) and E, [-] is the expectation conditioned on history up to round Then following the same arguments as Part of the proof of Theorem we get
where v, = KL(=141 ++11
E
(++1)] 0
(Aln
Σ wint + In (At)
Σ w
i=1
(define w - (1)=++1(1-0p (1)5))
50 (Ali the + (A(())))= - -0(Ann(A00)
Finally, following the arguments in Part III, we get
max [z) (zTGy Gy ZT Gy.) s (In(A)t +VE (K(L =0 (VA)) =
2.
D Last-Iterate Convergence Rate of Algorithm 2
D.1 On the Assumption of Irreducible Markov Game
Proposition 1. If Assumption 1 holds, then for any L' = 2L (o/2())8) consecutive steps, under any (non-stationary) policies of the two players, with probability at least 1 1- every state visited at least once.
Proof. We first show that for any pair of states 8',8", under any non-stationary policy pair, the expected time to reach s" from s' is upper bounded by L. For a particular pair of states 8"', consider the following modified MDP: let the reward be r(s, a) 1[s o s"). and the transition be the same as the original MDP on all & s", while ss a) = (i.e., making s" an absorbing state). Also, let s' be the initial state. By construction, the expected total reward of this MDP is the travelling time from s' to s". By Theorem 7.1.9 of [Put14], there exists a stationary optimal policy in this MDP. The optimal expected total value is then upper bounded by L by Assumption 1. Therefore, for any (possibly sub-optimal) non-stationary policies, the travelling time from s' to s" must also be upper bounded by L.
Divide steps into log, (5/5) intervals each of length 2L, and consider a particualr 8. Conditioned on not visited in all intervals 1,2, ...... 1, the probability of still not visiting in interval v is smaller than (because for any s', Pr[ 2L] < TTa 2T. L 1. where Ty- denotes
21.
the travelling time from s' to s). Therefore, the probability of not visiting in all (Sg intervals upper bounded by 2 (9/6)
S- Using union bound, we conclude that with probability least 1 - 8, every state is visited at least once within L' steps.
Corollary 1. If Assumption holds, then with probability 1- for any 2 players visit every state at least once in every 6L (n(tt/5) consecutive iterations before time t.
Proof. First, we fix time 1. > 1 and define t' = 3L. 3n /8). Let consider the following time intervals: [1, t"), [t', 2t'], 2 ... it t' Using Proposition 1, we known for each interval, with $ players visit every state s. Using a union bound over all intervals, we A. in every interval, players visit every state s. Since every 21' consecutive iterations must contain interval of length L', we have with probability at least 1 - f.. players visit every state s in every 2t' consecutive iterations until time t. Applying union bound over all t > completes the proof.
probability at least
have with probability at least
According to Corollary 1, the remaining of this section, we assume that for any 2 1, players visit every state least once in every 6L (St/6)) iterations until time t.

---

D.2 Part I. Basic Iteration Properties
Lemma 16. For any x² 27+11
(3 07) f2(2003) 07)
< [1-7G))K(( , KL(x",07 (7+11)
An
+ 2
(1 7)2 X + + + S2()).
"/r
71-
(see the proof for the definitions of 2 (,))
Proof. Consider a fixed s and fixed T, and let t (s) be the time when the players visit s the 7-th time.
( 0f) f,(x",
=(i; 2²) T G 7 [ve] 0z c,0(2+) - - =+0(x*)
(#f x²) G + +y [v-] [v=]] 0r + c in - (=K, ,4,)
(G+ +T (v=e]) 07
1 = al (or + (Vett)) $7,a Br
(i) x²) T ge e, KL 17) + (
1[a = al (o= + vV,
- G + Pe [v="]) 87
+ (x+)T
$7.a + Br
(x(²
1/ |K[(x",
KL(x" (9711)
T/r
10m,A A (AT)
A
X
Σ
1 = al
-1
+5- + +,(x*),
(1 7)
7)
A
G + Br
X
where we omit some calculation steps due to the similarity to Eq. (5).
D.3 Part II. Policy Convergence to the Nash of Regularized Game
Lemma 17. With probability least we have
(6)), for all S, t and 2 such that (=(s) S t.
KL 24) < 0
where kę = min ke, li - kg. ha kn 2k }.
Proof. In this proof, we abbreviate (+(4)) as S. By Lemma 16, for all i S T we have
KLL(E,, #f+1) S (1 7/4 KL ($f .f)
M) (A 0 f (+t. 2n A
A An2 (AT)
+ (1 71-
(1 - 11
d + n + +
Similarly, for all é S т, we have
KL KL Di+1) $ (1 7,t,)K(( 01)
- m/s (1) (2; 022 f7
10n A In? (AT)
2n A
X, + 5 +
(1 7)22
(1 (1-
25

---

Adding the two inequalities up, and using fi (i)e: 02) f2, I 0 because (#),. Vl.) is the equilibrium of fi we get for I T
A An
217 A (1 (1-7)² X + + + +nss + of.
KL (2) 1e: 2 $ [L) (2) 21) +
(1 21-
(11)
where vs. KL( 1(
KL 2/41) and -
for 0 E,. Gi-
Expanding Eq. (11), we get
KL 7+1+: (2 7++1 -
20A4 In (Ar) (1 7)²
W m +
2A
w 2 A
+
w n
+
Σ
w n
w us
(1
7)2
term
tering
termag
term
terms
These five terms correspond to those in Eq. (6), and can be handled in the same way. For term, to term4, we follow exactly the same arguments there, and bound their sum as with probability least
1-01 57
Σ term, = (AIIn((AAT/))) (F- + -2 + -hotk + + 2 + T No t-k ))
1=1
To bound termy, by Lemma 14 and Lemma 18, we have
| O (In(Ar)) || 2f+1xlli O ( Therefore, by Lemma 1,
terms = Σ
th 2hr
1=1
Combining all the terms with union bound over s E S and T 2 1 finishes the proof.
Lemma 18. For any sand T 2 0 such that L, (x() < 6, - 1 = o (In (SAA/8))) - -atth
Proof. The bound holds trivially when < 2T. Below focus the with > 2T. By exactly the same arguments as in the proof of Lemma 15, we have an inequality similar to Eq. (10):
|| -,
2 #1+11111
In 1/3
sup [V Vi (x", ,.,,"")) 10o + / 3/2
G+1 y
Vitn(0) (Er - ((())
In (
0
G+1 sup
V/s (+)))
VEr4773/2
6+1
VElos Viincos + In(AT)
0
67+1 sup
(12)
Since (s) < and we assume that every state is visited at least once 6L log(St/0) steps (Corollary 1), we have that for any state s', _ 2 (og(78)) 1. Thus, whenever V, updates
(-(#)
between (s) and (s), the change is upper bounded by 1 (og((1/) -1)- Besides,
(x())
between (s) and 1++1 (s), Ve change most (ottg(St times. Therefore,
(i(
Val (
1
(s) 6L log(St/)))
-1)
tha S
1 1-
GL log log(5//))
(olog(St/0)
1
(oogo))))
L In (St/8)7 Ra
=0
(13)
1-7
where the last inequality holds since hi < Combining Fq. (12) and Eq. (13) with the fact that ----- -ke finishes the proof.
D
26

---

D.4 Part III. Value Convergence
For positive integers 2 i, we define af a, IT (1 [a))
Lemma 19 (weighted regret bound). With probability - (()), for any s, any visitation count
7 2 To, and any E 9
Σ o - so (SAr/00
1-7
where k'' = min (ky, Ni, ka kg}
Proof. We will be considering a weighted sum of the instantaneous regret bound established in Lemma 16. However, notice that for Jú, Lemma 16 only provides regret bound with comparators in Q441. Therefore, for a fixed x² E f2,-1, we define the following auxiliary comparators for all i T:
Pi 1 (1 -p:)
A
where p; A (++1)2-(4+11² Since x E S7411 we have that for any a, 3f,0 2 P. (++))[[[+1)=-]]
+ 1-p. =
A A(r+1)²
A(FF1yz and thus 75 E 94+1-
Applying Lemma 16 and considering the weighted sum of the bounds, we get
(27,))) - 0())
7144 KL
10 (AA) (1 7)
2n,A
+
(1- -- Ai + E + (((35)
a mi (3)
=- 1,7)
(notice that - 1111) = 0)
11i
7K-1
termo
104 (AT))
Σ
a
2A
o
n
(1 7)
7)
term
terma
terms
term:
( [c
mhEs)
a
a
i-1
termo
Σ KL( 219)
+
(KL 2:
(L(";,4;))
17i
77--1
7
(0)
a i-1
0
0
(n(AA)
1 - 12
7/1-1
0
In(Ar)
P - psl
Th-1
1=
1-1
0
In(AT)
TK-1
(i 1)2
1=2
In(AT)
-0
2n--
(by Lemma 1)
1-7
where (a) is by Lemma 14 and the following calculation:
af T7C2))
X
17-1
1h-1 x
x
1 TTCs
i
ky tha
X
1 - i Ry Rr
<
<1x1=1
1)
07
0;-1
1 Oki
i
1 i ka
27

---

We proceed to bound other terms as follows: with probability at least 1 - 10(
(AT)
--h))
term =0
(Lemma 1)
1-7
Al1(SAA70))
0z x max iST BT
term = 0
(Lemma 7)
1-7
Aln(SAT/0))
No Ny tho
=0
(Lemma 2)
1-7
A
term 0
Beck
+
In(SAT/0) Σ (at)²
(Lemma 6)
1 1-
1-
A Aln(Ar) K/
1
7
=0
+
In(SAA /6)
Σ
a;
(Lemma 1)
1 1-7
1
1
i=1
AIn(SAT/O
(T-Ks + 4p
=0
(Lemma
1-
1
term
Σ
-1
+
Σ
at (x:)((a))
(by the linearity of (-)-)
A
==1
af
max SST
(Lemma 7)
1-7
Br
In(SSAT/0) tRath
(Lemma 2)
1-7
Combining all terms, get
af (ff(2))0) (( = Aln (T + - + -ha
-a mhathh)) )
(14)
1-7
1=1
Finally,
In((Ar
at (1 f(00000) 0
0f 1: x0'111
1-? ?
In(AAr
0 In(AT)
=0
pi
7(
(15)
Adding up Eq. (14) and Eq. (15) and applying union bound over all s E S and 5 finish the proof. Lemma 20. With probability at least 1 O6)), for any state & E and time t 2 1, we have
n
(nt(///)) A (St////
t
The L [II(St//))
VV V S O
In
71-
1 1-7
1
where k, = min (keys kg, Ki - ko ke).
Proof.
Fix and visitation count Let tú be the time index when the players visit for the i-th 8
time. Then with probability at least
377
Vr. =
a (o., + +TV
a (a" ( + (ve)) 0f +
Σ a 0r, + + - 2s5T (C'+>Bawp [V:'] 06
<
i=1
28

---

of (77 + - - +
In(ST/8)
Σ 1=1
(a
1-^1
1
(Azuma's inequality)
(n(SAA/00)T-- "
0 (. +0
Er In(A)
1-7
(x' 01 + ey (n((A)) Aln (SAA(557/)))
1-7
(k'' is defined Lemma 19 with R' S 2k3 + ki - -k3) = "2)
ASn((SAATO))
a (I¹)²
G* + wBB [i_"
D +0
1-y
(5AT/5AA/ -K-
a (x*) T G 7 p [v=¹]
13 +
Σ
a max Vid -V.¹ +0
1-7
i=1
(ATI"(AAr/0)T
S min max(x*)
+ [io1 1¹² + 7 Σ
a max Vi - V. +
1-7
i=1
+ + ( Al (SAT
S V of max V V.¹²
1-7
Similar inequality can be also obtained through the perspective of the other player: with probability least 1
2 E a max V V. -o - (SAT/00))))
1 1-7
which, combined with the previous inequality and union bound over 8 and 2 gives the following relation: with probability at least (6)), for any E S and 7 2 1,
A (SAT/8)T (
V22 V
o max Vi voll +o
(16)
1-7
Before continuing, first some auxiliary quantities. For fixed define
1-7 u(t)= (
16 x 6L (n(t/6)
t
In
for fixed (r,4) further define
t
v(T,1) 37 In
Now we continue to prove a bound for VV, V.7l Suppose that Eq. (16) can be written
max Vi v.¹
(SAr/6)r 1-7
V V
(17)
for universal constant C1 > 1. Below we use induction to show that for all f,
IV. V. s +(1)= & 8C1 (1 (6L (GLIn(S()0)((()) +1) 11Z11(((1/0((((())11) This is trivial for Suppose that Eq. (18) holds for all time 1...., and for s. Now we consider time and fixed state s. We denote L' 6L (t())/)). Let n and let 1 ti < < t, <
he
(18)
(1 (1-
t
29

---

be the time indices when the players visit state s. If < ['(((t) + 1), then Eq. (18) is trivial. If t シ L' L'(((t + 1), we have 2 f 1 u(t). Therefore,
IV2- V
VV - V
(t, is the last time up to time when Vi is updated)
In (SArr0)) k.
of max ViJ vel
(by Eq. (17))
1-7
((T.))
(SA7/0)
5
Σ
max
Vg -
-V
+7
Σ
of max ViL -V-l
1=1
im((T,))+
(0)
(1-7)3
1
(())+ (SAT00)
5 7T X
X
+
13
1-7
1-7
2C An (S /0)T k.
0 (1,)+ +
(induction hypothesis)
1-y
+=v(T,))+1
(
1
2C An (SAT/5)T -
Sy
(t) x
1-y
(=((,,))+1
(by the definition of t and that w(-) is increasing function)
(8)
1-7
(t) + 20, A
(5)
2
1 11
2
1-7
|+ 1
1-7
20, 20 A In (SAt/0)
L K.
Sy
((1) +
277
2
1 1-7
(t2 4 - 11 2
since t 2 L'(u(t) + 1) 2 2L')
1+ 1 1-?
((1) + 1-1#(1)
5>
2
2
=t)..
In (a) we use the following property: if r 2 u(t) and S v(T, t). then
a = ikk ke
(1 (1-j j ka) S (1 )
jmit1
< (1-- - kk)) 3r In - S eexp
(-p ku . 3o In
t
(1 - 7)
1 1-7)
t
In (b) we use the following calculation:
t t++
1,41 - t
L'(r + 1)
1
< 1
4
4
t
T ++1
11 T
((T,t)
T +
11+1'
47 In
1+
=1+1
1
1-
4rka-
In I-7
1
11+5'
Tol
1
1
1 1-7
11+1'
2L'
30

---

=1+ -7
2
where the first inequality is due to the fact that at time state s has only been visited for 7 times; the second inequality is because for any k > J. we have t, 2 j and f 1--1, 1, I [' - j); the third inequality by i 2 v(T, f); the fourth inequality is by the definition of v(T, t); the fifth inequality is because Ra In 1-7
16L., the last 1-7 since J 2 u(t), and + < m(t) v 1-7 since u(t) 2 1-7
41
1627
inequality is because
1+ 10a
S 11+ 2 for a E [0, 1].
D.5 Part IV. Combining
In this subsection, we combine previous lemmas to show last-iterate convergence rate of Algorithm 2 and prove Theorem Lemma 21. With probability least 1 -6)), for any time t 2 1,
AL 4+1// (SAL/8) In V (1/(1- xtoo
max (V 8,2-y
V (.Ne)) S O
z..y
(1 (1-7)2+1/=
Proof. Using Lemma 10, we can bound the duality gap of the whole game by the duality gap on an individual state:
2 max (1 (xy x² QQv(E)
max (V2... V V
y s,
2 G + +B pa (v=()s² - z² x0(6" + BB [v-1) [v=])eri)
max 8
1
2 G + TB p [io (vo)) - x² (c* 7B [100 (ee))u²)
<
max s,
y
4%
max VV V!-
With probability at least 1 - (6), for any s, x, y², and t 2 1, denote r the number of visitation to state s until time then
x G²
PP V." y² - x² (a - y-EEs V: 10)eet
2
G + PP (o))er 30 x² (G' + 77 PM [i:") 1)07 07
s #: 6 - YE
(6'
Vi((en) ) 30 20(6 (c' + +B (i((e)) )0²2 + 2m2xa//2 VVe' - Virioo
S 5at (a'
Vi (i( g0 x² (a" 71 p. (V(((s) 87. + max IV:" vil + O(1 25.11)
22, In(A) 0
11)
+ max IV,' VE'(o)
+ +O(V 22.))
(Lemma 9)
00 In(A) 1-7
L LIn(St/0)T -ha
-
+ +0
+ +0
(Lemma 17)
1-7
Combing the above two inequality with Lemma 17 and Lemma 20 and the choice of parameters I
ka D4c. kig = g+2. kg 3+2. and Kn g+z we have kg = min(ks - ke: kn - Kg. hig - kn
5
2 = g+e. hi- = min (kin; ka. hia ki hir = 945
1. and
max (V V....) s.
2
x (c(("" G + BBoaPs [Ve') (v=e))= 2¹ r0((" (0*+"Eewpp (roT)utt
47
max
max V22 V.2
I 1-
1 I-
1/2
LT min min(ki Haa)
Ann ( (L.
(I.t(/)/))
[IIn(S/88)
R.
< 0
+
(1 (1-
(1 7)3
1 1-7
1-7
t
31

---

AL (SAt/0)) In (t/(11 Fotee
0
(1 (1-7)8t1/E
E
Convergent Analysis of Algorithm 3
E.1 Part I. Basic Iteration Properties
Definition 1. Let (s) be the T-th time the players visit state s. Define &; = 2 (,) 07 Vi, (s)
a = at, (o), 87 (s)) Furthermore, define
(, xxT (a + Booup fx & T G BBew \ircol)) eδ(x) + eœ(y).
Li Ki(col c②(x) cb(y),
Lemma 22. For any x' E Ω
8 B (17 v7) - (z"
(z" 17)
me k (tt) KL(x",25H1) 22+11
10nA (n
2n
d E + +,((0
"
(1 - 7)2
(1 7)2
where
1 Σ
1 [ = a)
TTo + 3 -1).
l a] (or -VVel11
S = (#:)
(G" 4'g'
Vi
02
20 + B
1la al (or + +V+(11
(xx")= = =0))=(())) (I")T
G ps V2² 8r
# + 3
Proof.
The proof is exactly the same as that of Lemma 16.
E.2 Part II. Value Convergence
Lemma 23 (weighted regret bound). There exists large enough universal constant (used in the definition of bns,) such that with probability O(8), for any state s, visitation count т, and any xx Ω
Σ af ([ - (,"" < bns,-
Proof. Fix state s and visitation count 7 S T' Applying Lemma 22 and considering the weighted sum of the bounds, we get
(10)00))) f((x" 0.))
- (1-16) KL .2)) KL(x" (L_(,, (.411)
(07
27A
+ (1 7)2 di + + (7(0))))
(1-7)²
nTc)
KL(x", , +t) + Σ
a,((1 nc)
(I,"",
1-1
KL(x", " 23))
1
71
1=2
termo
32

---

10 (AT)
2nA 7)2
(x))
7)2
1=1
1=1
term:
term2
termy
term
Since a 1-1 oi ((1-aa) at 2 1 ac
[ nc)
at(1 - as))
termo
I - KL(x" (t1) +
KL(x",
11
7
1=2
a
a
In((AT)a
S IInAA)
=0
We proceed bound other terms as follows: wiht probability least
8
ST
A Alnn(AA))
term =0
E of =
(1 71
27/A
1 1 = al -1) )
term2
Σ at
A Σ
(1-7)²
2 + +
1=1
a
2A
max a'11 ist
x x0
In In(AST/0)
(by Lemma 7)
7)2
A AI(AAST/W))) /5)ar 3)
(1 7)
A
terma =0
Ba
In(AST/0) E(at)²
(by Lemma 6)
1
1-
AB 1
In(AST/6) Σ atas
1 1-7
1 1-71
Aln(AST/O)) (B (8+ +aa))
1-7
In(AAT/5) 1--
at max ISt 3
)
In( In((AT/8)a,
term =
(1-7)8
Combining all terms and applying a union bound over a E S and r, we get with probability - (
such that for any E S, visitation count T, and x E Ω
(f122) f((x9 = 0 Aln (AST/8)) (1+ 3 (n11 + 3- ()()(T)
(1 -)
(SAT(SA) + acr ktr/11)
(using 1) S 3)
(1-
This implies the conclusion of the lemma.
Lemma 24. For all I 8, V 2 Vi
Proof. We prove it by induction on t. The inequality clearly holds for = 1 by the initialization. Suppose that the inequality holds for 1,2,..., 1 and for all s. Now consider time and state Let mi, and let < t < t2 .... < t < t be the time indices when the players visit state s.
By the update rule,
V.
V Σat a (IV V + 2bns; > 0
1=1
33

---

where the inequality is by the induction hypothesis. Therefore,
- min [ig H} max [V',0} > 0.
In the last inequality we also use the fact that V:" S H and V 2 Note that by the induction
hypothesis and the update rule of and V we have 0 S V < V S H for all 8 and
S i H 6-1 1.
Thus Vi- a bns,, I H and similarly V 2 0.
Lemma 25. Letc = (c.....,rr) .r) be any non-negative sequence with S Cmax Vi and Σ Ct C. Then
T Σ
(A (1-
CmaxAS AS In (AST/00)
Vitt1 - Lett) 00
Cmax
1711 7)3
mm
Proof.
Ze Σ
(V(--1 Vini
C
t=1
T
T
Σ
C
Vi+1
- - +
Cmax ΣΣ
4+1 - Vi
+ +ikiit -
t=1
t=1
T
T
C--1
Vi V + 0
Cmax ΣΣ
ai
(shifting the indices and define co = 0)
1 1-7)
1=1
1=1
C-11
110t -V +
Cmax
x H IT
H+1 (using a; H+1
1 1-7
nr (()
((-(s)-1 (Vi(0) - -V),(s)) +0
(mmax T
(H = In T
(1 - 7/2
I-7
Σ (s)
J-1
.maxx S II2 T (1 - 7)2
Σ
(1)- Σ
V ((())
2 + +2bss: + (.)))
i=1
(s)=
(s)
29 Σ
Σ
Σ
(Puee ((()) (1()))
(7-1)-(0)-1
1=1
1=1+1
2,000
(s)
Cmax Sln? T (1 72
+
Σ
Σ
(t)(s) (-(s))-bns,-11 +
2=2
T
C./Cmax
Cmax Cmax Sln² T (1 - 7)2 71
de
+
Σ Σ
bnsx
1=1
T=1
(C. - Enr ((s)
(x)))-1)
Ca/Caux
Cmax Aln + ar/n))
Gmax In2 T
+0
Σ Σ
71-
(1 - (1-
T
(AST)((AS))8)5
Caax CuaxAASh 1n1mm(157 /8) 711-1)
vittt) +
(19)
(1 (1-
1=1
34

---

Note that c, is another sequence with
1
c S Émax I Cmax sup
Σ
a
S
1
Cmax
H
7=1
and
T
C
2m1
tm1
since gives
Di=1 af 1 for any 1. Thus, we can unroll the inequality Eq. (19) for H times, which
1
Cmax T
(AST/0))
Cmax AS An (AST//)
Ze S-" (1 H
H
1 1-7
(1-
771 711-))2
CAIn
CAIn((ASTOO)
Cmax AS In (AST/6)
=0 71-
1711 7)3
where in the inequality we use that (1+ H 3 S e and 7" = (1 (1-(1-7))) (1-(1-)) = 7.
"
Corollary 2. There exists universal conslunt C₁ 0 such that for any E>
CASTT(A with (1-7)²
probability least O(6),
T
It T
(Reewrer Vi - v=]) V/ 2 8 S0
(ASTN
n/ē(1 733
Proof. We apply Lemma 25 with the following definition of ce:
= 1 12 PB [Vi vz y8 26).
which gives
CAA (AST/5)3 (1 7)3
AS (AST//8)
Σ Cr Villl
veete)
SC2x x
(20)
n 7)
for some universal constant C2 and C= ΣT Ct. constant C3 > 0, with probability 55
By Azuma's inequality, for some universal
Σ
qat Evnnp
Vi -Vi 3/
Σ a (
V(ri))
T
C3
C3
5
In(S/o)
C
VIn(S/oJC
7
1-7
1=1
CB
In(5/5)
< C3
+ +Cs
(by AM-GM and that 1)
9)
1/(1
7)
(21)
Combining Eq. (20) and Eq. (21), we get
Σ qart (Rgwwn
[Vi - vi')) De-
CCAIn(AST/ON)
AS ASST((AST/))
< (C2 + Ca) x
+
(1 7)
111 7)3
35

---

By the definition of Ct. the left-hand side above is lower bounded by >Σ>= C 2(C₂ + Ca). Then by the condition on e', the right-hand side above above inequality is bounded by
C. = Cé. Define
Ce C1 (AST//)ST/)) 2
2
771 71
by the condition on é. Combining the upper bound and the lower bound, we get
AAS (AST//5)
C C
1/ē(1 -))3
Lemma 26. With probubility at least 1 - (δ), for any 1,
E (
E (II(AAT)
V -
2 -
1-7 (1-7)
Proof. Fix a and let (s), and let be the time index in which is visited the i-th time. With probability least 1 ST have
Vt Σ of (or, + +VVi bns.)
ar (1 vi.) ed(x?) eo(vi.))
+
Σ
a (ar, +V - x (G" + "E -Bgwp (
1=1
a bns;
d [e cd(x)) -e(vv)))
+
1
2
(a )2 log log(ST//)) (ST
bns,
i=1
(by Hoeffding's 's inequality)
1
ar
He vi.) cδ(xi) - eσ(vii) + v 2a, log( ST/6)
a b.bs
E of S 1)
(2 v(.) + cσ(xx)) e bns,
a
-=1
(E at 1 and bns, is decreasing, and ya, S a 11 G 9)
"
min xEf
a [( ..r() Σ a (cp(2i,) eσ(vi,))
(by Lemma 23)
a (x²) T (a + + V. vl. ((cn((AT) (xa 2 AT for any Q2)
min
Therefore, using union bound and I, have with probability 1 - 6, for all and
V max 0} min Σ ai (x")
(o* +rEE
(xa)) vi C4eln(AT)
(22)
t=1
for universal constant C4. Next, use induction to show the first inequality. Suppose that
C4e (n(T)
V V +
1 1-y
36

---

for all and t' < t. Then by Eq. (22),
a (20) G tyBpppa Va' 1-y [)
vi. (
Vi S min
Cieln(AT)
at (x*)* (o+ + BB [v=]] vi. + C4eIn(AT)
min
1-7
max a (z") T (G* +yBuNp [w:]]) 8² CacIn(AT)
< min
1-7
min max (x*) G + 7 [v:])>> 1² CacIn(AT)
1-7
CacIn(AT)
which proves the first desired inequality. The other incquality can be proven in the same way.
E.3 Part III. Policy Convergence to the Nash of the Regularized Game
Lemma 27. Let 0 $ p < 1 be arbitrarity chosen, and define
(= v²) _ (x" v²) + (1 - p)f=("", ,1"
=x²- + [pV ( + +1- ()V-)) 1² -c6(x") + cō(y").
Furthermore, let 2*+ (8t.1 be the equilibrium of (z, y). and define 2i- 24. where = (8). Then with probability at least 1 - -(8), the following holds for any 0 <<< < 1:
(1()))))
5 An
ΣΣ
1 KL 2)) 5o
1je2e' 7)
if 17 and B satisfy the following
Cs(1 Cs(1- Et
B <
(23)
A (AST/8)
Co(1 - 7)
n n A (ASTAA
3e'
(24)
with sufficiently small universal constant Cs Co 0.
Proof. In this proof, we write (²(2,)) as S.- By Lemma 22, we have
(l. (4+1) $ mH6 (I... +
[ (R 11)) (79.
10n A In2 (AT).
2 A
+
+
X + + + n
(1 7)
(1 71-
Similarly,
KL 01+1) S (1 (1- KL (L 0/) + 11 F. 02 F1 (21.
An?
21 A
+ 35. +
(1 (1-7)) 7)
(1 (1-7)²
Adding the two inequalities up, we get
( 1
20n, A In²
21
< (1 - M7 JL 2,) +
X + + + +5 +
(1 71
(1 - 7)2
37

---

17 (Fi 0f) - (2 0f) + (3 (f) -F.
(25)
where of KL 2 - KL 114 and D+ = =0 + By Lemma 24, we have
y)) - Ji (zz y) for all æ, y, and thus ( if)) [d vi.)) - (2 (2 27.) S O.
Therefore, Eq. (25) further implies
5 (L 22) + A0n2
21/² A (1-
Ai +zi + +n + ug + +
(1-
$ (1 Kl))L, 2/) + A0n)2Aln"(AY
21/²A
1
1 zne nce
1
(1 - 7)2 X + nx + n + ur + 2 nee +
71-
where A = T (2). 02) (at, ūf) and in the last step we use a S [a - b|+ + b.
for for all s and r (we show that the inequality holds for any fix s and T with probability - O (55)) and then apply the union bound over and T).
Unrolling the recursion, we get with probability at least 06)),
KL 127+111 #;+1)
20n2 An? J
20 A (1 7)2
$ (1 n7) KL 21) +
(1-mc))
Σ (1 (1-
-
(1 (1-7) 7)2
term:
term2
5: + (1 me) m
m7e) -
:=1
terms
term
m)6
+ 716"
+ 2
Σ (1 - -n)) -()
(1 nc) n))
--i
A
2 ce
1=1
1=1
terms (8.T)
terma(s.T)
(a)
(- n 1/2A
BA
1
in
O(e get In(AT)) + In (AST/8) O
(e(1-7)²
B(1 7)2
+
c(1-7)
+
V
E
3(1
+ term5(8,7) r) + z² + (s,
3 2 term5(s,7) terma
O (e ver (nT)))
(26)
4
where in (a) we use the following calculation:
n² (AT)
term; 0
S
(1- 7)2
T7C
c(1 - ([1-)}²
1
maxis (1 - m)
In In(AST/0)
1²AA
In (5SST/8)
term; 00
=0
(1 (1-
8
(1 7)2
(by Lemma 8)
terma 00
nA
Σ
3(1 n7) + 1 In(AST/8)
Σ (1 -e)
(by Lemma 6)
I 1-7 iml
2ml
-0
3A
+ ( (I((AS/))")
c(1 7))
maxisr (1 - -m)===+ (onT=InnASTTI (nST///))
= (010( (nIn(AST/O)
term 0
1-7
8
B(1
(by Lemma 8)
38

---

and in (b) we use the conditions Eq. (23) and Eq. (24).
We continue bound the sum of term; and term; over l. Note that
(s)
(erms(8,77) S Σ ( Σ (1- m76
1
Σ
Σ
(8)
of
s 0
(
5 (AT)
1Te2 7))3
1/6
i=1
(27)
where in the last inequality we use the following calculation:
my- (x)
(x)
b S ())((A))
1
1/111
2
(by Lemma 14)
(+)
ln(AT)
=((((T)) x
x
sup
p
V VV + (1 p) V. V f++1
x=1
o'
(by the same calculation as Eq. (12))
In (AT)
T
vs' V
+ Vid - -iiil)
0
Lit1
t=1
In (AT)
Sln T
x
(1 7
(V Vital S #47
x T = s) by the update rule)
(AT)
c(1 77)
and that
Σ (s)
Σ (s,
Σ (s)
1
7
(1 (1-mE) -
- ee'
(s)
(1))
<
Σ
m1 - mE)
A
2 ce'
1
(1))
EE
(+) Jmax
1 ( < < S ce'2)+1] ee'20+1
(define jmax logg (T-)) )
1 Jmax
A cc'2 cd'21+1
1=1
AS (ASTT()AST8) (AST/8) 1) - 7)3
0
Ee'2" 2
(by Corollary 2 with غ = cc'2 and the assumption that cc' 2 A Al
(1-7)3
AS (AST))))))8) 0
(without loss of generality, assume loga (a-)ee)) S log T)
nc(1 73
(28)
From Eq. (26), we have
Wir (s)
Σ Σ
1 IKL (3.,, 24) 2 el
T=1
39

---

"T (74)
ny ((s)
1
OlE ner In(AT)) 2
e'
Σ r=1
terms(#, r)
1
12
12
no++ (o)
+ Σ
Σ
1
terma
12
T=1
In(AT)
(*
S x8 0
+ +
nyEe'
T7EE (1-
00
1762ce(1 - 7)
where in the second-to-last inequality we use Eq. (27) and Eq. (28). This finishes the proof.
E.4 Part IV. Combining
Theorem 5. For any (0, 1 there exists proper choice of parameters ,,8,7 n such that
T
S "n
Σ max (z 12 V+ > u 0o
u 7)
with probability at least 1 (8))
Proof. We will choose such that u O7 1-7 with sufficiently large universal constant Cz. By Lemma 26, we have
cln(A7)
max stt Q.: x² Q+:x(") a.y
< max (zi² (GEL + -BBgs [i]]) 1rt zºd (cr +
<II(ATT
+
1 7
(z7² (G* + BBg (v:]) yrr yr - xel (Bewn
max 2.y
V
y/
+
Therefore, we can upper bound the left-hand side of the desired inequality by
max Vi
(G*
+7pew [V:]) 4 - 5xT (a+ + TRegs
V2 yi
2
4
(Gth Eeewpe [v:]) 11th - =za (Qth +7 [i']) v/.
u
max max
4
T Ft
(G + -ES (v-']) v8t x; (a# + +Bs V. 11²²
>
1
x +
(Gt
Wog [Va] (et)yr y*r min o7 +
Vc"
y¹ 2 "1
(29)
For the first term in Eq. (29), bound it by
(x) (s)
Σ
max ((())10)) (22. 2 u
#44 (ElI((AT)
(s)
Σ
Σ
max (27 - (1
40

---

(s)
In(AT)
max ((( S (8fe 1ii.) + 0
||a]
2 ||
1-7
(because (2,
similar the calculation in Eq. (9))
1-7
(here we choose (ii Vir) to be the equilibrium under (z, y))
In In(AT)
Σ
O
[ 21.111
7-7
(T)
KL (ix) 2/) 2 o ( x²2 7)
-
(AT(
5 (SAT11))) 50
(by Lemma with e'= 2 ( )
172u2(1 - 76
The third term in Eq. (29) can be bounded in the same way. The second term in Eq. (29) can be bounded using Corollary 2 by
SAln
n 77)3
Overall, we have
[m (t) Q" -x Q.
Σn max a.y
Q2'((') vi¹)
u <0
A (SAT1/0) nye2u2(1 - 7)
(30)
Notice that the parameters e, 33 needs to satisfy the conditions specified in this lemma and Lemma 27, with which we apply é' θ =²(11-) (under fixed u):
(SAT)5))) ) The constraints suggest the following parameter choice
In(SAT/8) )
111 (1)
c=
(1-7) 7-
2² 7)
66'
ASA
Aln A (SAT/O)
(1 (1-7)
u²(1 -7)
n= θ
Be'
An
A 1n (P
Using these parameters Eq. (30), we get
1S2 (5AA/10))))) u - 76
max 21
(5) Q. -~
Q.
<0
F Discussions Convergence Notions for General Markov Games
In general Markov games, learning the equilibrium policy pair on every state is impossible because some state might have exponentially small visitation probability under all policies. Therefore, reasonable definition of convergence is the convergence of the following quantity to zero:
T
Σ
max (V.L. 2e.y
(31)
T
J.y
x.
which is similar to the best-iterate convergence defined in Section 3, but over the state sequence visited by the players instead of taking max over 8. also a strict generalization of the sample complexity bound for single-player MDPs under the discounted criteria (see e.g., [LH14, WDCW20J).
41

---

The path convergence defined in our work is, on the other hand, that the following quantity converges zero:
T
Σ
max Ti Qu vai Q
(32)
Since maxy (x* is stronger than Eq. (32).
Q2y²) S maxy (x* [22y,0)) = maxy Vi.y for any æ, the convergence of Eq. (31)
Implications of Path Convergence Although Eq. (32) does not imply the more standard best-iterate guarantee Eq. (31), it still has meaningful implications. By definition, It implies that frequent visits to state bring players' policies closer to equilibrium, leading to both players using near-equilibrium policies for all but o(T) number of steps over time.
Path convergence also implies that both players have no regret compared the game value V.', which has been considered and motivated in previous works such as [BT02, TWYS20]. To see this more clearly, we apply the results the episodic setting, where in every step. with probability state is redrawn from s ~ for some initial distribution (every time the state is redrawn from we call it a new episode). We can show that if Eq. (32) vanishes, then every player's long-term average payoff is at least the game value. First, notice that if Eq. (32) converges to zero, then
7 the
T
(V
xx O V+ S max
Q:y0* x Q"
t=
max x."" y x Q Q²1,"" yt*² (T).
(33)
y
Now fix an i and let t; be time index at the beginning of episode i. Let E, = 1 indicate the event that episode has not ended time t. Then
T
E
v x Q Q2 vi
t=
1[E = 11 (v - x yf - -V
E
1=4
1 E[E 1] V x, 2 G G - 1 1 1]V
E
1=4
8
E
1[E 1)x G yt
T441
Egup
[V:1]
E
xt
G v
t=
Combining this with Eq. (33), we get
E
Σ
(Hepisodes in T Eemp [V.7] o(T)
(1-7)
Hence the one-step average reward least - 71 [V]]. ). symmetric analysis shows that is also most (1 - --a)) [V.]. This shows that both players have regret compared to the game value. Notice that this is only a loose implication of the path convergence guarantee because of the loose second inequality in Eq. (33).
Remark on the notion of "astt-tteaate convergence" general Markoy games While Eq. (31) corresponds to best-iterate convergence for general Markov games, an stronger notion one can
42

---

pursue after is "last-iterate convergence." As argued above, it is impossible to require that the policies on all states converge equilibrium. To address this issue, we propose study this problem under the episodic setting described above, in which the state is reset after every trajectory whose I-7' this case, last-iterate convergence will defined the convergence of the following quantity to zero when + OC:
expected length is -_
E Essp
max J.y 22,01/ V2.ue
V75
where we recall that i the episode index and (Itz Vt.)) are the policies used by the two players at the beginning of episode While last-iterate convergence seems reasonable and possibly achievable, we are unaware such results even for the degenerated case of single-player MDPs the standard regret bound corresponds to best-iterate convergence, while the techniques we are aware of to prove last-iterate convergence in MDPs require additional assumptions on the dynamics.
43


Reviews:
[{"review_id": "8Qkgl4nHej", "reviewer": "Reviewer_qUKM", "summary": "This paper studies algorithms for two-player zero-sum Markov games that are uncoupled, convergent, and rational. Previous attempts at designing such algorithms failed short in one aspect or the other. This work uses recent advances in entropy-based regularization to design new algorithms that overcome the inherent challenges. The other main contribution of this work is deriving the convergent rates of the proposed algorithms in Matrix, Markov, and general Markov games. \n", "questions": "1. How tight are the obtained convergence rates?\n2. My understanding of the proposed algorithm is that at time step $t,$ player 1 plays action $a _t$ and player 2 action $b_t.$ Thereafter, the local update steps are carried out. If this is correct, then why do the authors claim that their proposed algorithm does not require the two players to have synchronized policy updates? \n\n\n", "limitations": "The authors do address the limitations in their work adequately.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed algorithms do not require the two players to exchange information (including that related to entropy)\n2. The proposed ideas extend to general Markov games without any assumptions on the dynamics.\n3. The work seems technically sound. The algorithms follow easy-to-implement modifications of existing methods.\n4. The problem setup, the algorithms, and the proofs are written in an easy-to-digest format.\n\n\n\n", "weaknesses": "See Questions. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments. We address your questions below. \n\n*Q1:How tight are the obtained convergence rates?*\n\nA: The obtained convergence rates in this paper may not be tight. For example, Algorithm 1 achieves a high-probability $O(t^{-1/8})$ last-iterate convergence rate under bandit feedback for matrix games, while the known best lower bound of convergence rate in this setting is $\\Omega(t^{-1/2})$. However, we remark that no  last-iterate/path convergence *rates* are known in the considered setting (i.e., uncoupled algorithm with bandit feedback) prior to our work. Improving the convergence rates and establishing matching lower bounds are very interesting future directions.\n\n*Q2: My understanding of the proposed algorithm is that at time step $t$ player 1 plays action $a_t$ and player 2 action $b_t$.\nThereafter, the local update steps are carried out. If this is correct, then why do the authors claim that their proposed algorithm does not require the two players to have synchronized policy updates?*\n\nA: Thank you for pointing this out. Our interaction follows the standard model in the literature, where both players simultaneously choose an action in each round. This is not what we meant by ``synchronized policy updates.'' By \"synchronized policy updates,\" we refer to the method in [1] where each player is directed to use a fixed policy when interacting with others for an extended period. Afterward, all players update their policies simultaneously. We find such coordination between the players unnatural, and the subsequent algorithm no longer maintains the no-regret guarantee against adversaries. This drawback is also pointed out by [2] (see their second bullet point on Page 3). In contrast, the algorithm in our paper uses a very simple policy update rule and is robust (no-regret) even against an adversary. We will add the above explanation to the revised paper.  \n\n**References：**\n\n[1] Wei, C. Y., Lee, C. W., Zhang, M., & Luo, H. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games. COLT, 2021 \n\n[2]  Sayin, M., Zhang, K., Leslie, D., Basar, T., & Ozdaglar, A. Decentralized Q-learning in zero-sum Markov games. NeurIPS, 2021"}, {"review_id": "Ff0P1sFpqy", "reviewer": "Reviewer_Zo84", "summary": "This paper studies the problem of learning a Nash equilibrium in two-player zero-sum Markov games with bandit feedback.\nThe proposed algorithm introduces the entropy regularization technique into online mirror descent.\nFirst, the author proves that the proposed algorithm converges to an equilibrium in two-player normal-form games.\nThen, the last-iterate convergence rate for irreducible Markov games is provided.\nFinally, the paper presents a path convergence rate for Markov games without the irreducible assumption.", "questions": "* How much computational cost will it take to update the strategy (e.g., line 6 in Algorithm 1)?\n* Does the path convergence imply the average-iterate convergence?\n* What prior knowledge of the game is required for setting of $\\epsilon,\\beta$, and $\\eta$ in Theorem 3?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The problem is well-motivated. In many scenarios, the last-iterate convergence property under bandit feedback is more suitable than the average-iterate convergence property.\n* It seems novel to derive the last-iterate convergence rates under bandit feedback.\n* The proof sketch of Theorem 1 is intuitive and easy to follow.", "weaknesses": "* Since action probabilities of strategies in $\\Omega_t$ are lower bounded by $\\frac{1}{At^2}$, line 6 in Algorithm 1 seems to have no closed-form solution. I wonder how much computational cost it will take to update the strategy.\n* The term of $\\ln^{20}(SAT/\\delta)$ in Theorem 3 depends heavily on $T$. This term would be dominant practically and would not be able to be ignored.\n* I could not completely follow the sketch of Theorem 3. I would have appreciated more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive comments. We will include a more detailed proof sketch of Theorem 3 in the revised version. Your questions are addressed below. \n\n*Q1:How much computational cost will it take to update the strategy (e.g., line 6 in Algorithm 1)?*\n\nA: To update the strategy, the main computation happens in line 6 where we need to solve the following optimization problem over $\\Omega_t=\\{x \\in \\Delta^{|A|}: x[i] \\ge \\frac{1}{|A|t^2}\\}$: $x_{t+1} = \\arg \\min_{x \\in \\Omega_t} (g_t^\\top x + \\frac{1}{\\eta} D(x, x_t))$. By the KKT condition, we have $$x_{t+1}[i] =  \\frac{x_t[i] \\cdot \\exp(-\\eta_t(g_t[i] + \\lambda_i))}{\\sum_{i'\\in[|A|]} x_t[i'] \\cdot \\exp(-\\eta_t(g_t[i'] + \\lambda_{i'}))},$$ where $\\lambda_i = 0$ if $x_{t+1}[i] > \\frac{1}{|A|t^2}$ and $\\lambda_i\\le 0$ if $x_{t+1}[i] = \\frac{1}{|A|t^2}$. The computation of feasible $(x_{t+1}[i], \\lambda_i)$ can be done in $O(|A|\\log |A|)$ time as explained below. \n\nFor simplicity, let us denote $h[i] :=  x_t[i]\\cdot\\exp(-\\eta_t g_t[i])$. The algorithm works as follows: \n1. sort $i$'s based on the value of $h[i]$ in increasing order and let us assume w.o.l.g. that $h[1] \\le h[2] \\ldots \\le h[|A|]$; \n2. find $j \\in [|A|]$ such that $\\frac{j\\cdot h[j]}{j\\cdot h[j] + \\sum_{i=j+1}^{|A|} h[i]} \\le \\frac{j}{|A|t^2}$ and $\\frac{(j+1)\\cdot h[j+1]}{(j+1)\\cdot h[j+1] + \\sum_{i=j+2}^{|A|} h[i]} > \\frac{j+1}{|A|t^2}$; if such $j$ does not exist, then we can set $\\lambda_i = 0$ for all $i$ and compute $x_{t+1}$. \n3. compute $\\theta$ such that $\\frac{j\\cdot \\theta}{j\\cdot \\theta + \\sum_{i=j+1}^{|A|} h[i]} = \\frac{j}{|A|t^2}$ and set $x_{t+1}[i] = \\frac{1}{|A|t^2}$ for $1 \\le i \\le j$ and $x_{t+1}[i] = \\frac{h[i]}{j\\cdot \\theta + \\sum_{i=j+1}^{|A|} h[i]}$ for $j+1 \\le i \\le |A|$. \n\nSorting in step 1 can be done in $O(|A| \\log |A|)$ time and the computation in other steps can be done in time $O(|A|)$. \n\n*Q2: Does the path convergence imply the average-iterate convergence?*\n\nA: Path convergence does not imply averge-iterate convergence. Path convergence concerns visited states $\\{s_t\\}$ only and the policies on states that are not reached in learning can be arbitrary, so it does not imply average-iterate convergence. Similarly, path convergence does not imply best-iterate convergence. However, we would like to remark that path convergence excludes the possibility of cycling and has many interesting game-theoretical implications (see our discussion in section 6.1 and Appendix F).\n\n\n*Q3: What prior knowledge of the game is required for setting of $\\eta$, $\\epsilon$, and $\\beta$ in Theorem 3?*\n\nA: The prior knowledge of the game needed for setting $\\eta$, $\\epsilon$, and $\\beta$ is the number of states $S$, the number of actions $A$, and the discounting factor $\\gamma$. See Appendix G (page 39) for the concrete choice of these hyper parameters, and we plan to include it in the main body in the revised version. "}, {"review_id": "Zn37HVmpkA", "reviewer": "Reviewer_dxZp", "summary": "The paper studies the last iteration convergence of uncoupled learning in two player zero-sum markov game, with bandit feedback. The paper provide the first finite last-iterate convergence guarantee under the bandit feedback.\n\nThe paper studies the problem of two-player zero-sum Markov game and designs a new uncoupled learning algorithm that enjoys last iteration convergence, under bandit feedback. Along the way, the paper derive a couple of results, summarized below:\n\n(1) Even for the standard matrix game (where there is no Markov transition), the paper provides an algorithm with $O(T^{-1/8})$ convergence rate\n\n(2) For irreducible Markov game, the paper provides an algorithm with $O(T^{-1/9})$ convergence rate\n\n(3) For general irreducible Markov game, the paper provides an algorithm with $O(T^{-1/10})$ convergence rate, but only under the notion of path convergence.\n\nIt is worth noting that there is a large body of literature, but previous work [WLZL21] are either not uncoupled, or [BJY'20 ] not last iterate convergent.\n\nThe technical part seems novel, from a high level, the results are obtained by adding an entropy regularizer, but there are many subtle details to make it really work and the analysis is involved.\n\n------------------\nI have read the author response and want to keep my positive evaluation.\n\n", "questions": "I have a few minor questions:\n\n(1) Can you provide some intuition on the number of $T^{-1/8}$ or $T^{-1/9}$, ideally, it would be nice if one can written down a short explanation on how these magic exponent comes from.\n\n(2) The following paper seems very relevant?\n\n[1] Regret Minimization and Convergence to Equilibria in General-sum Markov Games", "limitations": "no.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper studies a fairly popular topic and provide the first finite last-iterate convergence guarantee for Markov game (two-player, zero-sum). The technical contribution seems to be novel. The paper is also well-written and the technical part is well explained.", "weaknesses": "No.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very positive comments! We address your questions below.\n\n*Q1: Can you provide some intuition on the number of $T^{-1/8}$ or $T^{-1/9}$, ideally, it would be nice if one can written down a short explanation on how these magic exponent comes from.*\n\nA: These exponents are a result of the parameters $k_\\eta, k_\\beta, k_\\epsilon$ (with $\\eta_t = t^{-k_\\eta}$, $\\beta_t = t^{-k_\\beta}$, $\\epsilon_t = t^{-k_\\epsilon}$) we choose to optimize the convergence rate. \nFor example, the analysis of Algorithm 1 (see Appendix B) shows that the last-iterate convergence rate  (ignoring log factors and other dependence) is $O(t^{\\frac{-k_\\eta + k_\\epsilon}{4}} + t^{\\frac{k_\\beta}{2} - k_\\eta} + t^{\\frac{-k_\\beta+k_\\epsilon}{2}} + t^{\\frac{-k_\\eta +k_\\beta}{2}} + t^{\\frac{-1+k_\\eta +k_\\epsilon}{2}} + t^{-k_\\epsilon})$. The choice of $k_\\eta = \\frac{5}{8}, k_\\beta = \\frac{3}{8}, k_\\epsilon = \\frac{1}{8}$ gives us the optimized $O(t^{-\\frac{1}{8}})$ rate. \n \n*Q2: The following paper seems very relevant? [1] Regret Minimization and Convergence to Equilibria in General-sum Markov Games*\n\nA: Thank you for pointing this out! We will add discussion about this paper in the revised version of our paper. Erez et al. (2022) [1] studies regret minimization in general-sum Markov games and provide algorithm with sublinear regret under self-play and *average-iterate* convergence to equibria, while our work focuses on *last-iterate* convergence rates to Nash equilibrium.   "}, {"review_id": "CR8uO3ARKb", "reviewer": "Reviewer_EDuK", "summary": "The paper introduces a new algorithm for learning in two-player zero-sum Markov games based on prior work that  is uncoupled (agents only need their own reward as feedback), convergent (to NE) and rational. The result is an algorithm that is similar in concept to a single agent RL algorithm, but guarantees convergence if both players use the same algorithm. For their initial algorithm, there is an additional assumption of irreducibility on the Markov games, and it achieves last iterate convergence to the Nash with rate O(t^{-1/9+eps}). Meanwhile, for general 2 player zero sum Markov games without additional assumptions, a modification of the algorithm that applies optimism achieves O(t^{-1/10}) path convergence rate.", "questions": "Have any experiments been run using Algorithms 1, 2 and 3 to compare them to current SOTA methods? If so, how does the empirical performance compare? It seems that your algorithms would scale well with larger games since players only use local information, how does this compare to existing methods?  ", "limitations": "The authors have adequately addressed the paper's limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The algorithm proposed takes inspiration from prior works and makes useful modifications that give practical and theoretical advantages. The structure of the results and explanation of analysis also flows well and is easy to read despite the technical complexity. The authors are also very clear in their motivation and I find the literature review to be quite comprehensive.", "weaknesses": "The paper does a good job of setting up motivation and prior ideas, but in my opinion the key takeaway from this work is that there is now a truly uncoupled and convergent algorithm for general Markov games, which is in my opinion the paper's most significant contribution. However, there is too much space dedicated to matrix and irreducible Markov games, when in my view, the section on general Markov games should be greatly expanded. In addition to the overview of the analysis, I am interested to know how optimism can be leveraged in this type of algorithm, and the thought process behind the modifications to Algorithm 3 compared to Algorithm 2. As a suggestion, perhaps part of the analysis overview for Thm 3 can be repurposed and moved forward to add to the explanation about Alg 3, which would improve the flow of the section in my opinion. Finally, there is a lack of experimental results which would make help frame the results better in comparison to prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive feedback! We will add more discussions on the optimism technique in Algorithm 3 in the revised version. We address your question below.\n\n*Q: Have any experiments been run using Algorithms 1, 2 and 3 to compare them to current SOTA methods? If so, how does the empirical performance compare? It seems that your algorithms would scale well with larger games since players only use local information, how does this compare to existing methods?* \n\nA: To our knowledge, our work presents the first provable algorithms that are uncouple, convergent, and have finite rate under bandit feedback. Therefore, there is no comparable previous state of the art in our setting. However, if we only care about computing Nash equilibrium under bandit feedback (without caring about uncoupledness and convergence), there are a few algorithms that have faster rates than our work. For example, the V-Learning algorithm [1] has a faster $O(t^{-1/2})$ *average-iterate* convergence rate but has no last-iterate/path convergence guarantee.  We do think that it is a very interesting future direction to evaluate the empirical performance of our algorithms in applications such as Game AI [2]. \n\n[1] Jin C, Liu Q, Wang Y, et al. V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent RL[J]. arXiv preprint arXiv:2110.14555, 2021\n\n[2] Perolat J, De Vylder B, Hennes D, et al. Mastering the game of Stratego with model-free multiagent reinforcement learning[J]. Science, 2022, 378(6623): 990-996."}, {"review_id": "r8xkZJaodz", "reviewer": "Reviewer_UUTi", "summary": "The paper studies the problem of designing uncoupled learning dynamics that provably converge to Nash equilibria in two-player zero-sum Markov games. As a preliminary result, the paper introduces the first dynamics that converge last iterate in self play in matrix games with bandit feedback. Then, the paper shows how to extend such last-iterate-convergent learning dynamics to the case of irreducible Markov games. Finally, the paper studies the case of general Markov games, proving that a variation of the previously-introduced learning dynamics achieve convergence according to a newly introduced definition of convergence (called path convergence).", "questions": "Please resolve my concerns in the weaknesses part.", "limitations": "None.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem studied in the paper is interesting and it has received considerable attention over the last years. The paper is well written and easy to follow.", "weaknesses": "A major weakness that I see is that the results presented in the paper seem minor adaptations and different analysis of already known techniques/tools. The authors should focus more on discussing the novelty of their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We address your concerns below.\n\n*Q: A major weakness that I see is that the results presented in the paper seem minor adaptations and different analysis of already known techniques/tools. The authors should focus more on discussing the novelty of their approach.*\n\nA : Although our algorithms share similarity with previous works that also use entropy regularization, we believe that both the design and the analysis of the algorithms are novel and non-trivial. To the best of our knowledge, all previous entropy regularized two-player zero-sum Markov game algorithms are coupled (e.g., [1,2,3]), while ours is the first that achieves uncoupledness under entropy regularization. We will further discuss this by comparing our algorithms to those in [3], highlighting the new technical challenges we encounter.\n\nThe entropy-regularized OMWU algorithm in [3] is tailored to the full-information setting and their value function updates require both players' entropy information: $ V^{t+1}(s)=(1 - \\alpha_{t+1})V^t(s) +\\alpha_{t+1} ( x^{t+1}(s)^\\top Q^{t+1}(s)y^{t+1}(s) + \\tau  \\phi(x^{t+1}(s)) - \\tau \\phi(y^{t+1}(s))$. This necessitates both players to know the entropy value of the other player's policy, which is unnatural. Indeed, the authors explicitly present the removal of this information sharing as an open question: *[the introduction of entropy regularization requires each agent to reveal the entropy of their current policy to each other, which prevents the proposed method from being fully decentralized. Can we bypass this by dropping the entropy information in value learning? We leave the answers to future work.]*\n\nWe answer this open question affirmatively by giving a fully decentralized algorithm for zero-sum Markov games with provable last-iterate convergence rates. In Algorithm 2, the update of the value function $V$ is simple and does not require any entropy information: $V_{t+1}^{s_t} \\leftarrow (1-\\alpha_\\tau)V_t^{s_t} + \\alpha_\\tau \\left(\\sigma_t + \\gamma V_t^{s_{t+1}}\\right)$. This modification results in a discrepancy between the policy update and the value update. While the policy now incorporates a regularization term, the value function does not. Such a mismatch is unprecedented in earlier studies and necessitates a non-trivial approach to resolve. \nAdditionally, Algorithm 2 operates on bandit feedback instead of full-information feedback, presenting further technical challenges\n\n[1] Cen, Shicong, Yuting Wei, and Yuejie Chi. \"Fast policy extragradient methods for competitive games with entropy regularization.\" NeurIPS 2021. \n\n[2] Ziyi Chen, Shaocong Ma, and Yi Zhou. Sample efficient stochastic policy extragradient algorithm for zero-sum\nmarkov game. ICLR 2021.\n\n[3] Cen, Shicong, Yuejie Chi, Simon S. Du, and Lin Xiao. \"Faster last-iterate convergence of policy optimization in zero-sum Markov games.\" ICLR 2023. "}]

Author rebuttal:

Meta-review: As its core contribution, the paper provides uncoupled learning algorithms for  zero-sum matrix games and irreducible Markov games with provable last-iterate convergence rates under bandit feedback. All reviewers are positive about the theoretical contributions. Notably though, there remains a wide gap between the last-iterate upper bound and best known lower bound in these settings. It is suggested that the authors could add discussions the limitation of their results and give some insights on why such a gap exists and possibly how to bridge them.
Decision: Accept (poster)