SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we define  $v_{t} = \mathrm{KL}(z_{t + 1}^{\star}, z_{t + 1}) - \mathrm{KL}(z_{t}^{\star}, z_{t + 1})$  (see Appendix B for definitions of  $\lambda_t, \xi_t, \zeta_t$ ) The instability penalty comes from some local-norm of the gradient estimator  $g_{t}$ . The estimation error comes from the bias between the gradient estimator  $g_{t}$  and the real gradient  $G y_{t}$ . We pay the last term  $v_{t}$  since the Nash equilibrium  $z_{t}^{*}$  of the regularized game  $f_{t}$  is changing over time.
Step 2: Strategy Convergence to NE of the Regularized Game Expanding the above recursion up to  $t_0$ , we get
$$
\mathrm {K L} \left(z _ {t + 1} ^ {\star}, z _ {t + 1}\right) \leq \mathcal {O} \left(\underbrace {\sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} ^ {2}} _ {\text {t e r m} _ {1}} + \underbrace {2 A \sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} ^ {2} \lambda_ {i}} _ {\text {t e r m} _ {2}} + \underbrace {\sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} \xi_ {i}} _ {\text {t e r m} _ {3}} + \underbrace {\sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} \zeta_ {i}} _ {\text {t e r m} _ {4}} + \underbrace {\sum_ {i = 1} ^ {t} w _ {t} ^ {i} v _ {i}} _ {\text {t e r m} _ {5}}\right), \tag {1}
$$
where  $w_{t}^{i}\triangleq \prod_{j = i + 1}^{t}(1 - \eta_{j}\epsilon_{j})$ . To upper bound  $\mathbf{term}_1$ -term4, we apply careful sequence analysis (Appendix A.1) and properties of the EXP3-IX algorithm with changing step size (Appendix A.2). The analysis of  $\mathbf{term}_5$  uses Lemma 13, which states  $v_{t} = \mathrm{KL}(z_{t + 1}^{\star},z_{t + 1}) - \mathrm{KL}(z_{t}^{\star},z_{t + 1})\leq \mathcal{O}(\ln (At)\| z_{t + 1}^{\star} - z_{t}^{\star}\|_{1}) = \mathcal{O}(\frac{\ln^{2}(At)}{t})$  and is slightly involved as  $\Omega_t$  and  $\epsilon_t$  are both changing. With these steps, we conclude that with probability at least  $1 - \mathcal{O}(\delta)$ ,  $\mathrm{KL}(z_t^\star ,z_t) = \mathcal{O}\left(A\ln^3 (At / \delta)t^{-\frac{1}{4}}\right)$ .
5 Irreducible Markov Games
We now extend our results on matrix games to two-player zero-sum Markov games. Similarly to many previous works, our first result makes the assumption that the Markov game is irreducible with bounded travel time between any pair of states. The assumption is formally stated below:
Assumption 1 (Irreducible Game). We assume that under any pair of stationary policies of the two players, and any pair of states  $s$ ,  $s'$ , the expected time to reach  $s'$  from  $s$  is upper bounded by  $L$ .
We propose Algorithm 2 for uncoupled learning in irreducible two-player zero-sum games, which is closely related to the Nash-V algorithm by [BJY20], but with additional entropy regularization. It can also be seen as players using Algorithm 1 on each state  $s$  to update the policies  $(x_{t}^{s},y_{t}^{s})$  whenever state  $s$  is visited, but with  $\sigma_t + \gamma V_t^{s_t + 1}$  as the observed loss to construct loss estimators. Importantly,  $V_{1}^{s},V_{2}^{s},\ldots$  is a slowly changing sequence of value estimations that ensures stable policy updates [BJY20, WLZL21a,  $\mathrm{SZL}^{+}21]$ . Note that in Algorithm 2, the updates of  $V_{t}^{s}$  only use players' local information (Line 8).
Algorithm 2 Irreducible Markov Game
1: Define:  $\eta_t = (1 - \gamma)t^{-k_\eta}$ ,  $\beta_t = t^{-k_\beta}$ ,  $\epsilon_t = \frac{1}{1 - \gamma}t^{-k_\epsilon}$ ,  $\alpha_t = t^{-k_\alpha}$  with  $k_\alpha, k_\epsilon, k_\beta, k_\eta \in (0,1)$ ,
2: Initialization:  $x_{1}^{s} \gets \frac{1}{A}\mathbf{1}$ ,  $n_{1}^{s} \gets 0$ ,  $V_{1}^{s} \gets \frac{1}{2(1 - \gamma)}$ ,  $\forall s$ .
3: for  $t = 1,2,\ldots ,\mathbf{d}\mathbf{o}$
4:  $\tau = n_{t + 1}^{s_t}\gets n_t^{s_t} + 1$  (the number of visits to state  $s_t$  up to time  $t$ ).
5: Draw  $a_{t} \sim x_{t}^{s_{t}}$ , observe  $\sigma_{t} \in [0,1]$  with  $\mathbb{E}\left[\sigma_{t}\right] = G_{a_{t},b_{t}}^{s_{t}}$ , and observe  $s_{t+1} \sim P_{a_{t},b_{t}}^{s_{t}}(\cdot)$ .
6: Compute  $g_{t}$  where  $g_{t,a} = \frac{\mathbf{1}[a_{t} = a](\sigma_{t} + \gamma V_{t}^{s_{t} + 1})}{x_{t,a}^{s_{t}} + \beta_{\tau}} + \epsilon_{\tau}\ln x_{t,a}^{s_{t}}$ ,  $\forall a\in \mathcal{A}$ .
7: Update  $x_{t+1}^{s_t} \gets \operatorname{argmin}_{x \in \Omega_{\tau+1}} \left\{ x^\top g_t + \frac{1}{\eta_\tau} \mathrm{KL}(x, x_t^{s_t}) \right\}$ .
8: Update  $V_{t + 1}^{s_t}\gets (1 - \alpha_\tau)V_t^{s_t} + \alpha_\tau (\sigma_t + \gamma V_t^{s_{t + 1}})$
9: For all  $s \neq s_t$ ,  $x_{t+1}^s \gets x_t^s$ ,  $n_{t+1}^s \gets n_t^s$ ,  $V_{t+1}^s \gets V_t^s$ .
Comparison to Previous Works Although Algorithm 2 shares similarity with previous works that also use entropy regularization, we believe that both the design and the analysis of our algorithm are novel and non-trivial. To the best of our knowledge, all previous entropy regularized two-player zero-sum Markov game algorithms are coupled (e.g., [CWC21, CMZ21, CCDX23]), while ours is the first that achieves uncoupledness under entropy regularization. We further discuss this by comparing our algorithm to those in [CCDX23], highlighting the new technical challenges we encounter.
The entropy-regularized OMWU algorithm in [CCDX23] is tailored to the full-information setting. Moreover, in the value function update step both players need to know the entropy value of the other player's policy, which is unnatural. Indeed, the authors explicitly present the removal of this information sharing as an open question. We answer this open question affirmatively by giving a fully decentralized algorithm for zero-sum Markov games with provable last-iterate convergence rates. In Algorithm 2 (Line 8), the update of the value function  $V$  is simple and does not require any entropy information:  $V_{t + 1}^{s_t} \gets (1 - \alpha_\tau)V_t^{s_t} + \alpha_\tau (\sigma_t + \gamma V_t^{s_t + 1})$ . This modification results in a discrepancy between the policy update and the value update. While the policy now incorporates a regularization term, the value function does not. Such a mismatch is unprecedented in earlier studies and necessitates a non-trivial approach to resolve. Additionally, Algorithm 2 operates on bandit feedback instead of full-information feedback, presenting further technical challenges.
Algorithm 2 also offers improvement over the uncoupled algorithm of [WLZL21a]. The algorithm of [WLZL21a] requires coordinated policy update where the players interact with each other using the current policy for several iterations to get an approximately accurate gradient (the number of iterations required depends on  $L$  as defined in Assumption 1), and then simultaneously update the policy pair on all states. We do not require such unnatural coordination between the players or prior knowledge on  $L$ .
Our main result is the following theorem on the last-iterate convergence rate of Algorithm 2.
Theorem 2 (Last-Iterate Convergence Rate). For any  $\varepsilon, \delta > 0$ , Algorithm 2 with  $k_{\alpha} = \frac{9}{9 + \varepsilon}$ ,  $k_{\epsilon} = \frac{1}{9 + \varepsilon}$ ,  $k_{\beta} = \frac{3}{9 + \varepsilon}$ , and  $k_{\eta} = \frac{5}{9 + \varepsilon}$  guarantees, with probability at least  $1 - \mathcal{O}(\delta)$ , for any time  $t \geq 1$ ,
$$
\max  _ {s, x, y} \left(V _ {x _ {t}, y} ^ {s} - V _ {x, y _ {t}} ^ {s}\right) \leq \mathcal {O} \Big (\frac {A L ^ {2 + 1 / \varepsilon} \ln^ {4 + 1 / \varepsilon} (S A t / \delta) \ln^ {1 / \varepsilon} (t / (1 - \gamma))}{(1 - \gamma) ^ {2 + 1 / \varepsilon}} \cdot t ^ {- \frac {1}{9 + \varepsilon}} \Big).
$$
5.1 Analysis Overview
We introduce some notations for simplicity. We denote by  $\mathbb{E}_{s' \sim P^s}[V_t^{s'}]$  the  $A \times A$  matrix such that  $(\mathbb{E}_{s' \sim P^s}[V_t^{s'}])_{a,b} = \mathbb{E}_{s' \sim P_{a,b}^s}[V_t^{s'}]$ . Let  $t_\tau(s)$  be the  $\tau$ -th time the players visit state  $s$ , and define  $\hat{x}_\tau^s = x_{t_\tau(s)}^s$  and  $\hat{y}_\tau^s = y_{t_\tau(s)}^s$ . Then, define the regularized game for each state  $s$  via the loss function  $f_\tau^s(x,y) = x^\top (G^s + \gamma \mathbb{E}_{s' \sim P^s}[V_{t_\tau(s)}^{s'}])y - \epsilon_\tau \phi(x) + \epsilon_\tau \phi(y)$ . Furthermore, let  $\hat{z}_{\tau\star}^s = (\hat{x}_{\tau\star}^s, \hat{y}_{\tau\star}^s)$  be the equilibrium of  $f_\tau^s(x,y)$  over  $\Omega_\tau \times \Omega_\tau$ . In the following analysis, we fix some  $t \geq 1$ .
Step 1: Policy Convergence to NE of Regularized Game Using similar techniques to Step 1 and Step 2 in the analysis of Algorithm 1, we can upper bound  $\mathrm{KL}(\hat{z}_{\tau +1\star}^{s},\hat{z}_{\tau +1}^{s})$  like Eq. (1) with similar subsequent analysis for term1-term4. The analysis for term5 where  $v_{i}^{s} = \mathrm{KL}(\hat{z}_{i + 1\star}^{s},\hat{z}_{i + 1}^{s}) - \mathrm{KL}(\hat{z}_{i\star}^{s},\hat{z}_{i + 1}^{s})$  is more challenging compared to the matrix game case since here  $V_{t_i(s)}^s$  is changing between two visits to state  $s$ . To handle this term, we leverage the following facts for any  $s'$ : (1) the irreducibility assumption ensures that  $t_{i + 1}(s) - t_i(s)\leq \mathcal{O}\bigl (L\ln (St / \delta)\bigr)$  thus the number of updates of the value function at state  $s^\prime$  is bounded; (2) until time  $t_i(s)\geq i$ , state  $s^\prime$  has been visited at least  $\Omega (\frac{i}{L\ln(St / \delta)})$  times thus each change of the value function between  $t_i(s)$  and  $t_{i + 1}(s)$  is at most  $\mathcal{O}((\frac{i}{L\ln(St / \delta)})^{-k_\alpha})$ . With these arguments, we can bound term5 by  $\mathcal{O}\left(\ln^4 (SAT / \delta)L\tau^{-k_\alpha +k_\eta +2k_\epsilon}\right)$ . Overall, we have the following policy convergence of NE of the regularized game (Lemma 17):  $\mathrm{KL}(\hat{z}_{\tau \star}^{s},\hat{z}_{\tau}^{s})\leq \mathcal{O}\left(A\ln^4 (SAT / \delta)L\tau^{-k_\sharp}\right)$ , where  $k_{\sharp} = \min \{k_{\beta} - k_{\epsilon},k_{\eta} - k_{\beta},k_{\alpha} - k_{\eta} - 2k_{\epsilon}\}$ .
Step 2: Value Convergence Unlike matrix games, policy convergence to NE of the regularized game is not enough for convergence in duality gap. We also need to bound  $|V_{t}^{s} - V_{\star}^{s}|$  since the regularized game is defined using  $V_{t}^{s}$ , the value function maintained by the algorithm, instead of the minimax game value  $V_{\star}^{s}$ . We use the following weighted regret quantities as a proxy:  $\mathrm{Reg}_{\tau}^{s} \triangleq \max_{x,y} \left( \sum_{i=1}^{\tau} \alpha_{\tau}^{i} \left( f_{i}^{s}(\hat{x}_{i}^{s}, \hat{y}_{i}^{s}) - f_{i}^{s}(x^{s}, \hat{y}_{i}^{s}) \right), \sum_{i=1}^{\tau} \alpha_{\tau}^{i} \left( f_{i}^{s}(\hat{x}_{i}^{s}, y_{i}^{s}) - f_{i}^{s}(\hat{x}_{i}^{s}, \hat{y}_{i}^{s}) \right) \right)$ , where  $\alpha_{\tau}^{i} = \alpha_{i} \prod_{j=i+1}^{\tau}(1 - \alpha_{j})$ . We can upper bound the weighted regret  $\mathrm{Reg}_{\tau}^{s}$  using a similar analysis as in Step 1 (Lemma 19). We then show a contraction for  $|V_{t_{\tau}(s)}^{s} - V_{\star}^{s}|$  with the weighted regret quantities:  $|V_{t_{\tau}(s)}^{s} - V_{\star}^{s}| \leq \gamma \sum_{i=1}^{\tau} \alpha_{\tau}^{i} \max_{s'} |V_{t_{i}(s)}^{s'} - V_{\star}^{s'}| + \tilde{\mathcal{O}}(\epsilon_{\tau} + \mathrm{Reg}_{\tau}^{s})$ . This leads to the following convergence of  $V_{t}^{s}$  (Lemma 20):  $|V_{t}^{s} - V_{\star}^{s}| \leq \tilde{\mathcal{O}}(t^{-k_{*}})$ , where  $k_{*} = \min \{k_{\eta}, k_{\beta}, k_{\alpha} - k_{\beta}, k_{\epsilon}\}$ .
Obtaining Last-Iterate Convergence Rate Fix any  $t$  and let  $\tau$  be the number of visits to  $s$  before time  $t$ . So far we have shown (1) policy convergence of  $\mathrm{KL}(\hat{z}_{\tau \star}^s, \hat{z}_\tau^s)$  in the regularized game; (2) and value convergence of  $|V_t^s - V_\star^s|$ . Using the fact that the regularized game is at most  $\mathcal{O}(\epsilon_\tau + |V_t^s - V_\star^s|)$  away from the minimax game matrix  $Q^\star$  and appropriate choices of parameters proves Theorem 2.
6 General Markov Games
In this section, we consider general two-player zero-sum Markov games without Assumption 1. We propose Algorithm 3, an uncoupled learning algorithm that handles exploration and has path convergence rate. Compared to Algorithm 2, the update of value function in Algorithm 3 uses a bonus term  $\mathtt{bns}_{\tau}$  based on the optimism principle to handle exploration.
Theorem 3 below implies that we can achieve  $\frac{1}{t}\sum_{\tau = 1}^{t}\max_{x,y}(x_{\tau}^{s_{\tau}}Q_{\star}^{s_{\tau}}y^{s_{\tau}} - x_{\tau}^{s_{\tau}}Q_{\star}^{s_{\tau}}y_{\tau}^{s_{\tau}}) = \mathcal{O}(t^{-\frac{1}{10}})$  path convergence rate if we use the doubling trick to tune down  $u$  at a rate of  $t^{-\frac{1}{10}}$ .
Theorem 3. For any  $u \in \left[0, \frac{1}{1 - \gamma}\right]$  and  $T \geq 1$ , there exists a proper choice of parameters  $\epsilon, \beta, \eta$  such that Algorithm 3 guarantees with probability at least  $1 - \mathcal{O}(\delta)$ ,
$$
\sum_ {t = 1} ^ {T} \mathbf {1} \left[ \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) > u \right] \leq \mathcal {O} \left(\frac {S ^ {2} A ^ {3} \ln^ {2 0} (S A T / \delta)}{u ^ {9} (1 - \gamma) ^ {1 6}}\right). \tag {2}
$$
Algorithm 3 General Markov Game
1: Input:  $\eta \leq \beta \leq \epsilon$  and  $T$ .
2: Define:  $\Omega = \{x\in \Delta_{\mathcal{A}}:x_a\geq \frac{1}{AT},\forall a\in \mathcal{A}\}$ ,  $\alpha_{\tau} = \frac{H + 1}{H + \tau}$ , where  $H = \frac{\ln(T)}{1 - \gamma}$ .
$\mathsf{bn}\mathsf{s}_{\tau} = \kappa A\ln^{2}(SAT / \delta)(\beta +\eta^{-1}\alpha_{\tau}) / (1 - \gamma)^{2}$  for a sufficiently large absolute constant  $\kappa >0$
3: Initialization:  $\underline{V}_1^s, n_1^s \gets 0, x_1^s \gets \frac{1}{A} \mathbf{1}, \forall s$ .
4: for  $t = 1,2,\ldots ,\mathbf{d}\mathbf{o}$
5:  $\tau = n_{t + 1}^{s_t} \gets n_t^{s_t} + 1$ .
6: Sample  $a_{t}\sim x_{t}^{s_{t}}$  , observe  $\sigma_t\in [0,1]$  with  $\mathbb{E}[\sigma_t] = G_{a_t,b_t}^{s_t}$  , and observe  $s_{t + 1}\sim P_{a_t,b_t}^{s_t}(\cdot)$
7: Compute  $g_{t}$  where  $g_{t,a} = \frac{\mathbf{1}[a_{t} = a]\left(\sigma_{t} + \gamma V_{t}^{s_{t} + 1}\right)}{x_{t,a}^{s_{t}} + \beta} +\epsilon \ln x_{t,a}^{s_{t}},\forall a\in \mathcal{A}.$
8: Update  $x_{t+1}^{s_t} \gets \operatorname{argmin}_{x \in \Omega} \left\{ x^\top g_t + \frac{1}{\eta} \mathrm{KL}(x, x_t^{s_t}) \right\}$ .
9: Update  $V_{\tilde{t} +1}^{s_t}\gets (1 - \alpha_\tau)\tilde{V}_t^{s_t} + \alpha_\tau (\sigma_t + \gamma \underline{V}_t^{s_{t + 1}} - \mathsf{bns}_{\tau})$  and  $\underline{V}_{t + 1}^{s_t}\gets \max \left\{V_{t + 1}^{s_t},0\right\}$
10: For all  $s \neq s_t$ ,  $x_{t+1}^s \gets x_t^s$ ,  $\underline{V}_{t+1}^s \gets \underline{V}_t^s$ ,  $\underline{V}_{t+1}^s \gets \underline{V}_t^s$ ,  $n_{t+1}^s \gets n_t^s$ .
1: end for
6.1 Path Convergence
Path convergence has multiple meaningful game-theoretic implications. By definition, It implies that frequent visits to a state bring players' policies closer to equilibrium, leading to both players using near-equilibrium policies for all but  $o(T)$  number of steps over time.
Path convergence also implies that both players have no regret compared to the game value  $V_{\star}^{s}$ , which has been considered and motivated in previous works such as [BT02, TWYS20]. To see this, we apply the results to the episodic setting, where in every step, with probability  $1 - \gamma$ , the state is redrawn from  $s \sim \rho$  for some initial distribution  $\rho$ . If the learning dynamics enjoys path convergence, then  $\mathbb{E}[\sum_{t=1}^{T} x_{t}^{s_{t}} G^{s_{t}} y_{t}^{s_{t}}] = (1 - \gamma) \mathbb{E}_{s \sim \rho}[V_{\star}^{s}] T \pm o(T)$ . Hence the one-step average reward is  $(1 - \gamma) \mathbb{E}_{s \sim \rho}[V_{\star}^{s}]$  and both players have no regret compared to the game value. A more important implication of path convergence is that it guarantees stability of players' policies, while cycling behaviour is inevitable for any FTRL-type algorithms even in zero-sum matrix games [MPP18, BP18]. We defer the proof and more discussion of path convergence to Appendix F.
Finally, we remark that our algorithm is built upon Nash V-learning [BJY20], so it inherits properties of Nash V-learning, e.g., one can still output near-equilibrium policies through policy averaging [JLWY21], or having no regret compared to the game value when competing with an arbitrary opponent [TWYS20]. We demonstrate extra benefits brought by entropy regularization regarding the stability of the dynamics.
6.2 Analysis Overview of Theorem 3
For general Markov games, it no longer holds that every state  $s$  is visited often, and thus the analysis is much more challenging. We first define two regularized games based on  $\underline{V}_t^s$  and the corresponding quantity  $\overline{V}_t^s$  for the  $y$ -player. Define  $t_\tau(s)$ ,  $\hat{x}_\tau^s$ ,  $\hat{y}_\tau^s$  the same way as in the previous section. Then define  $\underline{f}_\tau^s(x,y) \triangleq x^\top(G^s + \gamma \mathbb{E}_{s' \sim P^s}[\underline{V}_{t_\tau(s)}^{s'}]) y - \epsilon \phi(x) + \epsilon \phi(y)$ ,  $\overline{f}_\tau^s(x,y) \triangleq x^\top(G^s + \gamma \mathbb{E}_{s' \sim P^s}[\overline{V}_{t_\tau(s)}^{s'}]) y - \epsilon \phi(x) + \epsilon \phi(y)$  and denote  $J_t = \max_{x,y} (x_t^{s_t^\top}(G^{s_t} + \gamma \mathbb{E}_{s' \sim P^s t}[\overline{V}_t^{s'}]) y^{s_t} - x_t^{s_t^\top}(G^s + \gamma \mathbb{E}_{s' \sim P^s t}[\underline{V}_t^{s'}]) y^{s_t})$ . We first bound the "path duality gap" as follows
$$
\max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s} - x ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s}\right) \leq J _ {t} + \mathcal {O} \left(\max  _ {s ^ {\prime}} \left(V _ {\star} ^ {s ^ {\prime}} - \bar {V} _ {t} ^ {s ^ {\prime}}, \underline {{V}} _ {t} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}}\right)\right). \tag {3}
$$
Value Convergence: Bounding  $\underline{V}_t^s - V_\star^s$  and  $V_\star^s - \overline{V}_t^s$ . This step is similar to Step 2 in the analysis of Algorithm 2. We first show an upper bound of the weighted regret (Lemma 23):  $\sum_{i=1}^{\tau} \alpha_\tau^i (\underline{f}_i^s(\hat{x}_i^s, \hat{y}_i^s) - \underline{f}_i^s(x^s, \hat{y}_i^s)) \leq \frac{1}{2} \mathsf{bns}_\tau$ , where  $\alpha_\tau^i = \alpha_i \prod_{j=i+1}^\tau (1 - \alpha_j)$ . Note that the value function  $\underline{V}_t^s$  is updated using  $\sigma_t + \gamma \underline{V}_t^{s_t+1} - \mathsf{bns}_\tau$ . Thus when relating  $|\underline{V}_t^s - V_\star^s|$  to the regret, the regret term and the bonus term cancel out and we get  $\underline{V}_t^s \leq V_\star^s + \mathcal{O}\left(\frac{\epsilon \ln(AT)}{1 - \gamma}\right)$  (Lemma 26). The analysis for  $V_\star^s - \overline{V}_t^s$  is symmetric. By proper choice of  $\epsilon$ , both terms are bounded by  $\frac{1}{8} u$ . Combining
the above with Eq. (3), we can upper bound the left-hand side of the desired inequality Eq. (2) by  $\sum_{t=1}^{T} \mathbf{1}\left[J_t \geq \frac{3}{4} u\right]$ , which is further upper bounded in Eq. (29) by
$$
\begin{array}{l} \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \max  _ {y} \bar {f} _ {\tau} ^ {s} (\hat {x} _ {\tau} ^ {s}, y ^ {s}) - \bar {f} _ {\tau} ^ {s} (\hat {z} _ {\tau} ^ {s}) \geq \frac {u}{8} \right] + \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \underline {{f}} _ {\tau} ^ {s} (\hat {z} _ {\tau} ^ {s}) - \min  _ {x} \underline {{f}} _ {\tau} ^ {s} (x ^ {s}, \hat {y} _ {\tau} ^ {s}) \geq \frac {u}{8} \right] \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left[ x _ {t} ^ {s _ {t} ^ {\top}} \left(\gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \bar {V} _ {t} ^ {s ^ {\prime}} - \underline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}} \geq \frac {u}{4} \right]. \tag {4} \\ \end{array}
$$
Policy Convergence to NE of Regularized Games To bound the first two terms, we show convergence of the policy  $(\hat{x}_{\tau}^{s},\hat{y}_{\tau}^{s})$  to Nash equilibria of both games  $f_{\tau}^{s}$  and  $\overline{f}_{\tau}^{s}$ . To this end, fix any  $p\in [0,1]$ , we define  $f_{\tau}^{s} = pf_{\tau}^{s} + (1 - p)\overline{f}_{\tau}^{s}$  and let  $\hat{z}_{\tau \star}^{s} = (\hat{x}_{\tau \star}^{s},\hat{y}_{\tau \star}^{s})$  be the equilibrium of  $f_{\tau}^{s}(x,y)$ . The analysis is similar to previous algorithms where we first conduct single-step analysis (Lemma 22) and then carefully bound the weighted recursive terms. We show in Lemma 27 that for any  $0 < \epsilon^{\prime}\leq 1$ :  $\sum_{s}\sum_{\tau = 1}^{n_{T + 1}(s)}\mathbf{1}[\mathrm{KL}(\hat{z}_{\tau \star}^{s},\hat{z}_{\tau}^{s})\geq \epsilon^{\prime}]\leq \mathcal{O}(\frac{S^{2}A\ln^{5}(SAT / \delta)}{\eta\epsilon^{2}\epsilon^{\prime}(1 - \gamma)^{3}})$ . This proves policy convergence: the number of iterations where the policy is far away from Nash equilibria of the regularized games is bounded, which can then be translated to upper bounds on the first two terms.
Value Convergence: Bounding  $|\overline{V}_t^s - \underline{V}_t^s|$  It remains to bound the last term in Eq. (4). Define  $c_t = 1[x_{t}^{s_t}(\mathbb{E}_{s'\sim P^{s_t}}[\overline{V}_t^{s'} - \underline{V}_t^{s'}])y_t^{s_t}\geq \tilde{\epsilon} ]$  where  $\tilde{\epsilon} = \frac{u}{4}$ . Then we only need to bound  $C\triangleq \sum_{t = 1}^{T}c_{t}$ . We use the weighted sum  $P_T\triangleq \sum_{t = 1}^{T}c_tx_{t}^{s_t}(\mathbb{E}_{s'\sim P^{s_t}}[\overline{V}_t^{s'} - \underline{V}_t^{s'}])y_t^{s_t}$  as a proxy. On the one hand,  $P_T\geq C\tilde{\epsilon}$ . On the other hand, in Lemma 25, by recursively tracking the update of the value function and carefully choosing  $\eta$  and  $\beta$ , we upper bound  $P_T$  by  $\leq \frac{C\tilde{\epsilon}}{2} +\mathcal{O}(\frac{AS\ln^4(AST / \delta)}{\eta(1 - \gamma)^3})$ . Combining the upper and lower bound of  $P_T$  gives  $C\leq \mathcal{O}(\frac{AS\ln^4(AST / \delta)}{\eta u(1 - \gamma)^3})$  (Corollary 2). Plugging appropriate choices of  $\epsilon, \eta$ , and  $\beta$  in the above bounds proves Theorem 3 (see Appendix E).
In this work, we study decentralized learning in two-player zero-sum Markov games with bandit feedback. We propose the first uncoupled and convergent algorithms with non-asymptotic last-iterate convergence rates for matrix games and irreducible Markov games, respectively. We also introduce a novel notion of path convergence and provide algorithm with path convergence in Markov games without any assumption on the dynamics. Previous results either focus on average-iterate convergence or require stronger feedback/coordination or lack non-asymptotic convergence rates. Our results contribute to the theoretical understanding of the practical success of regularization and last-iterate convergence in multi-agent reinforcement learning.
Settling the optimal last-iterate convergence rate that is achievable by uncoupled learning dynamics is an important open question. The following directions are promising towards closing the gap between current upper bounds  $\mathcal{O}(T^{-1/8})$  and  $O(T^{-1/(9+\varepsilon)})$  and lower bound  $\Omega(T^{-\frac{1}{2}})$ , The impossibility result by [MPS20] demonstrates that certain algorithms with  $\mathcal{O}(\sqrt{T})$  regret diverge in last-iterate. Their result indicates that the current  $\Omega(\frac{1}{\sqrt{T}})$  lower bound on convergence rate may not be tight. On the other hand, our algorithms provide insights and useful templates to potential improvements on the upper bound. For instance, instead of using EXP3-IX update, adapting optimistic policy update or other accelerated first-order methods to the bandit feedback setting is an interesting future direction.
Acknowledgement We thank Chanwoo Park and Kaiqing Zhang for pointing out a mistake in our previous proof. We also thank the anonymous reviewers for their constructive feedback. HL is supported by NSF Award IIS-1943607 and a Google Research Scholar Award.