SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, we consider the case where the opponent of player  $i$  (where  $i \in \{1, 2\}$ ) plays the game with a stationary policy and provide a finite-sample bound for player  $i$  to find the best response.
Corollary 3.1.1. [Rationality $^3$ ] Given  $i \in \{1,2\}$ , suppose that player  $i$  follows the learning dynamics presented in Algorithm 2, but its opponent player  $-i$  follows a stationary policy, denoted by  $\pi^{-i}$ . Then, we have  $\max_{\hat{\pi}^i} U^i(\hat{\pi}^i, \pi^{-i}) - \mathbb{E}[U^i(\pi_{T,K}^i, \pi^{-i})] \leq \tilde{\mathcal{O}}\left(\omega_1 T\left(\frac{\gamma + 1}{2}\right)^T + \frac{\omega_2}{K^{1/2}} + \tau\right)$ , where  $\omega_1$  and  $\omega_2$  are constants that are exponential in  $\tau^{-1}$ , but polynomial in  $|\mathcal{S}|$ ,  $A_{\max}$ , and  $1/(1 - \gamma)$ .
Intuitively, the reason that our algorithm is rational is that it performs an on-policy update in RL. In contrast to an off-policy update, where the behavior policy can be arbitrarily different from the policy being generated during learning (such as in  $Q$ -learning [81]), in the on-policy update for games, each player is actually playing with the policy that is moving towards the best response to its opponent. As a result, when the opponent's policy is stationary, it reduces to a single-agent problem and the player naturally finds the best response (also up to a smoothing bias). This is an advantage of using symmetric and independent learning dynamics. One challenge of analyzing such on-policy learning dynamics is that the behavior policy is time-varying.
4 Conclusion and Future Directions
In this work, we consider payoff-based independent learning for zero-sum matrix games and stochastic games. In both settings, we establish the last-iterate finite-sample guarantees. Our approach, i.e., the coupled Lyapunov drift argument, provides a number of tools that are likely to be of interest more broadly for dealing with iterative algorithms with multiple sets of coupled and stochastic iterates.
Limitations and Future Directions. As mentioned before Corollary 2.1 and after Theorem 3.1, the convergence bounds involve constants that are exponential in  $\tau^{-1}$ , which arise due to the use of the smoothed best response to preserve the naturalness of the learning dynamics. An immediate future direction of this work is to remove such exponential factors by designing better exploration strategies. In the long term, we are interested to see if the algorithmic ideas and the analysis techniques developed in this work can be used to study other classes of games beyond zero-sum stochastic games.
3According to the definition in [27], a dynamics being rational means that the player following this dynamics will converge to the best response to its opponent when the opponent uses an asymptotically stationary policy. Since we are performing finite-sample analysis, we assume the opponent's policy is stationary, because otherwise, the convergence rate (which may be arbitrary) of the opponent's policy will also impact the bound.
The authors would like to thank the anonymous reviewers for the helpful feedback, especially for pointing out several related references. ZC acknowledges support from the PIMCO Postdoctoral Fellowship. KZ acknowledges support from the Northrop Grumman – Maryland Seed Grant Program. EM acknowledges support from NSF CAREER Award 2240110. AW acknowledges support from NSF Grants CNS-2146814, CPS-2136197, CNS-2106403, and NGSDI-2105648.
[1] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354, 2017.
[2] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. Preprint arXiv:1610.03295, 2016.
[3] Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang, Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319:103905, 2023.
[4] Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teptyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. Advances in neural information processing systems, 31, 2018.
[5] Lucian Busoniu, Robert Babuska, Bart De Schutter, et al. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C, 38(2):156-172, 2008.
[6] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.
[7] Gurdal Arslan and Serdar Yüksel. Decentralized  $Q$ -Learning for Stochastic Teams and Games. IEEE Transactions on Automatic Control, 62(4):1545-1558, 2017.
[8] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning, pages 5867-5876, 2018.
[9] Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Learning for Dynamics and Control, pages 256-266. PMLR, 2020.
[10] Yizhou Zhang, Guannan Qu, Pan Xu, Yiheng Lin, Zaiwei Chen, and Adam Wierman. Global convergence of localized policy iteration in networked multi-agent reinforcement learning. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 7(1):1-51, 2023.
[11] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on International Conference on Machine Learning, ICML'94, page 157-163, San Francisco, CA, USA, 1994. Morgan Kaufmann Publishers Inc.
[12] Michael L Littman. Friend-or-foe  $Q$ -learning in general-sum games. In International Conference on Machine Learning, volume 1, pages 322-328, 2001.
[13] Junling Hu and Michael P Wellman. Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research, 4(Nov):1039-1069, 2003.
[14] Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems, 33:5527-5540, 2020.
[88] Gen Li, Yuejie Chi, Yuting Wei, and Yuxin Chen. Minimax-optimal multi-agent RL in Markov games with a generative model. Advances in Neural Information Processing Systems, 35:15353-15367, 2022.
[89] Qiwen Cui and Simon S Du. When are offline two-player zero-sum Markov games solvable? Advances in Neural Information Processing Systems, 35:25779-25791, 2022.
[90] Qiwen Cui and Simon S Du. Provably efficient offline multi-agent reinforcement learning via strategy-wise bonus. Advances in Neural Information Processing Systems, 35:11739-11751, 2022.
[91] Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang, and Zhuoran Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In International Conference on Machine Learning, pages 27117-27142. PMLR, 2022.
[92] Zuguang Gao, Qianqian Ma, Tamer Başar, and John R. Birge. Sample complexity of decentralized tabular  $Q$ -learning for stochastic games. 2023 American Control Conference (ACC), pages 1098-1103, 2023.
[93] David A Levin and Yuval Peres. Markov Chains and Mixing Times, volume 107. American Mathematical Soc., 2017.
[94] Dimitri P Bertsekas and John N Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.
[95] John M Danskin. The Theory of Max-Min and Its Application to Weapons Allocation Problems, volume 5. Springer Science & Business Media, 2012.
[96] Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. Preprint arXiv:1704.00805, 2017.
[97] Amir Beck. First-order methods in optimization, volume 25. SIAM, 2017.
[98] Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. 10th Innovations in Theoretical Computer Science, 2019.
[99] Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, Kentaro Toyoshima, and Atsushi Iwasaki. Last-iterate convergence with full and noisy feedback in two-player zero-sum games. In International Conference on Artificial Intelligence and Statistics, pages 7999–8028. PMLR, 2023.
Continued from Section 1.3, we here discuss several other existing works that are relevant.
Recently, there has been an increasing study of MARL with sample efficiency guarantees recently [16, 82, 20, 17, 21, 83, 84, 22, 85]. Most of them focus on the finite-horizon episodic setting with online exploration, and perform regret analysis, which differs from our last-iterate finite-sample analysis under the stochastic approximation paradigm. Additionally, these algorithms are episodic due to the finite-horizon nature of the setting and are not best-response-type independent learning dynamics that are repeatedly run for infinitely long, which can be viewed as a non-equilibrating adaptation process. In fact, the primary focus of this line of work is a self-play setting where all the players can be controlled to perform centralized learning [86, 16, 82, 20, 17]. Beyond the online setting, finite-sample efficiency has also been established for MARL using a generative model [87, 88] or offline datasets [89–91, 67]. These algorithms tend to be centralized in nature and focus on equilibrium computation, instead of performing independent learning.
Finite-sample complexity has also been established for policy gradient methods, a popular RL approach when applied to solving zero-sum stochastic games [14, 36-38]. However, to ensure convergence, these methods are asymmetric in that the players update their policies at different timescales, e.g., one player updates faster than the other with larger step sizes, or one player fixes its policy while waiting for the other to update. Such asymmetric policy gradient methods are not independent, as some implicit coordination is required to enable such a timescale separation across agents. This style of implicit coordination is also required for the finite-sample analysis of decentralized learning in certain general-sum stochastic games, e.g., [92], which improves the asymptotic convergence in [7]. In contrast, our learning dynamics only require the update of each player's policy to be slower than the update of their  $q$ -functions, but crucially we do not assume a time-scale separation between the two players, making our learning dynamics symmetric.
B Proof Sketch of Theorem 3.1
In this section, we present the key steps and technical ideas used to prove Theorem 3.1. The core challenge here is that Algorithm 2 maintains 3 sets of iterates  $\left(\{q_{t,k}^i\}, \{\pi_{t,k}^i\}, \{v_t^i\}\right)$ , which are coupled. The coupling of their update equations means that it is not possible to separately analyze them. Instead, we develop a coupled Lyapunov drift approach to establish the finite-sample bounds of Algorithm 2. Specifically, we first show that the expected Nash gap can be upper bounded by a sum of properly defined Lyapunov functions, one for each set of the iterates (i.e., the  $v$ -functions, the policies, and the  $q$ -functions). Then, we establish a set of coupled Lyapunov drift inequalities - one for each Lyapunov function. Finally, we decouple the Lyapunov drift inequalities to establish the overall finite-sample bounds. We outline the key steps in the argument below.
To begin with, we show in Lemma D.1 that the  $q$ -functions  $\{q_{t,k}^i\}$  and the  $v$ -functions  $\{v_t^i\}$  generated by Algorithm 2 are uniformly bounded from above in  $\ell_{\infty}$ -norm by  $1 / (1 - \gamma)$  and the entries of the policies  $\{\pi_{t,k}^i\}$  are uniformly bounded below by  $\ell_{\tau} > 0$ . This result will be frequently used in our analysis. We next introduce the Lyapunov functions we use to analyze Algorithm 2. Specifically, for any  $t,k\geq 0$ , let  $\overline{q}_{t,k}^{i}\in \mathbb{R}^{|\mathcal{S}||\mathcal{A}^{i}|}$  be defined as  $\overline{q}_{t,k}^{i}(s) = \mathcal{T}^{i}(v_{t}^{i})(s)\pi_{t,k}^{-i}(s)$  for all  $s\in S$ . Let
$$
\mathcal {L} _ {\mathrm {s u m}} (t) = \| v _ {t} ^ {1} + v _ {t} ^ {2} \| _ {\infty}, \quad \mathcal {L} _ {v} (t) = \sum_ {i = 1, 2} \| v _ {t} ^ {i} - v _ {*} ^ {i} \| _ {\infty},
$$
$$
\mathcal {L} _ {q} (t, k) = \sum_ {i = 1, 2} \sum_ {s \in \mathcal {S}} \| q _ {t, k} ^ {i} (s) - \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, k} ^ {- i} (s) \| _ {2} ^ {2} = \sum_ {i = 1, 2} \| q _ {t, k} ^ {i} - \bar {q} _ {t, k} ^ {i} \| _ {2} ^ {2},
$$
$$
\mathcal {L} _ {\pi} (t, k) = \max _ {s \in \mathcal {S}} \sum_ {i = 1, 2} \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left\{(\mu^ {i} - \pi_ {t, k} ^ {i} (s)) ^ {\top} \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, k} ^ {- i} (s) + \tau \nu (\mu^ {i}) - \tau \nu (\pi_ {t, k} ^ {i} (s)) \right\}.
$$
Note that  $\mathcal{L}_{\mathrm{sum}}(t)$  is introduced to deal with the fact that the induced matrix game the inner loop of Algorithm 2 is designed to solve may not be a zero-sum game due to independent learning. See the discussion in Section 1.2 and the paragraph after Algorithm 2. At the core of our argument is the following inequality (cf. Lemma D.4):
$$
\mathrm {N G} \left(\pi_ {T, K} ^ {1}, \pi_ {T, K} ^ {2}\right) \leq \frac {4}{1 - \gamma} \left(2 \mathcal {L} _ {\text {s u m}} (T) + \mathcal {L} _ {v} (T) + \mathcal {L} _ {\pi} (T, K) + 2 \tau \log \left(A _ {\max }\right)\right), \tag {6}
$$
which motivates us to bound all the Lyapunov functions.
B.1 Analysis of the Outer Loop:  $v$ -Function Update
Motivated by Eq. (6), we need to bound  $\mathcal{L}_{\mathrm{sum}}(T)$  and  $\mathcal{L}_v(T)$ . To achieve that, we establish Lyapunov drift inequalities for them. Specifically, we show in Lemmas D.5 and D.6 that
$$
\mathcal {L} _ {v} (t + 1) \leq \underbrace {\gamma \mathcal {L} _ {v} (t)} _ {\text {D r i f t}} + \underbrace {4 \mathcal {L} _ {\text {s u m}} (t) + 2 \mathcal {L} _ {q} ^ {1 / 2} (t , K) + 4 \mathcal {L} _ {\pi} (t , K) + 6 \tau \log \left(A _ {\max }\right)} _ {\text {A d d i t i v e E r r o r s}}, \tag {7}
$$
$$
\mathcal {L} _ {\text {s u m}} (t + 1) \leq \underbrace {\gamma \mathcal {L} _ {\text {s u m}} (t)} _ {\text {D r i f t}} + \underbrace {2 \mathcal {L} _ {q} (t , K) ^ {1 / 2}} _ {\text {A d d i t i v e E r r o r s}}, \quad \forall t \geq 0. \tag {8}
$$
Suppose that the Additive Errors in the previous two inequalities were only functions of  $v_{t}^{1}$  and  $v_{t}^{2}$ , then these two Lyapunov drift inequalities can be repeatedly used to obtain a convergence bound for  $\mathcal{L}_{\mathrm{sum}}(T)$  and  $\mathcal{L}_v(T)$ . However, the coupled nature of Eqs. (7) and (8) requires us to analyze the policies and the  $q$ -functions in the inner loop, and establish their Lyapunov drift inequalities.
B.2 Analysis of the Inner Loop: Policy Update
As illustrated in Section 2.1 and Section 3.1, for each state  $s$ , the update equation of the policies can be viewed as a discrete and stochastic variant of the smoothed best-response dynamics for solving matrix games [29]. Typically, the following Lyapunov function is used to study such dynamics [53]:
$$
V _ {X} \left(\mu^ {1}, \mu^ {2}\right) = \sum_ {i = 1, 2} \max  _ {\hat {\mu} ^ {i} \in \Delta \left(\mathcal {A} ^ {i}\right)} \left\{\left(\hat {\mu} ^ {i} - \mu^ {i}\right) ^ {\top} X _ {i} \mu^ {- i} + \tau \nu \left(\hat {\mu} ^ {i}\right) - \tau \nu \left(\mu^ {i}\right) \right\}, \tag {9}
$$
where  $X_{1}$  and  $X_{2}$  are the payoff matrices for player 1 and player 2, respectively, and  $\nu(\cdot)$  is the entropy function. Specialized to our case, given a joint  $v$ -function  $v = (v^{1}, v^{2})$  from the outer loop<sup>4</sup> and a state  $s \in S$ , we would like to use
$$
V _ {v, s} (\pi^ {1} (s), \pi^ {2} (s)) = \sum_ {i = 1, 2} \max _ {\hat {\mu} ^ {i} \in \Delta (\mathcal {A} ^ {i})} \left\{(\hat {\mu} ^ {i} - \pi^ {i} (s)) ^ {\top} \mathcal {T} ^ {i} (v ^ {i}) (s) \pi^ {- i} (s) + \tau \nu (\hat {\mu} ^ {i}) - \tau \nu (\pi^ {i} (s)) \right\}
$$
as our Lyapunov function. Note that  $\max_{s\in S}V_{v_t,s}(\pi_{t,k}^1 (s),\pi_{t,k}^2 (s)) = \mathcal{L}_\pi (t,k)$ . A sequence of properties (e.g., strong convexity, smoothness, etc.) regarding the Lyapunov function  $V_{X}(\cdot ,\cdot)$  is established in Lemma D.7. In the end, we show in Lemma D.8 that
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k + 1) \right] \leq \underbrace {\left(1 - \frac {3 \beta_ {k}}{4}\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t , k) \right]} _ {\text {D r i f t}} \\ + \underbrace {2 L _ {\tau} \beta_ {k} ^ {2} + \frac {3 2 A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{\tau^ {3} \ell_ {\tau} ^ {2} (1 - \gamma) ^ {2}} \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t , k) \right] + \frac {1 6 A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{\tau} \mathcal {L} _ {\operatorname* {s u m}} (t) ^ {2}} _ {\text {A d d i t i v e E r r o r s}}, \tag {10} \\ \end{array}
$$
where  $\mathbb{E}_t[\cdot ]$  stands for conditional expectation conditioned on the history up to the beginning of the  $t$ -th outer loop. To interpret the above, suppose that we were considering the continuous-time smoothed best-response dynamics. Then, the additive error term would disappear in the sense that the time-derivative of the Lyapunov function along the trajectory of the ODE is strictly negative. Thus, the three terms in the Additive Errors can be interpreted as (1) the discretization error in the update equation, (2) the stochastic error in the  $q$ -function estimate, and (3) the error due to the non-zero-sum structure of the inner-loop auxiliary matrix game.
B.3 Analysis of the Inner Loop:  $q$ -Function Update
Our next focus is the  $q$ -function update. The  $q$ -function update equation is in the same spirit as TD-learning in RL, and a necessary condition for the convergence of TD-learning is that the behavior
<sup>4</sup>Due to the nested-loop structure of Algorithm 2, conditioned on the history up to the beginning of the  $t$ -th outer loop, the  $v$ -functions  $v_{t}^{1}$  and  $v_{t}^{2}$  are constants.
policy (i.e., the policy used to collect samples) should enable the agent to sufficiently explore the environment. To achieve this goal, since we show in Lemma D.1 that all joint policies from the algorithm trajectory have uniformly lower-bounded entries (with lower bound  $\ell_{\tau} > 0$ ), it is enough to restrict our attention to a "soft" policy class  $\Pi_{\tau} \coloneqq \{\pi = (\pi^{1}, \pi^{2}) \mid \min_{s,a^{1}} \pi^{1}(a^{1}|s) \geq \ell_{\tau}, \min_{s,a^{2}} \pi^{2}(a^{2}|s) \geq \ell_{\tau}\}$ . The following lemma, which is an extension of [10, Lemma 4], establishes a uniform exploration property under Assumption 3.1.
To present the result, we need the following notation. Under Assumption 3.1, the Markov chain induced by the joint policy  $\pi_b$  has a unique stationary distribution  $\mu_b \in \Delta(S)$  [93], the minimum component of which is denoted by  $\mu_{b,\min}$ . In addition, there exists  $\rho_b \in (0,1)$  such that  $\max_{s \in S} \left\| P_{\pi_b}^k(s,\cdot) - \mu_b(\cdot) \right\|_{\mathrm{TV}} \leq 2\rho_b^k$  for all  $k \geq 0$  [93], where  $P_{\pi_b}$  is the transition probability matrix of the Markov chain  $\{S_k\}$  under  $\pi_b$ . We also define the mixing time in the following. Given a joint policy  $\pi = (\pi^1, \pi^2)$  and an accuracy level  $\eta > 0$ , the  $\eta$ -mixing time of the Markov chain  $\{S_k\}$  induced by  $\pi$  is defined as
$$
t _ {\pi , \eta} = \min  \left\{k \geq 0: \max  _ {s \in \mathcal {S}} \| P _ {\pi} ^ {k} (s, \cdot) - \mu_ {\pi} (\cdot) \| _ {\mathrm {T V}} \leq \eta \right\}, \tag {11}
$$
where  $P_{\pi}$  is the  $\pi$ -induced transition probability matrix and  $\mu_{\pi}$  is the stationary distribution of  $\{S_k\}$  under  $\pi$ , provided that it exists and is unique. When the induced Markov chain mixes at a geometric rate, it is easy to see that  $t_{\pi,\eta} = \mathcal{O}(\log(1/\eta))$ .
Lemma B.1 (An Extension of Lemma 4 in [10]). Suppose that Assumption 3.1 is satisfied. Then we have the following results.
(1) For any  $\pi = (\pi^1, \pi^2) \in \Pi_{\tau}$ , the Markov chain  $\{S_k\}$  induced by the joint policy  $\pi$  is irreducible and aperiodic, hence admits a unique stationary distribution  $\mu_{\pi} \in \Delta(S)$ .
(2) It holds that  $\sup_{\pi \in \Pi_{\tau}} \max_{s \in S} \| P_{\pi}^{k}(s, \cdot) - \mu_{\pi}(\cdot) \|_{TV} \leq 2\rho_{\tau}^{k}$  for any  $k \geq 0$ , where  $\rho_{\tau} = \rho_{b}^{\ell_{\tau}^{2r_b}\mu_{b,\min}}$  and  $r_b := \min \{k \geq 0 : P_{\pi_b}^{k}(s, s') > 0, \forall (s, s')\}$ . As a result, we have
$$
t \left(\ell_ {\tau}, \eta\right) := \sup  _ {\pi \in \Pi_ {\tau}} t _ {\pi , \eta} \leq \frac {t _ {\pi_ {b} , \eta}}{\ell_ {\tau} ^ {2 r _ {b}} \mu_ {b , \min }}, \tag {12}
$$
where we recall that  $t_{\pi_b,\eta}$  is the  $\eta$  - mixing time of the Markov chain  $\{S_k\}$  induced by  $\pi_b$ .
(3) There exists  $L_{p} \geq 1$  (which was used in the statement of Theorem 3.1) such that
$$
\| \mu_ {\pi} - \mu_ {\bar {\pi}} \| _ {1} \leq L _ {p} \left(\max  _ {s \in \mathcal {S}} \| \pi^ {1} (s) - \pi^ {2} (s) \| _ {1} + \max  _ {s \in \mathcal {S}} \| \bar {\pi} ^ {1} (s) - \bar {\pi} ^ {2} (s) \| _ {1}\right)
$$
for all  $\pi = (\pi^1,\pi^2),\bar{\pi} = (\bar{\pi}^1,\bar{\pi}^2)\in \Pi_{\tau}$
(4)  $\mu_{\mathrm{min}}\coloneqq \inf_{\pi \in \Pi_{\tau}}\min_{s\in S}\mu_{\pi}(s) > 0.$
Remark. Lemma B.1 (1), (3), and (4) were previous established in [10, Lemma 4]. Lemma B.1 (2) enables us to see the explicit dependence of the "uniform mixing time" on the margin  $\ell_{\tau}$  and the mixing time of the benchmark exploration policy  $\pi_b$ .
In view of Lemma B.1 (2), we have fast mixing for all policies in  $\Pi_{\tau}$  if  $(i)$  the margin  $\ell_{\tau}$  is large, and  $(ii)$  the Markov chain  $\{S_k\}$  induced by the benchmark exploration policy  $\pi_b$  is well-behaved. By "well-behaved" we mean the mixing time is small (i.e., small  $t_{\pi_b,\eta}$ ) and the stationary distribution is relatively well-balanced (i.e., large  $\mu_{b,\min}$ ). Point  $(i)$  agrees with our intuition as a large margin encourages more exploration. To make sense of point  $(ii)$ , since  $\pi(a|s) \geq \ell_{\tau}^{2}\pi_{b}(a|s)$  for all  $s$  and  $a = (a^1, a^2)$ , we can write  $\pi$  as a convex combination between  $\pi_b$  and some residual policy  $\tilde{\pi}: \pi(\cdot|s) = \ell_{\tau}^{2}\pi_{b}(\cdot|s) + (1 - \ell_{\tau}^{2})\tilde{\pi}(\cdot|s)$  for all  $s \in S$ . Therefore, since any  $\pi \in \Pi_{\tau}$  has a portion of the benchmark exploration policy  $\pi_b$  in it, it makes intuitive sense that fast mixing of  $\{S_k\}$  under  $\pi_b$  implies, to some extent, fast mixing of  $\{S_k\}$  under  $\pi \in \Pi_{\tau}$ . Note that, as the margin  $\ell_{\tau}$  approaches zero, the uniform mixing time in Lemma B.1 (2) goes to infinity. This is not avoidable in general, as demonstrated by a simple MDP example constructed in Appendix E.
We define  $c_{\tau} = \mu_{\min}\ell_{\tau}$ , which was used in the statement of Theorem 3.1. With Lemma B.1 in hand, we are now able to analyze the behavior of the  $q$ -functions. We model the  $q$ -function update as a
stochastic approximation algorithm driven by time-inhomogeneous Markovian noise, and use the norm-square function
$$
\sum_ {i = 1, 2} \sum_ {s} \| q ^ {i} (s) - \mathcal {T} ^ {i} (v ^ {i}) (s) \pi^ {- i} (s) \| _ {2} ^ {2}
$$
as the Lyapunov function to study its behavior. Note that  $\mathcal{L}_q(t,k) = \sum_{i=1,2} \sum_s \|q_{t,k}^i(s) - T^i(v_t^i)(s)\pi_{t,k}^{-i}(s)\|_2^2$ . The key challenge to establishing a Lyapunov drift inequality is to control a difference of the form
$$
\mathbb {E} \left[ F ^ {i} \left(q ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \right] - \mathbb {E} \left[ F ^ {i} \left(q ^ {i}, \hat {S}, \hat {A} ^ {i}, \hat {A} ^ {- i}, \hat {S} ^ {\prime}\right) \right] \tag {13}
$$