SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although Theorem 2.1 is stated in terms of the expectation of the regularized Nash gap, it implies the mean-square convergence of the policy iterates  $(\pi_k^1,\pi_k^2)$ . To see this, note that the regularized Nash gap  $\mathrm{NG}_{\tau}(\pi^{1},\pi^{2})$  has a unique minimizer, i.e., the Nash distribution and is denoted by  $(\pi_{*,\tau}^{1},\pi_{*,\tau}^{2})$ . In addition, fixing  $\pi^1$  (respectively,  $\pi^2$ ), the function  $\mathrm{NG}_{\tau}(\pi^{1},\cdot)$  (respectively,  $\mathrm{NG}_{\tau}(\cdot ,\pi^{2}))$  is a  $\tau$ -strongly convex function with respect to  $\pi^2$  (respectively,  $\pi^1$ ). See Lemma D.7 for a proof. Therefore, by the quadratic growth property of strongly convex functions, we have
$$
\begin{array}{l} \mathrm {N G} _ {\tau} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right) = \mathrm {N G} _ {\tau} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right) - \mathrm {N G} _ {\tau} \left(\pi_ {* , \tau} ^ {1}, \pi_ {k} ^ {2}\right) + \mathrm {N G} _ {\tau} \left(\pi_ {* , \tau} ^ {1}, \pi_ {k} ^ {2}\right) - \mathrm {N G} _ {\tau} \left(\pi_ {* , \tau} ^ {1}, \pi_ {* , \tau} ^ {2}\right) \\ \geq \frac {\tau}{2} \left(\| \pi_ {k} ^ {1} - \pi_ {*, \tau} ^ {1} \| _ {2} ^ {2} + \| \pi_ {k} ^ {2} - \pi_ {*, \tau} ^ {2} \| _ {2} ^ {2}\right). \\ \end{array}
$$
As a result, up to a multiplicative constant, the convergence bound for  $\mathbb{E}[\mathrm{NG}_{\tau}(\pi_k^1,\pi_k^2)]$  implies a convergence bound of  $\mathbb{E}[\| \pi_k^1 -\pi_{*,\tau}^1\| _2^2 ] + \mathbb{E}[\| \pi_k^2 -\pi_{*,\tau}^2\| _2^2 ]$
Based on Theorem 2.1, we next derive the sample complexity of Algorithm 1 in the following corollary. See Appendix C.5 for the proof.
Corollary 2.1.1. Given  $\epsilon >0$ , to achieve  $\mathbb{E}[NG_{\tau}(\pi_K^1,\pi_K^2)]\leq \epsilon$ , the sample complexity is  $\mathcal{O}(\epsilon^{-1})$
To the best of our knowledge, Theorem 2.1 and Corollary 2.1.1 present the first last-iterate finite-sample analysis of Algorithm 1 [28]. Importantly, with only feedback in the form of realized payoffs, we achieve a sample complexity of  $\mathcal{O}(\epsilon^{-1})$  to find the Nash distribution. In general, for smooth and strongly monotone games, the lower bound for the sample complexity of payoff-based or zeroth-order algorithms is  $\mathcal{O}(\epsilon^{-2})$  [71]. We have an improved  $\mathcal{O}(\epsilon^{-1})$  sample complexity due to the bilinear structure of the game (up to a regularizer). In particular, with bandit feedback, the  $q$ -function is constructed as an efficient estimator for the marginalized payoff  $R_{i}\pi_{k}^{-i}$ , which can also be interpreted as the gradient. Therefore, Algorithm 1 enjoys the fast  $\mathcal{O}(\epsilon^{-1})$  sample complexity that is comparable to the first-order method [72].
The Dependence on the Temperature  $\tau$ . Although our finite-sample bound enjoys the  $\mathcal{O}(1 / K)$  rate of convergence, the stepsize ratio  $c_{\alpha,\beta}$  appears as  $c_{\alpha,\beta}^{-1}$  in the bound. Since  $c_{\alpha,\beta} = o(\ell_{\tau})$  (cf. Condition 2.1) and  $\ell_{\tau}$  is exponentially small in  $\tau$ , the finite-sample bound is actually exponentially large in  $\tau^{-1}$ . To illustrate this phenomenon, consider the update equation for the  $q$ -functions (cf. Algorithm 1 Line 5). Observe that the  $q$ -functions are updated asynchronously because only one component (which corresponds to the action taken at time step  $k$ ) of the vector-valued  $q_k^i$  is updated in the  $k$ -th iteration. Suppose that an action  $a^i$  is never taken in the algorithm trajectory, which means that  $q_k^i(a^i)$  is never updated during learning. Then, in general, we cannot expect the convergence of  $q_k^i$  or  $\pi_k^i$ . Similarly, suppose that an action is rarely taken in the learning dynamics, we would expect the overall convergence rate to be slow. Therefore, the finite-sample bound should depend on the quantity  $\min_{i \in \{1,2\}} \min_{0 \leq k \leq K} \min_{a^i} \pi_k^i(a^i)$ , which captures the exploration abilities of Algorithm 1. Due to the exponential nature of softmax functions, the parameter  $\ell_{\tau}$ , which we establish in Lemma C.2 as a lower bound of  $\min_{i \in \{1,2\}} \min_{0 \leq k \leq K} \min_{a^i} \pi_k^i(a^i)$ , is also exponentially small in  $\tau$ . This eventually leads to the exponential dependence in  $\tau^{-1}$  in the finite-sample bound.
A consequence of having such an exponential factor of  $\tau^{-1}$  in the sample complexity bound is that, if we want to have convergence to a Nash equilibrium rather than to the Nash distribution, the sample complexity can be exponentially large. To see this, note that the following bound holds regarding the Nash gap and the regularized Nash gap:
$$
\mathrm {N G} \left(\pi^ {1}, \pi^ {2}\right) \leq \mathrm {N G} _ {\tau} \left(\pi^ {1}, \pi^ {2}\right) + 2 \tau \log \left(A _ {\max }\right), \quad \forall (\pi^ {1}, \pi^ {2}), \tag {3}
$$
which, after combining with Theorem 2.1, gives the following corollary. For simplicity of presentation, we only state the result for using constant stepsizes.
Corollary 2.1.2. Under the same conditions stated in Theorem 2.1 (1), we have
$$
\mathbb {E} \left[ N G \left(\pi_ {K} ^ {1}, \pi_ {K} ^ {2}\right) \right] \leq B _ {i n} \left(1 - \frac {\beta}{4}\right) ^ {K} + 8 L _ {\tau} \beta + \frac {6 4 \alpha}{c _ {\alpha , \beta}} + 2 \tau \log \left(A _ {\max }\right). \tag {4}
$$
The last term on the RHS of Eq. (4) can be viewed as the bias due to using smoothed best-response. In view of Eq. (4), to achieve  $\mathbb{E}[\mathrm{NG}(\pi_K^1,\pi_K^2)]\leq \epsilon$ , we need  $\tau = \mathcal{O}(\epsilon)$ . Since  $c_{\alpha ,\beta}$  appears in the denominator of our finite-sample bound and is exponentially small in  $\tau$ , the overall sample complexity
for the convergence to a Nash equilibrium can be exponentially large in  $\epsilon^{-1}$ . In Appendix F, we conduct numerical experiments to investigate the impact of  $\tau$  on this smoothing bias.
In light of the discussion before Corollary 2.1.2, the reason for such an exponentially large sample complexity for finding a Nash equilibrium is due to the limitation of using the softmax policies in smoothed best-response for exploration. We kept the softmax policy without further modification to preserve the "naturalness" of the learning dynamics, which is part of the motivation for studying independent learning in games [73]. A future direction of this work is to remove such an exponential dependence on  $\tau$  by designing an improved exploration strategy.
3 Zero-Sum Stochastic Games
Moving to the setting of stochastic games, we consider an infinite-horizon discounted two-player zero-sum stochastic game  $\mathcal{M} = (\mathcal{S},\mathcal{A}^1,\mathcal{A}^2,p,R_1,R_2,\gamma)$ , where  $\mathcal{S}$  is a finite state space,  $\mathcal{A}^1$  (respectively,  $\mathcal{A}^2$ ) is a finite action space for player 1 (respectively, player 2),  $p$  represents the transition probabilities, in particular,  $p(s' \mid s,a^1,a^2)$  is the probability of transitioning to state  $s'$  after player 1 taking action  $a^1$  and player 2 taking action  $a^2$  simultaneously at state  $s$ ,  $R_1: \mathcal{S} \times \mathcal{A}^1 \times \mathcal{A}^2 \mapsto \mathbb{R}$  (respectively,  $R_2: \mathcal{S} \times \mathcal{A}^2 \times \mathcal{A}^1 \mapsto \mathbb{R}$ ) is player 1's (respectively, player 2's) reward function, and  $\gamma \in (0,1)$  is the discount factor. Note that we have  $R_1(s,a^1,a^2) + R_2(s,a^2,a^1) = 0$  for all  $(s,a^1,a^2)$ . We assume without loss of generality that  $\max_{s,a^1,a^2}|R_1(s,a^1,a^2)| \leq 1$ , and denote  $A_{\mathrm{max}} = \max(|\mathcal{A}^1|,|\mathcal{A}^2|)$ .
Given a joint policy  $\pi = (\pi^1, \pi^2)$ , where  $\pi^i: S \mapsto \Delta(\mathcal{A}^i)$ ,  $i \in \{1, 2\}$ , we define the local  $q$ -function  $q_{\pi}^{i} \in \mathbb{R}^{|\mathcal{S}| |\mathcal{A}^{i}|}$  of player  $i$  as  $q_{\pi}^{i}(s, a^{i}) = \mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k} R_{i}(S_{k}, A_{k}^{i}, A_{k}^{-i}) \mid S_{0} = s, A_{0}^{i} = a^{i}\right]$  for all  $(s, a^{i})$ , where we use the notation  $\mathbb{E}_{\pi}[\cdot]$  to indicate that the actions are chosen according to the joint policy  $\pi$ . In addition, we define the global value function  $v_{\pi}^{i} \in \mathbb{R}^{|\mathcal{S}|}$  as  $v_{\pi}^{i}(s) = \mathbb{E}_{A^{i} \sim \pi^{i}(\cdot|s)}[q_{\pi}^{i}(s, A^{i})]$  for all  $s$ , and the expected value function  $U^{i}(\pi^{i}, \pi^{-i}) \in \mathbb{R}$  as  $U^{i}(\pi^{i}, \pi^{-i}) = \mathbb{E}_{S \sim p_{o}}[v_{\pi}^{i}(S)]$ , where  $p_{o} \in \Delta(\mathcal{S})$  is an arbitrary initial distribution on the states. The Nash gap in the case of stochastic games is defined in the following.
Definition 3.1 (Nash Gap in Zero-Sum Stochastic Games). Given a joint policy  $\pi = (\pi^1, \pi^2)$ , the Nash gap  $\mathrm{NG}(\pi^1, \pi^2)$  is defined as  $\mathrm{NG}(\pi^1, \pi^2) = \sum_{i=1,2} \left( \max_{\hat{\pi}^i} U^i(\hat{\pi}^i, \pi^{-i}) - U^i(\pi^i, \pi^{-i}) \right)$ .
Similar to the matrix-game setting, a joint policy  $\pi = (\pi^1,\pi^2)$  satisfying  $\mathrm{NG}(\pi^1,\pi^2) = 0$  is called a Nash equilibrium, which may not be unique.
Additional Notation. In what follows, we will frequently work with the real vectors in  $\mathbb{R}^{|S||\mathcal{A}^i|}$ ,  $\mathbb{R}^{|S||\mathcal{A}^{-i}|}$ , and  $\mathbb{R}^{|S||\mathcal{A}^i||\mathcal{A}^{-i}|}$ , where  $i \in \{1,2\}$ . To simplify the notation, for any  $x \in \mathbb{R}^{|S||\mathcal{A}^i||\mathcal{A}^{-i}|}$  we use  $x(s)$  to denote the  $|\mathcal{A}^i| \times |\mathcal{A}^{-i}|$  matrix with the  $(a^i, a^{-i})$ -th entry being  $x(s, a^i, a^{-i})$ . For any  $y \in \mathbb{R}^{|S||\mathcal{A}^i|}$ , we use  $y(s)$  to denote the  $|\mathcal{A}^i|$ -dimensional vector with its  $a^i$ -th entry being  $y(s, a^i)$ .
3.1 Value Iteration with Smoothed Best-Response Dynamics
Our learning dynamics for stochastic games (cf. Algorithm 2) build on the dynamics for matrix games studied in Section 2.1, with the additional incorporation of minimax value iteration, a well-known approach for solving zero-sum stochastic games [74].
Algorithmic Ideas. To motivate the learning dynamics, we first introduce the minimax value iteration. For  $i \in \{1,2\}$ , let  $\mathcal{T}^i: \mathbb{R}^{|\mathcal{S}|} \mapsto \mathbb{R}^{|\mathcal{S}||\mathcal{A}^i||\mathcal{A}^{-i}|}$  be an operator defined as
$$
\mathcal {T} ^ {i} (v) \left(s, a ^ {i}, a ^ {- i}\right) = R _ {i} \left(s, a ^ {i}, a ^ {- i}\right) + \gamma \mathbb {E} \left[ v \left(S _ {1}\right) \mid S _ {0} = s, A _ {0} ^ {i} = a ^ {i}, A _ {0} ^ {- i} = a ^ {- i} \right]
$$
for all  $(s,a^i,a^{-i})$  and  $v\in \mathbb{R}^{|\mathcal{S}|}$ . Given  $X\in \mathbb{R}^{|\mathcal{A}^i|\times |\mathcal{A}^{-i}|}$ , we define  $\nu a^{i}:\mathbb{R}^{|\mathcal{A}^{i}|\times |\mathcal{A}^{-i}|}\mapsto \mathbb{R}$  as
$$
val^{i}(X) = \max_{\mu^{i}\in \Delta (\mathcal{A}^{i})}\min_{\mu^{-i}\in \Delta (\mathcal{A}^{-i})}\{(\mu^{i})^{\top}X\mu^{-i}\} = \min_{\mu^{-i}\in \Delta (\mathcal{A}^{-i})}\max_{\mu^{i}\in \Delta (\mathcal{A}^{i})}\{(\mu^{i})^{\top}X\mu^{-i}\} .
$$
Then, the minimax Bellman operator  $\mathcal{B}^i:\mathbb{R}^{|\mathcal{S}|}\mapsto \mathbb{R}^{|\mathcal{S}|}$  is defined as  $[\mathcal{B}^i (v)](s) = val^i (\mathcal{T}^i (v)(s))$  for all  $s\in S$ , where  $\mathcal{T}^i (v)(s)$  is an  $|\mathcal{A}^i |\times |\mathcal{A}^{-i}|$  matrix according to our notation. It is known that the operator  $\mathcal{B}^i (\cdot)$  is a  $\gamma$ -contraction mapping with respect to the  $\ell_{\infty}$ -norm [74], hence it admits a unique fixed point, which we denote by  $v_{*}^{i}$ .
A common approach for solving zero-sum stochastic games is to first implement the minimax value iteration  $v_{t+1}^{i} = \mathcal{B}^{i}(v_{t}^{i})$  until (approximate) convergence to  $v_{*}^{i}$  [75], and then solve the matrix game  $\max_{\mu^{i} \in \Delta(\mathcal{A}^{i})} \min_{\mu^{-i} \in \Delta(\mathcal{A}^{-i})} (\mu^{i})^{\top} \mathcal{T}^{i}(v_{*}^{i})(s) \mu^{-i}$  for each state  $s$  to obtain an (approximate) Nash equilibrium policy. However, implementing this algorithm requires complete knowledge of the underlying transition probabilities. Moreover, since it is an off-policy algorithm, the output is independent of the opponent's policy. Thus, it is not rational by the definition in [27]. To develop a model-free and rational learning dynamics, let us first rewrite the minimax value iteration:
$$
v _ {t + 1} ^ {i} = \hat {v}, \text {w h e r e} \hat {v} (s) = \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \mu^ {- i}, \forall s \in \mathcal {S}, \tag {5}
$$
where  $\hat{v} \in \mathbb{R}^{|S|}$  is a dummy variable. In view of Eq. (5), we need to solve a matrix game with payoff matrix  $\mathcal{T}^i(v_t^i)(s)$  for each state  $s$  and then update the value of the game to  $v_{t+1}^i(s)$ . In light of Algorithm 1, we already know how to solve matrix games with independent learning. Thus, what remains to do is to combine Algorithm 1 with value iteration. This leads to Algorithm 2, which is presented from player  $i$ 's perspective, where  $i \in \{1,2\}$ .
Algorithm 2 Value Iteration with Smoothed Best-Response (VI-SBR) Dynamics
1: Input: Integers  $K$  and  $T$ , initializations  $v_0^i = 0 \in \mathbb{R}^{|S|}$ ,  $q_{0,0}^i = 0 \in \mathbb{R}^{|S||A^i|}$ , and  $\pi_{0,0}^i (\cdot |s) = \mathrm{Unif}(\mathcal{A}^i)$  for all  $s \in S$ .  
2: for  $t = 0,1,\dots ,T - 1$  do  
3: for  $k = 0,1,\dots ,K - 1$  do  
4:  $\pi_{t,k + 1}^i (s) = \pi_{t,k}^i (s) + \beta_k(\sigma_\tau (q_{t,k}^i (s)) - \pi_{t,k}^i (s))$  for all  $s \in S$   
5: Play  $A_k^i \sim \pi_{t,k + 1}^i (\cdot |S_k)$  (against  $A_k^{-i}$ ) and observe  $S_{k + 1} \sim p(\cdot |S_k, A_k^i, A_k^{-i})$   
6:  $q_{t,k + 1}^i (s,a^i) = q_{t,k}^i (s,a^i) + \alpha_k\mathbb{1}_{\{(s,a^i) = (S_k,A_k^i)\}}(R_i(S_k,A_k^i,A_k^{-i}) + \gamma v_t^i (S_{k + 1}) - q_{t,k}^i (S_k,A_k^i))$  for all  $(s,a^i)$   
7: end for  
8:  $v_{t + 1}^i (s) = \pi_{t,K}^i (s)^\top q_{t,K}^i (s)$  for all  $s \in S$   
9:  $S_0 = S_K$ ,  $q_{t + 1,0}^i = q_{t,K}^i$ , and  $\pi_{t + 1,0}^i = \pi_{t,K}^i$   
10: end for
Algorithm Details. For each state  $s$ , the inner loop of Algorithm 2 is designed to solve a matrix game with payoff matrices  $\mathcal{T}^1 (v_t^1)(s)$  and  $\mathcal{T}^2 (v_t^2)(s)$  for each state  $s\in S$ , which reduces to Algorithm 1 when (1) the stochastic game has only one state, and (2)  $v_{t}^{1} = v_{t}^{2} = 0$ . However, in general, since  $v_{t}^{1}$  and  $v_{t}^{2}$  are independently maintained by players 1 and 2, the quantity
$$
\mathcal {T} ^ {1} (v _ {t} ^ {1}) (s, a ^ {1}, a ^ {2}) + \mathcal {T} ^ {2} (v _ {t} ^ {2}) (s, a ^ {2}, a ^ {1}) = \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} \mid s, a ^ {1}, a ^ {2}) (v _ {t} ^ {1} (s ^ {\prime}) + v _ {t} ^ {2} (s ^ {\prime}))
$$
is in general non-zero during learning. As a result, the auxiliary matrix game (with payoff matrices  $\mathcal{T}^1(v_t^1)(s)$  and  $\mathcal{T}^2(v_t^2)(s)$ ) at state  $s$  that the inner loop of Algorithm 2 is designed to solve is not necessarily a zero-sum matrix game, which presents a major challenge in the finite-sample analysis, as illustrated previously in Section 1.2.
The outer loop of Algorithm 2 is an "on-policy" variant of minimax value iteration. To see this, note that, ideally, we would synchronize  $v_{t+1}^i(s)$  with  $\pi_{t,K}^i(s)^\top \mathcal{T}^i(v_t^i)(s)\pi_{t,K}^{-i}(s)$ , which is an approximation of  $[\mathcal{B}^i(v)](s) = \text{val}^i(\mathcal{T}^i(v_t^i)(s))$  by design of our inner loop. However, player  $i$  has no access to  $\pi_{t,K}^{-i}$  in independent learning. Fortunately, the  $q$ -function  $q_{t,K}^i$  is precisely constructed as an estimate of  $\mathcal{T}^i(v_t^i)(s)\pi_{t,K}^{-i}(s)$ , as illustrated in Section 2.1, which leads to the outer loop of Algorithm 2. In Algorithm 2 Line 8, we set  $S_0 = S_K$  to ensure that the initial state of the next inner loop is the last state of the previous one; hence Algorithm 2 is driven by a single trajectory of Markovian samples.
3.2 Finite-Sample Analysis
We now state our main results, which, to the best of our knowledge, provide the first last-iterate finite-sample bound for best-response-type payoff-based independent learning dynamics in zero-sum stochastic games. Our results are based on the following assumption.
Assumption 3.1. There exists a joint policy  $\pi_{b} = (\pi_{b}^{1},\pi_{b}^{2})$  such that the Markov chain  $\{S_k\}_{k\geq 0}$  induced by  $\pi_{b}$  is irreducible and aperiodic.
One challenge in our finite-sample analysis is that the behavior policies used for taking the actions are time-varying, due to the best-response nature of the dynamics. Most, if not all, existing finite-sample guarantees of RL algorithms under time-varying behavior policies assume that the induced Markov chain of any policy, or any policy encountered along the algorithm trajectory, is uniformly geometrically ergodic [41, 76, 77, 59, 78-80]. Assumption 3.1 is weaker, since it assumes only the existence of one policy that induces an irreducible and aperiodic Markov chain.
We consider using either constant step sizes, i.e.,  $\alpha_{k} \equiv \alpha$  and  $\beta_{k} \equiv \beta = c_{\alpha,\beta}$ , or diminishing step sizes of  $\mathcal{O}(1/k)$  decay rate, i.e.,  $\alpha_{k} = \alpha/(k+h)$  and  $\beta_{k} = \beta/(k+h) = c_{\alpha,\beta}\alpha/(k+h)$ , where  $c_{\alpha,\beta} \in (0,1)$  is the step size ratio. In the stochastic-game setting, we redefine  $\ell_{\tau} = [1 + (A_{\max}-1)\exp(2/[(1-\gamma)\tau])]^{-1}$ , which, analogous to the matrix-game setting, is a uniform lower bound on the entries of the policies generated by Algorithm 2 (cf. Lemma D.1). We next state our requirement for choosing the step sizes.
Condition 3.1. When using either constant or diminishing step sizes, we choose  $\tau \leq 1 / (1 - \gamma)$  and the stepsize ratio  $c_{\alpha,\beta}$  to satisfy  $c_{\alpha,\beta} \leq \min \left(\frac{1}{60L_p|S|A_{\max}}, \frac{c_\tau\tau(1-\gamma)^2}{34|S|A_{\max}^2}, \frac{c_\tau\ell_\tau^2\tau^3(1-\gamma)^2}{144A_{\max}^2}\right)$ , where  $c_\tau \propto \ell_\tau$  and  $L_p > 0$  are defined in Appendix B.3. In addition, when using  $\alpha_k \equiv \alpha$  and  $\beta_k \equiv \beta$ , we require  $\alpha < 1/c_\tau$  and  $\beta < 1$ , and when using  $\alpha_k = \alpha/(k+h)$  and  $\beta_k = \beta/(k+h)$ , we require  $^2\beta = 4$ ,  $\alpha > 1/c_\tau$ , and  $h > 1$  such that  $\alpha_0 < 1/c_\tau$  and  $\beta_0 < 1$ .
We next state the finite-sample bound of Algorithm 2. For simplicity of presentation, we use  $a \lesssim b$  to mean that there exists an absolute constant  $c > 0$  such that  $a \leq bc$ .
Theorem 3.1. Suppose that both players follow Algorithm 2, Assumption 3.1 is satisfied, and the stepsizes  $\{\alpha_k\}$  and  $\{\beta_k\}$  satisfy Condition 3.1. Then, we have the following results.
(1) When using constant step sizes, there exists  $z_{\beta} = \mathcal{O}(\log (1 / \beta))$  such that the following inequality holds as long as  $K \geq z_{\beta}$ :
$$
\begin{array}{l} \mathbb {E} [ N G (\pi_ {T, K} ^ {1}, \pi_ {T, K} ^ {2}) ] \lesssim \underbrace {\frac {A _ {\operatorname* {m a x}} ^ {2} T}{\tau (1 - \gamma) ^ {3}} \left(\frac {1 + \gamma}{2}\right) ^ {T - 1}} _ {:= \mathcal {E} _ {1}} + \underbrace {\frac {A _ {\operatorname* {m a x}} ^ {2} L _ {i n} (K - z _ {\beta}) ^ {1 / 2}}{\tau (1 - \gamma) ^ {4}} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}}} _ {:= \mathcal {E} _ {2}} \\ + \underbrace {\frac {| \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {4} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha^ {1 / 2}} _ {:= \mathcal {E} _ {3}} + \underbrace {\frac {\tau \log (A _ {\max})}{(1 - \gamma) ^ {2}}} _ {:= \mathcal {E} _ {4}}, \\ \end{array}
$$
$$
w h e r e L _ {i n} = \frac {4}{(1 - \gamma)} + 2 \tau \log (A _ {\mathrm {m a x}}) + \frac {8 | S | A _ {\mathrm {m a x}}}{(1 - \gamma) ^ {2}}.
$$
(2) When using  $\alpha_{k} = \alpha /(k + h)$  and  $\beta_{k} = \beta /(k + h)$ , there exists  $k_0 > 0$  such that the following inequality holds as long as  $K\geq k_0$ :
$$
\mathbb {E} [ N G (\pi_ {T, K} ^ {1}, \pi_ {T, K} ^ {2}) ] \lesssim \frac {A _ {\max} ^ {2} T}{\tau (1 - \gamma) ^ {3}} \left(\frac {1 + \gamma}{2}\right) ^ {T - 1} + \frac {L _ {i n} | \mathcal {S} | A _ {\max} z _ {K} ^ {2} \alpha_ {K} ^ {1 / 2}}{(1 - \gamma) ^ {4} \alpha_ {k _ {0}} ^ {1 / 2} c _ {\alpha , \beta}} + \frac {\tau \log (A _ {\max})}{(1 - \gamma) ^ {2}},
$$
$$
w h e r e z _ {K} = \mathcal {O} (\log (K)).
$$
Remark. Analogous to [29, 15], our learning dynamics are symmetric between the two players in the sense that there is no time-scale separation between the two players, that is, they both implement the algorithm with the same step sizes.
A detailed proof sketch of Theorem 3.1 is provided in Appendix B and the complete proof is provided in Appendix D. Next, we discuss the result in Theorem 3.1 (1). The bound in Theorem 3.1 (1) involves a value iteration error term  $\mathcal{E}_1$ , an optimization error term  $\mathcal{E}_2$ , a statistical error term  $\mathcal{E}_3$ , and a smoothing bias term  $\mathcal{E}_4$  due to the use of smoothed best-response in the learning dynamics. Note that  $\mathcal{E}_1$  would be the only error term if we were able to perform minimax value iteration to solve
2The proof works as long as  $\beta >2$  . We here use  $\beta = 4$  to simplify the statement of the results.
the game. Since minimax value iteration converges geometrically, the term  $\mathcal{E}_1$  also goes to zero at a geometric rate. Notably, the terms  $\mathcal{E}_2$  and  $\mathcal{E}_3$  are orderwise larger compared to their matrix-game counterparts, see Corollary 2.1.2. Intuitively, the reason is that the induced auxiliary matrix game (with payoff matrices  $\mathcal{T}^1(v_t^1)(s)$  and  $\mathcal{T}^2(v_t^2)(s)$ ) that the inner loop of Algorithm 2 aims at solving does not necessarily have a zero-sum structure (see the discussion in Section 3.1 after Algorithm 2). Consequently, the error due to such a "non-zero-sum" structure propagates through the algorithm and eventually undermines the convergence bound.
Recall that in the matrix game setting, we proved convergence to the Nash distribution (or the Nash equilibrium of the entropy-regularized matrix game). In the stochastic-game setting, we do not have convergence to the Nash equilibrium of the entropy-regularized stochastic game. The main reason is that, in order to have such a convergence, our outer loop should be designed to approximate the entropy-regularized minimax value iteration rather than the vanilla minimax value iteration as in Algorithm 2 Line 8. However, in the payoff-based setting, since each player does not even observe the actions of their opponent, it is unclear how to construct an estimator of the entropy function of the opponent's policy, which is an interesting future direction to investigate.
Although the transient terms in Theorem 3.1 enjoy a desirable rate of convergence (e.g., geometric in  $T$  and  $\tilde{\mathcal{O}}(1 / K^{1/2})$  in  $K$ ), the stepsize ratio  $c_{\alpha, \beta}$  (which is exponentially small in  $\tau$ ) appears as  $c_{\alpha, \beta}^{-1}$  in the bound; see Theorem 3.1. Therefore, due to the presence of the smoothing bias (i.e., the term  $\mathcal{E}_4$  on the RHS of the bound in Theorem 3.1 (1)), to achieve  $\mathbb{E}[\mathrm{NG}(\pi_{T,K}^1, \pi_{T,K}^2)] \leq \epsilon$ , the overall sample complexity can also be exponentially large in  $\epsilon^{-1}$ . This is analogous to Corollary 2.1.2 for zero-sum matrix games. As illustrated in detail in Section 2, the reason here is due to the exploration limitation of using softmax as a means for smoothed best response, which we kept without further modification to preserve the naturalness of the learning dynamics. Removing such exponential factors by developing improved exploration strategies is an immediate future direction.