SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Using an union bound over  $t$ , we see that Eq. (7) holds for all  $t \geq t_0$  with probability at least  $1 - \mathcal{O}(\delta)$ .
Part III. Using Lemma 9 with  $f_{t}(x,y)$  and  $x^{\top}Gy$  with domains  $\Omega_t \times \Omega_t$  and  $\Delta_A \times \Delta_A$ , we get that for any  $(x,y) \in \Delta_A \times \Delta_A$ ,
$$
x _ {t} ^ {\star \top} G y - x ^ {\top} G y _ {t} ^ {\star} \leq \mathcal {O} \left(\epsilon_ {t} \ln (A) + \frac {1}{t}\right) = \mathcal {O} \left(\ln (A) t ^ {- k _ {\epsilon}}\right) = \mathcal {O} \left(\ln (A) t ^ {- \frac {1}{8}}\right).
$$
Further using Eq. (7), we get that with probability at least  $1 - 3\delta$ , for any  $t$  and any  $(x,y) \in \Delta_A \times \Delta_A$ ,
$$
\begin{array}{l} x _ {t} ^ {\top} G y - x ^ {\top} G y _ {t} \leq \mathcal {O} \left(\ln (A) t ^ {- \frac {1}{8}} + \| z _ {t} - z _ {t} ^ {\star} \| _ {1}\right) \stackrel {(a)} {=} \mathcal {O} \left(\ln (A) t ^ {- \frac {1}{8}} + \sqrt {\operatorname {K L} \left(z _ {t} ^ {\star} , z _ {t}\right)}\right) \\ = \mathcal {O} \left(\sqrt {A} \ln^ {3 / 2} (A t / \delta) t ^ {- \frac {1}{8}}\right) \\ \end{array}
$$
where  $(a)$  is by Pinsker's inequality. This completes the proof of Theorem 1.
Lemma 13.  $|v_{t}| = \mathcal{O}\left(\ln^{2}(At)t^{-1}\right)$ .
$$
\begin{array}{l} \left| v _ {t} \right| = \left| \mathrm {K L} \left(z _ {t + 1} ^ {\star}, z _ {t + 1}\right) - \mathrm {K L} \left(z _ {t} ^ {\star}, z _ {t + 1}\right) \right| \\ \leq \mathcal {O} \left(\ln (A t) \| z _ {t + 1} ^ {\star} - z _ {t} ^ {\star} \| _ {1}\right) (byLemma14) \\ = \mathcal {O} \left(\ln^ {2} (A t) t ^ {- 1}\right). (by Lemma 15) \\ \end{array}
$$
Lemma 14. Let  $x, x_1, x_2 \in \Omega_t$ . Then
$$
\left| K L \left(x _ {1}, x\right) - K L \left(x _ {2}, x\right) \right| \leq \mathcal {O} \left(\ln (A t) \| x _ {1} - x _ {2} \| _ {1}\right).
$$
$$
\begin{array}{l} \mathrm {K L} (x _ {1}, x) - \mathrm {K L} (x _ {2}, x) \\ = \sum_ {a} \left(x _ {1, a} \ln \frac {x _ {1 , a}}{x _ {a}} - x _ {2, a} \ln \frac {x _ {2 , a}}{x _ {a}}\right) \\ = \sum_ {a} \left(x _ {1, a} - x _ {2, a}\right) \ln \frac {x _ {1 , a}}{x _ {a}} + \sum_ {a} x _ {2, a} \left(\ln \frac {x _ {1 , a}}{x _ {a}} - \ln \frac {x _ {2 , a}}{x _ {a}}\right) \\ \leq \mathcal {O} \left(\ln (A t) \| x _ {1} - x _ {2} \| _ {1}\right) - \mathrm {K L} \left(x _ {2}, x _ {1}\right) \\ \leq \mathcal {O} \left(\ln (A t) \| x _ {1} - x _ {2} \| _ {1}\right). \\ \end{array}
$$
Similarly,  $\mathrm{KL}(x_2,x) - \mathrm{KL}(x_1,x)\leq \mathcal{O}\left(\ln (At)\| x_1 - x_2\| _1\right)$
Lemma 15.  $\| z_t^\star -z_{t + 1}^\star \| _1 = \mathcal{O}\left(\frac{\ln(At)}{t}\right)$
Proof. Notice that the feasible sets for the two time steps are different. Let  $(x_{t+1}', y_{t+1}')$  be such that  $x_{t+1}' = \frac{p_{t+1}}{A} \mathbf{1} + (1 - p_{t+1}) x_{t+1}^{\star}$  and  $y_{t+1}' = \frac{p_{t+1}}{A} \mathbf{1} + (1 - p_{t+1}) y_{t+1}^{\star}$  where  $p_{t+1} = \min \{1, 2t^{-3}\}$ . Since  $(x_{t+1}', y_{t+1}^{\star}) \in \Omega_{t+1} \times \Omega_{t+1}$ , we have that for any  $a$ ,  $x_{t+1,a}^{\prime} \geq \frac{p_{t+1}}{A} + (1 - p_{t+1}) \frac{1}{A(t+1)^2} \geq \frac{1}{At^2}$ . Hence,  $(x_{t+1}', y_{t+1}') \in \Omega_t \times \Omega_t$ .
Because  $(x_{t + 1}^{\star},y_{t + 1}^{\star})$  is the equilibrium of  $f_{t + 1}$  in  $\Omega_{t + 1}\times \Omega_{t + 1}$ , we have that for any  $(x,y)\in \Omega_{t + 1}\times \Omega_{t + 1}$
$$
\begin{array}{l} f _ {t + 1} (x, y _ {t + 1} ^ {\star}) - f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y\right) \\ = f _ {t + 1} \left(x, y _ {t + 1} ^ {\star}\right) - f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y _ {t + 1} ^ {\star}\right) + f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y _ {t + 1} ^ {\star}\right) - f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y\right) \\ \geq \epsilon_ {t + 1} \mathrm {K L} \left(x, x _ {t + 1} ^ {\star}\right) + \epsilon_ {t + 1} \mathrm {K L} \left(y, y _ {t + 1} ^ {\star}\right) \\ \geq \frac {1}{2} \epsilon_ {t + 1} \left(\| x - x _ {t + 1} ^ {\star} \| _ {1} ^ {2} + \| y - y _ {t + 1} ^ {\star} \| _ {1} ^ {2}\right) \quad \text {(P i n s k e r ' s i n e q u a l i t y)} \\ \geq \frac {1}{4} \epsilon_ {t + 1} \| z - z _ {t + 1} ^ {\star} \| _ {1} ^ {2}. \\ \end{array}
$$
where the first inequality is due to the following calculation:
$$
\begin{array}{l} \epsilon_ {t + 1} \mathrm {K L} (x, x _ {t + 1} ^ {\star}) = f _ {t + 1} (x, y _ {t + 1} ^ {\star}) - f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y _ {t + 1} ^ {\star}\right) - \nabla_ {\times} f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y _ {t + 1} ^ {\star}\right) ^ {\top} (x - x _ {t + 1} ^ {\star}) \\ \leq f _ {t + 1} (x, y _ {t + 1} ^ {\star}) - f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y _ {t + 1} ^ {\star}\right) \\ \end{array}
$$
where we use  $\nabla_{x}f_{t + 1}(x_{t + 1}^{\star},y_{t + 1}^{\star})^{\top}(x - x_{t + 1}^{\star})\geq 0$  since  $x_{t + 1}^{\star}$  is the minimizer of  $f_{t + 1}(\cdot ,y_{t + 1}^{\star})$  in  $\Omega_{t + 1}$ . Specially, we have
$$
f _ {t + 1} \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\star}\right) - f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y _ {t} ^ {\star}\right) \geq \frac {1}{4} \epsilon_ {t + 1} \| z _ {t} ^ {\star} - z _ {t + 1} ^ {\star} \| _ {1} ^ {2}. \tag {8}
$$
Similarly, because  $(x_{t}^{\star},y_{t}^{\star})$  is the equilibrium of  $f_{t}$  in  $\Omega_t\times \Omega_t$ , we have
$$
f _ {t} (x _ {t + 1} ^ {\prime}, y _ {t} ^ {\star}) - f _ {t} (x _ {t} ^ {\star}, y _ {t + 1} ^ {\prime}) \geq \frac {1}{4} \epsilon_ {t} \| z _ {t + 1} ^ {\prime} - z _ {t} ^ {\star} \| _ {1} ^ {2},
$$
$$
\begin{array}{l} f _ {t} \left(x _ {t + 1} ^ {\star}, y _ {t} ^ {\star}\right) - f _ {t} \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\star}\right) \\ = f _ {t} \left(x _ {t + 1} ^ {\prime}, y _ {t} ^ {\star}\right) - f _ {t} \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\prime}\right) + f _ {t} \left(x _ {t + 1} ^ {\star}, y _ {t} ^ {\star}\right) - f _ {t} \left(x _ {t + 1} ^ {\prime}, y _ {t} ^ {\star}\right) + f _ {t} \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\prime}\right) - f _ {t} \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\star}\right) \\ \geq \frac {1}{4} \epsilon_ {t} \| z _ {t + 1} ^ {\prime} - z _ {t} ^ {\star} \| _ {1} ^ {2} - \sup  _ {x \in \Omega_ {t + 1}} \| \nabla_ {\mathbf {x}} f _ {t} (x, y _ {t} ^ {\star}) \| _ {\infty} \| x _ {t + 1} ^ {\prime} - x _ {t + 1} ^ {\star} \| _ {1} - \sup  _ {y \in \Omega_ {t + 1}} \| \nabla_ {\mathbf {y}} f _ {t} (x _ {t} ^ {\star}, y) \| _ {\infty} \| y _ {t + 1} ^ {\prime} - y _ {t + 1} ^ {\star} \| _ {1} \\ \geq \frac {1}{8} \epsilon_ {t} \| z _ {t + 1} ^ {\star} - z _ {t} ^ {\star} \| _ {1} ^ {2} - \frac {1}{4} \epsilon_ {t} \| z _ {t + 1} ^ {\prime} - z _ {t + 1} ^ {\star} \| _ {1} ^ {2} - \mathcal {O} \left(\ln (A t) \times \frac {1}{t ^ {3}}\right) \\ \geq \frac {1}{8} \epsilon_ {t} \| z _ {t + 1} ^ {\star} - z _ {t} ^ {\star} \| _ {1} ^ {2} - \mathcal {O} \left(\frac {\ln (A t)}{t ^ {3}}\right). \tag {9} \\ \end{array}
$$
In the first inequality, we use the fact that  $f_{t}(x,y)$  is convex in  $x$  and concave in  $y$  and Hölder's inequality. In the second inequality, we use the triangle inequality,  $\| \nabla_{\mathbf{x}}f_t(x,y)\|_{\infty}\leq \max_a\{(Gy)_a + \ln (x_a)\} \leq \mathcal{O}(\ln (At))$ , and  $\| \nabla_{\mathbf{y}}f_t(x,y)\|_{\infty}\leq \max_b\{(G^\top x)_b + \ln (y_b)\} \leq \mathcal{O}(\ln (At))$ . In the second and third inequality, we use  $\| z_{t + 1}' - z_{t + 1}^{\star}\|_{1} = \mathcal{O}\left(\frac{1}{t^{3}}\right)$  by the definition of  $z_{t + 1}'$ .
Combining Eq. (8) and Eq. (9), we get
$$
\begin{array}{l} \frac {3}{8} \epsilon_ {t + 1} \| z _ {t} ^ {\star} - z _ {t + 1} ^ {\star} \| _ {1} ^ {2} \\ \leq f _ {t + 1} \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\star}\right) - f _ {t} \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\star}\right) - f _ {t + 1} \left(x _ {t + 1} ^ {\star}, y _ {t} ^ {\star}\right) + f _ {t} \left(x _ {t + 1} ^ {\star}, y _ {t} ^ {\star}\right) + \mathcal {O} \left(\frac {\ln (A t)}{t ^ {3}}\right) \\ = \left(f _ {t + 1} - f _ {t}\right) \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\star}\right) - \left(f _ {t + 1} - f _ {t}\right) \left(x _ {t + 1} ^ {\star}, y _ {t} ^ {\star}\right) + \mathcal {O} \left(\frac {\ln (A t)}{t ^ {3}}\right) \\ \leq \sup  _ {x, y \in \Omega_ {t + 1} \times \Omega_ {t + 1}} \| \nabla f _ {t + 1} (x, y) - \nabla f _ {t} (x, y) \| _ {\infty} \| \left(x _ {t} ^ {\star}, y _ {t + 1} ^ {\star}\right) - \left(x _ {t + 1} ^ {\star}, y _ {t} ^ {\star}\right) \| _ {1} + \mathcal {O} \left(\frac {\ln (A t)}{t ^ {3}}\right) \\ \end{array}
$$
$$
= \sup  _ {x, y \in \Omega_ {t + 1} \times \Omega_ {t + 1}} \| \nabla f _ {t + 1} (x, y) - \nabla f _ {t} (x, y) \| _ {\infty} \| z _ {t} ^ {\star} - z _ {t + 1} ^ {\star} \| _ {1} + \mathcal {O} \left(\frac {\ln (A t)}{t ^ {3}}\right)
$$
Solving the inequality, we get
$$
\begin{array}{l} \left\| z _ {t} ^ {\star} - z _ {t + 1} ^ {\star} \right\| _ {1} \leq \mathcal {O} \left(\frac {1}{\epsilon_ {t + 1}} \sup  _ {x, y \in \Omega_ {t + 1} \times \Omega_ {t + 1}} \| \nabla f _ {t + 1} (x, y) - \nabla f _ {t} (x, y) \| _ {\infty} + \frac {\ln^ {1 / 2} (A t)}{\sqrt {\epsilon_ {t + 1}} t ^ {3 / 2}}\right) \tag {10} \\ \leq \mathcal {O} \left(\frac {\left(\epsilon_ {t} - \epsilon_ {t + 1}\right) \ln (A t)}{\epsilon_ {t + 1}} + \frac {\ln^ {1 / 2} (A t)}{\sqrt {\epsilon_ {t + 1}} t ^ {3 / 2}}\right) \\ = \mathcal {O} \left(\frac {t ^ {- k _ {\epsilon} - 1} \ln (A t)}{t ^ {- k _ {\epsilon}}} + \frac {\ln^ {1 / 2} (A t)}{\sqrt {\epsilon_ {t + 1}} t ^ {3 / 2}}\right) \\ = \mathcal {O} \left(\frac {\ln (A t)}{t}\right). \\ \end{array}
$$
C Improved Last-Iterate Convergence under Expectation
In this section, we analyze Algorithm 4, which is almost identical to Algorithm 1 but does not involve the parameter  $\beta_{t}$ . The choices of stepsize  $\eta_t$  and amount of regularization  $\epsilon_t$  are also tuned differently to obtain the best convergence rate.
Algorithm 4 Matrix Game with Bandit Feedback
1: Define:  $\eta_t = t^{-k_\eta}$ ,  $\epsilon_t = t^{-k_\epsilon}$  where  $k_\eta = \frac{1}{2}$ ,  $k_\epsilon = \frac{1}{6}$ .
2: Initialization::  $x_{1} = \frac{1}{A}\mathbf{1}$
3: for  $t = 1,2,\ldots$  do
4: Sample  $a_{t} \sim x_{t}$ , and receive  $\sigma_t \in [0,1]$  with  $\mathbb{E}[\sigma_t] = G_{a_t,b_t}$ .
5: Compute  $g_{t}$  where  $g_{t,a} = \frac{\mathbf{1}[a_t = a]\sigma_t}{x_{t,a}} +\epsilon_t\ln x_{t,a},\forall a\in \mathcal{A}.$
6: Update  $x_{t + 1}\gets \mathrm{argmin}_{x\in \Omega_{t + 1}}\left\{x^{\top}g_{t} + \frac{1}{\eta_{t}}\mathrm{KL}(x,x_{t})\right\}$
7: end for
Theorem 4. Algorithm 4 guarantees  $\mathbb{E}\left[\max_{x,y\in \Delta_A}(x_t^\top Gy - x^\top Gy_t)\right] = \mathcal{O}\left(\sqrt{A}\ln^{3 / 2}(At)t^{-\frac{1}{6}}\right)$  for any  $t$
Proof. With the same analysis as in Part I of the proof of Theorem 1, we have
$$
\begin{array}{l} f _ {t} \left(x _ {t}, y _ {t}\right) - f _ {t} \left(x _ {t} ^ {\star}, y _ {t}\right) \\ \leq \frac {\left(1 - \eta_ {t} \epsilon_ {t}\right) \mathrm {K L} \left(x _ {t} ^ {\star} , x _ {t}\right) - \mathrm {K L} \left(x _ {t} ^ {\star} , x _ {t + 1}\right)}{\eta_ {t}} + 1 0 \eta_ {t} A \ln^ {2} (A t) + 2 \eta_ {t} A \underline {{\lambda}} _ {t} + \underline {{\xi}} _ {t} + \underline {{\zeta}} _ {t}. \\ \end{array}
$$
$$
\underline {{\xi}} _ {t} \triangleq \sum_ {a} x _ {t, a} \left((G y _ {t}) _ {a} - \frac {\mathbf {1} [ a _ {t} = a ] \sigma_ {t}}{x _ {t , a}}\right), \quad \zeta_ {t} \triangleq \sum_ {a} x _ {t, a} ^ {\star} \left(\frac {\mathbf {1} [ a _ {t} = a ] \sigma_ {t}}{x _ {t , a}} - (G y _ {t}) _ {a}\right),
$$
$$
\underline {{\lambda}} _ {t} \triangleq \frac {1}{A} \sum_ {a} \left(\frac {\mathbf {1} [ a _ {t} = a ]}{x _ {t , a}} - 1\right).
$$
Unlike in Theorem 1, here these three terms all have zero mean. Thus, following the same arguments that obtain Eq. (6) and taking expectations, we get
$$
\mathbb {E} _ {t} \left[ \mathrm {K L} \left(z _ {t + 1} ^ {\star}, z _ {t + 1}\right) \right] \leq \left(1 - \eta_ {t} \epsilon_ {t}\right) \mathrm {K L} \left(z _ {t} ^ {\star}, z _ {t}\right) + 2 0 \eta_ {t} ^ {2} A \ln^ {2} (A t) + \mathbb {E} _ {t} [ v _ {t} ]
$$
$$
\leq \left(1 - \eta_ {t} \epsilon_ {t}\right) \mathrm {K L} \left(z _ {t} ^ {\star}, z _ {t}\right) + \mathcal {O} \left(\eta_ {t} ^ {2} A \ln^ {2} (A t) + \frac {\ln^ {2} (A t)}{t}\right) \quad (\text {b y})
$$
where  $v_{t} = \mathrm{KL}(z_{t + 1}^{\star}, z_{t + 1}) - \mathrm{KL}(z_{t}^{\star}, z_{t + 1})$  and  $\mathbb{E}_t[\cdot]$  is the expectation conditioned on history up to round  $t$ . Then following the same arguments as in Part II of the proof of Theorem 1, we get
$$
\begin{array}{l} \mathbb {E} \left[ \mathrm {K L} \left(z _ {t + 1} ^ {\star}, z _ {t + 1}\right) \right] \leq \mathcal {O} \left(A \ln^ {2} (A t) \sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} ^ {2} + \ln^ {2} (A t) \sum_ {i = 1} ^ {t} w _ {t} ^ {i} t ^ {- 1}\right) \\ (\text {d e f i n e} w _ {t} ^ {i} \triangleq \prod_ {j = i + 1} ^ {t} (1 - \eta_ {j} \epsilon_ {j})) \\ \leq \mathcal {O} \left(A \ln^ {3} (A t) t ^ {- k _ {\eta} + k _ {\epsilon}} + \ln^ {3} (A t) t ^ {- 1 + k _ {\eta} + k _ {\epsilon}}\right) = \mathcal {O} \left(A \ln^ {3} (A t) t ^ {- \frac {1}{3}}\right). \\ \end{array}
$$
Finally, following the arguments in Part III, we get
$$
\mathbb {E} \left[ \max _ {x, y} \left(x _ {t} ^ {\top} G y - x ^ {\top} G y _ {t}\right) \right] \leq \mathcal {O} \left(\ln (A) t ^ {- k _ {\epsilon}} + \sqrt {\mathbb {E} \left[ \mathrm {K L} (z _ {t} ^ {\star} , z _ {t}) \right]}\right) = \mathcal {O} \left(\sqrt {A} \ln^ {3 / 2} (A t) t ^ {- \frac {1}{6}}\right).
$$
D Last-Iterate Convergence Rate of Algorithm 2
D.1 On the Assumption of Irreducible Markov Game
Proposition 1. If Assumption 1 holds, then for any  $L' = 2L\log_2(S / \delta)$  consecutive steps, under any (non-stationary) policies of the two players, with probability at least  $1 - \delta$ , every state is visited at least once.
Proof. We first show that for any pair of states  $s', s''$ , under any non-stationary policy pair, the expected time to reach  $s''$  from  $s'$  is upper bounded by  $L$ . For a particular pair of states  $(s', s'')$ , consider the following modified MDP: let the reward be  $r(s, a) = 1 [s \neq s'']$ , and the transition be the same as the original MDP on all  $s \neq s''$ , while  $P(s''|s'', a) = 1$  (i.e., making  $s''$  an absorbing state). Also, let  $s'$  be the initial state. By construction, the expected total reward of this MDP is the travelling time from  $s'$  to  $s''$ . By Theorem 7.1.9 of [Put14], there exists a stationary optimal policy in this MDP. The optimal expected total value is then upper bounded by  $L$  by Assumption 1. Therefore, for any (possibly sub-optimal) non-stationary policies, the travelling time from  $s'$  to  $s''$  must also be upper bounded by  $L$ .
Divide  $L'$  steps into  $\log_2(S / \delta)$  intervals each of length  $2L$ , and consider a particular  $s$ . Conditioned on  $s$  not visited in all intervals  $1, 2, \ldots, i - 1$ , the probability of still not visiting  $s$  in interval  $i$  is smaller than  $\frac{1}{2}$  (because for any  $s'$ ,  $\operatorname*{Pr}[T_{s' \to s} > 2L] \leq \frac{\mathbb{E}[T_{s' \to s}]}{2L} \leq \frac{L}{2L} = \frac{1}{2}$ , where  $T_{s' \to s}$  denotes the travelling time from  $s'$  to  $s$ ). Therefore, the probability of not visiting  $s$  in all  $\log_2(S / \delta)$  intervals is upper bounded by  $2^{-\log_2(S / \delta)} = \frac{\delta}{S}$ . Using a union bound, we conclude that with probability at least  $1 - \delta$ , every state is visited at least once within  $L'$  steps.
Corollary 1. If Assumption 1 holds, then with probability  $1 - \delta$ , for any  $t \geq 1$ , players visit every state at least once in every  $6L\ln (St / \delta)$  consecutive iterations before time  $t$ .
Proof. First, we fix time  $t \geq 1$  and define  $t' = 3L\ln (St^3 /\delta)$ . Let us consider the following time intervals:  $[1,t'],[t',2t'],\ldots ,[t - t',t]$ . Using Proposition 1, we know for each interval, with probability at least  $1 - \frac{\delta}{t^3}$ , players visit every state  $s$ . Using a union bound over all intervals, we have with probability at least  $1 - \frac{\delta}{t^2}$ , in every interval, players visit every state  $s$ . Since every  $2t'$  consecutive iterations must contain an interval of length  $L'$ , we have with probability at least  $1 - \frac{\delta}{t^2}$ , players visit every state  $s$  in every  $2t'$  consecutive iterations until time  $t$ . Applying union bound over all  $t \geq 1$  completes the proof.
According to Corollary 1, in the remaining of this section, we assume that for any  $t \geq 1$ , players visit every state at least once in every  $6L\ln (St / \delta)$  iterations until time  $t$ .
D.2 Part I. Basic Iteration Properties
Lemma 16. For any  $x^{s}\in \Omega_{\tau +1}$
$$
\begin{array}{l} f _ {\tau} ^ {s} \left(\hat {x} _ {\tau} ^ {s}, \hat {y} _ {\tau} ^ {s}\right) - f _ {\tau} ^ {s} \left(x ^ {s}, \hat {y} _ {\tau} ^ {s}\right) \\ \leq \frac {(1 - \eta_ {\tau} \epsilon_ {\tau}) K L (x ^ {s} , \hat {x} _ {\tau} ^ {s}) - K L (x ^ {s} , \hat {x} _ {\tau + 1} ^ {s})}{\eta_ {\tau}} + \frac {1 0 \eta_ {\tau} A \ln^ {2} (A \tau)}{(1 - \gamma) ^ {2}} + \frac {2 \eta_ {\tau} A}{(1 - \gamma) ^ {2}} \underline {{\lambda}} _ {\tau} ^ {s} + \underline {{\xi}} _ {\tau} ^ {s} + \underline {{\zeta}} _ {\tau} ^ {s} (x ^ {s}). \\ \end{array}
$$
(see the proof for the definitions of  $\underline{\lambda}_{\tau}^{s},\underline{\xi}_{\tau}^{s},\underline{\zeta}_{\tau}^{s}(\cdot))$
Proof. Consider a fixed  $s$  and a fixed  $\tau$ , and let  $t = t_{\tau}(s)$  be the time when the players visit  $s$  at the  $\tau$ -th time.
$$
\begin{array}{l} f _ {\tau} ^ {s} \left(\hat {x} _ {\tau} ^ {s}, \hat {y} _ {\tau} ^ {s}\right) - f _ {\tau} ^ {s} \left(x ^ {s}, \hat {y} _ {\tau} ^ {s}\right) \\ = \left(\hat {x} _ {\tau} ^ {s} - x ^ {s}\right) ^ {\top} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t} ^ {s ^ {\prime}} \right]\right) \hat {y} _ {\tau} ^ {s} - \epsilon_ {\tau} \phi \left(\hat {x} _ {\tau} ^ {s}\right) + \epsilon_ {\tau} \phi \left(x ^ {s}\right) \\ = \left(\hat {x} _ {\tau} ^ {s} - x ^ {s}\right) ^ {\top} \left[ \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t} ^ {s ^ {\prime}} \right]\right) \hat {y} _ {\tau} ^ {s} + \epsilon_ {\tau} \ln \hat {x} _ {\tau} ^ {s} \right] - \epsilon_ {\tau} \mathrm {K L} \left(x ^ {s}, \hat {x} _ {\tau} ^ {s}\right) \\ = \left(\hat {x} _ {\tau} ^ {s} - x ^ {s}\right) ^ {\top} g _ {t} - \epsilon_ {\tau} \mathrm {K L} \left(x ^ {s}, \hat {x} _ {\tau} ^ {s}\right) + \underbrace {\left(\hat {x} _ {\tau} ^ {s}\right) ^ {\top} \left(\left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t} ^ {s ^ {\prime}} \right]\right) \hat {y} _ {\tau} ^ {s} - \frac {\mathbf {1} \left[ \hat {a} _ {\tau} ^ {s} = a \right] \left(\sigma_ {t} + \gamma V _ {t} ^ {s _ {t + 1}}\right)}{\hat {x} _ {\tau , a} ^ {s} + \beta_ {\tau}}\right)} _ {\underline {{\xi}} _ {\tau} ^ {s}} \\ + \underbrace {\left(x ^ {s}\right) ^ {\top} \left(\frac {\mathbf {1} \left[ \hat {a} _ {\tau} ^ {s} = a \right] \left(\sigma_ {t} + \gamma V _ {t} ^ {s _ {t + 1}}\right)}{\hat {x} _ {\tau , a} ^ {s} + \beta_ {\tau}} - \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t} ^ {s ^ {\prime}} \right]\right) \hat {y} _ {\tau} ^ {s}\right)} _ {\underline {{\zeta}} _ {\tau} ^ {s} (x ^ {s})} \\ \leq \frac {\left(1 - \eta_ {\tau} \epsilon_ {\tau}\right) \mathrm {K L} \left(x ^ {s} , \hat {x} _ {\tau} ^ {s}\right) - \mathrm {K L} \left(x ^ {s} , \hat {x} _ {\tau + 1} ^ {s}\right)}{\eta_ {\tau}} \\ + \frac {1 0 \eta_ {\tau} A \ln^ {2} (A \tau)}{(1 - \gamma) ^ {2}} + \frac {2 \eta_ {\tau} A}{(1 - \gamma) ^ {2}} \times \underbrace {\frac {1}{| \mathcal {A} |} \sum_ {a} \left(\frac {\mathbf {1} [ \hat {a} _ {\tau} ^ {s} = a ]}{\hat {x} _ {\tau , a} ^ {s} + \beta_ {\tau}} - 1\right)} _ {\underline {{\lambda}} _ {\tau} ^ {s}} + \underline {{\xi}} _ {\tau} ^ {s} + \underline {{\zeta}} _ {\tau} ^ {s} (x ^ {s}), \\ \end{array}
$$
where we omit some calculation steps due to the similarity to Eq. (5).
D.3 Part II. Policy Convergence to the Nash of Regularized Game
Lemma 17. With probability at least  $1 - \mathcal{O}(\delta)$ , for all  $s\in S$ ,  $t\geq 1$  and  $\tau \geq 1$  such that  $t_\tau (s)\leq t$ , we have
$$
K L \left(\hat {z} _ {\tau \star} ^ {s}, \hat {z} _ {\tau} ^ {s}\right) \leq \mathcal {O} \left(A \ln^ {5} (S A t / \delta) L ^ {2} \tau^ {- k _ {\sharp}}\right),
$$
where  $k_{\sharp} = \min \{k_{\beta} - k_{\epsilon},k_{\eta} - k_{\beta},k_{\alpha} - k_{\eta} - 2k_{\epsilon}\}$
Proof. In this proof, we abbreviate  $\underline{\zeta}_i^s (\hat{x}_{i\star}^s)$  as  $\underline{\zeta}_i^s$ . By Lemma 16, for all  $i\leq \tau$  we have
$$
\begin{array}{l} \mathrm {K L} \left(\hat {x} _ {i \star} ^ {s}, \hat {x} _ {i + 1} ^ {s}\right) \leq \left(1 - \eta_ {i} \epsilon_ {i}\right) \mathrm {K L} \left(\hat {x} _ {i \star} ^ {s}, \hat {x} _ {i} ^ {s}\right) + \eta_ {i} \left(f _ {i} ^ {s} \left(\hat {x} _ {i \star} ^ {s}, \hat {y} _ {i} ^ {s}\right) - f _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right)\right) \\ + \frac {1 0 \eta_ {i} ^ {2} A \ln^ {2} (A \tau)}{(1 - \gamma) ^ {2}} + \frac {2 \eta_ {i} ^ {2} A}{(1 - \gamma) ^ {2}} \underline {{\lambda}} _ {i} ^ {s} + \eta_ {i} \underline {{\xi}} _ {i} ^ {s} + \eta_ {i} \underline {{\zeta}} _ {i} ^ {s}. \\ \end{array}
$$
Similarly, for all  $i \leq \tau$ , we have
$$
\begin{array}{l} \mathrm {K L} \left(\hat {y} _ {i \star} ^ {s}, \hat {y} _ {i + 1} ^ {s}\right) \leq \left(1 - \eta_ {i} \epsilon_ {i}\right) \mathrm {K L} \left(\hat {y} _ {i \star} ^ {s}, \hat {y} _ {i} ^ {s}\right) + \eta_ {i} \left(f _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) - f _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i \star} ^ {s}\right)\right) \\ + \frac {1 0 \eta_ {i} ^ {2} A \ln^ {2} (A \tau)}{(1 - \gamma) ^ {2}} + \frac {2 \eta_ {i} ^ {2} A}{(1 - \gamma) ^ {2}} \overline {{\lambda}} _ {i} ^ {s} + \eta_ {i} \overline {{\xi}} _ {i} ^ {s} + \eta_ {i} \overline {{\zeta}} _ {i} ^ {s}. \\ \end{array}
$$
Adding the two inequalities up, and using  $f_{i}^{s}(\hat{x}_{i\star}^{s}, \hat{y}_{i}^{s}) - f_{i}^{s}(\hat{x}_{i}^{s}, \hat{y}_{i\star}^{s}) \leq 0$  because  $(\hat{x}_{i\star}^{s}, \hat{y}_{i\star}^{s})$  is the equilibrium of  $f_{i}^{s}$ , we get for  $i \leq \tau$
$$
\mathrm {K L} \left(\hat {z} _ {i + 1 \star} ^ {s}, \hat {z} _ {i + 1} ^ {s}\right) \leq \left(1 - \eta_ {i} \epsilon_ {i}\right) \mathrm {K L} \left(\hat {z} _ {i \star} ^ {s}, \hat {z} _ {i} ^ {s}\right) + \frac {2 0 \eta_ {i} ^ {2} A \ln^ {2} (A \tau)}{(1 - \gamma) ^ {2}} + \frac {2 \eta_ {i} ^ {2} A}{(1 - \gamma) ^ {2}} \lambda_ {i} ^ {s} + \eta_ {i} \xi_ {i} ^ {s} + \eta_ {i} \zeta_ {i} ^ {s} + v _ {i} ^ {s}, \tag {11}
$$
where  $v_{i}^{s} = \mathrm{KL}(\hat{z}_{i + 1\star}^{s},\hat{z}_{i + 1}) - \mathrm{KL}(\hat{z}_{i\star}^{s},\hat{z}_{i + 1})$  and  $\square^s = \square^s +\overline{\square}^s$  for  $\square = \xi_i,\zeta_i$ .
Expanding Eq. (11), we get
$$
\mathrm{KL}(\hat{z}_{\tau +1\star}^{s},\hat{z}_{\tau +1}^{s})\leq \underbrace{\frac{20A\ln^{2}(A\tau)}{(1 - \gamma)^{2}}\sum_{i = 1}^{\tau}w_{\tau}^{i}\eta_{i}^{2}}_{\text{\bf term}_{1}} + \underbrace{\frac{2A}{(1 - \gamma)^{2}}\sum_{i = 1}^{\tau}w_{\tau}^{i}\eta_{i}^{2}\lambda_{i}^{s}}_{\text{\bf term}_{2}} + \underbrace{\sum_{i = 1}^{\tau}w_{\tau}^{i}\eta_{i}\xi_{i}^{s}}_{\text{\bf term}_{3}} + \underbrace{\sum_{i = 1}^{t}w_{\tau}^{i}\eta_{i}\zeta_{i}^{s}}_{\text{\bf term}_{4}} + \underbrace{\sum_{i = 1}^{\tau}w_{\tau}^{i}v_{i}^{s}}_{\text{\bf term}_{5}}.
$$
These five terms correspond to those in Eq. (6), and can be handled in the same way. For  $\mathbf{term}_1$  to  $\mathbf{term}_4$ , we follow exactly the same arguments there, and bound their sum as with probability at least  $1 - \mathcal{O}\left(\frac{\delta}{S\tau^2}\right)$
$$
\sum_ {j = 1} ^ {4} \mathbf {t e r m} _ {j} = \mathcal {O} \left(A \ln^ {3} (S A \tau / \delta) \left(\tau^ {- k _ {\eta} + k _ {\epsilon}} + \tau^ {- 2 k _ {\eta} + k _ {\beta}} + \tau^ {- k _ {\beta} + k _ {\epsilon}} + \tau^ {- \frac {1}{2} k _ {\eta} + \frac {1}{2} k _ {\epsilon}} + \tau^ {- k _ {\eta} + k _ {\beta}}\right)\right).
$$