SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
N _ {5} \leq \frac {1 6 L _ {p} | \mathcal {S} | A _ {\max} \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {4 \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}}.
$$
Combining the upper bounds we derived for the terms  $N_{1}$  to  $N_{5}$  in Eq. (56), we have
$$
\begin{array}{l} \mathbb {E} \left[ \left(F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right) ^ {\top} \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) \right] \right. \\ \leq \frac {3 6 L _ {p} | \mathcal {S} | A _ {\max z _ {k}} \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {8 \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {8 \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {4 \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} \\ + \frac {1 6 L _ {p} | \mathcal {S} | A _ {\max} \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {4 \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} \\ \leq \frac {6 0 L _ {p} | S | A _ {\max } z _ {k} \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {1 6 \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} \\ \leq \frac {1 7 z _ {k} \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}}, \\ \end{array}
$$
where the last line follows from  $\beta_{k} / \alpha_{k} = c_{\alpha ,\beta}\leq \frac{1}{60L_{p}|S|A_{\max}}$  (cf. Condition 3.1).
D.8 Proof of Corollary 3.1.1
The following proof idea was previous used in [15] to show the rationality of their decentralized  $Q$ -learning algorithm.
Observe that Theorem 3.1 can be easily generalized to the case where the reward is corrupted by noise. Specifically, suppose that player  $i$  takes action  $a^i$  and player  $-i$  takes action  $a^{-i}$ . Instead of assuming player  $i$  receives a deterministic reward  $R_{i}(s,a^{i},a^{-i})$ , we assume that player  $i$  receives a random reward  $r^i (s,a^i,a^{-i},\xi)$ , where  $\xi \in \Xi$  ( $\Xi$  is a finite set) is a random variable with distribution  $\mu_{\xi}(s)$ , and is independent of everything else. The proof is identical as long as  $r^i +r^{-i} = 0$ , and the reward is uniformly bounded, i.e.,  $\max_{s,a^i,a^{-i},\xi}|r^i (s,a^i,a^{-i},\xi)| < \infty$ . Now consider the case where player  $i$ 's opponent follows a stationary policy  $\pi^{-i}$ . We incorporate the randomness of player  $-i$ 's action into the model and introduce a fictitious opponent with only one action  $a^*$ . In particular, let the random reward function be defined as  $\hat{r}^i (s,a^i,a^*,A^{-i}) = R_i(s,a^i,A^{-i})$  for all  $(s,a^i)$ , where  $A^{-i}\sim \pi^{-i}(\cdot |s)$ , and let  $\hat{p} (s' | s,a^i,a^*) = \sum_{\pi^{-i}(a^{-i}|s)}p(s' | a^i,a^{-i},s)$ . Now the problem can be reformulated as player  $i$  playing against the fictitious player with a single action  $a^*$ , with reward function  $\hat{r}^i$  ( $i\in \{1,2\}$ ) and transition probabilities  $\hat{p}$ . Using the same proof for Theorem 3.1, we have the desired finite-sample bound.
E On the Mixing Time of MDPs with Almost Deterministic Policies
Consider an MDP with two states  $s_1, s_2$  and two actions  $a_1, a_2$ . The transition probability matrix  $P_1$  of taking action  $a_1$  is the identity matrix  $I_2$ , and the transition probability matrix  $P_2$  of taking action  $a_2$  is  $P_2 = [0, 1; 1, 0]$ . Given  $\alpha \in (1/2, 1)$ , let  $\pi_\alpha$  be a policy such that  $\pi(a_1|s) = \alpha$  and  $\pi(a_2|s) = 1 - \alpha$  for any  $s \in \{s_1, s_2\}$ . Denote  $P_\alpha$  as the transition probability matrix under  $\pi_\alpha$ . It is easy to see that
$$
P _ {\alpha} = \left[ \begin{array}{c c} \alpha & 1 - \alpha \\ 1 - \alpha & \alpha \end{array} \right].
$$
Since  $P_{\alpha}$  is a doubly stochastic matrix, and has strictly positive entries, it has a unique stationary distribution  $\mu = \mathbf{1}^{\top} / 2$ .
We next compute a lower bound of the mixing time of the  $\pi_{\alpha}$ -induced Markov chain. Let  $e_1 = [1,0]^\top$  be the initial distribution of the states, and denote  $[x_k, 1 - x_k]^\top$  as the distribution of the states at time step  $k$ . Then we have
$$
x _ {k + 1} = x _ {k} \alpha + (1 - x _ {k}) (1 - \alpha)
$$
$$
\begin{array}{l} = (2 \alpha - 1) x _ {k} + 1 - \alpha \\ = (2 \alpha - 1) ^ {k + 1} x _ {0} + \sum_ {i = 0} ^ {k} (1 - \alpha) (2 \alpha - 1) ^ {k - i} \\ = \frac {1}{2} + \frac {(2 \alpha - 1) ^ {k + 1}}{2}. \\ \end{array}
$$
$$
\begin{array}{l} t _ {\pi_ {\alpha}, \eta} = \min  _ {k \geq 0} \left\{\max  _ {\mu_ {0} \in \Delta^ {2}} \left\| \mu_ {0} ^ {\top} P _ {\alpha} ^ {k} - \mathbf {1} ^ {\top} / 2 \right\| _ {\mathrm {T V}} \leq \eta \right\} \\ \geq \min  _ {k > 0} \left\{\left\| e _ {1} ^ {\top} P _ {\alpha} ^ {k} - \mathbf {1} ^ {\top} / 2 \right\| _ {\mathrm {T V}} \leq \eta \right\} \\ = \min  _ {k \geq 0} \left\{\left(2 \alpha - 1\right) ^ {k} \leq 2 \eta \right\} \\ \geq \frac {\log (1 / 2 \eta)}{\log (1 / (2 \alpha - 1))} - 1, \\ \end{array}
$$
which implies  $\lim_{\alpha \to 1}t_{\alpha ,\eta} = \infty$ . Therefore, as the policies become deterministic, the mixing time of the associated Markov chain can approach infinity.
F Numerical Simulations
We first conduct numerical simulations to investigate the impact of choosing different  $\tau$ , which is used to define the softmax operator in Algorithms 1 and 2. Our theoretical results indicate that there is an asymptotically non-vanishing bias due to using a positive  $\tau$ . Intuitively, since a softmax policy always has strictly positive entries while a Nash equilibrium policy can have zero entries, we cannot, in general, expect the Nash gap to converge to zero.
To demonstrate this phenomenon, consider the following example of a zero-sum matrix game. Let
$$
R _ {1} = \left[ \begin{array}{c c c} N & 1 & - 1 \\ - 1 & 0 & 1 \\ 1 & - 1 & 0 \end{array} \right]
$$
be the payoff matrix for player 1, and let  $R_{2} = -(R_{1})^{\top}$ , where  $N > 0$  is a tunable parameter. Note that this matrix game has a unique Nash equilibrium, which goes to the joint policy  $\pi^{1} = (1/3, 2/3, 0)$ ,  $\pi^{2} = (0, 2/3, 1/3)$  as  $N \to \infty$ . In our simulations, we use constant stepsizes  $\alpha_{k} \equiv 0, 5$  and  $\beta_{k} \equiv 0.01$  and run Algorithm 1 for 100 trajectories (each has  $K = 2000$  iterations). Then, we plot the average Nash gap (averaged over the 100 trajectories) as a function of the number of iterations  $k$  in Figure 1 for different temperatures  $\tau$ . To enable a fair comparison, we use the normalized  $q$ -function to compute the softmax, that is, instead of directly using  $\sigma_{\tau}(q_k^i)$  in Algorithm 1, we use  $\sigma_{\tau}(q_k^i / \|q_k^i\|_2)$ . As we can see in Figure 1, as  $\tau$  increases, the asymptotic error also increases, which is consistent with our theoretical results.
F.1 Comparison with the Optimistic Multiplicative-Weights Update
The Optimistic Multiplicative-Weights Update (OMWU) was recognized as a popular learning algorithm for zero-sum matrix games [98]. Since OMWU in the payoff-based setting (or noisy-feedback setting) may not have last-iterate convergence [99], to enable a fair comparison, we will compare OMWU in the noiseless setting (which does enjoy last-iterate convergence [98]) with smoothed-best response dynamics. We start by writing down the algorithm.
OMWU: With initializations  $\pi_0^1, \pi_1^1$  (respectively,  $\pi_0^2, \pi_1^2$ ) that live in the interior of the probability simplex  $\Delta(\mathcal{A}^1)$  (respectively,  $\Delta(\mathcal{A}^2)$ ), OMWU updates  $(\pi_k^1, \pi_k^2)$  iteratively according to
$$
\pi_ {k + 1} ^ {i} (a ^ {i}) = \frac {\pi_ {k} ^ {i} (a ^ {i}) \exp (2 \eta [ R _ {i} \pi_ {k} ^ {- i} ] (a ^ {i}) - \eta [ R _ {i} \pi_ {k - 1} ^ {- i} (a ^ {i}) ])}{\sum_ {\tilde {a} ^ {i} \in \mathcal {A} ^ {i}} \pi_ {k} ^ {i} (\tilde {a} ^ {i}) \exp (2 \eta [ R _ {i} \pi_ {k} ^ {- i} ] (\tilde {a} ^ {i}) - \eta [ R _ {i} \pi_ {k - 1} ^ {- i} (\tilde {a} ^ {i}) ])}, \forall a ^ {i} \in \mathcal {A} ^ {i}, i \in \{1, 2 \},
$$
where  $\eta \in (0,1)$  is the stepsize.
Figure 1: The Nash Gap for Different Temperatures  $\tau$
Figure 2: The Nash Gap as a Function of the Number of Iterations  $k$
The Discrete Smoothed Best-Response Dynamics (DSBR): With arbitrary initializations  $\pi_0^1 \in \Delta(\mathcal{A}^1)$  and  $\pi_0^2 \in \Delta(\mathcal{A}^2)$ , the discrete smoothed best-response dynamics update  $(\pi_k^1, \pi_k^2)$  iteratively according to
$$
\pi_ {k + 1} ^ {i} = \left(1 - \beta_ {k}\right) \pi_ {k} ^ {i} + \beta_ {k} \sigma_ {\tau} \left(R _ {i} \pi_ {k} ^ {- i}\right), \quad \forall i \in \{1, 2 \},
$$
where  $\beta_{k}$  is the stepsize.
We perform two sets of numerical simulations to compare OMWU and DSBR. Our first experiment is implemented on the rock-paper-scissor game, where the payoff matrix for player 1 is
$$
R _ {1} = \left[ \begin{array}{c c c} 0 & 1 & - 1 \\ - 1 & 0 & 1 \\ 1 & - 1 & 0 \end{array} \right],
$$
and  $R_{2} = -(R_{1})^{\top}$ . As we see in Figure 2, the convergence rates of OMWU and DSBR are comparable. However, DSBR seems to be more stable compared with OMWU. Note that while we use softmax policies in DSBR, since the rock-paper-scissor game has a unique Nash equilibrium, which is also the unique Nash equilibrium of the entropy-regularized matrix game for any temperature  $\tau > 0$ , there is no smoothing bias and the Nash gap under DSBR does converge to zero.
Figure 3: The Nash Gap as a Function of the Number of Iterations  $k$
Figure 4: The Asymptotic Behavior of Figure 3
In our second numerical simulation, we set the payoff matrix of player 1 to be
$$
R _ {1} = \left[ \begin{array}{c c c} N & 1 & - 1 \\ - 1 & 0 & 1 \\ 1 & - 1 & 0 \end{array} \right],
$$
and  $R_{2} = -(R_{1})^{\top}$ , where we choose  $N = 100$ . Note that as  $N \to \infty$ , the unique Nash equilibrium goes to  $\pi^{1} = (1/3, 2/3, 0)$ ,  $\pi^{2} = (0, 2/3, 1/3)$ . In this case, we also see from Figure 3 that DSBR is more stable compared with OMWU. However, since in this case, the Nash equilibrium has zero entries, due to the use of softmax policies, DSBR suffers from an asymptotically non-vanishing bias. This is clear from Figure 4, which plots the asymptotic behavior of Figure 3. We see that OMWU converges to zero while the Nash gap from DSBR converges to a positive real number.