SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k + 1} ^ {i} - R _ {i} \pi_ {k + 1} ^ {- i} \| _ {2} ^ {2} ] \\ \leq \left(1 - 2 \ell_ {\tau} \alpha_ {k} + c _ {2} \beta_ {k} + \frac {2 A _ {\operatorname* {m a x}} ^ {2}}{\tau^ {2}} \left(\beta_ {k} ^ {2} + \frac {\beta_ {k}}{c _ {2}} + \alpha_ {k} \beta_ {k} c _ {1}\right)\right) \sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] \\ + \frac {8 A _ {\operatorname* {m a x}} ^ {2}}{\tau} \left(\beta_ {k} ^ {2} + \frac {\beta_ {k}}{c _ {2}} + \alpha_ {k} \beta_ {k} c _ {1}\right) \mathbb {E} \left[ V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right) \right] + 8 \left(\alpha_ {k} ^ {2} + \frac {\alpha_ {k} \beta_ {k}}{c _ {1}}\right) \\ = \left(1 - \frac {3 \ell_ {\tau} \alpha_ {k}}{2} + \frac {2 A _ {\operatorname* {m a x}} ^ {2}}{\tau^ {2}} \left(2 \beta_ {k} ^ {2} + \frac {2 \beta_ {k} ^ {2}}{\ell_ {\tau} \alpha_ {k}}\right)\right) \sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] \\ + \frac {8 A _ {\max } ^ {2}}{\tau} \left(2 \beta_ {k} ^ {2} + \frac {2 \beta_ {k} ^ {2}}{\ell_ {\tau} \alpha_ {k}}\right) \mathbb {E} \left[ V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right) \right] + 1 6 \alpha_ {k} ^ {2} \quad (\text {C h o o s i n g} c _ {1} = \frac {\beta_ {k}}{\alpha_ {k}} \text {a n d} c _ {2} = \frac {\ell_ {\tau} \alpha_ {k}}{2 \beta_ {k}}) \\ \leq \left(1 - \frac {\ell_ {\tau} \alpha_ {k}}{2}\right) \sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] + \frac {\beta_ {k}}{4} \mathbb {E} [ V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}) ] + 1 6 \alpha_ {k} ^ {2}, \\ \end{array}
$$
where the last line follows from  $c_{\alpha, \beta} \leq \min \left(\frac{\tau^2 \ell_\tau}{8A_{\max}^2}, \frac{\tau \ell_\tau}{128A_{\max}^2}\right)$  and  $\beta_0 \leq \frac{\tau}{128A_{\max}^2}$  (cf. Condition 2.1). The proof is complete.
C.4 Solving the Coupled Lyapunov Inequalities
For simplicity of notation, denote  $\mathcal{L}_q(k) = \sum_{i=1,2} \mathbb{E}[||q_k^i - R_i\pi_k^{-i}||_2^2]$  and  $\mathcal{L}_{\pi}(k) = \mathbb{E}[V_R(\pi_k^1, \pi_k^2)]$ . Then, Lemmas C.3 and C.4 state that
$$
\mathcal {L} _ {\pi} (k + 1) \leq \left(1 - \frac {\beta_ {k}}{2}\right) \mathcal {L} _ {\pi} (k) + \frac {\ell_ {\tau} \alpha_ {k}}{4} \mathcal {L} _ {q} (k) + 2 L _ {\tau} \beta_ {k} ^ {2},
$$
$$
\mathcal {L} _ {q} (k + 1) \leq \left(1 - \frac {\ell_ {\tau} \alpha_ {k}}{2}\right) \mathcal {L} _ {q} (k) + \frac {\beta_ {k}}{4} \mathcal {L} _ {\pi} (k) + 1 6 \alpha_ {k} ^ {2}.
$$
Adding up the previous two inequalities, we obtain
$$
\begin{array}{l} \mathcal {L} _ {q} (k + 1) + \mathcal {L} _ {\pi} (k + 1) \leq \left(1 - \frac {\beta_ {k}}{4}\right) \mathcal {L} _ {\pi} (k) + 2 L _ {\tau} \beta_ {k} ^ {2} + \left(1 - \frac {\ell_ {\tau} \alpha_ {k}}{4}\right) \mathcal {L} _ {q} (k) + 1 6 \alpha_ {k} ^ {2} \\ \leq \left(1 - \frac {\beta_ {k}}{4}\right) \left(\mathcal {L} _ {\pi} (k) + \mathcal {L} _ {q} (k)\right) + 2 L _ {\tau} \beta_ {k} ^ {2} + 1 6 \alpha_ {k} ^ {2}, \tag {25} \\ \end{array}
$$
where the second inequality follows from  $c_{\alpha, \beta} \leq \ell_{\tau}$  (cf. Condition 2.1).
Constant Stepsizes. When using constant step sizes, i.e.,  $\alpha_{k}\equiv \alpha$  and  $\beta_{k}\equiv \beta$ , repeatedly using Eq. (25), we have for all  $k\geq 0$  that
$$
\begin{array}{l} \mathcal {L} _ {q} (k) + \mathcal {L} _ {\pi} (k) \leq \left(1 - \frac {\beta}{4}\right) ^ {k} \left(\mathcal {L} _ {\pi} (0) + \mathcal {L} _ {q} (0)\right) + 8 L _ {\tau} \beta + 6 4 \alpha^ {2} / \beta \\ \leq \left(1 - \frac {\beta}{4}\right) ^ {k} \left(4 + 2 \tau \log \left(A _ {\max }\right) + 2 A _ {\max }\right) + 8 L _ {\tau} \beta + 6 4 \alpha^ {2} / \beta \\ \end{array}
$$
$$
= B _ {\mathrm {i n}} \left(1 - \frac {\beta}{4}\right) ^ {k} + 8 L _ {\tau} \beta + \frac {6 4 \alpha}{c _ {\alpha , \beta}}
$$
where the second inequality follows from
$$
\mathcal {L} _ {\pi} (0) \leq 4 + 2 \tau \log \left(A _ {\max }\right), \quad \text {a n d} \quad \mathcal {L} _ {q} (0) \leq 2 A _ {\max }
$$
Theorem 2.1 (1) follows by observing that  $\mathcal{L}_q(k) + \mathcal{L}_{\pi}(k)\geq \mathcal{L}_{\pi}(k) = \mathbb{E}[\mathrm{NG}_{\tau}(\pi_k^1,\pi_k^2)]$
**Diminishing Stepsizes.** Consider using  $\alpha_{k} = \frac{\alpha}{k + h}$  and  $\beta_{k} = \frac{\beta}{k + h}$ . Recursions of the form presented in Eq. (25) have been well studied in the existing literature for the convergence rates of iterative algorithms [44, 24, 65]. Since  $\beta > 4$ , using the same line of analysis as in [65, Appendix A.2], we have
$$
\mathcal {L} _ {q} (k) + \mathcal {L} _ {\pi} (k) \leq B _ {\mathrm {i n}} \left(\frac {h}{k + h}\right) ^ {\beta / 4} + \left(6 4 e L _ {\tau} \beta + 5 1 2 e \alpha / c _ {\alpha , \beta}\right) \frac {1}{k + h}.
$$
Theorem 2.1 (2) follows by observing that  $\mathcal{L}_q(k) + \mathcal{L}_{\pi}(k)\geq \mathcal{L}_{\pi}(k) = \mathbb{E}[\mathrm{NG}_{\tau}(\pi_k^1,\pi_k^2)]$
C.5 Proof of Corollary 2.1.1
We use Theorem 2.1 (1) to derive the sample complexity, and choose  $\beta = c_{\alpha,\beta}\alpha$  with  $c_{\alpha,\beta}$  satisfying Condition 2.1. To achieve  $\mathbb{E}[\mathrm{NG}_{\tau}(\pi_K^1,\pi_K^2)] \leq \epsilon$ , in view of Theorem 2.1 (1), it is sufficient that
$$
B _ {\mathrm {i n}} e ^ {- \beta K / 4} \leq \frac {\epsilon}{3}, \quad 8 L _ {\tau} \beta \leq \frac {\epsilon}{3}, \quad 6 4 \alpha / c _ {\alpha , \beta} \leq \frac {\epsilon}{3},
$$
which implies  $\beta = \mathcal{O}(\epsilon)$ . It follows that  $K = \mathcal{O}\left(\epsilon^{-1}\right)$ .
C.6 Supporting Lemmas
Lemma C.5. For any  $i\in \{1,2\}$  and  $\mu_1^i,\mu_2^i\in \{\mu^i\in \Delta (\mathcal{A}^i)\mid \min_{a^i\in \mathcal{A}^i}\mu^i (a^i)\geq \ell_\tau \}$ , we have
$$
\| \nabla \nu (\mu_ {1} ^ {i}) - \nabla \nu (\mu_ {2} ^ {i}) \| _ {2} \leq \frac {1}{\ell_ {\tau}} \| \mu_ {1} ^ {i} - \mu_ {2} ^ {i} \| _ {2}.
$$
Proof of Lemma C.5. For any  $i \in \{1,2\}$  and  $\mu^i \in \Delta(\mathcal{A}^i)$  such that  $\min_{a^i \in \mathcal{A}^i} \mu^i(a^i) \geq \ell_\tau$ , the Hessian of  $\nu(\cdot)$  satisfies
$$
\operatorname {H e s s i a n} _ {\nu} (\mu^ {i}) = \operatorname {d i a g} \left(\mu^ {i}\right) ^ {- 1} \leq \frac {I _ {| \mathcal {A} ^ {i} |}}{\operatorname* {m i n} _ {a ^ {i} \in \mathcal {A} ^ {i}} \mu^ {i} (a ^ {i})} \leq \frac {I _ {| \mathcal {A} ^ {i} |}}{\ell_ {\tau}}.
$$
Therefore, the gradient of the negative entropy function  $\nabla \nu (\cdot)$  is  $\frac{1}{\ell_{\tau}}$  - Lipschitz continuous with respect to  $\| \cdot \| _2$  on the set  $\{\mu^i\in \Delta (\mathcal{A}^i)\mid \min_{a^i\in \mathcal{A}^i}\mu^i (a^i)\geq \ell_\tau \}$ , which implies the  $\frac{1}{\ell_{\tau}}$  - smoothness of  $\nu (\cdot)$ .
Lemma C.6. For  $i\in \{1,2\}$ , we have for all  $\mu^i\in \Delta (\mathcal{A}^i)$  and  $\mu^{-i}\in \Delta (\mathcal{A}^{-i})$  that
$$
\| \sigma_ {\tau} (R _ {i} \mu^ {- i}) - \mu^ {i} \| _ {2} ^ {2} \leq \frac {2}{\tau} V _ {R} (\mu^ {1}, \mu^ {2}).
$$
Proof of Lemma C.6. Recall that the negative entropy  $\nu(\cdot)$  is 1-strongly concave with respect to  $\|\cdot\|_2$ . Therefore, given  $i \in \{1,2\}$ , fix  $\mu^{-i}$ , the function
$$
\max _ {\hat {\mu} ^ {i} \in \Delta (\mathcal {A} ^ {i})} \left\{\left(\hat {\mu} ^ {i} - \mu^ {i}\right) ^ {\top} R _ {i} \mu^ {- i} + \tau \nu \left(\hat {\mu} ^ {i}\right) - \tau \nu \left(\mu^ {i}\right) \right\}
$$
is  $\tau$ -strongly convex with respect to  $\mu^i$ . As a result, by the quadratic growth property of strongly convex functions, we have
$$
\| \sigma_ {\tau} (R _ {i} \mu^ {- i}) - \mu^ {i} \| _ {2} ^ {2} \leq \frac {2}{\tau} \max _ {\hat {\mu} ^ {i} \in \Delta (\mathcal {A} ^ {i})} \left\{(\hat {\mu} ^ {i} - \mu^ {i}) ^ {\top} R _ {i} \mu^ {- i} + \tau \nu (\hat {\mu} ^ {i}) - \tau \nu (\mu^ {i}) \right\} \leq \frac {2}{\tau} V _ {R} (\mu^ {1}, \mu^ {2}).
$$
Denote  $\Pi_{\tau} = \{(\pi^{1},\pi^{2})\in \Delta (\mathcal{A}^{1})\times \Delta (\mathcal{A}^{2})\mid \min_{a^{1}\in \mathcal{A}^{1}}\pi^{1}(a^{1})\geq \ell_{\tau},\min_{a^{2}\in \mathcal{A}^{2}}\pi^{2}(a^{2})\geq \ell_{\tau}\}$
Lemma C.7. The function  $V_{R}(\cdot ,\cdot)$  has the following properties.
(1) The function  $V_{R}(\mu^{1},\mu^{2})$  is  $L_{\tau}$ -smooth on  $\Pi_{\tau}$ , where  $L_{\tau} = \frac{\tau}{\ell_{\tau}} + \frac{A_{\max}^2}{\tau}$ .
(2) It holds for any  $(\mu^1,\mu^2)\in \Pi_\tau$  that
$$
\langle \nabla_ {1} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (R _ {1} \mu^ {2}) - \mu^ {1} \rangle + \langle \nabla_ {2} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (R _ {2} \mu^ {1}) - \mu^ {2} \rangle \leq - V _ {R} (\mu^ {1}, \mu^ {2}).
$$
(3) For any  $q^1 \in \mathbb{R}^{|\mathcal{A}^1|}$  and  $q^2 \in \mathbb{R}^{|\mathcal{A}^2|}$ , we have for all  $(\mu^1, \mu^2) \in \Pi_\tau$  that
$$
\begin{array}{l} \langle \nabla_ {1} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (q ^ {1}) - \sigma_ {\tau} (R _ {1} \mu^ {2}) \rangle + \langle \nabla_ {2} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (q ^ {2}) - \sigma_ {\tau} (R _ {2} \mu^ {1}) \rangle \\ \leq \frac {1}{2} V _ {R} (\mu^ {i}, \mu^ {- i}) + 4 \left(\frac {1}{\tau \ell_ {\tau} ^ {2}} + \frac {A _ {\mathrm {m a x}} ^ {2}}{\tau^ {3}}\right) \sum_ {i = 1, 2} \| q ^ {i} - R _ {i} \mu^ {- i} \| _ {2} ^ {2}. \\ \end{array}
$$
Proof of Lemma C.7. Recall the definition of  $V_{R}(\cdot, \cdot)$  in the following:
$$
V _ {R} (\mu^ {1}, \mu^ {2}) = \sum_ {i = 1, 2} \max  _ {\hat {\mu} ^ {i} \in \Delta (\mathcal {A} ^ {i})} \left\{(\hat {\mu} ^ {i} - \mu^ {i}) ^ {\top} R _ {i} \mu^ {- i} + \tau \nu (\hat {\mu} ^ {i}) - \tau \nu (\mu^ {i}) \right\}.
$$
By Danskin's theorem [95], we have
$$
\begin{array}{l} \nabla_ {1} V _ {R} \left(\mu^ {1}, \mu^ {2}\right) = - \tau \nabla \nu \left(\mu^ {1}\right) + \left(R _ {2}\right) ^ {\top} \sigma_ {\tau} \left(R _ {2} \mu^ {1}\right), \\ \nabla_ {2} V _ {R} (\mu^ {1}, \mu^ {2}) = - \tau \nabla \nu (\mu^ {2}) + (R _ {1}) ^ {\top} \sigma_ {\tau} (R _ {1} \mu^ {2}), \\ \end{array}
$$
both of which will be frequently used in our analysis.
(1) For any  $(\mu^1,\mu^2),(\bar{\mu}^1,\bar{\mu}^2)\in \Pi_\tau$  , we have
$$
\begin{array}{l} \left\| \nabla_ {1} V _ {R} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {1} V _ {R} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \right\| _ {2} \\ = \| \tau \nabla \nu (\bar {\mu} ^ {1}) - \tau \nabla \nu (\mu^ {1}) + (R _ {2}) ^ {\top} \sigma_ {\tau} (R _ {2} \mu^ {1}) - (R _ {2}) ^ {\top} \sigma_ {\tau} (R _ {2} \bar {\mu} ^ {1}) \| _ {2} \\ \leq \tau \| \nabla \nu (\bar {\mu} ^ {1}) - \nabla \nu (\mu^ {1}) \| _ {2} + \| R _ {2} \| _ {2} \| \sigma_ {\tau} (R _ {2} \mu^ {1}) - \sigma_ {\tau} (R _ {2} \bar {\mu} ^ {1}) \| _ {2} \\ \leq \frac {\tau}{\ell_ {\tau}} \| \mu^ {1} - \bar {\mu} ^ {1} \| _ {2} + \frac {\| R _ {2} \| _ {2} ^ {2}}{\tau} \| \mu^ {1} - \bar {\mu} ^ {1} \| _ {2} \\ \leq \left(\frac {\tau}{\ell_ {\tau}} + \frac {A _ {\operatorname* {m a x}} ^ {2}}{\tau}\right) \| \mu^ {1} - \bar {\mu} ^ {1} \| _ {2}, \\ \end{array}
$$
where the second last inequality follows from Lemma C.5 and  $\sigma_{\tau}(\cdot)$  being  $\frac{1}{\tau}$ -Lipschitz continuous with respect to  $\| \cdot \|_2$  [96], and the last inequality follows from  $\| R_i \|_2 \leq \sqrt{|\mathcal{A}^1| |\mathcal{A}^2|} \leq A_{\max}$  for  $i \in \{1, 2\}$ . Similarly, we also have
$$
\| \nabla_ {2} V _ {R} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {2} V _ {R} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \| _ {2} ^ {2} \leq \left(\frac {\tau}{\ell_ {\tau}} + \frac {A _ {\operatorname* {m a x}} ^ {2}}{\tau}\right) \| \mu^ {2} - \bar {\mu} ^ {2} \| _ {2}.
$$
It follows from the previous two inequalities that
$$
\begin{array}{l} \left\| \nabla V _ {R} \left(\mu^ {1}, \mu^ {2}\right) - \nabla V _ {R} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \right\| _ {2} ^ {2} \\ = \| \nabla_ {1} V _ {R} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {1} V _ {R} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \| _ {2} ^ {2} + \| \nabla_ {2} V _ {R} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {2} V _ {R} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \| _ {2} ^ {2} \\ \leq \left(\frac {\tau}{\ell_ {\tau}} + \frac {A _ {\operatorname* {m a x}} ^ {2}}{\tau}\right) ^ {2} \sum_ {i = 1, 2} \| \mu^ {i} - \bar {\mu} ^ {i} \| _ {2} ^ {2}, \\ \end{array}
$$
which implies that  $V_{R}(\cdot, \cdot)$  is an  $L_{\tau}$ -smooth function on  $\Pi_{\tau}$  [97], where  $L_{\tau} = \frac{\tau}{\ell_{\tau}} + \frac{A_{\max}^{2}}{\tau}$ .
(2) The result follows from Lemma D.7 by setting  $X_{i} = R_{i}$ ,  $i \in \{1, 2\}$ , and by observing that  $R_{1} + (R_{2})^{\top} = 0$ .
(3) Using the formula of the gradient of  $V_{R}(\cdot, \cdot)$  in the beginning of the proof, we have
$$
\begin{array}{l} \langle \nabla_ {1} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (q ^ {1}) - \sigma_ {\tau} (R _ {1} \mu^ {2}) \rangle \\ = \left\langle - \tau \nabla \nu \left(\mu^ {1}\right) + \left(R _ {2}\right) ^ {\top} \sigma_ {\tau} \left(R _ {2} \mu^ {1}\right), \sigma_ {\tau} \left(q ^ {1}\right) - \sigma_ {\tau} \left(R _ {1} \mu^ {2}\right) \right\rangle \\ = \tau \left\langle \nabla \nu \left(\sigma_ {\tau} \left(R _ {1} \mu^ {2}\right)\right) - \nabla \nu \left(\mu^ {1}\right), \sigma_ {\tau} \left(q ^ {1}\right) - \sigma_ {\tau} \left(R _ {1} \mu^ {2}\right) \right\rangle \\ + \left(\sigma_ {\tau} \left(R _ {2} \mu^ {1}\right) - \mu^ {2}\right) ^ {\top} R _ {2} \left(\sigma_ {\tau} \left(q ^ {1}\right) - \sigma_ {\tau} \left(R _ {1} \mu^ {2}\right)\right) \\ \end{array}
$$
(This follows from the order optimality condition:  $R_{1}\mu^{2} + \tau \nabla \nu (\sigma_{\tau}(R_1\mu^2)) = 0$
$$
\begin{array}{l} \leq \frac {\tau}{2 c _ {1}} \| \nabla \nu (\sigma_ {\tau} (R _ {1} \mu^ {2})) - \nabla \nu (\mu^ {1}) \| _ {2} ^ {2} + \frac {\tau c _ {1}}{2} \| \sigma_ {\tau} (q ^ {1}) - \sigma_ {\tau} (R _ {1} \mu^ {2}) \| _ {2} ^ {2} \\ + \frac {1}{2 c _ {2}} \| \sigma_ {\tau} (R _ {2} \mu^ {1}) - \mu^ {2}) \| _ {2} ^ {2} + \frac {c _ {2}}{2} \| R _ {2} (\sigma_ {\tau} (q ^ {1}) - \sigma_ {\tau} (R _ {1} \mu^ {2})) \| _ {2} ^ {2} \\ \end{array}
$$
(This follows from AM-GM inequality, where  $c_{1}, c_{2} > 0$  can be arbitrary)
$$
\begin{array}{l} \leq \frac {\tau}{2 c _ {1} \ell_ {\tau} ^ {2}} \| \sigma_ {\tau} (R _ {1} \mu^ {2}) - \mu^ {1} \| _ {2} ^ {2} + \frac {c _ {1}}{2 \tau} \| q ^ {1} - R _ {1} \mu^ {2} \| _ {2} ^ {2} \\ + \frac {1}{2 c _ {2}} \| \sigma_ {\tau} \left(R _ {2} \mu^ {1}\right) - \mu^ {2}) \| _ {2} ^ {2} + \frac {c _ {2} \| R _ {2} \| _ {2} ^ {2}}{2 \tau^ {2}} \| q ^ {1} - R _ {1} \mu^ {2} \| _ {2} ^ {2} \tag {Lemma C.5} \\ \end{array}
$$
$$
\begin{array}{l} \leq \left(\frac {1}{c _ {1} \ell_ {\tau} ^ {2}} + \frac {1}{\tau c _ {2}}\right) V _ {R} \left(\mu^ {1}, \mu^ {2}\right) + \frac {c _ {1}}{2 \tau} \| q ^ {1} - R _ {1} \mu^ {2} \| _ {2} ^ {2} + \frac {c _ {2} \| R _ {2} \| _ {2} ^ {2}}{2 \tau^ {2}} \| q ^ {1} - R _ {1} \mu^ {2} \| _ {2} ^ {2} (L e m m a C. 6) \\ \leq \frac {1}{4} V _ {R} (\mu^ {1}, \mu^ {2}) + \frac {4}{\tau \ell_ {\tau} ^ {2}} \| q ^ {1} - R _ {1} \mu^ {2} \| _ {2} ^ {2} + \frac {4 \| R _ {2} \| _ {2} ^ {2}}{\tau^ {3}} \| q ^ {1} - R _ {1} \mu^ {2} \| _ {2} ^ {2}, \\ \end{array}
$$
where the last line follows by choosing  $c_{1} = \frac{8}{\ell_{\tau}^{2}}$  and  $c_{2} = \frac{8}{\tau}$ . Similarly, we also have
$$
\begin{array}{l} \langle \nabla_ {2} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (q ^ {2}) - \sigma_ {\tau} (R _ {2} \mu^ {1}) \rangle \leq \frac {1}{4} V _ {R} (\mu^ {1}, \mu^ {2}) + \frac {4}{\tau \ell_ {\tau} ^ {2}} \| q ^ {2} - R _ {2} \mu^ {1} \| _ {2} ^ {2} \\ + \frac {4 \| R _ {1} \| _ {2} ^ {2}}{\tau^ {3}} \| q ^ {2} - R _ {2} \mu^ {1} \| _ {2} ^ {2}. \\ \end{array}
$$
Summing up the previous two inequalities, we obtain
$$
\begin{array}{l} \langle \nabla_ {1} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (q ^ {1}) - \sigma_ {\tau} (R _ {1} \mu^ {2}) \rangle + \langle \nabla_ {2} V _ {R} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (q ^ {2}) - \sigma_ {\tau} (R _ {2} \mu^ {1}) \rangle \\ \leq \frac {1}{2} V _ {R} (\mu^ {i}, \mu^ {- i}) + \left(\frac {4}{\tau \ell_ {\tau} ^ {2}} + \frac {4 A _ {\mathrm {m a x}} ^ {2}}{\tau^ {3}}\right) \sum_ {i = 1, 2} \| q ^ {i} - R _ {i} \mu^ {- i} \| _ {2} ^ {2}, \\ \end{array}
$$
where we used  $\| R_i\| _2\leq \sqrt{A_{\mathrm{max}}}$  for  $i\in \{1,2\}$ .
D Proof of Theorem 3.1
We begin by introducing a summary of notation in Appendix D.1. In Appendix D.2, we establish an important boundedness property regarding the  $q$ -functions, value functions, and the policies generated by Algorithm 2. In Appendix D.3, we bound the Nash gap in terms of the Lyapunov functions. In Appendices D.4 and D.5, we analyze the outer loop and the inner loop of Algorithm 2 and establish the Lyapunov drift inequalities. Finally, in Appendix D.6, we solve the coupled Lyapunov inequalities to obtain the finite-sample bound. The proof of all supporting lemmas are provided in Appendix D.7, and the proof of Corollary 3.1.1 is provided in Appendix D.8.
D.1 Notation
We begin with a summary of the notation that will be used in the proof.
(1) Given a pair of matrices  $\{X_{i}\in \mathbb{R}^{|\mathcal{A}^{i}|\times |\mathcal{A}^{-i}|}\}_{i\in \{1,2\}}$  and a pair of distributions  $\{\mu^i\in \Delta (\mathcal{A}^i)\}_{i\in \{1,2\}}$ , we define
$$
V _ {X} \left(\mu^ {1}, \mu^ {2}\right) = \sum_ {i = 1, 2} \max  _ {\hat {\mu} ^ {i} \in \Delta \left(\mathcal {A} ^ {i}\right)} \left\{\left(\hat {\mu} ^ {i} - \mu^ {i}\right) ^ {\top} X _ {i} \mu^ {- i} + \tau \nu \left(\hat {\mu} ^ {i}\right) - \tau \nu \left(\mu^ {i}\right) \right\}, \tag {26}
$$
where  $\nu (\cdot)$  is the entropy function. Note that  $V_{X}(\cdot ,\cdot)$  is similar to  $V_{R}(\cdot ,\cdot)$  defined in Appendix C.2 in the setting of matrix games. However, we do not assume that  $X_{1} + X_{2} = 0$
(2) Given a pair of value functions  $(v^{1}, v^{2})$  and a state  $s \in S$ , when  $X_{i} = \mathcal{T}^{i}(v^{i})(s)$ ,  $i \in \{1, 2\}$ , we write  $V_{v,s}(\cdot, \cdot)$  for  $V_{X}(\cdot, \cdot)$ .
(3) For any joint policy  $(\pi^1, \pi^2)$  and state  $s$ , given  $i \in \{1, 2\}$ , we define  $v_{*, \pi^{-i}}^i(s) = \max_{\hat{\pi}^i} v_{\hat{\pi}^i, \pi^{-i}}^i(s)$ ,  $v_{\pi^i, *}^i = \min_{\hat{\pi}^{-i}} v_{\pi^i, \hat{\pi}^{-i}}^i(s)$ ,  $v_{\pi^{-i}, *}^{-i}(s) = \min_{\hat{\pi}^i} v_{\pi^{-i}, \hat{\pi}^i}^{-i}(s)$ , and  $v_{*, \pi^i}^{-i}(s) = \max_{\hat{\pi}^{-i}} v_{\hat{\pi}^{-i}, \hat{\pi}^i}^{-i}(s)$ . Note that we have  $v_{*, \pi^2}^1 + v_{\pi^2, *}^2 = 0$  and  $v_{\pi^1, *}^1 + v_{*, \pi^1}^2 = 0$  because of the zero-sum structure.
(4) For  $i \in \{1, 2\}$ , denote  $v_{*}^{i}$  as the unique fixed point of the equation  $\mathcal{B}^i(v^i) = v^i$ , where  $\mathcal{B}^i(\cdot)$  is the minimax Bellman operator defined in Section 3. Note that we have  $v_{*}^{1} + v_{*}^{2} = 0$ .
(5) For any  $t, k \geq 0$  and  $i \in \{1,2\}$ , let  $\bar{q}_{t,k}^i \in \mathbb{R}^{|\mathcal{S}| |\mathcal{A}^i|}$  be defined as  $\bar{q}_{t,k}^i(s) = \mathcal{T}^i(v_t^i)(s)\pi_{t,k}^{-i}(s)$  for all  $s \in S$ . In addition, let
$$
\mathcal {L} _ {q} (t, k) = \sum_ {i = 1, 2} \sum_ {s \in \mathcal {S}} \| q _ {t, k} ^ {i} (s) - \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, k} ^ {- i} (s) \| _ {2} ^ {2} = \sum_ {i = 1, 2} \| q _ {t, k} ^ {i} - \bar {q} _ {t, k} ^ {i} \| _ {2} ^ {2},
$$
$$
\mathcal {L} _ {\pi} (t, k) = \max _ {s \in \mathcal {S}} V _ {v _ {t}, s} (\pi_ {t, k} ^ {1} (s), \pi_ {t, k} ^ {2} (s)),
$$
which will be the Lyapunov functions we use in the analysis.
(6) Given  $k_{1} \leq k_{2}$ , we denote  $\beta_{k_1,k_2} = \sum_{k = k_1}^{k_2}\beta_k$  and  $\alpha_{k_1,k_2} = \sum_{k = k_1}^{k_2}\alpha_k$ .
(7) Recall that  $z_{k} = t(\ell_{\tau}, \beta_{k})$  is the uniform mixing time defined in Lemma B.1 (2), where  $\ell_{\tau}$  is the uniform lower bound of the policies. When using constant step sizes,  $z_{k}$  is not a function of  $k$ , and is simply denoted by  $z_{\beta}$ . Observe that, due to the uniform geometric mixing property established in Lemma B.1 (2), we have  $z_{k} = \mathcal{O}(\log(k))$  when using  $\mathcal{O}(1/k)$  stepsizes and  $z_{\beta} = \mathcal{O}(\log(1/\beta))$  when using constant step sizes. Let  $k_{0} = \min k : k \geq z_{k}$ , which is well defined because  $z_{k}$  grows logarithmically with  $k$ .
D.2 Boundedness of the Iterates
We first show in the following lemma that the  $q$ -functions and the  $v$ -functions generated by Algorithm 2 are uniformly bounded from above, and the policies are uniformly bounded from below. In the context of stochastic games, we redefine  $\ell_{\tau} = [1 + (A_{\max} - 1)\exp (2 / [(1 - \gamma)\tau ])]^{-1}$ .
Lemma D.1. For all  $t, k$  and  $i \in \{1, 2\}$ , we have (1)  $\| v_t^i \|_\infty \leq 1 / (1 - \gamma)$  and  $\| q_{t,k}^i \|_\infty \leq 1 / (1 - \gamma)$ , and (2)  $\min_{s \in S, a^i \in \mathcal{A}^i} \pi_{t,k}^i(a^i \mid s) \geq \ell_\tau$ .
Proof of Lemma D.1. The proof uses induction arguments. Let  $i \in \{1, 2\}$ .
(1) Given  $t \geq 0$ , we first show by induction that, if  $\| v_{t}^{i}\|_{\infty} \leq \frac{1}{1 - \gamma}$  and  $\| q_{t,0}^{i}\|_{\infty} \leq \frac{1}{1 - \gamma}$ , we have  $\| q_{t,k}^{i}\|_{\infty} \leq \frac{1}{1 - \gamma}$  for all  $k \geq 0$ . The base case  $\| q_{t,0}^{i}\|_{\infty} \leq \frac{1}{1 - \gamma}$  holds by our assumption. Suppose that  $\| q_{t,k}^{i}\|_{\infty} \leq \frac{1}{1 - \gamma}$  for some  $k \geq 0$ . Then, by Algorithm 2 Line 6, we have for all  $(s,a^i)$  that
$$
\begin{array}{l} \left| q _ {t, k + 1} ^ {i} (s, a ^ {i}) \right| \\ = \left| q _ {t, k} ^ {i} (s, a ^ {i}) + \alpha_ {k} \mathbb {1} _ {\left\{\left(s, a ^ {i}\right) = \left(S _ {k}, A _ {k} ^ {i}\right) \right\}} \left(R _ {i} \left(S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right) + \gamma v _ {t} ^ {i} \left(S _ {k + 1}\right) - q _ {t, k} ^ {i} \left(S _ {k}, A _ {k} ^ {i}\right)\right) \right| \\ \leq \left(1 - \alpha_ {k} \mathbb {1} _ {\left\{\left(s, a ^ {i}\right) = \left(S _ {k}, A _ {k} ^ {i}\right) \right\}}\right) \left| q _ {t, k} ^ {i} \left(s, a ^ {i}\right) \right| \\ + \alpha_ {k} \mathbb {1} _ {\left\{(s, a ^ {i}) = \left(S _ {k}, A _ {k} ^ {i}\right) \right\}} \left| R _ {i} \left(S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right) + \gamma v _ {t} ^ {i} \left(S _ {k + 1}\right) \right| \\ \leq \left(1 - \alpha_ {k} \mathbb {1} _ {\left\{\left(s, a ^ {i}\right) = \left(S _ {k}, A _ {k} ^ {i}\right) \right\}}\right) \frac {1}{1 - \gamma} + \alpha_ {k} \mathbb {1} _ {\left\{\left(s, a ^ {i}\right) = \left(S _ {k}, A _ {k} ^ {i}\right) \right\}} \left(1 + \frac {\gamma}{1 - \gamma}\right) \tag {27} \\ = \frac {1}{1 - \gamma}, \\ \end{array}
$$
where Eq. (27) follows from the induction hypothesis  $\| q_{t,k}^i\|_\infty \leq \frac{1}{1 - \gamma}$ , our assumption that  $\| v_t^i\|_\infty \leq \frac{1}{1 - \gamma}$ , and  $\max_{s,a^i,a^{-i}}|R_i(s,a^i,a^{-i})|\leq 1$ . The induction is now complete and we have  $\| q_{t,k}^i\|_\infty \leq \frac{1}{1 - \gamma}$  for all  $k\geq 0$  whenever  $\| v_t^i\|_\infty \leq \frac{1}{1 - \gamma}$  and  $\| q_{t,0}^i\|_\infty \leq \frac{1}{1 - \gamma}$ .
We next again use induction to show that  $\| v_{t}^{i}\|_{\infty}\leq \frac{1}{1 - \gamma}$  and  $\| q_{t,0}^{i}\|_{\infty}\leq \frac{1}{1 - \gamma}$  for all  $t\geq 0$ . Our initialization ensures that  $\| v_0^i\|_\infty \leq \frac{1}{1 - \gamma}$  and  $\| q_{0,0}^{i}\|_{\infty}\leq \frac{1}{1 - \gamma}$ . Suppose that  $\| v_t^i\|_{\infty}\leq \frac{1}{1 - \gamma}$  and  $\| q_{t,0}^{i}\|_{\infty}\leq \frac{1}{1 - \gamma}$  for some  $t\geq 0$ . Using the update equation for  $v_{t + 1}^{i}$  (cf. Algorithm 2 Line 8) and the fact that  $\| q_{t,k}^{i}\|_{\infty}\leq \frac{1}{1 - \gamma}$  for all  $k\geq 0$  (established in the previous paragraph), we have for all  $s\in S$  that
$$
| v _ {t + 1} ^ {i} (s) | = \left| \sum_ {a ^ {i} \in \mathcal {A} ^ {i}} \pi_ {t, K} ^ {i} (a ^ {i} | s) q _ {t, K} ^ {i} (s, a ^ {i}) \right| \leq \sum_ {a ^ {i} \in \mathcal {A} ^ {i}} \pi_ {t, K} ^ {i} (a ^ {i} | s) \| q _ {t, K} ^ {i} \| _ {\infty} \leq \frac {1}{1 - \gamma},
$$