SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which implies  $\| v_{t + 1}^i\|_\infty \leq \frac{1}{1 - \gamma}$ . Moreover, we have by Algorithm 2 Line 9 that  $\| q_{t + 1,0}^i\|_\infty = \| q_{t,K}^i\|_\infty \leq \frac{1}{1 - \gamma}$ . The induction is now complete and we have  $\| v_t^i\|_\infty \leq \frac{1}{1 - \gamma}$  and  $\| q_{t,0}^i\|_\infty \leq \frac{1}{1 - \gamma}$  for all  $t \geq 0$ .
(2) We first use induction to show that, given  $t \geq 0$ , if  $\min_{s,a^i} \pi_{t,0}^i(a^i \mid s) \geq \ell_\tau$ , then we have  $\min_{s,a^i} \pi_{t,k}^i(a^i \mid s) \geq \ell_\tau$  for all  $k \in \{0,1,\dots,K\}$ . Since  $\min_{s,a^i} \pi_{t,0}^i(a^i \mid s) \geq \ell_\tau$  by our assumption, we have the base case. Now suppose that  $\min_{s \in S, a^i \in A^i} \pi_{t,k}^i(a^i \mid s) \geq \ell_\tau$  for some  $k \geq 0$ . Then we have by Algorithm 2 Line 4 that
$$
\begin{array}{l} \pi_ {t, k + 1} ^ {i} (a ^ {i} \mid s) = (1 - \beta_ {k}) \pi_ {t, k} ^ {i} (a ^ {i} \mid s) + \beta_ {k} \sigma_ {\tau} (q _ {t, k} ^ {i} (s)) (a ^ {i}) \\ \geq \left(1 - \beta_ {k}\right) \ell_ {\tau} + \beta_ {k} \ell_ {\tau} \\ = \ell_ {\tau}, \\ \end{array}
$$
where the inequality follows from (1) the induction hypothesis, and (2)  $\sigma_{\tau}(q_{t,k}^{i}(s))(a^{i})\geq \ell_{\tau}$ , which follows from Lemma D.1 (1) and Lemma C.1. The induction is complete.
We next again use induction to show that  $\min_{s,a^i}\pi_{t,0}^i (a^i\mid s)\geq \ell_\tau$  for all  $t\in \{0,1,\dots ,T\}$ . Since  $\min_{s,a^i}\pi_{0,0}^i (a^i\mid s)$  is initialized as a uniform policy, we have the base case. Now suppose that  $\min_{s,a^i}\pi_{t,0}^i (a^i\mid s)\geq \ell_\tau$  for some  $t\geq 0$ . Since this implies that  $\min_{s,a^i}\pi_{t,k}^i (a^i\mid s)\geq \ell_\tau$  for all  $k\in \{0,1,\dots ,K\}$ , and in addition,  $\pi_{t + 1,0}^{i} = \pi_{t,K}^{i}$  according to Algorithm 2 Line 9, we have  $\min_{s,a^i}\pi_{t + 1,0}^i (a^i\mid s)\geq \ell_\tau$ . The induction is complete.
D.3 Bounding the Nash Gap
Our ultimate goal is to bound the Nash gap
$$
\mathrm {N G} \left(\pi_ {T, K} ^ {1}, \pi_ {T, K} ^ {2}\right) = \sum_ {i = 1, 2} \left(\max  _ {\pi^ {i}} U ^ {i} \left(\pi^ {i}, \pi_ {T, K} ^ {- i}\right) - U ^ {i} \left(\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}\right)\right). \tag {28}
$$
We first bound the Nash gap using the value functions of the output policies from Algorithm 2.
Lemma D.2. It holds that
$$
\sum_ {i = 1, 2} \left(\max  _ {\pi^ {i}} U ^ {i} \left(\pi^ {i}, \pi_ {T, K} ^ {- i}\right) - U ^ {i} \left(\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}\right)\right) \leq \sum_ {i = 1, 2} \left\| v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} \right\| _ {\infty}. \tag {29}
$$
Proof of Lemma D.2. Using the definition of the expected value functions, we have
$$
\sum_ {i = 1, 2} \left(\max _ {\pi^ {i}} U ^ {i} (\pi^ {i}, \pi_ {T, K} ^ {- i}) - U ^ {i} (\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i})\right)
$$
$$
\begin{array}{l} = \sum_ {i = 1, 2} \left(\max  _ {\pi^ {i}} \mathbb {E} _ {S \sim p _ {o}} \left[ v _ {\pi^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} (S) - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} (S) \right]\right) \\ \leq \sum_ {i = 1, 2} \left(\mathbb {E} _ {S \sim p _ {o}} \left[ \max  _ {\pi^ {i}} v _ {\pi^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} (S) - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} (S) \right]\right) \quad (\text {J e s s e n ' s i n e q u a l i t y}) \\ = \sum_ {i = 1, 2} \left(\mathbb {E} _ {S \sim p _ {o}} \left[ v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} (S) - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} (S) \right]\right) \\ \leq \sum_ {i = 1, 2} \left\| v _ {* , \pi_ {T, K} ^ {- i}} ^ {i} - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} \right\| _ {\infty}. \\ \end{array}
$$
The next lemma bounds the RHS of Eq. (29) using the actual iterates generated by Algorithm 2.
Lemma D.3. It holds for  $i \in \{1,2\}$  that
$$
\left\| v _ {*, \pi_ {t, K} ^ {- i}} ^ {i} - v _ {\pi_ {t, K} ^ {i}, \pi_ {t, K} ^ {- i}} ^ {i} \right\| _ {\infty} \leq \frac {2}{1 - \gamma} \left(2 \mathcal {L} _ {s u m} (T) + \mathcal {L} _ {v} (T) + \mathcal {L} _ {\pi} (T, K) + 2 \tau \log \left(A _ {\max }\right)\right).
$$
Proof of Lemma D.3. For any  $s \in S$  and  $i \in \{1,2\}$ , we have
$$
\begin{array}{l} 0 \leq \left| v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} (s) - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} (s) \right| \\ = v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} (s) - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} (s) \\ \leq v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} (s) - v _ {\pi_ {T, K} ^ {i}, *} ^ {i} (s) \\ = - v _ {\pi_ {T, K} ^ {- i}, *} ^ {- i} (s) - v _ {\pi_ {T, K} ^ {i}, *} ^ {i} (s) \\ = v _ {*} ^ {i} (s) - v _ {\pi_ {T, K} ^ {- i}, *} ^ {- i} (s) + v _ {*} ^ {- i} (s) - v _ {\pi_ {T, K} ^ {i}, *} ^ {i} (s) \\ \leq \sum_ {j = 1, 2} \left\| v _ {*} ^ {- j} - v _ {\pi_ {T, K} ^ {- j}, *} ^ {- j} \right\| _ {\infty}. \\ \end{array}
$$
Since the RHS of the previous inequality does not depend on  $s$ , we have for  $i \in \{1, 2\}$  that
$$
\left\| v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} - v _ {\pi_ {T, K} ^ {i}, \pi_ {T, K} ^ {- i}} ^ {i} \right\| _ {\infty} \leq \sum_ {j = 1, 2} \left\| v _ {*} ^ {- j} - v _ {\pi_ {T, K} ^ {- j}, *} ^ {- j} \right\| _ {\infty}. \tag {30}
$$
It remains to bound the RHS of the previous inequality. Observe that for any  $s \in S$  and  $i \in \{1, 2\}$ , we have
$$
\begin{array}{l} 0 \leq v _ {*} ^ {- i} (s) - v _ {\pi_ {T, K} ^ {- i}, *} ^ {- i} (s) \\ = v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} (s) - v _ {*} ^ {i} (s) \\ = \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {*, \pi_ {T, K} ^ {- i}} ^ {i}) (s) \pi_ {T, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {*} ^ {i}) (s) \mu^ {- i} \\ \leq \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {*, \pi_ {T, K} ^ {- i}} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) \right| \\ + \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s) \mu^ {- i} \right| \\ \leq \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left| \left(\mu^ {i}\right) ^ {\top} \left(\mathcal {T} ^ {i} \left(v _ {*, \pi_ {T, K} ^ {- i}} ^ {i}\right) (s) - \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s)\right) \pi_ {T, K} ^ {- i} (s) \right| \\ + \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) \right| \\ + \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \mu^ {- i} \\ \end{array}
$$
$$
+ \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \mu^ {- i} - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s) \mu^ {- i} \right|. \tag {31}
$$
We next bound each term on the RHS of the previous inequality.
The 1st Term on the RHS of Eq. (31). Using the definition of  $\mathcal{T}^i (\cdot)$ , we have
$$
\begin{array}{l} \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left| \left(\mu^ {i}\right) ^ {\top} \left(\mathcal {T} ^ {i} \left(v _ {*}, \pi_ {T, K} ^ {- i}\right) (s) - \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s)\right) \pi_ {T, K} ^ {- i} (s) \right| \\ \leq \max _ {s, a ^ {i}, a ^ {- i}} \left| \mathcal {T} ^ {i} (v _ {\ast , \pi_ {T, K} ^ {- i}} ^ {i}) (s, a ^ {i}, a ^ {- i}) - \mathcal {T} ^ {i} (v _ {\ast} ^ {i}) (s, a ^ {i}, a ^ {- i}) \right| \\ = \gamma \max _ {s, a ^ {i}, a ^ {- i}} \left| \mathbb {E} \left[ v _ {*} ^ {i} (S _ {1}) - v _ {*}, \pi_ {T, K} ^ {- i} (S _ {1}) \right. \right| S _ {0} = s, A _ {0} ^ {i} = a ^ {i}, A _ {0} ^ {- i} = a ^ {- i} \Bigg ] \Bigg | \\ \leq \gamma \max  _ {s, a ^ {i}, a ^ {- i}} \mathbb {E} \left[ \left| v _ {*} ^ {i} (S _ {1}) - v _ {* , \pi_ {T, K} ^ {- i}} ^ {i} (S _ {1}) \right| \mid S _ {0} = s, A _ {0} ^ {i} = a ^ {i}, A _ {0} ^ {- i} = a ^ {- i} \right] \\ \leq \gamma \left\| v _ {*} ^ {i} - v _ {*}, \pi_ {T, K} ^ {- i} \right\| _ {\infty}. \\ \end{array}
$$
The 2nd Term on the RHS of Eq. (31). Using the definition of  $\mathcal{T}^i (\cdot)$ , we have
$$
\begin{array}{l} \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) \right| \\ \leq \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left| \left(\mu^ {i}\right) ^ {\top} \left(\mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s) - \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s)\right) \pi_ {T, K} ^ {- i} (s) \right| \\ \leq \max  _ {s, a ^ {i}, a ^ {- i}} \left| \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) \left(s, a ^ {i}, a ^ {- i}\right) - \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) \left(s, a ^ {i}, a ^ {- i}\right) \right| \\ \leq \gamma \left\| v _ {*} ^ {i} - v _ {T} ^ {i} \right\| _ {\infty}. \\ \end{array}
$$
The 3rd Term on the RHS of Eq. (31). Bounding the third term requires more effort. To begin with, we decompose it in the following way:
$$
\begin{array}{l} \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) \pi_ {T, K} ^ {- i} (s) - \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) \mu^ {- i} \\ \leq \left| \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) \pi_ {T, K} ^ {- i} (s) - \min _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \pi_ {T, K} ^ {i} (s) \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) \mu^ {- i} \right| \\ \leq \left| \max  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {- i}) ^ {\top} \mathcal {T} ^ {- i} (v _ {T} ^ {- i}) (s) \pi_ {T, K} ^ {i} (s) + \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {- i}) ^ {\top} \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) ^ {\top} \pi_ {T, K} ^ {i} (s) \right| \\ + \left| \sum_ {i = 1, 2} \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) \right|. \tag {32} \\ \end{array}
$$
We next bound each term on the RHS of the previous inequality. For the first term, we have by definition of  $\mathcal{T}^i (\cdot)$  that
$$
\begin{array}{l} \left| \max  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {- i}\right) ^ {\top} \mathcal {T} ^ {- i} \left(v _ {T} ^ {- i}\right) (s) \pi_ {T, K} ^ {i} (s) + \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {- i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) ^ {\top} \pi_ {T, K} ^ {i} (s) \right| \\ = \left| \max  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {- i}\right) ^ {\top} \mathcal {T} ^ {- i} \left(v _ {T} ^ {- i}\right) (s) \pi_ {T, K} ^ {i} (s) - \max  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {- i}\right) ^ {\top} \left[ - \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \right] ^ {\top} \pi_ {T, K} ^ {i} (s) \right| \\ \leq \max  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left| (\mu^ {- i}) ^ {\top} (\mathcal {T} ^ {- i} (v _ {T} ^ {- i}) (s) + \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) ^ {\top}) \pi_ {T, K} ^ {i} (s) \right| \\ \leq \max  _ {s, a ^ {i}, a ^ {- i}} \left| \mathcal {T} ^ {- i} \left(v _ {T} ^ {- i}\right) \left(s, a ^ {i}, a ^ {- i}\right) + \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) \left(s, a ^ {i}, a ^ {- i}\right) \right| \\ = \gamma \max _ {s, a ^ {i}, a ^ {- i}} \left| \mathbb {E} \left[ v _ {T} ^ {- i} (S _ {1}) + v _ {T} ^ {i} (S _ {1}) \right. \mid S _ {0} = s, A _ {0} ^ {i} = a ^ {i}, A _ {0} ^ {- i} = a _ {0} ^ {- i} \right] | \\ \leq \gamma \left\| v _ {T} ^ {- i} + v _ {T} ^ {i} \right\| _ {\infty}. \\ \end{array}
$$
For the second term on the RHS of Eq. (32), using the Lyapunov function  $V_{v,s}(\cdot ,\cdot)$  (defined in Appendix D.1), we have
$$
\begin{array}{l} \left| \sum_ {i = 1, 2} \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) \right| \\ = \sum_ {i = 1, 2} \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i} - \pi_ {T, K} ^ {i} (s)\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) + \left| \sum_ {i = 1, 2} \left(\pi_ {T, K} ^ {i} (s)\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) \right| \\ \leq \sum_ {i = 1, 2} \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left\{\left(\mu^ {i} - \pi_ {T, K} ^ {i} (s)\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) + \tau \nu \left(\mu^ {i}\right) - \tau \nu \left(\pi_ {T, K} ^ {i} (s)\right) \right\} \\ + 2 \tau \log (A _ {\max }) + \left| \sum_ {i = 1, 2} \left(\pi_ {T, K} ^ {i} (s)\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) \pi_ {T, K} ^ {- i} (s) \right| \\ \leq V _ {v _ {T}, s} \left(\pi_ {T, K} ^ {i} (s), \pi_ {T, K} ^ {- i} (s)\right) + 2 \tau \log \left(A _ {\max }\right) \\ + \max  _ {s, a ^ {i}, a ^ {- i}} \left| \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) \left(s, a ^ {i}, a ^ {- i}\right) + \mathcal {T} ^ {- i} \left(v _ {T} ^ {- i}\right) \left(s, a ^ {i}, a ^ {- i}\right) \right| \\ \leq V _ {v _ {T}, s} \left(\pi_ {T, K} ^ {i} (s), \pi_ {T, K} ^ {- i} (s)\right) + 2 \tau \log \left(A _ {\max }\right) + \gamma \| v _ {T} ^ {i} + v _ {T} ^ {- i} \| _ {\infty}. \\ \end{array}
$$
Using the previous two inequalities together in Eq. (32), we obtain
$$
\begin{array}{l} \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) \pi_ {T, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) \mu^ {- i} \\ \leq V _ {v _ {T}, s} \left(\pi_ {T, K} ^ {i} (s), \pi_ {T, K} ^ {- i} (s)\right) + 2 \gamma \| v _ {T} ^ {i} + v _ {T} ^ {- i} \| _ {\infty} + 2 \tau \log \left(A _ {\max }\right). \tag {33} \\ \end{array}
$$
The 4th Term on the RHS of Eq. (31). Using the definition of  $\mathcal{T}^i (\cdot)$ , we have
$$
\begin{array}{l} \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {T} ^ {i}) (s) \mu^ {- i} - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {*} ^ {i}) (s) \mu^ {- i} \right| \\ \leq \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left| \left(\mu^ {i}\right) ^ {\top} \left(\mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) (s) - \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) (s)\right) \mu^ {- i} \right| \\ \leq \max  _ {s, a ^ {i}, a ^ {- i}} \left| \mathcal {T} ^ {i} \left(v _ {T} ^ {i}\right) \left(s, a ^ {i}, a ^ {- i}\right) - \mathcal {T} ^ {i} \left(v _ {*} ^ {i}\right) \left(s, a ^ {i}, a ^ {- i}\right) \right| \\ \leq \gamma \| v _ {T} ^ {i} - v _ {*} ^ {i} \| _ {\infty}. \\ \end{array}
$$
Finally, using the upper bounds we obtained for all the terms on the RHS of Eq. (31), we have
$$
\begin{array}{l} \left\| v _ {*} ^ {- i} - v _ {\pi_ {T, K} ^ {- i}, *} ^ {- i} \right\| _ {\infty} \leq \gamma \left\| v _ {*, \pi_ {T, K} ^ {- i}} ^ {i} - v _ {*} ^ {i} \right\| _ {\infty} + 2 \gamma \| v _ {T} ^ {i} + v _ {T} ^ {- i} \| _ {\infty} + 2 \gamma \| v _ {T} ^ {i} - v _ {*} ^ {i} \| _ {\infty} \\ + \max  _ {s \in \mathcal {S}} V _ {v _ {T}, s} \left(\pi_ {T, K} ^ {i} (s), \pi_ {T, K} ^ {- i} (s)\right) + 2 \tau \log \left(A _ {\max }\right) \\ \leq \gamma \| v _ {*} ^ {- i} - v _ {\pi_ {T, K} ^ {- i}, *} ^ {- i} \| _ {\infty} + 2 \| v _ {T} ^ {i} + v _ {T} ^ {- i} \| _ {\infty} + 2 \| v _ {T} ^ {i} - v _ {*} ^ {i} \| _ {\infty} \\ + \max  _ {s \in \mathcal {S}} V _ {v _ {T}, s} (\pi_ {T, K} ^ {i} (s), \pi_ {T, K} ^ {- i} (s)) + 2 \tau \log (A _ {\max }) \\ \end{array}
$$
Rearranging terms and using  $\mathcal{L}_{\mathrm{sum}}(t)$  and  $\mathcal{L}_{\pi}(t,k)$  to simplify the notation, we obtain
$$
\left\| v _ {*} ^ {- i} - v _ {\pi_ {T, K} ^ {- i}, *} ^ {- i} \right\| _ {\infty} \leq \frac {1}{1 - \gamma} \left(2 \mathcal {L} _ {\operatorname {s u m}} (T) + 2 \| v _ {T} ^ {i} - v _ {*} ^ {i} \| _ {\infty} + \mathcal {L} _ {\pi} (T, K) + 2 \tau \log (A _ {\max})\right).
$$
Summing up both sides of the previous inequality for  $i \in \{1, 2\}$ , we have
$$
\sum_ {i = 1, 2} \left\| v _ {*} ^ {- i} - v _ {\pi_ {T, K} ^ {- i}, *} ^ {- i} \right\| _ {\infty} \leq \frac {2}{1 - \gamma} \left(2 \mathcal {L} _ {\mathrm {s u m}} (T) + \mathcal {L} _ {v} (T) + \mathcal {L} _ {\pi} (T, K) + 2 \tau \log (A _ {\mathrm {m a x}})\right).
$$
Using the previous inequality in Eq. (30), we have the desired result.
Combining the results in Lemma D.2 and Lemma D.3 in Eq. (28), we have the following result, which bounds the Nash gap in terms of the Lyapunov functions defined in Appendix D.1.
Lemma D.4. It holds that
$$
N G (\pi_ {T, K} ^ {1}, \pi_ {T, K} ^ {2}) \leq \frac {4}{1 - \gamma} \left(2 \mathcal {L} _ {s u m} (T) + \mathcal {L} _ {v} (T) + \mathcal {L} _ {\pi} (T, K) + 2 \tau \log (A _ {\max})\right).
$$
The next step is to bound the Lyapunov functions, which require us to analyze the outer loop and inner loop of Algorithm 2.
D.4 Analysis of the Outer Loop:  $v$ -Function Update
We first consider  $\mathcal{L}_v(t)$  and establish a one-step Lyapunov drift inequality for it.
Lemma D.5. It holds for all  $t \geq 0$  that
$$
\mathcal {L} _ {v} (t + 1) \leq \gamma \mathcal {L} _ {v} (t) + 4 \mathcal {L} _ {\text {s u m}} (t) + 2 \mathcal {L} _ {q} ^ {1 / 2} (t, K) + 4 \mathcal {L} _ {\pi} (t, K) + 6 \tau \log \left(A _ {\max }\right). \tag {34}
$$
Proof of Lemma D.5. For  $i \in \{1, 2\}$ , using the outer-loop update equation (cf. Algorithm 2 Line 8) and the fact that  $\mathcal{B}^i(v_*)^i = v_*^i$ , we have for any  $t \geq 0$  and  $s \in S$  that
$$
\begin{array}{l} v _ {t + 1} ^ {i} (s) - v _ {*} ^ {i} (s) = \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) - v _ {*} ^ {i} (s) \\ = \mathcal {B} ^ {i} (v _ {t} ^ {i}) (s) - \mathcal {B} ^ {i} (v _ {*} ^ {i}) (s) + \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) - \mathcal {B} ^ {i} (v _ {t} ^ {i}) (s). \\ \end{array}
$$
Since the minimax Bellman operator  $\mathcal{B}^i (\cdot)$  is a  $\gamma$ -contraction mapping with respect to  $\| \cdot \|_{\infty}$ , we have from the previous inequality that
$$
\begin{array}{l} \left| v _ {t + 1} ^ {i} (s) - v _ {*} ^ {i} (s) \right| \leq \left| \mathcal {B} ^ {i} \left(v _ {t} ^ {i}\right) (s) - \mathcal {B} ^ {i} \left(v _ {*} ^ {i}\right) (s) \right| + \left| \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) - \mathcal {B} ^ {i} \left(v _ {t} ^ {i}\right) (s) \right| \\ \leq \left\| \mathcal {B} ^ {i} \left(v _ {t} ^ {i}\right) - \mathcal {B} ^ {i} \left(v _ {*} ^ {i}\right) \right\| _ {\infty} + \left| \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) - \mathcal {B} ^ {i} \left(v _ {t} ^ {i}\right) (s) \right| \\ \leq \gamma \left\| v _ {t} ^ {i} - v _ {*} ^ {i} \right\| _ {\infty} + \left| \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) - \mathcal {B} ^ {i} \left(v _ {t} ^ {i}\right) (s) \right|. \tag {35} \\ \end{array}
$$
It remains to bound the second term on the RHS of Eq. (35). Using the definition of  $\mathcal{B}^i (\cdot)$ , we have
$$
\begin{array}{l} \left| \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) - \mathcal {B} ^ {i} \left(v _ {t} ^ {i}\right) (s) \right| \\ = \left| \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \mu^ {- i} \right| \\ \leq \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, K} ^ {- i} (s) - \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) \right| \\ + \left| \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, K} ^ {- i} (s) - \max _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} (\mu^ {i}) ^ {\top} \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \mu^ {- i} \right| \\ \leq \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i} - \pi_ {t, K} ^ {i} (s)\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, K} ^ {- i} (s) \\ + \left| \left(\pi_ {t, K} ^ {i} (s)\right) ^ {\top} \left(\mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, K} ^ {- i} (s) - q _ {t, K} ^ {i} (s)\right) \right| \\ + \left| \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, K} ^ {- i} (s) - \max  _ {\mu^ {i} \in \Delta (\mathcal {A} ^ {i})} \min  _ {\mu^ {- i} \in \Delta (\mathcal {A} ^ {- i})} \left(\mu^ {i}\right) ^ {\top} \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \mu^ {- i} \right| \\ \leq \left\| \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, K} ^ {- i} (s) - q _ {t, K} ^ {i} (s) \right\| _ {\infty} + 2 V _ {v _ {t}, s} (\pi_ {t, K} ^ {1} (s), \pi_ {t, K} ^ {2} (s)) \\ + 2 \gamma \| v _ {t} ^ {1} + v _ {t} ^ {2} \| _ {\infty} + 3 \tau \log (A _ {\max }), \\ \end{array}
$$
where the last line follows from Eq. (33). Using the previous inequality in Eq. (35), we obtain
$$
\begin{array}{l} \left\| v _ {t + 1} ^ {i} - v _ {*} ^ {i} \right\| _ {\infty} \leq \gamma \left\| v _ {t} ^ {i} - v _ {*} ^ {i} \right\| _ {\infty} + \max  _ {s \in \mathcal {S}} \left\| \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, K} ^ {- i} (s) - q _ {t, K} ^ {i} (s) \right\| _ {\infty} \\ + 2 \max _ {s \in \mathcal {S}} V _ {v _ {t}, s} (\pi_ {t, K} ^ {1} (s), \pi_ {t, K} ^ {2} (s)) + 2 \gamma \| v _ {t} ^ {1} + v _ {t} ^ {2} \| _ {\infty} + 3 \tau \log (A _ {\max}). \\ \end{array}
$$
$$
\mathcal {L} _ {v} (t + 1) \leq \gamma \mathcal {L} _ {v} (t) + 4 \mathcal {L} _ {\text {s u m}} (t) + 4 \mathcal {L} _ {\pi} (t, K) + 6 \tau \log \left(A _ {\max }\right)
$$
$$
+ \sum_ {i = 1, 2} \max  _ {s \in \mathcal {S}} \left\| \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, K} ^ {- i} (s) - q _ {t, K} ^ {i} (s) \right\| _ {\infty}.
$$
To bound the last term on the RHS of the previous inequality, observe that