SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} [ \mathrm {N G} (\pi_ {T, K} ^ {1}, \pi_ {T, K} ^ {2}) ] \lesssim \frac {A _ {\max } ^ {2} T}{\tau (1 - \gamma) ^ {3}} \left(\frac {1 + \gamma}{2}\right) ^ {T - 1} + \frac {L _ {\text {i n}} | \mathcal {S} | A _ {\max }}{(1 - \gamma) ^ {4} \alpha_ {k _ {0}} ^ {1 / 2} c _ {\alpha , \beta}} z _ {K} ^ {2} \alpha_ {K} ^ {1 / 2} + \frac {\tau \log (A _ {\max })}{(1 - \gamma) ^ {2}}.
$$
The proof of Theorem 3.1 (2) is complete.
D.7 Proof of All Supporting Lemmas
D.7.1 Proof of Lemma B.1
Lemma B.1 (1), (3), and (4) are identical to [10, Proposition 3]. We here only prove Lemma B.1 (2). Consider the Markov chain  $\{S_k\}$  induced by  $\pi_b$ . Since  $\{S_k\}$  is irreducible and aperiodic, there exists
a positive integer  $r_b$  such that  $P_{\pi_b}^{r_b}$  has strictly positive entries [93, Proposition 1.7]. Therefore, there exists  $\delta_b \in (0,1)$  such that
$$
P _ {\pi_ {b}} ^ {r _ {b}} (s, s ^ {\prime}) \geq \delta_ {b} \mu_ {b} (s ^ {\prime})
$$
for all  $(s, s')$ . In addition, the constant  $\rho_b$  introduced after Assumption 3.1 is explicitly given as  $\rho_b = \exp(-\delta_b / r_b)$ . The previous two equations are from the proof of the Markov chain convergence theorem presented in [93, Section 4.3]. Next, we consider the Markov chain  $\{S_k\}$  induced by an arbitrary  $\pi \in \Pi_{\tau}$ . Since
$$
\frac {\pi_ {b} (a | s)}{\pi (a | s)} = \frac {\pi_ {b} ^ {i} (a ^ {i} | s) \pi_ {b} ^ {- i} (a ^ {i} | s)}{\pi^ {i} (a ^ {i} | s) \pi^ {- i} (a ^ {i} | s)} \leq \frac {1}{\ell_ {\tau} ^ {2}}, \quad \forall a = (a ^ {i}, a ^ {- i}) \text {a n d} s,
$$
we have for any  $s, s' \in S$  and  $k \geq 1$  that
$$
\begin{array}{l} P _ {\pi_ {b}} ^ {k} (s, s ^ {\prime}) = \sum_ {s _ {0}} P _ {\pi_ {b}} ^ {k - 1} (s, s _ {0}) P _ {\pi_ {b}} (s _ {0}, s ^ {\prime}) \\ = \sum_ {s _ {0}} P _ {\pi_ {b}} ^ {k - 1} (s, s _ {0}) \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s _ {0}) P _ {a} (s _ {0}, s ^ {\prime}) \\ = \sum_ {s _ {0}} P _ {\pi_ {b}} ^ {k - 1} (s, s _ {0}) \sum_ {a \in \mathcal {A}} \frac {\pi_ {b} (a | s _ {0})}{\pi (a | s _ {0})} \pi (a | s _ {0}) P _ {a} (s _ {0}, s ^ {\prime}) \\ \leq \frac {1}{\ell_ {\tau} ^ {2}} \sum_ {s _ {0}} P _ {\pi_ {b}} ^ {k - 1} (s, s _ {0}) \sum_ {a \in \mathcal {A}} \pi (a | s _ {0}) P _ {a} \left(s _ {0}, s ^ {\prime}\right) \\ \leq \frac {1}{\ell_ {\tau} ^ {2}} \sum_ {s _ {0}} P _ {\pi_ {b}} ^ {k - 1} (s, s _ {0}) P _ {\pi} \left(s _ {0}, s ^ {\prime}\right) \\ = \frac {1}{\ell_ {\tau} ^ {2}} \left[ P _ {\pi_ {b}} ^ {k - 1} P _ {\pi} \right] (s, s ^ {\prime}). \\ \end{array}
$$
Since the previous inequality holds for all  $s$  and  $s'$ , we in fact have  $\ell_{\tau}^{2}P_{\pi_b}^k \leq P_{\pi_b}^{k-1}P_{\pi}$  (which is an entry-wise inequality). Repeatedly using the previous inequality, we obtain
$$
\ell_ {\tau} ^ {2 k} P _ {\pi_ {b}} ^ {k} \leq P _ {\pi} ^ {k},
$$
$$
\begin{array}{l} P _ {\pi} ^ {r _ {b}} (s, s ^ {\prime}) \geq \ell_ {\tau} ^ {2 r _ {b}} P _ {\pi_ {b}} ^ {r _ {b}} (s, s ^ {\prime}) \\ \geq \delta_ {b} \ell_ {\tau} ^ {2 r _ {b}} \mu_ {b} (s ^ {\prime}) \\ \geq \delta_ {b} \ell_ {\tau} ^ {2 r _ {b}} \frac {\mu_ {b} \left(s ^ {\prime}\right)}{\mu_ {\pi} \left(s ^ {\prime}\right)} \mu_ {\pi} \left(s ^ {\prime}\right) \\ \geq \delta_ {b} \ell_ {\tau} ^ {2 r _ {b}} \mu_ {b, \min } \mu_ {\pi} (s ^ {\prime}). \\ \end{array}
$$
Following the proof of the Markov chain convergence theorem in [93, Section 4.3], we have
$$
\left\| P _ {\pi} ^ {k} (s, \cdot) - \mu_ {\pi} (\cdot) \right\| _ {\mathrm {T V}} \leq \left(1 - \delta_ {b} \ell_ {\tau} ^ {2 r _ {b}} \mu_ {b, \min }\right) ^ {k / r _ {b} - 1}, \quad \forall s \in S, \pi \in \Pi_ {\tau}. \tag {55}
$$
Since  $A_{\mathrm{max}} \geq 2$  (otherwise there is no decision to make in this stochastic game), we have  $\ell_{\tau}^{2} \leq \frac{1}{2}$ . It follows that  $1 - \delta_b \ell_{\tau}^{2r_b} \mu_{b,\min} > 1/2$ . Using the previous inequality in Eq. (55), we have
$$
\begin{array}{l} \sup  _ {\pi \in \Pi_ {\tau}} \max  _ {s \in \mathcal {S}} \| P _ {\pi} ^ {k} (s, \cdot) - \mu_ {\pi} (\cdot) \| _ {\mathrm {T V}} \leq 2 \left(1 - \delta_ {b} \ell_ {\tau} ^ {2 r _ {b}} \mu_ {b, \min }\right) ^ {k / r _ {b}} \\ \leq 2 \exp \left(- \delta_ {b} \ell_ {\tau} ^ {2 r _ {b}} \mu_ {b, \min } k / r _ {b}\right) \\ = 2 \rho_ {b} ^ {\ell_ {\tau} ^ {2 r _ {b}} \mu_ {b, \min } k} \quad (\text {R e c a l l t h a t} \rho_ {b} = \exp (- \delta_ {b} / r _ {b})) \\ = 2 \rho_ {\tau} ^ {k}. \\ \end{array}
$$
We next compute the mixing time. Using the previous inequality and the definition of the total variation distance, we have
$$
\sup  _ {\pi \in \Pi_ {\tau}} \max  _ {s \in \mathcal {S}} \| P _ {\pi} ^ {k} (s, \cdot) - \mu_ {\pi} (\cdot) \| _ {\mathrm {T V}} \leq \eta
$$
$$
k \geq \frac {\log (2 / \eta)}{\log (1 / \rho_ {\delta})} = \frac {1}{\ell_ {\tau} ^ {2 r _ {b}} \mu_ {b , \mathrm {m i n}}} \frac {\log (2 / \eta)}{\log (1 / \rho_ {b})} \geq \frac {t _ {\pi_ {b} , \eta}}{\ell_ {\tau} ^ {2 r _ {b}} \mu_ {b , \mathrm {m i n}}}.
$$
D.7.2 Proof of Lemma D.11
For any  $k \geq z_k$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \left(F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right) ^ {\top} \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) \right] \right. \\ = \underbrace {\mathbb {E} \left[ \left(F ^ {i} \left(q _ {k - z _ {k}} ^ {i} , S _ {k} , A _ {k} ^ {i} , A _ {k} ^ {- i} , S _ {k + 1}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right)\right) ^ {\top} \left(q _ {k - z _ {k}} ^ {i} - \bar {q} _ {k - z _ {k}} ^ {i}\right) \right]} _ {N _ {1}} \\ + \underbrace {\mathbb {E} [ (F ^ {i} (q _ {k - z _ {k}} ^ {i} , S _ {k} , A _ {k} ^ {i} , A _ {k} ^ {- i} , S _ {k + 1}) - \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k - z _ {k}} ^ {i})) ^ {\top} (q _ {k} ^ {i} - q _ {k - z _ {k}} ^ {i}) ]} _ {N _ {2}} \\ + \underbrace {\mathbb {E} [ (F ^ {i} (q _ {k - z _ {k}} ^ {i} , S _ {k} , A _ {k} ^ {i} , A _ {k} ^ {- i} , S _ {k + 1}) - \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k - z _ {k}} ^ {i})) ^ {\top} (\bar {q} _ {k - z _ {k}} ^ {i} - \bar {q} _ {k} ^ {i}) ]} _ {N _ {3}} \\ + \underbrace {\mathbb {E} \left[ \left(F ^ {i} \left(q _ {k} ^ {i} , S _ {k} , A _ {k} ^ {i} , A _ {k} ^ {- i} , S _ {k + 1}\right) - F ^ {i} \left(q _ {k - z _ {k}} ^ {i} , S _ {k} , A _ {k} ^ {i} , A _ {k} ^ {- i} , S _ {k + 1}\right)\right) ^ {\top} \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) \right]} _ {N _ {4}} \\ + \underbrace {\mathbb {E} \left[ \left(\bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) - \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right)\right) ^ {\top} \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) \right]} _ {N _ {5}}. \tag {56} \\ \end{array}
$$
To bound the terms  $N_{1}$  to  $N_{5}$  on the RHS of the previous inequality, the following lemma is needed.
Lemma D.13. For any positive integers  $k_{1} \leq k_{2}$ , we have (1)  $\| q_{k_2}^i - q_{k_1}^i \|_\infty \leq \frac{2\alpha_{k_1,k_2-1}}{1-\gamma}$ , and (2)  $\max_{s \in \mathcal{S}} \| \pi_{k_2}^i(s) - \pi_{k_1}^i(s) \|_1 \leq 2\beta_{k_1,k_2-1}$ .
Proof of Lemma D.13. For any  $k \in [k_1, k_2 - 1]$ , we have by Eq. (41) that
$$
\| q _ {k + 1} ^ {i} - q _ {k} ^ {i} \| _ {\infty} = \alpha_ {k} \| F ^ {i} (q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}) \| _ {\infty} \leq \frac {2 \alpha_ {k}}{1 - \gamma}.
$$
It follows that  $\| q_{k_2}^i -q_{k_1}^i\|_\infty \leq \frac{2\alpha_{k_1,k_2 - 1}}{1 - \gamma}$ . Similarly, for any  $k\in [k_1,k_2 - 1]$  and  $s\in S$ , we have
$$
\left\| \pi_ {k + 1} ^ {i} (s) - \pi_ {k} ^ {i} (s) \right\| _ {1} = \beta_ {k} \left\| \sigma_ {\tau} \left(q _ {k} ^ {i} (s)\right) - \pi_ {k} ^ {i} (s) \right\| _ {1} \leq 2 \beta_ {k},
$$
which implies  $\max_{s\in S}\| \pi_{k_2}^i (s) - \pi_{k_1}^i (s)\| _1\leq 2\beta_{k_1,k_2 - 1}$ .
We next bound the terms  $N_{1}$  to  $N_{5}$ . Let  $\mathcal{F}_k$  be the  $\sigma$ -algebra generated the sequence of random variables  $\{S_0, A_0^i, A_0^{-i}, \dots, S_{k-1}, A_{k-1}^i, A_{k-1}^{-i}, S_k\}$ .
The Term  $N_{1}$ . Using the tower property of conditional expectations, we have
$$
\begin{array}{l} N _ {1} = \mathbb {E} \left[ \left(F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right)\right) ^ {\top} \left(q _ {k - z _ {k}} ^ {i} - \bar {q} _ {k - z _ {k}} ^ {i}\right) \right] \\ = \mathbb {E} \left[ \left(\mathbb {E} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \mid \mathcal {F} _ {k - z _ {k}} \right] - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right)\right) ^ {\top} \left(q _ {k - z _ {k}} ^ {i} - \bar {q} _ {k - z _ {k}} ^ {i}\right) \right] \\ \leq \mathbb {E} [ \| \mathbb {E} [ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \mid \mathcal {F} _ {k - z _ {k}} ] - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} \| q _ {k - z _ {k}} ^ {i} - \bar {q} _ {k - z _ {k}} ^ {i} \| _ {\infty} ] (57) \\ \leq \frac {2}{1 - \gamma} \mathbb {E} \left[ \left\| \mathbb {E} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \mid \mathcal {F} _ {k - z _ {k}} \right] - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \right\| _ {1} \right] \quad (\text {L e m m a D . 1}) \\ \leq \frac {2}{1 - \gamma} \mathbb {E} [ \| \bar {F} _ {k} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} ] \\ + \frac {2}{1 - \gamma} \mathbb {E} [ \| \mathbb {E} [ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \mid \mathcal {F} _ {k - z _ {k}} ] - \bar {F} _ {k} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} ]. (58) \\ \end{array}
$$
We next bound the two terms on the RHS of the previous inequality. Observe that
$$
\begin{array}{l} \left\| \bar {F} _ {k} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \right\| _ {1} \\ = \sum_ {s, a ^ {i}} | [ \bar {F} _ {k} ^ {i} (q _ {k - z _ {k}} ^ {i}) ] (s, a ^ {i}) - [ \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k - z _ {k}} ^ {i}) ] (s, a ^ {i}) | \\ = \sum_ {s, a ^ {i}} \left| \right.\left[ \right. \mathbb {E} _ {k} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S, A ^ {i}, A ^ {- i}, S ^ {\prime}\right)\right] (s, a ^ {i}) - \left[ \mathbb {E} _ {k - z _ {k}} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S, A ^ {i}, A ^ {- i}, S ^ {\prime}\right)\right] (s, a ^ {i}) \right.\left. \right|, \\ \end{array}
$$
where we use  $\mathbb{E}_k[\cdot ]$  to denote  $\mathbb{E}_{S\sim \mu_k(\cdot),A^i\sim \pi_k^i (\cdot |S_0),A^{-i}\sim \pi_k^{-i}(\cdot |S_0),S'\sim p(\cdot |S_0,A_0^i,A_0^{-i})}[\cdot ]$  for ease of presentation. To proceed, recall the following equivalent definition of total variation distance between probability measures  $p_1,p_2$ :
$$
\| p _ {1} - p _ {2} \| _ {\mathrm {T V}} = \frac {1}{2} \sup  _ {f: \| f \| _ {\infty} \leq 1} | \mathbb {E} _ {p _ {1}} [ f ] - \mathbb {E} _ {p _ {2}} [ f ] |.
$$
$$
\begin{array}{l} \left. \left| [ \mathbb {E} _ {k} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S, A ^ {i}, A ^ {- i}, S ^ {\prime}\right) \right] (s, a ^ {i}) - [ \mathbb {E} _ {k - z _ {k}} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S, A ^ {i}, A ^ {- i}, S ^ {\prime}\right) \right] (s, a ^ {i}) \right| \right. \\ \leq \max _ {\bar {s}, \bar {a} ^ {i}, \bar {a} ^ {- i}, \bar {s} ^ {\prime}} \left| \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, \bar {s}, \bar {a} ^ {i}, \bar {a} ^ {- i}, \bar {s} ^ {\prime}\right) \right] (s, a ^ {i}) \right| \\ \times \sum_ {\tilde {s}, \tilde {a} ^ {i}, \tilde {a} ^ {- i}} \left| \mu_ {k} (\tilde {s}) \pi_ {k} ^ {i} (\tilde {a} ^ {i} | \tilde {s}) \pi_ {k} ^ {- i} (\tilde {a} ^ {- i} | \tilde {s}) - \mu_ {k - z _ {k}} (\tilde {s}) \pi_ {k - z _ {k}} ^ {i} (\tilde {a} ^ {i} | \tilde {s}) \pi_ {k - z _ {k}} ^ {- i} (\tilde {a} ^ {- i} | \tilde {s}) \right| \\ \leq \frac {1}{1 - \gamma} \left(\| \mu_ {k} - \mu_ {k - z _ {k}} \| _ {1} + \max  _ {s} \| \pi_ {k} ^ {i} (s) - \pi_ {k - z _ {k}} ^ {i} (s) \| _ {1} + \max  _ {s} \| \pi_ {k} ^ {- i} (s) - \pi_ {k - z _ {k}} ^ {- i} (s) \| _ {1}\right) \\ \leq \frac {2 L _ {p}}{1 - \gamma} \left(\max  _ {s} \| \pi_ {k} ^ {i} (s) - \pi_ {k - z _ {k}} ^ {i} (s) \| _ {1} + \max  _ {s} \| \pi_ {k} ^ {- i} (s) - \pi_ {k - z _ {k}} ^ {- i} (s) \| _ {1}\right) (Lemma B.1) \\ \leq \frac {8 L _ {p} \beta_ {k - z _ {k} , k - 1}}{1 - \gamma}. (Lemma D.13) \\ \end{array}
$$
$$
\begin{array}{l} \| \bar {F} _ {k} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} = \sum_ {s, a ^ {i}} | [ \bar {F} _ {k} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) ] (s, a ^ {i}) - [ \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) ] (s, a ^ {i}) | \\ \leq \frac {8 | \mathcal {S} | A _ {\max } L _ {p} \beta_ {k - z _ {k} , k - 1}}{1 - \gamma}. \tag {59} \\ \end{array}
$$
It remains to bound the second term on the RHS of Eq. (58). Recall that we denote  $P_{\pi} \in \mathbb{R}^{|S| \times |S|}$  as the transition probability matrix of the Markov chain  $\{S_k\}$  induced by a joint policy  $\pi$ . Using the definition of conditional expectations, we have
$$
\begin{array}{l} \left\| \mathbb {E} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \mid \mathcal {F} _ {k - z _ {k}} \right] - \bar {F} _ {k} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \right\| _ {1} \\ = \left\| \sum_ {s \in \mathcal {S}} \left[ \left(\prod_ {j = k + 1} ^ {k + z _ {k}} P _ {\pi_ {j - z _ {k}}}\right) (S _ {k - z _ {k}}, s) - \mu_ {k} (s) \right] \sum_ {a ^ {i}} \pi_ {k} ^ {i} (a ^ {i} | s) \sum_ {a ^ {- i}} \pi_ {k} ^ {- i} (a ^ {- i} | s) \right. \\ \left. \times \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a ^ {i}, a ^ {- i}\right) F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, s, a ^ {i}, a ^ {- i}, s ^ {\prime}\right) \right\rVert_ {1} \\ \leq \frac {2}{1 - \gamma} \sum_ {s \in \mathcal {S}} \left| \left(\prod_ {j = k + 1} ^ {k + z _ {k}} P _ {\pi_ {j - z _ {k}}}\right) \left(S _ {k - z _ {k}}, s\right) - \mu_ {k} (s) \right| (Lemma D.9) \\ \leq \frac {2}{1 - \gamma} \left\{\sum_ {s \in \mathcal {S}} \left| \left(\prod_ {j = k + 1} ^ {k + z _ {k}} P _ {\pi_ {j - z _ {k}}}\right) \left(S _ {k - z _ {k}}, s\right) - P _ {\pi_ {k}} ^ {z _ {k}} \left(S _ {k - z _ {k}}, s\right) \right. \right| \\ \left. \right.\left. + \sum_ {s \in \mathcal {S}} \left| P _ {\pi_ {k}} ^ {z _ {k}} \left(S _ {k - z _ {k}}, s\right) - \mu_ {k} (s) \right|\right\} \\ \leq \frac {2}{1 - \gamma} \left\{\left\| \prod_ {j = k + 1} ^ {k + z _ {k}} P _ {\pi_ {j - z _ {k}}} - P _ {\pi_ {k}} ^ {z _ {k}} \right\| _ {\infty} + 2 \rho_ {\tau} ^ {z _ {k}} \right\}, (60) \\ \end{array}
$$
where the last line follows from Lemma B.1 (2). Observe that
$$
\left\| \prod_ {j = k + 1} ^ {k + z _ {k}} P _ {\pi_ {j - z _ {k}}} - P _ {\pi_ {k}} ^ {z _ {k}} \right\| _ {\infty} = \left\| \sum_ {\ell = 1} ^ {z _ {k}} \left(\prod_ {j = k + 1} ^ {k - \ell + 1 + z _ {k}} P _ {\pi_ {j - z _ {k}}} P _ {\pi_ {k}} ^ {\ell - 1} - \prod_ {j = k + 1} ^ {k - \ell + z _ {k}} P _ {\pi_ {j - z _ {k}}} P _ {\pi_ {k}} ^ {\ell}\right) \right\| _ {\infty}
$$
$$
\begin{array}{l} = \left\| \sum_ {\ell = 1} ^ {z _ {k}} \left(\prod_ {j = k + 1} ^ {k - \ell + z _ {k}} P _ {\pi_ {j - z _ {k}}} \left(P _ {\pi_ {k - \ell + 1}} - P _ {\pi_ {k}}\right) P _ {\pi_ {k}} ^ {\ell - 1}\right) \right\| _ {\infty} \\ \leq \sum_ {\ell = 1} ^ {z _ {k}} \left\| \prod_ {j = k + 1} ^ {k - \ell + z _ {k}} P _ {\pi_ {j - z _ {k}}} \right\| _ {\infty} \| P _ {\pi_ {k - \ell + 1}} - P _ {\pi_ {k}} \| _ {\infty} \| P _ {\pi_ {k}} ^ {\ell - 1} \| _ {\infty} \\ \leq \sum_ {\ell = 1} ^ {z _ {k}} \| P _ {\pi_ {k - \ell + 1}} - P _ {\pi_ {k}} \| _ {\infty}. \\ \end{array}
$$
Since  $P_{\pi}$  as a function of  $\pi$  is 1-Lipschitz continuous with respect to the  $\ell_{\infty}$ -norm, we have
$$
\begin{array}{l} \left\| \prod_ {j = k + 1} ^ {k + z _ {k}} P _ {\pi_ {j - z _ {k}}} - P _ {\pi_ {k}} ^ {z _ {k}} \right\| _ {\infty} \leq \sum_ {\ell = 1} ^ {z _ {k}} \max  _ {s \in \mathcal {S}} \| \pi_ {k - \ell + 1} (s) - \pi_ {k} (s) \| _ {1} \\ = \sum_ {\ell = 1} ^ {z _ {k}} \max  _ {s \in \mathcal {S}} \left(\| \pi_ {k - \ell + 1} ^ {- i} (s) - \pi_ {k} ^ {- i} (s) \| _ {1} + \| \pi_ {k - \ell + 1} ^ {i} (s) - \pi_ {k} ^ {i} (s) \| _ {1}\right) \\ \leq 4 z _ {k} \beta_ {k - z _ {k}, k - 1}. \tag {Lemma D.13} \\ \end{array}
$$
Using the previous inequality in Eq. (60), we have
$$
\begin{array}{l} \| \mathbb {E} \left[ F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \mid \mathcal {F} _ {k - z _ {k}} \right] - \bar {F} _ {k} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} \\ \leq \frac {2}{1 - \gamma} \left(4 z _ {k} \beta_ {k - z _ {k}, k - 1} + 2 \rho_ {\tau} ^ {z _ {k}}\right) \\ \leq \frac {2}{1 - \gamma} \left(4 z _ {k} \beta_ {k - z _ {k}, k - 1} + \beta_ {k}\right) \quad (\text {D e f i n i t i o n} z _ {k}) \\ \leq \frac {1 0 z _ {k} \beta_ {k - z _ {k} , k - 1}}{1 - \gamma}. \quad (z _ {k} \geq 1) \\ \end{array}
$$
Using the previous inequality and Eq. (59) together in Eq. (58), we obtain
$$
N _ {1} \leq \frac {1 6 L _ {p} | \mathcal {S} | A _ {\max} \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {2 0 z _ {k} \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} \leq \frac {3 6 L _ {p} | \mathcal {S} | A _ {\max} z _ {k} \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}}.
$$
The Term  $N_{2}$ . For any  $k \geq z_{k}$ , we have by Lemma D.13 that
$$
\begin{array}{l} N _ {2} \leq \mathbb {E} [ \| F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} \| q _ {k} ^ {i} - q _ {k - z _ {k}} ^ {i} \| _ {\infty} ] \\ \leq \frac {2 \alpha_ {k - z _ {k} , k - 1}}{1 - \gamma} \mathbb {E} [ \| F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \| _ {1} + \| \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} ]. \tag {61} \\ \end{array}
$$
Using the definition of  $F^i(\cdot)$ , we have
$$
\begin{array}{l} \left\| F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \right\| _ {1} \\ = \sum_ {s, a ^ {i}} \mathbb {1} _ {\left\{\left(s, a ^ {i}\right) = \left(S _ {k}, A _ {k} ^ {i}\right) \right\}} \left| R _ {i} \left(S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right) + \gamma v ^ {i} \left(S _ {k + 1}\right) - q _ {k} ^ {i} \left(S _ {k}, A _ {k} ^ {i}\right) \right| \\ = \left| R _ {i} \left(S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right) + \gamma v ^ {i} \left(S _ {k + 1}\right) - q _ {k} ^ {i} \left(S _ {k}, A _ {k} ^ {i}\right) \right| \\ \leq 1 + \frac {\gamma}{1 - \gamma} + \frac {1}{1 - \gamma} \\ = \frac {2}{1 - \gamma}. \tag {62} \\ \end{array}
$$
Moreover, we have by Jensen's inequality that
$$
\left\| \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \right\| _ {1} \leq \frac {2}{1 - \gamma}. \tag {63}
$$
Using Eqs. (62) and (63) together in Eq. (61), we have
$$
N _ {2} \leq \frac {8 \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}}.
$$
The Term  $N_{3}$ . For any  $k \geq z_k$ , we have
$$
\begin{array}{l} N _ {3} \leq \mathbb {E} [ \| F ^ {i} (q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}) - \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k - z _ {k}} ^ {i}) \| _ {1} \| \bar {q} _ {k - z _ {k}} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {\infty} ] \\ \leq \mathbb {E} [ (\| F ^ {i} (q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}) \| _ {1} + \| \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k - z _ {k}} ^ {i}) \| _ {1}) \| \bar {q} _ {k - z _ {k}} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {\infty} ] \\ \leq \frac {4}{1 - \gamma} \mathbb {E} \left[ \| \bar {q} _ {k - z _ {k}} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {\infty} \right]. \tag {Eqs. (62) and (63)} \\ \end{array}
$$
$$
\begin{array}{l} \| \bar {q} _ {k - z _ {k}} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {\infty} = \max  _ {s \in \mathcal {S}} \| \mathcal {T} ^ {i} (v ^ {i}) (s) (\pi_ {k} ^ {- i} (s) - \pi_ {k - z _ {k}} ^ {- i} (s)) \| _ {\infty} \\ \leq \max  _ {s \in \mathcal {S}} \| \mathcal {T} ^ {i} (v ^ {i}) (s) \| _ {1, \infty} \| \pi_ {k} ^ {- i} (s) - \pi_ {k - z _ {k}} ^ {- i} (s) \| _ {1} \\ \leq \frac {2 \beta_ {k - z _ {k} , k - 1}}{1 - \gamma}, \\ \end{array}
$$
where the last line follows from Lemma D.13 and
$$
\| \mathcal {T} ^ {i} (v ^ {i}) (s) \| _ {1, \infty} \leq \max  _ {s, a ^ {i}, a ^ {- i}} | \mathcal {T} ^ {i} (v ^ {i}) (s, a ^ {i}, a ^ {- i}) | \leq \frac {1}{1 - \gamma}.
$$
$$
N _ {3} \leq \frac {8 \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}}.
$$
The Term  $N_4$ . For any  $k \geq z_k$ , we have
$$
\begin{array}{l} N _ {4} \leq \mathbb {E} [ \| F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \| _ {1} \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {\infty} ] \\ \leq \frac {2}{1 - \gamma} \mathbb {E} [ \| F ^ {i} (q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}) - F ^ {i} (q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}) \| _ {1} ], \\ \end{array}
$$
where the last line follows from Lemma D.1. Using the definition of  $F^i(\cdot)$ , we have
$$
\begin{array}{l} \left\| F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - F ^ {i} \left(q _ {k - z _ {k}} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \right\| _ {1} \\ = \sum_ {s, a ^ {i}} \mathbb {1} _ {\left\{(s, a ^ {i}) = \left(S _ {k}, A _ {k} ^ {i}\right) \right\}} \left| q _ {k - z _ {k}} ^ {i} \left(S _ {k}, A _ {k} ^ {i}\right) - q _ {k} ^ {i} \left(S _ {k}, A _ {k} ^ {i}\right) \right| \\ = \left| q _ {k - z _ {k}} ^ {i} \left(S _ {k}, A _ {k} ^ {i}\right) - q _ {k} ^ {i} \left(S _ {k}, A _ {k} ^ {i}\right) \right| \\ \leq \left\| q _ {k - z _ {k}} ^ {i} - q _ {k} ^ {i} \right\| _ {\infty} \\ \leq \frac {2 \alpha_ {k - z _ {k} , k - 1}}{1 - \gamma}. \tag {Lemma D.13} \\ \end{array}
$$
$$
N _ {4} \leq \frac {4 \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}}.
$$
The Term  $N_{5}$ . For any  $k \geq z_{k}$ , we have
$$
\begin{array}{l} N _ {5} \leq \mathbb {E} \left[ \| \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {\infty} \right] \\ \leq \frac {2}{1 - \gamma} \mathbb {E} \left[ \| \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} \right] (Lemma D.1) \\ \leq \frac {2}{1 - \gamma} \mathbb {E} [ \| \bar {F} _ {k} ^ {i} (q _ {k} ^ {i}) - \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k} ^ {i}) \| _ {1} + \| \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k} ^ {i}) - \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k - z _ {k}} ^ {i}) \| _ {1} ] \\ \leq \frac {1 6 L _ {p} | \mathcal {S} | A _ {\max } \beta_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} + \frac {2}{1 - \gamma} \mathbb {E} [ \| \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k} ^ {i}\right) - \bar {F} _ {k - z _ {k}} ^ {i} \left(q _ {k - z _ {k}} ^ {i}\right) \| _ {1} ], (64) \\ \end{array}
$$
where the last line follows from the same analysis as we obtain Eq. (59). As for the second term on the RHS of Eq. (64), using the definition of  $\bar{F}_k^i (\cdot)$ , we have
$$
\| \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k} ^ {i}) - \bar {F} _ {k - z _ {k}} ^ {i} (q _ {k - z _ {k}} ^ {i}) \| _ {1} = \sum_ {s \in \mathcal {S}} \mu_ {k - z _ {k}} (s) \sum_ {a ^ {i}} \pi_ {k - z _ {k}} ^ {i} (a ^ {i} \mid s) | q _ {k} ^ {i} (s, a ^ {i}) - q _ {k - z _ {k}} ^ {i} (s, a ^ {i}) |
$$
$$
\begin{array}{l} \leq \left\| q _ {k} ^ {i} - q _ {k - z _ {k}} ^ {i} \right\| _ {\infty} \\ \leq \frac {2 \alpha_ {k - z _ {k} , k - 1}}{1 - \gamma}. \tag {Lemma D.13} \\ \end{array}
$$
Using the previous inequality in Eq. (64), we obtain