SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To bound  $\mathbf{term}_5$ , by Lemma 14 and Lemma 18, we have
$$
\left| v _ {\tau} ^ {s} \right| = \mathcal {O} \left(\ln (A \tau)\right) \cdot \left\| \hat {z} _ {\tau \star} ^ {s} - \hat {z} _ {\tau + 1 \star} ^ {s} \right\| _ {1} = \mathcal {O} \left(\ln^ {4} (S A t / \delta) L ^ {2} \cdot \tau^ {- k _ {\alpha} + k _ {\epsilon}}\right).
$$
Therefore, by Lemma 1,
$$
\mathbf {t e r m} _ {5} = \sum_ {i = 1} ^ {\tau} w _ {\tau} ^ {i} v _ {i} ^ {s} = \mathcal {O} \left(\ln^ {5} (S A t / \delta) L ^ {2} \cdot \tau^ {- k _ {\alpha} + k _ {\eta} + 2 k _ {\epsilon}}\right).
$$
Combining all the terms with union bound over  $s \in S$  and  $\tau \geq 1$  finishes the proof.
Lemma 18. For any sand  $\tau \geq 0$  such that  $t_{\tau}(s) \leq t$ ,  $\| \hat{z}_{\tau \star}^{s} - \hat{z}_{\tau + 1 \star}^{s}\|_{1} = \mathcal{O}\left(\ln^{3}(SA t / \delta)L^{2} \cdot \tau^{-k_{\alpha} + k_{\epsilon}}\right)$ .
Proof. The bound holds trivially when  $\tau \leq 2L$ . Below we focus on the case with  $\tau > 2L$ . By exactly the same arguments as in the proof of Lemma 15, we have an inequality similar to Eq. (10):
$$
\begin{array}{l} \left\| z _ {\tau \star} ^ {s} - z _ {\tau + 1 \star} ^ {s} \right\| _ {1} \\ = \mathcal {O} \left(\frac {1}{\epsilon_ {\tau + 1}} \sup  _ {x ^ {s}, y ^ {s}} \| \nabla f _ {\tau} ^ {s} \left(x ^ {s}, y ^ {s}\right) - \nabla f _ {\tau + 1} ^ {s} \left(x ^ {s}, y ^ {s}\right) \| _ {\infty} + \frac {\ln^ {1 / 2} (A \tau)}{\sqrt {\epsilon_ {\tau + 1}} \tau^ {3 / 2}}\right) \\ \leq \mathcal {O} \left(\frac {1}{\epsilon_ {\tau + 1}} \sup  _ {s ^ {\prime}} \left| V _ {t _ {\tau} (s)} ^ {s ^ {\prime}} - V _ {t _ {\tau + 1} (s)} ^ {s ^ {\prime}} \right| + \frac {(\epsilon_ {\tau} - \epsilon_ {\tau + 1}) \ln (A \tau)}{\epsilon_ {\tau + 1}} + \frac {\ln^ {1 / 2} (A \tau)}{\sqrt {\epsilon_ {\tau + 1}} \tau^ {3 / 2}}\right) \\ \leq \mathcal {O} \left(\frac {1}{\epsilon_ {\tau + 1}} \sup  _ {s ^ {\prime}} \left| V _ {t _ {\tau} (s)} ^ {s ^ {\prime}} - V _ {t _ {\tau + 1} (s)} ^ {s ^ {\prime}} \right| + \frac {\ln (A \tau)}{\tau}\right). \tag {12} \\ \end{array}
$$
Since  $t_{\tau}(s) \leq t$  and we assume that every state is visited at least once in  $6L\log (St / \delta)$  steps (Corollary 1), we have that for any state  $s^{\prime}$ ,  $n_{t_{\tau}(s)}^{s^{\prime}} \geq \frac{t_{\tau}(s)}{6L\log(St / \delta)} - 1$ . Thus, whenever  $V_{t}^{s^{\prime}}$  updates between  $t_{\tau}(s)$  and  $t_{\tau + 1}(s)$ , the change is upper bounded by  $\frac{1}{1 - \gamma}\left(\frac{t_{\tau}(s)}{6L\log(St / \delta)} - 1\right)^{-k_{\alpha}}$ . Besides, between  $t_{\tau}(s)$  and  $t_{\tau + 1}(s)$ ,  $V_{t}^{s^{\prime}}$  can change at most  $6L\log (St / \delta)$  times. Therefore,
$$
\begin{array}{l} \left| V _ {t _ {\tau} (s)} ^ {s ^ {\prime}} - V _ {t _ {\tau + 1} (s)} ^ {s ^ {\prime}} \right| \\ \leq \frac {1}{1 - \gamma} \times 6 L \log (S t / \delta) \times \left(\frac {t _ {\tau} (s)}{6 L \log (S t / \delta)} - 1\right) ^ {- k _ {\alpha}} \leq \frac {1}{1 - \gamma} \times 6 L \log (S t / \delta) \times \left(\frac {\tau}{6 L \log (S t / \delta)} - 1\right) ^ {- k _ {\alpha}} \\ = \mathcal {O} \left(\frac {L ^ {2} \ln^ {2} (S t / \delta) \tau^ {- k _ {\alpha}}}{1 - \gamma}\right), \tag {13} \\ \end{array}
$$
where the last inequality holds since  $k_{\alpha} < 1$ . Combining Eq. (12) and Eq. (13) with the fact that  $\epsilon_{\tau} = \frac{1}{1 - \gamma}\tau^{-k_{e}}$  finishes the proof.
D.4 Part III. Value Convergence
For positive integers  $\tau \geq i$ , we define  $\alpha_{\tau}^{i} = \alpha_{i}\prod_{j = i + 1}^{\tau}(1 - \alpha_{j})$
Lemma 19 (weighted regret bound). With probability  $1 - \mathcal{O}(\delta)$ , for any  $s$ , any visitation count  $\tau \geq \tau_0$ , and any  $x^s \in \Omega_{\tau + 1}$
$$
\sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(f _ {i} ^ {s} (\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}) - f _ {i} ^ {s} (x ^ {s}, \hat {y} _ {i} ^ {s})\right) \leq \mathcal {O} \left(\frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k ^ {\prime}}}{1 - \gamma}\right).
$$
where  $k^{\prime} = \min \left\{k_{\eta},k_{\beta},k_{\alpha} - k_{\beta}\right\}$
Proof. We will be considering a weighted sum of the instantaneous regret bound established in Lemma 16. However, notice that for  $f_{i}^{s}$ , Lemma 16 only provides a regret bound with comparators in  $\Omega_{i+1}$ . Therefore, for a fixed  $x^{s} \in \Omega_{\tau+1}$ , we define the following auxiliary comparators for all  $i = 1, \ldots, \tau$ :
$$
\widetilde {x} _ {i} ^ {s} = \frac {p _ {i}}{A} \mathbf {1} + (1 - p _ {i}) x ^ {s}
$$
where  $p_i \triangleq \frac{(\tau + 1)^2 - (i + 1)^2}{(i + 1)^2[(\tau + 1)^2 - 1]}$ . Since  $x^s \in \Omega_{\tau + 1}$ , we have that for any  $a, \widetilde{x}_{i,a}^s \geq \frac{p_i}{A} + \frac{1 - p_i}{A(\tau + 1)^2} = \frac{1}{A(i + 1)^2}$ , and thus  $\widetilde{x}_i^s \in \Omega_{i + 1}$ .
Applying Lemma 16 and considering the weighted sum of the bounds, we get
$$
\begin{array}{l} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(f _ {i} ^ {s} (\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}) - f _ {i} ^ {s} (\widetilde {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s})\right) \\ \leq \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(\frac {(1 - \eta_ {i} \epsilon_ {i}) \mathrm {K L} (\widetilde {x} _ {i} ^ {s} , \hat {x} _ {i} ^ {s}) - \mathrm {K L} (\widetilde {x} _ {i} ^ {s} , \hat {x} _ {i + 1} ^ {s})}{\eta_ {i}} + \frac {1 0 \eta_ {i} A \ln^ {2} (A \tau)}{(1 - \gamma) ^ {2}} + \frac {2 \eta_ {i} A}{(1 - \gamma) ^ {2}} \underline {{\lambda}} _ {i} ^ {s} + \underline {{\xi}} _ {i} ^ {s} + \underline {{\zeta}} _ {i} ^ {s} (\widetilde {x} _ {i} ^ {s})\right) \\ \leq \underbrace {\sum_ {i = 2} ^ {\tau} \left(\frac {\alpha_ {\tau} ^ {i} (1 - \eta_ {i} \epsilon_ {i})}{\eta_ {i}} \mathrm {K L} (\widetilde {x} _ {i} ^ {s} , \hat {x} _ {i} ^ {s}) - \frac {\alpha_ {\tau} ^ {i - 1}}{\eta_ {i - 1}} \mathrm {K L} (\widetilde {x} _ {i - 1} ^ {s} , \hat {x} _ {i} ^ {s})\right)} _ {\mathbf {t e r m} _ {0}} \\ + \underbrace {\frac {1 0 A \ln^ {2} (A \tau)}{(1 - \gamma) ^ {2}} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \eta_ {i}} _ {\mathbf {t e r m} _ {1}} + \underbrace {\frac {2 A}{(1 - \gamma) ^ {2}} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \eta_ {i} \underline {{\lambda}} _ {i} ^ {s}} _ {\mathbf {t e r m} _ {2}} + \underbrace {\sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \underline {{\xi}} _ {i} ^ {s}} _ {\mathbf {t e r m} _ {3}} + \underbrace {\sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \underline {{\zeta}} _ {i} ^ {s} (\widetilde {x} _ {i} ^ {s})} _ {\mathbf {t e r m} _ {4}}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {t e r m} _ {0} = \sum_ {i = 2} ^ {\tau} \mathrm {K L} (\widetilde {x} _ {i} ^ {s}, \hat {x} _ {i} ^ {s}) \left(\frac {\alpha_ {\tau} ^ {i} (1 - \eta_ {i} \epsilon_ {i})}{\eta_ {i}} - \frac {\alpha_ {\tau} ^ {i - 1}}{\eta_ {i - 1}}\right) + \sum_ {i = 2} ^ {\tau} \frac {\alpha_ {\tau} ^ {i - 1}}{\eta_ {i - 1}} \left(\mathrm {K L} (\widetilde {x} _ {i - 1} ^ {s}, \hat {x} _ {i} ^ {s}) - \mathrm {K L} (\widetilde {x} _ {i} ^ {s}, \hat {x} _ {i} ^ {s})\right) \\ \stackrel {(a)} {\leq} 0 + \mathcal {O} \left(\ln (A \tau) \sum_ {i = 2} ^ {\tau} \frac {\alpha_ {\tau} ^ {i - 1}}{\eta_ {i - 1}} \| \widetilde {x} _ {i - 1} ^ {s} - \widetilde {x} _ {i} ^ {s} \| _ {1}\right) \\ = \mathcal {O} \left(\ln (A \tau) \sum_ {i = 2} ^ {\tau} \frac {\alpha_ {\tau} ^ {i - 1}}{\eta_ {i - 1}} | p _ {i - 1} - p _ {i} |\right) \\ \leq \mathcal {O} \left(\ln (A \tau) \sum_ {i = 2} ^ {\tau} \frac {\alpha_ {\tau} ^ {i - 1}}{\eta_ {i - 1}} \frac {1}{(i - 1) ^ {2}}\right) \\ = \mathcal {O} \left(\frac {\ln (A \tau)}{1 - \gamma} \tau^ {k _ {\eta} - 2}\right) \tag {byLemma1} \\ \end{array}
$$
where  $(a)$  is by Lemma 14 and the following calculation:
$$
\frac {\alpha_ {\tau} ^ {i} (1 - \eta_ {i} \epsilon_ {i})}{\eta_ {i}} \times \frac {\eta_ {i - 1}}{\alpha_ {\tau} ^ {i - 1}} = \frac {\eta_ {i - 1}}{\eta_ {i}} \times \frac {\alpha_ {i}}{\alpha_ {i - 1}} \times \frac {1 - \eta_ {i} \epsilon_ {i}}{1 - \alpha_ {i}} = \left(\frac {i - 1}{i}\right) ^ {- k _ {\eta} + k _ {\alpha}} \times \frac {1 - i ^ {- k _ {\eta} - k _ {\epsilon}}}{1 - i ^ {- k _ {\alpha}}} \leq 1 \times 1 = 1.
$$
We proceed to bound other terms as follows: with probability at least  $1 - \mathcal{O}\left(\frac{\delta}{S\tau^2}\right)$
$$
\mathbf {t e r m} _ {1} = \mathcal {O} \left(\frac {A \ln^ {3} (A \tau)}{1 - \gamma} \tau^ {- k _ {\eta}}\right), \tag {Lemma 1}
$$
$$
\begin{array}{l} \mathbf {t e r m} _ {2} = \mathcal {O} \left(\frac {A \ln (S A \tau / \delta)}{1 - \gamma} \times \max  _ {i \leq \tau} \frac {\alpha_ {\tau} ^ {i} \eta_ {i}}{\beta_ {\tau}}\right) (Lemma7) \\ = \mathcal {O} \left(\frac {A \ln (S A \tau / \delta) \tau^ {- k _ {\alpha} - k _ {\eta} + k _ {\beta}}}{1 - \gamma}\right), (Lemma 2) \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {t e r m} _ {3} = \mathcal {O} \left(\frac {A}{1 - \gamma} \sum_ {i = 1} ^ {\tau} \beta_ {i} \alpha_ {\tau} ^ {i} + \frac {1}{1 - \gamma} \sqrt {\ln (S A \tau / \delta) \sum_ {i = 1} ^ {\tau} (\alpha_ {\tau} ^ {i}) ^ {2}}\right) (Lemma 6) \\ = \mathcal {O} \left(\frac {A \ln (A \tau) \tau^ {- k _ {\beta}}}{1 - \gamma} + \frac {1}{1 - \gamma} \sqrt {\ln (S A \tau / \delta) \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \alpha_ {i}}\right) (Lemma 1) \\ = \mathcal {O} \left(\frac {A \ln \left(S A \tau / \delta\right) \left(\tau^ {- k _ {\beta}} + \tau^ {- \frac {k _ {\alpha}}{2}}\right)}{1 - \gamma}\right), (Lemma 1) \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {t e r m} _ {4} = \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} p _ {i} \underline {{\zeta}} _ {i} ^ {s} \left(\frac {1}{A} \mathbf {1}\right) + \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(1 - p _ {i}\right) \underline {{\zeta}} _ {i} ^ {s} \left(x ^ {s}\right) \quad (\text {b y}) \\ = \mathcal {O} \left(\frac {\ln (S A \tau / \delta)}{1 - \gamma} \max  _ {i \leq \tau} \frac {\alpha_ {\tau} ^ {i}}{\beta_ {\tau}}\right) (Lemma7) \\ = \mathcal {O} \left(\frac {\ln (S A \tau / \delta) \tau^ {- k _ {\alpha} + k _ {\beta}}}{1 - \gamma}\right). (Lemma 2) \\ \end{array}
$$
Combining all terms, we get
$$
\sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(f _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) - f _ {i} ^ {s} \left(\tilde {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right)\right) = \mathcal {O} \left(\frac {A \ln^ {3} \left(S A \tau / \delta\right) \left(\tau^ {- k _ {\eta}} + \tau^ {- k _ {\beta}} + \tau^ {- k _ {\alpha} + k _ {\beta}}\right)}{1 - \gamma}\right). \tag {14}
$$
$$
\begin{array}{l} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(f _ {i} ^ {s} \left(\widetilde {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) - f _ {i} ^ {s} \left(x ^ {s}, \hat {y} _ {i} ^ {s}\right)\right) = \mathcal {O} \left(\frac {\ln (A \tau)}{1 - \gamma} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \| \widetilde {x} _ {i} ^ {s} - x ^ {s} \| _ {1}\right) \\ = \mathcal {O} \left(\frac {\ln (A \tau)}{1 - \gamma} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} p _ {i}\right) = \mathcal {O} \left(\frac {\ln (A \tau)}{\tau^ {2} (1 - \gamma)}\right). \tag {15} \\ \end{array}
$$
Adding up Eq. (14) and Eq. (15) and applying union bound over all  $s \in S$  and  $\tau$  finish the proof.
Lemma 20. With probability at least  $1 - \mathcal{O}(\delta)$ , for any state  $s\in S$  and time  $t\geq 1$ , we have
$$
| V _ {t} ^ {s} - V _ {\star} ^ {s} | \leq \mathcal {O} \left(\frac {A \ln (S A t / \delta)}{(1 - \gamma) ^ {2}} \left(\frac {L \ln (S t / \delta)}{1 - \gamma} \ln \frac {t}{1 - \gamma}\right) ^ {\frac {k _ {*}}{1 - k _ {\alpha}}} \left(\frac {L \ln (S t / \delta)}{t}\right) ^ {k _ {*}}\right),
$$
where  $k_{*} = \min \left\{k_{\eta}, k_{\beta}, k_{\alpha} - k_{\beta}, k_{\epsilon}\right\}$ .
Proof. Fix an  $s$  and a visitation count  $\tau$ . Let  $t_i$  be the time index when the players visit  $s$  for the  $i$ -th time. Then with probability at least  $1 - \frac{\delta}{S\tau^2}$ ,
$$
\begin{array}{l} V _ {t _ {\tau}} ^ {s} = \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(\sigma_ {t _ {i}} + \gamma V _ {t _ {i}} ^ {s _ {t _ {i} + 1}}\right) \\ \leq \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \hat {x} _ {i} ^ {s ^ {\top}} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t _ {i}} ^ {s ^ {\prime}} \right]\right) \hat {y} _ {i} ^ {s} + \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left[ \sigma_ {t _ {i}} + \gamma V _ {t _ {i}} ^ {s _ {t _ {i}} + 1} - \hat {x} _ {i} ^ {s ^ {\top}} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t _ {i}} ^ {s ^ {\prime}} \right]\right) \hat {y} _ {i} ^ {s} \right] \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \left(f _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) + \epsilon_ {i} \phi \left(\hat {x} _ {i} ^ {s}\right) - \epsilon_ {i} \phi \left(\hat {y} _ {i} ^ {s}\right)\right) + \mathcal {O} \left(\frac {1}{1 - \gamma} \sqrt {\ln (S \tau / \delta) \sum_ {i = 1} ^ {\tau} \left(\alpha_ {\tau} ^ {i}\right) ^ {2}}\right) \tag {Azuma's inequality} \\ \leq \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} f _ {i} ^ {s} (\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}) + \mathcal {O} \left(\epsilon_ {\tau} \ln (A) + \frac {\ln (S A \tau / \delta) \tau^ {- \frac {k _ {\alpha}}{2}}}{1 - \gamma}\right) \\ \leq \min  _ {x} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} f _ {i} ^ {s} (x ^ {s}, \hat {y} _ {i} ^ {s}) + \mathcal {O} \left(\epsilon_ {\tau} \ln (A) + \frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k ^ {\prime}}}{1 - \gamma}\right) \\ (k ^ {\prime} \text {i s d e f i n e d i n L e m m a 1 9 w i t h} k ^ {\prime} \leq \frac {1}{2} (k _ {\beta} + k _ {\alpha} - k _ {\beta}) = \frac {k _ {\alpha}}{2}) \\ \leq \min _ {x ^ {s}} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} (x ^ {s}) ^ {\top} (G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} [ V _ {t _ {i}} ^ {s ^ {\prime}} ]) \hat {y} _ {i} ^ {s} + \mathcal {O} \left(\frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma}\right) \\ \leq \min _ {x ^ {s}} \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} (x ^ {s}) ^ {\top} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {\star} ^ {s ^ {\prime}} \right]\right) \hat {y} _ {i} ^ {s} + \gamma \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \mathcal {O} \left(\frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma}\right) \\ \leq \min _ {x ^ {s}} \max _ {y ^ {s}} (x ^ {s}) ^ {\top} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {\star} ^ {s ^ {\prime}} \right]\right) y ^ {s} + \gamma \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \mathcal {O} \left(\frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma}\right) \\ \leq V _ {\star} ^ {s} + \gamma \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \mathcal {O} \left(\frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma}\right). \\ \end{array}
$$
Similar inequality can be also obtained through the perspective of the other player: with probability at least  $1 - \frac{\delta}{S\tau^2}$
$$
V _ {t _ {\tau}} ^ {s} \geq V _ {\star} ^ {s} - \gamma \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| - \mathcal {O} \left(\frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma}\right),
$$
which, combined with the previous inequality and union bound over  $s \in S$  and  $\tau \geq 1$ , gives the following relation: with probability at least  $1 - \mathcal{O}(\delta)$ , for any  $s \in S$  and  $\tau \geq 1$ ,
$$
\left| V _ {t _ {\tau}} ^ {s} - V _ {\star} ^ {s} \right| \leq \gamma \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max  _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \mathcal {O} \left(\frac {A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma}\right). \tag {16}
$$
Before continuing, we first some auxiliary quantities. For a fixed  $t$ , define
$$
u (t) = \left[ \left(\frac {1 6 \times 6 L \ln (S t / \delta)}{1 - \gamma} \ln \frac {t}{1 - \gamma}\right) ^ {\frac {1}{1 - k _ {\alpha}}} \right];
$$
for fixed  $(\tau ,t)$  we further define
$$
v (\tau , t) = \left\lfloor \tau - 3 \tau^ {k _ {\alpha}} \ln \frac {t}{1 - \gamma} \right\rfloor .
$$
Now we continue to prove a bound for  $|V_t^s - V_\star^s|$ . Suppose that Eq. (16) can be written as
$$
\left| V _ {t _ {\tau}} ^ {s} - V _ {\star} ^ {s} \right| \leq \gamma \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max  _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \frac {C _ {1} A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma} \tag {17}
$$
for a universal constant  $C_1 \geq 1$ . Below we use induction to show that for all  $t$ ,
$$
\left| V _ {t} ^ {s} - V _ {\star} ^ {s} \right| \leq \Phi (t) \triangleq \frac {8 C _ {1} A \ln^ {3} (S A t / \delta)}{(1 - \gamma) ^ {2}} \left(\frac {6 L \ln (S t / \delta) (u (t) + 1)}{t}\right) ^ {k _ {*}}. \tag {18}
$$
This is trivial for  $t = 1$ .
Suppose that Eq. (18) holds for all time  $1, \dots, t - 1$  and for all  $s$ . Now we consider time  $t$  and a fixed state  $s$ . We denote  $L' = 6L\ln (St / \delta)$ . Let  $\tau = n_{t + 1}^s$  and let  $1 \leq t_1 < t_2 < \dots < t_\tau \leq t$
be the time indices when the players visit state  $s$ . If  $t \leq L'(u(t) + 1)$ , then Eq. (18) is trivial. If  $t \geq L'(u(t) + 1)$ , we have  $\tau \geq \frac{t}{L'} - 1 \geq u(t)$ . Therefore,
$$
\begin{array}{l} \left| V _ {t} ^ {s} - V _ {\star} ^ {s} \right| \\ = \left| V _ {t _ {\tau}} ^ {s} - V _ {\star} ^ {s} \right| \quad \left(t _ {\tau} \text {i s t h e l a s t t i m e u p t o t i m e} t \text {w h e n} V _ {t} ^ {s} \text {i s u p d a t e d}\right) \\ \leq \gamma \sum_ {i = 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max  _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \frac {C _ {1} A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma} \tag {byEq.(17)} \\ \leq \gamma \sum_ {i = 1} ^ {v (\tau , t)} \alpha_ {\tau} ^ {i} \max _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \gamma \sum_ {i = v (\tau , t) + 1} ^ {\tau} \alpha_ {\tau} ^ {i} \max _ {s ^ {\prime}} \left| V _ {t _ {i}} ^ {s ^ {\prime}} - V _ {\star} ^ {s ^ {\prime}} \right| + \frac {C _ {1} A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma} \\ \stackrel {(a)} {\leq} \gamma \tau \times \frac {(1 - \gamma) ^ {3}}{t ^ {3}} \times \frac {1}{1 - \gamma} + \gamma \sum_ {i = v (\tau , t) + 1} ^ {\tau} \alpha_ {\tau} ^ {i} \Phi (t _ {i}) + \frac {C _ {1} A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma} \\ \leq \gamma \sum_ {i = v (\tau , t) + 1} ^ {\tau} \alpha_ {\tau} ^ {i} \Phi (t _ {i}) + \frac {2 C _ {1} A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma} \quad \text {(i n d u c t i o n h y p o t h e s i s)} \\ \leq \gamma \sum_ {i = v (\tau , t) + 1} ^ {\tau} \alpha_ {\tau} ^ {i} \Phi (t) \times \left(\frac {t}{t _ {i}}\right) ^ {k _ {*}} + \frac {2 C _ {1} A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma} \\ \end{array}
$$
(by the definition of  $\Phi$  and that  $u(\cdot)$  is an increasing function)
$$
\begin{array}{l} \stackrel {(b)} {\leq} \gamma \left(1 + \frac {1 - \gamma}{2}\right) \sum_ {i = v (\tau , t) + 1} ^ {\tau} \alpha_ {\tau} ^ {i} \Phi (t) + \frac {2 C _ {1} A \ln^ {3} (S A \tau / \delta) \tau^ {- k _ {*}}}{1 - \gamma} \\ \leq \gamma \left(1 + \frac {1 - \gamma}{2}\right) \Phi (t) + \frac {2 C _ {1} A \ln^ {3} (S A t / \delta) \left(\frac {t}{2 L ^ {\prime}}\right) ^ {- k _ {*}}}{1 - \gamma} \\ (t \geq \tau \geq \frac {t}{L ^ {\prime}} - 1 \geq \frac {t}{2 L ^ {\prime}} \text {s i n c e} t \geq L ^ {\prime} (u (t) + 1) \geq 2 L ^ {\prime}) \\ \leq \gamma \left(1 + \frac {1 - \gamma}{2}\right) \Phi (t) + \frac {1 - \gamma}{2} \Phi (t) \\ = \Phi (t). \\ \end{array}
$$
In  $(a)$  we use the following property: if  $\tau \geq u(t)$  and  $i\leq v(\tau ,t)$ , then
$$
\begin{array}{l} \alpha_ {\tau} ^ {i} = i ^ {- k _ {\alpha}} \prod_ {j = i + 1} ^ {\tau} \left(1 - j ^ {- k _ {\alpha}}\right) \leq \left(1 - \tau^ {- k _ {\alpha}}\right) ^ {\tau - i} \\ \leq \left(1 - \tau^ {- k _ {\alpha}}\right) ^ {3 \tau^ {k _ {\alpha}} \ln \frac {t}{1 - \gamma}} \leq \exp \left(- \tau^ {- k _ {\alpha}} \cdot 3 \tau^ {k _ {\alpha}} \ln \frac {t}{1 - \gamma}\right) = \frac {(1 - \gamma) ^ {3}}{t ^ {3}} \\ \end{array}
$$
In  $(b)$  we use the following calculation:
$$
\begin{array}{l} \frac {t}{t _ {i}} \leq \frac {t _ {\tau + 1}}{t _ {i}} = 1 + \frac {t _ {\tau + 1} - t _ {i}}{t _ {i}} \leq 1 + \frac {L ^ {\prime} (\tau + 1 - i)}{i} \\ \leq 1 + L ^ {\prime} \left(\frac {\tau + 1}{v (\tau , t)} - 1\right) \\ \leq 1 + L ^ {\prime} \left(\frac {\tau + 1}{\tau - 4 \tau^ {k _ {\alpha}} \ln \frac {t}{1 - \gamma}} - 1\right) \\ = 1 + L ^ {\prime} \left(\frac {1 + \frac {1}{\tau}}{1 - 4 \tau^ {k _ {\alpha} - 1} \ln \frac {t}{1 - \gamma}} - 1\right) \\ \leq 1 + L ^ {\prime} \left(\frac {1 + \frac {1 - \gamma}{1 6 L ^ {\prime}}}{1 - \frac {1 - \gamma}{4 L ^ {\prime}}} - 1\right) \\ \leq 1 + L ^ {\prime} \left(\frac {1 - \gamma}{2 L ^ {\prime}}\right) \\ \end{array}
$$
$$
= 1 + \frac {1 - \gamma}{2}
$$
where the first inequality is due to the fact that at time  $t$ , state  $s$  has only been visited for  $\tau$  times; the second inequality is because for any  $k > j$ , we have  $t_j \geq j$  and  $t_k - t_j \leq L'(k - j)$ ; the third inequality is by  $i \geq v(\tau, t)$ ; the fourth inequality is by the definition of  $v(\tau, t)$ ; the fifth inequality is because  $4\tau^{k_\alpha - 1} \ln \frac{t}{1 - \gamma} \leq \frac{1 - \gamma}{4L'}$  since  $\tau \geq u(t)$ , and  $\frac{1}{\tau} < \frac{1}{u(t)} \leq \frac{1 - \gamma}{16L'}$  since  $u(t) \geq \frac{16L'}{1 - \gamma}$ ; the last inequality is because  $\frac{1 + \frac{1}{16}a}{1 - \frac{1}{4}a} \leq 1 + \frac{1}{2}a$  for  $a \in [0,1]$ .
D.5 Part IV. Combining
In this subsection, we combine previous lemmas to show last-iterate convergence rate of Algorithm 2 and prove Theorem 2.
Lemma 21. With probability at least  $1 - \mathcal{O}(\delta)$ , for any time  $t \geq 1$ ,
$$
\max  _ {s, x, y} \left(V _ {x _ {t}, y} ^ {s} - V _ {x, y _ {t}} ^ {s}\right) \leq \mathcal {O} \left(\frac {A L ^ {2 + 1 / \varepsilon} \ln^ {4 + 1 / \varepsilon} (S A t / \delta) \ln^ {1 / \varepsilon} (t / (1 - \gamma))}{(1 - \gamma) ^ {2 + 1 / \varepsilon}} t ^ {- \frac {1}{9 + \varepsilon}}\right).
$$
Proof. Using Lemma 10, we can bound the duality gap of the whole game by the duality gap on an individual state:
$$
\begin{array}{l} \max  _ {s, x, y} \left(V _ {x _ {t}, y} ^ {s} - V _ {x, y _ {t}} ^ {s}\right) \leq \frac {2}{1 - \gamma} \max  _ {s, x, y} \left(x _ {t} ^ {s} Q _ {\star} ^ {s} y ^ {s} - x ^ {s} Q _ {\star} ^ {s} y _ {t} ^ {s}\right) \\ = \frac {2}{1 - \gamma} \max  _ {s, x, y} \left(x _ {t} ^ {s} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {\star} ^ {s ^ {\prime}} \right]\right) y ^ {s} - x ^ {s} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {\star} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s}\right) \\ \leq \frac {2}{1 - \gamma} \max  _ {s, x, y} \left(x _ {t} ^ {s} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t} ^ {s ^ {\prime}} \right]\right) y ^ {s} - x ^ {s} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s}} \left[ V _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s}\right) \\ + \frac {4 \gamma}{1 - \gamma} \max _ {s} | V _ {t} ^ {s} - V _ {\star} ^ {s} | \\ \end{array}
$$
With probability at least  $1 - \mathcal{O}(\delta)$ , for any  $s, x^s, y^s$ , and  $t \geq 1$ , denote  $\tau$  the number of visitation to state  $s$  until time  $t$ , then