SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ K L \left(\hat {z} _ {i \star} ^ {s}, \hat {z} _ {i} ^ {s}\right) \geq \epsilon^ {\prime} \right] \leq \mathcal {O} \left(\frac {S ^ {2} A \ln^ {5} (S A T / \delta)}{\eta \epsilon^ {2} \epsilon^ {\prime} (1 - \gamma) ^ {3}}\right)
$$
if  $\eta$  and  $\beta$  satisfy the following
$$
\beta \leq \frac {C _ {5} (1 - \gamma) ^ {3}}{A \ln^ {3} (A S T / \delta)} \epsilon \epsilon^ {\prime} \tag {23}
$$
$$
\eta \leq \frac {C _ {6} (1 - \gamma)}{A \ln^ {3} (A S T / \delta)} \beta \epsilon^ {\prime} \tag {24}
$$
with sufficiently small universal constant  $C_5, C_6 > 0$ .
Proof. In this proof, we write  $\underline{\zeta}_i^s (\hat{x}_{i\star}^s)$  as  $\underline{\zeta}_i$ . By Lemma 22, we have
$$
\begin{array}{l} \mathrm {K L} \left(\hat {x} _ {i \star} ^ {s}, \hat {x} _ {i + 1} ^ {s}\right) \leq (1 - \eta \epsilon) \mathrm {K L} \left(\hat {x} _ {i \star} ^ {s}, \hat {x} _ {i} ^ {s}\right) + \eta \left(\underline {{f}} _ {i} ^ {s} \left(\hat {x} _ {i \star} ^ {s}, \hat {y} _ {i} ^ {s}\right) - \underline {{f}} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right)\right) \\ + \frac {1 0 \eta^ {2} A \ln^ {2} (A T)}{(1 - \gamma) ^ {2}} + \frac {2 \eta^ {2} A}{(1 - \gamma) ^ {2}} \underline {{\lambda}} _ {i} ^ {s} + \eta \underline {{\xi}} _ {i} ^ {s} + \eta \underline {{\zeta}} _ {i} ^ {s}. \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {K L} \left(\hat {y} _ {i \star} ^ {s}, \hat {y} _ {i + 1} ^ {s}\right) \leq (1 - \eta \epsilon) \mathrm {K L} \left(\hat {y} _ {i \star} ^ {s}, \hat {y} _ {i} ^ {s}\right) + \eta \left(\bar {f} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) - \bar {f} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i \star} ^ {s}\right)\right) \\ + \frac {1 0 \eta^ {2} A \ln^ {2} (A T)}{(1 - \gamma) ^ {2}} + \frac {2 \eta^ {2} A}{(1 - \gamma) ^ {2}} \overline {{\lambda}} _ {i} ^ {s} + \eta \overline {{\xi}} _ {i} ^ {s} + \eta \overline {{\zeta}} _ {i} ^ {s}. \\ \end{array}
$$
Adding the two inequalities up, we get
$$
\begin{array}{l} \mathrm {K L} \left(\hat {z} _ {i + 1 \star} ^ {s}, \hat {z} _ {i + 1} ^ {s}\right) \\ \leq (1 - \eta \epsilon) \mathrm {K L} (\hat {z} _ {i \star} ^ {s}, \hat {z} _ {i} ^ {s}) + \frac {2 0 \eta^ {2} A \ln^ {2} (A T)}{(1 - \gamma) ^ {2}} + \frac {2 \eta^ {2} A}{(1 - \gamma) ^ {2}} \lambda_ {i} ^ {s} + \eta \xi_ {i} ^ {s} + \eta \zeta_ {i} ^ {s} + v _ {i} ^ {s} \\ \end{array}
$$
$$
+ \eta \left(\bar {f} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) - \underline {{f}} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) + \underline {{f}} _ {i} ^ {s} \left(\hat {x} _ {i \star} ^ {s}, \hat {y} _ {i} ^ {s}\right) - \bar {f} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i \star} ^ {s}\right)\right) \tag {25}
$$
where  $v_{i}^{s} = \mathrm{KL}(\hat{z}_{i + 1\star}^{s},\hat{z}_{i + 1}^{s}) - \mathrm{KL}(\hat{z}_{i\star}^{s},\hat{z}_{i + 1}^{s})$  and  $\square^s = \square^s +\overline{\square}^s$ . By Lemma 24, we have  $\underline{f}_i^s (x,y)\leq \overline{f}_i^s (x,y)$  for all  $x,y$ , and thus  $\underline{f}_i^s (\hat{x}_i^s,\hat{y}_i^s) - \overline{f}_i^s (\hat{x}_i^s,\hat{y}_i^s)\leq f_i^s (\hat{x}_i^s,\hat{y}_i^s) - f_i^s (\hat{x}_i^s,\hat{y}_i^s)\leq 0.$  Therefore, Eq. (25) further implies
$$
\mathrm {K L} \left(\hat {z} _ {i + 1 *} ^ {s}, \hat {z} _ {i + 1} ^ {s}\right)
$$
$$
\begin{array}{l} \leq (1 - \eta \epsilon) \mathrm {K L} (\hat {z} _ {i \star} ^ {s}, \hat {z} _ {i} ^ {s}) + \frac {2 0 \eta^ {2} A \ln^ {2} (A T)}{(1 - \gamma) ^ {2}} + \frac {2 \eta^ {2} A}{(1 - \gamma) ^ {2}} \lambda_ {i} ^ {s} + \eta \xi_ {i} ^ {s} + \eta \zeta_ {i} ^ {s} + v _ {i} ^ {s} + \eta \Delta_ {i} ^ {s} \\ \leq (1 - \eta \epsilon) \mathrm {K L} (\hat {z} _ {i \star} ^ {s}, \hat {z} _ {i} ^ {s}) + \frac {2 0 \eta^ {2} A \ln^ {2} (A T)}{(1 - \gamma) ^ {2}} + \frac {2 \eta^ {2} A}{(1 - \gamma) ^ {2}} \lambda_ {i} ^ {s} + \eta \xi_ {i} ^ {s} + \eta \zeta_ {i} ^ {s} + v _ {i} ^ {s} + \frac {1}{2} \eta \epsilon \epsilon^ {\prime} + \left[ \eta \Delta_ {i} ^ {s} - \frac {1}{2} \eta \epsilon \epsilon^ {\prime} \right] _ {+} \\ \end{array}
$$
where  $\Delta_i^s = \overline{f}_i^s (\hat{x}_i^s,\hat{y}_i^s) - \underline{f}_i^s (\hat{x}_i^s,\hat{y}_i^s)$  and in the last step we use  $a\leq [a - b]_{+} + b$
Unrolling the recursion, we get with probability at least  $1 - \mathcal{O}(\delta)$ , for all  $s$  and  $\tau$  (we show that the inequality holds for any fix  $s$  and  $\tau$  with probability  $1 - \mathcal{O}\left(\frac{\delta}{ST}\right)$  and then apply the union bound over  $s$  and  $\tau$ ),
$$
\begin{array}{l} \mathrm {K L} (\hat {z} _ {\tau + 1 \star} ^ {s}, \hat {z} _ {\tau + 1} ^ {s}) \\ \leq (1 - \eta \epsilon) ^ {\tau} \mathrm {K L} (\hat {z} _ {1 \star} ^ {s}, \hat {z} _ {1} ^ {s}) + \underbrace {\frac {2 0 \eta^ {2} A \ln^ {2} (A T)}{(1 - \gamma) ^ {2}} \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i}} _ {\text {t e r m} _ {1}} + \underbrace {\frac {2 \eta^ {2} A}{(1 - \gamma) ^ {2}} \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} \lambda_ {i} ^ {s}} _ {\text {t e r m} _ {2}} \\ + \underbrace {\eta \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} \xi_ {i} ^ {s}} _ {\text {t e r m} _ {3}} + \underbrace {\eta \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} \zeta_ {i} ^ {s}} _ {\text {t e r m} _ {4}} \\ + \underbrace {\sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} v _ {i} ^ {s}} _ {\triangleq \mathbf {t e r m} _ {5} (s, \tau)} + \frac {1}{2} \eta \epsilon \epsilon^ {\prime} \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} + \underbrace {\eta \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} \left[ \Delta_ {i} ^ {s} - \frac {1}{2} \epsilon \epsilon^ {\prime} \right] _ {+}} _ {\triangleq \mathbf {t e r m} _ {6} (s, \tau)} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \mathcal {O} (e ^ {- \eta \epsilon \tau} \ln (A T)) + \ln^ {3} (A S T / \delta) \times \mathcal {O} \left(\frac {\eta A}{\epsilon (1 - \gamma) ^ {2}} + \frac {\eta^ {2} A}{\beta (1 - \gamma) ^ {2}} + \frac {\beta A}{\epsilon (1 - \gamma)} + \frac {1}{1 - \gamma} \sqrt {\frac {\eta}{\epsilon}} + \frac {\eta}{\beta (1 - \gamma)}\right) \\ + \mathbf {t e r m} _ {5} (s, \tau) + \frac {1}{2} \epsilon^ {\prime} + \mathbf {t e r m} _ {6} (s, \tau) \\ \end{array}
$$
$$
\stackrel {(b)} {\leq} \mathcal {O} \left(e ^ {- \eta \epsilon \tau} \ln (A T)\right) + \frac {3}{4} \epsilon^ {\prime} + \operatorname {t e r m} _ {5} (s, \tau) + \operatorname {t e r m} _ {6} (s, \tau) \tag {26}
$$
where in  $(a)$  we use the following calculation:
$$
\mathbf {t e r m} _ {1} \leq \mathcal {O} \left(\frac {\eta^ {2} A \ln^ {2} (A T)}{(1 - \gamma) ^ {2}} \times \frac {1}{\eta \epsilon}\right) \leq \mathcal {O} \left(\frac {\eta A \ln^ {2} (A T)}{\epsilon (1 - \gamma) ^ {2}}\right).
$$
$$
\mathbf {t e r m} _ {2} \leq \mathcal {O} \left(\frac {\eta^ {2} A}{(1 - \gamma) ^ {2}} \frac {\operatorname* {m a x} _ {i \leq \tau} (1 - \eta \epsilon) ^ {\tau - i} \ln (A S T / \delta)}{\beta}\right) = \mathcal {O} \left(\frac {\eta^ {2} A}{(1 - \gamma) ^ {2}} \frac {\ln (A S T / \delta)}{\beta}\right)
$$
(by Lemma 8)
$$
\begin{array}{l} \mathbf {t e r m} _ {3} \leq \mathcal {O} \left(\frac {\eta A}{1 - \gamma} \sum_ {i = 1} ^ {\tau} \beta (1 - \eta \epsilon) ^ {\tau - i} + \eta \sqrt {\ln (A S T / \delta) \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i}}\right) \quad (\text {b y}) \\ = \mathcal {O} \left(\frac {\beta A}{\epsilon (1 - \gamma)} + \sqrt {\ln (A S / \delta) \frac {\eta}{\epsilon}}\right). \\ \end{array}
$$
$$
\mathbf {t e r m} _ {4} \leq \mathcal {O} \left(\frac {\eta}{1 - \gamma} \times \frac {\operatorname* {m a x} _ {i \leq \tau} (1 - \eta \epsilon) ^ {\tau - i} \ln (A S T / \delta)}{\beta}\right) = \mathcal {O} \left(\frac {\eta \ln (A S T / \delta)}{\beta (1 - \gamma)}\right),
$$
and in  $(b)$  we use the conditions Eq. (23) and Eq. (24).
We continue to bound the sum of  $\mathbf{term}_5$  and  $\mathbf{term}_6$  over  $t$ . Note that
$$
\sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {t e r m} _ {5} (s, \tau) \leq \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} v _ {i} ^ {s} \leq \frac {1}{\eta \epsilon} \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} v _ {i} ^ {s} \leq \mathcal {O} \left(\frac {S ^ {2} \ln^ {3} (A T)}{\eta \epsilon^ {2} (1 - \gamma) ^ {2}}\right), \tag {27}
$$
where in the last inequality we use the following calculation:
$$
\sum_ {i = 1} ^ {n _ {T + 1} (s)} \left| v _ {i} ^ {s} \right| \leq \mathcal {O} (\ln (A \tau)) \times \sum_ {i = 1} ^ {n _ {T + 1} (s)} \| \hat {z} _ {i \star} ^ {s} - \hat {z} _ {i + 1 \star} ^ {s} \| _ {1} \tag {byLemma14}
$$
$$
= \mathcal {O} \left(\ln (A T)\right) \times \frac {\ln (A T)}{\epsilon} \times \sum_ {i = 1} ^ {n _ {T + 1} (s)} \sup  _ {s ^ {\prime}} \left(p \left| \underline {{V}} _ {t _ {i}} ^ {s ^ {\prime}} - \underline {{V}} _ {t _ {i + 1}} ^ {s ^ {\prime}} \right| + (1 - p) \left| \overline {{V}} _ {t _ {i}} ^ {s ^ {\prime}} - \overline {{V}} _ {t _ {i + 1}} ^ {s ^ {\prime}} \right|\right)
$$
(by the same calculation as Eq. (12))
$$
\leq \mathcal {O} \left(\frac {\ln^ {2} (A T)}{\epsilon}\right) \times \sum_ {s ^ {\prime}} \sum_ {t = 1} ^ {T} \left(\left| \underline {{V}} _ {t} ^ {s ^ {\prime}} - \underline {{V}} _ {t + 1} ^ {s ^ {\prime}} \right| + \left| \overline {{V}} _ {t} ^ {s ^ {\prime}} - \overline {{V}} _ {t + 1} ^ {s ^ {\prime}} \right|\right)
$$
$$
\leq \mathcal {O} \left(\frac {\ln^ {2} (A T)}{\epsilon} \times \frac {S \ln T}{(1 - \gamma) ^ {2}}\right)
$$
$(|\underline{V}_t^s - \underline{V}_{t + 1}^s| \leq \frac{H + 1}{H + \tau} \times \frac{1}{1 - \gamma} \mathbf{1}[s_t = s]$  by the update rule)
$$
= \mathcal {O} \left(\frac {S \ln^ {3} (A T)}{\epsilon (1 - \gamma) ^ {2}}\right),
$$
$$
\begin{array}{l} \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {t e r m} _ {6} (s, \tau) \\ = \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \eta \sum_ {i = 1} ^ {\tau} (1 - \eta \epsilon) ^ {\tau - i} \left[ \Delta_ {i} ^ {s} - \frac {1}{2} \epsilon \epsilon^ {\prime} \right] _ {+} \\ \leq \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \sum_ {\tau = i} ^ {n _ {T + 1} (s)} \eta (1 - \eta \epsilon) ^ {\tau - i} \left[ \Delta_ {i} ^ {s} - \frac {1}{2} \epsilon \epsilon^ {\prime} \right] _ {+} \\ \leq \frac {1}{\epsilon} \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \left[ \Delta_ {i} ^ {s} - \frac {1}{2} \epsilon \epsilon^ {\prime} \right] _ {+} \\ = \frac {1}{\epsilon} \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \sum_ {j = - 1} ^ {j _ {\max }} \mathbf {1} \left[ \epsilon \epsilon^ {\prime} 2 ^ {j} \leq \Delta_ {i} ^ {s} \leq \epsilon \epsilon^ {\prime} 2 ^ {j + 1} \right] \epsilon \epsilon^ {\prime} 2 ^ {j + 1} \quad (\text {d e f i n e} j _ {\max } = \log_ {2} \left(\frac {1}{(1 - \gamma) \epsilon \epsilon^ {\prime}}\right)) \\ \leq \frac {1}{\epsilon} \sum_ {j = - 1} ^ {j _ {\max }} \sum_ {t = 1} ^ {T} \mathbf {1} \left[ \Delta_ {i} ^ {s _ {t}} \geq \epsilon \epsilon^ {\prime} 2 ^ {j} \right] \epsilon \epsilon^ {\prime} 2 ^ {j + 1} \\ \leq \frac {1}{\epsilon} \sum_ {j = - 1} ^ {j _ {\max }} \mathcal {O} \left(\frac {A S \ln^ {4} (A S T / \delta)}{\eta \epsilon^ {\prime} 2 ^ {j} (1 - \gamma) ^ {3}}\right) \times \epsilon^ {\prime} 2 ^ {j + 1} \\ \end{array}
$$
(by Corollary 2 with  $\tilde{\epsilon} = \epsilon \epsilon' 2^j$  and the assumption that  $\epsilon \epsilon' \gtrsim \frac{A \ln^3(AST / \delta) \beta}{(1 - \gamma)^3}$ )
$$
= \mathcal {O} \left(\frac {A S \ln^ {5} (A S T / \delta)}{\eta \epsilon (1 - \gamma) ^ {3}}\right) \quad \text {(w i t h o u t l o s s o f g e n e r a l i t y , a s s u m e} \log_ {2} \left(\frac {1}{(1 - \gamma) \epsilon \epsilon^ {\prime}}\right) \lesssim \log T) \tag {28}
$$
From Eq. (26), we have
$$
\sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \mathrm {K L} \left(\hat {z} _ {\tau \star} ^ {s}, \hat {z} _ {\tau} ^ {s}\right) \geq \epsilon^ {\prime} \right]
$$
$$
\begin{array}{l} \leq \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \mathcal {O} \left(e ^ {- \eta \epsilon \tau} \ln (A T)\right) \geq \frac {1}{1 2} \epsilon^ {\prime} \right] + \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \mathbf {t e r m} _ {5} (s, \tau) > \frac {1}{1 2} \epsilon^ {\prime} \right] \\ + \sum_ {s} \sum_ {\tau = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \mathbf {t e r m} _ {6} (s, \tau) > \frac {1}{1 2} \epsilon^ {\prime} \right] \\ \leq S \times \mathcal {O} \left(\frac {\ln (A T)}{\eta \epsilon \epsilon^ {\prime}}\right) + \mathcal {O} \left(\frac {S ^ {2} \ln^ {3} (A T)}{\eta \epsilon^ {2} \epsilon^ {\prime} (1 - \gamma) ^ {2}}\right) + \mathcal {O} \left(\frac {A S \ln^ {5} (A S T / \delta)}{\eta \epsilon \epsilon^ {\prime} (1 - \gamma) ^ {3}}\right) \\ \leq \mathcal {O} \left(\frac {S ^ {2} A \ln^ {5} (S A T / \delta)}{\eta \epsilon^ {2} \epsilon^ {\prime} (1 - \gamma) ^ {3}}\right) \\ \end{array}
$$
where in the second-to-last inequality we use Eq. (27) and Eq. (28). This finishes the proof.
E.4 Part IV. Combining
Theorem 5. For any  $u \in \left[0, \frac{1}{1 - \gamma}\right]$ , there exists a proper choice of parameters  $\epsilon, \beta, \eta$  such that
$$
\sum_ {t = 1} ^ {T} \mathbf {1} \left[ \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) > u \right] \leq \mathcal {O} \left(\frac {S ^ {2} A ^ {3} \ln^ {1 7} (S A T / \delta)}{u ^ {9} (1 - \gamma) ^ {1 3}}\right).
$$
with probability at least  $1 - \mathcal{O}(\delta)$
Proof. We will choose  $\epsilon$  such that  $u \geq C_7 \frac{\epsilon \ln(AT)}{1 - \gamma}$  with a sufficiently large universal constant  $C_7$ . By Lemma 26, we have
$$
\begin{array}{l} \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) \\ \leq \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \bar {V} _ {t} ^ {s ^ {\prime}} \right]\right) y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \underline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}}\right) + \mathcal {O} \left(\frac {\epsilon \ln (A T)}{1 - \gamma}\right) \\ \leq \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \bar {V} _ {t} ^ {s ^ {\prime}} \right]\right) y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \underline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}}\right) + \frac {u}{4}. \\ \end{array}
$$
Therefore, we can upper bound the left-hand side of the desired inequality by
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbf {1} \left[ \max  _ {x, y} \left(x _ {t} ^ {s _ {t}} ^ {\top} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \bar {V} _ {t} ^ {s ^ {\prime}} \right]\right) y ^ {s _ {t}} - x ^ {s _ {t}} ^ {\top} \left(G ^ {s} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \underline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}}\right) \geq \frac {3}{4} u \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left[ \max  _ {y} x _ {t} ^ {s _ {t} ^ {\top}} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \overline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \overline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}} \geq \frac {u}{4} \right] \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left[ x _ {t} ^ {s _ {t} ^ {\top}} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \bar {V} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \underline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}} \geq \frac {u}{4} \right] \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left[ x _ {t} ^ {s _ {t}} ^ {\top} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \underline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y ^ {s _ {t}} - \min  _ {x} x ^ {s _ {t} ^ {\top}} \left(G ^ {s _ {t}} + \gamma \mathbb {E} _ {s ^ {\prime} \sim P ^ {s _ {t}}} \left[ \underline {{V}} _ {t} ^ {s ^ {\prime}} \right]\right) y _ {t} ^ {s _ {t}} \geq \frac {u}{4} \right]. \tag {29} \\ \end{array}
$$
For the first term in Eq. (29), we can bound it by
$$
\begin{array}{l} \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \max  _ {y} \overline {{f}} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, y ^ {s}\right) - \overline {{f}} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) \geq \frac {u}{4} - \mathcal {O} (\epsilon \ln (A T)) \right] \\ \leq \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \max  _ {y} \bar {f} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, y ^ {s}\right) - \bar {f} _ {i} ^ {s} \left(\hat {x} _ {i} ^ {s}, \hat {y} _ {i} ^ {s}\right) \geq \frac {u}{8} \right] \\ \end{array}
$$
$$
\leq \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} {\bf 1} \left[ \max _ {y} \overline {{{f}}} _ {i} ^ {s} (\hat {x} _ {i \star} ^ {s}, y ^ {s}) - \overline {{{f}}} _ {i} ^ {s} (\hat {x} _ {i \star} ^ {s}, \hat {y} _ {i \star} ^ {s}) + \mathcal {O} \left(\| \hat {z} _ {i} ^ {s} - \hat {z} _ {i \star} ^ {s} \| _ {1} \frac {\ln (A T)}{1 - \gamma}\right) \geq \frac {u}{8} \right]
$$
(because  $\| \nabla \overline{f}_i^s (x,y)\|_\infty \leq \mathcal{O}\left(\frac{\ln(AT)}{1 - \gamma}\right)$  — similar to the calculation in Eq. (9))
(here we choose  $(\hat{x}_{i\star}^{s},\hat{y}_{i\star}^{s})$  to be the equilibrium under  $\overline{f}_i^s (x,y)$ )
$$
\begin{array}{l} \leq \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \mathcal {O} \left(\| \hat {z} _ {i} ^ {s} - \hat {z} _ {i \star} ^ {s} \| _ {1} \frac {\ln (A T)}{1 - \gamma}\right) \geq \frac {u}{8} \right] \\ \leq \sum_ {s} \sum_ {i = 1} ^ {n _ {T + 1} (s)} \mathbf {1} \left[ \mathrm {K L} (\hat {z} _ {i \star} ^ {s}, \hat {z} _ {i} ^ {s}) \geq \Omega \left(\frac {u ^ {2} (1 - \gamma) ^ {2}}{\ln^ {2} (A T)}\right) \right] \\ \leq \mathcal {O} \left(\frac {S ^ {2} A \ln^ {7} (S A T / \delta)}{\eta \epsilon^ {2} u ^ {2} (1 - \gamma) ^ {5}}\right) \quad (\text {b y}) \\ \end{array}
$$
The third term in Eq. (29) can be bounded in the same way. The second term in Eq. (29) can be bounded using Corollary 2 by
$$
\mathcal {O} \left(\frac {S A \ln^ {4} (S A T / \delta)}{\eta u (1 - \gamma) ^ {3}}\right).
$$
$$
\sum_ {t = 1} ^ {T} \mathbf {1} \left[ \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) > u \right] \leq \mathcal {O} \left(\frac {S ^ {2} A \ln^ {7} (S A T / \delta)}{\eta \epsilon^ {2} u ^ {2} (1 - \gamma) ^ {5}}\right). \tag {30}
$$
Notice that the parameters  $\epsilon, \beta, \eta$  needs to satisfy the conditions specified in this lemma and Lemma 27, with which we apply  $\epsilon' = \Theta\left(\frac{u^2(1 - \gamma)^2}{\ln^2(SAT / \delta)}\right)$ . The constraints suggest the following parameter choice (under a fixed  $u$ ):
$$
\epsilon = \Theta \left(\frac {u (1 - \gamma)}{\ln (S A T / \delta)}\right)
$$
$$
\beta = \Theta \left(\frac {(1 - \gamma) ^ {3}}{A \ln^ {3} (S A T / \delta)} \epsilon \epsilon^ {\prime}\right) = \Theta \left(\frac {u ^ {3} (1 - \gamma) ^ {6}}{A \ln^ {6} (S A T / \delta)}\right)
$$
$$
\eta = \Theta \left(\frac {(1 - \gamma)}{A \ln^ {3} (S A T / \delta)} \beta \epsilon^ {\prime}\right) = \Theta \left(\frac {u ^ {5} (1 - \gamma) ^ {9}}{A ^ {2} \ln^ {1 1} (S A T / \delta)}\right)
$$
Using these parameters in Eq. (30), we get
$$
\sum_ {t = 1} ^ {T} \mathbf {1} \left[ \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) > u \right] \leq \mathcal {O} \left(\frac {S ^ {2} A ^ {3} \ln^ {2 0} (S A T / \delta)}{u ^ {9} (1 - \gamma) ^ {1 6}}\right).
$$
F Discussions on Convergence Notions for General Markov Games
In general Markov games, learning the equilibrium policy pair on every state is impossible because some state might have exponentially small visitation probability under all policies. Therefore, a reasonable definition of convergence is the convergence of the following quantity to zero:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \max  _ {x, y} \left(V _ {x _ {t}, y} ^ {s _ {t}} - V _ {x, y _ {t}} ^ {s _ {t}}\right), \tag {31}
$$
which is similar to the best-iterate convergence defined in Section 3, but over the state sequence visited by the players instead of taking max over  $s$ . It is also a strict generalization of the sample complexity bound for single-player MDPs under the discounted criteria (see e.g., [LH14, WDCW20]).
The path convergence defined in our work is, on the other hand, that the following quantity converges to zero:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \max  _ {x, y} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right). \tag {32}
$$
Since  $\max_y(x^s^\top Q_\star^s y^s) \leq \max_y(x^{s^\top}Q_{x,y}^s y^s) = \max_y V_{x,y}^s$  for any  $x$ , the convergence of Eq. (31) is stronger than Eq. (32).
Implications of Path Convergence Although Eq. (32) does not imply the more standard best-iterate guarantee Eq. (31), it still has meaningful implications. By definition, It implies that frequent visits to a state bring players' policies closer to equilibrium, leading to both players using near-equilibrium policies for all but  $o(T)$  number of steps over time.
Path convergence also implies that both players have no regret compared to the game value  $V_{\star}^{s}$ , which has been considered and motivated in previous works such as [BT02, TWYS20]. To see this more clearly, we apply the results to the episodic setting, where in every step, with probability  $1 - \gamma$ , the state is redrawn from  $s \sim \rho$  for some initial distribution  $\rho$  (every time the state is redrawn from  $\rho$ , we call it a new episode). We can show that if Eq. (32) vanishes, then every player's long-term average payoff is at least the game value. First, notice that if Eq. (32) converges to zero, then
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(V _ {\star} ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) \leq \max  _ {y} \sum_ {t = 1} ^ {T} \left(x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\max  _ {y} x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) = o (T). \tag {33} \\ \end{array}
$$
Now fix an  $i$  and let  $t_i$  be time index at the beginning of episode  $i$ . Let  $E_t = 1$  indicate the event that episode  $i$  has not ended at time  $t$ . Then