SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \leq \int_ {s _ {h}} \sum_ {i = 1} ^ {h} \left| \int_ {s _ {2}, \dots , s _ {h - 1}} \right. \left| \left(P _ {i} ^ {\star , t}\right) ^ {\pi_ {i} ^ {\star , t - 1}} \left(s _ {i + 1} \mid s _ {i}\right) - \left(P _ {i} ^ {\star , t - 1}\right) ^ {\pi_ {i} ^ {\star , t - 1}} \left(s _ {i + 1} \mid s _ {i}\right) \right| \\ \prod_ {j = 1} ^ {i - 1} \left(P _ {j} ^ {\star , t}\right) ^ {\pi_ {j} ^ {\star , t - 1}} \left(s _ {j + 1} \mid s _ {j}\right) \prod_ {j = i + 1} ^ {h - 1} \left(P _ {j} ^ {\star , t - 1}\right) ^ {\pi_ {j} ^ {\star , t - 1}} \left(s _ {j + 1} \mid s _ {j}\right) d s _ {2} \dots d s _ {h - 1} \Bigg | d s _ {h} \\ \stackrel {(i)} {\leq} \int_ {s _ {h}} \sum_ {i = 1} ^ {h} \left| \int_ {s _ {2}, \dots , s _ {i}, s _ {i + 2}, \dots s _ {h - 1}} \int_ {s _ {i + 1}} \right| \left(P _ {i} ^ {\star , t}\right) ^ {\pi_ {i} ^ {\star , t - 1}} (s _ {i + 1} | s _ {i}) - (P _ {i} ^ {\star , t - 1}) ^ {\pi_ {i} ^ {\star , t - 1}} (s _ {i + 1} | s _ {i}) \Bigg | \\ \end{array}
$$
$$
\begin{array}{l} \max _ {s _ {i + 1} \in \mathcal {S}} \prod_ {j = 1} ^ {i - 1} \left(P _ {j} ^ {\star , t}\right) ^ {\pi_ {j} ^ {\star , t - 1}} (s _ {j + 1} | s _ {j}) \prod_ {j = i + 1} ^ {h - 1} \left(P _ {j} ^ {\star , t - 1}\right) ^ {\pi_ {j} ^ {\star , t - 1}} (s _ {j + 1} | s _ {j}) d s _ {2} \dots d s _ {h - 1} \Bigg | d s _ {h} \\ \stackrel {(i i)} {\leq} \int_ {s _ {h}} \sum_ {i = 1} ^ {h} \left| \int_ {s _ {2}, \ldots , s _ {i - 1}, s _ {i + 2}, \ldots s _ {h - 1}} \max _ {s _ {i} \in \mathcal {S}} \int_ {s _ {i + 1}} \right| (P _ {i} ^ {\star , t}) ^ {\pi_ {i} ^ {\star , t - 1}} (s _ {i + 1} | s _ {i}) - (P _ {i} ^ {\star , t - 1}) ^ {\pi_ {i} ^ {\star , t - 1}} (s _ {i + 1} | s _ {i}) \\ \int_ {s _ {i}} \max  _ {s _ {i + 1} \in \mathcal {S}} \prod_ {j = 1} ^ {i - 1} \left(P _ {j} ^ {\star , t}\right) ^ {\pi_ {j} ^ {\star , t - 1}} (s _ {j + 1} | s _ {j}) \prod_ {j = i + 1} ^ {h - 1} \left(P _ {j} ^ {\star , t - 1}\right) ^ {\pi_ {j} ^ {\star , t - 1}} (s _ {j + 1} | s _ {j}) d s _ {2} \dots d s _ {h - 1} \Bigg | d s _ {h} \\ \stackrel {(i i i)} {\leq} \int_ {s _ {h}} \sum_ {i = 1} ^ {h} \left| \max  _ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left\| P _ {i} ^ {\star , t} (\cdot | s, a) - P _ {i} ^ {\star , t - 1} (\cdot | s, a) \right\| _ {T V} \underbrace {\int_ {s _ {1} , \dots , s _ {i}} \prod_ {j = 1} ^ {i - 1} (P _ {j} ^ {\star , t}) ^ {\pi_ {j} ^ {\star , t - 1}} (s _ {j + 1} | s _ {j}) d s _ {1} \dots d s _ {i}} _ {= 1} \right. \\ \underbrace {\int_ {s _ {i + 2} , \dots , s _ {h - 1}} \max  _ {s _ {i + 1} \in \mathcal {S}} \prod_ {j = i + 1} ^ {h - 1} \left(P _ {j} ^ {\star , t - 1}\right) ^ {\pi_ {j} ^ {\star , t - 1}} \left(s _ {j + 1} | s _ {j}\right) d s _ {i + 2} \dots d s _ {h - 1}} _ {\leq 1} d s _ {h} \\ \leq \sum_ {h \in [ H ]} \max  _ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left\| P _ {i} ^ {\star , t} (\cdot | s, a) - P _ {i} ^ {\star , t - 1} (\cdot | s, a) \right\| _ {T V}, \tag {12} \\ \end{array}
$$
where  $(i)$  and  $(ii)$  follow from Holder's inequality, and  $(iii)$  follows from the definition of total variation distance.
Similarly for the second term and from Lemma A.19
$$
\int_ {s _ {h}} \left| \mathbb {P} _ {P ^ {\star}, t} ^ {\pi^ {\star , t}} \left(s _ {h}\right) - \mathbb {P} _ {P ^ {\star}, t} ^ {\pi^ {\star , t - 1}} \left(s _ {h}\right) \right| d s _ {h} \leq \sum_ {h \in [ H ]} \max  _ {s \in \mathcal {S}} \left\| \pi_ {h} ^ {\star , t} (\cdot | s) - \pi_ {h} ^ {\star , t - 1} (\cdot | s) \right\| _ {T V}. \tag {13}
$$
Plug Equation (12) and Equation (13) into Equation (11), we have
$$
\begin{array}{l} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \left(\mathbb {E} _ {P ^ {\star , k}, \pi^ {\star , k}} - \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}}\right) \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ \leq 2 \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \sum_ {t = (l - 1) \tau + 2} ^ {k} \left(\sum_ {i \in [ H ]} \max _ {s \in \mathcal {S}} \left\| \pi_ {i} ^ {\star , t} (\cdot | s) - \pi_ {i} ^ {\star , t - 1} (\cdot | s) \right\| _ {T V} \right. \\ \left. + \sum_ {i \in [ H ]} \max  _ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left\| P _ {i} ^ {\star , t} (\cdot | s, a) - P _ {i} ^ {\star , t - 1} (\cdot | s, a) \right\| _ {T V}\right) \\ \leq 2 \sum_ {h \in [ H ]} \left(\sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {t = (l - 1) \tau + 1} ^ {l \tau} \sum_ {i \in [ H ]} \left(\max  _ {s \in S} \left\| \pi_ {i} ^ {\star , t} (\cdot | s) - \pi_ {i} ^ {\star , t - 1} (\cdot | s) \right\| _ {T V} \right. \right. \\ \left. \left. + \max  _ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left\| P _ {i} ^ {\star , t} (\cdot | s, a) - P _ {i} ^ {\star , t - 1} (\cdot | s, a) \right\| _ {T V}\right)\right) \\ = 2 H \tau (\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}). \\ \end{array}
$$
A.3.3 Combining  $(a)$  and  $(b)$  Together
Lemma A.7. Let  $\eta = \sqrt{\frac{L\log A}{K}}$ , and we have
$$
\begin{array}{l} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {\pi^ {\star}, k} \left[ \langle \hat {Q} _ {h} ^ {\pi^ {k}, k} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ \leq 2 H K \sqrt {\log A / \tau} + 3 H \tau (\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}). \\ \end{array}
$$
Proof. We derive the following bound:
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h \in H} \mathbb {E} _ {s _ {h} \sim (P ^ {\star , k}, \pi^ {\star , k})} \left[ \langle Q _ {h, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ \stackrel {(i)} {\leq} \frac {1}{2} \eta K H + \frac {1}{\eta} L H \log A + \tau \Delta_ {[ H ], [ K ]} ^ {\pi} + 2 H \tau (\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}) \\ \stackrel {(i i)} {\leq} 2 H \sqrt {K L \log A} + 3 H \tau (\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}) \\ = 2 H K \sqrt {\log A / \tau} + 3 H \tau \left(\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}\right), \\ \end{array}
$$
where  $(i)$  follows from Lemmas A.5 and A.6, and  $(ii)$  follows from the choice of  $\eta = \sqrt{\frac{L\log A}{K}}$ .
A.4 Proof Theorem 4.4
Proof of Theorem 4.4. Combine Lemmas A.1 to A.3 and A.7, we have
$$
\begin{array}{l} \operatorname {G a p} _ {\mathrm {A v e}} (K) = \frac {1}{K} \sum_ {k = 1} ^ {K} V _ {P ^ {\star , k}, r ^ {k}} ^ {\star} - V _ {P ^ {\star , k}, r ^ {k}} ^ {\pi^ {k}} \\ \leq O \Big (\frac {H}{K} \left[ \right. \sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \Big [ H \sqrt {\frac {K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \Big ] + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} \Big ] \Big) \\ + O \left(\frac {2 H}{\sqrt {\tau}} + \frac {3 H \tau}{K} \left(\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}\right)\right) \\ = \underbrace {\tilde {O} \left(\sqrt {\frac {H ^ {4} d ^ {2} A}{W} (A + d ^ {2})} + \sqrt {\frac {H ^ {3} d A}{K} (A + d ^ {2}) W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} + \sqrt {\frac {H ^ {2} W ^ {3} A}{K ^ {2}}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right)} _ {(I)} \\ + \underbrace {\tilde {O} \left(\frac {2 H}{\sqrt {\tau}} + \frac {3 H \tau}{K} \left(\Delta_ {[ H ] , [ K ]} ^ {P} + \Delta_ {[ H ] , [ K ]} ^ {\pi}\right)\right)} _ {(I I)}. \tag {14} \\ \end{array}
$$
A.5 Supporting Lemmas
We first provide the following concentration results, which is an extension of Lemma 39 in Zanette et al. (2021).
Lemma A.8. There exists a constant  $\lambda_{k,W} = O(d\log (|\Phi |\min \{k,W\} TH / \delta))$ , the following inequality holds for any  $k\in [K],h\in [H],s_h\in \mathcal{S},a_h\in \mathcal{A}$  and  $\phi \in \Phi$  with probability at least  $1 - \delta$ :
$$
\frac {1}{5} \| \phi_ {h} (s, a) \| _ {(U _ {h, \phi} ^ {k, W}) ^ {- 1}} \leq \| \phi_ {h} (s, a) \| _ {(\hat {U} _ {h} ^ {k, W}) ^ {- 1}} \leq 3 \| \phi_ {h} (s, a) \| _ {(U _ {h, \phi} ^ {k, W}) ^ {- 1}}
$$
The following result follows directly from Lemma A.8.
Corollary A.9. The following inequality holds for any  $k \in [K], h \in [H], s_h \in S, a_h \in \mathcal{A}$  with probability at least  $1 - \delta$ :
$$
\min \left\{\frac {\tilde {\alpha} _ {k , W}}{5} \left\| \hat {\phi} _ {h} ^ {k} (s _ {h}, a _ {h}) \right\| _ {(U _ {h, \hat {\phi} ^ {k}} ^ {k, W}) ^ {- 1}}, 1 \right\} \leq \hat {b} _ {h} ^ {k} (s _ {h}, a _ {h}) \leq 3 \tilde {\alpha} _ {k, W} \left\| \hat {\phi} _ {h} ^ {k} (s _ {h}, a _ {h}) \right\| _ {(U _ {h, \hat {\phi} ^ {k}} ^ {k, W}) ^ {- 1}},
$$
where  $\tilde{\alpha}_{k,W} = 5\sqrt{2WA\zeta_{k,W} + \lambda_{k,W}d}$ .
We next provide the MLE guarantee for nonstationary RL, which shows that under any exploration policy, the estimation error can be bounded with high probability. Differently from Theorem 21 in Agarwal et al. (2020b), we capture the nonstationarity in the analysis.
Lemma A.10 (Nonstationary MLE Guarantee). Given  $\delta \in (0,1)$ , under Assumptions 4.1 to 4.3, let  $C_B = \sqrt{\frac{C_B}{p_{\min}}}$ , and consider the transition kernels learned from line 3 in Algorithm 2. We have the following inequality holds for any  $n,h\geq 2$  with probability at least  $1 - \delta /2$ :
$$
\frac {1}{W} \sum_ {i = 1 \vee (k - W)} ^ {k - 1} \underset { \begin{array}{c} s _ {h - 1}, a _ {h} \sim \mathcal {U} (\mathcal {A}) \\ s _ {h} \sim P ^ {\star , i} (\cdot | s _ {h - 1}, a _ {h - 1}) \end{array} } {\mathbb {E}} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) ^ {2} \right] \leq \zeta_ {k, W} + 2 C _ {B} \Delta_ {h, [ k - W, k - 1 ]} ^ {P}, \tag {15}
$$
where,  $\zeta_{k,W} := \frac{2\log(2|\Phi||\Psi|kH / \delta)}{W}$ . In addition, for  $h = 1$
$$
\underset {a _ {1} \sim \mathcal {U} (\mathcal {A})} {\mathbb {E}} \left[ f _ {1} ^ {k} (s _ {1}, a _ {1}) ^ {2} \right] \leq \zeta_ {k, W} + 2 C _ {B} \Delta_ {1, [ k - W, k - 1 ]} ^ {P}.
$$
Proof of Lemma A.10. For simplification, we denote  $x = (s,a) \in \mathcal{X}, \mathcal{X} = \mathcal{S} \times \mathcal{A}, y = p^{\star} \in \mathcal{Y}, \mathcal{Y} = \mathcal{S}$ . The model estimation process in Algorithm 1 can be viewed as a sequential conditional probability estimation setting with an instance space  $\mathcal{X}$  and a target space  $\mathcal{V}$ , where the conditional density is given by  $p^{i}(y|x) = P^{\star,i}(y|x)$  for any  $i$ . We are given a dataset  $D := \{(x_{i},y_{i})\}_{i=1 \vee (k-W)}^{k}$ , where  $x_{i} \sim \mathcal{D}_{i} = \mathcal{D}_{i}(x_{1:i-1},y_{1:i-1})$  and  $y_{i} \sim p^{i}(\cdot|x_{i})$ . Let  $D'$  denote a tangent sequence  $\{(x_{i}',y_{i}')\}_{i=1 \vee (k-W)}^{k}$  where  $x_{i}' \sim \mathcal{D}_{i}(x_{1:i-1},y_{1:i-1})$  and  $y_{i}' \sim p^{i}(\cdot|x_{i}')$ . Further, we consider a function class  $\mathcal{F} = \Phi \times \Psi: (\mathcal{X} \times \mathcal{V}) \to \mathbb{R}$  and assume that the reachability condition  $P^{\star,i} \in \mathcal{F}$  holds for any  $i$ .
We first introduce one useful lemma in Agarwal et al. (2020b) to decouple data.
Lemma A.11 (Lemma 24 of Agarwal et al. (2020b)). Let  $D$  be a dataset with at most  $W$  samples and  $D'$  be the corresponding tangent sequence. Let  $L(P, D) = \sum_{i=1}^{k} l(P, (x_i, y_i))$  be any function that decomposes additively across examples where  $l$  is any function. Let  $\widehat{P}(D)$  be any estimator taking as input random variable  $D$  and with range  $\mathcal{F}$ . Then
$$
\mathbb {E} _ {D} \left[ \exp \left(L (\widehat {P} (D), D) - \log \mathbb {E} _ {D ^ {\prime}} \left[ \exp (L (\widehat {P} (D), D ^ {\prime})) \right] - \log | \mathcal {F} |\right) \right] \leq 1.
$$
Suppose  $\widehat{f} (D)$  is learned from the following maximum likelihood problem:
$$
\widehat {P} (D) := \arg \max  _ {P \in \mathcal {F}} \sum_ {\left(x _ {i}, y _ {i}\right) \in D} \log f \left(x _ {i}, y _ {i}\right). \tag {16}
$$
Combining the Chernoff bound and Lemma A.11, we obtain an exponential tail bound, i.e., with probability at least  $1 - \delta$ ,
$$
- \log \mathbb {E} _ {D ^ {\prime}} \left[ \exp \left(L \left(\widehat {P} (D), D ^ {\prime}\right)\right) \right] \leq - L \left(\widehat {P} (D), D\right) + \log | \mathcal {F} | + \log (1 / \delta). \tag {17}
$$
To proceed, we let  $L(P, D) = \sum_{i=1}^{k-1} \frac{1}{2} \log (P^{\star, k}(x_i, y_i) / P(x_i, y_i))$  where  $D$  is a dataset  $\{(x_i, y_i)\}_{i=1}^k$  (and  $D' = \{(x_i', y_i')\}_{i=1}^k$  is tangent sequence). Then  $L(P^{\star, k}, D) = 0 \leq L(\hat{P}, D)$ .
Then, the RHS of Equation (17) can be bounded as
$$
\begin{array}{l} \text {R H S} (1 7) = \sum_ {i = 1 \vee (k - W)} ^ {k} \frac {1}{2} \log \left(P ^ {\star , k} \left(x _ {i}, y _ {i}\right) / \widehat {P} \left(x _ {i}, y _ {i}\right)\right) + \log | \mathcal {F} | + \log (1 / \delta) \\ \leq \log | \mathcal {F} | + \log (1 / \delta) = \log \left(| \Phi | | \Psi | / \delta\right), \tag {18} \\ \end{array}
$$
where the inequality follows because  $\widehat{P}$  is MLE and from the assumption of reachability, and the last equality follows because  $|\mathcal{F}| = |\Phi||\Psi|$ .
Consider for any distribution  $p$  and  $q$  over a domain  $\mathcal{Z}$ . Then the Hellinger distance  $H^{2}(p\|q) = \int \left( \sqrt{p(z)} - \sqrt{q(z)} \right)^{2} dz$  satisfies that
$$
\begin{array}{l} H ^ {2} (p \| q) \\ = \int (\sqrt {p (z)} - \sqrt {q (z)}) ^ {2} d z = \int p (z) + q (z) - 2 \sqrt {p (z)} \sqrt {q (z)} d z \\ = 2 \left(\int 1 - \sqrt {p (z)} \sqrt {q (z)} d z\right) = 2 \left(\int 1 - \sqrt {p (z)} \sqrt {q (z)} d z\right) = 2 \mathbb {E} _ {z \sim q} \left[ 1 - \sqrt {p (z) / q (z)} \right]. \tag {19} \\ \end{array}
$$
Next, the LHS of Equation (17) can be bounded as
LHS of Equation (17)
$$
\begin{array}{l} \stackrel {(i)} {=} - \log \mathbb {E} _ {D ^ {\prime}} \left[ \exp \left(\sum_ {i = 1 \lor (k - W)} ^ {k - 1} - \frac {1}{2} \log \left(\frac {P ^ {\star , k} (x _ {i} ^ {\prime} , y _ {i} ^ {\prime})}{\widehat {P} (x _ {i} ^ {\prime} , y _ {i} ^ {\prime})}\right)\right) \Bigg | D \right] \\ = \sum_ {i = 1 \vee (k - W)} ^ {k - 1} - \log \mathbb {E} _ {D} \left[ \exp \left(- \frac {1}{2} \log \left(\frac {P ^ {\star , k} \left(x _ {i} , y _ {i}\right)}{\widehat {P} \left(x _ {i} , y _ {i}\right)}\right)\right) \right] \\ = \sum_ {i = 1 \vee (k - W)} ^ {k - 1} - \log \mathbb {E} _ {D} \left[ \sqrt {\frac {\widehat {P} \left(x _ {i} , y _ {i}\right)}{P ^ {\star , k} \left(x _ {i} , y _ {i}\right)}} \right] \\ \geq \sum_ {i = 1 \vee (k - W)} ^ {(i i)} \left\{1 - \mathbb {E} _ {D} \left[ \sqrt {\frac {\widehat {P} \left(x _ {i} , y _ {i}\right)}{P ^ {\star , k} \left(x _ {i} , y _ {i}\right)}} \right] \right\} \\ = \sum_ {i = 1 \vee (k - W)} ^ {k - 1} \left\{\mathbb {E} _ {x _ {i} \sim D} \left[ 1 - \mathbb {E} _ {y _ {i} \sim P ^ {\star , i} (\cdot | x _ {i})} \left[ \sqrt {\frac {\widehat {P} (x _ {i} , y _ {i})}{P ^ {\star , k} (x _ {i} , y _ {i})}} \right] \right] \right\} \\ = \sum_ {i = 1 \vee (k - W)} ^ {k - 1} \left\{\mathbb {E} _ {x _ {i} \sim D} \left[ 1 - \mathbb {E} _ {y _ {i} \sim P ^ {\star , k} (\cdot | x _ {i})} \left[ \sqrt {\frac {\hat {P} (x _ {i} , y _ {i})}{P ^ {\star , k} (x _ {i} , y _ {i})}} \right] \right] \right\} \\ + \sum_ {i = 1 \vee (k - W)} ^ {k - 1} \mathbb {E} _ {x _ {i} \sim D} \left[ \mathbb {E} _ {y _ {i} \sim P ^ {\star , k} (\cdot | x _ {i})} \left[ \sqrt {\frac {\hat {P} (x _ {i} , y _ {i})}{P ^ {\star , k} (x _ {i} , y _ {i})}} \right] - \mathbb {E} _ {y _ {i} \sim P ^ {\star , i} (\cdot | x _ {i})} \left[ \sqrt {\frac {\hat {P} (x _ {i} , y _ {i})}{P ^ {\star , k} (x _ {i} , y _ {i})}} \right] \right] \\ \stackrel {(i i i)} {\geq} - W C _ {B} \Delta_ {h, [ k - W, k - 1 ]} ^ {P} + \sum_ {i = 1 \vee (k - W)} ^ {k - 1} \mathbb {E} _ {x _ {i} ^ {\prime} \sim \mathcal {D} _ {i}} \left[ \mathrm {H} ^ {2} \left(P ^ {\star , k} (\cdot | x _ {i} ^ {\prime}) \| \hat {P} (\cdot | x _ {i} ^ {\prime})\right) \right] \\ \stackrel {(i v)} {\geq} - W C _ {B} \Delta_ {h, [ k - W, k - 1 ]} ^ {P} + \frac {1}{2} \sum_ {i = 1 \vee (k - W)} ^ {k - 1} \mathbb {E} _ {x _ {i} \sim \mathcal {D} _ {i}} \left[ \left\| P ^ {\star , k} \left(x _ {i}, \cdot\right) - \widehat {P} \left(x _ {i}, \cdot\right) \right\| _ {T V} ^ {2} \right], \tag {20} \\ \end{array}
$$
where  $(i)$  follows from the above definition of  $L(f,D)$ ,  $(ii)$  follows from the fact that  $1 - x\leq -\log (x)$ ,  $(iii)$  follows from the definition of variation budgets and Equation (19), and  $(iv)$  follows from the fact that  $\| p(\cdot) - q(\cdot)\|_{TV}^2\leq \mathrm{H}^2 (p\| q)$  as indicated in Lemma 2.3 in Tsybakov (2009).
Combining Equations (17), (18) and (20), we have
$$
\frac {1}{W} \sum_ {i = 1 \vee (k - W)} ^ {k - 1} \underset { \begin{array}{c} s _ {h} \sim P ^ {\star , i} (\cdot | s _ {h - 1}, a _ {h - 1}) \\ s _ {h} \sim P ^ {\star , i} (\cdot | s _ {h - 1}, a _ {h - 1}) \end{array} } {\mathbb {E}} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) ^ {2} \right] \leq 2 C _ {B} \Delta_ {h, [ k - W, k - 1 ]} ^ {P} + \frac {2 \log \left(| \Phi | | \Psi | / \delta\right)}{W}. \tag {21}
$$
We substitute  $\delta$  with  $\delta /2nH$  to ensure Equation (21) holds for any  $h\in [H]$  and  $n$  with probability at least  $1 - \delta /2$ , which finishes the proof.
The following lemma extends Lemmas 12 and 13 under infinite discount MDPs under stationary case in Uehara et al. (2022) to episodic MDPs under nonstationary environment, which captures nonstationarity in analysis.
Lemma A.12 (Nonstationary Step Back). Let  $\mathcal{I} = [1 \vee (k - W), k - 1]$  be an index set and  $\{P_{h - 1}^i\}_{i \in \mathcal{I}} = \{\langle \phi_{h - 1}^i, \mu_{h - 1}^i \rangle\}$  be a set of generic MDP model, and  $\Pi$  be an arbitrary and possibly mixture policy. Define an expected Gram matrix as follows. For any  $\phi \in \Phi$ ,
$$
M_{h - 1,\phi} = \lambda_{W}I + \sum_{i\in \mathcal{I}}\underset { \begin{array}{c}s_{h - 1}\sim (P^{i,\star},\Pi)\\ a_{h - 1}\sim \Pi \end{array} }{\mathbb{E}}\left[\phi_{h - 1}(s_{h - 1},a_{h - 1})\left(\phi_{h - 1}(s_{h - 1},a_{h - 1})\right)^{\top}\right].
$$
Further, let  $f_{h - 1}^{i}(s_{h - 1},a_{h - 1})$  be the total variation between  $P_{h - 1}^{i,\star}$  and  $P_{h - 1}^{i}$  at time step  $h - 1$ . Suppose  $g\in \mathcal{S}\times \mathcal{A}\to \mathbb{R}$  is bounded by  $B\in (0,\infty)$ , i.e.,  $\| g\|_{\infty}\leq B$ . Then,  $\forall h\geq 2,\forall$  policy  $\pi_h$
$$
\begin{array}{l} \mathbb{E}_{\substack{s_{h}\sim P_{h - 1}^{k}\\ a_{h}\sim \pi_{h}}}\left[g(s_{h},a_{h})|s_{h - 1},a_{h - 1}\right] \\ \leq \left\| \phi_ {h - 1} ^ {k} \left(s _ {h - 1}, a _ {h - 1}\right) \right\| _ {\left(M _ {h - 1, \phi^ {k}}\right) ^ {- 1}} \times \\ \sqrt{\sum_{i\in\mathcal{I}}A\mathbb{E}_{s_{h}\sim(P^{i,\star},\Pi)}[g(s_{h},a_{h})^{2}] + WB^{2}\Delta_{h - 1,\mathcal{I}}^{P} + \lambda_{k,W}dB^{2} + \sum_{i\in\mathcal{I}}AB^{2}\mathbb{E}_{s_{h - 1}\sim(P^{i,\star},\Pi)}\left[f_{h - 1}^{k}(s_{h - 1},a_{h - 1})^{2}\right]}. \\ \end{array}
$$
Proof. We first derive the following bound:
$$
\begin{array}{l} \mathbb{E}_{\substack{s_{h}\sim P_{h - 1}^{k}\\ a_{h}\sim \pi_{h}}}\left[g(s_{h},a_{h})\big|s_{h - 1},a_{h - 1}\right] \\ = \int_ {s _ {h}} \sum_ {a _ {h}} g \left(s _ {h}, a _ {h}\right) \pi \left(a _ {h} \mid s _ {h}\right) \left\langle \phi_ {h - 1} ^ {k} \left(s _ {h - 1}, a _ {h - 1}\right), \mu_ {h - 1} ^ {k} \left(s _ {h}\right) \right\rangle d s _ {h} \\ \leq \left\| \phi_ {h - 1} ^ {k} (s _ {h - 1}, a _ {h - 1}) \right\| _ {(M _ {h - 1, \phi^ {k}}) ^ {- 1}} \left\| \int \sum_ {a _ {h}} g (s _ {h}, a _ {h}) \pi (a _ {h} | s _ {h}) \mu_ {h - 1} ^ {k} (s _ {h}) d s _ {h} \right\| _ {M _ {h - 1, \phi^ {k}}}, \\ \end{array}
$$
where the inequality follows from Cauchy-Schwarz inequality. We further expand the second term in the RHS of the above inequality as follows.
$$
\begin{array}{l} \left\| \int \sum_ {a _ {h}} g (s _ {h}, a _ {h}) \pi (a _ {h} | s _ {h}) \mu_ {h - 1} ^ {k} (s _ {h}) d s _ {h} \right\| _ {M _ {h - 1, \phi^ {k}}} ^ {2} \\ \stackrel {(i)} {\leq} \sum_ {i \in \mathcal {I}} \underset { \begin{array}{c} s _ {h - 1} \sim (P ^ {i, \star}, \Pi) \\ a _ {h - 1} \sim \Pi \end{array} } {\mathbb {E}} \left[ \left(\int_ {s _ {h}} \sum_ {a _ {h}} g (s _ {h}, a _ {h}) \pi_ {h} (a _ {h} | s _ {h}) \mu^ {k} (s _ {h}) ^ {\top} \phi^ {k} (s _ {h - 1}, a _ {h - 1}) d s _ {h}\right) ^ {2} \right] + \lambda_ {k, W} d B ^ {2} \\ = \sum_{i\in \mathcal{I}}\underset { \begin{array}{c}s_{h - 1}\sim (P^{i,\star},\Pi)\\ a_{h - 1}\sim \Pi \end{array} }{\mathbb{E}}\left[\left(\underset { \begin{array}{c}s_{h}\sim P^{k}_{h - 1}\\ a_{h}\sim \pi_{h} \end{array} }{\mathbb{E}}\left[g(s_{h},a_{h})\bigg|s_{h - 1},a_{h - 1}\right]\right)^{2}\right] + \lambda_{k,W}dB^{2} \\ \stackrel {(i i)} {\leq} \sum_ {i \in \mathcal {I}} \underset { \begin{array}{c} s _ {h - 1} \sim (P ^ {i}, \star , \Pi) \\ a _ {h - 1} \sim \Pi \end{array} } {\mathbb {E}} \left[ \underset { \begin{array}{c} s _ {h} \sim P _ {h - 1} ^ {k} \\ a _ {h} \sim \pi_ {h} \end{array} } {\mathbb {E}} \left[ g (s _ {h}, a _ {h}) ^ {2} \Big | s _ {h - 1}, a _ {h - 1} \right] \right] + \lambda_ {k, W} d B ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i i)} {\leq} \sum_ {i \in \mathcal {I}} \underset { \begin{array}{c} s _ {h - 1} \sim (P ^ {i, \star}, \Pi) \\ a _ {h - 1} \sim \Pi \end{array} } {\mathbb {E}} \left[ \underset { \begin{array}{c} s _ {h} \sim P _ {h - 1} ^ {i, \star} \\ a _ {h} \sim \pi_ {h} \end{array} } {\mathbb {E}} \left[ g (s _ {h}, a _ {h}) ^ {2} \Bigg | s _ {h - 1}, a _ {h - 1} \right] \right] + \lambda_ {k, W} d B ^ {2} \\ +\sum_{i\in \mathcal{I}}B^{2}\underset {s_{h - 1}\sim (P^{i,\star},\Pi)\\ a_{h - 1}\sim \Pi}{\mathbb{E}}\left[\left\| P^{i,\star}_{h - 1}(s_{h - 1},a_{h - 1}) - P^{k,\star}_{h - 1}(s_{h - 1},a_{h - 1})\right\|_{TV}\right] \\ +\sum_{i\in \mathcal{I}}B^{2}\underset { \begin{array}{c}s_{h - 1}\sim (Pi,\\ a_{h - 1}\sim \Pi \end{array} }{\mathbb{E}}\left[f_{h - 1}^{k}(s_{h - 1},a_{h - 1})^{2}\right] \\ \end{array}
$$
$$
\stackrel {(i v)} {\leq} \sum_ {i \in \mathcal {I}} A \mathbb {E} _ {s _ {h} \underset {a _ {h} \sim \mathcal {U}} {\sim (P ^ {i, \star}, \Pi)}} \left[ g (s _ {h}, a _ {h}) ^ {2} \right] + \lambda_ {k, W} d B ^ {2} + W B ^ {2} \Delta_ {h - 1, \mathcal {I}} ^ {P} + \sum_ {i \in \mathcal {I}} B ^ {2} \mathbb {E} _ {s _ {h - 1} \underset {a _ {h - 1} \sim \Pi} {\sim (P ^ {i, \star}, \Pi)}} \left[ f _ {h - 1} ^ {k} (s _ {h - 1}, a _ {h - 1}) ^ {2} \right],
$$
where  $(i)$  follows from the assumption that  $\| g\|_{\infty}\leq B$ $(ii)$  follows from Jensen's inequality,  $(iii)$  follows because  $f_{h - 1}^{k}(s_{h - 1},a_{h - 1})$  is the total variation between  $P_{h - 1}^{k,\star}$  and  $P_{h - 1}^{k}$  at time step  $h - 1$  and  $(iv)$  follows from importance sampling and the definition of  $\Delta_{h - 1,\mathcal{I}}^P$ . This finishes the proof.
Recall that  $f_h^k(s, a) = \|\hat{P}_h^k(\cdot|s, a) - P_h^{\star,k}(\cdot|s, a)\|_{TV}$ . Using Lemma A.12, we have the following lemma to bound the expectation of  $f_h^k(s, a)$  under estimated transition kernels.
Lemma A.13. Denote  $\alpha_{k,W} = \sqrt{2WA\zeta_{k,W} + \lambda_{k,W}d}$ . For any  $k \in [K]$ , policy  $\pi$  and reward  $r$ , for all  $h \geq 2$ , we have
$$
\begin{array}{l} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (\hat {P} ^ {k}, \pi)} \left[ f _ {h} ^ {k} (s _ {h}, a _ {h}) \Bigg | s _ {h - 1}, a _ {h - 1} \right] \\ \leq \min  \left\{\alpha_ {k, W} \left\| \hat {\phi} _ {h - 1} ^ {k} \left(s _ {h - 1}, a _ {h - 1}\right) \right\| _ {\left(U _ {h - 1, \hat {\phi} ^ {k}} ^ {k}\right) ^ {- 1}}, 1 \right\} + \sqrt {\frac {1}{2 d} W A C _ {B} \Delta_ {[ h - 1 , h ] , [ k - W , k - 1 ]} ^ {P}}, \tag {22} \\ \end{array}
$$
and for  $h = 1$ , we have
$$
\underset {a _ {1} \sim \pi} {\mathbb {E}} \left[ f _ {1} ^ {k} \left(s _ {1}, a _ {1}\right) \right] \leq \sqrt {A \left(\zeta_ {k , W} + \frac {1}{2} C _ {B} \Delta_ {1 , [ k - W , k - 1 ]} ^ {P}\right)}. \tag {23}
$$
Proof. For  $h = 1$ , we have