SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 2 (Appendix A.2): For the first part corresponding to model estimation error, first by Lemmas A.13 and A.15, we show that the model estimation error can be bounded by the average of the truncated value functions of the bonus terms i.e.  $\frac{1}{K}\sum_{k=1}^{K}\hat{V}_{\hat{P}^k,\hat{b}^k}^\pi$  plus a term w.r.t. variation budgets. We then upper bound the average of  $\hat{V}_{\hat{P}^k,\hat{b}^k}^\pi$  as in Lemma A.18. To this end, we divide the total  $K$  rounds into blocks with equal length of  $W$  and adopt an auxiliary anchor representation for each block to deal with the challenge arising from time-varying representations when using standard elliptical potential based methods.
Step 3 (Appendix A.3): For the second part corresponding to the performance difference bound, similarly to dealing with changing representations, since the optimal policy changes over time, we adopt an auxiliary anchor policy and decompose the performance difference bound into two terms as in Equation (10) and bound the two terms separately.
We further note that the above analysis techniques can also be applied to RL problems where model mis-specification exists, i.e.  $\phi^{\star ,k}\notin \Phi ,\mu^{\star ,k}\notin \Psi$
Organization of the Proof for Theorem 4.4. Our proof of Theorem 4.4 is organized as follows. In Appendix A.1, we provide the decomposition of the average dynamic suboptimality gap  $\mathrm{Gap}_{\mathrm{Ave}}$  in Equation (7); in Appendix A.2, we bound the first and third terms of  $\mathrm{Gap}_{\mathrm{Ave}}$ ; in Appendix A.3,
we bound the second term of  $\mathrm{Gap}_{\mathrm{Ave}}$ , and in Appendix A.4 we combine our results to complete the proof of Theorem 4.4. We provide all the supporting lemmas in Appendix A.5.
A.1 Average Suboptimality Gap Decomposition
Lemma A.1. We denote  $\pi_h^{\star ,k}(\cdot |s) = \arg \max_{\pi}V_{P^{\star ,k},r^{k}}^{\pi}$ . Then the average dynamic suboptimality gap has the following decomposition:
$$
\begin{array}{l} \operatorname {G a p} _ {\operatorname {A v e}} (K) = \frac {1}{K} \sum_ {k = 1} ^ {K} V _ {P ^ {\star , k}, r ^ {k}} ^ {\star} - V _ {P ^ {\star , k}, r ^ {k}} ^ {\pi^ {k}} \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h \in [ H ]} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \pi^ {\star , k})} \left[ \left\{P _ {h} ^ {\star , k} - \hat {P} _ {h} ^ {k} \right\} V _ {h + 1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} \right] \\ + \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h \in H} \mathbb {E} _ {s _ {h} \sim (P ^ {\star , k}, \pi^ {\star , k})} \left[ \langle Q _ {h, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ + \frac {1}{K} \sum_ {k = 1} ^ {K} V _ {\hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} - V _ {P ^ {\star , k}, r ^ {k}} ^ {\pi^ {k}} \tag {7} \\ \end{array}
$$
Proof. For any function  $f: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  and any  $(k, h, s) \in [K] \times [H] \times \mathcal{S}$ , define the following operators:
$$
\left(\mathbb {J} _ {k, h} ^ {\star} f\right) (s) = \langle f (s, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s) \rangle , \quad \left(\mathbb {J} _ {k, h} f\right) (s) = \langle f (s, \cdot), \pi_ {h} ^ {k} (\cdot | s) \rangle .
$$
We next consider the following decomposition:
$$
V _ {P ^ {\star}, k, r ^ {k}} ^ {\star} - V _ {P ^ {\star}, k, r ^ {k}} ^ {\pi^ {k}} = \underbrace {V _ {P ^ {\star} , k , r ^ {k}} ^ {\star} - V _ {\hat {P} ^ {k} , r ^ {k}} ^ {\pi^ {k}}} _ {G _ {1}} + V _ {\hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} - V _ {P ^ {\star}, k, r ^ {k}} ^ {\pi^ {k}}, \tag {8}
$$
The term  $G_{1}$  can be bounded as follows:
$$
\begin{array}{l} G _ {1} = V _ {P ^ {\star , k}, r ^ {k}} ^ {\star} - V _ {\hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} \\ = \left(\mathbb {J} _ {k, 1} ^ {\star} Q _ {1, P ^ {\star , k}, r ^ {k}} ^ {\pi^ {\star , k}}\right) - \left(\mathbb {J} _ {k, 1} Q _ {1, \hat {P} _ {k}, r ^ {k}} ^ {\pi^ {k}}\right) \\ = \left(\mathbb {J} _ {k, 1} ^ {\star} \left(Q _ {1, P ^ {\star}, k, r ^ {k}} ^ {\pi^ {\star , k}} - Q _ {1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}\right)\right) + \left(\left(\mathbb {J} _ {k, 1} ^ {\star} - \mathbb {J} _ {k, 1}\right) Q _ {1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}\right) \\ = \left(\mathbb {J} _ {k, 1} ^ {\star} \left(r _ {1} ^ {k} (s, \cdot) + P _ {1} ^ {\star , k} V _ {2, P ^ {\star , k}, r ^ {k}} ^ {\pi^ {\star , k}} - \left(r _ {1} ^ {k} (s, \cdot) + \hat {P} _ {1} ^ {k} V _ {2, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}\right)\right)\right) + \left(\left(\mathbb {J} _ {k, 1} ^ {\star} - \mathbb {J} _ {k, 1}\right) Q _ {1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}\right) \\ = (\mathbb {J} _ {k, 1} ^ {\star} (P _ {1} ^ {\star , k} V _ {2, P ^ {\star , k}, r ^ {k}} ^ {\pi^ {\star , k}} - \hat {P} _ {1} ^ {k} V _ {2, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}})) + ((\mathbb {J} _ {k, 1} ^ {\star} - \mathbb {J} _ {k, 1}) Q _ {1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}) \\ = \left(\mathbb {J} _ {k, 1} ^ {\star} \left(P _ {1} ^ {\star , k} \left\{V _ {2, P ^ {\star , k}, r ^ {k}} ^ {\pi^ {\star , k}} - V _ {2, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} \right\} + \left\{P _ {1} ^ {\star , k} - \hat {P} _ {1} ^ {k} \right\} V _ {2, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}\right)\right) + \left(\left(\mathbb {J} _ {k, 1} ^ {\star} - \mathbb {J} _ {k, 1}\right) Q _ {1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}\right) \\ = \Big (\mathbb {J} _ {k, 1} ^ {\star} \Big \{P _ {1} ^ {\star , k} - \hat {P} _ {1} ^ {k} \Big \} V _ {2, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} \Big) + \mathbb {E} _ {s _ {2} \sim (P ^ {\star}, k, \pi^ {\star}, k)} \left[ V _ {2, P ^ {\star}, k, r ^ {k}} ^ {\pi^ {\star}, k} (s _ {2}) - V _ {2, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} (s _ {2}) \right] + ((\mathbb {J} _ {k, 1} ^ {\star} - \mathbb {J} _ {k, 1}) Q _ {1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}}) \\ = \sum_ {h \in [ H ]} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \pi^ {\star , k})} \left[ \left\{P _ {h} ^ {\star , k} - \hat {P} _ {h} ^ {k} \right\} V _ {h + 1, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} \right] \\ + \sum_ {h \in [ H ]} \mathbb {E} _ {s _ {h} \sim \left(P ^ {\star , k}, \pi^ {\star , k}\right)} \left[ \left\langle Q _ {h, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} \left(s _ {h}, \cdot\right), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \right\rangle \right] \tag {9} \\ \end{array}
$$
Substituting the above result to Equation (8) completes the proof.
A.2 First and Third Terms of GapAve in Equation (7): Model Estimation Error Bound
A.2.1 First Term in Equation (7)
Lemma A.2. With probability at least  $1 - \delta$ , we have
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h \in [ H ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \pi^ {\star , k}\right)} \left[ \left\{P _ {h} ^ {\star , k} - \hat {P} _ {h} ^ {k} \right\} V _ {h + 1, \hat {P}, r} ^ {\pi^ {k}, k} \right]
$$
$$
\leq O \left(\frac {H}{K} \left[ \sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \left[ H \sqrt {\frac {K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} \right]\right).
$$
Proof. We proceed the proof by deriving the bound:
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h \in [ H ]} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \pi^ {\star , k})} \left[ \left\{P _ {h} ^ {\star , k} - \hat {P} _ {h} ^ {k} \right\} V _ {h + 1, \hat {P}, r} ^ {\pi^ {k}, k} \right] \\ \leq \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h \in [ H ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \pi^ {\star , k}\right)} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \right] \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {a _ {1} \sim \pi^ {\star , k}} \left[ f _ {1} ^ {k} \left(s _ {1}, a _ {1}\right) \right] + \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h = 2} ^ {H} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \pi^ {\star , k}\right)} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \right] \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {a _ {1} \sim \pi^ {\star , k}} \left[ f _ {1} ^ {k} \left(s _ {1}, a _ {1}\right) \right] + \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h = 2} ^ {H} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(\hat {P} ^ {k}, \pi^ {\star , k}\right)} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \right] \\ + \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h = 2} ^ {H} \left\{\mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \pi^ {\star , k}\right)} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \right] - \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(\hat {P} ^ {k}, \pi^ {\star , k}\right)} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \right] \right\} \\ \stackrel {(i)} {\leq} \frac {2}{K} \sum_ {h = 2} ^ {H} \sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi^ {\star , k}} + 2 \sum_ {h = 2} ^ {H} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \sqrt {W A \left(\zeta_ {k , W} + \frac {1}{2} C _ {B} \Delta_ {1 , [ k - W , k - 1 ]} ^ {P}\right)} \right. \\ \left. + \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h ^ {\prime} = 2} ^ {h} \sqrt {\frac {1}{2 d} W A C _ {B} \Delta_ {[ h ^ {\prime} - 1 , h ^ {\prime} ] , [ k - W , k - 1 ]} ^ {P}} \right] \\ \leq \frac {2 H}{K} \sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\tilde {\pi} ^ {k}} + \frac {2 H}{K} \left[ \sum_ {k = 1} ^ {K} \sqrt {W A \left(\zeta_ {k , W} + \frac {1}{2} C _ {B} \Delta_ {1 , [ k - W , k - 1 ]} ^ {P}\right)} \right. \\ \left. + \sum_ {k = 1} ^ {K} \sum_ {h = 2} ^ {H} \sqrt {\frac {1}{2 d} W A C _ {B} \Delta_ {[ h - 1 , h ] , [ k - W , k - 1 ]} ^ {P}} \right] \\ \stackrel {(i i)} {\leq} \frac {2 H}{K} \sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi^ {k}} + \frac {2 H}{K} \sum_ {k = 1} ^ {K} \sqrt {W A \zeta_ {k , W}} + \frac {2 H}{K} \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} \\ \stackrel {(i i i)} {\leq} O \Big (\frac {H}{K} \left[ \sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \Big [ H \sqrt {\frac {K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \Big ] + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} \right] \Big), \\ \end{array}
$$
where  $(i)$  follows from Lemmas A.13 and A.15,  $(ii)$  follows because  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}, \forall a, b \geq 0$  and  $\sum_{k=1}^{K} \Delta_{\{h\}, [k-W,k-1]}^{\sqrt{P}} \leq W \Delta_{\{h\}, [K]}^{\sqrt{P}}$ , and  $(iii)$  follows from Lemma A.18.
A.2.2 Third Term in Equation (7)
Lemma A.3. With probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 1} ^ {K} \left[ V _ {\hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} - V _ {P ^ {\star}, k, r ^ {k}} ^ {\pi^ {k}} \right] \\ \leq O \Big (\frac {1}{K} \sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \Big [ H \sqrt {\frac {K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \Big ] + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} \Big). \\ \end{array}
$$
Proof. We define the model error as  $f_h^k(s_h, a_h) = \left\| P_h^\star(\cdot | s_h, a_h) - \hat{P}_h(\cdot | s_h, a_h) \right\|_{TV}$ . We next derive the following bound:
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 1} ^ {K} \left[ V _ {\hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} - V _ {P ^ {\star}, k, r ^ {k}} ^ {\pi^ {k}} \right] \\ \stackrel {(i)} {\leq} \frac {1}{K} \sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi^ {k}} + \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h = 2} ^ {H} \sqrt {\frac {3}{\lambda_ {W}} W A C _ {B} \Delta_ {\{h - 1 , h \} , [ k - W , k - 1 ]} ^ {P}} \\ \end{array}
$$
$$
+ \frac {1}{K} \sum_ {k = 1} ^ {K} \sqrt {A \left(\zeta_ {k , W} + 2 C _ {B} \Delta_ {1 , [ k - W , k - 1 ]} ^ {P}\right)}
$$
$$
\begin{array}{l} \stackrel {(i i)} {\leq} \frac {1}{K} \sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\tilde {\pi} ^ {k}} + \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h = 2} ^ {H} \sqrt {\frac {3}{\lambda_ {W}} W A C _ {B} \Delta_ {\{h - 1 , h \} , [ k - W , k - 1 ]} ^ {P}} \\ + \frac {1}{K} \sum_ {k = 1} ^ {K} \sqrt {A \left(\zeta_ {k , W} + 2 C _ {B} \Delta_ {1 , [ k - W , k - 1 ]} ^ {P}\right)} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i i)} {\leq} O \left(\frac {1}{K} \sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \left[ H \sqrt {\frac {K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] \right. \\ \left. + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right), \\ \end{array}
$$
where  $(i)$  follows from Lemma A.15,  $(ii)$  follows from the definition of  $\tilde{\pi}^k$ , and  $(iii)$  follows from Lemma A.18.
A.3 Second Term of  $\mathrm{Gap}_{\mathrm{Ave}}$  in Equation (7): Performance Difference Bound
The second term in Lemma A.1  $\frac{1}{K}\sum_{k = 1}^{K}\sum_{h\in H}\mathbb{E}_{s_h\sim (P^{\star ,k},\pi^{\star ,k})}[\langle Q_{h,\hat{P}^{k},r^k}^{\pi^k}(s_h,\cdot),\pi_h^{\star ,k}(\cdot |s_h) - \pi_h^k (\cdot |s_h)\rangle ]$  can be further decomposed as
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {h \in H} \mathbb {E} _ {s _ {h} \sim (P ^ {\star , k}, \pi^ {\star , k})} \left[ \langle Q _ {h, \hat {P} ^ {k}, r ^ {k}} ^ {\pi^ {k}} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ = \frac {1}{K} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {P ^ {\star , k}, \pi^ {\star , k}} \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ = \underbrace {\frac {1}{K} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ \left\langle \hat {Q} _ {h} ^ {k} \left(s _ {h} , \cdot\right) , \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \right\rangle \right]} _ {(a)} \\ + \underbrace {\frac {1}{K} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \left(\mathbb {E} _ {P ^ {\star , k} , \pi^ {\star , k}} - \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}}\right) \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h} , \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] .} _ {(b)} \tag {10} \\ \end{array}
$$
A.3.1 Bound  $(a)$  in Equation (10)
We first present the following lemma of the descent result introduced in Cai et al. (2020).
Lemma A.4. For any distribution  $p^{\star}$  and  $p$  supported on  $\mathcal{A}$  and state  $s\in S$ , and function  $Q:\mathcal{S}\times \mathcal{A}\to [0,H]$ , it holds for a distribution  $p^\prime$  supported on  $\mathcal{A}$  with  $p^{\prime}(\cdot)\propto p(\cdot)\cdot \exp^{\eta \cdot Q(s,\cdot)}$  that
$$
\langle Q (s, \cdot), p ^ {\star} (\cdot) - p (\cdot) \rangle \leq \frac {1}{2} \eta + \frac {1}{\eta} \left[ D _ {K L} \left(p ^ {\star} (\cdot) \| p (\cdot)\right) - D _ {K L} \left(p ^ {\star} (\cdot) \| p ^ {\prime} (\cdot)\right) \right].
$$
Lemma A.5. Term (a) in Equation (10) satisfies the following bound:
$$
\begin{array}{l} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ \leq \frac {1}{2} \eta K H + \frac {1}{\eta} L H \log A + \tau \Delta_ {[ H ], [ K ]} ^ {\pi}. \\ \end{array}
$$
Proof. We first decompose  $(a)$  into two parts:
$$
\begin{array}{l} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ = \underbrace {\sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h} , \cdot) , \pi_ {h} ^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right]} _ {(I)} \\ + \underbrace {\sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ \left\langle \hat {Q} _ {h} ^ {k} \left(s _ {h} , \cdot\right) , \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \right\rangle \right]} _ {(I I)} \\ \end{array}
$$
I) Bound the term (I)
By Lemma A.4, we have
$$
\begin{array}{l} (I) \leq \frac {1}{2} \eta K H + \sum_ {h \in [ H ]} \frac {1}{\eta} \sum_ {l \in [ L ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \\ \times \left[ \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \left[ D _ {K L} \left(\pi^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \| \pi^ {k} (\cdot | s _ {h})\right) - D _ {K L} \left(\pi^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \| \pi^ {k + 1} (\cdot | s _ {h})\right) \right] \right] \\ \leq \frac {1}{2} \eta K H + \sum_ {h \in [ H ]} \frac {1}{\eta} \sum_ {l \in [ L ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \\ \times \left[ D _ {K L} \left(\pi^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \| \pi^ {(l - 1) \tau + 1} (\cdot | s _ {h})\right) - D _ {K L} \left(\pi^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \| \pi^ {l \tau + 1} (\cdot | s _ {h})\right) \right] \\ \leq \frac {1}{2} \eta K H + \sum_ {h \in [ H ]} \frac {1}{\eta} \sum_ {l \in [ L ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ D _ {K L} \left(\pi^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \| \pi^ {(l - 1) \tau + 1} (\cdot | s _ {h})\right) \right] \\ \leq \frac {1}{2} \eta K H + \frac {1}{\eta} L H \log A, \\ \end{array}
$$
where the last equation follows because
$$
\begin{array}{l} D _ {K L} \left(\pi^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \| \pi^ {(l - 1) \tau + 1} (\cdot | s _ {h})\right) = \sum_ {a \in \mathcal {A}} \pi^ {\star , (l - 1) \tau + 1} (a | s _ {h}) \log (A \cdot \pi^ {(l - 1) \tau + 1} (\cdot | s _ {h})) \\ = \log A + \sum_ {a} \pi^ {\star , (l - 1) \tau + 1} \left(a _ {h} \mid s _ {h}\right) \cdot \log \pi^ {\star , (l - 1) \tau + 1} \left(a _ {h} \mid s _ {h}\right) \\ \leq \log A. \\ \end{array}
$$
II) Bound the term (II)
$$
\begin{array}{l} \left. \left. (I I) \leq \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ \left\| \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {\star , (l - 1) \tau + 1} (\cdot | s _ {h}) \right\| _ {T V} \right] \right. \right. \\ \leq \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \sum_ {t = (l - 1) \tau + 2} ^ {k} \mathbb {E} _ {P ^ {\star , (l - 1) \tau + 1}, \pi^ {\star , (l - 1) \tau + 1}} \left[ \left\| \pi_ {h} ^ {\star , t} (\cdot | s _ {h}) - \pi_ {h} ^ {\star , t - 1} (\cdot | s _ {h}) \right\| _ {T V} \right] \\ \leq \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {t = (l - 1) \tau + 2} ^ {k} \sum_ {h \in [ H ]} \max  _ {s \in \mathcal {S}} \left[ \left\| \pi_ {h} ^ {\star , t} (\cdot | s) - \pi_ {h} ^ {\star , t - 1} (\cdot | s) \right\| _ {T V} \right] \\ \leq \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {t = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \max  _ {s \in \mathcal {S}} \left[ \left\| \pi_ {h} ^ {\star , t} (\cdot | s) - \pi_ {h} ^ {\star , t - 1} (\cdot | s) \right\| _ {T V} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \tau \sum_ {k \in [ K ]} \sum_ {h \in [ H ]} \max  _ {s \in \mathcal {S}} \left[ \left\| \pi_ {h} ^ {\star , k} (\cdot | s) - \pi_ {h} ^ {\star , k - 1} (\cdot | s) \right\| _ {T V} \right] \\ \leq \tau \Delta_ {[ H ], [ K ]} ^ {\pi}. \\ \end{array}
$$
A.3.2 Bound  $(b)$  in Equation (10)
Lemma A.6. The term (b) in Equation (10) can be bounded as follows:
$$
\begin{array}{l} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \left(\mathbb {E} _ {P ^ {\star , k}, \pi^ {\star , k}} - \mathbb {E} _ {P ^ {\star}, (l - 1) \tau + 1, \pi^ {\star}, (l - 1) \tau + 1}\right) \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ \leq 2 H \tau (\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}). \\ \end{array}
$$
Proof. Denote the indicator function of state  $s_h$  as  $\mathbb{I}(s_h)$ , and then we have
$$
\begin{array}{l} \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \left(\mathbb {E} _ {P ^ {\star , k}, \pi^ {\star , k}} - \mathbb {E} _ {P ^ {\star}, (l - 1) \tau + 1, \pi^ {\star}, (l - 1) \tau + 1}\right) \left[ \langle \hat {Q} _ {h} ^ {k} (s _ {h}, \cdot), \pi_ {h} ^ {\star , k} (\cdot | s _ {h}) - \pi_ {h} ^ {k} (\cdot | s _ {h}) \rangle \right] \\ \leq \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \int_ {s _ {h}} \left| \mathbb {P} _ {P ^ {\star}, k} ^ {\pi^ {\star}, k} \left(s _ {h}\right) - \mathbb {P} _ {P ^ {\star}, (l - 1) \tau + 1} ^ {\pi^ {\star}, (l - 1) \tau + 1} \left(s _ {h}\right) \right| d s _ {h} \\ \leq \sum_ {l \in [ L ]} \sum_ {k = (l - 1) \tau + 1} ^ {l \tau} \sum_ {h \in [ H ]} \sum_ {t = (l - 1) \tau + 2} ^ {k} \int_ {s _ {h}} \left| \mathbb {P} _ {P ^ {\star , t}} ^ {\pi^ {\star , t}} \left(s _ {h}\right) - \mathbb {P} _ {P ^ {\star , t - 1}} ^ {\pi^ {\star , t - 1}} \left(s _ {h}\right) \right| d s _ {h}, \tag {11} \\ \end{array}
$$
where  $\mathbb{P}_P^\pi (s)$  denotes the visitation probability at state  $s$  under model  $P$  and policy  $\pi$ .
Consider  $\int_{s_h}\left|\mathbb{P}_{P^{\star ,t}}^{\pi^{\star ,t}}(s_h) - \mathbb{P}_{P^{\star ,t - 1}}^{\pi^{\star ,t - 1}}(s_h)\right|ds_h$  can be further decomposed as
$$
\begin{array}{l} \int_ {s _ {h}} \left| \mathbb {P} _ {P ^ {\star}, t} ^ {\pi^ {\star}, t} (s _ {h}) - \mathbb {P} _ {P ^ {\star}, t - 1} ^ {\pi^ {\star}, t - 1} (s _ {h}) \right| d s _ {h} \\ \leq \int_ {s _ {h}} \left| \mathbb {P} _ {P ^ {\star}, t} ^ {\pi^ {\star , t - 1}} (s _ {h}) - \mathbb {P} _ {P ^ {\star}, t - 1} ^ {\pi^ {\star , t - 1}} (s _ {h}) \right| d s _ {h} + \int_ {s _ {h}} \left| \mathbb {P} _ {P ^ {\star}, t} ^ {\pi^ {\star , t}} (s _ {h}) - \mathbb {P} _ {P ^ {\star}, t} ^ {\pi^ {\star , t - 1}} (s _ {h}) \right| d s _ {h}. \\ \end{array}
$$
For the first term  $\int_{s_h}\left|\mathbb{P}_{P^{\star},t}^{\pi^{\star ,t - 1}}(s_h) - \mathbb{P}_{P^{\star},t - 1}^{\pi^{\star ,t - 1}}(s_h)\right|ds_h,$
$$
\int_ {s _ {h}} \left| \mathbb {P} _ {P ^ {\star , t}} ^ {\pi^ {\star , t - 1}} (s _ {h}) - \mathbb {P} _ {P ^ {\star , t - 1}} ^ {\pi^ {\star , t - 1}} (s _ {h}) \right| d s _ {h}
$$
$$
\begin{array}{l} \leq \int_ {s _ {h}} \sum_ {i = 1} ^ {h} \left| \left(P _ {1} ^ {\star , t}\right) ^ {\pi_ {1} ^ {\star , t - 1}} \dots \left(P _ {i} ^ {\star , t}\right) ^ {\pi_ {i} ^ {\star , t - 1}} \dots \left(P _ {h - 1} ^ {\star , t - 1}\right) ^ {\pi_ {h - 1} ^ {\star , t - 1}} (s _ {h}) \right. \\ - \left(P _ {1} ^ {\star , t}\right) ^ {\pi_ {1} ^ {\star , t - 1}} \dots \left(P _ {i} ^ {\star , t - 1}\right) ^ {\pi_ {i} ^ {\star , t - 1}} \dots \left(P _ {h - 1} ^ {\star , t - 1}\right) ^ {\pi_ {h - 1} ^ {\star , t - 1}} \left(s _ {h}\right) \Bigg | d s _ {h} \\ \end{array}
$$