SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} U _ {h} ^ {k} (s, a) = H \min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} + \widehat {\mathbb {P}} ^ {\dagger} (s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} U _ {h + 1} ^ {k} (\cdot , a ^ {\prime})\right) \\ = H \min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} + \sum_ {s ^ {\prime}, a ^ {\prime}} \widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \pi_ {h + 1} ^ {k} \left(a ^ {\prime} \mid s ^ {\prime}\right) U _ {h + 1} ^ {k} \left(s ^ {\prime}, a ^ {\prime}\right). \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{K _ {u c b}} \sum_ {k = 1} ^ {K _ {u c b}} U _ {1} ^ {k} (s _ {1}, a) \\ \leq \frac {1}{K _ {u c b}} \sum_ {k = 1} ^ {K _ {u c b}} \max  _ {a} U _ {1} ^ {k} (s _ {1}, a) \\ \leq \frac {H}{K _ {u c b}} \sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h = 1} ^ {H} \sum_ {(s, a)} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a) \min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} \tag {definitionA.9} \\ \end{array}
$$
$$
\leq \frac {4 H ^ {2}}{K _ {u c b}} \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \log \left(e \left(1 + \frac {K _ {u c b} H}{| \mathcal {S} |}\right)\right) \right] \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K _ {u c b}} \sum_ {(s, a)} \frac {\widehat {d} _ {\pi^ {k} , h} ^ {\dagger} (s , a)}{\max  \left\{1 , \sum_ {i = 1} ^ {k - 1} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i} , h} ^ {\dagger} (s , a) \right\}}
$$
We apply Lemma D.7 and Jensen's Inequality to obtain
$$
\begin{array}{l} \sum_ {h \in [ H ]} \sum_ {k = 1} ^ {K _ {u c b}} \sum_ {(s, a)} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a) \frac {1}{\max \left\{1 , \sum_ {i = 1} ^ {k - 1} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i} , h} ^ {\dagger} (s , a) \right\}} \\ \leq 4 \sum_ {(s, a)} \log \left(1 + \sum_ {i = 1} ^ {K _ {u c b}} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a)\right) \tag {lemma D.7} \\ \leq 4 \left| \mathcal {S} \right| \left| \mathcal {A} \right| \log \left(1 + \frac {1}{\left| \mathcal {S} \right| \left| \mathcal {A} \right|} \sum_ {i = 1} ^ {K _ {u c b}} \sum_ {h \in [ H ]} \sum_ {(s, a)} \hat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a)\right) \\ \leq 4 \left| \mathcal {S} \right| \left| \mathcal {A} \right| \log \left(1 + \frac {H K _ {u c b}}{\left| \mathcal {S} \right| \left| \mathcal {A} \right|}\right) \\ \end{array}
$$
Inserting the last display to the upper bound, we conclude the proof of the upper bound on  $U$ .
Then, to prove the sample complexity, we use lemma D.8. We take
$$
B = \frac {6 4 H ^ {2} | \mathcal {A} |}{\varepsilon^ {2}} \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \right] \text {a n d} x = \frac {K _ {u c b}}{| \mathcal {S} |}
$$
in Lemma D.8. From lemma D.8, we know there exists a universal constant  $C_1$  such that when  $x \geq C_1 B \log(B)^2$ , it holds that  $B \log(1 + x) (1 + \log(1 + x)) \leq x$ , which implies whenever
$$
K _ {u c b} \geq \frac {6 4 H ^ {2} | \mathcal {S} | | \mathcal {A} |}{\varepsilon^ {2}} \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \right],
$$
$$
\frac {6 4 H ^ {2} | \mathcal {S} | | \mathcal {A} |}{K _ {u c b}} \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \right] \log \left(1 + \frac {K _ {u c b}}{| \mathcal {S} |}\right) \left[ 1 + \log \left(1 + \frac {K _ {u c b}}{| \mathcal {S} |}\right) \right] \leq \varepsilon^ {2}. \tag {A.11}
$$
For notation simplicity, we define
$$
L = 1 6 H ^ {2} | \mathcal {S} | | \mathcal {A} | \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \right].
$$
Comparing the left hand side of (A.11) and the right hand side of (A.10), we know
Right hand side of (A.10)
$$
\begin{array}{l} = \frac {1 6 H ^ {2} | \mathcal {S} | | \mathcal {A} |}{K _ {u c b}} \log \left(1 + \frac {H K _ {u c b}}{| \mathcal {S} | | \mathcal {A} |}\right) \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | + | \mathcal {S} | \log \left(1 + \frac {K _ {u c b} H}{| \mathcal {S} |}\right) \right] \\ \leq \frac {1 6 H ^ {2} | \mathcal {S} | | \mathcal {A} |}{K _ {u c b}} \left[ \log (H) + \log \left(1 + \frac {K _ {u c b}}{| \mathcal {S} |}\right) \right] \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | + | \mathcal {S} | \log \left(1 + \frac {K _ {u c b} H}{| \mathcal {S} |}\right) \right] \\ \leq \frac {1 6 H ^ {2} | \mathcal {S} | | \mathcal {A} |}{K _ {u c b}} \left[ \log (H) + \log \left(1 + \frac {K _ {u c b}}{| \mathcal {S} |}\right) \right] \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \right] \left[ 1 + \log (H) + \log \left(1 + \frac {K _ {u c b}}{| \mathcal {S} |}\right) \right] \\ = \frac {L}{K _ {u c b}} \left[ \log (H) + \log \left(1 + \frac {K _ {u c b}}{| S |}\right) \right] \left[ 1 + \log (H) + \log \left(1 + \frac {K _ {u c b}}{| S |}\right) \right]. \\ \end{array}
$$
It is easy to see that  $\log (H)\leq \log \left(1 + K_{ucb} / |\mathcal{S}|\right)$ , so the right hand side of last inequality is upper bounded by
$$
\frac {4 L}{K _ {u c b}} \log \left(1 + \frac {K _ {u c b}}{| \mathcal {S} |}\right) \left[ 1 + \log \left(1 + \frac {K _ {u c b}}{| \mathcal {S} |}\right) \right].
$$
From (A.11), we know this is upper bounded by  $\varepsilon^2$  as long as  $K_{ucb} \geq 4L / \varepsilon^2$ . From the definition of L and equations (A.10) (A.11), we have when  $K_{ucb} \geq 4L / \varepsilon^2$ , it holds that
$$
\frac {1}{K _ {u c b}} \sum_ {k = 1} ^ {K _ {u c b}} U _ {1} ^ {k} (s _ {1}, a) \leq \text {r i g h t h a n d s i d e o f (A . 1 0)} \leq \text {l e f t h a n d s i d e o f (A . 1 1)} \leq \varepsilon^ {2}.
$$
In conclusion, this admits a sample complexity of
$$
K _ {u c b} = \frac {C H ^ {2} | \mathcal {S} | | \mathcal {A} |}{\varepsilon^ {2}} \left(\iota + | \mathcal {S} |\right) \operatorname {p o l y l o g} \left(| \mathcal {S} |, | \mathcal {A} |, H, \frac {1}{\varepsilon}, \frac {1}{\delta}\right).
$$
A.3 Omitted proofs
A.3.1 Proof for lemma A.4 (high probability event)
The lemma A.4 is proved by combining all the lemmas below via a union bound. Below, we always denote  $N(s, a, s')$  as the number of  $(s, a, s')$  in the offline dataset.
Lemma A.10.  $\mathcal{E}^P$  holds with probability at least  $1 - \delta /6$
Proof. For any fixed  $(s, a, s')$  such that  $N(s, a, s') \geq \Phi$ , we denote  $n_i(s, a)$  as the index of the i-th time when we visit  $(s, a)$ . For notation simplicity, we fix the state-action pair  $(s, a)$  here and write  $n_i(s, a)$  as  $n_i$  simply for  $i = 1, 2, \dots, N(s, a)$ . Notice that the total visiting time  $N(s, a)$  is random, so our argument is based on conditional probability. We denote  $X_i = \mathbb{I}\left(s_{n_i}' = s'\right)$  as the indicator of whether the next state is  $s'$  when we visited  $(s, a)$  at the i-th time. From the data generating mechanism and the definition for the reference MDP, we know conditional on the total number of visiting  $N(s, a)$  and all the indexes  $n_1 \leq \ldots \leq n_{N(s, a)}$ , it holds that  $X_1, X_2, \ldots, X_{N(s, a)}$  are independent Bernoulli random variables with successful probability being  $\mathbb{P}^{\dagger}(s' \mid s, a)$ . We denote  $\overline{X}$  as their arithmetic average. Using Empirical Bernstein Inequality (Lemma D.3), and the fact that  $\frac{1}{N(s, a)} \sum_{i=1}^{N(s, a)} \left(X_i - \overline{X}\right)^2 \leq \frac{1}{N(s, a)} \sum_{i=1}^{N(s, a)} X_i^2 = \frac{1}{N(s, a)} \sum_{i=1}^{N(s, a)} X_i = \widehat{\mathbb{P}}^{\dagger}(s' \mid s, a)$ , we have
$$
\mathbb {P} \left(\mathcal {E} ^ {P} \mid n _ {i} (1 \leq i \leq N (s, a)), N (s, a) = n\right) \geq 1 - \delta / 6.
$$
Taking integral of the conditional probability, we conclude that the unconditional probability of the event  $\mathcal{E}^P$  is at least  $1 - \delta /6$ . Therefore, we conclude.
Lemma A.11. For  $\delta > 0$ , it holds that  $\mathbb{P}\left(\mathcal{E}^4\right) \geq 1 - \delta / 6$ .
Proof. Consider for a fixed triple  $(s, a) \in S \times \mathcal{A}$ . Let  $m(s, a)$  denote the number of times the tuple  $(s, a)$  was encountered in total during the online interaction phase. We define  $X_i$  as follows. For  $i \leq m(s, a)$ , we let  $X_i$  be the subsequent state  $s'$  when  $(s, a)$  was encountered the  $i$ -th time in the whole run of the algorithm. Otherwise, we let  $X_i$  be an independent sample from  $\mathbb{P}^{\dagger}(s, a)$ . By construction,  $\{X_i\}$  is a sequence of i.i.d. categorical random variables from  $S$  with distribution  $\mathbb{P}^{\dagger}(\cdot \mid s, a)$ . We denote  $\widetilde{\mathbb{P}}_X^{\dagger,i} = \frac{1}{i} \sum_{j=1}^{i} \delta_{X_j}$  as the empirical probability mass and  $\mathbb{P}_X^{\dagger}$  as the probability mass of  $X_i$ . Then, from Lemma D.4, we have with probability at least  $1 - \delta/6$ , it holds that for any  $i \in \mathbb{N}$ ,
$$
\mathrm {K L} \left(\widetilde {\mathbb {P}} _ {X} ^ {\dagger , i}; \mathbb {P} _ {X} ^ {\dagger}\right) \leq \frac {1}{i} \left[ \log \left(\frac {6}{\delta}\right) + | \mathcal {S} | \log \left(e \left(1 + \frac {i}{| \mathcal {S} |}\right)\right) \right],
$$
$$
\operatorname {K L} \left(\widetilde {\mathbb {P}} _ {X} ^ {\dagger}; \mathbb {P} ^ {\dagger} (s, a)\right) \leq \frac {1}{m (s , a)} \left[ \log \left(\frac {6}{\delta}\right) + | \mathcal {S} | \log \left(e \left(1 + \frac {m (s , a)}{| \mathcal {S} |}\right)\right) \right].
$$
Using a union bound for  $(s,a)\in S\times A$  , we conclude.
Lemma A.12 (Lower Bound on Counters). For  $\delta > 0$ , it holds that  $\mathbb{P}\left(\mathcal{E}^2\right) \geq 1 - \delta / 6$  and  $\mathbb{P}\left(\mathcal{E}^5\right) \geq 1 - \delta / 6$ .
Proof. We denote  $n_h^k$  as the number of times we encounter  $(s, a)$  at stage  $h$  before the beginning of episode  $k$ . Concretely speaking, we define  $n_h^1(s, a) = 0$  for any  $(h, s, a)$ . Then, we define
$$
n _ {h} ^ {k} (s, a) = n _ {h} ^ {k} (s, a) + \mathbb {I} \left[ (s, a) = (s _ {h} ^ {k}, a _ {h} ^ {k}) \right].
$$
We fixed an  $(s,a,h)\in [H]\times \mathcal{S}\times \mathcal{A}$  and denote  $\mathcal{F}_k$  as the sigma field generated by the first  $k - 1$  episodes when running RF-UCB and  $X_{k} = \mathbb{I}\left[(s_{h}^{k},a_{h}^{k}) = (s,a)\right]$ . Then, we know  $X_{k}$  is  $\mathcal{F}_{k + 1}$ -measurable and  $\mathbb{E}\left[X_k\mid \mathcal{F}_k\right] = \widehat{d}_{\pi^k,h}^\dagger (s,a)$  is  $\mathcal{F}_k$ -measurable. Taking  $W = \ln (6 / \delta)$  in Lemma D.1 and applying a union bound, we know with probability  $1 - \delta /6$ , the following event happens:
$$
\forall k \in \mathbb {N}, (h, s, a) \in [ H ] \times \mathcal {S} \times \mathcal {A}, n _ {h} ^ {k} (s, a) \geq \frac {1}{2} \sum_ {i <   k} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a) - \ln \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right).
$$
To finish the proof, it remains to note that the event above implies the event we want by summing over  $h \in [H]$  for each  $k \in \mathbb{N}$  and each  $(s, a) \in S \times \mathcal{A}$ . For the  $\mathcal{E}^5$ , the proof is almost the same.
Lemma A.13 (Upper Bound on Counters). For  $\delta > 0$ , it holds that  $\mathbb{P}\left(\mathcal{E}^3\right) \geq 1 - \delta / 6$ .
$$
n _ {h} ^ {k} (s, a) = n _ {h} ^ {k} (s, a) + \mathbb {I} \left[ (s, a) = \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right].
$$
We fixed an  $(s,a,h)\in [H]\times \mathcal{S}\times \mathcal{A}$  and denote  $\mathcal{F}_k$  as the sigma field generated by the first  $k - 1$  episodes when running RF-UCB and  $X_{k} = \mathbb{I}\left[(s_{h}^{k},a_{h}^{k}) = (s,a)\right]$ . Then, we know  $X_{k}$  is  $\mathcal{F}_{k + 1}$ -measurable and  $\mathbb{E}\left[X_k\mid \mathcal{F}_k\right] = \widehat{d}_{\pi^k,h}^\dagger (s,a)$  is  $\mathcal{F}_k$ -measurable. Taking  $W = \ln (6 / \delta)$  in Lemma D.2 and applying a union bound, we know with probability  $1 - \delta /6$ , the following event happens:
$$
\forall k \in \mathbb {N}, (h, s, a) \in [ H ] \times \mathcal {S} \times \mathcal {A}, \quad n _ {h} ^ {k} (s, a) \leq 2 \sum_ {i <   k} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a) + \ln \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right).
$$
To finish the proof, it remains to note that the event above implies the event we want by summing over  $h\in [H]$  for each  $k\in \mathbb{N}$  and each  $(s,a)\in \mathcal{S}\times \mathcal{A}$ .
A.3.2 Proof for lemma A.14 (property of intermediate uncertainty function)
Lemma A.14 (Intermediate Uncertainty Function). We define for any  $(h,s,a)\in [H]\times S\times \mathcal{A}$  and any deterministic policy  $\pi$ ,
$$
\begin{array}{l} W _ {H + 1} ^ {\pi} (s, a, r) := 0, \\ W _ {h} ^ {\pi} (s, a, r) := \min  \left\{H - h + 1, \sqrt {\frac {8}{H ^ {2}} \overline {{\phi}} \left(m (s , a)\right) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} \left(s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi\right)\right)} \right. \\ \left. + 9 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) \left(\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}\right) W _ {h + 1} ^ {\pi} (s, a, r) \right\}, \\ \end{array}
$$
In addition, we define  $W_h^\pi(s^\dagger, a, r) = 0$  for any  $h \in [H]$ ,  $a \in \mathcal{A}$  and any reward function  $r$ . Then, under  $\mathcal{E}$ , for any  $(h, s, a) \in [H] \times \mathcal{S} \times \mathcal{A}$ , any deterministic policy  $\pi$ , and any deterministic reward function  $r$  (with its augmentation  $r^\dagger$ ), it holds that
$$
\left| Q _ {h} \left(s, a; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - Q _ {h} \left(s, a; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \leq W _ {h} ^ {\pi} (s, a, r).
$$
Proof. We prove by induction. For  $h = H + 1$ , we know  $W_{H + 1}(s, a, r) = 0$  by definition and the left hand side of the inequality we want also vanishes. Suppose the claim holds for  $h + 1$  and for any  $s, a$ . The Bellman equation gives us
$$
\begin{array}{l} \Delta_ {h} := Q _ {h} \left(s, a; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - Q _ {h} \left(s, a; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \leq \underbrace {\sum_ {s ^ {\prime}} \left(\widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s , a\right) - \mathbb {P} ^ {\dagger} \left(s ^ {\prime} \mid s , a\right)\right) V _ {h + 1} \left(s ^ {\prime} ; \mathbb {P} ^ {\dagger} , r ^ {\dagger} , \pi\right)} _ {I} \\ + \underbrace {\sum_ {s ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s , a\right) \left| V _ {h + 1} \left(s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi\right) - V _ {h + 1} \left(s ^ {\prime} ; \mathbb {P} ^ {\dagger} , r ^ {\dagger} , \pi\right) \right|} _ {I I} \\ \end{array}
$$
Under  $\mathcal{E}$ , we know that  $\mathrm{KL}\left(\widetilde{\mathbb{P}}^\dagger; \mathbb{P}^\dagger\right) \leq \frac{1}{H} \phi(m(s, a))$  for any  $(s, a)$ , so from Lemma D.5 we have
$$
\begin{array}{l} | I | \leq \sqrt {\frac {2}{H} \phi (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \mathbb {P} ^ {\dagger} (s , a)} \left(V _ {h + 1} (s ^ {\prime} ; \mathbb {P} ^ {\dagger} , r ^ {\dagger} , \pi)\right)} + \frac {2}{3} (H - h) \frac {\phi (m (s , a))}{H} \\ \leq \sqrt {\frac {2}{H} \phi (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \mathbb {P} ^ {\dagger} (s , a)} \left(V _ {h + 1} \left(s ^ {\prime} ; \mathbb {P} ^ {\dagger} , r ^ {\dagger} , \pi\right)\right)} + \frac {2}{3} \phi (m (s, a)), \\ \end{array}
$$
where  $\phi$  is the bonus defined as (A.4). Here, we let  $f$  in Lemma D.5 be  $V_{h + 1}\left(s';\mathbb{P}^{\dagger},r^{\dagger},\pi\right)$ . So the range will be  $H - h$  and the upper bound for KL divergence is given by high-probability event  $\mathcal{E}$ . We further apply Lemma D.6 to get
$$
\operatorname {V a r} _ {s ^ {\prime} \sim \mathbb {P} ^ {\dagger} (s, a)} \left(V _ {h + 1} \left(s ^ {\prime}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right)\right) \leq 2 \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s, a)} \left(V _ {h + 1} \left(s ^ {\prime}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right)\right) + 4 H \phi \left(m (s, a)\right)
$$
$$
\begin{array}{l} \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s, a)} \left(V _ {h + 1} \left(s ^ {\prime}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right)\right) \leq 2 \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s, a)} \left(V _ {h + 1} \left(s ^ {\prime}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right)\right) \\ + 2 H \sum_ {s ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \left| V _ {h + 1} \left(s ^ {\prime}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {h + 1} \left(s ^ {\prime}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right|. \\ \end{array}
$$
$$
\begin{array}{l} \left| I \right| \leq \frac {2}{3} \phi (m (s, a)) + \left[ \frac {8}{H} \phi (m (s, a)) \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s, a)} \left(V _ {h + 1} \left(s ^ {\prime}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right)\right) + 8 \phi (m (s, a)) ^ {2} \right. \\ \left. + 8 \phi \left(m (s, a)\right) \sum_ {s ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \left| V _ {h + 1} \left(s ^ {\prime}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {h + 1} \left(s ^ {\prime}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \right] ^ {1 / 2}. \\ \end{array}
$$
Using the fact that  $\sqrt{x + y} \leq \sqrt{x} + \sqrt{y}$  and  $\sqrt{xy} \leq \frac{x + y}{2}$  for positive  $x, y$ , and the definition of  $II$ , we obtain
$$
| I | \leq \sqrt {\frac {8}{H} \phi (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} \left(s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi\right)\right)} + 6 H \phi (m (s, a)) + \frac {1}{H} | I I |,
$$
$$
| \Delta_ {h} | \leq \sqrt {\frac {8}{H} \phi (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} (s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi)\right)} + 6 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) | I I |.
$$
If  $H\phi(m(s, a)) \leq 1$ , then we have by definition
$$
\begin{array}{l} \left| \Delta_ {h} \right| \leq \sqrt {\frac {8}{H ^ {2}} \cdot H \phi (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} (s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi)\right)} + 6 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) | I I | \\ \leq \sqrt {\frac {8}{H ^ {2}} \overline {{\phi}} (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} \left(s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi\right)\right)} + 6 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) | I I | \\ \end{array}
$$
by definition. Otherwise, if  $H\phi(m(s,a)) \geq 1$ , we have
$$
\sqrt {\frac {8}{H} \phi (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} (s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi)\right)} \leq \sqrt {8 H \phi (m (s , a))} \leq 3 H \phi (m (s, a)).
$$
Therefore, we have in either case
$$
| \Delta_ {h} | \leq \sqrt {\frac {8}{H ^ {2}} \overline {{\phi}} (m (s , a)) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} (s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi)\right)} + 9 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) | I I |.
$$
The induction hypothesis gives us that
$$
\begin{array}{l} \left| I I \right| \leq \sum_ {s ^ {\prime}} \sum_ {a ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \pi_ {h + 1} \left(a ^ {\prime} \mid s ^ {\prime}\right) \left| Q _ {h + 1} \left(s ^ {\prime}, a ^ {\prime}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - Q _ {h + 1} \left(s ^ {\prime}, a ^ {\prime}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \\ \leq \left(\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}\right) W _ {h + 1} ^ {\pi} (s, a, r). \\ \end{array}
$$
Inserting this into the upper bound of  $\Delta_h$ , we prove the case of  $h$  by the definition of  $W_h^\pi (s,a,r)$  and the simple fact that  $|\Delta_h|\leq H - h + 1$ .
A.3.3 Proof for lemma A.15 and lemma A.16 (properties of uncertainty function)
In this section, we prove some lemmas for upper bounding the uncertainty functions  $X_{h}(s,a)$ . We first provide a basic upper bound for  $X_{h}(s,a)$ . The uncertainty function is defined in definition A.6.
Lemma A.15. Under the high probability event  $\mathcal{E}$  (defined in appendix A.1.2) for all  $(h,s,a)\in [H]\times \mathcal{S}\times \mathcal{A}$ , it holds that
$$
X _ {h} (s, a) \leq 1 1 H \min  \left\{1, \phi \left(m (s, a)\right) \right\} + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right).
$$
In addition, for  $s = s^{\dagger}$ , from definition, we know the above upper bound naturally holds.
Proof. For any fixed  $(h,s,a)\in [H]\times S\times \mathcal{A}$ , from the definition of the uncertainty function, we know
$$
X _ {h} (s, a) \leq 9 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) \widetilde {\mathbb {P}} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right).
$$
To prove the lemma, it suffices to bound the difference between  $\widetilde{\mathbb{P}}^{\dagger}(\cdot \mid s,a)^{\top}(\max_{a'}X_{h + 1}(\cdot ,a'))$  and  $\mathbb{P}^{\dagger}(\cdot \mid s,a)^{\top}(\max_{a'}X_{h + 1}(\cdot ,a'))$ . Under  $\mathcal{E}$  (which happens with probability at least  $1 - \delta$ ), it holds that  $\mathrm{KL}\left(\widetilde{\mathbb{P}}_h^\dagger (\cdot \mid s,a);\mathbb{P}^\dagger (\cdot \mid s,a)\right)\leq \frac{1}{H}\phi (m(s,a))$ . Applying Lemma D.5 and a simple bound for variance gives us
$$
\begin{array}{l} \left| \widetilde {\mathbb {P}} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right) - \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right) \right| \\ \leq \sqrt {\frac {2}{H} \operatorname {V a r} _ {s ^ {\prime} \sim \mathbb {P} ^ {\dagger} (s , a)} \left(\max  _ {a ^ {\prime}} X _ {h + 1} \left(s ^ {\prime} , a ^ {\prime}\right)\right) \phi \left(m (s , a)\right)} + \frac {2}{3} \phi \left(m (s, a)\right) \\ \leq \sqrt {\left[ \frac {2}{H} \mathbb {P} ^ {\dagger} (\cdot \mid s , a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime}) ^ {2}\right) \right] \phi (m (s , a))} + \frac {2}{3} \phi (m (s, a)) \\ \leq \sqrt {\left[ \frac {2}{H} \mathbb {P} ^ {\dagger} (\cdot \mid s , a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right) \right] \cdot H \phi (m (s , a)) + \frac {2}{3} \phi (m (s , a))} \\ \leq \frac {1}{H} \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right) + 2 H \phi (m (s, a)). \\ \end{array}
$$
The last line comes from  $\sqrt{ab} \leq \frac{a + b}{2}$  for any positive  $a, b$ , and the fact that  $\frac{1}{2} H\phi(m(s, a)) + \frac{2}{3}\phi(m(s, a)) \leq 2H\phi(m(s, a))$ . Insert this bound into the definition of  $X_h(s, a)$  to obtain
$$
X _ {h} (s, a) \leq 1 1 H \phi (m (s, a)) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right).
$$
Noticing that  $X_{h}(s,a) \leq H - h + 1 \leq 11H$ , we conclude.