SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B.1 Comparison with other comparator policy 33
B.2 Comparison with offline reinforcement learning 33
B.3 Proof of corollary 5.2 35
B.4 Proof for lemma B.1 and lemma B.2 38
C Lower bound 39
D Technical lemmas and proofs 40
E Details of the planning phase 42
F More related works 42
A Proof of the main result
In this section, we define some crucial concepts that will be used in the proof of the main result.
A.1.1 Sparsified MDP
First, we restate the definition 4.1 in the main text.
Definition A.1 (Sparsified MDP). Let  $s^{\dagger}$  be an absorbing state, i.e., such that  $\mathbb{P}\left(s^{\dagger} \mid s^{\dagger}, a\right) = 1$  and  $r(s^{\dagger}, a) = 0$  for all  $a \in \mathcal{A}$ . The state space in the sparsified MDP  $\mathcal{M}^{\dagger}$  is defined as that of the original MDP with the addition of  $s^{\dagger}$ . The dynamics  $\mathbb{P}^{\dagger}$  of the sparsified MDP are defined as
$$
\mathbb {P} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \mathbb {P} \left(s ^ {\prime} \mid s, a\right) & i f N (s, a, s ^ {\prime}) \geq \Phi \\ 0 & i f N (s, a, s ^ {\prime}) <   \Phi , \end{array} \right. \quad \mathbb {P} ^ {\dagger} \left(s ^ {\dagger} \mid s, a\right) = \sum_ { \begin{array}{c} s ^ {\prime} \neq s ^ {\dagger} \\ N (s, a, s ^ {\prime}) <   \Phi \end{array} } \mathbb {P} \left(s ^ {\prime} \mid s, a\right). \tag {A.1}
$$
In the offline phase of our algorithm, we simulate the virtual episodes from the empirical version of sparsified MDP (See Algorithm 2). Now we formally this MDP.
Definition A.2 (Empirical sparsified MDP). Let  $s^{\dagger}$  be the absorbing state defined in the sparsified MDP. The state space in the empirical sparsified MDP  $\widehat{\mathcal{M}}^{\dagger}$  is defined as that of the original MDP with the addition of  $s^{\dagger}$ . The dynamics  $\widehat{\mathbb{P}}^{\dagger}$  of the sparsified MDP are defined as
$$
\widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \frac {N \left(s , a , s ^ {\prime}\right)}{N \left(s , a\right)} & i f N (s, a, s ^ {\prime}) \geq \Phi \\ 0 & i f N (s, a, s ^ {\prime}) <   \Phi , \end{array} \right. \quad \widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\dagger} \mid s, a\right) = \sum_ {\substack {s ^ {\prime} \neq s ^ {\dagger} \\ N \left(s, a, s ^ {\prime}\right) <   \Phi}} \frac {N \left(s , a , s ^ {\prime}\right)}{N (s , a)}. \tag{A.2}
$$
For any deterministic reward function  $r: \mathcal{S} \times \mathcal{A} \to [0,1]$ , the reward function on the empirical sparsified MDP is defined as  $r^{\dagger}(s,a) = r(s,a)$ ; for simplicity we only consider deterministic reward functions. Here, the counters  $N(s,a)$  and  $N(s,a,s')$  are the number of  $(s,a)$  and  $(s,a,s')$  in the offline data, respectively.
Finally, in the planning phase, after we interact with the true environment for many online episodes, construct a fine-estimated sparsified MDP, which is used to extract the optimal policy of given reward functions. We formally define it below.
Definition A.3 (Fine-estimated sparsified MDP). Let  $s^{\dagger}$  be the absorbing state defined in the sparsified MDP. The state space in the fine-estimated sparsified MDP  $\widetilde{\mathcal{M}}^{\dagger}$  is defined as that of the original MDP with the addition of  $s^{\dagger}$ . The dynamics  $\widetilde{\mathbb{P}}^{\dagger}$  of the sparsified MDP are defined as
$$
\widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \frac {m \left(s , a , s ^ {\prime}\right)}{m (s , a)} & i f N (s, a, s ^ {\prime}) \geq \Phi \\ 0 & i f N (s, a, s ^ {\prime}) <   \Phi , \end{array} \right. \quad \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\dagger} \mid s, a\right) = \sum_ { \begin{array}{c} s ^ {\prime} \neq s ^ {\dagger} \\ N \left(s, a, s ^ {\prime}\right) <   \Phi \end{array} } \frac {m \left(s , a , s ^ {\prime}\right)}{m (s , a)}. \tag {A.3}
$$
For any deterministic reward function  $r: \mathcal{S} \times \mathcal{A} \to [0,1]$ , the reward function on the fine-estimated sparsified MDP is defined as  $r^{\dagger}(s,a) = r(s,a)$ ; for simplicity we only consider deterministic reward functions. Here, the counters  $N(s,a)$  and  $N(s,a,s')$  are the number of  $(s,a)$  and  $(s,a,s')$  in the offline data, respectively, while  $m(s,a)$  and  $m(s,a,s')$  are the counters of  $(s,a)$  and  $(s,a,s')$  in online episodes.
A.1.2 High probability events
In this section, we define all high-probability events we need in order to make our theorem to hold. Specifically, we define
$$
\begin{array}{l} \mathcal {E} ^ {P}: = \left\{\forall (s, a, s ^ {\prime}) \text {s . t .} N (s, a, s ^ {\prime}) \geq \Phi , \left| \widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) - \mathbb {P} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \right| \right. \\ \leq \sqrt {\frac {2 \widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s , a\right)}{N (s , a)} \log \left(\frac {1 2 \left| \mathcal {S} \right| ^ {2} \left| \mathcal {A} \right|}{\delta}\right)} + \frac {1 4}{3 N (s , a)} \log \left(\frac {1 2 \left| \mathcal {S} \right| ^ {2} \left| \mathcal {A} \right|}{\delta}\right) \Bigg \}; \\ \end{array}
$$
$$
\mathcal {E} ^ {2} := \left\{\forall k \in \mathbb {N}, (s, a) \in \mathcal {S} \times \mathcal {A}, n ^ {k} (s, a) \geq \frac {1}{2} \sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a) - H \ln \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) \right\};
$$
$$
\mathcal {E} ^ {3} := \left\{\forall k \in \mathbb {N}, (s, a) \in \mathcal {S} \times \mathcal {A}, n ^ {k} (s, a) \leq 2 \sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a) + H \ln \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) \right\};
$$
$$
\begin{array}{l} \mathcal {E} ^ {4} := \left\{\forall (s, a) \in \mathcal {S} \times \mathcal {A}, \operatorname {K L} \left(\widetilde {\mathbb {P}} ^ {\dagger} (s, a); \mathbb {P} ^ {\dagger} (s, a)\right) \right. \\ \leq \frac {1}{m (s , a)} \left[ \log \left(\frac {6 | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \log \left(e \left(1 + \frac {m (s , a)}{| \mathcal {S} |}\right)\right) \right] \}; \\ \end{array}
$$
$$
\mathcal {E} ^ {5} := \left\{\forall (s, a) \in [ H ] \times \mathcal {S} \times \mathcal {A}, m (s, a) \geq \frac {1}{2} K _ {d e} \sum_ {h \in [ H ]} w _ {h} ^ {m i x} (s, a) - H \ln \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) \right\},
$$
$$
w _ {h} ^ {m i x} (s, a) := \frac {1}{K _ {u c b}} \sum_ {k = 1} ^ {K _ {u c b}} d _ {\pi^ {k}, h} ^ {\dagger} (s, a).
$$
Here,  $\mathrm{KL}(\cdot ;\cdot)$  denotes the Kullback-Leibler divergence between two distributions.  $K_{ucb}$  is the number of episodes of the sub-routine RF-UCB and  $\pi^i$  is the policy executed at the i-th episode of RF-UCB (See algorithm 2).  $K_{de}$  is the number of episodes executed in the online phase.  $n^{k}(s,a)$  is the counter of  $(s,a)$  before the beginning of the  $k$ -th episode in the offline phase and  $m(s,a)$  is the number of  $(s,a)$  samples in the online data (See definitions in section A.1). We denote  $d_{\pi ,h}(s,a)$ ,  $d_{\pi ,h}^{\dagger}(s,a)$  and  $\widehat{d}_{\pi ,h}^{\dagger}(s,a)$  as the occupancy measure of  $(s,a)$  at stage  $h$  under policy  $\pi$ , on  $\mathbb{P}$  (the true transition dynamics),  $\mathbb{P}^{\dagger}$  (the transition dynamics in the sparsified MDP) and  $\widehat{\mathbb{P}}^{\dagger}$  (the transition dynamics in the empirical sparsified MDP) respectively.
For the event defined above, we have the following guarantee, which is proved in appendix A.3.1.
Lemma A.4. For  $\delta > 0$ , we have
$$
\mathcal {E} := \left\{\mathcal {E} ^ {P} \cup \mathcal {E} ^ {2} \cup \mathcal {E} ^ {3} \cup \mathcal {E} ^ {4} \cup \mathcal {E} ^ {5} \right\}
$$
happens with probability at least  $1 - \delta$
A.1.3 Uncertainty function and empirical uncertainty function
In this section, we restate the definition of uncertainty function and empirical uncertainty functions, as well as some intermediate uncertainty functions which will be often used in the proof. First, we restate the definition of bonus function.
Definition A.5 (Bonus Function). We define
$$
\phi (x) = \frac {H}{x} \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \log \left(e \left(1 + \frac {x}{| \mathcal {S} |}\right)\right) \right]. \tag {A.4}
$$
$$
\bar {\phi} (x) = \min  \left\{1, H \phi (x) \right\}. \tag {A.5}
$$
Next, we restate the definition of uncertainty function and empirical uncertainty function. Notice that the uncertainty function does not depend on reward function or specific policy  $\pi$ .
Definition A.6 (Uncertainty function). We define for any  $(h,s,a)\in [H]\times \mathcal{S}\times \mathcal{A}$  and any deterministic policy  $\pi$ ,
$$
X _ {H + 1} (s, a) := 0,
$$
$$
X _ {h} (s, a) := \min  \left\{H - h + 1, 9 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) \widetilde {\mathbb {P}} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right) \right\},
$$
where we specify
$$
\widetilde {\mathbb {P}} ^ {\dagger} (\cdot \mid s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \{X _ {h + 1} (\cdot , a ^ {\prime}) \}\right) := \sum_ {s ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} (s ^ {\prime} \mid s, a) \left(\max  _ {a ^ {\prime}} \{X _ {h + 1} (s ^ {\prime}, a ^ {\prime}) \}\right).
$$
In addition, we define  $X_{h}(s^{\dagger},a) = 0$  for any  $h\in [H]$ ,  $a\in \mathcal{A}$ . Here, the notation  $\sum_{s'}$  above means summation over  $s'\in S\cup \{s^\dagger\}$ . But since  $X_{h}(s^{\dagger},a) = 0$  for any  $h\in [H]$ ,  $a\in \mathcal{A}$ , this is equivalent to the summation over  $s'\in S$ .  $\widetilde{\mathbb{P}}^{\dagger}$  is the transition probability of fine-estimated sparsified MDP defined in section A.1.1.
Then, for the reader to better understand the proof, we define some intermediate quantity, which also measure the uncertainty of value estimation, but they may be reward or policy dependent.
Definition A.7 (Intermediate uncertainty function). We define for any  $(h,s,a)\in [H]\times S\times \mathcal{A}$  and any deterministic policy  $\pi$ ,
$$
W _ {H + 1} ^ {\pi} (s, a, r) := 0,
$$
$$
\begin{array}{l} W _ {h} ^ {\pi} (s, a, r) := \min  \left\{H - h + 1, \sqrt {\frac {8}{H ^ {2}} \bar {\phi} \left(m (s , a)\right) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} \left(s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi\right)\right)} \right. \\ \left. + 9 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) \left(\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}\right) W _ {h + 1} ^ {\pi} (s, a, r) \right\}, \\ \end{array}
$$
$$
\left(\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}\right) W _ {h + 1} ^ {\pi} (s, a, r) := \sum_ {s ^ {\prime}} \sum_ {a ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \pi_ {h + 1} \left(a ^ {\prime} \mid s ^ {\prime}\right) W _ {h + 1} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}, r\right).
$$
In addition, we define  $W_h^\pi(s^\dagger, a, r) = 0$  for any  $h \in [H]$ ,  $a \in \mathcal{A}$  and any reward function  $r$ . Here,  $\widetilde{\mathbb{P}}^\dagger$  is the transition probability of fine-estimated sparsified MDP defined in section A.1.1.
The intermediate function  $W$  is reward and policy dependent and can be used to upper bound the value estimation error. Further, we need to define some policy-dependent version of the uncertainty function, denoted as  $X_{h}^{\pi}(s,a)$ , as well as another quantity  $Y_{h}^{\pi}(s,a,r)$ .
Definition A.8. We define  $X_{H + 1}^{\pi}(s,a) = Y_{H + 1}^{\pi}(s,a,r) = 0$  for any  $\pi, r, s, a$  and
$$
X _ {h} ^ {\pi} (s, a) := \min  \left\{H - h + 1, 9 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) \left(\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}\right) X _ {h + 1} ^ {\pi} (s, a) \right\};
$$
$$
Y _ {h} ^ {\pi} (s, a, r) := \sqrt {\frac {8}{H ^ {2}} \overline {{\phi}} (m (s , a)) \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} (s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi)\right)} + \left(1 + \frac {1}{H}\right) (\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}) Y _ {h + 1} ^ {\pi} (s, a, r)
$$
$$
\left(\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}\right) X _ {h + 1} ^ {\pi} (s, a) = \sum_ {s ^ {\prime}} \sum_ {a ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \pi_ {h + 1} \left(a ^ {\prime} \mid s ^ {\prime}\right) X _ {h + 1} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right);
$$
$$
\left(\widetilde {\mathbb {P}} ^ {\dagger} \pi_ {h + 1}\right) Y _ {h + 1} ^ {\pi} (s, a, r) = \sum_ {s ^ {\prime}} \sum_ {a ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \pi_ {h + 1} \left(a ^ {\prime} \mid s ^ {\prime}\right) Y _ {h + 1} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}, r\right).
$$
In addition, we define  $X_h^\pi(s^\dagger, a) = Y_h^\pi(s^\dagger, a, r) = 0$  for any  $a \in \mathcal{A}$ ,  $h \in [H]$  and any reward  $r$ , any policy  $\pi$ . Here,  $\widetilde{\mathbb{P}}^\dagger$  is the transition probability of fine-estimated sparsified MDP defined in section A.1.1.
Finally, we define the empirical uncertainty functions.
Definition A.9. We define  $U_{H + 1}^{k}(s,a) = 0$  for any  $k\in [K_{ucb}]$  and  $s\in S, a\in \mathcal{A}$ . Further, we define
$$
U _ {h} ^ {k} (s, a) = H \min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} + \widehat {\mathbb {P}} ^ {\dagger} (s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} U _ {h + 1} ^ {k} (\cdot , a ^ {\prime})\right), \tag {A.6}
$$
where  $\phi$  is the bonus function defined in eq. (A.4) and  $n^k(s, a)$  is the counter of the times we encounter  $(s, a)$  until the beginning of the  $k$ -the episode when running the sub-routine RF-UCB in the offline phase.
A.2 Proof of the key theorems and lemmas
A.2.1 Uncertainty Functions upper bounds the estimation error
Proof. This proof follows closely from the techniques in [Ménard et al., 2021], but here we consider the homogeneous MDP. We let  $\widetilde{d}_{\pi,h}^{\dagger}(s,a)$  be the probability of encountering  $(s,a)$  at stage  $h$  when running policy  $\pi$  on  $\widetilde{\mathbb{P}}^\dagger$ , starting from  $s_1$ . Now we assume  $\mathcal{E}$  happens and fix a reward function  $r$  and policy  $\pi$ . From lemma A.14, we know
$$
\begin{array}{l} \left| V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \leq \left| Q _ {h} \left(s _ {1}, \pi_ {1} (a _ {1}); \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - Q _ {h} \left(s _ {1}, \pi_ {1} (s _ {1}); \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \\ \leq W _ {h} ^ {\pi} (s _ {1}, \pi (s _ {1}), r), \\ \end{array}
$$
where  $W_h^\pi(s, a, r)$  is the intermediate uncertainty function defined in definition A.7. Here, we use the policy-dependent version of uncertainty function  $W^\pi(s, a, r)$ , which will then upper bounded using another two policy-dependent quantities  $X_h^\pi(s, a)$  and  $Y_h^\pi(s, a, r)$ . We define these policy-dependent uncertainty functions to upper bound the estimation error of specific policy, but since we are considering a reward-free setting, which entails a low estimation error for any policy and reward, these quantities cannot be directly used in the algorithm to update the policy. Next, we claim
$$
W _ {h} ^ {\pi} (s, a, r) \leq X _ {h} ^ {\pi} (s, a) + Y _ {h} ^ {\pi} (s, a, r),
$$
where  $X_h^\pi(s, a)$  and  $Y_h^\pi(s, a, r)$  are defined in definition A.8. Actually, this is easy from the definition of  $X_h^\pi(s, a)$ ,  $Y_h^\pi(s, a, r)$  and  $W_h^\pi(s, a, r)$ . For  $h = H + 1$ , this is trivial. Assume this is true for  $h + 1$ , then the case of  $h$  is given by the definition and the fact that  $\min\{x, y + z\} \leq \min\{x, y\} + z$  for any  $x, y, z \geq 0$ . Therefore, we have
$$
\left| V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \leq X _ {1} ^ {\pi} \left(s _ {1}, \pi_ {1} \left(s _ {1}\right)\right) + Y _ {1} ^ {\pi} \left(s _ {1}, \pi_ {1} \left(s _ {1}\right), r\right). \tag {A.7}
$$
Next, we eliminate the dependency to policy  $\pi$  and obtain an upper bound of estimation error using the policy-independent uncertainty function  $X_{h}(s,a)$ . This is done by bounding  $Y_{1}^{\pi}(s_{1},\pi_{1}(s_{1}),r)$  by  $X_{h}^{\pi}(s,a)$  and then upper bounding  $X_{h}^{\pi}(s,a)$  by  $X_{h}(s,a)$ . From definition A.8, the Cauchy-Schwarz inequality and the fact that  $r\in [0,1]$ , we have for any reward function and any deterministic policy  $\pi$ ,
$$
Y _ {1} ^ {\pi} \left(s _ {1}, \pi_ {1} (s _ {1}), r\right)
$$
$$
\leq \sum_ {s, a} \sum_ {h = 1} ^ {H} \widetilde {d} _ {\pi , h} ^ {\dagger} (s, a) \left(1 + \frac {1}{H}\right) ^ {h - 1} \sqrt {\frac {8}{H ^ {2}} \bar {\phi} \left(m (s , a)\right) \cdot \operatorname {V a r} _ {s ^ {\prime} \sim \widetilde {\mathbb {P}} ^ {\dagger} (s , a)} \left(V _ {h + 1} \left(s ^ {\prime} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi\right)\right)}
$$
(Induction by successively using the definition of  $Y$ )
$$
\begin{array}{l} \leq \frac{e}{H}\sqrt{8\sum_{s,a}\sum_{h = 1}^{H}\widetilde{d}_{\pi,h}^{\uparrow}(s,a)\operatorname{Var}_{s^{\prime}\sim\widetilde{\mathbb{P}}^{\dagger}(s,a)}\left(V_{h + 1}\left(s^{\prime};\widetilde{\mathbb{P}}^{\dagger},r^{\dagger},\pi\right)\right)}\cdot \sqrt{\sum_{s,a}\sum_{h = 1}^{H}\widetilde{d}_{\pi,h}^{\uparrow}(s,a)\overline{\phi}\left(m(s,a)\right)}\\ (\text{Cauchy-Schwarz Inequality}) \\ \leq \frac {e}{H} \sqrt {8 H ^ {2} \sum_ {s , a} \sum_ {h = 1} ^ {H} \widetilde {d} _ {\pi , h} ^ {\uparrow} (s , a)} \cdot \sqrt {\sum_ {s , a} \sum_ {h = 1} ^ {H} \widetilde {d} _ {\pi , h} ^ {\uparrow} (s , a) \overline {{\phi}} (m (s , a))} \\ \leq e \sqrt {8 \sum_ {s , a} \sum_ {h = 1} ^ {H} \tilde {d} _ {\pi , h} ^ {\uparrow} (s , a) \overline {{\phi}} (m (s , a))}. \\ \end{array}
$$
We notice that the right hand side of the last inequality above is the policy value of some specific reward function when running  $\pi$  on  $\widetilde{\mathbb{P}}$ . Concretely, if the transition probability is  $\widetilde{\mathbb{P}}$  and the reward function at  $(s,a)$  is  $\overline{\phi}(m(s,a))$ , then the state value function at the initial state  $s_1$  is  $\sum_{s,a} \sum_{h=1}^{H} \widetilde{d}_{\pi,h}^{\dagger}(s,a) \overline{\phi}(m(s,a))$ . This specific reward function is non-negative and uniformly bounded by one, so it holds that  $\sum_{s,a} \sum_{i=h}^{H} \widetilde{d}_{\pi,i}^{\dagger}(s,a) \overline{\phi}(m(s,a)) \leq H - h + 1$ . Moreover, from the definition of  $\phi$  and  $\overline{\phi}$  (eq. (A.4)), we know  $\overline{\phi}(m(s,a)) \leq H \phi(m(s,a))$ . Then, from the definition of  $X_h(s,a)$  in definition A.6, we apply an inductive argument to obtain
$$
\sum_ {s, a} \sum_ {h = 1} ^ {H} \tilde {d} _ {\pi , h} ^ {\uparrow} (s, a) \bar {\phi} (m (s, a)) \leq X _ {1} ^ {\pi} (s _ {1}, \pi (s _ {1}))
$$
for any deterministic policy  $\pi$ . So we have
$$
Y _ {1} ^ {\pi} \left(s _ {1}, \pi_ {1} \left(s _ {1}\right), r\right) \leq 2 \sqrt {2 e} \sqrt {X _ {1} ^ {\pi} \left(s _ {1} , \pi \left(s _ {1}\right)\right)}.
$$
Therefore, combining the last inequality with (A.7), we have
$$
\left| V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \leq X _ {1} ^ {\pi} \left(s _ {1}, \pi_ {1} \left(s _ {1}\right)\right) + 2 \sqrt {2} e \sqrt {X _ {1} ^ {\pi} \left(s _ {1} , \pi_ {1} \left(s _ {1}\right)\right)}.
$$
From the definition of  $X_{h}(s,a)$  (definition A.6) and  $X_{h}^{\pi}(s,a)$  (definition A.8), we can see
$$
X _ {h} ^ {\pi} (s, a) \leq X _ {h} (s, a)
$$
for any  $(h,s,a)$  and any deterministic policy  $\pi$ . Therefore, we conclude that
$$
\begin{array}{l} \left| V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \leq X _ {1} \left(s _ {1}, \pi_ {1} \left(s _ {1}\right)\right) + 2 \sqrt {2} e \sqrt {X _ {1} \left(s _ {1} , \pi_ {1} \left(s _ {1}\right)\right)} \\ \leq \max  _ {a} X _ {1} (s _ {1}, a) + 2 \sqrt {2} e \sqrt {\max  _ {a} X _ {1} (s _ {1} , a)}. \\ \end{array}
$$
A.2.2 Proof of lemma 6.2
Proof. It suffices to prove that for any  $k \in [K]$ ,  $h \in [H]$ ,  $s \in S$ ,  $a \in \mathcal{A}$ , it holds that
$$
X _ {h} (s, a) \leq C \left(1 + \frac {1}{H}\right) ^ {3 (H - h)} U _ {h} ^ {k} (s, a)
$$
since the theorem comes from an average of the inequalities above and the fact that  $(1 + 1 / H)^H \leq e$ . For any fixed  $k$ , we prove it by induction on  $h$ . When  $h = H + 1$ , both sides are zero by definition. Suppose the claim holds for  $h + 1$ ; we will prove that it also holds for  $h$ . We denote  $K_{ucb}$  as the
number of virtual episodes in the offline phase. From lemma A.16, the decreasing property of  $\phi (\cdot)$  (lemma A.17) and lemma A.21, we have
$$
\begin{array}{l} X _ {h} (s, a) \leq C H \phi \left(n ^ {K _ {u c b}} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) (Lemma A.16) \\ \leq C H \phi \left(n ^ {k} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) \quad (Lemma A.17) \\ \leq C H \phi \left(n ^ {k} (s, a)\right) + \left(1 + \frac {1}{H}\right) ^ {3} \widehat {\mathbb {P}} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) \\ (\text {L e m m a A . 2 1 a n d} 1 + 2 / H \leq (1 + 1 / H) ^ {2}) \\ \end{array}
$$
The inductive hypothesis gives us for any  $s'$ ,
$$
\left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} \left(s ^ {\prime}, a ^ {\prime}\right) \right\}\right) \leq C \left(1 + \frac {1}{H}\right) ^ {3 (H - h - 1)} \left(\max  _ {a ^ {\prime}} U _ {h + 1} ^ {k} \left(s ^ {\prime}, a ^ {\prime}\right)\right),
$$
$$
\begin{array}{l} X _ {h} (s, a) \leq C H \phi \left(n ^ {k} (s, a)\right) + \left(1 + \frac {1}{H}\right) ^ {3 (H - h)} \widehat {\mathbb {P}} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{U _ {h + 1} ^ {k} (\cdot , a ^ {\prime}) \right\}\right) \\ \leq C \left(1 + \frac {1}{H}\right) ^ {3 (H - h)} \left[ H \phi \left(n ^ {k} (s, a)\right) + \widehat {\mathbb {P}} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{U _ {h + 1} ^ {k} (\cdot , a ^ {\prime}) \right\}\right) \right]. \tag {A.8} \\ \end{array}
$$
Again, by definition of  $X_{h}(s,a)$ , we have
$$
X _ {h} (s, a) \leq H - h + 1 \leq C \left(1 + \frac {1}{H}\right) ^ {3 (H - h)} H. \tag {A.9}
$$
Combining (A.8) and (A.9), as well as the definition of  $U_h^k$  (definition A.9), we prove the case of stage  $h$  and we conclude the theorem by induction.
A.2.3 Upper bounding the empirical uncertainty function (lemma 6.3)
Proof. First, we are going to prove
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} U _ {1} ^ {k} (s, a) \leq \frac {H ^ {2} | \mathcal {S} | | \mathcal {A} |}{K} \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \log \left(e \left(1 + \frac {K H}{| \mathcal {S} |}\right)\right) \right] \log \left(1 + \frac {H K}{| \mathcal {S} | | \mathcal {A} |}\right). \tag {A.10}
$$
From the definition (see algorithm 2), we know an important property of uncertainty function is that  $\pi_h^k$  is greedy with respect to  $U_h^k$ . So we have