SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \left(\widetilde {P} _ {h} - P _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) \right| \leq \left| \left(\widetilde {P} _ {h} - \widehat {P} _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) \right| + \left| \left(\widehat {P} _ {h} - P _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) \right| \\ \leq H \left\| \widetilde {P} _ {h} (\cdot | s _ {h}, a _ {h}) - \widehat {P} _ {h} (\cdot | s _ {h}, a _ {h}) \right\| _ {1} + \left| \left(\widehat {P} _ {h} - P _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) \right| \tag {21} \\ \leq \frac {5 S H E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}} + \left| (\widehat {P} _ {h} - P _ {h}) \cdot \widetilde {V} _ {h + 1} (s _ {h}, a _ {h}) \right|, \\ \end{array}
$$
where the third inequality is due to Lemma C.4.
Next, recall  $\widehat{\pi}_{h+1}$  in Algorithm 1 is computed backwardly therefore only depends on sample tuple from time  $h+1$  to  $H$ . As a result,  $\widetilde{V}_{h+1} = \langle \overline{Q}_{h+1}, \widehat{\pi}_{h+1} \rangle$  also only depends on the sample tuple from time  $h+1$  to  $H$  and some Gaussian noise that is independent of the offline dataset. On the other side, by the definition,  $\widehat{P}_h$  only depends on the sample tuples from time  $h$  to  $h+1$ . Therefore  $\widetilde{V}_{h+1}$  and  $\widehat{P}_h$  are Conditionally independent (This trick is also used in [Yin et al., 2021] and [Yin and Wang,
2021b]), by Empirical Bernstein's inequality (Lemma E.4) and a union bound, with probability  $1 - \delta$ , for all  $s_h, a_h$  such that  $\widetilde{n}_{s_h, a_h} \geq 3E_{\rho}$ ,
$$
\left| \left(\widehat {P} _ {h} - P _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) \right| \leq \sqrt {\frac {2 \operatorname {V a r} _ {\widehat {P} _ {h} \left(\cdot \mid s _ {h} , a _ {h}\right)} \left(\widetilde {V} _ {h + 1} (\cdot)\right) \cdot \iota}{n _ {s _ {h} , a _ {h}}}} + \frac {7 H \cdot \iota}{3 n _ {s _ {h} , a _ {h}}}. \tag {22}
$$
$$
\begin{array}{l} \left| \left(\widetilde {P} _ {h} - P _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) \right| \leq \sqrt {\frac {2 \operatorname {V a r} _ {\widehat {P} _ {h} \left(\cdot \mid s _ {h} , a _ {h}\right)} \left(\widetilde {V} _ {h + 1} (\cdot)\right) \cdot \iota}{n _ {s _ {h} , a _ {h}}}} + \frac {7 H \cdot \iota}{3 n _ {s _ {h} , a _ {h}}} + \frac {5 S H E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}} \\ \leq \sqrt {\frac {2 \mathrm {V a r} _ {\widehat {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{n _ {s _ {h} , a _ {h}}}} + \frac {9 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}} \\ \leq \frac {9 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}} + \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{n _ {s _ {h} , a _ {h}}}} + 4 \sqrt {2} H \sqrt {\frac {S E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} \cdot n _ {s _ {h} , a _ {h}}}} \tag {23} \\ \leq \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{n _ {s _ {h} , a _ {h}}}} + \frac {1 6 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}} \\ \leq \sqrt {\frac {2 \mathrm {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}}} + \frac {1 6 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}}. \\ \end{array}
$$
The second and forth inequality is because when  $\widetilde{n}_{s_h,a_h} \geq 3E_{\rho}$ ,  $n_{s_h,a_h} \geq \frac{2\widetilde{n}_{s_h,a_h}}{3}$ . Specifically, these two inequalities are also because usually we only care about the case when  $SE_{\rho} \geq 1$ , which is equivalent to  $\rho$  being not very large. The third inequality is due to Lemma C.5. The last inequality is due to Lemma C.3.
Note that the previous Lemmas rely on the condition that  $\widetilde{n}$  is not very small ( $\widetilde{n}_{s_h,a_h} \geq 3E_\rho$ ). Below we state the Multiplicative Chernoff bound (Lemma C.7 and Remark C.8) to show that under our condition in Theorem 3.4, for  $(s_h,a_h) \in \mathcal{C}_h$ ,  $\widetilde{n}_{s_h,a_h}$  will be larger than  $3E_{\rho}$  with high probability.
Lemma C.7 (Lemma B.1 in [Yin and Wang, 2021b]). For any  $0 < \delta < 1$ , there exists an absolute constant  $c_{1}$  such that when total episode  $n > c_{1} \cdot 1 / \bar{d}_{m} \cdot \log(HSA / \delta)$ , then with probability  $1 - \delta$ ,  $\forall h \in [H]$
$$
n _ {s _ {h}, a _ {h}} \geq n \cdot d _ {h} ^ {\mu} \left(s _ {h}, a _ {h}\right) / 2, \quad \forall \left(s _ {h}, a _ {h}\right) \in \mathcal {C} _ {h}.
$$
Furthermore, we denote
$$
\mathcal {E} := \left\{n _ {s _ {h}, a _ {h}} \geq n \cdot d _ {h} ^ {\mu} \left(s _ {h}, a _ {h}\right) / 2, \forall \left(s _ {h}, a _ {h}\right) \in \mathcal {C} _ {h}, h \in [ H ]. \right\} \tag {24}
$$
then equivalently  $P(\mathcal{E}) > 1 - \delta$
In addition, we denote
$$
\mathcal {E} ^ {\prime} := \left\{n _ {s _ {h}, a _ {h}} \leq \frac {3}{2} n \cdot d _ {h} ^ {\mu} \left(s _ {h}, a _ {h}\right), \forall \left(s _ {h}, a _ {h}\right) \in \mathcal {C} _ {h}, h \in [ H ]. \right\} \tag {25}
$$
then similarly  $P(\mathcal{E}') > 1 - \delta$ .
Remark C.8. According to Lemma C.7, for any failure probability  $\delta$ , there exists some constant  $c_{1} > 0$  such that when  $n \geq \frac{c_{1}E_{\rho}\cdot\iota}{d_{m}}$ , with probability  $1 - \delta$ , for all  $(s_h,a_h) \in \mathcal{C}_h$ ,  $n_{s_h,a_h} \geq 4E_{\rho}$ . Therefore, under the condition of Theorem 3.4 and the high probability events in Lemma C.3 and Lemma C.7, it holds that for all  $(s_h,a_h) \in \mathcal{C}_h$ ,  $\widetilde{n}_{s_h,a_h} \geq 3E_{\rho}$  while for all  $(s_h,a_h) \notin \mathcal{C}_h$ ,  $\widetilde{n}_{s_h,a_h} \leq E_{\rho}$ .
Lemma C.9. Define  $(\mathcal{T}_hV)(\cdot ,\cdot)\coloneqq r_h(\cdot ,\cdot) + (P_hV)(\cdot ,\cdot)$  for any  $V\in \mathbb{R}^S$  . Note  $\widehat{\pi}$ $\overline{Q}_h$ $\widetilde{V}_h$  are defined in Algorithm 1 and denote  $\xi_{h}(s,a) = (\mathcal{T}_{h}\widetilde{V}_{h + 1})(s,a) - \overline{Q}_{h}(s,a)$  . Then it holds that
$$
V _ {1} ^ {\pi^ {\star}} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\star}} \left[ \xi_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] - \sum_ {h = 1} ^ {H} \mathbb {E} _ {\widehat {\pi}} \left[ \xi_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right]. \tag {26}
$$
Furthermore, (26) holds for all  $V_h^{\pi^*}(s) - V_h^{\widehat{\pi}}(s)$ .
Proof of Lemma C.9. Lemma C.9 is a direct corollary of Lemma E.8 with  $\pi = \pi^{\star}$ ,  $\widehat{Q}_h = \overline{Q}_h$ ,  $\widehat{V}_h = \widetilde{V}_h$  and  $\widehat{\pi} = \widehat{\pi}$  in Algorithm 1, we can obtain this result since by the definition of  $\widehat{\pi}$  in Algorithm 1,  $\langle \overline{Q}_h(s_h,\cdot),\pi_h(\cdot |s_h) - \widehat{\pi}_h(\cdot |s_h)\rangle \leq 0$ . The proof for  $V_{h}^{\pi^{\star}}(s) - V_{h}^{\widehat{\pi}}(s)$  is identical.
Next we prove the asymmetric bound for  $\xi_h$ , which is the key to the proof.
Lemma C.10 (Private version of Lemma D.6 in [Yin and Wang, 2021b]). Denote  $\xi_h(s,a) = (\mathcal{T}_h\widetilde{V}_{h + 1})(s,a) - \overline{Q}_h(s,a)$ , where  $\widetilde{V}_{h + 1}$  and  $\overline{Q}_h$  are the quantities in Algorithm 1 and  $\mathcal{T}_h(V) := r_h + P_h\cdot V$  for any  $V\in \mathbb{R}^S$ . Then under the high probability events in Lemma C.3 and Lemma C.6, for any  $h,s_h,a_h$  such that  $\widetilde{n}_{s_h,a_h} > 3E_{\rho}$ , we have
$$
\begin{array}{l} 0 \leq \xi_ {h} (s _ {h}, a _ {h}) = (\mathcal {T} _ {h} \widetilde {V} _ {h + 1}) (s _ {h}, a _ {h}) - \overline {{Q}} _ {h} (s _ {h}, a _ {h}) \\ \leq 2 \sqrt {\frac {2 \mathrm {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}}} + \frac {3 2 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}}, \\ \end{array}
$$
where  $\iota = \log (HSA / \delta)$
Proof of Lemma C.10. The first inequality: We first prove  $\xi_h(s_h, a_h) \geq 0$  for all  $(s_h, a_h)$ , such that  $\widetilde{n}_{s_h, a_h} \geq 3E_{\rho}$ .
Indeed, if  $\widehat{Q}_h^p (s_h,a_h) < 0$ , then  $\overline{Q}_h(s_h,a_h) = 0$ . In this case,  $\xi_{h}(s_{h},a_{h}) = (\mathcal{T}_{h}\widetilde{V}_{h + 1})(s_{h},a_{h})\geq 0$  (note  $\widetilde{V}_h\geq 0$  by the definition). If  $\widehat{Q}_h^p (s_h,a_h)\geq 0$ , then by definition  $\overline{Q}_h(s_h,a_h) = \min \{\widehat{Q}_h^p (s_h,a_h),H - h + 1\} ^+ \leq \widehat{Q}_h^p (s_h,a_h)$  and this implies
$$
\begin{array}{l} \xi_ {h} (s _ {h}, a _ {h}) \geq (\mathcal {T} _ {h} \widetilde {V} _ {h + 1}) (s _ {h}, a _ {h}) - \widehat {Q} _ {h} ^ {p} (s _ {h}, a _ {h}) \\ = (P _ {h} - \widetilde {P} _ {h}) \cdot \widetilde {V} _ {h + 1} (s _ {h}, a _ {h}) + \Gamma_ {h} (s _ {h}, a _ {h}) \\ \geq - \sqrt {\frac {2 \mathrm {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}}} - \frac {1 6 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}} + \Gamma_ {h} (s _ {h}, a _ {h}) = 0, \\ \end{array}
$$
where the second inequality uses Lemma C.6, and the last equation uses Line 5 of Algorithm 1.
The second inequality: Then we prove  $\xi_h(s_h, a_h) \leq 2\sqrt{\frac{2\mathrm{Var}_{\widetilde{P}_h(\cdot|s_h,a_h)}(\widetilde{V}_{h+1}(\cdot)\cdot\iota}{\widetilde{n}_{s_h,a_h} - E_\rho}} + \frac{32SHE_\rho\cdot\iota}{\widetilde{n}_{s_h,a_h}}$  for all  $(s_h, a_h)$  such that  $\widetilde{n}_{s_h,a_h} \geq 3E_\rho$ .
First, since by construction  $\widetilde{V}_h\leq H - h + 1$  for all  $h\in [H]$ , this implies
$$
\widehat {Q} _ {h} ^ {p} = \widetilde {Q} _ {h} - \Gamma_ {h} \leq \widetilde {Q} _ {h} = r _ {h} + (\widetilde {P} _ {h} \cdot \widetilde {V} _ {h + 1}) \leq 1 + (H - h) = H - h + 1
$$
which is because  $r_h \leq 1$  and  $\widetilde{P}_h$  is a probability distribution. Therefore, we have the equivalent definition
$$
\overline {{Q}} _ {h} := \min \{\widehat {Q} _ {h} ^ {p}, H - h + 1 \} ^ {+} = \max \{\widehat {Q} _ {h} ^ {p}, 0 \} \geq \widehat {Q} _ {h} ^ {p}.
$$
$$
\begin{array}{l} \xi_ {h} (s _ {h}, a _ {h}) = (\mathcal {T} _ {h} \widetilde {V} _ {h + 1}) (s _ {h}, a _ {h}) - \overline {{Q}} _ {h} (s _ {h}, a _ {h}) \leq (\mathcal {T} _ {h} \widetilde {V} _ {h + 1}) (s _ {h}, a _ {h}) - \widehat {Q} _ {h} ^ {p} (s _ {h}, a _ {h}) \\ = \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) - \widetilde {Q} _ {h} \left(s _ {h}, a _ {h}\right) + \Gamma_ {h} \left(s _ {h}, a _ {h}\right) \\ = \left(P _ {h} - \widetilde {P} _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) + \Gamma_ {h} \left(s _ {h}, a _ {h}\right) \\ \leq \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}}} + \frac {1 6 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}} + \Gamma_ {h} (s _ {h}, a _ {h}) \\ = 2 \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}}} + \frac {3 2 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}}. \\ \end{array}
$$
The proof is complete by combining the two parts.
C.3.3 Reduction to augmented absorbing MDP
Before we prove the theorem, we need to construct an augmented absorbing MDP to bridge  $\widetilde{V}$  and  $V^{\star}$ . According to Line 5 in Algorithm 1, the locations with  $\widetilde{n}_{s_h,a_h} \leq E_{\rho}$  is heavily penalized with penalty of order  $\widetilde{O}(H)$ . Therefore we can prove that under the high probability event in Remark C.8,  $d_{h}^{\widehat{\pi}}(s_h,a_h) > 0$  only if  $d_h^\mu(s_h,a_h) > 0$  by induction, where  $\widehat{\pi}$  is the output of Algorithm 1. The conclusion holds for  $h = 1$ . Assume it holds for some  $h > 1$  that  $d_{h}^{\widehat{\pi}}(s_h,a_h) > 0$  only if  $d_h^\mu(s_h,a_h) > 0$ , then for any  $s_{h+1} \in S$  such that  $d_{h+1}^{\widehat{\pi}}(s_{h+1}) > 0$ , it holds that  $d_{h+1}^\mu(s_{h+1}) > 0$ , which leads to the conclusion that  $d_{h+1}^{\widehat{\pi}}(s_{h+1},a_{h+1}) > 0$  only if  $d_{h+1}^\mu(s_{h+1},a_{h+1}) > 0$ . To summarize, we have
$$
d _ {h} ^ {\pi_ {0}} \left(s _ {h}, a _ {h}\right) > 0 \text {o n l y i f} d _ {h} ^ {\mu} \left(s _ {h}, a _ {h}\right) > 0, \pi_ {0} \in \left\{\pi^ {\star}, \widehat {\pi} \right\}. \tag {27}
$$
Let us define  $M^{\dagger}$  by adding one absorbing state  $s_h^\dagger$  for all  $h \in \{2, \dots, H\}$ , therefore the augmented state space  $S^{\dagger} = S \cup \{s_h^\dagger\}$  and the transition and reward is defined as follows: (recall  $\mathcal{C}_h := \{(s_h, a_h) : d_h^\mu(s_h, a_h) > 0\}$ )
$$
P _ {h} ^ {\dagger} (\cdot \mid s _ {h}, a _ {h}) = \left\{ \begin{array}{l l} P _ {h} (\cdot \mid s _ {h}, a _ {h}) & s _ {h}, a _ {h} \in \mathcal {C} _ {h}, \\ \delta_ {s _ {h + 1} ^ {\dagger}} & s _ {h} = s _ {h} ^ {\dagger} \text {o r} s _ {h}, a _ {h} \notin \mathcal {C} _ {h}, \end{array} \right.   r _ {h} ^ {\dagger} (s _ {h}, a _ {h}) = \left\{ \begin{array}{l l} r _ {h} (s _ {h}, a _ {h}) & s _ {h}, a _ {h} \in \mathcal {C} _ {h} \\ 0 & s _ {h} = s _ {h} ^ {\dagger} \text {o r} s _ {h}, a _ {h} \notin \mathcal {C} _ {h} \end{array} \right.
$$
and we further define for any  $\pi$
$$
V _ {h} ^ {\dagger \pi} (s) = \mathbb {E} _ {\pi} ^ {\dagger} \left[ \sum_ {t = h} ^ {H} r _ {t} ^ {\dagger} \mid s _ {h} = s \right], v ^ {\dagger \pi} = \mathbb {E} _ {\pi} ^ {\dagger} \left[ \sum_ {t = 1} ^ {H} r _ {t} ^ {\dagger} \right] \forall h \in [ H ], \tag {28}
$$
where  $\mathbb{E}^{\dagger}$  means taking expectation under the absorbing MDP  $M^{\dagger}$ .
Note that because  $\pi^{\star}$  and  $\widehat{\pi}$  are fully covered by  $\mu$  (27), it holds that
$$
v ^ {\dagger \pi^ {*}} = v ^ {\pi^ {*}}, v ^ {\dagger \widehat {\pi}} = v ^ {\widehat {\pi}}. \tag {29}
$$
Define  $(\mathcal{T}_h^\dagger V)(\cdot ,\cdot)\coloneqq r_h^\dagger (\cdot ,\cdot) + (P_h^\dagger V)(\cdot ,\cdot)$  for any  $V\in \mathbb{R}^{S + 1}$ . Note  $\widehat{\pi}$ ,  $\overline{Q}_h$ ,  $\widetilde{V}_h$  are defined in Algorithm 1 (we extend the definition by letting  $\widetilde{V}_h(s_h^\dagger) = 0$  and  $\overline{Q}_h(s_h^\dagger,\cdot) = 0$ ) and denote  $\xi_h^\dagger (s,a) = (\mathcal{T}_h^\dagger \widetilde{V}_{h + 1})(s,a) - \overline{Q}_h(s,a)$ . Using identical proof to Lemma C.9, we have
$$
V _ {1} ^ {\dagger \pi^ {\star}} (s) - V _ {1} ^ {\dagger \widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] - \sum_ {h = 1} ^ {H} \mathbb {E} _ {\widehat {\pi}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right], \tag {30}
$$
where  $V_1^{\dagger \pi}$  is defined in (28). Furthermore, (30) holds for all  $V_h^{\dagger \pi^\star}(s) - V_h^{\dagger \widehat{\pi}}(s)$ .
C.3.4 Finalize our result with non-private statistics
For those  $(s_h, a_h) \in \mathcal{C}_h$ ,  $\xi_h^\dagger(s_h, a_h) = r_h(s_h, a_h) + P_h \widetilde{V}_{h+1}(s_h, a_h) - \overline{Q}_h(s_h, a_h) = \xi_h(s_h, a_h)$ . For those  $(s_h, a_h) \notin \mathcal{C}_h$  or  $s_h = s_h^\dagger$ , we have  $\xi_h^\dagger(s_h, a_h) = 0$ .
Therefore, by (30) and Lemma C.10, under the high probability events in Lemma C.3, Lemma C.6 and Lemma C.7, we have for all  $t \in [H]$ ,  $s \in S$  ( $\mathcal{S}$  does not include the absorbing state  $s_t^\dagger$ ),
$$
\begin{array}{l} V _ {t} ^ {\dagger \pi^ {\star}} (s) - V _ {t} ^ {\dagger \widehat {\pi}} (s) \leq \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} \left(s _ {h}, a _ {h}\right) \mid s _ {t} = s \right] - \sum_ {h = t} ^ {H} \mathbb {E} _ {\widehat {\pi}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} \left(s _ {h}, a _ {h}\right) \mid s _ {t} = s \right] \\ \leq \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} \left(s _ {h}, a _ {h}\right) \mid s _ {t} = s \right] - 0 \\ \leq \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ 2 \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}}} + \frac {3 2 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}} \mid s _ {t} = s \right] \cdot \mathbb {1} \left(\left(s _ {h}, a _ {h}\right) \in \mathcal {C} _ {h}\right) \\ \leq \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ 2 \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{n _ {s _ {h} , a _ {h}} - 2 E _ {\rho}}} + \frac {3 2 S H E _ {\rho} \cdot \iota}{n _ {s _ {h} , a _ {h}} - E _ {\rho}} \mid s _ {t} = s \right] \cdot \mathbb {1} \left(\left(s _ {h}, a _ {h}\right) \in \mathcal {C} _ {h}\right) \tag {31} \\ \leq \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {*}} ^ {\dagger} \left[ 4 \sqrt {\frac {\operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{n _ {s _ {h} , a _ {h}}}} + \frac {1 2 8 S H E _ {\rho} \cdot \iota}{3 n _ {s _ {h} , a _ {h}}} \mid s _ {t} = s \right] \cdot \mathbb {1} \left(\left(s _ {h}, a _ {h}\right) \in \mathcal {C} _ {h}\right) \\ \leq \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ 4 \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}} + \frac {2 5 6 S H E _ {\rho} \cdot \iota}{3 n d _ {h} ^ {\mu} (s _ {h} , a _ {h})} \mid s _ {t} = s \right] \cdot \mathbb {1} \left(\left(s _ {h}, a _ {h}\right) \in \mathcal {C} _ {h}\right) \\ \end{array}
$$
The second and third inequality are because of Lemma C.10, Remark C.8 and the fact that either  $\xi^{\dagger} = 0$  or  $\xi^{\dagger} = \xi$  while  $(s_h,a_h)\in \mathcal{C}_h$ . The forth inequality is due to Lemma C.3. The fifth inequality is because of Remark C.8. The last inequality is by Lemma C.7.
Below we present a crude bound of  $\left|V_t^\dagger \pi^* (s) - \widetilde{V}_t(s)\right|$ , which can be further used to bound the main term in the main result.
Lemma C.11 (Self-bounding, private version of Lemma D.7 in [Yin and Wang, 2021b]). Under the high probability events in Lemma C.3, Lemma C.6 and Lemma C.7, it holds that for all  $t \in [H]$  and  $s \in S$ ,
$$
\left| V _ {t} ^ {\dagger \pi^ {\star}} (s) - \widetilde {V} _ {t} (s) \right| \leq \frac {4 \sqrt {2 \iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}}.
$$
where  $\bar{d}_m$  is defined in Theorem 3.4.
Proof of Lemma C.11. According to (31), since  $\mathrm{Var}_{\widetilde{P}_h(\cdot |s_h,a_h)}(\widetilde{V}_{h + 1}(\cdot))\leq H^2$ , we have for all  $t\in [H]$
$$
\left| V _ {t} ^ {\dagger \pi^ {\star}} (s) - V _ {t} ^ {\dagger \widehat {\pi}} (s) \right| \leq \frac {4 \sqrt {2 \iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}} \tag {32}
$$
Next, apply Lemma E.7 by setting  $\pi = \widehat{\pi}$ ,  $\pi' = \pi^{\star}$ ,  $\widehat{Q} = \overline{Q}$ ,  $\widehat{V} = \widetilde{V}$  under  $M^{\dagger}$ , then we have
$$
\begin{array}{l} V _ {t} ^ {\dagger \pi^ {\star}} (s) - \widetilde {V} _ {t} (s) = \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} (s _ {h}, a _ {h}) \mid s _ {t} = s \right] + \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ \langle \overline {{Q}} _ {h} (s _ {h}, \cdot), \pi_ {h} ^ {\star} (\cdot | s _ {h}) - \widehat {\pi} _ {h} (\cdot | s _ {h}) \rangle \mid s _ {t} = s \right] \\ \leq \sum_ {h = t} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} \left(s _ {h}, a _ {h}\right) \mid s _ {t} = s \right] \\ \leq \frac {4 \sqrt {2 \iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}}. \tag {33} \\ \end{array}
$$
Also, apply Lemma E.7 by setting  $\pi = \pi' = \widehat{\pi}$ ,  $\widehat{Q} = \overline{Q}$ ,  $\widehat{V} = \widetilde{V}$  under  $M^{\dagger}$ , then we have
$$
\widetilde {V} _ {t} (s) - V _ {t} ^ {\dagger \widehat {\pi}} (s) = - \sum_ {h = t} ^ {H} \mathbb {E} _ {\widehat {\pi}} ^ {\dagger} \left[ \xi_ {h} ^ {\dagger} \left(s _ {h}, a _ {h}\right) \mid s _ {t} = s \right] \leq 0. \tag {34}
$$
The proof is complete by combing (32), (33) and (34).
Now we are ready to bound  $\sqrt{\operatorname{Var}_{\widetilde{P}_h(\cdot|s_h,a_h)}(\widetilde{V}_{h+1}(\cdot))}$  by  $\sqrt{\operatorname{Var}_{P_h(\cdot|s_h,a_h)}(V_{h+1}^{\dagger\star}(\cdot))}$ . Under the high probability events in Lemma C.3, Lemma C.6 and Lemma C.7, with probability  $1 - \delta$ , it holds that for all  $(s_h,a_h) \in \mathcal{C}_h$ ,
$$
\begin{array}{l} \sqrt {\operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot))} \leq \sqrt {\operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\dagger \star} (\cdot))} + \left\| \widetilde {V} _ {h + 1} - V _ {h + 1} ^ {\dagger \pi^ {\star}} \right\| _ {\infty , s \in \mathcal {S}} \\ \leq \sqrt {\operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\dagger \star} (\cdot))} + \frac {4 \sqrt {2 \iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}} \\ \leq \sqrt {\operatorname {V a r} _ {\hat {P} _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\dagger \star} (\cdot))} + \frac {4 \sqrt {2 \iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}} + 4 H \sqrt {\frac {S E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}}} \\ \leq \sqrt {\operatorname {V a r} _ {\widehat {P} _ {h} (\cdot | s _ {h} , a _ {h})} \left(V _ {h + 1} ^ {\dagger \star} (\cdot)\right)} + \frac {4 \sqrt {2 \iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}} + 8 H \sqrt {\frac {S E _ {\rho}}{n \cdot \bar {d} _ {m}}} \tag {35} \\ \leq \sqrt {\operatorname {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\dagger \star} (\cdot))} + \frac {4 \sqrt {2 \iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}} + 8 H \sqrt {\frac {S E _ {\rho}}{n \cdot \bar {d} _ {m}}} + 3 H \sqrt {\frac {\iota}{n \cdot \bar {d} _ {m}}} \\ \leq \sqrt {\operatorname {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\dagger \star} (\cdot))} + \frac {9 \sqrt {\iota} H ^ {2}}{\sqrt {n \cdot \bar {d} _ {m}}} + \frac {2 5 6 S H ^ {2} E _ {\rho} \cdot \iota}{3 n \cdot \bar {d} _ {m}} + 8 H \sqrt {\frac {S E _ {\rho}}{n \cdot \bar {d} _ {m}}}. \\ \end{array}
$$
The second inequality is because of Lemma C.11. The third inequality is due to Lemma C.5. The forth inequality comes from Lemma C.3 and Remark C.8. The fifth inequality holds with probability  $1 - \delta$  because of Lemma E.5 and a union bound.
Finally, by plugging (35) into (31) and averaging over  $s_1$ , we finally have with probability  $1 - 4\delta$