SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} v ^ {\pi^ {*}} - v ^ {\widehat {\pi}} = v ^ {\dagger \pi^ {*}} - v ^ {\dagger \widehat {\pi}} \leq \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {*}} ^ {\dagger} \left[ 4 \sqrt {\frac {2 \mathrm {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot)) \cdot \iota}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}} + \frac {2 5 6 S H E _ {\rho} \cdot \iota}{3 n d _ {h} ^ {\mu} (s _ {h} , a _ {h})} \right] \\ \leq 4 \sqrt {2} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\star}} ^ {\dagger} \left[ \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\dagger \star} (\cdot)) \cdot \iota}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}} \right] + \widetilde {O} \left(\frac {H ^ {3} + S H ^ {2} E _ {\rho}}{n \cdot \bar {d} _ {m}}\right) \tag {36} \\ = 4 \sqrt {2} \sum_ {h = 1} ^ {H} \sum_ {(s _ {h}, a _ {h}) \in \mathcal {C} _ {h}} d _ {h} ^ {\pi^ {\star}} (s _ {h}, a _ {h}) \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\dagger \star} (\cdot)) \cdot \iota}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}} + \widetilde {O} \left(\frac {H ^ {3} + S H ^ {2} E _ {\rho}}{n \cdot \bar {d} _ {m}}\right) \\ = 4 \sqrt {2} \sum_ {h = 1} ^ {H} \sum_ {(s _ {h}, a _ {h}) \in \mathcal {C} _ {h}} d _ {h} ^ {\pi^ {\star}} (s _ {h}, a _ {h}) \sqrt {\frac {\mathrm {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\star} (\cdot)) \cdot \iota}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}} + \widetilde {O} \left(\frac {H ^ {3} + S H ^ {2} E _ {\rho}}{n \cdot \bar {d} _ {m}}\right), \\ \end{array}
$$
where  $\widetilde{O}$  absorbs constants and Polylog terms. The first equation is due to (29). The first inequality is because of (31). The second inequality comes from (35) and our assumption that  $n\cdot \bar{d}_m\geq c_1H^2$ . The second equation uses the fact that  $d_h^{\pi^*}(s_h,a_h) = d_h^{\dagger \pi^*}(s_h,a_h)$ , for all  $(s_h,a_h)$ . The last equation is because for any  $(s_h,a_h,s_{h + 1})$  such that  $d_h^{\pi^*}(s_h,a_h) > 0$  and  $P_{h}(s_{h + 1}|s_{h},a_{h}) > 0$ ,  $V_{h + 1}^{\dagger \star}(s_{h + 1}) = V_{h + 1}^{\star}(s_{h + 1})$ .
C.4 Put everything together
Combining Lemma C.1 and (36), the proof of Theorem 3.4 is complete.
D Proof of Theorem 4.1
D.1 Proof sketch
Since the whole proof for privacy guarantee is not very complex, we present it in Section D.2 below and only sketch the proof for suboptimality bound.
First of all, by extended value difference (Lemma E.7 and E.8), we can convert bounding the suboptimality gap of  $v^{\star} - v^{\widehat{\pi}}$  to bounding  $\sum_{h=1}^{H} \mathbb{E}_{\pi}\left[\Gamma_{h}(s_{h},a_{h})\right]$ , given that  $\left|\left(\mathcal{T}_{h}\widetilde{V}_{h+1}-\widetilde{\mathcal{T}}_{h}\widetilde{V}_{h+1}\right)(s,a)\right| \leq \Gamma_{h}(s,a)$  for all  $s,a,h$ . To bound  $(\mathcal{T}_{h}\widetilde{V}_{h+1}-\widetilde{\mathcal{T}}_{h}\widetilde{V}_{h+1})(s,a)$ , according to our analysis about the upper bound of the noises we add, we can decompose  $(\mathcal{T}_{h}\widetilde{V}_{h+1}-\widetilde{\mathcal{T}}_{h}\widetilde{V}_{h+1})(s,a)$  to lower order terms  $(\widetilde{O}\left(\frac{1}{K}\right))$  and the following key quantity:
$$
\phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left[ \sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right) - \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \right]. \tag {37}
$$
For the term above, we prove an upper bound of  $\left\| \sigma_{\widetilde{V}_{h + 1}}^2 -\widetilde{\sigma}_h^2\right\|_\infty$ , so we can convert  $\widetilde{\sigma}_h^2$  to  $\sigma_{\widetilde{V}_{h + 1}}^2$ . Next, since  $\mathrm{Var}\left[r_h^\tau +\widetilde{V}_{h + 1}\left(s_{h + 1}^\tau\right) - \left(\mathcal{T}_h\widetilde{V}_{h + 1}\right)\left(s_h^\tau ,a_h^\tau\right)\mid s_h^\tau ,a_h^\tau \right]\approx \sigma_{\widetilde{V}_{h + 1}}^2$ , we can apply Bernstein's inequality for self-normalized martingale (Lemma E.10) as in Yin et al. [2022] for deriving tighter bound.
Finally, we replace the private statistics by non-private ones. More specifically, we convert  $\sigma_{\widetilde{V}_{h + 1}}^2$  to  $\sigma_h^{\star 2}(\Lambda_h^{-1}$  to  $\Lambda_h^{\star -1}$ ) by combining the crude upper bound of  $\left\| \widetilde{V} -V^{\star}\right\|_{\infty}$  and matrix concentrations.
D.2 Proof of the privacy guarantee
The privacy guarantee of DP-VAPVI (Algorithm 2) is summarized by Lemma D.1 below.
Lemma D.1 (Privacy analysis of DP-VAPVI (Algorithm 2)). DP-VAPVI (Algorithm 2) satisfies  $\rho - zCDP$ .
Proof of Lemma D.1. For  $\sum_{\tau=1}^{K} \phi(\bar{s}_h^\tau, \bar{a}_h^\tau) \cdot \widetilde{V}_{h+1}(\bar{s}_{h+1}^\tau)^2$ , the  $\ell_2$  sensitivity is  $2H^2$ . For  $\sum_{\tau=1}^{K} \phi(\bar{s}_h^\tau, \bar{a}_h^\tau) \cdot \widetilde{V}_{h+1}(\bar{s}_{h+1}^\tau)$  and  $\sum_{\tau=1}^{K} \phi(s_h^\tau, a_h^\tau) \cdot (r_h^\tau + \widetilde{V}_{h+1}(s_{h+1}^\tau)) / \widetilde{\sigma}_h^2(s_h^\tau, a_h^\tau)$ , the  $\ell_2$  sensitivity is  $2H$ . Therefore according to Lemma 2.7, the use of Gaussian Mechanism (the additional noises  $\phi_1, \phi_2, \phi_3$ ) ensures  $\rho_0$ -zCDP for each counter. For  $\sum_{\tau=1}^{K} \phi(\bar{s}_h^\tau, \bar{a}_h^\tau) \phi(\bar{s}_h^\tau, \bar{a}_h^\tau)^\top + \lambda I$  and  $\sum_{\tau=1}^{K} \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^\top / \widetilde{\sigma}_h^2(s_h^\tau, a_h^\tau) + \lambda I$ , according to Appendix D in [Redberg and Wang, 2021], the per-instance  $\ell_2$  sensitivity is
$$
\left\| \Delta_ {x} \right\| _ {2} = \frac {1}{\sqrt {2}} \sup  _ {\phi : \| \phi \| _ {2} \leq 1} \left\| \phi \phi^ {\top} \right\| _ {F} = \frac {1}{\sqrt {2}} \sup  _ {\phi : \| \phi \| _ {2} \leq 1} \sqrt {\sum_ {i , j} \phi_ {i} ^ {2} \phi_ {j} ^ {2}} = \frac {1}{\sqrt {2}}.
$$
Therefore the use of Gaussian Mechanism (the additional noises  $K_{1}, K_{2}$ ) also ensures  $\rho_0$ -zCDP for each counter. $^{12}$  Combining these results, according to Lemma E.17, the whole algorithm satisfies  $5H\rho_0 = \rho$ -zCDP.
D.3 Proof of the sub-optimality bound
D.3.1 Utility analysis and some preparation
We begin with the following high probability bound of the noises we add.
<sup>12</sup>For more detailed explanation, we refer the readers to Appendix D of [Redberg and Wang, 2021].
Lemma D.2 (Utility analysis). Let  $L = 2H\sqrt{\frac{d}{\rho_0}\log(\frac{10Hd}{\delta})} = 2H\sqrt{\frac{5Hd\log(\frac{10Hd}{\delta})}{\rho}}$  and
[ E = \sqrt{\frac{2d}{\rho_0}} \left( 2 + \left( \frac{\log(5c_1H / \delta)}{c_2d} \right)^{\frac{2}{3}} \right) = \sqrt{\frac{10Hd}{\rho}} \left( 2 + \left( \frac{\log(5c_1H / \delta)}{c_2d} \right)^{\frac{2}{3}} \right) ] for some universal constants  $c_1, c_2$ . Then with probability  $1 - \delta$ , the following inequalities hold simultaneously:
$$
F o r a l l h \in [ H ], \| \phi_ {1} \| _ {2} \leq H L, \| \phi_ {2} \| _ {2} \leq L, \| \phi_ {3} \| _ {2} \leq L.
$$
For all  $h\in [H]$ $K_{1},K_{2}$  are symmetric and positive definite and  $\| K_i\| _2\leq E$ $i\in \{1,2\}$  (38
Proof of Lemma D.2. The second line of (38) results from Lemma 19 in [Redberg and Wang, 2021] and Weyl's Inequality. The first line of (38) directly results from the concentration inequality for Gaussian distribution and a union bound.
Define the Bellman update error  $\zeta_h(s, a) := (\mathcal{T}_h \widetilde{V}_{h+1})(s, a) - \widehat{Q}_h(s, a)$  and recall  $\widehat{\pi}_h(s) = \operatorname{argmax}_{\pi_h} \langle \widehat{Q}_h(s, \cdot), \pi_h(\cdot \mid s) \rangle_A$ , then because of Lemma E.8,
$$
V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \zeta_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] - \sum_ {h = 1} ^ {H} \mathbb {E} _ {\widehat {\pi}} \left[ \zeta_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right]. \tag {39}
$$
Define  $\widetilde{T}_h\widetilde{V}_{h + 1}(\cdot ,\cdot) = \phi (\cdot ,\cdot)^\top \widetilde{w}_h$ . Then similar to Lemma C.10, we have the following lemma showing that in order to bound the sub-optimality, it is sufficient to bound the pessimistic penalty.
Lemma D.3 (Lemma C.1 in [Yin et al., 2022]). Suppose with probability  $1 - \delta$ , it holds for all  $s, a, h \in S \times \mathcal{A} \times [H]$  that  $|(T_h \widetilde{V}_{h+1} - \widetilde{T}_h \widetilde{V}_{h+1})(s, a)| \leq \Gamma_h(s, a)$ , then it implies  $\forall s, a, h \in S \times \mathcal{A} \times [H]$ ,  $0 \leq \zeta_h(s, a) \leq 2\Gamma_h(s, a)$ . Furthermore, with probability  $1 - \delta$ , it holds for any policy  $\pi$  simultaneously,
$$
V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} 2 \cdot \mathbb {E} _ {\pi} \left[ \Gamma_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right].
$$
Proof of Lemma D.3. We first show given  $|(\mathcal{T}_h\widetilde{V}_{h + 1} - \widetilde{\mathcal{T}}_h\widetilde{V}_{h + 1})(s,a)| \leq \Gamma_h(s,a)$ , then  $0 \leq \zeta_h(s,a) \leq 2\Gamma_h(s,a), \forall s,a,h \in S \times \bar{\mathcal{A}} \times [H]$ .
Step 1: The first step is to show  $0 \leq \zeta_h(s, a), \forall s, a, h \in S \times \mathcal{A} \times [H]$ .
Indeed, if  $\bar{Q}_h(s,a)\leq 0$ , then by definition  $\widehat{Q}_h(s,a) = 0$  and therefore  $\zeta_h(s,a)\coloneqq (\mathcal{T}_h\widetilde{V}_{h + 1})(s,a) - \widehat{Q}_h(s,a) = (\mathcal{T}_h\widetilde{V}_{h + 1})(s,a)\geq 0$ . If  $\bar{Q}_h(s,a) > 0$ , then  $\widehat{Q}_h(s,a)\leq \bar{Q}_h(s,a)$  and
$$
\begin{array}{l} \zeta_ {h} (s, a) := \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \widehat {Q} _ {h} (s, a) \geq \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \bar {Q} _ {h} (s, a) \\ = \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \left(\widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) + \Gamma_ {h} (s, a) \geq 0. \\ \end{array}
$$
Step 2: The second step is to show  $\zeta_h(s,a)\leq 2\Gamma_h(s,a),\forall s,a,h\in S\times \mathcal{A}\times [H]$
Under the assumption that  $|(\mathcal{T}_h\widetilde{V}_{h + 1} - \widetilde{\mathcal{T}}_h\widetilde{V}_{h + 1})(s,a)|\leq \Gamma_h(s,a)$ , we have
$$
\bar {Q} _ {h} (s, a) = \left(\widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \Gamma_ {h} (s, a) \leq \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) \leq H - h + 1,
$$
which implies that  $\widehat{Q}_h(s,a) = \max (\bar{Q}_h(s,a),0)$ . Therefore, it holds that
$$
\begin{array}{l} \zeta_ {h} (s, a) := \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \widehat {Q} _ {h} (s, a) \leq \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \bar {Q} _ {h} (s, a) \\ = \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \left(\widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) + \Gamma_ {h} (s, a) \leq 2 \cdot \Gamma_ {h} (s, a). \\ \end{array}
$$
For the last statement, denote  $\mathfrak{F} := \{0 \leq \zeta_h(s, a) \leq 2\Gamma_h(s, a), \forall s, a, h \in S \times \mathcal{A} \times [H]\}$ . Note conditional on  $\mathfrak{F}$ , then by (39),  $V_1^\pi(s) - V_1^{\widehat{\pi}}(s) \leq \sum_{h=1}^{H} 2 \cdot \mathbb{E}_\pi[\Gamma_h(s_h, a_h) \mid s_1 = s]$  holds for any
policy  $\pi$  almost surely. Therefore,
$$
\begin{array}{l} \mathbb {P} \left[ \forall \pi , V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} 2 \cdot \mathbb {E} _ {\pi} \left[ \Gamma_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right]. \right] \\ = \mathbb {P} \left[ \forall \pi , V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} 2 \cdot \mathbb {E} _ {\pi} \left[ \Gamma_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] \middle | \mathfrak {F} \right] \cdot \mathbb {P} [ \mathfrak {F} ] \\ + \mathbb {P} \left[ \forall \pi , V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} 2 \cdot \mathbb {E} _ {\pi} \left[ \Gamma_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] \middle | \mathfrak {F} ^ {c} \right] \cdot \mathbb {P} [ \mathfrak {F} ^ {c} ] \\ \geq \mathbb {P} \left[ \forall \pi , V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \sum_ {h = 1} ^ {H} 2 \cdot \mathbb {E} _ {\pi} \left[ \Gamma_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] \middle | \mathfrak {F} \right] \cdot \mathbb {P} [ \mathfrak {F} ] = 1 \cdot \mathbb {P} [ \mathfrak {F} ] \geq 1 - \delta , \\ \end{array}
$$
D.3.2 Bound the pessimistic penalty
By Lemma D.3, it remains to bound  $|(\mathcal{T}_h\widetilde{V}_{h + 1})(s,a) - (\widetilde{\mathcal{T}}_h\widetilde{V}_{h + 1})(s,a)|$ . Suppose  $w_{h}$  is the coefficient corresponding to the  $\mathcal{T}_h\widetilde{V}_{h + 1}$  (such  $w_{h}$  exists by Lemma E.14), i.e.  $\mathcal{T}_h\widetilde{V}_{h + 1} = \phi^\top w_h$ , and recall  $(\widetilde{\mathcal{T}}_h\widetilde{V}_{h + 1})(s,a) = \phi (s,a)^\top \widetilde{w}_h$ , then:
$$
\begin{array}{l} \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) - \left(\widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) = \phi (s, a) ^ {\top} \left(w _ {h} - \widetilde {w} _ {h}\right) \\ = \phi (s, a) ^ {\top} w _ {h} - \phi (s, a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) + \phi_ {3}\right) \\ = \underbrace {\phi (s , a) ^ {\top} w _ {h} - \phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right)\right)} _ {\text {(i)}} \\ - \underbrace {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi_ {3}} _ {\text {(i i)}} + \underbrace {\phi (s , a) ^ {\top} \left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right) \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right) + \phi_ {3}\right)} _ {\text {(i i i)}}, \tag {40} \\ \end{array}
$$
where  $\widehat{\Lambda}_h = \widetilde{\Lambda}_h - K_2 = \sum_{\tau=1}^{K} \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^\top / \widetilde{\sigma}_h^2(s_h^\tau, a_h^\tau) + \lambda I$ .
Term (ii) can be handled by the following Lemma D.4
Lemma D.4. Recall  $\kappa$  in Assumption 2.2. Under the high probability event in Lemma D.2, suppose  $K \geq \max \left\{\frac{512H^4 \cdot \log\left(\frac{2Hd}{\delta}\right)}{\kappa^2}, \frac{4\lambda H^2}{\kappa}\right\}$ , then with probability  $1 - \delta$ , for all  $s, a, h \in S \times \mathcal{A} \times [H]$ , it holds that
$$
\left| \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi_ {3} \right| \leq \frac {4 H ^ {2} L / \kappa}{K}.
$$
Proof of Lemma D.4. Define  $\widetilde{\Lambda}_h^p = \mathbb{E}_{\mu,h}[\widetilde{\sigma}_h^{-2}(s,a)\phi(s,a)\phi(s,a)^\top]$ . Then because of Assumption 2.2 and  $\widetilde{\sigma}_h \leq H$ , it holds that  $\lambda_{\min}(\widetilde{\Lambda}_h^p) \geq \frac{\kappa}{H^2}$ . Therefore, due to Lemma E.13, we have with
probability  $1 - \delta$
$$
\begin{array}{l} \left| \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi_ {3} \right| \leq \| \phi (s, a) \| _ {\widehat {\Lambda} _ {h} ^ {- 1}} \cdot \| \phi_ {3} \| _ {\widehat {\Lambda} _ {h} ^ {- 1}} \\ \leq \frac {4}{K} \| \phi (s, a) \| _ {\left(\widetilde {\Lambda} _ {h} ^ {p}\right) ^ {- 1}} \cdot \| \phi_ {3} \| _ {\left(\widetilde {\Lambda} _ {h} ^ {p}\right) ^ {- 1}} \\ \leq \frac {4 L}{K} \left\| \left(\widetilde {\Lambda} _ {h} ^ {p}\right) ^ {- 1} \right\| \\ \leq \frac {4 H ^ {2} L / \kappa}{K}. \\ \end{array}
$$
The first inequality is because of Cauchy-Schwarz inequality. The second inequality holds with probability  $1 - \delta$  due to Lemma E.13 and a union bound. The third inequality holds because  $\sqrt{a^{\top} \cdot A \cdot a} \leq \sqrt{\|a\|_2 \|A\|_2 \|a\|_2} = \|a\|_2 \sqrt{\|A\|_2}$ . The last inequality arises from  $\|( \widetilde{\Lambda}_h^p)^{-1} \| = \lambda_{\max}((\widetilde{\Lambda}_h^p)^{-1}) = \lambda_{\min}^{-1} (\widetilde{\Lambda}_h^p) \leq \frac{H^2}{\kappa}$ .
The difference between  $\widetilde{\Lambda}_h^{-1}$  and  $\widehat{\Lambda}_h^{-1}$  can be bounded by the following Lemma D.5
Lemma D.5. Under the high probability event in Lemma D.2, suppose  $K \geq \frac{128H^4\log\frac{2dH}{\delta}}{\kappa^2}$ , then with probability  $1 - \delta$ , for all  $h \in [H]$ , it holds that  $\| \widehat{\Lambda}_h^{-1} - \widetilde{\Lambda}_h^{-1} \| \leq \frac{4H^4E / \kappa^2}{K^2}$ .
Proof of Lemma D.5. First of all, we have
$$
\begin{array}{l} \left\| \widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1} \right\| = \left\| \widehat {\Lambda} _ {h} ^ {- 1} \cdot \left(\widehat {\Lambda} _ {h} - \widetilde {\Lambda} _ {h}\right) \cdot \widetilde {\Lambda} _ {h} ^ {- 1} \right\| \\ \leq \left\| \widehat {\Lambda} _ {h} ^ {- 1} \right\| \cdot \left\| \widehat {\Lambda} _ {h} - \widetilde {\Lambda} _ {h} \right\| \cdot \left\| \widetilde {\Lambda} _ {h} ^ {- 1} \right\| \tag {41} \\ \leq \lambda_ {\min } ^ {- 1} (\widehat {\Lambda} _ {h}) \cdot \lambda_ {\min } ^ {- 1} (\widetilde {\Lambda} _ {h}) \cdot E. \\ \end{array}
$$
The first inequality is because  $\| A \cdot B \| \leq \| A \| \cdot \| B \|$ . The second inequality is due to Lemma D.2.
Let  $\widehat{\Lambda}_h^\prime = \frac{1}{K}\widehat{\Lambda}_h$ , then because of Lemma E.12, with probability  $1 - \delta$ , it holds that for all  $h\in [H]$
$$
\left\| \widehat {\Lambda} _ {h} ^ {\prime} - \mathbb {E} _ {\mu , h} \left[ \phi (s, a) \phi (s, a) ^ {\top} / \tilde {\sigma} _ {h} ^ {2} (s, a) \right] - \frac {\lambda}{K} I _ {d} \right\| \leq \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2},
$$
which implies that when  $K \geq \frac{128H^4\log\frac{2dH}{\delta}}{\kappa^2}$ , it holds that (according to Weyl's Inequality)
$$
\lambda_ {\min } \left(\widehat {\Lambda} _ {h} ^ {\prime}\right) \geq \lambda_ {\min } \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \widetilde {\sigma} _ {h} ^ {2} (s, a) ]\right) + \frac {\lambda}{K} - \frac {\kappa}{2 H ^ {2}} \geq \frac {\kappa}{2 H ^ {2}}.
$$
Under this high probability event, we have  $\lambda_{\mathrm{min}}(\widehat{\Lambda}_h) \geq \frac{K\kappa}{2H^2}$  and therefore  $\lambda_{\mathrm{min}}(\widetilde{\Lambda}_h) \geq \lambda_{\mathrm{min}}(\widehat{\Lambda}_h) \geq \frac{K\kappa}{2H^2}$ . Plugging these two results into (41), we have
$$
\| \widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1} \| \leq \frac {4 H ^ {4} E / \kappa^ {2}}{K ^ {2}}.
$$
Then we can bound term (iii) by the following Lemma D.6
Lemma D.6. Suppose  $K \geq \max \left\{\frac{128H^4\log\frac{2dH}{\delta}}{\kappa^2},\frac{\sqrt{2}L}{\sqrt{d\kappa}}\right\}$ , under the high probability events in Lemma D.2 and Lemma D.5, it holds that for all  $s,a,h \in S \times \mathcal{A} \times [H]$
$$
\left| \phi (s, a) ^ {\top} (\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}) \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) + \phi_ {3}\right) \right| \leq \frac {4 \sqrt {2} H ^ {4} E \sqrt {d} / \kappa^ {3 / 2}}{K}.
$$
Proof of Lemma D.6. First of all, the left hand side is bounded by
$$
\left\| \left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right) \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {2} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}}
$$
due to Lemma D.5. Then the left hand side can be further bounded by
$$
\begin{array}{l} H \sum_ {\tau = 1} ^ {K} \left\| \left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right) \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) / \widetilde {\sigma} _ {h} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \right\| _ {2} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}} \\ \leq H \sum_ {\tau = 1} ^ {K} \sqrt {T r \left((\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}) \cdot \frac {\phi (s _ {h} ^ {\tau} , a _ {h} ^ {\tau}) \phi (s _ {h} ^ {\tau} , a _ {h} ^ {\tau}) ^ {\top}}{\widetilde {\sigma} _ {h} ^ {2} (s _ {h} ^ {\tau} , a _ {h} ^ {\tau})} \cdot (\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1})\right)} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}} \\ \leq H \sqrt {K \cdot T r \left(\left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right) \cdot \widehat {\Lambda} _ {h} \cdot \left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right)\right)} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}} \\ \leq H \sqrt {K d \cdot \lambda_ {\max} \left(\left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right) \cdot \widehat {\Lambda} _ {h} \cdot \left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right)\right)} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}} \\ = H \sqrt {K d \cdot \left\| \left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right) \cdot \widehat {\Lambda} _ {h} \cdot \left(\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}\right) \right\| _ {2}} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}} \\ \leq H \sqrt {K d \cdot \left\| \widetilde {\Lambda} _ {h} ^ {- 1} \right\| _ {2} \cdot \left\| \widetilde {\Lambda} _ {h} - \widehat {\Lambda} _ {h} \right\| _ {2} \cdot \left\| \widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1} \right\| _ {2}} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}} \\ \leq \frac {2 \sqrt {2} H ^ {4} E \sqrt {d} / \kappa^ {3 / 2}}{K} + \frac {4 H ^ {4} E L / \kappa^ {2}}{K ^ {2}} \\ \leq \frac {4 \sqrt {2} H ^ {4} E \sqrt {d} / \kappa^ {3 / 2}}{K}. \\ \end{array}
$$
The first inequality is because  $\| a \|_2 = \sqrt{a^\top a} = \sqrt{Tr(aa^\top)}$ . The second inequality is due to Cauchy-Schwarz inequality. The third inequality is because for positive definite matrix  $A$ , it holds that  $Tr(A) = \sum_{i=1}^{d} \lambda_i(A) \leq d\lambda_{\max}(A)$ . The equation is because for symmetric, positive definite matrix  $A$ ,  $\| A \|_2 = \lambda_{\max}(A)$ . The forth inequality is due to  $\| A \cdot B \| \leq \| A \| \cdot \| B \|$ . The fifth inequality is because of Lemma D.2, Lemma D.5 and the statement in the proof of Lemma D.5 that  $\lambda_{\min}(\widetilde{\Lambda}_h) \geq \frac{K\kappa}{2H^2}$ . The last inequality uses the assumption that  $K \geq \frac{\sqrt{2}L}{\sqrt{d\kappa}}$ .
Now the remaining part is term (i), we have