SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \underbrace {\phi (s , a) ^ {\top} w _ {h} - \phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right)\right)} _ {\text {(i)}} \\ = \underbrace {\phi (s , a) ^ {\top} w _ {h} - \phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right) \cdot \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right)\right)} _ {\text {(i v)}} \\ - \underbrace {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right) - \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau} , a _ {h} ^ {\tau}\right)\right)} _ {\text {(v)}}. \tag {42} \\ \end{array}
$$
We are able to bound term (iv) by the following Lemma D.7.
Lemma D.7. Recall  $\kappa$  in Assumption 2.2. Under the high probability event in Lemma D.2, suppose  $K \geq \max \left\{\frac{512H^4 \cdot \log\left(\frac{2Hd}{\delta}\right)}{\kappa^2}, \frac{4\lambda H^2}{\kappa}\right\}$ , then with probability  $1 - \delta$ , for all  $s, a, h \in S \times \mathcal{A} \times [H]$
$$
\left| \phi (s, a) ^ {\top} w _ {h} - \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right| \leq \frac {8 \lambda H ^ {3} \sqrt {d} / \kappa}{K}.
$$
Proof of Lemma D.7. Recall  $\mathcal{T}_h\widetilde{V}_{h + 1} = \phi^\top w_h$  and apply Lemma E.13, we obtain with probability  $1 - \delta$ , for all  $s,a,h\in S\times \mathcal{A}\times [H]$
$$
\begin{array}{l} \left| \phi (s, a) ^ {\top} w _ {h} - \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right| \\ = \left| \phi (s, a) ^ {\top} w _ {h} - \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\top} w _ {h} / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right| \\ = \left| \phi (s, a) ^ {\top} w _ {h} - \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\widehat {\Lambda} _ {h} - \lambda I\right) w _ {h} \right| \\ = \left| \lambda \cdot \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} w _ {h} \right| \\ \leq \lambda \| \phi (s, a) \| _ {\widehat {\Lambda} _ {h} ^ {- 1}} \cdot \| w _ {h} \| _ {\widehat {\Lambda} _ {h} ^ {- 1}} \\ \leq \frac {4 \lambda}{K} \| \phi (s, a) \| _ {\left(\widetilde {\Lambda} _ {h} ^ {p}\right) ^ {- 1}} \cdot \| w _ {h} \| _ {\left(\widetilde {\Lambda} _ {h} ^ {p}\right) ^ {- 1}} \\ \leq \frac {4 \lambda}{K} \cdot 2 H \sqrt {d} \cdot \left\| \left(\tilde {\Lambda} _ {h} ^ {p}\right) ^ {- 1} \right\| \\ \leq \frac {8 \lambda H ^ {3} \sqrt {d} / \kappa}{K}, \\ \end{array}
$$
where  $\widetilde{\Lambda}_h^p \coloneqq \mathbb{E}_{\mu,h} \left[ \widetilde{\sigma}_h(s,a)^{-2} \phi(s,a) \phi(s,a)^\top \right]$ . The first inequality applies Cauchy-Schwarz inequality. The second inequality holds with probability  $1 - \delta$  due to Lemma E.13 and a union bound. The third inequality uses  $\sqrt{a^\top \cdot A \cdot a} \leq \sqrt{\|a\|_2 \|A\|_2 \|a\|_2} = \|a\|_2 \sqrt{\|A\|_2}$  and  $\|w_h\| \leq 2H \sqrt{d}$ . Finally, as  $\lambda_{\min}(\widetilde{\Lambda}_h^p) \geq \frac{\kappa}{\max_{h,s,a} \widetilde{\sigma}_h(s,a)^2} \geq \frac{\kappa}{H^2}$  implies  $\left\| (\widetilde{\Lambda}_h^p)^{-1} \right\| \leq \frac{H^2}{\kappa}$ , the last inequality holds.
For term (v), denote:  $x_{\tau} = \frac{\phi(s_h^\tau, a_h^\tau)}{\widetilde{\sigma}_h(s_h^\tau, a_h^\tau)}$ ,  $\eta_{\tau} = \left(r_h^\tau + \widetilde{V}_{h+1}(s_{h+1}^\tau) - \left(\mathcal{T}_h \widetilde{V}_{h+1}\right)(s_h^\tau, a_h^\tau)\right) / \widetilde{\sigma}_h(s_h^\tau, a_h^\tau)$ , then by Cauchy-Schwarz inequality, it holds that for all  $h, s, a \in [H] \times \mathcal{S} \times \mathcal{A}$ ,
$$
\begin{array}{l} \left| \phi (s, a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right) - \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} ^ {2} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right| \\ \leq \sqrt {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi (s , a)} \cdot \left\| \sum_ {\tau = 1} ^ {K} x _ {\tau} \eta_ {\tau} \right\| _ {\widehat {\Lambda} _ {h} ^ {- 1}}. \tag {43} \\ \end{array}
$$
We bound  $\sqrt{\phi(s,a)^{\top}\widehat{\Lambda}_{h}^{-1}\phi(s,a)}$  by  $\sqrt{\phi(s,a)^{\top}\widetilde{\Lambda}_{h}^{-1}\phi(s,a)}$  using the following Lemma D.8.
Lemma D.8. Suppose  $K \geq \max \left\{\frac{128H^4\log\frac{2dH}{\delta}}{\kappa^2},\frac{\sqrt{2}L}{\sqrt{d}\kappa}\right\}$ , under the high probability events in Lemma D.2 and Lemma D.5, it holds that for all  $s,a,h \in S \times \mathcal{A} \times [H]$
$$
\sqrt {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi (s , a)} \leq \sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a)} + \frac {2 H ^ {2} \sqrt {E} / \kappa}{K}.
$$
Proof of Lemma D.8.
$$
\begin{array}{l} \sqrt {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi (s , a)} = \sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a) + \phi (s , a) ^ {\top} (\widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1}) \phi (s , a)} \\ \leq \sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a) + \left\| \widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1} \right\| _ {2}} \tag {44} \\ \leq \sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a)} + \sqrt {\left\| \widehat {\Lambda} _ {h} ^ {- 1} - \widetilde {\Lambda} _ {h} ^ {- 1} \right\| _ {2}} \\ \leq \sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a)} + \frac {2 H ^ {2} \sqrt {E} / \kappa}{K}. \\ \end{array}
$$
The first inequality uses  $|a^\top A a| \leq \| a \|_2^2 \cdot \| A \|$ . The second inequality is because for  $a, b \geq 0$ ,  $\sqrt{a} + \sqrt{b} \geq \sqrt{a + b}$ . The last inequality uses Lemma D.5.
Remark D.9. Similarly, under the same assumption in Lemma D.8, we also have for all  $s, a, h \in S \times \mathcal{A} \times [H]$ ,
$$
\sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a)} \leq \sqrt {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi (s , a)} + \frac {2 H ^ {2} \sqrt {E} / \kappa}{K}.
$$
D.3.3 An intermediate result: bounding the variance
Before we handle  $\left\| \sum_{\tau = 1}^{K}x_{\tau}\eta_{\tau}\right\|_{\widehat{\Lambda}_{h}^{-1}}$ , we first bound  $\sup_h\left\| \widetilde{\sigma}_h^2 -\sigma_{\widetilde{V}_{h + 1}}^2\right\|_\infty$  by the following Lemma D.10.
Lemma D.10 (Private version of Lemma C.7 in [Yin et al., 2022]). Recall the definition of  $\widetilde{\sigma}_h(\cdot,\cdot)^2 = \max\{1, \widetilde{\mathrm{Var}}_h \widetilde{V}_{h+1}(\cdot,\cdot)\}$  in Algorithm 2 where  $[\widetilde{\mathrm{Var}}_h \widetilde{V}_{h+1}](\cdot,\cdot) = \langle \phi(\cdot,\cdot), \widetilde{\beta}_h \rangle_{[0,(H-h+1)^2]} - [\langle \phi(\cdot,\cdot), \widetilde{\theta}_h \rangle_{[0,H-h+1]}\]^2$  ( $\widetilde{\beta}_h$  and  $\widetilde{\theta}_h$  are defined in Algorithm 2) and  $\sigma_{\widetilde{V}_{h+1}}(\cdot,\cdot)^2 := \max\{1, \mathrm{Var}_{P_h} \widetilde{V}_{h+1}(\cdot,\cdot)\}$ . Suppose  $K \geq \max\left\{\frac{512 \log\left(\frac{2Hd}{\delta}\right)}{\kappa^2}, \frac{4\lambda}{\kappa}, \frac{128 \log\frac{2dH}{\delta}}{\kappa^2}, \frac{\sqrt{2}L}{H\sqrt{d\kappa}}\right\}$  and  $K \geq \max\left\{\frac{4L^2}{H^2d^3\kappa}, \frac{32E^2}{d^2\kappa^2}, \frac{16\lambda^2}{d^2\kappa}\right\}$ , under the high probability event in Lemma D.2, it holds that with probability  $1-6\delta$ ,
$$
\sup  _ {h} | | \widetilde {\sigma} _ {h} ^ {2} - \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} | | _ {\infty} \leq 3 6 \sqrt {\frac {H ^ {4} d ^ {3}}{\kappa K} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)}.
$$
Proof of Lemma D.10. Step 1: The first step is to show for all  $h, s, a \in [H] \times S \times \mathcal{A}$ , with probability  $1 - 3\delta$ ,
$$
\left| \langle \phi (s, a), \widetilde {\beta} _ {h} \rangle_ {[ 0, (H - h + 1) ^ {2} ]} - \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) ^ {2} (s, a) \right| \leq 1 2 \sqrt {\frac {H ^ {4} d ^ {3}}{\kappa K} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)}.
$$
Proof of Step 1. We can bound the left hand side by the following decomposition:
$$
\begin{array}{l} \left| \left\langle \phi (s, a), \widetilde {\beta} _ {h} \right\rangle_ {[ 0, (H - h + 1) ^ {2} ]} - \mathbb {P} _ {h} \left(\widetilde {V} _ {h + 1}\right) ^ {2} (s, a) \right| \leq \left| \left\langle \phi (s, a), \widetilde {\beta} _ {h} \right\rangle - \mathbb {P} _ {h} \left(\widetilde {V} _ {h + 1}\right) ^ {2} (s, a) \right| \\ = \left| \phi (s, a) ^ {\top} \widetilde {\Sigma} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \cdot \widetilde {V} _ {h + 1} (\bar {s} _ {h + 1} ^ {\tau}) ^ {2} + \phi_ {1}\right) - \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) ^ {2} (s, a) \right| \\ \leq \underbrace {\left| \phi (s , a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi \left(\bar {s} _ {h} ^ {\tau} , \bar {a} _ {h} ^ {\tau}\right) \cdot \tilde {V} _ {h + 1} \left(\bar {s} _ {h + 1} ^ {\tau}\right) ^ {2}\right) - \mathbb {P} _ {h} \left(\tilde {V} _ {h + 1}\right) ^ {2} (s , a) \right|} _ {(1)} + \underbrace {\left| \phi (s , a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \phi_ {1} \right|} _ {(2)} \\ + \underbrace {\left| \phi (s , a) ^ {\top} \left(\widetilde {\Sigma} _ {h} ^ {- 1} - \bar {\Sigma} _ {h} ^ {- 1}\right) \left(\sum_ {\tau = 1} ^ {K} \phi \left(\bar {s} _ {h} ^ {\tau} , \bar {a} _ {h} ^ {\tau}\right) \cdot \widetilde {V} _ {h + 1} \left(\bar {s} _ {h + 1} ^ {\tau}\right) ^ {2} + \phi_ {1}\right) \right|} _ {(3)}, \\ \end{array}
$$
where  $\bar{\Sigma}_h = \widetilde{\Sigma}_h - K_1 = \sum_{\tau=1}^{K} \phi(\bar{s}_h^\tau, \bar{a}_h^\tau) \phi(\bar{s}_h^\tau, \bar{a}_h^\tau)^\top + \lambda I$ .
Similar to the proof in Lemma D.5, when  $K \geq \max \left\{\frac{128\log\frac{2dH}{\delta}}{\kappa^2},\frac{\sqrt{2}L}{H\sqrt{d\kappa}}\right\}$ , it holds that with probability  $1 - \delta$ , for all  $h \in [H]$
$$
\lambda_ {\min } (\bar {\Sigma} _ {h}) \geq \frac {K \kappa}{2}, \lambda_ {\min } (\widetilde {\Sigma} _ {h}) \geq \frac {K \kappa}{2}, \left\| \widetilde {\Sigma} _ {h} ^ {- 1} - \bar {\Sigma} _ {h} ^ {- 1} \right\| _ {2} \leq \frac {4 E / \kappa^ {2}}{K ^ {2}}.
$$
(The only difference to Lemma D.5 is here  $\mathbb{E}_{\mu ,h}[\phi (s,a)\phi (s,a)^{\top}]\geq \kappa .$
Under this high probability event, for term (2), it holds that for all  $h,s,a\in [H]\times S\times \mathcal{A}$
$$
\left| \phi (s, a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \phi_ {1} \right| \leq \| \phi (s, a) \| \cdot \left\| \bar {\Sigma} _ {h} ^ {- 1} \right\| \cdot \| \phi_ {1} \| \leq \lambda_ {\min } ^ {- 1} (\bar {\Sigma} _ {h}) \cdot H L \leq \frac {2 H L / \kappa}{K}. \tag {45}
$$
For term (3), similar to Lemma D.6, we have for all  $h,s,a\in [H]\times \mathcal{S}\times \mathcal{A},$
$$
\left| \phi (s, a) ^ {\top} \left(\widetilde {\Sigma} _ {h} ^ {- 1} - \bar {\Sigma} _ {h} ^ {- 1}\right) \left(\sum_ {\tau = 1} ^ {K} \phi \left(\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}\right) \cdot \widetilde {V} _ {h + 1} \left(\bar {s} _ {h + 1} ^ {\tau}\right) ^ {2} + \phi_ {1}\right) \right| \leq \frac {4 \sqrt {2} H ^ {2} E \sqrt {d} / \kappa^ {3 / 2}}{K}. \tag {46}
$$
(The only difference to Lemma D.6 is that here  $\widetilde{V}_{h + 1}(s)^2\leq H^2,\| \phi_1\| _2\leq HL,\left\| \widetilde{\Sigma}_h^{-1}\right\| _2\leq \frac{2}{K\kappa}$  and
$$
\left\| \widetilde {\Sigma} _ {h} ^ {- 1} - \bar {\Sigma} _ {h} ^ {- 1} \right\| _ {2} \leq \frac {4 E / \kappa^ {2}}{K ^ {2}}.)
$$
We further decompose term (1) as below.
$$
\begin{array}{l} (1) = \left| \phi (s, a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \left(\sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \cdot \widetilde {V} _ {h + 1} (\bar {s} _ {h + 1} ^ {\tau}) ^ {2}\right) - \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) ^ {2} (s, a) \right| \\ = \left| \phi (s, a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \cdot \widetilde {V} _ {h + 1} (\bar {s} _ {h + 1} ^ {\tau}) ^ {2} - \phi (s, a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} (\sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) ^ {\top} + \lambda I) \int_ {\mathcal {S}} (\widetilde {V} _ {h + 1}) ^ {2} (s ^ {\prime}) d \nu_ {h} (s ^ {\prime}) \right| \\ \leq \underbrace {\left| \phi (s , a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau} , \bar {a} _ {h} ^ {\tau}) \cdot \left(\tilde {V} _ {h + 1} (\bar {s} _ {h + 1} ^ {\tau}) ^ {2} - \mathbb {P} _ {h} (\tilde {V} _ {h + 1}) ^ {2} (\bar {s} _ {h} ^ {\tau} , \bar {a} _ {h} ^ {\tau})\right) \right|} _ {(4)} + \underbrace {\lambda \left| \phi (s , a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \int_ {\mathcal {S}} (\tilde {V} _ {h + 1}) ^ {2} \left(s ^ {\prime}\right) d v _ {h} \left(s ^ {\prime}\right) \right|} _ {(5)}. \\ \end{array}
$$
For term (5), because  $K \geq \max \left\{\frac{512\log\left(\frac{2Hd}{\delta}\right)}{\kappa^2},\frac{4\lambda}{\kappa}\right\}$ , by Lemma E.13 and a union bound, with probability  $1 - \delta$ , for all  $h,s,a \in [H] \times \mathcal{S} \times \mathcal{A}$
$$
\begin{array}{l} \lambda \left| \phi (s, a) ^ {\top} \bar {\Sigma} _ {h} ^ {- 1} \int_ {\mathcal {S}} \left(\widetilde {V} _ {h + 1}\right) ^ {2} \left(s ^ {\prime}\right) d \nu_ {h} \left(s ^ {\prime}\right) \right| \leq \lambda \| \phi (s, a) \| _ {\bar {\Sigma} _ {h} ^ {- 1}} \left\| \int_ {\mathcal {S}} \left(\widetilde {V} _ {h + 1}\right) ^ {2} \left(s ^ {\prime}\right) d \nu_ {h} \left(s ^ {\prime}\right) \right\| _ {\bar {\Sigma} _ {h} ^ {- 1}} \\ \leq \lambda \frac {2}{\sqrt {K}} \| \phi (s, a) \| _ {\left(\Sigma_ {h} ^ {p}\right) ^ {- 1}} \frac {2}{\sqrt {K}} \left\| \int_ {\mathcal {S}} \left(\widetilde {V} _ {h + 1}\right) ^ {2} \left(s ^ {\prime}\right) d \nu_ {h} \left(s ^ {\prime}\right) \right\| _ {\left(\Sigma_ {h} ^ {p}\right) ^ {- 1}} \leq 4 \lambda \left\| \left(\Sigma_ {h} ^ {p}\right) ^ {- 1} \right\| \frac {H ^ {2} \sqrt {d}}{K} \leq 4 \lambda \frac {H ^ {2} \sqrt {d}}{\kappa K}, \tag {48} \\ \end{array}
$$
where  $\Sigma_h^p = \mathbb{E}_{\mu,h}[\phi(s,a)\phi(s,a)^\top]$  and  $\lambda_{\min}(\Sigma_h^p) \geq \kappa$ .
For term (4), it can be bounded by the following inequality (because of Cauchy-Schwarz inequality).
$$
(4) \leq \| \phi (s, a) \| _ {\bar {\Sigma} _ {h} ^ {- 1}} \cdot \left\| \sum_ {\tau = 1} ^ {K} \phi \left(\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}\right) \cdot \left(\widetilde {V} _ {h + 1} \left(\bar {s} _ {h + 1} ^ {\tau}\right) ^ {2} - \mathbb {P} _ {h} \left(\widetilde {V} _ {h + 1}\right) ^ {2} \left(\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}\right)\right) \right\| _ {\bar {\Sigma} _ {h} ^ {- 1}}. \tag {49}
$$
Bounding using covering. Note for any fix  $V_{h + 1}$ , we can define  $x_{\tau} = \phi (\bar{s}_h^\tau ,\bar{a}_h^\tau)$ $(\| \phi \| _2\leq 1)$  and  $\eta_{\tau} = V_{h + 1}(\bar{s}_{h + 1}^{\tau})^{2} - \mathbb{P}_{h}(V_{h + 1})^{2}(\bar{s}_{h}^{\tau},\bar{a}_{h}^{\tau})$  is  $H^2$ -subgaussian, by Lemma E.9 (where  $t = K$  and  $L = 1$ ), it holds that with probability  $1 - \delta$ ,
$$
\left\| \sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \cdot \big (V _ {h + 1} (\bar {s} _ {h + 1} ^ {\tau}) ^ {2} - \mathbb {P} _ {h} (V _ {h + 1}) ^ {2} (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \big) \right\| _ {\bar {\Sigma} _ {h} ^ {- 1}} \leq \sqrt {8 H ^ {4} \cdot \frac {d}{2} \log \left(\frac {\lambda + K}{\lambda \delta}\right)}.
$$
Let  $\mathcal{N}_h(\epsilon)$  be the minimal  $\epsilon$ -cover (with respect to the supremum norm) of
$$
\mathcal {V} _ {h} := \left\{V _ {h}: V _ {h} (\cdot) = \max  _ {a \in \mathcal {A}} \{\min  \{\phi (s, a) ^ {\top} \theta - C _ {1} \sqrt {d \cdot \phi (\cdot , \cdot) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (\cdot , \cdot)} - C _ {2}, H - h + 1 \} ^ {+} \} \right\}.
$$
That is, for any  $V \in \mathcal{V}_h$ , there exists a value function  $V' \in \mathcal{N}_h(\epsilon)$  such that  $\sup_{s \in \mathcal{S}} |V(s) - V'(s)| < \epsilon$ . Now by a union bound, we obtain with probability  $1 - \delta$ ,
$$
\sup  _ {V _ {h + 1} \in \mathcal {N} _ {h + 1} (\epsilon)} \left\| \sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \cdot \left(V _ {h + 1} (\bar {s} _ {h + 1} ^ {\tau}) ^ {2} - \mathbb {P} _ {h} (V _ {h + 1}) ^ {2} (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau})\right) \right\| _ {\bar {\Sigma} _ {h} ^ {- 1}} \leq \sqrt {8 H ^ {4} \cdot \frac {d}{2} \log \left(\frac {\lambda + K}{\lambda \delta} | \mathcal {N} _ {h + 1} (\epsilon) |\right)}
$$
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {K} \phi (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}) \cdot \left(\widetilde {V} _ {h + 1} (\bar {s} _ {h + 1} ^ {\tau}) ^ {2} - \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) ^ {2} (\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau})\right) \right\| _ {\bar {\Sigma} _ {h} ^ {- 1}} \\ \leq \sqrt {8 H ^ {4} \cdot \frac {d}{2} \log \left(\frac {\lambda + K}{\lambda \delta} | \mathcal {N} _ {h + 1} (\epsilon) |\right)} + 4 H ^ {2} \sqrt {\epsilon^ {2} K ^ {2} / \lambda} \\ \end{array}
$$
choosing  $\epsilon = d\sqrt{\lambda} /K$ , applying Lemma B.3 of [Jin et al., 2021] $^{13}$  to the covering number  $\mathcal{N}_{h + 1}(\epsilon)$  w.r.t.  $\mathcal{V}_{h + 1}$ , we can further bound above by
$$
\leq \sqrt {8 H ^ {4} \cdot \frac {d ^ {3}}{2} \log \left(\frac {\lambda + K}{\lambda \delta} 2 d H K\right)} + 4 H ^ {2} \sqrt {d ^ {2}} \leq 6 \sqrt {H ^ {4} \cdot d ^ {3} \log \left(\frac {\lambda + K}{\lambda \delta} 2 d H K\right)}
$$
Apply a union bound for  $h \in [H]$ , we have with probability  $1 - \delta$ , for all  $h \in [H]$
$$
\left\| \sum_ {\tau = 1} ^ {K} \phi \left(\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}\right) \cdot \left(\widetilde {V} _ {h + 1} \left(\bar {s} _ {h + 1} ^ {\tau}\right) ^ {2} - \mathbb {P} _ {h} \left(\widetilde {V} _ {h + 1}\right) ^ {2} \left(\bar {s} _ {h} ^ {\tau}, \bar {a} _ {h} ^ {\tau}\right)\right) \right\| _ {\bar {\Sigma} _ {h} ^ {- 1}} \leq 6 \sqrt {H ^ {4} d ^ {3} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)} \tag {50}
$$
and similar to term (2), it holds that for all  $h, s, a \in [H] \times S \times \mathcal{A}$
$$
\left\| \phi (s, a) \right\| _ {\bar {\Sigma} _ {h} ^ {- 1}} \leq \sqrt {\left\| \bar {\Sigma} _ {h} ^ {- 1} \right\|} \leq \sqrt {\frac {2}{\kappa K}}. \tag {51}
$$
Combining (45), (46), (47), (48), (49), (50), (51) and the assumption that  $K \geq \max \left\{\frac{4L^2}{H^2d^3\kappa},\frac{32E^2}{d^2\kappa^2},\frac{16\lambda^2}{d^2\kappa}\right\}$ , we obtain with probability  $1 - 3\delta$  for all  $h,s,a\in [H]\times S\times \mathcal{A}$
$$
\left| \langle \phi (s, a), \widetilde {\beta_ {h}} \rangle_ {[ 0, (H - h + 1) ^ {2} ]} - \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) ^ {2} (s, a) \right| \leq 1 2 \sqrt {\frac {H ^ {4} d ^ {3}}{\kappa K} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)}.
$$
Step 2: The second step is to show for all  $h, s, a \in [H] \times \mathcal{S} \times \mathcal{A}$ , with probability  $1 - 3\delta$
$$
\left| \langle \phi (s, a), \widetilde {\theta} _ {h} \rangle_ {[ 0, H - h + 1 ]} - \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) (s, a) \right| \leq 1 2 \sqrt {\frac {H ^ {2} d ^ {3}}{\kappa K} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)}. \tag {52}
$$
The proof of Step 2 is nearly identical to Step 1 except  $\widetilde{V}_h^2$  is replaced by  $\widetilde{V}_h$ .
Step 3: The last step is to prove  $\sup_h\| \widetilde{\sigma}_h^2 -\sigma_{\widetilde{V}_{h + 1}}^2\|_\infty \leq 36\sqrt{\frac{H^4d^3}{\kappa K}\log\left(\frac{(\lambda + K)2KdH^2}{\lambda\delta}\right)}$  with high probability.
Proof of Step 3. By (52),
$$
\begin{array}{l} \left| \left[ \left\langle \phi (\cdot , \cdot), \widetilde {\theta} _ {h} \right\rangle_ {[ 0, H - h + 1 ]} \right] ^ {2} - \left[ \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) (s, a) \right] ^ {2} \right| \\ = \left| \langle \phi (s, a), \widetilde {\theta} _ {h} \rangle_ {[ 0, H - h + 1 ]} + \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) (s, a) \right| \cdot \left| \langle \phi (s, a), \widetilde {\theta} _ {h} \rangle_ {[ 0, H - h + 1 ]} - \mathbb {P} _ {h} (\widetilde {V} _ {h + 1}) (s, a) \right| \\ \leq 2H\cdot \left|\langle \phi (s,a),\widetilde{\theta}_{h}\rangle_{[0,H - h + 1]} - \mathbb{P}_{h}(\widetilde{V}_{h + 1})(s,a)\right|\leq 24\sqrt{\frac{H^{4}d^{3}}{\kappa K}\log\left(\frac{(\lambda + K)2KdH^{2}}{\lambda\delta}\right)}. \\ \end{array}
$$
Combining this with Step 1, we have with probability  $1 - 6\delta, \forall h, s, a \in [H] \times S \times \mathcal{A}$ ,
$$
\left| \widetilde {\operatorname {V a r}} _ {h} \widetilde {V} _ {h + 1} (s, a) - \operatorname {V a r} _ {P _ {h}} \widetilde {V} _ {h + 1} (s, a) \right| \leq 3 6 \sqrt {\frac {H ^ {4} d ^ {3}}{\kappa K} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)}.
$$
Finally, by the non-expansiveness of operator  $\max \{1,\cdot \}$ , the proof is complete.
<sup>13</sup>Note that the conclusion in [Jin et al., 2021] hold here even though we have an extra constant  $C_2$ .
D.3.4 Validity of our pessimism
Recall the definition  $\widehat{\Lambda}_h = \sum_{\tau=1}^{K} \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^\top / \widetilde{\sigma}_h^2(s_h^\tau, a_h^\tau) + \lambda \cdot I$  and
$\Lambda_{h} = \sum_{\tau = 1}^{K}\phi (s_{h}^{\tau},a_{h}^{\tau})\phi (s_{h}^{\tau},a_{h}^{\tau})^{\top} / \sigma_{\widetilde{V}_{h + 1}}^{2}(s_{h}^{\tau},a_{h}^{\tau}) + \lambda I.$  Then we have the following lemma to bound
the term  $\sqrt{\phi(s,a)^{\top}\widehat{\Lambda}_{h}^{-1}\phi(s,a)}$  by  $\sqrt{\phi(s,a)^{\top}\Lambda_{h}^{-1}\phi(s,a)}$ .
Lemma D.11 (Private version of lemma C.3 in [Yin et al., 2022]). Denote the quantities  $C_1 = \max\{2\lambda, 128\log(2dH/\delta), \frac{128H^4\log(2dH/\delta)}{\kappa^2}\}$  and  $C_2 = \widetilde{O}(H^{12}d^3/\kappa^5)$ . Suppose the number of episode  $K$  satisfies  $K > \max\{C_1, C_2\}$  and the condition in Lemma D.10, under the high probability events in Lemma D.2 and Lemma D.10, it holds that with probability  $1 - 2\delta$ , for all  $h, s, a \in [H] \times S \times \mathcal{A}$ ,
$$
\sqrt {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi (s , a)} \leq 2 \sqrt {\phi (s , a) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (s , a)}.
$$
Proof of Lemma D.11. By definition  $\sqrt{\phi(s,a)^{\top}\widehat{\Lambda}_h^{-1}\phi(s,a)} = \| \phi (s,a)\|_{\widehat{\Lambda}_h^{-1}}$  . Then denote