SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {\Lambda} _ {h} ^ {\prime} = \frac {1}{K} \widehat {\Lambda} _ {h}, \quad \Lambda_ {h} ^ {\prime} = \frac {1}{K} \Lambda_ {h},
$$
where  $\Lambda_h = \sum_{\tau=1}^{K} \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^\top / \sigma_{\tilde{V}_{h+1}}^2(s_h^\tau, a_h^\tau) + \lambda I$ . Under the assumption of  $K$ , by the conclusion in Lemma D.10, we have
$$
\begin{array}{l} \left\| \widehat {\Lambda} _ {h} ^ {\prime} - \Lambda_ {h} ^ {\prime} \right\| \leq \sup  _ {s, a} \left\| \frac {\phi (s , a) \phi (s , a) ^ {\top}}{\widetilde {\sigma} _ {h} ^ {2} (s , a)} - \frac {\phi (s , a) \phi (s , a) ^ {\top}}{\sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s , a)} \right\| \\ \leq \sup  _ {s, a} \left| \frac {\widetilde {\sigma} _ {h} ^ {2} (s , a) - \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s , a)}{\widetilde {\sigma} _ {h} ^ {2} (s , a) \cdot \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s , a)} \right| \cdot \| \phi (s, a) \| ^ {2} \tag {53} \\ \leq \sup  _ {s, a} \left| \frac {\widetilde {\sigma} _ {h} ^ {2} (s , a) - \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s , a)}{1} \right| \cdot 1 \\ \leq 3 6 \sqrt {\frac {H ^ {4} d ^ {3}}{\kappa K} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)}. \\ \end{array}
$$
Next by Lemma E.12 (with  $\phi$  to be  $\phi / \sigma_{\widetilde{V}_{h + 1}}$  and therefore  $C = 1$ ) and a union bound, it holds with probability  $1 - \delta$ , for all  $h \in [H]$ ,
$$
\left\| \Lambda_ {h} ^ {\prime} - \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s, a) ] + \frac {\lambda}{K} I _ {d}\right) \right\| \leq \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2}.
$$
Therefore by Weyl's inequality and the assumption that  $K$  satisfies that
$K > \max \{2\lambda, 128\log(2dH/\delta), \frac{128H^4\log(2dH/\delta)}{\kappa^2}\}$ , the above inequality leads to
$$
\begin{array}{l} \left\| \Lambda_ {h} ^ {\prime} \right\| = \lambda_ {\max } \left(\Lambda_ {h} ^ {\prime}\right) \leq \lambda_ {\max } \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {\tilde {V} _ {h + 1}} ^ {2} (s, a) ]\right) + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ = \left\| \mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {\tilde {V} _ {h + 1}} ^ {2} (s, a) ] \right\| _ {2} + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ \leq \| \phi (s, a) \| ^ {2} + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \leq 1 + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \leq 2, \\ \end{array}
$$
$$
\begin{array}{l} \lambda_ {\min } \left(\Lambda_ {h} ^ {\prime}\right) \geq \lambda_ {\min } \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s, a) ]\right) + \frac {\lambda}{K} - \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ \geq \lambda_ {\min } \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {\tilde {V} _ {h + 1}} ^ {2} (s, a) ]\right) - \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ \geq \frac {\kappa}{H ^ {2}} - \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \geq \frac {\kappa}{2 H ^ {2}}. \\ \end{array}
$$
Hence with probability  $1 - \delta$ ,  $\| \Lambda_h' \| \leq 2$  and  $\| \Lambda_h'^{-1} \| = \lambda_{\min}^{-1}(\Lambda_h') \leq \frac{2H^2}{\kappa}$ . Similarly, one can show  $\left\| \widehat{\Lambda}_h'^{-1} \right\| \leq \frac{2H^2}{\kappa}$  with probability  $1 - \delta$  using identical proof.
Now apply Lemma E.11 and a union bound to  $\widehat{\Lambda}_h^\prime$  and  $\Lambda_h^\prime$ , we obtain with probability  $1 - \delta$ , for all  $h, s, a \in [H] \times \mathcal{S} \times \mathcal{A}$
$$
\begin{array}{l} \| \phi (s, a) \| _ {\widehat {\Lambda} _ {h} ^ {\prime - 1}} \leq \left[ 1 + \sqrt {\left\| \Lambda_ {h} ^ {\prime - 1} \right\| \cdot \left\| \Lambda_ {h} ^ {\prime} \right\| \cdot \left\| \widehat {\Lambda} _ {h} ^ {\prime - 1} \right\| \cdot \left\| \widehat {\Lambda} _ {h} ^ {\prime} - \Lambda_ {h} ^ {\prime} \right\|} \right] \cdot \| \phi (s, a) \| _ {\Lambda_ {h} ^ {\prime - 1}} \\ \leq \left[ 1 + \sqrt {\frac {2 H ^ {2}}{\kappa} \cdot 2 \cdot \frac {2 H ^ {2}}{\kappa} \cdot \left\| \widehat {\Lambda} _ {h} ^ {\prime} - \Lambda_ {h} ^ {\prime} \right\|} \right] \cdot \| \phi (s, a) \| _ {\Lambda_ {h} ^ {\prime - 1}} \\ \leq \left[ 1 + \sqrt {\frac {2 8 8 H ^ {4}}{\kappa^ {2}} \left(\sqrt {\frac {H ^ {4} d ^ {3}}{\kappa K} \log \left(\frac {(\lambda + K) 2 K d H ^ {2}}{\lambda \delta}\right)}\right)} \right] \cdot \| \phi (s, a) \| _ {\Lambda_ {h} ^ {\prime - 1}} \\ \leq 2 \left\| \phi (s, a) \right\| _ {\Lambda_ {h} ^ {\prime - 1}} \\ \end{array}
$$
where the third inequality uses (53) and the last inequality uses  $K > \widetilde{O}(H^{12}d^3/\kappa^5)$ . Note the conclusion can be derived directly by the above inequality multiplying  $1/\sqrt{K}$  on both sides.
In order to bound  $\left\| \sum_{\tau = 1}^{K}x_{\tau}\eta_{\tau}\right\|_{\widehat{\Lambda}_{h}^{-1}}$ , we apply the following Lemma D.12.
Lemma D.12 (Lemma C.4 in [Yin et al., 2022]). Recall  $x_{\tau} = \frac{\phi(s_h^\tau, a_h^\tau)}{\widetilde{\sigma}_h(s_h^\tau, a_h^\tau)}$  and
$$
\eta_ {\tau} = \left(r _ {h} ^ {\tau} + \widetilde {V} _ {h + 1} \left(s _ {h + 1} ^ {\tau}\right) - \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) / \widetilde {\sigma} _ {h} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right). D e n o t e
$$
$$
\xi := \sup  _ {V \in [ 0, H ], s ^ {\prime} \sim P _ {h} (s, a), h \in [ H ]} \left| \frac {r _ {h} + V \left(s ^ {\prime}\right) - \left(\mathcal {T} _ {h} V\right) (s , a)}{\sigma_ {V} (s , a)} \right|.
$$
Suppose  $K \geq \widetilde{O}\left(H^{12}d^3 /\kappa^5\right)^{14}$ , then with probability  $1 - \delta$
$$
\left\| \sum_ {\tau = 1} ^ {K} x _ {\tau} \eta_ {\tau} \right\| _ {\widehat {\Lambda} _ {h} ^ {- 1}} \leq \widetilde {O} \left(\max  \left\{\sqrt {d}, \xi \right\}\right),
$$
where  $\widetilde{O}$  absorbs constants and Polylog terms.
Now we are ready to prove the following key lemma, which gives a high probability bound for  $\left|\left(\mathcal{T}_h\widetilde{V}_{h + 1} - \widetilde{\mathcal{T}}_h\widetilde{V}_{h + 1}\right)(s,a)\right|$ .
Lemma D.13. Assume  $K > \max \{\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3, \mathcal{M}_4\}$ , for any  $0 < \lambda < \kappa$ , suppose  $\sqrt{d} > \xi$ , where  $\xi := \sup_{V \in [0,H], s' \sim P_h(s,a), h \in [H]}\left|\frac{r_h + V(s') - (\mathcal{T}_hV)(s,a)}{\sigma_V(s,a)}\right|$ . Then with probability  $1 - \delta$ , for all  $h, s, a \in [H] \times \mathcal{S} \times \mathcal{A}$
$$
\left| \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1} - \widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) \right| \leq \widetilde {O} \left(\sqrt {d} \sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a)}\right) + \frac {D}{K},
$$
where  $\widetilde{\Lambda}_h = \sum_{\tau=1}^K \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^\top / \widetilde{\sigma}_h^2(s_h^\tau, a_h^\tau) + \lambda I + K_2$ ,
$$
D = \widetilde {O} \left(\frac {H ^ {2} L}{\kappa} + \frac {H ^ {4} E \sqrt {d}}{\kappa^ {3 / 2}} + H ^ {3} \sqrt {d} + \frac {H ^ {2} \sqrt {E d}}{\kappa}\right) = \widetilde {O} \left(\frac {H ^ {2} L}{\kappa} + \frac {H ^ {4} E \sqrt {d}}{\kappa^ {3 / 2}} + H ^ {3} \sqrt {d}\right)
$$
and  $\widetilde{O}$  absorbs constants and Polylog terms.
<sup>14</sup>Note that here the assumption is stronger than the assumption in [Yin et al., 2022], therefore the conclusion of Lemma C.4 holds.
Proof of Lemma D.13. The proof is by combining (40), (42), Lemma D.4, Lemma D.6, Lemma D.7, Lemma D.8, Lemma D.12 and a union bound.  $\square$
Remark D.14. Under the same assumption of Lemma D.13, because of Remark D.9 and Lemma D.11, we have with probability  $1 - \delta$ , for all  $h, s, a \in [H] \times S \times \mathcal{A}$
$$
\begin{array}{l} \left| \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1} - \widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) \right| \leq \widetilde {O} \left(\sqrt {d} \sqrt {\phi (s , a) ^ {\top} \widetilde {\Lambda} _ {h} ^ {- 1} \phi (s , a)}\right) + \frac {D}{K} \\ \leq \widetilde {O} \left(\sqrt {d} \sqrt {\phi (s , a) ^ {\top} \widehat {\Lambda} _ {h} ^ {- 1} \phi (s , a)}\right) + \frac {2 D}{K} \tag {54} \\ \leq \widetilde {O} \left(2 \sqrt {d} \sqrt {\phi (s , a) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (s , a)}\right) + \frac {2 D}{K}. \\ \end{array}
$$
Because  $D = \widetilde{O}\left(\frac{H^2L}{\kappa} + \frac{H^4E\sqrt{d}}{\kappa^{3/2}} + H^3\sqrt{d}\right)$  and  $\widetilde{O}$  absorbs constant, we will write as below for simplicity:
$$
\left| \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1} - \widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) \right| \leq \widetilde {O} \left(\sqrt {d} \sqrt {\phi (s , a) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (s , a)}\right) + \frac {D}{K}. \tag {55}
$$
D.3.5 Finalize the proof of the first part
We are ready to prove the first part of Theorem 4.1.
Theorem D.15 (First part of Theorem 4.1). Let  $K$  be the number of episodes. Suppose  $\sqrt{d} > \xi$ , where  $\xi := \sup_{V \in [0, H], s' \sim P_h(s, a), h \in [H]} \left| \frac{r_h + V(s') - (\mathcal{T}_h V)(s, a)}{\sigma_V(s, a)} \right|$  and  $K > \max \{\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3, \mathcal{M}_4\}$ . Then for any  $0 < \lambda < \kappa$ , with probability  $1 - \delta$ , for all policy  $\pi$  simultaneously, the output  $\widehat{\pi}$  of Algorithm 2 satisfies
$$
v ^ {\pi} - v ^ {\widehat {\pi}} \leq \widetilde {O} \left(\sqrt {d} \cdot \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \left(\phi (\cdot , \cdot) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (\cdot , \cdot)\right) ^ {1 / 2} \right]\right) + \frac {D H}{K},
$$
where  $\Lambda_{h} = \sum_{\tau = 1}^{K}\frac{\phi(s_{h}^{\top},a_{h}^{\top})\cdot\phi(s_{h}^{\top},a_{h}^{\top})^{\top}}{\sigma_{V_{h + 1}(s_{h}^{\top},a_{h}^{\top})}^{2}} +\lambda I_{d}$ ,  $D = \widetilde{O}\left(\frac{H^{2}L}{\kappa} +\frac{H^{4}E\sqrt{d}}{\kappa^{3 / 2}} +H^{3}\sqrt{d}\right)$  and  $\widetilde{O}$  absorbs constants and Polylog terms.
Proof of Theorem D.15. Combining Lemma D.3 and Remark D.14, we have with probability  $1 - \delta$ , for all policy  $\pi$  simultaneously,
$$
V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) \leq \widetilde {O} \left(\sqrt {d} \cdot \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \left(\phi (\cdot , \cdot) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (\cdot , \cdot)\right) ^ {1 / 2} \mid s _ {1} = s \right]\right) + \frac {D H}{K}, \tag {56}
$$
now the proof is complete by taking the initial distribution  $d_{1}$  on both sides.
D.3.6 Finalize the proof of the second part
To prove the second part of Theorem 4.1, we begin with a crude bound on  $\sup_h\left\| V_h^\star -\widetilde{V}_h\right\|_\infty$
Lemma D.16 (Private version of Lemma C.8 in [Yin et al., 2022]). Suppose  $K \geq \max \{\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3, \mathcal{M}_4\}$ , under the high probability event in Lemma D.13, with probability at least  $1 - \delta$
$$
\sup  _ {h} \left\| V _ {h} ^ {\star} - \widetilde {V} _ {h} \right\| _ {\infty} \leq \widetilde {O} \left(\frac {H ^ {2} \sqrt {d}}{\sqrt {\kappa K}}\right).
$$
Proof of Lemma D.16. Step 1: The first step is to show with probability at least  $1 - \delta$ ,  $\sup_h\left\| V_h^\star -V_h^{\widehat{\pi}}\right\|_\infty \leq \widetilde{O}\left(\frac{H^2\sqrt{d}}{\sqrt{\kappa K}}\right)$ .
Indeed, combine Lemma D.3 and Lemma D.13, similar to the proof of Theorem D.15, we directly have with probability  $1 - \delta$ , for all policy  $\pi$  simultaneously, and for all  $s \in S$ ,  $h \in [H]$ ,
$$
V _ {h} ^ {\pi} (s) - V _ {h} ^ {\widehat {\pi}} (s) \leq \widetilde {O} \left(\sqrt {d} \cdot \sum_ {t = h} ^ {H} \mathbb {E} _ {\pi} \left[ \left(\phi (\cdot , \cdot) ^ {\top} \Lambda_ {t} ^ {- 1} \phi (\cdot , \cdot)\right) ^ {1 / 2} \mid s _ {h} = s \right]\right) + \frac {D H}{K}, \tag {57}
$$
Next, since  $K \geq \max \left\{\frac{512\log\left(\frac{2Hd}{\delta}\right)}{\kappa^2},\frac{4\lambda}{\kappa}\right\}$ , by Lemma E.13 and a union bound over  $h \in [H]$ , with probability  $1 - \delta$
$$
\sup  _ {s, a} \| \phi (s, a) \| _ {\Lambda_ {h} ^ {- 1}} \leq \frac {2}{\sqrt {K}} \sup  _ {s, a} \| \phi (s, a) \| _ {(\Lambda_ {h} ^ {p}) ^ {- 1}} \leq \frac {2}{\sqrt {K}} \sqrt {\lambda_ {\min } ^ {- 1} (\Lambda_ {h} ^ {p})} \leq \frac {2 H}{\sqrt {\kappa K}}, \forall h \in [ H ],
$$
where  $\Lambda_h^p = \mathbb{E}_{\mu,h}[\sigma_{\tilde{V}_{h+1}}^{-2}(s,a)\phi(s,a)\phi(s,a)^\top]$  and  $\lambda_{\min}(\Lambda_h^p) \geq \frac{\kappa}{H^2}$ .
Lastly, taking  $\pi = \pi^{\star}$  in (57) to obtain
$$
\begin{array}{l} 0 \leq V _ {h} ^ {\pi^ {\star}} (s) - V _ {h} ^ {\widehat {\pi}} (s) \leq \widetilde {O} \left(\sqrt {d} \cdot \sum_ {t = h} ^ {H} \mathbb {E} _ {\pi^ {\star}} \left[ \left(\phi (\cdot , \cdot) ^ {\top} \Lambda_ {t} ^ {- 1} \phi (\cdot , \cdot)\right) ^ {1 / 2} \mid s _ {h} = s \right]\right) + \frac {D H}{K} \tag {58} \\ \leq \widetilde {O} \left(\frac {H ^ {2} \sqrt {d}}{\sqrt {\kappa K}}\right) + \widetilde {O} \left(\frac {H ^ {3} L / \kappa}{K} + \frac {H ^ {5} E \sqrt {d} / \kappa^ {3 / 2}}{K} + \frac {H ^ {4} \sqrt {d}}{K}\right). \\ \end{array}
$$
This implies by using the condition  $K > \max \left\{\frac{H^2L^2}{d\kappa},\frac{H^6E^2}{\kappa^2},H^4\kappa \right\}$ , we finish the proof of Step 1.
Step 2: The second step is to show with probability  $1 - \delta$ ,  $\sup_h\left\| \widetilde{V}_h - V_h^{\widehat{\pi}}\right\|_{\infty}\leq \widetilde{O}\left(\frac{H^2\sqrt{d}}{\sqrt{\kappa K}}\right)$ .
Indeed, applying Lemma E.7 with  $\pi = \pi' = \widehat{\pi}$ , then with probability  $1 - \delta$ , for all  $s, h$
$$
\begin{array}{l} \left| \widetilde {V} _ {h} (s) - V _ {h} ^ {\widehat {\pi}} (s) \right| = \left| \sum_ {t = h} ^ {H} \mathbb {E} _ {\widehat {\pi}} \left[ \widehat {Q} _ {h} \left(s _ {h}, a _ {h}\right) - \left(\mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) \right\rvert   s _ {h} = s \right] \\ \leq \sum_ {t = h} ^ {H} \left\| \left(\widetilde {\mathcal {T}} _ {h} \widetilde {V} _ {h + 1} - \mathcal {T} _ {h} \widetilde {V} _ {h + 1}\right) (s, a) \right\| _ {\infty} + H \cdot \| \Gamma_ {h} (s, a) \| _ {\infty} \\ \leq \widetilde {O} \left(H \sqrt {d} \left\| \sqrt {\phi (s , a) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (s , a)} \right\| _ {\infty}\right) + \widetilde {O} \left(\frac {D H}{K}\right) \\ \leq \widetilde {O} \left(\frac {H ^ {2} \sqrt {d}}{\sqrt {\kappa K}}\right), \\ \end{array}
$$
where the second inequality uses Lemma D.13, Remark D.14 and the last inequality holds due to the same reason as Step 1.
Step 3: The proof of the lemma is complete by combining Step 1, Step 2, triangular inequality and a union bound.
Then we can give a high probability bound of  $\sup_h\| \sigma_{\tilde{V}_{h + 1}}^2 -\sigma_h^{\star 2}\|_{\infty}$
Lemma D.17 (Private version of Lemma C.10 in [Yin et al., 2022]). Recall  $\sigma_{\widetilde{V}_{h + 1}}^2 = \max \left\{1, \operatorname{Var}_{P_h} \widetilde{V}_{h + 1}\right\}$  and  $\sigma_h^{\star 2} = \max \left\{1, \operatorname{Var}_{P_h} V_{h + 1}^{\star}\right\}$ . Suppose  $K \geq \max \{\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3, \mathcal{M}_4\}$ , then with probability  $1 - \delta$ ,
$$
\sup  _ {h} \left\| \sigma_ {\tilde {V} _ {h + 1}} ^ {2} - \sigma_ {h} ^ {\star 2} \right\| _ {\infty} \leq \widetilde {O} \left(\frac {H ^ {3} \sqrt {d}}{\sqrt {\kappa K}}\right).
$$
Proof of Lemma D.17. By definition and the non-expansiveness of  $\max\{1, \cdot\}$ , we have
$$
\begin{array}{l} \left\| \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} - \sigma_ {h} ^ {\star 2} \right\| _ {\infty} \leq \left\| \operatorname {V a r} \widetilde {V} _ {h + 1} - \operatorname {V a r} V _ {h + 1} ^ {\star} \right\| _ {\infty} \\ \leq \left\| \mathbb {P} _ {h} \left(\widetilde {V} _ {h + 1} ^ {2} - V _ {h + 1} ^ {\star 2}\right) \right\| _ {\infty} + \left\| \left(\mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) ^ {2} - \left(\mathbb {P} _ {h} V _ {h + 1} ^ {\star}\right) ^ {2} \right\| _ {\infty} \\ \leq \left\| \widetilde {V} _ {h + 1} ^ {2} - V _ {h + 1} ^ {\star 2} \right\| _ {\infty} + \left\| \left(\mathbb {P} _ {h} \widetilde {V} _ {h + 1} + \mathbb {P} _ {h} V _ {h + 1} ^ {\star}\right) \left(\mathbb {P} _ {h} \widetilde {V} _ {h + 1} - \mathbb {P} _ {h} V _ {h + 1} ^ {\star}\right) \right\| _ {\infty} \\ \leq 2 H \left\| \widetilde {V} _ {h + 1} - V _ {h + 1} ^ {\star} \right\| _ {\infty} + 2 H \left\| \mathbb {P} _ {h} \widetilde {V} _ {h + 1} - \mathbb {P} _ {h} V _ {h + 1} ^ {\star} \right\| _ {\infty} \\ \leq \widetilde {O} \left(\frac {H ^ {3} \sqrt {d}}{\sqrt {\kappa K}}\right). \\ \end{array}
$$
The second inequality is because of the definition of variance. The last inequality comes from Lemma D.16.
We transfer  $\sqrt{\phi(s,a)^{\top}\Lambda_{h}^{-1}\phi(s,a)}$  to  $\sqrt{\phi(s,a)^{\top}\Lambda_{h}^{\star - 1}\phi(s,a)}$  by the following Lemma D.18.
Lemma D.18 (Private version of Lemma C.11 in [Yin et al., 2022]). Suppose  $K \geq \max\{\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3, \mathcal{M}_4\}$ , then with probability  $1 - \delta$ ,
$$
\sqrt {\phi (s , a) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (s , a)} \leq 2 \sqrt {\phi (s , a) ^ {\top} \Lambda_ {h} ^ {\star - 1} \phi (s , a)}, \quad \forall h, s, a \in [ H ] \times \mathcal {S} \times \mathcal {A},
$$
Proof of Lemma D.18. By definition  $\sqrt{\phi(s, a)^{\top} \Lambda_h^{-1} \phi(s, a)} = \|\phi(s, a)\|_{\Lambda_h^{-1}}$ . Then denote
$$
\Lambda_ {h} ^ {\prime} = \frac {1}{K} \Lambda_ {h}, \quad \Lambda_ {h} ^ {\star^ {\prime}} = \frac {1}{K} \Lambda_ {h} ^ {\star},
$$
where  $\Lambda_h^\star = \sum_{\tau=1}^K \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^\top / \sigma_{V_{h+1}^\star}^2(s_h^\tau, a_h^\tau) + \lambda I$ . Under the condition of  $K$ , by Lemma D.17, with probability  $1 - \delta$ , for all  $h \in [H]$ ,
$$
\begin{array}{l} \left\| \Lambda_ {h} ^ {\star^ {\prime}} - \Lambda_ {h} ^ {\prime} \right\| \leq \sup  _ {s, a} \left\| \frac {\phi (s , a) \phi (s , a) ^ {\top}}{\sigma_ {h} ^ {\star 2} (s , a)} - \frac {\phi (s , a) \phi (s , a) ^ {\top}}{\sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s , a)} \right\| \\ \leq \sup  _ {s, a} \left| \frac {\sigma_ {h} ^ {\star 2} (s , a) - \sigma_ {\tilde {V} _ {h + 1}} ^ {2} (s , a)}{\sigma_ {h} ^ {\star 2} (s , a) \cdot \sigma_ {\tilde {V} _ {h + 1}} ^ {2} (s , a)} \right| \cdot \| \phi (s, a) \| ^ {2} \tag {59} \\ \leq \sup _ {s, a} \left| \frac {\sigma_ {h} ^ {\star 2} (s , a) - \sigma_ {\widetilde {V} _ {h + 1}} ^ {2} (s , a)}{1} \right| \cdot 1 \\ \leq \widetilde {O} \left(\frac {H ^ {3} \sqrt {d}}{\sqrt {\kappa K}}\right). \\ \end{array}
$$
Next by Lemma E.12 (with  $\phi$  to be  $\phi / \sigma_{V_{h + 1}^{\star}}$  and  $C = 1$ ), it holds with probability  $1 - \delta$ ,
$$
\left\| \Lambda_ {h} ^ {\star^ {\prime}} - \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {V _ {h + 1} ^ {\star}} ^ {2} (s, a) ] + \frac {\lambda}{K} I _ {d}\right) \right\| \leq \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2}.
$$
Therefore by Weyl's inequality and the condition  $K > \max \{2\lambda, 128\log\left(\frac{2dH}{\delta}\right), \frac{128H^4\log(2dH / \delta)}{\kappa^2}\}$ , the above inequality implies
$$
\begin{array}{l} \left\| \Lambda_ {h} ^ {\star^ {\prime}} \right\| = \lambda_ {\max } \left(\Lambda_ {h} ^ {\star^ {\prime}}\right) \leq \lambda_ {\max } \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {V _ {h + 1} ^ {\star}} ^ {2} (s, a) ]\right) + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ \leq \left\| \mathbb {E} _ {\mu , h} \left[ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {V _ {h + 1} ^ {\star}} ^ {2} (s, a) \right] \right\| + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ \leq \| \phi (s, a) \| ^ {2} + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \leq 1 + \frac {\lambda}{K} + \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \leq 2, \\ \end{array}
$$
$$
\begin{array}{l} \lambda_ {\min } \left(\Lambda_ {h} ^ {\star^ {\prime}}\right) \geq \lambda_ {\min } \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {V _ {h + 1} ^ {\star}} ^ {2} (s, a) ]\right) + \frac {\lambda}{K} - \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ \geq \lambda_ {\min } \left(\mathbb {E} _ {\mu , h} [ \phi (s, a) \phi (s, a) ^ {\top} / \sigma_ {V _ {h + 1} ^ {*}} ^ {2} (s, a) ]\right) - \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \\ \geq \frac {\kappa}{H ^ {2}} - \frac {4 \sqrt {2}}{\sqrt {K}} \left(\log \frac {2 d H}{\delta}\right) ^ {1 / 2} \geq \frac {\kappa}{2 H ^ {2}}. \\ \end{array}
$$
Hence with probability  $1 - \delta$ ,  $\left\| \Lambda_h^{\star '} \right\| \leq 2$  and  $\left\| \Lambda_h^{\star ' - 1} \right\| = \lambda_{\min}^{-1}(\Lambda_h^{\star '}) \leq \frac{2H^2}{\kappa}$ . Similarly, we can show that  $\left\| \Lambda_h'^{-1} \right\| \leq \frac{2H^2}{\kappa}$  holds with probability  $1 - \delta$  by using identical proof.
Now apply Lemma E.11 and a union bound to  $\Lambda_h^{\star'}$  and  $\Lambda_h'$ , we obtain with probability  $1 - \delta$ , for all  $h, s, a \in [H] \times S \times \mathcal{A}$
$$
\begin{array}{l} \| \phi (s, a) \| _ {\Lambda_ {h} ^ {\prime - 1}} \leq \left[ 1 + \sqrt {\| \Lambda_ {h} ^ {\star^ {\prime} - 1} \| \cdot \| \Lambda_ {h} ^ {\star^ {\prime}} \| \cdot \| \Lambda_ {h} ^ {\prime - 1} \| \cdot \| \Lambda_ {h} ^ {\star^ {\prime}} - \Lambda_ {h} ^ {\prime} \|} \right] \cdot \| \phi (s, a) \| _ {\Lambda_ {h} ^ {\star^ {\prime} - 1}} \\ \leq \left[ 1 + \sqrt {\frac {2 H ^ {2}}{\kappa} \cdot 2 \cdot \frac {2 H ^ {2}}{\kappa} \cdot \left\| \Lambda_ {h} ^ {\star^ {\prime}} - \Lambda_ {h} ^ {\prime} \right\|} \right] \cdot \| \phi (s, a) \| _ {\Lambda_ {h} ^ {\star^ {\prime} - 1}} \\ \leq \left[ 1 + \sqrt {\frac {H ^ {4}}{\kappa^ {2}} \left[ \widetilde {O} \left(\frac {H ^ {3} \sqrt {d}}{\sqrt {\kappa K}}\right) \right]} \right] \cdot \| \phi (s, a) \| _ {\Lambda_ {h} ^ {\star^ {\prime} - 1}} \\ \leq 2 \left\| \phi (s, a) \right\| _ {\Lambda_ {h} ^ {\star^ {\prime} - 1}} \\ \end{array}
$$
where the third inequality uses (59) and the last inequality uses  $K \geq \widetilde{O}(H^{14}d / \kappa^5)$ . The conclusion can be derived directly by the above inequality multiplying  $1 / \sqrt{K}$  on both sides.
Finally, the second part of Theorem 4.1 can be proven by combining Theorem D.15 (with  $\pi = \pi^{\star}$ ) and Lemma D.18.
D.4 Put everything together
Combining Lemma D.1, Theorem D.15, and the discussion above, the proof of Theorem 4.1 is complete.
E Assisting technical lemmas
Lemma E.1 (Multiplicative Chernoff bound [Chernoff et al., 1952]). Let  $X$  be a Binomial random variable with parameter  $p, n$ . For any  $1 \geq \theta > 0$ , we have that
$$
\mathbb {P} [ X <   (1 - \theta) p n ] <   e ^ {- \frac {\theta^ {2} p n}{2}}, \quad a n d \quad \mathbb {P} [ X \geq (1 + \theta) p n ] <   e ^ {- \frac {\theta^ {2} p n}{3}}
$$
Lemma E.2 (Hoeffding's Inequality [Sridharan, 2002]). Let  $x_{1}, \ldots, x_{n}$  be independent bounded random variables such that  $\mathbb{E}[x_i] = 0$  and  $|x_i| \leq \xi_i$  with probability 1. Then for any  $\epsilon > 0$  we have
$$
\mathbb {P} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} \geq \epsilon\right) \leq e ^ {- \frac {2 n ^ {2} \epsilon^ {2}}{\sum_ {i = 1} ^ {n} \xi_ {i} ^ {2}}}.
$$
Lemma E.3 (Bernstein's Inequality). Let  $x_{1}, \ldots, x_{n}$  be independent bounded random variables such that  $\mathbb{E}[x_i] = 0$  and  $|x_i| \leq \xi$  with probability 1. Let  $\sigma^2 = \frac{1}{n} \sum_{i=1}^{n} \mathrm{Var}[x_i]$ , then with probability  $1 - \delta$  we have