SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {| \phi (s , a) ^ {\mathrm {T}} A _ {h , k} ^ {N _ {k}} \widehat {w} _ {h} ^ {k} |}{\sqrt {\frac {\gamma}{2} \left(1 - (1 - \frac {1}{2 \kappa_ {h}}) ^ {2 N _ {k}}\right) \phi (s , a) ^ {\mathrm {T}} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)}} \leq \frac {\sqrt {k + \lambda} \cdot (1 - \frac {1}{2 \kappa_ {h}}) ^ {N _ {k}} \cdot 2 H \sqrt {\frac {d k}{\lambda}}}{\sqrt {\frac {\gamma}{2} \left(1 - (1 - \frac {1}{2 \kappa_ {h}}) ^ {2 N _ {k}}\right)}} \leq \frac {1}{2}
$$
where the last inequality is by  $N_{k}\geq \log (\frac{32H^{2}(K + \lambda)dk}{\gamma\lambda} +1) / [2\log (1 / (1 - \frac{1}{2\kappa_{h}}))]$ .
For the second term above,
$$
\frac {| (r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}) (s , a) - \phi (s , a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} |}{\sqrt {\frac {\gamma}{2} \left(1 - (1 - \frac {1}{2 \kappa_ {h}}) ^ {2 N _ {k}}\right) \phi (s , a) ^ {\mathrm {T}} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)}} \leq \frac {C _ {\delta^ {\prime}}}{\sqrt {\frac {\gamma}{2} \left(1 - (1 - \frac {1}{2 \kappa_ {h}}) ^ {2 N _ {k}}\right)}} \leq \frac {C _ {\delta^ {\prime}}}{\sqrt {\frac {\gamma}{2} \left(1 - \frac {1}{2}\right)}} = \frac {1}{2}.
$$
Here the second inequality uses  $N_{k} \geq \frac{\log 2}{2\log(1 / (1 - \frac{1}{2\kappa_{h}}))}$  and the last equal sign comes from  $\gamma = 16C_{\delta'}^2$ .
Lemma C.5 (Optimism for Langevin Posterior Sampling). For any  $0 \leq \delta < 1$ , we set the input in Algorithm 2 as  $N_{k} \geq \max \{\log(\frac{32H^{2}(K + \lambda)dk}{\gamma\lambda} + 1) / [2\log(1 / (1 - \frac{1}{2\kappa_{h}}))], \frac{\log 2}{2\log(1 / (1 - \frac{1}{2\kappa_{h}}))}\}$ ,  $\gamma = 16C_{\delta/4}^{2}$  and  $M_{\delta} = \log(4HK/\delta) / \log(64/63)$ , then with probability  $1 - \delta/2$ , we have
Here  $C_{\delta}$  is defined in Lemma C.9.
Proof of Lemma C.5. Step1: Suppose the event
holds. Choose  $N_{k} \geq \max \{\log \left(\frac{32H^{2}(K + \lambda)dk}{\gamma\lambda} + 1\right) / \left[2\log \left(1 / \left(1 - \frac{1}{2\kappa_{h}}\right)\right)\right], \frac{\log 2}{2\log \left(1 / \left(1 - \frac{1}{2\kappa_{h}}\right)\right)}\}$ ,  $\gamma = 16C_{\delta^{\prime}}^{2}$  and  $M_{\delta} = \log (4HK / \delta) / \log (64 / 63)$ . Then we show, for any  $h \in [H]$ , with probability  $1 - \delta / 4$ ,  $\widetilde{Q}_{h}^{k}(s,a) \geq Q_{h}^{*}(s,a)$ ,  $\widetilde{V}_{h}^{k}(s) \geq V_{h}^{*}(s)$  for all  $(s,a) \in S \times A$ ,  $h \in [H]$ ,  $k \in [K]$ .
First, due to our choice of  $M_{\delta} = \log (4HK / \delta) / \log (64 / 63)$ , by Lemma C.3, with probability  $1 - \delta / 4$ ,
$$
\widetilde {Q} _ {h} ^ {k} (s, a) \geq (r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}) (s, a), \forall (s, a) \in \mathcal {S} \times \mathcal {A}, h \in [ H ], k \in [ K ],
$$
Step2: By Lemma C.9, with probability  $1 - \delta /4$ , for all  $k\in [K],h\in [H],s\in S,a\in \mathcal{A}$ , it holds
Therefore, in Step1, choose  $\delta' = \delta/4$ , and a union bound we obtain: for the choice  $N_k \geq \max \{\log \left( \frac{32H^2(K + \lambda)dk}{\gamma\lambda} + 1 \right) / \left[ 2\log \left( 1 / \left( 1 - \frac{1}{2\kappa_h} \right) \right) \right], \frac{\log 2}{2\log \left( 1 / \left( 1 - \frac{1}{2\kappa_h} \right) \right)}\}$ ,  $\gamma = 16C_{\delta/4}^2$  and  $M_\delta = \log (4HK/\delta) / \log (64/63)$ , then with probability  $1 - \delta/2$ , we have
C.3 Proofs of Concentration for Delayed-LPSVI
Lemma C.6 (Pointwise Concentration for Langevin Posterior Sampling). Choose  $N_{k} \geq \log\left(\frac{4HK^{3}}{\sqrt{\lambda/dK}}\right)/\log\left(1/(1-\frac{1}{2\kappa_{h}})\right)$ . Algorithm 2 guarantees that  $\forall k \in [K], h \in [H], s \in \mathcal{S}, a \in \mathcal{A}$ , the following holds with probability  $1-\delta$ ,
$$
\left| \left| \min  \left\{\widetilde {Q} _ {h} ^ {k} (s, a), H - h + 1 \right\} - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a) \right| \leq \beta \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}}. \right. \tag {21}
$$
where  $\beta := \sqrt{2\gamma\log(4C_dHMK / \delta)} + \sqrt{8H^2\left[\frac{d}{2}\log\left(\frac{k + \lambda}{\lambda}\right) + dM\log(1 + \frac{2\sqrt{8k^3}C_{H,d,k,M,\delta / 2}}{H\sqrt{\lambda}}) + \log\frac{4}{\delta}\right]}$ $+2\sqrt{\lambda}\sqrt{d} H.$  In particular, here  $\log C_d = d\log (1 + (16\sqrt{2\gamma\log(2 / \delta) / \lambda} +16H\sqrt{d})K^3)$  and  $C_{H,d,k,M,\delta} = 2H\sqrt{\frac{dk}{\lambda}} +\frac{\sqrt{2d\gamma} + \sqrt{2\gamma\log(M / \delta)}}{\sqrt{\lambda}}.$
Proof of Lemma C.6. Recall that  $|r_h^k + \mathbb{P}_h \widetilde{V}_{h+1}^k| \leq H - h + 1$ , therefore  $r_h^k + \mathbb{P}_h \widetilde{V}_{h+1}^k = \min\{r_h^k + \mathbb{P}_h \widetilde{V}_{h+1}^k, H - h + 1\}$ . This implies  $|\min\{\widetilde{Q}_h^k(s,a), H - h + 1\} - r_h^k - [\mathbb{P}_h \widetilde{V}_{h+1}^k](s,a)| = |\min\{\widetilde{Q}_h^k(s,a), H - h + 1\} - \min\{r_h^k + [\mathbb{P}_h \widetilde{V}_{h+1}^k](s,a), H - h + 1\}| \leq |\widetilde{Q}_h^k(s,a) - r_h^k - [\mathbb{P}_h \widetilde{V}_{h+1}^k](s,a)|$ . Hence
The proof then directly follows Lemma C.7 and Lemma C.9 to bound  $R_{1}$  and  $R_{2}$  respectively (together with a union bound).
Lemma C.7 (Concentration of  $R_{1}$  with Langevin Posterior Sampling). Suppose  $N_{k} \geq \log \left(\frac{4HK^{3}}{\sqrt{\lambda / dK}}\right) / \log \left(1 / \left(1 - \frac{1}{2\kappa_{h}}\right)\right)$ . For any  $0 < \delta < 1$ , define the event  $\widetilde{E}$  as
$$
\begin{array}{l} \widetilde {E} = \left\{\left| \widetilde {Q} _ {h} ^ {k} (s, a) - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| \leq \sqrt {2 \gamma \log (2 C _ {d} H M K / \delta)} \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} + \frac {1}{K ^ {3}}, \right. \\ \left. \forall k \in [ K ], h \in [ H ], s \in \mathcal {S}, a \in \mathcal {A} \right\}, \tag {22} \\ \end{array}
$$
then  $\widetilde{E}$  happens w.p.  $1 - \delta$ . Here  $\log C_d = d\log (1 + (16\sqrt{2\gamma\log(2 / \delta) / \lambda} + 16H\sqrt{d})K^3)$ .
Proof of Lemma C.7. In the Step1 and Step2, we abuse  $\widetilde{w}_h^k$  to denote  $\widetilde{w}_h^{k,m}$  for arbitrary  $m$  to avoid notation redundancy.
$$
\left| \phi (s, a) ^ {\mathrm {T}} \left(\widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k}\right) \right| \leq \sqrt {2 \gamma \log (2 / \delta)} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{2 K ^ {3}}.
$$
Indeed, by Lemma C.2 we have  $(\widetilde{w}_h^k - (I - A_{h,k}^{N_k})\widehat{w}_h^k) \sim \mathcal{N}(0, \Theta_h^k)$ , which gives,
Therefore,  $\phi(s, a)^{\mathrm{T}} (\widetilde{w}_h^k - (I - A_{h,k}^{N_k}) \widehat{w}_h^k)$  is  $\phi(s, a)^{\mathrm{T}} \Theta_h^k \phi(s, a)$ -sub-Gaussian. By concentration of sub-Gaussian random variables, we have
$$
\mathbb {P} \left(\left| \phi (s, a) ^ {\mathrm {T}} (\widetilde {w} _ {h} ^ {k} - (I - A _ {h, k} ^ {N _ {k}}) \widehat {w} _ {h} ^ {k}) \right| \geq t\right) \leq 2 \exp \left(- \frac {t ^ {2}}{2 \phi (s , a) ^ {\mathrm {T}} \Theta_ {h} ^ {k} \phi (s , a)}\right) := \delta
$$
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\widetilde {w} _ {h} ^ {k} - (I - A _ {h, k} ^ {N _ {k}}) \widehat {w} _ {h} ^ {k}) \right| \leq \sqrt {2 \log (2 / \delta)} \| \phi (s, a) \| _ {\Theta_ {h} ^ {k}} \leq \sqrt {2 \gamma \log (2 / \delta)} \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}},
$$
where the last inequality is by Lemma C.2, and by Lemma C.8, the above further implies
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k}) \right| \leq \sqrt {2 \gamma \log (2 / \delta)} \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} + \frac {1}{2 K ^ {3}}.
$$
Step2: we prove that for any  $0 < \delta < 1$ , define the event  $\widetilde{E}$  as
$$
\begin{array}{l} \widetilde {E} = \left\{\left| \phi (s, a) ^ {\mathrm {T}} \widetilde {w} _ {h} ^ {k} - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| \leq \sqrt {2 \gamma \log (2 C _ {d} H K / \delta)} \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} + \frac {1}{K ^ {3}}, \right. \\ \left. \forall k \in [ K ], h \in [ H ], s \in \mathcal {S}, a \in \mathcal {A} \right\}, \tag {23} \\ \end{array}
$$
In Lemma D.12, set  $\theta = \widetilde{w}_h^k -\widehat{w}_h^k$  and  $A = (\Omega_h^k)^{-1}$  and  $B = 1 / \lambda$ , and let  $\mathcal{V}$  be the  $\frac{1}{4K^3}$ -epsilon net for the class of values  $\{\lvert\langle\phi,\widetilde{w}_h^k-\widehat{w}_h^k\rangle\rvert-C\sqrt{\phi^\top(\Omega_h^k)^{-1}\phi}-\frac{1}{2K^3}:\| \phi \| \leq 1\}$  (where  $C=\sqrt{2\gamma\log(2/\delta)}$ ), then it must also be the  $\frac{1}{4K^3}$ -epsilon net for the class of values  $\mathcal{F} = \{\lvert\langle\phi(s,a),\widetilde{w}_h^k-\widehat{w}_h^k\rangle\rvert-C\sqrt{\phi(s,a)^\top(\Omega_h^k)^{-1}\phi(s,a)}-\frac{1}{2K^3}:(s,a)\in\mathcal{S}\times\mathcal{A}\}$ , let  $\bar{\mathcal{V}}$  is the smallest subset of  $\mathcal{V}$  such that it is  $\frac{1}{2K^3}$ -epsilon net for the class of values  $\mathcal{F}$ . Then we can select  $\mathcal{V}_{\mathcal{S}\times\mathcal{A}}$  to be the set of state-action pairs such that for any  $f_{\phi}:=|\langle\phi,\widetilde{w}_h^k-\widehat{w}_h^k\rangle| - C\sqrt{\phi^\top(\Omega_h^k)^{-1}\phi} \in \bar{\mathcal{V}} -\frac{1}{2K^3}$ , there exists  $(s,a)\in \mathcal{V}_{\mathcal{S}\times\mathcal{A}}$  satisfies  $\left|\left|\langle\phi(s,a),\widetilde{w}_h^k-\widehat{w}_h^k\rangle\right| - C\sqrt{\phi(s,a)^\top(\Omega_h^k)^{-1}\phi(s,a)}\right| - \frac{1}{2K^3} -f_{\phi}\left|\leq 1/4K^3\right.$ , then we have  $\mathcal{V}_{\mathcal{S}\times\mathcal{A}}$  is a  $1/(2K^3)$ -epsilon net of  $\mathcal{F}$  and  $|\mathcal{V}_{\mathcal{S}\times\mathcal{A}}|\leq |\bar{\mathcal{V}} |\leq |\mathcal{V}|$ . Therefore,
$$
\begin{array}{l} \sup _ {s, a} \left(| \langle \phi (s, a), \tilde {w} _ {h} ^ {k} - \hat {w} _ {h} ^ {k} \rangle | - C \sqrt {\phi (s , a) ^ {\top} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)} - \frac {1}{2 K ^ {3}}\right) \\ \leq \sup  _ {(s, a) \in \mathcal {V} _ {\mathcal {S} \times \mathcal {A}}} \left(| \langle \phi (s, a), \widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k} \rangle | - C \sqrt {\phi (s , a) ^ {\top} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)} - \frac {1}{2 K ^ {3}}\right) + 1 / (2 K ^ {3}) \\ \leq 1 / (2 K ^ {3}), \\ \end{array}
$$
where the last inequality is from Step1. Then by a union bound over  $H$ ,  $K$  and  $(1 + (16\sqrt{2\gamma\log(2 / \delta) / \lambda} + 16H\sqrt{d})K^3)^d$ , we have with probability  $1 - \delta$
$$
\begin{array}{l} \sup  _ {s, a, h, k} \left(| \langle \phi (s, a), \tilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k} \rangle | - \sqrt {2 \gamma \log (2 C _ {d} H K / \delta)} \sqrt {\phi (s , a) ^ {\top} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)}\right) \\ \leq \frac {1}{2 K ^ {3}} + \frac {1}{2 K ^ {3}} = \frac {1}{K ^ {3}}, \\ \end{array}
$$
where  $\log C_d = d\log (1 + (16\sqrt{2\gamma\log(2 / \delta) / \lambda} +16H\sqrt{d})K^3)$ .
Step3: We finish the proof. Note  $\widetilde{Q}_h^k = \max_m\phi^\mathrm{T}\widetilde{w}_h^{k,m}$ , hence by a union bound over  $M$ , we have
$$
\begin{array}{l} \left| \widetilde {Q} _ {h} ^ {k} (s, a) - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| = \left| \max  _ {m} \phi (s, a) ^ {\mathrm {T}} \widetilde {w} _ {h} ^ {k, m} - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| \\ \leq \max _ {m} | \phi (s, a) ^ {\mathrm {T}} \widetilde {w} _ {h} ^ {k, m} - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} | \\ \leq \sqrt {2 \gamma \log (2 C _ {d} H M K / \delta)} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}} \\ \end{array}
$$
for all  $k, h, s, a$  with probability  $1 - \delta$ . Here the last inequality uses Step2. This finishes the proof.
Lemma C.8. Let  $N_{k}\geq \log (\frac{4HK^{3}}{\sqrt{\lambda / dK}}) / \log (1 / (1 - \frac{1}{2\kappa_{h}}))$  and  $\eta = \frac{1}{4\lambda_{\mathrm{max}}(\Omega_h^k)}$ , then
$$
\left\| \phi (s, a) ^ {\mathrm {T}} A _ {h, k} ^ {N _ {k}} \widehat {w} _ {h} ^ {k} \right\| \leq \frac {1}{2 K ^ {3}}
$$
Proof of Lemma C.8. By direct calculation,
$$
\begin{array}{l} \left\| \phi (s, a) ^ {\mathrm {T}} A _ {h, k} ^ {N _ {k}} \widehat {w} _ {h} ^ {k} \right\| \leq \| \phi (s, a) \| \| A _ {h, k} \| ^ {N _ {k}} \| \widehat {w} _ {h} ^ {k} \| \leq \| A _ {h, k} \| ^ {N _ {k}} \| \widehat {w} _ {h} ^ {k} \| \\ \leq \left\| A _ {h, k} \right\| ^ {N _ {k}} 2 H \sqrt {\frac {d k}{\lambda}} \leq \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {N _ {k}} \cdot 2 H \sqrt {\frac {d k}{\lambda}} \leq \frac {1}{2 K ^ {3}} \\ \end{array}
$$
where the third inequality is by Lemma D.4 and the fourth inequality is by (20). The last inequality is by the choice of  $N_{k}$ .
Lemma C.9 (Concentration of  $R_{2}$  with Langevin Posterior Sampling). For any  $0 < \delta < 1$ , with probability  $1 - \delta$ , for all  $k \in [K]$ ,  $h \in [H]$ ,  $s \in S$ ,  $a \in \mathcal{A}$ , it holds
where  $C_{\delta} = \sqrt{8H^{2}\left[\frac{d}{2}\log\left(\frac{k + \lambda}{\lambda}\right) + dM\log(1 + \frac{2\sqrt{8k^{3}}C_{H,d,k,M,\delta}}{H\sqrt{\lambda}}) + \log\frac{2}{\delta}\right]} + 2\sqrt{\lambda}\sqrt{d}H$  and the quantity  $C_{H,d,k,M,\delta} = 2H\sqrt{\frac{dk}{\lambda}} + \frac{\sqrt{2d\gamma} + \sqrt{2\gamma\log(M / \delta)}}{\sqrt{\lambda}}$ .
Proof of Lemma C.9. For any  $(k,h)\in [K]\times [H]$  and  $(s,a)\in S\times \mathcal{A}$ , denote
Recall  $y_{h}^{\tau} = \mathbb{1}_{\tau ,k - 1}\cdot [r_{h}^{\tau}(s_{h}^{\tau},a_{h}^{\tau}) + \widetilde{V}_{h + 1}^{k}(s_{h + 1}^{\tau})]$  from Algorithm 1 and denote  $\bar{y}_h^\tau \coloneqq r_h^\tau (s_h^\tau ,a_h^\tau) +$ $\widetilde{V}_{h + 1}^{k}(s_{h + 1}^{\tau})$  . Then by definition,
$$
\widehat {w} _ {h} ^ {k} = \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) y _ {h} ^ {\tau} = \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \bar {y} _ {h} ^ {\tau}.
$$
By definition of  $\Omega_h^k$ , we have  $\Phi_h\Phi_h^{\mathrm{T}} = \Omega_h^k -\lambda I$ . Plug it into the definition of  $\widehat{w}_h^k$ , we have
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\mathbf {i}) \right| \leq \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {(\Omega_ {h} ^ {k}) ^ {- 1}}.
$$
Apply Lemma C.10, we have with probability at least  $1 - \delta$ , for any  $(k,h) \in [K] \times [H]$ , and  $(s,a) \in S \times A$ ,
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\mathbf {i}) \right| \leq C _ {1} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}, \tag {24}
$$
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\mathrm {i i}) \right| = \lambda \left| \phi (s, a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} w _ {h} ^ {k} \right| \leq 2 \sqrt {\lambda} \sqrt {d} H \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}. \tag {25}
$$
Combining (24), (25), we have with probability  $1 - \delta$ , for any  $(k, h) \in [K] \times [H]$  and  $(s, a) \in S \times \mathcal{A}$ ,
Lemma C.10. For any  $0 < \delta < 1$ , with probability  $1 - \delta$ , we have  $\forall (k, h) \in [K] \times [H]$ ,
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq 8 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log \left(1 + \frac {2 \sqrt {8 k ^ {3}} C _ {H , d , k , M , \delta}}{H \sqrt {\lambda}}\right) + \log \frac {2}{\delta} \right], \\ \end{array}
$$
here  $C_{H,d,k,M,\delta} = 2H\sqrt{\frac{dk}{\lambda}} + \frac{\sqrt{2d\gamma} + \sqrt{2\gamma\log(M / \delta)}}{\sqrt{\lambda}}$ .<sup>9</sup>
Proof of Lemma C.10. First note that
$$
\begin{array}{l} \widetilde {V} _ {h} ^ {k} (\cdot) := \max  _ {a} \min  \{\widetilde {Q} _ {h} ^ {k} (\cdot , a), (H - h + 1) \} = \max  _ {a} \min  \max  _ {m} \{\widetilde {Q} _ {h} ^ {k, m}, (H - h + 1) \} \\ = \max  _ {a} \min  \left\{\max  _ {m} \phi (\cdot , a) ^ {\mathrm {T}} \tilde {w} _ {h} ^ {k, m}, (H - h + 1) \right\}. \\ \end{array}
$$
Choosing  $w_0 = 0$ , then by Lemma C.2 and  $(\Theta_h^k)^{-1/2} (\widetilde{w}_h^{k,m} - (I - A_{h,k}^{N_k}) \widehat{w}_h^k) \sim \mathcal{N}(0, I_d)$ , and by Lemma D.7, with probability  $1 - \delta/2$ , we have
$$
\frac {\sqrt {\lambda}}{\sqrt {\gamma}} \left\| \widetilde {w} _ {h} ^ {k, m} - (I - A _ {h, k} ^ {N _ {k}}) \widehat {w} _ {h} ^ {k} \right\| \leq \left\| (\Theta_ {h} ^ {k}) ^ {- 1 / 2} (\widetilde {w} _ {h} ^ {k, m} - (I - A _ {h, k} ^ {N _ {k}}) \widehat {w} _ {h} ^ {k}) \right\| \leq \sqrt {2 d} + \sqrt {2 \log (1 / \delta)},
$$
<sup>9</sup>We will choose  $\gamma$  to be  $\mathrm{Poly}(H, d, K)$  and this will not affect the overall dependence of the guarantee since  $C_{H, d, k, M, \delta}$  is inside the log term.
where the first inequality uses Lemma C.2 again. Apply the union bound over all  $m$ , then above implies with probability  $1 - \delta / 2, \forall m \in [M]$
$$
\left\| \widetilde {w} _ {h} ^ {k, m} \right\| \leq \| \widehat {w} _ {h} ^ {k} \| + \frac {\sqrt {2 d \gamma} + \sqrt {2 \gamma \log (M / \delta)}}{\sqrt {\lambda}} \leq 2 H \sqrt {\frac {d k}{\lambda}} + \frac {\sqrt {2 d \gamma} + \sqrt {2 \gamma \log (M / \delta)}}{\sqrt {\lambda}} := C _ {H, d, k, M, \delta}. \tag {26}
$$
(where we used  $\left\| (I - A_{h,k}^{N_k})\widehat{w}_h^k\right\| \leq \left\| (I - A_{h,k}^{N_k})\right\| \left\| \widehat{w}_h^k\right\| \leq \left\| \widehat{w}_h^k\right\|$ ). Now consider the function class  $\bar{\mathcal{V}} := \{\max_a\max_m\phi (\cdot ,a)^\mathrm{T}w^m:\| w^m\| \leq C_{H,d,k,M,\delta}\}$ , so by Lemma D.13 the  $\epsilon$ -log covering number for  $\bar{\mathcal{V}}$  is  $dM\log (1 + \frac{2C_{H,d,k,M,\delta}}{\epsilon})$ . Since  $\min \{\cdot ,\cdot \}$  is a non-expansive operator, the  $\epsilon$ -log covering number for the function class  $\mathcal{V} := \{\max_a\min \{\max_m\phi (\cdot ,a)^\mathrm{T}w^m,(H - h + 1)\} : \| w^m\| \leq C_{H,d,k,M,\delta}\}$ , is at most  $dM\log (1 + \frac{2C_{H,d,k,M,\delta}}{\epsilon})$ . Hence, for any  $V\in \mathcal{V}$ , there exists  $V'$  in the  $\epsilon$ -covering such that  $V = V' + \Delta_V$  with  $\| \Delta_V\|_{\infty}\leq \epsilon$ . Then with probability  $1 - \delta /2$
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(V \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} V \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq 2 \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(V ^ {\prime} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} V ^ {\prime} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ + 2 \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\Delta_ {V} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} \Delta_ {V} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \tag {27} \\ \leq 2 \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(V ^ {\prime} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} V ^ {\prime} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} ^ {2} + \frac {8 k ^ {2} \epsilon^ {2}}{\lambda} \\ \leq 4 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log \left(1 + \frac {2 C _ {H , d , k , M , \delta}}{\epsilon}\right) + \log \frac {2}{\delta} \right] + \frac {8 k ^ {2} \epsilon^ {2}}{\lambda} \\ \end{array}
$$
where the second inequality can be conducted using a direct calculation and the third inequality uses Lemma D.9 and a union bound over the covering number. Now by (26) and (27) and a union bound, we have for any  $\epsilon > 0$ , with probability  $1 - \delta$ ,
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq 4 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log \left(1 + \frac {2 C _ {H , d , k , M , \delta}}{\epsilon}\right) + \log \frac {2}{\delta} \right] + \frac {8 k ^ {2} \epsilon^ {2}}{\lambda} \\ \leq 8 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log (1 + \frac {2 \sqrt {8 k ^ {3}} C _ {H , d , k , M , \delta}}{H \sqrt {\lambda}}) + \log \frac {2}{\delta} \right], \\ \end{array}
$$
D Auxiliary lemmas
D.1 Useful Norm Inequalities
Lemma D.1. Suppose  $v \in \mathbb{R}^d$ , and  $A$  is some positive definite matrix whose eigenvalues satisfy  $\lambda_{\max}(A) \geq \dots \geq \lambda_{\min}(A) > 0$ . It can be shown that
$$
\sqrt {\lambda_ {\min} (A)} \| v \| \leq \| v \| _ {A} \leq \sqrt {\lambda_ {\max} (A)} \| v \|.
$$
Proof of Lemma D.1. Consider the eigenvalue decomposition of  $A$ , which gives  $A = U\Lambda U^{\mathrm{T}}$ , where  $\Lambda = \mathrm{diag}(\lambda_{\max}(A),\ldots ,\lambda_{\min}(A))$ . Then
$$
\left\| v \right\| _ {A} = \sqrt {\sum_ {i = 1} ^ {d} \lambda_ {i} (A) \left(u _ {i} ^ {\mathrm {T}} v\right) ^ {2}} \leq \sqrt {\lambda_ {\max } (A) \left\| u _ {i} ^ {\mathrm {T}} v \right\| ^ {2}} = \sqrt {\lambda_ {\max } (A)} \| v \|.
$$
Similar argument shows  $\| v\| _A\geq \sqrt{\lambda_{\min}(A)}$
Lemma D.2 (Lemma D.1 of [35]). Let  $\Omega_h^k$  be the precision matrix of the posterior distribution of  $w_h^k$  at step  $h$  of episode  $k$ , where  $\Omega_h^k \coloneqq \sigma^{-2}\Phi_h\Phi_h^{\mathrm{T}} + \Sigma^{-1}$  with  $\Sigma^{-1} = \lambda I_d$  and  $\sigma^2 = 1$ . Then
$$
\sum_ {\tau = 1} ^ {k - 1} \left\| \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \leq d.
$$
Lemma D.3 (Bound on Weights of Q-function). Suppose the linear MDP assumption and at each step  $h \in [H]$ , rewards  $r_h$  are bounded between  $[0,1]$ , then the norm of the true parameter  $w_h^\pi$  under fixed policy  $\pi$  satisfies
$$
\forall h \in [ H ], \quad \| w _ {h} ^ {\pi} \| \leq 2 H \sqrt {d}.
$$
In addition, for any  $(s,a)\in \mathcal{S}\times \mathcal{A}$ , let  $\phi (s,a)^{\mathrm{T}}w_{h}^{k} := (r_{h} + \mathbb{P}_{h}\widetilde{V}_{h + 1}^{k})(s,a)$ , we also have
$$
\forall h \in [ H ], k \in [ K ], \quad \| w _ {h} ^ {k} \| \leq 2 H \sqrt {d}.
$$
Proof of Lemma D.3. By definition in Lemma A.1, the true parameter  $w_{h}$  at time step  $h$  is
$$
w _ {h} ^ {\pi} := \theta_ {h} + \mathbb {E} _ {s ^ {\prime} \sim \mu_ {h}} [ V _ {h + 1} ^ {\pi} (s ^ {\prime}) ].
$$
With bounded rewards  $r_h \in [0,1]$ , we have  $V_{h+1}^{\pi}(s) \leq H$ ,  $\forall s \in \mathcal{S}$ . Since  $\|\theta_h\| \leq \sqrt{d}$ , and  $\left\|\mathbb{E}_{\mu_h}[V_{h+1}^{\pi}(s')] \right\| \leq \left\|\int_{\mathcal{S}} H \mathrm{d}\mu_h(s')\right\| \leq H\sqrt{d}$ .
Similarly, by definition of the constructed weights  $w_{h}^{k}$ ,
$$
w _ {h} ^ {k} := \theta_ {h} + \int_ {\mathcal {S}} \widetilde {V} _ {h + 1} ^ {k} \left(s ^ {\prime}\right) \mathrm {d} \mu_ {h} \left(s ^ {\prime}\right).
$$
From Line 15 of Algorithm 1, for any  $h \in [H]$  and  $s \in S$ ,  $\widetilde{V}_h^k(s) = \max_a \min \{\widetilde{Q}_h^k(\cdot, a), H - h + 1\} \leq H$ . Applying triangle inequality, we have