SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here  $D_{\tau, \delta, H, K} \coloneqq 1 + 2\mathbb{E}[\tau] + 2\sqrt{2\mathbb{E}[\tau]\log(\frac{24KH}{\delta})} + \frac{4}{3}\log(\frac{24KH}{\delta}) + D_{\tau, K, \frac{\delta}{16H}}$  and  $D_{\tau, K, \delta}$  is defined in Lemma D.6.
Proof of Lemma B.4. Now Combine Lemma B.2 and Lemma B.3, we obtain
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \leq \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} + \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Lambda_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \\ \leq \underbrace{\sum_{k = 1}^{K}\left\| \phi_{h}^{k}\right\|_{(\Sigma_{h}^{k})^{-1}}} _{(*)} + \underbrace{\frac{1}{2}\sum_{k = 1}^{K}(1 + \max_{k\in[K]}U_{k} + \tau_{k})\left\| \phi_{h}^{k}\right\|_{(\Sigma_{h}^{k})^{-1}}^{2}}_{(**)}. \\ \end{array}
$$
For term  $(*)$ , since  $\lambda = 1$ , by Cauchy-Schwartz inequality and Elliptical Potential Lemma D.8,
$$
\sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} \leq \sqrt {K \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2}} \leq \sqrt {2 K \log \left(\frac {\det (\Sigma_ {h} ^ {K + 1})}{\det (\Sigma_ {h} ^ {1})}\right)} \leq \sqrt {2 d K \log ((d + K) / d)}
$$
For term  $(\ast \ast)$ , by Lemma D.15 and Lemma D.6 and a union bound, with probability  $1 - \delta / 8$ ,
$$
\begin{array}{l} \frac {1}{2} \sum_ {k = 1} ^ {K} \left(1 + \max _ {k \in [ K ]} U _ {k} + \tau_ {k}\right) \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq \frac {1}{2} \left(1 + \max  _ {k \in [ K ]} U _ {k} + \max  _ {k \in [ K ]} \tau_ {k}\right) \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq \frac {1}{2} \left(1 + \max  _ {k \in [ K ]} U _ {k} + \max  _ {k \in [ K ]} \tau_ {k}\right) 2 d \log (1 + K) \\ \leq d (1 + \mathbb {E} [ \tau ] + 2 \sqrt {2 \mathbb {E} [ \tau ] \log (\frac {2 4 K}{\delta})} + \frac {4}{3} \log (\frac {2 4 K}{\delta}) + \max _ {k \in [ K ]} \tau_ {k}) \log ((d + K) / d) \\ \leq d (1 + \mathbb {E} [ \tau ] + 2 \sqrt {2 \mathbb {E} [ \tau ] \log (\frac {2 4 K}{\delta})} + \frac {4}{3} \log (\frac {2 4 K}{\delta}) + \mathbb {E} [ \tau ] + D _ {\tau , \frac {\delta}{1 6}}) \log ((d + K) / d) \\ \end{array}
$$
Denote  $D_{\tau, \delta, K} := 1 + \mathbb{E}[\tau] + 2\sqrt{2\mathbb{E}[\tau]\log(\frac{24K}{\delta})} + \frac{4}{3}\log(\frac{24K}{\delta}) + \mathbb{E}[\tau] + D_{\tau, K, \frac{\delta}{16}}$ , then we have with probability  $1 - \delta/8$ ,
$$
\sum_ {k = 1} ^ {K} \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \leq \sqrt {2 d K \log ((d + K) / d)} + d D _ {\tau , \delta , K} \log ((d + K) / d),
$$
then apply a union bound over  $h \in [H]$  to obtained the stated result.
B.2 Proofs of Anti-concentration for Delayed-PSVI
In this section, we prove the optimism via anti-concentration for Delayed-PSVI. We first present two assisting lemmas.
Lemma B.5 (Anti-concentration for Optimism). Suppose the event
$$
E = \left\{\left| \widehat {Q} _ {h} ^ {k} (s, a) - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a) \right| \leq C _ {\delta^ {\prime}} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}, \forall s, a, h, k \right\}
$$
holds. Choose  $\nu = C_{\delta'}$  and  $M_{\delta} = \log(HK / \delta) / \log(64 / 63)$ . Then we have with probability  $1 - \delta$ ,
$$
\widetilde {Q} _ {h} ^ {k} (s, a) \geq \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \forall (s, a) \in \mathcal {S} \times \mathcal {A}, h \in [ H ], k \in [ K ].
$$
Proof of Lemma B.5. For the rest of the proof, we condition on the event
where  $\delta^\prime$  will be specified later and  $C_\delta$  is defined in the Lemma B.10. Also note
$$
\begin{array}{l} \widetilde {Q} _ {h} ^ {k, m} (s, a) - \widehat {Q} _ {h} ^ {k} (s, a) = \phi (s, a) ^ {\mathrm {T}} \left(\widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k}\right) \sim \mathcal {N} \left(0, \nu^ {2} \phi (s, a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s, a)\right) \\ \Leftrightarrow \frac {\widetilde {Q} _ {h} ^ {k , m} (s , a) - \widehat {Q} _ {h} ^ {k} (s , a)}{\sqrt {\nu^ {2} \phi (s , a) ^ {\mathrm {T}} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)}} \sim \mathcal {N} (0, 1). \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} \left(\widetilde {Q} _ {h} ^ {k, m} (s, a) \geq \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \forall s, a \mid \widehat {Q} _ {h} ^ {k}\right) \\ = \mathbb {P} \left(\frac {\widetilde {Q} _ {h} ^ {k , m} (s , a) - \widehat {Q} _ {H} ^ {k} (s , a)}{\sqrt {\nu^ {2} \phi (s , a) ^ {\mathrm {T}} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)}} \geq \frac {(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}) (s , a) - \widehat {Q} _ {h} ^ {k} (s , a)}{\sqrt {\nu^ {2} \phi (s , a) ^ {\mathrm {T}} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)}}, \forall s, a \mid \widehat {Q} _ {h} ^ {k}\right) \\ = \mathbb {P} \left(\mathcal {N} (0, 1) \geq \frac {\left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) (s , a) - \widehat {Q} _ {h} ^ {k} (s , a)}{\sqrt {\nu^ {2} \phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)}}, \forall s, a \mid \widehat {Q} _ {h} ^ {k}\right) \\ \geq \mathbb {P} \left(\mathcal {N} (0, 1) \geq C _ {\delta^ {\prime}} / \nu\right) \\ \geq \frac {1}{2 \sqrt {8 \pi}} e ^ {- 1 / 2} \geq \frac {1}{6 4}, \\ \end{array}
$$
where the first event uses the condition on  $E$  and the second inequality chooses  $\nu = C_{\delta'}$  and uses Lemma D.5. Apply Lemma B.6 with  $f = r_h + \mathbb{P}_h\widetilde{V}_{h + 1}^k$ , for  $M_{\delta} = \log (1 / \delta) / \log (64 / 63)$
$$
\mathbb {P} \left(\widetilde {Q} _ {h} ^ {k} (s, a) \geq \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \forall s, a \mid \widehat {Q} _ {h} ^ {k}\right) \geq 1 - \delta .
$$
By law of total expectation  $\mathbb{E}[\mathbb{E}[\mathbf{1}_A|X]] = \mathbb{E}[\mathbf{1}_A] = \mathbb{P}[A]$ , it implies
$$
\mathbb {P} \left(\widetilde {Q} _ {h} ^ {k} (s, a) \geq \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \forall s, a\right) \geq 1 - \delta .
$$
Apply a union bound for  $h, k$ , we have for  $M_{\delta} = \log (HK / \delta) / \log (64 / 63)$ , with probability  $1 - \delta$
$$
\mathbb {P} \left(\widetilde {Q} _ {h} ^ {k} (s, a) \geq \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \forall s, a, h, k\right) \geq 1 - \delta .
$$
The following lemma is used to prove Lemma B.5.
Lemma B.6. For any function  $f: \mathcal{S} \times \mathcal{A} \mapsto \mathbb{R}$ . For any  $0 < \delta < 1$ . Suppose for any  $(k, h, m) \in [K] \times [H] \times [M]$ ,  $\mathbb{P}\left(\widetilde{Q}_h^{k,m}(s, a) \geq f(s, a), \forall s, a \mid \widehat{Q}_h^k\right) \geq c$  for some constant  $c > 0$ . Let  $M = \log(1/\delta)/\log(1/(1-c))$ . Then
$$
\mathbb {P} \left(\widetilde {Q} _ {h} ^ {k} (s, a) \geq f (s, a), \forall s, a \mid \widehat {Q} _ {h} ^ {k}\right) \geq 1 - \delta .
$$
Proof of Lemma B.6. For any fixed  $(k,h)\in [K]\times [H]$ , we have
$$
\begin{array}{l} \mathbb {P} \left(\exists (s, a) s. t. \max  _ {m \in [ M ]} \widetilde {Q} _ {h} ^ {k, m} (s, a) \leq f (s, a) \mid \widehat {Q} _ {h} ^ {k}\right) \\ = \mathbb {P} \left(\exists (s, a) s. t. \forall m \in [ M ], \widetilde {Q} _ {h} ^ {k, m} (s, a) \leq f (s, a) \mid \widehat {Q} _ {h} ^ {k}\right) \\ \leq \mathbb {P} \left(\forall m \in [ M ], \exists \left(s _ {m}, a _ {m}\right) s. t. \widetilde {Q} _ {h} ^ {k, m} \left(s _ {m}, a _ {m}\right) \leq f \left(s _ {m}, a _ {m}\right) \mid \widehat {Q} _ {h} ^ {k}\right) \\ = \prod_ {m = 1} ^ {M} \mathbb {P} \left(\exists (s, a) s. t. \widetilde {Q} _ {h} ^ {k, m} (s, a) \leq f (s, a) \mid \widehat {Q} _ {h} ^ {k}\right) \\ = \prod_ {m = 1} ^ {M} \left[ 1 - \mathbb {P} \left(\widetilde {Q} _ {h} ^ {k, m} (s, a) \geq f (s, a), \forall s, a \mid \widehat {Q} _ {h} ^ {k}\right) \right] \leq (1 - c) ^ {M} = \delta , \\ \end{array}
$$
then this implies
With the above two lemmas, we are ready to prove the optimism achieved by Delayed-PSVI with respect to  $\widetilde{Q}_h^k$ .
Lemma B.7 (Optimism). For any  $0 \leq \delta < 1$ , we set the input in Algorithm 1 as  $\nu = C_{\delta/4}$  and  $M_{\delta} = \log(4HK/\delta)/\log(64/63)$ , then with probability  $1 - \delta/2$ , we have
$$
\widetilde {Q} _ {h} ^ {k} (s, a) \geq Q _ {h} ^ {*} (s, a), \widetilde {V} _ {h} ^ {k} (s) \geq V _ {h} ^ {*} (s) \quad \forall s, a \in \mathcal {S} \times \mathcal {A}, \forall h \in [ H ], k \in [ K ].
$$
Here  $C_{\delta}$  is defined in Lemma B.10.
Proof of Lemma B.7. Step1: Suppose the event
holds. Choose  $\nu = C_{\delta'}$  and  $M_{\delta} = \log(4HK/\delta) / \log(64/63)$ . Then we show, for any  $h \in [H]$ , with probability  $1 - \delta/4$ ,  $\widetilde{Q}_h^k(s,a) \geq Q_h^*(s,a)$ ,  $\widetilde{V}_h^k(s) \geq V_h^*(s)$  for all  $(s,a) \in S \times \mathcal{A}$ ,  $h \in [H]$ ,  $k \in [K]$ .
First, due to our choice of  $M_{\delta} = \log (4HK / \delta) / \log (64 / 63)$ , by Lemma B.5, with probability  $1 - \delta / 4$ ,
$$
\widetilde {Q} _ {h} ^ {k} (s, a) \geq \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \forall (s, a) \in \mathcal {S} \times \mathcal {A}, h \in [ H ], k \in [ K ],
$$
which we condition on.
Next, we finish the proof by backward induction. Base case: for  $h = H + 1$ , the value functions are zero, and thus  $\widetilde{Q}_{H + 1}^k \geq Q_{H + 1}^*$  holds trivially, which also implies  $\widetilde{V}_{H + 1}^k \geq V_{H + 1}^*$ . Suppose the conclusion holds true for  $h + 1$ . Then for time step  $h$  and any  $k \in [K]$ ,
$$
\begin{array}{l} \widetilde {Q} _ {h} ^ {k} - Q _ {h} ^ {*} = \widetilde {Q} _ {h} ^ {k} - \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) + \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) - Q _ {h} ^ {*} \\ \geq \widetilde {Q} _ {h} ^ {k} - \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) + \left(r _ {h} + \mathbb {P} _ {H} V _ {h + 1} ^ {*}\right) - Q _ {h} ^ {*} \\ = \widetilde {Q} _ {h} ^ {k} - \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) \geq 0 \\ \end{array}
$$
where the first inequality uses the induction hypothesis and the second inequality uses the condition. Lastly,  $\widetilde{V}_h^k (\cdot) = \max_a\min \{\widetilde{Q}_h^k (\cdot ,a),H - h + 1\} \leq \max_a\min \{Q_h^* (\cdot ,a),H - h + 1\} =$ $\max_{a}Q_{h}^{*}(\cdot ,a) = V_{h}^{*}(\cdot)$ . By induction, this finishes the Step1.
Step2: By Lemma B.10, with probability  $1 - \delta / 4$ , for all  $k \in [K], h \in [H], s \in S, a \in \mathcal{A}$ , it holds
$$
\left| \widehat {Q} _ {h} ^ {k} (s, a) - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a) \right| \leq C _ {\delta / 4} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}.
$$
Therefore, in Step1, choose  $\delta' = \delta/4$ , and a union bound we obtain: for the choice  $\nu = C_{\delta/4}$  and  $M_{\delta} = \log(4HK/\delta)/\log(64/63)$ , then with probability  $1 - \delta/2$ , we have
$$
\widetilde {Q} _ {h} ^ {k} (s, a) \geq Q _ {h} ^ {*} (s, a), \widetilde {V} _ {h} ^ {k} (s) \geq V _ {h} ^ {*} (s) \forall (s, a) \in \mathcal {S} \times \mathcal {A}, h \in [ H ], k \in [ K ].
$$
B.3 Proofs of Concentration for Delayed-PSVI
Lemma B.8 (Pointwise Concentration). Algorithm 1 guarantees that with probability  $1 - \delta$ ,  $\forall k \in [K], h \in [H], s \in S, a \in \mathcal{A}$ , it holds:
$$
\left| \left| \min  \left\{\widetilde {Q} _ {h} ^ {k} (s, a), H - h + 1 \right\} - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a) \right| \leq \beta \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}} \right. \tag {10}
$$
where  $\beta := \sqrt{2\nu^2\log(4C_dHMK / \delta)} + \sqrt{8H^2\left[\frac{d}{2}\log\left(\frac{k + \lambda}{\lambda}\right) + dM\log(1 + \frac{2\sqrt{8k^3}C_{H,d,k,M,\delta / 2}}{H\sqrt{\lambda}}) + \log\frac{4}{\delta}\right]} +$
$2\sqrt{\lambda}\sqrt{d}H$ . In particular, here  $\log C_d = d\log(1 + (8\sqrt{2\nu^2\log(4/\delta)/\lambda} + 8H\sqrt{d})K^3)$  and  $C_{H,d,k,M,\delta} = 2H\sqrt{\frac{dk}{\lambda}} + \frac{\nu\sqrt{2d} + \nu\sqrt{2\log(M/\delta)}}{\sqrt{\lambda}}$ .
Proof of Lemma B.8. Recall that  $|r_h^k + \mathbb{P}_h \widetilde{V}_{h+1}^k| \leq H - h + 1$ , therefore  $r_h^k + \mathbb{P}_h \widetilde{V}_{h+1}^k = \min\{r_h^k + \mathbb{P}_h \widetilde{V}_{h+1}^k, H - h + 1\}$ . This implies  $|\min\{\widetilde{Q}_h^k(s,a), H - h + 1\} - r_h^k - [\mathbb{P}_h \widetilde{V}_{h+1}^k](s,a)| = |\min\{\widetilde{Q}_h^k(s,a), H - h + 1\} - \min\{r_h^k + [\mathbb{P}_h \widetilde{V}_{h+1}^k](s,a), H - h + 1\}| \leq |\widetilde{Q}_h^k(s,a) - r_h^k - [\mathbb{P}_h \widetilde{V}_{h+1}^k](s,a)|$ . Hence
$$
\begin{array}{l} \left| \min  \{\widetilde {Q} _ {h} ^ {k} (s, a), H - h + 1 \} - r _ {h} ^ {k} - [ \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} ] (s, a) \right| \leq \left| \widetilde {Q} _ {h} ^ {k} (s, a) - r _ {h} ^ {k} - [ \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} ] (s, a) \right| \\ = \left| \widetilde {Q} _ {h} ^ {k} (s, a) - \widehat {Q} _ {h} ^ {k} (s, a) + \widehat {Q} _ {h} ^ {k} (s, a) - r _ {h} ^ {k} - [ \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} ] (s, a) \right| \\ \leq \underbrace {\left| \widetilde {Q} _ {h} ^ {k} (s , a) - \widehat {Q} _ {h} ^ {k} (s , a) \right|} _ {R _ {1}} + \underbrace {\left| \widehat {Q} _ {h} ^ {k} (s , a) - r _ {h} ^ {k} - [ \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} ] (s , a) \right|} _ {R _ {2}}. \\ \end{array}
$$
The proof then directly follows Lemma B.9 and Lemma B.10 to bound  $R_{1}$  and  $R_{2}$  respectively (together with a union bound).
Lemma B.9 (Concentration of  $R_{1}$ ). For any  $0 < \delta < 1$ , define the event  $\widetilde{E}$  as
$$
\begin{array}{l} \widetilde {E} = \left\{\left| \widetilde {Q} _ {h} ^ {k} (s, a) - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| \leq \sqrt {2 \nu^ {2} \log \left(2 C _ {d} H M K / \delta\right)} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}}, \right. \\ \left. \forall k \in [ K ], h \in [ H ], s \in \mathcal {S}, a \in \mathcal {A} \right\}, \tag {11} \\ \end{array}
$$
then  $\widetilde{E}$  happens with probability  $1 - \delta$ . Here  $\log C_d = d\log (1 + (8\sqrt{2\nu^2\log(2 / \delta) / \lambda} + 8H\sqrt{d})K^3)$ .
Proof of Lemma B.9. In the Step1 and Step2, we abuse  $\widetilde{w}_h^k$  to denote  $\widetilde{w}_h^{k,m}$  for arbitrary  $m$  to avoid notation redundancy.
In Step1: We first show for any  $k \in [K], h \in [H], (s, a) \in \mathcal{S} \times \mathcal{A}$ , with probability  $1 - \delta$
$$
\left| \phi (s, a) ^ {\mathrm {T}} \left(\widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k}\right) \right| \leq \sqrt {2 \nu^ {2} \log (2 / \delta)} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}.
$$
Indeed, by design of Algorithm 1,  $\widetilde{w}_h^k\sim \mathcal{N}(\widehat{w}_h^k,\nu^2 (\Omega_h^k)^{-1})$  , which gives,
$$
\phi (s, a) ^ {\mathrm {T}} (\widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k}) \sim \mathcal {N} (0, \nu^ {2} \phi (s, a) ^ {\mathrm {T}} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s, a)).
$$
Therefore,  $\phi(s, a)^{\mathrm{T}} (\widetilde{w}_h^k - \widehat{w}_h^k)$  is  $\nu^2 \phi(s, a)^{\mathrm{T}} (\Omega_h^k)^{-1} \phi(s, a)$ -sub-Gaussian. By concentration of sub-Gaussian random variables, we have
$$
\mathbb {P} \left(\left| \phi (s, a) ^ {\mathrm {T}} (\widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k}) \right| \geq t\right) \leq 2 \exp \left(- \frac {t ^ {2}}{2 \nu^ {2} \phi (s , a) ^ {\mathrm {T}} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)}\right) := \delta
$$
Solving for  $\delta$  gives with probability  $1 - \delta$ ,
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k}) \right| \leq \sqrt {2 \nu^ {2} \phi (s , a) ^ {\top} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a) \log (2 / \delta)} = \sqrt {2 \nu^ {2} \log (2 / \delta)} \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}}
$$
Step2: For any  $0 < \delta < 1$ , define the event  $\widetilde{E}$  as
$$
\begin{array}{l} \widetilde {E} = \left\{\left| \phi (s, a) ^ {\mathrm {T}} \widetilde {w} _ {h} ^ {k} - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| \leq \sqrt {2 \nu^ {2} \log (2 C _ {d} H K / \delta)} \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} + \frac {1}{K ^ {3}}, \right. \\ \left. \forall k \in [ K ], h \in [ H ], s \in \mathcal {S}, a \in \mathcal {A} \right\}, \tag {12} \\ \end{array}
$$
then  $\widetilde{E}$  happens with probability  $1 - \delta$ . Here  $\log C_d = d\log (1 + (8\sqrt{2\nu^2\log(2 / \delta) / \lambda} +8H\sqrt{d})K^3)$ . In Lemma D.12, set  $\theta = \widetilde{w}_h^k -\widehat{w}_h^k$  and  $A = (\Omega_h^k)^{-1}$  and  $B = 1 / \lambda$ , and let  $\mathcal{V}$  be the  $\frac{1}{2K^3}$ -epsilon net for the class of values  $\{| \langle \phi ,\widetilde{w}_h^k -\widehat{w}_h^k\rangle | - C\sqrt{\phi^\top(\Omega_h^k)^{-1}\phi}:\| \phi \| \leq 1\}$  (where  $C = \sqrt{2\nu^2\log(2 / \delta)}$ ), then it must also be the  $\frac{1}{2K^3}$ -epsilon net for the class of values  $\mathcal{F} = \{| \langle \phi (s,a),\widetilde{w}_h^k -\widehat{w}_h^k\rangle |$ $C\sqrt{\phi(s,a)^{\top}(\Omega_h^k)^{-1}\phi(s,a)}:(s,a)\in S\times \mathcal{A}\}$ , let  $\bar{\mathcal{V}}$  is the smallest subset of  $\mathcal{V}$  such that it is
$\frac{1}{2K^3}$ -epsilon net for the class of values  $\mathcal{F}$ . Then we can select  $\nu_{S\times A}$  to be the set of state-action pairs such that for any  $f_{\phi} := |\langle \phi, \widetilde{w}_h^k - \widehat{w}_h^k \rangle| - C \sqrt{\phi^\top(\Omega_h^k)^{-1}\phi} \in \bar{\nu}$ , there exists  $(s,a) \in \nu_{S\times A}$  satisfies  $|\langle \phi(s,a), \widetilde{w}_h^k - \widehat{w}_h^k \rangle|C \sqrt{\phi(s,a)^\top(\Omega_h^k)^{-1}\phi(s,a)| - f_{\phi}} \leq 1/2K^3$ , then we have  $\nu_{S\times A}$  is a  $1/K^3$ -epsilon net of  $\mathcal{F}$  and  $|\nu_{S\times A}| \leq |\bar{\nu}| \leq |\mathcal{V}|$ . Therefore,
$$
\begin{array}{l} \sup _ {s, a} | \langle \phi (s, a), \widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k} \rangle | - C \sqrt {\phi (s , a) ^ {\top} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)} \\ \leq \sup  _ {(s, a) \in \mathcal {V} _ {S \times \mathcal {A}}} | \langle \phi (s, a), \widetilde {w} _ {h} ^ {k} - \widehat {w} _ {h} ^ {k} \rangle | - C \sqrt {\phi (s , a) ^ {\top} (\Omega_ {h} ^ {k}) ^ {- 1} \phi (s , a)} + 1 / K ^ {3} \\ \end{array}
$$
Then by a union bound over  $\left(1 + \left(8\sqrt{2\nu^2\log(2 / \delta) / \lambda}\right) + 8H\sqrt{d}\right)K^3)^d$ ,  $H$  and  $K$ , we have the stated result.
Step3: Note  $\widetilde{Q}_h^k = \max_m\phi^\mathrm{T}\widetilde{w}_h^{k,m}$ , hence by a union bound over  $M$ , we have
$$
\begin{array}{l} \left| \widetilde {Q} _ {h} ^ {k} (s, a) - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| = \left| \max  _ {m} \phi (s, a) ^ {\mathrm {T}} \widetilde {w} _ {h} ^ {k, m} - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right| \\ \leq \max _ {m} | \phi (s, a) ^ {\mathrm {T}} \widetilde {w} _ {h} ^ {k, m} - \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} | \\ \leq \sqrt {2 \nu^ {2} \log (2 C _ {d} H M K / \delta)} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}} \\ \end{array}
$$
for all  $k, h, s, a$  with probability  $1 - \delta$ . Here the last inequality follows Step2, which completes the proof.
Lemma B.10 (Concentration of  $R_2$ ). For any  $0 < \delta < 1$ , with probability  $1 - \delta$ , for all  $k \in [K]$ ,  $h \in [H]$ ,  $s \in S$ ,  $a \in \mathcal{A}$ , it holds
$$
\left| \widehat {Q} _ {h} ^ {k} (s, a) - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a) \right| \leq C _ {\delta} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}
$$
where  $C_{\delta} = \sqrt{8H^{2}\left[\frac{d}{2}\log\left(\frac{k + \lambda}{\lambda}\right) + dM\log(1 + \frac{2\sqrt{8k^{3}}C_{H,d,k,M,\delta}}{H\sqrt{\lambda}}) + \log\frac{2}{\delta}\right]} + 2\sqrt{\lambda}\sqrt{d}H$  and the quantity  $C_{H,d,k,M,\delta} = 2H\sqrt{\frac{dk}{\lambda}} + \frac{\nu\sqrt{2d} + \nu\sqrt{2\log(M / \delta)}}{\sqrt{\lambda}}$ .
Proof of Lemma B.10. For any  $(k,h)\in [K]\times [H]$  and  $(s,a)\in S\times \mathcal{A}$ , denote
$$
\phi (s, a) ^ {\mathrm {T}} w _ {h} ^ {k} := \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \text {w h e r e} w _ {h} ^ {k} := \theta_ {h} + \int_ {\mathcal {S}} \widetilde {V} _ {h + 1} ^ {k} \left(s ^ {\prime}\right) \mathrm {d} \mu_ {h} \left(s ^ {\prime}\right).
$$
Recall  $y_h^\tau = \mathbb{1}_{\tau, k-1} \cdot [r_h^\tau(s_h^\tau, a_h^\tau) + \widetilde{V}_{h+1}^k(s_{h+1}^\tau)]$  from Algorithm 1 and denote  $\bar{y}_h^\tau := r_h^\tau(s_h^\tau, a_h^\tau) + \widetilde{V}_{h+1}^k(s_{h+1}^\tau)$ . Then by definition,
$$
\widehat {w} _ {h} ^ {k} = (\Omega_ {h} ^ {k}) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi (s _ {h} ^ {\tau}, a _ {h} ^ {\tau}) y _ {h} ^ {\tau} = (\Omega_ {h} ^ {k}) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi (s _ {h} ^ {\tau}, a _ {h} ^ {\tau}) \bar {y} _ {h} ^ {\tau}.
$$
From  $\Omega_h^k$  defined in line 7 of Algorithm 1, we have  $\Phi_h\Phi_h^{\mathrm{T}} = \Omega_h^k -\lambda I$ . Plug it into the definition of  $\widehat{w}_h^k$ , we have
$$
\begin{array}{l} \widehat {w} _ {h} ^ {k} = \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\bar {y} _ {h} ^ {\tau} - \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\mathrm {T}} w _ {h} ^ {k} + \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\mathrm {T}} w _ {h} ^ {k}\right) \\ = \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\bar {y} _ {h} ^ {\tau} - \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\mathrm {T}} w _ {h} ^ {k}\right) + \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \left(\Omega_ {h} ^ {k} - \lambda I\right) w _ {h} ^ {k}. \\ \end{array}
$$
We then proceed to bound  $\widehat{w}_h^k - w_h^k$ , which gives
$$
\begin{array}{l} \widehat {w} _ {h} ^ {k} - w _ {h} ^ {k} = \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\bar {y} _ {h} ^ {\tau} - \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\mathrm {T}} w _ {h} ^ {k}\right) - \lambda \left(\Omega_ {h} ^ {k}\right) ^ {- 1} w _ {h} ^ {k} \\ = \underbrace {(\Omega_ {h} ^ {k}) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi (s _ {h} ^ {\tau} , a _ {h} ^ {\tau}) \left(\widetilde {V} _ {h + 1} ^ {k} (s _ {h + 1} ^ {\tau}) - \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} (s _ {h} ^ {\tau} , a _ {h} ^ {\tau})\right)} _ {\text {(i)}} - \underbrace {\lambda (\Omega_ {h} ^ {k}) ^ {- 1} w _ {h} ^ {k}} _ {\text {(i i)}}. \\ \end{array}
$$
Term (i). Since  $\Omega_h^k$  is positive definite, multiplying the first term  $(i)$  with  $\phi(s, a)$  and by Cauchy-Schwartz inequality, we obtain,
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\mathbf {i}) \right| \leq \left\| \phi (s, a) \right\| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi (s _ {h} ^ {\tau}, a _ {h} ^ {\tau}) \left(\widetilde {V} _ {h + 1} ^ {k} (s _ {h + 1} ^ {\tau}) - \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} (s _ {h} ^ {\tau}, a _ {h} ^ {\tau})\right) \right\| _ {(\Omega_ {h} ^ {k}) ^ {- 1}}.
$$
Apply Lemma B.11, we have with probability at least  $1 - \delta$ , for any  $(k,h) \in [K] \times [H]$ , and  $(s,a) \in S \times \mathcal{A}$ ,