SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\mathrm {i}) \right| \leq C _ {1} \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}, \tag {13}
$$
where  $C_1 = \sqrt{8H^2\left[\frac{d}{2}\log\left(\frac{k + \lambda}{\lambda}\right) + dM\log(1 + \frac{2\sqrt{8k^3}C_{H,d,k,M,\delta}}{H\sqrt{\lambda}}) + \log\frac{2}{\delta}\right]}$ .
Term (ii). By Lemma B.12,  $\forall (s,a)\in \mathcal{S}\times \mathcal{A}$ , and  $(k,h)\in [K]\times [H]$ ,  $|\phi (s,a)^{\mathrm{T}}(\mathrm{ii})|$  can be bounded as
$$
\left| \phi (s, a) ^ {\mathrm {T}} (\mathrm {i i}) \right| = \lambda \left| \phi (s, a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} w _ {h} ^ {k} \right| \leq 2 \sqrt {\lambda} \sqrt {d} H \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}. \tag {14}
$$
Combining (13), (14), we have with probability  $1 - \delta$ , for any  $(k, h) \in [K] \times [H]$  and  $(s, a) \in S \times \mathcal{A}$ ,
$$
\begin{array}{l} \left| \widehat {Q} _ {h} ^ {k} (s, a) - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a) \right| = \left| \phi (s, a) ^ {\mathrm {T}} \left(\widehat {w} _ {h} ^ {k} - w _ {h} ^ {k}\right) \right| \leq \left| \phi (s, a) ^ {\mathrm {T}} (\mathrm {i}) \right| + \left| \phi (s, a) ^ {\mathrm {T}} (\mathrm {i i}) \right| \\ \leq \left(C _ {1} + 2 \sqrt {\lambda} \sqrt {d} H\right) \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}, \\ \end{array}
$$
Lemma B.11. For any  $0 < \delta < 1$ , with probability  $1 - \delta$ , we have  $\forall (k,h) \in [K] \times [H]$ ,
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \cdot \phi (s _ {h} ^ {\tau}, a _ {h} ^ {\tau}) \left(\widetilde {V} _ {h + 1} ^ {k} (s _ {h + 1} ^ {\tau}) - \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} (s _ {h} ^ {\tau}, a _ {h} ^ {\tau})\right) \right\| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} ^ {2} \\ \leq 8 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log \left(1 + \frac {2 \sqrt {8 k ^ {3}} C _ {H , d , k , M , \delta}}{H \sqrt {\lambda}}\right) + \log \frac {2}{\delta} \right], \\ \end{array}
$$
$$
h e r e C _ {H, d, k, M, \delta} = 2 H \sqrt {\frac {d k}{\lambda}} + \frac {\nu \sqrt {2 d} + \nu \sqrt {2 \log (M / \delta)}}{\sqrt {\lambda}}. ^ {8}
$$
Proof of Lemma B.11. First note that
$$
\begin{array}{l} \widetilde {V} _ {h} ^ {k} (\cdot) := \max  _ {a} \min  \{\widetilde {Q} _ {h} ^ {k} (\cdot , a), (H - h + 1) \} = \max  _ {a} \min  \max  _ {m} \{\widetilde {Q} _ {h} ^ {k, m}, (H - h + 1) \} \\ = \max  _ {a} \min  \{\max  _ {m} \phi (\cdot , a) ^ {\mathrm {T}} \widetilde {w} _ {h} ^ {k, m}, (H - h + 1) \}. \\ \end{array}
$$
Recall that  $(\Omega_h^k)^{1/2}(\widetilde{w}_h^{k,m} - \widehat{w}_h^k) / \nu \sim \mathcal{N}(0, I_d)$ , then by Lemma D.7, with probability  $1 - \delta/2$ , we have
$$
\frac {\sqrt {\lambda}}{\nu} \left\| \widetilde {w} _ {h} ^ {k, m} - \widehat {w} _ {h} ^ {k} \right\| \leq \frac {1}{\nu} \left\| (\Omega_ {h} ^ {k}) ^ {1 / 2} (\widetilde {w} _ {h} ^ {k, m} - \widehat {w} _ {h} ^ {k}) \right\| \leq \sqrt {2 d} + \sqrt {2 \log (1 / \delta)}.
$$
Note here  $\nu$  is in the line 10 of Algorithm 1. At the end we will choose  $\nu$  to be  $\mathrm{Poly}(H,d,K)$  and this will not affect the overall dependence of the guarantee since  $C_{H,d,k,M,\delta}$  is inside the log term.
Apply the union bound over all  $m$ , then above implies with probability  $1 - \delta / 2, \forall m \in [M]$
$$
\left\| \widetilde {w} _ {h} ^ {k, m} \right\| \leq \left\| \widehat {w} _ {h} ^ {k} \right\| + \frac {\nu \sqrt {2 d} + \nu \sqrt {2 \log (M / \delta)}}{\sqrt {\lambda}} \leq 2 H \sqrt {\frac {d k}{\lambda}} + \frac {\nu \sqrt {2 d} + \nu \sqrt {2 \log (M / \delta)}}{\sqrt {\lambda}} := C _ {H, d, k, M, \delta}. \tag {15}
$$
Now consider the function class  $\bar{\mathcal{V}} := \{\max_a \max_m \phi(\cdot, a)^\mathrm{T} w^m : \|w^m\| \leq C_{H,d,k,M,\delta}\}$ , so by Lemma D.13 the  $\epsilon$ -log covering number for  $\bar{\mathcal{V}}$  is  $dM \log(1 + \frac{2C_{H,d,k,M,\delta}}{\epsilon})$ . Since  $\min\{\cdot, \cdot\}$  is a non-expansive operator, the  $\epsilon$ -log covering number for the function class  $\mathcal{V} := \{\max_a \min\{\max_m \phi(\cdot, a)^\mathrm{T} w^m, (H - h + 1)\} : \|w^m\| \leq C_{H,d,k,M,\delta}\}$ , is at most  $dM \log(1 + \frac{2C_{H,d,k,M,\delta}}{\epsilon})$ . Hence, for any  $V \in \mathcal{V}$ , there exists  $V'$  in the  $\epsilon$ -covering such that  $V = V' + \Delta_V$  with  $\|\Delta_V\|_{\infty} \leq \epsilon$ . Then with probability  $1 - \delta/2$ ,
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(V \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} V \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} ^ {2} \\ \leq 2 \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(V ^ {\prime} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} V ^ {\prime} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ + 2 \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\Delta_ {V} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} \Delta_ {V} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \tag {16} \\ \leq 2 \left\| \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(V ^ {\prime} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} V ^ {\prime} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} + \frac {8 k ^ {2} \epsilon^ {2}}{\lambda} \\ \leq 4 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log \left(1 + \frac {2 C _ {H , d , k , M , \delta}}{\epsilon}\right) + \log \frac {2}{\delta} \right] + \frac {8 k ^ {2} \epsilon^ {2}}{\lambda} \\ \end{array}
$$
where the second inequality can be conducted using a direct calculation and the third inequality uses Lemma D.9 and a union bound over the covering number. Now by (15) and (16) and a union bound, we have for any  $\epsilon > 0$ , with probability  $1 - \delta$ ,
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(\widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq 4 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log (1 + \frac {2 C _ {H , d , k , M , \delta}}{\epsilon}) + \log \frac {2}{\delta} \right] + \frac {8 k ^ {2} \epsilon^ {2}}{\lambda} \\ \leq 8 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log (1 + \frac {2 \sqrt {8 k ^ {3}} C _ {H , d , k , M , \delta}}{H \sqrt {\lambda}}) + \log \frac {2}{\delta} \right], \\ \end{array}
$$
where the last step choose  $\epsilon^2 = H^2\lambda /8k^2$  so  $\frac{8k^2\epsilon^2}{\lambda}\leq 4H^2$ . Lastly, apply the union bound over  $H,K$  to obtain the stated result.
Lemma B.12.  $\forall (s,a)\in \mathcal{S}\times \mathcal{A},h\in [H],k\in [K],$  it holds that
$$
\left| \phi (s, a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} w _ {h} ^ {k} \right| \leq \frac {2}{\sqrt {\lambda}} \sqrt {d} H \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}}.
$$
Proof of Lemma B.12. Note that the precision matrix  $\Omega_h^k$  for any step  $h$  and episode  $k$  is always positive definite. By Cauchy-Schwartz inequality and Lemma D.1,
$$
\begin{array}{l} \left| \phi (s, a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} w _ {h} ^ {k} \right| = \left| \phi (s, a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1 / 2} \left(\Omega_ {h} ^ {k}\right) ^ {- 1 / 2} w _ {h} ^ {k} \right| \\ \leq \left\| \phi (s, a) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \left\| w _ {h} ^ {k} \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \\ \leq \| \phi (s, a) \| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} \sqrt {\left\| w _ {h} ^ {k} \right\| ^ {2} \left\| (\Omega_ {h} ^ {k}) ^ {- 1} \right\|} \\ \leq \| \phi (s, a) \| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \left\| w _ {h} ^ {k} \right\| \frac {1}{\sqrt {\lambda_ {\operatorname* {m i n}} \left(\Omega_ {h} ^ {k}\right)}} \\ \end{array}
$$
Note that  $\lambda_{\min}(\Omega_h^k) \geq \lambda$ . Applying Lemma D.3 for  $\| w_h^k\|$  completes the proof.
C Regret Analysis for Delayed-LPSVI
Proof of Theorem 2. The proof structure is similar to that of Theorem 1. We proceed by bounding  $\Delta_{opt}^{k}$  and  $\Delta_{est}^{k}$  respectively.
Step 1: bound regret from optimism. By Lemma C.5, with probability  $1 - \delta /2$
$$
\Delta_ {o p t} ^ {k} := V _ {1} ^ {*} \left(s _ {1} ^ {k}\right) - \widetilde {V} _ {1} ^ {k} \left(s _ {1} ^ {k}\right) \leq 0, \quad \forall k \in [ K ].
$$
Step 2: bound regret from estimation error. We first condition on the event that
$$
\mathcal {E} := \{\left| \right.\left| \min  \left\{\widetilde {Q} _ {h} ^ {k} (s, a), H - h + 1 \right\} - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a) \right| \leq \beta \left\| \phi (s, a) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}}, \forall s, a, h, k \},
$$
$$
\text {w i t h} \beta := \sqrt {2 \gamma \log (1 6 C _ {d} H M K / \delta)} + \sqrt {8 H ^ {2} \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + d M \log \left(1 + \frac {2 \sqrt {8 k ^ {3}} C _ {H , d , k , M , \delta / 8}}{H \sqrt {\lambda}}\right) + \log \frac {1 6}{\delta} \right]} +
$$
$2\sqrt{\lambda}\sqrt{d} H.$  Here  $C_d$  and  $C_{H,d,k,M,\delta}$  are defined in Lemma C.6.
Similarly, define
$$
\zeta_ {h} ^ {k} = \mathbb {E} \left[ \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {k}\right) - V _ {h + 1} ^ {\pi_ {k}} \left(s _ {h + 1} ^ {k}\right) \mid s _ {h} ^ {k}, a _ {h} ^ {k} \right] - \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {k}\right) + V _ {h + 1} ^ {\pi_ {k}} \left(s _ {h + 1} ^ {k}\right).
$$
Then by Lemma C.1,
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \Delta_ {e s t} ^ {k} = \sum_ {k = 1} ^ {K} \widetilde {V} _ {1} ^ {k} \left(s _ {1} ^ {k}\right) - V _ {1} ^ {\pi_ {k}} \left(s _ {1} ^ {k}\right) \\ \leq \sum_ {k = 1} ^ {K} \left(\widetilde {V} _ {2} ^ {k} \left(s _ {2} ^ {k}\right) - V _ {2} ^ {\pi_ {k}} \left(s _ {2} ^ {k}\right) + \zeta_ {1} ^ {k} + \beta \left\| \phi \left(s _ {1} ^ {k}, a _ {1} ^ {k}\right) \right\| _ {\left(\Omega_ {1} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}}\right) \tag {17} \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \zeta_ {h} ^ {k} + \beta \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {H}{K ^ {2}}. \\ \end{array}
$$
By definition,  $|\zeta_h^k| \leq 2H$  for all  $h \in [H], k \in [K]$ , therefore  $\{\zeta_h^k\}$  is a martingale difference sequence. By Azuma-Hoeffding's inequality,
$$
\mathbb {P} \left(\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \zeta_ {h} ^ {k} > t\right) \geq \exp \left(\frac {- t ^ {2}}{2 K \cdot H ^ {3}}\right) := \delta / 8, \quad \forall t > 0.
$$
Thus, with probability  $1 - \delta / 8$ ,
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \zeta_ {h} ^ {k} \leq \sqrt {2 K H ^ {3} \cdot \log (8 / \delta)} = \sqrt {2 H ^ {2} T \cdot \log (8 / \delta)}. \tag {18}
$$
$$
\beta \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \leq \beta H \sqrt {2 d K \log ((d + K) / d)} + \beta d H D _ {\tau , \delta , H, K} \log ((d + K) / d). \tag {19}
$$
Here  $D_{\tau, \delta, H, K} := 1 + 2\mathbb{E}[\tau] + 2\sqrt{2\mathbb{E}[\tau]\log(\frac{24KH}{\delta})} + \frac{4}{3}\log(\frac{24KH}{\delta}) + D_{\tau, K, \frac{\delta}{16H}}$  and  $D_{\tau, K, \delta}$  is defined in Lemma D.6. By Lemma C.6, event  $\mathcal{E}$  holds with probability  $1 - \delta/4$ , by a union bound with (18) and (19), we have with probability  $1 - \delta/2$ ,
where  $c > 0$  is some universal constant and  $\iota$  is a Polylog term of  $H, d, K, \delta$ . Similarly, we can bound  $\beta \leq C d H \iota_{\delta}$  for some universal constant  $C$ , and it is readily to verify  $D_{\tau, \delta, H, K}$  is bounded by  $c' \mathbb{E}[\tau] \iota + O(\iota)$ .
Lemma C.1. Define  $\zeta_h^k = \mathbb{E}[\widetilde{V}_{h + 1}^k (s_{h + 1}^k) - V_{h + 1}^{\pi_k}(s_{h + 1}^k)|s_h^k,a_h^k ] - \widetilde{V}_{h + 1}^k (s_{h + 1}^k) + V_{h + 1}^{\pi_k}(s_{h + 1}^k)$  and condition on the event (21) in Lemma C.6. Then for all  $k\in [K]$ ,  $h\in [H]$ , the following holds,
Proof of Lemma C.1. By the event defined in (21), the proof follows exactly as in that of Lemma B.1.
C.1 Convergence of Langevin Monte Carlo
The following lemma is crucial to prove the optimism and bound the error in Langevin analysis. For ease of notation, within the episode  $k$ , we simply use  $\eta$  to denote  $\eta_{k}$  for conciseness.
Lemma C.2 (Convergence of LMC). Denote  $\{\widetilde{w}_h^{k,m}\}_{m\in [M]}$  to be the weights returned by Line 6 of Algorithm 2. Set  $\eta = \frac{1}{4\lambda_{\max}(\Omega_h^k)}$ , we have
$$
\widetilde {w} _ {h} ^ {k, m} \sim \mathcal {N} \left(A _ {h, k} ^ {N _ {k}} w _ {0} + \left(I - A _ {h, k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k}, \Theta_ {h} ^ {k}\right) \quad \forall m \in [ M ]
$$
$$
A _ {h, k} := I - 2 \eta \Omega_ {h} ^ {k}
$$
$$
\Omega_ {h} ^ {k} := \lambda I + \sum_ {k = 1} ^ {K} \phi_ {h} \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \phi_ {h} \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {T}
$$
$$
\widehat {w} _ {h} ^ {k} := (\Omega_ {h} ^ {k}) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \phi_ {h} (s _ {h} ^ {\tau}, a _ {h} ^ {\tau}) y _ {h} ^ {\tau}
$$
$$
\Theta_ {h} ^ {k} := \gamma (I - A _ {h, k} ^ {2 N _ {k}}) (\Omega_ {h} ^ {k}) ^ {- 1} (I + A _ {h, k}) ^ {- 1}.
$$
$$
\frac {\gamma}{2} \left(1 - \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {2 N _ {k}}\right) \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \prec \Theta_ {h} ^ {k} \prec \gamma \left(\Omega_ {h} ^ {k}\right) ^ {- 1},
$$
where  $\kappa_h \coloneqq \frac{\lambda_{\max}(\Omega_h^k)}{\lambda_{\min}(\Omega_h^k)}$  is the condition number.
Proof of Lemma C.2. Let  $b_h^k \coloneqq \sum_{\tau=1}^{k-1} \phi(s_h^\tau, a_h^\tau) y_h^\tau$ , then
$$
\nabla L _ {h} ^ {k} (w) = 2 \Omega_ {h} ^ {k} w - 2 b _ {h} ^ {k}.
$$
Therefore, fix  $h,k,m$ , and within the Algorithm 3 we have
$$
\begin{array}{l} w _ {N} = w _ {N - 1} - 2 \eta (\Omega_ {h} ^ {k} \cdot w _ {N - 1} - b _ {h} ^ {k}) + \sqrt {2 \eta \gamma} \epsilon_ {N} \\ = (I - 2 \eta \Omega_ {h} ^ {k}) w _ {N - 1} + 2 \eta b _ {h} ^ {k} + \sqrt {2 \eta \gamma} \epsilon_ {N} \\ = A _ {h, k} w _ {N - 1} + 2 \eta b _ {h} ^ {k} + \sqrt {2 \eta \gamma} \epsilon_ {N} \\ = A _ {h, k} ^ {N} w _ {0} + 2 \eta \sum_ {l = 0} ^ {N - 1} A _ {h, k} ^ {l} b _ {h} ^ {k} + \sqrt {2 \eta \gamma} \sum_ {l = 0} ^ {N - 1} A _ {h, k} ^ {l} \epsilon_ {N - l} \\ = A _ {h, k} ^ {N} w _ {0} + \left(I - A _ {h, k} ^ {N}\right) \widehat {w} _ {h} ^ {k} + \sqrt {2 \eta \gamma} \sum_ {l = 0} ^ {N - 1} A _ {h, k} ^ {l} \epsilon_ {N - l} \\ \end{array}
$$
where the last equality uses  $(\Omega_h^k)^{-1}b_h^k = \widehat{w}_h^k$  and  $I\succ I - 2\eta \Omega_h^k\succ \mathbf{0}$ , so  $\sum_{l = 0}^{N - 1}A^l = (I - A^N)(I - A)^{-1}$ . Since  $\epsilon_{i}$  are i.i.d gaussian noise, from the above we directly have
$$
w _ {N} \sim \mathcal {N} (A _ {h, k} ^ {N} w _ {0} + (I - A _ {h, k} ^ {N}) \widehat {w} _ {h} ^ {k}, \Theta_ {h} ^ {k})
$$
$$
\begin{array}{l} \Theta_ {h} ^ {k} = \operatorname {C o v} [ \sqrt {2 \eta \gamma} \sum_ {l = 0} ^ {N - 1} A _ {h, k} ^ {l} \epsilon_ {N - l} ] = 2 \eta \gamma \cdot \operatorname {C o v} [ \sum_ {l = 0} ^ {N - 1} A _ {h, k} ^ {l} \epsilon_ {N - l} ] \\ = 2 \eta \gamma \cdot \sum_ {l = 0} ^ {N - 1} A _ {h, k} ^ {2 l} = 2 \eta \gamma (I - A _ {h, k} ^ {2 N}) (I - A _ {h, k} ^ {2}) ^ {- 1} \\ = \gamma (I - A _ {h, k} ^ {2 N _ {k}}) (\Omega_ {h} ^ {k}) ^ {- 1} (I + A _ {h, k}) ^ {- 1}. \\ \end{array}
$$
Next, due to the choice of  $\eta = \frac{1}{4\lambda_{\max}(\Omega_h^k)}$ , we have
$$
\begin{array}{l} \frac {1}{2} I \prec A _ {h, k} = I - 2 \eta \Omega_ {h} ^ {k} \prec (1 - 2 \eta \lambda_ {\min} (\Omega_ {h} ^ {k})) I \\ \Rightarrow \frac {1}{2 ^ {2 N}} I \prec A _ {h, k} ^ {2 N} \prec (1 - 2 \eta \lambda_ {\min } \left(\Omega_ {h} ^ {k}\right)) ^ {2 N} I \tag {20} \\ \Rightarrow \left(1 - \left(1 - 2 \eta \lambda_ {\min } \left(\Omega_ {h} ^ {k}\right)\right) ^ {2 N}\right) I \prec I - A _ {h, k} ^ {2 N} \prec \left(1 - \frac {1}{2 ^ {2 N}}\right) I \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{2} I \prec A _ {h, k} = I - 2 \eta \Omega_ {h} ^ {k} \prec (1 - 2 \eta \lambda_ {\min} (\Omega_ {h} ^ {k})) I \\ \Rightarrow \frac {3}{2} I \prec I + A _ {h, k} \prec (2 - 2 \eta \lambda_ {\min} (\Omega_ {h} ^ {k})) I \\ \Rightarrow \frac {1}{2 - 2 \eta \lambda_ {\min} \left(\Omega_ {h} ^ {k}\right)} I \prec \left(I + A _ {h, k}\right) ^ {- 1} \prec \frac {2}{3} I \\ \end{array}
$$
The above two implies
$$
\begin{array}{l} \gamma \frac {\left(1 - \left(1 - 2 \eta \lambda_ {\min } \left(\Omega_ {h} ^ {k}\right)\right) ^ {2 N}\right)}{2 - 2 \eta \lambda_ {\min } \left(\Omega_ {h} ^ {k}\right)} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \prec \Theta_ {h} ^ {k} \prec \gamma \frac {2}{3} \left(1 - \frac {1}{2 ^ {2 N}}\right) \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \\ \Rightarrow \gamma \frac {\left(1 - \left(1 - 2 \eta \lambda_ {\min } \left(\Omega_ {h} ^ {k}\right)\right) ^ {2 N}\right)}{2} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \prec \Theta_ {h} ^ {k} \prec \gamma \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \\ \end{array}
$$
Replacing  $N$  with  $N_{k}$  and  $w_{N}$  with  $\widetilde{w}_h^{k,m}$  for all  $m\in [M]$  completes the proof.
C.2 Proofs of optimism for Delayed-LPSVI
Lemma C.3 (Anti-concentration for Optimism). Suppose the event
holds. Choose  $N_{k} \geq \max \{\log \left(\frac{32H^{2}(K + \lambda)dk}{\gamma\lambda} + 1\right) / \left[2\log \left(1 / \left(1 - \frac{1}{2\kappa_{h}}\right)\right)\right], \frac{\log 2}{2\log \left(1 / \left(1 - \frac{1}{2\kappa_{h}}\right)\right)}\}$ ,  $\gamma = 16C_{\delta'}^2$  and  $M_{\delta} = \log (HK / \delta) / \log (64 / 63)$ . Then we have with probability  $1 - \delta$ ,
Proof of Lemma C.3. For the rest of the proof, we condition on the event
where  $\delta^\prime$  will be specified later and  $C_\delta$  is defined in the Lemma C.9.
$$
\phi (s, a) ^ {\mathrm {T}} \left(\widetilde {w} _ {h} ^ {k} - \left(I - A _ {h, k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k}\right) \sim \mathcal {N} \left(0, \phi (s, a) ^ {\mathrm {T}} \Theta_ {h} ^ {k} \phi (s, a)\right).
$$
Also note
$$
\begin{array}{l} \bar {Q} _ {h} ^ {k, m} (s, a) - \phi (s, a) ^ {\mathrm {T}} \left(I - A _ {h, k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k} \sim \mathcal {N} \left(0, \phi (s, a) ^ {\mathrm {T}} \Theta_ {h} ^ {k} \phi (s, a)\right) \\ \Leftrightarrow \frac {\widetilde {Q} _ {h} ^ {k , m} (s , a) - \phi (s , a) ^ {\mathrm {T}} (I - A _ {h , k} ^ {N _ {k}}) \widehat {w} _ {h} ^ {k}}{\sqrt {\phi (s , a) ^ {\mathrm {T}} \Theta_ {h} ^ {k} \phi (s , a)}} \sim \mathcal {N} (0, 1). \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} \left(\widetilde {Q} _ {h} ^ {k, m} (s, a) \geq \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) (s, a), \forall s, a\right) \\ = \mathbb {P} \left(\frac {\widetilde {Q} _ {h} ^ {k , m} (s , a) - \phi (s , a) ^ {\mathrm {T}} \left(I - A _ {h , k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k}}{\sqrt {\phi (s , a) ^ {\mathrm {T}} \Theta_ {h} ^ {k} \phi (s , a)}} \geq \frac {\left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) (s , a) - \phi (s , a) ^ {\mathrm {T}} \left(I - A _ {h , k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k}}{\sqrt {\phi (s , a) ^ {\mathrm {T}} \Theta_ {h} ^ {k} \phi (s , a)}}, \forall s, a\right) \\ = \mathbb {P} \left(\mathcal {N} (0, 1) \geq \frac {\left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) (s , a) - \phi (s , a) ^ {\mathrm {T}} \left(I - A _ {h , k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k}}{\sqrt {\phi (s , a) ^ {\mathrm {T}} \Theta_ {h} ^ {k} \phi (s , a)}}, \forall s, a\right) \\ \geq \mathbb {P} \left(\mathcal {N} (0, 1) \geq \frac {\left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) (s , a) - \phi (s , a) ^ {\mathrm {T}} \left(I - A _ {h , k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k}}{\sqrt {\frac {\gamma}{2} \left(1 - \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {2 N _ {k}}\right)} \phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)}, \forall s, a\right) \\ \geq \mathbb {P} \left(\mathcal {N} (0, 1) \geq 1\right) \geq \frac {1}{2 \sqrt {8 \pi}} e ^ {- 1 / 2} \geq \frac {1}{6 4}, \\ \end{array}
$$
where the first two inequalities follow Lemma C.2 and Lemma C.4 respectively and the thrid inequality results from Lemma D.5. Applying Lemma B.6 with  $f = r_h + \mathbb{P}_h\widetilde{V}_{h + 1}^k$  and without conditioning, for  $M_{\delta} = \log (1 / \delta) / \log (64 / 63)$ ,
Lemma C.4. Suppose the event
holds. Choose  $N_{k} \geq \max \left\{\log \left(\frac{32H^{2}(K + \lambda)dk}{\gamma\lambda} + 1\right) / \left[2\log \left(1 / \left(1 - \frac{1}{2\kappa_{h}}\right)\right)\right], \frac{\log 2}{2\log(1 / (1 - \frac{1}{2\kappa_{h}}))}\right\}$  and  $\gamma = 16C_{\delta^{\prime}}^{2}$ . Then
$$
\frac {\left| \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) (s , a) - \phi (s , a) ^ {\mathrm {T}} \left(I - A _ {h , k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k} \right|}{\sqrt {\frac {\gamma}{2} \left(1 - \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {2 N _ {k}}\right) \phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)}} \leq 1, \quad \forall s, a \in \mathcal {S} \times \mathcal {A}, h \in [ H ], k \in [ K ].
$$
Proof of Lemma C.4. By direct calculation,
$$
\begin{array}{l} \frac {\left| \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) (s , a) - \phi (s , a) ^ {\mathrm {T}} \left(I - A _ {h , k} ^ {N _ {k}}\right) \widehat {w} _ {h} ^ {k} \right|}{\sqrt {\frac {\gamma}{2} \left(1 - \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {2 N _ {k}}\right) \phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)}} \leq \frac {\left| \phi (s , a) ^ {\mathrm {T}} A _ {h , k} ^ {N _ {k}} \widehat {w} _ {h} ^ {k} \right|}{\sqrt {\frac {\gamma}{2} \left(1 - \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {2 N _ {k}}\right) \phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (S , a)}} \\ + \frac {\left| \left(r _ {h} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1}\right) (s , a) - \phi (s , a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k} \right|}{\sqrt {\frac {\gamma}{2} \left(1 - \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {2 N _ {k}}\right) \phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)}} \\ \end{array}
$$
For the first term above, by CS inequality we have
$$
\begin{array}{l} | \phi (s, a) ^ {\mathrm {T}} A _ {h, k} ^ {N _ {k}} \widehat {w} _ {h} ^ {k} | \leq \sqrt {\phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)} \cdot \left\| \left(\Omega_ {h} ^ {k}\right) ^ {1 / 2} A _ {h, k} ^ {N _ {k}} \widehat {w} _ {h} ^ {k} \right\| \\ \leq \sqrt {\phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)} \cdot \left\| \left(\Omega_ {h} ^ {k}\right) ^ {1 / 2} \right\| \| A _ {h, k} \| ^ {N _ {k}} \cdot 2 H \sqrt {\frac {d k}{\lambda}} \\ \leq \sqrt {\phi (s , a) ^ {\mathrm {T}} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi (s , a)} \cdot \sqrt {k + \lambda} \cdot \left(1 - \frac {1}{2 \kappa_ {h}}\right) ^ {N _ {k}} \cdot 2 H \sqrt {\frac {d k}{\lambda}} \\ \end{array}
$$
and this indicates