SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
[2] Marc Abeille and Alessandro Lazaric. Linear thompson sampling revisited. In Artificial Intelligence and Statistics, pages 176-184. PMLR, 2017.
[3] Milton Abramowitz and Irene A Stegun. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, volume 55. US Government Printing Office, 1964.
[4] Mridul Agarwal and Vaneet Aggarwal. Blind decision making: Reinforcement learning with delayed observations. Pattern Recognition Letters, 150:176-182, 2021.
[5] Priyank Agrawal, Jinglin Chen, and Nan Jiang. Improved worst-case regret bounds for randomized least-squares value iteration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6566-6573, 2021.
[6] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning, pages 127-135. PMLR, 2013.
[7] Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. Advances in Neural Information Processing Systems, 30, 2017.
[8] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008.
[9] Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning, pages 463-474. PMLR, 2020.
[10] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
[11] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023.
[12] Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Reinforcement learning with random delays. In International conference on learning representations, 2020.
[13] Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283-1294. PMLR, 2020.
[14] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems, 24, 2011.
[15] Baiming Chen, Mengdi Xu, Zuxin Liu, Liang Li, and Ding Zhao. Delay-aware multiagent reinforcement learning for cooperative and competitive environments. arXiv preprint arXiv:2005.05441, 2020.
[67] Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pages 10746-10756. PMLR, 2020.
[68] Yunchang Yang, Han Zhong, Tianhao Wu, Bin Liu, Liwei Wang, and Simon S Du. A reduction-based framework for sequential decision making with delayed feedback. arXiv preprint arXiv:2302.01477, 2023.
[69] Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 1567-1575. PMLR, 2021.
[70] Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. International Conference on Learning Representations, 2022.
[71] Ming Yin, Mengdi Wang, and Yu-Xiang Wang. Offline reinforcement learning with differentiable function approximation is provably efficient. International Conference on Learning Representations, 2023.
[72] Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics, pages 1954-1964. PMLR, 2020.
[73] Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning, pages 10978-10989. PMLR, 2020.
[74] Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022.
[75] Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021.
[76] Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning, pages 12793-12802. PMLR, 2021.
[77] Zhengyuan Zhou, Renyuan Xu, and Jose Blanchet. Learning in generalized linear contextual bandits with stochastic delays. Advances in Neural Information Processing Systems, 32, 2019.
[78] Julian Zimmert and Yevgeny Seldin. An optimal algorithm for adversarial bandits with arbitrary delays. In International Conference on Artificial Intelligence and Statistics, pages 3285-3294. PMLR, 2020.
A Some Properties
A.1 Properties of Linear MDPs
Lemma A.1. In linear MDPs, the action-value function is also linear in feature map.  $\forall (s,a)\in S\times \mathcal{A},h\in [H]$  and  $\phi \in \mathbb{R}^d$  , under any fixed policy  $\pi$
$$
Q _ {h} ^ {\pi} (s, a) = \phi (s, a) ^ {\mathrm {T}} w _ {h} ^ {\pi},
$$
where  $w_h^\pi \coloneqq \theta_h + \mathbb{E}_\mu [V_{h+1}^\pi(s')]$  and  $w_h \in \mathbb{R}^d$ . As a corollary, there exists  $w_h^*$  such that  $Q_h^* = \phi^{\mathrm{T}} w_h^*$ .
Proof of Lemma A.1. By Bellman equation,
$$
\begin{array}{l} Q _ {h} ^ {\pi} (s, a) = r _ {h} (s, a) + \mathbb {E} _ {s ^ {\prime} \sim \mathbb {P} _ {h} (\cdot | s, a)} \left[ V _ {t + 1} ^ {\pi} \left(s ^ {\prime}\right) \right] \\ = \phi (s, a) ^ {\mathrm {T}} \theta_ {h} + \int V _ {h + 1} ^ {\pi} \left(s ^ {\prime}\right) d \left(\phi (s, a) ^ {\mathrm {T}} \mu_ {h} \left(s ^ {\prime}\right)\right) \\ = \phi (s, a) ^ {\mathrm {T}} w _ {h} ^ {\pi}, \\ \end{array}
$$
where  $w_h^\pi \coloneqq \theta_h + \mathbb{E}_{\mu_h}[V_{h+1}^\pi(s')]$ .
A.2 Worst-case regret as a stronger criterion
We use Theorem 1 as an example. Using the worst-case result, i.e. with probability  $1 - \delta$ ,
Here  $\iota$  has the functional form  $\iota = \operatorname{Polylog}(d, K, H, \delta)$ . Then choosing  $\delta = 1/(HK)$  to obtain with probability  $1 - 1/(HK)$ ,
$$
R (T) \leq c \sqrt {d ^ {3} H ^ {3} T \iota} + c ^ {\prime} d ^ {2} H ^ {2} \mathbb {E} [ \tau ] \iota + c ^ {\prime \prime} \iota := A
$$
for  $\iota = \operatorname{Polylog}(d, K, H)$ . Therefore,
$$
\begin{array}{l} \mathbb {E} [ R (T) ] \leq \mathbb {E} [ R (T) \mathbb {1} _ {\{R (T) \leq A \}} ] + \mathbb {E} [ R (T) \mathbb {1} _ {\{R (T) \geq A \}} ] \\ \leq A \cdot 1 + H K \cdot \mathbb {P} (R (T) \geq A) \leq A + 1. \\ \end{array}
$$
This completes Corollary 1.
A.3 Discussion on the arbitrary delay
For completeness of our study, we also briefly discuss the case when delay is arbitrary. In general, the regret can be (roughly) bounded by  $\widetilde{O}\left(\frac{1}{q}\sqrt{d^3H^3T} + dH^2 d_\tau(q)\right)$  for  $d_\tau(q)$  to be the  $q$ -th quantile of delay  $\tau$ . This could be achieved by creating a low-switching variant of our Theorem 1/Theorem 2 and applying the reduction of the concurrent work [68]. We do not focus on this setting since there is a  $1/q$  blow-up in the main regret that many distributions (e.g. sub-exponential) do not need to sacrifice.
B Regret Analysis for Delayed-PSVI
To proceed with the regret analysis, we introduce some helpful notations. Besides  $\widetilde{Q}_h^k (s,a) =$ $\max_{m}\phi (s,a)^{\mathrm{T}},\widetilde{w}_{h}^{k,m},\widetilde{V}_{h}^{k}(s) = \max_{a}\widetilde{Q}_{h}^{k}(s,a)$  in Algorithm 1, we define
$$
\begin{array}{l} \widehat {Q} _ {h} ^ {k} (s, a) = \phi (s, a) ^ {\mathrm {T}} \widehat {w} _ {h} ^ {k}, \quad \widehat {V} _ {h} ^ {k} (s) = \max  _ {a} \widehat {Q} _ {h} ^ {k} (s, a), \quad \bar {Q} _ {h} ^ {k} = \min  \{\widetilde {Q} _ {h} ^ {k}, H - h + 1 \}; \\ (r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}) (s, a) := \phi (s, a) ^ {\mathrm {T}} w _ {h} ^ {k}, \text {w i t h} w _ {h} ^ {k} := \theta_ {h} + \int_ {\mathcal {S}} \widetilde {V} _ {h + 1} ^ {k} (s ^ {\prime}) \mathrm {d} \mu_ {h} (s ^ {\prime}). \\ \end{array}
$$
Regret decomposition: We start by rewriting regret in terms of value-function error decomposition following the standard analysis of optimistic algorithms [10]:
$$
R (T) = \sum_ {k = 1} ^ {K} \underbrace {V _ {1} ^ {*} (s _ {1} ^ {k}) - \widetilde {V} _ {1} ^ {k} (s _ {1} ^ {k})} _ {\Delta_ {o p t} ^ {k}} + \underbrace {\widetilde {V} _ {1} ^ {k} (s _ {1} ^ {k}) - V _ {1} ^ {\pi_ {k}} (s _ {1} ^ {k})} _ {\Delta_ {e s t} ^ {k}},
$$
where at each episode  $k$ ,  $\Delta_{opt}^{k}$  corresponds to the regret resulting from optimism, and  $\Delta_{est}^{k}$  tracks down the regret incurred from estimation error. Efficient RL algorithms thus need to strike a balance between both terms. More specifically, it is desirable to generate optimistic estimations over the true value function, while keeping estimation error relatively small. By cautious design of noise perturbation, we show in Theorem 1 that Algorithm 1 effectively achieves  $\sqrt{T}$  order regret in episodic MDPs with linear function approximation.
Proof of Theorem 1. The proof proceeds by bounding  $\Delta_{opt}^{k}$  and  $\Delta_{est}^{k}$  respectively.
Step 1: bound regret from optimism.
By Lemma B.7, the optimism provided by our algorithm guarantees with probability at least  $1 - \delta / 2$ , for all  $k \in [K]$ ,  $\Delta_{opt}^{k} \coloneqq V_{1}^{*}(s_{1}^{k}) - \widetilde{V}_{1}^{k}(s_{1}^{k}) \leq 0$ .
Step 2: bound regret from estimation error. To bound the estimation error, we first condition on the following event
$$
\mathcal {E} := \{\left| \right.\left| \min  \{\widetilde {Q} _ {h} ^ {k} (s, a), H - h + 1 \} - (r _ {h} ^ {k} + \mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}) (s, a) \right| \leq \beta \left\| \phi (s, a) \right\| _ {(\Omega_ {h} ^ {k}) ^ {- 1}} + \frac {1}{K ^ {3}}, \forall s, a, h, k \},
$$
with  $\beta := \sqrt{2\nu^2\log(16C_dHMK / \delta)} + \sqrt{8H^2\left[\frac{d}{2}\log\left(\frac{k + \lambda}{\lambda}\right) + dM\log(1 + \frac{2\sqrt{8k^3}C_{H,d,k,M,\delta / 8}}{H\sqrt{\lambda}}) + \log\frac{16}{\delta}\right]} + 2\sqrt{\lambda}\sqrt{d}H.$  Here  $C_d$  and  $C_{H,d,k,M,\delta}$  are defined in Lemma B.8.
Recall that  $\Delta_{est}^{k} := \widetilde{V}_{1}^{k}(s_{1}^{k}) - V_{1}^{\pi_{k}}(s_{1}^{k})$  and define  $\zeta_{h}^{k} = \mathbb{E}[\widetilde{V}_{h + 1}^{k}\left(s_{h + 1}^{k}\right) - V_{h + 1}^{\pi_{k}}\left(s_{h + 1}^{k}\right)|s_{h}^{k},a_{h}^{k}] - \widetilde{V}_{h + 1}^{k}\left(s_{h + 1}^{k}\right) + V_{h + 1}^{\pi_{k}}\left(s_{h + 1}^{k}\right)$ . Then by applying Lemma B.1 recursively, the total estimation error  $\sum_{k = 1}^{K}\Delta_{est}^{k}$  can be decomposed as:
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \Delta_ {e s t} ^ {k} = \sum_ {k = 1} ^ {K} \widetilde {V} _ {1} ^ {k} \left(s _ {1} ^ {k}\right) - V _ {1} ^ {\pi_ {k}} \left(s _ {1} ^ {k}\right) \\ \leq \sum_ {k = 1} ^ {K} \left(\widetilde {V} _ {2} ^ {k} \left(s _ {2} ^ {k}\right) - V _ {2} ^ {\pi_ {k}} \left(s _ {2} ^ {k}\right) + \zeta_ {1} ^ {k} + \beta \left\| \phi \left(s _ {1} ^ {k}, a _ {1} ^ {k}\right) \right\| _ {\left(\Omega_ {1} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}}\right) \tag {6} \\ \leq \dots \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \zeta_ {h} ^ {k} + \beta \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {H}{K ^ {2}}. \\ \end{array}
$$
On one hand, by definition,  $|\zeta_h^k| \leq 2H$  for all  $h \in [H], k \in [K]$ . Therefore,  $\{\zeta_h^k\}$  is a martingale difference sequence (since the computation of  $\widetilde{V}_h^k$  is independent of the new observation at episode  $k$ ). By Azuma-Hoeffding's inequality (for  $t > 0$ ),
$$
\mathbb {P} \left(\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \zeta_ {h} ^ {k} > t\right) \geq \exp \left(\frac {- t ^ {2}}{2 K \cdot H ^ {3}}\right) := \delta / 8,
$$
which implies with probability  $1 - \delta /8$
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \zeta_ {h} ^ {k} \leq \sqrt {2 K H ^ {3} \cdot \log (8 / \delta)} = \sqrt {2 H ^ {2} T \cdot \log (8 / \delta)}. \tag {7}
$$
<sup>7</sup>Note here the  $\delta$  equals  $\delta / 4$  as of Lemma B.8. Therefore, by Lemma B.8,  $\mathbb{P}(\mathcal{E}) \geq 1 - \delta / 4$ .
Step 3: bounding the delayed error. By Lemma B.4, with probability  $1 - \delta /8$
$$
\sum_ {h - 1} ^ {H} \sum_ {k = 1} ^ {K} \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \leq H \sqrt {2 d K \log ((d + K) / d)} + d H D _ {\tau , \delta , H, K} \log ((d + K) / d).
$$
Here  $D_{\tau, \delta, H, K} := 1 + 2\mathbb{E}[\tau] + 2\sqrt{2\mathbb{E}[\tau]\log(\frac{24KH}{\delta})} + \frac{4}{3}\log(\frac{24KH}{\delta}) + D_{\tau, K, \frac{\delta}{16H}}$  and  $D_{\tau, K, \delta}$  is defined in Lemma D.6. Consequently,
$$
\beta \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \leq \beta H \sqrt {2 d K \log ((d + K) / d)} + \beta d H D _ {\tau , \delta , H, K} \log ((d + K) / d). \tag {8}
$$
Note that by Lemma B.8, event  $\mathcal{E}$  holds with probability  $1 - \delta /4$ , and by a union bound with (7) and (8), we have with probability  $1 - \delta /2$
$$
\sum_ {k = 1} ^ {K} \Delta_ {e s t} ^ {k} \leq \sqrt {2 H ^ {2} T \cdot \log (8 / \delta)} + \beta H \sqrt {2 d K \log ((d + K) / d)} + \beta d H D _ {\tau , \delta , H, K} \log ((d + K) / d) + \frac {H}{K ^ {2}}.
$$
Finally, by a union bound over Step1, Step2 and Step3, we obtain with probability  $1 - \delta$
$$
\begin{array}{l} R (T) = \sum_ {k = 1} ^ {K} \Delta_ {o p t} ^ {k} + \sum_ {k = 1} ^ {K} \Delta_ {e s t} ^ {k} \leq \sum_ {k = 1} ^ {K} \Delta_ {e s t} ^ {k} \\ \leq \sqrt {2 H ^ {2} T \cdot \log (8 / \delta)} + \beta H \sqrt {2 d K \log ((d + K) / d)} + \beta d H D _ {\tau , \delta , H, K} \log ((d + K) / d) + \frac {H}{K ^ {2}} \\ \leq c \sqrt {d ^ {3} H ^ {3} T \iota} + c ^ {\prime} d ^ {2} H ^ {2} \mathbb {E} [ \tau ] \iota + O (\iota) \\ \end{array}
$$
where  $c > 0$  is some universal constant and  $\iota$  is a Polylog term of  $H, d, K, \delta$ . The last step is due to: by the choice of  $\lambda = 1$ ,  $\sigma^2 = 1$ ,  $\nu = C_{\delta/4}$  and  $M = \log(4HK/\delta)/\log(64/63)$ , we can bound  $C_{\delta}$  (in Lemma B.10) by  $C_{\delta} \leq c_0 H \sqrt{dM} \iota_{\delta}$  with  $c_0$  a universal constant and  $\iota_{\delta}$  contains only the Polylog terms. This implies  $\nu^2 \leq c_1 H^2 dM \iota_{\delta}$ . Note  $C_d \leq d\iota_{\delta}$ , therefore  $\beta$  is dominated by the first term  $\beta \leq C_2 \sqrt{2\nu^2 \log(16C_d H M K/\delta)} \leq C_3 dH \iota_{\delta}$  for some universal constants  $C_2, C_3$ . Since  $R(T)$  is dominated by the second term in the second to last inequality, plug back the upper bound for  $\beta$  gets the result. Finally, it is readily to verify  $D_{\tau, \delta, H, K}$  is bounded by  $c' \mathbb{E}[\tau] \iota + O(\iota)$ .
Lemma B.1. Define  $\zeta_h^k = \mathbb{E}[\widetilde{V}_{h + 1}^k (s_{h + 1}^k) - V_{h + 1}^{\pi_k}(s_{h + 1}^k)|s_h^k,a_h^k ] - \widetilde{V}_{h + 1}^k (s_{h + 1}^k) + V_{h + 1}^{\pi_k}(s_{h + 1}^k)$  and condition on the event (10) in Lemma B.8. Then for all  $k\in [K]$ ,  $h\in [H]$ , the following holds,
$$
\widetilde {V} _ {h} ^ {k} \left(s _ {h} ^ {k}\right) - V _ {h} ^ {\pi_ {k}} \left(s _ {h} ^ {k}\right) \leq \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {k}\right) - V _ {h + 1} ^ {\pi_ {k}} \left(s _ {h + 1} ^ {k}\right) + \zeta_ {h + 1} ^ {k} + \beta \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}}.
$$
Proof of Lemma B.1. Since  $a_h^k = \pi_k(s_h^k)$ , it implies  $V_h^k(x_h^k) = \bar{Q}_h^k(s_h^k, a_h^k)$  (recall  $\bar{Q}_h^k \coloneqq \min\{\widetilde{Q}_h^k, H - h + 1\}$ ) and  $V_h^{\pi_k}(x_h^k) = Q_h^{\pi_k}(s_h^k, a_h^k)$ . Hence,
$$
\begin{array}{l} \left| \left(\widetilde {V} _ {h} ^ {k} \left(s _ {h} ^ {k}\right) - V _ {h} ^ {\pi_ {k}} \left(s _ {h} ^ {k}\right)\right) - \left(\widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {k}\right) - V _ {h + 1} ^ {\pi_ {k}} \left(s _ {h + 1} ^ {k}\right)\right) - \zeta_ {h} ^ {k} \right| \\ = \left| \left(\widetilde {V} _ {h} ^ {k} \left(s _ {h} ^ {k}\right) - V _ {h} ^ {\pi_ {k}} \left(s _ {h} ^ {k}\right)\right) - \mathbb {E} \left[ \widetilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {k}\right) - V _ {h + 1} ^ {\pi_ {k}} \left(s _ {h + 1} ^ {k}\right) \mid s _ {h} ^ {k}, a _ {h} ^ {k} \right] \right| \\ = \left| \widetilde {V} _ {h} ^ {k} \left(s _ {h} ^ {k}\right) - r _ {h} ^ {k} - \left(\mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right| \\ = \left| \bar {Q} _ {h} ^ {k} \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) - r _ {h} ^ {k} - \left(\mathbb {P} _ {h} \widetilde {V} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right| \\ \leq \beta \left\| \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} + \frac {1}{K ^ {3}}, \\ \end{array}
$$
where the last step is by the event defined in (10).
B.1 Bounding the delayed error term  $\sum_{k=1}^{K}\left\|\phi(s_h^k,a_h^k)\right\|_{(\Omega_h^k)^{-1}}$
Recall the delayed covariance matrix  $\Omega_h^k = \sum_{\tau=1}^{k-1} \mathbb{1}_{\tau,k-1} \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^{\mathrm{T}} + \lambda I$  with  $\mathbb{1}_{s,t} := \mathbb{1}[s + \tau_s \leq t]$ , then we can define the full design matrix  $\Sigma_h^k$  and the complement matrix  $\Lambda_h^k$  as
$$
\Sigma_ {h} ^ {k} := \sum_ {\tau = 1} ^ {k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\mathrm {T}} + \lambda I, \quad \Lambda_ {h} ^ {k} := \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} [ s + \tau_ {s} > t ] \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\mathrm {T}}, \tag {9}
$$
then  $\Sigma_h^k = \Omega_h^k +\Lambda_h^k$  . Also, denote the number of missing episodes as:  $U_{k} = \sum_{s = 1}^{k}\mathbb{1}[s + \tau_{s} > k]$  Then we have the following Lemmas.
Lemma B.2. For  $\lambda > 0$ ,  $(\Omega_h^k)^{-1} = (\Sigma_h^k)^{-1} + (\Sigma_h^k)^{-1}\Lambda_h^k (\Omega_h^k)^{-1}$ .
Proof of Lemma B.2. Since  $\lambda > 0$ , both  $\Omega_h^k$  and  $\Sigma_h^k$  are invertible with:
$$
\begin{array}{l} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} = \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} + \left(\Omega_ {h} ^ {k}\right) ^ {- 1} - \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \\ = \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} + \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Sigma_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} - \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Omega_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \\ = \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} + \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Lambda_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \\ \end{array}
$$
Lemma B.3. Denote  $\phi_h^k \coloneqq \phi(s_h^k, a_h^k)$ . Let  $\lambda > 0$ , then
$$
\sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Lambda_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \leq \frac {1}{2} \sum_ {k = 1} ^ {K} \left(1 + \max  _ {k \in [ K ]} U _ {k} + \tau_ {k}\right) \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2}
$$
Proof of Lemma B.3. By definition and Trace of matrix, we have
$$
\begin{array}{l} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Lambda_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1}} = \sqrt {\left(\phi_ {h} ^ {k}\right) ^ {\mathrm {T}} \left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Lambda_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi_ {h} ^ {k}} \\ = \sqrt {\operatorname {T r} [ (\phi_ {h} ^ {k}) ^ {\mathrm {T}} (\Sigma_ {h} ^ {k}) ^ {- 1} \Lambda_ {h} ^ {k} (\Omega_ {h} ^ {k}) ^ {- 1} \phi_ {h} ^ {k} ]} = \sqrt {\operatorname {T r} [ (\Sigma_ {h} ^ {k}) ^ {- 1} \Lambda_ {h} ^ {k} (\Omega_ {h} ^ {k}) ^ {- 1} \phi_ {h} ^ {k} (\phi_ {h} ^ {k}) ^ {\mathrm {T}} ]} \\ \end{array}
$$
Denote  $A = (\Sigma_h^k)^{-1}\Lambda_h^k$  and  $B = (\Omega_h^k)^{-1}\phi_h^k (\phi_h^k)^\mathrm{T}$ , then  $A, B$  both have non-negative eigenvalues (by Lemma D.14) and this implies
$$
\operatorname {T r} (A B) = \operatorname {T r} \left(A B ^ {1 / 2} B ^ {1 / 2}\right) = \operatorname {T r} \left(B ^ {1 / 2} A B ^ {1 / 2}\right) \leq \operatorname {T r} \left(B ^ {1 / 2} (\operatorname {T r} (A)) I B ^ {1 / 2}\right) = \operatorname {T r} (A) \operatorname {T r} (B)
$$
$$
\begin{array}{l} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Lambda_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1}} = \sqrt {\operatorname {T r} [ A B ]} \leq \sqrt {\operatorname {T r} [ A ] \operatorname {T r} [ B ]} \leq \frac {1}{2} \operatorname {T r} (A) + \frac {1}{2} \operatorname {T r} (B) \\ = \frac {1}{2} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Omega_ {h} ^ {k}\right) ^ {- 1}} ^ {2} + \frac {1}{2} \sum_ {t = 1} ^ {k - 1} \mathbb {1} [ t + \tau_ {t} > k - 1 ] \left\| \phi_ {h} ^ {t} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq \frac {1 + \max _ {k \in [ K ]} U _ {k}}{2} \left\| \phi_ {h} ^ {k} \right\| _ {(\Sigma_ {h} ^ {k}) ^ {- 1}} ^ {2} + \frac {1}{2} \sum_ {t = 1} ^ {k - 1} \mathbb {1} [ t + \tau_ {t} > k - 1 ] \left\| \phi_ {h} ^ {t} \right\| _ {(\Sigma_ {h} ^ {k}) ^ {- 1}} ^ {2} \\ \end{array}
$$
where the last inequality uses Lemma D.15. Next, by changing the order summation, we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {t = 1} ^ {k - 1} \mathbb {1} [ t + \tau_ {t} > k - 1 ] \left\| \phi_ {h} ^ {t} \right\| _ {\left(\Sigma_ {h} ^ {t}\right) ^ {- 1}} ^ {2} = \sum_ {t = 1} ^ {K - 1} \sum_ {k = t + 1} ^ {K} \mathbb {1} [ t + \tau_ {t} > k - 1 ] \left\| \phi_ {h} ^ {t} \right\| _ {\left(\Sigma_ {h} ^ {t}\right) ^ {- 1}} ^ {2} \\ = \sum_ {t = 1} ^ {K - 1} \sum_ {s = 0} ^ {K - t - 1} \mathbb {1} [ \tau_ {t} > s ] \left\| \phi_ {h} ^ {t} \right\| _ {(\Sigma_ {h} ^ {t}) ^ {- 1}} ^ {2} \leq \sum_ {t = 1} ^ {K - 1} \sum_ {s = 0} ^ {\infty} \mathbb {1} [ \tau_ {t} > s ] \left\| \phi_ {h} ^ {t} \right\| _ {(\Sigma_ {h} ^ {t}) ^ {- 1}} ^ {2} = \sum_ {t = 1} ^ {K - 1} \tau_ {t} \left\| \phi_ {h} ^ {t} \right\| _ {(\Sigma_ {h} ^ {t}) ^ {- 1}} ^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \left| \left| \phi_ {h} ^ {k} \right| \right| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1} \Lambda_ {h} ^ {k} \left(\Omega_ {h} ^ {k}\right) ^ {- 1}} \\ \leq \frac {1 + \max  _ {k \in [ K ]} U _ {k}}{2} \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2} + \frac {1}{2} \sum_ {k = 1} ^ {K} \sum_ {t = 1} ^ {k - 1} \mathbb {1} \left[ t + \tau_ {t} > k - 1 \right] \left\| \phi_ {h} ^ {t} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2} \\ \leq \frac {1 + \max  _ {k \in [ K ]} U _ {k}}{2} \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2} + \frac {1}{2} \sum_ {k = 1} ^ {K} \sum_ {t = 1} ^ {k - 1} \mathbb {1} \left[ t + \tau_ {t} > k - 1 \right] \left\| \phi_ {h} ^ {t} \right\| _ {\left(\Sigma_ {h} ^ {t}\right) ^ {- 1}} ^ {2} \\ \leq \frac {1 + \max  _ {k \in [ K ]} U _ {k}}{2} \sum_ {k = 1} ^ {K} \left\| \phi_ {h} ^ {k} \right\| _ {\left(\Sigma_ {h} ^ {k}\right) ^ {- 1}} ^ {2} + \sum_ {t = 1} ^ {K - 1} \tau_ {t} \left\| \phi_ {h} ^ {t} \right\| _ {\left(\Sigma_ {h} ^ {t}\right) ^ {- 1}} ^ {2}, \\ \end{array}
$$
where the second step uses  $(\Sigma_h^k)^{-1} \succeq (\Sigma_h^t)^{-1}$  for  $k \geq t$ .
Lemma B.4 (Bounding the delayed error). With probability  $1 - \delta / 8$ ,