SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| w _ {h} ^ {k} \right\| \leq \| \theta_ {h} \| + \left\| \int_ {\mathcal {S}} \widetilde {V} _ {h + 1} ^ {k} \left(s ^ {\prime}\right) \mathrm {d} \mu_ {h} \left(s ^ {\prime}\right) \right\| \\ \leq \sqrt {d} + \left\| \int_ {\mathcal {S}} H \mathrm {d} \mu_ {h} \left(s ^ {\prime}\right) \right\| \\ \leq 2 H \sqrt {d}. \\ \end{array}
$$
Lemma D.4 (Bound on Estimated Weights of Algorithm 1). For any step  $h \in [H]$  and episode  $k \in [K]$ , the weight  $\widehat{w}_h^k$  output by Algorithm 1 satisfies,
$$
\left\| \widehat {w} _ {h} ^ {k} \right\| \leq 2 H \sqrt {\frac {d k}{\lambda}}.
$$
Proof of Lemma D.4. For any vector  $\mathbf{v} \in \mathbb{R}^d$ , it holds
$$
\begin{array}{l} \left| \mathbf {v} ^ {\top} \widehat {w} _ {h} ^ {k} \right| = \left| \mathbf {v} ^ {\top} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \sum_ {\tau = 1} ^ {k - 1} \phi_ {h} ^ {\tau} \left[ r \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) + \widetilde {V} _ {h} ^ {k} \left(s _ {h + 1} ^ {\tau}\right) \right] \right| \\ \leq \sum_ {\tau = 1} ^ {k - 1} \left| \mathbf {v} ^ {\top} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi_ {h} ^ {\tau} \right| \cdot 2 H \leq \sqrt {\left[ \sum_ {\tau = 1} ^ {k - 1} \mathbf {v} ^ {\top} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \mathbf {v} \right] \cdot \left[ \sum_ {\tau = 1} ^ {k - 1} \left(\phi_ {h} ^ {\tau}\right) ^ {\top} \left(\Omega_ {h} ^ {k}\right) ^ {- 1} \phi_ {h} ^ {\tau} \right] \cdot 2 H} \\ \leq 2 H \| \mathbf {v} \| \sqrt {d k / \lambda}, \\ \end{array}
$$
where the last step is by Lemma D.2. The above directly imply the stated result by the definition of  $l_{2}$  norm.
D.2 Concentration Inequalities
Lemma D.5 ([3]). Suppose  $Z$  is a random variable following a Gaussian distribution  $\mathcal{N}(\mu, \sigma^2)$ , where  $\sigma > 0$ . The following concentration and anti-concentration inequalities hold for any  $z \geq 1$ :
$$
\frac {1}{2 \sqrt {\pi} z} e ^ {- z ^ {2} / 2} \leq \mathbb {P} \left(| Z - \mu | > z \sigma\right) \leq \frac {1}{\sqrt {\pi} z} e ^ {- z ^ {2} / 2}.
$$
And for  $0 \leq z \leq 1$ , we have,
$$
\mathbb {P} \left(| Z - \mu | > z \sigma\right) \geq \frac {1}{\sqrt {8 \pi}} e ^ {- z ^ {2} / 2}.
$$
Lemma D.6 (Sub-exponential tail bound). Suppose  $\{\tau_k\}_{k=1}^{\infty}$  are  $(v, b)$ -sub-exponential random variables. denote  $D_{\tau, K, \delta} := \min \left\{\sqrt{2v^2\log\left(\frac{3K}{2\delta}\right)}, 2b\log\left(\frac{3K}{2\delta}\right)\right\}$ . Then with probability  $1 - \delta$
$$
\max  _ {k \in [ K ]} \tau_ {k} \leq \mathbb {E} [ \tau ] + D _ {\tau , K, \delta}.
$$
Lemma D.7 (Multivariate Gaussian Concentration). Suppose  $X \sim \mathcal{N}(0, I_d)$ . Then with probability  $1 - \delta$ ,
$$
\left\| X \right\| \leq \sqrt {2 d} + \sqrt {2 \log (1 / \delta)}.
$$
Proof. Apply Proposition 1 of [30], choose  $A = I_d$ , then  $\Sigma = I_d$  and  $Tr(\Sigma) = d$ ,  $\| \Sigma \| = 1$ . Then
$$
P \left[ \| X \| ^ {2} \geq d + 2 \sqrt {d t} + 2 t \right] \leq e ^ {- t} \Rightarrow P [ \| X \| ^ {2} \geq 2 (\sqrt {d} + \sqrt {t}) ^ {2} ] \leq e ^ {- t} := \delta
$$
which implies with probability  $1 - \delta$ ,  $\|X\| \leq \sqrt{2d} + \sqrt{2\log(1/\delta)}$ .
Lemma D.8 (Elliptical Potential Lemma [1]). Suppose  $\{\phi_t\}_{t=1}^{\infty}$  is an  $\mathbb{R}^d$ -valued sequence,  $\Omega_0 \in \mathbb{R}^{d \times d}$  is positive definite, and  $\Omega_t = \Omega_0 + \sum_{\tau=1}^{t-1} \phi_\tau \phi_\tau^{\mathrm{T}}$ . If  $\lambda_{\min}(\Omega_0) \geq 1$ , and  $\| \phi_\tau \|_2 \leq 1$  for all  $\tau \in \mathbb{Z}_+$ , then for any  $t \in \mathbb{Z}_+$ ,
$$
\log \left(\frac {\operatorname* {d e t} (\Omega_ {t + 1})}{\operatorname* {d e t} (\Omega_ {1})}\right) \leq \sum_ {\tau = 1} ^ {t} \phi_ {\tau} ^ {\mathrm {T}} (\Omega_ {\tau}) ^ {- 1} \phi_ {\tau} \leq 2 \log \left(\frac {\operatorname* {d e t} (\Omega_ {t + 1})}{\operatorname* {d e t} (\Omega_ {1})}\right).
$$
Lemma D.9 (Self-normalized process [1]). Let  $\{\mathcal{F}_t\}_{t=0}^{\infty}$  be a filtration, and  $\{\eta_t\}_{t=1}^{\infty}$  be a real-valued stochastic process such that  $\eta_t$  is  $\mathcal{F}_t$ -measurable and  $\eta_t|\mathcal{F}_{t-1}$  is zero-mean (i.e.,  $\mathbb{E}[\eta_t|\mathcal{F}_{t-1}] = 0$ ). Assume that conditioning on  $\mathcal{F}_t$ ,  $\eta_t$  is  $C$ -sub-Gaussian. Let  $\{\phi_t\}_{t=1}^{\infty}$  be an  $\mathbb{R}^d$  real-valued stochastic process such that  $\phi_t$  is  $\mathcal{F}_t$ -measurable. Let  $\Omega_0 \in \mathbb{R}^{d \times d}$  be a positive definite matrix and  $\Omega_t = \Omega_0 + \sigma^{-2}\sum_{\tau=1}^{t}\phi_\tau\phi_\tau^T$ . Then for  $\delta > 0$ , with probability at least  $1 - \delta$ , for all  $t \geq 0$ ,
$$
\left\| \sum_ {\tau = 1} ^ {t} \phi_ {\tau} \eta_ {\tau} \right\| _ {\Omega_ {t} ^ {- 1}} ^ {2} \leq 2 C ^ {2} \log \left(\frac {\det  (\Omega_ {t}) ^ {1 / 2} \det  (\Omega_ {0}) ^ {- 1 / 2}}{\delta}\right).
$$
Lemma D.10. Suppose  $\Omega_0 \coloneqq \lambda I_d$  is a positive definite matrix in  $\mathbb{R}^{d \times d}$  and  $\Omega_t = \Omega_0 + \sigma^{-2} \sum_{\tau=1}^{t-1} \phi_\tau \phi_\tau^T$ .
$$
\frac {\operatorname* {d e t} (\Omega_ {t + 1})}{\operatorname* {d e t} (\Omega_ {1})} \leq \left(\frac {\lambda + \sigma^ {- 2} t}{\lambda}\right) ^ {d}.
$$
Proof of Lemma D.10. By definition,  $\operatorname*{det}(\Omega_1) = \operatorname*{det}(\lambda I) = \lambda^d$ . For any  $\tau \in \mathbb{Z}_+$  and  $\phi_{\tau} \in R^{d}$ , notice that  $\phi_{\tau}\phi_{\tau}^{\mathrm{T}}$  is a rank-1 matrix with eigenvalues  $\| \phi_{\tau}\|$  and 0. By Definition 1 and triangle inequality,
$$
\left\| \sum_ {\tau = 1} ^ {t} \phi_ {\tau} \phi_ {\tau} ^ {\mathrm {T}} \right\| \leq \sum_ {\tau = 1} ^ {t} \left\| \phi_ {\tau} \phi_ {\tau} ^ {\mathrm {T}} \right\| \leq t.
$$
Consider the eigenvalue decomposition for  $\sum_{\tau = 1}^{t - 1}\phi_{\tau}\phi_{\tau}^{\mathrm{T}}$
$$
\sum_ {\tau = 1} ^ {t - 1} \phi_ {\tau} \phi_ {\tau} ^ {\mathrm {T}} = U \operatorname {d i a g} (\lambda_ {1}, \dots , \lambda_ {d}) U ^ {\mathrm {T}},
$$
which suggests
$$
\det (\Omega_ {t + 1}) = \det (\lambda I + \sigma^ {- 2} \sum_ {\tau = 1} ^ {t - 1} \phi_ {\tau} \phi_ {\tau} ^ {T}) = \prod_ {i = 1} ^ {d} (\sigma^ {- 2} \lambda_ {i} + \lambda) \leq (\sigma^ {- 2} \max _ {i} | \lambda_ {i} | + \lambda) ^ {d} \leq (\lambda + \sigma^ {- 2} t) ^ {d}.
$$
D.3 Covering Argument
Lemma D.11 (Covering number of Euclidean Ball). Consider an Euclidean ball  $B_R$  equipped with the Euclidean metric, whose radius is  $R > 0$ . The  $\epsilon$ -covering number of  $B_R$  satisfies,
$$
\mathcal {N} _ {\epsilon} (B _ {R}) \leq \left(1 + \frac {2 R}{\epsilon}\right) ^ {d}.
$$
Lemma D.12. Define  $\mathcal{V}$  to be a class of values with the parametric form
$$
f _ {\phi} := | \langle \phi , \theta \rangle | - C \sqrt {\phi^ {\top} A \cdot \phi}
$$
where the feature space is  $\{\phi : \| \phi \|_2 \leq 1\}$  and  $\| A \|_2 \leq B$ ,  $\|\theta\| \leq 2H\sqrt{d}$ . Let  $\mathcal{N}_{\epsilon}^{\mathcal{V}}$  be the covering number of  $\epsilon$ -net with respect to the absolute value distance, then we have
$$
\log \mathcal {N} _ {\epsilon} ^ {\nu} \leq d \log (1 + \frac {4 C \sqrt {B} + 4 H \sqrt {d}}{\epsilon}).
$$
Proof of Lemma D.12.
$$
\begin{array}{l} \left| f _ {\phi_ {1}} - f _ {\phi_ {2}} \right| \leq \left| \left| \langle \phi_ {1}, \theta \rangle \right| - C \sqrt {\phi_ {1} ^ {\top} A \cdot \phi_ {1}} - \left(\left| \langle \phi_ {2}, \theta \rangle \right| - C \sqrt {\phi_ {2} ^ {\top} A \cdot \phi_ {2}}\right) \right| \\ \leq \left\| \phi_ {1} - \phi_ {2} \right\| \cdot \left\| \theta \right\| + C \sqrt {\left| \phi_ {1} ^ {\top} A \cdot \phi_ {1} - \phi_ {2} ^ {\top} A \cdot \phi_ {2} \right|} \\ \leq \| \phi_ {1} - \phi_ {2} \| \cdot 2 H \sqrt {d} + C \sqrt {\| \phi_ {1} \| \| A \| \| \phi_ {1} - \phi_ {2} \|} + C \sqrt {\| \phi_ {1} - \phi_ {2} \| \| A \| \| \phi_ {2} \|} \\ \leq \left\| \phi_ {1} - \phi_ {2} \right\| \cdot 2 H \sqrt {d} + 2 C \sqrt {B \left\| \phi_ {1} - \phi_ {2} \right\|} \leq \left(2 C \sqrt {B} + 2 H \sqrt {d \left\| \phi_ {1} - \phi_ {2} \right\|}\right) \cdot \left\| \phi_ {1} - \phi_ {2} \right\| \\ \leq 2 C \sqrt {B \| \phi_ {1} - \phi_ {2} \|} \leq (2 C \sqrt {B} + 2 H \sqrt {d}) \cdot \| \phi_ {1} - \phi_ {2} \| \\ \end{array}
$$
Let  $\mathcal{C}_{\phi}$  be the  $\frac{\epsilon}{2C\sqrt{B} + 2H\sqrt{d}}$ -net of space  $\{\phi : \| \phi \|_2 \leq 1\}$ , then by Lemma D.11,
$$
| \mathcal {C} _ {\phi} | \leq (1 + \frac {4 C \sqrt {B} + 4 H \sqrt {d}}{\epsilon}) ^ {d}
$$
Therefore, the covering number of space  $\mathcal{V}$  satisfies
Lemma D.13. Let  $\mathcal{V}$  denote the function class from  $S$  to  $\mathbb{R}$
$$
V (\cdot) := \max  _ {a} \max  _ {m} \phi (\cdot , a) ^ {\mathrm {T}} w ^ {m}, \text {w h e r e} \| w ^ {m} \| \leq C _ {H, d, k, M, \delta}, \forall m \in [ M ]
$$
let  $\mathcal{N}_{\epsilon}$  be the  $\epsilon$ -covering number of  $\mathcal{V}$  with respect to the distance  $\text{dist}(V, V') = \sup_s |V(s) - V'(s)|$ . Then
$$
\log \mathcal {N} _ {\epsilon} \leq d M \log (1 + \frac {2 C _ {H , d , k , M , \delta}}{\epsilon}).
$$
Here  $C_{H,d,k,M,\delta} = 2H\sqrt{\frac{dk}{\lambda}} + \frac{\sqrt{2d} + \sqrt{2\log(M / \delta)}}{\sqrt{\lambda}}$ .
Proof. Let  $V_{1} = \max_{a}\max_{m}\phi (\cdot ,a)^{\mathrm{T}}w_{1}^{m}$  and  $V_{2} = \max_{a}\max_{m}\phi (\cdot ,a)^{\mathrm{T}}w_{2}^{m}$ . Then
$$
\begin{array}{l} \mathbf {d i s t} (V _ {1}, V _ {2}) = \max  _ {s} | \max  _ {a} \max  _ {m} \phi (\cdot , a) ^ {\mathrm {T}} w _ {1} ^ {m} - \max  _ {a} \max  _ {m} \phi (\cdot , a) ^ {\mathrm {T}} w _ {2} ^ {m} | \\ \leq \max _ {s, a, m} \| \phi (s, a) \| \cdot \| w _ {1} ^ {m} - w _ {2} ^ {m} \| \leq \max _ {s, a, m} \| w _ {1} ^ {m} - w _ {2} ^ {m} \|, \\ \end{array}
$$
For any  $m \in [M]$ , let  $\mathcal{C}^m$  be the  $\epsilon$ -net for  $\{w^m : \| w^m \| \leq C_{H,d,k,M,\delta}\}$ , then by Lemma D.11,  $|\mathcal{N}_{\epsilon}^{m}| \leq (1 + \frac{2C_{H,d,k,M,\delta}}{\epsilon})^d$ , implies the total log covering number
$$
\log | \mathcal {N} _ {\epsilon} | \leq \log \Pi_ {m = 1} ^ {M} | \mathcal {N} _ {\epsilon} ^ {m} | \leq d M \log (1 + \frac {2 C _ {H , d , k , M , \delta}}{\epsilon}).
$$
D.4 Delayed Feedback
Lemma D.14 (Lemma 9 of [28]). Let  $A, B \in \mathbb{R}^{d \times d}$  be two symmetric positive semi-definite matrices. Then,  $A^{\frac{1}{2}}BA^{\frac{1}{2}}$  and  $AB$  share the same set of eigenvalues. Further, these eigenvalues are all non-negative.
Lemma D.15. Let  $\Sigma_h^k, \Omega_h^k, \Lambda_h^k$  be the full design, delayed, and complement matrix respectively. Then  $(1 + \frac{U_k}{\lambda})(\Sigma_h^k)^{-1} \succeq (\Omega_h^k)^{-1}$ . In addition, with probability  $1 - \delta$ ,
$$
\max  _ {k \in [ K ]} U _ {k} \leq \mathbb {E} [ \tau ] + 2 \sqrt {2 \mathbb {E} [ \tau ] \log (3 K / 2 \delta)} + \frac {4}{3} \log (3 K / 2 \delta).
$$
Proof. The proof follows from Lemma 11 of [28] with  $\frac{U_k}{\lambda} (\Sigma_h^k)^{-1} \succeq (\Sigma_h^k)^{-1}\Lambda_h^k (\Omega_h^k)^{-1}$ , and then apply Lemma B.2 that  $(\Omega_h^k)^{-1} = (\Sigma_h^k)^{-1} + (\Sigma_h^k)^{-1}\Lambda_h^k (\Omega_h^k)^{-1}$ . The second part comes from Lemma 4 of [28].
In this section, we provide the experimental details of both simulated environments (synthetic linear MDP and RiverSwim) and discuss their results respectively.
E.1 Delayed-UCBVI
As shown in Table 1 and Section 2, there is no prior UCB method that concerns exactly the same delayed linear MDP setting without resorting to specific policy-switching schemes. To benchmark our posterior sampling algorithms, we modify the existing LSVI-UCB method to accommodate the delayed feedback, which is referred to as the Delayed-UCBVI. Below we include the algorithm of delayed-UCBVI for completeness.
Algorithm 4: Delayed Value Iteration with UCB (Delayed-UCBVI)
Input: bonus parameter  $\beta$ , regularization  $\lambda$ .  
1 Initialization:  $\forall k, h, \widetilde{Q}_{H+1}^{k}(\cdot, \cdot), \widetilde{V}_{H+1}(\cdot, \cdot), \widetilde{V}_{h}(\cdot, \cdot) \gets 0, \mathcal{D}_{h} \gets \emptyset$ .  
2 for episode  $k = 1, \ldots, K$  do  
3 Sample initial state  $s_{1}^{k}$   
4 for time step  $h = H, \ldots, 1$  do  
5  $y_{h} \gets [y_{h}^{1}, \ldots, y_{h}^{k-1}]$ , with  $y_{h}^{\tau} \gets \mathbb{1}_{\tau, k-1} \cdot [r_{h}^{\tau} + \widetilde{V}_{h+1}(s_{h+1}^{\tau})]$   
6  $\Phi_{h} \gets [\phi^{1}, \phi^{2}, \ldots, \phi^{k-1}]$  with  $\phi^{\tau} = \mathbb{1}_{\tau, k-1} \cdot \phi(s_{h}^{\tau}, a_{h}^{\tau})$   
7  $\Omega_{h}^{k} \gets \Phi_{h} \Phi_{h}^{\mathrm{T}} + \lambda I$   
8  $w_{h}^{k} \gets (\Omega_{h}^{k})^{-1} \Phi_{h} y_{h}^{\mathrm{T}}$   
9  $Q_{h}^{k}(\cdot, \cdot) \gets \phi(\cdot, \cdot)^{\mathrm{T}} w_{h}^{k} + \beta \sqrt{\phi(\cdot, \cdot)^{\mathrm{T}} (\Omega_{h}^{k})^{-1} \phi(\cdot, \cdot)}$   
10  $V_{h}(\cdot, \cdot) \gets \max_{a} \min\{Q_{h}^{k}(\cdot, a), H - h + 1\}$   
11 Update  $\pi_{h}^{k}(\cdot) \gets \operatorname{argmax}_{a \in A} \min\{Q_{h}^{k}(\cdot, a), H - h + 1\}$   
12 for time step  $h = 1, \ldots, H$  do  
13 Choose action  $a_{h}^{k} \sim \pi_{h}^{k}(s_{h}^{k})$   
14 Collect transitions  $D_{h} \gets D_{h} \cup \{(s_{h}^{k}, a_{h}^{k}, r_{h}^{k}, s_{h+1}^{k})\}$   
/* Feedback generated in episode  $k$  cannot be immediately observed in the presence of delay */
E.2 Synthetic Linear MDP Environment
In this section, we describe the further details in Section 5.1.
Environment Details. Following [44, 46, 70], we construct a set of synthetic linear MDP environments with  $|\mathcal{S}| = 2$ , feature dimension  $d = 10$ , planning horizon  $H = 20$ , and varying action space  $|\mathcal{A}| \in \{20, 50, 100\}$ . Each action  $a \in \mathcal{A} \subseteq \{0, 1\}^d$  is encoded with its 8-bit binary representation and represented by a vector  $\boldsymbol{b}_a \in \mathbb{R}^8$ . The feature map  $\phi(\cdot, \cdot)$  can then be defined as
$$
\phi (s, a) = \left[ \boldsymbol {b} _ {a} ^ {\mathrm {T}}, \delta (s, a), 1 - \delta (s, a) \right] ^ {\mathrm {T}} \in \mathbb {R} ^ {1 0}, \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A},
$$
$$
\delta (s, a) = \left\{ \begin{array}{l l} 1 & \text {i f} \mathbb {1} (s = 0) = \mathbb {1} (a = 0), \\ 0 & \text {o t h e r w i s e}. \end{array} \right.
$$
In addition, let  $\theta_h$  that induces the reward functions  $r$  be
$$
\theta_ {h} = [ 0, \dots , 0, r, 1 - r ] ^ {\mathrm {T}} \in \mathbb {R} ^ {1 0},
$$
with the choice of  $r = 0.99$ , and further define the measures  $\mu_h$  that govern the transition dynamics  $\mathbb{P}$  as
$$
\mu_ {h} (s) = [ 0, \dots , 0, (1 - s) \oplus \alpha_ {h}, s \oplus \alpha_ {h} ],
$$
where  $\{\alpha_h\}_{h\in [H]}\in \{0,1\} ^H$  is a sequence of integers taking values 0 or 1,  $\oplus$  is the XOR operator. By design, the set of environments with identical  $d$  and  $H$  has the same optimal value  $V_{1}^{*}(s_{1})$
Further Results and Discussions. Figure 2 depicts the empirical distributions of delays considered in section 5.1. Additionally, the average return achieved by each method upon convergence is reported in Table 2, corresponding to the results shown in Figure 1. Our empirical findings indicate that posterior sampling methods excel UCB-based methods in terms of both statistical accuracy and computational efficiency. More specifically, under different types of delays, both Delayed-PSVI and Delayed-LPSVI achieve higher return (lower regret) and exhibit faster convergence compared to Delayed-UCBVI.
While delays following multinomial distribution and Poisson distributions decay exponentially fast, Pareto delays are heavy-tailed. When computational budget is limited or when episodes are finite, feedback is only partially observable under long-tailed delays and is not guaranteed to be revealed to the agent. This setup captures the practical scenarios when small time windows are considered for decision-making or in online recommender systems, where only positive feedback (e.g. click, make a purchase) are often observed. As shown in Table 2, performance of Delayed-UCBVI can dramatically deteriorate in the presence of long-tailed delays.
Furthermore, our results presented in Table 4 and Table 3 illustrate the consistent behavior of posterior sampling in environments with delayed feedback, considering both statistical and computational aspects. When employing feature mapping, performance of the algorithms is much less dependent on the sizes of state and action space in contrast to tabular settings. It is worth noting that in large state and action space, the neighborhoods of a substantial number of state-action pairs may remain unvisited, leading to increased uncertainty in estimation. In such cases, adjusting the scale of exploration by decreasing the noise scaling factor  $\sigma$  for Delayed-PSVI can yield faster convergence. Finally, as shown in Table 3, Delayed-LPSVI achieves appealing performance as Delayed-PSVI while reducing computation through the use of approximate sampling with Langevin dynamics.
Multinomial Delay (10,20,30)	Poisson Delay (E[τ] = 50)	Pareto Delay (Shape 1.0, Scale 500)
Delayed-PSVI (σ = 0.1)	11.53 ± 0.76	11.48 ± 0.81	11.53 ± 0.74
Delayed-LPSVI (cη = 0.5)	11.56 ± 0.48	11.37 ± 0.48	10.98 ± 0.40
Delayed-UCBVI (cβ = 0.1)	10.61 ± 0.76	10.54 ± 0.81	7.20 ± 0.38
Table 2: Average return achieved by Delayed-PSVI, Delayed-LPSVI and Delayed-UCBVI upon convergence under different delays. Environment setup:  $|\mathcal{S}| = 2$ ,  $|\mathcal{A}| = 20$ ,  $d = 10$ ,  $H = 20$ . Optimal average return is  $V_{1}^{*}(s_{1}) = 11.96$ . Results are obtained over 10 experiments.
Figure 2: Empirical distributions of three types of delays. (a) Multinomial delays with delay categories  $\{10, 20, 30\}$ . (b) Poisson delays with rate  $\mathbb{E}[\tau] = 50$ . (c) Long-tail Pareto delays with shape 1.0, scale 500. The first two types of delays are well-behaved and decay exponentially fast, while pareto delays are heavy-tailed.
|S||A|=20	|S||A|=40	|S||A|=100	|S||A|=200
Delayed-PSVI (σ=0.3)	1418	1290	1669	2633
Delayed-PSVI (σ=0.2)	531	1114	1323	826
Delayed-PSVI (σ=0.1)	391	571	650	709
Delayed-LPSVI (cη=0.5)	293	246	517	566
Delayed-UCBVI (cβ=0.1)	3205	2713	3351	3694
Table 3: Number of episodes for each method to achieve its highest expected return. Different synthetic environments are examined with varied  $|\mathcal{S}|$  and  $|\mathcal{A}|$ . Optimal average return is  $V_{1}^{*}(s_{1}) = 11.96$  for all environments  $(d = 10, H = 20)$ . Results are obtained over 10 experiments with Poisson delays  $(\mathbb{E}[\tau] = 50)$ .
|S||A|=20	|S||A|=40	|S||A|=100	|S||A|=200
Delayed-PSVI (σ=0.3)	11.23 ± 1.00	11.07 ± 1.05	10.93 ± 1.11	10.80 ± 1.13
Delayed-PSVI (σ=0.2)	11.39 ± 0.91	11.28 ± 0.94	11.16 ± 1.02	11.11 ± 1.03
Delayed-PSVI (σ=0.1)	11.57 ± 0.74	11.48 ± 0.81	11.39 ± 0.86	11.33 ± 0.92
Delayed-LPSVI (cη=0.5)	11.31 ± 0.46	11.37 ± 0.48	11.57 ± 0.48	11.57 ± 0.78
Delayed-UCBVI (cβ=0.1)	10.98 ± 1.78	10.54 ± 0.81	9.67 ± 0.54	10.01 ± 0.16
Table 4: Average return achieved by Delayed-PSVI, Delayed-LPSVI and Delayed-UCBVI upon convergence in different linear MDP environments with varied  $|\mathcal{S}|$  and  $|\mathcal{A}|$ . Optimal average return is  $V_{1}^{*}(s_{1}) = 11.96$  for all environments ( $d = 10$ ,  $H = 20$ ). Results are obtained over 10 experiments with Poisson delays ( $\mathbb{E}[\tau] = 50$ ).
E.3 RiverSwim
RiverSwim environment is known to be a difficult exploration problem for least-squares value iteration with  $\epsilon$ -greedy exploration due to the sparse reward setting. It models an agent swimming in the river who can either swim towards the right (against the current) or towards the left (with the current). While trying to move rightwards may fail with some probability, moving leftwards always yield successful transition. We consider the environment with linear feature maps where  $|S| = 5$ ,  $d = 10$ ,  $H = 20$ , and Poisson delays. Accordingly, the tabular environment can be recovered with canonical basis in  $\mathbb{R}^d$  as its feature mapping:
$$
\phi (s, a) = \boldsymbol {e} _ {s, a} \in \mathbb {R} ^ {1 0}, \quad (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
Define  $\theta_h$  as
$$
\theta_ {h} (s, a) = [ 0. 0 0 5, 0, \dots , 0, 1. 0 ] ^ {\mathrm {T}} \in \mathbb {R} ^ {1 0},
$$
then reward functions induced by  $\theta_h$  are given by:
$$
r _ {h} (s, a) = \left\{ \begin{array}{l l} 0. 0 0 5 & \text {i f} s = 0, a = \text {l e f t}; \\ 1. 0 & \text {i f} s = 4, a = \text {r i g h t}; \\ 0. 0 & \text {o t h e r w i s e}. \end{array} \right.
$$
In this environment, We warm start LMC for Delayed-LPSVI by reusing the previous sample for initialization, and set  $M = 2$ ,  $N = 40$ ,  $\eta = c_{\eta} / \lambda_{\max}(\Omega_h^k)$ ,  $\gamma = c_{\gamma}^2 dMH^2$ . We set parameters  $M = 2$ ,  $\nu = 1.0$  for Delayed-PSVI, and the bonus coefficient in Delayed-UCBVI as  $\beta_h^k = c_\beta /2$ .
$d\sqrt{k} (H - h)$ . Optimal hyperparameters are determined by gridsearch and we fix  $c_{\beta} = 0.04$ ,  $c_{\eta} = 0.5$ ,  $c_{\gamma} = 0.005$ ,  $\sigma = 1.13$ . Experiments are repeated with 5 different random seeds. Cumulative regrets are then depicted in Figure 3.
Results and Discussions. Compared to the previous synthetic environment where dense rewards are available, posterior sampling methods are shown to be robust with spare rewards even in the presence of delays. Figure 3 shows that both Delayed-PSVI and Delayed-LPSVI outperform Delayed-UCBVI in delayed-feedback settings with linear function approximation. In particular, LMC (Algorithm 3) provides strong concentration such that Delayed-LPSVI is able to maintain the order-optimal regret as Delayed-PSVI when exploring the value-function space.
Figure 3: Delayed-PSVI and Delayed-LPSVI outperform Delayed-UCBVI in sparse-reward setting with Poisson delays  $(\mathbb{E}[\tau] = 5)$ . Results are reported over 5 experiments.