SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\Omega_h^k \coloneqq \Phi_h \Phi_h^{\mathrm{T}} + \lambda I_d$  and  $\Phi_h = [\phi(s_h^1, a_h^1), \phi(s_h^2, a_h^2), \dots, \phi(s_h^{k-1}, a_h^{k-1})]$ . Adding the scaling factors  $\sigma^2$  and  $\nu^2$  yields the Line 10 of Algorithm 1. It is important to note that while the induced likelihood  $\exp(-L(w_h^k, \boldsymbol{y}_h^k, \mathcal{D}_h^k))$  from (3) is Gaussian, we do not assume  $y_h^\tau = r_h^\tau(s_h^\tau, a_h^\tau)$  and  $\widetilde{V}_{h+1}(s_{h+1}^\tau)$  follows a Gaussian distribution. Instead, the above likelihood model can be used for non-Gaussian problems as we need not sample from the exact Bayesian posterior model [2, 74].
On the other hand, the  $\widehat{w}_h^k$  computed in Line 9 of Algorithm 1 together with the greedy choice  $\widetilde{V} (\cdot)\approx \max_a\widetilde{Q} (\cdot ,a)$  (Line 15) approximates the solution of Bellman optimality equation via the least-square ridge regression:  $\widehat{w}_h^k = \mathrm{argmin}_w\sum_{\tau = 1}^{k - 1}(\phi (s_h^\tau ,a_h^\tau)^\mathrm{T}w - (r + \max \bar{Q}_h^k))^2 +\lambda I_d.$  4 Consequently, Line 5-10 essentially performs the Posterior Sampling Value Iteration.
Optimism via multi-round sampling scheme. Unlike the Bayesian regret or the worst-case expected regret, the high-probability worst-case regret in (2) needs to control the sub-optimal gap with arbitrarily high probability of at least  $1 - \delta$ . However, sampling once at each time step only provides a constant-probability optimistic estimation, which breaks the high probability requirement. In addition, the estimation error incurred by sampling (i.e. constant-probability pessimistic estimation) at each timestep will propagate to the previous time steps during the backward posterior sampling value iteration. This phenomenon does not appear in the 1-horizon bandit problem due to a saturated-arm analysis [2, 6]. To remedy this issue, we design a multi-round sampling scheme that generates  $M$  estimates  $\{\widetilde{Q}^m\}_{m \in [M]}$  for  $Q$ -fuction through  $M$  i.i.d. sampling procedures, and constructs an optimistic estimate by setting  $\widetilde{Q} = \max_m \widetilde{Q}^m$ . Notably, our choice of  $M$  has order Polylog  $(H, K, d, \delta)$ , and thus makes our algorithm sample-efficient without increasing the overall complexity dependence. As shown in Line 11-14 of Algorithm 1, this scheme guarantees the optimistic estimates  $\widetilde{Q} \geq Q^*$  can be achieved as desired. Lastly, ensemble sampling methods enjoy empirical success and popularity in RL [21, 23, 31], including double q-learning [26] and bootstrapped DQN [42, 47]. We are among the first few works to explain its theoretical effectiveness.
Episodic delayed feedback model. Recall that by Definition 2, when delay  $\tau_{k}$  takes place, the feedback  $\{s_h^t,a_h^t,r_h^t,s_{h + 1}^t\}_{h\in [H]}$  of episode  $k$  cannot be observed until the beginning of the  $k + \tau_{k}$ -th episode. Accordingly, the delayed version of the fully observed  $y^{\tau},\Omega_{h}^{k}$  now becomes,
$$
y _ {h} ^ {\tau} \gets \mathbb {1} _ {\tau , k - 1} \cdot [ r _ {h} ^ {\tau} (s _ {h} ^ {\tau}, a _ {h} ^ {\tau}) + \widetilde {V} _ {h + 1} (s _ {h + 1} ^ {\tau}) ], \Phi_ {h} \gets [ \mathbb {1} _ {1, k - 1} \cdot \phi (s _ {h} ^ {1}, a _ {h} ^ {1}), \ldots , \mathbb {1} _ {k - 1, k - 1} \cdot \phi (s _ {h} ^ {k - 1}, a _ {h} ^ {k - 1}) ].
$$
As a result, episodic delays are considered during the posterior updates in subsequent episodes. This completes the design of Delayed-PSVI as presented in Algorithm 1. In the remainder of this section, we present the main theoretical guarantees of Delayed-PSVI and the proof sketch of Theorem 1.
Theorem 1. Suppose delays satisfy Assumption 1. In any episodic linear MDP with time horizon  $T = KH$ , where  $K$  is the total number of episodes, for any  $0 < \delta < 1$ , let  $\lambda = 1$ ,  $\sigma^2 = 1$ ,  $M = \log(4HK / \delta) / \log(64 / 63)$  and  $\nu = C_{\delta/4} \approx \widetilde{O}(\sqrt{dMH^2})$  ( $C_{\delta/4}$  in Lemma B.10). Then with probability at least  $1 - \delta$ , there exists some absolute constants  $c, c', c'' > 0$  such that the regret of Delayed-PSVI (Algorithm 1) satisfies:
$$
R (T) \leq c \sqrt {d ^ {3} H ^ {3} T \iota} + c ^ {\prime} d ^ {2} H ^ {2} \mathbb {E} [ \tau ] \iota + c ^ {\prime \prime} \iota .
$$
Here  $\iota$  is a Polylog term of  $H, d, K, \delta$ .
Here  $\bar{Q} \coloneqq \min \{\widetilde{Q}(\cdot, a), H - h + 1\}$  is the truncated version.
On the complexity bound. Theorem 1 provides the first analysis for PS algorithms under delay and answers the conjecture from [60]. Our result recovers the best-available frequentist regret of  $\widetilde{O}(\sqrt{d^3H^3T})$  for PS algorithms when there is no delay  $(\mathbb{E}[\tau] = 0)$ . According to [24], the worst-case regret of linear Thompson sampling is lower bounded by  $\Omega(\sqrt{d^3T})$ , and this implies our regret dependencies on parameter  $d$  and  $T$  are optimal under the class of PS algorithms. The order  $\sqrt{H^3}$  in our regret is  $\sqrt{H}$ -suboptimal to the optimal dependence in [27]. As an initial study for posterior sampling with delayed feedback, improving the horizon dependence is beyond our pursuit and we leave it for future work. Moreover, the presence of delay incurs an additive regret term  $\widetilde{O}(d^2H^2\mathbb{E}[\tau])$ . As  $T$  grows, the impact of delay will not dominate the overall regret. Furthermore, our high-probability regret bound directly implies the following worst-case expected regret.
Corollary 1. Under the setting of Theorem 1, the expected regret of Delayed-PSVI is bounded by
$$
\mathbb {E} [ R (T) ] \leq O (\sqrt {d ^ {3} H ^ {3} T \iota}) + O (d ^ {2} H ^ {2} \mathbb {E} [ \tau ] \iota) + O (\iota)
$$
Here  $\iota$  is a Polylog of  $H, d, K$ . The expectation is taken over the randomness in data and algorithm.
Proof of Corollary 1 is included in Appendix A.2. Additionally, we present the following corollary in linear bandits, whose main regret  $\widetilde{O}\sqrt{d^3T}$  is optimal for PS algorithms.
Corollary 2 (Delayed Posterior Sampling for Linear Bandits). For the linear bandit with  $y_{t} = x_{t}^{\mathrm{T}}\theta_{*} + \eta_{t}$ , where  $x_{t}\in D_{t}\subseteq \mathbb{R}^{d}$  and  $\eta_t$  be a mean-zero noise with B-subgaussian. Let  $T$  be the total number of steps. Under Assumption 1, for any  $0 < \delta < 1$ , with probability at least  $1 - \delta$ , the regret of Delayed-PSLB satisfies:
$$
R (T) \leq O (\sqrt {d ^ {3} T \iota}) + O (d ^ {2} \mathbb {E} [ \tau ] \iota) + O (\iota).
$$
Here  $\iota$  is a Polylog term of  $d, K, \delta$ .
3.1 Sketch of the analysis
Due to the space limit, we outline the key steps in our analysis and defer the complete proof of Theorem 1 in Appendix B. To bound the worst-case regret in (2), first note that
$$
R (T) = \sum_ {k = 1} ^ {K} \underbrace {V _ {1} ^ {*} (s _ {1} ^ {k}) - \widetilde {V} _ {1} ^ {k} (s _ {1} ^ {k})} _ {\Delta_ {o p t} ^ {k}} + \underbrace {\widetilde {V} _ {1} ^ {k} (s _ {1} ^ {k}) - V _ {1} ^ {\pi_ {k}} (s _ {1} ^ {k})} _ {\Delta_ {e s t} ^ {k}}.
$$
Our goal is to attain an optimistic estimation so that  $\Delta_{opt}^{k} \leq 0$  while controlling the estimation error  $\Delta_{est}^{k}$ . For optimistic PS algorithms, Gaussian anti-concentration is the main tool [6, 7, 65] to achieve optimism with constant probability. However, the probability of optimism will diminish as the algorithm back-propagates with respect to time. In contrast, we maintain  $m \in [M]$  independent ensembles  $Q^{m}$  so that roughly speaking,  $\mathbb{P}(Q^{m} \geq Q^{*}) \geq \frac{1}{64}$  for all valid  $m$ . For any  $0 < \delta < 1$ , with the choice  $M = \log(1/m)/\log(64/63)$ , the optimistic estimator  $Q = \max_{m} Q^{m}$  satisfies  $\mathbb{P}(Q \geq Q^{*}) \geq 1 - \delta$  (Lemma B.6). We can then proceed to prove  $\Delta_{opt}^{k} \leq 0$ .
To control  $\Delta_{est}^{k}$ , one key challenge is to bound the error term  $\sum_{k=1}^{K} \left\| \phi(s^{k}, a^{k}) \right\|_{(\Omega^{k})^{-1}}$ . Due to the presence of delays, we cannot directly apply the Elliptical Potential Lemma as in the non-delayed settings. Therefore, we decompose  $(\Omega^{k})^{-1}$  into  $(\Sigma^{k})^{-1} + M_{k}$ , where  $\Sigma^{k} := \sum_{\tau=1}^{k-1} \phi(s_{h}^{\tau}, a_{h}^{\tau}) \phi(s_{h}^{\tau}, a_{h}^{\tau})^{\mathrm{T}} + \lambda I$  is the full information matrix, and show
$$
\sum_ {k = 1} ^ {K} \left\| \phi \left(s ^ {k}, a ^ {k}\right) \right\| _ {M _ {k}} \lesssim \max  _ {k \in [ K ]} \tau_ {k} \sum_ {k = 1} ^ {K} \left\| \phi \left(s ^ {k}, a ^ {k}\right) \right\| _ {\left(\Sigma^ {k}\right) ^ {- 1}} ^ {2}.
$$
By doing so,  $\sum_{k=1}^{K}\left\|\phi(s^k,a^k)\right\|_{(\Sigma^k)^{-1}}^2$  can be upper bounded by  $\widetilde{O}(d\log(K))$  via the Elliptical Potential Lemma and  $\max_{k\in [K]}\tau_k$  can be upper bounded by  $\widetilde{O}(\mathbb{E}[\tau])$  via the sub-exponential tail bound. Combining all these steps completes the proof.
Note for non-sampling based on algorithms, e.g. UCB, the regret can attain  $\widetilde{O} (\sqrt{d^2T})$  [1].
Algorithm 2: Delayed Langevin Posterior Sampling Value Iteration (Delayed-LPSVI)
Input:  $w_{0},\eta_{k},N_{k},\gamma$  and rounds  $M,\lambda$  . Delayed loss  $L_h^k$  as (5).   
1 Initialization:  $\forall k\in [K],h\in [H],\widetilde{Q}_{H + 1}^{k}(\cdot ,\cdot)\gets 0,\widetilde{V}_{H + 1}^{k}(\cdot ,\cdot)\gets 0,\widetilde{V}_{h}^{0}(\cdot ,\cdot)\gets 0$    
2 for episode  $k = 1,\ldots ,K$  do   
3 Sample initial state  $s_1^k$    
4 for time step  $h = H,\dots ,1$  do   
5 for  $m = 1,\dots ,M$  do   
6  $\begin{array}{r}\widetilde{w}_h^{k,m}\gets LMC(L_h^k,w_0,\eta_k,N_k,\gamma)\\ \widetilde{Q}_h^{k,m}(\cdot ,\cdot)\gets \phi (\cdot)^{\mathrm{T}}\widetilde{w}_h^{k,m} \end{array}$  //LMC is given by Algorithm 3   
7 Update  $\widetilde{Q}_h^k (\cdot ,\cdot)\gets \max_m\widetilde{Q}_h^{k,m}$    
8  $\widetilde{V}_h^k (\cdot ,\cdot)\gets \max_a\min \{\widetilde{Q}_h^k (\cdot ,a),H - h + 1\}$    
9 Update policy  $\pi_h^k (\cdot)\gets \operatorname {argmax}_{a\in A}\min \{\widetilde{Q}_h^k (\cdot ,a),H - h + 1\}$    
11 for time step  $h = 1,\ldots ,H$  do   
12 Choose action  $a_h^k = \pi_h^k (s_h^k)$    
13 Collect trajectory observations  $\mathcal{D}_h\gets \mathcal{D}_h\cup \{(s_h^k,a_h^k,r_h^k,s_{h + 1}^k)\}$  /\* Feedback generated in episode  $k$  cannot be immediately observed in the presence of delay
4 Delayed Posterior Sampling via Langevin Dynamics
Delayed-PSVI performs noisy value iteration for linear MDPs by injecting randomness for exploration via Gaussian noise. From the Bayesian perspective, it constructs a Laplace approximation to obtain a Gaussian posterior given the observed data. However, sampling from a Gaussian distribution with a general covariance matrix  $\Omega_h^k$  can be computationally expensive in high-dimensional RL tasks. Specifically, Line 10 of Algorithm 1 is conducted via  $\widetilde{w} \coloneqq \widehat{w} + \nu \cdot \Omega^{-1/2}\zeta$ , where  $\zeta \sim \mathcal{N}(0, I_d)$ . The complexity of computing the matrix inverse involved (e.g. via Cholesky decomposition) is at least  $O(d^3)$ , which is prohibitively high for large  $d$ . More importantly, in complex problem domains, a flexible form of non-Gaussian noise perturbation may be desirable.
To tackle these challenges, we incorporate a gradient-based approximate sampling scheme via Langevin dynamics for PS algorithms, namely, LMC, and introduce the Delayed-Langevin Posterior Sampling Value Iteration (Delayed-LPSVI) in Algorithm 2. The update rule of LMC essentially performs the following noisy gradient update:
$$
w _ {t} \leftarrow w _ {t - 1} - \eta \nabla \mathcal {L} \left(w _ {t - 1}\right) + \sqrt {2 \eta \gamma} \epsilon_ {t},
$$
where  $\epsilon_t \stackrel{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, I_d)$ . It is based on the Euler-Murayama discretization of the Langevin stochastic differential equation (SDE):
$$
\mathrm {d} \boldsymbol {w} (t) = - \nabla L (\boldsymbol {w} (t)) \mathrm {d} t + \sqrt {2 \beta^ {- 1}} \mathrm {d} \boldsymbol {B} (t), \tag {4}
$$
where  $B(t) \in \mathbb{R}^d$  is a Brownian motion,  $\beta > 0$  and  $t > 0$ . Under certain regularity conditions on the drift term  $\nabla L(\boldsymbol{w}(t))$  in (4), it can be shown that the Langevin dynamics converges to a unique stationary distribution  $\pi(d\mathbf{w}) \propto \exp(-\beta L(\mathbf{w}))\mathrm{d}\mathbf{w}$ . As a result, LMC is capable of generating samples from arbitrarily complex distributions which can be intractable without closed form. With sufficient number of iterations, the posterior of  $w_t$  is in proportional to  $\exp(-\sqrt{1/\gamma}\mathcal{L}(w))$ .
In our problem, we specify  $\mathcal{L}$  to be the following delayed loss function
$$
L _ {h} ^ {k} (w) := \sum_ {\tau = 1} ^ {k - 1} \mathbb {1} _ {\tau , k - 1} \left(\left\langle \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right), w \right\rangle - \bar {y} _ {h} ^ {\tau}\right) ^ {2} + \lambda \| w \| _ {2} ^ {2}, \tag {5}
$$
where  $\bar{y}_h^\tau \coloneqq r_h^\tau +\widetilde{V}_{h + 1}^k (s_{h + 1}^\tau)$ . Compared to Delayed-PSVI, Algorithm 2 does not require the matrix inversion computation. Below we present the worst-case regret of Delayed-LPSVI and discuss the key insights in our analysis. The full proof is deferred to Appendix C.
Theorem 2. Suppose delays satisfy Assumption 1. In any episodic linear MDP with time horizon  $T = KH$ , where  $K$  is the total number of episodes and  $H$  is the fixed episode length, for any  $0 < \delta < 1$ , let  $\lambda = 1$ ,  $N_{k} = \max \{\log (\frac{32H^{2}(K + \lambda)dk}{\gamma\lambda} +1) / [2\log (1 / (1 - \frac{1}{2\kappa_{h}}))], \frac{\log 2}{2\log(1 / (1 - \frac{1}{2\kappa_{h}}))}, \log (\frac{4HK^{3}}{\sqrt{\lambda / dK}}) / \log (1 / (1 - \frac{1}{2\kappa_{h}}))\}$ ,  $\eta_{k} = \frac{1}{4\lambda_{\mathrm{max}}(\Omega_{h}^{k})}$ ,  $\gamma = 16C_{\delta /4}^{2} \approx \widetilde{O} (dMH^{2})$ ,
$w_{0} = \mathbf{0}$  and  $M = \log (4HK / \delta) / \log (64 / 63)$ . Then with probability at least  $1 - \delta$ , there exists some absolute constants  $c, c', c'' > 0$  such that the regret of Algorithm 2 satisfies:
Here  $\iota$  is a Polylog term of  $H, d, K, \delta$  and  $C_{\delta}$  is defined in Lemma C.9.
Neglecting the constants and Polylog factors, Delayed-LPSVI maintains the same order regret of  $\widetilde{O} (\sqrt{d^3H^3T} +d^2 H^2\mathbb{E}[\tau ])$  as Delayed-PSVI while significantly improving the computational efficiency. Precisely, LMC requires  $O(N)$  complexity to perform gradient steps in Line 6 of Algorithm 2 and an extra  $O(d)$  operations to compute  $\widetilde{Q}_b^{k,m}$  in Line
Algorithm 3: Langevin Monte Carlo LMC(L,  $w_0,\eta ,N,\gamma$
1	for t = 1...N-1 do
2	Draw εt ~ N(0,Id)
3	wt← wt-1 - η∇L(wt-1) + √2ηγεt
4	Output: wtN
7. Thus, the total computation complexity of LMC is  $O((N + d)MHK)$ . On the other hand, sampling without LMC (Line5-8 in Algorithm 1) requires  $O(d^3)$  operations, and the multi-round sampling (Line9-11) incurs  $O(dM)$  additional operations, which implies for a total computation complexity of  $O((d^3 + dM)HK)$ . As the choice of  $N$  in Algorithm 2 has logarithmic order, and  $M = \log(4HK/\delta)/\log(64/63)$ , the overall complexity of Delayed-LPSVI is  $\widetilde{O}(dHK)$ , whereas the overall computational complexity of Delayed-PSVI is  $\widetilde{O}(d^3 HK)$ . Notably, Delayed-LPSVI reduces the computational overhead of Delayed-PSVI by  $\widetilde{O}(d^2)$ .
On the analysis. The key step in the proof of Theorem 2 is to show the convergence guarantee of LMC. Indeed, by recursion, one can show
$$
w _ {N} = A _ {h, k} ^ {N} w _ {0} + \left(I - A _ {h, k} ^ {N}\right) \widehat {w} _ {h} ^ {k} + \sqrt {2 \eta \gamma} \sum_ {l = 0} ^ {N - 1} A _ {h, k} ^ {l} \epsilon_ {N - l},
$$
where  $A_{h,k} \coloneqq I - 2\eta_k\Omega_h^k$ . For any  $w_0$ , it implies  $w_N$  follows the Gaussian distribution  $\mathcal{N}\left(A_{h,k}^{N_k}w_0 + \left(I - A_{h,k}^{N_k}\right)\widehat{w}_h^k,\Theta_h^k\right)$ . With the choice of  $\eta_k = \frac{1}{4\lambda_{\max}\left(\Omega_h^k\right)}$ ,  $A_{h,k} \prec I_d$  and  $\frac{\gamma}{2}(1 - (1 - \frac{1}{2\kappa_h})^{2N_k})(\Omega_h^k)^{-1} \prec \Theta_h^k \prec \gamma (\Omega_h^k)^{-1}$ , which is the key to connect  $\Theta_h^k$  with  $(\Omega_h^k)^{-1}$  (Lemma C.2), the main analysis for Delayed-PSVI goes through by utilizing this connection.
On arbitrary delayed feedback. The current study considers the stochastic delays that are subexponential Assumption 1. What if delay has an arbitrary distribution (e.g. Cauchy distribution has unbounded mean)? Indeed, the regret can be (roughly) bounded by  $\widetilde{O}\left(\frac{1}{q}\sqrt{d^3H^3T} + dH^2 d_\tau(q)\right)$  for  $d_\tau(q)$  to be the  $q$ -th quantile of delay  $\tau$ . We do not focus on this setting since there is a  $1/q$  blow-up in the main regret that many distributions (e.g. sub-exponential) do not need to sacrifice. We include the discussion in Appendix A.3.
To validate whether our posterior sampling algorithms are competitive or outperform the non-sampling-based algorithms in the delayed setting, in this section, we examine their empirical performance in two simulated RL environments with different delayed feedback distributions. In particular, we consider a linear MDP environment following [44, 46], and a variant of the popular River-Swim [55]. In both environments, we benchmark Delayed-PSVI (Algorithm 1), Delayed-LPSVI (Algorithm 2) against LSVI-UCB [35] with delayed feedback, namely, Delayed-UCBVI. In this section, we discuss results in the first setting and defer the discussion of RiverSwim in Appendix E.
5.1 Synthetic Linear MDP
We construct a synthetic linear MDP instance with  $|\mathcal{S}| = 2$ ,  $|\mathcal{A}| = 50$ ,  $d = 10$ , and  $H = 20$ . The linear feature mapping embeds each state-action pair with its binary representation and induces the following reward function:  $r(s, a) = 0.99$  if  $s = 0$ ,  $a = 0$ ;  $r(s, a) = 0.01$  otherwise. The design of the environment results in the same optimal value  $V_1^*(s_1)$  when  $d$  and  $H$  are fixed. Algorithms are examined under three types of delays that are commonly encountered in real-world phenomena, including sub-exponential delays and long-tail delays:
Figure 1: Left:(a) Multinomial delay with delay categories  $\{10,20,30\}$ . (b) Poisson delay with rate  $\mathbb{E}[\tau] = 50$ . (c) Long-tail Pareto delay with shape 1.0, scale 500. Results are reported over 10 experiments. Delayed-PSVI and Delayed-LPSVI demonstrate robust performance under both well-behaved and long-tail delays.
- Multinomial delay. Delays follow a Multinomial distribution with three categories  $\{10, 20, 30\}$ , with the corresponding probabilities as  $\{0.5, 0.3, 0.2\}$ .
- Poisson delay. Delays follow a Poisson distribution with the expected delay as  $\mathbb{E}[\tau] = 50$ .
- Long-tail delay. Delays are discretized from a Pareto distribution with the shape parameter as 1.0 and the scale parameter as 500.
To run Delayed-LPSVI, we warm start LMC by initializing  $w_{0}$  at each time step with the previous sample, and let  $M = 2$ ,  $N = 40$ ,  $\eta = c_{\eta} / \lambda_{\max}(\Omega_h^k)$ . For Delayed-PSVI, we set parameters  $M = 2$ ,  $\nu = \sqrt{d} H$ . In the case of Delayed-UCBVI, we set the bonus coefficient as  $\beta = c_{\beta} / 2 \cdot dH \sqrt{\log(dH)}$ . To make a fair comparison, we perform a grid search to determine the optimal hyperparameter values and fix  $c_{\beta} = 0.1$ ,  $c_{\eta} = 0.5$ ,  $\gamma = 0.02$ . Experiments are repeated with 10 different random seeds, and the returns are averaged over episodes in Figure 1. Further elaboration on additional metrics is available in Appendix E.2.
Results and Discussions. Both Delayed-PSVI and Delayed-LPSVI exhibit consistent and robust performance with resilience, not only under the well-behaved delays that decay exponentially fast, as assumed in Assumption 1, but also under the heavy-tailed delays, such as those following Pareto distributions. Notably, when confronted with the challenge of long-tail delays, our algorithms excel. Delayed-UCBVI in terms of statistical accuracy (yielding higher return) and convergence rate. Specifically, the performance of Delayed-UCBVI degrades under long-tail delays, resulting from its computational inefficiency in iteratively constructing confidence intervals. In contrast, PS methods offer a higher degree of flexibility to adjust the range of exploration, owing to the inherent randomized algorithmic nature. To assess the computational advantages facilitated by LMC, we consider additional synthetic environments with varied dimensions for a more comprehensive analysis. For detailed statistics and further discussions, please refer to Appendix E.2. It is noteworthy that in practical high-dimensional RL tasks, the computational savings achieved by Delayed-LPSVI, in comparison to Delayed-PSVI, are considerably more significant.
In this paper, we study posterior sampling with episodic delayed feedback in linear MDPs. We introduce two novel value-based algorithms: Delayed-PSVI and Delayed-LPSVI. Both algorithms are proved to achieve  $\widetilde{O} (\sqrt{d^3H^3T} +d^2 H^2\mathbb{E}[\tau ])$  worst-case regret. Notably, by incorporating LMC for approximate sampling, Delayed-LPSVI reduces the computational cost by  $\widetilde{O} (d^{2})$  while maintaining the same order of regret. Our empirical experiments further validate the effectiveness of our algorithms by demonstrating their superiority over the UCB-based methods.
This work provides the first delayed-feedback analysis for posterior sampling algorithms in RL, paving the way to several promising avenues for future research. Firstly, it is interesting to extend the current results to settings with general function approximation [34, 71]. Additionally, leveraging the sharp analysis outlined in [27] to improve the suboptimal dependence on  $H$  for posterior sampling algorithms presents an intriguing avenue for exploration. Furthermore, addressing other types of delay (e.g. adversarial delay) that differ from stochastic one will contribute to the ongoing field of delayed feedback studies in online learning, and we leave the investigation in future works.
$^6$  Pareto distribution with shape parameter less than 5.0 are known to have hevy right tails.
Ming Yin and Yu-xiang Wang are gratefully supported by National Science Foundation (NSF) Awards #2007117 and #2003257. Nikki Kuang and Yi-An Ma are supported by the NSF SCALE MoDL-2134209 and the CCF-2112665 (TILOS) awards, as well as the U.S. Department of Energy, Office of Science, and the Facebook Research award. Mengdi Wang gratefully acknowledges funding from Office of Naval Research (ONR) N00014-21-1-2288, Air Force Office of Scientific Research (AFOSR) FA9550-19-1-0203, and NSF 19-589, CMMI-1653435.