Title: Posterior Sampling with Delayed Feedback for Reinforcement Learning with Linear Function Approximation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Posterior Sampling

Reinforcement Learning Theory

Linear Markov Decision Processes

Delayed Feedback

Langevin Monte Carlo
Abstract: Recent studies in reinforcement learning (RL) have made significant progress by leveraging function approximation to alleviate the sample complexity hurdle for better performance. Despite the success, existing provably efficient algorithms typically rely on the accessibility of immediate feedback upon taking actions. The failure to account for the impact of delay in observations can significantly degrade the performance of real-world systems due to the regret blow-up. In this work, we tackle the challenge of delayed feedback in RL with linear function approximation by employing posterior sampling, which has been shown to empirically outperform the popular UCB algorithms in a wide range of regimes. We first introduce \textit{Delayed-PSVI}, an optimistic value-based algorithm that effectively explores the value function space via noise perturbation with posterior sampling. We provide the first analysis for posterior sampling algorithms with delayed feedback in RL and show our algorithm achieves $\widetilde{O}(\sqrt{d^3H^3 T} + d^2H^2 \mathbb{E}[\tau])$ worst-case regret in the presence of unknown stochastic delays. Here $\mathbb{E}[\tau]$ is the expected delay. To further improve its computational efficiency and to expand its applicability in high-dimensional RL problems, we incorporate a gradient-based approximate sampling scheme via Langevin dynamics for \textit{Delayed-LPSVI}, which maintains the same order-optimal regret guarantee with $\widetilde{O}(dHK)$ computational cost. Empirical evaluations are performed to demonstrate the statistical and computational efficacy of our algorithms.

Posterior Sampling with Delayed Feedback for Reinforcement Learning with Linear Function Approximation

Abstract

Recent studies in reinforcement learning (RL) have made significant progress by leveraging function approximation to alleviate the sample complexity hurdle for better performance. Despite the success, existing provably efficient algorithms typically rely on the accessibility of immediate feedback upon taking actions. The failure to account for the impact of delay in observations can significantly degrade the performance of real-world systems due to the regret blow-up. In this work, we tackle the challenge of delayed feedback in RL with linear function approximation by employing posterior sampling, which has been shown to empirically outperform the popular UCB algorithms in a wide range of regimes. We first introduce Delayed-PSVI , an optimistic value-based algorithm that effectively explores the value function space via noise perturbation with posterior sampling. We provide the first analysis for posterior sampling algorithms with delayed feedback in RL and show our algorithm achieves ˜ O ( √ d 3 H 3 T + d 2 H 2 E [ τ ]) worst-case regret in the presence of unknown stochastic delays. Here E [ τ ] is the expected delay. To further improve its computational efficiency and to expand its applicability in high-dimensional RL problems, we incorporate a gradient-based approximate sampling scheme via Langevin dynamics for Delayed-LPSVI , which maintains the same order-optimal regret guarantee with ˜ O ( dHK ) computational cost. Empirical evaluations are performed to demonstrate the statistical and computational efficacy of our algorithms.

1 Introduction

Reinforcement Learning (RL) is the main workhorse for sequential decision-making problems where an agent needs to balance the trade-off between exploitation and exploration in the unknown environment. The flexible and powerful function approximation endowed by deep neural networks greatly contributes to the empirical success of RL in domains such as Large Language Models (LLMs) [50, 59], robotics [51], and AI for Science [37]. In general, collecting real-world training data from such practical systems can be expensive, which requires algorithms to be both sample efficient and computationally efficient. Recently, there have been growing efforts towards studying provably efficient RL algorithms in settings ranging from tabular Markov Decision Processes (MDPs) [29, 45, 69] to large-scale RL with function approximation [13, 35]. However, these algorithms typically rely

on the availability of immediate observations of states, actions and rewards in learning no-regret policies. Unfortunately, such an assumption is rarely satisfied in real-world domains, where delayed feedback is ubiquitous and fundamental. In recommender systems and online advertisement, for instance, responses from users (e.g. click, purchase) may not be immediately observable, which can take hours or days. In healthcare and clinical trials, medical feedback from patients on the effectiveness of treatments can only be determined at a deferred time frame. More examples exist in platforms that involve human interaction and evaluation, including human-robot collaboration in teleoperating systems and multi-agent systems [15, 39], aligning LLMs with human values [50, 63], and fine-tuning generative AI models using RL with human feedback (RLHF) [11, 41].

Despite the practical importance of addressing delays in decision-making problems, theoretical understanding of delayed feedback in RL remains limited. Recent parallel works study exploration under delayed feedback via upper confidence bound (UCB) algorithms [8] in tabular RL [29, 45], adversarial MDPs [36, 40], and RL with low policy-switching scheme [68] (see Table 1). Nevertheless, posterior sampling (PS) analysis that handles delayed feedback remains untackled in both bandit and RL literature. We aim to bridge the gap in this work.

PS is a randomized Bayesian algorithm that extends Thompson sampling (TS) [57] to RL, which selects an action according to its posterior probability of being the best. This philosophy inspires a number of promising exploration strategies that explicitly or implicitly adopt PS to explore [52], including bootstrapped DQN [42, 47] and RLSVI [49]. Compared to the popular UCB algorithms, it bears greater robustness in the presence of delays [14], and provides exceptional computational efficiency with competitive empirical performance [14, 65]. The fact that posteriors are often intractable in practice necessitates the use of approximate Bayesian inference such as ensemble sampling, variational inference (VI) and Markov Chain Monte Carlo (MCMC) [20, 38, 47].

In this paper, we provide the first analysis for the class of PS algorithms that handles delayed feedback in RL frameworks, in which the trajectory information is randomly delayed according to some unknown distribution. We highlight that delayed feedback model imposes new challenges that do not arise in standard RL settings. Algorithmically, it requires the computation of new posterior variance due to the weaker concentration arising from delays. Theoretically, it complicates the frequentist analysis of PS algorithms in several ways: (a) the lack of timely update in posterior learning can cause distribution shift, especially in the case of approximate sampling; (b) delays need to be carefully disentangled to quantify the penalty in regret decomposition and it prohibits the direct application of previous analysis; (c) balance between concentration and anti-concentration needs to be handled deliberately to achieve sub-linear regret.

To tackle these challenges, we introduce two novel value-based algorithms for linear MDPs under unknown stochastic delayed feedback. Developed upon Bayesian linear modeling with a multiround ensembling mechanism ( M ≈ Polylog ( H,K,d,δ ) round), our algorithms achieve a sublinear worst-case regret without requiring the knowledge of delay, thereby addressing the question raised in [60] that 'No frequentist analysis exists for posterior sampling with delayed feedback'. Empirical studies show that our algorithms outperform UCB-based methods in terms of both statistical accuracy and computational efficiency when delays are well-behaved or even long-tailed. We summarize our main contributions as follows.

We propose the Delayed Posterior Sampling Value Iteration (Delayed-PSVI, Algorithm 1) for linear MDPs. It achieves a high-probability worst-case regret of ˜ O ( √ d 3 H 3 T + d 2 H 2 E [ τ ]) 2 , where E [ τ ] is the expected delay.

We leverage Langevin Monte Carlo (LMC) for approximate inference and introduce Delayed Langevin Posterior Sampling Value Iteration (Delayed-LPSVI, Algorithm 2), which maintains the same order-optimal worst-case regret of ˜ O ( √ d 3 H 3 T + d 2 H 2 E [ τ ]) . To the best of our knowledge, this is the first analysis that provably incorporates LMC in linear MDPs and jointly considers the impact of delays.

Both algorithms achieve the optimal dependence on the parameters d and T in leading terms under the class of PS algorithms, and recover the best-available frequentist regret of ˜ O ( √ d 3 H 3 T ) [31, 72] as in non-delayed linear MDPs when E [ τ ] = 0 . In particular, Delayed-LPSVI reduces the computational complexity of Delayed-PSVI from ˜ O ( d 3 HK )

1.1 Related Work.

Delayed feedback. In bandit literature, delay is extensively studied in both stochastic [22, 56, 60, 77] and adversarial settings [32, 58, 78] for UCB-based methods. In comparison, while delay draws much attention in empirical RL studies [12, 17, 18], there is a lack of theoretical understanding until very recently. Parallel works focus on UCB-based methods in various RL settings [16, 28, 36, 40, 45, 68]. To provide the first analysis for PS algorithms in this context, we consider stochastic delays under linear function approximation without requiring any policy-switch scheme as in [68].

Posterior sampling. To encourage efficient exploration, PS is adopted in value-based methods to inject randomness in empirical Bellman update via Gaussian noise. From the Bayesian perspective, it is equivalent to maintaining an approximate Gaussian posterior for parameterized value function. Its sample complexity is studied in tabular settings [48, 49, 53], with the sharp worst-case regret of ˜ O ( H 2 S √ AT ) [5]. Under linear function approximation, frequentist regret of ˜ O ( √ d 3 H 3 T ) [31, 72] and Bayesian regret of ˜ O ( d √ H 3 T ) [19] are established. However, in complex problem domains that require higher computational efficiency and more refined surrogates, approximate inference is the remedy. Toward this end, we resort to a gradient-based MCMC method.

Langevin Monte Carlo. LMC is a class of MCMC methods tailored for large-scale online learning with strong convergence guarantee by utilizing the first-order gradient information [64]. It has been successfully applied to stochastic bandits [43], linear bandits [65] and tabular RL [38], In this work, we extend its usage in linear MDPs and demonstrate its convergent property under delay.

RL with Function Approximation. Function approximation is widely adopted to empower RL for large-scale applications. Fruitful results have been established for regret minimization in two types of MDPs under linear function approximation: linear mixture MDPs [9, 67], and linear MDPs [35, 66]. In linear mixture MDPs where transition kernel is parameterized as a linear combination of base models, provably efficient algorithms are discussed [13, 75, 76] and [75] provides the corresponding lower bound of Ω( dH √ T ) . In contrast, linear MDPs enjoy a linear structure in value functions by assuming a low-rank representation for both transitions and reward function, where algorithms are shown to enjoy polynomial sample complexity [27, 35, 62, 73]. When it comes to general function approximation, theoretical guarantees are developed based on measures of eluder dimension [54, 61] and Bellman rank [33]. In this work, we focus on delayed feedback in linear MDPs.

2 Preliminaries

We study the finite-horizon episodic MDP ( S , A , H, P , r ) , which is time-inhomogeneous, and denote by S , A the state and action spaces respectively, H the episode length, P = { P h } H h =1 the

transition dynamics, and r = { r h } H h =1 reward function. At each step h ∈ [ H ] , P h : S × A → ∆ S specifies the probabilities of transitioning from the current state-action pair into the next state, and r h : S×A → [0 , 1] emits a bounded reward. We adopt the prior protocol of linear MDPs as follows.

Definition 1 (Linear MDPs [35, 66]) . Suppose there exists a known feature map ϕ : S × A → R d that encodes each state-action pair into a d -dimensional feature vector. An MDP is a linear MDP 3 if for any time step h ∈ [ H ] , ∀ ( s, a ) ∈ S × A , both the transition dynamics P and reward function r are linear in ϕ :

where µ h : S → R d contains d unknown probability measures over S , and θ h ∈ R d . Furthermore, we assume that ∀ ( s, a ) ∈ S × A , ∥ ϕ ( s, a ) ∥ ≤ 1 , and ∀ h ∈ [ H ] , ∥ θ h ∥ ≤ √ d , ∥ ∥ ∫ S d µ h ( s ′ ) ∥ ∥ ≤ √ d , where ∥·∥ denotes the Euclidean norm.

A non-stationary policy π = { π h } H h =1 assigns the action to take at step h in state s h ∈ S . Accordingly, we define the value functions of a policy π as the expected rewards received under π :

We further denote by π ∗ the optimal policy whose value functions are defined as V ∗ h ( s ) := V π ∗ h ( s ) = sup π V π h ( s ) and Q ∗ h ( s, a ) := Q π ∗ h ( s, a ) = sup π Q π h ( s, a ) . Under Definition 1, the action-value functions are always linear in the feature map, and there exists some w ∗ h such that Q ∗ h = ϕ T w ∗ h (Lemma A.1). For ease of notation, , ∀ ( s, a ) , denote [ P h V π h +1 ]( s, a ) = E s ′ ∼ P h ( ·| s,a ) [ V ( s ′ )] . By Bellman equation and Bellman optimality equation,

The goal of the agent is to maximize the cumulative episodic rewards or equivalently, minimize the regret that quantifies the difference between the value of the optimal policy π ∗ and that of the executed policies. Formally, the worse-case regret over K episodes is given as:

Remark 1. Different types of regret are used in literature to measure the performance of PS algorithms. Bayesian regret E w ∗ ∼ p 0 ( · ) E [ R ( T ) | w ∗ ] is often considered when assuming a prior p 0 ( w ) over the true parameter w ∗ . Frequentist regret E [ R ( T )] is considered when w ∗ is fixed, where the expectation is taken over all the randomness over data and algorithm. As explained in Appendix A.2, the worst-case regret that we study is stronger than the frequentist regret.

2.1 Delayed Feedback Model

In this work, we consider stochastic delays across episodes. More specifically, the trajectory (i.e., sequence of states, actions and rewards) generated in each episode is not immediately observable in the presence of delay. The formal definition is given as follows.

Definition 2 (Episodic Delayed Feedback) . In each episode k ∈ [ K ] , the execution of a fixed policy π k generates a trajectory { s k h , a k h , r k h , s k h +1 } h ∈ [ H ] . Such trajectory information is called the feedback of episode k . Let τ k represent the random delay between the rollout completion of episode k and the time point at which its feedback becomes observable.

Remark 2. Various types of delays have been independently studied in the literature, including delays in states [4, 12, 16], delays in rewards [25, 45, 60], delays in actions[56], and delays in trajectories [29, 68]. We focus on the last scheme which facilitates the delayed analysis of valuebased methods in episodic linear MDPs.

Episodic delays do not disrupt the policy rollout within an episode, but alter the utilization of information in subsequent episodes. More precisely, the feedback of episode k remains inaccessible for the following τ k -1 episodes, becoming observable only at the onset of the ( k + τ k )-th episode. To track whether the feedback generated at episode k is revealed at episode k ′ , we utilize the indicator 1 k,k ′ := 1 { k + τ k ≤ k ′ } (where 1 denotes 'yes' and 0 denotes 'no'). We follow the standard assumption in literature in [28, 68] to assume delays are sub-exponential. It is crucial to note that this assumption primarily serves the purpose of theoretical analysis and is not a prerequisite for the effective functioning of our algorithms in practical settings. Without loss of generality, we discuss the performance bound under general random delays in Section 4 and empirically study the performance against different types of delays in Section 5.

Assumption 1 (Sub-exponential Episodic Delay) . The episodic delays { τ k } K k =1 are non-negative, integer-valued, independent and identically distributed ( v, b ) -subexponential random variables: τ k i.i.d. ∼ f τ ( · ) with f τ ( · ) being the probability mass function, and E [ τ ] being the expected value. For all k ∈ [ K ] , the moment generating function of τ k satisfies:

where v and b are non-negative, and | γ | ≤ 1 /b .

3 Delayed Posterior Sampling Value Iteration

In this section, we introduce a novel optimistic value-based algorithm, namely, Delayed Posterior Sampling Value Iteration (Delayed-PSVI), which efficiently explores the value function space in linear MDPs by embracing several critical components: posterior sampling that injects random noise when performing the least-square value iteration, optimism via multi-round sampling to achieve the optimal worst-case regret and delayed feedback model that encodes episodic trajectory delays.

Noisy value iteration via posterior sampling. At the beginning of each episode, we apply PS to sample an estimated value function from the posterior, which is maintained using the observed feedback D over the previous episodes. Specifically, at each time step, the Q -function is parameterized by some w ∈ Γ such that ˜ Q ( s, a ) = ϕ ( s, a ) T w is an approximation of the corresponding true optimal Q -function Q ∗ ( s, a ) . Let p 0 ( w ) be the prior of w , and p ( y | w, D ) be the likelihood of the observation y , then the posterior of w satisfies:

where L ( · ) is the log-likelihood. Unlike the case of model-based RL (MBRL), where PS is utilized to maintain an exact posterior over the environment model, we aim to adopt PS to perform noisy valueiteration by injecting randomness for efficient exploration of the value function space. Specifically, at each step h ∈ [ H ] , we consider Gaussian-noise perturbation in Delayed-PSVI by setting prior as

where y h = [ y 1 h , . . . , y k -1 h ] with y τ h = r τ h ( s τ h , a τ h ) + ˜ V h +1 ( s τ h +1 ) . Then for all step h ∈ [ H ] of episode k , the posterior of w k h follows a Gaussian distribution,

where Ω k h := Φ h Φ T h + λI d and Φ h = [ ϕ ( s 1 h , a 1 h ) , ϕ ( s 2 h , a 2 h ) , . . . , ϕ ( s k -1 h , a k -1 h )] . Adding the scaling factors σ 2 and ν 2 yields the Line 10 of Algorithm 1. It is important to note that while the induced likelihood exp( -L ( w k h , y k h , D k h )) from (3) is Gaussian, we do not assume y τ h = r τ h ( s τ h , a τ h ) + ˜ V h +1 ( s τ h +1 ) follows a Gaussian distribution. Instead, the above likelihood model can be used for non-Gaussian problems as we need not sample from the exact Bayesian posterior model [2, 74].

On the other hand, the ̂ w k h computed in Line 9 of Algorithm 1 together with the greedy choice ˜ V ( · ) ≈ max a ˜ Q ( · , a ) (Line 15) approximates the solution of Bellman optimality equation via the least-square ridge regression: ̂ w k h = argmin w ∑ k -1 τ =1 ( ϕ ( s τ h , a τ h ) T w -( r +max ¯ Q k h )) 2 + λI d . 4 Consequently, Line 5-10 essentially performs the Posterior Sampling Value Iteration.

Optimism via multi-round sampling scheme. Unlike the Bayesian regret or the worst-case expected regret, the high-probability worst-case regret in (2) needs to control the sub-optimal gap with arbitrarily high probability of at least 1 -δ . However, sampling once at each time step only provides a constant-probability optimistic estimation, which breaks the high probability requirement. In addition, the estimation error incurred by sampling (i.e. constant-probability pessimistic estimation) at each timestep will propagate to the previous time steps during the backward posterior sampling value iteration. This phenomenon does not appear in the 1 -horizon bandit problem due to a saturated-arm analysis [2, 6]. To remedy this issue, we design a multi-round sampling scheme that generates M estimates { ˜ Q m } m ∈ [ M ] for Q -fuction through M i.i.d. sampling procedures, and constructs an optimistic estimate by setting ˜ Q = max m ˜ Q m . Notably, our choice of M has order Polylog ( H,K,d,δ ) , and thus makes our algorithm sample-efficient without increasing the overall complexity dependence. As shown in Line 11-14 of Algorithm 1, this scheme guarantees the optimistic estimates ˜ Q ≥ Q ∗ can be achieved as desired. Lastly, ensemble sampling methods enjoy empirical success and popularity in RL [21, 23, 31], including double q-learning [26] and bootstrapped DQN [42, 47]. We are among the first few works to explain its theoretical effectiveness.

Episodic delayed feedback model. Recall that by Definition 2, when delay τ k takes place, the feedback { s t h , a t h , r t h , s t h +1 } h ∈ [ H ] of episode k cannot be observed until the beginning of the k + τ k -th episode. Accordingly, the delayed version of the fully observed y τ , Ω k h now becomes,

As a result, episodic delays are considered during the posterior updates in subsequent episodes. This completes the design of Delayed-PSVI as presented in Algorithm 1. In the remainder of this section, we present the main theoretical guarantees of Delayed-PSVI and the proof sketch of Theorem 1.

Theorem 1. Suppose delays satisfy Assumption 1. In any episodic linear MDP with time horizon T = KH , where K is the total number of episodes, for any 0 < δ < 1 , let λ = 1 , σ 2 = 1 , M = log(4 HK/δ ) / log(64 / 63) and ν = C δ/ 4 ≈ ˜ O ( √ dMH 2 ) ( C δ/ 4 in Lemma B.10). Then with probability at least 1 -δ , there exists some absolute constants c, c ′ , c ′′ > 0 such that the regret of Delayed-PSVI (Algorithm 1) satisfies:

Here ι is a Polylog term of H,d,K,δ .

On the complexity bound. Theorem 1 provides the first analysis for PS algorithms under delay and answers the conjecture from [60]. Our result recovers the best-available frequentist regret of ˜ O ( √ d 3 H 3 T ) for PS algorithms when there is no delay ( E [ τ ] = 0 ). According to [24], the worstcase regret of linear Thompson sampling is lower bounded by Ω( √ d 3 T ) , and this implies our regret dependencies on parameter d and T are optimal under the class of PS algorithms. 5 The order √ H 3 in our regret is √ H -suboptimal to the optimal dependence in [27]. As an initial study for posterior sampling with delayed feedback, improving the horizon dependence is beyond our pursuit and we leave it for future work. Moreover, the presence of delay incurs an additive regret term ˜ O ( d 2 H 2 E [ τ ]) . As T grows, the impact of delay will not dominate the overall regret. Furthermore, our high-probability regret bound directly implies the following worst-case expected regret.

Corollary 1. Under the setting of Theorem 1, the expected regret of Delayed-PSVI is bounded by

Here ι is a Polylog of H,d,K . The expectation is taken over the randomness in data and algorithm.

Proof of Corollary 1 is included in Appendix A.2. Additionally, we present the following corollary in linear bandits, whose main regret ˜ O √ d 3 T is optimal for PS algorithms.

Corollary 2 (Delayed Posterior Sampling for Linear Bandits) . For the linear bandit with y t = x T t θ ∗ + η t , where x t ∈ D t ⊆ R d and η t be a mean-zero noise with B -subgaussian. Let T be the total number of steps. Under Assumption 1, for any 0 < δ < 1 , with probability at least 1 -δ , the regret of Delayed-PSLB satisfies:

Here ι is a Polylog term of d, K, δ .

3.1 Sketch of the analysis

Due to the space limit, we outline the key steps in our analysis and defer the complete proof of Theorem 1 in Appendix B. To bound the worst-case regret in (2), first note that

Our goal is to attain an optimistic estimation so that ∆ k opt ≤ 0 while controlling the estimation error ∆ k est . For optimistic PS algorithms, Gaussian anti-concentration is the main tool [6, 7, 65] to achieve optimism with constant probability. However, the probability of optimism will diminish as the algorithm back-propagates with respect to time. In contrast, we maintain m ∈ [ M ] independent ensembles Q m so that roughly speaking, P ( Q m ≥ Q ∗ ) ≥ 1 64 for all valid m . For any 0 < δ < 1 , with the choice M = log(1 /m ) / log(64 / 63) , the optimistic estimator Q = max m Q m satisfies P ( Q ≥ Q ∗ ) ≥ 1 -δ (Lemma B.6). We can then proceed to prove ∆ k opt ≤ 0 .

To control ∆ k est , one key challenge is to bound the error term ∑ K k =1 ∥ ∥ ϕ ( s k , a k )∥ ∥ ( Ω k ) -1 . Due to the presence of delays, we cannot directly apply the Elliptical Potential Lemma as in the non-delayed settings. Therefore, we decompose (Ω k ) -1 into (Σ k ) -1 + M k , where Σ k := ∑ k -1 τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) T + λI is the full information matrix, and show

By doing so, ∑ K k =1 ∥ ∥ ϕ ( s k , a k ) ∥ ∥ 2 (Σ k ) -1 can be upper bounded by ˜ O ( d log( K )) via the Elliptical Potential Lemma and max k ∈ [ K ] τ k can be upper bounded by ˜ O ( E [ τ ]) via the sub-exponential tail bound. Combing all these steps completes the proof.

4 Delayed Posterior Sampling via Langevin Dynamics

Delayed-PSVI performs noisy value iteration for linear MDPs by injecting randomness for exploration via Gaussian noise. From the Bayesian perspective, it constructs a Laplace approximation to obtain a Gaussian posterior given the observed data. However, sampling from a Gaussian distribution with a general covariance matrix Ω k h can be computationally expensive in high-dimensional RL tasks. Specifically, Line 10 of Algorithm 1 is conducted via ˜ w := ̂ w + ν · Ω -1 / 2 ζ , where ζ ∼ N (0 , I d ) . The complexity of computing the matrix inverse involved ( e.g. via Cholesky decomposition) is at least O ( d 3 ) , which is prohibitively high for large d . More importantly, in complex problem domains, a flexible form of non-Gaussian noise perturbation may be desirable.

To tackle these challenges, we incorporate a gradient-based approximate sampling scheme via Langevin dynamics for PS algorithms, namely, LMC, and introduce the Delayed-Langevin Posterior Sampling Value Iteration (Delayed-LPSVI) in Algorithm 2. The update rule of LMC essentially performs the following noisy gradient update:

where ϵ t i.i.d. ∼ N (0 , I d ) . It is based on the Euler-Murayama discretization of the Langevin stochastic differential equation (SDE):

where B ( t ) ∈ R d is a Brownian motion, β > 0 and t > 0 . Under certain regularity conditions on the drift term ∇ L ( w ( t )) in (4), it can be shown that the Langevin dynamics converges to a unique stationary distribution π ( d w ) ∝ exp( -βL ( w )) d w . As a result, LMC is capable of generating samples from arbitrarily complex distributions which can be intractble without closed form. With sufficient number of iterations, the posterior of w t is in proportional to exp( -√ 1 /γ L ( w )) .

In our problem, we specify L to be the following delayed loss function

where ¯ y τ h := r τ h + ˜ V k h +1 ( s τ h +1 ) . Compared to Delayed-PSVI, Algorithm 2 does not require the matrix inversion computation. Below we present the worst-case regret of Delayed-LPSVI and discuss the key insights in our analysis. The full proof is deferred to Appendix C.

Theorem 2. Suppose delays satisfy Assumption 1. In any episodic linear MDP with time horizon T = KH , where K is the total number of episodes and H is the fixed episode length, for any 0 < δ < 1 , let λ = 1 , N k = max { log( 32 H 2 ( K + λ ) dk γλ + 1) / [2 log(1 / (1 -1 2 κ h ))] , log 2 2 log(1 / (1 -1 2 )) , log( 4 HK 3 √ λ/dK ) / log(1 / (1 -1 2 κ h )) } , η k = 1 4 λ max (Ω k h ) , γ = 16 C 2 δ/ 4 ≈ ˜ O ( dMH 2 ) ,

w 0 = 0 and M = log(4 HK/δ ) / log(64 / 63) . Then with probability at least 1 -δ , there exists some absolute constants c, c ′ , c ′′ > 0 such that the regret of Algorithm 2 satisfies:

Here ι is a Polylog term of H,d,K,δ and C δ is defined in Lemma C.9.

Neglecting the constants and Polylog factors, Delayed-LPSVI maintains the same order regret of ˜ O ( √ d 3 H 3 T + d 2 H 2 E [ τ ]) as Delayed-PSVI while significantly improving the computational efficiency. Precisely, LMC requires O ( N ) complexity to perform gradient steps in Line 6 of Algorithm 2 and an extra O ( d ) operations to compute ˜ Q k,m h in Line

7. Thus, the total computation complexity of LMC is O (( N + d ) MHK ) . On the other hand, sampling without LMC (Line5-8 in Algorithm 1) requires O ( d 3 ) operations, and the multi-round sampling (Line9-11) incurs O ( dM ) additional operations, which implies for a total computation complexity of O (( d 3 + dM ) HK ) . As the choice of N in Algorithm 2 has logarithmic order, and M = log(4 HK/δ ) / log(64 / 63) , the overall complexity of Delayed-LPSVI is ˜ O ( dHK ) , whereas the overall computational complexity of Delayed-PSVI is ˜ O ( d 3 HK ) . Notably, Delayed-LPSVI reduces the computational overhead of Delayed-PSVI by ˜ O ( d 2 ) .

On the analysis. The key step in the proof of Theorem 2 is to show the convergence guarantee of LMC. Indeed, by recursion, one can show

where A h,k := I -2 η k Ω k h . For any w 0 , it implies w N follows the Gaussian distribution N ( A N k h,k w 0 + ( I -A N k h,k ) ̂ w k h , Θ k h ) . With the choice of η k = 1 4 λ max ( Ω k h ) , A h,k ≺ I d and γ 2 (1 -(1 -1 2 κ h ) 2 N k ) ( Ω k h ) -1 ≺ Θ k h ≺ γ ( Ω k h ) -1 , which is the key to connect Θ k h with (Ω k h ) -1 (Lemma C.2), the main analysis for Delayed-PSVI goes through by utilizing this connection.

On arbitrary delayed feedback. The current study considers the stochastic delays that are subexponential Assumption 1. What if delay has an arbitrary distribution ( e.g. Cauchy distribution has unbounded mean)? Indeed, the regret can be (roughly) bounded by ˜ O ( 1 q √ d 3 H 3 T + dH 2 d τ ( q )) for d τ ( q ) to be the q -th quantile of delay τ . We do not focus on this setting since there is a 1 /q blow-up in the main regret that many distributions ( e.g. sub-exponential) do not need to sacrifice. We include the discussion in Appendix A.3.

5 Experiments

To validate whether our posterior sampling algorithms are competitive or outperform the nonsampling-based algorithms in the delayed setting, in this section, we examine their empirical performance in two simulated RL environments with different delayed feedback distributions. In particular, we consider a linear MDP environment following [44, 46], and a variant of the popular RiverSwim [55]. In both environments, we benchmark Delayed-PSVI (Algorithm 1), Delayed-LPSVI (Algorithm 2) against LSVI-UCB [35] with delayed feedback, namely, Delayed-UCBVI. In this section, we discuss results in the first setting and defer the discussion of RiverSwim in Appendix E.

5.1 Synthetic Linear MDP

We construct a synthetic linear MDP instance with |S| = 2 , |A| = 50 , d = 10 , and H = 20 . The linear feature mapping embeds each state-action pair with its binary representation and induces the following reward function: r ( s, a ) = 0 . 99 if s = 0 , a = 0 ; r ( s, a ) = 0 . 01 otherwise. The design of the environment results in the same optimal value V ∗ 1 ( s 1 ) when d and H are fixed. Algorithms are examined under three types of delays that are commonly encountered in real-world phenomena, including sub-exponential delays and long-tail delays:

12

12

12

10

10

10

eturn

Return

Ret

8

8

Ret

8

Average

6

Average

6

Average

6

4

4

4

2

D-PSVI

2

D-PSVI

2

D-PSVI

D-LPSVI

D-LPSVI

D-LPSVI

0

D-UCBVI

0

D-UCBVI

0

D-UCBVI

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

Episode

Episode

Episode

Multinomial delay. Delays follow a Multinomial distribution with three categories { 10 , 20 , 30 } , with the corresponding probabilities as { 0 . 5 , 0 . 3 , 0 . 2 } .

Poisson delay. Delays follow a Poisson distribution with the expected delay as E [ τ ] = 50 .

Long-tail delay. Delays are discretized from a Pareto distribution 6 with the shape parameter as 1 . 0 and the scale parameter as 500 .

To run Delayed-LPSVI, we warm start LMC by initializing w 0 at each time step with the previous sample, and let M = 2 , N = 40 , η = c η /λ max (Ω k h ) . For Delayed-PSVI, we set parameters M = 2 , ν = √ dH . In the case of Delayed-UCBVI, we set the bonus coefficient as β = c β / 2 · dH √ log( dH ) . To make a fair comparison, we perform a grid search to determine the optimal hyperparameter values and fix c β = 0 . 1 , c η = 0 . 5 , γ = 0 . 02 . Experiments are repeated with 10 different random seeds, and the returns are averaged over episodes in Figure 1. Further elaboration on additional metrics is available in Appendix E.2.

Results and Discussions. Both Delayed-PSVI and Delayed-LPSVI exhibit consistent and robust performance with resilience, not only under the well-behaved delays that decay exponentially fast, as assumed in Assumption 1, but also under the heavy-tailed delays, such as those following Pareto distributions. Notably, when confronted with the challenge of long-tail delays, our algorithms excel Delayed-UCBVI in terms of statistical accuracy (yielding higher return) and convergence rate. Specifically, the performance of Delayed-UCBVI degrades under long-tail delays, resulting from its computational inefficiency in iteratively constructing confidence intervals. In contrast, PS methods offer a higher degree of flexibility to adjust the range of exploration, owing to the inherent randomized algorithmic nature. To assess the computational advantages facilitated by LMC, we consider additional synthetic environments with varied dimensions for a more comprehensive analysis. For detailed statistics and further discussions, please refer to Appendix E.2. It is noteworthy that in practical high-dimensional RL tasks, the computational savings achieved by Delayed-LPSVI, in comparison to Delayed-PSVI, are considerably more significant.

6 Conclusion

In this paper, we study posterior sampling with episodic delayed feedback in linear MDPs. We introduce two novel value-based algorithms: Delayed-PSVI and Delayed-LPSVI. Both algorithms are proved to achieve ˜ O ( √ d 3 H 3 T + d 2 H 2 E [ τ ]) worst-case regret. Notably, by incorporating LMC for approximate sampling, Delayed-LPSVI reduces the computational cost by ˜ O ( d 2 ) while maintaining the same order of regret. Our empirical experiments further validate the effectiveness of our algorithms by demonstrating their superiority over the UCB-based methods.

This work provides the first delayed-feedback analysis for posterior sampling algorithms in RL, paving the way to several promising avenues for future research. Firstly, it is interesting to extend the current results to settings with general function approximation [34, 71]. Additionally, leveraging the sharp analysis outlined in [27] to improve the suboptimal dependence on H for posterior sampling algorithms presents an intriguing avenue for exploration. Furthermore, addressing other types of delay (e.g. adversarial delay) that differ from stochastic one will contribute to the ongoing field of delayed feedback studies in online learning, and we leave the investigation in future works.

References

Yasin Abbasi-Yadkori, D´ avid P´ al, and Csaba Szepesv´ ari. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Marc Abeille and Alessandro Lazaric. Linear thompson sampling revisited. In Artificial Intelligence and Statistics , pages 176-184. PMLR, 2017.

Milton Abramowitz and Irene A Stegun. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables , volume 55. US Government Printing Office, 1964.

Mridul Agarwal and Vaneet Aggarwal. Blind decision making: Reinforcement learning with delayed observations. Pattern Recognition Letters , 150:176-182, 2021.

Priyank Agrawal, Jinglin Chen, and Nan Jiang. Improved worst-case regret bounds for randomized least-squares value iteration. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 6566-6573, 2021.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. Advances in Neural Information Processing Systems , 30, 2017.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Mohammad Gheshlaghi Azar, Ian Osband, and R´ emi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 , 2023.

Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Reinforcement learning with random delays. In International conference on learning representations , 2020.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems , 24, 2011.

Baiming Chen, Mengdi Xu, Zuxin Liu, Liang Li, and Ding Zhao. Delay-aware multiagent reinforcement learning for cooperative and competitive environments. arXiv preprint arXiv:2005.05441 , 2020.

Minshuo Chen, Yu Bai, H Vincent Poor, and Mengdi Wang. Efficient rl with impaired observability: Learning to act with delayed and missing state observations. arXiv preprint arXiv:2306.01243 , 2023.

Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with nonstationary markov policies. In International Conference on Learning Representations , 2020.

Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 , 2019.

Ying Fan and Yifei Ming. Model-based reinforcement learning for continuous control with posterior sampling. In International Conference on Machine Learning , pages 3078-3087. PMLR, 2021.

Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. Virel: A variational inference framework for reinforcement learning. Advances in neural information processing systems , 32, 2019.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Manegueu Anne Gael, Claire Vernade, Alexandra Carpentier, and Michal Valko. Stochastic bandits with arm-dependent delays. In International Conference on Machine Learning , pages 3348-3356. PMLR, 2020.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Nima Hamidi and Mohsen Bayati. On frequentist regret of linear thompson sampling. arXiv preprint arXiv:2006.06790 , 2020.

Beining Han, Zhizhou Ren, Zuofan Wu, Yuan Zhou, and Jian Peng. Off-policy reinforcement learning with delayed rewards. In International Conference on Machine Learning , pages 82808303. PMLR, 2022.

Hado Hasselt. Double q-learning. Advances in neural information processing systems , 23, 2010.

Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear markov decision processes. In International Conference on Machine Learning , pages 12790-12822. PMLR, 2023.

Benjamin Howson, Ciara Pike-Burke, and Sarah Filippi. Delayed feedback in generalised linear bandits revisited. In International Conference on Artificial Intelligence and Statistics , pages 6095-6119. PMLR, 2023.

Benjamin Howson, Ciara Pike-Burke, and Sarah Filippi. Optimism and delays in episodic reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 6061-6094. PMLR, 2023.

Daniel Hsu, Sham Kakade, and Tong Zhang. A tail inequality for quadratic forms of subgaussian random vectors. 2012.

Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning , pages 4607-4616. PMLR, 2021.

Shinji Ito, Daisuke Hatano, Hanna Sumita, Kei Takemura, Takuro Fukunaga, Naonori Kakimura, and Ken-Ichi Kawarabayashi. Delay and cooperation in nonstochastic linear bandits. Advances in Neural Information Processing Systems , 33:4872-4883, 2020.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR, 2017.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418, 2021.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Tiancheng Jin, Tal Lancewicki, Haipeng Luo, Yishay Mansour, and Aviv Rosenberg. Near-optimal regret for adversarial mdp with delayed bandit feedback. arXiv preprint arXiv:2201.13172 , 2022.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin ˇ Z´ ıdek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Amin Karbasi, Nikki Lijing Kuang, Yian Ma, and Siddharth Mitra. Langevin thompson sampling with logarithmic communication: bandits and reinforcement learning. In International Conference on Machine Learning , pages 15828-15860. PMLR, 2023.

Parham M Kebria, Abbas Khosravi, Saeid Nahavandi, Peng Shi, and Roohallah Alizadehsani. Robust adaptive control scheme for teleoperation systems with delay and uncertainties. IEEE transactions on cybernetics , 50(7):3243-3253, 2019.

Tal Lancewicki, Aviv Rosenberg, and Yishay Mansour. Learning adversarial markov decision processes with delayed feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7281-7289, 2022.

Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023.

Ziniu Li, Yingru Li, Yushun Zhang, Tong Zhang, and Zhi-Quan Luo. Hyperdqn: A randomized exploration method for deep reinforcement learning. In International Conference on Learning Representations , 2021.

Eric Mazumdar, Aldo Pacchiano, Yian Ma, Michael Jordan, and Peter Bartlett. On approximate thompson sampling with langevin algorithms. In International Conference on Machine Learning , pages 6797-6807. PMLR, 2020.

Yifei Min, Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Variance-aware off-policy evaluation with linear function approximation. Advances in neural information processing systems , 34:7598-7610, 2021.

Washim Uddin Mondal and Vaneet Aggarwal. Reinforcement learning with delayed, composite, and partially anonymous reward. arXiv preprint arXiv:2305.02527 , 2023.

Thanh Nguyen-Tang, Ming Yin, Sunil Gupta, Svetha Venkatesh, and Raman Arora. On instance-dependent bounds for offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2211.13208 , 2022.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res. , 20(124):1-62, 2019.

Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning , pages 2377-2386. PMLR, 2016.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864 , 2023.

Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference on Learning Representations , 2018.

Daniel Russo. Worst-case regret bounds for exploration via randomized value functions. Advances in Neural Information Processing Systems , 32, 2019.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences , 74(8):1309-1331, 2008.

Wei Tang, Chien-Ju Ho, and Yang Liu. Bandit learning with delayed impact of actions. Advances in Neural Information Processing Systems , 34:26804-26817, 2021.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Tobias Sommer Thune, Nicol` o Cesa-Bianchi, and Yevgeny Seldin. Nonstochastic multiarmed bandits with unrestricted delays. Advances in Neural Information Processing Systems , 32, 2019.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Claire Vernade, Alexandra Carpentier, Tor Lattimore, Giovanni Zappella, Beyza Ermis, and Michael Brueckner. Linear bandits with stochastic delayed feedback. In International Conference on Machine Learning , pages 9712-9721. PMLR, 2020.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020.

Yining Wang, Ruosong Wang, Simon S Du, and Akshay Krishnamurthy. Optimism in reinforcement learning with generalized linear function approximation. arXiv preprint arXiv:1912.04136 , 2019.

Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966 , 2023.

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688, 2011.

Pan Xu, Hongkai Zheng, Eric V Mazumdar, Kamyar Azizzadenesheli, and Animashree Anandkumar. Langevin monte carlo for contextual bandits. In International Conference on Machine Learning , pages 24830-24850. PMLR, 2022.

Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning , pages 6995-7004. PMLR, 2019.

Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning , pages 10746-10756. PMLR, 2020.

Yunchang Yang, Han Zhong, Tianhao Wu, Bin Liu, Liwei Wang, and Simon S Du. A reduction-based framework for sequential decision making with delayed feedback. arXiv preprint arXiv:2302.01477 , 2023.

Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 1567-1575. PMLR, 2021.

Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. International Conference on Learning Representations , 2022.

Ming Yin, Mengdi Wang, and Yu-Xiang Wang. Offline reinforcement learning with differentiable function approximation is provably efficient. International Conference on Learning Representations , 2023.

Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , pages 1954-1964. PMLR, 2020.

Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science , 4(2):834-857, 2022.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning , pages 12793-12802. PMLR, 2021.

Zhengyuan Zhou, Renyuan Xu, and Jose Blanchet. Learning in generalized linear contextual bandits with stochastic delays. Advances in Neural Information Processing Systems , 32, 2019.

Julian Zimmert and Yevgeny Seldin. An optimal algorithm for adversarial bandits with arbitrary delays. In International Conference on Artificial Intelligence and Statistics , pages 32853294. PMLR, 2020.

Appendices

A Some Properties

A.1 Properties of Linear MDPs

Lemma A.1. In linear MDPs, the action-value function is also linear in feature map. ∀ ( s, a ) ∈ S × A , h ∈ [ H ] and ϕ ∈ R d , under any fixed policy π ,

where w π h := θ h + E µ [ V π h +1 ( s ′ )] and w h ∈ R d . As a corollary, there exists w ∗ h such that Q ∗ h = ϕ T w ∗ h .

Proof of Lemma A.1. By Bellman equation,

where w π := θ h + E µ h [ V π ( s ′ )]

A.2 Worst-case regret as a stronger criterion

We use Theorem 1 as an example. Using the worst-case result, i.e. with probability 1 -δ ,

Here ι has the functional form ι = Polylog ( d, K, H, δ ) . Then choosing δ = 1 / ( HK ) to obtain with probability 1 -1 / ( HK ) ,

for ι = Polylog ( d, K, H ) . Therefore,

This completes Corollary 1.

A.3 Discussion on the arbitrary delay

For completeness of our study, we also briefly discuss the case when delay is arbitrary. In genreal, the regret can be (roughly) bounded by ˜ O ( 1 q √ d 3 H 3 T + dH 2 d τ ( q )) for d τ ( q ) to be the q -th quantile of delay τ . This could be achieved by creating a low-switching variant of our Theorem 1/Theorem 2 and applying the reduction of the concurrent work [68]. We do not focus on this setting since there is a 1 /q blow-up in the main regret that many distributions ( e.g. sub-exponential) do not need to sacrifice.

B Regret Analysis for Delayed-PSVI

To proceed with the regret analysis, we introduce some helpful notations. Besides ˜ Q k h ( s, a ) = max m ϕ ( s, a ) T , ˜ w k,m h , ˜ V k h ( s ) = max a ˜ Q k h ( s, a ) in Algorithm 1, we define

Regret decomposition: We start by rewriting regret in terms of value-function error decomposition following the standard analysis of optimistic algorithms [10]:

where at each episode k , ∆ k opt corresponds to the regret resulting from optimism, and ∆ k est tracks down the regret incurred from estimation error. Efficient RL algorithms thus need to strike a balance between both terms. More specifically, it is desirable to generate optimistic estimations over the true value function, while keeping estimation error relatively small. By cautious design of noise perturbation, we show in Theorem 1 that Algorithm 1 effectively achieves √ T order regret in episodic MDPs with linear function approximation.

Proof of Theorem 1. The proof proceeds by bounding ∆ k opt and ∆ k est respectively.

Step 1: bound regret from optimism.

By Lemma B.7, the optimism provided by our algorithm guarantees with probability at least 1 -δ/ 2 , for all k ∈ [ K ] , ∆ k opt := V ∗ 1 ( s k 1 ) -˜ V k 1 ( s k 1 ) ≤ 0 .

Step 2: bound regret from estimation error. To bound the estimation error, we first condition on the following event

K , with β := √ 2 ν 2 log(16 C d HMK/δ )+ √ 8 H 2 [ d 2 log ( k + λ λ ) + dM log(1 + 2 √ 8 k 3 C H,d,k,M,δ/ 8 H √ λ ) + log 16 δ ] + 2 √ λ √ dH . 7 Here C d and C H,d,k,M,δ are defined in Lemma B.8.

Recall that ∆ k est := ˜ V k 1 ( s k 1 ) -V π k 1 ( s k 1 ) and define ζ k h = E [ ˜ V k h +1 ( s k h +1 ) -V π k h +1 ( s k h +1 ) | s k h , a k h ] -˜ V k h +1 ( s k h +1 ) + V π k h +1 ( s k h +1 ) . Then by applying Lemma B.1 recursively, the total estimation error ∑ K k =1 ∆ k est can be decomposed as:

On one hand, by definition, | ζ k h | ≤ 2 H for all h ∈ [ H ] , k ∈ [ K ] . Therefore, { ζ k h } is a martingale difference sequence (since the computation of ˜ V k h is independent of the new observation at episode k ). By Azuma-Hoeffding's inequality (for t > 0 ),

which implies with probability 1 -δ/ 8 ,

Step 3: bounding the delayed error. By Lemma B.4, with probability 1 -δ/ 8 ,

Here D τ,δ,H,K := 1 + 2 E [ τ ] + 2 √ 2 E [ τ ] log( 24 KH δ ) + 4 3 log( 24 KH δ ) + D τ,K, δ 16 H and D τ,K,δ is defined in Lemma D.6. Consequently,

Note that by Lemma B.8, event E holds with probability 1 -δ/ 4 , and by a union bound with (7) and (8), we have with probability 1 -δ/ 2 ,

Finally, by a union bound over Step1, Step2 and Step3, we obtain with probability 1 -δ ,

where c > 0 is some universal constant and ι is a Polylog term of H,d,K,δ . The last step is due to: by the choice of λ = 1 , σ 2 = 1 , ν = C δ/ 4 and M = log(4 HK/δ ) / log(64 / 63) , we can bound C δ (in Lemma B.10) by C δ ≤ c 0 H √ dMι δ with c 0 a universal constant and ι δ contains only the Polylog terms. This implies ν 2 ≤ c 1 H 2 dMι δ . Note C d ≤ dι δ , therefore β is dominated by the first term β ≤ C 2 √ 2 ν 2 log(16 C d HMK/δ ) ≤ C 3 dHι δ for some universal constants C 2 , C 3 . Since R ( T ) is dominated by the second term in the second to last inequality, plug back the upper bound for β gets the result. Finally, it is readily to verify D τ,δ,H,K is bounded by c ′ E [ τ ] ι + O ( ι ) .

Lemma B.1. Define ζ k h = E [ ˜ V k h +1 ( s k h +1 ) -V π k h +1 ( s k h +1 ) | s k h , a k h ] -˜ V k h +1 ( s k h +1 ) + V π k h +1 ( s k h +1 ) and condition on the event (10) in Lemma B.8. Then for all k ∈ [ K ] , h ∈ [ H ] , the following holds,

Proof of Lemma B.1. Since a k h = π k ( s k h ) , it implies V k h ( x k h ) = ¯ Q k h ( s k h , a k h ) (recall ¯ Q k h := min { ˜ Q k h , H -h +1 } ) and V π k h ( x k h ) = Q π k h ( s k h , a k h ) . Hence,

where the last step is by the event defined in (10).

B.1 Bounding the delayed error term ∑ K k =1 ∥ ∥ ϕ ( s k h , a k h ) ∥ ∥ (Ω k h ) -1 .

Recall the delayed covariance matrix Ω k h = ∑ k -1 τ =1 1 τ,k -1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) T + λI with 1 s,t := 1 [ s + τ s ≤ t ] , then we can define the full design matrix Σ k h and the complement matrix Λ k h as

then Σ k h = Ω k h +Λ k h . Also, denote the number of missing episodes as: U k = ∑ k s =1 1 [ s + τ s > k ] . Then we have the following Lemmas.

Proof of Lemma B.2. Since λ > 0 , both Ω k h and Σ k h are invertible with:

Lemma B.3. Denote ϕ k h := ϕ ( s k h , a k h ) . Let λ > 0 , then

Proof of Lemma B.3. By definition and Trace of matrix, we have

Denote A = (Σ k h ) -1 Λ k h and B = (Ω k h ) -1 ϕ k h ( ϕ k h ) T , then A,B both have non-negative eigenvalues (by Lemma D.14) and this implies

Tr( AB ) = Tr ( AB 1 / 2 B 1 / 2 ) = Tr ( B 1 / 2 AB 1 / 2 ) ≤ Tr ( B 1 / 2 (Tr( A )) IB 1 / 2 ) = Tr( A ) Tr( B ) and this implies

where the last inequality uses Lemma D.15. Next, by changing the order summation, we have

which implies

where the second step uses (Σ k h ) -1 ⪰ (Σ t h ) -1 for k ≥ t .

Lemma B.4 (Bounding the delayed error) . With probability 1 -δ/ 8 ,

Here D τ,δ,H,K := 1 + 2 E [ τ ] + 2 √ 2 E [ τ ] log( 24 KH δ ) + 4 3 log( 24 KH δ ) + D τ,K, δ 16 H and D τ,K,δ is defined in Lemma D.6.

Proof of Lemma B.4. Now Combine Lemma B.2 and Lemma B.3, we obtain

For term ( ∗ ) , since λ = 1 , by Cauchy-Schwartz inequality and Elliptical Potential Lemma D.8,

For term ( ∗∗ ) , by Lemma D.15 and Lemma D.6 and a union bound, with probability 1 -δ/ 8 ,

Denote D τ,δ,K := 1 + E [ τ ] + 2 √ 2 E [ τ ] log( 24 K δ ) + 4 3 log( 24 K δ ) + E [ τ ] + D τ,K, δ 16 , then we have with probability 1 -δ/ 8 ,

then apply a union bound over h ∈ [ H ] to obtained the stated result.

B.2 Proofs of Anti-concentration for Delayed-PSVI

In this section, we prove the optimism via anti-concentration for Delayed-PSVI. We first present two assisting lemmas.

Lemma B.5 (Anti-concentration for Optimism) . Suppose the event holds. Choose ν = C ′ and M = log( HK/δ ) / log(64 / 63) . Then we have with probability 1 -δ ,

Proof of Lemma B.5. For the rest of the proof, we condition on the event

where δ ′ will be specified later and C δ is defined in the Lemma B.10. Also note

Therefore,

where the first event uses the condition on E and the second inequality chooses ν = C δ ′ and uses Lemma D.5. Apply Lemma B.6 with f = r h + P h ˜ V k h +1 , for M δ = log(1 /δ ) / log(64 / 63) ,

By law of total expectation E [ E [ 1 A | X ]] = E [ 1 A ] = P [ A ] , it implies

Apply a union bound for h, k , we have for M δ = log( HK/δ ) / log(64 / 63) , with probability 1 -δ ,

The following lemma is used to prove Lemma B.5.

Proof of Lemma B.6. For any fixed ( k, h ) ∈ [ K ] × [ H ] , we have

then this implies

With the above two lemmas, we are ready to prove the optimism achieved by Delayed-PSVI with respect to ˜ Q k h .

Lemma B.7 (Optimism) . For any 0 ≤ δ < 1 , we set the input in Algorithm 1 as ν = C δ/ 4 and M δ = log(4 HK/δ ) / log(64 / 63) , then with probability 1 -δ/ 2 , we have

Here C δ is defined in Lemma B.10.

Proof of Lemma B.7. Step1: Suppose the event

holds. Choose ν = C δ ′ and M δ = log(4 HK/δ ) / log(64 / 63) . Then we show, for any h ∈ [ H ] , with probability 1 -δ/ 4 , ˜ Q k h ( s, a ) ≥ Q ∗ h ( s, a ) , ˜ V k h ( s ) ≥ V ∗ h ( s ) for all ( s, a ) ∈ S × A , h ∈ [ H ] , k ∈ [ K ] .

First, due to our choice of M δ = log(4 HK/δ ) / log(64 / 63) , by Lemma B.5, with probability 1 -δ/ 4 ,

which we condition on.

Next, we finish the proof by backward induction. Base case: for h = H + 1 , the value functions are zero, and thus ˜ Q k H +1 ≥ Q ∗ H +1 holds trivially, which also implies ˜ V k H +1 ≥ V ∗ H +1 . Suppose the conclusion holds true for h +1 . Then for time step h and any k ∈ [ K ] ,

where the first inequality uses the induction hypothesis and the second inequality uses the condition. Lastly, ˜ V k h ( · ) = max a min { ˜ Q k h ( · , a ) , H -h + 1 } ≤ max a min { Q ∗ h ( · , a ) , H -h + 1 } = max a Q ∗ h ( · , a ) = V ∗ h ( · ) . By induction, this finishes the Step1.

Step2: By Lemma B.10, with probability 1 -δ/ 4 , for all k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A , it holds

Therefore, in Step1, choose δ ′ = δ/ 4 , and a union bound we obtain: for the choice ν = C δ/ 4 and M δ = log(4 HK/δ ) / log(64 / 63) , then with probability 1 -δ/ 2 , we have

B.3 Proofs of Concentration for Delayed-PSVI

Lemma B.8 (Pointwise Concentration) . Algorithm 1 guarantees that with probability 1 -δ , ∀ k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A , , it holds:

Proof of Lemma B.8. Recall that | r k h + P h ˜ V k h +1 | ≤ H -h + 1 , therefore r k h + P h ˜ V k h +1 = min { r k h + P h ˜ V k h +1 , H -h +1 } . This implies | min { ˜ Q k h ( s, a ) , H -h +1 }-r k h -[ P h ˜ V k h +1 ]( s, a ) | = | min { ˜ Q k h ( s, a ) , H -h + 1 } -min { r k h + [ P h ˜ V k h +1 ]( s, a ) , H -h + 1 }| ≤ | ˜ Q k h ( s, a ) -r k h -[ P h ˜ V k h +1 ]( s, a ) | . Hence

The proof then directly follows Lemma B.9 and Lemma B.10 to bound R 1 and R 2 respectively (together with a union bound).

Lemma B.9 (Concentration of R 1 ) . For any 0 < δ < 1 , define the event ˜ E as

then ˜ E happens with probability 1 -δ . Here log C d = d log(1+(8 √ 2 ν 2 log(2 /δ ) /λ +8 H √ d ) K 3 ) .

Proof of Lemma B.9. In the Step1 and Step2, we abuse ˜ w k h to denote ˜ w k,m h for arbitrary m to avoid notation redundancy.

In Step1 : We first show for any k ∈ [ K ] , h ∈ [ H ] , ( s, a ) ∈ S × A , with probability 1 -δ ,

Indeed, by design of Algorithm 1, ˜ w k h ∼ N ( ̂ w k h , ν 2 (Ω k h ) -1 ) , which gives,

Therefore, ϕ ( s, a ) T ( ˜ w k h -̂ w k h ) is ν 2 ϕ ( s, a ) T (Ω k h ) -1 ϕ ( s, a ) -sub-Gaussian. By concentration of subGaussian random variables, we have

Solving for δ gives with probability 1 -δ ,

Step2: For any 0 < δ < 1 , define the event ˜ E as

√

then ˜ E happens with probability 1 -δ . Here log C d = d log(1+(8 √ 2 ν 2 log(2 /δ ) /λ +8 H d ) K 3 ) . In Lemma D.12, set θ = ˜ w k h -̂ w k h and A = (Ω k h ) -1 and B = 1 /λ , and let V be the 1 2 K 3 -epsilon net for the class of values {|⟨ ϕ, ˜ w k h -̂ w k h ⟩|-C √ ϕ ⊤ (Ω k h ) -1 ϕ : ∥ ϕ ∥ ≤ 1 } (where C = √ 2 ν 2 log(2 /δ ) ), then it must also be the 1 2 K 3 -epsilon net for the class of values F = {|⟨ ϕ ( s, a ) , ˜ w k h -̂ w k h ⟩| -C √ ϕ ( s, a ) ⊤ (Ω k h ) -1 ϕ ( s, a ) : ( s, a ) ∈ S × A} , let ¯ V is the smallest subset of V such that it is

1 2 K 3 -epsilon net for the class of values F . Then we can select V S×A to be the set of state-action pairs such that for any f ϕ := |⟨ ϕ, ˜ w k h -̂ w k h ⟩| -C √ ϕ ⊤ (Ω k h ) -1 ϕ ∈ ¯ V , there exists ( s, a ) ∈ V S×A satisfies |⟨ ϕ ( s, a ) , ˜ w k h -̂ w k h ⟩| C √ ϕ ( s, a ) ⊤ (Ω k h ) -1 ϕ ( s, a ) | -f ϕ ≤ 1 / 2 K 3 , then we have V S×A is a 1 /K 3 -epsilon net of F and |V S×A | ≤ | ¯ V| ≤ |V| . Therefore,

Then by a union bound over (1+(8 √ 2 ν 2 log(2 /δ ) /λ +8 H √ d ) K 3 ) d , H and K , we have the stated the result.

Step3: Note ˜ Q k h = max m ϕ T ˜ w k,m h , hence by a union bound over M , we have

for all k, h, s, a with probability 1 -δ . Here the last inequality follows Step2, which completes the proof.

LemmaB.10 (Concentration of R 2 ) . For any 0 < δ < 1 , with probability 1 -δ , for all k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A , it holds

Proof of Lemma B.10. For any ( k, h ) ∈ [ K ] × [ H ] and ( s, a ) ∈ S × A , denote

Recall y τ h = 1 τ,k -1 · [ r τ h ( s τ h , a τ h ) + ˜ V k h +1 ( s τ h +1 )] from Algorithm 1 and denote ¯ y τ h := r τ h ( s τ h , a τ h ) + ˜ V k h +1 ( s τ h +1 ) . Then by definition,

From Ω k h defined in line 7 of Algorithm 1, we have Φ h Φ T h = Ω k h -λI . Plug it into the definition of ̂ w k h , we have

We then proceed to bound ̂ w k h -w k h , which gives

Term (i). Since Ω k h is positive definite, multiplying the first term ( i ) with ϕ ( s, a ) and by CauchySchwartz inequality, we obtain,

Apply Lemma B.11, we have with probability at least 1 -δ , for any ( k, h ) ∈ [ K ] × [ H ] , and ( s, a ) ∈ S × A ,

Term (ii). By Lemma B.12, ∀ ( s, a ) ∈ S ×A , and ( k, h ) ∈ [ K ] × [ H ] , ∣ ∣ ϕ ( s, a ) T ( ii ) ∣ ∣ can be bounded as √ √

Combining (13), (14), we have with probability 1 -δ , for any ( k, h ) ∈ [ K ] × [ H ] and ( s, a ) ∈ S×A ,

This concludes the proof.

Lemma B.11. For any 0 < δ < 1 , with probability 1 -δ , we have ∀ ( k, h ) ∈ [ K ] × [ H ] ,

here C H,d,k,M,δ = 2 H √ dk λ + ν √ 2 d + ν 2 log( M/δ ) √ λ . 8

Proof of Lemma B.11. First note that

Recall that (Ω k h ) 1 / 2 ( ˜ w k,m h -̂ w k h ) /ν ∼ N (0 , I d ) , then by Lemma D.7, with probability 1 -δ/ 2 , we have √

Apply the union bound over all m , then above implies with probability 1 -δ/ 2 , ∀ m ∈ [ M ]

Now consider the function class ¯ V := { max a max m ϕ ( · , a ) T w m : ∥ w m ∥ ≤ C H,d,k,M,δ } , so by Lemma D.13 the ϵ -log covering number for ¯ V is dM log(1 + 2 C H,d,k,M,δ ϵ ) . Since min {· , ·} is a non-expansive operator, the ϵ -log covering number for the function class V := { max a min { max m ϕ ( · , a ) T w m , ( H -h + 1) } : ∥ w m ∥ ≤ C H,d,k,M,δ } , is at most dM log(1 + 2 C H,d,k,M,δ ϵ ) . Hence, for any V ∈ V , there exists V ′ in the ϵ -covering such that V = V ′ +∆ V with ∥ ∆ V ∥ ∞ ≤ ϵ . Then with probability 1 -δ/ 2 ,

where the second inequality can be conducted using a direct calculation and the third inequality uses Lemma D.9 and a union bound over the covering number. Now by (15) and (16) and a union bound, we have for any ϵ > 0 , with probability 1 -δ ,

where the last step choose ϵ 2 = H 2 λ/ 8 k 2 so 8 k 2 ϵ 2 λ ≤ 4 H 2 . Lastly, apply the union bound over H,K to obtain the stated result.

Lemma B.12. ∀ ( s, a ) ∈ S × A , h ∈ [ H ] , k ∈ [ K ] , it holds that

Proof of Lemma B.12. Note that the precision matrix Ω k h for any step h and episode k is always positive definite. By Cauchy-Schwartz inequality and Lemma D.1,

Note that λ min (Ω k h ) ≥ λ . Applying Lemma D.3 for ∥ ∥ w k h ∥ ∥ completes the proof.

C Regret Analysis for Delayed-LPSVI

Proof of Theorem 2. The proof structure is similar to that of Theorem 1. We proceed by bounding ∆ k opt and ∆ k est respectively.

Step 1: bound regret from optimism. By Lemma C.5, with probability 1 -δ/ 2 ,

Step 2: bound regret from estimation error. We first condition on the event that

2 λ dH . Here C d and C H,d,k,M,δ are defined in Lemma C.6.

Similarly, define

By definition, | ζ k h | ≤ 2 H for all h ∈ [ H ] , k ∈ [ K ] , therefore { ζ k h } is a martingale difference sequence. By Azuma-Hoeffding's inequality,

Thus, with probability 1 -δ/ 8 ,

Step 3: bounding the delayed error. By Lemma B.4, with probability 1 -δ/ 8 ,

Here D τ,δ,H,K := 1 + 2 E [ τ ] + 2 √ 2 E [ τ ] log( 24 KH δ ) + 4 3 log( 24 KH δ ) + D τ,K, δ 16 H and D τ,K,δ is defined in Lemma D.6. By Lemma C.6, event E holds with probability 1 -δ/ 4 , by a union bound with (18) and (19), we have with probability 1 -δ/ 2 ,

Finally, by a union bound over Step1, Step2 and Step3, we obtain with probability 1 -δ ,

where c > 0 is some universal constant and ι is a Polylog term of H,d,K,δ . Similarly, we can bound β ≤ CdHι δ for some universal constant C , and it is readily to verify D τ,δ,H,K is bounded by c ′ E [ τ ] ι + O ( ι ) .

Lemma C.1. Define ζ k h = E [ ˜ V k h +1 ( s k h +1 ) -V π k h +1 ( s k h +1 ) | s k h , a k h ] -˜ V k h +1 ( s k h +1 ) + V π k h +1 ( s k h +1 ) and condition on the event (21) in Lemma C.6. Then for all k ∈ [ K ] , h ∈ [ H ] , the following holds,

Proof of Lemma C.1. By the event defined in (21), the proof follows exactly as in that of Lemma B.1.

C.1 Convergence of Langevin Monte Carlo

The following lemma is crucial to prove the optimism and bound the error in Langevin analysis. For ease of notation, within the episode k , we simply use η to denote η k for conciseness.

Lemma C.2 (Convergence of LMC) . Denote { ˜ w k,m h } m ∈ [ M ] to be the weights returned by Line 6 of Algorithm 2. Set η = 1 4 λ max (Ω k h ) , we have

where

Furthermore, we have

where κ h := λ max (Ω k h ) λ min (Ω k h ) is the condition number.

Proof of Lemma C.2. Let b k h := ∑ k -1 τ =1 ϕ ( s τ h , a τ h ) y τ h , then

Therefore, fix h, k, m , and within the Algorithm 3 we have

where the last equality uses (Ω k h ) -1 b k h = ̂ w k h and I ≻ I -2 η Ω k h ≻ 0 , so ∑ N -1 l =0 A l = ( I -A N )( I -A ) -1 . Since ϵ i are i.i.d gaussian noise, from the above we directly have

where

Next, due to the choice of η = 1 4 λ max (Ω k h ) , we have

In addition,

The above two implies

Replacing N with N k and w N with w k,m for all m ∈ [ M ]

˜ h completes the proof.

C.2 Proofs of optimism for Delayed-LPSVI

Lemma C.3 (Anti-concentration for Optimism) . Suppose the event

holds. Choose N k ≥ max { log( 32 H 2 ( K + λ ) dk γλ + 1) / [2 log(1 / (1 -1 2 κ h ))] , log 2 2 log(1 / (1 -1 2 κ h )) } , γ = 16 C 2 δ ′ and M δ = log( HK/δ ) / log(64 / 63) . Then we have with probability 1 -δ ,

Proof of Lemma C.3. For the rest of the proof, we condition on the event

where δ ′ will be specified later and C δ is defined in the Lemma C.9.

Also note

Therefore,

where the first two inequalities follow Lemma C.2 and Lemma C.4 respectively and the thrid inequality results from Lemma D.5. Applying Lemma B.6 with f = r h + P h ˜ V k h +1 and without conditioning, for M δ = log(1 /δ ) / log(64 / 63) ,

Apply a union bound for h, k , we have for M δ = log( HK/δ ) / log(64 / 63) , with probability 1 -δ ,

holds. Choose N k ≥ max { log( 32 H 2 ( K + λ ) dk γλ + 1) / [2 log(1 / (1 -1 2 κ h ))] , log 2 2 log(1 / (1 -1 2 κ h )) } and γ = 16 C 2 δ ′ . Then

Proof of Lemma C.4. By direct calculation,

For the first term above, by CS inequality we have

and this indicates

where the last inequality is by N k ≥ log( 32 H 2 ( K + λ ) dk γλ +1) / [2 log(1 / (1 -1 2 κ h ))] . For the second term above,

Here the second inequality uses N k ≥ log 2 2 log(1 / (1 -1 2 κ h )) and the last equal sign comes from γ = 16 C 2 δ ′ .

Lemma C.5 (Optimism for Langevin Posterior Sampling) . For any 0 ≤ δ < 1 , we set the input in Algorithm 2 as N k ≥ max { log( 32 H 2 ( K + λ ) dk γλ +1) / [2 log(1 / (1 -1 2 κ h ))] , log 2 2 log(1 / (1 -1 2 κ h )) } , γ = 16 C 2 δ/ 4 and M δ = log(4 HK/δ ) / log(64 / 63) , then with probability 1 -δ/ 2 , we have

Here C δ is defined in Lemma C.9.

Proof of Lemma C.5. Step1: Suppose the event

holds. Choose N k ≥ max { log( 32 H 2 ( K + λ ) dk γλ + 1) / [2 log(1 / (1 -1 2 κ h ))] , log 2 2 log(1 / (1 -1 2 κ h )) } , γ = 16 C 2 δ ′ and M δ = log(4 HK/δ ) / log(64 / 63) . Then we show, for any h ∈ [ H ] , with probability 1 -δ/ 4 , ˜ Q k h ( s, a ) ≥ Q ∗ h ( s, a ) , ˜ V k h ( s ) ≥ V ∗ h ( s ) for all ( s, a ) ∈ S × A , h ∈ [ H ] , k ∈ [ K ] .

First, due to our choice of M δ = log(4 HK/δ ) / log(64 / 63) , by Lemma C.3, with probability 1 -δ/ 4 ,

which we condition on.

Next, we finish the proof by backward induction. Base case: for h = H + 1 , the value functions are zero, and thus ˜ Q k H +1 ≥ Q ∗ H +1 holds trivially, which also implies ˜ V k H +1 ≥ V ∗ H +1 . Suppose the conclusion holds true for h +1 . Then for time step h and any k ∈ [ K ] ,

where the first inequality uses the induction hypothesis and the second inequality uses the condition. Lastly, ˜ V k h ( · ) = max a min { ˜ Q k h ( · , a ) , H -h + 1 } ≤ max a min { Q ∗ h ( · , a ) , H -h + 1 } = max a Q ∗ h ( · , a ) = V ∗ h ( · ) . By induction, this finishes the Step1.

Step2: By Lemma C.9, with probability 1 -δ/ 4 , for all k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A , it holds

Therefore, in Step1, choose δ ′ = δ/ 4 , and a union bound we obtain: for the choice N k ≥ max { log( 32 H 2 ( K + λ ) dk γλ + 1) / [2 log(1 / (1 -1 2 κ h ))] , log 2 2 log(1 / (1 -1 2 κ h )) } , γ = 16 C 2 δ/ 4 and M δ = log(4 HK/δ ) / log(64 / 63) , then with probability 1 -δ/ 2 , we have

C.3 Proofs of Concentration for Delayed-LPSVI

Lemma C.6 (Pointwise Concentration for Langevin Posterior Sampling) . Choose N k ≥ log( 4 HK 3 √ λ/dK ) / log(1 / (1 -1 2 κ h )) . Algorithm 2 guarantees that ∀ k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A , the following holds with probability 1 -δ ,

Proof of Lemma C.6. Recall that | r k h + P h ˜ V k h +1 | ≤ H -h + 1 , therefore r k h + P h ˜ V k h +1 = min { r k h + P h ˜ V k h +1 , H -h +1 } . This implies | min { ˜ Q k h ( s, a ) , H -h +1 }-r k h -[ P h ˜ V k h +1 ]( s, a ) | = | min { ˜ Q k h ( s, a ) , H -h + 1 } -min { r k h + [ P h ˜ V k h +1 ]( s, a ) , H -h + 1 }| ≤ | ˜ Q k h ( s, a ) -r k h -[ P h ˜ V k h +1 ]( s, a ) | . Hence

The proof then directly follows Lemma C.7 and Lemma C.9 to bound R 1 and R 2 respectively (together with a union bound).

Lemma C.7 (Concentration of R 1 with Langevin Posterior Sampling) . Suppose N k ≥ log( 4 HK 3 √ λ/dK ) / log(1 / (1 -1 2 κ h )) . For any 0 < δ < 1 , define the event ˜ E as

then ˜ E happens w.p. 1 -δ . Here log C d = d log(1 + (16 √ 2 γ log(2 /δ ) /λ +16 H √ d ) K 3 ) .

Proof of Lemma C.7. In the Step1 and Step2, we abuse ˜ w k h to denote ˜ w k,m h for arbitrary m to avoid notation redundancy.

In Step1 : We first show for any k ∈ [ K ] , h ∈ [ H ] , ( s, a ) ∈ S × A , with probability 1 -δ ,

Indeed, by Lemma C.2 we have ( ˜ w k h -( I -A N k h,k ) ̂ w k h ) ∼ N (0 , Θ k h ) , which gives,

Therefore, ϕ ( s, a ) T ( ˜ w k h -( I -A N k h,k ) ̂ w k h ) is ϕ ( s, a ) T Θ k h ϕ ( s, a ) -sub-Gaussian. By concentration of sub-Gaussian random variables, we have

Solving for δ gives with probability 1 -δ ,

where the last inequality is by Lemma C.2, and by Lemma C.8, the above further implies

Step2: we prove that for any 0 < δ < 1 , define the event ˜ E as

√

then ˜ E happens w.p. 1 -δ . Here log C d = d log(1 + (16 √ 2 γ log(2 /δ ) /λ +16 H d ) K 3 ) . In Lemma D.12, set θ = ˜ w k h -̂ w k h and A = (Ω k h ) -1 and B = 1 /λ , and let V be the 1 4 K 3 -epsilon net for the class of values {|⟨ ϕ, ˜ w k h -̂ w k h ⟩| -C √ ϕ ⊤ (Ω k h ) -1 ϕ -1 2 K 3 : ∥ ϕ ∥ ≤ 1 } (where C = √ 2 γ log(2 /δ ) ), then it must also be the 1 4 K 3 -epsilon net for the class of values F = {|⟨ ϕ ( s, a ) , ˜ w k h -̂ w k h ⟩| -C √ ϕ ( s, a ) ⊤ (Ω k h ) -1 ϕ ( s, a ) -1 2 K 3 : ( s, a ) ∈ S × A} , let ¯ V is the smallest subset of V such that it is 1 2 K 3 -epsilon net for the class of values F . Then we can select V S×A to be the set of state-action pairs such that for any f ϕ := |⟨ ϕ, ˜ w k h -̂ w k h ⟩| -C √ ϕ ⊤ (Ω k h ) -1 ϕ ∈ ¯ V 1 2 K 3 , there exists ( s, a ) ∈ V S×A satisfies ∣ ∣ ∣ ∣ |⟨ ϕ ( s, a ) , ˜ w k h -̂ w k h ⟩| -C √ ϕ ( s, a ) ⊤ (Ω k h ) -1 ϕ ( s, a ) | -1 2 K 3 -f ϕ ∣ ∣ ∣ ∣ ≤ 1 / 4 K 3 , then we have V S×A is a 1 / (2 K 3 ) -epsilon net of F and |V S×A | ≤ | ¯ V| ≤ |V| . Therefore,

where the last inequality is from Step1. Then by a union bound over H , K and (1 + (16 √ 2 γ log(2 /δ ) /λ +16 H √ d ) K 3 ) d , we have with probability 1 -δ ,

where log C d = d log(1 + (16 √ 2 γ log(2 /δ ) /λ +16 H √ d ) K 3 ) .

Step3: We finish the proof. Note ˜ Q k h = max m ϕ T ˜ w k,m h , hence by a union bound over M , we have

for all k, h, s, a with probability 1 -δ . Here the last inequality uses Step2. This finishes the proof.

Proof of Lemma C.8. By direct calculation,

where the third inequality is by Lemma D.4 and the fourth inequality is by (20). The last inequality is by the choice of N k .

Lemma C.9 (Concentration of R 2 with Langevin Posterior Sampling) . For any 0 < δ < 1 , with probability 1 -δ , for all k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A , it holds

Proof of Lemma C.9. For any ( k, h ) ∈ [ K ] × [ H ] and ( s, a ) ∈ S × A , denote

Recall y τ h = 1 τ,k -1 · [ r τ h ( s τ h , a τ h ) + ˜ V k h +1 ( s τ h +1 )] from Algorithm 1 and denote ¯ y τ h := r τ h ( s τ h , a τ h ) + ˜ V k h +1 ( s τ h +1 ) . Then by definition,

By definition of Ω k h , we have Φ h Φ T h = Ω k h -λI . Plug it into the definition of ̂ w k h , we have

We then proceed to bound ̂ w k h -w k h , which gives

Term (i). Since Ω k h is positive definite, multiplying the first term ( i ) with ϕ ( s, a ) and by CauchySchwartz inequality, we obtain,

Apply Lemma C.10, we have with probability at least 1 -δ , for any ( k, h ) ∈ [ K ] × [ H ] , and ( s, a ) ∈ S × A ,

Term (ii). By Lemma B.12, ∀ ( s, a ) ∈ S ×A , and ( k, h ) ∈ [ K ] × [ H ] , ∣ ∣ ϕ ( s, a ) T ( ii ) ∣ ∣ can be bounded as √ √

Combining (24), (25), we have with probability 1 -δ , for any ( k, h ) ∈ [ K ] × [ H ] and ( s, a ) ∈ S×A ,

This concludes the proof.

Lemma C.10. For any 0 < δ < 1 , with probability 1 -δ , we have ∀ ( k, h ) ∈ [ K ] × [ H ] ,

here C H,d,k,M,δ λ λ

Proof of Lemma C.10. First note that

Choosing w 0 = 0 , then by Lemma C.2 and (Θ k h ) -1 / 2 ( ˜ w k,m h -( I -A N k h,k ) ̂ w k h ) ∼ N (0 , I d ) , and by Lemma D.7, with probability 1 -δ/ 2 , we have

9 We will choose γ to be Poly( H,d,K ) and this will not affect the overall dependence of the guarantee since C H,d,k,M,δ is inside the log term.

where the first inequality uses Lemma C.2 again. Apply the union bound over all m , then above implies with probability 1 -δ/ 2 , ∀ m ∈ [ M ]

(where we used ∥ ∥ ∥ ( I -A N k h,k ) ̂ w k h ∥ ∥ ∥ ≤ ∥ ∥ ∥ ( I -A N k h,k ) ∥ ∥ ∥ ∥ ∥ ̂ w k h ∥ ∥ ≤ ∥ ∥ ̂ w k h ∥ ∥ ). Now consider the function class ¯ V := { max a max m ϕ ( · , a ) T w m : ∥ w m ∥ ≤ C H,d,k,M,δ } , so by Lemma D.13 the ϵ -log covering number for ¯ V is dM log(1 + 2 C H,d,k,M,δ ϵ ) . Since min {· , ·} is a non-expansive operator, the ϵ -log covering number for the function class V := { max a min { max m ϕ ( · , a ) T w m , ( H -h +1) } : ∥ w m ∥ ≤ C H,d,k,M,δ } , is at most dM log(1 + 2 C H,d,k,M,δ ϵ ) . Hence, for any V ∈ V , there exists V ′ in the ϵ -covering such that V = V ′ +∆ V with ∥ ∆ V ∥ ∞ ≤ ϵ . Then with probability 1 -δ/ 2 ,

where the second inequality can be conducted using a direct calculation and the third inequality uses Lemma D.9 and a union bound over the covering number. Now by (26) and (27) and a union bound, we have for any ϵ > 0 , with probability 1 -δ ,

where the last step choose ϵ 2 = H 2 λ/ 8 k 2 so 8 k 2 ϵ 2 λ ≤ 4 H 2 . Lastly, apply the union bound over H,K to obtain the stated result.

D Auxiliary lemmas

D.1 Useful Norm Inequalities

Lemma D.1. Suppose v ∈ R d , and A is some positive definite matrix whose eigenvalues satisfy λ max ( A ) ≥ · · · ≥ λ min ( A ) > 0 . It can be shown that

Proof of Lemma D.1. Consider the eigenvalue decomposition of A , which gives A = U Λ U T , where Λ = diag ( λ max ( A ) , . . . , λ min ( A )) . Then

Similar argument shows ∥ v ∥ A ≥ √ λ min ( A ) .

Lemma D.2 (Lemma D.1 of [35]) . Let Ω k h be the precision matrix of the posterior distribution of w k h at step h of episode k , where Ω k h := σ -2 Φ h Φ T h +Σ -1 with Σ -1 = λI d and σ 2 = 1 . Then

Lemma D.3 (Bound on Weights of Q-function) . Suppose the linear MDP assumption and at each step h ∈ [ H ] , rewards r h are bounded between [0 , 1] , then the norm of the true parameter w π h under fixed policy π satisfies

In addition, for any ( s, a ) ∈ S × A , let ϕ ( s, a ) T w k h := ( r h + P h ˜ V k h +1 )( s, a ) , we also have

Proof of Lemma D.3. By definition in Lemma A.1, the true parameter w h at time step h is

With bounded rewards r h ∈ [0 , 1] , we have V π h +1 ( s ) ≤ H, ∀ s ∈ S . Since ∥ θ h ∥ ≤ √ d , and ∥ ∥ E µ h [ V π h +1 ( s ′ )] ∥ ∥ ≤ ∥ ∥ ∫ S H d µ h ( s ′ ) ∥ ∥ ≤ H √ d .

Similarly, by definition of the constructed weights w k h ,

From Line 15 of Algorithm 1, for any h ∈ [ H ] and s ∈ S , ˜ V k h ( s ) = max a min { ˜ Q k h ( · , a ) , H -h + 1 } ≤ H . Applying triangle inequality, we have

Lemma D.4 (Bound on Estimated Weights of Algorithm 1) . For any step h ∈ [ H ] and episode k ∈ [ K ] , the weight ̂ w k h output by Algorithm 1 satisfies,

Proof of Lemma D.4. For any vector v ∈ R d , it holds

where the last step is by Lemma D.2. The above directly imply the stated result by the definition of l 2 norm.

D.2 Concentration Inequalities

Lemma D.5 ([3]) . Suppose Z is a random variable following a Gaussian distribution N ( µ, σ 2 ) , where σ > 0 . The following concentration and anti-concentration inequalities hold for any z ≥ 1 :

And for 0 ≤ z ≤ 1 , we have,

∞

Lemma D.6 (Sub-exponential tail bound) . Suppose { τ k } k =1 are ( v, b ) -sub-exponential random variables. denote D τ,K,δ := min {√ 2 v 2 log ( 3 K 2 δ ) , 2 b log ( 3 K 2 δ ) } . Then with probability 1 -δ ,

Lemma D.7 (Multivariate Gaussian Concentration) . Suppose X ∼ N (0 , I d ) . Then with probability 1 -δ , √

Proof. Apply Proposition 1 of [30], choose A = I , then Σ = I and Tr (Σ) = d , ∥ Σ ∥ = 1

which implies with probability 1 -δ , ∥ X ∥ ≤ 2 d + √ 2 log(1 /δ )

.

Lemma D.8 (Elliptical Potential Lemma [1]) . Suppose { ϕ t } ∞ t =1 is an R d -valued sequence, Ω 0 ∈ R d × d is positive definite, and Ω t = Ω 0 + ∑ t -1 τ =1 ϕ τ ϕ T τ . If λ min (Ω 0 ) ≥ 1 , and ∥ ϕ τ ∥ 2 ≤ 1 for all τ ∈ Z + , then for any t ∈ Z + ,

Lemma D.9 (Self-normalized process [1]) . Let {F t } ∞ t =0 be a filtration, and { η t } ∞ t =1 be a real-valued stochastic process such that η t is F t -measurable and η t |F t -1 is zero-mean (i.e. E [ η t |F t -1 ] = 0 ). Assume that conditioning on F t , η t is C -sub-Gaussian. Let { ϕ t } ∞ t =1 be an R d real-valued stochastic process such that ϕ t is F t -measurable. Let Ω 0 ∈ R d × d be a positive definite matrix and Ω t = Ω 0 + σ -2 ∑ t τ =1 ϕ τ ϕ T τ . Then for δ > 0 , with probability at least 1 -δ , for all t ≥ 0 ,

Lemma D.10. Suppose Ω 0 := λI d is a positive definite matrix in R d × d and Ω t = Ω 0 + σ -2 ∑ t -1 τ =1 ϕ τ ϕ T τ .

Proof of Lemma D.10. By definition, det(Ω 1 ) = det( λI ) = λ d . For any τ ∈ Z + and ϕ τ ∈ R d , notice that ϕ τ ϕ T τ is a rank-1 matrix with eigenvalues ∥ ϕ τ ∥ and 0 . By Definition 1 and triangle inequality,

Consider the eigenvalue decomposition for ∑ t -1 τ =1 ϕ τ ϕ T τ :

which suggests

D.3 Covering Argument

Lemma D.11 (Covering number of Euclidean Ball) . Consider an Euclidean ball B R equipped with the Euclidean metric, whose radius is R > 0 . The ϵ -covering number of B R satisfies,

Lemma D.12. Define V to be a class of values with the parametric form

where the feature space is { ϕ : ∥ ϕ ∥ 2 ≤ 1 } and ∥ A ∥ 2 ≤ B , ∥ θ ∥ ≤ 2 H √ d . Let N V ϵ be the covering number of ϵ -net with respect to the absolute value distance, then we have

Let C ϕ be the ϵ 2 C √ B +2 H √ d -net of space { ϕ : ∥ ϕ ∥ 2 ≤ 1 } , then by Lemma D.11,

Therefore, the covering number of space V satisfies

Lemma D.13. Let V denote the function class from S to R

let N ϵ be the ϵ -covering number of V with respect to the distance dist ( V, V ′ ) = sup s | V ( s ) -V ′ ( s ) | . Then

Proof. Let V 1 = max a max m ϕ ( · , a ) T w m 1 and V 2 = max a max m ϕ ( · , a ) T w m 2 . Then

For any m ∈ [ M ] , let C m be the ϵ -net for { w m : ∥ w m ∥ ≤ C H,d,k,M,δ } , then by Lemma D.11, |N ϵ m | ≤ (1 + 2 C H,d,k,M,δ ϵ ) d , implies the total log covering number

D.4 Delayed Feedback

Lemma D.14 (Lemma 9 of [28]) . Let A,B ∈ R d × d be two symmetric positive semi-definite matrices. Then, A 1 2 BA 1 2 and AB share the same set of eigenvalues. Further, these eigenvalues are all non-negative.

Lemma D.15. Let Σ k h , Ω k h , Λ k h be the full design, delayed, and complement matrix respectively. Then (1 + U k λ )(Σ k h ) -1 ⪰ (Ω k h ) -1 . In addition, with probability 1 -δ ,

Proof. The proof follows from Lemma 11 of [28] with U k λ (Σ k h ) -1 ⪰ (Σ k h ) -1 Λ k h (Ω k h ) -1 , and then apply Lemma B.2 that (Ω k h ) -1 = (Σ k h ) -1 +(Σ k h ) -1 Λ k h (Ω k h ) -1 . The second part comes from Lemma 4 of [28].

E Experimental Details

In this section, we provide the experimental details of both simulated environments (synthetic linear MDP and RiverSwim) and discuss their results respectively.

E.1 Delayed-UCBVI

As shown in Table 1 and Section 2, there is no prior UCB method that concerns exactly the same delayed linear MDP setting without resorting to specific policy-switching schemes. To benchmark our posterior sampling algorithms, we modify the existing LSVI-UCB method to accommodate the delayed feedback, which is referred to as the Delayed-UCBVI. Below we include the algorithm of delayed-UCBVI for completeness.

E.2 Synthetic Linear MDP Environment

In this section, we describe the further details in Section 5.1.

Environment Details. Following [44, 46, 70], we construct a set of synthetic linear MDP environments with |S| = 2 , feature dimension d = 10 , planning horizon H = 20 , and varying action space |A| ∈ { 20 , 50 , 100 } . Each action a ∈ A ⊆ { 0 , 1 } d is encoded with its 8 -bit binary representation and represented by a vector b a ∈ R 8 . The feature map ϕ ( · , · ) can then be defined as

where

In addition, let θ h that induces the reward functions r be

with the choice of r = 0 . 99 , and further define the measures µ h that govern the transition dynamics P as

where { α h } h ∈ [ H ] ∈ { 0 , 1 } H is a sequence of integers taking values 0 or 1 , ⊕ is the XOR operator. By design, the set of environments with identical d and H has the same optimal value V ∗ 1 ( s 1 ) .

Further Results and Discussions. Figure 2 depicts the empirical distributions of delays considered in section 5.1. Additionally, the average return achieved by each method upon convergence is reported in Table 2, corresponding to the results shown in Figure 1. Our empirical findings indicate that posterior sampling methods excel UCB-based methods in terms of both statistical accuracy and computational efficiency. More specifically, under different types of delays, both Delayed-PSVI and Delayed-LPSVI achieve higher return (lower regret) and exhibit faster convergence compared to Delayed-UCBVI.

While delays following multinomial distribution and Poisson distributions decay exponentially fast, Pareto delays are heavy-tailed. When computational budget is limited or when episodes are finite, feedback is only partially observable under long-tailed delays and is not guaranteed to be revealed to the agent. This setup captures the practical scenarios when small time windows are considered for decision-making or in online recommender systems, where only positive feedback (e.g. click, make a purchase) are often observed. As shown in Table 2, performance of Delayed-UCBVI can dramatically deteriorate in the presence of long-tailed delays.

Furthermore, our results presented in Table 4 and Table 3 illustrate the consistent behavior of posterior sampling in environments with delayed feedback, considering both statistical and computational aspects. When employing feature mapping, performance of the algorithms is much less dependent on the sizes of state and action space in contrast to tabular settings. It is worth noting that in large state and action space, the neighborhoods of a substantial number of state-action pairs may remain unvisited, leading to increased uncertainty in estimation. In such cases, adjusting the scale of exploration by decreasing the noise scaling factor σ for Delayed-PSVI can yield faster convergence. Finally, as shown in Table 3, Delayed-LPSVI achieves appealing performance as Delayed-PSVI while reducing computation through the use of approximate sampling with Langevin dynamics.

Table 2: Average return achieved by Delayed-PSVI, Delayed-LPSVI and Delayed-UCBVI upon convergence under different delays. Environment setup: |S| = 2 , |A| = 20 , d = 10 , H = 20 . Optimal average return is V ∗ 1 ( s 1 ) = 11 . 96 . Results are obtained over 10 experiments.

0.14

Multi-nomial delays

0.07

Poisson delays

0.05

×10

Pareto delays

0.12

0.06

nsity

Dens

0.10

Density

0.05

0.04

bilit

b

P

0.08

0.06

0.04

ability

geqo

P

0.04

0.02

0.03

D

b

b

0.03

0.02

0.01

0.02

0.01

0.00

0.00

0.00

5

10

15

20

25

30

35

30

40

50

60

70

80

0

25

50

75

100

125

150

Delays

Delays

Delays

×10?

E.3 RiverSwim

RiverSwim environment is known to be a difficult exploration problem for least-squares value iteration with ϵ -greedy exploration due to the sparse reward setting. It models an agent swimming in the river who can either swim towards the right (against the current) or towards the left (with the current). While trying to move rightwards may fail with some probability, moving leftwards always yield successful transition. We consider the environment with linear feature maps where |S| = 5 , d = 10 , H = 20 , and Poisson delays. Accordingly, the tabular environment can be recovered with canonical basis in R d as its feature mapping:

Define θ h as then reward functions induced by θ h are given by:

In this environment, We warm start LMC for Delayed-LPSVI by reusing the previous sample for initialization, and set M = 2 , N = 40 , η = c η /λ max (Ω k h ) , γ = c 2 γ dMH 2 . We set parameters M = 2 , ν = 1 . 0 for Delayed-PSVI, and the bonus coefficient in Delayed-UCBVI as β k h = c β / 2 ·

d √ k ( H -h ) . Optimal hyperparameters are determined by gridsearch and we fix c β = 0 . 04 , c η = 0 . 5 , c γ = 0 . 005 , σ = 1 . 13 . Experiments are repeated with 5 different random seeds. Cumulative regrets are then depicted in Figure 3.

Results and Discussions. Compared to the previous synthetic environment where dense rewards are available, posterior sampling methods are shown to be robust with spare rewards even in the presence of delays. Figure 3 shows that both Delayed-PSVI and Delayed-LPSVI outperform Delayed-UCBVI in delayed-feedback settings with linear function approximation. In particular, LMC (Algorithm 3) provides strong concentration such that Delayed-LPSVI is able to maintain the order-optimal regret as Delayed-PSVI when exploring the value-function space.

250

×102

D-PSVI

D-LPSVI

200

D-UCBVI

gret

6

150

ative

100

50

0

0

10

20

30

40

50

60

Episode

×102

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and analyzes two \"posterior sampling\" algorithms for learning optimal policies in linear MDPs in a setting where feedback from the environment is delayed (by stochastic amounts of time). The algorithms, called Delayed-PSVI and Delayed-LPSVI (namely Delayed (Langevin) Posterior Sampling Value Iteration) explore by randomly sampling value functions from some notion of a posterior, which concentrates on the optimal value function. The Delayed-LPSVI algorithm circumvents some computational burdens in the inference and sampling procedure of Delayed-PSVI, and instead performs the \"posterior sampling\" with noisy gradient descent updates (akin to SGLD). It is established that the Delayed-[L]PSVI algorithms achieve sublinear regret with respect to the time horizon, which grows linearly with the mean feedback delay. Comparing to a simple modification to an existing optimism-based algorithm for linear MDPs to handle the delayed feedback, the authors demonstrate empirically in some simple environments that their posterior sampling algorithms tend to earn high return more quickly (and it is implied therefore that regret grows more slowly).\n\nSTRENGTHS:\nDelayed feedback is an interesting, relevant, and under-explored topic in RL research. This paper provides thorough analysis for posterior sampling algorithms with delayed feedback, with detailed analysis and with empirical evaluation. The algorithms appear to be reasonable and simple to implement.\n\nWEAKNESSES:\nFirstly, while I believe the problem of delayed feedback is an important one to study, I do not particularly understand the motivation and the logistics of the particular setting of delayed feedback that is studied in the paper. In this work, to my understanding, all data from an entire trajectory is kept hidden from the agent until a random future time. As I elaborate on further in the Questions section, I do not understand how hiding this information even allows any feedback control, and I cannot think of a real scenario that this delay models. There is very little discussion about the motivation for this setting.\n\nMoreover, it is not entirely clear to me how the delayed feedback makes the analysis of Delayed-[L]PSVI substantially different from the algorithms they're based on. Suppose that instead of the proposed algorithm, we employ [L]PSVI but only update the value function when new data becomes available. Intuitively, I would expect the regret to grow by a factor of $\\mathbb{E}[\\tau]$ roughly. It would be nice to have an intuitive understanding of how Delayed-[L]PSVI circumvents this multiplicative increase in regret.\n\nFurthermore, perhaps this is due to the fact that the paper proposes a new form of delayed feedback, but the baseline algorithm (to my understanding) is not actually rigorously justified. Thus, it is not clear that it is in fact the posterior sampling that explains the superior performance of Delayed-[L]PSVI.\n\nQUESTIONS:\nI am confused about how exactly the delayed feedback works. For example, in Algorithms 1, 2, 4, during the data collection phase, there is a comment saying \"Environment rollouts cannot be observed if there is a delay\". What does this mean? If the rollouts are not observed, which states are being fed to the policy to generate rollouts (i.e., how is $a_h^k\\sim\\pi_h^k(s_h^k)$ computed if $s_h^k$ is not observed)?\n\nThe only way I can see this working is if the agent can observe its trajectory online, but then cannot use the data it collects for updates until a random time in the future. But what is the motivation for such a model?\n\nLIMITATIONS:\nAs mentioned above, it is unclear what types of real problems are modeled by the particular delayed-feedback setting that is proposed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the challenges of delayed feedback in real-world systems for reinforcement learning and proposes an optimistic algorithm that utilizes noise perturbation via posterior sampling to effectively explore. The algorithm is shown to achieve a worse-case regret that depends on the delay and time horizon, and the paper provides theoretical analysis and empirical results to support the effectiveness of the proposed algorithm.\n\nSTRENGTHS:\n1. The paper addresses a practical problem in reinforcement learning which is the challenge of delayed feedback in real-world systems.\n2. The paper provides both theoretical analysis and empirical results to support the effectiveness of the algorithm.\n\nWEAKNESSES:\n1. Algorithm-wise, it is not clear what is the core idea to handle the delayed feedback problem, i.e., the key difference between delayed PSVI and normal PSVI (normal PSVI could be adjusted to update only when the feedback is received to handle the delayed feedback case) (See Q1).\n2. Theory-wise, it is not clear what is the improvement from previous work also designed to handle delayed feedback (see Q2).\n3. Experimental-wise, it is good to see a comparison with UCB, but LSVI-UCB is not designed to handle delayed feedback. So it might not be a fair comparison (see Q2).\n\nQUESTIONS:\n1. The proposed methods (encode value iteration, multi-round sampling) do not seem to be specifically designed for the delayed feedback case, could you explain why these techniques could improve the results with delayed feedback? Also, a straightforward approach could be just to apply PSRL with value iteration and only update the model when the feedback is received. How is that different from the current approach? If there are key differences, it would be a natural baseline missing from the current experimental section.\n2. In Table 1, there is no comparison with other algorithms that handle the delayed feedback problem. Is it possible to compare with other SOTA methods that also address this problem? Also, the same question applies to the experimental section.\n\nLIMITATIONS:\nThe paper could benefit from a comprehensive discussion about its limitations like the linear assumption, synthetic experiments, etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the frequentist regret of TS under stochastic delayed feedback with linear function approximation. They further proposed using LMC as a posterior sampling strategy to improve computational efficiency.\n\nSTRENGTHS:\n1. As far as I know, this is little work that develops frequentist regret bound with delayed feedback\n2. The paper provides a theoretical understanding of using LMC for posterior sampling\n\nWEAKNESSES:\n1. Overall, the theories look sound to me (I read Appendix B closely and skimmed Appendix C as it is very similar to Appendix B). However, the math is quite messy (such as missing/inconsistent notations, missing assumptions, incorrect references, incorrect constants, etc), which makes the proof hard to follow. Some examples (not exhaustive) are listed below but the manuscript should be proofread carefully\n    1. Definition 1 should include $r_h\\in[0,1]$ (given the value function is upper bounded by H-h+1)\n    2.  $\\tilde{V}$,  $\\tilde{Q}$ should be introduced or referred to algorithm 1 before using. in line 517, $\\tilde{V_h}^k$ is defined different from line 13 in algorithm 1.\n    3. In algorithm 1 line 7, $\\hat{w}$ should not be divided by $\\sigma^{-2}$\n    4. In theorem 2, $\\kappa_h$ is defined in appendix instead of the main manuscript. $\\eta_k$ is not used in theorem 2\n    5. In appendix section B, all Lemma C.5 should be B.8\n    6. Every lemma/theorem should use the phrase \"with probability **at least**\"\n    7. In Appendix line 648, please show that $\\theta=\\tilde{w}-\\hat{w}<=2H\\sqrt{d}$ as stated in the assumption of Lemma D.12  under event $\\tilde{E}$\n    8. In Lemma D.7, $d+2\\sqrt{dt}+2t \\neq 2(\\sqrt{d}+\\sqrt{t})^2$. The definition of constant $C_{H,d,k,M,\\delta}$ is wrong.\n2. The computational efficiency/overall performance of D-LPSVI should be demonstrated in high-dmensional domain\n3. Missing reference\n    - LMC is studied in general bandit setting (e.g. Mazumdar, Eric, et al. On Thompson Sampling with Langevin Algorithms)\n    - In the discussion of arbitrary delayed feedback, please include Wu, Han, and Stefan Wager. Thompson sampling with unrestricted delays.\n\nQUESTIONS:\n1. In Figure 1, why are there non-smooth jumps for average return of D-UCBVI\n2. Please provide some intuition on the bound of $N_k$ (how the condition number scales w.r.t K,H)\n3. To reduce computational complexity, one can also consider updating the parameters less frequently (the feedbacks come in with batches of episodes). I'm curious how this will scale the regret bound\n\nLIMITATIONS:\nLimitations are addressed as future work in Section 6", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of online reinforcement learning with delayed feedback. To address this problem, the authors proposed two Thompson sampling-based algorithms, Delayed Posterior Sampling Value Iteration (Delayed-PSVI) and Delayed-Langevin Posterior Sampling Value Iteration (Delayed-LPSVI). Delayed-PSVI adopts a Laplace approximation around the mode and thus it is computationally expensive due to the need of computing the covariance matrix. On the other hand, Delayed-LPSVI adopts the Langvein MCMC and thus is more tractable in computation when the dimension $d$ is large. \n\nTheoretically, the authors proved the worst-case regret bound for the algorithms, which attains a regret bound of $O(\\sqrt{d^3H^3T} + d^2H^2 E[\\tau])$ for linear MDP, and $O(\\sqrt{d^3 T})$ for linear bandits. A lower bound is also proved for linear bandits. Empirically, the authors conducted a comparison between Delayed-PSVI, Delayed-LPSVI, and a UCB variant D-UCBVI on a simulated MDP environment, demonstrating the advantages of Delayed-PSVI.\n\nSTRENGTHS:\n- The problem setup is useful for modeling many practical scenarios, where the conventional Thompson sampling framework cannot handle well. \n\n- A lower bound is proved for linear bandits with delayed feedback, showing that the first term in the worst-case regret bound is tight.\n\n- The authors also provided empirical evidence for demonstrating the effectiveness of their algorithm, though the setup is synthetic.\n\nWEAKNESSES:\n- The problem is not new, and it has been studied in prior bandit literature, for example, Pike-Burke et al. (2018). The technical contribution might be limited, given the results from bandits and Thompson sampling under the standard setup. It would be better if the authors could emphasize more on the key technical challenges and contrast them with the bandit literature.     \n\n- The assumption on the noise (sub-exponential delay) is not well-motivated and is overly strong. From a practical perspective, the distribution of delay should have a long tail, and it is possible that some users may not respond at all.     \n\n- Also, the delays should not be iid across different users or different groups of users in online recommendation. Thus the iid assumption is too strong.  \n\n-----\n\nThanks the authors for addressing my concerns, I will increase the score to 5.\n\nQUESTIONS:\nSee weakness and strengths.\n\nLIMITATIONS:\nSee weakness and strengths.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical study of a posterior sampling algorithm for linear MDPs with _delayed rewards_. The authors propose two algorithms, Delayed-PSVI and Delayed-LPSVI, which are based on sampling from a Gaussian posterior distribution over weights induced by a Bellman style likelihood function. The LPSVI variant is similar to PSVI, except the posterior sampling is done using Langevin Monte Carlo instead of sampling from the potentially high-dimensional Gaussian directly (motivation is reduced computational complexity). The authors provide worst-case regret bounds for both algorithms, with optimal dependence on on dimension $d$ and time horizon $T$, and slightly suboptimal dependence on the episode length $H$. The theoretical study is complemented by a toy experiment in which the proposed algorithms perform better than a UCB based baseline.\n\nSTRENGTHS:\n* To my knowledge, this paper provides the first worst-case regret bounds for Thompson sampling in linear MDPs with delays.\n\n* The bounds are tight up to a $\\sqrt{H}$ factor (and possibly polylog terms).\n\nWEAKNESSES:\n* The experimental setup is somewhat limited, with only two states and 10 feature dimensions.\n\n* Little detail on how LMC was tuned, and evidence it converged in practice.\n\nQUESTIONS:\n* Can you clarify your comment on page 5, where you claim that your analysis is relevant for double-Q learning? IIUC, the point of having two networks in double-Q learning is to alleviate overestimation of Q-values, whereas, in your analysis, taking multiple samples of the Q function is done to ensure that your estimates are higher than the true optimal Q value with high probability? (That sounds quite different.)\n\n* Could you please clarify why does it make sense to condition the posterior model on past estimates of $\\tilde{V}$? From a Bayesian perspective, the only observed quantities are rewards (and delays); it seems like conditioning on past estimates of $\\tilde{V}$ could lead the algorithm astray, especially if the number of samples required for convergence is large. How does your approach/analysis handle this?\n\nLIMITATIONS:\nThis is not my area of expertise. I am not aware of limitations beyond the already raised simplicity of experiments (which is ok for a theoretical paper like this).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Thompson sampling methods for reinforcement learning problems with delayed feedbacks. The authors tackle the challenge of delayed feedback under the linear function approximation setting. They proposed two new Thompson sampling algorithms, using the exact sampling mechanism and also the Langevin dynamics for an approximate posterior sampling. The regrets and the computation complexity for both algorithms are then analyzed.  They further conduct experiments to compare the empirical results of the proposed methods and the UCB-based algorithms.\n\nSTRENGTHS:\n1) The authors solve the reinforcement learning (RL) problem with delayed feedback from the angle of posterior sampling rather than the well-explored UCB-based algorithms, which is a new solution to this problem.\n\n2) The second posterior sampling algorithm based on Langevin dynamics outperforms the first algorithm in terms of computation complexity, which improves the efficiency of the posterior sampling methods for the problem of RL with delayed feedback. \n\n3) The authors conduct sufficient empirical comparisons to justify the superiority of the posterior sampling over UCB methods in RL with delayed feedback.\n\nWEAKNESSES:\n1) Although this is the first work for RL with delayed feedback by posterior sampling, the technical contribution of this work still seems not significant. In terms of the algorithm design, the methods seem to be combinations of existing posterior sampling methods for linear MDPs and the multi-round sampling scheme in [24]. The theoretical analysis, in my understanding, is also mainly dependent on the analysis for RL without delay and the decomposition of the covariance matrix to show the error caused by the delay. Can the authors further discuss in detail the major technical challenges in this work and the corresponding technical contributions?\n\n2) In order to clearly show the effectiveness of the posterior sampling methods, the authors need to further present a detailed comparison of the regret results in the existing UCB-based approaches and the proposed posterior sampling approach for RL with delayed feedback.\n\n3) Is there any possibility of further improving the regret term associated with the delay? For example, use some techniques to achieve a regret more robust to the delay $\\tau$, i.e., having a sublinear dependence on $\\tau$, or to improve the dependence on the multiplicative factors $H$ and $d$. I am wondering what the best-known result is for the UCB-based approaches under a similar setting.\n\nQUESTIONS:\nPlease see the Weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

comments: SUMMARY:
This paper considers the scenario where the feedback is delayed. The motivation for such a scenario is recommender/advertisement systems and clinical trials. They present a novel algorithm based on Thompson sampling idea: Delayed-PSVI. The authors present theoretical results for worst-case regret. They also show experimental results on synthetic linear MDP.

STRENGTHS:
Most of the works in RL are focused on instant feedback. This paper consider a real-time problem where the feedback may not instant. The theoretical results along with the experimental section make solid contribution.

WEAKNESSES:
-

QUESTIONS:
1. Posterior sample algorithms often are restricted to Gaussian, multinomial distribution due to posterior computation. The proposed algorithm considers normal distribution, which I have commonly seen in posterior sampling algorithms. Is such an assumption restrictive when it comes to real-world application?

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-01