SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nikki Lijing Kuang *
11kuang@ucsd.edu
Ming Yin *
my0049@princeton.edu
Mengdi Wang
mengdiw@princeton.edu
yianma@ucsd.edu
Recent studies in reinforcement learning (RL) have made significant progress by leveraging function approximation to alleviate the sample complexity hurdle for better performance. Despite the success, existing provably efficient algorithms typically rely on the accessibility of immediate feedback upon taking actions. The failure to account for the impact of delay in observations can significantly degrade the performance of real-world systems due to the regret blow-up. In this work, we tackle the challenge of delayed feedback in RL with linear function approximation by employing posterior sampling, which has been shown to empirically outperform the popular UCB algorithms in a wide range of regimes. We first introduce Delayed-PSVI, an optimistic value-based algorithm that effectively explores the value function space via noise perturbation with posterior sampling. We provide the first analysis for posterior sampling algorithms with delayed feedback in RL and show our algorithm achieves  $\widetilde{O} (\sqrt{d^3H^3T} +d^2 H^2\mathbb{E}[\tau])$  worst-case regret in the presence of unknown stochastic delays. Here  $\mathbb{E}[\tau ]$  is the expected delay. To further improve its computational efficiency and to expand its applicability in high-dimensional RL problems, we incorporate a gradient-based approximate sampling scheme via Langevin dynamics for Delayed-LPSVI, which maintains the same order-optimal regret guarantee with  $\widetilde{O} (dHK)$  computational cost. Empirical evaluations are performed to demonstrate the statistical and computational efficacy of our algorithms.
Reinforcement Learning (RL) is the main workhorse for sequential decision-making problems where an agent needs to balance the trade-off between exploitation and exploration in the unknown environment. The flexible and powerful function approximation endowed by deep neural networks greatly contributes to the empirical success of RL in domains such as Large Language Models (LLMs) [50, 59], robotics [51], and AI for Science [37]. In general, collecting real-world training data from such practical systems can be expensive, which requires algorithms to be both sample efficient and computationally efficient. Recently, there have been growing efforts towards studying provably efficient RL algorithms in settings ranging from tabular Markov Decision Processes (MDPs) [29, 45, 69] to large-scale RL with function approximation [13, 35]. However, these algorithms typically rely
on the availability of immediate observations of states, actions and rewards in learning no-regret policies. Unfortunately, such an assumption is rarely satisfied in real-world domains, where delayed feedback is ubiquitous and fundamental. In recommender systems and online advertisement, for instance, responses from users (e.g. click, purchase) may not be immediately observable, which can take hours or days. In healthcare and clinical trials, medical feedback from patients on the effectiveness of treatments can only be determined at a deferred time frame. More examples exist in platforms that involve human interaction and evaluation, including human-robot collaboration in teleoperating systems and multi-agent systems [15, 39], aligning LLMs with human values [50, 63], and fine-tuning generative AI models using RL with human feedback (RLHF) [11, 41].
Despite the practical importance of addressing delays in decision-making problems, theoretical understanding of delayed feedback in RL remains limited. Recent parallel works study exploration under delayed feedback via upper confidence bound (UCB) algorithms [8] in tabular RL [29, 45], adversarial MDPs [36, 40], and RL with low policy-switching scheme [68] (see Table 1). Nevertheless, posterior sampling (PS) analysis that handles delayed feedback remains untackled in both bandit and RL literature. We aim to bridge the gap in this work.
PS is a randomized Bayesian algorithm that extends Thompson sampling (TS) [57] to RL, which selects an action according to its posterior probability of being the best. This philosophy inspires a number of promising exploration strategies that explicitly or implicitly adopt PS to explore [52], including bootstrapped DQN [42, 47] and RLSVI [49]. Compared to the popular UCB algorithms, it bears greater robustness in the presence of delays [14], and provides exceptional computational efficiency with competitive empirical performance [14, 65]. The fact that posteriors are often intractable in practice necessitates the use of approximate Bayesian inference such as ensemble sampling, variational inference (VI) and Markov Chain Monte Carlo (MCMC) [20, 38, 47].
In this paper, we provide the first analysis for the class of PS algorithms that handles delayed feedback in RL frameworks, in which the trajectory information is randomly delayed according to some unknown distribution. We highlight that delayed feedback model imposes new challenges that do not arise in standard RL settings. Algorithmically, it requires the computation of new posterior variance due to the weaker concentration arising from delays. Theoretically, it complicates the frequentist analysis of PS algorithms in several ways: (a) the lack of timely update in posterior learning can cause distribution shift, especially in the case of approximate sampling; (b) delays need to be carefully disentangled to quantify the penalty in regret decomposition and it prohibits the direct application of previous analysis; (c) balance between concentration and anti-concentration needs to be handled deliberately to achieve sub-linear regret.
To tackle these challenges, we introduce two novel value-based algorithms for linear MDPs under unknown stochastic delayed feedback. Developed upon Bayesian linear modeling with a multiround ensembling mechanism ( $M \approx \mathrm{Polylog}(H, K, d, \delta)$  round), our algorithms achieve a sublinear worst-case regret without requiring the knowledge of delay, thereby addressing the question raised in [60] that "No frequentist analysis exists for posterior sampling with delayed feedback". Empirical studies show that our algorithms outperform UCB-based methods in terms of both statistical accuracy and computational efficiency when delays are well-behaved or even long-tailed. We summarize our main contributions as follows.
- We propose the Delayed Posterior Sampling Value Iteration (Delayed-PSVI, Algorithm 1) for linear MDPs. It achieves a high-probability worst-case regret of  $\tilde{O} (\sqrt{d^3H^3T} + d^2 H^2\mathbb{E}[\tau])^2$ , where  $\mathbb{E}[\tau]$  is the expected delay.
- We leverage Langevin Monte Carlo (LMC) for approximate inference and introduce Delayed Langevin Posterior Sampling Value Iteration (Delayed-LPSVI, Algorithm 2), which maintains the same order-optimal worst-case regret of  $\widetilde{O} (\sqrt{d^3H^3T} +d^2 H^2\mathbb{E}[\tau])$ . To the best of our knowledge, this is the first analysis that provably incorporates LMC in linear MDPs and jointly considers the impact of delays.
- Both algorithms achieve the optimal dependence on the parameters  $d$  and  $T$  in leading terms under the class of PS algorithms, and recover the best-available frequentist regret of  $\widetilde{O}(\sqrt{d^3H^3T})$  [31, 72] as in non-delayed linear MDPs when  $\mathbb{E}[\tau] = 0$ . In particular, Delayed-LPSVI reduces the computational complexity of Delayed-PSVI from  $\widetilde{O}(d^3HK)$
It provides a stronger guarantee as opposed to the weaker worst-case expected regret and Bayesian regret.
to  $\widetilde{O}(dHK)$ , expanding the applicability in complex high-dimensional RL tasks while potentially providing a more flexible form of approximation.
Algorithms	Setting	Exploration	Worst-case Regret	Computation
[28]	Linear Bandits	UCB	\(\widetilde{O}(d\sqrt{T}+d^{3/2}\mathbb{E}[\tau])\)	Confidence set optimization
[29]	Tabular MDPs	UCB	\(\widetilde{O}(\sqrt{SAH^{3}T}+S^{2}AH^{3}\mathbb{E}[\tau])\)	Active update
[68]	Linear MDPs	UCB	\(\widetilde{O}(\sqrt{d^{3}H^{3}T}+dH^{2}\mathbb{E}[\tau])\)	Multi-batch reduction
[40]	Adversarial MDPs	UCB	\(\widetilde{O}(H^{2}S\sqrt{AK}+H^{3/2}\sqrt{S\sum_{k=1}^{K}\tau_{k}})\)	Confidence set optimization
Delayed-PSVI (Thm 1)	Linear MDPs	PS	\(\widetilde{O}(\sqrt{d^{3}H^{3}T}+d^{2}H^{2}\mathbb{E}[\tau])\)	\(O((d^{3}+Md)HK)\)
Delayed-LPSVI (Thm 2)	Linear MDPs	PS	\(\widetilde{O}(\sqrt{d^{3}H^{3}T}+d^{2}H^{2}\mathbb{E}[\tau])\)	\(O((N+d)MHK)\)
Delayed-PSLB (Cor 2)	Linear Bandits	PS	\(\widetilde{O}(\sqrt{d^{3}T}+d^{2}\mathbb{E}[\tau])\)	\(O((N+d)MK)\)
UCB Lower bound [27]	Linear MDPs	UCB	\(\Omega(dH\sqrt{T})\)	—
PS Lower bound [24]	Linear Bandits	PS	\(\Omega(\sqrt{d^{3}T})\)	—
Table 1: Summary of regret bounds in linear bandits and episodic MDPs under stochastic delay. We denote by  $T$  the time horizon,  $K$  the number of episodes,  $H$  the episode length,  $d$  the dimension of feature space,  $M$  the number of sampling rounds, and  $N$  the total iterations in running LMC. Our choice of  $M$  and  $N$  has order of  $\operatorname{Polylog}(H, K, d, \delta)$ , ensuring both Delayed-PSVI and Delayed-LPSVI are computationally efficient and statistically sample-efficient. We remark that the gap in the frequentist regret between PS and best UCB-based methods is unavoidable by a factor of  $\sqrt{d}$  [24]. Thus, our dependencies on  $d$  and  $T$  are optimal for the class of PS algorithms. Our results fulfill the caveat [60] that no worst-case analysis exists for PS with delay.
1.1 Related Work.
Delayed feedback. In bandit literature, delay is extensively studied in both stochastic [22, 56, 60, 77] and adversarial settings [32, 58, 78] for UCB-based methods. In comparison, while delay draws much attention in empirical RL studies [12, 17, 18], there is a lack of theoretical understanding until very recently. Parallel works focus on UCB-based methods in various RL settings [16, 28, 36, 40, 45, 68]. To provide the first analysis for PS algorithms in this context, we consider stochastic delays under linear function approximation without requiring any policy-switch scheme as in [68].
Posterior sampling. To encourage efficient exploration, PS is adopted in value-based methods to inject randomness in empirical Bellman update via Gaussian noise. From the Bayesian perspective, it is equivalent to maintaining an approximate Gaussian posterior for parameterized value function. Its sample complexity is studied in tabular settings [48, 49, 53], with the sharp worst-case regret of  $\widetilde{O}(H^2 S\sqrt{AT})$  [5]. Under linear function approximation, frequentist regret of  $\widetilde{O}(\sqrt{d^3 H^3 T})$  [31, 72] and Bayesian regret of  $\widetilde{O}(d\sqrt{H^3 T})$  [19] are established. However, in complex problem domains that require higher computational efficiency and more refined surrogates, approximate inference is the remedy. Toward this end, we resort to a gradient-based MCMC method.
Langevin Monte Carlo. LMC is a class of MCMC methods tailored for large-scale online learning with strong convergence guarantee by utilizing the first-order gradient information [64]. It has been successfully applied to stochastic bandits [43], linear bandits [65] and tabular RL [38], In this work, we extend its usage in linear MDPs and demonstrate its convergent property under delay.
RL with Function Approximation. Function approximation is widely adopted to empower RL for large-scale applications. Fruitful results have been established for regret minimization in two types of MDPs under linear function approximation: linear mixture MDPs [9, 67], and linear MDPs [35, 66]. In linear mixture MDPs where transition kernel is parameterized as a linear combination of base models, provably efficient algorithms are discussed [13, 75, 76] and [75] provides the corresponding lower bound of  $\Omega(dH\sqrt{T})$ . In contrast, linear MDPs enjoy a linear structure in value functions by assuming a low-rank representation for both transitions and reward function, where algorithms are shown to enjoy polynomial sample complexity [27, 35, 62, 73]. When it comes to general function approximation, theoretical guarantees are developed based on measures of eluder dimension [54, 61] and Bellman rank [33]. In this work, we focus on delayed feedback in linear MDPs.
We study the finite-horizon episodic MDP  $(\mathcal{S},\mathcal{A},H,\mathbb{P},r)$ , which is time-inhomogeneous, and denote by  $\mathcal{S}$ ,  $\mathcal{A}$  the state and action spaces respectively,  $H$  the episode length,  $\mathbb{P} = \{\mathbb{P}_h\}_{h=1}^H$  the
transition dynamics, and  $r = \{r_h\}_{h=1}^H$  reward function. At each step  $h \in [H]$ ,  $\mathbb{P}_h: \mathcal{S} \times \mathcal{A} \to \Delta_S$  specifies the probabilities of transitioning from the current state-action pair into the next state, and  $r_h: \mathcal{S} \times \mathcal{A} \to [0,1]$  emits a bounded reward. We adopt the prior protocol of linear MDPs as follows.
Definition 1 (Linear MDPs [35, 66]). Suppose there exists a known feature map  $\phi : S \times \mathcal{A} \to \mathbb{R}^d$  that encodes each state-action pair into a  $d$ -dimensional feature vector. An MDP is a linear MDP if for any time step  $h \in [H]$ ,  $\forall (s, a) \in S \times \mathcal{A}$ , both the transition dynamics  $\mathbb{P}$  and reward function  $r$  are linear in  $\phi$ :
$$
\mathbb {P} _ {h} (\cdot | s, a) = \phi (s, a) ^ {\mathrm {T}} \mu_ {h} (\cdot), \quad r _ {h} (s, a) = \phi (s, a) ^ {\mathrm {T}} \theta_ {h}, \tag {1}
$$
where  $\mu_h: \mathcal{S} \to \mathbb{R}^d$  contains  $d$  unknown probability measures over  $\mathcal{S}$ , and  $\theta_h \in \mathbb{R}^d$ . Furthermore, we assume that  $\forall (s,a) \in \mathcal{S} \times \mathcal{A}, \| \phi(s,a) \| \leq 1$ , and  $\forall h \in [H]$ ,  $\| \theta_h \| \leq \sqrt{d}$ ,  $\left\| \int_{\mathcal{S}} \mathrm{d}\mu_h(s') \right\| \leq \sqrt{d}$ , where  $\|\cdot\|$  denotes the Euclidean norm.
A non-stationary policy  $\pi = \{\pi_h\}_{h=1}^H$  assigns the action to take at step  $h$  in state  $s_h \in S$ . Accordingly, we define the value functions of a policy  $\pi$  as the expected rewards received under  $\pi$ :
$$
Q _ {h} ^ {\pi} (s, a) = \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} | s _ {h} = s, a _ {h} = a \right], \quad V _ {h} ^ {\pi} (s) = \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} | s _ {h} = s \right].
$$
We further denote by  $\pi^{*}$  the optimal policy whose value functions are defined as  $V_{h}^{*}(s)\coloneqq V_{h}^{\pi^{*}}(s) = \sup_{\pi}V_{h}^{\pi}(s)$  and  $Q_{h}^{*}(s,a)\coloneqq Q_{h}^{\pi^{*}}(s,a) = \sup_{\pi}Q_{h}^{\pi}(s,a)$ . Under Definition 1, the action-value functions are always linear in the feature map, and there exists some  $w_{h}^{*}$  such that  $Q_{h}^{*} = \phi^{\mathrm{T}}w_{h}^{*}$  (Lemma A.1). For ease of notation,  $\forall (s,a)$ , denote  $[\mathbb{P}_hV_{h + 1}^\pi ](s,a) = \mathbb{E}_{s'\sim \mathbb{P}_h(\cdot |s,a)}[V(s')]$ . By Bellman equation and Bellman optimality equation,
$$
Q _ {h} ^ {\pi} (s, a) = \left(r _ {h} + \mathbb {P} _ {h} V _ {h + 1} ^ {\pi}\right) (s, a), \quad V _ {h} ^ {\pi} (s) = Q _ {h} ^ {\pi} \left(s, \pi_ {h} (s)\right),
$$
$$
Q _ {h} ^ {*} (s, a) = \left(r _ {h} + \mathbb {P} _ {h} V _ {h + 1} ^ {*}\right) (s, a), \quad V _ {h} ^ {*} (s) = \max  _ {a} \left(r _ {h} + \mathbb {P} _ {h} V _ {h + 1} ^ {*}\right) (s, a).
$$
The goal of the agent is to maximize the cumulative episodic rewards or equivalently, minimize the regret that quantifies the difference between the value of the optimal policy  $\pi^{*}$  and that of the executed policies. Formally, the worse-case regret over  $K$  episodes is given as:
$$
R (T) = \sum_ {k = 1} ^ {K} V _ {1} ^ {*} \left(s _ {1} ^ {k}\right) - V _ {1} ^ {\pi_ {k}} \left(s _ {1} ^ {k}\right). \tag {2}
$$
Remark 1. Different types of regret are used in literature to measure the performance of PS algorithms. Bayesian regret  $\mathbb{E}_{w^{*}\sim p_{0}(\cdot)}\mathbb{E}[R(T)|w^{*}]$  is often considered when assuming a prior  $p_0(w)$  over the true parameter  $w^{*}$ . Frequentist regret  $\mathbb{E}[R(T)]$  is considered when  $w^{*}$  is fixed, where the expectation is taken over all the randomness over data and algorithm. As explained in Appendix A.2, the worst-case regret that we study is stronger than the frequentist regret.
2.1 Delayed Feedback Model
In this work, we consider stochastic delays across episodes. More specifically, the trajectory (i.e., sequence of states, actions and rewards) generated in each episode is not immediately observable in the presence of delay. The formal definition is given as follows.
Definition 2 (Episodic Delayed Feedback). In each episode  $k \in [K]$ , the execution of a fixed policy  $\pi^k$  generates a trajectory  $\{s_h^k, a_h^k, r_h^k, s_{h+1}^k\}_{h \in [H]}$ . Such trajectory information is called the feedback of episode  $k$ . Let  $\tau_k$  represent the random delay between the rollout completion of episode  $k$  and the time point at which its feedback becomes observable.
Remark 2. Various types of delays have been independently studied in the literature, including delays in states [4, 12, 16], delays in rewards [25, 45, 60], delays in actions[56], and delays in trajectories [29, 68]. We focus on the last scheme which facilitates the delayed analysis of value-based methods in episodic linear MDPs.
3Linear MDPs recover tabular MDPs by taking  $d = |\mathcal{S}||\mathcal{A}|$ , where feature map is a one-one mapping for each state-action pair.
Algorithm 1: Delayed Posterior Sampling Value Iteration (Delayed-PSVI)
Input: priors  $p_0(w_h^k)\gets \mathcal{N}(0,\lambda I)$  , scaling factor  $\nu$  , multi-round paramter  $M$  , hyper parameters  $\lambda$  and  $\sigma^2$    
1 Initialization:  $\forall k,h,\widetilde{Q}_{H + 1}^{k}(\cdot ,\cdot),\widetilde{V}_{H + 1}(\cdot ,\cdot),\widetilde{V}_h(\cdot ,\cdot)\gets 0,\mathcal{D}_h\gets \emptyset .$    
for episode  $k = 1,\dots ,K$  do   
3 Sample initial state  $s_1^k$    
4 for time step  $h = H,\ldots ,1$  do   
5  $\boldsymbol {y}_h\gets [y_h^1,\dots ,y_h^{k - 1}]$  with  $y_{h}^{\tau}\gets \mathbb{1}_{\tau ,k - 1}\cdot [r_{h}^{\tau} + \widetilde{V}_{h + 1}(s_{h + 1}^{\tau})]$    
6  $\Phi_h\gets [\phi^1,\phi^2,\dots ,\phi^{k - 1}]$  with  $\phi^{\tau} = \mathbb{1}_{\tau ,k - 1}\cdot \phi (s_h^\tau ,a_h^\tau)$    
7  $\Omega_h^k\gets \sigma^{-2}\Phi_h\Phi_h^{\mathrm{T}} + \lambda I,\widehat{w}_h^k\gets \sigma^{-2}(\Omega_h^k)^{-1}\Phi_h\boldsymbol {y}_h^{\mathrm{T}}$    
8  $p(w_h^k\mid \mathcal{D}_h,y_h)\gets \mathcal{N}(\widehat{w}_h^k,\nu^2\cdot (\Omega_h^k)^{-1})$    
9 for  $m = I,\dots ,M$  do   
10 Sample  $\widetilde{w}_h^{k,m}\sim p(w_h^k\mid \mathcal{D}_h,y_h)$    
11  $\widetilde{Q}_h^{k,m}(\cdot ,\cdot)\gets \phi (\cdot ,\cdot)^{\mathrm{T}}\widetilde{w}_h^{k,m}$    
12 Update  $\widetilde{Q}_h^k (\cdot ,\cdot)\gets \max_m\widetilde{Q}_h^{k,m}$    
13  $\widetilde{V}_h(\cdot ,\cdot)\gets \max_a\min \{\widetilde{Q}_h^k (\cdot ,a),H - h + 1\}$    
14 Update  $\pi_h^k (\cdot)\gets \operatorname {argmax}_{a\in A}\min \{\widetilde{Q}_h^k (\cdot ,a),H - h + 1\}$    
for time step  $h = 1,\dots ,H$  do   
16 Choose action  $a_h^k = \pi_h^k (s_h^k)$    
17 Collect trajectory observations.  $D_h\gets D_h\cup \{(s_h^k,a_h^k,r_h^k,s_{h + 1}^k)\}$    
/\* Feedback generated in episode  $k$  cannot be immediately observed in the presence of delay
Episodic delays do not disrupt the policy rollout within an episode, but alter the utilization of information in subsequent episodes. More precisely, the feedback of episode  $k$  remains inaccessible for the following  $\tau_{k} - 1$  episodes, becoming observable only at the onset of the  $(k + \tau_{k})$ -th episode. To track whether the feedback generated at episode  $k$  is revealed at episode  $k'$ , we utilize the indicator  $\mathbb{1}_{k,k'} := \mathbb{1}\{k + \tau_k \leq k'\}$  (where 1 denotes "yes" and 0 denotes "no"). We follow the standard assumption in literature in [28, 68] to assume delays are sub-exponential. It is crucial to note that this assumption primarily serves the purpose of theoretical analysis and is not a prerequisite for the effective functioning of our algorithms in practical settings. Without loss of generality, we discuss the performance bound under general random delays in Section 4 and empirically study the performance against different types of delays in Section 5.
Assumption 1 (Sub-exponential Episodic Delay). The episodic delays  $\{\tau_k\}_{k=1}^K$  are non-negative, integer-valued, independent and identically distributed  $(v,b)$ -subexponential random variables:  $\tau_k \stackrel{i.i.d.}{\sim} f_\tau(\cdot)$  with  $f_\tau(\cdot)$  being the probability mass function, and  $\mathbb{E}[\tau]$  being the expected value. For all  $k \in [K]$ , the moment generating function of  $\tau_k$  satisfies:
$$
\mathbb {E} \left[ \exp \left(\gamma \left(\tau_ {k} - \mathbb {E} [ \tau ]\right)\right) \right] \leq \exp \left(\frac {1}{2} v ^ {2} \gamma^ {2}\right),
$$
where  $v$  and  $b$  are non-negative, and  $|\gamma |\leq 1 / b$ .
3 Delayed Posterior Sampling Value Iteration
In this section, we introduce a novel optimistic value-based algorithm, namely, Delayed Posterior Sampling Value Iteration (Delayed-PSVI), which efficiently explores the value function space in linear MDPs by embracing several critical components: posterior sampling that injects random noise when performing the least-square value iteration, optimism via multi-round sampling to achieve the optimal worst-case regret and delayed feedback model that encodes episodic trajectory delays.
Noisy value iteration via posterior sampling. At the beginning of each episode, we apply PS to sample an estimated value function from the posterior, which is maintained using the observed feedback  $\mathcal{D}$  over the previous episodes. Specifically, at each time step, the  $Q$ -function is parameterized by some  $w \in \Gamma$  such that  $\widetilde{Q}(s,a) = \phi(s,a)^{\mathrm{T}}w$  is an approximation of the corresponding true optimal  $Q$ -function  $Q^{*}(s,a)$ . Let  $p_{0}(w)$  be the prior of  $w$ , and  $p(\boldsymbol{y}|\boldsymbol{w},\mathcal{D})$  be the likelihood of the observation  $\boldsymbol{y}$ , then the posterior of  $w$  satisfies:
$$
p (w | \mathcal {D}, \boldsymbol {y}) \propto \exp (- L (w, \boldsymbol {y}, \mathcal {D})) p _ {0} (w),
$$
where  $L(\cdot)$  is the log-likelihood. Unlike the case of model-based RL (MBRL), where PS is utilized to maintain an exact posterior over the environment model, we aim to adopt PS to perform noisy value-iteration by injecting randomness for efficient exploration of the value function space. Specifically, at each step  $h \in [H]$ , we consider Gaussian-noise perturbation in Delayed-PSVI by setting prior as  $p_0(w_h) = \mathcal{N}(0, \lambda I_d)$ , and log-likelihood (with  $\mathcal{D}_h = \{s_h^\tau, a_h^\tau, r_h^\tau, s_{h+1}^\tau\}_{h \in [H]}^{\tau \in [k-1]}$ ) as
$$
L \left(w _ {h}, \boldsymbol {y} _ {h}, \mathcal {D} _ {h}\right) = \sum_ {\tau = 1} ^ {k - 1} \left(\phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\mathrm {T}} w _ {h} - y _ {h} ^ {\tau}\right) ^ {2}, \tag {3}
$$
where  $\pmb{y}_h = [y_h^1, \dots, y_h^{k-1}]$  with  $y_h^\tau = r_h^\tau(s_h^\tau, a_h^\tau) + \widetilde{V}_{h+1}(s_{h+1}^\tau)$ . Then for all step  $h \in [H]$  of episode  $k$ , the posterior of  $w_h^k$  follows a Gaussian distribution,
$$
p \left(w _ {h} ^ {k} \mid \mathcal {D} _ {h}, \boldsymbol {y} _ {h}\right) \propto \mathcal {N} \left(\left(\Omega_ {h} ^ {k}\right) ^ {- 1} \Phi_ {h} \boldsymbol {y} _ {h} ^ {\mathrm {T}}, \left(\Omega_ {h} ^ {k}\right) ^ {- 1}\right),
$$