Fast Value Tracking for Deep Reinforcement Learning
Authors: Frank Shih
Faming Liang
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement learning
SGMCMC
uncertainty quantification
Primary area: reinforcement learning
TL;DR: We develop a SGMCMC sampling framework for reinforcement learning algorithms, enabling us to track the uncertainties of the value functions in the sequential decision making process.

Abstract: Reinforcement learning (RL) tackles sequential decision-making problems by creating
agents that interacts with their environment. However, existing algorithms often view these problem as 
static, focusing on point estimates for model parameters to maximize expected rewards, neglecting the stochastic dynamics of agent-environment interactions and the critical role of uncertainty quantification.
Our research leverages the Kalman filtering paradigm to introduce a novel and scalable sampling algorithm called Langevinized Kalman Temporal-Difference (LKTD) for deep reinforcement learning. This algorithm, grounded in Stochastic Gradient Markov Chain Monte Carlo (SGMCMC), efficiently draws samples from the posterior distribution of deep neural network parameters. Under mild conditions, we prove that the posterior samples generated by the LKTD algorithm converge to a stationary distribution. This convergence not only enables us to quantify uncertainties associated with the value function and model parameters but also allows us to monitor these uncertainties during policy updates throughout the training phase. The LKTD algorithm paves the way for more robust and adaptable reinforcement learning approaches.

FAST VALUE TRACKING FOR DEEP REINFORCEMENT LEARNING
Frank Shih Department of Statistics Purdue University West. Lafayette, IN 47907, USA shih37@purdue, edu
Faming Liang
Department of Statistics
Purdue University
West Lafayette, IN 47907, USA
fml i imlianguuudue... edu
ABSTRACT
Reinforcement learning (RL) tackles sequential decision-making problems by creating agents that interacts with their environment. However, existing algorithms often view these problem static, focusing on point estimates for model parameters to maximize expected rewards, neglecting the stochastic dynamics of agent-environment interactions and the critical role of uncer- tainty quantification. Our research leverages the Kalman filtering paradigm introduce novel and scalable sampling algorithm called Langevinized Kalman Cemporal-Diffeeene algorithm, grounded in Stochastic Gradient Markov Chain Monte Carlo (SGMCMC), efficiently draws samples from the posterior distribution of deep neural network parameters. Under mild conditions, we prove that the posterior samples generated by the LKTD algorithm converge to stationary distribution. This convergence not only enables us to quantify uncertainties associated with the value function and model parameters but also allows us to monitor these uncertainties during policy updates throughout the training phasc. The LKTD algorithm paves the way for more robust and adaptable reinforcement learning approaches.
(LKTD) for deep reinforcement learning. This
INTRODUCTION
Over the last decade, RL. has achieved remarkable successes across a diverse array of tasks, including robotics (Kormushev et al. 2013) video games (Silver et al. 2016), bidding strate- gies (Jin et al. 2018), and ridesharing optimization (Xu et al. 20188). As a mathematical model, RL solves sequential decision-making problems by designing agent that interacts with the environment, the goal is to Iearn an optimal policy that maximizes the expected total reward for the agent. Prominent value-based algorithms, including Temporal-differene (TD) learning (Sutton, 1988), State action reward state action (SARSA) (Sutton & Barto 2018), and Q-learning, aim derive an optimal policy through learning values of states (or Q-values). Traditionally, these methods treat the state value (or Q-value) as deterministic function, focusing on calculating point estimates of model parameters, thereby overlooking the inherent stochasticity in agent-environnent interactions.
In the context of RL, fair algorithm should exhibit the features: (i) Uncertainty quantificn- tion, which addresses the stochastic nature of the agent-enviromment interactions, thereby enhancing the robustness of the learned policy; (ii) Dynamicity, which considers the dynamics of the agent-environment interaction system, thereby enhancing the practicality of the RI technique; (iii) Nonlinear approrimation, which employs, for example, a deep neural network to approximate the value function, thereby broadening the algorithm's 's applicability; (iv) Computational efficiency, which is scalable with respect to the model dimension and training sample size, facilitating online learning. Therefore, in RL, it is more suitable to treat values or model paramcters as random variables rather than fixed unknowns, focusing on tracking dynamic changes rather than achieving point convergence during the policy learning process.
To achieve these goals, the Kalman Temporal Difference (KTD) framework has been studied
for RL in the literature, as in references such e.g., Geist & Pietquin (2010). Tripp &

---

Shachter (2013) and Shashua & Mannor 2020 In these studies, values or their parameters are treated as random variables, and the focus is on the tracking property of the policy learning process. Specifically, KTD conceptualizes RL as state-space model:
8, 01-1 + W/:
(1)
r1 = h12,, 81) 7p.
where 0g E IR" denotes the parameters at time step t with dimension p. Wa E R' and ni E R" denote two independent multivariate Gaussian vectors, 2y denotes set of states and actions collected at time step t, E R" denotes a vector of rewards, denotes the mumber of samples, and h.)) is a function to be defined in Section 2.2] Within the framework of state-space models, the top equation in ! is reffered to as the state evolution equation, while the bottom equation known as the measurement equation. Under the normality assumption and for a linear measurement equation, where h(x, 0) is linear function of 0, the Kalman filter (Kalman 1960) is able to iteratively update the mean and variance estimates for 8g conditioned on the rewards (T2, Pt-1,..... P1), enabling proper quantification of the uncertainty associated with the dynamic agent-enviromment interaction system. However, when h( 0) becomes nonlinear, it necessitates the use of linearization techniques. Specifically, Geist & Pielquin (2010) employs Unscented Kalman Filter (UKF) (Wan & Van Der Merwel 2000), while Shashua & Mannor (2020) utilize Extended Kalman Filter
al, (1979)
to approximate the covariance matrices of 0,. Unfortunately, both UKF and EKF becomes computationally inefficient for high-dimensional parameter spaces, a common scenario when employing large-scale neural networks to approximate h(-.)) These filters require O(p²) additional space to store the covariance matrix and (np²) for matrix multiplications at each iteration. Moreover, the linearization operation involved in these algorithus can degrade the accuracy of estimation. To address the limitations encounteeed by KTD, we reformulate RL as the following state space model:
0: - 4 Vo (()-)) Wt.
2
(2)
Tt h(x,, + T/-
where We ~ e0,, =(0) represents a prior density function we impose on 0, and {Ex t = .. is a positive sequence decaying to zero. Additionally, we propose to update Og using the Langevinized Ensemble Kalman Filter (LEnKF) algorithm (Zhang et al., 2023). With the formulation ( and the LEnKF algorithm, converges to a proper distribution as the learning horizon + oo, enabling the uncertainty associated with the dynamic agent-environment interact ion system to be properly quantified. Inclusion of the prior information in the state evolution equation generally robustifies the performance of the RL algorithm. For instance, when employing a large-scale deep neural network to approximate the function h(--)) selecting an appropriate x(-), such as a mixture Gaussian distribution, can lead the sparsification of the neural network. This enhances the robustnoss of the learned policy according to the theory of sparsc deep lcarning (Sun et al. 2022). Compared existing KTD algorithms, the proposed algorithm can directly handle nonlinear function i(.,-) without the need for a linearization operator. The proposed algorithm enables fast value tracking al a complexity of O(mp) per iteration, scalable for large-scale neural networks. It also enhances memory-efficiene the covariance matrix with particles, representing samples of O. It is worth noting that both the sample size and the number of particles retained during the algorithm's execution are typically significantly smaller than the parameter size p. Lastly, we extend the convergence theory of our proposed algorithm to include scenarios that utilize replay buffers, thereby expanding its applicability beyond the on-policy framework.
show in Section 3 that 8,
it replaces the storage for
Compared to model variability of 0 through the prior (0), while still accounting for the dynamics of the system. However, these restrictions do not diminish the generality and adaptivity of the model (2). thanks to the universal approximation ability of deep neural networks that will be used in approximating h(-,-) in this paper.
our new formulation imposes slightly more restrictions on the
Related Works Bootstrapped DQN (Osband et al., 2016) and Quantile Regression DQN
(Bellemare et al.) (0017) also aim to learn the uncertainty estimates for the value function, but they are not formulated under the KTD framework.
2

---

2 BACKGROUND
2.1 MARKOV DECISION PROCESS
The standard RL procedure aims to learn an optimal policy from the interaction experiences between an agent and an environment, where the optimal policy maximizes the agent's expected total reward. The RL procedure can be described by a Markov Decision Process (MDP) represented by [S,A,P.) ?), where S is set of states, A is finite set of actions, P:S. x A S + R is the state transition probability from state s to state s' by taking action a, denoted by P(s'|s, a), r(s, a) is a random reward received from taking action a at state 8, and E (0, 1) is discount factor. At each time step t, the agent observes state 8, E S and takes action a, E A according to policy P with probability Po(als), then the environment returns reward rn r(s,, a.) and a new state 8141 E S. For given policy p, the performance is measured by the state value function Vy(s) = 1 7're|00 = s] and the state-action value function Qo(8, a) = E, -e= 7/re |80 8, 0g = al; which are called V-function and Q-function, respectively. Both functions satisfy the Bellman equation:
Vi(s) = Eplr(s, + (V)'''),
Qo(o,a) a) = Eplr(s, + ( where s' ~ a), a ~ Pp(-1s), a' ~ Pp(-1s', and the expectation EpH is taken over the transition probability distribution P for given policy p.
2.2
KALMAN TEMPORAL DIFFERENCE (KTD) ALGORITHMS
Let 8, = (s s/²) s...,,, a, = =aa "0
(2) " d(")),, and r, = (r. r ..... rf (aa
denote, respectively, a vector of n states, actions, rewards collected at time step Given the Bellman equation, the function h( 01) in ( can be expressed as
( (8)) - 7Vo (8++1),
for V-lunction, for Q-function,
h() 1 (8t, ar)
(4)
Q0,
(8141,
where 24. {s,, a2 88+1, a+t1), Vo, (s:)
(2) ),...,.. Vo, (s (o) and (6 (1) Vo, (0)
).... Sa......,, (8) (m) )) In this paper, we fo- indicated below. The KTD framework works under the Gaussian assumption, i.e., 6, follows a Gaussian distribution at each stage 1,2, To address the nonlinearity of the function h(x, 0), the KOVA algorithm, proposed by (Shahhua & Mannor (2020), employs the Extended Kalman Filter (EKF) technique for calculating the mean and covariance matrices of 9g. This approach involves linearizing h(x,0) 0) based on the first-order Taylor expansion, namely:
Qo, (8x, ar) = (s, (1) a(t),, (s. (a("))) cus on Q-functions only, however, our algorithm also works for V-functions
h( 0) RJ
(p ( fi--1,,
where f4-1 denotes the estimator for the mean of f--1. The KOVA algorithm, detailed in Algorithm S1 however, encounters several significant challenges: (i) the approximation accuracy for the true filtering distribution of 0g is unknown; (ii) it exhibits high computational complexity Ompp²) and (ili) it demands considerable memory complexity, necessitating O(p") additional space for the covariance matrix. Alternatively, Geist. & Pietquin (2010) recom- mended the implementation of KTD using Unscented Kalman Filter (UKF). Nonetheless, this alternative algorithm encounters similar challenges to those faced by KOVA, including issues related to approximation accuracy, computational complexity, and memory requirements.
3 LANGEVINIZED KALMAN TEMPORAL DIFFERENCE ALGORITHM
To overcome the limitations encountered by existing KTD algorithms, we introduce an approach that integrates KTD with the LEnKF algorithm, leading to the development of the Langevinized Kalman Temporal Difference (LKTD) algorithm. The LEnKF is a reformulation of the Eusemble Kalman filter (EnKF) (Evensen 1994) under the framework of Langevin dynamics. The LEnKF inherits the forecast-analysts procedure from the EnKF and the use of minibatch data from the Stochastic Gradient Langevin Dynamics (SGLD) algorithm (Welling& & Teh, 2011), making it scalable with respect to the state dimension and the mini-batch size Distinctively, the LEnKF algorithm is designed to converge to the accurate filtering distribution, setting it apart from the traditional EnKF.
3

---

3.1 THE LKTD ALGORITHM
The LKTD algorithm is designed to solve the RL problem by framing within the state-space model outlined in equation (2). As previously explained, this model differs from the model (I) by incorporating the prior information, (d), into the state evolution equation. This incorporation generally enhances the robustness of the algorithm, especially when using deep neural network to approximate the function A(x; 8). Next, we can apply the variance splitting technique (Zhang et al., 2023) to convert the model (1) into state-space model with a linear measurement equation, while allowing the state evolution equation to be nonlinear. The variance splitting technique can be described follows.
Without loss of generality, let's assume that ~ N(0,021n) for each stage t, where In is an n-identity matrix. By the state augmentation approach, we define
0g E
& h( 0s) ues 14 N(0, In),
where & is an n-dimensional vector, and 0 1 is pro-specified constant. Suppose
that 0, has a prior distribution (00) as specified previously, the joint density function of (P: = (0, (7) is given by ((pr) ((0.))((E (r),, where & 0. (t) Based
on Langevin dynamics, we can reformulate 2 as the following model:
Et n
6i = 91-1 +
V. e log (pp-1) + wr
2 N
(6)
rt- = Hisp + Ut,
where N > 0, We ~ N(0, N Bi), B, = alp. p= -p=po + is the dimension of Pe; H, = (0, I,) such that Hep, Eτ; N(0, (1- (1-a)o²Ia), which is independent of w. for all L. We call N the pseudo-population size, which scales uncertainty of the estimator of the system. Refer to Lemma [ and Theorem [ for mathematical justifications for this issue.
By
and 16 Ft, Or. & and r, form a hicrarchical model with the conditional distribution
&i Fr, O2 ~ ~N(ar, + -a---aloe ());; a() (() - I:))Inn)
(7)
which, as shown by Zhang et al. 2023 eventually leads to an efficient particle filtering algorithm for handling the state-space models with nonlinear measurement equation. Similar to LEnKF, we adopt the fbrecast-analysis procedure from EnKF to the model (6) leading to Algorithm [ It works in a single chain, different from particle filtering algorithms. The time complexity of the algorithm is (mp)). This attractive time complexity is due to the special structure of He, rendering the matrix Ke and equation [0) easily computed. Regarding the settings of IC and a, we make the following remarks: First, as indicated by the Kalman gain matrix KG only the É-component of Pr.k is updated at each analysis step. Generally, Ee, converges rapidly, benefiting from sccond-order gradient information. Thus, I. docs not need be excessively large. Based on the property (7), [Zhang et al.] (2023) demonstrated that LEnKF acts as variance reduction version of SGLD il 0.5 a << 1, recommending be set close to In this paper, we default X. 5 and a 0.9, initializing Éroo by r. al each time step t to enhance the convergence of the simulation.
3.2 CONVERGENCE THEORY
To study the convergence of Algorithm D it suffices study the convergence of Algorithm which ignores the inner iterations for imputing the latent variables &, and serves as the
S2
Lemma I shows that Algorithm SS2 is actually an accelerated preconditioned SGLD algorithun. Its proof follows Theorem SI of Zhang et al. (2023)
prototype of Algorithm [
Lemma 1. Algorithm
S2 implements a preconditioned SGLD algorithm, for which
+ C V. log í (1pf + et,
4 = -f-
(11)
2
where = vT) as defined in Algorithom Pt, et ((), and the gradient term
$2
N (T- K.H.) is a constant matrir given log ग ( 24) given by b log (ppp_11=.) =
Σ
N E 1= V log ( (2(1))18__))
(pp_1) + V. log ()p__))
4

---

Algorithm 1: Langevinized Kalman Temporal-Difference (LKTD)
Initialization: Draw Of E IRP drawn from the prior distribution a0)).
for t=1, T do
Sampling: With policy por generate a set of n transition tuples, denoted by
(1) (T), = (ree) 2t Jy=. where z; = (5% aat ,8411 a(41)" and
(5)
(1) (4 correspond to the choices of the Q-function and V-function ot (0( ai -8411
in 41, respectively. for k= -12,..,,
KK do
Presetting: Set B= Rτ = 2(1 - 1, and the Kalman gain matrix Ke, = B. H Be,HH + -Ra)-1
Forecast: Draw & N0 BB,,)) and calculate
f = Pi
kt n V log T (F(,n-1) + Wi
(8)
N
where fi,0 (0 (08-1, T ,TT)T if 1, and the gradient term given by
V V. log T(P[,A-1) = (Yo log π (0,,--1) "9 - (E,, h(vri 0c,k-1))
V 1)), h(a, Or, (e,,000,,-)))
(9)
Analysis: Draw BeR Nn N Rs) and calculate
Pi,k s4,6 + Ke,(rr HoPL, m(,k) = sh, + Ke,e(r,- - -r(.))
(10)
end
end
To establish the convergence of the preconditioned SGLD sampler it suffices to establish the convergence of the conventional SGLD sampler in the context of reinforcement learning by noting the positive definiteness of the preconditioned matrix. Specifically, we have
Σ.
(I (I--KHH))
n H (GH,H, RR))-'Ho)
N
N
which implies E, has bounded positive eigenvalues for all 2 1.
3.2.1 CONVERGENCE OF THE LKTD ALGORITHM UNDER THE ON-POLICY SETTING
With slight abuse of notations, we would prove the convergence of the following SGLD sampler in the RL context:
0₂ = 0 C( + /28
(12)
where e사 (0, Ia), the inverse temperature, and k indexes stages of the RL process. In pursuit of our objective, we introduce Assumption A detailed in the Appendix. This assumption aligns with the conditions outlined in Raginsky ct al. (2077) to demonstrate the convergence of SGLD in simulating a posterior distribution with fixed datasct. To tailor the sampler for RL, a context where the total sample size can be considered infinitely large, we introduce the invariant distribution of 6x, thereby reformulating RL as a sampling problem rather than an optimization problem. As result, the proposed method can perform robustly with respect to the dynamics of the distribution T=2(0x). However, under appropriate assumptions (see Remark 1, correct inference for the optimal policy can still be made based the stationary distribution (0) x exp(-3G(0)), where G(0) = O(N) as stated in Theorem [ Theorem [ and the followed Corollary [ establish the convergence of the LKTD algorithm under the general nonlinear setting for the value and Q-functions. Their proofs are given in Appendix.
size N to prevent the degeneration issue of the
Theorem 1. Consider the SGLD sampler 112 with a polynomailyy
learning rate ER to for some E 1). Suppose the enetronment is stationary and Assumption A1 holds. If (E(G((x-1,zz) g(0x-1) holds for any stage l (1,..., K), B 2 1 my then
5

---

there crist constants (Co, C1, C3) independent of the learning rates such that for all K E N, the 2- Wasserstein distance between μ. and VN can be upper bounded by
Wa(ux, uN) s (12 C2
1 kー-
[(C
2
5Col
1- ))-
co
1-
2
(13)
+ +C1ell 2t
2
5Col
1- T
c0 _ーt
] Cs exp
1 11-1---
BeLs
where ux(0) (0) denotes the probability law of 0k, vx( x exp(-39(0)) g(0)= O(N) is the anti- derivative of g(0), i.e., VeG(0)= g(0), and eLS denotes a logarithmic Sobolev constant satisfied by the ux- In addition, the constants (Co, Cr, C2, C3) are given by
=
Co (No 2(1V (b 2B 1 B²,
my
C 6L7 o 1) C2 = + 2b 2d.
3R
Muka
+ Byro + A b 2 log 3)).
C. (log 11oll + log myB + B1
3
Regarding statistical inference with the samples {0A 1,2 K}. we have the remark:
Remark 1. Let s(0)) be a lest function, which is bounded and differentiable. Suppose that the conditions of Theorem hold and (uvr(0), (()) salisfies the Laplace regularity condition
as given in Theorem 2.3 of Sun et al. 2022). Then, by Lapalce approrimation, we have
[ (00) exp( exp((8((0))B f eex((= = 600)) 0
ex(0)=
as K + 00,
(14)
N
where 8 denotes the marimizer vf) and thus the marimizer of Vox (0), and and denotes the connectivity of the sparse DNN learned for approrimating the value or Q-function. Therefore, when the number of total time steps K becomes large and N ra holds, we can make inference for the policy using the Monte Carlo average & = | by letting K + 00, forms consistent estimator for (00),, the true value of o(-) at the optimal policy.
exo(0x)) enl, which,
Remark 2. The choice of the pseudo population size N reflects our trade-off between optimization and sampling. It acts tempering factor for the system. As N + s0, have ((0) + ao and, consequently, the stationary distribution uN(() degenerates to a della function centered at 0*, defoved in Remark
We note that the proof for the convergence of Ô toward the population mean @x(()) requires that the learning rate sequence satisfies Assumption A2 This condition is readily met by the polynomailly-decay learning rate sequence outlined in Theorem [ Finally, we note that the
conclusions of Theorem [ remains valid for Algorithm [ Therefore, Remark I and Remark
2 also hold for this algorithm.
Corollary 1. The conchusions of Theorem and Remark
remains valid for Algorithm 1
3.2.2 COOPERATION WITH REPLAY BUFFER
In Section 3.2.1
we demonstrated the Wz-convergence of LKTD algorithm under on-policy framework, where transition tuples 2; are determined by the precoding parameter 0:-1- In contrast, off-policy algorithms obtain 2, from replay buffers, strategy that boosts data elficiency and is frequently adopted in Q-learning algorithms. This section is dedicated to establishing convergence theory for the LKTD algorithm when integrated with replay buffer. In practice, the replay buffer B stores transition {2is1/=1 drawn from the stationary distribution (2) at each time step t. Given its finite capacity, the replay buffer retains only the transition tuples from the last R time steps. That is, the replay buffer at time step f, denoted by B,, contains only the transition tuples generated from {x(2)0,))==11 In the population scope, we define the replay buffer time step f, denoted by ([ as

---

mixture of finite number of stationary distributions (z(2)0t-))] be explicitly written as
The replay buffer can Σ x(2(0-=)) R
R
==2
(15)
where 0-- [8 and R E N. Given the buffer's structure, we assume that the samples drawn from it are R-dependent. That is, 2 and 24 are independent for all 2 E Bt, 2y E By For some lest function Φ(0) of interest, we define the posterior average as: Φ (())vx(0)dd where uo( is the target distribution as defined in Theorem Let {0,)²=1 be the samples generated from LKTD algorithm, and the sample average Ø is as defined in Remark [ In Theorem P we show that although the gradient is biased due to replay buffer, the bias and variance of Φ vanish asymptotically. In other words, we can employ replay buffer to improve data-efficiency without losing the asymptotic consistency.
and |L - ['
Theorem 2. Let {0:)1=1 be a sequence of updates generated from LKTD with replay buffer. At each time i, the transition tuple 2, is sampled from the replay buffer =(2,)0 In addition to the assumptions in theorem 1 we further assume the following holds:
(i) (Lápschitz) fz (=|00) =z S L1J0 -1;; (ii) (Integrability) J2 ([((()))))) =)] S M and Jz ||G z) =>|0====0)d== & M, V0 E.
Then for bounded test function o, the bias of the LKTD can be bounded as:
Ei )
B( 5)² 0 ST + Di=
E3 = =0
(16)
ST
ST
87
S
EXPERIMENTS
This section compares LKTD with prominent RL algorithms such as DQN, BootDQN Osband et al. 2016), QR-DQN (Bellemare el al. 2017) and KOVA (Shashua & Mannor 20200. Using an simple indoor escape environment, we demonstrate the advantages of LKTD in three aspects: (1) accuracy of Q-value estimation, (2) uncertainty quantification of Q-values, and (3) optimal policy exploration. Furthermore, by employing more complex environments like OpenAT gym, we demonstrate that LKTD is capable of learning better and more stable policies for both training and testing.
4.1 INDOOR ESCAPE ENVIRONMENT
Figure depicts a simple indoor escape enviromment, for which the state space consists of 100 grids and the agent's objective is to navigate to the goal positioned at the top right corner. The agent starts its task from the bottom left grid at time = O. For every time step t, the agent identifies its current position, represented by the coordinate s = (x, y). Based on a policy, the agent chooses an action by the agent determines the adjacent grid to which it moves. Following each action, the agent is awarded an immediate reward, rı, drawn from the distributtin NN-1,0.01), It's worth noting environment that for most states, the Q-values for actions N and E are identical. This highlights the importance of exploring diverse optimal policies to achieve a consistent and resilient policy, Our experiment showeascs the sampling framework are capable of learning mixed optimal policy in a single run. We compare the proposed sampling framework LKTD against existing RL algorithms like DQN, Boot QR-DQN and KOVA in the training of the deep Q-network. Refer to section A.4.2 for the detailed experimental setup.
(N. E. S. W}. The action taken
Figure 1: Indoor escape
For each algorithm, we collect the last 3000 parameter updates to form a 0-sample pool, denoted by 0, = =(.),, which naturally induces a sample pool Q-functions Q. = (Qa(- 100 O.). We can obtain point estimate of the Q-value at (s,a) by calculating the sample average Q(s,a) 1 Ei= Qo (8) For uncertainty quantification, we can achieve one-step
7

---

LCTO 2500
Action
LCTD: 37 3000
Action N
Algonthin
LCTD
10
20a4
MSBIÓ1
Figure 2: Boxplots for MSE(Q.) (for E)))
0X10 2300
Agorthim
0.4
0.9
Coverage rate
Figure 3: Boxplots for coverage rates (for E))
value tracking by coustructing a 95% prediction interval with the Q-value samples. Due to the simplicity of this environment, the Q-values of optimal policy, denoted by (s,,,a), can be calculated by Monte Carlo simulations. For each algorithm and parameter setting, we conduct 100 runs and calculate two metrics: (1) the mean squared error (MSE) between ((s, a) and (s, a), denoted by MSE(Q.) for each action a, where the average is taken over all states and (2) the coverage rate (CR) of the 95% prediction intervals.
Figure presents a boxplot illustrating the distribution of (sF( (for a E (N. FE) across 100 experiments for each algorithm. Here, we consider only the actions (N, F}, since {S, W are sub-optimal actions at all states and the corresponding Q-values cannot be well approximated due to the lack enough transition tuples on them. Figure 2 indicates that LKTD yiclds notably higher Q-value estimation accuracy compared to all other algorithms. Morcover, the plot shows a clear trend that as the pscudo population size A grows, their accuracy correspondingly improves. For uncertainly quantification, Figure 3 shows that the coverage rates from the LKTD algorithm is close to the nominal 95% and independent of the choice of pseudo population size, whereas the DQN, BootDQN and KOVA algorithms fail to construct correct prediction intervals. Although QR-DQN approximates the distribution of the Q-function, does not provide the correct interval estimation for Q-values. These obscrvations on LKTD align well with the point we made in Remark 2
Effective policy exploration crucial for RL agents it empowers them to adeptly learn various optimal policies and navigate challenges like the local-trap issue. In this specific environment, the Q-values for actions N and are indist indistinguishabl.. Therefore, an algorithm excelling in policy exploration should, during training, give equal consideration to both optimal actions. To quantify policy exploration, we introduce the concept mean policy probability. For given policy pool g at state s, it's defined as:
(a(s))) = Σ (a(p)))) PEB
where e represents the policy pool derived from the O-sample pool obtained in run of the algorithm. In simpler terms, the mean policy probability measures the frequency of an action chosen by the policy sample at specific state. Figure that LKTD effectively explores the optimal policies across the majority of grids, whereas DQN sticks to one optimal policy, failing to explore others. This implies that DQN tends be locally trapped in RI, compared to LKTD.
From a computational aspect, the LKTD algorithm stands out for its efficiency and scalability, As detailed in Table A3 we have recorded the average computation time required by each algorithm to execute a single parameter update. The findings indicate that LKTD scale effectively in relation to network and batch size. Their time complexities align closely with that of DQN. Conversely, the KOVA algorithm, due to its reliance on the calculation of the Jacobian matrix and matrix inversion, proves to be computationally less efficient.
8

---

(a) Optimal solution
(b) LKTD
(c) DQN
Figure 4: Mean policy probabilities for the indoor escape environment: (a) known optimal
solution; (b) learned by LKTD; (c) learned by DQN, failing to explore different policies.
4.2 CLASSICAL CONTROL PROBLEMS
This section evaluates LKTD's performance on four OpenAl gym challenges: Acrobot-vl, CartPole-vl, LumarLander-v2, and MountainCar-vo, comparing it against DQN and QR- DQN based on RL Baselines3 Zoo (Raffin 2020 training framework. We conducted 100 replicates per experiment, with results in Figures 3 and A4 excluding the top and bottom 5% outliers. Mean reward curves are shown with solid lines, and a 90% confidence interval is indicated by the colored regions, showcasing LKTD's efficient exploration and robustness. For experimental details, see section A.5
Across all Lested environments, LKTD consistently surpasses DQN and slightly better than QR-DQN. LKTD's ability to achieve markedly higher training reward indicates its capacity to learn more robust policies against sub-optimal actions generated by random exploration. This superiority is particularly pronounced in sensitive environments like CartPole-vl (see Figure 5), where even single misstep can lead episode termination. Even when discounting The noise from random exploration, LKTD's evaluation rewards remain superior to DQN's. Additionally, when it comes to optimal policy exploration, LKTD can identify a more optimal model within equivalent computational timeframe DQN, highlighting its efficiency in policy exploration.
v1
tra ning reaned
evatustion reward
best model
000
MMM
2.9
100
DOW
400
064
Figure 5: CartPole-vl: The left plot shows the cumulative rewards obtained during the training process, the middle plot shows the testing performance without random exploration, and the right plot shows the performance of best model learned up the point
CONCLUSION
In this paper, we present novel sampling framework designed to enhance SGD optimizers for addressing deep reinforcement learning challenges. By redefining the state-space model and introducing a pseudo population size, we enable SGMCMC algorithms, such LKTD and SGLD, to converge to the accurate posterior distribution under mild conditions. Our approach outperforms existing value-basod algorithms in benchmarks. Specifically, our L.KTD algorithm demonstrates greater computational efficiency and circumvents potential matrix degeneration issues by eliminating the need for linearization, unlike the KOVA algorithm. Compared to DQN and its variants, our framework not only provides more precise point estimates of Q-values but also generates accurate prediction intervals for value tracking. In gym environments, LKTD surpasses DQN and QR-DQN in robustness and efficiency of optimal policy discovery. Overall, our framework signifies a significant advancement deep RL optimization, offering improvements in both efficiency and precision.
9

---

REFERENCES
Brian. D. O. Anderson, John B. Moore, and Mansour Eslami. Optimal filtering. IEEE Transactions on Systems, Man, and Cybernetics, 12:235 236, 1979.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on rein- forcement learning, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606. 000.000000 2016.
Changyou Chen, Nan Ding, and Lawrence Carin. On the convergence of stochastic gradient meme algorithms with high-order integrators. In Advonces in Newral Informalion Processing Systems, pp. 2278 2286, 2015.
G. Evensen. Sequential data assimilation with a nonlinear quasi--gesstropple model using monte carlo methods to forecast error statistics. Journal of Geophysical Reseorch: Oceans, 99(C55)1014331101))) -111111 1994.
Matthieu Geist and Olivier Pielquin. Kalman temporal differences. J. Arlif. Inl. Res., 39(1): sep 2010. ISSN 1076-9757.
Junqi Jin, Cheng-Nine Song, Han Li, Kun Gai, Jun Wang, and Weinan Zhang. Real-time bidding with multi-agent reinforcement learning in display advertising. Proceedings of the 27th ACM International Conference Information and Knowledge Management, 2018.
Rudolf E. Kalman, A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82:35-40 45, 1960.
Petar Kormushev, Sylvain Calinon, and Darwin Gordon Caldwell. Reinforcement learning in robotics: Applications and real-world challenges. Robotics, 2:122-148, 2013.
Chunyuan Li, Changyou Chen, David E. Carlson, and Lawrence Carin. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In AAAI, 2016.
Yi-An Ma, Tianqi Chen, and Emily B. Fox. A complete recipe for stochastic gradient meme. In NIPS, 2015.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https: https://proceedingg www.eene..ipp.eeee cc/ wwwppppr.pppppccceee hatppprrfieeeeppppp////////// pdf
Antonin Raflin. baselines3 2oo. 2020.
Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis. In Proceedings of the 2017 Conference on Learning Theory, pp. 1674-1703, 2017.
Shirli Di-Castro Shashua and Shie Mannor. Kalman meets bellman: Improving policy evaluation through value tracking. ArXiv, abs/2002.07171, 2020.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, L. Sifre, George van den Dricssche, Julian Schrittwicser, loannis Antonoglon, Vedavyas Pannecrshelvam, Mare Lanetot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484 48944 2016.
Yan Sun, Qifan Song, and Faming Liang, Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association, 117:1981-1999 2022.
10

---

Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9-44, August 1988. URL http://www cs. www.eaalllaaaaeeaaaaaaaaaa hwpppttaaccc--attttoo//aapr///// sutton-88.pdf
Richard S. Sutton and Andrew G. Barto. Reinforcement Leurning: An Introduction. The MIT Press, second edition, 2018. URL html
2nd httpp//coo...eeeeeeeeeeeeee////e
Charles Edison Tripp and Ross D. Shachter. Approximate kalman filter q-lcarning for continuous state-space mdpss. ArXiv, abs/1309.666 2013.
E.A. Wan and R. Van Der Merwe. The unscented kalman filter for nonlinear estimation. In Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Sympostum (Cat. No.00EX373), pp. 153-158, 2000. doi: 10.1109/ASSPCC. 2000.882463.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. (rXi::170.006116 2018a.
Zhe Xu, Zhixin Li, Qingwen Guan, Dingshui Zhang, Qiang Li, Junxiao Nan, Chunyang Liu, Wei Bian, and Jieping Ye. Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018b.
Peiyi Zhang, Qifan Song, and Faming Liang. A langevinized ensemble kalman filter for large- scale static and dynamic learning. Statislica Sinica, 2023. doi: 10.5705/ss. 20202222..
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning, 2020.
11

---

A APPENDIX
A.1 EXTENDED KALMAN TEMPORAL DIFFERENCE ALGORITIM
Let Ex denote the estimator for the covariance matrix of 0.. Additionally, for equation we let W, E _exp denote the covariance matrix of the Gaussian noise W2, and let T. E R denote the covariance matrix of the Gaussian noise tt- The resulting KTD algorithm is
Algorithu S1 given in the Appendix. The major issues with Algorithm SI are (i) unknown
approximation accuracy: it is unclear how well [( Σ.,) approximates the true distribution of 0;: (ii) computational complexity: it is ((pp²) per iteration; and (iii) memory complexity: it requires Og1²) additional space to store the covariance matrix Σ.
Algorithm S1: Extended Kalman Temporal Difference Algorithm (KOVA Algorithm);
Shashua & Mannor 2020
Initialize Ag;
for t=,,2..., do
(i) Set predictions: f = p P and Enje11 = Et-1 + W6:
(fi) Generate transition tuples (re, h( [}} from the system via the
agent-environment interaction;
(iii) Calculate (p n)-dim matrix Voh(a,,, A)) and the Kalman gain matrix
Kr - Vah(a, A)1."'
where r = Voh(ar, E Voh( A) + T+; (iv) Update the mean and covariance matrix estimators:
cu Puje- + áKi(r: h(xx, Paje-1), = E āK,I,,
where ā the learning rate.
end
A.2 THE PROTOTYPE OF THE LKTD ALGORITHM
By ignoring the detail of state augmentation, the model 6 can be simulated using Algorithm
$2 which is the single-chain version of Algorithm 2 of Zhang et al. 22023
Algorithm S2: Prototype of the LKTD A Algorithm
Initialization: Start with an initial parameter sample p8 RP, drawn from the prior distribution T(p);;
for t=1,2...,," do
Presetting: Set B, celp, R₁ = 20 In, and the Kalman gain matrix K, = B,H, (H,B,H, + +Re)
Sampling: With policy PPt_, generate a of n transition tuples from the
stationary distribution Pert_ denoted by z = (r,, xx) = { )=
Forecast:
Draw (2, 2 B.) and calculate
ol pi-1
6 v log uz.
(A1)
2
Analysis:
Draw 0, (0, #x4 R,) and calculate
o = of + KK Hard - vz) = of Ki(ri -r/).
(A2)
end
12

---

A.3 PROOFS FOR THEORETICAL RESULTS
A.3.1 ASSUMPTIONS
Assumption A1.
(C1) For any E θ we are able to generate tuples z from a unique stationary distribution =<2(0), the function G: ::0% x Z is measurable, and [1g(0)]l (20,)) < 20.
(C2) There evists a function G( which is an anti-derivative of g(0) with respect to 0 i.e., Vog(O) g(),, such that [g(0)] s A for some constant A 0; in addition, there exists some constant B > such that [|ş(O)]I s B.
(C3) There erists some constant Lu > 0 such that
ll9(0) .(0) - Lu 10 U|,, VO, 0 E e.
(C4) The function (()) is (mu. b) dissipative; that is, for some my > 0 and b 2 0,
(0.)(0)) 2 mo b, 40 E O.
(C5) There exist consland 8 and some conslants My and B such that. El|G(0,z) 2(0) = 20( |01 + B²), e. where the expectation is taken with respect to 2 ~ x=2()).
(C6) The probability law Po of the initial hypothesis Og has bounded and strictly positive density po with respect to the Lebesgue measure O, and
Ky := log /
e110 Po (0)dd <0.
In particular, the condition (C4) is quite standard for establishing the existence of an invariant distribution for 0, see e.g. Raginsky et al. (2017) and Xu et al. (2018a). It intuitively indicates that the dynamics stays inside bounded domain in high probability; if O2 is far away from the origin, then the dynamics are forced to get back around the origin.
Assumption A2. The learning rale sequence {Et} is decreasing, i.e., Ex41 < cks and satisfies that
Σ
20
ck 0o; 1)
lim
Ek=1 c =0
x Lk=1 Ek
A.3.2 PROOF OF THEOREM
Proof. The proof of Theorem (ollows from Theorem 2 of Zhang al. (0020). For convenience, we use t and k to index the continuous-ting and discrete- time respectively. Firstly, we consider the following SDE
d0, = -00( + V2aW,
(A3)
where ((0) f G(O, (z)(2))) Let de denote the distribution of 0,, and the stationary distribution of [A3b is denoted by Moe.
oc+1 0z C++1G(AR, 2k+1) v 2014118-1
(A4)
Further, let b4 denote the distribution of 0g and S₁ Σk= 64. Since
W2(uK, Px0) W2(uK vsx) + W2(Usx Moo),
(A5)
we need to bound these two terms respectively.
For the first term, Wz(KK V9,x), ) our proof is based the proof of Theorem in Zhang et al. (2020) with some modifications on learning rates. By definition, if 2 is sampled Trom the stationary distribution лt (=)0k), then G(Ok, #k+1) is an unbiased estimator of g(0x),
13

---

i... [[G [Ful g(0k), vOk θ C Rd. And we define p(t) which will be used in the following proof:
p(t) = {K Z|S, St Sk+1
(A6)
Then we focus on the following contimuous-tin time interpolation of 02:
((1) = 0o 1'" ((((Sp(e)))= (p00+1)ds + VB 1, aw!
12
(A7)
where G Gk for E [Sk, Sk+1). And for each k, ((Sk) and 0g have the same probability
law Pk- Since ((t)) is not Markov process, define the following process which has the same one-time marginals as ((t)
V(I) 0o ! (.((
V
2 V3..
aw!"
(A8)
with
(x)x) = E ((((( 21(1)+) [0(
(A9)
Let Pt- [[V(s): and P == (()(s) s S t) and according to the proof of Lemma 3.6 in Raginsky et al. (2017) we can derive similar result for the relative entropy of P. and p/
dP log dP? dP!
DxL (Pr =
8
I Bllg(V(ss) H.((
(A10)
4
8
/ Bl(g(((s)) (O(s))))
The last line follows the fact that (()(s)) (s)(s)) As. Then will let t E E and we can use the martingale property of the integral to derive:
Eme Et iP
EE
K-1
8/12
DK
p
Blls(((s)) H.(((s)))
1=0
K
Bllg([(s)) e(((s))))
2 1=0
8
S1+1
+
El(g((())) H. (((S))))
1=0
3L3
K-1
S/+1
B|(0(s) ((S))
(A11)
2
/=0
K-1
84+1
+
El(g(((s))) - (.(,
(A12)
Sq
1=0
For the first part A11 we consider some
[S,, Stt1, for which the following holds:
((s) 0(S,) = -(s - S,)G(Ok, 2k+1) + V (w!!)
2
- -
-
- - S))g((k)) + + (S-)))(((O)) -G(0,, x)) + v /2 (m - -W(())
(A13)
Thus, we can use Lemma 3.1 and 3.2 in Raginsky et al.
2017 for the following result:
E [((((s) (( < 3l) || 3e (0
G(O,, 2)+1)11 6Ef+1d
6E 6
(A14)
< 12 1 B²)
+ 0
14

---

Hence can bound the first part, (choosing 6o $ 1),
K
S5+1
L1
K
1
Gc j+1d d
[l) - 2(S)))
<
12 (1 B3) +
2
2
B
j ==0
5
1=0
K
3d
S 1
9+1
05isK-1
max
[6 0, 1| B²) +
B
j=0
(A15)
2 max 0SISK-1
3d
S 1
2
R0, + B²)
(A16)
3
The second part A12
can be bounded follows:
K
85+1
K-1
Bl(g(((s))) - -H,(0()))
Σ
()11B19(0,) - -G(0), 23+1)11
2
j=0
S
2 j=0
S 65k max
10 + B²)
0sišK-1
S 5c + log( K))
max 0SiSK-1
B + B²)
Due to the data-processing inequality for the relative entropy, have
DKL DKL(ux [[Sx)
S (PV
K
1
Si-11
K-1 1
S511
E
(s) - B(S))) ds
Σ
Blgg(((S))) H.(sss))))
2
1=
5
j=0
S;
K-1
3d
Li
(
2)+1)
max 0SisK-1
[6(L
BD)
88k
max
(13,,10,1 B²)
B
0Si5K-1
j=0
2
3d
$ Li 60
max
[6 E|0,11 212 + B²) +
2T
USJšK-
3
1
+ seol
K1
max 0sisK-1
(L B²)
1
According to the proof of Lemma 3.2 in Raginsky et al. (011 can bound the term 1|
26k+1d
B|10++112 s 2c + 422 E0O + 2cx+1d 422,, B² +
B
Similar to the statement of Lemma 3.2 in Raginsky et al. (2117) can fix eu (0,1A my 1 ). Then, we can know that
IMg me
d 3
0 2| (1 (4+1m)))E000 + 2 (b + 2B 2
(A17)
where Eσ is the minimum of the decreasing learning rate sequence. There are two cases to consider.
If 1 2cgmm + 4c2 M2 < 0, then from A17 it follows that
E 0 1 < 2co + B² + d B)
< E|10011 + 2(b + B² + d
3
15

---

If 0 0 -2ckmu + Mc 1 1, then iterating A177]
gives
E10,11 < (1 - 6 s, 1|2 2z (b + 2B +
B
d B
e - E E
2c 2B²
Se me E 0nl1 + + + 2B2 + d
Σ
10w0 (8)
i=1
<1110,112
2B2 + 2(6 +
mu
6e m
Sk mareo
$ <B/60 + 2(b + 2B2 +
St.
e dx
1
12|00/12 2(b 2B +
mu
muro
my
my
em
E|100 2 + 2(6 +
2B²
+
a YR
s 1 12 2(6 2B² +
Now, we have
max 0SJSK-1 1 (13
+ BB2 s (T2 (No 2(1 _) 2B² + ()) + B3) = Co
my
We denote the 6I2 4 C1 and derive
2 DKL S Ciei( 2 dCoeal K K1-0)
Then according to Lemma 3.3 in Raginsky et al. 2017) if we denote + 26 + 2d as C2 can derive the following result:
WalKK,Vs, S (12 C2Sk)* [D DKI(HKIlUS )*1
2
1 K 1)
(12 + C2ea(
1- K ))
2 1) + -SCoool
2w
(1 2
+ 6Cocol
1- K K1--))a]
2w
Now we derive the bound for W2(USx Moo) By following the results in Raginsky et al. 2017 that there exist some positive constants (C3,e1s).
SK
Wa(vsx,Po) C3 exp
CLS
A.3.3 PROOF OF COROLLARY I
Proof. By Theorem of Ma et al. (2015). Algorithm | works pre-conditioned SGLD algorithm with the pre-conditioner Σ, and it has the same stationary distribution as the SGLD algorithm (12). By (13). we have the 2-Wasserstein distance convergence for algorithm 12 under the given assumptions. Therefore, for Algorithm we also have W2(PT, V/x) + as by noting that E, is positive definite for any
16

---

A.3.4 PROOF OF THEOREM 2
Proof. By following the proof in Chen et al. 20015 we define the functional ei that solves the Poisson Equation:
(0,) - (0,) -
(A18)
And & satisfies the following smoothness condition
Assumption A3. v and its up 3rd-order derivatives, D are bounded by function V, i.e., ||D* I VR for k = (0.1,2, 3), Ck. Ps > O. Furthermore, the expectation of V on {0r} is bounded: supse(0.1) (P + (1 s)Y) C(V + V*(Y)), YO, Y. S max for some C 0.
Assumption A4. Let Et be the preconditioner, and assume that d S inf Amin (Ex) S supx Amax (Ex) >t,u for some de and Ae, where Amax(.) and Amin() demote the largest and smallest eigenvalues, respectively.
First let denote
: -,)0,,()-, Vo + + SEE V.V.
(A19)
2
the local generator of LKTD with replay buffer, where a b :- a b is the vector inner product, A: B = tr ( B} is the matrix double dot product. Furthermore, let L be the true generator of the LKTD without replay buffer, that is, replacing the stochastic gradient in &, with the true gradient. As a result, we have the relation:
i= C AVE.
(A20)
where AV, = (G(0--1,2) 9(fe-1)) EV,,, where g(0,-1) J2 2)0 )d2 is the true gradient, and G(O,-1, ze) is the stochastic gradient calculated using transition tuples sampled from the replay memory. By following the proof of Theorem 1 in Li et al. (2016) can derive the estimation error as follows:
5- Ex(0,) - (00) +
1 ST
Σ
(B6(0)-)- +
Σ AVA [=1
T
&
0 ST
Et= o
ST
ST
(A21)
By taking expectation on both side, we derived the bias as:
E c |EVV|
Et= 2
BB Φه 0 0 ST
(A22)
ST
ST
To prove the consistency of ô, we need bound the term 1 ΣT= & |BAV/l- By Assumption ST
A3 and A4 it is sufficient to prove that ΣT et ||ll is bounded, where Cr = G(O,-1, 2))
g(O,-1) is the gradient bias at time Let g(0kL)) = -I2 G(00,1,, () 0 (z ddz be the
expectation of the biased gradient given the replay buffer =[= By applying assumption and (ii), we can bound the conditional expectation of the gradient bias
[ = 11) g(0r-1)l1
< 1| G(0-1122) (T(2)0 (=(0
[IG(O,-1,)] [+(2 <<2/6, (1)]6,11)lm dz)²
Z
/ |=(=(0_1 (=(00-
<
(G(0,-1,2)]12
(A23)
8
R
1
$ M
(T(2(0)1) (2110-1))
R
R
ML
<
10-- - 0,l1
R
i=1
17

---

By Jensen's inequality,
|B
MI
R
$
1 - 0,1
R
R
(A24)
B - 02 12-1
R 1=1
R
0
1-1 0
R
Now, we want bound the expected square difference between parameter updates
E 00
01-1 G(0-,,)) + + 2
- 1e G(0, + 4e, |Ee/
(A25)
< 2 G(0,,0 =1]1 + 4e Aumax
Combine the results in A24 and A25
we have
[G]]]l
R Σ
1) 11-
-
R ΣΣ 1)0(E2)))
(A26)
R 0(+
< (( = =0(e)
By A26).. the operator norm |EVV|l is of order O(a)) Combining this result with A22 we can derive the bound for the bias
[B - =
e
IEVVV
ΣT
c
ST
ST
ST
2
2
01
+
-
21=1 9
(A27)
ST
ST
Sr
21=
0
ST
ST
Now, consider the 2 convergence of L. Since EAV, is nonzero under the setting of replay
buffer, we follow the proof of Theorem 3 in Chen et. al. (0055) with some modification. The MSE of & can be written as
(Ba)(O.) (lo)))
EL (E.-)),)) (0(-)))
T
B(Φ- 3)² < CE
(Vio)))))))
S2.
S7.
Sr
-1= c
+ C
Sr
(A28)
Let AV, EAV, + 6, Vo, where 6, = (G(()-1,.)) 0 1) has mean 0. Since 's are R-dependent due to the structure of replay buffer, Cou 6,, dz) = 0 for all |t 1t-ft f'| > R. By assumption (ii), 6 is bounded. Hence, we derive the following bound:
18

---

T
T Σ
T
E|
EL
AV 21
EL
E + 2E
Σ
EL
d, voll
ST
ST
ST
1=1
1=1
T
2 2(
|AV]
+
Σ
EEE Cou(dr, dr)
S
1=1
1=1
|1-F|KR
(A29)
C 2
RΣ 21166
0
ST
ST
T
Σ1
E
0
+
5
S7
Finally, we can derive the MSE as
E(O
3) 0
-1=1
T
2
-1
T
(A30)
ST.
ST
ST
A.4 MORE NUMERICAL RESULTS
A.4.1 IMPLEMENTATION OF THE SGLD AND SGHMC ALGORITHM
The notion of pseudo population introduced in the proposed LKTD algorithm can also be applied to SGLD and SGHMC algorithm. As implied by Lemma D can directly implement equation 111) with E, being restricted to an identity matrix, which leaves the same stationary distribution. The pscudocode of SGLD and SGHMC are given in Algorithm S3 and Algorithm S4 where is set to match the computation LKTD.
Algorithm S3: SGLD for RL sampling framework
Initialization: Draw 8ß I drawn from the prior distribution x(0).
for 1=1,2...., T do
Sampling: With policy por generate a set of n transition tuples, denoted by
(r1, = (1(2), 2t (i) Ji= where x.(') (s oz ,141 a(41)2 and
(
2.621 = (s( (((5)a (3) oT correspond to the choices of the Q-function and V-function
in 4 respectively. for 1= 1,2,... IC do
Presetting: Set BiG = E1,.p. Draw widb (0, N Br,k) and calculate
01, 01.
6LA N
Vo log (0t,a-1 ue
(A31)
where 6,,0 = 0:-1K if 1, and the gradient term is given by
1 v Voh(a 0, () M(x,;;,, 61,
V8 log 7 (0t |z)) = V log 7 (0c
0² n
(A32)
end
end
A.4.2 INDOOR ESCAPING ENVIRONMENT
This section serves complement to Section in the main text, offering more comprehen- sive experiment settings and numerical results to compare SGMCMC sampling algorithms with non-sampling algorithms. The SGMCMG sampling algorithms considered comprise
19

---

Algorithm S4: SGHMC for RL sampling framework
Initialization: Draw 0g E IP drawn from the prior distribution (0), momentum coefficient
for 1=1,2...., T do
Sampling: With policy po-_,> generate a set of n transition tuples, denoted by
(r), = (1()) 24 10 1i=1 where x! (5() a ,811 at41 and
( (5)
[.( -(s(
(3) , .8411) correspond to the choices of the Q-function and V-function
in 4 respectively.
Set 00
for 1,, K. do
Presetting: Set Bi, C.,.Ig
Draw Wi ~ (0, a 8 B1,b) and calculate
Ut = (1 Ut
6t.k
Vo log (0,.. + 0e
2 N
(A33)
01, = 81 + 1e.k
where 0r,0 = 1 if k: = 1, and the gradient term is given by
1 1N Vo 1oggo(00 = V log (ou + o² n
Veh(z 0c 1) h(xx; 0c,x-1))
(A34)
end
end
LKTD, SGLD, and SGHMC, while the non-sampling algorithms encompass DQN, BootDQN, QR-DQN, and KOVA.
In this experiment, the Q-function is approximated by deep neural network with two hidden layers of sizes (32, 32). The agent updates the network parameters every 10 interactions, for a total of 10 action steps. The replay buffer size is set to 10 For action selection, we
use e-greedy exploration with a final exploring rate of e = 0.01. The batch size is 100. To achieve sparse deep neural network, we follow the suggestion in Sun et al. (0222) let the deep neural network parameters be subject to a mixture Gaussian prior:
0 ~ (1 (1- - 1)W(O, o + x)
(A35)
where À E (0, is the mixture proportion and of is usually set to small number compare to of. We set 01 = 0.3, og 0.05 and À = 0.5 in all SGMCMC simulations. In equation (2) the reward r, is assumed to be Gaussian distribution with variance o². For indoor escape environment, the reward is given by (V11,.000 that is, we set 0² 0.01.
For BootDQN, the number of 'heads' is configured to 10, with the Bernoulli probability set at 0.5. For QR-DQN, the return distribution is approximated by 10 quantiles.
For this problem, the optimal policy is not unique, any policy that choose either action N or action at any inner state is an optimal policy. Despite that there are multiple optimal policies, they all share the same Q-uunction, denoted by (" Since adopt e-greedy exploration, we re-denoled (:),,)) by (:()) to indicates its dependence on the e-greedy exploration strategy. For all state-action pairs (s, a), the Q-value (s, a) can be estimated by Monte Carlo simulations. Note that Q-(-.)) .) is the target function that the deep neural network is approximate.
For cach algorithm, we collect from the last 3000 paramcter updates to form f-sample pool, denoted by 0, {6:}, which naturally induces a sample pool of Q-functions Q (Q O.). We can obtain point estimate of the Q-value at (s, a) by calculating the sample average Q(+, a) = + Σ Qg (s, a). For uncertainty quantification, we can achieve one-step value tracking by constructing a 95% prediction interval with the Q-value sample pool.
20

---

Arioo
LATE:: 1000C
SOLD PM00
SCLD 5000
Agurttin
G2IMC 5002
GCAN0 Ki- Inne
RJAR
10.
10
KSE(Ó)
Figure A1: Boxplots for MSF(Q.) (for E (N, EJ)
5002
LKTD: x' 1000C
SOLD AS 2500
SCLD AC- 5000
Algorttrm
SOLD AD
G2IMC 1000c
DON
BoNDON
0300K
0.2
0.5
Coverage rate
Figure A2: Boxplots for coverage rates
For each algorithm and parameter sctting, we conduct 100 runs and calculate two metrics for each action at each run: (i) the mean squared error (MSE) between Q(x, and (7 denoted by MSE(Qa), where the average is taken over all grids, that is,
MSF(Qu) 1 Σ ()( Q2
S
SES
and (ii) the coverage rate (CR) of the 95% prediction interval of (s, a), that is, the probability of (*, falling inside the prediction interval. Figure Al and Figure A2 show the boxplots of MSE(Qu) (with a E (N, F}) and coverage rates, respectively. In Figure A1 the SGMCMC algorithms exhibit significantly smaller MSEs compared to other algorithms. It worth noting that as the pscudo population increases, the MSE decreases, which supports the theoretical result in Remark [ As shown in Figure A2 the coverage rates of all SGMCMC algorithms achieve the nominal 95% and independent of the choice of pseudo population size, whereas the DQN, BootDQN and KOVA algorithms fail construct correct prediction intervals. Although the QR-DQN algorithm achieves a slightly higher coverage rate than DQN, the results prove that it does not converge to the correct return distribution.
In Table A1 we have recorded the trimmed mean (standard deviation) of MSE(Q.) (for E (N. ET) over 100 runs, where trimmed means are calculated by excluding the outliers. The outliers are the values that falls outside the interval (Q1-1.5IQR, Q3+1.5IQR), where Q1 and Q3 are, resepctively, the 1st and 3rd quartiles of the samples, and IQR = Q3 Q1. Both tables indicate that SGMCMC algorithms are more accurate than non-sampling algorithms in Q-function approximation. Regarding uncertainty quantification, Table A2 presents the trimmed mean of the coverage rates and lengths prediction intervals. is worth noting that the prediction interval shrinks the pscudo population size increases, which aligns with our thcory as mentioned Remark
From computational aspect, the LKTD and SGLD algorithms stand out for their efficiency and scalability, compared to the existing tracking algorithm KOVA. As detailed in Table A3 we have recorded the average computation time required by each algorithm to execute
21

---

5000
UKTD A 1000C
SOLD AC 2500
SOLD A 5000
Algortt
SOLE AD
MCTIMC 1000C
DOW
BANDON
014000
0.
Interval range
Figure A3: Boxplots for the range of prediction intervals
Table A1: Trimmed mean of (SE(Q.) (a E (N, over 100 runs
Algorithm
N
East
North
LKTD
le-5
2500
0.00031
(0.00018)
0.00035 (0.00026)
LKTD
le-5
5000
0.00023
(0.00015)
0.00023 (0.00015)
LKTD
1e-5
10000
0.00020
(0.00012)
0.00022
(0.00013)
SGLD
le-5
2500
0.00029
(0.00016)
0.00031
(0.00016)
SGLD
le-5
5000
0.00021
(0.00010)
0.00021
(0.00013)
SGLD
le-5
10000
0.00019
(0.00012)
0.00018
(0.00010)
SGHMC
le-5
2500
0.00020
(0.00014)
0.00021
(0.00021)
SGHMC
1e-5
5000
0.00016
(0.00009)
0.00017
(0.00011)
SGHMC
1e-5
10000
0.00016 (0.00010)
0.00016
(0.00010)
DQN
1e-3
0.10890 (0.14942)
0.08630 (0.10870)
BootDQN
le-3
0.11200 (0.18036)
0.08757 (0.14150)
QR-DQN
lc-2
0.00635 (0.00453)
0.00583
(0.00342)
KOVA
0.00584
(0.00812)
0.00533
(0.00771)
Table A2: Trimmed means of coverage rates and prediction interval widths over 100 runs
Description
East
North
Algorithm
.V
CR
Range
CR
Range
LKTD
le-5
2500
0.94419
(0.00946)
0.35163
(0.03169)
0.94394 (0.01004)
0.35271
(0.03257)
LKTD
To-5
0.94458
(0.00952)
0.32217
(0.03195)
(0.00989)
0.32079
(0.03223)
LKTD
1e-5
10000
0.94577
(0.01003)
0.31259
(0.03316)
0.94537
(0.00979)
0.31198
(0.03327)
SGLD
le-5
2500
0.94411
(0.01316)
0.38984
(0.02707)
0.94474
(0.01141)
0.38938
(0.02712)
SGLD
1e-5
0.9-1589
(0.01191)
0.35993
(0.02893)
0.94618 (0.0092:1)
0.35930
(0.02813)
SGLD
lo-5
10000
0.94636
(0.00878)
0.33496
(48)
0.94648 (0.00906)
0.33559
(0.02167)
SGHMC
le-5
2500
0.94633
(0.00890)
0.34193
(0.02125)
0 0.94578 (0.00931)
0.34144
(0.02140)
SCHMC
Io-5
5000
0.94704
(0.00854)
0.33553
(0.024439)
0.94622 (0.00885)
0.33586
(0.02458)
SGHMC
1e-5
10000
0.94682
(0.00893)
0.32756
(0.02000)
0.94659
(0.00874)
0.32659
(0.01990)
DQN
le:3
0.41132 (0.20317)
0.23791 (0.05742)
0.41142 (0. 19289)
0.23736 (0.06317)
BoodDQN
le-3
0.37996 (0.18066)
0.19207 (0.04146)
0.39634 (0.19053)
0.18339 (0.02263)
QR-DQN
1e-2
0.85690 (0.07660)
0.40063
(0.05326)
0.86395
(0.06097)
0.39800
(0.05206)
KOVA
0.21133 (0.15191)
0.04756 (0.02531)
0.25709 (0.15602)
0.04987 (0.02856)
22

---

single parameter update, utilizing 4-core AMD Epyc 7662 Rome processor. The findings indicate that both LKTD and SGLD scale effectively in relation to network and batch size. Their time complexities align closely with that of DQN. Conversely, the KOVA algorithm, due to its reliance on the calculation of the Jacobian matrix and matrix inversion, proves to be computationally less efficient.
Table A3: Computation time for the indoor escaping example
Algorithm
hidden layer
batch size
gradient steps (iterations)
cpu time (x10-3)
time per iteration
LKTD
32, 32
100
5
6.63
1.326
LKTD
32, 32
200
5
7.36
1.472
LKTD
64, 64
100
5
7.43
1.486
SGLD
32, 32
100
5
7.15
1,430
SGLD
32, 32
200
5
7.44
1.488
SGLD
64, 64
100
5
7.36
1.472
SCHMC
32, 32
100
5
7.47
1.494
SGHMC
32, 32
200
5
8.25
1.650
SGHMC
64, 64
100
5
8.08
1.616
DQN
32, 32
100
1.80
1.80
DQN
32, 32
200
2.32
2.32
DQN
64, 64
100
1.86
1.86
BootDQN
32, 32
100
2.29
2.29
BootDQN
32, 32
200
2.68
2.68
BootDQN
64, 64
100
2,26
2.26
QR-DQN
32, 32
100
2.41
2.41
OR-DQN
32, 32
200
2.95
2.95
QR-DQN
64, 64
100
2,51
2.51
KOVA
32, 32
100
44.20
44.20
KOVA
32, 32
200
87.00
87.00
KOVA
644 64
100
251.00
251.00
A.5
CLASSIC CONTROL PROBLEMS
This section evaluates the performance of LKTD on four classical control problems in OpenAl gym [Brockman et al.) 2016), including Acrobot-vl,, CartPole-vl, LunarLander-v2 and MountainCar-vO, We compare LKTD with DQN and QR-DQN under the framework of RL Basclines3 Zoo (Raffin 2020). The detailed hyperparameter setting is listed in Table A4 and Table A5 Each experiment is duplicated 100 times, and the training progress is recorded in Figure and Figure A4, At cach time step, the best and the worst 5% the rewards are considered as outliers and excluded in the plots. Due to the adaptability of our sampling framework, LKTD can be easily applied to DQN algorithm by modifying the state-space model in equation 2 as
0:1
+
El V 2
log x (0.-1) w/
(A36)
y: h(w,, 0z) + T),
where h(#;, 0,) = [Qa, (st,1, ir..)...., Qo, (st, ar,n)l and y, rt + (Q,, (8141, suitable constraints on the semi-gradient, we can modify Theorem [ to guarantee the convergence. In the four classic control problems, LKTD shows its strength in efficient exploration and robustness. In Figure A4, the lines represent the mean reward curves. For each algorithm, the colored area covers 90% of the reward curves. We consider lypes of reward measurements, training reward, evaluation reward and the best model reward. Training reward records the cumulative reward during training, which include the c-exploration errors. Evaluation reward calculates the mean reward over 5 testing trails at 100 time point throughout the training progress. The best evaluation reward records the performance of the best learned model.
ac+1). With
23

---

(a) Acrobot-vl
Acrobot-v1
briiing rensrd
unalattinn roward
bust boskmokiionnn ronard
-000
-30
D0n
Don 0R:OON
00.00
01.001
(b) LunarLander-ve v2
Lunar LunarLander-v2 nder-22
200
COM
-000
05
OV0 tnaiting proprem
0.4
04
takiing propme
(c) MountainCar-vo
MaunteinnCrr-v
tra rire reward
evaluation roward
best model reward
-120
COM
D0
605 06:0N
D00
210005
making
Figure A4: The first column shows the cumulative rewards obtained during the training process, the second column shows the testing performance without randoi exploration, and the third column shows the performance of best model learned up to the point
Table A4: Hyperparameters
Environment
CartPole-vl
MountainCar-vo
Hyperparameters
LKTD
DQN
QR-DQN
LKTD
DQN
QR-DQN
learning rate
2.5e-5
2.3e-3
2.3e-
1.0e-4
4.0--
4.0e-3
M (pseudo population)
20000
20000
og (prior)
1
0.5
(observation)
1
target update interval
10
10
100
600
600
(discount factor)
0.99
0.99
0.99
0.98
0.98
0.98
training steps
le5
le5
1e5
2e5
2e5
2e5
batch size buffer size learning starts train freq
64
64
128
128
128
le4
1e5
1e5
le4
1e5
1e5
1000
1000
1000
4
256
256
32
16
16
gradient steps exploration fraction
128
128
0.16
0.16
0.16
0.2
0.2
0.2
exploration final eps
0.04
0.04
0.04
0.07
0.07
0.07
24

---

Table A5: Hyperparameters (cont.)
Environment
LunarLander-v2
Acrobot-vi
Hyperparameters
LKTD
DQN
QR-DQN
LKTD
DQN
QR-DQN
learning rate
5.0e-6
6.30-4
1.5e-3
5.0c-5
6.3c-
6.3e-4
M (pseudo population)
20000
20000
(prior)
1
1
(observation)
1
target update interval
1
250
1
1
250
250
(discount factor)
0.99
0.99
0.995
0.99
0.99
0.99
training steps
2e5
2e5
2e5
1e5
1e5
1e5
batch size
128
128
128
128
128
128
buffer size learning starts train froq
2.5e4
5e4
1e5
5e4
5e4
5e4
0
10000
0
0
50000
256
gradient steps
4
4
256
4
4
4
exploration fraction
0.24
0.12
0.24
0.12
0.12
0.12
exploration final eps
0.05
0.10
0.18
0.05
0.1
0.1
25


Reviews:
[{"review_id": "UEQ43fhi1i", "reviewer": "Reviewer_5LPR", "summary": "This paper introduces a sampling algorithm called Langevinized Kalman Temporal-Difference (LKTD) for deep reinforcement learning. The algorithm leverages the Kalman filtering paradigm to efficiently draw samples from the approximate posterior distribution of deep neural network parameters, allowing for the quantification of uncertainties associated with the value function and model parameters. The LKTD algorithm improves the robustness and adaptability of reinforcement learning approaches.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper introduces the limitations of existing reinforcement learning algorithms that overlook the stochastic nature of the agent-environment interaction system. To address this, the authors propose a novel algorithm called Langevinized Kalman Temporal-Difference (LKTD) that leverages the Kalman filtering paradigm to draw samples from the posterior distribution of deep neural network parameters. The LKTD algorithm allows for quantifying uncertainties associated with the value function and model parameters, and enables monitoring of these uncertainties during policy updates throughout the training phase.\n\n### Technical route of the article (step by step):\n\nThe LKTD algorithm is based on Langevin-Koopman Dynamic Mode Decomposition (LKTD) and Stochastic Gradient Langevin Dynamics (SGLD) sampler.\nThe mathematical equations and algorithms used in the LKTD algorithm are described, including the formulas for forecasting and analysis.\nThe convergence of the LKTD algorithm under the on-policy setting is discussed, along with the conditions and assumptions required for the convergence.\nThe choice of the pseudo-population size N and its impact on the system is explained.\nThe cooperation of the LKTD algorithm with a replay buffer is discussed, and a convergence theory for this scenario is presented.", "weaknesses": "1. Comparison with existing posterior sampling value-based algorithms for exploration is missing. Say ensemble sampling, Bootstrapped DQN or HyperDQN.\n2. It is better to translate the theoretical guarantee to regret bound.\n3. Comparison for uncertainty quantification in deep neural network is missing.\n4. The empirical performance could be demonstrated through a wider range of benchmark problems, e.g. Arcade Learning benchmarks or behaviour suite.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UaE2MvPQd3", "reviewer": "Reviewer_dX3U", "summary": "This paper delves into the stochastic nature in RL by introducing the Langevinized Kalman Temporal-Difference (LKTD) algorithm, which relies on an approximate sampling technique grounded in the Kalman filtering paradigm. By leveraging stochastic gradient Markov chain Monte Carlo (SGMCMC), this algorithm efficiently draws samples from the posterior distribution of deep neural network parameters, offering robust uncertainty quantification for deep RL.", "questions": "1. Could authors comment on how many number of iterations are required in SGLD in order to achieve the convergence guarantee in Theorem 1? And how optimal is the convergence rate?\n\n2. I am also curious if we adopt second-order MCMC methods such as SGHMC, could we obtain faster convergence with extra computational benefits? Is Theorem 1 generalizable to include the convergence guarantee of SGHMC methods? What can be potential challenges compared to SGLD?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Existing RL algorithms often treat the value function as a deterministic entity, overlooking the system's inherent stochastic nature. The Kalman Temporal Difference (KTD) framework in RL treats the value function as a random variable, aiming to track the policy learning process by formulating RL as a state space model. However, KTD-based techniques employing linearization methods become inefficient with high-dimensional models.\n\nTo improve the computational efficiency in deep RL, this paper reformulates RL with a new state space model, and adopts Langevinized ensemble Kalman filter (LEnKF) algorithm to ensure that the empirical distribution of the parameters in value function will converge to the desired target distribution. This reformulation enhances robustness by including prior information in the state evolution equation, robustifying performance, and handling large-scale neural network approximations. The proposed algorithm introduces computational efficiency, directly handling nonlinear functions without requiring linearization. It achieves memory efficiency by replacing the storage for the covariance matrix with particles, showcasing scalability for large-scale neural network approximations.\n\nProviding the convergence guarantee in deep RL while incorporating approximate Bayesian inference can be challenging. This paper establishes promising performance guarantees in this regime.", "weaknesses": "1. The uncertainty quantification for Deep RL is not restricted to Kalman TD frameworks. In the literature of Bayesian RL, we do have works that adopt other approximate sampling techniques, e.g. ensembling in Bootstrapped DQN (Osband et.al. 2016), \"Deep Exploration via Randomized Value Functions\" (Osband et.al. 2019), and RLSVI with general function approximation (Ishfaq et.al., 2021). Although the above methods do not adopt MCMC methods as approximate sampling scheme, could authors compare the proposed method with the existing Bayesian RL methods, not just restricted to the Kalman TD context?\n\n2. Though the authors provide a clear comparison to the existing Kalman TD frameworks,  it is desirable to see a more thorough related work study in Bayesian RL / uncertainty quantification in deep RL, which would be beneficial.\n\n3. The proposed method requires a proper prior distribution to achieve the performance guarantee in LEnKF. It is unclear how the quality of the prior affects the performance through the current theorems. Is it possible to capture such quantity in the upper bounds provided in Theorem 1 or Theorem 2?\n\n4. The experiment environments are relatively simple, especially for the control tasks. It is suggested authors consider more complicated locomotion tasks to demonstrate the effectiveness of the proposed method.\n\n5. The proofs of Theorem 1 and 2 rely on a significant number of assumptions. It is suggested authors comment on whether those assumptions are imposed in order to achieve the convergence guarantee in sampling schemes, and how such assumptions can be satisfied in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EhaYeduNRW", "reviewer": "Reviewer_n3CS", "summary": "This paper proposed a novel and scalable sampling algorithm for reinforcement learning. It provides a systematic theoretical analysis of the convergence, identifying that the method is able to quantify uncertainties.", "questions": "The paper claims that:\n\n> it is more appropriate to consider the value or the model parameters as random variables rather than fixed unknowns, and it is preferable to focus on tracking rather than convergence during the policy learning process.\n\nHowever, could you please explain what tracking is? And how could (i)(ii)(iii)(iv) can lead to this conclusion or statement?\n\n\n\n\n\nHow's the paper related to some random value function-based method? such as [1].\n\n[1] Osband, Ian, et al. \"Deep Exploration via Randomized Value Functions.\" *J. Mach. Learn. Res.* 20.124 (2019): 1-62.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "This paper focuses on an important problem of RL: how to model the uncertainty during the interaction with the environment.\n\nThe paper proposed a new sampling framework for RL. The paper also gives a systematic theoretical analysis of the convergence of the proposed algorithm under the general nonlinear setting.", "weaknesses": "The writing of the paper is quite unclear to me. Some presentations need to be further clarified.\n\n- I don't quite understand why the title of the paper is \"fast value tracking\". The term \"tracking\" is only mentioned in the introduction without clearly explaining it. Why is it \"fast\"?\n\n- eq. 2,  introducing  $\\pi$ seems lack motivation to me. Why it can tackle the issues suffered by KTD.\n\n  \n\nSome terms need further explanation and clarification.\n\n- Please clarify what \"stage\" means.\n- $x_t$ denotes a set of states and actions, does it refer to a trajectory containing a set of states and actions?\n- It's weird to introduce a notation without explanation the first time, such as $h(\\cdot)$\n- \"Optimal policy exploration\"\n\nThe experiments are not sufficient\n\n- the paper only compares the Adam algorithm.\n- The improvement over Adam is not significant   according to Figure A1\n\n\n\n\nMinor comments:\n\n- It's kind of weird to say $r(s,a)$ is a random variable.\n- Markov chain => Markov Chain\n- It's kind of hard to get the main contribution of the paper from the introduction\n- such as e.g.?\n- The caption in Figure 5 is very akward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)