A sampling framework for value-based reinforcement learning
Authors: Frank Shih
Faming Liang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
Bayesian sampling
Stochastic gradient MCMC
Value function approximation

Abstract: Value-based algorithms have achieved great successes in solving Reinforcement Learning problems via minimizing the mean squared Bellman error (MSBE). Temporal-difference (TD) algorithms such as Q-learning and SARSA often use stochastic gradient descent based optimization approaches to estimate the value function parameters, but fail to quantify their uncertainties. In our work, under the Kalman filtering paradigm, we establish a novel and scalable sampling framework based on stochastic gradient Markov chain Monte Carlo, which allows us to efficiently generate samples from the posterior distribution of deep neural network parameters. For TD-learning with both linear and nonlinear function approximation, we prove that the proposed algorithm converges to a stationary distribution, which allows us to measure uncertainties of the value function and its parameters.

Under review as a conference paper at ICLR 2023
A SAMPLING FRAMEWORK FOR VALUE-BASED REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Value-based algorithms have achieved great successes in solving Reinforcement Learning problems via minimizing the mean squared Bellman error (MSBE). Temporal-difference (TD) algorithms such as Q-learning and SARSA often use stochastic gradient descent based optimization approaches to estimate the value function parameters, but fail to quantify their uncertainties. In our work, under the Kalman filtering paradigm, we establish novel and scalable sampling framework based on stochastic gradient Markov chain Monte Carlo, which allows us to efficiently generate samples from the posterior distribution of deep neural network parameters. For TD-learning with both linear and nonlinear function approximation, we prove that the proposed algorithm converges to a stationary distribution, which allows to measure uncertainties of the value function and its parameters.
INTRODUCTION
Reinforcement learning (RL) targets at learning an optimal policy for sequential decision problems in order to maximize the expected future reward. The value-based algorithms such as Temporal- difference (TD) learning (Sutton] 1988). faafe-atioo-reward--tt-------
(SARSA) (Sutton & Barto 2018). and Q-learning are frequently used, which play crucial role for policy improvement. TD- learning aims to estimate the value functions, including state-value function and eation-valuee function, by minimizing the mquan-quareed Bellman error, where the value functions are often approximated by function family with unknown parameters. Hence, it is critical to evaluate the accuracy and uncertainty of parameter estimation, which enables uncertainty quantification for the sequential decision at a sequence of states.
In the function approximation TD algorithms such as Deep Q-Network, the parameters are commonly optimized by stochastic gradient descent (SGD) based algorithms. The convergence of these algo- rithms, including both with linear function approximation (Schoknecht) 2002) and nonlinear function approximation (Fan al. 2020 Cai al.) 2019), has been extensively studied in the literature. How- ever, SGD suffers from the local trap issue while dealing with nonconvex function approximations such deep neural networks (DNNs). In order efficiently and effectively explore the landscape of the complex DNN model, Monte Carlo algorithms such as Stochastic Gradient Langevin Dynamics (SGLD) (Weliigg& Teh [011] Aicher al. 2019 Kamalaruban et al.) (220)) have shown their great potential in escaping from local traps. Moreover, under the Bayesian framework, the Monte Carlo algorithms generate samples from the posterior distribution, which naturally describes the uncertainty of the estimates.
Toward uncertainty quantification for reinforcement learning, is important to note that the reinforee- ment learning problem can be generally reformulated as state-space model. In consequence, the value function parameters can estimated with Kalman filtering methods such as Kalman Temporal Difference (KTD) (Geist & Pietquin 2010) and KOVA algorithm (Shashua & Mannor (022)). Under the normality assumption and for linear function approximation, the Kalman filter approaches are able to provide correct mean and variance of the value function, which enables uncertainty quantification for the sequential decision. However, for nonlinear function approximation, KTD and KOVA algo- rithms adopt unscented Kalman filter (UKF) (Wan & Van Der Merwe 2000) and extended Kalman filter (EKF) techniques to approximate the covariance matrices. Both algorithms are computationally inefficient for large scale neural networks. KTD requires Opp²) for covariance update, where p is the

---

Under review as a conference paper at ICLR 2023
number of parameters. In each iteration, KOVA calculates Jacobian matrix that grows linearly with batch size.
In this paper, we have two major contributions: (i) We develop Kalman filter-type algorithm for valued-based policy evaluation based on the Langevinized Ensemble Kalman filter (Zhang et al. The | Dong et al. 2022). The new algorithm scalable with respect the dimension of the parameter space, which has a computational complexity of O(p) for each iteration. (ii) We prove that even when the policy is not fixed, under some regularity conditions, the proposed algorithm converges to stationary distribution eventually.
2
BACKGROUND
2.1 MARKOV DECISION PROCESS FRAMEWORK
The standard RL procedure aims to learn an optimal policy from the interaction experiences be- tween an agent and an environment, where the optimal policy maximizes the agent's expected total reward. The RL procedure can be described by Markov decision process (MDP) represented where S the state transition probability from state s to state s' by taking action a, denoted by P(s'|s,a) a), discount A according
by (S,
of states, is finite set of actions, P S x x S - R is
r(s, a) is a random reward received from taking action at state 8, and 7 E (0, 1) S and takes action
factor. At each time stage I, the agent observes state to policy P with probability (a) then the environment returns a reward r₃ = r(s,, az) and a new state 81+1 E S. For given policy p. the performance is measured by the state value func- tion (V (V-function) V*(s) R 7 r = s] and the state-action value function (Q-function) (",s, a) = E 0 7 4, = a). Both functions satisfy the following Bellman equations:
V((s) = a)) + +VV*(s')), (e a) = [ 7Q"(6',")))
where s' ~ a(, Pp(.ls), a' Pp(-1s"), and the expectations are taken over the transition probability for a given policy p.
2.2 BAYESIAN FORMULATION
In this paper, we focus on learning optimal policy p via estimating Q". Suppose that Q-functions are parameterized by ((=:0) 8) with parameter e 0 c RP, Let He be the stationary distribution of the transition tuple = (s, a, 7, s', a') with respect to policy p. Q" can be estimated by minimizing the mean squared Bellman error (MSBE).
2Q(5 min MSBE min Ez-n, (( -r
where the expectation is taken over a fixed stationary distribution up. By imposing a prior density function nt (f) 0, we define a new objective function
F(0)= = Eznny [F)
= E Exnua
- -rr
- log π (0)
where F(B, 2) = (Q(s, a; 8) - ~ d': 0)) log π (0). Since the stationary distribution Mp is unkown, consider the empirical objective function
f.
2,),
on set of transition tuples = (2 Instead of minimizing F= directly, one can simulate sequence values using the SGLD algorithm by iterating the following equation:
0, 0 -ennF ((nF((0-11) + 2-,88-uus
2

---

Under review as a conference paper at ICLR 2023
where 1)(-1)) is conditionally unbiased estimator of VF= (0 wi Ip) is standard Gaussian random vector of dimension c > O is the learning rate at time t, and B > 0 is the constant inverse temperature. It has been proven that under weakly to the unique Gibbs measure V² exp(-3n However, in value-based RL algorithms, the policy p is dynamically updated along with parameter 8,. Therefore, the distribution H4 of the transition tuple also evolves from time to time 02 changes. In section 3, develop new sampling algorithm, Langivinized Kalman Temporal Difference (LKTD) algorithm, for value-based RL algorithms and establish the convergence of the proposed algorithm under the dynamic policy setting.
regularity assumptions. 8g converges
3 MAIN RESULTS
In this section, we first introduce the state-space model formulation and the proposed sampling algorithm under the setting of linear function approximation, and then extend the proposed sampling algorithm to the sctting of nonlinear function approximation. For simplicity, a full transition tuple with reward and reduced transition tuple without reward are denoted, respectively, by and
{ (s, (s,,,r,s',ar
and I = (s, a, a',aa'))))
(S)
(s,,,
(s. a,))
for which we often write 2 (r,, x.. The observation function (1x, 0) is defined follows:
h(2: 0) [Q(s, o; = QQ(s',a;; Q((s, a; 0) - 7 m Q(s', b), for the SARSA and Q-learning algorithm, where " is the discount factor.
(6)
3.1 LINEAR FUNCTION APPROXIMATION
Suppose that Q (Q(:0)) is family of linear Q-functions, where every Q-function be approximated in the form
(7) where ó : S A IP is p-dimensional vector-valued feature map. For example, é can be polynomial kernel, Gaussian kernel, etc. Let &; = (r,, xz) = {Tr,,, { be a batch transition tuples of size generated at stage t. For convenience, we bold symbols to represent either vectors or sets of transition tuples depending on the situation. By combining (6) and we define the observation matrix
Q(s, a: 0) = ((s To.
(IF,,1) : (
((xx)=
(8)
where each row vector is defined as
(o(st,1,0,1)) (06 d(.)))
+ (xx.)) (0(s1,)) (5(5 b)
(9)
for SARSA and Q-learning. Then it is easy to see that the minimization problem of MSBE in (1 can be reformulated as a Bayesian linear inverse problem
T=- = (xx;)0 + Tht T)t ~ 02), t 1=1,2.....7
(10)
where 7) is an additive Gaussian white noise with covariance matrix I and 0 is subject to the prior distribution π (0). The corresponding posterior distribution is given by x. (0) x e f(0).
To develop an efficient algorithm for simulating samples from the target distribution -. (0) - where 3 denotes the inverse temperature, we further reformulate the Bayesian linear inverse model
(00 as a state-space model through Langevin diffusion by following Zhang et al. (202) and Dong
ct al. (2022):
0: = 0,-1 V log (0.-1) + Wt.
2
(11)
r: = ((x;)) +
3

---

Under review as a conference paper at ICLR 2023
where W, c(,p, = (O, ie., 94 = and π) N( In the state-space model (1I). the state fg evolves a diffusion process that converges to the prior distribution (0). The new tormulation does not only allow to solve the Bayesian inverse problem by subsampling (r,, xx,) from given dataset at each stage (see Theorem S1 of ofang et al. (202T), but also allow us to model dynamic system where the policy po,-1 changes along with stage f. In order establish the convergence theory of the entire RL algorithm, we impose two fundamental assumptions on the data generating process and the normality structure.
Assumption (Data generating process) For each t, we are able to generate tuple 25 p according to policy pa.-1- Moreover, the stationary distribution pe__, has density function (2 (z)(0r-). which is differentiable with respect to 0.
3 Assumption (Normality structure) For each let 2t (= be a set of full transition tuples sampled po..1 the conditional distribution <(r, L2, 0,) is Gaussian:
rz|æ,, Br
(12)
That is, Te,5 O are independent Gaussian distributions.
Figure 1 depicts the RL updating scheme and data generating process. each stage i, the agent interacts with the environment according to the policy (9,_1 and generates batch of tran- sition tuples z, (æt, r1) from the stationary distribution 1 which refers to assumption I With the artificial normality structure in assump- tion 2 and the state-space model (11). we com- bine the RI. setting with the forecast-analysis procedure proposed in Zhang et al. (2021) ) and
8.
0,-1
0.12
Ph.-1
At
06.e
Xy-1
X-+2
Figure 1: Data generating process
introduce Algorithm [ In Theorem 3.1 we
prove that the our algorithm is equivalent to an accelerated preconditioned SGLD algorithm et al. 2016). Then, with some adjustment of Theorem 10 Raginsky al. (2017) and the general recipe of stochastic gradient MCMC (Ma et al. 2015). the chain {06 Jé-1 generated by Algorithm [ converges to stationary distribution p. (0) xx exp( 3G(()) as defined in Lemma A.1 in the Appendix. When p is fixed for policy evaluation, Algorithm converges to the target distribution (²)
Algorithm (Langevinized Kalman temporal difference learning for linear approximation)
0. (Initializationn) Start with an initial Q-function parameter 0% E R2, drawn from the prior distribution x (0). For each stage f. 1.2...., ,T, do steps 1-3:
(Sumpling) With policy pog_,' generate a set of n transition tuples from the stationary distribution pog_, denoted by 2: = (re, xx.) = (zt.}] where 2n, has the form of (5) Let 4, = =4(x,)-
Set Ω alp R₁ = I and the Kalman gain matrix Ke (2 R1
2. (Forecast) Draw ~ (, and calculate
of = 01-10 + o/2 " log (0g_1) wi.
(13)
3. (Analysis) Draw i Nn (0, R₂) and calculate
0g = of + Ke(r, - &.0 v.) = of + Ke(re -r/).
(14)
Theorem 3.1 Algorithml Ccan be reduced to a preconditioned SGLD algorithm.
32
C1 Σ2 2 V log T et.
0g = 02
(15)
where E, -II-KK is constant matrix given Eu e N(0, and V Vlog πT (88_112,,)
(8r,)))(r, (00,,,00_11 lo
4

---

Under review as a conference paper at ICLR 2023
Theorem 3.2 Consider the pre-conditioned SGLD algorithm:
0, ===0--1 01-1--e22,
+ 2er t= 1=,,,....,
(16)
where e, NO, E.), B is the inverse temperature, is the total iteration number, and eT constant learning rate depending on T. For this algorithm, we assume the conditions of Lemma A. (of the Appendix) hold. Further, we choose the learning rate eT such that Tet + xc, Tef and 8'4781/4 + then W2(Pr, P.) T 0o, where pr and pa are as defined in Lemma A.I of the Appendix.
Remark 1 For the LKTD algorithm, we have G(0,-1, 1.) ((Ee,)))-1)) + + 0, In addition, set Er to/ T for some constant to > 0 and E (4/5, 1).
- -i (+0 T (Tei
such that 81/61/4 + O.
The conditions required by Theorem 3.1 and Theorem 3.2 are verified by Lemma 3.1 given below.
Lemma 3.1 Let G(O, 2) - - Ei (+(,) (r. (x, [(), 0. Assume (i) is compact, and =uup[r] R] oc, (ii) (| a)ll S for all S, A, and (iii) 0g of).
with of }. then the conditions ((A) are satisfied.
The LKTD algorithm is very flexible. is not necessary use all samples (collected at each time t) at each iteration. Instead, a subsample can be used and multiple iterations can be performed for intergrating the available data information in the way of SGLD. Moreover, as explained in Zhang et al. (2021), the forecast-analysi. procedure enables the algorithm scalable with respect to the dimension of 6g. while enjoying the computational acceleration led by the pre-conditioner. In summary, the LKTD algorithm is scalable with respect both the data sample size and the dimension of the parameter space.
3.2 NONLINEAR FUNCTION APPROXIMATION
In this section, we further extend our algorithm to the setting of nonlinear function approximation. For each stage t, we consider the nonlinear inverse problem
h( f) + T)+ ~ N(0,
(17)
where ((::)) +
is a nonlinear differentiable observation function 0. With the state augmentation approach similar to LEnKF algorithm, we define the augmented state vector by
0: &
&, -h(x,;00 0z) + ues Ui ao0,a
where & is an n-dimensional vector, and < < is a pre-specified constant. Suppose that 0, has prior distribution (8) (8) as we defined in previous section, the joint density function (p: (0, ),T can be written as ( = ((f, )T (T (.). where Eg/d2 ~ (h((x, 0e), ao I). Based on Langevin
dynamics, we can reformulate (17) as the following dynamic system
E V log ((p=-1) + Wt: 9: 91-1
2
(18)
r: H1pp + Ut,
where wy NO, Q = etlp. p the dimension of ip; He (0, I) such that HeP: E ee NO, (1 T), which is independent of wy for all t. With the formulation in 18) transformed a nonlinear inverse problem to linear state-space model and thus the previous theoretical results still hold for the nonlinear inverse problem. The target distribution p.(O) can be easily obtained by marginalization from pa (5)).
Algorithm 2 (Langevinized Kalman temporal difference for nonlinear approximation)
O. (Initialization) Start with an initial Q the prior distribution n (0). For each stage i. 1.2,...7 ...T,, T, do the following steps 1-3:
Q-function parameter ensemble 8 E R,, drawn
5

---

Under review as a conference paper at ICLR 2023
1. (Sampling) With policy pot-1 defined in 440. generate set of n transition tuples from the stationary distribution -08_11 denoted by 2e vt) = {=t,]j=1 where 2n. has the form of ()) Let H (0.
For each iteration k 2, set Qi,k = ELppl. R = 2(1 - a)o² and the Kalman gain matrix Ke, = Qu (H.Qu + Ro) and do steps 2-3.
2. (Forecast) Draw W ~ N2 (2) and calculate
Pi,k = fi,k-1 + Et. v log (F(, Wr,ks
(19)
where if = 1, set Pi,0 = (0 ,T.).. More precisely, the gradient of two components can be written
V log π (P] V log π (01,--1) + 0, h(x:: 0t,
900 og (Et,--1 h(1;; 0t,k-1))
(20)
3. (Analysis) Draw Vr ~ (0, (0, Re) and calculate
-ix= + (rx - HoPL - v(,x) = Pix + Kexlre - -r(
(21)
4 EXPERIMENTS
In this section, we compare LKTD with Adam algorithm (Kingma & Ba 2014). With a simple indoor escape environment, exploration. Further, with a more complicated environments such as OpenAl gym, we show that LKTD is able to learn better and more stable policies for both training and testing.
show the ability of LKTD in uncertainty quantification and policy
4.1 INDOOR ESCAPE ENVIRONMENT
Consider simple indoor escape environment as shown in Figure The environment is in the square [0. (0, 1). and the goal is to reach the top right corner of size 0.1 x0.1 as fast as possible. At the beginning time O, the agent is randomly put in the square. At each time t, the agent observes the location coordinate
0.1- Gual
3.1
(x,y) y) as its current state, then chooses an action a E (N, S, E, W) according to policy p with a step size randomly drawn from [n, 0.3) The reward at each time t is -1 before the agent reaches the goal. The indoor escaping environment an example where the optimal policy is not unique. Observe that the Q-values of N and E have no difference in most states, except for the top and the right border. Hence, the ability to explore various optimal policies is critical for learning stable and robust policy. Through this experiment we show the ability of the LKTD algorithm to learn mixture optimal policy single run. We compare LKTD with the widely used Adam algorithm on training deep neural network with three
Wet
North
South
Figure 2: Indoor escape environ- ment
hidden layers of sizes 16,, 16). Agents update the network parameters every 50 interactions and 5 gradient steps per update for total of 10000 episodes. For action selection, the e- e-Boltzmann exploration as defined in B.I is used with an exploring rate e = 0.1 and an inverse action temperature of Ba 5. The batch size is 250. The last 1000 parameter updates are collected parameter ensemble, which induces a Q-value ensemble and a policy ensemble. With the Q-value ensemble, are able to draw the density plot of Q-values each point of the square illustrated by Figure To quantify uncertainty of the policy, we define the mean policy probability by
Pe (a =
(p(s)))
(22)
PEe
where @ is the policy ensemble induce from the parameter ensemble. Intuitively, the mean policy probability is the proportion of an action taken by the policy ensemble given state. We further

---

Under review as a conference paper at ICLR 2023
Ovvalie dieeiiy
Quvak deedity
Action M. 6.516
octin M: 6,038
25-
Actin si 202
25-
Actio 0.0.2
0:0.822
25-
25
10-
10-
60
60
(a) Q-value density at (5.5, 5.5)
(b) Q-value density at (2.5.9.5)
Figure 3: Q-value density plots and mean policy probabilities of LKTD
Adton Pressot
(a) Mean optimal action probability
(b) Close optimal
(c) Bad exploration
Figure 4: (a) is the mean optimal action probability of indoor escape environment. (b) is similar to (a) on both boundary and interior grids. (c) Fails to find the correct policy on the boundaries, which leads to high false optimal action rate for actions {N, E}.
define the mean optimal policy probability of an environment by taking expectation all optimal policies. The mean optimal policy probability of the indoor escape environment is shown in figure 4a
In figure|4 divide the state space into 100 grids of size 0.1 x 0.1, then compute the mean action probabilities of each grid center and each action. For a further comparison of the two algorithms, we calculated two metrics in table T (1) MSE between the mean action probability and the mean optimal action probability, denoted by MSE(p), where the MSE is taken over all grids. (2) The sub-optimal action rate (SOAR), which is defined by the probability of choosing the action that is sub-optimal for state. Table I shows sshows that in terms of MSE()), LKTD and Adam with large learning rate are more efficient in sample space exploration; however, terms of SOAR, LKTD is much smaller than Adam in actions (N, E). where the high SOAR comes from the top and right boundaries as in figure 4c In other words, LKTD can efficiently explore the optimal policies, while retaining its accuracy. In figure 3b the mean policy probability shows that LKTD can choose the correct policy the boundary grids.
Table 1: MSE()) and SOAR
Description
North
East
South
West
Name
ft
B
MSE(p)
SOAR
MSE())
SOAR
MSF(6)
SOAR
MSE(p)
SOAR
LKTD
lc-4
0.044
0.098
0.044
0.102
0.002
0.026
0.002
0.026
Adam
le-2
N/A
0.044
0.134
0.044
0.135
0.005
0.005
Adam
le-3
N/A
0.047
0.493
0.047
0.495
0
0
Adam
lc-4
N/A
0.058
0.502
0.058
0.495
0
0
0

---

Under review as a conference paper at ICLR 2023
4.2 CLASSICAL CONTROL PROBLEMS
In this section, we consider four classical control problems in OpenAl gym (Brockman et al. 2016), including CartPole-vI, MountainCar-vO, LunarL.ander-v2 and Acrobot-vl. We compare LKTD with Adam under the framework and parameter settings of RL Baselines3 Zoo (Raffin 2020). Each
experiment is duplicated 500 times, and the training progress is recorded in figure 5 At each time step. the best and the worst 1% of the rewards are considered as outliers and thus ignored in the plots. LKTD can also be applied DQN algorithm by modifying the state-space model in equation as
01-1 +
Et V
log क (0(-1) + Wrs
(23)
y: = o(,,, at) T + nt.
where 0(8 0z) 1 11,1), .... 2 n (t,a)) ) and y + r0(s), TT)
a0-)0 The new ((=,:)))) + o3 0--1, where the first term corresponds to the aani-gradient in DQN algorithm. With suitable constraints on the semi-gradient, we can modify lemma 3. guarantee the convergence. In the four classic control problems, LKTD shows its strength in efficient exploration and robustness without adopting common RL tricks such as gradient clipping and target network. The updating period of the target network is set to I for LKTD. The detail hyperparameter settings are given in section B.4
gradient can be written as (. =x) 1
In figure 5] the solid and dash lines represent the median and mean rewards, respectively. For each algorithm, the colored area covers 98% of the reward curves. We consider types of reward measurements, training reward, evaluation reward and the best evaluation reward. Training reward records the cumulative reward during training, which include the c-exploration errors. Evaluation reward calculates the mean reward over 10 testing trails at each time t. The best evaluation reward only records the best evaluation reward up to time t.
In CartPole-vl, LKTD outperforms Adam in all measurements, especially on the training and best evaluation rewards. During training, LKTD receives significantly higher rewards than Adam. In optimal policy exploration, almost 99% of the time LKTD achieves the optimal policy faster than the median of Adam.
In MountainCar-vO, the mean and median reward curves of LKTD and Adam are similar. However, LKTD is more robust during training and more efficient in exploration of good policies. From the best evaluation reward plot, we can observe that the 1% reward lower bound is close to -200 for Adam, which indicates that the agent fails find any good policies during the training.
In order to learn the optimal policies in Lunarlander-v2, the agent has to learn a correct way of landing instead of staying in the air. Due to the sampling nature of LKTD,, the exploration rate is increased from 0. 1212 to 0.25 for agent to collect enough landing experiences. Hence, LKTD converges slightly slower than Adam. However, with large exploration rate, LKTD is still able to obtain stable training rewards which are close to Adam with a much higher lower bound. Moreover, with a longer training period, LKTD will eventually perform better in evaluation.
In Acrobot-vl. 1, the training reward of LKTD converges slower in some experiments, but in most cases, the performance of LKTD dominates Adam.
According to the experiments, LKTDD has a more robust training process and finds the optimal polices faster than Adam. The experiments also indicate that Adam uses the experiences more efficiently, whereas LKTD needs to trade the training performance for the exploration of rare experiences.
5 CONCLUSION
This paper proposes LKTD as a new sampling framework for deep RL problems via state-space model reformulation. LKTD equivalent an accelerated preconditioned SGLD algorithm but with a self-dependent data generating process. For both linear and nonlinear function approximations, LKTD guaranteed converge to stationary distribution (0) under mild conditions. Our numerical experiments indicate that LKTD is comparable with Adam algorithm in optimal policy search, while outperforming Adam in robustness and optimal policy explorations. This implies great potential of LKTD in uncertainty quantification.
8

---

Under review as a conference paper at ICLR 2023
CartPole-v1
tram rewerdd
evall reward
beett evill reward
200
30-
300
(CChaome
2005
1000
10000
23906
4006
3009
3700
3000
4000
MauntainCar-vo
train roword
oval reward
bost ovall reward
LCO mai
LCD vaa
4002
1300
11:000
1M033
3300
1030
12:00
1800
4033
s00ce 1803
LunarLander-w2 eval reward
train reward
bast eval reward
400
-00
400
U man
MCU men
340K
4006
1209
8099
12900
24009
4006
2000
59000
12000
2430
4005
2000
1200
train reward
eval reward
best eval reward
LC0 m0n
UCD mon
2.00
4000
00L00
20,00
13000
20202
AU0UE
060E2
10000
20203
5003
2000
Figure 5: The first column shows the cumulative rewards obtained during the training process, the second column shows the testing performance without random exploration, and the third column shows the performance of best model learnt up time f.

---

Under review as a conference paper at ICLR 2023
REFERENCES
C. Aicher, S. Putcha, C. Nemeth, P. Fearhead, and E.B. Fox. Stochastic gradient meme for nonlinear state space models. NCKV::/901.10068000 2019.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 2016.
Qi Cai, Zhuoran Yang, Jason Lee, and Zhaoran Wang. Neural remporal-difference and q-learning provably converge to global optima. arXiv preprint arXiv: /0027, 2019.
Tianning Dong, Peiyi Zhang, and Faming Liang. A stochastic axproximation-langeviinn ensemble kalman filter algorithm for state space models with unknown parameters. Journal of Computational and Graphical Statistics, pp. press, 2022.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. theoretical analysis of deep q learning. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger (eds.), Proceedings of the 2nd Conference on Learning for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pp. 486-489. PMLR, 10-11 Jun 2020. URL https: http:///cccceeenn.scc/m /ttpts////prrrceiinngg.. wwtpeeediiiin.m........mm. mlr. www.pppwssss...//.
html
Matthicu Geist and Olivier Pietquin. Kalman temporal differences. J. Artif. Int. Res., 39(1):883-532, sep 2010. ISSN 1076-9757.
Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dy- namics. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 8127-8138. Curran Asso- ciates, Inc., 2020. URL htpp:ttttp///ppooeeelliggg 5cb0e249689ca6d32 0abbo222666c6b333333333333332-23
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1412. 2014.
Chunyuan Li, Changyou Chen, David E. Carlson, and Lawrence Carin. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In AAAI, 2016.
Yi-An Ma. Tianqi Chen, and Emily B. Fox. A complete recipe for stochastic gradient meme. In NIPS, 2015.
Francisco S. Melo and M. Isabel Ribeiro. Convergence of q-learning with linear function ap- proximation. 2007 European Control Conference (ECC), pp. 2671-2678, 2007. doi: 10.23919/ECC222077779999
Antonin Raffin. RI baselines3 zoo. httppess/ggttuub.com//Rtet 2020.
ine3--z0
Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis. Proceedings of the 2017 Conference on Learning Theory, pp. 1674-1703, 2017.
Ralf Schoknecht. Optimality of reinforcement learning algorithms with linear function approximation. In S. Becker, S. Thrun, and K. Obermayer (eds.), Advances in Neural Information Processing Sys- tems, volume 15. MIT Press, 2002. URL http:/tttss:://cceeeeeet
ngs www.ieerrs....
Shirli Di-Castro Shashua and Shie Mannor. Kalman meets bellman: Improving policy evaluation
through value tracking. ArXiv, abs/2002.07171,, 2020.
Richard S. Sutton. Learning predict by the methods of temporal differences. Machine Learn- ing. 3(1): 44, August 1988. URL http: wwww///wwww/////////////////// cs ualberta. ca/ suutoon/papers sutton-88.
10

---

Under review as a conference paper at ICLR 2023
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press. second edition, 2018. URL http:://inccmmaeeeeeeeaaaaaatt. toook 2nd html
E.A. Wan and R. Van Der Merwe. The unscented kalman filter for nonlinear estimation. In Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. No.00EX373), pp. 153-158, 2000. doi: 10.1 (L009/AASPCCC22008884446.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
Peiyi Zhang, Qifan Song. and Faming Liang. A langevinized ensemble kalman filter for large-scale static and dynamic learning, 2021. URL htt.:t/pps/aarx...ooooo///////// 05363

---

Under review as a conference paper at ICLR 2023
Appendix
A COMPLETE PROOFS
A.1 PROOF OF THEOREM 3.1
PROOF:
Following the proof of LEnKF theorem 3.1 (Zhang et al. 2021), 1. we consider the Kalman which can be rewritten as
gain matrix K, = (x,) (Rg (xx)))(,((0))
K₁ = (1 - (x (xx) T R² (*())) R (xx) +
R
(24)
Let of 08-1 + dg + wi, where 6t = e, V log (0 With the identity 24 the conditional expectation 0g can be written as
E[0 rt, x+) = 0-- + o, + Kr(r, - (xx)) - ((x))6))
0 + K2 - o(xx)0( + (x)(x)))
0/-1 + (1 - -
1 R2 (r: - (xxx) + (1 - (x)(xx))
0,-1 1 + (1 K2 )x), " (xx,) 1) + 22 _,
- + 2 (1 - K, 2 (x,)) VV (xx )/- + -//10.l
= 8 + " Σ (xx)) T (r. 4 (x, 08 1) + V log x (0p (8%_ 1)
2
(25)
where E, I (xb(x,,, 92 &,1, and R. 2V. For LKTD, we have
8% 0/ K2 (r. (xx, 00} v1.)
0/-1 + 5, - uh + K,(r, - (xx;) + 6, + wx) - we)
(26)
(10070001111110 I1, + we - K2 (x0(xx) - K
1( + ee,
where G = (x( )w, + + with mean E(e;) = 0 and covariance
Var(e;) = Var(w) w.) + K, (al.var(((x)))) + 21) K, 2Cooo K2 (x((,,,, + v.))
.4 + K, ( (x, (xx + RR -2,,0(x,)00
(27)
(I K2 (x))) = c.
By combining
24
25
26
and the assumption V o21, the update of 0g can be rewritten
8 = 0%
+ C
Σ (xx))
- ((x))0f111 v log π (0 ) + t
n
00-
+ 61
Σ
((1,,)) T(res - (xx, log (0 + +e
(28)
00
+ " Σ
V log (8, 18, ct.
i=1
where V log क (08_122),))
2/2 (xr,)) (rr, D(2,,) + A log ((f_1).
A.2 LEMMA FOR THIBRRE 3.2
We assume the following conditions hold:
(AI) For any E O the Markov transition kernel HI, has a single stationary distribution (p(=). G: ④ x Z is measurable, and [|s(()]| = [I G(O, <=]0)()d < co.
(A2) There exists function G(0, 2), which is an anti-derivative of G(l, 2) with respect to 0. i.e., VoG(0, 2) = G(,,2), such that (0, 2)| < A for some constant A and any 2 E Z: in addition, there exists some constant B such that |G((0,))] B for any 2.
12

---

Under review as a conference paper at ICLR 2023
(A3) There exists some constant M > 0 such that for any 2,
||G(0,2) z) G(0, z)|l - M| - 0|1, v0, 0 E θ
(A4) For each 2, the function G(+, 2) is (m, b) dissippative for some m and b 2 0,
(0, (() 2 m||01|2 b, e.
(A5) There exist constant d E [0, 1) and some constants M and B such that
E G 2) 00 25 + B²); vB E
(A6) The probability law di of the initial hypothesis &o has a bounded and strictly positive density Po with respect the Lebesgue measure θ and
Kg == log
e 191 Po (o)Odd xo.
Lemma A.1 (Proposition 10 of Raginsky et al. (2177) Consider the SGLD algorithm with constant learning rate 6,
0, = 01- - -G(0--- + /2eB-1ee,
(29)
where et. ~ NO. Ia), is the dimension of 0, and 3 is the inverse temperature. Assume the conditions (A1)-(A6) hold. If (0 g(0=-1) holds for any step 1 E 0<6 1A m then TM
EEN, B 2211 2 and
W2(pt.p.) S (Co61/4 1/4 te/8OLs
(30)
where po( denotes the density function of te: po( x exp( (0))), ((0) is the anti-derivative of g(f), Le.. Voğ(0) = =()); CLs denotes a logarithmic Sobolev constant satisfied by the p.. and the constants Co C₁ and C22 are given by
) (0+ (b+28² d 11) B²)
C = M No +
IV m
G = 6M2 + dd,
2d
(12 + 8(No 2b
13Co v/Co),
3
2d
Õ (12 +8K00 + 26
1) VC)),
3
d
3x
(Mro
MMo
+ 1 A b
2 log3 )
82 2CLS log IpPoll 00
log
+ B
mB
PROOF: The proof of Lemma A.I follows from Proposition 10 of Raginsky et al. (011
A.3 PROOF OF THEOREM 3.2
PROOF: By Theorem of Ma et al. (2015), Algorithm [66 (of the main text) works pre- conditioned SGLD algorithm with the pre-conditioner E,, and has the same stationary distribution as the algorithm (29 By 30 we have W2(PT, - for algorithm (2) under the given settings
of and Ny. Therefore, for Algorithm (16), we also have W W2(pT. P.) as T - oc by noting that E, is positive definite for any
A.4 PROOF OF LEMMA
PROOF: Since we assume that all samples are i.idd, it suffices to prove the lemma with the case n 1. In this case. G(0,z) G(0,2) 1 (x)())Te T (<=))) 1. Let g(0) Eg [G(, z)]
Jz G(O, (=)(0) be the expected gradient with respect to the stationary distribution nro
13

---

Under review as a conference paper at ICLR 2023
(A1) By assumption (i) and (ii), then by simple algebra
|[G(O, - -2 [+(+)| 11 -101-10-00000000 xx))001 + |0||
< (1+ () (1 + >)1101)) + 0g
|0]|
(31)
22 (1 +)) + - (1 (1+
|||||| < 0o.
Since the upper bound is independent of z, the expected gradient g(t) is well-defined with [[]( do for all 0 O.
(A2)
Given the explicit formulation of G(0, 2). the anti-derivative ( can be derived
1
(( z) 2x (r (z)))² + 20
1011
(32)
For any 2, derive the following bound
[g(0,=)] 20 2
20 p2
(33)
and
|G(0, z)|l S 0
(1 + ?
(34)
(A3) For any E 2,
1
[|G(0,2) G(0, 2|1l
((x) (xxx((( 0)
(0
-01
02
o
1
$
1|0 1
(35)
02
08
+
1||
19
(A4) For any 2,
(0,G((,,)) = (r - +(x)0)
|011
0
1
72
((())
+
|
(36)
0²
2
0
1
$2
10112
40
(A5) Since |[G(0, z)]l is uniformly bounded for all 2 Z given in (31 bounded by
the gradient bias is
2 [G(0, =) |(0) < 2 max [G 2)1 S 0² (1+ + 2 (1 7) +
1 |0l1 (37)
1
222
og
By some algebra, we can calculate the quadratic bound
|G(00=) - 0(0)1 < [10 B²), where M 4 >)² + ) + and B (2 )) Since the bound is uniform for all E Z, we can derive the desired bound for the expectation. (A6) By assumption (iii),
(38)
c010
3o8 1101
Ko log
de plog / 2
(39)
(1 20/ ||l11
< log
d0 < 00,
for any of < 3.
14

---

Under review as a conference paper at ICLR 2023
B MORE NUMERICAL RESULTS
B.1 SOFTMAX PROBABILISTIC POLICY
For the indoor escape environment, we adopt the Boltzmann exploration which selects an action with probability
exp(Bact a: ())
(als) = Enee (Ssp)
where Pan the action inverse temperature. When Ina small, the agent tends to explore random actions. In contrast, when Saa is large, the agent takes action greedily. Greedy Q-learning can be viewed as a special case of Boltzmann exploration, since po(s) argmax, o Q(s, a; 0) with probability 1 as Bact + c0. Moreover, the action probability ((p)(s) a) is differentiable with respect to 0. In this paper, we assume all RL algorithms follow the e-Boltzmann exploration with the action probability given by
PA (als) = + -)) 2 exp ( ((s
(40)
Eved
where is the random exploration rate. Note that as Bua > xo, c-Boltzmann exploration converges to e-greedy exploration.
B.2 STATE VALUE VISUALIZATION FOR THE INDOOR ESCAPING EXAMPLE
In this section, we demonstrate the state value approximation for both linear and nonlinear function approximations. By following Mclo & Ribeiro 2007), define the Q-function and the feature map as
((s)))=((sTT0) Σ a'EA
0g Ta) and ((s) (((t
(41)
where the parameter vector 0 = 00 05, 0g 0 is partitioned into 4 subspaces, 1() is the indicator function, and
R is basis function which be linear basis function, Gaussian kernel, etc. For each set of basis functions, the agent is allowed to update its parameter every 60 steps with 20 transition tuples as training data for 30.000 episodes. In every experiment, we collect the last 1,000 updates as the samples from the stationary distribution. In figure 6 we compare different function approximation of SARSA type LKTD. The optimal policy used simulate the true state-value is defined as
(z v) ( N, if 2. E, if x < y.
(42)
where the discount factor y = 0.9. In figure 6b 6c and 6d the state-value surfaces are estimated using the average state-values of last 1000 updates with respect to linear-based approximation, kernel-based approximation and deep neural network approximation respectively. We showed that LKTD algorithm successfully estimate the state-values for all three function approximations.
B.3 CONTINUATION OF UNCERTAINTY QUANTIFICATION FOR THE INDOOR ESCAPING EXAMPLE
This section is supplement to Section 4.1 of the main text, which includes more numerical results for comparison of the proposed LKTD algorithm and the popular Adam algorithm (Kingma & Ba 2014].
In each run of LKTD, we set the batch size to 250, fix the inverse temperature 3 1, and update the network parameters every 50 steps. For a and ag in Remark 1 we set و² and oj 25, where o² is estimated by the mean square TD error according to Assumption 2 and oă is chosen suitably for convergence and ignorable prior effects. Under the setting 3 = 1, LKTD converges to the stationary distribution p. (9) x exp( cxp(-ũ(0)). The parameters obtained in the last 1000 parameter updates are used parameter ensemble for performing the followed Bayesian inference tasks. The parameter ensemble naturally induces Q-value ensemble and a policy ensemble. More precisely, each parameter vector corresponds Q-function and greedy policy induced from the Q-function.
15

---

Under review as a conference paper at ICLR 2023
(a) Simulated State-value
(b) Linear-based approximation
(c) Gaussian-basse bassed approximation
(d) Deep neural network
Figure 6: State-value surface: (a) State-values of the center point in each grid with respect to the optimal policy. (b) Linear function approximation with linear basis. (c) Linear function approximation with Gaussian kernel. (d) Deep neural network with hidden-layers (16,16,16).
The experimental results are reported in Tables 2 and where each measurement is derived by averaging over 200 independent runs.
For comparison, Adam has been run with different learning rates, including 1.0e-2, 1.0e-3 and 1.0c-4. For each learning rate, it is run for 200 times independently, each run consisting of 10000 episodes. Similar to LKTD,, we use the parameters obtained in the last 1000 parameter updating steps parameter ensemble for performing the followed statistical inference tasks. The results are also summarized in Tables and
Table Bane@rrponn reports the estimation accuracy of Q-values, which is measured by the mean squared error between the mean of the Q-value ensemble and the Q-value of the optimal policy defined in easy to see that Adam produces about the same MSEs for all four actions with
cquation 42 learning rate of Ie-2, and it produces more varied MSEs with other learning rates. LKTD produces almost the same MSEs as the best run of Adam.
Table 2: MSE(Q) for the indoor escaping example
Description
North
East
South
West
Average
Name
Learning rate
MSE(Q)
MSE(Q)
MSE(Q)
MSE(Q)
MSE(Q)
LKTD
le-4
0.119
0.120
0.100
0.101
0.110
Adam
le-2
0.113
0.113
0.096
0.096
0.105
Adam
le-3.
0.130
0.139
0.354
0.355
0.245
Adam
le-4
0.120
0.119.
0.396
0.392
0.257
Table3 compares the performance of the two algorithms optimal policy exploration, which is measured by MSE(ğ), the mean squared error between the proportions of action votes from the policy ensemble and the probabilities of mean optimal actions. The probabilities of mean optimal actions describe the variety of optimal actions probabilities of mean optimal actions are the same across all optimal actions. For example, suppose that both action N and action E are optimal state, then each has optimal action probability
state. That is, if multiple actions are all optimal, the
16

---

Under review as a conference paper at ICLR 2023
of 0.5; therefore, policy ensemble that fails to explore all optimal policies (due to a local trap issue) might only vote for one of the two actions. For LKTD, the smaller values of MSE(p) in the north and east actions imply that it provides better optimal policy exploration than Adam.
is worth mentioning that Adam with a learning rate of 1c-2 also produces similar MSE(Ø) values to LKTD, but its SOAR in table 4 is worse than LKTD, which implies that LKTD provides more reliable policy than Adam.
Table MSE(ý) for the indoor escaping example
Description
North
East
South
West
Average
Name
Learning rate
MSE(p)
MSE(f)
MSE(i)
MSE(ý)
MSE(Ø)
LKTD
le-4
0.044
0.044
0.002
0.002
0.023
Adam
le-2
0.044
0.044
0
0.022
Adam
le-3
0.047
0.047
0
0
0.0235
Adam
le-4
0.058
0.058
0
0.029
Table Sub-optimal action rate for the indoor escaping example
Description
North
East
South
West
Name
Learning rate
SOAR
SOAR
SOAR
SOAR
Average
LKTD
le-4
0.098
0.102
0.026
0.026
0.063
Adam
le-2
0.134
0.135
0.005
0.005
0,070
Adam
le-3
0.493
0.495
0
0.247
Adam
lc-4
0.502
0.495
0
0.248
Table 5 compares the coverage rates the optimal Q-values by different algorithms. By considering calculate the coverage rate of optimal Q-values. Table 5 shows that LKTD has consistent coverage rates around 95% for all actions, while Adam with small learning rates failed cover over 50% of the optimal Q-values. Although Adam with large learning rate (le-2) can provide a good exploration for optimal policies, however, due to its optimization nature, it cannot provide correct confidence coverage for the Q-value.
100 grid points over the entire state space,
Table 5: Coverage of the optimal Q-value for the indoor escaping example
Description
North
East
South
West
Name
Learning rate
CR
CR
CR
CR
Average
LKTD
le-4
0.935
0.928
0.972
0.968
0.951
Adam
le-2
0.883
0.883
0.887
0.884
0.884
Adam
le-3
0.635
0.634
0.314
0.316
0.475
Adam
le-4
0.263
0.263
0.109
0.108
0.186
Remark 2 Instead of using semi-gradient, the true gradient is used in all of the indoor escaping experiments of LTKD and Adam. Note that the semi gradiet is biased, which can lead to incorrect stationary distribution. In order to keep the comparison subjective, the objective function for Adam at each time is given by
L(0) Σ
- -
(43)
B.4 HYPERPARAMETER SETTINGS FOR CLASSIC CONTROL PROBLEMS
Our experiment is based on the framework of RL Baselines3 Zoo. For Adam optimizer, the hyperpa- rameters provided by Zoo package. For LKTD, we and 1/B 0.01, and 0y can be
17

---

Under review as a conference paper at ICLR 2023
chosen suitably according to the parameter size. The DQN agents trained using 2-layer dense neural network with hidden layers of size (256, 256). All the hyperparameters are shown in table 6 Note that if the gradient step is set to -1, the agent conducts many gradient steps steps done the environment between two updates.
Table 6: Hyperparameters
Environment
CartPole-vl
MountainCar-vo
LunarLander-v2
Acrobot-vl
Hyperparameters
LKTD
Adam
LKTD
Adam
LKTD
Adam
LKTD
Adam
learning rate
2.5e-5
2.3e-3
le-4
4e-3
5e-6
6.3e-4
5e-5
6.3e-4
1/8 (temperature)
0.01
0.01
0.01
0.01
de (prior)
5
20
5
a (observation)
I
I
target update interval
1000
600
250
I
250
(discount factor)
0.99
0.98
0.99
0.99
training steps
5e4
2e5
1.2e5
le5
batch size
64
64
64
64
learning starts train freq
lc5
lc3
0
0
256
gradient steps
128
8
-1
-1
exploration fraction
0.16
0.2
0.12
0.12
exploration final cps
0.04
0.07
0.25
0.1
0.1
18


Reviews:
[{"review_id": "fdT6R8yqqb", "reviewer": "Reviewer_Je2E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Langivinized Kalman Temporal Difference (LKTD) learning, a new algorithm for value based reinforcement learning (RL). LKTD optimizes the mean squared Bellman error (MSBE) objective function using SGLD, in contrast to classical algorithm such as residual gradient which applies gradient descent, and temporal difference learning which uses semi-gradient update. The paper proves the convergence of LKTD and provides empirical studies in synthetic problems and simple control problems. ", "summary_of_the_review": "I recommend rejecting this paper as the advantage of the proposed algorithm is not clear at this point. I hope the authors can address my questions in the rebuttal. ", "strength_and_weaknesses": "Strength \n\nThe paper provides a novel perspective to study value-based RL. The main idea is to optimize MSBE using SGLD which is novel to me. Following this perspective, convergence guarantee of the proposed algorithm can be derived. The paper also provides empirical study to show the effectiveness of the algorithm. \n\nWeakness\n\nAs discussed above, in my opinion the main contribution of the paper is to use SGLD to optimize MSBE. So the key to decide if the contribution is significant enough is to understand if this new method can introduce some advantage from either empirical or theoretical perspective. However this is not clear to me as explained in details below. Please correct me if I missed anything important. I am happy to adjust my score based on how well the authors answer the questions in the rebuttal. \n\n1. One classical method in RL is the residual gradient (RG), which directly optimizes MSBE using gradient descent [1]. In fact, as discussed in Remark 1, the gradient is the same as RG (plus a regularization of the parameter). As a gradient-based method, the convergence of RG is robust with both linear and non-linear function approximation. I am almost sure that under the data collection assumption used in the paper, it can also be proved that RG converges. If that’s the case, what’s the advantage of LKTD compared to RG? \n\n2. It is known that one problem of RG is the double sampling issue. That is, in stochastic environment, to obtain an unbiased gradient estimate, we need two independent samples. It seems to me that LKTD has the same problem? \n\n3. The author suggests that equation 23 can be used to connect LKTD with DQN. However, besides the semi-gradient TD update, DQN also has other components such as replay buffer and target network. Also, the data stored in the replay buffer is collected from a very different way from Assumption 1. How do these fit the analysis of LKTD? \n\n4. It seems that one of the main contributions is to prove LKTD converges under Assumption 1. However, Assumption 1 basically considers an on-policy setting, in which case the convergence of TD with function approximation can also be proved, see [2,3] for example. Again, what’s the advantage of LKTD compared to TD?  \n\n5. There are no related works about RG and TD discussed in the paper. \n\n6. It’s good to see that the paper also test LKTD empirically. But from the experiment results, it is now clear if LKTD significantly outperforms a TD based algorithm. Also, the test domains might be too simple. I think it will be very convincing if the author can show competitive performance compared to TD based algorithm in more challenging  problems. \n\n7. In the experiments the baseline is Adam. But Adam is an optimization algorithm, what's the objective used in the baseline algorithm? For the control problem, does it mean DQN with Adam as the optimizer? \n\n[1] Baird, L., 1995. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995 (pp. 30-37). Morgan Kaufmann.\n\n[2] Bhandari, J., Russo, D. and Singal, R., 2018, July. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory (pp. 1691-1692). PMLR.\n\n[3] Tsitsiklis, J. and Van Roy, B., 1996. Analysis of temporal-diffference learning with function approximation. Advances in neural information processing systems, 9.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eWhLZz81oF", "reviewer": "Reviewer_2cQs", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Langevin-style approach to posterior approximation in TD learning.\nThe idea is that using this style of MCMC learning we can get useful uncertainty estimates, and better decision making as we scale to deep RL.", "summary_of_the_review": "Overall I think this paper is taking on an interesting problem, with a potentially interesting approach.\nHowever, there are some glaring holes in the related work... also, the experimental results are a bit muddled/unclear.\nI think that, at a minimum, revisiting the paper in light of that related work should be the first step for this paper.\nIn its current form, I do not recommend acceptance.", "strength_and_weaknesses": "There are a few things to like about the paper:\n- This paper is taking on a \"big\" problem that would be of very high value to the ML community.\n- The general idea of using Langevin-style updates is appealing, and may hold promise in the future.\n- The general progression of algorithms, theorems and then empirical results is a good structure.\n\nHowever, there are some significant shortcomings in this paper:\n- There are many pieces of this algorithm that don't quite make sense... why go through all the problems of Bayesian inference to then use \"epsilon-Boltzmann exploration\"? This is clearly an inefficient dithering scheme, if you were going to use this then you really give up the main lure of Bayesian RL.\n- There are some important holes in the related literature, to highlight two:\n  1. Deep Exploration via Randomized Value Functions (Osband et al)\n  2. Langevin DQN (Dwaracherla + Van Roy)\n  - reviewing these papers + related works in the area will be very important to help make sense of this work. Paper (1) gives an overview of the importance of \"deep epxloration\" and possible ways to have tractable posterior approximation in deep RL. Paper (2) explores an approach very linked to the LKTD that you propose.\n- The quality/clarity of the experimental results is lacking... none of the curves are separated beyond the confidence intervals + the the effects are confusing overall. I would suggest using something like the \"DeepSea\" environments available in opensource \"bsuite\" to give you something clear and simple to get started with... similar to paper (2)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "dFHz-O9YrsB", "reviewer": "Reviewer_ZkBy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " This paper proposed a sampling framework LKTD for value-based RL that converges to a stationary distribution and enables efficient sample generation from the posterior. They show numerical experiments that indicate LKTD is comparable to Adam while outperforming it in robustness.", "summary_of_the_review": "In all, although this paper is well-written and seems to be theoretically sound, I believe for now its novelty and significance (especially over KTD, KOVA) haven't been validated by its experiments. I would like to see more experiments focusing on these aspects in the revision if possible. ", "strength_and_weaknesses": "## Strengths\n- This paper is in general well-written and relatively easy to follow. The proof seems to be theoretically sound. \n## Weakness\nI worry about the novelty and significance of this work. \n\n- For instance, it is stated in the major contributions that the new algorithm is scalable to dimension, and is computationally efficient, but I don't see a clear comparison of this work with KTD, KOVA in terms of computational complexity. \n- Also, all experiments are comparing LKTD with Adam rather than KTD, KOVA. I wonder is there a significant improvement compared with these two methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4JKzjp6_Xim", "reviewer": "Reviewer_soqG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to quantify the uncertainty in value based RL algorithms such as TD learning and Q-learning.  The approach is based on prior work that uses a state-space modeling approach to derive a Kalman filtering algorithm for TD learning (KTD and KOVA). Unless the function approximation is linear and a normality assumption holds, this requires approximations that have a quadratic complexity in the number of parameters.  Instead, the current work has the goal of applying Langevinized ensemble approach to the value estimation setting, in order to get a lower complexity.", "summary_of_the_review": "I believe the assumptions are completely unreasonable as I mentioned above. Besides that, there are serious issues with the clarity and significance due to the incremental nature of applying prior work. The main delta compared to previous work appears to be in applying the prior work to the linear model defined by the TD loss, which seems minor even ignoring the assumptions and clarity. ", "strength_and_weaknesses": "The most serious issue in my opinion is clarity and the assumptions made wrt theory that seem hard to justify. For example, Assumption 2 appears to be completely unreasonable -- $\\theta_t$ is the parameter for the model, and the assumption says that the environment changes at each time step to fit the current model.\n\nIn connection with dynamic policy, what is the precise definition of the map that induces a stationary distribution for a given $\\theta$? This is never specified.\n\nThere is also a serious problem with clarity, where the density function in Assumption is called $\\pi(z_t | \\theta_{t-1})$. My understanding is that the authors use $\\pi$ for a prior on the parameter space $\\theta$. How did this prior in the param space morph into a distribution on the samples?\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject