Decorrelated Double Q-learning
Authors: GANG CHEN
Venue:  (2021)
Keywords: q-learning
control variates
reinforcement learning

Abstract: Q-learning with value function approximation may have the poor performance because of overestimation bias and imprecise estimate. Specifically, overestimation bias is from the maximum operator over noise estimate, which is exaggerated using the estimate of a subsequent state. Inspired by the recent advance of deep reinforcement learning and Double Q-learning, we introduce the decorrelated double Q-learning (D2Q). Specifically, we introduce the decorrelated regularization item to reduce the correlation between value function approximators, which can lead to less biased estimation and low variance. The experimental results on a suite of MuJoCo continuous control tasks demonstrate that our decorrelated double Q-learning can effectively improve the performance.

Under review as a conference paper at ICLR 2021
DECORRELATED DOUBLE Q-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Q-learning with value function approximation may have the poor performance because of overestimation bias and imprecise estimate. Specifically, overestima- tion bias is from the maximum operator over noise estimate, which exaggerated using the estimate of a subsequent state. Inspired by the recent advance of deep reinforcement learning and Double Q-learning, we introduce the decorrelated dou- ble Q-lcaring (D2Q). Specifically, introduce Q-value function utilizing con- trol variates and the decorrelated regularization to reduce the correlation between value function approximators, which lead less biased estimation and low variance. The experimental results on a suite of MuloCo continuous control tasks demonstrate that our decorrelated double Q-learning can effectively improve the performance.
INTRODUCTION
Q-learning Watkins & Dayan 1992) as a model free reinforcement learning approach has gained popularity, especially under the advance of deep neural networks Mnih et al. (2013). In general, it combines the neural network approximators with the actor-critic architectures Witten (1977): Konda & Tsitsiklis (1999), which has an actor network control how the agent behaves and critic to evaluate how good the action taken is.
The Deep Q-Network (DQN) algorithm Mnih et al. (2013) firstly applied the deep neural network to approximate the action-value function in Q-learning and shown remarkably good and stable results by introducing target network and Experience Replay buffer to stabilize the training. Lillicrap al. proposes DDPG Lillicrap et al. (2015), which extends Q-learning to handle continuous action space with target networks. Except the training stability, another issue Q-learning suffered overestima- tion bias, which was first investigated in Thrun & Schwartz (1933). Because of the noise in function approximation, the maximum operator in Q-learning can lead to overestimation of state-action val- ues. And, the overestimation property is also observed in deterministic continuous policy control Silver & Lever (2014). In particular, with the imprecise function approximation, the maximization of noisy value will induce overestimation to the action value function. This inaceuracy could be even worse (e.g. error accumulation) under temporal difference learning Sutton & Barto (ottoo(199)) (1998), in which bootstrapping method is used to update the value function using the estimate of subsequent state.
Given overestimation bias caused by maximum operator noise estimate, many methods have been proposed to address this issue. Double Q-learning van Hasselt (2010) mitigates the overestimation effect by introducing two independently crities estimate the maximum value of set of stochastic values. Averaged-Doo estimates, which results in a more stable training procedure, as well as reduces approximation error variance in the target values. Recently, Twin Delayed Deep Deterministic Policy Gradients (TD3) Fujimoto et al. (2018) extends the Double Q-learning, by using the minimum of two critics to limit the overestimated bias in actor-critic network. A soft Q-learning algorithm Haarnoja et al. (2018), called soft actor-critic, leverages the similar strategy as TD3, while including the maximum entropy to balance exploration and exploitation. Maxmin Q-larnnigg Lan et al. (2020) proposes the use of an ensembling scheme to handle overestimation bias in Q-Learning.
Anschel et al. (2017) takes the average of previously learned Q-values
This work suggests alternative solution to the overestimation phenomena, called decorrelated double Q-learning based on reducing the noise estimate in -aluues. On the one hand, we want to make the value function approximators independent as possible mitigate overestima-

---

Under review as a conference paper at ICLR 2021
tion bias. On the other hand, should reduce the variance caused by imprecise estimate. Our decorrelated double Q-learning proposes an objective function to minimize the correlation of two eritics, and meanwhile reduces the target approximation enror variance with control variate meth- ods. Finally, we provide experimental results on MuloCo games and show significant improvement compared to competitive baselines.
The paper is organized as follows. Section 2 we introduce reinforcement learning problems, notations and two existed Q-learning variants address overestimation bias. Then we present our D2Q algorithm in Section 3 and also prove that in the limit, this algorithm converges to the optimal solution. In Section 4 we show the experimental results on MuJoCo continuous control tasks, and compare it to the current state of the art. Some related work and discussion is presented in Section 5 and finally Section conttionnococcuudeee the paper.
BACKGROUND
In this section, introduce the reinforcement learning problems and Q-learning, as well as notions that will be used in the following sections.
2.1 PROBLEM SETTING AND NOTATIONS
We consider the model-free reinforcement learning problem (i.e. optimal policy existed) with quential interactions between an agent and environment (Suttnn & Barto (1998) in order to maxi- mize cumulative return. At every time step I, the agent selects action d, in the state 8; according its policy and receives a scalar reward iz (8t, Gz). and then transit to the next state 84-1- The problem modeled Markov decision process (MDP) with tuple: (8, A, p(so), p(s (s,())))))))))))))))())) ot), r(st, az), 7). Here, S and A indicate the state and action space respectively, p(so) is the initial state distribution. D($t+1|S;) az) the state transition probability to 8t-1 given the current state 8t and action as r(st, de) is reward from the environment after the agent taking action a in state and y is discount factor, which is necessary decay the future rewards ensuring finite returns. We model the agent's behavior with (a(s))) which is parametric distribution from neural network.
Suppose we have the finite length trajectory while the agent interacting with the environment. The return under the policy æ for a trajectory T = (st, az) )
[(0) Byyma(t) |r(r)] Ey (R))
E ET 7(((s,,))) at)l
where (T) denotes the distribution of trajectories,
p(T) (50, an, S1, ST,
T = p(s)) II To(a (s)
The goal of reinforcement learning learn policy æ which can maximize the expected returns = arg max (() = arg max
(3)
The action-value function describes what the expected return of the agent is in state and action under the policy т. The advantage of action value function is to make actions explicit, so we can select actions even in the model-free environment. After taking an action a; in state s; and thereafter following policy π, the action value function is formatted as:
Q=((st a.) Esirph Rd ar] Bacpp a
(4)
i=t
To get the optimal value function, max, Q=(ss. az), and the corresponding optimal policy æ can easily derived by x x'(s) E arg max,, Q* (S), at).
the maximum over actions, denoted as Q*(ss,
2

---

Under review as a conference paper at ICLR 2021
2.2 Q-LEARNING
O-learning, as an off-policy RL algorithm, has been extensively studied since it was proposed Watkins & Dayan (1992). Suppose neural network parametrized by 0 to approximate Q-value in the continuous environment. To update Q-value function, we minimize the follow loss:
L(G = Essrpe a [[Q(st, (0)(()))))))))) ax; 02) ye)
(5)
where 3/2 = r(tt, (4) + max4,+1 Q(++11, a++1; 09) is from Bellman equation, and its action a++1 taken from frozen policy network (actor) to stabilizing the learning. actor-critic methods, the
policy ग S A, known as the actor with parameters 0=, can be updated through the chain rule in the deterministic policy gradient algorithm Silver & Lever (014)
VJ(0") = Ex p- V Qs, a; 09) (a==<(=00))) Ve- (==;0=;;))))
(6)
where ((s, a) is the expected return while taking action in state s, and following π after.
One issue has attracted great attention is overestimation bias, which may exacerbate the situation into a more significant bias over the following updates if left unchecked. Moreover, an inaccurate value estimate may lead to poor policy updates. To address Double Q-Jearning van Hasselt (2010) use two independent critics qı (st, az) and q2 (s az), where policy selection uses a different critic network than value estimation
qi (3t, ar) = r(st, at) + 792(5 arg max qı (5t+1, ac+1; ( (22)
a++1
(2(3,, ar) = ((ss,ac) + 7b1 (5e+1, arg max q2 (8:41; ae+1; 09);; 0a))
ds+1
Recently, TD3 of them below:
TD3/Fujmmoo Fujimoto et al. (018) uses the similar two q-value functions, but taking the minimum
U/ (8t, ux) + ymin (21 (8++1, (8 )), (8 (8++1)))
Then the same square loss in Eq. can be used to learn model parameters.
3 DECORRELATED DOUBLE Q-LEARNING
In this section, we present Decorrelated Double Q-learning (D2Q) for continuous action control with attempt to address overestimation bias. Similar to Double Q-learning, to approximate (8t,, oz). Our main contribution to borrow the idea from control variates to decorrelate these value functions, which can further reduce the overestimation risk.
two q- vavue functions
3.1 Q- VALUE FUNCTION
Suppose we have two approximators qı (st, az) and q2(8,, az), D2Q uses the weighted difference of double q-value functions to approximate the action-value function at (st,a;). Thus, we define Q-value as following:
Q(st, = (st, a.) (gg2 ax) - E(gg(st. oz)))
(
where (2 aa)) ((g2(st, at)) is to model the noise in state and action a,, and / is the correlation coefficient of 91 (st, at) and qa (St, az). To understand the expectation E(ge (st, a,)), it is the average over all possible runs. Thus, the weighted difference between qn (st, 04) and qa(st, a.) attempts to reduce the variance and remove the noise effects in Q-learning.
To update qı and 92, we minimize the following loss:
T((0))
= Barpps -ov (8t, 0x: 6t) y,)² + T. Ber ard (1:((, az; 042) - y)
+ AlE aumn (corr(q) (st, a,; 0²)), (2( ari 002 )}1
(9)
where 80 {8m, 00,) and yr can be defined
y/ r(st, a+) (011+11,
(10)
3

---

Under review as a conference paper at ICLR 2021
where (sL+11, a(+1) is the action-value function defined in Eq. to decorrelate qı (5(+1: ac+1) and (2(1++1, ds+1), which are both from the frozen target networks. In addition, we want these two qr value functions as independent as possible. Thus, we introduce corr(gi (st, ae; 80) (, 92 92(ss, oe; 8 )), which measures similarity between these two q-value approximators. In the experiment, our method using Eq. T0 can get good results Halfcheetah, but it did not perform well on other MuJoCo tasks.
To stabilize the target value, we take the minimum of Q(s++11 ar4-1) and q2 1, a++1) 1) in Eq. 10
as TD3 Fujimoto et al. (208)). Then, it gives the target update of D2Q algorithm below
r(st, at) + (
(2( 0c+1))
(11)
And the action 0++1 is from policy 4++1 ( 05), which can take similar policy gradient as in Eq. [6] Our D2Q leverages the parametric actor-critic algorithm, which maintains two q-value approixmators and single actor. Thus, the loss in Eq. tries minimize the three terms below,
corr(qi (s ax; ). 92($,, ae; (2)) +0
(st, ax; 00:) +
42 (8t; Oe; 872) + y/
At each time step, we update the pair of critics towards the minimum target value in Eq. I1 while reducing the correlation between them. The purposes that we introduce control variate (r at) are following: (1) Since we use (2($): at) - (( 0z)) to model noise, if there is no noise, such that (2(S), 0z) E(q2 (())())))))) az)) O, then have y/ r(st, a,) + min(Q" (st, az), 92(s, dz)) rss,, de) + min(() (St, at), Q2 (st, az)) via Eq. I1 which is exactly the same as TD3. (2) In fact, because of the noise in value estimate, we have (5( ae) ((g2( a;)) j O. The purpose we introduce (2( at) is to mitigate overestimate bias in Q-learning. The control variate introduced by (2 a²) will reduce the variance of (s;,, at) stabilize the learning of value function.
we claim that our D2Q algorithm is to converge the optimal in the finite MDP settings. There is existed theorem in Jaakkola al. (1994), given the random process {A,} taking value in IR" and defined as
Convergence analysis:
A141(5;, az) (1 (s (.))) a.) + as az a()s,aa)
(12)
Then A, converges zero with probability 1 under the following assumptions:
1. 0 a4 < 1, do and E, az())) < xo
2. [ $ < c with 0 0<7 and a & 0
+ 3. var(F( & C11 +|A)))))) for C> 00
where Fe is a sequence of increasing a-field such that as az) and A, are F, measurable for t 1,2,...
Based on the theorem above, we provide sketch of proof which borrows heavily from the proof of convergence of Double Q-learning and TD3 below: Firstly, the learning rate da satisfies the condition 1. Secondly, variance of r(st, as) is limit, so condition holds. Finally, we will prove that
condition 2 holds below.
A441 a:) = (1 as at at)) Q" (st,
+ an (:) 7 at), (2 0z)) -
(1 (3),
+ (1
(13)
where ((() defined as:
F,(se, ar) = r. + min(Q(ss, ar), (2) ar)) - Q' ax)
+ min (2(8, ar)) (s), + (((s, ~Q(ss, at)
= rr + (Q(,,, at) Q* (xt, ar) + + a:), (
7Q( 01)
= =F, +
(14)
4

---

Under review as a conference paper at ICLR 2021
Since have E[F (s), 1 >|ااا| under Q-learning, the condition 2 holds. Then a.)) - Q(st, ae) converges to with probability 1.
need to prove 9₁ = m Q(st. ac)
min((((s,,44)))(()),)))) ( az) + qu (51, 01) - Q(s,,
0) - min((((s, (()
(Q( at) 92( ax))
= min((gis(, (e) - 92(3), at) ((02(1, ((,
+1 (5t. (2($1, ((g2(s), E(g2(st, a)))
(15)
Suppose there exists very small 61 and 62, such that / (st, 92(8) at) s 81 and (g2(st, at) Elg2(s;, az))l & 82, then we have
min(Q(st, at). (2( a²) Q(s,,
=2(19n at) - (8t, at) + Blgz(s,, (192(s),4))))
=2(61 -.82) 48
(16)
where d = max(61,00 52). Note that 381, lqi (st, az) (2, a, S ò1 holds because (1)1) =
19 (st, ae) d2 (st, ax) converges to zero. According Eq. both aı (s), and (2(5), are
updated with following
(1t1 at) q0(s,, at) a))s).a))() (((s),a))
(17)
Then we have 4 (q,, (2) = A,(y ( - as ax) converges 0 as the learning rate satisfies 0 (r az) 1.
= (1 - an a.))),((1
3.2 CORRELATION COEFFICIENT
The purpose we introduce corr(q (st, at); (8,, de)) ) in Eq. 9 is to reduce the correlation between two value approximators di and 92- In other words, we hope qi (8t, az) and q2(8t; 0z) to be as independent as possible. In this paper, we define corr(y1, 42) as:
corr (8t, at), (2(s), = cosine(fa (St, at), fax (st,
where cosine(a, b) is the cosine similarity between two vectors and b. fg( 0z) is the vector representation of the last hidden layer in the value approximator q(St, ae). In other words, we constrain the hidden representation learned from (z) and (2 ax) in the loss function, with attempt to make them independent.
According to control variates, the optimal B in Eq. 8 is:
coo(gi ( (sc, ac), a.))
var(gi (q) (sc. a4))
where cov is the symbol of covariance, and var represents variance. Considering it is difficult to estimate 3 continuous action space, we take an approximation here. addition, reduce the number of hyper parameters, we set 3 corr(qı (st, az), 92 az)) in Eq. 8 to approximate the correlation coefficient of qı (Be, 0e) and g2(ss, at) since hard to get covariance the continuous action space.
3.3 ALGORITHM
We summarize our approach in Algorithm. Similar to Double Q-learning, we use the target networks with slow updating rate keep stability under temporal difference learning. Our tributions are two folder: (1) introduce the loss to minimize the correlation between two critics, which can make qi (st, az) and 92(st, ar) random possible, and then effectively reduce the overestimation risk; (2) add control variates to reduce variance in the learning procedure.
4 EXPERIMENTAL RESULTS
In this section, we evaluate our method on the suite of MuJoCo continuous control tasks. We down- loaded the OpenAl Gym environment, and used the MuloCo v2 version of all tasks to test

---

Under review as a conference paper at ICLR 2021
Algorithm Decorrelated Double Q-learning
Initialize pair of critic networks Qi (s, a; 0)),, (2(s, a; 040) and actor (s;; 07) with weights 09
{0m, _8²) and 6
Initialize corresponding target networks for both critics and actor 0º' and 0=:: Initialize the total number of episodes N, batch size and the replay buffer R
Initialize the coefficient À in Eq. |
Initialize the updating rate т for target networks
for episode= = to N do
Receive initial observation state so from the environment for t = 0 to I" do
Select action according to a; = ( (*) 6,, (((,o)
Execute action 0₂ and receive reward it,, done, and further observe new state 81+1
Push the tuple (s,, at, r2, done, 9 into R
//sample from replay buffer
Sample a batch of = (8t, at, done, St+1) from R
0t+1 (8 8T) + with clip, ~ ~N(,
Compute Q(st, 0.) with target critic networks according to Eq.
Compute target value y via Eq. [1]
Update critics qi and @2 by minimizing L(09) in Eq. 9
Update actor (s; (T) by maximizing (5,,, ae) value in Eq. end for Update the target critics 0 (1 7)09' + 70 Update the target actor d (1 T)( T8
end for Return parameters 0 = 100,00 ).
method. We compared our approach against the state of the art off-policy continuous control algo- rithms, including DDPG, SAC and TD3. Since SAC requires the well-tuned hyperparameters to get the maximum reward across different tasks, we used the existed results from its training logs pub- lished by its authors. To obtain consistent results, we use the author's implementation for TD3 and DDPG. In practice, while we minimize the loss in Eq. 9 we constrain B E (0, 1). In addition, we add Gaussian noise to action selected by the target policy in Eq. [T] Specifically, the target policy (aip(N(O,,7)) cc with 0.5.
adds noise as a++1 T($++1:0T) + e, where
Without other specification, we use the same parameters below for all environments. The deep architecture for both actor and critic uses the same networks TD3 Fujimoto et al. (2018), with hidden layers [400, 300, 300]. Note that the actor adds the noise (0, to its action space to enhance exploration and the critic networks have two Q- functions 91 (1 a) and 92(s, a). The mini- batch size is 100, and both network parameters are updated with Adam using the learning rate 10-3 In addition, also target networks including the pair of critics and single actor improve the performance as in DDPG and TD3. The target policy is smoothed by adding Gaussian noise (O, 0.2) as in TD3, and both target networks are updated with = 0.005. We set the balance weight À = 2 for all tasks except Walker2d which we set X = 10. In addition, the off-policy algorithm uses the replay buffer with size 10 for all experiments.
We run each task for I million time steps and evaluate it every 5000 time steps with no exploration noise. We repeat each task 5 times with random seeds and get its mean and standard deviation respectively. And we report our evaluation results by averaging the returns with window size 10. The evaluation curves are shown Figures 1 and 3 Our D2Q consistently achieves much bet- ter performance than TD3 most continuous control tasks, including InvertedDoubleeendulun Walker2d, Ant, Halfchectah and Hopper environments. Other methods such as TD3 perform well on one task Reacher, but perform poorly other tasks compared our algorithm.
We also evaluated our approach on high dimensional continuous action space task. The Humanoid- v2 has 376 dimensional state space and 17 dimensional action space. In the task, we set the learning rate on Humanoid to be 3 x 10-4, and compared to DDPG and TD3. The result in Figure [bb demonstrates that our performance this task is on a par with TD33

---

Under review as a conference paper at ICLR 2021
2000
2000
20000
Reacher
(b) Humanoid
Figure I: shaded region represents the standard deviation of the average evaluation over nearby windows with size 10. On the MuJoCo tasks, our D2Q algorithm yields competitive results, compared to TD3 and DDPG.
The learning curves with exploration noise Reacher and Humanoid environments. The
7O0
3000
(a) InvertelDoubleeendulllm
Walker2d
Figure 2: The learning curves with exploration noise on the DivertedDoubleeenduuuu and Walker2d environments. The shaded region represents the standard deviation of the average evaluation over nearby windows with size 10. Our D2Q algorithm yields competitive results, compared to TD3 and DDPG.
(a) Ant
(b) Halfcheetah
(c) Hopper
Figure 3: The learning curves with exploration noise on the Ant, Halfcheetah and Hopper environ- ments. The shaded region represents the standard deviation of the average evaluation over nearby windows with size 10. Our D2Q algorithm yields significantly better results, compared to TD3 and DDPG.
The quantitative results over trials are presented in Table [ Compared SAC Haarnoja et al. (2018), our approach shows better performance with lower variance given the same size of training samples. demonstrates that our approach can yield competitive results, compared DDPG. Specifically, our D2Q method outperforms all other algorithms with much low variance on Ant, HalfCheetah, InvereedDoulleeendulnn and Walker2d. the Hopper task, our method achieve maximum reward competitive with the best methods such as TD3, with comparable variance.
TD3 and
RELATED WORK
Q-learning can suffer overestimation bias because it uses the maximum to estimate the maximum expected value. To address the overestimation issue Thrun & Schwartz (1993) in Q-learning, many approaches have been proposed to avoid the maximization operator a noisy value estimate. De- layed Q-learning Sttrehl al. (2006) tries to find c-optimal policy, which determines how frequent to update state-action function. However, it can suffer from overestimation bias, although it guarantees converge in polynomial time. Double Q-learning van Hasselt (2010) introduces indepen-

---

Under review as a conference paper at ICLR 2021
Table I: Comparison of Max Average Return over trials of 1 million samples. The maximum
value is marked bold for each task. = corresponds to a single standard deviation over trials.
Methods
Environments
D2Q
TD3
SAC
DDPG
PPO
HalfChoetah
9958.3
935.70 70
9636.95 859.06
2895.9644 # 3316.5
8577.29
1795.43
Hopper
3364.34 34 1 583.72 72
3223.75 514.2
2100.67 1 1051.6
2020.46
2164.70
Walker2d
4727.20 444.71
4582.82 525.60
3475.15 + 1508.71
1843.85
3317.69
5264.69
632.90
4373.44 + 1000.33
3250.49 1157.94 NA
1005.30
1082.20
Reacher
3.78 0.32
3.6 1 0.56
-6.51
-6.18
InyPendulum
1000 + 0.0
1000 + 0.0
NA
1000
1000
DnnDeubblePemaalmm
9200.6 186.22
8911.04 +: 750.58
NA
7741.28 +: 2195.87
8977.94
dently trained crities to mitigate the overestimation effect. Averaged-DQN Anschel et al. (2017) takes the average of previously learned Q-values estimates, which results in more stable training procedure, as well as reduces approximation error variance in the target values. A clipped Double Q- learning called TD3 Fujimoto et al. (2018) extends the deterministic policy gradient Siiver & Lever (014); Lillicrap et al. (2015) to address overestimation bias. In particular, TD3 uses the minimum of two independent critics to approximate the value function suffering from overestimation. Soft actor critic Haarnoja et al. (018) takes similar approach as TD3, but with better exploration with maximum entropy method. Maxmin Q-learning Lan et al. (2020) extends Double Q-learning and TD3 multiple critics to handle overestimation bias and variance.
Another side effect of consistent overestimation Thrun & Schwartz (1933) in Q-learning is that the accumulated error of temporal difference Sutton & Barto (998) can cause high variance. To reduce the variance, there are two popular approaches: baseline and actor-critic methods [Witten 1977);; Konda & Tsitsiklis (1999.. policy gradient, we can minus baseline in Q-value function to reduce variance without bias. Further, the advantage actor-critic (A²C) Mnih et al. (2016) introduces the average value to each state, and leverages the difference between value function and the average to update the policy parameters. Schulman et al proposed the generalized advantage value estimation Schulman al. (2016), which considered the whole episode with an exponentially- weiighted esti- mator of the advantage function that is analogous to TD()) to substantially reduce the variance of policy gradient estimates at the cost of some bias.
From another point of view, baseline and actor-critic methods can be categories into control variate methods Greensmith et al. (200I). Greensmith et al. analyze the two additive control variate meth- ods thcoretically including baseline and actor-critic method to reduce the variance of performance gradient estimates in reinforcement learning problems. Interpolated policy gradient (IPG) Gu et al.) (2017) based control variate methods merges and off-policy updates to reduce variance for deep reinforcement learning. Motivated by the Stein's 's identity, Liu et al. introduce more flexible and general eetion-dependent baseline functions Liu et al. (2018) by extending the previous control variate methods used in REINFORCE and advantage actor-critic. In this paper, we present a novel variant Double Q-learning to constrain possible overestimation. We limit the correlation between the pair of vavulue functions, and also introduce the control variates to reduce variance and improve performance.
CONCLUSION
In this paper, we propose the Decorrelated Double Q-learning approach for off-policy value-based reinforcement learning. We use pair of critics for value estimate, but we introduce a regulariza- tion term into the loss function to decorrelate these two approixmators. While minimizing the loss function, it constrains the two q-value functions be as independent as possible. In addition, con- sidering the overestimation derived from the maximum operator over positive noise, we leverage control variates to reduce variance and stabilize the learning procedure. The experimental results on suite of challenging tasks in the continuous control environment demonstrate our approach yields on par or better performance than competitive baselines. Although we leverage control variates in q- valul function, approximate the correlation coefficient with simple strategy based the similarity of these two q-functions. In the future work, we will consider a better estimation of correlation coefficient in control variate method.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-1 1 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 176-185. PMLR, 2017.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICML, volume 80 of JMLR Workshop and Conference Proceedings, pp. 1582-1591. 10L-11591.MM..000 JMLR.org. 2018.
Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. In Journal of Machine Learning Research, pp. 1471-1530.
MIT Press. In press, 2001.
Gu, T. Lillicrap, R. E. Turner, Z. Ghahramani, Schölkopf, and Levine. Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning. In Advances in Neural Information Processing Systems 30, pp. 3849-3858. Curran Associates, Inc., 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, volume 80 of JMLR Workshop and Conference Proceedings, pp. 1856-1865. JMLR.org, 2018.
T. Jaakkola, M. Jordan, and S. P. Singh. On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 616::18851 1994.
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Processing Systems, pp. 1008-1014. MIT Press, 1999.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In ICLR, 2020.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, Abs//509.00991. 2015.
Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng. and Qiang Liu. Action-dependent control variates for policy optimization via stein identity. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. Open 2018
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learn- ing Workshop. 2013.
Volodymyr Mnih, Adrià Puigdomenech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P. Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1928-1937. JMLR.org, 2016.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
David Silver and Guy Lever. Deterministic policy gradient algorithms. In ICML, 2014.
Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. Pac model- free reinforcement learning. In In: ICML-06: Proceedings of the 23rd international conference on Machine learning, pp. 881-888, 2006.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. Adaptive com- putation and machine learning. MIT Press, 1998.

---

Under review as a conference paper at ICLR 2021
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Michael Mozer, Paul Smolensky, David Touretzky, Jeffrey Elman, and Andreas Weigend (eds.), Proceedings of the 1993 Connectionist Models Summer School, pp. 255-263. Lawrence Erlbaum, 1993.
Hado van Hasselt. Double q-learning. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of meeting held 6-9 December 2010, Vancouver, British Cohumbia, Canada, 2613-26211 Curran Associates, Inc., 2010.
Christopher C. Watkins and Peter Dayan. 1992.
Q-learning.
Machine Learning, pp. 279-292,
lan H. Witten. An adaptive optimal controller for discrete-time markov environments. Information and Control, pp. 286-295, 1977.
A APPENDIX
We add additional experiments on how our model will perform by varying À in this Appendix. We set À = [1, 2, 5, 10] respectively to run Million steps and evaluate its performance every 5000 steps, while keeping all other parameters same.
(a) Halfcheetah
Walker2d
Figure 4: The figures show how method will perform while adjusting parameter A. The shaded region represents the standard deviation of the average evaluation over nearby windows with size 10.
10


Reviews:
[{"review_id": "rwAmhWey89L", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper suggests an improvement over double-Q learning by applying the control variates technique to the target Q, in the form of $(q1 - \\beta (q2 - E(q2))$ (eqn (8)). To minimize the variance, it suggests minimizing the correlation between $q1$ and $q2$. In addition, it applies the TD3 trick. The resulting algorithm, D2Q, outperforms DDPG and competes with TD3.\n\nRecommendation\nI hope I haven’t misunderstood this paper, but I’ve found neither the theory nor the experiment convincing. Therefore I recommend a rejection.\n\nStrengths\nThe proposed algorithm is simple and straightforward to use.\n\nWeaknesses\n1. Theory\n(a) Minimizing the variance of eqn (8) requires maximizing the correlation between q1 and q2. If they are independent, what’s the point of including q2? Check out https://en.wikipedia.org/wiki/Control_variates\n(b) $E(q2))$ is the “average over all possible runs”. It’s unclear how it’s calculated. Maybe run a few identical RL experiments with different random seeds, just to get $E(q2)$? Feels wasteful to me.\n(c) Why would minimizing the squared cosine between last layer feature vectors lead to minimum correlation? If the feature for q2 is obtained from that of q1 through a deterministic 90º rotation, wouldn’t that result in a zero cosine but really strong correlation?\n(d) Why is it ok to ignore $var(q1)$ while computing $\\beta$? No theory is given here.\n\n2. Experiments\nIn Fig 1 - 3, D2Q sometimes outperforms and sometimes underperforms TD3. Because these two algorithms are so similar, I can’t tell whether the comparison is statistically significant.\n\nOther feedbacks\nPlease address questions raised above. Perform additional experiments to make the paper more convincing. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qAUWHDfCDQa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed \"decorrelated double Q-learning\" algorithm combines a few techniques to improve the performance of model-free RL, including control variates for reducing variance, decorrelated regularization for reducing bias, and a technique from TD3 for stabilizing learning. \n\nOverall, the ideas of this work are interesting and bring some insights for tackling the overestimation issue of Q-learning. Empirically, the proposed method shows some improvements over the existing ones. However, a few major concerns are as follows.\n\n- The theoretical analysis of convergence seems hand-waving and confuses me. For example, does the analysis only apply to the tabular case? (The authors don't seem to state this explicitly.) How does Eq. (17) follow from Eq. (9) (are we missing the gradient of the decorrelated regularization term)?\n\n- All experimental results are about reward vs. iteration curves, which are not convincing or insightful enough. For example, is there empirical evidence showing that the proposed algorithm does learn two decorrelated critics? \n\n- The structure of Section 3 may need some adjustment. In particular, in the current version, the formal definition of the correlation term (Sec 3.2) and the description of the full algorithm itself (Sec 3.3) appear after the convergence analysis of the algorithm (Sec 3.1), which looks weird.\n\nBased on the above comments, I think substantial improvements are needed for publication of this work.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iLayvWD1I87", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method that modifies double Q-learning by eliminating a linearly correlated part of one Q. I am not familiar with the proof of Double Q-learning and TD3, and thus find the proof of this paper hard to read as it omits the majority of proof by claiming it is similar to the proof of the aforementioned two algorithms. To name some of the part that confused me while reading: what is the definition of F^Q_t and c_t in (14)? Why does a small delta_2 exist in (16)? Why does Delta_t converge to zero as claimed in the line after (16)? Why is the randomness of s_{t+1} not mentioned in the subscripts of E's in (5) and (9)? etc. Therefore, I suggest the author(s) write a thorough proof and put it in the appendix to make the convergence analysis readable. I am also curious how the de-correlation term helps to improve the convergence in the analysis as it is the main contribution of this paper. Besides,  double Q-learning and TD are mostly used in function approximations. I wonder if the analysis can extend to some simple case of parameterized Q functions, e.g. linear approximations. The experiment part looks good to me as it compares D2Q with several sota algorithms and get satisfying results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fftMr1r4arM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I vote for rejecting. The core idea of the new algorithm looks interesting, but this paper does not provide convincing evidence theoretically and empirically. \n\nPros:\n\n1. This paper introduces a new variant of Double Q-learning to reduce the overestimation risk and reduce variance. \n2. In part of the experiments, the new algorithm demonstrates better performance than previous ones.\n\nCons:\n\n1. This paper does not provides enough insightful intuition and theoretical guarantees on the design of the new algorithm. There should be more explanation and evidence to help reader understand why for instance, the definition of Q-value in equation (8) makes sense. \n2. This paper is not well-written. There are missing references links, typos and missing definitions of some notations. For example, in the second paragraph of introduction, “Lillicrap et al.” is just text and not linked to a paper in the references and there are many other cases like this. Typos can be seen in many places as well, such as in the first paragraph of section 3.3, it should be ‘two-fold’ instead of ‘two folder’; in the last paragraph of section 5, ‘categories’ should be ‘categorised’. Moreover, in the convergence analysis, the meaning of many notations are not explained at all (e.g. $\\alpha_t$). The author only says “we provide sketch of proof which borrows heavily from the proof of convergence of Double Q-learning and TD3”, but without the definitions of the notations, the completeness of the paper is greatly undermined. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates some variants of the double Q-learning algorithm and develops theoretical guarantees. In particular, it focuses on how to reduce the correlation between the two trajectories employed in the double Q-learning strategy, in the hope of rigorously addressing the overestimation bias issue that arises due to the max operator in Q-learning. However, the reviewers point out that the proofs are hard to parse (and often hand-waving with important details omitted). The experimental results are also not convincing enough.     
Decision: Reject