Deep Reinforcement Learning With Adaptive Combined Critics
Authors: Huihui Zhang
Wu Huang
Venue:  (2021)
Keywords: overestimation
continuous control
deep reinforcement learning
policy improvement

Abstract: The overestimation problem has long been popular in deep value learning, because function approximation errors may lead to amplified value estimates and suboptimal policies. There have been several methods to deal with the overestimation problem, however, further problems may be induced, for example, the underestimation bias and instability. In this paper, we focus on the overestimation issues on continuous control through deep reinforcement learning, and propose a novel algorithm that can minimize the overestimation, avoid the underestimation bias and retain the policy improvement during the whole training process. Specifically, we add a weight factor to adjust the influence of two independent critics, and use the combined value of weighted critics to update the policy. Then the updated policy is involved in the update of the weight factor, in which we propose a novel method to provide theoretical and experimental guarantee for future policy improvement. We evaluate our method on a set of classical control tasks, and the results show that the proposed algorithms are more computationally efficient and stable than several existing algorithms for continuous control.

Under review as a conference paper at ICLR 2021
DEEP REINFORCEMENT LEARNING WITH ADAPTIVE COMBINED CRITICS
Anonymous authors Paper under double-blind review
ABSTRACT
The overestimation problem has long been popular in deep value learning, because function approximation errors may lead to amplified value estimates and subop- timal policies. There have been several methods deal with the overestimation problem, however, further problems may be induced, for example, the underesti- mation bias and instability. In this paper, we focus on the overestimation issues on continuous control through deep reinforcement learning, and propose a novel al- gorithm that can minimize the overestimation, avoid the underestimation bias and retain the policy improvement during the whole training process. Specifically, we add a weight factor to adjust the influence of two independent critics, and use the combined value of weighted critics to update the policy. Then the updated policy is involved in the update of the weight factor, in which we propose a novel method to provide theoretical and experimental guarantee for future policy improvement. We evaluate our method on a set of classical control tasks, and the results show that the proposed algorithms are more computationally efficient and stable than several existing algorithms for continuous control.
INTRODUCTION
The task of deep reinforcement learning (DRL) is to learn good policies by optimizing a discounted cumulative reward through function approximation. Although variety of control tasks have gained success through DRL Mnih al. (2013) (015): (2013/2015): Van Hasselt al. (2016); [ et al. (2015); Schaul et al. (2015); Lillicrap et al. (2015); Mnih et al. (2016). there still exists biases caused by function approximation errors, studied prior work Mannor et al. (2007).
Overestimation bias is rooted in Q-learning by consistently maximizing noisy value estimate, and was originally reported to be present in the algorithms, typically, like deep Q-network (DQN) Mnih et al. (2015) which is aimed for discrete control tasks. Since DQN adopts neural networks to approximate a cumulative future reward, it is unavoidable that the noise evolved from function approximation accompanies the whole training. Besides, DQN updates its policy by choosing the action that maximizes the value function, which may bring the inaccurate value approximation that outweighes the true value, i.e., the overestimation bias. This bias further accumulates every step via bootstrapping of temporal difference learning, which estimates the value function using the value estimate of subsequent state.
When the overestimation bias is harmful to a certain task, it will cause instability, divergence and suboptimal policy updates. To solve this, Double deep Q-networks (DDQN) Van Hasselt et al. (016)) alter the policy update strategy by taking actions based on relatively independent target value function instead of maximizing the original Q network. DDQN not only yields more accu- rate value estimates, but leads to much higher scores on several games Van Hasselt et al. (2016). However, due to the randomness of approximation errors, DDQN provides no theoretical bounds for the overestimation, and the unresolved overestimation induced underestimation makes the performance of DDQN worse than DQN in some cases, reported in Van Hasselt et al. (2016).
For the case of high- dineeniinnsional,, continuous action spaces, Deep Deterministic Policy Gradient (D- DPG) Lillierap et al. (2015) provide a model-free, off-po actor-critic method using deterministic policy gradients. Due to the slow-chi slw-chhaggigg policy in actor-critic setting. the current and target value estimates will be too dependent to circumvent bias if the solution of DDQN to overestima- tion is directly under the continuous control setting. Inspired by Double Q-learning Hasseli (2010)

---

Under review as a conference paper at ICLR 2021
which employ pair of independently trained crities to achieve less biased value estimation, the twin delayed deep deterministic policy gradient algorithm (TD3) Fujimoto et al. (2018) propose a clipped Double Q-learning variant choosing the lower target value an approximate upper-bound to estimate the true current value, which favors underestimation biases that will not accumulate during training because actions with high value estimation lying in TD3, First, by taking actions towards the maximization of lower action- value (Q-value) at every step, the policy improvement cannot be guaranteed, which may cause potential suboptimal policies and instability. Second, using the same target value to update two critic networks will make them less independent.
preferred. However, there are two problems
The paper has the following contributions. First, we propose combined value of two independent critics connected by a weight factor, and use it to update the policy instead of serving as shared target estimate for the two critics, to avoid losing independence. Second, we propose sign multi- plier which determines whether the updated combined value has increased. The objective function for updating the weight factor is the product of the sign and the combined value evaluated by the updated policy. Third, we present a novel algorithm for continuous control, which can minimize the overestimation bias while providing guarantee for future policy improvement. And theoretical proofss show that the proposed algorithm has the property of asymptotical convergence and expected policy improvement. Fourth, we further apply the proposed algorithm to create another algorithm, which can remove the systematic bias due to the probability mismatch between the behavior policy and the target policy existing in off-policy methods. Fifth, extensive evaluations are conducted to compare the proposed algorithms with some baseline algorithms, in terms of computational efficiency, convergence rate and stability.
unbiased framework
BACKGROUND
The task of reinforcement learning (RL) is to learn an optimal policy that maximizes a single return, i.e., the expected discounted cumulative reward of episode. During an episode, the agent continu- ally receives a sequence of observations by interacting with the environment before encountering an terminated state or the timeout. The return calculated as the expected sum of future rewards. DRL combines the neural networks with RL so that the return can be approximated by the parameterized function. In DRL, the agent follows behavior policy to determine future rewards and states, and store these observations in a memory, which will be randomly sampled over to train the network parameters and update the target Q-values. The target updates can be immediate, delayed or "soft",
Generally, the Q-value of DRL is represented as the expected discounted cumulative reward, an estimate function with respect to the state and action, which given by
(= = Epg (M) (0.)
('r(s,, at) a0
where r(s, is the immediate reward which is usually connected to the state-action pair, (s, a) is the value of initial state-action pair, and 7 E (0, 1) is the discount horizon factor for future rewards. Under the guidance of behavior policy p" (h) so. ag) is the joint probability of all state-action pairs during an episode given the initial state-action pair (50.a)).
When neural networks are used to approximate Q-values, the update of behavior policy is closely related to the Q network under the setting of Markov Decision Process (MDP). The Q-value function in ( takes the state- acteencction pair (s, a) as input and maps it to the Q-value. The foundation for updates of network parameters is the Bellman equation. Most of existing DRL algorithms adopt an independent target Q network to approximate based on Bellman equation and organized to form the general loss function for DRL. as Lillicrap et al. 2015)
essential part of the target value, which can be set
L(w) (e.....))) [(r 7Q(s' u/s' (()(00)))))))) Q(,, al)))
(2)
where a is the action drawn from a behavior policy based on s, r and s' are the immediate reward and next state received by interacting with the MDP environment, respectively. Overall, (s, a, s') is the tuple stored in the replay buffer at every step. Besides, (us'((0') is the target policy mapping s to the next action a' through target actor network parameterized by 0' in deterministic actor-critic
2

---

Under review as a conference paper at ICLR 2021
methods, instead of taking actions from the replay buffer. Moreover, is the parameter of current Q network, which is normally different from the target network parameter w'.
updated, the objective at the current iteration is to optimize the actor parameter 0, which is updated by maximizing the expected return ( = Ey [Q, ((+]0)[) In the case of continuous control, can be updated by gradient descent VoJ(O)-
Once the Q networks
ADAPTIVE DELAYED DEEP DETERMINISTIC POLICY GRADIENT
In TD3, the clipped variant of Double Q-Learning is proposed for actor-critic to reduce the overes- timation bias. Instead of using a pair of actors and critics to learn twice, TD3 upper-bounds the less biased value estimate by taking the minimum between two critic networks to offer target estimation for the update of future critics. Inspired by this work, we also adopt two critic networks and their target estimates, which are parameterized by w. Q2 w/ and I2,, respectively. Since we plan to deal with our problem through DRL, we can substitute sampled minibatches into the loss functions of which are given by
N
L(w)
(nn + o' - Q(sn, an kn))))
(3)
N
==1
L(
(nn + 2(S,,, (( on 12))?
(4)
where d' pss",, ([ss,,10 10') is the action taken from the target actor network, and (8nx Ony Puy 8y)) is the n-th tuple of minibatches stored in replay buffer.
In TD3, the minimum of two critics employed to serve as the target estimation value. However, updating two Q networks according to the same target estimate will make them less independent, which will further negatively affect the training efficiency. Besides, is more reasonable to apply the clipped variant of Double Q-Learning the procedure of actor update. Inspired by the Lagrange relaxation applied in constrained MDP problems Tessler et al. (2018), we propose a dual form of combined value function via a weight factor to determine actions reducing the harmful effect from overestimation. The dual form of combined value function for policy update is given by
π = arg min max [(1 - (sW) + xQ2x M OSA51
(5)
where Qx(s(w) and (s are two critics following policy т, x' be the set of transient states in continuous compact state space X. and the weight factor À determines the influence of two Q functions. Specifically, (5) will reduce to a normal policy evaluation with one critic when X = or
Generally, (5) needs to be solved by two-timescale approach that may result in saddle point problem. i.e., on the faster timescale, the policy or its parameter is solved by (5) when fixing X, while on the slower timescale, is minimized without losing the optimal solution to (5). Due to the potential non-convexity of the action-valuu value function, the method may lead sub-optimal solution, and even cause instability when the convergence is slow.
slowly increased until the overestimation error
To avoid the potential saddle point problem, we first propose two-step separation method solve 5) for each policy update, which determines the optimal action to restrain overestimation based on the target weight factor X' and then use the up-to-date policy to update A. Second, although the underestimation bias accompanying the minimization operator is far preferable to overestimation and does not explicitly propagate through the policy update, it indeed negatively affect the policy improvement every iteration and further brings fluctuation on algorithm convergence. The policy improvement means that the optimized objective function in (5) should steadily increase during training. The insurance of this value improvement lies in the choice of weight factor X. The process
3

---

Under review as a conference paper at ICLR 2021
of this separation method is given by
1
j(0)
[11 X)Q(sn; anlw) + (xQm an
N
N
J(A)
1 N
Σ
1)Q anlw) + (2 (8n, an|s[]] Sign(A),
where 0 X 1, 0n - ((,, (0), which means actions in 6 and should be taken from the
policy of current actor instead of the replay buffer. We provide the guarantee of policy improvement by multiplying the averaged return for updating by a sign function, which is given by
A (8)
min ((n,, On ww, Pi, A,) Q( On i aa 2o-1. A
(Ar) =I
where
( alw, S2, = (1- ()00s a|w) + AQ(ss
an,i psss (0:) and On
(1-1) which come from actor networks parameterized by current states before and after updating respectively but not from the replay buffer. I(x:) produces
1 when x is negative, and vice versa. The sign denoted in ( actually the comparison between the minimum updated Q-values in two critics and the old (before the policy update) combined value defined in
Lemma 1. Denoting the converged values of two critics as Q(s, alw") and (s,, als²). respectively, then the convergence of combined value denoted in ( can be ensured by minimizing and
Different from updating in 7 the averaged return for updating 66 adopts the target weight factor X. Then (w", S2', 0', X²) are updated adopting the "soft" target updates Lillicrpp et al. 2015 by (w, S2 0, A), in the way
w/ arg min L(w;) + (1 7)
96 r arg min L(Ω.) (1 -)
Q
8 < r arg max J(0,) + (1 -T)0%
6.
X + arg min J(A,) + (1 - -)
(10)
A
where T < 1 is the factor to control the speed of policy updates for the sake of small value error each iteration, and X' updates following 0'. We organize the above procedures the adaptive delayed deep deterministic policy gradient (AD3) algorithm, whose pscudocode is described by Algorithm !
Theorem 1. AD3 algorithm asymptotically converges as the iteration i - oo with properly chosen learning rate.
Theorem 2. AD3 algorithm has the property of asymptotical expected policy improvement. Specif- ically, when the crities tend to converge, i.e., 3K, vi 2 K, VE > 0.
E(s.a) [Q(s,a/u41,, Q41, Ar) - Q(, alwi, 24 Ai) <e.
(11)
then
(s,a) [Q(s, alws+1, S4+1, ) 2 E(s,)) [Qss, alws, Qe Qe,
(12)
The proof of Theorem 1 and Theorem 2 can be found in the Appendix.
4

---

Under review as a conference paper at ICLR 2021
Algorithm AD3 Algorithm
Input: The batch size N, the maximum of updates M, the timeout step and the soft update parameter т.
2: Initialization: Initialize parameters (2, A,) (wo, So. 0u Ao). ("' (wog. Sn 6. X6) randomly; Initialize replay buffer R, the counter t- O.
3: while M do
4: Reset randomly the initial state 81.
5: = do
6:
Select action de according the current behavior policy, i.., plse(0.) added by exploration
noise:
7:
Execute actions dt. get next states St+1, and immediate reward rt;
8:
Store transition (st, ae Tt, St+1) in Rt
9:
if full then
10:
Randomly and uniformly sample the slot (8;, 02 Ti, 8(4-1) from Rt;
Minimize the Q1 loss function shown in (3) by gradient decent, and then update wit
12:
Minimize the Q2 loss function shown in (4) by gradient decent, and then update S2 Maximize the expected return shown in (6) by gradient ascent, and then update 0,;
14:
Minimize the product of 7 and (8) by gradient ascent, and then update X;; Execute the "soft" target updates shown in (TO) update 8%, w/, Dr, and X;: i i+
16:
17:
end
18:
end for
19:
end while
UNBIASED ADAPTIVE DELAYED DEEP DETERMINISTIC POLICY GRADIENT
To further improve the performance of proposed AD3 algorithm, employ AD3 under the frame- work of unbiased DRL (UDRL) Zhang & Huang (2020). UDRL attempts to solve the systematic bias induced by the approximation of MDP samples. This systematic bias happens in the experience
replay mechanism without importance sampling (IS) Precup et al. (2000; Hachiya et al. 2008);
Mahmood al. (2014); Thomas & Brunskill (2016); Jiang & Li (2016): Wang et (2016); Foer-
ster et al. (2017) Metelli et al. (2018), because there exists mismatch between distributions of the target policy and the behavior policy. Without IS weight the tuples with different probabilities in commonly applied off-policy methods, the experience replay that memorizes the past observations for random samples will accumulate the systematic errors and lower the convergence performance. When applying UDRL, the independently and identically distributed (IID) initial states are paral- lelly sampled to start respective tuples at the beginning of each iteration. Then the parallel virtual agents follow the same behavior policy to complete their tuples, which serve the observations to synchronously train and update the shared network parameters.
In the case of unbiased AD3 (UAD3) method, the parallelly sampled IID observations should be used to train the two critic networks, an actor network and a weight factor. At each iteration, the actions taken following the same behavior receive rewards and next states, that the achieved four-tuple transition slots are independent and follow the same joint probability. By this means, no IS and (8). The pseudocode of UAD3 is organized
required in the approximations of as Algorithm
EXPERIMENTS
CONTINUOUS MAZE
One of the benchmark tasks we choose the continuous maze which is filled with obstacles. The environment of the continuous maze problem includes infinite states and actions which is shown in Fig. ((a) At every step, the agent able to move towards all directions with its step size. Since the state-action space is continuous, the agent may travel through the obstacles represented by the gray grids if no effective guide provided during the whole training. The dark solid line at the edge of Fig. ((a) represented the wall to show the maze is closed except for the goal. The task of
5

---

Under review as a conference paper at ICLR 2021
Algorithm 2 UAD3 Algorithm
Input: The batch size N, the maximum of updates M, and the soft update parameter T.
2: Initialization: Initialize parameters (w, 0, f (wo, Lo, do. Ao), (w', St 0', 0') +
(w/b S7. 6. X6) randomly.
3: for i 1.M...... M do
4: Sample S, = (SL,1, , IID; ,0i,N) ) for S; according to the current actor network
5: Choose actions A; = ( 01,2;
((S))0)) added by exploration noise:
6: Execute actions A,, get next states S (5 " ,(,,n) and immediate rewards R,
(rr,1, ri. ... .rs,N);
7:
Minimize the Q1 loss function shown in by gradient decent, and then update wi; Minimize the Q2 loss function shown in by gradient decent, and then update Ω2; Maximize the expected return shown in 6 by gradient ascent, and then update 0,;
8:
9:
Minimize the product of 7 and (8) by gradient ascent, and then update X;;
11: Execute the "soft" target updates shown in (10) to update 0% w/ 24 and X;;
end for
this experiment move the agent from the start the goal colored yellow with block. This goal can be achieved by setting scores for the agent at every step. Specifically, the agent receives negative if it encounters any jam. If the agent reaches the goal, it will be rewarded 100 score. In other blank areas, the reward is set as the minus distance from the agent to the goal for the purpose of stimulating the agent to fulfill the task as soon as possible.
Gool
Froo
(a)
(b)
Figure 1: (a) The maze environment with continuous state-action space and lines of obstacles; (b) The robot arm environment with move and grasp task.
In this experiment, evaluate the performance of AD3 and UAD3 algorithms, using the baseline algorithms DDPG and TD3. The hyperparameters are shown in Table Every 500 iterations (update periods), an evaluation procedure is launched, which records 100 episodes and averages their results to improve accuracy, i.e., the average reward, where each episode observes the agent from the slart the goal and adds up the rewards without discount along the path. Similar evaluation procedures shared by all experiments in the context with the same cycle.
Figure 2 illustrates the average reward versus update periods of the continuous maze with barriers of different lines plotted in Fig. [(a) Fig. 2(a) shows that AD3 converges faster than DDPG and TD3. From Figs. 2 2(c) we see UAD3 robustly converges so that the agent can reach the goal and receives positive reward, however, other algorithms diverges and fails in their missions. The better performance is mainly due to the policy improvement clarifies in Theorem 2 Besides, adaptive
provides superior way reduce overestimation.

---

Under review as a conference paper at ICLR 2021
-90
-80
(a)
(b)
(C)
Figure 2: Convergence performance of the continuous maze with barriers (a) of 1 line; (b) of 2 lines; (c) of 3 lines.
ROBOT ARM
The "robot arm" experiment is move and grasp task which is shown in Fig, I(b) In this figure, we just show three sections to represent general arm that may contain several more sections. The aim of this task is to move the f'inger represented by the end of the arm to catch the goal, Specifically, the finger should get into the yellow "goal box", and then hold on to the moving goal for required steps to fulfill the grasp task. In the experiment setting, the goal is randomly moving, so the state representation includes both the positions of joints represented by the green circles and their relative positions the goal. Besides, the number of sections determines the dimension of action space which contains the rotation angle of each section. The reward is set as the negative distance from the finger to the goal outside the "goal box". If the finger is located within the "goal box", the reward is set as 1.
We also evaluate our proposed algorithms using the baseline algorithms DDPG and TD3 based on the hyperparameters in Table ! Fig. 3 shows the computational performance of the robot arm with 2 7 sections by fitting scatterplots run for 800 thousand iterations. Notably, one more section will raise the state dimension by 4, including the 2-dimensional coordinates of joints and their relative coordinates the goal. Increased state dimension needs more time for convergence and produces lower converged average reward. Throughout Fig. 3 we observe UAD3 can robustly converge to a value much higher than other algorithms, which shows the best performance because higher converged average reward means the agent can react more promptly to the moving goal. From Figs. ((a) (b) AD3 converges faster and more robustly to a higher value compared with DDPG and TD3 under the same circumstance. In Fig. ((c AD3 converges above the zero line with value higher than DDPG for 6 sections, however, TD3 diverges for both 6 and 7 sections. Overall, UAD3 and AD3 have better performance than their counterparts DDPG and TD3.
10
10
UK2)
(a)
(b)
(C)
Figure 3: Convergence performance of the robot arm with (a) 2-3 sections; (b) 4-5 sections; (c) 6-7 sections.
7

---

Under review as a conference paper at ICLR 2021
CLASSICAL CONTROL ENVIRONMENT
In this part, we adopt a series of classical control experiments including Pendulum, Acrobat, contin- uous mountain car and Caripole the benchmark tasks evaluate the proposed AD3 and UAD3 algorithms based on the hyperparameters in Table
is noteworthy that we revised the rewards of Cartpole environment to make them more challenging. Specifically, the distance and average velocity of the cart are added to the rewards to stimulate the cart to go farther and faster because staying at the origin trading for stability not enough tell the robustness of algorithms. The results of the average reward, average velocity and distance versus update periods for Cartpole are given by Fig- From Fig. (aa we see UAD3 and AD3 can converge much faster and more stably than DDPG and TD3. Besides, UAD3 and AD3 are able to simulate the cart to move faster and farther according to Figs. 44b) and [[c respectively. Figs. ((a) ((c) present the results of average reward versus update periods for Pendulum, Acrohat and continuous mountain UAD3 and AD3 over DDPG and TD3. Moreover, we
car, which further show the advantages reproduce the results of UDDPG Zhang Huang 2020 for a fair comparison with UAD3, From Figs. and 5 we see that UAD3 is even better than UDDPG in higher convergence speed, converged average reward, and stability.
(a)
(b)
(c)
Figure 4: (a) Average reward; (b) Average velocity; (c) Distance versus update periods in Cartpole.
2000
2000
K0
10070
(a)
(c)
Figure 5: Computational efficiency in (a) in Pendulum: (b) Acrobat; (c) continuous mountain
CONCLUSION
In this paper, we proposed the combined value of two independent critics connected by a weight factor to update the policy, which will in turn to update the combined value. We also proposed the objective function for updating the weight factor by multiplying the updated combined value by sign, which compares the minimum updated Q-value in two critics with the combined value before updating. Based on the above two components, bias and ensure future policy improvement at the same time. Furthermore, we apply AD3 to UDRL framework eliminate the systematic bias caused by the probability mismatch between the behavior policy and the target policy in experience replay, and present UAD3. The proposed AD3 algorith- is theoretically proved to possess the property of asymptotical convergence and expected policy
present AD3 to reduce the overestimation
8

---

Under review as a conference paper at ICLR 2021
improvement. Evaluation results show that our proposed algorithms can boost and stabilize the vergence. Although we represent the weight factor as a variable in the context, can be formulated function of states. It can be seen that all theorems and proofs can apply to it when lambda is state-dependent, and all the experimental results are based on the model of state-dependent weight factor ((s). The network architecture of weight factor concerning states is clarified in Appendix.
REFERENCES
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip II Torr, Pushmeet Kohli, and Shimon Whiteson, Stabilising experience replay for deep multi-agent reinforcement learning. pp- 1146-1155, 155, 2017.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. 10:1587-1596, 2018.
Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiyama, and Jan Peters. Adaptive importance sampling with automatic model selection in value function approximation. pp. 1351-1356, 2008.
Hado V Hasselt. Double g-learning. In Advances in neural information processing systems, pp. 2613-2621, 2010.
Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learning, pp. 652-661, 2016.
Timothy P "Lillcrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.0297) 2015.
A Rupam Mahmood, Hado Van Hasselt, and Richard S Sutton. Weighted importance sampling for off-policy learning with linear function approximation. pp. 3014-3022, 2014.
Shie Mannor, Duncan Simester, Peng Sun, and John N. Tsitsiklis. Bias and variance approximation in value function estimates. Monag. Sci., 53(2):308-322 302-22,,, 2007.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli, Policy optimization via impor- tance sampling. pp. 5442-5454, 2018. 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-533. 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Interna- tional conference machine learning, pp.
2016.
Doina Precup. Richard S Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation, 766. 2000.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.00 05952, 2015.
SATINDER SINGH, TOMMI JAAKKOLA, and MICHAEL L. L. LITTMAN, Convergence results for single-step on-policy reinforcement--eariine learnnng algorithms. Machine Learning, 38(3):..87-3088, 2000.
Csaba Szepesvári. Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning. 1003: 2010.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.1 1074, 2018
Philip S Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. pp. 2139-21111 2148, 2016.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Thirtieth AAAI conference on artificial intelligence, 2016.

---

Under review as a conference paper at ICLR 2021
Ziyu Wang. Tom Schaul, Matteo Hessel. Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv: 2015.
Ziyu Wang, Victor Bapst. Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando De Freitas. Sample efficient actor-critic with experience replay. arXiv: Learning, 2016.
Huihui Zhang and Wu Huang. Unbiased deep reinforcement learning: A general training framework for existing and future algorithms. CoRR, abs/2005.07782, 2020. URL https: http://aaxvvvv.. http://wwwwxvvvraa/a///aaa/a//// 2005 07788 07782
APPENDIX
PROOF OF LEMMA
Proef
((s.a) (Q(s, S2, A) al=," 22, A1)
[(11 - ()( alw) Q(s,al(())))))) + a[()) Q(s.
(..)
{[(1 x)(((s,oww) a(s.aw (A((s - Q(s, Q(s.a
+0
(13)
where the converged combined value als,a ,0", = (1 x)Q(s,alw") + a(())
PROOF THEOREM
Proof
This proof hased on Lemma of SINGH ct al. (2000). which moved originally helow for conve-
nience.
Lemma of SINGH et al. 2000 Consider stochastic process (ar, A., Fi), L 2 O, where au Do, Fi X -
R. which satislies the equations
A (2) (1 an)))) ar(s))((()) =,,,1=0,1,,
(14)
Let Pt be sequence of increasing a-fields such that cto, Ao are Po-measurable and a. D and Fi- P--measurable, 2. Assume that following hold:
1. the set of possible states X finite.
2. 0 ae(x)) L, (x(x) oc, E, (x) < w.p.l.
3. [[((F)()) S A/| + c,, where y E [0, 1) and ca converges to zero w.p.1.
4. Var [F) & ( ||), where K is some constant.
Then A converges
with probability one (w.p.1).
Within the scope of this paper, the MDP state space is finite, satisfying condition in Lemma I SINGH et al. (200),, and Lemma condition holds hy proper selection of learning rate. According [Sxepesvári (2010). even the commonly used constant learning rate can make algorithms converge in distribution.
We apply Lemma of SINGH et al. 2000 with Pt =
((Q(,11w)) Qt, -(2)), Sa,
0z
Following the update rule for optimizing and and using the current policy produce the action a:.1 puse+1 (0), have
4
Q(s,, (u(w))))))))))))))))))))())))( = (1 0z ((s,))((ss + a [re + 7Q 0441 Q(a,, as (011)) = (1 - au (((,, a. au Ire (St+ ac+1 [[+)].
(15)
(16)
Under the setting of our proposed algorithm, we denote A, Q(, Q, A) (+.)).)... which is the dif- ference between the combined value weighted critics denoted and optimal value function. Then we have
Aet1 ae) = Q( ale,,, (s,, at), at).
=(1 (0 + Xe(0s. Q (s))
=(1 an) [2 e we, Su, Qs., - arFi
=(1 as Ar) acF
(17)
10

---

Under review as a conference paper at ICLR 2021
where the third equality is due the substitution of 15 and 16 and
(i ac) = ra + (0 Qr 2) @*(,,a
(18)
Since the reward is bounded within the scope of this paper, the action-valuu are also bounded, then condition
Lemma ofSIG et al. (2000) holds. According to the proof in Theorem 2 of et al (2000). there
is || [Fr (#e, Palll S Aell. which satisfies condition 3 in Lemma of SINGH et al. (20X))
Finally, it can be concluded that Q(s lwr, Rz A) converges to (*(+.) with probability
PROOF OF THEOREM
Proof signi+1 2 O, then
(()))
E,
((s) aen, Aeta)]
Σ
(s.. ((5 S A+t1)
N
1=
N
min
Σ M(o
Σ ((s.(:4(4)) )(
N
((s,
Su De.
N
E ( al,,, S2 (2,
(19)
where the approximations the mean average is statistically equal to the expectation (or unbiased approximation for UAD3), the first inequality holds based (9). and the second inequality holds because (8) no less than
Otherwise, if sign:+1 < 0, then
T [..a)) [(s als (+t1)))t))
E. ((s0, R441, Xitti]
((sn, Qi+1, Ast1)
M(s.
N
Σ (1s.
N
N
(m. ((s, Aut1,
~ E. [Q( [(s,, alwi, Q, 2.)
X.]
E(s.a)
(20)
where the first inequality holds due to the fact that the update is to maximize 7 given a negative sign, the second inequality holds because the update of is to maximize (6). Although (19 and 20 are done for immediate target updates, the same conclusions can be achieved under the condition of delayed "soft" target
updates if the Q network is linear with respect the actor and critic parameters.

---

Under review as a conference paper at ICLR 2021
NETWORK ARCHITECTURE
We construct the critic network using fully-connectes MLP with two hidden layers. The input is composed of the state and action, outputting value representing the Q-value. The ReL.U function adopted to activate the first hidden layer. The setting of actor network is similar to that of the critic network, except that the input is the state and the output multiplied by the action supremum after tanh nonlinearity. The network of weight factor is constructed similar the actor network except replacing the tanh nonlinearity by clipping [0, 1]. The architecture of networks are plotted in Fig.
conneted KdL
Fulle conseced
State Toger
((s.a)
Artien Taper
Pully connected lager
Fulle consecodd
loper
State lagir
m(x
Bully cameent layer
Fuller conseced
Treat Tape
State lave
2(+)
Figure 6: Architecture of networks.
HYPERPARAMETERS
Table lists the common hyperparameters shared by experiments and their respective settings.
12

---

Under review as a conference paper at ICLR 2021
Table 1: List of hyperparameters Description
Shared Env
Value
Algorithm applied
LRaa
0.001
Learning rate of actor
DDPG, TD3, AD3, UAD3, UDDPG
LR c
0.001
Learning rate of critic Learning rate of criticl Learning rate of critic2 Learning rate of weight factor Soft update parameter of actor Soft update parameter of criticl Soft update parameter of critic2
DDPG, UDDPG
LR e
0.001
TD3, AD3, UAD3
LR_02
0.001
TD3, AD3, UAD3
LR_X
0.001
TD3, AD3, UAD3
T
0.01
DDPG, UDDPG
T.4I
0.01
TD3, AD3, UAD3
T_02
0.01
TD3, AD3, UAD3
7
0.9
Discount horizon factor
DDPG, TD3, AD3, UAD3, UDDPG
Interval
500
Eval period
DDPG, TD3, AD3, UAD3, UDDPG
Test
100
Episodes per eval period
DDPG, TD3, AD3, UAD3, UDDPG
Var.dr
0.9995
Exploration variance decay rate
DDPG, TD3, AD3, UAD3, UDDPG
Sample
200
Sample size of initial states
UAD3, UDDPG DDPG, TD3, AD3
Batch
200
Size of mini-batches Description
Cartpole
Value
Algorithm applied
Max Max_EPS
500
Maximal steps per episode training
DDPG, TD3. AD3
Runout
1000
Maximal steps per episode eval
DDPG, TD3, AD3, UAD3, UDDPG
Initial variance
10.0
Initial exploration variance
DDPG, TD3, AD3, UAD3, UDDPG
Memory
50000
Size of replay buffer
DDPG, TD3, AD3
Train num
100000
Updating iterations (Not episodes)
DDPG, TD3, AD3, UAD3, UDDPG
Acrobot
Value
Description
Algorithm applied
MountainCar
Max_EP
500
Maximal steps per episode training
DDPG, TD3, AD3
Runout
500
Maximal steps per episode eval
DDPG, TD3, AD3, UAD3, UDDPG DDPG, TD3. AD3, UAD3, UDDPG
Initial variance
10.0
Initial exploration variance
Memory
10000
Size of replay buffer
DDPG, TD3, AD3
Train num
800000
Updating iterations (Not episodes)
DDPG, TD3, AD3, UAD3, UDDPG
Robot Arm
Value
Description
Algorithm applied
Max_EPS
200
Maximal steps per episode training
DDPG, TD3, AD3
Runout
100
Maximal steps per episode eval
DDPG, TD3, AD3, UAD3, UDDPG
Initial variance
1.0
Initial exploration variance
DDPG, TD3, AD3, UAD3, UDDPG
Memory
30000
Size of replay buffer
DDPG, TD3, AD3
Train num
800000
Updating iterations (Not episodes)
DDPG, TD3, AD3, UAD3, UDDPG
Maze
Value
Description
Algorithm applied
Max_EPS
500
Maximal steps per episode training
DDPG, TD3, AD3
Runout
100
Maximal steps per episode eval
DDPG, TD3, AD3, UAD3, UDDPG
Initial variance
10.0
Initial exploration variance
DDPG, TD3, AD3, UAD3, UDDPG
Memory
10000
Size of replay buffer
DDPG, TD3, AD3
Train num
800000
Updating iterations (Not episodes)
DDPG, TD3, AD3, UAD3, UDDPG
Pendulum
Value
Description
Algorithm applied
Max_EPS
200
Maximal steps per episode training
DDPG, TD3, AD3
Runout
100
Maximal steps per episode eval
DDPG, TD3, AD3, UAD3, UDDPG DDPG, TD3, AD3, UAD3, UDDPG
Initial variance
10.0
Initial exploration variance
Memory
30000
Size of replay buffer
DDPG, TD3, AD3
Train num
800000
Updating iterations (Not episodes)
DDPG, TD3, AD3, UAD3, UDDPG
13

---

Under review as a conference paper at ICLR 2021
ESTIMATE OF Q-VALUE
We uniformly sample 10000 states from replay buffer every 500 update periods, and average computed Q-values based on the newly-updated current policy. The averaged Q-value can seen estimate of discounted cumulative return (value estimate). The value estimates of Cartpole, Robotarm of 2 sections and From these figures, we can see the converged value function
Pendulum experiments are plotted in Fig- (ssaalu") denoted in Lemma
Uptis B4002
(a)
(h)
(C)
Figure 7: Value estimate in (a) Cartpole; (b) Robotarm of 2 sections; (c) Pendulum.
14


Reviews:
[{"review_id": "j75GHmN48MX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes new value-based deep reinforcement learning algorithms (AD3 and UAD3) to address the overestimation bias issue of Q learning. The main contributions of this paper are three folds: 1) The authors propose a weighted sum of two state-action value functions that are trained separately. Then, it is used to update the policy. 2) The mixing weights are updated by the two-step separation method. 3) The original method (AD3) is integrated with the idea of unbiased DRL (Zhang and Huang, 2020). \nI have the following concerns for this paper:\n1) Although the algorithms are described clearly, I do not fully understand why the proposed method mitigates the overestimation bias issues. The authors claim that applying the clip variant of DDQN to the update procedure of the actor is more reasonable, but the experimental results did not support the authors’ claim. Is it possible to show value estimates by TD3 and the proposed methods?\n2) I agree that the two timescale approach may suffer from a saddle point problem when used to solve Equation (5). However, I am not sure whether the two-step separation method can avoid the saddle point problem. Does AD3 outperform the two-timescale approach?\n3) The framework of unbiased DRL (UDRL) is interesting, but it is not the main contribution of the paper. So, a comparison of UAD3 with DDPG and TD3 is not fair. Is it possible to apply UDRL to DDPG and TD3? Since Zhang and Huang (2020) proposed Unbiased DDPG, I think the comparison of UAD3 with UDDPG helps understand how AD3 contributes to learning efficiency. \n4) The authors compare the proposed methods with DDPG and TD3. However, DDPG does not consider the overestimation issue. It would be better to compare the proposed method with Averaged-DQN (Anschel et al., ICML 2017) and MaxMin Q-learning (Lan et al., ICLR 2020).\n5) Is Equation (5) correct? I think - \\lambda Q_\\pi^2 should be + \\lambda Q_\\pi^2. In addition, Q_\\pi^1 and Q_\\pi^2 are not defined explicitly. \nOverall, I think the idea of the paper is novel, but further improvements should be added to increase the score of the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "btT3XtCro3O", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a deterministic policy-gradient algorithm that extends the TD3 algorithm (Fujimoto 2018). The main claim is that it reduces overestimation issues in a more effective way. Two Q-critics are maintained with separate parameters, but updated using the same transitions. Then a convex combination of these critics is used in the deterministic policy gradient update. The mixture parameter is learned on a slower time-scale to minimize this convex combination over states (instead of taking the minimum of the 2 critics per batch as in TD3). Another contribution in the paper is the “Unbiased” variant of the algorithm (UAD3), which addresses the off-policy nature of the replay mechanism of the AD3 algorithm described above. My understanding is that this is simply a version of the algorithm that does not use any replay mechanism and samples the state iid from the on-policy distribution, so it isn’t a novel idea in itself.\n\nThere are two theorems given to justify the algorithm choices, but I want to question their validity. The first one says that AD3 converges asymptotically, but no formal statement of what this means is given and the proof for it in the appendix only states broad facts about stochastic approximation, but nothing specific that applies to the AD3 algorithm. Theorem 2 is misleading in another way, it says that AD3 has “the property of asymptotical expected policy improvement”, but it only really says that the critic value will be increasing, not that the actual policy value is increasing (and so an actual policy improvement step). Moreover, the proof contains some approximation steps which are not justified.\n\nThe approach is tested in two simple continuous control environments (maze + reacher task). (Are these using the full state as input?). There the proposed approaches perform better it seems than the baselines (TD3 and DDPG), but there isn’t any analysis to understand whether that was due to better critics - why not plot the estimated and true returns during learning to see whether AD3 indeed does better than the other critic update strategies? The experimental section is missing details to make these results reproducible and interpretable, for example what network architecture was used for the policy and critic?  All learning curves have rather strange oscillation patterns. Is that an artifact of the smoothing used? How many seeds were used to obtain each learning curve?\n\nAt the moment, the advantages of the proposed approach are neither demonstrated theoretically or empirically in a satisfactory way (see comments above). At least one of these aspects need to be improved significantly before the paper is ready for acceptance.\n\nOther comments:\n\nThe objective to decide how to mix the critics could be better motivated. Making sure that the critic increase may not be the best criterion to ensure stability and reduce over/underestimation for example. \n\nMinor things:\n\n* “In actor-critic methods, the policy is deterministic…” Actor-critic methods are actually more commonly described in a stochastic policy setting I believe. This work seems to be based on the deterministic policy gradient paper which is not cited.\n* Citation style should use parentheses around names in most cases in the text.\n* Gattami 2019 is not the right reference for the Bellman equations and derived RL updates.\n* Lots of equations are redundant and add unnecessary complexity. For example Eq 3 and Eq 4 are the same equations with different parameters, so why not call the two parameters w_1 and w_2 and write the equation once with w_i  for i \\in {1,2}\n* The hat on the \\lambda in Eq 6,7 are hard to see because they are attached to the left bracket.\n* adopts -> adopt\n* Transition slots -> tuples?\n* Minus distance/reward -> negative\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "M7tq491Xa6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe paper proposes a method to avoid the overestimation bias. They modify the TD3 algorithm such that, instead of regressing the two Q-values towards the minimum of both Q-value functions, the regression target is a convex combination of both, determined by a parameter lambda which is updated on a slower time-scale.\n\nI have three concerns due to which I am currently recommending rejection: First, the smaller concern is that the paper is currently difficult to understand, second, my major concern is that I am not sure about it's correctness, due to equation (9). Lastly, the experimental evaluation is insufficient. \n\nRegarding understanding: The paper currently has a significant number of grammatical mistakes and unusual or wrong word choices. While this often is without consequence, unfortunately, at times it does impact the clarity and creates uncertainty about the argument the authors are trying to make. However, overall I believe I was able to understand the algorithm the authors are proposing, so this is not my major concern.\n\nRegarding correctness of the algorithm: My main issue is with equation (9), i.e. with how lambda (the trade-off parameter in the convex combination between the two q-value functions) is updated. I believe equation (9) should almost always be negative because the minimum of two values is always smaller or equal than any convex combination of both, since lambda is restricted to 0 ≤ lambda ≤ 1. Which means that based on equation (11), lambda is actually optimized to *maximize* the value, i.e. it encourages overestimation instead of preventing it. \n\nOne caveat might be that equation (9) actually compares Q-values from before and after each update. It is not explained why this is done, but I believe that should  change the sign of (9) in only very rare cases as I'd expect Q to move only very little in each update.\n\nLastly, lambda is a global variable, whereas the minimum operation of TD3 is local, i.e. performed independently in each state. I believe this is a big difference in the proposed algorithms which is not discussed at all.  \n\nRegarding experimental evaluation: As the authors propose a novel, general purpose algorithm which aims to improve on TD3, it is important to evaluate it on a wide range of benchmarks, including the typical, widely used mujoco environments on which most current continuous control algorithms have been evaluated and compared. \n\nOther, minor, points:\n\n- In equation (5): What is the min over? What is the max over? Also, the Q functions have the wrong argument signature\n- Below equation (5): I don't understand what xi and xi' are\n- Very minor point for Equation (6) & (7): Personally I don't like the introduction of the \\bar{} version of lambda and would find (1-lambda) clearer in the equation", "extra_scores": {}, "extra_text": {}}, {"review_id": "BAjbhfwUVP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to mitigate the overestimation issue, which is quite common in RL algorithms whenever computing boostrap target is needed. The key idea is to introduce a weight parameter between two Q values and adopt a dual problem formulation to learn this weight and the policy parameters. The authors propose a two-step method to estimate these parameters and present some experiments to show the proposed algorithm's performance.\n\nOverall Quality/Clarity is poor. The reasons are as follows. \n\nFirst, the theorems are incorrect or meaningless. Theorem 1 says the proposed algorithm AD3 (i.e. adaptive delayed deep deterministic policy gradient) converges. But the proof is for the tabular case and it becomes trivial. Furthermore, the proof itself seems to be incorrect. There should be restrictions on the function \\mu as it would affect what type of operator it is. Theorem 2 is simply incorrect. The inequality (13) does not imply policy improvement at all. I guess the authors want to say the policy under the policy parameter of the i+1th step has a higher expected return. But the notations are not saying that. The proof itself uses approximation signs arbitrarily, and it is unclear what the meaning is. \n\nSecond, experiments lack details and it is unclear what message they are trying to convey (e.x. Fig 2). How were those hyperparameters chosen? How many runs? What is the standard deviation across runs?  \n\nThird, many of the statements and notations are not rigorous. \n\nThe authors never clearly state the objective function they are using. Is it the same as DDPG? And, it never makes a clear distinction between the objective used for updating the critic parameter and the actor parameter. \n\nThe definition of J(\\theta) is confusing, what is the underlying distribution of s (first paragraph on page 3)? In eq (6) and (7), the definition of J(\\theta) changes. \n\n“In actor-critic methods, the policy is deterministic and commonly parameterized as the actor network.” This is inaccurate. The policy does not have to be deterministic in actor-critic methods. \n\n“However, updating two Q networks according to the same target estimate will make them less independent, which will further negatively affect the training efficiency.” Why? \n\n“Although the underestimation bias accompanying the minimization operator is far preferable ..., it indeed negatively affect the policy improvement at every iteration and further brings fluctuation on algorithm convergence.” Why does it \"indeed affect the policy improvement\"?\n\nImportant detail missing in the background. The authors should cite the DDPG or the DPG papers at least when describing the updating rule (2). In the original work, there is also a slower moving network for the actor used for computing boostrap target to update the critic. For a paper concerning removing overestimation bias, it is a very important detail and should be clearly stated.  \n\nLast, those important derivations (i.e. those from eq (8))  do not have justification. \n\nOriginality/Significance. I did not see exactly the same work before. But there are similar works and many papers in the same research line are not appropriately discussed. Please see the citation part below. \n\nLack of citation & unrigorous citation. \n\n\"The foundation for updates of network parameters is the Bellman equation Gattami (2019).\" The cited paper is about multi-objective and constraint MDP; the credit of the Bellman equation should not go to this work. There is no need to cite anything here. Rigorously, Q-learning is based on the Bellman optimality equation. \n\nMaxmin Q-learning: Controlling the Estimation Bias of Q-learning by Lan et al. This work introduces a method that reduces bias to almost zero while also reducing variance. They also provide convergence proof under the tabular case. Since their approach does not have the issues as the author mentioned for DDQN, the authors should include such a citation and further discuss why their proposed approach is still interesting. The authors should also search for average Q-learning, ensemble Q-learning, etc. \n\nThe earliest work discussing the source overestimation should be Issues in Using Function Approximation for Reinforcement Learning by Sebastian Thrun and Anton Schwartz. 1993. \n\nThe following nice papers also discuss the harm of overestimation bias. \nIstva ́n Szita and Andra ́s Lo ̋rincz. The Many Faces of Optimism: A Unifying Approach. In International Conference on Machine learning, 2008. \n\nAlexander L. Strehl, Lihong Li, and Michael L. Littman. Reinforcement Learning in Finite MDPs: PAC Analysis. Journal of Machine Learning Research, 2009. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were in agreement that the paper is below the bar for acceptance, and the authors did not provide a response to reviewer concerns.
Decision: Reject