GRAC: Self-Guided and Self-Regularized Actor-Critic
Authors: Lin Shao
Yifan You
Mengyuan Yan
Shenli Yuan
Qingyun Sun
Jeannette Bohg
Venue: CoRL2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Q-learning

Abstract:  Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks. One dominant component of recent deep reinforcement learning algorithms is the target network which mitigates the divergence when learning the Q function. However, target networks can slow down the learning process due to delayed function updates. Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network. Additionally, we propose a self-guided policy improvement method by combining policy-gradient with zero-order optimization to search for actions associated with higher Q-values in a broad neighborhood. This makes learning more robust to local noise in the Q function approximation and guides the updates of our actor network. Taken together, these components define GRAC, a novel self-guided and self-regularized actor-critic algorithm. We evaluate GRAC on the OpenAI gym tasks, outperforming state of the art on four tasks and achieving competitive results on two environments. We also apply  GRAC to enable a non-anthropomorphic robotic hand to successfully accomplish an in-hand manipulation task in the real world.

GRAC: Self-Guided and Self-Regularized Actor-Critic
Lin Shao', Yifan You?, Mengyuan Yan', Shenli Yuan', Qingyun Sun Jeannette Bohg'
Stanford Lab, 'Department of Mathematics, Stanford University
{lins2, mengyuan, shenliy, qysun, bohg) @rr.anford...mmmmo *Department of Computer Science, University of California, Los Angeles harry473417 edu
Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on range of challenging decision making and control tasks. One dominant component of recent deep reinforcement learning algorithms is the target network which mitigates the divergence when learning the Q function. However, target networks can slow down the learning process due to delayed function updates. Our main contribution in this work is relf-regularized TD-learning method to address divergence without requiring a target network. Additionally, we propose a self-guided policy improvement method by combining policy-gradine with zero- order optimization to search for actions associated with higher Q-values in a broad neighborhood. This makes learning more robust local noise in the Q function approximation and guides the updates of our actor network. Taken together, these components define GRAC, novel self- self-guided and seff-regularized actor critic algorithm. We evaluate GRAC on the OpenAl gym tasks, outperforming popular methods such as TD3 [1] and SAC [2] on four tasks and achieving competitive re- sults on two environments. We also apply GRAC to enable a non-anttropomorphic robotic hand successfully accomplish in-hand manipulation task in the real world.
Abstract:
Keywords: Deep Reinforcement Learning, Q-learning
1 Introduction
Reinforcement learning (RL) studies decision-making with the goal of maximizing total discounted reward when interacting with environment. Leveraging high-capacity function approximators such as neural networks, Deep reinforcement learning (DRL) algorithms have been successfully applied to range of challenging domains, from video games [3] robotic control [4].
Actor-critic algorithms are among the most popular approaches in DRL, e.g. DDPG [5], TRPO [4], TD3 [1] and SAC [2]. These methods are based policy iteration, which alternates between policy evaluation and policy improvement [6]. Actor-critic methods jointly optimize the value function (critic) and the policy (actor) it is often impractical to either of these to convergence [2].
In DRL, both the actor and critic use deep neural networks as the function approximator. However, DRL is known to assign unrealistically high values to state-action pairs represented by the Q-function.
This is detrimental to the quality of the greedy control policy derived from Q [7]. Mnih al. [8] proposed to use a target network to mitigate divergence. A target network is copy of the current Q
function that is held fixed to serve as a stable target within the TD error update. The parameters the
target network are either infrequently copied [8] or obtained by Polyak averaging [5]. A limitation of
using a target network that it can slow down learning due to delayed function updates. We propose
TToyota Research Institute ("TRI") provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity. This article is also supported by Siemens and the HAI-AWS Cloud Credits.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

an approach that reduces the need for a target network in DRL while still ensuring stable learning and good performance in high- sigh-dimensional domains. We add a self-regularization term to encourage small changes to the target value while minimizing the Temporal Difference (TD)-error [6].
Evolution Strategies (ES) are a family of black-box optimization algorithms which are typically very stable, but scale poorly in high-dimensions gradient-based DRL methods, unlike evolutionary search methods, can continue sample previous experiences to improve value estimation, particularly in the off-policy setting. At the same time, these approaches can also be unstable and highly sensitive to papem-par tuning [9]. We propose a novel policy improvement method which combines both approaches to get the best of both worlds. Specifically, after the actor network first outputs an initial action, we apply the Cross Entropy Method (CEM) [10] to search the neighborhood of the initial action to find a second action associated with higher Q value. Then we leverage the second action in the policy improvement stage to speed up the learning process.
search spaces e.g. neural networks [9]. Policy
Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network that may slow down learning progress. In addition, we propose a self-guided policy improvement method which combines policy-gradients and zero-order optimiza- tion. This helps to speed up learning and is robust to local noise in the Q function approximation. Taken together, these components define GRAC, a novel self- self-guided and self-regularized actor critic algorithm. We evaluate GRAC on six continuous control domains from OpenAl gym [11], where GRAC outperforms popular methods such as TD3 [1] and SAC [2] on four tasks and achieves com- petitive results on two environments. We run our experiments across large number of seeds with fair evaluation metrics [12), perform extensive ablation studies, and open source both our code and learning curves. We also run GRAC to enable a non-anthropomorphic, real robotic hand to successfully rotate a cube to the target pose.
2 Related Work
The proposed algorithm incorporates key ingredients within the actor-critic method: self-regularized TD update and self-guided policy improvements based on evolution strategies. In this section, we review prior work related to these ideas.
Divergence in Deep Q-Learning In Deep Q-Learning, we use nonlinear function approximator such as a neural network to approximate the Q-function that represents the value of each state-action pair. Learning the Q-function in this way is known to suffer from divergence issues [13] such as assigning unrealistically high values to state-action pairs [7]. For the case when the control policy is greedily derived from Q [7]. unrealistically high Q values are detrimental. To mitigate the divergence issue, Mnih et al. [8] introduce a target network which is a copy of the estimated Q-function and is held fixed to serve
stable target for some number of steps. However, because of the delayed function updates, target networks can slow down learning [14]. Durugkar and Stone [15] propose Constrained Q-Learning, which uses constraint to prevent the average target value from changing after an update. Achiam et al. [16] give simple analysis based on a linear approximation of the Q function and develop stable Deep Q-Learning algorithm for continuous control without target networks. However, their proposed method requires separately calculating backward passes for each state-action pair in the batch, and solving a system of equations each timestep. Bhatt et al. [17] introduce a normalization called cross-oormaiiztioon which is regarded as an extension of batch normalization that re-centers data for on- and off-policy transitions. Peng et al. [18] uses Monte Carlo instead of TD error to update the Q network and removes the need for a target network. Our proposed GRAC algorithm adds a rellaaegulrrizatioo term to the TD-Learnin objective keep the change of the state-action value small.
Evolution Strategies in Deep Reinforcement Learning Evolution Strategies are typically stable but suffer from scaling to dimennsiniii search spaces. Policy gradient- based deep RL methods, such as DDPG [5] can continue to reuse previous experience to improve value estimations, particularly in the off-policy setting, but can be unstable and highly sensitive to hyper-parameter tuning [9]. Researchers have proposed to combine these approaches to get the best of both worlds. Pourchot and Sigaud [9] proposed CEM-RI. to combine CEM with either DDPG [5] or TD3 [1]. However, CEM-RL RL applies CEM within the actor parameter space which is extremely high-dimensional, making

---

the search not efficient. Kalashnikov et al. [19] introduce QT-Opt, which leverages CEM to search the landscape of the Q function, and enables Q-Learning continuous action spaces without using an actor. Based on QT-Opt, Simmons- Feller et al. [20] leverage CEM to search the landscape the Q-function but propose initialize CEM with Gaussian prior covering the action space, independent of observations. However, CEM does not scale well to high-dimensional action spaces [21]. such as in the Humanoid task. We first let the actor network output an initial Gaussian action distribution conditioned on current state. Then use CEM to search for action with higher Q value than the Q value of the Gaussian mean. Starting from the predicted distribution, we show that GRAC speeds up the learning process compared to popular actor-critic methods.
3 Preliminaries
In this section, we define the notation used in subsequent sections. Consider a Markov Decision Process (MDP), defined by the tuple (S,A, P,,, rr, (o, 7), where S is finite set of states, A is finite set of actions, S S A x S + R is the transition probability distribution, r x A - R is the reward function, Po : S + I is the distribution of the initial state sp, and y E [0, 1] is the discount factor. At each discrete time step t, with given state E S, the agent selects an action a₂ E A. receiving a reward r and the new state 8141 of the environment.
Let denote the policy which maps state to a probability distribution over the actions. T: : + P(A). The return from a state is defined as the sum of discounted reward R₃ = Ei=e? In reinforcement learning, the objective is to find the optimal policy x", with parameters s, which max- imizes the expected return (() = Σ, (=)...))))p (7"((,, at)] where Pa (8t) and Pr (8t,)))) denote the state and state-action marginals of the trajectory distribution induced by the policy (
a,).
We use the following standard definitions of the state-action value function Qr. It describes the expected discounted reward after taking an action 0e in state 8 and thereafter following policy
Qx ax) = E, [R (2, ar).
In this work we use CEM to find optimal actions with maximum Q values. CEM is randomized zero- order optimization algorithm. To find the action a that maximizes Q(s, CEM is initialized with paramaterized distribution over a, P(a; v). Then it iterates between the following two steps [22]: First generate d1...., aN ~ P(8; v). Retrieve their Q values Q(*, a;) and sort the actions to have decreasing Q values. Then keep the first actions, and solve for updated parameters v/::
K Σ
3 argmax & K
(a((P(a;;))) v))
In the following, let (QM, +), T (-1ss) denote the action found by CEM to maximize Q(+,) +), when CEM is initialized with the distribution predicted by the policy.
4 Technical Approach
4.1 Self-Regularized TD Learning
Reinforcement learning is prone to instability and divergence when a nonlinear function approximator such as neural network is used to represent the Q function [13]. Mnih et al. [8] identified several reasons for this. One is the correlation between the current cetion-values and the target value. Updates to Q(+;, a,) often also increase Q (s af+1) where a;+1 is the optimal next action. Hence, these updates also increase the target value ye which may lead to oscillations or the divergence of the policy.
More formally, given transitions (st, dt, rt. 8141) sampled from the replay buffer distribution B, the Q network can be trained by minimising the loss functions L(8:) at iteration d:
C(0.) E (l....)))88(((( az; 0.) 11)
(1)
where for now let us assume y; = rtz + max, Q( ,; o; 0.) to be the target for iteration computed based on the current Q network parameters 0. 4;+1 arg max, Q(s++1,)) a). ). If we update the
parameter 0,+1 to reduce the loss L(O.), it changes both Q(s,, ar; (:41) and Q( ((4411,0441500 00+1)
Assuming increase in both values, then the new target value yi+1 r. +7Q (5(411 a:41; 0(+1) for
3

---

Algorithm GRAC
Initialize critic network QB1. Qe2 and actor network Te with random parameters 01, 02 and Φ Initialize replay buffer B, Set a <
1: for == do
Tá. 2: Select action a TT,(s), (s), observe reward and new state
3: Store transition tuple (s, a, r,,)ss) in B
4: Sample mini-batch of N transitions (st, az Tt, St-1) from B.
5:
a++1
To, (8(+1)
6:
dr+1
( 02), To. (+1s+11))
7:
y - ro max min Q(s++1 0(+1:8)); min 0(+:=1.0=+1::)) 1=1,2
j=1,2
al - arg max min Q( 0j),
min Q(+ (++;;,1)
8:
(a.0) 1=1,2
j_1,2
9:
3/ (( at:0 ((:4+1,
10:
for to do
11:
Lx ly (8,, au 01 01 +1 ly - (s+, az 02) +2)1 Q(8++1, (110000)1+11
Q(s++1. a1;
81 A1 Aon 02 02 NV22Lt
13:
if L* aLı then
Break
15:
end if end for
17: āt To. (8.)
(1) = R ac) (x;
a CEM(Q (st, 01), (.,
20: Ø AV ( - - (o, (Q
( 01 - Q(s,, âz; 01)1+ V log o (āc(s;)))
end for
the next iteration will also increase leading explosion of the Q function. We demonstrated this behavior in an ablation experiment with results in Fig. 2. We also show how maintaining a separate target network [8] with frozen parameters f- compute Vi+1 Tz Q(s++1, ai+1; 00) delays the update of the target and therefore leads to more stable learning of the Q function. However, delaying the function updates also comes with the price of slowing down the learning process.
We propose a self- Regularedd TD-learning approach to minimize the TD-error while also keeping the change of Q(5++1 1, a4+1) small. This regularization mitigates the divergence issue [13], and no longer requires a target network that would otherwise slow down the learning process. Let o Q(s(+1((())))) ": 0.), and y: rı 78/. We define the learning objective
min minl|Q(ss ax: 0) - +=112+1|Q(s+++ ((++))),1,) vl
where the first term the original TD-Learning objective and the second term is the regularization term penalizing large updates to (( ([41). Note that when the current Q network updates its parameters 8, both Q(s,, at) and (s ai+1) change. Hence, the target value y; will also change which is different from the approach of keeping frozen target network for few iterations. We will demonstrate in our experiments that this self-regularized TD-Learnin approach removes the delays in the update of the target value thereby achieves faster and stable learning.
4.2 Self-Guided Policy Improvement with Evolution Strategies
The policy, known as the actor, can be updated through combination of two parts. The first part, which we call Q-loss policy update, improves the policy through local gradients of the current Q function, while the second part, which we call CEM policy update, finds a high-value action via CEM in a broader neighborhood of the Q function landscape and updates the action distribution to be around this high-value action. We formally describe the two parts below.
Given states s, sampled from the replay buffer, the Q-loss policy update maximizes the objective
Ja (Ø) E, (se, āz)],
(3)

---

where å, is sampled from the current policy x(-1s)) The gradient is taken through the reparameteri- zation trick. We reparameterize the policy using neural network transformation described by Haarnoja et al. [2].
à = fo(E)))))
where E2 is input noise vector, sampled from fixed distribution, such standard multivariate Normal distribution. Then the gradient of (Ø) is:
dQ St. fola(s)) @f((c))s))
(Ø) = BAI
(5)
Of
dΦ
For the CEM policy update, given a minibatch of states we first find a high-value action ā, for each state by running CEM on the current Q function, a, = QEM(Q is updated to increase the probability of this high-value action. The guided update on the parameter of π at iteration i
(-+ss))) +( Then the policy
Eg,~B,rr (Q(s), a.) (ss. 0c)]+ V log n. (ā,
66)
We used (st, âz) as a baseline term, since its expectation over actions å will give us the normal baseline V (s,):
E E.,~B(Q(ss, a²) - v(si)]+ V log (ā,ls))
In our implementation, we only perform update if the improvement on the Q function, Q(s,, a,) (sa, az). is non-negative, to guard against the occasional cases where CEM fails to find a better action.
Combining both parts, the final update rule on the parameter di of policy m. is
@ Φb AVoJa (0:) A B (0(s āc)
Q(( ax)l+ V (arl())
where À is the step size.
Let Q" be the state-action value function induced by the current policy. We can prove that the Q function has converged to Q" then both the Q-loss policy update and the CEM policy update will be guaranteed to improve the current policy. We formalize this result in Theorem and Theorem 2, and prove them in Appendix.
Theorem 1. Q-loss Policy Improvement Starling from the current policy maximize the objec- tive J+ (o) Q" (8, a). The muximization converges a critical point denoted as Tnem- Then the induced function, QThew, satisfies v(,,a), new (s,a) Z Q (s)
Theorem 2. CEM Policy Improvement Assuming the CEM process is able find the optimal
action of the state-action value function, a" (*) arg max, a Q" (s, a), where Q" is the Q function induced by the current policy n. By iteratively applying the update ((s.a))oo(ssd) (s,, a") Q(s,a)] V log (a=)<(aa))))) [s), the policy converges to Tnem. Then Qunew satisfies V(s,a) (s, Q TTaen (8,a) Z (s)
4.3 Max-min Double Q-Learning
Q-learning [23] is known to suffer from overestimation [24]. Hasselt [25] proposed Double-Q learning which uses two Q functions with independent sets of weights mitigate the overestimation problem. Fujimoto et al. [1] proposed Clipped Double Q-learning with two Q functions denoted as Q(s, a; 01) and (s, a; 02), or QI and Q2 short. Given a transition (st, de; Te, 8141), Clipped Double Q-learning uses the minimum between the two estimates of the Q functions when calculating target value in TD-error [6]:
リ= min Q(+++1;0441;0))
where ă is the predicted next action.
Fujimoto et al. [1] mentioned that such an update rule may induce an underestimation bias. In addition, â no (8++1) is the prediction of the actor network. The actor network's parameter
is optimized according to the gradients of Q1. In other words, âc+1 tends be selected according
5

---

to the Q1 network which consistently increases the discrepancy between the two Q-functions. In practice, we observe that the discrepancy between the two estimates of the Q-function, Q Q21 can increase dramatically leading to an unstable learning process.
We found that technique call Max-min Double Q-Learning reduces the discrepancy between the Q-functions. We first select 0r+1 according to the actor network ($t+)))). Then we run CEM to search the landscape of Q2 within broad neighborhood of uz+1 return a second action u,+1- Note that CEM selects an action Ø++1 that maximises Q2 while the actor network selects an action ār+1 that maximises Q1. We gather four different Q-values: Q(8++1, 0x+1; 01), Q(+++1, Ū++1; 02), (s+++1 @ f1), and ((1++1, a++1:02) We then run a max-min operation to compute the target value that cancels the biases induced by ōe+1 and ă++1-
Ti + max min Q(5++1,0++) 0z),
1=1,2
min 0($41,1(41::))) 1.2
The inner min-operation minj=1.2 Q(5++1, 1; 0,) is adopted from Eq. and mitigates overestima-
tion [24]. The outer operation helps reduce the difference between Q1 and Q2- In addition, the max operation provides a better approximation of the Bellman optimality operator [6]. We visualize Qi and Q2 during the learning process in the supplementary material. We formalize the convergence of the proposed Max-min Double Q-Learning approach in the finite MDP setting and prove this theorem in the Appendix.
5 Experiments
75:0
8020
5020
1630
4020
15:00
2010
0.0
0.6
0.0
0.2
0.4
0.8
(a) Ant-v2
(b) Hopper-v2
(c) Humanoid-v2
16.000
6000
12:00
8200
3000
4200
0.0
0.5
1.2
1.8
2.4
3.5
0.3
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.0
3.0
(d) HalfCheetah-v2
(e) S.Simmmr-22
(f) Walker2d-v2
Figure 1: Learning curves for the OpenAl gym continuous control tasks. For each task, we train 10 instances of each algorithm, using 10 different seeds. Evaluations are performed every 5000 interactions with the environment. Each evaluation reports the return (total reward), averaged over 10 episodes. For each training seed, we use a different seed for evaluation, which results in different start states. The solid curves and shaded regions represent the mean and standard deviation of the average return over seeds. We do not apply any smoothing to the learning curves. GRAC (orange) learns faster than other methods across all tasks. GRAC achieves comparable result to the popular methods on the Hopper-v2 task and the Ant-v2 task and outperforms prior methods on the other four tasks including the complex tighdiimensiona Humanoid-v2.
5.1 Comparative Evaluation
We present GRAC, a self-guided and selulregul actor-critic algorithm as summarized in Algo- rithm 1. To evaluate GRAC, we measure its performance on the suite of MuJoCo continuous control

---

tasks [26], interfaced through OpenAI Gym [11]. We compare our method with DDPG [5], TD3 [1], CEM-RL [9]. TRPO [4], SAC* [2], and SAC² [27]. We use the source code released by the original authors and adopt the same hyperparameters reported in the original papers and the number of training steps according to SAC* [2]. For CEM-RI. [9] and SAC [27], we use the results provided by their corresponding authors. SAC does not contain results on Swimmer-v2. CEM-RL. does not contain results on Humanoid-v2 and only runs one millions step on every tested task. Hyperparameters for all experiments are in the Appendix. Results are shown in Figure 1. GRAC outperforms or is comparable to all other algorithms in both final performance and learning speed across all tasks. On complex tasks with high state and action dimensions such as Humanoid-v2. GRAC outperforms all other algorithms by a large margin. Both CEM-RL and GRAC leverage CEM. CEM-RL RLL applies CEM within the actor parameter space which is extremely nigh-dimensional while GRAC utilizes CEM within the action space of the Q function. GRAC outperform CEM-RI. on tasks such as Ant-v2, HalfCheetah-v2, Swimmer-v2 by a large margin.
5.2
Ablation Study
DDFG
GOPG
DC#G
DOPG sewrad,
21K
13:0
31X
11%
2000 10300 CGX W0o 1ii0
32
2.4
12
(a) Returns on Hopper-v2
(b) Average of Q1 over training batch on Hopper-v2
(c) Returns on Hopper-v2 over one million steps
Figure 2: Learning curves and average Q1 values (yí in Alg. 1) on Hopper-v2. DDPG w/o target network quickly diverges seen by the unrealistically high Q values. DDPG is stable but progresses slowly. we remove the target network and add the proposed target regularization, we both maintain stability and achieve faster learning than DDPG.
In this section, we present ablation studies to under- stand the contribution of each proposed component: Self- Regularized TD-Learning (Section 4.1) and Self-Guided Policy Improvment (Section 4.2). We present results in Fig. 4 in which we compare the performance of GRAC with alternatives, each removing one component from GRAC. Additional learning curves can be found in the Ap- pendix. We also experiments examine how sensitive GRAC is to some hyperparameters such as a and K listed in Alg. and the results can be found in the Appendix.
300
200
1CM
20M
30M
4014
50M
Figure 3: Average returns for the BreakoutNo environment on OpenAl gym. Blue is DQN, red is DQN w/o target network w/ target regularization.
Self- Regllaeglr TD Learning To verify the effective- ness of the proposed self-regularized TD-learning method, we apply our method to DDPG (DDPG w/o target net- work w/ target regularization). We compare against two baselines: the original DDPG and DDPG without target networks for both actor and critic (DDPG w/o target network). We choose DDPG, because it does not have additional components such as Double Q-Learning, which may complicate the analysis of this comparison.
In Fig. 2, we visualize the average returns, and average Q1 values over training batches (3'1 in Alg. The Q1 values of DDPG w/o target network changes dramatically which leads to poor average returns. DDPG maintains stable Q values but makes slow progress. Our proposed DDPG w/o target network w/ target regularization maintains stable Q values. In addition, we compare the average returns of DDPG w/o target network, w/ targel regularization and DDPG within one million steps over four random seeds. DDPG w/o target network, w/ target regularization outperforms DDPG by large margins in five out of six Mujoco tasks (Fig. 2 shows results on Hopper-v2. The remaining results can be found in the Appendix). We also apply self-regularized TD-leamnig to DQN called DQN w/o target network w/ target regularization on the Atari Breakout environment and it outperforms DQN by 25%. The learning curve over 50 million steps is shown in Fig.3. These results on two different -elarrning methods demonstrate the effectiveness of our method and its potentials to be applied to a wide range of DRL methods.
7

---

Policy Improvement with Evolution Strategies The GRAC actor network uses combination of two actor loss functions, denoted QLoss and CEMLoss. QLoss refers to the unbiased gradient estimators which extend the DtPG-style policy gradients [5] to stochastic policies. CFMLoss represents the policy improvement guided by the action found with the zero-order optmization method CEM. We run another two ablation experiments on all six control tasks and compare it with our original policy training method denoted as GRAC. As seen in Fig. 4, in general GRAC achieves better performance compared to either using CEMLoss or QLoss, The significance of the improvements varies over the six control tasks. For example, CEMLoss plays dominant role in Swimmer while LLoss has major effect in HalfCheetah. This suggests that the CEMLoss and Losss are complementary.
GRAC
GRAC wis COE w/ ninQUpdate
GRAC w/o OOOCCEE
GRAC wo CEMLass
GRAC we GLoSs
Normalieed average raturn
0.6
0.4
0.2
HallCheetan--2
Ant-v2
Swimmer-y2
Walker2ddi22
Hlopper-22
Humanoidivv2
Figure 4: Final average returns, normalized w.r.t GRAC for all tasks. For each task, we train each ablation setting with 4 seeds, and average the last 10 evaluations of each seed (40 evaluations in total). The black lines represent one standard deviation. Actor updates without CEMLoss (GRAC w/o CEMLoss) and actor updates w.r.t minimum of both Q networks (GRAC w/o CriticCEM w/ minQUpdate) achieves slightly better performance on Walker2d-v2 and Hopper-v2. GRAC achieves the best performance on 4 out of 6 tasks, especially on more challenging tasks with higher-dimensional state and action spaces (Humanoid-v2, Ant-v2, HalfChectah-v2). This suggests that individual components of GRAC complement each other.
5.3 In Hand Manipulation
We evaluate our approach on the problem of in-hand manipulation which remains unsolved due the high dimensionality of the prob- lem and the complexity of multi-contact control [28]. Having this capability would allow robots to perform sophisticated tasks quiring repositioning and reorienting of grasped objects. We apply GRAC to a non-anthropomorphic robotic hand with 9 degrees of freedom [29]. We train the robotic hand in simulation to rotate a cube by 50 degrees around the gravity direction. The input states are the current, previous, initial, and target object position and orientation, and all nine gripper joint positions at the current time step. The actions are nine joint positions for the gripper at the next time step. GRAC learns a policy successfully accomplishing the task within 500k iterations. We test the learned policy on the real hand. Videos are included in the supplementary material.
Figure 5: Real-world Experi- ment Setup
6 Conclusion
Leveraging neural networks as function approximators, DRL has been successfully demonstrated on range of decision-making and control tasks. However, the nonlinear function approximators also introduce issues such as divergence and overestimation. As our main contribution, we proposed a self-regularized TD-lcarning method to address divergence without requiring target network that may slow down learning progress. The proposed method is agnostic to the specific Q-learning method and can be added any of them. In addition, we propose self-guided policy improvement by combining policy-gradien with zero-order optimization such as the Cross Entropy Method. This helps to search for actions associated with higher Q-values in broad neighborhood and is robust to local noise in the Q function approximation. Taken together, these components define GRAC, a novel self-guided and self-regularized actor critic algorithm. We evaluate GRAC on the OpenAl gym tasks, outperforming popular methods such as 7D3 [1] and SAC [2] on four tasks and achieving competitive results on two environments. We also run GRAC to enable a non-anthropomorphic robotic hand to successfully accomplish an in-hand manipulation task in the real world.
8

---

Acknowledgments
This work is supported by Toyota Research Institute (TRI). It is also supported by Siemens and the HAI-AWS Cloud Credits. We thank reviewers who gave useful comments.
References
[1] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. arXiv preprint wrxiv:1802.094777 2018.
[2] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:180010122000 2018.
[3] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
[4] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897, 2015.
[5] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference Learning Representations, 2016.
[6] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
[7] H. Van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv: 2018.
[8] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529, 2015.
[9] Pourchot and Sigaud. CEM-RL: Combining evolutionary and gradient-based methods for policy search. In International Conference Learning Representations, 2019. URL https: / //oenreview net/ feetfoot orwmmidd B-Bkke550cct.
[10] R. Y. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1999.
[11] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint 1606.0 2016.
[12] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. International Conference on Machine Leurning, pages 1329- 1338, 2016.
[13] N. Tsitsiklis and B. Van Roy. Analysis of temporal-diffference learning with function approximation. In Advances in neural information processing systems, pages 1075-1081, 1997.
[14] Kim, Asadi, M. Littman, and G. Konidaris. Deepmellow: Removing the need for target network in deep q-learning. In Proceedings of the Twenty-Eighth International Joinl Conference on Artificial Intelligence, IJCAI-19, pages 2733-2739 International Joint Conferences on Artificial Intelligence Organization, 7 2019, doi: 10. 0.0/4////ccciii2119//79999 URL https::/dpoo org/10. 24963 /iirs/ijjaaiaai 2019/379.
[15] I. Durugkar and P. Stone. Td learning with constrained gradients. In Proceedings of the Deep Reinforcement Learning Symposium, NISS 20/7, Long Beach, CA, USA, December 2017. URI http:/ :/wp:////ww///// cs. wtw.as.ea.......... www/eduuuueeeee -tae:r////a--abbiIIP--i---aaaaa
[16] Achiam, F. Knight, and P. Abbeel. Towards characterizing divergence deep q-learning. arXiv preprint 190.:::9900089444,,, 2019.

---

[17] A. Bhatt, M. Argus, A. Amiranashvili, and T. Brox. Crossnorm: Normalization for off- off-pollcy td reinforcement learning. arXiv preprint arXiv:1902.05603 2019.
[18] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. weiggta regression: Simple and scalable off-policy reinforcement learning. arXiv preprini wwXiv:9910000177 2019.
[19] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673 2018.
[20] R. Simmons-Edles B. Eisner, E. Mitchell, S. Seung, and Lee. Q-learning for continuous actions with cross-entropy guided policies, arXiv preprint arXiv: 10600, 2019.
[21] M. Yan, A. Li, M. Kalakrishnan, and P. Pastor. Learning probabilistic multi-modal actor models for vision-based robotic grasping. In 2019 International Conference on Robotics and Automation (ICRA), pages 1004-8800. IEEE, 2019.
[22] Z. Botev, D. Kroese, R. Rubinstein, and P. L'Ecuyer. The Cross-Entroph Method for Optimiza- tion, volume 31, pages 35-59. 12 2013.
[23] C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Oxford, 1989.
[24] S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learn- ing. In M. Mozer, P. Smolensky, D. Touretzky, J. Elman, and A. Weigend, editors, Pro- ceedings of the 1993 Connectionist Models Summer School, pages 255 263. Lawrence Erlbaum. 1993. URL http: http//wwwwwr/// ri. cmu. fttp:pubfflle//pulllep 1993_1/thrun__ sewerrunssbastiian_maea 1993_19911111 1dff.
sebastiaa
[25] H. V. Hasselt, Double q-learning, Advances in neural information processing systems, pages 2613-2621, 2010.
[26] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[27] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:7812.05905. 2018.
[28] O. M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L.. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020. 10::10. 111:10..1777777336119999777
[29] S. Yuan, L. Shao, C. L. Yako, A. Gruebele, and J. K. Salisbury. Design and control of roller grasper v2 for in-hand manipulation. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
10


Reviews:
[{"review_id": "u2z8Ro00brB", "reviewer": "Reviewer_wYEU", "summary": "This paper presented several contributions including:\n1. A self-regularized TD RL algorithm without conventional target network.\n2. A self-guided policy improvement method robust to noise (using CME).\n3. A Max-min Double Q-Learning introduced in 4.3\n\nThe proposed methods were tested in OpenAI gym tasks, although the relationship of each contribution is unclear. Section 5.3 presented a possible real task but the reviewer could not find any corresponding result. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses\n1. The link between each contribution is very weak. \n   For the self-regularized TD approach, the reviewer wonders whether the regularization term will delay the learning？\n   For the self-guided policy improvement method, it looks like a conventional policy based approach with CME, what is the contribution? What is the relationship between Theorems 1, 2 and the contribution of this paper? The CEM require the policy output a distribution of action, what is the policy network structure in GRAC?\n   For the Max-min Double Q-Learning, what is its relationship with contributions 1 and 2?\n2. No real robot experiment result in the paper. The results in attached video is too simple. The reviewer could not find how the proposed methods improved the learning performance in real robot experiment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The current paper has a mess structure. The reviewer could not clearly understand the relationship between each proposed methods and their theoretical contribution. The real experiment does not have details so the reviewer could not understand its contribution in robot learning.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please revise the paper following my comments in weaknesses. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "mM9Lp2leIel", "reviewer": "Reviewer_8B1m", "summary": "This paper presents a set of deep reinforcement learning techniques that could be exploited to improve the performance of deep reinforcement learning algorithms. The authors propose two techniques. \nThe first technique is the safe regularized TD learning, which consists of an added regularization term, preventing the network to change considerably the next state best q value. This technique frees from the necessity of having a target network.\nThe second technique is  Self-Guided Policy Improvement with Evolution Strategies, which consists of optimizing the current policy using local search with CEM, instead of using policy gradient updates. This technique instead seems to speed up the maximization of policy actions wrt the critic functions. This effect could be greatly beneficial in actor-critic methods, where the ideal condition is that both value function and policy are consistent, i.e. we want the q function to have a value close to the policy value and the policy close to being optimal w.r.t. the value function. The authors also claim that makes the update more robust to noise, which is also desirable in this context.\nFinally, the authors combine these two proposed techniques with Max-min Double Q-Learning to create GRAC, a novel deep RL algorithm. This algorithm shows very good performances in standard RL tasks. Authors also show that their approach is able to learn in simulation a manipulation task and transfer the resulting policy to the real robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes two very interesting techniques, that can be useful in Deep RL, particularly in the actor-critic settings.\nThe proposed algorithm seems to be sufficiently clean and easy to implement. The contribution looks very promising from the machine learning point of view.\nThe proposed ablation study is greatly executed and proves the claims of the paper and the advantages and complementarities of both methods.\nthe author proposes also a couple of interesting theorems on the optimality of the updates, even if the theoretical analysis should be extended (maybe in future works)\n\nThe major weakness of this paper is the experimental evaluation. The robotic experiment is a bit oversimplified, but I still think that the authors can prove the claim that this approach could be useful in a robotics environment.\nMy main concern is the Mujoco environment evaluation. After Henderson et. al \"Deep Reinforcement Learning that Matters\", it's unacceptable to see deep reinforcement learning experiments proposing new algorithms with less than 10 seeds per evaluation. While in the ablation study I can accept a more reduced statistical evaluation, it's unacceptable to have a novel algorithm with less than 10 seeds. Given that the tasks (excluding humanoid) are pretty easy to solve with moderate computational power, I suggest running 25 seeds to compute mean and confidence intervals.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has very good potential if a proper experimental campaign with statistically relevant experiments is performed.\nThe common excuse \"in other works few seeds are used\" is not acceptable, as it's important to have a solid evaluation in deep learning approaches to have solid progress in this field.\nThe algorithm looks very promising and tackles important problems of deep reinforcement learning. \nWhile I don't think that these RL approaches can be used to learn directly on a robot, I still think that the authors proved that these methods could be used to deploy a policy on the real system. \nCurrently, I propose a rejection, however, I'm willing to drastically change my recommendation if a proper experimental campaign is conducted. This can be easily achieved during the rebuttal period, as for most of the environments the computation time is very reduced.\n\nAfter the response of the authors, I'll raise my evaluation to \"weak accept\", as the experimental evaluation has been/will be fixed.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "- Do a proper experimental campaign on Mujoco tasks (except Humanoid, which is quite heavy computationally) with 25 seeds for every algorithm. The computation time for running almost every State-of-the-art approach in these environments is roughly 1 day per environment, so it's totally doable, with a normal university cluster.\n- Remove the rolling window cleanup from the graphs, it's not needed if you use a sufficient amount of seeds. Also, it could hide some instability of the method.\n- Complete a proper evaluation for the CEM baseline. It's unacceptable to see a cut line in a graph. I would rather remove the comparison from the available ones than leaving this approach with incomplete experiments. \n- In line 41, remove the \"But\" conjunction after the point. Use another more appropriate word."}}, {"review_id": "aAQczDBoJDz", "reviewer": "Reviewer_4Buh", "summary": "The authors introduce GRAC, an actor-critic RL framework that automatically regularizes the learning to prevent large spikes in Q value estimation (while obviating the need for target Q networks) and uses cross-entropy-guided search for improving action exploration economy. The proposed algorithm is theoretically justified and out-competes several current popular baselines on a number of OpenAI gym benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\nS1) As far as I can tell, this is indeed a novel framework and is theoretically compatible with any actor-critic implementation (though, realistically, it feels like it just stands on its own as an AC algorithm, given that adding blindly adding GRAC to existing AC algorithms could fundamentally break how they're meant to operate).\n\nS2) The performance of the algorithm, in my opinion, does enough to justify it as another in the playing field.\n\nS3) I appreciate the ablation study on the introduced elements as they help establish the value of the introduced algorithm elements.\n\nS4) Overall, I think the paper maintains decent flow, barring a few structural issues (see below)\n\n### Weaknesses:\n\nW1) While a reasonable case can be made for the improved convergence speed of the proposed GRAC framework, it also introduces a number of additional computational elements - there are still multiple Q networks and there's the added compute burden of the CEM optimization of action choice. To make a claim of 'faster' training, I think it would be important to also show how compute cost changes under the proposed framework relative to baselines.\n\nW2) A comparison of algorithms on the in-hand manipulation task seems to have been omitted but this omission was not justified in text. Is there some reason why the other algorithms could not be run on this task for a comparison? Given the relative performances on other tasks, I'd assume the other algorithms would have been able learn to at least somewhat handle this task if GRAC could.\n\nW3) The authors claim in the abstract that GRAC is more robust to local noise in the Q-function but I felt this is never really quantitatively or theoretically substantiated.\n\nW4) The authors claim Deep RL algorithms to be 'sample efficient' a number of times in the paper, but this is not a substantiated claim. While we can say that any one algorithm is less or more sample efficient than others, I do not believe we can make the claim yet that any deep RL methods are 'sample efficient' in an absolute sense - if anything, it's probably fair to say they're *not* sample efficient. I do not believe there has been a clear and comprehensive study on the sample efficiency of deep RL, but [1] and [2] do address it to a degree, as examples (this is honestly arguably more of a nitpick on semantics, but I feel like clarity is important for readers).\n\nW5) The compared methods (TD3, SAC, DDPG, TRPO) are referred to as state-of-the-art, though they are relatively old now (by deep learning research standards), and newer methods have been developed and open-sourced, which perform better (eg. TQC [3]). I do not begrudge the authors their choice of baselines, but perhaps 'popular' is more appropriate a term than 'state of the art'.\n\n#### Structural Issues:\nSI1) It is unclear what you mean by gradient-based RL algorithms. Aren't all deep RL algorithms gradient-based, in a sense? Are you referring to policy gradients?\n\nSI2) There is a lot of almost verbatim repetition in the text - eg. lines 64 - 68 and 79-80. It's fine to restate points, but at least paraphrasing them would help the flow - it was momentarily confusing to encounter verbatim sentences from earlier in the paper as it confused my sense of where I was in the text.\n\nSI3) Missing '+' subscript in equation 7?\n\nSI4) Line 199 and equation 9 -> change in notation from Q_1, Q_2 to Q(s,a,theta_1/2) feels unnecessary. It would read better to pick one format and stick to it.\n\nSI5) It might be better to have the algorithm appear after the end of Section 4 as it would be easier to parse after the full description has been read.\n\nSI6) Line 100: discount factor can also be 0 or 1 so \\in [0,1] would be the more appropriate notation.\n\n### Would be nice to have:\n\nI realize it's rarely welcome when a reviewer asks for more data and/or analysis but it would be good to see how the regularization term's components (as introduced in section 4.1) change as training progresses with and without GRAC. It might help make a more visual case for the value of the regularization terms, though the ablation study in 5.2 is already perhaps sufficient, hence filing this as a 'would be nice to have'\n\n#### References:\n[1] Donner, F. Measuring Progress in Deep Reinforcement Learning Sample Efficiency, 2021\n\n[2] Lee, S.Y. et al., Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update, NeurIPS 2019\n\n[3] Kuznetsov, A. et al., Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics, 2020 (also in stable baselines 3 contrib)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has its merits, and as far as I can tell, is novel work that is likely competitive with current state of the art methods in RL. The lack of clear evaluation on the robotic task and various weaknesses listed do make me feel that it isn't yet what I'd consider a strong paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I imagine all the structural issues are fairly easy to address. Weaknesses W1-3 would also need to be addressed for me to consider improving my recommendation.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}]

Author rebuttal:

Meta-review: === comments before the discussion ===

The reviewers 4Buh and 8B1m agrees that the proposed method is novel. However, reviewers raised concerns related to experiments.
Especially, please address the following points:
- do experiments with some more random seeds and update the results (I would not request results with 25 random seeds, but 10 random seeds in total are necessary.)

- address the concerns related to experiments (please refer to comments from reviewers 4Buh and 8B1m)

Although reviewer wYEU criticized that the paper does not contain any real robot experiment, CoRL can accept  papers without real-robot experiments as long as papers try to address important problems for robotics.

=== comments after the discussion ===

This paper presents novel techniques for deep reinforcement learning. Authors improved the experimental results and addressed concerns raised by reviewers. The area chair recommends the acceptance of the paper.
Decision: Accept (Poster)