Towards Simplicity in Deep Reinforcement Learning: Streamlined Off-Policy Learning
Authors: Che Wang
Yanqiu Wu
Quan Vuong
Keith Ross
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Sample Efficiency
Off-Policy Algorithms

Abstract: The field of Deep Reinforcement Learning (DRL) has recently seen a surge in the popularity of maximum entropy reinforcement learning algorithms.  Their popularity stems from the intuitive interpretation of the maximum entropy objective and their superior sample efficiency on standard benchmarks. In this paper, we seek to understand the primary contribution  of the entropy term to the performance of maximum entropy algorithms. For the Mujoco benchmark, we demonstrate that the entropy term in Soft Actor Critic (SAC) principally addresses the bounded nature of the action spaces. With this insight, we propose a simple normalization scheme which allows a streamlined algorithm without entropy maximization match the performance of SAC. Our experimental results demonstrate a need to revisit the benefits of entropy regularization in DRL. We also propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training.  We further show that the streamlined algorithm with the simple non-uniform sampling scheme outperforms SAC and achieves state-of-the-art performance on challenging continuous control tasks.

Under review as a conference paper at ICLR 2020
TOWARDS SIMPLICITY IN DEEP REINFORCEMENT LEARNING: STREAMLINED OFF-POLICY LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The field of Deep Reinforcement Learning (DRL) has recently seen surge in the popularity of maximum entropy reinforcement learning algorithms. Their pop- ularity stems from the intuitive interpretation of the maximum entropy objective and their superior sample efficiency on standard benchmarks. In this paper, we seek to understand the primary contribution of the entropy term to the perfor- mance of maximum entropy algorithms. For the Mujoco benchmark, we demon- strate that the entropy term in Soft Actor Critic (SAC) principally addresses the bounded nature of the action spaces. With this insight, we show how streamlined algorithms without entropy maximization can match the performance of SAC. We also propose simple non-uniform sampling method for selecting transitions from the replay buffer during training. We further show that the streamlined algorithm with the simple non-unifee unnuuiform sampling scheme outperforms SAC and achieves state-of-the-ar performance on challenging continuous control tasks.
INTRODUCTION
Off-policy Deep Reinforcement Learning (RL) algorithms aim improve sample efficiency by reusing past experience. Recently a number of new off-policy Deep Reinforcement Learning algo- rithms have been proposed for control tasks with continuous state and action spaces, including Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3) (Lillicrap et al., 2015 Fu- jimoto et al., 2018). TD3, which introduced clipped double-Q learning. delayed policy updates and target policy smoothing, has been shown to be significantly more sample efficient than popular on-policy methods for wide range Mujoco benchmarks.
The field of Deep Reinforcement Learning (DRL) has also recently seen surge in the popularity of maximum entropy RL algorithms. Their popularity stems from the intuitive interpretation of the maximum entropy objective and their superior sample efficiency on standard benchmarks. In particular, Soft Actor Critic (SAC), which combines off-policy learning with maximum-entee entropy RL, not only has many attractive theoretical properties, but can also give superior performance wide-range of Mujoco environments, including on the high-dimensional environment Humanoid for which both DDPG and TD3 perform poorly (Haarnoja al., 2018a b
Langlois et al.) 2019). SAC has a similar structure to TD3, but also employs maximum entropy reinforcement learning.
In this paper, we first seek to understand the primary contribution of the entropy term to the per- formance of maximum entropy algorithms. For the Mujoco benchmark, we demonstrate that when using the standard objective without entropy along with standard additive noise exploration, there is often insufficient exploration due to the bounded nature of the action spaces. Specifically, the outputs of the policy network are often way outside the bounds of the action space, so that they need to be squashed to fit within the action space. The squashing results in actions persistently taking on their maximal values, so that there is insufficient exploration. In contrast, the entropy term in the SAC objective forces the outputs to have sensible values, that even with squashing. exploration is maintained. We conclude that the entropy term in the objective for Soft Actor Critic principally addresses the bounded nature of the action spaces in the Mujoco environments.
With this insight, we propose Streamlined Off Policy (SOP), a streamlined algorithm using the standard objective without the entropy term. SOP employs simple normalization scheme to ad- dress the bounded nature of the action spaces, allowing satisfactory exploration throughout training. We also consider replacing the aforementioned normalization scheme with inverting gradients (IG)

---

Under review as a conference paper at ICLR 2020
Hausknecht & Stone (2015). Our results show that SOP and IG match the sample-efficieng and ro- bustness performance of SAC, including on the more challenging Ant and Humanoid environments. This demonstrates need to revisit the importance of entropy maximization in DRL.
Keeping with the theme of simplicity with the goal of meeting Occam's 's principle, we also propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training. In vanilla SOP (as well as in DDPG, TD3, and SAC), samples from the replay buffer are chosen uniformly random during training. Our method, called Emphasizing Recent Experience (ERE), samples more aggressively recent experience while not neglecting past experience. Unlike Priority Experience Replay (PER) (Schaul et 2015). popular non- non-uniform sampling scheme for the Atari environments, ERE is only few lines of code and does not rely on any sophisticated data structures. We show that SOP combined with ERE out-performs SAC and provides state of the art performance. For example, for Ant and Humanoid, it improves over SAC by 21% and 24%, respectively, with one million samples. Furthermore, we also investigate combining SOP with PER, and show SOP+ERE also out-performs the more complicated SOP+PER scheme.
The contributions of this paper are thus threefold. First, we uncover the primary contribution of the entropy term of maximum entropy RL algorithms when the environments have bounded action spaces. Second, we propose streamlined algorithm which do not employ entropy maximization but nevertheless matches the sampling efficiency and robustness performance of SAC for the Mu- joco benchmarks. And third, we combine our streamlined algorithms with a simple non-uniform sampling scheme to achieve state-of-the art performance for the Mujoco benchmarks. We provide anonymized code for reproducibility [
PRELIMINARIES
We represent an environment as a Markov Decision Process (MDP) which is defined by the tuple (S, A, r.p. r.) where and A are continuous multi-dimensional state and action spaces, r(s, a) is bounded reward function, p(s'ls, a) is transition function, and y is the discount factor. Let s(t) and a(t) respectively denote the state the environment and the action chosen at time t. Let (Ta) s), S,a E A denote the policy. We further denote K for the dimension of the action space, and write aş for the kth component of an action a E A, that is, a = (a1....,
The expected discounted return for policy π beginning in state is given by:
(+) E,
((((((),
Standard MDP and RL problem formulations seek to maximize Va (s) over policies n. For finite state and action spaces, under suitable conditions for continuous state and action spaces, there exists an optimal policy that is deterministic (Puterman 2014 Bertsekas & Tsitsiklis 1996). In RL with unknown environment, exploration is required to learn a suitable policy.
In DRL with continuous action spaces, typically the policy is modeled by parameterized policy network which takes as input a state 8 and outputs a value {=(;(0), f). where represents the current parameters of the policy network (Schulman et al., 2015) 2017] Vuong et al., 2018 Lillierap al., 2015 Fujimoto et al., (018). During training, typically additive random noise is added for explo- ration, so that the actual action taken when in state takes the form a pss; 0) + e where is a K- K-dimensional Gaussian random vector with each component having zero mean and variance a. During testing, is set zero.
2.1 ENTROPY MAXIMIZATION RL
Maximum entropy reinforcement learning takes different approach than II by optimizing policies
to maximize both the expected return and the expected entropy of the policy (Ziebart et al. 2008
Ziebart
2010 Todorov 2008 Rawlik et al. 2013: Levine & Koltun 2013 Levine al. 2016
Nachum et al. 2017 Haarnoja et al. 2017 2018a
https://ancnymouss 4oreess.opppe https//eeeeccccccce//////aaaaaa
1a:-888---666-- a001
2

---

Under review as a conference paper at ICLR 2020
In partieular, with maximization entropy RL, the objective is maximize
Vi(()
Σ 7 ('Ex)r(s(t),a(t))) + (H = 1=0
(2)
where (=((ss)) is the entropy of the policy when in state s, and the temperature parameter À determines the relative importance of the entropy term against the reward.
For entropy maximization DRL, when given state the policy network will typically output K- dimensional vector (s;; 8) in addition to the vector ju(s; 8). The action selected when in state is then modeled as (s;; 0) + where (0, ( (s; 0)).
Maximum entropy RL has been touted to have a number of conceptual and practical advantages for DRL (Haarnoja al., 2018a b). For example, it has been argued that the policy is incentivized to explore more widely, while giving up on clearly unpromising avenues. It has also been argued that the policy can capture multiple modes of near-optimal behavior, that is, in problem settings where multiple actions seem equally attractive, the policy will commit equal probability mass to those actions. In this paper, we show for the Mujoco benchmarks that the standard additive noise exploration suffices and can achieve the same performance as maximum entropy RL.
3
THE SQUASHING EXPLORATION PROBLEM
3.1 BOUNDED ACTION SPACES
Continuous environments typically have bounded action spaces, that is, along each action dimension k there is a minimum possible action value ob and maximum possible action value og When selecting an action, the action needs to be selected within these bounds before the action can be taken. DRL algorithms often handle this by squashing the action so that it fits within the bounds. For example, if along any one dimension the value {us;;) 6) + e exceeds Omax. the action is set (clipped) to 4max- Alternatively, a smooth form of squashing can employed. For example, suppose ag - MM and mmax + M for some positive number M, then a smooth form of squashing could use a = .M (anh(u(s; s; 0) + e) in which tanh() is being applied to each component of the K -ddimensional vector. DDPG (Hou et al., 2017) and TD3 (Fujimoto et al. (0018) use clipping, and SAC (Haarnoja ct al. 2018a uses smooth squashing with the tanh() function. For concreteness, henceforth we will assume that smooth squashing with the tanh() is employed.
We note that an environment may actually allow the agent to input actions that are outside the bounds. In this case, the environment will typically first clip the actions internally before passing them on to the "actual" environment (Fujita & Maeda 2018).
We now make simple but crucial observation: squashing actions to fit into bounded action space can have a disastrous effect on additive-noise exploration strategies. To see this, let the output of the policy network be u()) = (μ (s)...., (K((..,)(s)))))) Consider an action taken along one dimension k. and suppose Pa (s) >> and lea| is relatively small compared to (s). Then the action ds M (E)) will be very close (essentially equal) to M. If the condition (s)s) >> > persists over many consecutive states, then 0g will remain close to for all these states. and consequently there will be essentially no exploration along the keth dimension. We will refer to this problem as the squashing exploration problem. A similar observation was made in Hausknecht & Stone (2015). We will argue that algorithms such as DDPG and TD3 based on the standard objective (I) with additive noise exploration can be greatly impaired by squashing exploration.
3.2 WHAT DOES ENTROPY MAXIMIZATION BRING TO SAC FOR THE MUJUCO ENVIRONMENTS?
SAC is a maximum-entropy based off-policy DRL algorithm which provides good performance across all of the Mujuco benchmark environments. To the best of our knowledge, it currently pro- vides state of the art performance for the Mujoco benchmark. In this section, we argue that the prin- cipal contribution of the entropy term in the SAC objective is to resolve the squashing exploration problem, thereby maintaining sufficient exploration when facing bounded action spaces. To argue this, consider two DRI. algorithms: SAC with adaptive temperature (Haarnoja al. (018b). and
3

---

Under review as a conference paper at ICLR 2020
SAC with entropy removed altogether (temperature set zero) but everything clse the same. We refer to them as SAC and as SAC without entropy. For SAC without entropy, for exploration we use additive zero-mean Gaussian noise with fixed at 0.3. Both algorithms use tanh squashing. We compare these two algorithms on two Mujoco environments: Humanoid-v2 and Walker-v2.
shows Sshoss the performance of the two algorithms with 10 seeds. For Humanoid, SAC performs much better than SAC without entropy. However, for Walker, SAC without entropy performs nearly as well SAC, implying maximum entropy RL is not critical for this environment.
Figure
5006
SAC
average fest return
GAC Affiict wntropy
overage lest return
SAC at setince
-200
1000
1030
1200
0.2
04
36
09
100
9.2
06
1.
environment Intoractions
environment Interactions
(a) Humanoid-v2
(b) Walker2d-v2
Figure 1: SAC performance with and without entropy maximization
To understand why entropy maximization is important for one environment but less so for another, examine the actions selected when training these two algorithms. Humanoid and Walker have action dimensions K = 17 and K = 6, respectively. Here we show representative results for one dimension for both environments, and provide the full results the Appendix. The top and bottom rows of Figure shows results for Humanoid and Walker, respectively. The first column shows the (4 values for interval of 1, 1,000 consecutive time steps, namely, for time steps 599,000 to 600,000. The second column shows the actual action values passed to the environment for these time steps. The third and fourth columns show a concatenation of 10 such intervals of 1000 time steps, with each interval coming from a larger interval of 100,000 time steps.
The top and bottom rows of Figure strikingly different. For Humanoid using SAC with entropy, the | values are small, mostly in the range [-1.5,1.5], and fluctuate significantly. This allows the action values to also fluctuate significantly, providing exploration in the action space. On the other hand, for SAC without entropy the | values are typically huge, most of which well outside the interval [-10,10 10]. This causes the actions ag to be persistently clustered at either M or -M, leading to essentially exploration along that dimension. As shown in the Appendix, this property (lack of exploration for SAC without entropy maximization) holds for all 17 action dimensions. For Walker, see that for both algorithms, the /4 values sensible, mostly in the range [-1,1] and therefore the actions chosen by both algorithms exhibit exploration.
In conclusion, the principle benefit of maximum entropy RL in SAC for the Mujuco environments is that it resolves the squashing exploration problem. For some environments (such as Walker), the outputs of the policy network take on sensible values, so that sufficient exploration is maintained and overall good performance is achieved without the need for entropy maximization. For other environments (such as Humanoid), entropy maximization is needed to reduce the magnitudes of the outputs so that exploration is maintained and overall good performance is achieved.
4 STREAMLINED OFF-POLICY (SOP) ALGORITHM
Given the observations in the previous section, a natural question is: is it possible to design stream- lined off policy algorithm that does not employ entropy maximization but offers performance com- parable to SAC (which has entropy maximization)?
As we observed the previous section, without entropy maximization, in some environments the policy network output values [ua]. k = I,..., K can become persistently huge, which leads to insufficient exploration due to the squashing. A simple solution is to modify the outputs of the policy network by normalizing the output values when they collectively (across the action dimensions) become too large. To this end, let μ = (Mi,.... pk) be the output of the original policy network, and let G Ex | K. The G is simply the average of the magnitudes of the components of p. The
4

---

Under review as a conference paper at ICLR 2020
101 1011 1011
101
1n1
model ouegut
actton 02
model outpus
1n1
-10"
-10"
son She Mi Bouon time steps
504 St4 Spe Saa Soa
700 2a Sae
1200 200
time steps
time steps
time steps
(a) Humanoid-v2
model output
model outgut
0.23
2.00
-10
-1.00
599.200 3yn ".... " time steps
50t "on Sn time steps
y08 Sa Miga time steps
time steps
(b) Walker2d-v2
Figure 2: pa and ag values from SAC and SAC without entropy maximization
normalization procedure is as follows. If G > 1, then we reset μ4 + M for all k = 1...., otherwise, we leave μ unchanged. With this simple normalization, we are assured that the average of the normalized magnitudes is never greater than one. Henceforth we assume the policy network has been modified with the simple normalization scheme just described.
Our Streamlined Off Policy (SOP) algorithm is described in Algorithm The algorithm is es- sentially DDPG plus the normalization described above, plus clipped double Q-learning and target policy smoothing (Fujimoto et al., 2018). Another way of looking at it is as TD3 plus the normal- ization described above, minus the delayed policy updates and the target policy parameters. SOP also uses tanh squashing instead of clipping, since tanh gives somewhat better performance in experiments. The SOP algorithm is "streamlined" as has no entropy terms, temperature adapta- tion, target policy parameters delayed policy updates. In experiments, we also consider TD3 plus the simple normalization, and also another streamlined algorithm which we replace the sim- ple normalization scheme described above with the inverting gradients (IG) scheme described in Hausknecht & Stone 2015). The basic idea is: when gradients suggest increasing the action mag- nitudes, gradients will be downscaled if actions within the boundaries, and inverted entirely actions are outside the boundaries. More implementation details can be found in the Appendix.
Algorithm Streamlined Off-Policy
Input: initial policy parameters 0, Q-function parameters Φ1, Φ2, empty replay buffer D
2: Set target parameters equal to main parameters Otarg, + 6; for = 1,2
3: repeat
4: Generate episode using actions Mtann((uas)) + where (1)
5: for j in range(however many updates) do
6: Randomly sample batch of transitions, B {(s, from D
7: Compute targets for Q functions:
Dg (r, 8') = + min;=1,2 Qebun. (s', (too +)) (~(00)))))
8:
Update Q-functions by one step of gradient descent using
Vo. jiy E(s,o.r,')) (Q s. a) (r, s')) for = 1,2
9:
Update policy by one step of gradient ascent using Ve V 1
( E.En B Qo, (s, M tanh(ue (s)))
10:
Update target networks with Puag + powag + p) for 1,2
4.1 EXPERIMENTAL RESULTS FOR SOP
Figure 3 compares SAC (with temperature adaptation (Haamoja et al. 201 Sa b)) with SOP, TD3+
(that TD3 plus the simple normalization), and inverting gradients (IG) for five of the most chal-
5

---

Under review as a conference paper at ICLR 2020
lenging Mujuco environments. Using the same baseline code, we train with ten different random seeds for each of the two algorithms. Each algorithm performs five evaluation rollouts every 5000 environment steps. The solid curves corresoond to the mean, and the shaded region to the standard deviation of the returns over the ten seeds.
Results show that SOP, SAC and IG have similar sample-efficiency performance and robustness across all environments. TD3+ has slightly weaker asymptotic performance for Walker and Hu- manoid. IG initially learns slowly for Humanoid with high variance across random seeds, but gives similar asymptotic performance. This confirms that with a simple output normalization scheme in the policy network, the performance of SAC can be achieved without maximum entropy RL.
In the Appendix we provide an ablation study for SOP, which shows major performance drop when removing either double Q-learning or normalization, whereas removing target policy smoothing Fujimoto et al. 2018) results in only a small performance drop in some environments.
1220
1000
E90
0000
2000
3003
00000000
1300
1300
SAC
2002
4000
182
2000
32
64
08
02
8-1
64
09
04
06
envrrnment interactions
ervioomme interactions
endiionmnttnn Lnneracnntiieraciioon
Hopper-v2
(b) Walker2d-v2
(c) HalfChectah-v2
7830
exe0ge hest retum
woseage Vel whine
00:0
2000
"00
TDS-
010
1.5
1.0
24
15
21
34
(d) Ant-v2
(c) Humanoid-v2
Figure 3: Streamlined Off-Policy (SOP) versus SAC, TD3+ and IG
NON-UNIFORM SAMPLING
We now show how a small change in the sampling scheme for SOP can achieve state of the art performance for the Mujoco benchmark. We call this sampling scheme Emphasizing Recent Ex- perience (ERE). ERE has 3 core features: (i) It is a general method applicable to any off-policy algorithm; (ii) It requires no special data structure, is very simple to implement, and has near-zero computational overhead; (iii) It only introduces one additional important hyper-parameter
The basic idea is: during the parameter update phase, the first mini-batch is sampled from the entire buffer, then for each subsequent mini-batch we gradually reduce our range of sampling to sample more aggressively from more recent data. Specifically, assume that in the current update phase we are to make 1000 mini-batee updates. Let N be the max size of the buffer. Then for the 1t update, we sample uniformly from the most recent e data points, where с₁ N 17² and 17 E (0, is hyper-parameter that determines how much emphasis we put recent data. n 1 is uniform sampling. When 7) < 1, C decreases as we perform each update. 1) can made to adapt to the learning speed of the agent that we do not have to tune it for each environment.
The effect of such a sampling formulation is twofold. The first is recent data have a higher chance of being sampled. The second is that we do this in ordered way: we first sample from all the data in the buffer, and gradually shrink the range of sampling to only sample from the most recent data. This scheme reduces the chance of wwer-wniting parameter changes made by data with parameter changes made by old data (French) 1999, McClelland et al. 1995] McCloskey & Cohen, 1989 Rateliff 1990. Robins 1995). This process allows us to quickly obtain new information

---

Under review as a conference paper at ICLR 2020
from recent data, and better approximate the value functions near recently-visited states, while still maintaining an acceptable approximation near states visited in the more distant past.
What is the effect of replacing uniform sampling with ERE? First note if we do uniform sampling on a fixed buffer, the expected number of times a data point is sampled is the same for all data points. Now consider scenario where we have buffer of size 1000 (FIFO queue), time, and then perform one update with mini-batch size of one. If we start with an empty buffer and sample uniformly, as data fills the buffer, each data point gets less and less chance of being sampled. Specifically, over a period of 1000 updates, the expected number of times the /th data is sampled is: 1/t+ + 1/(t + 1) + + 1/T. Figure 44 shows the expected number of times a data is sampled as a function of its position in the buffer. We see that older data are expected to get sampled much more than newer data. This is undesirable because when the agent is improving and exploring new areas of the state space; new data points may contain more interesting information than the old ones, which have already been updated many times.
collect one data at
When we apply the ERE scheme, we effectively skew the curve towards assigning higher expected number of samples for the newer data, allowing the newer data to be frequently sampled soon after being collected, which can accelerate the learning process. In the Appendix, we provide further algorithmic detail and analysis on ERE, and compare ERE to two other sampling schemes: an exponential sampling scheme and Prioritized Experience Replay (Schaul et al. 2015).
5.1 EXPERIMENTAL. RESULTS FOR SOP+ERE
Figure 4 compares the performance of SOP, SOP+ERE, SAC and SAC+ERE. With ERE, both SAC and SOP gain a significant performance improvement in all environments. SOP+ERE learns faster than SAC and vanilla SOP in all Mujoco environments. SOP+ERE also greatly improves overall performance for the two most challenging environments, Ant and Humanoid, and has the best per- we show the mean test episode return and std across 10 random
formance for Humanoid. In table seeds at IM timesteps for all environments. The last column displays the percentage improvement of SOP+ERE over SAC, showing that SOP+ERE achieves state of the art performance. In Ant and Humanoid, SOP+ERE improves performance by 21% and 24% over SAC at I million timesteps, respectively. As for the std, SOP+ERE gives lower values, and for Humanoid higher value.
I
2500
400
1900
2000
300
1000
800
SAC
DGP
SAS
Son
6000
SAC SORIERE
20
SDP-
1000
SOTHERE
2000
50
211
0/2
08
08
anteacione
(a) Hopper-v2
(b) Walker2d-v2
(c) HallCheetah-v2
2000
1000
aepecting tel # detappint sampledd
ficel
evpty
5000
5000
sechs
w00
2000
300
SOP
2001
son SACIERS SOF
SACEERE
cedformeet inecacione
M
M
time datapoint
collected
(d) Ant-v2
(e) Humanoid-v2
(f Uniform and ERE sampling
Figure 4: (a) to (c) show the performance of SOP and SAC with ERE sampling. (f) shows over period of 1000 updates, the expected number of times the 7th data point sampled (with r = 0.996). ERE allows new data to be sampled many times soon after being collected.
7

---

Under review as a conference paper at ICLR 2020
Table 1: Performance comparison at one million samples. Last column shows percentage improve- ment of SOP+ERE over SAC.
Environment
SAC Adaptive
SOP
SOP+ERE
Improvement
Hopper
3161.2 + 381.0
3317.3 + 133.9
3201.5 + 248.7
1.3%
Walker
4801.5 514.5
4
$145.9
7.2%
HalfCheetah
10963.7 512.4
9968.0 497.4
11335. 478.3
3.4%
Ant
4153.7 + 925.0
4674.0 588.8
5023.3
891.6
21.0%
Humanoid
5076.2 = 148.1
4900.9 316.6
6297.7 =
24.1%
6 RELATED WORK
In recent years, there has been significant progress in improving the sample efficiency of DRL for continuous robotic locomotion tasks with off-policy algorithms (Lillicrap et al. 2015 Fujimoto
jtt al. 2018: Haarnoja et al. 2018a b). There also significant body of research on maximum
entropy RL methods (Ziebart et al. 2008 Ziebart 2010 Todorov 2008; Rawlik ct al. 2013 Levine
& Koltun 2013; Levine et al., 2016, Nachum et al., 2017 Haarnoja et al. 2017 201 8a b). Ahmed
al."
(2019) very recently shed light how entropy leads smoother optimization landscape.
By taking clipping in the Mujoco environments explicitly into account, Fujita & Maeda (2018)
modified the policy gradient algorithm to reduce variance and provide superior performance among on-policy algorithms. Eisenach et al. (2018) extend the work of Fujita & Maedal (2018) for when an action may be direction. Hausknecht & Stone (2015) introduce Inverting Gradients, for which we provide expermintal results in this paper for the Mujoco environments. Chou et al.) (2017) also
explores DRL the context of bounded action spaces. Dalal et al. (2018) consider safe exploration in the context of constrained action spaces.
Uniform sampling is the most common way to sample from a replay buffer. One of the most well- known alternatives is prioritized experience replay (PER) (Schaul et al., 2013). PER uses the abso- lute TD-error of data point the measure for priority, and data points with higher priority will have a higher chance of being sampled. This method has been tested on DQN (Mnih et al., 2015) and double DQN (DDQN) (Van Hasselt et al., 2016) with significant improvement and applied suc- cessfully in other algorithms (Wang et al.] 2015] 1 Schulze & Schulze 2018 Hessel et al. [018 Hou (ct al. (0117) and can be implemented in a distributed manner (Horgan et al. 2018). There are other methods proposed to make better use of the replay buffer. The ACER algorithm has on-policy part and an off- off-cy part, with a hyper-parameter controlling the ratio of off off-policy to on-policy updates (Wang et al. 2016). The RACER algorithm (Novati & Koumoutsakos 2018) selectively removes data points from the buffer, based on the degree of loff-policyness", bringing improve- ment to DDPG (Lillicrap et al.) 2015), NAF (Gu al. 2016) and PPO (Schulman al. (017). In
De Bruin et al. (2015). replay buffers of different sizes were tested, showing large butfer with data diversity can lead to better performance. Finally, with Hindsight Experience Replay(Andie (aglay(Andrychoowizz iet al., (017). priority can be given to trajectories with lower density estimation (Zaoo & Tresp 2019) to tackle multi-goal sparse reward environments.
7 CONCLUSION
In this paper we first showed that the primary role of maximum entropy RL. for the Mujoco bench- mark is to maintain satisfactory exploration in the presence of bounded action spaces. We then de- veloped a new streamlined algorithm which does not employ entropy maximization but nevertheless matches the sampling efficiency and robustness performance of SAC for the Mujoco benchmarks. Our experimental results demonstrate need to revisit the benefits of entropy regularization in DRL. Finally, we combined our streamlined algorithm with a simple non-uniform sampling scheme to achieve sfate-of-the art performance for the Mujoco benchmark.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. International Conference on Machine Learning, pp- 151-160, 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming, volume Athena Scien- tific Belmont, MA, 1996.
Po-Wei Chou, Daniel Maturana, and Scbastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In Proceedings of the 34th International Conference on Machine Learning-Voie Volume 70, pp. 834 843. JMLR. org, 2017.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.00 1888000777,.7 2018.
Tim De Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. The importance of experience replay database composition in deep reinforcement learning. In Deep reinforcement learning workshop, NIPS, 2015.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016.
Carson Eisenach, Haichuan Yang. Ji Liu, and Han Liu. Marginal policy gradients: A unified family of estimators for bounded action spaces with applications. arXiv preprint 10Xiv:8006.05 2018.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms. arXiv preprint arXiv: 1902. 10250, 2019.
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. arXiv preprint
2018.
Yasuhiro Fujita and Shin-ichi Maeda. arXiv 2018.
Clipped action policy gradient.
arXiv preprint
Shixiang Gu, Timothy Lillicrap. Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-basee acceleration. In International Conference on Machine Learning, pp. 2829- 2838, 2016.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference Machine Learning-Volle Volume 70, pp. 1352-1361. JMLR. org, 2017.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor- critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 1801.0199. 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishck Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.0590) 2018b.
Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv: /51 151411144,, 2015.

---

Under review as a conference paper at ICLR 2020
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intel- ligence, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv: 1803:0093,, 2018.
Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu, and Chunlin Chen. A novel ddpg method with prioritized experience replay. In 20/7 IEEE International Conference Systems, Man, and Cybernetics (SMC), pp. 316-321. IEEE, 2017.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.041333 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.8.. 29900, 2014.
Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.020577 2019.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference Machine Learning, pp. 1-9, 2013.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1373 2016.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint Stt::::509009777 2015.
James L McClelland, Bruce L McNaughton, and Randall C O'rreilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances Neural Information Processing Systems, pp. 2775 27885 2017.
Guido Novati and Petros Koumoutsakos, Remember and forget for experience replay, arXiv preprint arXiv:1807.05827, 2018.
Martin L Puterman. Markov Decision Processes.: Discrete Stochastic Dynamic Programming. John Wiley Sons, 2014.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. In Twenty- Third International Joint Conference on Artificial Intelligence, 2013.
10

---

Under review as a conference paper at ICLR 2020
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connection Science, 7(2): 123-146, 1995.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.00 0.9952, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Christopher Schulze and Marcus Schulze. Vizdoom: Dran with prioritized experience replay, double-q learning and snapshot ensembling. In Proceedings of SAI Intelligent Systems Confer- ence, pp. 1-17. Springer. 2018.
Emanuel Todorov. General duality between optimal control and estimation. In 2008 47th IEEE Conference on Decision and Control, pp. 4286 IEEE, 2008
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI, volume 2, pp. 5. Phoemix, AZ, 2016.
Quan Vuong, Yiming Zhang, and Keith W Ross. Supervised policy update for deep reinforcement learning. arXiv preprint
2018.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint 1.00588,, 2015.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv: 161 1.01224, 2016.
Rui Zhao and Volker Tresp. Curiosity-driven experience prioritization via density estimation. arXiv preprint arXiv:1902.08039 2019.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, figshare, 2010.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008

---

Under review as a conference paper at ICLR 2020
A ABLATION STUDY
In this ablation study we separately examine the importance of (2) the normalization at the output of the policy network; (ii) the double Q networks; (iii) and randomization used in the line 8 of the SOP algorithm (that is, target policy smoothing (Fujimoto et al. 2018)).
Figure 5 shows the results for the five environments considered in this paper. In Figure "no normalization" is SOP without the normalization of the outputs of the policy network; "single Q" SOP with one Q-network instead of two; and "no smoothing" is SOP without the randomness in line 8 of the algorithm.
Figure confirms that double Q-networks are critical for obtaining good performance (Van Hasselt et al. 2016; Fujiioto et al., 2018 Haarnoja et al., 2018a). Figure 5 also shows that output normal- ization also critical. Without output normalization, performance fluctuates wildly, and average performance can decrease dramatically, particularly for Humanoid and HalfCheetah. Target policy smoothing improves performance by relatively small amount.
6X0
2000
600
2000
2000
2303
anerage
1300 SOP angae
3.80
130
2000
envrrnmment interactions
125
enstonmment interccions
14
envronment www.mmeeeniittmmtiicocooooo interaction
(a) Hopper-v2
(b) Walker2d-v2
(c) HallCheetah-v2
6300
105 retun
1000
awwoage hed retum
4300
1000
1000
20
environment interrcton perraction
envrrrment Interactions
(d) Ant-v2
(c) Humanoid-v2
Figure 5: Ablation Study
12

---

Under review as a conference paper at ICLR 2020
B HYPERPARAMETERS
Table 2 shows hyperparameters used for SOP, SOP+ERE and SOP+PER. For adaptive SAC, we use our own PyTorch implementation for the comparisons. Our implementation uses the same hy- perparameters as used in the original paper (Haarnoja et al. 2018b). Our implementation of SOP variants and adaptive SAC share most of the code base. For TD3, our implementation uses the same hyperparamters the original paper (Fujimoto et al. 2018). They claimed that the new set of hyperparamters can improve performance for TD3. We now discuss hyperparameter search for better clarity, fairness and reproducibility (Henderson et al. 2018 Duan al. 2016 Islam al. 2017).
used in the authors' implementation, which is different from the ones in
For the 7) value in the ERE scheme, in our early experiments we tried the values (0.993, 0.994, 0.995, 0.996, 0.997, 0.998) on the Ant and found 0.995 to work well. This initial range of values was decided by computing the ERE sampling range for the oldest data. We found that for smaller values, the range would simply be too small. For the PER scheme, we did some informal preliminary search, then searched Ant for 31 in (0, 0.4, 0.6, 0.8), 32 in (0, 0.4, 0.5, 0.6, 1), and learning rate in (le-4, 2e-4, 3e-4, 5e-4, 8e-4, le-3), we decided to search these values because the original paper used B1 0.6, A2 0.4 and with reduced learning rate. For the exponential sampling scheme, we searched the À value in (3c-7, le-6, 3e-6, 5e-6, le-5, 3e-5, 5c-5, lc-4) in Ant, this search range was decided by plotting out the probabilities of sampling, and then pick a of values that not too extreme. For a in SOP, in some of our early experiments with SAC, we accidentally found that 0.3 gives good performance for SAC without entropy and with Gaussian noise. We searched values (0.27, 0.28, 0.29, 0.3). For values for TD3+, we searched values (0.1, 0.15, 0.2, 0.25, 0.3).
Table 2: SOP Hyperparameters
Parameter
Value
Shared
optimizer
Adam Kingma & Ba 2014)
learning rate
3-10 10
discount (7)
0.99
target smoothing coefficient (p)
0.005
target update interval
replay buffer size
106
number of hidden layers for all networks
2
number of hidden units per layer mini-batch size
256
256
nonlinearity
ReLU
SAC adaptive
entropy target
dim(.A) (e.g., for HalfCheetah-v2)
SOP
gaussian noise std a 0
0.29
TD3
gaussian noise std for data collection o guassian noise for target policy smoothing ō
0.1 action limit
0.2
TD3+
gaussian noise std for data collection
0.15
guassian noise std for target policy smoothing ō
0.2
ERE
ERE initial v/
0.995
PER
PER 31 (a in PER paper)
0.4
PER 32 (3 in PER paper)
0.4
EXP
Exponential A
5e - 06
13

---

Under review as a conference paper at ICLR 2020
C ERE PSEUDOCODE
Algorithm SOP with Emphasizing Recent Experience
Input: initial policy parameters 0, Q-function parameters é1, d2, empty replay buffer D of size
N, initial No. recent and max performance improvement Lecent = Iyax = 0.
2: Set target parameters equal to main parameters Ourri + 9; for 1,2
3: repeat
4: Generate an episode using actions a Mpann((pos)) + c) where c ((,
5: update Fecent, Iman with training episode returns, let K = length of episode compute 170 Leacent + (1 Iman
6:
Imae
7:
for j in range(K) K) do
8:
Compute = . 17 "
R
9:
Sample batch of transitions, B ((8, r, from most recent eg data in D
10:
Compute targets for Q functions:
(r, 5') r min4=1.2 Qdw.. (s', (taan (5)))++++)+) +)) 8 8~
11:
Update Q-functions by one step of gradient descent using
Vo. TET E(saa,,s') (Q, a) (r)r)'))) for 1,2
Update policy by one step of gradient ascent using Vo (ol EeEB Qs,1 M tanh(Hu(s)))
13:
Update target networks with
Puurg.i f potug. + (1 - p)o; for i = 1,2
14

---

Under review as a conference paper at ICLR 2020
D INVERTING GRADIENT METHOD
In this section we discuss the details of the Inverting Gradient method.
Hausknecht & Stone (2013) discussed three different methods for bounded parameter space learning: Zeroing Gradients, Squashing Gradients and Inverting Gradients, they analyzed and tested the three methods and found that Inverting Gradients method can achieve much stronger performance than the other two. In our implementation, we remove the tanh function from SOP and use Inverting Gradients instead bound the actions. Let p indicate the output of the last layer of the policy network. During exploration will be the mean of normal distribution that we sample actions from, the IG approach can be summarized by the following equation (Hausknecht & Stone, 2015):
v Vp { (Pmax (Pmmx Pmin
if suggests increasing p otherwise
(p- Punin)/ (Pmax
Pmin)
Where vp the gradient of the policy loss w.r. p. During policy network update, we first backpropagate the gradients from the outputs of the Q network to the output of the policy network for each data point in the batch,
then compute the ratio (Pmax - -p) (P Pmin) or (Pmax p)/ (Puax Puuin) for each p value (each action dimension), depending on the sign of the gradient. We then backpropagate from the output of the policy network parameters of the policy network, and we modify the gradients in the policy network according to the ratios we computed. We made an efficient implementation and further discuss the computation efficiency of IG the implementation details section.
E SOP WITH OTHER SAMPLING SCHEMES
We also investigate the effect of other interesting sampling schemes.
E.
SAC WITH PRIORITIZED EXPERIENCE REPLAY
We also implement the proportional variant of Prioritized Experience Replay (Schaul et al. 2015
with SOP.
Since SOP has two Q-networks, we redefine the absolute TD error |6] of transition (s, a, 8') to be the average absolute TD error in the Q network update:
[6]
wa(r, Qo.(s,a)
(4)
Within the sum, the first term (r s') f Qures "es,ttaeh((ss)) +0)), 8 ~~ NO 02)
is simply the target for the Q network, and the term Qo.(s, a) is the current estimate of the 1t Q network. For the th data point, the definition of the priority value p. is p. 18. +.. The probability of sampling a data point P(i) is computed as:
P()) P
(5)
Σ
where 31 is a hyperparameter that controls how much the priority value affects the sampling proba- bility, which denoted by in Schaul et al. (2015), but avoid confusion with the SAC, denote it as 31. The importance sampling (IS) weight w, for data point is computed as:
1
1
Wi
(6)
N
P(i)
where B2 is denoted as @ in Schaul et al. 2015).
Based on the SOP algorithm, change the sampling method from uniform sampling sampling using the probabilities P(i), and for the Q updates we apply the IS weight w, This gives SOP with Prioritized Experience Replay (SOP+PER). We note that as compared with SOP+PER, ERE
15

---

Under review as a conference paper at ICLR 2020
does require special data structure and has negligible extra cost, while PER uses sum-tree structure with some additional computational cost. We also tried several variants of SOP+PER, but preliminary results show that it is unclear whether there is improvement in performance, we kept the algorithm simple.
E.2 SOP WITH EXPONENTIAL SAMPLING
The ERE scheme is similar to an exponential sampling scheme where we assign the probability of sampling according to the probability density function of an exponential distribution. Essentially, in such a sampling scheme, the more recent data points get exponentially more probability of being sampled compared to older data.
For the tt most recent data point, the probability of sampling data point P(t) is computed as:
P(i)
Ae-
We apply this sampling scheme to SOP and refer this variant as SOP+EXP.
E.3 PER AND EXP EXPERIMENT RESULTS
Figure shows a performance comparison of SOP, SOP+ERE, SOP+EXP and SOP+PER. Results show that the exponential sampling scheme gives boost to the performance of SOP, and especially the Humanoid environment, although not as good as ERE. Surprisingly, SOP+PER does not give significant performance boost SOP (if any boost at all). We also found that it is difficult to find hyperparameter settings for SOP+PER that work well for all environments. Some of the other hyperparameter settings actually reduce performance. It is unclear why PER does not work well for SOP. A similar result has been found in another recent paper (Fu et al. (019), showing that PER can significantly reduce performance on TD3. Further research is needed understand how PER can be successfully adapted to environments with continuous action spaces and dense reward structure.
4000
1980
6000
0000
1000
RCP
0:03
S0P
1000
500
100
2010
SOPHEXP
D4
intractions
endforment Pntaaclion
(a) Hopper-v2
(b) Walker2d-v2
(c) Halfchectah-v2
7000
5000
4000
2000
KgP S0PrE3P
1002
1000
83
35
12
12
25
30
30
05
13
1.5
20
25
30
(d) Ant-v2
(c) Humanoid-v2
Figure 6: Streamlined Off-Policy (SOP), with ERE and PER sampling schemes
F ADDITIONAL ERE ANALYSIS
Figure 7 shows, for fixed n, how 77 affects the data sampling process, under the ERE sampling scheme. Recent data points have a much higher probability of being sampled compared to older data, and smaller 7) value gives emphasis recent data.
16

---

Under review as a conference paper at ICLR 2020
Different 7) values are desirable depending how fast the agent is learning and how fast the past experiences become obsolete. So to make ERE work well in different environments with different reward scales and learning progress, performance to be the training episode return. Define recent to be how much performance improved from N/2 timesteps ago, and maz to be the maximum improvement throughout training, where N is the buffer size. Let the hyperparameter 7/ be the initial value. We then adapt 1) according to the formula: 1) 7/0 /recent//nnnn + 1 (Irecent/ (aaeennt/mmaa>
adapt 7 to the the speed of learning. To this end, define
Under such an adaptive scheme, when the agent learns quickly, the value is low in order to learn quickly from new data. When progress slow, 77 is higher to make use of the stabilizing effect of uniform sampling from the whole buffer.
0.1
0.
08
596
0.066
0.215
01 131 001
k 160
XE,00 401330 MEC ROURES SUKEACS
pohnt
(b)
Figure 7: Effect of different y values. The plots assume replay buffer with I million samples, and 1,000 mini-batches size 256 in an update phase. Figure 7a plots og (ranging from to million) function of is (ranging from to 1,000). Figure 7b plots the expected number of times data point in the buffer sampled, with the data points ordered from most to least recent.
G ADDITIONAL IMPLEMENTATION DETAILS
G.1 ERE IMPLEMENTATION
In this section we discuss some programming details. These details are not necessary for under- standing the algorithm, but they might help with reproducibility.
In the ERE scheme, the sampling range always starts with the entire buffer (IM data) and then gradually shrinks. This is true points the buffer, we compute C based as if there are 1M data points in the buffer. One can also modify the design slightly to obtain variant that uses the current amount of data points compute cg. In addition to the reported scheme, we also tried shrinking the sampling range linearly, but it gives less performance gain.
when the buffer is not full. So even if there are not many data
In our implementation we set the number of updates after an episode to be the same as the number of timesteps in that episode. Since environments do always end at 1000 timesteps, we can give a more general formula for Ck- Let K be the number of mini-batch updates, let N be the max size of the replay buffer, then:
C N n7
1000
With this formulation, the range of sampling shrinks in more or less the same way with varying number of mini-batch updates. We always do uniform sampling in the first update, and we always have 7 1980
17" 1000 in the last update.
When r) is small, eg can also become small for some of the mini- nini-batches. To prevent getting a mini- batch with too many repeating data points, we set the minimum value for e to 5000. We did not find this value to be too important and did not find the need to tune it. also does not have any effect for any 7 0.995 since the sampling range cannot be lower than 6000.
In the adaptive scheme with buffer of size 1M, the recent performance improvement is computed as the difference of the current episode return compared to the episode return 500,000 timesteps earlier.
17

---

Under review as a conference paper at ICLR 2020
Before reach 500,000 timesteps, simply use no. The exact way of computing performance improvement does not have significant effect on performance as long as it is reasonable.
G.2 PROGRAMMING AND COMPUTATION COMPLEXITY
In this section we give analysis on the additional programming and computation complexity brought by ERE and PER.
In terms of programming complexity, ERE is a clear winner since only requires a small adjustment to how sample mini-batches. It does not modify how the buffer stores the data, and does not require a special data structure to make work efficiently. Thus the implementation difficulty is minimal. PER (proportional variant) requires sum-tree data structure make The implementation is not too complicated, but compared to ERE it is a lot more work.
efficiently.
The exponential sampling scheme is very easy implement, although a naive implementation will incur a significant computation overhead when sampling from a large buffer. To improve its com- putation efficiency, we instead uses an approximate sampling method. We first sample data indexes from segments of size 100 from the replay buffer, and then for each segment sampled, we sample one data point uniformly from that segment.
In terms of computation complexity (not sample efficiency), and wall-clock time, ERE's extra com- putation is negligible. In practice we observe no difference computation time between SOP and SOP+ERE. PER needs to update the priority of its data points constantly and compute sampling probabilities for all the data points. The complexity for sampling and updates is (log((N), and the rank-based variant is similar (Schaul et al., 2015). Although this is not too bad, does impose a significant overhead on SOP: SOP+PER runs twice as long as SOP. Also note that this overhead grows linearly with the size of the mini-batch. The overhead for the Mujoco environments is higher compared to Atari, possibly because the Mujoco environments have a smaller state space dimension while larger batch size is used, making PER take up larger portion of computation cost. For the exponential sampling scheme, the extra computation is also close to negligible when using the approximate sampling method.
In terms of the proposed normalization scheme and the Inverting Gradients (IG) method, the nor- malization is very simple and be easily implemented and has negligible computation overhead. IG has a simple idea, but its implementation is slightly more complicated than the normalization scheme. When implemented naively, IG can have a large computation overhead, but it can be largely avoided by making sure the gradient computation is still done in batch-manner We have made very efficient implementation and our code is publicly available so that interested reader can easily reproduce it.
H ADDITIONAL EXPERIMENTAL RESULTS
H.1 INVERTING GRADIENTS WITH ERE
In Figure we show additional results on applying ERE to SOP+IG. The result shows that after applying the ERE scheme, SOP and IG both get performance boost. The performance of the SOP+ERE and IG+ERE are similar.
H.2 TD3 VERSUS TD3+
In figure we show additional results comparing TD3 with TD3 plus our normalization scheme, which we refer as TD3+. The results show that after applying normalization scheme. TD3+ has a significant performance boost in Humanoid, while in other environments, both algorithms achieve similar performance.
18

---

Under review as a conference paper at ICLR 2020
3000
4000
2000
6000
50
stacaction
Hopper-v2
(b) Walker2d-v2
(c) Halfcheetah-v2
7003
4000
2000
6.03
2000
SCP
507 CACRE
100
60
05
1.
1.
30
00
05
1.
1.5
20
26
30
(d) Ant-v2
(e) Humanoid-v2
Figure 8: SOP and inverting gradients with ERE sampling scheme
300
10000
4000
2000
2000
000
4000
103+
TEs+
100
26
08
10
02
044
02
(a) Hopper-v2
(b) Walker2d-v2
(c) HalfCheetahhv2
rstum
2000
0000000
TO
100
100+
100
10
28
Interactions
(d) Ant-v2
(e) Humanoid-v2
Figure TD3 versus TD3+ (TD3 plus the normalization scheme)
19

---

Under review as a conference paper at ICLR 2020
I ADDITIONAL ANALYSIS AND RESULTS COMPARING SAC WITH AND WITHOUT ENTROPY
To understand why entropy maximization is important for environment but less for another, we examine the actions selected when training SAC with and without entropy. Humanoid and Walker2d have action dimensions K 17 and K 6, respectively. In addition to the repre- sentative results shown for one dimension for both environments in Section 3.2 the results for all the dimensions are provided here in Figures and 11
From Figure 10 we see that for Humanoid using SAC (which uses entropy maximization), the |px| values are small and fluctuate significantly for all 17 dimensions. On the other hand, for SAC without entropy the |ux| values are typically huge, again for all 17 dimensions. This causes the actions ag be persistently clustered either M or or-M. As for both algorithms for all 6 dimensions, shown in figure I1
Walker, the |pg] values are sensible
20

---

Under review as a conference paper at ICLR 2020
mode output pix
model output
1o1
action
-10
-301
-u0 -101
-10"
-101
Sy Be / /o M
20. yo Mh Galt
200 La Bee 60 time steps
time steps
time
time steps
(a) ==
to1.
mode cutput pi:
model output
1o1
action
1n1 -200
-301
-301
withot Intropp
-20"
-30"
une 200 Bas Main Plly Plgaa timm
1y MMgg Ma time steps
pal Ba (200000
20 Maa 2Bn time steps
steps
(b) k=2
model output ptx
model output
action
-107
-307
20 Bya Maga Mlya Plaanno time steps
/ag Vlga Aliga Maa time steps
Ba ( steps
200 Mgr "Ba time
(c) k=
entropp
model output st:
model output
action
-10" -tol
20 Byn Maga Plan Poa steps
23. San Viga diaa Ma time steps
1oM Boate 1300.000
20 Se Ben
time
(d) k=
model output pte
100
model output Atu
101
-107
action
100 _
Pgg "Baa Segeeeeeg Pa steps
Uo. Son V'aa d'g doa time steps
100M Baatt 1300.00
700 Pe Be time
steps
(e) k=
200
model output 53x
model output
101.
action
100.
-101
-307
20 Bgg Mlia Plyy Mog time steps
23g Myy. Miga Mig M00 time steps
200 Maa "Bn
steps
time steps
(f k=
model cutpat st:
model output,
oction
-107
20 Bgg Miga Al Piga steps
299 20 A50 time
time steps
(g) k=
Figure 10: Humanoid-v2: aa and az values from SAC and SAC without entropy maximization
21

---

Under review as a conference paper at ICLR 2020
mode output pt:
model output
action Du
to1 10
action
-101
-z0
withot entrops
-1o
101
or 7/0 By on Mlly Migan time steps
3y Ba Ba
200 Ben BBe
time
time steps
time steps
(h) k= 8
mode output pi:
mode cutput
100
action
-10
EAC
-101
-10
pl Bag MM Mlig Ma Mog time steps
1p9 Mlgga Man time steps
290 Mga 2en time steps
time steps
(i) k=
161
model outpptt pt:
101 tot
model output
action
-10 -107
-300
una 20 Byg Mllg Allyn Plaaano steps
wtw. /o Alga Algga Maa time steps
palhere Roa 1500,000
200 Mer "Ba time
steps
k= 10
model output
model output
action
100
-301
20
-300
100
20 Byn Maga May Piaaa steps
Sp Syn Viga di Mo time steps
1300.00
20 2e Bes
time
(k) k=)=1
100
101
model outtput
model output
1n1
action
-1o1
-101
200
300
-100
-00
200 Bgg Miaa Magaeeee Pag steps
Up. Soa. V'aa d'g doa.0 time steps
100M Maate 2000000
500 Bor BBs time
steps
()) k= 12
model output 53>
model output
action
-708
f0 Bga Mia Plyy Plga time
22.. Myy Viga Aiga M0 time steps
y0 Mag MBn cam time steps
(m) k = 13
107.
C6C thout estrary
madel output Ai
model cutputt
1n).
oction
-1ol
-267
-307
-100
-301
20 Pgga Mfga Ma Pog time steps
Cggo MByn -ilga A'gga Ml time steps
290 20 M5e time
steps
(n) k 14
Figure 10: Humanoid-v2: aa and az values from SAC and SAC without entropy maximization
22

---

Under review as a conference paper at ICLR 2020
model output Jlr
mode: cutput
action 3u
-16.
thet ertrayy
-101
-101
20 Byg Alla Ally Plaa steps
Bn 1500.000
200 BBart
time steps
time
(0) = 15
model output pi:
model output pt
101
action
-101
200 Ban Maga Mla Pia steps
2o Myn Viga dlaa Mo time steps
1300.000
20 Se Be
time
(p) k 16
model outputt pi:
model output pa
oction
-1o1
-1o'
-30F
-101
20 Bgn
MC Pip steps
Co. Son. V'aaa d'ga doo time steps
yo8M 130.00000000
200 Phe Bos
(q) k=)== 17
Figure 10: Humanoid-v2: aa and az. values from SAC and SAC without entropy maximization
23

---

Under review as a conference paper at ICLR 2020
model output plr
moddel output
action D2
-325
-20n
-0.0
-101
300. Ba Baa L200000
300 2a 1,00
time steps
time steps
time steps
time steps
(a) 1=
model output ptc
mobtel output
actio
107
-30%
-100
-050
300
on 8 oi Mo
-0.5
300
100 Baa Ma Bo 1,000,000 time steps
time steps
time steps
time steps
(b) 2
100
2.00
mode output pt:
model outgut
action
-307
une 20 Bag M /2lg Mo
200 ga Peal 2,0 time steps
tirme
time
steps
k=)
1.00
mode output pt:
101
model output
100
action
-000
24G BETTROY
-0.0
0.75
20 Myg Mlt Mlyn Plyaant time steps
Ba Baaie Bar (500,000
ge Ma time steps
time
(d)k=4
220
model outpat pte
301
model output
209
oction
thot
-301
-101
20 Bag Miga Mlya Piaa
(300.000
g time steps
steps
time
steps
k=)
model output, pia
10/.
1.00
madel outputt Dt:
107-
oction
-309
-00
1300.00
St Po time steps
steps
k=
Figure 11: Walker2d-v2: μa and 0g values from SAC and SAC without entropy maximization
24


Reviews:
[{"review_id": "HyeO-W-0YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of this paper is a normalization scheme to avoid saturating the squashing function typically used to constrain actions within a bounded range in continuous control problems. It is argued that algorithms like DDPG and TD3 suffer from such saturation, which prevents proper exploration during training, while maximum entropy algorithms like Soft Actor-Critic (SAC) avoid it thanks to their entropy bonus. The main reason behind the success of SAC would thus be its ability to keep exploring throughout training, by avoiding saturation. A second contribution is a new experience replay sampling scheme, named Emphasizing Recent Experience (ERE), based on the idea that most recently added transitions should be given higher weights when sampling mini-batches from the replay bufffer. Combining both ideas leads to the SOP (Streamlined Off-Policy)+ERE algorithm, which is shown to consistently outperform SAC on Mujoco tasks.\n\nAlthough this paper presents interesting insights and very good empirical results, I am currently leaning towards rejection mostly due to missing some important empirical comparisons, which hopefully can be added in a revised version.\n\nThe first key missing comparison (IMO) is to the Inverting Gradients approach from Hausknecht & Stone (2016), which the authors know about since it is cited in the related work section. Note that in that paper, the problem of saturating squashing functions preventing proper exploration was already mentioned, although not investigated in as much depth as in this submission («(…) squashing functions quickly became saturated. The resulting agents take the same discrete action with the same maximum/minimum parameters each timestep »). Their proposed Inverting Gradients technique was found to work significantly better than squashing functions, which is why I believe it should be an obvious baseline to compare to.\n\nThe other important experiments which I think need to be added are simply to implement the proposed normalization scheme within DDPG & TD3 to demonstrate its usefulness as a standalone improvement over existing algorithms. This would strengthen the claim that « algorithms such as DDPG and TD3 based on the standard objective with additive noise exploration can be greatly impaired by squashing exploration ». Without this comparison on the same benchmark, it is difficult to fully grasp the impact of this normalization.\n\nFinally, regarding the ERE sampling scheme, I would appreciate to see SAC+ERE as well, to (hopefully) show that it can benefit SAC too (since this second contribution is orthogonal to the SOP algorithm).\n\nMinor points:\n•\tI would tone down a bit the claims for « the need to revisit the benefits of entropy maximization in DRL », since better exploration has always been put forward as a major benefit (« the maximum entropy formulation provides a substantial improvement in exploration and robustness », as written in « Soft Actor-Critic Algorithms and Applications »). To me, what this submission shows is essentially that naive implementation of additive noise exploration in e.g. DDPG is very bad for exploration, more than uncovering some novel properties of SAC.\n•\tBelow eq. 1: « the optimal policy is deterministic » => should be replaced with « there exists an optimal policy that is deterministic »\n•\t« principle contribution » => principal\n•\tThe normalization scheme does not appear in Alg. 1\n•\tIn Alg. 1 there are a Q_phi,i and a Q_phi,1 that should probably be Q_phi_i and Q_phi_1\n•\tThe results from section E in the Appendix should be mentioned in the main text\n•\tIn Fig. 4f the y axis’ label is a bit clipped\n\nUpdate based on new revision: thank you for adding more results. From what I can see, it is difficult to conclude on the benefits of SOP over IG, which I find really problematic. It seems to me that the most impactful result is related to the improvements brought by the ERE sampling scheme, which could probably be worth a paper on its own (by showing its benefits over a wider range of algorithms, e.g. TD3 & DQN+variants), but this would be a different paper. As a result I am sticking to \"Weak Reject\".", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1x1YN36Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\nThe paper identifies a problem with TD3 related to action clipping. The authors notice that SAC alleviates this problem by means of entropy regularization. Given the insight that action clipping is crucial, the authors propose an alternative approach to avoid action clipping in TD3, which is empirically shown to yield the same results as SAC. Surprisingly, with this improvement, even several parts from TD3 can be removed, such as delayed policy updates and the target policy parameters. In addition, a straightforward-to-implement experience replay scheme is proposed, that emphasizes recent experiences, which propels the proposed algorithm to achieve state-of-the-art results on MuJoCo.\n\n# Decision\nThis is a great paper: accept. The proposed Streamlined Off Policy (SOP) algorithm is thoroughly evaluated, ablation studies performed, code made available. Nevertheless, there are a few suggestions below that may further improve the paper.\n\n# Suggestions\n1) It is said that entropy regularization leads to action not being saturated in SAC. I feel that this causal relation is very indirect. Maybe SAC with entropy just discovers better policies that do not go crazy between extremes? For example, if you would leave the entropy term but remove tanh saturation from SAC, don't you think you would also get a bang-bang policy? Adding such an ablation study could further strengthen the argument that entropy leads to no constraint violation, if it turns out true.\n\n2) The Emphasizing Recent Experience (ERE) replay scheme seems reminiscent of sampling according to a distribution exponentially decaying into the past. It is said that physically shrinking the allowed sampling range by dropping old experiences is better because then very old experiences cannot be used at all. It would be interesting to see a comparison to sampling according to exponential distribution from the replay queue.\n\n# AFTER REBUTTAL\nTaking into account the concerns of other reviewers and the newly added evaluations, I lower my score to weak accept. Since now it seems that ERE is quite crucial, the argument of SPO outperforming SAC becomes weaker. Therefore, the authors should tone down the claims of outperforming SAC. Nevertheless, I still find the contribution of the paper valuable and think that it should be accepted, albeit with the aforementioned modifications in the camera-ready version.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gpjwjOOS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors investigate the role of entropy maximization in SAC and show that entropy regularization does not do what is usually thought: in the examples they investigate, where the output of the policy network needs to be squashed to fit in the action space domain, squashing would result in having only action at the boundaries, but entropy regularization maintains some intermediate values, hence exploration. From this insight, the authors replace entropy regularization by a simpler normalization process and show equivalent performance with their simpler Streamlined Off-Policy (SOP) algorithm. Then they introduce a second \"Emphasizing Recent Experience\" mechanism and show that SOP+ERE performs better than SAC.\n\nA good point for the paper is that the entropy regularization  study is very nice, more papers in the field should show similar detailed analyses of internal processes. But the paper suffers from a few serious weaknesses:\n\n- The TD3 mechanism goes beyond the Double Q-learning (or DDQN) mechanism of Van Hasselt et al: it takes the min over two critics. This should be explained properly.\n- the title, abstract and introduction insist more on SOP, but performance improvement seem to result more from ERE. If this is possible, studying the performance of SAC + ERE would disambiguate the relative contribution of both mechanisms.\n\nAbout gradient squashing issues, the authors main mention de gradient inverter idea from this paper:\n\n@article{hausknecht2015deep,\n  title={Deep reinforcement learning in parameterized action space},\n  author={Hausknecht, Matthew and Stone, Peter},\n  journal={arXiv preprint arXiv:1511.04143},\n  year={2015}\n}\n\nThe authors should also probably also cite (and read the latest arxiv version of):\n@inproceedings{ahmed2019understanding,\n  title={Understanding the impact of entropy on policy optimization},\n  author={Ahmed, Zafarali and Le Roux, Nicolas and Norouzi, Mohammad and Schuurmans, Dale},\n  booktitle={International Conference on Machine Learning},\n  pages={151--160},\n  year={2019}\n}\n\n\nMore local points:\n- \"without performing a careful hyper-parameter search\": so how did you choose these hyper-parameters? I see what you mean, but this is a very vague and slippery statement.\n- I do not find the 23*4 images in Appendix B much useful\n- Fig 3 seems to be repeated in Fig 4. Can't you just remove Fig 3?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the role of entropy in maximum entropy RL, particularly in soft actor-critic, and proposes an action normalization scheme that leads to a new algorithm, called Streamlined Off-Policy (SOP), that does not maximize entropy, but retains or exceeds the performance of SAC. Independently from SOP, the paper also introduces Emphasizing Recent Experience (ERE) that samples minibatches from the replay buffer by prioritizing the most recent samples. After rounds of discussion and a revised version with added experiments, the reviewers viewed ERE as the main contribution, while had doubts regarding the claimed benefits of SOP. However, the paper is currently structured around SOP, and the effectiveness of ERE, which can be applied to any off-policy algorithm, is not properly studied. Therefore, I recommend rejection, but encourage the authors to revisit the work with an emphasis on ERE.
Decision: Reject