Continuous Control With Ensemble Deep Deterministic Policy Gradients
Authors: Piotr Januszewski
Mateusz Olko
Michał Królikowski
Jakub Swiatkowski
Marcin Andrychowicz
Łukasz Kuciński
Piotr Miłoś
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
reinforcement learning
ensemble learning
MuJoCo
continuous control
stable performance
deterministic policy gradient

Abstract: The growth of deep reinforcement learning (RL) has brought multiple exciting tools and methods to the field. This rapid expansion makes it important to understand the interplay between individual elements of the RL toolbox. We approach this task from an empirical perspective by conducting a study in the continuous control setting. We present multiple insights of fundamental nature, including: a commonly used additive action noise is not required for effective exploration and can even hinder training; the performance of policies trained using existing methods varies significantly across training runs, epochs of training, and evaluation runs; the critics' initialization plays the major role in ensemble-based actor-critic exploration, while the training is mostly invariant to the actors' initialization; a strategy based on posterior sampling explores better than the approximated UCB combined with the weighted Bellman backup; the weighted Bellman backup alone cannot replace the clipped double Q-Learning. As a conclusion, we show how existing tools can be brought together in a novel way, giving rise to the Ensemble Deep Deterministic Policy Gradients (ED2) method, to yield state-of-the-art results on continuous control tasks from \mbox{OpenAI Gym MuJoCo}. From the practical side, ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing RL toolbox.

Under review as a conference paper at ICLR 2022
CONTINUOUS CONTROL WITH ENSEMBLE DEEP DE- TERMINISTIC POLICY GRADIENTS
Anonymous authors Paper under double-bling review
ABSTRACT
The growth of deep reinforcement learning (RL) has brought multiple exciting tools and methods to the field. This rapid expansion makes it important to under- stand the interplay between individual elements of the RL toolbox. We approach this task from an empirical perspective by conducting study in the continuous control setting. We present multiple insights of fundamental nature, including: a commonly used additive action noise is not required for effective exploration and can even hinder training; the performance of policies trained using existing methods varies significantly across training runs, epochs of training, and evaluation runs; the critics' initialization plays the major role in ensemble-based actor-critic exploration, while the training is mostly invariant to the actors' initialization; a strategy based on posterior sampling explores better than the approximated UCB combined with the weighted Bellman backup; the weighted Bellman backup alone cannot replace the clipped double Q-Learning. As a conclusion, we show how existing tools can be brought together in novel way, giving rise to the Ensemble Deep Deterministic Policy Gradients (ED2) method, to yield state-of-the continuous control tasks from OpenAl Gym MuJoCo. From the practical side, ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing RI. toolbox.
results
INTRODUCTION
Recently, deep reinforcement learning (RL) has achieved multiple breakthroughs in a range of challenging domains (e.g. Silver al. (201)); |erner et al. (2019); Andrychowicz et al. (2020b); Vinyals et al. (2019)) part of this success is related to an ever wver-growing toolbox of tricks and observed to boost the RL algorithms' performance (e.g. Hessel al. 2018); [Osbad et of affairs benefits the field but also brings challenges related to often unclear interactions between the individual improvements and the credit assignment related to the overall performance of the
methods that
Haarnoja et al. 20188); : Fujimoto et al. 2018); Wang et al. (2000))))
(2019)). This state
algorithm Andrychowicz al. 2020a) Ilyas al. (2020).
In this paper, we present comprehensive empirical study of multiple tools from the RL toolbox applied the continuous control in the OpenAl Gym MuJoCo setting. These are presented in Section and Appendix B Our insights include:
The normally distributed action noise, commonly used for exploration, hinders training.
The current iate-of-the-art methods are unstable under several stability criteria.
The critics' initialization plays a major role in ensemble-based actor-critic exploration, while the training is mostly invariant the actors' initialization.
The approximated posterior sampling exploration (Osband et al. 2013) outperforms approx- imated UCB exploration combined with weighted Bellman backup (Lee et al.) 2020).
The weighted Bellman backup (Lee et al. 2020) can not replace the clipped double Q-
Learning (Fujimoto et al. 2018).
To address some of the issues listed above, we introduce the Ensemble Deep Deterministic Policy Gradient (ED2) algorithm see Section 3 ED2 brings together existing RL tools novel way: is
!Our code is based on SpinningUp (Achiam 2018 We open-source it at: https: //gi thub. com/
ed2 122-paper/Hii

---

Under review as a conference paper at ICLR 2022
an off- off-policy algorithm for continuous control, which constructs ensemble of streamlined versions of TD3 agents and achieves the statc-of-the art performance in OpenAl Gym MuloCo, substantially improving the results the two hardest tasks Ant and Humanoid. Consequently, ED2 does not require knowledge outside of the existing RI toolbox, is conceptually straightforward, and easy to code.
BACKGROUND
We model the environment (S,A, R. P. 7, Po). where is a continuous multi-dimensional state space, A denotes a contin- uous multi-dimensional action space, P is transition kemnel, 7 [0, stands for discount factor, Po refers to an initial state distribution, and R is a reward function. The agent learns a policy from sequences of transitions
Markov Decision Process (MDP). It is defined by the tuple
[(st. de, Tas $141, d)};- called episodes trajectories, where az (-)))),, 81+1 P(-|8;, , rt R(s,, dt, 814-1) 1) is a terminal signal, and is the terminal time-step. A stochastic policy T(als) maps each state distribution over actions. A deterministic policy pr: S + A assigns each state an action.
All algorithms that consider in this paper use a different policy for collecting data (exploration) and a different policy for evaluation (exploitation). In order to keep track of the progress, the evaluation runs are performed every ten thousand environment interactions. Because of the en- vironments' stochasticity, we run the evaluation policy multiple times. Let {R; be a set of (undiscounted) returns from evaluation episodes {Ti) i.e. Rg = ==EEE, tr. We evaluate the policy using the average test return = Ei=1
R2 and the standard deviation of the test returns
V - E
R22.
We run experiments on four continuous control tasks and their variants, introduced in the appropriate sections, from the OpenAl Gym MuJoCo suite (Brockman et al. (2016) presented in Figure The agent observes vectors that describe the kinematic properties of the robot and its actions specify torques to be applied on the robot joints. See Appendix for the details on the experimental setup.
Ant
Hopper
Walker2d
Humanoid
Figure 1: The OpenAl Gym MuJoCo tasks benchmark method on.
3 ENSEMBLE DEEP DETERMINISTIC POLICY GRADIENTS
For completeness of exposition, we present ED2 before the experimental section. The ED2 ar-
chitecture is based an ensemble of Streamlined Off- Off-Policy (SOP) agents (Wang et al.) [0220).
meaning that our agent is an ensemble of TD3-like agents (Fujimoto et al. 2018) with the action normalization and the ERE replay buffer. The pseudo-code listing can be found in Algorithm [ while the implementation details, including more verbose version of pseudo-code (Algorithm) 3 can be found Appendix E In the data collection phase (Lines 9 ED2 selects one actor from the ensemble uniformly at random (Lines and (9) and run its deterministic policy for the course of one episode (Line 4b. In the evaluation phase (not shown in Algorithm TT. the evaluation policy averages all the actors' output actions. We train the ensemble every 50 environment steps with 50 stochastic gradient descent updates (Lines 10 ED2 concurrently learns K 2 Q-functions, Q and Q where & E K, by mean square Bellman error minimization. in almost the same way that SOP learns its two Q-functions. The only difference is that we have K critic pairs that are initialized with different random weights and then trained independently with the same batches of data. Because of the different initial weights, each O-function has different bias its Q-values. The K actors, Te, train maximizing their corresponding first critic, Q just like SOP.
2

---

Under review as a conference paper at ICLR 2022
Algorithm ED2 Ensemble Deep Deterministic Policy Gradients
Input: init. params for policy 0₂ and Q-functions ØR.1, 9k,2, & E [1. replay buffer D;
1: Sample the current policy index [[)...)).
2: Reset the environment and observe the state s.
3: repeat
4: Execute action a = -oo
b f uses the action normalization
5: Observe and store (8, (,, s', ,d) in the replay buffer D.
6: Set 8 f- s'
7:
if' episode is finished then
8:
Reset the environment and observe initial state
9:
Sample the current policy index c ~ [][
10:
if time to update then for many as steps done in the environment do Sample batch of transitions B = {(s, a, s', d)} C
11:
12:
uses ERE
Update the parameters 02, Φk,1 and Φ by one gradient step.
14:
until convergence
Utilizing the ensembles requires several design choices, which we summarize below. The ablation study of ED2 elements is provided in Appendix C
Ensemble
Used: We train the ensemble of 5 actors and 5 critics; each actor learns from its critic and the whole ensemble is trained on the same data.
Not used: We considered different actor-critic configurations, initialization schemes and relations, as well as the use of random prior networks (Osband et al. 2018), data bootstrap (Osband et al. 2016), and different ensemble sizes. We also change the SOP network sizes and training intensity instead of using the ensemble. Besides the prior networks some special cases, these turn out to be inferior as shown in Section 4 and Appendix B.T
Exploration
Used: We pick actor uniformly at random to collect the data for the course of one episode. The actor is deterministic (no additive action noise is applied). These two choices ensure coherent and temporally-extended exploration similarly to Osband al. (2016).
Not used: We tested several approaches to exploration: using the ensemble of actors, UCB (Lee al. 2020), and adding the action noise in different proportions. These experiments presented in Appendix B.2
Exploitation
Used: The evaluation policy averages all the actors' output actions to provide stable performance.
Not used: We tried picking action with the biggest value estimate (average of the critics' Q- functions) evaluation (Huang et al.) (017).
Interestingly, both policies had similar results, Appendix B.3
Action normalization
Used: We use the action normalization introduced by Wang al. (2020).
Not used: We experimented with the observations and rewards normalization, which turned out to be unnecessary. The experiments are presented in Appendix B.4
Q-function updates
Used: We do 50 SGD updates (ADAM optimizer (Kingma and Ba 2015). MSE loss) to the actors and the critics every 50 environment interactions, use Clipped Double Q- L-Learning (Fujimoto et al. 2018).
3

---

Under review as a conference paper at ICLR 2022
Not used: We also examined doing the updates at the end of each episode (with the proportional number of updates), using the Hubert loss, and doing weighted Bellman backups (Lee et al. 2020). However, we found them to bring no improvement to our method, presented in Appendix B.5
4 EXPERIMENTS
In this section, we present our comprehensive study and the resulting insights. The rest of the experiments verifying that design choices perform better than alternatives in Appendix B Unless stated otherwise, solid line in the figures represents an average, while a shaded region shows 95% bootstrap confidence interval. We used 30 seeds for ED2 and the basclines and seeds for the ED2 variants.
4.1 THE NORMALLY DISTRIBUTED ACTION NOISE, COMMONLY USED FOR EXPLORATION, HINDERS TRAINING
In this experiment, we deprive SOP of its exploration mechanism, namely additive normal action noise, and call this variant deterministic SOP (det. SOP). causes relatively minor deterioration in the Humanoid performance, has no significant influence on the Hopper or Walker performance, and substantially improves the Ant performance, see Figure 2 This result shows that no additional exploration mechanism, often in form of an exploration noise (Lillicrap et al. 2016 Fujimoto et al. 2018
Wang et al. 2020), is required for the diverse data collection and can even hinder training.
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
Averagg best roturn
3000
8000
3000
0000
2000
4030
4200
4000
.000
1000
2000
5900
ytar
pror
1000
pritoor
proor
Total samples
Total semples
146 Total samoles
Total samples
Figure 2:
The average test returns across the training of SOP and SOP without the exploration noise in two without and with the prior network. All metrics were computed over 30 seeds.
variants:
ED2 leverages this insight and constructs an ensemble of deterministic SOP agents presented in Section 3 Figure 3 shows that ED2 magnifies the beneficial effect coming from the deterministic exploration.
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
Averge best poturn
0000
2000
4000
Nethod ED2
Wethod ED2
Mathed
1000
(oun)
2000
SDP
Total samples
Total samples
166
Total samples
Total samples
Figure 3:
The average test returns across the training ED2, SOP and SOP without the exploration noise and without the prior network. All metrics computed over 30 seeds.
ED2 achieves state-of-the-ar performance on the OpenAI Gym MuloCo suite. Figure 4 shows the results of ED2 contrasted with three strong baselines: SUNRISE (Lee et al. 2020), SOP Waag et al. 2020), and SAC(Haamoja et al. 2018b).
For completeness, we plot the Humanoid velocities in Figure which shows that our method accelerates much higher velocity than the baselines.
4

---

Under review as a conference paper at ICLR 2022
Hopper-v2
Walker2d-12
Ant-v2
Humanoid-v2
Aestagg last ratur
1000
5000
2000
terhhd
4030
Metttd
4c00
Verthd
Mathod
3000
2000
2000
5
Total samples
Total samples
188
Total samples
Total samples
Figure 4: The average test returns across the training of ED2 and the three baselines. The solid lines represent averages, whereas the shaded regions show 95% bootstrap confidence intervals, each being computed over 30 seeds.
Humanoid-v2
slgo
Velocity
1.5
ED2
SUNRIS
1.0
0.-
600
Time-step
Figure 5: The Humanoid velocity of ED2 and the three baselines. The solid line represents the average velocity and the shaded region 95% bootstrap confidence interval over runs the trained agents with the median performance across 30 seeds.
4.2 THE CURRENT STATE-OF-THIEAAR METHODS ARE UNSTABLE UNDER SEVERAL STABILITY CRITERIA
We consider three notions of stability: inference stability, asymptotic performance stability, and training stability. ED2 outperforms baselines in each of these notions, as discussed below. Similar metries were also studied in Chan et al. (2020).
Inference stability We say that agent is inference stable if, when run multiple times, it achieves similar test performance every time. We measure inference stability using the standard deviation of test returns explained in Section 2 We found that that the existing methods train policies that are surprisingly sensitive to the randomness in the environment initial condition Figure 4 and Figure show that ED2 successfully mitigates this problem. By the end of the training. ED2 produces results within 1 1% of the average performance on Humanoid, while the performance of SUNRISE, SOP, and SAC may vary as much as 11%
Hopper-v2
Walker22d-22
Anl-v2
Humanod-v2
Std dev of tast retum
Nerhod
Mathad
1500
Nethod
200
En2 (sum)
E02 (um))
ED2 (141))
<00
SUNE
SUNnisE
SUN
sam
sor
SAC
M
430
SUNRISE
200
250
BNC
200
samples
100
samples
samples
100
samples
100
Figure 6: The standard deviation of test returns across training (lower is better), see Section
Thh MuloCo suite is overall deterministie, nevertheless, little stochasticity is injected at the heginning of each trajectory, see Appendix for details.
5

---

Under review as a conference paper at ICLR 2022
Asymptotic performance stability We say that an agent achieves asymptotic performance stability achieves similar test performance across multiple training runs starting from different initial networks weights. Figure 7 shows that ED2 has significantly smaller variance than the other methods while maintaining high performance.
Hopper-v2
Walker2d-V v2
Art.v v2
Humanoid. v2
Avorage tost return
4000
8000
0000
2000
4000
8000
2500
2000
2000
1500
Figure 7: The dots are the average test returns after training (3M samples) of each seed. The distance between each box's top and bottom edges is the interquartile range (IQR). The whiskers spread across all values.
Training stability We will consider training stable performance does not severely deteriorate from one evaluation to the next. We define the root mean squared deterioration metric (RMSD) as follows:
M
max R,, - R, R,,0))
RMSD
M
where M is the number of the evaluation phases during training and R; is the average test return at the i-th evaluation phase (described in Section 2}. We compare returns 20 evaluation phases apart to
ensure that the deterioration in performance doesn't t stem from the evaluation variance. ED2 has the lowest RMSD across all tasks, see Figure
Rod maznn squa deterioratio
Hoppar-42
Whiker2d-42
Ant-v2
Hu Humannid-422
1900
1000
1350
1500
1000
1002
400
SDP
SIC
E00 NRSS
3dP
s/c
(oun) BUIRISE
5/C
ED2 (uuss
SDF
Figure 8: RMSD, the average and the 95% bootstrap confidence interval 30 seeds.
4.3 THE CRITICS' INITIALIZATION PLAYS A MAJOR ROLE IN ENSEMBLE-BASED ACTOR-CRITIC EXPLORATION, WHILE THE TRAINING MOSTLY INVARIANT THE ACTORS' INITIALIZATION
In this experiment, actors' weights are initialized with the same random values (contrary to the standard case of different initialization). Moreover, we test a corresponding case with critics' weights initialized with the same random values or simply training only single critic.
Figure indicates that the choice of actors initialization does not matter in all tasks but Humanoid. Although the average performance on Humanoid seems to be better, it also less stable. This is quite interesting because the actors are deterministic. Therefore, the exploration must come from the fact that each actor is trained to optimize his own critic.
On the other, Figure 9 shows that the setup with the single critic severely impedes the agent perfor- mance. We suspect that using the single critic impairs the agent exploration capabilities as its actors' policies, trained to maximize the same critic's 's Q-function, become very similar.

---

Under review as a conference paper at ICLR 2022
Hopper-v2
Walker2d-12
Ant-v2
Humanoid-v2
0000
Aasragg last ratum
3000
0000
2000
(oans)
Wethod
2002
Netthd
Jears adore est)
Vethod
2000
2000
E02 (9ngle amto
2000
EC2
Enoyysnge
5
Total samples
Total samples
148
Total samples
Total samples
Figure 9: The average test returns across the training of ED2, ED2 with actors initialized to the same random values, and ED2 with the single critic.
4.4 THE APPROXIMATED POSTERIOR SAMPLING EXPLORATION OUTPERFORMS APPROXIMATED UCB EXPLORATION COMBINED WITH WEIGHTED BELLMAN BACKUP
ED2 uses posterior sampling based exploration method (Osband et al. 2016). SUNRISE, on the other hand, approximates the Upper Confidence Bound (UCB) exploration technique and does weighted Bellman backups (Lee et al. 2020). For the fair comparison between ED2 and SUNRISE, substitute the SUNRISE base algorithm SAC for the SOP algorithm used by ED2. call this variant SUNRISE-SOP.
We test both methods on the standard MuJoCo benchmarks as well as delayed (Zheng et al. 201 20) and sparse (Plappert et al. 2018) rewards variants. Both variations make the environments harder from the exploration standpoint. In the delayed version, the rewards are accumulated and returned to the agent only every 10 time-steps. In the sparse version, the reward for the forward motion returned to the agent only after it crosses the threshold of one unit on the x-axis. For a better perspective, a fully trained Humanoid is able to move to around five units until the end of the episode. All the other reward components (living reward, control cost, and contact cost) remain unchanged. The results are presented in Table
Improvement over
Improvement over SUNRISE-SOP
Environment
SUNRISE-SOP
SOP
ED2
SOP
H2
3350
3083
3512
+5%
+14%
Wa2ker-v2
5458
5730
6222
+14%
+9%
Ant-v2
5507
771
6862
+25%
790%
Humanoid-v2
6456
5738
7307
+13%
+27%
DelayedHopper-v2
2990
2708
3315
+11%
+22%
DelayedWalker-v2
4907
4505
5940
+21%
+32%
DelayedAnt-v2
4216
1486
3611
- 11%
+43%
DelayedHumannid-v2
770
1129
3823
397%
239%
SparseHopper-v2
2798
2644
3397
+21%
+29%
SparseWalker-v2
4830
4664
5936
+23%
+27%
SparseAnt-v2
5573
3625
5466
-2%
+51%
SparseHuumanodd-v2
5433
5225
7438
+37%
+-42%
Table 1: The test returns after training (3M samples) median across 30 seeds for the standard MuJoCo and seeds for the delayed/sparse variants.
ED2 outperforms the non-ensemble method SOP, supporting the argument of coherent and temporally- extended exploration ED2. Moreover, we observe that performance in MuJoCo environments benefits from the ED2 approximate Bayesian posterior sampling exploration (Osband et al.) (013) in
contrast to the approximated UCB in SUNRISE, which follows the OFU principle. The posterior sampling is proved to be theoretically superior the OFU strategy (Osband and Van Roy 2017)
7

---

Under review as a conference paper at ICLR 2022
The experiment where the ED2's exploration mechanism is replaced for UCB is in Appendix [B.2 This variant also achieves worse results than ED2. The additional exploration efficiency experiment in the custom Humanoid environment, where agent has to find and reach goal position, is in Appendix A
4.5 THE WEIGHTED BELLMAN BACKUP CAN NOT REPLACE THE CLIPPED DOUBLE Q-LEARNING
We applied the weighted Bellman backups proposed by Lee et al. (020) to our method. It is suggested that the method mitigates error propagation in Q-learning by re-weighting the Bellman backup based on uncertainty estimates from an ensemble of target Q-functions (i.e. variance of predictions). Interestingly, Figure I0 does not show this positive effect on ED2.
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
Auamaga bost return
6030
6000
0090
0000
2000
4000
4000
1000
ED2 (sum)
Velliad
ED2 (oun)
2000
E02 (sun)
EDE (ouns)
Meiths
E02 wiight
502 (er wiight)
E02 weigh
ED0 (ar (arwaghh)
Total semples
Total semples
Total samoles
Total samples
Figure 10: The average test returns across the training of method and ED2 with the weighted Bellman backup.
Our method uses clipped double Q-Leamnnn mitigate overestimation in Q-functions (Fujimoto et al. 2018). We wanted to check if it is required and if it can be exchanged for the weighted Bellman backups used by Lee al. (2020). Figure I shows that clipped double Q-Learning is required and that the weighted Bellman backups can not replace it.
Hopper-v2 v2
Walker2d.vk
Ant.v2
Humanoid-vo
Averaga lest roturn
0030
1000
(surs)
5000
EG2 cwib
2002
2000
4000
EG2 owlb waggge
4000
(sart)
Method
3000
3000
del)
E02 (Wo dip)
1000
F/2P (pxzs)
2000
dip, weight
CD2 (wis dip whinint
(seght)
1000
5
Total samples
Total somples
148
Total samples
Total samples
Figure 11: The average test returns across the training of our method and ED2 without clipped double O-Learning in two variants without and with the weighted Bellman backups.
RELATED WORK
Off-policy RL Recently, multiple deep RL algorithms for continuous control have been proposed,
e.g. DDPG (Lillicrap et al. 2016). TD3 (Fujimoto et al. 2018). SAC (Haarnoja et al. (2118)), SOP (Wang et al. 2020). SUNRISE (Lee et al., 2020). They provide a variety of methods for improving training quality, including double-Q bias reduction van Hasselt et al. (2016), target policy smoothing or different update frequencies for actor and critic
al. (2018), entropy regularization
Haarnoja et al. (2018b), action normalization Wang et al. (2020), prioritized experience replay Wang et al. (2020). weighted Bellman backups Kumar et al. (2020) Lee et al. (2020), and use of ensembles Osband et al. (2019): Lee et al. (2020); Kurutach et al. (2018): Chua et al. (2018).
Ensembles Deep ensembles are practical approximation of a Bayesian posterior, offering im- proved accuracy and uncertainty estimation Lakshminrrayanan al. 2017); Fort et al. (2019). They
8

---

Under review as a conference paper at ICLR 2022
inspired variety of methods in deep RL. They are often used for eemporall--ettendee exploration; see the next paragraph. Other than that, ensembles of different TD-leaming algorithms were used to calculate better Q-learning targets (Chen et al. (018). Others proposed to combine the actions and value functions of different RI. algorithms Wiering and van Hasselt 2008) or the same algorithm with different nyper-parameters Huang et al. (2017). For mixing the ensemble components, complex self-adaptive confidence mechanisms were proposed in Zheng et al. (0118). Our method is simpler: uses the same algorithm with the same hyper-parameters without any complex learnt mixing mechanism. [ee et al. (2200) proposed unified framework for ensemble learning in deep RL (SUNRISE) which uses bootstrap with random initialization Osband al. (2016) similarly to our work. We achieve better results than SUNRISE and show in Appendix B that their UCB exploration and weighted Bellman backups do not aid our algorithm performance.
Exploration Various frameworks have been developed to balance exploration and exploitation in RL. The optimism the face of uncertainty principle and Robbins (1985); Bellemare et al. (2016) assigns overly optimistic value each state-action pair, usually the form of an exploration bonus reward, to promote visiting unseen areas of the environment. The maximum entropy method Haarnoja al. (20T8a) encourages the policy be stochastic, hence boosting exploration. In the parameter space approach Plappert et al. 2018): Fortunato et al. (2018). noise is added to the network weights, which can lead to camporall--etennae
exploration and richer set of behaviours. Posterior sampling SStrens (2000); Osband et al. (2016) 2018) methods have similar motivations. They stem from the Bayesian perspective and rely on selecting the maximizing action among sampled and statistically plausible set of action values. The ensemble approach Lowrey et al. 22; Milos et al. (2019): Lee et al. (2020) trains multiple versions the agent, which yields a diverse set behaviours and can be viewed an instance of posterior sampling RL.
CONCLUSIONS
We conduct comprehensive empirical amalysis of multiple tools from the RL toolbox applied the continuous control in the OpenAl Gym MuJoCo setting. We believe that the findings can be useful to RL researchers. Additionally, we propose Ensemble Deep Deterministic Policy Gradients (ED2), an ensemble-base off-policy RL algorithm, which achieves state-of-the-ar performance and addresses several issues found during the aforementioned study.
REPRODUCIBILITY STATEMENT
We have made significant effort make results reproducible. We 30 random seeds, which is above the currently popular choice in the field (up to 5 seeds). Furthermore, we systematically explain our design choices in Section and we provide detailed pseudo-code of our method in Algorithm 3 in the Appendix B Additionally, we open-sourced the code for the project together with examples of how reproduce the main experiments. The implementation details are explained in Appendix E and extensive information about the experimental setup is given in Appendix D
httpp:///iithuu.cc///ee popp:///////ddppppp////

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. GitHub repository, 2018.
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What Matters in On-Policy Reinforcement Learning? A Large-Scale Empirical Study, 2020a. ISSN 23318422.
OpenAl: Marcin Andrychowicz. Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in- in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020b.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, edi- tors, Advances in Neural Information Processing Systems 29; Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages wttpss/ppppppppp///ee/eoppppppps Awt.tt--bstttttta html
1471-1479, 2016. URL https:t//prcceeeinnnnoto// .ew..e.rriiin.................
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0666 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Stephanic C. Y. Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the Reliability of Reinforcement Learning Algorithms. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Xi Liang Chen, Lei Cao, Chen Xi Li, Zhi Xiong Xu, and Jun Lai. Ensemble Network Architecture for Deep Reinforcement Learning. Mathematical Problems in Engineering, 2018. ISSN 15635147. doi: 10.1 15/1151//////29999..
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine Deep reinforcement learning in handful of trials using probabilistic dynamics models. In NeurIPS 2018, pages 4779, 2018.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. CoRR, abs/1912.02757,, 2019. httt:://awww...rg/////
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In 6th International Conference on Learning Representations, ICLR 2018 Conference Track Proceedings, 2018
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1592-1591. PMLR, 2018. URL
mll ptw.s/ppess/vv/vm wtwt:///mmmmmmmma//mmmaamooo .tm.mml
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with stochastic actor. In 35th International Conference on Machine Learning, ICMI. 2018, 2018a. ISBN 97815108
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05990 2018b. URL sttt:://axat orq/ abs abaaaaas1100 /8812 05905
10

---

Under review as a conference paper at ICLR 2022
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018. pages 3215-3222, oct 2018. ISBN 9781577358008. URL 02298
Zhewei Huang, Shuchang Zhou, Bo Er Zhuang, and Xinyu Zhou. Learning to run with actor-critic ensemble, 2017. ISSN 23318422.
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras. Firdaus Janoos, Larry Rudolph, and Aleksander Madry, closer look at deep policy gradients. In 8th International Conference on Learning Representations, JCLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. /
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL bttt::/aarxis wwg.orgg////ooooooooo als./111.... 6980
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Informattion Processing Systems 2020, NeurIPS 2020,
December 6-12, 2020, virtual, 2020. URL http:/tttps//prroceedegg///e/e/s neurips.eeeppp.....m httpp//cccpppr///////p 2020/hashi G
httphh/a7444cccccc 42444626666668 66b 7ee66677622330 2ee3777e35555525 -bb.ecc2cccc.aaaaatttt........ html
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS 2017, 2017.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning, 2020. ISSN 2331 23318422.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Truck Proceedings, 2016. URL httt:://arxise org/ /a00s.159..000
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model- control. CoRR, abs/1811.0184g, 2018
Piotr Mitoš, Lukasz Kuciński, Konrad Czechowski, Piotr Kozakowski, and Macick Klimek. Uncertainty-sensiiivee learning and planning with ensembles. arXiv preprint 191:v:191..09996. 2019.
Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In 34th International Conference on Machine Learning, ICML 2017, 2017. ISBN 9781510855144.
Ian Osband, Benjamin Van Roy, and Daniel Russo. (More) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, 2013.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via
bootstrapped DQN. In Advances in Neural Information Processing Systems, 2016.

---

Under review as a conference paper at ICLR 2022
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In NeurlPS, 2018.
Ian Osband, Benjamin Van Roy, Daniel J. Russo, and Zheng Wen. Deep exploration via randomized value functiinn. J. Mach. Learn. Res., 20:124:1-124:6,, 2019, URL http://imlr.on org
papers/v20/18
html]
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In 6th International Conference Learning Representations, ICLR 2018 Conference Track Proceedings, 2018.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4444889 2016.
Malcolm J. A. Strens. A bayesian framework for reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 July 2, 2000, pages 943-950, 2000.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q' learning. In Dale Schuurmans and Michael P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2094 2100. AAAI Press, 2016. URL httt:://wwx aaai. .
3AAT16/japer///rre///1233
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi- agent reinforcement learning. Nature, 575(7782):350 354, 2019.
Che Wang, Yanqiu Wu, Quan Vuong, and Keith Ross. Striving for simplicity and performance in off-policy DRL: Output normalization and non-uniform sampling. Proceedings of the 37th International Conference on Machine Learning, 119:10070-10080 13-18 Jul 2020. http
//proceedings mlr www......esss......m...o press http:/vv11//w//1
html
Marco A. Wiering and Hado van Hasselt. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 2008. ISSN 10834419, doi: 10.1109TTSMCB. 2008.92023..
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On Learning Intrinsic Rewards for Policy Gradient Methods. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6649-4659 2018a
Zhuobin Zheng, Chun Yuan, Zhihui Lin, Yangyang Cheng, and Hanghao Wu. Self-adaptive double bootstrapped DDPG, In IJCAI International Joint Conference on Artificial Intelligence, 2018b. ISBN 9780999241127, doi: 10.24963/jcai.20118444
12

---

Under review as a conference paper at ICLR 2022
A EXPLORATION EFFICIENCY IN THE CUSTOM HUMANOID ENVIRONMENT
To check the exploration capabilities of our method, constructed environments based Humanoid where the goal is not only to move forward as fast as possible but to find and get to the specific region. The environments are described in Figure 12
CC
03-
01-
20
02
30
comples
Total camplo
This is top-down view. Hu- manoid starts the origin (marked 'x'). reward in each time-step equal to the number of circles for which the agent is inside. Being in the most nested circle (or either of two) solves the task.
Figure 12:
Figure 13: The fraction of episodes in which the task is finished for ED2 and two baselines. The average and the 95% bootstrap confidence interval over seeds.
Because the Humanoid initial state slightly perturbed every rum, we compare solved rates over multiple runs, see details in Appendix D [ the solved rates of our method and the three baselines. Our method outperforms the baselines. For this experiment, our method uses the prior networks (Osband et al. 2018).
B DESIGN CHOICES
In this section, we summarize the empirical evaluation of various design choices grouped by topics related to an ensemble agents (B.I). exploration (B.2), exploitation (B.3). normalization (B.4). and Q-function updates (B.5). In the plots, solid line and shaded region represent average and a 95% bootstrap confidence interval over 30 seeds in case ED2 (ours) and 7 seeds otherwise. All of these experiments test ED2 presented in Section with Algorithm 2 used for evaluation (the ensemble critie variant). We call Algorithm 2 'vote policy'.
Algorithm 2 Vote policy
1: Input: ensemble size K; policy O% and Q-function Φx,1 parameters where & E [1...., KK; max action scale M;
2: function VOTE_ (OTE_POLICY(s.e)
ax M tanh (noy (s)) for l E (1,..., K]
(1)
3: if use arbitrary critic then
for & K] y = Qo. (s, (s,ax)
(2)
4: else use ensemble critic
Σ
(s) for h: E [l,..., K]
qk
K
[E[1...K]
5: return 0% for k = argmaxx Uk
13

---

Under review as a conference paper at ICLR 2022
B.1 ENSEMBLE
Prior networks We tested if our algorithm can benefit from prior networks (Osband et al. 2018 turned out that the results are very similar on OpenAl Gym MuJoCo tasks, see Figure [4 However, the prior networks are useful on our crafted hard-exploration Humanoid environments, see Figure 15
Hopper-v2
Walker2d-v2
And-v2
Humanoid-v2
Resrago lest raturs
0030
1000
1000
2000
4030
4200
3000
2000
1000
Vothod
2000
2000
Vethod
(M4)))
(sung
(pens) (pecr pecennteett)
getr
joros
1000
yoior netwerio)
Total samples
Total semples
145
Total samoles
Total samples
Figure 14: The average test returns across the training of ED2 without (ours) and with prior networks.
HurnanoddiieDDedisstooossss v
10-
00-
04-
0.4
02-
Larted tett prol
nateato
Netbou ecaions)
00
8.5
1.0
20
Total samples
Total samples
Figure 15: The average test returns across the training of ED2 without (ours) and with prior networks.
Ensemble size Figure 16 shows ED2 with different ensemble sizes. As can be seen, the ensemble of size (which we use in ED2) achieves good results, striking a balance between performance and computational overhead.
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
1000
Avaroga best poturn
X000
200
ED2 (422
Mittor
4030
E02 y0u0 1)
no2
Welhvad
4200
Miliiod
E52 1920
4000
Velhor
E12 paiw
ED2 (400
1000
2000
ED2
2000
2000
ED0 (d0t
EDC (sibo
508
Total semples
Total semples
Total samples
Total samples
Figure 16: The average test returns across the training of ED2 with a different number of actor-critics.
Data bootstrap Osband et al. (2016) and Lee et al. (020) remark that training an ensemble of agents using the same training data but with different initialization achieves, in most cases, better performance than applying different training samples to each agent. We confirm this observation in Figure 17
Data bootstrap assigned each transition each agent the ensemble with 50% probability.
14

---

Under review as a conference paper at ICLR 2022
Hopper-v2
Walker2d-12
Ant-v2
Humanoid-v2
Aostagg last ratur
0030
2000
5000
2000
4030
4000
Eit2 (ous))
Withos
2000
1000
F02 (dits bostttop))
2000
2002
garnt
cours]
boalehag)
1000
hooterap)
5
5
Total samples
Total samples
Total samples
Total samples
Figure 17: The average test returns across the training of ED2 without (ours) and with data bootstrap.
SOP bigger networks and training intensity We checked if simply training SOP with bigger networks with higher training intensity number of updates made for each collected transition) can get it close to the ED2 results. Figure 18 compares ED2 to SOP with different network sizes, while Figure 19 compares ED2 SOP with one or five updates per environment step. It turns out that bigger networks or higher training intensity does not improve SOP performance.
Hopper-v2
Walker2d-12
Ant-v2
Humanoid-42
Aeorgg testt poturn
6000
1000
5000
0000
0000
2000
ED2 (oun)
4030
E02 (punc)
4000
E02 (ourS)
Lettha
ED2 lours)
Metwwll
SK2 (2t0 2000
2000
552 204]
-000
(poet 2001]
500 (2306 220]
1000
SDP (577) 670)
SCP (5-00 645)
SCF (144) s40g
SCF (1571. t77)
(1706 5700
(100, 5e0g
SDP (5:45, m600)
(1177, 670)
ADP grea 7400
RDP (2748
(160)
(244, 7000
Total samples
Total semples
Total samples
Total samples
Figure 18: The average test returns across the training of ED2 and SOP with different network sizes.
Hoppar-v2
Walker2d-v2
Anl-v2
Humanoid-v2
8000
Average tast retum
0000
3000
5000
2000-
2000
4000
(oun)
2000
SGI"
(purs)
2007
552
E02
2000
purs)
19 xiitenbyy
1000
Intensty)
100
9GP (Sxintensh))
Total samplas
168
Total samples
166
Total samoles
14t
Total samples
166
Figure 19: The average test returns across the training of ED2 and SOP with one or five updates for every step in an environment.
B.2 EXPLORATION
Vote policy In this experiment, we used the so-called "vote policy" described Algorithm We use it for action selection in step 5f Algorithm 3 in two variations: (1) where the random critic, chosen for the duration of one episode, evaluates each actor's 'ss action or (2) with the full ensemble of
critics for actors actions evaluation. Figure shows that the arbitrary critic is not much different from our method. However, in the case of the ensemble critic, we observe a significant performance drop suggesting deficient exploration.
15

---

Under review as a conference paper at ICLR 2022
Hopper-v2
Walker2d-12
Ant-v22
Humanoid-v2
Fastagi last ratum
0030
1000
5000
4030
2000
tarhed
2000
Mertod
Methos
1000
2000
2000
EDC
pelkgg. ettiteey
pallyy, wchitery
(yole jollcy, arbilbery
poliyy aberney polliy,
polloyy
pollyy, onte)
(ust pallyy,
3 148
Total samples
Total samples
Total samples
Total samples
Figure 20: The average test returns across the training of ED2 with and without the vote policy for exploration.
UCB We tested the UCB exploration method from Lee et al. (2020). This method defines an upper-confidence bound (UCB) based on the mean and variance of Q-functions in an ensemble and selects actions with the highest UCB for efficient exploration. Figure 21 shows that the UCB exploration method makes the results of our algorithm worse.
Hopper-v2
Walker2d-v2 v2
Ant.v2
Humanoid .2
Auarage tost return
0000
1000
0000
2000
4000
4000
1000
FTP (p4n)]
Ketttod
2030
F02 parn
Nethod
FTP (xun)
2000
ETP (RP
Metoot
Ea2 (UCC repiaatioon
E:02 (qcloao)
E.0 (UE2 aeparsdann)
ESP2UCC wedtowton)
Total samples
Total samples
145
Total samples
Total samples
1ed
Figure 21: The average test returns across the training of our method and ED2 with the UCB exploration.
Gaussian noise While our method uses ensemble-based temporally coherent exploration, the most popular choice of exploration injecting i.i.d. noise (Fujimoto et al. 2018 Wang et al. 2020). We evaluate if these two approaches can be used together. We used Gaussian noise with the standard deviation of 0.29, it is the default value in Wang et al. (0000). We found that the effects are task- specific, barely visible for Hopper and Walker, positive in the Ant - see Figure [2] In a more refined experiment, we varied the noise level. With more noise the Humanoid results are better, whereas the And results are worse see Figure
of Humanoid, and negative for
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
8000
Avaroga best poturn
0030
5030
2002
2000
4200
2000
4000
1000
E02
Malteal
Unittdd
2000 EG2
1e texd
Melted
jours)
1000
K.0
Total semples
Total semples
Total samples
Total samples
Figure 22: The average test returns across the training ED2 with and without the additive Gaussian noise for exploration.
16

---

Under review as a conference paper at ICLR 2022
Hopper-v2
Walker2d-12
Ant-v2
Humanoid-v2
8000
Aastage last ratum
1000
5000
0000
2000
Mithod
4030
Mothod
4000
lethot
3000
J0000
2000
2002
2000
1002
5
5
Total samples
Total samples
148
Total samples
Total samples
Figure 23: The average test returns across the training of ED2 with and without the additive Gaussian noise for exploration. Different noise standard deviations.
B.3 EXPLOITATION
We used the vote policy, see Algorithm 2 as the evaluation policy in 2T of Algorithm 3 Figure 24 shows that the vote policy does worse on the OpenAl Gym MuJoCo tasks. However, custom Humanoid tasks introduced in Section improves our agent performance see Figure 25
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
Aeorgg bast paturn
0031
5000
0000
2000
1000
EGP
MMtiod
2000
E30
NMtted
Micbod joung perti
Rho
Fho
Total samples
Total semples
Total samples
Total samples
Figure 24: The average test returns across training our method and ED2 with the vote policy evaluation.
HimannooChhlDeeassttooo on-vol
02
E22 bunt
02
(pab)
E22
Total
14c
Total
Figure 25: The average test returns across the training of our method and ED2 with the vote policy for evaluation.
B.4 NORMALIZATION
We validated rewards or observations normalization (Andrychowicz et al. 2020a) help our method.
In both cases, keep the empirical mean and standard deviation of each reward/observation coordinate, based on all rewards/observations seen so far, and normalize rewards/observations. by subtracting the empirical mean and dividing by the standard deviation. It turned out that only the
observations normalization significantly helps the agent on Humamoid, see Figures 26 and 27 The action normalization influence tested in Appendix C
17

---

Under review as a conference paper at ICLR 2022
Hopper-v2
Walker2d-12
Ant-v2
Humanoid-v2
Aostagg fast ratum
0030
2000
5030
0000
2000
4030
4000
4000
2000
1000
2000
2002
joung)
(p.ag)
(sung
1002
5
Total samples
Total samples
Total samples
Total samples
Figure 26: The average test returns across the training of our method and ED2 with the rewards normalization.
Hopper-v2 v2
Walker2d
Ant.v2
Humanoid
Avaa aegragooo teatgllettin roturn
.000
6000
5030
2000
4000
4200
0000
2000
4000
1000
Mettod
E32
Serthd
2000
Mehhel
2000
ED2
Unitot
1000
Total samples
Total samples
1e6
Total samples
Total samples
Figure 27: The average test returns across the training of our method and ED2 with the observations normaliza- tion.
B.5 Q-FUNCTION UPDATES
Huber loss
We tried using the Huber loss for the Q-function training. It makes the results on all
tasks worse, see Figure 28
Hopper-v2
Walker2d-v2
Antov2
Humanoid-V/ v2
8000
Averae best return
0000
1000
Mbthod (pand)
2000
4000
1000
Sechod
2000
Nethoc
Nithed
Li2 (ams)
L13 (ours]
cours)
ED2 mube
E02 muuer
EDS thutse boat
Total samples
Total somples
Total samples
Total samples
Figure 28: The average test returns across the training of our method and ED2 with the Huber loss.
C ABLATION STUDY
In this section, we ablate the ED2 components to see their impact on performance and stability. We start with the ensemble exploration and exploitation and then move to the action normalization and the ERE replay buffer. In all plots, a solid line and a shaded region represent an average and 95% bootstrap confidence interval over 30 seeds in all but action normalization and ERE replay buffer experiments, where we run seeds.
Exploration & Exploitation In the first experiment we wanted to isolate the effect of ensemble- based temporally coherent exploration the performance and stability of ED2. Figures 29 32
compare the performance and stability of ED2 and one baseline, SOP, to ED2 with the single actor (the first one) used for evaluation in 22 Algorithm It is worth noting that the action selection during the data collection, step55in in Algorithm g is left unchanged - the ensemble of actors is used for ex elporation and each actor is trained on all the data. This should isolate the effect of exploration on
18

---

Under review as a conference paper at ICLR 2022
the test performance of every actor. The results show that the performance improvement and stability of ED2 does not come solely from the efficient exploration. ED2 ablation performs comparably to the baseline and is even less stable.
Hopper-v2
Walker2d-v2
And-v2
Humanoid-v2
Average test restrmm retum
3000
0000
5000
2000
0030
4200
4000
3000
(QuR))
pangle ador
2000
bums cangle swal)
242 jamgle voa.)
(pund)
2000
langle ado svil)
(suns)
1000
9OP
Total samples
108
Total samples
160
Total samples
168
Total samples
100
Figure 29: the baseline.
The average test returns across the training of ED2, ED2 with the single actor for exploitation, and
Hopper-V2
Walker2d-12
Anl--12
Humano
Sid day of tast neturn
800
Metied
T003
200
10.00
1000
1000
250
kistood
E32 (045)
600
220
E52 (ningle ador exaui pll
SOP
000
Total samples
Total samples
Total samples
166
Total semples
Figure 30: The standard deviation of test returns across the training of ED2, ED2 with the single actor for exploitation, and baseline.
Hopper-v2
Walker2d-12
Anl-v22
Humanoid-v2
Average tost retur
0000
0000
2000
0000
1000
2000
2000
4500
802 (suta)
EDI pango woter eval) spon
100 (nos)
EDR tsagle wast ent hoth
E02 (ouns)
EAR (snge actnt ort)) goo
E02 (osts)
EM tengle adter evei laoe
Figure 31:
The dots are the average test returns after training (3M samples) of each seed of ED2, ED2 with the single actor for exploitation. and the baseline. The distance between each box's top and bottom edges is the interquartile range (IQR). The whiskers spread across all values.
Rool mean squand catetirrtton
Hopper.v2
Walker2d-v2
Antov2
Humanoid-V
1000
1500
1250
600
150
400
450
6C2 (ssee)
EDR ginge wits evth) Igon
102 greN) EDR tsrgl wdor wal wat.
(SAA)
EX3 (sige site wvd wnd-) 5op
(SSO))
E2R (snglo wotes evil
Figure 32: RMSD of ED2, ED2 with the single actor for exploitation, and the baseline average and the 95% bootstrap confidence interval over 30 seeds.
19

---

Under review as a conference paper at ICLR 2022
In the next experiment, wanted to check if the ensemble evaluation is all we need in that event. Figure [3 compares the performance of ED2 and one baseline, SOP, to ED2 with the single actor (the first one) used for the data collection in step of Algorithm B The action selection during the evaluation, step 21 in A AlgorithmB: [] is left unchanged - the ensemble of actors is trained on the data collected only by one of the actors. We add Gaussian noise to the single actor's actions for exploration as described in Appendix B.2 The results show that the ensemble actor test performance collapses, possibly because of training on the out of distribution data. This implies that the ensemble of actors, used for evaluation, improves the test performance and stability. However, it is required that the same ensemble of actors also used for exploration, during the data collection.
Hopper-v2
Walker2d-12
Anl-v2
Humanoid-v2
average tast rastralum
8030
1000
5000
2000
4030
4200-
(Duns)
Maltodd
Kethen
tangle soter expll
1000
ED2
2000
lengle aqui]
jangpe sopll
right
1002
0
5
Total samplas
Total somples
168
Total samples
Total samples
Figure 33: The average test returns across the training of ED2, ED2 with the single actor for exploration, and the bascline.
The implementation details of the action normalization are described in Appendix E Figure 34 shows that the action normalization is especially required on the Ant and Humanoid environments, while not disrupting the training on the other tasks.
Action normalization
Hopper-v2
Walker2d-v2
Anl-v2
Humanoid-22
Auoraga test retur
6000
2000
5000
4000
2000
3000
1000
7SP (os wdtton rom)
E52 (pan))
2000
1030
ETP tyo onnooo
ED2 (ouns)
Metoop
2000
1000
ED2 (cum)
TTP (rovetton nem)
Methoc
2000
FnF jeo nom!
BD2 jouns
3
1
Total semples
148
semples
145
Total samples
1ec
Total samples
Figure 34: The average test returns across the training of ED2 with and without the action normalization.
ERE replay buffer The implementation details of the ERE replay buffer are described in Appendix In Figure 35 we observe that it improves the final performance of ED2 on all tasks, especially on Walker2d and Humanoid.
Hopper.v2 .2
Walker2d.v2 v2
Ant.v2
Humanoid v2
Aearage test testtgeeettooo return
6000
3000
2002
8000
2000
4220
4000
3000
1000
E02 (surs)
Mitiod
E32 (ou1s)
Mothod
ED8
Mchhod
2000
ED2 (ours)
Nethod
L03 tno US0
L33 L.R.1
LXX tro LM
Total semples
Total semples
166
Total sambles
Total samples
Figure 35: The average test returns across the training of ED2 with and without the ERE replay buffer.
20

---

Under review as a conference paper at ICLR 2022
D EXPERIMENTAL SETUP
Plots In all evaluations, we used 30 evaluation episodes to better access the average performance of each policy, as described in Section 2 For a more pleasant look and easier visual assessment, smoothed the lines using an exponential moving average with a smoothing factor equal 0.4.
OpenAI Gym MuJoCo In MuJoCo environments, that used, a state is defined by (I, y. position and velocity of the robot's 's'sis root, and angular position and velocity of each of its joints. The observation holds almost all information from the state except the x and y position of the robot's root. The action is torque that should be applied to each joint of the robot. Sizes of those spaces for each environment are summarised Table 2
MuJoCo is deterministic physics engine thus all simulations conducted inside it are deterministic. This includes simulations of our environments. However, to simplify the process of data gathering and to counteract weer-fitting the authors of OpenAl Gym decided to introduce some stochasticity. Each episode starts from a slightly different state initial positions and velocities are perturbed with random noise (uniform normal depending the particular environment).
Environment name
Action space size
Observation space size
Hopper-v2
3
11
Walker2d-v2
6
17
Ant-v2
8
111
Humanoid-v2
17
376
Table 2: Action and observation space sizes for used environments.
E IMPLEMENTATION DETAILS
Architecture and syper-parameters In our experiments, we use deep neural networks with two hidden layers, each of them with 256 units. All of the networks use ReLU an activation, except on the final output layer, where the activation used varies depending on the model: critic networks use no activation, while actor networks use tanh() multiplied by the max action scale. Table 3 shows the hyper-paranen pper-pprrmeters used for the tested algorithms.
Parameter
SAC
SOP
SUNRISE
ED2
discounting
0.99
0.99
0.99
0.99
optimizer
Adam
Adam
Adam
Adam
learning rate
10-3
10-1
10-3
10-4
replay buffer size
106
106
106
106
batch size ensemble size entropy coefficient
256
256
256
256
5
0.2
0.2
update interval 17u (ERE)
50
50
50
50
0.995
0.995
Number of environment interactions between updates.
Table 3: Default values of hyper-parameterd were used in our experiments.
Action normalization Our algorithm employs action normalization proposed by WWan et (2020). means that before applying the squashing function (e.g. lanh()), the outputs of each actor network are normalized in the following way: let μ = (И1 ........ MA) be the output of the actor's network and let = E |pal I/A be the average magnitude of this output, where A is the action 's dimensionality. If G then normalize the output by setting μ4 p4/ G for all i = 1, ... A. Otherwise, leave the output unchanged. Each actor's outputs are normalized independently from other actors in the ensemble.
21

---

Under review as a conference paper at ICLR 2022
Algorithm 3 ED2 Ensemble Deep Deterministic Policy Gradients
Input: ensemble size K; init. policy 0j. and Q-functions Φ Φx.2 param. where E [l,..., K];
replay buffer D; max action scale M; target smoothing std. dev. a; interpolation factor p:
1: Set the target parameters Φ &- @ 6k.2 + ΦR.2
2: Sample the current policy index ~ [[I,...... K]).
3: Reset the environment and observe the state
4: repeat
5: Execute action M tanh (ua,(s))
/ the action normalization
6: Observe and store (s, a, 7; s', d) in the replay buffer D.
7: Set 4' s'
8:
if episode is finished then
9:
Reset the environment and observe initial state Sample the current policy index с ~ U(II.... K]).
10:
11:
if time update then
12:
for as many as steps done in the environment do
13:
Sample batch of transitions B (ss,,,,, d)) C D Compute targets
ERE
14:
(k s d) = 7(1 min Qs (s' a(²) =11
ak M tanh (mo, (5') e) ((,
15:
Update the Q-functions by one step of gradient descent using
Σ
(Qoi, ux(r, d))² for E (1, [I...., K]
do
BB
".a.r,e'ddjBB
16:
Update the policies by step of gradient ascent using
VoL
Σ
Qan. (o, for & E (1,...,K
B K
=EB
17:
Update target parameters with
tz.t - pōk.; + (1 p)ou. for {1,2), k [1..... ......KK
if time evaluate then
19:
for specified number of evaluation runs do Reset the environment and observe the state s. Execute policy 1 k Σ M tanh (No, (s)) until the terminal state.
21:
22:
Record and log the return.
23:
until convergence
22

---

Under review as a conference paper at ICLR 2022
Emphasizing Recent Experience We implement the Emphasizing Recent Experience (ERE) mech- anism from Wang et al. (2020). ERE samples non-uni non-unifomlly from the most recent experiences stored in the replay buffer. Let B be the number of mini-batch updates and [D] be the size of the replay buffer. When performing the gradient updates, we sample from the most recent Co data points stored in the replay buffer, where
[D] -7 1 for b - , B.
The hyper-parameter v) starts off with set value of r/o and later adapted based the improvements in the agent training performance. Let Irecett be the improvement in terms of training episode returns made over the last /2/2 time-steps and mex be the maximum of such improvements over the course of the training. We adapt 17 according to the formula:
Trecent
recent
77 170
1
Imoz
Imo
Our implementation uses the exponentially weighted moving average store the value of Trecent- More concretely, define Lrecent based on two additional parameters Rrecent and Rprew so that reeent Ryecent - Ryrev. Those parameters are then updated whenever we receive a new training episode return ep_ret:
Rreeent Arecent ep_ret + (1 Arecent) Brecent
Rpree Apree ep_ret + (1 - Aprev) Rprev
where Aprev T/|
Arecent =
10 Aprev and T is the maximum length of an episode.
Hardware During the training of our models, we employ only CPUs using cluster where each node has 28 available cores of 2.6 GHz, alongside at least 64 GB of memory. The running time of typical experiment did not exceed 24 hours.
23


Reviews:
[{"review_id": "pDiASP6k0H", "reviewer": "Reviewer_thDH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well-written, flows smoothly and is a pleasure to read. The ideas are well articulated and clear.\n\nResult in Fig 2 is surprising as it suggests that the additive normal action noise is entirely unnecessary. This has been a fixture is most DRL algorithms. However, looking in Appendix B.2, the authors did not test Ornstein-Uhlenbeck noise (DDPG paper) as one of the baselines. Adding this popular choice would complete this empirical evaluation. I do not think that OU noise would change the conclusions, but it would be nice to include for the sake of completeness - considering that it is used in some of the seminal works in the field.\n\nThe results for the stability experiments in Fig 6-8 on inference stability, asymptotic performance stability and training stability do not seem that surprising to me. Wouldn’t an ensemble method fundamentally lead to more stable learning? SUNRISE seems to be an exception here for some domains. However, ED2 being more ‘stable’ that a lone network algorithm (like SAC baseline) run seems to be rather obvious to me. An ensemble method should fundamentally be more stable as it has the advantage of N=5 random initializations. Would perhaps a better comparison be against an ensemble of SAC runs? \n\nThe main contribution of the paper are (1) the empirical study into the various design choices within DRL algorithms used for continuous control settings and (2) an ensemble approach that integrates these learnings. While the ensemble method achieves SOTA results on many tasks and the empirical study presents some riveting results, the novelty in the paper is quite limited. \n", "summary_of_the_paper": "The paper presents an empirical study evaluating the commonly accepted design choice in off-policy Deep RL algorithms in continuous control settings. The use of additive exploration noise, initialization choices, update frequency, and precision for retraining are tested empirically highlighting some interesting results. The paper also introduces ED2 - an ensemble method utilizing the design choices from the study which is demonstrated to achieve SOTA results on Mujoco benchmarks.", "summary_of_the_review": "The paper presents some interesting learnings via a diligent evaluation of the design choices used in Deep RL algorithms for continuous control settings. The paper presents an ensemble method using these learnings and posts some SOTA results. I believe the community would benefit from these learnings and the resulting ED2 approach. However, I do note that the novelty for the method is quite limited as ED2 is fundamentally an ensemble of previous method (SOP) with well-evaluated design parameters for the target task of Mujoco benchmarks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VhX9kO8aP3I", "reviewer": "Reviewer_ZQ4d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The code is open-sourced, which is excellent for reproducibility \n- There is significant focus on the stability of RL algorithms, which is great to see. Often RL papers just present the reward curves and information on the stability of these algorithms is important. \n- Using 30 random seeds (instead of the usual 3-6) is great and strengthens any empirical claims\n\nCons:\n- The empirical results don’t seem to justify the algorithm. Figure 4 represents the main empirical argument for ED2 but the empirical gains shown are minor. On Walker it appears as though SUNRISE actually outperforms ED2. Humanoid and Hopper both have ED2 on top, but only by a very slim margin (one that has overlapping confidence intervals with other algorithms). \n- Small number of environments. The algorithms are only compared on 4 environments (Ant, Hopper, Humanoid, Walker) when MuJoCo/OpenAI Gym offers many more (e.g. HalfCheetah, InvertedDoublePendulum, Swimmer, Reacher, etc.). The empirical results could be more compelling if the scope was increased. If you expanded the results presented in Table 1 for all algorithms, this could be a more compelling argument. \n- Much of the paper is dedicated to information that would be better in the appendix. Information about non-essential ablation studies (e.g. Figure 2 shows the impacts are pretty minor) are not necessary in the main body. These figures could be replaced with more directly relevant information, like to aid claims of stability, e.g. by injecting varying degrees of noise/randomness into the environment and evaluating relative performances, or evaluating sensitivity to other hyperparameters (network structure, learning rates, etc.) \n- Lack of theoretical understanding. I am well aware of the problems of equation packing and the disconnect that often occurs in RL papers between the algorithm and the theoretical justification, but at least providing some insight into comparisons of UCB, posterior sampling, priors, etc. would be nice. \n- The empirical comparisons don’t seem to offer an even playing field for all algorithms. Based on (Haarnoja et al., 2019), SAC should reach ~8000 on Humanoid-v2 (after 10e6 steps). However, the experiments are only conducted till 3e6 steps with multiple algorithms appearing to still have upward trajectories. There seems to be no mention of a focus on the speed of learning/early learning performance vs final performance. \n- The relationship between Figure 4 and Figure 6 is not clear. Figure 6 seems to indicate that (e.g.) pretty much all algorithms have >1,000 STD on Humanoid and Ant early on in training. However, the bootstrapped CI in Figure 4 is substantially smaller. This discrepancy is uncommon and should be explained. \n- Reinforcement learning is notoriously brittle and I would encourage references to the previous work done evaluating the effect of random seeds/initialization on agent performance e.g. the famous (Henderson et al., 2019).\n- Figure 5 does not provide any support or meaningful insight. The velocity of humanoid is not what the algorithm is optimizing. It is included for “completeness”, but this seems to be a cherry-picked statistic that doesn’t convey anything meaningful (while the speed is incorporated in the reward function, since the rewards are much closer than the velocities we don’t see what the tradeoff is). \n- Figure 1 does not really help clarify anything. If one already knows the environments, then the figure is unnecessary, if one is unfamiliar with them, the figure doesn’t show what actually transpires in the environment and doesn’t clear anything up.\n- To claim ED2 is really SotA, further analysis is necessary across environments like Agarwal et al. 2021, Barreto et al. 2010, Jordan et al. 2020, etc. \n- There are minor typographical inconsistencies, e.g. differing usages of \\cite{} and \\citep{}\n- Inconsistent background information. The paper provides a definition of standard deviation, an extremely common statistical measurement, but not for much more niche terms such as approximated UCB. To be clear, I have no problem with giving the formula for STD, but not giving definitions for much less widely known terms is something that could be fixed. \n\nMisc/Note:\nThis paper seems like it’s trying to do two things at once: (1) provide a review of RL techniques (e.g. exploitation techniques), evaluate their impact and report on the key takeaways of this empirical analysis, and (2) introduce a novel RL algorithm and justify it’s empirical construction and performance. Both of these are papers that are perfectly fine, but by trying to do both it leaves something lacking from each of them. If this was a review of techniques, I would like to see more continuous control algorithms evaluated and more techniques experimented with. If this is just introducing a novel algorithm, I would like to see more theoretical explanations of the techniques (e.g. theoretical derivations and insights into the effect of K)  and more extensive ablation studies (e.g. evaluating on more MuJoCo tasks or other continuous control environments that have different properties such as RLBench, Industrial control benchmark, assistive gym, DM Control suite, etc.). \n", "summary_of_the_paper": "This paper presents a deep reinforcement algorithm, Ensemble Deep Deterministic Policy Gradients (ED2), for continuous control tasks. The algorithm is empirically derived and is claimed to represent SotA performance on several tasks and while providing more stable results. These claims are justified based primarily on the (reward and stability) results on 4 MuJoCo environments. \n", "summary_of_the_review": "Given that the algorithm is entirely empirically derived and the empirical results are not compelling, I have given this paper a reject. I appreciate that there are important ideas represented here, but it currently isn’t up to the ICLR standard. \n\nAfter author rebuttal, the score has moved from 3 to 5 (see comment for more information). \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ue9AbLA4V-", "reviewer": "Reviewer_M5Fw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe main strength of the work is introducing a straightforward extension of an existing base actor-critic method that substantially outperforms existing algorithms in standard benchmark tasks. The new algorithm also has some desirable properties for policy optimization such as not having random additive noise and providing stable performance.\n\nMoreover, some key insights on deep policy gradient methods are presented such as the contribution of actor and critic initialization.\n\nAnother strength of the paper is its focus on various details such as ideas that didn’t work as well as ablative studies in various manners.\n\nWeakness:\n\nThe main weakness of the work is the fairness of the experiments. This deficiency is common in many papers including those that get published in top conferences but acceptance doesn’t justify wrong choices. I would be willing to know from the authors their thoughts on it.\n\nThe first issue with such experiments is that only a single hyper-parameters are used for the methods to show comparison. However, the claims concluded from such results are that one algorithm outperformed the others. However, to make such a claim, different hyper-parameter values should be tried for all algorithms. Otherwise, the claim should be humbler such as our method outperforms the competing methods with default choices of hyper-parameters and such. Such hyper-parameter search is also necessary for ablative studies. When we are removing one component at a time, we cannot assume that the default hyper-parameter configuration of the original method will still be effective for the subsequent variants.\n\nI understand that it will make deep RL experiments much more expensive. However, it isn’t necessary to perform a grid search over hyper-parameters. It has been shown before that random search can give close to the best performance within a handful of trials of configurations, which will considerably reduce the search cost.\n\nIn a similar vein, it has been common to compare algorithms with different computational profiles. However, when a new algorithm is computationally way more expensive than the competitors, is it fair to compare them with such computational disparity and claim one algorithm is better than the other?\n\nDetails on the computational expense of ED2 are not given. How many actors are used for ED2? How much more expensive ED2 is compared to its competitors such as SAC or SOP?\n\nOther comments:\n- To understand more clearly, det. SOP uses no exploration whatsoever and still performs well on these tasks?\n\n- Both the action averaging and greedy choice among actors yielded similar results. This leads me to suspect whether the actors either converged to similar performant behavior or stationary behavior with zero torque, which upon averaging gives a behavior similar to the greedy one.\n\n- Considering the performance and the computational expense compared to ED2, det. SOP seems a strong contender. Why is it not added to Figures 6, 7, or 8?\n\n- Figure 9 somewhat makes sense except that there is a puzzle. When ED2 is reduced to single critic, the diversity is reduced considerably, which hurts possibly the exploration and reduces performance substantially. But if we reduce ED2 single critic further by also having a single actor, then don’t we get det. SOP, which wasn’t doing as badly as ED2 single critic? How can that be explained?\n\n- What's really the motivation behind having a separate evaluation phase just to measure performance for plot when learning online? If these algorithms are deployed to learn online say on a robot, their online performance is the actual evaluation. Creating an additional evaluation phase to measure performance will only delay its learning in real-time. In what case, such a separate evaluation is useful other than because many other works repeat it? Even if there is a case, isn't it quite restrictive? Wouldn't it be important to see the online performance of ED2 as it randomly draws actors to interact? If that performance is also good, it would be a more interesting and stronger result.", "summary_of_the_paper": "This paper has two main contributions: it introduces an ensemble-based actor-critic method, and it answers some pertinent questions in policy optimization by focusing on its different components. The ensemble is different from multi-actor learners that interact with multiple environments simultaneously, violating the standard RL setup. Instead, the learner of this paper maintains multiple actors and critics but uses only a single actor at a time to interact with the environment. All actors and critics are trained on a common replay buffer. The base method is the streamlined off-policy (SOP) method, which unlike soft actor-critic (SAC) doesn’t use an entropy bonus. Additionally, no exploration noise is added, resulting in their Ensemble Deep Deterministic (ED2) method.\n\nThe proposed algorithm ED2 is shown to be superior and more stable in performance according to different measures compared to existing methods. It is also revealed that actor initialization affects performance less than critic initialization. ED2 uses deterministic actors, and its exploration comes from sampling among the actors. Such a form of exploration is also shown to be superior to UCB-style exploration.\n", "summary_of_the_review": "The paper’s strength is a straightforward performant policy optimization method and the insights developed through experiments. However, hyper-parameter search isn’t performed to substantiate the strong claims and it isn’t clear how much more computationally expensive ED2 is compared to its competitors.\n\n*** updated ***\n\nI will update later. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KL2jvGzFgMO", "reviewer": "Reviewer_2e3U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n1. The paper is also well-written and easy to understand\n2. I believe the paper studies a problem that's important and significant in the rl community and very relevant to the venue\n\nweaknesses/questions\nIn general, I think the results should be treated more carefully and some theoretical motivation is needed. Following are some of my questions. \n1.in section 4.1, the conclusion of \"This result shows that no additional exploration mechanism, often in a form of an exploration noise (Lillicrap et al., 2016; Fujimoto et al., 2018; Wang et al., 2020), is required for the diverse data collection and it can even hinder training.\" seems rather strong judging from fig.2 (I would say they are roughly the same). \n2. \"Figure 3 shows that ED2 magnifies the beneficial effect coming from the deterministic exploration.\", wouldn't it make better sense if you compare against the ones that are also ensemble, e.g. ensemble SOP instead of SOP? How do you know what brings the performance, is it the ensemble or the determinist action, or both? \n3. fig.4 again, why not compare the ensemble baselines? I think it's important to ablate the design choices that are actually important\n\nSome minor problems. \n1. maybe more introduction for SOP (e.g. what's the ere replay buffer?) \n2.sec.3, \"These two choices ensure coherent and temporally-extended exploration similarly to Osband et al. (2016).\", I do not understand why is the exploration \"coherent and temporally-extended\". \n3.in sec.3, I guess the used and not used should be comparable and matched, but why is action normalization compared with \"observations and rewards normalization\"? \n4.fig.5, what does it mean for the Humanoid velocities? could you elaborate?", "summary_of_the_paper": "This paper presents an empirical study between different commonly-used tricks and elements of off-policy RL algorithms and tries to understand the interplay between those elements. The authors propose a new method called ED2 that utilizes their insights from the empirical study.", "summary_of_the_review": "I believe the paper should present a more careful analysis of these design elements and put more effort into justifying their choices used for comparison.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2oYcEsHZsC", "reviewer": "Reviewer_hF9t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While this paper seems to show some interesting new results related to ensemble reinforcement learning, there are several issues with this paper in this current shape:\n\n1. The technical innovation of this paper remains largely unclear. It was claimed by the authors in this paper that ED2 brings together existing RL tools in a novel way. However, it is unclear which part of the design of ED2 is truly novel. As far as I am aware, ED2 mainly used existing training techniques and ensemble tricks. A series of experiments were carried out to justify the use of several different tricks in ED2. While the combined use of these tricks might be new in ED2, it is not clear why such a combination is potentially more superior than other possible combinations. Furthermore, since the experiments focus mainly on four benchmark problems, it is questionable whether ED2 can achieve clearly better performance over other ensemble baseline algorithms on a much wider range of reinforcement learning problems. Hence the novelty and technical contribution of this paper may need to be improved.\n\n2. This paper lacks theoretical depth. The experiment results in the paper only revealed some insights. However, no further theoretical analysis was conducted to verify (or at least partially explain) the experimental observations. For example, on page 6, the authors conjectured that good exploration may come more from the critic. While this sounds interesting, it is unclear why the critic will play such a critical role to induce effective exploration and what the corresponding conditions are for this to happen.\n\n3. Some experiment findings and the corresponding claims do not appear to be consistent. For example, on page 4, the authors found experimentally that additive normal action noise can substantially improve the Ant performance. They subsequently concluded that additive noise is not required for effective learning. These two claims do not sound consistent. Accordingly, the main findings discovered in the paper may need to be further verified.\n\n4. Some experiment findings appear to be well-known a priori in the literature. For example, as acknowledged by the authors, posterior sampling techniques can be more effective than the OFU strategy for action selection. Consequently, the technical contribution of the corresponding experiment results does not seem to be sufficiently strong.", "summary_of_the_paper": "This paper conducted an experimental study over a range of tricks that are often exploited to facilitate ensemble deep reinforcement learning. The experiment results show several interesting findings. For example, it was found that commonly used additive action noise may not be necessary for effective exploration. Meanwhile, experiments show that the initialization of critics perhaps has a higher impact on learning performance than the initialization methods adopted for actors. These findings can be quite important to guide future design of more effective ensemble reinforce learning algorithms.", "summary_of_the_review": "This paper conducted an experimental study over a range of tricks that are often exploited to facilitate ensemble deep reinforcement learning. The corresponding empirical findings can be quite important to guide future design of more effective ensemble reinforce learning algorithms. Meanwhile, the technical novelty and theoretical depth of this paper may need to be strengthened.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies improving continuous control. The paper suggests a practical, beneficial combination approach that does well in the presented experiments. It also provides some overview and comparison over several recent insights in RL. While both are valuable, multiple reviewers had concerns that the paper has some limitations on both. In particular, the proposed ensemble approach is quite simple though valuable, and that reviewers generally felt that raised their expectations as to the strength of the empirical results which was not yet there. The reviewers’ provided a lot of detailed feedback that may be useful in revising the contribution.
Decision: Reject