Continuous Control With Ensemble Deep Deterministic Policy Gradients

1 INTRODUCTION . Recently , deep reinforcement learning ( RL ) has achieved multiple breakthroughs in a range of challenging domains ( e.g . Silver et al . ( 2016 ) ; Berner et al . ( 2019 ) ; Andrychowicz et al . ( 2020b ) ; Vinyals et al . ( 2019 ) ) . A part of this success is related to an ever-growing toolbox of tricks and methods that were observed to boost the RL algorithms ’ performance ( e.g . Hessel et al . ( 2018 ) ; Haarnoja et al . ( 2018b ) ; Fujimoto et al . ( 2018 ) ; Wang et al . ( 2020 ) ; Osband et al . ( 2019 ) ) . This state of affairs benefits the field but also brings challenges related to often unclear interactions between the individual improvements and the credit assignment related to the overall performance of the algorithm Andrychowicz et al . ( 2020a ) ; Ilyas et al . ( 2020 ) . In this paper , we present a comprehensive empirical study of multiple tools from the RL toolbox applied to the continuous control in the OpenAI Gym MuJoCo setting . These are presented in Section 4 and Appendix B . Our insights include : • The normally distributed action noise , commonly used for exploration , hinders training . • The current state-of-the-art methods are unstable under several stability criteria . • The critics ’ initialization plays a major role in ensemble-based actor-critic exploration , while the training is mostly invariant to the actors ’ initialization . • The approximated posterior sampling exploration ( Osband et al. , 2013 ) outperforms approx- imated UCB exploration combined with weighted Bellman backup ( Lee et al. , 2020 ) . • The weighted Bellman backup ( Lee et al. , 2020 ) can not replace the clipped double Q- Learning ( Fujimoto et al. , 2018 ) . To address some of the issues listed above , we introduce the Ensemble Deep Deterministic Policy Gradient ( ED2 ) algorithm1 , see Section 3 . ED2 brings together existing RL tools in a novel way : it is 1Our code is based on SpinningUp ( Achiam , 2018 ) . We open-source it at : https : //github.com/ ed2-paper/ED2 . an off-policy algorithm for continuous control , which constructs an ensemble of streamlined versions of TD3 agents and achieves the state-of-the-art performance in OpenAI Gym MuJoCo , substantially improving the results on the two hardest tasks – Ant and Humanoid . Consequently , ED2 does not require knowledge outside of the existing RL toolbox , is conceptually straightforward , and easy to code . 2 BACKGROUND . We model the environment as a Markov Decision Process ( MDP ) . It is defined by the tuple ( S , A , R , P , γ , p0 ) , where S is a continuous multi-dimensional state space , A denotes a continuous multi-dimensional action space , P is a transition kernel , γ ∈ [ 0 , 1 ) stands for a discount factor , p0 refers to an initial state distribution , and R is a reward function . The agent learns a policy from sequences of transitions τ = [ ( st , at , rt , st+1 , d ) ] Tt=0 , called episodes or trajectories , where at ∼ π ( ·|st ) , st+1 ∼ P ( ·|st , at ) , rt = R ( st , at , st+1 ) , d is a terminal signal , and T is the terminal time-step . A stochastic policy π ( a|s ) maps each state to a distribution over actions . A deterministic policy µ : S −→ A assigns each state an action . All algorithms that we consider in this paper use a different policy for collecting data ( exploration ) and a different policy for evaluation ( exploitation ) . In order to keep track of the progress , the evaluation runs are performed every ten thousand environment interactions . Because of the environments ’ stochasticity , we run the evaluation policy multiple times . Let { Ri } Ni=1 be a set of ( undiscounted ) returns from N evaluation episodes { τi } Ni=1 , i.e . Ri = ∑ rt∈τi rt . We evaluate the policy using the average test return R̄ = 1N ∑N i=1Ri and the standard deviation of the test returns σ = √ 1 N−1 ∑N i=1 ( Ri − R̄ ) 2 . We run experiments on four continuous control tasks and their variants , introduced in the appropriate sections , from the OpenAI Gym MuJoCo suite ( Brockman et al. , 2016 ) presented in Figure 1 . The agent observes vectors that describe the kinematic properties of the robot and its actions specify torques to be applied on the robot joints . See Appendix D for the details on the experimental setup . 3 ENSEMBLE DEEP DETERMINISTIC POLICY GRADIENTS . For completeness of exposition , we present ED2 before the experimental section . The ED2 architecture is based on an ensemble of Streamlined Off-Policy ( SOP ) agents ( Wang et al. , 2020 ) , meaning that our agent is an ensemble of TD3-like agents ( Fujimoto et al. , 2018 ) with the action normalization and the ERE replay buffer . The pseudo-code listing can be found in Algorithm 1 , while the implementation details , including a more verbose version of pseudo-code ( Algorithm 3 ) , can be found in Appendix E. In the data collection phase ( Lines 1-9 ) , ED2 selects one actor from the ensemble uniformly at random ( Lines 1 and 9 ) and run its deterministic policy for the course of one episode ( Line 4 ) . In the evaluation phase ( not shown in Algorithm 1 ) , the evaluation policy averages all the actors ’ output actions . We train the ensemble every 50 environment steps with 50 stochastic gradient descent updates ( Lines 10-13 ) . ED2 concurrently learns K · 2 Q-functions , Qφk,1 and Qφk,2 where k ∈ K , by mean square Bellman error minimization , in almost the same way that SOP learns its two Q-functions . The only difference is that we have K critic pairs that are initialized with different random weights and then trained independently with the same batches of data . Because of the different initial weights , each Q-function has a different bias in its Q-values . The K actors , πθk , train maximizing their corresponding first critic , Qφk,1 , just like SOP . Algorithm 1 ED2 - Ensemble Deep Deterministic Policy Gradients Input : init . params for policy θk and Q-functions φk,1 , φk,2 , k ∈ [ 1 ... K ] ; replay buffer D ; 1 : Sample the current policy index c ∼ U ( [ 1 ... K ] ) . 2 : Reset the environment and observe the state s. 3 : repeat 4 : Execute action a = µθc ( s ) . µ uses the action normalization 5 : Observe and store ( s , a , r , s′ , d ) in the replay buffer D. 6 : Set s← s′ 7 : if episode is finished then 8 : Reset the environment and observe initial state s. 9 : Sample the current policy index c ∼ U ( [ 1 ... K ] ) . 10 : if time to update then 11 : for as many as steps done in the environment do 12 : Sample a batch of transitions B = { ( s , a , r , s′ , d ) } ⊂ D . uses ERE 13 : Update the parameters θk , φk,1 and φk,2 by one gradient step . 14 : until convergence Utilizing the ensembles requires several design choices , which we summarize below . The ablation study of ED2 elements is provided in Appendix C . Ensemble . Used : We train the ensemble of 5 actors and 5 critics ; each actor learns from its own critic and the whole ensemble is trained on the same data . Not used : We considered different actor-critic configurations , initialization schemes and relations , as well as the use of random prior networks ( Osband et al. , 2018 ) , data bootstrap ( Osband et al. , 2016 ) , and different ensemble sizes . We also change the SOP network sizes and training intensity instead of using the ensemble . Besides the prior networks in some special cases , these turn out to be inferior as shown in Section 4 and Appendix B.1 . Exploration . Used : We pick one actor uniformly at random to collect the data for the course of one episode . The actor is deterministic ( no additive action noise is applied ) . These two choices ensure coherent and temporally-extended exploration similarly to Osband et al . ( 2016 ) . Not used : We tested several approaches to exploration : using the ensemble of actors , UCB ( Lee et al. , 2020 ) , and adding the action noise in different proportions . These experiments are presented in Appendix B.2 . Exploitation . Used : The evaluation policy averages all the actors ’ output actions to provide stable performance . Not used : We tried picking an action with the biggest value estimate ( average of the critics ’ Qfunctions ) in evaluation ( Huang et al. , 2017 ) . Interestingly , both policies had similar results , see Appendix B.3 . Action normalization . Used : We use the action normalization introduced by Wang et al . ( 2020 ) . Not used : We experimented with the observations and rewards normalization , which turned out to be unnecessary . The experiments are presented in Appendix B.4 . Q-function updates . Used : We do 50 SGD updates ( ADAM optimizer ( Kingma and Ba , 2015 ) , MSE loss ) to the actors and the critics every 50 environment interactions , use Clipped Double Q-Learning ( Fujimoto et al. , 2018 ) . Not used : We also examined doing the updates at the end of each episode ( with the proportional number of updates ) , using the Hubert loss , and doing weighted Bellman backups ( Lee et al. , 2020 ) . However , we found them to bring no improvement to our method , as presented in Appendix B.5 . 4 EXPERIMENTS . In this section , we present our comprehensive study and the resulting insights . The rest of the experiments verifying that our design choices perform better than alternatives are in Appendix B . Unless stated otherwise , a solid line in the figures represents an average , while a shaded region shows a 95 % bootstrap confidence interval . We used 30 seeds for ED2 and the baselines and 7 seeds for the ED2 variants . 4.1 THE NORMALLY DISTRIBUTED ACTION NOISE , COMMONLY USED FOR EXPLORATION , HINDERS TRAINING . In this experiment , we deprive SOP of its exploration mechanism , namely additive normal action noise , and call this variant deterministic SOP ( det . SOP ) . It causes relatively minor deterioration in the Humanoid performance , has no significant influence on the Hopper or Walker performance , and substantially improves the Ant performance , see Figure 2 . This result shows that no additional exploration mechanism , often in a form of an exploration noise ( Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ; Wang et al. , 2020 ) , is required for the diverse data collection and it can even hinder training . ED2 leverages this insight and constructs an ensemble of deterministic SOP agents presented in Section 3 . Figure 3 shows that ED2 magnifies the beneficial effect coming from the deterministic exploration . ED2 achieves state-of-the-art performance on the OpenAI Gym MuJoCo suite . Figure 4 shows the results of ED2 contrasted with three strong baselines : SUNRISE ( Lee et al. , 2020 ) , SOP ( Wang et al. , 2020 ) , and SAC ( Haarnoja et al. , 2018b ) . For completeness , we plot the Humanoid velocities in Figure 5 which shows that our method accelerates to a much higher velocity than the baselines .

Summary: This paper has two main contributions: it introduces an ensemble-based actor-critic method, and it answers some pertinent questions in policy optimization by focusing on its different components. The ensemble is different from multi-actor learners that interact with multiple environments simultaneously, violating the standard RL setup. Instead, the learner of this paper maintains multiple actors and critics but uses only a single actor at a time to interact with the environment. All actors and critics are trained on a common replay buffer. The base method is the streamlined off-policy (SOP) method, which unlike soft actor-critic (SAC) doesn’t use an entropy bonus. Additionally, no exploration noise is added, resulting in their Ensemble Deep Deterministic (ED2) method.  The proposed algorithm ED2 is shown to be superior and more stable in performance according to different measures compared to existing methods. It is also revealed that actor initialization affects performance less than critic initialization. ED2 uses deterministic actors, and its exploration comes from sampling among the actors. Such a form of exploration is also shown to be superior to UCB-style exploration.