What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study

1 INTRODUCTION . Deep reinforcement learning ( RL ) has seen increased interest in recent years due to its ability to have neural-network-based agents learn to act in environments through interactions . For continuous control tasks , on-policy algorithms such as REINFORCE [ 2 ] , TRPO [ 10 ] , A3C [ 14 ] , PPO [ 17 ] and off-policy algorithms such as DDPG [ 13 ] and SAC [ 21 ] have enabled successful applications such as quadrupedal locomotion [ 20 ] , self-driving [ 30 ] or dexterous in-hand manipulation [ 20 , 25 , 32 ] . Many of these papers investigate in depth different algorithmic ideas , for example different loss functions and learning paradigms . Yet , it is less visible that behind successful experiments in deep RL there are complicated code bases that contain a large number of low- and high-level design decisions that are usually not discussed in research papers . While one may assume that such “ choices ” do not matter , there is evidence that they are in fact crucial for or even driving good performance [ 27 ] . While there are open-source implementations available that can be used by practitioners , this is still unsatisfactory : Research publications often contain one-to-one comparisons of different algorithmic ideas based on implementations in different code bases . This makes it impossible to assess whether improvements are due to the underlying algorithmic idea or due to the implementation . In fact , it is hard to assess the performance of high-level algorithmic ideas without an understanding of lower-level choices as performance may strongly depend on the tuning of hyperparameters and implementationlevel details . Overall , this makes it hard to attribute progress in reinforcement learning and slows down further research [ 15 , 22 , 27 ] . Our contributions . Our key goal in this paper is to investigate such lower level choices in depth and to understand their impact on final agent performance . Hence , as our key contributions , we ( 1 ) implement > 50 choices in a unified on-policy deep actor-critic implementation1 , ( 2 ) conducted a large-scale ( more than 250 ’ 000 agents trained ) experimental study that covers different aspects of the training process , and ( 3 ) analyze the experimental results to provide practical insights and recommendations for the training of on-policy deep actor-critic RL agents . 1The implementation is available at https : //github.com/google-research/seed_rl . Most surprising finding . While many of our experimental findings confirm common RL practices , some of them are quite surprising , e.g . the policy initialization scheme significantly influences the performance while it is rarely even mentioned in RL publications . In particular , we have found that initializing the network so that the initial action distribution has zero mean , a rather low standard deviation and is independent of the observation significantly improves the training speed ( Sec . 3.2 ) . Paper outline . The rest of of this paper is structured as follows : We describe our experimental setup and performance metrics used in Sec . 2 . Then , in Sec . 3 we present and analyse the experimental results and finish with related work in Sec . 4 and conclusions in Sec . 5 . The appendices contain the detailed description of all design choices we experiment with ( App . B ) , default hyperparameters ( App . C ) and the raw experimental results ( App . D - K ) . 2 STUDY DESIGN . Considered setting . In this paper , we consider the setting of on-policy deep actor-critic reinforcement learning for continuous control . We define on-policy learning in the following loose sense : We consider policy iteration algorithms that iterate between generating experience using the current policy and using that experience to improve the policy . This is the standard modus operandi of algorithms usually considered on-policy such as PPO [ 17 ] . However , we note that algorithms often perform several model updates and thus may operate technically on off-policy data within a single policy improvement iteration . As benchmark environments , we consider five widely used continuous control environments from OpenAI Gym [ 12 ] of varying complexity : Hopper-v1 , Walker2d-v1 , HalfCheetah-v1 , Ant-v1 , and Humanoid-v1 2 . Unified on-policy deep actor-critic gradient algorithm . We took the following approach to create a highly configurable unified on-policy deep actor-critic gradient algorithm with as many choices as possible : 1 . We researched prior work and popular code bases to make a list of commonly used choices , i.e. , different loss functions ( both for value functions and policies ) , architectural choices such as initialization methods , heuristic tricks such as gradient clipping and all their corresponding hyperparameters . 2 . Based on this , we implemented a single , unified on-policy deep actor-critic agent and corresponding training protocol starting from the SEED RL code base [ 28 ] . Whenever we were faced with implementation decisions that required us to take decisions that could not be clearly motivated or had alternative solutions , we further added such decisions as additional choices . 3 . We verified that when all choices are selected as in the PPO implementation from OpenAI baselines , we obtain similar performance as reported in the PPO paper [ 17 ] . We chose PPO because it is probably the most commonly used on-policy deep actor-critic RL algorithm at the moment . The resulting agent implementation is detailed in Appendix B . The key property is that the implementation exposes all choices as configuration options in an unified manner . For convenience , we mark each of the choice in this paper with a number ( e.g. , C1 ) and a fixed name ( e.g . num_envs ( C1 ) ) that can be easily used to find a description of the choice in Appendix B . Difficulty of investigating choices . The primary goal of this paper is to understand how the different choices affect the final performance of an agent and to derive recommendations for these choices . There are two key reasons why this is challenging : First , we are mainly interested in insights on choices for good hyperparameter configurations . Yet , if all choices are sampled randomly , the performance is very bad and little ( if any ) training progress is made . This may be explained by the presence of sub-optimal settings ( e.g. , hyperparameters of the wrong scale ) that prohibit learning at all . If there are many choices , the probability of such failure increases exponentially . 2It has been noticed that the version of the Mujoco physics simulator [ 5 ] can slightly influence the behaviour of some of the environments — https : //github.com/openai/gym/issues/1541 . We used Mujoco 2.0 in our experiments . Second , many choices may have strong interactions with other related choices , for example the learning rate and the minibatch size . This means that such choices need to be tuned together and experiments where only a single choice is varied but interacting choices are kept fixed may lead to misleading conclusions . Basic experimental design . To address these issues , we design a series of experiments as follows : We create groups of choices around thematic groups where we suspect interactions between different choices , for example we group together all choices related to neural network architecture . We also include Adam learning rate ( C24 ) in all of the groups as we suspect that it may interact with many other choices . Then , in each experiment , we train a large number of models where we randomly sample the choices within the corresponding group 3 . All other settings ( for choices not in the group ) are set to settings of a competitive base configuration ( detailed in Appendix C ) that is close to the default PPOv2 configuration4 scaled up to 256 parallel environments . This has two effects : First , it ensures that our set of trained models contains good models ( as verified by performance statistics in the corresponding results ) . Second , it guarantees that we have models that have different combinations of potentially interacting choices . We consider two different analyses for each choice ( e.g , for advantage_estimator ( C6 ) ) : Conditional 95th percentile : For each potential value of that choice ( e.g. , advantage_estimator ( C6 ) = N-Step ) , we look at the performance distribution of sampled configurations with that value . We report the 95th percentile of the performance as well as a confidence interval based on a binomial approximation 5 . Intuitively , this corresponds to a robust estimate of the performance one can expect if all other choices in the group were tuned with random search and a limited budget of roughly 20 hyperparameter configurations . Distribution of choice within top 5 % configurations . We further consider for each choice the distribution of values among the top 5 % configurations trained in that experiment . The reasoning is as follows : By design of the experiment , values for each choice are distributed uniformly at random . Thus , if certain values are over-represented in the top models , this indicates that the specific choice is important in guaranteeing good performance . Performance measures . We employ the following way to compute performance : For each choice configuration , we train 3 models with independent random seeds where each model is trained for one million ( Hopper , HalfCheetah , Walker2d ) or two million environment steps ( Ant , Humanoid ) . We evaluate trained policies every hundred thousand steps by freezing the policy and computing the average undiscounted episode return of 100 episodes ( with the stochastic policy ) . We then average these score to obtain a single performance score of the seed which is proportional to the area under the learning curve . This ensures we assign higher scores to agents that learn quickly . The performance score of a hyperparameter configuration is finally set to the median performance score across the 3 seeds . This reduces the impact of training noise , i.e. , that certain seeds of the same configuration may train much better than others . Robustness of results . While we take 3 random seeds to compute the performance measure for a single choice configuration , it is important to note that all the experimental results reported in this paper are based on more than 3 random seeds : The reported conditional 95th percentile and distribution of choice within top 5 % configurations are computed based upon the performance of hundreds of choice configurations . Furthermore , we also report confidence intervals for the conditional 95th percentile . 3Exact details for the different experiments are provided in Appendices D - K. 4https : //github.com/openai/baselines/blob/master/baselines/ppo2/ defaults.py 5We compute confidence intervals with a significance level of α = 5 % as follows : We find il = icdf ( α 2 ) and ih = icdf ( 1− α 2 ) where icdf is the inverse cumulative density function of a binomial distribution with p = 0.95 ( as we consider the 95th percentile ) and the number of draws equals the number of samples . We then report the ilth and ihth highest scores as the confidence interval . 3 EXPERIMENTS . We run experiments for eight thematic groups : Policy Losses ( Sec . 3.1 ) , Networks architecture ( Sec . 3.2 ) , Normalization and clipping ( Sec . 3.3 ) , Advantage Estimation ( Sec . 3.4 ) , Training setup ( Sec . 3.5 ) , Timesteps handling ( Sec . 3.6 ) , Optimizers ( Sec . 3.7 ) , and Regularization ( Sec . 3.8 ) . For each group , we provide a full experimental design and full experimental plots in Appendices D - K so that the reader can draw their own conclusions from the experimental results . Moreover , the raw data from all training runs and a script used to generate all plots for this paper can be found online6 . In the following sections , we provide short descriptions of the experiments , our interpretation of the results , as well as practical recommendations for agent training for continuous control .

Summary: This paper carries out a large-scale study for understanding of on-policy deep actor-critic. The study looks into a large choices of many implementation settings and design decisions, and investigate their impact on the task performance. The evaluations are done with 250000 RL agents on 5 different continuous control tasks. For each evaluation category, there is a finding summary that provides practical recommendations.