What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study

1 INTRODUCTION . Deep reinforcement learning ( RL ) has seen increased interest in recent years due to its ability to have neural-network-based agents learn to act in environments through interactions . For continuous control tasks , on-policy algorithms such as REINFORCE [ 2 ] , TRPO [ 10 ] , A3C [ 14 ] , PPO [ 17 ] and off-policy algorithms such as DDPG [ 13 ] and SAC [ 21 ] have enabled successful applications such as quadrupedal locomotion [ 20 ] , self-driving [ 30 ] or dexterous in-hand manipulation [ 20 , 25 , 32 ] . Many of these papers investigate in depth different algorithmic ideas , for example different loss functions and learning paradigms . Yet , it is less visible that behind successful experiments in deep RL there are complicated code bases that contain a large number of low- and high-level design decisions that are usually not discussed in research papers . While one may assume that such “ choices ” do not matter , there is evidence that they are in fact crucial for or even driving good performance [ 27 ] . While there are open-source implementations available that can be used by practitioners , this is still unsatisfactory : Research publications often contain one-to-one comparisons of different algorithmic ideas based on implementations in different code bases . This makes it impossible to assess whether improvements are due to the underlying algorithmic idea or due to the implementation . In fact , it is hard to assess the performance of high-level algorithmic ideas without an understanding of lower-level choices as performance may strongly depend on the tuning of hyperparameters and implementationlevel details . Overall , this makes it hard to attribute progress in reinforcement learning and slows down further research [ 15 , 22 , 27 ] . Our contributions . Our key goal in this paper is to investigate such lower level choices in depth and to understand their impact on final agent performance . Hence , as our key contributions , we ( 1 ) implement > 50 choices in a unified on-policy deep actor-critic implementation1 , ( 2 ) conducted a large-scale ( more than 250 ’ 000 agents trained ) experimental study that covers different aspects of the training process , and ( 3 ) analyze the experimental results to provide practical insights and recommendations for the training of on-policy deep actor-critic RL agents . 1The implementation is available at https : //github.com/google-research/seed_rl . Most surprising finding . While many of our experimental findings confirm common RL practices , some of them are quite surprising , e.g . the policy initialization scheme significantly influences the performance while it is rarely even mentioned in RL publications . In particular , we have found that initializing the network so that the initial action distribution has zero mean , a rather low standard deviation and is independent of the observation significantly improves the training speed ( Sec . 3.2 ) . Paper outline . The rest of of this paper is structured as follows : We describe our experimental setup and performance metrics used in Sec . 2 . Then , in Sec . 3 we present and analyse the experimental results and finish with related work in Sec . 4 and conclusions in Sec . 5 . The appendices contain the detailed description of all design choices we experiment with ( App . B ) , default hyperparameters ( App . C ) and the raw experimental results ( App . D - K ) . 2 STUDY DESIGN . Considered setting . In this paper , we consider the setting of on-policy deep actor-critic reinforcement learning for continuous control . We define on-policy learning in the following loose sense : We consider policy iteration algorithms that iterate between generating experience using the current policy and using that experience to improve the policy . This is the standard modus operandi of algorithms usually considered on-policy such as PPO [ 17 ] . However , we note that algorithms often perform several model updates and thus may operate technically on off-policy data within a single policy improvement iteration . As benchmark environments , we consider five widely used continuous control environments from OpenAI Gym [ 12 ] of varying complexity : Hopper-v1 , Walker2d-v1 , HalfCheetah-v1 , Ant-v1 , and Humanoid-v1 2 . Unified on-policy deep actor-critic gradient algorithm . We took the following approach to create a highly configurable unified on-policy deep actor-critic gradient algorithm with as many choices as possible : 1 . We researched prior work and popular code bases to make a list of commonly used choices , i.e. , different loss functions ( both for value functions and policies ) , architectural choices such as initialization methods , heuristic tricks such as gradient clipping and all their corresponding hyperparameters . 2 . Based on this , we implemented a single , unified on-policy deep actor-critic agent and corresponding training protocol starting from the SEED RL code base [ 28 ] . Whenever we were faced with implementation decisions that required us to take decisions that could not be clearly motivated or had alternative solutions , we further added such decisions as additional choices . 3 . We verified that when all choices are selected as in the PPO implementation from OpenAI baselines , we obtain similar performance as reported in the PPO paper [ 17 ] . We chose PPO because it is probably the most commonly used on-policy deep actor-critic RL algorithm at the moment . The resulting agent implementation is detailed in Appendix B . The key property is that the implementation exposes all choices as configuration options in an unified manner . For convenience , we mark each of the choice in this paper with a number ( e.g. , C1 ) and a fixed name ( e.g . num_envs ( C1 ) ) that can be easily used to find a description of the choice in Appendix B . Difficulty of investigating choices . The primary goal of this paper is to understand how the different choices affect the final performance of an agent and to derive recommendations for these choices . There are two key reasons why this is challenging : First , we are mainly interested in insights on choices for good hyperparameter configurations . Yet , if all choices are sampled randomly , the performance is very bad and little ( if any ) training progress is made . This may be explained by the presence of sub-optimal settings ( e.g. , hyperparameters of the wrong scale ) that prohibit learning at all . If there are many choices , the probability of such failure increases exponentially . 2It has been noticed that the version of the Mujoco physics simulator [ 5 ] can slightly influence the behaviour of some of the environments — https : //github.com/openai/gym/issues/1541 . We used Mujoco 2.0 in our experiments . Second , many choices may have strong interactions with other related choices , for example the learning rate and the minibatch size . This means that such choices need to be tuned together and experiments where only a single choice is varied but interacting choices are kept fixed may lead to misleading conclusions . Basic experimental design . To address these issues , we design a series of experiments as follows : We create groups of choices around thematic groups where we suspect interactions between different choices , for example we group together all choices related to neural network architecture . We also include Adam learning rate ( C24 ) in all of the groups as we suspect that it may interact with many other choices . Then , in each experiment , we train a large number of models where we randomly sample the choices within the corresponding group 3 . All other settings ( for choices not in the group ) are set to settings of a competitive base configuration ( detailed in Appendix C ) that is close to the default PPOv2 configuration4 scaled up to 256 parallel environments . This has two effects : First , it ensures that our set of trained models contains good models ( as verified by performance statistics in the corresponding results ) . Second , it guarantees that we have models that have different combinations of potentially interacting choices . We consider two different analyses for each choice ( e.g , for advantage_estimator ( C6 ) ) : Conditional 95th percentile : For each potential value of that choice ( e.g. , advantage_estimator ( C6 ) = N-Step ) , we look at the performance distribution of sampled configurations with that value . We report the 95th percentile of the performance as well as a confidence interval based on a binomial approximation 5 . Intuitively , this corresponds to a robust estimate of the performance one can expect if all other choices in the group were tuned with random search and a limited budget of roughly 20 hyperparameter configurations . Distribution of choice within top 5 % configurations . We further consider for each choice the distribution of values among the top 5 % configurations trained in that experiment . The reasoning is as follows : By design of the experiment , values for each choice are distributed uniformly at random . Thus , if certain values are over-represented in the top models , this indicates that the specific choice is important in guaranteeing good performance . Performance measures . We employ the following way to compute performance : For each choice configuration , we train 3 models with independent random seeds where each model is trained for one million ( Hopper , HalfCheetah , Walker2d ) or two million environment steps ( Ant , Humanoid ) . We evaluate trained policies every hundred thousand steps by freezing the policy and computing the average undiscounted episode return of 100 episodes ( with the stochastic policy ) . We then average these score to obtain a single performance score of the seed which is proportional to the area under the learning curve . This ensures we assign higher scores to agents that learn quickly . The performance score of a hyperparameter configuration is finally set to the median performance score across the 3 seeds . This reduces the impact of training noise , i.e. , that certain seeds of the same configuration may train much better than others . Robustness of results . While we take 3 random seeds to compute the performance measure for a single choice configuration , it is important to note that all the experimental results reported in this paper are based on more than 3 random seeds : The reported conditional 95th percentile and distribution of choice within top 5 % configurations are computed based upon the performance of hundreds of choice configurations . Furthermore , we also report confidence intervals for the conditional 95th percentile . 3Exact details for the different experiments are provided in Appendices D - K. 4https : //github.com/openai/baselines/blob/master/baselines/ppo2/ defaults.py 5We compute confidence intervals with a significance level of α = 5 % as follows : We find il = icdf ( α 2 ) and ih = icdf ( 1− α 2 ) where icdf is the inverse cumulative density function of a binomial distribution with p = 0.95 ( as we consider the 95th percentile ) and the number of draws equals the number of samples . We then report the ilth and ihth highest scores as the confidence interval . 3 EXPERIMENTS . We run experiments for eight thematic groups : Policy Losses ( Sec . 3.1 ) , Networks architecture ( Sec . 3.2 ) , Normalization and clipping ( Sec . 3.3 ) , Advantage Estimation ( Sec . 3.4 ) , Training setup ( Sec . 3.5 ) , Timesteps handling ( Sec . 3.6 ) , Optimizers ( Sec . 3.7 ) , and Regularization ( Sec . 3.8 ) . For each group , we provide a full experimental design and full experimental plots in Appendices D - K so that the reader can draw their own conclusions from the experimental results . Moreover , the raw data from all training runs and a script used to generate all plots for this paper can be found online6 . In the following sections , we provide short descriptions of the experiments , our interpretation of the results , as well as practical recommendations for agent training for continuous control .

Summary: The paper presents an empirical evaluation of many algorithmic choices made in the implementations of on-policy actor-critic algorithms in deep reinforcement learning (RL). The authors group those choices in clusters in which they expect some interactions. For each cluster, they test sets of randomly made choices while assuming that choices outside a cluster are set to competitive default values. Based on those experimental results, the authors formulate recommendations about how to make those choices for each cluster.