What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study
Authors: Marcin Andrychowicz
Anton Raichuk
Piotr Stańczyk
Manu Orsini
Sertan Girgin
Raphaël Marinier
Leonard Hussenot
Matthieu Geist
Olivier Pietquin
Marcin Michalski
Sylvain Gelly
Olivier Bachem
Venue: ICLR 2021 Oral (2021)
Keywords: Reinforcement learning
continuous control

Abstract: In recent years, reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``"choices" in a unified on-policy deep actor-critic framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for the training of on-policy deep actor-critic RL agents.

Published as a conference paper at ICLR 2021
WHAT MATTERS FOR ON-POLICY DEEP ACTOR- CRITIC METHODS? A LARGE-SCALE STUDY
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, Olivier Bachem
Google Research, Brain Team
ABSTRACT
In recent years, reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their sate-of-the-ee art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RI and slows down overall progress [27]. As a step towards filling that gap, we implement >50 such "choices" in a unified on-policy deep actor-critic framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250' 5000000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for the training of on-policy deep actor-critic RL agents.
INTRODUCTION
Deep reinforcement learning (RL) has seen increased interest in recent years due to its ability to have neural-nettwrkk-based agents learn in environments through interactions. For continuous control tasks, on-policy algorithms such as REINFORCE [2], TRPO [10], A3C [14), PPO [17] and off- off-policy algorithms such as DDPG [13] and SAC [21] have enabled successful applications such as quadrupedal locomotion [20]. self-driving [30] or dexterous in-hand manipulation [20, 25, 32].
Many of these papers investigate in depth different algorithmic ideas, for example different loss functions and learning paradigms. Yet, it is less visible that behind successful experiments in deep RL there are complicated code bases that contain large number of low- and high-level design decisions that are usually not discussed in research papers. While one may assume that such "choices" do not matter, there evidence that they are fact crucial for or even driving good performance [27].
While there are open-source implementations available that can be used by practitioners, this is still unsatisfactory: Research publications often contain one-to-one comparisons of different algorithmic ideas based implementations in different code bases. This makes it impossible to assess whether improvements are due to the underlying algorithmic idea or due to the implementation. In fact, is hard to assess the performance of high-level algorithmic ideas without understanding of lower-level choices as performance may strongly depend on the tuning of hyperparameters and implementation level details. Overall, this makes hard to attribute progress in reinforcement learning and slows down further research [15, 22, 27].
Our contributions. Our key goal in this paper is investigate such lower level choices in depth and to understand their impact on final agent performance. Hence, as our key contributions, we (I) implement >50 choices
unified on-policy deep actor-critic implementation's (2) conducted large-scale (more than 250'000 000 agents trained) experimental study that covers different aspects of the training process, and (3) analyze the experimental results to provide practical insights and recommendations for the training of on-policy deep actor-critic RL agents.
'The implementation is available https: / //i tig.t.......... com/goog e- e-research caaccch/seerrree

---

Published as a conference paper at ICLR 2021
Most surprising finding. While many of our experimental findings confirm common RL practices, some of them are quite surprising, e.g. the policy initialization scheme significantly influences the performance while it is rarely even mentioned in RL publications. In particular, have found that initializing the network so that the initial action distribution has zero mean, a rather low standard deviation and is independent of the observation significantly improves the training speed (Sec. 3.2).
We describe our experimental setup and performance metrics used in Sec. 2. Then, in Sec. we present and analyse the experimental results and finish with related work in Sec. 4 and conclusions in Sec. 5. The appendices contain the detailed description of all design choices we experiment with (App. B), default hyperparameters (App. C) and the raw experimental results (App. D K).
Paper outline. The rest of of this paper is structured as follows:
2 STUDY DESIGN
Considered setting. In this paper, we consider the setting of on-policy deep actor-critic reinforce- ment learning for continuous control. We define on-policy learning in the following loose sense: We consider policy iteration algorithms that iterate between generating experience using the current policy and using that experience to improve the policy. This is the standard modus operandi of algorithms usually considered on-policy such as PPO [17]. However, perform several model updates and thus may operate technically on off-policy data within a single policy improvement iteration. As benchmark environments, we consider five widely used continuous control environments from OpenAl Gym [12] of varying complexity: Hopper-v1. Walker2d-v1, HalfChectah 11, Ant-v1, and Humanoid-vl
note that algorithms often
Unified on-policy deep actor-critic gradient algorithm. We took the following approach to create highly configurable unified on-policy deep actor-critic gradient algorithm with as many choices as possible:
1. We researched prior work and popular code bases to make a list of commonly used choices, ie.. different loss functions (both for value functions and policies), architectural choices such as initialization methods, heuristic tricks such as gradient clipping and all their corresponding hyperparameters.
2. Based on this, we implemented a single, unified on-policy deep actor-critic agent and correspond- ing training protocol starting from the SEED RI code base [28]. Whenever we were faced with implementation decisions that required to take decisions that could not be clearly motivated or had alternative solutions, we further added such decisions as additional choices.
We verified that when all choices are selected as in the PPO implementation from OpenAl baselines, we obtain similar performance as reported in the PPO paper [17]. We chose PPO because is probably the most commonly used on-policy deep actor-critic RL algorithm at the moment.
The resulting agent implementation is detailed in Appendix B. The key property is that the implemen- tation exposes all choices configuration options an unified manner. For convenience, we mark each of the choice in this paper with a number (e.g.. C1) and fixed name (e.g. num envs (C1)) that can be easily used find a description of the choice in Appendix B.
Difficulty of investigating choices. The primary goal of this paper is to understand how the different choices affect the final performance an agent and to derive recommendations for these choices. There are two key reasons why this is challenging:
First, we are mainly interested in insights on choices for good hyperparameter configurations. Yet, all choices are sampled randomly, the performance is very bad and little (if any) training progress is made. This may be explained by the presence of sub-optimal settings (c.g.. hyperparameters of the wrong scale) that prohibit learning at all. If there are many choices, the probability of such failure increases exponentially.
It has been noticed that the version of the Mujoco physics simulator [5] can slightly influence the behaviour of some of the environments - https: thu.. com/ opena /gmm/ ssues/ 1.11, We used Mujeco 2.0 in our experiments.
2

---

Published as a conference paper at ICLR 2021
Second, many choices may have strong interactions with other related choices, for example the learning rate and the minibatch size. This means that such choices need to be tuned together and experiments where only single choice is varied but interacting choices are kept fixed may lead to misleading conclusions.
Basic experimental design. To address these issues, we design series of experiments as follows: We create groups choices around thematic groups where we suspect interactions between different choices, for example we group together all choices related to neural network architecture. We also include Adam learn learning rate (C24) ) all the groups as we suspect that it may interact with many other choices.
Then, in each experiment, we train a large number of models where we randomly sample the choices within the corresponding group 9, All other settings (for choices not in the group) are set settings of a competitive base configuration (detailed Appendix C) that is close to the default PPOv2 configuration" scaled up 256 parallel environments. This has two effects: First, ensures that set of trained models contains good models (as verified by performance statistics in the corresponding results). Second, it guarantees that have models that have different combinations of potentially interacting choices.
We consider different analyses for each choice (e.g, for advantage_estiima
(C6) ):
Conditional 95th percentile: For each potential value of that choice (e.g., advantage _etet__stttte imator (C6) N- St ep), look the performance distribution of sampled configurations with that value. We report the 95th percentile of the performance as well as confidence interval based on a binomial approximation" 5. Intuitively, this corresponds to a robust estimate of the performance one can expect if all other choices in the group were tuned with random search and limited budget of roughly 20 hyperparameter configurations.
Distribution of choice within top 5% configurations. We further consider for each choice the distribution of values among the top 5% configurations trained in that experiment. The reasoning as follows: By design of the experiment, values for each choice are distributed uniformly at random. Thus, if certain values are over-represented in the top models, this indicates that the specific choice important in guaranteeing good performance.
Performance measures. We employ the following way compute performance: For each choice configuration, we train 3 models with independent random seeds where each model is trained for one million (Hopper. HalfCheetah, Walker2d) or million environment steps (Ant, Humanoid). We evaluate trained policies every hundred thousand steps by freezing the policy and computing the average undiscounted episode return of 100 episodes (with the stochastic policy). We then average these score to obtain single performance score of the seed which is proportional to the area under the learning curve. This ensures we assign higher scores to agents that learn quickly. The performance score of a hyperparameter configuration is finally set to the median performance score across the 3 seeds. This reduces the impact of training noise, i.c., that certain seeds the same configuration may train much better than others.
Robustness of results. While take 3 random seeds to compute the performance measure for a single choice configuration, it is important to note that all the experimental results reported in this paper are based on more than 3 random seeds: The reported conditional 95th percentile and distribution of choice within top 5% configurations are computed based upon the performance of hundreds of choice configurations. Furthermore, we also report confidence intervals for the conditional 95th percentile.
'Exact details for the different experiments are provided in Appendices D- K.
'https: //gi //ggttguhs thub. com opena i /ww./aaeeeeeeeeeeeeee.aeeeeaaeae http://lllbbmm/t///teeeee/aee//e http////te/bbbellemmmeee wtwp://bbbaeenneepppee../..
de faafuutss... py
'We compute confidence intervals with significance level of a 5% as follows: We find i icdf (9) where iedf is the inverse cumulative density function of a binomial distribution with p 0.95 (as consider the 95th percentile) and the number of draws equals the number of samples. We then report the i,th and th highest scores as the confidence interval.
and ig icdf (1
3

---

Published as a conference paper at ICLR 2021
3 EXPERIMENTS
We run experiments for eight thematic groups: Policy Losses (Sec. 3.1), Networks architecture (Sec. 3.2), Normalization and clipping (Sec. 3.3), Advantage Estimation (Sec. 3.4), Training senup (Sec. 3.5), Timesteps handling (Sec. 3.6), Optimizers (Sec. 3.7), and Regularization (Sec. 3.8). For each group, we provide full experimental design and full experimental plots in Appendices so that the reader draw their conclusions from the experimental results. Morcover, the raw data from all training runs and a script used to generate all plots for this paper can be found online In the following sections, provide short descriptions of the experiments, our interpretation of the results, as well as practical recommendations for agent training for continuous control.
3.1 POLICY LOSSES (BASED ON THE RESULTS IN APPENDIX D)
Study description. We investigate different policy losses (C14): vanilla policy gradient (PG), V- trace [19], PPO [17], AWR [33], V-MPO" [34] and the limiting case of AWR (3 - and V-MPO (cn + 0) which we call Repeat Positive Advantages (RPA) as it is equivalent to the negative log- probability of actions with positive advantages. See App. B.3 for a detailed description of the different losses. We further sweep the hyperparameters of each of the losses (C15, C16, C1 81 C17, C1 9), the learning rate (C2 4) and the number passes over the data (C3).
The goal of this study is to better understand the importance of the policy loss function in the on-policy deep actor-critic setting considered this paper. The goal is not to provide a general statement that one of the losses is better than the others some of them were specifically designed for other settings (e.g., s.. the V-trace loss is targeted at near-on-policy data a distributed setting).
Hopper-v1
Humanoid-vi
Walker2d-v1
HalfCheetah-v1
Ant-v1
3002
1300
2500
1500
1200
1250
1.000
600
1500
1500
750
600
1000
250
500
200
500
AWA
107
2000
10
PRO
RPA
PrO
P5
PPO
VTTroo
Figure 1: Comparison of different policy losses (C14).
Interpretation. Fig. shows the 95-th percentile of the average policy score during training for different policy losses (C14). 4). We observe that PPO performs better than the other losses on 4 out of 5 environments and is one of the top performing losses on HalfCheetah. As we randomly sample the loss specific hyperparameters in this analysis, one might argue that our approach favours choices that are not too sensitive to hyperparameters. At the same time, there might losses that are sensitive to their hyperparameters but for which good settings may be easily found. Fig. 5 shows that even we condition on choosing the optimal loss hyperparameters for each loss PPO still outperforms the other losses the two hardest tasks Humanoid and Ant and is one of the top performing losses on the other 3 tasks. Moreover, we show the empirical cumulative density functions of agent performance conditioned on the policy loss used in Fig. 4.
Perhaps unsurprisingly, PG and V-trace perform worse on all tasks. This is likely caused by their inability handle data that becomes off-policy in one iteration, either due multiple passes (C3)
'https:/ http::::///tttubb.. www.ttaub.conngooooto http://c//oooooeeeeeeeeeeeeeeeee seed r ht/blcb/mastete hwtps//aaterrreeeeeee htteeermmjjjoocc whatt....
matters in on.eennppoo. policy rl. ipynb
"e used the V-MPO policy loss without the decoupled KL constraint as we investigate the effects of
different policy regularizers separately in Sec. 3.8.
the temperature (C1 and the weight clipping coefficient uhax (C17). 7). We only condition on 3 which is more important.
AWR loss has two hyperparameters
"hese two tasks were not included in the original PPO paper [17] the hyperparameters were not
tuned for them.
4

---

Published as a conference paper at ICLR 2021
over experience (which can be seen in Fig. 14) a large experience buffer (C2) in relation to the batch size (C4). While V-Trace contains an off-policy correction, was designed for "slightly" off off-policy experience arising in ansynchronous, distributed setups and becomes more and more biased as experience becomes more off-policy. Overall, these results show that trust-region optimization (preventing the current policy from diverging too much from the behavioral one) which is present in all the other policy losses is crucial for good sample complexity.
For PPO and its clipping threshold e (C1 6), we further observe that e 0.2 and 0.3 perform reasonably well in all environments but that lower (e 0.1) or higher (e = 0.5) values give better performance on some of the environments (See Fig. 10 and Fig. 32).
Recommendation. Use the PPO policy loss. Start with the clipping threshold set to 0.25 but also try lower and higher values if possible.
3.2 NETWORKS ARCHITECTURE (BASED ON THE RESULTS IN APPENDIX E)
Study description. We investigate the impact of differences in the policy and value function neural network architectures. We consider choices related to the network structure and size (C47, C48, C49, C50, C51, C52, C52), activation functions (C55), and initialization of network weights (C56, C57, C58). We further include choices related to the standard deviation of actions (C59, C60, C61, C62) and transformations of sampled actions (C63).
Interpretation. Separate value and policy networks (C47) appear to lead to better performance on four out of five environments (Fig. 15). To avoid analyzing the other choices based on bad models, we thus focus for the rest of this experiment only on agents with separate value and policy networks. Regarding network sizes, the optimal width of the policy MLP depends on the complexity of the environment (Fig. 18) and too low or too high values can cause significant drop in performance while for the value function there seems to be no downside in using wider networks (Fig. 21). Moreover, on some environments it is beneficial to make the value network wider than the policy one, e.g. on HalfCheetah the best results are achieved with 16 32 units per layer the policy network and 256 in the value network. Two hidden layers appear to work well for policy (Fig. 22) and value networks (Fig. 20) in all tested environments. As for activation functions, we observe that tanh activations perform best and relu ll worst. (Fig. 30).
Interestingly, the initial policy appears to have a surprisingly high impact on the training performance. The key recipe is to initialize the policy at the beginning of training that the action distribution is centered around 0 regardless the observation and has a rather small standard deviation. This can be achieved by initializing the policy MLP with smaller weights in the last layer (c57, Fig. 24, this alone boosts the performance on Humanoid by 66%) so that the initial action distribution is almost independent of the observation and by introducing an offset in the standard deviation of actions (C61). Fig. 2 shows that the performance is very sensitive to the initial action standard deviation with 0.5 performing best on all environments except Hopper where higher values perform better.
Hopper-v1
Humanoid-v1
Walker2d-v1
HalfCheetah-v1
Ant-v1
3002
800
2500
2500
2000
2000
2000
1500
1500
1002
10000
1000
200
200
500
500
10
10
Figure 2: Comparison of different initial standard deviations of actions (C61).
Fig. 17 compares two approaches to transform unbounded sampled actions into the bounded [-1, 1] domain expected by the environment (C63): clipping and applying tanh function. tanh performs slightly better overall (in particular it improves the performance on HalfCheetah by 30% ). Comparing
All environments expect normalized actions in [-1,1].
5

---

Published as a conference paper at ICLR 2021
Fig. 17 and Fig. 2 suggests that the difference might be mostly caused by the decreased magnitude of initial actions".
Other choices appear to be less important: The scale of the last layer initialization matters much less for the value MLP (C58) than for the policy MLP (Fig. 19). Apart from the last layer scaling, the network initialization scheme (C56) does not matter too much (Fig. 27). Only he_normal 1 and rm [7] appear to be suboptimal choices with the other options performing very similarly. There also appears to be clear benefits if the standard deviation of the policy learned for each state (i.e. outputted by the policy network) or once globally for all states (C59, 59, Fig. 23). For the transformation of policy output into action standard deviation (C60), softplus and exponentiation perform very similarly (Fig. 25). Finally, the minimum action standard deviation (C62) seems to matter little, if it not set too large 30).
he_uni form
Recommendation Initialize the last policy layer with 100x smaller weights. Use softplu lus to transform network output into action standard deviation and add a (negative) offset to its input to decrease the initial standard deviation of actions. Tune this offset if possible. Use tanh both as the activation function (if the networks are not too deep) and to transform the samples from the normal distribution to the bounded action space. Use a wide value MLP (no layers shared with the policy) but tune the policy width (it might need to be narrower than the value MLP).
3.3 NORMALIZATION AND CLIPPING (BASED ON THE RESULTS IN APPENDIX F)
Study description. We investigate the impact of different normalization techniques: observation normalization (C64), value function normalization (C66), per-minibatch advantage normalization (C67), as well as gradient (C68) and observation (C65) clipping.
Interpretation. Input normalization (C64) is crucial for good performance on all environments apart from Hopper (Fig. 33). Quite surprisingly, value function normalization (C66) also influences the performance very strongly - it is crucial for good performance on HalfCheetah and Humanoid, helps slightly on Hopper and Ant and significantly hurts the performance on Walker2d (Fig. 37). We are not sure why the value function scale matters that much but suspect that it affects the performance by changing the speed of the value function fitting. 13 contrast to observation and value function normalization, per-minibatch advantage normalization (C67) seems not to affect the performance too much (Fig. 35). Similarly, we have found little evidence that clipping normalized observations (C65) helps (Fig. 38) but it might be worth using if there is risk of extremely high observations due to simulator divergence. Finally, gradient clipping (C 68) provides a small performance boost with the exact clipping threshold making little difference (Fig. 34).
Recommendation. Always use observation normalization and check if value function normalization improves performance. Gradient clipping might slightly help but is of secondary importance.
3.4 ADVANTAGE ESTIMATION (BASED ON THE RESULTS IN APPENDIX G)
Study description. We compare the most commonly used advantage estimators (C6): N-step [3], GAE [9] and V-trace [19] and their hyperparameters (C7, C8, C9, C10). We also experiment with applying PPO-style pessimistic clipping (C13) to the value loss (present in the original PPO implementation but not mentioned in the PPO paper [17]) and using Huber loss [1] instead of MSE for value learning (C11, 1, C1 2).). Moreover, we varied the number of parallel environments used (C1) as it changes the length of the experience fragments collected in each step.
Interpretation. GAE and V-trace appear to perform better than N-step returns (Fig. 44 and 40) which indicates that is beneficial to combine the value estimators from multiple timesteps. We have not
11 "tanh can also potentially perform better with entropy regularization (not used in this experiment) as it bounds the maximum possible policy entropy.
We noticed that some of the training runs with exponentiation resulted in NaNs but clipping the exponent solves this issue (See Sec. B.8 for the details).
13 Another explanation could be the interaction between the value function normalization and PPO-style value clipping (C13 3). We have, however, disabled the value clipping this experiment avoid this interaction. The disabling of the value clipping could also explain why our conclusions are different from [27] where a form of value normalization improved the performance on Walker.
We only applied clipping if input normalization was enabled.

---

Published as a conference paper at ICLR 2021
found significant performance difference between GAE and V-trace in experiments. À 0.9 (C8, C9) performed well regardless of whether GAE (Fig. 45) or V-trace (Fig. 49) was used on all tasks but tuning this value per environment may lead to modest performance gains. We have found that PPO-style value loss clipping (C13) hurts the performance regardless of the clipping threshold (Fig. 43). Similarly, the Huber loss (c11) 1) performed worse than MSE in all environments (Fig. 42) regardless of the value of the threshold (C12) used (Fig. 48).
Recommendation. Use GAE with X 0.9 but neither Huber loss nor PPO-style value loss clipping.
3.5 TRAINING SETUP (BASED ON THE RESULTS IN APPENDIX H)
Study description. We investigate choices related to the data collection and minibatch handling: the number of parallel environments used (C1), the number of transitions gathered in each iteration (C2), the number of passes over the data (C3), minibatch size (C4) and how the data is split into minibatches (C5).
For the last choice, in addition to standard choices, we also consider a new small modification of the original PPO approach: The original PPO implementation splits the data in each policy iteration step into individual transitions and then randomly assigns them to minibatches (C5). This makes it impossible to compute advantages as the temporal structure is broken. Therefore, the advantages are computed once at the beginning of each policy iteration step and then used in minibatch policy and value function optimization. This results in higher diversity of data in each minibatch at the cost of using slightly stale advantage estimations. As remedy to this problem, we propose to recompute the advantages at the beginning of each pass over the data instead of just once per iteration.
Results. Unsurprisingly, going over the experience multiple times appears to be crucial for good sam- ple complexity (Fig. 54). Often, this is computationally cheap due to the simple models considered, in particular machines with accelerators such as GPUs and TPUs. As we increase the number of parallel environments (C1), performance decreases sharply on some of the environments (Fig. 55). This is likely caused by shortened experience chunks (See Sec. B.1 for the detailed description of the data collection process) and earlier value bootstrapping. Despite that, training with more environments usually leads to faster training in wall-clock time if enough CPU cores are available. Increasing the batch size (C4) does not appear to hurt the sample complexity in the range we tested (Fig. 57) which suggests that it should be increased for faster iteration speed. On the other hand, the number of transitions gathered in each iteration (C2) influences the performance quite significantly (Fig. 52). Finally, we compare different ways to handle minibatches (See App. B. for the detailed description of different variants) in Fig. 53 and 58. The plots suggest that stale advantages can in fact hurt performance and that recomputing them at the beginning of each pass at least partially mitigates the problem and performs best among all variants.
Recommendation. Go over experience multiple times. Shuffle individual transitions before assigning them to minibatches and recompute advantages once per data pass (See App. B.1 for the details). For faster wall-clock time training use many parallel environments and increase the batch size (both might hurt the sample complexity). Tune the number of transitions in each iteration (C2) if possible.
3.6 TIMESTEPS HANDLING (BASED ON THE RESULTS IN APPENDIX I)
Study description. We investigate choices related to the handling of timesteps: discount fac- 16 (C20), frame skip (C21), 1). and how episode termination due to timestep limits are handled (C22).
The latter relates to a technical difficulty explained in App. B.4 where one assumes for the algorithm an infinite time horizon but then trains using a finite time horizon [16].
Interpretation. Fig. 60 shows that the performance depends heavily on the discount factor y (C20) with y = 0.99 performing reasonably well in all environments. Skipping every other frame (C21) improves the performance on 2 out of environments (Fig. 61). Proper handling of episodes abandoned due to the timestep limit seems not to affect the performance (C22, Fig. 62) which
15 This is consistent with prior work [27].
While the discount factor is sometimes treated as part of the environment, we assume that the real goal is maximize undiscounted returs and the discount factor a part of the algorithm which makes learning casier.
7

---

Published as a conference paper at ICLR 2021
is probably caused by the fact that the timestep limit is quite high (1000 transitions) in all the environments we considered.
Recommendation. Discount factor y is of the most important hyperparameters and should be tuned per environment (start with 7 = 0.99). Try frame skip if possible. There is no need to handle environments step limits in special way for large step limits.
3.7 OPTIMIZERS (BASED ON THE RESULTS IN APPENDIX J)
Study description. We investigate two gradient-based optimizers commonly used in RL: (C23)- Adam [8] and RMSprop - as well as their hyperparameters (C24, C25, C26, C27, C2B, C29, C30) and linear learning rate decay schedule (C31).
Interpretation. The differences in performance between the optimizers (C23) appear to be rather small with optimizer consistently outperforming the other across environments (Fig. 66). Unsur- prisingly, the learning rate influences the performance very strongly (Fig. 69) with the default value of 0.0003 for Adam (C2 4) performing well on all tasks. Fig. 67 shows that Adam works better with momentum (C2 6). For RMSprop, momentum (C27) makes less difference (Fig. 71) but our results suggest that might slightly improve performance Whether the centered or uncentered version of RMSprop is used (C30) makes no difference (Fig. 70) and similarly we did not find any difference between different values of the c coefficients (C2 B, C29, Fig. 68 and 72). Linearly decaying the learning rate to O increases the performance on 4 out of 5 tasks but the gains are very small apart from Ant, where leads to 15% higher scores (Fig. 65).
Recommendation. Use Adam [8] optimizer with momentum 31 = 0.9 and a tuned learning rate (0.0003 is a safe default). Linearly decaying the learning rate may slightly improve performance but is of secondary importance.
3.8 REGULARIZATION (BASED ON THE RESULTS IN APPENDIX K)
Study description. We investigate different policy regularizers (C32), which can have either the form of a penalty (C33, e.g. bonus for higher entropy) or a soft constraint (C34, e.g. entropy should not be lower than some threshold) which is enforced with Lagrange multiplier. In particular, consider the following regularization terms: entropy (C40, C46), the Bullback-Leible divergence (KL) between a reference (0, 1) action distribution and the current policy (C37, 34 and the KL divergence and reverse KI. divergence between the current policy and the behavioral one (C35, C41, 6, C42), well the "decoupled" KI. divergence from [18, 34] (C38, C39, 444, C45).
Interpretation. We do not find evidence that regularization helps significantly on our environments with the exception of HalfCheetah on which all constraints (especially the entropy constraint) help (Fig. 76 and 77). However, the performance boost is largely independent on the constraint threshold (Fig. 83, 84, 87, 89, 90 and 91) which suggests that the effect is caused by the initial high strength of the penalty (before it gets adjusted) and not by the desired constraint. While it is surprising that regularization does not help at all (apart from HalfCheetah), we conjecture that regularization might be less important in our experiments because: (1) the PPO policy loss already enforces the trust region which makes KL penalties or constraints redundant; and (2) the careful policy initialization (Sce Sec. 3.2) is enough to guarantee good exploration making the entropy bonus or constraint redundant.
RELATED WORK
Islam et al. [15] and Henderson et al. [22] point out reproducibility issues in RL including the performance differences between different code bases, the importance of hyperparameter tuning and the high level of stochasticity due to random seeds. Tucker et al. [26] showed that the gains, which had been attributed to one of the recently proposed policy gradients improvements, were, in fact, caused by the implementation details. The most closely related work to ours is probably Engstrom et al. [27] where the authors investigate code-level improvements in the PPO [17] code base and conclude that they are responsible for the most of the performance difference between PPO
I'pprrtntly switching from momentum to momentum 0.9 increases the R.SSprpp step size by approxi- mately 10x and requires an appropriate adjustment to the learning rate (Fig. 74).
8

---

Published as a conference paper at ICLR 2021
and TRPO [10]. Our work is also similar to other large-scale studies done in other fields of Deep Learning, e.g. model-based RL [31], GANs [24], NLP [35], disentangled representations [23] and convolution network architectures [36].
CONCLUSIONS
In this paper, we investigated the importance of a broad set of high- and low-level choices that need to be made when designing and implementing on-policy deep actor-critic RL algorithms. Based on more than 250'000 experiments in five continuous control environments, different choices and provide practical recommendations. One of the surprising insights is that the initial action distribution plays an important role agent performance. We expect this be fruitful avenue for future research.
evaluate the impact of
REFERENCES
[]] Peter Huber. "Robust estimation of a location parameter". In: Breakthroughs in statistics. Springer, 1992, pp. 492-518.
[2] Ronald J Williams. "Simple statistical cradient-followigg algorithms for connectionist reinforcement leamiin".. In: Machine learning 8.3-4 (1992), pp. 229-256.
[3] Richard S Sutton, Andrew G Barto, al. Introduction reinforcement learning. Vol. 135, MIT press Cambridge, 1998.
[4] Brian D Zichart. "Modeling purposeful adaptive behavior with the principle of maximum causal entropy".
In: (2010).
[5] Emanuel Todorov, Tom Erez, Yuval Tassa. "Mujoco: A physics engine for model based control". In: 2012 JEEE/RSJ International Conference on Intellggett Robots and Systems. IEEE. 2012, pp. 5026-50333
[6] Volodymyr Mnih et al. "Playing atari with deep reinforcement learning". In: arXiv preprint
arXiv:1312.5602 (2013).
[7] Kaiming He et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification". In: Proceedings of the IEEE international conference on computer vision. 2015, pp. 1026- 1034.
[8] Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization", In: 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL: http: arrivv.. or g/ abs 1112/411114 6980.
[9] John Schulman al. arXiv preprint arXiv: 100.:v:55000000 02438 (2015).
continuous control using generalized advantage estimation". In:
[10] John Schulman al. "Trust region policy optimization". In: International conference on machine learning. 2015, pp. 1889-1897.
[11] Martin Abadi al. "Tensorflow: A system for large-scale machine Icarning", In: 12th {USENIX) Symposium on Operating Systems Design and Implementation (OODDI) 16). 2016, pp. 265-283.
[12] Greg Brockman "Openai gym"". arXiv preprint urXiv:1606.01544 (2016).
[13] Timothy P Lillicrap al. "Continuous control with deep reinforcement learning.. In: International
Conference on Learning Representations. 2016.
[14] Volodymyr Mnih al. "Asynchronous methods deep reinforcement leaminn".. In: International
conference on machine learning. 2016, pp. 1928-1937.
[15] Riashat Islam et al. "Reproducibilltty benchmarked deep reinforcement learning tasks continuous control". In: arXiv preprint arxiv: 04133 (2017).
[16] Fabio Pardo et al. "Time limits reinforcement learning". In: arXiv preprint urXiv:1712.003378 (2017).
[17] John Schulman et al. "Proximal policy optimization algorithms", In: arXiv preprint 17Xiv:1707.06347
(2017).
[18] Abbas Abdolmaleki et "Maximum a posteriori policy optimisation". Xrv: 06900006900006 (2018).
arXiv preprint
[19] Lasse Espeholt et al. "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures".
In: International Conference on Machine Learning. 2018, pp. 1406-1415.
[20] Tuomas Haarojj et al. "Soft actor-critic algorithms and applications".
arXiv preprint
A8/2 0.900. (2018).
[21] Tuomas Haarnoja et al. "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor", In: International Conference on Machine Learning. 2018, pp. 1861-1870.

---

Published as a conference paper at ICLR 2021
[22] Peter Henderson et al. "Deep reinforcement learning that matters". In: Thirty-Second AAAI Conference Artificial Intelligence. 2018.
[23]
Francesco Locatello al. "Challenging common assumptions in the unsupervised learning of disentan- gled representations". arXiv preprint 17X1::1811.12359 (2018).
[24] Mario Lucic al. "Are gans created equal? large-scale study". In: Advances in neural information processing systems. 2018, pp- 700-709.
[25]
Andrychowicz OpenAl et "Learning dexterous in-hand manipulation". In: arXiv preprint arXiv:1888.0000 001777788011771800 (2018).
[26] George Tucker "The mirage action-dependent baselines in reinforcement learning". arXiv preprint arXiv: 0000000300000 (2018).
[27] Logan Engstrom et al. "Implementation Matters in Deep RL: A Case Study on PPO and TRPO". In: International Conference on Learning Representations. 2019.
[28]
Lasse Espeholt et al. "SEED RL: Scalable and Efficient Deep-RL. RL with Accelerated Central Inference". In: arXiv preprint 1910 06599 (2019).
[29]
Michael Janner et al. "When to trust your model: Model-based policy optimization". In: Advances in Neural Information Processing Systems. 2019, pp. 22498-12500
[30] Alex Kendall ct al. "Learning to drive in day". In: 2019 International Conference on Robotics and Automation (ICRA). IEEE. 2019, pp. 8248-8254.
[31] Eric Langlois et al. "Benchmarking model-based reinforcement learning". In: arXiv preprint 17X1v:1907.02057 (2019).
[32] Ilge OpenAl et al. "Solving Rubik's Cube with Robot Hand". In: arXiv preprint arXiv11910.07133 (2019).
[33] Xue Bin Peng et al. "Advantage-Weigge Regression: Simple and Scalable Off-Policy Reinforcement Learning". In: arXiv preprint arXiv: 1910.00177 (2019).
[34] H Francis Song et al. "V-MPO: On-Policy Maximum Posteriori Policy Optimization for Discrete and Continuous Control", In: arXiv preprint arXiv:1909. 12238 (2019).
[35] Jared Kaplan al. "Scaling laws for neural language models". In: arXiv preprint arXiv:2001.8836) (2020).
[36] Ilija Radosavovic et al. "Designing Network Design Spaces". In: arXiv preprint arXiv:2003. 13678 (2020).
10


Reviews:
[{"review_id": "R9kNv-9434Z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an empirical evaluation of many algorithmic choices made in the implementations of on-policy actor-critic algorithms in deep reinforcement learning (RL). The authors group those choices in clusters in which they expect some interactions. For each cluster, they test sets of randomly made choices while assuming that choices outside a cluster are set to competitive default values. Based on those experimental results, the authors formulate recommendations about how to make those choices for each cluster.\n\n\nPROS\nThe paper is well-written and clear. This paper is part of the string of recent papers that discuss the difficulty of evaluating deep RL algorithms. I appreciate the breadth of the choices that the authors consider. The justification for their overall experimental design (i.e., evaluating per choice clusters) is reasonable. While some findings are as expected, others are indeed unexpected and not discussed in the deep RL literature. \n\n\nCONS\nI have a doubt about the robustness of the results. The authors decided to use the median over 3 seeds for the evaluation. Although the median is used, is it reliable given the observations made by Henderson et al., which implies that performance can vary a lot with respect to seeds? Could the authors comment on that point?\n\nI think one important missing experiment is the evaluation of the combinations of all the recommendations made in the paper. Do the recommendations depend on the default setting for other choices, do they have a synergetic effect or could there be some negative interactions?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HBC7V6IvvI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThis paper conducted a large-scale empirical study that provides insights and practical recommendations for the training of on-policy deep actor-critic RL agents\n\n##########################################################################\n\nReasons for score: \n \nOverall, I'd vote for acceptance to the paper. The paper is informative and practical; however, I'm not sure that the paper meets ICLR's requirement.\n\nPros:\n\t1. Reproducibility is one of the main issues for various RL algorithms. This paper conducts a large-scale empirical study for popular on-policy algorithms.\n\t2. The paper is well-written, and the suggestion is useful to me.\n\n\nSorry, but I didn't go through all the details in the appendix.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5nkNQUTGEd0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors survey a wide variety of implementation-level and hyperparameter decisions in reinforcement learning for continuous control tasks. They train over 250.000 agents with different settings and suggest empirical guidelines.\n\nAs the authors indicate, there's not too much related work so one could call this work pioneering: The sort of work conducted by the authors is crucially important for a field afloat with tricks and tweaks, many of which are typically not discussed in the scientific literature due to a misplaced conceit around this being \"not research, just engineering\" entirely absent from established fields of science such as experimental Physics. It is also typically not done as it is just plain hard to do. Combined with the often lackluster response from the community, the cost-benefits trade-off has just not been worth it, especially for junior researchers. It's all the more commendable that the authors have engaged with this formidable task of bringing some of the \"secret sauce\" out of the heads of senior engineers in the various labs and into published and peer-reviewed science.\n\nThe authors compare various choices of configurations obtained from the Cartesian product of 8 factors which they call thematic groups: Policy Losses (Sec. 3.1), Networks architecture (Sec. 3.2), Normalization and clipping (Sec. 3.3), Advantage Estimation (Sec. 3.4), Training setup (Sec. 3.5), Timesteps handling (Sec. 3.6), Optimizers (Sec. 3.7), and Regularization (Sec. 3.8). The high-variance nature of training RL agents makes it such that the individual factors in these configurations often have surprising non-linear cross-relations such that the problem space cannot be evaluated incrementally (i.e., it's often not possible to establish \"the best\" architecture first and select the right learning rate afterwards). The authors propose a novel approach of considering for each choice the distribution of values among the top 5% configurations trained in that experiment. Their experimental design is such that the values for each choice are distributed uniformly at random and thus if certain values are over-represented in the top models this indicates that the specific choice is important in guaranteeing good performance.\n\nAs for improvements on the paper, I have one major and only a few minor comments. My major comment is that the paper does not indicate anywhere that the research code is released, only that it's based on SEED RL. I believe an authoritative public implementation of the configurations considered would be extremely worthwhile, both for the community and the authors. If they haven't already done so (there's no supplements to this submission and I refrained from doing any web searches to preserve anonymity), I'd urge the authors to invest the time to release a (possibly cleaned-up) version of their code.\n\nAs for minor comments, I'm not clear about the philosophical distinction of something being \"due to the algorithms or due to their implementations\" (in the Introduction). I very much see the point the authors are making, which is an important one -- what makes RL results work is often \"nitty-gritty\" details not mentioned in the main part of the relevant publications (and often just barely mentioned in appendices). However, in the strictest sense, the algorithm very much is the implementation -- that's what produces a given result. It's worthwhile to keep the distinction between an idea (say, PPO) and a given implementation of that idea (e.g., presumably the authors had to re-implement PPO in TF2 when using SEED RL and couldn't use OpenAI's original implementation). It's also fine to call the idea \"the algorithm\", but I'd have preferred to see this distinction more clearly defined.\n\nSomewhat related: The authors are very much correct about what they call \"standard modus operandi of algorithms [...] such as PPO\", namely iterating between generating experience using the current policy and using the experience to improve the policy. I'd add that strictly speaking no _iteration_ is necessary, as for instance IMPALA, coming from the A3C line of development, does both asynchronously in parallel, and I suspect so do the authors given their use of SEED RL. My suggestion would be to slightly rephrase this sentence and mention IMPALA along with PPO. Perhaps there could also be a comment somewhere about what constitutes \"PPO\" (or \"IMPALA\") -- e.g., IMPALA consists of (1) an asynchronous actor/learner split [with further choices of when/how the weights are copied from learner to actor, see e.g. [this comment](https://github.com/deepmind/scalable_agent/blob/master/experiment.py#L508)], (2) a specific type of policy-gradient loss, v-trace, (3) a specific neural network architecture, optionally including recurrence via an LSTM and potentially even (4) a specific type of preprocessing for environments such as Atari or DMLab [according to some papers, swapping out the implementation of the frame-downscaling algorithm in Atari has measurable impact on final performance -- this will matter a lot when evaluating re-implementations of an \"algorithm\"]. I'd like for the authors to take on this opportunity and propose a common language to discuss these distinctions, which in practise are often confusing to junior researchers (and some senior researchers, too).\n\nFurther related to IMPALA and v-trace, I was surprised about the word \"unsurprisingly\" and the explanation in \"Perhaps unsurprisingly, PG and V-trace perform worse on all tasks. This is likely caused by their inability to handle data that become off-policy in one iteration, either due to multiple passes over experience [...] or a large experience buffer in relation to the batch size.\" While the results speak for themselves, my understanding of v-trace was that it was specifically designed for the very goal of dealing with the \"slight\" off-policiness produced by asynchronous actor/learner splits in a PG setting. Perhaps the authors have an intuition I'm lacking at this point, but if so I'd appreciate further elaboration.\n\nAs a final and perhaps trivial comment, I was slightly irritated by the notation/typography for the inverse cumulative density function of a binomial distribution. In $\\rm\\LaTeX$, the symbols $icdf$ read as the in-context nonsensical $i\\cdot c\\cdot d\\cdot f$ while the authors would presumably want to use $\\mathrm{icdf}$ (compare $exp(x)$ vs $\\exp(x)$ or $sin(x)$ vs $\\sin(x)$). I'd propose `\\mathrm{icdf}`  as the correct syntax for this in $\\rm\\LaTeX$.\n\nIn follow-up work, I'd like to see a similar paper for various \"discrete RL\" tasks (a subset of Atari, VizDoom, DMLab, MiniGrid, BabyAI, ProcGen, and perhaps even Obstacle Tower, Minecraft, StarCraft (I or II) or the recent NetHack environment) with similar factors of configurations. I assume this is a task yet more daunting, but no less useful to the overall community of researchers.\n\nOverall, this is a strong paper and I recommend it for publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-alWI4hbVo8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper carries out a large-scale study for understanding of on-policy deep actor-critic. The study looks into a large choices of many implementation settings and design decisions, and investigate their impact on the task performance. The evaluations are done with 250000 RL agents on 5 different continuous control tasks. For each evaluation category, there is a finding summary that provides practical recommendations.\n\nIn overall, this study is exhaustive and helpful to both RL researchers and practitioners. The experiment organization which separates all design choices into 7 main categories is very excellent in a systematic way. They cover most design choices in recent works of on-policy RL methods. The reports and the interpretation of results are very interesting and easy to read. The main and important findings are summarized concisely and expected to play important hints.\n\n\nThe only performance metric studied in the paper is a score that is proportional to the area under the learning curve. I was wondering if there should be an additional metric, i.e the final policy or an average reward of the final 100 policies? Would the final or best policy be of more interest to the choice of a practitioner?\n\nAs many recent work investigates the design choice of only on-policy RL methods, it would be interesting if in introduction there is discussion on why off-policy methods are not considered or should it be addressed in a different way in another research?\n\nBeside the focus on only the performance in terms of rewards, it would be interesting if the discussion can be expanded to look at other matters, e.g. numerical stability of design/hyper-parameter choices, convergence behaviors (it might requires plot to see if a method show premature convergence, fast learning but sub-optimal, fluctuating, etc.). \n\nAlthough the paper only uses Mujoco simulator, would the hyperparameters' domains be subjective to it, e.g. inertia, fiction, joint limits, contacts, etc.? It would be helpful if the discussion can show if such those factors play any role in the results? It would lead to more helpful finding summary.\n\n\nAs a final comment, this is a solid work and will be very helpful to the community. Given that it is implemented on the new SEED RL framework, so it would be better if the implementation code can be published.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a clear consensus over all reviewers that this is a very strong empirical analysis, with actionable insights that should prove quite useful both to researchers and practitioners. I have no doubt that many will use it as a reference when implementing and using RL algorithms (especially since the authors said they would release their code).

This is thus a clear accept, that in my opinion would deserve an oral presentation, so as to better disseminate its key findings.
Decision: Accept (Oral)