Implementation Matters in Deep RL: A Case Study on PPO and TRPO
Authors: Logan Engstrom
Andrew Ilyas
Shibani Santurkar
Dimitris Tsipras
Firdaus Janoos
Larry Rudolph
Aleksander Madry
Venue:  (2020)
Keywords: deep policy gradient methods
deep reinforcement learning
trpo
ppo

Abstract: We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty, and importance, of attributing performance gains in deep reinforcement learning.


Published as a conference paper at ICLR 2019
IMPLEMENTATION MATTERS IN DEEP POLICY GRADIENTS: A CASE STUDY ON PPO AND TRPO
Logan Engstrom", Andrew Ilyas*, Shibani Santurkar', Dimitris Tsipras', Firdaus Janoos?, Larry Rudolph'2, and Aleksander Madry!
MIT Two Sigma
(engstrrom, ai ailyas yas, shibani, tsi pras, madry) @mi Smmit.s. edu rudolphécsail
f janoos com
firdaus.
ABSTRACT
We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations Seemingly of secondary importance, such optimizations turn out to have a major impact agent behavior. Our results show that they (a) responsible for most of PPO's s gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty, and importance, of attributing performance gains in deep reinforcement learning.
described auxiliary details the core algorithm.
INTRODUCTION
Deep reinforcement learning (RL) algorithms have fueled many of the most publicized achievements in modern machine learning (Silver et al.) 2017] OpenAl| 2018 Abbeel & Schulman (016)] Mnih eet al., 2013). However, despite these accomplishmente deep RL methods still are not nearly as reliable as their (deep) supervised learning counterparts. Indeed, recent research found the existing deep RL methods to be brittle (Henderson al., 2017 ZZang al., 2018), hard reproduce (Hen-
derson et al.) 2017 Tucker et 2018). unreliable across runs (Henderson et al. 2017 2018). and sometimes outperformed by simple baselines (Mania al. 2018).
The prevalence of these issues points to a broader problem: we do not understand how the parts comprising deep RL algorithms impact agent training. either separately or as a whole. This unsat- isfactory understanding suggests that we should re-evaluate the inner workings of our algorithms. Indeed, the overall question motivating our work is: how do the multitude of mechanisms used in deep RL training algorithms impact agent behavior?
Our contributions. We analyze the underpinnings of agent behavior both through the traditional metric of cumulative reward, and by measuring more ane-grained algorithmic properties. As first step towards tackling this question, we conduct case study of two of the most popular deep policy- gradient methods: Trust Region Policy Optimization (TRPO) (Schulman et al., (2015)) and Proximal Policy Optimization (PPO) (Schulman et al. 2017). These two methods are closely related: PPO was originally developed
refinement of TRPO.
We find that much of the PPO's observed improvement in performance comes from seemingly small modifications to the core algorithm that either can be found only paper's original implementa- tion, or are described as auxiliary details and are not present in the corresponding TRPO baselines, We pinpoint these modifications, and perform an ablation study demonstrating that they are instru- mental to the PPO's 's performance.
Equual contribution. Work done in part as an intern at Two Sigma.
Note that these code-level optimizations separate from "implementation choices" like the choice of PyTorch versus TeosorFlow in that they intentionally change the training algorithm's operation.

---

Published as a conference paper at ICLR 2019
This observation prompts us study how such code-level optimizations change agent training dy- namics, and whether we can truly think of them as merely auxiliary improvements. Our results indicate that these optimizations fundamentally change algorithms' operation, and go even beyond improvements in agent reward. We find that they majorly impact a key algorithmic principle behind TRPO and PPO's operations: trust region enforcement.
Ultimately, we discover that the PPO rode-optimizations are more important in terms of final re- ward achieved than the choice of general training algorithm (TRPO vs. PPO). This result is in stark contrast to the previous view that the central PPO clipping method drives the gains seen in Schulman al.] (017). In doing so, we demonstrate that the algorithmic changes imposed by such optimiza- tions make rigorous comparisons of algorithms difficult. Without a rigorous understanding of the full impact of code-level optimizations, we cannot hope to gain any reliable insight from comparing algorithms on benchmark tasks.
Our results emphasize the importance of building RL methods in modular manner. To progress towards more performant and reliable algorithms, need to understand each component's 's impact on agent behavior and performance both individually, and as part of whole.
Code for all the results shown in this work is available at httpt: http::::///itttb..co.. com www.eoooo/aaaaaaaa.a
imp ementa tataenttttitmmmmtrrrtara
2 RELATED WORK
The idea of using gradient estimates to update neural network-based RL agents dates back at least to the work of Williams (1992). who proposed the REINFORCE algorithm. Later, SStton et al. (1999) established unifying framework that casts the previous algorithms as instances of the policy gradient method.
Our work focuses on proximal policy optimization (PPO) (Schulman al. 2017) and trust region policy optimization (TRPO) (Schulman et al., (005a)). which are two of the most prominent policy gradient algorithms used in deep RL. Much of the original inspiration for the usage of the trust regions stems from the conservative policy update of Kakade (2001). This policy update, similarly TRPO, uses a natural gradient descent-based greedy policy update. TRPO also bears similarity to the relative policy entropy search method of [Ptters et al. (2010), which constrains the distance between marginal action distributions (whereas TRPO constrains the conditionals of such action distributions).
Notably, Henderson et al. (2017) points out number of brittleness, reproducibility. and experimen- tal practice issues in deep RL algorithms. Importantly, we build on the observation of [Henderoon jct al. (2017) that final reward for given algorithm is greatly influenced depending on the code base used. Rajeswaran et al. (017) and Mania et al. (2018) also demonstrate that on many of the benchmark tasks, the performance of PPO and TRPO can be matched by fairly elementary random- ized search approaches. Additionally, Tucker et al. (2018) showed that one of the recently proposed extensions of the policy gradient framework, i.e., the usage of baseline functions that are also action- dependent (in addition being state-dependent), might not lead to better policies after all.
ATTRIBUTING SUCCESS IN PROXIMAL POLICY OPTIMIZATION
Our overarching goal is to better understand the underpinnings of the behavior of deep policy gra- dient methods. We thus perform careful study of two tightly linked algorithms: TRPO and PPO (recall that PPO is motivated as TRPO with a different trust region enforcement mechanism). To better understand these methods, we start by thoroughly investigating their implementations in prac- tice. We find that in comparison to TRPO, the PPO implementation contains many non-trivial op- timizations that are not (or only barely) described its corresponding paper. Indeed, the standard implementation of PPO CroPPonntiin the following additional optimizations:
From the OpenAl haselines GitHub repository:
hwt.:/wwwwooooo/oaa.aaaaaaa (bapeennnaabaaeell /basel ines
2

---

Published as a conference paper at ICLR 2019
1. Value function clipping: Schulman et al. (2017) originally suggest fitting the value net- work via regression to target values:
L =y=0 -Vo. Viarg)?
but the standard implementation instead fits the value network with a PPO-like objective:
LV min
(Vo. Viarg) (clip (Vo,, Vo._, E, Vo:-, + E) Viarg)
where Vi is clipped around the previous value estimates (and is fixed to the same value the value used in 2 to clip the probability ratios).
2. Reward scaling: Rather than feeding the rewards directly from the environment into the objective, the PPO implementation performs a certain discount-based scaling scheme. In this scheme, the rewards are divided through by the standard deviation of a rolling dis- counted sum of the rewards (without subtracting and re-adding the mean) see Algorithm in Appendix A.2
3. Orthogonal initialization and layer scaling: Instead of using the default weight initial- ization scheme for the policy and value networks, the implementation uses an orthogonal initialization scheme with scaling that varies from layer to layer.
4. Adam learning rate annealing: Depending the task, the implementation sometimes anneals the learning rate of Adam (Kingma & Ba 2014) (an already adaptive method) for optimization.
5. Reward Clipping:
The implementation also clips the rewards within preset range (usu- ally [-5,5] or [-10, 10)).
6. Observation Normalization: In a similar manner to the rewards, the raw states are also not fed into the optimizer. Instead, the states are first normalized mean-zero, variance-one vectors.
7. Observation Clipping: Analagously to rewards, the observations are also clipped within range, usually [-10, 10).
8. Hyperbolic tan activations: As also observed by Henderson et al. (017)). implementa- tions of policy gradient algorithms also also use hyperbolic tangent function activations between layers in the policy and value networks.
9. Global Gradient Clipping: After computing the gradient with respect to the policy and the value networks, the implementation clips the gradients such the "global &g norm" (i.e. the norm of the concatenated gradients of all parameters) does not exceed 0.5.
These optimizations may appear merely surface-level or insignificant algorithmic changes the core policy gradient method at hand. However, we find that they dramatically affect the performance of PPO. To demonstrate this. we start by performing a full ablation study on the four optimizations mentioned above f Figure [ shows histogram of the final rewards of agents trained with every each configuration, grid search for the optimal learning rate is performed, and we measure the reward of random agents trained using the identified learning rate. Our findings suggest that many code-level optimizations are necessary for PPO to attain its claimed performance.
possible configuration of the above optimizations--ie
The above findings show that our ability to understand PPO from an algorithmic perspective hinges on the ability distill out its fundamental principles from such algorithm-independee independent (in the sense that these optimizations can be implemented for any policy gradient method) optimizations. We thus consider a variant of PPO called PPO-MINIMAL (PPO-M) which implements only the core of the algorithm. PPO-M uses the standard value network loss, no reward scaling, the default network initialization, and Adam with fixed learning rate. Importantly, PPO-M ignores all the code-level optimizations listed above in the beginning of Section 3 We then explore PPO-M alongside PPO and TRPO. We list all the algorithms we study and their defining properties in Table !
Overall, our results on the importance of these optimizations both corroborate results demonstrating the brittleness of deep policy gradient methods, and demonstrate that even beyond environmental brittleness, the algorithms themselves exhibit high sensitivity to implementation choices
Due to restrictions on computational resources, we could only perform a full ablation on the first four of the identified oprimizations.
"Tiis might also explain the difference hetween different codehases observed in Henderson al. 2017
3

---

Published as a conference paper at ICLR 2019
Humanoid-v2
clipping True Fa
norm_r rewards returns nore
CDF(Reward)
anneal_r Ir
initialization
0.8
I-CDFiReward
0.6
83
True False
CCCF(Reeaad))
0.8
orthegonal xavier
0.6
0.4
0.4
0.1
0.6
0.0
1000
500
1280
400 Reward
1060
Reward
Reward
Reward
Walker2d-v2
1.4
valuee_cllpaeeee clipping True
1.0
normur rewaards
10
anneal_rr True
1.5
initialization orthogonal xavier
CDF(Reeard))
C-FF(Rewaard
0.8
returms
CDF(Reward)
I-CDF(Reward))
False
0.6
0.6
0.6
6.2
0.2
1900
2002 1600 1000 5000
1930
2000 1000 4000 5932
3000
2000 9:03 1932
5000
1000
2002 3000
1600
5000
Reward
Reward
Reward
Reward
Figure 1: An ablation study on the first four optimizations described in Section 3 (value clipping. reward scaling, network initialization, and learning rate annealing). For each of the 2 possible con- figurations of optimizations, we train a Humanoid-v2 (top) and Walker2d-v2 (bottom) agent using PPO with five random seeds and grid of learning rates, and choose the learning rate which gives the best average reward (averaged over the random seeds). We then consider all rewards from the "best learning rate" runs (a total of 5 x 2 agents), and plot histograms in which agents are parti- tioned based on whether each optimization is on or off. Our results show that reward normalization, Adam annealing, and network initialization each significantly impact the rewards landscape with re- spect to hyperparameters, and were necessary for attaining the highest PPO reward within the tested hyperparameter grid. We detail our experimental setup in Appendix A.I
Table 1: List of algorithms studied in this work, with their crucial properties. Step method refers to the method used to build each training step, PPO clipping refers to the use of clipping in the step (as in Equation and PPO optimizations refer to the optimizations listed in Section
Algorithm
Section
Step method
Uses PPO clipping?
Uses PPO optimizations?
PPO PPO-M
PPO
As in (Dhariwal et al. 2017
Sec.
B
PPO
PPO-NOCLIP
Sec.
PPO
Found via grid search
TRPO
TRPO
X
TRPO+
Sec. 5
TRPO
Found via grid search
4 CODE-LEVEL OPTIMIZATIONS HAVE ALGORITHMIC EFFECTS
In the previous section, found that canonical implementations of PPO contain many code-level optimizations: implementation choices that are not integral to the method but profoundly impact performance.
The seemingly disproportionate effect of code-level optimizations identified in our ablation study may lead us to ask: how do these seemingly superficial code-level optimizations impact underlying agent behavior? In this section, we demonstrate that the code-level optimizations fundamentally alter agent behavior. Rather than merely improving ultimate cumulative award, such optimizations directly impact the principles motivating the core algorithms.
Trust Region Optimization. A key property of policy gradient algorithms is that update steps computed at any specific policy Tg, are only guaranteed predictiveness in a neighborhood around 0. Thus, ensure that the update steps we derive remain predictive many policy gradient algo-
4

---

Published as a conference paper at ICLR 2019
rithms ensure that these steps stay in the vicinity of the current policy. The resulting "trust region" methods (Kakade) 2001] SSchulman et al. 2215aa 2017) try to constrain the local variation of the parameters in policy-space by restricting the distributional distance between successive policies.
A popular method in this class is trust region policy optimization (TRPO) Schulman et al. (2015a). TRPO constrains the KL divergence between successive policies the optimization trajectory, leading to the following problem:
[(auls)) (x ai)
max ( 8
s.t. (RL( 8)][T(( 8)) S vs
In practice, maximize this objective with a second-order approximation of the KL divergence and natural gradient descent, and replace the worst-case KL constraints over all possible states with an approximation of the mean KL based on the states observed the current trajectory.
Proximal policy optimization. One disadvantage of the TRPO algorithm is that it can be compu- tationally costly-the step direction is estimated with nonlinear conjugate gradients, which requires the computation of multiple Hessian-vector products. To address this issue, /Schulman et al. (2017) propose proximal policy optimization (PPO), which tries to enforce a trust region with a different objective that does not require computing projection. Concretely, PPO proposes replacing the KL-constrained objective (T) of TRPO by clipping the objective function directly as:
max E (s, min (clip (pe, E, + E) As(si,a)) (1 at))
(2)
where
To Te(a: St))
P: π < Se)
Note that this objective can be optimized without an explicit projection step, leading to a simpler parameter update during training. In addition its simplicity, PPO is intended to be faster and more sample-efficient than TRPO (Schulman et al. 2017).
Enforcing a trust region core algorithmic property of different policy gradient methods. However, whether or not trust region is enforced is not directly observable from the final rewards. So, how does this algorithmic property vary across state-of-the-an policy gradient methods?
Trust regions in TRPO and PPO.
In Figure 2 we measure the mean KL divergence between successive policies a training run of both TRPO and PPO-M (PPO without code-level optimizations). Recall that TRPO is designed specifically to constrain this KL-bbased trust region, while the clipping mechanism PPO attempts to approximate it. Indeed, we find that TRPO precisely enforces this trust region (this is unsuprising, and nearly by construction).
We thus turn attention the trust regions induced by training with PPO and PPO-M. First, consider mathematically the contribution of a single state-action pair to the gradient of the PPO objective, which is given by
VoLo
mefals) [(ass [1 - + or Ig < To
VoLrro
0
otherwise
Ta(als) <[aas)
A Az (s,
where
Lg (o..)EET
Toe [T(a))) [[ass)
and Lg = --EEGGaEENN
clip
1- e, 1 An
are respectively the standard and clipped versions the surrogate objective. As result, since we initialize To (and thus the ratios start all equal one) the first step take is identical to maximization step over the unclipped surrogate objective. It thus stands to reason that the nature of the trust region enforced is heavily dependent on the method with which the clipped PPO objective
5

---

Published as a conference paper at ICLR 2019
800
Meee Reward
PP
15.0
0.06
PHD
0.0
12.5
10.0
400
0.04
200
0.00
100
2:00
400
300
400
2:00 303 Iteeaaions
Iterations
Iterati ons
Figure 2: Per step mean reward, maximum ratio (c.f. 22), mean KL, and mean KL for agents trained solve the MuJoCo Humanoid-v2 task. The quantities are measured over the state-action pairs collected in the training step. Each line represents training curve from separate agent. The black dotted line represents the 1 + c ratio constraint in the PPO algorithm, and we measure each quantity every twenty five steps. We take mean and max KL over the KL divergences between the conditional distributions induced by the current and previous policy on the observed states training (at each step). In the left plot we see the reward for each trained agent. From in the middle plot, we can see that the PPO variants' maximum ratios consistently violates the ratio "trust region." " In the right plot, we that both PPO and PPO-M constraint the KL well (compared to the TRPO bound of 0.07). The two methods exhibit different behavior: while PPO-M KI trends up as the number of iterations increases, PPO KL peaks halfway through training before trending down again. We measure the quantities over a heldout set of state-action pairs and find little qualitative difference in the results (seen in Figure in the appendix), suggesting that TRPO does indeed enforce mean KL trust region. We show plots for additional tasks in the Appendix in Figure 3 experimental setup in Appendix A.1
We detail our
is optimized, rather than on the objective itself. Therefore, the size of step take is determined solely be the steepness of the surrogate landscape (i.e. Lipschitz constant of the optimization prob- lem we solve), and we can end up moving arbitrarily far from the trust region. We hypothesize that this dependence of PPO on properties of the optimizer rather than the optimization objective con- tributes to the brittleness of the algorithm to hyperparameters such as learning rate and momentum, as observed by Henderson et al. (2018) and others.
The results we observe (shown in Figure 2 corroborate this intuition. For agents trained with op- timal parameters, all three algorithms are able to maintain a KL--based trust region. First, we note that all three algorithms fail to maintain a ratio-based trust region, despite PPO and PPO-M being trained directly with ratio-clipping objective. Furthermore, the nature of the KL trust region en- forced differs between PPO and PPO-M, despite the fact that the core algorithm remains constant between the two methods; while PPO-M KL trends up as the number of iterations increases, PPO KL peaks halfway through training before trending down again.
The findings from this experiment and the corresponding calculations demonstrate that perhaps key factor in the behavior PPO-trained agents even from an algorithmic viewpoint comes from auxiliary optimizations, rather than the core methodology.
IDENTIFYING ROOTS OF ALGORITHMIC PROGRESS
State-of-the-art deep policy gradient methods are comprised of many interacting components. At what is generally described as their core, these methods incorporate mechanisms like trust region- enforcing steps, time-dependent value predictors, and advantage estimation methods for controlling the exploitation/exploraiin trade-off (Schulman et al. 2015b). However, these algorithms also incorporate many less oft-dixcussed optimizations (cf. Section 3 that ultimately dictate much of agent behavior (cf. Section 4. Given the need to improve on these algorithmss, the fact that such optimizations are so important begs the question: how do we identify the true roots of algorithmic progress deep policy gradient methods?
Unfortunately, we find that answering this question is not easy. Going back to our study of PPO and TRPO, it is widely believed (and claimed) that the key innovation of PPO responsible for its improved performance over the baseline of TRPO is the ratio clipping mechanism discussed in Section[4] However, we have already shown that this clipping mechanism is insufficient theoretically

---

Published as a conference paper at ICLR 2019
Table 2: Full ablation of step choices (PPO or TRPO) and presence of code-level optimizations measuring agent performance on benchmark tasks. TRPO+ is a variant of TRPO that uses PPO inspired code-level optimizations, and PPO-M level optimizations (cf. S.) We find that varying the use of code-level optimizations impacts performance significantly more than varying whether the PPO or TRPO step is used. We detail our experimental setup in Appendix [ We train at least 80 agents for each estimate (more for some high- high-variance cases). We present 95% confidence intervals computed via a 1000~sample bootstrap. We also present the AAI and ACLI metries discussed in Section 5 which attempt to quantify the relative contribution of algorithmic choice vs. use of code-level optimizations respectively.
variant of PPO that does not use PPO's code-
MuJoCo TASK
STEP
WALKER2D-V2
HOPPER-V2
HUMANOID-V?
PPO
3292 [3157, 3426]
2513 [2391, 2632]
806 [785, 827]
PPO-M
2735 [2602. 2866]
2142 [2008. 2279]
674 [656, 695]
TRPO
2791 [2709, 2873]
2043 [1948, 2136]
586 [576, 596]
TRPO+
3050 [2976, 3126]
2466 [2381, 2549]
1030 [979, 1083]
AAI
242
99
224
ACLI
557
421
444
maintain a trust region, and also that the method by which the objective is optimized appears to have significant effect on the resulting trust region. If code-level optimizations are thus (at least partially) responsible for algorithmic properties of PPO, possible that they are also a key factor in PPO's improved performance?
To address this question, we set out to further disentangle the impact of PPO's core clipping mecha- nism from its code-level optimizations by once again considering variations on the PPO and TRPO algorithms. Specifically, we examine how employing the core PPO and TRPO steps changes model performance while controlling for the effect of code-level optimizations identified in standard im- plementations of PPO (in particular, we focus on those covered in Section 3) (Note that these code-level optimizations are largely algorithm algorithm-independdee independent: they be straightforwardly applied or lightly adapted to any policy gradient method.) The previously introduced PPO-M algorithm corresponds to PPO without these optimizations. To further account for their effects, we study an additional algorithm which we denote as 7RPO+, consisting of the core algorithmic contribution of TRPO in combination with PPO's code-level optimizations as identified in Section 3 We note that TRPO+ together with the other three algorithms introduced (PPO, PPO-M, and TRPO; all listed in Table now capture all combinations of core algorithms and code-level optimizations, allowing us to study the impact of each in a fine- ann-igrined manner.
As our results show in Table it turns out that code-level optimizations contribute algorithms' increased performance often significantly more than the choice of algorithm (i.e., using PPO vs. TRPO). For example, on Hopper-v2, PPO and TRPO see 17% and 21 % improvements (respectively) when equipped with code-level optimizations. At the same time, for all tasks after fixing the choice to use or not use optimizations, the core algorithm employed does not seem to have a significant impact reward. In Table we quantify this contrast through the following two metrics, which we denote average algorithmic improvement (AAI) and average code-level improvement (ACLI):
AAI = m TRPO+|, PPO-M TRPO
ACLI = max{ PPO - PPO-M| -I, TRPO+ TRPO]}
In short, AAI measures the maximal effect of switching step algorithms (from PPO to TRPO or vice-versa), whereas ACLI measures the maximal effect of adding in code-level optimizations for a fixed choice of step algorithm.
PPO without clipping. Given the relative insignificance of the step mechanism compared to the use of code-level optimizations, we are prompted to ask: to what extent is the clipping mechanism
'ee also add new code-level optimization, KT. decay, inapplicable to PPO hut meant to serve as the analog of Adam learning rate annealing.
7

---

Published as a conference paper at ICLR 2019
Table 3: Comparison of PPO performance PPO without clipping. We find that there is little difference between the rewards attained between the two algorithms on the benchmark tasks. Note that both algorithms use code-level optimizations; our results indicate that the clipping mechanism often of comparable or lesser importance to the use of code-level optimizations. We detail our exper- imental setup Appendix A We train at least 80 agents for each estimate (for some high-variance cases, more agents were used). We present 95% confidence intervals computed via a 1000-sample bootstrap. We also present results from the OpenAl baselines (Dhariwal et al. 2017) repository where available.
WALKER2D-V2
HOPPER-V2
HUMANOID-V2
PPO PPO (BASELINES))
3292 [3157, 3426]
2513 [2391, 2632]
806 [785, 827]
3424
2316
PPO-M PPO-NOCLIP
2735 [2602, 2866]
2142 [2008. 2279]
674 [656, 695]
2867 [2701, 3024]
2371 [2316, 2424]
831 [798, 869]
of PPO actually responsible for the algorithm'ss success? In Table R]
assess this by considering algorithm which makes use of common code-level optimizations (by gridding over the best possible combination of such optimizations) but does not employ a clipping mechanism (this is the same algorithm we studied in Section 4 in the context of trust region enforcement) rrecall that we list all the algorithms studied in Table
a
It turns out that the clipping mechanism is not necessary to achieve high performance-vve we find that PPO-NOC1.IP performs uniformly better than PPO-M, despite the latter employing the core PPO clipping mechanism. Our results suggest that the introduction of code-level optimizations outweighs even the core PPO algorithm in terms effect on rewards. sufficient hyperparameter tuning, PPO-NOCLIP often matches the performance of standard PPO, which includes a standard configuration of code-level optimizations" [6) We also include benchmark PPO numbers from the OpenAl basel ines repository (Dhariwal al. 2017) where available to put results into context.
fact, we find that with
Our results suggest that it is difficult to attribute success to different aspects of policy gradient algorithms without careful analysis.
6 CONCLUSION
In this work, we take first step in examining how the mechanisms powering deep policy gradi- methods impact agents both in terms of achieved reward and underlying algorithmic behavior. Wanting to understand agent operation from the ground up, we take a deep dive into the operation of two of the most popular deep policy gradient methods: TRPO and PPO. In doing so, we identify -aaiiittiioosalooriiiii
number of "code-level mentations or described auxiliary details in their presentation and find that these optimizations have a drastic effect on agent performance.
augmentations found only in algorithms' imple-
In fact, these seemingly unimportant optimizations fundamentally change algorithm operation in ways unpredicted by the conceptual policy gradient framework. Indeed, the optimizations have a profound effect on the nature of the trust region enforced by policy gradient algorithms, even controlling for the surrogate objective being optimized. We go to test the importance of code- level optimizations in agent performance, and find that PPO's marked improvement over TRPO (and even stochastic gradient descent) can be largely attributed to these optimizations.
Overall, our results highlight the necessity of designing deep RI. methods in a modular manner. should understand precisely how each component impacts agent
When building algorithms, training- both in terms of overall performance and underlying algorithmic behavior. It is impos- sible to properly attribute successes and failures in the complicated systems that make up deep RL methods without such diligence. More broadly, our findings suggest that developing an RL toolkit
'Not that it is possible that further refinement on the code-level optimizations could be added on top of PPO to perhaps improve its performance to an even greater extent (after all, PPO-NOCI.IP can only express subset the training algorithms covered by PPO, as the latter leaves the clipping severity to he free parameter)
8

---

Published as a conference paper at ICLR 2019
will require moving beyond the current benchmark-driver evaluation model to more tne-grained understanding of deep RL methods.
REFERENCES
Pieter Abbeel and John Schulman. Deep reinforcement learning through policy optimization. Tuto- rial Neural Information Processing Systems, 2016.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
//github. wwwp://p/////aaaaaaeaeeeaoaa
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint 170Xiv:700.000 2017.
Peter Henderson, Joshua Romoff, and Joelle Pineau. Where did my optimum go?: An empirical analysis of gradient descent optimization in policy gradient methods, 2018.
Sham M. Kakade. A natural policy gradient. In NIPS, 2001.
Diederik P. Kingma and Jimmy Ba. abs/1412.6980, 2014.
Adam: A method for stochastic optimization. CoRR,
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach reinforcement learning. CoRR, abs/1803.07055, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NeurIPS Deep Learning Workshop, 2013.
OpenAl. Openai five.
https: /bwll.ll...////////..o..oooooo.a wweooag.ooeniii..o..o wawpenaii.coooppeiiiiiiiii 2018.
Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. AAAI, 2010.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade. Towards general- ization and simplicity in continuous control. In NIPS, 2017.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.022 024388 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00677 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):355 2017.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, 1999.
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E. Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. In ICML, 2018.
J. Williams. Simple statistical learning. Machine Learning, 8:229-255 256, 1992.
algorithms for conmeciioistreeiffrreenntt
Amy Zhang, Yuxin Wu, and Joelle Pincau. Natural environment benchmarks for reinforcement learning, 2018

---

Published as a conference paper at ICLR 2019
A APPENDIX
A. EXPERIMENTAL SETUP
All the hyperparameters used this paper were obtained through grid searches. For PPO the exact code-level optimizations and their associated hyperparameters (e.g. coefficients for entropy regular- ization, reward clipping, etc.) were taken from the OpenAl baselines repository 7 and gridding performed over the value function learning rate, the clipping constant, and the learning rate sched- ule. TRPO, we grid over same parameters (replacing learning rate schedule with the KL constraint), but omit the code-level optimizations. For PPO-NoClip, grid over the same param- eters as PPO, in addition to the configuration code-level optimizations (since we lack a good reference for what the optimal configuration of these optimizations is). For TRPO+ over the code-level optimizations, and also implement a "KI. schedule" whereby the KI. constraint change over training (analogous the learning rate annealing optimization in PPO). Finally, for PPO-M, we grid over the same parameters as PPO (just learning rate schedules), without any code-level optimizations. The final parameters for each algorithm are given below:
also grid
Table 4: Hyperparameters for all algorithms for Walker2d-v2.
PPO
TRPO
PPO-NoClip
PPO-M
TRPO+
Timesteps per iteration
2048
2048
2048
2048
2048
Discount factor (7)
0.99
0.99
0.99
0.99
0,99
GAE discount (A)
0.95
0.95
0.85
0.95
0.95
Value network LR
0.0003
0.0003
0,0006
0.0002
0,0001
Value network num. epochs
10
10
10
10
10
Policy network hidden layers
[64, 64]
[64, 64]
164, 64]
[64, 64]
164, 64]
Value network hidden layers
[64, 64]
164, 64]
164, 64]
164, 64]
164, 64]
KI. constraint (ō)
N/A
0.04
N/A
N/A
0.07
Fisher estimation fraction
N/A
0.1
N/A
N/A
0.1
Conjugate gradient steps
N/A
10
N/A
N/A
10
Conjugate gradient damping
N/A
0.1
N/A
N/A
0.1
Backtracking steps
N/A
10
N/A
N/A
10
Policy LR (Adam)
0.0004
N/A
7.25e-05
0.0001
N/A
Policy epochs
10
N/A
N/A
PPO Clipping
0.2
N/A
le+32
0.2
N/A
Entropy coeff.
0
-0.01
Reward clipping
[-10.0, 10.0]
[-30, 30]
[-10.0, 10.0]
Gradient clipping (&g norm)
-1
0.1
-1
Reward normalization
returns
none
rewards
none
returns
State clipping
[-10.0, 10.0]
[-30, 30]
[-10.0, 10.0]
All error bars plot are 95% confidence intervals, obtained via bootstrapped sampling.
httpp:://githuu.ccommmaaa /base1
10

---

Published as a conference paper at ICLR 2019
Table 5: Hyperparameters for all algorithms for Humanoid-v2.
PPO
TRPO
PPO-NoClip
PPO-M
TRPO+
Timesteps per iteration
2048
2048
2048
2048
2048
Discount factor (r)
0.99
0.99
0.99
0.99
0.99
GAE discount (A)
0.95
0.95
0.85
0.95
0.85
Value network LR
0.0001
0.0003
5e-05
0.0004
5e-05
Value network num. epochs
10
10
10
10
10
Policy network hidden layers
[64, 64]
[64, 64]
[64, 64]
[64, 64]
164, 64]
Value network hidden layers
[64, 64]
[64, 64]
164, 64]
164, 64]
[64, 64]
KL constraint (d)
N/A
0.07
N/A
N/A
0.1
Fisher estimation fraction
N/A
0.1
N/A
N/A
0.1
Conjugate gradient steps
N/A
10
N/A
N/A
10
Conjugate gradient damping
N/A
0.1
N/A
N/A
0.1
Backtracking steps
N/A
N/A
N/A
10
Policy LR (Adam)
0,00015
N/A
2e-05
9e-05
N/A
Policy epochs
10
N/A
N/A
PPO Clipping
0.2
N/A
1c+32
0.2
N/A
Entropy coeff.
0
0.005
Reward clipping
[-10.0, 10.0]
[10.0, 10.0]
[-10.0, 10.0]
Gradient clipping (&2 norm)
-1
0.5
0.5
Reward normalization
returns
none
returns
none
returns
State clipping
[-10.0, 10.0]
[-10.0, 10.0]
[-10.0, 10.0]
Table 6: Hyperparameters for all algorithms for Hopper-v2.
PPO
TRPO
PPO-NoClip
PPO-M
TRPO+
Timesteps per iteration
2048
2048
2048
2048
2048
Discount factor (y)
0.99
0.99
0.99
0.99
0,99
GAE discount (A)
0.95
0.95
0.925
0.95
0.95
Value network LR
0.00025
0.0002
0.0004
0.0004
0,0002
Value network num. epochs
10
10
10
10
10
Policy network hidden layers
[64, 64]
[64, 64]
[64, 64]
[64, 64]
[64, 64]
Value network hidden layers
[64, 64]
[64. 64]
[64, 64]
[64, 64]
[64, 64]
KL constraint (ö)
N/A
0.13
N/A
N/A
0.04
Fisher estimation fraction
N/A
0.1
N/A
N/A
0.1
Conjugate gradient steps
N/A
N/A
N/A
Conjugate gradient damping
N/A
0.1
N/A
N/A
0.1
Backtracking steps
N/A
N/A
N/A
Policy LR (Adam)
0.0003
N/A
6c-05
0.00017
N/A
Policy epochs
10
N/A
10
10
N/A
PPO Clipping
0.2
N/A
lc+32
0.2
N/A
Entropy coeff.
0
-0.005
Reward clipping
[10000 10.0]
1-2.5, 2.5]
100.0, 10.0]
Gradient clipping (f2 norm)
Reward normalization
returns
none
rewards
none
returns
State clipping
[-10.0, 10.0]
[22,5, 2.5]
110.0 10.0]
11

---

Published as a conference paper at ICLR 2019
A.2 PPO CODE-LEVEL OPTIMIZATIONS
Algorithm PPO scaling optimization.
1: procedure INTTIALIZESSALLNNG)
2: Ro f
3: RS RUNNINGSTATTISTICS)
New running stats class that tracks mean, standard
deviation
4: procedure SCALIF-OBSFRVATIONNN.)
Input: reward rt is the reward discount
5: R4 4- RRt11 +r
6: ADD(RS, Rτ)
7: return D/SSANDAAR--DVVAITO(((5S
Returns scaled reward
12

---

Published as a conference paper at ICLR 2019
A.3
TRUST REGION OPTIMIZATION
4000
TBP0
1850
2,035
Beean Reward
3000
2000
2,020
1000
5.055
(a) Walker2d-v2 (train)
0.12
Meen Reward
1PD
PPO
PRO
2000
0.08
15
0.06
1000
10
0.04
100
403
3:0
200
Iterations
Iterationn
Iterations
(b) Hopper-v2 (train)
Figure 3: Per step mean reward, maximum ratio (c.f. mean KI, and maximum versus mean KL for agents trained to solve the MuJoCo Humanoid task. The quantities are measured over the state-action pairs collected in the training step. Each line represents training curve from separate agent. The black dotted line represents the + e ratio constraint the PPO algorithm, and we measure each quantity every twenty five steps. Compare the results here with Figure 4 they qualitatively nearly identical.
13

---

Published as a conference paper at ICLR 2019
800
9.06
Mean Reward
MON
MMN
maxinulh
400
0.03
200
200
300
300
400
100
Iterations
Iterations
Iterations
(a) Humanoid-v2 (heldout)
7EPO
mh7o APo
TA70
FPC
2.035
PAO
Mean Reward
7PD
mo M
PON
maXirults)
5.000
2,000
100
203
300
403
100
200
300
400
100
200
300
400
Iterations
# Iter ations
Iterations
(b) Walker2d-v2 (heldout)
3000
7890
15.0
TR3O
0.150
TARO
110
me
0.325
Mean Reward
190
PRC
PRONN
2000
0.300
1500
0.075
7.3
0.050
500
2.5
100
300
100
350
100
Iterations
Ibr ons
# Iterations
(c) Hopper-v2 (heldout)
Figure 4: Per step mean reward, maximum ratio (c.f. 2 mean KL, and maximum versus mean KL for agents trained to solve the MuJoCo Humanoid task. The quantities are measured over state- action pairs collected from heldout trajectories. Each line represents curve from separate agent. The black dotted line represents the + ratio constraint in the PPO algorithm, and measure each quantity every twenty five steps. See that the mean KL for TRPO nearly always stays within the desired mean KL trust region (at 0.06).
14


Reviews:
[{"review_id": "HJeISRZAKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the impact of implementation \"details\", with existing implementations of TRPO and PPO as examples. The main takeaway is that the performance gains observed in PPO (compared to TRPO) are actually caused by differences in implementation, and not by the differences between the two learning algorithms. In particular, adding to the TRPO code the same implementation changes as in PPO makes TRPO on par with (and possibly even better than) PPO. The clipping objective of PPO is also found to have no significant impact on its performance. This calls for more careful comparisons between algorithms (by minimizing implementation changes and more in-depth ablation studies) than has typically been done until now in the RL research community.\n\nAlthough this paper is pretty straightforward and does not bring meaningful algorithmic improvements, I still believe it should be accepted as reproducibility and evaluation are a major issue in RL, and people need to be aware of these kinds of implementation differences that can affect the reported results.\n\nMy only important concern is that I could not find a link to the code, which I believe is a must for such a paper focusing on implementation. Could the authors please confirm that they will release their code?\n\nOther small remarks:\n\t- Fig. 1 is hard to read, I think more synthetic results could have easily conveyed more clearly the intended message\n\t- When referring to Fig. 2 and 3 please specify \"left\", \"middle\" or \"right\"\n\t- Fig. 2's caption should describe the plots in left to right order (also what does \"maximum versus mean KL\" mean?)\n\t- Fig. 3's caption lists mean KL twice on its first line\n\t- \"The trust region for PPO-NoClip bounds KL to a lesser degree\": this is confusing as it sounds like it is \"less bounded\" while it is actually \"more bounded\" (as said in Fig. 3's caption)\n\t- It would help comparing Fig. 2 and Fig. 3 if they both used the same y axis range\n\t- Typo: \"enforcing\" => enforces\n\nUpdate after author feedback: increasing score to \"Accept\" thanks to the release of the code", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xMvclFtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper calls to attention the importance of specifying all performance altering implementation details that are current inherent in the state-of-the-art deep policy gradient community. Specifically, this paper builds very closely on the work started by  Henderson et al. 2017, building a conversation around the importance of more rigorous and careful scientific study of published algorithms. This paper identifies many \"code-level optimizations\" that account for the differences between the popular TRPO and PPO deep policy gradient algorithms. The paper then subselects four of these optimizations and carefully investigates their impact on the final performance of each algorithm. The clear conclusion from the paper is that the touted algorithmic improvement of PPO over TRPO has negligible effect on performance, and any previously reported differences are due only to what were considered unimportant implementation details.\n\nReview\n\nThis paper investigates the claims made by Schulman et al. 2017 carefully, by investigating the impact of PPO's clipping mechanism on maintaining a valid trust-region; the central claim made by PPO's originating paper. The empirical results suggest that PPO is not sufficient for maintaining a valid trust-region, however the \"code-level optimizations\" that differ between the TRPO implementation the PPO implementation are sufficient. The ablation study of the four optimizations studied by the paper shows dramatic and clear results suggesting that annealing stepsizes and normalize rewards make very strong differences in learning performance; much more effect than demonstrated by the differences between TRPO and PPO's core algorithmic contribution as demonstrated in Figure 2 and even more strongly in Figure 3. I find the work included in this paper to be novel and a valuable contribution to the field.\n\nFor the above reasons, I recommend to accept this paper for publication at ICLR. In the following paragraphs I will discuss why I only recommend a weak accept instead of a strong accept.\n\nMy primary concern with the empirical study is the use of only three random seeds. As demonstrated in Henderson et al. 2017 (which is heavily cited in this paper), using such a small number of random seeds can have very misleading results. Although the effects appear very strong in the empirical studies in this paper, the effects likewise appear strong in Henderson et al.'s Figure 6 where 10 random seeds were split into two groups for the same algorithm. For this paper to make such strong claims about the negligence of the careful scientific study on TRPO and PPO, it would be best if this paper included far more random seeds in its investigation.\n\nMy second concern is with the discussion and conclusions drawn from Tables 1 and 2. It appears that the inclusion of clipping plays a strong role in the variance of each algorithm on every domain except Hopper. Specifically, the algorithms that include clipping appear to be much lower variance than the algorithms including clipping. Admittedly using only 3 seeds means that investigating the variance appropriately is near impossible (see the above paragraph), however variance should be considered and discussed in a conversation about the effects of the core contribution of PPO. If clipping leads to more consistent results across runs, even if those results are a little worse, it is still a valid and important contribution.\n\nThe paper cites Henderson et al. 2017 in several places. I would point out (perhaps in the introduction) that this paper builds on work already done in Henderson et al. 2017. Specifically, Henderson et al. 2017 investigates the effects of using different codebases for TRPO and shows that these different codebases result in dramatically different performance. The similarity to the investigation in this paper to too close to be unreported. However, I find that the investigation in this paper is much more complete and insightful than that of Henderson et al. 2017 (this paper has a more narrow focus), thus contributes significantly and meaningfully to this ongoing conversation.\n\nAdditional Comments (do not affect score)\n\nIt might be worthwhile to move the related work section to the beginning of the paper, either merged with the introduction or immediately after. This section is of critical importance to understanding the scope of this paper and for understanding why you are studying what you study. In fact, there is already a bit of duplication between the related works and introduction sections, so the paper could likely gain some additional real-estate by combining these.\n\nI disagree with the terminology \"code-level optimizations\" and I find that it is misleading. This caused a bit of confusion on my first pass reading the paper, as I originally was expected the code differences to be more akin to using Tensorflow vs PyTorch or switching hash table functions, etc. Instead the changes focused on in this paper are changes to the problem specification and algorithm implementation. These are not simply implementation details as \"code-level optimizations\" suggests, but are rather details that necessarily must be included in peer-reviewed works. I don't have a suggested name to switch to, but felt strongly enough to mention it.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeHgxi6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\nThe papers studies the effects of code-level optimization on the performance of TRPO and PPO. Details, usually considered as implementation-level particularities, are shown to be of crucial importance for the algorithms' performance.\n\n# Decision\nThe paper makes an important point, it is written clearly, and the body of evidence is convincing. Therefore, I recommend this paper for publication.\n\n# Suggestions\nMake it more clear what is meant by code-level optimizations.\n    - In Sec. 2, there is a link to Appendix A.2 for a \"full list\", but the list in A.2 does not contain all points from Sec. 2.\n    - For PPO-M, it is said \"implements only the core of the algorithm\". What exactly does that mean?\n    - PPO-NoClip is defined as \"PPO without clipping\". Does it mean that it includes all other tricks apart from clipping? Please, be explicit in such places.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a careful and well-executed evaluation of the code-level details of two leading policy search algorithms, which are typically considered implementation details and therefore often unstated or brushed aside in papers. These are revealed to have major implications for the performance of both algorithms.

The reviewers are all in agreement that this paper has important reproducibility and evaluation implications for the field, and adds substantially to our body of knowledge on policy gradient algorithms. I therefore recommend it be accepted.

However, a serious limitation is that only 3 random seeds were used to get average performance in the first, key experiment. Experiments are expensive, but that result is not meaningful without more runs, and arguably could be misleading rather than informative. The authors should increase the number of runs as much as possible, at least to 10 but ideally more.
Decision: Accept (Talk)