Implementation Matters in Deep RL: A Case Study on PPO and TRPO

1 INTRODUCTION . Deep reinforcement learning ( RL ) algorithms have fueled many of the most publicized achievements in modern machine learning ( Silver et al. , 2017 ; OpenAI , 2018 ; Abbeel & Schulman , 2016 ; Mnih et al. , 2013 ) . However , despite these accomplishments , deep RL methods still are not nearly as reliable as their ( deep ) supervised learning counterparts . Indeed , recent research found the existing deep RL methods to be brittle ( Henderson et al. , 2017 ; Zhang et al. , 2018 ) , hard to reproduce ( Henderson et al. , 2017 ; Tucker et al. , 2018 ) , unreliable across runs ( Henderson et al. , 2017 ; 2018 ) , and sometimes outperformed by simple baselines ( Mania et al. , 2018 ) . The prevalence of these issues points to a broader problem : we do not understand how the parts comprising deep RL algorithms impact agent training , either separately or as a whole . This unsatisfactory understanding suggests that we should re-evaluate the inner workings of our algorithms . Indeed , the overall question motivating our work is : how do the multitude of mechanisms used in deep RL training algorithms impact agent behavior ? Our contributions . We analyze the underpinnings of agent behavior—both through the traditional metric of cumulative reward , and by measuring more fine-grained algorithmic properties . As a first step towards tackling this question , we conduct a case study of two of the most popular deep policygradient methods : Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015a ) and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) . These two methods are closely related : PPO was originally developed as a refinement of TRPO . We find that much of the PPO ’ s observed improvement in performance comes from seemingly small modifications to the core algorithm that either can be found only in a paper ’ s original implementation , or are described as auxiliary details and are not present in the corresponding TRPO baselines . 1 We pinpoint these modifications , and perform an ablation study demonstrating that they are instrumental to the PPO ’ s performance . ∗Equal contribution . Work done in part as an intern at Two Sigma . 1Note that these code-level optimizations are separate from “ implementation choices ” like the choice of PyTorch versus TensorFlow in that they intentionally change the training algorithm ’ s operation . This observation prompts us to study how such code-level optimizations change agent training dynamics , and whether we can truly think of them as merely auxiliary improvements . Our results indicate that these optimizations fundamentally change algorithms ’ operation , and go even beyond improvements in agent reward . We find that they majorly impact a key algorithmic principle behind TRPO and PPO ’ s operations : trust region enforcement . Ultimately , we discover that the PPO code-optimizations are more important in terms of final reward achieved than the choice of general training algorithm ( TRPO vs. PPO ) . This result is in stark contrast to the previous view that the central PPO clipping method drives the gains seen in Schulman et al . ( 2017 ) . In doing so , we demonstrate that the algorithmic changes imposed by such optimizations make rigorous comparisons of algorithms difficult . Without a rigorous understanding of the full impact of code-level optimizations , we can not hope to gain any reliable insight from comparing algorithms on benchmark tasks . Our results emphasize the importance of building RL methods in a modular manner . To progress towards more performant and reliable algorithms , we need to understand each component ’ s impact on agent behavior and performance—both individually , and as part of a whole . Code for all the results shown in this work is available at https : //github.com/MadryLab/ implementation-matters . 2 RELATED WORK . The idea of using gradient estimates to update neural network–based RL agents dates back at least to the work of Williams ( 1992 ) , who proposed the REINFORCE algorithm . Later , Sutton et al . ( 1999 ) established a unifying framework that casts the previous algorithms as instances of the policy gradient method . Our work focuses on proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) and trust region policy optimization ( TRPO ) ( Schulman et al. , 2015a ) , which are two of the most prominent policy gradient algorithms used in deep RL . Much of the original inspiration for the usage of the trust regions stems from the conservative policy update of Kakade ( 2001 ) . This policy update , similarly to TRPO , uses a natural gradient descent-based greedy policy update . TRPO also bears similarity to the relative policy entropy search method of Peters et al . ( 2010 ) , which constrains the distance between marginal action distributions ( whereas TRPO constrains the conditionals of such action distributions ) . Notably , Henderson et al . ( 2017 ) points out a number of brittleness , reproducibility , and experimental practice issues in deep RL algorithms . Importantly , we build on the observation of Henderson et al . ( 2017 ) that final reward for a given algorithm is greatly influenced depending on the code base used . Rajeswaran et al . ( 2017 ) and Mania et al . ( 2018 ) also demonstrate that on many of the benchmark tasks , the performance of PPO and TRPO can be matched by fairly elementary randomized search approaches . Additionally , Tucker et al . ( 2018 ) showed that one of the recently proposed extensions of the policy gradient framework , i.e. , the usage of baseline functions that are also actiondependent ( in addition to being state-dependent ) , might not lead to better policies after all . 3 ATTRIBUTING SUCCESS IN PROXIMAL POLICY OPTIMIZATION . Our overarching goal is to better understand the underpinnings of the behavior of deep policy gradient methods . We thus perform a careful study of two tightly linked algorithms : TRPO and PPO ( recall that PPO is motivated as TRPO with a different trust region enforcement mechanism ) . To better understand these methods , we start by thoroughly investigating their implementations in practice . We find that in comparison to TRPO , the PPO implementation contains many non-trivial optimizations that are not ( or only barely ) described in its corresponding paper . Indeed , the standard implementation of PPO 2 contains the following additional optimizations : 2From the OpenAI baselines GitHub repository : https : //github.com/openai/baselines 1 . Value function clipping : Schulman et al . ( 2017 ) originally suggest fitting the value network via regression to target values : LV = ( Vθt − Vtarg ) 2 , but the standard implementation instead fits the value network with a PPO-like objective : LV = min [ ( Vθt − Vtarg ) 2 , ( clip ( Vθt , Vθt−1 − ε , Vθt−1 + ε ) − Vtarg ) 2 ] , where Vθ is clipped around the previous value estimates ( and ε is fixed to the same value as the value used in ( 2 ) to clip the probability ratios ) . 2 . Reward scaling : Rather than feeding the rewards directly from the environment into the objective , the PPO implementation performs a certain discount-based scaling scheme . In this scheme , the rewards are divided through by the standard deviation of a rolling discounted sum of the rewards ( without subtracting and re-adding the mean ) —see Algorithm 1 in Appendix A.2 . 3 . Orthogonal initialization and layer scaling : Instead of using the default weight initialization scheme for the policy and value networks , the implementation uses an orthogonal initialization scheme with scaling that varies from layer to layer . 4 . Adam learning rate annealing : Depending on the task , the implementation sometimes anneals the learning rate of Adam ( Kingma & Ba , 2014 ) ( an already adaptive method ) for optimization . 5 . Reward Clipping : The implementation also clips the rewards within a preset range ( usually [ −5 , 5 ] or [ −10 , 10 ] ) . 6 . Observation Normalization : In a similar manner to the rewards , the raw states are also not fed into the optimizer . Instead , the states are first normalized to mean-zero , variance-one vectors . 7 . Observation Clipping : Analagously to rewards , the observations are also clipped within a range , usually [ −10 , 10 ] . 8 . Hyperbolic tan activations : As also observed by Henderson et al . ( 2017 ) , implementations of policy gradient algorithms also also use hyperbolic tangent function activations between layers in the policy and value networks . 9 . Global Gradient Clipping : After computing the gradient with respect to the policy and the value networks , the implementation clips the gradients such the “ global ` 2 norm ” ( i.e . the norm of the concatenated gradients of all parameters ) does not exceed 0.5 . These optimizations may appear as merely surface-level or insignificant algorithmic changes to the core policy gradient method at hand . However , we find that they dramatically affect the performance of PPO . To demonstrate this , we start by performing a full ablation study on the four optimizations mentioned above 3 . Figure 1 shows a histogram of the final rewards of agents trained with every possible configuration of the above optimizations—for each configuration , a grid search for the optimal learning rate is performed , and we measure the reward of random agents trained using the identified learning rate . Our findings suggest that many code-level optimizations are necessary for PPO to attain its claimed performance . The above findings show that our ability to understand PPO from an algorithmic perspective hinges on the ability to distill out its fundamental principles from such algorithm-independent ( in the sense that these optimizations can be implemented for any policy gradient method ) optimizations . We thus consider a variant of PPO called PPO-MINIMAL ( PPO-M ) which implements only the core of the algorithm . PPO-M uses the standard value network loss , no reward scaling , the default network initialization , and Adam with a fixed learning rate . Importantly , PPO-M ignores all the code-level optimizations listed above in the beginning of Section 3 . We then explore PPO-M alongside PPO and TRPO . We list all the algorithms we study and their defining properties in Table 1 . Overall , our results on the importance of these optimizations both corroborate results demonstrating the brittleness of deep policy gradient methods , and demonstrate that even beyond environmental brittleness , the algorithms themselves exhibit high sensitivity to implementation choices 4 . 3Due to restrictions on computational resources , we could only perform a full ablation on the first four of the identified optimizations . 4This might also explain the difference between different codebases observed in Henderson et al . ( 2017 ) 4 CODE-LEVEL OPTIMIZATIONS HAVE ALGORITHMIC EFFECTS . In the previous section , we found that canonical implementations of PPO contain many code-level optimizations : implementation choices that are not integral to the method but profoundly impact performance . The seemingly disproportionate effect of code-level optimizations identified in our ablation study may lead us to ask : how do these seemingly superficial code-level optimizations impact underlying agent behavior ? In this section , we demonstrate that the code-level optimizations fundamentally alter agent behavior . Rather than merely improving ultimate cumulative award , such optimizations directly impact the principles motivating the core algorithms . Trust Region Optimization . A key property of policy gradient algorithms is that update steps computed at any specific policy πθt are only guaranteed predictiveness in a neighborhood around θt . Thus , to ensure that the update steps we derive remain predictive many policy gradient algo- rithms ensure that these steps stay in the vicinity of the current policy . The resulting “ trust region ” methods ( Kakade , 2001 ; Schulman et al. , 2015a ; 2017 ) try to constrain the local variation of the parameters in policy-space by restricting the distributional distance between successive policies . A popular method in this class is trust region policy optimization ( TRPO ) Schulman et al . ( 2015a ) . TRPO constrains the KL divergence between successive policies on the optimization trajectory , leading to the following problem : max θ E ( st , at ) ∼π [ πθ ( at|st ) π ( at|st ) Âπ ( st , at ) ] s.t . DKL ( πθ ( · | s ) ||π ( · | s ) ) ≤ δ , ∀s . ( 1 ) In practice , we maximize this objective with a second-order approximation of the KL divergence and natural gradient descent , and replace the worst-case KL constraints over all possible states with an approximation of the mean KL based on the states observed in the current trajectory . Proximal policy optimization . One disadvantage of the TRPO algorithm is that it can be computationally costly—the step direction is estimated with nonlinear conjugate gradients , which requires the computation of multiple Hessian-vector products . To address this issue , Schulman et al . ( 2017 ) propose proximal policy optimization ( PPO ) , which tries to enforce a trust region with a different objective that does not require computing a projection . Concretely , PPO proposes replacing the KL-constrained objective ( 1 ) of TRPO by clipping the objective function directly as : max θ E ( st , at ) ∼π [ min ( clip ( ρt , 1− ε , 1 + ε ) Âπ ( st , at ) , ρtÂπ ( st , at ) ) ] ( 2 ) where ρt = πθ ( at|st ) π ( at|st ) . ( 3 ) Note that this objective can be optimized without an explicit projection step , leading to a simpler parameter update during training . In addition to its simplicity , PPO is intended to be faster and more sample-efficient than TRPO ( Schulman et al. , 2017 ) . Trust regions in TRPO and PPO . Enforcing a trust region is a core algorithmic property of different policy gradient methods . However , whether or not a trust region is enforced is not directly observable from the final rewards . So , how does this algorithmic property vary across state-of-the-art policy gradient methods ? In Figure 2 we measure the mean KL divergence between successive policies in a training run of both TRPO and PPO-M ( PPO without code-level optimizations ) . Recall that TRPO is designed specifically to constrain this KL-based trust region , while the clipping mechanism of PPO attempts to approximate it . Indeed , we find that TRPO precisely enforces this trust region ( this is unsuprising , and nearly by construction ) . We thus turn our attention to the trust regions induced by training with PPO and PPO-M. First , we consider mathematically the contribution of a single state-action pair to the gradient of the PPO objective , which is given by ∇θLPPO = { ∇θLθ πθ ( a|s ) π ( a|s ) ∈ [ 1− , 1 + ] or L C θ < Lθ 0 otherwise , where Lθ : = E ( s , a ) ∈τ∼π [ πθ ( a|s ) π ( a|s ) Aπ ( s , a ) ] , and LCθ : = E ( s , a ) ∈τ∼π [ clip ( πθ ( a|s ) π ( a|s ) , 1− ε , 1 + ε ) Aπ ( s , a ) ] are respectively the standard and clipped versions of the surrogate objective . As a result , since we initialize πθ as π ( and thus the ratios start all equal to one ) the first step we take is identical to a maximization step over the unclipped surrogate objective . It thus stands to reason that the nature of the trust region enforced is heavily dependent on the method with which the clipped PPO objective is optimized , rather than on the objective itself . Therefore , the size of step we take is determined solely be the steepness of the surrogate landscape ( i.e . Lipschitz constant of the optimization problem we solve ) , and we can end up moving arbitrarily far from the trust region . We hypothesize that this dependence of PPO on properties of the optimizer rather than the optimization objective contributes to the brittleness of the algorithm to hyperparameters such as learning rate and momentum , as observed by Henderson et al . ( 2018 ) and others . The results we observe ( shown in Figure 2 ) corroborate this intuition . For agents trained with optimal parameters , all three algorithms are able to maintain a KL-based trust region . First , we note that all three algorithms fail to maintain a ratio-based trust region , despite PPO and PPO-M being trained directly with a ratio-clipping objective . Furthermore , the nature of the KL trust region enforced differs between PPO and PPO-M , despite the fact that the core algorithm remains constant between the two methods ; while PPO-M KL trends up as the number of iterations increases , PPO KL peaks halfway through training before trending down again . The findings from this experiment and the corresponding calculations demonstrate that perhaps a key factor in the behavior of PPO-trained agents even from an algorithmic viewpoint comes from auxiliary optimizations , rather than the core methodology .

Summary: This paper calls to attention the importance of specifying all performance altering implementation details that are current inherent in the state-of-the-art deep policy gradient community. Specifically, this paper builds very closely on the work started by  Henderson et al. 2017, building a conversation around the importance of more rigorous and careful scientific study of published algorithms. This paper identifies many "code-level optimizations" that account for the differences between the popular TRPO and PPO deep policy gradient algorithms. The paper then subselects four of these optimizations and carefully investigates their impact on the final performance of each algorithm. The clear conclusion from the paper is that the touted algorithmic improvement of PPO over TRPO has negligible effect on performance, and any previously reported differences are due only to what were considered unimportant implementation details.