A Closer Look at Deep Policy Gradients

1 INTRODUCTION . Deep reinforcement learning ( RL ) is behind some of the most publicized achievements of modern machine learning ( Silver et al. , 2017 ; OpenAI , 2018 ; Dayarathna et al. , 2016 ; OpenAI et al. , 2018 ) . In fact , to many , this framework embodies the promise of the real-world impact of machine learning . However , the deep RL toolkit has not yet attained the same level of engineering stability as , for example , the current deep ( supervised ) learning framework . Indeed , recent studies demonstrate that state-of-the-art deep RL algorithms suffer from oversensitivity to hyperparameter choices , lack of consistency , and poor reproducibility ( Henderson et al. , 2017 ) . This state of affairs suggests that it might be necessary to re-examine the conceptual underpinnings of deep RL methodology . More precisely , the overarching question that motivates this work is : To what degree does current practice in deep RL reflect the principles informing its development ? . Our specific focus is on deep policy gradient methods , a widely used class of deep RL algorithms . Our goal is to explore the extent to which state-of-the-art implementations of these methods succeed at realizing the key primitives of the general policy gradient framework . Our contributions . We take a broader look at policy gradient algorithms and their relation to their underlying framework . With this perspective in mind , we perform a fine-grained examination of key RL primitives as they manifest in practice . Concretely , we study : • Gradient Estimation : we find that even when agents improve in reward , their gradient estimates used in parameter updates poorly correlate with the “ true ” gradient . We additionally show that gradient estimate quality decays with training progress and task complexity . Finally , we demonstrate that varying the sample regime yields training dynamics that are unexplained by the motivating framework and run contrary to supervised learning intuition . • Value Prediction : our experiments indicate that value networks successfully solve the supervised learning task they are trained on , but do not fit the true value function . Additionally , employing a value network as a baseline function only marginally decreases the ∗Equal contribution . Work done in part as an intern at Two Sigma . variance of gradient estimates compared to using true value as a baseline ( but still dramatically increases agent ’ s performance compared to using no baseline at all ) . • Optimization Landscapes : we show that the optimization landscape induced by modern policy gradient algorithms is often not reflective of the underlying true reward landscape , and that the latter is frequently poorly behaved in the relevant sample regime . Overall , our results demonstrate that the motivating theoretical framework for deep RL algorithms is often unpredictive of phenomena arising in practice . This suggests that building reliable deep RL algorithms requires moving past benchmark-centric evaluations to a multi-faceted understanding of their often unintuitive behavior . We conclude ( in Section 3 ) by discussing several areas where such understanding is most critically needed . 2 EXAMINING THE PRIMITIVES OF DEEP POLICY GRADIENT ALGORITHMS . In this section , we investigate the degree to which our theoretical understanding of RL applies to modern methods . We consider key primitives of policy gradient algorithms : gradient estimation , value prediction and reward fitting . In what follows , we perform a fine-grained analysis of state-ofthe-art policy gradient algorithms ( PPO and TRPO ) through the lens of these primitives—detailed preliminaries , background , and notation can be found in Appendix A.1 . 2.1 GRADIENT ESTIMATE QUALITY . A central premise of policy gradient methods is that stochastic gradient ascent on a suitable objective function yields a good policy . These algorithms use as a primitive the gradient of that objective function : ĝ = ∇θE ( st , at ) ∼π0 [ πθ ( at|st ) π0 ( at|st ) Âπ0 ( st , at ) ] = E ( st , at ) ∼π0 [ ∇θπθ ( at|st ) π0 ( at|st ) Âπ0 ( st , at ) ] , ( 1 ) where in the above we use standard RL notation ( see Appendix A.1 for more details ) . An underlying assumption behind these methods is that we have access to a reasonable estimate of this quantity . This assumption effectively translates into an assumption that we can accurately estimate the expectation above using an empirical mean of finite ( typically ∼ 103 ) samples . Evidently ( since the agent attains a high reward ) these estimates are sufficient to consistently improve reward—we are thus interested in the relative quality of these gradient estimates in practice , and the effect of gradient quality on optimization . How accurate are the gradient estimates we compute ? To answer this question , we examine two of the most natural measures of estimate quality : the empirical variance and the convergence to the “ true ” gradient . To evaluate the former , we measure the average pairwise cosine similarity between estimates of the gradient computed from the same policy with independent rollouts ( Figure 1 ) . We evaluate the latter by first forming an estimate of the true gradient with a large number of state-action pairs . We then examine the convergence of gradient estimates to this “ true ” gradient ( which we once again measure using cosine similarity ) as we increase the number of samples ( Figure 2 ) . We observe that deep policy gradient methods operate with relatively poor estimates of the gradient , especially as task complexity increases and as training progresses ( contrast Humanoid-v2 , a “ hard ” task , to other tasks and contrast successive checkpoints in Figures 1 and 2 ) . This is in spite of the fact that our agents continually improve throughout training , and attain nowhere near the maximum reward possible on each task . In fact , we sometimes observe a zero or even negative correlation in the relevant sample regime1 . While these results might be reminiscent of the well-studied “ noisy gradients ” problem in supervised learning ( Robbins & Monro , 1951 ; d ’ Aspremont , 2008 ; Kawaguchi , 2016 ; Safran & Shamir , 2018 ; Livni et al. , 2014 ; Keskar et al. , 2016 ; Hochreiter & Schmidhuber , 1997 ) , we have very little understanding of how gradient quality affects optimization in the substantially different reinforcement learning setting . For example : • The sample regime in which RL algorithms operate seems to have a profound impact on the robustness and stability of agent training—in particular , many of the sensitivity issues reported by Henderson et al . ( 2017 ) are claimed to disappear ( Sutskever , 2018 ) in highersample regimes . Understanding the implications of working in this sample regime , and more generally the impact of sample complexity on training stability remains to be precisely understood . • Agent policy networks are trained concurrently with value networks ( discussed more in the following section ) meant to reduce the variance of gradient estimates . Under our conceptual framework , we might expect these networks to help gradient estimates more as training progresses , contrary to what we observe in Figure 1 . The value network also makes the now two-player optimization landscape and training dynamics even more difficult to grasp , as such interactions are poorly understood . 1Deep policy gradient algorithms use gradients indirectly to compute steps—in Appendix A.4 we show that our results also hold true for these computed steps . • The relevant measure of sample complexity for many settings ( number of state-action pairs ) can differ drastically from the number of independent samples used at each training iteration ( the number of complete trajectories ) . The latter quantity ( a ) tends to be much lower than the number of state-action pairs , and ( b ) decreases across iterations during training . All the above factors make it unclear to what degree our intuition from classical settings transfer to the deep RL regime . And the policy gradient framework , as of now , provides little predictive power regarding the variance of gradient estimates and its impact on reward optimization . Our results indicate that despite having a rigorous theoretical framework for RL , we lack a precise understanding of the structure of the reward landscape and optimization process . 2.2 VALUE PREDICTION . Our findings from the previous section motivate a deeper look into gradient estimation . After all , the policy gradient in its original formulation ( Sutton et al. , 1999 ) is known to be hard to estimate , and thus algorithms employ a variety of variance reduction methods . The most popular of these techniques is a baseline function . Concretely , an equivalent form of the policy gradient is given by : ĝθ = Eτ∼πθ ∑ ( st , at ) ∈τ ∇θ log πθ ( at|st ) · ( Qπθ ( st , at ) − b ( st ) ) ( 2 ) where b ( st ) is some fixed function of the state st. A canonical choice of baseline function is the value function Vπ ( s ) , the expected return from a given state ( more details and motivation in A.1 ) : Vπθ ( st ) = Eπθ [ Rt|st ] . ( 3 ) Indeed , fitting a value-estimating function ( Schulman et al. , 2015c ; Sutton & Barto , 2018 ) ( a neural network , in the deep RL setting ) and using it as a baseline function is precisely the approach taken by most deep policy gradient methods . Concretely , one trains a value network V πθt such that : θt = min θ E [ ( V πθ ( st ) − ( V πθt−1 ( st ) +At ) ) 2 ] ( 4 ) where V πθt−1 ( st ) are estimates given by the last value function , andAt is the advantage of the policy , i.e . the returns minus the estimated values . ( Typically , At is estimated using generalized advantage estimation , as described in ( Schulman et al. , 2015c ) . ) Our findings in the previous section prompt us to take a closer look at the value network and its impact on the variance of gradient estimates . Value prediction as a supervised learning problem . We first analyze the value network through the lens of the supervised learning problem it solves . After all , ( 4 ) describes an empirical risk minimization , where a loss is minimized over a set of sampled ( st , at ) . So , how does V πθ perform as a solution to ( 4 ) ? And in turn , how does ( 4 ) perform as a proxy for learning the true value function ? Our results ( Figure 3a ) show that the value network does succeed at both fitting the given loss function and generalizing to unseen data , showing low and stable mean relative error ( MRE ) . However , the significant drop in performance as shown in Figure 3 indicates that the supervised learning problem induced by ( 4 ) does not lead to V πθ learning the underlying true value function . Does the value network lead to a reduction in variance ? Though evaluating the V πθ baseline function as a value predictor as we did above is informative , in the end the sole purpose of the value function is to reduce variance . So : how does using our value function actually impact the variance of our gradient estimates ? To answer this question , we compare the variance reduction that results from employing our value network against both a “ true ” value function and a trivial “ zero ” baseline function ( i.e . simply replacing advantages with returns ) . Our results , captured in Figure 4 , show that the “ true ” value function yields a much lower-variance estimate of the gradient . This is especially true in the sample regime in which we operate . We note , however , that despite not effectively predicting the true value function or inducing the same degree of variance reduction , the value network does help to some degree ( compared to the “ zero ” baseline ) . Additionally , the seemingly marginal increase in gradient correlation provided by the value network ( compared to the “ true ” baseline function ) turns out to result in a significant improvement in agent performance . ( Indeed , agents trained without a baseline reach almost an order of magnitude worse reward . ) Our findings suggest that we still need a better understanding of the role of the value network in agent training , and raise several questions that we discuss in Section 3 .

Summary: This paper empirically studies the behavior of deep policy gradient algorithms during the optimization. The conclusion is that, while these methods generally improve the policy, their behavior does not comply with the underlying theoretical framework. First, sample gradients obtained with a reasonable batch size have little correlation with each other and with the true gradient. Second, a larger batch size requires a smaller step-size. Third, the value baseline is far from true values and only marginally reduces variance, yet it considerably helps with optimization. Finally, the optimization landscape highly varies with the choice of objective function and the number of samples used to estimate it.