Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications

1 INTRODUCTION . Recent work in deep reinforcement learning ( RL ) has achieved human level-control for complex tasks like Atari 2600 games and the ancient game of Go . Mnih et al . ( 2015 ) show that it is possible to learn to play Atari 2600 games using end to end reinforcement learning . Other authors ( Silver et al. , 2014 ) derive algorithms tailored to continuous action spaces , such as appear in problems of robotics control . Today , solving RTS games is a major open problem in RL ( Foerster et al. , 2016 ; Usunier et al. , 2017 ; Vinyals et al. , 2017 ) ; these are more challenging than previously solved game domains because the action and state spaces are far larger . In RTS games , actions are no longer chosen from a relatively small discrete action set as in other game types . Neither is the objective solely learning a continuous control . Instead the action space typically consists of many discrete actions each of which has a continuous parameter . For example , a discrete action in an RTS game might be moving the player controlled by the agent with a parameter specifying the movement direction . Because the agent must learn a continuous parameter for each discrete action , a policy gradient method is a natural approach to an RTS game . Unfortunately , obtaining stable , sample-efficient performance from policy gradients remains a key challenge in model-free RL . * These authors contributed equally . †This work was done while at the Tencent AI Lab , Bellevue , WA 98004 . ‡ Correspondence to : eisenach @ princeton.edu . Just as robotics control tasks often have actions restricted to a bounded interval , Multi-player Online Battle Arena ( MOBA ) games , an RTS sub-genre , often have actions restricted to the unit sphere which specify a direction ( e.g . to move or attack ) . The current practice , despite most continuous control problems having bounded action spaces , is to use a Gaussian distribution to model the policy and then apply a transformation T to the action a before execution in the environment . This support mismatch between the sampling action distribution ( i.e . the policy π ) , and the effective action distribution can both introduce bias to and increase the variance of policy gradient estimates ( Chou et al. , 2017 ; Fujita & Maeda , 2018 ) . For an illustration of how the distribution over actions a is transformed under T ( a ) = a/||a|| , see Figure 1 in Section 3 . In this paper , motivated by an application to a MOBA game , we study policy gradient methods in the context of directional actions , something unexplored in the RL literature . Like CAPG for actions in an interval [ α , β ] , our proposed algorithm , termed angular policy gradient ( APG ) , uses a variance-reduced , unbiased estimated of the true policy gradient . Since the key step in APG is an update based on an estimate of the policy gradient , it can easily be combined with other state-of-the art methodology including value function approximation and generalized advantage estimation ( Sutton et al. , 2000 ; Schulman et al. , 2016 ) , as well as used in policy optimization algorithms like TRPO , A3C , and PPO ( Schulman et al. , 2015 ; Mnih et al. , 2016 ; Schulman et al. , 2017 ) . Beyond new methodology , we also introduce the marginal policy gradients ( MPG ) family of estimators ; this general class of estimators contains both APG and CAPG , and we present a unified analysis of the variance reduction properties of all such methods . Because marginal policy gradient methods have already been shown to provide substantial benefits for clipped actions ( Fujita & Maeda , 2018 ) , our experimental work focuses only on angular actions ; we use a marginal policy gradient method to learn a policy for the 1 vs. 1 map of the King of Glory game and the Platform2D-v1 navigation task , demonstrating improvement over several baseline policy gradient approaches . 1.1 RELATED WORK . Model-Free RL . Policy based methods are appealing because unlike value based methods they can support learning policies over discrete , continuous and parametrized action spaces . It has long been recognized that policy gradient methods suffer from high variance , hence the introduction of trust region methods like TRPO and PPO ( Schulman et al. , 2015 ; 2017 ) . Mnih et al . ( 2016 ) leverage the independence of asynchronous updating to improve stability in actor-critic methods . See Sutton & Barto ( 2018 ) for a general survey of reinforcement learning algorithms , including policy based and actor-critic methods . Recent works have applied policy gradient methods to parametrized action spaces in order to teach an agent to play RoboCup soccer ( Hausknecht & Stone , 2016 ; Masson et al. , 2016 ) . Formally , a parametrized action space A over K discrete , parametrized actions is defined as A : = ⋃k { ( k , ω ) : ω ∈ Ωk } , where k ∈ [ K ] and Ωk is the parameter space for the kth action . See Appendix B.5 for rigorous discussion of the construction of a distribution over parametrized action spaces and the corresponding policy gradient algorithms . Bounded Action Spaces . Though the action space for many problems is bounded , it is nonetheless common to model a continuous action using the multivariate Gaussian , which has unbounded support ( Hausknecht & Stone , 2016 ; Florensa et al. , 2017 ; Finn et al. , 2017 ) . Until recently , the method for dealing with this type of action space was to sample according to a Gaussian policy and then either ( 1 ) allow the environment to clip the action and update according to the unclipped action or ( 2 ) clip the action and update according to the clipped action ( Chou et al. , 2017 ) . The first approach suffers from unnecessarily high variance , and the second approach is off-policy . Recent work considers variance reduction when actions are clipped to a bounded interval ( Chou et al. , 2017 ; Fujita & Maeda , 2018 ) . Depending upon the way in which the Q-function is modeled , clipping has also been shown to introduce bias ( Chou et al. , 2017 ) . Previous approaches are not applicable to the case when T is the projection onto the unit sphere ; in the case of clipped actions , unlike previous work , we do not require that each component of the action is independent and obtain much stronger variance reduction results . Concurrent work ( Fellows et al. , 2018 ) also considers angular actions , but their method can not be used as a drop in replacement in state of the art methods and the a special form of the critic qπ is required . Integrated Policy Gradients . Several recent works have considered , as we do , exploiting an integrated form of policy gradient ( Ciosek & Whiteson , 2018 ; Asadi et al. , 2017 ; Fujita & Maeda , 2018 ; Tamar et al. , 2012 ) . Ciosek & Whiteson ( 2018 ) introduces a unified theory of policy gradients , which subsumes both deterministic ( Silver et al. , 2014 ) and stochastic policy gradients ( Sutton et al. , 2000 ) . They characterize the distinction between different policy gradient methods as a choice of quadrature for the expectation . Their Expected Policy Gradient algorithm uses a new way of estimating the expectation for stochastic policies . They prove that the estimator has lower variance than stochastic policy gradients . Asadi et al . ( 2017 ) propose a similar method , but lack theoretical guarantees . Fujita & Maeda ( 2018 ) introduce the clipped action policy gradient ( CAPG ) which is a partially integrated form of policy gradient and provide a variance reduction guarantee , but their result is not tight . By viewing CAPG as a marginal policy gradient we obtain tighter results . Variance Decomposition . The law of total variance , or variance decomposition , is given by Var [ Y ] = E [ Var ( Y |X ) ] + Var [ E [ Y |X ] ] , where X and Y are two random variables on the same probability space . Our main result can be viewed as a special form of law of total variance , but it is highly non-trivial to obtain the result directly from the law of total variance . Also related to our approach is Rao-Blackwellization ( Blackwell , 1947 ) of a statistic to obtain a lower variance estimator . 2 PRELIMINARIES . Notation and Setup . For MDP ’ s we use the standard notation . S is the state space , A is the action space , p denotes the transition probability kernel , p0 the initial state distribution , r the reward function . A policy π ( a|s ) is a distribution over actions given a state s ∈ S. A sample trajectory under π is denoted τπ : = ( s0 , a0 , r1 , s1 , a1 , . . . ) where s0 ∼ p0 and at ∼ π ( ·|st ) . The statevalue function is defined as vπ ( s ) : = Eπ [ ∑∞ t=0 γ trt+1|s0 = s ] and the action-value function as qπ ( s , a ) : = Eπ [ ∑∞ t=0 γ trt+1|s0 = s , a0 = a ] . The objective is to maximize expected cumulative discounted reward , η ( π ) = Ep0 [ vπ ( s0 ) ] . ρπ denotes the improper discounted state occupancy distribution , defined as ρπ : = ∑ t γ tEp0 [ P ( st = s|s0 , π ) ] . We make the standard assumption of bounded rewards . We consider the problem of learning a policy π parametrized by θ ∈ Θ . All gradients are with respect to θ unless otherwise stated . By convention , we define 0 · ∞ = 0 and 00 = 0 . A measurable space ( A , E ) is a set A with a sigma-algebra E of subsets of A . When we refer to a probability distribution of a random variable taking values in ( A , E ) we will work directly with the probability measure on ( A , E ) rather than the underlying sample space . For a measurable mapping T from measure space ( A , E , λ ) to measurable space ( B , F ) , we denote by T∗λ the push-forward of λ. Sd−1 denotes the unit sphere in Rd and for any space A , B ( A ) denotes the Borel σ-algebra on A . The notation µ ν signifies the measure µ is absolutely continuous with respect to ν . The function clip is defined as clip ( a , α , β ) = min ( β , max ( α , a ) ) for a ∈ R. If a ∈ Rd , it is interpreted element-wise . Variance of Random Vectors . We define the variance of a random vector y as Var ( y ) = E [ ( y − Ey ) > ( y − Ey ) ] , i.e . the trace of the covariance of y ; it is easy to verify standard properties of the variance still hold . This definition is often used to analyze the variance of gradient estimates ( Greensmith et al. , 2004 ) . Stochastic Policy Gradients . In Section 4 we present marginal policy gradient estimators and work in the very general setting described below . Let ( A , E , µ ) be a measure space , where as before A is the action space of the MDP . In practice , we often encounter ( A , E ) = ( Rd , B ( Rd ) ) with µ as the Lebesgue measure . The types of policies for which there is a meaningful notation of stochastic policy gradients are µ-compatible measures ( see remarks 2.3 and 2.4 ) . Definition 2.1 ( µ-Compatible Measures ) . Let ( A , E , µ ) be a measure space and consider a parametrized family of measures Π = { π ( · , θ ) : θ ∈ Θ } on the same space . Π is a µ-compatible family of measures if for all θ : ( a ) π ( · , θ ) µ with density of the form fπ ( · , θ ) , ( b ) fπ is differentiable in θ , and ( c ) π satisfies the conditions to apply the Leibniz integral rule for each θ , so that ∇ ∫ A fπ ( a ) dµ = ∫ A∇fπ ( a ) dµ . For µ-compatible policies , Theorem 2.2 gives the stochastic policy gradient , easily estimable from samples . When µ is the counting measure we recover the discrete policy gradient theorem ( Sutton et al. , 2000 ) . See Appendix A.1 for a more in depth discussion and a proof of Theorem 2.2 , which we include for completeness . Theorem 2.2 ( Stochastic Policy Gradient ) . Let ( A , E , µ ) be a measure space and let Π = { π ( · , θ|s ) : θ ∈ Θ } be a family of µ-compatible probability measures . Denoting by fπ the density with respect to µ , we have that ∇η = ∫ S dρπ ( s ) ∫ A qπ ( s , a ) ∇ log fπ ( a|s ) dπ ( ·|s ) . In general we want an estimate g of∇η such that it is unbiased ( E [ g ] = ∇η ) and that has minimal variance , so that convergence to a ( locally ) optimal policy is as fast as possible . In the following sections , we explore a general approach to finding a low variance , unbiased estimator . Remark 2.3 . Under certain choices of T ( e.g . clipping ) the effective action distribution is a mixture of a continuous distribution and point masses . Thus , although it adds some technical overhead , it is necessary that we take a measure theoretic approach in this work . Remark 2.4 . Definition 2.1 is required to ensure the policy gradient is well defined , as it stipulates the existence of an appropriate reference measure ; it also serves to clarify notation and to draw a distinction between π and its density fπ . Though these details are often minimized they are important in analyzing the interaction between T and π .

Summary: In this paper the authors proposed a new policy gradient method, which is known as the angular policy gradient (APG), that aims to provide provably lower variance in the gradient estimate. Here they presented a stochastic policy gradient method for directional control. Under the set of parameterized Gaussian policies, they presented a unified analysis of the variance of APG and showed how it theoretically outperform (in terms of having lower variance) than other state-of-the art methods. They further evaluated the APG algorithms on a grid-world navigation domain as well as the King of Glory task, and showed that the APG estimator significantly out-performs the standard policy gradient.