Adaptive Discretization for Continuous Control using Particle Filtering Policy Network

1 INTRODUCTION . In the last few years , impressive results have been obtained by deep reinforcement learning ( DRL ) both on physical and simulated articulated agents for a wide range of motor tasks that involve learning controls in high-dimensional continuous action spaces ( Lillicrap et al. , 2015 ; Levine et al. , 2016 ; Heess et al. , 2017 ; Haarnoja et al. , 2018c ; Rajeswaran et al. , 2018 ; Tan et al. , 2018 ; Peng et al. , 2018 ; 2020 ) . Many methods have been proposed that can improve the performance of DRL for continuous control problems , e.g . distributed training ( Mnih et al. , 2016 ; Espeholt et al. , 2018 ) , hierarchical learning ( Daniel et al. , 2012 ; Haarnoja et al. , 2018a ) , and maximum entropy regularization ( Haarnoja et al. , 2017 ; Liu et al. , 2017 ; Haarnoja et al. , 2018b ) . Most of such works , though , focus on learning mechanisms to boost performance beyond the basic distribution that defines the action policy , where a Gaussian-based policy or that with a squashing function is the most common choice as the basic policy to deal with continuous action spaces . However , the unimodal form of Gaussian distributions could experience difficulties when facing a multi-modal reward landscape during optimization and prematurely commit to suboptimal actions ( Daniel et al. , 2012 ; Haarnoja et al. , 2017 ) . To address the unimodality issue of Gaussian policies , people have been exploring more expressive distributions than Gaussians , with a simple solution being to discretize the action space and use categorical distributions as multi-modal action policies ( Andrychowicz et al. , 2020 ; Jaśkowski et al. , 2018 ; Tang & Agrawal , 2019 ) . However , categorical distributions can not be directly extended to many off-policy frameworks as their sampling process is not reparameterizable . Importantly , the performance of the action space discretization depends a lot on the choice of discrete atomic actions , which are usually picked uniformly due to lack of prior knowledge . On the surface , increasing the resolution of the discretized action space can make fine control more possible . However , in practice , this can be detrimental to the optimization during training , since the policy gradient variance increases with increasing number of atomic actions ( Tang & Agrawal , 2019 ) . Our work also focuses on action policies defined by an expressive , multimodal distribution . Instead of selecting fixed samples from the continuous action space , though , we exploit a particlebased approach to sample the action space dynamically during training and track the policy represented as a mixture distribution with state-independent components . We refer to the resulting policy network as Particle Filtering Policy Network ( PFPN ) . We evaluate PFPN on state-of-the-art on-policy and off-policy baselines using high-dimensional tasks from the PyBullet Roboschool en- vironments ( Coumans & Bai , 2016–2019 ) and the more challenging DeepMimic framework ( Peng et al. , 2018 ) . Our experiments show that baselines using PFPN exhibit better overall performance and/or speed of convergence and lead to more robust agent control . as compared to uniform discretization and to corresponding implementations with Gaussian policies . Main Contributions . Overall , we make the following contributions . We propose PFPN as a general framework for providing expressive action policies dealing with continuous action spaces . PFPN uses state-independent particles to represent atomic actions and optimizes their placement to meet the fine control demand of continuous control problems . We introduce a reparameterization trick that allows PFPN to be applicable to both on-policy and off-policy policy gradient methods . PFPN outperforms unimodal Gaussian policies and the uniform discretization scheme , and is more sampleefficient and stable across different training trials . In addition , it leads to high quality motion and generates controls that are more robust to external perturbations . Our work does not change the underlying model architecture or learning mechanisms of policy gradient algorithms and thus can be applied to most commonly used policy gradient algorithms . 2 BACKGROUND . We consider a standard reinforcement learning setup where given a time horizonH and the trajectory τ = ( s1 , a1 , · · · , sH , aH ) obtained by a transient modelM ( st+1|st , at ) and a parameterized action policy πθ ( at|st ) , with st ∈ Rn and at ∈ Rm denoting the state and action taken at time step t , respectively , the goal of learning is to optimize θ that maximize the cumulative reward : J ( θ ) = Eτ∼pθ ( τ ) [ rt ( τ ) ] = ∫ pθ ( τ ) r ( τ ) dτ . ( 1 ) Here , pθ ( τ ) denotes the state-action visitation distribution for the trajectory τ induced by the transient model M and the action policy πθ with parameter θ , and r ( τ ) = ∑ t r ( st , at ) where r ( st , at ) ∈ R is the reward received at time step t. We can maximize J ( θ ) by adjusting the policy parameters θ through the gradient ascent method , where the gradient of the expected reward can be determined according to the policy gradient theorem ( Sutton et al. , 2000 ) , i.e . ∇θJ ( θ ) = Eτ∼πθ ( ·|st ) [ At∇θ log πθ ( at|st ) |st ] . ( 2 ) whereAt ∈ R denotes an estimate to the reward term rt ( τ ) . In DRL , the estimator ofAt often relies on a separate network ( critic ) that is updated in tandem with the policy network ( actor ) . This gives rise to a family of policy gradient algorithms known as actor-critic . On-Policy and Off-Policy Actor-Critics . In on-policy learning , the update policy is also the behavior policy based on which a trajectory is obtained to estimate At . Common on-policy actor-critic algorithms include A3C ( Mnih et al. , 2016 ) and PPO ( Schulman et al. , 2017 ) , and directly employ Equation 2 for optimization . In off-policy learning , the policy can be updated without the knowledge of a whole trajectory . This results in more sample efficient approaches as samples are reusable . While algorithms such as Retrace ( Munos et al. , 2016 ) and PCL ( Nachum et al. , 2017 ) rely on Equation 2 , many off-policy algorithms exploit a critic network to estimate At given a state-action pair ( Q- or soft Q-value ) . Common off-policy actor-critic methods include DDPG ( Lillicrap et al. , 2015 ) , SAC ( Haarnoja et al. , 2018b ; d ) and their variants ( Haarnoja et al. , 2017 ; Fujimoto et al. , 2018 ) . These methods perform optimization to maximize a state-action value Q ( st , at ) . In order to update the policy network with parameter θ , they require the action policy to be reparameterizable such that the sampled action at can be rewritten as a function differentiable to the parameter θ , and the optimization can be done through the gradient∇atQ ( st , at ) ∇θat . Policy Representation . Given a multi-dimensional continuous action space , the most common choice in current DRL baselines is to model the policy πθ as a multivariate Gaussian distribution with independent components for each action dimension ( DDPG , SAC and their variants typically use Gaussian with a monotonic squashing function to stabilize the training ) . For simplicity , let us consider a simple case with a single action dimension and define the action policy as πθ ( ·|st ) : = N ( µθ ( st ) , σ2θ ( st ) ) . Then , we can obtain log πθ ( at|st ) ∝ − ( at − µθ ( st ) ) 2 . Given a sampled action at and the estimate of cumulative rewards At , the optimization process based on the above expression can be imagined as that of shifting µ θ ( st ) towards the direction of at ifAt is higher than the expectation , or to the opposite direction if At is smaller . Such an approach , though , can easily converge to a suboptimal solution , if , for example , the reward landscape has a basis between the current location of µ θ ( st ) and the optimal solution , or hard to be optimized if the reward land- scape is symmetric around µ θ ( st ) . These issues arise due to the fact that the Gaussian distribution is inherently unimodal , while the reward landscape could be multi-modal ( Haarnoja et al. , 2017 ) . Similar problems also could occur in Q-value based optimization , like DDPG and SAC . We refer to Appendix F for further discussion about the limitations of unimodal Gaussian policies and the value of expressive multimodal policies . 3 PARTICLE FILTERING POLICY NETWORK . In this section , we describe our Particle Filtering Policy Network ( PFPN ) that addresses the unimodality issues from which typical Gaussian-based policy networks suffer . Our approach represents the action policy as a mixture distribution obtained by adaptively discretizing the action space using state-independent particles , each capturing a Gaussian distribution . The policy network , instead of directly generating actions , it is tasked with choosing particles , while the final actions are obtained by sampling from the selected particles . 3.1 PARTICLE-BASED ACTION POLICY . Generally , we define P : = { 〈µi , k , wi , k ( st|θ ) 〉|i = 1 , · · · , n ; k = 1 , · · · , m } as a weighted set of particles for continuous control problems with a m-dimension action space and n particles distributed on each action space , where µi , k ∈ R representing an atomic action location on the kth dimension of the action space , and wi , k ( st|θ ) , satisfying ∑ i wi , k ( st|θ ) = 1 , denotes the associated weight generated by the policy network with parameter θ given the input state st. Let pi , k ( ai , k|µi , k , ξi , k ) denote the probability density function of the distribution defined by the location µi , k and a noise process ξi , k . Given P , we define the action policy as factorized across the action dimensions : πPθ ( at|st ) = ∏ k ∑ i wi , k ( st|θ ) pi , k ( at , k|µi , k , ξi , k ) , ( 3 ) where at = { at,1 , · · · , at , m } , at , k ∈ R is the sampled action at the time step t for the action dimension k , and wi , k ( ·|θ ) can be obtained by applying a softmax operation to the output neurons of the policy network for the k-th dimension . The state-independent parameter set , { µi , k } , gives us an adaptive discretization scheme that can be optimized during training . The choice of noise ξi , k relies on certain algorithms . It can be a scalar , e.g. , the Ornstein–Uhlenbeck noise in DDPG ( Lillicrap et al. , 2015 ) or an independent sample drawn from the standard normal distribution N ( 0 , 1 ) in soft Q-learning ( Haarnoja et al. , 2017 ) , or be decided by a learnable variable , for example , a sample drawn fromN ( 0 , ξ2i , k ) with a learnable standard deviation ξi , k . In the later case , a particle become a Gaussian componentN ( µi , k , ξ2i , k ) . Without loss of generality , we define the parameters of a particle as φi , k = [ µi , k , ξi , k ] for the following discussion . While the softmax operation gives us a categorical distribution defined by w· , k ( st|θ ) , the nature of the policy for each dimension is a mixture distribution with state-independent components defined by φi , k . The number of output neurons in PFPN increases linearly with the increase in the number of action dimensions and thus makes it suitable for high-dimensional control problems . Drawing samples from the mixture distribution can be done in two steps : first , based on the weights w· , k ( st|θ ) , we perform sampling on the categorical distribution to choose a particle jk for each dimension k , i.e . jk ( st ) ∼ P ( ·|w· , k ( st ) ) ; then , we can draw actions from the components represented by the chosen particles with noise as at , k ∼ pjk ( st ) ( ·|φjk ( st ) ) . Certain algorithms , like A3C and IMPALA , introduce differential entropy loss to encourage exploration . However , it may be infeasible to analytically evaluate the differential entropy of a mixture distribution without approximation ( Huber et al. , 2008 ) . As such , we use the entropy of the categorical distribution if a differential entropy term is needed during optimization . We refer to Appendix C for the action policy representation in DDPG and SAC where an invertible squashing function is applied to Gaussian components .

Summary: This paper presents an approach to multimodal policies based on Gaussian mixtures. The policy is parameterized as a set of Gaussian distributions (with state-invariant mean and variance) weighted by state-dependent mixture weights, which are the output of a (softmaxed) network. The weighting network and the means and covariances of the Gaussians are updated with standard RL losses. The authors propose a resampling scheme for mixture elements that consistently have low weight, in the style of resampling in particle filters. The authors evaluate the method with several RL algorithms including PPO and SAC, and on a variety of environments.