D3PG: Deep Differentiable Deterministic Policy Gradients

1 INTRODUCTION . In recent years , deep reinforcement learning ( DRL ) has emerged as a flexible and robust means of teaching simulated robots to complete complex tasks , from manipulation ( Kumar et al. , 2016 ) and locomotion ( Haarnoja et al. , 2018b ) , to navigating complex terrain ( Peng et al. , 2016 ) . Compared with more direct optimization methods such as gradient descent or second-order optimization , DRL naturally incorporates exploration into its planning , allowing it to learn generalizable policies and robust state value estimations across simulated environments . Perhaps the most salient reason for DRL ’ s surge in popularity is its ability to operate on black-box simulators where the underlying dynamics model is not available . DRL ’ s model-free , Monte-Carlo-style methods have made it applicable to a wide range of physical ( and non-physical ) simulation environments , including those where a smooth , well-behaved dynamical model does not exist . This comes at two striking costs . First , such sampling procedures may be inefficient , requiring a large number of samples for adequate learning . Second , in order to be generally applicable to any model-free environment , underlying dynamical gradients are not used , even if they are available . In other words , valuable information that could greatly aid control tasks is not taken advantage of in these schemes . When an accurate model of robot dynamics is given , model-based methods such as model-predictive control ( MPC ) or trajectory optimization have historically been employed . These methods can solve tasks with higher sample efficiency than model-free DRL algorithms . Models provide access to ground-truth , analytical gradients of robot physics without the need for sample-based estimation . However , such methods don ’ t incorporate exploration or learning into their procedures , and are especially prone to becoming trapped in poor local minima . While there has been a recent surge in fast and accurate differentiable simulators not previously available , most applications for control have relied on established local methods such as MPC ( de Avila Belbute-Peres et al. , 2018 ) , gradient descent ( Degrave et al. , 2019 ) , or trajectory optimization ( Hu et al. , 2019 ) to solve control tasks . An ideal algorithm would exploit the efficiency of model-based methods while maintaining DRL ’ s relative robustness to poor local minima . In this paper , we propose an actor-critic algorithm that leverages differentiable simulation and combines the benefits of model-based methods and DRL . We build our method upon standard actor-critic DRL algorithms and use true model gradients in order to improve the efficacy of learned critic models . Our main insights are twofold : First , gradients of critics play an important role in certain DRL algorithms , but optimization of these critics ’ gradients has not been explored by previous work . Second , the emergence of differentiable simulators enables computation of advantage estimation ( AE ) gradients with little additional computational overhead . Based on these observations , we present an algorithm that uses AE gradients in order to co-learn critic value and gradient estimation , demonstrably improving convergence of both actor and critic . In this paper , we contribute the following : 1 ) An efficient hybrid actor-critic method which builds upon deep deterministic policy gradients ( DDPG , ( Lillicrap et al. , 2015 ) ) , using gradient information in order to improve convergence in a simple way . 2 ) A principled mathematical framework for fitting critic gradients , providing a roadmap for applying our method to any deterministic policy gradient method , and 3 ) Demonstrations of our algorithm on seven control tasks , ranging from contact-free classic control problems to complex tasks with accurate , hard contact , such as the HalfCheetah , along with comparisons to both model-based control and DRL baselines . 2 RELATED WORK . Model-based control methods peer into the underlying dynamics of a system in order to optimize control parameters . Such approaches have especially grown in popularity with the rise of differentiable simulators , whose application now spans rigid body simulation ( de Avila Belbute-Peres et al. , 2018 ; Degrave et al. , 2019 ) , soft body simulation ( Hu et al. , 2019 ) , and fluid dynamics ( Schenck & Fox , 2018 ; Li et al. , 2019 ) . Such simulators allow for backward propagating through an entire simulation , and even the simplest optimization algorithms ( gradient descent ) can be employed to a great effect . By treating robotic control tasks as a nonlinear numerical optimization problem , two more sophisticated strategies , trajectory optimization ( Posa et al. , 2014 ; Manchester & Kuindersma , 2017 ; Winkler et al. , 2018 ; Marchese et al. , 2016 ; Bern et al. , 2019 ) and model-predictive control ( MPC ) ( Todorov & Li , 2005 ; Tassa et al. , 2012 ; 2014 ; Lenz et al. , 2015 ) , employ established numerical optimization techniques to efficiently solve complex , even high-dimensional control tasks . But , these methods are are prone to becoming trapped in poor local minima , especially in the presence of complex dynamics or terrain . Separate from model-based control methods , which assume the model of dynamics is known , modelbased learning methods ( Gu et al. , 2016 ; Yamaguchi & Atkeson , 2016 ) attempt to learn a prior unknown model so as to directly optimize policies . In this paper , we do not focus on model-based learning , but rather the complementary problem , where we assume an accurate model of dynamics is given by a differentiable simulator and we use it to efficiently explore optimal control parameters . In contrast to the above local methods , stochastic global optimization methods , such as evolutionary algorithms ( e.g. , ( Hansen et al. , 2003 ) ) or simulated annealing , explore the solution space with a large number of samples in order to find global optima . Such methods , however , can be quite slow , due to the large number of samples they require . RL , while technically local , naturally incorporates sampling , exploration , and value function learning in a similar way that greatly mitigates optimization ’ s propensity to becoming trapped in poor local minima . Our algorithm similarly incorporates exploration to mitigate the likelihood of poor local minima , while using gradient information to do so with high sample efficiency . 3 METHOD . 3.1 PRELIMINARIES . Markov-Decision Problems ( MDP ) In this paper , we consider the rigid body simulation as a discrete-time Markov Decision Process ( MDP ) defined by ( S , A , Γ , R ) , where S ⊆ Rn is the state space , A ⊆ Rm is the action space , Γ : S ×A → S is the deterministic dynamic transition function , andR : S ×A× S → R is the reward function . The optimal control policy π : S → A maximizes the γ-discounted sum of reward η ( π ) over a finite time horizon T , which is defined as : η ( π ) = Eρ0 , π [ T−1∑ t=0 γtrt ] . Here ρ0 specifies the distribution of the initial state s0 and rt = r ( st , at , st+1 ) represents the reward collected at time step t. We represent the policy function π as a neural network parameterized by θ and assume πθ is a deterministic policy that takes as input the current state st and predicts the next action at = πθ ( st ) . The action-value function Qπ ( st , at ) , which describes the expected return given the action at and state st at step t , is defined as Qπ ( st , at ) = Est+1 , at+1 , ··· [ T−1∑ l=t γl−trl ( sl , al , sl+1 ) ] , ( 1 ) in which al ∼ π ( sl ) and sl ∼ Γ ( sl−1 , al−1 ) . The expectation can be removed in our case since both Γ and π are deterministic . This function is approximated by the critic network Qφ ( s , a ) parameterized with φ. Differentiable simulators In this paper , we focus on differentiable and deterministic rigid body simulators with fully observable states . At each time step t , the state st fully characterizes the current status of the robot . The simulator takes current state st and an action at as input and returns the next state st+1 by solving the governing physical equations.Additionally , we assume a differentiable simulator also returns a Jacobian matrix ∇st , atst+1 . For all the differentiable rigid body simulators in this paper , the time and space complexities of computing the Jacobian matrix are the same as computing st+1 in one forward step . 3.2 ALGORITHM . Our hybrid algorithm takes inspiration from a subclass of modern actor-critic algorithms which seek to improve actor policies via critic gradients ( such as DDPG ) . These algorithms iteratively improve the model of the critic , or estimated future return , using recorded data . Such algorithms implicitly assume that if the critic is well-fit to reality , then the gradients must also be well-fit . If we want to prioritize accurate critic gradients and better bootstrap our actor learning , it is best to ensure this directly . Specifically , we augment DDPG ’ s critic gradient fitting step with a regularization term which prioritizes fitting accurate gradients in addition to accurate returns . Our choice of working in the domain of deterministic policy gradients is deliberate ; as will be seen , this framework allows us to use our differentiable simulation engine to back propagate through the state-action pairs of our MDP , generating ground-truth reward gradients to fit . Algorithm 1 summarizes our method with our modifications to the original DDPG highlighted in color . Like actor-critic methods , our algorithm populates a replay buffer of state-action pairs from which to learn . At the same time , it records the dynamical Jacobian ∇st , atst+1 corresponding to those state-action pairs , without degrading its asymptotic time or space complexity . Like the state-action pairs , these Jacobians need only be computed during rollouts ; they can be re-used during the embedded supervised training steps to efficiently improve the actor and critic networks . With additional Jacobian information about the dynamics at hand , we compute the gradients of the target network Q̂i . Since Q̂i defines the target value that the critic Qφ attempts to fit to , ∇Q̂i provides a good estimation to the desired ∇Q as well . This motivates us to use it as a regularizer in the critic loss ( Line 10 in Algorithm 1 ) with two additional weights w1 and w2 corresponding to the weights on the two partial derivatives . The definition of ∇si , aiQ̂i is provided by the following equation ( For brevity , we use the network names to refer to their output values in the equation ) : ∇si , aiQ̂i =∇si , airi +∇si+1ri · ∇si , aisi+1 + γ ( ∇si+1Q′ +∇π′Q′ · ∇si+1π′ ) ∇si , aisi+1 , ( 2 ) which can be evaluated in constant time ( Figure 1 ) . It is important to note that this computation is only possible when a differentiable simulator is available due to the need for∇si , aisi+1 . Time and space complexity Our modifications require extra storage space for two Jacobian matrices of the size dim ( s ) × dim ( s ) and dim ( s ) × dim ( a ) associated with each si in the replay buffer , which is tolerable as long as the dimensions of s and a are small constants . This is the case , for example , in rigid robot control tasks . In terms of the time cost , computing ∇Q̂i makes efficient use of the cached data . As a result , its computation requires only a handful of matrix multiplications and back-propagation through Q′ and π′ . Algorithm 1 Hybrid DDPG algorithm Require : Differentiable simulator Γ , actor network µθ , critic network Qφ , weighting factors w1 , w2 , damping constant τ 1 1 : Initialize two target networks π′ = πθ and Q′ = Qφ 2 : Initialize a replay buffer B 3 : Get the initial state s0 4 : while True do 5 : Generate a random process N 6 : Simulate the agent using at = π ( st ) +Nt at each time step t until termination 7 : Store all ( st , at , rt , st+1 , ∇stst+1 , ∇atst+1 ) to B 8 : Sample a random minibatch of n transitions ( si , ai , ri , si+1 , ∇sisi+1 , ∇aisi+1 ) from B 9 : Set Q̂i = ri + γQ′ ( si+1 , π′ ( si+1 ) ) . 10 : Update the critic parameter φ by minimizing the loss : L = 1 n ∑ i ∥∥Qφ ( si , ai ) − Q̂i∥∥2 + w1∥∥∇aiQφ −∇aiQ̂i∥∥2 + w2∥∥∇siQφ −∇siQ̂i∥∥2 11 : Update the actor parameter θ using the following gradient : ∇θη ≈ 1 n ∑ i ∇aQφ ( si , a ) |a=πθ ( si ) ∇θπ ( si ) 12 : Update target networks : π′ = τπθ + ( 1− τ ) π′ Q′ = τQφ + ( 1− τ ) Q′ 13 : end while

Summary: DDPG is a popular RL method for continuous control problems. It is more widely applicable than traditional model-based approaches like MPC, since it doesn't require differentiable models of the dynamics. However, in many environments, dynamics are differentiable. This paper proposes a method for extending DDPG to exploit simulator gradients. In particular, the Bellman error objective (which is defined in terms of critic values) used for training the critic is augmented with additional terms defined in terms of gradients of the critic. This leads to faster convergence in practice on a range of benchmarks.