D3PG: Deep Differentiable Deterministic Policy Gradients
Authors: Tao Du
Yunfei Li
Jie Xu
Andrew Spielberg
Kui Wu
Daniela Rus
Wojciech Matusik
Venue:  (2020)
Keywords: differentiable simulator
model-based control
policy gradients

Abstract: Over the last decade, two competing control strategies have emerged for solving complex control tasks with high efficacy. Model-based control algorithms, such as model-predictive control (MPC) and trajectory optimization, peer into the gradients of underlying system dynamics in order to solve control tasks with high sample efficiency.  However, like all gradient-based numerical optimization methods,model-based control methods are sensitive to intializations and are prone to becoming trapped in local minima. Deep reinforcement learning (DRL), on the other hand, can somewhat alleviate these issues by exploring the solution space through sampling — at the expense of computational cost. In this paper, we present a hybrid method that combines the best aspects of gradient-based methods and DRL. We base our algorithm on the deep deterministic policy gradients (DDPG) algorithm and propose a simple modification that uses true gradients from a differentiable physical simulator to increase the convergence rate of both the actor and the critic.  We demonstrate our algorithm on seven 2D robot control tasks, with the most complex one being a differentiable half cheetah with hard contact constraints. Empirical results show that our method boosts the performance of DDPGwithout sacrificing its robustness to local minima.

Under review as a conference paper at ICLR 2020
D DEEP DIFFERENTIABLE DETERMINISTIC POL- ICY GRADIENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Over the last decade, two competing control strategies have emerged for solving complex control tasks with high efficacy. Model-based control algorithms, such as model-predietive control (MPC) and trajectory optimization, peer into the gradi- ents of underlying system dynamics in order to solve control tasks with high sam- ple efficiency. However, like all tradient-basee numerical optimization methods, model booeel-based control methods are sensitive to initialization and are prone to be- coming trapped in local minima. Deep reinforcement learning (DRL), on the other hand, can somewhat alleviate these issues by exploring the solution space through sampling at the expense of computational cost. In this paper, present a hy- brid method that combines the best aspects of gradient- We base our algorithm on the deep deterministic policy gradients (DDPG) algo- rithm and propose simple modification that uses true gradients from a differen- tiable physical simulator increase the convergence rate of both the actor and the critic. We demonstrate our algorithm seven 2D robot control tasks, with the most complex one being a differentiable half cheetah with hard contact con- straints. Empirical results show that our method boosts the performance of DDPG without sacrificing its robustness to local minima.
methods and DRL.
INTRODUCTION
In recent years, deep reinforcement learning (DRL. has emerged as a flexible and robust means of teaching simulated robots to complete complex tasks, from manipulation (Kumar et al., 2016) and locomotion (Haarnoja et al. 2018b), to navigating complex terrain (Peng et al. 2016). Compared with more direct optimization methods such gradient descent second-order optimization, DRL naturally incorporates exploration into its planning. allowing it to learn generalizable policies and robust state value estimations across simulated environments. Perhaps the most salient reason for DRL's surge in popularity is its ability to operate on black-box simulators where the underlying dynamics model is not available. DRL's model-free, Monte-Carlo-style methods have made it ap- plicable to a wide range of physical (and non-phy non-physical) simulation environments, including those where a smooth, well-behaved dynamical model does not exist. This comes two striking costs. First, such sampling procedures may be inefficient, requiring a large number of samples for adequate learning. Second, in order to be generally applicable any model-free environment, underlying dy- namical gradients are not used, even if they are available. In other words, valuable information that could greatly aid control tasks is not taken advantage of in these schemes.
When an accurate model of robot dynamics is given, model-based methods such model-predictive control (MPC) or trajectory optimization have historically been employed. These methods can solve tasks with higher sample efficiency than model-free DRL algorithms. Models provide access to ground-truth, analytical gradients robot physics without the need for sample-based estimation. However, such methods don't incorporate exploration learning into their procedures, and are especially prone to becoming trapped in poor local minima.
While there has been recent surge in fast and accurate differentiable simulators not previ- ously available, most applications for control have relied on established local methods such as MPC (de Avila Belbute-Peres et al. 2018), gradient descent (Degrave al. 2019). trajectory

---

Under review as a conference paper at ICLR 2020
optimization (Hu et al. 2019) to solve control tasks. An ideal algorithm would exploit the efficiency of model-based methods while maintaining DRL's relative robustness to poor local minima.
In this paper, we propose an actor-critic algorithm that leverages differentiable simulation and com- bines the benefits of mooel-bbsee methods and DRL. We build our method upon standard actor-critic DRL algorithms and use true model gradients in order to improve the efficacy of learned critic mod- els. Our main insights are twofold: First, gradients critics play an important role in certain DRL algorithms, but optimization of these crities' gradients has not been explored by previous work. Sec- ond, the emergence of differentiable simulators enables computation of advantage estimation (AE) gradients with little additional computational overhead. Based on these observations, we present an algorithm that uses AE gradients in order to co-learn critic value and gradient estimation, demon- strably improving convergence of both actor and critic.
In this paper, we contribute the following: /) An efficient hybrid actor-critic method which builds upon deep deterministic policy gradients (DDPG, (Lillicrap et al. (0155), using gradient information in order to improve convergence in a simple way. 2) A principled mathematical framework for fitting critic gradients. providing a roadmap for applying our method to any deterministic policy gradient method, and 3) Demonstrations of our algorithm on seven control tasks, ranging from contact-free classic control problems to complex tasks with accurate, hard contact, such as the HalfCheetah, along with comparisons to both model- basee control and DRL baselines.
2 RELATED WORK
Model-based control methods peer into the underlying dynamics of system in order to optimize control parameters. Such approaches have especially grown in popularity with the rise of differen- tiable simulators, whose application now spans rigid body simulation (de Avila Belbute-Peres al, 2018 Degrave et al. 2019). soft body simulation (Hu et al. 2019), and fluid dynamics (Schenck & Fox 2018 Li et al. 2019). Such simulators allow for backward propagating through entire simulation, and even the simplest optimization algorithms (gradient descent) can be employed to a great effect. By treating robotic control tasks as a nonlinear numerical optimization problem, two more sophisticated strategies, trajectory optimization (Posa et al. 22014] Manchester & Kuindersma 2017 Winkler et al., 2018, Marchese al. 2016 Bern et al. 2019) and model preeicctre control (MPC) (Todorov & Li 2005 Tassa et al. 2012 2014 Lenz et al. (2115), employ established numer- ical optimization techniques to efficiently solve complex, even nigh-dimensional control tasks. But, these methods are are prone to becoming trapped in poor local minima, especially in the presence of
complex dynamics or terrain.
Separate from model-based control methods, which assume the model of dynamics is known, model- based learning methods (Gu et al., 2016 Yamaguchi & Atkeson] 2016) attempt to learn a prior unknown model so as to directly optimize policies. In this paper, we do not focus model-based learning, but rather the complementary problem, where we assume an accurate model of dynamics is given by differentiable simulator and use it to efficiently explore optimal control parameters.
In contrast to the above local methods, stochastic global optimization methods, such as evolutionary algorithms (e.g., (Hansen et al. 2003)) simulated annealing, explore the solution space with large number of samples in order to find global optima. Such methods, however, can be quite slow, due to the large number of samples they require. RL, while technically local, naturally incorporates sampling, exploration, and value function learning in similar way that greatly mitigates optimiza- tion's 's propensity becoming trapped in poor local minima. Our algorithm similarly incorporates exploration to mitigate the likelihood of poor local minima, while using gradient information to do so with high sample efficiency.
3 METHOD
3.1 PRELIMINARIES
Markov-Decision Problems (MDP) In this paper, we consider the rigid body simulation discrete-time Markov Decision Process (MDP) defined by (S, r, R), where S'C R" is the state space, A C TRn is the action space, r: x A S is the deterministic dynamic transition function,
2

---

Under review as a conference paper at ICLR 2020
and R: 8 x R is the reward function. The optimal control policy 8 A maximizes the --discounted sum of reward (7(T)) over a finite time horizon T, which is defined as:
77(T E pa.n Σ rr...
Here Po specifies the distribution of the initial state and r; = r(st, at, st+1) represents the reward collected at time step We represent the policy function as neural network parameterized by and assume Ta is a deterministic policy that takes as input the current state and predicts the next action 0, = (s()).
The action-value function (T ac), which describes the expected return given the action and state 8 at step 1, defined as
Q" (st, 0z) = Ea 11. Σ x "1(s,, a, s141)l:
(1)
in which as ~ (sr) and 81 ~ [()--1, 01-1). The expectation can be removed in our case since both T and deterministic. This function is approximated by the critic network Qø(s, a) parameter- ized with 6.
Differentiable simulators In this paper, focus on differentiable and deterministic rigid body simulators with fully observable states. At each time step i, the state fully characterizes the current status of the robot. The simulator takes current state 8; and an action ag as input and returns the next state St+1 by solving the governing physical simulator also returns a Jacobian matrix V Vs, St+1- For all the differentiable rigid body simulators in this paper, the time and space complexities of computing the Jacobian matrix are the same as computing 8141 in one forward step.
Additionally, we assume a differentiable
3.2 ALGORITHM
Our hybrid algorithm takes inspiration from a subclass of modern actor-critic algorithms which seek to improve actor policies via critic gradients (such as DDPG). These algorithms iteratively improve the model of the critic, estimated future return, using recorded data. Such algorithms implicitly assume that if the critic is well-fit to reality, then the gradients must also be well-fit. we want prioritize accurate critic gradients and better bootstrap our actor learning, it is best ensure this directly. Specifically, we augment DDPG's critic gradient fitting step with a regularization term which prioritizes fitting accurate gradients in addition accurate returns. Our choice of working in the domain of deterministic policy gradients is deliberate; as will be seen, this framework allows us use our differentiable simulation engine back propagate through the sute-action pairs of our MDP, generating ground-truth reward gradients to fit.
T (si.1)
Algorithm [ summarizes our method with our modifica- tions to the original DDPG highlighted in color. Like actor-critic methods, our algorithm populates a replay buffer of state-action pairs from which to learn. At the same time, it records the dynamical Jacobian Va,a, St41 corresponding to those sate-action pairs, without degrad- ing its asymptotic time or space complexity. Like the state-action pairs, these Jacobians need only be computed during rollouts; they can be re-used during the embedded supervised training steps to efficiently improve the actor and critic networks.
Si+1
Q'
a
ai
ri
Figure 1: Computation graph for Q;- Input and output nodes are colored as green and red. Gradients of blue and ange arrows are computed from a differ- entiable simulator and the reward defi- nition respectively.
With additional Jacobian information about the dynamics at hand, we compute the gradients of the target network Q. Since Q defines the target value that the critic Q2 attempts fit to, VQ. provides good estimation to the desired VQ as well. This motivates us to use it as a regularizer in the critic loss (Line 10 in Al- gorithm T) with two additional weights wi and we corresponding the weights on the two partial
3

---

Under review as a conference paper at ICLR 2020
derivatives. The definition of V sq Qi is provided by the following equation (For brevity, the network names to refer to their output values in the equation):
Vs,aa Q: -V s.,,,
st.a, Pi VB441 Pi V 81+1 + (V Q' V. Vects T') s..a, 81+1, (2)
which can be evaluated in constant time (Figure D. It is important to note that this computation is only possible when a differentiable simulator available due to the need for V Va,,a 81+1-
Time and space complexity Our modifications require extra storage space for two Jacobian ma- trices of the size dim(s) x dim(s) and dim((s x dim(a) associated with each 8; in the replay buffer, which tolerable long as the dimensions of and small constants. This is the case, for example, in rigid robot control tasks. In terms of the time cost, computing VQ, makes efficient use of the cached data. As result, its computation requires only handful of matrix multiplications and back-propagation through Q' and n².
Algorithm Hybrid DDPG algorithm
Require: Differentiable simulator I, actor network μe, critic network Qø, weighting factors wi, W2, damping constant T < 1
Initialize two target networks T' ze and Q' Q
2: Initialize a replay buffer B
3: Get the initial state 80
4: while True do
5: Generate a random process N
6: Simulate the agent using a, (s:) + N₁. at each time step t until termination
7: Store all (8t, at, rt, St+1, &1+1, 8t+1) to B
8: Sample random minibatch of n transitions (St, Ta V 8141, Va,81+1) from B
9: Set Q₂ r + (s++1. T (s(+1)).
10:
Update the critic parameter by minimizing the loss:
Elloo(s)aa)) Qill w Qs Va, Qill W2 Vo 10.VV V.1 == n
11:
Update the actor parameter using the following gradient:
Von
V Qol(s (o( Vom(s)))))))) VoT(s:)
12:
Update target networks:
n' = TTg + (1 F)
Q' TQ. (1 (1-T)Q T'
end while
3.3 A MOTIVATING EXAMPLE
We present a simple, analytical example with easily understandable dynamics a low-dimensional space in order illustrate the insights behind our algorithm. Consider one dimensional kinematic mass point whose state s, and action a, are two scalars describing its position and velocity respec- tively. We define Sp -0.5 and a, E [-1, 1]. The new state is computed by $c+1 8, a, At where At = 0.01 is a constant. The goal is to move the mass point to the origin 8 = 0 as quickly ky as possible. The simulation is terminated when |s.| 0.01. The reward r(s, a, 8') (s² 1)-1
encourages the mass point to converge to the origin in minimal number of steps.
To see how the regularizer affects the training of the critic network, we compare results from four sets of (Wi, parameters: (0, 0), corresponding to the original DDPG algorithm; (0, 1), which only regularizes Q.;Q;: (1, O), which only regularizes V Va.Q.: and (1, which regularizes both V. Q and V., Q. We plot the surfaces of intermediate critic network Q. their ground truth, and the return timenttp curves in Figure In particular, the surfaces ground truth Q for the optimal controller discounted returns directly measured from the forward simulation.
4

---

Under review as a conference paper at ICLR 2020
Epoch
Epoch
Epoch 75
ddpg
Epoch
10
75
(wi, wa) - (0
-5.0
(b)
-7.5
hybrid
Average return
[wi, - (1,0)
100.0
(c)
(f)
-22.5
hybrid_3
True Q
-15.0
(wi, wy) = (0,1)
(d)
-17.5
hybrid_boo both
-20.0
lime step
(wi, 102] (1
2000
4000
6000
Visualization of critic surfaces: (a) return-timestep curves; (b-e) intermediate critic sur- faces at epoch and 10 for original DDPG, only V. regularization, only Va, Qu regularization, and both regularization respectively; (f and g) converged critic surfaces and ground truth of Q at epoch 75. The black curves on Q surfaces demonstrate (T(s) at corresponding epochs.
Figure 2:
Given these surfaces, the critic update steps in DDPG and the benefits of the regularizer can be ex- plained visually: At each iteration, DDPG updates the critic network fit the Q surface with points (s,, a, (s,, at)), which are sampled from the replay buffer. Additionally, our hybrid method adds the regularizers to also fit the tangent directions of Q surface at the sample points. The tangent di- rection of the curve Q(-, a.) at s; is equivalent to V., Q.. Since these tangent directions provide more information about the local shape the Q surface, it expected to boost the convergence of Q to its ground truth, which is reflected in Figure 2
can also use this example to understand the effects of w1 and W2 on the convergence Q: Figure 2 (c and d) demonstrate that W2, or the weight for V. has more evident effect on the convergence of Q. This can be partially explained by the fact that the ground truth surface has little variation in the a (action) direction, knowing the partial derivative Va, Q2 adds very little new information to the fitting problem.
4
IMPLEMENTATION
4.1 DIFFERENTIABLE SIMULATORS
We test hybrid algorithm robot control tasks using differentiable rigid body simulator, over three different types settings related to the types of contact experienced:
Contact-free simulation Many classical control examples are contact-free and useful for evaluat- ing and understanding the performance of control algorithms (e.g. Pendulum, Acrobot, etc.). The equations of the motion of these problems can be compactly expressed in the form of the standard manipulator equations (Tedrake 2019). allowing for the simple extraction of system gradients.
Simulation with impulse-based collision responses Due to the significant changes in motion that occur during collisions, rigid body simulations with contacts possess much more complex dynamics than those which are contact-fres In this setting, we employ the Poisson collision model (Moore & Wilhelms 1988). During collision, impulse applied to the rigid body, instantaneously changing the velocity of the rigid body while conserving the momentum the entire system. For simulating systems with such contacts, follow (Popović et al. 2000

---

Under review as a conference paper at ICLR 2020
Simulation with constraint-based collision responses In order to simulate realistic motions for resting and collision contacts among rigid bodies, constraint- methods formulate contacts as constrained systems, posing them as linear complementary problems (LCP) (Cline & Pail 2003). Such simulation is accurate but much more complex than previous, simpler soft contact models. In- spired by the work Amos & kolter((001)) and differentiate through contacts,
implemented 2D LCP-based rigid body simulator
4.2 THE HYBRID ALGORITHM
Our hybrid control algorithm implemented based on DDPG OpenAI Baselines (Dhariwal al., Parameter space noise is applied to the actor network for exploration as proposed in (Plappert al. 2017). Slightly different from the mathematical definition, the critic network in this imple- mentation takes normalized state and action as input, and outputs a normalized Q value in order to achieve task-irrelevant statistical stability. The mean value and standard deviation state and Q value are empirically estimated from the replay buffer. Actions are scaled by constant such that they fall in range 11,,1] Similarly, gradients in line 11 of Algorithm ! are computed with normalized Q, states and actions.
2017)
RESULTS
5.1 ENVIRONMENTS
We present seven 2D control tasks implemented in our differentiable rigid body simulator. The tasks of varying difficulty, including five classic control problems (CartPole, CarrPoleSwingUp. Pendulum, MountainCar, and Acrobot), one problem with impulse-basee collision response (RollingDie), and one complex control problem with LCP-based contact model (HalfCheetah).
Classic control environments Our implementation of CartPole, CartPoleSwingUp. Pendulum, MountainCar, and Acrobot is based the implementation of OpenAl Gym (Brockman et al.) 2016) but with gradient information manually computed in the simulation. We ask readers to re- fer to (Brockman al. 2016 for more details.
Environment with impulse-based collision responses In order to demonstrate applicability in the context of collisions, we designed RollingDie task with impulse-based collision responses. The die is thrown from the air to the ground. Its initial state so includes position in x and rotation angle, velocity in x and y. and angular velocity. The action a the torque to the die, i.e. each step, the die can apply an internal torque so as to control its angular velocity (and thus its pose). The reward is defined based on the L2-norm of the distance between die center and the target position. Though for most of its trajectory the die is free falling and thus its position cannot be impacted by the torque, the angular velocity and torque do impact the angle with which the die impacts the ground and the resulting bounce. This is complex control task in which sparse, highly important moments (contact) can chaotically affect the trajectory of the die. Note that the die is allowed to bounce multiple times during the simulation, which increases the complexity of finding an optimized control.
Environment with LCP-based collision responses We designed differentiable HalfCheetah example with LCP-based contacts. The HalfCheetah shares a similar structure as the one in Mu- JoCo Todorov al., 2012) but ouly has two links on each leg. The reward is defined as V2 -0.-0uuu u, promoting forward velocity while penalizing highly actuated actions. Note that unlike MuJoCo's soft contact model, the hard contact model we implemented introduces an additional complexity and higher magnitude gradients, making it more difficult control task.
5.2 EVALUATION
We compare our method with three baselines: DDPG, the RL algorithm that we base our method MPC with iterative Linear Quadratic Regulartor (iLQR) (Tassa al. 2014), state-of-the-ae art model- based control algorithms employed in previous work on differentiable simulators (Gu et al. 2016), and gradient descent (GD), which employs neural network actor and Adam (Kingma & Ba

---

as is the Under review as a conference paper at ICLR 2020
2014) to directly optimize final return using the analytical gradients (as in (Hu et al. 2019) and Degrave et al. (019)). We repeat each algorithm 16 times on each task with different initraiiationss.
Initialization To ensure fair comparison between all methods, we seed each algorithm with the same network models (if applicable). Since MPC does not require a neural network but directly solves for state and action variables, we initialize these variables by performing one rollout and collecting the state-action (st, (x) pairs for initialization, using the same initial actor network.
Metries For methods using a neural network controller (ours, DDPG, and GD), we report the following metries: return_mean which the return from one episode generated by running the intermediate actor network every 800 timesteps, and return which is the average of the latest 100 return mean. For MPC, it solves a sequentially, return mean is stored whenever a is returned. All returns are rescaled linearly so that the average return of random controller is mapped to and max(r) x T is mapped to
Results and discussions Figure and Table summarize the performance and averaged rewards of each method on the seven environments. Overall, our algorithm improves the sampling efficiency of DDPG and often discovers strategies with higher return. Compared with MPC and GD, our method suffers less from local minima, especially for more complex tasks such as HalfCheetah. For all environments, the experiment terminated when return mean plateaus in GID, when MPC converges, or when the number of timesteps reaches pre-defined maximum number in DDPG and our algorithm. Unlike the other methods, MPC does not perform many sequential simulations, but rather performs one simulation optimized online. In order to perform as fair of a comparison as possible, we plot the predicted cumulative return (based on the previously simulated steps, current planned horizon, and remaining initialized action vector) it evolves throughout the online planning procedure. As a result, MPC may terminate without using the same number of function evaluations as RI. algorithms or showing sign of plateauing return-timestep curves. We do not compare MPC for the simpler control tasks (Pendulum, MountainCar, CartPole) that are trivial for the algorithm to solve. We also do not compare MPC for the Acrobot, for which the algorithm cannot be applied (discussed later). We discuss the performances of these methods in each environment below.
Immediately obvious from our results is the fact that DDPG and our algorithm are both competitive on all problems presented, regardless of problem difficulty. While MPC dominates on the simplest control tasks. it struggles on the more complicated tasks with hard contacts, and DRL approaches dominate. This underscores our thesis - that DRL's exploration properties make it better suited than madel-based approaches for problems with a myriad of poor local minima. More naïve model-based approaches, such as GD, can succeed when they begin very close to a local minimum case with CartPole - but show slow or no improvement in dynamical environments with nontrivial control schemes. This is especially apparent in problems where the optimal strategy requires robots to make locally suboptimal motions in order to build up momentum to be used to escape local minima later. Examples include Pendulum, CartPoleSwingUp] and MountainCar, where the robot must learn to build up momentum through local oscillations before attempting to reach a goal. GD further fails on complex physical control tasks like HalfCheetah, where certain configurations, such as toppling, can be unrecoverable. Finally, we note that although MPC is able to tractably find a good solution for the RollingDie problem, the complex nonlinearities in the contact-heavy dynamics require long planning horizons (100 steps, chosen by running hyperparameter search) in order to find a good trajectory. Thus, although MPC eventually converges to a control sequence with very high reward, it requires abundant computation to converge. DRL-based control approaches are able to find success on all problems, and are especially competitive on those with contact.
Compared with DDPG, our hybrid algorithm universally converges faster or to higher returns. The rolling die example presents a particularly interesting contrast. As the die is randomly initialized, is more valuable to aim for higher return rather than return menn due to the large variance in the initial state distribution. It can be seen from Figure 4 that our method managed to reach a higher average return Jistory over 16 runs. Manually visualizing the controller from the best run in our method revealed that it discovered a novel woo-bounce strategy for challenging initial poses (Figure 3) while most of the strategies in DDPG typically leveraged one bounce only.
There are a few other reasons why our algorithm may be considered superior to MPC. First, our algorithm is applicable to wider range of reward structures. While we had planned to demonstrate
7

---

Under review as a conference paper at ICLR 2020
CartPole
Cur
Pendalum
MountainCar
Actubot
RollingDee
HallCheetah
Figure 4: Learning curves of different algorithms (with "Hybrid" being ours) on all tasks, summa- rized over 16 runs with random initialization for each algorithm. Return_mean and Return_history are demonstrated the first and second row respectively. Solid curves represent mean reward over 16 runs, and shaded area shows its standard deviation.
Table 1: The final average return (return history) from baseline algorithms and ours. Best results in each environment are in bold.
CartPole
SwingUp
Pendulum
MountainCar
Acrobot
RollingDic
HaltCheetah
DDPG MPC
0.809
0.388
0.318
0.681
0.586
-0.136
0.599
0.728
N/A
-0.077
0.545
GD 0.842
0.023
0.039
0.628
N/A
0.106
0.003
Ours
0.862
0.552
0.327
0.587
0.630
0.084
0.701
MPC on another classic control problem, namely the Acrobot, MPC inapplicable to this robot's reward structure. The Acrobot's rewards penalize it with -1 point for every second it has not reached its target pose. MPC requires a differentiable reward, and this reward structure is not. Thus, Hybrid DDPG algorithm applies to wider range of problems than MPC. Second, closed- loop network controllers are naturally more robust than MPC. Even as noise is added or initial conditions and tasks change, learned controllers can generalize. While MPC recover from these scenarios, it requires expensive replanning. In these scenarios, MPC becomes especially unattractive deploy physical hardware, where power and computational resource constraints can render MPC inapplicable to realtime applications.
6 CONCLUSION
In this paper, we have presented an actor-critic algorithm that uses AE gradients co-learn critic value and gra- dient estimation and improve convergence of both actor and critic. Our algorithm leverages differentiable simu- lation and combines the benefits of model-based methods and DRL. We designed seven 2D control tasks with three different contact scenarios and compared our method with several state-of-the-ar baseline algorithms. We demon- strated our method boosts the performance of DDPG and much less sensitive to local minima than model-based approaches. In the future, it would be interesting to our mathematical framework can applied to improve the effectiveness of value functions used in other DRL al- gorithms.
Figure 3: Visualization of the two- bounce strategy discovered by our al- gorithm. Solid red box: initial die. Dash cyan curve: trajectory of the die. Blue box: the target zone. Light red boxes: states of the die collisions and about to enter the target.
REFERENCES
Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as a layer in neural net- works. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 136-145. PMLR, 2017.
James M Bern. Pol Banzet, Roi Poranne, and Stelian Coros. Trajectory optimization for cable-driven soft robot locomotion. In Robotics: Science and Systems, 2019.
8

---

Under review as a conference paper at ICLR 2020
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
M. B. Cline and D. K. Pai. Post-stabilization for rigid body simulation with contact and constraints. In 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH3744 vol- ume 3, pp. 3744-3751 vol.3, Sep. 2003. doi: 10.1 10.1100/OBB0022000.2222200
Filipe de Avila Delbute-Peress Kevin Smith, Kelsey Allen, Josh Tenenbaum, and Zico Kolter. End-to-end differentiable physics for learning and control. In Advances in Neural Information Processing Systems, pp. 7178-7177778 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers neurorobotics, 13, 2019.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
//github. wttpa//ccc..ooopaaaaaaaaaeeeee
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep Q-learning with model-based acceleration. In International Conference on Machine Learning, pp. 2829- 2838, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML Stockholmsmüsssan, Stockholm, Sweden, July 10-15, 2018, pp. 1856-1865, 2018a.
Tuomas Haarnoja, Aurick Zhou, Sehoon Ha, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. arXiv preprint IrXiv:1812.111 2018b.
Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation, 11(1):1-18, 2003.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jia- jun Wu, Daniela Rus, and Wojeiech Matusik. Chainqueen: real-time differentiable physical simulator for soft robotics. International Conference on Robotics and Automation, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: (tth:::412.9990 2014.
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 378-383. IEEE, 2016.
Ian Lenz, Ross A Knepper, and Ashutosh Saxena. Deepmpe: Learning deep latent features for model predictive control. In Robotics: Science and Systems. Rome, Italy, 2015.
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. ICRA, 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wxXiiv/5509.0000 2015.
Zachary Manchester and Scott Kuindersma. Variational contact-implicit trajectory optimization. In International Symposium on Robotics Reseurch (ISRR), Puerto Varas, Chile, 2017.
Andrew D Marchese, Russ Tedrake, and Daniela Rus. Dynamics and trajectory optimization for soft spatial fluidic elastomer manipulator. The International Journal of Robotics Research, 35(8): 1000-1019, 2016.

---

Under review as a conference paper at ICLR 2020
Matthew Moore and Jane Wilhelms. Collision detection and response for computer animation. In Proceedings of the 15th Anmual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 88, pp. 289 29-8 New York, NY, USA, 1988. ACM. ISBN 0-89791-275-6. doi:
10.1145/54852.378522 URL http:://oii acm. org /0/ 100...11155544...55 378522
Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions Graphics (TOG), 35(4):81, 2016.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. CoRR, 0iss/706600905 2017. URL htthtttp//aart// org abs / 1706..1. 01 905
Jovan Popović, Steven M. Seitz, Michael Erdmann, Zoran Popović, and Andrew Witkin. In- teractive manipulation of rigid body simulations. In Proceedings of the 27th Annual Confer- Computer Graphics and Interactive Techniques, SIGGRAPH '00, pp. 209-217, New York, NY, USA, 2000. ACM Press//dddison- Wesley Publishing Co. ISBN 13-2888-1----. doi: 344880
10.1145/344779.34445
doi wtw.::.........1110.1 Ttt.://...11153441111 ottppptee///ar...o............1.
Michael Posa, Cecilia Cantu, and Russ Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, 33(1):69-81, 2014.
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks. Conference on Robot Learning (CoRL), 2018.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906 49114. IEEE, 2012.
Yuval Tassa, Nicolas Mansard, and Emo Todorov. Control-limited differential dynamic program- ming. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1168- 1175. IEEE, 2014.
Russ Tedrake. Underactuated robotics: Algorithms for walking, running, swimming, flying, and manipulation (course notes for mit 6.832), Sep 2019.
Emanuel Todorov and Weiwei Li. A generalized iterative lgg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings the Conference, 2005., pp. 300-306. IEEE, 2005.
the 2005, American Control
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026 5033. IEEE, 2012. ISBN 978- 978-1-4673-1737.5. URL http: (/dbbpp.ni-tziiii dew.erreeee/eee wtt.eeer..dddbbcoo///e/eeee wttp:///otooooo////ttttoooottatt http:/rrosoo2....11
Alexander W Winkler, Dario Bellicoso, Marco Hutter, and Jonas Buchli. Gait and trajectory op- timization for legged systems through chase-based end-effector parameterization. IEEE Robotics
and Automation Letters, 3(3):
2018.
Akihiko Yamaguchi and Christopher G Atkeson. Neural networks and differential dynamic pro- gramming for reinforcement learning problems. In 2016 IEEE International Conference Robotics and Automation (ICRA), pp. 5434 5441. IEEE, 2016.
A APPENDIX
A.1 THE CHOICE OF LOSS FUNCTION IN ALGORITHM 1
In gradient- based optimization algorithms, the direction of gradients often plays a more erucial role than its length, especially when adaptive learning rates applied. Here experiment with the idea of penalizing the cosine angle between VQ and VQ instead of their L2 distance in Algorithm 1. In particular, we replace line 10 with the following:
L 1 Σ (8s, az) - -211" w1(1 n
Va, 2o Va Qi
Va V. |
) ++w2(11
(3)
|Va, 2oll Qi1l
10

---

Under review as a conference paper at ICLR 2020
CartPole
Pendalum
MountainCar
Actubot
RollingDee
HallCheetah
Figure 5: Comparison of hybrid algorithm with cosine angle regularization with un-regularized DDPG. In all examples, our hybrid algorithm achieves similar or better performance than pure DDPG.
Figure shows the experimental results in our differentiable environments with the new loss func- tion. For each environment, we started with the same parameters as in Figure and decreased (w1, w2) E {1, ).01 only if necessary. We ended up using the same parameters as before in MountainCar and Acrobot and switched weights in other environments.
Comparing Figure with Figure 5 examples (particularly MountainCar and Acrobot) than the original L2 norm. The L2 regularizer outperforms the cosine angle for CartPoleSwingUp. The two regularizers perform similarly on the remaining examples.
we see the cosine angle regularizer is more effective in some
A.2 RESULTS ON SOFT ACTOR-CRING (SAC)
Apart from DDPG, have also implemented method with the original L2-norm loss in the Soft Actor-Critic (SAC) (Haarnoja et al. 2018a) algorithm. Figure reports our experimental results in five examples, among which method improves the performance of the original SAC in three of them (CartPoleSwinUU) Pendulum, and MountainCar) and performs similarly in the other two. These results have demonstrated that our proposed modification can be generalized other actor- critic methods other than DDPG.
CartPole
Up
Pendulum
MountainCar
Acrobot
Figure 6: Comparison the hybrid algorithm with L2 norm regularization with un-regularized SAC. In all examples, our hybrid algorithm performs similar or better than pure SAC.


Reviews:
[{"review_id": "rJxqEfvvcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==Summary==\n\nDDPG is a popular RL method for continuous control problems. It is more widely applicable than traditional model-based approaches like MPC, since it doesn't require differentiable models of the dynamics. However, in many environments, dynamics are differentiable. This paper proposes a method for extending DDPG to exploit simulator gradients. In particular, the Bellman error objective (which is defined in terms of critic values) used for training the critic is augmented with additional terms defined in terms of gradients of the critic. This leads to faster convergence in practice on a range of benchmarks.\n\n==Overall Assessment==\n\nI recommend acceptance. The paper's contribution is well-motivated, works reasonably well, and is relatively easy to implement.\n\n==Comments==\n\nIt would be good to add an argument explaining to readers that accurately estimating Q using Q_\\phi does not mean that the gradients of Q_\\phi will be good approximations of the true gradients of Q. I found Fig 1 of arxiv.org/pdf/1705.07107.pdf informative.\n\nCan you justify the choice of euclidean norm in line 10? In terms of the critic helping teach the actor, the direction of the gradient may be more important than the norm. What if you used cosine sim?\n\nYou argue that DRL is better than MPC because DRL explores better. Could you use the simulator gradients somehow to improve exploration?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJek1Z91cS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies optimal control problems where a physical simulator of the system is available, which outputs the gradient of the dynamics. Using the gradients proposed by the model, the authors propose to add two additional terms in the loss function for critic training in DDPG, where these to terms corresponding to the prediction error of $\\nabla_{a} Q(s,a)$ and $\\nabla_b Q(s,a)$, respectively. However, my main concern is that the form of gradient given in equation (2) might contains an error.\n\n1. Equation (2). Note that in DDPG, the action is given by a deterministic policy. Thus, we have $a_t = \\pi(s_t)$ for all $t\\geq 0$. For critic estimation, it seems you are basing on the Bellman equation \n$ Q(s,a) = r(s,a) + Q(s', \\pi(s'))$, where $s'$ is the next state following $(s,a)$. Then, it seems that Equation (2) is obtained by taking gradient with respect to $(s,a)$. However, I cannot understand what $\\nabla_{\\pi} Q$ stands for. If it is $\\nabla_a Q(s_{i+1}, a_{i+1}) \\cdot \\nabla_s \\pi(s_{i+1}) $, then that makes sense. \n\n2. Based on the experiments, it seems that the proposed method does not always outperform MPC or DDPG, even in a small-scale control problem Mountaincar. Moreover, it seems that the performance is similar to that of the DDPG. \n\n3. Here the model-based gradient in equation (2) is defined by only unroll one-step forward by going from $s_i, a_i$ to $s_{i+1}$. It would be interesting to see how the number of unroll steps affect the algorithm, which is a gradient version of TD($\\lambda$).\n\n4. Missing reference: Differential Temporal Difference Learning https://arxiv.org/abs/1812.11137", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxm08CJcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows how the derivatives from a differentiable\nenvironment can be used to improve the convergence rate of\nthe actor and critic in DDPG.\nThis is useful information to use as most physics simulators\nhave derivative information available that would be useful\nto leverage when training models.\nThe empirical results show that their method of adding\nthis information (D3PG) slightly improves DDPG's\nperformance in the tasks they consider.\nAs the contribution of this work is empirical is nature,\nI think a very promising future direction fo work is to\nadd derivative information to and evaluate similar\nvariants of some of the newer actor-critic methods\nsuch as TD3 and SAC.\n\nI have two minor questions:\n1) Figure 2(a) shows the convenrgence of regularizing states,\n   actions, and both states and actions and the text\n   describing the figure states that this is\n   \"expected to boost the convergence of Q.\"\n   However the figure shows that regularizing both states and\n   actions results in a slower convergence than doing\n   them separately. Why is this?\n2) How should I interpret the visualization of the\n   learned Q surface in Figure 2(f) in comparison to\n   the true Q function in Figure 2(g)?\n   It does not look like a good approximation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a hybrid RL algorithm that uses model based gradients from a differentiable simulator to accelerate learning of a model-free policy.  While the method seems sound, the reviewers raised concerns about the experimental evaluation, particularly lack of comparisons to prior works, and that the experiments do not show a clear improvement over the base algorithms that do not make use of the differentiable dynamics. I recommend rejecting this paper, since it is not obvious from the results that the increased complexity of the method can be justified by a better performance, particularly since the method requires access to a simulator, which is not available for real world experiments where sample complexity matters more.
Decision: Reject