PODS: Policy Optimization via Differentiable Simulation
Authors: Miguel Angel Zamora Mora
Momchil Peychev
Sehoon Ha
Martin Vechev
Stelian Coros
Venue:  (2021)
Keywords: Reinforcement Learning
Decision and Control
Planning
Robotics.

Abstract: Current reinforcement learning (RL) methods use simulation models as simple black-box oracles. In this paper, with the goal of improving the performance exhibited by RL algorithms, we explore a systematic way of leveraging the additional information provided by an emerging class of differentiable simulators. Building on concepts established by Deterministic Policy Gradients (DPG) methods, the neural network policies learned with our approach represent deterministic actions. In a departure from standard methodologies, however, learning these policy does not hinge on approximations of the value function that must be learned concurrently in an actor-critic fashion. Instead, we exploit differentiable simulators to directly compute the analytic gradient of a policy's value function with respect to the actions it outputs. This, in turn, allows us to efficiently perform locally optimal policy improvement iterations. Compared against other state-of-the-art RL methods, we show that with minimal hyper-parameter tuning our approach consistently leads to better asymptotic behavior across a set of payload manipulation tasks that demand high precision.

Under review as a conference paper at ICLR 2021
PODS: POLICY OPTIMIZATION VIA DIFFERENTIABLE SIMULATION
Anonymous authors Paper under double-blind review
ABSTRACT
Current reinforcement learning (RL) methods simulation models simple black-box oracles. In this paper, with the goal of improving the performance exhibited by RL algorithms, we explore systematic way of leveraging the ad- ditional information provided by an emerging class of differentiable simulators. Building on concepts established by Deterministic Policy Gradients (DPG) meth- ods, the neural network policies learned with our approach represent deterministic actions. In a departure from standard methodologies, however, learning these pol- icy does not hinge approximations of the value function that must be learned concurrently in an actor-critic fashion. Instead, we exploit differentiable simu- lators to directly compute the analytic gradient of policy's 's value function with respect to the actions it outputs. This, in turn, allows us to efficiently perform locally optimal policy improvement iterations. Compared against other state-of- the-art RL methods, we show that with minimal hyper-parameter tuning our ap- proach consistently leads better asymptotic behavior across a set of payload manipulation tasks that demand high degree of accuracy and precision.
INTRODUCTION
The main goal in RL is to formalize principled algorithmic approaches to solving sequential decision-makin maaiing problems. As defining characteristic of RL methodologies, agents gain experi- ence by acting in their environments in order to learn how to achieve specific goals. While learning directly the real world (Haarnoja et al., 22099] Kalashnikov et al. 2018) is perhaps the holy grail in the field, this remains fundamental challenge: RL is notoriously data hungry, and gathering real-world experience is slow, tedious and potentially unsafe. Fortunately, recent years have seen exciting progress in simulation technologies that create realistic virtual training grounds, and sim-2- real efforts Tan et al. 2018 Hwangbo et al. 2019) are beginning to produce impressive results.
A class of differentiable simulators (Zimmermann et al.] 2019 Liang al., 2019 de Avila Belbute-Peres et al.] 2018;; Degrave et (019)) is currently emerging. These simula- tors not only predict the outcome of particular action, but they also provide derivatives that capture the way in which the outcome will change due to infinitesimal changes in the action. Rather than using simulators as simple black box oracles, we therefore ask the following question: how can the additional information provided by differentiable simulators be exploited to improve RL algorithms?
To provide an answer this question, we propose novel method to efficiently learn control policies for finite horizon problems. The policies learned with our approach use neural networks to model deterministic actions. In a departure from established methodologies. learning these policies does not hinge on learned approximations of the system dynamics or of the value function. Instead, we leverage differentiable simulators to directly compute the analytic gradient of policy's value function with respect to the actions it outputs for specific set of points sampled in state space. We show how to use this gradient information to compute first and second order update rules for locally optimal policy improvement iterations. Through simple line search procedure, the process of updating a policy avoids instabilities and guarantees monotonic improvement of its value function.
To evaluate the policy optimization scheme that we propose, apply it to set of control problems that require payloads to be manipulated via stiff or elastic cables. We have chosen to focus our attention on this class of high-precision dynamic manipulation tasks for the following reasons:

---

Under review as a conference paper at ICLR 2021
they are inspired by real-world applications ranging from cable-driven parallel robots and crane systems to UAV-based transportation to (Figure
the systems we need to learn control policies for ex- hibit rich, highly non-linear dynamics;
the specific tasks we consider constitute a challeng- ing benchmark because they require very precise se- quences of actions. This is feature that RL algo- rithms often struggle with, as the control policies they learn work well on average but tend to output noisy ac- tions. Given that sub-optimal control signals can lead to significant oscillations in the motion of the payload, these manipulation tasks therefore make it possible to provide an easy-to- cos-tto-itterpret comparison of the quality of the policies generated with different approaches;
by varying the configuration of the payloads and actu- ation setups, we can finely control the complexity of the problem to test systematically the way in which we focus in this paper our method scales.
Figure I: Real-world applications that inspire the control problems
The results of our experiments confirm theoretical derivations and show that our method con- sistently outperforms two state-of-the-art (SOTA) model-free RL. algorithms, Proximal Policy Op- timization(PPOO) (Wang et al., 2019) and Soft Actor-Critic(AAC)) (Haarnoja al. 2018), as well as the model-based approach of Backpropagation Through Time (BPTT). Although our policy opti- mization scheme (PODS) can be interleaved within the algorithmic framework of most RL methods (e.g. by periodically updating the means of the probability distributions represented by stochas- tic policies), we focused our efforts on evaluating in isolation pinpoint the benefits brings. This allowed us to show that with minimal byper-parameter tuning, the second order update rule that we derive provides an excellent balance between rapid, reliable convergence and computational complexity. In conjunction with the continued evolution of accurate differentiable simulators, our method promises to significantly improve the process of learning control policies using RL.
2 RELATED WORK
Deep Reinforcement Learning. Deep RL (DRL) algorithms have been increasingly more suc-
cessful in tackling challenging continuous control problems in roboties (Kober et al.. 2013 Li
2018). Recent notable advances include applications in robotic locomotion (Tan et al. 2018
Haarnoja et al. 2019), manipulation (OpenAI et al. 2018 Zhu et al. 2019 Kalashnikov ct al. 2018
Gu ct al.] 6016). and navigation (Anderson al. 2018]] 20 Kempka et al. 20 Mirowski et al.
2016
to mention a few. Many model-free DRL algorithms have been proposed over the years, which can
be roughly divided into two classes, off-policy methods (Mnih et al. 20]] Lillicrap al. 2016 Fu-
jimoto et al. 2018 Haarnoja et al. 20188) and on-policy methods (Schulman et 201 20
Wang
al. 2019). based on whether the algorithm can learn independently from how the samples were generated. Recently, moodel-based RL algorithms (Nagabandi et al. 2017:; Kurutach et al. 2018 Clavera al. 2018 Nagabandi et al., 2019) have emerged as a promising alternative for improving the sample efficiency. Our method can be considered as an on-policy algorithm as it computes first or second-order policy improvements given the current policy's 's experience.
Policy Update as Supervised Learning. Although policy gradient methods are some of the most popular approaches for optimizing a policy (Kurutach et al. 2018)) Wang et al.) 2011)), many DRL al- gorithms also update the policy in a supervised learning (SI.) fashion by explicitly aiming to mimic expert demonstration (Ross et al., (011) or optimal trajectories (Levine & Koltun 2013a b) Mor- datch Todorov 2015). Optimal trajectories, in particular, can be computed using numerical meth- ods such iterative linear-quadratte regulators (Levine & Koltun 2013a b) or contact invariant optimization (Mordatch & Todorov] 2015). The solutions they provide have the potential to improve the sample efficiency of RL methods either by guiding the learning process through meaningful sam- ples (Levine & Koltun 2013a) or by explicitly matching action distributions (Mordatch & Todorov] 2015). Importantly, these approaches are not only evaluated in simulation but have also been shown
2

---

Under review as a conference paper at ICLR 2021
to be effective for many real- world robotic platforms, including manipulators (Schenck & Fox 2016
Levine et al. 2016) and exoskeletons (Dubureq et al. 2019). Recently, Peng et al. (2019) proposed an off- oft-policy RL algorithm that uses SL both to learn the value function and to fit the policy to the Lavaantage-weigheee
target actions. While our method shares some similarities with this class of approaches that interleave SI. and RI., the updates of our policy do not rely on optimal trajectories that must be given as input. Rather, we show how to leverage differentiable simulators to compute locally optimal updates policy. These updates are computed by explicitly taking the gradient of the value function with respect to the actions output by the policy. As such, our method also serves to reinforce the bridge between the fields of trajectory optimization and reinforcement learning.
Differentiable Models. Our approach does not aim to learn a model of the system dynamics, but rather leverages differentiable simulators that explicitly provide gradients of simulation outcomes with respect to control actions. We note that traditional physics simulators such as ODE Drumwwiiii jet al. (2100) or HyBulletCoumaant
& Bail (0199) are not designed to provide this information. We build, in particular, on recent class of analytically differentiable simulators that have been shown to effectively solve trajectory optimization problems, with a focus on sim-2-rea transfer, for both manipulation Zimmermann et al. 2019) and locomotion tasks (Bern et al. 2019).
Degrave et al. (2019) embed differentiable rigid body simulator within a recurrent neural network to concurrently perform simulation steps while learning policies that minimize a loss corresponding to the control objective. While their goal is related to ours, we show how to leverage explicitly- computed gradients to formulate second order policy updates that have significant positive effect on convergence. Furthermore, in contrast to Degrave et al. (2019), we show that PODS consistently outperforms two common RL baselines, PPO (Wang et al. 2019) and SAC (Haarnoja al. 2018).
Also related to our method is the very recent work of Clavera et al. (2020). Their observation is that while most model-based RL algorithms or as a black-box oracle to sample from (Nagabandi et al. 2017), the differentiability of learned dynamics models can and should be exploited further. In an approach that is related to ours, they propose a policy optimization algorithm based on derivatives of the learned model. In contrast, we directly use differentiable simulators for policy optimization, bypassing altogether the need to learn the dynamics- including all the hyperparameters that are involved in the process, as well as the additional strategies required to account for the inaccuracies introduced by the learned dynamics (Boney et al. 2019). Thanks to the second order update rule that we derive, our method consistently outperforms SOTA model-free RL algorithms in the tasks we proposed. In contrast, their method only matches the asymptotic performance of model-free RL (which is a feat for model-based RL). It is also worth pointing out that while model-based approaches hold the promise of enabling learning directly in the real world, with continued progress in sim-2-real transfer, methods such as ours that rely on accurate simulation technologies will continue to be indispensable in the field of RL.
models simply as source of data augmentation
A common approach to leverage differentable models is that of backpropagating through time (BPTT) as is the main focus of Grzeszezuk et al. (1998), Deisenroth & Rasmussen (2011). Par- mas (2088), Degrave et al. 2019), and Clavera et al. (020)) where a policy we parametrized by 0 is optimized directly in parameter space (PS), coupling the actions at each time step by the policy parameters. In contrast, our approach alternates between optimizing in trajectory space (TS), fol- lowing gradient information of the value function for an independent set of actions a; = wo (s)) and in parameter space (PS) by doing imitation learning of the monotonically improved actions a, by Te. Alternating between TS and PS allows PODS to avoid the well-know problems of BPTT (vanishing and exploding gradients), that have been reported for long time (Bengio et al. 1994).
POLICY OPTIMIZATION ON DIFFERENTIABLE SIMULATORS
Following the formulation employed by DPG methods, for a deterministic neural network policy me parameterized by weights O, the RL objective J J'Ta) and its gradient VeJ (TTO) are defined as:
/ p(so) (so)dso.
J(TO)
(1)
[
VoJ(TO) = / p(so) Vo ve (so)ddo ~
1 E V (o
(2)
k
3

---

Under review as a conference paper at ICLR 2021
where p(su) is the initial probability distribution over states, V is the value function for ne. and the second expression in Eq. approximates the integral with a sum over a batch of ho initial states sampled from S, as is standard.
Restricting our attention to an episodic problem setup with fixed time horizon N and deterministic state dynamics 81+1 f(s az), the value function gradient simplifies to:
N
VoVaO (80) = Vo (1(80.T0(5))) To(s))) +
Σ r(se, To(i))
(3)
f(s (T(s)=1)) the compu- tation of the gradient 3 equivalent to backpropagating through time (BPTT) into the policy parameters. However, BPTT can be challenging due to well known problems of vanishing or ex- ploding gradients (Degrave et al. 2019). We therefore turn our focus to the task of performing policy improvement iterations. In particular, our goal is to find a new policy â, in trajectory space, such that (TT((s)) < va (so) for batch of initial states sampled according to so ~ p(so).
Noting that the state can be specified as a recursive function
3.1 FIRST ORDER POLICY IMPROVEMENT
While the parametrization of Te is given in terms of 9 (the weights of the neural net), will choose TS policy a to directly have as parameters the actions that are executed at each time step. By representing the actions independently of each other, rather than having them coupled through 0, BPTT is therefore not required. Moreover, at the start of each policy improvement step, we initialize the TS policy the actions executed during rollout of Te(s))==ss_ Thus, VT (so) = (*)) initially, The value function gradient of policy à is then:
[ao, d1,.... aN-1 to match the output of πæ, where the individual terms a
VaV V (aVa)s Va (V(()(00,))) as) + Σ (s)((a-1))
(4)
where (a) = [s0, 81 (ao)....,, (xNNN-1)) 1)] is the vector of the state trajectory associated to the policy rollout. For the sake of clarity we now switch notation from Va d(.).
da
d (8o)
ava ava ds
da + Ös dā
(5)
da
For known, differentiable reward, the terms
av8 and ava can be easily computed analytically. In
contrast, the Jacobian da that represents the way in which the state trajectory changes the policy da à changes, is the first piece of information that we will require from a differentiable simulator. Furthermore, notice that even though we are not BPTT, the lower triangular structure of de encodes da the dependency of a particular point in state space on all the previous actions during a rollout (see the Appendix A.. for more details on the Jacobian structure.
The first order update rule for policy à is then computed as:
dV((s))
+ Oa dā
(6)
Since this update rule uses the policy gradient (i.e. the direction of local steepest ascent), there exists a value Qa > O such that (su) < y (su). In practice, we use the simulator to run a standard line-search on aa ensure the inequality holds. We note, however, that if desired, Oa, can also be treated as a hyperparameter that is tuned to a sufficiently small value.
Once the policy has been improved, the parameters of the neural net policy тo by running gradient descent on the following loss:
the corresponding state trajectories s(a) to update
b
N
Lo
1
ΣΣ
(T(.s))) a1.11
k
2
4

---

Under review as a conference paper at ICLR 2021
where the gradient and update rule given by:
& N
VoLo = - E
V
VoTo(s))(To(sL.) ac,),,
(8)
k:
0 = 00 VoLo.
Here, indexes the batch of initial states used to approximate the integral in Eq 2 Notice that gradients VTJ( and VeLa are closely related for the first iteration in the policy improvement operation, where:
(Vo da
VoLo = -000 V (o
(10)
which explains why minimizing Eq improves the value function formulated in Eq. It is also worth noting that the stability of the policy improvement process is guaranteed by the parameter Ou. which is found through line search procedure such that V (so) < (so)). as well through the intermediate targets of Eq. 7 which eliminate potential overshooting problems that might occur if the gradient direction in Eq 10 was followed too aggressively.
3.2 SECOND ORDER POLICY IMPROVEMENT
For second order policy update rule, the Hessian (sv) is required. A brief derivation of this expression can be found in the Appendix and is summarized as follows:
dø²
(ZVa(su)
d Tva a ds
da +
(11)
d
dā
0s dō
0
ds T 0 ds
θ ds
ds T
(0 ds 82va
+
+ Os² da
(12)
da
Ös da
da da
da
Osda
da²
The second order tensors à de provide. As described Zimmermann et al. (2019), these terms can be computed analytically. However, they computationally expensive compute, and they often lead to the Hessian be- coming indefinite. As a consequence, ignoring these terms from the equation above results a Gauss-Newton approximation of the Hessian:
0x and de additional terms that differentiable simulator must
( d
ds gv ds 0va
H
(13)
da ds² da da²
In the expression above we assume that the rewards do not couple and a. As long as the second derivatives of the rewards with respect states and actions are positive definite, which is almost always the case, the Gauss-Newton approximation H also guaranteed be positive semi-definite. A second order update rule for a can therefore be computed as:
(VV(ss)
no + Q ...
(14)
da
Analogous to the first order improvements discussed in the previous section, the same loss Lg can be used to perform policy update on To to strictly improve its value function. In this case, Le incorporates the second order policy updates of Eq. [4] without the need to compute the Hessian of the neural network policy, and with the additional benefit of allowing the use of well-defined acceleration methods such Adam (Kingma & Ba 2015
3.3 MONOTONIC POLICY IMPROVEMENT
The combination of simple line search on a, together with the use of La to update To provides a simple and very effective way of preventing overshooting 0 is updated. PODS therefore features
5

---

Under review as a conference paper at ICLR 2021
monotonic increases in performance, shown through experiments. As summarized in Figure for the task controlling a 2D pendulum such that it goes to stop as quickly as possible (see the experiments section for detailed description of task), both the first and second order policy improvement methods are well-behaved. Nevertheless, there is a drastic difference in convergence rates, with the second order method winning by significant margin.
Algorithm 1: PODS: Policy Optimization via Differentiable
Simple Pendulum
Simulators
for epoch = 1, M do
for sample = 1, k: do
Roowed
Sample initial condition 80,i
Collect πe by rolling out me starting from 50.;
1,003
Compute improved policy a; (Eq 6
Eq
end
Run gradient descent on La (Eq 7 such that the output of ao matches a; for the entire sequence of states s(@.)
Stepe tutof)
Performance of first and second order update rules.
Figure 2:
end
In contrast other approaches such as PPO (Wang et al. 2019) and SAC (Haarnoja et al., 2018). our policy update scheme does not need to be regularized by a KL-divergence metric, demonstrating its numerical robustness. Our method is only limited by the expressive power of policy mas needs to approximate a well. For reasonable network architectures, this is not problem, especially since a corresponds to local improvements. The overall PODS algorithm is summarized above. For the experiments we present in the next section, we collected k = 4000 rollouts for each epoch, and performed 50 gradient descent steps on Le for each policy optimization iteration.
4 EXPERIMENTS
Handle a
Elastic cables or inextensible rods
Payload x
Figure 3: Experiments left to right; 2D pendulum, 3D double pendulum, Cable driven payload 2D,
Discretized 3D rope
The environments used in our experiments set up cable-driven payload manipu- lation control problems that are inspired by the types of applications visualized in Figure [[] For all these examples, illustrated in Figure the action space is defined by the veloeity of more handles, which are assumed to be directly controlled by a robot, and the state space is defined by the position of the handle as well the position and velocity of the payload. We model our dynamical systems as mass-spring networks by connecting payloads to handles or to each other via stiff bilateral or unilateral springs. Using simulation engine that follows closely the description in Zimmermann et al. 2219)) we use BDF2 integration scheme, as it exhibits very little numerical damping and stable even under large time steps. Although this is not a common choice for RL environments, the use of higher order integration schemes also improves simulation quality and ac- curacy, as pointed out by Zhong et al. (2020). The Jacobian ds which is used for both the first order and second order policy updates, computed analytically dø sensitivity analysis, as described in detail Zimmermann et al. (2018). The computational cost of computing this Jacobian is significantly less than performing the sequence of simulation steps needed for policy rollout.
Environments:
The control problems we study here are deceptively simple. All the environments fall in the category of underactuated systems and, in consequence, policies for such environments must fully leverage the system's dynamics to successfully achieve task. The lack of numerical damping in the motion's payload, in particular, necessitates control policies that very precise, small emors lead to

---

Under review as a conference paper at ICLR 2021
noticeable oscillations. These environments also enable us incrementally increase the complexity of the tasks in order to study the scalability of our method, as well as that of the RL algorithms we compare against. For comparison purposes, in particular,
three different types of dynami- cal systems; 2D Simple Pendulum, 3D Simple Pendulum, and 3D Double Pendulum. A detailed description of these environments is presented in Appendix A.2
For all the environments, the action space describes instantaneous velocities of the handles, which are restricted to remain within physically reasonable limits.
Tasks: In order to encode our tasks, used continuous rewards that are function of the fol- lowing state variables: the position of the handle (p), the position of the mass points representing the payloads relative tains a term that a function of the actions which are taken. This term takes the form of a simple regularizer that aims to discourage large control actions.
target position (æ)), and their global velocities (v). The reward also
r(st, = uuplppel wn 22 2
(15)
1|11 + Wuallay
w
where the coefficients up: Way Was Wa allow each sub-objective to be weighted independently, as is commonly done. This very general reward formulation allows us to define two different tasks that apply to each of the three systems described above:
Go stop: Starting from an initial state with non-zero velocity, the pendulum must go to stop as quickly as possible in a downward configuration. For this task the weights " = w²y = 0.
Go to stop at the origin: In addition to stopping as fast as possible, the system must come to rest at target location, which, without loss of generality, is chosen be the origin.
The architecture of the neural network policies that we used is detailed in Appendix |A.3] For a fair comparison, the neural network policies for PODS, PPO and SAC were initialized with the same set of initial weights. We fine tuned hyper parameters of PPO and SAC to get the best performance we could, and otherwise ran standard implementations provided in Achiam (2018).
4.1 RESULTS
The monotonically improving behaviour of PODS can be seen in Figure [5] The reward reported is the result of averaging the reward of T000 rollouts started from test bed of unseen initial states. Even if the initial progress of PODS is not always as fast as PPO SAC, it consistently leads to higher reward after a small number of epochs. We note that the standard deviations visualized in this figure are indicative of a large variation in problem difficulty for the different state-space points that seed the test rollouts (e.g. double pendulum that has little momen- tum is easier to be brought to a stop than one that is swinging wildly). As can be seen, the tasks that demand the payloads to be brought to stop specific location are considerably more challenging. The supplementary video illustrates the result the rollouts to provide an intuition into the quality of the control policies learned with our method. Furthermore, Appendix A. presents convergence plots for the cable driven payload 2D, and the discretized 3D rope environments.
2D Simple Pendulum
4.000
aco
3,00
Rowegt
ico
1.000
Steps cstof)
Suegs (xlli))
PODS
Figure 4: Comparison of PODS update rules against BPTT
To further explore the benefits of the PODS second order update rule, we com- pared against the approach of BPTT which naturally leverages the differentiability of the model. We found BPTT to be highly sensitive to the weight initialization of the policy. In Figure 4 we report results using the weight initialization that we found to favor BPTT the most. When training neural network policies, doing BPTT for 100 steps rollout is effectively equivalent to backpropagating through a network that is 100 times deeper than the actual network policy, which is itself a feat considering that despite introducing a terminal cost function stabilize BPPT, Clavera et al. (2020)
PODS vs BPTT:
7

---

Under review as a conference paper at ICLR 2021
only reports results of effectively BPTT for maximum of 10 steps. Nontheless, BPTT is able to outperform PODS with the 1st order update rule. However, PODS with the 2nd order update rule is able to significantly outperform BPTT both in terms on convergence rates and final performance. Even though, a second order formulation BPTT could be derived, it's deployment would involve the hessian of the neural network policy which computationally expensive. In contrast, PODS first order and second order formulations are equally easy to deploy.
2D Simple Pendulum
3D Simple Pendulum
3D Double Pendulum
4,000
3,203
3,000
7,013
Rewad
Roward
2,000
0,793
20
Stepe Is
Strpe (xlif)
Stepe cot
Simple (Stop Origin)
Simple Pendulum (Stop Origin)
Pendulum (Stop Origin)
Rosua
Rewuad
Rewud
83
68.5
37
15
(=10))
St
(cllf)
SAL
FP12
Figure 5: Comparison of reward curves. Our algorithm, PODS, achieves better performance com- pared to other algorithms, PPO and SAC
To better understand the relative performance of the control policies learned with PODS, SAC and PPO, we report the terminal kinetic energy (KE) of the payload (Fig- urel)), the average magnitude of control action (Figure (Appurig-Appendxx)) and the average distance to the target location for the Stop At Origin tasks (Figure 7) note, lower is better, and upon convergence, control policies learned with PODS adequately solve each individual problem our randomized test bed. The shaded areas represent half the standard deviation of each metric. For figures with logarithmic scale only the upper side of the standard deviation is presented.
PODS, SAC, and PPO:
For the task of stopping as fast as possible, PODS leads to terminal kinetic energy that is typically orders of magnitude better than the other approaches (Top row Figure 6). For the tasks of stopping the origin, SAC achieves very good terminal KE. The policies SAC learns, however, output large, high frequency handle motions, as seen in the high control cost in Figure 8 These actions end up counteracting the natural dynamic oscillations of the payload. The same strategy for the 3D double pendulum, however, is unsuccessful. In contrast, PODS learns control policies that use less effort to solve the control tasks than both SAC and PPO. This indicates that our policies learn to leverage the dynamics of the payload much more effectively. a characteristic that we attribute to the local improvement steps which, by design, monotonically improve the value functions of the control policies. Furthermore, it should also be noted that the class of fine manipulation tasks that we are dealing with represents a challenge for policies that output noisy actions.
5 CONCLUSION AND FUTURE WORK
In this paper, presented highly effective strategy for policy optimization. As core idea be- hind our approach, we exploit differentiable simulators to directly compute the analytic gradient of policy's value function with respect to the actions it outputs. Through specialized update rules,
8

---

Under review as a conference paper at ICLR 2021
Simple Pendulum
3D Simple Penduluum
3D Double Pendulum
105
100
Terninal Kiretis haary
Tominal Kizzate launge
20
Torntal Kiecia
10.2
10-1
10--2
10
Stepo (=lid)
Supp (xi0f)
Steps (s
Simple Pendolum (Stop Origin)
Simple Penchllum (Stop Origin)
3D Double Pendullmm (Stop Origin)
10-77
tn:
in
Mpl
Temital
un
Temitall
10
Figure 6: Final Kinetic Energy (averaged over period of 10 time-steps after the policy is rolled
out)
2D Simple Panduluum (Stop Origin)
3D Simple Pendulum (Stop Origin)
3D Douile Pendulum (Stop Origin)
10".
302
101
Distrce (nggggnn))
[niinnn))
:0"
Dinate 20 otgin
in-
in
My
10.9
20
.0-4
10-7
in
30--
-0-9
10"
10m4
Stape (xl0")
Sueps bxE0")
Stape (x10')
Figure 7: Final distance to Origin
this gradient information used monotonically improve the policy's value function. We demon- strated the efficacy of our approach by applying it to series of increasingly challenging payload manipulation problems, and we showed that it outperforms two SOTA RL methods both terms of convergence rates, and in terms of quality of the learned policies.
Our work opens up exciting avenues future investigations. For example, although we evaluated PODS in isolation in order to best understand its strengths, it would be interesting to interleave it with existing RL methods. This will require extensions of our formulation to stochastic policies, and it would allow the relative strengths of different approaches be effectively combined (e.g. explo- ration vs exploitation, with PODS excelling in the latter but not being designed for the former). We also excited about the prospect of applying PODS other types of control problems, particularly ones that include contacts (e.g. locomotion, grasping, etc). Although the need for a specialized sim- ulator makes the application to standard RL benchmark suites (Brockman al. 2016 Tassa al. 2018) challenging, we note that sim-2-ree success with differentiable simulatt lator has been recently reported in the context soft locomoting robots (Bern al.) 2019). With continued evolution of such simulation technologies, we are excited about the prospect of creating new benchmark suite applicable to approaches such as PODS that use differentiable simulators their core.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, al. On evaluation of embodied navigation agents. arXiv preprint 1 2018.
Y. Bengio, Simard, and Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-116, 1994.
James Bern, Pol Banzet, Roi Poranne, and Stelian Coros. Trajectory optimization for cable-driven soft robot locomotion. In Proc. Robot. Sci. Syst. 2019.
Rinu Boney, Norman Di Palo, Mathias Berglund, Alexander Ilin, Juho Kannala, Antti Rasmus, and Harri Valpola. Regularizing trajectory optimization with denoising autoencoders. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bur Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 2859 28699. Curran Associates, Inc., 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym CoRR, abs/1606.01540, 2016. URL httl:attp//axxi> org/ abs abs/1606.015c
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-Based Reinforcement Learning via Meta- Vielcy Optimization In Aude Billard, Anca Dra- gan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 617 629. PMLR, 29-31 Oct 2018.
URL http::/ http://aaaa........eoeee///////. woress/v87/claveeeee
m.
html
Ignasi Clavera, Yao Fu, and Pieter Abbeel. Model-augmented actor-critic: Backpropagating through paths. In International Conference Learinng Representations, 2020. URL https:
// //openreview..o. httt/forrmmdd=kk1n2222
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, 2016-2019.
robotics and machine learning. http::/pybullet...
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey R. Allen, Josh Tenenbaum, and J. Zico Kolter. End-to-end differentiable physics for learning and control. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Confer- ence on Neural Information Processing Systems 2018, NewrIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 7178-7189, 2018. URL http: papprrsppppassee nips . ce ccppaper/ www.eerennaaableepysiii...o for Soo--leerninee Lewaninngg---e.ee...ee.eee and-iinoono-ooooooooooo
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis wyffels. A differentiable physics engine for deep learning in robotics. Frontiers in Newrorobotics, 13:6, 2019. ISSN 1662-5218. doi: 10. 3389/fnbot.2019 2019000006 URL http://www.......rrraooooorrraaa
Enbot. 2019 00006
Mare Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-hased and data-efficient ap- proach to policy search. In Proceedings of the 28th International Conference International Conference on Machine Learning, ICML'11, pp. 465-472, Madison, WI, USA, 2011. Omnipress. ISBN 978 175030666..
Evan Drumwright, John Hsu, Nathan P. Koenig, and Dylan A. Shell. Extending open dynam- ics engine for robotics simulation. In Noriaki Ando, Stephen Balakirsky, Thomas Hemker, Monica Reggiani, and Oskar von Stryk (eds.), Simulation, Modeling, and Programming for Autonomous Robots Second International Conference, SIMPAR 2010, Darmstadt, Germany, November 15-18. 2010. Proceedings, volume 6472 of Lecture Notes in Computer Science, pp- 38-50. Springer, 2010. doi: 10. 10.1/97/97 3-62- 17319-6) 7. URL https: //oi.. org/1C. 1007
10

---

Under review as a conference paper at ICLR 2021
Alexis Dubureq, Yann Chevaleyre, Nicolas Bredech, and Guilhem Boéris. Online trajectory plan- ning through combined trajectory optimization and function approximation: Application to the exoskeleton atalante. arXiv preprint (rXfi:1910.005:00 2019.
Scott Fujimoto, Herke Hoof, and David Meger. AAdrrssiig Function Approximation Error in Critic Methods CoRR, abs/1 802.09477, 2018. URL httttt//aaaxii www.wawwww.a...aaaaaoo.o.oorrort orq/abs.
1802. 09477
Radek Grzeszczuk, Demetri Terzopoulos, and Geoffrey Hinton. Neuroanimator: Fast neural net- work emulation and control physics-based models. In Proceedings of the 25th Anmual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '98, pp. 9-20, New York, NY, USA, 1998. Association for Computing Machinery, ISBN 0897919998, doi: 10.1145/280814.28888 280011. htllhttss///do.org//// 1twp:////...11/11111111111
Shixiang Gu, Ethan Holly, Timothy P. Lillicrap, and Sergey Levine. Deep Reinforcement Learning For Robotic Manipulation, CoRR, abs/1610.000333, 2016. URL http: http::/axxiv..ot/ wrg.:or.gaaa... abs
1610.00633
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. [Soft Actor-Critic Al- gorithms and Applications CoRR, abs/1812.059005 2018.
1612.05900
Tuomas Haarnoja, Schoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to Walk Via Deep Reinforcement Learning In Proceedings of Robotics: Science and Systems, FreiburgimBreisggau, Germany, June 2019. doi: (0.15607/RSS.2219.XV000
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26)_eaa5587, 2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine QT- Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation CoRR, abs/1806.10293. 2018. URL [allat////axxi org/ abs 1806. 10293
Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojciech Jaskowski. Viz- doom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua
Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http:
6980
Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A sur- vey The International Journal of Robotics Research, 32(11):1238 1274, 2013. doi: 10.1177/ 0278364913495721. URLpttts::
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model--Enemmble Region Policy Optimization CoRR, 0a2///802.119222 2018. URl.http://arxiv.oo abs/ 1802. 10592
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, pp. 1-9, 2013a.
Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimization. In Ad- vances in neural information processing systems, pp. 207-215, 2013b.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):333-1373, 2016.
Yuxi Li. Deep Reinforcement Learning
CoRR, abs/1810.06339, 2018.
URL http: arxiv.
brg/abs/ /000s/18110000 06339

---

Under review as a conference paper at ICLR 2021
Junbang Liang, Ming C. Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buce Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 Decem- ber 2019, Vancouver, BC, Canada, pp. 771-780, 2019. URL http: http:::/pappprs..em nips cc/ paper 8365 www.aaiffeeetaaiieeeeeee -able- Sertthssing simulatiionn wwweoo--oo-nneeeeeeeeeeee Soeverse-probleee
for
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:
02977
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukeuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv: 1.03633, 2016.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pp. 1937, New York, New York, USA, 20-22 Jun 2016. PMLR. htt:: //proceedings.. mlr. press / v48 / mwwaannillllllmmmmmmmmmmmmmmmmmm 6. html
Igor Mordatch and Emo Todorov. Combining the benefits of function approximation and trajectory optimization. 2015. doi: 10. 201xxx00s..
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural Network Dy- namics for Model-Based Deep Reinforcement Learning with Model-Free Fine- TineTTnnng CoRR,
2017. http: orq/ abs/ 1 708.0000000 02596
Anusha Nagabandi, Kurt Konoglie, Sergey Levine, and Vikash Kumar. Deep Dynamics Models for Learning Dexterous Manipulation In Conference on Robot Learning (CoRL), 2019.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning Dexterous In-Hand Manipulation CoRR, abs/1808.00117 2018. URL
http: / /tt.xxxivvooro www.://.rrgaab///8000 00177
Paavo Parmas. Total stochastic gradient algorithms and applications in reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems 31, pp. 10204-10214, Curran Associates, Inc., 2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wdvintage-weiggtted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019. 2019.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. Structured Prediction No- No-Regret Online Learning
A Reduction of Imitation Learning and In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourleenth International Conference Artificial In- telligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL http: pttp:://oocceinngaaa mlr. press/v15/rossl11 html
Connor Schenck and Dieter Fox. Guided policy search with delayed sensor measurements, arXiv preprint 1rXhv:1609.00370 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz, Trust Region Policy Optimization In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-188888 1897, Lille, France, 07-09 Jul 2015. PMLR. URL httt:://orrcceedngg... mir. press/v37/st w3//sch/ waa.anhhhhmmmmmmnmmmmmmmmmmmmm1m htmmll
12

---

Under review as a conference paper at ICLR 2021
John Schulman, Philipp Moritz, Sergey Levine, Michael 1. Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation In 4th International Conference Learning Representations, ICLR 2016. San Juan, Puerto Rico, May 2-4, 2016, /506. 02438
Conference Track Proceedings, 2016. Utt.htt:::/arswwwooggg
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:180..13332. 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite Technical report, DeepMind, January 2018. URL https://ara
/arxiv.vvvvvv..////v wtt/ /1001.1101..10 00690
Yuhui Wang, Hao He, Xiaoyang Tan, and Yaozhong Gan. Trust region-guided proxi- mal policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems 32, pp. 626-636. Curran Associates, Inc., 2019. URL http: /apprrs nips oc paper 8352 1.52-trust.. 1352-trust-reeee regi on- ded- ma mal--pllioee pol--polccyoop..ooooool wwt.toppiiiiiiiiiiiimiiio wtt.iiittiiio...............
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. International Conference on Learning Representations, 2020.
H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost In 2019 International Conference on Roboties and Automation (ICRA), pp. 3651-3657, May 2019. doi: 10.1109/ICRA.2019. 20199.999..99..9
Simon Zimmermann, Roi Poranne, and Stelian Coros. Optimal control via second order sensitivity abs / 905 08534
analysis. CoRR, abs/1905.08555 2018.
Simon Zimmermann, Roi Poranne, James M. Bern, and Stelian Coros. PuppetMaster: Robotic animation of marionettes, ACM Trans. Graph. 38(4), July 2019. ISSN 0730-0301, doi: 10. 145// URL /ttp:///////////////////////////
13

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1
VALUE FUNCTION HESSIAN
(s))
d
0V ds
da²
da
da
Ös da
d Tava
d
@v
ds
+ dā
dā da
0s dā
css vm
d
0V
ds
0V ds
dā Osda da dā
0s dā 0s dā da V ds
ds vo 0 ds 0v 0s
0Ve
ds
T 0 ds
a ds
+
da Osda da²
da
dads da Ös da 8
Ös da
da da
av ds
0 ds
0 ds
ds
02va ds
+ 8
+
0s
dă
0s
dā
dā
dā
dā
Og dā
Osda
Oa
A.2 DETAILED DESCRIPTION OF ENVIROMENTS
2D Simple Pendulum: This system corresponds to a cable-driven pendulum in 2D (Figure)
left). The handle of the pendulum is constrained to move only along the horizontal axis in order
the degree to which control policy can exploit the natural dynamics of the system.
3D Simple Pendulum: For this system the pendulum is free move in 3D, but the handle is restricted to moving along horizontal plane.
3D Double Pendulum: Extending the dynamical system above, the payload for this problem consists of two mass points that coupled to each other via stiff bilateral spring. The dimen- sionality of the state space doubles, and the system exhibits very rich and dynamic motions.
Cable drive payload 2D: For this environment we have a densely connected network point masses and two handles that are constrained to move along the horizontal axis.
Rope in 3D: For this environment we use 5 point masses to descretize rope in 3D and one handle that is constrained to move on the horizontal plane.
Table 1: Summary of environments
State space
Action space
Total mass
Environment
Constraints
Additional Info
2D Simple
Handle along
50gr
Pendulum
horizontal axis
3D Simple
Handle on horizontal plane
2
50gr
Pendulum
3D Double
Handle on horizontal plane
100gr
Pendulum
Cable driven
Handles along horizontal axis
Attachment to handle uses deformable cables
2
200gr
payload
Handle on horizontal plane
All conections are deformable cables
Rope in 3D
3"5=15
250gr
A.3 ARCHITECTURE OF NEURAL NETWORK POLICIES
The neural networks representing the control policies for all our environments share the same archi- tecture, fully connected layers of 256 units each with ReLU activations and one output layer with Tanh activation, to ensure that the policy only outputs commands that are within the velocity limits.
14

---

Under review as a conference paper at ICLR 2021
A.4 PODS: ADDITIONAL FIGURES
Simple Pendulum
Simple
Double Pendulum
1072
Artoasoss BleeP
30
Attica eoe SllnPP
-0-4
-0-4
10n4
10
Steps (s
Suqpett (10)))
Sugpe (xitf))
2D Simple Pendulum (Stop Origin)
3D Simple Pendulum (Stop Origin)
3D Double Pendulum (Stop Origin)
12
10
10-
cost
13
10
coe Ell!!!?
13-1
10
Mn WWr wmwa
13-4
10"
13:0
[0"1
10-13
Stepo (x10")
Supps (x10f)
Stepe (s10f)
Figure 8: Average handle velocity (control effort)
2D Simple Pendulura
3D3 Simple Pendulum
3D Doaile Pendulum
3,30
5,000
2,023
Rewed
Ronad
2,000
0,700
20
1,000
35
Stops
Stope
Stepe
2D Simple Pendulum (Stop
Simple Pendulum (Stop Origin)
3D Double Pendalum (Stop
270
Boood
3.55
60
60.9
S0
15
36
Staps CU10")
Steps DLt0h)
Steps o010")
SAC
PPO
Clur
Figure 9: Extended Reward
15

---

Under review as a conference paper at ICLR 2021
A.5 DIPFERANTIABLE SIMULATOR
Following the approach in Zimmermann et al. (2018). the sensitivity de has the structure of the da
figure below.
ds
-1
dG
dā
0s
dā
Figure 10:
Structure of sensitivity matrix ds that encodes the dependency of state on all the da
previous actions
A.6 ADDITIONAL DEMOS
See the accompanying video for more details of the following environments.
Cable-driven payload (Stop Origin)
Cable-driven payload 2D (Stop Origin)
120
10
10-1
90
Terminal Kinetic Energy
10-2
Reward
10-3
10-1
10->
10-6
21
12
18
20
Steps (x10")
Steps (x10*)
Cable-driven payload 2D (Stop Origin)
Cable-driven payload 2D (Stop Origin)
10'
10a
Actions cod ||ul
Distance to origin [m]
10-1
10-2
10-1
10-5
0
10-1
Steps (x10")
Steps (x10))
Own
Figure 11: Cable driven payload 2D
Figure reproduced with authorization of the authors
http://arxxvv.rg/aabe// 195.995.00000 08534
16

---

Under review as a conference paper at ICLR 2021
Rope (Stop Origin)
Rope (Stop Origin)
40
10
Terminal Kinetic Energy [J]
10
30
10-1
Reward
10 2
20
a
10-1
10
10-3
0
18
27
10
27
36
Steps (x10))
Steps (x10*)
Rope (Stop Origin)
Rope (Stop Origin)
10
10
10
Actions cost I[1 12
Distance to origin [m]
10a
10
10
10-5
10-3
10-1
9
36
15
Steps (x10%)
Steps (x10')
Omn
Figure 12: Rope in 3D
17


Reviews:
[{"review_id": "Qwiv3JadCL9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pretext: Physics simulators are evolving and pose as a good approximation of the real world. Differentiable simulators are evolving and can provide analytic gradients.\n\nProblem formulation: Can analytical gradients form differentiable simulators be leveraged to formulate better policy optimization algorithms?\n\nApproach: Alternate between - optimization in trajectory space by explicitly taking gradients of the value function and - optimization in policy space by performing imitation learning over the improved actions. \n\nStrengths: Tractable and stable optimization procedure due to the decoupling of the trajectory and policy optimization. Seconds order monotonic efficient method requiring minimal hyper parameter searches. \n\nDiscussions:\n1. Inconsistencies between the positioning of the paper and experimental analysis\n- Under the lens that the PODS primary contribution should be viewed as a new technique for policy optimization when environment-dynamics is differentiable, viewing SAC and POLO merely as a policy optimization algorithm (independent of training and test dynamics differences) is justified. Expectation under this viewpoint warrants comparison to similar (interleaving supervisedlearning with trajectory optimization) optimization paradigms like POLO(Plan online, learn offline: Efficient learning and exploration via model-based control\nK Lowrey, A Rajeswaran, S Kakade, E Todorov), IGOR (Interactive control of diverse complex characters with neural networks. I Mordatch, K Lowrey, G Andrew, Z Popovic), GPS (Learning neural network policies with guided policy search under unknown dynamics S Levine, P Abbeel), etc. \n- Under the premise that PODS' primary contribution should be viewed as a paradigm for sim2real, we need to account for the training and test (true) dynamics being different. While SAC and PPO should train using samples from true dynamics for training and report performance on true dynamics as well. PODS' should train only using samples from train-dynamics but report performance using true dynamics. Sample complexity should be compared only wrt to the sample drawn from the true dynamics (i.e. training sample for model-free methods and sample required for system-ID for PODS and similar methods)\n\n2. No results on commonly accepted benchmarks tasks are presented. I recognize that the authors claimed that these tasks can pose for future benchmark tasks. But in this case, the focus and rigor for an evaluation will be completely different. With both algorithms, as well as the tasks, being new, its hand to establish the strength and credibility of both using one another. My suggestion will be to implement one/ two commonly accepted benchmark tasks and compare against the reported performance on them. \n\nSuggestions: \n1. Inline with point1 above, an interesting idea will be to perform an experiment in the spirit of sim2real. Where few established benchmarks tasks can be implemented with the differential engine. This is used as training-dynamics for training policies, and the resulting policy is evaluated for performance on equivalent OpenAI-Gym-mujoco as true dynamics.\n2. \"a\" and \"pi\" are interchangeably called policies. Introducing a more crisp definition separating the two will help with clarity.\n2. Missing figure reference in the appendix A2\n3. Information on the line search for the experiments reported will be helpful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LNnUBkQndLN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe paper argues for using differentiable simulators for policy optimization. To avoid back propagation through time, the paper splits the policy optimization problem into two steps: i) find improved action sequence for a set of initial conditions, ii) fit a parametric policy to the set of improved action sequences. First and second order methods are presented and evaluated on a version of payload-on-crane stabilization problem.\n\nDecision\n-----------\nI vote for rejecting this paper for the following reasons.\n\n1. There is little novelty in this work. The method can be seen as a version of Guided Policy Search (GPS) [1] where the trajectory optimization part is done using analytical gradients.\n2. The evaluations don't seem to be reliable. In Fig. 5, PPO and SAC show very bad performance. Given that the task is quite low-dimensional, it is surprising that they work so badly. If that is nevertheless the case, then it would be beneficial to add experiments at least on the Classic Control environments from OpenAI Gym, such that one can compare to other papers.\n3. The choice of the step size alpha is questionable. The paper says that unlike PPO and SAC, the proposed method does not require any policy update regularization. However, setting a small value of alpha effectively limits the policy update step. So, the proposed algorithm is still a version of conservative policy update algorithms, such as [1].\n\nReferences\n---------------\n[1] Montgomery, W., & Levine, S. (2016). Guided policy search as approximate mirror descent. arXiv preprint arXiv:1607.04614.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7X9t8ngevK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper presents a class of RL algorithm based on analytic gradients of the objective, \ncoming directly from a fully differentiable simulator and differentiable rewards.\nThe analytic gradients are used to optimize trajectories.\nThe known states and actions from the optimized trajectories are then used to improve the deterministic policy.\nOverall, this is an interesting paper that maximally leverages analytic gradient information\nthat come directly from a differentiable simulator, rather than a learned differentiable dynamics model.\nThe methods prove to be highly effective for the given class of problems they are tested on.\n\n## Strengths\n- This work provides strong insights into the benefits of differentiable simulators for\n  trajectory-guided reinforcement learning.  The paper provides a solid discussion and evaluation of\n  the use of first and second-order methods.\n- It demonstrates effective and efficient results for a difficult class of problem that has direct application,\n  i.e., cable-suspended loads\n\n## Weaknesses\n- The foundations of the trajectory-based optimization method exist in prior art [Zimmermann 2019]\n- There is perhaps a restricted class of tasks can benefit from the approach:  contact collisions, infinite-horizons,\n  and non-differentiable rewards may be off limits?\n\n## Recommendation\nThe RL community can benefit from this work in several ways, as outlined with the various strengths given above.\n\n## Questions\n- Is the method restricted to fixed-length episodic tasks?  Fine if it is, but readers will want to know.\n- Do local minima pose a problem, given that the method effectively does no true exploration?\n- Can PODS be viewed as a form of fast analytic-derivative trajectory optimization followed by behavior cloning?\n- Can hard constraints be handled, e.g., obstacles or joint limits that are to be avoided?\n- What are the tradeoffs between working in handle-space vs robot joint space?\n\n## Additional Feedback\nWhat would be the impact of using many fewer rollouts per epoch?\n\nWhy not also compare the results for some tasks which are commonly used for RL, e.g., pendulum and acrobot swing-up?\n\nAppendix A.2 only provides very minimal details on the environments.\nWhat are the dimensions and masses of the various pendulum systems?\n\nThe following is an early paper (1998) that uses back-prop-through-time to directly optimize a control objective:\n\"NeuroAnimator:Fast Neural Network Emulation and Control of Physics-Based Models\"\n\nThe proposed work has connections with Guided Policy Search (Levine and Koltun, IMCL 2013),\nwhich also uses optimized trajectories as training data, only in their case for a stochastic policy\nand using differential dynamic programming, instead of the deterministic policy setting used here\ntogether with a Hessian-dependent Newton's method. Explaining these connections could be helpful\nto many.\n\n[ Rebuttal / question responses are acknowledged, and also the other reviews.\n    I think the algorithm is sufficiently novel, and it does really well on some difficult new problems.\n    I see the paper as being about a new policy optimization algorithm leveraging differential dynamics, and not about sim2real.\n   However, the biggest limitation is pointed about by both R3 and R4, i.e., per R3: \"With both algorithms, as well as the tasks, being new, its hand to establish the strength and credibility of both using one another. \"  And so with this in mind, I am changing my score to a 6, i.e., marginally in favor of accept. ] ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was evaluated by 3 knowledgeable reviewers, where 2 reviewers were leaning for acceptance and one reviewer argued for rejection, rendering the paper a borderline paper. The positive negative points that were raised about the paper during the discussion are summarized below:

Strength:
- The presented policy optimization method provides strong results
- It provided strong insights into the benefits of differentiable simulators for trajectory-guided reinforcement learning
- The direction of differentiable physics simulators is very promising and the provided benchmarks are interesting

Weak points:
- (i) The main contribution of the paper is a novel trajectory optimization method that uses analytical gradients. In a second step, a neural network is fitted to generalize the control from the single trajectories. The given approach is very much related to existing methods such as GPS or IGOR, just that the trajectory optimization is different. A comparison to these methods is needed. For example, how does the algorithm compare to using iLQG as trajectory optimization method ( we could also use analytical gradients for the linierazations used in iLQG)?
- (ii) While the presented tasks are very interesting, there is no benchmark on a more well known task. Hence, it is hard to evaluate the performance in comparison to other algorithms.

The paper defintely has interesting contributions in terms of the new trajectory optimization method and I could live with (ii) as the presented experiments are challenging and interesting, the contribution needs to be better evaluated as comparisions to other trajectory optimization methods are missing. I am sure that the paper will be accepted at another conference with this additional experiments, however, without it the paper is incomplete and I can unfortunately not recommend acceptance.


Decision: Reject