PODS: Policy Optimization via Differentiable Simulation

1 INTRODUCTION . The main goal in RL is to formalize principled algorithmic approaches to solving sequential decision-making problems . As a defining characteristic of RL methodologies , agents gain experience by acting in their environments in order to learn how to achieve specific goals . While learning directly in the real world ( Haarnoja et al. , 2019 ; Kalashnikov et al. , 2018 ) is perhaps the holy grail in the field , this remains a fundamental challenge : RL is notoriously data hungry , and gathering real-world experience is slow , tedious and potentially unsafe . Fortunately , recent years have seen exciting progress in simulation technologies that create realistic virtual training grounds , and sim-2real efforts ( Tan et al. , 2018 ; Hwangbo et al. , 2019 ) are beginning to produce impressive results . A new class of differentiable simulators ( Zimmermann et al. , 2019 ; Liang et al. , 2019 ; de Avila Belbute-Peres et al. , 2018 ; Degrave et al. , 2019 ) is currently emerging . These simulators not only predict the outcome of a particular action , but they also provide derivatives that capture the way in which the outcome will change due to infinitesimal changes in the action . Rather than using simulators as simple black box oracles , we therefore ask the following question : how can the additional information provided by differentiable simulators be exploited to improve RL algorithms ? To provide an answer to this question , we propose a novel method to efficiently learn control policies for finite horizon problems . The policies learned with our approach use neural networks to model deterministic actions . In a departure from established methodologies , learning these policies does not hinge on learned approximations of the system dynamics or of the value function . Instead , we leverage differentiable simulators to directly compute the analytic gradient of a policy ’ s value function with respect to the actions it outputs for a specific set of points sampled in state space . We show how to use this gradient information to compute first and second order update rules for locally optimal policy improvement iterations . Through a simple line search procedure , the process of updating a policy avoids instabilities and guarantees monotonic improvement of its value function . To evaluate the policy optimization scheme that we propose , we apply it to a set of control problems that require payloads to be manipulated via stiff or elastic cables . We have chosen to focus our attention on this class of high-precision dynamic manipulation tasks for the following reasons : • they are inspired by real-world applications ranging from cable-driven parallel robots and crane systems to UAV-based transportation to ( Figure 1 ) ; • the systems we need to learn control policies for exhibit rich , highly non-linear dynamics ; • the specific tasks we consider constitute a challenging benchmark because they require very precise sequences of actions . This is a feature that RL algorithms often struggle with , as the control policies they learn work well on average but tend to output noisy actions . Given that sub-optimal control signals can lead to significant oscillations in the motion of the payload , these manipulation tasks therefore make it possible to provide an easy-to-interpret comparison of the quality of the policies generated with different approaches ; • by varying the configuration of the payloads and actuation setups , we can finely control the complexity of the problem to test systematically the way in which our method scales . Figure 1 : Real-world applications that inspire the control problems we focus on in this paper The results of our experiments confirm our theoretical derivations and show that our method consistently outperforms two state-of-the-art ( SOTA ) model-free RL algorithms , Proximal Policy Optimization ( PPO ) ( Wang et al. , 2019 ) and Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) , as well as the model-based approach of Backpropagation Through Time ( BPTT ) . Although our policy optimization scheme ( PODS ) can be interleaved within the algorithmic framework of most RL methods ( e.g . by periodically updating the means of the probability distributions represented by stochastic policies ) , we focused our efforts on evaluating it in isolation to pinpoint the benefits it brings . This allowed us to show that with minimal hyper-parameter tuning , the second order update rule that we derive provides an excellent balance between rapid , reliable convergence and computational complexity . In conjunction with the continued evolution of accurate differentiable simulators , our method promises to significantly improve the process of learning control policies using RL . 2 RELATED WORK . Deep Reinforcement Learning . Deep RL ( DRL ) algorithms have been increasingly more successful in tackling challenging continuous control problems in robotics ( Kober et al. , 2013 ; Li , 2018 ) . Recent notable advances include applications in robotic locomotion ( Tan et al. , 2018 ; Haarnoja et al. , 2019 ) , manipulation ( OpenAI et al. , 2018 ; Zhu et al. , 2019 ; Kalashnikov et al. , 2018 ; Gu et al. , 2016 ) , and navigation ( Anderson et al. , 2018 ; Kempka et al. , 2016 ; Mirowski et al. , 2016 ) to mention a few . Many model-free DRL algorithms have been proposed over the years , which can be roughly divided into two classes , off-policy methods ( Mnih et al. , 2016 ; Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) and on-policy methods ( Schulman et al. , 2015 ; 2016 ; Wang et al. , 2019 ) , based on whether the algorithm can learn independently from how the samples were generated . Recently , model-based RL algorithms ( Nagabandi et al. , 2017 ; Kurutach et al. , 2018 ; Clavera et al. , 2018 ; Nagabandi et al. , 2019 ) have emerged as a promising alternative for improving the sample efficiency . Our method can be considered as an on-policy algorithm as it computes first or second-order policy improvements given the current policy ’ s experience . Policy Update as Supervised Learning . Although policy gradient methods are some of the most popular approaches for optimizing a policy ( Kurutach et al. , 2018 ; Wang et al. , 2019 ) , many DRL algorithms also update the policy in a supervised learning ( SL ) fashion by explicitly aiming to mimic expert demonstration ( Ross et al. , 2011 ) or optimal trajectories ( Levine & Koltun , 2013a ; b ; Mordatch & Todorov , 2015 ) . Optimal trajectories , in particular , can be computed using numerical methods such as iterative linear–quadratic regulators ( Levine & Koltun , 2013a ; b ) or contact invariant optimization ( Mordatch & Todorov , 2015 ) . The solutions they provide have the potential to improve the sample efficiency of RL methods either by guiding the learning process through meaningful samples ( Levine & Koltun , 2013a ) or by explicitly matching action distributions ( Mordatch & Todorov , 2015 ) . Importantly , these approaches are not only evaluated in simulation but have also been shown to be effective for many real-world robotic platforms , including manipulators ( Schenck & Fox , 2016 ; Levine et al. , 2016 ) and exoskeletons ( Duburcq et al. , 2019 ) . Recently , Peng et al . ( 2019 ) proposed an off-policy RL algorithm that uses SL both to learn the value function and to fit the policy to the advantage-weighted target actions . While our method shares some similarities with this class of approaches that interleave SL and RL , the updates of our policy do not rely on optimal trajectories that must be given as input . Rather , we show how to leverage differentiable simulators to compute locally optimal updates to a policy . These updates are computed by explicitly taking the gradient of the value function with respect to the actions output by the policy . As such , our method also serves to reinforce the bridge between the fields of trajectory optimization and reinforcement learning . Differentiable Models . Our approach does not aim to learn a model of the system dynamics , but rather leverages differentiable simulators that explicitly provide gradients of simulation outcomes with respect to control actions . We note that traditional physics simulators such as ODE Drumwright et al . ( 2010 ) or PyBullet Coumans & Bai ( 2016–2019 ) are not designed to provide this information . We build , in particular , on a recent class of analytically differentiable simulators that have been shown to effectively solve trajectory optimization problems , with a focus on sim-2-real transfer , for both manipulation ( Zimmermann et al. , 2019 ) and locomotion tasks ( Bern et al. , 2019 ) . Degrave et al . ( 2019 ) embed a differentiable rigid body simulator within a recurrent neural network to concurrently perform simulation steps while learning policies that minimize a loss corresponding to the control objective . While their goal is related to ours , we show how to leverage explicitlycomputed gradients to formulate second order policy updates that have a significant positive effect on convergence . Furthermore , in contrast to Degrave et al . ( 2019 ) , we show that PODS consistently outperforms two common RL baselines , PPO ( Wang et al. , 2019 ) and SAC ( Haarnoja et al. , 2018 ) . Also related to our method is the very recent work of Clavera et al . ( 2020 ) . Their observation is that while most model-based RL algorithms use models simply as a source of data augmentation or as a black-box oracle to sample from ( Nagabandi et al. , 2017 ) , the differentiability of learned dynamics models can and should be exploited further . In an approach that is related to ours , they propose a policy optimization algorithm based on derivatives of the learned model . In contrast , we directly use differentiable simulators for policy optimization , bypassing altogether the need to learn the dynamics – including all the hyperparameters that are involved in the process , as well as the additional strategies required to account for the inaccuracies introduced by the learned dynamics ( Boney et al. , 2019 ) . Thanks to the second order update rule that we derive , our method consistently outperforms SOTA model-free RL algorithms in the tasks we proposed . In contrast , their method only matches the asymptotic performance of model-free RL ( which is a feat for model-based RL ) . It is also worth pointing out that while model-based approaches hold the promise of enabling learning directly in the real world , with continued progress in sim-2-real transfer , methods such as ours that rely on accurate simulation technologies will continue to be indispensable in the field of RL . A common approach to leverage differentable models is that of backpropagating through time ( BPTT ) as is the main focus of Grzeszczuk et al . ( 1998 ) , Deisenroth & Rasmussen ( 2011 ) , Parmas ( 2018 ) , Degrave et al . ( 2019 ) , and Clavera et al . ( 2020 ) , where a policy πθ parametrized by θ is optimized directly in parameter space ( PS ) , coupling the actions at each time step by the policy parameters . In contrast , our approach alternates between optimizing in trajectory space ( TS ) , following gradient information of the value function for an independent set of actions at = πθ ( s ) |s=st , and in parameter space ( PS ) by doing imitation learning of the monotonically improved actions at by πθ . Alternating between TS and PS allows PODS to avoid the well-know problems of BPTT ( vanishing and exploding gradients ) , that have been reported for a long time ( Bengio et al. , 1994 ) .

Summary: The paper argues for using differentiable simulators for policy optimization. To avoid back propagation through time, the paper splits the policy optimization problem into two steps: i) find improved action sequence for a set of initial conditions, ii) fit a parametric policy to the set of improved action sequences. First and second order methods are presented and evaluated on a version of payload-on-crane stabilization problem.