Universal Controllers with Differentiable Physics for Online System Identification

1 INTRODUCTION . In order for robots to shine in real-world applications , they need to handle ever-changing and unpredictable situations in real environments . For instance , a robot waiter should be able to serve a new type of dish without spilling food , and an autonomous vehicle should take a person safely to an unvisited destination . Creating artificial agents that can operate in changing and unknown environments is a longstanding problem in the robotics community . The collective wisdom of the robotic research community in recent years indicates that enabling learning agents to work in changing and unknown environments is not about making one big breakthrough , but rather making many small but informed decisions . One general approach advances control policies such that they can operate more robustly ( Tan et al. , 2018 ) or more adaptively ( Cully et al. , 2015 ) in testing environments . However , these methods usually exhibit sub-optimal task performance or require additional fine-tuning in the target environment . Alternative approaches advance simulation techniques to bring the training environment closer to the testing one prior to learning a control policy , such as training a dynamics model ( Jiang et al. , 2021 ) or identifying simulation parameters ( Tan et al. , 2018 ) from data . These methods are often used in offline settings as learning or identifying an accurate simulation model can be time consuming . Thus , they can not handle changing environments naturally . Work such as Yu et al . ( 2017 ) have also investigated methods that learn models to perform online system identification . However , a learned model often does not generalize well to unknown environments or those not seen during training . Recent developments in differentiable physics simulation potentially offer a more effective way to address these challenges by advancing both control and simulation techniques . By utilizing fast computation of analytical gradients , one can devise more computationally and sample efficient optimal control and system identification algorithms . Recent differentiable physics simulators , such as NimblePhysics ( Werling et al. , 2021 ) , provide fast computation of analytical gradients in the face of constraint satisfaction and non-differentiable contact handling . These enable generic gradient-based optimizers to solve contact-rich optimal control problems . While promising , differentiable physics simulation does not solve the fundamental problem of multiple local minima due to ill-conditioned cost functions , often exacerbated by long-horizon and highly nonlinear differential equations . This paper introduces a new approach for creating resilient and adaptive agents by combining differentiable physics simulation for online system identification and reinforcement learning for offline policy training . Online system identification can be formulated as a short-horizon , local optimization problem , but must be solved fast . This plays to the strength of differentiable physics simulation which provides analytical gradients efficiently , while avoiding the pitfalls of poor cost function landscapes . On the other hand , for challenging control problems with long-horizon cost functions , we resort to a reinforcement learning approach leveraging samples ( “ rollouts ” ) generated offline at scale to train a control policy . We explore many possible situations the agent might encounter when operating in the testing environment by varying the simulation parameters during training and learning a Universal Controller ( UC ) conditioned on the simulation parameters . At test time , we use differentiable physics simulation to continuously optimize the simulation parameters based on the most recent history of observations ( DiffOSI ) . The optimal simulation parameters will “ modulate ” the universal policy to output the optimal action for the currently identified environment . We evaluate our approach on two robotic control tasks , a cartpole balancing problem and a robot arm table wiping task involving rich contact phenomena . We show that our proposed approach combining a Universal Controller and a Differentiable physics-based Online System Identification module ( UC-DiffOSI ) can outperform pure learning-based or traditional system identification methods . Finally , we demonstrate that our approach can be applied to environments with changing dynamics or un-modeled effects . 2 RELATED WORK . Deep Reinforcement Learning and Domain Randomization . Deep reinforcement learning has been proven to be effective in learning complex motor skills for simulated robots , such as running ( Yu et al. , 2018 ) , parkour ( Heess et al. , 2017 ) , and dressing ( Clegg et al. , 2018 ) . However , these controllers often perform poorly on real robot hardware due to the discrepancies between the simulated and real environment , also known as the sim-to-real gap ( Neunert et al. , 2017 ) . Domain randomization of the simulation physics parameters has been extensively explored to help the simulation-trained controller transfer to a different target environment , where a robust control policy is trained to perform well for a wide variety of simulated environments ( Peng et al. , 2017 ; Tan et al. , 2018 ; Hwangbo et al. , 2019 ; Exarchos et al. , 2020 ; OpenAI et al. , 2019 ) . However , policies trained from domain randomization often exhibit over-conservative behaviors , leading to sub-optimal performance ( Tan et al. , 2018 ) . Different from these methods , we develop a domain adaptation approach by training adaptive controllers that can adjust behavior for different environments using an estimation of the environment parameters . This enables our controller to achieve better performance than a domain randomization controller . Domain Adaptation . To achieve better task performance in novel situations , researchers have developed adaptive controllers that can adjust behavior for different environments . Szita et al . ( 2003 ) showed that Q-learning , using event-learning , can find near-optimal policies in varying environments . Heess et al . ( 2015 ) demonstrated that control policies modified to use recurrent networks are also capable of dealing with unknown kinematic parameters such as link lengths . Xu et al . ( 2020 ) presented a deep reinforcement learning method that encodes the dynamic context online to achieve a stable non-planar pushing task controller . Yu et al . ( 2017 ) proposed a system using a Universal Policy and Online System Identification ( OSI ) function to explicitly incorporate model parameters to adapt to varying environments . These methods usually identify the environment parameters ( explicitly or implicitly ) and then adjust the controllers to adapt to the new environment . Differentiable Simulation . In recent years , researchers have built more efficient and featurecomplete differentiable physics engines . These engines support 3D rigid body and contact constraints between spheres and planes ( Degrave et al. , 2016 ) , analytic differentiation of a linear com- plementarity problem ( de Avila Belbute-Peres et al. , 2018 ) , modeling soft bodies via a differentiable real-time differentiable Material Point Method ( Hu et al. , 2018 ) , support differentiable cloth simulation ( Liang et al. , 2019 ) , optimize for large numbers of objects and contact interactions ( Qiao et al. , 2020 ) , and support articulated rigid bodies with contact ( Werling et al. , 2021 ) . Prior work , such as Toussaint et al . ( 2018 ) and Heiden et al . ( 2019 ) , also showed that differentiable physics can be integrated for end-to-end controller learning , in addition to parameter learning . Jatavallabhula et al . ( 2021 ) further integrated differentiable rendering to remove dependency on 3D vision in an end-to-end learning pipeline . 3 METHODS . Our goal is to design a system that can handle changing or unknown dynamics in the environment . The true dynamics in the target environment can be described by xt+1 = fµ ( xt , τt ) , where x = ( q , q̇ ) denotes the robot ’ s sensed states and their time derivatives , and τ denotes the control actions . fµ evolves the system from timestep t to t + 1 with dynamics parameters µ . We aim to predict optimal controls τ ∗t which maximize task performance . The controls are predicted in the first part of our system , the universal controller ( UC ) : ( x , µ ) 7→ τ . The second part is a differentiable physics engine that performs online system identification ( DiffOSI ) : { ( xi , xi+1 , τi ) } 7→ µ . The overview of the system is shown in Figure 1 . Together , they form a robust controller capable of handling unknown or changing environment dynamics . 3.1 LEARNING A UNIVERSAL CONTROLLER . Universal Controller ( UC ) augments a regular robotic controller by conditioning it on parameters of the environment µ , such as friction coefficient or robot payload . This information is crucial for the controller to select appropriate actions for different environments , yet are non-trivial to infer directly from sensory input . By providing this additional information to the UC , we expect it to outperform a regular policy given the true environment parameters . A successful UC should perform near-optimally for a wide range of µ ’ s . Given that the best way to obtain a control policy can be different across tasks , the training of UC largely depends on the task to be performed . In this work , we tailor the training of UC to two control tasks of interest : cart pole balancing and table wiping . For the cart-pole balancing problem , we want to obtain a controller that directly sends torque commands to the robot at high-frequency . Thus , we directly apply a reinforcement learning approach to obtain a Universal Control Policy as done in Yu et al . ( 2017 ) . On the other hand , for the table wiping problem , we adopt a hierarchical control structure where the learned UC needs to modulate the parameters of a low-level admittance controller per wiping motion . A black-box optimization technique is more suitable for this low-frequency problem . More details on how we train our UPs can be found in Section 4 . Note that we train our UC with a set of training environments gµ̂ , which approximates the target environment fµ̄ . µ̄ and µ̂ need not represent the same set of parameters , and neither do fµ̄ and gµ̂ need to represent the same model , as the exact governing equations of fµ̄ is usually unknown . We aim to use a gµ̂ diverse enough to train a robust UC and expressive enough to approximate all possible trajectories evolved with fµ̄ . 3.2 DIFFERENTIABLE PHYSICS FOR ONLINE SYSTEM IDENTIFICATION . We use differentiable physics to identify some unknown physics parameters µ̂ ∈ Rg that parameterize the dynamics of the system . The numerical modeling , xt+1 = gµ̂ ( xt , τt ) , in the differentiable physics engine should be the same as in UC training approximating the target environment dynamics , xt+1 = fµ̄ ( xt , τt ) . This way , the nature of the UC ’ s inputs stays consistent . To perform system identification , DiffOSI requires first collecting a small number of samples X̄ = { x̄t , τ̄t } from target environment fµ̄ . DiffOSI uses these samples to optimize for a µ̂ that minimizes the differences between the resulted trajectory { x̂t , τ̂t } and the target state-action history { x̄t , τ̄t } . DiffOSI requires a minimum of two samples , but if the problem is nondeterministically underconstrained ( e.g. , in the presence of contact ) , more samples ( e.g. , 30-50 ) may be required to exercise all dynamics of the system . At the beginning ( first iteration k = 0 ) , we initialize µ̂ = µ0 ( e.g. , mean of expected distribution ) . For each iteration of DiffOSI optimization k , we execute the UC for a certain number of steps Tk ≤ |X̄k| using actions predicted with UC ( x̄k , µ̂k ) . With the collected samples X̄k , we use DiffOSI to optimize for the µ̂k that minimizes the following objective function : L ( X̄k ) = ∑ t∈X̄k φ ( q̂t+1 , q̄t+1 ) + φ ( ˆ̇qt+1 , ¯̇qt+1 ) , ( 1 ) where ( q̂t+1 , ˆ̇qt+1 ) = gµ̂k ( q̄t , ¯̇qt , τ̄t ) ( 2 ) and φ is any differentiable distance function . A differentiable physics engine enables the computation of gradients of L with respect to the unknown parameters µ : ∂L ( X̄k ) ∂µ̂k ( 3 ) Our system is agnostic to the choice of the differentiable physics engine . We use the Nimble differentiable physics engine ( based on DART ) by Werling et al . ( 2021 ) , which has the advantage of being able to handle articulated rigid bodies and differentiate through contact .

Summary: The paper tackles the problem of learning robot controllers that can handle changing or unknown environments. It proposes to use differentiable physics for online system identification and reinforcement learning for offline policy training. The differentiable physics module estimates simulation parameters from robot history and feeds this to the controller that is parameterized by these simulation parameters. They use domain randomization to ensure the universal controller conditioned on simulation parameters is robust to changing environments (simulation parameters). At test time, the differential physics simulation is used to estimate the simulation parameters to bias the controller to output controls for the 'correct' simulation parameters.  The proposed approached is evaluated against several benchmarks on the cartpole and a tabletop wiping tasks and has been shown to outperform the baselines, which include domain randomization and some domain adaptation approaches.