gradSim: Differentiable simulation for system identification and visuomotor control

1 INTRODUCTION . Accurately predicting the dynamics and physical characteristics of objects from image sequences is a long-standing challenge in computer vision . This end-to-end reasoning task requires a fundamental understanding of both the underlying scene dynamics and the imaging process . Imagine watching a short video of a basketball bouncing off the ground and ask : “ Can we infer the mass and elasticity of the ball , predict its trajectory , and make informed decisions , e.g. , how to pass and shoot ? ” These seemingly simple questions are extremely challenging to answer even for modern computer vision models . The underlying physical attributes of objects and the system dynamics need to be modeled and estimated , all while accounting for the loss of information during 3D to 2D image formation . Depending on the assumptions on the scene structre and dynamics , three types of solutions exist : black , grey , or white box . Black box methods ( Watters et al. , 2017 ; Xu et al. , 2019b ; Janner et al. , 2019 ; Chang et al. , 2016 ) model the state of a dynamical system ( such as the basketball ’ s trajectory in time ) as a learned embedding of its states or observations . These methods require few prior assumptions about the system itself , but lack interpretability due to entangled variational factors ( Chen et al. , 2016 ) or due to the ambiguities in unsupervised learning ( Greydanus et al. , 2019 ; Cranmer et al. , 2020b ) . Recently , grey box methods ( Mehta et al. , 2020 ) leveraged partial knowledge about the system dynamics to improve performance . In contrast , white box methods ( Degrave et al. , 2016 ; Liang et al. , 2019 ; Hu et al. , 2020 ; Qiao et al. , 2020 ) impose prior knowledge by employing explicit dynamics models , reducing the space of learnable parameters and improving system interpretability . ∗Equal contribution Most notably in our context , all of these approaches require precise 3D labels – which are laborintensive to gather , and infeasible to generate for many systems such as deformable solids or cloth . We eliminate the dependence of white box dynamics methods on 3D supervision by coupling explicit ( and differentiable ) models of scene dynamics with image formation ( rendering ) 1 . Explicitly modeling the end-to-end dynamics and image formation underlying video observations is challenging , even with access to the full system state . This problem has been treated in the vision , graphics , and physics communities ( Pharr et al. , 2016 ; Macklin et al. , 2014 ) , leading to the development of robust forward simulation models and algorithms . These simulators are not readily usable for solving inverse problems , due in part to their non-differentiability . As such , applications of black-box forward processes often require surrogate gradient estimators such as finite differences or REINFORCE ( Williams , 1992 ) to enable any learning . Likelihood-free inference for black-box forward simulators ( Ramos et al. , 2019 ; Cranmer et al. , 2020a ; Kulkarni et al. , 2015 ; Yildirim et al. , 2017 ; 2015 ; 2020 ; Wu et al. , 2017b ) has led to some improvements here , but remains limited in terms of data efficiency and scalability to high dimensional parameter spaces . Recent progress in differentiable simulation further improves the learning dynamics , however we still lack a method for end-to-end differentiation through the entire simulation process ( i.e. , from video pixels to physical attributes ) , a prerequisite for effective learning from video frames alone . We present∇Sim , a versatile end-to-end differentiable simulator that adopts a holistic , unified view of differentiable dynamics and image formation ( cf . Fig . 1,2 ) . Existing differentiable physics engines only model time-varying dynamics and require supervision in state space ( usually 3D tracking ) . We additionally model a differentiable image formation process , thus only requiring target information specified in image space . This enables us to backpropagate ( Griewank & Walther , 2003 ) training signals from video pixels all the way to the underlying physical and dynamical attributes of a scene . Our main contributions are : • ∇Sim , a differentiable simulator that demonstrates the ability to backprop from video pixels to the underlying physical attributes ( cf . Fig . 2 ) . • We demonstrate recovering many physical properties exclusively from video observations , including friction , elasticity , deformable material parameters , and visuomotor controls ( sans 3D supervision ) • A PyTorch framework facilitating interoperability with existing machine learning modules . We evaluate ∇Sim ’ s effectiveness on parameter identification tasks for rigid , deformable and thinshell bodies , and demonstrate performance that is competitive , or in some cases superior , to current physics-only differentiable simulators . Additionally , we demonstrate the effectiveness of the gradients provided by ∇Sim on challenging visuomotor control tasks involving deformable solids and cloth . 2 RELATED WORK . Differentiable physics simulators have seen significant attention and activity , with efforts centered around embedding physics structure into autodifferentiation frameworks . This has enabled differentiation through contact and friction models ( Toussaint et al. , 2018 ; de Avila Belbute-Peres et al. , 1Dynamics refers to the laws governing the motion and deformation of objects over time . Rendering refers to the interaction of these scene elements – including their material properties – with scene lighting to form image sequences as observed by a virtual camera . Simulation refers to a unified treatment of these two processes . 2018 ; Song & Boularias , 2020b ; a ; Degrave et al. , 2016 ; Wu et al. , 2017a ; Research , 2020 ( accessed May 15 , 2020 ) , latent state models ( Guen & Thome , 2020 ; Schenck & Fox , 2018 ; Jaques et al. , 2020 ; Heiden et al. , 2019 ) , volumetric soft bodies ( Hu et al. , 2019 ; 2018 ; Liang et al. , 2019 ; Hu et al. , 2020 ) , as well as particle dynamics ( Schenck & Fox , 2018 ; Li et al. , 2019 ; 2020 ; Hu et al. , 2020 ) . In contrast , ∇Sim addresses a superset of simulation scenarios , by coupling the physics simulator with a differentiable rendering pipeline . It also supports tetrahedral FEM-based hyperelasticity models to simulate deformable solids and thin-shells . Recent work on physics-based deep learning injects structure in the latent space of the dynamics using Lagrangian and Hamiltonian operators ( Greydanus et al. , 2019 ; Chen et al. , 2020 ; Toth et al. , 2020 ; Sanchez-Gonzalez et al. , 2019 ; Cranmer et al. , 2020b ; Zhong et al. , 2020 ) , by explicitly conserving physical quantities , or with ground truth supervision ( Asenov et al. , 2019 ; Wu et al. , 2016 ; Xu et al. , 2019b ) . Sensor readings have been used to predicting the effects of forces applied to an object in models of learned ( Fragkiadaki et al. , 2016 ; Byravan & Fox , 2017 ) and intuitive physics ( Ehsani et al. , 2020 ; Mottaghi et al. , 2015 ; 2016 ; Gupta et al. , 2010 ; Ehrhardt et al. , 2018 ; Yu et al. , 2015 ; Battaglia et al. , 2013 ; Mann et al. , 1997 ; Innamorati et al. , 2019 ; Standley et al. , 2017 ) . This also includes approaches that learn to model multi-object interactions ( Watters et al. , 2017 ; Xu et al. , 2019b ; Janner et al. , 2019 ; Ehrhardt et al. , 2017 ; Chang et al. , 2016 ; Agrawal et al. , 2016 ) . In many cases , intuitive physics approaches are limited in their prediction horizon and treatment of complex scenes , as they do not sufficiently accurately model the 3D geometry nor the object properties . System identification based on parameterized physics models ( Salzmann & Urtasun , 2011 ; Brubaker et al. , 2010 ; Kozlowski , 1998 ; Wensing et al. , 2018 ; Brubaker et al. , 2009 ; Bhat et al. , 2003 ; 2002 ; Liu et al. , 2005 ; Grzeszczuk et al. , 1998 ; Sutanto et al. , 2020 ; Wang et al. , 2020 ; 2018a ) and inverse simulation ( Murray-Smith , 2000 ) are closely related areas . There is a rich literature on neural image synthesis , but we focus on methods that model the 3D scene structure , including voxels ( Henzler et al. , 2019 ; Paschalidou et al. , 2019 ; Smith et al. , 2018b ; Nguyen-Phuoc et al. , 2018 ) , meshes ( Smith et al. , 2020 ; Wang et al. , 2018b ; Groueix et al. , 2018 ; Alhaija et al. , 2018 ) , and implicit shapes ( Xu et al. , 2019a ; Chen & Zhang , 2019 ; Michalkiewicz et al. , 2019 ; Niemeyer et al. , 2020 ; Park et al. , 2019 ; Mescheder et al. , 2019 ) . Generative models condition the rendering process on samples of the 3D geometry ( Liao et al. , 2019 ) . Latent factors determining 3D structure have also been learned in generative models ( Chen et al. , 2016 ; Eslami et al. , 2018 ) . Additionally , implicit neural representations that leverage differentiable rendering have been proposed ( Mildenhall et al. , 2020 ; 2019 ) for realistic view synthesis . Many of these representations have become easy to manipulate through software frameworks like Kaolin ( Jatavallabhula et al. , 2019 ) , Open3D ( Zhou et al. , 2018 ) , and PyTorch3D ( Ravi et al. , 2020 ) . Differentiable rendering allows for image gradients to be computed w.r.t . the scene geometry , camera , and lighting inputs . Variants based on the rasterization paradigm ( NMR ( Kato et al. , 2018 ) , OpenDR ( Loper & Black , 2014 ) , SoftRas ( Liu et al. , 2019 ) ) blur the edges of scene triangles prior to image projection to remove discontinuities in the rendering signal . DIB-R ( Chen et al. , 2019 ) applies this idea to background pixels and proposes an interpolation-based rasterizer for foreground pixels . More sophisticated differentiable renderers can treat physics-based light transport processes ( Li et al. , 2018 ; Nimier-David et al. , 2019 ) by ray tracing , and more readily support higher-order effects such as shadows , secondary light bounces , and global illumination . 3 ∇Sim : A UNIFIED DIFFERENTIABLE SIMULATION ENGINE Typically , physics estimation and rendering have been treated as disjoint , mutually exclusive tasks . In this work , we take on a unified view of simulation in general , to compose physics estimation and rendering . Formally , simulation is a function Sim : RP × [ 0 , 1 ] 7→ RH × RW ; Sim ( p , t ) = I . Here p ∈ RP is a vector representing the simulation state and parameters ( objects , their physical properties , their geometries , etc . ) , t denotes the time of simulation ( conveniently reparameterized to be in the interval [ 0 , 1 ] ) . Given initial conditions p0 , the simulation function produces an image I of height H and width W at each timestep t. If this function Sim were differentiable , then the gradient of Sim ( p , t ) with respect to the simulation parameters p provides the change in the output of the simulation from I to I + ∇Sim ( p , t ) δp due to an infinitesimal perturbation of p by δp . This construct enables a gradient-based optimizer to estimate physical parameters from video , by defining a loss function over the image space L ( I , . ) , and descending this loss landscape along a direction parallel to −∇Sim ( . ) . To realise this , we turn to the paradigms of computational graphs and differentiable programming . ∇Sim comprises two main components : a differentiable physics engine that computes the physical states of the scene at each time instant , and a differentiable renderer that renders the scene to a 2D image . Contrary to existing differentiable physics ( Toussaint et al. , 2018 ; de Avila Belbute-Peres et al. , 2018 ; Song & Boularias , 2020b ; a ; Degrave et al. , 2016 ; Wu et al. , 2017a ; Research , 2020 ( accessed May 15 , 2020 ; Hu et al. , 2020 ; Qiao et al. , 2020 ) or differentiable rendering ( Loper & Black , 2014 ; Kato et al. , 2018 ; Liu et al. , 2019 ; Chen et al. , 2019 ) approaches , we adopt a holistic view and construct a computational graph spanning them both .

Summary: This paper presents a framework for performing both differentiable physics simulations and differentiable rendering. This fully differentiable simulation and rendering pipeline is then employed to perform system identification tasks, directly from video frames, being able to match or outperform both visual-based and state-based baselines. Moreover, the potential of this framework to be applied for visuomotor control is also demonstrated.