gradSim: Differentiable simulation for system identification and visuomotor control
Authors: J. Krishna Murthy
Miles Macklin
Florian Golemo
Vikram Voleti
Linda Petrini
Martin Weiss
Breandan Considine
Jérôme Parent-Lévesque
Kevin Xie
Kenny Erleben
Liam Paull
Florian Shkurti
Derek Nowrouzezahrai
Sanja Fidler
Venue: ICLR 2021 Poster (2021)
Keywords: Differentiable simulation
System identification
Physical parameter estimation
3D scene understanding
3D vision
Differentiable rendering
Differentiable physics

Abstract: In this paper, we tackle the problem of estimating object physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current best solutions to the problem require precise 3D labels which are labor intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. In this work we present gradSim, a framework that overcomes the dependence on 3D supervision by combining differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This unique combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Furthermore, our unified computation graph across dynamics and rendering engines enables the learning of challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to/better than techniques that require precise 3D labels.

Published as a conference paper at ICLR 2021
Sim: DIFFERENTIABLE SIMULATION FOR SYSTEM IDENTIFICATION AND VISUOMOTOR CONTROL
https: : //ggddiimmmmmmmmm github. io
Krishna Murthy Jatavallabhula" 1.3.4 Miles Macklin Florian Golemo 1.3 Vikram Voleti Linda Petrini?, Martin Weiss Breandan Considine Jérôme Parent-Lévesquees Kevin Xie Kenny Erleben Liam Paull Florian Shkurti6.7 Derek Nowrouzezahrai2s, and Sanja Fidler267
Montreal Robotics and Embodied AI Lab, NVIDIA, *Mila, Université de Montréal, MceGill, University of Toronto, Vector Institute, University Copenhagen
Figure I: V.im a unified diffcrentiable rendering and multiphysics framework that allows solving range of control and parameter estimation tasks (rigid bodies, deformable solids, and cloth) directly from images/video.
ABSTRACT
We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible create for many systems such as deformable solids cloth. We present VSim, framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, unified computation graph -saanning from the dynamics and through the rendering process enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
INTRODUCTION
Accurately predicting the dynamics and physical characteristics of objects from image sequences is long-standing challenge in computer vision. This end-to-end reasoning task requires a fundamental understanding of both the underlying scene dynamics and the imaging process. Imagine watching short video of a basketball bouncing off the ground and ask: "Can we infer the mass and elasticity of the ball, prediet its trajectory, and make informed decisions. e.g., how to pass and shoot?" These seemingly simple questions are extremely challenging to answer even for modern computer vision models. The underlying physical attributes of objects and the system dynamics need to be modeled and estimated, all while accounting for the loss of information during 3D to 2D image formation.
Depending on the assumptions on the scene structre and dynamics, three types of solutions exist: black, grey, or white box. Black box methods (Watters et al., 2017; Xu et al., 2019b; Janner et al., 2019; Chang al., 2016) model the state of dynamical system (such as the basketball's 's trajectory in time) as learned embedding of its states or observations. These methods require few prior assumptions about the system itself. but lack interpretability due entangled variational factors (Chen al., 2016) due to the ambiguities in unsupervised learning (Greydanus et al., 2019; Cranmer et al., 2020b). Recently, grey box methods (Mehta et al., 2020) leveraged partial knowledge about the system dynamics improve performance. In contrast, white box methods (Degrave et al., 2016; Liang et al., 2019; Hu al., 2020; Qiao et al., 2020) impose prior knowledge by employing explicit dynamics models, reducing the space of learnable parameters and improving system interpretability.
Equual contribution

---

Published as a conference paper at ICLR 2021
Rollout
Gradient
Camera
Colors
Lights
Textures
Mass
Vertices
Friction
Forces
States
Loss
Elasticity
(d) Predicted
(e) Ground
(a) Object
(b) Physics Engine
(4) Renderer
Video
Video
Existing Methods Supervision)
gradsim [Image Supervision)
Figure 2: VSim: Given video observations an evolving physical system (c), we randomly initialize scene object properties (a) and evolve them over time using differentiable physics engine (b), which generates states. Our renderer (c) processes states, object vertices and global rendering parameters produce image frames for computing loss. We backprop through this computation graph estimate physical attributes and controls. Existing methods rely solely on differentiable physics engines and require supervision state-space (f), while Sim only needs image-space supervision (g).
Most notably in our context, all of these approaches require precise 3D labels which are labor- intensive to gather, and infeasible to generate for many systems such as deformable solids or cloth.
We eliminate the dependence of white box dynamics methods on 3D supervision by coupling explicit (and differentiable) models scene dynamics with image formation (rendering).
Explicitly modeling the end-to-end dynamics and image formation underlying video observations challenging, even with access to the full system state. This problem has been treated in the vision, graphics, and physics communities (Pharr et al., 2016; Macklin et al., 2014), leading to the development of robust forward simulation models and algorithms. These simulators are not readily usable for solving inverse problems, due in part their non- cin-dfffrentiabblity.. As such, applications of black-box forward processes often require surrogate gradient estimators such as finite differences REINFORCE (Williams, 1992) enable any learning. Likelihood freeeooo--feeeeeeeee inference for black-box forward simulators (Ramos et al., 2019; Cranmer et al., 2020a; Kulkarni et al., 2015; Yildirim et al., 2017; 2015; 2020; Wu al., 2017b) has led some improvements here, but remains limited in terms of data efficiency and scalability to high dimensional parameter spaces. Recent progress in differentiable simulation further improves the learning dynamics, however we still lack a method for end-to-end differentiation through the entire simulation process (i.e., from video pixels to physical attributes), a prerequisite for effective learning from video frames alone.
We present Sim, versatile end-to-end differentiable simulator that adopts a holistic, unified view of differentiable dynamics and image formation(cf.. Fig. 1,2). Existing differentiable physics engines only model time-vrrying dynamics and require supervision in state space (usually 3D tracking). We additionally model differentiable image formation process, thus only requiring target information specified in image space. This enables signals from video pixels all the way to the underlying physical and dynamical attributes scene.
backpropagate (Griewank & Walther, 2003) training
Our main contributions
V Sim, a differentiable simulator that demonstrates the ability to backprop from video pixels to the underlying physical attributes (ef. Fig. 2).
We demonstrate recovering many physical properties exclusively from video observations, including friction, elasticity, deformable material parameters, and visuomotor controls (sans 3D supervision)
A PyTorch framework facilitating interoperability with existing machine learning modules.
We evaluate VSim's s effectiveness on parameter identification tasks for rigid, deformable and thin- shell bodies, and demonstrate performance that is competitive, in some physics-only differentiable simulators. Additionally, we demonstrate the effectiveness of the gradients provided by Sim challenging visuomotor control tasks involving deformable solids and cloth.
superior, to current
RELATED WORK
Differentiable physics simulators have seen significant attention and activity, with efforts centered around embedding physics structure into autodifferentiation frameworks. This has enabled differ- entiation through contact and friction models (Toussaint et al., 201 8; de Avila Belbute- PPres et al.,
Dynamics refers to the laws governing the motion and deformation of objects over time. Rendering refers to the interaction of these scene elements including their material properties with scene lighting to form image sequences as observed by virtual camera. Simulation refers unified treatment these two processes.
2

---

Published as a conference paper at ICLR 2021
2018; Song & Boularias, 2020b;a; Degrave et al., 2016; Wu et al., 2017a; Research, 2020 (accessed May 15, 2020), latent state models (Guen & Thome, 2020; Schenck & Fox, 2018; Jaques et al., 2020; Heiden et al., 2019). volumetric soft bodies (Hu et al., 2019; 2018; Liang et al., 2019; Hu et al., 2020), as well as particle dynamics (Schenck & Fox, 2018; Li et al., 2019; 2020; Hu et al., 2020). In contrast, Sim addresses superset of simulation scenarios, by coupling the physics simulator with differentiable rendering pipeline. It also supports tetrahedral FEM-based hyperelasticity models to simulate deformable solids and thin-shells.
Recent work on physics- baseed deep learning injects structure in the latent space of the dynamics using Lagrangian and Hamiltonian operators (Greydanus et al., 2019; Chen et al., 2020; Toth et al., 2020; Sanchez-Gonzalez et al., 2019; Cranmer et al., 2020b; Zhong et al., 2020), by explicitly conserving physical quantities, or with ground truth supervision (Asenov et al., 2019; Wu et al., 2016; Xu et al., 2019b).
Sensor readings have been used to predicting the effects of forces applied to an object in models of learned (Fragkiadaki et al., 2016; Byravan & Fox, 2017) and intuitive physics (Ehsani et al., 2020; Mottagii et al., 2015; 2016; Gupta et al., 2010; Ehrhardt et al., 2018; Yu et al., 2015; Battaglia et al., 2013; Mann et al., 1997; Innamorati et al., 2019; Standley et al., 2017). This also includes approaches that learn to model multi-object interactions (Watters et al., 2017; Xu et al., 2019b; Janner et al., 2019; Ehrhardt et al., 2017; Chang et al., 2016; Agrawal et al., 2016). In many cases, intuitive physies approaches are limited in their prediction horizon and treatment of complex scenes, as they do not sufficiently accurately model the 3D geometry nor the object properties. System identification based on parameterized physics models (Salzmann & Urtasun, 2011; Brubaker et al., 2010; Kozlowski, 1998; Wensing et al., 2018; Brubaker et al., 2009; Bhat et al., 2003; 2002; Liu al., 2005; Grzeszezuk et al., 1998; Sutanto et al., 2020; Wang et al., 2020; 2018a) and inverse simulation (Murray-Sni -mith,,, 2000) are closely related areas.
There is a rich literature on neural image synthesis, but we focus on methods that model the 3D scene structure, including voxels (Henzler al., 2019; Paschalidou al., 2019; Smith et al., 2018b; Nguyen-Phune et al., 2018), meshes (Smith et al., 2020; Wang et al., 201 8b; Groueix et al., 2018; Alhaija et al., 2018), and implicit shapes (Xu et al., 2019a; Chen & Zhang, 2019; Michalkiewicz et al., 2019; Niemeyer et al., 2020; Park et al., 2019; Mescheder et al., 2019). Generative models condition the rendering process on samples of the 3D geometry (Liao et al., 2019). Latent factors determining 3D structure have also been learned in generative models (Chen et al., 2016; Eslami et al., 2018). Additionally, implicit neural representations that leverage differentiable rendering have been proposed (Mildenhall et al., 2020; 2019) for realistic view synthesis. Many of these representations have become easy to manipulate through software frameworks like Kaolin (Jatavallabhula et al., 2019), Open3l (Zhou et al., 2018), and PyTorch3D (Ravi et 2020).
Differentiable rendering allows for image gradients to be computed w.r.t. the scene geometry, camera, and lighting inputs. Variants based on the rasterization paradigm (NMR (Kato et al., 2018), OpenDR (Loper& & Black, 2014), SoftRas (Liu et al., 2019)) blur the edges of scene triangles prior to image projection to remove discontinuities the rendering signal. DIB-R (Chen et al., 2019) applies this idea to background pixels and proposes an interpolation based rasterizer for foreground pixels. More sophisticated differentiable renderers can treat physics- based light transport processes (Li et al., 2018; Nimier- David et al., 2019) by ray tracing, and more readily support higher-order effects such as shadows, secondary light bounces, and global illumination.
3 Sim: A UNIFIED DIFFERENTIABLE SIMULATION ENGINE
Typically, physics estimation and rendering have been treated as disjoint, mutually exclusive tasks. In this work, we take on a unified view of sinulation in general, to compose physics estimation and rendering. Formally, simulation is a function Sim IR" x [0, 1] + R x HW; Sim (p, t) = Z. Here p E T a vector representing the simulation state and parameters (objects, their physical properties, their geometries, etc.). denotes the time of simulation (conveniently reparameterized to be in the interval [0, 1)). Given initial conditions Po. the simulation function produces an image I of height H and width W at each timestep t. If this function Sim were differentiable, then the gradient of Sim(p, t) with respect to the simulation parameters P provides the change in the output of the simulation from I to I + VSim(p. t)6p due to an infinitesimal perturbation of p by óp This construct enables a gradient- defining a loss function over the image space C(I,-), and descending this loss landscape along
optimizer to estimate physical parameters from video, by
3

---

Published as a conference paper at ICLR 2021
direction parallel to - Vim(()) To realise this, we turn to the paradigms of computational graphs and differentiable programming.
Sim comprises two main components: differentiable physics engine that computes the physical states of the scene at each time instant, and a differentiable renderer that renders the scene to a 2D image. Contrary existing differentiable physics (Toussaint et al., 2018; de Avila Belbute-Peres et al., 2018; Song & Boularias, 2020b;a; Degrave et al., 2016; Wu et al., 2017 7a; Research, 2020 (accessed May 15, 2020; Hu al., 2020; Qiao et al., 2020) or differentiable rendering (Loper & Black, 2014; Kato et al., 2018; 8; Liu et al., 2019; Chen et al., 2019) approaches, we adopt a holistic view and construct a computational graph spanning them both.
3.1 DIFFERENTIABLE PHYSICS ENGINE
Under Lagrangian mechanics, the state of a physical system can be described in terms of generalized coordinates q. generalized velocities q = u. and design/model parameters 0. For the purpose of exposition, we make distinction between rigid bodies, or deformable solids, or thin-shell models of cloth, etc. Although the specific choices of coordinates and parameters vary, the simulation procedure virtually unchanged. We denote the combined state vector by s(t) [q(t), u(t)]
The dynamic evolution of the system is governed by second order differential equations (ODEs) of the form M(s, 0 ) f(s, 0), where M is mass matrix that depends on the state and parameters.
The forces on the system may be parameterized by design parameters (e.g. Young's 's modulus). Solutions to these ODEs may be obtained through black box numerical integration methods, and their derivatives calculated through the continuous adjoint method (Chen et al., 2018). However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector s s(t) at the start of time step, and the updated state at the end of the time step s+ s(t + At). An arbitrary discrete time integration scheme can be then be abstracted as the function g(s sst 0) 0 relating the initial and final system state and the model parameters A.
Gradients through this dynamical system can be computed by graph- based autodiff frame- works (Paszke et al., 2019; Abadi et al., 2015; Bradbury et al., 2018), by program transformation approaches (Hu et al., 2020; van Merriënboer et al., 2018). Our framework is agnostic to the specifics of the differentiable physics engine, however in Appendices A through D detail an efficient approach based on the source-code transformation of parallel kernels, similar to Diih'aichii (Hu et al., 2020). In addition, we describe extensions to this framework support mesh-based tetrahedral finite-element models (FEMs) for deformable and thin-shell solids. This is important since we require surface meshes to perform differentiable rasterization as described in the following section.
3.2 DIFFERENTIABLE RENDERING ENGINE
A renderer expects scene description inputs and generates color image outputs, all according to sequence of image formation stages defined by the forward graphics pipeline. The scene description includes complete geometric descriptor of scene elements, their associated material/reflectance properties, light source definitions, and virtual camera parameters. The rendering process is not generally differentiable,, as visibility and occlusion events introduce discontinuities. Most interactive renderers, such as those used in real-time applications, employ rasterization process to project 3D geometric primitives onto 2D pixel coordinates, resolving these visibility events with non- differentiable operations.
Our experiments employ two differentiable alternatives to traditional rasterization, SoftRas (Liu et al., 2019) and DIB-R (Chen et al., 2019), both of which replace discontinuous triangle mesh edges with smooth sigmoids. This has the effect of blurring triangle edges into semi-transparent boundaries, thereby removing the non-differentiable discontinuity of traditional rasterization. DIB- R distinguishes between foreground pixels (associated to the principal object being rendered in the scene) and background pixels (for all other objects, if any). The latter are rendered using the same technique as SoftRas while the former are rendered by bilinearly sampling a texture using differentiable UV coordinates.
Sim performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us trade computation for accuracy by rendering fewer frames than dynamics updates.
4

---

Published as a conference paper at ICLR 2021
4 EXPERIMENTS
We conducted multiple experiments to test the efficacy of Vim on physical parameeer identification from video and visuomotor control, address the following questions:
Can we accurately identify physical parameters by backpropagating from video pixels, through the simulator? (Ans: Yes, very accurately, cf. 4.1)
What the performance gap associated with using VSi (2D supervision) vs. differentiable
physics-only engines (3D supervision)? (Ans: Sim is compelitive/superior, cf. Tables 1, 3)
How do loss landscapes differ across differentiable simulators (ii)) and their non-differentiable counterparts? (Ans: Loss landscapes for Vim are smooth, cf. 4.1.3)
Can we use Vim for visuomotor control tasks? (Ans: Yes, without any 3D supervision, cf. 4.2)
How sensitive is Sim to modeling assumptions system level? (Ans: Moderately, cf. Table 4)
Each of our experiments comprises an environment & that applies partieular set of physical forces and/or constraints, a (differentiable) loss function L that implicitly specifies an objective, and an initial guess &g of the physical state the simulation. The goal is to recover optimal physics parameters l that minimize L, by backpropagating through the simulator.
4.1 PHYSICAL PARAMETER ESTIMATION FROM VIDEO
First, we assess the capabilities of Sim to accurately identify a variety of physical attributes such as mass, friction, and elasticity from image/video observations. To the best of our knowledge, Vim is the first study to jointly infer such fine-graine grained parameters from vidco observations. We also implement set of competitive baselines that use strictly more information on the task.
4.1.1 RIGID BODIES (RIGID)
Our first environment-rigg seiggid-evvllatte the accu- racy of estimating of physical and material attributes of rigid objects from videos. We curate dataset of 10000 simulated videos generated from varia- tions of 14 objects, comprising primitive shapes such as boxes, cones, cylinders, as well as non-convex shapes from ShapeNet (Chang al., 2015) and DexNet (Mahler et al., 2017). With uniformly sam- pled initial dimensions, poses, velocities, and phys-
Intollsation
Our Ervel and mmde
droune trats
ical properties (density, elasticity, and friction pa- rameters), we apply known impulse to the object and record a video of the resultant trajectory. Infer- ence with Vim is done by guessing an initial mass (uniformly random in the range [2, 12)kg/ 22]kg/m²), m, un- rolling a differentiable simulation using this guess, comparing the rendered out video with the (pixelwise mean-squared error MSE), and perform- ing gradient descent updates. We refer the interested reader to the appendix (Sec. G) for more details.
(81)
deformabe
rigil
rigid
Figure
Parameter Estimation: For
formabble experiments, we optimize the material properties of a beam to match a video of a beam hanging under gravity. In the rigic d experiments, we estimate contact parameters (clasticity/triitioon) and object density match video (GT). We visualize entire time sequences (1) with color-coded blends.
video
Table shows the results for predicting the mass of an object from video, with a known impulse applied it. We use EfficientNet (BO) (Tan & Le. 2019) and resize input frames to 64 x 64. Feature maps at resoluition of 4 x 4 x 32 are concatenated for all frames and fed to an MLP with 4 linear layers, and trained with an MSE loss. We compare Vim with three other baselines: PyBullet + REINFORCE (Ehsani et al., 2020; Wu et al., 2015), diff. physics only (requiring 3D supervision), and a ConvLSTM baseline adopted from Xu et al. (2019b) but with a stronger backbone. The DiffPhysics baseline is a strict subset of Sim, it only inolves the differentiable physics engine. However, it needs precise 3D states as supervision, which is the primary factor for its superior performance. Nevertheless, Sim is able to very precisely estimate mass from video, to absolute relative error of 9.01e-5, nearly two orders of magnitude better than the ConvLSTM baseline. Two other baselines also used: the "Average" baseline always predicts the dataset mean and the "Random" baseline
5

---

Published as a conference paper at ICLR 2021
Approuch
Mean abs.
Abs, Rel.
muss
clasticity
Friction
ere. (ke)
est.
Appronch
kg
ConvLSTM TTM Xo al (2019h)
PyBallen + REINFORCE Ehami et al. (2020)
Rundon
Aserge:
0.2222
0.1347
0.2663
0.0025
0.3665
0,0001
0.3364
0.1031
CTM Xu al. (2019h)
Average
Random
10,0007
I,7713
0.029
3.7145
0.14
4.18
2.5154
2.3410
0.14
4.1157
5.0241
0.17
0.4863
0.5558
0.096
DeffPhysiss (D sup.)
1.55e-9
5.17e-9
DiMPhysies (31) sup.)
1,70e-8
0.036
0.0020
0.007
0.0107
2.36: 5
9.0
VSin
2,57c-1
0.4
0.0026
0.0017
0.0073
Table 1: Mass estimation: Sim obtains precise
Table 2: Rigid-body parameter estimation: VSim esti- mates contact parameters (elasticity, friction) high degree of accuracy, despite estimating them from video. Diffphys, requires accurate 3D ground-truth at 30 FPS. We report absolute relative errors for each approach evaluated.
estimates, comparing favourably even with ap- proaches that require 3D supervision (diffphysics). We report the mean abolute error and absolute rela- tive errors for all approaches evaluated.
Deformable solid PEM
Thin Thillshell (cloth)
Per-particle mass
Material properties
Per-particle velocity
X
Approach
Rel. MAE
Rel. MAE
Rel. MAE
Rel. MAE 0.127
DiflPhysics (3D Sup.)
0.032
0.0025
0.0024
Sbn
0.048
0.0054
0.0056
0.026
Table Parameter estimation of deformable objects: estimate per-particle masses material properties (for solid def. objects) and per-particle velocities for cloth. In the case of cloth, there is a perceivable performance drop in diffphysics, as the center of mass of a cloth is often outside the body, which results in ambiguity.
predicts a random parameter value from the test distribution. All baselines and training details can be found in Sec. H of the appendix.
To investigate whether analytical differentiabilityy is required, our PyBullet + REINFORCE baseline ap- plies black box gradient estimation (Williams, 1992) through a non con-differentiable simulator (Coumans Bai, 2016-2019), similar to Ehsani et al. (2020). We find this baseline particularly sensitive to several simulation parameters, and thus worse- corse-performing In Table 2, we jointly estimate friction and clasticity parameters of our compliant contact model from video observations alone. The trend is similar to Table I, and V Sim is able to precisely recover the parameters of the simulation. A few examples can be seen in Fig. 3.
4.1.2 DEFORMABLE BODIES (DEFORMABLE)
We conduct series of experiments investigate the ability of Sim to recover physical parameters of deformable solids and thin-shell solids (cloth). Our physical model parameterized by the per- particle mass, and Lamé elasticity parameters, as described in in Appendix C.I. Fig. 3 illustrates the recovery of the elasticity parameters of beam hanging under gravity by matching the deformation given by input video sequence. We found our method is able accurately recover the parameters of 100 instances of deformable objects (cloth, balls, beams) as reported in Table 3 and Fig. 3.
4.1.3 SMOOTHNESS OF THE LOSS LANDSCAPE IN Sim
Since Sim is complex combination of differentiable non-linear components, analyze the loss landscape to verify the validity of gradients through the system. Fig. 4 illustrates the loss landscape when optimizing for the mass of rigid body when all other physical properties are known.
We examine the image-space mean- -quured error (MSE) of a mass cube (1 kg) for a range of initializations (0.1 kg to 5 kg). Notably, the loss landscape of V is well-behaved and conducive to momentum-based optimizers. Applying MSE to the first and last frames of the predicted and true videos provides the best gradients. However, for naive gradient estimator applied non- differentiable simulator (PyBullet + REINFORCE), multiple local minima exist resulting in very narrow region of convergence. This explains V Sim's superior performance in Tables 1, 2,
4.2 VISUOMOTOR CONTROL
To investigate whether the gradients computed by VSim are meaningful for vision-based tasks, we conduct range of visuomotor control experiments involving the actuation of deformable objects towards a visual target pose (a single image). In all cases, we evaluate against diffphysics, which uses goal specification and reward, both defined over the 3D state-space.

---

Published as a conference paper at ICLR 2021
Panm Peghiie SOE SAN eitim Abea 1A000
03
Scoeelund
Coundenut
KilE on fine and fraina
Eslimattdd masa
Estimaned materall poramittr valus
(a) Loss landscape (rigid)
(b) Loss landscape (deformable)
Figure 4: Loss landscapes when optimizing for physical attributes using Sim. (Left) When estimating the mass of rigid bodd with known shape using Vim, despite images being formed by a highly nonlinear process (simulation), the loss landscape is remarkably smooth, for a range of initialization errors. (Right) when optimizing for the elasicity parameters 1000, where the MSE loss has unique, dominant minimum. Note that, for fair comparison, the ground-truth for our TyBullet+REINFORCE baseline was generated using the PyBullet engine.
deformable FEM solid. Both
Lamé parameters and are set to
4.2. DEFORMABLE SOLIDS (CONIROL WALKER, CONTROL-FEN)
The first example (contro) rol -wa walker) involves a 2D walker model. Our goal is train neural net- work (NN) control policy to actuate the walker to reach a target pose on the right-hand side of image. Our NN consists of one fully connected layer and a tanh activation. The network input is a set of 8 time-varying sinusoidal signals, and the output is a scalar activation value per-tetrahedron. Vii able
Initaitaaion
Our final estimate
FGA IA
Ground
Eruih
to solve this environment within three iterations of gradient descent, by minimizing pixelwise MSE between the last frame of the rendered video and the goal image as shown in Fig. 5 (lower left).
control-callerr
control aem
cantrdiddath
t:1
tI
Figure 5: Visuomotor Control:
Vim provides gradients suitable for diverse, complex visuo- motor control tasks. For control- control-walle
In second test, formulate ing 3D control problem roontro--fed where the goal is actuate a soft-body FEM object (a gear) consisting of 1152 tetrahedral elements to move to a target position shown Fig. 5 (center). We use the same NN architecture as in the 2D walker example, and use the Adam (Kingma & Ba, 2015) optimizer to minimize pixelwise MSE loss. We also train a privileged baseline (diffphysics) that uses strong supervision and minimizes the MSE between the target position and the precise 3D location of the center-of-mass mass (COM) of the FEM model at each time step (i.c. dense reward). We test both diffphysics and Sim against a naive baseline that generates random activations and plot convergence behaviors in Fig. 6a.
challeng-
and experiments, we train neu- network to actuate soft body towards target image (GT). For cont -cl olohh we optimize the cloth's 's initial velocity to hit target (GT) (spec- ified an image), under nonlinear lift/drag forces.
While diffphysics appears to be a strong performer on this task, it is important to note that it uses explicit 3D supervision at each timestep (i.e. 30 FPS). In contrast, Sim single image as implicit target, and yet manages to achieve the goal state, albeit taking a longer number of iterations.
4.2.2 CLOTH (CONTROL-CLOTH)
We design an experiment to control a piece of cloth by optimizing the initial velocity such that reaches pre-specified target. In each episode, random cloth is spawned, comprising between 64 and 2048 triangles, and a new start/goal combination is chosen.
In this challenging setup, notice that state-based MPC (diffphsicss) is often unable accurately reach the target. We believe this is due to the underdetermined nature of the problem, since, for objects such cloth, the COM by itself does not uniquely determine the configuration of the object. Visuomotor control on the other hand, provides a more well-defined problem. An illustration of the task presented in Fig. (column 3), and the convergence of the methods shown in Fig. 6b.
7

---

Published as a conference paper at ICLR 2021
sigge mage superviion jyressm))
38-
Sing superteee (gradami
30 persicce 1aff phylica)
10 ruperv den prtpppy
Aandoniy actured model
fondumny adtates mode
woodta orror Opweentt
Optimbratio Iterations
Oytimezzionn Moraluss
(a) Results of various approaches the control-fem environment randomseeds; each randomseed corre- sponds to a different goal configuration). While diff- physics performs well, it assumes strong 3D supervision. In contrast, Vim is able to solve the task by using just single image of the target configuration.
(b) Results on control-clote environment (5 randomseeds; cach controls the dimensions and ini- tial/target poses of cloth). diffphysics converges to a suboptimal solution due to ambiguity in specify- ing the pose of a cloth via its center-of-mass. VSin solves the environment using single target image.
Figure 6: Convergence Analysis: Performance of Vin on visuomotor control using image-based supervision, 3D supervision, and random policies.
4.3 IMPACT OF IMPERFECT DYNAMICS AND RENDERING MODELS
Being a white box method, the performance of Vim relies on the choice of dynamics and rendering models employed. An immediate question that arises is "how would the performance of Sim be impacted (if at all) by such modeling choices." We conduct multiple experiments targeted at investigating modelling errors and summarize them in Table (left).
We choose a dataset comprising 90 objects equally representing rigid, deformable, and cloth types. By not modeling specific dynamics and rendering phenomena, we create the following 5 variants of our simulator.
1. Unmodeled friction: We model all collisions as being frictionless.
2. Unmodeled elasticity We model all collisions as perfectly elastic.
Rigid-as as -efl--a-dfforable: All rigid objects in the dataset are modeled deformable objects. Deformable- as as-riiid: All deformable objects in the dataset are modeled as rigid objects.
5. Photorealistic render: We employ photorealistic renderer-as opposed to VSim's 's differentiable rasterizers-inn generating the target images.
In all cases, we evaluate the accuracy with which the mass of the target object estimated from target video sequence devoid of modeling discrepancies. In general, we observe that imperfect dynamics models (i.e. unmodeled friction and elasticity, or modeling a rigid object as deformable or vice-versa) have more profound impact on parameter identification compared to imperfect renderers.
4.3.1 UNMODELED DYNAMICS PHENOMENON
From Table (left), we observe a noticeable performance drop when dynamics effects go unmodeled. Expectedly, the repureussions of incorrect object type modeling (Rigid-as- digiidaas-eeoommable Deformable- as-rigid) are more severe compared to unmodeled contact parameters (friction, elasticity). Modeling deformable body rigid body results in irrecoverable deformation parameters and has the most severe impact on the recovered parameter set.
4.3.2 UNMODELED RENDERING PHENOMENON
We also independently investigate the impact of unmodeled rendering effects (assuming perfect dynamics). We indepenently render ground-truth images and object foreground masks from a photorealistic renderer (Pharr al., 2016). We use these photorealistic renderings for ground-truth and perform physical parameter estimation from video. We notice that the performance obtained under this setting superior compared ones with dynamics model imperfections.
8

---

Published as a conference paper at ICLR 2021
Mean Kel. Als. lir.
Tetrahedra (8)
Forward (DF)
Purward (DR)
Bockward (DP)
Bockword (DP B3R)
Hnmodeled friction
(00)
9057 Hrr
1504 Hb
3721 Hx
s007 Hir
Unmodeled clasiiity
228
200
9257 Hz
3478 He
3780 H3x
2963 H2
Sags--s-defonmohl
0.34662
400
8753 Ht
3357 He
3750 H2
1360 H2
Photorvaisticc
Defoniahle-s-riiid
0.49974
5000
2000
1000
4174 Htz
357] Hz
3907 Htr
1660 Ht
1529 Hz
1554 Ho
1644 Ht
1553 E2x
1665 H0
1043 Ht
698 Htx
424 EI
Perfect model
0.1071
10000
3721 Hz
1500 Hz
1429 H2z
248 Hz
Table 4: (Left) Impact of imperfect models: The accuracy of physical parameters estimated by Viim is
impacted by the choice of dynamics and graphies (rendering) models. We find that the system is more sensitive the choice of dynamics models than the rendering engine used. (Right) Timing analysis: We report runtime simulation steps/ second (Hz). V Sim significantly faster than real-time, even for complex geometries.
4.3.3 IMPACT OF SHADING AND TEXTURE CUES
Although our work does not attempt to bridge the reality gap, we show early prototypes to assess phenomena such as shading/texture. Fig. 7 shows the accuracy over time for mass estimation from video. We evaluate three variants of the renderer "Only color", "Shading", and "Texture.. The "Only color" variant renders each mesh element in the same color regardless of the position and orientation of the light source. The "Shading" variant implements a Phong shading model and can model specular and diffuse reflections. The "Texture" variant also applies non-uniform texture sampled from ShapeNet (Chang et al., 2015). We notice that shading and texture cues significantly improve convergence speed. This is expected, as vertex colors often have very little appearance cues inside the object boundaries, leading to poor correspondences between the rendered and ground-truth images. Furthermore, textures seem to offer slight improvements in convergence speed over shaded models, as highlighted by the inset (log scale) plot in Fig. 7.
4.3.4 TIMING ANALYSIS
Table (right) shows simulation rates for the forward and backward passes of cach module. We report forward and backward pass rates sep- arately for the differentiable physics (DP) and the differentiable rendering (DR) modules. The
Shading
Teakre
emror i1 mose CrSITaston
1.1 Ieroor mraano wndirriitt
time complexity of Sim is funetion of the number of tetrahedrons and/or triangles. We illustrate the arguably more complex case of deformable object simulation for varying num- bers of tetrahedra (ranging from 100 to 10000). Even in the case of 10000 tetrahedra-aen enough to contruct complex mesh models of multiple moving objects V Sim enables faster-than-real. time simulation (1500 steps/second).
Deratows
SGD Banatoon
Figure 7: Including shading and texture cues lead faster convergence. Inset plot has logarithmic Y-axis.
CONCLUSION
We presented Sim, a versatile differentiable simulator that enables system identification from videos by differentiating through physical processes governing dyanmics and image formation. We demonstrated the benefits of such holistic approach by estimating physical attributes for time- evolving scenes with complex dynamics and deformations, all from raw video observations. We also demonstrated the applicability of this efficient and accurate estimation scheme end-to-end visuomotor control tasks. The latter case highlights VSim's 's efficient integration with PyTorch, facilitating interoperability with existing machine learning modules. Interesting avenues for futuree work include extending our differentiable simulation to contact-rich motion, articulated bodies and higher-fidelity physically-basd renderers doing so takes us closer to operating in the real-worid.
ACKNOWLEDGEMENTS
KM and LP thank the IVADO fundamental research project grant for funding. FG thanks CIFAR for project funding under the Catalyst program. FS and I.P acknowledge partial support from NSERC.

---

Published as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http:://tnsofffoooo org/. Software available from tensorflow.org. 4, 17
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning poke by poking: Experiential learning of intuitive physics. Neural Information Processing Systems, 2016.
Hassan Abu Alhaija, Siva Karthik Mustikovela, Andreas Geiger, and Carsten Rother. Geometric image synthesis. In Proceedings of Computer Vision and Pattern Recognition, 2018. 3
Martin Asenov, Michael Burke, Daniel Angelov, Todor Davchev, Kartic Subr, and Subramanian Ramamoorthy. Vid2Param: Modelling of dynamics parameters from video. IEEE Robotics and Automation Letters, 2019. 3
Peter W. Battaglia, Jessica Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding, Proceedings of the National Academy of Sciences, 1 110(45):18327-1833) 2013. ISSN 0027-8424, doi: 10. 10.10733puas.330652211000
Kiran S Bhat, Steven M Seitz, Jovan Popović, and Pradeep K Khosla. Computing the physical parameters of rigid-body motion from video. In Proceedings of the European Conference Computer Vision, 2002. 3
Kiran S Bhat, Christopher D Twigg, Jessica K Hodgins, Pradeep Khosla, Zoran Popovic, and Steven M Seitz. Estimating cloth simulation parameters from video. In ACM SIGGRAPPf/Eangrrpphhie Symposium on Computer Animation, 2003. 3
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, and Skye Wanderman- Milne, JAX: composable transformations of Python+NumPy programs, 2018. URL http:://gitttt
com/ http///ooeeeeeeeee jax. 4, 17
Robert Bridson, Sebastian Marino, and Ronald Fedkiw. Simulation of clothing with folds and wrinkles. In ACM SIGGRAPH 2005 Courses, 2005.
Marcus Brubaker, David Flect, and Aaron Hertzmann. Physics-based person tracking using the anthropomorphic walker. International Journal of Computer Vision, 17:140-155, 03 2010.
Mareus A Brubaker, Leonid Sigal, and David J Fleet. Estimating contact dynamics. In Proceedings of International Conference on Computer Vision, 2009. 3
Arunkumar Byravan and Dieter Fox. SE3-Nets: Learning rigid body motion using deep neural networks. IEEE International Conference on Robotics and Automation (ICRA), 2017.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 2015. 5,9
Michael Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum. A compositional object-based approach to learning physical dynamics. International Conference on Learning Representations, 2016. 1,3
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen-
tial equations. In Neural Information Processing Systems, 2018. 4, 17
Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation- based differentiable renderer. Neural Information Processing Systems, 2019. 3, 4, 19
10

---

Published as a conference paper at ICLR 2021
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, llya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. Neural Information Processing Systems, 2016. 3,3
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. In International Conference on Learning Representutions, 2020. 3
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of Computer Vision and Pattern Recognition, 2019. 3
Erwin Coumans and Yunfei Bai. PyBullet, python module for physies simulation for games, robotics and machine learning.
2016-2015 6, 22
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation- based inference. In National Academy of Sciences (NAS), 2020a.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR Workshops, 2020b. 1,3, 20
Filipe de Avila Belbute-Peree Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J. Zico Kolter. End-to-end differentiable physics for learning and control. In Neural Information Processing Systems, 2018. 2, 4, 19, 20, 25
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis Wyffels. A differentiable physics engine for deep learning in robotics. Neural Information Processing Systems, 2016. 1,3, 4, 20
Sébastien Ehrhardt, Aron Monszpart, Niloy J. Mitra, and Andrea Vedaldi. Learning physical long-term predictor. arXiv, 2017.
Sébastien Ehrhardt, Aron Monszpart, Niloy J. Mitra, and Andrea Vedaldi. Unsupervised intuitive physics from visual observations. Asian Conference on Computer Vision, 2018. 3
Kiana Ehsani, Shubham Tulsiani, Saurabh Gupta, Ali Farhadi, and Abhinav Gupta. Use the Force, Luke! learning predict physical forces by simulating effects. In Proceedings of Computer Vision and Pattern Recognition, 2020. 3, 5, 6, 23
Tom Erez, Yuval Tassa, and Emanuel Todorov. Simulation tools for model-based robotics: Compari- son of Bullet, Havok, MuJoCo, ODE, and PhysX. In JEEE International Conference on Roboties and Automation (ICRA), 2015. 17, 18, 19
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 2018. 3
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. In International Conference Learning Representations, 2016. 3
Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Neural Information Processing Systems, 2019. 1,3, 20
Andreas Griewank and Andrea Walther. Introduction to automatic differentiation. PAMM, 2(1): 45-49, 2003.
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. Atlasnet: A papier-måché approach to learning 3d surface generation. In Proceedings of Computer Vision and Pattern Recognition, 2018. 3
Radek Grzeszezuk, Demetri Terzopoulos, and Geoffrey Hinton. Neuroanimator: Fast neural network emulation and control of physics- based models. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 1998. 3

---

Published as a conference paper at ICLR 2021
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of Computer Vision and Pattern Recognition, 2020.
Abhinav Gupta, Alexei A. Efros, and Martial Hebert. Blocks world revisited: Image understanding using qualitative geometry and mechanics. In Proceedings of the European Conference Computer Vision, 2010. 3
Ernst Hairer, Christian Lubich, and Gerhard Wanner. Geometric numerical integration: structure- preserving algorithms for ordinary difserential equations, volume 31. Springer Science & Business Media, 2006. 18
Eric Heiden, David Millard, Hejia Zhang, and Gaurav S. Sukhatme. Interactive differentiable simulation. arXiv, 2019.
Philipp Henzler, Niloy J. Mitra, and Tobias Ritschel. Escaping plato's cave using adversarial training: 3d shape from unstructured 2d image collections. In Proceedings of International Conference on Computer Vision, 2019. 3
Yuanming Hu, Yu Fang. Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Gruphics, 37(4), 2018.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B. Tenenbaum, William T. Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simulator for soft robotics. In IEEE International Conference on Robotics and Automation (ICRA), 2019. 3,
17
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelleg, Kelley, and Frédo Durand. DiffTaichi: Differentiable programming for physical simulation. International Conference on Learning Representations, 2020. 1, 3,4, 17, 19, 20
Carlo Innamorati, Bryan Russell, Danny Kaufman, and Niloy Mitra. Neural re-simulation for generating bounces in single images. In Proceedings of International Conference on Computer Vision, 2019. 3
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. International Conference Learning Representations, 2019. 1,
Miguel Jaques, Michael Burke, and Timothy M. Hospedales. Physiie-as-inverse--rapiii Joint unsupervised learning of objects and physics from video. International Conference on Learning Representations, 2020. 3
Krishna Murthy Jatavallabhula, Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. Kaolin: A pytorch library for accelerating 3d deep learning research. In arXiv, 2019.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of Computer Vision and Pattern Recognition, 2018. 3.4
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. 24
David Kirk et al. Nvidia cuda software and gpu parallel computing architecture. ISMM, volume 7, pp. 103-104, 2007. 17
Krzysztof Kozlowski. Modelling and Identification in Robotics. Advances in Industrial Control. Springer, London, 1998. ISBN 111114444-11199) 3
T. D. Kulkarni, P. Kohli, J. B. Tenenbaum, and V. Mansinghka. Picture: A probabilistic programming language for scene perception. In Proceedings of Computer Vision and Pattern Recognition, 2015.
12

---

Published as a conference paper at ICLR 2021
Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. SIGGRAPH Asia, 37(6:22211-22::11, 2018. 3
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. International Conference on Learning Representations, 2019.
Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel L. K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International Conference on Machine Learning. 2020.
Junbang Liang, Ming Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. In Neural Information Processing Systems, 2019. 1,3
Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proceedings of Computer Vision and Pattern Recognition, 2019. 3
C Karen Liu, Aaron Hertzmann, and Zoran Popović. Learning physics- motion style with nonlinear inverse optimization. ACM Transactions on Graphics (TOG), 140)) 2005.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. Proceedings of International Conference on Computer Vision, 2019. 4, 19
Matthew M. Loper and Michael J. Black. Opendr: An approximate differentiable renderer. In Proceedings of the European Conference on Computer Vision, 2014. 3,4
Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae- Yong Kim. Unified particle physics for real-time applications. ACM Transactions on Graphics (TOG), 33(4):1-12, 2014. 2
Dougal Maclaurin, David Duvenaud, Matt Johnson, and Jamie Townsend. Autograd, 2015. URL
https: //tp::///ghhhtt com HHIS autograd. 20
Jeffrey Mahler, Jacky Liang. Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg. Dex-net 2.0: Deep learning plan robust grasps with synthetic point clouds and analytic grasp metrics. In Robotics Science and Systems, 2017.
Richard Mann, Allan Jepson, and Jeffrey Mark Siskind. The computational perception of scene dynamics. Computer Vision and Image Understanding, 65(2):1 13 128, 1997.
Charles C Margossian, A review of automatic differentiation and its efficient implementation. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(4)ee1305 2019. 20
Viraj Mehta, Ian Char, Willie Neiswanger, Youngseog Chung, Andrew Oakleigh Nelson, Mark D Boyer, Egemen Kolemen, and Jeff Schneider. Neural dynamical systems: Balancing structure and flexibility in physical prediction. ICLR Workshops, 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. Proceedings of Computer Vision and Pattern Recognition, 2019.
Mateusz Michalkiewiez, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In Proceedings of International Conference on Computer Vision, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019. 3
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference Computer Vision, 2020. 3
13

---

Published as a conference paper at ICLR 2021
Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. Newtonian image understanding: Unfolding the dynamics of objects in static images. Proceedings of Computer Vision and Pattern Recognition, 2015.
Roozbeh Mottaghí, Mohammad Rastegari, Abhinav Gupta, and Ali Farhadi. "what happens if..." learning predict the effect of forces in images. In Proceedings of the European Conference Computer Vision, 2016. 3
DJJ. Murray-Smith, The inverse simulation approach: focused review of methods and applications. Mathematics and Computers in Simulation, 53(4):239 247, 2000. ISSN 0378-4754.
Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yong-Liang Yang. Rendernet: A deep con- volutional network for differentiable rendering from 3d shapes. Newral Information Processing Systems, 2018.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of Computer Vision and Pattern Recognition, 2020. 3
Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. Transactions on Graphics (Proceedings of SIGGRAPH Asia), 38(6), 2019. 3
Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of Computer Vision and Pattern Recognition, 2019. 3
Despoina Paschalidou, Ali Osman Ulusoy, Carolin Schmitt, Luc van Gool, and Andreas Geiger. Raynet: Learning volumetric 3d reconstruction with ray potentials. In Proceedings of Computer
Vision and Pattern Recognition, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- -egh-perforancce deep learning library. In Neural Information Processing Systems, 2019. 4, 17
Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physicully Based Rendering: From Theory to Implementaation. Morgan Kaufmann Publishers Inc., 2016. ISBN 0128006455. 2,
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Scalable differentiable physics for learning and control. International Conference Machine Learning. 2020. 1,
Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization via probabiliisiic inference for robotics simulators. Robotics Science and Systems, 2019.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501, 2020. 3
Google Research. Tiny Differentiable Simulator, 2020 (accessed May 15, 2020). URL https: //ggitubb... com/ maapoggeereeseercttiiiooe
fferentiail s simulator. 3,4
Danilo Jimenez Rezende, SM Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In Neural Information Processing Systems, 2016. 23
Mathieu Salzmann and Raquel Urtasun. Physically-basee motion models for 3d tracking: convex formulation. In Proceedings of International Conference Computer Vision, 2011. 3
Alvaro Sanchez-Gonee Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. In arXiv, 2019. 3, 20
14

---

Published as a conference paper at ICLR 2021
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks, In International Conference on Robot Learning, 2018. 3
Eftyhhoo Sifakis and Jernej Barbic. Fem simulation of 3d deformable solids: a practitioner's guide to theory, discretization and model reduction. In ACM SIGGRAPH 2012 courses, 2012. 17
Breannan Smith, Fernando De Goes, and Theodore Kim. Stable neo-hookean flesh simulation. ACM Transactions on Graphics, 37(2):1-155 2018a. 17, 19
Edward Smith, Scott Fujimoto, and David Meger. Multi-view silhouette and depth decomposition for high resolution 3d object representation. In Neural Information Processing Systems, 201 8b.
Edward J. Smith, Scott Fujimoto, Adriana Romero, and David Meger. Geometries: Exploiting geometric structure for graph-encoded objects. International Conference on Machine Learning. 2020. 3
Changkyu Song and Abdeslam Boularias. Identifying mechanical models through differentiable simulations. In Learning for Dynamical Systems and Control (L4DC), 2020a. 3,4
Changkyu Song and Abdeslam Boularias. Learning to slide unknown objects with differentiable physics simulations. In Robotics Science and Systems, 2020b. 3,
Jos Stam. Stable fluids. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 121-128, 1999. 18, 20
Trevor Standley, Ozan Sener, Dawn Chen, and Silvio Savarese. image2mass: Estimating the mass of an object from its image. In International Conference on Robot Learning, 2017. 3
Giovanni Sutanto, Austin S. Wang, Yixin Lin, Mustafa Mukadam, Gaurav S. Sukhatme, Akshara Rai, and Franziska Meier. Encoding physical constraints in differentiable newton-euler algorithm. In Learning for Dynamical systems and Control (L4DC), 2020.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, 2019. 5, 24
Emanuel Todorov. Convex and maaytically-invertin dynamics with contacts and constraints: Theory and implementation in mujoco. In IEEE International Conference on Robotics and Automation (ICRA), 2014. 18
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Representations, 2020. 3, 20
Mare Toussaint, Kelsey Allen, Kevin Smith, and Joshua Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Robotics Science and Systems, 2018. 2,
Bart van Merričnboer, Alexander B Wiltschko, and Dan Moldovan. Tangent: automatic differentiation using source code transformation in python. In Neural Information Processing Systems, 2018.
17
Bin Wang, Paul G. Kry, Yuanmin Deng, Uri M. Ascher, Hui Huang, and Baoquan Chen. Neural material: Learning clastic constitutive material and damping models from sparse data. arXiv, 2018a. 3
Kun Wang, Mridul Aanjaneya, and Kostas Bekris. A first principles approach for data-efficient system identification of spring-rod systems via differentiable physics engines. In arXiv, 2020.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu- Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. Proceedings of the European Conference on Computer Vision, 2018b. 3
Nicholas Watters, Daniel Zoran, Theophane Weber. Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. In Neural Information Processing Systems, 2017. 1,3
15

---

Published as a conference paper at ICLR 2021
P. M. Wensing, S. Kim, and J. E. Slotine. Linear matrix inequalities for physically consistent inertial parameter identification: A statistical perspective on the mass distribution. IEEE Robotics and Automation Letters, 3(1):60-67, 2018.
Ronald Williams. Simple statistical gradient-foloowing algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256, 1992. ISSN 0885-6125. 2, 6, 22
Jiajun Wu, IIker Yildirim, Joseph J Lim, William T Freeman, and Joshua B Tenenbaum. Galileo: Perceiving physical object properties by integrating a physies engine with deep learning. In Neural Information Processing Systems, 2015. 5, 23
Jiajun Wu, Joseph Lim, Hongyi Zhang, Joshua B Tenenbaum, and William T Freeman. Physics 101: Learning physical object properties from unlabeled videos. In British Machine Vision Conference, 2016. 3
Jiajun Wu, Erika Lu, Pushmeet Kohli, William T Freeman, and Joshua B Tenenbaum. Learning to see physics via visual de-animation. In Neural Information Processing Systems, 2017a. 3,
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of Computer Vision and Puttern Recognition, 2017b.
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In Neural Information Processing Systems, 2019a. 3
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B. Tenenbaum, and Shuran Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. In Robotics Science and Systems, 2019b.
1, 3. 5,
Ilker Yildirim, Tejas Kulkarni, Winrich Freiwald, and Joshua Tenenbaum. Efficient analysis-by- synthesis in vision: computational framework, behavioral tests, and comparison with neural representations. In CogSci, 2015. 2
llker Yildirim, Michael Janner, Mario Belledonne, Christian Wallraven, W. A. Freiwald, and Joshua B. Tenenbaum. Causal and compositional generative models in online perception. In CogSci, 2017.
Ilker Yildirim, Mario Belledonne, Winrich Freiwald, and Josh Tenenbaum. Efficient inverse graphics in biological face processing. Science Advances, 6(10), 2020. doi: 10.1 1...11///ciadv.aaxx5599 2
L. Yu, Duncan, and Yeung. Fill and transfer: A simple physics-based approach for containability reasoning. In Proceedings of International Conference Computer Vision, 2015,
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference on Learning Representations, 2020. 3
Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018. 3
16

---

Published as a conference paper at ICLR 2021
A APPENDIX
A DIFFERENTIABLE PHYSICS ENGINE
Under Lagrangian mechanics, the state of a physical system can be described in terms of generalized coordinates q, generalized velocities q = u, and design, or model parameters 9. For the purposes of exposition, we make no distinction between rigid-bodies, deformable solids, or thin-shell models of cloth and other bodies. Although the specific choices of coordinates and parameters vary, the simulation procedure is virtually unchanged. We denote the combined state vector by s(t) (q(t), u.t)].
The dynamic evolution of the system is governed by second order differential equations (ODE) of the form M = f(s), where M is a mass matrix that may also depend on our state and design parameters 8. Solutions to ODEs of this type may be obtained through black box numerical integration methods, and their derivatives calculated through the continuous adjoint method Chen et al. (201 8). However, we instead consider our physics engine implicit relationship between a state vector s = s(t) at the start of a time step. and the updated state at the end of the time step s+ s(t + At). An arbitrary discrete time integration scheme can be then be abstracted as the function g(s ss 0) = O, relating the initial and final system state and the model parameters 8. By the implicit function theorem, if we can specify loss function I the output of the simulator, we can compute öe at T and likewise for the model parameters 0.
differentiable operation that provides an
01 c _x, where c the solution the linear system ös
0g T 0s c = as
While the partial derivatives 0g og dg can be computed by graph-based automatic differentation Ue ag 3 frameworks Paszke et al. (2019); Abadi et al. (2015); Bradbury et al. (2018), program transformation approaches such as DiffTaichi, and Google Tangent Hu et al. (2020); van Merriënboer et al. (2018) are particularly well-suited to simulation code. We use an embedded subset of Python syntax. which computes the adjoint of each simulation kernel at runtime, and generates C++/CUDA Kirk et al. (2007) code. Kernels are wrapped as custom autograd operations on PyTorch tensors, which allows users to focus on the definition of physical models, and leverage the Py PyTorch tape-based autodiff to track the overall program flow. While this formulation is general enough to represent explicit, multi-step, or fully implicit time-integration schemes, we employ semi-implicit Euler integration, which is the preferred integration scheme for most simulators Erez et al. (2015).
A.1 PHYSICAL MODELS
We now discuss some of the physical models available in VSim.
Deformable Solids: In contrast with existing simulators that use grid-based methods for differentiable soft-body simulation Hu et al. (2019; 2020), we adopt finite element (FEM) model with constant strain tetrahedral elements common in computer graphics Sifakis & Barbic (2012). We use the stable Neo-Hookean constitutive model of Smith et al. Smith al. (2018a) that derives per-element forces from the following strain energy density:
X 1" (Ic - 3) -3)+ (J aa²
1 loog[Tc + 1),
(g 0)
(1)
2
2
where Tc, are invariants of strain, A = Ju, >] are the Lamé parameters, and a is a per-element actuation value that allows the element expand and contract.
Numerically integrating the energy density over each tetrahedral mesh element with volume V,, gives the total elastic potential energy, U(q 0) Σ V. V, The forces due to this potential fs 0) -. Ug. 0), can computed analytically, and their gradients obtained using the adjoint method (cf. Section 3.
Deformable Thin-Shells: To model thin-shells such as clothing. we use constant strain triangular elements embedded in 3D. The Neo-Hookean constitutive model above is applied to model in in-plane elastic deformation, with the addition of a bending energy fs 9) kysin( _ + a)d, where k is the bending stiffness, ò is the dihedral angle between two triangular faces, a is a per-edge actuation value that allows the mesh to flex inwards or outwards, and dl is the force direction given by Bridson
17

---

Published as a conference paper at ICLR 2021
al. (2005). We also include lift/drag model that approximates the effect of the surrounding air the surface mesh.
Rigid Bodies: We represent the state of 3D rigid body q [xx r| consisting of a position
x E IR', and quaternion r IR+, The generalized velocity of a body is u, = [v,w] and the dynamics of each body is given by the Newton-Fuler equations,
f
m
w
x Iw
where the mass and inertia matrix parameters
I (expressed at the center of mass) considered design
Contact: We adopt a compliant contact model that associates elastic and damping forces with each nodal contact point. The model is parameterized by four scalars 0 = [Ke, ka, ky, 14], corresponding elastic stiffness, damping, frictional stiffness, and friction coefficient respectively. To prevent interpenetration we a proportional penalty-based force, f, (s. is a contact normal, and C' is a gap function measure of overlap projected on to Rt. We model friction using relaxed Coulomb model Todorov (2014) (s, 0) -D min [,,,,, k uyu))] where D is basis of the contact plane, and u, D u is the sliding velocity at the contact point. While these forces are only C continuous, we found that this was sufficient for optimization over variety of objectives.
-n k: ((q) kaC(,)), where
More physical simulations: We also implement a number of other differentiable simulations such as pendula, mass-springs, and incompressible fluids Stam (1999). We note these systems have already been demonstrated in prior art, and thus focus on the more challenging systems in our paper.
B DISCRETE ADJOINT METHOD
Above, presented a formulation of time-intee inte-integration using the discrete adjoint method that repre- sents an arbitrary time-stepping scheme through the implicit relation,
g(s st 0) = 0.
This formulation is general enough to represent both explicit or implicit time-stepping methods. While explicit methods are often simple to implement, they may require extremely small time-steps for stability, which is problematic for reverse-mode mode automatic differentiation frameworks that must explicitly store the input state for each discrete timestep invocation of the integration routine. On the other hand, implicit methods can introduce computational overhead or unwanted numerical dissipation Hairer et al. (2006). For this reason, many real-time physics engines employ a semi- implicit (symplectic) Euler integration scheme Erez et al. (2015), due to its case of implementation and numerical stability in most meaningful scenarios (conserves energy for systems where the Hamiltonian is time-invariant)
We now give a concrete example of the discrete adjoint method applied to semi-implicit Euler. For the state variables defined above, the integration step may be written follows,
g(s (t = Iut u - -AM- 1IM-If(s-) 0.
(4)
q - - Atu
Note that in general, the mass matrix M is function of and 0. For conciseness we only consider the dependence on f, although the overall procedure unchanged in the general case. We provide a brief sketch of computing the gradients of g(s .s :)). In the case of semi-implicit integration above, these are given by the following equations:
A-M-1 -1 OF 0g Ōs
3q
A&M 1
Julf)
0g
0g 00
A/
8M
-I
0st
I
AAII
(5)
18

---

Published as a conference paper at ICLR 2021
In the case of semi-implicit Euler, the triangular structure of these Jacobians allows the adjoint variables to be computed explicitly. For fully implicit methods such as backwards Euler, the Jacobians may create a linear system that must be first solved to generate adjoint variables.
C PHYSICAL MODELS
We now undertake a more detailed discussion of the physical models implemented in Vim.
(a) Triangular FEM element
(b) Tetrahedral FEM element
Figure 8: Mesh Discretization: We use triangular (a) and tetrahedral (b) FEM models with angle-based and volumetric activation parameters, a. These mesh-based discretizations natural fit for our differentiable rasterization pipeline, which is designed to operate on triangles.
C.I FINITE ELEMENT METHOD
As described in section 3.2 ("Physical models"), hyperelastic constitutive model based the neo-Hookean model of Smith et al. Smith et al. (2018a):
(g 0) = - (lc - 3) 2 a)²
# log(Ic + 1).
(6)
The Lamé parameters, A, ns control the element's resistance to shearing and volumetric strains. These may be specified on a per-element basis, allowing us represent heterogeneous materials. In contrast to other work using particle-base models Hu et al. (2020), we adopt a mesh-based discretization for deformable shells and solids. For thin-shells, such as cloth, the surface is represented by triangle mesh as in Figure 8a, enabling straightforward integration with our triangle mesh-based differentiable rasterizer Liu al. (2019); Chen et al. (2019). For solids, we use a tetrahedral FEM model as illustrated in Figure 8b. Both these models include per-element activation parameter which for thin-shells, allows us to control the relative dihedral angle between two connected faces. For tetrahedral meshes, this enables changing the element's volume, enabling locomotion, as in the control fem example.
C.2 CONTACT
Implicit contact methods based on linear complementarity formulations (LCP) of contact may be used to maintain hard non-penetration constraints de Avila Belbute-Pern al. (2018). However, we found relaxed models of contact-uue used in typical physics engines Erez et al. (2015) were sufficient for our experiments. this approach, contact forces are derived from a one-sided quadratic potential, giving rise to penalty forces of the form 9a. While Coulomb friction may also be modeled an LCP, we use a relaxed model where the stick regime represented by a stiff quadratic potential around the origin, and a linear portion in the slip regime, shown in Figure 9b. To generate contacts, test each vertex of mesh against a collision plane and introduce a contact within some distance threshold d.
19

---

Published as a conference paper at ICLR 2021
0.5
0.5
Force
Force
Exact Relaxed
Coutomb Ralaxed
-1
-0.5
-0.5
0.5
Penetration Depth
Slip Velocity
(a)
(b)
To model non-interpenetration constraints relaxed model of contact that replaces delta function with a linear hinge corresponding to quadratic penalty energy (a). model friction we use relaxed Coulomb model, that replaces the step function with symmetric hinge (b).
Figure 9: Contact Model:
C.3 PENDULA
We also implement simple and double pendula, toy examples of well-behaved and chaotic systems respectively, and estimate the parameters of the system (i.e., the length(s) of the rod(s) and initial angular displacement(s)),
comparing the rendered videos (assuming uniformly random initial guesses) with the true videos. As pendula have extensively been studied in the context of differentiable physies simulation Degrave et al. (2016); de Avila Belbute-Peree Toth et al. (2020); Greydanus et al. (2019); Sanchez Sanchez-Gonzalez Gonzalez et al. (2019), we focus on more challenging systems which have not been studied in prior art.
al. (2018); Crammer et al. (2020b);
C.4 INCOMPRESSIBLE FLUIDS
As an example of incompressible fluid simulation, we implement a smoke simulator following the popular Lagranggnn advection scheme of Stam al. Stam (1999). At 2:20 in our supplementary video attachment, we show an experiment which optimizes the initial velocities of smoke parti- cles to form desired pattern. Similar schemes have already been realized differentiably, e.g. in DiffTaichi Hu et al. (2020) and autograd Maclaurin et al. (2015).
D SOURCE-CODE TRANSFORMATION FOR AUTOMATIC DIFFERENTIATION
The discrete adjoint method requires computing gradients of physical quantities with respect state and design parameters. To do so, we adopt a source code transformation approach to perform reverse mode automatic differentiation Hu et al. (2020); Margossian (2019). We a domain-specinf subset of the Python syntax extended with primitves for representing vectors, matrices, and quaternions. Each type includes functions for acting them, and the corresponding adjoint method. An example simulation kernel is then defined as follows:
3keznel
def integrate_parrtee particlest
tensor (floct31,
tensor (f 31
= Lensoz (Eloat31,
tenser (flost),
gravity Censoz (Cloat3),
: float,
x_new sensortfloatt
10
v_new tensor (f
11
12 thread 10
13
14
chread_id = tid()
16 soad stcte variables and parameters
load Ix, thread_id!
20

---

Published as a conference paper at ICLR 2021
It, thread_id
19
F0 load if, thread_id]
20
inv_mass load (w, thread_id}
22
_cad external forces (graviryyy
24
Semi-implicit
26
v1 = vo (f0 inv_mass (top dt
27
x1 x3 vI
28
29
Store results
(x_new, thread_1d, x1) store (V_new, thread_id, v1)
31
Listing I: Particle Integration Kernel
At runtime, the kernel's abstract syntax tree (AST) is parsed using Python's built-in ast module. We then generate C++ kernel code for forward and reverse mode, which may be compiled to a CPU or GPU executable using the PyTorch torch utill cpp_extens
mechanism.
This approach allows writing imperative code, with fine-grained indexing and implicit operator fusion (since all operations in a kernel execute as one GPU kernel launch). Each kernel is wrapped as PyTorch autograd operation that it fits natively into the larger computational graph.
E
MPC CONTROLLER ARCHITECTURE
For model predictive control examples, illustrated in Figure 10. With simulation time t as input we generate N phase-shifted sinusoidal signals which passed fully-connected layer (zero-bias), and final activation layer. The output a vector of per-element activation values as described in the previous section.
simple 3-layer neural network architecture
Control
a
7.5m
SinCos
Dense
Tanh
Figure 10: Our simple network architecture used the for control -wal ker and control - -fem tasks.
F LOSS LANDSCAPES FOR PARAMETER ESTIMATION OF DEFORMABLE SOLIDS
Sim integrates several functional blocks, many of which contain nonlinear operations. Furthermore, we employ a pixelwise mean-squared error (MSE) loss function for estimating physical parameters from video. To demonstrate whether the gradients obtained from Vim relevant for the task of physical parameter estimation, in Figure 2 of the main paper, present an analysis of the MSE loss landscape for mass estimation.
ELASTICITY PARAMETER
We now present similar analysis for elasticity parameter estimation in deformable solids. Figure 1a shows the loss landscape when optimizing for the Lamé parameters of a deformable solid FEM. In this case, both parameters À and y are 1000. As can be seen in the plot, the loss landscape has
21

---

Published as a conference paper at ICLR 2021
unique, dominant minimum at 1000. We believe the well-behaved nature of our loss landscape is key contributing factor to the precise physical -arameetrr estimation ability of Vim.
F.2 Loss LANDSCAPE IN PYBULLET (REINFORCE)
Figure 11 shows how optimization using REINFORCE can introduce complications. As the simu- lation becomes unstable with masses close zero, poor local optimum arise the mean of the current estimated mass. This illustrates that optimization through REINFORCE only possible after careful tuning step size, sampling noise and sampling range. This reduces the utility of this method in a realistic setting where these hyperparameters are not known a priori.
Semund trah wilan
Pootee MMSE between eccoranntto +00e
Ove inch mat
Estinated materia porsiettr vaMe
Extimated mass
(a) Lamé loss landscape
(b) PyBullet loss landscape
Figure 11: Loss Landscapes: (left) when optimizing for the clasicity parameters a deformable FEM solid. Both the Lamé parameters X and are set to 1000, where the MSE loss has unique, dominant minimum. (right) when optimizing for the mass, the reward (negative normalized MSE) has maximum close to the ground truth maximum but the negative log likelihond of each sample that's 's multiplied with the reward only shows a local minimum that's sensitive to the center of current mass estimate.
F.3 IMPACT OF THE LENGTH OF VIDEO SEQUENCE
To assess the impact of the length of a video on the quality of our solution, we plot the loss landscapes for videos of varying lengths in Fig. 12. We find that shorter videos tend to have steeper loss landscapes compared to longer ones. The frame-rate also has an impact the steepness of the landscape. In all cases though, the loss landscape smooth and has the same unique minimum.
G DATASET DETAILS
For the rigid-body task of physical parameter estimation from video, we curated a dataset comprising of 14 meshes, as shown in Fig. 13. The objects include a combination of primitive shapes, fruits and vegetables, animals, office objects, and airplanes. For each experiment, we select object random, and sample its physical attributes from predefined range: densities from the range [2, 12) kg/ /m contact parameters ke, ka, ky from the range [1, 500), and coefficient of friction from the range [0.2, 1.0J. The positions, orientations, (anisotropic) scale factors, and initial velocities are sampled uniformly random from cube of side-length 13m centered on the camera. Across all rigid-body experiments, we use 800 objects for training and 200 objects for testing.
H BASELINES
In this section, we present implementation details of the baselines used in our experiments.
H.1 PYBULLET + REINFORCE
To explore whether existing non-differentiable simulators be employed for physical parameter estimation, we take PyBullet Coumans & Bai (2016-2013 - a popular physics engine- and make it trivially differentiable, by gradient estimation. We employ the REINFORCE Williams (1992)
22

---

Published as a conference paper at ICLR 2021
Pixelwise MSE between estimate/t fr/errueett video
0.4
Ground-truth
MSE on first and last frames
on every frame
0.3
MSE on every 5th frame
MSE on first and middle frames
0.0
2.5
Estimated mass
Figure 12: Impact the length video sequence on the loss landscape. Notice how loss landscape much steeper for smaller videos (e.g., MSF of first and last frames). Nonetheless, all cases have a smooth loss landscape with the same unique minimum.
Figure 13: Objects used our rigid-body experiments. these meshes have been simplified to contain 250 or fewer vertices, for faster collision detection times.
technique to acquire approximate gradient through the otherwise non-differentiable environment. The implementation was inspired by Wu et al. (2015) and Rezende et al. (2016). In concurrent work, similar idea was explored Ehsani al. (2020).
In PyBullet, the mass parameter of the object is randomly initialized in the range [0, Nu). where the same starting position and orientation in
N,, the number of vertices, the object the dataset, and the camera parameters are identical to those used in the dataset. This configuration ensures that if the mass was correct, the video frames rendered out by PyBullet would perfectly align with those generated by Vim.. Each episode is rolled out for the same duration in the dataset (60 frames, corresponding
2 seconds of motion). In PyBullet this is achieved by running the simulation at 240 Hz and skipping 7 frames between observations. The REINFORCE reward is calculated by summing the individual L2 losses between ground truth frames and PyBullet frames, then multiplying each by 1 to establish global maximum at the correct mass, in contrast with global minimum as in Sim. When all individual frame rewards have been calculated, all trajectory rewards normalized before calculating the loss. This ensures that the reward is scaled correctly with respect to REINFORCE's negative sample log likelihood, but when the mass value approaches the local optimum, this leads instability in the optimization process. To mitigate this instability,
23

---

Published as a conference paper at ICLR 2021
introduce reward decay, which hyperparameter that slowly decreases the reward values as optimization progresses, a similar manner to learning rate decay. Before each optimization step, all normalized frame reward values multiplied by reward decay. After the optimization step, the decay is updated by reward decay = reward_decap * decay_faclon The hyperparameters used in this baseline be found in Table 5.
Parameter
Value
Meaning
samples
5
How often was the mass sampled at every step
optimization_ste _tps
125
Total number of optimization steps
sample_nof
0.05
Std. dev. of normal distribution that mass is sampled from
decay_factor
0.925
Factor that reward decay is multiplied with after optimizer step
dataset_size size
200
Number of bodies that the method was evaluated
Table 5: RyBullez-REINFORCE hyperparameters.
H.2 CNN FOR DIRECT PARAMETER ESTIMATION
In the rigid-body parameter estimation experiments, we train a ConvNet baseline, building on the EfficientNet-Bl architecture Tan & Le (2019). The ConvNet consists of convolutional layers with parameters (Py (PyTorch convention): (1280, 128, 1). (128, 32, 1). followed by linear layers and ReLU activations with sizes [7680, 1024, 100, 100, 100, 5]. No activation applied over the output of the ConvNet. We train the model to minimize the rean-e error between the estimated and the true parameters, and use the Adam optimizer Kingma Ba (2015) with learning rate of 0.0001. Each model was trained for 100 epochs on V100 GPU. The input image frames were preprocessed by resizing them to 64 64 pixels (to reduce GPU memory consumption) and the features were extracted with pretrained EfficientNet-BO..
I COMPUTE AND TIMING DETAILS
Most of the models presented in Sim can be trained and evaluated on modern laptops equipped with graphics processing units (GPUs). We find that, on a laptop with an Intel i7 processor and GeForce GTX 1060 GPU, parameter estimation experiments for rigid/nonrigid bodies can run in under 5-20 minutes object on CPU and in under minute on the GPU. The visuomotor control experiments (control- fem, control. wwwrroolllllttt cloth) take about 30 minutes per episode on the CPU and under 5 minutes per episode the GPU.
OVERVIEW OF AVAILABLE DIFFERENTIABLE SIMULATIONS
Table 6 presents an overview of the differentiable simulations implemented in Vim, and the optimiz- able parameters therein.
pos
vel
mass
rot
rest
stiff
damp
actuation
ext forces
Rigid body
Simple pendulum
I
Double pendulum
V
Deformable object
Y
Cloth
Fluid (Smoke) (2D)
Table 6: An overview optimizable parameters in Vim. Table columns (in order, from left right): Initial particle positions (pos), Initial particle velocities (vel), Per-particle mass (mass), Initial object orientation (rot), Spring rest lengths (rest), Spring stiffnesses (stiff), Spring damping coefficients (damp), Actuation parameters (actuation), Gravity (g), Friction parameters μ4, Elasticity parameters (e), External force parameters (ext forces).
24

---

Published as a conference paper at ICLR 2021
K LIMITATIONS
While providing a wide range of previously inaccessible capabilities, Vim has a few limitations that we discuss in this section. These shortcomings also form interesting avenues for subsequent research.
Sim (and equivalently VPyBullet) are inept at handling tiny masses (100g and less). Optimizing for physical parameters for such objects requires a closer look at the design of physics engine and possibly, numerical stability.
Articulated bodies are not currently implemented in Sim. Typically, articulated bodies are composed of multiple prismatic joints which lend additional degrees freedom to the system. While capable of modeling contacts with simple geometries (such as between arbitrary triangle meshes and planar surfaces), Vim has limited capability to handle contact-rich motion that introduces a large number of discontinuities. One way to handle contacts differentiably could be employ more sophisticated contact detection techniques and solve a linear complementarity problem (LCP) at each step, as done in de Avila Belbute-Peree et al. (2018).
Aside from the aforementioned drawbacks, we note that physics engines are adept at modeling phenomena which can be codified. However, there are several unmodeled physical phenomena that occur in real-world videos which must be studied in order for Vim evolve scalable framework capable of operating in the wild.
L BROADER IMPACT
Much progress has been made on end-to-end learning in visual domains. If successful, image and video understanding promises far-reaching applications from safer autonomous vehicles more realistic computer graphics, but relying on these tools for planning and control poses substantial risk.
Neural information processing systems have shown experimentally promising results visuomotor tasks, yet fail in unpredictable and unintuitive ways when deployed in real-world applications. embodied learning agents are play a broader role in the physical world, they must be held higher standard of interpretability. Establishing trust requires not just empirical, but explanatory evidence in the form of physically grounded models.
Our work provides bridge between gradient- and model- based optimization, Explicitly modeling visual dynamics using well-understood physical principles has important advantages for human explainability and dehuggability.
Unlike end-to-end neural architectures which distribute bias across a large set of parameters, VSi trades their flexibility for physical interpretability. This does not eliminate the risk of bias in simulation, but allows us to isolate bias to physically grounded variables. Where discrepancy occurs, users can probe the model to obtain end-to-end gradients with respect to variation in physical orientation and material properties, or pixelwise differences. Differentiable simulators like Vim afford number of opportunities for use and abuse. We envision the following scenarios.
technician could query a trained model, "What physical parameters is the steering controller most sensitive to?". "What happens if friction were slightly lower on that stretch of roadway?" An energy-conscion organization could use Vim to accelerate convergence of reinforcement learning models, reducing the energy consumption required for training.
Using differentiable simulation, an adversary could efficiently construct physically plausible scene causing the model to produce an incorrect prediction or take an unsafe action.
Video understanding is world-building exercise with inherent modeling bias. Using physically well-studied models makes those modeling choices explicit, however mitigating the risk of bias still requires active human participation in the modeling process. While growing number of physically-based rendering and animation efforts are currently underway, our approach does require high upfront engineering cost in simulation infrastructure. To operationalize these tools, anticipate practitioners will need to devote significant effort to identifying and replicating unmodeled dynamics from real world-trajcctories. Differentiable simulation offers a computationally tractable and physically interpretable pathway for doing so, allowing users to estimate physical trajectories and the properties which govern them.
25


Reviews:
[{"review_id": "v2AWkPsL5S0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on the problem of estimating object physical properties from video sequences. \n\nThe proposed framework combines differentiable physical simulations and differentiable rendering to map physical parameters into images differentiably. This paradigm is then used to recover physical parameters from image sequences by means of gradient based optimisation.\n\nValidation of the proposed method is carried through two main synthetic applications, parameter identification and visuomotor control. \n\nAlthough the proposed approach still requires 3D ground truth information to yield reliable estimates, it is and encouraging step towards unsupervised physics understanding from image/video data. \n\nPositive:\n\n-Crucially and differently from previous attempts, the proposed approach does not require 3D supervision - except for geometry and appearance of the static scene (i.e. at t=0). \n\n-Approach is clever, simple and yields interpretable representation\n\n-First step towards physics understanding from videos\n\nNegative:\n\n- I would improve the quality of the visualisations and plots in the paper (e.g. I found Figure 6 impossible to read)\n\n-  How to differentiate through the physical simulator was not obvious to me. I would have appreciated a more detailed explanation of how that is done in practice for one of the physical problems studied in the paper to be included in the main manuscript, in an effort to make the paper more readable. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kx4hfiZJtpi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary\n\nThis paper presents a framework for performing both differentiable physics simulations and differentiable rendering. This fully differentiable simulation and rendering pipeline is then employed to perform system identification tasks, directly from video frames, being able to match or outperform both visual-based and state-based baselines. Moreover, the potential of this framework to be applied for visuomotor control is also demonstrated.\n \n\n- Pros\n\nThis method unified advances in the differentiation of both physics simulation and rendering.\n\nThe experimental results demonstrate a good ability to perform system identification for diverse parameters and control directly from videos.\nThe ability to identify parameters or direct control tasks directly from images is useful, since it reduces the need for direct supervision/annotation in the form of state information.\n\nThe presented simulator supports a variety of \"domains\", such as rigid and deformable body dynamics, cloth simulation, and these are efficient enough to be run faster than real time (at least for simple tasks).\n\n\n\n- Cons\n\nOverall, the proposed method is mostly a unification of pre-existing techniques from different fields, such as differentiable rigid and deformable body dynamics, differentiable rendering.\n\nThe paper itself admits that a limitation of this method is that it currently \"has limited capability to handle contact-rich motion that introduces a large number of discontinuities\", which limits its applicability to real-world scenes. It cannot also currently handle joints. All of these would be important for possible robotic applications, for example.\n\nThe tasks demonstrated in the experiments are simple, and issues from model mismatch does not seem to have been thoroughly evaluated (see comments below for more).\n\n\n\n- Reasons for score\n\n[Edit: Score updated, see discussion below]\n\nOverall, given the \"pros\" described above, notably the interesting results achieved for system identification and control directly from video frames by combining differentiable physics and rendering into a single framework, I recommend this paper for acceptance. Given some of the concerns raised in the \"cons\" and in more detail in the comments below, I for now will score this paper as a little above the acceptance threshold.\n\n \n\n- Additional comments\n\nThe scenarios used for the system identification and control tasks are fairly simple, with usually only a single object and few contact points.\nWas the ground truth for the scenarios in the system identification tasks generated using gradsim itself? If so, isn't it unfair that it is compared to other models (e.g., pybullet), for which there would be model mismatch? (While not mismatch would be present for gradsim)\n\nAlong the same direction, the experiments present a section on \"Impact of imperfect dynamics and rendering models\". It would also be interesting to see a quantification of the impact of model mismatch (possibly both while using the same renderer, i.e. only dynamics mismatch, or also different renderers)\n\nIn the experiments section, it is said that \"Inference ... is done by picking an initial guess of the mass (at random)\". From what distribution is this random initial guess picked from? What are these starting guesses in relation to the true parameters?\n\nThe section on \"Impact of shading and texture cues\" seems a little too short, which renders it hard to understand in detail what is going on.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MKPyV_I8NfR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a fully differentiable physics simulation coupled with neural rendering such that input video can be used to estimate object properties or find control policies to move those objects by trying to generate the same video at the output.\n\nThe paper is well motivated by presenting a natural progression of ideas from this literature and it does a thorough job discussing related work. The paper is light on details in section 3 and it is necessary to refer to the appendix to get a complete picture. Overall, the technical contribution is solid and thus worth accepting the paper even if the validation is with relatively simpler experiments since they are sufficient to motivate this direction to be further researched. Below are a few comments to aid in improving the current work:\n\n- All experiments use what I am guessing are input (desired) videos from the same pipeline and then later hiding some parameters (to be learned). While this is a good validation the learning done here is still 'in distribution'. It would be useful to see if video (even simplistic) from a different simulator or simplified from a real world video could be applied. To what extent is this possible and are there any fundamental limitations that prevent this at the moment?\n\n- Analysis is mostly with one object in an empty scene. Are there technical limitations to handling realistic scenes where there are multiple objects and those objects interact with each other as well the environment? How does this affect performance wrt forward and backward pass timings? With such experiments, it would be helpful to understand if the released code can be easily extended to such (more complex) settings or if someone would need to start a new implementation from scratch.\n\n- The scale on the loss landscape is quite small, '0.4 pixelwise mse'. How good does the initial guess need to be to stay in the range, do the curves in fig 3 continue the trend beyond these values for larger error?\n\n- Reality gap: while this is discuss in reference to visual appearances, since the current experiments deal with synthetic scenes, the more relevant topic to discuss is the reality gap wrt physics and object motions. Experiments designed to study this would boost confidence in this approach.\n\n\nOther comments:\n\n- How much does the performance depend on good initial guess?\n\n- Currently a single impulse is used to set things in motion, can this be extended to handle more continuous actions?\n\n- Presenting qualitative results for baselines would be helpful\n\n- Some baselines not clearly explained: average, random, ConvLSTM\n\n- How does performance scale with the length of the video?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework for joint differentiable simulation of physics and image formation for inverse problems. It brings together ideas from differentiable physics and differentiable rendering in a compelling framework.
Decision: Accept (Poster)