RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation

1 INTRODUCTION . Reconstructing dynamic information about a physical system directly from a video has received considerable attention in the robotics , machine learning , computer vision , and graphics communities . This problem is fundamentally challenging because of its deep coupling among physics , geometry , and perception of a system . Traditional solutions like motion capture systems ( Vicon ; OptiTrack ; Qualisys ) can provide high-quality results but require prohibitively expensive external hardware platforms . More recent development in differentiable simulation and rendering provides an inexpensive and attractive alternative to the motion capture systems and has shown promising proof-of-concept results ( Jatavallabhula et al. , 2021 ) . However , existing methods in this direction typically assume the videos come from a known renderer . Such an assumption limits their usefulness in inferring dynamic information from an unknown rendering domain , which is common in real-world applications due to the discrepancy between rendering and real-world videos . Existing techniques for aligning different rendering domains , e.g. , CycleGAN ( Zhu et al. , 2017 ) , may help alleviate this issue . However , they typically require access to the target domain with massive data , which is not always available . To our best knowledge , inferring dynamic parameters of a physical system directly from videos under unknown rendering conditions remains far from being solved , and our work aims to fill this gap . Our work proposes a novel approach by combining three key ideas to address this challenging problem : domain randomization , state estimation , and rendering gradients . Domain randomization is a classic technique for transferring knowledge between domains , e.g. , deploying control policy trained in simulation to hardware platforms . The key idea is to let the algorithm in the source domain see massive samples under varying rendering configurations so that it is robust to discrepancies between source and target domains . We borrow this classic idea but upgrade it with two key innovations that substantially improve its effectiveness in our problem . First , we notice that image differences 1Project page : https : //sites.google.com/view/risp-iclr-2022/home . are sensitive to changes in rendering configurations . Therefore , they hardly reflect those renderinginvariant , dynamics-related parameters that we genuinely aim to match . This observation motivates us to propose a rendering-invariant state predictor ( RISP ) that extracts state information of a physical system from videos . By comparing in the state space instead of the image space , we rule out the interference of varying rendering configurations in videos from different rendering domains . The second innovation in our approach is to leverage rendering gradients from a differentiable renderer . Traditionally , domain randomization exploits data in the source domain without using their first-order information . Now that we have access to a differentiable renderer , their rendering gradients provide rich information that we should make full use of to train RISP . Essentially , requiring the output of RISP to be agnostic to rendering configurations equals enforcing its gradients for rendering parameters to be zero . Based on this idea , we propose a new loss function using rendering gradients and show an efficient method for integrating it into modern deep learning frameworks . Putting all these ideas together , we provide a powerful pipeline that effectively infers parameters of a physical system directly from video input under random rendering configurations . We demonstrate the efficacy of our approach on a variety of challenging tasks , including state estimation , system identification , imitation learning , and visuomotor control , with input videos generated from unknown rendering conditions . We evaluate these tasks on rigid-body systems with and without contact , an articulated body system , and a deformable-body system ( Fig . 1 ) , including a challenging dexterous hand ( Xu et al. , 2021 ) . The experimental results show that our approach outperforms the state-of-theart technique ( Jatavallabhula et al. , 2021 ) by a large margin in most of these tasks , which is primarily due to the inclusion of rendering gradients in the training process . In summary , our work makes the following contributions . First , we investigate and identify the bottleneck in inferring state , system , and control parameters of physical systems from videos under various rendering configurations ( Sec . 3.1 ) . Second , we propose a novel solution that combines domain randomization , state estimation , and differentiable rendering gradients to achieve generalizability across multiple rendering domains ( Sec . 3.2 ) . Third , we demonstrate the efficacy of our approach on several challenging tasks in rigid-body and deformable-body environments , including state estimation , system identification , imitation learning , and visuomotor control ( Sec . 4 ) . We will release our code and data along with the paper upon publication for readers to reproduce our results . 2 RELATED WORK . Differentiable simulation Differentiable simulation is a family of simulation methodology that equips traditional simulation with gradient information for simulation inputs . This additional gradient information connects simulation tasks with classic numerical optimization techniques . Previous works have demonstrated the power of gradients from a differentiable simulator in rigid-body dynamics ( Geilinger et al. , 2020 ; Degrave et al. , 2019 ; de Avila Belbute-Peres et al. , 2018 ; Xu et al. , 2021 ) , deformable-body dynamics ( Du et al. , 2021b ; a ; Hu et al. , 2019 ; Hahn et al. , 2019 ) , fluids ( Du et al. , 2020 ; McNamara et al. , 2004 ; Hu et al. , 2020 ) , and co-dimensional objects ( Qiao et al. , 2020 ; Liang et al. , 2019 ) , many of which show successful applications in system identification ( Hahn et al. , 2019 ) , trajectory optimization ( Hu et al. , 2019 ) , and closed-loop control ( Du et al. , 2021b ) . Our work is closely related to the progress in this field as we make heavy use of differentiable simulators in our pipeline . However , our contribution is orthogonal to them : we treat differentiable simulation as a black box , and our proposed approach is agnostic to the choice of simulators , as demonstrated by our environments involving rigid bodies , articulated bodies , and deformable bodies . Differentiable rendering Differentiable rendering methods offer gradient information for rendering inputs , e.g. , lighting , materials , cameras , or shapes ( Ramamoorthi et al. , 2007 ; Li et al. , 2015 ; Jarosz et al. , 2012 ) . The state-of-the-art differentiable renderers ( Li et al. , 2018 ; Nimier-David et al. , 2019 ) are very powerful in handling gradients even with discontinuities from occlusion or advanced global illumination effects . Our work leverages these differentiable renderers but with a quite different focus : our work instead uses rendering gradients as guidance to optimize parameters external to the renderer , which we hope can broaden the potential applications for differentiable rendering in the future . Domain randomization Our approach is closely related to domain randomization in the learning and robotics communities for transferring knowledge between domains having discrepancies ( Tobin et al. , 2017 ; Peng et al. , 2018 ; Andrychowicz et al. , 2020 ; Sadeghi & Levine , 2017 ; Tan et al. , 2018 ) . The intuition is that a model can hopefully cross the domain discrepancy by seeing a large amount of random data in the source domain . This often requires tedious data generation and leads to robust but conservative performances in the target domain . Our core idea in this work is distinctive from this line of research in that the generalizability of our method comes from a more accurate model that aims to match first-order gradient information , while the generalizability of domain randomization comes from a more robust model that attempts to absorb domain discrepancies by behaving conservatively . 3 METHOD . Given a video showing the dynamic motion of a physical system , our goal is to infer the unknown state , system , or control parameters directly from the video , with partial knowledge about the physics model and rendering conditions . Specifically , we assume we know the governing equations of the physical system ( e.g. , Newton ’ s law for rigid-body systems ) and the camera position in the video , but the exact system , control , or rendering parameters are not exposed . To solve this problem , we propose a pipeline that consists of two components : 1 ) a differentiable simulation and rendering engine ; 2 ) a novel rendering-invariant state-prediction network ( RISP ) . Our differentiable simulation and rendering engine simulates and renders the state of a physical system into an image output , and RISP learns to reconstruct state information of the physical system from images generated under varying rendering configurations . Putting these two components together , we have a pipeline that can faithfully recover dynamic information of a physical system from a new video with unseen rendering configurations . We give an overview of our approach in Fig . 2 . 3.1 DIFFERENTIABLE SIMULATION AND RENDERING ENGINE . Given a physical system with known dynamic modelM , we first use a differentiable simulator to simulate its states based on action inputs at each time step after time discretization : si+1 =Mφ ( si , ai ) , ∀i = 0 , 1 , · · · , N − 1 , ( 1 ) where N is the number of time steps in a rollout of physics simulation , and si , si+1 and ai represent the state and action vectors at the corresponding time steps , respectively . The φ vector encodes the system parameters in the model , e.g. , mass , inertia , and elasticity . Next , we apply a differentiable rendererR to generate an image Ii for each state si : Ii = Rψ ( si ) , ∀i = 0 , 1 , · · · , N. ( 2 ) Here , ψ is a vector encoding rendering parameters whose gradients are available in the renderer R. Examples of ψ include light intensity , material reflectence , or background color . By abuse of notation , we re-write the workflow of our simulation and rendering engine to a compact form : { Ii } = Rψ [ Mφ ( s0 , { ai } ) ︸ ︷︷ ︸ { si } ] . ( 3 ) In other words , given an initial state s0 and a sequence of actions { ai } , we generate a sequence of states { si } from simulation and renders the corresponding image sequence { Ii } . The task of recovering unknown information from a reference video { Irefi } can be formulated as follows : min s0 , { ai } , φ , ψ L ( { Irefi } , { Ii } ) , ( 4 ) s.t . { Ii } = Rψ [ Mφ ( s0 , { ai } ) ] , ( 5 ) where L is a loss function penalizing the difference between the generated images and their references . Assuming that the simulatorM and the renderer R are differentiable with respect to their inputs , we can run gradient-based optimization algorithms to solve Eqn . ( 4 ) . This is essentially the idea proposed in∇Sim , the state-of-the-art method for identifying parameters directly from video inputs ( Jatavallabhula et al. , 2021 ) . Specifically , ∇Sim defines L as a norm on pixelwise differences . One major limitation in Eqn . ( 4 ) is that it expects reasonably similar initial images { Ii } and references { Irefi } to successfully solve the optimization problem . Indeed , since the optimization problem is highly nonlinear due to its coupling between simulation and rendering , local optimization techniques like gradient-descent can be trapped into local minima easily if { Ii } and { Irefi } are not close enough . While ∇Sim has reported promising results when { Ii } and { Irefi } are rendered with moderately different ψ , we found in our experiments that directly optimizing L defined on the image space rarely works when the two rendering domains are vastly different ( Fig . 1 ) . Therefore , we believe it requires a fundamentally different solution , motivating us to propose RISP in our method .

Summary: This paper presents a new method of predicting physics simulation parameters and rendering configurations from an RGB video. Unlike previous methods that calculate the loss function in image space, this work proposes to calculate in the simulation state space to avoid the issues (e.g. being stuck in a local minimum) when the reference video is very different from the generated video. Specifically, a rendering-invariant state-prediction (RISP) network is pretrained with the generated data from a differentiable renderer under various rendering conditions.  The RIST network is then appended to the output of a differentiable renderer to make the pipeline from simulation and rendering parameters to states predicted from images fully differentiable. Besides a state prediction loss term for training this pipeline, a novel regularization term is used to enforce the RISP network rendering invariant. In addition, a new training strategy to efficiently calculate the gradient for the loss function is proposed. The experiments have shown that the proposed method significantly outperforms the state-of-the-arts and the proposed components contribute to the final results with big improvements.