PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics

1 INTRODUCTION . Virtual environments , such as Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) , MuJoCo ( Todorov et al. , 2012 ) , and OpenAI Gym ( Brockman et al. , 2016 ) , have significantly benefited the development and evaluation of learning algorithms on intelligent agent control and planning . However , existing virtual environments for skill learning typically involves rigid-body dynamics only . Research on establishing standard soft-body environments and benchmarks is sparse , despite the wide range of applications of soft bodies in multiple research fields , e.g. , simulating virtual surgery in healthcare , modeling humanoid characters in computer graphics , developing biomimetic actuators in robotics , and analyzing fracture and tearing in material science . Compared to its rigid-body counterpart , soft-body dynamics is much more intricate to simulate , control , and analyze . One of the biggest challenges comes from its infinite degrees of freedom ( DoFs ) and the corresponding high-dimensional governing equations . The intrinsic complexity of soft-body dynamics invalidates the direct application of many successful robotics algorithms designed for rigid ∗This work was done during an internship at the MIT-IBM Watson AI Lab . 1Project page : http : //plasticinelab.csail.mit.edu bodies only and inhibits the development of a simulation benchmark for evaluating novel algorithms tackling soft-body tasks . In this work , we aim to address this problem by proposing PlasticineLab , a novel benchmark for running and evaluating 10 soft-body manipulation tasks with 50 configurations in total . These tasks have to be performed by complex operations , including pinching , rolling , chopping , molding , and carving . Our benchmark is highlighted by the adoption of differentiable physics in the simulation environment , providing for the first time analytical gradient information in a soft-body benchmark , making it possible to conduct supervised learning with gradient-based optimization . In terms of the soft-body model , we choose to study plasticine ( Fig . 1 , left ) , a versatile elastoplastic material for sculpturing . Plasticine deforms elastically under small deformation , and plastically under large deformation . Compared to regular elastic soft bodies , plasticine establishes more diverse and realistic behaviors and brings challenges unexplored in previous research , making it a representative medium to test soft-body manipulation algorithms ( Fig . 1 , right ) . We implement PlasticineLab , its gradient support , and its elastoplastic material model using Taichi ( Hu et al. , 2019a ) , whose CUDA backend leverages massive parallelism on GPUs to simulate a diverse collection of 3D soft-bodies in real time . We model the elastoplastic material using the Moving Least Squares Material Point Method ( Hu et al. , 2018 ) and the von Mises yield criterion . We use Taichi ’ s two-scale reverse-mode differentiation system ( Hu et al. , 2020 ) to automatically compute gradients , including the numerically challenging SVD gradients brought by the plastic material model . With full gradients at hand , we evaluated gradient-based planning algorithms on all soft-robot manipulation tasks in PlasticineLab and compared its efficiency to RL-based methods . Our experiments revealed that gradient-based planning algorithms could find a more precious solution within tens of iterations with the extra knowledge of the physical model . At the same time , RL methods may fail even after 10K episodes . However , gradient-based methods lack enough momentum to resolve long-term planning , especially on multi-stage tasks . These findings have deepened our understanding of RL and gradient-based planning algorithms . Additionally , it suggests a promising direction of combining both families of methods ’ benefits to advance complex planning tasks involving soft-body dynamics . In summary , we contribute in this work the following : • We introduce , to the best of our knowledge , the first skill learning benchmark involving elastic and plastic soft bodies . • We develop a fully-featured differentiable physical engine , which supports elastic and plastic deformation , soft-rigid material interaction , and a tailored contact model for differentiability . • The broad task coverage in the benchmark enables a systematic evaluation and analysis of representative RL and gradient-based planning algorithms . We hope such a benchmark can inspire future research to combine differentiable physics with imitation learning and RL . 2 RELATED WORK . Learning in virtual environments Recently , several simulation platforms and datasets have been developed to facilitate the research and development of new algorithms on RL and robotics . An incomplete list includes RL Benchmark ( Duan et al. , 2016 ) , DeepMind Lab ( Beattie et al. , 2016 ) , OpenAI Gym ( Brockman et al. , 2016 ) , AI2-THOR ( Kolve et al. , 2017 ) , VirtualHome ( Puig et al. , 2018 ) , Gibson ( Xia et al. , 2018 ) , Habitat ( Savva et al. , 2019 ) , SAPIEN ( Xiang et al. , 2020 ) , and TDW ( Gan et al. , 2020 ) . We observe a tendency to use full-physics simulators with realistic dynamics . However , most of these virtual environments are based on rigid-body physical engines , such as MuJoCo ( Todorov et al. , 2012 ) and PyBullet ( Coumans & Bai , 2016 ) . While some support soft-body dynamics in theory ( e.g. , TDW and SAPIEN is based on NVIDIA PhysX ( PhysX ) that supports particle simulation ) , none has provided the assets and tasks for soft-body manipulation . Differentiable information is also missing in these engines . We fill in this gap with our PlasticineLab benchmark . Differentiable physics engines Differentiable physics engines for machine learning have gained increasing popularity . One family of approaches approximates physical simulators using neural networks , which are naturally differentiable ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Mrowca et al. , 2018 ; Li et al. , 2018 ) . A more direct and accurate approach is to implement physics-based simulators using differentiable programming systems , e.g. , standard deep learning frameworks equipped with automatic differentiation tools ( Degrave et al. , 2016 ; de Avila Belbute-Peres et al. , 2018 ; Schenck & Fox , 2018 ; Heiden et al. , 2019 ) . These systems are typically redistricted to explicit time integration . Other approaches of evaluating simulation gradient computation include using the adjoint methods to differentiate implicit time integrators ( Bern et al. , 2019 ; Geilinger et al. , 2020 ) , LCP ( de Avila Belbute-Peres et al. , 2018 ) and leveraging QR decompositions ( Liang et al. , 2019 ; Qiao et al. , 2020 ) . Closely related to our work is ChainQueen ( Hu et al. , 2019b ) , a differentiable simulator for elastic bodies , and DiffTaichi ( Hu et al. , 2020 ) , a system to automatically generate high-performance simulation gradient kernels . Our simulator is originated from ChainQueen but with significant modifications in order to add our novel support for plasticity and contact gradients . Trajectory optimization Our usage of differentiable simulation in planning soft-body manipulation is closely related to trajectory optimization , a topic that has been extensively studied in robotics for years and has been applied to terrestrial robots ( Posa et al. , 2014 ; Erez & Todorov , 2012 ; de Avila Belbute-Peres et al. , 2018 ) , aerial robots ( Foehn et al. , 2017 ; Tang & Kumar , 2015 ; Sreenath et al. , 2013 ) , and , closest to examples in our work , robotic manipulators ( Marchese et al. , 2016 ; Li et al. , 2015 ) . Both trajectory optimization and differentiable physics formulate planning as an optimization problem and derive gradients from governing equations of the dynamics ( Tedrake , 2020 ) . Still , the problem of motion planning for soft-body manipulation is under exploration in both communities because of two challenges : first , the high DoFs in soft-body dynamics make traditional trajectory optimization methods computationally prohibitive . Second , and more importantly , contacts between soft bodies are intricate to formulate in a concise manner . Our differentiable physics simulator addresses both issues with the recent development of DiffTaichi ( Hu et al. , 2020 ) , unlocking gradientbased optimization techniques on planning for soft-body manipulation with high DoFs ( > 10 , 000 ) and complex contact . Learning-based soft body manipulation Finally , our work is also relevant to prior methods that propose learning-based techniques for manipulating physics systems with high degrees of freedom , e.g . cloth ( Liang et al. , 2019 ; Wu et al. , 2020 ) , fluids ( Ma et al. , 2018 ; Holl et al. , 2020 ) , and rope ( Yan et al. , 2020 ; Wu et al. , 2020 ) . Compared to our work , all of these prior papers focused on providing solutions to specific robot instances , while the goal of our work is to propose a comprehensive benchmark for evaluating and developing novel algorithms in soft-body research . There are also considerable works on soft manipulators ( George Thuruthel et al. , 2018 ; Della Santina et al. , 2018 ) . Different from them , we study soft body manipulation with rigid manipulators . 3 THE PLASTICINELAB LEARNING ENVIRONMENT . PlasticineLab is a collection of challenging soft-body manipulation tasks powered by a differentiable physics simulator . All tasks in PlasticineLab require an agent to deform one or more pieces of 3D plasticine with rigid-body manipulators . The underlying simulator in PlasticineLab allows users to execute complex operations on soft bodies , including pinching , rolling , chopping , molding , and carving . We introduce the high-level design of the learning environment in this section and leave the technical details of the underlying differentiable simulator in Sec . 4 . 3.1 TASK REPRESENTATION . PlasticineLab presents 10 tasks with the focus on soft-body manipulation . Each task contains one or more soft bodies and a kinematic manipulator , and the end goal is to deform the soft body into a target shape with the planned motion of the manipulator . Following the standard reinforcement learning framework ( Brockman et al. , 2016 ) , the agent is modeled with the Markov Decision Process ( MDP ) , and the design of each task is defined by its state and observation , its action representation , its goal definition , and its reward function . Markov Decision Process An MDP contains a state space S , an action space A , a reward function R : S × A × S → R , and a transition function T : S × A → S. In PlasticineLab , the physics simulator determines the transition between states . The goal of the agent is to find a stochastic policy π ( a|s ) to sample action a ∈ A given state s ∈ S , that maximizes the expected cumulative future return Eπ [ ∑∞ t=0 γ tR ( st , at ) ] where 0 < γ < 1 is the discount factor . State The state of a task includes a proper representation of soft bodies and the end effector of the kinematic manipulator . Following the widely used particle-based simulation methodology in previous work , we represent soft-body objects as a particle system whose state includes its particles ’ positions , velocities , and strain and stress information . Specifically , the particle state is encoded as a matrix of size Np × dp where Np is the number of particles . Each row in the matrix consists of information from a single particle : two 3D vectors for position and velocities and two 3D matrices for deformation gradients and affine velocity fields ( Jiang et al. , 2015 ) , all of which are stacked together and flattened into a dp-dimensional vector . Being a kinematic rigid body , the manipulator ’ s end effector is compactly represented by a 7D vector consisting of its 3D position and orientation represented by a 4D quaternion , although some DoFs may be disabled in certain scenes . For each task , this representation results in an Nm × dm matrix encoding the full states of manipulators , whereNm is the number of manipulators needed in the task and dm = 3 or 7 depending on whether rotation is needed . Regarding the interaction between soft bodies and manipulators , we implement one-way coupling between rigid objects and soft bodies and fix all other physical parameters such as particle ’ s mass and manipulator ’ s friction . Observation While the particle states fully characterize the soft-body dynamics , its high DoFs are hardly tractable for any planning and control algorithm to work with directly . We thus downsample Nk particles as landmarks and stack their positions and velocities ( 6D for each landmark ) into a matrix of size Nk × 6 , which serves as the observation of the particle system . Note that landmarks in the same task have fixed relative locations in the plasticine ’ s initial configuration , leading to a consistent particle observation across different configurations of the task . Combining the particle observation with the manipulator state , we end up having Nk × 6 + Nm × dm elements in the observation vector . Action At each time step , the agent is instructed to update the linear ( and angular when necessary ) velocities of the manipulators in a kinematic manner , resulting in an action of size Nm × da where da = 3 or 6 depending on whether rotations of the manipulators are enabled in the task . For each task , we provide global Amin , Amax ∈ Rda , the lower and upper bounds on the action , to stabilize the physics simulation . Goal and Reward Each task is equipped with a target shape represented by its mass tensor , which is essentially its density field discretized into a regular grid of size N3grid . At each time step t , we compute the mass tensor of the current soft body St. Discretizing both target and current shapes into a grid representation allows us to define their similarity by comparing densities at the same locations , avoiding the challenging problem of matching particle systems or point clouds . The complete definition of our reward function includes a similarity metric as well as two regularizers on the high-level motion of the manipulator : R = −c1Rmass− c2Rdist− c3Rgrasp + c4 , whereRmass measures the L1 distance between the two shapes ’ mass tensors as described above , Rdist is the dot product of the signed distance field ( SDF ) of the target shape and the current shape ’ s mass tensor , and Rgrasp encourages the manipulators to be closer to the soft bodies . Positive weights c1 , c2 , c3 are constant for all tasks . The bias c4 is selected for each environment to ensure the reward is nonnegative initially .

Summary: The paper introduces a new open-source simulation benchmark for soft robotics. The simulation environment builds on top of DiffTaichi, an existing differentiable simulator which enables end-to-end differentiability. The paper proposes 10 different tasks, each with 5 variations and evaluates both RL-based policy learning methods and gradient-based optimization methods on those tasks. The results suggests neither current RL-based methods nor gradient-based method can solve most of the tasks efficiently, especially for those require long-term planning.