PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics

1 INTRODUCTION . Virtual environments , such as Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) , MuJoCo ( Todorov et al. , 2012 ) , and OpenAI Gym ( Brockman et al. , 2016 ) , have significantly benefited the development and evaluation of learning algorithms on intelligent agent control and planning . However , existing virtual environments for skill learning typically involves rigid-body dynamics only . Research on establishing standard soft-body environments and benchmarks is sparse , despite the wide range of applications of soft bodies in multiple research fields , e.g. , simulating virtual surgery in healthcare , modeling humanoid characters in computer graphics , developing biomimetic actuators in robotics , and analyzing fracture and tearing in material science . Compared to its rigid-body counterpart , soft-body dynamics is much more intricate to simulate , control , and analyze . One of the biggest challenges comes from its infinite degrees of freedom ( DoFs ) and the corresponding high-dimensional governing equations . The intrinsic complexity of soft-body dynamics invalidates the direct application of many successful robotics algorithms designed for rigid ∗This work was done during an internship at the MIT-IBM Watson AI Lab . 1Project page : http : //plasticinelab.csail.mit.edu bodies only and inhibits the development of a simulation benchmark for evaluating novel algorithms tackling soft-body tasks . In this work , we aim to address this problem by proposing PlasticineLab , a novel benchmark for running and evaluating 10 soft-body manipulation tasks with 50 configurations in total . These tasks have to be performed by complex operations , including pinching , rolling , chopping , molding , and carving . Our benchmark is highlighted by the adoption of differentiable physics in the simulation environment , providing for the first time analytical gradient information in a soft-body benchmark , making it possible to conduct supervised learning with gradient-based optimization . In terms of the soft-body model , we choose to study plasticine ( Fig . 1 , left ) , a versatile elastoplastic material for sculpturing . Plasticine deforms elastically under small deformation , and plastically under large deformation . Compared to regular elastic soft bodies , plasticine establishes more diverse and realistic behaviors and brings challenges unexplored in previous research , making it a representative medium to test soft-body manipulation algorithms ( Fig . 1 , right ) . We implement PlasticineLab , its gradient support , and its elastoplastic material model using Taichi ( Hu et al. , 2019a ) , whose CUDA backend leverages massive parallelism on GPUs to simulate a diverse collection of 3D soft-bodies in real time . We model the elastoplastic material using the Moving Least Squares Material Point Method ( Hu et al. , 2018 ) and the von Mises yield criterion . We use Taichi ’ s two-scale reverse-mode differentiation system ( Hu et al. , 2020 ) to automatically compute gradients , including the numerically challenging SVD gradients brought by the plastic material model . With full gradients at hand , we evaluated gradient-based planning algorithms on all soft-robot manipulation tasks in PlasticineLab and compared its efficiency to RL-based methods . Our experiments revealed that gradient-based planning algorithms could find a more precious solution within tens of iterations with the extra knowledge of the physical model . At the same time , RL methods may fail even after 10K episodes . However , gradient-based methods lack enough momentum to resolve long-term planning , especially on multi-stage tasks . These findings have deepened our understanding of RL and gradient-based planning algorithms . Additionally , it suggests a promising direction of combining both families of methods ’ benefits to advance complex planning tasks involving soft-body dynamics . In summary , we contribute in this work the following : • We introduce , to the best of our knowledge , the first skill learning benchmark involving elastic and plastic soft bodies . • We develop a fully-featured differentiable physical engine , which supports elastic and plastic deformation , soft-rigid material interaction , and a tailored contact model for differentiability . • The broad task coverage in the benchmark enables a systematic evaluation and analysis of representative RL and gradient-based planning algorithms . We hope such a benchmark can inspire future research to combine differentiable physics with imitation learning and RL . 2 RELATED WORK . Learning in virtual environments Recently , several simulation platforms and datasets have been developed to facilitate the research and development of new algorithms on RL and robotics . An incomplete list includes RL Benchmark ( Duan et al. , 2016 ) , DeepMind Lab ( Beattie et al. , 2016 ) , OpenAI Gym ( Brockman et al. , 2016 ) , AI2-THOR ( Kolve et al. , 2017 ) , VirtualHome ( Puig et al. , 2018 ) , Gibson ( Xia et al. , 2018 ) , Habitat ( Savva et al. , 2019 ) , SAPIEN ( Xiang et al. , 2020 ) , and TDW ( Gan et al. , 2020 ) . We observe a tendency to use full-physics simulators with realistic dynamics . However , most of these virtual environments are based on rigid-body physical engines , such as MuJoCo ( Todorov et al. , 2012 ) and PyBullet ( Coumans & Bai , 2016 ) . While some support soft-body dynamics in theory ( e.g. , TDW and SAPIEN is based on NVIDIA PhysX ( PhysX ) that supports particle simulation ) , none has provided the assets and tasks for soft-body manipulation . Differentiable information is also missing in these engines . We fill in this gap with our PlasticineLab benchmark . Differentiable physics engines Differentiable physics engines for machine learning have gained increasing popularity . One family of approaches approximates physical simulators using neural networks , which are naturally differentiable ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Mrowca et al. , 2018 ; Li et al. , 2018 ) . A more direct and accurate approach is to implement physics-based simulators using differentiable programming systems , e.g. , standard deep learning frameworks equipped with automatic differentiation tools ( Degrave et al. , 2016 ; de Avila Belbute-Peres et al. , 2018 ; Schenck & Fox , 2018 ; Heiden et al. , 2019 ) . These systems are typically redistricted to explicit time integration . Other approaches of evaluating simulation gradient computation include using the adjoint methods to differentiate implicit time integrators ( Bern et al. , 2019 ; Geilinger et al. , 2020 ) , LCP ( de Avila Belbute-Peres et al. , 2018 ) and leveraging QR decompositions ( Liang et al. , 2019 ; Qiao et al. , 2020 ) . Closely related to our work is ChainQueen ( Hu et al. , 2019b ) , a differentiable simulator for elastic bodies , and DiffTaichi ( Hu et al. , 2020 ) , a system to automatically generate high-performance simulation gradient kernels . Our simulator is originated from ChainQueen but with significant modifications in order to add our novel support for plasticity and contact gradients . Trajectory optimization Our usage of differentiable simulation in planning soft-body manipulation is closely related to trajectory optimization , a topic that has been extensively studied in robotics for years and has been applied to terrestrial robots ( Posa et al. , 2014 ; Erez & Todorov , 2012 ; de Avila Belbute-Peres et al. , 2018 ) , aerial robots ( Foehn et al. , 2017 ; Tang & Kumar , 2015 ; Sreenath et al. , 2013 ) , and , closest to examples in our work , robotic manipulators ( Marchese et al. , 2016 ; Li et al. , 2015 ) . Both trajectory optimization and differentiable physics formulate planning as an optimization problem and derive gradients from governing equations of the dynamics ( Tedrake , 2020 ) . Still , the problem of motion planning for soft-body manipulation is under exploration in both communities because of two challenges : first , the high DoFs in soft-body dynamics make traditional trajectory optimization methods computationally prohibitive . Second , and more importantly , contacts between soft bodies are intricate to formulate in a concise manner . Our differentiable physics simulator addresses both issues with the recent development of DiffTaichi ( Hu et al. , 2020 ) , unlocking gradientbased optimization techniques on planning for soft-body manipulation with high DoFs ( > 10 , 000 ) and complex contact . Learning-based soft body manipulation Finally , our work is also relevant to prior methods that propose learning-based techniques for manipulating physics systems with high degrees of freedom , e.g . cloth ( Liang et al. , 2019 ; Wu et al. , 2020 ) , fluids ( Ma et al. , 2018 ; Holl et al. , 2020 ) , and rope ( Yan et al. , 2020 ; Wu et al. , 2020 ) . Compared to our work , all of these prior papers focused on providing solutions to specific robot instances , while the goal of our work is to propose a comprehensive benchmark for evaluating and developing novel algorithms in soft-body research . There are also considerable works on soft manipulators ( George Thuruthel et al. , 2018 ; Della Santina et al. , 2018 ) . Different from them , we study soft body manipulation with rigid manipulators . 3 THE PLASTICINELAB LEARNING ENVIRONMENT . PlasticineLab is a collection of challenging soft-body manipulation tasks powered by a differentiable physics simulator . All tasks in PlasticineLab require an agent to deform one or more pieces of 3D plasticine with rigid-body manipulators . The underlying simulator in PlasticineLab allows users to execute complex operations on soft bodies , including pinching , rolling , chopping , molding , and carving . We introduce the high-level design of the learning environment in this section and leave the technical details of the underlying differentiable simulator in Sec . 4 . 3.1 TASK REPRESENTATION . PlasticineLab presents 10 tasks with the focus on soft-body manipulation . Each task contains one or more soft bodies and a kinematic manipulator , and the end goal is to deform the soft body into a target shape with the planned motion of the manipulator . Following the standard reinforcement learning framework ( Brockman et al. , 2016 ) , the agent is modeled with the Markov Decision Process ( MDP ) , and the design of each task is defined by its state and observation , its action representation , its goal definition , and its reward function . Markov Decision Process An MDP contains a state space S , an action space A , a reward function R : S × A × S → R , and a transition function T : S × A → S. In PlasticineLab , the physics simulator determines the transition between states . The goal of the agent is to find a stochastic policy π ( a|s ) to sample action a ∈ A given state s ∈ S , that maximizes the expected cumulative future return Eπ [ ∑∞ t=0 γ tR ( st , at ) ] where 0 < γ < 1 is the discount factor . State The state of a task includes a proper representation of soft bodies and the end effector of the kinematic manipulator . Following the widely used particle-based simulation methodology in previous work , we represent soft-body objects as a particle system whose state includes its particles ’ positions , velocities , and strain and stress information . Specifically , the particle state is encoded as a matrix of size Np × dp where Np is the number of particles . Each row in the matrix consists of information from a single particle : two 3D vectors for position and velocities and two 3D matrices for deformation gradients and affine velocity fields ( Jiang et al. , 2015 ) , all of which are stacked together and flattened into a dp-dimensional vector . Being a kinematic rigid body , the manipulator ’ s end effector is compactly represented by a 7D vector consisting of its 3D position and orientation represented by a 4D quaternion , although some DoFs may be disabled in certain scenes . For each task , this representation results in an Nm × dm matrix encoding the full states of manipulators , whereNm is the number of manipulators needed in the task and dm = 3 or 7 depending on whether rotation is needed . Regarding the interaction between soft bodies and manipulators , we implement one-way coupling between rigid objects and soft bodies and fix all other physical parameters such as particle ’ s mass and manipulator ’ s friction . Observation While the particle states fully characterize the soft-body dynamics , its high DoFs are hardly tractable for any planning and control algorithm to work with directly . We thus downsample Nk particles as landmarks and stack their positions and velocities ( 6D for each landmark ) into a matrix of size Nk × 6 , which serves as the observation of the particle system . Note that landmarks in the same task have fixed relative locations in the plasticine ’ s initial configuration , leading to a consistent particle observation across different configurations of the task . Combining the particle observation with the manipulator state , we end up having Nk × 6 + Nm × dm elements in the observation vector . Action At each time step , the agent is instructed to update the linear ( and angular when necessary ) velocities of the manipulators in a kinematic manner , resulting in an action of size Nm × da where da = 3 or 6 depending on whether rotations of the manipulators are enabled in the task . For each task , we provide global Amin , Amax ∈ Rda , the lower and upper bounds on the action , to stabilize the physics simulation . Goal and Reward Each task is equipped with a target shape represented by its mass tensor , which is essentially its density field discretized into a regular grid of size N3grid . At each time step t , we compute the mass tensor of the current soft body St. Discretizing both target and current shapes into a grid representation allows us to define their similarity by comparing densities at the same locations , avoiding the challenging problem of matching particle systems or point clouds . The complete definition of our reward function includes a similarity metric as well as two regularizers on the high-level motion of the manipulator : R = −c1Rmass− c2Rdist− c3Rgrasp + c4 , whereRmass measures the L1 distance between the two shapes ’ mass tensors as described above , Rdist is the dot product of the signed distance field ( SDF ) of the target shape and the current shape ’ s mass tensor , and Rgrasp encourages the manipulators to be closer to the soft bodies . Positive weights c1 , c2 , c3 are constant for all tasks . The bias c4 is selected for each environment to ensure the reward is nonnegative initially .

Summary: In this work, the authors present PlasticineLab, a new framework for soft-body manipulation tasks for Reinforcement Learning and planning algorithms. The environment consists of a novel soft (i.e., deformable, plastic) material termed Plasticine which is complex to model and manipulate because of the inherently complex high-dimensional governing equations and the large number of degrees of freedom associated with soft materials. The PlasticineLab framework proposes 10 novel tasks involving manipulation of the soft plasticine material. The authors show thorough empirical analysis that traditional state of the art model-free reinforcement learning algorithms fail to effectively learn the task even after a substantial amount of training. Thus effectively showcasing the complexity of the proposed tasks and the inability of state of the art RL models to model the proposed tasks.