Year: 2021

Abstract:
Simulated virtual environments serve as one of the main driving forces behind developing and evaluating skill learning algorithms. However, existing environments typically only simulate rigid body physics. Additionally, the simulation process usually does not provide gradients that might be useful for planning and control optimizations. We introduce a new differentiable physics benchmark called PasticineLab, which includes a diverse collection of soft body manipulation tasks. In each task, the agent uses manipulators to deform the plasticine into a desired configuration. The underlying physics engine supports differentiable elastic and plastic deformation using the DiffTaichi system, posing many underexplored challenges to robotic agents. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark. Experimental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently; 2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solution within tens of iterations, but still fall short on multi-stage tasks that require long-term planning. We expect that PlasticineLab will encourage the development of novel algorithms that combine differentiable physics and RL for more complex physics-based skill learning tasks. PlasticineLab is publicly available 1.

heading: 1 INTRODUCTION
text: Virtual environments, such as Arcade Learning Environment (ALE) (Bellemare et al., 2013), MuJoCo (Todorov et al., 2012), and OpenAI Gym (Brockman et al., 2016), have significantly benefited the development and evaluation of learning algorithms on intelligent agent control and planning. However, existing virtual environments for skill learning typically involves rigid-body dynamics only. Research on establishing standard soft-body environments and benchmarks is sparse, despite the wide range of applications of soft bodies in multiple research fields, e.g., simulating virtual surgery in healthcare, modeling humanoid characters in computer graphics, developing biomimetic actuators in robotics, and analyzing fracture and tearing in material science.
Compared to its rigid-body counterpart, soft-body dynamics is much more intricate to simulate, control, and analyze. One of the biggest challenges comes from its infinite degrees of freedom (DoFs) and the corresponding high-dimensional governing equations. The intrinsic complexity of soft-body dynamics invalidates the direct application of many successful robotics algorithms designed for rigid
∗This work was done during an internship at the MIT-IBM Watson AI Lab. 1Project page: http://plasticinelab.csail.mit.edu
bodies only and inhibits the development of a simulation benchmark for evaluating novel algorithms tackling soft-body tasks.
In this work, we aim to address this problem by proposing PlasticineLab, a novel benchmark for running and evaluating 10 soft-body manipulation tasks with 50 configurations in total. These tasks have to be performed by complex operations, including pinching, rolling, chopping, molding, and carving. Our benchmark is highlighted by the adoption of differentiable physics in the simulation environment, providing for the first time analytical gradient information in a soft-body benchmark, making it possible to conduct supervised learning with gradient-based optimization. In terms of the soft-body model, we choose to study plasticine (Fig. 1, left), a versatile elastoplastic material for sculpturing. Plasticine deforms elastically under small deformation, and plastically under large deformation. Compared to regular elastic soft bodies, plasticine establishes more diverse and realistic behaviors and brings challenges unexplored in previous research, making it a representative medium to test soft-body manipulation algorithms (Fig. 1, right).
We implement PlasticineLab, its gradient support, and its elastoplastic material model using Taichi (Hu et al., 2019a), whose CUDA backend leverages massive parallelism on GPUs to simulate a diverse collection of 3D soft-bodies in real time. We model the elastoplastic material using the Moving Least Squares Material Point Method (Hu et al., 2018) and the von Mises yield criterion. We use Taichi’s two-scale reverse-mode differentiation system (Hu et al., 2020) to automatically compute gradients, including the numerically challenging SVD gradients brought by the plastic material model. With full gradients at hand, we evaluated gradient-based planning algorithms on all soft-robot manipulation tasks in PlasticineLab and compared its efficiency to RL-based methods. Our experiments revealed that gradient-based planning algorithms could find a more precious solution within tens of iterations with the extra knowledge of the physical model. At the same time, RL methods may fail even after 10K episodes. However, gradient-based methods lack enough momentum to resolve long-term planning, especially on multi-stage tasks. These findings have deepened our understanding of RL and gradient-based planning algorithms. Additionally, it suggests a promising direction of combining both families of methods’ benefits to advance complex planning tasks involving soft-body dynamics. In summary, we contribute in this work the following:
• We introduce, to the best of our knowledge, the first skill learning benchmark involving elastic and plastic soft bodies.
• We develop a fully-featured differentiable physical engine, which supports elastic and plastic deformation, soft-rigid material interaction, and a tailored contact model for differentiability.
• The broad task coverage in the benchmark enables a systematic evaluation and analysis of representative RL and gradient-based planning algorithms. We hope such a benchmark can inspire future research to combine differentiable physics with imitation learning and RL.

heading: 2 RELATED WORK
text: Learning in virtual environments Recently, several simulation platforms and datasets have been developed to facilitate the research and development of new algorithms on RL and robotics. An incomplete list includes RL Benchmark (Duan et al., 2016), DeepMind Lab (Beattie et al., 2016), OpenAI Gym (Brockman et al., 2016), AI2-THOR (Kolve et al., 2017), VirtualHome (Puig et al., 2018), Gibson (Xia et al., 2018), Habitat (Savva et al., 2019), SAPIEN (Xiang et al., 2020), and
TDW (Gan et al., 2020). We observe a tendency to use full-physics simulators with realistic dynamics. However, most of these virtual environments are based on rigid-body physical engines, such as MuJoCo (Todorov et al., 2012) and PyBullet (Coumans & Bai, 2016). While some support soft-body dynamics in theory (e.g., TDW and SAPIEN is based on NVIDIA PhysX (PhysX) that supports particle simulation), none has provided the assets and tasks for soft-body manipulation. Differentiable information is also missing in these engines. We fill in this gap with our PlasticineLab benchmark.
Differentiable physics engines Differentiable physics engines for machine learning have gained increasing popularity. One family of approaches approximates physical simulators using neural networks, which are naturally differentiable (Battaglia et al., 2016; Chang et al., 2016; Mrowca et al., 2018; Li et al., 2018). A more direct and accurate approach is to implement physics-based simulators using differentiable programming systems, e.g., standard deep learning frameworks equipped with automatic differentiation tools (Degrave et al., 2016; de Avila Belbute-Peres et al., 2018; Schenck & Fox, 2018; Heiden et al., 2019). These systems are typically redistricted to explicit time integration. Other approaches of evaluating simulation gradient computation include using the adjoint methods to differentiate implicit time integrators (Bern et al., 2019; Geilinger et al., 2020), LCP (de Avila Belbute-Peres et al., 2018) and leveraging QR decompositions (Liang et al., 2019; Qiao et al., 2020). Closely related to our work is ChainQueen (Hu et al., 2019b), a differentiable simulator for elastic bodies, and DiffTaichi (Hu et al., 2020), a system to automatically generate high-performance simulation gradient kernels. Our simulator is originated from ChainQueen but with significant modifications in order to add our novel support for plasticity and contact gradients.
Trajectory optimization Our usage of differentiable simulation in planning soft-body manipulation is closely related to trajectory optimization, a topic that has been extensively studied in robotics for years and has been applied to terrestrial robots (Posa et al., 2014; Erez & Todorov, 2012; de Avila Belbute-Peres et al., 2018), aerial robots (Foehn et al., 2017; Tang & Kumar, 2015; Sreenath et al., 2013), and, closest to examples in our work, robotic manipulators (Marchese et al., 2016; Li et al., 2015). Both trajectory optimization and differentiable physics formulate planning as an optimization problem and derive gradients from governing equations of the dynamics (Tedrake, 2020). Still, the problem of motion planning for soft-body manipulation is under exploration in both communities because of two challenges: first, the high DoFs in soft-body dynamics make traditional trajectory optimization methods computationally prohibitive. Second, and more importantly, contacts between soft bodies are intricate to formulate in a concise manner. Our differentiable physics simulator addresses both issues with the recent development of DiffTaichi (Hu et al., 2020), unlocking gradientbased optimization techniques on planning for soft-body manipulation with high DoFs (> 10, 000) and complex contact.
Learning-based soft body manipulation Finally, our work is also relevant to prior methods that propose learning-based techniques for manipulating physics systems with high degrees of freedom, e.g. cloth (Liang et al., 2019; Wu et al., 2020), fluids (Ma et al., 2018; Holl et al., 2020), and rope (Yan et al., 2020; Wu et al., 2020). Compared to our work, all of these prior papers focused on providing solutions to specific robot instances, while the goal of our work is to propose a comprehensive benchmark for evaluating and developing novel algorithms in soft-body research. There are also considerable works on soft manipulators (George Thuruthel et al., 2018; Della Santina et al., 2018). Different from them, we study soft body manipulation with rigid manipulators.

heading: 3 THE PLASTICINELAB LEARNING ENVIRONMENT
text: PlasticineLab is a collection of challenging soft-body manipulation tasks powered by a differentiable physics simulator. All tasks in PlasticineLab require an agent to deform one or more pieces of 3D plasticine with rigid-body manipulators. The underlying simulator in PlasticineLab allows users to execute complex operations on soft bodies, including pinching, rolling, chopping, molding, and carving. We introduce the high-level design of the learning environment in this section and leave the technical details of the underlying differentiable simulator in Sec. 4.

heading: 3.1 TASK REPRESENTATION
text: PlasticineLab presents 10 tasks with the focus on soft-body manipulation. Each task contains one or more soft bodies and a kinematic manipulator, and the end goal is to deform the soft body into a target shape with the planned motion of the manipulator. Following the standard reinforcement
learning framework (Brockman et al., 2016), the agent is modeled with the Markov Decision Process (MDP), and the design of each task is defined by its state and observation, its action representation, its goal definition, and its reward function.
Markov Decision Process An MDP contains a state space S, an action space A, a reward function R : S × A × S → R, and a transition function T : S × A → S. In PlasticineLab, the physics simulator determines the transition between states. The goal of the agent is to find a stochastic policy π(a|s) to sample action a ∈ A given state s ∈ S, that maximizes the expected cumulative future return Eπ [ ∑∞ t=0 γ
tR(st, at)] where 0 < γ < 1 is the discount factor. State The state of a task includes a proper representation of soft bodies and the end effector of the kinematic manipulator. Following the widely used particle-based simulation methodology in previous work, we represent soft-body objects as a particle system whose state includes its particles’ positions, velocities, and strain and stress information. Specifically, the particle state is encoded as a matrix of size Np × dp where Np is the number of particles. Each row in the matrix consists of information from a single particle: two 3D vectors for position and velocities and two 3D matrices for deformation gradients and affine velocity fields (Jiang et al., 2015), all of which are stacked together and flattened into a dp-dimensional vector.
Being a kinematic rigid body, the manipulator’s end effector is compactly represented by a 7D vector consisting of its 3D position and orientation represented by a 4D quaternion, although some DoFs may be disabled in certain scenes. For each task, this representation results in an Nm × dm matrix encoding the full states of manipulators, whereNm is the number of manipulators needed in the task and dm = 3 or 7 depending on whether rotation is needed. Regarding the interaction between soft bodies and manipulators, we implement one-way coupling between rigid objects and soft bodies and fix all other physical parameters such as particle’s mass and manipulator’s friction.
Observation While the particle states fully characterize the soft-body dynamics, its high DoFs are hardly tractable for any planning and control algorithm to work with directly. We thus downsample Nk particles as landmarks and stack their positions and velocities (6D for each landmark) into a matrix of size Nk × 6, which serves as the observation of the particle system. Note that landmarks in the same task have fixed relative locations in the plasticine’s initial configuration, leading to a consistent particle observation across different configurations of the task. Combining the particle observation with the manipulator state, we end up having Nk × 6 + Nm × dm elements in the observation vector.
Action At each time step, the agent is instructed to update the linear (and angular when necessary) velocities of the manipulators in a kinematic manner, resulting in an action of size Nm × da where da = 3 or 6 depending on whether rotations of the manipulators are enabled in the task. For each task, we provide global Amin, Amax ∈ Rda , the lower and upper bounds on the action, to stabilize the physics simulation.
Goal and Reward Each task is equipped with a target shape represented by its mass tensor, which is essentially its density field discretized into a regular grid of size N3grid. At each time step t, we compute the mass tensor of the current soft body St. Discretizing both target and current shapes into a grid representation allows us to define their similarity by comparing densities at the same locations, avoiding the challenging problem of matching particle systems or point clouds. The complete definition of our reward function includes a similarity metric as well as two regularizers on the high-level motion of the manipulator: R = −c1Rmass− c2Rdist− c3Rgrasp + c4, whereRmass measures the L1 distance between the two shapes’ mass tensors as described above, Rdist is the dot product of the signed distance field (SDF) of the target shape and the current shape’s mass tensor, and Rgrasp encourages the manipulators to be closer to the soft bodies. Positive weights c1, c2, c3 are constant for all tasks. The bias c4 is selected for each environment to ensure the reward is nonnegative initially.

heading: 3.2 EVALUATION SUITE
text: PlasticineLab has a diverse collection of 10 tasks (Fig. 2). We describe four representative tasks here, and the remaining six tasks are detailed in Appendix B.
These tasks, along with their variants in different configurations, form an evaluation suite for benchmarking performance of soft-body manipulation algorithms. Each task has 5 variants (50 config-
urations in total) generated by perturbing the initial and target shapes and the initial locations of manipulators.
Rope The agent needs to wind a rope, modeled as a long plasticine piece, around a rigid pillar with two spherical manipulators. The pillar’s position varies in different configurations.
Writer The agent manipulates a “pen” (represented using a vertical capsule), to sculpt a target scribble on cubic plasticine. For each configuration, we generate the scribble by drawing random 2D lines on the plasticine surface. The three-dimensional action controls the tip of the pen.
Chopsticks The agent uses a pair of chopsticks, modeled as two parallel capsules, to pick up the rope on the ground and rotate it into the target location. The manipulator has 7 DoFs: 6 for moving and rotating the pair of chopsticks and 1 for controlling the distance between them.
RollingPin The agent learns to flatten a “pizza dough”, which is modeled as a plasticine box, with a rigid rolling pin. We simulate the rolling pin with a 3-DoF capsule: 1) the pin can descend vertically to press the dough; 2) the pin can rotate along the vertical axis to change its orientation; 3) the agent can also roll the pin over the plasticine to flatten it.

heading: 4 DIFFERENTIABLE ELASTOPLASTICITY SIMULATION
text: The simulator is implemented using Taichi (Hu et al., 2019a) and runs on CUDA. Continuum mechanics is discretized using the Moving Least Squares Material Point Method (MLS-MPM) (Hu et al., 2018), a simpler and more efficient variant of the B-spline Material Point Method (MPM) in computer graphics (Stomakhin et al., 2013). Both Lagrangian particles and Eulerian background grids are used in the simulator. Material properties, include position, velocity, mass, density, and deformation gradients, are stored on Lagrangian particles that move along with the material, while particle interactions and collisions with rigid bodies are handled on the background Eulerian grid. We refer the reader to ChainQueen (Hu et al., 2019b) and DiffTaichi (Hu et al., 2020) for more details on differentiable MPM with elastic materials. Here we focus on extending the material model with (differentiable) plasticity, a defining feature of plasticine. We leverage Taichi’s reverse-mode automatic differentiation system (Hu et al., 2020) for most of the gradient evaluations.
von Mises yield criterion We use a simple von Mises yield criterion for modeling plasticity, following the work of Gao et al. (2017). According to the von Mises yield criterion, a plasticine particle yields (i.e., deforms plastically) when its second invariant of the deviatoric stress exceeds a certain threshold, and a projection on the deformation gradient is needed since the material “forgets” its rest state. This process is typically called return mapping in MPM literature.
Return mapping and its gradients Following Klár et al. (2016) and Gao et al. (2017), we implement the return mapping as a 3D projection process on the singular values of the deformation gra-
dients of each particle. This means we need a singular value decomposition (SVD) process on the particles’ deformation gradients, and we provide the pseudocode of this process in Appendix A. For backpropagation, we need to evaluate gradients of SVD. Taichi’s internal SVD algorithm (McAdams et al., 2011) is iterative, which is numerically unstable when automatically differentiated in a bruteforce manner. We use the approach in Townsend (2016) to differentiate the SVD. For zeros appearing in the denominator when singular values are not distinct, we follow Jiang et al. (2016) to push the absolute value of the denominators to be greater than 10−6.
Contact model and its softened version for differentiability We follow standard MPM practices and use grid-base contact treatment with Coulomb friction (see, for example, Stomakhin et al. (2013)) to handle soft body collision with the floor and the rigid body obstacles/manipulators. Rigid bodies are represented as time-varying SDFs. In classical MPM, contact treatments induce a drastic non-smooth change of velocities along the rigid-soft interface. To improve reward smoothness and gradient quality, we use a softened contact model during backpropagation. For any grid point, the simulator computes its signed distance d to the rigid bodies. We then compute a smooth collision strength factor s = min{exp(−αd), 1}, which increases exponentially when d decays until d = 0. Intuitively, collision effects get stronger when rigid bodies get closer to the grid point. The positive parameter α determines the sharpness of the softened contact model. We linearly blend the grid point velocity before and after collision projection using factor s, leading to a smooth transition zone around the boundary and improved contact gradients.

heading: 5 EXPERIMENTS

heading: 5.1 EVALUATION METRICS
text: We first generate five configurations for each task, resulting in 50 different reinforcement learning configurations. We compute the normalized incremental IoU score to measure if the state reaches the goal. We apply the soft IoU (Rahman & Wang, 2016) to estimate the distance between a state and the goal. We first extract the grid mass tensor S, the masses on all grids. Each value Sxyz stores how many materials are located in the grid point (x, y, z), which is always nonnegative. Let two states’ 3D mass tensors be S1 and S2. We first divide each tensor by their maximum magnitude to normalize its values to [0, 1]: S̄1 = S1maxijk Sijk1 and S̄2 = S2maxijk Sijk2 . Then the softened IoU of the
two state is calculated as IoU(S1, S2) = ∑
ijk S̄1S̄2∑ ijk S̄1+S̄2−S̄1S̄2
. We refer readers to Appendix F for a better explanation for the soft IoU. The final normalized incremental IoU score measures how much IoU increases at the end of the episode than the initial state. For the initial state S0, the last state St at the end of the episode, and the goal state Sg , the normalized incremental IoU score is defined as IoU(St,Sg)−IoU(S0,Sg)
1−IoU(S0,Sg) . For each task, we evaluate the algorithms on five configurations and report an algebraic average score.

heading: 5.2 EVALUATIONS ON REINFORCEMENT LEARNING
text: We evaluate the performance of the existing RL algorithms on our tasks. We use three SOTA modelfree reinforcement learning algorithms: Soft Actor-Critic (SAC) (Haarnoja et al., 2017), Twin Delayed DDPG (TD3) (Fujimoto et al., 2018), and Policy Proximal Optimization (PPO) (Schulman et al., 2017). We train each algorithm on each configuration for 10000 episodes, with 50 environment steps per episode.
Figure 3 shows the normalized incremental IoU scores of the tested reinforcement learning algorithms on each scene. Most RL algorithms can learn reasonable policies for Move. However, RL algorithms can hardly match the goal shape exactly, which causes a small defect in the final shape matching. We notice that it is common for the RL agent to release the objects during exploration, leading to a free-fall of plasticine under gravity. Then it becomes challenging for the agent to regrasp the plasticine, leading to training instability and produces unsatisfactory results. The same in Rope, agents can push the rope towards the pillar and gain partial rewards, but they fail to move the rope around the pillar in the end. Increasing the numbers of manipulators and plasticine boxes causes significant difficulties in TripleMove for RL algorithms, revealing their deficiency in scaling to high dimensional tasks. In Torus, the performance seems to depend on the initial policy. They could
sometimes find a proper direction to press manipulators, but occasionally, they fail as manipulators never touch the plasticine, generating significant final score variance. Generally, we find that PPO performs better than the other two. In RollingPin, both SAC and PPO agents find the policy to go back and forth to flatten the dough, but PPO generates a more accurate shape, resulting in a higher normalized incremental IoU score. We speculate that our environment favors PPO over algorithms dependent on MLP critic networks. We suspect it is because PPO benefits from on-policy samples while MPL critic networks might not capture the detailed shape variations well.
In some harder tasks, like Chopsticks that requires the agent to carefully handle the 3D rotation, and Writer that requires the agent to plan for complex trajectories for carving the traces, the tested algorithm seldom finds a reasonable solution within the limited time (104 episodes). In Assembly, all agents are stuck in local minima easily. They usually move the spherical plasticine closer to the destination but fail to lift it up to achieve an ideal IoU. We expect that a carefully designed reward shaping, better network architectures, and fine-grained parameter tuning might be beneficial in our environments. In summary, plasticity, together with the soft bodies’ high DoFs, poses new challenges for RL algorithms.

heading: 5.3 EVALUATIONS ON DIFFERENTIABLE PHYSICS FOR TRAJECTORY OPTIMIZATION
text: Thanks to the built-in differentiable physics engine in PlasticineLab, we can apply gradient-based optimization to plan open-loop action sequences for our tasks. In gradient-based optimization, for a
certain configuration starting at state s, we initialize a random action sequence {a1, . . . , aT }. The simulator will simulate the whole trajectory, accumulate the reward at each time step, and do backpropagation to compute the gradients of all actions. We then apply a gradient-based optimization method to maximize the sum of rewards. We assume all information of the environment is known. This approach’s goal is not to find a controller that can be executed in the real world. Instead, we hope that differentiable physics can help find a solution efficiently and pave roads for other control or reinforcement/imitation learning algorithms.
In Figure 4, we demonstrate the optimization efficiency of differentiable physics by plotting the reward curve w.r.t. the number of environment episodes and compare different variants of gradient descent. We test the Adam optimizer (Adam) and gradient descent with momentum (GD). We use the soft contact model to compute the gradients. We compare the Adam optimizer with a hard contact model (Adam-H). For each optimizer, we modestly choose a learning rate of 0.1 or 0.01 per task to handle the different reward scales across tasks. Notice that we only use the soft contact model for computing the gradients and search for a solution. We evaluate all solutions in environments with hard contacts. In Figure 4, we additionally plot the training curve of reinforcement learning algorithms to demonstrate the efficiency of gradient-based optimization. Results show that optimization-based methods can find a solution for challenging tasks within tens of iterations. Adam outperforms GD in most tasks. This may be attributed to Adam’s adaptive learning rate scaling property, which fits better for the complex loss surface of the high-dimensional physical process. The hard contact model (Adam-H) performs worse than the soft version (Adam) in most tasks, which validates the intuition that a soft model is generally easier to optimize.
Table 1 lists the normalized incremental IoU scores, together with the standard deviations of all approaches. The full knowledge of the model provides differentiable physics a chance to achieve more precious results. Gradient descent with Adam can find the way to move the rope around the pillar in Rope, jump over the sub-optimal solution in Assembly to put the sphere above the box, and use the chopsticks to pick up the rope. Even for Move, it often achieves better performance by better aligning with the target shape and a more stable optimization process.
Some tasks are still challenging for gradient-based approaches. In TripleMove, the optimizer minimizes the particles’ distance to the closet target shape, which usually causes two or three plasticines to crowd together into one of the target locations. It is not easy for the gradient-based approaches, which have no exploration, to jump out such local minima. The optimizer also fails on the tasks that require multistage policies, e.g., Pinch and Writer. In Pinch, the manipulator needs to press the objects, release them, and press again. However, after the first touch of the manipulator and the plasticine, any local perturbation of the spherical manipulator doesn’t increase the reward immediately, and the optimizer idles at the end. We also notice that gradient-based methods are sensitive to initialization. Our experiments initialize the action sequences around 0, which gives a good performance in most environments.

heading: 6 POTENTIAL RESEARCH PROBLEMS TO STUDY USING PLASTICINELAB
text: Our environment opens ample research opportunities for learning-based soft-body manipulation. Our experiments show that differential physics allows gradient-based trajectory optimization to solve simple planning tasks extremely fast, because gradients provide strong and clear guidance to improve the policy. However, gradients will vanish if the tasks involve detachment and reattachment between the manipulators and the plasticine. When we fail to use gradient-based optimization that is based on local perturbation analysis, we may consider those methods that allow multi-step exploration and collect cumulative rewards, e.g., random search and reinforcement learning. Therefore, it is interesting to study how differentiable physics may be combined with these sampling-based methods to solve planning problems for soft-body manipulation.
Beyond the planning problem, it is also interesting to study how we shall design and learn effective controllers for soft-body manipulation in this environment. Experimental results (Sec. 5.2) indicate that there is adequate room for improved controller design and optimization. Possible directions include designing better reward functions for RL and investigating proper 3D deep neural network structures to capture soft-body dynamics.
A third interesting direction is to transfer the trained policy in PlasticineLab to the real world. While this problem is largely unexplored, we believe our simulator can help in various ways: 1. As shown in Gaume et al. (2018), MPM simulation results can accurately match the real world. In the future, we may use our simulator to plan a high-level trajectory for complex tasks and then combine with low-level controllers to execute the plan. 2. Our differential simulator can compute the gradient to physical parameters and optimize parameters to fit the data, which might help to close the sim2real gap. 3. PlasticineLab can also combine domain randomization and other sim2real methods (Matas et al., 2018). One can customize physical parameters and the image renderer to implement domain randomization in our simulator. We hope our simulator can serve as a good tool to study real-world soft-body manipulation problems.
Finally, generalization is an important exploration direction. Our platform supports procedure generation and can generate and simulate various configurations with different objects, evaluating different algorithms’ generalizability. PlasticineLab is a good platform to design rich goal-condition tasks, and we hope it can inspire future work.

heading: 7 CONCLUSION AND FUTURE WORK
text: We presented PlasticineLab, a new differentiable physics benchmark for soft-body manipulation. To the best of our knowledge, PlasticineLab is the first skill-learning environment that simulates elastoplastic materials while being differentiable. The rich task coverage of PlasticineLab allows us to systematically study the behaviors of state-of-the-art RL and gradient-based algorithms, providing clues to future work that combines the two families of methods.
We also plan to extend the benchmark with more articulation systems, such as virtual shadow hands2. As a principled simulation method that originated from the computational physics community (Sulsky et al., 1995), MPM is convergent under refinement and has its own accuracy advantages. However, modeling errors are inevitable in virtual environments. Fortunately, apart from serving as a strong supervision signal for planning, the simulation gradient information can also guide systematic identification. This may allow robotics researchers to “optimize” tasks themselves, potentially simultaneously with controller optimization, so that sim-to-real gaps are automatically minimized.
We believe PlasticineLab can significantly lower the barrier of future research on soft-body manipulation skill learning, and will make its unique contributions to the machine learning community.
Acknowledgement This work is in part supported by ONR MURI N00014-16-1-2007, the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF-1231216), and IBM Research.
2https://en.wikipedia.org/wiki/Shadow_Hand

heading: A SIMULATOR IMPLEMENTATION DETAILS
text: von Mises plasticity return mapping pseudo code Here we list the implementation of the forward return mapping (Gao et al., 2017). Note the SVD in the beginning leads to gradient issues that need special treatments during backpropagation. def von_Mises_return_mapping(F):
# F is the deformation gradient before return mapping
U, sig, V = ti.svd(F)
epsilon = ti.Vector([ti.log(sig[0, 0]), ti.log(sig[1, 1])]) epsilon_hat = epsilon - (epsilon.sum() / 2) epsilon_hat_norm = epsilon_hat.norm() delta_gamma = epsilon_hat_norm - yield_stress / (2 * mu)
if delta_gamma > 0: # Yields! epsilon -= (delta_gamma / epsilon_hat_norm) * epsilon_hat sig = make_matrix_from_diag(ti.exp(epsilon)) F = U @ sig @ V.transpose() return F
Parameters We use a yield stress of 50 for plasticine in all tasks except Rope, where we use 200 as the yield stress to prevent the rope from fracturing. We use α = 666.7 in the soft contact model.
Parallelism and performance Our parallel mechanism is based on ChainQueen (Hu et al.,2019b). A single MPM substep involves three stages: particle to grid transform (p2g), grid boundary conditions (gridop), and grid to particle transform (g2p). In each stage, we use a parallel for-loop to loop over all particles (p2g) or grids (for gridop and g2p) to do physical computations and ap-ply atomic operators to write results back to particles/grids. Gradient computation, which needs to reverse the three stages, is automatically done by DiffTaich (Hu et al., 2020).
We benchmark our simulator’s performance on each scene in Table 2. Note one step of our environment has 19 MPM substeps.
Table 2: Performance on an NVIDIA GTX 1080 Ti GPU. We show the average running time for a single forward or forward + backpropagation step for each scene.

heading: B MORE DETAILS ON THE EVALUATION SUITE
text: Move The agent uses two spherical manipulators to grasp the plasticine and move it to the target location. Each manipulator has 3 DoFs controlling its position only, resulting in a 6D action space.
TripleMove The agent operates three pairs of spherical grippers to relocate three plasticine boxes into the target positions. The action space has a dimension of 18. This task is challenging to both RL and gradient-based methods.
Torus A piece of cubic plasticine is fixed on the ground. In each configuration of the task, we generate the target shape by randomly relocating the plasticine and push a torus mold towards it. The agent needs to figure out the correct location to push down the mold.
Pinch In this task, the agent manipulates a rigid sphere to create dents on the plasticine box. The target shape is generated by colliding the sphere into the plasticine from random angles. To solve this task, the agent needs to discover the random motion of the sphere.
Assembly A spherical piece of plasticine is placed on the ground. The agent first deforms the sphere into a target shape and then moves it onto a block of plasticine. The manipulators are two spheres.
Table This task comes with a plasticine table with four legs. The agent pushes one of the table legs towards a target position using a spherical manipulator.

heading: C REINFORCEMENT LEARNING SETUP
text: We use the open-source implementation of SAC, PPO and TD3 in our environments. We list part of the hyperparameters in Table 3 for SAC, Table 5 for TD3 and Table 4 for PPO. We fix c1 = 10, c2 = 10 and c3 = 1 for all environments’ reward.
Table 4: PPO Parameters
update steps 2048 lr 0.0003 entropy coef 0 value loss coef 0.5 batch size 32 horizon 2500
Table 5: TD3 Parameters
start timesteps 1000 batch size 256 gamma 0.99 tau 0.005 policy noise 0.2 noise clip 0.5

heading: D ABLATION STUDY ON YIELD STRESS
text: To study the effects of yield stress, we run experiments on a simple Move configuration (where SAC can solve it well) with different yield stress. We vary the yield stress from 10 to 1000 to generate 6 environments and train SAC on them. Figure 5 plots the agents’ performances w.r.t. the number of training episodes. The agents achieve higher reward as the yield stress increase, especially in the beginning. Agents in high yield stress environments learn faster than those in lower yield stress environments. We attribute this to the smaller plastic deformation in higher yield stress environments. If we train the agents for more iterations, those in environments with yield stress larger than 100 usually converge to a same performance level, close to solving the problem. However, materials in environments with yield stress smaller than 100 tend to deform plastically, making it hard to grasp and move an object while not destroying its structure. This demonstrates a correlation between yield stress and task difficulty.

heading: E DETAILED RESULTS ON ALL CONFIGURATIONS
text: We run each algorithm with 3 random seeds and each scene has 5 configurations. Therefore, 3×5 = 15 random seeds in total are used for each algorithm on each scene. We report the normalized incremental IoU scores and the standard deviations for each configuration in Table 6. We also show the optimization efficiency of differentiable physics in Figure 6.
.

heading: F EXPLANATION OF SOFT IOU
text: Let a, b be two numbers and can only be 0 or 1, then ab = 1 if and only if a = b = 1; a+ b−ab = 1 if and only if at least one of a, b is 1. If the two mass tensors only contain value of 0 or 1, ∑ S1S2 equals the number of grids that have value 1 in both tensors, i.e., the “Intersection”. For the same reason ∑ S1 + S2 − S1S2 counts the grids that S1 = 1 or S2 = 1, the “Union”. Thus, the formula∑
S1S2/ ∑ S1 + S2 − S1S2 computes the standard Intersection over Union (IoU). In our case, we assume the normalized mass tensor S1 and S2 are two tensors with positive values. Therefore, we first normalize them so that their values are between 0 and 1, then apply the previous formula to compute a “soft IoU,” which approximately describes if two 3D shapes match.

Review summaries:
The paper introduces a new open-source simulation benchmark for soft robotics. The simulation environment builds on top of DiffTaichi, an existing differentiable simulator which enables end-to-end differentiability. The paper proposes 10 different tasks, each with 5 variations and evaluates both RL-based policy learning methods and gradient-based optimization methods on those tasks. The results suggests neither current RL-based methods nor gradient-based method can solve most of the tasks efficiently, especially for those require long-term planning.

In this work, the authors present PlasticineLab, a new framework for soft-body manipulation tasks for Reinforcement Learning and planning algorithms. The environment consists of a novel soft (i.e., deformable, plastic) material termed Plasticine which is complex to model and manipulate because of the inherently complex high-dimensional governing equations and the large number of degrees of freedom associated with soft materials. The PlasticineLab framework proposes 10 novel tasks involving manipulation of the soft plasticine material. The authors show thorough empirical analysis that traditional state of the art model-free reinforcement learning algorithms fail to effectively learn the task even after a substantial amount of training. Thus effectively showcasing the complexity of the proposed tasks and the inability of state of the art RL models to model the proposed tasks.