PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
Authors: Zhiao Huang
Yuanming Hu
Tao Du
Siyuan Zhou
Hao Su
Joshua B. Tenenbaum
Chuang Gan
Venue: ICLR 2021 Spotlight (2021)
Keywords: Soft Body
Differentiable Physics
Benchmark

Abstract: Simulated virtual environments serve as one of the main driving forces behind developing and evaluating skill learning algorithms. However, existing environments typically only simulate rigid body physics. Additionally, the simulation process usually does not provide gradients that might be useful for planning and control optimizations. We introduce a new differentiable physics benchmark called PasticineLab, which includes a diverse collection of soft body manipulation tasks. In each task, the agent uses manipulators to deform the plasticine into a desired configuration. The underlying physics engine supports differentiable elastic and plastic deformation using the DiffTaichi system, posing many under-explored challenges to robotic agents. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark. Experimental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently;  2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solution within tens of iterations, but still fall short on multi-stage tasks that require long-term planning. We expect that PlasticineLab will encourage the development of novel algorithms that combine differentiable physics and RL for more complex physics-based skill learning tasks. PlasticineLab will be made publicly available.

Published as a conference paper at ICLR 2021
PLASTICINELAB: A SOFT-BODY MANIPULATION BENCHMARK WITH DIFFERENTIABLE PHYSICS
Zhiao Huang UC San Diego
Yuanming Hu
Tao Du MIT taoduêcsail 1.mit.t...... edu
MIT yuanmi ng@mi edu
z2huangeeng. ucsd. edu
Siyuan Zhou
Hao Su
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL jbt
Peking University elegyaunter@gmai1.ccm
UC San Diego
haosußeng uesd. edu
Chuang Gan
MIT-IBM Watson AI Lab
ganchuang@csail mit. edu
ABSTRACT
Simulated virtual environments serve as one the main driving forces behind developing and evaluating skill learning algorithms. However, existing envi- ronments typically only simulate rigid body physics. Additionally, the simula- tion process usually does not provide gradients that might be useful for planning and control optimizations. We introduce a new differentiable physics benchmark called PasticineLab, which includes a diverse collection of soft body manipula- tion tasks. In each task, the agent uses manipulators to deform the plasticine into desired configuration. The underlying physics engine supports differentiable elastic and plastic deformation using the DiffTaichi system, posing many under- explored challenges robotic agents. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark. Experi- mental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently; 2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solu- tion within tens of iterations, but still fall short multi-stage tasks that require long-term planning. We expect that Plasticinel.a aa will encourage the development of novel algorithms that combine differentiable physics and RI. for more complex srysics-bbsed skill learning tasks. PlasticineLab is publicly available !
INTRODUCTION
Virtual environments, such Arcade Learning Environment (ALE) (Bellemare al., 2013), Mu- JoCo (Todorov et al., 2012), and OpenAl Gym (Brockman et al., 2016), have significantly benefited the development and evaluation of learning algorithms on intelligent agent control and planning. However, existing virtual environments for skill learning typically involves rigid-body dynamics only. Research establishing standard soft-body environments and benchmarks sparse, despite the wide range of applications of soft bodies in multiple research fields, e.g., simulating virtual surgery in healthcare, modeling humanoid characters in computer graphics, developing biomimetic actuators in robotics, and analyzing fracture and tearing in material science.
Compared to its rigid-body counterpart, soft-body dynamics is much more intricate to simulate, con- trol, and analyze. One of the biggest challenges comes from its infinite degrees of freedom (DoFs) and the corresponding high-dimensional governing equations. The intrinsic complexity of soft-body dynamics invalidates the direct application of many successful robotics algorithms designed for rigid
"This work was done during an internship at the MIT-IBM Watson AI Lab.
Project page: http:/ /p1 ast nel ab. csai 1.mi edu

---

Published as a conference paper at ICLR 2021
bodies only and inhibits the development of simulation benchmark for evaluating novel algorithms tackling soft-body tasks.
In this work, aim to address this problem by proposing PlasticineLab, novel benchmark for running and evaluating soft-body manipulation tasks with 50 configurations in total. These tasks have to be performed by complex operations, including pinching, rolling, chopping, molding, and carving. Our benchmark is highlighted by the adoption differentiable physics in the simulation environment, providing for the first time analytical gradient information soft-body benchmark, making it possible to conduct supervised learning with gradient-based optimization. In terms of the soft-body model, we choose to study plasticine (Fig. 1. left), versatile elastoplastic material for sculpturing. Plasticine deforms elastically under small deformation, and plastically under large de- formation. Compared regular elastic soft bodies, plasticine establishes more diverse and realistic behaviors and brings challenges unexplored in previous research, making it representative medium test soft-body manipulation algorithms (Fig. right).
We implement PlasticineLab, its gradient support, and its elastoplastic material model using Taichi (Hu et al., 2019a), whose CUDA backend leverages massive parallelism on GPUs simulate diverse collection of 3D soft-bodies in real time. We model the elastoplastic material using the Moving Least Squares Material Point Method (Hu et al., 2018) and the von Mises yield criterion. We use Taichi's two-scale pverse-mode differentiation system (Hu et al., 2020) automatically compute gradients, including the numerically challenging SVD gradients brought by the plastic ma- terial model. With full gradients hand, evaluated gradient-base planning algorithms all soft-robot manipulation tasks in PlasticineLab and compared its efficiency to RL-based methods. Our experiments revealed that gradient-based planning algorithms could find more precious solu- tion within tens of iterations with the extra knowledge of the physical model. At the same time, RL methods may fail even after 10K episodes. However, gradient- based methods lack enough momen- tum to resolve long-term planning, especially on multi-stage tasks. These findings have deepened our understanding of RI. and gradient- planning algorithms. Additionally, it suggests a promis- ing direction of combining both families of methods' benefits to advance complex planning tasks involving soft-body dynamics. In summary. we contribute in this work the following:
We introduce, to the best of our knowledge, the first skill learning benchmark involving elastic and plastic soft bodies.
We develop a fullu-featured differentiable physical engine, which supports elastic and plastic deformation, soft-rigid material interaction, and a tailored contact model for differentiabilityy
The broad task coverage in the benchmark enables a systematic evaluation and analysis of rep- resentative RL and gradient-based planning algorithms. We hope such benchmark can inspire future research to combine differentiable physics with imitation learning and RL.
Manipulator
Plasticine
Human
Initial State
Manipulation
Target Plasticine Shape
Figure 1: Left: A child deforming a piece of plasticine into a thin pie using rolling pin. Right: The challenging RollingPin scene in PlasticineLab. The agent needs to flatten the material by rolling the pin back and forth, so that the plasticine deforms into the target shape.
RELATED WORK
Learning in virtual environments Recently, several simulation platforms and datasets have been developed to facilitate the research and development of new algorithms on RL and robotics. An incomplete list includes RL Benchmark (Duan et al., 2016), DeepMind Lab (Beattie et al., 2016), OpenAl Gym (Brockman et al., 2016), AI2-THOR (Kolve et al., 2017), VirtualHome (Puig et al., 2018), Gibson (Xia al., 2018), Habitat (Savva al., 2019), SAPIEN (Xiang et al., 2020), and
2

---

Published as a conference paper at ICLR 2021
TDW (Gan et al., 2020). We observe a tendency to use full-physics simulators with realistic dynam- ics. However, most of these virtual environments are based on rigid -ody physical engines, such as MuJoCo (Todorov et all, 2012) and PyBullet (Coumans & Bai, 2016). While some support soft-body dynamics in theory (e.g., TDW and SAPIEN is based on NVIDIA PhysX (PhysX) that supports par- ticle simulation), none has provided the assets and tasks for soft-hody manipulation. Differentiable information is also missing in these engines. We fill in this gap with our PlasticineLab benchmark.
Differentiable physics engines Differentiable physics engines for machine learning have gained increasing popularity. One family of approaches approximates physical simulators using neural net- works, which are naturally differentiable (Battaglia et al., 2016; Chang al., 2016; Mrowca al., 2018; Li et al., 2018). A more direct and accurate approach is to implement physics physics-based simulators using differentiable programming systems, e.g., standard deep learning frameworks equipped with automatic differentiation tools (Degrave et al., 2016; de Avila Belbute-Peree et al., 201 8; Schenck & Fox, 2018; Heiden et al., 2019). These systems are typically redistricted to explicit time integration. Other approaches of evaluating simulation gradient computation include using the adjoint methods to differentiate implicit time integrators (Bern et al., 2019; Geilinger et al., 2020), LCP (de Avila Belbute-Peree et al., 2018) and leveraging QR decompositions (Liang et al., 2019; Qiao et al., 2020). Closely related to our work is ChainQueen (Hu et al., 2019b), a differentiable simulator for elas- tic bodies, and DiffTaichi (Hu al., 2020), system automatically generate high-performance simulation gradient kernels. Our simulator is originated from ChainQueen but with significant mod- ifications in order to add our novel support for plasticity and contact gradients.
Trajectory optimization Our usage of differentiable simulation in planning soft-body manipulation closely related to trajectory optimization, a topic that has been extensively studied in robotics for years and has been applied to terrestrial robots (Posa et al., 2014; Erez & Todorov, 2012; de Avila Belbute-Peree al., 2018), aerial robots (Foehn al., 2017: Tang & Kumar, 2015; Sreenath al., 2013), and, closest to examples in our work, robotic manipulators (Marchese et al., 2016; Li et al., 2015). Both trajectory optimization and differentiable physics formulate planning as an optimization problem and derive gradients from governing equations of the dynamics (Tedrake, 2020). Still, the problem of motion planning for soft-body manipulation is under exploration in both communities because of two challenges: first, the high DoFs in soft-body dynamics make traditional trajectory optimization methods computationally prohibitive. Second, and more importantly, contacts between soft bodies intricate to formulate in concise manner. Our differentiable physics simulator ad- dresses both issues with the recent development of DiffTaichi (Hu et al., 2020), unlocking gradient- based optimization techniques on planning for soft-body manipulation with high DoFs (> 10, 000) and complex contact.
Learning-based soft body manipulation Finally, our work is also relevant to prior methods that propose baarning-based techniques for manipulating physics systems with high degrees of freedom, e.g. cloth (Liang et al., 2019; Wu et al., 2020), fluids (Ma et al., 2018; Holl al., 2020), and rope (Yan et al., 2020; Wu et al., 2020). Compared to our work, all of these prior papers focused on providing solutions to specific robot instances, while the goal of our work is to propose a compre- hensive benchmark for evaluating and developing novel algorithms in soft-body research. There are also considerable works on soft manipulators (George Thuruthel et al., 2018; Della Santina et al., 2018). Different from them, we study soft body manipulation with rigid manipulators.
3 THE PLASTICINELAB LEARNING ENVIRONMENT
PlasticineLab is a collection of challenging soft- body manipulation tasks powered by a differentiable physics simulator. All tasks in PlasticineLab require an agent to deform or more pieces of 3D plasticine with rigid-body manipulators. The underlying simulator in Plasticinel.at Lab allows users to execute complex operations soft bodies, including pinching, rolling, chopping, molding, and carving. We introduce the high-level design of the learning environment in this section and leave the technical details of the underlying differentiable simulator in Sec. 4.
3.1 TASK REPRESENTATION
PlasticineLab presents 10 tasks with the focus soft-body manipulation. Each task contains one or more soft bodies and a kinematic manipulator. and the end goal is to deform the soft body into target shape with the planned motion of the manipulator. Following the standard reinforcement
3

---

Published as a conference paper at ICLR 2021
learning framework (Brockman al., 2016), the agent is modeled with the Markov Decision Process (MDP), and the design of each task is defined by its state and observation, its action representation, its goal definition, and its reward function.
Markov Decision Process An MDP contains a state space S, an action space A, a reward function R : S x A x S - R. and transition function T S A + S. In PlasticineLab,, the physics simulator determines the transition between states. The goal of the agent is to find a stochastic policy (als) to sample action a A given state s E S, that maximizes the expected cumulative future return Ex EI= 7 ('(( at)] where 0 < 7 < 1 is the discount factor.
State The state of task includes proper representation of soft bodies and the end effector of the kinematic manipulator. Following the widely used particle-based simulation methodology in previous work, we represent soft-body objects a particle system whose state includes its particles' positions, velocities, and strain and stress information. Specifically, the particle state is encoded as a matrix of size Np x dp where Np is the number of particles. Each row in the matrix consists of information from single particle: two 3D vectors for position and velocities and two 3D matrices for deformation gradients and affine velocity fields (Jiang et al., 2015), all of which are stacked together and flattened into dp dd-dmennioonal vector.
Being kinematic rigid body, the manipulator's end effector compactly represented by 7D vector consisting of its 3D position and orientation represented by 4D quaternion, although some DoFs may be disabled in certain scenes. For each task, this representation results in an Nm x dm matrix encoding the full states of manipulators, where N,, is the number of manipulators needed in the task and dm 3 or 7 depending on whether rotation is needed. Regarding the interaction between soft bodies and manipulators, we implement one coupling between rigid objects and soft bodies and fix all other physical parameters such as particle's mass and manipulator's friction.
Observation While the partiele states fully characterize the soft-body dynamics, its high DoFs hardly tractable for any planning and control algorithm to work with directly. We thus downsample No particles as landmarks and stack their positions and velocities (6D for each landmark) into a matrix of size N₂ x 6, which serves as the observation of the particle system. Note that landmarks in the same task have fixed relative locations in the plasticine's initial configuration, leading to a consistent particle observation across different configurations of the task. Combining the particle observation with the manipulator state, we end up having N2 x 6 + Nm dm elements in the observation vector.
Action At each time step, the agent instructed to update the linear (and angular when necessary) velocities of the manipulators in kinematic manner, resulting in an action of size Nm. da where da = or depending on whether rotations of the manipulators are enabled in the task. For each task, we provide global Amin, Amax Rdd,, the lower and upper bounds on the action, stabilize the physics simulation.
Goal and Reward Each task is equipped with target shape represented by its mass tensor, which is essentially its density field discretized into a regular grid of size N grid- At each time step 4, we compute the mass tensor of the current soft body St. Discretizing both target and current shapes into grid representation allows us to define their similarity by comparing densities at the same locations, avoiding the challenging problem of matching particle systems or point clouds. The complete definition of our reward function includes similarity metric as well as two regularizers on the high-level motion of the manipulator: R = -c Rmass C2 Raist C3 Rgaspp + C4, where Reass measures the L1 distance between the two shapes' mass tensors as described above, Rais is the dot product of the signed distance field (SDF) of the target shape and the current shape's 's mass tensor, and Reass encourages the manipulators be closer to the soft bodies. Positive weights C1 C2 C3 are constant for all tasks. The bias e4 is selected for each environment to ensure the reward is nonnegative initially.
3.2 EVALUATION SUITE
PlasticineLabb has a diverse collection of 10 tasks (Fig. 2). We describe four representative tasks here, and the remaining six tasks are detailed in Appendix B.
These tasks, along with their variants in different configurations, form an evaluation suite for bench- marking performance of soft-body manipulation algorithms. Each task has 5 variants (50 config-
4

---

Published as a conference paper at ICLR 2021
Trip
Torus
Rope
Writer
Pinch
Chopsticks
RollingPin
Assembly
Table
Figure 2: Tasks and reference solutions of Plasticinel.at Lab. Certain tasks require multi-stage planning.
urations in total) generated by perturbing the initial and target shapes and the initial locations of manipulators.
Rope The agent needs wind rope, modeled long plasticine piece, around rigid pillar with two spherical manipulators. The pillar's position varies in different configurations.
Writer The agent manipulates "pen" (represented using a vertical capsule), to sculpt a target scribble on cubic plasticine. For each configuration, we generate the scribble by drawing random 2D lines on the plasticine surface. The firre-dimensional action controls the tip of the pen.
Chopsticks The agent uses pair of chopsticks, modeled as two parallel capsules, to pick up the rope on the ground and rotate into the target location. The manipulator has 7 DoFs: for moving and rotating the pair of chopsticks and for controlling the distance between them.
RollingPin The agent learns to flatten "pizza dough", which is modeled as plasticine box, with rigid rolling pin. We simulate the rolling pin with a 3-DoF capsule: 1) the pin can descend vertically to press the dough; 2) the pin can rotate along the vertical axis to change its orientation; 3) the agent can also roll the pin the plasticine to flatten it.
4 DIFFERENTIABLE ELASTOPLASTICIT SIMULATION
The simulator is implemented using Taichi (Hu et al., 2019a) and runs CUDA. Continuum chanics is discretized using the Moving Least Squares Material Point Method (MLS-MPM) (Hu al., 2018), simpler and efficient variant of the B-spline Material Point Method (MPM) in computer graphics (Stomakhin et al., 2013). Both Lagrangian particles and Eulerian background grids are used in the simulator. Material properties, include position, velocity, mass, density, and deformation gradients, are stored on Lagrangian particles that move along with the material, while particle interactions and collisions with rigid bodies are handled on the background Eulerian grid. We refer the reader to ChainQueen (Hu et al., 2019b) and DiffTaichi (Hu et al., 2020) for more de- tails on differentiable MPM with elastic materials. Here we focus on extending the material model with (disferentiable) plasticity, defining feature of plasticine. We leverage Taichi's reverse-mode automatic differentiation system (Hu et al., 2020) for most of the gradient evaluations.
von Mises yield criterion We simple Mises yield criterion for modeling plasticity, follow- ing the work of Gao et al. (2017). According to the von Mises yield criterion, a plasticine particle yields (i.e., deforms plastically) when its second invariant of the deviatoric stress exceeds a certain threshold, and a projection on the deformation gradient is needed since the material "forgets" its rest state. This process is typically called return mapping in MPM literature.
Return mapping and its gradients Following Klár et al. (2016) and Gao et al. (2017), we imple- ment the return mapping 3DD projection process the singular values of the deformation gra-
5

---

Published as a conference paper at ICLR 2021
dients of each particle. This means we need singular value decomposition (SVD) process on the particles' deformation gradients, and we provide the pseudocode of this process in Appendix A. For backpropagaatin, we need to evaluate gradients of SVD. Taichi's internal SVD algorithm (McAdams et al., 201 1) is iterative, which is numerically unstable when automatically differentiated in a brute- force manner. We the approach in Townsend (2016) differentiate the SVD. For zeros appear- ing in the denominator when singular values are not distinct, we follow Jiang et al. (2016) to push the absolute value of the denominators to be greater than 10-6
Contact model and its softened version for differentiability We follow standard MPM practices and use grid-base contact treatment with Coulomb friction (see, for example, Stomakhin et al. (2013)) to handle soft body collision with the floor and the rigid body obstacles/maniiplltoos Rigid bodies are represented as time-varying SDFs. In classical MPM, contact treatments induce a drastic non-smooth change of velocities along the rigid-soft interface. To improve reward smoothness and gradient quality, we use a softened contact model during backpropagation. For any grid point, the simulator computes its signed distance df to the rigid bodies. We then compute smooth collision strength factor 8 = miin(ex(--adi Intuitively, collision effects get stronger when rigid bodies get closer to the grid point. The positive parameter a determines the sharpness of the softened contact model. We linearly blend the grid point velocity before and after collision projection using factor 8, leading smooth zone around the boundary and improved contact gradients.
1}, which increases exponentially when d decays until = O.
transition
EXPERIMENTS
5.1 EVALUATION METRICS
We first generate five configurations for each task, resulting in 50 different reinforcement learning configurations. We compute the normalized incremental IoU score to measure if the state reaches the goal. We apply the soft IoU (Rahman & Wang, 2016) to estimate the distance between a state and the goal. We first extract the grid mass tensor S, the masses on all grids. Each value S stores how many materials are located in the grid point (Ir, y, z). which is always nonnegative. Let two states' 3D mass tensors be S1 and S2. We first divide each tensor by their maximum magnitude to normalize its values to [0, 1]: S
S1
and S2
maxxss STz. Then the softened IoU of the
Sa
maxiyk STTE
En,. S.Sa
two state is calculated as IoU (S1, = Eej -,+, We refer readers to Appendix for a better explanation for the soft IOU. The final normalized incremental IoU score measures how much IoU increases at the end of the episode than the initial state. For the initial state So, the last state S the end of the episode, and the goal state Sy, the normalized incremental IoU score is defined as For each task, we evaluate the algorithms on five configurations and report an algebraic average score.
(S,.U(
5.2 EVALUATIONS ON REINFORCEMENT LEARNING
We evaluate the performance of the existing RL algorithms on our tasks. We use three SOTA model- free reinforcement learning algorithms: Soft Actor-Critic (SAC) (Haarnoja et al., 2017), Twin De- layed DDPG (TD3) (Fujimoto et al., 2018), and Policy Proximal Optimization (PPO) (Schulman et al., 2017). We train each algorithm on each configuration for 10000 episodes, with 50 environ- ment steps per episode.
Figure 3 shows the normalized incremental IoU scores of the tested reinforcement learning algo- rithms on each scene. Most RL algorithms can learn reasonable policies for Move. However, RL algorithms can hardly match the goal shape exactly, which causes a small defect in the final shape matching. We notice that it is common for the RL agent to release the objects during exploration, leading to free-fall of plasticine under gravity. Then it becomes challenging for the agent to regrasp the plasticine, leading to training instability and produces unsatisfactory results. The same in Rope, agents can push the rope towards the pillar and gain partial rewards, but they fail to move the rope around the pillar in the end. Increasing the numbers of manipulators and plasticine boxes causes sig- nificant difficulties in Move for RL algorithms, revealing their deficiency in scaling to high dimensional tasks. In Torus, the performance seems depend on the initial policy. They could
6

---

Published as a conference paper at ICLR 2021
1.0
Normalized incremental lou.
SAC
TD3
PPO
0.8
0.6
0.4
0.2
0.0
Move
TripleMove
Torus
Rope
Writer
Pinch
RollingPin
Chopsticks
Assembly
Table
Figure 3: The final normalized incremental IoU score achieved by RL methods within 10 epochs. Scores lower than O are clamped. The dashed orange line indicates the theoretical upper limit.
sometimes find proper direction to press manipulators, but occasionally, they fail as manipulators never touch the plasticine, generating significant final score variance. Generally, we find that PPO performs better than the other two. In RollingPin, both SAC and PPO agents find the policy to go back and forth flatten the dough, but PPO generates normalized incremental loU score. We speculate that our environment favors PPO over algorithms dependent MLP critic networks. We suspect it is because PPO benefits from on-policy samples while MPL. critic networks might not capture the detailed shape variations well.
accurate shape, resulting in higher
In harder tasks, like Chopsticks that requires the agent carefully handle the 3D rotation, and Writer that requires the agent to plan for complex trajectories for carving the traces, the tested algorithm seldom finds a reasonable solution within the limited time (104 episodes). In Assembly, all agents are stuck in local minima easily. They usually move the spherical plasticine closer to the destination but fail to lift it up to achieve an ideal IoU, We expect that carefully designed reward shaping, better network architectures, and fine-grained parameter tuning might be beneficial our environments. summary, plasticity, together with the soft bodies' high DoFs, poses new challenges for RL algorithms.
5.3
EVALUATIONS DIFFERENTIABLE PHYSICS FOR TRAIECTORY OPTIMIZATION
Move
TripleMove
Torus
Rope
Writer
200
Epiendee
Spaodes
Epleades
Epinurctes
Epesodes
Pinch
RollingPin
Chopsticks
Assembly
Table
Rowcos
150
Eptoodos
sp-sodes
Estaodas
EpGodds
Ep-sodas
SAC
TD3
PPO
Adam
GD
Adam-H
Figure 4: Rewards and their variances in each task w.r.t. the number of episodes spent training. We clamp the reward to be greater than for better illustration.
Thanks to the built-in differentiable physics engine PlasticineLab. we can apply gradient-based optimization to plan open-loop action sequences for tasks. In gradient-based optimization, for

---

Published as a conference paper at ICLR 2021
Env
Move
Tri. Move
Torus
Rope
Writer
SAC
0.27 + 0.27
0.12 1 0.11
0.53 1 0.42
0.29 1 0.20
0.41 1 0.23
TD3
0.31 + 0.20
0.00 0.02
0.42 + 0.42
0.12 0.15
0.19 = 0.18
PPO
0.69 = 0.15
0.13 - 0.09
0.44 + 0.38
0.38 + 0.19
0.37 = 0.17
Adam
0.90 + 0.12
0.35 + 0.20
0.77 + 0.39
0.59 + 0.13
0.62 + 0.27
GD
0.51 + 0.39
0.24 + 0.17
0.79 1 0.37
0.41 + 0.17
0.69 + 0.29
Adam-H
0.05 + 0.15
0.26 0.15
0.72 + 0.23
0.21 + 0.09
0.00 0.00
Env
Pinch
RollingPin
Chopsticks
Assembly
Table
SAC
0.05 + 0.08
0.36 + 0.30
0.13 + 0.08
0.00 + 0.00
0.01 + 0.12
TD3
0.01 = 0.02
0.11 0.02
0.11 + 0.07
0.00 = 0.00
0.10 = 0.16
PPO
0.06 + 0.09
0.86 0.10
0.14 0.09
0.06 + 0.17
0.29 + 0.28
Adam
0.08 + 0.08
0.93 0.04
0.88 + 0.08
0.90 + 0.10
0.01 = 0.01
GD
0.03 + 0.05
0.89 + 0.11
0.03 + 0.04
0.27 + 0.36
0.00 + 0.00
Adam-H
0.00 = 0.02
0.26 0.12
0.02 + 0.06
0.03 + 0.03
0.00 = 0.01
Table I: The averaged normalized incremental IoU scores and the standard deviations of cach method. Adam-H stands for optimizing the hard contact model with Adam optimizer. We train RL agent for 10000 episodes and optimizing for 200 episodes for gradient- hased approaches.
certain configuration starting at state s, we initialize a random action sequence {aI..... aT The simulator will simulate the whole trajectory, accumulate the reward each time step, and do back- propagation to compute the gradients of all actions. We then apply gradient-based optimization method to maximize the sum of rewards. We assume all information of the environment is known. This approach's 's goal is not to find a controller that can be executed in the real world. Instead, we hope that differentiable physics can help find a solution efficiently and pave roads for other control or reinforcement/imitaiiin
learning algorithms.
In Figure 4, we demonstrate the optimization efficiency of differentiable physics by plotting the reward curve w.r.t. the number of environment episodes and compare different variants of gradi- ent descent. We test the Adam optimizer (Adam) and gradient descent with momentum (GD). We use the soft contact model to compute the gradients. We compare the Adam optimizer with a hard contact model (Adam-H) (a).m-H). For each optimizer, we modestly choose a learning rate of 0.1 or 0.01 per task to handle the different reward scales across tasks. Notice that we only the soft contact model for computing the gradients and search for a solution. We evaluate all solutions in envi- ronments with hard contacts. In Figure 4, we additionally plot the training curve of reinforcement learning algorithms to demonstrate the efficiency of gradient- based optimization. Results show that optimization- based methods can find a solution for challenging tasks within tens of iterations. Adam outperforms GD in most tasks. This may be attributed to Adam's 's adaptive learning rate scaling prop- erty. which fits better for the complex loss surface of the high-dimensional physical process. The hard contact model (Adam-H) performs worse than the soft version (Adam) in most tasks, which validates the intuition that soft model generally easier to optimize.
Table lists the normalized incremental IoU scores, together with the standard deviations of all approaches. The full knowledge of the model provides differentiable physics chance to achieve more precious results. Gradient descent with Adam can find the way to move the rope around the pillar in Rope, jump over the sub-optimal solution in Assembly to put the sphere above the box, and use the chopsticks pick up the rope. Even for Move, it often achieves better performance by better aligning with the target shape and a more stable optimization process.
Some tasks are still challenging for gradient-based approaches. In TripleMove, the optimizer mini- mizes the particles' distance to the closet target shape, which usually causes two or three plasticines to crowd together into one of the target locations. It is not easy for the gradient-based approaches, which have no exploration, to jump out such local minima. The optimizer also fails on the tasks that require multistage policies, e.g., Pinch and Writer. In Pinch, the manipulator needs press the objects, release them, and press again. However, after the first touch of the manipulator and the plas- ticine, any local perturbation of the spherical manipulator doesn't increase the reward immediately, and the optimizer idles at the end. We also notice that gradient- based methods are sensitive to initial- ization. Our experiments initialize the action sequences around O, which gives a good performance in most environments.
8

---

Published as a conference paper at ICLR 2021
6 POTENTIAL RESEARCH PROBLEMS TO STUDY USING PLASTICINELAB
Our environment opens ample research opportunities for learning-based soft-hody manipulation. Our experiments show that differential physics allows gradient- trajectory optimization to solve simple planning tasks extremely fast, because gradients provide strong and clear guidance to improve the policy. However, gradients will vanish if the tasks involve detachment and reattachment between the manipulators and the plasticine. When we fail to use gradient-basee optimization that is based on local perturbation analysis, we may consider those methods that allow multi-step explo- ration and collect cumulative rewards, e.g., random search and reinforcement learning. Therefore, it is interesting to study how differentiable physics may be combined with these sampling-base methods to solve planning problems for soft-body manipulation.
Beyond the planning problem, it is also interesting to study how we shall design and learn effective controllers for soft-body manipulation in this environment. Experimental results (Sec. 5.2) indicate that there is adequate room for improved controller design and optimization. Possible directions include designing better reward functions for RL and investigating proper 3D deep neural network structures to capture soft-body dynamics.
A third interesting direction is to transfer the trained policy in PlasticineLab to the real world. While this problem is largely unexplored, we believe our simulator can help in various ways: As shown in Gaume et al. (2018), MPM simulation results can accurately match the real world. In the future, may use our simulator plan a high-level trajectory for complex tasks and then combine with low-level controllers to execute the plan. 2. Our differential simulator can compute the gradient to physical parameters and optimize parameters to fit the data, which might help to close the sim2real gap. 3. PlasticineLab can also combine domain randomization and other sim2real methods (Matas al., 2018). One can customize physical parameters and the image renderer to implement domain randomization in our simulator. We hope our simulator can serve good tool to study real-world soft-body manipulation problems.
Finally, generalization is an important exploration direction. Our platform supports procedure gen- eration and can generate and simulate various configurations with different objects, evaluating dif- ferent algorithms' generalizability. PlasticineLab is good platform to design rich goal-condition tasks, and we hope can inspire future work.
7 CONCLUSION AND FUTURE WORK
We presented PlasticineL.ab, Lab, new differentiable physics benchmark for soft-body manipulation. To the best of our knowledge, PlasticineLab is the first skill-learning environment that simulates elastoplastic materials while being differentiable. The rich task coverage of PlasticineLab allows us to systematically study the behaviors of state-of-the-arr RL and gradient-based algorithms, providing clues to future work that combines the two families of methods.
We also plan to extend the benchmark with more articulation systems, such as virtual shadow hands?. As principled simulation method that originated from the computational physics community (Sul- sky et al., 1995), MPM is convergent under refinement and has its own accuracy advantages. How- ever, modeling errors are inevitable in virtual environments. Fortunately, apart from serving as a strong supervision signal for planning, the simulation gradient information can also guide system- atic identification. This may allow robotics researchers to "optimize" tasks themselves, potentially simultaneously with controller optimization, so that sim-to-real gaps are automatically minimized.
We believe PlasticineL can significantly lower the barrier of future research on soft-body manip- ulation skill learning, and will make its unique contributions to the machine learning community.
Acknowledgement This work is in part supported by ONR MURI 1800114-1-1120000 the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF-1231216), and IBM Research.
hhtpps://en..iikipdddii
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et el. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp. 4502-4510, 2016. 3
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab, arXiv preprint arXiv:1612.03801, 2016. 2
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
James M Bern, Pol Banzet, Roi Poranne, and Stelian Coros. Trajectory optimization for cable-driven soft robot locomotion. In Robotics: Science and Systems, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016. 1,
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. ICLR, 2016.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and Zico Kolter. End-to-end differentiable physics for learning and control. In Advances in Neural Information Processing Systems, pp. 7178-7189 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. arXiv preprint arXiv:1611.01655 2016. 3
Cosimo Della Santina, Robert K Katzschmann, Antonio Biechi, and Daniela Rus. Dynamic control of soft robots interacting with the environment. In 2018 IEEE International Conference on Soft Robotics (RoboSoft), pp. 46-53. IEEE, 2018. 3
Yan Duan, Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference Machine Learning, pp. 1329-1338, 2016. 2
Tom Erez and Emanuel Todorov. Trajectory optimization for domains with contacts using inverse dynamics. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp- 4914 4919. IEEE, 2012.
Philipp Foehn, Davide Falanga, Naveen Kuppuswamy, Russ Tedrake, and Davide Scaramuzza. Fast trajectory optimization for agile quadrotor maneuvers with cable-suspended payload. Robotics: Science and System, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1582-1591, 2018. 6
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04955 049544, 2020. 3
Ming Gao, Andre Pradhana Tampubolon, Chenfanfu Jiang, and Eftychios Sifakis. An adaptive gen- eralized interpolation material point method for simulating elastoplastic materials. ACM Trans- actions Graphics (TOG), 36(6):1-1 2017. 5, 14
Johan Gaume, T Gast, J Teran, A van Herwijnen, and C Jiang. Dynamic anticrack propagation in snow. Nature communications, 91):1-10, 2018. 9
10

---

Published as a conference paper at ICLR 2021
Moritz Geilinger. David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski. and Stelian Coros. Add: Analytically differentiable dynamics for multi- body systems with frictional contact. arXiv preprint arXiv:2007.00987, 2020.
Thomas George Thuruthel, Yasmin Ansari, Egidio Falotico, and Cecilia Laschi. Control strategies for soft robotic manipulators: A survey. Soft robotics, ((2):149-163, 2018. 3
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. 2017. 6
Eric Heiden, David Millard, Hejia Zhang, and Gaurav S Sukhatme. Interactive differentiable simu- lation. arXiv preprini arXiv: /905. 10706, 2019.
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning to control pdes with differentiable physics. International Conference Learning Representations, 2020. 3
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(4): 1-14, 2018. 2,5
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: language for high-perfrmmance computation on spatially sparse data structures. ACM Transac- tions on Graphics (TOG), 38(6):201, 2019a. 2,
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojeiech Matusik. Chainqueen: A real-time differentiable physical simu- lator for soft robotics. In 2019 International Conference on Robotics and Automation (ICRA), pp- 0265-6271. IEEE, 2019b. 3.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan DegannKelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020. 2, 3,5
Chenfanfu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. The affine particle-in-cell method. ACM Transactions Graphics (TOG), 34(4):1-10, 2015. 4
Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The ma- terial point method for simulating continuum materials. In ACM SIGGRAPH 2016 Courses, pp- 1-52. 2016. 6
Gergely Klár, Theodore Gast, Andre Pradhana, Chuyuan Fu, Craig Schroeder, Chenfanfu Jiang, and Joseph Teran. Drucker-pras Drucker-prager elastoplasticity for sand animation. ACM Trunsactions on Graphics (TOG), 35(4):1-12, 2016. 5
Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2- thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 2017.
Yinxiao Li, Yonghao Yue, Danfei Xu, Eitan Grinspun, and Peter K Allen. Folding deformable objects using predictive simulation and trajectory optimization. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6000-6006. IEEE, 2015. 3
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566. 2018. 3
Junbang Liang, Ming Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse prob- lems. Advances in Neural Information Processing Systems, 2019. 3
Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. Fluid directed rigid body control using deep reinforcement learning. ACM Trans. Graph., 37(4), July 2018. ISSN 0730-0301. doi: 10.1145/1195117.220333 URL https: //cii htt///ii.org0/00/ orr/10..145/ 3197517 3201334.

---

Published as a conference paper at ICLR 2021
Andrew D Marchese, Russ Tedrake, and Daniela Rus. Dynamics and trajectory optimization for soft spatial fluidic elastomer manipulator. The International Journal of Robotics Research, 35(8): 1000-1019, 2016. 3
Jan Matas, Stephen James, and Andrew J Davison. Sim-to-real reinforcement learning for de- formable object manipulation. arXiv preprint arXiv:1806.078) 2018. 9
Aleka McAdams, Andrew Selle, Rasmus Tamstorf, Joseph Teran, and Eftychios Sifakis. Computing the singular value decomposition of 3x3 matrices with minimal branching and elementary floating point operations. Technical report, University of Ausconsin-Madisee Department of Computer Sciences, 2011.6
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural representation for physies prediction. arxiv preprint 1eXiv:806.00844 2018.
PhysX. "PhysX SDK". URL "https: /tttpp::/deeeee http///wwe..ppee.....idoooooooaa
Michael Posa, Cecilia Cantu, and Russ Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, -3(1):69-81, 2014.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 8494-8502, 2018.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.02166. 2020. 3
Md Atiqur Rahman and Yang Wang. Optimizing intersection-ovve--nioo in deep neural networks for image segmentation. In International symposium on visual computing, pp. 234-244. Springer, 2016. 6
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. ICCV, 2019.
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks. arXiv preprint
2018. 3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: ToXivv/707... 170..06347 2017. 6
Koushil Sreenath, Nathan Michael, and Vijay Kumar. Trajectory generation and control of a quadro- tor with cable-suspended load-a differentially- -fla hybrid system. In 2013 IEEE International Conference on Robotics and Automation, pp. 4888 4895. IEEE, 2013. 3
Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation. ACM Transactions on Graphics (TOG), 32(4):1-10, 2013. 5,
Deborah Sulsky, Shi-Jian Zhou, and Howard L. Schreyer. Application of particle-in-cell method to solid mechanics. Computer physics communications, 87(1-2):23--553 1995.
Sarah Tang and Vijay Kumar. Mixed integer quadratic program trajectory generation for a quadro- with a cable-suspended payload. In 2015 IEEE International Conference on Roboties and Automation (ICRA), pp. 2216-2222. IEEE, 2015. 3
Russ Tedrake. Underactuated robotics: Algorithms for walking, running, swimming, flying, and manipulation (course notes for mit 6.832). http://underncuatted.mi..ee 2020. Downloaded on
2020-09-30.7 3
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. 1,3
12

---

Published as a conference paper at ICLR 2021
James Townsend. Differentiating the singular value decomposition. Technical report, Technical Report 2016, https://j-towns.. github. lolpapers/svddddrivatiie ..., 2016.
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations, 2020. 3
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In CVPR, pp. 2068-9079, 2018. 2
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanx- iao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11097-11107, 2020. 2
Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg. Self-supervised learning of state estima- tion for manipulating deformable linear objects. IEEE Robotics and Automation Letters, 5(2): 2372-2379, 2020. 3
13

---

Published as a conference paper at ICLR 2021
A SIMULATOR IMPLEMENTATION DETAILS
von Mises plasticity return mapping pseudo code Here we list the implementation of the for- ward return mapping (Gao et al., 2017). Note the SVD in the beginning leads to gradient issues that need special treatments during backpropagation.
def von Mises aeissesretuun_neeee mappingg (E):
# is the deformation gradient before return mapping
U, sig, V ti. (F)
epsilon -i..i.eee vector ([ti (oggg00) 01), ti. 1.go 1111) epsilon hat = epsilon (epsilon. / 2)
epsilon hat norm = epsilon_hat.norm delta delta_gamma epsilen hat_norm yield_stress stress
hat norm)
mu)
if delta_gamma > 0: # Yields!
opsilon - (dclta_ganma / epsilon_hat_nenn) sig = wtt__maari_rreeee____ cerrom_diage (ti (ti.exp(epsilcn) F U sig e v.transpose return
epsilon_hat
Parameters We use yield stress of 50 for plasticine in all tasks except Rope, where 200 as the yield stress to prevent the rope from fracturing. We use a = 666.7 in the soft contact model.
Parallelism and performance Our parallel mechanism is based on ChainQueen (Hu et 20,,0019)). single MPM substep involves three stages: particle to grid transform (p2g), grid boundary condi- tions (gridop), and grid particle transform (g2p). In each stage. over all particles (p2g) or grids (for gridop and g2p) to do physical computations and ap-ply atomic operators write results back to particles/grids. Gradient computation, which needs reverse the three stages, is automatically done by Diff (Hu et al., 2020).
a parallel for-loop loop
We benchmark simulator's performance each scene in Table 2. Note step of our environ-
ment has 19 MPM substeps.
Env
Forward
Forward Backward
Move
14.26 ms (70 FPS)
35.62 ms (28 FPS)
Tri.Move
17.81 (56 FPS)
41.88 (24 FPS)
Torus
13.77 ms (73 FPS)
35.69 ms (28 FPS)
Rope
15.05 ms (66 FPS)
38.70 ms (26 FPS)
Writer
14,00 ms (71 FPS)
36.04 ms (28 FPS)
Pinch
12.07 ms (83 FPS)
27.03 (37 FPS)
RollingPin
14.14
FPS)
36.45 (27 FPS)
Chopsticks
14.24 ms (70 FPS)
35.68 ms (28 FPS)
Assembly
14.43 ms (69 FPS)
36.51 ms (27 FPS)
Table
14.00 ms (71 FPS)
35.49 ms (28 FPS)
Table 2: Performance on an NVIDIA GTX 1080 Ti GPU, show the average running time for single forward or forward backpropagation step for each scene.
B MORE DETAILS ON THE EVALUATION SUITE
Move The agent uses two spherical manipulators to grasp the plasticine and move it to the target location. Each manipulator has 3 DoFs controlling its position only, resulting in 6D action space.
TripleMove The agent operates three pairs of spherical grippers to relocate three plasticine boxes into the target positions. The action space has dimension of 18. This task is challenging both RL and gradient-basse based methods.
Torus A piece of cubic plasticine is fixed on the ground. In each configuration of the task, generate the target shape by randomly relocating the plasticine and push a torus mold towards it. The agent needs figure out the correct location push down the mold.
14

---

Published as a conference paper at ICLR 2021
Pinch In this task, the agent manipulates rigid sphere to create dents the plasticine box. The target shape is generated by colliding the sphere into the plasticine from random angles. To solve this task, the agent needs to discover the random motion of the sphere.
Assembly spherical piece of plasticine is placed on the ground. The agent first deforms the sphere into target shape and then moves onto block of plasticine. The manipulators are two spheres.
Table This task comes with a plasticine table with four legs. The agent pushes one of the table legs towards a target position using spherical manipulator.
C REINFORCEMENT LEARNING SETUP
We use the open-source implementation of SAC, PPO and TD3 in our environments. We list part of the hyperparameters in Table 3 for SAC, Table for TD3 and Table 4 for PPO. We fix 10,02 10 and C3 = for all environments' reward.
Table 3: SAC Parameters
Table 4: PPO Parameters
Table 5: TD3 Parameters
gamma
0.99
update steps
2048
start timesteps
1000
policy Ir
0.0003
Ir
0.0003
batch size
256
entropy Ir
0.0003
entropy coef
0
gamma
0.99
target update coef
0.0003
value loss coef
0.5
tau
0.005
batch size
256
batch size
32
policy noise
0.2
memory size
1000000
horizon
2500
noise clip
0.5
start steps
1000
D ABLATION STUDY ON YIELD STRESS
To study the effects of yield stress, we run experiments on simple Move configuration (where SAC can solve it well) with different yield stress. We vary the yield stress from 10 to 1000 to generate environments and train SAC on them. Figure plots the agents' performances w.rt. the number of training episodes. The agents achieve higher reward as the yield stress increase, especially the beginning. Agents in high yield stress environments learn faster than those in lower yield stress environments. We attribute this to the smaller plastic deformation in higher yield stress environments. If we train the agents for more iterations, those in environments with yield stress larger than 100 usually converge to same performance level, close solving the problem. However, materials in environments with yield stress smaller than 100 tend to deform plastically, making it hard grasp and move object while not destroying its structure. This demonstrates correlation between yield stress and task difficulty.
300
Rewards
200
20
50
100
100
200
1000
0
2000 Episodes
4000
Figure 5: Rewards w.rt. the number of training episode on 6 environments. Their yield stresses are
10, 20. 50, 100, 200, and 1000.
15

---

Published as a conference paper at ICLR 2021
E DETAILED RESULTS ON ALL CONHIGURATIONS
We run each algorithm with 3 random seeds and each scene has 5 configurations. Therefore, 3 x 15 random seeds in total are used for each algorithm on each scene. We report the normalized incremental IoU scores and the standard deviations for each configuration in Table 6. We also show optimization efficiency of differentiable physics in Figure 6.
Method
SAC
TD3
PPO
Adam
GD
Adam-H
Env
Move-v1
0.19 = 0.26
0.30 - 0.11
0.72 = 0.11
0.97 + 0.01
0.04 = 0.08
0.00 + 0.00
Move-v2
0.20 1: 0.28
0.38 1 0.27
0.60 0.10
0.67 1 0.08
0.67 1 0.02
0.00 1 0.01
Move-v3
0.48 + 0.35
0.29 0.27
0.21
0.97 +.00
0.21 + 0.32
0.00 0.00
Move-v4
0.19 + 0.17
0.33 1:3300.000 0.07
0.79
0.99 0.001
0.73 - 0.30
0.02 0.04
Move-v5
0.29 = 0.06
0.23
0.71
0.92 0.02
0.90 - 0.06
0.22 0.28
Trv.Move-v
0.10 + 0.09
0.00
0.00
0.26 1 0.22
0.05 1 0.08
0.17 0.12
MoveMove-2
0.09 + 0.09
0.00
0.05
0.18 0.09
- 0.00
0.24 0.02
Trv.Move-v3
0.04 + 0.06
0.00
0.05
0.23 + 0.01
0.15 0.08
0.15 + 0.05
MoveMoev-
0.12 0.09
0.00
0.07
0.45 0.45 0.11
0.27 0.00 0.00
0.23 0.03
M5
0.23 = 0.09
0.02 0.03
0.24
0.62 + 0.01
0.49
0.52 0.02
Torus-vl
0.48 = 0.28
0.58
0.36
0.90 + 0.00
0.91 0.01
0.88 0.03
Torus-v2
0.37 = 0.35
0.00
0.08
0.30
0.07 = 0.05
0.39 + 0.10
Torus-v3
0.40 1 0.49
0.67 0.47
0.20
1.00 0.000
1.00
1.00 0.000
Torus-v4
0.78 + 0.32
0.55 0.41
0.66+0441
1.00 +.00+00001
+.00
0.64 0.17
Torus-v5
0.63 + 0.45
0.29
0.43 0.39
0.67 0.47
1.00
0.70 0.08
Ropc-v1
0.29 = 0.16
0.10
0.29 0.29:0009
0.66 + 0.01
0.55 + 0.06
0.20 + 0.08
Rope-v2
0.33 + 0.22
0.18 0.19
0.53 0.05
0.71 1: 0.02
0.40
0.20 1 0.01
Rope-v3
0.35 + 0.06
0.20
0.52 - 0.06
0.43
0.34 0.06
Rope-v4
0.09 +: 0.07
0.00
0.18 0.04
0.37 0.00
0.42
0.13 4 0.01
Rope-v5
0.38 = 0.27
0.10
0.08
0.70 0.05
0.25
0.18 0.03
Writer-vl
0.46 + 0.02
0.27
0.60 0.10
0.78 + 0.04
1.00 + 0.00
0.00 + 0.00
Writer-v2
0.75 = 0.10
0.24
0.18
0.16
0.98 - 0.03
0.00 + 0.00
Writer-v3
0.39 = 0.03
0.34 0.08
0.08
0.81 0.09
0.63
0.00 + 0.00
Writer-v4
0.15 1 0.12
0.02
0.28 0.06
1.00
0.00 1 0.00
Writer-v5
0.31 + 0.22
0.12
0.07 0.38
0.35 + 0.03
0.34 0.06
0.00 0.00
Pinch-v1
0.00 + 0.00
0.00
0.01 0.02
0.16 1 0.01
0.11 + 0.04
0.02 1 0.03
Pinch-v2
0.16 + 0.10
0.02
0.16 0.13
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
Pinch-v3
0.06 + 0.05
0.00
0.03 0.04
0.09 1 0.02
0.02 1 0.03
0.00 + 0.00
Pinch-v4
0.01 + 0.02
0.01
0.00 0.00
0.15 1 0.09
0.00 + 0.00
0.00 + 0.00
Pinch-v5
0.03 -: 0.03
0.00
0.10 0.07
0.02 1 0.00
0.00 + 0.00
0.00 0.01
RollingPin-vl
0.47 +: 0.26
0.10
0.83 0.04
0.91 1 0.04
0.89 0.11
0.22 1 0.07
RollingPin-v2
0.34 + 0.22
0.09
0.74 0.10
0.92 + 0.02
0.87 + 0.04
0.23 + 0.09
RollingPin-v3
0.28 + 0.34
0.12 0.01
0.96 0.05
0.95 - 0.03
0.82 - 0.18
0.36 + 0.03
RollingPin-v4
0.42 0.30
0.87 - 0.07
0.96 + 0.03
0.88 + 0.07
0.35 + 0.15
RollingPin-vS
0.27 1 0.29
0.12 0.00
0.90 0.05
0.91 1 0.01
0.98 1 0.02
0.16 1 0.08
Chopsticks-vl
0.12 + 0.04
0.08 + 0.06
0.12 0.06
0.92 + 0.02
0.04 + 0.05
0.04 + 0.03
Chopsticks-v2
0.23 1: 0.11
0.22 1 0.09
0.22 1 0.11
0.98 1 0.02
0.05 1 0.05
0.08 1 0.11
Chopsticks-v3
0.13 + 0.03
0.09 0.01
0.14 0.08
0.88 + 0.04
0.00 + 0.00
0.00 + 0.00
Chopsticks-v4
0.05 + 0.03
0.08 + 0.03 0.03
0.13 - 0.07
0.81 + 0.07
0.02 - 0.02
0.00 +: 0.00
Chopsticks-v5
0.13 - 0.03
0.09 = 0.04
0.10 0.08
0.81 + 0.03
0.03 - 0.03
0.00 + 0.00
Assembly-v1
0.00 + 0.00
0.00
0.00 = 0.00
0.84 + 0.03
0.55 0.400
0.05 0.03
Assembly-v2
0.00 +: 0.00
0.00 0.00
0.01 - 0.00
0.93 1 0.02
1.34
0.03 1: 0.03
Assembly-v3
0.00 + 0.00
0.00
0.26 + 0.31
0.83 + 0.18
0.01 + 0.02
0.02 + 0.02
Assembly-v4
0.00 = 0.01
0.00
0.04 0.06
0.94 0.02
0.02
0.20 + 0.27
0.02 + 0.02
Assembly-v5
0.00 = 0.00
1.0000000
0.00 = 0.00
0.94 + 0.03
0.24 0.30
0.01 + 0.00
Table-v1
0.01 + 0.04
0.05 0.11
0.57 0.26
0.00 + 0.00
0.00 - 0.00
0.00 + 0.00
Table-v2
0.01 + 0.02
0.10
0.14
0.00
+ 0.00
0.00 = 0.00
0.00 + 0.00
Table-v3
0.14 1 0.19
0.14 0.15
0.33 25
0.02 1 0.00
0.00
1 0.00
0.02 1 0.01
Table-v4
0.05 + 0.13
0.19
0.34 + 0.24
0.00
+ 0.00
0.00 0.00
0.00 + 0.00
Table-v5
0.01 + 0.01
0.02 - 0.04
0.04 0.03
0.01 1 0.00
0.00 0.00
0.00 +: 0.00
Table 6: The normalized incremental IoU scores of each method on each configuration. We run each experiment with 3 random seeds and report their averages and standard deviations (mean + std)
16

---

Published as a conference paper at ICLR 2021
Move v1
Nove x2
Move-va v3
Move v4
Move
431
Rewar
800
Rewart
Reward
200
Rewards
Episodes
Episodes
poissdds
pisooles
Upisodes
TripleRove-v1
TripleMove-??
Tripleoveeu
Tr Kinvevv
TripleMnve--5
200
Ro
120
100
Epicodes
Episodes
Eo lo0
Episodies
Ep 20:
Torus v1
rus v2
Torus v3
Torus v4
Tonus V5
Rowarrss
Rewares
Rewerrs
Episodess
Epiindess
Ep lo
priccdes
Roppeiyl
Rope-12
Ropauu3
Rope-od
Rope-es
Rewands
Rewands
Rowards
300
100
100
160
2006
1.90
Epicodes
Epicodes
Epico des
Episocies
Ep opeso
Records
Rewards
Rewards
Rewards
150
100 Episodes
100
150 Epriacdes Pinch-v5
Episodes
Epiioo
Episocees
Pinch-v3
Pinch-ed
Rowads
Rewards
Rowands
Rewnds
6o
100 Epicodes
100
200
100 Episodes
153
300
53
100 Episo
150
200
160 Eprsodies
3:00
200
300 Episodes
1.00
200
Rc
RollingPini22
RollinPiniv3
RollingP-vvl
Rollingfi-v5
100
Rewards
Rewards
Rewards
50
100
350
200
50
100
150
200
50
300
250
200
50
300
250
200
200 Chopsticks-v5
150
200
Chapsticks-v1
Chepsticks-12
Chopsticks.v3
Chopsticks.vt
Rowards
Rewards
800
Rowards
200
100
150
Episodes
Episodes
Episodes
As
Assembly-v2
Assemblyy-v3
Assembly-v4
Assembly-v3
Rewordss
Bowwars
Rewards
200
Episodes
Episodes
Ep des
Episidess
Eprisdess
Table-u1
Tablevv2
Table-v3
Table v4
Table-v5
Rowsnds
Rowards
Rewards
Epicodes
Episodes
Ep lolloddess des
Episodies
Ep co:sodees
Figure 6: Rewards and variances each configuration w.r.t the number of episodes spent on training. We clamp the reward be greater than 0 for better illustration.
17

---

Published as a conference paper at ICLR 2021
F EXPLANATION OF SOFT IOU
Let a, be two numbers and can only be or 1, then ab = 1 and only if = 1; if and only if least one of a, 6 is 1. If the two mass tensors only contain value of 0 or Σ S2S2 equals the number of grids that have value 1 in both tensors, i.e., the "Intersection". For the same reason 5 S2 S1S² counts the grids that S1 Σ / +1+ - S1S₂ computes the standard Intersection over Union (IoU). In our case, we assume the normalized mass tensor S₁ and Sn are two tensors with positive values. Therefore, first normalize them so that their values are between 0 and 1, then apply the previous formula to compute a "soft loU,"" " which approximately describes if 3D shapes match.
b- ab
S 1, the "Union". Thus, the formula
18


Reviews:
[{"review_id": "nWo-xjNqkd7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "PlasticineLab\n\nThe paper presents a new soft-body manipulation benchmark for RL and differentiable planning.\nThe presented simulation suite is very interesting and the contribution is solid.\n\nStrength:\n- new simulation benchmark with features that are not yet well explored\n- differentiable physics to open up possibilities for planning methods\n- tasks are difficult enough to be challenging for a while\n- baseline results are provided\n\nWeaknesses:\n- only the computation times would be good to add\n\nPresentation:\nThe paper is clearly written and easy to follow.\n\nWays to improve the paper:\n- wall-clock times would really be very useful. Both for the forward pass as well as a backward pass through the entire horizon with Adam. Maybe also some notes on how it can be parallelized since you have a CUDA implementation.\n\n\nDetails:\n- p5 last paragraph: \"For any grid points with a signed distance d..:\" The formulation is not clear enough. \n Do you mean with positive signed distance. Prob not, because you can also have penetration. But why would a point then not have a distance to the rigid body?  \n- same paragraph: \"By definition, s decays exponentially with d until d becomes negative (when penetration occurs)\" Well it decays with increasing distance (and then it cannot become negative if it increases...)\n- 5.1: IoU definition: Are S always positive? I don't exactly understand what the mass tensor S is. I know it from rigid body dynamics, but this does not seem to be the same here. Can you clarify this better such that it becomes clear why the formula describes and IoU.\n- Fig 4. Consider removing the grey background that seaborn uses automatically. The plots will look much cleaner and better visible. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kkaV_yTUCS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new open-source simulation benchmark for soft robotics. The simulation environment builds on top of DiffTaichi, an existing differentiable simulator which enables end-to-end differentiability. The paper proposes 10 different tasks, each with 5 variations and evaluates both RL-based policy learning methods and gradient-based optimization methods on those tasks. The results suggests neither current RL-based methods nor gradient-based method can solve most of the tasks efficiently, especially for those require long-term planning.\n\nOverall the paper is well-written and the contribution is well-argued. I have a few comments / questions as follows:\n- The simulator only considers the state of the end-effector of the manipulator. It would be great to consider higher DOF soft manipulators (e.g. [1][2]) which would further benefit the soft robotics community.\n- Given the randomness and the nature of the RL algorithms [3], the evaluation in section 5.2 should be done with at least multiple random seeds with multiple trials per seed to make the benchmarking results  statistically significant.\n- Similarly for Section3, In Table 1, it would be great to see the standard deviation in addition to the average value. It's also not clear how many trials were conducted in order to get the numbers shown in both Table 1 and the plots in Fig. 4.\n\n[1] Della Santina, Cosimo, et al. \"Dynamic control of soft robots interacting with the environment.\" 2018 IEEE International Conference on Soft Robotics (RoboSoft). IEEE, 2018.\n\n[2] George Thuruthel, Thomas, et al. \"Control strategies for soft robotic manipulators: A survey.\" Soft robotics 5.2 (2018): 149-163.\n\n[3] Khimya Khetarpal, Zafarali Ahmed, Andre Cianflone, Riashat Islam, Joelle Pineau. Reproducibility in Machine Learning Workshop, ICML 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "YfLQq9D19dA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "####  Summary\n\nIn this work, the authors present PlasticineLab, a new framework for soft-body manipulation tasks for Reinforcement Learning and planning algorithms. The environment consists of a novel soft (i.e., deformable, plastic) material termed Plasticine which is complex to model and manipulate because of the inherently complex high-dimensional governing equations and the large number of degrees of freedom associated with soft materials. The PlasticineLab framework proposes 10 novel tasks involving manipulation of the soft plasticine material. The authors show thorough empirical analysis that traditional state of the art model-free reinforcement learning algorithms fail to effectively learn the task even after a substantial amount of training. Thus effectively showcasing the complexity of the proposed tasks and the inability of state of the art RL models to model the proposed tasks.\n\n#### Positives:\n\n1. Novel tasks: Each task poses a different challenge: E.g., some tasks involve flattening the plasticine, other involve pinching the plasticine while yet other tasks involve grasping one or multiple plasticine objects (at one or multiple points) and deforming it or moving it in some required manner. \n\n2. The variety of tasks test various facets of RL like long-term planning especially in the case of multi stage tasks.\n\n3. Another major effort prelavent in the paper is that the authors have chosen to use a differentiable physics engine using the DiffTaichi system thereby making the gradients available for planning and control algorithms. \n\n4. The paper highlights through empirical results, the superiority of gradient-based approaches (over model-free RL approaches) that leverage the underlying differentiable physics engine toward learning the required tasks.\n\n5. An important facet of a benchmark is to propose tasks that are sufficiently complex for the current state of the art procedures. The authors employ 3 state of the art model-free RL algorithms and show that these RL models perform poorly in a majority of the 10 tasks. Torus, RollingPin, Move tasks are the only three tasks where the model-free procedures are able to perform somewhat comparably with the gradient-based planning approaches which themselves perform well in all but the Writer, Pinch and TripleMove tasks. The last task involves multi-object manipulation and requries long-term planning and hence  \n\n#### Concerns:\n\n1. In figure 4. the Adam optimizer and the GD which both seem to consistently accumulate the greatest rewards also have high variance, some commentary about how this can be explained would help the reader better contextualize the results.\n\n\n2. As one of the main claims of the paper is the challenge of soft-body manipulation and the proposal of a framework for the same, it is imperative to demonstrate the variation of the degree of difficulty with increase (or decrease) in rigidity of the materials being manipulated. A comparative analysis such as this, demonstrating for example the variation in IOU error of the best performing RL model with increasing in yield stress for plasticine would serve to showcase the actual challenge posed by soft-body material mainpulation in the context of the current proposed framework. Ofcourse since decreasing softness and increasing rigidity is most likely not as simple as increasing a single number such as yield stress, this is a minor concern and more a suggestion toward a holistic analysis of the proposed framework.\n\n\n#### Minor Details & Suggestions:\n\n1. Extrapolation is an important facet of learning algorithms in general. Since one of the suggestions of the current work is to present the PlasticineLab framework as a way to not only characterize RL and gradient-based algorithms but also combine these two families of methods, it is also important to evaluate the performance of these models on unseen but related tasks e.g., manipulating a table with fewer or grater number of legs, trying to place more than 3 objects at specified locations.\n\n2. The citation relating to the paper by Avila et al. titled End-to-end differentiable physics for learning and control published in the Advances in Neural Information Processing Systems conference in 2018 seems to be repeated. \n\n3. Another potential direction of the current framework could be using PlasticineLab to learn policies which might be transferred to the real-world (similar to the task mentioned in [1]). If feasible, adding some brief commentary about this in the context of [1] might open up further avenues of exploration for plasticinelab.\n\n#### References:\n\n1. Matas J, James S, Davison AJ. Sim-to-real reinforcement learning for deformable object manipulation. arXiv preprint arXiv:1806.07851. 2018 Jun 20.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WavSMaWfnFG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper presents PlasticineLab, a differentiable physics environment geared towards softbody manipulation. By implementing a softened rigid-deformable contact interface, and by leveraging recent advances in softbody dynamics simulation (DiffTaichi, ChainQueen), PlasticineLab is able to provide analytical gradients which seem to outperform gradient-estimation based approaches (SAC, TD3, PPO) on few tasks.\n\n\n## Strengths\n\n**S1** The central problem tackled here is quite interesting (and challenging)! There is a growing interest in the ML community wrt differentiable simulation techniques and in particular, their applicability to learning dynamics.\n\n**S2** The paper is extremely well-written and easy to follow. While this builds heavily on DiffTaichi and ChainQueen, it is commendable that this paper came across as self-contained.\n\n**S3** I believe the characterization of related work is fair. An interesting point made here was that TDW and SAPIEN do not provide assets for soft-body simulation. I am unsure if that's entirely true, but the lack of assets is indeed an issue for softbody simulation. It will be a welcome contribution if this paper were to make these 3D assets publicly available.\n\n**S4** Conceptually, this paper claims that inductive biases (arising from simulation of deformable objects) should be exploited wherever possible. In particular Fig. 4 and Table 1 seem to indicate that gradient-based optimization (using differentiable simulation) consistently outperforms RL techniques in 8/10 tasks. To one's anticipation, gradient-based optimization seems to achieve significantly faster convergence (seems like two orders of magnitude), which is an impressive feat.\n\n**S5** It is interesting to see that it is possible to differentiate through contacts across rigid and deformable objects. To my knowledge, this has not been demonstrated before (has been tangentially discussed in [C]) and is a significant contribution.\n\n\n## Weaknesses\n\n**W1** The paper could benefit from an explicit exposition of critical design choices that affect differentiability. While PlasticineLab uses a particle-based model for representing and simulating soft-bodies, alternatives in the form of (usually tetrahedral) mesh-based representations exist. It appears that particle systems are chosen to enable trivial differentiability (e.g. the material-point method in the absense of contact forces is analytically differentiable)\n\n**W2** An important detail which I couldn't find in the paper and/or supplementary material is how many of the parameters are simultaneously observable. For example, if the masses of the particles and the manipulator contact parameters are both unspecified, wouldn't this lead to problems in observability (i.e., both quantities cannot be simultaneously solved for, resulting in ill-behaved gradients)?\n\n**W3** Does the approach assume a one-one correspondence between the predicted and target shape? While this might seem a reasonable assumption, I believe gradient computation is cumbersome (and perhaps ambiguous?) were this to be relaxed?\n\n**W4** Another crucial detail that the paper does not seem to get through. As with most other \"differentiable physics\" approaches, unmodelled effects in the dynamical system might limit the applicability of the system. Since the physics engine only implements forces and softbody dynamics that are \"predetermined\", I would imagine it is hard to emulate real-world effects such as wear-and-tear, sophisticated contacts, and material properties. In favor of the paper though, I feel this detail might also be out of scope to an extent. (Recent approaches such as Neural dynamical systems [A] and Learning physical constraints by neural projections [B] come to mind, to handle some of these concerns).\n\n\n## Summary\n\nWhile the differentiable simulation aspect of the paper is not substantially novel (building atop DiffTaichi, ChainQueen, and soft contact models - Stomakhin et al. 2013), the overall system is impressive and addresses a gap in the differentiable physics community. Simulating differentiable softbody dynamics, as well as interaction with (a limited class of) rigid bodies could open up interesting avenues in reinforcement learning and softbody manipulation.\n\n[A] Neural dynamical systems: balancing structure and flexibility in physical prediction. arXiv 2020\n\n[B] Learning physical constraints with neural projections. arXiv 2020\n\n[C] Scalable differentiable physics for learning and control. ICML 2020\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new differentiable physics benchmark for soft-body manipulation. The proposed benchmark is based on the  DiffTaichi system. Several existing reinforcement learning algorithms are evaluated on this benchmark. The paper identify a set of key challenges that are posed by this specific benchmark to RL algorithms. Short horizon tasks are shown to be feasible by optimizing the physics parameters via gradient descent. The reviewers agree that this paper is very well-written, the  problem tackled in it is quite interesting and challenging, and the use of differentiable physics in RL for manipulating soft objects quite intriguing.
Decision: Accept (Spotlight)