DaxBench: Benchmarking Deformable Object Manipulation with Differentiable Physics
Authors: Siwei Chen
Yiqing Xu
Cunjun Yu
Linfeng Li
Xiao Ma
Zhongwen Xu
David Hsu
Venue: ICLR 2023 notable top 5% (2023)
Keywords: deformable object manipulation
differentiable physics
benchmark

Abstract: Deformable object manipulation (DOM) is a long-standing challenge in robotics and has attracted significant interest recently. This paper presents DaXBench, a differentiable simulation framework for DOM. While existing work often focuses on a specific type of deformable objects, DaXBench supports fluid, rope, cloth ...; it provides a general-purpose benchmark to evaluate widely different DOM methods, including planning, imitation learning, and reinforcement learning. DaXBench combines recent advances in deformable object simulation with JAX, a high-performance computational framework. All DOM tasks in DaXBench are wrapped with the OpenAI Gym API for easy integration with DOM algorithms. We hope that DaXBench provides to the research community a comprehensive, standardized benchmark and a valuable tool to support the development and evaluation of new DOM methods. The code and video are available online.

Published as a conference paper at ICLR 2023
DAXBENCH: BENCHMARKING DEFORMABLE OBJECT MANIPULATION WITH DIFFERENTIABLE PHYSICS
Siwei Chen', Yiqing Xu't, Cunjun Yu Linfeng Li', Xiao Ma², Zhongwen Xu², David Hsu
National University of Singapore Sea Al Lab?
ABSTRACT
Deformable object manipulation (DOM) long-standing challenge in robotics and has attracted significant interest recently. This paper presents DaXBench, a differentiable simulation framework for DOM. While existing work often fo- cuses a specific type of deformable objects, DaXBench supports fluid, rope. cloth ...: it provides a general-purpost benchmark to evaluate widely differ- ent DOM methods, including planning. imitation learning, and reinforcement learning. DaXBench combines recent advances in deformable object simula- tion with JAX, high-performance computational framework. All DOM tasks in DaXBench are wrapped with the OpenAl Gym API for easy integration with DOM algorithms. We hope that DaXBench provides the research community a comprehensive, standardized benchmark and valuable tool to support the devel- opment and evaluation of new DOM methods. The code and video are available online".
INTRODUCTION
Deformable object manipulation (DOM) is crucial area of research with broad applications, from household Maiiin-Shhppaa et al. 2010 Miller et al.) [201 Ma et al. 2022) to industrial set- tings (Miller et 2011 Zhu et al. 2022). To aid in algorithm development and prototyping, several
DOM benchmarks (Lin et al. 202 Huang et al., (022)) have been developed using deformable ob-
ject simulators. However, the high dimensional state and action spaces remain a significant challenge to DOM.
Differentiable physics is a promising direction for developing control policies for deformable ob- jects. It implements physical laws as differentiable computational graphs (Freeman et al. 2021] Hu (ct al. 2020), enabling the optimization of control policies with analytical gradients and therefore improving sample efficiency. Recent studies have shown that differentiable physics-based DOM methods can benefit greatly from this approach (Huang et al. 2021 Heiden et al.. 2021 Xu et al,,
2022 Chen et al. 2023). To facilitate fair comparison and further advancement of DOM techniques, it is important to develop a general-purpose simulation platform that accommodates these methods.
We present DaXBench, differentiable simulation framework for deformable object manipulation (DOM). In contrast with current single-task benchmarks, DaXBench encompasses diverse range of object types, including rope, cloth, liquid, and elasto-plastic materials. The platform includes tasks with varying levels of difficulty and well-defined reward functions, such as liquid pouring, cloth folding, and rope wiping. These tasks are designed support high-level macro actions and low-level controls, enabling comprehensive evaluation of DOM algorithms with different action spaces.
We benchmark cight competitive DOM methods across different algorithmic paradigms, includ- ing sampling-based planning. reinforcement learning (RL), and imitation learning (IL.). For plan- ning methods, we consider model predictive control with the Cross Entropy Method (CEM-MPC) Richardss] 2005), differentiable model predictive control (Hu et al. 2020), and a combination of
These authors contributed equally.
*This work is partially completed at the SEA Al Lab.
*The link of the project is https: / / thub. , com /Aaaacooppu////aaaeann.

---

Published as a conference paper at ICLR 2023
the two. For RL domains, we consider Proximal Policy Optimization (PPO) (Schulman et al. 2017) with non-differentiable dynamics, and Sbort-Horizoo CritrrCritic (SHAC) (Xu et al. 2022 and An- alytic Policy Gradients (APG) (Freeman et al. 2021) with analytical gradients. For IL, Transporter networks (Seita et al. 2021) with non-differentiable dynamics and Imitation Learning via Differen- tiable physics (IID) Chen al. 2023) compared. Our experiments compare algorithms with and without analytic gradients on each task, providing insights into the benefits and challenges of iiterrettible--pyssis----eee DOM methods.
DaXBench provides deformable object simulator, DaX, which combines recent advances in de- formable object simulation algorithmss (Xu et al. computational framework JAX (Bradbury et al. 2018). This integration allows for efficient auto- differentiation and parallelization across multiple accelerators, such as multiple GPUs. All task environments are wrapped with the OpenAl Gym API (Brockman et al. 2016), enabling seamless integration with DOM algorithms for fast and easy development. By providing a comprehensive and standardized simulation framework, we aim to facilitate algorithm development and advance the state of the art in DOM. Moreover, our experimental results show that the dynamics model in DaXBench enables direct sim-to-real transfer to a real robot for rope manipulation, indicating the potential applicability of our simulation platform to real-world problems.
2022 Chen et al. [2023) with the nigh-performance
2
RELATED WORKS
2.1 DEFORMABLE OBJECT SIMULATORS
Recent advancements in Deformable Object Manipulation (DOM) have been partially attributed to the emergence of DOM simulators in recent years. SoftGym (Lin al., 2021) the first DOM benchmark that models all liquid, fabric, and elastoplastic objects and introduces wide range of standardized DOM tasks. ThreeDWorld (Gan et al. 2020) enables agents to physically interact with various objects in rich 3D environments, but it is non-differentiabll and does not support methods based on differentiable physics.
On the other hand, several differentiable DOM simulators emerge, such as ChainQueen (Hu et al. 2019, Diff-PDE (Holl et al.] 2020), PlasticineLab (Huang et al., 2021), DiSECt (Heiden et al, 202TI, and DiffSim (Qiao et al. (020). However, each simulator specializes in modeling a single type of deformable object and supports only a limited range of object-specific tasks. For example, PlasticineLab (Huang et al. 2021) and DiSECt (Heiden et al.) 2021) model only the elasto-plastic objects, including tasks such as sculpting, rolling and cutting the deformable objects. Since the dynamics of each deformable object are vastly different, the physic engine specialized in modeling one type of deformable object cannot be easily extended to another. DaXBench bridges this gap by offering general-purpose simulation framework that covers wide range of deformable objects and the relevant tasks. We provide a fairground for comparing and developing all types of DOM methods, especially the differentiable ones.
2.2 DEFORMABLE OBJECT MANIPULATION ALGORITHMS
These challenges make difficult apply existing methods for rigid object manipulation directly to DOM. Methods designed for DOM must be able to handle a large number of degrees of freedom in the state space and the complexity of the dynamics. Despite these challenges, many interesting approaches have been proposed for DOM. Depending on the algorithmic paradigms, we categorize DOM methods into three groups: planning, Imitation Learning (IL), and Reinforcement Learning (RL). Our discussion below focuses on the general DOM methods for a standardized set of tasks. We refer to (Sanchez et al., 2018 Khalil & Payeur 2010) for a more detailed survey on prior methods for robot manipulation of deformable objects.
Reinforcement Learning algorithms using differentiable physics such as SHAC (Xu et al. 2022) and APG (Freeman et al. 2021) are also explored. These methods use the ground-truth gradients on the dynamics to optimize the local actions directly, thereby bypassing the massive sampling for the policy gradient estimation.
Imitation Learning is another important learning paradigm for DOM. To overcome the "curse of history", the existing works such as (Seita al. 2021. Sundaresan et al. 2020 Ganapathi et al.)
2

---

Published as a conference paper at ICLR 2023
Application
Parallelization
Differen- tiability
Engine
Elasto-
of simulations
Multi-Device
Liquid
Fabric
Mixture
plastic
per process
Training
PlasticineLab
x
x
V
Single
x
DiSECt x
x
/
x
V
Single
x
DiftSim
x
v
x
x
V
Single
x
SoftGym
/
x
X
Single
x
DaXBench
V
V
/
Multiple
Table 1: Comparison among the simulators for Deformable Object Manipulation.
2021
overcome the enormous state/action space by learning the low dimensional latent space and simplifying the primitive actions to only pick-and-place. ILD (Chen et al. (023) is another line of IL. method that utilizes the differentiable dynamies to match the entire trajectory all at once; therefore, it reduces the state coverage requirement by the prior works.
Motion planning for DOM has been explored in (Wi et al. 2022 Shen al. 2022 Lippi et al,
2020 Nair& & Finn 2019 Yan et al. 2021 Ma et al. 2022) These methods overcome the enormous DoHs the original state space by learning a low oiw-dimesiional latent state representation and a cor- responding dynamic model for planning. The success of these planning methods critically depends on the dimensionality and quality the learned latent space and the dynamic model, which in itself
challenging.
We believe that the development of a general-purpose purpose simulation framework with a differentiable simulator provide a platform for comparing existing DOM methods. Such framework enables researchers to gain a better understanding of the progress and limitations of current DOM methods, which in turn provide valuable insights for advancing DOM methods across all paradigms.
3 DAXBENCH: DEFORMABLEE-ND-JAN BENCHMARK
DaXBench is ceneral-purpont differentiable DOM simulation platform that covers wide range of deformable objects and manipulation tasks. To facilitate benchmarking of DOM methods from all paradigms, particularly differentiable methods, we developed our own differentiable simulator, Deformable-and-IA (DaX), which is both highly efficient and parallelizable. In this section, we first provide a brief overview our simulator, DaX, and then focus on introducing the benchmark tasks we have implemented for DaXBench.
3.1 DAX
DaX differentiable simulator designed to model various types of deformable object manipulation with high parallelization capa- bilities. comparison with other existing simulators can be found Table DaX is implemented using JAX (Bradbury et al. (2018). which supports highly parallelized computation with excellent au- tomatic differentiation directly optimized the GPU kernel level. Moreover, DaX can be seamlessly integrated with the numerous learning algorithms implemented using JAX, allowing for the en- tire computation graph to be highly parallelized end-to-end.
An example of bowl of soup with tofu and vegetables. All materials including the soup are simulated by DaX using par- ticles. Moreover, the entire sim- ulation is fully differentiable and highly parallelizable.
Figure 1:
State Representations and Dynamic Models DaX adopts differ- ent state representations and dynamic systems to model different types of deformable objects with distinct underlying deformation and dynamics. To model liquid and rope, which undergo significant their states using particles and uses Moving Least Square Material Point Method (MLS-MPM) (Hu et al. 2018 Sulsky et al., [1994 to model their dynamics the particle level. For cloth, which only undergoes limited deformation, DaX models state using mesh and uses a less computationally expensive mass-spring system model its dynamics. These modeling choices allow DaX to strike
deformation. DaX represents
3

---

Published as a conference paper at ICLR 2023
(a) Pour Soup.
(b) Push Rope.
(c) Whip Rope.
(d) Fold T-shirt.
(e) Unfold Cloth.
Figure 2: We illustrate different deformable object manipulation tasks in this work. A template is
provided to readily expand this set to many other similar tasks.
a balance between modeling capacity and complexity. For more information on the object models used in DaX please refer to Appendix section A.]
Designed for Customization DaX allows full customization of new tasks based on the existing framework. Objects with arbitrary shapes and different rigidness can be added to the simulator with case. Morcover, primitive actions can also be customized. For a code example how to customize environment in DaX, please refer to Appendix section A.2
Saving Memory DaX uses two tricks to optimize the efficiency and memory consumption of the end-to-end gradient computation, namely lazy dynamic update and "'heckpointing method" (Qiao et al., 2021: Chen et al. 2016 Griewank & Walther, (000). To optimize the time and memory consumption of MPM for single timestep, DaX lazily updates the dynamic of a small region affected by the manipulation at that step. Moreover, DaX only stores the values of few states (i.e.. checkpoints) during the forward pass. During the backward propagation, DaX re-computes segments of the forward values starting from every stored state in reverse order sequentially for the overall gradient. For more details on these techniques, please refer to Appendix section A.4
3.2 BENCHMARK TASKS
DaXBench provides comprehensive set of Deformable Object Manipulation tasks, including liq- uid pouring, rope wiping, cloth folding, and sculpting elastoplastic objects, as illustrated in Figure 2 These tasks are representative of the common deformable objects encountered in daily life. Ad- ditionally, we propose several new challenging long- horizon tasks that require finer-grained control,
allowing for better benchmarking of the scalability of DOM methods. The horizon of each task is determined by how many steps human expert takes to finish the task. DaXBench provides standardized platform for comparing existing methods and future work on DOM. this section, we provide detailed information on the tasks supported by DaXBench.
Long Horizon Tasks The time complexity for the search-based planning method grows exponen- tially with the planning horizon, known as the "curse of history"; this is exacerbated by the large state space deformable objects. However, the existing DOM benchmark tasks typically have rela- tively short horizons. For example, the cloth folding and unfolding tasks in SoftGym consist of only 1-3 steps. DaXBench addresses this limitation by providing tasks with longer horizons, enabling better evaluation of the scalability of search-based planning methods.
Tasks without Macro-Actions macro-action can be seen as executing raw actions for multiple steps. For example, vector (x, y, 2, x ,'','' z') representing the start (æ, z) and end (,',v/==)) positions of the gripper, and the raw actions Cartesian velocities of the gripper. Macro-actions reduce the dimension of the action space and the effective horizon, and hence make scaling easier for the DOM methods. However, macro-actions might be too coarse for some tasks. For example, when whipping a rope to hit target location, the momentum needs to be adjusted at relatively high frequency throughout the execution, making macro-actions unsuitable. DaXBench includes tasks that do not have suitable macro-actions to provide a more comprehensive evaluation of DOM methods.
define pick-and-plaa
push macro-action
6-dimensional real
3.2.1 LIQUID MANIPULATION
Pour- Water Pour bowl of water into the target bowl as quickly as possible. The agent directly controls the velocity and rotation (in Cartesian space) of the gripper that holds the bowl. In particular,
4

---

Published as a conference paper at ICLR 2023
the action space of the agent is 4-dimensional vector (Uzz Uy: U2, w), for which (ez, By: Uz) represents the linear velocity of the gripper, and w denotes the angular velocity around the wrist.
Pour-Soup Pour bowl of soup with various solid ingredients into the target bowl fast as possible. The actions are the same as those of Pour-Water, This task is more challenging than Pour- Water due to the additional interactions between the liquid and the solid ingredients. In particular, the agent needs to promptly react when a solid ingredient transferred, so as to 1) adjust for the sudden decrease in load the gripper and 2) control the momentum of the ingredient when hitting the soup surface to minimize spilling.
Neither tasks have suitable high-level macro-actions and the task horizons 100 steps.
3.2.2 ROPE MANIPULATION
Push-Rope Push the rope to the pre-specified configuration as quickly as possible. We randomly initialize the configuration of the rope. The agent uses a single gripper to execute the push macro- actions. The horizon for this task is steps.
Whip-Rope Whip the rope into target configuration. The agent holds one end of the rope and controls the gripper's velocity to whip the rope. In particular, the action space of this task is a 3- dimensional vector = (Uas Uy, U2) that denotes the velocity of the gripper in the Cartesian space. This task cannot be completed by the abstract macro-actions since the momentum and the deforma- tion of the rope has to be adjusted at relatively high frequency. The task horizon is 70 steps.
3.2.3 CLOTH MANIPULATION
Fold-Cloth Fold piece of flattened cloth and move it to target location. This task has two variants: the easy version requires the agent to execute fold and the difficult version 3 folds. For both variants, the task begins with the cloth lying flat on the table an arbitrary position. The agent executes pick-and-place macro-actions. The task horizon is 3 and 4 for the easy and difficult versions respectively.
Fold-T-shin --s-r- Fold a T-shirt to a target location. The task begins with the T-shirt lying flat on the table at an arbitrary position. The agent executes pick-and-plan macro-actions. The task horizon 4.
Unfold-Cloth Flatten a piece of folded cloth to a target location as fast as possible. This task also has two variants that differ in the initial configurations of the folded cloth: the cloth in the easy version has been folded once and that in the difficult version has been folded three times. This task has the same action space as the Fold-Clon task. The task horizon is 10 steps for both variants.
All of these tasks share similar APIS to the OpenAI Gym's standard environments. We provide general template to define tasks in our simulator. This not only better organizes the existing task environments but also allows the users to customize the existing task environments to better suit their research requirements. The variables and constants defined in the template easily interpretable quantities that correspond to real physical semantics. This allows the users to better understand the environment, which can also help with the development and debugging process.
EXPERIMENTS
In this section, we aim to answer the following two questions: 1) Are the existing representative methods capable of solving the tasks in DaXBench? 2) Within each paradigm, will the diffcrentiable simulator help in completing the tasks?
To investigate these two questions, we benchmark eight representative methods of deformable object manipulation, covering planning, imitation learning, and reinforcement learning. Each paradigm has different assumptions and requirements for input knowledge. We summarize the baseline methods and the relevant information in Table
5

---

Published as a conference paper at ICLR 2023
Paradigmss
Reinforcement Learning
Inuiation Learning
Planning
Methods
PPO
SHAC
APG
Transporter
ILD
CEM-MPC
diff-MPC
din-CCMM-PPC
expert demos
10
Reward function
/
V
V
Differentiabiity
Table Methods on Deformable Object Manipulation with respect different setups.
Fold-T-Shirt
Fold-Cloth-3
Unfold-Cloth-1
Unfold-Cloth-3
Reward aa
0.7
0.6
Reward
0.5-
Reward
0.7
0.6
0.5
0.5
0.4
0.2
0.0
0.1
0.0
0.1
0.2
0.0
0.1
0.0
2000
2000
2000
Iteration
Iteration
Iteration
Iteration
Shape Rope
Pour-Soup
Pour-Weter
Whip Rope
0.75
Reward
0.70
0. 0:40
0.35
0.25
Rgwarg
0.6
0 30
0.20
0.65
PRO
0.60
0.10
0.2
0.55
0.10
0.0
2000
2000
2000
2000
Iteration
Iter Iteration
Iteration
Iteration
Figure 3: Learning curves of three benchmarked RL algorithms: PPO, APG, and SHAC. The y-axis shows the episode reward, scaled from 0 to 1, with larger values being closer to the goal state. The x-axis the number of training iterations. All methods are trained in the same number of batched environments. We report the mean and variance of the performance for 5 seeds and each seed has 20 rollouts with different initializations. We refer the readers Table 4 for the numerical results in the appendix.
4.1 EXPERIMENTAL SETUP
Reward function For each task, the goal is specified by the desired final positions the set of particles, g. representing the deformable object. In this setup, the reward can be intuitively defined by how well the current object particles match with those of the goal. Hence, we define the ground- truth reward as (s, a) cxp( AD((s' (g), where s' is the next state resulted from at the current state 8, and (ss,1) g) is a non-negative distance measure between the positions of the object's particles and those of the goal g.
However, using the ground-truth reward alone inefficient for learning. The robot's 's end-effector can only change the current object pose by contacting Therefore, to most efficiently manipulate the object to the desired configuration, we add an auxiliary reward to encourage the robot to contact the object. We define the auxiliary reward as Taux (s, a) exp(- L2((s, a)), where L2(s, a) measures the L2 distance between the end-effector and the object. During training, we use the sum of the ground- truth and auxiliary reward functions Ty + Tauxs and during evaluation, we only use the ground-truth reward Tgt-
4.2 REINFORCEMENT LEARNING
Methods. We benchmark PPO (Schulman et al. 2017). SHAC (Xu et al. 2022 and APG (Freeman et al. 202T) as the representative RI. methods. PPO widely used model-free non- differentiable RL method with competitive performance, while SHAC and APG are the two latest RL methods that utilize the differentiable simulator. The computation graph of SHAC illustrated in Figure 4a By comparing the performance of the DaXBench tasks across these three methods, aim to further study the benefits and limitations of applying the differentiable physics to RL.
Overall Performance. The learning curves of RL methods are shown in Figure The performance difference between PPO and the differentiable RL methods shows opposite trends the high-level macro-action-based and local-level control tasks. To our surprise, for most of the short-horizon

---

Published as a conference paper at ICLR 2023
Imitation Learning
Planning
Task Type
Task
Transporter
ILD
Expert
CEM-MPC
Diff-MPC
Diff-CEMMMPC
Fold Cloth-1
0.19
0.76.10.04
0.90
0.74
0,29.10.07
0.80
Fold-Clon 3
0.40=0.05
0.82+0.05
0.89=0.00
0.62=008
0.14+00.02
0.66=0.07
High-level
Fold-T-she --sirt
0.46:0.02
0.59-0.11
0.8510.00
0.58-0044
0.29:10.06
0.66:10.01
Unfold-Clotl->
0.4810.01
0.64:10.03
0.87100.00
0.57=0.04
0.30
0.67
Unfold-Cloth-3
0.30=00.0
0.52+0002
0.87+0.00
0.57=0011
0.26:+:0.44
0.58==0.1
Push Soph-Rope
0.70.10.00
0.76.10.02
0.9310.00
0.71
0.70.10.01
0.71100.01
Whip-Rope
0.70:1:0066
1.00 0.00
0.34=0001
0.37+:0.00
0.33
Low-level
Pour-Water
0.32++0.03
0.91= 0.06
0.58=0011
0.30++00.00
0.58=0.01
Pour-Soup
0.42:2.12
0.85# 0.00
0.56=0.01
0.44::0.00
0.55
Table 3:
Task performance for the planning and imitation learning methods. We report the mean and standard error for the policy/control sequences evaluated under 20 seeds.
macro-action-base based tasks, PPO performs consistently better than the differentiable RL methods. This seems to contradict the experiment results of SHAC (Xu et al., 2022), which shows stronger performance than PPO in its own simulator on wide range of context-rich MuJoCo tasks. How- ever, for aag-horizon tasks with low-level controls, the differentiable RL methods, especially APG, outperform PPO by a large margin.
RL methods: Exploration. We argue that, on high-level macro-action--asse tasks, the main limiting factor of the performance of differentiable-phh physics- based RL is the lack of exploration during training. The need to balance this trade-off is exaggerated by the enormous DoF's and the complex dynamics unique to the DOM tasks. Especially for the high-level macro-action-baset tasks, their non-smmothhddicconinuuuusoooo convex optimization landscape necessitates using a good exploration strategy. However, the existing differentiable-phyy phaooll--ppisss---eeee based RL methods rely exclusively on the simulation gradient to optimize the policy. They are not entropy regularized; hence, they may quickly collapse to the local opti- mal. Without explicit exploration, the pitterenttable-pyyiis--bbaae near-optimal state using the simulation gradients alone. Taking Fold-Cloth-3 as an example, if the randomly initialized policy never touches the cloth, the local simulation gradients cannot guide the agent to touch the cloth since the rewards and, consequently, the gradients for all not-in-contact actions are zero.
Main Challenge for the hilferentiablle-hyysse--bae
RL methods fail to reach the
pifferentibble--hysee--bbe RL are sensitive to the Optimization Landscape. Differentiable- maysics-based RL can largely improve the performance some low-level control tasks if the opti- mization landscape is smooth and convex. Taking Whip-Rope control sequence
an example, the optimal low-level smooth motion trajectory that lies on a relatively smooth and plane for the gradient- maadeent-based methods. This explains why APG outperforms the non- non-differentiable. physics-based PPO by a large margin.
4.3 IMITATION LEARNING
Methods. We benchmark Transporter Seetaa et al. 2021 and ILD (Chen al. 2023 the two representative Imitation Learning DOM methods. Transporter is a popular DOM method that well-known for its simple implementation and competitive performance. We use particles as input to evaluate the performance of Transporter in our experiment for fair comparison. The original implementation of Transporter uses RGBD input. We refer the readers the Appendix section for the performance of the Transporter with RGB channels. ILD is the latest differentiable- physics- based IL method; it utilizes the simulation gradients to reduce the required number of expert demonstrations and to improve performance. Note that since Transporter abstracts its action space to pick-and-place actions to simplify the learning, it cannot be extended to the low-level control tasks. Our comparison of the IL methods will first focus the high-level macro-action-bug based tasks, then we will analyze the performance of ILD with the expert.
Overall performance For the high-level macro-action-basse tasks, ILD outperforms Transporter by a large margin for all tasks. For example, ILD outperforms Transporter in Fold-Cloth-1, Fold-Cloti We highlight that this significant performance gain is achieved by using
3, and Unfold-Clotha much smaller number of expert demonstrations. We attribute this improvement to the additional information provided by the simulation gradients, where now ILD reason over the ground-truth

---

Published as a conference paper at ICLR 2023
4
9o
g
Expert Trajectory
0g
Leamer Trajectory
Forward
V V(S,,
ag
Qg
Backward
(c) Diff' CEM-MPC. Figure 4: Computation graphs for the differentiable algorithms. We use SHAC, ILD, and Diff' CEM-MPC represent the differentiable algorithms for planning, RL, and IL respectively. The black/green arrows correspond to the forward/backward propagation respectively.
(a) SHAC.
(b) ILD.
dynamics to bring the trajectory closer to the expert, at least locally. As illustrated in Figure 4 the gradients used optimize each action comes from the globally optimal expert demonstrations 1) at the current step and 2) in all the steps after. In contrast, the gradients used in SHAC only count the reward at the current step. The differentiable RL faces the challenge of exploding/vanishing gradients and lack of exploration. Differentiable IL alleviates these problems by having step-wise, globally optimal guidance from expert demonstrations.
Challenge for ILD: Unbalanced State Representations. ILD does not perform well on tasks like Pour-Water and Pour-Soup with unbalanced state representations, for example, 1000 features of particles and 6 gripper states. The learned signals from the particles overwhelm the signals from the gripper states. Therefore, ILD cannot learn useful strategies with unbalanced learning signals. Softgym (Lin et al. (021) also concurs with the results that policies with reduced state representation perform better than those with the full state. more representative representation of the states needs to be considered and we leave it for future study.
In conclusion, differentiable methods can optimize the policies efficiently if the optimization land- scape is well-behaved. For tasks that require tremendous explorations with sparse rewards, gradient-based methods suffer severely from the local minima and gradient instability issues.
4.4 PLANNING
Methods. We benchmark the classic CEM-MPC (Richards) 2005) as representative random shoot- ing method that uses a gradient-free method to optimize the action sequences for the highest final reward. To study the effect of using differentiable physics, implement two versions of differ- entiable MPC. The basic computation graph for both is illustrated in Figure 4c particular, both differentiable MPC baselines maximize the immediate reward of each action using differentiable physics. The difference lies in the initial action sequences: diff-MPC initialized with a random action sequence, while Liff-CEM-MER with an action sequence optimized by CEM-MPC.
Overall Performance. The diff-CEM-MPC performs consistently better than or on par with CEM- MPC, while Diff-MPC fails plan any reasonable control sequences for most of the tasks.
Simulation Gradients as Additional Optimization Signals. The vastly different performance be- tween the two versions of the differentiable baselines brings us insights into how differentiable physics helps in plamning. Simulation gradients provide additional signals in the direction to opti- mize the actions, thereby reducing the sample complexity. However, similar any gradient-based optimization methods, differentiable planning methods are sensitive to the non-convex/non-oe smooth optimization landscape and face a major limitation of being stuck at the local optimal solution. DOM's enormous DoFss of DOM and complex dynamics exacerbate these challenges for the differ- entiable DOM planning methods.
Differentiable Planning needs Good Initialization. We attribute the poor performance of Diff- MPC to its random control sequence initialization. In particular, this control sequence may sig- nificantly deviate from the optimal one over the non-smoott/noon-conve landscape, such that the simulation gradients alone cannot bring the control sequence back to the optimal. In contrast, Diff- CEM-MPC is initialized with the optimized control sequence by CEM-MPC, which relies on itera- tive sampling followed by clite selection to escape the local optimality. Hence, the control sequence fed into the differentiable step is already sufficiently close to the globally optimal solution. The
8

---

Published as a conference paper at ICLR 2023
execution:
next state
Simulation:
State
aciion
CEM-MPC
?
>
>
>
USO
Dax (simulator)
(a)
(b)
Figure 5: We deploy CEM-MPC in Push-Rope task to straighten the rope in steps on real robot. (a) The system structure. CEM-MPC uses our DaX simulator as the predictive model. Given the current state estimated from a point cloud image, CEM-MPC plans the best action to be executed by the robot. (b) The states after each the pushes. We compare the dynamics in reality and in simulation. Top: state trajectory in real robot experiment. Bottom: the next state predicted by DaX given the estimated current state and the corresponding planned action.
simulation gradients help diff-CEM-MPC optimize the action sequences further locally, which explains the performance gain.
To summarize, the differentiable simulator can further optimize the near-optimal control sequences locally with much low sample complexity. Yet, its performance critically relies on good initializa- tion and locally smooth optimization landscape.
4.5 S-M-2-REAL GAP
Granted that the physical engine of DaX is built upon state-of-the-an simulation method, the fidelity and correctness still need to be tested. To testify to the correctness of our simulated dynamics to real-world physics,
a Kinova Gen 3 robot. We deploy CEM-MPC with DaX as the predictor model to a Push-Rops task, as shown in Figure 5 Our study finds that the resultant state trajectories fidelity of the tasks implemented based on our engine, and the performance of the DOM methods our simulator provides hints about their performance on the real robot. We refer the reader the appendix for details. In addition, the differentiabillty of DaX enables system identification that can reduce the sim- --mall gap (Le Lidec et al.) 2021] Murthy et al. (021). We can get an error term by comparing the observations in simulation and in reality; by differentiating this error with respect to simulator parameters, we get gradients for parameter tuning.
carry
real robot experiment
similar in simulation and in reality. This validates the
CONCLUSION
We introduced DaXBench, general-purpose differentiable simulation framework for various de- formable object manipulation tasks. To our knowledge, DaXBench is the first platform that sim- ulates liquid, rope, fabric, and elastoplastic materials while being differentiable and highly par- allelizable. In addition to common DOM tasks, we also proposed several novel tasks, including WhipRope and PourSoup, to study the performance of DOM methods on long-horizon tasks with low-level controls. Our task coverage provides comprehensive platform to compare existing DOM methods from all paradigms, and our empirical study sheds light on the benefits and limitations of differentiable-pyysee seeeee-hhiissbbbaaee based methods in DOM. We believe that DaXBenchcan assist in the algorith- mic development of DOM methods, particularly those based on differentiable physics, across all paradigms.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research is supported by the National Research Foundation, Singapore under its Al Singa- pore Programme (AISG Award No: A1SG2-PhD-2021-08--00 AISGG-PhID-0022-01--35 and AISG2-PhD-2021-00-- and A STAR under the National Robotics Program (Grant No. 192 25 00054).
ETHICS STATEMENT
DaXBench provides a benchmark for deformable object manipulation (DOM). We believe that DaXBench and its simulator DaX have no potential negative societal impacts. In addition, our work is on benchmarking the DOM methods simulator, so we do not collect any data or conduct experiments with human subjects. In summary, we have read the ICLR Code of Ethics and ensured that our work conforms to them.
REPRODUCIBILITY STATEMENT
In this paper, imitation learning and planning experiments are averaged over 20 random rollouts and RL approaches use 100 rollouts. We have included our source code an easy-to-install package in the supplementary material.
REFERENCES
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Siwei Chen, Xiao Ma, and Zhongwen Xu. Imitation learning via differentiable physics. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, 2023.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: arXiv:/604.060 6664061744 2016.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax A differentiable physics engine for large scale rigid body simulation. Advances in Neural Information Processing Systems, 2021.
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber. Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh McDermott, and Daniel L. K. Yamins. Threedworld: A platform for interactive multi-mode physical simulation. In Advances in Neural Information Processing Systems, 2020.
Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jen- nifer Grannen, Minho Hwang, Ryan Hoque, Joseph E Gonzalez, Nawid Jamali, et al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In Proc. IEEE Inf. Conf. on Robotics & Automation, pp. 11515-11522. IEEE, 2021.
Andreas Griewank and Andrea Walther. Algorithm 799: Revolve: An implementation of check- pointing for the reverse or adjoint mode of computational differentiation. ACM Trans. Math. Softw., 26(1), mar 2000.
Eric Heiden, Miles Macklin, Yashraj Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. Disect: A differentiable simulation engine for autonomous robotic cutting. In Proc. Robotics: Science & Systems, 2021.
10

---

Published as a conference paper at ICLR 2023
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning control pdes with differentiable physics. In Proc. Int. Conf. on Learning Representations, 2020.
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(4), 2018.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua Joshua Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojeiech Matusik. Chainqueen: A real-time differentiable physical simu- lator for soft robotics, In Proc. IEEE Int. Conf. on Robotics & Automation, pp. 6265-6271. IEEE, 2019.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan- Kelle,,lley, and Fredo Durand. Difftaichi: Differentiable programming for physical simulation. In Proc. Int. Conf. on Learning Representations, 2020.
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In Proc. Int. Conf. on Learning Representations, 2021.
Fouad F. Khalil and Pierre Payeur. Dexterous robotic manipulation of deformable objects with multi-sensory feedback review. In Agustin Jimenez and Basil M Al Hadithi (eds.), Robot Manipulators, chapter 28. IntechOpen, Rijeka, 2010. doi: 10.5772/9183. UtLhhtpps://do
arg/10.57722
Quentin Le Lidec, Igor Kalevatykh, Ivan Laptev, Cordelia Schmid, and Justin Carpentier. Differen- tiable simulation for physical system identification. IEEE Robotics & Automation Letters, 6(2), 2021.
Xingyu Lin, Yufei Wang. Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pp. 432-448 PMLR, 2021.
Martina Lippi, Petra Poklukar, Michael C. Welle, Anastasiaa Varava, Hang Yin, Alessandro Marino, and Danica Kragic. Latent space roadmap for visual action planning of deformable and rigid object manipulation. Proc. IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, 2020.
Xiao Ma, David Hsu, and Wee Sun Lee. Learning latent graph dynamics for visual manipulation of deformable objects. In Proc. IEEE Int. Conf. Robotics & Automaatin, pp. 2022.
IEEE,
Jeremy Maitin-Shepard, Marco Toosue,,--Toweer, Jinna Lei, and Pieter Abbeel. Cloth grasp point detection based multiple-view geometric cues with application to robotic towel folding. In Proc. IEEE Int. Conf. on Robotics & Automation. IEEE, 2010.
Stephen Miller, Mario Fritz, Trevor Darrell, and Pieter Abbeel. Parametrized shape models for clothing. In Proc. IEEE Int. Conf. on Robotics Automation. IEEE, 2011.
Stephen Miller, Jur Van Den Berg, Mario Fritz, Trevor Darrell, Ken Goldberg, and Pieter Abbeel. A geometric approach to robotic laundry folding. Int. J. Robotics Research, 31(2), 2012.
J. Krishna Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-Lévesque Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differentiable simulation for system identification and visuomotor control. In Pruc. Inf. Conf. on Learning Representations, 2021.
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint arXiv: (tt::1909008222 2019.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Scalable differentiable physics for learning and control. In Proc. Int. Conf. on Machine Learning, 2020.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Efficient differenniable simulation of articulated bodies. In Proc. Int. Conf. Machine Learning, 2021.

---

Published as a conference paper at ICLR 2023
Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute Technology, 2005.
Jose Sanchez, Juan-Antonn Corrales, Belhassen-Chedli Bouzgarrou, and Youcef Mezouar. Robotic manipulation and sensing of deformable objects in domestic and industrial applications: survey. Int. J. Robotics Research, 37(7), 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (10.::1777.66444 2017.
Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, and Andy Zeng. Learning rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. Proc. IEEE Int. Conf. on Robotics & Automation, pp. 4568 4575. IEEE, 2021.
Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas Guibas, Silvio Savarese, Anima Anand- kumar, and Yuke Zhu. Acid: Action-conditional implicit visual dynamics for deformable object manipulation. In Proc. Robotics: Science & Systems, 2022.
Deborah Sulsky, Zhen Chen, and Howard Schreyer. A particle method for history-dependent materials. Computer methods in applied mechanics and engineering. 118(1-2), 1994.
Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph E. Gonzalez, and Ken Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In Proc. IEEE Int. Conf. Robotics & Automation, pp. 9411 1-9418. IEEE, 2020.
Youngsun Wi, Pete Florence, Andy Zeng, and Nima Fazeli. Virdo: Visio-tactile implicit represen- tations deformable objects. Proc. IEEE Int. Conf. on Robotics Automation, 2022.
Jie Xu, Miles Macklin, Viktor Makoviychuk, Yashraj Narang, Animesh Garg, Fabio Ramos, and Wojeiech Matusik. Accelerated policy learning with parallel differentiable simulation. In Proc. Int. Conf. on Learning Representations. 2022.
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive representa- tions for deformable objects using contrastive estimation. Conference on Robot Learning, pp. 564-574 PMLR, 2021.
Jihong Zhu, Andrea Cherubini, Claire Dune, David Navarro-Alarcon, Farshid Alambeigi. Dmitry Berenson, Fanny Ficuciello, Kensuke Harada, Jens Kober, Xiang Li, al. Challenges and outlook in robotic manipulation of deformable objects. IEEE Robotics & Automation Magazine, 29(3): 67-77, 2022.
12

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A.1 OBJECTS
DaXBench models a variety of deformable objects of different dimensionalities, ranging from O- dimensional liquid, 1-dimensional rope, to 2-dimensional cloth. The underlying deformation and the dynamics vary distinctly for each dimension. In general, the lower the dimension is, the higher the degrees of freedom (DoFs) of the system, thus the more complex modeling that system becomes. We use two different representations to model the deformable objects to balance the modeling capacity with its complexity: water and rope modeled using Material Point Method (MPM), while the cloth is modeled using a sprinngpring system.
Modelling Water and Rope using MPM Material Point Method (MPM) demonstrates strong mod- eling capacity and flexibility. MPM models the deformable objects as set of particles; each particle represented by its position and some properties such as mass. The dynamics and the deformation of the object are computed by simulating the interactions among all individual particles, conforming pre-defined physics rules. This Particle-level representation and the dynamics simulation allow MPM to model the object under severe deformation. We, therefore, choose to use MPM to model water and rope that can undergo severe deformation, for they can exploit the model capacity of MPM to its full potential.
Modelling Cloth using Mass-Spring System Cloth only undergoes limited deformation as it has to conform to the canonical shape of the initial cloth through manipulation. This limited deformable does not require the strong modeling capacity of MPM, which is computationally expensive. There- fore, rather than modeling the deformation the Particle-level, we choose to model the cloth using mass-spring system, which is simpler model that uses the physical constraints as prior.
A.2 CODE EXAMPLE
import numpy np
from core. registration import env_functions
env env_ eoy_functions
bbatch_sie==2,, seed=1) obs, state env reset (env. wwn.v.immlattr... (env.siiullterr...y.....o
actions np. random form 1, (env. honv.batc_ss ze, env.action_size) obs, reward, done, info env. anv.step_diff (act state)
Listing 1: Example of Running Cloth Environment
import jax
import numpy as np from core. envs. wor..env.. registraaitn impert env_functions
env env env_functions (rrriionssffoddclttttto obs, state env reset (env. www..simmattor........... key_goball
tbtech_ aux_roward=True)
actions random. (n 1, cew.battchsize, env.action_size)
@jax.jit
10 @jax.grad
11 def compute compute_grad (actionss state)
obs, reward, done, info = env. diff (actions, state)
return -reward. 14
I5 print (*action gradients", compute_gradd(attion state)) Listing 2: Example of Computing Action Gradients via DaX
A.3 JAX TO PARALLELIZE
One major limitation the existing simulators is the lack support for highly paralleled sampling. The key challenges for Deformable Object Manipulation are the complex governing equations and
13

---

Published as a conference paper at ICLR 2023
(a) Detect the active region.
(b) Update the defor-
Shift back the up- date active region.
mation.
Figure 6: We demonstrate how to use the active region to reduce the computation complexity for the simulation. Taking rope for example, the original environment, denoted as the black background, is of size 128 128 x 128, while the active region, where the region that will be actively changed due to the current deformation highlighted in red, is of size 32 x 6 x 32, as illustrated in (a). To avoid updating all the particles in the environment, in b), shift the active region to the top corner, the origin of the environment, and update the dynamic and observation. Finally, in ( c), we shift the updated active region back to align its original coordinates.
the enormous state space. The complex governing equations invalidate most of the analytical ap- proaches since solving the complex governing equations intractable. On the other hand, the suc- cess of the sampling-based methods hinges on the reasonable coverage of the state space. The lack of support for highly paralleled sampling to better cover the state space a major impediment to the works on Deformable Object Manipulation. DaXBench fills in this gap by implementing its simulator using JAX, which supports highly paralleled sampling with great automatic differentia- tion. Using our implementation, we finish iteration (both forward and backward pass) for 128 rollouts with 80 timesteps on a server with 2080-Ti GPUs in 3 seconds.
A.4 SAVING MEMORY
Lazy Dynamic Update DaXBench optimizes the time and memory consumption of MPM by only updating the dynamic of small region affected by the manipulation at each step. MPM models and updates the state and dynamics on the Particle-level; its time and memory consumption is pro- portional to the size of the environment, which can be arbitrarily large. Our insight is that, though the deformable object can potentially interact with the entire environment, only a small region will be affected at each time. This inspires DaXBench to optimize the time and memory consumption of running MPM by exclusively considering the active region during the interactions. Using this trick, DaXBench effectively reduces the computation time by two orders of magnitude on average. Taking the rope for example,
original environment is size 128 x 128 x 128. However, grid of size 32 x 6 32 is sufficient to cover the region of interaction between the environment and the rope. Therefore, we first detect the active region of size 32 x 6 x 32, then we perform a linear transformation shift the active region the origin, perform the update due to deformation and interaction, finally we shift the updated active region to align with its original coordinates.
Checkpointing method DaXBench only stores the values of few states (i.e. checkpoints), and re-evaluates the sub-step gradients online during the back-propag to reduce the memory con- sumption for long horizon problems. The checkpointing method has been proven useful in saving the memory consumption for differentiable simulations in many prior works (Qiao et al. 2021 Chen al. 2016 Griewank & Walther, 2000). To make the simulator end-to-end differentiable. every value in the neural network that is computed during the forward pass has to be saved, as they are needed to compute the gradient during the backward pass. The computation complexity scales up quickly w.r.t. the length of the task horizon. This is significant problem for our simulator for high DoF' deformable object manipulation, as the memory required for each timestep's update enor- mous compared to its rigid body counterpart. To overcome the large memory consumption issue, DaXBench trade-offs the computation complexity slightly for large memory consumption improve- ment: DaXBench only stores the values of a few states (i.e. checkpoints) at the forward pass; during the backward propagation, DaXBench re-compute segments of the forward values starting from ev- ery stored state in the reverse order sequentially for the overall gradient. With this trick, DaXBench
14

---

Published as a conference paper at ICLR 2023
Task Type
Task
PPO
APG
SHAC
Fold Cloth
0.40=10.13
0.36:40.06
0.34=0.07
Fold Cloth
0.21110.11
0.19100.99
0.22
High-level
Unfold Cloth
Fold T-shirt
0.61::0.88
0.72.1
0.42:10.02
0.40+10.05
0.44=0.09
0.50
Unfold Cloth
0.56::0.00
0.39:10.02
0.48:00.03
Rope Push
0.75==0.02
0.72+10.02
0.75=0.01
Wipe Rope
0.25 10
0.83=+0.01
0.66=0.03
Low-level
0.27-40.02 0.27.10.08
Pour Soup
0.27:10.00
0.32 0.13
Table 4: Reinforcement Learning Methods Task Results. We report the mean and standard enror over 20 random seeds for each entry.
now doubles the computation complexity, but saves an arbitrary amount of memory, depending on the intervals the states saved.
A.5 DISCONTINUOUS GRADIENT
As an end-to-end differentiable simulator, the effectiveness of DaXBench rests on the accuracy of the gradient computed. We found that for deformable object manipulation, the action not in contact with the object will cause the gradient to be discontinuous, which causes the estimate of the gradient after this action suffer from high variance and poor accuracy in the backward propagation. To mitigate this problem, for every action input by the agent or the human user, DaXBench will perform local adjustment such that the action will always make the gripper contact with the deformable objects. This ensures that our gradient is end-to-end continuous therefore improving the accuracy of the estimated gradient.
B SIM-2-REAL GAP
To verify that the dynamics of our simulated tasks have high fidelity and small Sim2Real gap, we earry out real robot experiment. We deploy CEM-MPC the PushRope task on Kinova Gen3 robot. CEM-MPC uses our DaX simulator as the predictive model. Given a state, CEM-MPC plans the next best push action (I, yJ, 2, ,', z').. The state is estimated from point cloud image, and the Cartesian space push action is transformed to the joint space trajectory via an inverse kinematics module. Note we are not verifying the sim2real gap caused by the inaccuracy state estimation or kinematics; we are interested in whether the dynamics of DaX can correctly guide the CEM-MPC to succeed in the task. The experiment is included in our supplementary video.
C RL ALGORITHMS NUMERICAL EXPERIMENT RESULTS
We report the numerical experiment performance for the RL algorithms in Table 4 This is to sup- plement the learning curve in the main text. These numerical results report the final performance of the learning curves in Figure 3
Imitation Leaming
Task Type
Task
Transporter
Transporter RGB
I1.D)
Expert
Fold-Cloth-1
0.19++0.04
0. 0.86=0.07
0.76++0.04
0.91=0.00
Fold-Cloth-3
0.40-1:000
0.36:1:.25
0.82-10.05
0.89::0.0
level
Fold-T-shirt
0.46::0.02
0.83=20.05
0.5940..11
0.85+00.00
Unfold-Cloth-
0.48-10.01
0.76-20.06
0.64-10.03
0.8720.00
Unfold- Cloth
0.30-10.00
0.70-10.06
0.52-10.02
0.87-10.00
Push-Rope
0.70+40.00
0.86=0.09
0,76+:0.02
0.9340.00
Whip-Rope
0.70-100006
1.00+ 0.00
Low-level
Pour
0.32-10.03
Pour-Soup
0.42:10.12
0.85:1 0.00
Table 5; Task performance for the planning and imitation learning methods. We report the mean and standard error for the policy/control sequences evaluated under 20 seeds.
15

---

Published as a conference paper at ICLR 2023
D TRANSPORTER WITH IMAGE AS INPUT
We reported the performance of two different versions of Transporter in Table Transporter, which uses 3D particles and projects them onto an image to form height map; and Transporter-RGB, which adds RGB channels and projects the particles onto the image plane, giving it white color with a black background. We have found that an explicit process of rendering particles into RGB images significantly enhances the performance of Transporter.
16


Reviews:
[{"review_id": "n3ZwUiaujOP", "reviewer": "Reviewer_5D3R", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduce DaXBench, which is a very strong differentiable simulator for various DOM tasks. The authors examined various algorithms for the given tasks.", "summary_of_the_review": "Very nice paper. I think it's a strong paper and definitely worth acceptance. I wish the authors can address the weakness i mentioned above.", "strength_and_weaknesses": "Strength:\n- A very respectable implementation efforts was put into this work. I think this simulator will be of interest to many in the community.\n- A large range of learning algorithms are studied. Offering some interesting insights.\n\n\nWeakness:\n- Some of the more recent DOM works are not referenced to, all of these works offer insights / simulator for DOM:\n* ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation, Gan et al, Neurips 2021\n* ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, Yuke Zhu, RSS, 2022\n* VIRDO: Visio-tactile Implicit Representations of Deformable Objects., Youngsun Wi, Pete Florence, Andy Zeng, Nima Fazeli. ICRA 2022.\n- I'm not confident about the sim2real part. Saying the simulator has sim2real ability is a pretty big claim. However, only a demo video is provided without any systematic evaluation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UtI1xPIqaFA", "reviewer": "Reviewer_4Kpq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a benchmark that can simulate and evaluate a variety of manipulation tasks on deformable objects, including fluid, cloth, rope, and elastoplastic objects. The simulator is implemented using Jax, therefore it can also provide analytic gradients, making it compatible with recent popular control algorithms designed for differentiable simulation. \n\nA series of comparisons between differentiable-simulation-physics methods and gradient-free ones are further conducted. The experiments cover a wide range of categories of algorithms, e.g. reinforcement learning, imitation learning, and planning. Based on these comprehensive evaluations, the authors conclude the limitation and advantages of the differentiable-physics-based methods.\n\nIn the end, the authors successfully conduct a real-world Push-Rope task using the control policy from the simulation, which can validate the fidelity of the simulator.\n", "summary_of_the_review": "In my view, this paper provides a unified platform to evaluate the soft-body manipulation tasks and differentiable physics algorithms. It conducts comprehensive experiments and provides useful insights for future work. The sim-to-real experiment makes it more convincing. One thing it can improve is its differentiation from previous methods. In total, I think it is a good paper.", "strength_and_weaknesses": "Strength\n\nIn my view, the experiments are the highlights of this paper. The authors implement the recently proposed differentiable simulation papers in their benchmark DaX, and compare them with gradient-free methods. The experiments include various robotic learning algorithms. This systematic evaluation indicates that the differentiable-physics-based method can indeed have better performance in some cases, but they also suffer from limitations as many gradient-based methods do, i.e. sensitive to the optimization landscape and initialization. And it also reiterates the importance of exploration in those differentiable-physics-based methods. Moreover, the authors perform real-world experiments, making the benchmark more convincing.\n\nWeaknesses\n\nSince the research of differentiable physics is progressing fast, the authors might better illustrate and clarify its difference from related literature. For example, PlasticineLab is another platform for differentiable soft body manipulation. Since both DaX (fluid, elastoplastic, and mixture) and PlasticineLab are using MPM as the governing dynamics, it is not apparent to me why PlasticineLab cannot support liquid and mixture (in Table 1) while DaX can. Moreover, it is unclear to me how the solid ingredients are represented in Pour-Soup. Are they elastoplastic objects or purely rigid bodies? Is there a coupling issue between the fluid particles and the ingredients?\nFurthermore, the Saving Memory strategy in Section 3.1 is the same as the checkpoint scheme in [1] (Section 4.2), where the back-propagation trades time for memory usage by recomputing the intermediate variables in each step.\n\n[1] Qiao, Yi-Ling, Junbang Liang, Vladlen Koltun, and Ming C. Lin. \"Efficient differentiable simulation of articulated bodies.\" In International Conference on Machine Learning, pp. 8661-8671. PMLR, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ShXlfcoRA7", "reviewer": "Reviewer_yxby", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposed a benchmark platform for deformable object manipulation tasks: DaXBench. DaxBench contains several environments with a mixture of fluid, fabric, and elastoplastic materials. Due to the support of JAX, DaXBench seems to support multi-GPU simulation and training easily. Also, it promotes differentiable simulation. The authors compared a set of controller optimization methods on the proposed benchmark platform and shared their insights.", "summary_of_the_review": "This manuscript proposed DaxBench, the first deformable object manipulation benchmark platform based on JAX. I appreciate the work from the authors to build the simulation framework and share it with the community, but I expect some explanations and comparisons to taichi/warp.", "strength_and_weaknesses": "Strength:\n- As far as I know, DaXBench is the first deformable object manipulation benchmark platform based on JAX.\n- This manuscript compared a large set of methods on their proposed method, which provides the readers with a good sense of the performance of their simulator and the compared methods.\n- The provided code represents the visible reproducibility of the work.\n\nWeaknesses:\n- I wonder how the *mixture* is handled in these demos: are they two-way coupled, one-way coupled, or any other coupling was used?\n- I personally did not follow the *Lazy Dynamic Update* part: MPM will require a grid for simulation. Does it mean the overall grid is 32*6*32, or is the grid actually 128*128*128, but only a small region of it is activated? My personal feeling is this setting only applies to a small fraction of the proposed environments, is it?\n- My main reservation is the comparison at the framework level. JAX has a very similar position to taichi and Nvidia warp, which are more famous for physical simulation. And I would expect they have similar performance to JAX-implemented environments, is it? Is DaXBench distinguishable from them because of user-friendliness? Or better interface to deep learning modules? Or speed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%