DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics
Authors: Sizhe Li
Zhiao Huang
Tao Chen
Tao Du
Hao Su
Joshua B. Tenenbaum
Chuang Gan
Venue: ICLR 2023 poster (2023)
Keywords: Deformable Object Manipulation
Dexterous Manipulation
Differentiable Physics

Abstract: In this work, we aim to learn dexterous manipulation of deformable objects using multi-fingered hands. Reinforcement learning approaches for dexterous rigid object manipulation would struggle in this setting due to the complexity of physics interaction with deformable objects. At the same time, previous trajectory optimization approaches with differentiable physics for deformable manipulation would suffer from local optima caused by the explosion of contact modes from hand-object interactions. To address these challenges, we propose DexDeform, a principled framework that abstracts dexterous manipulation skills from human demonstration, and refines the learned skills with differentiable physics. Concretely, we first collect a small set of human demonstrations using teleoperation. And we then train a skill model using demonstrations for planning over action abstractions in imagination. To explore the goal space, we further apply augmentations to the existing deformable shapes in demonstrations and use a gradient optimizer to refine the actions planned by the skill model. Finally, we adopt the refined trajectories as new demonstrations for finetuning the skill model. To evaluate the effectiveness of our approach, we introduce a suite of six challenging dexterous deformable object manipulation tasks. Compared with baselines, DexDeform is able to better explore and generalize across novel goals unseen in the initial human demonstrations. Additional materials can be found at our project website: https://sites.google.com/view/dexdeform.

Published as a conference paper at ICLR 2023
DEXDEFORM: DEXTEROUS DEFORMABLE OBJECT MA- NIPULATION WITH HUMAN DEMONSTRATIONS AND DIFFERENTIABLE PHYSICS
Sizhe Li Zhiao Huang"*, Tao Chen', Tao D
Hao Su², Joshua B. Tenenbaum', Chuang Gan
I MITT PC San Diego, Tssinguua University, *Shanghai Qi Zhi Institute,
'IIT BCS, CBMM, CSAIL, UMass Amherst, 'MIT-IBM Watson Al Lab sizheli@csail. wwt.miiteeee edd, z ucsd. taochen@mit t. edu, edu. en, haosudeng ucsd. edu, jbt@mit.
ABSTRACT
In this work, we aim to learn dexterous manipulation of deformable objects using multi-finggerd hands. Reinforcement learning approaches for dexterous rigid object manipulation would struggle in this setting due to the complexity of physics inter- action with deformable objects. At the same time, previous trajectory optimization approaches with differentiable physics for deformable manipulation would suffer from local optima caused by the explosion of contact modes from hand-object interactions. To address these challenges, we propose DexDeform, a principled framework that abstracts dexterous manipulation skills from human demonstration, and refines the learned skills with differentiable physies. Concretely, we first collect small set of human demonstrations using teleoperation. And we then train skill model using demonstrations for planning over action abstractions imagination. To explore the goal space, we further apply augmentations to the existing deformable shapes in demonstrations and use a gradient optimizer to refine the actions planned by the skill model. Finally, we adopt the refined trajectories new demonstrations for finetuning the skill model. To evaluate the effectiveness our approach, we introduce suite of six challenging dexterous deformable object manipulation tasks. Compared with baselines, DexDeform is able to better explore and generalize across novel goals unseen in the initial human demonstrations. Additional materials can be found at our project website
INTRODUCTION
The recent success of learning-based approaches for dexterous manipulation has been widely observed on tasks with rigid objects (OpenAl et al., 2020; Chen et al., 2022: Nagabandi et al., 2020). However, substantial portion of human dexterous manipulation skills comes from interactions with deformable objects (e.g., making bread, stuffing dumplings, and using sponges). Consider the three simplified variants of such interactions shown Figure 1. Folding in row requires the cooperation of the front four fingers of downward-facine hand to carefully lift and fold the dough. Bun in row 4 requires two hands to simultaneously pinch and push the wrapper. Row shows Flip, an in-hand manipulation task that requires the fingers to flip the dough into the air and deform it with agility.
In this paper, we consider the problem of deformable object manipulation with a simulated Shadow Dexterous hand (ShadowRobot, 2013). The benefits of human-level dexterity can be seen through the lens of versatility (Feix et al., 2015; Chen et al., 2022). When holding fingers together, the robot hands can function fingertips, we can arrive at a stable grip on the object while manipulating the shape of the object (Fig. 1, row 2). Using spherical grasp, the robot hands able quickly squeeze the dough into
a spatula to fold deformable objects (Fig. 1, row 1). When pinching with
"Equal Contribution Project website: https: / //ai tes. lales.gooolle con/ http://ewwwdddeedefeee.m

---

Published as a conference paper at ICLR 2023
We present framework for learning dexterous manipulation of deformable objects, covering tasks with single hand (Folding and Wrap, row 1-2), in-hand manipulation (Flip, row 3), and dual hands (Bun, Rope, Dumpling, 4-6). Images in the rightmost column represent goals.
Figure 1:
folded shape (Fig. row Therefore, it is necessary and critical learn manipulation policy that autonomously controls the robot hand with human-like dexterity, with the potential for adapting various scenarios. Additionally, using multi multi-finngere hand adds convenience demonstration collection: (1) controlling deformable objects with hands is natural choice for humans, resulting in an easy-to-adapt teleoperation pipeline. (2) there exists a vast amount of in-the-wild human videos for dexterous deformable object manipulation (e.g., building sand castle, making bread). Vision-based teleoperation techniques can be employed for collecting demonstrations at scale (Sivakumar et al., 2022). As with any dexterous manipulation task, the contact modes associated with such tasks are naturally complex. With the inclusion soft bodies, additional difficulties arise with the tremendous growth in the dimension of the state space. Compared the rigid-body counterparts, soft body dynamics carries infinite degrees of freedom (DoFs). Therefore, it remains challenging to reason over the complex transitions in the contact state between the fingers and the objects.
Given the high dimensionality of the state space, the learning manipulation policy typically requires a large number of samples. With or an insufficient amount of demonstrations. interactions with the environment are needed to improve the policy. Indeed, past works in dexterous manipulation have leveraged reinforcement learning (RL) approaches for this purpose (Rajeswaran al., 2017; Chen et al., 2022). However, the sample complexity of most RL algorithms becomes a limitation under the deformable object manipulation scenarios due the large state space. Recent works have found trajectory optimization with the first-order gradient from differentiable simulator to be an alternative solution for soft body manipulation (Huang et al., 2021; Li al., 2022a; Lin et al., 2022). However, the gradient-based optimizers found to be sensitive to the initial conditions, such as contact points. It remains unclear how to leverage the efficiency of the gradient- -bseed optimizer and overcome its sensitivity to initial conditions at the same time.
In this work, we aim to learn dexterous manipulation of deformable objects using multi-fingee cinti--fineeed hands. To address the inherent challenges posed by the high dimensional state space, propose DexDeform, a principled framework that abstracts dexterous manipulation skills from human demonstrations and refines the learned skills with differentiable physics. DexDeform consists of three components:
2

---

Published as a conference paper at ICLR 2023
(I) collecting small number of human demonstrations (10 per task variant) with teleoperation for initializing the training data. (2) extracting abstractions of the dexterous action sequences from demonstrations with a skill model. This model decomposes the manipulation process and allows for planning for a novel goal with a learned skill dynamics predictor. (3) using differentiable physics refine trajectories planned by the skill model augmented goals, which adds new trajectories to further fine-tune the skill model. Hence, DexDeform is capable of avoiding local minima of the gradient-based optimizer by initializing trajectories with the abstractions of dexterous actions. At the same time, DexDeform enjoys the efficiency of the gradient- based optimizer to augment demonstrations for bootstrapping the learned skill model.
To evaluate the effectiveness of DexDeform, we propose suite of six challenging dexterous de- formable object manipulation tasks with differentiable simulator. Extensive experiment results suggest that DexDeform can successfully accomplish the proposed tasks and explore different goal shapes on set of dexterous deformable object manipulation tasks. In summary, our work makes the following contributions:
We perform, to the best of our knowledge, the first investigation on the learning-based dexterous manipulation of deformable objects.
We build a platform that integrates a low-cost teleoperation system with soft-body simulation that is differentiable, allowing humans to provide demonstration data.
We propose Deform,,, a principled framework that abstracts dexterous manipulation skills from human demonstration, and refines the learned skills with differentiable physics.
Our approach outperforms the baselines and successfully accomplishes six challenging tasks such as Flip, learning complex soft-body manipulation skills from demonstrations.
2
METHOD
(1) Initialize training data using human teleoperation.
(2) Learning skill abstractions from demonstrations.
Skill Encoder
[ Ro Stgg Ueek)
Skill Dynamies
Ze
Skill
(0,,
(Deegi ac+g)
(0.42x 4442k)
Space
Skill Decoder
(3) Exploration with planning and differentiable physics on novel goals.
Plan skill with dynamics (s 5.)
Decoding skill (o..2.)...
Gradient Descent
Add
L
Dataset
existing goal
So
Repeat (2)-(3)
3a,
novel goal
Figure 2: An overview of DexDeform (Sec.2). (1): We first collect human demonstrations using hand tracking teleoperation. (2): We then train skill model from demonstrations, which consists of a skill sequence encoder, a skill dynamics predictor, and skill action decoder. (3): To explore the high dimensional state space, we use the skill model to plan for novel goals, and apply a gradient-bar based optimizer refine the actions planned by the skill model. Lastly, we store the successful trajectories as new demonstrations and repeat (2)-(3).
Given goal configuration of deformable shapes, objective is plan actions perform dexterous deformable object manipulation using the Shadow Dexterous hand. We assume we know the full point cloud observation of the scene, which includes the multi-inggeee hand(s) and the object(s).
To tackle this problem, we propose DexDeform (Fig. 2). a framework that consists three com- ponents: (1) small set of human demonstrations for policy initialization (Sec. 2.1); (2) learning
3

---

Published as a conference paper at ICLR 2023
skill abstractions of dexterous actions from demonstrations (Sec. 2.2); (3) exploring novel goals with planning and differentiable physics (Sec. 2.3).
2.1 COLLECTION OF HUMAN MANIPULATION DEMONSTRATION
Dexterous manipulation with high degrees-of-freedis robot hand poses challenges for policy learning, since the state dimension is high and dexterous tasks involve frequent contact making and breaking between the hand and the objects. Learning such a policy from scratch would be extremely time consuming (OpenAl et al., 2020). One way to overcome the exploration challenge is by providing human demonstration data. However, many prior works use a complex and expensive system such as motion capture system (Rajeswaran et al., 2017: Gupta et al., 2016) or many cameras (Handa et al., 2020) for capturing human demonstration data. We built a low-cost (S100) and simple teleoperation system that allows human operator to control the simulated hand in real time to perform dexterous manipulation tasks. Our system is built based on the Leap Motion Controller (Spiegelmock, 2013) which is optical hand tracking device. By constructing inverse kinematics model based on the Shadow hand, our system re-targets the detected human finger positions into the joint positions of the simulated robot hand that is controlled via a position-based based PD controller. More details on teleoperation setup can be found in Appendix B.
2.2 LEARNING ABSTRACTIONS OF DEXTEROUS SKILLS
Humans execute abstractions of dexterous skills interact with deformable objects instead of plan- ning every finger muscle movement involved. In the same spirit, we would like to learn abstractions of actions present in the collected human demonstrations. Our skill model consists of three components, a skill encoder, a skill dynamics predictor, and a skill action decoder. The skill model uses dynamics predictor for planning. and action decoder to predict actions from skill embeddings. The skill model is built on the implicit scene representations of point clouds, which we will describe first.
Implicit Representation of the Scene. We leverage the translational equivariance offered by the Convolutional Occupancy Network (Peng et al., 2020), or ConvONet, to build a continuous implicit representation of the scene. Concretely, let or E O describe the unordered point cloud observation R (the 3D position and 3D color label). The
at time 6, where Or = {21, ..., In with encoder of ConvONet Venc. : O + R Hx x maps a point cloud to set of 2D feature maps. Given a query point p E R3, we get its point feature (enc(De))p from the feature maps (o)) via bilinear interpolation. An occupancy decoder (dec((, (ene( R X RD - R3 is then used to map a query point and its point feature (en)(Ot))p into the occupancy probabilities of being free space, hand, and the deformable object, based on one-hot encoding. Our ConvONet is trained with sufe-uuperv in simulation. We use the 2D feature maps from the encoder as the translational-equivrrinn representation of the scene.
Latent encoding of the implicit scene representation. As will be explained later, our choice of implicit representation of the scene can be naturally integrated with our skill model for planning for target deformable shapes. To extract compact scene representation for dynamics modeling and planning in the latent space, we train a VAE to reconstruct the 2D feature maps from ConvONet encoder outputs. The VAE includes an encoder &ene that encodes the scene representation into a latent vector 8; and a decoder Φdec that decodes the latent back into the scene representation space. Specifically, = Φene (Wen(WVec(O)))) and (dec(()) == (o.).
Skill Encoder. Using our learned latent encoding of the scene, we encode the point cloud observation at each timestep into 8t. We then train a skill encoder ʃ that maps a sequence of K-step observation-action pairs (st, (t,,a,,... Kt+ at+*) x) into a skill embedding space containing 2;; i.e.,
21
f(t, Ot..... St+K. 0ttK) K). We use 2, for decoding actions between timesteps & and t + K.
Skill Dynamics and Skill Decoder. For each skill embedding, we follow SkiMo (Shi et al., 2022b) to jointly learn to predict the resulting dynamics of applying the skill and decode the actions responsible for forming the skill. Concretely, we train a skill dynamics predictor $ = T $1) that predicts the future latent scene encoding K steps away in imagination. We also train skill action decoder a (a,l 20,2,,55 s,) that recovers the actions corresponding to the skill abstractions. We refer the readers to Appendix C for the training details and objective functions.
4

---

Published as a conference paper at ICLR 2023
Long-horizon planning in the space of skill abstractions. Our choice of implicit representation of the scene allows us to decode the latent encoding for computing the shape occupancy loss for planning. Given target shape g described by point cloud and horizon H, hope to find the sequence of skills 2k, .... žI such that the final predicted scene encoding 8 is occupied by the points in target shape under the object category. Let (oce (1gg)) be the sum of the cross entropy loss computed between each point T; within target shape g described as point cloud and the predicted occupancy probabilities (a dee ($ ($*+K) when z; queried in the decoded scene representation. We formulate our planning problem as continuous optimization problem.
arg min ,1 8
C(g. 2) = Loce T(=n, ((,,-
Here, z = 2K, 2K, 21 is the sequence of skills we are optimizing over, and we iteratively apply T in forward manner HHK times to predict the resulting scene encoding $ In practice, we optimize batch of initial solutions (K, "..,, =H 1 and choose the best one based on C(g, z). We refer the readers Appendix C for more details on skill planning.
2.3 DIFFERENTIABLE PHYSICS GUIDED EXPLORATION
Given that the skill model could be limited interactions captured in the current demonstration set, more interactions are needed for the skill model to generalize across novel goals. Two challenges exist: (1) Given the high degrees of freedom of soft bodies and human demonstrations, distribution cannot be easily defined in closed-form expressions. How can we sample novel goals in the first place? (2) Suppose that novel target shape is provided and is not closely captured by the demonstration set the skill model is trained on. How can we efficiently enable the skill model to achieve the novel shape, which would allow us expand our demonstration set? We present two ideas for overcoming the two challenges.
shape
Shape augmentation for novel goal generation. To tackle the intractability of the distribution of deformable shapes and sample new shapes, we explore the space of deformable shapes based on the shapes covered by the current demonstrations. We employ two simple geometric transformations: translation on the xz-plane and rotation around the y-axis, which is similar to data augmentation practices in training neural networks for image classification. We randomly sample target shapes from the existing demonstrations and apply augmentations to generate new target shapes.
lefferentiabll-phyiiiee based trajectory refinement. We use trajectories planned by the skill model optimization initialization overcome the local optima caused by the complex contacts, and use the gradient-base optimizer to refine the planned trajectories within tens of iterations.
The DexDeform algorithm. Putting all ingredients together, we present the DexDeform algorithm (Algo. 2) in Appendix D. During training, our framework learns implicit scene representations and the skill model. During exploration, framework leverages differentiable physics optimizer to expand the demonstrations to new goals.
3 EXPERIMENTS
In this section, we conduct experiments aiming to answer the following questions:
Q1: How does DexDeform compare against trajectory optimization, imitation learning, and RL approaches?
Q2: How much improvement does differentiable physics guided exploration bring for solving dexterous deformable manipulation tasks?
Q3: What are the benefits of the skill model?
Q4: Are skill dynamics predictions consistent with the resulting states from applying the decoded actions?
Q5: What does the latent space of skill embedding look like?
3.1
ENVIRONMENTAL. SETUP
Tasks and Environments. Inspired by human dexterous deformable object manipulation tasks, design six tasks (Fig. 1): three single-hand tasks (Folding, Wrap, Flip), including in-hand
5

---

Published as a conference paper at ICLR 2023
manipulation, and three dual-hand tasks (Rope, Dumpling, Bun). Detailed descriptions of our environments and tasks can be found in Appendix A. Each Shadow hand has 28 degrees of freedom with movable base.
Human Demonstration Collection. Using our telcoperation setup described in Section 2.1, We collected 10 demonstrations for each task variant. There exist 4 variants for Folding, corresponding to left, right, front, and back folding directions. All other tasks have 1 task variant. The total demonstration amounts to approximately 60, 000 environment steps, or hours of human interactions with the environment.
Evaluation metric. We report the normalized improvement (i.e., decrease) in Earth Mover distance (EMD) computed as d(()) do-d: where do and de are the initial and current values of EMD. Hence, de
normalized improvement score of 0 represents a policy that results in decrease in the EMD, while a score of indicates that the policy is able to result in a shape that matches perfectly with the goal. We threshold the minimum of the score be O, as negative distances could occur if the policy results in shapes further away from the goal shape than the initial one. We approximate the EMI) using Sinkhorn Divergence (Séjourné et al., 2019) between the source and target particles to quantify the fine-grained difference between state and a goal.
Baselines. We consider four categories of baselines:
Model-free Reinforcement Learning. We compare against Proximal Policy Optimization (PPO) (Schulman et al., 2017), an model-free RL method. The RL agent takes point cloud as input. Behavior Cloning. We compare with baseline that directly learns goal-conditioned policy with Behavior Cloning (BC) and hindsight relabeling. The agent is trained with the same human demonstration set and takes point cloud as input.
Model-free Reinforcement Learning with Data Augmentation. We compare with demonstration augmented policy gradient (DAPG), method that combines demonstrations with an RL agent (PPO). We train the agent using the same demonstration set with point cloud as input observations. Trajectory Optimization. We compare against the trajectory optimization (TrajOpt) that uses first-order gradients from a differentiable simulator to solve for an open-loop action sequence (Kelley, 1960). This method takes the full state of the simulation as the observation at each timestep.
3.2 OBJECT MANIPULATION RESULTS
Given a goal configuration of deformable shapes, our objective is to perform dexterous deformable object manipulation using the Shadow Dexterous hand. We created five goal configurations for each task to evaluate different approaches. We report the mean and standard deviation of the normalized improvement (Q1). We show the quantitative results in Table and the qualitative results in Figure 3.
We find that DexDeform is capable of completing the challenging long-horizon dexterous manip- ulation tasks, and significantly outperforms the compared baselines. On the challenging in-hand manipulation task Flip, we find that all baseline approaches fail to complete the task, while DexDe- form is able to swiftly flip the wrapper into the air with fingertips and deform the dough. We hypothesize that the success of DexDeform comes from the ability to leverage the skill model for decomposing the high dimensional state space, which allows for efficient planning. On single-hand task Folding, we find that the BC agent would fold the dough in the wrong direction, while such behavior is not found for DexDeform. We hypothesize that this because DexDeform able to leverage the translational equivariance offered by the implicit representation during planning. DAPG agent is able to squeeze the dough and move towards a location that best matches the general shape, but is unable to dexterously fold the dough over. PPO agent and TrajOpt agents are unable to squeeze or create meaningful shapes, and would slightly move the initial dough towards the target shape. On dual-hand task Rope, we find that DexDeform is able to match the shape with fine- fine-grained details. The BC agent is able to generally match the shape, while DAPG, PPO, and TrajOpt agents fail to create meaningful shapes due to the high dimensional space created by two Shadow hands and two deformable objects. Due to the sample complexity of RL approaches, the speed of soft-body simulation limits the speed of convergence. We believe that with large amount of samples, the performance of RL agents should improve, constituting a novel future direction.

---

Published as a conference paper at ICLR 2023
Initial State
DexDeform
BC
DAPG
TrajOpt
Goal
Figure 3: Qualitative results of each method four environments: Folding, Rope, Dumpling, Flip (from top to bottom). The robot hand not rendered for the first three environments to avoid occlusion of the final shape.
Env
Folding
Rope
Bun
TrajOpt
0.032 + 0.061
0.079 + 0.026
0.000 + 0.000
PPO
0.361 + 0.173
0.460 + 0.257
0.069 + 0.117
DAPG
0.538 + 0.308
0.246 + 0.626
0.460 + 0.079
BC
0.685 0.388
0.557 0.377
0.379 + 0.258
DexDeform
0.970 0.021
0.972 0.010
0.874 - 0.078
Env
Dumpling
Wrap
Flip
TrajOpt
0.000 + 0.000
0.000 +: 0.000
0.195 + 0.275
PPO
0.000 + 0.000
0.000 + 0.000
0.223 + 0.328
DAPG
0.000 + 0.000
0.000 +: 0.000
0.000 + 0.000
BC
0.506
0.134 0.595
0.253 0.359
DexDeform
0.888 + 0.055
0.845 + 0.050
0.842 + 0.057
Table 1: The averaged normalized improvements and the standard deviations of each method.
3.3 ABLATION ANALYSIS OF DEXDEFORM
To quantify the improvement brought by differentiable physics (Q2), perform ablatively compare DexDeform against a baseline, named Skill-Only, that does not use differentiable physics for explo- ration. In a fashion similar to previous table, report the normalized improvement in Table 2. We find that the Skill-Only agent, trained entirely on the initial human demonstrations, unable to generalize across evaluation goals that are uncovered by the initial dataset. Gradient-basee trajectory optimization leveraged the interactions with the environment and exploited the gradient information achieve fine-grained control the soft-body shapes.
To evaluate the benefits of the skill model (Q3), we perform an ablation that compares Dex with a baseline (NN-TrajOpt) that replaces the skill model with a heuristic. Given a goal shape, NN-TrajOpt uses EMD to find the nearest neighbor of that shape from the initial human demonstration data. NN-TrajOpt then uses gradient- based optimizer to refine the corresponding trajectory of this nearest neighbor. We report the qualitative comparison in Figure 4. We illustrate that pure EMD might not be a good measure for soft bodies with large topological variations Feydy (2020). contrast, Dexorm leverages skill embedding and is able to compositionally represent the deformation process, allowing for finding the suitable policy.

---

Published as a conference paper at ICLR 2023
DexDeform NN-TrajOpt
Initial State
NN-found Goal
Figure 4: Ablative comparison with NN-TrajOpt that replaces skill model with heuristic.
Eny
Folding
Rope
Bun
Dumpling
Skill-Only
0.908 + 0.058
0.914 + 0.023
0.820 + 0.008
0.725 + 0.244
DexDeform
0.970 + 0.021
0.972 + 0.010
0.874 + 0.078
0.888 + 0.055
Table 2: Ablative comparison with Skill-Only that does not use the gradient- -aaed optimizer. We show the averaged normalized improvements and the standard deviation of each method.
3.4 SKILL MODEL VISUALIZATION
To see whether skill dynamics predictions are consistent with action rollouts (Q4), we visualize the latent encoding of the scene $t. predicted by the skill dynamics model given skill embedding, as well as the ground truth state S. obtained from actions predicted by the skill decoder. Ideally, the two visualizations should show consistency. As shown in Figure 7, we observe high level of consistency between the skill dynamics and the skill decoder. To find out what the latent space of skill embedding looks like (Q5), we visualize the skill embeddings using t-distributed stochastic neighbor embedding (t-SNE) (van der Maaten & Hinton, 2008) on Folding. We label each embedding based on the location of the final shape achieved by the corresponding skill sequence. We partition the ground plane into five parts: left, right, front, back, and center. As shown in Figure 8, the skill embeddings correlated with the label categories. Details and visualizations can be found in Appendix E.
RELATED WORK
Dexterous Manipulation. Dexterous manipulation has been long-standing challenge in roboties, with the early works dating back to Salisbury & Craig (1982); Mason et al. (1989). Different from parallel-jaw grasping, dexterous manipulation typically continuously controls force to the object through the fingertips of robotic hand (Dafle et al., 2014). There have been many prior works on using trajectory optimization (Mordatch al., 2012: Bai & Liu, 2014; Sundaralingam & Hermans, 2019) kinodynamic planning (Rus, 1999) to solve for the controllers. However, to make the optimization or planning tractable, prior works usually make assumptions on known dynamics properties and simple geometries. Another line of works uses reinforcement learning to train the controller. Some model-based RL works learned a dynamics model from the rollout data (Kumar al., 2016; Nagabandi et al., 2020), and used online optimal control to rotate pen Baoding balls on a Shadow hand. OpenAl et al. (2020; 2019) uses model-free RL to learn a controller to reorient cube and transfer the controller to the real world. To speed up the policy learning when using model-free RL, Chen et al. (2022) uses a teacher-student framework to learn a controller that can reorient thousands of geometrically different objects with both the hand facing upward and downward. (Radosavovic et al., 2020; Zhu et al., 2019; Rajeswaran et al., 2017; Jeong et al., 2020; Gupta et al., 2016; Qin al., 2021) bootstraps the RL policy learning from demonstration data for reorienting pen, opening door, assembling LEGO blocks, etc. Handa et al. (2020); Arunachalam et al. (2022); Sivakumar et al. (2022) developed a teleoperation system for dexterous manipulation by tracking hand pose and re-targeting it to robot hand. Unlike previous works with rigid bodies, our work performs the first investigation on the learning- based dexterous manipulation of soft bodies that carries infinite degrees of freedom, and provides a differentiable simulation platform for teleoperation.
8

---

Published as a conference paper at ICLR 2023
Learning Skills from Demonstrations. Our skill model shares the same spirits with hierarchical
imitation learning (Fang et al., 2019; Shi et al., 2022b; Gupta et al., 2019; Lynch et al.. 2020)
and motion synthesis (Peng et al., 2018; 2022), which view skill learning sequential modeling tasks (Janner et al., 2021; Chen et al., 2021) in a low-dimensional space. Following Shi et al. (2022b), we learn a latent dynamic model compose skills with model-based planning (Hafner al., 2019; 2020) in the latent space. We employ these ideas for deformable object manipulation, where we integrate skill abstraction and latent dynamies into our pipeline. Our additional innovation is an exploration phase guided by the gradient- based trajectory optimizer, learning dexterous soft-body manipulation skills with a small number of demonstrations.
Deformable Object Manipulation. Deformable object manipulation have attracted great attention because of its wide range of applications in the real world. Previous works have explored manipulating different materials from objects humans interact with on a daily basis, including cloth (Maitin-Shepard et al., 2010; Hoque et al., 2021; Lin et al., 2021; Huang et al., 2022; Weng al., 2021; Liang et al., 2019; Wu et al., 2020). rope (Sundaresan et al., 2020; Mitrano et al., 2021; Yan et al., 2020; Wu et al., 2020), and fluid materials (Ma et al., 2018; Holl et al., 2020; Li et al., 2022b; Schenck & Fox, 2017; Gautham et al., 2022). Our work is built upon Huang et al. (2020), which uses the MPM (Jiang et al., 2016) to simulate elastoplastic objects (Huang et al., 2020; Li et al., 2019; Shi et al., 2022a; Figueroa al., 2016; Mail & Bajesy, 2021; Heiden et al., 2021), and is able to represent materials such as dough and clay. Different from previous works, we investigate how to interact with deformable objects using multi- fingered hands, which carry versatility across different scenarios.
Differentiable physics. The development of differentiable simulator (Bern et al., 2019; Geilinger al., 2020; Liang al., 2019; Hu et al., 2019b;;; Huang et al., 2020; Qiao al., 2021: Du et al., 2021; Heiden et al., 2019; Geilinger et al., 2020; Werling et al., 2021; Howell et al., 2022) enables fast planning (Huang et al., 2020), demonstration generation (Lin et al., 2022) and adaptation (Murthy et al., 2020). Systems have been developed to generate high-performance simulation code for the support of automatic differentiation (Hu et al., 2019a; 9a; Macklin, 2022; Freeman et al., 2021). However, many works have discovered that trajectory optimizers with first-order gradients are sensitive to local optima (Li et al., 2022a; Suh et al., 2022; Xu et al., 2022; Antonova et al., 2022). Many have found that the gradient-based optimizer can benefit from the integration of sampling-based methods, which enables global search to escape from local optima. The skill model employed by our method can be viewed as form of planning. Different from previous methods, the skill model can decompose the high dimensional policy space, which enables efficient planning in the latent skill space.
5 CONCLUSION
In this work, we perform, to the best of our knowledge, the first investigation of the learning- based dexterous manipulation of deformable objects. We build platform that integrates low-cost teleoperation with a soft-body simulation that is differentiable. We propose DexDeform, a principled framework that abstracts dexterous manipulation skills from human demonstrations, and refines the learned skills with differentiable physics. We find that DexDeform outperforms the baselines and accomplishes all six challenging tasks.
There are few interesting directions for future work. With our simulation platform, it would be interesting to leverage the vast amount of in-the-wild videos (e.g., making bread, stuffing dumpling. building sand castle) for learning dexterous deformable manipulation policies in the future. It is also intriguing to speed up the soft-body simulation for large-scale learning with RL. Our work assumes full point cloud observation. Although our choice of implicit representation has been shown to transfer from the simulation into real-world robotic deployment by Shen et al. (2022), we would like to work with real-world observations in the future.
Acknowledgement. This project was supported by the DARPA MCS program, MIT-IBM Watson AI Lab, and gift funding from MERL, Cisco, and Amazon.
REFERENCES
Rika Antonova, Jingyum Yang, Krishna Murthy Jatavallabhula, and Jeannette Bohg. Rethinking opti- mization with differentiable simulation from a global perspective. arXiv preprint arXiv:2207.00167, 2022. 9

---

Published as a conference paper at ICLR 2023
Sridhar Pandian Arunachalam, Sneha Silwal, Ben Evans, and Lerrel Pinto. Dexterous imitation made easy: A learning-ber basedd framework for efficient dexterous manipulation. arXiv preprint arXiv:2203.13251, 2022. 8
Yunfei Bai and C Karen Liu. Dexterous manipulation using both palm and fingers. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1560-1565. IEEE, 2014. 8
James M Bern, Pol Banzet, Roi Poranne, and Stelian Coros. Trajectory optimization for cable-driven soft robot locomotion. In Robotics: Science and Systems, 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp- 15084-15000 Curran As- sociates, Inc., 2021. URL https: / /ppoce/ee/ccooceoeeeeeeooo neurin neurips cc/ httppppppr/2//////// 7f489f642
Pape:e paf. 9
Tao Chen, Jie Xu, and Pulkit Agrawal. system for general in-hand object re-orientation. In Conference on Robot Learning, pp. 297-307. PMLR, 2022. 1, 8
Özgün Çişek, Ahmed Abdulkadir, Soeren S Lienkamp. Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In International conference medical image computing and computer-assisted intervention, pp. 424 432. Springer, 2016. 16
Nikhil Chavan Dafle Alberto Rodriguez, Robert Paolini, Bowei Tang, Siddhartha Srinivasa, Michael Erdmann, Matthew T Mason, Ivan Lundberg, Harald Staab, and Thomas Fuhlbrigge. Extrinsic dexterity: In-hand manipulation with external forces. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1578-1588 1585. IEEE, 2014. 8
Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojeiech Matusik. Diffpd: Diffcrentiable projective dynamics. ACM Transactions on Graphics (TOG), 41 (2):1-21, 2021.
Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Dynamics learning with cascaded variational inference for multi-step manipulation. arXiv preprint arXiv:1910. 13395, 2019.
Thomas Feix, Javier Romero, Heinz-Bode Schmiedmayer, Aaron M Dollar, and Danica Kragic. The grasp taxonomy of human grasp types. IEFE Transactions on tuman-machine systems, 46(1): 66-77, 2015.
Jean Feydy. Geometric data analysis, beyond convolutions. PhD thesis, Université Paris Sacla- Yvette France, 2020.
Nadia Figueroa, Ana Lucia Pais Ureche, and Aude Billard. Learning complex sequential tasks from demonstration: A pizza dough rolling case study. In 2016 11th ACM/IEEE International Conference on
Interaction (HRI), pp. 611-612. Ieee, 2016. 9
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.1 1 13281, 2021. 9
Narasimhan Gautham, Zhang Kai, Eisner Ben, Lin Xingyu, and Held David. Transparent liquid segmentation for robotic pouring. In International Conference on Robotics and Automation (ICRA), 2022. 9
Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski. and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. arXiv preprint 00Xiv:2007.00988, 2020. 9
Abhishek Gupta, Clemens Eppner. Sergey Levine, and Pieter Abbeel. Learning dexterous manip- ulation for soft robotic hand from human demonstrations. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3786-3793, IEEE, 2016. 4,
10

---

Published as a conference paper at ICLR 2023
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- ong-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956. 2019. 9
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01600 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02//90 2020. 9
Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu- Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff,, and Dieter Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand- arm system. In 2020 IEFE International Conference Robotics and Automation (ICRA), pp. 9170- IEEE, 2020. 4,
Eric Heiden, David Millard, Hejia Zhang, and Gaurav S Sukhatme. Interactive differentiable simulation. arXiv preprint arXiv: 1905. 10706, 2019. 9
Eric Heiden, Miles Macklin, Yashraj S Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Robotics:
Science and Systems, July 2021. doi: N0.155607RSS.000 20...2222.1..... XVII1067. 9
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997. 16
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning to control pdes with differentiable physics. International Conference on Learning Representations, 2020. 9
Ryan Hoque, Daniel Seita, Ashwin Balakrishna, Aditya Ganapathi, Ajay Tanwani, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. VisuoSpatial Foresight for Physical Sequential Fabric Manipulation. Autonomous Robots (AURO) journal, 2021. 9
Taylor A Howell, Simon Le Cleac' h. Zico Kolter, Mac Schwager, and Zachary Manchester. Dojo: A differentiable simulator for robotics. arXiv preprint arXiv:2203. 00806, 2022. 9
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions Gruphics (TOG), 37(4): 2018. 15
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Bagan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint McXiv:191000093: 2019a. 9
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simulator for soft robotics. In 2019 International Conference on Robotics and Automation (ICRA), pp. 6265- 6271. IEEE, 2019b. 9
Zhiao Huang, Yuanming Hu, Tao Du Siyuan Zhou, Hao Su, Joshua Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In International Conference on Learning Representations, 2020. 9
Zhiao Huang, Yuamming Hu, Tao Du. Siyuan Zhou, Hao Su, Joshua Tenenbaum, and Chuang Gan. Plasticinelab: A body manipulation benchmark with differentiable physics. arXiv preprint (04i:2/04.0333) 2021. 2, 15
Zixuan Huang, Xingyu Lin, and David Held. Mesh-based dynamics model with occlusion reasoning for cloth manipulation. Robotics: Science and Systems (RSS), 2022. 9
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 14431133-828 2021. 9
Rae Jeong, Jost Tobias Springenberg, Jackie Kay, Daniel Zheng, Yuxiang Zhou, Alexandre Galashov, Nicolas Heess, and Francesco Nori. Learning dexterous manipulation from suboptimal experts. arXiv preprint arX00:2010.08587, 2020. 8

---

Published as a conference paper at ICLR 2023
Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials. In ACM SIGGRAPH 2016 Courses, pp. 1-52. 2016. 9
Henry J Kelley. Gradient theory of optimal flight paths. Ars Journal, 30(10):947-954, 1960. 6, 20
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 378-383. IEEE, 2016.
Sizhe Li, Zhiao Huang. Tao Du Hao Su, Joshua B Tenenbaum, and Chuang Gan. Contact points discovery for soft-body manipulations with differentiable physics. In International Conference Learning Representations, 2022a. 2,
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019. 9
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. representations for visuomotor control. In Conference Robot Learning, pp. 11 112-123, PMLR, 2022b.
3d neural scene
Junbang Liang, Ming C Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. Advances in Neural Information Processing Systems, 2019. 9
Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, 2021. 9
Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B Tenenbaum, David Held, and Chuang Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. arXiv preprint ar 12X.:2203..1273. 2022. 9.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113-1132. PMLR, 2020. 9
Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. Fluid directed rigid body control using deep reinforcement learning. ACM Trans. Graph., 37(4), July 2018. ISSN 0730-0301. doi: 10. 10.1145/3197511.2220334 URL https: //tp:::////oii/// wwt.:/oo.ooooooooooooooooooooeoe org / 11 /11111/31119 9157177.977. 3201334.
Miles Macklin. Warp: A high-performance python framework for gpu simulation and graphics. https: //ghttutb././//////////// cww/moooiiii/ceeeeeooo com/nvidia/warp March 2022. NVIDIA GPU Technology Confer- ence (GTC). 9
Jeremy Maitin-Shepard, Marco Cusumano-Towner, Jinna Lei, and Pieter Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pp. 2308-2315, IEEE, 2010. URL
https: eeeexpllre.eeeeeoeeeeeeeeeeeeeee ieee. org/ abstract documentt /5509499.99 9
Matthew T Mason, Kenneth Salisbury, and Joey K Parker. Robot hands and the mechanics of manipulation. 1989. 8
Carolyn Matl and Ruzena Bajesy. Deformable elasto -pasttc object shaping using an elastic hand and model-based reinforcement learning. In 202/ IEEF/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3955-3962. IEEF, 2021. 9
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF/CVF conference computer vision and pallern recognition, pp. 4460-4470, 2019. 18
Peter Mitrano, Dale McConachie, and Dmitry Berenson. Learning where to trust unreliable models in an unstructured world for deformable object manipulation. Science Robotics, 6(54):eabd8 170, 2021. 9
12

---

Published as a conference paper at ICLR 2023
Igor Mordatch, Zoran Popović, and Emanuel Todorov. invariatn optimization for hand manipulation. In Proceedings of the ACM SIGGRAPHHEungraphiis symposium on computer animation. pp. 137-144, 2012. 8
J Krishna Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-Lévesque Kevin Xie, Kenny Erleben, et al. gradsim: Differ- entiable simulation for system identification and visuomotor control. In International Conference on Learning Representations, 2020. 9
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1111 PMLR, 2020.
1, 8
OpenAI, llge Akkaya, Marein Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint
13, 2019. 8
OpenAI, Marcin Andrychowicz. Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020. 1, 8, 8
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, pp. Springer,
2020. 4
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4))1-14, 2018.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. arXiv preprint 2rXXix2205.01906 2022. 9
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017. 15
Yiling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft mody systems. Advances in Neural Information Processing Systems, 14:17123-17135, 2021.
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. arXiv preprint 1XXi:22008.0587, 2021. 8
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. arXiv preprint arXiv:2004. 200.:200000005500, 2020. 8
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 2017. 2, 4, 21
Daniela Rus. In-hand dexterous manipulation of piecewise-smooth 3-d objects. The International Journal of Robotics Research, 18(4):355-381, 1999. 8
J Kenneth Salisbury and John Craig. Articulated hands: Force control and kinematic issues. The International journal of Robotics research, 1(1):4-17, 1982. 8
Connor Schenck and Dieter Fox. Visual closed-loop control for pouring liquids. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2629-2636. IEEE, 2017. 9
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017, 6
13

---

Published as a conference paper at ICLR 2023
Thibault Séjourné, Jean Feydy, Trançois-Xavier Vialard, Alain Trouvé, and Gabriel Peyré. Sinkhorn divergences for unbalanced optimal transport. arXiv preprint arXiv:I910. 12958, 2019.
ShadowRobot. Shadow dexterous hand. 2013.
Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J Guibas, Silvio Savarese, Anima Anand- kumar, and Yuke Zhu. Acid: cetion-conditional implicit visual dynamics for deformable object manipulation. arXiv preprint arXiv:2203.0688. 2022. 9
Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning simulate, and shape clasto-plastic objects with graph networks. Robotics: Science and Systems (RSS), 2022a. 9
Lucy Xiaoyang Shi, Joseph Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07566 2022b. 4, 9, 16
Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: learning a robotic hand
imitator by watching humans on youtube. arXiv preprint arXiv:2202. 10448, 2022. 2, 8
Mischa Spiegelmock. Leap motion development essentials. Packt Publishing Ltd, 2013. 4, 15
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference Machine Learning, pp. PMLR, 2022. 9
Balakumar Sundaralingam and Tucker Hermans. Relaxed-rigidity constraints: kinematic trajectory optimization and collision avoidance for in-grasp manipulation. Autonomous Robots, 43(2): 469-4833 2019. 8
Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph E Gonzalez, and Ken Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9411 1411-94 IEEE, 2020. 9
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma- chine Learning Research, 9:2579-2600 2605, 2008. URL http:/ /www. jmlr. org. htgg/papers/v//
wandermaaten html. 8, 18
Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, and David Held. Fabricflownet: Bimanual cloth manipulation with flow-based policy. Conference on Robot Learning, 2021.
Keenon Werling, Dalton Omens, Jeongseok Lee, Ioannis Exarchos, and Karen Liu. Fast and feature-complete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103.16021, 2021. 9
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations, 2020.
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint 001:022004071377, 2022. 9
Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg. Self-supervised learning of state estimation for manipulating deformable linear objects. IEEE Robotics and Automation Letters, 5(2):3372- 2379, 2020. 9
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous ma- nipulation with deep reinforcement learning: Efficient, general, and low-cost. 20/9 International Conference on Robotics and Automation (ICRA), pp. 3651-3657. IEFE, 2019. 8
14

---

Published as a conference paper at ICLR 2023
A ENVIRONMENT DETAILS
Here, we provide descriptions of each task covered in Figure 1. These descriptions (e.g., which hand use first, where to grasp) are made based how human demonstrations are collected. There could exist many other solutions for each task.
Folding The agent is spawned above the dough wrapper, and needs to fold the wrapper to four general directions, including front, back, left, and right. The task length is 250 steps.
Wrap The agent needs to first grasp and move the plasticine ball onto the rope, and then pinch on the side of the rope to wrap the ball over. The task length 500 steps.
Flip The agent needs to swiftly flip the dough wrapper into the air, and deform and reorient it with agility. The task length is 500 steps
Bun The agent needs use two hands simultaneously pinch and push the wrapper into bun-shaped object. The task length is 250 steps
Rope The agent needs to use the right hand to grasp onto the rope on the right, lift and place above the rope on the left. Finally, the left hand needs to bend the rope that was originally on the left. The task length is 250 steps.
Dumpling The agent needs first the right hand to grasp onto the right side of the wrapper. While holding the dumpling with the right hand, the left hand needs to lift the left side. Finally, the hands need bring together the dough into a dumpling-shaps object. The task length 250
steps.
We build our simulation enviromments on top of PlasticineLab (Huang et al., 2021), differentiable physics simulator based on the MLS-MPM algorithm (Hu et al., 2018).
For single hand environments, Folding and Wrap have an action dimension of 26 (20 for actuators for finger joints and wrists, and 6 for base). For in-hand manipulation environments, we assume single hand with fixed base that has 20 action dimension. All dual hand environments have action dimension of 52 with movable bases.
A.1 DETAILS ON ROBOT ACTIONS
The robot actions in our simulated environment correspond the relative change in the joint angles on each actuated joint. Le., 9:+1 oe joine + az
A.2 DETAILS ON POINT CLOUD OBSERVATIONS
For the point clouds, we assume full point cloud observation of the scene from MPM simulation, which includes the multi- multi-fingered hand(s) and the object(s). We have also conducted experiments taking partial point clouds observed through four RGBD camera viewpoints as inputs. The experiment results and interpretations reported in Appendix H.
B DETAILS ON TELEOPERATION
Our teleoperation system is based on the Leap Motion Controller (Spiegelmock, 2013), an optical hand tracking device. By constructing an inverse kinematics model based on the Shadow hand, our system re-targets the detected human finger positions into the joint positions of the simulated robot hand that is controlled via a position-based PD controller. Our system runs teleoperation, simulation, and rendering with multiprocessing, and achieves 15-20 FPS on a laptop with NVIDIA GeForce RTX3070 Laptop GPU. Our system setup is illustrated below in Figure 5.
C DETAILS ON SKILL MODEL TRAINING AND PLANNING
C. IMPLICIT SCENE REPRESENTATION
ConvONet Encoder. A shallow PointNet encoder Qi al. (2017) first maps the three-dimensional input coordinates and their rgb colors into feature space. The PointNet encoded features are then
15

---

Published as a conference paper at ICLR 2023
Leap Motion Tracker
Figure The hardware setup of our teleoperation system with the Leap Motion Tracker with human operator. The computer is hidden for better illustration of the system.
projected orthographically onto three canonical planes that represent xz, xy, and yz planes. The triplane 2D feature grids are then passed into U-Net Çięek al. (2016). Concretely, choose use triplane resolution of 128 with hidden dimension equals to 64, and the depth of U-Net is 5.
ConvONer Decoder. Since the triplane feature grids represent parameterization of the scene, we can obtain the occupancy prediction (empty space, hand, or object) at any query point. The implicit scene decoder first uses bilinear interpolation to retrieve the local feature of the query point from each of the triplanes. The interpolated features from different planes are then summed and passed to a multi-layer perceptron (MLP) to decode into the final occupancy prediction. In practice, we use 5-layer MLP with a hidden dimension of 32.
Latent Encoding of the Scene. We use convolutional VAE to encode the triplanes from ConvONet encoder into latent encoding, and to decode latent code back into the space of implicit scene representations. Concretely, the VAE has an encoder with 6 convolutional layers with a kernel size of 3 and stride of 2, and channel size of (64, 64, 128, 256, 512, 512), followed by an MLP that maps the feature into a latent code 128 dimensions.
Training Details. We train the ConvONet model for 10, 000 iterations, and pretrain the latent scene VAE for 50, 000 iterations.
C.2 SKILL MODEL
Input State-Action Sequence. We use K 10 in practice, which is the abstraction length. During training, each sample is sequence of 50 consecutive state-action pair, which can be encoded into skill abstractions.
Skill Encoder.
We use 5 layer LSTM Hochreiter & Schmidhuber (1997) with dimension of 128 to encode the input sequence of state-action pairs. Following the VAE training paradigm, the encoded feature then passed through MLP that maps the feature into latent code of 128 dimensions.
Skill Dynamics Predictor. The dynamics predictor 3-layer MLP with dimensions (512, 256, 128).
Skill Action Decoder. The action decoder is 3-layer MI.P with dimensions (512, 256, act_ act_dim), Where the action dimension depends whether the task uses single hand or dual hands.
Training Details. Following SkiMo (Shi et al., 2022b), we jointly train the skill encoder, dynamics predictor, and action decoder with the following objectives for every skill embedding We first present the objective function for the skill encoder and skill action decoder trained under the VAE setting.
16

---

Published as a conference paper at ICLR 2023
LVAA (s.a))
(z) - a:)² + + (x)
(2)
Let N = 5 be the number of skill abstractions obtained from each sample that is of 50 steps. We show the objective function for skill dynamies
DDyn x(
Σ [[(441))
5(++1)++1lp
Combining the objectives above, we jointly train the skill encoder, dynamics predictor, and action decoder.
Sam
LSAA + LDyn
Planning Details. Given a target shape, we iteratively plans actions using the optimization problem described in Equation 1. We apply the following algorithm to plan and execute actions. The initial batch of skill sequences is sampled from the standard Gaussian distribution.
Algorithm 1 Planning actions with skill model
Input: Goal shape g, approximate task horizon H, abstraction length K, number of abstractions to execute each time M
1: remain < [H/K]
2: while remain > 0 do:
3: Sample batch of {21, 25, ) from standard Gaussian initialize;
4: Optimize batch of {=1, ,,,... ZHr 1,j=1 according to Eqn. and choose the best sequence; )
5: Use to decode the first M skills into M M actions to execute;
6: remain +- remain -M;
7: end while
D DETAILS OF DEXDEFORM
(A.1) Learning implicit representation of the scene.
(B.1) Planning with skill abstractions.
YZ plane
Viase (p. (ane)t))
S
Initialbe z's from standard normal
(B.2) Decoding actions
XZ plane
plane
Zt
Skill Decoder
Learning Occupancy
Forward
Gradient
Scene (a)))
Senc
Skill Dynamics
Kistap action trajectory
Learning latent
SttK
scene embedding
(A.2) Learning skill abstractions from demonstrations.
Zt+K
Skill Decoder
Forward
Gradient
(S Se4x.
Skill Dynamics (
Skill Dynamics
K-step
trajectory
Sr+2K
Optimize s using Eqn.
Goal
Latent Skill
Space
Skill Decoder m(0.k...5)
Figure 6: An overview of training (A) and inference (B) DexDeform. Demonstration collection using teleoperation and differentiable physics guided exploration are omitted.
The DexDeform algorithm presented in Algo. 2, contains the training phase and exploration phase. The training phase involves learning implicit representation of the scene and the skill model, while the exploration phase leverages differentiable physics optimizer for expanding the demonstration set
17

---

Published as a conference paper at ICLR 2023
to goals. We start with collection of human demonstrations and iteratively execute the training phase and the exploration phase to explore novel goals. In practice, for each iteration, we generate trajectory from each original human demonstration based goal augmentations, In Fig. we illustrate the details of training and inference procedures Deform...
Algorithm 2 Iterative learning with DexDmform
Input: human demonstrations, number of iterations N, new trajectories per iteration M,
1: Initialize dataset D with human demonstrations, do
2: for 4-
3: 'Train implicit representations and skill model J.T, using D until convergence;
4: m <- 0;
5: while < M do:
Exploration Phase
6:
Sample goal from D and apply augmentation to get g:
7:
Optimize batch of {21, ........ 21 ) according to Eqn. and choose the best sequence;
Use and Algo. to decode skill sequence into trajectory Refine the trajectory with differentiable physics to get F; 10: if Dema (ŌH, g) < then
8:
{01 41, OH, 0z )
9:
If the shape difference threshold is satisfied
11:
Store F in dataset D) m 6 + 1; 13: end if
end while
15:
end for
E SKILL VISUALIZATION DETAILS
E. SKILL DYNAMICS
In Figure 7, to create the visualizations of 6t. decode the latent encoding of the scene into 2D feature grids. Since the feature grids represent an implicit parameterization of the scene with occupancy information, we apply Multiresolution IsoSurface Extraction (MISE) (Mescheder et al., 2019) to extract meshes of the scene predicted in imagination.
Skill Dynamics
Skill Decoder
$10
$40
8
50
Initial State
Goal
$10
San
S
Sgo
Figure 7: We visualize the skill dynamic predictions on Folding, and show consistency between the predicted state and the resulting states from actions predicted by the skill decoder.
E.2 TSNE VISUALIZATION
We visualize the skill embeddings using t-distributed stochastic neighbor embedding (t-SNE) (van der Maaten & Hinton, 2008) on Folding. We label each embedding based on the location final shape achieved. We partition the ground plane into five parts: left, right, front, back, and center. As shown in Figure 8, we see that the skill embeddings are correlated with the label categories. Our t-SNE is with million iterations.
18

---

Published as a conference paper at ICLR 2023
0.8
0.4
left
back
0.2
right
center
0.0
0.2
0.4
0.6
0.8
1.0
Figure
Skill Tsne Visualization on Folding
F RESULTS ON DEXDEFORM WITH PARTIAL POINT CLOUD
We have conducted experiments taking partial point clouds observed through four RGBD camera viewpoints as inputs. The four cameras are placed on the front, back, left, and right sides of the tabletop (ground plane). We report the averaged normalized improvements Table
Env
Folding
Rope
Bun
Dumpling
Wrap
Flip
DexDeform w/ perfect pointeloud
0.970
0.972
0.874
0.888
0.845
0.842
DexDeform w/ partial pointcloud
0.954
0.961
0.814
0.785
0.809
0.683
Table 3: The averaged normalized improvements of each method.
In general, we find that our use of implicit scene representation allows for generalization across partial point clouds with varying number of points observed in each frame.
G RESULTS ON BASELINES WITH LATENT SCENE EMBEDDING
We have conducted experiments where the baseline models use the same latent embeddings as our proposed methods. We report the averaged normalized improvements each approach Table For convenience, we have included the original baselines and our approach.
Eny
Folding
Rope
Bun
Dumpling
Wrap
Flip
PPO w/o latent
0.361
0.460
0.069
0.000
0.000
0.223
PPO w/ latent
0.630
0.531
0.000
0.100
0.082
0.270
DAPG w/o latent
0.538
0.246
0.460
0.000
0.000
0.000
DAPG latent
0.459
0.544
0.294
0.272
0.183
0.000
BC w/o latent
0.685
0.557
0.379
0.506
0.131
0.253
BC w/ latent
0.672
0,524
0,000
0.145
0.433
0.326
DexDeform
0.970
0.972
0.874
0.888
0.845
0.842
Table 4: The averaged normalized improvements each method.
In general, find that the inclusion of latent embeddings did not significantly improve the performance the baseline methods. We speculate that the policy characteristics the baselines are more responsible for the performance than the designs of their perception modules.
Specifically, for the RL approaches (PPO & DAPG), the sample complexity becomes a limitation under dexterous deformable object manipulation due the large state space. Behavior cloning
19

---

Published as a conference paper at ICLR 2023
struggles to generalize across different goals. Because BC lacks the ability to compositionally represent and reason about the deformation process, which is an advantage of skill-based planning used by method.
H RESULTS ON SKILL MODEL WITH LEARNED PRIOR
We have conducted additional experiments where prior network fp(2s]/er (te, Qe ae...., 1.. K ax-K) is learned jointly with the posterior network [g(2, h,, at...., llt K: 04+K)) (i.e., the skill encoder) using KL divergence. Concretely,
implemented the prior network using LSTM architecture. During our iterative planning (Algo. I), we maintain a 10-step history of past state embeddings and actions from executing M skill latents in the previous planning iteration. For the current planning iteration, we use the learned prior to sample the batch of future skill latents for initializing the optimization in Eqn. For the initial planning iteration, we use the standard Gaussian prior to sample the skill latents for optimization.
We observed that the two approaches arrive at similar performances. We think an intriguing future direction to incorporate temporal skill prior and consider other forms of exploration for tackling deformable object manipulation.
Env
Folding
Rope
Bun
Dumpling
Wrap
Flip
DexDeform w/o learned prior
0.970
0.972
0.874 0.907
0.888 0.814
0.845 0.881
0.842
DexDeform w/ learned prior
0.933
0.739
0.773
Table 5: The averaged normalized improvements of each method.
I DETAILS ON BASELINE EXPERIMENTS
We describe the input type, training (if applicable), and runtime usage of each baseline method below.
Trajectory Optimization (TrajOpt).
Input type: This method takes the full state of simulation as the observation each timestep.
Runtime usage: TrajOpt is not dearning-based. TrajOpt uses first-order gradients from differen- tiable simulator to solve for an open-loop action sequence Kelley (1960).
Model-free Reinforcement Learning (PPO)
Input type: Our PPO agent takes goal-conditioned input. Concretely, suppose that Waeene the point cloud observation of the scene at timestep f,, and Wooo is the point cloud of the goal shape. The input to these baselines at timestep is the goal-conditioned point cloud W. = W.seene goal For each point in the concatenated point cloud, binary label indicates whether the point belongs to the goal.
Training: We use a batch size of 400 and learning rate of 10-4 4 train the agent for 3M steps or till convergence.
Runtime usage: Given goal specified in a point cloud, we create the goal-conditioned input at each timestep to perform rollout using the task-specific horizon.
Behavior Cloning (BC)
Input type: Same PPO described above, our BC agent takes goal-conditioned point clouds as inputs.
Training: We use batch size of 128 and learning rate of 10-4 train the agent for 3M steps or till convergence.
Runtime usage: Given goal specified in a point cloud, we create the goal-conditioned input at each timestep to perform a rollout using the task-specific horizon.
20

---

Published as a conference paper at ICLR 2023
Model-free Reinforcement Learning with Data Augmentation We compare against Demo Augmented Policy Gradient (DAPG) from Rajeswaran et al. (2017), method that combines behavior cloning and online RL.
Input Type: Same as PPO described above, our DAPG agent takes goal-conditioned point clouds inputs.
Training: We initialize the policy with behavior cloning described above Part C. We then apply RL finetuning to the initialized policy using PPO. The DAPG objective in Rajeswaran et al. (2017) (Eq. 6) involves two hyperparameters (Λo, A1) for the weighting function. We use Ap = 0. 0.995. We finetune for 3M steps or till convergence, using a batch size of 400 and learning rate of 3 x 10-4.
Runtime: Given goal specified in a point cloud, we create the goal-conditioned input at each timestep to perform rollout using the task-specific horizon.
21


Reviews:
[{"review_id": "l8--z6NXZqI", "reviewer": "Reviewer_uqMy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a framework for deformable object manipulation with a dexterous multi-finger hand. It consider six tasks in simulation. The proposed framework solves these tasks by first collecting a few demonstrations by teleoperation, then it trains a skill dynamics predictor and action decoder for planning towards a given goal. The skill abstraction is defined by a fixed temporal interval. Finally, the planned action sequence is refined by gradient descent since the simulator is differentiable. The authors demonstrate success in the six tasks and it outputperform several alternatives such as reinforcement learning and imitation learning.", "summary_of_the_review": "In summary, this paper is worth reading for a large part of the community. Though it focuses more on a system design and implementation, it can still benefit the community a lot.\n\n----\nScore Increase from 6 to 8 after discussion.", "strength_and_weaknesses": "Strength:\n1) This paper is (to the best of my knowledge) the first work studies how to do deformable object manipulation with a multi-fingered hand. Previous methods either do deformable object manipulation using parallel jaw grippers or rigid object manipulation with multi-fingered hand.\n2) The writing is pretty clear and complete. It’s easy to read and understand the main message of the paper. The figures are also nice and well-designed.\n\nWeakness:\n1) Since I consider this paper as an engineering effort to build a practical system, I’m worried about if the results can be easily transferred to the real world. Specifically, this paper assumes perfect point clouds both for input and goals. However, it is unlikely to obtain perfect point cloud observations in the real world due to sensor noise. It is also hard for the user to specify point clouds as the end state.\n2) The authors does not promise code release. I think realeasing code upon publication can greatly improve the paper’s impact.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TUlR6Rnf91o", "reviewer": "Reviewer_whUB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the task of dexterous manipulation of deformable objects and introduces a pipeline involving gathering human demonstrations, learning representations for the scene and the demonstrations, and augmenting novel data samples via a differentiable simulator. The proposed pipeline, DexDeform, starts with a small set of demonstrations to learn scene representations and skill abstractions, a dynamics model, and an initial policy. It then iteratively runs training and exploration stages where a gradient-based trajectory optimization is applied via a differentiable simulator to augment available samples. The paper presents evaluations on a variety of tasks, including soft object folding, wrapping, and flipping with a single or dual Shadow hand. ", "summary_of_the_review": "Despite the weaknesses I mentioned, I think it is an overall good paper. It provides useful insights into the problem as well as intriguing solutions. I am leaning toward accepting this paper. I'm looking forward to getting further clarification and experimental evidence from the authors.\n\n-----\n**Post rebuttal update:** The rebuttal addresses my concerns, and I do not have any further questions. I think this is an interesting paper for the community. I am in favor of accepting this paper. I raised my score from 6 to 8.", "strength_and_weaknesses": "Strengths:\n\n1- The paper is well-motivated. The concerns raised at the beginning are later addressed. \n\n2- The optimization objective in Eq. 1 leverages the implicit scene representation, nicely overcoming the challenges of modeling soft objects.\n\n3- Few demonstrations are enough to initialize the models, and it is straightforward to gather them by using the proposed teleoperation setup.\n\n---\n\nWeaknesses:\n\n1- What is the design choice behind using a non-dynamic skill abstraction network? I think a temporal variant with a learnable prior would be more useful and allow for sampling of the next latent sample z. A sampled latent trajectory could be used in the exploration phase instead of augmenting the targets and running optimization in the observation space.\n\n2- It is not clear if the prior of the skill abstraction network is used. I assume the latent skill samples in Eq. 1 are randomly initialized from the standard Gaussian prior. Could the authors clarify this?\n\n3- I think the evaluations could be fairer. The baseline models take the input from the scene in point cloud representation. The implicit scene representation learned by the ConvONet and the scene embeddings learned by the VAE are trained in isolation by using standard training objectives. I believe the baseline models could use the same latent embeddings as the proposed method.\n\n4- This is in line with my previous comment. The baselines and how they are implemented seem rather straightforward. A goal-conditioned policy (see [1*] for grasping rigid objects as an example) could be well suited for the tasks introduced in this paper. The goals could be represented by latent scene embeddings. This is not a weakness per se, but a suggestion. I think stronger baselines would improve the quality of the submission.\n\n[1*] Christen, Sammy, et al. \"D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9E7vKUbyI9Z", "reviewer": "Reviewer_WWJM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the task of dexterous manipulation with human demonstrations. The paper introduces an interesting set of simulated benchmarking tasks, along with a baseline approach to solve these. The benchmark builds on prior work (Plasticinelab), using a simulated shadow robot hand. The proposed baseline relies on human demonstrations captured using a leap motion controller, and mapped to shadow hand using an ik solver. An implicit representation of the scene (observed in point cloud form) is learned using a point cloud encoder decoder model. this representation is then combined with actions, and a sequence of these mapped into another latent space, a dynamics model is learned using this representation, and used for goal-based planning.", "summary_of_the_review": "This paper introduces an interesting benchmark for goal oriented dexterous deformable object manipulation tasks, alongside a latent learning-based planning approach. The explanation of the latter needs to be improved.\n\n---- Post rebuttal updates ----\n\nI think the authors have added sufficient clarity and addressed most of my concerns, hence am raising my score.", "strength_and_weaknesses": "Strengths:\n\nThe proposed set of benchmarks is interesting, and the results appear to show that the proposed approach performs well at these.\n\nWeaknesses:\n\nThe biggest concern I have here is that the description of the proposed approach is rather unclear, and I struggled to follow section 2.4. It is not at all clear to be how the different components in the model connect, and how the baseline approach is trained or used at run-time. It would be helpful to provide a figure illustrating this, and to standardise notation to align with Fig 2. Notation needs a lot of care, for example Locc is seemingly also referred to L(g,h) on page 4, and this makes it hard to keep up with and follow much of the method. I appreciate that some of this is in the appendix, but feel section 2.4 could do with a bit of a rewrite in service of the reader, as at present the complexity here limits reproducibility and does not do the method justice.\n\n\nMinor points:\n\nPar 2: \"...consider the problem of deformable object manipulation with the Shadow Dexterous hand.\" please rephrase to a *simulated* shadow dexterous hand.\n\nThe paper repeatedly claims to provide the first dexterous manipulation benchmark for deformable objects (to the best of the authors knowledge), but learning based approaches have been widely used in robotics literature, along with many deformable manipulation tasks (rope, cables, cloth folding). Plasticinelab, which this work builds on, requires the manipulation of deformable objects. Is the definition of dexterous manipulation used here specifically down to the use of a multi-fingered hand?\n\nCan you provide additional detail about the environments? Eg. actions (are these joint torques?) and point clouds (are these rendered with respect to the camera view point? do they factor in occlusions) etc.\n\nSection 2.3 \"Given a novel deformable shape, how do we solve it...\" Rephrase - I don't think you can solve a shape.\n\nSection 3.3: \"We illustrate that pure EMD might not be a good measure for soft bodies with large topological variations Feydy (2020)\" - I don't see an illustration of this. If EMD is not a good metric, shouldn't the benchmark report alternatives too?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster