DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
Authors: Xingyu Lin
Zhiao Huang
Yunzhu Li
Joshua B. Tenenbaum
David Held
Chuang Gan
Venue: ICLR 2022 Poster (2022)
Keywords: Deformable Object Manipulation
Differentiable Physics

Abstract: We consider the problem of sequential robotic manipulation of deformable objects using tools.
Previous works have shown that differentiable physics simulators provide gradients to the environment state and help trajectory optimization to converge orders of magnitude faster than model-free reinforcement learning algorithms for deformable object manipulation. However, such gradient-based trajectory optimization typically requires access to the full simulator states and can only solve short-horizon, single-skill tasks due to local optima. In this work, we propose a novel framework, named DiffSkill, that uses a differentiable physics simulator for skill abstraction to solve long-horizon deformable object manipulation tasks from sensory observations. In particular, we first obtain short-horizon skills using individual tools from a gradient-based optimizer, using the full state information in a differentiable simulator; we then learn a neural skill abstractor from the demonstration trajectories which takes RGBD images as input. Finally, we plan over the skills by finding the intermediate goals and then solve long-horizon tasks. We show the advantages of our method in a new set of sequential deformable object manipulation tasks compared to previous reinforcement learning algorithms and compared to the trajectory optimizer. 

Published as a conference paper at ICLR 2022
DIFESKILL: SKILL ABSTRACTION FROM DIFFEREN- TIABLE PHYSICS FOR DEFORMABLE OBJECT MANIP- ULATIONS WITH TOOLS
Xingyu Lin" Carnegie Mellon University x1 xlin3@andrew. cmu, edu
Zhiao Huang
Yunzhu Li MIT liyunzhu@mit.co
UC San Diego z 22haaggeegg ucsd. edu
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL jbt @ mit.edu
David Held
Chuang Gan MIT-IBM Waston AI Lab chuanggemit.com
Carnegie Mellon University dheld@andrew. cmu edu
ABSTRACT
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators pro- vide gradients to the environment state and help trajectory optimization converge orders of magnitude faster than model-free reinforcement learning algorithms for deformable object manipulation. However, such gradient-basee baseed trajectory opti- mization typically requires access to the full simulator states and can only solve short-horizon, single-skill tasks due local optima. In this work, propose a novel framework, named DiffSkill, that uses a differentiable physics simulator for skill abstraction solve long-horizon deformable object manipulation tasks from sensory observations. In particular, we first obtain short-horizon skills using individual tools from a gradient-based optimizer, using the full state information in a differentiable simulator; we then learn a neural skill abstractor from the demon- stration trajectories which takes RGBD images as input. Finally, plan over the skills by finding the intermediate goals and then solve long-horizon tasks. We set of sequential deformable object
show the advantages of method in manipulation tasks compared to previous reinforcement learning algorithms and compared to the trajectory optimizer. Videos are available at our project page'.
INTRODUCTION
Robot manipulation of deformable objects is a fundamental research problem in the field of robotics and Al and has many real-world applications such as laundry making (Maitin-Shepard et al., 2010), cooking (Bollini et al., 2013), and caregiving tasks like assistive dressing, feeding or bed bathing (Er- ickson et al., 2020). The recent development of differentiable physics simulators for deformable objects has shown promising results for solving soft-body control problems (Hu et al., 201 9019b; Murthy et al., 2020; Heiden et al., 2021; Huang et al., 2021). These differentiable simulators have facilitated gradient-based trajectory optimizers find motion trajectory with much fewer samples, compared with black box optimizers such as CEM or reinforcement learning algorithms (Huang et al., 2021; Heiden et al., 2021; Geilinger et al., 2020).
However, the usage of these simulators is often limited in two ways. First, most differentiable physics simulators are only applied to solve short-horizon, single-skill tasks, such as cutting (Heiden et al., 2021), ball throwing (Geilinger et al., 2020), or locomotion (Hu et al., 2019b). This is partly because the gradient only provides local information, and thus, gradient-based optimizers often get stuck in local optima, preventing them from solving long horizon tasks. For example, consider a two-stage task of first gathering scattered dough on table and then putting it on the cutting board using spatula (Figure l). If the objective function is minimize the distance between all the dough
"This work was done during an internship at the MIT-IBM Watson AI Lah. 'htpps: /xi ngyu- 1 in thub. io/di ffski11/

---

Published as a conference paper at ICLR 2022
rolling
gripper
gripper
kmife
spatu
scattered dough
Task 1: Liftppred
Task 2: Gatssrrransport
Task 3: CutRearrange
Figure 1: Humans use various tools to manipulate deformable objects much more effectively than state-of thettrr robotic systems. This work aims to narrow the gap and develop method named DiftSkill that learns like a rolling pin, spatula, knife, etc., to accomplish complicated dough manipulation tasks. Our method learns skill abstraction using a diffcrentiable physies simulator, composes the skills for long-horizon manipulation of the dough, and evalulated in three challenging sequential deformable object manipulations tasks: LiftSpread. Transporr,,,, and CutRearrange.
tools
and the cutting board, gradient-based trajectory optimizer would directly use the spatula to put small amount of dough on the cutting board without ever using the gathering operation. Second, the agent needs to know the full simulation state and relevant physical parameters in order to perform gradient-based trajectory optimization, which is great limiting factor for real-world generalization, as reliable full state estimation of deformable objects from sensory data like RGB-D images can often be challenging and sometimes impossible due ambiguities caused by self-occlusions.
Planning over the temporal abstraction of skills is a powerful approach to tackle the first issue of solving long horizon tasks. However, finding a suitable set of skills is challenging. For example, while standard skills such as grasping an object or moving the robot arm from one pose to another may be manually specified (Toussaint et al., 2018), more complex skills such as cutting gathering can be difficult to define manually. Therefore, it is desirable to learn these skills automatically.
In this work, we aim to solve a novel set of sequential dough manipulation tasks that involve operating various tools and multiple stages of actions (Figure 1). including lifting and spreading, gathering and transporting, cutting and rearranging the dough. To extend the use of differentiable physics models to these mong-oorizon tasks and enable the agent directly consume visual observations, we propose DiftSkill: a novel framework where the agent learns skill abstraction using the differentiable physics model and composes them accomplish complicated manipulation tasks. Our method consists of three components, (1) a trajectory optimizer that acts as an expert that applies gradient-bate based optimization on the differentiable simulator to obtain demonstration trajectories, which requires the full state information from the simulator (2) a neural skill abstractor that is instantiated as a conlicondi policy, taking RGBD images input and imitating the demonstration trajectories, and (3) a planner that learns to assemble the abstracted skills by opgimizing the intermediate goals to reach and solve the long horizon tasks. Experiments show that method, operating the nigh-dimensional RGB-D images, can successfully accomplish a set of the dough manipulation tasks, which greatly outperforms the model-free RL baselines and the standalone gradient-basee based trajectory optimizer.
METHOD
Our goal is to learn a policy to perform sequential deformable object manipulation using tools from sensory observations. We utilize a differentiable physics simulator during training. Since is not feasible to directly use a standalone differentiable physics solver to find an optimal solution for long-horizontal tasks, we propose to first learn to abstract primitive skills from this differentiable physics simulator; we then plan on top of these skills to solve long-horizon tasks. An overview of our framework is shown in Figure 2.
2

---

Published as a conference paper at ICLR 2022
(a) Collecting demonstration trajectories
(b) Neural skill abstraction
differentiable physics simulator
skill
skill 0
back propagation
policy
Tx (o 0g)
sim
sim
ST
observation
feasibility predictor
fu og)
Ap
goal
r(ot.0g)
rewand predictor
Planning the abstracted skills
skill O
r(22. 2g)
fee 21
fal21 22)
Figure 2: (a) Collecting demonstration trajectories by running gradient-based trajectory optimizer in differentiable simulator. (b) Neural abstraction by imitating the expert demonstration, which consist of goal-conditioned policy, feasibility predictor and reward predictor. Planning for both skill combination the intermediate goals to solve long-horizon tasks.
2.1 PROBLEM FORMULATION
We consider Markov Decision Process (MDP) defined by set of states E S, actions E A and deterministic, differentiable transition dynamics $1+1 p($,, ae), with indexing the discrete time. At each timestep, the agent only has access to an observation o E O (such an image) instead of directly observing the state. For any goal state 8g G, distance function from the state is given as D(s, sg). The objective is to find policy (STT, sy), where T is the length of an episode.
(o, 0g) that minimizes the final distance to the goal
2.2 COLLECTING DEMONSTRATION TRAJECTORIES WITH DIFFERENTIABLE PHYSICS
Previous work has shown that differentiable physics solvers can acquire short-horizon skills for deformable object manipulation in tens of iterations (Huang al., 2021). Inspired by this work, we first collect demonstration trajectories of using each tool to achieve short-term goal. Given an initial state so. goal state Sy and the transition dynamics p of a differentiable simulator, gradient-based based trajectory optimization to solve for an open-loop action sequence (Kelley, 1960):
T
arg min ..at.....rr))) arg min ap...r-1
Σ D(s 8g) +A 07...07-11 1=1
Σ F(m(s:), d(s))) where 8141 p(8e; at),
(1) In the case of deformable object manipulation, we represent the current and target shape of the deformable object as two sets of particles and use Earth Mover Distance (EMD) between the two particle sets as the distance metric D(st, which we approximate with the Sinkhorn Divergence (Séjourné al., 2019) To encourage the manipulator to approach the deformable object, we additionally add into the objective the Euclidean distance between the manipulator and the deformable object ((m(s)). d(st)), with weight X, where m(st) and d(s;) are the positions of the center of mass of the manipulator and deformable object at time t, respectively. We solve Equation 1 by updating the action sequence using V L, O... T, with an Adam optimizer (Kingma & Ba, 2014), initialized using action sequence of all zeros.
In this paper, define "skill" policy that starting from an initial state 8p. Each skill corresponds to the use of one tool and can be applied to different observations. For example, in the LiftSpread task, the skill of using the rolling pin can be
single tool to achieve short-horizon goal 8g-

---

applied to either when the dough is on the right when the dough is on the cutting board. Assuming that we have K tools, the action space over all tools can be written as .(k)) where A(k) is the action space of the t tool. When collecting demonstration for learning the skills, for each Published as a conference paper at ICLR 2022
short-horizon goal Sz, we run the trajectory optimizer for each tool separately, by masking the actions for other tools to be zero at each timestep.
2.3 NEURAL SKILL ABSTRACTION
Although the trajectory optimizer able to provide solutions for short-horizon tasks, it unable to solve long-howe nogghhoizzon tasks due to local optima. Furthermore, running the trajectory optimizer requires knowing the full state of the environment, including particle positions and mass distributions of the deformable objects and the physical parameters of these objects. This information is difficult to obtain during real-world deployment, where the observations available to robot are from sensory information such as RGB-D images. Additionally, the trajectory optimizer takes minutes to run, which is too slow during evaluation for real-time applications.
As such, we propose to learn neural skill abstractor that learns skills from the demonstration videos of trajectory optimizer; we will then leverage these skills for solving long horizon tasks. Our neural skill abstraction consists of a goal-conditioned policy that takes a sensory observation (RGB-D images in our case) input, feasibility and reward predictor, well variational auto-encoder (VAE).
Goal conditioned policy: For each tool, we learn goal-conditioned neural policy de = T (or, 0g) E AA)), using behavior cloning from the corresponding demonstration trajectories collected for the ttth tool. Given the demonstration trajectory (so. ......... ST) and the corresponding RGB-D sensory observations (Oo. ... oT). as well as an observation of the goal Og. we train a policy using the MSE loss Lpoliey (o,, 0g) a.113- Furthermore, to make the policy more robust to goal observations outside of the training goal images, we adopt hindsight relabeling (Andrychowicz et al., 2017). When sampling from the demonstration buffer, with probability Phand, we will relabel the original goal Og with a hindsight goal o; where o; ~ Uniform(ot- (t. )r). The intuition is that, for any goal that will be achieved by the current action sequence, the policy should imitate the current action.
Variational Auto-Encoder: As will be explained in Section 2.4, we will plan to compose skills in latent space instead of optimizing directly in the image space. To learn the latent space, we train a generative model of the observation space, specifically a VAE Kingma & Welling (2013). The VAE includes an encoder = (()) that encodes an observation into fixed length latent vector, and a decoder o = G(2) that decodes back into the observation space. The VAE is trained such that G(=), (o, I) reproduces the observation distribution.
For each skill, we learn a feasibility predictor fa(Ze; =g) E R to determine the feasibility of reaching from oz to 0g using the trajectory optimizer, where 2, = Q(ot), &g Q(0²). For training the feasibility predictor, we obtain positive pairs by sampling ( from the same demonstration trajectory and negative pairs by sampling pairs of observations from different trajectories. We additionally add negative pairs (21, z2) where both 2 and ø2 are sampled from a unit Gaussian distribution in the VAE latent space. We assign a label of for positive pairs and a label of 0 for negative pairs. We use an MSE loss T.fea for model training, which was shown empirically to work better than a cross-entropy loss. While there may be false negative pairs, we find the feasibility predictor to perform reasonably well, as different trajectories have different initial and goal configurations and it is less likely that the state of a different configuration (c.g. mass of the dough) can be achieved from the current state within one skill. This intuition was similarly used for goal relabeling in previous work by Lin et al. (2019).
Feasibility Predictor:
We further train a reward predictor r(o,, E IR that predicts the negative of the Sinkhorn divergence between the corresponding states - DD(st, 8y) using an MSE loss Ly. The reward predictor does not depend on any specific skill or tool.
Reward Predictor:
Training We first pretrain the VAE and then freeze the weights. We then jointly optimize the fcasibility predictor, reward predictor and the goal conditioned policy.
2.4 LONG-HORIZON PLANNING WITH SKILL ABSTRACTION
To apply the skill abstraction learned above sequentially, we need to determine 1) which skill to use at each stage; 2) what intermediate goal to specify for each skill. Given oo, 0g the initial and goal
4

---

Published as a conference paper at ICLR 2022
observations, we plan over H steps. By using the feasibility and reward predictor, we formulate problem as a hybrid discrete and continuous optimization problem:
H-1
arg min C(k, = II ==0
((41))) =g); s.t. || S M, Vi.
(2)
Here, k is the index of the tool used at step i, is the feasibility predictor and F(ZH, 2g) exp(-D(zH, (g)) which can be computed from the reward predictor, and 20, 2g are the VAE encoded latent vectors of the initial and goal observations (RGB-D image) o0, 0g- The optimization variables include the discrete variables k ki,... kH, representing the index of the skills to use at each step and the continuous variables
21, ................................ 2 that latent vectors that represent the intermediate goals. The term ||=;|| in the constraint is proportional to the log likelihood of the latent vectors under unit normal distribution and M is threshold to ensure that the latent vectors correspond to actual intermediate goals in the observation space.
To optimize for both the discrete and continuous variables in Eqn. 2, all possible combinations of the discrete variables. For the continuous variables, we start with N initial solutions {zi,... z J. j .. and use Adam with projected gradient on the loss for all the initial solutions in parallel, since the constraint convex. Specifically, after each gradient update step of Adam, we project the current to the constraint set by setting
exhaustive search over
max(||=:la///M))) Once we have solved for k1; Ri,..., kH, 2H, we can decode the latent vectors back to images , 0H O1.....
G(z1), .... G(zH), and call the corresponding coal-conditioned policies sequentially: The (0o, oi.,.... Thy (:. o.). To simplify the execution, we move each tool to its initial pose at the beginning of executing each skill. The pseudocode is shown in Algorithm 1.
Algorithm 1: Solve long-horizee planning with DiftSkill
Input Trajectory optimizer, skill horizon 7', planning horizon H
Initialize modules for neural skill abstraction TT.. fk. r, Q,Q;
Generate N demonstration trajectories differentiable physics {(on, ap, ro...., or, rr, or, Train neural skill abstraction Wk, fa, r, G, using loss Laku until convergence;
for k /0 ... KJ* do
Initialize = [=1,-.. zul (00, T);
Optimize 21,, 2H according to Eqn. to obtain cost C(k, z)
Choose k, that minimizes C(k,z):
i + 0 to H do
Reset tools to initial poses ;
Decode intermediate goal images: Og, + G(z)):
Execute policy Th, 0g.s) in the environment for 7' steps:
3
EXPERIMENTS
In this section, we will discuss our experimental setup, implementation details and comparison results.
3.1 EXPERIMENTAL SETUP
Tasks and environments We experiment with set of sequential deformable object manipulation tasks with dough. We build our simulation environments on top of Plasticinel.ab (Huang et al., 2021), a differentiable physics benchmark using the DiffTaichi system (Hu et al., 2019a) that could simulate plasticine-like like objects based on the MLS-MPM algorithm (Hu et al., 2018). Inspired by the dumpling making process, we design three novel tasks that require long-horizen planning and usage of multiple tools:
The agent needs to first use a spatula (modeled as thin surface) to lift a dough onto the cutting board and then adopt rolling pin roll over the dough to flatten it. The rolling pin is simulated as a 3-Dof capsule that can rotate along the long axis and the vertical axis and translate along the vertical axis to press the dough.
LiftSpread:
FatherTranspoor Initially, residual of dough is scattered over the table. First, the agent needs to gather the dough with extended parallel gripper and place top of spatula. Then the agent
5

---

Published as a conference paper at ICLR 2022
needs to use the spatula to transport the dough onto the cutting board. The spatula can translate and rotate along the vertical plane. The gripper can translate along the horizontal plane, rotate around its center and open or close the gripper.
CutRearrange: This is a three-step task. Given an initial pile of dough, the agent needs to first cut the dough in half using knife. Inspired by the recent cutting simulation (Heiden et al., 2021), 1), we model the knife using a thin surface as the body and a prism as the blade. Next, the agent needs to use the gripper to transport each piece of the cut dough to target locations.
visualization of these tasks can be found in Figure 1. For all the taks, the agent receives RGB-D image resized to 64x64 from a camera and return velocity control commands directly on the tools.
Evaluation metric We report the normalize decrease in Sinkhorn diverge computed s(l) 6am&. where 8o, 8t are the initial and current. Sinkhorn divergence. In this way, normalized performance of 0 representing a policy that does nothing and a normalized performance of representing an upper bound of policy that perfectly match the two distributions. The maximum normalized performance of I may not always be possible. For example, due to the incompressibility of the dough, certain target shape may be too small for a large dough to fit into. As such, we additionally set threshold on the Sinkhorn divergence for each task such that the task is assume to be completed successfully when EMD is below the threshold. This threshold is manually picked by observing the performance gap between successful and failed trajectories. For each task, initial and target configurations are given for evaluation, which all require multi-stage manipulation of the dough. We report both the normalized performance metric and the success rate for comparisons.
to
3.2 IMPLEMENTATION DETAILS OF DIFFSKILL
For each task, we first generate 1000 different initial and goal configurations, varying the initial and target shape of the dough as well as poses of the manipulators. For each configuration, we run the optimizer for each tool separately to generate trajectories of length 50. The Adam optimizer starts with initial action sequence of all zero and is then run for 200 iterations to optimize each trajectory,
We then train our VAE, policy, feasibility and score predictors over this demonstration video dataset. These modules share the same encoder architecture: 4 convolutional layers with a kernel size of 4 and a stride of 2 and a channel size of (32, 64, 128, 256), followed by an MLP that maps the feature into a fixed 8 dimension vector. The VAE, fcasibility and the score predictor also share the same weights for the encoder.
After training, we find the feasibility and score predictor to perform well on the held out trajectories, achieving L2 error of less than 0.05 for the score predictor and an accuracy of over 0.95 for the feasibility trajectory, across different environments. We found behavior cloning to be sufficient for learning short-horizer
skills from the demonstration dataset. In Table 3, we can that the learned skills (labeled as Behavior Cloning) approach the normalized performance of the trajectory optimization (Trajectory Opt) on single-tool use, although they cannot solve the long-horizer tasks.
Given the skill abstraction, DiffSkill iterates through all skill combination. For each skill combination, we randomly sample 1000 initial solutions for the latent vectors 21,.... -., and then perform 500 iterations of Adam optimization for each of them. Since we are optimizing in the latent space, and with the help of parallel computation of GPU, this optimization can be done efficiently in around 10 seconds for each skill combination on a single NVIDIA 2080Ti GPU.
3.3 BASELINES
We compare with three strong baselines:
Model-free Reinforcement Learning (RL) We compare with two model-free RL methods: TD3 (Fu- jimoto et al., 2018) and SAC (Haarnoja et al., 2018). The RL methods use the same encoder architec- ture as DiffSkill and receive the negative of the cost function used for the trajectory optimizer as the reward function. We train the RL agents with either a single tool, or with both tools where the agent controls them simultaneously.
Behavior Cloning We compare with another baseline that directly trains a goal-conditioned policy with Behavior Cloning (BC) and hindsight relabeling using all tools. The agent is trained with the same dataset and jointly controls all tools.

---

Published as a conference paper at ICLR 2022
Trajectory Opt (Oracle) We compare with the trajectory optimizer used to generate the demon- stration, with the same parameters. Note that this method requires full state of the simulation and multiple forward and backward passes through the simulator during evaluation time.
Since DiftSkill sequentially apply multiple skills, we move any used tools to its initial pose after executing each skill with a manual policy.
3.4 RESULT ANALYSIS
We show that DiffSkill is able to solve the challenging long- horizon, tool-use tasks from the sensory observation (RGB-D) while the baselines cannot. The quantitative results can be found in Table 1. Tool A refers to the spatula for LiftSpread, the parallel gripper for GatherTransppr and the knife for the CutRearrange environment, while tool B refers to the rolling pin, the spatula and the gripper for each task respectively.
First, we can compare the rows for single-tool use. All baselines are able to solve one-stage of the task, achieving reasonable performance, if the correct tool is used (e.g. tool for LiftSpread and GatherTransport) although for cutting, the performance is still close to zero with cutting only, since cutting itself does not reduce the distance between the current dough and the desired dough locations. BC can well approach the performance of the corresponding trajectory optimizer despite being a policy that only receives RGB-D input. SAC can also solve some of the single-stage tasks such as lifting or transporting, while TD3 performs worse.
Second, with multiple tools, Remarkably, DiftSkill even outperforms the trajectory optimizer that controls both tools at the same time, which uses the full simulation state during evaluation time. This indicates that high-level planning over skills is necessary for achieving these sequential manipulation tasks. Furthermore, we show visualization of the found solutions of DiftSkill in Figure 3. We can see that DiffSkill is able to find reasonable intermediate goal state such as first putting the dough on the cutting board, or gather dough onto the spatula. CutRearrange is shown to be harder task, with all methods performing poorly. This is because it requires three stages of manipulation; further, is non-trivial to transport the dough without deforming it too much. Still, DiffSkill is able to achieve higher success, even though the trajectory optimizer oracle achieves a higher average score. This is because the trajectory optimizer is more reliable at finding partial solutions that transport part of the dough to the target locations but does not complete the full task. Please see our website for visualizations of different methods.
that DiffSkill significantly outperforms the single-skill policy,
Task (H)
LiftSpread (2)
GatherTransport (2)
CutRearrange (3)
Method
Trajectory Opt (Oracle)
0.755 / 0%
0.386 / 0%%
0.033 0%
Tool A only
Behavior Cloning
0.754/ / 0%
0.356 0%
0.018 0%
Model-free RL (TD3)
0.766 0%
0.103 0%
0.015 0%
Model-free RL (SAC)
0.815 / 0%
0.141 0%
-0.004 / 0%
Trajectory Opt (Oracle)
0.092 / 0%
0.494 0%
0.239 00%
Tool B only
Behavior Cloning
0.095 0%
0.530 0
0.168 0%
Model-free RL (TD3)
0.001 0%
0.000 0%
0.121 0%
Model-free RL (SAC)
-0.016 0%
0.592 / 20%
0.131 0%
Trajectory Opt (Oracle)
0.818 / 44%
0.403 / 70%
0.353 / 0%
Multi-tool
Model-free RL (TD3)
0,781/ 0%
0.423 / 0%
0.133 0%
Model-free RL (SAC)
0.797 / 0%
0.567 / 20%
0.131 / 0%
DiffSkill (Ours)
0.920 / 100%
0.683 60%
0.297 20 %
Table 1: Normalized improvement of all methods and the success rate different tasks. Each entry shows the normalized improvement/ success rate, The top bar shows H, the planning horizon for each environment.
3.5 ABLATION ANALYSIS
We perform two ablations on DiffSkill. First, we try removing the planning over the discrete variables that decides which tool to use at each step. Instead, we train a tool-independent policy, the feasibility and score predictor, where the action space is the joint-tool action space. Then during planning, we only need to optimize for the intermediate goals. This ablation is labeled as No Discrete Planning.
7

---

Published as a conference paper at ICLR 2022
ohservation genersted intermediate goals given goal
Tool
Tool
0.00
0.71
0.89
0.94
0.00
0.01
0.02
0.55
0.73
7
0.00
0.60
0.87
observation
generated intermediate goals Optimized Plan
given goal
Execution
Figure 3: Visualization of the generated plan and the corresponding execution. The plan generated by DiffSkill is shown in the left, where the first and the last image are the given initial and goal observation and in between are the generated intermediate goals. The color blocks indicate which tool is needed to reach the sub-goal. The right shows sampled frames during the execution of the generated plan using the corresponding goal conditioned policy. The numbers on bottom right shows the achieved normalized improvement metric at that time.
Second, we try to remove the planning over the continuous variables, i.e. the intermediate goals. Without the intermediate goals, we directly use the final target image as the goal for sequentially executing each skill. This ablation labeled as Direct Execution. Since we do not have the intermediate goals anymore, we try two different ways of choosing the skills execute at each stage: Randomly pick one skill and an oracle that execute each combination of skills in the simulator and choose the best one in hindsight. The results are shown in Table 2. Without discrete planning, the policy performs poorly. This is probably because during training, a single policy and feasibility predictor is used for learning two modes of skills that are very different and the policy is unable to differentiable for different modes or decide when to switch modes. On the other hand, if we do not optimize for the intermediate goals, we also cannot determine which tools to use at evaluation time, since both the feasibility predictor requires intermediate goals as input. In this case, we can see that using random skills at each stage results in poor performance. Even if the oracle skill order is used, there is still a drop performance as the policy may not work well given only a future goal instead of an immediate goal.
Task
LiftSpread
GatherTranspor
CutRearrange
Method
No Discrete Planning
0.758 200%
0.312/0%
0.118 / 70%
Direct Execution (Random)
0.593 / 15%
0.369 / 0%
0.018 / 2.5%
Direct Execution (Oracle)
0.865 / 60%
0.501 / 0%
0.334 / 20%
DiftSkill (Ours)
0.920 / 100 %
0.683 / 60 %
0.297 20%
Table 2: Normalized improvement and success rate of ablation methods.
4 RELATED WORK
Motion Planning with Differentiable Physics. Differentiable physics models provide the gradients of the future states with respect to the initial state and the input actions. Compared with black-bor box dynamics models, planning with differentiable physics models can often make more accurate updates on the action sequences and deliver better sample efficiency (Huang et al., 2021). Over the past few years, researchers have built differentiable physics models from first principles rooted in our understanding of physics for various physical systems, ranging from multi-body systems (Degrave et al., 2019; Tedrake & the Drake Development Team, 2019; de Avila Belbute-PPres et al., 2018 8;; Geilinger al., 2020), articulated bodies (Werling al., 2021: Qiao al., 2021), cloth (Qiao et al., 2020; Liang et al., 2019), fluid (Schenck & Fox, 2018), plasticine (Huang et al., 2021), to soft robots (Hu al., 2019 1019b;;; Du al., 2021). They have shown impressive performance in using the
8

---

Published as a conference paper at ICLR 2022
gradient to solve inverse problems like parameter identification and control synthesis. A complement thread of methods tried to relax the assumption that we have to know the full state and the physics equations; instead, they employ deep neural networks to learn the dynamics models directly from observation data and use the gradients from the learned models to aid motion planning or policy learning (Battaglia et al., 2016; 6; Chang et al., 2016; Mrowca et al., 2018; 8; Ummenhofer et al., 2019; Sonchez-Gonzalz et al., 2020; Pfaff et al., 2020; Sanchez-Gonzaler et al., 2018; Li et al., 2019; 2018). Our method differs from prior works in that we use the differentiable physics model for skill abstraction and compose the learned skills for long-horizon manipulation of deformable objects.
Deformable Object Manipulation. Deformable objects have more degrees of freedom and are typically more challenging to manipulate than rigid objects. Extensive work has been done in this area, focusing on different types of deformable objects. including plasticine (Li et al., 2018; Huang al., 2021), 1), fluid (Li al., 2021), cloth (Maitin-Shepard et al., 2010; Lin et al., 2021; Weng et al., 2021; Ha & Song, 2021; Wu et al., 2019; Yan et al., 2020; Ganapathi et al., 2020), rope (Nair et al., 2017; Sundaresan
al., 2020), and object piles (Suh & Tedrake, 2020). People have also come up with standard benchmarks to establish a fairer comparison between various algorithms (Lin et al., 2020). Our method aims to manipulate dough using tools similar to how humans would do in a kitchen environment (Figure I). There are also a series of works on tool-using for object manipulation (Toussaint et al., 2018; Qin et al., 2020; Fang et al., 2020; Xie et al., 2019), but most of them focus on rigid objects, whereas we take a step further and tackle deformable objects.
Solving Long Horizon Tasks, Our work aims to solve long-horizon manipulation tasks; thus, it is also closely related to task and motion planning (TAMP) that contains elements of discrete task planning and continuous motion planning. (Gravot et al., 2005; Kaelbling & Lozano-Péree Pérez, 2010; 2013: Toussaint et al., 2018; Garrett et al., 2021). However, most TAMP approaches assume a fully specified state space, dynamics model as well as set of pre- pre-defined skills such as grasping, pushing, or placing (Toussaint et al., 2018). Differing from prior works, our method learns a policy over RGB-D inputs and generates motion trajectories and skill abstractions by applying mrudient-based trajectory optimization techniques using a differentiable simulator.
Our feasibility predictor also relates to previous works that plan over the temporal abstraction of options (Sutton et al., 1999) or skills. In order to do planning, these approaches define multi-time model (Precup & Sutton, 1997) or equivalently a skill effect model (Kaelbling & Jozano-Pérez, 2017; Liang et al., 2021) to model the end state after executing a skill. In contrast, model such dynamics implicitly by predicting the feasibility between two latent vectors which allows direct optimization over the sub-goals in latent space. The most similar to ours is (Nasiriany et al., 2019), which learns a value function as the implicit dynamics. Our method differs from this work by learning skills from trajectory optimizer instead of using RL. Additionally, encode multiple skills in different neural networks and perform discrete optimization over the skills for planning.
5 CONCLUSIONS
In this paper, we propose DiffSkill, novel framework for learning skill abstraction from differentiable physics and compose them to solve sensory observation. We evaluate model on series of challenging tasks for sequential dough manipulation using different tools and demonstrate that it outperforms both reinforcement learning and standalone differentiable physics solver. We hope our work will be first step towards more common application of differentiable simulators beyond solving short-horizer tasks.
deformable object manipulations tasks from
There are few interesting directions for future work. First, currently DiffSkill uses exhaustive search for planning over the discrete space. As the planning horizon and the number of skills grow larger for more complex tasks, exhaustive search quickly becomes infeasible. An interesting direction to incorporate a heuristic policy or value function for more efficient planning. Second, similar to many other data-driven methods, while neural skill abstraction gives good prediction results on region where a lot of data are available, it performs worse when tested on situations that are more different from training. This can be remedied by either collecting more diverse training data in simulation, for example, under an online reinforcement learning framework, by using a more structured representation beyond RGBD images, such as using an object-centric representation. Third, we hope to extend our current results to the real world, by using more transferrable representation such as just a depth map or point-cloud representation. Finally, we hope to see DiffSkill be applied to other similar tasks, such as those related to cloth manipulation.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank Sizhe Li for the initial implementation of the CutRearrange environment. This material is based upon work supported by the National Science Foundation under Grant No. IIS-1849154, IIS- 2--4444491, LG Electronics, MIT-IBM Watson AI Lab and its member company Nexplore, ONR MURI (N00014-13- 1-0333,,, DARPA Machine Common Sense program, ONR (N00014-18-1-2847)) and MERL.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv: 170700149, 2017. 4
Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:16/2.00222 2016. 9
Mario Bollini, Stefanie Tellex. Tyler Thompson, Nicholas Roy. and Daniela Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics, pp. 481-495 Springer, 2013. 1
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.0034/ 2016.
Filipe de Avila Belbute-Peres Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. Advances in neural information processing systems, 31: 7178-7189, 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, et al. differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, 13:6, 2019.
Tao Du, Kui Wu, Pingchuan Ma. Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Diffpdd: Differentiable projective dynamics. arXiv preprint arXiv:2 105917 2021. 8
Zackory Erickson, Vamsee Gangaram, Ariel Kapusta, C Karen Liu, and Charles C Kemp. Assistive gym: A physies simulation framework for assistive robotics. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 10169-10176 76. IEEE, 2020.
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mchta, Li Fei-Fei, and Silvio Savarese. Learning task-oriented grasping for tool manipulation from simulated self-supervision. The International Journal Robotics Research, 39(2-3):202 216. 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMIR, 2018.
Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph E. Gonzalez, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. Learning dense visual correspondences in simulation to smooth and fold real fabries. arXiv preprint arXiv:2003. /2698, 2020.
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay. Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling. and Tomás Lozano-Pérez Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021. 9
Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: analytically differentiable dynamics multi-body systems with frictional contact. ACM Trunsactions on Graphics (TOG), -15,, 2020. 1,
Fabien Gravot, Stephane Cambon, and Rachid Alami. asymov: planner deals with intricate symbolic and geometric problems. In Robotics Research. The Eleventh International Symposium, pp. 100-110. Springer. 2005. 9
Huy Ha and Shuran Song. Flingbot: The unrcasonable effectiveness of dynamic manipulation for cloth unfolding.
arXiv preprint arXiv:2105.03633 2021. 9
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning. pp. 1861-1870. PMLR, 2018. 6
10

---

Published as a conference paper at ICLR 2022
Eric Heiden, Miles Macklin, Yashraj S Narang. Dieter Fox, Animesh Garg, and Fabio Ramos. DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Roboties: Science and Systems, Virtual, July 2021. doi:
2000://RSS.2221XXX...555 1,
Yuanming Hu, Yu Fang, Zäheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(4):150, 2018.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaich:: Differentiable programming for physical simulation. arXiv preprint arXiv: 1910,00935, 2019a. 5,
Yuanming Hu, Jiancheng Liu, Andrew Spielherg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simulator for soft robotics. In 2019 International conference on robotics and automation (ICRA), pp. 6265-6271 JEEE, 2019b. 1, 8
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua Tenenbaum, and Chuang Gan. Plas- ticinelab: A soft-body manipulation benchmark with differentiable physics. International Conference on Learning Representations (ICLR), 2021. 1, 5, 8,9
Leslic Pack Kaclbling and Tomás Lozano-Pérez. Hicrarchical planning in the now. In Workshops at the Twenty-Fourth AAAI Conference Artificial Intelligence, 2010. 9
Leslie Pack Kaelbling and Tomás Lozano-Pérez. Integrated task and motion planning in helief space. The International Journal Robotics Research, 32(9-10):1194 1227, 2013. 9
Leslie Pack Kaelbling and Tomás Lozano-Pérez Learning composable models of parameterized skills. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 886-893 IEEE, 2017. 9
Henry J Kelley. Gradient theory of optimal flight paths. Ars Journal, 30(10):947 9544, 1960.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014, 3
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1t..ivv3112..114.. 2013.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv: 1810000600,,,66, 2018. 9
Yunzhu Li, Jiajun Wu, Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pp. 1205-1211 IEEE, 2019.
Yunzhu Shuang Li, Vincent Sitzmann. Pulkit Agrawal, and Antonio Torralba. neural scene representations for visuomotor control. arXiv preprint 20x::22007.00000 2021. 9
Jacky Liang. Mohit Sharma, Alex LaGrassa, Shivam Vats, Saumya Saxena, and Oliver Kroemer. Search- based task planning with learned skill effect models for lifelong robotic manipulation. arXiv preprint arXiv:2109.08771 2021.
Junbang Liang, Ming Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. NewrIPS. 2019. 8
Xingyu Lin, Harjatin Singh Baweja, and David Held. Reinforcement learning without ground- state. arXiv preprint arXiv:/905.07866. 2019.
Xingyu Lin, Yufei Wang. Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. Conference on Robot Learning, 2020.
Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth
smoothing. In Conference on Robot Learning, 2021. 9
Jeremy Maitin-Shepard, Marco Cusumano Husumano-Towner, Jinna Lei, and Pieter Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pp. 2308 2315. IEEE, 2010. 1,9
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural representation for physics prediction. arXiv preprint arXiv: 1806,08047, 2018.
11

---

Published as a conference paper at ICLR 2022
J Krishma Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jéröme Parent- Lévesque, Kevin Xie, Kenny Erleben, al. gradsim: Differentiable simulation for system identification and visuomotor control. International Conference on Learning Representations, 2020. 1
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Com- bining self-supervised learning and imitation for rision-based rope manipulation. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2146-2153. IEEE, 2017. 9
Soroush Nasiriany. Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Bue, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32, 2019. 9
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez- Gonzalez, Sanzhez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020. 9
Doina Precup and Richard S Sutton. Multi-time models for temporally abstract planning. Advances in neural information processing systems, 10, 1997. 9
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.022 2020. 8
Yi-Ling Qiao, Junbang Liang. Vladlen Koltun, and Ming C. Lin. Efficient differentiable simulation of articulated bodies. In ICMI., 2021. 8
Zengyi Qin, Kuan Fang. Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Keto: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 7278-7285. IEEF, 2020. 9
Alvaro
Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470 4479. PMLR, 2018.
Alvaro Sanchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying. Jure Leskovee, and Peter Battaglia. Learning simulate complex physics with graph networks. In International Conference Machine Learning, pp. 8459 8468. PMLR, 2020. 9
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks. In Conference on Robot Learning, pp. 317-335. PMI.R, 2018.
Thibault Séjourné, Jean Feydy, François-Xavier Vialard, Alain Trouvé, and Gabriel Peyré. Sinkhorn divergences for unbalanced optimal transport. arXiv preprint arXiv:1910.12958 2019.
HJ Suh and Russ Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. arXiv preprint arXi::2002.0902) 2020. 9
Priya Sundaresan, Jennifer Gramnen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph E. Gonzalez. and Ken Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. arXiv preprint 233 2020. 9
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 (12(12))181-2211 1999. 9
Russ Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019.
URL https:/ // /draee..... .nit. edu. 8
Marc A Toussaint, Kelsey Rebecca Allen, Kevin Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. Robotics: Science and Systems Foundation, 2018. 2, 9
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladien Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Representations, 2019. 9
Thomas Weng, Sujay Man Bajracharya, Yufei Wang, and David Held. Fabricflownet: Bimanual cloth manipula- tion with flow-based policy. In 5th Annual Conference on Robot Learning, 2021. 9
Keenon Werling, Dalton Omens, Jeongseok Lee, Ioannis Exarchos, and C. Karen Liu. Fast and feature-complete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103. 16021, 2021. 8
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations, arXiv preprint arXiv:/9100 133499, 2019. 9
12

---

Published as a conference paper at ICLR 2022
Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical understanding: Using novel objects tools with visual foresight. arXiv preprint arXiv: w9X4v:190400533,, 2019.
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive representations for deformable objects using contrastive estimation. arXiv preprint arXiv:2003.05436, 2020. 9
13

---

Published as a conference paper at ICLR 2022
A
IMPLEMENTATION DETAILS
A list of the hyperparameters used can be found in Table 3.
Model parameter
Value
dimension of latent space
MLP hidden node number
1024
Training parameters
Value
learning rate
0.001
batch size
128
optimizer
Adam
betal
0.9
beta2
0,999
weight decay
0
hindsight relabeling ratio Phind
0.5
Feasibility predictor loss weighting Aea
10
Reward predictor loss weighting A,
VAE loss weighting Awae
Trajectory optimizer
learning rate
0.02
number of iteration
200
episode length T
50
action sequence initialization
all initialized zero
Planning over continuous variables using Adam
number of iteration
1000
learning rate
0.5
number of initial seeds log likelihood constraint
1000
* H
Table 3: Summary of all
B COMPARISON WITH MODEL-FREE RL ON SINGLE-TOOL TASKS
In this work, focus solving long-horizon multi-tool tasks. But our proposed method is limitied to the multi-tool setup. For the single-tool task, or more specifically the single-skill task, our method can directly applied without planning over the discrete space which tool to use. show this, we show results on two single-tool, single-stage tasks: Lift and Spread, which corresponds to the two sub-tasks in LiftSpread, but with a different initial and target dough shapes. For both environments, only one tool can be used: the spatula for Lift and the rolling pin for Spread. The results are shown in Table 4. As these solve both tasks. SAC also solve the simpler lift task, but has some trouble learning the spread task. Visualization both methods on these environments can found on our website.
simpler tasks, DifiSkill
Task
Lift
Spread
Method
SAC 0.890 / 100 -0.049 70%
DifiSkill (Ours) 0.904/ 100%
0.495 100%
Table 4: Normalized improvement methods and the success rate on single-tool tasks, Each entry shows the normalized improvement / success rate.
14


Reviews:
[{"review_id": "p27nmmssp97", "reviewer": "Reviewer_DHQA", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n- I really like the idea of transfer-learning between diff-simulator and image-based policy, as well as searching over the latent for planning intermediate goals.\n- The video results are quite impressive.\n- The paper is very focussed - there's enough theory to understand the idea, there's enough implementation detail to get an understanding how to implement this, there aren't many experiments but the ones that are there are reasonably novel.\n\n**Weaknesses:**\n\n- W.1 Maybe this is just me but the writing could be a bit more polished for accessibility. It took me 2 reads for it to \"click\" that when you write \"neural skill abstractor\" in the abstract and intro, what you mean is that you're learning image-based control from a state-based (\"expert\") policy with the purpose of applying this in an image-based setting. I think just adding a sentence or two to explain this or create more of an intuition why we need to learn this would go a long way. The same is true imho for the fact that you're discovering intermediate goals via searching over z. \n- W.2 There's a couple typos in the main body of the text but I'm sure you can fix those. Also in the first sentence, calling elder care \"deformable object manipulation\" is... interesting. \n- Honestly, that's all my criticism. The paper is pretty good.\n\n**Nitpicks/Questions:**\n\n- When mentioning differentiable physics and planning therein, it would be nice to also mention the recent work on GradSim (Jatavallabhula, Krishna Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine et al. \"gradSim: Differentiable simulation for system identification and visuomotor control.\" ICLR 2021.)\n- Why did you decide to train all network components simultaneously with one big loss term? Did you try training them separately? Are there any upsides/downsides to that?", "summary_of_the_paper": "The paper provides a framework for learning how handle deformable objects. Specifically, the authors introduce a 3-step process where (1) in a differentiable simulator low-level skills are learned, then (2) the learned skill are converted from state observations to image observations via behavioral cloning, while training a feasibility detector that helps finding the right skill and tool for a goal state and lastly, (3) planning a long-horizon task by searching for both the low-level skill and latent variable (representing an intermediate goal).", "summary_of_the_review": "UPDATED (2021-11-21): The authors answered my questions and updated a few sections of the paper. I think the work is now in a great state and should be highlighted.\n\nOLD: I'm pretty happy with the paper. It's very novel, well-written, and would make for a great contribution to the robotic manipulation literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jyvBuTQmnxF", "reviewer": "Reviewer_wVGi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper designs a pipeline to learn multistage deformable body manipulation tasks. It is hard to simply apply differentiable physics in this task since there are lots of local optima and even discrete variables - the choice of different tools. This task is also challenging for reinforcement learning because the process can be complex, the reward function can be sparse, and the control variables contain both continuous and discrete values. \n\nThe authors find a clever way to circumvent the challenges. 1. Differentiable physics is used to optimize local trajectories, such that demonstrations are generated efficiently and then used to train an NN-based controller. 2. A global planner is articulated as optimization of both the intermediate states and the choice of tools. The state is in a low-dimensional continuous latent space encoded by a VAE network. 3. Two NNs are learned to approximate the feasibility and reward function, which are used when scheduling the global motion.\n\nWhen reading this paper, I have several concerns,\n1. Typos: 'psedocode' -> 'pseudocode', 'there strong baselines' -> 'three strong baselines'.\n2. In Equation 2, should it be maximizing $C(k,z)$? and should $\\bar{r}$ be negative or positive?\n3. When optimizing $z_i$ in equation (2), is it efficient/correct to first treat the problem as unconstrained and then project variables back to the constraint set? Can you directly solve it as a constrained optimization? \n4. What could be the possible limitation and future work of this method.", "summary_of_the_paper": "This paper proposes to combine a local controller with a global planner for long-horizon deformable object manipulation with different tools. The controller is modeled as a neural network that takes an RGBD image as input and outputs the control signal. The global planner iterates all possible tools and optimizes a utility function, which is the multiplication of predicted feasibility and reward function. Differentiable physics is used to generate demonstrations for short-horizon skills. All observations are embedded in a continuous latent space, on which the optimization is further performed.", "summary_of_the_review": "This paper formulates a pipeline of applying differentiable physics to long-horizon tasks. The experiment setting is very challenging. It deals with sensory observation, deformable body, variable tools, and complex tasks. The proposed method solves those problems reasonably well as shown in the video. I think their method has the potential of generalizing to other similar tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QK6NO6spYhy", "reviewer": "Reviewer_H1Qf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The paper is well written and easy to follow. The provided videos showing the results highlight the difficulty of each task and the method ability to solve them.\n- Splitting the long-horizon and multi-tool nature of the task into different short-horizon single-tool stages is a natural and intuitive way to manage to solve long and complex tasks.\n- Using a VAE to learn a low dimensional latent space to encode the RGB-D observations is a clever way to allow generating intermediate goals and thus splitting the long-horizon task into multiple more manageable short-horizon tasks.\n- The proposed tasks are fairly complex and convincing to show how their approach work for the defined problem.\n\n### Weaknesses\n\n- For the optimization problem in (2), shouldn't you maximize the cost function C(k,z) instead of minimizing it?\n- A video comparison of the resulting policies to the chosen baselines would be nice (RL, BC and trajectory optimizer). This could illustrate where the other methods fail.\n- What are the possible limitations of this approach in practice? Is the goal representation as an RGB-D image practical for example for real robot applications? How critical is resetting tools to initial poses at the end of each skill policy execution?\n\n### Typos\n- Page 6: \"We also present the psedocode of our method in Algorithm\" → **psedocode** to **pseudocode**.\n- Page 8. \"On the other hand, RL performs poorly on tasks, showing that it is difficult to learn A good policy directly from visual input.\" → Capital **A** to **a**", "summary_of_the_paper": "This work proposes a new framework to learn long-horizon control policies on multi-tool deformable object manipulation tasks. A differentiable physics simulator is combined with a trajectory optimizer to gather expert demonstrations of short-term skills. The skill expert trajectories are then used to learn a neural skill abstractor. Finally, a planner learns to assemble the skill policies to solve the long-term task by solving intermediate goals.", "summary_of_the_review": "The paper proposes a novel approach to solve a challenging problem: long-horizon multi-tool deformable object manipulators tasks. The idea can be beneficial for the robotics community and seems to be extendable to quite different tasks in the same setting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6xvNnDTRGD", "reviewer": "Reviewer_ifUw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n - The method uses differentiable physics for obtaining trajectories that are required for the imitators to learn, which eases the difficulty of designing a good reward function during RL.\n - Different skills and tools here are separately considered, decoupling the problem into smaller ones.\n\nWeaknesses:\n\n - Is a skill dependent on the tool alone or the observations together as well? Specifically, if you have different start/end positions of the same short-horizon task using only one tool, are they considered the same skill?\n - Why not training the VAE alone? Generally speaking, this module should be standalone.\n - How do you determine H in each long-horizon task? How does different H in one task impact the performance?\n - The method exhaustively search over all combinations of different skills in each small step. Does it mean that in some specific configurations where H and num_skills are large, the proposed method can actually be even slower than the trajectory optimization provided by differentialbe physics? I would like to see more discussions or experiments regarding this matter.\n - How did DiffSkill perform on single tool experiments?\n - Why Trajectory Opt has 0.544 improvement and 0% success rate in 'Tool A Only' while the numbers changed to 0.385/20% in 'Multi-Tool'? Aren't they positively correlated?\n - It seems that all previous baselines somewhat fails in the experiments designed. I wonder how they would perform in simpler ones where they actually have achieved something meaningful (i.e. success rate>0), and how the proposed method would compare in these examples.", "summary_of_the_paper": "This paper proposes a method to learn long-horizon control policies. It uses differentiable physics simulator to generate skills for the latter imitator to learn. To complete the long-horizon task, the proposed method first exhaustively search for tools/skills to be used in each step. Next, it optimizes the intermediate observation states to maximize the feasibility and the reward through the trajectory. At last, the policy of each small step is obtained by feeding start/end state pairs to the imitator. the results show that the proposed method is much better than both traditional RL and the trajectory optimization offered by differentiable physics alone.", "summary_of_the_review": "The paper addresses the challenging problem of long-horizon RL by using differentiable physics and imitation learning. It is a inspirational method in terms of the way differentiable physics is coupled in large-scale complex problems and the decoupling of different sub-steps. However, there are some issues described above that are not clearly demonstrated in the main text nor in the supplementary document. So, I decided to give a score of 5 as the initial rating. I am happy to raise my score if my concerns (together with those from other reviewers) are properly addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Manipulating deformable objects is an up-and-coming topic in robotics and machine learning, and it creates many interesting scientific and real-world challenges. The paper looks into long horizon tasks of manipulation of deformable objects, using an interesting mix of more local trajectory optimization and differentiable physics. The reviewers agree on the interesting significance of the suggest work, all above acceptance threshold, but also a bit bimodal in terms of “just above” vs. “solidly good”. Thus, the paper appears an useful and discussion-provoking accept for ICLR.
Decision: Accept (Poster)