Year: 2021

Abstract:
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to the environment state and help trajectory optimization to converge orders of magnitude faster than model-free reinforcement learning algorithms for deformable object manipulation. However, such gradient-based trajectory optimization typically requires access to the full simulator states and can only solve short-horizon, single-skill tasks due to local optima. In this work, we propose a novel framework, named DiffSkill, that uses a differentiable physics simulator for skill abstraction to solve long-horizon deformable object manipulation tasks from sensory observations. In particular, we first obtain short-horizon skills using individual tools from a gradient-based optimizer, using the full state information in a differentiable simulator; we then learn a neural skill abstractor from the demonstration trajectories which takes RGBD images as input. Finally, we plan over the skills by finding the intermediate goals and then solve long-horizon tasks. We show the advantages of our method in a new set of sequential deformable object manipulation tasks compared to previous reinforcement learning algorithms and compared to the trajectory optimizer. Videos are available at our project page1.

heading: 1 INTRODUCTION
text: Robot manipulation of deformable objects is a fundamental research problem in the field of robotics and AI and has many real-world applications such as laundry making (Maitin-Shepard et al., 2010), cooking (Bollini et al., 2013), and caregiving tasks like assistive dressing, feeding or bed bathing (Erickson et al., 2020). The recent development of differentiable physics simulators for deformable objects has shown promising results for solving soft-body control problems (Hu et al., 2019b; Murthy et al., 2020; Heiden et al., 2021; Huang et al., 2021). These differentiable simulators have facilitated gradient-based trajectory optimizers to find a motion trajectory with much fewer samples, compared with black box optimizers such as CEM or reinforcement learning algorithms (Huang et al., 2021; Heiden et al., 2021; Geilinger et al., 2020).
However, the usage of these simulators is often limited in two ways. First, most differentiable physics simulators are only applied to solve short-horizon, single-skill tasks, such as cutting (Heiden et al., 2021), ball throwing (Geilinger et al., 2020), or locomotion (Hu et al., 2019b). This is partly because the gradient only provides local information, and thus, gradient-based optimizers often get stuck in local optima, preventing them from solving long horizon tasks. For example, consider a two-stage task of first gathering scattered dough on a table and then putting it on the cutting board using a spatula (Figure 1). If the objective function is to minimize the distance between all the dough and the cutting board, a gradient-based trajectory optimizer would directly use the spatula to put a small amount of dough on the cutting board without ever using the gathering operation. Second, the agent needs to know the full simulation state and relevant physical parameters in order to perform gradient-based trajectory optimization, which is a great limiting factor for real-world generalization, as reliable full state estimation of deformable objects from sensory data like RGB-D images can often be challenging and sometimes impossible due to ambiguities caused by self-occlusions.
1https://sites.google.com/view/iclr2022diffskill
Planning over the temporal abstraction of skills is a powerful approach to tackle the first issue of solving long horizon tasks. However, finding a suitable set of skills is challenging. For example, while standard skills such as grasping an object or moving the robot arm from one pose to another may be manually specified (Toussaint et al., 2018), more complex skills such as cutting or gathering can be difficult to define manually. Therefore, it is desirable to learn these skills automatically.
In this work, we aim to solve a novel set of sequential dough manipulation tasks that involve operating various tools and multiple stages of actions (Figure 1), including lifting and spreading, gathering and transporting, cutting and rearranging the dough. To extend the use of differentiable physics models to these long-horizon tasks and enable the agent to directly consume visual observations, we propose DiffSkill: a novel framework where the agent learns skill abstraction using the differentiable physics model and composes them to accomplish complicated manipulation tasks. Our method consists of three components, (1) a trajectory optimizer that acts as an expert that applies gradient-based optimization on the differentiable simulator to obtain demonstration trajectories, which requires the full state information from the simulator (2) a neural skill abstractor that is instantiated as a goal-conditioned policy, taking RGBD images as input and imitating the demonstration trajectories, and (3) a planner that learns to assemble the abstracted skills by opgimizing the intermediate goals to reach and solve the long horizon tasks. Experiments show that our method, operating on the high-dimensional RGB-D images, can successfully accomplish a set of the dough manipulation tasks, which greatly outperforms the model-free RL baselines and the standalone gradient-based trajectory optimizer.

heading: 2 RELATED WORK
text: Motion Planning with Differentiable Physics. Differentiable physics models provide the gradients of the future states with respect to the initial state and the input actions. Compared with black-box dynamics models, planning with differentiable physics models can often make more accurate updates on the action sequences and deliver better sample efficiency (Huang et al., 2021). Over the past few years, researchers have built differentiable physics models from first principles rooted in our understanding of physics for various physical systems, ranging from multi-body systems (Degrave et al., 2019; Tedrake & the Drake Development Team, 2019; de Avila Belbute-Peres et al., 2018; Geilinger et al., 2020), articulated bodies (Werling et al., 2021; Qiao et al., 2021), cloth (Qiao et al., 2020; Liang et al., 2019), fluid (Schenck & Fox, 2018), plasticine (Huang et al., 2021), to soft robots (Hu et al., 2019b;a; Du et al., 2021). They have shown impressive performance in using the gradient to solve inverse problems like parameter identification and control synthesis. A complement thread of methods tried to relax the assumption that we have to know the full state and the physics equations; instead, they employ deep neural networks to learn the dynamics models directly from observation data and use the gradients from the learned models to aid motion planning or policy
learning (Battaglia et al., 2016; Chang et al., 2016; Mrowca et al., 2018; Ummenhofer et al., 2019; Sanchez-Gonzalez et al., 2020; Pfaff et al., 2020; Sanchez-Gonzalez et al., 2018; Li et al., 2019; 2018). Our method differs from prior works in that we use the differentiable physics model for skill abstract and compose the learned skills for long-horizon manipulation of deformable objects.
Deformable Object Manipulation. Deformable objects have more degrees of freedom and are typically more challenging to manipulate than rigid objects. Extensive work has been done in this area, focusing on different types of deformable objects, including plasticine (Li et al., 2018; Huang et al., 2021), fluid (Li et al., 2021), cloth (Maitin-Shepard et al., 2010; Lin et al., 2021; Weng et al., 2021; Ha & Song, 2021; Wu et al., 2019; Yan et al., 2020; Ganapathi et al., 2020), rope (Nair et al., 2017; Sundaresan et al., 2020), and object piles (Suh & Tedrake, 2020). People have also come up with standard benchmarks to establish a fairer comparison between various algorithms (Lin et al., 2020). Our method aims to manipulate dough using tools similar to how humans would do in a kitchen environment (Figure 1). There are also a series of works on tool-using for object manipulation (Toussaint et al., 2018; Qin et al., 2020; Fang et al., 2020; Xie et al., 2019), but most of them focus on rigid objects, whereas we take a step further and tackle deformable objects.
Solving Long Horizon Tasks. Our work aims to solve long-horizon manipulation tasks; thus, it is also closely related to task and motion planning (TAMP) that contains elements of discrete task planning and continuous motion planning. Typical solutions to TAMP involve formulating the problem as mixed-integer programming (MIP) over state or belief space and solving the program using branch-and-bound techniques (Gravot et al., 2005; Kaelbling & Lozano-Pérez, 2010; 2013; Srivastava et al., 2014; Toussaint, 2015; Toussaint et al., 2018; Driess et al., 2020; Garrett et al., 2020). Please refer to Garrett et al. (2021) for a detailed and comprehensive survey of the related works. Differs from prior works, our method directly operates on the pixel space and generates motion trajectories and skill abstractions by applying gradient-based trajectory optimization techniques using the differentiable physics model. The abstracted skills are then composed sequentially to accomplish long-horizon manipulation tasks according to a higher-level task planner.
On the other hand, hierarchical reinforcement learning provides a general framework for learning both the low-level policy and a high-level controller at the same time through exploration (Dietterich, 2000; Bacon et al., 2017; Levy et al., 2017), where the two-level controllers make decisions at a different temporal scale. However, learning the temporal structure proves difficult, especially from high-dimension visual observation. Compared to hierarchical RL, our method takes advantage of the skills learned with differentiable physics and greatly reduces the difficulty of learning a high-level controller or planner.

heading: 3 METHOD
text: Our goal is to learn a policy to perform sequential deformable object manipulation using tools from sensory observations. We assume that we are learning these skills in a differentiable physics simulator. Since it is not feasible to directly use a standalone differentiable physics solver to find an optimal solution for long-horizontal tasks, we propose to first learn to abstract primitive skills from this differentiable physics simulator; we then plan on top of these skills to solve long-horizon tasks. An overview of our framework is shown in Figure 2.

heading: 3.1 PROBLEM FORMULATION
text: We consider a Markov Decision Process (MDP) defined by a set of states s ∈ S , actions a ∈ A and a deterministic, differentiable transition dynamics st+1 = p(st, at), with t indexing the discrete time. At each timestep, the agent only has access to an observation o ∈ O (such as an image) instead of directly observing the state. For any goal state sg ∈ G, a distance function from the state s is given as D(s, sg). The objective is to find a policy at = π(o, og) that minimizes the final distance to the goal D(sT , sg), where T is the length of an episode.

heading: 3.2 COLLECTING DEMONSTRATION TRAJECTORIES WITH DIFFERENTIABLE PHYSICS
text: Previous work has shown that differentiable physics solvers can acquire short-horizon skills for deformable object manipulation in tens of iterations (Huang et al., 2021). Inspired by this work, we first collect demonstration trajectories of using each tool to achieve a short-term goal. Concretely,
given an initial state s0, a goal state sg and the transition dynamics p of a differentiable simulator, we use gradient-based trajectory optimization to solve for an open-loop action sequence (Kelley, 1960). Specifically, we solve
arg min a0,...aT−1 L(a0, . . . aT−1) = arg min a0,...aT−1 T∑ t=1 D(st, sg)+λ T∑ t=1 E(m(st), d(st)), where st+1 = p(st, at), (1) In the case of deformable object manipulation, we represent the current and target shape of the deformable object as two sets of particles. We use the Earth Mover Distance (EMD) between the two particle sets as the distance metric D(st, sg), which we approximate with the Sinkhorn Divergence (Séjourné et al., 2019) . To encourage the manipulator to approach the deformable object, we additionally add into the objective the Euclidean distance between the manipulator and the deformable object E(m(st), d(st)), with a weight λ, where m(st) and d(st) are the positions of the center of mass of the manipulator and deformable object at time t, respectively. We solve Equation 1 by updating the action sequence using∇atL, t = 0 . . . T, with an Adam optimizer (Kingma & Ba, 2014), initialized using an action sequence of all zeros.
In this paper, we define a “skill" as using a single tool to achieve a short-horizon goal sg , starting from an initial state s0. Skills are based on the tool alone and not the observations. A single skill is applied to different observations. For example, in the LiftSpread task, the skill of using the rolling pin can be applied to either when the dough is on the right or when the dough is on the cutting board. Assuming that we have K tools, the action space over all tools can be written as [A(1), . . .A(K)], where A(k) is the action space of the kth tool. During demonstration collection, for each short-horizon goal sg , we run the trajectory optimizer for each tool separately, by masking the actions for other tools to be zero at each timestep.

heading: 3.3 NEURAL SKILL ABSTRACTION
text: Although the trajectory optimizer is able to provide solutions for short-horizon tasks, it is unable to solve long-horizon tasks due to local optima. Furthermore, running the trajectory optimizer requires knowing the full state of the environment, including particle positions and mass distributions of the deformable objects and the physical parameters of these objects. This information is difficult to
obtain during real-world deployment, where the observations available to a robot are from sensory information such as RGB-D images. Additionally, the trajectory optimizer takes minutes to run, which is too slow during evaluation for real-time applications.
As such, we propose to learn a neural skill abstractor that learns skills from the demonstration videos of a trajectory optimizer; we will then leverage these skills for solving long horizon tasks. Our neural skill abstraction consists of a goal-conditioned policy that takes a sensory observation (RGB-D images in our case) as input, a feasibility and reward predictor, as well as a variational auto-encoder (VAE).
Goal conditioned policy: For each tool, we learn a goal-conditioned neural policy at = πk(ot, og) ∈ A(k), using behavior cloning from the corresponding demonstration trajectories collected for the kth tool. Given the demonstration trajectory (s0, a0, . . . , sT ) and the corresponding RGB-D sensory observations (o0, . . . oT ), as well as an observation of the goal og, we train a policy using the MSE loss Lpolicy = ||π(ot, og)− at||22. Furthermore, to make the policy more robust to goal observations outside of the training goal images, we adopt hindsight relabeling (Andrychowicz et al., 2017). When sampling from the demonstration buffer, with a probability phind, we will relabel the original goal og with a hindsight goal ōi, where ōi ∼ Uniform{ot+1, . . . oT }. The intuition is that, for any goal that will be achieved by the current action sequence, the policy should imitate the current action.
Feasibility Predictor: For each skill, we also learn a feasibility predictor fk(ot, og) ∈ R to determine the feasibility of reaching from ot to og using the trajectory optimizer. For training the feasibility predictor, we obtain positive pairs by sampling (ot1 , ot2) from the same demonstration trajectory and negative pairs by sampling pairs of observations from different trajectories. We assign a label of 1 for positive pairs and a label of 0 for negative pairs. We use an MSE loss Lfea for model training, which was shown empirically to work slightly better than a cross-entropy loss. While there may be false negative pairs, we find the feasibility predictor to perform reasonably well, as different trajectories have different initial and goal configurations and it is less likely that the state of a different configuration (e.g. mass of the dough) can be achieved from the current state within one skill. This intuition was similarly used for goal relabeling in previous work by Lin et al. (2019).
Reward Predictor: We further train a reward predictor r(ot, og) ∈ R that predicts the negative of the Sinkhorn divergence between the corresponding states −D(st, sg) using an MSE loss Lr. The reward predictor does not depend on any specific skill or tool.
Variational Auto-Encoder: As explained in Section 3.4, we will plan to compose skills in a latent space instead of optimizing directly in the image space. To learn the latent space, we train a generative model of the observation space, specifically a VAE. The VAE includes an encoder z = Q(o) that encodes an observation into a fixed length latent vector, and a decoder o = G(z) that decodes back into the observation space. The VAE is trained such that G(z), z ∼ N (0, I) reproduces the observation distribution. The feasibility and reward predictor shares the encoder with the VAE. Thus, we can also write the feasibility and reward predictor as
fk(ot, og) = f̄k(Q(ot), Q(og)) = f̄k(zt, zg), r(ot, og) = r̄(Q(ot), Q(og)) = r̄(zt, zg).
As such, the feasibility and reward predictor can be used with only latent vectors from the VAE encoder as input.
Training We jointly optimize the loss function Lskill = Lpolicy + λfeaLfea + λrLr + λvaeLvae, where Lvae is the standard ELBO objective for VAE and λfea, λr, λvae are hyperparameters.

heading: 3.4 LONG-HORIZON PLANNING WITH SKILL ABSTRACTION
text: To apply the skill abstraction learned above sequentially, we need to determine 1) which skill to use at each stage; 2) what intermediate goal to specify for each skill. Given o0, og the initial and goal observations, we plan over H steps. By using the feasibility and score predictor, we formulate our problem as a hybrid discrete and continuous optimization problem:
arg min k1,z1,...,kH ,zH C(k, z) = − H−1∏ i=0 f̄ki(zi, zi+1)r̄(zH , zg), s.t. ||zi||22 ≤M, ∀i. (2)
Here, ki is the index of the tool used at step i, f̄ , r̄ are the feasibility and reward predictors that take latent vectors as input, and z0, zg are the VAE encoded latent vectors of the initial and goal observations (RGB-D image) o0, og. We assume the reward is positive. The optimization variables
include the discrete variables k = k1, . . . , kH , representing the index of the skills to use at each step and the continuous variables z = z1, . . . , zH that are latent vectors that represent the intermediate goals. The term ||zi||22 in the constraint is proportional to the log likelihood of the latent vectors under a unit normal distribution and M is a threshold to ensure that the latent vectors correspond to actual intermediate goals in the observation space.
To optimize for both the discrete and continuous variables in Eqn. 2, we use exhaustive search over all possible combinations of the discrete variables. For the continuous variables, we start with N initial solutions {z1, . . . zH}j , j = 1, . . . , N and use Adam with projected gradient on the loss for all the initial solutions in parallel, since the constraint is convex. Specifically, after each gradient update step of Adam, we project the current zi to the constraint set by setting zi = zimax(||zi||2/ √ M),1)
. Once we have solved for k1, z1, . . . , kH , zH , we can decode the latent vectors back to images o1, . . . , oH = G(z1), . . . , G(zH), and call the corresponding goal-conditioned policies sequentially: πk1(o0, o1), . . . , πkH (·, oH). To simplify the execution, we move each tool to its initial pose at the beginning of executing each skill. We also present the pseudocode of our method in Algorithm 1.
Algorithm 1: Solve long-horizon planning with DiffSkill Input : Trajectory optimizer, skill horizon T , planning horizon H Initialize modules for neural skill abstraction πk, fk, r, G,Q ; Generate N demonstration trajectories in differentiable physics τ = {(o0, a0, r0, . . . , oT , rT , og)}i=1,...,N ; Train neural skill abstraction πk, fk, r, G,Q using loss Lskill until convergence; for k ∈ {0 . . .K}H do
Initialize z = [z1, . . . zH ] ∼ N (0, I) ; Optimize z1, . . . zH according to Eqn. 2 to obtain cost C(k, z) ;
Choose k, z that minimizes C(k, z); for i← 0 to H do
Reset tools to initial poses ; Decode intermediate goal images: og,i ← G(zi) ; Execute policy πki (·, og,i) in the environment for T steps ;

heading: 4 EXPERIMENTS
text: In this section, we will discuss our experimental setup, implementation details and comparison results.

heading: 4.1 EXPERIMENTAL SETUP
text: Tasks and environments We experiment with a set of sequential deformable object manipulation tasks with dough. We build our simulation environments on top of PlasticineLab (Huang et al., 2021), a differentiable physics benchmark using the DiffTaichi system (Hu et al., 2019a) that could simulate plasticine-like objects based on the MLS-MPM algorithm (Hu et al., 2018). Inspired by the dumpling making process, we design three novel tasks that require long-horizon planning and usage of multiple tools:
• LiftSpread: The agent needs to first use a spatula (modeled as a thin surface) to lift a dough onto the cutting board and then adopt a rolling pin to roll over the dough to flatten it. The rolling pin is simulated as a 3-Dof capsule that can rotate along the long axis and the vertical axis and translate along the vertical axis to press the dough. • GatherTransport: Initially, residual of dough is scattered over the table. First, the agent needs to gather the dough with an extended parallel gripper and place it on top of a spatula. Then the agent needs to use the spatula to transport the dough onto the cutting board. The spatula can translate and rotate along the vertical plane. The gripper can translate along the horizontal plane, rotate around its center and open or close the gripper. • CutRearrange: This is a three-step task. Given an initial pile of dough, the agent needs to first cut the dough in half using a knife. Inspired by the recent cutting simulation (Heiden et al., 2021), we model the knife using a thin surface as the body and a prism as the blade. Next, the agent needs to use the gripper to transport each piece of the cut dough to target locations.
A visualization of these tasks can be found in Figure 1. For all the taks, the agent receives RGBD image resized to 64x64 from a camera and return velocity control commands directly on the tools.
Evaluation metric We report the normalize decrease in Sinkhorn diverge computed as s(t) = s0−sts0 , where s0, st are the initial and current Sinkhorn divergence. In this way, a normalized performance of 0 representing a policy that does nothing and a normalized performance of 1 representing an upper bound of a policy that perfectly match the two distributions. Note that the normalized performance can be negative if the policy makes the Sinkhorn divergence larger compared to the initial state. The maximum normalized performance of 1 may not always be possible. For example, due to the incompressibility of the dough, certain target shape may be too small for a large dough to fit into. As such, we additionally set a threshold on the Sinkhorn divergence for each task such that the task is assume to be completed successfully when EMD is below the threshold. This threshold is manually picked by observing the performance gap between successful and failed trajectories. For each task, 5 initial and target configurations are given for evaluation, which all require multi-stage manipulation of the dough. We report both the normalized performance metric and the success rate for comparisons.

heading: 4.2 IMPLEMENTATION DETAILS OF DIFFSKILL
text: For each task, we first generate 1000 different initial and goal configurations, varying the initial and target shape of the dough as well as poses of the manipulators. For each configuration, we run the optimizer for each tool separately to generate trajectories of length 50. The Adam optimizer starts with a initial action sequence of all zero and is then run for 200 iterations to optimize each trajectory.
We then jointly train our VAE, policy, feasibility and score predictors over this demonstration video dataset. These modules share the same encoder architecture: 4 convolutional layers with a kernel size of 4 and a stride of 2 and a channel size of (32, 64, 128, 256), followed by an MLP that maps the feature into a fixed 16 dimension vector. The VAE, feasibility and the score predictor also share the same weights for the encoder.
After training, we find the feasibility and score predictor to perform well on the held out trajectories, achieving a L2 error of less than 0.05 for the score predictor and an accuracy of over 0.95 for the feasibility trajectory, across different environments. We found behavior cloning to be sufficient for learning short-horizon skills from the demonstration dataset. In Table 3, we can see that the learned skills (labeled as Behavior Cloning) approach the normalized performance of the trajectory optimization (Trajectory Opt) on single-tool use, although they cannot solve the long-horizon tasks.
Given the skill abstraction, DiffSkill iterates through all skill combination. For each skill combination, we randomly sample 400 initial solutions for the latent vectors z1, . . . , zH , and then perform 1500 iterations of Adam optimization for each of them. Since we are optimizing in the latent space, and with the help of parallel computation of GPU, this optimization can be done efficiently in around 10 seconds for each skill combination on a single NVIDIA 2080Ti GPU.

heading: 4.3 BASELINES
text: We compare with three strong baselines:
Model-free Reinforcement Learning (RL) We compare with two model-free RL methods: TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018). The RL methods use the same encoder architecture as DiffSkilland receive the negative of the cost function used for the trajectory optimizer as the reward function. We train the RL agents with either a single tool, or with both tools where the agent controls them simultaneously.
Behavior Cloning We compare with another baseline that directly trains a goal-conditioned policy with Behavior Cloning (BC) using all tools. The agent is trained with the same dataset and jointly controls all tools.
Trajectory Opt (Oracle) We compare with the trajectory optimizer used to generate the demonstration, with the same parameters. Note that this method requires full state of the simulation and multiple forward and backward passes through the simulator during evaluation time.
For all baselines that sequentially apply multiple skills (BC, DiffSkill), we move any used tools to its initial pose after executing each skill with a manual policy.

heading: 4.4 RESULT ANALYSIS
text: We show that DiffSkill is able to solve the challenging long-horizon, tool-use tasks from the sensory observation (RGBD) while the baselines cannot. The quantitative results can be found in Table 3. Tool A refers to the spatula for LiftSpread, the parallel gripper for GatherTransport and Knife for the CutRearrange environment, while tool B refers to the rolling pin, the spatula and the gripper for each task respectively.
First, we can compare the rows for single-tool use. Trajectory Opt is able to solve one-stage of the task, achieving a reasonable performance, if the correct tool is used (e.g. tool A for LiftSpread and GatherTransport), although for cutting, the performance is still close to zero with cutting only, since cutting itself does not reduce the distance between the current dough and the desired dough locations. BC can well approach the performance of the corresponding trajectory optimizer despite being a policy that only receives RGBD input. On the other hand, RL performs poorly on tasks, showing that it is difficult to learn A good policy directly from visual input.
Second, with multiple tools, we can see that DiffSkill significantly outperforms the single-skill policy. Remarkably, DiffSkill even outperforms the trajectory optimizer that controls both tools at the same time, which uses the full simulation state during evaluation time. This indicates that high-level planning over skills is necessary for achieving these sequential manipulation tasks. Furthermore, we show visualization of the found solutions of DiffSkill in Figure 3. We can see that DiffSkill is able to find reasonable intermediate goal state such as first putting the dough on the cutting board, or gather dough onto the spatula. CutRearrange is shown to be a harder task, with all method performing poorly. This is because it requires three stages of manipulation and both the cutting and transporting dough without deforming the dough too much are non-trivial skills. Still, DiffSkill is able to achieve higher normalized improvement.

heading: 4.5 ABLATION ANALYSIS
text: We perform two ablations on DiffSkill. First, we try removing the planning over the discrete variables that decides which tool to use at each step. Instead, we train a tool-independent policy, the feasibility and score predictor, where the action space is the joint-tool action space. Then during planning, we only need to optimize for the intermediate goals. This ablation is labeled as No Discrete Planning.
Second, we try to remove the planning over the continuous variables, i.e. the intermediate goals. Without the intermediate goals, we directly use the final target image as the goal for sequentially executing each skill. This ablation is labeled as Direct Execution. Since we do not have the intermediate goals anymore, we try two different ways of choosing the skills to execute at each stage: Randomly pick one skill and an oracle that execute each combination of skills in the simulator and choose the best one in hindsight. The results are shown in Table 2. Without discrete planning, the policy performs poorly. This is probably because during training, a single policy and feasibility
predictor is used for learning two modes of skills that are very different and the policy is unable to differentiable for different modes or decide when to switch modes. On the other hand, if we do not optimize for the intermediate goals, we also cannot determine which tools to use at evaluation time, since both the feasibility predictor requires intermediate goals as input. In this case, we can see that using random skills at each stage results in poor performance. Even if the oracle skill order is used, there is still a drop in performance as the policy may not work well given only a future goal instead of an immediate goal.

heading: 5 CONCLUSIONS
text: In this paper, we propose DiffSkill, a novel framework for learning skill abstraction from differentiable physics and compose them to solve long-horizontal deformable object manipulations tasks from sensory observation. We evaluate our model on a series of challenging tasks for sequential dough manipulation using different tools and demonstrate that it outperforms both reinforcement learning and standalone differentiable physics solver. We hope our work will be a first step towards more common application of differentiable simulators beyond solving short-horizon tasks.
There are a few interesting directions for future work. First, currently DiffSkill uses exhaustive search for planning over the discrete space. As the planning horizon and the number of skills grow larger for more complex tasks, exhaustive search quickly becomes infeasible. An interesting direction is to incorporate a heuristic policy or value function for more efficient planning. Second, similar to many other data-driven methods, while neural skill abstraction gives good prediction results on region where a lot of data are available, it performs worse when tested on situations that are more different from training. This can be remedied by either collecting more diverse training data in simulation, for example, under an online reinforcement learning framework, or by using a more structured representation beyond RGBD images, such as using an object-centric representation. Third, we hope to extend our current results to the real world, by using a more transferrable representation such as just a depth map or a point-cloud representation. Finally, we hope to see DiffSKill be applied to other similar tasks, such as those related to cloth manipulation.

heading: B COMPARISON WITH MODEL-FREE RL ON SINGLE-TOOL TASKS
text: In this work, we focus on solving long-horizon multi-tool tasks. But our proposed method is not limitied to the multi-tool setup. For the single-tool task, or more specifically the single-skill task, our method can be directly applied without planning over the discrete space of which tool to use. To show this, we show results on two single-tool, single-stage tasks: Lift and Spread, which corresponds to the two sub-tasks in LiftSpread, but with a different initial and target dough shapes. For both environments, only one tool can be used: the spatula for Lift and the rolling pin for Spread. The results are shown in Table 4. As these are two simpler tasks, DiffSkill can solve both tasks. SAC can also solve the simpler lift task, but has some trouble learning the spread task. Visualization of both methods on these environments can be found on our website.

heading: C COMPARISON OF JOINTLY TRAINING VAE VERSUS PRE-TRAINING VAE
text: We compare with two ways of training VAE: one is our method which jointly trains the VAE along with other modules and the other is to train the VAE first and then freeze the VAE encoder when training other modules. The planning performance on different tasks are shown in Table 5. We find that pre-training VAE provides a slight performance gain, although it takes longer to train overall since we need to train the VAE first.

Review summaries:
This paper proposes a method to learn long-horizon control policies. It uses differentiable physics simulator to generate skills for the latter imitator to learn. To complete the long-horizon task, the proposed method first exhaustively search for tools/skills to be used in each step. Next, it optimizes the intermediate observation states to maximize the feasibility and the reward through the trajectory. At last, the policy of each small step is obtained by feeding start/end state pairs to the imitator. the results show that the proposed method is much better than both traditional RL and the trajectory optimization offered by differentiable physics alone.

This work proposes a new framework to learn long-horizon control policies on multi-tool deformable object manipulation tasks. A differentiable physics simulator is combined with a trajectory optimizer to gather expert demonstrations of short-term skills. The skill expert trajectories are then used to learn a neural skill abstractor. Finally, a planner learns to assemble the skill policies to solve the long-term task by solving intermediate goals.

This paper proposes to combine a local controller with a global planner for long-horizon deformable object manipulation with different tools. The controller is modeled as a neural network that takes an RGBD image as input and outputs the control signal. The global planner iterates all possible tools and optimizes a utility function, which is the multiplication of predicted feasibility and reward function. Differentiable physics is used to generate demonstrations for short-horizon skills. All observations are embedded in a continuous latent space, on which the optimization is further performed.