DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools

1 INTRODUCTION . Robot manipulation of deformable objects is a fundamental research problem in the field of robotics and AI and has many real-world applications such as laundry making ( Maitin-Shepard et al. , 2010 ) , cooking ( Bollini et al. , 2013 ) , and caregiving tasks like assistive dressing , feeding or bed bathing ( Erickson et al. , 2020 ) . The recent development of differentiable physics simulators for deformable objects has shown promising results for solving soft-body control problems ( Hu et al. , 2019b ; Murthy et al. , 2020 ; Heiden et al. , 2021 ; Huang et al. , 2021 ) . These differentiable simulators have facilitated gradient-based trajectory optimizers to find a motion trajectory with much fewer samples , compared with black box optimizers such as CEM or reinforcement learning algorithms ( Huang et al. , 2021 ; Heiden et al. , 2021 ; Geilinger et al. , 2020 ) . However , the usage of these simulators is often limited in two ways . First , most differentiable physics simulators are only applied to solve short-horizon , single-skill tasks , such as cutting ( Heiden et al. , 2021 ) , ball throwing ( Geilinger et al. , 2020 ) , or locomotion ( Hu et al. , 2019b ) . This is partly because the gradient only provides local information , and thus , gradient-based optimizers often get stuck in local optima , preventing them from solving long horizon tasks . For example , consider a two-stage task of first gathering scattered dough on a table and then putting it on the cutting board using a spatula ( Figure 1 ) . If the objective function is to minimize the distance between all the dough and the cutting board , a gradient-based trajectory optimizer would directly use the spatula to put a small amount of dough on the cutting board without ever using the gathering operation . Second , the agent needs to know the full simulation state and relevant physical parameters in order to perform gradient-based trajectory optimization , which is a great limiting factor for real-world generalization , as reliable full state estimation of deformable objects from sensory data like RGB-D images can often be challenging and sometimes impossible due to ambiguities caused by self-occlusions . 1https : //sites.google.com/view/iclr2022diffskill Planning over the temporal abstraction of skills is a powerful approach to tackle the first issue of solving long horizon tasks . However , finding a suitable set of skills is challenging . For example , while standard skills such as grasping an object or moving the robot arm from one pose to another may be manually specified ( Toussaint et al. , 2018 ) , more complex skills such as cutting or gathering can be difficult to define manually . Therefore , it is desirable to learn these skills automatically . In this work , we aim to solve a novel set of sequential dough manipulation tasks that involve operating various tools and multiple stages of actions ( Figure 1 ) , including lifting and spreading , gathering and transporting , cutting and rearranging the dough . To extend the use of differentiable physics models to these long-horizon tasks and enable the agent to directly consume visual observations , we propose DiffSkill : a novel framework where the agent learns skill abstraction using the differentiable physics model and composes them to accomplish complicated manipulation tasks . Our method consists of three components , ( 1 ) a trajectory optimizer that acts as an expert that applies gradient-based optimization on the differentiable simulator to obtain demonstration trajectories , which requires the full state information from the simulator ( 2 ) a neural skill abstractor that is instantiated as a goal-conditioned policy , taking RGBD images as input and imitating the demonstration trajectories , and ( 3 ) a planner that learns to assemble the abstracted skills by opgimizing the intermediate goals to reach and solve the long horizon tasks . Experiments show that our method , operating on the high-dimensional RGB-D images , can successfully accomplish a set of the dough manipulation tasks , which greatly outperforms the model-free RL baselines and the standalone gradient-based trajectory optimizer . 2 RELATED WORK . Motion Planning with Differentiable Physics . Differentiable physics models provide the gradients of the future states with respect to the initial state and the input actions . Compared with black-box dynamics models , planning with differentiable physics models can often make more accurate updates on the action sequences and deliver better sample efficiency ( Huang et al. , 2021 ) . Over the past few years , researchers have built differentiable physics models from first principles rooted in our understanding of physics for various physical systems , ranging from multi-body systems ( Degrave et al. , 2019 ; Tedrake & the Drake Development Team , 2019 ; de Avila Belbute-Peres et al. , 2018 ; Geilinger et al. , 2020 ) , articulated bodies ( Werling et al. , 2021 ; Qiao et al. , 2021 ) , cloth ( Qiao et al. , 2020 ; Liang et al. , 2019 ) , fluid ( Schenck & Fox , 2018 ) , plasticine ( Huang et al. , 2021 ) , to soft robots ( Hu et al. , 2019b ; a ; Du et al. , 2021 ) . They have shown impressive performance in using the gradient to solve inverse problems like parameter identification and control synthesis . A complement thread of methods tried to relax the assumption that we have to know the full state and the physics equations ; instead , they employ deep neural networks to learn the dynamics models directly from observation data and use the gradients from the learned models to aid motion planning or policy learning ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Mrowca et al. , 2018 ; Ummenhofer et al. , 2019 ; Sanchez-Gonzalez et al. , 2020 ; Pfaff et al. , 2020 ; Sanchez-Gonzalez et al. , 2018 ; Li et al. , 2019 ; 2018 ) . Our method differs from prior works in that we use the differentiable physics model for skill abstract and compose the learned skills for long-horizon manipulation of deformable objects . Deformable Object Manipulation . Deformable objects have more degrees of freedom and are typically more challenging to manipulate than rigid objects . Extensive work has been done in this area , focusing on different types of deformable objects , including plasticine ( Li et al. , 2018 ; Huang et al. , 2021 ) , fluid ( Li et al. , 2021 ) , cloth ( Maitin-Shepard et al. , 2010 ; Lin et al. , 2021 ; Weng et al. , 2021 ; Ha & Song , 2021 ; Wu et al. , 2019 ; Yan et al. , 2020 ; Ganapathi et al. , 2020 ) , rope ( Nair et al. , 2017 ; Sundaresan et al. , 2020 ) , and object piles ( Suh & Tedrake , 2020 ) . People have also come up with standard benchmarks to establish a fairer comparison between various algorithms ( Lin et al. , 2020 ) . Our method aims to manipulate dough using tools similar to how humans would do in a kitchen environment ( Figure 1 ) . There are also a series of works on tool-using for object manipulation ( Toussaint et al. , 2018 ; Qin et al. , 2020 ; Fang et al. , 2020 ; Xie et al. , 2019 ) , but most of them focus on rigid objects , whereas we take a step further and tackle deformable objects . Solving Long Horizon Tasks . Our work aims to solve long-horizon manipulation tasks ; thus , it is also closely related to task and motion planning ( TAMP ) that contains elements of discrete task planning and continuous motion planning . Typical solutions to TAMP involve formulating the problem as mixed-integer programming ( MIP ) over state or belief space and solving the program using branch-and-bound techniques ( Gravot et al. , 2005 ; Kaelbling & Lozano-Pérez , 2010 ; 2013 ; Srivastava et al. , 2014 ; Toussaint , 2015 ; Toussaint et al. , 2018 ; Driess et al. , 2020 ; Garrett et al. , 2020 ) . Please refer to Garrett et al . ( 2021 ) for a detailed and comprehensive survey of the related works . Differs from prior works , our method directly operates on the pixel space and generates motion trajectories and skill abstractions by applying gradient-based trajectory optimization techniques using the differentiable physics model . The abstracted skills are then composed sequentially to accomplish long-horizon manipulation tasks according to a higher-level task planner . On the other hand , hierarchical reinforcement learning provides a general framework for learning both the low-level policy and a high-level controller at the same time through exploration ( Dietterich , 2000 ; Bacon et al. , 2017 ; Levy et al. , 2017 ) , where the two-level controllers make decisions at a different temporal scale . However , learning the temporal structure proves difficult , especially from high-dimension visual observation . Compared to hierarchical RL , our method takes advantage of the skills learned with differentiable physics and greatly reduces the difficulty of learning a high-level controller or planner . 3 METHOD . Our goal is to learn a policy to perform sequential deformable object manipulation using tools from sensory observations . We assume that we are learning these skills in a differentiable physics simulator . Since it is not feasible to directly use a standalone differentiable physics solver to find an optimal solution for long-horizontal tasks , we propose to first learn to abstract primitive skills from this differentiable physics simulator ; we then plan on top of these skills to solve long-horizon tasks . An overview of our framework is shown in Figure 2 . 3.1 PROBLEM FORMULATION . We consider a Markov Decision Process ( MDP ) defined by a set of states s ∈ S , actions a ∈ A and a deterministic , differentiable transition dynamics st+1 = p ( st , at ) , with t indexing the discrete time . At each timestep , the agent only has access to an observation o ∈ O ( such as an image ) instead of directly observing the state . For any goal state sg ∈ G , a distance function from the state s is given as D ( s , sg ) . The objective is to find a policy at = π ( o , og ) that minimizes the final distance to the goal D ( sT , sg ) , where T is the length of an episode . 3.2 COLLECTING DEMONSTRATION TRAJECTORIES WITH DIFFERENTIABLE PHYSICS . Previous work has shown that differentiable physics solvers can acquire short-horizon skills for deformable object manipulation in tens of iterations ( Huang et al. , 2021 ) . Inspired by this work , we first collect demonstration trajectories of using each tool to achieve a short-term goal . Concretely , given an initial state s0 , a goal state sg and the transition dynamics p of a differentiable simulator , we use gradient-based trajectory optimization to solve for an open-loop action sequence ( Kelley , 1960 ) . Specifically , we solve arg min a0 , ... aT−1 L ( a0 , . . . aT−1 ) = arg min a0 , ... aT−1 T∑ t=1 D ( st , sg ) +λ T∑ t=1 E ( m ( st ) , d ( st ) ) , where st+1 = p ( st , at ) , ( 1 ) In the case of deformable object manipulation , we represent the current and target shape of the deformable object as two sets of particles . We use the Earth Mover Distance ( EMD ) between the two particle sets as the distance metric D ( st , sg ) , which we approximate with the Sinkhorn Divergence ( Séjourné et al. , 2019 ) . To encourage the manipulator to approach the deformable object , we additionally add into the objective the Euclidean distance between the manipulator and the deformable object E ( m ( st ) , d ( st ) ) , with a weight λ , where m ( st ) and d ( st ) are the positions of the center of mass of the manipulator and deformable object at time t , respectively . We solve Equation 1 by updating the action sequence using∇atL , t = 0 . . . T , with an Adam optimizer ( Kingma & Ba , 2014 ) , initialized using an action sequence of all zeros . In this paper , we define a “ skill '' as using a single tool to achieve a short-horizon goal sg , starting from an initial state s0 . Skills are based on the tool alone and not the observations . A single skill is applied to different observations . For example , in the LiftSpread task , the skill of using the rolling pin can be applied to either when the dough is on the right or when the dough is on the cutting board . Assuming that we have K tools , the action space over all tools can be written as [ A ( 1 ) , . . .A ( K ) ] , where A ( k ) is the action space of the kth tool . During demonstration collection , for each short-horizon goal sg , we run the trajectory optimizer for each tool separately , by masking the actions for other tools to be zero at each timestep .

Summary: This work proposes a new framework to learn long-horizon control policies on multi-tool deformable object manipulation tasks. A differentiable physics simulator is combined with a trajectory optimizer to gather expert demonstrations of short-term skills. The skill expert trajectories are then used to learn a neural skill abstractor. Finally, a planner learns to assemble the skill policies to solve the long-term task by solving intermediate goals.