DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools

1 INTRODUCTION . Robot manipulation of deformable objects is a fundamental research problem in the field of robotics and AI and has many real-world applications such as laundry making ( Maitin-Shepard et al. , 2010 ) , cooking ( Bollini et al. , 2013 ) , and caregiving tasks like assistive dressing , feeding or bed bathing ( Erickson et al. , 2020 ) . The recent development of differentiable physics simulators for deformable objects has shown promising results for solving soft-body control problems ( Hu et al. , 2019b ; Murthy et al. , 2020 ; Heiden et al. , 2021 ; Huang et al. , 2021 ) . These differentiable simulators have facilitated gradient-based trajectory optimizers to find a motion trajectory with much fewer samples , compared with black box optimizers such as CEM or reinforcement learning algorithms ( Huang et al. , 2021 ; Heiden et al. , 2021 ; Geilinger et al. , 2020 ) . However , the usage of these simulators is often limited in two ways . First , most differentiable physics simulators are only applied to solve short-horizon , single-skill tasks , such as cutting ( Heiden et al. , 2021 ) , ball throwing ( Geilinger et al. , 2020 ) , or locomotion ( Hu et al. , 2019b ) . This is partly because the gradient only provides local information , and thus , gradient-based optimizers often get stuck in local optima , preventing them from solving long horizon tasks . For example , consider a two-stage task of first gathering scattered dough on a table and then putting it on the cutting board using a spatula ( Figure 1 ) . If the objective function is to minimize the distance between all the dough and the cutting board , a gradient-based trajectory optimizer would directly use the spatula to put a small amount of dough on the cutting board without ever using the gathering operation . Second , the agent needs to know the full simulation state and relevant physical parameters in order to perform gradient-based trajectory optimization , which is a great limiting factor for real-world generalization , as reliable full state estimation of deformable objects from sensory data like RGB-D images can often be challenging and sometimes impossible due to ambiguities caused by self-occlusions . 1https : //sites.google.com/view/iclr2022diffskill Planning over the temporal abstraction of skills is a powerful approach to tackle the first issue of solving long horizon tasks . However , finding a suitable set of skills is challenging . For example , while standard skills such as grasping an object or moving the robot arm from one pose to another may be manually specified ( Toussaint et al. , 2018 ) , more complex skills such as cutting or gathering can be difficult to define manually . Therefore , it is desirable to learn these skills automatically . In this work , we aim to solve a novel set of sequential dough manipulation tasks that involve operating various tools and multiple stages of actions ( Figure 1 ) , including lifting and spreading , gathering and transporting , cutting and rearranging the dough . To extend the use of differentiable physics models to these long-horizon tasks and enable the agent to directly consume visual observations , we propose DiffSkill : a novel framework where the agent learns skill abstraction using the differentiable physics model and composes them to accomplish complicated manipulation tasks . Our method consists of three components , ( 1 ) a trajectory optimizer that acts as an expert that applies gradient-based optimization on the differentiable simulator to obtain demonstration trajectories , which requires the full state information from the simulator ( 2 ) a neural skill abstractor that is instantiated as a goal-conditioned policy , taking RGBD images as input and imitating the demonstration trajectories , and ( 3 ) a planner that learns to assemble the abstracted skills by opgimizing the intermediate goals to reach and solve the long horizon tasks . Experiments show that our method , operating on the high-dimensional RGB-D images , can successfully accomplish a set of the dough manipulation tasks , which greatly outperforms the model-free RL baselines and the standalone gradient-based trajectory optimizer . 2 RELATED WORK . Motion Planning with Differentiable Physics . Differentiable physics models provide the gradients of the future states with respect to the initial state and the input actions . Compared with black-box dynamics models , planning with differentiable physics models can often make more accurate updates on the action sequences and deliver better sample efficiency ( Huang et al. , 2021 ) . Over the past few years , researchers have built differentiable physics models from first principles rooted in our understanding of physics for various physical systems , ranging from multi-body systems ( Degrave et al. , 2019 ; Tedrake & the Drake Development Team , 2019 ; de Avila Belbute-Peres et al. , 2018 ; Geilinger et al. , 2020 ) , articulated bodies ( Werling et al. , 2021 ; Qiao et al. , 2021 ) , cloth ( Qiao et al. , 2020 ; Liang et al. , 2019 ) , fluid ( Schenck & Fox , 2018 ) , plasticine ( Huang et al. , 2021 ) , to soft robots ( Hu et al. , 2019b ; a ; Du et al. , 2021 ) . They have shown impressive performance in using the gradient to solve inverse problems like parameter identification and control synthesis . A complement thread of methods tried to relax the assumption that we have to know the full state and the physics equations ; instead , they employ deep neural networks to learn the dynamics models directly from observation data and use the gradients from the learned models to aid motion planning or policy learning ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Mrowca et al. , 2018 ; Ummenhofer et al. , 2019 ; Sanchez-Gonzalez et al. , 2020 ; Pfaff et al. , 2020 ; Sanchez-Gonzalez et al. , 2018 ; Li et al. , 2019 ; 2018 ) . Our method differs from prior works in that we use the differentiable physics model for skill abstract and compose the learned skills for long-horizon manipulation of deformable objects . Deformable Object Manipulation . Deformable objects have more degrees of freedom and are typically more challenging to manipulate than rigid objects . Extensive work has been done in this area , focusing on different types of deformable objects , including plasticine ( Li et al. , 2018 ; Huang et al. , 2021 ) , fluid ( Li et al. , 2021 ) , cloth ( Maitin-Shepard et al. , 2010 ; Lin et al. , 2021 ; Weng et al. , 2021 ; Ha & Song , 2021 ; Wu et al. , 2019 ; Yan et al. , 2020 ; Ganapathi et al. , 2020 ) , rope ( Nair et al. , 2017 ; Sundaresan et al. , 2020 ) , and object piles ( Suh & Tedrake , 2020 ) . People have also come up with standard benchmarks to establish a fairer comparison between various algorithms ( Lin et al. , 2020 ) . Our method aims to manipulate dough using tools similar to how humans would do in a kitchen environment ( Figure 1 ) . There are also a series of works on tool-using for object manipulation ( Toussaint et al. , 2018 ; Qin et al. , 2020 ; Fang et al. , 2020 ; Xie et al. , 2019 ) , but most of them focus on rigid objects , whereas we take a step further and tackle deformable objects . Solving Long Horizon Tasks . Our work aims to solve long-horizon manipulation tasks ; thus , it is also closely related to task and motion planning ( TAMP ) that contains elements of discrete task planning and continuous motion planning . Typical solutions to TAMP involve formulating the problem as mixed-integer programming ( MIP ) over state or belief space and solving the program using branch-and-bound techniques ( Gravot et al. , 2005 ; Kaelbling & Lozano-Pérez , 2010 ; 2013 ; Srivastava et al. , 2014 ; Toussaint , 2015 ; Toussaint et al. , 2018 ; Driess et al. , 2020 ; Garrett et al. , 2020 ) . Please refer to Garrett et al . ( 2021 ) for a detailed and comprehensive survey of the related works . Differs from prior works , our method directly operates on the pixel space and generates motion trajectories and skill abstractions by applying gradient-based trajectory optimization techniques using the differentiable physics model . The abstracted skills are then composed sequentially to accomplish long-horizon manipulation tasks according to a higher-level task planner . On the other hand , hierarchical reinforcement learning provides a general framework for learning both the low-level policy and a high-level controller at the same time through exploration ( Dietterich , 2000 ; Bacon et al. , 2017 ; Levy et al. , 2017 ) , where the two-level controllers make decisions at a different temporal scale . However , learning the temporal structure proves difficult , especially from high-dimension visual observation . Compared to hierarchical RL , our method takes advantage of the skills learned with differentiable physics and greatly reduces the difficulty of learning a high-level controller or planner . 3 METHOD . Our goal is to learn a policy to perform sequential deformable object manipulation using tools from sensory observations . We assume that we are learning these skills in a differentiable physics simulator . Since it is not feasible to directly use a standalone differentiable physics solver to find an optimal solution for long-horizontal tasks , we propose to first learn to abstract primitive skills from this differentiable physics simulator ; we then plan on top of these skills to solve long-horizon tasks . An overview of our framework is shown in Figure 2 . 3.1 PROBLEM FORMULATION . We consider a Markov Decision Process ( MDP ) defined by a set of states s ∈ S , actions a ∈ A and a deterministic , differentiable transition dynamics st+1 = p ( st , at ) , with t indexing the discrete time . At each timestep , the agent only has access to an observation o ∈ O ( such as an image ) instead of directly observing the state . For any goal state sg ∈ G , a distance function from the state s is given as D ( s , sg ) . The objective is to find a policy at = π ( o , og ) that minimizes the final distance to the goal D ( sT , sg ) , where T is the length of an episode . 3.2 COLLECTING DEMONSTRATION TRAJECTORIES WITH DIFFERENTIABLE PHYSICS . Previous work has shown that differentiable physics solvers can acquire short-horizon skills for deformable object manipulation in tens of iterations ( Huang et al. , 2021 ) . Inspired by this work , we first collect demonstration trajectories of using each tool to achieve a short-term goal . Concretely , given an initial state s0 , a goal state sg and the transition dynamics p of a differentiable simulator , we use gradient-based trajectory optimization to solve for an open-loop action sequence ( Kelley , 1960 ) . Specifically , we solve arg min a0 , ... aT−1 L ( a0 , . . . aT−1 ) = arg min a0 , ... aT−1 T∑ t=1 D ( st , sg ) +λ T∑ t=1 E ( m ( st ) , d ( st ) ) , where st+1 = p ( st , at ) , ( 1 ) In the case of deformable object manipulation , we represent the current and target shape of the deformable object as two sets of particles . We use the Earth Mover Distance ( EMD ) between the two particle sets as the distance metric D ( st , sg ) , which we approximate with the Sinkhorn Divergence ( Séjourné et al. , 2019 ) . To encourage the manipulator to approach the deformable object , we additionally add into the objective the Euclidean distance between the manipulator and the deformable object E ( m ( st ) , d ( st ) ) , with a weight λ , where m ( st ) and d ( st ) are the positions of the center of mass of the manipulator and deformable object at time t , respectively . We solve Equation 1 by updating the action sequence using∇atL , t = 0 . . . T , with an Adam optimizer ( Kingma & Ba , 2014 ) , initialized using an action sequence of all zeros . In this paper , we define a “ skill '' as using a single tool to achieve a short-horizon goal sg , starting from an initial state s0 . Skills are based on the tool alone and not the observations . A single skill is applied to different observations . For example , in the LiftSpread task , the skill of using the rolling pin can be applied to either when the dough is on the right or when the dough is on the cutting board . Assuming that we have K tools , the action space over all tools can be written as [ A ( 1 ) , . . .A ( K ) ] , where A ( k ) is the action space of the kth tool . During demonstration collection , for each short-horizon goal sg , we run the trajectory optimizer for each tool separately , by masking the actions for other tools to be zero at each timestep .

Summary: This paper proposes a method to learn long-horizon control policies. It uses differentiable physics simulator to generate skills for the latter imitator to learn. To complete the long-horizon task, the proposed method first exhaustively search for tools/skills to be used in each step. Next, it optimizes the intermediate observation states to maximize the feasibility and the reward through the trajectory. At last, the policy of each small step is obtained by feeding start/end state pairs to the imitator. the results show that the proposed method is much better than both traditional RL and the trajectory optimization offered by differentiable physics alone.