Title: DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differentiable physics; Soft body manipulation
Abstract: Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems.
Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory.
However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users.
We introduce DiffVL, a method that enables non-expert users to communicate soft-body manipulation tasks -- a combination of vision and natural language, given in multiple stages -- that can be readily leveraged by a differential physics solver. 
We have developed GUI tools that enable non-expert users to specify 100 tasks inspired by real-life soft-body manipulations from online videos, which we'll make public.
We leverage large language models to translate task descriptions into machine-interpretable optimization objectives. The optimization objectives can help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines.

DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics

Abstract

Combining gradient-based trajectory optimization with differentiable physics simulation is an accurate and efficient technique for solving soft-body manipulation problems. Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory. However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users. We introduce DiffVL, a framework that integrates the process from task collection to trajectory generation leveraging a combination of visual and linguistic task descriptions. A DiffVL task represents a long horizon soft-body manipulation problem as a sequence of 3D scenes (key frames) and natural language instructions connecting adjacent key frames. We built GUI tools and tasked non-expert users to transcribe 100 soft-body manipulation tasks inspired by real-life scenarios from online videos. We also developed a novel method that leverages large language models to translate task language descriptions into machine-interpretable optimization objectives, which can then help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines. Experiments show that existing baselines cannot complete complex tasks, while our method can solve them well. Videos can be found on the website https://sites.google.com/view/diffvl/home .

1 Introduction

This paper focuses on soft body manipulation, a research topic with a wide set of applications such as folding cloth [52, 90, 34], untangling cables [88, 87], and cooking foods [77, 72]. Due to their complicated physics and high degree of freedom, soft body manipulations raise unique opportunities and challenges. Recent works such as [32] and [99] have heavily leveraged various differentiable physics simulators to make these tasks tractable. Towards generalizable manipulation skill learning, such approaches have the potential to generate data for learning from demonstration

algorithms [51, 43]. However, to enable differentiable physics solvers to generate a large scale of meaningful trajectories, we must provide them with suitable tasks containing the scene and the optimization objectives to guide the solver. Previous tasks are mostly hand-designed [32] or procedure-generated [51], resulting in a lack of diverse and realistic soft-body manipulation tasks for researchers.

This paper takes another perspective by viewing tasks as data, or more precisely, task specifications as data. Each data point contains a pair of an initial scene and an optimization objective depicting the goal of the task. Taking this perspective, we can annotate meaningful tasks like annotating data of other modalities, like texts, images, videos, or each action trajectory, by leveraging non-expert human labor, providing possibilities for scaling up the task space of differentiable physics solvers.

Acore problem in building a task collection framework with trajectory generation is finding a suitable representation of the tasks. The representation should be intuitive for non-expert annotators while being accurate enough to describe the complex shapes and motions of the soft bodies that appear in various soft body manipulation processes. Last but not least, it should be reliably interpreted by differentiable physics solvers to yield a valid trajectory.

sample("RollingPin",

y(    ) > y(max(  ))

)

optimize(

keep(touch(  )),

require(simiar(  ,   ))

)

scene 1

D) Optimization Programs

Large

Language

Model

NL2: Put the rolling pin above the white dough on the

right and deform it into the target shape

scene 2

B) Task Annotation

E) Sample and Optimize

C) DiffVL Task

scene 3

A) Dumpling Video

NL2

NL1

NL1: ...

Add Object Mooule, which can control fhe

color, shape, and material of the object

Storage, and Basie

CameraControl,

Tools, Lighting. and

Sofbody Ink

a

V

Studies such as [3, 1, 60, 89] have shown that humans are proficient at defining goal-driven, spatial object manipulation tasks using either sketches, natural languages, or both. Figure 1(A), for instance, depicts the complete process of dumpling creation, supplemented by textual instructions. Taking this as an inspiration, we present DiffVL, a framework that enables non-expert users to specify soft-body manipulation tasks to a differentiable solver using a combination of vision and language. Specifically, each DiffVL task consists of a sequence of 3D scenes (keyframes), with natural language instruction connecting adjacent keyframes. The sequence of key frames specifies the sequence of subgoals of the manipulation task, and the natural language instructions provides suggestions on how to use the actuators to manipulate the objects through this sequence of subgoals. See Figure 1.

We develop tools to ease the annotation for non-expert users. With our interactive simulator equipped with a GUI, the user can edit, manipulate, draw, or carve shapes easily like other 3D editing tools and observe the consequence through simulation in an intuitive manner. Meanwhile, when it is tedious for users to edit all the intermediate steps of a complex motion, they can use natural language to describe the goal instead of drawing them step by step. This enables us to build SoftVL100 , a vision-language dataset with 100 diverse tasks. DiffVL uses LLM to compile the natural language instructions to an optimization program - which enforces a set of constraints during the soft-body manipulations from one keyframe to the next. This optimization program is then used by a differentiable physics solver to generate a working trajectory.

To summarize, our work makes the following contributions:

We propose a new multi-stage vision-language representation for defining soft-body manipulation tasks that is suitable for non-expert user annotations.

We developed a corresponding GUI, and curated SoftVL100, consisting of 100 realistic soft-body manipulation tasks from online videos 3 .

We develop a method, DiffVL, which marries the power of a large-language model and differentiable physics to solve a large variety of challenging long-horizon tasks in SoftVL100.

2 Related Work

Differentiable simulation for soft bodies Differentiable physics [30, 93, 69, 19, 29] has brought unique opportunities in manipulating deformable objects of various types [58, 28, 53, 33, 92, 49, 95, 83, 63, 100, 95, 56, 27, 47, 74, 23], including plasticine [32], cloth [45], ropes [54] and fluids [96] simulated with mass-spring models [30], Position-based Dynamics [65], Projective Dynamics [69, 16], MPM [36] or Finite Element Method [26], which could be differentiable through various techniques [30, 16, 57, 7]. It has been shown that soft bodies have smoother gradients compared to rigid bodies [32, 30, 93, 6, 82], enabling better optimization. However, the solver may still suffer from non-convexity [44] and discontinuities [82], motivating approaches to combine either stochastic sampling [44], demonstrations [43, 6] or reinforcement learning [98, 64] to overcome the aforementioned issues. DiffVL capitalizes on an off-the-shelf differentiable physics solver to annotate tasks. It incorporates human priors through a novel vision-language task representation. This allows users to employ natural language and GUI to guide the differentiable solver to solve long-horizon tasks, distinguishing DiffVL from previous methods.

Task representation for soft body manipulation There are various ways of defining tasks for soft body manipulation. To capture the complex shape variations of soft bodies, it is natural to use either RGB images [95, 51, 73] or 3D point cloud [78, 47] to represent goals. However, given only the final goal images, it may be hard for the solver to find a good solution without further guidance. Besides, generating diverse images or point clouds that adhere to physical constraints like gravity and collision avoidance in varied scenes can be challenging without appropriate annotation tools. Our GUI tools are designed to address these issues, simplifying the process. Other works use hand-defined features [10, 88] or formalized languages [41]. These methods necessitate specialist knowledge and require tasks to be defined on a case-by-case basis. Learning from demonstrations [43, 73] avoids defining the task explicitly but requires agents to follow demonstration trajectories and learn to condition on various goal representations through supervised learning or offline reinforcement learning [20, 18, 79, 24, 55, 35, 11, 2, 5]. However, collecting demonstrations step-by-step [70, 101, 43] requires non-trivial human efforts, is limited to robots with human-like morphologies, and may be challenging for collecting tasks that are non-trivial for humans (e.g., involving complex dynamics or tool manipulation). Our annotation tool emphasizes the description of tasks over the identification of solutions. Annotators are merely required to define the task, not execute the trajectories themselves, thereby simplifying the annotation process. Our method is also related to [46], but we consider a broader type of tasks and manipulation skills.

Language-driven robot learning Many works treat language as subgoals and learn languageconditioned policies from data [80, 21, 66, 42, 8, 62, 61] or reinforcement learning [37, 17, 84, 12]. It has been shown that conditioning policies on languages lead to improved generalization and transferability [80, 37]. [76] turns languages into constraints to correct trajectories. Our method distinguishes itself by integrating language with differentiable physics through machine-interpretable optimization programs. Recent works aim at leveraging large language models [4, 15, 94, 50, 13] to facilitate robot learning. Most use language or code [48] to represent policies rather than goals and focus on high-level planning while relying on pre-defined primitive actions or pre-trained policies for low-level control. In contrast, our language focuses on low-level physics by compiling language instructions to optimization objectives rather than immediate commands. It's noteworthy that VIMA [38] also introduces multimodal prompts, embodying a similar ethos to our visionlanguage task representation. Their approach views multimodal representation as the task prompts to guide the policy. However, it still requires pre-programmed oracles to generate offline datasets for learning, which is hard to generalize to soft-body tasks with complex dynamics. Contrastingly, we

adopt a data annotation perspective, providing a comprehensive framework for task collection, natural language compiling, and trajectory optimization, enabling us to harness the potential of differentiable physics effectively.

Task and motion planning Our optimization program compiled from natural language is closely connected to the field of Task and Motion Planning (TAMP) [22], particularly Logic Geometric Programming (LGP) [85, 86]. We draw inspiration from LGP's multiple logical states in designing our DiffVL task representation. However, unlike LGP, our approach asks human annotators to assign 'logic states' and facilitates the manipulation of soft bodies through our novel vision-language representation. Some recent works have explored the intersection of TAMP and language models [14, 81, 59] while primarily focusing on generating high-level plans.

3 SoftVL100: Vision-Language Driven Soft-body Manipulation Dataset

Grasp the back end of rope. Then Wind the rope and place the salmon ball.

Grasp the front end of rope. Then Wind the rope and place the black ball.

Grasp the front end of top yellow by gripper and place it on the bottom.

Then grasp the ropes by gripper to wind and place them into theirs goal shape.

Finally, grasp the red by gripper to place it on the top.

(A)Multistage Tasks

Put the rolling pin above the white and manipulate the white into goal.

Then grasp the yellow by gripper and place them into goal places.

Finally grasp the white and wrap it as well as the yellow in it.

3.8

D

(B) Scene examples

In this representation, the original tasks are divided into a sequence of 3D scenes, or key frames, along with text descriptions that detail the steps for progressing to the next scene. Figure 2(A) illustrates our representation. Each key frame consists of multiple soft bodies with different positions and shapes. By grouping two consecutive key frames together, we form a manipulation stage. Natural language instructions may guide the selection of an actuator and provide guidance on how to control it to reach the subsequent key frames. In Section 3.1, we develop annotation tools to facilitate the task representation annotation process. These tools make it easy to construct a diverse set of tasks, referred to as SoftVL100, as discussed in Section 3.2.

3.1 Vision-Language Task Annotator

Our annotation tool is based on PlasticineLab [32], a simulation platform that utilizes the Material Point Method (MPM) [31] for elastoplastic material simulation. To enhance the user experience and support scene creation, we integrate it into SAPIEN [97], a framework that enables the development of customized graphical user interfaces.

Our simulation runs on GPU servers and the interface is accessible as a web service through VNC, allowing users to interact with the simulator directly from their web browser. When

Select the object to edit.

Add a new sphere

Lift it up

Press it into a cylinder

Cut it into two objects

Annotator

The chosen sub-goal

</>

starting a new task annotation, users can open the simulator, perform actions, and save the current scene. Our GUI tools provide comprehensive support for creating, editing, managing, and simulating objects, as illustrated in Figure 3. Scene creation: The GUI offers a variety of primitive shapes, such as ropes, spheres, cubes, and cylinders, that users can add to the scene. Before adding a soft-body shape, users can adjust its size, color, and materials (e.g., rubber, fiber, dough, iron) if needed. Shape editing: Users have the ability to select, edit, or delete shapes within the scene. The GUI tool includes a range of shape operators, including moving, rotating, carving, and drawing. We provide a complete list of supported operations in the appendix. Simulation: Since the interface is built on a soft body simulator, users can simulate and interact with the soft body using rigid actuators or magic forces. They can also run simulations to check for stability and collision-free states. Object management: Users can merge connected shapes to create a single object or divide an object into two separate ones with distinct names. The annotation tool keeps tracking the identities of objects across scenes.

Once users have finished editing the scene and creating a new key frame, they can save the scene into the key frame sequence. The key frames are displayed below the simulator, visualizing the current task annotation progress. Users have the option to open any of these saved scenes in the simulator, delete a specific scene if needed, and add text annotations for each scene. Please refer to the appendix for detailed information on the functionalities and usage of the GUI tool.

3.2 The SoftVL100 Dataset

Our annotation tool simplifies the task creation process, enabling us to hire non-expert users to collect new tasks to form a new task set, SoftVL100. The task collection involves several stages, starting with crawling relevant videos from websites like YouTube that showcase soft body manipulation, particularly clay-making and dough manipulation. We developed tools to segment these videos and extract key frames to aid in task creation.

We hired students to annotate tasks, providing them with a comprehensive tutorial and step-by-step guidelines for using our annotation tools. It took approximately two hours for each annotator to become proficient in using the annotation tool. Subsequently, they were assigned a set of real-world videos to create similar tasks within the simulator. After collecting the keyframes, we proceed to relabel the scenes with text descriptions. Annotators are provided with a set of example text descriptions. We encourage users to include detailed descriptions of the actuators, their locations, moving directions, and any specific requirements, such as avoiding shape breakage for fragile materials. The annotation process for each task typically takes around 30 minutes. Using our task annotation tool, we have created a dataset called SoftVL100, which consists of 100 tasks, and there are more than 4 stages on average. The tasks cover a wide range of skills as illustrated in Figure 2 (A). Sample tasks from the dataset can be seen in Figure 2 (B).

4 Optimization with Vision-Language Task Description

We propose DiffVL to tackle the challenging tasks in SoftVL100. Given a stage of a task depicted in Figure 4(A), DiffVL utilizes large language models to compile the natural language instructions into a machine-interpretable optimization program in Figure 4(B). The program comprises of the names of visible elements and Python functions, effectively capturing the essence of the language instructions. We introduce the design of our DSL in Section 4.1 and outline the DiffVL compiler based on large language models (LLM) in Section 4.2. The resulting optimization program includes crucial information for selecting and locating actuators and can facilitate a differentiable physics solver to generate valid trajectories, as discussed in Section 4.3.

4.1 Optimization Program

The optimization program is formulated using a specialized domain-specific language (DSL) that incorporates several notable features. Firstly, it includes functions that extract the names of visible elements from the 3D scenes and support operations on these elements, represented as 3D point clouds. This enables manipulation and analysis of the visual information within the optimization program. Secondly, the DSL provides various functions to express geometric and temporal relations, encompassing common geometric and motion constraints. These functions facilitate the representation and handling of natural language instructions. Furthermore, the DSL clauses are interpreted

A) One Stage of A Task

RollingPin

= "RollingPin"

Dough

= get("white right dough")

Goal

= goal("white right dough")

sample(

RollingPin

,                      // use the rolling pin

y(

RollingPin

) > y(max(pcd(

Dough

)))  // above the dough

)

optimize(

keep(touch(

Dough

)),                 // deform the dough

require(similar(

Dough

,

Goal

))       // into the target shape

)

require(similar(

Dough

,

Goal

))

y(

RollingPin

) > y(max(

Dough)

)

Put the rolling pin above the white

dough on the right. Roll the pin to

deform it into the target shape.

keep(touch(

Dough

))

Current Frame

Next Frame

Large

Language

Model

RollingPin

B) Optimization Program

Dough

Goal

Extracted Visual

Elements

C) Smapling and Optimiation

Sampling Actuators

Rollout Trajectory

Ending states

y(

RollingPin

)

y(max(

Dough

))

>

dist(

RollingPin, Dough)

Dough

Goal

Earth Mover's Distance

VL

into PyTorch [68], making the constraints automatically differentiable and directly applicable for differentiable physics solvers.

The example in Figure 4 already exposes multiple components of our DSL. In the program, we can refer to the names of visible elements through get and goal functions, which can look for and extract the 3D objects that satisfy the description, identified by their color, shapes, and locations (we ensured that these attributes are sufficient to identify objects in our tasks). The program will then start with a sample statement, which is used to specify the actuators and their associated constraints over initial positions. Actuators refer to the entities utilized for manipulation, such as a robot gripper, a knife, or the rolling pin depicted in Figure 4. The constraint y(RollingPin) > y(max(pcd(Dough))) within the sample statement represents a specific constraint. It ensures that the height of the actuator (RollingPin) is greater than the highest y coordinate of the white dough in the scene, reflecting the meaning of 'put bove' of the language instructions. Following the sample statement, an optimize statement is included. This statement plays a vital role in defining the constraints and objectives for the manipulation trajectories. These constraints and objectives serve as the differentiable optimization objectives that the differentiable physics solver aims to maximize. In an optimization program, two special functions, namely require and keep , can be used to specify the temporal relationship of the optimization objectives. By default, the require function evaluates the specified condition at the end of the trajectories. On the other hand, the keep function is applied to each frame of the trajectories, ensuring that the specified conditions are maintained throughout the trajectory. In the example program shown in Figure 4(B), the optimization program instructs the actuator to continuously make contact with the white dough. The function similar computes the Earth Mover distance [71] between two objects' point clouds, which aims to shape the dough into a thin pie, the goal shape extracted from the next key frame. We illustrate more example optimization programs in Figure 5.

4.2 Compiling Natural Languages with LLM

We utilize the power of the Large Language Model (LLM) to compile text instructions into optimization programs. This approach capitalizes on the few-shot capabilities of the LLM and employs various few-shot prompts [91, 9]. The prompts begin by defining the 'types' and 'functions' within our DSL. Each function is accompanied by a language explanation to clarify its purpose. Additionally, examples are given to demonstrate how primitive functions can be combined to form complex clauses. To facilitate the translation, we provide the language model with both the scene description of the

A) Grasp the front of the rope and then grasp the back

of the rope. Deform the rope into its target shape.

sample("gripper", grasp(frontend(get("rope"))))

optimize(require(emd(get("rope"), goal("rope"))))

sample("gripper", grasp(backend(get("rope"))))

optimize(require(similar(get("rope"), goal("rope"))))

B) Use the gripper to first lift up the white cube above the

black wall and then move it to the target location

sample("gripper", grasp(get("white")))

optimize(

require(y(com("white"))>y(max(pcd("black"))), 0.5),

require(l2(com(get("white")), com(goal("white")))),

)

C) Put the knife above the white object. Cut the knife

down to match the target shape of the white object.

Ensure that the knife does not rotate.

sample("knife", roll(0.),

y("knife") > y(max(pcd("white"))))

optimize(

keep(touch(leftpart("white"))),

keep(roll() < 0.1), keep(pitch() < 0.1),

require(similar("white", goal("white"))),

)

D) Pick up the front part of object 'top left mat' and place

it into the goal without breaking it.

sample("gripper", touch(frontpart("top left mat"))

optimize(

keep(touch(frontpart("top left mat"))),

require(similar("top left mat",

goal("top left mat"))),

keep(no_break("top left mat")),

)

objects that it contains and the natural language instructions that we wish to compile. We then prompt the model with the instruction, 'Please translate the instructions into a program.' This step is repeated for all frames, resulting in their respective optimization programs. Through our experimentation, we have observed that GPT-4 [67] surpasses other models in terms of performance. Please refer to the appendix for a more comprehensive understanding of the translation process.

4.3 Solving the Optimization through Differentiable Physics

We leverage the differentiable physics solver in [32], which has been proven accurate and efficient in solving the generated optimization programs. The process is illustrated in Figure 4(C). The sample function is employed within a sampling-based motion planner. Initially, it samples the pose of the specified actuator type to fulfill the provided constraints. Subsequently, an RRT planner [40] determines a path toward the target location, and a PD controller ensures that the actuators adhere to the planned trajectory. With the actuator suitably initialized, the optimize clause is then passed to a gradient-based optimizer to optimize the action sequence for solving the task. The solver performs rollouts, as depicted in Figure 4(C), where the conditions enclosed by keep operators are evaluated at each time step. Conversely, conditions enclosed by require are evaluated only at specified time frames. The resulting differentiable values are accumulated to calculate a loss, which is utilized to compute gradients for optimizing the original trajectories. For handling multistage tasks involving vision-language representation, we can solve them incrementally, stage by stage.

5 Experiments

In this section, we aim to justify the effectiveness of our vision-language task representation in guiding the differentiable physics solver. For a detailed analysis of the ablation study on the language translator, we direct readers to the appendix. To better understand the performance of different solvers, we evaluate the tasks of two tracks. The first track focuses on tasks that only require agents to transition from one key frame to another. Tasks in this track are often short-horizon. This setup allows us to evaluate the efficiency and effectiveness of the differentiable physics solver in solving shorthorizon soft body manipulation tasks, as described in Section 5.2. In the second track, we validate the agent's capability to compose multiple key frames in order to solve long-horizon manipulation tasks.

These tasks involve complex scenarios where agents need to switch between different actuators and manipulate different objects. The evaluation of this composition is discussed in Section 5.3.

5.1 Initialization for tasks

Weclarify that our approach does not use a gradient-based optimizer to optimize trajectories generated by the RRT planner. Motion planning and optimization occur in separate temporal phases. Initially, we plan and execute a trajectory to position the actuator at the specified initial pose. Subsequently, we initialize a new trajectory starting from this pose, and the actuator remains fixed in this trajectory as we initialize the policy with zero actions to maintain its position. The sample function generates poses in line with the annotators' guidance.

5.2 Mastering Short Horizon Skills using Differentiable Physics

Before evaluating the full task set, we first study how our vision-language representation can create short-horizon tasks that are solvable by differentiable physics solvers. We picked 20 representative task stages as our test bed from the SoftVL100. We classify these tasks into 5 categories, with 4 tasks in each category. The categories are as follows: Deformation involves the deformation of a large single 3D shape, achieved through pinching and carving; Move asks the movement of an object or stacking one object onto another; Winding involves the manipulation of a rope-like object to achieve different configurations. Fold focuses on folding a thin mat to wrap other objects. Cut involves splitting an object into multiple pieces. The selected tasks encompass a wide spectrum of soft body manipulation skills, ranging from merging and splitting to shaping. We evaluate different methods' ability to manipulate the soft bodies toward their goal configuration. The agents are given the goal configuration in the next frame, and we measure the 3D IoU of the ending state as in [32]. We set a threshold for each scene's target IoU score to measure successful task completion, accommodating variations in IoU scores across different scenes.

We compare our method against previous baselines, including two representative reinforcement learning algorithms, Soft Actor-Critic (SAC) [25] and Proximal Policy Gradient (PPO) [75]. The RL agents perceive a colored 3D point cloud as the scene representation and undergo 10 6 steps to minimize the shape distance towards the goal configuration while contacting objects. Additionally, we compare our approach with CPDeform [44], which employs a differentiable physics solver. However, CPDeform uses the gradient of the Earth mover's distance as a heuristic for selecting the initial actuator position. It differs from DiffVL which utilizes the text description to determine ways to initialize the actuators and the optimization objectives. For differentiable physics solvers, we run Adam [39] optimization for 500 gradient steps using a learning rate of 0 . 02 .

Table 1 presents the success rates and mean IoU of various approaches. It is evident that, except for specific deformation and cutting tasks where SAC shows success, the RL baselines struggle to solve most tasks. Although RL agents demonstrate an understanding of how to manipulate the soft bodies correctly, they lack the precision required to match the target shape accurately. Additional experimental results can be found in the appendix. CPDeform achieves partial success in each task category. It surpasses the RL baselines by utilizing a differentiable physics solver and employing a heuristic that helps identify contact points to minimize shape differences. However, CPDeform's heuristic lacks awareness of physics principles and knowledge about yielding objects to navigate around obstacles. Consequently, CPDeform fails to perform effectively in other tasks due to these limitations. We want to emphasize that integrating language into RL baselines entails non-trivial challenges. For example, many reward functions in our optimization program have temporal aspects, making the original state non-Markovian. As we focus on enhancing the optimization baseline through language-based experiments, we've chosen to defer tackling the complexities of language-conditioned RL to future research.

Our approach surpasses all other methods, achieving the highest overall success rate. To examine the impact of different components and illustrate their effects, we conduct ablation studies. First, we remove the constraints in sample named '- Sample' in the table and ask the agent to sample random tools without using hints from the language instructions. As a result, the performance drops significantly, either getting stuck at the local optima or failing to select suitable tools to finish the task. For example, in the cutting task, it failed to select knives and split the white material into two pieces. On the other hand, the variant '- Optimize' replaces the statement in optimize by

-Optimize

Start

-Sample

(a) Move a single object to another side

Ours

Goal

(b) Pick up the front end of top left mat and use it to cover the brown core

(c) First pick up the back end of the yellow rope and move it across the orange one

(d) Use knife to cut out the left part of the white mat

(a)Multistage Task

(b) Our solution

(d) No actuator sample

(e) Single stage

(c) final only

(1)

(2)

品名

an objective focusing solely on optimizing shape distance, removing additional guidance for the physical process. The effects are lesser compared to the previous variant, demonstrating that with a suitable tool initialization method, a differentiable physics solver is already able to solve many tasks. However, as illustrated in Figure 6(A), it does not lift the white cube to avoid the black wall in a moving task and breaks the mat shown in the second row. In the winding task, it fails to find the relatively complex lifting-then-winding plan directly from shape minimization. In contrast, our method can successfully solve these tasks after leveraging the trajectory guidance compiled from the natural language instructions. We find that our solver is quite efficient. For a single-stage task, it takes 10 minutes for 300 gradient descent steps on a machine with NVIDIA GeForce RTX 2080, for optimizing a trajectory with 80 steps. For most tasks, 300 gradient steps are sufficient for finding good solutions.

5.3 Driving Multi-Stage Solver using Vision-Language

Having evaluated short-horizon tasks, we apply our method to long-horizon tasks in SoftVL100. We leverage the natural language instructions to generate the evaluation metrics for each task, which might measure the relationships between objects, check if soft materials rupture, or if two shapes are similar. We refer readers to the appendix for more details.

Our multistage vision-language task representation decomposes the long-horizon manipulation process into multiple short-horizon stages, allowing our method to solve them in a stagewise manner. As baseline algorithms fail to make notable progress, we apply ablation to our method: We first compare

our method with a single-stage approach (single), which refers to solving long-horizon tasks like shorthorizon tasks, by dropping the key frames and directly optimizing for the final goal. We then dropped the actuator sampling process in the middle (no actuator sample). The solver is still told to optimize for key frames (sub-goals) at each stage but was not informed to switch actuators and objects to manipulate. As expected, dropping the stage information and the language instructions in our vision task representation significantly degrades the performance, making solvers unable to complete most tasks.

After leveraging the language instructions, the FinalOnly solver removes the vision subgoals in each sub-stage but provides the agent's actuators to choose the objects to manipulate. In this case, similar to the '- Sample' variant, the differentiable physics solver can solve certain tasks by only minimizing shape distance. However, it may have troubles for tasks that require objects to deform into multiple shapes. For example, as illustrated in Figure 6(B), the solver needs to first compress the purple shape into a thin mat to warp the yellow dough.

The solver fails to discover the compression process after removing the subgoals but directly moves the purple toward its final configurations, resulting in the failure in the

end. This showcases the importance of the introduction of the key frames in our vision-language task annotation.

6 Limitations and Conclusion

We introduce DiffVL, a method that enables non-expert users to design specifications of soft-body manipulation tasks through a combination of vision and natural language. By leveraging large language models, task descriptions are translated into machine-interpretable optimization objectives, allowing differentiable physics solvers to efficiently solve complex and long-horizon multi-stage tasks. The availability of user-friendly GUI tools enables a collection of 100 tasks inspired by real-life manipulations. However, it is important to acknowledge some limitations. Firstly, the approach relies on human labor for dataset creation. Additionally, the utilization of large language models incurs high computational costs. Nevertheless, We anticipate being able to gather a larger set of tasks in the future. Furthermore, leveraging the generated trajectories to guide learning methods holds significant potential as a promising avenue for future exploration.

7 Future Works

We believe that crowdsourcing is vital in achieving this goal. We have taken the first step to demonstrate how non-experts can contribute to scaling up robot learning, which may inspire the field to explore ways of involving individuals from various backgrounds in expanding robot manipulation capabilities. It would be out of the capabilities of our group but can be readily adopted by other groups with better resources eventually.

Transferring the policies found by the differentiable simulator to the real world is important. We believe that various components of our framework can offer valuable contributions to real-world scenarios: one such contribution lies in our vision language representation, which naturally represents tasks in the real world; leveraging large language models aids in generating rewards for evaluating real-world results; our GUI tool can effectively manipulate real-world materials, particularly in constrained tasks, serving as a direct interface between humans and robot controllers.

References

Samuel Acquaviva, Yewen Pu, Marta Kryven, Catherine Wong, Gabrielle E Ecanow, Maxwell Nye, Theodoros Sechopoulos, Michael Henry Tessler, and Joshua B Tenenbaum. Communicating natural programs to humans and machines. arXiv preprint arXiv:2106.07824 , 2021.

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pages 104-114. PMLR, 2020.

Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Hanrahan, and Barbara Tversky. Designing effective step-by-step assembly instructions. ACM Transactions on Graphics (TOG) , 22(3):828-837, 2003.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022.

Rika Antonova, Jingyun Yang, Krishna Murthy Jatavallabhula, and Jeannette Bohg. Rethinking optimization with differentiable simulation from a global perspective. arXiv preprint arXiv:2207.00167 , 2022.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:18771901, 2020.

Lawrence Yunliang Chen, Baiyu Shi, Daniel Seita, Richard Cheng, Thomas Kollar, David Held, and Ken Goldberg. Autobag: Learning to open plastic bags and insert objects. arXiv preprint arXiv:2210.17217 , 2022.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 15084-15097. Curran Associates, Inc., 2021.

Cédric Colas, Ahmed Akakzia, Pierre-Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Language-conditioned goal generation: a new approach to language grounding for rl. arXiv preprint arXiv:2006.07043 , 2020.

Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference , pages 893-905. PMLR, 2022.

Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247 , 2023.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378 , 2023.

Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Diffpd: Differentiable projective dynamics. ACM Transactions on Graphics (TOG) , 41(2):1-21, 2021.

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.08853 , 2022.

Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Dynamics learning with cascaded variational inference for multi-step manipulation. arXiv preprint arXiv:1910.13395 , 2019.

C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281 , 2021.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. Lisa: Learning interpretable skill abstractions from language. arXiv preprint arXiv:2203.00054 , 2022.

Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems , 4:265-293, 2021.

Narasimhan Gautham, Zhang Kai, Eisner Ben, Lin Xingyu, and Held David. Transparent liquid segmentation for robotic pouring. In International Conference on Robotics and Automation (ICRA) , 2022.

Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956 , 2019.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Eric Heiden, Miles Macklin, Yashraj S Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Robotics: Science and Systems , July 2021.

Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning to control pdes with differentiable physics. International Conference on Learning Representations , 2020.

Ryan Hoque, Daniel Seita, Ashwin Balakrishna, Aditya Ganapathi, Ajay Tanwani, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. VisuoSpatial Foresight for Physical Sequential Fabric Manipulation. Autonomous Robots (AURO) journal , 2021.

Taylor A Howell, Simon Le Cleac'h, J Zico Kolter, Mac Schwager, and Zachary Manchester. Dojo: A differentiable simulator for robotics. arXiv preprint arXiv:2203.00806 , 2022.

Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:1910.00935 , 2019.

Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG) , 37(4):1-14, 2018.

Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In International Conference on Learning Representations , 2020.

Zixuan Huang, Xingyu Lin, and David Held. Mesh-based dynamics model with occlusion reasoning for cloth manipulation. Robotics: Science and Systems (RSS) , 2022.

Zixuan Huang, Xingyu Lin, and David Held. Mesh-based dynamics with occlusion reasoning for cloth manipulation, 2022.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:12731286, 2021.

Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials. In ACM SIGGRAPH 2016 Courses , pages 1-52. 2016.

Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxiv-2210.03094 , 2022.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 12 2014.

Steven M LaValle and James J Kuffner. Rapidly-exploring random trees: Progress and prospects: Steven m. lavalle, iowa state university, a james j. kuffner, jr., university of tokyo, tokyo, japan. Algorithmic and computational robotics , pages 303-307, 2001.

Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto MartínMartín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning , pages 80-93. PMLR, 2023.

Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems , 35:31199-31212, 2022.

Sizhe Li, Zhiao Huang, Tao Chen, Tao Du, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physics. arXiv preprint arXiv:2304.03223 , 2023.

Sizhe Li, Zhiao Huang, Tao Du, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Contact points discovery for soft-body manipulations with differentiable physics. arXiv preprint arXiv:2205.02835 , 2022.

Yifei Li, Tao Du, Kui Wu, Jie Xu, and Wojciech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Transactions on Graphics (TOG) , 42(1):1-20, 2022.

Yulong Li, Shubham Agrawal, Jen-Shuo Liu, Steven K Feiner, and Shuran Song. Scene editing as teleoperation: A case study in 6dof kit assembly. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4773-4780. IEEE, 2022.

Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning , pages 112123. PMLR, 2022.

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 , 2022.

Junbang Liang, Ming C Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. Advances in Neural Information Processing Systems , 2019.

Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153 , 2023.

Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B Tenenbaum, David Held, and Chuang Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. arXiv preprint arXiv:2203.17275 , 2022.

Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning , 2021.

Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning , 2021.

Fei Liu, Entong Su, Jingpei Lu, Mingen Li, and Michael C Yip. Robotic manipulation of deformable rope-like objects using differentiable compliant position-based dynamics. IEEE Robotics and Automation Letters , 2023.

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning , pages 1113-1132. PMLR, 2020.

Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. Fluid directed rigid body control using deep reinforcement learning. ACM Trans. Graph. , 37(4), July 2018.

Miles Macklin. Warp: A high-performance python framework for gpu simulation and graphics. https://github.com/nvidia/warp , March 2022. NVIDIA GPU Technology Conference (GTC).

Jeremy Maitin-Shepard, Marco Cusumano-Towner, Jinna Lei, and Pieter Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation , pages 2308-2315. IEEE, 2010.

Jiayuan Mao, Tomás Lozano-Pérez, Josh Tenenbaum, and Leslie Kaelbling. Pdsketch: Integrated domain programming, learning, and planning. Advances in Neural Information Processing Systems , 35:36972-36984, 2022.

William P McCarthy, Robert D Hawkins, Haoliang Wang, Cameron Holdaway, and Judith E Fan. Learning to communicate about shared procedural abstractions. arXiv preprint arXiv:2107.00077 , 2021.

Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters , 7(4):11205-11212, 2022.

Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters , 2022.

Peter Mitrano, Dale McConachie, and Dmitry Berenson. Learning where to trust unreliable models in an unstructured world for deformable object manipulation. Science Robotics , 6(54):eabd8170, 2021.

Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning , pages 7805-7817. PMLR, 2021.

Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff. Position based dynamics. Journal of Visual Communication and Image Representation , 18(2):109-118, 2007.

Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning languageconditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning , pages 1303-1315. PMLR, 2022.

OpenAI. Gpt-4 technical report, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Yiling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft multi-body systems. Advances in Neural Information Processing Systems , 34:17123-17135, 2021.

Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters , 7(4):10873-10881, 2022.

Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision , 40(2):99, 2000.

Md. Sadman Sakib, David Paulius, and Yu Sun. Approximate task tree retrieval in a knowledge network for robotic cooking. IEEE Robotics and Automation Letters , 7(4):11492-11499, 2022.

Gautam Salhotra, I-Chun Arthur Liu, Marcus Dominguez-Kuhne, and Gaurav S Sukhatme. Learning deformable object manipulation from expert demonstrations. IEEE Robotics and Automation Letters , 7(4):8775-8782, 2022.

Connor Schenck and Dieter Fox. Visual closed-loop control for pouring liquids. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 2629-2636. IEEE, 2017.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05186 , 2022.

Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools, 2023.

Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning to see, simulate, and shape elasto-plastic objects with graph networks. Robotics: Science and Systems (RSS) , 2022.

Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560 , 2022.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022.

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302 , 2022.

Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning , pages 20668-20696. PMLR, 2022.

Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph E Gonzalez, and Ken Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 9411-9418. IEEE, 2020.

Allison Tam, Neil Rabinowitz, Andrew Lampinen, Nicholas A Roy, Stephanie Chan, DJ Strouse, Jane Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. Advances in Neural Information Processing Systems , 35:25377-25389, 2022.

Marc Toussaint. Logic-geometric programming: An optimization-based approach to combined task and motion planning. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015.

Marc A Toussaint, Kelsey Rebecca Allen, Kevin A Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. Robotics: Science and Systems Foundation , 2018.

Vainavi Viswanath, Kaushik Shivakumar, Jainil Ajmera, Mallika Parulekar, Justin Kerr, Jeffrey Ichnowski, Richard Cheng, Thomas Kollar, and Ken Goldberg. Learning to trace and untangle semi-planar knots (tusk), 2023.

Vainavi Viswanath, Kaushik Shivakumar, Justin Kerr, Brijen Thananjeyan, Ellen Novoseller, Jeffrey Ichnowski, Alejandro Escontrela, Michael Laskey, Joseph E Gonzalez, and Ken Goldberg. Autonomously untangling long cables. arXiv preprint arXiv:2207.07813 , 2022.

Ruocheng Wang, Yunzhi Zhang, Jiayuan Mao, Chin-Yi Cheng, and Jiajun Wu. Translating a visual lego manual to a machine-executable plan. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII , pages 677-694. Springer, 2022.

Yufei Wang, Zhanyi Sun, Zackory Erickson, and David Held. One policy to dress them all: Learning to dress people with diverse poses and garments, 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.

Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, and David Held. Fabricflownet: Bimanual cloth manipulation with a flow-based policy. In Conference on Robot Learning , 2021.

Keenon Werling, Dalton Omens, Jeongseok Lee, Ioannis Exarchos, and C Karen Liu. Fast and feature-complete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103.16021 , 2021.

Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658 , 2023.

Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations, 2020.

Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao-Yu Tung, Antonio Torralba, Katerina Fragkiadaki, and Chuang Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. arXiv preprint arXiv:2303.02346 , 2023.

Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint arXiv:2204.07137 , 2022.

Zhenjia Xu, Zhou Xian, Xingyu Lin, Cheng Chi, Zhiao Huang, Chuang Gan, and Shuran Song. Roboninja: Learning an adaptive cutting policy for multi-material objects. arXiv preprint arXiv:2302.11553 , 2023.

Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg. Self-supervised learning of state estimation for manipulating deformable linear objects. IEEE Robotics and Automation Letters , 5(2):2372-2379, 2020.

Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 , 2023.

Supplementary files

We provide a video illustrating our GUI tools, example task, and solutions. We also provide a tutorial for helping users to use our annotation system as a supplementary file.

A How to choose keyframes

The selection of keyframes is determined by the annotators, as we believe humans can naturally decompose complex tasks into simpler ones for communication. When annotators manipulate a scene within the GUI, they can save the current scene as a key frame within the task representation. They can also add, modify, or delete key frames using the editors within the web server interface.

The annotators were instructed on what kind of decomposition would likely result in a working trajectory, i.e., segmenting the manipulation processes at contact point changes., which is a simple and effective way for the annotator to provide high-quality labels, without having any expert understanding of the physics and the solver. If the agent is required to manipulate a new object or the actuator needs to establish contact on a different face of the object, annotators would introduce a new keyframe. Additionally, we employ heuristic methods to segment YouTube videos. This segmentation occurs when there is a significant disparity between two frames, aiming to simplify the keyframe selection for annotators.

B DSL for Optimization Programs

Table 2 lists the functions and primitives in our domain-specific language.

We rely on the large language model's few-shot ability to compile the natural language description like 'lift up the white cube above the black wall' into an executable optimization program. The prompt start with the introduction to the DSL, listing the functions as well as the explanations, followed by several examples illustrating how to compose those functions to implement complex functions, e.g., by checking the x coordinates of two objects to decide if one is on the left of the other. Then we provide several pairs of natural language inputs and corresponding programs. In the end, we provide the natural language input to compile. Below is an example prompt.

B.1 Comparing GPT3.5 and GPT4

We notice that GPT4 has a better performance compared with GPT3.5. Below we provide the results of GPT3.5 and GPT4 of example tasks shown in the main text. Generally, we have found that it is common for GPT3.5 to misinterpret the instructions, add/ignore constraints, and write incorrect commands with incorrect grammar. In contrast, GPT4 can generate pretty good results.

Although GPT4 already outperforms GPT3.5 in the translation task, we observed several failure cases in the translation process. For example, the output of GPT4 will contain objects that do not exist. In a task that lifts a sphere above a box, the output may contain an invalid sentence like require(similar('sphere', goal('sphere above box'))) where 'sphere above box' which doesn't correspond to a proper object. It may also generate a statement with a type error. For example, sometimes it may output require(similar('sphere', y(max(pcd('box'))))), which would trigger a compilation error as we can not compute the shape distance between an object and a coordinate. Fortunately, those compilation errors could be detected during the compilation and can be resolved by resampling. However, there are also rare cases where LLM removes or adds more constraints to the instructions. For example, it may ask the solver to fix the position of an object that needs to move. In these cases, our solver may fail to find a suitable solution. Restricting the output to fit a particular syntax, for instance, using context-free grammars would resolve such problems.

Test Case 1 Grasp the front of the rope and then grasp the back of the rope. Deform the rope into its target shape.

Test case 2 Use the gripper to first lift up the white cube above the black wall and then move it to the target location.

GPT4

Test case 3 Put the knife above the white object. Cut the knife down to match the target shape of the white object. Ensure that the knife does not rotate.

GPT3.5: It uses z instead y to extract the objects ´ height while the latter is true. It also misinterprets the meaning to deform the knife into the goal.

GPT4

Test case 4 Pick up the front part of object 'top left mat' and place it into the goal without breaking it.

GPT3.5: It adds an incorrect command to let the gripper to move away from the object in the beginning.

C Reinforcement learning baseline

We use the SAC and PPO implementation from stable-baselines3 with hyperparameters in Table 3. Figure 8 compares the optimization performance of RL algorithms and the Adam optimizer with respect to the IOU score of the ending state with different numbers of sampled episodes for the 20 single-stage tasks of five categories.

SAC

Adam

PPO

0.65

(B) Move-vo

0.30

(C) wina-vo

0.70

(D) Fold-v0

0.45

(E) Cut-v0

0.5

0.60

0.25

0.65

0.40

0.4

0.55

0.20

0.60

SE0

20.3

0.50

0E0

0.2

0.45

0.40

0.10

0.15

0.50

0.55

0.25

0.20

0.1

0

50

100150200250300

50100150200250300

0.05

0

50100150200250300

0.45

0

50

100150200250300

0

50

100150200250300

(F) Deform-v1

(G) Move-v1

(H) Wind-v1

(1) Fold-v1

() Cut-v1

IOU

0.40

0.35

20.45

0.60

0.55

0.50

20.20

0.30

0.25

0.35

0.50

0.60

0.55

0.35

0.45

0.40

0.30

0.40

0.35

0.15

0.45

0.25

0.25

0.30

0.10

0.40

0.20

0

100150200250300

Episodes

50100150200250300

50100150200250300

50100150200250300

50

100150200250300

Episodes

0.5

(K) Deform-v2

(L) Move-v2

(M) Wind-v2

(N) Fold-v2

(O)Cut-v2

0.7

0.40

0.6

0.44

0.4

0.6

0.35

0.5

0.42

0.3

0.30

20.40

0.2

0.1

0.4

0.25

0.20

0.2

0.1

0.3

0.38

0.36

0.34

50

100150200250300

100150200250300

100150200250300

Episodes

50

100150200250300

0

100150200250300

0.60

0.58

(P) Deform-v3

0.56

0.58

(Q)Move-v3

0.45

(R) Wind-v3

0.5

0.6

(S) Fold-v3

0.45

0.50

(T) Cut-v3

no

0.56

0.52

0.54

0.40

20.3

0.4

0.35

0.40

0.52

0.54

0.48

0.50

0.35

0.2

0.30

0.25

0.50

0.46

0.30

0.1

0.20

50

100150200250300

Episodes

0

50

100150200

Episodes

250300

50

100150200250300

50

100150200250300

0

50

100150200250300

D Evaluation metrics

We notice that in some tasks, a simple Intersection Over Union (IOU) score comparing shapes does not effectively explain the human intuition of task completion. For example, a solver might 'cut' two

objects similar to the target shape without truly separating them, as the space between the two parts might only contain a minimal amount of space. In other circumstances, the solver may violate the requirements of no_break and break a rope into two or destroy a mat. To this end, we add checkers to check if two shapes are split and if a shape is not broken for success evaluation. For no-breaking constraint, we track particles and ensure their distance to their initial nearest neighbor does not change too much. On the other hand, we can check if two shapes are separated from each other by finding the connected components of particles.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper works on trajectory generation for soft body manipulation with differentiable simulation. To address the key challenge of representing task goals for optimization, the authors propose to use natural language descriptions to lower the barrier for annotation, where a framework that utilizes LLM for translating natural language into optimization programs is developed to facilitate data collection.\n\nSTRENGTHS:\nThe idea of task specification with natural language is an interesting idea; \n\nThe overall framework seems to be reasonable and technical sound, it is smart to leverage LLM for converting natural language descriptions into optimization programs; \n\nThe paper is well-presenting and easy to read.\n\nWEAKNESSES:\nMore evaluations can be performed to validate the robustness of the trajectory generation; \n\nThere is no discussion of the failure cases; \n\nSome missing citations in line 21; \n\nTypo in figure 4: smapling -> sampling; \n\nSome works that leverage natural language for task goal specifications, which may need to be discussed in the related work:  \n\n[1] StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects, ICRA 2022; \n\n[2] Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement, ICRA 2023\n\nQUESTIONS:\nIn section 4.3, the authors say that a sampling-based RRT planner is used to determine the path for the actuator, and then the trajectory is refined with a gradient-based optimizer. Could the authors provide more details on how the trajectory is initially calculated and how it is refined for execution? Besides, it would be better to provide more snapshots for the rollouts to show the difference between the initial trajectory and the final ones; \n\nWould the trajectory optimization be sensitive to the initialization? It would be interesting to conduct some experiments for demonstrating the robustness of the initialization strategy; \n\nAre there any failure cases that are caused by the LLM translation? It would be good to understand the failure mode in which situations the LLM failed to generate appropriate optimization programs.\n\nLIMITATIONS:\nLimitations are discussed in the paper, but no failure cases are presented and analyzed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study focuses on soft-body manipulation problems such as flattening dough using a rolling pin, cutting deformable objects, and more. It introduces a method that engages non-expert users to provide detailed annotations and identify sub-goal states within key frames of the task video. Despite requiring additional human intervention, this approach generates a rich learning signal for manipulation tasks involving deformable objects. This significantly eases the learning difficulty associated with such tasks. The proposed method has been evaluated based on six fundamental deformable object manipulation skills, demonstrating its effectiveness.\n\nSTRENGTHS:\nThis work proposes a unique representation of task specifications. It incorporates human-labeled annotations for deformable object tasks to shape an objective/reward function. Such a reward function provides far richer reward signals than just a goal state, potentially greatly reducing policy learning difficulties.\n\nFurthermore, this study offers a new dataset that includes 100 deformable object manipulation tasks, complemented by human annotations on key frames. This contribution could be beneficial to the broader community and greatly aid studies related to deformable object manipulation.\n\nThe presentation of this research is both clear and organized.\n\nWEAKNESSES:\nHuman annotation offers a valuable reward signal, which could potentially simplify the learning process. However, it also imposes certain constraints on the policy strategy. When states deviate from the demonstrated trajectory, adhering strictly to specific action commands could exacerbate the situation. Hence, at its current stage, it lacks the flexibility needed to handle unseen states not accounted for in the instructions/annotations.\n\nMoreover, each task necessitates considerable human involvement in creating subgoal states and corresponding annotations. Although software solutions have been developed to assist in this process, it still represents a significant involvement of human resources and incurs considerable costs.\n\nQUESTIONS:\nQ1: How is a key frame chosen? Also, can you explain how the decomposition of long-horizon tasks occurs?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to soft body manipulation employing the strengths of the Large Language Model (LLM). The key innovation is viewing tasks as data, with each data point consisting of an initial scene and an optimization objective. To tackle the challenge of task representation, the authors introduce DiffVL, a framework enabling non-expert users to define soft-body manipulation tasks to a differentiable solver using a combination of vision and language. The users specify tasks via an interactive simulator with a sequence of 3D scenes (keyframes) connected by natural language instructions. The authors also developed a corresponding GUI and curated SoftVL100, a vision-language dataset with 100 diverse tasks. They further developed a method that combines the power of a large-language model and differentiable physics to solve a wide variety of challenging long-horizon tasks in SoftVL100. This study's contributions lie in its new task representation, the developed GUI, the curated dataset, and the DiffVL method.\n\nSTRENGTHS:\nThe paper under discussion is a robust, well-rounded work, bringing to the table significant contributions across multiple facets. It excels in terms of novel methodology, tool development and proposes a new dataset:\n\n• The authors introduce an innovative multi-stage vision-language representation. This new approach simplifies the definition of soft-body manipulation tasks, making it accessible for non-expert user annotations - a noteworthy contribution to research in this area.\n\n• In terms of tool development, the authors have crafted a corresponding Graphical User Interface (GUI), enhancing the user experience and overall accessibility of their proposed approach. They have also curated SoftVL100, a compilation of 100 realistic soft-body manipulation tasks derived from online videos. This dataset stands as a valuable resource for further research and application in this domain.\n\n• Moreover, the authors have devised DiffVL, a method that bridges the gap between a large-language model and differentiable physics. This combination of strengths is adeptly applied to tackle a wide variety of challenging long-horizon tasks presented in SoftVL100, which marks a significant advancement in the field.\n\nThe experimental section of the paper is another key strength, featuring a thorough ablation study.\n\nWEAKNESSES:\nThe relative weakness of the approach can be considered the amount of human labor required for dataset collection. Also, DSL could sometimes be too constrained to define arbitrary new tasks. Despite these points, they in no way diminish the overall quality of the work presented in the paper. The minor limitations identified simply highlight areas for potential future refinement. The project stands as an excellent example of research and tool development, and I am eager to see how it evolves.\n\nQUESTIONS:\n• It would be good to have more info about time constraints - simulation and training time for different tasks, not only training progress per epoch.\n• IWere there any simulation failure cases? How challenging was simulator tuning for different tasks?\n\nLIMITATIONS:\nLimitations were addressed fairly well.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates curating a set of 100 soft-body manipulation tasks and provides expert policies for them by using a mix of: annotators that provide supervision in the form of keyframes and/or natural langauge annotation, translating the annotations into programs via an LLM, and solving an optimization.\n\nSTRENGTHS:\nOne strength of this paper is that obtaining manipulation policies for soft bodies is undoubtedly hard, and so anything that can do is has some potential value to the community.\n\nAdditionally, the dataset if released by the authors could be valuable.  The tasks and expert policies could be used as data to study methods that require some amount of expert data (imitation / offline RL / etc), and it's hard to both get these types of policies, as well as build simulation environments, in order to study these well. Good datatsets and tasks can be considered a bottleneck of manipulation research, especially with soft body manipulation, and the dataset could be valuable here.\n\nWEAKNESSES:\nThe first sentence of this paper has unfinished citations. It reads, verbatim: \"This paper focuses on soft body manipulation, a research topic with a wide set of applications such as folding cloth[cite], untangling cables [cite], and cooking foods[cite].\"  This is a pity because it makes it hard to take the work too seriously given an incomplete first sentence.  Meanwhile, the rest of the paper seems relatively polished, so I'm doing my best to evaluate it seriously.\n\nOne limitation that should be kept in mind is that in multiple ways I only see the methods of this paper as being relevant to simulation, and not directly to the real world.  This applies both to the 3D GUI annotation process, as well as the process for converting into something that the differentiable physics solver can solve.  The work could help enable further studies in simulation that could eventually impact the real-world, indirectly either through helping provide evaluation tools or perhaps eventually sim-to-real, but not directly.\n\nOverall, I think the main potential interesting thing is that the authors have actually provided expert policies for 100 or so soft-body manipulation tasks. This is hard to achieve.  I personally find that the existence of the policies and tasks is the main valuable and interesting thing, and less so the methods used to obtain them. \n\nAdditionally, 100 tasks is not that large for a crowdsourced effort.  This is an important point in my opinion because it's likely the process might have just gone faster if the first author designed all the tasks themselves rather than teaching others to do so.  One might need to get to 1,000 or 10,000 tasks in order to really start to see the benefits of a crowdsourced effort.  Accordingly this in my opinion undermines the care taken in the paper to make the annotation process usable by non-experts.\n\nAnother lens to look at this paper is: what actual results are shown?  There are two results tables/figures: Table 1, and Figure 7.  Table 1 shows that it is hard for SAC or PPO to solve the tasks, demonstrating that they are indeed not easy.  It also shows that their method, which has access to considerably more info than SAC/PPO due to its use of the annotations, can solve the tasks.  This is not a fully fair comparison, and that's fine.  The second results table shows that the multi-stage annotations and optimizations are useful.  This also makes sense.  It is hard to pinpoint what specific contribution claim this results figure supports though, since I don't think there is clear novelty in the multi-step framework presented by the authors.\n\nQUESTIONS:\nDo each of the points discussed in the weaknesses make sense?\n\nLIMITATIONS:\nThe authors do discuss limitations in the conclusion. They should add that the methods are limited to simulation and impact on the real-world is left as a future question to address.\nThey don't discuss societal impacts but I think that's okay here.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes DiffVL, a novel framework that tackles soft-body manipulation, which consists of a GUI for users to specify tasks easily and a large language model (LLM) for translating text instructions to programs for policy learning and execution.

STRENGTHS:
- An intuitive user interface for task specification
- A novel combination of LLM and a differentiable simulator for program generation and optimization
- Good empirical success rate
- Clear writing

WEAKNESSES:
- Heavy user input needed on the magnitude of hours to specify task
- Comparison in Table 1 is a bit unfair since other pure RL baselines do not leverage language guidance
- Writing is not polished (e.g., missing citations in line 21)

QUESTIONS:
How does this approach differ from other recent robot learning papers utilizing LLM for instruction generation/high-level planning? The novelty in this aspect is obscure to me.

LIMITATIONS:
The authors discussed limitations adequately in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-08