SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiao Huang*
z2huang@ucsd.edu
Feng Chen*
Institute for Interdisciplinary Information Sciences
chenf20@mails.tsinghua.edu.cn
Yewen Pu
yewen.pu@autodesk.com
Chunru Lin
UMass Amherst
chunrulin@umass.edu
Hao Su†
haosu@ucsd.edu
Chuang Gan†
MIT-IBM Watson AI Lab, UMass Amherst
chuangg@umass.edu
Combining gradient-based trajectory optimization with differentiable physics simulation is an accurate and efficient technique for solving soft-body manipulation problems. Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory. However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users. We introduce DiffVL, a framework that integrates the process from task collection to trajectory generation leveraging a combination of visual and linguistic task descriptions. A DiffVL task represents a long horizon soft-body manipulation problem as a sequence of 3D scenes (key frames) and natural language instructions connecting adjacent key frames. We built GUI tools and tasked non-expert users to transcribe 100 soft-body manipulation tasks inspired by real-life scenarios from online videos. We also developed a novel method that leverages large language models to translate task language descriptions into machine-interpretable optimization objectives, which can then help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines. Experiments show that existing baselines cannot complete complex tasks, while our method can solve them well. Videos can be found on the website https://sites.google.com/view/diffvl/home.
This paper focuses on soft body manipulation, a research topic with a wide set of applications such as folding cloth [52, 90, 34], untangling cables [88, 87], and cooking foods [77, 72]. Due to their complicated physics and high degree of freedom, soft body manipulations raise unique opportunities and challenges. Recent works such as [32] and [99] have heavily leveraged various differentiable physics simulators to make these tasks tractable. Towards generalizable manipulation skill learning, such approaches have the potential to generate data for learning from demonstration
$\dagger$  Equal contribution
algorithms [51, 43]. However, to enable differentiable physics solvers to generate a large scale of meaningful trajectories, we must provide them with suitable tasks containing the scene and the optimization objectives to guide the solver. Previous tasks are mostly hand-designed [32] or procedure-generated [51], resulting in a lack of diverse and realistic soft-body manipulation tasks for researchers.
This paper takes another perspective by viewing tasks as data, or more precisely, task specifications as data. Each data point contains a pair of an initial scene and an optimization objective depicting the goal of the task. Taking this perspective, we can annotate meaningful tasks like annotating data of other modalities, like texts, images, videos, or each action trajectory, by leveraging non-expert human labor, providing possibilities for scaling up the task space of differentiable physics solvers.
A core problem in building a task collection framework with trajectory generation is finding a suitable representation of the tasks. The representation should be intuitive for non-expert annotators while being accurate enough to describe the complex shapes and motions of the soft bodies that appear in various soft body manipulation processes. Last but not least, it should be reliably interpreted by differentiable physics solvers to yield a valid trajectory.
Figure 1: (A) A dumpling making video; (B) The annotator interacts with our GUI tool to create DiffVL tasks; (C) A DiffVL task contains a sequence of 3D scenes along with natural language instructions to guide the solver; (D) DiffVL leverages a large language model to compile instructions into optimization programs consisting of vision elements; (E) The optimization program guides the solver to solve the task in the end.
Studies such as [3, 1, 60, 89] have shown that humans are proficient at defining goal-driven, spatial object manipulation tasks using either sketches, natural languages, or both. Figure 1(A), for instance, depicts the complete process of dumpling creation, supplemented by textual instructions. Taking this as an inspiration, we present DiffVL, a framework that enables non-expert users to specify soft-body manipulation tasks to a differentiable solver using a combination of vision and language. Specifically, each DiffVL task consists of a sequence of 3D scenes (keyframes), with natural language instruction connecting adjacent keyframes. The sequence of key frames specifies the sequence of subgoals of the manipulation task, and the natural language instructions provide suggestions on how to use the actuators to manipulate the objects through this sequence of subgoals. See Figure 1.
We develop tools to ease the annotation for non-expert users. With our interactive simulator equipped with a GUI, the user can edit, manipulate, draw, or carve shapes easily like other 3D editing tools and observe the consequence through simulation in an intuitive manner. Meanwhile, when it is tedious for users to edit all the intermediate steps of a complex motion, they can use natural language to describe the goal instead of drawing them step by step. This enables us to build SoftVL100, a vision-language dataset with 100 diverse tasks. DiffVL uses LLM to compile the natural language instructions to an optimization program – which enforces a set of constraints during the soft-body manipulations from one keyframe to the next. This optimization program is then used by a differentiable physics solver to generate a working trajectory.
To summarize, our work makes the following contributions:
- We propose a new multi-stage vision-language representation for defining soft-body manipulation tasks that is suitable for non-expert user annotations.
- We developed a corresponding GUI, and curated SoftVL100, consisting of 100 realistic soft-body manipulation tasks from online videos<sup>3</sup>.
- We develop a method, DiffVL, which marries the power of a large-language model and differentiable physics to solve a large variety of challenging long-horizon tasks in SoftVL100.
Differentiable simulation for soft bodies Differentiable physics [30, 93, 69, 19, 29] has brought unique opportunities in manipulating deformable objects of various types [58, 28, 53, 33, 92, 49, 95, 83, 63, 100, 95, 56, 27, 47, 74, 23], including plasticine [32], cloth [45], ropes [54] and fluids [96] simulated with mass-spring models [30], Position-based Dynamics [65], Projective Dynamics [69, 16], MPM [36] or Finite Element Method [26], which could be differentiable through various techniques [30, 16, 57, 7]. It has been shown that soft bodies have smoother gradients compared to rigid bodies [32, 30, 93, 6, 82], enabling better optimization. However, the solver may still suffer from non-convexity [44] and discontinuities [82], motivating approaches to combine either stochastic sampling [44], demonstrations [43, 6] or reinforcement learning [98, 64] to overcome the aforementioned issues. DiffVL capitalizes on an off-the-shelf differentiable physics solver to annotate tasks. It incorporates human priors through a novel vision-language task representation. This allows users to employ natural language and GUI to guide the differentiable solver to solve long-horizon tasks, distinguishing DiffVL from previous methods.
Task representation for soft body manipulation There are various ways of defining tasks for soft body manipulation. To capture the complex shape variations of soft bodies, it is natural to use either RGB images [95, 51, 73] or 3D point cloud [78, 47] to represent goals. However, given only the final goal images, it may be hard for the solver to find a good solution without further guidance. Besides, generating diverse images or point clouds that adhere to physical constraints like gravity and collision avoidance in varied scenes can be challenging without appropriate annotation tools. Our GUI tools are designed to address these issues, simplifying the process. Other works use hand-defined features [10, 88] or formalized languages [41]. These methods necessitate specialist knowledge and require tasks to be defined on a case-by-case basis. Learning from demonstrations [43, 73] avoids defining the task explicitly but requires agents to follow demonstration trajectories and learn to condition on various goal representations through supervised learning or offline reinforcement learning [20, 18, 79, 24, 55, 35, 11, 2, 5]. However, collecting demonstrations step-by-step [70, 101, 43] requires non-trivial human efforts, is limited to robots with human-like morphologies, and may be challenging for collecting tasks that are non-trivial for humans (e.g., involving complex dynamics or tool manipulation). Our annotation tool emphasizes the description of tasks over the identification of solutions. Annotators are merely required to define the task, not execute the trajectories themselves, thereby simplifying the annotation process. Our method is also related to [46], but we consider a broader type of tasks and manipulation skills.
Language-driven robot learning Many works treat language as subgoals and learn language-conditioned policies from data [80, 21, 66, 42, 8, 62, 61] or reinforcement learning [37, 17, 84, 12]. It has been shown that conditioning policies on languages lead to improved generalization and transferability [80, 37]. [76] turns languages into constraints to correct trajectories. Our method distinguishes itself by integrating language with differentiable physics through machine-interpretable optimization programs. Recent works aim at leveraging large language models [4, 15, 94, 50, 13] to facilitate robot learning. Most use language or code [48] to represent policies rather than goals and focus on high-level planning while relying on pre-defined primitive actions or pre-trained policies for low-level control. In contrast, our language focuses on low-level physics by compiling language instructions to optimization objectives rather than immediate commands. It's noteworthy that VIMA [38] also introduces multimodal prompts, embodying a similar ethos to our vision-language task representation. Their approach views multimodal representation as the task prompts to guide the policy. However, it still requires pre-programmed oracles to generate offline datasets for learning, which is hard to generalize to soft-body tasks with complex dynamics. Contrastingly, we
both the GUI and dataset will be made public
adopt a data annotation perspective, providing a comprehensive framework for task collection, natural language compiling, and trajectory optimization, enabling us to harness the potential of differentiable physics effectively.
Task and motion planning Our optimization program compiled from natural language is closely connected to the field of Task and Motion Planning (TAMP) [22], particularly Logic Geometric Programming (LGP) [85, 86]. We draw inspiration from LGP's multiple logical states in designing our DiffVL task representation. However, unlike LGP, our approach asks human annotators to assign "logic states" and facilitates the manipulation of soft bodies through our novel vision-language representation. Some recent works have explored the intersection of TAMP and language models [14, 81, 59] while primarily focusing on generating high-level plans.
3 SoftVL100: Vision-Language Driven Soft-body Manipulation Dataset
Put the rolling pin above the white and manipulate the white into goal.
Then grasp the yellow by gripper and place them into goal places.
Finally grasp the white and wrap it as well as the yellow in it.
Grasp the back end of rope. Then Wind the rope and place the salmon ball.
Grasp the front end of rope. Then Wind the rope and place the black ball.
Grasp the front end of top yellow by gripper and place it on the bottom.
Then grasp the ropes by gripper to wind and place them into theirs goal shape.
Finally, grasp the red by gripper to place it on the top.
(A)Multistage Tasks
(B) Scene examples
Figure 2: (A) Example of multi-stage tasks and their text annotations; (B) Snapshots of scenes in SoftVL100 dataset.
In this representation, the original tasks are divided into a sequence of 3D scenes, or key frames, along with text descriptions that detail the steps for progressing to the next scene. Figure 2(A) illustrates our representation. Each key frame consists of multiple soft bodies with different positions and shapes. By grouping two consecutive key frames together, we form a manipulation stage. Natural language instructions may guide the selection of an actuator and provide guidance on how to control it to reach the subsequent key frames. In Section 3.1, we develop annotation tools to facilitate the task representation annotation process. These tools make it easy to construct a diverse set of tasks, referred to as SoftVL100, as discussed in Section 3.2.
3.1 Vision-Language Task Annotator
Our annotation tool is based on PlasticineLab [32], a simulation platform that utilizes the Material Point Method (MPM) [31] for elastoplastic material simulation. To enhance the user experience and support scene creation, we integrate it into SAPIEN [97], a framework that enables the development of customized graphical user interfaces.
Our simulation runs on GPU servers and the interface is accessible as a web service through VNC, allowing users to interact with the simulator directly from their web browser. When
Figure 3: Example operations in GUI tools
starting a new task annotation, users can open the simulator, perform actions, and save the current scene. Our GUI tools provide comprehensive support for creating, editing, managing, and simulating objects, as illustrated in Figure 3. Scene creation: The GUI offers a variety of primitive shapes, such as ropes, spheres, cubes, and cylinders, that users can add to the scene. Before adding a soft-body shape, users can adjust its size, color, and materials (e.g., rubber, fiber, dough, iron) if needed. Shape editing: Users have the ability to select, edit, or delete shapes within the scene. The GUI tool includes a range of shape operators, including moving, rotating, carving, and drawing. We provide a complete list of supported operations in the appendix. Simulation: Since the interface is built on a soft body simulator, users can simulate and interact with the soft body using rigid actuators or magic forces. They can also run simulations to check for stability and collision-free states. Object management: Users can merge connected shapes to create a single object or divide an object into two separate ones with distinct names. The annotation tool keeps tracking the identities of objects across scenes.
Once users have finished editing the scene and creating a new key frame, they can save the scene into the key frame sequence. The key frames are displayed below the simulator, visualizing the current task annotation progress. Users have the option to open any of these saved scenes in the simulator, delete a specific scene if needed, and add text annotations for each scene. Please refer to the appendix for detailed information on the functionalities and usage of the GUI tool.
3.2 The SoftVL100 Dataset
Our annotation tool simplifies the task creation process, enabling us to hire non-expert users to collect new tasks to form a new task set, SoftVL100. The task collection involves several stages, starting with crawling relevant videos from websites like YouTube that showcase soft body manipulation, particularly clay-making and dough manipulation. We developed tools to segment these videos and extract key frames to aid in task creation.
We hired students to annotate tasks, providing them with a comprehensive tutorial and step-by-step guidelines for using our annotation tools. It took approximately two hours for each annotator to become proficient in using the annotation tool. Subsequently, they were assigned a set of real-world videos to create similar tasks within the simulator. After collecting the keyframes, we proceed to relabel the scenes with text descriptions. Annotators are provided with a set of example text descriptions. We encourage users to include detailed descriptions of the actuators, their locations, moving directions, and any specific requirements, such as avoiding shape breakage for fragile materials. The annotation process for each task typically takes around 30 minutes. Using our task annotation tool, we have created a dataset called SoftVL100, which consists of 100 tasks, and there are more than 4 stages on average. The tasks cover a wide range of skills as illustrated in Figure 2 (A). Sample tasks from the dataset can be seen in Figure 2 (B).
4 Optimization with Vision-Language Task Description
We propose DiffVL to tackle the challenging tasks in SoftVL100. Given a stage of a task depicted in Figure 4(A), DiffVL utilizes large language models to compile the natural language instructions into a machine-interpretable optimization program in Figure 4(B). The program comprises of the names of visible elements and Python functions, effectively capturing the essence of the language instructions. We introduce the design of our DSL in Section 4.1 and outline the DiffVL compiler based on large language models (LLM) in Section 4.2. The resulting optimization program includes crucial information for selecting and locating actuators and can facilitate a differentiable physics solver to generate valid trajectories, as discussed in Section 4.3.
4.1 Optimization Program
The optimization program is formulated using a specialized domain-specific language (DSL) that incorporates several notable features. Firstly, it includes functions that extract the names of visible elements from the 3D scenes and support operations on these elements, represented as 3D point clouds. This enables manipulation and analysis of the visual information within the optimization program. Secondly, the DSL provides various functions to express geometric and temporal relations, encompassing common geometric and motion constraints. These functions facilitate the representation and handling of natural language instructions. Furthermore, the DSL clauses are interpreted
Figure 4: (A) One stage of the DiffVL task of two keyframes and a natural language instruction. (B) The compiled optimization program. (C) The sampling and optimization process.
into PyTorch [68], making the constraints automatically differentiable and directly applicable for differentiable physics solvers.
The example in Figure 4 already exposes multiple components of our DSL. In the program, we can refer to the names of visible elements through get and goal functions, which can look for and extract the 3D objects that satisfy the description, identified by their color, shapes, and locations (we ensured that these attributes are sufficient to identify objects in our tasks). The program will then start with a sample statement, which is used to specify the actuators and their associated constraints over initial positions. Actuators refer to the entities utilized for manipulation, such as a robot gripper, a knife, or the rolling pin depicted in Figure 4. The constraint  $y(\text{RollingPin}) > y(\max(\text{pcd}(\text{Dough}))$  within the sample statement represents a specific constraint. It ensures that the height of the actuator (RollingPin) is greater than the highest  $y$  coordinate of the white dough in the scene, reflecting the meaning of "put bove" of the language instructions. Following the sample statement, an optimize statement is included. This statement plays a vital role in defining the constraints and objectives for the manipulation trajectories. These constraints and objectives serve as the differentiable optimization objectives that the differentiable physics solver aims to maximize. In an optimization program, two special functions, namely require and keep, can be used to specify the temporal relationship of the optimization objectives. By default, the require function evaluates the specified condition at the end of the trajectories. On the other hand, the keep function is applied to each frame of the trajectories, ensuring that the specified conditions are maintained throughout the trajectory. In the example program shown in Figure 4(B), the optimization program instructs the actuator to continuously make contact with the white dough. The function similar computes the Earth Mover distance [71] between two objects' point clouds, which aims to shape the dough into a thin pie, the goal shape extracted from the next key frame. We illustrate more example optimization programs in Figure 5.
4.2 Compiling Natural Languages with LLM
We utilize the power of the Large Language Model (LLM) to compile text instructions into optimization programs. This approach capitalizes on the few-shot capabilities of the LLM and employs various few-shot prompts [91, 9]. The prompts begin by defining the "types" and "functions" within our DSL. Each function is accompanied by a language explanation to clarify its purpose. Additionally, examples are given to demonstrate how primitive functions can be combined to form complex clauses. To facilitate the translation, we provide the language model with both the scene description of the
A) Grasp the front of the rope and then grasp the back of the rope. Deform the rope into its target shape.
B) Use the gripper to first lift up the white cube above the black wall and then move it to the target location
C) Put the knife above the white object. Cut the knife down to match the target shape of the white object. Ensure that the knife does not rotate.
Figure 5: A) Decomposing a stage into sub-stages: manipulating a rope into a circle using a single gripper requires the agent to first grasp the upper part of the rope and then manipulate the other side to form the circle. The program counterpart contains two pairs of sample and optimize, reflecting the two sub-stages in the language instruction. backend extracts the back end of 3D objects; B) Guiding the motion of the objects to avoid local optima: we use com to compute the center of the mass of objects and pcd to get the clouds; relation operators compare two objects' coordinate. The first require takes an additional argument 0.5, meaning the inner clause, which asks the white object above the black one, should be evaluated halfway through the trajectory to represent meanings of 'first do A and then do B.' C) Selecting a suitable tool to split the objects; D) The program can include additional constraints like not breaking the shape, which is critical for manipulating fragile materials.
D) Pick up the front part of object "top left mat" and place it into the goal without breaking it.
objects that it contains and the natural language instructions that we wish to compile. We then prompt the model with the instruction, "Please translate the instructions into a program." This step is repeated for all frames, resulting in their respective optimization programs. Through our experimentation, we have observed that GPT-4 [67] surpasses other models in terms of performance. Please refer to the appendix for a more comprehensive understanding of the translation process.
4.3 Solving the Optimization through Differentiable Physics