DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
Authors: Zhiao Huang
Feng Chen
Yewen Pu
Chunru Lin
Hao Su
Chuang Gan
Venue: NeurIPS 2023 poster (2023)
Keywords: Differentiable physics; Soft body manipulation
TL;DR: We introduce DiffVL, a method that enables non-expert users to communicate soft-body manipulation tasks -- a combination of vision and natural language, given in multiple stages -- that can be readily leveraged by a differential physics solver.

Abstract: Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems.
Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory.
However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users.
We introduce DiffVL, a method that enables non-expert users to communicate soft-body manipulation tasks -- a combination of vision and natural language, given in multiple stages -- that can be readily leveraged by a differential physics solver. 
We have developed GUI tools that enable non-expert users to specify 100 tasks inspired by real-life soft-body manipulations from online videos, which we'll make public.
We leverage large language models to translate task descriptions into machine-interpretable optimization objectives. The optimization objectives can help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines.

DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
Zhiao Huang"
Feng Chen'
Computer Science & Engineering University of California, San Diego
Institute for Interdisciplinary Information Sciences Tsinghua University chenf20@mails tsinghua edu.
z2huang@ucsd.cm edu
Yewen Pu Autodesk yewen. www.pppuuuuuu...cooom
Chunru Lin
UMass Amherst chunrulin@unae edu
Hao Sul
Chuang Gan
University of California, San Diego haosu@ucsd. edu
MIT-IBM Watson AI Lab, UMass Amherst
chuanggdumass.cm edu
Abstract
Combining gradient-based trajectory optimization with differentiable physics sim- ulation is an accurate and efficient technique for solving soft-body manipulation problems. Using a well-crafted optimization objective, the solver can quickly verge onto a valid trajectory. However, writing the appropriate objective functions requires expert knowledge, making difficult collect large of naturalistic problems from non-expert users. We introduce DiffVL, a framework that integrates the process from task collection trajectory generation leveraging combination of visual and linguistic task descriptions. A DiffVL task represents a long horizon sequence of 3D scenes (key frames) and natural language instructions connecting adjacent key frames. We built GUI tools and tasked non-expert users to transcribe 100 soft-body manipulation tasks inspired by real-life scenarios from online videos. We also developed novel method that leverages large language models to translate task language descriptions into machine-interpretabll optimization objectives, which can then help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines. Experiments show that existing baselines cannot complete complex tasks, while our method can solve them well. Videos can be found on the website https: /ttps://sae /tsp::///stess /sites.tssss/////e//////////et// google, wwoo//iww///// http///ifvllllll hwfddlfffffooooe/oo/
soft-body manipulation problem
Introduction
This paper focuses on soft body manipulation, a research topic with wide set of applications such as folding cloth [52, 90, 34], untangling cables [88, 87], and cooking foods [77, 72]. Due their complicated physics and high degree of freedom, soft body manipulations raise unique opportunities and challenges. Recent works such as [32] and [99] have heavily leveraged various differentiable physics simulators make these tasks tractable. Towards generalizable manipulation skill learning, such approaches have the potential to generate data for learning from demonstration
"Equal contribution
1 Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

algorithms [51, 43]. However, to enable differentiable physics solvers to generate large scale of meaningful trajectories, we must provide them with suitable tasks containing the scene and the optimization objectives to guide the solver. Previous tasks are mostly hand-designed [32] or pooceuur--generater [51], resulting in a lack of diverse and realistic soft-body manipulation tasks for researchers.
This paper takes another perspective by viewing tasks as data, or more precisely, task specifications as data. Each data point contains a pair of an initial scene and an optimization objective depicting the goal of the task. Taking this perspective, we can annotate meaningful tasks like annotating data of other modalities, like texts, images, videos, or each action trajectory. by leveraging non-expert human labor, providing possibilities for scaling up the task space of differentiable physies solvers.
core problem in building a task collection framework with trajectory generation is finding a suitable representation of the tasks. The representation should be intuitive for non-expert annotators while being accurate enough to describe the complex shapes and motions of the soft bodies that appear in various soft body manipulation processes. Last but not least, should be reliably interpreted by differentiable physics solvers to yield a valid trajectory.
A] Dumpling Video
B) Task Annotation
Optimization Programs
Hanginn.
ytnox
optinize
Soopptouchh 13,
Large
Language
c) DINVL Task
Model
NLB Fut the railing pi abose the white cough on the right and deform it reto the sorget shope
E) Sample and Optimize
MU
T VC
scene
scana 2
Scane
Figure 1: (A) A dumpling making video; (B) The annotator interacts with our GUI tool to create DiffVL tasks; (C) A DiffVL task contains a sequence of 3D scenes along with natural language instructions to guide the solver; (D) DiffVL leverages large language model to compile instructions into optimization programs consisting of vision elements; (E) The optimization program guides the solver solve the task in the end.
Studies such as [3, 1, 60, 89] have shown that humans are proficient at defining goal-driven, spatial object manipulation tasks using either sketches, natural languages, or both. Figure 1(A), for instance, depicts the complete process of dumpling creation, supplemented by textual instructions. Taking this as an inspiration, we present DiffVL, a framework that enables non-expert users to specify soft-body manipulation tasks to a differentiable solver using a combination of vision and language. Specifically, each DiffVL task consists of sequence of 3D scenes (keyframes), with natural language instruction connecting adjacent keyframes. The sequence of key frames specifies the sequence of subgoals of the manipulation task, and the natural language instructions provides suggestions on how to use the actuators to manipulate the objects through this sequence of subgoals. See Figure 1.
We develop tools to ease the annotation for non-expert users. With our interactive simulator equipped with GUI, the user can edit, manipulate, draw, or carve shapes easily like other 3D editing tools and observe the consequence through simulation in an intuitive manner. Meanwhile, when it is tedious for users to edit all the intermediate steps of a complex motion, they can use natural language to describe the goal instead of drawing them step by step. This enables us to build SoftVL100, a vision-language dataset with 100 diverse tasks. DiffVL uses LLM to compile the natural language instructions to an optimization program which enforces set of constraints during the soft-body manipulations from one keyframe to the next. This optimization program is then used by differentiable physics solver to generate working trajectory.
2

---

To summarize, our work makes the following contributions:
We propose a new multi-stage vision-language representation for defining soft-body manipulation tasks that is suitable for non-expert user annotations.
We developed a corresponding GUI, and curated SoftVL100, consisting of 100 realistic soft-body manipulation tasks from online videos?.
We develop a method. DisfVL, which marries the power of a large-language model and differentiable physics to solve a large variety of challenging long-horizur tasks in SoftVL100.
2 Related Work
Differentiable simulation for soft bodies Differentiable physics [30, 93, 69, 19, 29] has brought
unique opportunities in manipulating deformable objects of various types [58, 28, 53, 33, 92, 49, 95, 83, 63, 100, 95, 56, 27,47, 47, 74, 23], including plasticine [32], cloth [45], ropes [54] and fluids [96] simulated with mass-spring models [30], Position-based Dynamics Projective Dynamics [69, 16). MPM [36] or Finite Element Method [26], which could be differentiable through various
techniques [30, 16, 57, 7]. has been shown that soft bodies have smoother gradients compared to rigid bodies [32, 30, 93, 6, 82], enabling better optimization. However, the solver may still suffer from non-convexity [44] and discontinuities [82], motivating approaches to combine either stochastic sampling [44], demonstrations [43, 6] reinforcement learning [98, 64] to overcome the aforementioned issues. DifIVL capitalizes on an off-the-shelf differentiable physics solver to annotate tasks. It incorporates human priors through novel vision-language task representation. This allows users to employ natural language and GUI to guide the differentiable solver to solve long-horizon tasks, distinguishing Di from previous methods.
Task representation for soft body manipulation There are various ways of defining tasks for soft body manipulation. To capture the complex shape variations of soft bodies, it is natural to use either RGB images [95, 51, 73] or 3D point cloud [78, 47] to represent goals. However, given only the final goal images, it may be hard for the solver to find good solution without further guidance. Besides, generating diverse images or point clouds that adhere to physical constraints like gravity and collision avoidance in varied scenes can be challenging without appropriate annotation tools. Our GUI tools are designed to address these issues, simplifying the process. Other works use hand-defined features [10, 88] or formalized languages [41]. These methods necessitate specialist knowledge and require tasks to be defined on a case-by-case basis. Learning from demonstrations [43, 731 avoids defining the task explicitly but requires agents to follow demonstration trajectories and learn to condition on various goal representations through supervised learning or offline reinforcement learning [20, 18, 79, 24, 55, 35, 11, 5]. However, collecting demonstrations step-by-step [70, 101, 43] requires non-trivial human efforts, is limited to robots with human-like morphologies, and may be challenging for collecting tasks that are non-trivial for humans (c.g., involving complex dynamics or tool manipulation). Our annotation tool emphasizes the description of tasks over the identification of solutions. Annotators are merely required to define the task, not execute the trajectories themselves, thereby simplifying the annotation process, Our method is also related to [46], but we consider a broader type of tasks and manipulation skills.
Language-driven robot learning Many works treat language as subgoals and learn language- conditioned policies from data [80, 21, 66, 42, 8, 62, 61] or reinforcement learning [37, 17, 84, 12]. It has been shown that conditioning policies on languages lead to improved generalization and transferability [80, 37]. [76] turns languages into constraints to correct trajectories. Our method distinguishes itself by integrating language with differentiable physics through machine-interpretabbl optimization programs. Recent works aim at leveraging large language models [4, 15, 94, 50, 13] to facilitate robot learning. Most use language or code [48] to represent policies rather than goals and focus on high-level planning while relying on pre-defined primitive actions or pre-trained policies for low-level control. In contrast, our language focuses on low-level physics by compiling language instructions to optimization objectives rather than immediate commands. It's noteworthy that VIMA [38] also introduces multimodal prompts, embodying a similar ethos to our vision- language task representation. Their approach views multimodal representation as the task prompts to guide the policy. However, it still requires pre- prepprogrameed oracles to generate offline datasets for learning, which is hard to generalize to soft-body tasks with complex dynamics. Contrastingly, we
both the GUI and dataset will be made public
3

---

adopt a data annotation perspective, providing a comprehensive framework for task collection, natural language compiling, and trajectory optimization, enabling us harness the potential of differentiable physics effectively.
Task and motion planning Our optimization program compiled from natural language is closely connected to the field of Task and Motion Planning (TAMP) [22], particularly Logic Geometric Programming (LGP) [85, 86]. We draw inspiration from LGP's multiple logical states in designing our DiffVL task representation. However, unlike LGP, our approach asks human annotators to assign "logic states" and facilitates the manipulation soft bodies through our novel aision-language representation. Some recent works have explored the intersection of TAMP and language models [14, 81, 59] while primarily focusing on generating high-level plans.
SoftVL100: Vision-Language Driven Soft-body Manipulation Dataset
Put the roiling pin above the whot and manipuiste the wh into goal. places.
Then grasp yellow grisper
Finally graep wrap yellow
Grasp the back end rope. Then Wind the rope and place sa saimon ball. Grasp the front end ropn. Then Wind the rope and place the black ball.
Then the Finally (AMullistage
them theirs
examples
Figure 2: (A) Example of multi-stage tasks and their text annotations; (B) Snapshots of scenes in SoftVL100 dataset.
In this representation, the original tasks are divided into sequence 3D scenes, or key frames, along with text descriptions that detail the steps for progressing to the next scene. Figure 2(A) illustrates our representation. Each key frame consists of multiple soft bodies with different positions and shapes. By grouping two consecutive key frames together, we form manipulation stage. Natural language instructions may guide selection of an actuator and provide guidance on how to control it to reach the subsequent key frames. In Section 3. we develop annotation tools to facilitate the task representation annotation process. These tools make easy to construct a diverse set of tasks, referred to as SoftVL100, discussed in Section 3.2.
3.1 Vision-Langinn Task Annotator
Our annotation tool is based on Plasticinel.at Lab [32], simulation platform that utilizes the Material Point Method (MPM) [31] for clastoplastic material simulation. To enhance the user experience and support creation, integrate it into SAPIEN [97], framework that enables the development of customized graphical user interfaces.
Our simulation runs on GPU servers and the inter- face is accessible service through VNC, al- lowing users to interact with the simulator directly from their web browser. When
choson Sub goo
web
Annotetre
Select the object wdit
Add aitheew
Cuil il into two steeds
Preas: it into cylrdder
Lift
Figure 3: Example operations in GUI tools

---

starting a new task annotation, users can open the simulator, perform actions, and save the current scene. Our GUI tools provide comprehensive support for creating, editing, managing, and simulating objects, as illustrated in Figure 3. Scene creation: The GUI offers a variety of primitive shapes, such as ropes, spheres, cubes, and cylinders, that users can add to the scene. Before adding soft-body shape, users can adjust its size, color, and materials (e.g., rubber, fiber, dough, iron) if needed. Shape editing: Users have the ability to select, edit, or delete shapes within the scene. The GUI tool includes range of shape operators, including moving, rotating, carving, and drawing. We provide complete list of supported operations in the appendix. Simulation: Since the interface built on a soft body simulator, users can simulate and interact with the soft body using rigid actuators or magic forces. They can also run simulations to check for stability and collision-fre states. Object management: Users merge connected shapes to create single object divide object into two separate ones with distinct names. The annotation tool keeps tracking the identities of objects across scenes.
Once users have finished editing the scene and creating new key frame, they can save the scene into the key frame sequence. The key frames are displayed below the simulator, visualizing the current task annotation progress. Users have the option to open any of these saved scenes in the simulator, delete specific scene if needed, and add text annotations for each scene, Please refer to the appendix for detailed information on the functionalities and usage of the GUI tool.
3.2 The SoftVL100 Dataset
Our annotation tool simplifies the task creation process, enabling hire non-expert users to collect new tasks to form a new task set, SoftVL100. The task collection involves several stages, starting with crawling relevant videos from websites like YouTube that showcase soft body manipulation, particularly clay-making and dough manipulation. We developed tools to segment these videos and extract key frames to aid in task creation.
We hired students to annotate tasks, providing them with a comprehensive tutorial and step-by-step guidelines for using our annotation tools. It took approximately two hours for each annotator to become proficient in using the annotation tool. Subsequently, they were assigned a set of real-world videos to create similar tasks within the simulator. After collecting the keyframes, we proceed to relabel the scenes with text descriptions. Annotators are provided with a set of example text descriptions. We encourage users to include detailed descriptions of the actuators, their locations, moving directions, and any specific requirements, such as avoiding shape breakage for fragile materials. The annotation process for each task typically takes around 30 minutes. Using our task annotation tool, we have created a dataset called SoftVL.100 100, which consists of 100 tasks, and there more than 4 stages on average. The tasks cover wide range of skills as illustrated in Figure (A). Sample tasks from the dataset can be seen in Figure 2 (B).
4 Optimization with Vision-Language Task Description
We propose DiffVL to tackle the challenging tasks in SoftVL100. Given stage of task depicted in Figure 4(A), DiffVL utilizes large language models to compile the natural language instructions into auchine-interppetablle optimization program in Figure 4(B). The program comprises of the names of visible elements and Python functions, effectively capturing the essence of the language instructions. We introduce the design of our DSL in Section 4.1 and outline the DiffVI. compiler based on large language models (LLM) in Section 4.2. The resulting optimization program includes crucial information for selecting and locating actuators and can facilitate differentiable physics solver to generate valid trajectories, as discussed in Section 4.3.
4.1 Optimization Program
The optimization program is formulated using a specialized domain-specife language (DSL) that incorporates several notable features. Firstly, includes functions that extract the names of visible elements from the 3D scenes and support operations on these elements, represented as 3D point clouds. This enables manipulation and analysis of the visual information within the optimization program. Secondly, the DSL provides various functions to express geometric and temporal relations, encompassing common geometric and motion constraints. These functions facilitate the represen- tation and handling of natural language instructions. Furthermore, the DSL clauses are interpreted
5

---

A) One Stage of Task
Current Frame
Next Frame
BollingPin
((RollingPin)
Dough
y Dough))
(io Dough)
Distance
rolling pin above. dough on the right. Roll the pin To deform it into the torget shape.
6oal
Gj Smapling and Optimiation
Seuuire((imiler(euuuee Gowl))
Lorge
Extracted Visual Elements
y(Rol1ingPin) ((rax(())gh)))
Madel
B) Optimization Program
Hollingiin
"Rollinghin"
VC
Dough
get( white right dough")
goall right dough
// above the
y(RollingFin)
Splimize
Sampling Actuators
Rollout Trajectory
Endlngg
Keep
seform sough Larget
(Dough, Goal))
(MDgDe Goal)
diat Rollinggn, Dough)
Figure 4: (A) One stage of the DiffVL task of two keyframes and natural language instruction. (B)
The compiled optimization program. (C) The sampling and optimization process.
into PyTorch [68], making the constraints automatically differentiable and directly applicable for differentiable physics solvers.
The example in Figure 4 already exposes multiple components of our DSL. In the program, fer to the names of visible elements through get and goal functions, which can look for and extract the 3D objects that satisfy the description, identified by their color, shapes, and locations (we ensured that these attributes are sufficient to identify objects in our tasks). The program will then start with a sample statement, which used to specify the actuators and their associated constraints over initial positions. Actuators refer to the entities utilized for manipulation, such as robot gripper, knife, or the rolling pin depicted in Figure 4. The constraint y (RollingPin) y (max (pod (Dough))) within the sample statement represents specific constraint. It ensures that the height of the actuator (RollingPin) is greater than the highest y coordinate of the white dough in the scene, re- flecting the meaning of "put bove" of the language instructions. Following the sample statement, an optimize statement is included. This statement plays vital role in defining the constraints and objectives for the manipulation trajectories. These constraints and objectives serve as the dif- ferentiable optimization objectives that the differentiable physics solver aims to maximize. In an optimization program, two special functions, namely require and keep can be used to specify
the temporal relationship of the optimization objectives. By default, the require function evaluates
the specified condition at the end of the trajectories. On the other hand, the keep function is applied to each frame of the trajectories, ensuring that the specified conditions are maintained throughout the trajectory. In the example program shown in Figure 4(B), the optimization program instructs the actuator to continuously make contact with the white dough. The function similar computes the Earth Mover distance [71] between two objects' point clouds. which aims to shape the dough into thin pie, the goal shape extracted from the next key frame. We illustrate more example optimization programs in Figure 5.
4.2 Compiling Natural Languages with LLM
We utilize the power of the Large Language Model (LLM) to compile text instructions into opti- mization programs. This approach capitalizes on the few-shot capabilities of the LLM and employs various few-shot prompts [91,9] 9]. The prompts begin by defining the "types" and "functions" within our DSL. Each function is accompanied by a language explanation to clarify its purpose. Additionally, examples are given demonstrate how primitive functions can be combined form complex clauses. To facilitate the translation, we provide the language model with both the scene description of the

---

A) Grosp the frent the rope and then group the back of the rope. befonn the rape into its target shope.
B3 Use the gripper firet lift the white cubs, chove the block wall and then move it the tanget location
"gripper", gross frosttooootttteett "rope")13
samplet"prisee"" grasplgat( (hatte)))))))))))) ostimized
compled "prppeerrr proestarkmddig (repe"))33
requirebyccom "hitee)) (oy ("Baact113,, 0.5), requlined A Aoro ("anite")), contgoal ("uhite"))))
oot (eogrnneeerreesstelee pngestttaargggnn (orptarr""""""""""" (hrop")) goal (*1op=")33)
Put the kniife above the white object.
knite
motch
shops of the white
b) Pick up the front part of dbject 'top left mat' and place
that Inife
rotote.
the gool without breoking
camplo("inife", rollce.),
camplet" tariaeenn touchi fronspart) lart mxt"))
yf ("rrsfe)) X
brtial zeç
optimised
thepth (ootrroten teft (3t7)
eetporn (Natse)))) keepdnalit)) reguire(simiaer "httttt goald (inase")))
requiretaistaar left wat",
keeptro prreoooo_rrreeeoo nart)).
Figure 5: A) Decomposing a stage into sub-stages: manipulating a rope into a circle using a single gripper requires the agent first grasp the upper part of the rope and then manipulate the other side to form the circle. The program counterpart contains two pairs of sample and optimize reflecting
the two sub-stages in the language instruction. backend extracts the back end of 31D objects; B) Guiding the motion of the objects to avoid local optima: we use com to compute the center of the mass of objects and pcd to get the clouds; relation operators compare two objects' coordinate. The first require takes an additional argument 0.5, meaning the inner clause, which asks the white object above the black one, should be evaluated halfway through the trajectory to represent meanings of first do A and then do B. C) Selecting a suitable tool to split the objects; D) The program can include additional constraints like not breaking the shape, which is critical for manipulating fragile materials.
objects that it contains and the natural language instructions that we wish to compile. We then prompt the model with the instruction, "Please translate the instructions into program." " This step is repeated for all frames, resulting in their respective optimization programs. Through our experimentation, we have observed that GPT-4 [67] surpasses other models in terms of performance. Please refer to the appendix for more comprehensive understanding of the translation process.
4.3 Solving the Optimization through Differentiable Physics
We leverage the differentiable physies solver in [32], which has been proven accurate and efficient in solving the generated optimization programs. The process is illustrated in Figure 4(C). The sample function employed within a sampling-basedd motion planner. Initially, it samples the pose of the specified actuator type to fulfill the provided constraints. Subsequently, an RRT planner [40] determines a path toward the target location, and a PD controller ensures that the actuators adhere to the planned trajectory. With the actuator suitably initialized, the optimize clause then passed to a gradient- based optimizer to optimize the action sequence for solving the task. The solver performs rollouts, as depicted in Figure 4(C), where the conditions enclosed by keep operators are evaluated
at each time step. Conversely, conditions enclosed by require are evaluated only at specified time frames. The resulting differentiable values are accumulated to calculate a loss, which is utilized to compute gradients for optimizing the original trajectories. For handling multistage tasks involving vision-language representation, we can solve them incrementally, stage by stage.
Experiments
In this section, we aim justify the effectiveness of our vision-language task representation in guiding the differentiable physics solver. For detailed analysis of the ablation study on the language translator, we direct readers to the appendix. To better understand the performance of different solvers, we evaluate the tasks of two tracks. The first track focuses on tasks that only require agents to transition from one key frame to another. Tasks in this track are often short-horizon. This setup allows us to evaluate the efficiency and effectiveness of the differentiable physics solver in solving short- horizon soft body manipulation tasks, as described in Section 5.2. In the second track, we validate the agent's 's capability to compose multiple key frames in order to solve long- horizon manipulation tasks.
7

---

These tasks involve complex scenarios where agents need to switch between different actuators and manipulate different objects. The evaluation of this composition is discussed in Section 5.3,
5.1 Initialization for tasks
We clarify that our approach does not use a gradient-based optimizer to optimize trajectories generated by the RRT planner. Motion planning and optimization occur in separate temporal phases. Initially, we plan and execute trajectory to position the actuator at the specified initial pose. Subsequently, we initialize new trajectory starting from this pose, and the actuator remains fixed in this trajectory as we initialize the policy with zero actions to maintain its position. The sample function generates poses in line with the annotators' guidance.
5.2 Mastering Short Horizon Skills using Differentiable Physics
Before evaluating the full task set, we first study how our vision-language representation can create short-horizon tasks that are solvable by differentiable physics solvers. We picked 20 representative task stages as test bed from the SoftVL We classify these tasks into categories, with tasks in each category. The categories are as follows: Deformation involves the deformation of a large single 3D shape, achieved through pinching and carving: Move asks the movement of object or stacking one object onto another; Winding involves the manipulation of rope-like object to achieve different configurations. Fold focuses folding thin mat to wrap other objects. Cut involves splitting an object into multiple pieces. The selected tasks encompass a wide spectrum of soft body manipulation skills, ranging from merging and splitting shaping. We evaluate different methods' ability to manipulate the soft bodies toward their goal configuration. The agents are given the goal configuration in the next frame, and we measure the 3D IoU of the ending state as in [32]. We set a threshold for each scene's 's target IoU score to measure successful task completion, accommodating variations in IoU scores across different scenes.
We compare our method against previous baselines, including two representative reinforcement learning algorithms, Soft Actor-Critic (SAC) [25] and Proximal Policy Gradient (PPO) [75]. The RL. agents perceive a colored 3D point cloud as the scene representation and undergo 10 steps to minimize the shape distance towards the goal configuration while contacting objects. Additionally, we compare our approach with CPDeform [44], which employs a differentiable physics solver. However, CPDeform uses the gradient of the Earth mover's distance as a heuristic for selecting the initial actuator position. It differs from Dift'L which utilizes the text description to determine ways to initialize the actuators and the optimization objectives. For differentiable physics solvers, we run Adam [39] optimization for 500 gradient steps using learning rate of 0.02.
Table presents the success rates and mean IoU of various approaches. It is evident that, except for specific deformation and cutting tasks where SAC shows success, the RI. baselines struggle to solve most tasks. Although RL agents demonstrate an understanding of how manipulate the soft bodies correctly, they lack the precision required to match the target shape accurately. Additional experimental results can be found in the appendix. CPDeform achieves partial success in each task category. It surpasses the RL baselines by utilizing a differentiable physics solver and employing heuristic that helps identify contact points to minimize shape differences. However, CPDeform's heuristic lacks awareness of physics principles and knowledge about yielding objects to navigate around obstacles. Consequently, CPDeform fails to perform effectively in other tasks due these limitations. We want to emphasize that integrating language into RL baselines entails non- non-trivial challenges. For example, many reward functions our optimization program have temporal aspects, making the original state non- non-Markovian. As we focus on enhancing the optimization baseline through language-bar based experiments, we' 'e'ee chosen defer tackling the complexities of aanuuag--condttoone RL to future research.
Our approach surpasses all other methods, achieving the highest overall success rate. To examine the impact of different components and illustrate their effects, we conduct ablation studies. First, we remove the constraints in sample named "- Sample" in the table and ask the agent to sample random tools without using hints from the language instructions. As result, the performance drops significantly, either getting stuck at the local optima or failing to select suitable tools to finish the task. For example, in the cutting task, it failed to select knives and split the white material into two pieces. On the other hand, the variant " Optimize" replaces the statement in optimize by
8

---

previous baselines
DiffVL
SAC
PPO
CPDeform
Sample
Optimize
Ours
SR/IOU
SR/IOU
SR/OU
SR/IOU
SR/IOU
SR/OU
Deform
0.25 (0.504)
0.00/0.392
0.25/0.532
0.11/0.489
0,44/0.524
1.00/0.564
Move
0.00/0.541
0.00
0.08 /0.570
0.00/0.533
0.33/0.618
1.00/0.641
Wind
0.00/0.308
0.00/0.289
0.21/0.362
0.25 /0.351
0.08 /0.408
0.59/0.446
Fold
0.00/0.572
0.00 /0.457
0.33/0.612
0.00/0.550
1.00 /0.650
0,94/0,64 643
Cut
0.33/0.449
0.00/0.408
0.89/0.482
0.33/0.357
0.56/0.447
0.87/0.490
Total
0.12/0.475
0.00/0.415
0.35/0.512
0.14/0.456
0.48/0.529
0.88/0.557
Table I: Single stage experiment results.
The metrics we use are Success Rate (SR) and 3D
Intersection Over Union (IOU).
Start
-ample
Optimize
Curs
Goal
singia object another
back
final
(4) Use knffe to cut out the left part of the white mat
No acluator samale
[o] Single stage
(2)
Figure 6: (1) Performance on single stage tasks. (2)rerorrmance multistage task that includes
moving, packing, and pressing.
objective focusing solely optimizing shape distance, removing additional guidance for the physical process. The effects are lesser compared to the previous variant, demonstrating that with suitable tool initialization method, differentiable physics solver is already able solve many tasks. However, as illustrated in Figure 6(A), it does not lift the white cube to avoid the black wall in moving task and breaks the mat shown in the second row. In the winding task, fails to find the relatively complex littingtheenwinddng plan directly from shape minimization. In contrast, our method successfully solve these tasks after leveraging the trajectory guidance compiled from the natural language instructions. We find that our solver is quite efficient. For a single-stage task, takes minutes for 300 gradient descent steps on machine with NVIDIA GeForce RTX 2080, for optimizing trajectory with 80 steps. For most tasks, 300 gradient steps are sufficient for finding good solutions.
5.3 Driving Multi-Stage Solver using Vision-Languase
Having evaluated short-horizon tasks, apply our method to long-horizon tasks in SoftVL100. We leverage the natural language instructions to generate the evaluation metrics for each task, which might measure the relationships between objects, check if soft materials rupture, or if two shapes similar. We refer readers to the appendix for more details.
Our multistage vision-language task representation decomposes the long-horizon manipulation pro- cess into multiple short-horizon stages, allowing our method to solve them stagewise manner. baseline algorithms fail to make notable progress, apply ablation method: We first compare

---

our method with single-stage approach (single), which refers to solving long- long-horizon tasks like short- horizon tasks, by dropping the key frames and directly optimizing for the final goal. We then dropped the actuator sampling process in the middle (no actuator sample). The solver is still told to optimize for key frames (sub-goals) at each stage but was not informed switch actuators and objects to ma- nipulate. As expected, dropping the stage information and the language instructions in our vision task representation significantly degrades the performance, making solvers unable to complete most tasks. After leveraging the language instructions, the FinalOnly solver removes the vision subgoals in each sub-stage but provides the agent's s actuators to choose the objects ma- nipulate. In this case, similar to the "- Sample" variant, the differentiable physies solver can solve certain tasks by only minimizing shape distance. However, it may have troubles for tasks that require objects to deform into multi- ple shapes. For example, as illustrated in Figure 6(B), the solver needs first compress the purple shape into thin mat to warp the yellow dough.
10
00
Mullstagge task succees rate
The solver fails to discover the compression process after removing the subgoals but directly moves the purple to- ward its final configurations, resulting in the failure in the end. This showcases the importance of the introduction of the key frames in our vision- Laiionl task annotation.
Figure 7: Success rate of multi-star multi-stage tasks of different ablations
6 Limitations and Conclusion
We introduce DiffVL, method that enables non-expert users to design specifications of soft-body manipulation tasks through combination of vision and natural language. By leveraging large language models, task descriptions are translated into machine- maachne-interpretabblee optimization objectives, allowing differentiable physics solvers efficiently solve complex and long- ang-horizon multi-stage tasks. The availability of user- sser-friendly GUI tools enables a collection of 100 tasks inspired by real-life manipulations. However, it is important to acknowledge some limitations. Firstly, the approach relies on human labor for dataset creation. Additionally, the utilization of large language models incurs high computational costs. Nevertheless, We anticipate being able to gather larger set of tasks in the future. Furthermore, leveraging the generated trajectories to guide learning methods holds significant potential as promising avenue for future exploration.
7 Future Works
We believe that crowdsourcing is vital in achieving this goal. We have taken the first step to demonstrate how non-experts can contribute to scaling up robot learning, which may inspire the field to explore ways of involving individuals from various backgrounds in expanding robot manipulation capabilities. It would be out of the capabilities of group but be readily adopted by other groups with better resources eventually.
Transferring the policies found by the differentiable simulator to the real world is important. We believe that various components of our framework can offer valuable contributions real-world scenarios: one such contribution lies in our vision language representation, which naturally represents tasks in the real world; leveraging large language models aids in generating rewards for evaluating real-world results; our GUI tool can effectively manipulate real-world materials, particularly in constrained tasks, serving as a direct interface between humans and robot controllers.
Acknowledgements
This work is supported by DSO grant DSOCO2107, MIT-IBM Watson AI Lab, and gift funding from MERL,, Qualcomm, and Cisco. The exchange program of Feng Chen gifts funding from Tsinghua University, Institute for Interdisciplinary Information Sciences.
10

---

References
[1] Samuel Acquaviva, Yewen Pu, Marta Kryven, Catherine Wong, Gabrielle E Ecanow, Maxwell Nye, Theodoros Sechopoulos, Michael Henry Tessler, and Joshua Tenenbaum. Com- municating natural programs to humans and machines. arXiv preprint arXiv:2106.07824, 2021.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, pages 114. PMIR, 2020.
[3] Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Han- rahan, and Barbara Tversky. Designing effective step-by-step assembly instructions. ACM Transactions on Graphics (TOG), 22(3):828 837, 2003.
[4] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint 2rXiv:2204.01691, 2022.
[5] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making arXiv preprint arXiv:2211.15657 2022.
[6] Rika Antonova, Jingyun Yang, Krishna Murthy Jatavallabhula, and Jeannette Bohg. Rethink- ing optimization with differentiable simulation from global perspective. arXiv preprint arXiv:2207.00167, 2022.
[7] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Honderman-Milng and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[8] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068117 2022.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[10] Lawrence Yunliang Chen, Baiyu Shi, Daniel Seita, Richard Cheng, Thomas Kollar, David Held, and Ken Goldberg, Autobag: Learning to open plastic bags and insert objects. arXiv preprint arXiv:2210.12217, 2022.
[11] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15084-15097. Curran Associates, Inc., 2021.
[12] Cédric Colas, Ahmed Akakzia, Pierre-Y Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Language-condition goal generation: a new approach to language grounding for rl. arXiv preprint Xii:2006.07044 2020.
[13] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pages 893-905. PMLR, 2022.
[14] Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint 47X1v22303.06247, 2023.

---

[15] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson. Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.032: 2023.
[16] Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojeiech Matusik. Diffpd: Differentiable projective dynamics. ACM Transactions on Graphics (TOG), 41(2):1-21, 2021.
[17] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang. De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with snternet-scale knowledge. arXiv preprint arXiv:2206. 08853, 2022.
[18] Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Dynamics learning with cascaded variational inference for multi-step manipulation. arXiv preprint arXiv: 1910. 13395, 2019.
[19] C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint 1rXiv:2106.13281, 2021.
[20] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[21] Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. Lisa: Learning interpretable skill abstractions from language. arXiv preprint arXiv:2203.00054 2022.
[22] Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265 293, 2021.
[23] Narasimhan Gautham, Zhang Kai, Eisner Ben, Lin Xingyu, and Held David. Transparent liquid segmentation for robotic pouring. In International Conference on Robotics and Automation (ICRA), 2022.
[24] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horize tasks via imitation and reinforcement learning, arXiv preprint 191iv: 1956,, 2019.
[25] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[26] Eric Heiden, Miles Macklin, Yashraj S Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Robotics: Science and Systems, July 2021.
[27] Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning to control pdes with differentiable physics. International Conference on Learning Representations, 2020.
[28] Ryan Hoque, Daniel Seita, Ashwin Balakrishna, Aditya Ganapathi, Ajay Tanwani, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. VisuoSpatial Foresight for Physical Sequential Fabric Manipulation. Autonomous Robots (AURO) journal, 2021.
[29] Taylor A Howell, Simon Le Cleac'h, J Zico Kolter, Mac Schwager, and Zachary Manchester. Dojo: A differentiable simulator for robotics. arXiv preprint arXiv:2203.008066 2022.
[30] Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan- Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint
2019.
12

---

[31] Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(44)1-14, 2018.
[32] Zhiao Huang, Yuamming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In International Conference on Learning Representations, 2020.
[33] Zixuan Huang, Xingyu Lin, and David Held. Mesh-based dynamics model with occlusion reasoning for cloth manipulation. Robotics: Science and Systems (RSS), 2022.
[34] Zixuan Huang, Xingyu Lin, and David Held. Mesh-based dynamics with occlusion reasoning for cloth manipulation, 2022.
[35] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[36] Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials. In ACM SIGGRAPH 2016 Courses, pages 1-52. 2016.
[37] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances Neural Information Processing Systems, 32, 2019.
[38] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxin-2210.03094, 2022.
[39] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
[40] Steven M Valle and James Kuffner. Rapidly-explorins random trees: Progress and prospects: Steven m. lavalle, iowa state university, a james j. kuffner, jr., university of tokyo, tokyo, japan. Algorithmic and computational robotics, pages 303-307, 2001.
[41] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- lk: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80-93. PMLR, 2023.
[42] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances Neural Information Processing Systems, 35:31 199-3122,,,,,, 2022.
[43] Sizhe Li, Zhiao Huang, Tao Chen, Tao Du Hao Su, Joshua Tenenbaum, and Chuang Gan. Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physics. arXiv preprint arXiv:2304. 230.22333,...2,, 2023.
[44] Sizhe Li, Zhiao Huang, Tao Du, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Contact points discovery for soft-body manipulations with differentiable physics. arXiv preprint urXiv:2205.0288 02220.. 2022.
[45] Yifei Li, Tao Du, Kui Wu, Jie Xu, and Wojciech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Transactions Gruphics (TOG), 42(1):1-20, 2022.
[46] Yulong Li, Shubham Agrawal, Jen-Shuo Liu, Steven K Feiner, and Shuran Song. Scene editing as teleoperation: case study in 6dof kit assembly. In 2022 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2022.
13

---

[47] Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 112- 123. PMLR, 2022.
[48] Jacky Liang. Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprini arXiv:2209.0777.
2022.
[49] Junbang Liang, Ming C Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. Advances in Neural Information Processing Systems, 2019.
[50] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions feasible plans. arXiv preprint 1rX1v:2303.12133, 2023.
[51] Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B Tenenbaum, David Held, and Chuang Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. arXiv preprint arXiv:2203.17277 2022.
[52] Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning. 2021.
[53] Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, 2021.
[54] Fei Liu, Entong Su, Jingpei Lu, Mingen Li, and Michael C Yip. Robotic manipulation of deformable rope-like objects using differentiable compliant position-bant dynamics. IEEE Robotics and Automation Letters, 2023.
[55] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132 PMLR, 2020.
[56] Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. Fluid directed rigid body control using deep reinforcement learning. ACM Trans. Graph., 37(4), July 2018.
[57] Miles Macklin. Warp: A high-performanee python framework for gpu simulation and graphics. https://github www.ooo///oo/o wwwiii/wwaaaaaaaaaaa March 2022. NVIDIA GPU Technology Conference (GTC).
[58] Jeremy Maitin-Shepard Marco Cusumano- Towner, Jinna Lei, and Pieter Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference Robotics and Automation, pages 2308-2315 IEEE, 2010.
[59] Jiayuan Mao, Tomás Lozano-Pérez, Josh Tenenbaum, and Leslie Kaelbling, Pdsketch: In- tegrated domain programming, learning. and planning. Advances in Neural Information Processing Systems, 15:33672-366844 2022.
[60] William P McCarthy, Robert D Hawkins, Haoliang Wang, Cameron Holdaway, and Ju- dith E Fan. Learning to communicate about shared procedural abstractions. arXiv preprint arXiv:2107.0001 2021.
[61] Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 124551 1212, 2022.
[62] Oier Mees, Lukas Hermann, Erick Rosete-Beat Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditione policy learning for lorg-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 2022.
[63] Peter Mitrano, Dale McConachie, and Dmitry Berenson. Learning where to trust unreliable models in an unstructured world for deformable object manipulation. Science Robotics, 6654):cabd8170, 2021.
14

---

[64] Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference Machine Learning, pages 7805-7817. PMLR, 2021.
[65] Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Rateliff. Position based dynamics. Journal of Visual Communication and Image Representation, 18(2):109-118, 2007.
[66] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
[67] OpenAL. Gpt-4 technical report, 2023.
[68] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[69] Yiling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft multi-body systems. Advances in Neural Information Processing Systems, 34:17123-17135, 2021.
[70] Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters, 7(4):10873-100 10881, 2022.
[71] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as metric for image retrieval. International journal of computer vision, 40(2):99, 2000.
[72] Md. Sadman Sakib, David Paulius, and Yu Sun. Approximate task tree retrieval in a knowledge network for robotic cooking. IEEE Roboties and Automation Letters, 7(4): 1499:111111111 11499, 2022.
[73] Gautam Salhotra, 1-Chun Arthur Liu, Marcus Dominguez-Kuhne, and Gaurav S Sukhatme. Learning deformable object manipulation from expert demonstrations. IEEE Robotics and Automation Letters, 7(4):8775-8882, 2022.
[74] Connor Schenck and Dieter Fox. Visual closed-loop control for pouring liquids. In 2017 IEEE International Conference Robotics and Automation (ICRA), pages 2629-26334 IEEE, 2017.
[75] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1100100777.777 2017.
[76] Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv preprint urXiv:2204.05. 086,0 2022.
[77] Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools, 2023.
[78] Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning to see, simulate, and shape elasto-plastic objects with graph networks. Robotics: Science and Systems (RSS), 2022.
[79] Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model- based reinforce- ment learning, arXiv preprint XIXX::22770075600 2022.
[80] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages 994-006. PMIR, 2022.
[81] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint 2209v 11302, 2022.
15

---

[82] Hyung Ju Suh, Max Simchowitz. Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning. pages 20668-20696. PMLR, 2022.
[83] Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph E Gonzalez, and Ken Goldberg. Learning rope manipulation poli- cies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 941 1919.8 IEEE, 2020.
[84] Allison Tam, Neil Rabinowitz, Andrew Lampinen, Nicholas A Roy, Stephanic Chan, DJ Strouse, Jane Wang, Andrea Banino, and Felix Hill. Semantic exploration from lan- guage abstractions and pretrained representations. Advances in Neural Information Processing Systems, 35:25377-25389, 2022.
[85] Mare Toussaint. Logic-geomentt programming: An optimization- based approach to combined task and motion planning. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
[86] Mare A Toussaint, Kelsey Rebecca Allen, Kevin A Smith, and Joshua B Tenenbaum. Differen- tiable physics and stable modes for tool-use and manipulation planning. Robotics: Science and Systems Foundation, 2018.
[87] Vainavi Viswanath, Kaushik Shivakumar, Jainil Ajmera, Mallika Parulekar, Justin Kerr, Jeffrey Ichnowski, Richard Cheng. Thomas Kollar, and Ken Goldberg. Learning to trace and untangle semi-planar knots (tusk), 2023.
[88] Vainavi Viswanath, Kaushik Shivakumar, Justin Kerr, Brijen Thananjeyan, Ellen Novoseller, Jeffrey Ichnowski, Alejandro Escontrela, Michael Laskey, Joseph E Gonzalez, and Ken Goldberg. Autonomously untangling long cables. arXiv preprint arXiv:2207.078133 2022.
[89] Ruocheng Wang, Yunzhi Zhang, Jiayuan Mao, Chin-Yi Cheng, and Jiajun Wu, Translating visual lego manual to a machine-executaie plan. In Computer Eisi 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 677-694. Springer, 2022.
[90] Yufei Wang, Zhanyi Sun, Zackory Erickson, and David Held. One policy to dress them all: Learning to dress people with diverse poses and garments, 2023.
[91] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[92] Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, and David Held, Fab- ricflownet: Bimanual cloth manipulation with flow-based policy. Conference on Robot Learning, 2021.
[93] Keenon Werling, Dalton Omens, Jeongseok Lee, Ioannis Exarchos, and C Karen Liu. Fast and feature-complete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103.1666 16021, 2021.
[94] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658 2023.
[95] Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations, 2020.
[96] Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao- Yu Tung, Antonio Torralba, Katerina Fragkiadaki, and Chuang Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. arXiv preprins arXiv:2303.023444 2023.
[97] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part- based interactive environment. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
16

---

[98] Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint arXiv:220.007137, 2022.
[99] Zhenjia Xu, Zhou Xian, Xingyu Lin, Cheng Chi, Zhiao Huang, Chuang Gan, and Shuran Song. Robonimja: Learning an adaptive cutting policy for multi multi-material objects. arXiv preprint arXiv:2302.11553 2023.
[100] Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg. Self-supervised learning of state estimation for manipulating deformable linear objects. IEEE Robotics and Automation Letters, 5(2):2372-2379, 2020.
[101] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
17

---

Supplementary files
We provide video illustrating GUT tools, example task, and solutions. We also provide tutorial for helping users to use our annotation system as supplementary file.
A How to choose keyframes
The selection of keyframes is determined by the annotators, we believe humans can naturally decompose complex tasks into simpler ones for communication, When annotators manipulate a scene within the GUT, they can save the current scene as key frame within the task representation. They can also add, modify, or delete key frames using the editors within the web server interface.
The annotators were instructed on what kind of decomposition would likely result in a working trajectory, i.e., segmenting the manipulation processes contact point changes., which is simple and effective way for the annotator to provide high-quality labels, without having any expert understanding of the physics and the solver. If the agent is required manipulate new object the actuator needs to establish contact on different face of the object, annotators would introduce a new keyframe. Additionally, we employ heuristic methods segment YouTube videos. This segmentation occurs when there is a significant disparity between two frames, aiming to simplify the keyframe selection for annotators.
B DSL for Optimization Programs
Table 2 lists the functions and primitives our domain-specific language.
We rely on the large language model's few-shot ability to compile the natural language description like "lift up the white cube above the black wall" into an executable optimization program. The prompt start with the introduction to the DSL, listing the functions well as the explanations, followed by several examples illustrating how to compose those functions to implement complex functions, e.g., by checking the coordinates of two objects decide if one is the left of the other. Then we provide several pairs of natural language inputs and corresponding programs. In the end, we provide the natural language input compile. Below is example prompt.
Here the functions for obtaining objects and target objects. Their return types are all one single point-cloue object. desc can only include strings that contain shape ["rope", "sphere", "box", "mat"], colors like ["white", "gray", " "green", "red", "blue", black"], and position like ["left", "right", "top", "bottom"]: (desc) Get the point cloud of the objects with the description, for example, get white mat"). If the desc is 'all', then get all the objects.
goal (desc) # Get the point cloud of the goal for the objects with the given name. The name also be 'all', others (desc) Get given name.
other point clouds except the object with the
otherssgoale (deac) Get all other point clouds of the goals except the object with the given name.
Here are the functions that use constraints. Note that the obj. and goal can only be one single point-cloud:
(cond, start=s, end=t) # Minimize the reward from time to time t. It only takes one cond as argument.
require(cond, end=t) Reach the condition time t. By default end =1.
similar (obj, goal) # Compare the shape earth mover distance of the objects to the given goal shapes.
(obj) # Minimize the distance between the tool and the object. fix_place (obj) Ensure that the shape of the given object and its positions do not change. It only takes one argument.
no_break (obj) # Do not break the objects.
() # check if the actuator away from all objects
18

---

Category Objects
FuncName get(desc)
Explanation
Get the point cloud of the objects with the description, for example, get("left white mat"). the dese 'all', then get all the objects.
goal(deser)
Get the point cloud the goal for the objects with the given name. The name can also be 'all'. Minimize the cond from time time t. It only takes one cond as an argument.
keep(cond, start=O, end=l)
Temporal conditions
require(cond, end=1)
Reach the condition at time By default end is 1.
and(condl, cond2, com(shape)
Needs to satisfy all constraints
Compute the center of the mass of the point cloud
Shape operators
similar(A, B)
Compare the shape distance using EMD distance
pcd(shape)
Get the PyTorch tensor that repre- sents the point cloud of the shape Get part of the point cloud based the description.
leftpart, rightend, etc.
Actuator
(Sape())
Minimize the signed distance func- tion of the actuators and the shape Check if the actuator is far away from the shape
away
roll, vertical, etc.
Get the rotation angle of the actua- tor
Tensor operators
Relation >, 2,
Compare relationships of tors or scalar
Algebra +,- (2(A, B)
Add values something. Compare the la distance
min, max
Get the min max of tensor
Soft Body Constraints
fix shape
Do not change the shape of the ob- jects for too much
fix_place
Keep the object not moving
no_break
Do not break the objects
Special
stage(sample_fn fn, optimize_fin fn)
An optimization stage that may in- volve a motion planning procedure and optimization procedure. is only used for compiling.
Cimplettoo__name, *conds)
Select the tool of tool_name We provide "knife", "board", "rolling pin" and "gripper". The conds are list of conditions that we hope the tool satisfy
optimize( (>omis
Optimize the trajectory to satisfy the conditions.
Table 2: Elements in the optimization program
19

---

roll() # actuators' rotation about pitch actuator's 's rotation about y
yaw() actuator's rotation about y
vertical # actuator is vertical
horizontal() # actuator is horizontal
Here are the example functions to obtain additional information: com())j) Center of the objects.
(ob() # point clouds of the objects. max (ax(ppc(bb)))
max of the point cloud of the objects.
min (ped (obj)) min the point cloud of the objects. (<com (objo)) (pcd (objo)) # smallest X coordinate of the boundary of the points.
coordinate of the points.
We can compose those functions to implement various functions. Below are examples: The coordinatte of the objects
x x(com (com())))))
Objo is on the left of obj1.
1t (x (com (com (objjj))),, x x(com (obji)))
the front end the Objo above obj1. gt (y (com (frontend (objo))), y (max (pbd((t))))))
Objo is in front of objl. It (2(co (objo)), 2min (pdd(())c))))))))
deform the rope named by 'blue' into its goal shape require (similar ('blue', goal ('blue
deform object 'blue' into its goal shape while fixing others
goal bluu peeptfix (o (bblue')
))
first do then do require end=0. 5), require (B, end=1.0)
not rotate the actuator aboux axis keep (roll 0.1)
Here special function for
stage (sample_clause optimize_clause)
Each program may have multiple stage and each stage must have a 'sample' function and an 'optimize' function. There are at most three stages. Start new stage if and only if need change the actuators manipulate parts.
different objects *args) # sample the actuator with the given sample (actuator_nane name in the beginning, args denotes for the conditions or requirements of the actuator's 's pose.
(#aggs) conditions needs satiesfied during the manipulation.
Below are examples of the scenes and the program to solve the scenes.
Input: grasp the front end the blue rope vertically then deform into its goal pose and please do not break it. Then grasp the
20

---

back of the mat and then pick up the mat and move it into its goal position.
Program:
blue_rope get ("blue rope")
stage
sample ("gripper", grasp (frontend (bluu (blue rope)), vertical ()),
optimize(
require (similar (blue_rope, rope, goal ('blue rope )))),
keep (touch (blue_ (blue_rope)), keep (no (no_break (blue_rope))
mat get ("mat") stage
sample (" ("gripper", grasp (maa))at(tmtt)))) optimize(
require (12 (com (ga)())((aa))),, com (goal (aroaooooaaa"""""" ("mat")))),
keep (touch (backpart (mat)))
Input: cut the mat into its goal shapes and then move the knife avay. Program:
mat get ("mat")
stage
sample ("knife", touch (mat)), optimize(
require (similar (mat, goal ('mat )))) keep (touch (mat), end=0.5), requre (away ())
Input:
move the object 'A> the left then move to the right.
Program:
A = ()A*)
(eB')
stage
sample ("gripper", (A)),
optimize(
require (1t (1 (A)), x(com (B))), end=0.5), require (gt (com (A)), (com (B))), end-1), keep (touch (A))
Input: Put the board above all objects and deform them into their goal shapes and keep them not broken
Program:
stage(
sample ("board", gy (t ("borrdd)"" y (max optimize(
(Pall ))))),
require (similar goal ('all)))),
keep (touch ('all')),
keep (ne_break (?all))),
)
Input:
Use the gripper to grasp the right part of the object on the right and deform it into its target shape while firing others.
Program:
21

---

left get ('left stage(
sample ("gripper", rightpart (left)),
optimize(
require (sinilarrftt goal ('left
keep placcc ()))
keep (touch (rightpart (left))),
)
Input:
objects of different colors "red", "green". Use the board to touch them one by one. Deform them into their target shape.
There
Program:
stage
sample ("board", touch ("red")),
optimize(
keep ('
require (similar ("red", goal ("red")))
stage(
sample ("board", touch (* green")), optimize(
keep (touch 'ggren')),,
require (similar (" ("gree",, goal ("green")))
Please compile the input into a program. The generated programs satisfy aeveral requirements: Do not use functions not mentioned before; the program should be concise; only use 'touch' at most within single 'optimize' and if need manipulate multiple objects, please use touch('all)
Please include an 'optimize' for each stage Do not use more than one stage if it does not need to manipulate different objects parts Here the input compile:
B.1 Comparing GPT3.5 and GPT4
We notice that GPT4 has better performance compared with GPT3.5. Below provide the results of GPT3.5 and GPT4 of example tasks shown in the main text. Generally, we have found that is common for GPT3.5 to misinterpret the instructions, add/ignore constraints, and write incorrect commands with incorrect grammar. In contrast, GPT4 can generate pretty good results.
Although GPT4 already outperforms GPT3.5 in the translation task, we observed several failure cases in the translation process. For example, the output of GPT4 will contain objects that do not exist. In task that lifts sphere above box, the output may contain invalid sentence like require(similar((S) ('pphrre'' goal(" sphere above box'))) where 'sphere above box" which doesn't correspond to proper object. It may also generate statement with type error. For example, sometimes may output require 'sphere', ymmax(ped(" (obo(("oo)))) )))), which would trigger a compilation error Fortunately, those compilation errors could be detected during the compilation and can be resolved by resampling. However, there are also the instructions. For example, it may ask the solver to fix the position of an object that needs to move. In these cases, solver may fail to find suitable solution. Restricting the output fit particular syntax, for instance, using context-free grammars would resolve such problems.
not compute the shape distance between object and coordinate.
cases where LLM removes adds more constraints
Test Case Grasp the front of the rope and then grasp the back of the rope. Deform the rope into its target shape.
22

---

GPT3.5: added constraints not mentioned in the language
rope
get (" ("rope")
stage
sample (" ("gripper", grasp (frontend (ropp) ))
optimize
keep (touch (rope))))))))
require (similar (rope, goal goal( ('rope ))),
stage(
sample (" ("gripper,, grasp (backpart optimize(
keep (touch (rope)),
require (similar (rope, goal ('rope )))),
keep (no_break (rope))
rope
get (" ("rope")
stage(
sample ( ("gripper", grasp (ropteddrroooo)
optimize(
keep (touch (ropee)
require (similar (rope, goal ("rope")), end=0.5)
stage
sample (" ("gripper", grasp (backend (rope)))) optimize
keep (touch (backend (rope)))
require (similar (rope, goal ("rope")) end=1.0)
Test case 2 the target location.
Use the gripper to first lift up the white cube above the black wall and then move it to
GPT3.5: it creates two stages for single stage task (not need to regrasp the object).
white_cube get ("white cube") black_vall wall get ("black wall") stage
sample ("gripper", lift (1fitee__ccuub optimize(
require (gt (com (white_cube)) (max (bcc(bbcccwwwal)))) keep (touch (white, (uciite__ubb)))))_ keep (no_break break (white_cube))
stage(
sample sample ("gripper", gt (x (com (ube), , (ccmmblack.vall)) optimize(
require (12 (com (white_ cube), com (goal ("white cube keep (touch (white_cubel keep (no break (shite_cube))
23

---

GPT4
white_ cube get ("white cube") black_wall get ("blackk wall")
stage(
sample (" ("gripper", (raiteeeeteccube))
optimize
require (gt (ycoccyccitteeee (ubete_cube)), (max (po(kkkka_wwal))) end =0.5), require (12 (com (ubete_cube), com (goal ("white cube"))), end=1),
keep (touch (ubite_cube))
Test case the knife above the white object. Cut the knife down to match the target shape of the white object. Ensure that the knife does not rotate.
GPT3.5: It uses instead y extract the objectsheight while the latter is true. It also misinterprets the meaning to deform the knife into the goal.
white_obj get ("white") stage(
sample ("knife", ("xencem""i"ee")) (max <mmxx(pa((witte_bb))))] optimize(
require (12 (com ("knife"), require (similar ("knife", goal (**hhite))) (pitchh) 0.1, yau() 0.1)
(chite__bj))),
GPT4
stage(
sample ( ("knife", above (get ("white"))), optimize(
require
goal ("white"))
keep (touch ("shite")),
keep (roll < 0.1),
keep (pitch () < 0.1),
keep (yaw 0.1)
Test case 4 Pick up the front part of object 'top left mat' and place into the goal without breaking it.
GPT3.5; It adds an incorrect command to let the gripper to move away from the object in the beginning.
get ("top left mat") stage
sample ("gripper", grasp (frontpart (mat)), away(), optimize(
require (similar (mat, goal (' left mat?))),
keep (touch (mat)), keep (no_break break (mat))
24

---

GPT4
stage
sample ( gripper", grasp (froptprrttt (get ("top left mat")))),
optimize(
require (12 (com (get (" too left mat")), con com(goal ("top left mat"))
)
keep (touch (frontpart (get ("top left mat")))),
keep (no_break (get totop left mat")))
C Reinforcement learning baseline
We use the SAC and PPO implementation from basel-ba with hyperparameters Table 3. Figure 8 compares the optimization performance of RL algorithms and the Adam optimizer with respect to the IOU score of the ending state with different numbers of sampled episodes for the 20 single-stage tasks of five categories.
Table 3: Parameters for Reinforcement Learning
Table 4: SAC
Table 5: PPO Parameters
gamma
0.95
update steps
2048
learning rate
3 x 10
learning rate
3 x 10
buffer size
106
entropy coef
0
target update coef
0.005
value loss coef
0.5
batch size
256
batch size
64
Adam
ITO
SAC
4:5
B430
20
56
200
130
25
200
30
14 Marl)
3.2
0.29
3.5
3
3.30
20
20
100 100
27
20 12
18. 27
25 Dh-hhan...
171
111Ex
0.5
3.C
0-
0.0
Figure 8: Comparison between RL and differentiable physics solver
D Evaluation metries
We notice that in some tasks, a simple Intersection Over Union (IOU) score comparing shapes does not effectively explain the human intuition of task completion. For example, solver might "cut" two
25

---

objects similar to the target shape without truly separating them, as the space between the two parts might only contain a minimal amount of space. In other circumstances, the solver may violate the requirements of no_ bo_brak and break rope into two or destroy mat. To this end, we add checkers to check if two shapes are split and a shape is not broken for success evaluation. For no-breaking constraint, we track particles and ensure their distance to their initial nearest neighbor does not change too much. On the other hand, we can check if two shapes are separated from each other by finding the connected components of particles.
26


Reviews:
[{"review_id": "efbtMRmv8S", "reviewer": "Reviewer_Scwt", "summary": "This paper proposes DiffVL, a novel framework that tackles soft-body manipulation, which consists of a GUI for users to specify tasks easily and a large language model (LLM) for translating text instructions to programs for policy learning and execution. ", "questions": "How does this approach differ from other recent robot learning papers utilizing LLM for instruction generation/high-level planning? The novelty in this aspect is obscure to me.", "limitations": "The authors discussed limitations adequately in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- An intuitive user interface for task specification\n- A novel combination of LLM and a differentiable simulator for program generation and optimization\n- Good empirical success rate\n- Clear writing", "weaknesses": "- Heavy user input needed on the magnitude of hours to specify task\n- Comparison in Table 1 is a bit unfair since other pure RL baselines do not leverage language guidance\n- Writing is not polished (e.g., missing citations in line 21)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely thank you for your thorough assessment and encouraging remarks regarding our manuscript. We are pleased to learn that you appreciated the intuitiveness of our user interface and the novelty of our approach to integrating LLM. Additionally, we are grateful for your recognition of our empirical study and the clarity of our writing.\n\n> Heavy user input needed on the magnitude of hours to specify task\n\nThank you for bringing this issue to our attention. We acknowledge that minimizing human efforts is definitely an important research direction. However, most of the slowdowns can be significantly improved given more time and attention to details in the UI -- which is the realm of Human–computer interaction (HCI). For example, using advanced tools such as Augmented Reality, 3D mice, and even language-based instructions may accelerate the process. Our system can be much better with more attention to detail on the HCI front. However, we respectfully argue that while a better UI is relevant, it isn't the focus of our core contribution. Our current workflow, involving keyframes and natural language, serves as a guiding example for constructing UI systems that generate intuitive and machine-learnable data. This bridges the realms of HCI and robot learning and may inspire future research.\n\n> Comparison in Table 1 is a bit unfair since other pure RL baselines do not leverage language guidance\n\n\nAlthough we agree that it is worth integrating language into RL baselines, we would like to clarify that our RL baselines are set up in a very similar manner to the `-Optimize` approach. In both cases, we sample good actuator initialization poses and optimize policies with the sole objective of minimizing the shape distance. This setup ensures a fair comparison between our RL baselines and the `-Optimize` method. However, our RL baselines show significantly worse performance compared to the trajectory optimization baseline, which benefits from an analytical world model for optimization. This outcome is reasonable, given that previous work~[1] has already demonstrated the superiority of model-based trajectory optimization. As a result, we have focused our attention on conducting language-based experiments mainly on the optimization baseline, allowing us to concentrate on exploring the potential of our language-based approach.\n\nIn addition, integrating language into RL baselines may entail non-trivial challenges. For example, many reward functions in our optimization program have temporal aspects, making the original state non-Markovian. As we focus on enhancing the optimization baseline through language-based experiments, we've chosen to defer tackling the complexities of language-conditioned RL to future research. We'll elaborate on this in the revised manuscript.\n\n[1] Huang, Zhiao, et al. \"Plasticinelab: A soft-body manipulation benchmark with differentiable physics.\" arXiv preprint arXiv:2104.03311 (2021).\n\n> differ from other recent robot learning papers\n \nWe acknowledge that several papers have emerged recently exploring the application of LLM in robots. However, at the time of submitting our paper, our method distinguished itself from other approaches in several key aspects. We would like to highlight the following points that make our work novel:\n- First, our method distinguishes itself from other language-conditioned policies by focusing on translating natural language into rewards rather than generating a direct plan, instructions, or actions, as in previous approaches such as R11, PALM-E, and CLIPort. The fundamental concept underlying our approach is that verifying if a generated trajectory accomplishes the task is inherently simpler than generating the right plan. Consequently, our method only requires the LLM to perform the task of compiling a reward program, which is considerably simpler compared to generating a plan, which may exceed the capabilities of the LLM.\n- Secondly, the ability to control low-level physics emerges after integrating a model-based trajectory optimizer, which differs from most existing work that mainly works on high-level actions like picking certain objects.\n- Thirdly, our research specifically targets soft body problems, whereas the majority of existing work primarily focuses on rigid bodies. Softbody manipulation is a unique domain where our approach of using both vision and language guidance as a way of communicating a task is best suited. In comparison, in a rigid body setting, a language-only task definition such as \"put the cup on the table\" might be already sufficient.\n- Lastly, it is crucial to emphasize that our framework has a distinctive objective of collecting tasks as data, which sets it apart from existing work that assumes the existence of pre-defined tasks. This novel formulation enables the injection of human feedback into the trajectory generation process, providing a more interactive and adaptive approach. By incorporating human feedback and iteratively refining the system, we aim to create a more robust and flexible framework that can adapt to a wide range of scenarios and tasks.\n\n> Writing is not polished (e.g., missing citations in line 21)\n\nThanks for pointing out the issue. We will publish it carefully.\n\n\nWe hope our explanation helps alleviate your concerns. If you have any further questions, please do not hesitate to ask.\n"}, {"review_id": "M6krlwoXqp", "reviewer": "Reviewer_kids", "summary": "This paper demonstrates curating a set of 100 soft-body manipulation tasks and provides expert policies for them by using a mix of: annotators that provide supervision in the form of keyframes and/or natural langauge annotation, translating the annotations into programs via an LLM, and solving an optimization.", "questions": "Do each of the points discussed in the weaknesses make sense?", "limitations": "The authors do discuss limitations in the conclusion. They should add that the methods are limited to simulation and impact on the real-world is left as a future question to address.\nThey don't discuss societal impacts but I think that's okay here.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "One strength of this paper is that obtaining manipulation policies for soft bodies is undoubtedly hard, and so anything that can do is has some potential value to the community.\n\nAdditionally, the dataset if released by the authors could be valuable.  The tasks and expert policies could be used as data to study methods that require some amount of expert data (imitation / offline RL / etc), and it's hard to both get these types of policies, as well as build simulation environments, in order to study these well. Good datatsets and tasks can be considered a bottleneck of manipulation research, especially with soft body manipulation, and the dataset could be valuable here.", "weaknesses": "The first sentence of this paper has unfinished citations. It reads, verbatim: \"This paper focuses on soft body manipulation, a research topic with a wide set of applications such as folding cloth[cite], untangling cables [cite], and cooking foods[cite].\"  This is a pity because it makes it hard to take the work too seriously given an incomplete first sentence.  Meanwhile, the rest of the paper seems relatively polished, so I'm doing my best to evaluate it seriously.\n\nOne limitation that should be kept in mind is that in multiple ways I only see the methods of this paper as being relevant to simulation, and not directly to the real world.  This applies both to the 3D GUI annotation process, as well as the process for converting into something that the differentiable physics solver can solve.  The work could help enable further studies in simulation that could eventually impact the real-world, indirectly either through helping provide evaluation tools or perhaps eventually sim-to-real, but not directly.\n\nOverall, I think the main potential interesting thing is that the authors have actually provided expert policies for 100 or so soft-body manipulation tasks. This is hard to achieve.  I personally find that the existence of the policies and tasks is the main valuable and interesting thing, and less so the methods used to obtain them. \n\nAdditionally, 100 tasks is not that large for a crowdsourced effort.  This is an important point in my opinion because it's likely the process might have just gone faster if the first author designed all the tasks themselves rather than teaching others to do so.  One might need to get to 1,000 or 10,000 tasks in order to really start to see the benefits of a crowdsourced effort.  Accordingly this in my opinion undermines the care taken in the paper to make the annotation process usable by non-experts.\n\nAnother lens to look at this paper is: what actual results are shown?  There are two results tables/figures: Table 1, and Figure 7.  Table 1 shows that it is hard for SAC or PPO to solve the tasks, demonstrating that they are indeed not easy.  It also shows that their method, which has access to considerably more info than SAC/PPO due to its use of the annotations, can solve the tasks.  This is not a fully fair comparison, and that's fine.  The second results table shows that the multi-stage annotations and optimizations are useful.  This also makes sense.  It is hard to pinpoint what specific contribution claim this results figure supports though, since I don't think there is clear novelty in the multi-step framework presented by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful evaluation. We genuinely appreciate your recognition of the significance of our soft body manipulation dataset and the challenges it presents. Your criticism is invaluable to us, and we would be more than happy to discuss your concerns in greater detail.\n\n> I personally find that the existence of the policies and tasks is the main valuable and interesting thing, and less so the methods used to obtain them.\n\nWe want to emphasize that our contribution goes beyond just the dataset itself. Our work highlights that not only the dataset itself but also the task representation and methodology employed in creating the dataset play a crucial role in robot learning. With our approach, other researchers will be able to efficiently curate their own datasets in a similar manner, beyond what we have already collected.\n- We advocate the benefits of using crowd-sourcing for task collection, as opposed to conventional methods like gathering demonstration trajectories. The latter \"requires non-trivial human efforts, is limited to robots with human-like morphologies, and may be challenging for collecting tasks that are non-trivial for humans.\" On the other hand, it also helps to avoid the tedious reward engineering and pave the path toward a large-scale dataset with diverse tasks.\n- We use a large language model to compile instructions for rewards, setting us apart from recent studies that use LLMs to generate high-level plans. This approach is not only more straightforward for the LLM but also allows us to leverage existing solvers for precise control over soft bodies, offering finer-grained manipulation capabilities.\n- Our approach introduces a novel way of representing tasks by combining vision and language, enabling the creation of a differentiable program capable of describing multi-stage temporal procedures. It simplifies trajectory annotation and seamlessly integrates with trajectory optimization; introducing the vision subgoal allows us to express complex deformations of soft bodies.\n- We also want to emphasize the importance and the potential of the GUI tool. It facilitates more straightforward communication between the robot and human annotators, enhancing the overall annotation process. The current GUI tool has some room for improvement -- for instance, using a 3D mouse to ease the manipulation of 3D objects.\n\nThrough the tables/figures, we illustrated that \"considerably more info\" and \"the multi-stage annotations and optimization\" can solve the tasks, **which evidences the importance of suitable annotation and the design of the annotation tools**. This is why we designed the tool to help annotators gather and use that information to solve the tasks.\n\n\n> being relevant to simulation; they should add that the methods are limited to simulation and impact on the real-world is left as a future question to address.\n\nWe acknowledge the gap between the simulator we used and real-world environments, yet we want to emphasize that we focus on both automatic tasks and policy generation. The goal is to develop a method to scale up the demonstration collection process and enable robots to master more diverse skills. Transferring the policies found by the differentiable simulator to the real world~[1] is important but orthogonal to our current study. Moreover, we believe that various components of our framework can offer valuable contributions to real-world scenarios: one such contribution lies in our vision language representation, which naturally represents tasks in the real world; leveraging large language models aids in generating rewards for evaluating real-world results; our GUI tool can effectively manipulate real-world materials, particularly in constrained tasks, serving as a direct interface between humans and robot controllers. Thus, we consider real-world experiments compatible with our approach instead of the limitation of the existing method.\n\n[1] Xu, Zhenjia, et al. \"Roboninja: Learning an adaptive cutting policy for multi-material objects.\" arXiv preprint arXiv:2302.11553 (2023).\n\n\n> One might need to get to 1,000 or 10,000 tasks in order to really start to see the benefits of a crowdsourced effort. It's likely the process might have just gone faster if the first author designed all the tasks themselves\n\nWe respectfully push back your point. Our framework has already benefited us by creating DiffVL100, the largest soft body manipulation task set so far. Notice how challenging it is for a single author to create 100 tasks, which may take $100\\times 0.5=50$ hours, while involving several students, e.g., 10, in parallelizing the collection process can finish the time in one day ($50/10=5$ hours) and capable of generating a broader range of diverse tasks. Thus, we believe that crowdsourcing is vital in achieving this goal.  We have taken the first step to demonstrate how non-experts can contribute to scaling up robot learning, which may inspire the field to explore ways of involving individuals from various backgrounds in expanding robot manipulation capabilities. This direction is clearly promising and not constrained by the number of tasks. Scaling up to 1000 tasks requires in total $1000\\times 0.5=500$ hours. It would be out of the capabilities of our group but can be readily adopted by other groups with better resources eventually.\n\n\nOnce again, thank you for taking the time to review our manuscript. We will polish it further and include the discussions above. We're eager to discuss and address any concerns you may have."}, {"review_id": "PVktJeUTCr", "reviewer": "Reviewer_xC3q", "summary": "This paper presents a novel approach to soft body manipulation employing the strengths of the Large Language Model (LLM). The key innovation is viewing tasks as data, with each data point consisting of an initial scene and an optimization objective. To tackle the challenge of task representation, the authors introduce DiffVL, a framework enabling non-expert users to define soft-body manipulation tasks to a differentiable solver using a combination of vision and language. The users specify tasks via an interactive simulator with a sequence of 3D scenes (keyframes) connected by natural language instructions. The authors also developed a corresponding GUI and curated SoftVL100, a vision-language dataset with 100 diverse tasks. They further developed a method that combines the power of a large-language model and differentiable physics to solve a wide variety of challenging long-horizon tasks in SoftVL100. This study's contributions lie in its new task representation, the developed GUI, the curated dataset, and the DiffVL method.", "questions": "• It would be good to have more info about time constraints - simulation and training time for different tasks, not only training progress per epoch.\n• IWere there any simulation failure cases? How challenging was simulator tuning for different tasks?\n", "limitations": "Limitations were addressed fairly well.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper under discussion is a robust, well-rounded work, bringing to the table significant contributions across multiple facets. It excels in terms of novel methodology, tool development and proposes a new dataset:\n\n• The authors introduce an innovative multi-stage vision-language representation. This new approach simplifies the definition of soft-body manipulation tasks, making it accessible for non-expert user annotations - a noteworthy contribution to research in this area.\n\n• In terms of tool development, the authors have crafted a corresponding Graphical User Interface (GUI), enhancing the user experience and overall accessibility of their proposed approach. They have also curated SoftVL100, a compilation of 100 realistic soft-body manipulation tasks derived from online videos. This dataset stands as a valuable resource for further research and application in this domain.\n\n• Moreover, the authors have devised DiffVL, a method that bridges the gap between a large-language model and differentiable physics. This combination of strengths is adeptly applied to tackle a wide variety of challenging long-horizon tasks presented in SoftVL100, which marks a significant advancement in the field.\n\nThe experimental section of the paper is another key strength, featuring a thorough ablation study.\n", "weaknesses": "The relative weakness of the approach can be considered the amount of human labor required for dataset collection. Also, DSL could sometimes be too constrained to define arbitrary new tasks. Despite these points, they in no way diminish the overall quality of the work presented in the paper. The minor limitations identified simply highlight areas for potential future refinement. The project stands as an excellent example of research and tool development, and I am eager to see how it evolves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're thrilled about your positive feedback on our work. Your acknowledgment of our innovation, robustness, and solid foundation is truly gratifying. Your encouraging words inspire us to pursue excellence in all our efforts. Thank you.\n\n>  the amount of human labor required for dataset collection\n\nWe agree that reducing the amount of human labor is an important research direction. We foresee that better UI design, although it is not our core contribution, can significantly improve the annotation process with more attention on the HCI frontier. For example, we can develop better tools like augmented reality (AR), 3D mice, and even language instruction. Our current workflow, involving keyframes and natural language, demonstrated how to construct the appropriate data format (namely, keyframes and natural language annotations), along with the UI system, which makes it feasible to both (a) enable end users to provide high-quality annotations and (b) having the data be readily usable by machines. We hope our work can bridge the realms of HCI and robot learning and bring better synergies between humans and robots.\n\n> DSL could sometimes be too constrained to define arbitrary new tasks.\n\nWe acknowledge that our DSL is not designed to cover arbitrary natural language. For instance, our DSL assumes a fixed number of stages and often deals with one object per stage, making it less suitable for scenarios involving multiple objects moving simultaneously. Additionally, the requirement to specify exact time makes it challenging to implement temporal logic like `do A until B.` Moreover, the DSL supports only a limited range of spatial and dynamic relationships and lacks coverage for the various verbs used in everyday language. \n\nHowever, although rudimentary, our proposed DSL does not need to be \"complete\" as natural language. Notice that it serves as an auxiliary loss function for completing the correct trajectory, and the lack of certain DSL primitives does not make coming up with a trajectory infeasible but only more difficult. This difficulty can be overcome elegantly by further decomposing the task into sub-tasks, simply by providing more keyframes to guide the trajectory step by step, effectively making a motivated user always possible to solve the task with a physics backend. \n\nMoreover, it is possible to refine our DSL for a wider range of scenarios. An interesting approach is to conduct a more thorough user study as in [1], which can help us identify common elements and distill valuable insights to create a more refined DSL, ensuring that our DSL remains relevant and tailored to the needs of our users and the tasks they encounter.\n\n[1] Acquaviva, Sam, et al. \"Communicating natural programs to humans and machines.\" Advances in Neural Information Processing Systems 35 (2022): 3731-3743.\n\n> Were there any simulation failure cases? How challenging was simulator tuning for different tasks?\n\nOur simulator is constructed using the MPM algorithm from PlasticineLab, which prioritizes time efficiency over simulation accuracy. Consequently, a frequent issue observed is that when the soft body moves at high speeds, it may penetrate through the rigid actuators due to the lower MPM resolution used for fast simulations. However, we did not yet fine-tune the simulator for specific tasks to keep the simulator tuning simple.\n\n>  It would be good to have more info about time constraints - simulation and training time for different tasks, not only training progress per epoch.\n\nFor a single-stage task, it evenly takes 10 minutes for 300 training steps on a machine with NVIDIA GeForce RTX 2080 Super, which contains 80 steps in the simulation. For most tasks, 300 steps in training is sufficient. We will include a task-level task time analysis in our revised manuscript."}, {"review_id": "nQyNYdNmtE", "reviewer": "Reviewer_EcLH", "summary": "This study focuses on soft-body manipulation problems such as flattening dough using a rolling pin, cutting deformable objects, and more. It introduces a method that engages non-expert users to provide detailed annotations and identify sub-goal states within key frames of the task video. Despite requiring additional human intervention, this approach generates a rich learning signal for manipulation tasks involving deformable objects. This significantly eases the learning difficulty associated with such tasks. The proposed method has been evaluated based on six fundamental deformable object manipulation skills, demonstrating its effectiveness.", "questions": "Q1: How is a key frame chosen? Also, can you explain how the decomposition of long-horizon tasks occurs?\n", "limitations": "See weaknesses and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work proposes a unique representation of task specifications. It incorporates human-labeled annotations for deformable object tasks to shape an objective/reward function. Such a reward function provides far richer reward signals than just a goal state, potentially greatly reducing policy learning difficulties.\n\nFurthermore, this study offers a new dataset that includes 100 deformable object manipulation tasks, complemented by human annotations on key frames. This contribution could be beneficial to the broader community and greatly aid studies related to deformable object manipulation.\n\nThe presentation of this research is both clear and organized.\n", "weaknesses": "Human annotation offers a valuable reward signal, which could potentially simplify the learning process. However, it also imposes certain constraints on the policy strategy. When states deviate from the demonstrated trajectory, adhering strictly to specific action commands could exacerbate the situation. Hence, at its current stage, it lacks the flexibility needed to handle unseen states not accounted for in the instructions/annotations.\n\nMoreover, each task necessitates considerable human involvement in creating subgoal states and corresponding annotations. Although software solutions have been developed to assist in this process, it still represents a significant involvement of human resources and incurs considerable costs.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comprehensive evaluation and positive feedback on our paper. It's truly gratifying to know that you found value in our unique task representation, the novel manner in which we incorporated LLM, and the clarity throughout our writing.\n\n> Imposes constrains on policy strategies; Strict adherence to action commands can worsen trajectory deviations; Lacks flexibility for unseen states outside annotations.\n\n\nThank you for bringing this issue to our attention. If we understand correctly, it refers to the case when our solver fails to reach a certain keyframe, making the reward signal in the subsequent stage useless. This might happen when users need to provide better annotations. We acknowledge it as a limitation of our current solver, and could in theory introduce a DAGGER-like approach to allow users to actively adjust annotations by adding or editing keyframes based on the found trajectory and offering corrective input. This integration of the annotation and optimization could provide a \"closed-loop\" control to help the agent recover from those unseen states. \n\n\n> Necessitates considerable human involvement\n\nThank you for bringing this issue to our attention. While we recognize our current labeling system can be significantly improved, we would like to point out that our framework is already sufficient to help us build SoftVL100 --  a compact yet diverse dataset with many distinct tasks and high-quality annotations on how to achieve the desired manipulations. We believe considerable human involvement is a good thing and is synergetic with having an efficient annotation experience so that non-expert annotators can provide information-rich annotations for robotic tasks without unsatisfying overheads. We are confident that we can significantly reduce human efforts with enhanced toolchains. For example, using advanced tools such as Augmented Reality, 3D mice, and even language-based instructions can facilitate the operation of soft bodies. However, the further improvement of the toolchain lies in the realm of HCI and is beyond our scope. We would like to leave it as future work.\n\n> How is a key frame chosen? Also, can you explain how the decomposition of long-horizon tasks occurs?\n\nThe selection of keyframes is determined by the annotators, as we believe humans can naturally decompose complex tasks into simpler ones for communication. When annotators manipulate a scene within the GUI, they can save the current scene as a key frame within the task representation. They can also add, modify, or delete key frames using the editors within the web server interface. \n\nThe annotators were instructed on what kind of decomposition would likely result in a working trajectory, i.e., segmenting the manipulation processes at contact point changes., which is a simple and effective way for the annotator to provide high-quality labels, without having any expert understanding of the physics and the solver. If the agent is required to manipulate a new object or the actuator needs to establish contact on a different face of the object, annotators would introduce a new keyframe. Additionally, we employ heuristic methods to segment YouTube videos. This segmentation occurs when there is a significant disparity between two frames, aiming to simplify the keyframe selection for annotators.\n\n\nWe hope our explanation help alleviate your concerns. If you have any further questions, please do not hesitate to ask."}, {"review_id": "wusnfEXdf0", "reviewer": "Reviewer_C43D", "summary": "The paper works on trajectory generation for soft body manipulation with differentiable simulation. To address the key challenge of representing task goals for optimization, the authors propose to use natural language descriptions to lower the barrier for annotation, where a framework that utilizes LLM for translating natural language into optimization programs is developed to facilitate data collection.", "questions": "In section 4.3, the authors say that a sampling-based RRT planner is used to determine the path for the actuator, and then the trajectory is refined with a gradient-based optimizer. Could the authors provide more details on how the trajectory is initially calculated and how it is refined for execution? Besides, it would be better to provide more snapshots for the rollouts to show the difference between the initial trajectory and the final ones; \n\nWould the trajectory optimization be sensitive to the initialization? It would be interesting to conduct some experiments for demonstrating the robustness of the initialization strategy; \n\nAre there any failure cases that are caused by the LLM translation? It would be good to understand the failure mode in which situations the LLM failed to generate appropriate optimization programs.", "limitations": "Limitations are discussed in the paper, but no failure cases are presented and analyzed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of task specification with natural language is an interesting idea; \n\nThe overall framework seems to be reasonable and technical sound, it is smart to leverage LLM for converting natural language descriptions into optimization programs; \n\nThe paper is well-presenting and easy to read.", "weaknesses": "More evaluations can be performed to validate the robustness of the trajectory generation; \n\nThere is no discussion of the failure cases; \n\nSome missing citations in line 21; \n\nTypo in figure 4: smapling -> sampling; \n\nSome works that leverage natural language for task goal specifications, which may need to be discussed in the related work:  \n\n[1] StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects, ICRA 2022; \n\n[2] Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement, ICRA 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough assessment and positive feedback concerning our paper. It is genuinely rewarding to learn that you recognize the significance of our distinctive task representation, the novel method to integrate LLM, and the overall clarity evident in our writing. Your insights are invaluable to us.\n\n\n> Could the authors provide more details on how the trajectory is initially calculated and how it is refined for execution?\n\n\nWe clarify that our approach does not use a gradient-based optimizer to optimize trajectories generated by the RRT planner. Motion planning and optimization occur in separate temporal phases. Initially, we plan and execute a trajectory to position the actuator at the specified initial pose. Subsequently, we initialize a new trajectory starting from this pose, and the actuator remains fixed in this trajectory as we initialize the policy with zero actions to maintain its position. We then only optimize the second trajectory to operate the actuator to complete the task. The optimizer does not touch the previous trajectory and keeps it unaltered. \n\n\n> More evaluations can be performed to validate the robustness of the trajectory generation & sensitivity to the initialization\n\nWe emphasize that our vision-language task representation is designed to provide a good initial pose for trajectory optimization.  The `sample` function generates poses in line with the annotators' guidance, and the optimization trajectory starts with zero actions. This yields a robust optimizer for different optimization programs.\n\nThe table below illustrates the solver's performance across several sampled tasks. The standard deviation, calculated across five seeds, highlights that the observed variance is not significant.\n\n|                   | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 |\n| ----------------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| Average IOU Score | 0.575   | 0.453   | 0.684   | 0.529   | 0.474   | 0.604   |\n| Variance          | 0.026   | 0.029   | 0.052   | 0.039   | 0.016   | 0.007   |\n\n\n\n> Discussion of the failure cases; failure cases caused by the LLM translation\n\nAlthough GPT4 already outperforms GPT3.5 in the translation task, we observed several failure cases in the translation process. For example, the output of GPT4 will contain objects that do not exist. In a task that lifts a sphere above a box, the output may contain an invalid sentence like `require(similar('sphere', goal('sphere_above_box')))` where `sphere_above_box` which doesn't correspond to a proper object. It may also generate a statement with a type error. For example, sometimes it may output `require(similar('sphere', y(max(pcd('box')))))`, which would trigger a compilation error as we can not compute the shape distance between an object and a coordinate. Fortunately, those compilation errors could be detected during the compilation and can be resolved by resampling. However, there are also rare cases where LLM removes or adds more constraints to the instructions. For example, it may ask the solver to fix the position of an object that needs to move. In these cases, our solver may fail to find a suitable solution. Restricting the output to fit a particular syntax, for instance, using context-free grammars would resolve such problems~[1].  We will conduct a more comprehensive analysis of these failure cases and elaborate on them in the revised manuscript.\n\n[1] Shin, Richard, et al. \"Constrained language models yield few-shot semantic parsers.\" arXiv preprint arXiv:2104.08768 (2021).\n\n> Some works that leverage natural language for task goal specifications, which may need to be discussed in the related work\n\nThanks for pointing out the related literature. Our work aligns closely with leveraging natural language for task goal specifications. Our vision-language task representation can be considered a richer extension of prior language goal representations in the following aspects:\n- Our vision-language task representation introduces a temporal dimension, offering a more detailed description of the *physical process* rather than just the *scene*. The description of the process aids the solver in synthesizing the trajectory beyond single-goal guidance.\n- We provide the GUI tool to directly create vision goals instead of only relying on language, yielding a more intuitive and accurate means of specifying goals.\n- This flexibility empowers us to focus on low-level physics and manipulate soft bodies effectively.\n\nWe believe it is possible to integrate ideas from language-guided goal specifications into our work. We look forward to exploring the synergy between these approaches and will include more discussion in our revised manuscripts.\n\nWe will improve our writing and polish the manuscripts. If you have any further questions, please do not hesitate to ask."}]

Author rebuttal:
# General response

We thank all reviewers and ACs for their time and effort in reviewing the paper. We are glad that the reviewers generally recognized the following contributions.

**Problem and dataset** The paper curated a hard and valuable dataset (`kids`, `xC3q`), which is beneficial to the broader community and greatly aids studies related to deformable object manipulation (`EcLH`).

**Method** The paper proposes a novel framework combining LLM and a differentiable simulator(`Scwt`). The multi-stage vision-language representation is innovative (`xC3q`), interesting (`C43D`) and unique (`EcLH`). The overall framework is reasonable and technically sound (`C43D`). It is smart to leverage LLM(`C43D`), and the reward signal is valuable(`EcLH`).

**Tool development** Intuitive interface (`Scwt`) and GUI that enhances the user experience (`xC3q`).

**Experiments** The paper achieved a good empirical success rate (`Scwt`) and provided a thorough ablation study (`xC3q`).

**Presentation** The writing is clear (`Scwt`). The paper is organized (`EcLH`) and easy to read (`C43D`).

We have carefully considered all the questions and concerns raised by the reviewers and provided a detailed response to each reviewer's questions and concerns. We have polished our manuscripts and fixed typos.

We hope our responses have convincingly addressed all reviewers’ concerns. We thank all reviewers’ time and efforts again! Please don’t hesitate to let us know of any additional comments on the manuscript or the changes.

Best Regards,

Authors

Meta-review: This submission proposes a novel method related to soft-body manipulation using LLM. The submission also provides a GUI to specify tasks and also a vision-langauge soft-body manipulation dataset too. While the submission proposes a rather heavy engineering system method, it questions a timely important problem. However, there are several legitimate concerns from reviewers: if it is scalable to the real world (both the method and the gui), and also if the provided dataset is really useful in terms of its scale. Summing them up, we believe it is good to be shared to the community, but if the authors wants this paper to be more impactful, we suggest to increase the dataset size at the least.
Decision: Accept (poster)