SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We leverage the differentiable physics solver in [32], which has been proven accurate and efficient in solving the generated optimization programs. The process is illustrated in Figure 4(C). The sample function is employed within a sampling-based motion planner. Initially, it samples the pose of the specified actuator type to fulfill the provided constraints. Subsequently, an RRT planner [40] determines a path toward the target location, and a PD controller ensures that the actuators adhere to the planned trajectory. With the actuator suitably initialized, the optimize clause is then passed to a gradient-based optimizer to optimize the action sequence for solving the task. The solver performs rollouts, as depicted in Figure 4(C), where the conditions enclosed by keep operators are evaluated at each time step. Conversely, conditions enclosed by require are evaluated only at specified time frames. The resulting differentiable values are accumulated to calculate a loss, which is utilized to compute gradients for optimizing the original trajectories. For handling multistage tasks involving vision-language representation, we can solve them incrementally, stage by stage.
In this section, we aim to justify the effectiveness of our vision-language task representation in guiding the differentiable physics solver. For a detailed analysis of the ablation study on the language translator, we direct readers to the appendix. To better understand the performance of different solvers, we evaluate the tasks of two tracks. The first track focuses on tasks that only require agents to transition from one key frame to another. Tasks in this track are often short-horizon. This setup allows us to evaluate the efficiency and effectiveness of the differentiable physics solver in solving short-horizon soft body manipulation tasks, as described in Section 5.2. In the second track, we validate the agent's capability to compose multiple key frames in order to solve long-horizon manipulation tasks.
These tasks involve complex scenarios where agents need to switch between different actuators and manipulate different objects. The evaluation of this composition is discussed in Section 5.3.
5.1 Initialization for tasks
We clarify that our approach does not use a gradient-based optimizer to optimize trajectories generated by the RRT planner. Motion planning and optimization occur in separate temporal phases. Initially, we plan and execute a trajectory to position the actuator at the specified initial pose. Subsequently, we initialize a new trajectory starting from this pose, and the actuator remains fixed in this trajectory as we initialize the policy with zero actions to maintain its position. The sample function generates poses in line with the annotators' guidance.
5.2 Mastering Short Horizon Skills using Differentiable Physics
Before evaluating the full task set, we first study how our vision-language representation can create short-horizon tasks that are solvable by differentiable physics solvers. We picked 20 representative task stages as our test bed from the SoftVL100. We classify these tasks into 5 categories, with 4 tasks in each category. The categories are as follows: Deformation involves the deformation of a large single 3D shape, achieved through pinching and carving; Move asks the movement of an object or stacking one object onto another; Winding involves the manipulation of a rope-like object to achieve different configurations. Fold focuses on folding a thin mat to wrap other objects. Cut involves splitting an object into multiple pieces. The selected tasks encompass a wide spectrum of soft body manipulation skills, ranging from merging and splitting to shaping. We evaluate different methods' ability to manipulate the soft bodies toward their goal configuration. The agents are given the goal configuration in the next frame, and we measure the 3D IoU of the ending state as in [32]. We set a threshold for each scene's target IoU score to measure successful task completion, accommodating variations in IoU scores across different scenes.
We compare our method against previous baselines, including two representative reinforcement learning algorithms, Soft Actor-Critic (SAC) [25] and Proximal Policy Gradient (PPO) [75]. The RL agents perceive a colored 3D point cloud as the scene representation and undergo  $10^{6}$  steps to minimize the shape distance towards the goal configuration while contacting objects. Additionally, we compare our approach with CPDeform [44], which employs a differentiable physics solver. However, CPDeform uses the gradient of the Earth mover's distance as a heuristic for selecting the initial actuator position. It differs from DiffVL which utilizes the text description to determine ways to initialize the actuators and the optimization objectives. For differentiable physics solvers, we run Adam [39] optimization for 500 gradient steps using a learning rate of 0.02.
Table 1 presents the success rates and mean IoU of various approaches. It is evident that, except for specific deformation and cutting tasks where SAC shows success, the RL baselines struggle to solve most tasks. Although RL agents demonstrate an understanding of how to manipulate the soft bodies correctly, they lack the precision required to match the target shape accurately. Additional experimental results can be found in the appendix. CPDeform achieves partial success in each task category. It surpasses the RL baselines by utilizing a differentiable physics solver and employing a heuristic that helps identify contact points to minimize shape differences. However, CPDeform's heuristic lacks awareness of physics principles and knowledge about yielding objects to navigate around obstacles. Consequently, CPDeform fails to perform effectively in other tasks due to these limitations. We want to emphasize that integrating language into RL baselines entails non-trivial challenges. For example, many reward functions in our optimization program have temporal aspects, making the original state non-Markovian. As we focus on enhancing the optimization baseline through language-based experiments, we've chosen to defer tackling the complexities of language-conditioned RL to future research.
Our approach surpasses all other methods, achieving the highest overall success rate. To examine the impact of different components and illustrate their effects, we conduct ablation studies. First, we remove the constraints in sample named “- Sample” in the table and ask the agent to sample random tools without using hints from the language instructions. As a result, the performance drops significantly, either getting stuck at the local optima or failing to select suitable tools to finish the task. For example, in the cutting task, it failed to select knives and split the white material into two pieces. On the other hand, the variant “- Optimize” replaces the statement in optimize by
previous baselines	DiffVL	Ours
SAC	PPO	CPDeform	- Sample	- Optimize
SR/IOU	SR/IOU	SR/IOU	SR/IOU	SR/IOU	SR/IOU
Deform	0.25 (0.504)	0.00/0.392	0.25/0.532	0.11/0.489	0.44/0.524	1.00/0.564
Move	0.00/0.541	0.00 /0.527	0.08 /0.570	0.00/0.533	0.33/0.618	1.00/0.641
Wind	0.00/0.308	0.00/0.289	0.21/0.362	0.25 /0.351	0.08 /0.408	0.59/0.446
Fold	0.00/0.572	0.00 /0.457	0.33/0.612	0.00/0.550	1.00 /0.650	0.94/0.643
Cut	0.33/0.449	0.00/0.408	0.89/0.482	0.33/0.357	0.56/0.447	0.87/0.490
Total	0.12/0.475	0.00/0.415	0.35/0.512	0.14/0.456	0.48/0.529	0.88/0.557
Table 1: Single stage experiment results. The metrics we use are Success Rate (SR) and 3D Intersection Over Union (IOU).
(a) Move a single object to another side
(a) Multistage Task
(b) Pick up the front end of top left mat and use it to cover the brown core
(b) Our solution
(c) First pick up the back end of the yellow rope and move it across the orange one
(c) final only
(d) Use knife to cut out the left part of the white mat
(1)
Figure 6: (1) Performance on single stage tasks. (2)Performance on a multistage task that includes moving, packing, and pressing.
(d) No actuator sample
(2)
(e) Single stage
an objective focusing solely on optimizing shape distance, removing additional guidance for the physical process. The effects are lesser compared to the previous variant, demonstrating that with a suitable tool initialization method, a differentiable physics solver is already able to solve many tasks. However, as illustrated in Figure 6(A), it does not lift the white cube to avoid the black wall in a moving task and breaks the mat shown in the second row. In the winding task, it fails to find the relatively complex lifting-then-winding plan directly from shape minimization. In contrast, our method can successfully solve these tasks after leveraging the trajectory guidance compiled from the natural language instructions. We find that our solver is quite efficient. For a single-stage task, it takes 10 minutes for 300 gradient descent steps on a machine with NVIDIA GeForce RTX 2080, for optimizing a trajectory with 80 steps. For most tasks, 300 gradient steps are sufficient for finding good solutions.
5.3 Driving Multi-Stage Solver using Vision-Language
Having evaluated short-horizon tasks, we apply our method to long-horizon tasks in SoftVL100. We leverage the natural language instructions to generate the evaluation metrics for each task, which might measure the relationships between objects, check if soft materials rupture, or if two shapes are similar. We refer readers to the appendix for more details.
Our multistage vision-language task representation decomposes the long-horizon manipulation process into multiple short-horizon stages, allowing our method to solve them in a stagewise manner. As baseline algorithms fail to make notable progress, we apply ablation to our method: We first compare
our method with a single-stage approach (single), which refers to solving long-horizon tasks like short-horizon tasks, by dropping the key frames and directly optimizing for the final goal. We then dropped the actuator sampling process in the middle (no actuator sample). The solver is still told to optimize for key frames (sub-goals) at each stage but was not informed to switch actuators and objects to manipulate. As expected, dropping the stage information and the language instructions in our vision task representation significantly degrades the performance, making solvers unable to complete most tasks.
After leveraging the language instructions, the FinalOnly solver removes the vision subgoals in each sub-stage but provides the agent's actuators to choose the objects to manipulate. In this case, similar to the "- Sample" variant, the differentiable physics solver can solve certain tasks by only minimizing shape distance. However, it may have troubles for tasks that require objects to deform into multiple shapes. For example, as illustrated in Figure 6(B), the solver needs to first compress the purple shape into a thin mat to warp the yellow dough.
Figure 7: Success rate of multi-stage tasks of different ablations
The solver fails to discover the compression process after removing the subgoals but directly moves the purple toward its final configurations, resulting in the failure in the
end. This showcases the importance of the introduction of the key frames in our vision-language task annotation.
We introduce DiffVL, a method that enables non-expert users to design specifications of soft-body manipulation tasks through a combination of vision and natural language. By leveraging large language models, task descriptions are translated into machine-interpretable optimization objectives, allowing differentiable physics solvers to efficiently solve complex and long-horizon multi-stage tasks. The availability of user-friendly GUI tools enables a collection of 100 tasks inspired by real-life manipulations. However, it is important to acknowledge some limitations. Firstly, the approach relies on human labor for dataset creation. Additionally, the utilization of large language models incurs high computational costs. Nevertheless, We anticipate being able to gather a larger set of tasks in the future. Furthermore, leveraging the generated trajectories to guide learning methods holds significant potential as a promising avenue for future exploration.
7 Future Works
We believe that crowdsourcing is vital in achieving this goal. We have taken the first step to demonstrate how non-experts can contribute to scaling up robot learning, which may inspire the field to explore ways of involving individuals from various backgrounds in expanding robot manipulation capabilities. It would be out of the capabilities of our group but can be readily adopted by other groups with better resources eventually.
Transferring the policies found by the differentiable simulator to the real world is important. We believe that various components of our framework can offer valuable contributions to real-world scenarios: one such contribution lies in our vision language representation, which naturally represents tasks in the real world; leveraging large language models aids in generating rewards for evaluating real-world results; our GUI tool can effectively manipulate real-world materials, particularly in constrained tasks, serving as a direct interface between humans and robot controllers.
This work is supported by DSO grant DSOCO2107, MIT-IBM Watson AI Lab, and gift funding from MERL, Qualcomm, and Cisco. The exchange program of Feng Chen gifts funding from Tsinghua University, Institute for Interdisciplinary Information Sciences.
[98] Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint arXiv:2204.07137, 2022.
[99] Zhenjia Xu, Zhou Xian, Xingyu Lin, Cheng Chi, Zhiao Huang, Chuang Gan, and Shuran Song. Roboninja: Learning an adaptive cutting policy for multi-material objects. arXiv preprint arXiv:2302.11553, 2023.
[100] Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg. Self-supervised learning of state estimation for manipulating deformable linear objects. IEEE Robotics and Automation Letters, 5(2):2372-2379, 2020.
[101] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
We provide a video illustrating our GUI tools, example task, and solutions. We also provide a tutorial for helping users to use our annotation system as a supplementary file.
A How to choose keyframes
The selection of keyframes is determined by the annotators, as we believe humans can naturally decompose complex tasks into simpler ones for communication. When annotators manipulate a scene within the GUI, they can save the current scene as a key frame within the task representation. They can also add, modify, or delete key frames using the editors within the web server interface.
The annotators were instructed on what kind of decomposition would likely result in a working trajectory, i.e., segmenting the manipulation processes at contact point changes., which is a simple and effective way for the annotator to provide high-quality labels, without having any expert understanding of the physics and the solver. If the agent is required to manipulate a new object or the actuator needs to establish contact on a different face of the object, annotators would introduce a new keyframe. Additionally, we employ heuristic methods to segment YouTube videos. This segmentation occurs when there is a significant disparity between two frames, aiming to simplify the keyframe selection for annotators.
B DSL for Optimization Programs
Table 2 lists the functions and primitives in our domain-specific language.
We rely on the large language model's few-shot ability to compile the natural language description like "lift up the white cube above the black wall" into an executable optimization program. The prompt starts with the introduction to the DSL, listing the functions as well as the explanations, followed by several examples illustrating how to compose those functions to implement complex functions, e.g., by checking the  $x$  coordinates of two objects to decide if one is on the left of the other. Then we provide several pairs of natural language inputs and corresponding programs. In the end, we provide the natural language input to compile. Below is an example prompt.
Here are the functions for obtaining objects and target objects. Their return types are all one single point-cloud object. A desc can only include strings that contain shape ["rope", "sphere", "box", "mat"], colors like ["white", "gray", "green", "red", "blue", "black"], and position like ["left", "right", "top", "bottom']: - get desc) # Get the point cloud of the objects with the description, for example, get("left white mat"). If the desc is 'all', then get all the objects. - goal desc) # Get the point cloud of the goal for the objects with the given name. The name can also be 'all'. - others (desc) # Get all other point clouds except the object with the given name. - others goals (desc) # Get all other point clouds of the goals except the object with the given name. Here are the functions that use constraints. Note that the obj, and goal can only be one single point-cloud: - keep (cond, start=s, end=t) # Minimize the reward from time s to time t. It only takes one cond as an argument. - require (cond, end=t) # Reach the condition at time t. By default end =1. - similar(obj, goal) # Compare the shape earth mover distance of the objects to the given goal shapes. - touch(obj) # Minimize the distance between the tool and the object. - fix_place(obj) # Ensure that the shape of the given object(s) and its positions do not change. It only takes one argument. - no_break(obj) # Do not break the objects. - away() # check if the actuator is away from all objects
Category	FuncName	Explanation
Objects	get desc)	Get the point cloud of the objects with the description, for example, get("left white mat"). If the desc is 'all', then get all the objects.
goal (descr)	Get the point cloud of the goal for the objects with the given name. The name can also be 'all'.
Temporal conditions	keep (cond, start=0, end=1)	Minimize the cond from time s to time t. It only takes one cond as an argument.
require (cond, end=1)	Reach the condition at time t. By default end is 1.
and (cond1, cond2, ...)	Needs to satisfy all constraints
Shape operators	com(shape)	Compute the center of the mass of the point cloud
similar(A, B)	Compare the shape distance using EMD distance
pcd(shape)	Get the PyTorch tensor that repre-sents the point cloud of the shape
leftpart, rightend, etc.	Get a part of the point cloud based on the description.
Actuator	touch (ShapeA)	Minimize the signed distance func-tion of the actuators and the shape
away	Check if the actuator is far away from the shape
roll, vertical, etc.	Get the rotation angle of the actua- tor
Tensor operators	Relation>,<,≥,≤	Compare relationships of two vec-tors or scalar
Algebra+, -l2(A, B)min, max	Add values to something. Compare the l2distance
Get the min max of a tensor
Soft Body Constraints	fix_shape	Do not change the shape of the ob-jects for too much
fix_placeno_break	Keep the object not moving Do not break the objects
Special	stage (sample_fn, optimize_fn)	An optimization stage that may in-volve a motion planning procedure and an optimization procedure. It is only used for compiling.
sample (tool_name, *conds)	Select the tool of tool_name. We provide "knife", "board", "rolling pin" and "gripper". Theconds are list of conditions that we hope the tool satisfy
optimize (*conds)	Optimize the trajectory to satisfy the conditions.
Table 2: Elements in the optimization program
- roll() # actuators' rotation about x
- pitch() # actuator's rotation about y
- yaw() # actuator's rotation about y
- vertical() # actuator is vertical
- horizontal() # actuator is horizontal
Here are the example functions to obtain additional information:
- com(obj) # Center of the objects.
- pcd(obj) # point clouds of the objects.
- max(pcd(obj)) # max of the point cloud of the objects.
- min(pcd(obj)) # min of the point cloud of the objects.
- x(com(obj0)) # X coordinate of the points.
- x(min(pcd(obj0))) # smallest X coordinate of the boundary of the points.
We can compose those functions to implement various functions. Below are examples:
# The x coordinate of the objects
x(com(obj0))
# Obj0 is on the left of obj1.
lt(x(com(obj0)), x(com(obj1)))
# the front end of the Obj0 is above obj1.
gt(y(com(frontend(obj0))), y(max(pcd(obj1)))))
# Obj0 is in front of obj1.
lt(z(com(obj0)), z(min(pcd(obj1)))))
# deform the rope named by 'blue' into its goal shape
require(similar('blue', goal('blue')))
# deform object 'blue' into its goal shape while fixing others
require(similar('blue', goal('blue')), keep(fix_place(others('blue')))
# first do A then do B
require(A, end=0.5), require(B, end=1.0)
# not rotate the actuator aboux x axis
keep(roll() < 0.1)
Here are special function for
- stage(sample_clause, optimize_clause) # Each program may have multiple stage and each stage must have a 'sample' function and an 'optimize' function. There are at most three stages. Start a new stage if and only if we need to change the actuators or manipulate different objects or parts.
- sample(actuator_name, *args) # sample the actuator with the given name in the beginning, args denotes for the conditions or requirements of the actuator's pose.
- optimize(*args) # conditions needs to satiesfied during the manipulation.
Below are examples of the scenes and the program to solve the scenes.
Input: grasp the front end of the blue rope vertically and then deform into its goal pose and please do not break it. Then grasp the
back of the mat and then pick up the mat and move it into its goal position.   
Program:   
blue_ripe  $\equiv$  get("blue_ripe")   
stage( sample("gripper",grasp(frontend(blue_ripe)),vertical(), optimize( require(similar(blue_ripe,goal('blue_ripe'))keep(touch(blue_ripe)keep(no_break(blue_ripe)) 1   
)   
mat  $=$  get("mat")   
stage( sample("gripper",grasp(backpart(mat))), optimize( require(12(com(get(mat)),com(goal("mat"))), keep(touch(backpart(mat))) 1   
)   
Input: cut the mat into its goal shapes and then move the knife away.   
Program:   
mat  $=$  get("mat")   
stage( sample("knife",touch(mat)), optimize( require(similar(mat,goal('mat'))keep(touch(mat),end=0.5), require-away()) 1   
)   
Input: move the object 'A' to the left of 'B' then move it to the right.   
Program:   
A  $=$  get('A')   
B  $=$  get('B')   
stage( sample("gripper",grasp(A)), optimize( require(1t(x(com(A)),x(com(B)),end=0.5), require(gt(x(com(A)),x(com(B)),end=1), keep(touch(A)) 1   
)   
Input: Put the board above all objects and deform them into their goal shapes and keep them not broken   
Program:   
stage( sample("board",gt(y("board"),y(max(pcd('all'))))) , optimize( require(similar('all',goal('all'))), keep(touch('all')), keep(no_break('all')) 1   
)   
Input: Use the gripper to grasp the right part of the object on the right and deform it into its target shape while fixing others. Program:
left  $=$  get('left')   
stage( sample("gripper", rightpart(left)), optimize( require(similar(left, goal('left'))), keep(fix_place(others('left'))), keep(touch(rightpart(left))), ）   
Input: There are two objects of different colors "red", "green". Use the board to touch them one by one. Deform them into their target shape.   
Program:   
stage( sample("board", touch("red")) , optimize( keep-touch('red')), require(similar("red", goal("red")) )   
）   
stage( sample("board", touch("green")) , optimize( keep(touch('green')), require(similar("green", goal("green")) )   
）   
Please compile the input into a program. The generated programs satisfy several requirements: Do not use functions not mentioned before; the program should be concise; only use 'touch' at most once within a single 'optimize' and if we need to manipulate multiple objects, please use touch('all').   
Please include an 'optimize' for each stage. Do not use more than one stage if it does not need to manipulate different objects or parts. Here is the input to compile:
B.1 Comparing GPT3.5 and GPT4