SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We notice that GPT4 has a better performance compared with GPT3.5. Below we provide the results of GPT3.5 and GPT4 of example tasks shown in the main text. Generally, we have found that it is common for GPT3.5 to misinterpret the instructions, add/ignore constraints, and write incorrect commands with incorrect grammar. In contrast, GPT4 can generate pretty good results.
Although GPT4 already outperforms GPT3.5 in the translation task, we observed several failure cases in the translation process. For example, the output of GPT4 will contain objects that do not exist. In a task that lifts a sphere above a box, the output may contain an invalid sentence like require(similar('sphere', goal('sphere above box')))) where 'sphere above box' which doesn't correspond to a proper object. It may also generate a statement with a type error. For example, sometimes it may output require(similar('sphere', y(max(pcd('box')))), which would trigger a compilation error as we can not compute the shape distance between an object and a coordinate. Fortunately, those compilation errors could be detected during the compilation and can be resolved by resampling. However, there are also rare cases where LLM removes or adds more constraints to the instructions. For example, it may ask the solver to fix the position of an object that needs to move. In these cases, our solver may fail to find a suitable solution. Restricting the output to fit a particular syntax, for instance, using context-free grammars would resolve such problems.
Test Case 1 Grasp the front of the rope and then grasp the back of the rope. Deform the rope into its target shape.
GPT3.5: added constraints not mentioned in the language
rope  $=$  get("rope")   
stage( sample("gripper",grasp(frontend(lope))), optimize( keep(touch(lope)), require(similar(lope,goal('rope'))), keep(no_break(lope)) ）   
）   
stage( sample("gripper",grasp(backpart(lope))), optimize( keep-touch(lope)), require(similar(lope,goal('rope'))), keep(no_break(lope)) ）   
）
rope  $=$  get("rope")   
stage( sample("gripper",grasp(frontend(lope))), optimize( keep(touch(frontend(lope))), require(similar(lope,goal("rope"，  $\mathrm{end} = 0.5)$  ）   
）   
stage( sample("gripper",grasp(backend(lope))), optimize( keep-touch(backend(lope))， require(similar(lope,goal("rope"），  $\mathrm{end} = 1.0$  ）   
）
Test case 2 Use the gripper to first lift up the white cube above the black wall and then move it to the target location.
GPT3.5: it creates two stages for single a stage task (not need to regrasp the object).
white_cube = get("white cube")
black_cube = get("black cube")
stage(
    sample("gripper", lift(white_cube)), optimize(
        require(gt(y(com(white_cube)), y(max(pcd(black_cube)))) keep(touch(white_cube)), keep(no_break(white_cube))
) stage(
    sample("gripper", gt(x(com(white_cube)), x(com(black_cube))), optimize(
        require(l2(com(white_cube), com(goal("white cube"))),
        keep(touch(white_cube)), keep(no_break(white_cube))
)
GPT4
white_cube = get("white cube")
black_cube = get("black cube")
stage(
    sample("gripper", grasp(white_cube)), optimize(
        require(gt(y(com(white_cube)), y(max(pcd(black_cube))), end = 0.5),
        require(l2(com(white_cube), com(goal("white cube))), end = 1),
        keep(touch(white_cube))
    )
)
Test case 3 Put the knife above the white object. Cut the knife down to match the target shape of the white object. Ensure that the knife does not rotate.
GPT3.5: It uses z instead y to extract the object's height while the latter is true. It also misinterprets the meaning to deform the knife into the goal.
white_obj = get("white")
stage(
    sample("knife", gt(z(com("knife")), z(max(pcd(white_obj))))),
    optimize(
        require(l2(com("knife"), com(white_obj)),
        require(similar("knife", goal("white)))
    ), 
    keep(pitch() < 0.1, yaw() < 0.1)
)
GPT4
stage(
    sample("knife", above(get("white"))),
    optimize(
        require(similar(get("white"), goal("white"))),
        keep(touch("white"),
        keep(roll() < 0.1),
        keep(pitch() < 0.1),
        keep(yaw() < 0.1)
    )
)
Test case 4 Pick up the front part of object 'top left mat' and place it into the goal without breaking it.
GPT3.5: It adds an incorrect command to let the gripper to move away from the object in the beginning.
mat = get("top left mat")  
stage(  
    sample("gripper", grasp(frontpart(mat)), away(), optimize(  
        require(similar(mat, goal('top left mat'))),  
        keep(touch(mat)),  
        keep(no_break(mat))  
    )  
)
GPT4
stage(
    sample("gripper", grasp(frontpart(get("top left mat")))),
    optimize(
        require(l2(com(get("top left mat")), com(goal("top left mat)))
    ),
    keep(touch(frontpart(get("top left mat"))),
    keep(no_break(get("top left mat)))
)
)
C Reinforcement learning baseline
We use the SAC and PPO implementation from stable-baselines3 with hyperparameters in Table 3. Figure 8 compares the optimization performance of RL algorithms and the Adam optimizer with respect to the IOU score of the ending state with different numbers of sampled episodes for the 20 single-stage tasks of five categories.
Table 3: Parameters for Reinforcement Learning
Table 4: SAC	Table 5: PPO Parameters
gamma	0.95	update steps	2048
learning rate	3 × 10-4	learning rate	3 × 10-4
buffer size	106	entropy coef	0
target update coef	0.005	value loss coef	0.5
batch size	256	batch size	64
Figure 8: Comparison between RL and differentiable physics solver
D Evaluation metrics
We notice that in some tasks, a simple Intersection Over Union (IOU) score comparing shapes does not effectively explain the human intuition of task completion. For example, a solver might "cut" two
objects similar to the target shape without truly separating them, as the space between the two parts might only contain a minimal amount of space. In other circumstances, the solver may violate the requirements of no_break and break a rope into two or destroy a mat. To this end, we add checkers to check if two shapes are split and if a shape is not broken for success evaluation. For no-breaking constraint, we track particles and ensure their distance to their initial nearest neighbor does not change too much. On the other hand, we can check if two shapes are separated from each other by finding the connected components of particles.