SlowGuess/NeurIPS_2023_OCR huggingface dataset:
* At state 3-4, the robot moved mezzaluna_1 from sink_1 to sink_2.
* At state 4-5, the robot moved peeler:potato_1 from sink_1 to sink_2.
* At state 5-6, the robot picked up peeler:potato_1.
* At state 6-7, the robot soaped peeler:potato_1 until it became soapy
* At state 7-8, the robot moved peeler:potato_1 from sink_1 to sink_2.
* At state 8-9, the robot moved knife_1 from sink_1 to sink_2.
* At state 9-10, the robot picked up knife_1
* At state 10-11, the robot soaped knife_1 until it became soapy.
* At state 11-12, the robot moved knife_1 from sink_1 to sink_2.
* At state 12-13, the robot moved board:cutting_1 from sink_1 to sink_2.
* At state 13-14, the robot picked up board:cutting_1.
* At state 14-15, the robot soaped board:cutting_1 until it became soapy.
* At state 15-16, the robot moved board:cutting_1 from sink_1 to sink_2.
* At state 16-17, the robot turned on tap_1.
* At state 17-18, the robot picked up board:cutting_1.
* At state 18-19, the robot rinsed board:cutting_1 until it became clean.
* At state 19-20, the robot placed board:cutting_1 in dryingrack_1.
* At state 20-21, the robot picked up mezzaluna_1
* At state 21-22, the robot rinsed mezzaluna_1 until it became clean.
* At state 22-23, the robot placed mezzaluna_1 in dryingrack_1.
* At state 23-24, the robot picked up peeler:potato_1.
* At state 24-25, the robot rinsed peeler:potato_1 until it became clean.
* At state 25-26, the robot placed peeler:potato_1 in dryingrack_1.
* At state 26-27, the robot picked up knife_1.
* At state 27-28, the robot rinsed knife_1 until it became clean.
* At state 28-29, the robot placed knife_1 in dryingrack_1.
* At state 29-30, the robot turned off tap_1.
========summarizingdemo \(= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
[[Reasoning:]]
The input trajectory contains low-level actions because the trajectory mentions "picked up", "moved", "soaped", "rinsed", etc.
I will summarize the low-level action trajectory into high-level subtasks.
High-level subtasks are compacted enough because they cannot be combined together without losing important information.
The new trajectory will be sufficiently summarized.
[[Is the new trajectory sufficiently summarized? (yes/no):]] yes
[[Summarized trajectory:]]
* In [Scenario 1], at state 1-3, the robot brought mezzaluna_1 from sink_1 to sink_2.
* At state 3-4, the robot brought peeler:potato_1 from sink_1 to sink_2.
* At state 4-5, the robot picked up peeler:potato_1.
* At state 5-6, the robot soaped peeler:potato_1 until it became soapy
* At state 6-7, the robot moved peeler:potato_1 from sink_1 to sink_2.
* At state 7-8, the robot moved knife_1 from sink_1 to sink_2.
* At state 8-9, the robot picked up knife_1.
* At state 9-10, the robot soaped knife_1 until it became soapy.
* At state 10-11, the robot moved knife_1 from sink_1 to sink_2.
* At state 11-12, the robot moved board:cutting_1 from sink_1 to sink_2.
* At state 12-13, the robot picked up board:cutting_1.
* At state 13-14, the robot soaped board:cutting_1 until it became soapy.
* At state 14-15, the robot moved board:cutting_1 from sink_1 to sink_2.
* At state 15-16, the robot turned on tap_1.
* At state 16-19, the robot rinsed board:cutting_1, mezzaluna_1, peeler:potato_1, and knife_1 until they became clean.
* At state 19-20, the robot placed board:cutting_1 in dryingrack_1.
* At state 20-21, the robot picked up mezzaluna_1.
* At state 21-22, the robot rinsed mezzaluna_1 until it became clean.
* At state 22-23, the robot placed mezzaluna_1 in dryingrack_1.
* At state 23-24, the robot picked up peeler:potato_1.
* At state 24-25, the robot rinsed peeler:potato_1 until it became clean.
* At state 25-26, the robot placed peeler:potato_1 in dryingrack_1.
* At state 26-27, the robot picked up knife_1.
* At state 27-28, the robot rinsed knife_1 until it became clean.
* At state 28-29, the robot placed knife_1 in dryingrack_1.
* At state 29-30, the robot turned off tap_1.
============== summary -> spec==============
* There are 4 objects that need to be washed: [mezzaluna_1, peeler: potato_1, knife_1, board:cutting_1]
* The list of high-level actions that happened in order is: [move object from A to B, pick up object, soap object, move object from A to B, turn on tap, rinse objects, place object in drying rack, turn off tap]
* The actions [pick up, soap, move object from A to B] are repeated for each object, so we can use a loop.
* Rinsing involves the use of tap water, which is why it is turned on at some point before the rinse cycle, and turned off after. [[Task Specification:]]
Wash objects in sink. All clean objects must be placed in drying rack. Specifically -
Get a list of all objects to wash  
For each object in all objects:  
    Move object from sink_1 to sink_2  
    Pick up object  
    Soap object  
    Move object from sink_1 to sink_2  
Turn on tap_1  
For each object in all objects:  
    Rinse object  
    Place object in dryingrack_1  
Turn off tap_1
17.3.3 Output Code
This is the final output of Demo2Code, which is the task code to solve this problem. The LLM captures that the objects areSOAPed first, then rinsed one by one after. The code also illustrates the LLM's ability to write complex code that has control flows (e.g. for loop and if-else statements.)
objects = get_all Objects()
for object in objects:
    bring_objects_to_location([object], "sink_2")
    pick_up(object)
    soap(object)
    bring_objects_to_location([object], "sink_1")
    turn_on("tap_1")
for object in objects:
    rinse(object)
    place(object, "dryingrack_1")
turn_off("tap_1")
18 Prompts
All prompts are inyaml file representation. For the most updated prompt, please refer to our code base: https://portal-cornell.github.io/demo2code/.
18.1 Tabletop Manipulation Task Prompts
18.1.1 Recursive Summarization
recursive_summarization: main: | You are a helpful summarizer that recursively summarizes a trajectory into a more and more compacted form. <end_of_system_message> You are given a trajectory. Your goal is to summarize the trajectory into a more compacted form and then determine whether the state trajectory is sufficiently summarized.
You must respond using the following form (you must generate [[end of response]] at the very end):
You should first identify what type of trajectory this is. Then, you should determine what type you will summarize the trajectory into. Finally, you should determine whether the new type of trajectory is sufficiently summarized or not.
[[Is the new trajectory sufficiently summarized? (yes/no):]]
You should only respond using either "yes" or "no", and nothing else.
You should actually summarize the input trajectory and output it here.
[[end of response]]
You must follow these rules when you are summarizing a trajectory. Rules:
* You must slowly summarize the trajectory from one type to another following this order: a state trajectory > a low-level action trajectory.
* A low-level action trajectory is represented as an unordered list. Each line in the unordered list should only contain one low-level action.
* The low-level actions must be one of the following: "move <item1>", "picked up and placed <item1> <relative_position Relation> <item2>", "picked up and placed <item1> at <loc1>", "picked up and placed <item1> on top of 'table' ", "picked up and placed <item1>
on top of <item2>". You should never define new low-level actions. examples:
- |  
[[Input Trajectory:]]  
[Scenario 1]  
Place the green cylinder next to the yellow cylinder.
'yellow cylinder' is to the right of 'green cylinder'
'green cylinder' has moved
'green cylinder' is to the left of 'yellow cylinder'
<end_of_example_user_query>
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('yellow cylinder' is to the right of 'green cylinder').
I will summarize the state trajectory into low-level actions. The low-level actions are the most compact form, so I should stop summarizing after this.
* In [Scenario 1], at state 2, the robot picked up and placed 'green cylinder' to the left of 'yellow cylinder'.
[[Input Trajectory:]]
[Scenario 1]
Place the purple cylinder next to the orange block.
'orange block' is in front of 'purple cylinder'
'purple cylinder' is at 'top edge'
'purple cylinder' is behind 'orange block'
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('orange block' is in front of 'purple cylinder').
* In [Scenario 1], at state 2, the robot picked up and placed 'purple cylinder' behind 'orange block'.
Place the blue block at the corner.
'blue block' has moved
'blue block' is at 'top right corner'
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('blue block' has moved).
* In [Scenario 1], at state 2, the robot picked up and placed 'blue block' at 'top right corner'.
Place the cyan block on the blue block.
'orange cylinder' is on top of 'table'
'orange cylinder' has moved
'orange cylinder' is not on top of 'red block'
'orange cylinder' is not on top of 'cyan block'
'orange cylinder' is not on top of 'yellow block'
'yellow block' is on top of 'table'
'yellow block' has moved
'yellow block' is not on top of 'red block'
'yellow block' is not on top of 'cyan block'
'red block' is on top of 'table'
'red block' has moved
'red block' is not on top of 'cyan block'
'red block' is to the right of 'orange cylinder'
'orange cylinder' is to the left of 'red block'
'cyan block' is not on top of 'table'
'cyan block' has moved
'cyan block' is on top of 'blue block'
<end_of_example_user_query>
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('orange cylinder' is on top of 'table').
* In [Scenario 1], at state 2, the robot picked up and placed the 'orange cylinder' on top of 'table'.
* At state 3, the robot picked up and placed 'yellow block' on top of 'table'.
* At state 4, the robot picked up and placed 'red block' on top of 'table'.
* At state 5, the robot picked up and placed 'cyan block' on top of 'blue block'.
Stack all blocks, where enforcing the maximum stack height if there is a requirement.
'orange block' has moved
'orange block' is not behind 'cyan cylinder'
'cyan cylinder' is not in front of 'orange block'
'blue block' is not on top of 'table'
'blue block' has moved
'blue block' is on top of 'orange block'
State 4:
'brown block' is not on top of 'table'
'brown block' has moved
'brown block' is on top of 'orange block'
'brown block' is on top of 'blue block'
'yellow block' has moved
'yellow block' is behind 'cyan cylinder'
'cyan cylinder' is in front of 'yellow block'
State 6:
'gray block' is not on top of 'table'
'gray block' has moved
'gray block' is on top of 'yellow block'
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('orange block' has moved).
* In [Scenario 1], at state 2, the robot moved 'orange block'.
* At state 3, the robot picked up and placed 'blue block' on top of 'orange block'.
* At state 4, the robot picked up and placed 'brown block' on top of 'blue block'.
* At state 5, the robot moved 'yellow block'.
* At state 6, the robot picked up and placed 'gray block' on top of 'yellow block'.
Stack all blocks into one stack, while enforcing the order between blocks if there is a requirement.
'red block' is not on top of 'table'
'red block' has moved
'red block' is on top of 'blue block'
'red block' is not to the right of 'red cylinder'
'red cylinder' is not to the left of 'red block'
'yellow block' is not on top of 'table'
'yellow block' is on top of 'red block'
'yellow block' is on top of 'blue block'
'cyan block' is not on top of 'table'
'cyan block' has moved
'cyan block' is on top of 'yellow block'
'cyan block' is on top of 'red block'
'cyan block' is on top of 'blue block'
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('pink block' has moved).
* In [Scenario 2], at state 2, the robot moved 'blue block'.
* At state 3, the robot picked up and placed 'red block' on top of 'blue block'.
* At state 4, the robot picked up and placed 'yellow block' on top of 'red block'.
* At state 5, the robot picked up and placed 'cyan block' on top of 'yellow block'.
Stack all objects into two stacks, and objects may need to be categorized in the stacks.
'brown block' is to the left of 'brown cylinder'
'brown cylinder' has moved
'brown cylinder' is to the right of 'brown block'
'brown cylinder' is to the left of 'blue cylinder'
'blue cylinder' is to the right of 'brown cylinder'
'orange block' is not on top of 'table'
'orange block' is on top of 'brown cylinder'
'yellow block' is on top of 'orange block'
'yellow block' is on top of 'brown cylinder'
'pink cylinder' is on top of 'yellow block'
'pink cylinder' is on top of 'brown cylinder'
State 7:
'brown cylinder' is not to the left of 'blue cylinder'
'blue cylinder' is on top of 'blue block'
'blue cylinder' is not to the right of 'brown cylinder'
State 8:
'orange cylinder' is not on top of 'table'
'orange cylinder' has moved
'orange cylinder' is on top of 'blue block'
'orange cylinder' is on top of 'blue cylinder'
State 9:
'brown block' is not on top of 'table'
'brown block' has moved
'brown block' is on top of 'blue block'
'brown block' is not to the left of 'brown cylinder'  
'brown block' is on top of 'orange cylinder'  
'brown block' is on top of 'blue cylinder'  
'brown cylinder' is not to the right of 'brown block'  
<end_of_example_user_query>  
[[Reasoning:]]  
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('brown block' is to the left of 'brown cylinder').  
I will summarize the state trajectory into low-level actions. The low-level actions are the most compact form, so I should stop summarizing after this.  
The new trajectory will be sufficiently summarized.  
[[Is the new trajectory sufficiently summarized? (yes/no):]]  
yes  
[[Summarized trajectory:]]  
* In [Scenario 1], at state 2, the robot moved 'brown cylinder'.  
* At state 3, the robot picked up and placed 'orange block' on top of 'brown cylinder'.  
* At state 4, the robot picked up and placed 'yellow block' on top of 'orange block'.  
* At state 5, the robot picked up and placed 'pink cylinder' on top of 'yellow block'.  
* At state 6, the robot moved 'blue block'.  
* At state 7, the robot picked up and placed 'blue cylinder' on top of 'blue block'.  
* At state 8, the robot picked up and placed 'orange cylinder' on top of 'blue cylinder'.  
* At state 9, the robot picked up and placed 'brown block' on top of 'orange cylinder'.  
[[end of response]]
summary_2_spec:  
main: |  
You are a helpful assistant who analyzes the trajectories and summarizes them into a concise pseudocode-style task specification.  
<end_of_system_message>  
You are given (1) a high-level goal and (2) one or more trajectories where each one represents a different scenario. Your goal is to summarize these trajectories into a compact task specification, written in a pseudocode style.
You must respond using the following format (you must generate [end of response] at the very end):
You should first list out the order of the high-level subtask trajectories in all scenarios. Then, you should consider if certain sections of the subtasks are repeated in the scenario and can be represented by a loop. Two scenarios having exactly the same list does not mean that there is a loop. A loop only exists among subtasks in one individual scenario.
Overall, your task specification should work for all scenarios. You should make sure that the task specification matches the subtasks ordering across all scenarios. You should also make sure that the task specification uses loops when there is any repetition.
You should first state the high-level goal. Then, you should say "Specifically:" before outputting the pseudocode-style task specification.
You must follow these rules when you are writing the task specifications.
* You must write the task specifications in pseudocode style. You should not write the task specification as a list. You cannot number any line.
- |  
    [[High-Level Goal:]]  
    Place the green cylinder next to the yellow cylinder.  
    [[Trajectories:]]  
* In [Scenario 1], at state 2, the robot picked up and placed 'green cylinder' to the left of 'yellow cylinder'.  
    <end_of_example_user_query>  
    [[Reasoning:]]  
With only 1 scenario, we assume that the 'green cylinder' must always be to the left of 'yellow cylinder'.  
    [[Task Specification:]]  
Place the green cylinder to the left of the yellow cylinder.  
    [[end of response]]  
- |  
    [[High-Level Goal:]]  
    Place the purple cylinder next to the orange block.  
    [[Trajectories:]]  
* In [Scenario 1], at state 2, the robot picked up and placed the 'purple cylinder' behind 'orange block'.  
* In [Scenario 2], at state 2, the robot picked up and placed the 'purple cylinder' in front of 'orange block'.  
    <end_of_example_user_query>  
    [[Reasoning:]]  
Because two scenarios have two different relative positions between 'purple cylinder' and 'orange block', we assume that the 'purple cylinder' can be anywhere next to 'orange block'.  
    [[Task Specification:]]  
Place the purple cylinder at any where next to the orange block.  
    [[end of response]]  
- |  
    [[High-Level Goal:]]  
    Place the blue block at the corner.  
    [[Trajectories:]]  
* In [Scenario 1], at state 2, the robot picked up and placed the 'blue block' at 'bottom right corner'.  
* In [Scenario 2], at state 2, the robot picked up and placed the 'blue block' at 'top right corner'  
    <end_of_example_user_query>  
    [[Reasoning:]]  
Because two scenarios have two corners ('bottom right corner', 'top right corner'), we assume that the 'blue block' can be at any random corner.  
    [[Task Specification:]]  
Place the blue block at any random corner.  
    [[end of response]]  
- |  
    [[High-Level Goal:]]  
    Place the cyan block on the blue block.  
    [[Trajectories:]]  
* In [Scenario 1], at state 2, the robot picked up and placed the 'orange cylinder' on top of 'table'.  
* At state 3, the robot picked up and placed 'yellow block' on top of 'table'.  
* At state 4, the robot picked up and placed 'red block' on top of 'table'.  
* At state 5, the robot picked up and placed 'cyan block' on top of 'blue block'.
Although the goal is to place "cyan block" on the "blue block", the trajectories show that it needs to move other blocks ('orange cylinder', 'yellow block', 'red block') before finally placing 'cyan block' on top of 'blue block'.
1. place the orange cylinder on the table
2. place the yellow block on the table
3. place the red block on the table
4. place the cyan block on the blue block
[[High-Level Goal:]]
[[Trajectories:]]
<end_of_example_user_query>  
[[Reasoning:]]
Because the task is "Stack all blocks, where enforcing the maximum stack height...", we focus how high the stacks are.
Based on States 2-4, in the first stack, the blocks from bottom to top is ['orange block', 'blue block', 'brown block']. This is 3 blocks high.
Based on States 5-6, in the second stack, the blocks from bottom to top is ['yellow block', 'gray block']. This is 2 blocks high.
Because there are 2 stacks and the tallest stack is 3 block high, we assume that each stack needs to be at most 3 blocks high.
Stack all blocks. However, the maximum height of a stack is 3.
* In [Scenario 1], at state 2, the robot moved 'red block'.
* At state 3, the robot picked up and placed 'yellow block' on top of 'red block'.
* At state 4, the robot picked up and placed 'cyan block' on top of 'yellow block'.
* At state 5, the robot picked up and placed 'blue block' on top of 'cyan block'.
Because the task is "Stack all blocks, while enforcing the order ...", we focus on whether there is any ordering requirement.
In [Scenario 1], based on States 2-5, in the first stack, the blocks from bottom to top is ['red block', 'yellow block', 'cyan block', 'blue block'].
In [Scenario 2], based on States 2-5, in the first stack, the blocks from bottom to top is ['blue block', 'red block', 'yellow block', 'cyan block'].
Between the order of blocks in these two scenarios, the ordering that stays the same is ['red block', 'yellow block', 'cyan block'].
Stack all blocks into one stack. Also make sure that the stacking enforces the bottom to top order between these objects: ['red block', 'yellow block', 'cyan block'].
* In [Scenario 1], at state 2, the robot moved 'brown cylinder'.
* At state 3, the robot picked up and placed 'orange block' on top of 'brown cylinder'.
* At state 4, the robot picked up and placed 'yellow block' on top of 'orange block'.
* At state 5, the robot picked up and placed 'pink cylinder' on top of 'yellow block'.
* At state 6, the robot moved 'blue block'.
* At state 7, the robot picked up and placed 'blue cylinder' on top of 'blue block'.
* At state 8, the robot picked up and placed 'orange cylinder' on top of 'blue cylinder'.
* At state 9, the robot picked up and placed 'brown block' on top of 'orange cylinder'.
Because the task is "Stack all objects into two stacks, and objects may need to be categorized in the stacks", we focus on whether the objects are stacked by type.
Based on States 2-5, in the first stack, the blocks from bottom to top is ['brown cylinder', 'orange block', 'yellow block', 'pink cylinder'].
Based on States 6-9, in the first stack, the blocks from bottom to top is ['blue block', 'blue cylinder', 'orange cylinder', 'brown block'].
Because each stack has both blocks and cylinders, we assume that it doesn't matter whether the objects are categorized.
Stack all objects into two stacks. It doesn't matter whether the objects are categorized.
18.1.2 Recursive Expansion
spec_2_highlevelcode:  
main: |  
You are a Python code generator for robotics. The users will first provide the imported Python modules. Then, for each code they want you to generate, they provide the requirements and pseudocode in Python docstrings.  
<end_of_system_message>
You need to write robot control scripts in Python code. The Python code should be general and applicable to different environments.
Below are the imported Python libraries and functions that you can use. You CANNOT import new libraries.
Python kitchen robot control script  
import numpy as np  
from robot_utils import put_first_on_second,  
stackWithout_height_limit, stack_with_height_limit  
from env_utils import get_obj_names,  
get_all_obj_names_that_MATCH_type, determine_finalStacking_order,  
parse_position  
ALL_CORNERS_LIST = ['top left corner', 'top right corner', 'bottom left corner', 'bottom right corner']  
ALL_EDGES_LIST = ['top edge', 'bottom edge', 'left edge', 'right edge']  
ALL_POSITION_RELATION_LIST = ['left of', 'right of', 'behind', 'in front of']
Below shows the docstrings for these imported library functions that you must follow. You CANNOT add additional parameters to these functions.
* robot_utils Specifications: put_first_on_second(arg1, arg2)
You must not write things like: put_first_on_second ("red block", "left of yellow block") put_first_on_second ("red block", "top left corner") put_first_on_second ("red block", "top edge")   
You can write something like: put_first_on_second ("red block", "yellow block") put_first_on_second ("red block", "table")
Pick up an object (arg1) and put it at arg2. If arg2 is an object, arg1 will be on top of arg2. If arg2 is 'table', arg1 will be somewhere random on the table. If arg2 is a list, arg1 will be placed at location [x, y].
Parameters:  
arg1 (str): A string that defines the object by its color and type (which is either "block" or "cylinder"). For example, "red block", "orange cylinder".
arg2 (list or str): If it's a list, it needs to be a list of floats, and it represents the [x, y] position to place arg1. If it's a string, it can either be "table" or a string that defines the object by its color and type. arg2 must not be a free-from string that represents a description of a position. For example, it cannot be relative position (e.g. "left of yellow block"), or corner name (e.g. "top left corner"), or edge name (e.g. "top edge").
stack_without_height_limit(objects_to_stack)