SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stack the list of objects_to_stack into one stack without considering height limit. The first object (which is the object at the bottom of the stack) will also get moved and placed somewhere on the table.
Parameters: objects_to_stack (list): a list of strings, where each defines the object by its color and type.
stack_with_height_limit(objectst_to_stack，height_limit)
Stack the list of objects_to_stack into potentially multiple stacks, and each stack has a maximum height based on height_limit. The first object (which is the object at the bottom of the stack) will also get moved and placed somewhere on the table.
Parameters:   
objects_to_stack (list): a list of strings, where each string   
defines the object by its color and type.   
height_limit (int): an integer representing the maximum height   
for each stack.   
""   
\*env_utils Specifications:   
get_obj_names()   
""   
Return: a list of objects in the environment   
""   
get_all_obj_names_that_MATCH_type(type_name, objects_list)   
""   
Return: a list of objects in the environment that match the type_name   
""   
determine_finalStacking_order(objectsto_enforce_order,   
objects Without_order)   
""   
Return: a sorted list of objects to stack. The first object in the list   
would be at the bottom of the stack.   
""   
parse_position(description)   
""   
You must use parse_position for relative position (e.g. "left of yellow block") - corner position (e.g. "top left corner") - edge position (e.g. "top edge")   
Return: a list [x, y] that represents the position described by the description.   
""   
You must follow these rules when you are generating Python code. \* You MUST ONLY use Python library functions imported above. You MUST follow the docstrings and specification for these functions. \* You must follow the instructions provided by the user. You   
CANNOT add additional steps, conditionals, or loops that are not in the instruction.   
examples:   
1 objects  $=$  ['orange block', 'yellow cylinder', 'green cylinder']   
"" Place the green cylinder to the left of the yellow cylinder.   
""   
... <end_of_example_user_query>   
# must use parse position to get relative position location_pos = parse_position('left of the yellow cylinder') put_first_on_second('green cylinder', location_pos)   
- 1 objects  $=$  ['orange block', 'purple cylinder']
<end_of_example_user_query>
    position_relation = np.random.choice(ALL_POSITION_RELATION_LIST) # must use parse position to get relative position
    location_pos = parse_position(f{'position_relation} the orange block')
    put_first_on_second('purple cylinder', location_pos)
...
- ! ...
objects=['blue block', 'brown block', 'orange cylinder', 'pink cylinder', 'purple cylinder']
    Place the blue block at any random corner.
    ...
    ...
    <end_of_example_user_query>
    corner_name = np.random.choice(ALL_CORNERS_LIST) # must use parse position to get corner position
    corner_pos = parse_position(corner_name)
    put_first_on_second('blue block', corner_pos)
...
- ! ...
objects=['red block', 'blue block', 'cyan block', 'yellow block',
    'orange cylinder']
    ...
1. place the orange cylinder on the table
2. place the yellow block on the table
3. place the red block on the table
4. place the cyan block on the blue block
    ...
    ...
    <end_of_example_user_query>
    items_to_place_first_in_order = ['orange cylinder', 'yellow block',
    'red block']
    for item in items_to_place_first_in_order:
        put_first_on_second(item, 'table')
        put_first_on_second('cyan block', 'blue block')
...
- ! ...
objects=['orange block', 'blue block', 'brown block', 'yellow block',
    'gray block', 'pink block', 'cyan cylinder', 'blue cylinder']
    ...
    Stack all blocks. However, the maximum height of a stack is 3.
    ...
    ...
    <end_of_example_user_query>
    block_names = get_all_obj_names_that-match_type(type_name="block",
                    objects_list=get_obj_names())
    stack_with_height_limit(objs_to_stack=block_names,
                    height_limit=3)
...
- ! ...
objects=['yellow block', 'cyan block', 'red block', 'blue block',
    'green cylinder', 'red cylinder', 'blue cylinder']
Stack all blocks into one stack. Also make sure that the stacking enforces the bottom to top order between these objects: ['red block', 'yellow block', 'cyan block'].  
```
```
...
<end_of_example_user_query>
...
# Based on the instruction, first define the blocks that have strict ordering
block_names_with_strick_order_from_bottom_to_top = ['red block', 'yellow block', 'cyan block']
# Then, find the rest of the blocks that don't have strict order
block_names = get_all_obj_names-that-match_type(type_name="block",
" objects_list=get_obj_names())
block_names Without_order = []
for block in block_names:
    if block not in
block_names_with_strick_order_from_bottom_to_top:
        block_names Without_order.append(block)
# Call helper function to determine the final stacking order before stacking the blocks
stack_order_from_bottom_to_top = determine_finalStacking_order(
objects_to_enforce_order=
block_names_with_strick_order_from_bottom_to_top,
objectsWithout_order=block_names Without_order)
stackWithout_height_limit(objs_to_stack=
stack_order_from_bottom_to_top)
...
-!
objects=['blue block', 'yellow block', 'brown block', 'orange block', 'pink cylinder', 'brown cylinder', 'orange cylinder',
...
"..." 
Stack all objects into two stacks (where each stack has maximum height of 4). It doesn't matter whether the objects are categorized.
...
...
<end_of_example_user_query>
...
object_names = get_obj_names()
# split the objects into 2 stacks.
stack_1 = object_names[:4]
stack_2 = object_names[4:] 
stackWithout_height_limit(objs_to_stack=stack_1)
stackWithout_height_limit(objs_to_stack=stack_2)
18.2 Robotouille Task Prompts
18.2.1 Recursive Summarization
recursive_summarization: main: | You are a helpful summarizer that recursively summarizes a trajectory into a more and more compacted form. <end_of_system_message> You are given a trajectory. Your goal is to summarize the trajectory into a more compacted form and then determine whether the state trajectory is sufficiently summarized. You must respond using the following form (you must generate [[end of response]] at the very end):
[[Is the new trajectory sufficiently summarized? (yes/no):]] You should only respond using either "yes" or "no", and nothing else.
In summary, you must follow these rules when you are summarizing a trajectory.
* You must slowly summarize the trajectory from one type to another following this order: a state trajectory > a low-level action trajectory --> a high-level subtask trajectory.
* You cannot skip a type (e.g. you cannot directly summarize a low-level action trajectory into a high-level subtask trajectory).
* The low-level actions must be one of the following: "move from location1 to location2", "pick up item1", "place down item1 at location1 ", "stack item1 on top of item2", "unstack item1 from item 2", "cook item1", "cut item1". You should never define new low-level actions.
* A high-level subtask trajectory is represented as an unordered list. Each line in the unordered list should only contain one high-level subtask. This high-level subtask should refer to one continuous section of the states. For example, you cannot say "at states 1-5, and states 10-15, the robot did". There can only be one interval of states.
* The high-level subtask must be one of the following: "cook [ITEM]", "cut [ITEM]", "stack [ITEM] on top of [ITEM]", and "unstack [ITEM] from [ITEM]". [ITEM] must be one of the following: "patty", "lettuce", "top bun", "bottom bun", "cheese", "tomato".
* For the high-level subtask, you cannot define any other subtasks that are not: cook, cut, stack, or unstack. You must use choose from the list above.
Cook a patty and cut a lettuce.
'robot1' is at 'table6'
'robot1' is not at 'table7'
'lettuce6' is not at 'table6'
'robot1' is holding 'lettuce6'
'robot1' is at 'cutting_board6'
'robot1' is not at 'table6'
'lettuce6' is at 'cutting_board6'
'robot1' is not holding 'lettuce6'
'lettuce6' is cut
'robot1' is not at 'cutting_board6'
'robot1' is at 'table7'
State 10:
'patty6' is not at 'table7'
'robot1' is holding 'patty6'
State 11:
'robot1' is at 'stove6'
State 12:
'patty6' is at 'stove6'
'robot1' is not holding 'patty6'
State 13:
State 14:
State 15:
State 16:
'patty6' is cooked
The input trajectory contains state predicates because the trajectory talks about the status of the robot ('robot1' is at ' table6') and the status of the objects ('lettuce6' is not at ' table6').
Low-level actions are not compacted enough yet because low-level actions can still be combined into high-level subtasks.
The new trajectory will NOT be sufficiently summarized.
[[Is the new trajectory sufficiently summarized? (yes/no):]] no
* In [Scenario 2], at state 2, the robot moved from 'table7' to 'table 6'.
* At state 3, the robot picked up 'lettuce6'.
* At state 4, the robot moved from 'table6' to 'cutting_board6'.
* At state 5, the robot placed 'lettuce6' at location 'cutting_board6'.
* At state 6-8, the robot had cut 'lettuce6'.
* At state 9, the robot moved from 'cutting_board6' to 'table7'.
* At state 10, the robot picked up 'patty6'.
* At state 11, the robot moved from 'table7' to 'stove6'.
* At state 12, the robot placed 'patty6' at 'stove6'.
* At state 13-16, the robot had cooked 'patty6'
- |  
[[Input Trajectory:]]
* In [Scenario 2], at state 2, the robot moved from 'table5' to 'table9'.
* At state 3, the robot picked up 'patty9'.
* At state 4, the robot moved from 'table9' to 'stove5'.
* At state 5, the robot placed 'patty9' on 'stove5'.
* At state 6-9, the robot had cooked 'patty9'.
* At state 10, the robot moved from 'stove5' to 'table5'.
* At state 11, the robot picked up 'patty3
* At state 12, the robot moved from 'table5' to 'stove8'.
* At state 13, the robot placed 'patty3' on 'stove8'
* At state 14-17, the robot had cooked 'patty3'.
* In [Scenario 2], at state 2-9, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty9' (state 2-3)
2. moving to place 'patty9' on 'stove5' (state 4-5) 3. cooking patty9' until it is cooked (state 6-9)
* At state 10-17, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty3' (state 10-11) 2. moving to place 'patty3' on 'stove8' (state 12-13) 3. cooking 'patty3' until it is cooked (state 14-17)
Cut a lettuce before stacking it on top of a bottom bun. Then stack a top bun on top of the lettuce.
'lettuce1' is not at 'table6'
'robot1' is holding 'lettuce1'
'robot1' is at 'cutting_board1'
'robot1' is not holding 'lettuce1'
'lettuce1' is at 'cutting_board1'
'lettuce1' is cut
'lettuce1' is not at 'cutting_board1'
'robot1' is not at 'cutting_board1'
'robot1' is at 'table2'
'lettuce1' is at 'table2'
'lettuce1' is on top of 'bottom_bun1'
'robot1' is not at 'table2'
'robot1' is at 'table5'
'top_bun2' is not at 'table5'
'robot1' is holding 'top_bun2'
'robot1' is not at 'table5'
top_bun2' is at 'table2'
'top_bun2' is on top of 'lettuce1'
'robot1' is not holding 'top_bun2'
The input trajectory contains state predicates because the trajectory talks about the status of the robot ('robot1' is holding 'lettuce1') and the status of the objects ('lettuce1' is not at 'table6').
* In [Scenario 1], at state 2, the robot picked up 'lettuce1'.
* At state 3, the robot moved from 'table6' to 'cutting_board1'.
* At state 4, the robot placed 'lettuce1' at location '
cutting_board1'
* At state 5-7, the robot had cut 'lettuce1'.
* At state 8, the robot picked up 'lettuce1'.
* At state 9, the robot moved from 'cutting_board1' to 'table2'.
* At state 10, the robot placed 'lettuce1' on top of 'tTom_bun1' at location 'table2'.
* At state 11, the robot moved from 'table2' to 'table5'.
* At state 12, the robot picked up 'top_bun2'.
* At state 13, the robot moved from 'table5' to 'table2'.
* At state 14, the robot placed 'top_bun2' on top of 'lettuce1' at location 'table2'.
I [[Input Trajectory:]]
* At state 4, the robot placed 'lettuce1' at location cutting_board1'.
* At state 5-7, the robot had cut 'lettuce1'.
* At state 8, the robot picked up 'lettuce1'.
* At state 9, the robot moved from 'cutting_board1' to 'table2'.
* At state 10, the robot placed 'lettuce1' on top of 'bottom_bun1' at location 'table2'.
* At state 11, the robot moved from 'table2' to 'table5'.
* At state 12, the robot picked up 'top_bun2'.
* At state 13, the robot moved from 'table5' to 'table2'.
* At state 14, the robot placed 'top_bun2' on top of 'lettuce1' at location 'table2'.
The input trajectory contains low-level actions because the trajectory mentions "picked up", "moved", etc.
* In [Scenario 1], at state 2-7, the subtask is "cut lettuce". This subtask contains: 1. pick up 'lettuce1' (state 2) 2. moving to place 'lettuce1' on 'cutting_board1' (state 3-4) 3. cutting 'lettuce1' until it is cut (state 5-7)
* At state 8-10, the subtask is "stack lettuce on top of bottom bun". This subtask contains: 1. picking up 'lettuce1' (state 8) 2. moving to stack 'lettuce1' on 'bottom_bun1' (state 9-10)
* At state 11-14, the subtask is "stack top bun on top of lettuce". This subtask contains: 1. moving to pick up 'top_bun2' (state 11-12) 2. moving to stack 'top_bun2' on 'lettuce1' (state 13-14)
summary_2_spec:
You are a helpful assistant that analyzes the high-level subtask trajectories and summarizes them into a concise pseudocode-style task specification.
<end_of_system_message>
You are given (1) a high-level goal and (2) one or more high-level subtask trajectories where each one represents a different scenario. Your goal is to summarize these trajectories into a compact task specification, written in a pseudocode style.
* When checking for loops, you cannot compare the subtasks across multiple scenarios. Even if two scenarios have the exact same list of subtasks, there is NOT any loop. Loops can only exist within the list of subtasks for one individual scenario. Do not consider loops across multiple scenarios.
* In [Scenario 2], at state 2-8, the subtask is "cut lettuce". This subtask contains: 1. moving to pick up 'lettuce4' (state 2-3) 2. moving to place 'lettuce4' on 'cutting_board3' (state 4-5) 3. cutting 'lettuce4' until it is cut (state 6-8)
* At state 9-16, the subtask is "cook Patty". This subtask contains: 1. moving to picking up 'patty2' (state 9-10) 2. moving to place 'patty2' at location 'stove8' (state 11-12) 3. cooking 'patty2' until it is cooked (state 13-16)
* In [Scenario 2], at state 2-8, the subtask is "cut lettuce". This subtask contains: 1. moving to pick up 'lettuce6' (state 2-3) 2. moving to place 'lettuce6' on 'cutting_board6' (state 4-5) 3. cutting 'lettuce6' until it is cut (state 6-8)
* At state 9-16, the subtask is "cook patty". This subtask contains: 1. moving to picking up 'patty6' (state 9-10) 2. moving to place 'patty6' at location 'stove6' (state 11-12) 3. cooking 'patty6' until it is cooked (state 13-16)
* In [Scenario 1]'s unique kitchen environment, the subtasks were executed in this order: ['cut lettuce', 'cook Patty'].
* In [Scenario 2]'s unique kitchen environment, the subtasks were executed in this order: ['cut lettuce', 'cook Patty'].
* Both scenarios are executing the subtasks in the same order.
* There is no repetition or loop in the subtask ordering for any individual scenario.
Cook two patties.
* In [Scenario 1], at state 2-9, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty4' (state 2-3) 2. moving to place 'patty4' on 'stove10' (state 4-5) 3. cooking 'patty4' until it is cooked (state 6-9)
* At state 10-17, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty6' (state 10-11) 2. moving to
place 'patty6' on 'stove11' (state 12-13) 3. cooking 'patty6' until it is cooked (state 14-17)
* In [Scenario 2], at state 2-9, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty9' (state 2-3) 2. moving to place 'patty9' on 'stove5' (state 4-5) 3. cooking 'patty9' until it is cooked (state 6-9)
* In [Scenario 1]'s unique kitchen environment, the subtasks were executed in this order: ['cook patty', 'cook patty'].
* In [Scenario 2]'s unique kitchen environment, the subtasks were executed in this order: ['cook patty', 'cook patty'].
* In both scenarios, we see that the subset ['cook patty'] got repeated 2 times consecutively, so we can use a for-loop in our specification.
For two patties, do:
* In [Scenario 2], at state 2-7, the subtask is "cut lettuce". This subtask contains: 1. pick up 'lettuce4' (state 2) 2. moving to place 'lettuce4' on 'cutting_board2' (state 3-4) 3. cutting 'lettuce4' until it is cut (state 5-7)
* At state 8-10, the subtask is "stack lettuce on top of bottom bun". This subtask contains: 1. picking up 'lettuce4' (state 8) 2. moving to stack 'lettuce4' on 'bottom_bun3' (state 9-10)
* At state 11-14, the subtask is "place top bun on top of lettuce". This subtask contains: 1. moving to pick up 'top_bun1' (state 11-12) 2. moving to place 'top_bun1' on top of 'lettuce4' (state 13-14)
* In [Scenario 1]'s unique kitchen environment, the subtasks were executed in this order: ['cut lettuce', 'stack lettuce on top of bottom bun', 'stack top bun on top of lettuce'].
* In [Scenario 2]'s unique kitchen environment, the subtasks were executed in this order: ['cut lettuce', 'stack lettuce on top of bottom bun', 'place top bun on top of lettuce'].
* 'stack lettuce on top of bottom bun' and 'place top bun on top of lettuce' are essentially the same subtask.
Stack the top bun on top of the lettuce.
18.2.2 Recursive Expansion
spec_2_highlevelcode:   
main: | You are a Python code generator for robotics. The users will first provide the imported Python modules. Then, for each code they want you to generate, they provide the requirements and pseudocode in Python docstrings. <end_of_system_message> You need to write kitchen robot control scripts in Python code. The Python code should be general and applicable to different kitchen environments.
Python kitchen robot control script from env_utils import get_all_obj_names_that-match_type, get_all_location_names那個 match_type, isCut, is_cooked
\*env_utils Specifications: get_all_obj_names_that_MATCH_type(obj_type)   
""   
Parameters: obj_type (str): A string that has to be one of 'patty', 'lettuce', 'bottom bun' ,top bun'.   
Return: (list) a list of string of objects that match the obj_type   
""   
get_all_location_names那個 match_type location_type)   
""   
Parameters:
location_type(str): A string that has to be one of 'table', 'cutting_board', 'stove'.  
Return:  
(list) a list of string of locations that match the location_type  
"..."  
is_cut(obj)  
"..."  
Parameters:  
obj(str): A string that has the type of a cuttable object (must be 'lettuce') and an id. For example: 'lettuce3', 'lettuce1'  
Return:  
boolean true if obj is cut  
"..."  
is_cooked(obj)  
"..."  
Parameters:  
obj(str): A string that has the type of a cookable object (must be 'patty') and an id. For example: 'patty1', 'patty2'  
Return:  
boolean true if obj is cooked  
"..."
You must follow these rules when you are generating Python code. * You can use Python library functions imported above. You MUST follow the docstrings and specification for these functions.
* You can also call new functions that are not yet defined. However, these functions must be completing general, high-level subtasks (cook, cut, place on top, unstack). These functions cannot be doing low-level actions.
* You must follow the pseudocode provided by the user. You CANNOT add additional steps, conditionals, or loops that are not in the pseudocode.
- !  
...  
"""  
Cook a patty and cut a lettuce.  
Specifically:  
Get a list of all the lettuces in the kitchen.  
Get a list of all the patties in the kitchen.  
Decide a lettuce to use.  
Get a list of all the available cutting boards in the kitchen.  
Decide a cutting board to go to.  
Cut that lettuce at that cutting board.
Decide a patty to use.   
Get a list of all the available stoves in the kitchen.   
Decide a stove to go to.   
Cook that patty at that stove.   
""""   
...   
<end_of_example_user_query>
Get a list of all the lettuces in the kitchen.  
lettuces = get_all_obj_names_that_MATCH_type('lettuce')  
# Get a list of all the patties in the kitchen.  
patties = get_all_obj_names_that_MATCH_type('patty')
# Decide a lettuce to use. lettuce_to_use = lettuces[0] # Get a list of all the available cutting boards in the kitchen.
cutting Boards  $\equiv$  get_all_location_names_that_match_type(' cutting_board') # Decide a cutting board to go to. cutting_board_toCut_at  $=$  cutting Boards[0] # Cut that lettuce at that cutting board. cut_object_at_location(obj  $\equiv$  lettuce_to_use, location= cutting_board_toCut_at) # Decide a patty to use. patty_to_use  $=$  patties[O] # Get a list of all the available stoves in the kitchen. stoves  $=$  get_all_location_names_that_MATCH_type('stove') # Decide a stove to go to. stove_to_cook_at  $=$  stoves[O] # Cook that patty at that stove. cook_object_at_location(obj  $\equiv$  patty_to_use, location= stove_to_cook_at) -! .. Cook two patties. Specifically: Get a list of all the patties in the kitchen. Decide a patty to use. Get a list of all the available stoves in the kitchen. Decide a stove to go to. Cook that patty at that stove. Repeat the above steps for the second patty. "" ... <end_of_example_user_query> # Get a list of all the patties in the kitchen. patties  $=$  get_all_obj_names_that-match_type('patty') # For two patties, do for i in range(2): # Decide a patty to use. patty_to_use  $=$  patties[i] # Get a list of all the available stoves in the kitchen. stoves  $=$  get_all_location_names那個 match_type('stove') # Decide a stove to go to. stove_to_cook_at  $=$  stoves[i] # Cook that patty at that stove. cook_object_at_location(obj  $\equiv$  patty_to_use, location= stove_to_cook_at) -! .. """ Cut a lettuce before stacking it on top of a bottom bun. Then stack a top bun on top of the lettuce. Specifically: Get a list of all the lettuces in the kitchen. Get a list of all the bottom buns in the kitchen. Get a list of all the top buns in the kitchen. Decide a lettuce to use. Get a list of all the available cutting boards in the kitchen. Decide a cutting board to go to.
Cut that lettuce at that cutting board.   
Decide a bottom bun to use.   
Stack the lettuce on top of the bottom bun.   
Decide a top bun to use.   
Stack the top bun on top of the lettuce.   
````   
...   
<end_of_example_user_query>   
Get a list of all the lettuces in the kitchen.   
lettuces  $=$  get_all_obj_names_that-match_type('lettuce')   
# Get a list of all the bottom buns in the kitchen.   
bottom_buns  $=$  get_all_obj_names那個 match_type('bottom_bun')   
# Get a list of all the top buns in the kitchen.   
top_buns  $=$  get_all_obj_names那個 match_type('top_bun')   
# Decide a lettuce to use.   
lettuce_to_use  $=$  lettuces [0]   
# Get a list of all the available cutting boards in the kitchen.   
cutting Boards  $=$  get_all_location_names那個 match_type(' cutting_board')   
# Decide a cutting board to go to.   
cutting_board_to-cut_at  $=$  cutting Boards [0]   
# Cut that lettuce at that cutting board.   
cut_object_at_location(obj=lettuce_to_use, location= cutting_board_to-cut_at)   
# Decide a bottom bun to use.   
bottom_bun_to_use  $=$  bottom_buns [0]   
# Stack the lettuce on top of the bottom bun.   
# obj1 should be the lettuce, obj2 should be the bottom bun. stack_obj1_on_obj2(obj1=lettuce_to_use, obj2=bottom_bun_to_use)   
# Decide a top bun to use.   
top_bun_to_use  $=$  top_buns [0]   
# Stack that top bun on top of the lettuce.   
# obj1 should be the top bun, obj2 should be the lettuce. stack_obj1_on_obj2(obj1=top_bun_to_use, obj2=lettuce_to_use)
Step 2: Define composite actions Given a function header, the LLM outputs code that may contain undefined functions.