SlowGuess/NeurIPS_2023_OCR huggingface dataset:
objects  $=$  ['pink block', 'yellow block', 'purple cylinder', 'cyan cylinder', 'pink cylinder', 'blue cylinder']   
""[Scenario 1]   
Stack all cylinders into one stack, while enforcing the order between cylinders if there is a requirement.   
State 2: 'blue cylinder' has moved   
State 3: 'purple cylinder' is not on top of table' 'purple cylinder' has moved 'purple cylinder' is on top of blue cylinder'   
State 4: 'cyan cylinder' is not on top of table' 'cyan cylinder' has moved
'cyan cylinder' is on top of 'purple cylinder'  
'cyan cylinder' is on top of 'blue cylinder'
State 5:
'pink cylinder' is not on top of 'table'
'pink cylinder' has moved
'pink cylinder' is on top of 'purple cylinder'
'pink cylinder' is on top of 'cyan cylinder'
'pink cylinder' is on top of 'blue cylinder'
[Scenario 2]
Stack all cylinders into one stack, while enforcing the order between cylinders if there is a requirement.
State 2:
'cyan cylinder' is not on top of 'table'
'cyan cylinder' has moved
'cyan cylinder' is on top of 'pink cylinder'
State 3:
'blue cylinder' is not on top of 'table'
'blue cylinder' has moved
'blue cylinder' is on top of 'cyan cylinder'
'blue cylinder' is on top of 'pink cylinder'
State 4:
'purple cylinder' is not on top of 'table'
'purple cylinder' has moved
'purple cylinder' is on top of 'cyan cylinder'
'purple cylinder' is on top of 'pink cylinder'
'purple cylinder' is on top of 'blue cylinder'
17.1.2 Intermediate output
In stage 1 recursive summarization, for each demonstration, Demo2Code summarized the trajectories into low-level actions (e.g. moved 'blue cylinder', picked up and placed 'red cylinder' on top of 'blue cylinder'). Then, because tabletop's domain has short demonstrations, the LLM determined that a low-level action trajectory was sufficiently summarized.
Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM first identified the order of the stack in each scenario. Then, it compared the two orders in the two scenarios before concluding that the order that stays the same is ['blue cylinder', 'purple cylinder']. It finally produced the correct specification that will get passed to stage 2 to produce the correct code.
Stage 1 Recursive Summary: recursive call # 1 with state-based trajectories as input
============== summarizing demo==============
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('red cylinder' is not on top of 'table').
I will summarize the state trajectory into low-level actions.
The low-level actions are the most compact form, so I should stop summarizing after this.
The new trajectory will be sufficiently summarized.
[[Is the new trajectory sufficiently summarized? (yes/no):]] yes
[[Summarized trajectory:]]
* In [Scenario 1], at state 2, the robot moved 'blue cylinder'.
* At state 3, the robot picked up and placed 'red cylinder' on top of 'blue cylinder'.
* At state 4, the robot picked up and placed 'orange cylinder' on top of 'red cylinder'.
* At state 5, the robot picked up and placed 'cyan cylinder' on top of 'orange cylinder'.
========summarizingdemo  $= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = [[\text{Reasoning} :]]$
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('orange cylinder' has moved).
I will summarize the state trajectory into low-level actions.
The low-level actions are the most compact form, so I should stop summarizing after this.
The new trajectory will be sufficiently summarized.
[[Is the new trajectory sufficiently summarized? (yes/no):]] yes
[[Summarized trajectory:]]
* In [Scenario 2], at state 2, the robot moved 'orange cylinder'.
* At state 3, the robot picked up and placed 'blue cylinder' on top of 'orange cylinder'.
* At state 4, the robot picked up and placed 'red cylinder' on top of 'blue cylinder'.
* At state 5, the robot picked up and placed 'cyan cylinder' on top of 'red cylinder'.
Summarized Demonstrations -> Task Specification
Because the task is "Stack all cylinders into one stack, while enforcing the order...", we focus on whether there is any ordering requirement.
In [Scenario 1], based on States 2-5, in the first stack, the cylinders from bottom to top is ['blue cylinder', 'red cylinder', 'orange cylinder', 'cyan cylinder'].
In [Scenario 2], based on States 2-5, in the first stack, the cylinders from bottom to top is ['orange cylinder', 'blue cylinder', 'red cylinder', 'cyan cylinder'].
Between the order of cylinders in these two scenarios, the ordering that stays the same is ['blue cylinder', 'red cylinder', 'cyan cylinder'].
[[Task Specification:]]
Stack all cylinders into one stack. Also make sure that the stacking enforces the bottom to top order between these cylinders: ['blue cylinder', 'red cylinder', 'cyan cylinder'].
17.1.3 Output Code
This is the final output of Demo2Code, which is the task code to solve this problem. The code illustrates the LLM's ability to write complex code that has control flows (e.g. for loop and if-else statements.)
Based on the instruction, first define the cylinders that have strict ordering  
cylinder_names_with_strick_order_from_bottom_to_top = ['blue cylinder', 'purple cylinder']  
# Then, find the rest of the cylinders that don't have strict order  
cylinder_names = get_all_obj_names_that_MATCH_type(type_name="cylinder", objects_list=get_obj_names())
cylinder_names Without_order = []
for cylinder in cylinder_names:  
    if cylinder not in  
        cylinder_names_with_strick_order_from_bottom_to_top:
cylinder_names Without_order.append (cylinder) # Call helper function to determine the final stacking order before stacking the cylinders stack_order_from_bottom_to_top = determine_finalStacking_order( objects_to_enforce_order= cylinder_names_with_strick_order_from_bottom_to_top, objectsWithout_order=cylinder_names Without_order) stackWithout_height_limit(objectsto_stack  $\equiv$  stack_order_from_bottom_to_top)
17.2 Robotouille Example
17.2.1 Query
The example we will consider is Make a burger stacking patty atop lettuce immediately. This is a task that contains many different actions and is successfully solved by Demo2Code, making it a comprehensive example. The query for this task consists of two demonstrations each of which has the same language instruction to make a burger. Because each demonstration has 25 states, we have placed the query at the end of the appendix in section 19.1.
17.2.2 Intermediate output
Below, in stage 1 recursive summary, for each demonstration, Demo2Code summarizes each trajectory from a state-based trajectory to a low-level action trajectory to a high-level subtask trajectory, before the LLM decides that the trajectory is sufficiently summarized.
Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM first identified the order of the subtasks in each scenario. Then, it determined that the lettuce needs to be cut and stacked on top of the bottom bun before cooking the patty and stacking the patty on top of the lettuce. It finally produced the correct specification that will get passed to stage 2 to produce the correct code.
Stage 1 Recursive Summary: recursive call # 1 with state-based trajectories as input
$= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 1$    
The input trajectory contains state predicates because the trajectory talks about the status of the robot ('robot1' is at 'table2') and the status of the objects ('lettuce1' is not at 'table2'). I will summarize the state trajectory into low-level actions. Low-level actions are not compacted enough yet because low-level actions can still be combined into high-level subtasks. The new trajectory will NOT be sufficiently summarized. [Is the new trajectory sufficiently summarized? (yes/no):]] no   
[[Summarized trajectory:]]   
\* In [Scenario 1], at state 2, the robot moved from 'table1' to ' table2'.   
\* At state 3, the robot picked up 'lettuce1'.   
\* At state 4, the robot moved from 'table2' to 'cutting_board1'. \* At state 5, the robot placed 'lettuce1' at location 'cutting_board1'.   
\* At state 6-8, the robot had cut 'lettuce1'.   
\* At state 9, the robot picked up 'lettuce1'.   
\* At state 10, the robot moved from 'cutting_board1' to 'table3'. \* At state 11, the robot placed 'lettuce1' on top of 'bottom_bun1' at location 'table3'.   
\* At state 12, the robot moved from 'table3' to 'stove1'.   
\* At state 13, the robot picked up 'patty1'.   
\* At state 14, the robot moved from 'stove1' to 'stove4'. \* At state 15, the robot placed 'patty1' at 'stove4'.   
\* At state 16-19, the robot had cooked 'patty1'.   
\* At state 20, the robot picked up 'patty1'.
* At state 21, the robot moved from 'stove4' to 'table3'.
* At state 22, the robot placed 'patty1' on top of 'lettuce1' at location 'table3'.
* At state 23, the robot moved from 'table3' to 'table4'.
* At state 24, the robot picked up 'top_bun1'.
* At state 25, the robot moved from 'table4' to 'table3'.
* At state 26, the robot placed 'top_bun1' on top of 'patty1' at location 'table3'.
========summarizingdemo \(= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the robot ('robot1' is at 'cutting_board5') and the status of the objects ('lettuce3' is not at 'cutting_board5').
I will summarize the state trajectory into low-level actions.
Low-level actions are not compacted enough yet because low-level actions can still be combined into high-level subtasks.
The new trajectory will NOT be sufficiently summarized.
[[Is the new trajectory sufficiently summarized? (yes/no):]] no
[[Summarized trajectory:]]
* In [Scenario 2], at state 2, the robot moved from 'table7' to 'cutting_board5'.
* At state 3, the robot picked up 'lettuce3'.
* At state 4, the robot moved from 'cutting_board5' to 'cutting_board3'.
* At state 5, the robot placed 'lettuce3' at location 'cutting_board3'
* At state 6-8, the robot had cut 'lettuce3'.
* At state 9, the robot picked up 'lettuce3'.
* At state 10, the robot moved from 'cutting_board3' to 'table4'.
* At state 11, the robot placed 'lettuce3' on top of 'bottom_bun3' at location 'table4'.
* At state 12, the robot moved from 'table4' to 'table7'.
* At state 13, the robot picked up 'patty3'.
* At state 14, the robot moved from 'table7' to 'stove3'.
* At state 15, the robot placed 'patty3' on 'stove3'.
* At state 16-19, the robot had cooked 'patty3'.
* At state 20, the robot picked up 'patty3'.
* At state 21, the robot moved from 'stove3' to 'table4'.
* At state 22, the robot placed 'patty3' on top of 'lettuce3' at location 'table4'.
* At state 23, the robot moved from 'table4' to 'cutting_board4'.
* At state 24, the robot picked up 'top_bun3'.
* At state 25, the robot moved from 'cutting_board4' to 'table4'.
* At state 26, the robot placed 'top_bun3' on top of 'patty3' at location 'table4'.
Stage 1 Recursive Summary: recursive call # 2 with low-level action trajectories as input
The input trajectory contains low-level actions because the trajectory mentions "moved", "picked up", etc.
I will summarize the low-level action trajectory into high-level subtasks.
High-level subtask trajectory is the most compacted form that cannot be summarized anymore.
* In [Scenario 1], at state 2-8, the subtask is "cut lettuce". This subtask contains: 1. moving from 'table1' to 'cutting_board1' (
state 2-4) 2. placing 'lettuce1' at 'cutting_board1' (state 5) 3. cutting 'lettuce1' until it is cut (state 6-8)
* At state 9-11, the subtask is "stack lettuce on top of bottom bun". This subtask contains: 1. picking up 'lettuce1' (state 9) 2. moving to stack 'lettuce1' on 'bottom_bun1' (state 10-11)
* At state 12-19, the subtask is "cook patty". This subtask contains: 1. moving from 'stove1' to 'stove4' (state 12-14) 2. placing 'patty1' at 'stove4' (state 15) 3. cooking 'patty1' until it is cooked (state 16-19)
* At state 20-22, the subtask is "stack patty on top of lettuce". This subtask contains: 1. picking up 'patty1' (state 20) 2. moving to stack 'patty1' on 'lettuce1' (state 21-22)
* At state 23-26, the subtask is "stack top bun on top of patty". This subtask contains: 1. moving from 'table3' to 'table4' (state 23) 2. picking up 'top_bun1' (state 24) 3. moving to stack 'top_bun1' on 'patty1' (state 25-26)
========summarizingdemo \(= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
* In [Scenario 2], at state 2-8, the subtask is "cut lettuce". This subtask contains: 1. moving to pick up 'lettuce3' (state 2-3) 2. moving to place 'lettuce3' on 'cutting_board3' (state 4-5) 3. cutting 'lettuce3' until it is cut (state 6-8)
* At state 9-11, the subtask is "stack lettuce on top of bottom bun". This subtask contains: 1. picking up 'lettuce3' (state 9) 2. moving to stack 'lettuce3' on 'bottom_bun3' (state 10-11)
* At state 12-19, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty3' (state 12-13) 2. moving to place 'patty3' on 'stove3' (state 14-15) 3. cooking 'patty3' until it is cooked (state 16-19)
* At state 20-22, the subtask is "stack patty on top of lettuce". This subtask contains: 1. picking up 'patty3' (state 20) 2. moving to stack 'patty3' on 'lettuce3' (state 21-22)
* At state 23-26, the subtask is "stack top bun on top of patty". This subtask contains: 1. moving to pick up 'top_bun3' (state 23-24) 2. moving to stack 'top_bun3' on 'patty3' (state 25-26)
Summarized Demonstrations  $\rightarrow$  Task Specification
* In [Scenario 1]'s unique kitchen environment, the subtasks were executed in this order: ['cut lettuce', 'stack lettuce on top of bottom bun', 'cook patty', 'stack patty on top of lettuce', 'stack top bun on top of patty'].
* In [Scenario 2]'s unique kitchen environment, the subtasks were executed in this order: ['cut lettuce', 'stack lettuce on top of bottom bun', 'cook patty', 'stack patty on top of lettuce', 'stack top bun on top of patty'].
* Both scenarios are executing the subtasks in the same order.
* There is no repetition or loop in the subtask ordering for any individual scenario.
Make a burger.
Specifically:
Get a list of all the lettuces in the kitchen.  
Get a list of all the bottom buns in the kitchen.  
Get a list of all the patties in the kitchen.  
Get a list of all the top buns in the kitchen.
Decide a lettuce to use.
Get a list of all the available cutting boards in the kitchen. Decide a cutting board to go to. Cut that lettuce at that cutting board.
Decide a bottom bun to use.
Stack the lettuce on top of the bottom bun.
Decide a patty to use.
Get a list of all the available stoves in the kitchen.
Decide a stove to go to.
Cook that patty at that stove.
Stack the patty on top of the lettuce.
Decide a top bun to use.
Stack the top bun on top of the patty.
17.2.3 Output Code
This is the final output of the LLM, which is the task code to solve this problem. The code illustrates the LLM's ability to write complex code that has control flows (e.g. for loop and if-else statements.)
Task Specification -> High-Level Code
Get a list of all the lettuces in the kitchen.
lettuces = get_all_obj_names_that_MATCH_type('lettuce')
Get a list of all the bottom buns in the kitchen.
bottom_buns = get_all_obj_names_that_MATCH_type('bottom_bun')
Get a list of all the patties in the kitchen.
patties = get_all_obj_names_that_MATCH_type('patty')
Get a list of all the top buns in the kitchen.
top_buns = get_all_obj_names_that_MATCH_type('top_bun')
lettuce_to_use = lettuces[0]
Get a list of all the available cutting boards in the kitchen.
cutting Boards = get_all_location_names_that_MATCH_type('cutting_BOARD')
Decide a cutting board to go to.
cutting_board_to_cut_at = cutting Boards[0]
Cut that lettuce at that cutting board.
cut_object_at_location(obj=lettuce_to_use, location=cutting_board_to-cut_at)
bottom_bun_to_use = bottom_buns[0]
obj1 should be the lettuce, obj2 should be the bottom bun.
stack_obj1_on_obj2(obj1=lettuce_to_use, obj2=bottom_bun_to_use)
patty_to_use = patties[0]
stoves = get_all_location_names_that_MATCH_type('stove')
stove_to_cook_at = stoves[0]
cook_object_at_location(obj = Patty_to_use, location = stove_to_cook_at)
Stack the patty on top of the lettuce.  
# obj1 should be the patty, obj2 should be the lettuce.  
stack_obj1_on_obj2(obj1 = patty_to_use, obj2 = lettuce_to_use)
# Decide a top bun to use.  
top_bun_to_use = top_buns[0]  
# Stack the top bun on top of the patty.  
# obj1 should be the top bun, obj2 should be the patty.  
stack_obj1_on_obj2(obj1=top_bun_to_use, obj2=patty_to_use)
Step 2: Define composite actions
def cook_object_at_location(obj, location):
    # To cook an object, the robot first needs to be holding obj
    if not is_holding(obj):
        # If the robot is not holding obj, there are 2 scenarios:
            # (1) if obj is in a stack, unstack obj
            # (2) else, pick up obj.
        if is_in_a_stack(obj):
            # Because obj is in a stack, robot need to move then
            unstack the obj from the obj_at_bottom first
                obj_at_bottom = get_obj_that_is_underneath(obj_at_top=obj)
            # move then unstack: first you move to the obj_at_bottom's location, then you unstack obj from obj_at_bottom
                moveThen_unstack(obj_to_unstack=obj, obj_at_bottom=
obj_at_bottom, unstack_location=get_obj_location(obj_at_bottom))
        else:
            # moveThen_pick: first you move to obj's location, then you pick obj up
                moveThen_pick(obj=obj, pick_location=get_obj_location(obj))
        # move then place: first you move to the location to cook at, then you place obj at that location
movethen_place(obj=obj, place_location=location)
# cook the object until it is cooked
cook_until_is_cooked(obj=obj)
def stack_obj1_on_obj2(obj1, obj2):
    # To stack obj1 on obj2, the robot needs to be holding obj1
    if not is_holding(obj1):
        # If the robot is not holding obj1, there are 2 scenarios:
            # (1) if obj1 is in a stack, unstack obj1
            # (2) else, pick up obj1.
            if is_in_a_stack(obj1):
                # Because obj1 is in a stack, robot need to move then
            unstack the obj from the obj_at_bottom first
                obj_at_bottom = get_obj_that_is_underneath(obj_at_top=obj1)
            # move then unstack: first you move to the obj_at_bottom's location, then you unstack obj from obj_at_bottom
                moveThen_unstack(obj_to_unstack=obj1, obj_at_bottom=
obj_at_bottom, unstack_location=get_obj_location(obj_at_bottom))
            else:
                # move THEN_pick: first you move to obj's location, then you pick obj up
                move THEN_pick(obj=obj1, pick_location=get_obj_location(obj1))
            # determine the location of obj2 to stack on
            obj2_location = get_obj_location(obj2)
            # move then stack: first you move to obj2's location, then you stack obj1 on obj2
            move THEN_stack(obj_to_stack=obj1, obj_at_bottom=obj2,
            stack_location=obj2_location)
def cut_object_at_location(obj, location):
    # To cut an object, the robot first needs to be holding obj
    if not is_holding(obj):
        # If the robot is not holding obj, there are 2 scenarios:
            # (1) if obj is in a stack, unstack obj
            # (2) else, pick up obj.
        if is_in_a_stack(obj):
            # Because obj is in a stack, robot need to move then
            unstack the obj from the obj_at_bottom first
                obj_at_bottom = get_obj_that_is_underneath(obj_at_top=obj)
            # move then unstack: first you move to the obj_at_bottom's location, then you unstack obj from obj_at_bottom
                moveThen_unstack(obj_to_unstack=obj, obj_at_bottom=
obj_at_bottom, unstack_location=get_obj_location(obj_at_bottom))
    else:
        # moveThen_pick: first you move to obj's location, then you pick obj up
        moveThen_pick(obj=obj, pick_location=get_obj_location(obj))
    # move then place: first you move to the location to cut at, then you place obj at that location
    moveThen_place(obj=obj, place_location=location)
    # cut the object until it is cut
    cut_until_isCut(obj=obj)
Step 3: Define low-level actions
def move_unstack(obj_to_unstack, obj_at_bottom, unstack_location):
    if get_curr_location() != unstack_location:
        move(get_curr_location(), unstack_location)
    unstack(obj_to_unstack, obj_at_bottom)  
def move_unstack(obj, pick_location):
    if get_curr_location() != pick_location:
        move(get_curr_location(), pick_location)
    pick_up(obj, pick_location)  
def move_until_is_cooked(obj):
    start_cooking(obj)
    while not is_cooked(obj):
        noop()
def move_until_is_stack(obj):
    while not is_stack(obj):
        cut(obj)
17.3 EPIC-Kitchens Example
17.3.1 Query
We present an example of the demonstration given as a query to the LLM. The annotations are made for video id  $P22\_07$ . This example shows the LLM generalizing to multiple new objects, new
Figure 14: Example of annotations for video P22_07 in 14 frames
locations, and a different dishwashing style. Figure 14 shows visualizations of the video with the respective state annotations. Because each demonstration has 25 states, we have placed the query at the end of the appendix in section 19.2.
17.3.2 Intermediate Output
Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM was able to identify that it should use a for-loop because [pick up, soap, move object from A to B] are repeated. It finally produced the correct specification that will get passed to stage 2 to produce the correct code.
$= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 1$    
The input trajectory contains state predicates because the trajectory talks about the status of the objects (is_in_hand('sponge_1'), dirty('mezzaluna_1'), etc.).   
I will summarize the state trajectory into low-level actions.   
Low-level actions are not compacted enough yet because low-level actions can still be combined into high-level subtasks.   
The new trajectory will NOT be sufficiently summarized.   
[[Is the new trajectory sufficiently summarized? (yes/no):]]   
no   
[[Summarized trajectory:]]   
\* In [Scenario 1], at state 1-2, the robot picked up mezzaluna_1 from sink_1.   
\* At state 2-3, the robot soaped mezzaluna_1 until it became soapy.