SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We examine all four models' outputs on the task: placing A next to B. There are 2 states per demonstration and 1 demonstration in total. The query and the Full approach's correctly generated code for a particular instance of the task are shown below:
1 objects=['yellow block', 'blue block', 'green block', 'cyan cylinder', 'gray cylinder', 'red cylinder']  
2 '''  
3 Place the gray cylinder next to the green block.  
4  
5 Initial State (State 1):  
6 'green block' is not behind 'gray cylinder'  
7 'gray cylinder' has not moved  
8 'gray cylinder' is not in front of 'green block'  
9  
10 [Scenario 1]  
11 State 2:  
12 'green block' is behind 'gray cylinder'  
13 'gray cylinder' has moved  
14 'gray cylinder' is in front of 'green block'  
15 '''  
1 say("Place the gray cylinder in front of the green block.")
2 location_pos = parse_position('in front of the green block')
3 put_first_on_second ('gray cylinder', location_pos)
For this particular example, the LLM needs to be able to identify that the robot needs to specifically place the gray cylinder in front of the green block. Only No-Cot failed to generate the correct code,
while 1-step and 2-steps were able to identify the specification and generate the code exactly the same as Full.
Why No-Cot failed?
No-Cot seemed to overfit to the prompt. For this example where the specific requirement is to place A to the right of B, it outputs "to the right of the green block." We hypothesize that because the specific requirement of placing A to the left of B appears in the prompt, the LLM just assumes that any state that differs from the example in the prompt is placing A to the right of B. Below is the code outputted by No-Cot:
say("Place the gray cylinder to the right of the green block.")
2 location_pos = parse_position('right of the green block')
3 put_first_on_second('gray cylinder', location_pos)
12.4 Qualitative example for a medium-horizon task
We examine all four models' output on the task: placing A on top of B. There are 5 states per demonstration and 1 demonstration in total. The query and the Full approach's correctly generated code for a particular instance of the task are shown below:
objects  $=$  ['orange block', 'brown block', 'red cylinder', 'purple
cylinder', 'pink cylinder', 'yellow cylinder']
3 Place the brown block on the purple cylinder.
5 Initial State (State 1):
6 'orange block' is not on top of 'table'
7 'orange block' has not moved
8 'orange block' is on top of 'brown block'
9 'orange block' is not to the right of 'yellow cylinder'
10 'brown block' is on top of 'table'
11 'brown block' has not moved
'brown block' is not on top of 'purple cylinder'
13 'red cylinder' is not on top of 'table'
14 'red cylinder' has not moved
15 'red cylinder' is on top of 'orange block'
16 'red cylinder' is on top of 'brown block'
17 'red cylinder' is on top of 'pink cylinder'
' pink cylinder ' is not on top of ' table '
19 'pink cylinder' has not moved
'pink cylinder' is on top of 'orange block'
'pink cylinder' is on top of 'brown block'
'yellow cylinder' is not to the left of 'orange block'
24 [Scenario 1]
25 State 2:
26 'red cylinder' is on top of 'table'
27 'red cylinder' has moved
28 'red cylinder' is not on top of 'orange block'
29 'red cylinder' is not on top of 'brown block'
'red cylinder' is not on top of 'pink cylinder'
31
32 State 3:
33 'pink cylinder' is on top of 'table'
'pink cylinder' has moved
'pink cylinder' is not on top of 'orange block'
36 'pink cylinder' is not on top of 'brown block'
37
38 State 4:
39 'orange block' is on top of 'table'
40 'orange block' has moved
41 'orange block' is not on top of 'brown block'
'orange block' is to the right of 'yellow cylinder'
43 'yellow cylinder' is to the left of 'orange block'
44
45 State 5:
46 'brown block' is not on top of 'table'
47 'brown block' has moved
'brown block' is on top of 'purple cylinder'
49
1 say("First, place the red cylinder, the pink cylinder, and the orange block on the table. Then, place the brown block on the purple cylinder.")
2 items_to_place_first_in_order = ['red cylinder', 'pink cylinder', 'orange block']
3 for item in items_to_place_first_in_order:
4 put_first_on_second(item, "table")
5 put_first_on_second('brown block', 'purple cylinder')
For this particular example, the LLM needs to recognize that there are 3 additional objects on top of the brown block. In order to place the brown block on top of the purple cylinder, the robot needs to first place those 3 additional objects on the table. No-Cot and 1-step approaches failed to generate the correct code, while 2-steps was able to identify the specification and generate the code exactly the same as Full.
Why No-Cot and 1-step failed?
Both No-Cot and 1-step seem to be over-reliant on the language instruction instead of reasoning through the state transitions. Specifically, examining the specification outputted by 1-step suggests that the LLM simply copied the language instruction instead of recognizing that more objects are moved than just moving the brown block to be on top of the purple cylinder.
The detailed requirement is: <Place the brown block on the purple cylinder.
Below show the identical code generated by No-Cot and 1-step.
I say("Place the brown block on the purple cylinder.")
2 put_first_on_second('brown block', 'purple cylinder')
12.5 Qualitative example for a long-horizon task
We examine all four models' output on the long-horizon task: making a burger with patty, lettuce, and tomato. This task has 71 states per demonstration and 2 demonstrations in total.
The Full pipeline's approach outputs the correct code as shown below. Next, we analyze how other models failed to produce the correct code.
1 # Get a list of all the patties in the kitchen.  
2 patties = get_all_obj_names_that_match_type('patty')  
3 # Get a list of all the lettuces in the kitchen.  
4 lettuces = get_all_obj_names那個 match type('lettuce')  
5 # Get a list of all the tomatoes in the kitchen.  
6 tomatoes = get_all_obj_names那個 match type('tomato')  
7 # Get a list of all the bottom buns in the kitchen.  
8 bottom_buns = get_all_obj_names那個 match type('bottom bun')  
9 # Get a list of all the top buns in the kitchen.  
10 top_buns = get_all_obj_names那個 match type('top bun')  
11 # Get a list of all the stoves in the kitchen.  
12 stoves = get_all_location_names那個 match type('stove')  
13 # Get a list of all the cutting boards in the kitchen.  
14 cutting Boards = get_all_location_names那個 match type('cutting_board')  
15 # Get a list of all the tables in the kitchen.  
16 tables = get_all_location_names那個 match type('table')  
17
18 # Decide a stove to use.  
19 stove_to_use = stoves[0]  
20 # Decide a cutting board to use.  
21 cutting-board_to_use = cutting Boards[0]  
22 # Decide a table to use.  
23 table_to_use = tables[0]  
24  
25 # Repeat the following twice:  
26 for i in range(2):  
27 # Decide a patty to cook.  
28 Patty_to_cook = patties[i]  
29 # Cook that patty at that stove.  
30 cook_object_at_location(obj=patty_to_cook, location=stove_to_use)  
31 # Decide a bottom bun to use.  
32 bottom_bun_to_use = bottom_buns[i]  
33 # Stack the patty on top of the bottom bun.  
34 stack_obj1_on_obj2(obj1=patty_to_cook, obj2=bottom_bun_to_use)  
35  
36 # Decide a tomato to cut.  
37 tomato_toCut = tomatoes[i]  
38 # Cut that tomato at that cutting board.  
39 cut_object_at_location(obj=tomato_toCut, location=cutting_board_to_use)  
40 # Stack the tomato on top of the patty.  
41 stack_obj1_on_obj2(obj1=tomato_toCut, obj2=patty_to_cook)  
42  
43 # Decide a lettuce to cut.  
44 lettuce_toCut = lettuces[i]  
45 # Cut that lettuce at that cutting board.  
46 cut_object_at_location(obj=lettuce_toCut, location=cutting_board_to_use)  
47 # Stack the lettuce on top of the tomato.  
48 stack_obj1_on_obj2(obj1=lettuce_toCut, obj2=tomato_toCut)  
49  
50 # Stack the top bun on top of the lettuce, tomato, and patty stack.  
51 top_bun_to_use = top_buns[i]  
52 stack_obj1_on_obj2(obj1=top_bun_to_use, obj2=lettuce_to-cut)
The main issue that No-Cot suffers from is the maximum token length issue because the prompt needs to include demonstrations as the example query and solution code as the example response. Thus, the LLM was not able to finish completely the code as seen below (line 37). In addition, although the LLM showed some ability to use the control flow, it was only creating a for loop to cook the patties but it did not use a for-loop to assemble the burger. Specifically, the LLM generated code that index to specific items: (e.g. line 23 bottom_buns [0], line 25 lettuces [0], etc.)
1 # Get a list of all the patties in the kitchen.  
2 patties = get_all_obj_names_that-match_type('patty')  
3 # Get a list of all the stoves in the kitchen.  
4 stoves = get_all_location_names那個 match_type('stove')  
5 # Decide a stove to go to.  
6 stove_to_cook_at = stoves[0]  
7  
8 # Repeat the following twice  
9 for i in range(2):  
10 # Decide a patty to cook.  
11 patty_to_cook = patties[i]  
12 # Cook that patty at that stove.  
13 cook_object_at_location(obj=patty_to_cook, location= stove_to_cook_at)  
14  
15 # Get a list of all the bottom buns in the kitchen.
16 bottom_buns = get_all_obj_names_that_MATCH_type('bottom bun')
17 # Get a list of all the lettuces in the kitchen.
18 lettuces = get_all_obj_names那個 match_type('lettuce')
19 # Get a list of all the tomatoes in the kitchen.
20 tomatoes = get_all_obj_names那個 match_type('tomato')
21
22 # Decide a bottom bun to use.
23 bottom_bun_to_use = bottom_buns[0]
24 # Decide a lettuce to use.
25 lettuce_to_use = lettuces[0]
26 # Decide a tomato to use.
27 tomato_to_use = tomatoes[0]
28
29 # Stack the patty on top of the bottom bun.
30 # obj1 should be the patty, obj2 should be the bottom bun.
31 stack_obj1_on_obj2(obj1=patties[0], obj2=bottom_bun_to_use)
32
33 # Stack the lettuce on top of the patty.
34 # obj1 should be the lettuce, obj2 should be the patty on top of the bottom bun.
35 stack_obj1_on_obj2(obj1=lettuce_to_use, obj2=patties[0])
36
37 #
Why 1-step failed?
In Fig. 11's result, 1-step actually performed worse than No-Cot. The worsened performance is caused by low-quality specifications produced by the LLM. Because stage 2 recursive expansion is highly reliant on the specification that gets passed in, any mistaken or missing specification will be reflected in the code as well. Below shows the intermediate reasonings and specification outputted by the LLM. Notice that it does not capture any information about the lettuce and the tomato needing to be cut before being added to the burger (see lines 22-23 and lines 25-26).
1 \* The order of high level actions is: ['cook', 'assemble', 'cook', ' assemble']   
2 \* Because there are two cook' actions and two assembl' actions, we assume that we must cook and assemble each burger separately.   
3 Thus:   
4 <Make two burgers.   
5   
6 Specifically:   
7 Get a list of all the patties in the kitchen.   
8 Get a list of all the stoves in the kitchen.   
9 Get a list of all the buns in the kitchen.   
10 Get a list of all the lettuces in the kitchen.   
11 Get a list of all the tomatoes in the kitchen.   
12   
13 Repeat the following twice:   
14 Decide a patty to use..   
15 Decide a stove to go to.   
16 Cook that patty at that stove.   
17   
18 Decide a bottom bun to use.   
19 Place that bottom bun on a plate.   
20 Place that patty on top of the bottom bun.   
21   
22 Decide a lettuce to use.   
23 Place that lettuce on top of the patty.   
24   
25 Decide a tomato to use.   
26 Place that tomato on top of the lettuce.   
27   
28 Decide a top bun to use.   
29 Place that top bun on top of the tomato.
Consequently, although the code (shown below) closely matched the specification and included the for-loop, because the specification was wrong, the code also failed to solve the problem.
1 #Get a list of all the patties in the kitchen.   
2 patties  $=$  get_all_obj_names_that_match_type('patty')   
3 #Get a list of all the stoves in the kitchen.   
4 stoves  $=$  get_all_locationnames_that_match_type('stove')   
5 #Get a list of all the buns in the kitchen.   
6 buns  $=$  get_all_obj_names那個 match_type('bottom bun')  $^+$  get_all_obj_names那個 match_type('top bun')   
7 #Get a list of all the lettuces in the kitchen.   
8 lettuces  $=$  get_all_obj_names那個 match_type('lettuce')   
9 #Get a list of all the tomatoes in the kitchen.   
10 tomatoes  $=$  get_all_obj_names那個 match_type('tomato')   
11   
12 #Repeat the following twice:   
13 for i in range(2):   
14 # Decide a patty to use.   
15 patty_to_use  $=$  patties[i]   
16 # Decide a stove to go to.   
17 stove_to_cook_at  $=$  stoves[i]   
18 # Cook that patty at that stove.   
19 cook_object_at_location(obj  $\equiv$  patty_to_use, location= stove_to_cook_at)   
20   
21 # Decide a bottom bun to use.   
22 bottom_bun_to_use  $=$  buns[i\*2]   
23 # Place that bottom bun on a plate.   
24 place_obj_on_obj(obj1  $\equiv$  bottom_bun_to_use, obj2  $\equiv$  'plate')   
25 # Place that patty on top of the bottom bun.   
26 place_obj_on_obj(obj1  $\equiv$  patty_to_use, obj2  $\equiv$  bottom_bun_to_use)   
27   
28 # Decide a lettuce to use.   
29 lettuce_to_use  $=$  lettuces[i]   
30 # Place that lettuce on top of the patty.   
31 place_obj_on_obj(obj1  $\equiv$  lettuce_to_use, obj2  $\equiv$  patty_to_use)   
32   
33 # Decide a tomato to use.   
34 tomato_to_use  $=$  tomatoes[i]   
35 # Place that tomato on top of the lettuce.   
36 place_obj_on_obj(obj1  $\equiv$  tomato_to_use, obj2  $\equiv$  lettuce_to_use)   
37   
38 # Decide a top bun to use.   
39 top_bun_to_use  $=$  buns[i\*2+1]   
40 # Place that top bun on top of the tomato.   
41 place_obj_on_obj(obj1  $\equiv$  top_bun_to_use, obj2  $\equiv$  tomato_to_use)
Why 2-steps failed?
2-steps improved its specification compared to 1-step's. Specifically, its specification was able to identify that lettuce and tomato require cutting before they are stacked as a part of the burger. However, it still fails to identify the physical constraint that in order to cut an object, it also needs to specify the location. While it is able to identify the stove used for cooking (line 17), it fails to identify the cutting board (line 22 and line 25).
1 * The order of high level actions is: ['cook', 'assemble', 'cook', 'assemble']
2 * Because 'cook' is before 'assemble', we assume that we must cook the patties before assembling the burgers.
3 * Because 'assemble' is repeated twice, we assume that we must assemble two burgers.
4 Thus:
5 <Make two burgers.
6
7 Specifically:
8 Get a list of all the patties in the kitchen.  
9 Get a list of all the buns in the kitchen.  
10 Get a list of all the lettuces in the kitchen.  
11 Get a list of all the tomatoes in the kitchen.  
12  
13 Repeat the following twice:  
14 Decide a patty to cook.  
15 Get a list of all the stoves in the kitchen.  
16 Decide a stove to use.  
17 Cook that patty on that stove.  
18 Decide a bun to use as the bottom bun.  
19 Place that bun on a plate.  
20 Stack that patty on top of that bun.  
21 Decide a tomato to use.  
22 Cut that tomato.  
23 Place that tomato on top of that patty.  
24 Decide a lettuce to use.  
25 Cut that lettuce.  
26 Place that lettuce on top of that tomato.  
27 Decide a bun to use as the top bun.  
28 Place that bun on top of that lettuce.
Consequently, in the code that the LLM produced based on the specification, it created a function cut_object(obj). Then, because the function signature does not specify any location to cut the object at, the downstream function definition also fails (see the definition starting from line 43).
1 # Get a list of all the patties in the kitchen.  
2 patties = get_all_obj_names_thatmatch_type('patty')  
3 # Get a list of all the buns in the kitchen.  
4 buns = get_all_obj_names那個 match_type('bottom bun') + get_all_obj_names那個 match_type('top bun')  
5 # Get a list of all the lettuces in the kitchen.  
6 lettuces = get_all_obj_names那個 match_type('lettuce')  
7 # Get a list of all the tomatoes in the kitchen.  
8 tomatoes = get_all_obj_names那個 match_type('tomato')  
9  
10 # Repeat the following twice:  
11 for i in range(2):  
12 # Decide a patty to cook.  
13 Patty_to_cook = patties[i]  
14 # Get a list of all the stoves in the kitchen.  
15 stoves = get_all_locationnames那個 match_type('stove')  
16 # Decide a stove to use.  
17 stove_to_use = stoves[i]  
18 # Cook that patty on that stove.  
19 cook_object_at_location(obj= Patty_to_cook, location= stove_to_use)  
20 # Decide a bun to use as the bottom bun.  
21 bottom_bun_to_use = buns[i]  
22 # Place that bun on a plate.  
23 place_obj_on_obj(obj1=bottom_bun_to_use, obj2='plate')  
24 # Stack that patty on top of that bun.  
25 stack_obj1_on_obj2(obj1=Patty_to_cook, obj2=bottom_bun_to_use)  
26 # Decide a tomato to use.  
27 tomato_to_use = tomatoes[i]  
28 # Cut that tomato.  
29 cut_object(obj=tomato_to_use)  
30 # Place that tomato on top of that patty.  
31 place_obj_on_obj(obj1=tomato_to_use, obj2=patty_to_cook)  
32 # Decide a lettuce to use.  
33 lettuce_to_use = lettuces[i]  
34 # Cut that lettuce.  
35 cut_object(obj=lettuce_to_use)  
36 # Place that lettuce on top of that tomato.  
37 place_obj_on_obj(obj1=lettuce_to_use, obj2=tomato_to_use)  
38 # Decide a bun to use as the top bun.
39 top_bun_to_use  $=$  buns[i+2]   
40 # Place that bun on top of that lettuce.   
41 place_obj_on_obj(obj1  $\equiv$  top_bun_to_use, obj2  $\equiv$  lettuce_to_use)   
42   
43 def cut_object(obj):   
44 if not is_holding(obj):   
45 # To cut obj, the robot needs to hold obj first.   
46 if is_in_a_stack(obj):   
47 # Because obj is in a stack, robot need to move then   
48 unstack the obj from the obj_at_bottom first obj_at_bottom  $=$  get_obj_that_is_underneath(obj_at_top  $\equiv$  obj)   
49 move_then_unstack(obj_to_unstack  $\equiv$  obj, obj_at_bottom= obj_at_bottom, unstack_location  $\equiv$  get_obj_location(obj_at_bottom))   
50 else:   
51 # Since obj is not in a stack, robot can just move then pick it up   
52 move_then_pick(obj  $\equiv$  obj)   
53 # cut the object   
54 cut_until_is_cut(obj  $\equiv$  obj)
13 Intermediate Reasoning Ablation Experiment
This experiment studies how the thoroughness of intermediate reasoning (in stage 3 of recursive summarization) affects the LLM's performance. We found:
- Our approach may be improved by ablating away some part of reasoning (e.g. listing the high-level actions)
- Having intermediate reasoning of any form is helpful for hard cooking tasks that have many sub-tasks
- Future work is necessary to engineer better prompts
13.1 Experiment detail
This experiment compares Demo2Code (labeled as Full) with three additional ablation models each with differing levels of reasoning
- No reasoning: The LLM generates the specification directly from step 2 of recursive summarization with no intermediate reasoning.
- Only List: The LLM generates the specification after intermediate reasoning which lists the high-level actions in common with the scenarios from step 2 of recursive summarization.
- Only Analyze: The LLM generates the specification after intermediate reasoning which describes the repetition and ordering of high-level actions from step 2 of recursive summarization.
These models are tested on all the Robotouille tasks. We use 3 clusters of tasks based on the number of high-level actions/sub-tasks they have.
- Easy cooking tasks ( $\leq$  2 high-level actions/sub-tasks): "cook and cut", "cook two patties", and "cut two lettuces"
- Normal cooking tasks (between 2-7 high-level actions/sub-tasks): "make a burger" and "assemble two burgers with already cooked patties"
- Hard cooking tasks ( $\geq 8$  high-level actions/sub-tasks): "make two burgers"
13.2 Quantitative result
For each task and for each of that task's specific requirements, we tested each model on 10 randomly generated environments and took an average of the unit test pass rate. Fig. 12 shows that the Only Analyze model outperforms the other models. All methods easily achieve  $100\%$  pass rate on Easy
Figure 12: Ablation of different amount and style of intermediate reasoning at step 3 of recursive summarization across Robotouille tasks. These tasks are clustered into easy cooking tasks, normal cooking tasks, and hard cooking tasks based on the number of high-level actions/sub-tasks each task has.
tasks. There is a trend that including any kind of reasoning improves performance on hard tasks. Only List only slightly suffers in Normal performance while increasing significantly for Hard performance relative to No Reasoning. Full's Normal performance suffers significantly yet its Hard performance increases as well, suggesting a tradeoff between Normal performance and Hard performance; however, Only Analyze has the highest Normal and Hard performance of all models. Since Full combines both Only List and Only Analyze, this shows that concatenating reasoning methods doesn't necessarily mean better performance.
13.3 Qualitative example
In this section, we show a qualitative example of all models' output on the long-horizon task "make two burgers" with the specific requirement "stack lettuce atop patty immediately." We choose this example to show the effects of reasoning on the code as well as to shed light on why Full underperforms relative to Only Analyze. Only Analyze is the only model with 1.0 success rate while the other models all have 0.0 success rates. For each model's output, the helper functions can be found in section 19.3.
13.3.1 Why No Reasoning failed?
No Reasoning is the only unsuccessful model that generates code that executes. Due to the lack of any reasoning, it performs the task correctly but with the incorrect requirement; rather than preparing one burger at a time and assembling the burger immediately as ingredients are prepared, it prepares all ingredients before assembling both burgers in parallel.
1 # Get a list of all the patties in the kitchen.  
2 patties = get_all_obj_names_that_match_type('patty')  
3 # Get a list of all the lettuces in the kitchen.  
4 lettuces = get_all_obj_names那個 match type('lettuce')  
5 # Get a list of all the bottom buns in the kitchen.  
6 bottom_buns = get_all_obj_names那個 match type('bottom bun')  
7 # Get a list of all the top buns in the kitchen.  
8 top_buns = get_all_obj_names那個 match type('top bun')  
9 # Get a list of all the stoves in the kitchen.  
10 stoves = get_all_location_names那個 match type('stove')  
11 # Get a list of all the cutting boards in the kitchen.  
12 cutting Boards = get_all_location_names那個 match type('cutting_board')  
13  
14 # Decide two patties to cook.  
15 patties_to_cook = patties[:2]  
16 # Cook those patties at two different stoves.  
17 for i in range(2):
18 cook_object_at_location(obj=patties_to_cook[i], location  $\equiv$  stoves[i ])   
19   
20 # Decide two lettuces to cut.   
21 lettuces_to-cut  $=$  lettuces[:2]   
22 # Cut those lettuces at two different cutting boards.   
23 for i in range(2): cut_object_at_location(obj  $\equiv$  lettuces_toCut[i], location= cutting Boards[i])   
25   
26 # Decide two bottom buns to use.   
bottom_buns_to_use  $=$  bottom_buns[:2]   
28 # Stack a patty on top of each bottom bun.   
for i in range(2): patty_to_use  $=$  patties_to_cook[i] bottom_bun_to_use  $=$  bottom_buns_to_use[i] stack_obj1_on_obj2(obj1  $\equiv$  patty_to_use, obj2  $\equiv$  bottom_bun_to_use)   
33   
4 # Decide two lettuces to use.   
lettuces_to_use  $=$  lettuces_to-cut[:2]   
# Stack a lettuce on top of each patty.   
for i in range(2): lettuce_to_use  $=$  patties_to_use[i] patty_to_use  $=$  patties_to_cook[i] stack_obj1_on_obj2(obj1  $\equiv$  lettuce_to_use, obj2  $\equiv$  patty_to_use)   
41   
42 # Decide two top buns to use.   
top_buns_to_use  $=$  top_buns[:2]   
44 # Stack a top bun on top of each lettuce.   
for i in range(2): top_bun_to_use  $=$  top_buns_to_use[i] lettuce_to_use  $=$  lettuces_to_use[i] stack_obj1_on_obj2(obj1  $\equiv$  top_bun_to_use, obj2  $\equiv$  lettuce_to_use)
13.3.2 Why Only List failed?