SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Only List improves on No Reasoning by generating code in the correct order of the specification. Patties are cooked and then stacked on the bottom bun before lettuces are cut and stacked on the patties before finally the top buns. While the order is now correct, just specifying the order wasn't effective enough because the reasoning missed making burgers one by one rather than all in parallel. In addition, this code doesn't execute due to the attempt of cutting lettuces back to back at the same cutting board. This violates the environment's constraints since only one object can be at one station at a time.
1 # Get a list of all the patties in the kitchen.  
2 patties = get_all_obj_names_that-match_type('patty')  
3 # Get a list of all the lettuces in the kitchen.  
4 lettuces = get_all_obj_names那個 match type('lettuce')  
5 # Get a list of all the bottom buns in the kitchen.  
6 bottom_buns = get_all_obj_names那個 match type('bottom bun')  
7 # Get a list of all the top buns in the kitchen.  
8 top_buns = get_all_obj_names那個 match type('top bun')  
9 # Get a list of all the stoves in the kitchen.  
10 stoves = get_all_location_names那個 match type('stove')  
11 # Get a list of all the cutting boards in the kitchen.  
12 cutting Boards = get_all_location_names那個 match type('cutting_board')  
13  
14 # Decide two patties to cook.  
15 patties_to_cook = patties[:2]  
16 # Cook those patties at two stoves.  
17 for i in range(2):  
    cook_object_at_location(obj=patties_to_cook[i], location=stoves[i])
19   
20 # Decide two bottom buns to use.   
21 bottom_buns_to_use  $=$  bottom_buns[:2]   
22 # Stack a patty on top of each bottom bun.   
23 for i in range(2): stack_obj1_on_obj2(obj1=patties_to_cook[i], obj2= bottom_buns_to_use[i])   
25   
26 # Decide two lettuces to use.   
27 lettuces_to_use  $=$  lettuces[:2]   
28 # Cut those lettuces at a cutting board.   
29 for lettuce in lettuces_to_use: cut_object(objl  $\equiv$  lettuce, location  $\equiv$  cutting Boards [0])   
31 # Stack a lettuce on top of each patty.   
32 for i in range(2): stack_obj1_on_obj2(obj1  $\equiv$  lettuces_to_use[i], obj2  $\equiv$  patties_to_cook[i ])   
34   
35 # Decide two top buns to use.   
36 top_buns_to_use  $=$  top_buns[:2]   
37 # Stack a top bun on top of each lettuce.   
38 for i in range(2): stack_obj1_on_obj2(obj1  $\equiv$  top_buns_to_use[i], obj2  $\equiv$  lettuces_to_use[i ])
13.3.3 Why Full failed?
Full gets much closer to correctly solving the task than Only List. Unlike Only List, Full has the correct ordering for the requirement; it stacks the patty and lettuce immediately after preparation and it does so for an entire burger before moving on. The issue is that while with the analysis of ordering and repetition it manages to put all the code in one for loop rather than multiple, it retained some of the poor aspects of Only List by attempting to cook two patties at once. Doing this within the for loop is incorrect logic. In addition, the cooking is attempted on the same stove twice which violates the environment's constraints of one item being on a station at a time.
1 #Get a list of all the bottom buns in the kitchen.   
2 bottom_buns  $=$  get_all_obj_names_that-match_type('bottom bun')   
3 #Get a list of all the lettuces in the kitchen.   
4 lettuces  $=$  get_all_obj_names那個 match_type('lettuce')   
5 #Get a list of all the patties in the kitchen.   
6 patties  $=$  get_all_obj_names那個 match_type('patty')   
7 #Get a list of all the top buns in the kitchen.   
8 top_buns  $=$  get_all_obj_names那個 match_type('top bun')   
9 #Get a list of all the stoves in the kitchen.   
10 stoves  $=$  get_all_location_names那個 match_type('stove')   
11 #Get a list of all the cutting boards in the kitchen.   
12 cutting Boards  $=$  get_all_location_names那個 match_type('cutting_board '）   
13   
14 #Repeat the following twice   
15 for i in range(2):   
16 # Decide a stove to use.   
17 stove_to_use  $=$  stoves[i]   
18 #Cook two patties at that stove.   
19 for j in range(2): cook_object_at_location(obj  $\equiv$  patties[j], location  $\equiv$  stove_to_use)   
20   
21 # Decide a bottom bun to use.   
22 bottom_bun_to_use  $=$  bottom_buns[i]   
24 #Stack the patty on top of the bottom bun.   
25 #obj1 should be the patty, obj2 should be the bottom bun. stack_obj1_on_obj2(obj1  $\equiv$  patties[i], obj2  $\equiv$  bottom_bun_to_use)   
27   
28 #Decide a lettuce to use.
lettuce_to_use  $=$  lettuces[i]   
# Cut that lettuce at a cutting board.   
cut_object_at_location(obj  $\equiv$  lettuce_to_use，location  $\equiv$  cutting Boards [i])   
# Stack the lettuce on top of the patty.   
# obj1 should be the lettuce, obj2 should be the patty on top of the bottom bun. stack_obj1_on_obj2(obj1  $\equiv$  lettuce_to_use，obj2  $\equiv$  patties[i])   
# Decide a top bun to use.   
top_bun_to_use  $=$  top_buns[i]   
# Stack the top bun on top of the lettuce.   
# obj1 should be the top bun, obj2 should be the lettuce on top of the patty on top of the bottom bun. stack_obj1_on_obj2(obj1  $\equiv$  top_bun_to_use，obj2  $\equiv$  lettuce_to_use)
13.3.4 Only Analyze Output
Only Analyze is the only model to adhere to the environment's constraints. Note that while the same stove is used for cooking twice, the second time a patty is cooked the first patty has already been removed from the stove and stacked into the first burger.
1 #Get a list of all the patties in the kitchen.   
2 patties  $=$  get_all_obj_names_that_match_type('patty')   
3 #Get a list of all the lettuces in the kitchen.   
4 lettuces  $=$  get_all_obj_names那個 match_type('lettuce')   
5 #Get a list of all the bottom buns in the kitchen.   
6 bottom_buns  $=$  get_all_obj_names那個 match_type('bottom bun')   
7 #Get a list of all the top buns in the kitchen.   
8 top_buns  $=$  get_all_obj_names那個 match_type('top bun')   
9 #Get a list of all the stoves in the kitchen.   
10 stoves  $=$  get_all_location_names那個 match_type('stove')   
11 #Get a list of all the cutting boards in the kitchen.   
12 cutting Boards  $=$  get_all_location_names那個 match_type('cutting_board '）   
13   
14 # Decide a stove to use.   
15 stove_to_use  $=$  stoves[0]   
16   
17 # Repeat the following twice   
18 for i in range(2):   
19 # Decide a patty to cook.   
20 patty_to_cook  $=$  patties[i]   
21 # Cook that patty at that stove.   
22 cook_object_at_location(obj= Patty_to_cook, location  $\equiv$  stove_to_use)   
23   
24 # Decide a bottom bun to use.   
25 bottom_bun_to_use  $=$  bottom_buns[i]   
26 # Stack the patty on top of the bottom bun.   
27 # obj1 should be the patty, obj2 should be the bottom bun. stack_obj1_on_obj2(obj1=Patty_to_cook, obj2=bottom_bun_to_use)   
29   
30 # Decide a lettuce to use.   
31 lettuce_to_use  $=$  lettuces[i]   
32 # Cut that lettuce at that cutting board.   
33 cut_object_at_location(obj=lettuce_to_use, location=cutting Boards [i])   
34 # Stack the lettuce on top of the patty.   
35 # obj1 should be the lettuce, obj2 should be the patty on top of the bottom bun. stack_obj1_on_obj2(obj1=lettuce_to_use, obj2=patty_to_cook)   
37   
38 # Decide a top bun to use.   
39 top_bun_to_use  $=$  top_buns[i]
Stack the top bun on top of the lettuce.
obj1 should be the top bun, obj2 should be the lettuce on top of the patty on top of the bottom bun.
stack_obj1_on_obj2(obj1 = top_bun_to_use, obj2 = lettuce_to_use)
14 Recursive Expansion Ablation Experiment
This experiment studies how the number of recursive code expansion steps (in stage 2 recursive expansion) affects the LLM's performance. We found:
- It is helpful to guide the LLM to slowly expand the code instead of asking it to directly generate all the code at once using only the low-level imported APIs.
- The initial code that LLM uses to expand the rest of the functions should align closer to the given specifications.
14.1 Experiment detail
This experiment compares Demo2Code (labeled as Full) with three additional ablation models each with a different amount of recursive code expansion steps:
- 1-layer: Given a specification, the LLM directly outputs the code for the task using only low-level action and perception APIs from the import statement. The LLM is not allowed to define any helper function.
- 2-layer (Comp): Given a specification, the LLM first outputs corresponding code that can call undefined "composite functions." Each composite functions contain at most two low-level actions, e.g. move_then_pick, move_then_stack. In the next step, the LLM defines these composite functions using only low-level action and perception APIs from the import statement.
- 2-layer (High): Given a specification, the LLM first outputs corresponding high-level code that can call undefined "high-level functions." This step is the same as Demo2Code. Then, the LLM defines these high-level functions using only low-level action and perception APIs from the import statement.
These models are tested on all the Robotouille tasks. Because section 13 on Intermediate Reasoning Ablation Experiments also tested only on Robotouille, we use the same 3 clusters of tasks based on the number of high-level actions/sub-tasks they have.
- Easy cooking tasks ( $\leq 2$  high-level actions/sub-tasks): "cook and cut", "cook two patties", and "cut two lettuces"
- Normal cooking tasks (between 2-7 high-level actions/sub-tasks): "make a burger" and "assemble two burgers with already cooked patties"
- Hard cooking tasks ( $\geq 8$  high-level actions/sub-tasks): "make two burgers"
14.2 Quantitative result
For each task and for each of that task's specific requirements, we tested each model on 10 randomly generated environments and took an average of the unit test pass rate. Fig. 13 shows how our approach Demo2Code (Full) outperforms the other models. Overall, as the number of recursive code generation steps increases, the LLM's performance also increases. Interestingly, 2-Layers (Comp) performs worse than 2-Layer's (High) despite having the same amount of recursive steps. The two model's performance difference suggests that the granularity of the initial code also affects the performance. While 2-Layers (High) first asks the LLM to generate high-level code, which is the same approach as Demo2Code, 2-Layers (Comp) first asks the LLM to generate code that calls composite functions. Consequently, for 2-Layers (Comp), the LLM needs to first output longer, more complicated code the LLM has to produce in one-step, which is more prone to error.
40   
41   
42
Figure 13: Ablation of different amount of recursive code expansion steps across Robotouille tasks. These tasks are clustered into easy cooking task, normal cooking task, and hard cooking task based on the number of high-level actions/sub-tasks each task has.
14.3 Qualitative example
In this section, we show a qualitative example of all models' output on the short-horizon task "cut two lettuces." Note that cutting lettuce has not been shown in the prompt, so the LLM needs to be able to generalize to this new task and properly use a for-loop to write efficient code. 1-layer and 2-layer (Comp) had 0.0 success rate. 2-layer (High has 0.2 success rate, and Demo2Code has 1.0 success rate. All models are able to generate the for-loop, as evident in the outputted code below. However, all ablation models struggle to generate code that adheres to physical constraints and applies to any random environment.
14.3.1 Why 1-Layer failed?
1-Layer generates the correct code to cut the lettuce. Despite not seeing how the low-level action cut () should be used in the prompt, it correctly writes the code that will cut the lettuce until the lettuce becomes cut. However, the LLM misses the physical constraints associated with cooking, e.g. in order to cut the lettuce at a cutting board, the robot needs to make sure that there is a lettuce at the cutting board before starting to cut. If there is no lettuce on the cutting board, the robot first needs to go pick up a lettuce and bring it to the cutting board. Another issue that 1-Layer encounters is reaching the LLM maximum token length. For longer tasks, it cannot finish generating all the code, so its overall execution success rate is only 0.04.
1 #Get a list of all the lettuces in the kitchen.   
2 lettuces  $=$  get_all_obj_names_that-match_type('lettuce')   
3 #Get a list of all the cutting boards in the kitchen.   
4 cutting Boards  $\equiv$  get_all_location_names那個 match_type('cutting_board ）   
5   
6 #Repeat the following twice   
7 for i in range(2):   
8 # Decide a cutting board to use.   
9 cutting_board_to_use  $=$  cutting Boards[i]   
10 # Decide a lettuce to cut.   
11 lettuce_toCut  $=$  lettuces[i]   
12 # Cut that lettuce at that cutting board.   
13 #To cut lettuce_to-cut, the robot first needs to be at the same location as cutting_board_to_use   
14 if getCurr_location()  $! =$  cutting_board_to_use:   
15 move(getCurr_location(),cutting_board_to_use)   
16 while not isCut(lettuce_toCut):   
17 cut(lettuce_toCut)
14.3.2 Why 2-Layers (Comp) failed?
2-Layer (Comp) faces the same issue of not adhering to all the physical constraints. For example, it successfully identifies that it needs to pick up the lettuce before going to the cutting board and cutting it. However, it does not place the lettuce on the cutting board before it starts cutting (see line 27).
In addition, all the composite functions defined in the recursive step are technically correct even though they are not efficient or compact. For example, move_then_cut unnecessarily moves to the object to cut before actually starting to cut (see line 19). It then moves to the cut location inside the while loop of checking if the object has been cut yet or not (see line 22). A more optimal code will be to move to the cut location before the while loop, then just have the call to cut function in the while loop.
A core issue with 2-Layer (Comp) is that the LLM needs to account for the physical constraints when generating the code in the first step. However, the specification that the LLM receives only includes information about the order between high-level actions and high-level control flows (e.g. for loops). The LLM is more prone to mistakes when it has to extrapolate the physical constraints from the high-level specification. Meanwhile, 2-Layer (High), which has the same number of recursive steps, has generated the correct initial code that the LLM can expand on. Its initial code aligns closer to the specification (see the code in section 14.3.3).
1 # Get a list of all the lettuces in the kitchen.  
2 lettuces = get_all_obj_names-that_MATCH_type('lettuce')  
3 # Get a list of all the cutting boards in the kitchen.  
4 cutting Boards = get_all_location_names-that_MATCH_type('cutting_BOARD')  
5  
6 # Decide a cutting board to use.  
7 cutting_board_to_use = cutting Boards[0]  
8  
9 # Repeat the following twice  
10 for i in range(2):  
11 # Decide a lettuce to cut.  
12 lettuce_to-cut = lettuces[i]  
13 # Cut that lettuce at that cutting board.  
14 # To cut lettuce_to-cut, the robot first needs to be holding lettuce_to-cut  
15 if not is_holding(lettuce_to-cut):  
16 # If the robot is not holding lettuce_to-cut, there are 2 scenarios:  
17 # (1) if lettuce_to-cut is in a stack ,unstack lettuce_to-cut  
18 # (2) else, pick up lettuce_to-cut.  
19 if is_in_a_stack(lettuce_to-cut):  
20 # Because lettuce_to-cut is in a stack, robot need to move then unstack the lettuce_to-cut from the obj_at_bottom first obj_at_bottom = get_obj_that_is_underneath(obj_at_top= lettuce_to-cut)  
21 movethen_unstack(obj_to_unstack  $\equiv$  lettuce_to-cut, obj_at_bottom  $=$  obj_at_bottom, unstack_location  $\equiv$  get_obj_location( obj_at_bottom))  
22 else:  
23 # Since lettuce_to-cut is not in a stack, robot can just move then pick it up  
24 move_then_pick(obj  $\equiv$  lettuce_to-cut)  
25 # move to the cutting-board_to_use's location then cut lettuce_to-cut  
26 move_thenCut(obj_toCut  $\equiv$  lettuce_to-cut, cut_location  $\equiv$  cutting_board_to_use)  
27
32 if get_curr_location() != bottom_location:  
33 move(get_curr_location(), bottom_location)  
34 unstack(obj_to_unstack, obj_at_bottom)  
35 if get_curr_location() != unstack_location:  
36 move(get_curr_location(), unstack_location)  
37  
38 def movethen_pick(obj):  
39 obj_location = getobj_location(obj)  
40 if get_curr_location() != obj_location:  
41 move(get_curr_location(), obj_location)  
42 pick_up(obj, obj_location)  
43  
44 def movethen.cut(obj_to_cut, cut_location):  
45 obj_location = getobj_location(obj_to_cut)  
46 if get_curr_location() != obj_location:  
47 move(get_curr_location(), obj_location)  
48 while not is.cut(obj_to.cut):  
49 if get_curr_location() != cut_location:  
50 move(get_curr_location(), cut_location)  
51 cut(obj_to.cut)
14.3.3 Why 2-Layers (High) failed?
2-Layers (High) generated the same initial high level code as Demo2Code. However, the definition of cut_object_at_location in the recursive step is not exactly correct because the LLM is not taking account of all the physical constraints. It again assumes that the object to cut is already on the cutting board to cut. It moves to the object's location before starting to cut the object (line 18). Although 2-Layers (High) has 0.2 unit test success rate, it achieved this because many of the lettuces happened to be placed on cutting boards in the randomly initialized environments.
1 # Get a list of all the lettuces in the kitchen.  
2 lettuces = get_all_obj_names_that-match_type('lettuce')  
3 # Get a list of all the cutting boards in the kitchen.  
4 cutting Boards = get_all_location_names那個 match_type('cutting_board')  
5  
6 # Repeat the following twice:  
7 for i in range(2):  
8 # Decide a lettuce to cut.  
9 lettuce_toCut = lettuces[i]  
10 # Decide a cutting board to use.  
11 cutting-board_to_use = cutting Boards[i]  
12 # Cut that lettuce at that cutting board.  
13 cut_object_at_location(obj=lettuce_to-cut, location=cutting_BOARD_to_use)  
14  
15 def cut_object_at_location(obj, location):  
16 # To cut an object, the robot first needs to be at the same location as obj  
17 if get_curr_location() != get_obj_location(obj):  
18 move(get_curr_location(), get_obj_location(obj))  
19 # cut the object  
20 while not is CUT(obj):  
21 cut(obj)
14.3.4 Demo2Code Output
Demo2Code is able to adhere to physical constraints and correctly defines all the helper function in its recursive expansion steps.
def cut_object_at_location(obj, location):
    # To cut an object, the robot first needs to be holding obj
    if not is_holding(obj):
        # If the robot is not holding obj, there are 2 scenarios:
5 # (1) if obj is in a stack ,unstack obj   
6 # (2) else, pick up obj.   
7 if is_in_a_stack(obj):   
8 # Because obj is in a stack, robot need to move then   
9 unstack the obj from the obj_at_bottom first obj_at_bottom  $=$  get_obj_that_is_underneath(obj_at_top  $\equiv$  obj) move_then_unstack(obj_to_unstack  $\equiv$  obj, obj_at_bottom= obj_at_bottom, unstack_location  $\equiv$  get_obj_location(obj_at_bottom)) else:   
12 # Since obj is not in a stack, robot can just move then pick it up move_then_pick(obj  $\equiv$  obj)   
14 # place the object at the location to cut at move_then_place(obj  $\equiv$  obj, place_location  $\equiv$  location) # cut the object cut_until_is.cut(obj  $\equiv$  obj)   
18   
19 def cut_until_is.cut(obj): while not is.cut(obj): cut(obj)   
22   
23 def move_then_unstack(obj_to_unstack, obj_at_bottom, unstack_location) .   
24 # For unstacking, we need to move to the location of the bottom object   
25 if get_curr_location()  $! =$  get_obj_location(obj_at_bottom): move(get_curr_location(), get_obj_location(obj_at_bottom))   
27 unstack(obj_to_unstack, obj_at_bottom)   
28 # After unstacking, we need to move to the unstack_location   
29 if get_curr_location()  $! =$  unstack_location: move(get_curr_location(), unstack_location)   
31   
32 def move_then_pick(obj):   
33 obj_location  $=$  get_obj_location(obj)   
34 if get_curr_location()  $! =$  obj_location: move(get_curr_location(), obj_location)   
35 pick_up(obj, obj_location)   
37   
38 def move_then_place(obj, place_location):   
39 if get_curr_location()  $! =$  place_location: move(get_curr_location(), place_location)   
40 place(obj, place_location)
15 Broader Impact
Our approach is a step towards making collaborative robots more accessible in different settings, such as homes, factories, and logistics operations. The broader implications of our research are manifold, touching upon societal, economic, ethical, technical, and educational aspects.
Societal Impacts: Currently, robots can only be programmed by engineers, limiting the tasks the robot can do to design choices made by engineers. This limits the accessibility for everyday users who require personalization to their individual needs and use cases. Our work tackles this problem head-on by allowing robots to be easily programmable via intuitive, user-friendly interfaces like vision and language. This could lead to increased creativity in the types of tasks that robots can be used for, potentially enabling novel applications and solutions.
Economic Impacts: Our approach has the potential to dramatically decrease the cost of programming robots, thus lowering the barriers to entry for businesses interested in incorporating robotics into their workflows. This could boost productivity and efficiency across a variety of industries, such as manufacturing, agriculture, and healthcare, among others. In the long term, this may contribute to economic growth and job creation in sectors that are currently behind on automation.
Ethical and Legal Impacts: Our approach has many ethical and legal considerations that need to be addressed. For instance, the widespread use of automation may lead to job displacement in certain sectors. Furthermore, it's crucial to ensure that the task code generated from demonstrations respects privacy laws and does not inadvertently encode biased or discriminatory behavior. Future work in this area will require close collaboration with ethicists, policymakers, and legal experts to navigate these issues.
Technical Impacts: Our work has the potential to accelerate the development of more intuitive and efficient human-robot interaction paradigms. For instance, demonstrations can be extended to interventions and corrections of the user. On the algorithmic side, our approach is a first step to connecting LLMs to Inverse Reinforcement Learning, and can spur advances in the fields of imitation learning, reinforcement learning, and natural language processing.
Educational Impacts: Lastly, our work could contribute to educational initiatives. The ability to generate task code from demonstrations could be utilized as an effective teaching tool in schools and universities, promoting a more experiential and intuitive approach to learning about robotics and coding. This could inspire and enable more students to pursue careers in STEM fields.
16 Reproducibility
We ran all our experiments using GPT-3.5 (gpt-3.5-turbo) with temperature 0. We have provided our codebase in https://github.com/portal-cornell/demo2code and all the prompts in section 18. The queries used to produce our results are available in the code base. Note that although we set the temperature to 0, which will make the LLM output mostly deterministic, the output might still slightly vary even for identical input.
17 Demo2Code Example Output
We provide an example for each domain and explain Demo2Code's intermediate and final output.
17.1 Tabletop Simulator Example
This section shows an example of how Demo2Code solves the task: stacking all cylinders into one stack. The hidden user preference is that certain objects might need to have a fixed stack order, while other objects can be unordered. For this example, two objects ('blue cylinder' and 'purple cylinder') must follow the order that 'blue cylinder' should always be directly under the 'purple cylinder.'
17.1.1 Query
The query has 3 components: (1) a list of objects that are in the environment, (2) a language instruction describing the goal of the task, and (3) two demonstrations.