SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In table 4, we characterize the complexity of the tasks in terms of the demonstrations' length, the code's length, and the expected code's complexity (i.e. how many loops/conditionals/functions are needed to solve this task).
Table 4: For tabletop tasks, we group them by cluster and report: 1. number of states in demonstrations (range and average) 2. number of predicates in demonstrations (range and average) 3. number of lines in the oracle Spec2Code's generated code (range and average) 4. average number of loops 5. average number of conditionals 6. average number of functions
Task	Input Demo Length	Code Length	# of loops	# of conditionals	# of functions
# of states	# of predicates
Place A next to B	1-1 (1.00)	2-5 (3.53)	3-7 (3.38)	0.00	0.02	1.00
Place A at corner/edge	1-1 (1.00)	1-5 (2.09)	2-4 (3.03)	0.00	0.00	1.00
Place A on top of B	1.0-4.0 (2.50)	3-19 (9.40)	2-6 (3.65)	0.10	0.00	1.00
Stack all blocks/cylinders	2-7 (4.43)	4-33 (14.09)	3-15 (4.44)	0.24	0.06	1.00
Stack all blocks/cylinders into one stack	3.5-4 (3.98)	12-23 (14.77)	12-12 (12)	1.00	1.00	1.00
Stack all objects into two stacks	6-8 (6.95)	16-42 (23.90)	7-25 (8.1)	0.05	0.20	1.00
9 Robotouille Simulator Pipeline
9.1 Overview
9.1.1 Simulator Description
In Robotouille, a robot chef performs cooking tasks in a kitchen environment. The state of the kitchen environment consists of items such as buns, lettuce, and patties located on stations which could be tables, grills, and cutting boards. The actions of the robot consist of moving around from one station to another, picking items from and placing items on stations, stacking items atop and unstacking items from another item, cooking patties on stoves, and cutting lettuce on cutting boards. The state and actions are described through the Planning Domain Description Language (PDDL).
These PDDL files consist of a domain and a problem. The domain file defines an environment; it contains the high-level predicates that describe the state of the world as well as the actions of the world including their preconditions and effects on the world's predicate state. The problem file describes a configuration of an environment; it contains the domain name for the environment, the initial objects and true predicates, and the goal state. These files are used with PDDLGym [59] as a backend to create an OpenAI Gym [6] environment which given a state and action can be stepped through to produce the next state.
There are 4 problem files for different example scenarios including cooking a patty and cutting lettuce, preparing ingredients to make a burger, preparing ingredients to make two burgers, and assembling a burger with pre-prepared ingredients. In a scenario, various different tasks can be carried out, such as varying the order and ingredients for making a burger. These problem files contain the minimum number of objects necessary to complete the scenario for any specified task.
One issue with having pre-defined problem files for each scenario is that the code produced in code generation could be hardcoded for a scenario. This is avoided by procedurally generating the problem files. There are two types of procedural generation: noisy randomization and full randomization. Noisy randomization, which is used for every Robotouille experiment in this paper, ensures that the minimum required objects in a problem file appear in an environment in the same grouped arrangement (so an environment with a robot that starts at a table with a patty on it and a cutting board with lettuce on it will maintain those arrangements) but the locations are all randomized and extra stations and items are added (noise). The location of stations and items determines the ID suffix which prevents code generation from always succeeding using hardcoded code.
Full randomization does everything except enforcing that the minimum required objects in a problem file appear in the same grouped arrangement. This would require code that handles edge cases as simple as utilizing ingredients that are already cooked or cut in the environment rather than preparing new ones to more extreme cases such as the kitchen being cluttered with stacked items requiring solving a puzzle to effectively use the kitchen. The simpler case is more appropriate in a real setting and we leave it to future work to remove initial arrangement conditions.
9.1.2 Pipeline Overview
In stage 1 recursive summarization, the LLM first recursively summarizes the provided demonstrations, which are represented as state changes since the previous state, until it determines that the trajectories are sufficiently summarized. For this domain, the LLM in general terminates after it summarizes the trajectory into a series of high-level subtasks. Then, Demo2Code concatenates all trajectories together before prompting the LLM to reason about invariant in subtask's order before generating the task specification.
In stage 2 recursive expansion, there are 3 steps that occur for Demo2Code. First, (1) the task specification is converted directly to code which uses provided helper functions and may use undefined higher-level functions. Second, (2) the undefined higher-level functions are defined potentially including undefined lower-level functions. Finally, (3) the undefined lower-level functions are unambiguously defined.
The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.2.
Figure 8: Examples of goal states with the respective tasks underneath.
9.2 Experiment Setup
In the paper, we categorized the Robotouille simulator into 4 example scenarios. Below are all the scenarios as well as possible tasks, visualized in Fig. 8.
Cook a Patty and cut lettuce
- Cook a patty
- Cut a lettuce
- Cook first then cut
- Cut first then cook
- Assemble two burgers from prepared ingredients
- Assemble two burgers one by one
- Assemble two burgers in parallel
Make a burger
- Stack a top bun on top of a cut lettuce on top of a bottom bun
- Make a burger stacking lettuce atop patty immediately
- Make a burger stacking patty atop lettuce immediately
- Make a burger stacking lettuce atop patty after preparation
- Make a burger stacking patty atop lettuce after preparation
- Make a cheese burger
- Make a chicken burger
- Make a lettuce tomato burger
- Make two burgers
- Cook two patties
- Cut two lettuces
- Make two burgers stacking lettuce atop patty immediately
- Make two burgers stacking patty atop lettuce immediately
- Make two burgers stacking lettuce atop patty after preparation
- Make two burgers stacking Patty atop lettuce after preparation
- Make two cheese burgers
- Make two chicken burgers
- Make two lettuce tomato burgers
9.2.1 Provided Low-Level APIs
- get_all_obj_names_that_MATCH_type(obj_type): return a list of string of objects that match the obj_type.
- get_all_location_names_that_MATCH_type location_type): return a list of string of locations that match the location_type.
- is-cut(obj): return true if obj is cut.
- is_cooked(obj): return true if obj is cooked.
- is_holding(obj): return true if the robot is currently holding obj.
- is_in_a_stack(obj): return true if the obj is in a stack.
- get_obj_that_is_underneath(obj_at_top): return the name of the object that is underneath obj_at_top.
- get_obj_location(obj): return the location that obj is currently at.
- get_curr_location(): return the location that the robot is currently at.
- move(curr_location, target_location): move from the curr_location to the target_location.
- pick_up(obj, loc): pick up the obj from the loc.
- place(obj, loc): place the obj on the loc.
- cut(obj): make progress on cutting the obj. Need to call this function multiple times to finish cutting the obj.
- start_cooking(obj): start cooking the obj. Only need to call this once. The obj will take an unknown amount before it is cooked.
- noop(): do nothing.
- stack(obj_to_stack, obj_at_bottom)): stack obj_to_stack on top of obj_at_bottom.
- unstack(obj_to_unstack, obj_at_bottom): unstack obj_to_unstack from obj_at_bottom.
9.3 Characterize Robotouille's Tasks' Complexity
In table 5, we characterize the complexity of the tasks in terms of the demonstrations' length, the code's length, and the expected code's complexity (i.e. how many loops/conditionals/functions are needed to solve this task).
Table 5: For Robotouille's tasks, we group them by cluster and report the following: 1. number of states in demonstrations (range and average) 2. number of predicates in demonstrations (range and average) 3. number of lines in the oracle Spec2Code's generated code (range and average) 4. average number of loops 5. average number of conditionals 6. average number of functions
Task	Input Demo Length	Code Length	# of loops	# of conditionals	# of functions
# of states	# of predicates
Cook and cut	7-15 (10.75)	8-19 (13.5)	98-98 (98.0)	2.00	12.0	8.00
Cook two patties / cut two lettuces	14-16 (24.3)	19-19 (19.0)	50-54 (52.0)	1.50	6.00	6.00
Assemble two burgers	15-15 (15.0)	36-36 (36.0)	58-62 (60.0)	1.5	6.00	5.00
Make a burger	32-55 (42.6)	26-55 (40.5)	109-160 (146.3)	1.86	17.1	9.86
Make two burgers	38-70 (52.3)	68-114 (86.85)	112-161 (149)	2.86	17.1	9.86
In addition, to bridge the different evaluation standards between planners that generate code and classical planners, we also characterize the Robotouille's tasks based on [41]'s taxonomy in table 6
Table 6: For each Robotouille task, we check if it contains the specification pattern defined in [41].
Task	Global Avoidance	Lower/Exact Restriction Avoidance	Wait	Instantaneous Reaction	Delayed Reaction
Cook a patty	✓	✓
Cook two patties	✓	✓
Stack a top bun on top of a cut lettuce on top of a bottom bun	✓
Cut a lettuce	✓
Cut two lettuces	✓
Cook first then cut	✓	✓	✓
Cut first then cook	✓	✓	✓
Assemble two burgers one by one
Assemble two burgers in parallel
Make a cheese burger	✓	✓	✓	✓
Make a chicken burger	✓	✓	✓	✓
Make a burger stacking lettuce atop patty immediately	✓	✓	✓
Make a burger stacking patty atop lettuce immediately	✓	✓	✓
Make a burger stacking lettuce atop patty after preparation	✓	✓	✓	✓
Make a burger stacking patty atop lettuce after preparation	✓	✓	✓	✓
Make a lettuce tomato burger	✓	✓	✓	✓
Make two cheese burgers	✓	✓	✓	✓
Make two chicken burgers	✓	✓	✓	✓
Make two burgers stacking lettuce atop patty immediately	✓	✓	✓
Make two burgers stacking patty atop lettuce immediately	✓	✓	✓
Make two burgers stacking lettuce atop patty after preparation	✓	✓	✓	✓
Make two burgers stacking patty atop lettuce after preparation	✓	✓	✓	✓
Make two lettuce tomato burgers	✓	✓	✓	✓
10 EPIC-Kitchens Pipeline
10.1 Annotations
We take 9 demonstrations of dishwashing by users 4, 7, 22 and 30, and use 2 of these as in-context examples for the LLM, by writing down each intermediate step's expected output.
Figure 9: Example of annotations for video ID P07_10 in 6 frames
Figure 10: Example of annotations for video ID P04_101 in 6 frames
Predicates are of the form - foo (^<obj>_, ...) where foo is a predicate function like adjective (is\_dirty, is\_soapy etc) or preposition (at, is\_in\_hand, on etc). Each argument is a combination of object name and unique id, the latter added to distinguish multiple objects of the same kind. Note that these annotations or object ids are not available in the EPIC-Kitchens dataset.
Not all predicates are enumerated exhaustively, because this can be difficult for a human annotator, as well as useless and distracting for the LLM. The state predicate annotations in the demonstrations are limited to incremental changes to the observable environment. For example, is\_in\_hand('plate_1') comes after in('plate_1', 'sink_1')
Examples of these incremental state predicate annotations are described in figures 9 and 10. We avoid annotating unnecessary human-like actions like picking up something then immediately placing it back, or turning on a tap momentarily.
10.2 Pipeline Overview
In stage 1 recursive summarization, the LLM first recursively summarizes the provided demonstrations, which are represented as state changes since the previous state, until it determines that the
trajectories are sufficiently summarized. For this domain, the LLM in general terminates after it summarizes the trajectory into a series of high-level subtasks, which each consist of multiple states and low-level actions. For example, low-level actions "Pick up spoon_1", "Pick up fork_1", and "Go from countertop_1 to sink_1" get combined as the subtask "bring spoon_1 and fork_1 from countertop_1 to the sink_1." Then, Demo2Code concatenates all trajectories together before prompting the LLM to reason about the control flow (e.g. whether a for-loop is needed) before generating the task specification.
In stage 2 recursive expansion, because the dishwashing does not use that many unique actions, the LLM is asked to directly use all the low-level actions that are provided as APIs to output the task code given a specification.
The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.3.
10.2.1 Provided Low-Level APIs
- Perception Library
- get_all Objects(): return a list of objects in the environment.
- Low-level Skill Library
- bringobjects_to_location(obj,loc):bringalltheobjectstoloc.
- turn_off(tap_name): turn off tap.
- turn_on(tap_name): turn on tap.
soap(obj): soap the object.
- rinse(obj): rinse the object.
- pick_up(obj): pick up the object.
- place(obj, loc): pick up the object at loc.
- clean_with(obj, tool): clean the object with the tool, which could be a sponge or a towel.
11 Noisy Demonstration Ablation Experiment
As seen in our own annotations for EPIC-Kitchens demonstrations, human annotations or annotations generated by automatic scene summarizers and object detectors may not be noise-free. They may omit some predicates or completely missed predicates in an entire timestep. They may contain objects that the users did not interact with during the demonstration, so predicates about these objects are of little importance to the robot task plan. Thus, we conducted two noisy demonstration ablations:
1. Randomly removing predicates/states from the demonstrations (tested in Robotouille)
2. Randomly adding predicates about irrelevant objects to the demonstrations (tested in EPIC-Kitchens).
We found that:
- Randomly removing predicates/states
- Removing predicates reduces Demo2Code's performance for tasks with short horizons.
- Surprisingly, it does not significantly worsen the performance for tasks with long horizons.
- Randomly adding irrelevant predicates
- Additional irrelevant predicates worsen Demo2Code's performance for correctly generating code for 5 users to 2 users.
11.1 Randomly removing predicates/states
11.1.1 Experimental Details
For each task in Robotouille, we modified the demonstrations in two ways:
1. for each predicate in the demonstration, there is a  $10\%$  probability that the predicate would be removed from the demonstration.
2. for each state (which could consist of multiple predicates), there is a  $10\%$  probability that the entire state would be removed from the demonstration.
We ran the experiment on 4 seeds to report the average and the variance.
11.1.2 Qualitative Result
We analyze a qualitative example (making a burger where the Patty needs to be stacked on top of the lettuce immediately after it is cooked) where removing predicates did not affect Demo2Code's performance.
When each predicate has  $10\%$  probability of being removed, the demonstration is missing 6 predicates, Half of them omits information such as picking up the lettuce, moved from one location to another location, etc. However, the other half does not omit any information. For example, one of the predicate that gets removed is “'robot1’ is not holding ‘top_bun3’”.
State 26: 'top_bun3' is at 'table4' 'top_bun3' is on top of 'patty3' >>>'robot1' is not holding 'top_bun3'<
Removing this predicate does not lose key information because “’top_bun3’ is on top of ‘patty3’” still indicates that ‘top_bun3’ has been placed on top of ‘patty3’. Consequently, the LLM is still able to summarize for that state:
* At state 26, the robot placed 'top_bun3' on top of 'patty3' at location 'table4'.
Thus, Demo2Code is able to generate identical predicates
Using the same seed, when each state has  $10\%$  probability of being completely removed, the demonstration is missing 5 states (9 predicates). Because all the predicate in a selected state gets removed, the LLM misses more context. For example, because the following two states are randomly removed, the LLM does not know that the demonstration has moved and placed 'lettuce1' at 'cuttingboarding1'.
State 3: 'lettuce1' is not at 'table2' 'robot1' is holding 'lettuce1'  
>>> State 4:  
>>> 'robot1' is at 'cutting_board1'  
>>> 'robot1' is not at 'table2'  
>>> State 5:  
>>> 'lettuce1' is at 'cutting_board1'  
>>> 'robot1' is not holding 'lettuce1'
Consequently, it causes the LLM to incorrectly summarize the states and misses the subtask of cutting the lettuce.
* In [Scenario 1], at state 2, the robot moved from 'table1' to 'table2'.
* At state 3-4, the subtask is "pick up lettuce". This subtask contains: 1. picking up 'lettuce1' (state 3)
11.2 Randomly removing predicates/states
11.2.1 Experimental Details
For each EPIC-Kitchens task, we add additional predicates (i.e. showing the position of additional objects in the scene) in at least 2 separate states in the demonstrations. We also do the same
Table 7: Results for Demo2Code's performance on the original EPIC-Kitchens demonstrations v.s. on the demonstrations with additional irrelevant predicates. The unit test pass rate is evaluated by a human annotator, and BLEU score is calculated between each method's code and the human annotator's reference code.
P4-101 (7)	P7-04 (17)	P7-10 (6)	P22-05 (28)	P22-07 (30)	P30-07 (11)	P30-08 (16)
Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.
Demo2Code	1.00	0.33	0.00	0.19	1.00	0.63	1.00	0.43	1.00	0.66	1.00	0.58	0.00	0.24
Demo2Code + additional states	0.00	0.21	0.00	0.15	1.00	0.27	0.00	0.22	0.00	0.49	1.00	0.67	0.00	0.22
modification for the training examples, while keeping the rest of the prompt identical. We expect the LLM to weed out these additional states during recursive summarization.
For example, for annotations of video ID  $P22\_07$  as seen in appendix 17.3.1, we add distractions in 2 states -
State 3:  
at('sink_1')  
is_in_hand('sponge_1')  
+++++ is_in('brush_1', 'sink_1')+++++  
+++++ is_in('bowl_1', 'sink_2')+++++
State 4:  
is_on('tap_1')  
+++++on('jar_1', 'microwave_1')+++++
11.3 Quantitative Analysis
We see that in table 7 that Demo2Code suffers from degradation on most demonstrations when distracting states are added, resulting in only being able to generate correct code for 2 out of 7 demonstrations instead of 5 out of 7 demonstrations.
11.4 Qualitative Analysis
When adding distracting states, the LLM mostly ignores the distracting states and only shows variation in output as a result of change in input. However, the distracting states can interfere with the final code, as can be seen in the output for demonstration P44_101.
Code using clean demonstrations:
1 objects = get_allobjects()
2 for object in objects:
3 pick_up(object)
4 go_to("sink_1")
5 pick_up("sponge_1")
6 turn_on("tap_1")
7 soap(object)
8 rinse(object)
9 turn_off("tap_1")
10 go_to("kitchentop_1")
11 place(object, "kitchentop_1")
Code generated with demonstration that has additional irrelevant predicates:
1 objects = get_all Objects()
2 for object in objects:
3 bring_objects_to_location([object], "sink_1")
4 pick_up("sponge_1")
5 turn_on("tap_1")
6 place("brush_1", "sink_1")
7 place("bowl_1", "sink_2")
8 soap(object)
9 rinse(object)
10 turn_off("tap_1")
11 go_to("kitchentop_1")
When compared to the generated output with clean annotations, we see that while the on(^jar_1^, `microwave_1^) was ignored, in(^brush_1^, `sink_1^) and in(^bowl_1^, `sink_2^) result in the LLM generating additional code that does not align with the demonstrations. Specifically, even though brush_1 and bowl_1 were objects that the users were not interested in interacting with, the LLM has generated a place() code (lines 6-7) for these two objects. This type of mistake could be avoided by adding reasoning during recursive summarization. The LLM can be guided to ignore irrelevant objects and avoid hallucinating actions relating to these objects - for example, ground place action only when both is_in_hand(...) and on(..., loc) are seen one after the other.
12 Chain-of-thought Ablation Experiment
This experiment studies the effect of the chain-of-thought's length (in stage 1 recursive summarization) on the LLM's performance. We found:
- It is helpful to guide the LLM to take small recursive steps when summarizing demonstrations (especially for tasks with long demonstrations).
- The LLM performs the worst if it is asked to directly generate code from demonstrations.
12.1 Experiment Detail
We defined 3 ablation models listed below from the shortest chain-of-thought length to the longest chain length. In addition, because the tabletop's Demo2Code pipeline is different from Robotouille's pipeline, we also describe how these pipelines are adapted to each ablation model:
- No-Cot: Tabletop and Robotouille has exactly the same process of prompting the LLM ONCE to generate code given the language model and the demonstrations.
1-Step
- Tabletop: First, the LLM receives all the demonstrations concatenated together as input to generate the specification without any intermediate reasoning. Next, the LLM generates the code given the specification.
- Robotouille: First, the LLM receives all the demonstrations concatenated together as input to generate the specification. It can have intermediate reasoning because the tasks are much more complex. Next, the LLM generates the high-level code given the specification and recursively expands the code by defining all helper functions.
2-Steps
- Tabletop: First, the LLM classifies the task into either placing task or stacking task. Second, the LLM receives all the demonstrations concatenated together as input to generate the specification without any intermediate reasoning. Finally, the LLM generates the code given the specification.
- Robotouille: First, for each demonstration, the LLM gets its state trajectories as input to identify a list of the low-level action that happened at each state. Second, all the low-level actions from each scenario are concatenated together and used by the LLM to generate the specification. The LLM can have intermediate reasoning at this step because the tasks are much more complex. Finally, the LLM generates the high-level code given the specification and recursively expands the code by defining all helper functions.
We identified 3 clusters of tasks based on the number of states they have, and for each cluster, we selected two tasks to test. For each task and for each of that task's specific requirements, we tested the approach 10 times and took an average of the unit test pass rate.
- Short-horizon tasks (around 2 states): "Place A next to B" and "Place A at a corner"
- Medium-horizon tasks (around 5-10 states): "Place A on top of B" and "Stack all blocks/cylinders (where there might be a maximum stack height)"
- Long-horizon tasks (more than 15 states): "Make a burger" and "Make two burgers"
12 place(object, "kitchenop_1")
Figure 11: Ablation of different degrees of chain-of-thought (recursive summarization) across tasks with short, medium, long horizon.
12.2 Quantitative Result
We include the quantitative result again here as Fig. 11. Overall, Demo2Code/Full performs the best, and there's the general trend that as the length of the chain of length increases, the LLM also generates code that has a higher unit test pass rate. For short-horizon tasks, the difference in the chain-of-thought's length has a small effect on the pass rate because these tasks have short demonstrations that are easy to process without intermediate chain-of-thought steps. In contrast, both medium-horizon tasks and long-horizon tasks see great improvement when the LLM is prompted to take smaller recursive summarization steps. In the next section, we will examine one example from each cluster of tasks to analyze how the length of the chain-of-thought prompts affects the LLM's output.
12.3 Qualitative example for a short-horizon task