Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought
Authors: Huaxiaoyue Wang
Gonzalo Gonzalez-Pumariega
Yash Sharma
Sanjiban Choudhury
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Model
Code Generation
Robotics

Abstract: Language instructions and demonstrations are two natural ways for users to teach robots personalized tasks. Recent progress in Large Language Models (LLMs) has shown impressive performance in translating language instructions into code for robotic tasks. However, translating demonstrations into task code continues to be a challenge due to the length and complexity of both demonstrations and code, making learning a direct mapping intractable. This paper presents Demo2Code, a novel framework that generates robot task code from demonstrations via an extended chain-of-thought and defines a common latent specification to connect the two. Our framework employs a robust two-stage process: (1) a recursive summarization technique that condenses demonstrations into concise specifications, and (2) a code synthesis approach that expands each function recursively from the generated specifications. We conduct extensive evaluation on various robot task benchmarks, including a novel game benchmark Robotouille, designed to simulate diverse cooking tasks in a kitchen environment.

Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought
Huaxiaoyue Wang
Gonzalo Sennzalz-Puuariege
Yash Sharma Cornell University ys749@cornell. edu
Cornell University yukiwang@es.
Cornell University gg387@cornell. edu
wwwkvaangcs.cornel..eee cornell.edu
Sanjiban Choudbury
Coreell University sanjibanc@cornell.edd
Abstract
Language instructions and demonstrations are two natural ways for users to teach robots personalized tasks. Recent progress in Large Language Models (LLMs) has shown impressive performance translating language instructions into code for robotic tasks. However, translating demonstrations into task code continues be challenge due to the length and complexity of both demonstrations and code, making learning direct mapping intractable. This paper presents Demo2Code, a novel framework that generates robot task code from demonstrations via an extended chain-of-thee of-thought and defines common latent specification connect the two. Our framework employs robust two-stage process: (1) a recursive summarization technique that condenses demonstrations into concise specifications, and (2) code synthesis approach that expands each function recursively from the generated specifications. We conduct extensive evaluation on various robot task benchmarks, including novel game benchmark Robotouille, designed to simulate diverse cooking tasks kitchen environment. The project's website is at https://portall-comeelll haa.iall--cooeel...hhbbbb. io/eem22cded/
Introduction
How do we program home robots to perform a wide variety of personalized everyday tasks? Robots must learn such tasks online, through natural interactions with the end user. A user typically communicates a task through a combination of language instructions and demonstrations. This paper addresses the problem of learning robot task code from those two inputs. For instance. in Fig. [ the user teaches the robot how they prefer to make a burger through both language instructions, such as "make a burger", and demonstrations, which shows the order which the ingredients are used.
61 35 have shown that Large Language Models (LLMs) are highly effective in using language instructions prompts to plan robot tasks. However, extending LLMs to take demonstrations as input presents two fundamental challenges. The first challenge comes from demonstrations for long-horizon tasks. Naively concatenating and including all demonstrations in the LLM's prompt would easily exhaust the model's context length. The second challenge that code for long-horizon robot tasks can be complex and require control flow. It also needs check for physics constraints that a robot may have and be able to call custom perception and action libraries. Directly generating such code in single step is error-prone.
Recent works 24 23 33
80
Our key insight is that while demonstrations are long and code is complex, they both share latent task specification that the user had in mind. This task specification is detailed language
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

'Make a burger"
Lang Spec High-Levee Code
Loc):
1f not 1e to_hoddingtob)t
Make
then placa(ch), loc) cook_uctil to_co_edddooooop
Nobest hold
bettyy
Make hurger
def matn():
bettyl
Specificlly
cook p_____ppoppattyy
det move then te____he____eee_ee (lace( loc]
Stade S
betty/l cooked
Cook patty at stovn.
curr_ 1oe i get_nurr__on s4 oure Jao lees
State
lokest boldiig
Stack that letuace
lettuce)
move
place(obs, 100)
betty?l
bottor, Jest
Language Demonstrations
Recursive Summarization
Recursive Expansion
(Demos
Spec)
(High- Level Code + Define Helper Functions)
Overview of Demo2Code that converts language instruction and demonstrations to task code that the robot can execute. The framework recursively summarizes both down a specification, then recursively expands the specification to an executable task code with the helper functions defined.
Figure 1:
description of how the task should be completed. is latent because the end user might not provide all the details about the desired task via natural language. We build an extended chain-of-thougee [73] that recursively summarizes demonstrations to a compact specification, maps it to high-level code, and recursively expands the code by defining all the helper functions. Each step in the chain is small and easy for the LLM to process.
We propose novel framework, Demo2Code, that generates robot task code from language instructions and demonstrations through a two-stage process (Fig. [T] (I) Summarizing demonstrations to task specifications: Recursive summarization first works on each demonstration individually. Once all demonstrations are compactly represented, they are then jointly summarized in the final step as the task specification. This approach helps prevent each step from exceeding the LLM's maximum context length. (2) Synthesizing code from the task specification: Given task specification, the LLM first generates high-level task code that can call undefined functions. It then recursively expands each undefined function until eventually terminating with only calls to the existing APIs imported from the robot's 's low-level action and perception libraries. These existing libraries also encourage the LLM to write reusable, composable code.
Our key contributions are:
1. A method that first recursively summarizes demonstrations specification and then recursively expands specification to robot code via an extended chain-of-thought prompt.
2. A novel game simulator, Robotouille, designed to generate cooking tasks that are complex, long-horizon, and involve diverse food items, for benchmarking task code generation.
3. Comparisons against a range of baselines, including prior state of the [33], on manipulation benchmark, Robotouille as well as real-world human activity dataset.
Related Work
Controlling robots from natural language has a rich history
74
66
37
primarily because it provides a
natural means for humans to interact with robots [5]
30
Recent work on this topic can be categorized
as semantic parsing [39 30
69
67
55
40
68 planning 60, 22
24
61
35
34
28
task
specification 64 19
58 12
reward learning 466
56
7) learning level policies 46
2 57 56 7
imitation learning 25
1. However, these approaches fall in one of two categories: generating open-loop action sequences, or learning closed- loop, but thort-horizon policies. In contrast, we look to generate task code, which is promising in solving Long-horizon tasks with control flows. The generated code also presents an interpretable way to control robots while maintaining the ability to generalize by composing existing functions.
38
58
64 and reinforcement learning 26
18
10
45
Synthesizing code from language too has rich history. Machine learning approaches offer powerful More recently, these tasks are extended to general- and program specifications are fully described in natural English text [21 3, [1].. Pretrained language models have shown great promise in code
techniques for program synthesis 49
14
purpose programming languages 79
78
8
generation by exploiting the contextual representations learned from massive data of codes and
texts [6
9 47). These models can be trained on non-ML_ objectives (20), such as RL 32 to pass unit tests. Alternatively, models can also be improved through prompting methods

---

such as Least-to-Most [2), Think-Step-by-S55 22 or Chain-of- Thain-of-Thought [73], which we leverage in our approach. Closest to our approach is CodeAsPolicies [33], that translates language robot code. We build on it to address the more challenging problem of going from few demonstrations to code.
We broadly view our approach inverting the output of code. This is closely related to inverse graphics, where the goal is to generate code that has produced given image or 3D model [76] 36
15 70 17.. Similar to our approach (65) trains an LSTM model that takes as input multiple demonstrations, compresses it to a latent vector and decodes it to domain specific code. Instead of training custom models to generate custom code, we leverage pre-trained LLMs that can generalize much more broadly, and generate more complex Python code, even create new functions. Closest to our approach [77] uses pre-trained LLMs summarize demonstrations as rules in one step before generating code that creates sequence of pick-then-ples and pick- pick-then-tee then-toss actions. However, they show results on short-horizon tasks with small number of primitive actions. We look at more complex, long-horizon robot tasks, where demonstrations cannot be summarized in one step. We to recursively summarize demonstrations until they are compact.
draw inspiration from 75 50
3 Problem Formulation
We look at the concrete setting where a robot must perform a set of everyday tasks in a home, like cooking recipes or washing dishes, although our approach can be easily extended other settings. We formalize such tasks as a Markov Decision Process (MDP), < S, R >, defined below:
State (s E S) is the set of all objects in the scene and their propositions, e.g. open (obj) ("obj
open"), " on-top (obj1, obj2) ("obj1 1 is on top of obj2").
Action (a E A) is primitive action, c.g. pick(obj ("pick up obj"), place loc)
("place obj on loc"). move (1oc1, 1oc2) ("move from loc1 to loc2").
Transition function (T (.|ss, a)) specifies how objects states and agent changes upon executing an action. The transition is stochastic due to hidden states, e.g. cut ('lettuce') must be
called variable number of times till the state changes is_cut ( ('lettuce').
Reward function (r(s, a)) defines the task, i.e. the subgoals that the robot must visit and constraints that must not be violated.
We assume access to state-based demonstrations because most roboties system have perception modules that can parse raw sensor data into predicate states 42 27 We also assume that system engineer provides a perception library and an action library. The perception library uses sensor observations to maintain a set of state predicates and provides helper functions that use (obj). is cooked(bb)) ). Meanwhile, the action library
these predicates (e.g. bet_obbj_ocatee defines a of actions that correspond to low-level policies, similar to B3 61 77 80).
The goal is to learn a policy uo that maximizes cumulative reward J(TO) = ET [E a.l] 0 being the parameters of the policy. We choose to represent the policy as code 0 for number of reasons: code is interpretable, composable, and verifiable.
at)]
In this setting, the reward function r(s, a) is not explicit, but implicit in the task specification that the user has in mind. Unlike typical Reinforcement Learning (RL), where the reward function is hand designed, it is impractical to expect everyday users to program such reward functions for every new task that they want to teach their robots. Instead, they are likely to communicate tasks through natural means of interaction such language instructions (e.g. "Make burger"). We can either use model to generate reward r(s, a) from / (31 or directly generate the optimal code 0 (33
However, language instructions I from everyday users can be challenging to map to precise robot instructions [63] 44 8I]: they may be difficult to ground, may lack specificity, and may not capture users' intrinsic preferences or hidden constraints of the world. For example, the user may forget to specify how they wanted their burger done, what toppings they preferred, etc. Providing such level of detail through language every time is taxing. A more scalable solution is to pair the language instruction I with demonstrations D = {s1, 82, .r. of the user doing the task. The state at step- t only contains the propositions that have changed from t 1 to t. Embedded in the states are specific details of how the user wants a task done.
Our goal is to infer the most likely code given both the language and the demonstrations: arg max, P(O]1,D). For long-horizon task like cooking, each demonstration can become long. Naively concatenating all demonstrations together to query the LLM can either exhaust the model's
3

---

Algorithm Demo2Code: Generating task code from language instructions and demonstrations Input:
Language instructions lang, Demonstrations demos
Output: Final task code final_code that can be executed def summarize (demos)
if ____sumarr' cemmarrizeed ize (demos):
all_demos
join (demos)
return 11m (swmmary_pppppp proopt,,,pppp all_s
elae:
sunmarized_ _emoedddeeee- []
for demo in demos:
summarized, semmarrrdddddmmss...oo demos.appeni append (11m (Lmmsumarryyprrop., demo))
return summarize (summarized, demos)
def expand_code cooe : if is_ () return code
else:
expanded_ code code for fun in pert_undefing
(code)
fun_code = 11m (onn(cod__ppoonn fun)
expanded_
expand_code (fun_code)
return expanded_code
def main():
apec summarize (demos)
high_level_code 11m pcooode_prropt,, lang spec)
final_code expand_ code leve___eve__code
context length make directly generating the code challenging. We propose approach that overcomes these challenges.
4 Approach
We present framework, Demo2Code, that takes both language instructions and set of demonstrations from a user as input to generate robot code. The key insight is that while both input and output can be quite long, they share latent, compact specification of the task that the user had in mind. Specifically, the task specification is detailed language description of how the task should be completed. Since our goal is to generate code, its structure is similar to a pscudocode that specifies the desired code behavior. The specification is latent because we assume that users do not explicitly define the task specification and do not provide detailed language instructions on how to complete the task.
Our approach constructs an extended ofain-of-thoughh that connects the users' demonstrations to latent task specification, and then connects the generated specification the code. Each step is small and easy for the LLM to process. Algorithm describes descrrbbe our overall approach, which contains two main stages. Stage recursively summarizes demonstrations down to a specification. The specification and language instruction is then converted to high-level code with new, undefined functions. Stage 2 recursively expands this code, defining more functions along the way.
4.1 Stage Recursively Summarize Demonstrations to Specifications
(Scenssii
The order of high level subtaske ls: high level subtask is "cook", becnux::...
Sufficiently Summarized Demonstrations
Input: Demonstrations
Make buzger.
Outpur: Specification
Stats 'robots'
In [Seenarie 1),
3-9,
'tables'
'cobuts" is as "eables"
Spectfically:
State
'puttyd' cooked
state 24-26, the high level subtaak "stack",...
Decide patty to cook.
Cook that patty at that stove.
In (Scenarte 2],
...
stack that top ban that lettuce,
Figure 2: Recursive summarization of input demonstrations compact specification. (Stage 1)

---

The goal of this stage is to summarize the set of demonstrations provided by the user into a compact specification (refer to summarize (demos) in Algorithm[[] Each demonstration is first independently summarized until the L.M determines that the demonstration can no longer be compressed, then the summaries are concatenated and summarized together. Fig. shows example interim outputs during this stage. First, states in each demonstration get summarized into low-level actions (e.g. "patty6 is cooked" is summarized as "robotl cooked patty6.") Then, low-level actions across time-steps are summarized into high-level subtasks, such stacking, cutting, (e.g. "At state 3-8, the high level subtask is cook..."). "). The LLM determines to stop recursively summarizing after the entire demonstration gets converted to high-level subtasks, but this can have different stopping condition (e.g. setting a maximum step) for task settings different than Fig. 2i.. s. Next, these demonstrations' summaries concatenated together for the LLM to generate the task specification. The LLM is prompted to first perform some intermediate reasoning to extract details on personal preferences, possible control loops, etc. For instance, the L.LM aggregates high-level subtasks into ordered list, which empirically helps the model to identify repeated subsets in that list and reason about control loops. An example final specification is shown in Fig. 2 which restates the language instruction first, then states "Specifically: followed by a more detailed instruction of the task.
4.2 Stage 2: Recursively Expand Specification to Task Code
import relavant librariex toon sobit_ inport pic.. from uttle teport get_ahd_
Partially expanded code
del ca at loscess, loo): 1f tos is so.hllingge33):
esRt Jectat scove)
Output: Task Code
Input: Specification
def cook_obs book_objjat__oo st__a__leee lo_[oo loo): if not 1a (obdd] satichh_
ales: sircttr
Make burger. specifically:
nove Le___tte______e_ then_piel((ob]) curr loc - get__urr, loe() obj_loc get_ebi_loocsbb)
Decide patty to cook, Cook that patty at that stove.
clse:
w____hhe___________ piok(obi)
it curr_loc I= obi_loc: nove(cure lsc, obj 1oc)
Stack that
that lettuce.
move move_ then pla.ppaccbebb lec)
cook _oo__aniil_s_ cboked((bb))
pioi_uppob), obi_lon)
Figure 3: Recursive expansion of the high-level code generated from the specification, where new functions are defined by the LLM along the way. (Stage 2)
The goal of this stage is to use the generated specification from stage 1 to define all the code required for the task (see expand_ code (code) in Algorithm [T]. The LLM is prompted to first generate high- level code that calls functions that may be undefined. Subsequently, each of these undefined functions in the code is recursively expanded. Fig. 3 shows an example process of the code generation pipeline. The input is the specification formatted as docstring. We import custom robot perception and control libraries for the LLM and also show examples of how to use such libraries in the prompt. The LLM first generates high-level code, that contain new functions, e.g. book_bbj___ at __loc, that it has not seen in the prompt or import statements before. It expands this code by calling additional functions (e.g. move tere__he___pp (icck, which it defines in the next recursive step. The LLM eventually reaches the base case when it only uses imported APIs to define a function (e.g. move move_then then_ fatten_pickk...
5 Experiments
5.1 Experimental Setup
Baselines and Metries We compare our approach Demo2Code against prior work, CodeAsPoli- cics (33), which we call Lang2Code. This generates code only from language instruction. We also compare against DemoNoLang2Codee that generates code from demonstrations without language instruction, which is achieved by modifying the LLM prompts to redact the language. Finally, we also compare to oracle Spec2Code, which generates task code from detailed specifications on how to complete a task. We use 100-333--uurb--6630 for all experiments with temperature O.
We evaluate the different methods across three metrics. Execution Success Rate is the average 0/1 success of whether the generated code can run without throwing an error. It is independent from whether the goal was actually accomplished. Unit Test Pass Rate is based on checking whether all subgoals are achieved and all constraints are satisfied. The unit test module checks by examining the state transitions created from executing the generated code. Code BLEU score is the BLEU
5

---

Table 1: Results for Tabletop Manipulation simulator. The tasks are categories into 3 clusters: Specificity ("Specific"), Hidden World Constraint (" ("Hidden"), and Personal Preference ("Pref").
Task
Lang2Oode
33
DemoNoLangg2ode
Demo2Code(ours)
Exec.
Pass.
BLEU.
Exec.
Pass.
BLEU.
Exec.
Pass.
BLEU.
Specttic
Place A next to B
1.00
0.33
0.73
0,90
0.80
0.82
1.00
1.00
0.98
Place A at corner of the table
1.00
0.30
0.08
1.00
1.00
0.85
1.00
1.00
1.00
Place A edge of the table
1.00
0.20
0.59
1.00
0.95
0.84
1.00
1.00
0.84
Hidden
Place A on top of B
1.00
0.03
0.23
0.60
0.70
0.56
0.90
0.40
0.40
Stack all blocks
1.00
0.20
0.87
1.00
0.70
0.50
1.00
0.70
0.50
Stack all cylinders
1.00
0.37
0.89
1.00
0.83
0.49
1.00
1.00
1.00
Stack all blocks into one stack
1.00
0.13
0.07
1.00
0.67
0.52
1.00
0.87
0.71
Prefs
Stack all cylinders into one stack
1.00
0.13
0.00
0.90
0.77
0.19
1.00
0.90
0.58
Stack all objects into two stacks
1.00
0.00
0.00
1.00
0.90
0.68
1.00
0.90
0.65
Overall
1.00
0.19
0.39
0.93
0.81
0.60
0.99
0.88
0.77
score [48]
between a method's generated code and the oracle Spec2Code's generated code. We tokenize each code by the spaces, quotations, and new lines.
We build upon a physics simulator from which simulates robot arm manipulating blocks and cylinders different configurations. The task objectives are to place objects at specific locations or stack objects on top of each other. The LLM has access to action primitives (e.g. pick and place) and perception modules (e.g. to get all the objects in the scene). We create range of tasks that vary in complexity and specificity, use the oracle Spec2Code to generate reference code, and execute that code to get demonstrations for other methods. For each task, we test the generated code for 10 random initial conditions of objects.
Tabletop Manipulation Simulator 80 23
80 23
Table 2: Results for the Robotouille simulator. The training tasks in the prompt are at the top of the table and highlighted in gray. All tasks are ordered by the horizon length (the number of states). Below the table shows four Robotouille tasks where the environments gradually become
complex.
Task
Lang2Code
DeneYooaang@Code
Dewe2Caden(wur))
Horizon Length
Exec.
Pass.
BLEB.
Exec.
Pass.
BLEU.
Esec.
Poss.
BLEU.
Cook patty
L.00
1.00
0.90
1.00
1.00
0.90
1.00
1.00
0.90
8.0
Cook patties
0.80
0.80
0:92
0.80
0.80
0.92
0.80
0.80
0.92
16.0
Stack a lop bun on top of cut lettoce on lop of bottom bun
L.00
1.00
0.70
0.00
0.00
0.75
1.00
1,00
0.60
14,0
Oat lefince
1.00
1.00
0.87
0.00
0.00
0.76
1.00
1.000
0.87
7.0
two lettices
0.80
0.00
0.72
Cook first then cut
L.00
1.00
0.88
1.00
1.00
0.85
1.00
1.00
0.88
14,0
then
0.82
Assemble two burgers one by one
0.00
0.00
0.34
1.00
1.00
0.77
1.00
1.00
0.76
15.0
Assemble burgers parnilel
0.00
0.00
0.25
1.00
1.00
Make cheese burger
L.00
0.00
0.24
1.00
1.00
0.69
1.00
1.00
0.69
18.0
Make chicken burger
0.00
0.00
0.57
0.00
0.00
0.61
0.90
0.90
0.69
25.0
Make burger stacking lettace alop potly immodiately
1.00
0.00
0.74
0.20
0.00
0.71
0.00
0.00
0.71
24.5
Make burger stacking patty atop lerrace immediately
0.00
0.00
0.74
0.20
0.00
0.71
1.00
1.000
0.74
25.0
burger stacking
atop party after preparation
0.00
0.10
0.65
Make burger stacking patty stop lettace after preparation
L.00
0.00
0.67
0.00
0.00
0.53
1.00
0.00
0.69
27.0
Make
burger
1,00
0.85
Make two choose burgers
0.00
0.00
0.63
1.000
1.000
0.68
1.000
1.00
0.68
38.01
Make chicken hurgers
0.00
0.00
0.52
0.00
0.68
0.00
0.56
50.0
Make two burgers stncking lettuce stop patty immediately
0.80
0.00
0.66
0.80
1.00
0.69
0.00
0.00
0.66
50,0
Make burgers stacking patty stop lettuce immediattll
0.80
0.00
0.67
1.00
0.00
0.18
1.00
1.00
0.73
50.0
Make two bungers stucking letiuce stop putty after proporation
0.80
0.00
0.66
0.60
0.00
0.66
0.80
0.00
0.67
$4.0
Make two hungers stacking party stop lettice after preparation burgers
0.80
0.00
0.67
0.50
0.00
0.71
0.80
0.00
0.68
$4.0
two
Overall
0.64
0.29
0.64
0.49
0.38
0.71
0.79
0.59
0.74
28.8
Cook and cut
Assemble 2 burgers
Make burger
Make 2 burgers

---

Table 3: Results for EPIC- Kitchens dataset on different user demonstrations of dish-washing (length of demonstration in parentheses). The unit test pass rate is evaluated by a human annotator, and BLEU score calculated between each method's code and the human annotator's 's reference code.
P4-101 (7)
P7-04 (17)
P7-10
P22-05
(28)
P22-07
P30-07 (11)
P30-08 (16)
Pass.
BLEI.
Pass.
BLEU
Pass.
BLEU.
Pass.
BLEU.
Pass.
BLEU
Pass.
BLEUU
Pass.
BLEU.
Lang2Code 55
1.00
0.58
0.00
0.12
0.00
0.84
0.00
0:48
0.00
0.37
1.00
0.84
0.90
0.66
DenelloLang2oode
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.37
0.00
0.51
1.00
0.57
0,00
0.00
Demo2Code
1.00
0.33
0.00
0.19
1.00
0.63
1.00
0.43
1.00
0.66
1.00
0.58
0.00
0.24
Cooking Task Simulator: Robotouille [e introduce a novel, open-source simulator to simulate complex, long-horizon cooking tasks for a robot, e.g. making a burger by cutting lettuces and cooking patties. Unlike existing simulators that focus on simulating physics or sensors, Robotouille focuses on high level task planning and abstracts away other details. We build on a standard backend, PDDLGym (59), with a user-friendly game as the front end to easily collect demonstrations. For the experiment, we create set of tasks, where each associated with a set of preferences (e-g. what a user wants in the burger, how the user wants the burger cooked). For each task and each associated preference, we procedurally generate 10 scenarios.
EPIC-Kitchens Dataset [13 EPIC-Kitchens is real-world, egocentric video dataset of users doing tasks in their kitchen. We use this to test if Demo2Code can infer users' preferences from real videos, with the hopes of eventually applying our approach to teach real robot personalized tasks. We focus on dish washing as we found preferences in it easy to qualify. While each video has annotations of low-level actions, these labels are insufficient for describing the tasks. Hence, we choose 7 videos of 4 humans washing dishes and annotate each demonstration with dense state information. We compare the code generated by Lang2Code, DemoNoLang2Code and Demo2Code on whether it satisfies the annotated preference and how well it matches against the reference code.
5.2 Results and Analysis
Overall, Demo2Code has the closest performance to the oracle (Spec2Code). Specifically, our approach has the highest unit test pass rates in all three benchmarks, as well as the highest execution success in Robotouille (table 2 and EPIC-Kitchens (table 33.. Meanwhile, Lang2Code (33 has a higher overall execution success than Demo2Code for the Tabletop simulator (table However, Lang2Code has the lowest unit test pass rate among all baselines because it cannot fully extract users' specifications without demonstrations. DemoNoLang2Code has relatively higher pass rate, but it sacrifices execution success because it is difficult to output plausible code without context from language. We provide prompts, detailed results and ablations in the Appendix [ We now ask series of questions of the results to characterize the performance difference between the approaches.
How well does Demo2Code generalize to unseen objects and tasks? Demo2Code exhibits its generalization ability in three axes. First, Demo2Code generalizes and solves unseen tasks with longer horizons and more predicates compared to examples in the prompt at train time. For Robotoui llle table 2 shows the average horizon length for each training task (highlighted in gray) and testing task. Overall, the training tasks have an average of 12.7 states compared the testing tasks (31.3 states). Compared to the baselines, Demo2Code performs the best for long ourger-making tasks (an average of 32 states) even though the prompt does not show this type of task. Second, Demo2Code uses control flow, defines hierarchical code, and composes multiple subtasks together to solve these long-horizon tasks. The appendix details the average number of loops, conditionals, and helper functions that Demo2Code generates for tabletop simulator (in section (.3) and Robotouille (in section 99.3). Notably, Demo2Code generates code that uses a for- for the longest task (making two lettuce tomato burgers with 70 states), which requires generalizing to unseen subtasks (e.g. cutting tomatoes) and composing 7 distinct subtasks. Third, Demo2Code solves tasks that contain unseen objects or different number of objects compared to the training tasks in the prompt. For Robotoui 11e, the prompt only contains examples of preparing burger patties and lettuce, but Demo2Code still has the highest unit test pass rate for making burgers with unseen ingredients: cheese, chicken, and
I 'Codebase and usage guide for Robotouille 1e is available here: http:///giibbbooo/ooooo////////a "Codebase is available here: http://github.commpoottllll http://///ooott.toooo//////eeeea comnell/eemo22odee
7

---

LangãCode
Deno2Cade
DenoNoLangioooo
Block
'grass blreh
right
"porple oyt
lomen black
Uiddrr
(prrple
Marck
puppa 9342
sec. block
[Ronsacio 3]
Bomaria
geess 2lech ppopie
Hunder
green block
purple cyih
blckk
1o0 pos parse position
100_pos paxse position
100_poe parrseposition
gightt of the green block')
"left of the green blook')
"gight. of the purple cyl inder""
pat first on second I ppople cylinder', loe_
put firafirattc. on second purple
put firet on, second green block
cylindec', loc_poa)
loe_pox)
Figure 4: Demo2Code successfully extracts specificity in tabletop tasks. Lang2Code lacks demonstrations and lacks context in what the demonstrations are for.
randomly chooses a spatial location while
User Prefers
pally
polrr
Pick patty]
Cook
Pick
lnturel
main( by
det moin():
berpet.
run pitty
pasty pastieafo)
paity pastles[]]
paicy cosc.
cosL cb Joc(pstty, storex(3))
at stoves(d])
chease
letture lettures(0)
cheess chreses(e)
ont att__tt___tttttte Merrerrltttsee boarda(o])
(hat wwwt_annneeeeeeee www.nnnnnneehheeeeeee pasty)
patty.
oagt (ljpslesse,,,, patsy)
ceookeeeeeLnneesstttoooooo cheese)
ttack tha tep bes that owees.
stuck cn wttpoottttttpp.oo bun, lettuce)
Puach that ty hes insture
User 2: Prefers cheese on patty
A
Jentes' cokke
baldnn vimee
yolest Stack cheese!
Tee eac
Pick up patsyl
Cook pattyl
Stack
Pick up cheesel
Stack top bun]
Figure 5: Demo2Code summarizes demonstrations and identify different users' preferences how to make burger (e.g. whether include lettuce cheese) in Robotouille simulator. Then, it generates personalized burger cooking code to use the user's preferred ingredients.
tomatoes. Similarly, for tabletop (table D although the prompt only contains block-stacking tasks, our approach maintains high performance for cylinder-stacking tasks.
Is Demo2Code able to ground its tasks using demonstrations? Language instructions sometimes cannot ground the tasks with specific execution details. Since demonstrations provide richer informa- tion about the task and the world, we evaluate whether Demo2Code can utilize them to extract details. Tasks under the "Specific" cluster in Table sssow cases when the LLM needs to use demonstrations to ground the desired goal. Fig. dillustrates that although the language instruction ("Place the purple cylinder next the green block" does not ground the desired spatial relationship between the two objects, approach is able to infer the desired specification ('to the left"). In contrast, Lang2Code can only randomly guess a spatial relationship. while DemoNoLang2Code can determine the relative position, but moved the green block because it does not have language instruction to ground the overall task. Similarly, tasks under the "Hidden" cluster in Table how Demo2Code outperforms others in inferring hidden constraints (e.g the maximum height of a stack) ground its tasks.
outpes.
Is Demo2Code able to capture individual user preference? As pipeline for users to teach robots personalized tasks, Demo2Code is evaluated on its ability to extract a user's preference. Table shows that our approach performs better than Lang2Code in generating code that matches each EPIC-Kitchens user's dish washing preference, without overfitting to the demonstration like in DemoNoLang22ode. Because we do not have simulator that completely matches the dataset, human annotators have manually inspect the code. The code passes the inspection if has correct syntax, does not violate any physical constraints (e.g. does not rinse dish without turning on the tap), and matches the user's 's dish-washing preference. Qualitatively, Fig. shows that our approach is able to extract the specification and generate the correct code respectively for user 22, who prefers to soap

---

User 22: Prefers to first soap all objects and then rinse
Most you 13 aayy pecc
we siaad WmL.33
Soapl bowl
soapt glass 11
rinsa bowl
placel bood
plcoo 11)
get get_aal_cbjja()
pisa_upt""ccub_..
objj art_ailLotjee
for obj aboss pick obs))
tick up accib,2
Fee each object objectsr
unep object
place(obs, "einn_ t"ag__"")
geuset "wank
turn "tap_A")
Thr and dojecs in degaoie:
fuom sarvess Sounsden KAMLS
Darn leg.1
sosp(obs)
Fer object objector
riseeç
Biase objact
place(obs, "dtatsankk_"" tum_aft("apps""
turn_ot(("tap ("tpp_"""""))
19y2
place(ab)) "counter__""
Zare off tep
User 30: Prefers to soap and rinse each object
Med
S0ap: mag
nisso mug 1')
placol mug.
pickup[
placo[]ux 11)
Figure 6: Demo2Code summarizes different styles of users washing dishes from demonstration (how soap rinse objects) in EPIC-Kitchens, and generates personalized dish washing code.
Short
Medium
Long
0.87
0.30
Unit Test Pass Rate
0.68
0.75
original
predicates, drop prob
drop
0.60
Cook and cut
1.00
0.81 (0.24)
0.94 (0.13)
Cook two putties / Cut two lettuces
0.80
0.75 (0.17)
0.73 (0.22)
0.43
Assemble two burgers
0.50
0.38 (0.25)
0.50 (0.00)
Make burger
0.49
0.65 (0.07)
0,38 (0.31)
Make burgers
(0.13)
(0.14)
0.19
Overall
0.59
0.62 (0.05)
0.59 (0.12)
0.03
0.00
0.06
No-COT 1-Step
2-Steps
Full
Figure 7: (Left) Unit test result for ablating different degrees of chain-of- across tasks with short, medium, long horizon. (Right) Demo2Code's 's unit test result for Robotouille demonstrations with different level of noises: (1) each predicate has 10% chance of being dropped, and (2) cach has 10% chance of being completely dropped. We ran the experiment 4 times and report the average and variance.
all objects before rinsing them, and user 30, who prefers to soap then rinse each object individually. Similarly, Fig. provides an example of how Demo2Code is able to identify a user's 's preference of using cheese vs lettuce even when the language instruction is just "make a burger." Quantitatively, Table 2 shows more examples of our approach identifying user's preference in cooking order, ingredient choice, etc, while Table aalso shows our approach performing well in tabletop tasks.
How does thain-of-thought compare to directly generating code from demonstrations? To evaluate the importance of our extended chain-of-thon thoutht pipeline, conduct ablation by varying the length of the chain on three clusters of tasks: short-horizon (around 2 states), codiummhhriizon (5-10 states), and long-horizon (2 15 states). We compare the unit test pass rate four different chain lengths, ranging from No chain-of-though (the shortest), which directly generates code from demonstrations, to Full (the longest), which represents approach Demo2Code. The left bar plot in Fig. 7 shows that directly generating code from demonstrations is not effective, and the LLM performs better as the length of the chain increases. The chain length also has larger effect on tasks with longer horizons. For short-horizon tasks, the LLM can easily process the short demonstrations and achieve high performances by just using 1-step. Meanwhile, the stark difference between 2-steps and Full's 's results on long-horizon tasks emphasizes the importance of taking as many small steps as the LLM needs in summarizing long demonstrations so that it will not lose key information.
study how Demo2Code performs (1) when each predicate has a 10% chance to be removed from the demonstrations, and (2) when each state has a 10% chance to completely removed. Fig. 7s table shows that Demo2Code's overall performance does not degrade even though demonstrations missing information. While
How do noisy demonstrations affect Demo2Code's performance?

---

removing predicates or states worsen Demo2Code's performance for shorter tasks (e.g. cook and cut). they surprisingly increase the performance for longer tasks. Removing any predicates can omit essential information in shorter tasks' demonstrations. Meanwhile, for longer tasks, the removed predicates are less likely to be key information, while reducing the length of demonstrations, Similarly, for the longest tasks to make two burgers, one burger's missing predicates or states can be explained by the other burger's demonstration. In section I1 we show a specific example of this phenomenon. We also study the effect of adding additional predicates to demonstrations, which has degraded Demo2Code's s performance from satisfying 5 users' preferences to 2 users' in SPIC-Kitchens.
Discussion
In this paper, we look the problem of generating robot task code from combination of lan- guage instructions and demonstrations. We propose Demo2Code that first recursively summarizes demonstrations into a latent, compact specification then recursively expands code generated from that specification to a fully defined robot task code. We evaluate our approach against prior state- of-the-art [33 that generates code only from language instructions, across distinct benchmarks: a tabletop manipulation benchmark, a novel cooking game Robotouille, and annotated data from EPIC-Kitchens, a real-world human activity dataset. We analyze various capabilities of Demo2Code, such as grounding language instructions, generalizing across tasks, and capturing user preferences.
Demo2Code can generalize across complex, long-horizon tasks. Even though Demo2Code was shown only short-horizon tasks, it's able to generalize to complex, long demonstrations. Recursive summarization compresses long chains of demonstrations and recursive expansion generates complex, multi-layered code.
Demo2Code leverages demonstrations to ground ambiguous language instructions and infer hidden preferences and constraints. The latent specification explicitly searches for missing details in the demonstrations, ensuring they do not get explained away and are captured explicitly in the specification.
Demo2Code strongly leverages chain-of-thought Given the complex mapping between demon- strations and code, seaii-of-thhugeee plays a critical role in breaking down computation into small manageable steps during summarization, specification generation and code expansion.
In future directions, we are looking close the loop on code generation learn from failures, integrate with a real home robot system and run user studies with Robotouille.
7 Limitations
Demo2Code limited by the capability of LLMs. Recursive summarization assumes that once all the demonstrations are sufficiently summarized, they can be concatenated to generate specification. However, in extremely long horizon tasks (e-g. making burgers for an entire day), possible that the combination of all the sufficiently summarized demonstrations can still exceed the maximum context length. A future work direction is to prompt the LLM with chunks of the concatenated demonstrations and incrementally improve the specifications based on each new chunk. In recursive expansion, our approach assumes that all low-level action primitives are provided. Demo2Code currently cannot automatically update its prompt to include any new action. Another direction is to automatically build the low-level skill libraries by learning low-level policy via imitation learning and iteratively improve the code- getereeenraiiinn prompt over time. Finally, since LLMs not completely reliable and can hallucinate facts, it is important to close the loop by providing feedback to the LLM when they fail. One solution 62 52) is to incorporate feedback in the query and reprompt the language model. Doing this in a self-supervised manner with a verification system remains an open challenge.
In addition, the evaluation approach for Demo2Code other planners that generate code [33 6 different from the one for classical planners [53] 54). Planners that generate code measure a task's complexity by the horizon length, the number of control flows, whether that task is in the training dataset, etc. Meanwhile, many classical planners use domain specific languages such as Linear Temporal Logic (LTL) to specify tasks [4I,, which leads to categorizing tasks and measuring the task complexity based on LTL. Future work needs to resolve this mismatch in evaluation standards.
10

---

Acknowledgements
We sincerely thank Nicole Thean for creating assets for Robotouille. This work was supported in part by the National Science Foundation FRR (#2327973).
References
[1] Ahmed Akakzia, Cédric Colas, Pierre- Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-aqquiee skills via goal generation. arXiv:2006.077 07/88, 2020.
[2] Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. arXiv:1711.0048.. 2017.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv:2108.077: 077773 2021.
[4] Matej Balog, Alexander L Gaunt, Mare Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989 2016.
[5] Cynthia Breazeal, Kerstin Dautenhahn, and Takayuki Kanda. handbook of robotics, 2016.
Social robotics. Springer
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jic Tang, and Wojciech Zaremba. Openai gym, 2016.
[7] Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning Generalizable Robotic Reward Functions from "In-The-Wild Human Videos. In Proceedings of Roboties: Science and Systems, Virtual, July 2021.
[8] Mark Chen, Jerry Tworek, Heewoo Jun. Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.
[9] Qibin Chen, Jeremy Lacomis, Edward J. Schwartz, Graham Neubig, Bogdan Vasilescu, and Claire Le Goues. Varclr: Variable semantic representation pre-training via contrastive learning, 2021.
[10] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Self-educated language agent with hindsight experience replay for instruction following. DeepMind, 2019.
[11] Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 9062-9065, Online, November 2020. Association for Computational Linguistics.
[12] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pages 893-905. PMLR, 2022.
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evan- gelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-11 International Journal of Computer Vision (IJV), 130:33-55, 2022.
[14] Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference machine learning, pages 990 998. PMLR, 2017.
[15] Kevin Ellis, Daniel Ritchie, Armando Solar- and Josh Tenenbaum. Learning to infer graphies programs from hand-drawn images. Advances neural information processing systems, 31, 2018.

---

[16] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: pre-trained model for pro- gramming and natural languages. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 1536-1547, Online, November 2020. Association for Computational Linguistics.
[17] Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Ali Eslami, and Oriol Vinyals. Synthe- sizing programs for images using reinforced adversarial learning. In International Conference on Machine Learning, pages 1666-1675. PMLR, 2018.
[18] Prasoon Goyal, Scott Nickum, and Raymond Mooney. PixI2r: Guiding reinforcement learning using natural language by mapping pixels to rewards. 2047: 2020.
[19] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (CoRL), 2019.
[20] Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. From language to pro- grams: Bridging reinforcement learning and maximum marginal likelihood. arXiv preprint arXiv:104.007926. 2017.
[21] Dan Hendrycks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puraník, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.099 2021.
[22] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv:2201.07207, 2022.
[23] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv:2207.05600 2022.
[24] Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do i can, not as say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022.
[25] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lyneh, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In CoRL, 2022.
[26] Yiding Jiang, Shixiang Shane Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning, NeurIPS, 2019.
[27] Kei Kase, Chris Paxton, Hammad Mazhar, Tetsuya Ogata, and Dieter Fox. Transferable task execution from pixels through deep planning domain learning, 2020.
[28] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks, 2023.
[29] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. 20X.: 2022.
[30] Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In HRI, 2010.
12

---

[31] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023.
[32] Hung Le, Yue Wang, Akhilesh Deepak Goumare, Silvio Savarese, and Steven CH Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. arXiv:2207.017800 2022.
[33] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.0775.. 2022.
[34] Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models, 2023.
[35] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions feasible plans, 2023.
[36] Yunchao Liu, Jiajun Wu, Zheng Wu, Daniel Ritchie, William T. Freeman, and Joshua B. Tenenbaum. Learning to describe scenes with programs. In International Conference on Learning Representations, 2019.
[37] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, S. Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In IJCAI, 2019.
[38] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv:2005.0766 076488 2020.
[39] Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. AAAI, 2006.
[40] Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse natural language commands to a robot control system. In Experimental robotics, 2013.
[41] Claudio Menghi, Christos Tsigkanos, Patrizio Pelliccione, Carlo Ghezzi, and Thorsten Berger. Specification patterns for robotic missions, 2019.
[42] Toki Migimatsu and Jeannette Bohg. Grounding predicates through actions, 2022.
[43] Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading com- prehension through question decomposition and rescoring. arXiv preprint arXiv: arXiv:1906.022: 1906.0296. 2019.
[44] Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions 3d environments with visual goal prediction. arXiv preprint arXiv: 1809.00786 2018
[45] Dipendra Kumar Misra, John Langford, and Yoav Artzi Mapping instructions and visual observations to actions with reinforcement learning. CoRR, abs//104.00875, 2017.
[46] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. CoRL, 2022.
[47] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 1347,4 2022.
[48] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
[49] Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. arXiv preprint arXiv:1611.01855, 2016.
13

---

[50] Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho. and Douwe Kiela. Unsupervised question decomposition for question answering. arXiv preprint arXiv:2002.09758, 2020.
[51] Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In Internationaa Conference on Learning Representations, 2022.
[52] Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting, 2022.
[53] Ankit Shah, Pritish Kamath, Julie Shah, and Shen Li. Bayesian inference of temporal task specifications from demonstrations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. (esa-Bianchi and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[54] Ankit Shah, Shen Li, and Julie Shah. Planning with uncertain specifications (PUnS). IEEE Robotics and Automation Letters, 512):3414-3421, apr 2020.
[55] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action, 2022.
[56] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Leaninn manipulation concepts from instructions and human demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2020.
[57] Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv:2204.00 086, 2022.
[58] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In CoRL, 2021.
[59] Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems, 2020.
[60] Tom Silver, Varun Hariprasad, Reece Shuttleworth, Nishanth Kumar, Tomás Lozano-Péres and Leslie Pack Kaelbling. PDDL planning with pretrained large language models. In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
[61] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
[62] Marta Skreta, Naruki Yoshikawa, Sebastian Deelano-RRubchh Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, 2023.
[63] Shawn Squire, Stefanie Tellex Dilip Arumugam, and Lei Yang. Grounding english commands to reward functions. In Robotics: Science and Systems, 2015.
[64] Simon Stepputtis, Joseph Camphell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Longuage-condditoone imitation learning for robot manipulation tasks, NeurIPS. 2020.
[65] Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthesis from diverse demonstration videos. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages
PMLR, 10-15 Jul 2018.
[66] Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gard and Cynthia Matuszek. Robots that use language. Review of Control, Robotics, and Autonomous Systems, 2020.
[67] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011.
14

---

[68] Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker. Yuqian Jiang. Harel Yedidsion, Justin Hart, Peter Stone, and Raymond Mooney. Jointly improving parsing and perception for natural language commands through human-robot dialog. JAIR, 2020.
[69] Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the 2015 International Joint Conference on Artificial Intelligence (IJCAI), pages 1923-1929, Buenos Aires, Argentina, July 2015.
[70] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning infer and execute 3d shape programs. In International Conference on Learning Representations, 2019.
[71] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. http:////ggatabb..ooo/o/ooooooo/
pransformer-J] May 2021.
[72] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, 2021.
[73] Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Brian Ichter. Fei Xia, Ed Chi, Quoc Lc. and Denny Zhou Chain of thought prompting elicits reasoning in large language models. arXiv:2201. / 2022.
[74] Terry Winograd. Procedures representation for data in computer program for understand- ing natural language. MIT PROJECT MAC, 1971.
[75] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv:2109. 2021.
[76] Jiajun Wu, Joshua B. Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[77] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng. Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models, 2023.
[78] Xiaojun Xu, Chang Liu, and Dawn Song. SQLNet: Generating structured queries from natural language without reinforcement learning, 2018.
[79] Pengcheng Yin and Graham Neubig. syntactic neural model for general-purpose code gen- eration. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 440-450, Vancouver, Canada, July 2017. Associa- tion for Computational Linguistics.
[80] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv:2204.00559, 2022.
[81] Luke Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv: /420, 2012.
[82] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. 2rXiv:2205.1062) 2022.
15

---

Appendix
Table of Contents
8
Tabletop Manipulation Simulator Pipeline
17
8.1
Pipeline Overview
17
8.2
Experiment Setup
17
8.3
Characterize Tabletop Tasks' Complexity
18
9
Robotouille Simulator Pipeline
19
9.1
Overview
19
9.2
Experiment Setup
................................
21
9.3
Characterize Robotouille's Tasks' Complexity
................................
22
PPIC-Kitchens Pipeline
22
10.1
Annotations
................................
22
10.2
Pipeline Overview
................................
23
11
Noisy Demonstration Ablation Experiment
24
Randomly removing predicates/states
24
11.2
Randomly removing predicates/states
25
11.3
Quantitative Analysis
26
11.4
Qualitative Analysis
26
12
Chain-of-thought Ablation Experiment
27
12.1
Experiment Detail
27
12.2
Quantitative Result
28
12.3
Qualitative example for a honzonn task
................................
28
12.4
Qualitative example for a medium-horizon task
................................
29
12.5
Qualitative example for a long-horizon task
30
Intermediate Reasoning Ablation Experiment
35
13.1
Experiment detail
................................
35
13.2
Quantitative resull
................................
35
13.3
Qualitative example
36
14
Recursive Expansion Ablation Experiment
40
14.1
Experiment detail
40
14.2
Quantitative result
40
14.3
Qualitative example
41
15
Broader Impact
44
16
Reproducibility
45
Demo2Code Example Output
45
17.1 Tabletop Simulator Example
45
17.2 Robotouille Example
48
17.3 EPIC-Kitchens Example
53
18 Prompts
57
18.1 Tabletop Manipulation Task Prompts
57
18.2 Robotouille Task Prompts
69
16

---

18.3 EPIC Kitchens Task Prompis
87
I9 Other Long Examples
97
19.1
Example Robotouille Query
97
19.2 Example EPIC- Kitchens Query
19.3
Intermediate Reasoning Ablation Helper Functions
103
8
Tabletop Manipulation Simulator Pipeline
8.1 Pipeline Overview
The tabletop manipulation simulator contains simple tasks. Consequently, the demonstrations do not have too many states (S states) and the code is not complex. Thus, Demo2Code 's prompt for this domain does need a long extended chain-of-thought In stage recursive summarization, the LLM just needs to summarize each states into sentences that describes the low-level action (e.g. move, pick, place, etc.) In stage recursive expansion, because the code simple, the LLM can directly use all the low-level actions that are provided to output the task code given specification.
The prompt demonstrating this pipeline is listed at the end of the appendix in section [8.1
8.2 Experiment Setup
In the paper, we categorize the tabletop tasks into three clusters. For each cluster, we list all the tasks and their variants of possible requirements below. The tasks that are used in the prompt are bolded.
Specificity
Place A next to B
* No hidden specificity: A can be placed in any relative position next B
must be the left of
A must be the right of B
* A must be behind B
* A must be in front of B
Place A at corner of the table
No hidden specificity: A can be placed at any corner.
* A must be at the top left corner
* A must be at the top right corner
* A must be at the bottom left corner
* must be the bottom right corner
Place at an edge the table
* No hidden specificity: A can be placed at any corner.
* must be at the top edge
# must be the bottom edge
must be the left edge
A must be at the right edge
Hidden Constraint
Place on top of B
* No hidden constraint: A can be directly placed on top of B in one step
* There additional object on top of A, so that needs to be removed before placing
A top of B.
* There are 2 additional objects on top of A.
*
There are 3 additional objects on top of A.
Stack all blocks
No hidden constraint: All blocks be stacked into stack
* Each stack can be at most 2 blocks high
17

---

*
Each stack can be at most blocks high
*
Each stack can be at most 4 blocks high
Stack all cylinders (Same set of hidden constraints "stack all blocks." None of the examples appears in the prompt.)
Personal Preference
Stack all blocks into one stack
* 2 blocks must stacked a certain order, and the rest can be unordered
* 3 blocks must stacked
certain order
* All blocks must be stacked in certain order
Stack all cylinders into one stack (Same set of hidden constraints as "stack all blocks into stack" None of the examples appears in the prompt.)
-Stack all objects
* No hidden preference: The objects not need to be stacked in to different
stacks based on their type
* All the blocks should be stacked in one stack, and all the cylinders should be stacked another stack
8.2.1 Provided Low-Level APIs
We have provided the following APIs for the perception library and low-level skill library:
Perception Library
get get_obb__nmmeel (): return a list of objects in the environment
get_all_oby aaal_loj__aame_____aaaaaaaaaeeee _oooi___mm_____eae__ ttat_thtt_taae that_match__tpppp type (type_name) objects_ list): return list of objects the environment that match the type_name.
ooreinngoorrrrroe (objectobjett_ttee catrs__t__eefrrreeeee enforce_orderr
brbecta_without.. witeeetttttttoorrrr........ return sorted list of objects stack.
Low-level Skill Library
put put_ufirtt_o__e sist__oonsoocotteeee (argi, arg2): pick up object (arg1) and put at arg2. If arg2 an object, arg1 will be on top of arg2. If arg2 is 'table', argi will be somewhere random the table. If arg2 is list, argi will be placed at location [x, y]. stack the list of objects_to_stacee into one stack without considering height limit. st_ck,, height_limit): stack the list of @bjects_to_stadt into potentially multiple stacks, and each stack has a maximum height based on neight_limit. limit.
wwwrithotthhhghhhte limit teteeess_tsstttteeeeeeeeeeesssse
stack hta._witthhheiggeee httptt_lmmiittieeeeettt
limit):
8.3 Characterize Tabletop Tasks' Complexity
In table 4 we characterize the complexity of the tasks in terms of the demonstrations' length, the code's length, and the expected code's complexity (i.e. how many loops/codiitional//tuntiissss needed to solve this task).
Table 4: For tabletop tasks, group them by cluster and report: 1. number of states in demonstrations (range and average) number predicates in demonstrations (range and average) number lines in the oracle Spec2Code's '' generated code (range and average) 4. average numher of loops average number conditionals 6. average number of functions
Tisk
Input Demo Length of predicates
Code Length
of loops
conditionals
of lunctions
of states
Place next to B
1-1 (1.00)
235(3
3-7 (3.38)
0.00
0.02
1.00
Place comeniedge top ofB
1-1 (1.00)
12.09))))
(3.03)
0.00
0.00
1.00
23.65)
Stack all blocksleylinders
(4.43)
4-33 (14.09)
3-15 (4.44)
0.24
0.06
1.00
Stack all blocks/eylinder into one stack
3.5.4 (3.98)
12:23 (14.77)
(22)
1.00
1.00
1.00
objects two stacks
(6.95)
(23.90)
7-25
18

---

9 Robotouille Simulator Pipeline
9.1
Overview
9.1.1 Simulator Description
In Robotouille, robot chef performs cooking tasks in a kitchen environment. The state of the kitchen environment consists of items such as buns, lettuce, and patties located on stations which could be tables, grills, and cutting boards. The actions of the robot consist of moving around from one station to another, picking items from and placing items on stations, stacking items atop and unstacking items from another item, cooking patties stoves, and cutting lettuce cutting boards. The state and actions are described through the Planning Domain Description Language (PDDL).
These PDDL files consist of domain and a problem. The domain file defines environment; it contains the high-level predicates that describe the state of the world as well as the actions of the world including their preconditions and effects the world's predicate state. The problem file describes a configuration of an environment; it contains the domain name for the environment, the initial objects and true predicates, and the goal state. These files are used with PDDLGym [59] as backend to create an OpenAl Gym [6] environment which given a state and action can be stepped through produce the next state.
There are problem files for different example scenarios including cooking patty and cutting lettuce, preparing ingredients make a burger, preparing ingredients to make two burgers, and assembling burger with pre- prepared ingredients. In a scenario, various different tasks can carried out, such as varying the order and ingredients for making burger. These problem files contain the minimum number objects necessary to complete the scenario for any specified task.
One issue with having pre-defined problem files for each scenario is that the code produced in code generation could be hardcoded for a scenario. This is avoided by procedurally generating the problem files. There are two types of procedural generation: noisy randomization and full randomization. Noisy randomization, which used for every Robotouille experiment this paper, ensures that the minimum required objects in problem file appear in an environment in the same grouped arrangement (so an environment with robot that starts at a table with a patty on and a cutting board with lettuce on it will maintain those arrangements) but the locations are all randomized and extra stations and items are added (noise). The location stations and items determines the ID suffix which prevents code generation from always succeeding using hardcoded code.
Full randomization does everything except enforeing that the minimum required objects problem file appear in the same grouped arrangement. This would require code that handles edge cases as simple as utilizing ingredients that are already cooked or cut in the environment rather than preparing new ones to more extreme cases such the kitchen being cluttered with stacked items requiring solving puzzle effectively use the kitchen. The simpler case is more appropriate in a real setting and we leave it to future work to remove initial arrangement conditions.
9.1.2 Pipeline Overview
In stage recursive summarization, the LLM first recursively summarizes the provided demonstra- tions, which are represented as state changes since the previous state, until it determines that the trajectories are sufficiently summarized. For this domain, the I.LM in general terminates after summarizes the trajectory into a series of high-level subtasks. Then, Demo2Code concatenates all trajectories together before prompting the LLM to reason about invariant in subtask's 's order before generating the task specification.
In stage 2 recursive expansion, there are 3 steps that occur for Demo2Code. First, (1) the task specification is converted directly to code which uses provided helper functions and may undefined ligeer-level functions. Second, (2) the undefined higher-level functions are defined potentially including undefined lower-level functions. Finally, (3) the undefined lower-level functions unambiguously defined.
The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.2
19

---

1. Cook first then cut
Make burger stacking lettuce atop patty immedistely
1. Make burger stacking patty atop lettuce immediataly
Make cheese burger
2. Out first then cook
burger stacking lettuce atop patty after preparation
burger stacking atop lettuce after preparation
1. Make chicken burger
1. Make letluce tomato burger
1. Cook two patties
Cut two lettuces
Make burgers stacking lettuce atop patty immediataly
Make burgers stacking patty atop lettuce immedistely
1. Make cheese burgers
Make two chicken burgers
burgers stacking lettuce atop patty after preparation
burgers stacking patty atop lettuce after preparation
1. Make two lettuoe tomato burgers
1. Assemble two burgers one by
Assemble burgers parallel
Figure 8: Examples of goal states with the respective tasks underneath.
20

---

9.2 Experiment Setup
In the paper, we categorized the Robotouille simulator into example scenarios. Below are all the scenarios as well as possible tasks, visualized in Fig. g
Cook patty and cut lettuce
Cook patty
Cut lettuce
Cook first then
Cut first then cook
Assemble two burgers from prepared ingredients
Assemble two burgers one by one
Assemble two burgers in parallel
Make burger
Stack a top bun on top of a cut lettuce on top of a bottom bun
Make a burger stacking lettuce atop patty immediately
Make burger stacking patty atop lettuce immediately
Make burger stacking lettuce atop patty after preparation
Make burger stacking patty atop lettuce after preparation
Make cheese burger
Make chicken burger
Make a lettuce tomato burger
Make burgers
Cook two patties
Cut two lettuces
Make two burgers stacking lettuce atop patty immediately
Make two burgers stacking patty atop lettuce immediately
Make two burgers stacking lettuce atop patty after preparation
Make two burgers stacking patty atop lettuce after preparation
Make two cheese burgers
Make two chicken burgers
Make two lettuce tomato burgers
9.2.1 Provided Low-Level APIs
We have provided the following APIs for the perception library and low-level skill library:
Perception Library
get get_all_obj a_l___oj__aameee______a_aaea_a__ aabj_names__eeeeeeee_eee ttatttatt_____ sact__matc_ttttt thattmmatch_tyyeeooooo..... (obji_ttyy)))): return a list of string of objects that match the obj_type.
all__loccation nam_nnam_____ nttee_atatttttaaaa__aa_aaraaaaat ttattthattaaa www.sattchhheeee type (location_typaa typee): return list of string of locations that match the location_type.
is_cut (obj): return true if obj is cut.
is_ ces_coeedddoo (obj): return true if obj cooked.
is ss_hoodiiggoobi] return true the robot currently holding obj.
is iiiiinn_staaa in_na__sttaccobb) return true if the obj is stack.
get get__ojj_ttt htt.__bb__haa_ieaamdeeeeeeaaaal
return the name of the object that is
underneath obj_at_top.
get__chj_locationnobb)).. return the location that obj is currently at.
gett_curr_locatioonn return the location that the robot is currently at.
Low-level Skill Library
moveeccur:.. target_loc): move from the curr uur_loc to the target_loc
21

---

pick_up(obj, loc): pick up the obj from the loc.
(obc loc): place the obj on the loc.
cut (obj): make progress cutting the obj. Need to call this function multiple times to finish cutting the obj.
start cooking the obj. Only need to call this once. The obj will take an unknown amount before cooked.
noop(): (): do nothing.
bgj_at_bottom)>
stack
obj_ tottt_sstck on top of
obj_at_bottom
(usttack(oo__to_uuaee unstack, obj_at_botton) unstack obj_to_unster from abj_at_botton
9.3 Characterize Robotouille's Tasks' Complexity
In 5 characterize the complexity of the tasks in terms of the demonstrations' length, the code's length, and the expected code's complexity (i.e. how many loopsconddiionaas/tuuniiisss are needed solve this task).
Table For Robotouille's tasks, we group them by cluster report the following: number of states in demonstrations (range and average) 2. number of predicates in demonstrations (range and average) 3. number of lines in the oracle Spec2Code's generated code (range and average) 4. average number of loops 5, average number of conditionals 6. average number of functions
Task
Tnput Demo Langth
Code Length
# of Inops
of conditionals
of functions
# of states
of predicates
Cook cut
7-15 (10.75)
8-19 (13.5)
98-98 (98.0)
2.00
12.0
8.00
Cook patties
(24.3)
(19.0)
(52.0)
Assemble two burgers
15-15 (15.0)
36-36 (36.0)
58-62 (60.0)
1.5
6.00
5,00
Make burger Make two burgers
32-55 (42.6)
26-55 (40.5)
109-160 (146.3)
1.86
17.1
9.86
(52.3)
68-114 (86.85)
112-161 (149)
2.86
9,86
In addition, to bridge the different evaluation standards between planners that generate code and classical planners, we also characterize the Robotouille' tasks based on [41 taxonomy in table
Table 6: For each Robotouille task, we check if contains the specification pattern defined in
Taak
Global Avoiálace
LowerExust Bestictius Arsidance
Wis
Instastancoua Reaction
Delayal Reaction
Cook
Cook top
top
Cur two lemuces
Cook fixt thes out
Cut fint thes cock
Assemble two hurgrs by
Assemble burgens
bergrr
burger
burgr stacking alop paaly inmedistely
Male buger stacking patty sup letuuee immedianely
Mikrn hurger saaking letice siop porty afte properatio
Make burger stacking party lettuce after preperation
Make letroce
Make
hurgers
Make hurges stacking lestuee stop pully inmellittly
Make twn burgers stacking party stop lemice immedunely
Make two buzgers stacking lenuce stop party alter properation
Make two burgers stacinng party stop letuce after preparation
Make twn latuce tamto bungess
10
EPIC-Kitchens Pipeline
10.1
Annotations
We take demonstrations of dishwashing by users 4, 7, 22 and 30, and use of these as in-context examples for the LLM, by writing down each intermediate step's expected output.
22

---

pand)
bact) lao_1 pan_!: cedraci 11
clean par_))
muspyl part Jaant tap_1)
Figure 9: Example of annotations for video ID PO7_ 10 in 6 frames
inhardo mag. "eddryy mug.311
cook E')
shi l'Ijhhand'scnbb Tissort tap. 11
S0aps! mug. 1
Idhhntooo Thar mug_1. 'MBchemop_
tsot/uup11)
muq_1)
Figure 10: Example of annotations for video ID 101 in frames
Predicates are of the form foo(" <obj>_<idtt <i__'iid>', .). where foo is predicate function like adjective (is\_dirty, is\_soapy etc) or preposition (at, ses__in__haama etc). Each argument is combina- tion of object name and unique id, the latter added to distinguish multiple objects of the same kind. Note that these annotations object ids are not available in the EPIC-Kitcher dataset.
Not all predicates enumerated exhaustively, because this can be difficult for human annotator, well as useless and distracting for the LLM. The state predicate annotations in the demonstrations limited to incremental changes to the observable environment. For example, "el_inn__aand' plate_1") comes after in (" ("plate_1", *sink_1")
Examples of these incremental state predicate annotations described in figures 9 and 10 annotating unnecessary human-like actions like picking something then immediately placing back, turning tap momentarily.
We avoid
10.2 Pipeline Overview
In stage recursive summarization, the L.LM first recursively summarizes the provided demonstra- tions, which are represented as state changes since the previous state, until it determines that the
23

---

trajectories are sufficiently summarized. For this domain, the LLM in general terminates after it summarizes the trajectory into series of high-level subtasks, which each consist multiple states and low-level actions. For example, low-level actions "Pick up spoon_ 1n",,,", "Pick up fork_ I1, and "Go from countertop_
sink_1" 1" get combined as the subtask "bring spoon_ and fork_1 from counter- top_1 to the sink_1. 1. Then, Demo concatenates all trajectories together before prompting the LLM to reason about the control flow (e.g. whether for-loop is needed) before generating the task specification.
In stage 2 recursive expansion, because the dishwashing does use that many unique actions, the LLM is asked to directly use all the low-level actions that are provided as APIs to output the task code given a specification.
The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.3
10.2.1 Provided Low-Level APIs
We have provided the following APIs for the perception library and low-level skill library:
Perception Library
get_all_objects(] O): return list of objects in the environment.
Low-level Skill Library
bring_objects_ (ob_tttt loc): bring all the objects loc.
turn_ off :ame): turn off tap.
turn_ore onn_on(e (tap__tmee) turn tap.
sobp() soap the object.
rinse (obj): rinse the object.
pick_up(obj)) pick up the object.
(obj loc): pick up the object at loc.
(obj tool): clean the object with the tool, which could be sponge a towel.
11 Noisy Demonstration Ablation Experiment
As seen in our own annotations for EPIC-Kitcher demonstrations, human annotations or annotations generated by automatic scene summarizers and object detectors may not be noise-free. They may omit some predicates or completely missed predicates in entire timestep. They may contain objects that the users did not interact with during the demonstration, so predicates about these objects are of little importance to the robot task plan. Thus, we conducted two noisy demonstration ablations:
1. Randomly removing predicates/states from the demonstrations (tested in Robotouille)
2. Randomly adding predicates about irrelevant objects to the demonstrations (tested in EPIC- Kitchens).
We found that:
Randomly removing predicates/states
Removing predicates reduces Demo2Code's performance for tasks with short horizons.
Surprisingly, does not significantly worsen the performance for tasks with long horizons.
Randomly adding irrelevant predicates
Additional irrelevant predicates worsen Demo2Code's performance for correctly gener- ating code for 5 users to 2 users.
11.1 Randomly removing predicates/states
11.111
Experimental Details
For each task in Robotouille, modified the demonstrations in two ways:

---

1. for each predicate in the demonstration, there is 10% probability that the predicate would removed from the demonstration.
2. for each state (which could consist of multiple predicates), there is a 10% probability that the entire state would be removed from the demonstration.
We ran the experiment on 4 seeds to report the average and the variance.
11.1.2 Qualitative Result
We analyze a qualitative example (making a burger where the patty needs to be stacked on top of the lettuce immediately after it is cooked) where removing predicates did not affect Demo2Code s performance.
When each predicate has 10% probability of being removed, the demonstration is missing 6 predicates, Half of them omits information such as picking up the lettuce, moved from one location another location, etc. However, the other half does not omit any information. For example, one of the predicate that gets removed is "'robot1" is not holding "top_bun3"".
State 26:
'top_ bun3 is at 'table4'
'top_bun3' is top of 'patty3' >>> >>> robott is not holding 'top_bun3 <<<
Removing this predicate does not lose key information because bun3' is top of "patty3"" still indicates that 'top_bun' has been placed on top of 'patty3'. Consequently, the LLM is still able to summarize for that state:
At state 26, location 'table4'.
robot placed 'top_bun' on top 'patty3' at
Thus, Demo2Code able to generate identical predicates
Using the same seed, when each state has 10% probability of being completely removed, the demonstration is missing states predicates). Because all the predicate in a selected state gets removed, the LLM misses more context. For example, because the following two states are randomly removed, the LLM does not know that the demonstration has moved and placed 'lettuce1> "cutting__or boordingg'.
State 3:
'lettucel" 1
'table2'
'robot1' is holding 'lettucel'
>>>State 4:<<<
>>>'robot1' is nuttiingbbardy '<<<
roborob
at 'table2 <<<
>>>State 5:<<<
>>> >>>'tecuu is at boarting_boardl <<<< >>>'robot1 is not holding 'lettucel '<<<
Consequently, it causes the LLM to incorrectly summarizes the states and misses the subtask of cutting the lettuce.
In [Scenario 1], at state 2, the robot moved from 'tablel to table2 At state 3-4, the subtask "pick up lettuce", This subtask contains: picking up 'lettucei" (state 3)
11.2 Randomly removing predicates/states
11.2.1
Experimental Details
For each EPIC-Kitchens task, we add additional predicates (i.e. showing the position of additional objects in the scene) in at least 2 separate states in the demonstrations. We also do the same
25

---

Table 7: Results for Demo2Code's performance the original EPIC-Kitchens demonstrations v.s. on demonstrations with additional imelevant predicates. The unit test pass rate is evaluated by a human annotator, and BLEU score is calculated between each method's code and the human annotator's reference code.
P7-10(6)
(30-08(16)
Pass.
BLEG.
Pass.
BLEU.
Pass.
BLEU.
Pass
DLCU.
Pies.
DLEU.
Pas..
BLEU.
Pass.
BLDU
Demo222dde
1.00
0.33
0.00
0.19
1,00
0.63
1.00
0.43
1.00
0.66
1.00
0.58
0.00
0.24
Desuu2odd addicional
027
modification for the training examples, while keeping the rest of the prompt identical. We expect the LLM to weed out these additional states during recursive summarization.
For example, for annotations of video ID P22_07 as seen in appendix 17.3. we add distractions in 2 states
State 3:
at ( sink_ 1")
is is_in_hard hand sponge_1")
**++++is_ ('brssh_1 "sink_1 ++++++
bowl_1', "sink_2 ++++++
State 4:
in_on taap_1') ++++++on jar_1", "microwave__ ++++++
11.3 Quantitative Analysis
We see that table that Demo2Code suffers from degradation on most demonstrations when distracting states are added, resulting in only being able to generate correct code for 2 out of 7 demonstrations instead of 5 out of 7 demonstrations.
11.4 Qualitative Analysis
When adding distracting states, the LLM mostly ignores the distracting states and only shows variation in output as result change in input. However, the distracting states can interfere with the final code, as can be seen in the output for demonstration P44 101.
Code using clean demonstrations:
objects get_all_objetts ()
2 for object in objects:
pick up (object)
go_to to ("sink___"11 pick_up (""poonge"__"1""" turn_on ("tta__1)))) soap soap(object) rinse (object) turn_off("tap_1"
10
to ("kitcchhheet
" place (object, "kitchentop_1")
Code generated with demonstration that has additional irreeevnt predicates:
objects = get_all.ojeects()
for object in objects:
bring__bbects_tooo ([object], "sink_1") pick_ (asponge_e"")
turn_on ("tap_1")
place ("brush__" "sink_1")
place ("bovl_1", "sink_2")
soap(object) rinse (inse(object)
10
turn_off("tap.it
go_to ("Aaiicccchht)))p_1"))
26

---

place (object, "kitchentop_11")
When compared to the generated output with clean annotations, we see that while the onj"jar_ "micronave_1" was ignored, in( brush_1', "sink_1') and in ['bowl_1', "sink_2" 22) result in the LLM generating additional code that does not align with the demonstrations. Specifically, even though brush_1 and bowl_1 were objects that the users were not interested in interacting with, the LLM has generated place() code (lines 6-7) for these two objects. This type of mistake could be avoided by adding reasoning during recursive summarization. The LLM can be guided to ignore irrelevant objects and avoid hallucinating actions relating to these objects for example, ground place action only when both is_in_ ii_dinnand.... and on...., loc) are seen one after the other.
12 Chain-of-thought Ablation Experiment
This experiment studies the effect of the chain-of-thought length (in stage recursive summarization) on the LLM's performance. We found:
It is helpful to guide the LLM to take small recursive steps when summarizing demonstrations (especially for tasks with long demonstrations).
The LLM performs the worst if it is asked to directly generate code from demonstrations.
12.1 Experiment Detail
We defined 3 ablation models listed below from the shortest chain-of-thoughe length the longest chain length. In addition. because the tabletop's 's Demo2Code pipeline is different from Robotouille's pipeline, we also deseribe how these pipelines are adapted to each ablation model:
No-Cot:
Tabletop and Robotouille has exactly the same process of prompting the LLM ONCE to generate code given the language model and the demonstrations,
1-Step
Tabletop:
First, the LLM receives all the demonstrations concatenated together as input to generate the specification without any intermediate reasoning. Next, the LLM generates the code given the specification.
bobotouille: First, the LLM receives all the demonstrations concatenated together input to generate the specification. It can have intermediate reasoning because the tasks are much more complex. Next, the LLM generates the high-level code given the specification and recursively expands the code by defining all helper functions.
2-Steps
Tabletop: First, the LLM classifies the task into either placing task or stacking task. Second, the LLM receives all the demonstrations concatenated together input to generate the specification without any intermediate reasoning. Finally, the LLM generates the code given the specification.
Robotouille: lle: First, for each demonstration, the LLM gets its state trajectories as input to identify a list of the low-level action that happened at each state. Second, all the low-level actions from each scenario are concatenated together and used by the LLM to generate the specification. The LLM can have intermediate reasoning at this step because the tasks are much more complex. Finally, the LLM generates the high-level code given the specification and recursively expands the code by defining all helper functions.
We identified clusters tasks based on the number of states they have, and for each cluster, we selected two tasks to test. For each task and for each of that task's specific requirements, we tested the approach 10 times and took an average of the unit test pass rate.
Short-horizon tasks (around 2 states): "Place A next to B" and "Place A corner"
Medium-horizon tasks (around 5-10 states): "Place A on top of B" and "Stack all blocks/- cylinders (where there might be maximum stack height)"
Long-horizon horizon tasks (more than 15 states): "Make burger" and "Make two burgers"
27

---

Short
Medium
Long
0.90
0.86
0.87
0.80
0.8
0.75
Unit Test Pass Rate
0.68
0.60
06"
0.43
04-
0.19
02
0.06
0.03
0.00
No-COT 1-Step 2-Steps
Full
Figure 11:
Ablation of different degrees of shain-of-thougee (recursive summarization) across tasks with short, medium, long horizon.
12.2 Quantitative Result
We include the quantitative result again here as Fig. [] Overall, Demo2Code/Full performs the best, and there's 's the general trend that the length of the chain of length increases, the LLM also generates code that has a higher unit test pass rate. For short-horizon tasks, the difference in the thain-of-thoughtt length has small effect the pass rate because these tasks have short demonstrations that are easy to process without intermediate chain-of-thor tf-ho-ugttthh steps. In contrast, both rredium-horizon tasks and long-hone long-horizon tasks great improvement when the LLM is prompted to take smaller recursive summarization steps. In the next section, we will examine one example from each cluster of tasks to analyze how the length of the chain-of-thought prompts affects the LLM's output.
12.3 Qualitative example for short-horizon task
We examine all four models' outputs the task: placing A next to B. There are 2 states per demonstration and demonstration in total. The query and the Full approach's 's correctly generated code for particular instance of the task are shown below:
objects=[ ['yelloy block', blue block', 'green block', 'cyan cylinder', .** gray cylinder', 'red cylinder
Place the gray cylinder next to the green block.
Initial State (State 1):
'green block' is not behind 'gray cylinder' 'gray cylinder' has not moved ggray cylinder' is not in front of 'green block'
10 [Scenario 1]
State 2:
12 green block' is behind ggra cylinder'
13 gray cylinder' has moved grra cylinder' is in front of 'green block'
say ("Place the gray cylinder front the green block.") 2 location_pos parse_position ('in front of the green block')
cettfirrst_o_secomm (' gray cylinder', location.pos)
For this particular example, the L.M needs to be able to identify that the robot needs to specifically place the gray cylinder in front of the green block. Only No-Cot failed to generate the correct code,
28

---

while 1-step and 2-steps were able to identify the specification and generate the code exactly the same as Full.
Why No-Cot failed?
No-Cot seemed overfit the prompt. For this example where the specific requirement place A to the right of B, it outputs "to the right of the green block." We hypothesize that because the specific requirement of placing to the left of B appears in the prompt. the LLM just assumes that any state that differs from the example in the prompt is placing A to the right of B. Below is the code outputted by No-Cot:
say ("Place the gray cylinder to the right of the green block.") location_pos - parse_position ('right of the green block') put_firsst_on_econ ('gray cylinder', location_pos)
12.4 Qualitative example for medium-horizoon task
We examine four models' output on the task: placing A on top of B. There are 5 states per demonstration and I demonstration in total. The query and the Full approach's correctly generated code for particular instance of the task are shown below:
ob objects- [ orange block', 'bronn block', 'red cylinder', 'purple cylinder', 'pink cylinder', 'yellow cylinder
Place the brown block on the purple cylinder.
Initial State (State 1):
'orange block' is not top of 'table'
'orange block' has not moved
oraage block' is on top of 'brown block'
orange block' is not to the right of 'yellow cylinder' 'bron block' is
top of 'table'
not moved 12 'brown block' is not on top
11 'brown block'
'purple cylinder'
13 'red cylinder' is not on top of 'table'
14 'red cylinder' has not moved
'red cylinder' is top of 'orange block'
16 cylinder'
on top of 'brown block'
17 'red cylinder' is on top of 'pink cylinder'
18 'pink cylinder' is not on top of 'table' 'pink cylinder' has not moved
21 'pink cylinder' on top
20 'pink cylinder'
'orange block'
on top 'brown block'
22 'yellow cylinder' is not to the left of 'orange block'
[Scenario 1]
25 State 2:
26 'red cylinder' is on top of 'table'
27 'red cylinder' has moved
'red cylinder! is not top of 'orange block'
'red cylinder' is not top of 'brown block'
30 'red cylinder' not on top of 'pink cylinder'
31
32 State 3:
'pink cylinder' is on top of 'table'
34 'pink cylinder' has moved
35 'pink cylinder' is not on top of 'orange block'
36 'pink cylinder' is not on top of 'brown block'
State 4:
39 'orange block' on top of 'table'
10 'orange block' has moved
4 'orange block' is not on top of 'brovn block'
oraane block' is to the right of 'yellow cylinder'
29

---

'yellow cylinder' is to the left of 'orange block'
45 State 5:
16 'brown block' is not on top of 'table'
" 'brown block' has moved
'brown block' is top of 'purple cylinder' anm
say ("First, place the red cylinder, the pink cylinder, and the orange block on the table. Then, place the brown block on the purple cylinder.") ") wtw.e.tooopace..ftttiin...o orange block']
= ['red cylinder', 'pink cylinder',
for item in items_ tow.llcce.firsttttcoeeoeeeesesse inderr wet_firsst_o_secooee (item, "table")
put_firrst_on_eeee b'oww block', 'purple cylinder')
For this particular example, the LLM needs to recognize that there are 3 additional objects top of the brown block. order to place the brown block on top of the purple cylinder, the robot needs to first place those additional objects the table. No-Cot and 1-step approaches failed to generate the correct code, while 2-steps was able to identify the specification and generate the code exactly the same as Full.
Why No-Cot and 1-step failed?
Both No-Cot and 1-step seem to be over-reliant on the language instruction instead of reasoning through the state transitions. Specifically, examining the specification outputted by I-step suggests that the LLM simply copied the language instruction instead of recognizing that more objects are moved than just moving the brown block to be on top the purple cylinder.
The detailed requirement is: <Place the brown block on the purple cylinder.
Below show the identical code generated by No-Cot and 1-step.
say ("Place the brown block on the purple cylinder. ") fat__firat_on_eeeae ( broon block', 'purple cylinder'
12.5 Qualitative example for long-horizon task
We examine all four models' output on the long-horizon task: making a burger with patty, lettuce, and tomato. This task has 71 states per demonstration and demonstrations in total.
The Full pipeline's s approach outputs the correct code as shown below. Next, we analyze how other models failed to produce the correct code.
list of all hat__ammstttttaaaeeeee whattattthhhhhttteeeaaeta eett__atcc__tteeeeeteee ('patty') 2 patties get_all_obj wwwe_names_.e
patties the kitchen.
Get a list of all the lettuces in the kitchen. lettuces =
Get list of all the tomatoes the kitchen. tomatoes get aha.lloooooaaaeeeeaaaaaaattt.tte ("tommatto 7 Get a list of all the bottom buns in the kitchen. botton_buns
('botton bun !)
Get list of all the top buns in the kitchen.
10 top_buns get_ ceo.aall..0j.naee..tat.tm.
type bun')
11 Get a list of all the stoves in the kitchen.
12 stoves =
htt.aaiioonamssstattttooooo baatch_type('stora ('stove')
13 Get list of all the cutting boards in the kitchen. cutting. 1)
let_alll whw. cccttnngg
1s Get a list of all the tables in the kitchen. 16 tables =
taatssstttttmtttthhheee type ( ('table')
30

---

18 # Decide stove to use.
stove_to_use stoves
20 Decide a cutting board to use.
21 betiing_bard_to..
outting.boards [0]
22 Decide a table to use.
table_to_use - tables
25 Repeat the following twice:
26 for i in range (2):
27
# Decide a patty to cook. patty_to_cook
patties[i]
29
Cook that patty at that stove. cook_
30
att_locationnobj-patteee # Decide a bottom bun to use. bottom_ bun_to_use
locatia=sstoe_.to_ue
31
bottom_buns(i]
Stack the patty top of the bottom bun.
34
stack_obji_on_oo0 (objl=patty_
obj2=bottoo__uu_to_uue
35
36
# Decide a tomato to cut.
comato_to_cet
tomatoes [1]
38
Cut that tomato that cutting board.
39
cut board_to__se)
tobtstttttttttoooooouu location=
Stack the tomato top of the patty
ol tom
obj2=patty.to_cook
42
43
Decide a lettuce to cut.
44
lettuce, lettuce_to_cut cut = lettuces
Cut that lettuce at that cutting board. cut_ cut_
46
(tw.::attiiioiiettttttee...ee (obj=lettuue__tocuu, location
47
Stack the lettuce on top of the tomato. stack howp:////ooiillettuttt.oo.oo
obj2=tomato_to__utt
50
Stack the top bun on top of the lettuce, tomato, patty stack
bun_ton 51 top to use = top_buns [i]
stack stack_obji_oo (obji=tooop bun_ use, obj2=lettuce.to_cut))
Why No-Cot failed?
The main issue that No-Cot suffers from is the maximum token length issue because the prompt needs to include demonstrations as the example query and solution code the example response. Thus, the LLM was not able to finish completely the code as seen below (line 37). addition, although the LLM showed some ability but it did not use for-loop to assemble the burger. Specifically, the LLM generated code that index specific items: (e.g. line 23 bottom, buns [0], line 25 lettuces [0], etc.)
the control flow, it only creating for loop to cook the patties
Get a list of all the patties in the kitchen. _ij__nmme___ in the kitchen.
patties = get_all_ obj.ojjjj.
Get list of all the that_match type ('patty!)
stoves get_all.locaioonnaanss taaessstttttmtmtttpppppee stove
Decide a stove to go to.
stove_to_cook.at stoves
Repeat the following tvice for in range(2):
10
# Decide a patty to cook.
patty_to_cook [atties[i]
12
Cook that patty that stove
13
at_.lectttttoooaiiooooooo tob-attttooo-tttttt_ooooo.o location- stove_to_cook.ct
cook_object_at__]
14
Get list of all the bottom buns in the kitchen.
31

---

16 bottom_buns
('bottom bun ')
Get list of all the lettuces the kitchen.
18 lettuces oet. tereelttyeeeeeeteee ('lettuce')
19 Get a list of all the tomatoes in the kitchen.
20 tomatoes
htw.eelsooonaaesttaaateee type ('tomatoo)
Decide bottom bun
23 bottom_ bun_to_use bottom_bune buns [0]
24 Decide a lettuce to use.
25 bettuce_to_use
lettuces
Decide tomato to
27 tomato_to_use tomatoes [0]
28
29 Stack the patty on top of the botton bun. obj1 should be the patty, obj2 should be the bottom bun.
stack_obji_oonoooppe (obj1=pattiees 0 obj2=bottom.bun.touuu
32
13 Stack the lettuce on top of the patty.
34 obj1 should be the lettuce, obj2 should be the patty on top of the
bottom bun.
35 octecp_..011...oooooopooooetree.
obj2-patties [0])
36
37
Why 1-step failed?
In Fig. II result, 1-step actually performed worse than No-Cot. The worsened performance is caused by low-quality specifications produced by the LLM. Because stage 2 recursive expansion is highly reliant on the specification that gets passed in, any mistaken or missing specification will be reflected in the code as well. Below shows the intermediate reasonings and specification outputted by the LLM. Notice that it does capture any information about the lettuce and the tomato needing to be cut before being added to the burger (see lines 22-23 and lines 25-26),
The order of high level actions is: [' cook 'assemble', 'cook assemble')
Because there are two 'cook' actions and two 'assemble' actions, assume that must cook and assemble each burger separately.
Thus:
<Make two burgers.
Specifically:
Get list of all the patties in kitchen. Get a list of all the stoves in the kitchen. Get a list of all the buns in the kitchen.
10 Get list of all the lettuces in the kitchen. Get list of all the tomatoes in the kitchen.
12
13 Repeat the following twice:
14
Decide patty to use. Decide stove to go to. Cook that patty that stove.
16
17
18
Decide a bottom bun to use. Place that bottom bun on plate. Place that patty top of the bottom bun.
22
Decide a lettuce to use. Place that lettuce on top of the patty.
23
Decide tomato Place that tomato on top of lettuce.
26
27
28
Decide top bun to use.
Place that top bun top of the tomato.
32

---

Consequently, although the code (shown below) closely matched the specification and included the for-loop, because the specification was wrong, the code also failed to solve the problem.
Get list of all the patties the kitchen. patties =
(erpe('pattee patttyy)
Get list of all the stoves in the kitchen.
stoves
htt.:oocttto..masa......eeeeeeee Get list of all the buns in kitchen. bun')
('stove!)
buns
bun')
Get list of all the lettuces in the kitchen. lettuces tha.all.tj..amme.thatt.
('Ietttuc))
Get list of all the tomatoes the kitchen.
10 tomatoes get_all_obl m
thatea tthaatmatch_type tomato
12 Repeat the following twice:
13 for range(2):
14
Decide a patty to use.
15
patty_to_use _ue patties[i] # Decide a stove to go to
stove stove_to_ cook__t at stoves Cook that patty at that stove.
18
19
aocattittttotottiiinoooooooooo cotppoootttttptttttttoo to__se, location stove_to_cook.c
Decide bottom bun use.
22
tou_to_uuue buns[i+2] Place that bottom bun on a plate.
23
obac__ob__o__bbjoo (obii-boottt booji=-totom_buu_toooooooo
obj2- abj2='plater plate')
Place that patty top of the bottom bun.
26
place_obj_on_ce (onjptttttttpttttttoouuu
obj2-bottom, www.bbttoombuuntt..ooo to__to_
27
28
# Decide a lettuce to use...
lettuce_to_us = lettuces
Place that lettuce top of the patty.
31
of tob
obj2=patty.to_us)
32
33
# Decide a tomato to use.
tomato_to__
tomatoes [1]
35
Place that tomato on top of the lettuce.
36
obb.ii/..toobiooiiiittttttttte use,, Djj==lettuce_to_uaa
37
38
# Decide top bun to use. bun_to_se buns [i+21]
40
Place that top bun on top the tomato.
41
(objl=top ton_to__sse obf2=tomato_
Why 2-steps failed?
2-steps improved its specification compared to 1-step's. Specifically, its specification able to identify that lettuce and tomato require cutting before they are stacked as a part the burger. However, it still fails to identify the physical constraint that in order to cut object, it also needs to specify the location. While is able to identify stove used for cooking (line 17), fails to identify the cutting board (line 22 and line 25).
The order of high level actions is: [ cooo''' 'assemble', 'cook', assemble ']
Because 'cook' is before 'assemble', we assume that we must cook the patties before assembling the burgers.
* Because 'assemble' is repeated twice, we assume that must assemble two burgers.
Thus:
<Make two burgers.
Specifically:
33

---

Get list of all the patties in the kitchen. Get list of all the buns in the kitchen. t0 Get list of all the lettuces in the kitchen. II Get a list of all the tomatoes in the kitchen.
12
Repeat the following twice:
Decide patty to cook.
15
Get a list all the stoves the kitchen.
16
Decide stove to use.
17
Cook that patty on that stove. Decide bun
the bottom bun. Place that bun on plate.
19
20
Stack that patty on top of that bun. Decide tomato to use.
Cut that tomato. Place that tomato top of that patty. Decide lettuce use.
24
25
Cut that lettuce.
26
Place that lettuce on top of that tomato. Decide bun to on top that lettuce.
the top bun.
28
Place that
Consequently, in the code that the LLM produced based on the specification, it created a function cut_oby cut_object (obj). Then, because the function signature does not specify any location to cut the object at, the downstream function definition also fails (see the definition starting from line 43).
Get list of all the patties in the kitchen. patties = get_all obj _am_mmmm_____ that_match__m type ( ('patty') !)
Get list of all the buns in the kitchen.
buns get_all_obj_nanee caamessthat_mmaee type ('top bun')
('botton bun')
tattses__tattmmtchhtttte Get list of all the lettuces in the kitchen.
lettuces get_all_obj_net thaame_that_.__ thattmmtch__tee Get list of all the tomatoes the kitchen. tomatoes
('lettuce')
the
thats_tttt_mmtttte type toomato')
10 Repeat the following tvice: for in range (2):
12
Decide a patty cook.
13
patty_to_cook
patties
14
Get a list of all the stoves in the kitchen. stoves Decide stove use.
low.alloooatinnneeaaa....a that match_typee ('stove')
16
17
stove_to_us
[i]
18
Cook that patty on that stove. cook Decide a bun to nottom_bun_to__. bun_to_use buns[i]
(obj=patty_to... tieppatt_tto_cok cook, location=store_to_us)))
the bottom bun.
21
22
Place that bun on a plate.
23
place_obji obj__o_oojjj____oooooooo__p___oe babji=bottom_bbn_too use, (plate) Stack that patty top of that bun. stack_ objj__on_obj(obillppppp Decide a tomato use.
25
obj2=bottom_bun ttrtttomuuuu_uuuuuutu
26
27
tomatoes[i]
# Cut that tomato.
30
Place that tomato on top that patty place place_obj_on. obj_oo_objj___eeeooo tomo # Decide a lettuce to use.
31
obj2-patyy.to_cook
32
lettuce_to_use = lettuces Cut that lettuce.
35
36
Place that lettuce on top of that tomato. place_obj_on_ee cobji=lettuce_to_uu Decide bun to the top bun.
37
bbj2=tomato_to_uu
34

---

top_bun_to_use = Duns[i+]]
Place that bun top of that lettuce.
41
place_obj.on__bj (objjooopppbbuttooooo. to__se, obj2=lettuue_to_use))
42
43
def cut_object(ob]]
1f not is hs_holding (obj)
# cut obj, the robot needs hold obj first. if ss_in_
46
47
# Because obj is in a stack, robot need to move then unstack the obj from the obj_at_bottom first obj_at_bottor ae.oo...aa.....aataaaeeh..o..... move_then_unse (tw.bbee__uuaaccccoo..ooo.ooaeee obj at _t_at_bottion bottom, oot. else:
49
90
# Since obj is not in stack, robot just move then pick it up
52
move_ pove_ccee.
53
cut the object
54
(objj-obb))
13
Intermediate Reasoning Ablation Experiment
This experiment studies how the thoroughness of intermediate reasoning (in stage 3 of recursive summarization) affects the LLM's performance. We found:
Our approach may be improved by ablating away some part of reasoning (e.g. listing high-level actions)
Having intermediate reasoning of any form is helpful for hard cooking tasks that have many sub-tasks
Future work is necessary to engineer better prompts
13.1 Experiment detail
This experiment compares Demo2Code (labeled as Full) with three additional ablation models each with differing levels of reasoning
No reasoning: The LLM generates the specification directly from step of recursive summarization with no intermediate reasoning.
Only List: The L.M generates the specification after intermediate reasoning which lists the high-level actions in common with the scenarios from step 2 of recursive summarization.
Only Analyze: The LLM generates the specification after intermediate reasoning which describes the repetition and ordering of high-level actions from step 2 of recursive summa- rization.
These models are tested on all the Robotouille tasks. We use clusters of tasks based on the number of high-level actions/sub-tatt they have.
Easy cooking tasks (< 2 high-level actions/sub-taskk)) "cook and cut", "cook two patties", and "cut two lettuces"
Normal cooking tasks (between 2-7 high-level actions/sub-ttasks: "make a burger" and "assemble two burgers with already cooked patties"
Hard cooking tasks (2 8 high-level actions/sub-taske "make two burgers"
13.2
Quantitative result
For each task and for each of that task's specific requirements, we tested each model on 10 randomly generated environments and took an average of the unit test pass rate. Fig. 12 shows that the Only Analyze model outperforms the other models. All methods easily achieve 100% pass rate Easy
35

---

Easy
Normal
Hard
1.00
1,00
1.00
1.00
1.0
Unit Test Pass Rate
0.8
0.6
0.54
0.51
0.56
0.43
0.40..3
0.4
0.23
0.2
0.14
0.0
No
Only List
Only Analyze
Full
Reasoning
Figure 12: Ablation of different amount and style of intermediate reasoning step 3 of recursive summarization across Robotouill ooille tasks. These tasks are clustered into easy cooking tasks, normal cooking tasks, and hard cooking tasks based the number of high-level actions/sub-tant each task has.
tasks. There is a trend that including any kind of reasoning improves performance on hard tasks. Only List only slightly suffers Normal performance while increasing significantly for Hard performance relative to No Reasoning. Full's Normal performance suffers significantly yet its Hard performance increases as well, suggesting a tradeoff between Normal performance and Hard performance; however, Only Analyze has the highest Normal and Hard performance of all models. Since Full combines both Only List and Only Analyze, this shows that concatenating reasoning methods doesn't necessarily mean better performance.
13.3 Qualitative example
In this section, we show a qualitative example all models' output on the long-horizon task "make two burgers" with the specific requirement "stack lettuce atop patty immediately." " We choose this example to show the effects of reasoning on the code as well as to shed light on why Full underperforms relative to Only Analyze. Only Analyze is the only model with 1.0 success rate while the other models all have 0.0 success rates. For each model's output, the helper functions can be found in section 19.3
13.3.1
Why No Reasoning failed?
No Reasoning is the only unsuccessful model that generates code that executes. Due to the lack of any reasoning, performs the task correctly but with the incorrect requirement; rather than preparing one burger at a time and assembling the burger immediately as ingredients are prepared, it prepares all ingredients before assembling both burgers in parallel.
Get list of 2 patties get_all_obj www.nass_tht_ntttee that_aatchhhhh_teeeeee maatt_aatch_typee ('patty') Get a list of all the lettuces in the kitchen. lettuces get_all_obb aal.aalloojaammsss.
the patties
the kitchen.
Get list of all the bottom buns in the kitchen. botton_buns
('botton bun')
7 Get a list of all the top buns in the kitchen.
top_buns get oot.all.oo.nnmeetttt.aa...... Get list of all the stoves in the kitchen. 10 stoves get_all..octionnnaaee mathh ('stove')
bun')
=1 Get a list of all the cutting boards in the kitchen.
12 butting_boards = get_all wtw.aall.ocattin_aaaaaa marttiioonnamms..mm www.saatatttaaaeeeeeeea thwattaatchhttteeeeeeeeeeeettttt type d'cuutingg ))
13
14 Decide two patties to cook.
15 patties_to_comt patties[:2]
16 Cook those patties at two different stoves. for in range
36

---

location=stoves
19
20 Decide two lettuces to cut.
lettuces [:2]
Cut those lettuces two different cutting boards.
for in range (2) (2):
24 (ot.
[i], location=
cutting.boards [i])
25
Decide bottom buns to
27 trroo__uus_tt__ue
bottom_bung [:2]
28 Stack a patty on top of each bottom bun.
29 for i in range (2)
patty_to_
terries_to_cookll
bottom_ tou
32
stack_obji_on.on prtp::ooo_.oooooiiirrrrraatttttp (oty
obj2=botton_bun.t.uuu
33
34 Decide two lettuces to use. lettuces_to_ust
lettuces_to. paaces_to_cutt [:2]
36 Stack a lettuce on top of each patty.
37 for in r2)ge()):
38 lettuce_ toccce_to_uss use lettuces_ to uoe
patty_to_use use betties__t_ccookie otackk.0ii.oom000ff00o1-------ee
obj2-patty_to_use
41
42 Decide two top buns to use.
43 top_buusss Duns_to_use top_bune [:2]
Stack a top bun top of each lettuce.
45 for range (2):
46 top_bun_to_us
top_buns_to_useefi
47 lettuce_to_ use = lettuces_to__selle
obj2=lettuce_to_use)
13.3.2
Why Only List failed?
Only List improves on Reasoning by generating code the correct order the specification. Patties are cooked and then stacked on the bottom bun before lettuces are cut and stacked on the patties before finally the top buns. While the order is now correct, just specifying the order wasn't effective enough because the reasoning missed making burgers by rather than all in parallel. In addition, this code doesn't It execute due to the attempt cutting lettuces back to back at the same cutting board. This violates the environment's constraints since only one object can be one station at a time.
Get list of all the patties in the kitchen. patties = Get list of all the lettuces lettuces get_all_obj_nanee wwtaaaassaaattteeeeeeeeeee that that_matchh_type that. ('lettuce') Get a list of all the bottom buns in the kitchen. bottom_bre buns get_all_objjnanee Get list of all the top buns the kitchen. top_buns Get a list of all the stoves in the kitchen. stores artalllioatioo_aaeetttt...mm Get list of all the cutting boards in the kitchen. 12 cutting.
('pattyyptty)y) 'patty')
the kitchen.
bun')
('oop bun')
('stove))
www.messtttt_mttcchh.o.m type cuttinng.
13
14 Decide two patties to cook.
15 patties_to__tccc
patties [:2]
16 Cook those patties two stoves.
17 for in range(2). (2):
18
[i], location=stoves [i
})
37

---

Decide bottom buns to
21 otcc::o0hh...o...11.oooroaaaaaaa
bottom_bure cooom_uussti [:2]
22 Stack a patty on top of each bottom bun.
23 for 1 in range (2):
obj2-
botton_buns_ use [i])
25
26 Decide two lettuces to use.
27 bettuces_to_ure Cut those lettuces
lettuces [:2]
cutting board.
29 for lettuce in lettuces_to_use:
30 (ut_object(ob--ettteee
locatin=cuting [0])
31 Stack a lettuce on top of each patty.
for in range (2)
obj2=patties_to_com cook
34
35 Decide two top buns to use.
top_ bop_buus_to.uu
bun_buns [:2]
37 Stack a top bun on top of each lettuce.
88 for in range(2):
19 stack_ otact_ buns, use [i], obj2=lettuces.to_uas
])
13.3.3 Why Full failed?
Full gets much closer to correctly solving the task than Only List. Unlike Only List, Full has the correct ordering for the requirement; it stacks the patty and lettuce immediately after preparation and does so for an entire burger before moving on. The issue is that while with the analysis ordering and repetition it manages to put all the code in one for loop rather than multiple, it retained some of the poor aspects of Only List by attempting to cook two patties at once. Doing this within the for loop is incorrect logic. In addition, the cooking is attempted on the same stove twice which violates the environment's constraints of one item being on a station at a time.
Get list of all the bottom buns in the kitchen. botton_buns get_all get_all_obj_nanes that_hattteeee thht_aatchhaeeeeeeeeeeeeeeteeeee _at_ch_ttppeeeeeee (' botton bun')
Get list of all the lettuces the kitchen. lettuces = get_all Get a list of all the patties in the kitchen. patties =
("lettucc)
('patty')
Get list of all the top buns the kitchen. ('top bun')
top_buns =
Get a list of all the stoves in the kitchen. taatt_mach_ttyeessttt
stoves = lrt_aalllocattionnaaaeee
Get list of all the cutting boards in the kitchen.
12 cutting_boarde
get_all,.ocation_nane
wwtpssttttttthamaaaeeeeeea type (a cuttinngbbbart
13
14 Repeat the following twice
15 for range (2):
16
Decide a stove use.
17
stove_to_use stoves
# Cook tuo patties at that stove.
for in (2)ge(2): cook. (objjattioooooopptttt [j], loatton==ttve.to_uuee
20
21
22
Decide a bottom bun to use.
bottom_bun_to_use botton_buns [i]
Stack the patty top of the bottom bun.
25
obj1 should be the patty, obj2 should the bottom bun.
26
wew-botom__buuttoet
27
Decide lettuce to use.
38

---

lettuce_to_use = lettuces[i]
Cut that lettuce at cutting board.
31
oobaaaiiccoppleetttee......ss locaion=autingg
[i])
32
# Stack the lettuce on top of the patty
obj1 should be the lettuce, obj2 should be the patty top of the bottom bun.
34
use, obj2=patties
35
36
Decide a top bun to use. top_bun_to_use
son_buns [i]
38
Stack the top bun on top the lettuce. obj1 should be the top bun, obj2 should be the lettuce on top of the patty on top of the bottom bun. oob tou obj2=lettuce.to_uue))
19
13.3.4 Only Analyze Output
Only Analyze is the only model to adhere to the environment's constraints. Note that while the same stove is used for cooking twice, the second time a patty is cooked the first patty has already been removed from the stove and stacked into the first burger.
Get list of all the patties in the kitchen. patties cet_all_obj_mantt waamessthat_attch...yoo ('pattyy Get a list of all the lettuces the kitchen. lettuces =
('Ieettuce
Get list of all the bottom buns in the kitchen. bottom_buns
h.g.allooo_.aree__tttttttttttt Get a list of all the top buns the kitchen. bun_buns =
('bottom bun')
hit.oobj_arees.hattmeetttt Get list of all the stoves in the kitchen. stoves get_all_.ocation_nae waw.ioonnnamms_tttmttttttet type ('stove')
bun')
n1 Get list of all the cutting boards the kitchen.
12 cutting.boarde get_all. www.ooattioonnmssss.see. names_ taaaate_tttatte. tatttttttch (tp.ttycteeeeeeeeeeeeeee butting-board
Decide stove 15 stove_to_use stoves [0]
16
17 # Repeat the following tvice for in range (2)
19
Decide a patty cook. patty_to_cook patties[i]
20
21
Cook that patty at that stove.
cobt (oojpattiiiiljjpppttttoooooooo locationsstove.oouse))
24
Decide a bottom bottom_buns [i]
to use.
25
bottom_ pactommbun_tor to__to_use # Stack the patty on top of the botton bun.
26
obj1 should be the patty, obj2 should be the bottom bun. stack_obj1_ (on
obj2=bottom, bun_to_us_
29
30
Decide a lettuce to use. lettuce_to_use lettuces [i]
Cut that lettuce at that cutting board.
33
aett_obeett_t__ocaaiie (aocatiooooo=leettee (obj=lettuce_to_use, locatonn [i])
34
Stack the lettuce on top of the patty. obj1 should be the lettuce, obj2 should be the patty top of the bottom bun.
36
stack_objl_on. oaceLoooo..oooooofoooooeeeeeeeee
obj2-patty_to_cookt
37
38
# Decide a top bun to use. top_bun_to_use tuns [i]
39

---

4 # Stack the top bun on top of the lettuce.
obj1 should be the top bun, obj2 should be the lettuce top of the patty on top the bottom bun.
2
ob3--lettuc.to.use))
14 Recursive Expansion Ablation Experiment
This experiment studies how the number of recursive code expansion steps (in stage recursive expansion) affects the LLM's performance. We found:
It is helpful to guide the LLM to slowly expand the code instead of asking it to directly generate all the code at once using only the low-level imported APIs.
The initial code that LLM uses to expand the rest of the functions should align closer to the given specifications.
14.1 Experiment detail
This experiment compares Demo2Code (labeled as Full) with three additional ablation models each with different amount of recursive code expansion steps:
lllayer: Given specification, the LLM directly outputs the code for the task using only low-level action and perception APIs from the import statement. The LLM not allowed to define any helper function.
2-layer (Comp): Given a specification, the LLM first outputs corresponding code that can call undefined "composite functions." Each composite functions contain at most two low-level actions, e.g. Sove_then_pii move_then_ ahen_stack. In the next step, the LLM defines these composite functions using only low-level action and perception APIs from the import statement.
2-layer (High): Given specification, the LLM first outputs corresponding high-level code that can call undefined "high-level functions." " This step same as Demo2Code, Then, the LLM defines these high-level functions using only low-level action and perception APIs from the import statement.
These models are tested all the Robotouille tasks. Because section D3 on Intermediate Reasoning Ablation Experiments also tested only on Robotouille, we use the same 3 clusters of tasks based on the number of high-level actions/sub-taske they have.
Easy cooking tasks (< high-level actions/sub-tasks): "cook and cut", "cook two patties", and "cut two lettuces"
Normal cooking tasks (between 2-7 high-level actions/sub-tasks)) "make burger" and "assemble two burgers with already cooked patties"
Hard cooking tasks (> 8 high-level actions/sub-tain tasks): "make two burgers"
14.2 Quantitative result
For each task and for each of that task's specific requirements, we tested each model on 10 randomly generated environments and took an average of the unit test pass rate. Fig. [3]shows how our approach Demo2Code (Full) outperforms the other models. Overall, the number of recursive code generation steps increases, the LLM's performance also increases. Interestingly, 2-Layers (Comp) performs worse than 2-Layer's (High) despite having the same amount of recursive steps. The two model's performance difference suggests that the granularity of the initial code also affects the performance. While 2-Layers (High) first asks the LLM to generate high-level code, which as Demo2Code, Layes (Comp) first asks the LLM to generate code that calls composite functions. Consequently, for 2-Layers (Comp), the LLM needs to first output longer, more complicated code the LLM has to produce in one-step, which is more prone to error.
same approach
40

---

Easy
Normal
Hard
1.00
1.0
Unit Test Pass Rate
0.8
0.6
0.40
0.4
0.30
0.23
0.2
0
0
0.0
1-Layer
2-Layers (Comp)
2-Layers (High)
Full
Figure 13: Ablation different amount of recursive code expansion steps across Robotouil lle tasks. These tasks are clustered into easy cooking task, normal cooking task, and hard cooking task based on the number of high-level actions/sub-tasks each task has.
14.3 Qualitative example
In this section, show a qualitative example of all models' output on the short-horizon task "cut two lettuces." Note that cutting lettuce has not been shown in the prompt, so the LLM needs to be able to generalize to this new task and properly use for-loop write efficient code. 1-layer and 2-layer (Comp) had 0.0 success rate. 2-layer (High has 0.2 success rate, and Demo2Code has 1.0 success rate. All models are able generate the for-loop, evident in the outputted code below. However, all ablation models struggle to generate code that adheres to physical constraints and applies to any random environment.
14.3.1 Why L-Layer failed?
I-Layer generates the correct code cut the lettuce. Despite not seeing how the low-level action cut() () should be used in the prompt, it correctly writes the code that will cut the lettuce until the lettuce becomes cut. However, the LLM misses the physical constraints associated with cooking, e.g. in order to cut the lettuce at a cutting board, the robot needs to make sure that there is lettuce at the cutting board before starting cut. If there is lettuce the cutting board, the robot first needs to pick up lettuce and bring it to the cutting board. Another issue that 1-Layer encounters is reaching the LLM maximum token length. For longer tasks, cannot finish generating all the code, so its overall execution success rate is only 0.04.
Get list of all the lettuces the kitchen. lettuces
('lettuce')
Get a list of all the cutting boards in the kitchen. cutting_boards - get_all.locatinn names_ wttpsettttttttthheeeeeeeee type (1 cutting.
Repeat the following twice
7 for i in range (2):
# Decide a cutting board to use. cutting-boards
butting_boad_to_use Decide a lettuce cut. pictuce_to_one # Cut that lettuce at that cutting board.
10
11
lettuces[i]
12
# To cut lettuce_to_cut the robot first needs to be at the same location as cutting.board_toouuss get_curr_ move (eweeettcurr_locatiooo while not is_ (iettuce_tttt cut (lettuce_to_cut)
14
!= nutting_board_to.n
15
16
41

---

14.3.2 Why 2-Layers (Comp) failed?
2-Layer (Comp) faces the same issue of not adhering to all the physical constraints. For example, it successfully identifies that needs to pick up the lettuce before going to the cutting board and cutting it. However, it does not place the lettuce the cutting board before it starts cutting (see line 27).
In addition, all the composite functions defined in the recursive step are technically correct though they are not efficient or compact. For example, move _eoe_then____ then_ cut unnecessarily moves to the object cut before actually starting to cut (see line 19). then moves to the cut location inside the while loop of checking if the object has been cut yet or not (see line 22) A more optimal code will be move the location before the while loop, then just have the call to function in the while loop.
core issue with 2-Layer (Comp) is that the LLM needs account for the physical constraints when generating the code in the first step. However, the specification that the LLM receives only includes information about the order between high-level actions and high-level control flows (e.g. for loops). The LLM is more prone to mistakes when it has to extrapolate the physical constraints from the high-level specification. Mcanwhile, 2-Layer (High), which has the same number recursive steps. has generated the correct initial code that the LLM can expand on. Its initial code aligns closer the specification (see the code in section 14.3.3).
Get list of all the lettuces in the kitchen. lettuces - get get_all_obj_nane htt__aaessttt_mmeeeeeeeee thaat_mtchh_ypp ('lettuce')
Get list of all the cutting boards in the kitchen. cutting.boarde
thapeeessttttttthttttaae type outting-board
Decide cutting board to use. cutting.board
cutting. boards [0]
Repeat the following twice 10 for i in range (2) (2):
Decide a lettuce to cut.
12 lettuce_to_cur
lettuces
13 Cut that lettuce at that cutting board. To cut lettuce_to_cut, the robot first needs to be holding lettuce_to_cut
14
if not is is_holding (lettuce_to_cut)
16
# the robot is not holding lettuce_to there are 2 scenarios:
17
# (1) if lettuce_to_cut cut is in stack unstack
lettuce_to_cut
18
# (2) else, pick up lettuce_to_cut.
19
if is_in_ (1etuceetttt
20
# Because lettuce_to_cnt is in a stack, robot need to move then unstack the lettuce_to_cu from the abjtot_bottom first abj_at_botton lettuce_to_cut)
22
move teentheeeuueee wtwee_thee_nnsu((OO ooo.ecccooooooutttteeeeee.o..... obj abj_at_bottee Abj_at_bottom=ohy at_obttootoooto unstack mastackklocationsgee locatioon
obj abj_at_bol boottm))
else:
24
Since lettuce_ is not in stack, robot can just move then pick it www..bae.ppccccojjllatuueeeee.
location then cut move to the cutting.board_toouus lettuce_to_cut
27
move then, cut cut (obj_to_ -et cut_location- betting_boar__tooo.
def move move_then_
unsttkkk abj_at_bottan unstack_location
30
For unstacking, we need to move to the location of the bottom object
bottom_location get.ool_.occttioocoo.......
42

---

if get_curr_location( := botton_location: move move (), cnttom_location unstack (obj_oo._ooot. obj_at_bottom)
36
35
1f get_curz_locaation !- unstack_location:
36
move(get.curr_locata)
instack_location
def move move pove_then..ickkob))).
39
obj_location
_et_obj_locatio (obj)
40
if
!= obj
O), obj_location)
pick_up obj_location)
43
44
def move move_then, tttteettteenuutteeeeeee chhenncut(bbj.to.cuut cut_location):
45
obj obj_location get.chh_.ooationnobj.ccc tobj_to_cutt
if pet__cur_location() obj_location:
move [ere@ettccrr..ooatioon)] oto__p_r__aaaaioooooooooooaaall while not is_cut(obb__o_cuut)
obj.location)
48
49
if get_curr_locaane move ler._ettrr__ottttttooo cut (objo.to_cuut)
!= cut_location
90
cut_location)
14.3.3 Why 2-Layers (High) failed?
2-Layers (High) generated the same initial high level code as Demo2Code. However, the definition of cut_obje objjeb__ttttt.....o.oo act_at_location in the recursive step is not exactly correct because the LLM is not taking account of all the physical constraints. It again assumes that the object cut already on the cutting board to cut. It moves to the object's location before starting to cut the object (line 18). Although 2-Layers (High) has 0.2 unit test success rate, it achieved this because many of the lettuces happened to be placed on cutting boards in the randomly initialized environments.
Get a list of all the lettuces in the kitchen. lettuces = get_all_obj_aa Get list of all the cutting boards the kitchen. cutting_boarde
('lettuce')
nawes_naaa_________ thamts_that_ssee wtt.htttttttthhtpppteee cuttinggboard
Repeat the following twice: for range (2):
Decide a lettuce to cut.
tettuce_to_cut = lettuces[i]
# Decide a cutting board to use. bacting_board_to_us Cut that lettuce at that cutting board.
cutting_boards [i]
12
13
cut aet_-obeett_a__lcatte cutting.board_to.
location
14
15 def cut cut_ objecttjet_att lo_aaatioon (obj. location): To cut an object, the robot first needs to at the same location as obj
16.
17
if loctatonn) I= let_obj_location((0))) location O,, get_obj location (ocation(ob))) cut the object
18
move move(get_curr coctttaioooooo
19
20
while not is_cut (obj):
cut (obj)
14.3.4 Demo2Code Output
Demo2Code is able to adhere to physical constraints and correctly defines all the helper function in its recursive expansion steps.
def cut_
(obj location): To cut an object, the robot first needs to be holding obj
if not is is_holding (obj):
# If the robot is not holding obj, there 2 scenarios:
43

---

# (1) if obj is in stack unstack obj
# (2) else, pick up obj. if as_in
Because obj is in a stack, robot need to move then unstack the obj from the obj_at_bottom first obj_at_botton move_then, move_then_unstack (obj abj_at_ at _ottomm unstack ott.aeee...aatorrrrooooooooooooa
obj_at_botton
11
elae:
12
# Since obj is not in stack, robot can just move then
pick it up
13
pove_then_.iccoobjeeee place the object at the location to cut at
14
15
move hove_then_plar place place_locacionnlcccctt:)) cut the object
cut_until_is_cou (obj=obj)
18
19
def cut wewuuniiliiie cut (obj):
20
while not cu_ (ob)):
cut (obj)
22
23
def move then unstack to_nsack,,,,, obj_at_ unstack_locatiin
For unstacking, need to move the location of the bottom object
25
get_curr_ getcoh__oocciiioooo.o...ttttttot
26
cove@fet_.urr_loctiio (obj..toobbbtoo... _t_ttac__stkkk abj_at_bottont
get get..bb_.ccatiooccbb.....ooo.
27
After unstacking, need move to the unstack_location
29
get_curr.location umstack_location: more(get.ccrr..lcatiin unstack,location))
30
31
32
def move (wt.thhenniccxo)))
obj_tocation _ot_oob__occtionn(0
34
get_curr__ocationn
obj_location:
35
(), obj.location)
36
pick_up(obj obj_location
38
def move_ tove_ place_ alace_locatoon:
39
get_curr_ != place_location
40
move Cove seret__cur___eeeee cowrrrrcctaiionno (), place_locatio
place (obj, place_location
15 Broader Impact
Our approach a step towards making collaborative robots more accessible in different settings, such as homes, factories, and logistics operations. The broader implications of our research are manifold, touching upon societal, economic, ethical, technical, and educational aspects.
Currently, robots can only programmed by engineers, limiting the tasks the robot can do to design choices made by engineers. This limits the accessibility for everyday users who require personalization to their individual needs and use cases. Our work tackles this problem head-on by allowing robots to be easily programmable via intuitive, user-friendly interfaces like vision and language. This could lead increased creativity in the types of tasks that robots can be used for, potentially enabling novel applications and solutions.
Societal Impacts:
Economic Impacts: Our approach has the potential to dramatically decrease the cost of program- ming robots, thus lowering barriers entry for businesses interested in incorporating robotics into their workflows. This could boost productivity and efficiency across variety of industries, such as manufacturing, agriculture, and healthcare, among others. In the long term, this may contribute to economic growth and job creation in sectors that are currently behind on automation.
44

---

Ethical and Legal Impacts: Our approach has many ethical and legal considerations that need to be addressed. For instance, the widespread use of automation may lead to job displacement in certain sectors. Furthermore, it's crucial to ensure that the task code generated from demonstrations respects privacy laws and does not inadvertently encode biased or discriminatory behavior. Future work in this area will require close collaboration with ethicists, policymakers, and legal experts to navigate these issues.
Technical Impacts: Our work has the potential to accelerate the development of more intuitive and efficient human-robot interaction paradigms. For instance, demonstrations can be extended to interventions and corrections of the user. On the algorithmic side, our approach is first step to connecting LLMs to Inverse Reinforcement Learning, and can spur advances in the fields of imitation learning, reinforcement learning, and natural language processing.
Educational Impacts: Lastly, our work could contribute to educational initiatives. The ability to effective teaching tool in schools
generate task code from demonstrations could be utilized and universities, promoting a more experiential and intuitive approach learning about roboties and coding. This could inspire and enable more students to pursue careers in STEM fields.
16 Reproducibility
We ram all our experiments using GPT-3.5 (gpt-3. 3. b-rbbo) with temperature 0. We have provided codebase in http:///ww....oo//oooooo////////
and all the prompts in section [8 The queries used to produce our results are available in the code base. Note that although we set the temperature which will make the LLM output mostly deterministic, the output might still slightly vary even for identical input.
17
Demo2Code Example Output
We provide an example for each domain and explain Demo2Code's intermediate and final output.
17.1 Tabletop Simulator Example
This section shows an example of how Demo2Code solves the task: stacking all cylinders into one stack. The hidden preference is that certain objects might need to have fixed stack order, while other objects can be unordered. For this example, two objects ('blue cylinder' and 'purple cylinder') must follow the order that 'blue cylinder' should always be directly under the 'purple cylinder.'
17.1.1 Query
The query has components: (1) a list of objects that are in the environment, (2) language instruction describing the goal of the task, and (3) two demonstrations.
objects- pink block', 'yellow block', 'purple cylinder', 'cyan cylinder', 'pink cylinder', 'blue cylinder']
[Scenario 1]
Stack all cylinders into stack, while enforcing the order between cylinders if there a requirement.
State 2: bbuu cylinder' has moved
State 3:
'purple cylinder' is not on top of 'table' 'purple cylinder' has moved
'prrple cylinder' is top of 'blue cylinder'
State 4:
'cyan cylinder' is not on top of 'table' 'yaan cylinder' has moved
45

---

cyan cylinder' is on top of 'purple cylinder' cyaa cylinder' is top of 'blue cylinder'
State 5:
'pink cylinder' is not on top of 'table' 'pink cylinder' has moved
'pink cylinder'
top of 'purple cylinder'
'pink cylinder' on top 'cyan cylinder' 'pink cylinder' is on top of 'blue cylinder'
[Scenario 2]
Stack all cylinders into one stack, while enforcing the order between cylinders if there is a requirement.
State 2:
"yaa cylinder' is not top of 'table'
'cyan cylinder' has moved
"cyan cylinder' is on top of 'pink cylinder'
State 3:
not on top of 'table' 'blue cylinder' has moved
'blue cylinder'
bblue cylinder' is on top of 'cyan cylinder'
blue cylinder' is on top of 'pink cylinder!
State 4:
'purple cylinder' is not on top of 'table' 'purple cylinder' has moved
'purple cylinder' is top of 'cyan cylinder' 'purple cylinder' is top 'pink cylinder'
'purple cylinder' is on top 'blue cylinder'
17.1.2 Intermediate output
In stage 1 recursive summarization, for each demonstration, Demo2Code summarized the trajectories into low-level actions (e.g. moved "blue cylinder', picked up and placed 'red cylinder' on top of 'blue cylinder'). Then, because tabletop's domain has short demonstrations, the L.M determined that low-level action trajectory was sufficiently summarized.
Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM first identified the order of the stack in each scenario. Then, it compared the two orders in the two scenarios before concluding that the order that stays the same [ b'luue cylinder', 'purple cylinder'] It finally produced the correct specification that will get passed to stage 2 to produce the correct code.
Stage Recursive Summary: recursive call # with state-based trajectories as input
sum o
[[Reasoning:]]
The input trajectory contains state predicates because trajectory
talks about the status of the objects ('red cylinder' is not on top of ")abee).
will summarize the state trajectory into lou-level actions. The level actions are the most compact form, should stop summarizing after this.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently aummarized? (yes/no) :]] yes
[[Sumnarized trajectory:]]
In [Scenario 1], at state 2, the robot moved 'blue cylinder'.
At state 3, the robot picked up and placed 'red cylinder' on top of 'blue cylinder'.
46

---

At state 4, the robot picked up and placed 'orange cylinder' top of 'red cylinder'. At state 5, the robot picked up and placed 'cyan cylinder' on top 'orange cylinder'.
sumaarzing demo
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('orange cylinder' has moved )
will summarize the state trajectory into low-level actions.
The low-level actions are most compact form, so should stop aummarizing after this.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no):]) yes
[[Summarized trajectory:ll
In [Scenario 2], at state 2, the robot moved 'orange cylinder's At state 3, the robot picked up and placed 'blue cylinder' on top of 'orange cylinder',
At state 4, the robot picked up and placed 'red cylinder' on top 'blue cylinder'.
At state 5, the robot picked up and placed 'cyan cylinder' on top of 'red cylinder'.
Summarized Demonstrations -> Task Specification
[[Reasoning:]]
Because the task is "Stack all cylinders into one stack, while enforcing the order...", we focus on whether there is any ordering requirement.
In [Scenario 1], based on States 2-5, the first stack, the cylinders from bottom to top is ['blue cylinder', 'red cylinder', 'orange cylinder', 'cyan cylinder 'J. In [Scenario 2], based States 2-5, in the first stack, the cylinders from bottom to top ['orange cylinder', 'blue cylinder 'red cylinder', 'cyan cylinder '].
Between the order of cylinders in these two scenarios, the ordering that stays the same is ['blue cylinder', 'red cylinder', 'cyan cylinder
[[Task Specification:1]
Stack all cylinders into one stack. Also make sure that the stacking enforces the bottom to top order between these cylinders: ['blue cylinder', 'red cylinder', 'cyan cylinder '].
17.1.3 Output Code
This is the final output of Demo2Code, which the task code to solve this problem. The code illustrates the LLM's ability write complex code that has control flows (e.g. for loop and if-else statements.)
Based on the instruction, first define the cylinders that have strict ordering
['blue cylinder'
purple cylinder
Then, find the rest of the cylinders that don't have strict order cylinder_names= tth.all.oo..aamss.ttttaaaaaaa cylimeer.naees.tthoutts for cylinder in cylinder_name:: if cylinder not in sylinder.namos.wtthho wtt.:sstccccccccrrreeoeee httpiic..rrrrr.fommoottteeee
cylineer
())
47

---

append (cylinder)
Call helper function determine the final stacking order before stacking the cylinders
stack
to_
wtt.:/aaaaaattttttorrrorooo ordeeottoorderr
stack_ http::/ttoot...ghh...iiiiiiiiioa
stack, order from ardder_fronbbottmmttoo----
17.2
Robotouille Example
17.2.1 Query
The example we will consider Make burger stacking patty alop lettuce immediately. This task that contains many different actions and is successfully solved by Demo2Code, making it comprehensive example. The query for this task consists two demonstrations each of which has the same language instruction to make burger. Because each demonstration has 25 states, we have placed the query at the end of the appendix in section 19.1
17.2.2 Intermediate output
Below, in stage recursive summary, for each demonstration, Demo2Code summarizes each trajectory from state-based trajectory to low-level action trajectory high-level subtask trajectory, before LLM decides that the trajectory is sufficiently summarized.
Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM first identified the order of the subtasks in each scenario. Then, it determined that the lettuce needs to be cut and stacked on top of the bottom bun before cooking the patty and stacking the patty on top of the lettuce. It finally produced the correct specification that will get passed to stage 2 to produce the correct code.
Stage Recursive Summary: recursive call # with state- based trajectories as input
c...s.aa*aeaeete weg.eeeeeeoeeeennnnnniniiin demo demo===tt
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the robot ('roboti' is at 'table2') and the status of the objects ('lettucel is not at 'table2"). I will summarize the state trajectory into lou-level actions.
levelevel actions are not compacted enough yet because level actions can still combined into high-level subtasks. The new trajectory will NOT be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yea/no) :]]
[[Sumnarized trajectory:]]
In [Scenario 1], at state 2, the robot moved from 'table1 to table2'.
At state 3, the robot picked up 'lettucel',
At state 4, the robot moved from 'table2' 1 cuttiing.board
At state 5, the robot placed 'lettucei" at location 'cutting_board
At state 6-8, the robot had cut 'lettucel', At state 9, the robot picked up 'lettucel
state 10,
robot moved from "cutting.board" to 'table3
At state 11, the robot placed 'lettucel' on top of 'bottom_bun buni at location 'table3'.
At state 12, the robot moved from 'table3' to 'stovel',
At state 13, the robot picked up 'pattyl'.
At state 14,
robot moved from 'stovel' 'stove4'.
At state 15, the robot placed 'pattyl' at 'stove"',
At state 16-19, the robot had cooked 'pattyl'.
At state 20, the robot picked up 'pattyl'.
48

---

At state 21, the robot moved from 'stoved' to 'table3'.
At state 22, the robot placed 'pattyl" top of 'lettuce1 location 'table3'.
At state 23, the robot moved from 'table3' to 'tabled'.
At state 24, the robot picked
"top_bun''.
At state 25, the robot moved from 'table4' to 'table3'.
At state 26, the robot placed 'top_bunl' top of 'pattyl' location 'table3'.
summarizinn demo
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status the robot ('robot1' at becting.board5) and the status of the objects ('lettuce3' is not at boatinggboard5").
will summarize the state trajectory into low-level actions. Low-level actions are not compacted enough yet because low-level actions can still be combined into high-level subtasks. The new trajectory will NOT be sufficiently summarized. [[Is the trajectory sufficiently summarized? (yes/no) :]] no
[[Summarized trajectory:ll
In [Scenario 2], at state 2, the robot moved from 'table7' to boardd.
At state 3, the robot picked up 'lettuce3', At state 4, the robot moved from 'cutting_board5 to 'cutting_boarde
At state 5, the robot placed 'lettuce3' at location 'cutting.board3
robot had cut 'lettuce3', At state 6-8,
At state 9, the robot picked up 'lettuce3',
At state 10, the robot moved from 'cutting.board3' to 'table4
At state 11, the robot placed 'lettuce3' on top of bun3 at location 'table4'.
At state 12,
robot moved from 'table4' 'table7'.
At state 13, the robot picked up 'patty3',
At state 14, the robot moved from 'table7' to 'stove3'.
* At state 15, the robot placed 'patty3' on 'stove3'.
At state 16-19, the robot had cooked 'patty3'.
At state 20,
robot picked up 'patty3'.
At state 21, the robot moved from 'stove3' to 'table4'.
* At state 22, the robot placed 'patty3' on top of 'lettuce3' at location 'table4'.
At state 23,
robot moved from 'table4' 'outting_bbard4"
At state 24, the robot picked up 'top_bun3',
At state 25, the robot moved from 'cutting_board4' to 'table4',
* At state 26, the robot placed 'top_bun3' on top of 'patty3' at location 'table4'.
Stage Recursive Summary: recursive call # with low-level action trajectories as input
r*seesnneaaeeee sunnaaniiin demo dem
[[Reasoning:]]
The input trajectory contains lov-level actions because the trajectory mentions "moved", "picked up", etc.
I will summarize the lov-level action trajectory into high- subtasks.
High-leve subtask trajectory is the most compacted form that cannot be summarized anymore.
The new trajectory will be sufficiently summarized. [[Is the trajectory aufficiently aummarized? (yes/no):1) :]1 yes
[[Summarized :rajectory:]]
In [Scenario 1], at state 2-8, the subtask is "cut lettuce", This aubtask contains: 1. moving from 'tablei to buttinng.borrdi
49

---

state 2-4) 2. placing 'lettucel' at 'cutting_board1 (state 5) 3. cutting 'lettucel until it is cut (state 6-8)
At state 9-11, the subtask is "stack lettuce on top of bottom bun".
This subtask contains: 1. picking up 'lettucel' (state 9) 2. moving to stack 'lettucel' on "bottom_bunl' (state 10-11)
At state 12-19, the subtask is "cook patty". This subtask contains: moving from 'stovel' pattyl at 'stove4' (state 15) 3. cooking 'pattyl' until is cooked (state 16-19)
'stove4' (state 12-14) 2. placing
At state 20-22, the subtask is "stack patty on top of lettuce", This subtask contains: 1. picking up 'pattyi' (state 20) 2. moving stack 'patty1" on 'lettuce1' (state 21-22)
At state 23-26, the subtask is "stack top bun on top patty". This subtask contains: 1. moving from 'table3 to 'tabled' (state 23)
2. picking up sonpbbun'' (state 24) 3. moving to stack 'top_bunl 'pattyl' (state 25-26)
summarizing demo
[[Reasoning:]]
The input trajectory contains low-level actions because the trajectory mentions "moved", "picked up", etc. will summarize the low-level action trajectory into high-level subtasks.
High- subtask trajectory is the most compacted form that cannot be summarized anymore.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) yes
[[Summarized trajectory:]]
In [Scenario 2], at state 2-8, the subtask "cut lettuce", This subtask contains: 1. moving to pick up 'lettuce3' (state 2-3) moving to place 'lettuce3' on 'cutting_board3" (state 4-5) 3. cutting '1ettuce3' until it is cut (state 6-8)
At state 9-11, the subtask is "stack lettuce This subtask contains: moving to stack 'lettuce3' on 'bottom_bun3" (state 10-11)
top of bottom bun". picking up 'lettuce3' (state 9)
At state 12-19, the aubtask is "cook patty". This subtask contains:
1. moving to pick up 'patty3' (state 12-13) 2. moving to place patty3'
'stove3' (state 14-15) cooking 'patty3' until it cooked (state 16-19)
At state 20-22, the subtask is "stack patty on top of lettuce", This subtask contains: 1. picking up 'patty3' (state 20) 2. moving to stack 'patty3' on 'lettuce3' (state 21-22)
At state 23-26, the subtask is "stack top on top patty", This subtask contains: 1. moving to pick up 'unp_bun3' (state 23-24) 2. moving to stack 'top_bun3' on 'patty3' (state 25-26)
Summarized Demonstrations -> Task Specification
[[Reasoning:]]
In [Scenario
unique kitchen environment, the subtasks executed in this order: ['cut lettuce', 'stack lettuce on top of botton bun', 'cook patty', 'stack patty on top of lettuce', 'stack top bun on top of patty'
In [Scenario 2s unique kitchen environment, the subtasks were executed this order: ['cut lettuce', 'stack lettuce on top bottom bun', 'cook patty', 'stack patty on top of lettuce', stack top bun on top of 'j.
Both scenarios are executing the subtasks in the same order. There is repetition individual scenario.
loop the subtask ordering for any [[Task Specification:]] Make burger.
50

---

Specifically:
Get list of all the lettuces in the kitchen. Get a list of all the botton buns in the kitchen. Get a list of all the patties in the kitchen. Get list of all the top buns in the kitchen.
Decide lettuce
Get a list all the available cutting boards in the kitchen.
Decide a cutting board to go to.
Cut that lettuce at that cutting board.
Decide a botton bun to use. Stack the lettuce on top of the bottom bun.
Decide patty Get list of all the available stores the kitchen. Decide a stove to go to. Cook that patty at that stove.
Stack the patty top of the lettuce.
Decide a top bun to use. Stack the top bun on top of the patty.
17.2.3 Output Code
This is the final output of the LLM, which is task code to solve this problem. The code illustrates the LLM's ability to write complex code that has control flows (e.g. for loop and if-else statements.)
Task Specification -> High-Level Code
Get list of all the lettuces in the kitchen. lettuces
www..bonaansstaaatteeeeee.e type ('lettuce')
Get a list of all the bottom buns in the kitchen. bottom_bume
www.aatttyppebbttttecc Get list of all the patties in the kitchen. patties
('patty')
Get list of all the top buns the kitchen. 'unp_bun')
top_buns = get_all.obj_nanes. thatees_thtt_machhhe
Decide lettuce to use.
lettuce_to_use lettuces [0]
Get a list of all the available cutting boards the kitchen.
cutting_boards = haw.ioootatoonnmmetttatttt (at.aaaaaaaataeeetaaaaaaoaooono ('cutting
Decide cutting board to go pertinn.board_to.. Cut that lettuce at that cutting board.
cutting_boards [0]
hot.ctaatttccaiiinoooooeeeee cutting.bard
location=
Decide a bottom bun to use.
peetoom_uu__to use botton_buns [0]
Stack the lettuce on top of the bottom bun.
obji should be the lettuce, obj2 should be the bottom bun. otbpao.....oooc22ecceeeoeeeeaaaa
obj2-bottom ton_to_use!
Decide a patty to use.
patty_to_use batties[o]
Get list of all the available stoves in the kitchen. stoves
stove_to_comd haallboattooonmmeehhtt...... Cook that patty at that stove.
('stove))))
Decide a stove to go to.
stoves [0]
51

---

cook_objject.at potyyaaattttoooopatatttttoto..o
Loattoorsstoe__tocoob....
Stack the patty on top of the lettuce. obji should be the patty, obj2 should be the lettuce.
aoj=-lettuce_t__see
Decide top bun to top_bun_to_use top_buns [0]
Stack the top bun on top of the patty.
obji should be the top bun, obj2 should be the patty.
obt.eooooi_oo_.oooeeeiiiiiii to obf2-patty_to_uso
Step 2: Define composite actions
def cejjeet_at_loaattioo
location):
To cook an object, the robot first needs to holding obj not is is_holding (obj):
# If the robot is not holding obj, there are 2 scenarios: # (1) if obj is in stack unstacck obj (2) else, pick up obj.
if is_in_a_stent stnaa
# Because obj is in a stack, robot need to move then unstack the obj from the obj_at_bottom first itn (otjaatttoobjttttoppop move then unstack: first you move to the obj_at_bottom location, then you unstack obj from abj_at_bottom nove_then_ tob obj tottooo else:
obj_at_botton
obj_at_bottom.
move_then first you move to obj's location, then
you pick obj up nove_then_picke jiv__teenpiic(ooj-op. pick
)
move then place: first you move the location to cook at, then you place obj at that location move move_then_plie thentthlaceeeeeepppppppptttteeee (obj=obj, (objooob,,, cook the object until it is cooked coot_uafi..ie_cootd((oo....))))
place_lccttoon:occcciin))
def stack_objj_onn.ojf[(0
obj2):
# To stack obj1 on obj2, the robot needs to be holding obj1 if not is hs_holding (objl):
the robot is
obj1 is holding obji, there are scenarios: stack unstack obj1
# (1) # (2) else, pick obj1.
if as_in_a__ttccxobbpp # Because obj1 is in stack, robot need to then unstack the obj from the obj_at_ first abj_at_botton
move then unstack: first you move to the abj_at_bottom's location, then you
obj from ab obj_at_bottom:
else:
nove_then.pick first you move to obj's location, then you pick obj up obj1))
lick,oocationggtt.co l lbj2_iocation get_obj_location (objj)
move_then hove_then.piccooj-ooo
# determine the location of obj2 to stack
move then stack: first you move obj2's location, then you stack obji on obj2 move teos_thhnsstccke sta stack stcct_loattioo-oo]]llaa
obj abj_at_bottommobb..
52

---

def Cut_objeet.._.Jcattiooo(0( (obj, location):
To cut an object, the robot first needs to holding obj if not is_holding (obj):
# If the robot is not holding obj, there are 2 scenarios: # (1) if obj is in stack unstack obj (2) else, pick up obj. if as_in_a_stack
# Because obj is in a stack, robot need to move then unstack the obj from the obj_at_bottom first
obj_at_botton
get_ojj.thtt_issudeeeeee topp
move then unstack: first you move to the obj_at_botton location, then you unstack obj from abj_at_bottom seve_then_unstacee obj st__botoom, (obe unstaakkooott obj_at_bottom locattoon (cbj
else:
move_then.pickk first you move to obj location, then
you pick obj up move_then_plcccobj-ott pleh.locatongget.....ootttt (obj
)
move then place: first you move the location to cut at, then you place obj at that location
move _ thennppace (obj=ob,,
place_locationslccaa:in))
# cut the object until it is cut (obj=obj)
Step 3: Define low-level actions
def move then unstack bebjj_to_unstack obj_at_botton, unstack_location)
if get_curr_locattine != unstack.
move cove O), lnstack.location)
unstack
abj_at_botton.
def thoe_ pick__ocatioo):
get_curr_locationn != pick_ pick_location:
move move(geet.urr._Jocation pick_location) pick_up piccpooooo..cc..occttiooooo pick_location)
def move_then__laace(0)), place_location): get_curr_
place_
O,, locatioo
move place place_location
def uo (obj) start_cooking
while not is (obje ()
def move tove__the_stacccobjj..o. stackk obj_at_botton, stack_location):
1f get_ cet_cuur_locatiine
!- stack laackioocatonn:
(wtrteeeetrrrrrcaeeeeeee O), stack_location
stack tobck obj_at_bottom
def cut_until_is_cuu (obj): while not cut (obj): cut (obj)
17.3 EPIC-Kitchens Example
17.3.1 Query
present an example of demonstration given as a query to the L.M.. The annotations are made for video id p22.2002 07. This example shows the LLM generalizing to multiple new objects, new
53

---

ahardl) merzakure
2):
sonpy, posse
Interdi preferzotato. ('aadinyy
diant beard cuttin_ 1
150n| TAP_13
soapy! coard.cutingg.
peetee petono boadrrstinn
"'sccity boarrtiutiing
bowdrarting
dryingracc
lnband]
1')
dryinggack (1)
inhadd poster poiato, 1%. dean; pee poant
Iaph 17)
In pesdeepppo ddyingrack 1)
Figure 14: Example of annotations for video P22_07 in 14 frames
locations, and different dishwashing style. Figure 14 shows visualizations of the video with the respective state annotations. Because each demonstration has states, we have placed the query at the end of the appendix in section 19.2
17.3.2 Intermediate Output
Below, in stage recursive summary, each demonstration, Demo2Code summarizes each trajectory from a state-based trajectory to low-level action trajectory to high-level subtask trajectory, before the LLM decides that the trajectory is sufficiently summarized.
Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM was able identify that should use a for-loop because [pick up, soap, move object from A to B] are repeated. It finally produced the correct specification that will get passed stage 2 to produce the correct code.
Stage Recursive Summary: recursive call # with btste trajectories as input
r summarizinn demo =mo
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects etc.).
(ert((mmzzaluna_11))
will summarize the state trajectory into lov-level actions.
lov-level actions are not compacted enough yet because level actions can still be combined into high-level subtasks.
The new trajectory will NOT be sufficiently summarized. [[Is the new trajectory sufficiently aummarized? (yes/no) :]]
[[Summarized trajectory:]]
In [Scenario 1], at state 1-2, the robot picked up mezzaluna_1 from sink_1.
At state 2-3, the robot soaped mezzaluna_1 until it became soapy.
54

---

At state 3-4, the robot moved mezzaluma_1 from sink_1 to sink_2. At state 4-5, the robot moved peeler peeler:potato. potato_1 from sink_1 to sink_2. At state 5-6, the robot picked up peeler: : potato...
At state 6-7, the robot soaped peeler:potaton
until it became soapy
At state 7-8, the robot moved peeler:
from sink_1 to 2ink_2.
At state 8-9, the robot moved knife_1 from sink_1 sink_2.
At state 9-10, the robot picked up knife_1.
At state 10-11, the robot soaped knife_1 until it became soapy.
At state 11-12, the robot moved knife_1 from sink_1 to sink_2. At state 12-13, the robot moved board: from sink_1 sink_2.
At state 13-14, the robot picked up cutting.
At state 14-15, the robot soaped board: cutting. until it became soapy.
At state 15-16, the robot moved board:cutting. from sink_1 sink_2.
At state 16-17, the robot turned on tap_1.
At state 17-18, the robot picked up board: cutting....
At state 18-19, the robot rinsed board: until it became clean.
At state 19-20, the robot placed board:cutting.1
dryingrack_1.
At state 20-21, the robot picked up mezzaluna_1.
At state 21-22, the robot rinsed mezzaluna_1 until it became clean.
At state 22-23, the robot placed mezzaluna_1 in dryingrack_1.
At state 23-24, the robot picked up :npllerpotato__.
åt state 24-25, the robot rinsed peeler peeler:potato._ potato_1 until it became clean.
At state 25-26, the robot placed
in 1ryingrack_1. At state 26-27, the robot picked up knife_1.
At state 27-28, the robot rinsed knife_1 until became clean.
At state 28-29, the robot placed knife_1 in dryingrack_1.
* At state 29-30, the robot turned off tap_1.
Stage Recursive Summary: recursive call # with low-level action trajectories as input
[[Reasoning:]]
The input trajectory contains lov-level actions because trajectory mentions "picked up", "moved", "soaped", "rinsed", etc.
I will summarize the lov-level action trajectory into high-level subtasks.
High-level subtasks combined together without losing important information.
compacted enough because they cannot
The new trajectory will be sufficiently summarized, [[Is the new trajectory sufficiently summarized? (yes/no) :]] yes
[[Summarized trajectory:]]
In [Scenario 1], at state 1-3, the robot brought mezzaluna_1 from sink_1 to sink_2.
At state 3-4, the robot brought peeler potato_1 from sink_1 to sink_2.
At state 4-5, the robot picked up peeler:potato.1: At state 5-6, the robot soaped peeler:potato.1 until it became soapy
At state 6-7, the robot moved peeler:potato. from sink_1 1 to sink_2. robot
state 7-8,
knife_1 from sink_1 to sink_2.
At state 8-9, the robot picked up knife_1.
At state 9-10, the robot soaped knife_1 until it became soapy.
* At state 10-11, the robot moved knife_1 from sink_1 to sink_2.
At state 11-12, the robot moved board: from sink_1
sink_2.
At state 12-13, the robot picked up board: board:cutting...
At state 13-14, the robot soaped board: board:cutting_1 until it became soapy.
55

---

At state 14-15, the robot moved board:cutting. from sink_1 to sink_2.
At state 15-16, the robot turned on tap_1.
At state 16-19, the robot rinsed board: cutting., mezzaluna_l,
and knife_1 until they became clean.
At state 19-20, the robot placed boarinngtting.. in dryingrack.1.
At state 20-21, the robot picked up mezzaluna_1.
At state 21-22, the robot rinsed mezzaluna_1 until it became clean.
At state 22-23, the robot placed mezzaluna_1 in dryingrack_1.
At state 23-24, the robot picked up peeler: potatoo_.
At state 24-25, the robot rinsed peeler potatoo_1 until it became clean.
At state 25-26, the robot placed peeler: :otatoo_ in dryingrack.1.
At state 26-27, the robot picked up knife_1.
At state 27-28, the robot rinsed knife_1 until it became clean.
At state 28-29, the robot placed knife_1 in dryingrack_1.
At state 29-30, the robot turned off tap_1.
Summarized Demonstrations >> Task Specification
summary ->
[[Reasoning]]]
There are objects that need to be washed: [mezzaluna_1, peeler: potato_1, knife_1, boardicutting.ll
The list of high-level actions that happened in order is: [move object from A to B, pick up object, soap object, move object from tap, rinse objects, place object in drying rack,
A to turn off tap]
The actions [pick for each object, so we can use loop.
soap, move object from A to B] are repeated
Rinsing involves the use of tap water, which is why it is turned some point before the rinse cycle, and turned off after. [[Task Specification:]]
Wash objects in sink. All clean objects must be placed in drying rack. Specifically
Get list all objects to wash For each object in all objects: Move object from sink to sink_2 Pick up object
Soap object
Move object from sink_1 to sink_2 Turn on tap_1 For each object in all objects:
Rinse object
Place object 1
Turn off tap_1
17.3.3 Output Code
This is the final output of Demo2Code, which is the task code to solve this problem. The LLM captures that the objects are soapped first, then rinsed one by one after. The code also illustrates the LLM's ability to write complex code that has control flows (e.g. for loop and if-else statements.)
objects = set_all_objecte ()
for object objects:
bring_ob srjjngobjjectsttor tocets_tt_lloc ( ([object], "sink_2")
pick_up (object)
sobp(tb))ct) ourn_on("tap_1")
"sink_1")
for object in objects:
rinse robject)) (object 1dryingrack_1")
56

---

turn_off ("tap_1")
18
Prompts
All prompts are in yaml file representation. For the most updated prompt, please refer code base:
18.1
Tabletop Manipulation Task Prompts
18.1.1
Recursive Summarization
scarsivessumarratiion
main: I
You are helpful summarizer that recursively summarizes trajectory into a more and more compacted form. letd_fff_sstt__meeeeeee_ee You are given trajectory. Your goal is to summarize the trajectory into the state trajectory is sufficiently summarized.
compacted form and then determine whether
You must respond using the following form (you must generate [[end of responsel] at the very end): [[Reasoning:]]
You should first identify what type of trajectory this is. Then, you should determine what type you will summarize the trajectory into. Finally, you should determine whether the new type of not.
trajectory is sufficiently summarized [[Is new trajectory sufficiently summarized? (yes/no): :1] You should only reapond using either "yes" or "no", and nothing else.
[[Summarized trajectory:]]
You should actually summarize the input trajectory output here.
[[end of response]]
You must follow these rules when you summarizing trajectory.
Rules:
You must slovly summarize the trajectory from one type to another following this order: a state trajectory a lov-level action trajectory.
level action trajectory is represented list. Each line in the unordered list should only contain one low- level action.
unordered
* The lov-level actions must be one of the following: "move <item1 >", "picked up and placed <iteml> rellaire.posttioorreaaatoo item2>", "picked up and placed <iteml> at <loci>", "picked up and placed <itemi> on top of 'table"" "picked up and placed <itemi> on top of <item2>", You should never define new levelevel actions. examples:
<
[[Input Trajectory:]]
[Scenario 1] Place the green cylinder next to the yellow cylinder.
State 2:
'yellow cylinder' is to the right of green cylinder' 'green cylinder' has moved 'green cylinder' is to the left of 'yellow cylinder' <end_of of wmmd_off.xxmplle.ssr.ueeee [[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('yellow cylinder is to the right of 'green cylinder
57

---

I will summarize the state trajectory into low-level actions. The low-level actions the compact form, I should stop summarizing after this.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :]] yes
[[Summarized trajectory:ll
In [Scenario 1], at state 2, the robot picked up and placed green cylinder' to the left of 'yellow cylinder'. [[end of response]]
[[Input Trajectory:l] [Scenario 1] Place the purple cylinder next to the orange block.
State 2:
'orange block in front 'purple cylinder' 'purple cylinder' has moved
'purple cylinder' is at 'top edge'
'purple cylinder' is behind 'orange block' cendoof..xample.user..oo [[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('orange block is in front of 'purple cylinder will summarize the state trajectory into low-level actions. The lov-level actions are the most compact form, so I should stop summarizing after this.
The trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no):: :]] yes
[[Summarized trajectory:]l
In [Scenario 1], at state 2, the robot picked up and placed purple cylinder' behind 'orange block'. [[end of response]]
[[Input Trajectory:
[Scenaric 2]
Place the blue block the corner.
State 2:
'blue block' has moved 'blue block' is 'top right corner exaapoof. [[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('blue block' has moved).
will summarize the state trajectory into lov-level actions.
The lov-level actions are the most compact form, so I should stop summarizing after this.
The trajectory will be sufficiently summarized.
[[Is the trajectory sufficiently summarized? (yes/no): :11 yes
[[Summarized trajectory:]l
* In [Scenario 1], at state 2, the robot picked up and placed blue block 'top right corner'. [[end of response]]
[[Input Trajectory:1 :]] [Scenario 1]
Place the cyan block the blue block.
State 2:
'orange cylinder' is top of 'table'
58

---

'orange cylinder' has moved
orange cylinder' is not top 'red block'
orange cylinder' is not on top "cyan block"
orange cylinder' is not on top of yellow block
State 3:
'yellow block'
top of 'table'
'yellow block' has moved
'yellow block' is not on top of 'red block'
'yellow block is not on top of cyaan block
State 4:
'red block is on top of 'table'
'red block has moved
red block! is not top of 'cyan block'
'red block' is the right of orange cylinder'
"orange cylinder' is the left 'red block
State 5:
'cyan block' is not top of 'table' 'cyan block' has moved 'cyan block' is on top of 'blue block' <end_of of ofnd_of_exampp www.oofeaaaaeeuurr.uuaaa query? [[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status the objects ('orange cylinder is on top of 'table'),
I will summarize the state trajectory into lov-level actions.
The lovelevel actions stop summarizing after this.
the most compact form, I should
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :]] yes
[[Summarized trajectory:ll
In [Scenario 1], at state 2, robot picked up and placed the 'orange cylinder' on top of 'table'.
At state 3, the robot picked up and placed 'yellow block on top of 'table'.
At state 4, the robot picked up and placed 'red block! top table',
At state 5, the robot picked up and placed 'cyan block' on top of 'blue block'.
[[end of response]]
[[Input Trajectory: :]] (Scenario 1]
Stack all blocks, where enfocing the maximum stack height if there is requirement.
State 2:
'orange block has moved 'orange block' not behind 'yan cylinder' 'yan cylinder'
front of 'orange block'
State 3:
'blue block' is not on top of 'table'
'blue block' has moved
'blue block' is on top of 'raanee block'
State 4:
'brown block' 1 is not top of 'table' 'brown block' has moved 'brown block' is on top of 'orange block
'bronn block' is on top of 'blue block'
59

---

State 6:
'yellow block' has moved 'yellow block behind 'cyan cylinder' "cyan cylinder' is in front of yeellow block
State 6:
"gray block' is 'gray block' has moved 'gray block' is on top of 'yellow block' Dond_of_example_usee [[Reasoning:]]
top 'table'
The input trajectory contains state predicates because trajectory talks about the status of the objects ('orange block has moved).
I will summarize the state trajectory into low-level actions. The low-level actions stop summarizing after this.
the most compact form, I should
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :]] yes
[[Summarized trajectory:]]
In [Scenario 1l, at state 2, the robot moved 'orange block' At state 3, the robot picked up and placed 'blue block on top of 'orange block'.
At state 4, the robot picked up and placed 'brown block top of 'blue block'.
At state 5, the robot moved 'yellow block', At state 6, the robot picked up and placed 'gray block' on top of 'yellow block'.
[[end of response]]
[[Input Trajectory:]] :]]
[Scenario 2] Stack all blocks into stack, while enforcing the order between blocks if there a requirement.
State 2:
bblue block has moved
State 3:
'red block' is not on top of 'table'
'red block' has moved
'red block is top of 'blue block'
'red block' is to the right 'red cylinder'
'red cylinder' is not to the left of 'red block
State 4:
'yellow block'
top of 'table'
'yellow block' has moved
'yellow block' is on top of 'red block' 'yellow block is on top of 'blue block'
State 5:
'cyan block' is not on top 'table'
'cyan block' has moved
'cyan block' is top of 'yellow block'
'cyan block' is top of 'red block'
'cyan block' is on top of 'blue block
oxanpl.fexa [[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects ('pink block has moved).
I will summarize the state trajectory into lov-level actions.
60

---

The low-level actions are the most compact form, I should stop summarizing after this.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :]] yes
[[Summarized trajectory:]]
In [Scenario 2], at state 2, the robot moved 'blue block'. At state 3, the robot picked up and placed 'red block on top of 'blu block'.
At state 4, the robot picked up and placed 'yellow block top of 'red block',
At state 5, the robot picked up and placed 'cyan block' on top 'yellow block'.
[[end of response]]
[[Input Trajectory :]]
[Scenario 1] Stack all objects into two stacks, and objects may need to be categorized in the stacks.
State 2:
'broun block is to the left of 'brown cylinder
'brown cylinder' has moved 'brown cylinder' is to the right of 'brown block' 'brown cylinder' is to the left of 'blue cylinder'
'blue cylinder! to the right 'brown cylinder'
State 3:
orannee block' is not top of 'table' oranng block' 1 has moved
"orange block
on top of 'brown cylinder'
State 4:
'yellow block' 1 is not top of 'table' 'yellow block has moved
'yellow block is on top of orange block' 'yellow block is on top of 'brown cylinder'
State 5:
'pink cylinder
not on top of 'table'
'pink cylinder' has moved 'pink cylinder' is on top of 'yellow block' 'pink cylinder! is top of orange block' pink cylinder'
on top 'brown cylinder!
State 6:
'blue block' has moved
State 7:
'brown cylinder' is not to the left of 'blue cylinder' bbuu cylinder' is not on top of 'table'
'blu cylinder' has moved
bblu cylinder' on top 'blue black'
bblue cylinder! not to the right of 'brown cylinder'
State 8:
top 'table' "orange cylinder' has moved orange cylinder' is on top of 'blue block' 'orange cylinder' is on top of 'blue cylinder'
"orange cylinder' is
State 9:
'brown block is not on top of 'table'
'brown block' has moved
brown block' is on top of blue block'
61

---

brown block' is not to the left of 'brown cylinder' brown block' is top of 'orange cylinder' brown block is on top of 'blue cylinder' bbron cylinder' is not to the right of 'broun block' oand_of_example_uee [[Reasoning:]]
The imput trajectory contains trajectory. talks about the status of the objects ('brown block' to the left of 'brown cylinder
predicates because the
I will summarize the state trajectory into 1ow-level actions. The low-level actions stop summarizing after this.
the most compact form, I should
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :1] yes
[[Summarized trajectory:ll
In [Scenario 1], at state 2, the robot moved 'brown cylinder'. At state 3, the robot picked and placed 'orange block on top of 'brown cylinder'.
At state 4, the robot picked up and placed 'yellow block
top of 'orange block',
At state 5, the robot picked up and placed 'pink cylinder' on top of 'yellow block'.
At state 6, the robot moved 'blue block'.
At state 7. the robot picked up and placed 'blue cylinder' top of 'blue block'.
At state 8, the robot picked up and placed oranne cylinder' on top of 'blue cylinder'.
At state 9, the robot picked up and placed 'bronn block top of 'orange cylinder
[[end of response]]
Summarized Demonstrations -> Task Specification
1ammary_2_spee
main:
You are helpful assistant who analyzes the trajectories and summarizes then into a concise p task specification
of messaem
You are given (1) level goal and (2) one or more trajectories where each one represents a different scenario. Your goal is to summarize these trajectories into a compact task specification, written in pseudocode style.
You must respond using the following format (you must generate [[ end of response]] at the very end) [[Reagoning:]]
You should first list out the order of the high-level aubtask trajectories all scenarios. Then, you should consider if certain sections can be represented by a loop. Two scenarios having exactly the same list does not mean that there is a loop. A loop only exists among subtasks in one individual scenario.
the subtasks are repeated in the scenario and
Overall, your task specification should work You should make sure that the task specification matches the subtasks ordering across all scenarios. You should also make sure that the task specification uses loops when there is any repetition.
all scenarios.
[[Task Specification:]]
You should first state the high-level goal. Then, you should say Specifically:" before outputting the pseudocode style task specification.
62

---

[[end of responsel]
You must follow these rules when you are writing the task specifications.
You must write the task specifications in pseudocode style. You should not write the task specification list. You cannot number any line. examples:
[[High-Level Goal []]
Place the green cylinder [[Trajectories:]]
to the yellow cylinder.
In [Scenario 1l, at state 2, the robot picked up and placed green cylinder to the left of 'yellow cylinder'. exan_oofexample.uueerr [[Reasoning:]]
With only scenario, we assume that the 'green cylinder' must always be to the left of 'yellow cylinder'.
[[Task Specification:l] Place the green cylinder to the left of the yellow cylinder. [[end of response]]
[[High-Level foal:]]
Place the purple cylinder next to the orange block. [[Trajectories:]]
In [Scenario 1], at state 2, the robot picked up and placed the 'purple cylinder' behind 'orange block', In [Scenario 2], at state 2, the robot picked up and placed the 'purple cylinder' in front of 'orange block!. exxeooffxxmmleeuueee [[Reasoning:]]
Because two scenarios have two different relative positions between 'purple cylinder and 'orange block', assume that the purple cylinder'
be anyuhere next to orange block'.
[[Task Specification:]]
Place the purple cylinder at anywhere next to the orange block. [[end of response]]
[[vigh-Level Goal:]l
Place the blue block the corner. [[Trajectories::]]
In [Scenario i], at state 2, the robot picked up and placed the 'blue block' 'bottom right corner
In [Scenario 2], at state 2, robot picked up and placed the 'blue block' at 'top right corner
[[Reasoning:]]
Because scenarios have top right corner we assume that the 'blue block' can be at any randon corner.
('bottom right corner
[[Task Specification:]]
Place the blue block any random corner.
[[end of response]]
[[High-Leve] Goal:]]
Place the cyan block the blue block. [[Trajectories:]]
In [Scenario 1], at state 2, the robot picked up and placed the 'orange cylinder' on top of 'table'. At state 3, the robot picked and placed 'yellow block on top of 'table'.
At state 4, the robot picked up and placed 'red block' top 'table'
At state 5, the robot picked and placed 'cyan block on top of 'blue block'.
63

---

<end_of lerddoof_exmmle_usrreooo [[Reasoning:]]]
Although the goal is to place "cyan block" on the "blue block", the trajectories show that it needs to move other blocks ('orange cylinder's 'yellow block', 'red block') before finally placing top of 'blue block'. [[Task Specification:]]
eyan block'
1. place the orange cylinder on the table
2. place the yellov block on the table
3. place the red block on the table
4. place the cyan block the blue block [[end of response]]
[[High-Level Goal :]l
Stack all blocks, where enforcing the maximum stack height if there requirement.
[[Trajectories:]]]
In [Scenario 1l, at state 2, the robot moved 'orange block'. At state 3, the robot picked up and placed 'blue block' on top of 'oaanee block',
At state 4, the robot picked up and placed 'brown block' on top of 'blue block'.
At state 5, the robot moved 'yellow block'.
At state 6, the robot picked up and placed 'gray block' on top of 'yellow block '
<end_of_ arndooff.eampll..ssr.uuerr
[[Reasoning:]]
Because the task is "Stack all blocks, where enfocing the maximum stack height.... we focus how high the stacks are.
Based on States 2-4, the first stack, the blocks from bottom to top is ['orange block', 'blue block', 'brown block '1. This is 3 blocks high.
Based States 5-6, in the second stack, the blocks from bottom top ['yellow block', 'gray block 'J. This 2 blocks high.
Because there are 2 stacks and the tallest stack is 3 block high assume that each stack needs to be at most blocks high. [[Task Specification:1] Stack all blocks. However, the maximum height of stack is [[end of response]]
[[High Level []al:]] Stack all blocks into one stack, while enforcing order between blocks if there is a requirement. [[Trajectories:]]
* In [Scenario 1], at state 2, the robot moved 'red block', At state 3, the robot picked up and placed 'yellow block top of 'red block'.
At state 4, the robot picked up and placed 'cyan block' on top of yellow block',
At state 5, the robot picked up and placed 'blue block' top cyan block'.
In [Scenario 2], at state 2, the robot moved 'blue block', At state 3, the robot picked up and placed 'red block on top of bbue block',
At state 4, the robot picked up and placed 'yellow block! top of 'red block'
At state 5, the robot picked up and placed 'cyan block' on top of 'yellow block'.
<end_of exxplee_usruuue [[Reasoning:]]
Because the task is "Stack all blocks, while enforcing the order we focus on vhether there is any ordering requirement
64

---

In [Scenario 1], based on States 2-5, in the first stack, the blocks from bottom top is ['red block', 'yellow block', 'cyan block' 'blue block 'J.
In (Scenario 2], based on States 2-5, in the first stack, the blocks from bottom to top is ('blue block', 'red block', 'yellow block', 'cyan block '].
Between the order of blocks in these two scenarios, the ordering that stays the same is ['red block', 'yellow block', 'cyan block
[[Task Specification:ll
Stack all blocks into one stack. Also make sure that the stacking enforces the bottom to top order between these objects: ['red block', 'yellow block', 'cyan block']. [[end of response]]
[[High-Level Goal:]]
Stack all objects into two stacks, and objects may need to be categorized in the stacks.
[[Trajectories:]]
In [Scenario 1], at state 2, the robot moved 'brown cylinder'. At state 3, the robot picked up and placed 'orange block on top of 'brown cylinder'.
At state 4, the robot picked up and placed 'yellow block top of 'orange block',
At state 5, the robot picked up and placed 'pink cylinder on top of 'yellow block'.
At state 6, the robot moved 'blue block'.
At state 7, the robot picked up and placed 'buue cylinder' top of 'blue block'.
At state 8, the robot picked up and placed 'orange cylinder' on top of 'blue cylinder'.
At state 9, the robot picked up and placed 'bronn block top of 'orange cylinder'.
<end_of_ <end_of..xxmllouusrr..eeee" [[Reasoning:]]
Because the task is "Stack all objects into two stacks, and objects may need to be categorized in the stacks", we focus whether the objects
stacked by type.
Based on States 2-5, in the first stack, the blocks from botton to top is ['brown cylinder', 'oaange block', 'yellow block', 'pink cylinder '].
Based on States 6-9, the first stack, the blocks from bottom to top is ['blue block', 'blue cylinder', 'orange cylinder', brown block']
Because each stack has both blocks and cylinders, doesn 't matter whether the objects are categorized.
that
[[Task Specification:ll Stack all objects into two stacks. It doesn 't matter whether the objects are categorized.
[[end of responsel]
18.1.2 Recursive Expansion
Task Specification High-Level Code
_pec_2_hist h
main: I You
Python code generator for robotica. The will first provide the imported Python modules. Then, for each code they want you to generate, they provide the requirements and pseudocode in Python docstrings. Band_of__yattem_messggg
65

---

You need to write robot control scripts in Python code. The Python code should be general and applicable to different environments.
Below are the imported Python libraries and functions that you can use. You CANNOT import new libraries.
Python kitchen robot control script import numpy as np
from robot_utils import put_firatton_second, wtt.__wwtthtuthhhhhhhhoo from env_utils import get_obj_names, L..aal__oj_.arrs_hhttaattt
stack
determineffill_ssccciie
parse_position
= ['top left corner', 'top right corner', 'bottom left corner's 'bottom right corner
ALL_EDGES_LIST LIST ['top edge', 'bottom edge', 'left edge', 'right edge']
ALL_ _AL_POSITION. DDSITION_RELATTON. front ']
= ['left of', 'right of', 'behind', 'in
Below shows the docstrings for these imported library functions that you must follow. You CANNOT add additional parameters to these functions.
robot_utils Specifications:
put_firrtt.on_eccon (argi, arg2)
You must not write things like:
pccccon r"red block", "left of yellow block") put_first.onn.ecomm ("red block", "top left corner") put_ffirst.on_eceee
block", "top edge")
You can write something like:
ant_firstt_on_econd rerd block", "yellow block") ser_ffrrst_on_secom r"red block", "table")
Pick up an object (arg1) and put it at arg2. arg2 is an object, argi will be on top of arg2. If arg2 is 'table', , argi will be somewhere randon on the table. If arg2 is list, argl will be placed at location [x, yl.
Parameters:
argi (str); string that defines the object by its color and type (which is either "block" or "cylinder"). For example, "red block "orange cylinder".
arg2 (list or str): If it's list, floats, and it represents the [x, y] position to place argi. If it 's a string, it can either be "table" or string that defines the object by its color and type. arg? must not be free-from string that representa
needs be list
description of position. For example, it cannot relative position (e.g. "left yellow block"), or corner name (e.g "top left corner"), or edge name (e.g. "top edge ").
Stack the list of objects_to_ to_ into one stack without considering height limit. The first object (which is the object at the bottom of the stack) will also get moved and placed somewhere on the table.
Parameters:
dojects_to_stade (list): list of strings, where each defines the object by its color and type.
een
stack with www.:httthhhhhtittteeeeeeeeeee tonjeeitggtt..itttoooootttt.... stack,, height_limit)
66

---

Stack the list of objacts_to_stack into potentially multiple stacks, and each stack has maximum height based height_limit, The first object (which
object at the bottom the stack) will also get moved and placed somevhere on the table.
Parameters:
objecta_ _tt_a_tttccckkk (list): list strings, where each string defines the object by its color and type. height_limit (int): an integer representing the maximum height for each stack.
env_utils Specifications: nem_os)_names(
Return:
list of objects in the environment
get_all obj names_ thates_that_eee ttattthaaa______eee_eeeeeeeeeeee mattt_attc_tppp ( (type_name, objects_list
Return:
list of objects in environment that match the type_name
determine fimal_lnaaaaeeeee www.aaaatacinggoooooooo order tenjects_tt_ennooreeee objects_ order)
Return:
a sorted list of objects to stack. The first object in the list would be at the bottom of the stack.
parse_position (description)
You must use parse_postion for
relative position (e.g. "left of yellow block") corner position (e.g. "top left corner")
edge position (e.g. "top edge")
Return:
a list [x, y] that represents the position described by the description.
***
You must follow these rules when you are generating Python code. You MUST ONLY use Python library functions imported above. You MUST follow the docstrings and specification for these functions. You must follow the instructions provided by the user. You CANNOT add additional steps, conditionals, or loops that are not in the instruction. examples:
objects=[ orange (orange block', 'yellow cylinder', 'green cylinder ']
unl
Place the green cylinder to the left of the yellow cylinder. ann
<end_ wxw._eeeemleeeer..c.. querry
must use parse position to get relative position location_pon parse_position ('left of the yellow cylinder') put_first_ onttt_on_eeeee on_rrssdd___eecddeee green cylinder', location.pos)
objects=[ orange block', 'purple cylinder
und
Place the purple cylinder at anyvhere next to the orange block.
67

---

<end_ oent_o...rammlee.err..ss
position_relation up. random. choice BALLCCLALLLLSSSTOLAAAAATTTILLLLL # must use parse position to get relative position
location_pos - parse_position (or.pppsttittttppppptoooooooooo relatio the orange block')
on ('prrple purple cylinder', location.pos)
objects=[ blue block', 'brown block', 'orange cylinder', 'pink cylinder', 'purple cylinder
Place the blue block any random corner. una
<end of www.pleeeeeeeeeeeeeeeee
corner name np. random choice (ALL_CORNERS_LLA _LSTT)
must use parse position to get corner position corner_poo parse_ porse.positionccorre (corner_name) he..first_onsseconde ('blue block', corner_pos)
objects=['res red block', 'blue block', 'cyan block', 'yellow block'
orange cylinder
orange cylinder on table 2. place the yellow block on the table
1. place
3. place the red block on the table place the cyan block on the blue block
4.
<end_ of wtw.ooffeaalleuueeeerroo
items_ to wo.placce_... wttplllllfrssstiieeeppp firsttii__orde order ['orange cylinder', 'yellow block', 'red block']
for item itms.to.place.iitt...
put_first_on__ee put_first__oossccnnieeeee (icedd "table")
ont_firstt.on_eccom (' cyan block', 'blue block')
objects-[ orange block', 'blue block', 'brovn block', 'yellow block', gray block', 'pink block', 'cyan cylinder', 'blue cylinder unn
Stack all blocks. Hovever, the maximum height of a stack is 3.
<end_ of of
namek_
names__amm_____ thateesstttttttttttttt naae blokk pepecta_littgge_oojj.ooo ())
bhack_vith_heighhee limit limit htt.res_tt_ttcccclooooooaaa height_limitss
objects=[ yeellow block', 'cyan block', 'red block', 'blue block' 'green cylinder', 'red cylinder', 'blue cylinder
68

---

Stack all blocks into stack. Also make sure that the stacking enforces the bottom top order between these objects: l'red block', 'yellow block', 'cyan block 'J. unn
Based on the instruction, first define the blocks that have s strict ordering
l'red block',
yellow block', 'cyan block'] Then, find the rest the blocks that don't 't have strict order block_names
wames__tha__match..oo (type tttppeeemm=="llloo for block block_names:
block, www.naae__itthhte
[]
if block not
Block_names strchsssrrccccrrr_ffeeee
to_tppo
block_names baock_nanes_kithutttoomm appeedd (block)
Call helper function to determine the final stacking order before stacking the blocks
frddr__foombbotom. objects_to_ to wwwtttooeeeeerreeeeeeeoeeeeeeeee order=
determine_finallstttc
block Cot.ckanaa__iittt.trritt.to.oor wrt.:/rrrrrrrr.ooooooooooooo.
objects_ bojoct..wthoott..rderbbboo...ae hawes_sitthouutooreert
stack stack ordor hro.eer_froobbbtttotttootoooo
objects= ('bects=''buue blue block', 'yellow block', 'brown block', 'orange block', 'pink cylinder', 'brown cylinder', 'orange cylinder', blue cylinder
Stack all objects into two stacks (vhere each stack has maximun height of 4). It doesn't matter whether the objects are categorized.
<end ofnd_of_exaaee paadoof.erample_usrr.uue
object_names = get_obj_names ()
split the objects into 2 stacks.
stack_1 object_names [:4]
stack_2 object_ [4:]
stack srrthouthheighh.. limight (objects to stasskkktttcck_1 stack
18.2
Robotouille Task Prompts
18.2.1 Recursive Summarization
main:
You are helpful summarizer that recursively summarizes trajectory into a more and more compacted form.
You are given trajectory. Your goal is to summarize the trajectory into
compacted form and then determine whether the state trajectory is sufficiently summarized.
You must respond using the following form (you must generate [[end of responsel] at the very end):
69

---

[[Reasoning:]]
You should first identify what type of trajectory this is Then, you should determine what type you will summarize the trajectory into. Finally, you should determine whether the new type of trajectory is sufficiently summarized or not. [[Is the trajectory sufficiently summarized? (yes/no) :]] You should only reapond using either "yes" "no", and nothing else.
[[Summarized trajectory:1]
You should actually summarize the input trajectory and output it here.
[[end responsel]
In summary, you must follow these rules when are summarizing a trajectory.
Rules:
You must slowly summarize the trajectory from one type
another folloving this order: state trajectory > lov-level action trajectory --> high-lever subtask trajectory.
You cannot skip type (e.g. you cannot directly summarize low level action trajectory into high-level subtask trajectory).
low-level action trajectory is represented as an unordered list. Each line in the unordered list should only contain one low- level action.
The low-level actions must location1 to location2", "pick up itemi", "place down itemi at locationi ", "stack itemi on top of item2", "unstack itemi from item 2", "cook itemi", "cut iteml". You should never define new lov-level actions.
of the following: "move from
represented as unordered list. Each line in the unordered list should only contain one high -level subtask. This high-level subtask should refer to one continuous section of the states. For example, you cannot say "at states 1-5, and states 10-15, the robot did " There can only one interval of states.
high-level subtask trajectory
The high- subtask must be one of the folloving: "coo [ITEM ]" "cut [ITEM]", "stack [ITEM] on top of [ITEM]", and "unstack [ ITEM] from [ITEM]". [ITEX] must be one of the following: "patty", "lettuce", "top bun", "bottom bun", "cheese", "tomato". For the high-levei subtask, you cannot define any other subtasks that are not: cook, cut, stack, or unstack. You must use choose from the list above. examples:
[[Input Trajectory:]] (Scenario 2] Cook patty and cut lettuce.
State 2:
'roboti' is at 'table6' 'roboti' is not at 'table7'
State 3:
'lettuce6' is not at 'table6' 'robot1' is holding 'lettuce6'
State 4:
'roboti' at cattiing_bardd 'roboti' is not at 'table"'
State 5:
'lettuce6' is boart 'robot1' not holding 'lettuce6'
State 6:
70

---

State 7:
State 8:
'lettuce6' is
State 9:
'roboti' is not at 'cutting_board6 'roboti' is at 'table7'
State 10:
'patty6' is not at 'table7' 'robot1' is holding 'patty6'
State 11:
'roboti' is at 'stove6' 'roboti' is not at 'talle'
State 12:
'patty6" at 'stove6'
'roboti' is not holding 'patty6'
State 13:
State 14:
State 15:
State 16: 'patty6' cooked <end_ of lond_of_exanee wtw.oooccammle.uuuuuuuyy... [[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the table6') and the status the objects ('1ettuce6 table6').
of the robot ('roboti' 1 is not
I will summarize the state trajectory into low-level actions. Low level actions are not compacted enough yet because level actions can still combined into high-level subtasks. The new trajectory will NOT be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no): :]] no [[Summarized trajectory:
In [Scenario 2], at state 2, the robot moved from 'table7' ttable 6'.
At state 3, the robot picked up 'lettuce6', At state 4, the robot moved from 'table6' At state 5, the robot placed 'lettuce6' location secting_boardd
coardng
At state 6-8, the robot had cut 'lettuce6', At state 9, the robot moved from 'cutting.board6" to 'table7',
At state 10, the robot picked up 'patty6'.
At state 11, the robot moved from 'table7' to 'stove6'.
At state 12, the robot placed 'patty6' at 'stove6'.
At state 13-16, the robot had cooked 'patty6" [[end of response]]
[[Input Trajectory: :11
In [Scenario 2]. at state 2, the robot moved from 'table5' to 'table9'
71

---

At state 3, the robot picked up 'patty9',
At state 4, the robot moved from 'table9' 'stove5',
At state 5, the robot placed 'patty9' on 'stoveb'.
At state 6-9, the robot had cooked 'pattys
At state 10, the robot moved from 'stove5' to 'table5',
At state 11, the robot picked up 'patty3',
At state 12, the robot moved from 'table5' 'stoves',
At state 13, the robot placed 'patty3' on 'stove8' At state 14-17, the robot had cooked 'patty3'.
pmnd_of_examlee_ssrrr. [[Reasoning:]]
The input trajectory contains lov-level actions because the trajectory mentions "moved", "picked up", etc.
I will summarize the low-level action trajectory into high-lever subtasks.
High-level subtask trajectory is the compacted form that cannot summarized anymore.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :]] yes
[[Summarized trajectory:]]
In [Scenario 2], at state 2-9, the subtask "cook patty". This aubtask contains: 1. moving to pick 'pattyg' (state 2-3) 2. moving to place 'patty9'
'stove5' (state 4-5) 3. cooking patty9 until it is cooked (state 6-9)
At state 10-17, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty3' (state 10-11) 2. moving to place 'patty3' on 'stove8' (state 12-13) 3. cooking 'patty3' until it is cooked (state 14-17) [[end of response]]
[[Input Trajectory:1]
[Scenario 1]
Cut lettuce before stacking it top of bottom bun. Then stack top on top the lettuce.
State 2:
'lettucel' is not at 'table6' rrooot1' 1 holding 'lettucel'
State 3:
'roboti' is not at 'table6' 'robot1' is at 'oarting.board!'
State 4:
'roboti' is not holding 'lettucel" 'lettucel" is at boardn
State 5:
State 6:
State 7:
'lettuce1 is cut
State 8:
'lettucel" is not at cuttinggbbardd 'robot1' is holding 'lettucel'
State 9:
'roboti' not at bcutting_ 'roboti' is at 'table2'
72

---

State 10:
'lettucel' is 'table2'
'lettuce1 is on top 'bottom_buni buni 'robot1 is not holding 'lettucel'
State 11:
'roboti' is not 'table2' 'roboti' at 'table5'
State 12:
'top_bun2' is at 'table'' 'robot1' holding 'top_bun2'
State 13:
'roboti' is not 'table5'
'roboti' is at 'table2
State 14:
'top_bun2' is at 'table2'
'top_bun2' is top of 'lettucel' 'roboti' not holding 'top_bun2' fend_of_erample_uerrr [[Reasoning:]]
The imput trajectory contains state predicates because the trajectory talks about the status of the robot ('robot1' is holding 'lettucel !) and the status the objects ('lettucel is not at 'table6")
I will summarize the state trajectory into lov-level actions. Low-level actions are not compacted enough yet because low-level actions can still be combined into high-level subtasks. The new trajectory will NOT be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no): :]] no
[[Summarized ]rajectory:]]
In [Scenario 1], at state 2, the robot picked up 'lettucel At state 3, the robot moved from 'table6' to battinggboardd At state 4, the robot placed 'lettucel" at location "octinggbbaadd"
At state 5-7, the robot had At state 8, the robot picked up 'lettucel At state 9, the robot moved from cutting_boardi: to 'table2'.
'lettucel'.
At state 10, the robot placed 'lettucei" on top of bottom_bunl
location 'table2'.
robot moved from 'table2' 'table5', At state 12, the robot picked up 'top_bun2', At state 13, the robot moved from 'table5' to 'table2'. At state 14, the robot placed 'top_bun2' top of 'lettucel' location 'table2'. [[end of response]]
state 11,
[[Input Trajectory: :]]
In [Scenario 1], at state 2, the robot picked up 'lettucel ' At state 3, the robot moved from 'table6' At state 4, the robot placed 'lettucel' at location socting.bbard".
bartinng.bardd.
* At state 5-7, the robot had cut 'lettucel', At state 8, the robot picked up 'lettucel At state 9, the robot moved from 'cutting_boardl to 'table2'
At state 10, the robot placed 'lettucel" on top of bottom_bunl at location 'table2'
At state 11, the robot moved from 'table2' 'table5', At state 12, the robot picked up 'top_bun2 At state 13, the robot moved from 'table5' to 'table2'. At state 14, the robot placed 'un2_bun2' on top of 'lettucel' at location '
73

---

<end_ lttp_oof_exampe_userroooo [[Reasoning:]]
The input trajectory contains lov-level actions because the trajectory mentions "picked up", "moved", etc.
I will summarize the low-level action trajectory into high- subtasks.
High-lever subtask trajectory is the compacted form that cannot summarized anymore.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :]] yes
[[Sumaarized trajectory:ll
In [Scenario 1], at state 2-7, the subtask is "cut lettuce". This subtask contains: 1. pick up 'lettucel' (state 2) 2. moving to place 'lettucel' on conttingbboarde lettucei' until it At state 8-10, the subtask is "stack lettuce on top botton 'lettucel' (state 8) 2.
(state 3-4) 3. cutting
(state 5-7)
bun". This subtask contains: 1. picking moving to stack 'lettucel' on 'bottom_bunl' (state 9-10) At state 11-14, the subtask is "stack top bun top of lettuce", This subtask contains: 1. moving to pick up 'top_bun2' state 11-12) 2. moving to stack 'top_bun2 on 'lettucel' (state 13-14)
[[end of response]]
Summarized Demonstrations -> Task Specification
cummary-2_spon
main:
You are helpful assistant analyzes the high-level level subtask trajectories and summarizes them into a concise paeudocode- -tyyee task apecification.
<end_ send_off.ysstemmmsssee
You given (1) high- goal and (2) subtask trajectories where each one represents different scenario. Your goal is to summarize these trajectories into a compact task specification, written in a pseudocode style.
high-level
respond using following format (you must generate [[ end of responsell [[Reasoning:]]
the very end)
You should first list out the order of the high- subtask trajectories in all scenarios. Then, you should consider if certain sections
the scenario and can be represented by a loop. Two scenarios having exactly the same list does not mean that there is a loop. A loop only erists among subtasks in one individual scenario.
the subtasks are repeated
Overall, your task specification should work for all scenarios. You should make sure that the task specification matches subtasks ordering across all scenarios. You should also make sure that the task specification uses loops when there is any repetition.
[[Task Specification::]
You should first state the high-level goal. Then, you should say Specifically." before outputting the pseudocode style task specification.
[[end response]]
You must follow these rules when you are writing the task specifications. Rules:
74

---

You must write the task specifications in pseudocode style. You should not write the task specification number any line.
list. You cannot
When checking for loops, you cannot compare the subtasks across multiple scenarios. Even if two scenarios have the exact same list of subtasks, there is NOT any loop. Loops only exist within the list of subtaske for individual scenario. Do not consider loops across multiple scenarios. examples:
[[High-Leve) Goal:]]
Cook patty and cut lettuce. [[Trajectories:]]
In [Scenario 2], at state 2-8, the subtask is "cut lettuce".
This subtask contains: 1. moving to pick up 'lettuce4' (state 2-3) 2. moving place 'lettuce4' cutting 'lettuce4' until it is cut (state 6-8)
"cutting_board3" (state 4-5) 3.
At state 9-16, the subtask is "cook patty". This subtask contains: 1. moving to picking
'patty2' (state 9-10) 2. moving
to place 'patty2'
location 'stove8' (state 11-12) 3. cooking
patty2' until it is cooked (state 13-16)
In [Scenario 2], at state 2-8, the subtask "cut lettuce". This aubtask contains: 1. moving to pick 2. moving to place 'lettuce6'
'lettuce6' (state 2-3) 'cutting.board6" (state 4-5) 3. cutting 'lettuce6" until it is cut (state 6-8)
At state 9-16, the subtask is "cook patty". This subtask contains: 1. moving to picking up 'patty6' (state 9-10) 2. moving
to place 'patty6' at location 'stove6 (state 11-12) 3. cooking patty6' until it is cooked (state 13-16) exapooffeeamleeuuuer.oo [[Reasoning:]]
In [Scenario 'l's unique kitchen environment, the subtasks were executed in this order: ['cut lettuce', 'cook patty ']. In [Scenario 2] unique kitchen environment, the subtasks were executed
this order: ['out lettuce', 'cook patty ']. Both scenarios are executing the subtasks in the same order. There is no repetition or loop in the subtask ordering for any individual scenario.
[[Task Specification :]]
Cook patty and cut lettuce.
Specifically:
Get list of all the lettuces in the kitchen. Get list
the patties in kitchen.
Decide a lettuce to use. Get list of all the available cutting boards in the kitchen. Decide cutting board to go to. Cut that lettuce at that cutting board.
Decide patty to use. Get list of all the available stoves in the kitchen. Decide
go to. Cook that patty at that stove.
[[High-Level Goal:]] Cook two patties.
[[Trajectories:]]
In [Scenario 1], at state 2-9, the subtask is "cook patty", This subtask contains: 1. moving to pick 'patty4' (state 2-3) 2. moving to place 'patty4'
'stove10' (state 4-5) 3. cooking patty4' until it is cooked (state 6-9)
At state 10-17, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty6' (state 10-11) 2. moving to
75

---

place 'patty6' 'stovell' (state 12-13) 3. cooking 'patty6' until it is cooked (state 14-17)
In [Scenario 2], at state 2-9, the subtask is "cook patty", This subtask contains: 1. moving to pick up 'pattyg' (state 2-3) 2. moving to place 'patty9' on 'stove5' (state 4-5) 3. cooking pattyg until it is cooked (state 6-9)
At state 10-17, the subtask is "cook patty". This aubtask contains: 1. moving to pick up 'patty3' (state 10-11) 2. moving place 'patty3' on 'stove8' (state 12-13) 3. cooking 'patty3' until it is cooked (state 14-17)
<end_of <end_of_example.. wwwppeeeeeeee..eeurr...... [[Reasoning:]]
In [Scenario 1l unique kitchen environment, the subtasks were executed in this order: ['cook patty', ccook patty 'I. In [Scenario 2s's unique kitchen environment, the subtasks executed Both scenarios are executing the subtasks In both scenarios, we see that the subset ['cook patty '] got repeated 2 times consecutively, so we can use a for- in our specification.
this order: ['cook patty', 'cook patty the same order
[[Task Specification:)] Cook two patties.
Specifically:
Get list of all the patties in the kitchen.
For two patties, do: Decide a patty to use.
Get list of all the available stoves in the kitchen. Decide stove go to. Cook that patty at that stove.
[[High-Level Goal:]]
Cut lettuce before stacking it top of bottom bun. Then stack a top on top the lettuce. [[Trafectories:]]
In [Scenario 1], at state 2-7, the subtask is "cut lettuce". This subtask contains: 1. pick up 'lettucel' (state 2) 2. moving place 'lettucel" lettucei' until it cut (state 5-7)
outting.boardy (state 3-4) cutting
At state 8-10, the subtask is "stack lettuce on top of botton bun", This subtask contains: 1. picking up 'lettucel' (state 8) 2. 'bottom_bunl' (state 9-10)
moving to stack 'lettucel'
At state 11-14, the subtask is "stack top on top lettuce", This subtask contains: 1. moving to pick up bun2' state 11-12) 2. moving to stack 'top_bun2 on 'lettucel' (state 13-14)
In [Scenario 2], state 2-7, the subtask "cut lettuce". This subtask contains: to place 'lettuce4" on 'cutting_board2" lettuce4 until it is cut (state 5-7)
pick up 'lettuce4 (state 2) 2. moving
(state 3-4) 3. cutting
At state 8-10, the subtask is "stack lettuce top of botton picking up 'lettuce4' (state 8) 2. (state 9-10)
bun". This subtask contains: moving to stack 'lettuced' on bottom_bung At state 11-14, the subtask is "place top bun on top of lettuce", This subtask contains: 1. moving to pick up "uni_buni' state 11-12) 2. moving place 'top_buni" top of 'lettuce4" state 13-14)
<end of example_
[[Reasoning:]]
In [Scenario is unique kitchen environment, the subtasks executed this order: ['cut lettuce', 'stack lettuce top bottom bun', 'stack top bun on top of lettuce
76

---

In [Scenario 2] 's unique kitchen environment, the subtasks executed in this order: ['cut lettuce', 'stack lettuce top bottom bun', place top bun on top of lettuce 'J.
'stack lettuce on top of bottom bun' and 'place top bun on top of lettuce are essentially the same subtask.
Both scenarios are executing the subtasks in the same order. There is repetition individual scenario.
loop the subtask ordering for any
[[Task Specification:]]
Cut lettuce before stacking it top of bottom bun. Then stack top bun top of the lettuce.
Specifically:
Get list of all the lettuces in the kitchen. Get list of all the bottom buns in the kitchen. Get list of all the top buns in the kitchen.
Decide a lettuce to use. Get list of all the available cutting boards in the kitchen. Decide cutting board to go to. Cut that lettuce at that cutting board.
Decide a bottom bun to use.
Stack the lettuce on top of the bottom bun.
Decide a top bun to use.
Stack the top bun on top of the lettuce.
18.2.2 Recursive Expansion
spec_2 _ highheeeleeecde:
main:
You are a Python code generator for robotics. The users will first provide the imported Python modules. Then, for each code they want you to generate, they provide the requirements and pseudocode in Python docstrings
You need to write kitchen robot control scripts in Python code. The Python code should be general and applicable to different kitchen environments.
Below are the imported Python libraries functions that you can use. You CANNOT import new libraries.
Python kitchen robot control script from env_utils import getaall_occtionnnam.ss taaties_thhtmmttctttppee is_cut, is_cooked
Below shows the docstrings for these imported library functions that you follow. You CANNOT add additional parameters these functions.
env_utils Specifications:
net_all.obj.nnmos_tte www.aaassssaatteeeeeeeeeeeaeeeee ctt.eessshhttttc.ttppppoeeoo tyb)_type)
Parameters:
oyj_type (str): : A string that has to be one of 'patty', 'lettuce', 'boton bun't 'topp bun'.
Return:
(list) list of string of objects that match the obj_type
***
get_alll
taatsssttattmmtthh type (location_type) t
Parameters:
77

---

location_type (str): A string that has to be of 'table', "cutting_boare board', 'stove'.
Return:
(list) a list of string of locations that match the location_type
[e_cu((bb)
Parameters:
obj (str); A string that has the type of cuttable object (
must be 'lettuce and id. For example: 'lettuce3', 'lettucel'
Return:
(boolean) true if obj cut
(obj))
Parameters:
obj (str): A string that has the type of cookable object must be 'patty') and an id. For example: 'patty1', 'patty2'
Return:
(boolean) true if obj cooked
You must follow these rules when you generating Python code. You can use Python library functions imported above. You MUST follow the docstrings and specification for these functions. You can also call new functions that are not yet defined. Hosever, these functions must be completing general, high-level subtasks (cook, cut, place on top, unstack). These functions cannot doing low-level actions.
You must follow the paeudocode provided by the user. You CANNOT add additional ateps, conditionals, loops that are not in the pseudocode.
examples:
Cook patty and lettuce.
Specifically:
Get list of all the lettuces in the kitchen. Get list of all the patties in the kitchen.
Decide a lettuce to use. Get list of all the available cutting boards in the kitchen. Decide a cutting board to to. Cut that lettuce
that cutting board.
Decide a patty to use. Get list of all the available stoves in the kitchen. Decide store go to. Cook that patty that stove. unn
<end_of of Lind_of_exapple www.ooeeewapeeeuerreoooo
Get a list of all the lettuces the kitchen. lettuces
aab__namssttatt_mttteeeeee thhat_mtch__tyee thatt.macchtypp((lettcc llettuce
Get a list of all the patties in the kitchen. patties
'
Decide a lettuce to use. lettuce_ to_use lettuces [0]
# Get list of all the available cutting boards in the kitchen.
78

---

cutting_boards
outting_board"
Decide a cutting board to go to.
secting_boarde [0]
# Cut that lettuce at that cutting board.
aut__bjeet_at_lccatioo (obj=lettuce. location-
# Decide a patty to use.
patty_to_use patties [0]
Get list of all the available stoves in the kitchen. stoves
Decide a stove to go to. toove_to_cook..
stoves
# Cook that patty at that stove. attpe_obbect_tt_ocaeeeeeee stove_ toree__t__ook__e cook_at)
(obj=patty_ tse location=
Cook two patties.
Specifically:
Get list of all the patties in the kitchen.
Decide a patty to use.
Get list of all the available stoves in the kitchen. Decide a stove go to. Cook that patty that stove.
Repeat the above steps for the second patty.
<end www.eeeaaaleeeeeeeeeee.o
Get a list of all the patties in the kitchen. patties = get_all_obj_ names that_ that_ (ppatty))
For two patties, do for in range(2))
# Decide patty to use. patty_to_use patties [i]
# Get list all the available stoves the kitchen. stoves oot. wtw.ammssstaatmmtcc..ppp... ('sttpeee'ooee' ('stove') # Decide stove to go to.
stove_to_cook.. = stoves
# Cook that patty that cook, abject_ (obpssttttooapptttttttoooooo _o__pptttttt_ueeee use, location
stove to stove_to_ cook__aaa_tt
unn
Cut lettuce before stacking it on top of a bottom bun. Then stack top bun on top of the lettuce.
Specifically:
Get a list of all the lettuces in the kitchen. Get list of all the bottom buns in the kitchen. Get list of all the top buns in the kitchen.
Decide a lettuce to use.
Get list of all the available cutting boards in the kitchen.
Decide a cutting board to to.
79

---

Cut that lettuce at that cutting board.
Decide a bottom bun to use.
Stack the lettuce on top of the bottom bun.
Decide top bun to Stack the top bun top of the lettuce. ann
<end_ of www.eeeeeeeeeeeeeeeeeeee
Get list of all the lettuces the kitchen. lettuces get_all_obe www.n. thaatmmtchh... type ('lettucc) Get a list of all the bottom buns in the kitchen. bontom_buns get sr__aal_obj__aaa aat___mmsstat_matteete wyt.matcchtppeebott....... Get list of all the top buns the kitchen. ton_buns set_all_obj_nant thamte_thtt__se thtttthtthhhtttppee
('tppbbuu))
# Decide lettuce to use. lettuce_to_ Get list of all available cutting boards the kitchen.
lettuces (0]
botting.boards get_all..ocation namen that type outting.boardy
# Decide cutting board to go to. nutting.board_to_coo Cut that lettuce at that cutting board.
burting-boards [0]
aatt_ojjct_at_locaite (obj=letuuc_to__ss location
Decide bottom bun use.
botton__un_toouee bottom_buns [0]
# Stack the lettuce on top of the bottom bun.
obj1 should be the lettuce, obj2 should be the bottom bun. oot.koooj_.oooo002ooooooeeeeeeee
boj--bottom_bunttouuus
Decide a top bun to use.
top_bun_tee hop_bun_to__ae to__tt__eee = top_buns[0 [0] [0]
Stack that top bun top of the lettuce.
obj1 should be the top bun, obj2 should be the lettuce. obj2=letruce_to_us))
Step 2: Define composite actions Given a function header, the LLM outputs code that may contain undefined functions.
step2:
main:
You are a Python code generator for robotics. The users will first provide the imported Python modules. Then, for each code that they want you to generate, they provide the requirement in Python docatrings.
<end_of_systens <end_oo_ooyteur_mssaga
# Python kitchen robot control script from env_utils import get_obj_ is_holding, is_in_a_stack,
# **
All the code should follow the specification.
env_utils Specifications:
get.obj_locction(bb]]
Paraneters:
obj (str): A string that has the type of object (one of lettuce', 'patty', "bottom_bun bunop_bun ?) and an id. For example 'lettuce5', 'patty7', 'botton_buni', 'un4_bun4'
80

---

Return:
(str) location that the object is currently at. A string that has the type location (one 'table' 'cutting_boan board', stove') and an id. For example: 'table2', butting.boardll. stove5'
(s_holding(ob))
Parametera:
obj (str): string that has the type object (one of lettuce', 'patty', 'bottom_bur bun ,'top_bun') and an id. For example 'lettuces', 'patty7', 'bottom_bunl', 'top_bun4'
Return:
(bool) true if the robot is currently holding obj is a
Parameters:
obj (str): string that has the type of object (one of lettuce', 'patty', 'bottom_bun ,'top_bun') and id. For example 'lettuce5', 'patty7' bottom_bunl', 'top_bun4"
Return:
(bool) true if the obj is in a stack and the robot needs to unstack the obj
Parameters:
obj_at_top (str): A string that has the type of object ( of 'lettuce', 'patty', bunttom_bun' 'ttopbuu and an id. For example: 'lettuce5', 'patty7', 'bottom_bunl', 'top_bun4' Return:
(str) The name of the object that is directly underneath the obj_at_top A string that has the type of object (one of lettuce', 'patty', 'bottom_bun .'top_bun') ') and an id. For example 'lettuce5', 'patty7', nottom_bunl', 'top_bun4' ean
The robot is only capable of these basic actions: move, pick, place, stack, unstack, cook, cut. These functions require the robot hold the object: place, stack These functions require the robot to not hold anything: pick, unstack
You define functions that combine of these actions, but you cannot define new actions.
examples:
unn
You can use functions imported above and also call new functions
The robot might not be any object any location specified in the function parameters.
Define the function: Loctcia location location) anm
<end
def cut
(obj, location);
# cut object, the robot first needs be holding obj not is_ (holddnggobj)::: (obj):
# If the robot is not holding obj, there are scenarios:
# (1) if obj is in stack unstack obj
(2) else, pick up obj.
if is_iina.ettcck(bb)
Because obj is a stack, robot need to move then unstack the obj from the abj_at_botton first
81

---

abj_at_botton = itt.:oo__hatt.i...drraaaaaaa
obj)
move then unstack: first you move to abj_at_bottom bottom's location, then you unstack obj from obj_at_botte
nove_then, tot. obj_at_bottom, uas.eeel.oaattrrree...oooooooooo http::oo__..aaaaooooooo.aa..bbba else:
obj_at_botton,
move_ hove_then_piikk first you move to obj location,
then you pick obj
pove_then..iccxoojjooo
(obj))
# move then place: first you move to the location to cut at, then you place obj at that location
move paace # cut the object until it is cut
cut_until_is_cm (obj=obj) (objoooob
You can use functions imported above and also call new functions
The robot might not be any object or near any location specified in the function parameters.
Define the function: stack_objj1.o_ obj2)
<end cend.o..errmlee..errcuur
def stack stack_obj1_on_obj2(cb. obj2):
# To stack obj1 obj2, the robot needs to be holding
obj1
if not is [._hodding(obj1) If the robot is not holding obji, there are
acenarios:
(1) if obj1 is in stack unstack obj1
(2) else, pick up obj1.
# Because obj1 is in a stack, robot need to move
then unstack the obj from the obj_at_bottom first obj-at_bottom
obj obj.at_top-objt)
# move then unstack: first you move to the obj.at_bottom' location, then you unstack obj from at tove__ten_unsttck tom abj_ abj_at_bottom=obj__ bottom, unstack unstack cet location obj obj_at_bottom))
else:
# move_then_pick: first you move to obj's 's location then you pick obj up
get lot_tooj tove_thenn.ickk(o (obj))
pick_location-
# determine the location of obj2 to stack on obj2_location = get_ooj..
then stack: first you to obj2's location, then you stack obj1 on obj2
tove_then_steet to to_cctt obj_at_ obj_att stack_locatte
You can functions imported above and also call new functions
82

---

The robot might not be any object any location specified in the function parameters.
Define the function: ustack..0j...tro..c2220000 obj2)
of ceedooff.xaplee.eer.evee
def
(obj1, obj2):
To unstack obj1 from obj2, the robot needs not hold anything yet.
is.holding(abj1));
# Because the robot is holding obj1, unstacking must have been successful already
determine the location of obj2 to unstack from obj2_location = get_ooi_location (obj2)
move then unstack: first you move to obj2's location, then you unstack obj1 from obj2
move www._tttec_ssttcc unstack, unstack mmttack_ooattinooo....oaaaa
obj_at.botton-ob)2,
You can use functions imported above and also call new functions
The robot might not be near any object near any location specified in the function parameters.
Define the function:
cook cook_location)
<end cmmboot_xxaplll.uuss.qer
def cook Sboject_a__locatt
(obj, cook_location).
# To cook an object, the robot first needs to be holding obj if not is_holding (obj): (obj))
not holding obj, there are scenarios: # (1) if obj is in a stack unstacck obj
If the robot
# (2) else, pick up obj.
if is stack stackk
Because obj is a stack, robot need move then unstack the obj from the abj_at_botton first obj_at_botton
obj)
then unstack: firat you move the abj_at_bottom bottom location, then you unstack obj from obj_ obj_at_botton obj Djj_atbottoon oos (objaaatttttott.att.ttttttt. else:
move_thea. tave_then_unstaaee
move then_ichenppck first you move obj location,
then you pick obj up (obj))
move shve_then.iccc(bb-aaa)
# then place: first you to the location cook at, then you place obj at that location move thee_ttee_ppaeeeeeeeeeeeee (obj-ob,, (obb # cook the object until it is cooked is
step3:
83

---

main:
You Python code generator for robotics. The will first provide the imported Python modules. Then, for each code that they want you to generate, they provide the requirement in Python docstrings.
Python kitchen robot control script import numpy as np
from robot_utils import move, pick_up, place, cut, start_cooking, noop, stack, unstack from env_utils import is_cut, is_cooked, get_obj_locatin get_curr_locatin
All the code should follow the specification.
robot_utils Specifications:
move (curr_loc, target_loc)
Requirement:
The curr_loc cannot be the same as target_loc.
Move from the curr_loc to the target_loc.
Parameters:
curr_loc (str): string that has the type of location ( of 'table', outting.board', 'store') and id. For example: table2', 'cutting.boardd', 'stove5'
target_loc (str): a string that has the type of location board', 'stove and an id. For example: "table2', sutting.boardd', 'stove5' pick_up(obj, loc)
one of 'table',
Requirement:
The robot must have moved to loc already, and it cannot be holding anything else.
Pick up the obj from the loc.
Parameters:
obj (str): object to pick. string that has the type of object (one of 'lettuce', 'patty', 'bottom bun 'topp bun') and " id. For example: 'lettuceb', 'patty7', 'buni' loc (str): location to pick the object from. string that has the type of location (one of 'table', cutting-boarde 'stove 1) and id. For example: 'table2', place (obj, loc)
boardl' 'stove5'
Requirement:
The robot must have moved to loc already, and it cannot be holding anything else.
Place the obj on the loc.
Paraneters:
obj (str): object to place. string that has the type of object (one of 'lettuce', 'patty', 'botton bun 'top bun') and id. For example: 'lettuce5', 'patty7', 'buni' loc (str): location to place the object at. string that has the type of location (one of 'table', coatting_board" 'stove ") and id. For example: 'table2', 'cutting_boardd" 'stove5'
cut (obj)
Requirement:
The robot must be at the same location as obj.
Make progress cutting the obj You need call this function multiple times finish cutting the obj.
Parameters:
84

---

obj (str): object to cut, A string that has the type of cuttable object (must be 'lettuce') and id. For example: lettuce3' 'lettucel'
start. start_cooking(ob))
Requirement:
The robot must be at the same location obj.
Start cooking the obj. You only need to call this once. The obj will take an unknown amount before it is cooked.
Parameters:
obj (str): object to cook, string that has the type cookable object (must 'patty') and an id. For example: 'pattyl 'patty5'
noop ()
Do nothing
stack (obj_to_stackk, obj_at_botton
Requirement:
The robot must be at the same location as obj_at_bottom.
Stack obj_to_stack on top obj_at_bottom
Parametera:
obj_ko_stack (str): object to stack. A string that has the type of object (one of 'lettuce', 'patty', 'bottom bun't t'oop bun )) and an id. For example: 'lettuce5', 'patty?', 'bun1' obj_at_bottoo
(str): object to stack on top of. A string that has the type of object (one of 'lettuce', 'patty', 'botton bun't 'top bun') and an id. For example: 'lettuce5', 'patty7', bunl
unstack to
Requirement:
The robot must be at the same location obj obj_at_botton
Unstack obj_to_unstas from obj_at_bottom
Parameters:
toj unotuck (str): object to unstack. A string that has the type of object (one of 'lettuce', 'patty', 'bottom bun' 'op bun') and an id. For example: 'lettuceb', 'patty7', 'buni' (str): object to unstack from. A string that has the type of object (one of 'lettuce', 'patty', 'botton bun" 'top bun ') and an id. For example: 'lettuce5', 'patty7', 'buni'
obj_at_botton
env_utils Specifications:
(s_cut(ob))
Parameters:
obj (str): string that has the type cuttable object (must 'lettuce') and an id. For example: 'lettuce3', 'lettuce1
Return:
(boolean)
is cooke
Parameters:
obj (str): A string that has the type of a cookable object (must be 'patty') and an id. For example: 'pattyl', 'patty2'
Return:
(boolean)
get_curr.location(
Return:
(str) location that the robot is currently A string that has the type location (one of 'table', 'cutting. board', stove') and an id. For example: 'table2', 'ourting_boardd', stove5
get_obj_location(ok
85

---

Parameters:
obj (str): string that has the type cuttable object (must 'lettuce') and an id. For example: 'lettuce3', 'lettucei
Return:
(str) location that the object is currently at. A string that has the type location (one of 'table', 'cutting.boue board', stove and an id. For example: 'table2', "cutting.board! stove5.
examples:
You only use the functions imported in the header.
Define the function: basic_move (targett_ooc)
Move to any location specified the target_loc.
<end_ oondoo...xmmloo.usrrrvvu
def basic_ (tai___tvettrggt__11o.
if get_curr_locations != target_loc: (gtt_ccur target_loc)
unu
You can only use the functions imported the header.
Define the function: cook_until_ is__ll_i___ooo (o
<end cmm.oo..rrapllo..ssr..eee
def cook_untinn (oo.uttil_i__cooeddd]))) start_ start_cooking(ob)) while (8_ccooed
()
You can only use the functions imported in the header.
Define the function: move staetthhh_atacooooooee
abj_at_bott
un
<end_ of pnnd_of.exmple_uurrquuu
def move_ stack (obj_to_stacd abj_at_botton
For stacking, because the location is not provided, we need to determine the stack_location
stack_location = (ot_::___occaiiiot_______o...
get_ccurr_occation( != stack_location, move [ore@getcur_loocttioo (), stack_location)
abj_at_bottom,
86

---

18.3 EPIC Kitchens Task Prompts
18.3.1 Recursive Summarization
recursive_ssmmariaattt
main:
You are a helpful summarizer that recursively summarizes trajectory into more and more compacted form.
You are given a trajectory. Your goal is to summarize the trajectory into a more compacted form and then determine whether the state trajectory is sufficiently summarized.
must respond using of response]] at the very end): [[Reasoning:]]
following form (you must generate [[end
You should first identify what type of trajectory this is Then, you should determine what type you will summarize the trajectory into. Finally, you should determine whether the new type trajectory is sufficiently summarized or not. [[Is the new trajectory sufficiently summarized? (yes/no) :]] You should only respond using either "yes" or "no", and nothing else.
[[Summarized trajectory:ll
You should actually summarize the input trajectory and output it here.
[[end response]]
You must follow these rules when you are summarizing a trajectory.
Rules:
You must slowly summarize the trajectory from type dow-lewe
another folloving this order: state trajectory
action trajectory --> high-level subtask trajectory.
You cannot skip type (e.g. you cannot directly summarize a low
levell action trajectory into high-level subtask trajectory). lowelevel action trajectory is represented list.
unordered
The level actions must be one of the following: "the robot moved from locationi to location2", "the robot picked up itemi", the robot placed down iteml at location1 ", "the robot soaped iteml until rinsed item1 until became clean", "the robot turned on tap1", the robot turned off tap2". You should never define new low-level actions.
became soapy because iteml was dirty", "the robot
hige-level subtask trajectory is represented list. This high-leve! subtask should refer to one continuous section of the states. For example, you cannot say "at states 1-5, and states 10-15, the robot did interval of states.
unordered
There can only be one
The high- subtask must one of the following: "brought ITEMS] from [LOCATION 1] to [LOCATION 2]", "soaped [ITEM] until became soap", "rinsed [ITEM] until it became clean", "picked up ITEM]", "placed [ITEMS] in [LOCATION]", "turned [TAP]", "turned off [TAP]". You should never define new high-level subtasks. You choose from the list above. examples:
[[Input Trajectory :]] [Scenario 1] Wash objects at sink. All cleaned objects should placed the sink. Objects to clean
spoon_1 at countertop_1
87

---

fork_1 at countertop_1
Initial Condition (State : at (" (Ccuuteerto__1" is ("sron"""poo_11
is_dirty('fori near (Saa(("_oo_ 1') near (' fork__1")
State 2:
is ga_in_hand f is_in_hand spoon_ far ("sink__1)
State 3: at ("sink_1")
State 4: is_soapy (" ('fork_1') is_on tap_1")
State 5: is_soapy (" spoon_1')
State 6: is_clean (" ("spoon_1")
State 7
is_clean (' ('fork_1')
State is (' tap_1')
in in eink_1') in un('spoo__1. "sink_1') <end_ot_example__uer.
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of robot (a coottttrter))) (esdirtt(''ponn111))
and the objects
will summarize the state trajectory into lov-level actions. Low-level actions are not compacted enough yet because low-level actions can still be combined into high-level subtasks. The trajectory will NOT be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :]] no
[[Summarized trajectory:]l
In [Scenario 1], at state 1-2, the robot picked up spoon_1. The robot picked up fork_1.
At state 2-3, the robot moved from countertop_1 to sink_1. At state 3-4, the robot turned on tap_1. The robot soaped fork_1 until it became soapy because fork_1 was dirty.
At state 4-5, the robot soaped spoon_ until it became soapy because spoon_1
dirty.
At state 5-6, the robot rinsed spoon_1 until it became clean. At state 6-7, the robot rinsed fork_1 until it became clean. At state 7-8, the robot turned off tap_1. The robot placed spoon_1 in sink_1. The robot placed fork_1 in sink_1. [[end of response]]
[[Input :rajectory:]]
[Scenario 1] Wash objects at sink. All cleaned objects should be placed the sink.
Objects to clean
spoon_1 at dishwasher_1
88

---

plate_1 at sink_1 plate_2 at sink_1
Initial Condition (State 1): in (" spoon_1", "dishwashe__1 in (" plate_1", "sink_1") in (n plaat__2',, "sink_1")
State 2: is_in_hand ( spoon_1")
State 3: at (tssikk_1 is is_dirty (' spoon_1
State 4: is_on ((' tap_1')
State 5: is_foamy (" sponge_1')
State 6: is_soapy ( spoon_1')
State 7: is_clean (" spoon_
State 8: in (" ('spoon__', 'sink_1')
State 9: is_off ('ta_p11')))) _s_in_hand ( plate_1")
is_dirty( (" plate_1")
State 10:
is_on (('tap_1')
is frinn_hande (" sponge_1") is_foamy (' spongee_1")
State 11: is_soapy ("plate_1")
State 12: clenn plate_1')
State 13: in (" plate_1', 'sink_1')
State 14: is_off (" tap_1"))
State 15: is_in is_in_hame ( plate_2" 2")
is is_dirty (' plate_2")
State 16: is_on ((" 1a)_1")
State 17: fs_foamy ( sponge_1")
State 18: is_scapy (' ('plate_2))
89

---

State 19:
is_clean ( plate_2")
in (" ("late__2 sink_1') <end_of_example_pn
[[Reasoning:]]
The input trajectory contains state predicates because the trajectory talks about the status of the objects (in un"'spoo__1", dishvasher_1"))
I will summarize the state trajectory into lov-level actions. Low-level actions are not compacted enough yet because low- actions still be combined into high-level subtasks. The new trajectory will NOT be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no) :1] no
[[Summarized trajectory:]l
In [Scenario 11, at state 1-2, the robot picked up spoon_1 from dishwasher_1.
At state 2-3, the robot moved from dishwasher_ to sink_1. At state 3-4, the robot turned on tap_1.
At state 4-5, the robot picked up sponge_1, which was foamy. At state 5-6, the robot soaped spoon_1 until it became soapy because spoon_1 was dirty.
At state 6-7, the robot rinsed spoon_1 until it became clean. At state 7-8, the robot placed spoon_1 in sink_1. The robot placed sponge_1 in sink_1.
At state 8-9, the robot turned off tap_1. The robot picked up plate_1.
At state 9-10, the robot turned on tap_1. The robot picked sponge_l, which was foamy.
At state 10-11 the robot soaped plate_1 until became soapy because plate_1 was dirty.
At state 11-12, the robot rinsed plate_1 until it became clean
At state 12-13, the robot placed plate_1 in sink_1. The robot placed sponge_1 in sink_1.
At state 13-14, the robot turned off tap_1.
At state 14-15, the robot picked up plate_2. At state 15-16, the robot turned on tap.1.
At state 16-17, the robot picked up sponge_1, which foany. At state 17-18, the robot soaped plate_2 until became soapy because plate_2 was dirty.
At state 18-19, the robot rinsed plate_2 until it became clean The robot placed plate_2 in sink_l. The robot turned off tap_1. [[end of response]]
[[Input Trajectory: :]]
[Scenario 1]
In [Scenario 11, state 1-2, the robot picked up spoon_1. The robot picked up fork_1.
At state 2-3, the robot moved from countertop_1 to sink_1. At state 3-4, the robot turned tap_1. The robot soaped fork_1 until
became soapy because fork_1 dirty. At state 4-5, the robot soaped spoon_1 until it became soapy because spoon_1 was dirty.
At state 5-6, the robot rinsed spoon_1 until it became clean. At state 6-7, the robot rinsed fork_1 until it became clean. At state 7-8, the robot turned off tap_1. The robot placed spoon_1 in sink_1. The robot placed fork_1 in sink_1.
[[Reasoning:]]
The input trajectory contains lov-level actions because the trajectory mentions "picked up" "moved", etc. will summarize the -evelevel action trajectory into high- subtasks.
90

---

High -leeel subtasks compacted enough because they cannot be combined together without losing important information.
The new trajectory will be sufficiently summarized. [[Is the new trajectory sufficiently summarized? (yes/no). :]] yes
[[Summarized trajectory:]]
In [Scenario 11, at state 1-3, the robot brought spoon_1 1 and fork_1 from countertop_1 to the sink_1.
At state 3-4, turned on tap_1
Åt state 3-4, soaped fork_1 until it became soapy At state 4-5, soaped spoon_1 until it became soapy At state 5-6, rinsed spoon_1 until
became clean.
At state 6-7, rinsed fork.1 until it became clean. At state 7-8, turned off tap_1, and placed spoon_1 and fork_1 in sink_l.
[[end of response]]
[[Input Trajectory:]] :]]
[Scenaric 1]
In [Scenario 1], at state 1-2, the robot picked up spoon_1 dishwasher_1,
At state 2-3, the robot moved from dishwasher_ to sink.1. At state 3-4, the robot turned on tap.1.
At state 4-5, the robot picked up sponge_1, which was foamy. At state 5-6, the robot soaped spoon_1 until it became soapy because spoon_1 was dirty.
At state 6-7, the robot rinsed spoon_1 until it became clean. At state 7-8, the robot placed spoon_1 in sink_l. The robot placed sponge_1 in sink_l.
robot turned off tap_1. robot picked up At state 8-9, plate_1.
At state 9-10, the robot turned on tap_1. The robot picked sponge_1, which was foamy.
At state 10-11 the robot soaped plate_1 until it became soapy because plate_1 was dirty.
At state 11-12, the robot rinsed plate_1 until it became clean
At state 12-13, the robot placed plate_1 1 in sink_1. The robot placed sponge_1 in sink_1.
At state 13-14, the robot turned off tap-1. At state 14-15, the robot picked up plate_2.
At state 15-16, the robot turned on tap_1.
At state 16-17, the robot picked up sponge_1, which foamy. state 17-18, the robot soaped plate_2 until became soapy because plate_2 was dirty.
At state 18-19, the robot rinsed plate_2 until it became clean The robot placed plate_2 in sink_1. The robot turned off tap_1. <end_of peaddfffexampeequueee [[Reasoning:]]
The input trajectory contains lov-level actions because the trajectory mentions "picked "m",,moved", etc. will summarize the lew-level action trajectory into high- subtasks.
High-leved subtasks are compacted enough because they cannot
combined together vithout losing important information.
The new trajectory will be sufficiently summarized. [[Is the trajectory sufficiently summarized? (yes/no) :]1 yes
[[Summarized trajectory:]l
In (Scenario 1], at state 1-3, brought spoon_1 from dis dishwasher_1
sink_1
At state 3-4, turned tap_1.
At state 4-6, soaped spoon_1 until it became soapy At state 6-7, rinsed spoon_1 until it became clean At state 7-8, placed spoon_1 in sink_1
91

---

At state 8-9,
turned off tap-1 and picked up plate_1
At state 9-10,
turned on tap-1.
At state 10-11,
soaped plate.1 until became soapy
At state 11-12,
rinsed plate_1 until it became clean
At state 12-13,
placed plate_1 in sink_1
At state 13-15,
turned off tap_1 and picked up plate_2 tap_1
At state 15-16,
turned
At state 16-18,
soaped plate_2 until became soapy
At state 18-19,
rinsed plate_2 until it became clean, turned
off tap_1, and placed plate_2 in sink_1
[[end of response]]
Summarized Demonstrations -> Task Specification
hummary-2_spee
main:
You are helpful assistant who analyzes the trajectories and summarizes them into concise paeudocode style task specification
You are given (1) high -lvel goal and (2) one or more trajectories where each represents different scenario. Your goal is to summarize these trajectories into a compact task apecification, written in a peeudocode style.
You must respond using the following format (you must generate [[ end of responsell [[Reasoning:]]
the very end)
You should first list out the order of the high- subtask trajectories in all scenarios. Then, you should consider if certain sections of the subtasks are repeated in the scenario and can be represented by a loop. Two scenarios having exactly same list does not mean that there among subtasks in one individual acenario.
loop. loop only exists
Overall, your task specification should work for all acenarios. You should make subtasks ordering across all scenarios. You should also make sure that the task specification uses loops when there is any repetition.
that the task specification matches the
[[Task Specification:1]
You should first state the high-level goal. Then, you should say Specifically:" before outputting the pseudocode style task specification.
[[end of response]]
must follow these rules you are writing the task specifications.
You must write the task apecifications in pseudocode style. You should not write the task specification list. You cannot number any line. examples:
[[iigh-Level Goal:]]
Wash objects at sink. All cleaned objects should be placed in the sink.
Objects to clean
spoon_1 at countertop_1
fork_1 at countertop_1
Initial Condition (State 1): at (" ('cunterroop11" is is_dirty(" (* ("spon__11)
is is_dirty ('fork_1')
92

---

near (' spoon_1")
near ( fork_1") [[Trajectories :1]
In [Scenario 1l, at state 1-3, the robot brought spoon_1 and fork_1 from countertop_1 to the sink_l.
At state 3-4, turned At state 3-4, soaped fork.1 until it became sоapy At state 4-5, soaped spoon_1 until
tap-1
became soapy At state 5-6, rinsed spoon_1 until it became clean.
At state 6-7, rinsed fork_1 until it became clean.
At state 7-8, turned off tap_1, and placed spoon_1 and fork_1 sink_1.
Send_ofe www.pleeuser. [[Reasoning:]]
There are 2 dishes that got washed: [fork_1, spoon 1] The list of high level actions that happened in order is: [ move dishes from to B, turn on tap, soap, soap, rinse, rinse, turn off tap, placed dishes in sink_1]
Because in [soap, soap], the high -leell action of soapping got use for-
repeated twice (once for each dish brought) loop.
Because [rinse, rinsel, the high-level action rinsing gets repeated tvice (once for each dish), we can use for-loop Since the dishes can be kept in hand, there is rinsing.
place
and pick upp before soapping Rinsing involves the use tap water, which is why was turned on at some point before the rinse cycle, and turned off after
[[Task Specification:1l
Wash objects at sink. All cleaned objects should placed the sink.
Specifically
Get list of all objects wash
Bring all objects from countertop_1 to sink_1 Turn on tap For each object Soap object For each object
Rinse object Turn off tap For each object
Place object in sink_1 [[end of response]]
[[High-Leve] Goal:]] Wash objects at sink. All cleaned objects should be placed in the sink.
Objects to clean
spoon_1 at countertop_1
fork_1 at countertop_1
Initial Condition (State 1): in ('spoon_1', "dishaasher_1") in ( 1'plae_e_1", 'sink_1')
in ("pplate_2", 'sink_1")
[[Trajectories:1]
In [Scenario 1], at state 1-3, brought spoon_1 from dishwasher_1 to sink_1
At state 3-4, turned on tap_1.
At state 4-6, soaped spoon_1 until it became soapy
At state 6-7, rinsed spoon_1 until it became clean
At state 7-8, placed spoon_1 in sink_
At state 8-9, turned off tap.1 and picked plate_1 At state 9-10, turned on tap_1.
93

---

point before the rinse cycle, and turned off At state 10-11,
soaped plate_1 until it became soapy
At state 11-12,
rinsed plate_1 until became clean
At state 12-13,
placed plate_1
sink_1
At state 13-15,
turned off tap_1 and picked up plate_2
At state 15-16,
turned on tap_1
At state 16-18,
soaped plate_2 until it became soapy
At state 18-19,
rinsed plate_2 until became clean, turned
off tap_1, and placed plate_2 sink_1
<end_of_ www.ooeexaameeeeeerreeeeee [[Reasoning:]]
There 3 dishes got washed: [spoon_1, plate_1, plate_2] The list high level actions that happened in order is: [ move dish from A to B, turn on tap, soap, rinse, place, turn off tap, pick up, turn on tap, soap, rinse, place, turn off tap, pick up, turn on tap, soap, rinse, place, turn off tap]
Only spoon_1 is brought dishes are already in the sink.
the sink from the dishwasher, other
The spoon_1 does not have pick_up action associated with it
because its already in hand when brought from dishvasher_1 to sink_1. The action without side effect.
be added to the code for generalizing
The actions [pick_up, turn on tap, soap, rinse, place, turn off tap] are repeated for each dish, so we can use a loop.
Rinsing involves the use of tap water, which is why it is turned at after.
[[Task Specification:1l
Wash objects at sink. All cleaned objects should be placed in the sink.
Specifically
Get list of all objects to wash Bring all objects to sink_1 For each object in all objects:
Pick_up object Turn on tap_1
Soap object
Rinse object Place object in sink_1 Turn off tap_1
[[end of response]]
18.3.2 Recursive Expansion
spec_ h. his.ee..hhhheeeecodee: main:
You are a Python code generator for robotics. The users will first provide the imported Python modules. Then, for each code they want you to generate, they provide the requirements and pseudocode Python docstrings
send_of__ystem_meesee You need to write robot control scripts in Python code. The Python code should be general and applicable to different environments.
Below are imported Python libraries functions that you can use. You CANNOT import new libraries.
# Python kitchen robot control script from env_utila import get_all.objecta from robot_utils import bring turn_off, turn_on, soap, rinse, pick_up, place, go_to, clean_with
94

---

Below shows the docstrings for these imported library functions that you must follow. You CANNOT add additional parameters these functions.
All the code should follow the specification. Follow descriptions when writing code.
Specifications:
get_all_obfects()
Return:
(list) list of string of objects that need be cleaned (objj, loc)
involves calling pick_up, go_to and place within the function.
Parameters:
obj (List Strings of the form "object_id" (e.g. plate_1") that need to brought to the location loc loc(str): location string of the form "loc_id" (e.g. "sink_1
")
Return:
(void)
turn_off (tap_name)
turns off tap
Parameters:
tap_name (str); A string that has the type of tap (must be 'tap') and an id. For example: 'tapi', 'tap2'
turn_on (tap_name) turns on tap
Parameters:
'tap') ) tap_name (str): string that has type of tap (must and an id. For example: 'tap1', 'tap2"
soap(obj)
soap (obj)
soap the object "obj". It must be in hand before calling this function.
Parameters:
obj (str): String of the form "object_id" (e.g. "plate_1") that needs to be saoped rinse (obj)
rinses the object "obj". It must be in hand before calling this function, tap must turned on before calling this function, and switched off after
Parameters:
obj (str): String of the form "object_id" (e.g. "plate_1")
that needs to rinsed (obj)
Parameters:
obj (str): String of the form "object_id" (e.g. "plate_1")
pick up
place(obj, loc)
Parameters:
obj (str); String of the form "object_id" (e.g. "plate_1") to pick up
loc (str): Location string (e.g. "sink_1") where obj is placed (loc):
Parameters
loc (str): Location string (e.g. "sink_1") to go to from current location.
[hean_withobbi] tool)
Parameters:
obj (str): Object strings (e.g. "plate") to clean
tool (str): Tool strings (e.g- "sponge" "tovel")
clean with
95

---

You must follow these rules when you are generating Python code. You MUST ONLY use Python library functions imported above. You MUST follow the docstrings and specifications for these functions.
You CANNOT call define new functions. You CANNOT use functions that NOT imported above.
You muat follow the instructions provided by the CANNOT add additional steps, conditionals, or loops that are not in the instruction. examples:
You
unn
Wash objects at sink. All cleaned objects should be placed in the sink. Specifically
Get a list of all objects to wash Bring all objects from countertop_1 to sink_1 Turn tap_1 For each object Soap object For each object Rinse object Turn off tap_1 For each object Place object in sink_1
ofnd_of_user_que
objects = get_all httpggoobeects__tooooooooooo "sink_1")
("tt__nn""ap_111
for object objects: soap(object)
for object in objects: rinse curn_off("tap.11
for object objects: place "sink_1")
unn
Wash objects at sink. All cleaned objects should be placed in the sink. Specifically
Get a list of all objects to wash Bring all objects to sink_1 For each object in all objects:
Pick_up object
Turn on tap_1
Soap object Rinse object Place object in sink_1 Turn off tap_1
unn
<end_of_use ufserrquuerr
objects get_all. laang_oojectstttto loc "sink_1") for object in objects:
96

---

pick_up (object)
turn_on ("tap_1") soap (object)
rinse (object)
place (object, "sink_1") turn_ off ("tap_1")
19
Other Long Examples
19.1 Example Robotouille Query
[Scenario 1] Make burger.
State 2:
'pattyl' is not at 'table1' 'roboti' is holding 'patty1'
State 3:
'roboti' at 'stove2'
'roboti' is not at 'tablel'
State 4:
'pattyl' at 'stove2'
'roboti' is not holding 'patty1'
State 5:
State 6:
State 7:
State 8: 'pattyl' cooked
State 9:
'pattyl' is not at 'stove2' 'robot1' is holding 'pattyl'
State 10:
'robot1' is not at 'stove2' rrobot1 is at 'table3'
State 11:
'patty1' is at 'table3'
'pattyl' is on top of 'bottom_bunt' 'roboti is not holding 'pattyl'
State 12:
'robot1' is not at 'table3' 'roboti' is at 'table6'
State 13:
'tomato'' is not at 'table6' 'roboti' is holding 'tomatol'
97

---

State 14:
'robot1' is not 'table6'
robot1 at 'cutting.board"
State 15:
'tomato'' is at boarting.boardd
'robot1 ! is not holding 'tomatol
State 16:
State 17:
State 18: 'tomatol' cut
State 19:
'tomatoo' is not at 'cutting_board1" 'robot1' ! is holding 'tomatol"
State 20:
'robot1' is at 'table3' 'roboti is not at 'cutting. board1
State 21:
'tomatoi' is at 'table3'
'tomato"" is on top of 'pattyl'
'robot1' is not holding 'tomatol
State 22:
'roboti' is at 'table5' 'roboti' is not at 'table3'
State 23:
'lettuce1" is not at 'table5' 'roboti' is holding 'lettucei'
State 24:
'roboti' not at 'table5' 'robot1" is at 'cutting_board1
State 25:
'lettucel at "cutting_boan board1' 'robot1' is not holding 'lettucel'
State 26:
State 27:
State 28:
'lettucel is cut
State 29:
'lettucel' is at coattingbboardl. 'robot1' holding 'lettucei"
State 30:
'roboti' is at 'table3' 'roboti not 'curting.board
State 31: 'lettucel' is at 'table3'
98

---

'lettucel" is on top of 'tomato1 'robot1' ! is not holding 'lettuce1"
State 32:
'roboti' is at 'table4' rroboti is not 'table3'
State 33:
buni is not at 'table4' 'robot1' is holding 'top_bunl'
State 34:
'roboti' not at 'tabled' 'roboti' is at 'table3'
State 35:
'top_buni' at 'table3'
'top_buni' is on top of 'lettucel' 'robot1' is not holding 'top_bunl'
[Scenario 2] Make burger.
State 2: 'patty3' ! is not 'table6' 'robot1 holding pattty3'
State 3:
'roboti' 1 is at 'stove3' 'robot1 not 'table6'
State 4:
'patty3' is at 'stove3'
'robot1' is not holding 'patty3'
State 5:
State 6:
State 7:
State 8: 'patty3' is cooked
State 9:
'patty3' not at 'stove3' 'robot1" is holding 'patty3'
State 10:
'roboti' 1 is 'roboti at 'table5'
'stove3'
State 11:
'patty3' 1 is at 'table5'
'patty3' on top of 'bottom_bun3 rroboti' is not holding 'patty3'
State 12:
'roboti' 1 is at 'table3' 'robot1' not at 'table5'
State 13:
99

---

'tomato3' is not at 'table3' 'robot1 is holding 'tomato3
State 14:
'roboti' is not at 'table3' rrbbot1 is at "outting_board3
State 15:
"tomato3" is at 'cutting.board3'
'robot1' is not holding 'tomato3'
State 16:
State 17:
State 18:
'tomato'' is cut
State 19:
'tomato3' not at 'cutting.boate b 'roboti' is holding 'tomato3
State 20:
'robot1' at 'table5' 'roboti' is not at 'cutting.board3'
State 21: 'tomato3' 'tomato3' on top of 'patty3' 'robot1' is not holding 'tomato3'
'table5'
State 22:
'roboti' at 'table7'
'roboti' is not at 'table5'
State 23: 'lettuce3'
'table7'
'robot1' holding 'lettuce3'
State 24:
'roboti' ! is not 'table7' 'robot1' at coutting_board3
State 25:
'lettuce3' is at Saatting_board3 'robot1' ' is not holding 'lettuce3
State 26:
State 27:
State 28:
'lettuce3' is
State 29:
'lettuce3' is not at 'cutting_boards 'roboti' ! is holding 'lettuce3
State 30:
'roboti' is at 'table5' 'robot1' is not at bcurting_board3!
100

---

State 31:
'lettuce3' at 'table5'
llettuce3" is on top of 'tomato3' 'robot1' is not holding 'lettuce3'
State 32:
'roboti' at 'table9'
'roboti' is not at 'table5'
State 33:
'top_bun3' not at 'table9' 'roboti' holding 'top_bun3"
State 34:
'robot1' ' is not 'table9' 'roboti' at 'table5
State 35:
'top_bun3' is at 'table5'
ttop_bun3" on top 'lettuce3'
'roboti' not holding 'top_bun3
19.2 Example EPIC-Kitchens Query
[Scenario
Wash objects in sink. All clean objects must be placed in drying rack. Objects to clean
mezzaluna_1 in sink_1
potter:potato_1 knife_1 in sink_1
sink_1
board: in sink_1
Initial Condition (State 1); is us_in_hand sponge_1") in (''nezzaluna_"" "sink_1") at ("sink_1")
State 2:
is_in_ hand ("enaanmmmezauuaa.1" dirty ("mezzaluna,11)
mezzalunaa
State 3: is_soapy (" ("mezzaluna_1")
State 4: in m''zezzaluna_1', 'sink_2")
State 5: in peppeeer:otato.... "sink_1")
State 6: is_in_ hand peeler: potato__")
State 7: is_soapy ("
State 8: in ('peeeer::: :otator "sink_2')
101

---

State 9: in (n k "sink_1")
State 10: is_in_hand ("knife_1")
State 11: is_soapy "
State 12: in knife_1", "sink_2")
State 13: in('borrd: cutting.1', 'sink_1')
State 14: _s_in_hand board: board: cutting_1")
State 15: ss_so board: cutting.
State 16: ("oard:: cutting_1', 'sink_2')
State 17: is_on is_on(" ("tap_1")
State 18: is_in_hand board: cutting..)
State 19: is_clean (" board: outtinggt
State 20: in board: cutting_1", 1dryingrack_11)
State 21: is iandinnndd( ('mezzaluna_1')
State 22: is_clean (" "mmzzalunaa11)
State 23: m'ezzaluna_.", "dryingrack_1")
State 24: is (nnd ('ppeee 1""
State 25: isenl
State 26: in :otatoo,,, 1dryingrack_1")
State 27: is is_in_hand hand (in"i_eeee
knife_1")
State 28: clenn "nife_1'")
State 29: in knife_1', 1"ryingrack_1")
State 30: is_off (" ta)_1")
102

---

19.3
Intermediate Reasoning Ablation Helper Functions
19.3.1
No Reasoning Helper Functions
def move _ove_then_a unstack (obj tobj_tk__ unstack, obj_at_bottom,
unstack,location
For unstacking, we need to move to the location of the bottom object
if ret_curr_location != get_obl bott_oottmm) coreegeet.crr.bccaion (), ott. unstack unstack [obj_to_unsteeet obj_at_botton
# After unstacking, we need to move to the instack_locatid if get_curr.location( != unstack_location:
cove(get_curr_oocati
unstack_loaatinn
10 def move_then_ pove_tten..ick(ob)))
obj_location
get_curr () !- obj_location:
13
move moveegeet.crr__ocatiio O,, obj_location)
14
pick_up obj_location objeoootooooo]
15
def move then whw.thhee_ppacecceee (obj, pace_location):
get_cuur.location != locationn
18
move coneegeet.crr.lccciinn place place_location,
place_location
19
20
def cook wookkuntillisseom wtw.unttiiss_ccoeeddooo (obj):
22
start_cooking (ob))
23
while not is (obo) ()
24
def cook_ at_ location);
27
To cook an object, the robot first needs to holding obj if not is hs_holding (obj):
28
29
# If the robot is not holding obj, there are 2 scenarios: # (1) if obj is in stack unstack obj (2) else, pick up obj.
32
if asain
33
# Because obj is in a stack, robot need to move then unstack the obj from the obj_at_ first obj_at_botton gec.oo..he...suuudeeetth........ move_then_unse obj at aa_tttttttoomm else:
35
obj_at_bottomt oorethoe._.saccccoooooooooossesr
36
# Since obj is not in stack, robot can just move then
pick it up nove_then.picctobjooo
38
39
place the object at the location to cook at
phapehp place_locationn-loctioo cook the object
42
cook_until_is_comm
(obj=obj)
43
44
def cut cicii_is_ee iss (obj):
while not cut (obj):
cub())j)
47
18 def cut_ cut_object__. at wtt.obeecttt__lcattoooooooooo (obj, location);
# To cut an object, the robot first needs to be holding obj 19
if not hs_hinding (obj):
103

---

# If the robot is not holding obj, there are 2 scenarios:
# (1) if obj is in stack unstack obj
53
# (2) else, pick up obj.
54
if as_in_a_stack (obj):
55
# Because obj is in stack, robot need to move then
unstack the obj from the obj_at_botton first obj_at_bottor move_then_ (og obj at bottoot unstack_ lot.accc.ooatiioonee....oo----- (objatttooooottttottttotooo
get.obj.that_ wnd..neeeeneaahcccmmmmme aot top==bb)
57
obj_at_botton
58
else:
Since obj is not in stack, robot just then
pick it up
60
move move_then_ picen_
61
# place the object at the location to cut at
move. peen.plac(oob-oott cut the object cut_untill_s_cutt (objj_soo (obj=obj)
64
65
66
def move thtttttthn_sstteeeee shov__the_sstccccbbbttooooe
obj_at_bottom, stack_location):
get_ [bt_curr_locationn !- stack stack_ locationn
68
nove(get..urr._ocaa O), socaton) stack ttack abj_at_botton
69
70
def stack_objilon__obj2(00
obj2):
To stack obj1 obj2, the robot needs be holding obj1 not (..holding(obj]]
13
14
# If the robot is not holding obji, there are 2 scenarios: # (1) if obj1 is in a stack unstack obj1
75
# (2) else, pick up objl. if sta
77
78
Because obj1 in stack, robot need to move then unstack the obj from the obj_at_bottom first obj_at_botton= =
80
obj at bottom,
81
else:
Since obj1 is not in stack, robot just move then pick it up nove_tuen_pickkooo-oojj.
83
84
determine the location of obj2 to stack on obj2_location get_obj_ (obj)) move to obj2's location then stack obj1 obj2 abj_at_bottom-obb2.
87
move move_ sta.thhhnnncanaeneoe (ob_o stack_ ttack_occatioonoof_.loooo
19.3.2 Only List Helper Functions
def move shenaunstakk tobj_to_unstack, obj_at_botton instack_location
For unstacking, we need to move the location of the bottom object
if get.curr_locationl I= gtto0g_ieaatiancbb...e...tttaa move wocattooooooooooo0 get.ooj._Lbcccttooooooo_________ unstack (oottackoob..touusstcc.. After unstacking, we need to move to the unstack_locat if
abj_at_bottom]
I= unstack move move@geet.urr.lcatioo
instack_locatinn
10 def move move_thenn_pcck(ojj
11
obj obc_tocation get_obj_location (obj)
12
if get_curr_locationn !- obj obj_location:
move cereeeeecrr_loccatiee (), obj_location)
104

---

(pbjj obj_location)
16
def tove_thenn.lacceob. place_location]
17
1f (rt_:gttcuur__oaaiioo place_location)
!- place_location
18
place place_location)
21
def cook_
22
caart_cookingooo while not (obo) m)
start_ (obj)
25
26
def cook objectt at_ Itt_locationotte (obj, location):
27
# To cook an object, the robot first needs to be holding obj if not is (olding(obj)>
# the robot is not holding obj, there 2 scenarios: obj is # (2) else, pick up obj.
30
# (1)
stack unstack obj
31
32
if ss_in_aastacckob)))
Because obj is in stack, robot need to then unstack the obj from the obj_at_bottom first wttp::/ttt....eemmmmaaaaaa..oooo
34
obj_at_botton move_ther corp obj at__a_bbttton unstack_ orrtacd_oooatoongeee..o........ ao.j else:
35
abj_at_botton,
37
Since obj is not in stack, robot can just move then
pick it up
38
move t___the_________ pick (icjn---coobo-bb
place the object at the location cook at move (ben_ place_ cook the object ioo.eei
40
41
42
def cove_then_.stacc(ct)tttotttt get_curr_ move (ove (), stack_location
obj_at_bottom, stack_location); stack_location
45
46
47
stack (tbjk(obbtto_stacck abj_at_botton
def stack_obji..nn_db obbjo obj2):
50
To stack obj1 on obj2, the robot needs to be holding obj1 if not is (chooding(objt)).
51
# If the robot is not holding obji, there are 2 scenarios: # (1) if obji is in stack unstack obj1
54
(2) else, pick up obj1.
55
if (tan_a_stackoooil)
56
# Because obj1 is in stack, robot need to move then
unstack the obj from the obttat_bottom first ab aet.ob...tat......mddmaaaa......
58
obj at jj_at_botton else:
Since obj1 is not stack, robot can just then pick it up
61
move tove_then.pickkobj-oo
determine the location of obj2 to stack
lbja_iocation get_obj_location (obj2)
64
move to obj2's location then stack obj1 on obj2 move move_then_ hhr
65
obj_at_botton-obb
s trach_loaattoon-b322..aa.....
def move_to_locatiine cove_to..lcatioon(loaaaaaa] get_ get_ [et_curr__lccation(] location: move move (), location)
68
69
70
105

---

def cut_object location):
To cut object, the robot first needs to be holding obj not (holding(obo]
13
74
# If the robot is not holding obj, there are 2 scenarios: # #1) (1) if obj is in stack .nstack obj (2) (2) else, pick
15
obj.
if ie_.na..ttac
78
Because obj is in a stack, robot need to move then unstack the obj from the obj_at_botton first (opt
19
obj_at_bottor move_heee..uatooooooo......tttea
obj_at_bottom= at _tj_ttoo ott.occoooaiiorrge.ooooooooooo.o oag
81
elae:
82
# Since obj is not in stack, robot can just move then
pick it up (ockthhen.icckobooooo) move to the location cut at move toee_tooloctton (location=location)
84
85
86
cut the object
cut (obj-obj)
19.3.3 Full Helper Functions
def move www.tthennnnaaeeeeeeee unstack toooj_tt_unstae
unstack_locationn
For unstacking, need to move the location of the bottom object
get_curr_location
(), get_ get.cchh.locccooocooo..tttttttt.
unstack unstack After unstacking, need move to the unstack_locatid
obj_at_bottom]
get.curr.locationd unstack_location:
unstack,location
def move (ove_tthen.pcck(bb))
obj_tocation wtt.aoooooocaooooeet
12
get_curr obj_location:
13
nove O), obj_location) pick_up lbj_ccctioo)
14
16
def move tove__then_placeeCn place_location):
17
get_curr_location] != place_location
18
move place place_location
(), place_location
21
def cook cook_until_is. cooked
22
start start_cooking(00)
23
while not is_ ie_cooeed ()
26
def cook_object itaijicct_t__oociioooot location);
27
To cook an object, the robot first needs to be holding obj if not is is_holding (obj):
# If the robot is not holding obj, there 2 scenarios: stack unstackk obj
30
(1)
obj
31
# (2) else, pick up obj.
32
if is_in_a_stack
# Becauss obj is in stack, robot need to move then unstack the obj from the obj_at_botton first obj_at_bottor get.oo..tha.....umeerett........
34
35
oov__chen_uustcccoooooooo... @bj_at_botton unstack haw.oeoootttorrroooooooooooorooo else:
obj_at_bottom-
106

---

# Since obj is not in stack, robot can just move then pick it up
38
nove_then_ pire_tthe_icccoobjeeeeeee (obj=ob))
39
place the object at the location to cook at move then_thace (obj-obb, place_location-lccarr cook the object coo__umil_ji_.coce((((30) (obj=ob))
40
43
44
def move taareettee__ttccceee Sooe__then_staccccbttttooo.
stack_location.
45
if get_ _et_curr__ocatioon) !- stack location:
move nove(ggt_curr__ccatiio O),
stack stack obj_at_bottto 47
48
49 def bback_objl_on_obp22
obj2):
To stack obj1 on obj2, the robot needs to be holding obj1 if not is_holding (obj1):
52
# the robot is not holding obji, there are 2 scenarios: # (1) if obj1 is in a stack unstack obj1
53
54
# (2) else, pick
obj1.
if as_inna_stacc
56
Because obj1 in stack, robot need to move then unstack the obj from the obj_at_bottoon first abj_at_botton = trt.obb,that.BBBuddeeettttt atb top=obi
57
move_then cow.
obj_at_botton
obj _tj_a_bbotton ott.eeeeeeettrrrooooooooooooooa.
59
else:
60
# Since obj1 is not in stack, robot can just move then
pick it up (ove_
62
determine the location of obj2 to stack on (obj))
63
obj2_location = get_obj__ocation (obj2) move to obj2's location then stack obj1 obj2
move wtw.eeoo...cccccc....oootaaaaaaa stack_locattoor..b]..locaaa.
obj_at_botton-obj..
66
67
def cut wnm.until., until_is_cuuttt (obj): while not cut (obj):
cubj))))
20
71
def cut_objer ot (obj, location):
wwl_loocaionnooo # To cut an object, the robot first needs to be holding obj if not is_holding (obj):
74
the robot is not holding obj, there are scenarios:
15
# (1) if obj is in a stack unstack obj # (2) else, pick
76
obj.
if is_inna.sttacx
78
Because obj is in stack, robot need then unstack the obj from the obj_at_botton first obj_at_botton oow....bee..mnooooooooo.ooasseee obj aotttot ott.aaaaeaaotrrrrooooo..oooo.oaa else:
79
obj_attbotoom
83
Since obj is not in stack, robot can just move then
pick it pive_then_pcccobbjoott
move_then_n
place the object the location cut
85
move www.thhheacccooieeeeeeee cut the object
place_locations placi
86
87
cut_until__ (ottil)-__cutobbo-aee
19.3.4 Only Analyze Helper Functions
107

---

def _ove_then_ uestack tobtuuttkck.. obj_at_bottom, unstack_location
For unstacking, we need to move the location of the bottom object
1f vet_curr_locationd !- get ott.00__occtiono00..tt.oooo bbttbbb novelget.curr..aattion)) 8et.00/_.aattooocooo.....tttooo (obj. After unstacking, we need to move to the unstack instack_location if get_curr_location != mmstack_location. nove[get.curr_locata
obj_at_botton)
unstack_locatinn
10 def move move (ove_then_.ickkobb)))
11
obj abj_location get_ob] get.obj_occaiin (obj)
12
if vet_cuur_locatee
!= obj_location:
(), obj.location) pick_up obj_location)
15
16
def move move_ther tt__then__ppee Sachen_place__eee (obj, place_ [lace_location):
17
1f get_curr_location !- place_location,
gove@get_cur__occattt place place_location)
place_locatioe
19
20
21 def cook_until (o___iid__oookedob_)) ctart__ooking
while not is_ (obj): ()
24
25
26 def cook_object at locatiocn (obj, location):
To cook object, the robot first needs to be holding obj not is_holding (obj):
28
29
# the robot is not holding obj, there are 2 scenarios: # (1) if obj is in a stack unstack obj # (2) else, pick
30
obj.
if as__n_
33
in a stack, robot need to move then unstack the obj from the obj_at_botton first aobj
Because obj
34
obj_at_ move_then mo (obj obj at bottom, unstack Losaccatttccc..cctooooeeeeee.e ooa. bobjaattoooob else:
=
bottom=
36
37
# Since obj is not in stack, robot can just move then
pick it up
move Love_then_pae (ice_
39
place the object at location cook at move_ phen
40
place_.ocation
41
cook the object cook_
(objeoobb)))) (obj=obj)
44
def move move_ tove__then.staccccch.tooooo stack,
stack_location)
45
if get_curr_
!- stack locatioocation:
46
move(get.cur..locati (), stack abotat_
49 def stack_objil on________b2oo_]]] obj2):
50
# To stack obj1 on obj2, the robot needs to be holding obj1 if not is (holdiggoojj])
# If the robot is not holding obji, there 2 scenarios: obj1 is # (2) else, pick up obj1.
53
# (1)
stack unstack obj1
54
55
if (tt_n_aastaccoooj]))
Because obj1 is in stack, robot need to then unstack the obj from the abj_at_botton first
57
(obj. abj_at_botton tt._ob__tat.... oot
108

---

oor. obj_at_bottom=
obj at__t_bbottt _t_ttomm ott.
59
else:
60
Since obj1 is not in a stack, robot can just move then pick it up (oee_then
determine the location of obj2 to stack obj2_location get_obj_ (obj2) move to obj2's location then stack obj1 on obj2
63
64
move move_ (tabensstaaccbbbttootttaaaa sace_locattorr-oo3..loototoo
obj_at_bottommobb22,,
66
67
def cut cut_ uit_uutil__isccut (obj): while not cut (obj):
68
cut (obj)
71
def cut_object_at__ue
(obj, location):
72
To cut an object, the robot first needs to be holding obj if not is_holding (obj):
13
# If the robot is not holding obj, there 2 scenarios: obj
15
(1)
stack ,nstack obj
76
# (2) else, pick up obj.
77
if as_
# Because obj is in stack, robot need to move then
unstack the obj from the obj_at_bottom first obj_at_botton obj at bottbottom unstack orgaepccocaatooree....ooo.oooood
19
80
obj_at_
else:
82
Since obj is not in stack, robot can just move then
pick it up
83
thee_then, (itpteeeepccpooooobppoo
# place the object at the location to cut at
phw._tteeeplaceooo cut the object
llace_locttooorllaaaiaa
86
87
cut_unte intii_is_cuu cut (obj=obj)
109


Reviews:
[{"review_id": "huh69mIcVW", "reviewer": "Reviewer_Gc7K", "summary": "This paper presents a method that can take both demo and language in and teach LLM to perform new tasks. The idea makes sense and the algorithm is easy to understand and works very well. Evaluation results and ablations show improvement over existing works.", "questions": "It looks like the method does not involve any network training / fine tuning, right?\nIf so, do you expect to get better performance by training / fine tuning LLM?", "limitations": "As the author said, the ability of this framework is limited by the capability of LLM.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the idea is clear and easy to understand.", "weaknesses": "It's probably better to define what \"task\" is. For example, the authors claim \"Demo2Code can generalize across complex, long-horizon tasks.\" However, for real world robotics tasks, I would imagine there will be a lot of corner cases and the code needs to handle it and therefore it won't generalize to those tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer is excited about the capability of our approach and finds our paper to be clear and easy to understand. We also thank the reviewer for suggestions to make our paper clearer. We would like to answer the questions and then address the concerns.\n\n### Questions\n#### **Q1: It looks like the method does not involve any network training / fine-tuning, right?**\nCorrect, we are currently doing in-context learning, where we provide the LLM example query and responses [8]. \n\nWe do expect fine-tuning and updating the weights of the LLMs to increase the performance. However, fine-tuning large models is challenging because:\n- it requires access to the parameters\n- the process is computationally intensive\n- it requires a large amount of data due to the large number of parameters LLMs have\n\nMoreover, open-source models currently do not match the code-generating abilities of GPT3.5 [9]. \n\nHence, in-context learning with GPT-3.5 allows quick reusability across different environments without needing large amounts of data and computational power. \n\n### Weakness\n#### **Clarification on tasks**\nWe define a task as a goal that contains a series of sub-goals, which correspond to sub-tasks that the robot needs to complete. For example, the make-a-burger tasks contain sub-goals on the item's status (e.g. the lettuce needs to be cut, the lettuce must be on top of the patty) and on the order of sub-tasks to execute (e.g. preparing all the ingredients before assembling the burger). \n\n#### **Clarification on generalization ability to complex, long-horizon tasks**\nIn the global response, we clarify how Demo2Code is able to generalize to tasks that are more complex and have a longer horizon than the tasks in the prompt, e.g. for Robotouille, an average of 11 states and 15 predicates at train time v.s. 32 states and 53 predicates at test time.\n\nIn addition, Demo2Code is able to solve tasks with longer horizons (up to 71 states) compared to other existing LLM-based planners [1-7], which have an average of around 20 states. These planners mostly solve tasks that require chaining multiple low-level actions. In contrast, the tasks that Demo2Code can resolve require chaining multiple high-level subtasks, which each contains multiple low-level actions with loops and conditionals. \n\n#### **Clarification on how the code handles corner cases**\nWe acknowledge the importance of ensuring that the generated policy can handle corner cases. That is why we developed Robotouille, which procedurally generates environments with different initial conditions to test our code in. This variability includes: \n- the robot starts in a new, unseen position, \n- items are in novel arrangements,\n- key items exist among other irrelevant items, etc. \n\nAs future work, we are working on integrating Demo2Code into an end-to-end robotic system that would surface more realistic corner cases.\n\n[1] Jacky Liang, et al. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.\n\n[2] Ishika Singh, et al. Progprompt: Generating situated robot task plans using large language models, 2022.\n\n[3] Jimmy Wu, et al. Tidybot: Personalized robot assistance with large language models, 2023.\n\n[4] Andy Zeng, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv:2204.00598, 2022.\n\n[5] Brian Ichter, et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022.\n\n[6] Wenlong Huang, et al. Inner monologue: Embodied reasoning through planning with language models. In arXiv:2207.05608, 2022.\n\n[7] Kevin Lin, et al. Text2motion: From natural language instructions to feasible plans, 2023.\n\n[8] Mark Chen, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.\n\n[9] Hugo Touvron, et al. Llama: Open and efficient foundation language models, 2023."}, {"review_id": "VrwpXUtwW7", "reviewer": "Reviewer_sxrs", "summary": "The authors propose an LLM-based completion framework to translate natural language instructions, in addition to transcribed state sequences of demonstrations (as PDDL (or other strips-like) predicates), into code for executing the task with a robot.\n\nThe method is based on recursively summarizing the demonstrations into a 'specification', and then recursively expanding the 'specification' into python(?) code. The recusrsion stops in the first phase when no further summarization is possible, and in the code expansion loop, when no function is undefined.\n\nThe authors demonstrate this is three different domains with upto 10 distinct high level actions occuring in each demonstration.", "questions": "To strengthen the paper the authors should carefully define the following:\n1. The role of the user/system developer. This includes definition of skill primitives that can be executed in the environment, definition of predicates that are sufficient to track the progress of the tasks. Developing perception systems that accurates record the predicate states\n2. Quantify task complexity using metrics from logic (expressing instructions in temporal logic, and measuring the size of the automaton). Using number of predicates, and actions and reporting them for each planning domain.\n3. Evaluate on diverse set of instructions taking inspiration from Menghi et al. [1] to come up with specification templates beyond sequenced visit.\n4. Report comparative performance against state-of-the-art symbolic planners. in comparably sized planning domains.", "limitations": "Please see above", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Multimodal learning from demonstrations plus language instructions is an important problem that will enable consuming a wider modality of data sources to generate robot/agent programs and behavior. Using the summarization/translation capabilities of LLMs will likely be a key component of such a system.", "weaknesses": "**Methodology**: The paper does not define what a task specification is concretely, nor does it define what it means for the demonstrations to be adequately summarized, in particular in Alg 1 the function is_summarized() is not defined, nor is its operationalization defined anywhere in the text of the paper. Specification traditionally refers to a formal statement whose semantics are well known and the satisfaction of the specification with respect to an output is computationally well-defined and consistent. By not having a concrete definition of specification, the evaluations set up in the latter part of the paper suffer from lack of diversity and a lack of quantification of task difficulty.\n\n**Code as output language**: This work relies on the output code being interpretable, and this involves providing a set of primitive parametric functions to the LLM to use. The part of sensing the environment, and acting using perception feedback is something that is already programmed into the primitives. Coming up with an adequate and a competent set of parameterized primitives is challenging, and in this case is entirely the responsibility of the system designer.\n\n**Claims of generalization** Tthe generalization shown is only generalizing towards named entity substitutions, not to complex control flows and temporal specifications. Specifically, all the tasks shown here are a sequence of subgoals where performing any subgoal out of order will not preclude the agent from completing the subsequent actions. Further, there is no reactivity in the task specifications, and there are no avoidance tasks. All of these task specifications notions are very common in robotics and planning problems that this system has not been evaluated on. Refer to [1] for a survey on robotics mission types (also relevant to symbolic planning). All the tasks here are limited to the visit or sequenced visit type. Further while the submission claims that demo2code can generalize for complex long-horizon tasks, the maximum task length is quite smaller than state of the art for symbolic planning. Automatically translating textual domain descriptions to a formal domain description followed by the use of automated planners has already shown more reliable performance on harder problems [5]. Further there is quite a bit of evidence that LLMs cannot plan beyond the simplest of domains [6], and this line of research is unacknowledged in the submission.\n\n**Issues on evaluations of learning from demonstrations:** Generalization from demonstrations and language is a tricky subject. Usually demonstrations and language contain complementary sources of information. Therefore none of the system behaviors are incorrect in Figure 5, the core issue is that inductive learning is by definition an ill-posed problem, and many approaches to inductive learning have relied on Bayesian inference in the past. [2],[3]. Specifically, the issue of where to place the purple block (fig 5a) given the language description is underspecified, and the system had to forcibly ground the placement to any of the valid options to generate a trajectory. Committing to a valid assignment as was done by the lang2code model is one approach, and asking for resolution of referential ambiguity is another approach [4]. One might argue that learning to overconstrain the output based on a single demonstration is an example of overspecification.\n\n\n\n[1] - Menghi, C., Tsigkanos, C., Pelliccione, P., Ghezzi, C. and Berger, T., 2019. Specification patterns for robotic missions. IEEE Transactions on Software Engineering, 47(10), pp.2208-2224.\n\n[2] - Tenenbaum, J.B., 1999. A Bayesian framework for concept learning (Doctoral dissertation, Massachusetts Institute of Technology).\n\n[3] - Shah, A., Kamath, P., Shah, J.A. and Li, S., 2018. Bayesian inference of temporal task specifications from demonstrations. Advances in Neural Information Processing Systems, 31.\n\n[4] - Williams, Tom, Rafael C. Núñez, Gordon Briggs, Matthias Scheutz, Kamal Premaratne, and Manohar N. Murthi. \"A dempster-shafer theoretic approach to understanding indirect speech acts.\" In Advances in Artificial Intelligence--IBERAMIA 2014: 14th Ibero-American Conference on AI, Santiago de Chile, Chile, November 24-27, 2014, Proceedings 14, pp. 141-153. Springer International Publishing, 2014.\n\n[5] - Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J. and Stone, P., 2023. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477.\n\n[6] - Valmeekam, K., Olmo, A., Sreedharan, S. and Kambhampati, S., 2022. Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback on how we can strengthen our work. We also thank the reviewer for seeing our approach's capability to summarize and handle learning from demonstrations!\n\nPlease find below our responses to the questions and concerns:\n\n## Questions\n#### **Q1:**\nWe assume that a system developer does the following:\n- Provides a perception library. This library defines a set of predicates, similar to [8-9], and maintains this set based on sensor observations. It also contains helper functions that use these predicates, e.g. get_obj_location(obj), is_cooked(obj).\n- Provides an action library. This library defines a set of actions that correspond to low-level policies, similar to [1-7].\n\nThese are common modules in a robotics stack and seem reasonable to assume. Given such a system, Demo2Code can take demonstrations from any user (not a system developer) to generate robot code that uses functions from these libraries. We list the set of low-level actions and predicate-based functions for each planning domain in Table 3 of the attached pdf.\n\n#### **Q2:**\nWe present the number of unique predicates and low-level actions for each planning domain in Table 2 of the attached pdf. We also cluster the tasks based on their type to report the demonstration length and code complexity (by code length, average number of loops, average number of conditionals, and average number of functions defined) in Table 4.\n\n#### **Q3:**\nWe clarify our rationale for our selection of tasks, which has a different motivation than Menghi et al. [10] Since we focus on learning tasks from demonstrations rather than task planning efficiency, we chose our tasks to test the following:\n- **Recovering hidden world constraints:** The tabletop tasks require identifying constraints not specified in language but implicit in the demonstrations.\n- **Inferring conditionals and control flows:** The Robotuille tasks require identifying conditionals (e.g. picking up the object if not holding it) and loops (e.g. cutting an object until it's cut, using for loops to cook multiple burgers, etc).\n- **Extract user preferences:** The Epic Kitchen tasks require identifying the user’s preference, e.g. different ways to wash dishes.  \n\n#### **Q4:**\nWe adapted our pipeline to run LLM with a symbolic planner, LLM+P [11]. Since LLM+P does not have a recursive summarization pipeline to handle long demonstrations, we provide the task specification generated by Demo2Code.\n\nWe made these observations:\n- LLM+P fails to capture user preferences if they are not observable in the goal state. Fig. 1 in the attached pdf shows a qualitative example. LLM+P produces the left plan which has a lower cost, but misses the preference that the user wants the robot to prepare all ingredients before assembling them into a burger because the order of subtasks is not captured in the goal state.\n- LLM+P needs to be called every time the initial condition changes. In contrast, Demo2Code needs to generate the code once, which generalizes for different initial conditions.\n\n## Weakness\n### Methodology\nIn our work, the task specification is described in language rather than a formal structure. We empirically found the following format to work reliably.\n- Header: 1-2 sentences that define the overall goal to ground the code generation step.\n- Body: descriptions in pseudocode format to help the LLM generate the high-level task code more easily.\n\nA demonstration is sufficiently summarized when it has been distilled down to a task specification that can “explain away the demos”, i.e. P(code | spec) = P(code | spec, demo).\n\nIn our implementation, we leverage the LLM to rely on instructions and examples in the prompt to determine whether a demo is sufficiently summarized.\n\n### Claims of generalization\nPlease refer to the global response where we clarify:\n- The difference between the train and test tasks\n- The code complexity of the different test tasks\n\nWe show that Demo2Code can generalize to tasks that have longer horizons, more states, and more control flows than examples in the prompt. Our long-horizon claims refer to the result that Demo2Code can solve longer tasks (up to 114 states) compared to existing LLM-based planners [1-7] (up to 20 states).\n\n#### **On the planning ability of LLMs**\n\nPlease refer to the \"Code vs other policy representation\" section where we clarify that we focus on using LLMs to generate robot task codes from demonstrations rather than LLMs' planning abilities. We acknowledge the importance of symbolic planners and their complementary strengths to our summarization framework.\n\n### Issues on evaluations of learning from demonstrations\nWe evaluate solutions in a manner similar to how imitation learning evaluates policies:\n- When we generate demonstrations, we also generate held-out reward functions, captured as unit tests, that check for satisfying a set of sub-goals and constraints.\n- We use these held-out unit tests to evaluate the code.\n\n[1] Jacky Liang, et al. Code as policies: Language model programs for embodied control\n\n[2] Ishika Singh, et al. Progprompt: Generating situated robot task plans using large language models\n\n[3] Jimmy Wu, et al. Tidybot: Personalized robot assistance with large language models\n\n[4] Andy Zeng, et al. Socratic models: Composing zero-shot multimodal reasoning with language\n\n[5] Brian Ichter, et al. Do as i can, not as i say: Grounding language in robotic affordances\n\n[6] Wenlong Huang, et al. Inner monologue: Embodied reasoning through planning with language models\n\n[7] Kevin Lin, et al. Text2motion: From natural language instructions to feasible plans\n\n[8] Toki Migimatsu, et al. Grounding predicates through actions\n\n[9] Kei Kase, et al. Transferable task execution from pixels through deep planning domain learning\n\n[10] Claudio Menghi, et al. Specification patterns for robotic missions\n\n[11] Bo Liu, et al. LLM+P: Empowering large language models with optimal planning proficiency"}, {"review_id": "qjTuFGQjQS", "reviewer": "Reviewer_ujZb", "summary": "This paper proposes Demo2Code, a new method for generating code given a natural language description and demonstrations of the task. Demo2Code recursively summarizes demonstrations using a language model (LM) to create a task specification. The task specification is concatenated to the description and then recursively synthesized into code using a LM. Demo2Code is shown to outperform previous SoTA on an object manipulation environment, an author-designed cooking simulator, and the EPIC Kitchens dataset. Finally, qualitative results are shown demonstrating OOD generalization, grounding, and understanding of user preferences. \n\n", "questions": "Questions:\n* Is there intuition for why the pass performance of DemoNoLang2Code is greater than the pass performance of Demo2Code on the \"Make a burger stacking lettuce atop patty immediately\" and the \"Make two burgers stacking patty atop lettuce after preparation\" tasks in Table 2?", "limitations": "In general the limitations are well-addressed. The following two limitations were also addressed as weaknesses above: \n* Demonstrations are complete descriptions of low-level actions. In practice it seems difficult to obtain complete descriptions of all actions and state of an environment, as actions are often noisy and difficult to clearly delineate. E.g., given some video of a human washing the dishes, it would be unclear how to appropriately parse the action space (in the paper it was done by hand). \n* The natural language description of the task may be misspecified by the human. This seems like a fundamental limitation of the current method, and might make the generated code incorrect or misspecified.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "*Originality:* a new long-range sequential decision-making benchmark, Robotouille, was developed that focuses more on high-level actions rather than manipulation and navigation. This may be useful for future work studying agents that learn from task specification.\n\n*Quality:* the experiments are comprehensive and cover a range of tasks and domains, suggesting that the method is robust to distributions of tasks. Furthermore, the results show a marked improvement over the previous SoTA on all the tasks, which likely demonstrates the efficacy of leveraging demonstrations.\n\n*Clarity:* the paper is well-written and the figures are self-explanatory.\n\n*Significance:* generating code from demonstrations is an important step towards developing agents that can efficiently interact with humans. Moreover, this work demonstrates a working end-to-end setup from natural language demonstrations to code, likely encouraging future work in this area.", "weaknesses": "There are two implicit assumptions of the method that are not evaluated and may make the method difficult to use in practice. I believe that these assumptions may be hard to overcome, so I am leaning towards giving a 6.5 but am rounding up because of the high-quality execution.\n\n1. *The method assumes that demonstrations are complete descriptions of each state in the trajectory.* I would imagine that in many real-world environments (which is the setting that such a method would likely be deployed in), obtaining a complete description of each state in the trajectory is noisy. Some actions may be occluded or unable to be clearly delineated. Moreover, it may not even be clear a priori how to canonically parse actions from video or a natural language description of the environment. Would it be possible to develop an ablation where some of the actions are noisily parsed or even entirely omitted from the demonstration? I suspect that GPT-3.5 may not be able to handle the perturbation, but GPT-4 may be able to.\n\n2. *The method assumes that the task description is well-specified.* Humans can often provide demonstrations and descriptions of the task that are under-specified or misspecified. For example, when booking airline tickets, a human may forget to describe their preference for red-eye flights and there may not be enough demonstrations to determine their underlying preferences [1]. In its current form, the method appears unequipped to handle such cases and it is unclear what solution it would generate. As mentioned in the work, one possible remedy is to provide feedback to the LM, but it is unclear how successful such an approach would be.\n\n[1]  Lin, J., Fried, D., Klein, D., & Dragan, A. (2022). Inferring Rewards from Language in Context. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8546–8560). Association for Computational Linguistics.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's enthusiasm for our cooking game Robotouille and how our approach shows improvement over a wide range of tasks and domains. We also appreciate the reviewer's feedback on our assumptions. We would like to address the questions and concerns raised:\n\n### Questions\n#### **Q1: Why is the unit pass rate of DemoNoLang2Code sometimes greater than Demo2Code?**\nThe two instances in Table 2 of the paper where DemoNoLang2Code performed better than Demo2Code are due to errors made by the summarization module in Demo2Code. For example, when Demo2Code summarizes the demonstrations for the \"make two burgers stacking patty atop lettuce after preparation\" task, the LLM accidentally makes a mistake and misses the high-level subtask of stacking the patty on top of the lettuce. Consequently, the task specification also misses this subtask. We are working on better verification techniques to catch such mistakes and improve code generation. \n\n### Weakness\n#### **Clarification of assuming access to state-based demonstrations**\nWe refer the reviewer to the global response where we clarified the scope. We envision that our pipeline will exist in conjunction with other perception modules that parse the environment and demonstrations into symbolic states. As a part of future work, we are integrating Demo2Code into an end-to-end robotics system. \n\n#### **Ablations where some states are noisily parsed or entirely omitted?**\nPlease refer to the appendix (section G Table 1), where we have run an ablation study with the EPIC-KITCHENS dataset. \n- We try adding noisy, distracting predicates (e.g. showing the position of additional objects in the scene) to at least two separate states in the demonstrations to confuse the LLM. \n- We find that Demo2Code's performance suffers from degradation. \n- Specifically, it originally can correctly extract 5 out of 7 users' preferences, but with noisy states, it can only correctly solve for 3 users. \n\nWe also present new noisy state ablation experiments for Robotouille in the attached pdf (Table 1).\n- Specifically, for a demonstration, we study the effect of randomly removing 10% of the predicates and the effect of randomly removing 10% of states completely. \n- We find that missing 10% of the predicates in demonstrations only slightly worsens the performance (from 0.465 to 0.42).\n- Meanwhile, because removing 10% of the states essentially removed more than 10% of the predicates, it has more greatly affected the performance (from 0.465 to 0.327).\n\nTable 1 in the attached pdf contains detailed results for individual Robotouille tasks. \n\n#### **Clarification of using language description and demonstrations as input**\nWe agree with the reviewer that language descriptions of a task can be under-specified or misspecified. We offer the following two arguments.\n1. Demonstrations may capture complementary information that language description omits. Even if demonstrations and language are both noisy, as long as one modality captures the user’s preference, Demo2Code can recover it in the specification. Using the airline ticket booking examples, if a user shows a demonstration where a red-eye flight is picked over other ones that are cheaper, our pipeline can extract that preference from the demonstrations even if the user never explicitly states that they prefer red-eye flights. \n2. In the event that both demonstrations and language miss such information, we would need feedback from the user to add what's missing. Recent works [1-4] have shown such feedback schemes are able to correct LLMs' output successfully. We should be able to extend Demo2Code to add such feedback, and we will explore this in future work. \n\n[1] Marta Skreta, et al. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, 2023.\n\n[2] Shreyas Sundara Raman, et al. Planning with large language models via corrective re-prompting, 2022.\n\n[3] Debjit Paul, et al. Refiner: Reasoning feedback on intermediate representations, 2023.\n\n[4] Noah Shinn, et al. Reflexion: Language agents with verbal reinforcement learning, 2023."}, {"review_id": "asTKDKPp7F", "reviewer": "Reviewer_325h", "summary": "This paper presents demo2code, a framework that takes as input user's language instructions as well as demonstrations, and outputs synthesized code for completing the tasks. It first iteratively summarizes given demonstrations to a compact task specification, then reasons by incorporating user preferences etc, and lastly output expanded execution code. The method is evaluated on a range of tasks, including table top manipulation, a simple cooking simulator, and real-world epic kitchen dataset.", "questions": "- the authors says the language instruction and demonstration shares a common latent space after summarizing, but i don't see any details on this. Did I miss something?\n- also see weaknesses.", "limitations": "- assumption on access to privileged low-level state and relations\n- additional annotation on real-world data\n- other limitations are discussed in the last section", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The framework is novel in that it proposes to summarize demonstrations and language instructions using LLM, which is then used for action generation via code synthesis.\n- the recursive and hierachical way of summarizing demonstrations and generating code is reasonable\n- the idea that using LLM to reason about user preferences makes sense", "weaknesses": "- Upon reading the introduction i was excited to see how the approach is able to handle both instruction and demonstration: the latter usually comes in a visual space, but then i realize the authors made a big assumption that they can query the simulator to get state-based demonstration. This assumption presents a few issues: 1) such privileged information hinders the application in realistic settings. In fact, the author had to manually densely label the epic kitchen dataset, which leads to the question of how this can be used for real-world settings 2) even in simulation, it's not straightforawrd to obtain these state information. for example, as opposed to `op-top`, relations such as `in(obj, microwave)` is hard to obtain easily. Also, binary spatial relations loses dense geometric informations. 3) if access to step-by-step low-level state is assumed, and the LLM can summarize and generate step-by-step specifications, why not directly use task and motion planning(TAMP) to solve the task?\n- how does the oracle spec2code works? does it use TAMP? if yes, what's the advantage of demo2code over it?\n- the experiment which uses epickitchen, but need additional manual annotation, loses the point of evaluating on real-world dataset", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer acknowledges our novelty in recursively summarizing demonstrations and hierarchically generating code. We would like to respond to the reviewer's helpful feedback and questions.\n\n### Questions\n#### **Q1: Do you have any detail on the latent task specification?**\nIn the paper's introduction section (line 30-31), we stated that \"while demonstrations are long and code is complex, they both share a latent task specification that the user had in mind.\" Specifically, this task specification is a detailed language description of how the task should be completed. It is latent because we assume that the user does not directly provide it as the language instruction. Please refer to our global response for more details. \n\n### Weakness\n#### **Clarification of assuming access to state-based demonstrations**\nWe clarify the focus of our work in the global response. We envision that our pipeline will exist in conjunction with other perception modules that parse the environment and demonstrations into symbolic states. We make this assumption so that we can focus on our goal of generating robot task code from demonstrations. As a part of future work, we are integrating Demo2Code into an end-to-end robotics system.\n\n#### **Clarification on code v.s. using TAMP**\nWe refer the reviewer to the global response on our justification of generating code. We chose code as a flexible closed-loop policy representation that can check conditionals, contain loops, and use existing utility functions. In contrast, TAMP produces an open loop plan that needs replanning if the environment changes. \n\nHowever, note that once Demo2Code's pipeline is used to summarize the demonstrations into a task specification, the downstream policy can be modified to other types of policy representation, e.g. PDDL planner, TAMP. \n- For example, we show the possibility of integrating Demo2Code with a symbolic planner LLM+P [1], a pipeline that uses LLMs to generate PDDL before calling a planner to solve the problem. Please refer to Figure 1 of the attached pdf for a qualitative example.  \n\n#### **Clarification of the oracle Spec2Code**\nThe oracle Spec2Code and our approach Demo2Code are different in the following way:\n- Spec2Code assumes having access to the latent task specification, which is the user's hidden information on how they want the tasks to be completed. \n- Demo2Code doesn't know the task specification. It needs to generate the specification by summarizing the input demonstrations.\n\nHowever, the second stage of Spec2Code and Demo2Code are the same: given a task specification, generating the robot task code and recursively defining any helper function. \n\nSpec2Code sets an upper bound for how good Demo2Code can be. Demo2Code attempts to generate a good enough task specification from demonstrations and language to achieve the same performance as Spec2Code.\n\n#### **What is the advantage of using Demo2Code over Spec2Code?**\nUsing Spec2Code directly is cumbersome because it requires the human to give a very detailed description of how to do a task. In contrast, Demo2Code just needs the user to demonstrate the task. Thus, Demo2Code reduces the amount of effort that users need to spend in order to teach and interact with a robot. \n\n#### **Clarification of running experiments with EPIC-KITCHENS data**\nWhile we acknowledge relying on manual annotations for EPIC-KITCHEN is limiting, we argue that the experiments we provide are still quite valuable because:\n- The EPIC-KITCHENS is a real-world dataset, analogous to how a typical user would provide demonstrations once Demo2Code is integrated into an end-to-end robotics system.\n- The data covers a wide range of unique kitchens with different kitchenware and a diverse set of users who each has a preferred way of washing dishes.\n- The manually annotated states and actions have a wide range of objects, predicates, and actions. \n\nIn our future work, we are working to integrate Demo2code with a vision language model that can automatically extract states and actions from the video data. In the meantime, our current experiments have verified that Demo2Code can extract different real-world users' unique preferences from the output of such perception models. \n\nWe also ran ablation studies where we added noisy predicates (e.g. positions of irrelevant objects in the background) that a perception module might automatically identify in real-world demonstrations. Table 1 in Section G of the appendix shows that noisy demonstrations can worsen LLM's performance. Thus, the EPIC-KITCHENS dataset emphasizes the importance of future works on using feedback to iteratively improve Demo2Code's output [2-5].\n\n[1] Bo Liu, et al. LLM+P: Empowering large language models with optimal planning proficiency, 2023.\n\n[2] Marta Skreta, et al. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, 2023.\n\n[3] Shreyas Sundara Raman, et al. Planning with large language models via corrective re-prompting, 2022.\n\n[4] Debjit Paul, et al. Refiner: Reasoning feedback on intermediate representations, 2023.\n\n[5] Noah Shinn, et al. Reflexion: Language agents with verbal reinforcement learning, 2023."}, {"review_id": "SAe7szXwuw", "reviewer": "Reviewer_yYo2", "summary": "This work presents a recursive method to summarize demonstrations into programs through LLM.  The idea is interesting in that it uses spec as the bottleneck to connect complex demonstrations and complex robot task code, encoded and decoded through chain of thoughts. The method is evaluated on three different benchmarks involving table-top manipulation, novel kitchen text tasks, and EpicKitchen. The method outperforms naive language-to-code baselines and can generalize to longer-horizon tasks as well as learning user intents.", "questions": "1. The formulation of MDP seem a bit unnecessary to me.\n2. The spec is called “latent” multiple times in the paper, but shouldn’t it be clearly defined for each task?\n3. To confirm on the evaluation metric: Is the unit-test pass rates the same as task success rates? I.e. does the generated code satisfy a set of instructions (task) defined with the demonstrations? Or is it more narrowed?\n4. What is the quality of the spec2code, and why matching it is a necessary metric? It seems to me that there are many ways to write code for each task.\n5. Is hallucination a problem? Because the method introduces such a pipeline instead of end-to-end, is it possible that the LLM introduces unnecessary steps in the instruction summary stage and then introduce more functions to complete those substeps in the expanding stage?\n6. What is the reference code in EpicKitchen dataset in line 190. And what is the user scoring process?\n7. Given that there could be multiple ways to do the breakdown and the LLM are not deterministic, is stochasty a concern? It seems that variance is not provided in the table for each experiment.\n8. Have the authors tried the new GPT3.5 model with longer-context or GPT-4 model?", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem is challenging in that demonstrations and codes are both complex: demonstrations have lots of details and multimodality, yet codes are abstract and need to follow strict requirements. \n2. The method is sound: it uses divide and conquer to tackle some limitations of the current LLM. \n3. The new benchmark can be interesting to researchers that want to attempt the high-level planning problem in kitchen tasks.\n4. The figures and the pseudo codes are helpful\n5. The method generalize to longer-horizon tasks as well as learning user intents.\n", "weaknesses": "1. Have several concerns on the evaluation metrics \n2. Need more details on discussing the tabletop benchmark, the EpicKitchen experiment and the new proposed benchmark. Looks like epic kitchen is closer to diverse raw data such as Youtube and yet the new kitchen simulator and the table-top tasks has more predicates as well as low-level relationships\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the questions which help us improve our paper's clarity. We are excited that the reviewer acknowledges how our approach solves a challenging problem and Robotouille as a promising benchmark. Please see below for our answers to the questions: \n\n### Questions\n#### **Q2: Why is \"latent task specification\" not clearly defined for each task?**\nWe wish to clarify the sentence (line 30-31): “while demonstrations are long and code is complex, they both share a latent task specification that the user had in mind.” \n\nWe use \"latent task specification\" because:\n- “Latent” evokes the analogy with an encoder-decoder architecture that encodes demonstrations with the summarizer and decodes into code with the code expansion module.\n- \"Task specification” refers to the detailed language description of how the task should be completed. \n\nThis specification is not defined for each task because we assume that the user doesn't need to provide detailed instructions on how to complete a task. Instead, Demo2Code is responsible for \"encoding\" the demonstrations and generating this latent task specification. \n\n#### **Q3: Is the unit-test pass rates the same as the execution success rate?**\nNo, they are different. \n- The execution success rate shows whether a code can be executed in the simulator without any error. \n- The unit-test pass rate is whether a code has completed the task in the same way that the user wants. The unit test checks whether the code has completed all the subgoals and satisfied all the constraints successfully.\n\nFor example:\n- **When execution success rate = 0:** The generated code has syntax error, or if it violates the physical constraint in the simulator (e.g. trying to pick up item A from table B even though table B is empty). \n- **When unit-test pass rate = 0:** Consider a task to make a burger with a patty and lettuce. Not known to our pipeline, the user wants to prepare all the ingredients before assembling the burger, and they also want the lettuce to be on top of the patty. Even if a code makes a burger without having any error in the simulator (thereby having an execution success rate of 1), if it cooks the patty then immediately adds it to the burger instead of cutting the lettuce first, or if it puts the patty on top of the lettuce, the code will fail the unit test. \n\n#### **Q4: Why is matching Spec2Code a necessary metric?**\nSpec2Code assumes access to the latent task specification, so it knows exactly how to complete the task. We agree with the reviewer that matching Spec2Code isn't a necessary metric, as there are many possible ways to solve the task while satisfying the user's hidden requirements. \n\nHowever, we empirically found this metric to be a useful proxy metric of the quality of a code without having to actually run the code.\n- A high match rate indicates that the code is likely to succeed in execution and pass the unit test. \n- However, a low match rate doesn't imply the code is wrong. Hence, we rely on the execution success rate and unit-test pass rate to evaluate the correctness of the code. \n\n#### **Q5: Is hallucination a problem?**\nFor our experiments, we didn't encounter explicit issues with hallucination although LLMs like GPT-3.5 do hallucinate at times [1]. We are able to avoid hallucination because when we show examples of how to summarize demonstrations, we make sure that the LLM would cite parts of the demonstrations that make up the summarized line. \n\nHowever, we do run into issues with the LLMs making mistakes during summarization, which consequently causes the task specification to be incorrect. \n- e.g. the LLMs can accidentally omit a high-level subtask like stacking a patty on top of the lettuce, thereby causing the specification to also miss this subtask. \n\nThis limitation shows the value of future work to add a verification and improvement step so that our approach can catch errors it makes and improve its code [2-5].\n\n#### **Q6: What is the reference code and the pass rate for the EPIC-KITCHENS dataset?**\nThe reference code for the EPIC-KITCHENS dataset is written by a human annotator after watching the demonstrations, and reviewed by another human to avoid inconsistency. Then, the LLM's generated code is compared against this reference code to compute the BLEU score (match score). \n\nThe pass rate is similar to the unit-test pass rate in the tabletop environment and Robotouille. However, because we don't have access to a simulator to run the generated code, we rely on human annotators to check if the demonstrations fit as a trace of the generated code. \n\n#### **Q7: Is stochasticity a concern?**\nIt is not a major concern because we use temperature = 0 for GPT-3.5, which keeps our generated output and the final code mostly deterministic.\n\n#### **Q8: Have the authors tried the new GPT-3.5 model with longer contexts or GPT-4 model?**\nWhile we didn't try the new GPT3.5 model with longer contexts or GPT-4, we hypothesize that the increased context length will help solve more complicated tasks with longer and larger amount of demonstrations. \n\nThe multimodal reasoning capabilities of models like GPT-4 (which came out in parallel to us running the experiments) can also be more robust to noisy demonstrations and better solve real-world tasks like the ones in the EPIC-KITCHENS's dataset. \n\n[1] Mark Chen, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.\n\n[2] Marta Skreta, et al. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, 2023.\n\n[3] Shreyas Sundara Raman, et al. Planning with large language models via corrective re-prompting, 2022.\n\n[4] Debjit Paul, et al. Refiner: Reasoning feedback on intermediate representations, 2023.\n\n[5] Noah Shinn, et al. Reflexion: Language agents with verbal reinforcement learning, 2023."}]

Author rebuttal:
We thank all reviewers for their time, energy, and helpful feedback! We are excited to see that reviewers view the problem as important and challenging *(Reviewer sxrs, ujZb, 325h, yYo2)*, find our LLM summarization framework to be novel *(325h)* and important *(sxrs, yYo2)*, and view Robotouille as a new benchmark for high-level task planning *(ujZb, yYo2)*. We are also pleased to see that reviewers find the paper well written *(Gc7K, ujZb)* and figures easy to understand *(ujZb, yYo2)*.

**Please find an attached pdf with new ablation studies, baseline comparison, and experiment details.**

### Clarification of scope
We build on prior works [1-4] that leverage LLMs to generate robot code from detailed language instructions. In contrast, Demo2Code generates robot task code from state-based demonstrations. People often cannot provide detailed language instructions and tend to underspecify. With Demo2Code, we show that even with little or no language input, we are competitive with prior works [1] that have access to detailed language instructions.

### Access to state-based demonstrations
Most robotic systems have perception modules [5-6] that can parse raw sensor data into predicate states. We think assuming access to state-based demonstrations is reasonable so that we can focus on generating code, which is nontrivial given how long demonstrations can get. As future work, we are working on integrating Demo2Code into an end-to-end robotic system.

We also conducted several ablations:
- Table 1 in Section G of the appendix shows ablation where we add noisy predicates to the EPIC-KITCHENS demonstrations. We find that the LLM's performance worsens and goes from correctly extracting 5 users' preferences to 3.
- We also conduct new studies on Robotouille where 10% of the predicates or 10% of the states are randomly removed. Missing 10% of the predicates only slightly worsens the performance (0.462 to 0.42), and missing 10% of the states (which could remove more than 10% of predicates) have a larger effect (0.462 to 0.327). More details are in Table 1 of the attached pdf.

### Why code?
We choose code as the output representation because:
- Much of high-level robot tasks are programmed as code.
- Code offers a flexible, concise way of expressing control flows, calling external libraries like perception and planning, etc.
- Code allows composability.
- Code is interpretable to engineers and verifiable through static analysis.
- LLMs have been trained on code and can generalize well in this output space [8-9].

### Code v.s. other policy representation
We don't claim our approach is an alternative to a symbolic planner. In fact, many problems such as re-arrangement tasks are more suited to be solved as a planning problem. For such cases, we can modify Demo2Code to summarize demonstrations to a task specification for a (PDDL) planner.

We experiment combining Demo2Code with LLM+P [7] and present a qualitative example in Figure 1 of the attached pdf. We find that while this combination produces valid plans:
- LLM+P fails to capture user preferences that aren’t observable in the final state.
- LLM+P needs to be run for every new initial condition, while the code generalizes to different environments.

Still, both code and PDDL are valid ways to represent a policy, and they can even co-exist, e.g. the code can call a planner. However, for the reasons stated in "Why code?", we chose to generate code.

### Clarification of generalization results
We clarify how Demo2Code can generalize and solve unseen complex tasks with longer horizons and more predicates compared to examples in the prompt at train time.

For Robotouille, the tasks at train time are significantly different from the ones at test time:
- Mean horizon length: 11 states (train) v.s. 32 states (test)
- Mean number of predicates: 15 predicates (train) v.s. 53 predicates (test)

Compared to baselines, Demo2Code performs the best for long burger-making tasks (an average of 26 states and 43 predicates) even though the prompt doesn't show this type of task. Our pipeline also shows its ability to identify control loops, e.g. solving a task to make two burgers with patties, lettuce, and tomatoes (which has 71 states and 115 predicates). Demo2Code used a for-loop to make two burgers, generalized to unseen subtasks (e.g. cut tomatoes), and composed 7 distinct subtasks together to make one burger.

We also test the code against various initial conditions procedurally generated by the Robotouille simulator, where the robot starts in a new position, items are in novel arrangements, etc. These environments check the code against corner cases, e.g. when a key item is in a stack, the robot should unstack the item instead of directly picking it up.

### Clarification of the definition of "latent task specification"
We wish to clarify the sentence: “Our key observation is that the input (demonstrations) and the output (code) share a latent task specification”. We use the term “latent” to evoke the analogy with an encoder-decoder architecture (encoder being the summarizer and decoder being the code expansion). "Task specification” refers to the detailed language description of how the task should be completed.

[1] Jacky Liang, et al. Code as policies: Language model programs for embodied control

[2] Ishika Singh, et al. Progprompt: Generating situated robot task plans using large language models

[3] Jimmy Wu, et al. Tidybot: Personalized robot assistance with large language models

[4] Andy Zeng, et al. Socratic models: Composing zero-shot multimodal reasoning with language

[5] Toki Migimatsu, et al. Grounding predicates through actions

[6] Kei Kase, et al. Transferable task execution from pixels through deep planning domain learning

[7] Bo Liu, et al. LLM+P: Empowering large language models with optimal planning proficiency

[8] Mark Chen, et al. Evaluating large language models trained on code

[9] OpenAI. Gpt-4 technical report

Meta-review: The paper presents Demo2Code, a method to generate robot task code from natural language instructions and demonstrations. It recursively summarizes demonstrations into a "task specification" using a language model, then expands the specification into executable code.

Reviewers found the problem important and challenging, and recommended acceptance across the board. They praised the novelty of the hierarchical summarization approach and the new cooking environment benchmark Robotouille. However, some questioned the assumptions of having full state access and well-specified language instructions.

In their rebuttal, the authors clarified the scope - the focus of the paper is on code generation given state-based demos, leaving state estimation to other modules, which I believe is a reasonable assumption given a typical robotics stack contains focused state estimation algorithms. Furthermore, the expectation is that code is a flexible policy representation for robot tasks. They added ablation studies showing performance degrades with noisy state inputs. For underspecified language, they proposed integrating user feedback loops to refine the generated code, which is similar to the conversational ability of mainstream chat-LLMs. 

Overall, the authors appear to have satisfactorily addressed major reviewer concerns, with some differences in opinion around novelty and assumptions. The experiments demonstrate solid results on multiple domains. I would recommend acceptance given the authors expand on the scope and evaluation details in the camera ready version.
Decision: Accept (poster)