Title: Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Model

Code Generation

Robotics
Abstract: Language instructions and demonstrations are two natural ways for users to teach robots personalized tasks. Recent progress in Large Language Models (LLMs) has shown impressive performance in translating language instructions into code for robotic tasks. However, translating demonstrations into task code continues to be a challenge due to the length and complexity of both demonstrations and code, making learning a direct mapping intractable. This paper presents Demo2Code, a novel framework that generates robot task code from demonstrations via an extended chain-of-thought and defines a common latent specification to connect the two. Our framework employs a robust two-stage process: (1) a recursive summarization technique that condenses demonstrations into concise specifications, and (2) a code synthesis approach that expands each function recursively from the generated specifications. We conduct extensive evaluation on various robot task benchmarks, including a novel game benchmark Robotouille, designed to simulate diverse cooking tasks in a kitchen environment.

Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought

Abstract

Language instructions and demonstrations are two natural ways for users to teach robots personalized tasks. Recent progress in Large Language Models (LLMs) has shown impressive performance in translating language instructions into code for robotic tasks. However, translating demonstrations into task code continues to be a challenge due to the length and complexity of both demonstrations and code, making learning a direct mapping intractable. This paper presents Demo2Code , a novel framework that generates robot task code from demonstrations via an extended chain-of-thought and defines a common latent specification to connect the two. Our framework employs a robust two-stage process: (1) a recursive summarization technique that condenses demonstrations into concise specifications, and (2) a code synthesis approach that expands each function recursively from the generated specifications. We conduct extensive evaluation on various robot task benchmarks, including a novel game benchmark Robotouille , designed to simulate diverse cooking tasks in a kitchen environment. The project's website is at https://portal-cornell.github.io/demo2code/

1 Introduction

How do we program home robots to perform a wide variety of personalized everyday tasks? Robots must learn such tasks online, through natural interactions with the end user. A user typically communicates a task through a combination of language instructions and demonstrations. This paper addresses the problem of learning robot task code from those two inputs. For instance, in Fig. 1, the user teaches the robot how they prefer to make a burger through both language instructions, such as 'make a burger', and demonstrations, which shows the order in which the ingredients are used.

Recent works [24, 23, 33, 80, 61, 35] have shown that Large Language Models (LLMs) are highly effective in using language instructions as prompts to plan robot tasks. However, extending LLMs to take demonstrations as input presents two fundamental challenges. The first challenge comes from demonstrations for long-horizon tasks. Naively concatenating and including all demonstrations in the LLM's prompt would easily exhaust the model's context length. The second challenge is that code for long-horizon robot tasks can be complex and require control flow. It also needs to check for physics constraints that a robot may have and be able to call custom perception and action libraries. Directly generating such code in a single step is error-prone.

Our key insight is that while demonstrations are long and code is complex, they both share a latent task specification that the user had in mind. This task specification is a detailed language

def main():

...

cook_obj_at_loc

stove)

...

stack_objects(top_bun,

lettuce)

'Make a burger'

Make a burger.

…..

State 5:

'robot1' is not holding

'patty1'

'patty1' is at 'stove1'

…..

State 6:

'patty1' is cooked

…..

State 11:

'robot1' is not holding

'patty1'

'patty1' is on top of

'bottom_bun1'

……

[Demonstration 1]

…

[Demonstration N]

Recursive Summarization

→

Spec)

Recursive Expansion

(High-Level Code

→

De

fi

ne Helper Functions)

def

cook_obj_at_loc

(obj,

if not is_holding(obj):

...

move_then_place

(obj, loc)

cook_until_is_cooked(obj)

def

move_then_place

(obj,

if curr_loc != loc:

move(curr_loc, loc)

place(obj, loc)

...

def main():

...

Language +

Demonstrations

Make a burger.

Speci

fi

cally:

…

Cook a patty at a stove.

…

Stack a top bun on that lettuce.

Lang + Spec

→

High-Level Code

description of how the task should be completed. It is latent because the end user might not provide all the details about the desired task via natural language. We build an extended chain-of-thought [73] that recursively summarizes demonstrations to a compact specification, maps it to high-level code, and recursively expands the code by defining all the helper functions. Each step in the chain is small and easy for the LLM to process.

Wepropose a novel framework, Demo2Code , that generates robot task code from language instructions and demonstrations through a two-stage process (Fig. 1). (1) Summarizing demonstrations to task specifications: Recursive summarization first works on each demonstration individually. Once all demonstrations are compactly represented, they are then jointly summarized in the final step as the task specification. This approach helps prevent each step from exceeding the LLM's maximum context length. (2) Synthesizing code from the task specification: Given a task specification, the LLM first generates high-level task code that can call undefined functions. It then recursively expands each undefined function until eventually terminating with only calls to the existing APIs imported from the robot's low-level action and perception libraries. These existing libraries also encourage the LLM to write reusable, composable code.

Our key contributions are:

Amethod that first recursively summarizes demonstrations to a specification and then recursively expands specification to robot code via an extended chain-of-thought prompt.

A novel game simulator, Robotouille , designed to generate cooking tasks that are complex, long-horizon, and involve diverse food items, for benchmarking task code generation.

Comparisons against a range of baselines, including prior state of the art [33], on a manipulation benchmark, Robotouille , as well as a real-world human activity dataset.

2 Related Work

Controlling robots from natural language has a rich history [74, 66, 37], primarily because it provides a natural means for humans to interact with robots [5, 30]. Recent work on this topic can be categorized as semantic parsing [39, 30, 69, 67, 55, 40, 68], planning [60, 22, 23, 24, 61, 35, 34, 28], task specification [64, 19, 58, 12], reward learning [46, 56, 7], learning low-level policies [46, 2, 57, 56, 7], imitation learning [25, 38, 58, 64] and reinforcement learning [26, 18, 10, 45, 1]. However, these approaches fall in one of two categories: generating open-loop action sequences, or learning closedloop, but short-horizon, policies. In contrast, we look to generate task code , which is promising in solving long-horizon tasks with control flows. The generated code also presents an interpretable way to control robots while maintaining the ability to generalize by composing existing functions.

Synthesizing code from language too has a rich history. Machine learning approaches offer powerful techniques for program synthesis [49, 4, 14]. More recently, these tasks are extended to generalpurpose programming languages [79, 78, 8], and program specifications are fully described in natural English text [21, 3, 51]. Pretrained language models have shown great promise in code generation by exploiting the contextual representations learned from massive data of codes and texts [16, 11, 72, 71, 9, 47]. These models can be trained on non-MLE objectives [20], such as RL [32] to pass unit tests. Alternatively, models can also be improved through prompting methods

such as Least-to-Most [82], Think-Step-by-Step [29] or Chain-of-Thought [73], which we leverage in our approach. Closest to our approach is CodeAsPolicies [33], that translates language to robot code. We build on it to address the more challenging problem of going from few demonstrations to code.

We broadly view our approach as inverting the output of code. This is closely related to inverse graphics , where the goal is to generate code that has produced a given image or 3D model [76, 36, 15, 70, 17]. Similar to our approach [65] trains an LSTM model that takes as input multiple demonstrations, compresses it to a latent vector and decodes it to domain specific code. Instead of training custom models to generate custom code, we leverage pre-trained LLMs that can generalize much more broadly, and generate more complex Python code, even create new functions. Closest to our approach [77] uses pre-trained LLMs to summarize demonstrations as rules in one step before generating code that creates a sequence of pick-then-place and pick-then-toss actions. However, they show results on short-horizon tasks with small number of primitive actions. We look at more complex, long-horizon robot tasks, where demonstrations cannot be summarized in one step. We draw inspiration from [75, 50, 43] to recursively summarize demonstrations until they are compact.

3 Problem Formulation

We look at the concrete setting where a robot must perform a set of everyday tasks in a home, like cooking recipes or washing dishes, although our approach can be easily extended to other settings. We formalize such tasks as a Markov Decision Process (MDP), < S , A , T , R > , defined below:

State ( s ∈ S ) is the set of all objects in the scene and their propositions, e.g. open(obj) (' obj is open"), on-top(obj1, obj2) (' obj1 is on top of obj2 ").

Action ( a ∈ A ) is a primitive action, e.g. pick(obj) ('pick up obj "), place(obj, loc) ('place obj on loc "), move(loc1, loc2) ('move from loc1 to loc2 ").

Transition function ( T ( . | s, a ) ) specifies how objects states and agent changes upon executing an action. The transition is stochastic due to hidden states, e.g. cut('lettuce') must be called a variable number of times till the state changes to is_cut('lettuce') .

Reward function ( r ( s, a ) ) defines the task, i.e. the subgoals that the robot must visit and constraints that must not be violated.

We assume access to state-based demonstrations because most robotics system have perception modules that can parse raw sensor data into predicate states [42, 27]. We also assume that a system engineer provides a perception library and an action library. The perception library uses sensor observations to maintain a set of state predicates and provides helper functions that use these predicates (e.g. get_obj_location(obj) , is_cooked(obj) ). Meanwhile, the action library defines a set of actions that correspond to low-level policies, similar to [33, 61, 77, 80].

The goal is to learn a policy π θ that maximizes cumulative reward J ( π θ ) = E π θ [ ∑ T t =1 [ r ( s t , a t )] ] , θ being the parameters of the policy. We choose to represent the policy as code θ for a number of reasons: code is interpretable, composable, and verifiable.

In this setting, the reward function r ( s, a ) is not explicit, but implicit in the task specification that the user has in mind. Unlike typical Reinforcement Learning (RL), where the reward function is hand designed, it is impractical to expect everyday users to program such reward functions for every new task that they want to teach their robots. Instead, they are likely to communicate tasks through natural means of interaction such as language instructions l (e.g. 'Make a burger'). We can either use a model to generate reward r ( s, a ) from l [31] or directly generate the optimal code θ [33].

However, language instructions l from everyday users can be challenging to map to precise robot instructions [63, 44, 81]: they may be difficult to ground, may lack specificity, and may not capture users' intrinsic preferences or hidden constraints of the world. For example, the user may forget to specify how they wanted their burger done, what toppings they preferred, etc. Providing such level of detail through language every time is taxing. A more scalable solution is to pair the language instruction l with demonstrations D = { s 1 , s 2 , . . . , s T } of the user doing the task. The state at time-step t only contains the propositions that have changed from t -1 to t . Embedded in the states are specific details of how the user wants a task done.

Our goal is to infer the most likely code given both the language and the demonstrations: arg max θ P ( θ | l, D ) . For a long-horizon task like cooking, each demonstration can become long. Naively concatenating all demonstrations together to query the LLM can either exhaust the model's

context length or make directly generating the code challenging. We propose an approach that overcomes these challenges.

4 Approach

Wepresent a framework, Demo2Code , that takes both language instructions and a set of demonstrations from a user as input to generate robot code. The key insight is that while both input and output can be quite long, they share a latent, compact specification of the task that the user had in mind. Specifically, the task specification is a detailed language description of how the task should be completed. Since our goal is to generate code, its structure is similar to a pseudocode that specifies the desired code behavior. The specification is latent because we assume that users do not explicitly define the task specification and do not provide detailed language instructions on how to complete the task.

Our approach constructs an extended chain-of-thought that connects the users' demonstrations to a latent task specification, and then connects the generated specification to the code. Each step is small and easy for the LLM to process. Algorithm 1 describes our overall approach, which contains two main stages. Stage 1 recursively summarizes demonstrations down to a specification. The specification and language instruction is then converted to a high-level code with new, undefined functions. Stage 2 recursively expands this code, defining more functions along the way.

4.1 Stage 1: Recursively Summarize Demonstrations to Specifications

The goal of this stage is to summarize the set of demonstrations provided by the user into a compact specification (refer to summarize(demos) in Algorithm 1). Each demonstration is first independently summarized until the LLM determines that the demonstration can no longer be compressed, then the summaries are concatenated and summarized together. Fig. 2 shows example interim outputs during this stage. First, states in each demonstration get summarized into low-level actions (e.g. 'patty6 is cooked" is summarized as 'robot1 cooked patty6.') Then, low-level actions across time-steps are summarized into high-level subtasks, such as stacking, cutting, (e.g. 'At state 3-8, the high level subtask is cook...'). The LLM determines to stop recursively summarizing after the entire demonstration gets converted to high-level subtasks, but this can have a different stopping condition (e.g. setting a maximum step) for task settings different than Fig. 2's. Next, these demonstrations' summaries are concatenated together for the LLM to generate the task specification. The LLM is prompted to first perform some intermediate reasoning to extract details on personal preferences, possible control loops, etc. For instance, the LLM aggregates high-level subtasks into an ordered list, which empirically helps the model to identify repeated subsets in that list and reason about control loops. An example final specification is shown in Fig. 2, which restates the language instruction first, then states "Specifically: .." followed by a more detailed instruction of the task.

4.2 Stage 2: Recursively Expand Specification to Task Code

The goal of this stage is to use the generated specification from stage 1 to define all the code required for the task (see expand_code(code) in Algorithm 1). The LLM is prompted to first generate highlevel code that calls functions that may be undefined. Subsequently, each of these undefined functions in the code is recursively expanded. Fig. 3 shows an example process of the code generation pipeline. The input is the specification formatted as a docstring. We import custom robot perception and control libraries for the LLM and also show examples of how to use such libraries in the prompt. The LLM first generates a high-level code, that can contain new functions , e.g. cook_obj_at_loc , that it has not seen in the prompt or import statements before. It expands this code by calling additional functions (e.g. move_then_pick ), which it defines in the next recursive step. The LLM eventually reaches the base case when it only uses imported APIs to define a function (e.g. move_then_pick ).

5 Experiments

5.1 Experimental Setup

Baselines and Metrics We compare our approach Demo2Code against prior work, CodeAsPolicies [33], which we call Lang2Code . This generates code only from language instruction. We also compare against DemoNoLang2Code that generates code from demonstrations without a language instruction, which is achieved by modifying the LLM prompts to redact the language. Finally, we also compare to an oracle Spec2Code , which generates task code from detailed specifications on how to complete a task. We use gpt-3.5-turbo-16k for all experiments with temperature 0.

We evaluate the different methods across three metrics. Execution Success Rate is the average 0/1 success of whether the generated code can run without throwing an error. It is independent from whether the goal was actually accomplished. Unit Test Pass Rate is based on checking whether all subgoals are achieved and all constraints are satisfied. The unit test module checks by examining the state transitions created from executing the generated code. Code BLEU score is the BLEU

score [48] between a method's generated code and the oracle Spec2Code 's generated code. We tokenize each code by the spaces, quotations, and new lines.

Tabletop Manipulation Simulator [80, 23] We build upon a physics simulator from [80, 23], which simulates a robot arm manipulating blocks and cylinders in different configurations. The task objectives are to place objects at specific locations or stack objects on top of each other. The LLM has access to action primitives (e.g. pick and place) and perception modules (e.g. to get all the objects in the scene). We create a range of tasks that vary in complexity and specificity, use the oracle Spec2Code to generate reference code, and execute that code to get demonstrations for other methods. For each task, we test the generated code for 10 random initial conditions of objects.

Task

Lang2Code

[33]

DemoNoLang2Code

Demo2Code

(

ours

)

Horizon

Exec.

Pass.

BLEU.

Exec.

Pass.

BLEU.

Exec.

Pass.

BLEU.

Length

1.00

1.00

0.90

1.00

1.00

0.90

1.00

1.00

0.90

8.0

Cook two patties

0.80

0.80

0.92

0.80

0.80

0.92

0.80

0.80

0.92

16.0

Stack a top bun on top of a cut lettuce on top of a bottom bun

1.00

1.00

0.70

0.00

0.00

0.75

1.00

1.00

0.60

14.0

Cut a lettuce

1.00

1.00

0.87

0.00

0.00

0.76

1.00

1.00

0.87

7.0

Cut two lettuces

0.80

0.80

0.92

0.00

0.00

0.72

0.80

0.80

0.92

14.0

Cook first then cut

1.00

1.00

0.88

1.00

1.00

0.88

1.00

1.00

0.88

14.0

Cut first then cook

1.00

1.00

0.88

0.00

0.00

0.82

1.00

1.00

0.88

15.0

Assemble two burgers one by one

0.00

0.00

0.34

1.00

1.00

0.77

1.00

1.00

0.76

15.0

Assemble two burgers in parallel

0.00

0.00

0.25

1.00

1.00

0.51

0.00

0.00

0.71

15.0

Make a cheese burger

1.00

0.00

0.24

1.00

1.00

0.69

1.00

1.00

0.69

18.0

Make a chicken burger

0.00

0.00

0.57

0.00

0.00

0.64

0.90

0.90

0.69

25.0

1.00

0.00

0.74

0.20

0.00

0.71

0.00

0.00

0.71

24.5

0.00

0.00

0.74

0.20

0.00

0.71

1.00

1.00

0.74

25.0

Make a burger stacking lettuce atop patty after preparation

1.00

0.00

0.67

0.10

0.00

0.65

0.00

0.00

0.66

26.5

1.00

0.00

0.67

0.00

0.00

0.53

1.00

0.00

0.69

27.0

Make a lettuce tomato burger

0.00

0.00

0.13

1.00

1.00

0.85

1.00

0.00

0.66

34.0

Make two cheese burgers

0.00

0.00

0.63

1.00

1.00

0.68

1.00

1.00

0.68

38.0

Make two chicken burgers

0.00

0.00

0.52

0.00

0.00

0.68

1.00

0.00

0.56

50.0

0.80

0.00

0.66

0.80

1.00

0.69

0.00

0.00

0.66

50.0

0.80

0.00

0.67

1.00

0.00

0.48

1.00

1.00

0.73

50.0

Make two burgers stacking lettuce atop patty after preparation

0.80

0.00

0.66

0.60

0.00

0.66

0.80

0.00

0.67

54.0

Make two burgers stacking patty atop lettuce after preparation

0.80

0.00

0.67

0.50

0.00

0.71

0.80

0.00

0.68

54.0

Make two lettuce tomato burgers

1.00

0.00

0.55

0.00

0.00

0.70

1.00

1.00

0.84

70.0

Overall

0.64

0.29

0.64

0.49

0.38

0.71

0.79

0.59

0.74

28.8

Assemble 2 burgers

Make a burger

Make 2 burgers

Cooking Task Simulator: Robotouille 1 We introduce a novel, open-source simulator to simulate complex, long-horizon cooking tasks for a robot, e.g. making a burger by cutting lettuces and cooking patties. Unlike existing simulators that focus on simulating physics or sensors, Robotouille focuses on high level task planning and abstracts away other details. We build on a standard backend, PDDLGym [59], with a user-friendly game as the front end to easily collect demonstrations. For the experiment, we create a set of tasks, where each is associated with a set of preferences (e.g. what a user wants in the burger, how the user wants the burger cooked). For each task and each associated preference, we procedurally generate 10 scenarios.

EPIC-Kitchens Dataset [13] EPIC-Kitchens is a real-world, egocentric video dataset of users doing tasks in their kitchen. We use this to test if Demo2Code can infer users' preferences from real videos, with the hopes of eventually applying our approach to teach a real robot personalized tasks. We focus on dish washing as we found preferences in it easy to qualify. While each video has annotations of low-level actions, these labels are insufficient for describing the tasks. Hence, we choose 7 videos of 4 humans washing dishes and annotate each demonstration with dense state information. We compare the code generated by Lang2Code , DemoNoLang2Code and Demo2Code on whether it satisfies the annotated preference and how well it matches against the reference code.

5.2 Results and Analysis

Overall, Demo2Code has the closest performance to the oracle ( Spec2Code ). Specifically, our approach has the highest unit test pass rates in all three benchmarks, as well as the highest execution success in Robotouille (table 2) and EPIC-Kitchens (table 3). Meanwhile, Lang2Code [33] has a higher overall execution success than Demo2Code for the Tabletop simulator (table 1). However, Lang2Code has the lowest unit test pass rate among all baselines because it cannot fully extract users' specifications without demonstrations. DemoNoLang2Code has a relatively higher pass rate, but it sacrifices execution success because it is difficult to output plausible code without context from language. We provide prompts, detailed results and ablations in the Appendix. 2 We now ask a series of questions of the results to characterize the performance difference between the approaches.

How well does Demo2Code generalize to unseen objects and tasks? Demo2Code exhibits its generalization ability in three axes. First, Demo2Code generalizes and solves unseen tasks with longer horizons and more predicates compared to examples in the prompt at train time. For Robotouille , table 2 shows the average horizon length for each training task (highlighted in gray) and testing task. Overall, the training tasks have an average of 12.7 states compared the testing tasks (31.3 states). Compared to the baselines, Demo2Code performs the best for long burger-making tasks (an average of 32 states) even though the prompt does not show this type of task. Second, Demo2Code uses control flow, defines hierarchical code, and composes multiple subtasks together to solve these long-horizon tasks. The appendix details the average number of loops, conditionals, and helper functions that Demo2Code generates for tabletop simulator (in section 8.3) and Robotouille (in section 9.3). Notably, Demo2Code generates code that uses a for-loop for the longest task (making two lettuce tomato burgers with 70 states), which requires generalizing to unseen subtasks (e.g. cutting tomatoes) and composing 7 distinct subtasks. Third, Demo2Code solves tasks that contain unseen objects or a different number of objects compared to the training tasks in the prompt. For Robotouille , the prompt only contains examples of preparing burger patties and lettuce, but Demo2Code still has the highest unit test pass rate for making burgers with unseen ingredients: cheese, chicken, and

"""

Initial State (State 1):

'green block' is not to the right of 'purple cylinder'

'purple cylinder' has not moved

'purple cylinder' is not to the left of 'green block'

[Scenario 1]

State 2:

'green block' is to the right of 'purple cylinder'

'purple cylinder' has moved

'purple cylinder' is to the left of 'green block'

"""

"""

Place the purple cylinder next to the green block.

Initial State (State 1):

'green block' is not to the right of 'purple cylinder'

'purple cylinder' has not moved

'purple cylinder' is not to the left of 'green block'

[Scenario 1]

State 2:

'green block' is to the right of 'purple cylinder'

'purple cylinder' has moved

'purple cylinder' is to the left of 'green block'

"""

loc_pos = parse_position(

'right of the green block')

put_first_on_second('purple

cylinder', loc_pos)

"""

Place the purple cylinder next to the green

block.

"""

loc_pos = parse_position(

'right of the purple cylinder')

put_first_on_second('green block',

loc_pos)

loc_pos = parse_position(

'left of the green block')

put_first_on_second('purple

cylinder', loc_pos)

Lang2Code

Demo2Code

DemoNoLang2Code

def main():

...

cook_obj_at_loc(patty, stoves[0])

...

lettuce = lettuces[0]

cut_obj_at_loc(lettuce, boards[0])

stack_obj1_on_obj2(lettuce, patty)

...

stack_obj1_on_obj2(top_bun, lettuce)

def main():

...

cook_obj_at_loc(patty, stoves[0])

...

cheese = cheeses[0]

stack_obj1_on_obj2(cheese, patty)

...

stack_obj1_on_obj2(top_bun, cheese)

robot is at 'table2',…

'robot1' is holding 'patty1',…

Make a burger.

...

Decide a patty to cook.

Cook that patty at that stove.

...

Decide a lettuce to cut.

Cut that lettuce on that

cutting board.

Stack that lettuce on that

...

Stack that top bun on that lettuce.

Make a burger.

...

Decide a patty to cook.

Cook that patty at that stove.

...

Decide a cheese to use.

Stack that cheese on that

...

Stack that top bun on that cheese.

User 1: Prefers lettuce on patty

User 2: Prefers cheese on patty

`patty 1' is cooked,…

'patty1' is on top of

'bottom_bun1',…

'robot1' is holding 'lettuce1',…

'lettuce1' is on top of

`patty1',…

'top_bun1' is on top of

'lettuce1',…

robot is at 'table1',…

'robot1' is holding 'patty1',…

'patty1' is cooked

'patty1' is on top of

'bottom_bun1,…

'robot1' is holding 'cheese1'

'cheese1' is on top of

'patty1',…

'top_bun1' is on top of

'cheese1',…

Pick up cheese1

Stack cheese1

Pick up patty1

Stack patty1

Pick up patty1

Stack patty1

Pick up lettuce1

Stack lettuce1

Stack top_bun1

Stack top_bun1

tomatoes. Similarly, for tabletop (table 1), although the prompt only contains block-stacking tasks, our approach maintains high performance for cylinder-stacking tasks.

Is Demo2Code able to ground its tasks using demonstrations? Language instructions sometimes cannot ground the tasks with specific execution details. Since demonstrations provide richer information about the task and the world, we evaluate whether Demo2Code can utilize them to extract details. Tasks under the 'Specific" cluster in Table 1 show cases when the LLM needs to use demonstrations to ground the desired goal. Fig. 4 illustrates that although the language instruction ('Place the purple cylinder next to the green block') does not ground the desired spatial relationship between the two objects, our approach is able to infer the desired specification ('to the left"). In contrast, Lang2Code can only randomly guess a spatial relationship, while DemoNoLang2Code can determine the relative position, but it moved the green block because it does not have language instruction to ground the overall task. Similarly, tasks under the 'Hidden" cluster in Table 1 show how Demo2Code outperforms others in inferring hidden constraints (e.g the maximum height of a stack) to ground its tasks.

Is Demo2Code able to capture individual user preference? As a pipeline for users to teach robots personalized tasks, Demo2Code is evaluated on its ability to extract a user's preference. Table 3 shows that our approach performs better than Lang2Code in generating code that matches each EPIC-Kitchens user's dish washing preference, without overfitting to the demonstration like in DemoNoLang2Code . Because we do not have a simulator that completely matches the dataset, human annotators have to manually inspect the code. The code passes the inspection if it has correct syntax, does not violate any physical constraints (e.g. does not rinse a dish without turning on the tap), and matches the user's dish-washing preference. Qualitatively, Fig. 6 shows that our approach is able to extract the specification and generate the correct code respectively for user 22, who prefers to soap

place('bowl_1')

rinse('bowl_1')

ison(`tap_1`);inhand(`bowl_1`);

soapy(`bowl_1`); …

inhand(`bowl_1`); clean(`bowl_1`); …

Wash objects at the sink.

...

Get a list of all objects to wash

Pick up scrub_1

For each object in all objects:

soap object

Move object from sink_1 to sink_2

Turn on tap_1

For each object in all objects:

Rinse object

Place object in dishrack_1

Turn off tap_1

objs = get_all_objs()

for obj in objs:

pick_up(obj)

go_to('sink_1")

turn_on('tap_1")

soap(obj)

rinse(obj)

turn_off("tap_1")

place(obj, "counter_1")

objs = get_all_objs()

pick_up("scrub_1")

for obj in objs:

soap(obj)

place(obj, "sink_2")

turn_on("tap_1")

for obj in objs:

rinse(obj)

place(obj, "dishrack_1")

turn_off("tap_1")

Wash objects at the sink.

...

Get a list of all objects to wash

For each object in all objects:

Pick up object

Move from current location to sink_1

Turn on tap_1

Soap object

Rinse object

Turn off tap_1

Place object on counter_1

User 22: Prefers to

fi

rst soap all objects and then rinse

User 30: Prefers to soap and rinse each object

soap('bowl_1')

inhand(`mug_1`);

isdirty(`mug_1`)

; …

rinse('mug_1')

soap('mug_1')

ison(`tap_1`); inhand(`mug_1`);

soapy(`mug_1`); …

clean(`mug_1`); at(`countertop_1`); …

soap('glass_1')

pickup('jug_1')

rinse('jug_1')

place('mug_1')

place('jug_1')

at(`countertop_1`);on(`jug_1`,

countertop_1`); …

ison(`tap_1`); inhand(`jug_1`);

isnotdirty(`jug_1`); …

clean(`jug_1`);at(`countertop_1`); …

inhand(`bowl_1`); isdirty(`bowl_1`); …

inhand(`glass_1`); isdirty(`glass_1`); …

place('glass_1') …

rinse('glass_1')

inhand(`glass_1`); soapy(`glass_1`); …

inhand(`glass_1`);clean(`glass_1`); …

Task

original

predicates, 10% drop prob

states, 10% drop prob

1.00

0.81 (0.24)

0.94 (0.13)

Cook two patties / Cut two lettuces

0.80

0.75 (0.17)

0.73 (0.22)

Assemble two burgers

0.50

0.38 (0.25)

0.50 (0.00)

Make a burger

0.49

0.65 (0.07)

0.38 (0.31)

Make two burgers

0.43

0.49 (0.13)

0.59 (0.14))

Overall

0.59

0.62 (0.05)

0.59 (0.12)

Short

Medium

Long

0.90

0.86

0.87

0.80

Rate

0.8

0.75

0.68

0.60

Pass

0.6

Test

0.43

0.4

Unit

0.19

0.2

0.03

0.00

0.06

0.0

No-COT 1-Step

2-Steps

Full

all objects before rinsing them, and user 30, who prefers to soap then rinse each object individually. Similarly, Fig. 5 provides an example of how Demo2Code is able to identify a user's preference of using cheese vs lettuce even when the language instruction is just 'make a burger." Quantitatively, Table 2 shows more examples of our approach identifying a user's preference in cooking order, ingredient choice, etc, while Table 1 also shows our approach performing well in tabletop tasks.

How does chain-of-thought compare to directly generating code from demonstrations? To evaluate the importance of our extended chain-of-thought pipeline, we conduct ablation by varying the length of the chain on three clusters of tasks: short-horizon (around 2 states), medium-horizon (5-10 states), and long-horizon ( ≥ 15 states). We compare the unit test pass rate on four different chain lengths, ranging from No chain-of-thought (the shortest), which directly generates code from demonstrations, to Full (the longest), which represents our approach Demo2Code . The left bar plot in Fig. 7 shows that directly generating code from demonstrations is not effective, and the LLM performs better as the length of the chain increases. The chain length also has a larger effect on tasks with longer horizons. For short-horizon tasks, the LLM can easily process the short demonstrations and achieve high performances by just using 1-step . Meanwhile, the stark difference between 2-steps and Full 's results on long-horizon tasks emphasizes the importance of taking as many small steps as the LLM needs in summarizing long demonstrations so that it will not lose key information.

How do noisy demonstrations affect Demo2Code 's performance? We study how Demo2Code performs (1) when each predicate has a 10% chance to be removed from the demonstrations, and (2) when each state has a 10% chance to be completely removed. Fig. 7's table shows that Demo2Code 's overall performance does not degrade even though demonstrations are missing information. While

removing predicates or states worsen Demo2Code 's performance for shorter tasks (e.g. cook and cut), they surprisingly increase the performance for longer tasks. Removing any predicates can omit essential information in shorter tasks' demonstrations. Meanwhile, for longer tasks, the removed predicates are less likely to be key information, while reducing the length of demonstrations. Similarly, for the longest tasks to make two burgers, one burger's missing predicates or states can be explained by the other burger's demonstration. In section 11, we show a specific example of this phenomenon. We also study the effect of adding additional predicates to demonstrations, which has degraded Demo2Code 's performance from satisfying 5 users' preferences to 2 users' in EPIC-Kitchens.

6 Discussion

In this paper, we look at the problem of generating robot task code from a combination of language instructions and demonstrations. We propose Demo2Code that first recursively summarizes demonstrations into a latent, compact specification then recursively expands code generated from that specification to a fully defined robot task code. We evaluate our approach against prior stateof-the-art [33] that generates code only from language instructions, across 3 distinct benchmarks: a tabletop manipulation benchmark, a novel cooking game Robotouille , and annotated data from EPIC-Kitchens, a real-world human activity dataset. We analyze various capabilities of Demo2Code , such as grounding language instructions, generalizing across tasks, and capturing user preferences.

Demo2Code can generalize across complex, long-horizon tasks. Even though Demo2Code was shown only short-horizon tasks, it's able to generalize to complex, long demonstrations. Recursive summarization compresses long chains of demonstrations and recursive expansion generates complex, multi-layered code.

Demo2Code leverages demonstrations to ground ambiguous language instructions and infer hidden preferences and constraints. The latent specification explicitly searches for missing details in the demonstrations, ensuring they do not get explained away and are captured explicitly in the specification.

Demo2Code strongly leverages chain-of-thought. Given the complex mapping between demonstrations and code, chain-of-thought plays a critical role in breaking down computation into small manageable steps during summarization, specification generation and code expansion.

In future directions, we are looking to close the loop on code generation to learn from failures, integrate with a real home robot system and run user studies with Robotouille .

7 Limitations

Demo2Code is limited by the capability of LLMs. Recursive summarization assumes that once all the demonstrations are sufficiently summarized, they can be concatenated to generate a specification. However, in extremely long horizon tasks (e.g. making burgers for an entire day), it is possible that the combination of all the sufficiently summarized demonstrations can still exceed the maximum context length. A future work direction is to prompt the LLM with chunks of the concatenated demonstrations and incrementally improve the specifications based on each new chunk. In recursive expansion, our approach assumes that all low-level action primitives are provided. Demo2Code currently cannot automatically update its prompt to include any new action. Another direction is to automatically build the low-level skill libraries by learning low-level policy via imitation learning and iteratively improve the code-generation prompt over time. Finally, since LLMs are not completely reliable and can hallucinate facts, it is important to close the loop by providing feedback to the LLM when they fail. One solution [62, 52] is to incorporate feedback in the query and reprompt the language model. Doing this in a self-supervised manner with a verification system remains an open challenge.

In addition, the evaluation approach for Demo2Code or other planners that generate code [33, 61, 77] is different from the one for classical planners [53, 54]. Planners that generate code measure a task's complexity by the horizon length, the number of control flows, whether that task is in the training dataset, etc. Meanwhile, many classical planners use domain specific languages such as Linear Temporal Logic (LTL) to specify tasks [41], which leads to categorizing tasks and measuring the task complexity based on LTL. Future work needs to resolve this mismatch in evaluation standards.

References

Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acquired skills via goal generation. arXiv:2006.07185 , 2020.

Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. arXiv:1711.00482 , 2017.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv:2108.07732 , 2021.

Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989 , 2016.

Cynthia Breazeal, Kerstin Dautenhahn, and Takayuki Kanda. Social robotics. Springer handbook of robotics , 2016.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning Generalizable Robotic Reward Functions from 'In-The-Wild' Human Videos. In Proceedings of Robotics: Science and Systems , Virtual, July 2021.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv:2107.03374 , 2021.

Qibin Chen, Jeremy Lacomis, Edward J. Schwartz, Graham Neubig, Bogdan Vasilescu, and Claire Le Goues. Varclr: Variable semantic representation pre-training via contrastive learning, 2021.

Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Self-educated language agent with hindsight experience replay for instruction following. DeepMind , 2019.

Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 9052-9065, Online, November 2020. Association for Computational Linguistics.

Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference , pages 893-905. PMLR, 2022.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV) , 130:33-55, 2022.

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning , pages 990-998. PMLR, 2017.

Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. Advances in neural information processing systems , 31, 2018.

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1536-1547, Online, November 2020. Association for Computational Linguistics.

Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Ali Eslami, and Oriol Vinyals. Synthesizing programs for images using reinforced adversarial learning. In International Conference on Machine Learning , pages 1666-1675. PMLR, 2018.

Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards. arXiv:2007.15543 , 2020.

Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (CoRL) , 2019.

Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. arXiv preprint arXiv:1704.07926 , 2017.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 , 2021.

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv:2201.07207 , 2022.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv:2207.05608 , 2022.

Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning , 2022.

Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In CoRL , 2022.

Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. NeurIPS , 2019.

Kei Kase, Chris Paxton, Hammad Mazhar, Tetsuya Ogata, and Dieter Fox. Transferable task execution from pixels through deep planning domain learning, 2020.

Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks, 2023.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv:2205.11916 , 2022.

Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In HRI , 2010.

Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001 , 2023.

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven CH Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. arXiv:2207.01780 , 2022.

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 , 2022.

Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models, 2023.

Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans, 2023.

Yunchao Liu, Jiajun Wu, Zheng Wu, Daniel Ritchie, William T. Freeman, and Joshua B. Tenenbaum. Learning to describe scenes with programs. In International Conference on Learning Representations , 2019.

Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, S. Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In IJCAI , 2019.

Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv:2005.07648 , 2020.

Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. AAAI , 2006.

Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse natural language commands to a robot control system. In Experimental robotics , 2013.

Claudio Menghi, Christos Tsigkanos, Patrizio Pelliccione, Carlo Ghezzi, and Thorsten Berger. Specification patterns for robotic missions, 2019.

Toki Migimatsu and Jeannette Bohg. Grounding predicates through actions, 2022.

Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading comprehension through question decomposition and rescoring. arXiv preprint arXiv:1906.02916 , 2019.

Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3d environments with visual goal prediction. arXiv preprint arXiv:1809.00786 , 2018.

Dipendra Kumar Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. CoRR , abs/1704.08795, 2017.

Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning languageconditioned robot behavior from offline data and crowd-sourced annotation. In CoRL , 2022.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 , 2022.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.

Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. arXiv preprint arXiv:1611.01855 , 2016.

Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. arXiv preprint arXiv:2002.09758 , 2020.

Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations , 2022.

Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting, 2022.

Ankit Shah, Pritish Kamath, Julie A Shah, and Shen Li. Bayesian inference of temporal task specifications from demonstrations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Ankit Shah, Shen Li, and Julie Shah. Planning with uncertain specifications (PUnS). IEEE Robotics and Automation Letters , 5(2):3414-3421, apr 2020.

Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action, 2022.

Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. In Proceedings of Robotics: Science and Systems (RSS) , 2020.

Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv:2204.05186 , 2022.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In CoRL , 2021.

Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems, 2020.

Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop , 2022.

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models, 2022.

Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, 2023.

Shawn Squire, Stefanie Tellex, Dilip Arumugam, and Lei Yang. Grounding english commands to reward functions. In Robotics: Science and Systems , 2015.

Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. NeurIPS , 2020.

Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthesis from diverse demonstration videos. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4790-4799. PMLR, 10-15 Jul 2018.

Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Review of Control, Robotics, and Autonomous Systems , 2020.

Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI , 2011.

Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond Mooney. Jointly improving parsing and perception for natural language commands through human-robot dialog. JAIR , 2020.

Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the 2015 International Joint Conference on Artificial Intelligence (IJCAI) , pages 1923-1929, Buenos Aires, Argentina, July 2015.

Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. In International Conference on Learning Representations , 2019.

Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax , May 2021.

Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv:2201.11903 , 2022.

Terry Winograd. Procedures as a representation for data in a computer program for understanding natural language. MIT PROJECT MAC , 1971.

Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv:2109.10862 , 2021.

Jiajun Wu, Joshua B. Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models, 2023.

Xiaojun Xu, Chang Liu, and Dawn Song. SQLNet: Generating structured queries from natural language without reinforcement learning, 2018.

Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 440-450, Vancouver, Canada, July 2017. Association for Computational Linguistics.

Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv:2204.00598 , 2022.

Luke S Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv:1207.1420 , 2012.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv:2205.10625 , 2022.

Appendix

Table of Contents

8 Tabletop Manipulation Simulator Pipeline

8.1 Pipeline Overview

The tabletop manipulation simulator contains simple tasks. Consequently, the demonstrations do not have too many states ( ≤ 8 states) and the code is not complex. Thus, Demo2Code 's prompt for this domain does not need a long extended chain-of-thought. In stage 1 recursive summarization, the LLM just needs to summarize each states into a sentences that describes the low-level action (e.g. move, pick, place, etc.) In stage 2 recursive expansion, because the code is simple, the LLM can directly use all the low-level actions that are provided to output the task code given a specification.

The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.1.

8.2 Experiment Setup

In the paper, we categorize the tabletop tasks into three clusters. For each cluster, we list all the tasks and their variants of possible requirements below. The tasks that are used in the prompt are bolded.

Specificity

-Place A next to B

No hidden specificity: A can be placed in any relative position next to B

A must be to the left of B

A must be to the right of B

A must be behind B

A must be in front of B

-Place A at a corner of the table

No hidden specificity: A can be placed at any corner.

A must be at the top left corner

A must be at the top right corner

A must be at the bottom left corner

A must be at the bottom right corner

-Place A at an edge of the table

No hidden specificity: A can be placed at any corner.

A must be at the top edge

A must be at the bottom edge

A must be at the left edge

A must be at the right edge

Hidden Constraint

-Place A on top of B

No hidden constraint: A can be directly placed on top of B in one step

There is 1 additional object on top of A, so that needs to be removed before placing A on top of B.

There are 2 additional objects on top of A.

There are 3 additional objects on top of A.

-Stack all blocks

No hidden constraint: All blocks can be stacked into one stack

Each stack can be at most 2 blocks high

Each stack can be at most 3 blocks high

Each stack can be at most 4 blocks high

-Stack all cylinders (Same set of hidden constraints as 'stack all blocks." None of the examples appears in the prompt.)

Personal Preference

-Stack all blocks into one stack

2 blocks must be stacked in a certain order, and the rest can be unordered

3 blocks must be stacked in a certain order

All blocks must be stacked in a certain order

-Stack all cylinders into one stack (Same set of hidden constraints as 'stack all blocks into one stack" None of the examples appears in the prompt.)

-Stack all objects

No hidden preference: The objects do not need to be stacked in to different stacks based on their type

All the blocks should be stacked in one stack, and all the cylinders should be stacked in another stack

8.2.1 Provided Low-Level APIs

We have provided the following APIs for the perception library and low-level skill library:

Perception Library

-get_obj_names() : return a list of objects in the environment

-get_all_obj_names_that_match_type(type_name, objects_list) : return a list of objects in the environment that match the type_name .

-determine_final_stacking_order(objects_to_enforce_order, objects_without_order) : return a sorted list of objects to stack.

Low-level Skill Library

-put_first_on_second(arg1, arg2) : pick up an object ( arg1 ) and put it at arg2 . If arg2 is an object, arg1 will be on top of arg2 . If arg2 is 'table', arg1 will be somewhere random on the table. If arg2 is a list, arg1 will be placed at location [x, y].

-stack_without_height_limit(objects_to_stack) : stack the list of objects_to_stack into one stack without considering height limit.

-stack_with_height_limit(objects_to_stack, height_limit) : stack the list of objects_to_stack into potentially multiple stacks, and each stack has a maximum height based on height_limit .

8.3 Characterize Tabletop Tasks' Complexity

In table 4, we characterize the complexity of the tasks in terms of the demonstrations' length, the code's length, and the expected code's complexity (i.e. how many loops/conditionals/functions are needed to solve this task).

9 Robotouille Simulator Pipeline

9.1 Overview

9.1.1 Simulator Description

In Robotouille , a robot chef performs cooking tasks in a kitchen environment. The state of the kitchen environment consists of items such as buns, lettuce, and patties located on stations which could be tables, grills, and cutting boards. The actions of the robot consist of moving around from one station to another, picking items from and placing items on stations, stacking items atop and unstacking items from another item, cooking patties on stoves, and cutting lettuce on cutting boards. The state and actions are described through the Planning Domain Description Language (PDDL).

These PDDL files consist of a domain and a problem. The domain file defines an environment; it contains the high-level predicates that describe the state of the world as well as the actions of the world including their preconditions and effects on the world's predicate state. The problem file describes a configuration of an environment; it contains the domain name for the environment, the initial objects and true predicates, and the goal state. These files are used with PDDLGym [59] as a backend to create an OpenAI Gym [6] environment which given a state and action can be stepped through to produce the next state.

There are 4 problem files for different example scenarios including cooking a patty and cutting lettuce, preparing ingredients to make a burger, preparing ingredients to make two burgers, and assembling a burger with pre-prepared ingredients. In a scenario, various different tasks can be carried out, such as varying the order and ingredients for making a burger. These problem files contain the minimum number of objects necessary to complete the scenario for any specified task.

One issue with having pre-defined problem files for each scenario is that the code produced in code generation could be hardcoded for a scenario. This is avoided by procedurally generating the problem files. There are two types of procedural generation: noisy randomization and full randomization. Noisy randomization, which is used for every Robotouille experiment in this paper, ensures that the minimum required objects in a problem file appear in an environment in the same grouped arrangement (so an environment with a robot that starts at a table with a patty on it and a cutting board with lettuce on it will maintain those arrangements) but the locations are all randomized and extra stations and items are added (noise). The location of stations and items determines the ID suffix which prevents code generation from always succeeding using hardcoded code.

Full randomization does everything except enforcing that the minimum required objects in a problem file appear in the same grouped arrangement. This would require code that handles edge cases as simple as utilizing ingredients that are already cooked or cut in the environment rather than preparing new ones to more extreme cases such as the kitchen being cluttered with stacked items requiring solving a puzzle to effectively use the kitchen. The simpler case is more appropriate in a real setting and we leave it to future work to remove initial arrangement conditions.

9.1.2 Pipeline Overview

In stage 1 recursive summarization, the LLM first recursively summarizes the provided demonstrations, which are represented as state changes since the previous state, until it determines that the trajectories are sufficiently summarized. For this domain, the LLM in general terminates after it summarizes the trajectory into a series of high-level subtasks. Then, Demo2Code concatenates all trajectories together before prompting the LLM to reason about invariant in subtask's order before generating the task specification.

In stage 2 recursive expansion, there are 3 steps that occur for Demo2Code . First, (1) the task specification is converted directly to code which uses provided helper functions and may use undefined higher-level functions. Second, (2) the undefined higher-level functions are defined potentially including undefined lower-level functions. Finally, (3) the undefined lower-level functions are unambiguously defined.

The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.2.

Make a burger stacking lettuce atop patty immediately

Make a burger stacking lettuce atop patty after preparation

Cook first then cut

Cut first then cook

Make a burger stacking patty atop lettuce immediately

Make a burger stacking patty atop lettuce after preparation

Make a cheese burger

Make a chicken burger

Make a lettuce tomato burger

Cook two patties

Cut two lettuces

Make two burgers stacking lettuce atop patty immediately

Make two burgers stacking lettuce atop patty after preparation

Make two burgers stacking patty atop lettuce immediately

Make two burgers stacking patty atop lettuce after preparation

Make two cheese burgers

Make two chicken burgers

Make two lettuce tomato burgers

Assemble two burgers one by one

Assemble two burgers in parallel

9.2 Experiment Setup

In the paper, we categorized the Robotouille simulator into 4 example scenarios. Below are all the scenarios as well as possible tasks, visualized in Fig. 8.

Cook a patty and cut lettuce

-Cook a patty

-Cut a lettuce

-Cook first then cut

-Cut first then cook

Assemble two burgers from prepared ingredients

-Assemble two burgers one by one

-Assemble two burgers in parallel

Make a burger

-Stack a top bun on top of a cut lettuce on top of a bottom bun

-Make a burger stacking lettuce atop patty immediately

-Make a burger stacking patty atop lettuce immediately

-Make a burger stacking lettuce atop patty after preparation

-Make a burger stacking patty atop lettuce after preparation

-Make a cheese burger

-Make a chicken burger

-Make a lettuce tomato burger

Make two burgers

-Cook two patties

-Cut two lettuces

-Make two burgers stacking lettuce atop patty immediately

-Make two burgers stacking patty atop lettuce immediately

-Make two burgers stacking lettuce atop patty after preparation

-Make two burgers stacking patty atop lettuce after preparation

-Make two cheese burgers

-Make two chicken burgers

-Make two lettuce tomato burgers

9.2.1 Provided Low-Level APIs

We have provided the following APIs for the perception library and low-level skill library:

Perception Library

-get_all_obj_names_that_match_type(obj_type) : return a list of string of objects that match the obj_type .

-get_all_location_names_that_match_type(location_type) : return a list of string of locations that match the location_type .

-is_cut(obj) : return true if obj is cut.

-is_cooked(obj) : return true if obj is cooked.

-is_holding(obj) : return true if the robot is currently holding obj .

-is_in_a_stack(obj) : return true if the obj is in a stack.

-get_obj_that_is_underneath(obj_at_top) : return the name of the object that is underneath obj_at_top .

-get_obj_location(obj) : return the location that obj is currently at.

-get_curr_location() : return the location that the robot is currently at.

Low-level Skill Library

-move(curr_loc, target_loc) : move from the curr_loc to the target_loc .

-pick_up(obj, loc) : pick up the obj from the loc .

-place(obj, loc) : place the obj on the loc .

-cut(obj) : make progress on cutting the obj . Need to call this function multiple times to finish cutting the obj .

-start_cooking(obj) : start cooking the obj . Only need to call this once. The obj will take an unknown amount before it is cooked.

-noop() : do nothing.

-stack(obj_to_stack, obj_at_bottom)) : stack obj_to_stack on top of obj_at_bottom .

-unstack(obj_to_unstack, obj_at_bottom) : unstack obj_to_unstack from obj_at_bottom .

9.3 Characterize Robotouille 's Tasks' Complexity

In table 5, we characterize the complexity of the tasks in terms of the demonstrations' length, the code's length, and the expected code's complexity (i.e. how many loops/conditionals/functions are needed to solve this task).

In addition, to bridge the different evaluation standards between planners that generate code and classical planners, we also characterize the Robotouille 's tasks based on [41]'s taxonomy in table 6

10 EPIC-Kitchens Pipeline

10.1 Annotations

We take 9 demonstrations of dishwashing by users 4, 7, 22 and 30, and use 2 of these as in-context examples for the LLM, by writing down each intermediate step's expected output.

at(`stove_1`);on(`pan_1`, `stove_1`)

inhand(`pan_1`);isdirty(`pan_1`)

soapy(`pan_1`);ison(`tap_1`)

clean(`pan_1`)

isoff(`tap_1`);on(`pan_1`, `dishrack_1`)

at(`desk_1`)

inhand(`mug_1`);isdirty(`mug_1`)

at(`sink_1`);inhand(`scrub_1`);ison(`tap_1`)

soapy(`mug_1`)

clean(`mug_1`)

isoff(`tap_1`)

at(`kitchentop_1`);on(`mug_1`, `kitchentop_1`)

Predicates are of the form foo(`<obj>_<id>`, ...) where foo is a predicate function like adjective ( is\_dirty , is\_soapy etc) or preposition ( at , is\_in\_hand , on etc). Each argument is a combination of object name and unique id, the latter added to distinguish multiple objects of the same kind. Note that these annotations or object ids are not available in the EPIC-Kitchens dataset.

Not all predicates are enumerated exhaustively, because this can be difficult for a human annotator, as well as useless and distracting for the LLM. The state predicate annotations in the demonstrations are limited to incremental changes to the observable environment. For example, is\_in\_hand(` plate_1`) comes after in(`plate_1`, `sink_1`)

Examples of these incremental state predicate annotations are described in figures 9 and 10. We avoid annotating unnecessary human-like actions like picking up something then immediately placing it back, or turning on a tap momentarily.

10.2 Pipeline Overview

In stage 1 recursive summarization, the LLM first recursively summarizes the provided demonstrations, which are represented as state changes since the previous state, until it determines that the

trajectories are sufficiently summarized. For this domain, the LLM in general terminates after it summarizes the trajectory into a series of high-level subtasks, which each consist of multiple states and low-level actions. For example, low-level actions 'Pick up spoon_1", 'Pick up fork_1", and 'Go from countertop_1 to sink_1" get combined as the subtask 'bring spoon_1 and fork_1 from counter-top_1 to the sink_1." Then, Demo2Code concatenates all trajectories together before prompting the LLM to reason about the control flow (e.g. whether a for-loop is needed) before generating the task specification.

In stage 2 recursive expansion, because the dishwashing does not use that many unique actions, the LLM is asked to directly use all the low-level actions that are provided as APIs to output the task code given a specification.

The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.3.

10.2.1 Provided Low-Level APIs

We have provided the following APIs for the perception library and low-level skill library:

Perception Library

-get_all_objects() : return a list of objects in the environment.

Low-level Skill Library

-bring_objects_to_loc(obj, loc) : bring all the objects to loc .

-turn_off(tap_name) : turn off tap.

-turn_on(tap_name) : turn on tap.

-soap(obj) : soap the object.

-rinse(obj) : rinse the object.

-pick_up(obj) : pick up the object.

-place(obj, loc) : pick up the object at loc .

-clean_with(obj, tool) : clean the object with the tool , which could be a sponge or a towel.

11 Noisy Demonstration Ablation Experiment

As seen in our own annotations for EPIC-Kitchens demonstrations, human annotations or annotations generated by automatic scene summarizers and object detectors may not be noise-free. They may omit some predicates or completely missed predicates in an entire timestep. They may contain objects that the users did not interact with during the demonstration, so predicates about these objects are of little importance to the robot task plan. Thus, we conducted two noisy demonstration ablations:

Randomly removing predicates/states from the demonstrations (tested in Robotouille)

Randomly adding predicates about irrelevant objects to the demonstrations (tested in EPICKitchens).

We found that:

Randomly removing predicates/states

-Removing predicates reduces Demo2Code 's performance for tasks with short horizons.

-Surprisingly, it does not significantly worsen the performance for tasks with long horizons.

Randomly adding irrelevant predicates

-Additional irrelevant predicates worsen Demo2Code 's performance for correctly generating code for 5 users to 2 users.

11.1 Randomly removing predicates/states

11.1.1 Experimental Details

For each task in Robotouille , we modified the demonstrations in two ways:

for each predicate in the demonstration, there is a 10% probability that the predicate would be removed from the demonstration.

for each state (which could consist of multiple predicates), there is a 10% probability that the entire state would be removed from the demonstration.

We ran the experiment on 4 seeds to report the average and the variance.

11.1.2 Qualitative Result

We analyze a qualitative example (making a burger where the patty needs to be stacked on top of the lettuce immediately after it is cooked) where removing predicates did not affect Demo2Code 's performance.

When each predicate has 10% probability of being removed, the demonstration is missing 6 predicates, Half of them omits information such as picking up the lettuce, moved from one location to another location, etc. However, the other half does not omit any information. For example, one of the predicate that gets removed is ' 'robot1' is not holding 'top_bun3' ".

Removing this predicate does not lose key information because ' 'top_bun3' is on top of 'patty3' " still indicates that 'top_bun3' has been placed on top of 'patty3' . Consequently, the LLM is still able to summarize for that state:

Thus, Demo2Code is able to generate identical predicates

Using the same seed, when each state has 10% probability of being completely removed, the demonstration is missing 5 states (9 predicates). Because all the predicate in a selected state gets removed, the LLM misses more context. For example, because the following two states are randomly removed, the LLM does not know that the demonstration has moved and placed 'lettuce1' at 'cutting_boarding1'.

Consequently, it causes the LLM to incorrectly summarizes the states and misses the subtask of cutting the lettuce.

11.2 Randomly removing predicates/states

11.2.1 Experimental Details

For each EPIC-Kitchens task, we add additional predicates (i.e. showing the position of additional objects in the scene) in at least 2 separate states in the demonstrations. We also do the same

modification for the training examples, while keeping the rest of the prompt identical. We expect the LLM to weed out these additional states during recursive summarization.

For example, for annotations of video ID P 22 _ 07 as seen in appendix 17.3.1, we add distractions in 2 states -

11.3 Quantitative Analysis

We see that in table 7 that Demo2Code suffers from degradation on most demonstrations when distracting states are added, resulting in only being able to generate correct code for 2 out of 7 demonstrations instead of 5 out of 7 demonstrations.

11.4 Qualitative Analysis

When adding distracting states, the LLM mostly ignores the distracting states and only shows variation in output as a result of change in input. However, the distracting states can interfere with the final code, as can be seen in the output for demonstration P 44 _ 101 .

Code using clean demonstrations:

Code generated with demonstration that has additional irrelevant predicates:

When compared to the generated output with clean annotations, we see that while the on(`jar_1`, `microwave_1`) was ignored, in(`brush_1`, `sink_1`) and in(`bowl_1`, `sink_2`) result in the LLM generating additional code that does not align with the demonstrations. Specifically, even though brush_1 and bowl_1 were objects that the users were not interested in interacting with, the LLM has generated a place() code (lines 6-7) for these two objects. This type of mistake could be avoided by adding reasoning during recursive summarization. The LLM can be guided to ignore irrelevant objects and avoid hallucinating actions relating to these objects - for example, ground place action only when both is_in_hand(...) and on(..., loc) are seen one after the other.

12 Chain-of-thought Ablation Experiment

This experiment studies the effect of the chain-of-thought's length (in stage 1 recursive summarization) on the LLM's performance. We found:

It is helpful to guide the LLM to take small recursive steps when summarizing demonstrations (especially for tasks with long demonstrations).

The LLM performs the worst if it is asked to directly generate code from demonstrations.

12.1 Experiment Detail

We defined 3 ablation models listed below from the shortest chain-of-thought length to the longest chain length. In addition, because the tabletop's Demo2Code pipeline is different from Robotouille 's pipeline, we also describe how these pipelines are adapted to each ablation model:

No-Cot : Tabletop and Robotouille has exactly the same process of prompting the LLM ONCE to generate code given the language model and the demonstrations.

1-Step

-Tabletop: First, the LLM receives all the demonstrations concatenated together as input to generate the specification without any intermediate reasoning. Next, the LLM generates the code given the specification.

-Robotouille : First, the LLM receives all the demonstrations concatenated together as input to generate the specification. It can have intermediate reasoning because the tasks are much more complex. Next, the LLM generates the high-level code given the specification and recursively expands the code by defining all helper functions.

· 2-Steps

-Tabletop: First, the LLM classifies the task into either placing task or stacking task. Second, the LLM receives all the demonstrations concatenated together as input to generate the specification without any intermediate reasoning. Finally, the LLM generates the code given the specification.

-Robotouille : First, for each demonstration, the LLM gets its state trajectories as input to identify a list of the low-level action that happened at each state. Second, all the low-level actions from each scenario are concatenated together and used by the LLM to generate the specification. The LLM can have intermediate reasoning at this step because the tasks are much more complex. Finally, the LLM generates the high-level code given the specification and recursively expands the code by defining all helper functions.

We identified 3 clusters of tasks based on the number of states they have, and for each cluster, we selected two tasks to test. For each task and for each of that task's specific requirements, we tested the approach 10 times and took an average of the unit test pass rate.

Short-horizon tasks (around 2 states): 'Place A next to B" and 'Place A at a corner"

Medium-horizon tasks (around 5-10 states): 'Place A on top of B" and 'Stack all blocks/cylinders (where there might be a maximum stack height)"

Long-horizon tasks (more than 15 states): 'Make a burger" and 'Make two burgers"

Short

Medium

Long

0.90

0.86

0.87

0.80

0.8F

0.75

Rate

0.68

0.60

Unit Test Pass

0.6

0.43

0.4

0.19

0.2

0.06

0.03

0.00

0.0

No-COT 1-Step

2-Steps

Full

12.2 Quantitative Result

We include the quantitative result again here as Fig. 11. Overall, Demo2Code / Full performs the best, and there's the general trend that as the length of the chain of length increases, the LLM also generates code that has a higher unit test pass rate. For short-horizon tasks, the difference in the chain-of-thought's length has a small effect on the pass rate because these tasks have short demonstrations that are easy to process without intermediate chain-of-thought steps. In contrast, both medium-horizon tasks and long-horizon tasks see great improvement when the LLM is prompted to take smaller recursive summarization steps. In the next section, we will examine one example from each cluster of tasks to analyze how the length of the chain-of-thought prompts affects the LLM's output.

12.3 Qualitative example for a short-horizon task

We examine all four models' outputs on the task: placing A next to B. There are 2 states per demonstration and 1 demonstration in total. The query and the Full approach's correctly generated code for a particular instance of the task are shown below:

For this particular example, the LLM needs to be able to identify that the robot needs to specifically place the gray cylinder in front of the green block. Only No-Cot failed to generate the correct code,

while 1-step and 2-steps were able to identify the specification and generate the code exactly the same as Full .

Why No-Cot failed?

No-Cot seemed to overfit to the prompt. For this example where the specific requirement is to place A to the right of B, it outputs 'to the right of the green block." We hypothesize that because the specific requirement of placing A to the left of B appears in the prompt, the LLM just assumes that any state that differs from the example in the prompt is placing A to the right of B . Below is the code outputted by No-Cot :

12.4 Qualitative example for a medium-horizon task

We examine all four models' output on the task: placing A on top of B. There are 5 states per demonstration and 1 demonstration in total. The query and the Full approach's correctly generated code for a particular instance of the task are shown below:

For this particular example, the LLM needs to recognize that there are 3 additional objects on top of the brown block. In order to place the brown block on top of the purple cylinder, the robot needs to first place those 3 additional objects on the table. No-Cot and 1-step approaches failed to generate the correct code, while 2-steps was able to identify the specification and generate the code exactly the same as Full .

Why No-Cot and 1-step failed?

Both No-Cot and 1-step seem to be over-reliant on the language instruction instead of reasoning through the state transitions. Specifically, examining the specification outputted by 1-step suggests that the LLM simply copied the language instruction instead of recognizing that more objects are moved than just moving the brown block to be on top of the purple cylinder.

Below show the identical code generated by No-Cot and 1-step .

12.5 Qualitative example for a long-horizon task

We examine all four models' output on the long-horizon task: making a burger with patty, lettuce, and tomato. This task has 71 states per demonstration and 2 demonstrations in total.

The Full pipeline's approach outputs the correct code as shown below. Next, we analyze how other models failed to produce the correct code.

Why No-Cot failed?

The main issue that No-Cot suffers from is the maximum token length issue because the prompt needs to include demonstrations as the example query and solution code as the example response. Thus, the LLM was not able to finish completely the code as seen below (line 37). In addition, although the LLM showed some ability to use the control flow, it was only creating a for loop to cook the patties but it did not use a for-loop to assemble the burger. Specifically, the LLM generated code that index to specific items: (e.g. line 23 bottom_buns[0] , line 25 lettuces[0] , etc.)

Why 1-step failed?

In Fig. 11's result, 1-step actually performed worse than No-Cot . The worsened performance is caused by low-quality specifications produced by the LLM. Because stage 2 recursive expansion is highly reliant on the specification that gets passed in, any mistaken or missing specification will be reflected in the code as well. Below shows the intermediate reasonings and specification outputted by the LLM. Notice that it does not capture any information about the lettuce and the tomato needing to be cut before being added to the burger (see lines 22-23 and lines 25-26).

Consequently, although the code (shown below) closely matched the specification and included the for-loop, because the specification was wrong, the code also failed to solve the problem.

Why 2-steps failed?

2-steps improved its specification compared to 1-step 's. Specifically, its specification was able to identify that lettuce and tomato require cutting before they are stacked as a part of the burger. However, it still fails to identify the physical constraint that in order to cut an object, it also needs to specify the location. While it is able to identify the stove used for cooking (line 17), it fails to identify the cutting board (line 22 and line 25).

Consequently, in the code that the LLM produced based on the specification, it created a function cut_object(obj) . Then, because the function signature does not specify any location to cut the object at, the downstream function definition also fails (see the definition starting from line 43).

13 Intermediate Reasoning Ablation Experiment

This experiment studies how the thoroughness of intermediate reasoning (in stage 3 of recursive summarization) affects the LLM's performance. We found:

Our approach may be improved by ablating away some part of reasoning (e.g. listing the high-level actions)

Having intermediate reasoning of any form is helpful for hard cooking tasks that have many sub-tasks

Future work is necessary to engineer better prompts

13.1 Experiment detail

This experiment compares Demo2Code (labeled as Full ) with three additional ablation models each with differing levels of reasoning

No reasoning: The LLM generates the specification directly from step 2 of recursive summarization with no intermediate reasoning.

Only List: The LLM generates the specification after intermediate reasoning which lists the high-level actions in common with the scenarios from step 2 of recursive summarization.

Only Analyze: The LLM generates the specification after intermediate reasoning which describes the repetition and ordering of high-level actions from step 2 of recursive summarization.

These models are tested on all the Robotouille tasks. We use 3 clusters of tasks based on the number of high-level actions/sub-tasks they have.

Easy cooking tasks ( ≤ 2 high-level actions/sub-tasks): 'cook and cut", 'cook two patties", and 'cut two lettuces"

Normal cooking tasks (between 2-7 high-level actions/sub-tasks): 'make a burger" and 'assemble two burgers with already cooked patties"

Hard cooking tasks ( ≥ 8 high-level actions/sub-tasks): 'make two burgers"

13.2 Quantitative result

For each task and for each of that task's specific requirements, we tested each model on 10 randomly generated environments and took an average of the unit test pass rate. Fig. 12 shows that the Only Analyze model outperforms the other models. All methods easily achieve 100% pass rate on Easy

Easy

Normal

Hard

1.00

1.00

1.00

1.00

1.0

Rate

0.8

Test PassI

0.6

0.54

0.56

0.51

0.43

0.400.37

0.4

Unit'

0.23

0.2

0.14

0.0

No

Only

Only

Full

Reasoning

List

Analyze

tasks. There is a trend that including any kind of reasoning improves performance on hard tasks. Only List only slightly suffers in Normal performance while increasing significantly for Hard performance relative to No Reasoning . Full 's Normal performance suffers significantly yet its Hard performance increases as well, suggesting a tradeoff between Normal performance and Hard performance; however, Only Analyze has the highest Normal and Hard performance of all models. Since Full combines both Only List and Only Analyze , this shows that concatenating reasoning methods doesn't necessarily mean better performance.

13.3 Qualitative example

In this section, we show a qualitative example of all models' output on the long-horizon task "make two burgers" with the specific requirement "stack lettuce atop patty immediately." We choose this example to show the effects of reasoning on the code as well as to shed light on why Full underperforms relative to Only Analyze . Only Analyze is the only model with 1.0 success rate while the other models all have 0.0 success rates. For each model's output, the helper functions can be found in section 19.3.

13.3.1 Why No Reasoning failed?

No Reasoning is the only unsuccessful model that generates code that executes. Due to the lack of any reasoning, it performs the task correctly but with the incorrect requirement; rather than preparing one burger at a time and assembling the burger immediately as ingredients are prepared, it prepares all ingredients before assembling both burgers in parallel.

13.3.2 Why Only List failed?

Only List improves on No Reasoning by generating code in the correct order of the specification. Patties are cooked and then stacked on the bottom bun before lettuces are cut and stacked on the patties before finally the top buns. While the order is now correct, just specifying the order wasn't effective enough because the reasoning missed making burgers one by one rather than all in parallel. In addition, this code doesn't execute due to the attempt of cutting lettuces back to back at the same cutting board. This violates the environment's constraints since only one object can be at one station at a time.

13.3.3 Why Full failed?

Full gets much closer to correctly solving the task than Only List . Unlike Only List , Full has the correct ordering for the requirement; it stacks the patty and lettuce immediately after preparation and it does so for an entire burger before moving on. The issue is that while with the analysis of ordering and repetition it manages to put all the code in one for loop rather than multiple, it retained some of the poor aspects of Only List by attempting to cook two patties at once. Doing this within the for loop is incorrect logic. In addition, the cooking is attempted on the same stove twice which violates the environment's constraints of one item being on a station at a time.

13.3.4 Only Analyze Output

Only Analyze is the only model to adhere to the environment's constraints. Note that while the same stove is used for cooking twice, the second time a patty is cooked the first patty has already been removed from the stove and stacked into the first burger.

14 Recursive Expansion Ablation Experiment

This experiment studies how the number of recursive code expansion steps (in stage 2 recursive expansion) affects the LLM's performance. We found:

It is helpful to guide the LLM to slowly expand the code instead of asking it to directly generate all the code at once using only the low-level imported APIs.

The initial code that LLM uses to expand the rest of the functions should align closer to the given specifications.

14.1 Experiment detail

This experiment compares Demo2Code (labeled as Full ) with three additional ablation models each with a different amount of recursive code expansion steps:

1-layer: Given a specification, the LLM directly outputs the code for the task using only low-level action and perception APIs from the import statement. The LLM is not allowed to define any helper function.

2-layer (Comp): Given a specification, the LLM first outputs corresponding code that can call undefined 'composite functions." Each composite functions contain at most two low-level actions, e.g. move_then_pick , move_then_stack . In the next step, the LLM defines these composite functions using only low-level action and perception APIs from the import statement.

2-layer (High): Given a specification, the LLM first outputs corresponding high-level code that can call undefined 'high-level functions." This step is the same as Demo2Code . Then, the LLM defines these high-level functions using only low-level action and perception APIs from the import statement.

These models are tested on all the Robotouille tasks. Because section 13 on Intermediate Reasoning Ablation Experiments also tested only on Robotouille , we use the same 3 clusters of tasks based on the number of high-level actions/sub-tasks they have.

Easy cooking tasks ( ≤ 2 high-level actions/sub-tasks): 'cook and cut", 'cook two patties", and 'cut two lettuces"

Normal cooking tasks (between 2-7 high-level actions/sub-tasks): 'make a burger" and 'assemble two burgers with already cooked patties"

Hard cooking tasks ( ≥ 8 high-level actions/sub-tasks): 'make two burgers"

14.2 Quantitative result

For each task and for each of that task's specific requirements, we tested each model on 10 randomly generated environments and took an average of the unit test pass rate. Fig. 13 shows how our approach Demo2Code ( Full ) outperforms the other models. Overall, as the number of recursive code generation steps increases, the LLM's performance also increases. Interestingly, 2-Layers (Comp) performs worse than 2-Layer's (High) despite having the same amount of recursive steps. The two model's performance difference suggests that the granularity of the initial code also affects the performance. While 2-Layers (High) first asks the LLM to generate high-level code, which is the same approach as Demo2Code , 2-Layers (Comp) first asks the LLM to generate code that calls composite functions. Consequently, for 2-Layers (Comp) , the LLM needs to first output longer, more complicated code the LLM has to produce in one-step, which is more prone to error.

Easy

Normal

Hard

1.00

1.0

0.8

Rat

Test Pass

0.6

0.400.37

0.4

Unit?

0.30

0.23

0.2

0

0

0

0

0

0

0

0.0

1-Layer

2-Layers

2-Layers

Full

(Comp)

(High)

14.3 Qualitative example

In this section, we show a qualitative example of all models' output on the short-horizon task 'cut two lettuces." Note that cutting lettuce has not been shown in the prompt, so the LLM needs to be able to generalize to this new task and properly use a for-loop to write efficient code. 1-layer and 2-layer (Comp) had 0.0 success rate. 2-layer (High has 0.2 success rate, and Demo2Code has 1.0 success rate. All models are able to generate the for-loop, as evident in the outputted code below. However, all ablation models struggle to generate code that adheres to physical constraints and applies to any random environment.

14.3.1 Why 1-Layer failed?

1-Layer generates the correct code to cut the lettuce. Despite not seeing how the low-level action cut() should be used in the prompt, it correctly writes the code that will cut the lettuce until the lettuce becomes cut. However, the LLM misses the physical constraints associated with cooking, e.g. in order to cut the lettuce at a cutting board, the robot needs to make sure that there is a lettuce at the cutting board before starting to cut. If there is no lettuce on the cutting board, the robot first needs to go pick up a lettuce and bring it to the cutting board. Another issue that 1-Layer encounters is reaching the LLM maximum token length. For longer tasks, it cannot finish generating all the code, so its overall execution success rate is only 0.04.

14.3.2 Why 2-Layers (Comp) failed?

2-Layer (Comp) faces the same issue of not adhering to all the physical constraints. For example, it successfully identifies that it needs to pick up the lettuce before going to the cutting board and cutting it. However, it does not place the lettuce on the cutting board before it starts cutting (see line 27).

In addition, all the composite functions defined in the recursive step are technically correct even though they are not efficient or compact. For example, move_then_cut unnecessarily moves to the object to cut before actually starting to cut (see line 19). It then moves to the cut location inside the while loop of checking if the object has been cut yet or not (see line 22) A more optimal code will be to move to the cut location before the while loop, then just have the call to cut function in the while loop.

A core issue with 2-Layer (Comp) is that the LLM needs to account for the physical constraints when generating the code in the first step. However, the specification that the LLM receives only includes information about the order between high-level actions and high-level control flows (e.g. for loops). The LLM is more prone to mistakes when it has to extrapolate the physical constraints from the high-level specification. Meanwhile, 2-Layer (High) , which has the same number of recursive steps, has generated the correct initial code that the LLM can expand on. Its initial code aligns closer to the specification (see the code in section 14.3.3).

14.3.3 Why 2-Layers (High) failed?

2-Layers (High) generated the same initial high level code as Demo2Code . However, the definition of cut_object_at_location in the recursive step is not exactly correct because the LLM is not taking account of all the physical constraints. It again assumes that the object to cut is already on the cutting board to cut. It moves to the object's location before starting to cut the object (line 18). Although 2-Layers (High) has 0.2 unit test success rate, it achieved this because many of the lettuces happened to be placed on cutting boards in the randomly initialized environments.

14.3.4 Demo2Code Output

Demo2Code is able to adhere to physical constraints and correctly defines all the helper function in its recursive expansion steps.

15 Broader Impact

Our approach is a step towards making collaborative robots more accessible in different settings, such as homes, factories, and logistics operations. The broader implications of our research are manifold, touching upon societal, economic, ethical, technical, and educational aspects.

Societal Impacts: Currently, robots can only be programmed by engineers, limiting the tasks the robot can do to design choices made by engineers. This limits the accessibility for everyday users who require personalization to their individual needs and use cases. Our work tackles this problem head-on by allowing robots to be easily programmable via intuitive, user-friendly interfaces like vision and language. This could lead to increased creativity in the types of tasks that robots can be used for, potentially enabling novel applications and solutions.

Economic Impacts: Our approach has the potential to dramatically decrease the cost of programming robots, thus lowering the barriers to entry for businesses interested in incorporating robotics into their workflows. This could boost productivity and efficiency across a variety of industries, such as manufacturing, agriculture, and healthcare, among others. In the long term, this may contribute to economic growth and job creation in sectors that are currently behind on automation.

Ethical and Legal Impacts: Our approach has many ethical and legal considerations that need to be addressed. For instance, the widespread use of automation may lead to job displacement in certain sectors. Furthermore, it's crucial to ensure that the task code generated from demonstrations respects privacy laws and does not inadvertently encode biased or discriminatory behavior. Future work in this area will require close collaboration with ethicists, policymakers, and legal experts to navigate these issues.

Technical Impacts: Our work has the potential to accelerate the development of more intuitive and efficient human-robot interaction paradigms. For instance, demonstrations can be extended to interventions and corrections of the user. On the algorithmic side, our approach is a first step to connecting LLMs to Inverse Reinforcement Learning, and can spur advances in the fields of imitation learning, reinforcement learning, and natural language processing.

Educational Impacts: Lastly, our work could contribute to educational initiatives. The ability to generate task code from demonstrations could be utilized as an effective teaching tool in schools and universities, promoting a more experiential and intuitive approach to learning about robotics and coding. This could inspire and enable more students to pursue careers in STEM fields.

16 Reproducibility

We ran all our experiments using GPT-3.5 ( gpt-3.5-turbo ) with temperature 0. We have provided our codebase in https://github.com/portal-cornell/demo2code and all the prompts in section 18. The queries used to produce our results are available in the code base. Note that although we set the temperature to 0, which will make the LLM output mostly deterministic, the output might still slightly vary even for identical input.

17 Demo2Code Example Output

We provide an example for each domain and explain Demo2Code 's intermediate and final output.

17.1 Tabletop Simulator Example

This section shows an example of how Demo2Code solves the task: stacking all cylinders into one stack. The hidden user preference is that certain objects might need to have a fixed stack order, while other objects can be unordered. For this example, two objects ('blue cylinder' and 'purple cylinder') must follow the order that 'blue cylinder' should always be directly under the 'purple cylinder.'

17.1.1 Query

The query has 3 components: (1) a list of objects that are in the environment, (2) a language instruction describing the goal of the task, and (3) two demonstrations.

17.1.2 Intermediate output

In stage 1 recursive summarization, for each demonstration, Demo2Code summarized the trajectories into low-level actions (e.g. moved 'blue cylinder', picked up and placed 'red cylinder' on top of 'blue cylinder'). Then, because tabletop's domain has short demonstrations, the LLM determined that a low-level action trajectory was sufficiently summarized.

Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM first identified the order of the stack in each scenario. Then, it compared the two orders in the two scenarios before concluding that the order that stays the same is ['blue cylinder', 'purple cylinder'] . It finally produced the correct specification that will get passed to stage 2 to produce the correct code.

Stage 1 Recursive Summary: recursive call # 1 with state-based trajectories as input

17.2 Robotouille Example

17.2.1 Query

The example we will consider is Make a burger stacking patty atop lettuce immediately . This is a task that contains many different actions and is successfully solved by Demo2Code , making it a comprehensive example. The query for this task consists of two demonstrations each of which has the same language instruction to make a burger. Because each demonstration has 25 states, we have placed the query at the end of the appendix in section 19.1.

17.2.2 Intermediate output

Below, in stage 1 recursive summary, for each demonstration, Demo2Code summarizes each trajectory from a state-based trajectory to a low-level action trajectory to a high-level subtask trajectory, before the LLM decides that the trajectory is sufficiently summarized.

Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM first identified the order of the subtasks in each scenario. Then, it determined that the lettuce needs to be cut and stacked on top of the bottom bun before cooking the patty and stacking the patty on top of the lettuce. It finally produced the correct specification that will get passed to stage 2 to produce the correct code.

Stage 1 Recursive Summary: recursive call # 1 with state-based trajectories as input

Stage 1 Recursive Summary: recursive call # 2 with low-level action trajectories as input

state 2-4) 2. placing 'lettuce1 ' at 'cutting_board1 ' (state 5) 3. cutting 'lettuce1 ' until it is cut (state 6-8)

At state 12-19, the subtask is "cook patty". This subtask contains: 1. moving from 'stove1 ' to 'stove4 ' (state 12-14) 2. placing ' patty1 ' at 'stove4 ' (state 15) 3. cooking 'patty1 ' until it is cooked (state 16-19)

At state 9-11, the subtask is "stack lettuce on top of bottom bun". This subtask contains: 1. picking up 'lettuce1 ' (state 9) 2. moving to stack 'lettuce1 ' on 'bottom_bun1 ' (state 10-11)

At state 20-22, the subtask is "stack patty on top of lettuce". This subtask contains: 1. picking up 'patty1 ' (state 20) 2. moving to stack 'patty1 ' on 'lettuce1 ' (state 21-22)

At state 23-26, the subtask is "stack top bun on top of patty". This subtask contains: 1. moving from 'table3 ' to 'table4 ' (state 23) 2. picking up 'top_bun1 ' (state 24) 3. moving to stack 'top_bun1 ' on 'patty1 ' (state 25-26)

==========================summarizing demo==========================

The input trajectory contains low-level actions because the trajectory mentions "moved", "picked up", etc.

[[Reasoning:]]

I will summarize the low-level action trajectory into high -level subtasks.

The new trajectory will be sufficiently summarized.

High -level subtask trajectory is the most compacted form that cannot be summarized anymore.

[[Is the

new trajectory

sufficiently

[[Summarized trajectory:]]

yes summarized?

(yes/no):]]

In [Scenario 2], at state 2-8, the subtask is "cut lettuce". This subtask contains: 1. moving to pick up 'lettuce3 ' (state 2-3) 2. moving to place 'lettuce3 ' on 'cutting_board3 ' (state 4-5) 3. cutting 'lettuce3 ' until it is cut (state 6-8)

At state 12-19, the subtask is "cook patty". This subtask contains: 1. moving to pick up 'patty3 ' (state 12-13) 2. moving to place ' patty3 ' on 'stove3 ' (state 14-15) 3. cooking 'patty3 ' until it is cooked (state 16-19)

At state 9-11, the subtask is "stack lettuce on top of bottom bun". This subtask contains: 1. picking up 'lettuce3 ' (state 9) 2. moving to stack 'lettuce3 ' on 'bottom_bun3 ' (state 10-11)

At state 20-22, the subtask is "stack patty on top of lettuce". This subtask contains: 1. picking up 'patty3 ' (state 20) 2. moving to stack 'patty3 ' on 'lettuce3 ' (state 21-22)

At state 23-26, the subtask is "stack top bun on top of patty". This subtask contains: 1. moving to pick up 'top_bun3 ' (state 23-24) 2. moving to stack 'top_bun3 ' on 'patty3 ' (state 25-26)

Summarized Demonstrations -> Task Specification

[[Reasoning:]]

In [Scenario 2]'s unique kitchen environment , the subtasks were executed in this order: ['cut lettuce ', 'stack lettuce on top of bottom bun', 'cook patty ', 'stack patty on top of lettuce ', 'stack top bun on top of patty '].

In [Scenario 1]'s unique kitchen environment , the subtasks were executed in this order: ['cut lettuce ', 'stack lettuce on top of bottom bun', 'cook patty ', 'stack patty on top of lettuce ', 'stack top bun on top of patty '].

Both scenarios are executing the subtasks in the same order.

[[Task Specification:]]

There is no repetition or loop in the subtask ordering for any individual scenario.

Make a burger.

17.2.3 Output Code

This is the final output of the LLM, which is the task code to solve this problem. The code illustrates the LLM's ability to write complex code that has control flows (e.g. for loop and if-else statements.)

Task Specification -> High-Level Code

17.3 EPIC-Kitchens Example

17.3.1 Query

We present an example of the demonstration given as a query to the LLM. The annotations are made for video id P 22 _ 07 . This example shows the LLM generalizing to multiple new objects, new

soapy(`mezzaluna_1`)

in(`mezzaluna_1`, `sink_2`);

inhand(`peeler:potato_1`);isdirty(`peeler:potato_1`)

inhand(`mezzaluna_1`);

isdirty(`mezzaluna_1`)

soapy(`peeler:potato_1`)

soapy(`board:cutting_1`)

in(`peeler:potato_1`, `sink_2`);

inhand(`board:cutting_1`);isdirty(`board:cutting_1`)

ison(`tap_1`)

isoff(`tap_1`)

clean(`board:cutting_1`)

in(`board:cutting_1`, `dryingrack_1`)

inhand(`mezzaluna_1`);clean(`mezzaluna_1`)

in(`mezzaluna_1`, `dryingrack_1`)

in(`peeler:potato_1`, `dryingrack_1`)

inhand(`peeler:potato_1`);

clean(`peeler:potato_1`)

locations, and a different dishwashing style. Figure 14 shows visualizations of the video with the respective state annotations. Because each demonstration has 25 states, we have placed the query at the end of the appendix in section 19.2.

17.3.2 Intermediate Output

Below, in stage 1 recursive summary, for each demonstration, Demo2Code summarizes each trajectory from a state-based trajectory to a low-level action trajectory to a high-level subtask trajectory, before the LLM decides that the trajectory is sufficiently summarized.

Then, Demo2Code concatenates all summarized trajectories together to identify the user's preference and generate an appropriate task specification. Specifically, the LLM was able to identify that it should use a for-loop because [pick up, soap, move object from A to B] are repeated. It finally produced the correct specification that will get passed to stage 2 to produce the correct code.

18 Prompts

All prompts are in yaml file representation. For the most updated prompt, please refer to our code base: https://portal-cornell.github.io/demo2code/.

18.1 Tabletop Manipulation Task Prompts

18.1.1 Recursive Summarization

18.1.2 Recursive Expansion

Task Specification -> High-Level Code

18.2 Robotouille Task Prompts

18.2.1 Recursive Summarization

place 'patty6 ' on 'stove11 ' (state 12-13) 3. cooking 'patty6 '

step3:

18.3 EPIC Kitchens Task Prompts

18.3.1 Recursive Summarization

Summarized Demonstrations -> Task Specification

other [[end of response]] 18.3.2 Recursive Expansion spec_2_highlevelcode:

You must follow these rules when you are generating Python code. * You MUST ONLY use Python library functions imported above. You MUST follow the docstrings and specifications for these functions. * You CANNOT call define new functions. You CANNOT use functions that are NOT imported above.

You must follow the instructions provided by the user. You CANNOT add additional steps , conditionals , or loops that are not

19 Other Long Examples

19.1 Example Robotouille Query

"""

19.3 Intermediate Reasoning Ablation Helper Functions

19.3.1 No Reasoning Helper Functions

19.3.3 Full Helper Functions

19.3.4 Only Analyze Helper Functions

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a recursive method to summarize demonstrations into programs through LLM.  The idea is interesting in that it uses spec as the bottleneck to connect complex demonstrations and complex robot task code, encoded and decoded through chain of thoughts. The method is evaluated on three different benchmarks involving table-top manipulation, novel kitchen text tasks, and EpicKitchen. The method outperforms naive language-to-code baselines and can generalize to longer-horizon tasks as well as learning user intents.\n\nSTRENGTHS:\n1. The problem is challenging in that demonstrations and codes are both complex: demonstrations have lots of details and multimodality, yet codes are abstract and need to follow strict requirements. \n2. The method is sound: it uses divide and conquer to tackle some limitations of the current LLM. \n3. The new benchmark can be interesting to researchers that want to attempt the high-level planning problem in kitchen tasks.\n4. The figures and the pseudo codes are helpful\n5. The method generalize to longer-horizon tasks as well as learning user intents.\n\nWEAKNESSES:\n1. Have several concerns on the evaluation metrics \n2. Need more details on discussing the tabletop benchmark, the EpicKitchen experiment and the new proposed benchmark. Looks like epic kitchen is closer to diverse raw data such as Youtube and yet the new kitchen simulator and the table-top tasks has more predicates as well as low-level relationships\n\nQUESTIONS:\n1. The formulation of MDP seem a bit unnecessary to me.\n2. The spec is called “latent” multiple times in the paper, but shouldn’t it be clearly defined for each task?\n3. To confirm on the evaluation metric: Is the unit-test pass rates the same as task success rates? I.e. does the generated code satisfy a set of instructions (task) defined with the demonstrations? Or is it more narrowed?\n4. What is the quality of the spec2code, and why matching it is a necessary metric? It seems to me that there are many ways to write code for each task.\n5. Is hallucination a problem? Because the method introduces such a pipeline instead of end-to-end, is it possible that the LLM introduces unnecessary steps in the instruction summary stage and then introduce more functions to complete those substeps in the expanding stage?\n6. What is the reference code in EpicKitchen dataset in line 190. And what is the user scoring process?\n7. Given that there could be multiple ways to do the breakdown and the LLM are not deterministic, is stochasty a concern? It seems that variance is not provided in the table for each experiment.\n8. Have the authors tried the new GPT3.5 model with longer-context or GPT-4 model?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents demo2code, a framework that takes as input user's language instructions as well as demonstrations, and outputs synthesized code for completing the tasks. It first iteratively summarizes given demonstrations to a compact task specification, then reasons by incorporating user preferences etc, and lastly output expanded execution code. The method is evaluated on a range of tasks, including table top manipulation, a simple cooking simulator, and real-world epic kitchen dataset.\n\nSTRENGTHS:\n- The framework is novel in that it proposes to summarize demonstrations and language instructions using LLM, which is then used for action generation via code synthesis.\n- the recursive and hierachical way of summarizing demonstrations and generating code is reasonable\n- the idea that using LLM to reason about user preferences makes sense\n\nWEAKNESSES:\n- Upon reading the introduction i was excited to see how the approach is able to handle both instruction and demonstration: the latter usually comes in a visual space, but then i realize the authors made a big assumption that they can query the simulator to get state-based demonstration. This assumption presents a few issues: 1) such privileged information hinders the application in realistic settings. In fact, the author had to manually densely label the epic kitchen dataset, which leads to the question of how this can be used for real-world settings 2) even in simulation, it's not straightforawrd to obtain these state information. for example, as opposed to `op-top`, relations such as `in(obj, microwave)` is hard to obtain easily. Also, binary spatial relations loses dense geometric informations. 3) if access to step-by-step low-level state is assumed, and the LLM can summarize and generate step-by-step specifications, why not directly use task and motion planning(TAMP) to solve the task?\n- how does the oracle spec2code works? does it use TAMP? if yes, what's the advantage of demo2code over it?\n- the experiment which uses epickitchen, but need additional manual annotation, loses the point of evaluating on real-world dataset\n\nQUESTIONS:\n- the authors says the language instruction and demonstration shares a common latent space after summarizing, but i don't see any details on this. Did I miss something?\n- also see weaknesses.\n\nLIMITATIONS:\n- assumption on access to privileged low-level state and relations\n- additional annotation on real-world data\n- other limitations are discussed in the last section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Demo2Code, a new method for generating code given a natural language description and demonstrations of the task. Demo2Code recursively summarizes demonstrations using a language model (LM) to create a task specification. The task specification is concatenated to the description and then recursively synthesized into code using a LM. Demo2Code is shown to outperform previous SoTA on an object manipulation environment, an author-designed cooking simulator, and the EPIC Kitchens dataset. Finally, qualitative results are shown demonstrating OOD generalization, grounding, and understanding of user preferences.\n\nSTRENGTHS:\n*Originality:* a new long-range sequential decision-making benchmark, Robotouille, was developed that focuses more on high-level actions rather than manipulation and navigation. This may be useful for future work studying agents that learn from task specification.\n\n*Quality:* the experiments are comprehensive and cover a range of tasks and domains, suggesting that the method is robust to distributions of tasks. Furthermore, the results show a marked improvement over the previous SoTA on all the tasks, which likely demonstrates the efficacy of leveraging demonstrations.\n\n*Clarity:* the paper is well-written and the figures are self-explanatory.\n\n*Significance:* generating code from demonstrations is an important step towards developing agents that can efficiently interact with humans. Moreover, this work demonstrates a working end-to-end setup from natural language demonstrations to code, likely encouraging future work in this area.\n\nWEAKNESSES:\nThere are two implicit assumptions of the method that are not evaluated and may make the method difficult to use in practice. I believe that these assumptions may be hard to overcome, so I am leaning towards giving a 6.5 but am rounding up because of the high-quality execution.\n\n1. *The method assumes that demonstrations are complete descriptions of each state in the trajectory.* I would imagine that in many real-world environments (which is the setting that such a method would likely be deployed in), obtaining a complete description of each state in the trajectory is noisy. Some actions may be occluded or unable to be clearly delineated. Moreover, it may not even be clear a priori how to canonically parse actions from video or a natural language description of the environment. Would it be possible to develop an ablation where some of the actions are noisily parsed or even entirely omitted from the demonstration? I suspect that GPT-3.5 may not be able to handle the perturbation, but GPT-4 may be able to.\n\n2. *The method assumes that the task description is well-specified.* Humans can often provide demonstrations and descriptions of the task that are under-specified or misspecified. For example, when booking airline tickets, a human may forget to describe their preference for red-eye flights and there may not be enough demonstrations to determine their underlying preferences [1]. In its current form, the method appears unequipped to handle such cases and it is unclear what solution it would generate. As mentioned in the work, one possible remedy is to provide feedback to the LM, but it is unclear how successful such an approach would be.\n\n[1]  Lin, J., Fried, D., Klein, D., & Dragan, A. (2022). Inferring Rewards from Language in Context. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8546–8560). Association for Computational Linguistics.\n\nQUESTIONS:\nQuestions:\n* Is there intuition for why the pass performance of DemoNoLang2Code is greater than the pass performance of Demo2Code on the \"Make a burger stacking lettuce atop patty immediately\" and the \"Make two burgers stacking patty atop lettuce after preparation\" tasks in Table 2?\n\nLIMITATIONS:\nIn general the limitations are well-addressed. The following two limitations were also addressed as weaknesses above: \n* Demonstrations are complete descriptions of low-level actions. In practice it seems difficult to obtain complete descriptions of all actions and state of an environment, as actions are often noisy and difficult to clearly delineate. E.g., given some video of a human washing the dishes, it would be unclear how to appropriately parse the action space (in the paper it was done by hand). \n* The natural language description of the task may be misspecified by the human. This seems like a fundamental limitation of the current method, and might make the generated code incorrect or misspecified.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an LLM-based completion framework to translate natural language instructions, in addition to transcribed state sequences of demonstrations (as PDDL (or other strips-like) predicates), into code for executing the task with a robot.\n\nThe method is based on recursively summarizing the demonstrations into a 'specification', and then recursively expanding the 'specification' into python(?) code. The recusrsion stops in the first phase when no further summarization is possible, and in the code expansion loop, when no function is undefined.\n\nThe authors demonstrate this is three different domains with upto 10 distinct high level actions occuring in each demonstration.\n\nSTRENGTHS:\nMultimodal learning from demonstrations plus language instructions is an important problem that will enable consuming a wider modality of data sources to generate robot/agent programs and behavior. Using the summarization/translation capabilities of LLMs will likely be a key component of such a system.\n\nWEAKNESSES:\n**Methodology**: The paper does not define what a task specification is concretely, nor does it define what it means for the demonstrations to be adequately summarized, in particular in Alg 1 the function is_summarized() is not defined, nor is its operationalization defined anywhere in the text of the paper. Specification traditionally refers to a formal statement whose semantics are well known and the satisfaction of the specification with respect to an output is computationally well-defined and consistent. By not having a concrete definition of specification, the evaluations set up in the latter part of the paper suffer from lack of diversity and a lack of quantification of task difficulty.\n\n**Code as output language**: This work relies on the output code being interpretable, and this involves providing a set of primitive parametric functions to the LLM to use. The part of sensing the environment, and acting using perception feedback is something that is already programmed into the primitives. Coming up with an adequate and a competent set of parameterized primitives is challenging, and in this case is entirely the responsibility of the system designer.\n\n**Claims of generalization** Tthe generalization shown is only generalizing towards named entity substitutions, not to complex control flows and temporal specifications. Specifically, all the tasks shown here are a sequence of subgoals where performing any subgoal out of order will not preclude the agent from completing the subsequent actions. Further, there is no reactivity in the task specifications, and there are no avoidance tasks. All of these task specifications notions are very common in robotics and planning problems that this system has not been evaluated on. Refer to [1] for a survey on robotics mission types (also relevant to symbolic planning). All the tasks here are limited to the visit or sequenced visit type. Further while the submission claims that demo2code can generalize for complex long-horizon tasks, the maximum task length is quite smaller than state of the art for symbolic planning. Automatically translating textual domain descriptions to a formal domain description followed by the use of automated planners has already shown more reliable performance on harder problems [5]. Further there is quite a bit of evidence that LLMs cannot plan beyond the simplest of domains [6], and this line of research is unacknowledged in the submission.\n\n**Issues on evaluations of learning from demonstrations:** Generalization from demonstrations and language is a tricky subject. Usually demonstrations and language contain complementary sources of information. Therefore none of the system behaviors are incorrect in Figure 5, the core issue is that inductive learning is by definition an ill-posed problem, and many approaches to inductive learning have relied on Bayesian inference in the past. [2],[3]. Specifically, the issue of where to place the purple block (fig 5a) given the language description is underspecified, and the system had to forcibly ground the placement to any of the valid options to generate a trajectory. Committing to a valid assignment as was done by the lang2code model is one approach, and asking for resolution of referential ambiguity is another approach [4]. One might argue that learning to overconstrain the output based on a single demonstration is an example of overspecification.\n\n\n\n[1] - Menghi, C., Tsigkanos, C., Pelliccione, P., Ghezzi, C. and Berger, T., 2019. Specification patterns for robotic missions. IEEE Transactions on Software Engineering, 47(10), pp.2208-2224.\n\n[2] - Tenenbaum, J.B., 1999. A Bayesian framework for concept learning (Doctoral dissertation, Massachusetts Institute of Technology).\n\n[3] - Shah, A., Kamath, P., Shah, J.A. and Li, S., 2018. Bayesian inference of temporal task specifications from demonstrations. Advances in Neural Information Processing Systems, 31.\n\n[4] - Williams, Tom, Rafael C. Núñez, Gordon Briggs, Matthias Scheutz, Kamal Premaratne, and Manohar N. Murthi. \"A dempster-shafer theoretic approach to understanding indirect speech acts.\" In Advances in Artificial Intelligence--IBERAMIA 2014: 14th Ibero-American Conference on AI, Santiago de Chile, Chile, November 24-27, 2014, Proceedings 14, pp. 141-153. Springer International Publishing, 2014.\n\n[5] - Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J. and Stone, P., 2023. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477.\n\n[6] - Valmeekam, K., Olmo, A., Sreedharan, S. and Kambhampati, S., 2022. Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498.\n\nQUESTIONS:\nTo strengthen the paper the authors should carefully define the following:\n1. The role of the user/system developer. This includes definition of skill primitives that can be executed in the environment, definition of predicates that are sufficient to track the progress of the tasks. Developing perception systems that accurates record the predicate states\n2. Quantify task complexity using metrics from logic (expressing instructions in temporal logic, and measuring the size of the automaton). Using number of predicates, and actions and reporting them for each planning domain.\n3. Evaluate on diverse set of instructions taking inspiration from Menghi et al. [1] to come up with specification templates beyond sequenced visit.\n4. Report comparative performance against state-of-the-art symbolic planners. in comparably sized planning domains.\n\nLIMITATIONS:\nPlease see above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents a method that can take both demo and language in and teach LLM to perform new tasks. The idea makes sense and the algorithm is easy to understand and works very well. Evaluation results and ablations show improvement over existing works.

STRENGTHS:
The paper is well written and the idea is clear and easy to understand.

WEAKNESSES:
It's probably better to define what "task" is. For example, the authors claim "Demo2Code can generalize across complex, long-horizon tasks." However, for real world robotics tasks, I would imagine there will be a lot of corner cases and the code needs to handle it and therefore it won't generalize to those tasks.

QUESTIONS:
It looks like the method does not involve any network training / fine tuning, right?
If so, do you expect to get better performance by training / fine tuning LLM?

LIMITATIONS:
As the author said, the ability of this framework is limited by the capability of LLM.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06