SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huaxiaoyue Wang
yukiwang@cs.cornell.edu
Gonzalo Gonzalez-Pumariega
gg387@cornell.edu
Yash Sharma
ys749@cornell.edu
Sanjiban Choudhury
sanjibanc@cornell.edu
Language instructions and demonstrations are two natural ways for users to teach robots personalized tasks. Recent progress in Large Language Models (LLMs) has shown impressive performance in translating language instructions into code for robotic tasks. However, translating demonstrations into task code continues to be a challenge due to the length and complexity of both demonstrations and code, making learning a direct mapping intractable. This paper presents Demo2Code, a novel framework that generates robot task code from demonstrations via an extended chain-of-thought and defines a common latent specification to connect the two. Our framework employs a robust two-stage process: (1) a recursive summarization technique that condenses demonstrations into concise specifications, and (2) a code synthesis approach that expands each function recursively from the generated specifications. We conduct extensive evaluation on various robot task benchmarks, including a novel game benchmark Robotouille, designed to simulate diverse cooking tasks in a kitchen environment. The project's website is at https://portal-cornell.github.io/demo2code/
How do we program home robots to perform a wide variety of personalized everyday tasks? Robots must learn such tasks online, through natural interactions with the end user. A user typically communicates a task through a combination of language instructions and demonstrations. This paper addresses the problem of learning robot task code from those two inputs. For instance, in Fig. 1, the user teaches the robot how they prefer to make a burger through both language instructions, such as "make a burger", and demonstrations, which shows the order in which the ingredients are used.
Recent works [24, 23, 33, 80, 61, 35] have shown that Large Language Models (LLMs) are highly effective in using language instructions as prompts to plan robot tasks. However, extending LLMs to take demonstrations as input presents two fundamental challenges. The first challenge comes from demonstrations for long-horizon tasks. Naively concatenating and including all demonstrations in the LLM's prompt would easily exhaust the model's context length. The second challenge is that code for long-horizon robot tasks can be complex and require control flow. It also needs to check for physics constraints that a robot may have and be able to call custom perception and action libraries. Directly generating such code in a single step is error-prone.
Our key insight is that while demonstrations are long and code is complex, they both share a latent task specification that the user had in mind. This task specification is a detailed language
Figure 1: Overview of Demo2Code that converts language instruction and demonstrations to task code that the robot can execute. The framework recursively summarizes both down to a specification, then recursively expands the specification to an executable task code with all the helper functions defined.
description of how the task should be completed. It is latent because the end user might not provide all the details about the desired task via natural language. We build an extended chain-of-thought [73] that recursively summarizes demonstrations to a compact specification, maps it to high-level code, and recursively expands the code by defining all the helper functions. Each step in the chain is small and easy for the LLM to process.
We propose a novel framework, Demo2Code, that generates robot task code from language instructions and demonstrations through a two-stage process (Fig. 1). (1) Summarizing demonstrations to task specifications: Recursive summarization first works on each demonstration individually. Once all demonstrations are compactly represented, they are then jointly summarized in the final step as the task specification. This approach helps prevent each step from exceeding the LLM's maximum context length. (2) Synthesizing code from the task specification: Given a task specification, the LLM first generates high-level task code that can call undefined functions. It then recursively expands each undefined function until eventually terminating with only calls to the existing APIs imported from the robot's low-level action and perception libraries. These existing libraries also encourage the LLM to write reusable, composable code.
Our key contributions are:
1. A method that first recursively summarizes demonstrations to a specification and then recursively expands specification to robot code via an extended chain-of-thought prompt.
2. A novel game simulator, Robotouille, designed to generate cooking tasks that are complex, long-horizon, and involve diverse food items, for benchmarking task code generation.
3. Comparisons against a range of baselines, including prior state of the art [33], on a manipulation benchmark, Robotouille, as well as a real-world human activity dataset.
Controlling robots from natural language has a rich history [74, 66, 37], primarily because it provides a natural means for humans to interact with robots [5, 30]. Recent work on this topic can be categorized as semantic parsing [39, 30, 69, 67, 55, 40, 68], planning [60, 22, 23, 24, 61, 35, 34, 28], task specification [64, 19, 58, 12], reward learning [46, 56, 7], learning low-level policies [46, 2, 57, 56, 7], imitation learning [25, 38, 58, 64] and reinforcement learning [26, 18, 10, 45, 1]. However, these approaches fall in one of two categories: generating open-loop action sequences, or learning closed-loop, but short-horizon policies. In contrast, we look to generate task code, which is promising in solving long-horizon tasks with control flows. The generated code also presents an interpretable way to control robots while maintaining the ability to generalize by composing existing functions.
Synthesizing code from language too has a rich history. Machine learning approaches offer powerful techniques for program synthesis [49, 4, 14]. More recently, these tasks are extended to general-purpose programming languages [79, 78, 8], and program specifications are fully described in natural English text [21, 3, 51]. Pretrained language models have shown great promise in code generation by exploiting the contextual representations learned from massive data of codes and texts [16, 11, 72, 71, 9, 47]. These models can be trained on non-MLE objectives [20], such as RL [32] to pass unit tests. Alternatively, models can also be improved through prompting methods
such as Least-to-Most [82], Think-Step-by-Step [29] or Chain-of-Thought [73], which we leverage in our approach. Closest to our approach is CodeAsPolicies [33], that translates language to robot code. We build on it to address the more challenging problem of going from few demonstrations to code.
We broadly view our approach as inverting the output of code. This is closely related to inverse graphics, where the goal is to generate code that has produced a given image or 3D model [76, 36, 15, 70, 17]. Similar to our approach [65] trains an LSTM model that takes as input multiple demonstrations, compresses it to a latent vector and decodes it to domain specific code. Instead of training custom models to generate custom code, we leverage pre-trained LLMs that can generalize much more broadly, and generate more complex Python code, even create new functions. Closest to our approach [77] uses pre-trained LLMs to summarize demonstrations as rules in one step before generating code that creates a sequence of pick-then-place and pick-then-toss actions. However, they show results on short-horizon tasks with small number of primitive actions. We look at more complex, long-horizon robot tasks, where demonstrations cannot be summarized in one step. We draw inspiration from [75, 50, 43] to recursively summarize demonstrations until they are compact.
We look at the concrete setting where a robot must perform a set of everyday tasks in a home, like cooking recipes or washing dishes, although our approach can be easily extended to other settings. We formalize such tasks as a Markov Decision Process (MDP),  $< S, \mathcal{A}, \mathcal{T}, \mathcal{R} >$ , defined below:
- State  $(s \in S)$  is the set of all objects in the scene and their propositions, e.g. open(obj1) ("obj1 is open"), on-top(obj1, obj2) ("obj1 is on top of obj2").
- Action  $(a \in \mathcal{A})$  is a primitive action, e.g. pick(obj) ("pick up obj"), place(obj, loc) ("place obj on loc"), move(loc1, loc2) ("move from loc1 to loc2").
- Transition function  $(\mathcal{T}(|s, a))$  specifies how objects states and agent changes upon executing an action. The transition is stochastic due to hidden states, e.g. cut('lettuce') must be called a variable number of times till the state changes to is_cut('lettuce').
- Reward function  $(r(s, a))$  defines the task, i.e. the subgoals that the robot must visit and constraints that must not be violated.
We assume access to state-based demonstrations because most robotics system have perception modules that can parse raw sensor data into predicate states [42, 27]. We also assume that a system engineer provides a perception library and an action library. The perception library uses sensor observations to maintain a set of state predicates and provides helper functions that use these predicates (e.g. get_obj_location(obj), is_cooked(obj)). Meanwhile, the action library defines a set of actions that correspond to low-level policies, similar to [33, 61, 77, 80].
The goal is to learn a policy  $\pi_{\theta}$  that maximizes cumulative reward  $J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}}\left[\sum_{t=1}^{T}[r(s_t, a_t)]\right]$ ,  $\theta$  being the parameters of the policy. We choose to represent the policy as code  $\theta$  for a number of reasons: code is interpretable, composable, and verifiable.
In this setting, the reward function  $r(s,a)$  is not explicit, but implicit in the task specification that the user has in mind. Unlike typical Reinforcement Learning (RL), where the reward function is hand designed, it is impractical to expect everyday users to program such reward functions for every new task that they want to teach their robots. Instead, they are likely to communicate tasks through natural means of interaction such as language instructions  $l$  (e.g. "Make a burger"). We can either use a model to generate reward  $r(s,a)$  from  $l$  [31] or directly generate the optimal code  $\theta$  [33].
However, language instructions  $l$  from everyday users can be challenging to map to precise robot instructions [63, 44, 81]: they may be difficult to ground, may lack specificity, and may not capture users' intrinsic preferences or hidden constraints of the world. For example, the user may forget to specify how they wanted their burger done, what toppings they preferred, etc. Providing such level of detail through language every time is taxing. A more scalable solution is to pair the language instruction  $l$  with demonstrations  $\mathcal{D} = \{s_1, s_2, \ldots, s_T\}$  of the user doing the task. The state at time-step  $t$  only contains the propositions that have changed from  $t - 1$  to  $t$ . Embedded in the states are specific details of how the user wants a task done.
Our goal is to infer the most likely code given both the language and the demonstrations: arg  $\max_{\theta} P(\theta | l, \mathcal{D})$ . For a long-horizon task like cooking, each demonstration can become long. Naively concatenating all demonstrations together to query the LLM can either exhaust the model's
Input: Language instructions lang, Demonstrations demos
Output: Final task code final_code that can be executed
Algorithm 1 Demo2Code: Generating task code from language instructions and demonstrations
def summarize(demos):
    if is_summarized(demos):
        all_demos = ""
    return llm.summary_prompt, all_demos)
    else:
        summarized_demos = []
        for demo in demos:
            summarized_demos.append(llm.summary_prompt, demo))
        return summarize.summary��emos)
context length or make directly generating the code challenging. We propose an approach that overcomes these challenges.
We present a framework, Demo2Code, that takes both language instructions and a set of demonstrations from a user as input to generate robot code. The key insight is that while both input and output can be quite long, they share a latent, compact specification of the task that the user had in mind. Specifically, the task specification is a detailed language description of how the task should be completed. Since our goal is to generate code, its structure is similar to a pseudocode that specifies the desired code behavior. The specification is latent because we assume that users do not explicitly define the task specification and do not provide detailed language instructions on how to complete the task.
Our approach constructs an extended chain-of-thought that connects the users' demonstrations to a latent task specification, and then connects the generated specification to the code. Each step is small and easy for the LLM to process. Algorithm 1 describes our overall approach, which contains two main stages. Stage 1 recursively summarizes demonstrations down to a specification. The specification and language instruction is then converted to a high-level code with new, undefined functions. Stage 2 recursively expands this code, defining more functions along the way.
4.1 Stage 1: Recursively Summarize Demonstrations to Specifications
Figure 2: Recursive summarization of input demonstrations to a compact specification. (Stage 1)
The goal of this stage is to summarize the set of demonstrations provided by the user into a compact specification (refer to summarize (demo) in Algorithm 1). Each demonstration is first independently summarized until the LLM determines that the demonstration can no longer be compressed, then the summaries are concatenated and summarized together. Fig. 2 shows example interim outputs during this stage. First, states in each demonstration get summarized into low-level actions (e.g. "patty6 is cooked" is summarized as "robot1 cooked patty6.") Then, low-level actions across time-steps are summarized into high-level subtasks, such as stacking, cutting, (e.g. "At state 3-8, the high level subtask is cook..."). The LLM determines to stop recursively summarizing after the entire demonstration gets converted to high-level subtasks, but this can have a different stopping condition (e.g. setting a maximum step) for task settings different than Fig. 2's. Next, these demonstrations' summaries are concatenated together for the LLM to generate the task specification. The LLM is prompted to first perform some intermediate reasoning to extract details on personal preferences, possible control loops, etc. For instance, the LLM aggregates high-level subtasks into an ordered list, which empirically helps the model to identify repeated subsets in that list and reason about control loops. An example final specification is shown in Fig. 2, which restates the language instruction first, then states "Specifically: ..." followed by a more detailed instruction of the task.
4.2 Stage 2: Recursively Expand Specification to Task Code
Figure 3: Recursive expansion of the high-level code generated from the specification, where new functions are defined by the LLM along the way. (Stage 2)
The goal of this stage is to use the generated specification from stage 1 to define all the code required for the task (see expand_code (code) in Algorithm 1). The LLM is prompted to first generate high-level code that calls functions that may be undefined. Subsequently, each of these undefined functions in the code is recursively expanded. Fig. 3 shows an example process of the code generation pipeline. The input is the specification formatted as a docstring. We import custom robot perception and control libraries for the LLM and also show examples of how to use such libraries in the prompt. The LLM first generates a high-level code, that can contain new functions, e.g. cook_obj_atLoc, that it has not seen in the prompt or import statements before. It expands this code by calling additional functions (e.g. movethen_pick), which it defines in the next recursive step. The LLM eventually reaches the base case when it only uses imported APIs to define a function (e.g. move then pick).
Baselines and Metrics We compare our approach Demo2Code against prior work, CodeAsPolicies [33], which we call Lang2Code. This generates code only from language instruction. We also compare against DemoNoLang2Code that generates code from demonstrations without a language instruction, which is achieved by modifying the LLM prompts to redact the language. Finally, we also compare to an oracle Spec2Code, which generates task code from detailed specifications on how to complete a task. We use gpt-3.5-turbo-16k for all experiments with temperature 0.
We evaluate the different methods across three metrics. Execution Success Rate is the average 0/1 success of whether the generated code can run without throwing an error. It is independent from whether the goal was actually accomplished. Unit Test Pass Rate is based on checking whether all subgoals are achieved and all constraints are satisfied. The unit test module checks by examining the state transitions created from executing the generated code. Code BLEU score is the BLEU
Table 1: Results for Tabletop Manipulation simulator. The tasks are categories into 3 clusters: Specificity ("Specific"), Hidden World Constraint ("Hidden"), and Personal Preference ("Pref").
Task	Lang2Code[33]	DemoNoLang2Code	Demo2Code(ours)
Exec.	Pass.	BLEU.	Exec.	Pass.	BLEU.	Exec.	Pass.	BLEU.
Specific	Place A next to B	1.00	0.33	0.73	0.90	0.80	0.82	1.00	1.00	0.98
Place A at a corner of the table	1.00	0.30	0.08	1.00	1.00	0.85	1.00	1.00	1.00
Place A at an edge of the table	1.00	0.20	0.59	1.00	0.95	0.84	1.00	1.00	0.84
Hidden	Place A on top of B	1.00	0.03	0.23	0.60	0.70	0.56	0.90	0.40	0.40
Stack all blocks	1.00	0.20	0.87	1.00	0.70	0.50	1.00	0.70	0.50
Stack all cylinders	1.00	0.37	0.89	1.00	0.83	0.49	1.00	1.00	1.00
Prefix	Stack all blocks into one stack	1.00	0.13	0.07	1.00	0.67	0.52	1.00	0.87	0.71
Stack all cylinders into one stack	1.00	0.13	0.00	0.90	0.77	0.19	1.00	0.90	0.58
Stack all objects into two stacks	1.00	0.00	0.00	1.00	0.90	0.68	1.00	0.90	0.65
Overall	1.00	0.19	0.39	0.93	0.81	0.60	0.99	0.88	0.77
score [48] between a method's generated code and the oracle Spec2Code's generated code. We utilize each code by the spaces, quotations, and new lines.
Tabletop Manipulation Simulator [80, 23] We build upon a physics simulator from [80, 23], which simulates a robot arm manipulating blocks and cylinders in different configurations. The task objectives are to place objects at specific locations or stack objects on top of each other. The LLM has access to action primitives (e.g. pick and place) and perception modules (e.g. to get all the objects in the scene). We create a range of tasks that vary in complexity and specificity, use the oracle Spec2Code to generate reference code, and execute that code to get demonstrations for other methods. For each task, we test the generated code for 10 random initial conditions of objects.
Table 2: Results for the Robotouille simulator. The training tasks in the prompt are at the top of the table and highlighted in gray. All tasks are ordered by the horizon length (the number of states). Below the table shows four Robotouille tasks where the environments gradually become more complex.
Task	Lang2Code[33]	DemoNoLang2Code	Demo2Code(ours)	Horizon Length
Exec.	Pass.	BLEU.	Exec.	Pass.	BLEU.	Exec.	Pass.	BLEU.
Cook a patty	1.00	1.00	0.90	1.00	1.00	0.90	1.00	1.00	0.90	8.0
Cook two patties	0.80	0.80	0.92	0.80	0.80	0.92	0.80	0.80	0.92	16.0
Stack a top bun on top of a cut lettuce on top of a bottom bun	1.00	1.00	0.70	0.00	0.00	0.75	1.00	1.00	0.60	14.0
Cut a lettuce	1.00	1.00	0.87	0.00	0.00	0.76	1.00	1.00	0.87	7.0
Cut two lettuces	0.80	0.80	0.92	0.00	0.00	0.72	0.80	0.80	0.92	14.0
Cook first then cut	1.00	1.00	0.88	1.00	1.00	0.88	1.00	1.00	0.88	14.0
Cut first then cook	1.00	1.00	0.88	0.00	0.00	0.82	1.00	1.00	0.88	15.0
Assemble two burgers one by one	0.00	0.00	0.34	1.00	1.00	0.77	1.00	1.00	0.76	15.0
Assemble two burgers in parallel	0.00	0.00	0.25	1.00	1.00	0.51	0.00	0.00	0.71	15.0
Make a cheese burger	1.00	0.00	0.24	1.00	1.00	0.69	1.00	1.00	0.69	18.0
Make a chicken burger	0.00	0.00	0.57	0.00	0.00	0.64	0.90	0.90	0.69	25.0
Make a burger stacking lettuce atop patty immediately	1.00	0.00	0.74	0.20	0.00	0.71	0.00	0.00	0.71	24.5
Make a burger stacking patty atop lettuce immediately	0.00	0.00	0.74	0.20	0.00	0.71	1.00	1.00	0.74	25.0
Make a burger stacking lettuce atop patty after preparation	1.00	0.00	0.67	0.10	0.00	0.65	0.00	0.00	0.66	26.5
Make a burger stacking patty atop lettuce after preparation	1.00	0.00	0.67	0.00	0.00	0.53	1.00	0.00	0.69	27.0
Make a lettuce tomato burger	0.00	0.00	0.13	1.00	1.00	0.85	1.00	0.00	0.66	34.0
Make two cheese burgers	0.00	0.00	0.63	1.00	1.00	0.68	1.00	1.00	0.68	38.0
Make two chicken burgers	0.00	0.00	0.52	0.00	0.00	0.68	1.00	0.00	0.56	50.0
Make two burgers stacking lettuce atop patty immediately	0.80	0.00	0.66	0.80	1.00	0.69	0.00	0.00	0.66	50.0
Make two burgers stacking patty atop lettuce immediately	0.80	0.00	0.67	1.00	0.00	0.48	1.00	1.00	0.73	50.0
Make two burgers stacking lettuce atop patty after preparation	0.80	0.00	0.66	0.60	0.00	0.66	0.80	0.00	0.67	54.0
Make two burgers stacking patty atop lettuce after preparation	0.80	0.00	0.67	0.50	0.00	0.71	0.80	0.00	0.68	54.0
Make two lettuce tomato burgers	1.00	0.00	0.55	0.00	0.00	0.70	1.00	1.00	0.84	70.0
Overall	0.64	0.29	0.64	0.49	0.38	0.71	0.79	0.59	0.74	28.8
Table 3: Results for EPIC-Kitchens dataset on 7 different user demonstrations of dish-washing (length of demonstration in parentheses). The unit test pass rate is evaluated by a human annotator, and BLEU score is calculated between each method's code and the human annotator's reference code.
P4-101 (7)	P7-04 (17)	P7-10 (6)	P22-05 (28)	P22-07 (30)	P30-07 (11)	P30-08 (16)
Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.	Pass.	BLEU.
Lang2Code [33]	1.00	0.58	0.00	0.12	0.00	0.84	0.00	0.48	0.00	0.37	1.00	0.84	0.00	0.66
DemoNoLang2Code	0.00	0.00	0.00	0.00	1.00	0.00	0.00	0.37	0.00	0.51	1.00	0.57	0.00	0.00
Demo2Code	1.00	0.33	0.00	0.19	1.00	0.63	1.00	0.43	1.00	0.66	1.00	0.58	0.00	0.24
Cooking Task Simulator: Robotouille  ${}^{1}$  We introduce a novel,open-source simulator to simulate complex, long-horizon cooking tasks for a robot, e.g. making a burger by cutting lettuces and cooking patties. Unlike existing simulators that focus on simulating physics or sensors, Robotouille focuses on high level task planning and abstracts away other details. We build on a standard backend, PDDLGym [59], with a user-friendly game as the front end to easily collect demonstrations. For the experiment, we create a set of tasks, where each is associated with a set of preferences (e.g. what a user wants in the burger, how the user wants the burger cooked). For each task and each associated preference, we procedurally generate 10 scenarios.
EPIC-Kitchens Dataset [13] EPIC-Kitchens is a real-world, egocentric video dataset of users doing tasks in their kitchen. We use this to test if Demo2Code can infer users' preferences from real videos, with the hopes of eventually applying our approach to teach a real robot personalized tasks. We focus on dish washing as we found preferences in it easy to qualify. While each video has annotations of low-level actions, these labels are insufficient for describing the tasks. Hence, we choose 7 videos of 4 humans washing dishes and annotate each demonstration with dense state information. We compare the code generated by Lang2Code, DemoNoLang2Code and Demo2Code on whether it satisfies the annotated preference and how well it matches against the reference code.