SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Overall, Demo2Code has the closest performance to the oracle (Spec2Code). Specifically, our approach has the highest unit test pass rates in all three benchmarks, as well as the highest execution success in Robotouille (table 2) and EPIC-Kitchens (table 3). Meanwhile, Lang2Code [33] has a higher overall execution success than Demo2Code for the Tabletop simulator (table 1). However, Lang2Code has the lowest unit test pass rate among all baselines because it cannot fully extract users' specifications without demonstrations. DemoNoLang2Code has a relatively higher pass rate, but it sacrifices execution success because it is difficult to output plausible code without context from language. We provide prompts, detailed results and ablations in the Appendix. We now ask a series of questions of the results to characterize the performance difference between the approaches.
How well does Demo2Code generalize to unseen objects and tasks? Demo2Code exhibits its generalization ability in three axes. First, Demo2Code generalizes and solves unseen tasks with longer horizons and more predicates compared to examples in the prompt at train time. For Robotouille, table 2 shows the average horizon length for each training task (highlighted in gray) and testing task. Overall, the training tasks have an average of 12.7 states compared the testing tasks (31.3 states). Compared to the baselines, Demo2Code performs the best for long burger-making tasks (an average of 32 states) even though the prompt does not show this type of task. Second, Demo2Code uses control flow, defines hierarchical code, and composes multiple subtasks together to solve these long-horizon tasks. The appendix details the average number of loops, conditionals, and helper functions that Demo2Code generates for tabletop simulator (in section 8.3) and Robotouille (in section 9.3). Notably, Demo2Code generates code that uses a for-loop for the longest task (making two lettuce tomato burgers with 70 states), which requires generalizing to unseen subtasks (e.g. cutting tomatoes) and composing 7 distinct subtasks. Third, Demo2Code solves tasks that contain unseen objects or a different number of objects compared to the training tasks in the prompt. For Robotouille, the prompt only contains examples of preparing burger patties and lettuce, but Demo2Code still has the highest unit test pass rate for making burgers with unseen ingredients: cheese, chicken, and
$^{1}$ Codebase and usage guide for Robotouille is available here: https://github.com/portal-cornell/robotouille
$^{2}$ Codebase is available here: https://github.com/portal-cornell/demo2code
Figure 4: Demo2Code successfully extracts specificity in tabletop tasks. Lang2Code lacks demonstrations and randomly chooses a spatial location while DemoNoLang2Code lacks context in what the demonstrations are for.
Figure 5: Demo2Code summarizes demonstrations and identify different users' preferences on how to make a burger (e.g. whether to include lettuce or cheese) in Robotouille simulator. Then, it generates personalized burger cooking code to use the user's preferred ingredients.
tomatoes. Similarly, for tabletop (table 1), although the prompt only contains block-stacking tasks, our approach maintains high performance for cylinder-stacking tasks.
Is Demo2Code able to ground its tasks using demonstrations? Language instructions sometimes cannot ground the tasks with specific execution details. Since demonstrations provide richer information about the task and the world, we evaluate whether Demo2Code can utilize them to extract details. Tasks under the "Specific" cluster in Table 1 show cases when the LLM needs to use demonstrations to ground the desired goal. Fig. 4 illustrates that although the language instruction ("Place the purple cylinder next to the green block") does not ground the desired spatial relationship between the two objects, our approach is able to infer the desired specification ("to the left"). In contrast, Lang2Code can only randomly guess a spatial relationship, while DemoNoLang2Code can determine the relative position, but it moved the green block because it does not have language instruction to ground the overall task. Similarly, tasks under the "Hidden" cluster in Table 1 show how Demo2Code outperforms others in inferring hidden constraints (e.g. the maximum height of a stack) to ground its tasks.
Is Demo2Code able to capture individual user preference? As a pipeline for users to teach robots personalized tasks, Demo2Code is evaluated on its ability to extract a user's preference. Table 3 shows that our approach performs better than Lang2Code in generating code that matches each EPIC-Kitchens user's dish washing preference, without overfitting to the demonstration like in DemoNoLang2Code. Because we do not have a simulator that completely matches the dataset, human annotators have to manually inspect the code. The code passes the inspection if it has correct syntax, does not violate any physical constraints (e.g. does not rinse a dish without turning on the tap), and matches the user's dish-washing preference. Qualitatively, Fig. 6 shows that our approach is able to extract the specification and generate the correct code respectively for user 22, who prefers to soap
Figure 6: Demo2Code summarizes different styles of users washing dishes from demonstration (how to soap and rinse objects) in EPIC-Kitchens, and generates personalized dish washing code.
Figure 7: (Left) Unit test result for abating different degrees of chain-of-thought across tasks with short, medium, long horizon. (Right) Demo2Code's unit test result for Robotouil1e demonstrations with different level of noises: (1) each predicate has  $10\%$  chance of being dropped, and (2) each state has  $10\%$  chance of being completely dropped. We ran the experiment 4 times and report the average and variance.
all objects before rinsing them, and user 30, who prefers to soap then rinse each object individually. Similarly, Fig. 5 provides an example of how Demo2Code is able to identify a user's preference of using cheese vs lettuce even when the language instruction is just "make a burger." Quantitatively, Table 2 shows more examples of our approach identifying a user's preference in cooking order, ingredient choice, etc, while Table 1 also shows our approach performing well in tabletop tasks.
How does chain-of-thought compare to directly generating code from demonstrations? To evaluate the importance of our extended chain-of-thought pipeline, we conduct ablation by varying the length of the chain on three clusters of tasks: short-horizon (around 2 states), medium-horizon (5-10 states), and long-horizon ( $\geq$  15 states). We compare the unit test pass rate on four different chain lengths, ranging from No chain-of-thought (the shortest), which directly generates code from demonstrations, to Full (the longest), which represents our approach Demo2Code. The left bar plot in Fig. 7 shows that directly generating code from demonstrations is not effective, and the LLM performs better as the length of the chain increases. The chain length also has a larger effect on tasks with longer horizons. For short-horizon tasks, the LLM can easily process the short demonstrations and achieve high performances by just using 1-step. Meanwhile, the stark difference between 2-steps and Full's results on long-horizon tasks emphasizes the importance of taking as many small steps as the LLM needs in summarizing long demonstrations so that it will not lose key information.
How do noisy demonstrations affect Demo2Code's performance? We study how Demo2Code performs (1) when each predicate has a  $10\%$  chance to be removed from the demonstrations, and (2) when each state has a  $10\%$  chance to be completely removed. Fig. 7's table shows that Demo2Code's overall performance does not degrade even though demonstrations are missing information. While
removing predicates or states worsen Demo2Code's performance for shorter tasks (e.g. cook and cut), they surprisingly increase the performance for longer tasks. Removing any predicates can omit essential information in shorter tasks' demonstrations. Meanwhile, for longer tasks, the removed predicates are less likely to be key information, while reducing the length of demonstrations. Similarly, for the longest tasks to make two burgers, one burger's missing predicates or states can be explained by the other burger's demonstration. In section 11, we show a specific example of this phenomenon. We also study the effect of adding additional predicates to demonstrations, which has degraded Demo2Code's performance from satisfying 5 users' preferences to 2 users' in EPIC-Kitchens.
In this paper, we look at the problem of generating robot task code from a combination of language instructions and demonstrations. We propose Demo2Code that first recursively summarizes demonstrations into a latent, compact specification then recursively expands code generated from that specification to a fully defined robot task code. We evaluate our approach against prior state-of-the-art [33] that generates code only from language instructions, across 3 distinct benchmarks: a tabletop manipulation benchmark, a novel cooking game Robotouille, and annotated data from EPIC-Kitchens, a real-world human activity dataset. We analyze various capabilities of Demo2Code, such as grounding language instructions, generalizing across tasks, and capturing user preferences.
Demo2Code can generalize across complex, long-horizon tasks. Even though Demo2Code was shown only short-horizon tasks, it's able to generalize to complex, long demonstrations. Recursive summarization compresses long chains of demonstrations and recursive expansion generates complex, multi-layered code.
Demo2Code leverages demonstrations to ground ambiguous language instructions and infer hidden preferences and constraints. The latent specification explicitly searches for missing details in the demonstrations, ensuring they do not get explained away and are captured explicitly in the specification.
Demo2Code strongly leverages chain-of-thought. Given the complex mapping between demonstrations and code, chain-of-thought plays a critical role in breaking down computation into small manageable steps during summarization, specification generation and code expansion.
In future directions, we are looking to close the loop on code generation to learn from failures, integrate with a real home robot system and run user studies with Robotouille.
Demo2Code is limited by the capability of LLMs. Recursive summarization assumes that once all the demonstrations are sufficiently summarized, they can be concatenated to generate a specification. However, in extremely long horizon tasks (e.g. making burgers for an entire day), it is possible that the combination of all the sufficiently summarized demonstrations can still exceed the maximum context length. A future work direction is to prompt the LLM with chunks of the concatenated demonstrations and incrementally improve the specifications based on each new chunk. In recursive expansion, our approach assumes that all low-level action primitives are provided. Demo2Code currently cannot automatically update its prompt to include any new action. Another direction is to automatically build the low-level skill libraries by learning low-level policy via imitation learning and iteratively improve the code-generation prompt over time. Finally, since LLMs are not completely reliable and can hallucinate facts, it is important to close the loop by providing feedback to the LLM when they fail. One solution [62, 52] is to incorporate feedback in the query and reprompt the language model. Doing this in a self-supervised manner with a verification system remains an open challenge.
In addition, the evaluation approach for Demo2Code or other planners that generate code [33, 61, 77] is different from the one for classical planners [53, 54]. Planners that generate code measure a task's complexity by the horizon length, the number of control flows, whether that task is in the training dataset, etc. Meanwhile, many classical planners use domain specific languages such as Linear Temporal Logic (LTL) to specify tasks [41], which leads to categorizing tasks and measuring the task complexity based on LTL. Future work needs to resolve this mismatch in evaluation standards.
We sincerely thank Nicole Thean for creating our art assets for Robotouille. This work was supported in part by the National Science Foundation FRR (#2327973).
[1] Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acquired skills via goal generation. arXiv:2006.07185, 2020.
[2] Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. arXiv:1711.00482, 2017.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv:2108.07732, 2021.
[4] Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989, 2016.
[5] Cynthia Breazeal, Kerstin Dautenhahn, and Takayuki Kanda. Social robotics. Springer handbook of robotics, 2016.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[7] Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos. In Proceedings of Robotics: Science and Systems, Virtual, July 2021.
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.
[9] Qibin Chen, Jeremy Lacomis, Edward J. Schwartz, Graham Neubig, Bogdan Vasilescu, and Claire Le Goues. VarCLR: Variable semantic representation pre-training via contrastive learning, 2021.
[10] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Self-educated language agent with hindsight experience replay for instruction following. DeepMind, 2019.
[11] Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9052–9065, Online, November 2020. Association for Computational Linguistics.
[12] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pages 893-905. PMLR, 2022.
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV), 130:33-55, 2022.
[14] Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990–998. PMLR, 2017.
[15] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. Advances in neural information processing systems, 31, 2018.
[68] Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidzion, Justin Hart, Peter Stone, and Raymond Mooney. Jointly improving parsing and perception for natural language commands through human-robot dialog. JAIR, 2020.
[69] Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the 2015 International Joint Conference on Artificial Intelligence (IJCAI), pages 1923–1929, Buenos Aires, Argentina, July 2015.
[70] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. In International Conference on Learning Representations, 2019.
[71] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
[72] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, 2021.
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv:2201.11903, 2022.
[74] Terry Winograd. Procedures as a representation for data in a computer program for understanding natural language. MIT PROJECT MAC, 1971.
[75] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv:2109.10862, 2021.
[76] Jiajun Wu, Joshua B. Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[77] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models, 2023.
[78] Xiaojun Xu, Chang Liu, and Dawn Song. SQLNet: Generating structured queries from natural language without reinforcement learning, 2018.
[79] Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[80] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv:2204.00598, 2022.
[81] Luke S Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv:1207.1420, 2012.
[82] Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv:2205.10625, 2022.
8 Tabletop Manipulation Simulator Pipeline 17
8.1 Pipeline Overview 17
8.2 Experiment Setup 17
8.3 Characterize Tabletop Tasks' Complexity 18
9 Robotouille Simulator Pipeline 19
9.1 Overview 19
9.2 Experiment Setup 21
9.3 Characterize Robotouille's Tasks' Complexity 22
10 EPIC-Kitchens Pipeline 22
10.1 Annotations 22
10.2 Pipeline Overview 23
11 Noisy Demonstration Ablation Experiment 24
11.1 Randomly removing predicates/states 24
11.2 Randomly removing predicates/states 25
11.3 Quantitative Analysis 26
11.4 Qualitative Analysis 26
12 Chain-of-thought Ablation Experiment 27
12.1Experiment Detail 27
12.2 Quantitative Result 28
12.3 Qualitative example for a short-horizon task 28
12.4 Qualitative example for a medium-horizon task 29
12.5 Qualitative example for a long-horizon task 30
13 Intermediate Reasoning Ablation Experiment 35
13.1 Experiment detail 35
13.2 Quantitative result 35
13.3 Qualitative example 36
14 Recursive Expansion Ablation Experiment 40
14.1 Experiment detail 40
14.2 Quantitative result 40
14.3 Qualitative example 41
15 Broader Impact 44
16 Reproducibility 45
17 Demo2Code Example Output 45
17.1 Tabletop Simulator Example 45
17.2 Robotouille Example 48
17.3 EPIC-Kitchens Example 53
18 Prompts 57
18.1 Tabletop Manipulation Task Prompts 57
18.2 Robotouille Task Prompts 69
18.3 EPIC Kitchens Task Prompts 87
19 Other Long Examples 97
19.1 Example Robotouille Query 97
19.2 Example EPIC-Kitchens Query 101
19.3 Intermediate Reasoning Ablation Helper Functions 103
8 Tabletop Manipulation Simulator Pipeline
8.1 Pipeline Overview
The tabletop manipulation simulator contains simple tasks. Consequently, the demonstrations do not have too many states ( $\leq 8$  states) and the code is not complex. Thus, Demo2Code's prompt for this domain does not need a long extended chain-of-thought. In stage 1 recursive summarization, the LLM just needs to summarize each state into a sentences that describes the low-level action (e.g. move, pick, place, etc.) In stage 2 recursive expansion, because the code is simple, the LLM can directly use all the low-level actions that are provided to output the task code given a specification.
The prompt demonstrating this pipeline is listed at the end of the appendix in section 18.1.
8.2 Experiment Setup
In the paper, we categorize the tabletop tasks into three clusters. For each cluster, we list all the tasks and their variants of possible requirements below. The tasks that are used in the prompt are bolded.
- Specificity
- Place A next to B
* No hidden specificity: A can be placed in any relative position next to B
* A must be to the left of B
* A must be to the right of B
* A must be behind B
* A must be in front of B
- Place A at a corner of the table
* No hidden specificity: A can be placed at any corner.
* A must be at the top left corner
* A must be at the top right corner
* A must be at the bottom left corner
* A must be at the bottom right corner
- Place A at an edge of the table
* No hidden specificity: A can be placed at any corner.
* A must be at the top edge
* A must be at the bottom edge
* A must be at the left edge
* A must be at the right edge
- Hidden Constraint
- Place A on top of B
* No hidden constraint: A can be directly placed on top of B in one step
* There is 1 additional object on top of A, so that needs to be removed before placing A on top of B.
* There are 2 additional objects on top of A.
* There are 3 additional objects on top of A.
- Stack all blocks
* No hidden constraint: All blocks can be stacked into one stack
* Each stack can be at most 2 blocks high
* Each stack can be at most 3 blocks high
* Each stack can be at most 4 blocks high
- Stack all cylinders (Same set of hidden constraints as "stack all blocks." None of the examples appears in the prompt.)
- Personal Preference
- Stack all blocks into one stack
* 2 blocks must be stacked in a certain order, and the rest can be unordered
* 3 blocks must be stacked in a certain order
* All blocks must be stacked in a certain order
- Stack all cylinders into one stack (Same set of hidden constraints as "stack all blocks into one stack" None of the examples appears in the prompt.)
- Stack all objects
* No hidden preference: The objects do not need to be stacked in to different stacks based on their type
* All the blocks should be stacked in one stack, and all the cylinders should be stacked in another stack
8.2.1 Provided Low-Level APIs
We have provided the following APIs for the perception library and low-level skill library:
- Perception Library
- get_obj_names(): return a list of objects in the environment
- get_all_obj_names_that_MATCH_type(type_name, objects_list): return a list of objects in the environment that match the type_name.
-determine_final_stacking_order(objects_to_enforce_order, objects Without_order):return a sorted list of objects to stack.
- Low-level Skill Library
- put_first_on_second(arg1, arg2): pick up an object (arg1) and put it at arg2. If arg2 is an object, arg1 will be on top of arg2. If arg2 is 'table', arg1 will be somewhere random on the table. If arg2 is a list, arg1 will be placed at location [x, y].
stack_without_height_limit(object_to_stack): stack the list of objects_to_stack into one stack without considering height limit.
- stack_with_height_limit(objects_to_stack, height_limit): stack the list of objects_to_stack into potentially multiple stacks, and each stack has a maximum height based on height_limit.
8.3 Characterize Tabletop Tasks' Complexity