Teaching Robots with Show and Tell: Using Foundation Models to Synthesize Robot Policies from Language and Visual Demonstration
Authors: Michael Murray
Abhishek Gupta
Maya Cakmak
Venue: CoRL 2024 (2024)
Keywords: learning from demonstration
language model planning
neuro-symbolic reasoning

Abstract: We introduce a modular, neuro-symbolic framework for teaching robots new skills through language and visual demonstration. Our approach, ShowTell, composes a mixture of foundation models to synthesize robot manipulation programs that are easy to interpret and generalize across a wide range of tasks and environments. ShowTell is designed to handle complex demonstrations involving high level logic such as loops and conditionals while being intuitive and natural for end-users. We validate this approach through a series of real-world robot experiments, showing that ShowTell out-performs a state-of-the-art baseline based on GPT4-V, on a variety of tasks, and that it is able to generalize to unseen environments and within category objects.

Teaching Robots with Show and Tell: Using Foundation Models to Synthesize Robot Policies from Language and Visual Demonstrations
Michael Murray
Abhishek Gupta
Maya Cakmak
Paul G. Allen School of Computer Science and Engineering
University of Washington (mmurr, abhgupta, mcakmak @cs. washington. edu
Abstract: We introduce a modular, neuro-symbolic framework for teaching robots new skills through language and visual demonstration. Our approach, SHOWTELL, composes a mixture of foundation models to synthesize robot ma- nipulation programs that are easy interpret and generalize across a wide range of tasks and environments. SHOWTELL designed handle complex demonstra- tions involving high level logic such loops and conditionals while being intu- itive and natural for end-users. We validate this approach through series of real- world robot experiments, showing that SHOWTEL out out-performs state-of-the-art baseline based on GPT4-V, on a variety of tasks, and that it is able to generalize to unseen environments and within category objects. Supplementary materials and videos are available on our webpage:
Keywords: learning from demonstration, language model planning, neuro- symbolic reasoning
1 Introduction
General purpose robots have the potential to enhance productivity and reliability in human-centric, task-oriented settings such as kitchens, warehouses, and offices, but one of the key challenges to achieving this potential is that each environment, user, and task combination demands tailored be- havior from the robot. Programming by Demonstration (PbD) a popular approach to this chal- lenge, enabling end-users to program robots for personalized tasks and environments by providing demonstrations of the desired behavior. However, existing PbD techniques typically require large unable to extract high-level task information related to control
number of demonstrations flow and action parameterization from the demonstrations. Natural language is another popular method for personalizing robot behavior, but low-level details can be cumbersome and error prone to communicate with language alone, and most works assume fixed set of low-level primitives are available to be composed by the language instructions, limiting the scale and complexity of tasks that can be programmed.
Studies in human observational learning show that humans both language and demonstration when teaching each other new tasks, with both communication modalities playing important roles [1, 2]. Language allows us to transmit abstract information, while demonstrations instantiate that information in concrete examples [3]. Inspired by this insight, we envision a more natural and intuitive PbD system, where end-users can program robots for personalized tasks and environments by flexibly using both language and visual demonstrations.
In this work, we propose SHoWTELL system that enables end-users to teach robots new tasks the same way they would teach another person, by visually demonstrating and verbally describing what they are doing as they demonstrate. SHOWTELL is modular, composing a set of pre-trained large language models (LLMs) and vision-language models (VLMs) to synthesize robot policies that
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

can jointly reason about the language and visual components of the demonstrations and execute the demonstrated behavior in novel scenes. SHOWTELL is designed to be generalizable across a wide range of tasks and environments, and to be intuitive and natural for end-users to use, while requiring only a single demonstration with no additional training or fine-tuning.
While both language and demonstration are traditionally popular interfaces for task planning, there has been comparatively little attention paid to combining the two. Most existing works attempt to train end-to-end models, an approach that is difficult to scale because it requires enough training examples to implicitly learn to understand all demonstrations and perform all tasks within the for- ward pass of a neural network. In contrast, our approach requires no additional training, utilizing the vast knowledge encoded in off-the-shelf foundation models. Our approach is neuro-symbolic in that it aims combine the strengths of neural networks, which excel at learning from data, with symbolic reasoning, which excels manipulating symbols and logical rules, create more robust robot PbD system that is modular, allowing us to scale to new tasks by composing existing mod- ules in novel ways, and also also interpretable, allowing us to understand the reasoning behind the generated policies and to easily modify or extend them.
Through real-world robot experiments, we validate our approach, showing that SHOWTELL is able to synthesize robot policies from language and visual demonstrations including tasks that require high level logic such as conditions, iteration, and segmentation. We show that our approach out- performs a state-of-the-an baseline on a variety of tasks, and that it is able to generalize to new objects and environments, while requiring only a single demonstration with no additional training or fine-tuning. We believe that our approach has the potential to significantly improve the usability and performance robot PbD systems, and to enable robots to learn from demonstrations that are more natural and intuitive for end-users.
2 Related Work
Programming by Demonstration. Programming by Demonstration, also referred to as Learning from Demonstration or Imitation Learning, has been the subject of four decades of robotics research [4, 5]. Approaches are often categorized based on the method of providing demonstrations and in contrast to methods that require moving the robot (e.g. through teleoperation [6, 7, 8] kinesthetic teaching [9, 10, I1J, or the use of specialized demonstration hardware [12], in this work we focus on programming by passive observation, where the robot is programmed by observing a human perform the desired behavior [13, 14, 15, 16, 17, 18]. Most existing works attempt to train end- to-end models, an approach that is difficult to scale because it requires enough training examples to implicitly learn to understand all demonstrations and perform all tasks within the forward pass of a neural network. Instead, we propose leverage the prior knowledge encoded in pre-trained foundation models to synthesize programs that can both reason about demonstrations and execute the demonstrated behavior on a robot with novel scenes and objects.
LLMs for Task and Motion Planning. With the advent of large-scale pre-trained language models, there has been growing interest in using these models for robotics tasks. large body work has focused on planning and reasoning from text-based natural language instructions [19, 20, 21, 22, 23]. These works typically output their plans as a sequence of robot actions, but recent approaches show the benefits of using LLMs to synthesize code with logical constructs that be executed by robot [24, 25, 26). While much progress has been made in synthesizing code policies from text- based instructions, there has been comparatively little work on synthesizing code policies from visual demonstrations. Wang et al. [27] assume that a demonstration has been converted to a textual description, and focus on generating policies from the text. In contrast, we focus on generating policies directly from visual demonstrations. Most similar to our work, Wake et al. [28] propose to use a VLM summarize visual demonstrations and generate policies from the resulting summaries. In contrast, propose synthesize programs that reason about the visual demonstration, which allows us to handle more complex demonstrations, better align the language with the visual

---

LLM
Processing
a,
Perception APIs
botle tabrer
Language Narration
Viual Dono
Osservatio
Figure 1: An overview of the SHOWTELL framework. First, the visual and spoken components of the demo are processed and fed into an LLM. The LLM synthesizes modular program that can jointly reason about the provided demonstration and novel observations transfer the demonstrated skill to new scenes.
components of the demonstration, and more easily interpret the reasoning behind the generated policies.
LLMs for Visual Reasoning. Visual reasoning approaches have typically used end-to-end trained models, but recent works have shown that pre-trained LLMss can be leveraged to accomplish state-of- the-art for many visual reasoning tasks. Early iterations represent the visual information from images as text via captions, objects, and attributes and feed this textual representation to LLMs along with task instructions and in-context examples [29]. Zeng al. [30] propose a modular approach wherein the LLM leverages other pre-trained models such as vision-language models (VLMs) and audio- language models through program generation. More recent works have scaled this idea additional tasks including knowledge tagging, image editing, and causal/temporil reasoning over videos [31, 32). In this work we seek to leverage LLMs for visual reasoning in the context of robot PhD by using the LLM to generate modular programs that jointly reason over video demonstrations and spoken language instructions to understand the task being demonstrated and ground that understanding to robot actions in new scene observations.
3 Method
We present SHOWTELL, a neuro-symbolic robot PbD framework for synthesizing modular, gen- eralizable, and interpretable robot manipulation programs from visual demonstrations and natural language. Our approach, illustrated in Figure requires only single demonstration with no ad- ditional training or fine-tuning, it utilizes combination of pre-trained foundation models and hand-engineered components to reason about observed demonstrations and to transfer learned skills to new scenes. In the following sub-sections we first formalize the problem setting, then we provide a high-level overview of the approach, and finally we describe each phase of the approach.
3.1 Problem Formulation
In this work, we consider multi-modal PbD for robotic manipulation tasks. Let A be the set of robot actions, and S the set of world states. We assume access to human demonstration consisting of visual demonstration component D = (do, d1, dr.)). ). where each demonstration frame de is an RGB-D image at time f, and a spoken language component L. Given the demonstration D, language L, and an initial state so E S, the goal is to generate an execution & (So- ag. $1, aI,..., ........,y ST,, aT, ) where 0e E A is an action taken by the robot at time t, s, E S the state before taking a,, and 8+11 = T (s:,, a:) under environment dynamies environment layout, the poses and states of objects, and the pose and state of the robot. The robot does not directly have access to the state st, but only to observation or- An observation ot (Tz, K,) includes an RGB-D camera image I, and the robot's proprioceptive state K,, The task
A - The state 8, defined by the

---

is considered successful if the goal coal-conditions corresponding to the demonstration D and spoken language I. are true at the final state STg'
3.2 Overview
Our method utilizes mixture of pre-trained foundation models and hand-engineered modules to both understand the provided demonstration and to transfer that understanding to new scenes. We first pre-process the demonstration to make relevant information readily accessible for program syn- thesis. Next, an L.M is used to compose a set of modules into program. The modules available to the LLM include tools for visual and spatio-temporal reasoning about the video demonstration frames, tools for aligning the spoken language of the demonstration with the visual demonstration, tools for transferring the demonstrated skill to the current environment, and tools for controlling the robot.
3.3 Pre-prose pre-processing
The first step our approach is to pre-process the demonstration to make relevant information readily accessible the LLM. The spoken component of the demonstration, L, transcribed using the Whisper speech-to-tex model [33]. The transcription is saved and uttered words indexed by their timestamp relative the start of the demonstration. The visual component of the demonstration is processed to detect human hands and their interactions with objects in the scene. We utilize the Mediapipe [34] hand landmark detection model, and detect the human hand pose represented as 21 landmarks following the topology in [35]. The two landmarks on the thumb and another two on the index finger are used to represent a parallel jaw robot gripper.
For robot manipulation tasks, timesteps in which the end-effector interacts with objects in the envi- ronment are particularly important, or clusters of timestamps where the hand is in contact with an object, from the demonstration D. We extract information about the hands in the scene and the objects that they contact using 100DOH [36], hand-object interaction model that has been pre-trained on 100K images extracted from large-scale video dataset of humans interacting with objects. We use 100DOH to extract, for each demonstration frame, hand bounding box, in-contact object bounding box, and boolean contact variable indicating whether the hand is in contact with object not. We obtain fine grained masks for both hands and objects using Segment Anything Model (SAM) [37] with bounding boxes provided as prompts. 3D perception of the scene is crucial for manipulation tasks, we additionally produce point cloud for each demonstration frame using the RGB-D image and camera intrinsics.
seek to identify and extract contiguous contact sequences,
3.4 Program Synthesis
SHOWTELL uses powerful code-generating language model (GPT-4 [38]) to synthesize pro- grams that can both reason about a demonstra- tion and execute the demonstrated behavior in new scene. The LLM provided with prompt that consists of import statements and API doc- umentation that specifies the available func- tions, a brief summary of contact sequences ex- tracted during the pre-processing phase, and the full transcribed narration. The program is run with the Python interpreter, allowing the use of control flow tools like for loops, conditionals like if/else, built in functions like sort, and built in modules such datetime math. All of the information extracted during the pre- reeprocessing phase is made available in addition suite of modules. The additional modules made available to the program include mixture of pre- pre-trained
Perception
@cthandLardeario, GettianSbb (enteaeeeeeeenennneeeeeeeet GetGenturaftef, DetectCh Cectss Trackfesk, VQA
Spenchnligmentt
GelT Tapeacaapp Ge GenTineeeepo
Novel Scons Transfer
Gettheerustion. Transferiitox. TransferMark
Transie /EBCofa Trans TransferContectSeqp.co Anyärasp
PoveEE,oPsse, NoveEERel, NoveESGuarded,
Adjusttr ipperNidth
Figure 2: Taxonomy of modules available to the
synthesized cyurolyymbolic programs.

---

foundation models and hand-engineered modules, including modules for perception, speech align- ment, transferring to novel scenes, and controlling the robot. In the following sections we describe each the custom modules and their APIs and a taxonomy of the modules is provided in Figure 2.
3.4.1 Perception
The perception modules made available to the neuro-symbolic program enable extracting informa- tion about both the visual demonstration, D, and the new scene observations, (0o. O1...., or, ) The perception modules include APIs for accessing visual information made available in the pre- processing step via SeclaandDDjerrContattteeeeee
The perception modules also include APIs for free-form visual reasoning, DetectObjects, TrackMask, and VQA. The DetectObjecta module uses ViLD [39], pre-trained open-world object masking VLM. Detected masks can be tracked to new frames and scenes with the TrackMask module, which uses the XMem mask track- ing model under the hood [40]. The VQA module is a pre-trained visual question answering VLM, BLIP-v2 [41], that can be used to answer questions about the visual scene.
3.4.2 Language Alignment
To understand a multi-modal demonstration, the robot must align the spoken language of the demon- stration with the visual demonstration. For this purpose, provide the LLM with two modules, and
which are used to extract the start and end timestamps of spoken words from the language component of the demonstration, L, and enable the program to align the language with timestamps corresponding to frames in the visual demonstration, D.
3.4.3 Novel Scene Transfer
Ultimately, the goal is to transfer the policy generated from the demonstration to new environments. To do this, the robot must match the visual demonstration D) to the current environment observed in or. The Transf erons, TransferMask, TransferEEConfigg modules are used to transfer detections to new scene observations. These modules find correspond- ing reference points by leveraging features from Stable Diffusion [42], a pre-trained image diffusion model which have been shown implicitly encode rich information about the structure of objects within image, and have been shown to be highly effective for finding corresponding points for visual reasoning tasks [43, 44, 45). Using these features, we can match similar points within- category objects in addition to exact points. For example, after demonstrating how to pick up a mug by its handle, the robot should be able to repeat this skill for visually distinct mugs and mugs of different sizes.
In order to facilitate grasping of objects in the new scene, we provide the AnyGrasp module, which is used to find the best grasp configuration for particular object mask in the scene. This module uses pre-trained grasp prediction model [46] to predict the best grasp configuration for particular object based on its point cloud, and can be used to find grasp configurations for objects that were not present in the demonstration.
3.4.4 Robot Control
To control the robot we provide set of modules to specify end-effector goal poses and configu- rations. The modules MoveεEToPose is used to specify goal EE pose in 3D space, MoveEERel used to specify a goal relative to the current EE pose, and MoveEEGuarded performs a guarded movement along a given vector until contact occurs. The module RotateEE is used to specify goal EE rotation. And finally, AdjustGriiperWideh is used to specify desired gripper width to open and close the gripper. These modules are used to specify the robot's end-effector goal poses and con- figurations, which used to generate the robot's 's actions during program execution described in Section 3.5.
5

---

Figure 3: We evaluate SHOW TELL using Stretch RE2 robot to perform 16 real world manipulation
tasks across visually distinct environments including a conference room, a kitchen, a classroom, an office lounge, and cluttered workbench. We show that this approach is effective for teaching manipulation tasks requiring high level logic such as conditions, iteration, and segmentation.
3.5
Skill Execution
After program synthesis, the resulting program is run with Python interpreter and its execution is simple Python call. During execution, correspondence matching modules evaluated transfer reference points identified in the demonstration the new scene observation, and all waypoints redefined relative to the reference points in the new scene. To interpolate robot motion between end-effector goal poses, we use collision-free motion planner to generate trajectory of robot actions for reaching the next desired waypoint goal. Specifically, we use the GPU accelerated motion generation library cuRobo [47].
4 Experimental Setup
Hardware and Environments:
To evaluate approach, conduct series of real world exper- iments with Stretch RE2 robot [48] across 5 indoor environments illustrated in Figure 3. The robot's mobile base, arm lift, and telescoping arm are moved in conjunction to reach 6-DOF target waypoints. The robot's end effector is parallel-jaw gripper with rubber fingertips. An Intel Re- alSense D435i RGB-D camera is mounted to the frame which used both to record demonstrations and to provide observations during execution.
Baselines and Experiments: We evaluate our approach against CPTT4V-Robal [28], state-of-the- art method for robot task planning from language and visual demonstrations using LLMs. In this baseline approach, demonstration frames are directly fed to VLM (GPT4-V [38]), which generates a summary that is used by the LLM to generate policy. To study the contribution of the visual and spoken components the demonstration we perform series of ablation experiments. First we evaluate a language-only baseline, ShowTelll-N with no access to the visual demonstration, similar to the approach proposed by Liang et al. [24]. We also include a vision-only baseline, ShowTell-NoLane which uses the visual demonstration only generate policies without access to the language component.
We evaluate SHOWTELL using a set of 16 real world manipulation tasks across visually distinct environments. We group evaluation tasks based on the challenge conditions they present. First we evaluate with simple demonstrations that are straightforward to follow. including pick and place, stacking cubes, and opening drawers. Next we evaluate tasks involving control flow including iterative tasks, for example "move all of the blocks over to this container". and tasks with conditionals, for example "open the drawer only if it is closed". Finally. we evaluate tasks that require segmenting the demonstration, for example if the demonstrator makes an error and corrects themselves, the robot is required to segment the demonstration into correct and incorrect segments to generate a policy that correctly imitates only the desired behavior. All the tasks, including the simple tasks, can require the robot to jointly reason about the visual and spoken components the demonstration to resolve ambiguities. For example, an instruction like "pick up this mug and move it over here" requires aligning the spoken instruction with the visual demonstration to determine which mug to pick up and where to move it.
Evaluation Tasks:

---

Human Gemonstration
Synthesieed Program
the drawer lkk this, but only the drawer already open"
(a)
cta un carg, ang contage:
(:venneor crp.ptas.
*Pick all the bottles up like this end put them in this container
betrr porert
for bettle bettice
(b)
target desttttlee.isttttttttt
Re-escto cra craupees)
"Grab the like Wait, actually, misiaake.
first valid case out Takeer sapa "speeciinn' chone" sustake?
Trandferdenacttee
Aesustte (peertetts) wodthh Mevattllnna)
Figure 4: Qualitative examples of code synthesized by SHOWTELL for a set of representative demonstrations. The representative demonstrations show the ability to follow high level logic in- cluding (a) conditionals (b) iteration and (c) segmentation.
ShowTell--No
ShowTelll-N Vis
GPT4-V-Robol [28]
ShowTTel (ours)
Demo Type
GCR
SR
GCR
SR
GCR
SR
GCR
SR
Simple
0.89
0.85
0.86
0.81
0.88
0.85
0.96
0.94
Iterative
0.31
0.12
0.61
0.59
0.43
0.22
0.94
0.85
Conditional
0.41
0.39
0.64
0.64
0.41
0.41
0.93
0.93
Segmented
0.12
0.06
0.48
0.42
0.20
0.18
0.93
0.91
Table I: Real-world robot manipulation task performance across different task families.
We use two primary metrics evaluate system performnace: Success Rate (SR) and Goal Condition Recall (GCR). The task-relevant goal conditions are the set of state changes that must be satisfied at the end of an episode for the task to be considered successful. SR is the fraction of rollouts for which the object positions and state changes completely satisfy the task goal-conditions at the end of the action sequence. GCR is the fraction of goal-conditions successfully completed at the end of an episode to those necessary to have finished a task.
Metrics:
Results
In experiments seek to answer the following research questions: Is SHOWTELL practi- cal for teaching range of robot manipulation skills including those that require high level logic like conditions, iteration, and segmentation? 2) How does SHOWTELL compare to existing state- of-the-art methods for generating robot policies from language and visual demonstrations? 3) Is SHOWTELL able to generalize to unseen environments and within category objects? 4) To what extent do the visual and spoken components the demonstration contribute the performance?
To evaluate the performance of SHOWTELL, demonstrations are provided (2 demonstrations in each environment) and rollouts are performed for each demonstration, resulting in a total of 50 rollouts per task. We group tasks based on challenge conditions as described in section and for each task family, we report the average SR and GCR across all rollouts.
perform total of 50 rollout trials per task: 10
The results summarized in Table show that SHOWTELL outperforms existing state-of-the- art methods for generating robot policies from language and visual demonstrations. Even for
7

---

simple tasks, SHOWTELL outperforms the baseline methods, the modular, neuro-symbolic programs are more easily able to reason about the demonstration and to resolve ambigui- ties than the monolothic VLM used to summarize demonstrations the baseline method. This advantage is even more pronounced for the other task families as the purely sequential structure of the baseline method is not well suited to tasks that require high level logic like conditions, iteration, and segmentation. The qualitative examples in Figure 4 illustrate the ability of SHOWTELLI to synthesize programs from such demonstrations. Comparing the ablation experiments, we see that the visual and spoken components of the demonstration are both important for gener- ating effective policies, the ablation experiments ShowTell- NoVis and ShowTell-NoLang perform poorly compared the full SHOWTELL approach, even for simple tasks. This is be- cause the visual and spoken components of the demonstration are often complementary, and help to resolve ambiguities present in the individual components of the demonstration. To better un- derstand the failure cases, we analyze the failures in Section C of the appendix and illustrate the distribution failures in Figure 7. We finally evaluate the performance of SHOWTELL across roll- outs in the canonical scene (the scene used for demonstration), with unseen objects, and in unseen environments. The results summarized in Table 2 indicate that SHOWTELL is able generalize to unseen environments and within category objects, while requiring only single demonstration with no additional training fine-tuning.
Rollout
SR
Canonical
0.92
Unseen environment
0,84
Unseen objects
0.88
Table 2: Performance across roll-
outs: in the canonical scene, with unseen objects, and in unseen en- vironment.
6 Limitations
Our approach has several limitations. Most crucially, the approach is constrained by the limitations of the pre-trained models it uses, although we note that the modularity of the framework allows for easy integration of new models as they become available. As parallel fields progress and new models are developed, we expect that the performance of SHOWTELL will improve and the variety of applicable skills will expand. Additionally, ane-tuning of pre-trained models mitigate this lim- itation and reduce the need for explicit pre-processing is an exciting direction for future work. Our approach is also limited by the quality of the demonstrations provided, assumes demonstrated grasps can be mapped to robot gripper, and may struggle with demonstrations that are incomplete dif- ficult to perceive due to occlusion or poor quality. Future work could leverage the interpretability of the approach to mitigate this limitation through interactive program repair. The inclusion both visual and language inputs may introduce ambiguities that could complicate the approach. Future works could incorporate interactive dialog with the user to disambiguate inputs that are ambiguous or unclear. Finally, the approach uses closed loop execution to reach each goal waypoint and may struggle with dynamic disturbances changes in the environment during execution.
7 Conclusion
In this work, we present SHOWTELL, neuro-symbolic robot PbD framework for synthesizing mod- ular, generalizable, and interpretable robot manipulation programs from visual demonstrations and natural language. Our approach can teach robot manipulation skills from single demonstration, without requiring any additional training or fine-tuning, by utilizing combination of pre-trained foundation models and -and-ennieeered components to reason about observed demonstrations and to transfer learned skills to new scenes. We evaluate SHOWTELL on a set of 16 real world ma- nipulation tasks across visually distinct environments and show that this approach is effective for teaching wide range of manipulation tasks. We show that SHOWTELL is able reason about demonstrations that require high level logic like conditions, iteration, and segmentation, and that outperforms existing sfate-of-the methods for generating robot policies from language and visual demonstrations.

---

Acknowledgments
This research is partially funded by the UW Amazon Science Hub. We would like thank Entong Su for help with cuRobo, Varad Dhat for robot hardware maintenance, Sylvia Dai, Michael Wolf, Joshua Smith, Markus Grotz, and Nick Walker for helpful discussions.
References
[1] S. A. Al-Abood, K. Davids, and S. Bennett. Specificity of task constraints and effects of visual demonstrations and verbal instructions in directing learners' search during skill acquisi- tion. Journal of motor behavior, 33(3):295-3055 2001.
[2] T. R. Sumers, M. K. Ho, R. D. Hawkins, and T. L.. Griffiths. Show or tell? exploring when (and why) teaching with language outperforms demonstration. Cognition, 232:105326, 2023.
[3] G. Csibra and R. Shamsudheen. Nonverbal generics: Human infants interpret objects sym- bols of object kinds. Annual review of psychology, 66:689-710, 2015.
[4] A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Survey: Robot programming by demon- stration. Technical report, Springrer, 2008.
[5] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
[6] W. Si, N. Wang, and C. Yang. review on manipulation skill acquisition through teleoperation- based learning from demonstration. Cognitive Computation and Systems, 3(1):1-16, 2021.
[7] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. 6th Annual Conference on Robot Learning (CoRL), 2022.
[8] Y. Zhu, Z. Jiang, Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric representations. 7th Annual Conference on Robot Learning, 2023.
[9] T. Lozano-Perez Robot programming. Proceedings of the IEEE, 71(7):821 1--81, 1983.
[10] M. Hersch, F. Guenter, S. Calinon, and A. Billard. Dynamical system modulation for robot learning via kinesthetic demonstrations. IEEE Transactions on Robotics, 2466):1463-1467, 2008.
[11] B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz. Trajectories and keyframes for kines- thetic teaching: A human-robot interaction perspective. In Proceedings of the seventh annual ACM/IEEE international conference on Htmman-Robol Interaction, pages 391-398, 2012.
[12] N. M. Mahi Shafiullah, A. Rai, H. Etukuru, Liu, Misra, S. Chintala, and Pinto. bringing robots home. arXiv e-prints, pages arXiv-2311, 2023.
[13] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1118-1125. IEEE, 2018
[14] Huang, D. Fox, and M. Cakmak. Synthesizing robot manipulation programs from a single observed human demonstration. In 20/9 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 4585-4592, IEEE, 2019.
[15] S. Bahl, A. Gupta, and D. Pathak. Hluman-to-robie imitation in the wild. arXiv preprint arXiv:2207.0944 09450, 2022.
[16] M. Xu, Z. Xu, C. Chi, M. Veloso, and S. Song. Xskill: Cross embodiment skill discovery. In Conference Robot Learning, pages 3536-3555. PMLR, 2023.

---

[17] S. A. Sontakke, J. Zhang, S. M. Amold, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti. Roboelip: one demonstration enough to learn robot policies. arXiv preprint arXiv:2310.07899 2023.
[18] V. Jain, M. Attarian, N. J. Joshi, A. Wahid, D. Driess, Q Vuong, P. R. Sanketi, P. Sermanet, S. Welker, C. Chan, al. Vid2robot: End-to-end video-conditioned policy learning with cross- attention transformers. arXiv preprint arXiv:2403.1294 2024.
[19] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, Xu, S. Xu, M. Yan, and A. Zeng. Do as can and not as say: Grounding language in robotic affordances. In Xrxiv preprint 21ti::2004.019/// 2022.
[20] W. Huang, F. Xia, T. Xiao, H. Chan, Liang, Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 2022.
[21] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928 2023.
[22] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. Au- tonomous Robots, 2023.
[23] Liu, Y. Orru, C. Paxton, N. M. M. Shafiullah, and Pinto. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint 12Xv: 2024.
[24] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1493-9500. IEEE, 2023.
[25] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523- 11530. IEEE, 2023.
[26] W. Huang, Wang. Zhang. Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[27] Y. Wang, G. Gonzalez-Pumariegge Y. Sharma, and S. Choudhury. Demo2code: From summa- rizing demonstrations to synthesizing code via extended chain-of-thought. Advances in Neural Information Processing Systems, 36, 2024.
[28] N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi. Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration. arXiv preprint 2rX1::3311.11011, 2023.
[29] Z. Yang, Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang. An empirical study of gpt-
3 for few-shot knowledge-baset vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3081-3089, 2022.
[30] A. Zeng, M. Attarian, Ichter, K. Choromanski, A. Wong, S. Welker, F, Tombari, A. Purohit, M. Ryoo, V. Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint 0000092004.00598 2022.
10

---

[31] T. Gupta and A. Kembhavi. Visual programmiigg Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 11953-14922 2023.
[32] D. Surís, Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 11888-1 11898, 2023.
[33] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learn- ing, pages 28492-28518. PMLR, 2023.
[34] C. Lugaresi, J. Tang. H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. Yong, J. Lee, et al. Mediapipe: A framework for perceiving and processing reality, In Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR), volume 2019, 2019.
[35] T. Simon, H. Joo, Matthews, and Sheikh, Hand keypoint detection in single images using multiview bootstrapping, In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1145-1153, 2017.
[36] D. Shan, J. Geng, M. Shu, and D. F. Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869-9878, 2020.
[37] A. Kirillov, E. Mintun, N. Ravi, Mao, Rolland, L. Gustafson, Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. rrXiv:2304.02643, 2023.
[38] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, L. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[39] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. arXiv preprint vrXiv:2104.1322) 2021.
[40] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffri memory model. In European Conference on Computer Vision, pages 640- 658. Springer, 2022.
[41] Li, Li, Savarese, and Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[42] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[43] L. Tang, M. Jia, Q Wang, P. Phoo, and B. Hariharan, Emergent correspondence from image diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https::/openrrrriee.. wtt.://wwww..e.eeoooorrreeoeoood
[44] G. Luo, L. Dunlap, D. H. Park, A. Holynski, and T. Darrell Diffusion hyperfcatures: Searching through time and space for semantic correspondence. In Advances in Neural Information Processing Systems, 2023.
[45] E. Hedlin, G. Sharma, Mahajan, H. Isack, A. Kar, A. Tagliasacchi, and K. M. Yi. Un- supervised semantic correspondence using stable diffusion. Advances in Neural Information Processing Systems, 36, 2024.

---

[46] H.-S. Fang. C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023.
[47] B. Sundaralingam, S. K. Hari, A. Fishman, C. Garrett, K. V. Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, Ramos, N. Ratliff, and Fox. curobo: Parallelized collision-Fie free minimum-jerk robot motion generation, 2023.
[48] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The design of stretch: A com- pact, lightweight mobile manipulator for indoor human environments. In 2022 International Conference on Robotics and Automation (ICRA), pages 3150-3157. IEEE, 2022.
Appendix
A Implementation Details
A.1 Perception APIs
The perception APIs provide access to the visual information made available in the pre-processing step, in addition to open world perception of both the demonstration scene and the novel scene via pre-trained foundation models. The APIs include:
GetContaatteequeceess () Returns all of the hand-object contact sequences detected in the demo by 100DOH [36] during the preprocessing step.
GetHanddanddarrss (timestamp) Returns the thumb and index finger hand landmarks detected in the demo by Mediapipe [34] during the preprocessing step.
CestGestureRet (timestanm Returns mask of the object being referred to at the given timestamp of the demo. Any in-hand object used, otherwise vector is produced from the hand landmarks and the closest object to the vector used.
DetectObjects (obj_class, rgb_ frame) Returns a mask of the object class in the given RGB frame using ViLD [39].
TrackMask (mask, origin_frant new_frae vew_frame) Tracks the given mask to the given RGB frame using XMem [40].
VQA (query, rgb_frame, output_tyye type, bbox) Returns the answer to the given query about the given RGB frame using BLIP-v2 [41].
A.2 Speech Alignment APIs
The speech alignment APIs provide access to the spoken language of the demonstration and the timestamps of the spoken words. The spoken language is transcribed using the Whisper speech-to- text model [33]. The transcription is saved and uttered words are indexed by their timestamp relative to the start of the demonstration. The APIs include:
GeTTimestamppfSpeecttettt (speech, occurance) Returns the timestamp of the start of the given occurance of the given speech in the spoken language of the demo.
(etTimmettanpOOSpeechnnn (speech, occurance) Returns the timestamp of the end of the given occurance of the given speech in the spoken language of the demo.
A.3 Scene Transfer APIs
The scene transfer APIs provide access to the tools necessary to transfer the demonstrated skill to the current environment observed in the new scene. The APIs include:
12

---

Reference Point
Predicted Corresponding Points
Figure 5: During scene transfer, corresponding reference points are found in new observation using features from pre-trained image diffusion model. On the left, reference points are shown in the demonstration scene. On the right, corresponding reference points are shown as detected in novel viewpoints, scenes, and with novel within category objects.
TransferronntactSequer (aopiicceecotttt (contact_seq, target_mask) Transfers the given con- tact sequence the new scene. Points corresponding to each reference point in the contact sequence identified in the scene by using features from pre-trained image diffu- sion model. This process can be performed over the entire scene optionally guided by semantic/instance masks.
TransferEEConf ig (t (timestamp, reference_poor _pointt, mask) Transfers the end- effector configuration from the given timestamp the new scene using the given reference point anchor. A corresponding reference point in the contact sequence is identified in the new scene by using features from pre-trained image diffusion model. This process can be performed over the entire scene or optionally guided by semantic/instance masks.
Transfer (bbox, frame) Transfers the given bounding box the given RGB frame using SAM [37] acquire mask from the bounding box and XMem [40] transfer the mask to the new frame before converting back to bounding box.
AnyGrasp (mask) Returns the best grasp configuration for the given object mask using AnyGrasp [46].
A.4 Robot Control APIs
The robot control APIs provide access to the tools necessary to specify end-effector goal poses and configurations. All motion generated using the motion planner implemented by cuRobo [47]. The APIs include:
MoveEEToPose (pose))) Moves the end-effector to the given pose in 3D space.
MoveEERel (rel_pose) Moves the end-effector to the given pose relative to the current end-effector pose.
MoveeEEGuardeaadirtiiooo Moves the end-effector along the given direction until con- tact occurs.
MoveEEHome Moves the end-effector to the home position.
RotateEE(rotation Rotates the end-effector to the given rotation.
AdjustGripperWidthh (width) Adjusts the gripper width to the given width.
13

---

B Experimental Details
B.1 Benchmark Tasks
We design evaluation tasks to wide range of contact-rich manipulation behaviors involv- ing prehensile and non-prehensile motions. The tasks range from rearranging objects, to multi-step extraction from cluttered scenes, to tool use, to manipulation of deformable and articulated objects. The tasks are grouped based on reasoning challenge conditions they present, including simple tasks, iterative tasks, conditional tasks, and segmented tasks. This taxonomy can be seen in Table 3. Be- low we provide a brief description of each task and the conditions used to determine successful completion of the task.
Pick-and-place: In this task, the robot picks up a bottle by its top and places into a bowl. The task successful if the robot grasps from the top of the bottle and the bottle contained inside of the bowl the end of execution.
Open drawer: In this task, the robot is required to open a drawer. This requires a precise grasp of the drawer handle and careful imitation of the demonstrated trajectory to open the drawer. The task is successful if the drawer open the end of execution.
Stack blocks: This task demonstrates manipulation program with a multi-step horizon. The robot must stack a of three colored blocks in the same order as the demonstration. The task is successful when the blocks are stacked in stable column following the order given by the demonstration.
Fold towel: This task demonstrates manipulation program with deformable objects. The robot must first grasp the corner of towel, then follow the demonstrated trajectory to fold the towel. The task is successful when the towel is folded.
Wipe whiteboard: This task demonstrates manipulation program with tool use. The robot must first grasp cloth, then follow the demonstrated trajectory to clean a marking off of whiteboard using the cloth. The task is successful when the whiteboard is cleaned.
Unplug charger: In this task the robot must first grasp charger, then follow the demon- strated trajectory to unplug the charger from wall outlet. The task is successful when the charger is unplugged.
Move all bottles to container: This task demonstrates an interative manipulation program with variable number of objects. For each bottle in the scene, the robot must grasp the bottle and move to the receptacle as demonstrated. The task is successful when all bottles are contained inside of the container.
Clear table: This task demonstrates an iterative manipulation program with variable num- ber of objects. For each object on the table, the robot must grasp the object and move it to the receptacle as demonstrated. The task is successful when the table clear.
Sort items: This task demonstrates an iterative manipulation program with variable number objects and conditional logic. The robot must sort the objects on the table into two groups based on their color. The task successful when all objects are sorted correctly.
Mechanical search: This task demonstrates an iterative manipulation program with variable number of objects and conditional logie. The robot must search for a target object among set of distractor objects. The task is successful when the target object is found.
Unstack to grasp: This task demonstrates an iterative manipulation program with variable number of objects and conditional logic. The robot must unstack set of blocks reveal target object, then grasp the target object. The task is successful when the target object is grasped.
Conditional pick and place: This task demonstrates manipulation program with condi- tional logic. The robot must move object to receptacle only if the receptacle is empty. The task is successful when the object is contained inside of the receptacle.
14

---

Figure 6: Benchmark items used in the evaluation of SHOWTELL.
Open drawer if not opened: This task demonstrates manipulation program with condi- tional logic. The robot must open drawer only it is closed. The task successful when the drawer open.
Conditional grasp: This task demonstrates manipulation program with conditional logic. The robot must grasp object in one of two ways based on the object's color. The task is successful when the object is grasped correctly.
Assemble bento: This task demonstrates a manipulation program with segmentation. The robot must assemble bento box by placing each item in the correct compartment. The task is successful when the bento box is assembled correctly.
Fixed grab mug: This task demonstrates manipulation program with segmentation. The robot must grasp mug by its handle. The task is successful when the mug is grasped correctly.
B.2 Evaluation Details
evaluate the performance of SHOWWTELL with a total of 50 rollout trials per task: 10 demon- strations are provided demonstrations in each environment) and 5 rollouts are performed for each demonstration, resulting in total of 50 rollouts per task. For tasks that require iteration vary the number of objects in the scene for each demonstration. For tasks that require conditionals we vary the state of the environment for each demonstration. To evaluate the performance of SHOWTELL across rollouts with unseen objects we perform an additional 10 rollouts each for the pick-and- andcee clear table, and conditional grasp tasks with objects that were not present in the demonstration. To evaluate the performance of SHOWTELL across rollouts unseen environments we perform an ad- ditional 10 rollouts each for the pick-and-plac and open drawer tasks in distinct environment from the demonstration.
15

---

Task Name
Simple
Iterative
Conditional
Segmented
Pick and place
Open drawer
Stack blocks
Fold Towel
Wipe whiteboard
Unplug charger
Move all bottles to container
Clear table
Sort items
Mechanical search
Unstack to grasp
Conditional pick and place
Open drawer if not opened
Conditional grasp
Assemble bento
Fixed grab mug
Table 3: Real-world robot manipulation tasks across different task families.
Hand-Object Perception (35%)
Scene Transfer (19%)
Failed Task (12%)
Other Perception (B%)
Motion Planning (4%)
Execution (23%)
Figure 7: The distribution of failures across SHOWTELL benchmark tasks. Executions can fail due to errors in hand-object perception, errors scene transfer correspondence matching, other perception failures, failure to motion plan, control execution failures, or failure meet the task requirements.
C Failure Analysis
We analyze the failures of SHOWTELL across the benchmark tasks. The distribution of failures across the tasks is shown in Figure 7. We divide failures into six categories:
Hand-Object Perception: Errors in hand-object perception, where the robot fails to rectly identify the hand or hand-object contact.
Scene Transfer Correspondence Matching: Errors in scene transfer correspondence matching, where the robot fails to correctly match reference points in the demonstration to the new scene.
Other Perception Failures: Errors in object detection, tracking, other perception tasks.
16

---

Failure to Motion Plan: Errors in motion planning, where the robot fails to plan collision-free trajectory.
Control Execution Failures: Errors in control execution, where motion plan is generated but the robot fails to execute the plan correctly.
Failure to Meet Task Requirements: Errors in the manipulation program, incorrect exe- cution, or other factors that prevent the robot from successfully completing the task.
The most common failure mode is due to errors in hand-object perception, indicating that improve- ments in hand-object perception could lead to significant improvements in overall performance. Execution another common source of failure, typically due to imprecise control of the robot's wheeled base. More dynamic control involving visual servoing could mitigate this failure mode in future work. The other failure modes are less common, but still contribute overall performance.
17

---

System Prompt:
You are writing Python code to control robot based a natural language inatruction and visual desonstration. The natural language instruction is provided in the comsents and you can query information about the visual denonstration. You rust write code to understand what is being desonstrated in the input video and execute the demonstrated behavior with the robot. Please outout only valid Python code with other explanations. Do not output narkdown or comments, just the executable python code.
User Message:
The ApTs wost lable are: @ctContactSesuence
Descr bpt ipt ion: beturns wll of the hand-abjact contact sequences detected in the deeoo Each contect suquance hax (xtert_time, and time, ref_frane. ce.conf:gs)
Getthrdd andaerka Linestwep)
Deser iption: beturns
landwarks detected at the given Linesturp, one (u,y) point the index Cinger and the
detobservation!
Descr ton: Returne current cbservstion frase from the robot csners, RGn array and one depth.
Descr ipt ion: beturne e eask of the object being referenced at the given tirestwpp, wither in-twnd or being gestured to. jceettibjeets(oo bettstoe_.ciass, rgb, franfranc)
Deser ipt lon: open-world abject detector returns a makk bourding the object detected in righ TrackMusk (eusk, origin, frase, new, frase)
Desoription:
bbox-honn)
Descript ton:
vimual question about the fibufrrew. outpot_type be bool, number, striing, bbox is optional.
detTinestampon tapoosteeeeeeeeeeeeee arrtsspech, occurence 1dk)
Deser ipt ion: returne the to wap when the given speach started for the ocnurence_ids time Speenntd Get andeeeennndtspeeene (speesh, occurence 16x)
Deser ipt lunt returns
Liwalerp shwe the given spewch unded the occurence, jide Line
PranteeSontsettequuee (conteet requeece,,,, target..o) .rask-More)
Desoription: Transter
given contact sequence to the nen scone, Optionally provice mask the target object acene.
Transfer (8oxbbor.
the given frnm TransforEEoonfgg (rrdeefa (tinestaxo, reference point, mask-Hoone)
bescription: Transfer the bounding
observation.
bescript pnn::: Transfar the and affector pase at the given tirestwap to the new scane observation ton.
AnyGrasppposntt cloud)
beaer iplinn: Chonses group conf configuaation for an wrb arbiirary point cloud
(oveETooPseeppoe)
the specsiied goal. pose
NoveEEhellde ddelaa
Descript Nove EE the specified reletive
MoveEEQuareddj yecturdddd//eoor
Descript iptiion: Rove FP wlong x given vector until contact
NovcEEhone!)
Deser ipt ion: feeturn to position, Uouilly done the the task.
Adsuetärtsperksschiiee
Desoription: Acjust
gripper
(ueed grasp ungrase)
color_ frr.rfrfttttetttanpppp to acceas the frase for tteestarp in the video You can use death franes| [inestaap] to access the depth frame for any tinestaup in the deno video You can use pointelmutel tide//inneeeepp tn annees the point claud for any timstxep in the dwas video You can use obs = Getobservationl to get the current deservation frane fron the rebot's s comera
You
not outout narkdown (e.g. python ") or add any oode comeence, just the exccutable python code.
For example, in the recorded
following PTOK AMD_ target_objecte"ppe bottle".
And the natural language narration of the demn ix ax follmw:
Grase: the pill bottle and move St to the bokl.
can write (Bqueenee
execute Semonotrated behavtor robot cimilar to this:
Guttoserration
Transferbontee Betoonttetteeeeeeere.....o for ofg in targ sea, es confige:
dejustfir pepppppdheeee..... NoveEE NoveEEToPosse ToPosel ec. pose)
wicth)
Roved Hove))))oww)
recorded (contact, sequenon_ssararr
following sequincea
The natural language carration of the dess is as fallnes: (denc. narration)
The prompt includes API documentation that specifies the available functions, a brief summary of contact sequences extracted during the pre-processing phase, and the full transcribed narration of the demonstration.
Figure 8:
18


Reviews:
[{"review_id": "RpPgcVHkA9", "reviewer": "Reviewer_sbTj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work tackles the challenge of learning a policy for a task by passively observing a human performing it. This problem is difficult because understanding human actions requires joint reasoning about the visual and language cues in the demonstration. Additionally, the learned policy should generalize to similar environments and objects. The proposed approach combines the visuo-lingual reasoning capabilities of vision-language models with task-specific neural networks and other hand-engineered components to learn a policy from human demonstrations and transfer it to new scenes and environments. The authors first preprocess the demonstration into a form suitable for the vision-language model to output the policy as Python code composed of high-level APIs. The evaluation is conducted in real-world manipulation tasks using the Stretch RE2 robot.", "summary_of_recommendation": "The paper proposes an interesting method for using vision-language models (VLMs) to learn a policy from human demonstration. In particular, it makes progress toward learning long-horizon policies that require concepts like loops and conditionals. This work could inspire similar research in the future. While the evaluation setting is limited compared to baseline, the inclusion of hardware experiments is indeed an advantage. Overall, I feel the strengths outweigh the weaknesses. I recommend that the paper be accepted, provided the authors address the concerns raised in the review during the rebuttal", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "0) Address the weakness section in the reviews. \n\nAdditional questions: \n\n1) I found the neuro-symbolic aspect of the work unclear. Although the term \"neuro-symbolic\" is used broadly, I am having difficulty relating this work to existing neuro-symbolic studies in the literature. It would be helpful if the authors could clarify why they consider this work specifically neuro-symbolic and clearly delineate what elements of their approach are neural versus symbolic. This discussion would enhance understanding and situate the work within the broader context of neuro-symbolic research.\n\n2) In Figure 4(c), the \"cube\" seems to be a typo. It should be \"bottle\" instead. \n\n3) what happens when multiple instances of an object are present in the scene? How the feature mapping will be done?\n\n4) Are you considering releasing the code?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VMvL4hrcU6", "reviewer": "Reviewer_F4Hd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes ShowTell, a method to perform Programming by Demonstration by receiving instructions both in the form of language and video. Through a large series of off-the-shelf models, predefined primitives, and APIs, all coordinates by a large VLM, the method can ingest this multimodal data and generate the desired behaviour in new settings-", "summary_of_recommendation": "I suggest a weak accept, but I will take into account opinions from other reviewers and make the final decision after the discussion session. To recap, I believe the paper is interesting and the experiments and method are well developed and executed, however it is difficult to really understand what novelty it can present in the wide range of VLM-based papers, and the engineering aspects can surpass the scientific ones, making it look more like a system paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "As stated in the review, I would like the reviewers to better clarify the scientific impact of the paper with respect to its engineering impact, also considering if possible the existing papers I mentioned.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbpsYwYxSD", "reviewer": "Reviewer_2euh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces SHOWTELL, a neuro-symbolic framework that enables robots to learn manipulation skills from a single visual demonstration and natural language instruction, without requiring additional training. It emphasizes generalizability across various tasks and environments, allowing robots to adapt to new objects and settings effectively. SHOWTELL's modular design enhances interpretability and scalability, enabling it to handle complex tasks that involve high-level reasoning. Empirical results demonstrate that SHOWTELL outperforms existing state-of-the-art methods, highlighting its potential to improve the usability and performance of robot programming by demonstration systems.", "summary_of_recommendation": "In summary, I appreciate the contributions and innovative approach presented in this paper. The results are convincing and demonstrate the effectiveness of the proposed framework. It appears to be a strong fit for the conference, and I believe it should be accepted. Additionally, it would be valuable to explore some failure cases and discuss potential strategies for addressing them, as this could further enhance the robustness of the approach.", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Curious how the model performs in unseen environments with unseen objects (both).\n2. What would be the implications of fine-tuning certain modules within the framework for handling more complex tasks? Will that reduce the reliance on hand-engineered inputs and the need for pre-processing demonstrations to some degree?\n3. Can you provide examples of failure cases encountered during experiments, and what insights have been gained regarding the limitations of the approach and potential solutions? \n4. How does the framework adapt to dynamic disturbances or changes in the environment during execution, and what measures are taken to ensure reliable performance under such conditions?\n5. Can this framework be adapted to handle long-horizon tasks that involve multiple sub-goals, such as: \"If the drawer is open, then retrieve the pen inside it; otherwise, bring me the pen that is on the table\"?\n6. Is this framework capable of extending to utilising multiple video demonstrations as a sequence of demonstrations for a long horizon task like - Bring me a glass of water and throw this paper in the bin where first part of instruction is shown in one video and the second in another. If we are able to do this then we can compose very complicated tasks by combining short videos for demonstration increase or data as a whole. \n7. Is the framework capable of leveraging multiple video demonstrations as a sequence for executing a long-horizon task, such as: \"Bring me a glass of water and throw this paper in the bin,\" where the first part of the instruction is shown in one video and the second in another? If so, how could this capability enable the composition of more complex tasks by combining shorter video demonstrations?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a programming by demonstration approach to generate robot programs, relying on a base set of foundation models to generate policies (program synthesis) for new tasks.

Strengths:
* Well written, method attempts to solve a challenging task, a good systems demo about an approach to integrate and generate programs by composing a set of lower level api calls provided by various base models and tools.
* A nice real-world demo

Weaknesses:
* There is a lack of clarity around the specific engineering implementations required to implement this system (what are the prompts, what information is required, what data is collected and what form etc. how were the low level primitives and controllers designed and implemented)
* There is insufficient detail around the tasks used and experimental details.
* While the method does produce a general program to solve complex tasks, this rules based approach requires all existing controllers and modules to be well engineered and functional
* The method does not seem to scale to handle faults or uncertainties, and scalability to longer horizon tasks seems limited
* The scientific contribution here is limited, particularly when considered with regard to prior work in program synthesis.

Reviewers provide a number of questions to be addressed in the rebuttal, in particular around missing information required for reproducibility and greater clarity around the novelty of the contribution.

Post rebuttal:
- The authors added additional detail around apis and prompts and experimental settings, along with greater discussion of limitations and failure analysis, and included more discussion of related work, addressing the bulk of the remaining weaknesses with this paper.
Decision: Accept