Teaching Robots with Show and Tell: Using Foundation Models to Synthesize Robot Policies from Language and Visual Demonstration

Authors: michael  murray
abhishek  gupta
maya  cakmak
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 23:56:45

Abstract: We introduce a modular, neuro-symbolic framework for teaching robots new skills through language and visual demonstration. Our approach, ShowTell, composes a mixture of foundation models to synthesize robot manipulation programs that are easy to interpret and generalize across a wide range of tasks and environments. ShowTell is designed to handle complex demonstrations involving high level logic such as loops and conditionals while being intuitive and natural for end-users. We validate this approach through a series of real-world robot experiments, showing that ShowTell out-performs a state-of-the-art baseline based on GPT4-V, on a variety of tasks, and that it is able to generalize to unseen environments and within category objects.

Main paper: 
 introduction: 
 General purpose robots have the potential to enhance productivity and reliability in human-centric, task-oriented settings such as kitchens, warehouses, and offices, but one of the key challenges to achieving this potential is that each environment, user, and task combination demands tailored behavior from the robot. Programming by Demonstration (PbD) is a popular approach to this challenge, enabling end-users to program robots for personalized tasks and environments by providing demonstrations of the desired behavior. However, existing PbD techniques typically require a large number of demonstrations or are unable to extract high-level task information related to control flow and action parameterization from the demonstrations. Natural language is another popular method for personalizing robot behavior, but low-level details can be cumbersome and error prone to communicate with language alone, and most works assume a fixed set of low-level primitives are available to be composed by the language instructions, limiting the scale and complexity of tasks that can be programmed.Studies in human observational learning show that humans use both language and demonstration when teaching each other new tasks, with both communication modalities playing important roles [1, 2] . Language allows us to transmit abstract information, while demonstrations instantiate that information in concrete examples [3] . Inspired by this insight, we envision a more natural and intuitive PbD system, where end-users can program robots for personalized tasks and environments by flexibly using both language and visual demonstrations.In this work, we propose SHOWTELL -a system that enables end-users to teach robots new tasks the same way they would teach another person, by visually demonstrating and verbally describing what they are doing as they demonstrate. SHOWTELL is modular, composing a set of pre-trained large language models (LLMs) and vision-language models (VLMs) to synthesize robot policies that can jointly reason about the language and visual components of the demonstrations and execute the demonstrated behavior in novel scenes. SHOWTELL is designed to be generalizable across a wide range of tasks and environments, and to be intuitive and natural for end-users to use, while requiring only a single demonstration with no additional training or fine-tuning.While both language and demonstration are traditionally popular interfaces for task planning, there has been comparatively little attention paid to combining the two. Most existing works attempt to train end-to-end models, an approach that is difficult to scale because it requires enough training examples to implicitly learn to understand all demonstrations and perform all tasks within the forward pass of a neural network. In contrast, our approach requires no additional training, utilizing the vast knowledge encoded in off-the-shelf foundation models. Our approach is neuro-symbolic in that it aims to combine the strengths of neural networks, which excel at learning from data, with symbolic reasoning, which excels at manipulating symbols and logical rules, to create a more robust robot PbD system that is modular, allowing us to scale to new tasks by composing existing modules in novel ways, and also also interpretable, allowing us to understand the reasoning behind the generated policies and to easily modify or extend them.Through real-world robot experiments, we validate our approach, showing that SHOWTELL is able to synthesize robot policies from language and visual demonstrations including tasks that require high level logic such as conditions, iteration, and segmentation. We show that our approach outperforms a state-of-the-art baseline on a variety of tasks, and that it is able to generalize to new objects and environments, while requiring only a single demonstration with no additional training or fine-tuning. We believe that our approach has the potential to significantly improve the usability and performance of robot PbD systems, and to enable robots to learn from demonstrations that are more natural and intuitive for end-users. 
related work: 
 Programming by Demonstration. Programming by Demonstration, also referred to as Learning from Demonstration or Imitation Learning, has been the subject of four decades of robotics research [4, 5] . Approaches are often categorized based on the method of providing demonstrations and in contrast to methods that require moving the robot (e.g. through teleoperation [6, 7, 8] or kinesthetic teaching [9, 10, 11] ), or the use of specialized demonstration hardware [12] , in this work we focus on programming by passive observation, where the robot is programmed by observing a human perform the desired behavior [13, 14, 15, 16, 17, 18] . Most existing works attempt to train endto-end models, an approach that is difficult to scale because it requires enough training examples to implicitly learn to understand all demonstrations and perform all tasks within the forward pass of a neural network. Instead, we propose to leverage the prior knowledge encoded in pre-trained foundation models to synthesize programs that can both reason about demonstrations and execute the demonstrated behavior on a robot with novel scenes and objects.LLMs for Task and Motion Planning. With the advent of large-scale pre-trained language models, there has been growing interest in using these models for robotics tasks. A large body of work has focused on planning and reasoning from text-based natural language instructions [19, 20, 21, 22, 23] . These works typically output their plans as a sequence of robot actions, but recent approaches show the benefits of using LLMs to synthesize code with logical constructs that can be executed by a robot [24, 25, 26] . While much progress has been made in synthesizing code policies from textbased instructions, there has been comparatively little work on synthesizing code policies from visual demonstrations. Wang et al. [27] assume that a demonstration has been converted to a textual description, and focus on generating policies from the text. In contrast, we focus on generating policies directly from visual demonstrations. Most similar to our work, Wake et al. [28] propose to use a VLM to summarize visual demonstrations and generate policies from the resulting summaries.In contrast, we propose to synthesize programs that can reason about the visual demonstration, which allows us to handle more complex demonstrations, better align the language with the visual LLMs for Visual Reasoning. Visual reasoning approaches have typically used end-to-end trained models, but recent works have shown that pre-trained LLMs can be leveraged to accomplish state-ofthe-art for many visual reasoning tasks. Early iterations represent the visual information from images as text via captions, objects, and attributes and feed this textual representation to LLMs along with task instructions and in-context examples [29] . Zeng et al. [30] propose a modular approach wherein the LLM leverages other pre-trained models such as vision-language models (VLMs) and audiolanguage models through program generation. More recent works have scaled this idea to additional tasks including knowledge tagging, image editing, and causal/temporal reasoning over videos [31, 32] . In this work we seek to leverage LLMs for visual reasoning in the context of robot PbD by using the LLM to generate modular programs that jointly reason over video demonstrations and spoken language instructions to understand the task being demonstrated and ground that understanding to robot actions in new scene observations. 
method: 
 We present SHOWTELL, a neuro-symbolic robot PbD framework for synthesizing modular, generalizable, and interpretable robot manipulation programs from visual demonstrations and natural language. Our approach, illustrated in Figure 1 , requires only a single demonstration with no additional training or fine-tuning, as it utilizes a combination of pre-trained foundation models and hand-engineered components to reason about observed demonstrations and to transfer learned skills to new scenes. In the following sub-sections we first formalize the problem setting, then we provide a high-level overview of the approach, and finally we describe each phase of the approach.In this work, we consider multi-modal PbD for robotic manipulation tasks. Let A be the set of robot actions, and S the set of world states. We assume access to a human demonstration consisting of visual demonstration component D = ⟨d 0 , d 1 , . . . , d T D ⟩, where each demonstration frame d t is an RGB-D image at time t, and a spoken language component L. Given the demonstration D, language L, and an initial state s 0 ∈ S, the goal is to generate an execution ξ = s 0 , a 0 , s 1 , a 1 , . . . , s T ξ , a T ξ , where a t ∈ A is an action taken by the robot at time t, s t ∈ S is the state before taking a t , and s t+1 = T (s t , a t ) under environment dynamics T : S × A → S. The state s t is defined by the environment layout, the poses and states of all objects, and the pose and state of the robot. The robot does not directly have access to the state s t , but only to an observation o t . An observation o t = (I t , K t ) includes an RGB-D camera image I t and the robot's proprioceptive state K t . The task is considered successful if the goal-conditions corresponding to the demonstration D and spoken language L are true at the final state s T ξ .Our method utilizes a mixture of pre-trained foundation models and hand-engineered modules to both understand the provided demonstration and to transfer that understanding to new scenes. We first pre-process the demonstration to make relevant information readily accessible for program synthesis. Next, an LLM is used to compose a set of modules into a program. The modules available to the LLM include tools for visual and spatio-temporal reasoning about the video demonstration frames, tools for aligning the spoken language of the demonstration with the visual demonstration, tools for transferring the demonstrated skill to the current environment, and tools for controlling the robot.The first step in our approach is to pre-process the demonstration to make relevant information readily accessible to the LLM. The spoken component of the demonstration, L, is transcribed using the Whisper speech-to-text model [33] . The transcription is saved and uttered words are indexed by their timestamp relative to the start of the demonstration. The visual component of the demonstration is processed to detect human hands and their interactions with objects in the scene. We utilize the Mediapipe [34] hand landmark detection model, and detect the human hand pose represented as 21 landmarks following the topology in [35] . The two landmarks on the thumb and another two on the index finger are used to represent a parallel jaw robot gripper.For robot manipulation tasks, timesteps in which the end-effector interacts with objects in the environment are particularly important, so we seek to identify and extract contiguous contact sequences, or clusters of timestamps where the hand is in contact with an object, from the demonstration D. We extract information about the hands in the scene and the objects that they contact using 100DOH [36] , a hand-object interaction model that has been pre-trained on 100K images extracted from a large-scale video dataset of humans interacting with objects. We use 100DOH to extract, for each demonstration frame, a hand bounding box, in-contact object bounding box, and a boolean contact variable indicating whether the hand is in contact with an object or not. We obtain fine grained masks for both hands and objects using Segment Anything Model (SAM) [37] with bounding boxes provided as prompts. 3D perception of the scene is crucial for manipulation tasks, so we additionally produce a point cloud for each demonstration frame using the RGB-D image and camera intrinsics. SHOWTELL uses a powerful code-generating language model (GPT-4 [38] ) to synthesize programs that can both reason about a demonstration and execute the demonstrated behavior in a new scene. The LLM is provided with a prompt that consists of import statements and API documentation that specifies the available functions, a brief summary of contact sequences extracted during the pre-processing phase, and the full transcribed narration. The program is run with the Python interpreter, allowing the use of control flow tools like for loops, conditionals like if/else, built in functions like sort, and built in modules such as datetime or math. All of the information extracted during the pre-processing phase is made available in addition to a suite of modules. The additional modules made available to the program include a mixture of pre-trained foundation models and hand-engineered modules, including modules for perception, speech alignment, transferring to novel scenes, and controlling the robot. In the following sections we describe each of the custom modules and their APIs and a taxonomy of the modules is provided in Figure 2 .The perception modules made available to the neuro-symbolic program enable extracting information about both the visual demonstration, D, and the new scene observations, o 0 , o 1 , . . . , o T ξ . The perception modules include APIs for accessing visual information made available in the preprocessing step via GetHandObjectContactSequences. The perception modules also include APIs for free-form visual reasoning, DetectObjects, TrackMask, and VQA. The DetectObjects module uses ViLD [39] , a pre-trained open-world object masking VLM. Detected masks can be tracked to new frames and scenes with the TrackMask module, which uses the XMem mask tracking model under the hood [40] . The VQA module is a pre-trained visual question answering VLM, BLIP-v2 [41] , that can be used to answer questions about the visual scene.To understand a multi-modal demonstration, the robot must align the spoken language of the demonstration with the visual demonstration. For this purpose, we provide the LLM with two modules, GetTimestampOfSpeechStart and GetTimestampOfSpeechEnd, which are used to extract the start and end timestamps of spoken words from the language component of the demonstration, L, and enable the program to align the language with timestamps corresponding to frames in the visual demonstration, D.Ultimately, the goal is to transfer the policy generated from the demonstration to new environments. To do this, the robot must match the visual demonstration D to the current environment observed in o t . The TransferBBox, TransferMask, TransferEEConfig, TransferContactSequence modules are used to transfer detections to new scene observations. These modules find corresponding reference points by leveraging features from Stable Diffusion [42] , a pre-trained image diffusion model which have been shown to implicitly encode rich information about the structure of objects within an image, and have been shown to be highly effective for finding corresponding points for visual reasoning tasks [43, 44, 45] . Using these features, we can match similar points on withincategory objects in addition to exact points. For example, after demonstrating how to pick up a mug by its handle, the robot should be able to repeat this skill for visually distinct mugs and mugs of different sizes.In order to facilitate grasping of objects in the new scene, we provide the AnyGrasp module, which is used to find the best grasp configuration for a particular object mask in the scene. This module uses a pre-trained grasp prediction model [46] to predict the best grasp configuration for a particular object based on its point cloud, and can be used to find grasp configurations for objects that were not present in the demonstration.To control the robot we provide a set of modules to specify end-effector goal poses and configurations. The modules MoveEEToPose is used to specify a goal EE pose in 3D space, MoveEERel is used to specify a goal relative to the current EE pose, and MoveEEGuarded performs a guarded movement along a given vector until contact occurs. The module RotateEE is used to specify a goal EE rotation. And finally, AdjustGripperWidth is used to specify a desired gripper width to open and close the gripper. These modules are used to specify the robot's end-effector goal poses and configurations, which are used to generate the robot's actions during program execution as described in Section 3.5.Figure 3 : We evaluate SHOWTELL using a Stretch RE2 robot to perform 16 real world manipulation tasks across 5 visually distinct environments including a conference room, a kitchen, a classroom, an office lounge, and a cluttered workbench. We show that this approach is effective for teaching manipulation tasks requiring high level logic such as conditions, iteration, and segmentation.After program synthesis, the resulting program is run with a Python interpreter and its execution is a simple Python call. During execution, correspondence matching modules are evaluated to transfer reference points identified in the demonstration to the new scene observation, and all waypoints are redefined relative to the reference points in the new scene. To interpolate robot motion between end-effector goal poses, we use a collision-free motion planner to generate a trajectory of robot actions for reaching the next desired waypoint goal. Specifically, we use the GPU accelerated motion generation library cuRobo [47] . 
experimental setup: 
 Hardware and Environments: To evaluate our approach, we conduct a series of real world experiments with a Stretch RE2 robot [48] across 5 indoor environments as illustrated in Figure 3 . The robot's mobile base, arm lift, and telescoping arm are moved in conjunction to reach 6-DOF target waypoints. The robot's end effector is a parallel-jaw gripper with rubber fingertips. An Intel Re-alSense D435i RGB-D camera is mounted to the frame which is used both to record demonstrations and to provide observations during execution.Baselines and Experiments: We evaluate our approach against GPT4-V-Robot [28] , a state-of-theart method for robot task planning from language and visual demonstrations using LLMs. In this baseline approach, demonstration frames are directly fed to a VLM (GPT4-V [38] ), which generates a summary that is used by the LLM to generate a policy. To study the contribution of the visual and spoken components of the demonstration we perform a series of ablation experiments. First we evaluate a language-only baseline, ShowTell-NoVis, with no access to the visual demonstration, similar to the approach proposed by Liang et al. [24] . We also include a vision-only baseline, ShowTell-NoLang, which uses the visual demonstration only to generate policies without access to the language component. Evaluation Tasks: We evaluate SHOWTELL using a set of 16 real world manipulation tasks across 5 visually distinct environments. We group evaluation tasks based on the challenge conditions they present. First we evaluate with simple demonstrations that are straightforward to follow, including pick and place, stacking cubes, and opening drawers. Next we evaluate tasks involving control flow including iterative tasks, for example "move all of the blocks over to this container", and tasks with conditionals, for example "open the drawer only if it is closed". Finally, we evaluate tasks that require segmenting the demonstration, for example if the demonstrator makes an error and corrects themselves, the robot is required to segment the demonstration into correct and incorrect segments to generate a policy that correctly imitates only the desired behavior. All of the tasks, including the simple tasks, can require the robot to jointly reason about the visual and spoken components of the demonstration to resolve ambiguities. For example, an instruction like "pick up this mug and move it over here" requires aligning the spoken instruction with the visual demonstration to determine which mug to pick up and where to move it. We use two primary metrics to evaluate system performnace: Success Rate (SR) and Goal Condition Recall (GCR). The task-relevant goal conditions are the set of state changes that must be satisfied at the end of an episode for the task to be considered successful. SR is the fraction of rollouts for which the object positions and state changes completely satisfy the task goal-conditions at the end of the action sequence. GCR is the fraction of goal-conditions successfully completed at the end of an episode to those necessary to have finished a task. 
results: 
 In our experiments we seek to answer the following research questions: 1) Is SHOWTELL practical for teaching a range of robot manipulation skills including those that require high level logic like conditions, iteration, and segmentation? 2) How does SHOWTELL compare to existing stateof-the-art methods for generating robot policies from language and visual demonstrations? 3) Is SHOWTELL able to generalize to unseen environments and within category objects? 4) To what extent do the visual and spoken components of the demonstration contribute to the performance?To evaluate the performance of SHOWTELL, we perform a total of 50 rollout trials per task: 10 demonstrations are provided (2 demonstrations in each environment) and 5 rollouts are performed for each demonstration, resulting in a total of 50 rollouts per task. We group tasks based on challenge conditions as described in section 4 and for each task family, we report the average SR and GCR across all rollouts.The results summarized in Table 1 show that SHOWTELL outperforms existing state-of-theart methods for generating robot policies from language and visual demonstrations. Even for simple tasks, SHOWTELL outperforms the baseline methods, as the modular, neuro-symbolic programs are more easily able to reason about the demonstration and to resolve ambiguities than the monolothic VLM used to summarize demonstrations in the baseline method.Rollout SR Canonical 0.92 Unseen environment 0.84 Unseen objects 0.88 Table 2 : Performance across rollouts: in the canonical scene, with unseen objects, and in unseen environment.This advantage is even more pronounced for the other task families as the purely sequential structure of the baseline method is not well suited to tasks that require high level logic like conditions, iteration, and segmentation. The qualitative examples in Figure 4 illustrate the ability of SHOWTELL to synthesize programs from such demonstrations. Comparing the ablation experiments, we see that the visual and spoken components of the demonstration are both important for generating effective policies, as the ablation experiments ShowTell-NoVis and ShowTell-NoLang perform poorly compared to the full SHOWTELL approach, even for simple tasks. This is because the visual and spoken components of the demonstration are often complementary, and help to resolve ambiguities present in the individual components of the demonstration. To better understand the failure cases, we analyze the failures in Section C of the appendix and illustrate the distribution of failures in Figure 7 . We finally evaluate the performance of SHOWTELL across rollouts in the canonical scene (the scene used for demonstration), with unseen objects, and in unseen environments. The results summarized in Table 2 indicate that SHOWTELL is able to generalize to unseen environments and within category objects, while requiring only a single demonstration with no additional training or fine-tuning. 
limitations: 
 Our approach has several limitations. Most crucially, the approach is constrained by the limitations of the pre-trained models it uses, although we note that the modularity of the framework allows for easy integration of new models as they become available. As parallel fields progress and new models are developed, we expect that the performance of SHOWTELL will improve and the variety of applicable skills will expand. Additionally, fine-tuning of pre-trained models to mitigate this limitation and reduce the need for explicit pre-processing is an exciting direction for future work. Our approach is also limited by the quality of the demonstrations provided, assumes demonstrated grasps can be mapped to a robot gripper, and may struggle with demonstrations that are incomplete or difficult to perceive due to occlusion or poor quality. Future work could leverage the interpretability of the approach to mitigate this limitation through interactive program repair. The inclusion of both visual and language inputs may introduce ambiguities that could complicate the approach. Future works could incorporate interactive dialog with the user to disambiguate inputs that are ambiguous or unclear. Finally, the approach uses closed loop execution to reach each goal waypoint and may struggle with dynamic disturbances or changes in the environment during execution. 
conclusion: 
 In this work, we present SHOWTELL, a neuro-symbolic robot PbD framework for synthesizing modular, generalizable, and interpretable robot manipulation programs from visual demonstrations and natural language. Our approach can teach robot manipulation skills from a single demonstration, without requiring any additional training or fine-tuning, by utilizing a combination of pre-trained foundation models and hand-engineered components to reason about observed demonstrations and to transfer learned skills to new scenes. We evaluate SHOWTELL on a set of 16 real world manipulation tasks across 5 visually distinct environments and show that this approach is effective for teaching a wide range of manipulation tasks. We show that SHOWTELL is able to reason about demonstrations that require high level logic like conditions, iteration, and segmentation, and that it outperforms existing state-of-the-art methods for generating robot policies from language and visual demonstrations.• GetContactSequences() Returns all of the hand-object contact sequences detected in the demo by 100DOH [36] during the preprocessing step.• GetHandLandmarks(timestamp) Returns the thumb and index finger hand landmarks detected in the demo by Mediapipe [34] during the preprocessing step.• GestGestureRef(timestamp) Returns a mask of the object being referred to at the given timestamp of the demo. Any in-hand object is used, otherwise a vector is produced from the hand landmarks and the closest object to the vector is used.• DetectObjects(obj_class, rgb_frame) Returns a mask of the object class in the given RGB frame using ViLD [39] .• TrackMask(mask, origin_frame, new_frame) Tracks the given mask to the given RGB frame using XMem [40] .• VQA(query, rgb_frame, output_type, bbox) Returns the answer to the given query about the given RGB frame using BLIP-v2 [41] . 
 
 Appendix: The speech alignment APIs provide access to the spoken language of the demonstration and the timestamps of the spoken words. The spoken language is transcribed using the Whisper speech-totext model [33] . The transcription is saved and uttered words are indexed by their timestamp relative to the start of the demonstration. The APIs include:• GetTimestampOfSpeechStart(speech, occurance) Returns the timestamp of the start of the given occurance of the given speech in the spoken language of the demo.• GetTimestampOfSpeechEnd(speech, occurance) Returns the timestamp of the end of the given occurance of the given speech in the spoken language of the demo.The scene transfer APIs provide access to the tools necessary to transfer the demonstrated skill to the current environment observed in the new scene. The APIs include:We design our evaluation tasks to cover a wide range of contact-rich manipulation behaviors involving prehensile and non-prehensile motions. The tasks range from rearranging objects, to multi-step extraction from cluttered scenes, to tool use, to manipulation of deformable and articulated objects.The tasks are grouped based on reasoning challenge conditions they present, including simple tasks, iterative tasks, conditional tasks, and segmented tasks. This taxonomy can be seen in Table 3 . Below we provide a brief description of each task and the conditions used to determine successful completion of the task.• Pick-and-place: In this task, the robot picks up a bottle by its top and places it into a bowl. The task is successful if the robot grasps from the top of the bottle and the bottle is contained inside of the bowl at the end of execution.• Open drawer: In this task, the robot is required to open a drawer. This requires a precise grasp of the drawer handle and careful imitation of the demonstrated trajectory to open the drawer. The task is successful if the drawer is open at the end of execution.• Stack blocks: This task demonstrates a manipulation program with a multi-step horizon.The robot must stack a set of three colored blocks in the same order as the demonstration.The task is successful when the blocks are stacked in a stable column following the order given by the demonstration.• Fold towel: This task demonstrates a manipulation program with deformable objects. The robot must first grasp the corner of a towel, then follow the demonstrated trajectory to fold the towel. The task is successful when the towel is folded.• Wipe whiteboard: This task demonstrates a manipulation program with tool use. The robot must first grasp a cloth, then follow the demonstrated trajectory to clean a marking off of a whiteboard using the cloth. The task is successful when the whiteboard is cleaned.• Unplug charger: In this task the robot must first grasp a charger, then follow the demonstrated trajectory to unplug the charger from a wall outlet. The task is successful when the charger is unplugged.• Move all bottles to container: This task demonstrates an interative manipulation program with variable number of objects. For each bottle in the scene, the robot must grasp the bottle and move to the receptacle as demonstrated. The task is successful when all bottles are contained inside of the container.• Clear table: This task demonstrates an iterative manipulation program with variable number of objects. For each object on the table, the robot must grasp the object and move it to the receptacle as demonstrated. The task is successful when the table is clear.• Sort items: This task demonstrates an iterative manipulation program with variable number of objects and conditional logic. The robot must sort the objects on the table into two groups based on their color. The task is successful when all objects are sorted correctly.• Mechanical search: This task demonstrates an iterative manipulation program with variable number of objects and conditional logic. The robot must search for a target object among a set of distractor objects. The task is successful when the target object is found.• Unstack to grasp: This task demonstrates an iterative manipulation program with variable number of objects and conditional logic. The robot must unstack a set of blocks to reveal a target object, then grasp the target object. The task is successful when the target object is grasped.• Conditional pick and place: This task demonstrates a manipulation program with conditional logic. The robot must move an object to a receptacle only if the receptacle is empty. The task is successful when the object is contained inside of the receptacle. • Conditional grasp: This task demonstrates a manipulation program with conditional logic. The robot must grasp an object in one of two ways based on the object's color. The task is successful when the object is grasped correctly.• Assemble bento: This task demonstrates a manipulation program with segmentation. The robot must assemble a bento box by placing each item in the correct compartment. The task is successful when the bento box is assembled correctly.• Fixed grab mug: This task demonstrates a manipulation program with segmentation. The robot must grasp a mug by its handle. The task is successful when the mug is grasped correctly.We evaluate the performance of SHOWTELL with a total of 50 rollout trials per task: 10 demonstrations are provided (2 demonstrations in each environment) and 5 rollouts are performed for each demonstration, resulting in a total of 50 rollouts per task. For tasks that require iteration we vary the number of objects in the scene for each demonstration. For tasks that require conditionals we vary the state of the environment for each demonstration. To evaluate the performance of SHOWTELL across rollouts with unseen objects we perform an additional 10 rollouts each for the pick-and-place, clear table, and conditional grasp tasks with objects that were not present in the demonstration. To evaluate the performance of SHOWTELL across rollouts in unseen environments we perform an additional 10 rollouts each for the pick-and-place and open drawer tasks in a distinct environment from the demonstration. We analyze the failures of SHOWTELL across the benchmark tasks. The distribution of failures across the tasks is shown in Figure 7 . We divide failures into six categories:• Hand-Object Perception: Errors in hand-object perception, where the robot fails to correctly identify the hand or hand-object contact.• Scene Transfer Correspondence Matching: Errors in scene transfer correspondence matching, where the robot fails to correctly match reference points in the demonstration to the new scene.• Other Perception Failures: Errors in object detection, tracking, or other perception tasks.• Failure to Motion Plan: Errors in motion planning, where the robot fails to plan a collision-free trajectory.• Control Execution Failures: Errors in control execution, where a motion plan is generated but the robot fails to execute the plan correctly.• Failure to Meet Task Requirements: Errors in the manipulation program, incorrect execution, or other factors that prevent the robot from successfully completing the task.The most common failure mode is due to errors in hand-object perception, indicating that improvements in hand-object perception could lead to significant improvements in overall performance. Execution is another common source of failure, typically due to imprecise control of the robot's wheeled base. More dynamic control involving visual servoing could mitigate this failure mode in future work. The other failure modes are less common, but still contribute to overall performance.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: This paper introduces a programming by demonstration approach to generate robot programs, relying on a base set of foundation models to generate policies (program synthesis) for new tasks.

Strengths:
* Well written, method attempts to solve a challenging task, a good systems demo about an approach to integrate and generate programs by composing a set of lower level api calls provided by various base models and tools.
* A nice real-world demo

Weaknesses:
* There is a lack of clarity around the specific engineering implementations required to implement this system (what are the prompts, what information is required, what data is collected and what form etc. how were the low level primitives and controllers designed and implemented)
* There is insufficient detail around the tasks used and experimental details.
* While the method does produce a general program to solve complex tasks, this rules based approach requires all existing controllers and modules to be well engineered and functional
* The method does not seem to scale to handle faults or uncertainties, and scalability to longer horizon tasks seems limited
* The scientific contribution here is limited, particularly when considered with regard to prior work in program synthesis.

Reviewers provide a number of questions to be addressed in the rebuttal, in particular around missing information required for reproducibility and greater clarity around the novelty of the contribution.

Post rebuttal:
- The authors added additional detail around apis and prompts and experimental settings, along with greater discussion of limitations and failure analysis, and included more discussion of related work, addressing the bulk of the remaining weaknesses with this paper.

Reviews: review: title: title: Good paper with limited experiments.
paper_summary: paper_summary: The work tackles the challenge of learning a policy for a task by passively observing a human performing it. This problem is difficult because understanding human actions requires joint reasoning about the visual and language cues in the demonstration. Additionally, the learned policy should generalize to similar environments and objects. The proposed approach combines the visuo-lingual reasoning capabilities of vision-language models with task-specific neural networks and other hand-engineered components to learn a policy from human demonstrations and transfer it to new scenes and environments. The authors first preprocess the demonstration into a form suitable for the vision-language model to output the policy as Python code composed of high-level APIs. The evaluation is conducted in real-world manipulation tasks using the Stretch RE2 robot.
main_review: main_review: ## Strengths
- The paper is solid and well-written.
- It addresses the challenging problem of learning control policies for complex tasks, such as "moving all bottles in a tray," from video demonstrations and offers a reasonable solution using vision-language models (VLMs).
- The proposed pipeline significantly outperforms the baseline, with approximately double the performance on iterative and conditional tasks.

## Weakness 
The following are the concerns in decreasing order of importance:

- The paper does not include the prompts used to obtain the plan. Since the method of prompting and the choice of in-context examples can significantly impact performance, this omission poses a challenge to the reproducibility of the results.

- While the authors discuss the implementation of some important APIs, details of others are missing (especially the scene transfer APIs). To understand the applicability and limitations of the work, it is important to discuss the implementation of all the APIs (at least in the appendix).

- It appears that the robot is controlled using end-effector poses. However, the process by which waypoints from the demonstrations are transferred to the end-effector positions in the current scene is not clear. Furthermore, the hand-engineered transfer functions may fail if the object pick-and-place distances vary significantly from those seen in the demonstration. This raises concerns about the generalizability of the learned policy.

- The preprocessing seems to assume the human uses two fingers to manipulate the object, which is the case in all examples and figures provided in the paper. This makes the preprocessing and perception API tailored for such videos and may affect performance if the assumptions are violated.

- The authors state that the evaluation is done on 16 tasks. However, the details of the tasks are missing. Additional information on the type of tasks and how the data is collected would help in understanding the evaluation setup better.

- The experimental setup seems weak and limited compared to the baseline (GPT-4V(ision) for Robotics).

- The authors could provide a video of the hardware demonstration to help understand the qualitative performance of the model and assess the effectiveness of the learned policy.
questions: questions: 0) Address the weakness section in the reviews. 

Additional questions: 

1) I found the neuro-symbolic aspect of the work unclear. Although the term "neuro-symbolic" is used broadly, I am having difficulty relating this work to existing neuro-symbolic studies in the literature. It would be helpful if the authors could clarify why they consider this work specifically neuro-symbolic and clearly delineate what elements of their approach are neural versus symbolic. This discussion would enhance understanding and situate the work within the broader context of neuro-symbolic research.

2) In Figure 4(c), the "cube" seems to be a typo. It should be "bottle" instead. 

3) what happens when multiple instances of an object are present in the scene? How the feature mapping will be done?

4) Are you considering releasing the code?
limitations: limitations: 2
review_summary: review_summary: The paper proposes an interesting method for using vision-language models (VLMs) to learn a policy from human demonstration. In particular, it makes progress toward learning long-horizon policies that require concepts like loops and conditionals. This work could inspire similar research in the future. While the evaluation setting is limited compared to baseline, the inclusion of hardware experiments is indeed an advantage. Overall, I feel the strengths outweigh the weaknesses. I recommend that the paper be accepted, provided the authors address the concerns raised in the review during the rebuttal
title: title: Review
paper_summary: paper_summary: The paper proposes ShowTell, a method to perform Programming by Demonstration by receiving instructions both in the form of language and video. Through a large series of off-the-shelf models, predefined primitives, and APIs, all coordinates by a large VLM, the method can ingest this multimodal data and generate the desired behaviour in new settings-
main_review: main_review: The paper is well-written and clearly specifies the problem at hand, why it is important, how it is addressed, and lists a set of interesting questions in the experiments section. 

Strenghts:
- The method tackles a non trivial problem of teaching robots via multimodal demonstrations in the form of language and video. The authors study the effect of each of these modalities and what improvement they can bring to robot learning.
- The method seems to be effective in transferring the behaviour to new scenes, and a series of interesting real-world experiments are conducted.

Weaknesses:
- The method relies on a very large set of pre-defined models, primitives, and hand-defined APIs. While they allow the method to work and are well orchestrated by GPT-4, this can become a bottleneck to generalise even more to more complex movements and behavior that go beyond pick and place or relatively linear movements.
- A plethora of recent works have used this formulation of a large VLM + code/off-the-shelf models/motion primitives to tackle similar problems, making it difficult to clearly understand the impact and novelty that this method can bring. Nevertheless, I do know that many papers are concurrent work and it would be unfair to assume the authors can both keep track of them all, especially considering the time between submissions, reviews, etc.
Minor point: I do not see any real benefit, scientifically, in using speech as an input with respect to written language. While I appreciate the authors ability to use it as an input modality, which is more natural than writing, it is becoming more and more a slight engineering challenge instead of a fundamental research questions. I would therefore advocate for simplicity instead of over-engineering, as the paper is already very engineering/system heavy.

Overall, I believe this is an interesting project and an interesting contribution, and a certainly challenging method to develop, however it is difficult to distinguish clear scientific contribution with respect building a system with existing tools (similarly to papers like Dobb-E [1] and OK-Robot [2]: while I know they differ in the kind of inputs, the general concept of building a system with many off-the-shelf models is similar). The experimental section is well executed, and the method is certainly valid. The lack of videos, websites, or supplementary material however makes it harder to more deeply understand the nuances of the method, or see the actual inputs and outputs.

[1] Nur Muhammad "Mahi" Shafiullah*, Anant Rai*, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, Lerrel Pinto, Dobb-E: On Bringing Robots Home
[2] Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto, OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics
questions: questions: As stated in the review, I would like the reviewers to better clarify the scientific impact of the paper with respect to its engineering impact, also considering if possible the existing papers I mentioned.
limitations: limitations: 3
review_summary: review_summary: I suggest a weak accept, but I will take into account opinions from other reviewers and make the final decision after the discussion session. To recap, I believe the paper is interesting and the experiments and method are well developed and executed, however it is difficult to really understand what novelty it can present in the wide range of VLM-based papers, and the engineering aspects can surpass the scientific ones, making it look more like a system paper.
title: title: Interesting Approach and Great Results
paper_summary: paper_summary: The paper introduces SHOWTELL, a neuro-symbolic framework that enables robots to learn manipulation skills from a single visual demonstration and natural language instruction, without requiring additional training. It emphasizes generalizability across various tasks and environments, allowing robots to adapt to new objects and settings effectively. SHOWTELL's modular design enhances interpretability and scalability, enabling it to handle complex tasks that involve high-level reasoning. Empirical results demonstrate that SHOWTELL outperforms existing state-of-the-art methods, highlighting its potential to improve the usability and performance of robot programming by demonstration systems.
main_review: main_review: The paper presents a well-structured and methodologically sound approach to teaching robots through the SHOWTELL framework. The experiments conducted are thorough, involving real-world scenarios that validate the effectiveness of the proposed method. The writing is generally clear and accessible, with a logical flow that guides the reader through the problem statement, methodology, and results. The integration of language and visual inputs in a modular neuro-symbolic framework to perform robotic manipulation tasks without any fine-tuning or additional training is a novel and significant contribution.

**Strengths**
1. The framework demonstrates the capability to generalize to new objects and environments without requiring any fine-tuning or additional training, relying solely on a single demonstration.
2. Comprehensive real-world experiments offer compelling evidence supporting the effectiveness of the SHOWTELL framework.
3. The modular architecture facilitates the seamless integration of new pre-trained models as they become available, allowing for straightforward updates to leverage state-of-the-art foundation models.
4. The comparison with strong baseline models yields promising results, demonstrating improvements over existing approaches such as GPT4-V-Robot.



**Weaknesses**

1. The framework may face challenges when dealing with incomplete or poorly executed demonstrations, potentially restricting its effectiveness in real-world applications.
2. The reliance on both visual and language inputs may introduce ambiguities that could complicate the program synthesis if not properly managed. Additionally, there is no mechanism to disregard one of the less reliable or confusing inputs in situations where both inputs are contradictory or unclear.
3. The paper does not discuss or present any failure cases or provide insights on how to resolve them, which limits the understanding of the framework's limitations and potential areas for improvement.
4. The proposed method is closed-loop, which will be a limitation in real-world deployment with dynamic changes and noise.
5. The tasks in the experiments remain short-horizon tasks rather than long, complex tasks that involve conditional multi-step instructions, such as: "If the drawer is open, then retrieve the pen inside it; otherwise, bring me the pen that is on the table."
questions: questions: 1. Curious how the model performs in unseen environments with unseen objects (both).
2. What would be the implications of fine-tuning certain modules within the framework for handling more complex tasks? Will that reduce the reliance on hand-engineered inputs and the need for pre-processing demonstrations to some degree?
3. Can you provide examples of failure cases encountered during experiments, and what insights have been gained regarding the limitations of the approach and potential solutions? 
4. How does the framework adapt to dynamic disturbances or changes in the environment during execution, and what measures are taken to ensure reliable performance under such conditions?
5. Can this framework be adapted to handle long-horizon tasks that involve multiple sub-goals, such as: "If the drawer is open, then retrieve the pen inside it; otherwise, bring me the pen that is on the table"?
6. Is this framework capable of extending to utilising multiple video demonstrations as a sequence of demonstrations for a long horizon task like - Bring me a glass of water and throw this paper in the bin where first part of instruction is shown in one video and the second in another. If we are able to do this then we can compose very complicated tasks by combining short videos for demonstration increase or data as a whole. 
7. Is the framework capable of leveraging multiple video demonstrations as a sequence for executing a long-horizon task, such as: "Bring me a glass of water and throw this paper in the bin," where the first part of the instruction is shown in one video and the second in another? If so, how could this capability enable the composition of more complex tasks by combining shorter video demonstrations?
limitations: limitations: 1
review_summary: review_summary: In summary, I appreciate the contributions and innovative approach presented in this paper. The results are convincing and demonstrate the effectiveness of the proposed framework. It appears to be a strong fit for the conference, and I believe it should be accepted. Additionally, it would be valuable to explore some failure cases and discuss potential strategies for addressing them, as this could further enhance the robustness of the approach.