Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration
Authors: Jennifer Grannen
Siddharth Karamcheti
Suvir Mirchandani
Percy Liang
Dorsa Sadigh
Venue: CoRL 2024 (2024)
Keywords: Continual Learning
Multimodal Teaching
Human-Robot Interaction
TL;DR: We present Vocal Sandbox: a framework for situated human-robot collaboration that adapts to users in real-time, learning new behaviors and low-level skills from mixed-modality feedback.

Abstract: We introduce Vocal Sandbox, a framework for enabling seamless human-robot collaboration in situated environments. Systems in our framework are characterized by their ability to *adapt and continually learn* at multiple levels of abstraction from diverse teaching modalities such as spoken dialogue, object keypoints, and kinesthetic demonstrations. To enable such adaptation, we design lightweight and interpretable learning algorithms that allow users to build an understanding and co-adapt to a robot's capabilities in real-time, as they teach new behaviors. For example, after demonstrating a new low-level skill for "tracking around" an object, users are provided with trajectory visualizations of the robot's intended motion when asked to track a new object. Similarly, users teach high-level planning behaviors through spoken dialogue, using pretrained language models to synthesize behaviors such as "packing an object away" as compositions of low-level skills -- concepts that can be reused and built upon. We evaluate Vocal Sandbox in two settings: collaborative gift bag assembly and LEGO stop-motion animation. In the first setting, we run systematic ablations and user studies with 8 non-expert participants, highlighting the impact of multi-level teaching. Across 23 hours of total robot interaction time, users teach 17 new high-level behaviors with an average of 16 novel low-level skills, requiring 22.1% less active supervision compared to baselines. Qualitatively, users strongly prefer Vocal Sandbox systems due to their ease of use (+31.2%), helpfulness (+13.0%), and overall performance (+18.2%). Finally, we pair an experienced system-user with a robot to film a stop-motion animation; over two hours of continuous collaboration, the user teaches progressively more complex motion skills to produce a 52 second (232 frame) movie. Videos & Supplementary Material: https://vocal-sandbox.github.io

Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration
Jennifer Grannen*, Siddharth Karamcheti", Suvir Mirchandani, Percy Liang, Dorsa Sadigh Stanford University, Stanford, CA https://vocal-sant -sandbox gi thub. io
Abstract: We introduce Vocal Sandbox, a framework for enabling seamless human- robot collaboration in situated environments. Systems in our framework are charac- terized by their ability to adapt and continually learn at multiple levels of abstrac- tion from diverse teaching modalities such as spoken dialogue, object keypoints, and kinesthetic demonstrations. To enable such adaptation, and interpretable learning algorithms that allow users to build an understanding and co-adapt to robot's capabilities in real-time, as they teach new behaviors. For example, after demonstrating a new low-level skill for "tracking around" an object, users are provided with trajectory visualizations of the robot's intended motion when asked to track a new object. Similarly, users teach high-level planning behaviors through spoken dialogue, using pretrained language models synthesize behaviors such as "packing object away" compositions of low-level skills -concepts that can be reused and built upon. We evaluate Vocal Sandbox in two settings: collaborative gift bag assembly and LEGO stop-motion animation. In the first setting, we run systematic ablations and user studies with 8 non-expee non-expert participants, highlighting the impact of multi-level teaching. Across 23 hours total robot interaction time, users teach 17 new high-level behaviors with an average 16 novel low-level skills, requiring 22. 1% less active supervision compared baselines. Qualitatively, users strongly prefer Vocal Sandbox systems due to their ease of use (+31.2%) helpfulness (+13.0%), and overall performance (+18.2%). Finally, we pair an experienced system-user with robot to film stop-motion ani- mation; over two hours of continuous collaboration, the user teaches progressively more complex motion skills to produce a 52 second (232 frame) movie.
design lightweight
Keywords: Continual Learning, Multimodal Teaching, Human--Robo Interaction
Introduction
Effective human-robot collaboration requires systems that can seamlessly work with and learn from people [1-3]. This is especially important for situated interactions [4-7] where robots and people share the same space, working together achieve complex goals. Such settings require robots that concepts online. Yet, recent work [8-15] remain limited in the types of teaching and generalization they permit. For example, many systems use language models to map user utterances to sequences of skills from a static, predefined library [16-18]. While these systems may generalize at the plan level, they trivially fail when asked to execute new low-level skills - regardless of how complex that skill might be (e.g., "hold this still"). Instead, we argue that situated human-robot collaboration requires learning and adaptation multiple levels of abstraction, empowering collaborators to continuously teach new high-level planning behaviors and low-level skills over the course of an interaction.
continually adapt from diverse feedback, learning and grounding
Building on this insight, we present Vocal Sandbox, a framework for situated human-robot collabora- tion that enables users to teach via diverse modalities such as spoken dialogue, object keypoints, and kinesthetic demonstrations. Systems in our framework consist of language model (LM) planner [19] that maps user utterances to sequences of high-level behaviors, and family of low-level skills that
EEuual Contribution. Correspondence to Cigrannen, skaramchttt festanford.edu.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Let's get tracking shot around the Hulk?
Ok now push on Loki then track around the tower,
I'm sarry. don't know how to do that!
tOk_ let' try teaching)
InGtatr IMM Proom)
Leastand)
DEAP tas peas Aarati
cracac ip IMPLgnol- Tomer Tomer Arrcc))
Ok have Learned track!
Figure Motivating Example. present Vocal Sandbox. framework for human robot collaboration that enables robots to adapt and continually learn from situated interactions. this example, user arranges individual LEGO structures for each frame of stop- dtop-motion film [Bottom]. while robot arm controls the camera. The user teaches the robot new behaviors through feedback modalities such as language and demonstrations [Left]] The robot learns online, scaling more complex tasks the collaboration continues [Right].
ground each behavior to robot actions. Crucially, we design lightweight and interpretable learning algorithms to incorporate user's teaching feedback, dynamically growing the capabilities of the system in real-time. Consider user and robot collaborating film LEGO stop-motion animation The user is the animator, articulating LEGO figures and structures rig each keyframe (Fig. Bottom), while the robot carries out precise camera motions (c.g., tracking smoothly to zoom into a character). Early in the interaction, the user asks: "Let's get tracking shot around the Hulk?" In response, the LM attempts to generate a plan, but fails - "tracking" is not known concept. The robot vocalizes its failure, deferring to the user for next steps. Empowered by our framework, the user chooses to explicitly teach the robot how to "track" by providing a kinesthetic demonstration. Using this single example as supervision (Fig. 1; Left), we immediately parameterize a new skill (i.e., (tacc)) and synthesize the corresponding behavior (i.e., track (loc: Location)). Later (Fig. Right), when the user directs to robot to "push in on Loki and then track around the tower," the robot can immediately use what it has learned to generate both the full task plan Ezoom_inLooki] track (Tower)), as well as a visualization of the motion trajectory (shown via custom interface; $3.3). On user confirmation, the robot executes in the real-world, capturing the desired shot.
(Fig. 1).
We evaluate Vocal Sandbox through systematic ablations and long-horizon user studies. In our first setting, non-expert participants work with Franka Emika fixed-arm manipulator for the task of collaborative gift bag assembly; run within-subjects user study with N 8 users spanning total of 23 hours of robot interaction time, comparing a Vocal Sandbox system with two baselines: static variant of our system without learning at either the high- and low-level [8]. as well as a variant of our system with only high-level teaching. We show that across all three methods, our contributions enable users to teach 17 new high-level behaviors and an average of new low-level skills, resulting in 22.15 % faster collaborative task performance compared to baselines; qualitatively, participants strongly prefer our system over all baselines due its ease of (+29.4%) helpfulness 10.8%) and overall performance (+13,9%). We then scale to more advanced setting where experienced system-user (an author of this work) works with robot to shoot a stop-motion animation (Fig. 1); over two hours of continuous collaboration the user teaches progressively more complex concepts, building a rich library of skills and behaviors to produce a 52 second (232 frame) movie.
Vocal Sandbox: Framework for Continual Learning and Adaptation
Vocal Sandbox characterized by a language model planner that maps user utterances to sequences of high-level behaviors and family of skill policies that ground behaviors robot actions (Fig. 2). In this section, first formalize planning ($2.1) and skill execution ($2.2), then introduce our contributions for teaching new behaviors from diverse modes of feedback ($2.3).
Wikipedia provides thorough overview of the rich history and different styles of stop motion animation, with dedicated page "Brickfilm" (e.g.. using LEGOs) the sub-genre focus in this work.

---

class lrttaattiirrtiee
79bot: Rebotinterface
Can you grab the Jelly sandwich thing?
Interpretable Grounding
Fanotions
def goo hooseeeeee - None:
my plan: piok
"Baturnn to neutral home position."
f- "pm_hmme", sage [Loostion value]
OK, now grab the green toy car?
def gotoCohj: ChjjosRaf) - Mona:
"More abova the apeciteed okd"
Ok but you need to teach me what TOY CAR looks Uke!
Teaching New Arguments
relt steil(
F- "goto", ango vahuu
Ofijecs (auuunn)
(Onjeup TOY CAR)
CANDY gummy..
Functions
Now can pack the cendy in the bag?
"do picli the spectfied "bj"
pictecp(ott OhjectRet) None:
Siy missing concent Teach other (bcaback rroduilles
goto(ott); grapp:
Im sorry, don' know how ppack' things Teach me?
= Argaments
class ChjertBaf(Rouzm)
Sure, lots try:
gummo, sandwieb shaped condy.. GIPT BAG gift bag that Itemms"
Pick up the candy
Teaching New Functions
above the bag Drop it.
portttt ObjectBal) - Xone.
clacs (ooossso/EEumm)
*Setrriera the rpecnted oa] and place i5 indde the gift bag"
HOME ([0.86 0.00, 0.403,
packoup(ck)ss (ww(((jeetbuut PAA), relesse(
Figure 2: Planning & Teaching with Language Models. We use language models to parse user utterances to plans executable programs with functions and arguments defined by an API [Left]. Given a successful parse. we visualize an interpretable trace of both the plan and robot's intended behavior on custom GUI (53.3). In the case of failure, we clicit teaching fcedback from users to synthesize new functions and arguments ($2.3).
2.1 High-Level Planning with Language Models
We implement high-level planning using a language model (LM) prompted with an API specificcatin A, (F,C) (Fig. 2; Left) that consists of a set of functions F (synonymous with "behaviors") and argument literals C, indexed by interaction timestep / this indexing is important as users can teach new behaviors (add to the API) over the course of a collaboration. Each argument typed literal - for example, we define Enums such as Location that associate a canonical name (e.g.. Location. HOME in Fig. 2) to values that are used by the robot when executing skill. We define each function f E F as a tuple (n, a, d, b) consisting of a semantically meaningful name n (e.g.. goto), a type signature a (e.g.. [obj: ObjectRef]] + None), human-readable descriptive docstring d (e.g., "Move above the specified obj"), and function body b. Crucially, we assume we can define higher-order functions for example pickup ObjectRef) as goto(obj); grasp().
C is
Given user utterance at time f, the language model generates a plan Pt that is comprised of a program, or sequence of function invocations. For example, a valid plan for an in- struction such as "place the candy in the gift bag" subject to the API in Fig. 2 would be Pt = pickup (ObjectRef. CANDY); gobj(Obeet CFFFF.IIITBGG) release We formalize planning as generating pe = LM(- ue, Aq, h4), where d4 is the user's utterance, Ar is the current API, and h, corresponds to the full interaction history through I. Concretely, he (Uprompt, [[u1, P1), (2), .... (u--1, Pt-1)]); Uprompe is the system prompt that describes the setting and constraints (e.g., "return plans as well-formed Python"; see our project page for full prompts).
Note that there are cases where the language model may not be able to generate a valid plan - for example, given an utterance describing a behavior that does not exist in the API (e.g., "can you pack a gift bag with three candies"). In such a situation, we rely on the commonsense understanding of the LM to generate a helpful error message that indicates the failure mode (c.g., "I am not sure how to pack; could you teach me?"). These eπror messages (as well as the successfully generated plans) are shown to users via a custom graphical interface (GUI; $3.3) to inform their next action.
2.2 Low-Level Skills for Action Execution
To ground plans P; to robot actions, we assemble skill policies ya | Or, [c|, .I) that define the exccution of a function f E F as a mapping from a state observation ON E RR" and individual "resolved" arguments [[1, (2, to a sequence of robot actions a E Tx (e.g., end-effector poses). Of particular note is the resolution operator [-I that maps an LM-generated plan P: to a sequence of skill policy invocations; to do this, first iterate through each function invocation in Pe and recursively expand higher-order functions necessary. In our "place the candy in the gift bag"
3

---

example, this means expanding the initial pickup (Objeuptb Candy) to its resolved function body [[ goto (ObjectRef Candy) and release (). We then resolve each argument, substituting each literal (variable name) with its corresponding value for example [objectRef. Candy] = "A gummy, sandwich-shaped candy" while [Location. HOME] = ((0.36, 0.00, 0.49 pos, [1, O, O, mm.)
The corresponding values are passed to the underlying policy implementation z.-
In this work, we look at three different classes of skill policies: hand-coded primitives (e.g., GO GOLHOME or GRASP), visual ceypoint-conditioned policies that identify end-effector poses given natural language object descriptions, and dynamic movement primitives ($3.2). Note that in general, our formulation permits arbitrary policy classes, including learned closed-loop control policies [20-22].
2.3 Teaching via Program Synthesis
core component of Vocal Sandbox is the ability to synthesize new behaviors and skills from diverse modalities of teaching feedback such as spoken dialogue, object keypoints, and kinesthetic demonstrations. To do this, we leverage the commonsense priors and strong generalization ability of our underlying language model to synthesize new functions and arguments, updating the API A, in real-time. Given an failed plan, each "teaching" step first uses the language model to vocalize the "missing" concept(s) to be taught, followed by a interaction that prompts the user to provide targeted feedback. The language model then synthesizes new functions and arguments for the API. Fig. (Right) shows the two types of teaching we develop: 1) argument teaching and 2) function teaching.
The goal of argument teaching is to teach and ground new literals è E C for example, identifying "green toy car" as new argument given the utterance "Grab the green toy car"' in Fig. 2 (Right). To do this, the LM parses as much of the utterance as possible subject to the current API, mapping "grab" to pi pick.p Because is able to identify the correct function (but not the arguments), the LM then uses the corresponding type signature to infer that "green toy car" should be of type ObjectRef: then automatically synthesizes an API update, adding new literal ObjectRef. TOY_CAR. This addition is then shown to the user (as part of the second stage of teaching), who then then provides the supervision needed to successfully ground the literal (i.c., providing the keypoint supervision to localize the object). On successful execution, the LM commits this change, yielding A4+1-
The goal of fumction teaching is to teach new functions f E F - for example, defining pack from the utterance "now can you pack the candy in the bag" in Fig. 2 (Right). Here, the LM cannot partially parse the utterance - "pack" does not have an associated function, there is no reliable way infer type signature. Instead, the LM highlights "pack" as new behavior, and explicitly asks the user to teach its meaning through decomposition [23, 24], breaking "pack" down into a chain of existing skills. In this case, the user says: "Pick up the candy; go above the bag; drop it" with program (ObjectRef CANDY); goto the new function f = (f, ā, d). with name ñ = pack, signature â = obj: ObjectRef - None, and docstring = "Retrieve the object and place it in the gift bag." We also generate the "lifted" body dickup(obj). goto ; release via first-order abstraction [24, 25].
release () The LM then explicitly synthesizes
This combination of argument and function teaching enables the expressivity and real-time adaptivity of our framework; in defining lightweight algorithms for learning and synthesizing new API specifi- cations from interaction, we provide users with reliable method of growing and reasoning over the robot's capabilities during the course of a collaboration.
3 Implementation & Design Decisions
While we introduce Vocal Sandhox as a general learning framework, this section provides implemen- tation details specific to the experimental settings we explore in our experiments. Notably, our first experimental setting involves a collaborative gift-bag assembly task ($4.1) where a non-expert user and robot work together to assemble a gift bag with set of known objects (visualized in Fig. 2). Our second setting ($4.2) pairs an experienced system user (an author of this work) with robot for the task of LEGO stop-motion animation (visualized in Fig. 1). For all settings, we use a Franka Emika Panda arm equipped with Robotig 2F-85 gripper following DROID [26]; we also assume access to an overhead ZED 2 RGB-D camera with known intrinsics and extrinsics to obtain point clouds.

---

Using Taught Behaviors
Teaching New Skills via Keypoint Supervision
Pack the toy car gift bag.
Did find the cer?
find the bug?
Does this plan look gnod?
Yeaf
pask(TOY CAR) piokup(Tay CAR) gnto(AIPT_BAG)) Nelesse()
Click to Teach
VISS
D CAB)
pickup( TTO CAR) (LIT valeasa))
plokupxTOY._CA goto CETTGBAAAA
plokugrtov CAR) (oSa(G1PP_BA)) relosac()
(d (RAO)
relatse
relesse)
1-13
( 35
[-38
t- 5-
Collaborative Gift-Bag Assembly, In this example, study participant ($4.)) 1) verbally asks the robot to "pack the toy car in the gift bag." leveraging pack, a newly taught behavior, to minimize his time supervising [Left]. When the robot fails to localize the "car" in the image, the user corrects this by clicking on the interactive GUI, producing keypoint label, teaching new argument and grounding corresponding skill [Right].
Figure 3:
3.1 Language Models for High-Level Planning
We use GPT-3.5 Turbo with function calling v11-06; 27, 28] due its high latency and cost-effective pricing. We encode A, as a Python code block (formatted as Markdown) the system prompt Uprompt- To constrain the LM outputs to well-formed programs, by the OpenAl chat completion endpoint [28], formatting each function in A, "tool" that the LM can invoke in response to user endpoint. Full code and prompts are on our project page.
the function calling feature provided
3.2 Skill Policies for Object Manipulation and Dynamic Motions
We implement different families of skill policies f for each of our two experimental settings. In our first setting ($4.1) we implement skills using a visual keypoints model, while for our second setting ($4.2) we implement skills as dynamic movement primitives [DMPs; 29, 30].
Visual Keypoint- Conditt-oodd Policies for Object Manipulation. For the collaborative gift-bag assembly setting ($4.1), we implement skills Tgoto and Tpickup via learned keypoint-conditions models that ground object referring expressions (e.g., "a green toy car") to point clouds in scene [3]] Specifically, given an (image, language) input, we first predict a 2D keypoint (z. y) that corresponds to the centroid of the desired object pixel space, then use off-the-shelf models (i.e., FastSAM; 32] to produce an object segmentation mask. Finally, we deproject this mask through our calibrated camera obtain the object's point cloud to inform manipulation. To ensure consistency, use Iearned mask propagation model [XMem; 33] to maintain a dictionary mapping language expressions to existing object masks this allows to ensure that referring expressions such as "the green toy car" always refer to the same object instance for the duration of the interaction (as opposed to predictions changing over time, confusing users) We provide further detail in $B.3.
Note that we adopt this modular approach (predict keypoints from language, then extract segmen- tation mask) for two reasons. First, found learning visual keypoints model to be extremely data efficient (requiring only couple dozen examples) and reliable, significantly outperforming existing open-vocabulary detection and end-to-end models such as OWL-v2 [34, 35] as well as vision- language foundation models such as GPT-4V [36]; we quantify these results via offline evaluation in SB.3. Second, we found keypoints to be the right interface for interpretability and teaching: to specify new object, users need only "click" on the relevant part of the image via our GUI ($3.3).
Learning Dynamic Movement Primitives from Demonstration. For our stop-motion animation setting ($4.2), the robot is tasked with controlling the camera to execute different cinematic motions such as panning, tracking, or zooming (amongst others); due to the dynamic nature of these motions, we implement skills as (discrete) DMPs [29, 30], where users teach new motions by providing a single kinesthic demonstration; using DMPs allows us to generalize motions to novel start and goal positions, while also providing nice affordances for re-timing trajectories (e.g., speeding up a motion by a factor of 2x, or constraining that motion executes in K steps). Furthermore, as rolling out DMP generates entire trajectory,
provide users with visualization of the robot's intended path via our custom GUI ($3.3; visualized in Fig. 6). To learn a DMP, we divide user's
5

---

Robot Supervision Time
Behavior Complexity
Skill Fallures
VS- Blow, High)
64
Skils Execuite per Betawker
S411 Paluros
Timo (Minutes!
Bag 1
Bag
Bag
Bog
Dag
Deg
Dag
Baga
User
VS- Blow, Hight
VS-
Low
VS (Ours)
Time Spont
Instrutting
Confimming
Perellelzing
Figure 4: User Study Quantitative Results. We report robot supervision time [Left], behavior complexity [Middle]. and skill failures [Right] across assembling individual gift bags in our user study ($4.1). Over time, users working with Vocal Sandbox (VS) systems teach more complex high-level behaviors, see fewer skill failures, and need to supervise the robot for shorter periods of time compared to baselines.
demonstration into fixed sequence of waypoints, and learn a of radial basis function weights Strack from Fig. 1) that smoothly interpolate between them; we provide further detail in $B.4.
3.3 A Structured Graphical User Interface for Transparency and Intervention
Finally, we design a graphical user interface (GUI; Fig. 5) to provide users with transparency into the system state and to enable targeted teaching. Consider an utterance "pack the candy" and a failure mode that has the robot packing different object (e.g., the ball) instead. Without any additional information, it is impossible for the user to identify whether this failure failure in the skill policy
result of
the LM planner (e.g., generating the incorrect plan pack (ball)), (e.g., incorrectly predicting keypoint on the ball instead of the candy). The GUI counteracts this confusion by explicitly visualizing the plan and the interpretable traces produced by each skill (e.g.. the predicted keypoints and segmentation mask, or the robot's intended path as output by a DMP) these interpretable traces also double as interfaces for eliciting teaching feedback (e.g.. "clicks" to designate keypoints). The GUT also provides 1) the transcribed user utterance, 2) the current "mode" (e.g., normal execution, teaching, etc.), and 3) the prior interaction history.
Experiments
We evaluate the usability, adaptability,, and helpfulness of Vocal Sandbox systems through two experimental settings: 1) real-world human-subjects study with N = 8 inexperienced participants for the task of collaborative gift bag assembly ($4.1), and 2) more complex setting that pairs an experienced system-user (author of this work) and robot to film a stop-motion animation ($4.2). All studies were IRB-approved with all participants providing informed consent.
4.1 User Studies: Collaborative Gift Bag Assembly
This study has participant work with the robot to assemble four gift bags with fixed set of objects (candy, Play-Doh, and toy car), along with hand-written card (transcribed from 96-word script). This task is repetitive and time-intensive, serving to study how well users teach the robot maximally helpful behaviors to parallelize work and minimize their time spent supervising the robot.
Participants and Procedure. We conduct a within-subjects study with 8 non-expert user (3 female/5 male, ages 25.2 1.22). Each user assembled four bags with three different systems, with a random ordering of methods across users. Prior to engaging with the robot, we gave users a sheet describing the robot's 's capabilities (i.e., the base API functionality), and instructions for using any teaching interfaces (iff applicable). Prior to starting the bag assembly task, users were allowed to practice using each method for the unrelated task of throwing (disjoint) objects from a table into a trash can.
Independent Variables Robot Control Method. We compare the full Vocal Sandbox system (VS) to two baselines. First, VS (Low, High) (without Low, High), static version of Vocal Sandbox that ablates both low-level skill teaching and high-level plan teaching, reflecting prior work like MOSAIC [8] that assume fixed skill library and LM planner seeded with base API. Our second bascline VS
Low ablates lowelevel skill teaching, but allows for teaching new high-level behaviors.

---

Qualitative User Rankings
Graphical User Interface
hward:
Ostrut: Clikkon Teech
Object
User Rankings
200 Sng
3
2D
Ease
Helpful
Intuitive
Prodictable
Trust
Use Again
Finsk andi
VS (Low, High!
VS Low
VS (Ours)
Figure 5: User Study Subjective Results and System GUL We report qualitative user rankings for Vocal Sandhox (VS) and two baselines. With high significance (p < 0.05), we ohserve that VS outperforms the VS- (Low, High) baseline across all measures except predictability and trust [Left]] We also visualize the graphical user interface (GUT; $3.3) displayed to users when interacting with the system [Right].
Dependent Measures. We consider objective metrics such as time supervising the robot, number of commands spoken, number of low-level skills exccuted per command (behavior complexity), as well as the number of teaching interactions at both the high- and low-level. Intuitively, we expect that methods capable of learning high-level behaviors from user feedback require less active supervision time and yield more complex behaviors than those using static (naive) planner; similarly, expect methods that permit teaching new low-level skills see fewer low-level skill execution errors. In addition to objective metries, we asked users to fill out a qualitative survey after engaging with each method, describing their experience and ranking each method.
Results Objective Metrics. We report objective results in Fig. We measure robot supervision time (Fig. 4; Left) proxy for robot capability more capable robot should require less supervision (as it performs more actions autonomously). We observe that Vocal Sandbox (VS) systems outperform both the VS - (Low, High) and VS Low baselines in terms of supervision time, demonstrating the capabilities afforded by the ability to teach at multiple levels of abstraction; the strengths of VS systems are made further evident in visualizing how users spend their time during a collaboration across instructing, confirming, and parallelizing (Fig. 4; Bottom-Left).
We additionally measure the complexity of (taught) high-level behaviors, measures the number of low-level skills exccuted per language utterance (Fig. 4: Middle). We find that VS systems yield increasingly more complex behaviors compared to both baselines, with significantly (p < 0.05) more complex behaviors taught (and used) compared to the VS - (Low, High) baseline. This further highlights the importance of high-level teaching via structured function synthesis ($2.3). Of equal importance is the ability of the robot to learn low-level skills, exhibiting a declining rate of skill execution failures over time (Fig. 4; Right). We see that VS systems show significantly (p < 0.05) fewer skill failures than both baselines, stressing the isolated importance of teaching the low-level. We further note that VS systems show fewer skill failures function of numbers of bags assembled, demonstrating the ability of VS systems to improve over time.
Results Subjective Metrics. We report subjective survey results in Fig. 5 (Left), where users ranked the three methods across six different measures. terms of ease, helpfulness, intuitiveness, and willingness to use again, we find that users prefer our VS system over both baselines due to its transparency and teachability, as users noted "teaching is useful" and "I loved how was able to teach the robot certain skills". We also note that users significantly (p < 0.05) prefer VS the naive VS- (Low, High) baseline, as it "felt chunkier to use". For predictability and trust, we observe that users prefer our VS system over baselines, however this trend is less pronounced because any autonomous exccution incurs some loss of predictability (due to imperfect robot execution with learned policies).
4.2 Scaling Experiment: LEGO Stop Motion Animation
Finally, to push the limits of our framework, we consider LEGO stop motion animation setting, where an experienced system-user (author of this work) works with the robot over two hours of continuous collaboration shoot ttopoootion film. The user is the director, leading the creative vision for the film by directing footage and arranging LEGO set pieces, while the robot controls the
7

---

Limitations and Future Work. As execution relies on low-level skills that are quickly adapted from sparse feedback, this framework struggles dexterous settings (e.g., assistive bathing) where more data necessary capture behavior nuances. Another shortcoming is that the collaborations enabled camera to capture different types of dynamic shots. Specifically, the user teaches the robot to perform cinematic concepts including "tracking", "zooming" and "panning" via kinesthetic demonstrations, which we use to fit different DMPs (as described in $3.2). The user iteratively builds on these behaviors to shoot progressively more complex frame sequences, ultimately resulting in a 52-second stop-motion film, consisting of 232 individual frames. Of the total frame count, 43% were shot with completely autonomous dynamic camera motions taught by the user that is, camera motinos where the LEGO scene remained fixed, but the camera moved subject to a DMP rollout (e.g.. a "zoom-in" shot). We found the taught skills were able to generalize across different start and end positions, as well as different timing constraints - for example "pan around slowly" commanding a arouud motion with more frames (N = 30; 8 seconds), while "pan around quickly" modulated the same skill to execute in fewer frames (N = 8; 1.33 seconds). Over the two-hour long interaction, the robot executed on 40 novel commands commands such as "let's frame the tower in this shot" to "zoom into Iron Man" (see Appx. and our project page for more examples).
5 Related Work
Vocal Sandbox engages with large body of work proposing systems for human-robot collaboration that pair task planning with with learned skill policies for executing actions; we provide an extended treatment of related work in SC.3, focusing here on prior work that center language models for task planning, or introduce generalizable approaches for learning skills from different feedback modalities.
Task Planning with Language Models. Recent work investigates methods for using LMs such as GPT-4 and PaLM [37-39] for general-purpose task planning [9, 16-18]. Especially relevant are approaches that use LMs to generate plans as programs [19, 40-43]. While some methods explore using LMs to generate new skills e.g.. by parameterizing reward functions [44, 45] they require expensive simulation and offline learning. In contrast, Vocal Sandbox designs lightweight learning algorithms to learn new behaviors online, from natural user interactions.
Learning Generalizable Skills from Mixed-Modality Feedback. A litany of prior approaches in robotics study methods for interpreting diverse feedback modalities, from intent inference in HRI [8, 46, 47] to learning from implicit expressions [11, 48] or explicit gestures [12, 49, 50]. With the advent of LMs, language has become an increasingly popular interaction modality; however, most methods are limited to specific types of language feedback such as goal specifications [13, 51] or corrections [14, 52, 533. In contrast, Vocal Sandbox demonstrates that language alone is not sufficient when teaching new behaviors especially for teaching new object groundings or dynamic motions. Instead, framework leverages multiple feedback modalities simultaneously guide learning.
6 Discussion
We present Vocal Sandbox, framework for situated human-robot collaboration that continually learns from mixed-modality interaction feedback in real-time. Vocal Sandbox has two components: a high-level language model planner, and a family of skill policies that ground plans to actions. This decomposition allows users to give targeted feedback at the correct level of abstraction, in the relevant modality. We evaluate Vocal Sandbox system in a user study across N = participants, and observe that our Vocal Sandbox system is preferred by users (+18.2%), requires minimal supervision (-22.1%) and yields more complex autonomous performance (+19.7%) with fewer failures (-67.1%) compared to non-adaptive baselines.
by our system relatively homogeneous users teachers and robots are followers which is not suited for all settings. Future work will explore algorithms for cross-user improvement well as sample-efficient algorithms to learn more expressive skills. We will also further probe the user's model of robot capabilities to investigate questions about human-robot trust and collaboration styles.
8

---

Acknowledgments
This work was supported by the Toyota Research Institute (TRI), the DARPA Friction for Account- ability in Conversational Transactions (FACT) Program, the AFOSR Young Investigator Program, the Cooperative AI Foundation, the NSF (Awards # #1941722, #2006388, #2125511), 1), the Office of Naval Research (ONR Award #N0001421 12298), and DARPA (Grant #W91 INF22102144 Jen- nifer Grannen is grateful to supported by the NSF Graduate Research Fellowship Program (GRFP). Siddharth Karamcheti is grateful to be supported by the Open Philanthropy Project AI Fellowship. Any opinions, findings, and conclusions expressed in this article reflect those the authors and do necessarily reflect the views of the sponsoring agencies. Finally, we thank Yuchen Cui, Suneel Belkhale, David Hall, and our anonymous reviewers for their thoughtful feedback and suggestions.
References
[1] C. L. Breazeal, G. Hoffman, and A. L. Thomaz. Teaching and working with robots as collaboration. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2004.
[2] A. Ajoudani, A. M. Zanchettin, S. Ivaldi, A. O. Albu-Schäffer, K. Kosuge, and O. Khatib. Progress and prospects of the human-robot collaboration. Autonomous Robots (AURO), pages 957-975, 2017.
[3] S. Tellex, N. Gopalan, H. Kress-Gazie and C. Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):25-55, 2020.
[4] G. Hoffman and C. Breazeal. Collaboration in human-robot teams. In AIAA Intelligent Systems Technical Conference (AIAA-IST), 2004.
[5] P. E. Rybski, K. Yoon, J. Stolarz, and M. M. Veloso. Interactive robot task training through dialog and demonstration. In ACM/IEEE International Conference on Human Robot Interaction (HRI), 2007.
[6] J. Y. Chai, L. She, R. Fang, S. Ottarson, C. Littley, C. Liu, and K. Hanson. Collaborative effort towards common ground in situated human-robot dialogue. In ACM/EEE International Conference on Human Robot Interaction (HRI), 2014.
[7] Brawer, O. Mangin, A. Roncone, S. Widder, and B. Scassellati. Situated human- collaboration: predicting intent from grounded natural language. International Conference on Intelligent Robots and Systems (IROS), 2018.
[8] H. Wang, K. Kedia, Ren, R. Abdullah, Bhardwaj. A. Chao, K. Y. Chen, N. Chin, Dan, X. Fan, G. Gonzalez-Pumariega,, A. Kompella, M. A. Pace, Y. Sharma, X. Sun, N. Sunkara, and S. Choudhury, Mosaic: A modular system for assistive and interactive cooking, arXiv preprint arXiv:2402. 18796, 2024.
[9] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine. K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning (CoRL), 2022.
[10] A. Z. Ren, A. Dixit, A. Bodrova, Singh, S. Tu, Brown, P. Xu, L. Takayama, F. Xia, Varley, 7. Xu, Sadigh, A. Zeng, and A. Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In Conference on Robot Learning (CoRL), 2023.
[11] Y. Cui, Q. Zhang, A. Allievi, Stone, S. Nickum, and W. B. Knox. The EMPATHIC framework for task learning from implicit human feedback. In Conference Robot Learning (CoRL), 2020.

---

[12] L.-H. Lin, Y. Cui, Y. Hao, F. Xia, and D. Sadigh. foundation models. In Conference on Robot Learning (CoRL.), 2023.
robot assistance via
[13] C. Lynch, A. Wahid, J. Tompson, I. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters (RA-L), 1, 2023.
[14] L. X. Shi, Z. Hu, T. Zhao, A. Sharma, K. Pertsch, J. Luo, S. Levine, and C. Finn. Yell your robot: Improving on-the-fly from language corrections. In Robotics; Science and Systems (RSS), 2024.
[15] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, K. Choromanski, T. Ding, D. Driess, C. Finn, P. R. Florence, C. Fu, M. G. Aremas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, J. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, I. Leal, S. Levine, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. S. Ryoo, G. Salazar, P. R. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. H. Vuong, A. Wahid, S. Welker, P. Wohlhart, T. Xiao, T. Yu, and B. Zitkovich. RT-2: Vision-language- action models transfer web knowledge robotic control. arXiv preprint arXiv:2307.15818 2023.
[16] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, Herzog, D. Ho, Hsu, J. Iharz, Ichter, A. Irpan, E. Jang, J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, Quiambao, K. Rao, J. Rettinghouse, D. M. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[17] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: from natural language instructions to feasible plans. Autonomous Robots (AURO), 2023.
[18] Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In International Conference on Robotics and Automation (ICRA), 2022.
[19] Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and Zeng. Code as policies: Language model programs for embodied control. In International Conference on Robotics and Automation (ICRA), 2023.
[20] Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research (JMLR), 17, 2016.
[21] S. Stepputtiss, J. Campbell, M. Phielipp. S. Lee, C. Baral, and B. Amor. Language-condttioned imitation learning for robot manipulation tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[22] C. Chi, S. Feng, Y. Du, Z. Xu, E. A. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023.
[23] S. I. Wang, S. Ginn, P. Liang, and C. D. Manning. Naturalizing programming language via interactive learning. In Association for Computational Linguistics (ACL), 2017.
[24] S. Karamcheti, D. Sadigh, and P. Liang. Learning adaptive language interfaces through decom- position. In EMNLP Workshop for Interactive and Executable Semantic Parsing (IntEx-SemPPr), 2020.
[25] Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Empirical Methods in Natural Language Processing (EMNLP), pages 1223-1233, 2010.
10

---

[26] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, I.. Y. Chen, K. Ellis, P. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Y. Park, Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, Lin, Lu, J.-P. Mercat, A. Rehman, R. Sanketi, A. Sharma, C. B. Simpson, Q. H. Vuong, H. Walke, B. Wulfe. T. Xiao, H. Yang, A. Yavary, T. Zhao, C. Agia, R. Baijal, M. G. Castro, D. L. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O O'Neill, R. M. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman,
Lim, Malik, R. Martin-Martin, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip. Zhu, T. Kollar, S. Levine, and C. Finn. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems (RSS), 2024.
[27] OpenAL. Introducing ChatGPT and Whisper APIs. hhaprrcng-cchattt-----------eepe
https:///openai.com/inni
2023.
[28] OpenAL. GPT-3.5 function calling and other updates. https://opeeaii.cm//inni function-callin--add-- otttatother-r--pppppassssss//// 2023.
[29] S. Schaal. Dynamic movement primitives a framework for motor control in humans and humanoid robotics. Adaptive Motion of Animals and Machines, 1, 2006.
[30] A. J. Ijsppent, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25, 2013.
[31] Sundaresan, S. Belkhale, D. Sadigh, and Bohg. KITE: Keypoint-conditioned policies for semantic manipulation. In Conference on Robot Learning (CoRL), 2023.
[32] X. Zhao, W. Ding, Y. An, Y. Du, T. Yu, Li, M. Tang, and J. Wang. Fast segment anything. arXiv preprint ar arXiv: 12/56, 2023.
[33] H. K. Cheng and A. Schwing. XMem: Long-term video object segmentation with an atkinson-
shiffrin memory model. In European Conference on Computer Vision (ECCV), 2022.
[34] M. Minderer, A. A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy,
A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby. Simple open- open-vocabulary object detection with vision transformers. European Conference on Computer Vision (ECCV), 2022.
[35] M. Minderer, A. A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[36] OpenAL.
GPT-4v(ision) system card. gpt- gpt-4v- ppt-4y-system-carrd 2023.
https://openal http://pppeaii.ooommeeeeeeet cww/research
[37] OpenAl. Introducing ChatGPT. https://openaal http:://ppeniii.ccmmmbbo///aaggp 2022.
[38] OpenAI, Achiam, S. Adler, Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, Babuschkin, S. Balaji, V. Balcom, P. Bal- tescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapirg C. Berner, L. Bogdonofff, O. Boiko, M. Boyd, A.L Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Dai, C. Decareaux, Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Bcoffet, A. Eleti, T. Eloundou,
D. Farhi, L. Fedus, N. Felix, S. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson,
V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene,

---

J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke,
C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang. H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaf- tan, L. Kaiser, A. Kamali, Kanitscheider, N. Keskar, T. Khan, L. Kilpatrick, W. Kim, C. Kim, Kim, H. Kirchner, J. R. Kiros, M. Knight, D. Kokotajlo, L. Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez. R. Lowe, P. Lue, A. A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mchta, J. Menick, L. Metz, A. Mishchenko, Mishkin, V. Monaco, E. Morikawa, D. P. Mossing, T. Mu, M. Murati, O. Murk, D. Mely, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, O. Long, C. O'Keefe, J. W. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, M. Pokorny, M. Pokrass, V. H. Pong. T. Powell, A. Power, B. Power, F. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. D. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. D. Sokolowsky, Y. Song, N. Staudacher, F. Such, N. Summers, Sutskever, Tang, N. A. Tezak, M. Thompson, Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang. J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng. M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. 7ellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[39] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. M. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Brradburyy J. Austin, M. Isard, G. Gur-Ari, P. Yin. T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. García, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellsterm, D. Eck, J. Dean, S. Petrov, and N. Fiedel. PaLM: Scaling language modeling with pathways. arXiv, 2022.
[40] P. Liu, Y. OrTu, C. Paxton, N. M. M. Shafiullah, and L. Pinto. OK-Robot: What really matters in integrating open-knowledge models for robotics. In Robotics: Science and Systems (RSS), 2024.
[41] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. J. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions of Machine Learning Research (TMLR), 2023.
[42] W. Huang, C. Wang, R. Zhang, Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning (CoRL), 2023.
[43] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. A. Funkhouser. Tidybot: Personalized robot assistance with large language models. In International Conference on Intelligent Robots and Systems (IROS), 2023.
[44] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, Erez, L. Hasenclever, J. Humplik, B. Ichter, T. Xiao, P. Xu, A. Zeng, T. Zhang, N. M. O. Heess, D. Sadigh, J. Tan, Y. Tassa, and F. Xia. Language rewards for robotic skill synthesis. arXiv preprint arXiv:2306 08647, 2023.
12

---

[45] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. In International Conference on Learning Representations (ICLR), 2024.
[46] K. Kedia, A. Bhardwaj, P. Dan, and S. Choudhury. InteRACT: Transformer models for human intent prediction conditioned on robot actions. In International Conference on Robotics and Automation (ICRA), 2024.
[47] K. Kedia, Dan, A. Bhardwaj. and S. Choudhury. Manicast: Collaborative manipulation with cost-aware human forecasting. In Conference on Robot Learning (CoRL), 2023.
[48] R. Arakawa, S. Kobayashi, Y. Unno, Y. Tsuboi, and S. ichi Maeda. DQN-TAMER: Human-in- the-loop reinforcement learning with intractable feedback. arXiv preprint arXiv:/810.11744 2018.
[49] D. Whitney, M. Eldon, J. G. Oberlin and S. Tellex. Interpreting multimodal referring expressions in real time. In International Conference on Robotics and Automation (ICRA), pages 3331-3338, 2016.
[50] C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. In Association for the Advancement of Artificial Intelligence (AAAI), 2014.
[51] Sharma, B. Sundaralingam, Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and D. Fox. Correcting robot plans with natural language feedback. In Robotics: Science and Systems (RS5), 2022.
[52] Y. Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. "No, to the right"- online language corrections for robotic manipulation via shared autonomy. In ACM/IEEE International Conference on Human Robot Interaction (HRI), 2023.
[53] L. Zha, Y. Cui, L.-H. Lin, M. Kwon, M. G. Arenas, A. Zeng, F. Xia, and D. Sadigh. Distilling and retrieving generalizable knowledge for robot manipulation via language corrections. In International Conference on Robotics and Automation (ICRA), 2024.
[54] A. Radford, J, W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scall scale weak supervision. arXiv preprint arXiv:22/2.043556 2022.
[55] OpenAL. Text-to-speech models.
https://platformm. http::///ppaaffooooppaaa.. openai. com/ coom/docs//a gggg /
com/docs/guidess
[56] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems (IROS), pages 5026-5033, 2012.
[57] M. Shridhar, Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning (CoRL), 2021.
[58] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), volume 139, pages 8748-8763, 2021.
[59] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
[60] G. Jocher, Stoken, J. Borovec, NanoCode012, ChristopherSTAN, Laughing, tkianai,
A. Hogan, lorenzomammana, yxNONG, Alex 1900, Diaconu, Mare, wanghaoyang0106. ml5ah. Doug, F. Ingham, Frederik, Guilhen, Hatovix, J. Poznanski, J. Fang. L. Yu, changyu98, M. Wang, N. Gupta, O. Akhtar, PetrDvoracck, and P. Rai. YOLO-v5 repository, 2020.
13

---

[61] Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and F. Meier. Polymetis. https://faacebookresettt github. wwtoaiirooppee stt/frr/oollmmmtss///////// 2021.
[62] S. Belkhale, T. Ding, T. Xiao, P. Sermaner, Q Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. RT-H: Action hierarchies using language. In Robotics: Science and Systems (RSS), 2024.
[63] K. Burn,, Z. Witzel, J. I. Hamid, T. Yu, C. Finn, and Hausman. What makes pre-trained visual representations successful for robust manipulation? arXiv preprint arXiv:2312. 12444 2023.
[64] A. Xie, L.. Lee, Xiao, and C. Finn. Decomposing the generalization gap imitation learning for visual robotic manipulation. In International Conference on Robotics and Automation (ICRA), 2024.
[65] Liu, K. Fang, P. Abbeel, and S. Levine. MOKA: Open-vocabulary robotic manipulation through mark-based visual prompting. In Robotics: Science and Systems (RSS), 2024.
[66] M. Natarajan, E. Seraj, B. Altundas, R. R. Paleja, S. Ye, L. Chen, R. Jensen, K. C. Chang, and
M. C. Gombolay, Human-robot teaming: Grand challenges. Current Robotics Reports, 4, 2023.
[67] A. D. Dragan and S. Srinivasa. policy- policy-blending formalism for shared control. International Journal of Robotics Research (IJRR), 32:790-805, 2013.
[68] S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell. Shared autonomy via hindsight optimization for teleoperation and teaming. International Journal of Robotics Research (IRR), 37:717-742, 2018.
[69] S. Karamcheti, M. Srivastava, P. Liang, and D. Sadigh. LILA: Language-informed latent actions. In Conference on Robot Learning (CoRL), 2021.
[70] B. Driessen, H. Evers, and J. A. Woerden. MANUS a wheelchair-mounce rehabilitation robot. Proceedings of the Institution of Mechanical Engineers: Journal of Engineering in Medicine, 215, 2001.
[71] B. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. RAS, 2009.
[72] D. Losey, C. G. McDonald, E. Battaglia, and M. K. O'Malley. A review of intent detection, arbitration, and communication aspects of shared control for physical human-robo interaction. Applied Mechanics Reviews, 70, 2018.
[73] S. Tellex, Kollar, Dickerson, M. R. Walter, A. G. Banerjee, J. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Association for the Advancement of Artificial Intelligence (AAAI), 2011.
[74] S. Tellex, R. Knepper, A. Li, D. Rus, and N. Roy. Asking for help using inverse semantics. In Robotics: Science and Systems (RSS), 2014.
14

---

Overview
In the appendices below, we provide additional details around the implementation the various components of both Vocal Sandbox systems (i.e., for both the collaborative gift-bag assembly and the LEGO stop-motion settings), including details around the rest of the system architecture (e.g., speech recognition, text-to-speech). We then extend discussion from the main body of paper, building additional opportunities provided by our framework for future research.
large portion of our paper is grounded user studies and extended interactions; videos of these studies can be found on our project website: https://vocal sandbox. gi tiub.
Furthermore, our work involves prompting GPT-3.5 Turbo with function calling [v11-06; 27, 28] through the public OpenAl API; we provide these prompts directly (as code) on our website as well: https://vocee -anddbox gi thub. 1o/ prooutt
An overview of each appendix as follows:
Appx. A - Motivating Questions
We index a list of motivating questions that may arise from reading the main text and that expand on further here (e.g.. "why adopt modular vs. an end-to-end approach?"".
Our answers here are direct, linking to concrete sections further on the appendices.
Appx. B Implementing Vocal Sandbox
We provide complete implementation details for both Vocal Sandbox systems we present in the main body the paper; we additionally include details around the rest of the system architecture (e.g., speech-to-texx robot control, etc.):
$B.1 - System Architecture
Additional system details around implementing real-time speech recognition and text-to- speech, with latency and pricing statistics.
$B.2 - Language Models for Task Planning and Skill Induction
GPT-3.5 Turbo prompting details, generation hyperparameters, and latency statistics.
BB.3 Visual Conioiii-Codiiioneeed Policy Implementation
Additional implementation details and static evaluations for our visual keypoint- conditioned policy (for robust object manipulation).
$B.4 SB.4-Learite Learning Discrete Dynamic Movement Primitives from Demonstration
Formalism of discrete dynamic movement primitives (DMP) and learning algorithm: additional details about the DMP features we employ (re-timing and goal editing).
$B.5 Physical Robot Platform & Controller Parameters
Robot platform and controller implementation; ensuring compliant control and safety.
Appx. Extended Discussion & Future Work
We provide an extended discussion on themes from the paper, from the importance of modularity and transparency in developing Vocal Sandbox, the broader impact of Vocal Sandbox in the context of human-robot collaboration, and directions for future work.
Appx. D - Additional Experiment Visualizations
We provide additional visualizations of experiment settings and results, including stills from the LEGO stop motion interaction and final film product.
15

---

A Motivating Questions
Q1. If wanted to implement Vocal Sandbox from scratch, what components would need? How do the current experiments handle real-time speech-to-text and text-to-speech? What about pricing what was the cost of running the Gift-Bag Assembly User Study (N 8)?
Beyond the language model task planner that uses GPT-3.5 Turbo with function calling [v1 27, 28], and the (lightweight) learned skill policy, we use a combination of Whisper [54] for real-time speech recognition (mapping user utterances to text), and the OpenAl text-to-speech (TTS) API [55] for vocalizing confirmation prompts and querying users for teaching feedback. All models, cameras, and API calls
through single laptop equipped with NVIDIA RTX 4080 GPU (12 GB).
For the gift-bag assembly user study (NV 8), the total cost of all external APIs (Whisper, OpenAI TTS, GPT-3.5 Turbo) amounted to S0.47 + $0.08 + $1.24 = $1.79. For the entirety of the project, GPT-3.5 API spend was $5.79, with ~$4.0 spent on Whisper and TTS (< $10.00 total).
Q2. Given the use of powerfill closed-source foundation models such GPT-3.5/GPT-4, why adopt modular approach for implementing the visual keypoints (and similarly dynamic movement primitives for learning policies)? Why not adopt an end-to-end approach building on top of GPT-4 with Vision, or existing pretrained multitask policies?
We choose to adopt modular approach in this work for two reasons. First, existing end-to-end models are still limited when it comes to fine-grainee perception and grounding; quantify this more explicitly through tead-to-head static evaluations of our keypoint model vs. pretrained models such as OWLv2 [34, 35] in $B.3. Second, we argue that modularity allows users to systematically isolate failures and address them via multimodal feedback, at the right level of abstraction. We expand on this further in SC.I.
Q3. The baseline methods in the user study ($4. 1) are framed as ablations, rather than instances of existing systems that combine language model planning with learning from multiple modalities (e.g., on-the-fly language corrections, gestures). How were these ablations chosen? What is their explicit relationship to prior work?
In user studies, we constructed baselines in a way that best represented the contributions of prior work while still fulfilling the necessary prerequisites perform in situated human-roba collaboration setting. Though we labeled these "ablations" in the paper, each one is representative of prior work connections we make explicit in $C.2.
Q4. How does Vocal Sandbox fit into the context of prior human-robot interaction works? What are the capabilities Vocal Sandbox is bringing to the table?
While the main body of the paper situates our framework against prior work in task planning and skill learning from different modalities, Vocal Sandbox builds on a rich history of work that develops systems for different modes of human-robot interaction. We provide an extended treatment of related work, as well directions for future work in SC.3.
16

---

B Implementing Vocal Sandbox
Implementing a system in the Vocal Sandbox framework requires not only the learned components for language-basse task planning and low-level skill execution, but broader support for interfacing with users via automated speech-to-text, text-to-speech for vocalizing failures confirmation prompts, as well as screen for visualizing the graphical user interface. We describe these additional components, as well as provide more detail around the implementation of the learned components of the systems instantiated in our paper over the following sections.
B.1 System Architecture
For robust and cheap automated speech recognition (mapping user utterances to text), we use Whisper [27, 54], accessed via the OpenAI API endpoint. Whisper is a state-of-the-ari model meant for natural speech transcription, and we find that the latency for given transcription request (< 0.5s round-trip) is more than enough for all of our use-cases. API pricing is also affordable, with the Whisper API (through OpenAl) charging $0.006 / minute of transcription (less than $0.50 to run our entire gift-bag assembly user study). Note that we implement speech-to-text via explicit "push-to-talk" interface, rather than an alternative halways-listeningg to keep cost and word-error rate down, but improves user experience. By gating the listening and stop-listening features with explicit audio cues, users are more aware of what the system doing, and can more quickly localize any failures stemming from malformed speech transcriptions.
approach; we find that this not only allows us
In addition to automated speech-to-text, we adopt off off-the-shelf solutions for real-time timettmme text-to-speech; this is mostly for implementing confirmation prompts ("does this plan look ok to you?") and for vocalizing the system state, but also includes an adaptive component when probing users to teach new visual concepts or behaviors ("I'm m sorry, I'm not sure what the 'jelly-candy thing' looks like, could you teach me?"). For these queries, we use the OpenAI TTS API [55] with a similarly affordable pricing scheme of $15.00 per 1M characters (or approximately 200K words); to run our gift-bag assembly study, this cost fewer than $0.08. For hardware (for both speech recognition and text-to-speech), we use standard USB
(the Anker PowerConf S3). To visualize the graphical user interface to users, we use an external monitor (27 inches), placed outside of the robot's workspace. We drive the GUL, all API calls (speech recognition, text-to-speech. and language modeling via GPT-3.5 Turbo), ZED 2 camera, and all our learned models including visual keypoint-condine conditioned policy, FastSAM [32], and XMem [33]- from single Alienware M16 laptop with an NVIDIA RTX 4080 GPU with 12 GB of VRAM; this laptop was purchased matching the DROID platform specification [26].
Modifications for Gift-Bag Assembly User Study. For the gift-bag assembly user study (N 8) we implement the "push-to-talle speech recognition interface with physical buttons placed on the table; users are provided two buttons- one for "talking" and one for "cancelling" the prior actions (which serves dual function as secondary, software-based emergency stop when the robot is moving). These buttons are placed on the side of the user's 's non-dominant hand, always within reach.
Modifications for LEGO Stop-Motion Animation. For the LEGO stop-motion animation study, we use the same components as above, with two additions. As the expert user is directing and framing individual camera shots during the course of the collaboration, they add additional laptop (a MacBook, rumning Stop Motion Studio) to the workspace (disconnected from the rest of the system). As the user requires both hands free for this study (for articulating LEGO minifigures and structures, or editing the clip on their laptop), we replace the tabletop "push-to-talk buttons with USB-connected foot pedal with two switches with the same recognition and cancel functionality.
B.2 Language Models for Task Planning and Skill Induction
As described in the main body of the paper, we use GPT-3.5 Turbo with function calling [v11-06; 27, 28] as our base language model for the task planner. This the latest, most affordable, and highest latency language model at the time began this work (prior to the release of GPT-4 and
17

---

Comparing Visual Keypoint Models
Mujoco GUI for DMP Visualization
Vyslos boy car"
The doh"
Ensembici
GPT-WV
Figure 6: Visual Keypoint Static Evaluation & DMP Visualizer. We visualize visual keypoint predictions for object locations across clean environment [Left] and more difficult, cluttered environment [Middle). We also highlight the trace generated by our Dynamic Movement Primitive (DMP) skills, rendered a Mujoco [56] simulation environment to display the robot path before execution.
GPT-4o), with a response time between 1-3s on average. and a cost of $2.00 / 1M tokens; for this work, the total cost spent on GPT 3.5 API calls (including development) was $5.79, with the gift-bag assembly user study itself only amounting to $1.24 of the total spend.
We use the GPT-3.5 function calling capabilities throughout our work, which requires formatting our API specification following a custom JSON schema set by OpenAl; we provide these function calling prompts (and all GPT-3.5 prompts for our work) on our supplementary website for easy visualization: https: /vocall procal-sandien sandbbox. git..iio ioli/l//l1 rllangage-prompee outputs were generated with low-temperature sampling (0.2).
All language model
B.3 Visual Keypoint-Conditioned Policy Implementation
As described in the main body of the paper, use three components implement Vocal Sandbox's object manipulation skills: (1) learned sanguage-condttoons keypoints model, (2) a pretrained mask propagatation model [XMem; 33], and (3) point-conditioned segmentation model [FastSAM; 32).
Our learned keypoint model predicts object centroids from language, enabling us to generalize across object instances where XMem struggles. Given an RGB image o E HR x3 and natural language literal Ger from the high-level language planner, it predicts matrix of per-pixel scores H. E [0, 1| We take the coordinate-ve wissi argmax of 7 as the predicted keypoint. We implement model with a two-stream architecture following Shridhar et al. [57] that fuses pretrained CLIP [58] textual embeddings with fully-convolutional architecture. We train this model on small, cheap-to-collect dataset of 25 unique images each annotated with 3 keypoints (75 examples total). To fit our model, we create heatmaps from each ground- -ruth label, centering a 2D Gaussian around each keypoint with fixed standard deviation of pixels; we train our model by minimizing the binary cross-entropy between model predictions and these heatmaps, augmenting images with various label-preserving affine transformations (e.g., random crops, shears, rotations).
Our mask propagation model, XMem [33] tracks object segmentation masks from one image frame the next; we provide brief overview here. XMem is comprised of three convolutional networks query encoder decoder d, and a value encoder v) and three memory modules short-term sensory memory, working memory, and a long-term memory). For given image I,, the query encoder outputs query q e(T,) and performs beeentoon-based memory reading from working and long-term memory stores to extract features Fe... where Get is the language utterance (e.g.. "candy"). The decoder then takes as input q. F, and h (the short-term sensory memory) to output a predicted mask Mc. Finally, the value encoder v(I,, MM) outputs new features to be added to the memory history h,, The query encoder e and value encoder u are instantiated with ResNet-50 and R8 [59] respectively. The decoder d concatenates the short-term memory history h,-1 with the extracted features F, upsampling by factor of 2x until reaching stride of 4. While upsampling, the decoder fuses skip connections from the query encoder at every level. The final feature map is
18

---

passed through a 3 x 3 convolution to output single channel logit which is upsampled to the image size. See Cheng and Schwing [33] for additional details.
Finally, our point-conditionel segmentation model, FastSAM [32], is used to extract an object mask from predicted keypoint. It has two components: YOLOv8 [60] segmentation model for all-instance segmentation, and a point prompt-guided selection for identifying the object mask in which the point lies. From given predicted keypoint p. the segmentation model outputs the mask M from s that encompasses We refer to [32] for additional details. This predicted mask M is subsequently added to the XMem memory storage after being passed through the value encoder v.
Robot actions coded as parameterized primitives (i.e., uick_up or goto) that take object locations input and output trajectories.
Static Evaluations Robust Object Grounding. To highlight the need for data-efficient, domain- specific vision system, we evaluate the performance of our vision module implementation (as described above) compared to existing closed-source foundation models such as OWLv2 and GPT-4V. To compare, we consider an (image, annotation) dataset of all visual queries from the N user study, where the annotation is where the user confirmed manually selected correct object location. We report measures for accuracy and precision keypoint mean squared error in pixel distance and success counts for predictions within a toy-car radius (14 pixels) from the annotation. For the Vocal Sandbox predicted mask, the centroid of the mask is used for these point-to-poing calculations. We observe that while the mean squared error across all three methods are comparable, our Vocal Sandbox vision module greatly outperforms the foundation model baselines in the precision metric. This is because all the objects are clustered together on table (Fig. 6) - randomly selecting between these objects yields low MSE predictions, however nearby prediction is not sufficient identify and isolate the correct object for grasping.
Keypoint MSE (px)
Precision
OWLv2 Ensemble (ViT-L/14)
35.3 = 1.01
1.83 + 0.91
GPT-4-Turbo (w/ Vision)
36.39 + 1.73
15.94 + 2.55
Vocal Sandbox (Ours)
30.46 + 3.61
69.41 + 3.12
B.4 Learning Discrete Dynamic Movement Primitives from Demonstration
For our LEGO stop-motion animation setting, we implement our low-level skill policy as a library of discrete Dynamic Movement Primitives [29, 30]. We adopt the traditional discrete DMP formulation from ljspeert et al. [30], defining a second-order point dynamical system in terms of the system state y. goal g. and phase variable x such that:
TØ= = (7, (g y) -0- (xxg);
T = -
where a and y define gain terms, E (0, 1] denotes a temporal scaling factor, and [xr g) is the learned forcing function that drives DMP to follow a specific trajectory to the goal g: f(x, g) is implemented as a learned linear combination of radial basis functions and the phase variable x such that:
f(z E/= Wbr2i E/=1' x(g - -
き = exp(-h,(<z cc,)²)
where ey and hy are the heuristically chosen centers and heights of the basis functions, respectively. We fit the DMP weights 3 = {wi, W2... w.} with locihly- regression (LWR) from the provided kinesthetic demonstration. For all DMPs in this work, we use J = 32, with gain values and basis functions parameters set following prior work [30].
ag = 25, 7, 4
We choose (discrete) DMPs to implement skill learning they permit efficient learning from kinesthetic demonstration, and have two properties that enable rich generalization to 1) new goals (by
19

---

specifying new g) and 2) arbitrary temporal scaling (by rescaling 7). This lets us induce a simple algebra for parameterizing our policy Td,3 (Gret 1, N), indexing each learned DMP with a learned referent Cres, a new goal location 1, and a number of waypoints (used to set 7) - in other words, allowing us to learn a new DMP track (loc: Location) - that we can call with arbitrary new locations (from novel initial states) with arbitrary timing parameters (e.g., "can you track around Loki in 30 frames" or "I need a tracking shot around the tower.. let's try seconds").
Visualizing DMP Rollouts. Another advantage of using DMPs for parameterizing control is that they allow us to visualize entire trajectories prior to execution. Similar to how we visualize the keypoints and object segmentation masks in the collaborative gift-bag assembly setting, we provide GUI that shows the robot and the planned path (and end-effector poses) via simple MuJoCo-basee viewer. plot the original kinesthetic demonstration relative the
Fig. 6 (Right) provides example current robot pose in green (for reference), and the planned DMP trajectory in blue, along with the end-effector orientation frames the beginning and end of the trajectory. Users additionally can dynamically advance the simulation to visualize the entire rollout (at the actual speed of execution).
B.5 Physical Robot Platform & Controller Parameters
We use a Franka Emika Panda 7-DoF robot arm with Robotiq 2F-85 parallel jaw gripper following the platform specification from DROID [26]. The robot and its base are positioned one side of 3' x 5' table, across from the user, such that the user and robot share the tabletop workspace. We overhead ZED 2 RGB-D camera with known intrinsics and extrinsics. For robot control, we use modified version of the DROID control stack based on Polymetis [61]. Low-level policies command joint positions at 10 Hz to joint impedance controller which runs at I kHz. We implement two compliance modes: a stiff" mode which is activated when the robot is executing a low-level skill, and a compliant mode for when the user provides a kinesthetic demonstration.
Safety. We include multiple safeguards ensure safety. Users have the option cancel any proposed behavior when an interpretable trace is presented with physical Cancel button as described in $B.1 this prevents execution and immediately backtracks the Vocal Sandbox system. Second, during execution of any low-level skill, the user can interrupt the robot's motion with this button as well. This halts the robot's 's motion and immediately becomes fully compliant. Lastly, during user studies, both the user and proctor have access to the hardware emergency stop button which cuts the robot's power supply and mechanically locks the robot arm.
20

---

C Extended Discussion & Future Work
The following sections expand on the discussion from the main body of the paper, with a specific focus on the benefits of Vocal Sandbox's s modular design, before providing an extended treatment of our contributions and future directions in the broader context of systems for human-robot collaboration.
C.1 On Modular vs. End-to-End Approaches
We develop Vocal Sandbox as modular framework; the decoupled nature of the high-level language behavior planner from the low-level skill policies is explicit, and characterizes the rest of our contributions. Yet, this choice poses an important question - why not an end-to-end approach?
An initial answer stems from limitations in current models; our user studies show that "flat" planning with language models has several failure modes when it comes to reliability, while our static evalua- tions in $B.3 indicate deficiencies in ability for current multimodal models (e.g., GPT-4 Turbo with Vision) for
language grounding in cluttered scenes. Yet, even if we consider a future where we have stronger end-to-end approaches that unify language, vision, and action [e.g.. building on top of RT-2 or RT-H; 15, 62], we argue that modularity is an important feature in allowing users to isolate system failures and localize their feedback at the right level of abstraction.
Consider common failure mode of end-to-end policies learned from data: visual robustness. Different degrees of distribution shift (e.g., introducing new distractor objects, or even perturbing the scene in small ways) not only hurt success rate [63], but they also affect the closed-loop execution in arbitrary ways [64], leading to suboptimal or unsafe trajectories. Worse is that errors only cascade as the robot or scene go further out of distribution, leading to even more unpredictable behavior.
Conversely, one of the more salient observations from user study was how quickly users were able to not only identify failure modes in our system, but co-adapt to them. For example, within assembling the first two gift hags in the study, many users identified that the learned keypoint model was especially poor at predicting one category of object (Play-Doh). Rather than let this failure completely derail the task, users leaned on the modularity of our system to isolate this failure to the specific module (i.e., the visual keypoints- based policy) and sequence their feedhack - teaching new high-level behaviors while expecting when and where the robot would fail. Specifically, users opted to teach a high-level behavior "assemble bag() that would always attempt to pack the Play-Doh into the gift bag as the final step, affording them the ability to maximally "disengage" from actively supervising the robot for the bulk of execution, only intervening at the last step. In other words, modularity in Vocal Sandbox gives users the leverage to to quickly understand the robot's capabilities, as well as the power to meaningfully build on its strengths, and adapt around its weaknesses.
C.2 Explicit Connections Between System Ablations and Prior Work
Our user studies ($4.1) compares Vocal Sandbox systems two baseline methods that we frame different "ablations": in this section, we explicitly link each of these ablations to prior work that builds systems for human-robor collaboration using language model planners, and that learn skills from different feedback modalities.
To make this link, we first want restate the contributions of Vocal Sandbox: 1) the ability to continually learn new high-level behaviors and low-level skills from mixed-modality teaching feedback, grounded in 2) situated numan-robot collaborative setting where we evaluate the ability of our system to provide more utility (assist the user) over time. The setting is important; the bulk of prior works frame human-robot collaboration as turn-taking between a human providing instructions corrections from afar and a robot performing task [12, 17, 51]. These works evaluate binary success rate given user input and a task, did the robot succeed or fail.
In contrast, our setting assigns the robot and human clear roles within a collaboration (e.g., shoot a stop-motion film, with robot controlling the camera and the human directing). Here, task success is necessary condition if the user could not direct the robot to perform a minimal set of actions, they would not use it (and instead perform the task completely their own, or with a human partner). As
21

---

result, we needed to develop a "baseline" system that ensured the robot remains a viable partner at every point during the collaboration. This is why we minimally define set of primitives that cover the full space of robot capabilities needed, as well as any interfaces (e.g., the GUI) the user might need to teach. In other words, viability and recovery are prerequisites for all systems we evaluate. Given these prerequisites, we looked at our hypothesis specifically that learning at both the high- and low-level through mixed modality feedback leads to more effective human-robot collaboration. We looked at prior works in collaborative task planning, approaches for handling on-the-fly language corrections, and methods for learning from other feedback modalities, identifying the following system "templates" to compare:
1. "Static" Planning with Fixed Low-Level Skills. These systems define human-robor collaboration
as translation from user instructions to fixed set of predefined skills. This system is reflective of the core contributions of recent work such as Text2Motion [17], ProgPrompt [18], and MOSAIC [8] the latter of which we consider to be closest to ours.
Crucially, these works use LMs to map user intent to realizable robot plans specified as programs. We specifically build on MOSAIC, as we believe this system to be the most general - does not assume access to full-state information priori (i.e., the LM is not priori prompted with all objects in the scene), instead invoking learned perception model to localize objects if possible (as we do; $3.2). Combining this with our "prerequisite" system, get VS- (Low, High) system reflective of MOSAIC that uses an LM to translate language input to skills from a fixed library, some of which rely on the output of a learned perception module.
On-the-Fly Corrections: "Adaptive" Planning with Fixed Skills. In works such as Inner Monologue [9], Code as Policies [19] and Yell at Your Robot [YAY; 14], robot systems go beyond static planning. Instead, they exhibit forms of reactivity: while the sets of skills in these works remain static, the planners incorporate user environment feedback (as corrections, action feasibility, subtask success) to synthesize plans. Of these works, YAY adds an iterative component, explicitly adding structured behaviors to retrain high-level policy in batches (similar to how we induce new high-level behaviors during teaching). Adding this reactivity results in VS- Low - a system that combines InnerMonologue with the iterative updates of YAY reactive LM planner that conditions on user corrections or other feedback to iteratively learn new high-level behaviors to guide planning.
3. Iteratively Improving Low-Level Skills. Finally, to explore the impact of adaptively growing our library of low-level skills learned from mixed-modality feedback we looked to prior work that learn skills from keypoint supervision as in KITE [31] and MOKA [65], gestures as in GIRAF [12], or as DMPs from kinesthetic demonstrations [30]. These works do not focus on collaborative multi-turn settings, instead proposing methods that are evaluated offline, given "static" input (e.g., evaluating success rate of a fully autonomous policy that is conditioned on gestures vs. not). As our contributions are focused more the nature of collaboration when we allow users to teach new low-level skills, and not on the actual methods for skill learning, we use these methods fixed components of our system. This results in our final Vocal Sandbox systems (SS)-whrre in our gift-bag assembly setting ($4.1, VS specifies skills via a learned visual keypoints model, and in the stop-motion setting ($4.2), VS specifics skills via DMPs. Note that for the keypoints model, we performed a static evaluation of different (pretrained) models $B.3, finding our simple two-stream CLIP-basee architecture to outperform existing open- vocabulayy detectors like OWLv2 and foundation models like GPT-4V.
Structuring our ablations this way gives fair, apples-to-apple comparison against prior works such as MOSAIC (VS- Low, High), InnerMonologue and YAY (VS 0 Low), while demonstrating our novel contributions for enabling continual learning at both the high- and low-levels.
C.3 Broader Context and Future Work
While the main body of our paper situates our framework against methods that use language models for task planning and learning low-level skills from multimodal feedback, Vocal Sandbox builds off much larger body of work that build systems for different forms of human-robot interaction. The systems differ in the modes of collaboration they enable, from explicit human-robot teaming in
22

---

situated environments [1, 4, 6, 7, 66], to learned methods for shared autonomy [67-69], to platforms for assistive robotics [70-72], amongst many others [2].
While Vocal Sandbox is heavily inspired by this prior work, especially those that learn language interfaces for grounding intent to low-level robot behavior [8, 73, 74], this only the beginning. Future iterations of our framework will build the types of interactions and learning we permit (e.g., multi-robot teaming integrating modalities such as touch nonverbal feedback), all driving towards general and seamless human-robot collaboration.
D Additional Experiment Visualizations
present additional visualizations of the LEGO stop-motion animation experiment, including examples of user-taught camera motions and stills from the two hour interaction during filming.
"OK, now zoom into the neot"
Figure 7: LEGO User-Taught Camera Motions. We present examples of user-taught camera motions "Let's frame tower this shot" and "OK, now zoom into the Hulk next" along with their corresponding robot trajectories. Of the 232 frames shot for the film, 99 were shot with complex, user-taught camera motions such as the ones visualized here.
23

---

Human-Robot Collaboration
LEGO Stop Motion Film
THLLS
Figure Stills from LEGO Stop Motion Interaction. We present stills from the two-hour continuous collaboration for shooting the LEGO stop motion animation [Left], as well as corresponding frames from the final film [Right]. The user directs and arranges LEGO figures while the rohot controls the camera.
24


Reviews:
[{"review_id": "RLFNdd1Kq9", "reviewer": "Reviewer_gPa9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a framework for human-robot interaction that allows a user to continuously provide corrective and instructional feedback across multiple modalities and hierarchies. The proposed framework allows a user to teach both low-level skills, which can be taught via visual keypoints (conditioned policy) or kinesthetic demonstrations (DMP), and high-level behaviors taught via language. This framework also allows for corrective feedback in the event the system's plan does not match user expectations. The framework is evaluated through a user study in which the robot assembles a gift bag where it out-performs ablated methods in both subjective and objective metrics.", "summary_of_recommendation": "I think this paper is interesting and addresses a timely and relevant problem, but I do have some concerns about the methodology itself and am unsure how useful the proposed framework actually is.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Can the user correct/override the proposed skill parameters?\n\n2. How are these parameters specified when providing a language instruction (does the user need to say \"pan around slowly with N=30 frames\" or does the language model infer parameters)? How can a user correct parameters if they are inferred and are incorrect?\n\n3. Can a user provide corrections to behaviors purely via language or do they always have to do so manually through the GUI? I am wondering whether language is a feedback modality or simply used only for instructions (and therefore to define new skills/behaviors).\n\n4. What do complex user-defined behaviors look like? Can a user modify existing behaviors (see example above with gift bag)?\n\n5. Do users prefer one modality over another? When is each modality preferred? Is there value in even providing multiple modalities, as opposed to existing approaches which only use one? Does this depend on the underlying model/task and the type of mistakes that might occur?\n\n6. Can a user modify an existing skill? For example, let's suppose that the user already taught the robot the \"camera pan\" skill. Can the user modify this skill to either add an additional parameter or create some sort of variation of it? Or do they have to create a completely new skill from scratch?\n\n7. Does \"(w/o Low, High)\" in the experiments indicate that indicate that users can teach neither high-level behaviors nor low-level skills? The text states high-level only but that seems odd given the name.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zGVG83v4gB", "reviewer": "Reviewer_nEiK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper Vocal Sandbox (abbreviated as VS), a framework where users can teach both new high-level planning behaviors through spoken dialogue and provide teaching feedback to learn new low-level skills in real-time. The authors evaluate this framework in a user study with 8 non-expert participants and find that the system outperforms baselines both objectively and subjectively. Finally, the authors conclude by showing a human-robot collaboration example where the human and robot create a 30-second stop motion animation, an impressive feat.", "summary_of_recommendation": "Overall, this paper is very well-written, contains an impressive demonstration of a few combined techniques, and is validated via a user study. The contribution is incremental and a step forward for HRI, combining different techniques into a system for human-robot collaboration.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Please address the questions in the weaknesses section of the main review.\n\nFurther Questions: \n- How do you ensure the learned low-level skills generalize well to other objects? \n- Did users within the human-subject study need to reteach skills at any point or delete learned skills for any reason?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjoKoFIq5s", "reviewer": "Reviewer_etUa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Summary  This paper presents Vocal Sandbox, a framework for users to teach high-level behaviors through mixed-modality interactions. The framework enables users to give feedback and instruction through multiple modalities online during an collaboration. The Sandbox interface is over high-level behaviors and low-level skills. A high-level LLM planner maps language utterances to behavior plans, and learned low-level policies grounds these plans to robot actions. The low-level skills are trained through DMPs and keypoint-conditioned learned policies.  Online, during the interaction, the user gives the robot a natural language command (ie. track the hulk). The robot explains its failure, deferring to the user for choosing a next step. The user can either move on, or explicitly teach the robot what the new skill of tracking is. When teaching the new skill, the human defines a new behavior function, which can be composed of requisite skills. If a low-level skill is needed, the user can provide a demonstration. If a perceptual concept is needed, the user can label object segmentation masks. Finally, the system provides a visualization of the resultant robot trajectory.  Vocal Sandbox uses a interface for users to teach new skills, perceptual concepts, and visualize robot traces during teaching.  The sandbox is tested across two tasks, gift bag assembly across multiple non-expert participants, and stop motion animation with an expert user. Users strongly preferred the system over its ablated baselines in ease of user, helpfulness, and performance.", "summary_of_recommendation": "Overall, Vocal Sandbox presents a system for online behavior corrections to robots via multi-modal feedback. This is a very interesting paradigm and learning from online, multi-modal human interventions is an important research direction. The framework presented offers very promising system insights and results on favorable perceptions from novice users. Aspects of the evaluation and system performance can be further analyzed, but overall I recommend a weak accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Co-adaptation is presented in the abstract as one of the main motivations of the visualization traces. What signs of co-adaptation were seen in the user study? How were users adjusting to the visualization traces? Did you notice shift in their teaching strategies over time? \n\nDid users see a monotonic increase in performance as a result of further teaching? Where there instances where this was not the case?\n\nPlease see above section for further questions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Paper summary**

This paper contributes a framework for users to provide mixed-modality feedback (language, demonstrations, and labels) to robots in order to learn/improve both high-level and low-level behaviors. The paper contributes methods for integrating multi-modal feedback and providing transparency to the user about its behavior. 

**Review summary**

Summary of strengths:
+ The paper addresses a key problem for human-robot collaboration: adaptation based on human feedback.
+ The proposed method is evaluated with a user study with non-experts.
+ The proposed method makes uses of multi-modal interactions, and allows a robot to learn both low-level and high-level information about a task.

Summary of weaknesses:
- More detail is needed about how users can (and cannot) interact with the interface at each step. Furthermore, the reviewers have noted several opportunities to highlight the users' experiences with system failures, approving/disapproving the robot's behavior, and preferences between modalities. **[Edit: The rebuttal now includes more details about the interface and user comments.]**
- The evaluation is missing comparisons to non-ablation baselines. **[Edit: The rebuttal clarifies that the ablated versions of the system are representative of SOTA baselines.]**
- The evaluation is missing detail about the user study participants and their familiarity with robots. **[Edit: The rebuttal clarifies the backgrounds of the participants.]**


**Response to rebuttal**

The authors all maintain their positive rating, and one reviewer has increased their rating from weak to strong accept. The reviewers agree that the rebuttal addresses their concerns. However, the clarifications have not yet been integrated into a revision.
Decision: Accept