Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration

Authors: jennifer  grannen
siddharth  karamcheti
suvir  mirchandani
percy  liang
dorsa  sadigh
j  gross
s s gu
y  guo
c  hallacy
j  han
j  harris
y  he
m  heaton
j  heidecke
c  hesse
a  hickey
w  hickey
p  hoeschele
b  houghton
k  hsu
s  hu
x  hu
j  huizinga
s  jain
j  jang
a  jiang
r  jiang
h  jin
d  jin
s  jomoto
b  jonn
h  jun
t  kaf- tan
l  kaiser
a  kamali
i  kanitscheider
n s keskar
t  khan
l  kilpatrick
j w kim
c  kim
y  kim
h  kirchner
j r kiros
m  knight
d  kokotajlo
l  kondraciuk
a  kondrich
a  konstantinidis
k  kosic
g  krueger
v  kuo
m  lampe
i  lan
t  lee
j  leike
j  leung
d  levy
c m li
r  lim
m  lin
s  lin
m  litwin
t  lopez
r  lowe
p  lue
a a makanju
k  malfacini
s  manning
t  markov
y  markovski
b  martin
k  mayer
a  mayne
b  mcgrew
s m mckinney
c  mcleavey
p  mcmillan
j  mcneil
d  medina
a  mehta
j  menick
l  metz
a  mishchenko
p  mishkin
v  monaco
e  morikawa
d p mossing
t  mu
m  murati
o  murk
d  mèly
a  nair
r  nakano
r  nayak
a  neelakantan
r  ngo
h  noh
o  long
c  o'keefe
j w pachocki
a  paino
j  palermo
a  pantuliano
g  parascandolo
j  parish
e  parparita
a  passos
m  pavlov
a  peng
a  perelman
f  de avila
belbute  peres
m  petrov
h p de
oliveira  pinto
m  pokorny
m  pokrass
v h pong
t  powell
a  power
b  power
e  proehl
r  puri
a  radford
j  rae
a  ramesh
c  raymond
f  real
k  rimbach
c  ross
b  rotsted
h  roussez
n  ryder
t  saltarelli
s  sanders
g  santurkar
h  sastry
d  schmidt
j  schnurr
d  schulman
k  selsam
t  sheppard
j  sherbakov
s  shieh
p  shoker
s  shyam
e  sidor
m  sigler
j  simens
k  sitkin
i  slama
b d sohl
y  sokolowsky
n  song
f p staudacher
n  such
i  summers
j  sutskever
n a tang
m  tezak
p  thompson
a  tillet
e  tootoonchian
p  tseng
n  tuggle
j  turley
j f c tworek
a  uribe
a  vallone
c  vijayvergiya
c  voss
j j wainwright
a  wang
b  wang
j  wang
j  ward
c  wei
a  weinmann
p  welihinda
j  welinder
l  weng
m  weng
d  wiethoff
c  willner
s  winter
h  wolrich
l  wong
s  workman
j  wu
m  wu
k  wu
t  xiao
s  xu
k  yoo
q  yu
w  yuan
r  zaremba
c  zellers
m  zhang
s  zhang
t  zhao
  zheng
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:35:46

Abstract: We introduce Vocal Sandbox, a framework for enabling seamless human-robot collaboration in situated environments. Systems in our framework are characterized by their ability to *adapt and continually learn* at multiple levels of abstraction from diverse teaching modalities such as spoken dialogue, object keypoints, and kinesthetic demonstrations. To enable such adaptation, we design lightweight and interpretable learning algorithms that allow users to build an understanding and co-adapt to a robot's capabilities in real-time, as they teach new behaviors. For example, after demonstrating a new low-level skill for "tracking around" an object, users are provided with trajectory visualizations of the robot's intended motion when asked to track a new object. Similarly, users teach high-level planning behaviors through spoken dialogue, using pretrained language models to synthesize behaviors such as "packing an object away" as compositions of low-level skills -- concepts that can be reused and built upon. We evaluate Vocal Sandbox in two settings: collaborative gift bag assembly and LEGO stop-motion animation. In the first setting, we run systematic ablations and user studies with 8 non-expert participants, highlighting the impact of multi-level teaching. Across 23 hours of total robot interaction time, users teach 17 new high-level behaviors with an average of 16 novel low-level skills, requiring 22.1% less active supervision compared to baselines. Qualitatively, users strongly prefer Vocal Sandbox systems due to their ease of use (+31.2%), helpfulness (+13.0%), and overall performance (+18.2%). Finally, we pair an experienced system-user with a robot to film a stop-motion animation; over two hours of continuous collaboration, the user teaches progressively more complex motion skills to produce a 52 second (232 frame) movie. Videos & Supplementary Material: https://vocal-sandbox.github.io

Main paper: 
 introduction: 
 Effective human-robot collaboration requires systems that can seamlessly work with and learn from people [1] [2] [3] . This is especially important for situated interactions [4] [5] [6] [7] where robots and people share the same space, working together to achieve complex goals. Such settings require robots that continually adapt from diverse feedback, learning and grounding new concepts online. Yet, recent work [8] [9] [10] [11] [12] [13] [14] [15] remain limited in the types of teaching and generalization they permit. For example, many systems use language models to map user utterances to sequences of skills from a static, predefined library [16] [17] [18] . While these systems may generalize at the plan level, they trivially fail when asked to execute new low-level skills -regardless of how complex that skill might be (e.g., "hold this still"). Instead, we argue that situated human-robot collaboration requires learning and adaptation at multiple levels of abstraction, empowering collaborators to continuously teach new high-level planning behaviors and low-level skills over the course of an interaction.Building on this insight, we present Vocal Sandbox, a framework for situated human-robot collaboration that enables users to teach via diverse modalities such as spoken dialogue, object keypoints, and kinesthetic demonstrations. Systems in our framework consist of a language model (LM) planner [19] that maps user utterances to sequences of high-level behaviors, and a family of low-level skills that ground each behavior to robot actions. Crucially, we design lightweight and interpretable learning algorithms to incorporate a user's teaching feedback, dynamically growing the capabilities of the system in real-time. Consider a user and robot collaborating to film a LEGO stop-motion animation (Fig. 1 ). 1 The user is the animator, articulating LEGO figures and structures to rig each keyframe (Fig. 1 ; Bottom), while the robot carries out precise camera motions (e.g., tracking smoothly to zoom into a character). Early in the interaction, the user asks: "Let's get a tracking shot around the Hulk?" In response, the LM attempts to generate a plan, but fails -"tracking" is not a known concept. The robot vocalizes its failure, deferring to the user for next steps. Empowered by our framework, the user chooses to explicitly teach the robot how to "track" by providing a kinesthetic demonstration. Using this single example as supervision (Fig. 1 ; Left), we immediately parameterize a new skill (i.e., β track ), and synthesize the corresponding behavior (i.e., track(loc: Location)). Later (Fig. 1 ; Right), when the user directs to robot to "push in on Loki and then track around the tower," the robot can immediately use what it has learned to generate both the full task plan (zoom_in(Loki); track(Tower)), as well as a visualization of the motion trajectory (shown via a custom interface; §3.3). On user confirmation, the robot executes in the real-world, capturing the desired shot.We evaluate Vocal Sandbox through systematic ablations and long-horizon user studies. In our first setting, non-expert participants work with a Franka Emika fixed-arm manipulator for the task of collaborative gift bag assembly; we run a within-subjects user study with N = 8 users spanning a total of 23 hours of robot interaction time, comparing a Vocal Sandbox system with two baselines: a static variant of our system without learning at either the high-and low-level [8] , as well as a variant of our system with only high-level teaching. We show that across all three methods, our contributions enable users to teach 17 new high-level behaviors and an average of 16 new low-level skills, resulting in 22.1% faster collaborative task performance compared to baselines; qualitatively, participants strongly prefer our system over all baselines due to its ease of use (+29.4%), helpfulness (+10.8%) and overall performance (+13.9%). We then scale to a more advanced setting where an experienced system-user (an author of this work) works with a robot to shoot a stop-motion animation (Fig. 1 ); over two hours of continuous collaboration the user teaches progressively more complex concepts, building a rich library of skills and behaviors to produce a 52 second (232 frame) movie. 
vocal sandbox: a framework for continual learning and adaptation: 
 Vocal Sandbox is characterized by a language model planner that maps user utterances to sequences of high-level behaviors and a family of skill policies that ground behaviors to robot actions (Fig. 2 ).In this section, we first formalize planning ( §2.1) and skill execution ( §2.2), then introduce our contributions for teaching new behaviors from diverse modes of feedback ( §2.3). . Given a successful parse, we visualize an interpretable trace of both the plan and robot's intended behavior on a custom GUI ( §3.3). In the case of failure, we elicit teaching feedback from users to synthesize new functions and arguments ( §2.3).We implement high-level planning using a language model (LM) prompted with an API specification Λ t = (F, C) (Fig. 2 ; Left) that consists of a set of functions F (synonymous with "behaviors") and argument literals C, indexed by interaction timestep t -this indexing is important as users can teach new behaviors (add to the API) over the course of a collaboration. Each argument c ∈ C is a typed literal -for example, we define Enums such as Location that associate a canonical name (e.g., Location.HOME in Fig. 2 ) to values that are used by the robot when executing a skill. We define each function f ∈ F as a tuple (n, σ, d, b) consisting of a semantically meaningful name n (e.g., goto), a type signature σ (e.g., [obj: ObjectRef] → None), human-readable descriptive docstring d (e.g., "Move above the specified obj"), and function body b. Crucially, we assume we can define higher-order functions -for example pickup(obj: ObjectRef) as goto(obj); grasp().Given a user utterance at time t, the language model generates a plan p t that is comprised of a program, or sequence of function invocations. For example, a valid plan for an instruction such as "place the candy in the gift bag" subject to the API in Fig. 2 would be p t = pickup(ObjectRef.CANDY); goto(ObjectRef.GIFT_BAG); release(). We formalize planning as generatingp t = LM(• | u t , Λ t , h t ), where u t is the user's utterance, Λ t is the current API, and h t corresponds to the full interaction history through t. Concretely, h t = (u prompt , [(u 1 , p 1 ), (u 2 , p 2 ), . . . (u t-1 , p t-1 )]); u prompt is the system prompt that describes the setting and constraints (e.g., "return plans as well-formed Python"; see our project page for full prompts).Note that there are cases where the language model may not be able to generate a valid plan -for example, given an utterance describing a behavior that does not exist in the API (e.g., "can you pack a gift bag with three candies"). In such a situation, we rely on the commonsense understanding of the LM to generate a helpful error message that indicates the failure mode (e.g., "I am not sure how to pack; could you teach me?"). These error messages (as well as the successfully generated plans) are shown to users via a custom graphical interface (GUI; §3.3) to inform their next action.To ground plans p t to robot actions, we assemble skill policies π f (a | o t , c 1 , c 2 , . . . ) that define the execution of a function f ∈ F as a mapping from a state observation o t ∈ R n and individual "resolved" arguments c 1 , c 2 , . . . to a sequence of robot actions a ∈ R T ×D (e.g., end-effector poses). Of particular note is the resolution operator • that maps an LM-generated plan p t to a sequence of skill policy invocations; to do this, we first iterate through each function invocation in p t and recursively expand higher-order functions as necessary. In our "place the candy in the gift bag" example, this means expanding the initial pickup(ObjectRef.Candy) to its resolved function body b pickup goto(ObjectRef.Candy) and release(). We then resolve each argument, substituting each literal (variable name) with its corresponding value -for example ObjectRef.Candy = "A gummy, sandwich-shaped candy" while Location.HOME = ([0.36, 0.00, 0.49] pos , [1, 0, 0, 0] rot ).The corresponding values are passed to the underlying policy implementation π f .In this work, we look at three different classes of skill policies: hand-coded primitives (e.g., GO_HOME or GRASP), visual keypoint-conditioned policies that identify end-effector poses given natural language object descriptions, and dynamic movement primitives ( §3.2). Note that in general, our formulation permits arbitrary policy classes, including learned closed-loop control policies [20] [21] [22] .A core component of Vocal Sandbox is the ability to synthesize new behaviors and skills from diverse modalities of teaching feedback such as spoken dialogue, object keypoints, and kinesthetic demonstrations. To do this, we leverage the commonsense priors and strong generalization ability of our underlying language model to synthesize new functions and arguments, updating the API Λ t in real-time. Given an failed plan, each "teaching" step first uses the language model to vocalize the "missing" concept(s) to be taught, followed by a interaction that prompts the user to provide targeted feedback. The language model then synthesizes new functions and arguments for the API. Fig. 2 (Right) shows the two types of teaching we develop: 1) argument teaching and 2) function teaching.The goal of argument teaching is to teach and ground new literals c ˆ∈ C -for example, identifying "green toy car" as a new argument given the utterance "Grab the green toy car" in Fig. 2 (Right). To do this, the LM parses as much of the utterance as possible subject to the current API, mapping "grab" to pickup. Because it is able to identify the correct function (but not the arguments), the LM then uses the corresponding type signature to infer that "green toy car" should be of type ObjectRef; it then automatically synthesizes an API update, adding a new literal ObjectRef.TOY_CAR. This addition is then shown to the user (as part of the second stage of teaching), who then then provides the supervision needed to successfully ground the literal (i.e., providing the keypoint supervision to localize the object). On successful execution, the LM commits this change, yielding Λ t+1 .The goal of function teaching is to teach new functions f ˆ∈ F -for example, defining pack from the utterance "now can you pack the candy in the bag" in Fig. 2 (Right). Here, the LM cannot partially parse the utterance -"pack" does not have an associated function, so there is no reliable way to infer a type signature. Instead, the LM highlights "pack" as a new behavior, and explicitly asks the user to teach its meaning through decomposition [23, 24] , breaking "pack" down into a chain of existing skills. In this case, the user says: "Pick up the candy; go above the bag; drop it" with program pickup(ObjectRef.CANDY); goto(GIFT_BAG); release(). The LM then explicitly synthesizes the new function f ˆ= (n ˆ, σ ˆ, d ˆ), with name n ˆ= pack, signature σ ˆ= obj: ObjectRef → None, and docstring d ˆ= "Retrieve the object and place it in the gift bag." We also generate the "lifted" body pickup(obj); goto(GIFT_BAG); release() via first-order abstraction [24, 25] . This combination of argument and function teaching enables the expressivity and real-time adaptivity of our framework; in defining lightweight algorithms for learning and synthesizing new API specifications from interaction, we provide users with a reliable method of growing and reasoning over the robot's capabilities during the course of a collaboration. 
implementation & design decisions: 
 While we introduce Vocal Sandbox as a general learning framework, this section provides implementation details specific to the experimental settings we explore in our experiments. Notably, our first experimental setting involves a collaborative gift-bag assembly task ( §4.1) where a non-expert user and robot work together to assemble a gift bag with a set of known objects (visualized in Fig. 2 ). Our second setting ( §4.2) pairs an experienced system user (an author of this work) with a robot for the task of LEGO stop-motion animation (visualized in Fig. 1 ). For all settings, we use a Franka Emika Panda arm equipped with a Robotiq 2F-85 gripper following DROID [26] ; we also assume access to an overhead ZED 2 RGB-D camera with known intrinsics and extrinsics to obtain point clouds. We use GPT-3.5 Turbo with function calling [v11-06; 27, 28] due to its high latency and cost-effective pricing. We encode Λ t as a Python code block (formatted as Markdown) in the system prompt u prompt .To constrain the LM outputs to well-formed programs, we use the function calling feature provided by the OpenAI chat completion endpoint [28] , formatting each function in Λ t as a "tool" that the LM can invoke in response to a user endpoint. Full code and prompts are on our project page.We implement different families of skill policies π f for each of our two experimental settings. In our first setting ( §4.1) we implement skills using a visual keypoints model, while for our second setting ( §4.2) we implement skills as dynamic movement primitives [DMPs; 29, 30] .Visual Keypoint-Conditioned Policies for Object Manipulation. For the collaborative gift-bag assembly setting ( §4.1), we implement skills π goto and π pickup via learned keypoint-conditioned models that ground object referring expressions (e.g., "a green toy car") to point clouds in a scene [31] . Specifically, given an (image, language) input, we first predict a 2D keypoint (x, y) that corresponds to the centroid of the desired object in pixel space, then use off-the-shelf models [i.e., FastSAM; 32] to produce an object segmentation mask. Finally, we deproject this mask through our calibrated camera to obtain the object's point cloud to inform manipulation. To ensure consistency, we use a learned mask propagation model [XMem; 33] to maintain a dictionary mapping language expressions to existing object masks -this allows us to ensure that referring expressions such as "the green toy car" always refer to the same object instance for the duration of the interaction (as opposed to predictions changing over time, confusing users). We provide further detail in §B.3.Note that we adopt this modular approach (predict keypoints from language, then extract a segmentation mask) for two reasons. First, we found learning a visual keypoints model to be extremely data efficient (requiring only a couple dozen examples) and reliable, significantly outperforming existing open-vocabulary detection and end-to-end models such as OWL-v2 [34, 35] as well as visionlanguage foundation models such as GPT-4V [36] ; we quantify these results via offline evaluation in §B.3. Second, we found keypoints to be the right interface for interpretability and teaching: to specify new object, users need only "click" on the relevant part of the image via our GUI ( §3.3).Learning Dynamic Movement Primitives from Demonstration. For our stop-motion animation setting ( §4.2), the robot is tasked with controlling the camera to execute different cinematic motions such as panning, tracking, or zooming (amongst others); due to the dynamic nature of these motions, we implement skills as (discrete) DMPs [29, 30] , where users teach new motions by providing a single kinesthic demonstration; using DMPs allows us to generalize motions to novel start and goal positions, while also providing nice affordances for re-timing trajectories (e.g., speeding up a motion by a factor of 2x, or constraining that a motion executes in K steps). Furthermore, as rolling out a DMP generates an entire trajectory, we can provide users with a visualization of the robot's intended path via our custom GUI ( §3.3; visualized in Fig. 6 ). To learn a DMP, we divide a user's demonstration into a fixed sequence of waypoints, and learn a set of radial basis function weights (e.g., β track from Fig. 1 ) that smoothly interpolate between them; we provide further detail in §B.4.Finally, we design a graphical user interface (GUI; Fig. 5 ) to provide users with transparency into the system state and to enable targeted teaching. Consider an utterance "pack the candy" and a failure mode that has the robot packing a different object (e.g., the ball) instead. Without any additional information, it is impossible for the user to identify whether this failure is a result of the LM planner (e.g., generating the incorrect plan pack(ball)), or a failure in the skill policy (e.g., incorrectly predicting a keypoint on the ball instead of the candy). The GUI counteracts this confusion by explicitly visualizing the plan and the interpretable traces produced by each skill (e.g., the predicted keypoints and segmentation mask, or the robot's intended path as output by a DMP)these interpretable traces also double as interfaces for eliciting teaching feedback (e.g., "clicks" to designate keypoints). The GUI also provides 1) the transcribed user utterance, 2) the current "mode" (e.g., normal execution, teaching, etc.), and 3) the prior interaction history. 
experiments: 
 We evaluate the usability, adaptability, and helpfulness of Vocal Sandbox systems through two experimental settings: 1) a real-world human-subjects study with N = 8 inexperienced participants for the task of collaborative gift bag assembly ( §4.1), and 2) a more complex setting that pairs an experienced system-user (author of this work) and robot to film a stop-motion animation ( §4.2). All studies were IRB-approved, with all participants providing informed consent.This study has participant work with the robot to assemble four gift bags with a fixed set of objects (candy, Play-Doh, and a toy car), along with a hand-written card (transcribed from a 96-word script). This task is repetitive and time-intensive, serving to study how well users can teach the robot maximally helpful behaviors to parallelize work and minimize their time spent supervising the robot.Participants and Procedure. We conduct a within-subjects study with 8 non-expert user (3 female/5 male, ages 25.2 ± 1.22). Each user assembled four bags with three different systems, with a random ordering of methods across users. Prior to engaging with the robot, we gave users a sheet describing the robot's capabilities (i.e., the base API functionality), and instructions for using any teaching interfaces (if applicable). Prior to starting the bag assembly task, users were allowed to practice using each method for the unrelated task of throwing (disjoint) objects from a table into a trash can.Independent Variables -Robot Control Method. We compare the full Vocal Sandbox system (VS) to two baselines. First, VS -(Low, High) (without Low, High), a static version of Vocal Sandbox that ablates both low-level skill teaching and high-level plan teaching, reflecting prior work like MOSAIC [8] that assume a fixed skill library and LM planner seeded with a base API. Our second baseline VS -Low ablates low-level skill teaching, but allows for teaching new high-level behaviors. Dependent Measures. We consider objective metrics such as time supervising the robot, number of commands spoken, number of low-level skills executed per command (behavior complexity), as well as the number of teaching interactions at both the high-and low-level. Intuitively, we expect that methods capable of learning high-level behaviors from user feedback require less active supervision time and yield more complex behaviors than those using a static (naive) planner; similarly, we expect methods that permit teaching new low-level skills see fewer low-level skill execution errors. In addition to objective metrics, we asked users to fill out a qualitative survey after engaging with each method, describing their experience and ranking each method.Results -Objective Metrics. We report objective results in Fig. 4 . We measure robot supervision time (Fig. 4 ; Left) as a proxy for robot capability -a more capable robot should require less supervision (as it performs more actions autonomously). We observe that Vocal Sandbox (VS) systems outperform both the VS -(Low, High) and VS -Low baselines in terms of supervision time, demonstrating the capabilities afforded by the ability to teach at multiple levels of abstraction; the strengths of VS systems are made further evident in visualizing how users spend their time during a collaboration across instructing, confirming, and parallelizing (Fig. 4 ; Bottom-Left).We additionally measure the complexity of (taught) high-level behaviors, measures as the number of low-level skills executed per language utterance (Fig. 4 ; Middle). We find that VS systems yield increasingly more complex behaviors compared to both baselines, with significantly (p < 0.05) more complex behaviors taught (and used) compared to the VS -(Low, High) baseline. This further highlights the importance of high-level teaching via structured function synthesis ( §2.3). Of equal importance is the ability of the robot to learn low-level skills, exhibiting a declining rate of skill execution failures over time (Fig. 4 ; Right). We see that VS systems show significantly (p < 0.05) fewer skill failures than both baselines, stressing the isolated importance of teaching at the low-level. We further note that VS systems show fewer skill failures as a function of numbers of bags assembled, demonstrating the ability of VS systems to improve over time.Results -Subjective Metrics. We report subjective survey results in Fig. 5 (Left) , where users ranked the three methods across six different measures. In terms of ease, helpfulness, intuitiveness, and willingness to use again, we find that users prefer our VS system over both baselines due to its transparency and teachability, as users noted "teaching is useful" and "I loved how I was able to teach the robot certain skills". We also note that users significantly (p < 0.05) prefer VS to the naive VS -(Low, High) baseline, as it "felt chunkier to use". For predictability and trust, we observe that users prefer our VS system over baselines, however this trend is less pronounced because any autonomous execution incurs some loss of predictability (due to imperfect robot execution with learned policies).Finally, to push the limits of our framework, we consider a LEGO stop motion animation setting, where an experienced system-user (author of this work) works with the robot over two hours of continuous collaboration to shoot a stop-motion film. The user is the director, leading the creative vision for the film by directing footage and arranging LEGO set pieces, while the robot controls the camera to capture different types of dynamic shots. Specifically, the user teaches the robot to perform cinematic concepts including "tracking", "zooming" and "panning" via kinesthetic demonstrations, which we use to fit different DMPs (as described in §3.2). The user iteratively builds on these behaviors to shoot progressively more complex frame sequences, ultimately resulting in a 52-second stop-motion film, consisting of 232 individual frames. Of the total frame count, 43% were shot with completely autonomous dynamic camera motions taught by the user -that is, camera motinos where the LEGO scene remained fixed, but the camera moved subject to a DMP rollout (e.g., a "zoom-in" shot). We found the taught skills were able to generalize across different start and end positions, as well as different timing constraints -for example "pan around slowly" commanding a pan_around motion with more frames (N = 30; 8 seconds), while "pan around quickly" modulated the same skill to execute in fewer frames (N = 8; 1.33 seconds). Over the two-hour long interaction, the robot executed on 40 novel commands -commands such as "let's frame the tower in this shot" to "zoom into Iron Man" (see Appx. D and our project page for more examples). 
related work: 
 Vocal Sandbox engages with a large body of work proposing systems for human-robot collaboration that pair task planning with with learned skill policies for executing actions; we provide an extended treatment of related work in §C.3, focusing here on prior work that center language models for task planning, or introduce generalizable approaches for learning skills from different feedback modalities.Task Planning with Language Models. Recent work investigates methods for using LMs such as GPT-4 and PaLM [37] [38] [39] for general-purpose task planning [9, [16] [17] [18] . Especially relevant are approaches that use LMs to generate plans as programs [19, [40] [41] [42] [43] . While some methods explore using LMs to generate new skills -e.g., by parameterizing reward functions [44, 45] -they require expensive simulation and offline learning. In contrast, Vocal Sandbox designs lightweight learning algorithms to learn new behaviors online, from natural user interactions.Learning Generalizable Skills from Mixed-Modality Feedback. A litany of prior approaches in robotics study methods for interpreting diverse feedback modalities, from intent inference in HRI [8, 46, 47] to learning from implicit expressions [11, 48] or explicit gestures [12, 49, 50] . With the advent of LMs, language has become an increasingly popular interaction modality; however, most methods are limited to specific types of language feedback such as goal specifications [13, 51] or corrections [14, 52, 53] . In contrast, Vocal Sandbox demonstrates that language alone is not sufficient when teaching new behaviors -especially for teaching new object groundings or dynamic motions. Instead, our framework leverages multiple feedback modalities simultaneously to guide learning. 
discussion: 
 We present Vocal Sandbox, a framework for situated human-robot collaboration that continually learns from mixed-modality interaction feedback in real-time. Vocal Sandbox has two components: a high-level language model planner, and a family of skill policies that ground plans to actions. This decomposition allows users to give targeted feedback at the correct level of abstraction, in the relevant modality. We evaluate a Vocal Sandbox system in a user study across N = 8 participants, and observe that our Vocal Sandbox system is preferred by users (+18.2%), requires minimal supervision (-22.1%) and yields more complex autonomous performance (+19.7%) with fewer failures (-67.1%) compared to non-adaptive baselines.Limitations and Future Work. As execution relies on low-level skills that are quickly adapted from sparse feedback, this framework struggles in dexterous settings (e.g., assistive bathing) where more data is necessary to capture behavior nuances. Another shortcoming is that the collaborations enabled by our system are relatively homogeneous -users are teachers and robots are followers -which is not suited for all settings. Future work will explore algorithms for cross-user improvement as well as sample-efficient algorithms to learn more expressive skills. We will also further probe the user's model of robot capabilities to investigate questions about human-robot trust and collaboration styles. Beyond the language model task planner that uses GPT-3.5 Turbo with function calling [v11-06; 27, 28], and the (lightweight) learned skill policy, we use a combination of Whisper [54] for real-time speech recognition (mapping user utterances to text), and the OpenAI text-to-speech (TTS) API [55] for vocalizing confirmation prompts and querying users for teaching feedback. All models, cameras, and API calls are run through a single laptop equipped with an NVIDIA RTX 4080 GPU (12 GB).For the gift-bag assembly user study (N = 8), the total cost of all external APIs (Whisper, OpenAI TTS, GPT-3.5 Turbo) amounted to $0.47 + $0.08 + $1.24 = $1.79. For the entirety of the project, GPT-3.5 API spend was $5.79, with ∼$4.00 spent on Whisper and TTS (< $10.00 total). Q2. Given the use of powerful closed-source foundation models such as GPT-3.5/GPT-4, why adopt a modular approach for implementing the visual keypoints (and similarly dynamic movement primitives for learning policies)? Why not adopt an end-to-end approach building on top of GPT-4 with Vision, or existing pretrained multitask policies?We choose to adopt a modular approach in this work for two reasons. First, existing end-to-end models are still limited when it comes to fine-grained perception and grounding; we quantify this more explicitly through head-to-head static evaluations of our keypoint model vs. pretrained models such as OWLv2 [34, 35] in §B.3. Second, we argue that modularity allows users to systematically isolate failures and address them via multimodal feedback, at the right level of abstraction. We expand on this further in §C.1.The baseline methods in the user study ( §4.1) are framed as ablations, rather than instances of existing systems that combine language model planning with learning from multiple modalities (e.g., on-the-fly language corrections, gestures). How were these ablations chosen? What is their explicit relationship to prior work?In our user studies, we constructed our baselines in a way that best represented the contributions of prior work while still fulfilling the necessary prerequisites to perform in our situated human-robot collaboration setting. Though we labeled these "ablations" in the paper, each one is representative of prior work -connections we make explicit in §C.2.While the main body of the paper situates our framework against prior work in task planning and skill learning from different modalities, Vocal Sandbox builds on a rich history of work that develops systems for different modes of human-robot interaction. We provide an extended treatment of related work, as well as directions for future work in §C.3.Implementing a system in the Vocal Sandbox framework requires not only the learned components for language-based task planning and low-level skill execution, but broader support for interfacing with users via automated speech-to-text, text-to-speech for vocalizing failures or confirmation prompts, as well as a screen for visualizing the graphical user interface. We describe these additional components, as well as provide more detail around the implementation of the learned components of the systems instantiated in our paper over the following sections.For robust and cheap automated speech recognition (mapping user utterances to text), we use Whisper [27, 54] , accessed via the OpenAI API endpoint. Whisper is a state-of-the-art model meant for natural speech transcription, and we find that the latency for a given transcription request (< 0.5s round-trip) is more than enough for all of our use-cases. API pricing is also affordable, with the Whisper API (through OpenAI) charging $0.006 / minute of transcription (less than $0.50 to run our entire gift-bag assembly user study). Note that we implement speech-to-text via explicit "push-to-talk" interface, rather than an alternative "always-listening" approach; we find that this not only allows us to keep cost and word-error rate down, but improves user experience. By gating the listening and stop-listening features with explicit audio cues, users are more aware of what the system is doing, and can more quickly localize any failures stemming from malformed speech transcriptions.In addition to automated speech-to-text, we adopt off-the-shelf solutions for real-time text-to-speech; this is mostly for implementing confirmation prompts ("does this plan look ok to you?") and for vocalizing the system state, but also includes an adaptive component when probing users to teach new visual concepts or behaviors ("I'm sorry, I'm not sure what the 'jelly-candy thing' looks like, could you teach me?"). For these queries, we use the OpenAI TTS API [55] with a similarly affordable pricing scheme of $15.00 per 1M characters (or approximately 200K words); to run our gift-bag assembly study, this cost fewer than $0.08. For hardware (for both speech recognition and text-to-speech), we use a standard USB speaker-microphone (the Anker PowerConf S3).To visualize the graphical user interface to users, we use an external monitor (27 inches), placed outside of the robot's workspace. We drive the GUI, all API calls (speech recognition, text-to-speech, and language modeling via GPT-3.5 Turbo), ZED 2 camera, and all our learned models -including our visual keypoint-conditioned policy, FastSAM [32] , and XMem [33] -from a single Alienware M16 laptop with an NVIDIA RTX 4080 GPU with 12 GB of VRAM; this laptop was purchased matching the DROID platform specification [26] .Modifications for Gift-Bag Assembly User Study. For the gift-bag assembly user study (N = 8) we implement the "push-to-talk" speech recognition interface with physical buttons placed on the table; users are provided two buttons -one for "talking" and one for "cancelling" the prior actions (which serves a dual function as a secondary, software-based emergency stop when the robot is moving). These buttons are placed on the side of the user's non-dominant hand, always within reach.Modifications for LEGO Stop-Motion Animation. For the LEGO stop-motion animation study, we use the same components as above, with two additions. As the expert user is directing and framing individual camera shots during the course of the collaboration, they add an additional laptop (a MacBook, running Stop Motion Studio) to the workspace (disconnected from the rest of the system). As the user requires both hands free for this study (for articulating LEGO minifigures and structures, or editing the clip on their laptop), we replace the tabletop "push-to-talk" buttons with a USB-connected foot pedal with two switches with the same recognition and cancel functionality.As described in the main body of the paper, we use GPT-3.5 Turbo with function calling [v11-06; 27, 28] as our base language model for the task planner. This was the latest, most affordable, and highest latency language model at the time we began this work (prior to the release of GPT-4 and GPT-4o), with a response time between 1-3s on average, and a cost of $2.00 / 1M tokens; for this work, the total cost we spent on GPT 3.5 API calls (including development) was $5.79, with the gift-bag assembly user study itself only amounting to $1.24 of the total spend.We use the GPT-3.5 function calling capabilities throughout our work, which requires formatting our API specification following a custom JSON schema set by OpenAI; we provide these function calling prompts (and all GPT-3.5 prompts for our work) on our supplementary website for easy visualization: https://vocal-sandbox.github.io/#language-prompts. All language model outputs were generated with low-temperature sampling (0.2).As described in the main body of the paper, we use three components to implement Vocal Sandbox's object manipulation skills: (1) a learned language-conditioned keypoints model, (2) a pretrained mask propagatation model [XMem; 33], and (3) a point-conditioned segmentation model [FastSAM; 32] .Our learned keypoint model predicts object centroids from language, enabling us to generalize across object instances where XMem struggles. Given an RGB image o t ∈ R H×W ×3 and natural language literal c ref from the high-level language planner, it predicts a matrix of per-pixel scores H ∈ [0, 1] H×W . We take the coordinate-wise argmax of H as the predicted keypoint. We implement our model with a two-stream architecture following Shridhar et al. [57] that fuses pretrained CLIP [58] textual embeddings with a fully-convolutional architecture. We train this model on an small, cheap-to-collect dataset of 25 unique images each annotated with 3 keypoints (75 examples total). To fit our model, we create heatmaps from each ground-truth label, centering a 2D Gaussian around each keypoint with a fixed standard deviation of 6 pixels; we train our model by minimizing the binary cross-entropy between model predictions and these heatmaps, augmenting images with various label-preserving affine transformations (e.g., random crops, shears, rotations).Our mask propagation model, XMem [33] tracks object segmentation masks from one image frame to the next; we provide a brief overview here. XMem is comprised of three convolutional networks (a query encoder e, a decoder d, and a value encoder v) and three memory modules (a short-term sensory memory, a working memory, and a long-term memory). For a given image I t , the query encoder outputs a query q = e(I t ) and performs attention-based memory reading from working and long-term memory stores to extract features F cref , where c ref is the language utterance (e.g., "candy"). The decoder d then takes as input q, F , and h t-1 (the short-term sensory memory) to output a predicted mask M t . Finally, the value encoder v(I t , M t ) outputs new features to be added to the memory history h t . The query encoder e and value encoder v are instantiated with ResNet-50 and ResNet-18 [59] respectively. The decoder d concatenates the short-term memory history h t-1 with the extracted features F , upsampling by a factor of 2x until reaching a stride of 4. While upsampling, the decoder fuses skip connections from the query encoder e at every level. The final feature map is passed through a 3 × 3 convolution to output a single channel logit which is upsampled to the image size. See Cheng and Schwing [33] for additional details.Finally, our point-conditioned segmentation model, FastSAM [32] , is used to extract an object mask from a predicted keypoint. It has two components: a YOLOv8 [60] segmentation model s for all-instance segmentation, and a point prompt-guided selection for identifying the object mask in which the point lies. From a given predicted keypoint p, the segmentation model outputs the mask M from s that encompasses p. We refer to [32] for additional details. This predicted mask M is subsequently added to the XMem memory storage after being passed through the value encoder v.Robot actions are coded as parameterized primitives (i.e., pick_up or goto) that take object locations as input and output trajectories.Static Evaluations -Robust Object Grounding. To highlight the need for a data-efficient, domainspecific vision system, we evaluate the performance of our vision module implementation (as described above) compared to existing closed-source foundation models such as OWLv2 and GPT-4V. To compare, we consider an (image, annotation) dataset of all visual queries from the N = 8 user study, where the annotation is where the user confirmed or manually selected a correct object location. We report measures for accuracy and precision -keypoint mean squared error in pixel distance and success counts for predictions within a toy-car radius (14 pixels) from the annotation. For the Vocal Sandbox predicted mask, the centroid of the mask is used for these point-to-point calculations. We observe that while the mean squared error across all three methods are comparable, our Vocal Sandbox vision module greatly outperforms the foundation model baselines in the precision metric. This is because all the objects are clustered together on a table (Fig. 6 ) -randomly selecting between these objects yields low MSE predictions, however a nearby prediction is not sufficient to identify and isolate the correct object for grasping.OWLv2 Ensemble (ViT-L/14) 35 For our LEGO stop-motion animation setting, we implement our low-level skill policy as a library of discrete Dynamic Movement Primitives [29, 30] . We adopt the traditional discrete DMP formulation from Ijspeert et al. [30] , defining a second-order point dynamical system in terms of the system state y, a goal g, and phase variable x such that:τ y ¨= α y (γ y (g -y) -y ̇) + f (x, g); τ x ¨= -α x xwhere α and γ define gain terms, τ ∈ (0, 1] denotes a temporal scaling factor, and f (x, g) is the learned forcing function that drives a DMP to follow a specific trajectory to the goal g; f (x, g) is implemented as a learned linear combination of J radial basis functions and the phase variable x such that:f (x, g) = ∑︁ J j=1 ψ j w j ∑︁ J j=1 ψ jx(g -y 0 );ψ j = exp(-h j (x -c j ) 2 )where c j and h j are the heuristically chosen centers and heights of the basis functions, respectively. We fit the DMP weights β = {w 1 , w 2 . . . w J } with locally-weighted regression (LWR) from the provided kinesthetic demonstration. For all DMPs in this work, we use J = 32, with gain values α y = 25, γ y = 25 4 and basis functions parameters set following prior work [30] .We choose (discrete) DMPs to implement skill learning as they permit efficient learning from a kinesthetic demonstration, and have two properties that enable rich generalization to Visualizing DMP Rollouts. Another advantage of using DMPs for parameterizing control is that they allow us to visualize entire trajectories prior to execution. Similar to how we visualize the keypoints and object segmentation masks in the collaborative gift-bag assembly setting, we provide a GUI that shows the robot and the planned path (and end-effector poses) via a simple MuJoCo-based viewer. Fig. 6 (Right) provides an example -we plot the original kinesthetic demonstration relative to the current robot pose in green (for reference), and the planned DMP trajectory in blue, along with the end-effector orientation frames at the beginning and end of the trajectory. Users additionally can dynamically advance the simulation to visualize the entire rollout (at the actual speed of execution).We use a Franka Emika Panda 7-DoF robot arm with a Robotiq 2F-85 parallel jaw gripper following the platform specification from DROID [26] . The robot and its base are positioned at one side of a 3' x 5' table, across from the user, such that the user and robot share the tabletop workspace. We use an overhead ZED 2 RGB-D camera with known intrinsics and extrinsics. For robot control, we use a modified version of the DROID control stack based on Polymetis [61] . Low-level policies command joint positions at 10 Hz to a joint impedance controller which runs at 1 kHz. We implement two compliance modes: a stiff mode which is activated when the robot is executing a low-level skill, and a compliant mode for when the user provides a kinesthetic demonstration.Safety. We include multiple safeguards to ensure user safety. Users have the option to cancel any proposed behavior when an interpretable trace is presented with a physical Cancel button as described in §B.1 -this prevents execution and immediately backtracks the Vocal Sandbox system. Second, during execution of any low-level skill, the user can interrupt the robot's motion with this button as well. This halts the robot's motion and it immediately becomes fully compliant. Lastly, during user studies, both the user and proctor have access to the hardware emergency stop button which cuts the robot's power supply and mechanically locks the robot arm.a result, we needed to develop a "baseline" system that ensured the robot remains a viable partner at every point during the collaboration. This is why we minimally define a set of primitives that cover the full space of robot capabilities needed, as well as any interfaces (e.g., the GUI) the user might need to teach. In other words, viability and recovery are prerequisites for all systems we evaluate. Given these prerequisites, we looked at our hypothesis -specifically that learning at both the highand low-level through mixed modality feedback leads to more effective human-robot collaboration. We looked at prior works in collaborative task planning, approaches for handling on-the-fly language corrections, and methods for learning from other feedback modalities, identifying the following system "templates" to compare:1. "Static" Planning with Fixed Low-Level Skills. These systems define human-robot collaboration as translation from user instructions to a fixed set of predefined skills. This system is reflective of the core contributions of recent work such as Text2Motion [17] , ProgPrompt [18] , and MOSAIC [8] the latter of which we consider to be closest to ours.Crucially, these works use LMs to map user intent to realizable robot plans specified as programs.We specifically build on MOSAIC, as we believe this system to be the most general -it does not assume access to full-state information a priori (i.e., the LM is not a priori prompted with all objects in the scene), instead invoking a learned perception model to localize objects if possible (as we do; §3.2). Combining this with our "prerequisite" system, we get VS -(Low, High) -a system reflective of MOSAIC that uses an LM to translate language input to skills from a fixed library, some of which rely on the output of a learned perception module.2. On-the-Fly Corrections: "Adaptive" Planning with Fixed Skills. In works such as Inner Monologue [9] , Code as Policies [19] and Yell at Your Robot [YAY; 14], robot systems go beyond static planning. Instead, they exhibit forms of reactivity: while the sets of skills in these works remain static, the planners incorporate user or environment feedback (as corrections, action feasibility, subtask success) to synthesize plans. Of these works, YAY adds an iterative component, explicitly adding structured behaviors to retrain a high-level policy in batches (similar to how we induce new high-level behaviors during teaching). Adding this reactivity results in VS -Low -a system that combines InnerMonologue with the iterative updates of YAY -a reactive LM planner that conditions on user corrections or other feedback to iteratively learn new high-level behaviors to guide planning.3. Iteratively Improving Low-Level Skills. Finally, to explore the impact of adaptively growing our library of low-level skills learned from mixed-modality feedback we looked to prior work that learn skills from keypoint supervision as in KITE [31] and MOKA [65] , gestures as in GIRAF [12] , or as DMPs from kinesthetic demonstrations [30] . These works do not focus on collaborative multi-turn settings, instead proposing methods that are evaluated offline, given "static" input (e.g., evaluating success rate of a fully autonomous policy that is conditioned on gestures vs. not). As our contributions are focused more on the nature of collaboration when we allow users to teach new low-level skills, and not on the actual methods for skill learning, we use these methods as fixed components of our system. This results in our final Vocal Sandbox systems (VS) -where in our gift-bag assembly setting ( §4.1, VS specifies skills via a learned visual keypoints model, and in the stop-motion setting ( §4.2), VS specifies skills via DMPs. Note that for the keypoints model, we performed a static evaluation of different (pretrained) models in §B.3, finding our simple two-stream CLIP-based architecture to outperform existing open-vocabulary detectors like OWLv2 and foundation models like GPT-4V.Structuring our ablations this way gives us a fair, apples-to-apples comparison against prior works such as MOSAIC (VS -Low, High), InnerMonologue and YAY (VS 0 Low), while demonstrating our novel contributions for enabling continual learning at both the high-and low-levels.While the main body of our paper situates our framework against methods that use language models for task planning and learning low-level skills from multimodal feedback, Vocal Sandbox builds off a much larger body of work that build systems for different forms of human-robot interaction. The systems differ in the modes of collaboration they enable, from explicit human-robot teaming in situated environments [1, 4, 6, 7, 66] , to learned methods for shared autonomy [67] [68] [69] , to platforms for assistive robotics [70] [71] [72] , amongst many others [2] .While Vocal Sandbox is heavily inspired by this prior work, especially those that learn language interfaces for grounding user intent to low-level robot behavior [8, 73, 74] , this is only the beginning. Future iterations of our framework will build on the types of interactions and learning we permit (e.g., multi-robot teaming or integrating modalities such as touch or nonverbal feedback), all driving towards general and seamless human-robot collaboration.We present additional visualizations of the LEGO stop-motion animation experiment, including examples of user-taught camera motions and stills from the two hour interaction during filming. Wikipedia provides a thorough overview of the rich history and different styles of stop-motion animation, with a dedicated page on "Brickfilm" (e.g., using LEGOs) -the sub-genre we focus on in this work. 
 
 Appendix:  


Decision: **Paper summary**

This paper contributes a framework for users to provide mixed-modality feedback (language, demonstrations, and labels) to robots in order to learn/improve both high-level and low-level behaviors. The paper contributes methods for integrating multi-modal feedback and providing transparency to the user about its behavior. 

**Review summary**

Summary of strengths:
+ The paper addresses a key problem for human-robot collaboration: adaptation based on human feedback.
+ The proposed method is evaluated with a user study with non-experts.
+ The proposed method makes uses of multi-modal interactions, and allows a robot to learn both low-level and high-level information about a task.

Summary of weaknesses:
- More detail is needed about how users can (and cannot) interact with the interface at each step. Furthermore, the reviewers have noted several opportunities to highlight the users' experiences with system failures, approving/disapproving the robot's behavior, and preferences between modalities. **[Edit: The rebuttal now includes more details about the interface and user comments.]**
- The evaluation is missing comparisons to non-ablation baselines. **[Edit: The rebuttal clarifies that the ablated versions of the system are representative of SOTA baselines.]**
- The evaluation is missing detail about the user study participants and their familiarity with robots. **[Edit: The rebuttal clarifies the backgrounds of the participants.]**


**Response to rebuttal**

The authors all maintain their positive rating, and one reviewer has increased their rating from weak to strong accept. The reviewers agree that the rebuttal addresses their concerns. However, the clarifications have not yet been integrated into a revision.

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper introduces a framework for human-robot interaction that allows a user to continuously provide corrective and instructional feedback across multiple modalities and hierarchies. The proposed framework allows a user to teach both low-level skills, which can be taught via visual keypoints (conditioned policy) or kinesthetic demonstrations (DMP), and high-level behaviors taught via language. This framework also allows for corrective feedback in the event the system's plan does not match user expectations. The framework is evaluated through a user study in which the robot assembles a gift bag where it out-performs ablated methods in both subjective and objective metrics.
main_review: main_review: Strengths:
* Developing human-robot interaction frameworks which allow continuous adaptation are very important for fluid collaboration. Even in the era of large data-based foundation models, we can't expect pre-trained policies/frameworks to have perfect performance in all tasks that may be encountered in real world settings. I think the approach taken here which attempts to incorporate multiple feedback modalities across multiple levels of abstraction is very promising.

* The paper is easy to read and the provided examples are fairly intuitive. Given the complexity of the framework though, I feel like it might benefit from some sort of flow chart-based figure that indicates all the options available to a user to provide feedback.

Weaknesses:
* Aspects of the methodology are unclear and so it is hard to tell how useful this framework is in practice. For example, in Sec. 2.1 it is stated that the language model infers the parameters of skills being taught. But it's not clear a) whether the user can correct/override the proposed parameters, b) how these parameters are specified when providing a language instruction (does the user need to say "pan around slowly with N=30 frames" or does the language model infer parameters?), and c) how a user can correct parameters if they are inferred and are incorrect. Similarly, can a user provide corrections to behaviors purely via language or do they always have to do so manually through the GUI?\
What I am trying to get at is that it's unclear to me how much utility the framework (as proposed) offers as a whole as it's not clear how much user involvement is required at each step. For example, at risk of being overly reductive, is this a glorified GUI which simply exposes convenient utilities for training additional low-level skills and exposing them to a language model-based planner or is there something more useful/novel here?

* I think the paper would benefit from further elaboration/analysis on the complexity of user-defined behaviors. For example, Fig. 4 shows the complexity of behaviors increasing as users perform more tasks, but what do these look like? I am curious about things like how easy it is to modify existing behaviors. For example, if I am assembling gift bags using the behavior in Fig. 3, can I give an instruction like "Assemble another gift bag but also pack the book this time" which then adds another pack_object behavior to the end? Or does this require user intervention with the creation of a new behavior?

* For the amount of emphasis placed on mixed-modality feedback, there is very little discussion on how well it works. I don't see any analysis on feedback modalities in the experimental results section. Do users prefer one modality over another? When is each modality preferred? Is there value in even providing multiple modalities, as opposed to existing approaches which only use one? Does this depend on the underlying model/task and the type of mistakes that might occur?
questions: questions: 1. Can the user correct/override the proposed skill parameters?

2. How are these parameters specified when providing a language instruction (does the user need to say "pan around slowly with N=30 frames" or does the language model infer parameters)? How can a user correct parameters if they are inferred and are incorrect?

3. Can a user provide corrections to behaviors purely via language or do they always have to do so manually through the GUI? I am wondering whether language is a feedback modality or simply used only for instructions (and therefore to define new skills/behaviors).

4. What do complex user-defined behaviors look like? Can a user modify existing behaviors (see example above with gift bag)?

5. Do users prefer one modality over another? When is each modality preferred? Is there value in even providing multiple modalities, as opposed to existing approaches which only use one? Does this depend on the underlying model/task and the type of mistakes that might occur?

6. Can a user modify an existing skill? For example, let's suppose that the user already taught the robot the "camera pan" skill. Can the user modify this skill to either add an additional parameter or create some sort of variation of it? Or do they have to create a completely new skill from scratch?

7. Does "(w/o Low, High)" in the experiments indicate that indicate that users can teach neither high-level behaviors nor low-level skills? The text states high-level only but that seems odd given the name.
limitations: limitations: 3
review_summary: review_summary: I think this paper is interesting and addresses a timely and relevant problem, but I do have some concerns about the methodology itself and am unsure how useful the proposed framework actually is.
title: title: Review of Submission 599
paper_summary: paper_summary: This paper Vocal Sandbox (abbreviated as VS), a framework where users can teach both new high-level planning behaviors through spoken dialogue and provide teaching feedback to learn new low-level skills in real-time. The authors evaluate this framework in a user study with 8 non-expert participants and find that the system outperforms baselines both objectively and subjectively. Finally, the authors conclude by showing a human-robot collaboration example where the human and robot create a 30-second stop motion animation, an impressive feat.
main_review: main_review: Overall, this paper is very well-written, contains an impressive demonstration of a few combined techniques, and is validated via a user study.


Strengths
+ This paper combines several techniques well and allows for a new capability, improving human-robot collaboration.
+ The robot demonstration to create the 30-second stop-motion animation is impressive. It would be nice to attach this short film outside of the 3-minute as a standalone supplementary file. 
+ This paper is well-written and contains sufficient information to understand the proposed approach.




Weaknesses
- There has been prior work (https://www.roboticsproceedings.org/rss18/p028.pdf) in humans teaching abstractions to robots via learning from demonstration. This paper's results display that humans have difficulty teaching robots different skills and behavior without oversight. Could you comment on why/how your approach and experiment lead to this capability?
- The term "expert" is used to describe the collaborator for the stop motion animation. As this "expert" is an author of this work, I would recommend grounding the expertise needed to perform this type of human-robot collaboration. Could you also comment on the expertise of the users in the human-subject study? Were these graduate students studying robotics and if so, are these results limited to this set of users?

Post-Rebuttal - The author's rebuttal was well-written and comprehensive. I appreciate the careful description regarding VS's relation to the results in Gopalan et al., the clarification regarding user expertise, and the anecdote about a user's experience with the proposed system. After reading all the reviews and their respective replies (and further discussions), I decided to maintain my score of Weak Accept.
questions: questions: - Please address the questions in the weaknesses section of the main review.

Further Questions: 
- How do you ensure the learned low-level skills generalize well to other objects? 
- Did users within the human-subject study need to reteach skills at any point or delete learned skills for any reason?
limitations: limitations: 3
review_summary: review_summary: Overall, this paper is very well-written, contains an impressive demonstration of a few combined techniques, and is validated via a user study. The contribution is incremental and a step forward for HRI, combining different techniques into a system for human-robot collaboration.
title: title: Submission599
paper_summary: paper_summary: Summary  This paper presents Vocal Sandbox, a framework for users to teach high-level behaviors through mixed-modality interactions. The framework enables users to give feedback and instruction through multiple modalities online during an collaboration. The Sandbox interface is over high-level behaviors and low-level skills. A high-level LLM planner maps language utterances to behavior plans, and learned low-level policies grounds these plans to robot actions. The low-level skills are trained through DMPs and keypoint-conditioned learned policies.  Online, during the interaction, the user gives the robot a natural language command (ie. track the hulk). The robot explains its failure, deferring to the user for choosing a next step. The user can either move on, or explicitly teach the robot what the new skill of tracking is. When teaching the new skill, the human defines a new behavior function, which can be composed of requisite skills. If a low-level skill is needed, the user can provide a demonstration. If a perceptual concept is needed, the user can label object segmentation masks. Finally, the system provides a visualization of the resultant robot trajectory.  Vocal Sandbox uses a interface for users to teach new skills, perceptual concepts, and visualize robot traces during teaching.  The sandbox is tested across two tasks, gift bag assembly across multiple non-expert participants, and stop motion animation with an expert user. Users strongly preferred the system over its ablated baselines in ease of user, helpfulness, and performance.
main_review: main_review: Strengths

This is a very interesting paradigm and learning from online, multi-modal human interventions is an important research direction. The framework presented offers very promising system insights and results on favorable perceptions from novice users.

Weaknesses

The two baselines in this work are ablations of the system itself, each of which remove capabilities of the overall VS system. When the ability to correct skills and plans is removed, it is not surprising that users found the system easier to use, intuitive, and more performant. It would be effective to compare to existing works that combine learning from multiple modalities (ie. on-the-fly language corrections and learning from different feedback types).  

Comments/Questions

How accurate is the skill trace? Is the learned robot skill demonstrated in simulation, and it would be great to show a snapshot of the GUI depicting how the robot trajectory looks while the trace is shown on screen when the user evaluates the trace. How often did the user disapprove of the skill trace? Is the user able to go back in and resume training if they disapprove of the skill?

What were the common failure modes when it comes to the ways users interacted with the system? Despite outperforming the baselines, did users struggle with any aspects of the VS system in particular. It would be excellent to elaborate qualitatively on why users found the system to be helpful and intuitive, and what parts of the system were not.

Why did VS outperform the VS (w/o Low, High) baseline on all measures except predictability and trust? Do you have any hypothesis as to why this is the case? Is the GUI driving predictability (via traces) and trust?
questions: questions: Co-adaptation is presented in the abstract as one of the main motivations of the visualization traces. What signs of co-adaptation were seen in the user study? How were users adjusting to the visualization traces? Did you notice shift in their teaching strategies over time? 

Did users see a monotonic increase in performance as a result of further teaching? Where there instances where this was not the case?

Please see above section for further questions.
limitations: limitations: 2
review_summary: review_summary: Overall, Vocal Sandbox presents a system for online behavior corrections to robots via multi-modal feedback. This is a very interesting paradigm and learning from online, multi-modal human interventions is an important research direction. The framework presented offers very promising system insights and results on favorable perceptions from novice users. Aspects of the evaluation and system performance can be further analyzed, but overall I recommend a weak accept.